🔍 Diagnose with Copilot — Usage Dashboard

The Diagnose with Copilot button is shown when three conditions are all true: an active Spark session, the cell run failed, and the user has Copilot permission. This dashboard tracks how often the button is shown, clicked, and the AI-generated fix reverted.
Numbers are aggregated across both standardized-report clusters — pbiclients.eastus (US) and pbiclientseu.northeurope (EU).

Window: — · Clusters: US + EU · Source: CopilotFixErrorButton.tsx · Companion: /fix End-to-End Eval Report → · ← back to docs

Executive summary (last 30 days)

Button shown

—

failed Spark cells where the user has Copilot permission

Button clicked

—

Click-through rate

—

share of impressions that resulted in a click

Fix reverted (Undo ≤10 min)

—

Active clickers

—

Notebooks reached

—

distinct notebooks where the button was shown

Tenants reached

—

Avg clicks per day

—

Daily trends

How usage changes day by day

Each chart is interactive — hover any point for the exact value, and click a series name in the legend to hide or show it. Weekends typically show lower counts (fewer Spark sessions).

Events per day

Impressions (left axis) vs Clicks & Undos (right axis).

Tip: click Impressions in the legend to focus on the click vs undo trend.

Unique users per day

Distinct users who saw, clicked, or undid the fix.

The gap between "saw" and "clicked" is the conversion opportunity.

Unique notebooks per day

Distinct notebooks where the button was shown / interacted with.

Notebooks where the button was shown only — click/undo counts are not broken down per notebook (telemetry limitation).

Unique tenants per day

Distinct customer tenants reached.

Compare reach across enterprises — how broadly the feature is exposed each day.

Click outcomes

What happens after a click?

Outcome	Events	% of clicks	Share

Undo events are counted as reverts of a Copilot fix only when an UndoChanges activity happens within 10 minutes of a FixWithCopilot click in the same browser tab.

Top tenants by clicks

Which customers use it most?

Tenant	Clicks	Users

Customer tenants are anonymous IDs from SCR.CustomerTenantId. 72f988bf… is Microsoft's own tenant (dogfood usage).

Why fixes fail

Failure analysis — —

—

Failure rate

—

Median time-to-fail

—

most failures hit the 60 s chat-pane-ready timeout

Median time-to-succeed

—

successful clicks dispatch in well under a second

🔍 Top finding. —

Failure breakdown by likely cause

Stacked daily failures grouped by duration profile. Click a series in the legend to hide it.

Tip: hide Chat-pane init timeout to see the underlying trend of all other failure modes.

Share of failures by cause

Each bucket as a share of the {{failTotal}} failed clicks in the window.

A handful of root causes dominate — mostly chat-pane initialization and downstream LLM errors.

Likely cause	Failures	% of fails	Users	Tenants	What it means / where it comes from

How we classify a failure. The FixWithCopilot activity does not record an error code in ResultCode or ActivityAttributes — only DurationMs. The good news: the code path has well-defined timeout boundaries (60 s for chat-pane init, plus the LLM call latency window), so duration buckets map cleanly to root causes. See actionUtils.ts — functions sendFixCellMessageIfCan, sendChatPaneCellMessageIfCan, sendChatPaneMessage.

Failure rate by operating system

OS	Clicks	Failed	Fail %

Recommended next steps

Instrument the error reason — pass the caught error.message / error.name into endActivity as a custom property so we can group exactly.
Investigate the 60 s timeout — profile chatPaneState transitions, identify the long-tail bundle loads / AAD token calls.
Consider a "still loading…" UX — surfacing progress instead of failing silently after 60 s.
Cap retry-able LLM errors — for the 24% post-ready failures, surface a "try again" affordance instead of an opaque error.

Where is it being shown?

Impressions by environment

Environment	Impressions	Distinct users	Distinct notebooks	Share	30-day distribution

PROD = public production. MSIT = Microsoft internal. DAILY/DXT/INT/TEST = pre-production rings. Clicks & undos are PROD-only in our telemetry, so the env split applies to impressions.

Reach & dispersion

Funnel and unique-counts

Stage	Events	Distinct users	Distinct notebooks	Distinct tenants	Distinct browser tabs

A single user can show in all three stages with different counts — this is a funnel, not a hard subset. Workspace / capacity dimensions are intentionally omitted because TL_capacityId is always "1" in the underlying dataset (an upstream telemetry quirk).

⚠ How impressions are measured. The button has no direct impression telemetry. We approximate impressions by counting failed Spark cell runs where all three render conditions are met: statementMeta != null (Spark session); the cell run finished with state == 'failed'; and the user has Copilot permission for that artifact / hour — rejecting tabs that ran Python / T-SQL in the same hour, and excluding (artifact, hour) windows where LLMEndpointTenantSettings.checkAccess logged a Disallowed, DisallowedForCrossGeo, or final license-check failure. The result is close to the true impression count; the residual error comes from the cell-level canShowFixButton(…) visibility check, which is not re-applied at query time. For an exact count, see the Q7 · Path A recipe below.

Reproducible queries

Re-validate these numbers in Kusto

Clusters pbiclients.eastus + pbiclientseu.northeurope · Database appinsights.
Standardized-report and App-Insights data are partitioned regionally. Each query below uses union across both clusters so totals match the dashboard. Click any tab to view the exact KQL.

Q1 · Impressions (Spark + permission)

Q2 · FixWithCopilot clicks

Q3 · Fix → Undo (10 min)

Q4 · Impressions by env

Q5 · Top tenants

Q6 · Failure analysis

Q7 · Path A (exact impressions)

// === Path B: approximate "Diagnose with Copilot" impressions (Spark + permission, 30 d) ===
// Standardized-report data is split between two regional clusters. Use the union form
// so the query covers both. If the cross-cluster channel is unavailable for your
// principal, run the query against each cluster separately and sum the result.
let lookback = 30d;
let scr =
    union
        cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
        cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting;
let ev =
    union
        cluster('pbiclients.eastus').database('appinsights').customEvents,
        cluster('pbiclientseu.northeurope').database('appinsights').customEvents;
let failedCells =
    ev
    | where timestamp > ago(lookback)
    | where name == 'DiagnosticTrace'
    | extend
        componentName = tostring(customDimensions.TL_componentName),
        message       = tostring(customDimensions.TL_message),
        state         = tostring(customDimensions.TU_state),
        artifactId    = tostring(customDimensions.TU_artifactId),
        tabId         = tostring(customDimensions.browserTabId),
        tenantId      = tostring(customDimensions.customerTenantId)
    | where componentName == 'Notebook.NotebookFileView'
    | where message == 'Finish to run Notebook code cell'
    | where state == 'failed'
    | extend hourBucket = bin(timestamp, 1h);
let sparkTabHours =
    scr
    | where Timestamp > ago(lookback)
    | where ActivityName == 'RunNotebookCodeCell'
    | extend hourBucket = bin(Timestamp, 1h)
    | distinct BrowserTabId, hourBucket;
let nonSparkTabHours =
    scr
    | where Timestamp > ago(lookback)
    | where ActivityName in ('RunPythonNotebookCodeCell', 'runTSQLNotebookCodeCell')
    | extend hourBucket = bin(Timestamp, 1h)
    | distinct BrowserTabId, hourBucket;
// (artifact, hour) windows where the Copilot license check confirmed "no access"
// for the user. The button cannot render in these windows.
// Note: FullAccess is not logged — only the Disallowed / region-limited / final-error cases
// are written, so this list is the precise set of windows we need to exclude.
let noCopilotArtifactHours =
    ev
    | where timestamp > ago(lookback)
    | where tostring(customDimensions.TL_extensionName) == 'DES'
    | where tostring(customDimensions.TL_componentName) == 'LLMEndpointTenantSettings.checkAccess'
    | extend level   = tostring(customDimensions.TL_level),
             message = tostring(customDimensions.TL_message)
    | where (level == 'warning' and message in ('Copilot is currently limited to certain regions for your organization',
                                                 'Valid copilot license, but one or more models are disallowed'))
         or (level == 'error'   and message == 'failed to check copilot license')
    | extend artifactId = tostring(customDimensions.TU_artifactId),
             hourBucket = bin(timestamp, 1h)
    | where isnotempty(artifactId)
    | distinct artifactId, hourBucket;
failedCells
| join kind=leftsemi sparkTabHours        on $left.tabId == $right.BrowserTabId, hourBucket
| join kind=leftanti nonSparkTabHours     on $left.tabId == $right.BrowserTabId, hourBucket
| join kind=leftanti noCopilotArtifactHours on artifactId, hourBucket
| summarize
    Impressions = count(),
    Users       = dcount(user_Id),
    Notebooks   = dcount(artifactId),
    Tabs        = dcount(tabId),
    Tenants     = dcount(tenantId)
    by dayBucket = bin(timestamp, 1d)
| order by dayBucket asc

// === FixWithCopilot clicks (30 d), union of both standardized-report clusters ===
union
    cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
    cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting
| where Timestamp > ago(30d)
| where ActivityName == 'FixWithCopilot'
| where FeatureName  == 'NotebookCopilot'
| summarize
    Clicks    = count(),
    Succeeded = countif(ActivityStatus == 'Succeeded'),
    Failed    = countif(ActivityStatus == 'Failed'),
    Users     = dcount(ExecutingUserObjectId),
    Tenants   = dcount(CustomerTenantId),
    Tabs      = dcount(BrowserTabId)
    by dayBucket = bin(Timestamp, 1d)
| order by dayBucket asc

// === Fix -> Undo correlation (10-min window, 30 d) ===
let lookback = 30d;
let scr =
    union
        cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
        cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting;
let fixes =
    scr
    | where Timestamp > ago(lookback)
    | where ActivityName == 'FixWithCopilot' and FeatureName == 'NotebookCopilot'
    | project FixTime = Timestamp, BrowserTabId, FixUser = ExecutingUserObjectId, FixTenant = CustomerTenantId;
let undos =
    scr
    | where Timestamp > ago(lookback)
    | where ActivityName == 'UndoChanges' and FeatureName == 'NotebookCopilot'
    | extend UndoId = strcat(BrowserTabId, '|', tostring(Timestamp))
    | project UndoId, UndoTime = Timestamp, BrowserTabId, UndoUser = ExecutingUserObjectId, UndoTenant = CustomerTenantId;
undos
| join kind=inner fixes on BrowserTabId
| where FixTime <= UndoTime and datetime_diff('minute', UndoTime, FixTime) <= 10
| summarize arg_max(FixTime, FixUser, FixTenant) by UndoId, UndoTime, BrowserTabId, UndoUser, UndoTenant
| summarize
    UndoAfterFix = count(),
    Users        = dcount(UndoUser),
    Tabs         = dcount(BrowserTabId),
    Tenants      = dcount(UndoTenant)
    by dayBucket = bin(UndoTime, 1d)
| order by dayBucket asc

// === Impressions by env (Spark + permission, 30 d) ===
let lookback = 30d;
let scr =
    union
        cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
        cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting;
let ev =
    union
        cluster('pbiclients.eastus').database('appinsights').customEvents,
        cluster('pbiclientseu.northeurope').database('appinsights').customEvents;
let failedCells =
    ev
    | where timestamp > ago(lookback)
    | where name == 'DiagnosticTrace'
    | extend
        componentName = tostring(customDimensions.TL_componentName),
        message       = tostring(customDimensions.TL_message),
        state         = tostring(customDimensions.TU_state),
        artifactId    = tostring(customDimensions.TU_artifactId),
        tabId         = tostring(customDimensions.browserTabId),
        env           = tostring(customDimensions.TL_environment)
    | where componentName == 'Notebook.NotebookFileView'
    | where message == 'Finish to run Notebook code cell' and state == 'failed'
    | extend hourBucket = bin(timestamp, 1h);
let sparkTabs    = scr | where Timestamp > ago(lookback) | where ActivityName == 'RunNotebookCodeCell' | extend hourBucket = bin(Timestamp, 1h) | distinct BrowserTabId, hourBucket;
let nonSparkTabs = scr | where Timestamp > ago(lookback) | where ActivityName in ('RunPythonNotebookCodeCell','runTSQLNotebookCodeCell') | extend hourBucket = bin(Timestamp, 1h) | distinct BrowserTabId, hourBucket;
let noCopilotArtifactHours =
    ev
    | where timestamp > ago(lookback)
    | where tostring(customDimensions.TL_extensionName) == 'DES'
    | where tostring(customDimensions.TL_componentName) == 'LLMEndpointTenantSettings.checkAccess'
    | extend level = tostring(customDimensions.TL_level), message = tostring(customDimensions.TL_message)
    | where (level == 'warning' and message in ('Copilot is currently limited to certain regions for your organization',
                                                 'Valid copilot license, but one or more models are disallowed'))
         or (level == 'error'   and message == 'failed to check copilot license')
    | extend artifactId = tostring(customDimensions.TU_artifactId), hourBucket = bin(timestamp, 1h)
    | where isnotempty(artifactId)
    | distinct artifactId, hourBucket;
failedCells
| join kind=leftsemi sparkTabs              on $left.tabId == $right.BrowserTabId, hourBucket
| join kind=leftanti nonSparkTabs           on $left.tabId == $right.BrowserTabId, hourBucket
| join kind=leftanti noCopilotArtifactHours on artifactId, hourBucket
| summarize Impressions = count(), Users = dcount(user_Id), Notebooks = dcount(artifactId) by env
| order by Impressions desc

// === Top tenants by FixWithCopilot clicks (30 d), both clusters ===
union
    cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
    cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting
| where Timestamp > ago(30d)
| where ActivityName == 'FixWithCopilot' and FeatureName == 'NotebookCopilot'
| where isnotempty(CustomerTenantId)
| summarize Clicks = count(), Users = dcount(ExecutingUserObjectId), Tabs = dcount(BrowserTabId) by CustomerTenantId
| order by Clicks desc
| take 10

// === FixWithCopilot failure analysis (30 d), both clusters ===
let lookback = 30d;
let scr =
    union
        cluster('pbiclients.eastus').database('appinsights').StandardizedClientReporting,
        cluster('pbiclientseu.northeurope').database('appinsights').StandardizedClientReporting;
scr
| where Timestamp > ago(lookback)
| where ActivityName == 'FixWithCopilot' and FeatureName == 'NotebookCopilot'
| where ActivityStatus == 'Failed'
| extend bucket = case(
    DurationMs <= 5000,    'A. Quick failure (\u22645s)      \u2014 synchronous throw',
    DurationMs <= 50000,   'B. Backend failure (5-50s)   \u2014 sendMessage rejected early',
    DurationMs <= 70000,   'C. Chat-pane init timeout (~60s)',
    DurationMs <= 180000,  'D. sendMessage failure (70-180s)',
    DurationMs <= 600000,  'E. Stuck request (3-10 min)',
                           'F. Extreme (>10 min)')
| summarize
    Failures = count(),
    Users    = dcount(ExecutingUserObjectId),
    Tenants  = dcount(CustomerTenantId)
    by bucket
| order by bucket asc

// Duration overview: median ok vs median fail (the 60s spike is unmistakable)
scr
| where Timestamp > ago(30d)
| where ActivityName == 'FixWithCopilot' and FeatureName == 'NotebookCopilot'
| summarize
    Events  = count(),
    p50_ms  = percentile(DurationMs, 50),
    p90_ms  = percentile(DurationMs, 90),
    p99_ms  = percentile(DurationMs, 99)
    by ActivityStatus

// Per-OS failure rate
scr
| where Timestamp > ago(30d)
| where ActivityName == 'FixWithCopilot' and FeatureName == 'NotebookCopilot'
| summarize
    Clicks = count(),
    Failed = countif(ActivityStatus == 'Failed')
    by OperatingSystemName
| extend FailPct = round(100.0 * Failed / Clicks, 2)
| where Clicks >= 10
| order by Clicks desc

// === Path A: exact impression query (after instrumentation) ===
//
// Inside packages/@synapse/spark-notebook-extensions/src/components/cellMonitor/CopilotFixErrorButton.tsx:
//
//   const logger = getLogger('CellMonitor.CopilotFixErrorButton');
//   const showFixButton = useMemo(() => {
//       const visible = isCopilotEnabledForUser
//           && statementMeta?.statementId
//           && copilotState !== CopilotState.Hidden
//           && canShowFixButton(...);
//       if (visible) logger.event('CopilotFixButtonShown', { artifactId, cellId });
//       return visible;
//   }, [...]);
//
union
    cluster('pbiclients.eastus').database('appinsights').customEvents,
    cluster('pbiclientseu.northeurope').database('appinsights').customEvents
| where timestamp > ago(30d)
| where name == 'DiagnosticTrace'
| extend componentName = tostring(customDimensions.TL_componentName),
         message       = tostring(customDimensions.TL_message),
         env           = tostring(customDimensions.TL_environment)
| where componentName has 'CellMonitor.CopilotFixErrorButton'
| where message == 'CopilotFixButtonShown'
| summarize Impressions = count(), Users = dcount(user_Id) by env, dayBucket = bin(timestamp, 1d)
| order by dayBucket asc, env asc

Methodology

How these numbers are produced

Show logic. The button renders when three conditions are all true (see CopilotFixErrorButton.tsx): (a) a Spark session is attached — statementMeta != null; (b) the cell run failed — canShowFixButton(…) sees an output error or Spark job failure; and (c) the user has Copilot permission — isChatEnabled() returns true, gated by LLMEndpointTenantSettings.checkAccess.
Impressions = proxy. The component does not currently emit a "shown" event, so we count failed Spark cell runs in browser tabs that ran Spark in that hour (rejecting Python / T-SQL tabs), and then exclude (artifact, hour) windows where checkAccess logged a Disallowed, DisallowedForCrossGeo, DisallowedForStoreDataCrossGeo, or final license-check error. Because the FullAccess outcome is intentionally not logged, the negative-log list is the exact set we need to subtract.
Permission filter is artifact-scoped. Copilot license is checked per artifact (per capacity / region). The permission filter rejects only the specific (artifact, hour) where the license check returned a non-FullAccess outcome — a user who has Copilot on one workspace but not another is still counted on the permitted workspace.
Clicks. FixWithCopilot activity in StandardizedClientReporting (emitted by sendFixCellMessageIfCan in apps/de-ds-extension/src/chat/actionUtils.ts).
Undos. UndoChanges activity (emitted by handleUndoChanges in apps/de-ds-extension/src/chat/authoring/chatParts/chatChanges/changesHandlers.ts). Counted only when the undo happens within 10 minutes of a Fix in the same tab — tighter than 1 hour to better reflect "the user immediately walked back the AI's change".
Env coverage. StandardizedClientReporting only holds PROD data, so clicks / undos are PROD-only. The env breakdown applies to impressions (which come from customEvents).
Cross-cluster aggregation. Standardized-report and customEvents data is partitioned across two regional clusters — pbiclients.eastus (US) and pbiclientseu.northeurope (EU). Every metric on this page combines both. Counts (clicks, impressions, undos) are summed exactly. Distinct-counts (users, tenants, notebooks) are summed across clusters — a user who works in both regions will be double-counted, but regional affinity makes the resulting over-count empirically < 1% for our 30-day windows.