How the Notebook Collab & Save Reliability Dashboard works

A weekly reliability snapshot for Fabric Notebook collaboration join, save success, and save-leader yield. Auto-generated by an Azure DevOps pipeline every Tuesday morning Beijing time and deployed here. This page explains where the data comes from, what each section means, and how to read it.

Refresh: weekly · Tue 03:00 UTC+8 Window: previous Mon–Sun week (UTC) Environment: PROD Data: Kusto telemetry

📊 Open the latest dashboard 📚 Browse the weekly archive

What does it measure?

Three reliability surfaces are tracked side-by-side so you can spot regressions and correlate failures between layers:

Collaboration Reliability — join success rate, session reliability, content consistency between collaborators, and per-component-version slices.
Save Reliability — save success rate (3 different denominators), where saves fail (frontend vs backend, error-type breakdown), and an HTTP-500 deep-dive.
Save-Leader Yield — when the active save-leader yields, who picks it up, whether the new leader's save succeeds within 20 seconds, and why yields fail.

When is it refreshed?

Cron 0 19 * * 1 — Mondays 19:00 UTC = Tuesdays 03:00 Beijing. Each run regenerates the dashboard from the most-recently-completed Monday→Sunday week (UTC).

The pipeline runs on Microsoft-hosted Linux agents (ubuntu-latest), takes about 3–5 minutes end-to-end, and writes both the latest snapshot and a permanent per-week archive entry. You can also manually trigger a refresh from the pipeline page if a week's data needs to be re-run.

Where does the data come from?

The dashboard issues 16 KQL queries against three Kusto clusters:

Cluster	Database	What it provides
`pbiclients.eastus`	`appinsights`	Frontend (DES extension) telemetry: collab join events, save outcomes, yield-leader decisions.
`pbiclientseu.northeurope`	`appinsights`	Same shape as eastus, EU region. Unioned in to give global coverage.
`azurenb.southcentralus`	`Prod`	Backend (aznbServices) telemetry: server-side save outcomes, HTTP status codes, error messages.

Authentication is via the trident-pipeline-service-connection workload identity in msdata/A365, which has read access on the three databases above. There are no credentials stored in code.

All 16 queries are also visible (and copy-pasteable) at the bottom of every dashboard page under the Source Queries (KQL) card.

How to read each section

Weekly Verdict (top of the page)

Big traffic-light banner with the week-over-week delta on the headline metric. Green = healthy (above threshold), Yellow = watch (mild degradation), Red = action required.

Collaboration Reliability

Daily Collaboration Trends shows two line charts — Collab Join Success Rate (clients that successfully joined / clients that attempted) and Collab Session Reliability (sessions that completed without error / sessions started). Look for sudden drops on specific days.

Collab Join Rate by Component Version slices the join rate per shipped DES version, so a regression introduced by a deployment shows up as one version row diverging from the others.

Collab Content Consistency tracks how many instances emit a dataIntegrity/error event — i.e., the collaborative state diverged between clients. Uses a 14-day window (longer than the rest) because these events are rare and need more data to be meaningful. The table below the chart lists the top inconsistency messages.

Save Reliability

Save Success Rate (3 views) renders the same week three ways: by attempts, by instances, and by clients. If only one view drops, it usually points at a long-tail of retries from a small population.

Where is the failure? A sortable table classifying failures by error-type family. Click the column header to sort. Rows tagged SystemError are highlighted because they're the most actionable.

HTTP 500 Deep-Dive isolates server-side 500 errors specifically, which are almost always the most actionable signal for the backend team. Top error messages with counts and affected clients.

Failure Chain (Frontend → Backend) joins the same instance's frontend save-failure event to its backend log entry within a short time window, and shows a small sample of the latest error message text so the on-call can paste it straight into a search.

Save-Leader Yield

In collaboration sessions one client is the "save leader" responsible for writing to backend. After repeated save failures while joined, it can yield to another client. This whole section tells you whether that yield mechanism is healthy.

How the decision works — short decision tree explaining when yield fires.
Yield Outcome Summary — count of yields by outcome (succeeded / failed / not-needed).
Successful saves on new leader (≤ 20 s after yield) — proves the yield actually rescued the user. Includes uniqueness columns so you can tell 50 events from 50 clients apart from 50 events from 1 retrying client.
Why yields fail — top error messages when the yield itself errors out.
Save failures on new leader (≤ 20 s after yield) — yields that succeeded but the new leader's subsequent save also failed. These rows include unique-notebook and unique-instance columns to make retry storms easy to spot.

Cross-Week Trends (archive only)

The archive page overlays the last several weeks of each headline metric on a single sparkline, so multi-week regressions are obvious at a glance.

How is it built?

Python generator (generate.py) — opens an authenticated Kusto session via az CLI, fires the 16 queries, and merges the results with the previous week's data so deltas and sparklines can be computed locally.
HTML renderer (render.py) — pure-Python templating; produces a single self-contained HTML file with inline SVG charts (no JS frameworks, no external assets). Easy to inspect via View Source.
Azure DevOps pipeline (.azure-pipelines.yml) — cron-scheduled every Monday 19:00 UTC, restores prior history from a pipeline artifact, runs the generator, and publishes the output to this Static Web App.
Static Web App — same site as sharing.notebook.windows-int.net; AAD-gated so only Microsoft accounts can view.

Where's the code?

📂 Repo	dev.azure.com/msdata/A365/_git/notebook-reliability-dashboard
🚀 Pipeline	notebook-reliability-weekly (id 56355)
🔗 Service connection	`trident-pipeline-service-connection` (WIF; pre-configured Kusto reader)

Running it locally

The generator is fully runnable on a dev box if you want to experiment without waiting for the weekly cron:

git clone https://dev.azure.com/msdata/A365/_git/notebook-reliability-dashboard
cd notebook-reliability-dashboard
pip install -r requirements.txt
az login            # uses your AAD identity, must have Kusto reader on the 3 clusters above
python generate.py --source live
# → output/collabsave_dashboard_latest.html

Add --source sample to render against the committed sample data (no Kusto access needed) for layout work.

Customizing the dashboard

You want to…	Edit this
Add a new metric	Add `q_<name>()` in `queries.py`, append to `ALL_QUERIES`, render in `render.py`
Change a threshold (healthy / watch)	`HEALTHY_THRESHOLD` / `WATCH_THRESHOLD` in `render.py`
Tune the action playbook	`ACTION_PLAYBOOK` dictionary in `render.py`
Move the weekly boundary	`_last_completed_week_utc()` in `generate.py`
Add a new chart shape	Clone `_svg_line_chart()` / `_svg_multi_line_chart()` in `render.py`

Heads up: changes pushed to main take effect the next time the cron fires (Tue 03:00 Beijing). To validate sooner, hit Run pipeline on the pipeline page after your push.

Troubleshooting

Symptom	Likely cause / fix
Dashboard hasn't refreshed this Tuesday	Check the pipeline runs page. Look for a build dated this week. If none, the schedule may not have fired (cluster availability, repo perms). Click Run pipeline manually.
A specific KQL query fails	The pipeline log shows the query name and error. Most often a Kusto cluster timeout — re-run the pipeline; the 20-minute request timeout is usually enough.
SWA deploy step fails	The deploy token (`SWA_TOKEN` pipeline variable) may have rotated. Regenerate via `az staticwebapp secrets list -n notebook-skill-docs -g ltianrg` and update the variable.
"Page not found" on this site	The SWA enforces AAD auth — make sure you're signed in with your Microsoft account.

Who owns this?

Maintained by the Fabric Notebook team. For changes, file a PR against msdata/A365/notebook-reliability-dashboard. For data issues (a metric looks wrong, a query is misclassifying failures), open a bug in the same repo with a screenshot and the affected week.