Programmatic reproduction of the "Diagnose with Copilot" button in Fabric Notebooks.
Each case runs a failing cell against a live Spark kernel, captures the real outputs, sends
/fix to the spark-diagnostics agent over the
Notebook Service WebSocket, and persists the full streamed response. The wire format mirrors
trident-de-ds-app/getCopilotContext() exactly.
copilot_fix_cell() helper and the new slash_command: fix case schema in nb-eval.4b351b50-d95b-4a72-b184-3b5101880166 · 4 cases · 0 hard-failures · avg 10.2s per case.89663e7e-fd1a-4303-8912-1b10ee21676a).
① Input — the failing cell as the user sees it.
② Captured cell outputs — what the Spark kernel returned (statement-meta + error + traceback).
③ Backend context — the exact WebSocket payload sent to the spark-diagnostics agent.
④ LLM response — the rendered markdown streamed back by the agent, plus any tool calls.
Simulates clicking "Diagnose with Copilot" on a Python cell that references an undefined variable. Triggers a plain Python NameError (no Spark statement-meta). The runner captures the real outputs + traceback and sends /fix to the spark-diagnostics agent to verify how the agent handles a non-Spark error.
The cell that the user clicked "Diagnose with Copilot" on. Executed via KernelChannelClient against a live MSIT Spark kernel before /fix was sent.
result = totally_undefined_variable_xyz + 1
print(result)
After execution, these outputs (statement-meta + error) became the cell_outputs field inside the additional context payload.
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[20], line 1
----> 1 result = totally_undefined_variable_xyz + 1
2 print(result)
NameError: name 'totally_undefined_variable_xyz' is not defined
Sent over the Notebook Service WebSocket as UserPromptContent. The shape exactly matches the production trident-de-ds-app getCopilotContext() output (snake_case, .data nested inside additional_context).
| question | /fix |
| agent | spark-diagnostics |
| location | cellOutput |
| intents | [{"type": "/", "value": "fix"}] |
{
"data": [
{
"type": "cell",
"data": {
"cell_id": "62526e39-0c0d-4d1f-84d4-5b34f2bf5f7c",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:03.1614074Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "a4730f3b-a6ec-44ea-89aa-7432182d4b2c"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "NameError",
"evalue": "name 'totally_undefined_variable_xyz' is not defined",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[20], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m result \u001b[38;5;241m=\u001b[39m totally_undefined_variable_xyz \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28mprint\u001b[39m(result)\n",
"\u001b[0;31mNameError\u001b[0m: name 'totally_undefined_variable_xyz' is not defined"
]
}
]
}
}
]
}{
"question": "/fix",
"agent": "spark-diagnostics",
"location": "cellOutput",
"intents": [
{
"type": "/",
"value": "fix"
}
],
"context": {
"current_notebook_context": {
"editor_context": {
"user_focused_cell_id": "62526e39-0c0d-4d1f-84d4-5b34f2bf5f7c"
},
"run_context": {
"run_failed_cell_ids": [
"62526e39-0c0d-4d1f-84d4-5b34f2bf5f7c"
]
}
},
"fabric_context": {
"notebook_artifact_info": {
"artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"workspace_id": "89663e7e-fd1a-4303-8912-1b10ee21676a"
}
},
"additional_context": {
"data": [
{
"type": "cell",
"data": {
"cell_id": "62526e39-0c0d-4d1f-84d4-5b34f2bf5f7c",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:03.1614074Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "a4730f3b-a6ec-44ea-89aa-7432182d4b2c"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "NameError",
"evalue": "name 'totally_undefined_variable_xyz' is not defined",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[20], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m result \u001b[38;5;241m=\u001b[39m totally_undefined_variable_xyz \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28mprint\u001b[39m(result)\n",
"\u001b[0;31mNameError\u001b[0m: name 'totally_undefined_variable_xyz' is not defined"
]
}
]
}
}
]
}
}
}Streamed as 230 markdown chunks (1077 chars total) over the Copilot WebSocket. Rendered below; raw markdown is available in the JSON.
totally_undefined_variable_xyz, which has not been defined anywhere above or in the current notebook context. This results in a standard Python NameError.
# Key Changes:
- Removed usage of undefined variable.
- Added a defined variable as an example for successful execution.
defined_variable = 10 # Example definition
result = defined_variable + 1
print(result)
defined_variable = 10 with your intended value or reference, as needed.*
Suggested code fix was not applied. You can copy the code above and apply it manually.
Reason: The cell failure was due to a NameError, not a Spark/job failure. No running Spark session context is available for programmatic cell replacement and there is no evidence of this cell containing necessary non-demo logic. The provided fix corrects the undefined variable usage. Please update the cell with the code above and re-run.### 2. Error Summary **Error Type**: NameError (Undefined Variable) **Root Cause**: The code in the cell attempts to use the variable `totally_undefined_variable_xyz`, which has not been defined anywhere above or in the current notebook context. This results in a standard Python `NameError`. --- ### 3. Suggested Code Fix ```python # Key Changes: # - Removed usage of undefined variable. # - Added a defined variable as an example for successful execution. defined_variable = 10 # Example definition result = defined_variable + 1 print(result) ``` *Replace `defined_variable = 10` with your intended value or reference, as needed.* --- Suggested code fix was not applied. You can copy the code above and apply it manually. **Reason:** The cell failure was due to a NameError, not a Spark/job failure. No running Spark session context is available for programmatic cell replacement and there is no evidence of this cell containing necessary non-demo logic. The provided fix corrects the undefined variable usage. Please update the cell with the code above and re-run.
Simulates clicking "Diagnose with Copilot" on a PySpark cell that selects a column that doesn't exist on the DataFrame. Triggers an AnalysisException about an unresolved column reference. The runner captures the real outputs + traceback and sends /fix to the spark-diagnostics agent.
The cell that the user clicked "Diagnose with Copilot" on. Executed via KernelChannelClient against a live MSIT Spark kernel before /fix was sent.
from pyspark.sql import Row
df = spark.createDataFrame([Row(name="alice", age=30), Row(name="bob", age=25)])
df.select("nonexistent_column_zzz").show()
After execution, these outputs (statement-meta + error) became the cell_outputs field inside the additional context payload.
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
Cell In[23], line 3
1 from pyspark.sql import Row
2 df = spark.createDataFrame([Row(name="alice", age=30), Row(name="bob", age=25)])
----> 3 df.select("nonexistent_column_zzz").show()
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py:3229, in DataFrame.select(self, *cols)
3184 def select(self, *cols: "ColumnOrName") -> "DataFrame": # type: ignore[misc]
3185 """Projects a set of expressions and returns a new :class:`DataFrame`.
3186
3187 .. versionadded:: 1.3.0
(...)
3227 +-----+---+
3228 """
-> 3229 jdf = self._jdf.select(self._jcols(*cols))
3230 return DataFrame(jdf, self.sparkSession)
File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
... (22 more lines truncated)
Sent over the Notebook Service WebSocket as UserPromptContent. The shape exactly matches the production trident-de-ds-app getCopilotContext() output (snake_case, .data nested inside additional_context).
| question | /fix |
| agent | spark-diagnostics |
| location | cellOutput |
| intents | [{"type": "/", "value": "fix"}] |
{
"data": [
{
"type": "cell",
"data": {
"cell_id": "e93ac37c-50c4-4460-8adf-a26d15e056b3",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:13.7211536Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "65bbecc4-fe33-4855-90ad-fa2a515d9184"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "AnalysisException",
"evalue": "[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `nonexistent_column_zzz` cannot be resolved. Did you mean one of the following? [`name`, `age`].;\n'Project ['nonexistent_column_zzz]\n+- LogicalRDD [name#730, age#731L], false\n",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAnalysisException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[23], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msql\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Row\n\u001b[1;32m 2\u001b[0m df \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39mcreateDataFrame([Row(name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124malice\u001b[39m\u001b[38;5;124m\"\u001b[39m, age\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m30\u001b[39m), Row(name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbob\u001b[39m\u001b[38;5;124m\"\u001b[39m, age\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m25\u001b[39m)])\n\u001b[0;32m----> 3\u001b[0m df\u001b[38;5;241m.\u001b[39mselect(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonexistent_column_zzz\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py:3229\u001b[0m, in \u001b[0;36mDataFrame.select\u001b[0;34m(self, *cols)\u001b[0m\n\u001b[1;32m 3184\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mselect\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39mcols: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mColumnOrName\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDataFrame\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m 3185\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Projects a set of expressions and returns a new :class:`DataFrame`.\u001b[39;00m\n\u001b[1;32m 3186\u001b[0m \n\u001b[1;32m 3187\u001b[0m \u001b[38;5;124;03m .. versionadded:: 1.3.0\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 3227\u001b[0m \u001b[38;5;124;03m +-----+---+\u001b[39;00m\n\u001b[1;32m 3228\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m-> 3229\u001b[0m jdf \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jdf\u001b[38;5;241m.\u001b[39mselect(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jcols(\u001b[38;5;241m*\u001b[39mcols))\n\u001b[1;32m 3230\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m DataFrame(jdf, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msparkSession)\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mAnalysisException\u001b[0m: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `nonexistent_column_zzz` cannot be resolved. Did you mean one of the following? [`name`, `age`].;\n'Project ['nonexistent_column_zzz]\n+- LogicalRDD [name#730, age#731L], false\n"
]
}
]
}
}
]
}{
"question": "/fix",
"agent": "spark-diagnostics",
"location": "cellOutput",
"intents": [
{
"type": "/",
"value": "fix"
}
],
"context": {
"current_notebook_context": {
"editor_context": {
"user_focused_cell_id": "e93ac37c-50c4-4460-8adf-a26d15e056b3"
},
"run_context": {
"run_failed_cell_ids": [
"e93ac37c-50c4-4460-8adf-a26d15e056b3"
]
}
},
"fabric_context": {
"notebook_artifact_info": {
"artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"workspace_id": "89663e7e-fd1a-4303-8912-1b10ee21676a"
}
},
"additional_context": {
"data": [
{
"type": "cell",
"data": {
"cell_id": "e93ac37c-50c4-4460-8adf-a26d15e056b3",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:13.7211536Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "65bbecc4-fe33-4855-90ad-fa2a515d9184"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "AnalysisException",
"evalue": "[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `nonexistent_column_zzz` cannot be resolved. Did you mean one of the following? [`name`, `age`].;\n'Project ['nonexistent_column_zzz]\n+- LogicalRDD [name#730, age#731L], false\n",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAnalysisException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[23], line 3\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpyspark\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msql\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m Row\n\u001b[1;32m 2\u001b[0m df \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39mcreateDataFrame([Row(name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124malice\u001b[39m\u001b[38;5;124m\"\u001b[39m, age\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m30\u001b[39m), Row(name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbob\u001b[39m\u001b[38;5;124m\"\u001b[39m, age\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m25\u001b[39m)])\n\u001b[0;32m----> 3\u001b[0m df\u001b[38;5;241m.\u001b[39mselect(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonexistent_column_zzz\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py:3229\u001b[0m, in \u001b[0;36mDataFrame.select\u001b[0;34m(self, *cols)\u001b[0m\n\u001b[1;32m 3184\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mselect\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39mcols: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mColumnOrName\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDataFrame\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[1;32m 3185\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Projects a set of expressions and returns a new :class:`DataFrame`.\u001b[39;00m\n\u001b[1;32m 3186\u001b[0m \n\u001b[1;32m 3187\u001b[0m \u001b[38;5;124;03m .. versionadded:: 1.3.0\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 3227\u001b[0m \u001b[38;5;124;03m +-----+---+\u001b[39;00m\n\u001b[1;32m 3228\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m-> 3229\u001b[0m jdf \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jdf\u001b[38;5;241m.\u001b[39mselect(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jcols(\u001b[38;5;241m*\u001b[39mcols))\n\u001b[1;32m 3230\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m DataFrame(jdf, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msparkSession)\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mAnalysisException\u001b[0m: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `nonexistent_column_zzz` cannot be resolved. Did you mean one of the following? [`name`, `age`].;\n'Project ['nonexistent_column_zzz]\n+- LogicalRDD [name#730, age#731L], false\n"
]
}
]
}
}
]
}
}
}Streamed as 313 markdown chunks (1300 chars total) over the Copilot WebSocket. Rendered below; raw markdown is available in the JSON.
nonexistent_column_zzz from the DataFrame, but this column does not exist. The DataFrame only contains the columns name and age, as shown in the exception’s suggestion.
# Import Row to create structured data for DataFrame
from pyspark.sql import Row
Create a DataFrame with two columns: 'name' and 'age'
df = spark.createDataFrame([
Row(name="alice", age=30),
Row(name="bob", age=25)
])
Select an EXISTING column — options: 'name' or 'age'
Example: Selecting the 'name' column (as 'nonexistent_column_zzz' does not exist)
df.select("name").show()
df.select("nonexistent_column_zzz") with df.select("name"), as name exists in the DataFrame. Alternatively, you could use age.Suggested code fix was not applied automatically due to Spark session/context issues. You can copy the code above and apply it manually.
select() statement.### 2. Error Summary
**Error Type**: Column Not Found (AnalysisException)
**Root Cause**: The code attempts to select a column named `nonexistent_column_zzz` from the DataFrame, but this column does not exist. The DataFrame only contains the columns `name` and `age`, as shown in the exception’s suggestion.
---
### 3. Suggested Code Fix
```python
# Import Row to create structured data for DataFrame
from pyspark.sql import Row
# Create a DataFrame with two columns: 'name' and 'age'
df = spark.createDataFrame([
Row(name="alice", age=30),
Row(name="bob", age=25)
])
# Select an EXISTING column — options: 'name' or 'age'
# Example: Selecting the 'name' column (as 'nonexistent_column_zzz' does not exist)
df.select("name").show()
```
**Key Changes:**
- Replaced `df.select("nonexistent_column_zzz")` with `df.select("name")`, as `name` exists in the DataFrame. Alternatively, you could use `age`.
---
### 4. Confirmation
Suggested code fix was not applied automatically due to Spark session/context issues. You can copy the code above and apply it manually.
---
**Note:** The direct diagnostics for the Spark session could not be retrieved due to missing or inactive context, but the error is fully explained and solved by correcting the column name in the `select()` statement.
Simulates clicking "Diagnose with Copilot" on a PySpark cell that tries to read a parquet file from a path that doesn't exist in any attached lakehouse. Triggers an AnalysisException / PathNotFoundException from the Spark catalyst. The runner captures the real outputs + traceback and sends /fix to the spark-diagnostics agent.
The cell that the user clicked "Diagnose with Copilot" on. Executed via KernelChannelClient against a live MSIT Spark kernel before /fix was sent.
df = spark.read.parquet("abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table")
df.show()
After execution, these outputs (statement-meta + error) became the cell_outputs field inside the additional context payload.
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
Cell In[26], line 1
----> 1 df = spark.read.parquet("abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table")
2 df.show()
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:544, in DataFrameReader.parquet(self, *paths, **options)
533 int96RebaseMode = options.get("int96RebaseMode", None)
534 self._set_opts(
535 mergeSchema=mergeSchema,
536 pathGlobFilter=pathGlobFilter,
(...)
541 int96RebaseMode=int96RebaseMode,
542 )
--> 544 return self._df(self._jreader.parquet(_to_seq(self._spark._sc, paths)))
File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
1317 self.command_header +\
1318 args_command +\
... (17 more lines truncated)
Sent over the Notebook Service WebSocket as UserPromptContent. The shape exactly matches the production trident-de-ds-app getCopilotContext() output (snake_case, .data nested inside additional_context).
| question | /fix |
| agent | spark-diagnostics |
| location | cellOutput |
| intents | [{"type": "/", "value": "fix"}] |
{
"data": [
{
"type": "cell",
"data": {
"cell_id": "e0f0bf3d-ae4c-4a62-8f3d-2d6ecaded458",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:25.1360898Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "2c400593-c18b-463f-808e-2ca290fe8da7"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "AnalysisException",
"evalue": "[PATH_NOT_FOUND] Path does not exist: abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table.",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAnalysisException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[26], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m df \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39mread\u001b[38;5;241m.\u001b[39mparquet(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mabfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 2\u001b[0m df\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:544\u001b[0m, in \u001b[0;36mDataFrameReader.parquet\u001b[0;34m(self, *paths, **options)\u001b[0m\n\u001b[1;32m 533\u001b[0m int96RebaseMode \u001b[38;5;241m=\u001b[39m options\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mint96RebaseMode\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m 534\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_set_opts(\n\u001b[1;32m 535\u001b[0m mergeSchema\u001b[38;5;241m=\u001b[39mmergeSchema,\n\u001b[1;32m 536\u001b[0m pathGlobFilter\u001b[38;5;241m=\u001b[39mpathGlobFilter,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 541\u001b[0m int96RebaseMode\u001b[38;5;241m=\u001b[39mint96RebaseMode,\n\u001b[1;32m 542\u001b[0m )\n\u001b[0;32m--> 544\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_df(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jreader\u001b[38;5;241m.\u001b[39mparquet(_to_seq(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_spark\u001b[38;5;241m.\u001b[39m_sc, paths)))\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mAnalysisException\u001b[0m: [PATH_NOT_FOUND] Path does not exist: abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table."
]
}
]
}
}
]
}{
"question": "/fix",
"agent": "spark-diagnostics",
"location": "cellOutput",
"intents": [
{
"type": "/",
"value": "fix"
}
],
"context": {
"current_notebook_context": {
"editor_context": {
"user_focused_cell_id": "e0f0bf3d-ae4c-4a62-8f3d-2d6ecaded458"
},
"run_context": {
"run_failed_cell_ids": [
"e0f0bf3d-ae4c-4a62-8f3d-2d6ecaded458"
]
}
},
"fabric_context": {
"notebook_artifact_info": {
"artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"workspace_id": "89663e7e-fd1a-4303-8912-1b10ee21676a"
}
},
"additional_context": {
"data": [
{
"type": "cell",
"data": {
"cell_id": "e0f0bf3d-ae4c-4a62-8f3d-2d6ecaded458",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:25.1360898Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "2c400593-c18b-463f-808e-2ca290fe8da7"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "AnalysisException",
"evalue": "[PATH_NOT_FOUND] Path does not exist: abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table.",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAnalysisException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[26], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m df \u001b[38;5;241m=\u001b[39m spark\u001b[38;5;241m.\u001b[39mread\u001b[38;5;241m.\u001b[39mparquet(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mabfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 2\u001b[0m df\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py:544\u001b[0m, in \u001b[0;36mDataFrameReader.parquet\u001b[0;34m(self, *paths, **options)\u001b[0m\n\u001b[1;32m 533\u001b[0m int96RebaseMode \u001b[38;5;241m=\u001b[39m options\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mint96RebaseMode\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m 534\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_set_opts(\n\u001b[1;32m 535\u001b[0m mergeSchema\u001b[38;5;241m=\u001b[39mmergeSchema,\n\u001b[1;32m 536\u001b[0m pathGlobFilter\u001b[38;5;241m=\u001b[39mpathGlobFilter,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 541\u001b[0m int96RebaseMode\u001b[38;5;241m=\u001b[39mint96RebaseMode,\n\u001b[1;32m 542\u001b[0m )\n\u001b[0;32m--> 544\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_df(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jreader\u001b[38;5;241m.\u001b[39mparquet(_to_seq(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_spark\u001b[38;5;241m.\u001b[39m_sc, paths)))\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mAnalysisException\u001b[0m: [PATH_NOT_FOUND] Path does not exist: abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table."
]
}
]
}
}
]
}
}
}Streamed as 334 markdown chunks (1436 chars total) over the Copilot WebSocket. Rendered below; raw markdown is available in the JSON.
abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table
This is either a placeholder, an incorrect workspace/lakehouse/table, or the file/table has not been created.
# Key Changes:
- The original path referenced a non-existent workspace and table.
- You must update this path to a valid table or file that exists in your current Lakehouse.
- If you want to read from a real table, use the Files/ or Tables/ location from your default Lakehouse.
- Here is a template example. Replace 'your_table' and optionally 'dbo' (for schema-enabled) with your existing Lakehouse table name.
Example: Read from an existing default Lakehouse table 'your_table' (schema 'dbo')
df = spark.read.table("Tables/dbo/your_table")
To discover existing tables in your Lakehouse, use:
tables = spark.sql("SHOW TABLES")
display(tables)
Replace 'your_table' with a table you verified exists.
display(df)
Suggested code fix was not applied. You need to manually update the path with a real table or file that exists in your environment. Use the "SHOW TABLES" command or the Lakehouse UI to find an available table, then re-run the cell with the correct table name.
---
## Error Summary
**Error Type**: AnalysisException – Path Not Found
**Root Cause**: The code attempts to read a Parquet file from the following path, which does not exist:
`abfss://nonexistent_workspace@onelake.dfs.fabric.microsoft.com/bogus.lakehouse/Tables/missing_table`
This is either a placeholder, an incorrect workspace/lakehouse/table, or the file/table has not been created.
---
## Suggested Code Fix
```python
# Key Changes:
# - The original path referenced a non-existent workspace and table.
# - You must update this path to a valid table or file that exists in your current Lakehouse.
# - If you want to read from a real table, use the Files/ or Tables/ location from your default Lakehouse.
# - Here is a template example. Replace 'your_table' and optionally 'dbo' (for schema-enabled) with your existing Lakehouse table name.
# Example: Read from an existing default Lakehouse table 'your_table' (schema 'dbo')
df = spark.read.table("Tables/dbo/your_table")
# To discover existing tables in your Lakehouse, use:
# tables = spark.sql("SHOW TABLES")
# display(tables)
# Replace 'your_table' with a table you verified exists.
display(df)
```
---
Suggested code fix was not applied. You need to manually update the path with a real table or file that exists in your environment. Use the "SHOW TABLES" command or the Lakehouse UI to find an available table, then re-run the cell with the correct table name.
Simulates clicking the "Diagnose with Copilot" (/fix) button after a Spark SQL cell fails because it queries a table that does not exist. The runner executes the failing query on the live Spark kernel, captures the real ename/evalue/traceback + statement-meta, then sends a /fix message to the spark-diagnostics agent with the captured outputs in additional_context.data (matching the trident UX button payload).
The cell that the user clicked "Diagnose with Copilot" on. Executed via KernelChannelClient against a live MSIT Spark kernel before /fix was sent.
spark.sql("SELECT * FROM nonexistent_table_xyz_nbeval_fix_test LIMIT 1").show()
After execution, these outputs (statement-meta + error) became the cell_outputs field inside the additional context payload.
---------------------------------------------------------------------------
UnsupportedOperationException Traceback (most recent call last)
Cell In[29], line 1
----> 1 spark.sql("SELECT * FROM nonexistent_table_xyz_nbeval_fix_test LIMIT 1").show()
File /opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py:1631, in SparkSession.sql(self, sqlQuery, args, **kwargs)
1627 assert self._jvm is not None
1628 litArgs = self._jvm.PythonUtils.toArray(
1629 [_to_java_column(lit(v)) for v in (args or [])]
1630 )
-> 1631 return DataFrame(self._jsparkSession.sql(sqlQuery, litArgs), self)
1632 finally:
1633 if len(kwargs) > 0:
File ~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +\
1317 self.command_header +\
1318 args_command +\
1319 proto.END_COMMAND_PART
1321 answer = self.gateway_client.send_command(command)
... (15 more lines truncated)
Sent over the Notebook Service WebSocket as UserPromptContent. The shape exactly matches the production trident-de-ds-app getCopilotContext() output (snake_case, .data nested inside additional_context).
| question | /fix |
| agent | spark-diagnostics |
| location | cellOutput |
| intents | [{"type": "/", "value": "fix"}] |
{
"data": [
{
"type": "cell",
"data": {
"cell_id": "5fe02731-3d59-466b-9673-a5e5afdb6268",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:35.3762825Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "29ae78e0-d3d3-48e9-be3d-aaf7f1a640d5"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "UnsupportedOperationException",
"evalue": "No default context found, please attach a lakehouse before running spark sql queries with partial namespaces.",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mUnsupportedOperationException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[29], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m spark\u001b[38;5;241m.\u001b[39msql(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSELECT * FROM nonexistent_table_xyz_nbeval_fix_test LIMIT 1\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py:1631\u001b[0m, in \u001b[0;36mSparkSession.sql\u001b[0;34m(self, sqlQuery, args, **kwargs)\u001b[0m\n\u001b[1;32m 1627\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jvm \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 1628\u001b[0m litArgs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jvm\u001b[38;5;241m.\u001b[39mPythonUtils\u001b[38;5;241m.\u001b[39mtoArray(\n\u001b[1;32m 1629\u001b[0m [_to_java_column(lit(v)) \u001b[38;5;28;01mfor\u001b[39;00m v \u001b[38;5;129;01min\u001b[39;00m (args \u001b[38;5;129;01mor\u001b[39;00m [])]\n\u001b[1;32m 1630\u001b[0m )\n\u001b[0;32m-> 1631\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m DataFrame(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jsparkSession\u001b[38;5;241m.\u001b[39msql(sqlQuery, litArgs), \u001b[38;5;28mself\u001b[39m)\n\u001b[1;32m 1632\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m 1633\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(kwargs) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m:\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mUnsupportedOperationException\u001b[0m: No default context found, please attach a lakehouse before running spark sql queries with partial namespaces."
]
}
]
}
}
]
}{
"question": "/fix",
"agent": "spark-diagnostics",
"location": "cellOutput",
"intents": [
{
"type": "/",
"value": "fix"
}
],
"context": {
"current_notebook_context": {
"editor_context": {
"user_focused_cell_id": "5fe02731-3d59-466b-9673-a5e5afdb6268"
},
"run_context": {
"run_failed_cell_ids": [
"5fe02731-3d59-466b-9673-a5e5afdb6268"
]
}
},
"fabric_context": {
"notebook_artifact_info": {
"artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"workspace_id": "89663e7e-fd1a-4303-8912-1b10ee21676a"
}
},
"additional_context": {
"data": [
{
"type": "cell",
"data": {
"cell_id": "5fe02731-3d59-466b-9673-a5e5afdb6268",
"cell_index": 0,
"cell_type": "code",
"notebook_artifact_id": "967ec516-01d0-48f4-95c3-0a6421d72312",
"notebook_name": "diagnose-with-copilot-eval-target",
"cell_outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.livy.statement-meta+json": {
"spark_pool": null,
"statement_id": -1,
"statement_ids": null,
"state": "waiting",
"livy_statement_state": null,
"spark_jobs_updating": true,
"spark_jobs": null,
"session_id": null,
"normalized_state": "waiting",
"queued_time": "2026-05-19T13:50:35.3762825Z",
"session_start_time": null,
"execution_start_time": null,
"execution_finish_time": null,
"parent_msg_id": "29ae78e0-d3d3-48e9-be3d-aaf7f1a640d5"
},
"text/plain": "StatementMeta(, , -1, Waiting, , Waiting, True)"
}
},
{
"output_type": "error",
"ename": "UnsupportedOperationException",
"evalue": "No default context found, please attach a lakehouse before running spark sql queries with partial namespaces.",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mUnsupportedOperationException\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[29], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m spark\u001b[38;5;241m.\u001b[39msql(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSELECT * FROM nonexistent_table_xyz_nbeval_fix_test LIMIT 1\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mshow()\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py:1631\u001b[0m, in \u001b[0;36mSparkSession.sql\u001b[0;34m(self, sqlQuery, args, **kwargs)\u001b[0m\n\u001b[1;32m 1627\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jvm \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 1628\u001b[0m litArgs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jvm\u001b[38;5;241m.\u001b[39mPythonUtils\u001b[38;5;241m.\u001b[39mtoArray(\n\u001b[1;32m 1629\u001b[0m [_to_java_column(lit(v)) \u001b[38;5;28;01mfor\u001b[39;00m v \u001b[38;5;129;01min\u001b[39;00m (args \u001b[38;5;129;01mor\u001b[39;00m [])]\n\u001b[1;32m 1630\u001b[0m )\n\u001b[0;32m-> 1631\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m DataFrame(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jsparkSession\u001b[38;5;241m.\u001b[39msql(sqlQuery, litArgs), \u001b[38;5;28mself\u001b[39m)\n\u001b[1;32m 1632\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m 1633\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(kwargs) \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m:\n",
"File \u001b[0;32m~/cluster-env/trident_env/lib/python3.11/site-packages/py4j/java_gateway.py:1322\u001b[0m, in \u001b[0;36mJavaMember.__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 1316\u001b[0m command \u001b[38;5;241m=\u001b[39m proto\u001b[38;5;241m.\u001b[39mCALL_COMMAND_NAME \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcommand_header \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1318\u001b[0m args_command \u001b[38;5;241m+\u001b[39m\\\n\u001b[1;32m 1319\u001b[0m proto\u001b[38;5;241m.\u001b[39mEND_COMMAND_PART\n\u001b[1;32m 1321\u001b[0m answer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client\u001b[38;5;241m.\u001b[39msend_command(command)\n\u001b[0;32m-> 1322\u001b[0m return_value \u001b[38;5;241m=\u001b[39m get_return_value(\n\u001b[1;32m 1323\u001b[0m answer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mgateway_client, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtarget_id, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mname)\n\u001b[1;32m 1325\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m temp_arg \u001b[38;5;129;01min\u001b[39;00m temp_args:\n\u001b[1;32m 1326\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(temp_arg, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_detach\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n",
"File \u001b[0;32m/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:185\u001b[0m, in \u001b[0;36mcapture_sql_exception.<locals>.deco\u001b[0;34m(*a, **kw)\u001b[0m\n\u001b[1;32m 181\u001b[0m converted \u001b[38;5;241m=\u001b[39m convert_exception(e\u001b[38;5;241m.\u001b[39mjava_exception)\n\u001b[1;32m 182\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(converted, UnknownException):\n\u001b[1;32m 183\u001b[0m \u001b[38;5;66;03m# Hide where the exception came from that shows a non-Pythonic\u001b[39;00m\n\u001b[1;32m 184\u001b[0m \u001b[38;5;66;03m# JVM exception message.\u001b[39;00m\n\u001b[0;32m--> 185\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m converted \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 186\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 187\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m\n",
"\u001b[0;31mUnsupportedOperationException\u001b[0m: No default context found, please attach a lakehouse before running spark sql queries with partial namespaces."
]
}
]
}
}
]
}
}
}Streamed as 146 markdown chunks (694 chars total) over the Copilot WebSocket. Rendered below; raw markdown is available in the JSON.
1. In the Fabric notebook interface, click the "Add Lakehouse" button at the top. 2. Select or create a Lakehouse to attach to your notebook. 3. Once your lakehouse is attached, re-run this cell.
Suggested code fix was not applied. You can copy the code above and apply it manually.
--- ## Error Summary **Error Type**: Lakehouse Context Not Attached **Root Cause**: SparkSQL cannot execute queries without an attached lakehouse context. Your notebook has no default lakehouse attached, which is required for running SQL queries with partial namespaces. --- ## Resolution (User Action Required) **No code changes needed.** This error requires you to perform the following action: 1. In the Fabric notebook interface, click the "Add Lakehouse" button at the top. 2. Select or create a Lakehouse to attach to your notebook. 3. Once your lakehouse is attached, re-run this cell. --- Suggested code fix was not applied. You can copy the code above and apply it manually.