Pre-M3 Performance

Item 1: Web Worker Streaming Execution

Design

New file: src/workers/worker-manager.ts

Singleton class owning Worker lifecycle and orchestrating execution.

• Instantiate Worker: new Worker(new URL('./stats.worker.ts', import.meta.url), { type: 'module' })
• Called lazily on first use (any public method calls init() if not yet initialized)

• Posts LOAD_DATASET { csv } to Worker
• Worker parses CSV via PapaParse, stores dataset, responds with DATASET_LOADED { dataset }
• Resolves with the parsed Dataset for main-thread store state (column names, row counts, UI display)

• Runs topologicalSort() on main thread (reuse exported function from executor.ts)
• Builds edge lookup (same logic as current executePipeline)
• For each node in topological order:
  1. Collect input data from already-completed nodes (tracked locally in a Map<string, unknown>)
  2. Post EXECUTE_NODE { node, inputData } to Worker
  3. Await NODE_COMPLETE or NODE_ERROR response
  4. On complete: store result locally, call onNodeComplete(nodeId, result)
  5. On error: store error, call onNodeError(nodeId, error), mark dependents as upstream-error (skip them)
• Resolves when all nodes are processed

• Kill the Worker

Data Flow

CSV loading (currently freezes UI):

Pipeline execution (currently synchronous):

Changes to src/ui/store/data.ts

loadCSVFile() delegates CSV parsing to the Worker. No more parseCSV() call on main thread.

Changes to src/ui/store/pipeline.ts

runPipeline(dataset) becomes async:

1. Mark all nodes as running, set executing: true (unchanged)
2. Call workerManager.executePipeline() with streaming callbacks:
   • onComplete(nodeId, result) → set() to update that node to complete
   • onError(nodeId, error) → set() to update that node to error
3. When promise resolves, set executing: false

Changes to src/core/pipeline/executor.ts

• Export topologicalSort() (currently module-private) for WorkerManager to reuse
• Synchronous executePipeline() stays as-is — used by Vitest tests and as Worker fallback

No changes to Worker files

The existing protocol and Worker implementation already support exactly this flow. LOAD_DATASET accepts CSV string, parses it, responds with Dataset. EXECUTE_NODE accepts a node + inputs, executes, responds with result.

Testing

• Unit test worker-manager.ts: Mock Worker via a minimal MessagePort shim. Verify topological execution order, streaming callbacks, error propagation to dependents, fallback to sync on Worker failure.
• Async integration test worker-manager.integration.test.ts: Mock the Worker transport layer but use real executors. Feed a realistic multi-node pipeline (data-load → linear-model → comparison-table) with a real CSV fixture. Verify: (1) nodes execute in topological order, (2) onNodeComplete callbacks fire with correct typed results matching the synchronous path, (3) node error propagates to skip dependents, (4) final promise resolves after all nodes.
• Existing integration tests: pipeline/integration.test.ts continues to pass unchanged (synchronous path).
• E2E: Existing Playwright tests cover the full upload → execute → results flow and should pass without changes.

Item 2: Spec Curve Hover Performance

Design

Remove hoveredCellCoordinate from renderPlot dependencies

Change dependency array from [data, hoveredCellCoordinate] to [data]. SVGs rebuild only when underlying data changes.

Tag SVG elements with data-coordinate

After each Plot.plot() call, post-process the returned SVG:

1. Use the existing sortedColumns map (sort index → coordinate)
2. Query mark elements:
   • Estimate panel: circle (dots), line (CI rules)
   • Indicator panel: rect (cells)
3. Bucket elements by x-position (cx or x attribute), sort by x, assign data-coordinate from sortedColumns[i].coordinate. Positions are deterministic because Plot uses the same linear scale with a shared xDomain.

Hover effect via direct DOM mutation

const containerRef = useRef<HTMLDivElement>(null);

useEffect(() => {
  const container = containerRef.current;
  if (!container) return;
  const elements = container.querySelectorAll('[data-coordinate]');
  for (const el of elements) {
    const coord = el.getAttribute('data-coordinate');
    const match = hoveredCellCoordinate === null
      || coord === hoveredCellCoordinate;
    (el as HTMLElement).style.opacity = match ? '' : '0.25';
  }
}, [hoveredCellCoordinate]);

Remove opacity from Plot mark data

Lines 99 and 106 currently compute per-mark opacity based on hoveredCellCoordinate. Remove these — all marks render at full opacity. Hover highlighting is handled entirely by DOM mutation.

No changes to chart-container.tsx

ChartContainer stays unchanged. It re-renders when renderPlot changes, which now only happens on data changes.

After fix:

Zero SVG rebuilds. Direct DOM mutation on <200 elements is effectively free.

Testing

• E2E: Existing spec curve tests assert on rendered content and edges — should pass without changes.
• Manual: Hover over spec curve columns, confirm smooth highlighting without jank.

Execution Order

These two items are independent and can be implemented in parallel. Recommended sequence: Worker first (higher impact), then spec curve hover (polish).

Files Changed