Interlyse V2 — Cloud Verification Service (Pivot Design)

Date: 2026-08-19 Status: Draft for review Supersedes: ROADMAP.md milestones M6 (Replication Dashboard) and M7 (Authoring Foundation) as written; the “no backend” and “no server-side compute” architecture principles; the browser-execution strategy (TS engine + WebR) as the product’s execution layer.

1. Decision summary

Interlyse pivots from a browser-executed specification explorer to a cloud verification service for empirical research. Replication packages execute in containers running real R (and licensed Stata) on rented infrastructure. The parser/DAG stack — the project’s genuinely novel asset — is repurposed from execution engine to evidence layer: the thing that makes a verification report legible, auditable, and clickable down to source.

The product is certification, not compute. In an era where any author can ask an AI agent to run their package in a sandbox, the durable value is a third party’s signed, structured, reproducible verdict: these models ran, these numbers match the printed tables, here is the lineage. Every roadmap decision is checked against “does this make the signature more trusted?”, not “does this run more code?”

Why now, in one sentence each:

The browser architecture required reimplementing R semantics in TypeScript — an unbounded backlog that consumed the last three months and permanently capped coverage at ~50% of the corpus (Stata cannot run in a browser; it runs fine in a licensed container).
AEA and the Econometric Society have started paying AI-verification vendors (Refine), proving the procurement path.
Runtime instrumentation of real R replaces most of the static-recognition workload at higher fidelity and near-zero marginal cost.

2. Product spine

A paper’s empirical results, continuously verified — from data to code to tables to manuscript.

Stage	Ships	Buyer / motion
V1 — Runner + Report	Containerized execution (R, then Stata), environment reconstruction, instrumented trace, scorecard + evidence DAG, printed-value matching, signed certificate. Free gallery of reproduced published papers.	Authors, self-serve pre-flight (“pass the data editor on the first try”). Gallery = marketing.
V1.5 — Manuscript loop	Table generation + Overleaf/git sync; result matching against LaTeX we generate. Nothing in the manuscript is ever stale.	Authors, subscription. Converts one-shot verification into a recurring relationship.
V2 — Programmatic	Public HTTP API + MCP server (`replicate`, `get_dag`, `run_spec`, `verify_tables`); GitHub push triggers; credentials-passthrough data resolvers (FRED/IPUMS/WRDS with the user’s own keys); exportable verifier for restricted-data enclaves.	AI agents, institutions with restricted data.
V2.5 — Exploration	Fork/clone a gallery paper; robustness-on-demand (server-side spec sweeps rendered with the existing spec-curve machinery).	Referees, discussants, researchers. The growth engine.
Journals (parallel track)	Same report, institutional billing, editorial-workflow integration. Sold from V1 usage evidence, never load-bearing for survival.	Data editors at journals without in-house teams.

3. Business case (condensed)

Full reasoning lives in the 2026-08-19 strategy conversation; the load-bearing facts:

Beachhead TAM is small and grant-shaped. ~20–40 journals worldwide computationally verify, spending ~$30–150k/yr each (≈ $2–5M TAM); pre-flight authors add ≈ $0.5–1M. The dominant “competitors” are grad students (AEA in-house) and philanthropy (I4R reproduces 250 studies/yr on Open Philanthropy money and is building its own AI Replication Engine on a $95k grant). Code Ocean raised ~$40M, employs ~51, and plateaued — the market can’t feed 51 people. It can feed 1–2 at AI-era costs.
Pricing rail: authors already pay $3–5k grant-billed open-access fees per accepted paper. A $200–500 pre-flight + certificate rides the same rail. Journal contracts ($10–50k/yr) are gravy, modeled at zero for survival.
Financing: revenue + non-dilutive grants (Sloan, Arnold Ventures, Open Philanthropy, NSF POSE — the same funders already paying for this activity). No venture path is assumed or needed.
Moat: the certification apparatus — signed DOI’d reports, badge registry, corpus of curated printed-value gold files, standing with journals/funders, Stata licensing, and the evidence DAG no competitor has. Compute is explicitly not the moat.
Conditions attached to the verdict (self-imposed): first paid pre-flight within 2 quarters of V1 launch; one grant application submitted in parallel with V1 build; I4R partnership conversation opened before gallery launch; certification framing present in the product from day one (reports are signed and immutable from R1).

4. Architecture

4.1 The runner

Containers: one image family per language/vintage. R images pin R version + Posit Package Manager date snapshot; Stata images follow the AEA Data Editor’s docker-stata pattern with our licenses (seats are a few $k/yr; a single-user license covers development).
Orchestration (buy, don’t build): Fly Machines for V1 — scale-to-zero, per-second billing, API-driven, already scoped in the old roadmap’s Future Horizons. A run = create machine → mount package → execute → collect artifacts → destroy. Revisit (AWS Batch / Hetzner) only if volume or GPU needs demand it. No Kubernetes.
Job flow: upload (ZIP or repo URL) → environment reconstruction → instrumented execution → artifact collection (trace JSONL, generated tables/figures, logs) → report generation → signed report stored; workspace destroyed.
Environment reconstruction is a first-class capability, not plumbing: honor renv.lock when present; otherwise solve for a working vintage (package census from library() calls → PPM snapshot nearest the paper’s date → bisect on failure). Every report embeds the resolved manifest — this is both reproducibility metadata and the moat’s unglamorous half.
Security & data policy: untrusted code runs only inside the throwaway machine with no credentials and egress restricted to package repositories + user-authorized data APIs. Default policy: delete everything after the run except the report and the artifacts the user marks for retention. No restricted data is ever stored or redistributed; the gallery contains only open-data papers.

The container executes the author’s code verbatim in real R. We inject a shim that shadows/traces estimation and output functions (lm, glm, feols, felm, ivreg, lm_robust, fixest family, stargazer/modelsummary/etable, ggsave, file writers). Each traced call emits one JSONL record: call text + span-locatable source position, resolved formula, broom::tidy() coefficients/SEs, N, vcov type, data fingerprint (dims + column hash), output file if any. Stata: wrapper ado + log parsing capturing e(b)/e(V) after estimation commands.

Consequences:

A feols call inside any loop/lapply/paste0 contraption is captured when it fires, with ground-truth arguments. Loop expansion, paste-folding, .SD forms, the bounded expression evaluator — all become optional enrichment. The ~250-item TS-semantics backlog is closed, not paused.
The failure mode flips from “unrecognized ⇒ can’t execute” to “unrecognized ⇒ slightly less annotated report.”
Static recognition (the existing R recognizer, a future Stata recognizer) remains for: pre-run preview, source-anchored annotation of the trace, and fork-time editing. It needs to be good, not complete.

4.3 The evidence DAG

Output-centric by default: tables/figures ← models ← datasets ← files. Dozens of nodes, not thousands. Built by joining the runtime trace (ground truth of what executed) with static recognition (source anchoring, lineage between capture points).
Statement-level drill-down: the existing statement-level pipeline machinery becomes the zoomed-in lineage view for a selected model, not the default rendering.
Keep: AnalysisCall, Span, React Flow rendering, grouping/comparison-table/spec-curve components. Drop: typed ports as an execution contract, the executor, worker protocol, WebR dispatch.

4.4 The match engine

Match targets, in priority order per paper:

Deposit-shipped table sources (output/paper/tables/*.tex, logs) — deterministic LaTeX parsing; machine-generated stargazer/esttab output is regular.
Author-provided manuscript source (pre-flight flow; later, the Overleaf git sync).
LLM extraction from PDF — offline curation at gallery ingestion only, never a runtime product feature. Extract twice, cross-check, human spot-check on disagreement. A systematic mismatch pattern in the match step flags a bad extraction.

Extracted values persist as structured gold files with provenance (the existing replicate-*-gold.json convention, extended with extraction provenance). Matching = tolerance-aware diff of traced coefficients/SEs/N against gold, rendered as the scorecard: X of Y scripts executed · M of N models matched printed values · per-cell green/amber/red with drill-down to the evidence DAG.

4.5 The report & certificate

A report is an immutable, signed JSON artifact + rendered view: run summary, environment manifest, model inventory, match scorecard, evidence DAG, blockers list (the run-status playbook’s punch list, machine-generated). Certificates: content-hash of report + our signature; DOI-able for gallery entries; badge embeddable by authors. The existing React app becomes the report viewer + gallery; accounts/billing/job-status are the new UI surface (managed auth + Stripe; build nothing custom).

5. V1 scope — concrete

In: R-language packages end-to-end (upload → reconstruct → run → trace → match → report → certificate); Fly Machines orchestration; instrumentation shim for the estimation/output functions listed in §4.2; env reconstruction with PPM snapshots; match engine paths 1–2 (deposit .tex, user-supplied manuscript); gallery seeded with the existing 10 replicate papers + expansion toward ~50 open-data papers (LLM-assisted curation, path 3); self-serve accounts + Stripe; signed reports.

Out (deliberately): Stata execution (R4, fast-follow — licensing is solved, the wrapper ado is the work); Overleaf sync (V1.5); API/MCP (V2); fork/robustness (V2.5); any TS-side execution work; PDF parsing as a runtime feature; journal-specific workflow integration.

Build sequence:

Phase	Deliverable	Validation
R1 — Runner MVP	Container + shim + trace + env reconstruction; report JSON (no UI polish); works headless via CLI	The 10 `replicate` corpus papers run end-to-end; traced coefficients match the existing gold files
R2 — Match + report UI	Match engine (paths 1–2), evidence DAG view, scorecard, signed immutable reports	≥ 25 gallery papers with green scorecards; blockers list auto-generated matches hand-written run-status entries on overlap
R3 — Self-serve launch	Accounts, billing, upload flow, gallery public, certificates/badges	First external pre-flight run completes without founder intervention; gallery ≥ 50 papers
R4 — Stata	docker-stata images + wrapper ado trace + Stata recognizer (annotation-grade)	≥ 3 Stata-primary corpus papers reproduce printed values

Pricing hypothesis at R3 launch (to be tested, not defended): gallery free; pre-flight $249/run or $499 until-you-pass bundle, certificate included.

6. Disposition of the existing codebase

Asset	Disposition
R lexer/parser/AST, `AnalysisCall`, `Span`, recognizer	Keep — annotation/enrichment layer; coverage bar drops from “complete” to “good”
DAG rendering, grouping, comparison tables, spec curve, LaTeX export	Keep — report UI now; robustness product later
Run-status corpus (142 entries), audit CLI, gold-file conventions, playbook	Keep/repurpose — gallery seed, regression corpus for the report generator, machine-generated-blockers spec
TS stats engine (regression, 2SLS, GLM, demean, sandwich…)	Freeze — keep passing tests, accept no new work; becomes an in-browser demo/preview nicety, never a correctness path
WebR stack (dispatcher, marshalers, VFS bridge, RData extraction)	Freeze — demo tech; no new marshalers
TS-semantics backlog (~250 items: `.SD` variants, `frollmean`, evaluator ops, …)	Close — mark BACKLOG sections superseded by this spec, do not delete history
PDF layout parsing plan (old M6)	Drop
Bidirectional code↔︎DAG sync (old vision / M7)	Drop as foundation; fork-time “generate a code patch” may return in V2.5
“No backend / static CDN / no server-side compute” principles	Rescinded by this spec

New no-go list (additions): no reimplementation of R/Stata semantics for execution — instrumentation only; no storage or redistribution of restricted data; no custom auth/billing/orchestration where a managed service exists; no journal sales dependency for survival; no unsigned or mutable reports.

7. Risks

Risk	Mitigation
Agent commoditization of “running code”	Certification framing from R1: signed immutable reports, badge registry, curated gold corpus. Sell the signature.
Refine expands from prose to code	Speed + wedge focus; their editorial position also makes them a natural partner/acquirer.
I4R’s free AI Replication Engine	Partner, don’t fight: offer the runner as their tooling; their corpus and credibility seed the gallery.
Tiny beachhead TAM	The ladder (V1.5 manuscript loop, V2.5 robustness, adjacent fields) is mandatory roadmap, not option value.
Operational gravity (service, not site)	Managed everything; delete-after-run default minimizes data liability; scale-to-zero keeps idle cost ~$0.
Gallery skews to open-data papers	Stated honestly on the gallery; exportable verifier (V2) turns the restriction into a product.
Env reconstruction rabbit holes	Time-box per paper during seeding; “could not reconstruct” is itself a valid, honest report outcome.

8. Success metrics (12 months post-R3)

≥ 50-paper public gallery at launch; ≥ 150 by month 12.
First paid pre-flight within 2 quarters of R3; ≥ $3k MRR by month 12 (survival floor, not ambition ceiling).
One grant application submitted during V1 build; I4R conversation opened before gallery launch.
≥ 1 journal pilot conversation sourced from gallery/badge traffic (inbound, not cold).