Engineering

Built for
the record.

How Stonewall stores the record, searches it, cites its answers, and tests every change before release. Open any chapter for the detail.

This page, one line per chapter
  1. 1One record, two lanes. The full record stays private; only counts and neutral labels reach this site.
  2. 2Answers cite the record. Each fact is cited by document ID, and the documents an answer cites are listed beside it.
  3. 3Patterns, measured. How often two of the ten most common patterns share a document.
  4. 4Three checks, one queue. Every change passes three required checks before it reaches production.
  5. 5Six services, one cloud. Each part deploys on its own, and rolls back on its own if a live check fails.
  6. 6The production bar. Reliability, security, performance, operations and cost, each tied to a check.

One record, two lanes. The full record stays private. Only counts and neutral labels reach this site.

Architecture: one canon, two lanesSources and signed ingest callbacks become verbatim sidecars, then rows in one manifest beside the codices. From that canon a public lane builds an aggregate snapshot served by stonewall-public. A private lane syncs the full record into Neon Postgres on merge for the cockpit, the Telegram bot and the chat agent, and the corpus API serves it from the repository files behind a service key. An export guard between the lanes stops the public build if the snapshot holds anything else.buildsync on mergeINSourcesdocuments, email,transcripts,screenshotsCALLBACKSIngest webhookINTAKESidecars300 dpi OCR floor,500 dpi vision goldCANONManifest + codicescatalog/manifest.mdPUBLIC LANEAggregate snapshotcounts, neutral labelsPRIVATE LANENeon Postgresfull text, pgvector,memoryEXPORT GUARD · THE BUILD FAILS CLOSEDstonewall-publicstonewall-cockpitstonewall-portalstonewall-botchat-agent-service-c2e9Architecture: one canon, two lanesSources and signed ingest callbacks become verbatim sidecars, then rows in one manifest beside the codices. From that canon a public lane builds an aggregate snapshot served by stonewall-public. A private lane syncs the full record into Neon Postgres on merge for the cockpit, the Telegram bot and the chat agent, and the corpus API serves it from the repository files behind a service key. An export guard between the lanes stops the public build if the snapshot holds anything else.buildsync onmergeINSourcesdocuments, transcripts,email, screenshotsINGESTWebhooksignedcallbacksINTAKESidecars300 dpi OCR floor, 500 dpi vision goldCANONManifest + codicescatalog/manifest.mdPUBLIC LANESnapshotcounts, neutrallabels onlyPRIVATE LANENeon Postgresfull text,pgvector,memoryEXPORT GUARD · FAILS CLOSEDstonewall-publicstonewall-cockpitstonewall-portalstonewall-botchat-agent-service-c2e9
FIG 1 The wall between the lanes is a build step: if the public snapshot holds anything beyond the allowed fields, the build stops.

Every document gets an ID, a type, a date and the people and patterns it touches. The private side holds the full text, searchable by word and by meaning. The public side gets counts and neutral labels only, and a build check refuses anything else.

Inside the data plane
Corpus projection
One row per manifest artifact, full source bodies, and the pattern taxonomy, all full-text indexed.
Semantic layer
Every canonical text file, chunked, with 1536-dimension embeddings under an HNSW cosine index.
Memory
Conversation threads and a long-term operator memory, distilled in the background. The workspace and the Telegram bot keep their own threads and share the long-term memory.
Access
Row-level security, default-deny for generic roles, a read-only query role, and an idempotent, schema-versioned migrator.

PostgreSQL 17 · pgvector · HNSW cosine · numbered migrations

Answers cite the record. Shelby searches the documents before it answers and cites each fact by its document ID.

The citation spineEach factual claim in an answer is bound to the manifest artifact ID and the repo-relative source path, with a line range where the claim requires one.ANSWER“The record shows the expertconceded the point on cross.”A0071artifact IDsources/…/*.pdf.mdsource pathL23line range,where required
FIG 2 A citation names the document ID and its file. For claims about who someone is, when something happened or where a case stands, the format calls for the exact lines.

Shelby answers from the record, not from what the model remembers. Every passage its search returns carries the document’s ID and file, and Shelby is told to cite each fact by that ID and never to invent one. In the workspace, the documents an answer cites are listed beside the conversation, and each answer is saved with its citations. A fictional example, cited claim by claim, is on the landing page.

What is checked, and what is not yet

Checked on every change: the search tools still attach a document ID and file to each passage, and the IDs they return point at real catalog entries. When search by meaning is unavailable, it falls back to word search and says so.

Not automatic yet: a check that compares each answer’s citations with the passages Shelby actually retrieved, and a link from each citation to its source.

How documents become searchable text

Each document’s text is transcribed in two grades.

Searchable floor
Machine-read text: good enough to find the page, not to quote.
  1. page image at 300 dpi or more
  2. machine reading (OCR)
  3. searchable text
Gold
A verbatim transcript: every word, marker and reaction.
  1. page image at 500 dpi or more
  2. visual transcription
  3. second machine check
  4. verbatim transcript
A fictional sample transcript. Nothing is summarized: a quoted reply nests above the reply that quotes it, and anything that is not text is bracketed and named.
Speaker B @ 09:16
> [quoted: Speaker A @ 09:14]
> Can you send the exhibit list before the call?
Sent. Two attachments below.
[IMAGE: exhibit list, page 1 of 3, four-column table]
[reaction: thumbs up · Speaker A]

Patterns, measured. How often two of the record’s ten most common patterns turn up in the same document. The counts are real; the names are stand-ins.

Drawing the Jaccard matrix…

Strongest pairs

  1. Motion Posture Witness SequencingJaccard 0.27
  2. Calendar Control Billing HygieneJaccard 0.23
  3. Witness Sequencing Billing HygieneJaccard 0.23
  4. Exhibit Chain Service of ProcessJaccard 0.17
  5. Calendar Control Deadline CadenceJaccard 0.16
FIG 3 The ten most common patterns in the record, under stand-in names; 22 of the 45 possible pairs shown, across 939 documents that carry a pattern. Strongest: Motion Posture and Witness Sequencing, sharing 26 documents (J = 0.27). The stronger the overlap, the stronger the cell; the strongest pair is outlined.
How the score works

The score, Jaccard similarity, divides the documents two patterns share by the documents that carry either one: 1 means they always appear together, 0 means they never do. Pairs that share fewer than two documents are not drawn.

Three checks, one queue. Every change passes three required checks, then waits its turn in the merge queue.

The merge contractA pull request must pass verify, protected code/config scan, CodeQL gate. It then lands through the merge queue (squash, one at a time) onto main, where a path-scoped Cloud Build trigger starts the service's deploy lane.CHANGEPull requestverifyprotected code/config scanCodeQL gateMERGE QUEUESquash, one at a timeall green, built on mainBRANCHmainCLOUD BUILDTriggerpath-scoped,one per serviceThe merge contractA pull request must pass verify, protected code/config scan, CodeQL gate, then lands through the merge queue onto main, where a path-scoped Cloud Build trigger starts the service's deploy lane.CHANGEPull requestRULESETThree required checksMERGE QUEUESquash, one at a timeall green, built on mainBRANCHmainCLOUD BUILDTriggerpath-scoped, one per service
FIG 4 The queue tests each change on top of everything already merged. AI review tools can comment but cannot block a merge. After a merge, each service’s own build deploys it.
The three checks, and the rules behind them
verify
Checks follow the change scope. Code changes run lint, types and tests; changes affecting the public app, cockpit or command worker also run that surface's build and checks, including applicable bundle budgets. Corpus-only changes run the record checks without the lint/test stack. Record checks cover manifest consistency, the ontology, freshness of gated tracked indexes, content-hash stamps, and identity and rendering audits. Corpus cross-reference freshness is advisory in ordinary verification.
protected code/config scan
Secret scanning on code and configuration, on every pull request and every queue entry. An advisory scan runs over the corpus beside it.
CodeQL gate
Three analyzers: GitHub Actions, JavaScript and TypeScript, Python. The gate passes only when all three finish.

The rules it was built around

  1. A search result points to a document; it is not proof of what the document says.
  2. A correct record can still be displayed wrongly, so displays are checked too.
  3. If a fact decides whether code merges, a machine checks it.

And every check must be one a change can pass, one that can actually fail, and one that fails only on a real problem.

What the lane refuses

Every service deploys through the same eight-stage lane. The numerals mark where it can refuse a deploy.

  1. Build the image, then a filesystem gate
  2. Scan Trivy, fixable high and critical findingsi
  3. Guard the main tip is unchangedii
  4. Lease one build per serviceiii
  5. Candidate tagged, at zero traffic
  6. Probe the exact commit, readyiv
  7. Promote full traffic, production probesv
  8. Finalize rollback held, lease released
MarkConditionCaught byResult
iA fixable high- or critical-severity finding in the imageTrivy, before pushThe build stops; nothing is published.
iiMain advanced with changes to this serviceThe main-tip guardThe build stops before publishing; the newer build deploys.
iiiAnother build holds the serviceThe deploy leaseThis build waits for the owner to finish.
ivThe candidate serves the wrong commit or fails readinessThe candidate probeTraffic never moves; production is untouched.
vA production probe fails after promotionPromote and finalizeThe prior revision is restored.
—Derived artifacts sit behind their sourcesverify, on the pull request and in the queueRefused until they are regenerated and committed.

Six services, one cloud. Each part of Stonewall runs and deploys on its own, in one Google Cloud region.

stonewall-publicPublic lane
stnwl.ai — this sitePublic lane
stonewall-cockpitPrivate lane
app.stnwl.ai — the operator cockpit and ShelbyPrivate lane
stonewall-portalPrivate lane
api.stnwl.ai — the workspace’s document API, behind a service keyPrivate lane
stonewall-botPrivate lane
Shelby on TelegramPrivate lane
knowledge-base-webhookCallbacks
Signed ingest callbacksCallbacks
chat-agent-service-c2e9Private lane
The operator’s Google Chat agent, bounded toolsPrivate lane
stonewall-commandPrivate lane · Cloudflare Workers
The operator’s command agent, behind Cloudflare AccessPrivate lane · Cloudflare Workers
Triggers, edge and uptime

The roster above is read from the one surface file CI probes live. One more surface runs beside the fleet on Cloudflare Workers, deployed by hand. The apex sits behind a global external HTTPS load balancer with managed certificates. The stnwl.ai DNS zone uses Google Cloud DNS; stonewall.esq retains Cloudflare DNS-only nameservers. Eleven uptime checks run at a five-minute cadence, with alert policies for uptime-check failures.

ServiceDeploy configTrigger
stonewall-publiccloudbuild.public.yamlstonewall-public-main-push
stonewall-cockpitweb/cloudbuild.cockpit.yamlstonewall-cockpit-main-push
stonewall-portalcloudbuild.yamlstonewall-main-push
stonewall-botscripts/stonewall_bot/cloudbuild.bot.yamlstonewall-bot-main-push
knowledge-base-webhookcloudbuild.webhook.yamlstonewall-kb-webhook-main-push
chat-agent-service-c2e9apps/chat-agent/cloudbuild.chat-agent.yamlstonewall-chat-agent-main-push
stonewall-commandapps/stonewall-command/wrangler.jsoncwrangler deploy, operator-run
The full stack
Languages
Python 3.12 / 3.13 / 3.14, by service · TypeScript · Node 24 · Markdown
Canon
catalog/manifest.md · Markdown + JSON · stdlib stonewall.py CLI
Data plane
Neon Postgres 17 · pgvector, HNSW · Row-level security
Embeddings
Gemini embedding, the embedding model · gemini-embedding-001, 1536 dimensions
Public site
Next.js 16 standalone · Distroless Node 24 · Self-hosted Geist + Newsreader · GA4, deferred
Corpus API
FastAPI · Uvicorn · Repo JSON + in-memory search index
Cockpit
Next.js 16 · React 19 · Tailwind · Clerk · AI SDK
Shelby
Anthropic Claude · Claude Fable 5.1, the deploy-lane pin · A smaller Claude tier for distillation
Voice and audio
Whisper · Gemini 2.5 Pro, listening · TTS-1-HD
Chat agent
Gemini 2.5 Flash on Vertex AI · Bounded tools, read-only SQL
Intake
pypdfium2 · Tesseract · extract-msg · Two-pass vision
CI
GitHub Actions · pytest · Vitest · Playwright · CodeQL · gitleaks · Trivy
Infrastructure
Cloud Run · Cloud Build · Artifact Registry · Cloud DNS · HTTPS load balancer · Terraform · Cloud Monitoring

The production bar. Five areas, each tied to the file or check that enforces it. The files are in our private repository.

Reliability
Zero-traffic candidates, rollback held, hashed assets, live probes.
Security
Secret and image scanning, CodeQL, an export guard, default-deny data.
Performance
Invariants in CI, per-route bundle budgets, self-hosted type.
Operations
One service per workload; triggers and merge rules as code.
Cost
Scope-aware CI, frozen lockfiles, duplicate runs cancelled.
Files and gates

Reliability

  • Zero-traffic candidate, rollback heldscripts/cloud_run_safe_promote.pyCloud Build probes, every lane
  • Content-hashed assetsscripts/asset_version_stamp.pyverify
  • Declared surfaces, probed livecatalog/intake/state/surfaces.tomlaudit_surface_state.py
  • Cockpit reads with the API coldweb/lib/api.tsbundled snapshot fallback

Security

  • Secret scanning.github/workflows/gitleaks.ymlprotected code/config scan
  • Static analysis.github/workflows/codeql.ymlCodeQL gate
  • Public export guardscripts/assert_public_portal_payload.pyfails the build
  • Operator boundaryweb/middleware.tsallowlist, fails closed
  • Default-deny data accessweb/migrations/postgres/0028-data-api-default-deny.sqlrow-level security
  • Security headers, pinned actionsnext.config.jsactionlint and zizmor in verify

Performance

  • Performance invariantsscripts/check_web_perf_invariants.pyverify
  • Per-route bundle budgetsweb/perf-budgets.jsonpnpm check:bundles
  • Self-hosted type, deferred analyticstests/test_public_perf_invariants.pyverify

Operations

  • Triggers declared in codeterraform/edge.tftrigger roster audit, live
  • Merge rules as codedocs/ruleset-policy.jsonapply_ruleset_policy.py
  • Gates tested on a clean treescripts/audit_gate_lane_parity.py--check and --canary

Cost

  • Corpus-only verify for prosescripts/detect_pr_change_scope.pyverify scope
  • Frozen lockfilesDockerfile.publicpnpm install --frozen-lockfile
  • Duplicate runs cancelled.github/workflows/cancel-runs-on-janitor-close.ymljanitor
  • Warm capacity per servicedocs/deployment-current-state.mdpublic 2 · portal, cockpit, bot 1 · webhook, chat agent 0
How it was built

How the system grew, corrected to how it runs today; the milestones follow.

  1. The catalog is the database. Every document has a numbered row: its type, its date, the people and patterns it involves, and whether it has been analyzed. A small command-line tool queries it, and the build checks the indexes against it.catalog/manifest.mdscripts/stonewall.py
  2. Verification is scoped to each change. A change to prose runs the quick record checks; a change to code runs everything. Names, displays and drift between pages each have their own audit.scripts/verify_all.pyscripts/pre_pr_check.py
  3. One file lists every live service. A single file names every live service, hostname, trigger and probe; the deployment docs are generated from it, and CI probes it live.catalog/intake/state/surfaces.tomlscripts/render_surface_docs.py
  4. Two grades of transcription. Page images at 500 dpi or better, transcribed and then checked by machine, give the verbatim transcripts; machine reading at 300 dpi or better is the searchable floor. Reply chains nest; nothing is summarized.references/bubble_render_doctrine.mdscripts/batch_sidecar_gold_uplift.py
  5. The workspace keeps reading when a service is down. Sign-in gated, with a fallback from the live API to a bundled snapshot to public data, so a document still opens when an upstream service is cold.web/lib/api.tsscripts/build_cockpit_operator_snapshot.py
  6. A scan flags pages that cite changed code. An on-demand scanner lists the pages that still cite changed paths or names; a scheduled documentation pass opens ordinary pull requests through the queue, and a janitor keeps one per topic.scripts/docs_api_drift.py.github/workflows/close-duplicate-doc-prs.yml
  1. Groundwork: the document reader, task reading lists, duplicate detection and drift checks.
  2. Google Cloud Run becomes the deploy plane.
  3. The merge queue settles on main.
  4. The last external hosts retire; stnwl.ai DNS moves to Google Cloud DNS. The compatibility domain retains Cloudflare DNS.
  5. The public site is consolidated into fewer pages.
  6. A sixth service joins the fleet: the operator’s chat agent.
  7. Review threads leave the merge gate; three checks and the queue are the whole contract.
  8. Every service deploys through the guarded lane.
  9. Search by meaning moves to Google’s Gemini embeddings; the record is re-embedded.

These checks describe how the system is built; they are not a certification.

Go deeper. The demonstration, the founders, and what the record shows.