Internal R&D

Case Study — GetDev Labs Build Log

Company Brain

We kept hearing the same brief in different words: "we have the data, we have the models, why can't the AI just do the job." We wanted to find out where that breaks before we quoted anyone for it.

GBrain (MIT)

FastAPI

Postgres + pgvector

Frontier LLM API

Next.js console

What this is

An operator console that sits next to a support team, drafts the next action on every incoming case, cites where each part of the draft came from, and records whether the human agreed. Skills — the procedures it runs — are induced from the company's own resolved cases rather than written by hand.

We built it because we kept getting the same brief from clients in different words. We wanted to find out where it breaks before we quoted anyone for it.

Operator console showing a drafted refund action with a provenance ribbon connecting one step to its evidence — The review. Every drafted action traces to the evidence that produced it — hover a step, follow the line. Northline Supply is a synthetic company built from our test corpus; the numbers shown are from the prototype, not from a customer.

1. We started building an autopilot. We shipped a copilot.

The first version executed. It worked in the demo and was unusable in practice, for a reason that took a week to accept: it was right most of the time and there was no way to know which times. The number that killed it was consistency, not accuracy — run the same case five times, get four identical answers and one that quietly picks a different refund path. Averaged over a queue that reads as 85% accurate. To an operations manager it reads as "I have to check all of them," which means the system saves nobody anything.

So we inverted it. The agent proposes, a human presses one key, and we log the disagreement. It sidesteps the reliability problem, because a human is the gate. It sidesteps the security conversation, because nothing autonomous touches a production system. And every keystroke is a labelled training example — the product generates its own evaluation set by being used. That last one is why the copilot isn't a stepping stone to the autopilot. It's the machine that builds the autopilot.

2. We wrote the first skills by hand, and stopped pretending that was a shortcut.

The plan was auto-induction from day one: cluster the resolved cases, prompt a model to synthesise the procedure, done. What came back were plausible-sounding documents that were subtly wrong in the way that matters — they described the happy path and had no idea where the cliff edges were.

The fix was to stop feeding it only successes. A procedure induced from clean resolutions has no concept of when to stop; add the cases a human escalated and it learns the boundary. The escalated cases are the highest-signal data in the whole corpus, and they were sitting unused in the ticket system, flagged and ignored. We hand-wrote the first skills anyway — read fifty cases, drafted with an LLM, corrected. It felt like cheating. It wasn't: you can't automate authoring a thing until you know what a good one looks like.

3. Every claim carries its source, and that turned out to be the whole product.

Early builds showed the drafted action and nothing else. Operators didn't trust it and couldn't say why. When we added provenance — hover a step, see the exact ticket, Slack message or SOP clause it came from — trust moved immediately.

Mechanically this is a chain-of-custody problem, not a UI problem. Every ingested event is stored append-only and never mutated; facts derived from it keep a pointer back. If you extract knowledge and throw away the span, you can't answer "why did it believe that" three weeks later when it gets something wrong — and it will get something wrong. We under-budgeted this. It looks like plumbing and it's the debugger for the entire system.

Evidence page showing compiled truth on top of an append-only timeline, with an untrusted buyer email entry dated 11 August — The evidence page. Current best understanding on top, the record that produced it underneath, unrewritable. The entry dated 11 August is a buyer email asserting a 90-day returns window — recorded, searchable, and marked untrusted, so it's structurally incapable of changing what a skill does.

4. Time is not metadata.

We shipped a version that answered questions about February cases using March's policy. Confidently. It took a client-side reviewer about four minutes to find it. Every edge in the knowledge store now carries two clocks: when the fact was true in the world, and when we learned it. Superseded facts are marked invalid, never deleted — more annoying than it sounds, since entity resolution across systems is already ugly, and a validity interval on every assertion roughly doubled the complexity of the write path. It's also non-negotiable in anything that gets audited: "which version of the policy applied on the date of this case" is the first question a compliance reviewer asks.

5. We built our own retrieval layer. Then we deleted it.

Hybrid retrieval — BM25 plus vectors, fused with reciprocal rank fusion, then a rerank. We'd built this before, from scratch and without a framework, so building it again was tempting and we did it anyway. It worked. It also produced nothing a competitor couldn't reproduce in a fortnight.

We threw it out and put GBrain underneath instead — MIT-licensed, the memory layer Garry Tan runs his own agents on. It had already made four decisions we'd have had to make ourselves: the git repo as system of record with history/diff/revert for free; compiled truth above an append-only timeline; graph extraction with no LLM in the write path; brains and sources as separate axes for scoped collaboration.

What we kept from our own work is the part that mattered: recall was never the constraint, precision was, and precision came from the structure around retrieval. Structure beat embedding quality by a wide margin. GBrain gave us the substrate; the case-shaped structure on top of it is ours. Deleting working code you wrote is unpleasant. It was still the right call — it moved a meaningful share of our planned codebase into a dependency and freed the time for the induction and eval work nobody gives you for free.

Where GBrain stops

Worth being precise about — "we used GBrain" is not the same as "GBrain does this."

Capability	Comes from GBrain	We built it
Hybrid retrieval, RRF, rerank	✓
Typed knowledge graph, zero-LLM extraction	✓
Compiled truth + append-only evidence trail	✓
Page versioning, diff, revert	✓
Team mounts, scoped access	✓
MCP server into Claude Code and friends	✓
Brain health: stale pages, orphans, dead links	✓
Bi-temporal validity — when true vs. when learned		✓
Cross-system trace assembly into episodes		✓
Outcome labelling		✓
Skill induction from traces		✓
Replay evals, agreement rate, pass^k		✓
Trust levels on ingested spans		✓
Capability grants, spend caps, autonomy ladder		✓

The top half is a solved problem someone gave away. The bottom half is the entire reason this is a project and not an afternoon. Two rows cost real time: bi-temporality (GBrain versions a page but doesn't natively answer "which policy was in force on this date" — we added a validity interval and reconcile on read), and multi-tenancy (GBrain's design centre is one operator's brain, not infrastructure for thousands of end users — a non-issue for a lab build and early deployments at one brain per client, a real decision at scale).

6. The system reads untrusted text and writes executable instructions.

Support tickets are written by strangers. Skills are executable. If ingested content could author a skill, an attacker could email you a new refund policy. Every evidence span carries a trust level from the moment it lands — external content can inform a draft, it can never originate a constraint or an action. Human-in-the- loop gives you a second gate almost for free, which is the argument for shadow mode we'd have missed if we'd rushed to autonomy.

7. Autonomy is earned per skill, not granted per system.

There's no global "turn on the AI" switch and there never will be. Each skill sits on a five-rung ladder from suggest-only to autonomous, and it climbs on evidence: agreement rate on live cases, consistency across repeated runs, whether it escalates the right things, and how bad the worst possible execution is. It falls automatically when live agreement degrades.

Paired with that: capability grants. An agent never has "access to the refund system." It has permission to issue credit notes up to a per-case cap and a cumulative hourly cap, expiring in days, revocable by any of three named people — borrowed wholesale from how crypto treasuries handle multisig, minus the crypto, and the single reason a finance lead will sign off on any of this.

Weekly report showing a skill's live agreement rate falling and the skill demoting itself — The most useful week we've had. A GST rate change silently broke invoice disputes; live agreement fell from 0.88 to 0.71 and the skill demoted itself before a human noticed. Continuous measurement caught it — a one-time certification never would have.

What broke

Case studies that only contain wins are advertisements.

Stitching a case across four systems

A single episode is a WhatsApp thread, a ticket, a credit note in the accounting system, and an outcome that only exists as the absence of a reopen. Nothing shares an ID. This was most of the engineering and none of the demo — and it's the part a competitor can't copy, because it's different in every company.

Outcome labels

An episode without a known outcome is close to worthless for induction. Only a minority of historical cases had anything you could honestly call a label. Half the work was inferring outcomes from side effects — a reopened ticket, a reversed credit note, a second complaint from the same account within fourteen days.

The political layer

The system surfaced that the written returns policy and the team's actual behaviour diverge on roughly a quarter of damaged-goods cases. That's the most valuable output it produced and the most uncomfortable. Findings that implicate a named person's discretion now route to that person first, as a draft.

Latency

An operator who can resolve a ticket in forty seconds will not wait nine seconds for a draft. Retrieval, rerank, generation and provenance assembly all have to fit inside the time it takes them to read the customer's message, or they route around you. We cut ours materially by pre-computing retrieval on ticket arrival rather than on operator open.

Sources page showing four connected systems and a 71% case reconstruction rate — Four sources in, 71% of cases reconstructed end to end. The 12% that never link were resolved on the phone: the ticket says “called buyer, sorted” and nothing else. We surface that rather than hide it — it's a decision the customer has to make, not a number we should be embarrassed by.

Open questions page generated by comparing the written policy against actual team behaviour — Questions generated by comparing the written record against actual behaviour, routed to the person most likely to know. The first has been open six days, which is itself informative — the uncomfortable questions are the valuable ones, and they're the ones that sit.

The same brain, three other places it shows up

Build the measurement before the model. Agreement rate — does the agent's draft match what the human actually did — is the only number that moves a conversation with a buyer, and it takes an afternoon to instrument. We'd also pick a narrower process next time: "support" isn't a process, "damaged-goods returns on trade accounts" is.

Incident response at 2:47am, a runbook assembled from two prior similar incidents — On call three weeks, an alert he's never seen, a runbook assembled from the two prior times it happened. “Don't restart Postgres. Someone did, in March” is negative knowledge that exists nowhere else in the company.

A coding agent checking the brain over MCP before implementing a retry policy — Asked to add a retry, the agent checks the brain first and finds two incidents saying a naive retry makes it worse. The codebase says what the worker does; it never said why the retry policy is shaped like this.

Merging a personal brain into a team mount, showing five conflict types with two requiring human review — Merging a personal brain into a team mount. Five conflict types, only two needing a human — the Friday-deployment case is contextual rather than contradictory, and collapsing it into one averaged rule would destroy information.

Where it stands

Running on a small set of skills across the damaged-goods returns process, in shadow mode. Agreement is climbing week over week as corrections feed back. Nothing autonomous yet, by design — the top skill needs more consistency before it graduates, and we'd rather it took three more weeks than reverse a credit note we shouldn't have.

Prior work this leans on: Cortex, our agentic retrieval system — hybrid BM25 and vector search with RRF fusion, LLM reranking, multi-step planning and self-correction, written without LangChain or LlamaIndex. The retrieval and planning substrate here is a descendant of it.

If you're carrying knowledge in your team's heads and want to know what it would take to get it out — that's the conversation we like having.

Ready to Get Started?

Book a free discovery call and let's explore how Dev can accelerate your business with AI-first solutions.