second open
← All case studiesPrevious work · 2025 to now

Multi-agent AI platform

An AI sales assistant and growth suite for online merchants. It answers shoppers on the merchant's site and on WhatsApp, looks up orders, fills carts and hands over to a person, then turns every closed conversation into leads and market insight. We architected the AI system from zero.

Role
AI engineer, architecting the AI system
Surface
Site widget, WhatsApp, merchant web app
Status
In production, paying customers
Company
Unnamed here by choice
77.5%of input tokens served from prompt cache, over 3,276 chat turns
157golden eval cases; the checks written in code gate the build
120klive chat sockets held by one 8-core node, load-tested
31AI operations metered and priced per call
The challenge

A wrong assistant costs a sale. An expensive one costs the business.

Merchants pay a monthly plan; the assistant pays per token. One chat turn carries about 67,000 input tokens of catalogue, policy and history for roughly 500 tokens of answer. It had to be right about products and orders, safe with side effects like carts and emails, and cheap enough per turn that the plan still made money.

What we did

Retrieval first, tools second, and the bill measured from day one.

Every shopper turn starts with a forced knowledge search, lets the model choose from twelve tools, then forces a written reply. Closed conversations go through up to six structured calls. Every call is logged with its tokens and priced from a table, so the cost of any feature for any merchant is one query away.

How one chat turn works

  1. 01
    Every turn retrieves first
    Step zero is always a knowledge search: pgvector and full-text search fused by rank, filtered by price, stock and category before scoring.
  2. 02
    The model picks from twelve tools
    Order lookups behind a one-time code, cart changes, lead capture, escalation to a person. Up to six tool steps.
  3. 03
    The last step must answer
    No tools are allowed on the final step, so every turn ends in a written reply, streamed to the widget over a socket.
  4. 04
    One fallback, never after a side effect
    If the primary model fails before its first token, the turn retries once on the fallback. Never after a cart change or an email.
  5. 05
    Every call is metered
    Tokens logged per merchant, widget and operation, priced from a table, drawn from the plan allowance and then a prepaid wallet.
  6. 06
    Closed chats become insight
    Queued in Postgres, each conversation is classified, summarised, qualified as a lead and searched for demand the shop could not meet.

The four tests, on this project

We judge every product on the same four questions. This is how this one answered each of them.

01 · Design
Does it feel right?
  • Replies stream over a socket that stays open from the AI engine to the widget.
  • A person can take over live, and the AI goes quiet while they hold the chat.
  • Proactive greetings cached per page, so they stay instant and never cost a model call per pageview.
02 · Mobile and backend
Does it hold up?
  • Analysis jobs written in the same transaction that closes the chat, claimed with SKIP LOCKED.
  • No model fallback after a side effect: a cart change or an email never runs twice.
  • Load-tested: 500+ clients pooled onto 50 database connections, webhook bursts of 8,000 a second with no failures.
03 · AI, retrieval and evals
Led here
Is it right?
  • Hybrid search with a similarity threshold lowered from 0.5 to 0.35 after measuring what it missed.
  • Every number in an analyst answer is checked against the data that turn read. If it fails, it is regenerated once.
  • The merchant id comes from the session, never from the model, so no answer can reach another merchant's data.
04 · Metering and pricing
Does it pay?
  • Six models tiered by task: the smallest writes thread titles, the largest only where it earns its place.
  • The system prompt kept byte-identical across turns, with the date moved to the end, so the cache holds.
  • A token wallet on an append-only ledger, with alerts at 80, 90 and 100% of the plan.

Three decisions we made

01
Retrieve first, every turn
Step zero is always a knowledge search. It takes 3 to 9% of a turn's time and removes a whole class of confident guesses about products.
02
Code checks gate, model judges advise
Grounding and routing are scored by code and can fail the build. The model-judged score cannot: a drifting judge that fails builds is worse than no judge.
03
Budgets from percentiles
The forced first step is capped at 2,048 output tokens, set from real traffic: p50 of 190, p95 of 704, p99 of 4,605.

What we would buy today

About half of it. Tracing today is logs and dashboards wired by hand; now we would buy an LLM observability tool. The eval runner would be a product, not a script.

What we would build again: the step policy, the grounding check and the usage ledger. No tool does those the way a merchant's bill needs them done.

What this means for you
If you are putting an assistant in front of customers
We have built the parts that decide whether it is right, safe and affordable, and measured them in production. In two weeks we can tell you which of them you are missing, and build the one that matters most.
Closest track: Fix retrieval, add evalsSee the two weeks
What we can build for you
  • Hybrid retrieval, measured and scoped per customer
  • Tool-calling agents that are safe with side effects
  • Eval sets that gate your CI
  • Metering, token wallets and model tiering
  • Prompt-cache layout that cuts input cost
Stack
TypeScriptNestJSVue 3Vercel AI SDKOpenAIGeminipgvectorPostgreSQLRedisRabbitMQDocker SwarmTerraform

Want this in your product?

Thirty minutes, no deck. Tell us what you are building and which of the four tests worries you most. We say no when we are not the right people.

Book a 30-minute callNext: Voice AI writing app