AI customer service the model cannot over-promise on

ConciergeAI

If the model can issue the refund, then one prompt injection away, it will. So the model never holds the authority to commit the business.

Role: Sole engineer. Research, architecture, policy engine, guardrails, adversarial suite.
Shape: Next.js and TypeScript, roughly 12k lines, with a full research to SRS to HLD to LLD design set.
Links: Private repo

ConciergeAI restaurant demo explaining that allergen answers are read from a signed-off ingredient record and never generated by the model.

Architecture

AUTHORITY BOUNDARY

The problem

Four public incidents shape almost every decision in this codebase. A tribunal found an airline liable after its chatbot invented a bereavement-fare policy, rejecting the argument that the bot was a separate legal entity responsible for its own actions. A dealership assistant was talked into agreeing to sell a vehicle for a dollar and calling it a binding offer. A courier's bot was jailbroken into writing a poem attacking its own employer. A fintech announced an assistant doing the work of hundreds of agents, then reversed and rehired humans, naming quality and the absence of reliable escalation as the cause.

The common thread is not that the models were bad. It is that each system granted the model authority it should never have had, and then tried to constrain that authority with instructions. Instructions are not a security boundary.

The approach

The architecture puts a deterministic policy engine between understanding and action. The model reads the customer, retrieves context, and proposes an action. A policy engine with no model inside it decides. The model is then handed the decision and narrates it back to the customer.

The consequence is the whole point. If the model hallucinates that it has refunded four hundred pounds, no refund happened, because it never had the capability to move money. The failure degrades from an unauthorised financial commitment into a contradiction in the transcript, which the output rail then catches and escalates.

Retrieval is deliberately verbatim-only. The assistant can quote a policy but cannot paraphrase one into existence, which closes the invented-policy failure mode directly rather than hoping a system prompt holds.

Decisions

Allergens are safety-critical, not an intent

A peer-reviewed evaluation found that of 41 sources a chatbot supplied for food-allergy questions, only 39 percent were accurate. The rest were expired pages, miscited articles, and fabricated journal references. So the system never generates an allergen answer, never infers one from a dish name, and never asserts that a dish is safe. The only permitted speech act is that the records list a given ingredient, always with the cross-contamination warning, always escalating on uncertainty.

The money is recomputed server-side

The model proposes a refund; it never supplies the amount that gets processed. The engine recomputes the figure from the order record. Combined with idempotency derived from the tool-call id, a retry cannot produce a double refund, and a manipulated conversation cannot produce a manipulated number.

Escalate on leading indicators, not on anger

Most systems hand over once the customer is already furious, which is the point at which the handover no longer helps. The policy engine evaluates leading indicators first, such as repeated reformulation and failed resolution attempts, and treats sentiment last. Sentiment is a lexical heuristic, and it is deliberately the weakest signal in the chain rather than the trigger.

The adversarial corpus is frozen

Attack cases are committed, not regenerated per run. If the corpus were generated fresh each time, a regression and a newly-discovered attack would be indistinguishable in CI. Freezing it means a red build always means something broke. Tests assert on structure and tool-call shape against a mocked model, because asserting on prose is how you get a test suite that fails on a synonym.

Evidence

9: enforced controls, each traced to an incident
0: actions the model can take unilaterally
3: adversarial suites: injection, exfiltration, PII
1: environment variable from memory to Postgres

What it does not do

Retrieval is lexical only. BM25 with structural chunking is strong on exact identifiers and policy clauses, and weaker on paraphrase.

Streams are not resumable, so backgrounding a mobile browser mid-answer loses that answer.

Operator auth is a shared secret, so the audit trail records that an operator acted, not which one.

The Postgres rate limiter reads its bucket without a row lock, so two exactly simultaneous requests can both be admitted at a burst edge.

ConciergeAI

Architecture

The problem

The approach

Decisions

Allergens are safety-critical, not an intent

The money is recomputed server-side

Escalate on leading indicators, not on anger

The adversarial corpus is frozen

Evidence

What it does not do

Other systems

Anvaya

AegisQuery

Dialoft

Virtual Buddy