The AI Retrofit Log: Six Features We Added to Products Already in Production (and the Two We Turned Off)

What makes an AI retrofit different from an AI build

A retrofit adds AI to a system that already has users, data, a fixed schema, a signed privacy policy, and an ops team with habits. You cannot design around the model. You insert it into an existing code path, budget its latency against a page that already works, and keep the old path alive as a fallback.

Greenfield AI work lets you shape the data model around the model's needs. Retrofits do the opposite. On GlycanAge's platform, which we have maintained for six years, the consent scope was defined by a privacy policy written long before anyone shipped an LLM feature, so the question was never "what could the model do with this data" but "which fields are we permitted to send at all". On Plodovi's marketplace, six interconnected systems already exchanged product data through settled contracts, so adding AI-generated copy meant writing into an existing field with existing validation, not adding a new one.

The practical consequence is that most of the engineering in a retrofit is not the prompt. It is the insertion point, the timeout, the fallback, the review gate, and the off switch. We treat every one of those as a named requirement before writing the first call.

• • No schema changes: AI output lands in existing fields or a sidecar collection, never a migration on a live table
• • Consent scope is fixed: if the current policy does not cover third-party processing of a field, that field is redacted or the feature is scoped down
• • The old path stays: every AI call has a deterministic fallback that produced acceptable results yesterday
• • Ops owns the workflow: features are built for the non-engineer who will actually run them daily
• • Every feature ships with an acceptance threshold and a documented condition for turning it off

The six retrofits, feature by feature

Six features across three live products: support ticket triage, product description drafting, order note parsing, community event description cleanup, internal document search, and result-report summarisation for internal reviewers. Each was inserted behind an existing entry point, given a latency budget, and shipped with a fallback and a human gate.

Support ticket triage sits on the marketplace's inbound queue. The call happens asynchronously after the ticket is written to the database, so the customer-facing write path is untouched and the added latency is invisible to the person submitting. The model returns a category and a suggested priority; the ops team sees them as suggestions on the ticket, never as an automatic routing decision. If the call fails or exceeds four seconds, the ticket lands in the default queue exactly as it did before.

Product description drafting is a batch job, not a request-path feature. Retailers submit products through the retailer portal with a short title and attributes. Overnight, unpublished products get a drafted description written into a draft field, and the retailer approves or edits before publish. Nothing publishes without a human click. Order note parsing turns free-text delivery instructions into structured hints for the logistics system, with a strict rule: if confidence markers are missing or the output fails schema validation, the raw note passes through to the driver app as plain text, which is what it did before the feature existed. Event description cleanup in the community app is an editor-side assist that the organiser triggers explicitly, so latency is user-initiated and a spinner is acceptable. Internal document search is retrieval over an internal knowledge base with citations back to the source document, used only by staff. Report summarisation drafts a plain-language paragraph for an internal reviewer, who edits and signs off before anything reaches a customer.

• • Async post-write triage: no added latency on the user path, suggestion-only output, default queue on failure
• • Overnight batch copy drafting: writes to a draft field, human approval required to publish
• • Free-text note parsing: schema-validated, raw text passthrough on any validation failure
• • User-triggered editor assist: explicit action, visible loading state, original text always recoverable
• • Staff-only retrieval search: answers cite source documents, no generation without a retrieved passage
• • Reviewer-facing summary draft: never customer-visible until a named human approves it

Latency budgets, cost per call, and where they come from

Set the latency budget from the existing page, not the model. If a screen renders in a few hundred milliseconds today, an AI call on that path is either asynchronous, cached, or not on that path at all. We classify every retrofit as request-path, user-triggered, or batch, and each class gets a different budget and a different timeout.

Request-path features are the hardest and we avoid them where we can. When one is unavoidable, the call gets a hard timeout well under the page's existing budget and the fallback renders the pre-AI experience rather than an error or an empty state. User-triggered features can afford seconds because the user asked and is watching a spinner, but they still need a cancel path and a way to keep the original content. Batch features have effectively no latency constraint and should absorb as much of the workload as possible, which is why we moved product copy generation to an overnight job instead of generating on submit.

Cost is modelled the same way: per call, then per month at real volume, then compared against the manual cost of the same work. The variable that dominates is not price per token, it is how many calls the feature makes per unit of business activity. A feature that fires once per published product is cheap and predictable. A feature that fires on every keystroke, every page view, or every retry is where budgets break. Before shipping, we write the expected monthly call count, the token ceiling per call, and a hard spend cap that disables the feature and falls back rather than silently overspending.

• • Request-path: timeout under the page's existing budget, deterministic fallback, cache aggressively
• • User-triggered: seconds are acceptable, but always cancellable with the original preserved
• • Batch: no latency constraint, batch prompts, run off-peak, retry with backoff
• • Cost model: calls per business event x tokens per call x volume, plus a hard monthly cap with automatic fallback

The two features we turned off, and why

Two features shipped and were later removed. One was an automated first-response draft for support that cost more per resolved ticket than the human handling it replaced. The other was a longer-form generated content feature whose output quality drifted below the review threshold, so reviewers spent more time correcting drafts than writing from scratch.

The support draft looked good in testing. In production, the arithmetic went the wrong way. Only a minority of tickets were resolvable by the drafted response; the rest still needed an agent, who now also had to read and discard a draft. The cost per genuinely resolved ticket, once you divide total spend by the tickets that actually closed without human editing, exceeded what the ops team spent handling the whole queue manually. That is a measurable threshold, so we removed the feature rather than tuning it indefinitely. The triage suggestion feature, which is cheaper and does not attempt to write the reply, stayed.

The content feature failed differently. We set an acceptance criterion up front: the share of drafts a reviewer could publish with only light edits. It cleared the bar at launch and slipped over the following weeks as inputs got more varied than the ones we prompted against. Once reviewers were rewriting more than they were approving, the feature was net negative work and we turned it off. The general lesson is not that generation is unreliable. It is that a retrofit needs a numeric acceptance criterion agreed before launch, and someone whose job it is to check it, otherwise a degrading feature just quietly becomes overhead.

The constraints that decide architecture in a live product

Four constraints decide most retrofit architecture: you cannot change the schema, consent scope is already fixed by an old privacy policy, PII must be redacted before the call, and the workflow is owned by an ops team rather than engineers. Each one removes options before any model is chosen.

The no-schema-change rule is practical, not dogmatic. On a system with live users and multiple consuming services, a migration to accommodate an experimental feature is a large blast radius for something you might turn off in a month. So AI output goes into a sidecar collection keyed by the existing record ID, or into an existing draft or notes field with a source marker. Removing the feature then means deleting a collection, not reversing a migration. Consent works the same way. If the policy users agreed to does not describe sending a field to a third-party processor, the honest options are to redact the field, restrict the feature to internal use, or update the policy and re-consent. On health data we default to the first two. Redaction happens server-side before the outbound call, and what leaves the system is a reduced payload with identifiers replaced by placeholders that are restored on the way back.

The ops ownership constraint changes the interface more than the backend. If the person running the workflow is a coordinator, not an engineer, the feature needs a visible on/off toggle in the admin panel, an obvious way to see what the model produced versus what a human wrote, and no state where the system does something irreversible on the model's say-so. In practice that means AI output is always labelled, always editable, and always preceded by a human action for anything customer-facing.

The retrofit checklist we run before shipping any AI feature

Before any AI feature goes into a production codebase we answer eight questions in writing: insertion point, latency class, fallback behaviour, redaction rule, review gate, acceptance criterion, monthly cost ceiling, and removal procedure. If any answer is missing, the feature is not ready, regardless of how good the demo looked.

The removal procedure matters more than it sounds. Two of six features here were removed, and the reason that was cheap is that each was designed to be reversible: sidecar storage, an ops-facing toggle, and an untouched original code path that never stopped working. A retrofit you cannot undo in an afternoon is a retrofit you will tolerate long after it stopped paying for itself.

None of this requires a model-first architecture. It requires treating the model as one unreliable dependency among several in a system that already works, with a budget, a timeout, and a documented condition under which you switch it off.

• • Insertion point: exactly which existing function or job the call is added to, and what it does not touch
• • Latency class: request-path, user-triggered, or batch, with a numeric timeout
• • Fallback: the deterministic behaviour that fires on timeout, error, or failed validation
• • Redaction rule: which fields leave the system, which are replaced by placeholders, and where restoration happens
• • Review gate: the named human step before anything is customer-visible or irreversible
• • Acceptance criterion: the number that decides whether the feature is working after launch
• • Cost ceiling: expected monthly spend, plus an automatic cutoff that reverts to fallback
• • Removal procedure: what has to be deleted or toggled to take the feature out, and who can do it

For the provider side of these same features, see our model selection matrix, which covers how consent scope, volume, and self-hosting decided OpenAI versus Anthropic versus a local model.