Shipping AI Features Without an ML Engineer: The Four Ownership Roles We Assign Instead

Why "we don't have an ML engineer" is the wrong blocker

Most companies shipping LLM features do not need an ML engineer. They need someone accountable for model behaviour. Almost every AI feature we have built for clients calls a hosted model over HTTP, so the real work is prompt design, output evaluation, cost control, and failure handling, not training or fine-tuning.

The confusion is understandable. Job listings and conference talks describe AI work as model selection, embeddings, and evaluation harnesses, which sounds like a specialist discipline. In practice, when the model is a vendor API, the engineering surface looks like any other third-party integration: a request, a response you cannot fully trust, a bill, and a dependency that can degrade without notice. That is work a competent backend developer already does for payment gateways and shipping APIs.

What genuinely breaks is ownership. An integration with a payment provider has an obvious owner because it fails loudly. An LLM feature fails quietly. The output is still a paragraph of plausible English. Nobody gets paged. So the failure mode of AI features in small teams is not incompetence, it is that the prompt sits in a repository with no name attached to it and slowly stops matching the product around it.

The four roles: Prompt Owner, Eval Owner, Cost Owner, Fallback Owner

We split responsibility for every production AI feature into four named roles, each held by a specific person, not a team. Prompt Owner controls the wording and versioning. Eval Owner decides whether output quality is acceptable. Cost Owner watches spend per call and per month. Fallback Owner defines what the product does when the model is slow, down, or wrong.

Two things matter more than the exact split. First, the names go in the repository, in a short OWNERS block at the top of the prompt file or in the feature's README, with a date. Second, one person can hold more than one role on a small feature, but no role can be unassigned. On a MOVE & MEET feature, one backend developer held three of the four; the Eval Owner was the community manager, because she was the only person who could tell whether a generated event description sounded like the brand.

The roles are not equal in difficulty. Prompt Owner and Cost Owner are learnable in an afternoon. Fallback Owner is standard engineering. Eval Owner is the hard one, and it is the role that has to belong to someone who understands the domain rather than the stack.

• • Prompt Owner: owns the prompt text, its version history, and the changelog entry for every edit. Approves any change that alters instructions, tone, or output schema.
• • Eval Owner: owns the eval set and the pass bar. Signs off before a prompt or model change reaches production. Usually a domain person, not a developer.
• • Cost Owner: owns the per-call and monthly budget, the alert thresholds, and the decision to downgrade model tier or cache more aggressively.
• • Fallback Owner: owns timeouts, retries, degraded modes, and the rule for what the user sees when the call fails. Owns the kill switch.

The eval spreadsheet that replaced a data scientist

Our standard eval artifact is a spreadsheet with one row per test input and one column per prompt version. A domain person scores each output pass or fail against written criteria, and a prompt change ships only if it does not regress any previously passing row. It is not sophisticated, and it catches the problems that actually occur.

Building it takes a day. You collect 30 to 60 real inputs from production or from the client's existing manual process, deliberately weighted toward the awkward ones: the product with a missing description, the customer question that mixes two topics, the record with a typo in a name. You write the pass criteria in plain sentences before you look at any output, because criteria written after the fact always describe what the model already produced. Then the Eval Owner scores every row for the current prompt, and that becomes the baseline.

The discipline is in the regression rule, not the scoring. Every prompt edit produces a new column. If a row that passed on version 4 fails on version 5, version 5 does not ship, even if it improves ten other rows. That rule is what stops the slow quality drift that nobody notices until a customer complains. It also gives a non-technical Eval Owner real authority: she is not arguing about model behaviour in the abstract, she is pointing at row 23.

What a backend developer could judge, and what they could not

Backend developers reliably judge structure: whether the output is valid JSON, whether required fields exist, whether values fall inside allowed enums, whether length is within bounds. They cannot reliably judge whether the content is correct, appropriate, or on-brand in a domain they do not work in. That line decided who we assigned as Eval Owner on every feature.

On Plodovi, a farm-to-table marketplace, generated product copy passed every structural check a developer could write. What a developer could not see was that the model kept describing seasonal produce in ways that implied year-round availability, which is exactly the claim a short supply chain platform must not make. The person who caught it was the operations lead who talks to growers. After that, the Eval Owner on that feature was permanently a marketplace operations person, and the developer's job narrowed to enforcing the schema and shipping the harness.

In regulated work the split is sharper. On GlycanAge, anything touching interpretation of results sits behind human review by design, and the eval criteria are written by people who understand what a clinician can and cannot say. A developer can verify that the pipeline redacted personal data before the call and that the output never reaches a customer without a review step. A developer cannot sign off on whether a phrasing crosses from explanation into medical advice, and should never be asked to.

Two prompts that drifted, and what it cost

Both production incidents we have had with AI features came from prompts with no assigned owner. In one case the surrounding data model changed and the prompt kept referencing a field that no longer existed. In the other, a model version upgrade changed output formatting and nothing failed loudly, so the degraded output ran for weeks.

The first was on a marketplace copy feature. A category restructure renamed and merged some product attributes. The prompt template still interpolated the old attribute, which now resolved to an empty string, so the model started inventing plausible-sounding attributes to fill the gap. No exception was thrown; the JSON was valid. It surfaced when a retailer noticed a claim about their own product that they had never made. The fix took an hour. The lesson was that prompt files must be listed as consumers in the same place any other consumer of a data model is listed, so a schema change forces a review.

The second was subtler. A provider's default model pointer moved to a newer version, and the newer version started returning markdown emphasis inside a field that was rendered as plain text. Users saw asterisks. Nothing alerted because the response parsed fine and stayed inside length limits. After that we pinned model versions explicitly, added a rendering assertion to the eval sheet, and made "a model version bump is a prompt change" a rule: it goes through the Eval Owner like any edit.

When we told a client not to build it

Some AI features require expertise the client does not have and should not hire for a single feature. The test we apply: if a wrong output causes harm that is invisible to everyone on the team, and the correct answer requires a specialist to verify, the four roles are not enough and the feature should not ship. We have said this more than once.

The pattern that fails this test most often is anything that produces a confident interpretation in a domain where being wrong is expensive: health guidance, legal or regulatory phrasing, financial recommendation, and anything a customer would reasonably treat as authoritative. In those cases the Eval Owner role has no valid holder inside the company. Hiring a specialist to review each output turns a low-cost automation into an ongoing labour cost with worse economics than the manual process it replaced.

The alternative we usually propose is to keep the model on the side of the work where errors are cheap and visible. Draft generation with mandatory human editing before publication. Classification and routing, where the fallback is a human queue. Extraction from documents, where a person confirms the extracted fields. Same model, same integration cost, and the Eval Owner role can genuinely be held by someone already on staff.

How the roles transfer at handover

At handover we assign each of the four roles to a named client employee, walk them through their specific artifact, and leave a one-page runbook per role. Nothing is handed to "the team". If a role has no plausible holder on the client side, we either keep it under a support arrangement or recommend turning the feature off.

The handover is uneven by role, and we plan for that. Cost Owner transfers cleanly: it is a dashboard, two alert thresholds, and a documented decision about what to do when spend crosses them. Fallback Owner transfers to whoever already owns on-call, because the artifacts are timeouts, retries, and a kill switch that they can test in staging. Prompt Owner transfers to whoever owns the feature in the product, along with the rule that no prompt edit merges without a changelog line and an eval run.

Eval Owner is the role that decides whether the handover is real. It has to go to a domain person with time protected for it, and it needs a standing cadence, not a promise to check occasionally. Where a client has taken that seriously, the feature is still running and the eval sheet has columns we never wrote. Where nobody claimed it, the honest recommendation is to reduce the feature's autonomy: put a human gate in front of the output, so drift produces a small annoyance rather than a customer-facing claim nobody signed off on.

For the feature-level detail behind these engagements, including latency budgets, cost per call, and the two features we turned off, see our AI Retrofit Log.