OpenAI vs. Anthropic vs. a Local Model: The Selection Matrix We Ran for Three Client Features

Why model benchmarks almost never decide the choice

For production features, the deciding constraint is rarely model quality. In our experience it is one of five things: where the data is legally allowed to go, cost per call at real volume, latency inside a request the user is waiting on, structured output and tool-calling fidelity, and how much eval work a vendor's next model release forces on you.

Benchmark roundups compare models on tasks nobody ships. A three-point difference on a reasoning benchmark is invisible when your actual job is extracting six fields from a support email into a JSON schema, or writing 40 words of product copy that reads like a human wrote it. All three frontier-class options clear that bar. What does not clear the bar uniformly is what happens around the model: whether you can sign a data processing agreement that satisfies your consent language, whether 200,000 calls a month fits the unit economics, whether the model returns valid JSON on the first attempt 99% of the time or 92% of the time.

So the useful exercise is not ranking models. It is identifying which single constraint dominates the feature, then picking the option that satisfies it and treating everything else as a tiebreaker. Below are three features where a different constraint won each time.

Feature 1: consent-bound health data, where data residency dominated

When the input contains sample-linked personal health data covered by a specific consent scope, the question stops being which model is best and becomes which processing is lawful. The consent text a user signed defines the permitted purposes and recipients. If a US-hosted inference API is not inside that scope, no benchmark score makes it acceptable.

In regulated health-tech work, the consent record is a first-class object in the database, not a checkbox. It has a version, a timestamp, a scope, and a withdrawal path. Sending sample-linked data to a third-party model provider is a new processing purpose and a new sub-processor, which means either the consent language and the record of processing activities cover it explicitly, or you do not send it. Retroactively re-consenting a live user base to enable a feature is a product decision with a real cost, and usually a bad trade for a convenience feature.

That leaves two workable paths. The first is redaction before the call: strip direct and indirect identifiers in a deterministic pipeline, send only the de-identified payload, and re-attach identity locally after the response comes back. This works well for free-text tasks like summarising a clinician note, and poorly for anything where the identifiers are the signal. The second is self-hosting an open-weight model inside the same infrastructure boundary as the rest of the regulated data, which keeps the processing entirely internal. We ran the redaction path first because it ships in days rather than weeks, and treated self-hosting as the escalation for workloads where redaction destroyed too much of the input.

• • Check the consent scope and record of processing activities before evaluating a single model
• • If redaction preserves the signal, redact and use a hosted API with a DPA in place
• • If identifiers are the signal, self-host inside the existing data boundary
• • Log which model version processed which record, because auditors will ask
• • Never let a debugging log capture the un-redacted prompt

Feature 2: marketplace product copy, where cost per call and latency dominated

For generating vendor and product descriptions across a marketplace catalogue, the deciding factors were cost per 1,000 calls and throughput, not writing quality. A catalogue with thousands of SKUs, each regenerated when a vendor edits attributes, turns a per-token price difference into a monthly line item that a finance person will ask about.

The task itself is narrow: take structured attributes (product name, producer, category, seasonality, unit) and produce two or three sentences in the right tone and language. That is a task where the gap between a frontier model and a small, cheap model is small once you invest in the prompt and a handful of few-shot examples. So the sensible sequence is to write the evaluation set first, twenty or thirty real products with acceptable and unacceptable outputs, then walk down the price ladder until quality actually degrades. Most teams skip this and default to the most capable model, which is how you end up paying frontier prices for a sentence-completion job.

Latency mattered differently here than in an interactive feature. Bulk generation runs as a background job, so a two-second response is fine. But the same feature had an interactive path where a vendor edits a product and expects to see suggested copy before they finish the form, and that path has a budget measured in hundreds of milliseconds. We solved it by splitting the two: a batched, cheap, high-throughput path for the catalogue backfill, and a smaller fast model for the in-form suggestion. One feature, two model choices, one abstraction layer behind both.

Feature 3: support and ops triage, where structured output fidelity dominated

For internal triage that classifies an incoming message, extracts fields, and calls a tool to route it, the deciding factor was schema conformance and tool-calling reliability. A model that writes beautifully but returns malformed JSON on one call in twenty creates a retry queue, an alerting problem, and eventually a human doing the job manually again.

The evaluation here is mechanical rather than subjective. Run several hundred real historical tickets through each candidate, validate every response against the Zod or JSON schema, and count three failure classes separately: invalid structure, valid structure with hallucinated enum values, and valid structure with the wrong routing decision. The first two are engineering problems with clear fixes (strict schema modes, constrained decoding, enum validation with a retry). The third is a quality problem that prompt work can move. Vendors differ meaningfully in how well their strict structured-output and tool-calling modes hold up under messy real input, and this is one area where the frontier hosted models were clearly ahead of what we could get from a small self-hosted model without significant constrained-decoding work.

The second-order factor was tool-calling behaviour under ambiguity. When a ticket genuinely does not fit any category, a good model calls the escalate tool. A weaker one picks the closest category with high confidence, which is worse than doing nothing because it silently routes real problems into a queue nobody watches. We test for that explicitly with deliberately ambiguous inputs in the eval set.

What a local model actually costs once you include everything

A self-hosted model is not free because the weights are free. The real cost is GPU hosting billed by the hour whether or not you have traffic, plus engineering time for deployment, quantisation, batching, load testing, upgrades, and on-call. For most features under a few hundred thousand calls a month, that total exceeds the API bill it was supposed to replace.

The economics flip in two situations. The first is genuinely high, steady volume, where a reserved GPU running near capacity beats per-token pricing. The second is when data residency makes hosted APIs unavailable at any price, in which case the cost comparison is irrelevant and self-hosting is simply the price of shipping the feature at all. Outside those two cases, choosing a local model for the privacy pitch alone usually means paying more for a worse model and a new operational surface.

The failure modes we watch for are specific: idle GPU spend on a feature with bursty traffic, nobody owning the upgrade path when a better open-weight model ships six months later, and quality regressions on structured output that the team discovers in production because the eval suite was written for the hosted model and never re-run. If nobody on the team wants to own a GPU deployment for the next two years, do not start one.

• • Wrong answer when: traffic is bursty and low volume, so the GPU idles
• • Wrong answer when: the task needs strong tool calling and you have no constrained-decoding work planned
• • Wrong answer when: no one on the team wants to own inference infrastructure long term
• • Right answer when: data cannot legally leave your boundary, at any price
• • Right answer when: volume is high and steady enough to keep a reserved GPU busy

The provider abstraction that keeps the decision reversible

Every LLM feature we ship sits behind a thin internal interface: a typed request with a task name, an input payload, and an output schema, resolved to a provider by configuration rather than by import. The point is not vendor neutrality as a principle. It is that we have changed providers after launch, and we wanted the change to be a config edit plus an eval run rather than a refactor.

The layer is deliberately thin. It handles provider selection per task, retries and timeouts, schema validation with a bounded repair loop, token and cost accounting tagged by task, and prompt versioning so you can attribute a quality change to a prompt edit or a model change. It does not try to normalise every provider feature, because that is where these abstractions go wrong. Provider-specific capabilities like extended reasoning modes or prompt caching are exposed as opt-in escape hatches for the tasks that need them.

Two things broke when we actually switched. First, prompts tuned hard against one provider's formatting habits produced subtly different output on the other, mostly in tone and in how aggressively the model followed length instructions, which the schema validator happily passed. That is why the eval set matters more than the abstraction: without it, you cannot tell whether a switch was neutral. Second, structured output semantics differed in the edge cases, particularly around optional fields and how a model represents 'no value found', where one provider returned null and the other omitted the key or returned an empty string. Both were caught by the eval suite, and neither would have been caught by unit tests.

The redaction pipeline referenced in the health-data section is covered in detail in our privacy-preserving AI blueprint, which walks through PII redaction before LLM calls, tenant isolation, and retention enforcement.