Shipping LLM Features Without a Platform Team

Adding an LLM feature to an existing product is rarely blocked by the first API call. The difficult part starts afterwards: deciding where model calls belong, keeping user data out of the wrong logs, handling slow or malformed responses, and giving the team a way to turn a feature off when the model behaves unexpectedly.

At SocioDigitals, we work with distributed engineers across four countries and often add AI capabilities to products that were not originally designed around machine learning. We do not always have the luxury of creating a separate platform or MLOps team first. The practical answer has been to build a small set of boundaries into the application itself, then add infrastructure only when the traffic and failure modes justify it.

This article describes those boundaries. It is not a recipe for a universal AI platform. It is a way to ship a useful LLM feature without creating an operational mystery that only one person understands.

Start with a narrow responsibility

The first design decision is not which model to use. It is what the model is allowed to do.

A reliable first version normally has one narrow responsibility:

• • classify or route an incoming request;
• • extract structured fields from a document;
• • draft a response for a human to approve; or
• • answer questions over a known set of documents.

That responsibility should be represented by an application service with a normal interface. Controllers, queue jobs, and scheduled commands should not contain provider-specific prompt construction scattered across the codebase.

For example, an application can expose a service like this:

final class SupportAnswer
{
    public function __construct(
        public readonly string $text,
        public readonly string $status,
        public readonly array $citations = [],
    ) {}
}

interface SupportAssistant
{
    public function answer(string $question, array $context): SupportAnswer;
}

The rest of the application can then handle status = "needs_review" without knowing whether the implementation used one provider, a fallback provider, or a retrieval pipeline.

This boundary makes model changes boring. Boring is good. A model upgrade should not require rewriting the billing flow, the support dashboard, and the queue worker at the same time.

Keep the synchronous request small

A model call inside a customer-facing HTTP request creates a fragile dependency chain. The user waits for network latency, provider latency, retrieval latency, and any retry attempts. A timeout becomes a product error even when the rest of the application is healthy.

For anything that does not need an immediate answer, we put the work behind a queue:

1. 1. accept the user action;
2. 2. persist the input and a processing state;
3. 3. dispatch an idempotent job;
4. 4. call the model outside the request lifecycle;
5. 5. save the result, evidence, and status;
6. 6. notify the user or make the result available for review.

The job needs an idempotency key. If a worker retries after a timeout, it must not create two invoices, send two messages, or attach two generated summaries to the same record.

A useful status model is more informative than a boolean:

queued -> running -> completed
                 -> needs_review
                 -> failed_retryable
                 -> failed_permanent

The needs_review state is particularly important for LLM features. A response can be syntactically valid and still be unsafe to use automatically.

Treat the model as an untrusted dependency

The model is not a database and its output is not a trusted command. We validate it just as we would validate a response from an external payment or shipping API.

For structured output, validate against a schema before writing anything important. Reject unknown fields where possible, enforce lengths, and make required decisions explicit.

$payload = $llm->complete($messages);
$data = json_decode($payload, true, flags: JSON_THROW_ON_ERROR);

$validated = $supportAnswerSchema->validate($data);

if (!$validated->isValid()) {
    return new SupportAnswer('', 'needs_review');
}

Schema validation does not prove that the answer is correct. It only prevents a malformed answer from silently becoming application state. Domain rules still need to run afterwards: permissions, numerical limits, account status, duplication checks, and human approval gates.

The same principle applies to tool use. A model may propose an action, but the application decides whether that action is permitted. Tool calls should be allow-listed, parameter-validated, authenticated as the application user, and logged with a correlation ID.

Make retrieval boring and inspectable

A retrieval-augmented assistant usually has four separate operations:

• • identify the tenant and user permissions;
• • select candidate documents;
• • build a bounded context window;
• • generate an answer with citations or document references.

Do not let retrieval become an opaque helper called askTheAI(). Keep the selected document IDs, scores, filters, and retrieval timestamp available for inspection. When a user reports a bad answer, the first question should be: what context did the model actually receive?

The context builder should also enforce limits. A very large context is not automatically better. It increases cost, latency, and the chance that relevant instructions are buried among unrelated text.

For multi-tenant systems, permission filtering must happen before content reaches the model. Never retrieve broadly and ask the model to ignore records belonging to another customer. Authorization belongs in the query layer, not in a prompt.

Separate prompts from business rules

Prompts are part of the application, but they should not become the application’s only policy layer. Store prompt templates with a version identifier and keep business rules in code.

A prompt can say “be concise,” but code should enforce a maximum output length. A prompt can say “do not reveal private data,” but the retrieval query and redaction layer must enforce access control.

We typically record:

• • prompt version;
• • model and provider name;
• • temperature or reasoning setting, if applicable;
• • input and output token counts when available;
• • retrieval IDs;
• • latency and retry count;
• • final status and human override.

That metadata makes a later comparison possible. Without it, “the model got worse” is difficult to distinguish from “the prompt changed,” “the retrieved documents changed,” or “the provider was slow and the request timed out.”

Build a fallback that changes the status, not just the model

A second model is useful, but blindly retrying with another model can create duplicate side effects or hide a systemic failure. Fallbacks should be used for tasks where the operation is safe to repeat, and they should preserve the same validation and approval path.

For example:

• • a failed classification can be retried with a fallback model;
• • a draft can be regenerated and marked for review;
• • a payment or account mutation should not be replayed merely because the model timed out.

The fallback result should record why it was used. A provider outage and a low-confidence answer are different operational events. They should not be collapsed into a generic “AI completed” log entry.

Instrument the edges first

A small team does not need a giant observability platform on day one. It does need enough information to answer five questions:

1. 1. How many requests are reaching the model?
2. 2. How often do they time out or fail validation?
3. 3. How much does each workflow cost?
4. 4. Which prompt/model/version produced the result?
5. 5. How often does a human override it?

Start with structured application logs and a small dashboard. Redact prompts and responses that contain sensitive data, or store only hashes and operational metadata where the content is not required for debugging. Do not log API keys, full customer documents, or raw authorization headers.

Useful initial metrics include:

• • request count by workflow and provider;
• • p50 and p95 latency;
• • retry and fallback rate;
• • schema rejection rate;
• • human-review rate;
• • average input/output tokens;
• • cost per successful workflow;
• • user feedback or correction rate.

These metrics are more valuable than a single aggregate “AI accuracy” number, because production workflows usually contain several different tasks with different definitions of success.

Give the team a kill switch

Every LLM feature should have a way to reduce its blast radius without a deployment. A feature flag can disable automatic actions while leaving draft generation available. A provider flag can route new requests to a fallback. A tenant-level flag can pause a feature for one customer while an incident is investigated.

The kill switch should be checked in the application service and in queue workers. Disabling a button in the UI is not enough; old jobs may still be waiting in the queue.

We also prefer an explicit human-review mode for new workflows. It lets the team inspect real failures before allowing the system to perform actions automatically.

What a small team should not build first

A distributed development team can waste months building infrastructure that the product does not yet need. We generally avoid starting with:

• • a custom model gateway for one low-volume workflow;
• • a vector database before measuring whether normal database search is sufficient;
• • an elaborate evaluation platform before defining a handful of representative cases;
• • autonomous tool use for irreversible operations;
• • a generic prompt management product that nobody owns.

The better sequence is to build one workflow end to end, capture its failure modes, and extract reusable components only after a second workflow demonstrates the need.

A pragmatic shipping checklist

Before releasing an LLM feature, we ask:

• • Is the model’s responsibility narrow and explicit?
• • Is the workflow asynchronous where latency does not need to be user-facing?
• • Are inputs and outputs validated independently of the prompt?
• • Are tenant permissions enforced before retrieval?
• • Can we replay a safe job without duplicating side effects?
• • Are prompt, model, retrieval, latency, and cost metadata recorded?
• • Is there a human-review state?
• • Is there a kill switch that reaches queued work?
• • Can we explain what context produced a disputed answer?
• • Can we roll back the feature without rolling back unrelated product code?

If the answer to several of these is no, adding more model capability will not fix the operational gap. A smaller model behind clear boundaries is usually more useful than a stronger model embedded in an untestable request path.

The goal is not to pretend a small team has a platform organization. The goal is to make the few boundaries that matter explicit, observable, and replaceable. That is enough to ship useful LLM features today while leaving room to add more infrastructure when the evidence says it is needed.