Architecture at the model boundary

Documented failure modes in AI-native SaaS

Research Essay. Independent research. Not peer reviewed.

---

Abstract

Placing a language model inside a request path that users depend on breaks four assumptions that conventional SaaS architecture is built on: that the same input produces the same output, that behaviour is fixed by the deployed artifact, that correctness can be asserted, and that latency and cost per request are bounded and known.

This note reviews the published software engineering literature on machine learning systems, retrieval architectures and stability patterns, and argues that most of the resulting failure modes were documented before large language models became widely deployed. The synthesis produced here is a four-axis versioning model, a three-outcome contract for the model boundary, and a set of seven findings about where the boundary should sit.

No original empirical research was conducted. No systems were benchmarked and no measurements were taken.

Research question

When a probabilistic component is placed inside a request path that users depend on, what changes structurally in the architecture, and which of the resulting failure modes are already documented in the software engineering literature rather than novel?

The second half of the question matters more than the first. There is a widespread assumption in current practice that AI-native architecture is unmapped territory. The premise examined here is that a substantial portion of it was mapped between 2015 and 2020, in papers about machine learning systems, and that the practical failure of current products is a failure to read rather than a failure of available knowledge.

Background

Conventional SaaS architecture rests on a set of properties that are rarely stated because they are rarely violated.

Determinism. The same request against the same state produces the same response. This property is what makes a test suite possible, what makes a bug reproducible, and what makes a cache correct.

Behaviour fixed by the artifact. What the system does is determined by the code that was deployed. A behavioural change requires a deployment, which means it appears in version control and can be reverted.

Assertable correctness. Correctness is expressible as equality or as a predicate. A test either passes or fails, and a passing suite has a defined meaning.

Bounded cost and latency. A request costs a known amount of compute and returns within a distribution the team has measured. Reads are cheap and idempotent, so they can be cached and retried freely.

A model in the request path violates all four. Sampling makes output non-deterministic. Behaviour depends on a provider's weights that can change without any deployment on the application side. Correctness becomes a graded judgement about acceptability rather than a predicate. And a single request has variable, per-token cost with a latency distribution whose tail is long and provider-controlled.

The question is not whether these violations are inconvenient. It is what architecture follows from them.

Methodology

Literature and secondary-source analysis, meaning synthesis of published research, specifications and documentation. The corpus was selected on three criteria: papers about the engineering rather than the modelling of machine learning systems, papers about retrieval and tool-use architectures, and prior work on stability patterns for unreliable remote dependencies.

Two limits on the method are declared here rather than in the limitations section, because they affect how the evidence should be read.

First, the central papers on machine learning systems debt predate large language models. Their subject is supervised learning pipelines with training in the loop. Their transfer to systems that consume a third-party model through an API is argued in this note, not demonstrated by the papers themselves.

Second, where this note refers to build experience, that is declared as practitioner observation from a single-operator sample. It is context, not evidence.

Evidence

Sculley et al., 2015, "Hidden Technical Debt in Machine Learning Systems" (NeurIPS). The foundational text, and still the most useful. Several of its named failure modes transfer directly.

The CACE principle: changing anything changes everything. Because a model's behaviour is a function of all its inputs jointly, no input can be treated as independent. Applied to a system that assembles a prompt from several sources, this predicts that changing one section of the prompt alters behaviour attributable to another.

Correction cascades. A model is corrected by a downstream filter, then a second correction is applied to the filtered output, and the resulting stack has no owner and no clean version. This is the observed shape of most prompt-engineering codebases after six months.

Undeclared consumers. Model output is written somewhere, another feature reads it, and there is now a dependency nobody recorded. This is the mechanism by which probabilistic output leaks into parts of a product that assume authored data.

Configuration debt. Sculley and colleagues note that configuration is under-tested and under-reviewed relative to its effect on behaviour. Prompts are configuration in exactly this sense, and they are frequently held in string literals with no review process at all.

Amershi et al., 2019, "Software Engineering for Machine Learning: A Case Study" (ICSE-SEIP). A study of Microsoft teams and a nine-stage workflow model. Three differences from conventional software engineering are identified, and the third is the one that matters here: AI components are harder to treat as distinct modules, because model behaviour is entangled with data and with other components in ways that resist encapsulation.

That finding is a direct challenge to the standard architectural instinct, which is to hide the model behind an interface and proceed. Encapsulation is still correct, and the paper's finding is that it is harder to achieve than the instinct assumes.

Ribeiro et al., 2020, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" (ACL). Held-out accuracy overstates capability. The authors propose a matrix of linguistic capabilities against test types, including minimum functionality tests, invariance tests and directional expectation tests, and report finding substantial failures in commercial systems that scored well on their benchmarks.

The architectural consequence is that a model's test suite is a different artifact from an application's test suite, with a different pass criterion. It reports a rate, not a boolean.

Lewis et al., 2020, "Retrieval-Augmented Generation" (NeurIPS). Retrieval as an architectural component, combining parametric memory in the weights with non-parametric memory in an index. The operational consequence is under-discussed: once knowledge lives in a retrievable corpus, that corpus has its own deployment cycle, its own freshness properties and its own failure modes, none of which are synchronised with either the application or the model.

Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (TACL, 2024). Performance depends on where relevant information sits in the context window, with a U-shaped curve: material at the beginning and end is used more reliably than material in the middle.

This converts context assembly from plumbing into a functional parameter. The order in which retrieved documents are concatenated affects output quality, which means it is a design decision that requires a test.

Yao et al., 2023, ReAct, and Schick et al., 2023, Toolformer. Both establish the pattern in which model output selects and parameterises a call to external code. This is the precise point at which text becomes an effect on the world, and it is therefore the point at which the application, not the model, must enforce every constraint that matters.

Shankar et al., 2024, "Who Validates the Validators?" Work on aligning LLM-based evaluators with human judgement, reporting that evaluation criteria are themselves unstable and require grounding against human labels. The architectural reading: an automated evaluation harness is a component with its own version, its own drift and its own error rate, and treating its output as ground truth reproduces the problem it was built to solve one level up.

Nygard, Release It! Stability patterns for remote dependencies: timeouts on every call, circuit breakers to stop cascading failure, bulkheads to isolate resource pools. None of this is about machine learning. It is about calling something over a network that can be slow or unavailable, which describes a model provider exactly, and it is frequently absent from AI-native codebases that treat the provider call as if it were a local function.

Analysis

The evidence supports grouping the structural change into five areas.

The output contract

Model output arrives as text. Everything downstream expects structure. The boundary between them is where most production failures occur, and the common implementation is a parse inside a try block, which handles the easy case and hides the interesting ones.

The literature suggests a three-outcome contract instead of a two-outcome one. Output is either valid against a declared schema, repairable by a bounded retry or coercion, or a refusal, in which case the system must have a deterministic path that does not involve the model. The third outcome is the one usually missing, and its absence is what turns a model failure into a product failure.

The ReAct and Toolformer line makes the stronger version of this point. Constraint enforcement cannot live in the instructions to the model, because instructions are requests. Anything that must be true has to be checked on the application side after the output arrives.

The versioning problem

A conventional deployable has one version axis. An AI-native one has four, and only the first is in version control by default.

1. Application code.
2. The instruction set, meaning prompts, tool descriptions and system messages.
3. The model identifier, including provider-side changes to a pinned name.
4. The retrieval corpus, including its chunking and embedding parameters.

Any of the four can change observable behaviour with no change to the other three. Sculley's configuration debt applies to the second axis literally: prompts are configuration, they are under-reviewed, and they are the most common location of behavioural regressions that nobody can find in a diff.

The practical implication is that a behavioural incident report needs all four versions recorded, or it is not reproducible.

The evaluation problem

Because correctness is graded, the test suite reports a rate. This has three consequences the literature makes explicit.

CheckList establishes that aggregate accuracy hides capability-specific failure, so the suite needs to be organised by capability with per-capability thresholds. Shankar and colleagues establish that an automated grader is itself a component requiring human grounding. And a graded suite requires a policy decision that a boolean suite does not: what pass rate ships, and who decides.

A suite with no declared threshold is not a gate. It is a dashboard.

Cost and latency

Per-request cost is variable and proportional to tokens, which makes cost a function of user behaviour rather than of infrastructure sizing. Latency has a long, provider-controlled tail. Neither is cacheable by the usual mechanisms, because the cache key is a natural-language input with effectively unbounded cardinality.

Nygard's patterns are the available answer and they are conventional: timeouts, circuit breakers, bulkheads, and a degraded path that returns something useful without the model. The additional AI-specific levers are semantic caching at the level of intent rather than string, and tier routing, meaning a deliberate per-operation choice of model capability rather than a single default for the whole product.

Entanglement and containment

CACE and the Amershi finding together argue that a model cannot be cleanly encapsulated, only contained. Containment has two concrete requirements.

Model-derived data must be distinguishable from authored data at the storage layer, permanently, by a provenance field rather than by convention. Without this, Sculley's undeclared-consumer problem is guaranteed: a later feature will read the column, assume it was authored, and inherit an error whose source is three systems away.

And the model's interface to the rest of the application should be narrow enough that the application does not depend on the shape of the model's reasoning. Products that expose reasoning traces to downstream logic have coupled themselves to an implementation detail of a third party.

Findings

1. Most documented failure modes in AI-native systems are not new. The CACE principle, correction cascades, undeclared consumers and configuration debt were named in 2015 for a different class of system and transfer with minor translation.

2. The model boundary requires a three-outcome contract. Systems with no deterministic refusal path convert a model failure into a product failure by construction.

3. Constraints that must hold cannot be expressed as instructions to the model. They are application-side checks on output. Instructions are requests, not guarantees.

4. An AI-native deployable has four independent version axes, and behavioural reproducibility requires recording all four.

5. Prompts are configuration, and the under-review of configuration relative to its behavioural effect is a documented debt category rather than a new problem.

6. Evaluation reports a rate, which requires a declared threshold and an owner. An automated grader is a versioned component with its own error rate, not a source of truth.

7. Model providers are remote dependencies with unbounded latency, and the stability patterns for that situation predate the technology by two decades.

Limitations

This note is a synthesis of published work with no measurement behind it, and several of its weaknesses are structural rather than incidental.

The transfer argument is not demonstrated. The machine learning systems papers describe organisations that train and serve their own models. A team consuming a third-party API has a different topology: no training pipeline, no feature store, no data collection loop. This note argues that the debt categories transfer, and that argument is reasoning by analogy. A reader who rejects the analogy is left with much less.

No system was measured. There are no benchmarks, no incident data, no comparison of architectures under load. Every claim about what fails in practice is either cited from a paper or reasoned from premises.

The corpus is small and selected by the author. No systematic search protocol was followed, no inclusion criteria were pre-registered, and papers were chosen because they were known to be relevant. This is a review in the informal sense, not a systematic review, and selection bias is unaddressed.

The literature is young and moves faster than the review. Findings about context position sensitivity are properties of particular model families at particular sizes and may not hold for later ones. Architectural advice derived from a transient model property has an expiry date this note cannot state.

None of the recommended patterns are demonstrated in the author's own work. The most uncomfortable limitation. TapQR publishes an AI fraud detection pillar that is not built. No project in the author's record contains an evaluation harness, a provenance field on model-derived data, or a circuit breaker on a provider call. The practitioner observation available here is that of someone who has read the literature and has not yet implemented it, which is worth less than either reading or implementing alone would suggest.

The four-axis versioning model and the three-outcome contract are not findings. They are this note's framings. They have not been tested against a real incident corpus, and their usefulness is currently a claim.

Implications

For teams building AI-native products. Read the 2015 debt paper before designing the architecture. Record all four version axes in every incident report. Give the model boundary a schema, a bounded repair path and a deterministic refusal path. Add a provenance field to every column that can hold model output, on the first migration rather than the fortieth.

For evaluation practice. Organise tests by capability rather than by aggregate score, declare the shipping threshold in writing, and treat an automated grader as a component with a version and an error rate.

For operations. Apply the ordinary stability patterns for unreliable remote dependencies. Set a latency budget per user-facing task rather than per provider call, and choose model tier per operation.

For further work. The gap this note cannot fill is empirical. What is missing from the public literature is incident data from AI-native products: what actually broke, at which boundary, and whether the four version axes were recoverable after the fact. That is a study someone with production systems could run and this note cannot.

Conclusion

The architectural problem is not that models are unpredictable. It is that four properties conventional architecture relies on are simultaneously withdrawn, and that the resulting failure modes were documented for an earlier generation of machine learning systems and are being rediscovered at cost.

The available response is unglamorous and mostly conventional: a contract at the boundary with a refusal path, four version axes recorded, a graded evaluation suite with a declared threshold, provenance on stored output, and the same stability patterns any unreliable remote dependency has warranted since 2007.

The literature is ahead of the practice. That is a solvable problem, and it is solvable by reading rather than by invention.

---

References

1. D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo and D. Dennison, "Hidden Technical Debt in Machine Learning Systems", Advances in Neural Information Processing Systems 28, 2015.
2. S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi and T. Zimmermann, "Software Engineering for Machine Learning: A Case Study", ICSE-SEIP, 2019.
3. M. T. Ribeiro, T. Wu, C. Guestrin and S. Singh, "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList", ACL 2020.
4. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel and D. Kiela, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020.
5. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni and P. Liang, "Lost in the Middle: How Language Models Use Long Contexts", Transactions of the Association for Computational Linguistics 12, 2024.
6. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan and Y. Cao, "ReAct: Synergizing Reasoning and Acting in Language Models", ICLR 2023.
7. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda and T. Scialom, "Toolformer: Language Models Can Teach Themselves to Use Tools", NeurIPS 2023.
8. S. Shankar, J. D. Zamfirescu-Pereira, B. Hartmann, A. Parameswaran and I. Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences", UIST 2024.
9. M. T. Nygard, Release It! Design and Deploy Production-Ready Software, Pragmatic Bookshelf, second edition 2018.
10. tapqr.live, published product pillars including "AI Fraud Detection", with the AI layer documented as future work. Recorded as A25 and B9 in SOURCE_INVENTORY.md.

Related reading in this archive

• Features versus native: what actually compounds
• What AI-native product management actually means
• Agentic workflow architectures: the same boundary problem across multiple steps
• Latency, trust and the two variables everyone conflates
• TapQR