A Two-Speed Architecture for Persistent Local AI Agents

1Motivation

The goal is a personal assistant that genuinely knows its user: one that remembers what the user cares about, learns their habits, and grows more useful over time, while running privately on the user's own machine with no user data sent to an external service.

Local deployment imposes a constraint that shapes the entire design. On consumer hardware, the most capable open models are slow — large models whose weights are paged from disk may produce output on the order of a single token per second. Such a model cannot be used interactively; no user will wait through a conversation at that rate. Conversely, the models small enough to respond with low latency are too limited to build and maintain a deep model of a person. Naively, one must choose between depth and speed.

The central observation is that depth does not have to be delivered in real time. A slow-deep model's understanding remains valuable if it is produced ahead of when it is needed and stored for fast retrieval. The architecture therefore separates the two timescales: expensive, deep cognition runs in the background and writes its conclusions down; cheap, fast cognition runs in the conversation and reads those conclusions. Depth and speed are obtained together by never requiring them from the same component at the same moment.

2Overview

The system follows a compiler/runtime split. A slow-deep model acts as a compiler that turns the raw record of interaction into a clean, structured knowledge base; a fast model acts as a runtime that reads that knowledge base to act. The knowledge base is both the interface between the two models and the actual product of the system.

Where a component has a close counterpart in evolutionary biology, that counterpart is given in brackets alongside it — gene pool, allele, selection, genome. This is a deliberate anchor for readers approaching the design from artificial life or biology: familiar handles onto each mechanism. The bracketed terms are not ornament — each names something specific — and §10 gives the full mapping, along with the single point where the analogy stops.

The principal components are:

• The slow-deep model — a capable, offloaded model that runs in batch, never in the interactive path, and writes structured notes.
• The fast model — a small, low-latency model that answers the user by retrieval rather than by reasoning about the user from scratch.
• The knowledge base — a structured, human-readable store; the compiled artifact and the single interface between the models. The fittest version of each fact, gathered here, is the genome the fast model reads.
• The short-term buffer — a small verbatim record of the most recent turns.
• The candidate pool (a gene pool) — a sub-layer of the knowledge base holding, for each contested fact (a gene), its competing versions (alleles).
• The resource governor — a scheduler that decides when the slow-deep model may run.

3Components

3.1The two models

The slow-deep model is a compiler. Given time, it reads back over accumulated interaction logs and writes what it extracts into the knowledge base as clean, organised pages. It is never placed in the interactive path; its latency is irrelevant because it operates behind the conversation rather than within it.

The fast model is the runtime and the only component the user speaks to. When it responds, it retrieves relevant material from the knowledge base rather than deriving an understanding of the user on the spot. This is what allows a small model to behave as though it understands the user deeply: the depth was supplied earlier, by the slow-deep model, and is being spent now.

3.2The knowledge base

The knowledge base is a structured store — in effect a private, single-user wiki — and it is human-readable by design. Its legibility is a functional requirement, not a convenience: because a person can read it, a person can also audit and correct it, which is the system's final line of defence against error (§6) and the basis of its auditability (§8). Pages carry metadata used elsewhere in the design: provenance, a timestamp, a confidence tier, and a stakes classification.

3.3The short-term buffer

The most recent turns are held verbatim in a short buffer that functions as working memory. When the fast model responds, it reads two sources: the buffer, for what was said recently, and the knowledge base, for everything older that has already been consolidated. Over time the slow-deep model drains the buffer, working its contents into knowledge-base pages, after which the verbatim turns can age out because their meaning now resides in the store.

4Operation

Two data paths run at different speeds.

The response path is fast and synchronous. On each turn, the runtime reads the buffer and retrieves from the knowledge base, then answers. This path never invokes the slow-deep model and so meets interactive latency.

The consolidation path is slow and asynchronous. The compiler processes the tail of the buffer, produces candidate pages, and files them into the knowledge base. This path is where the system's understanding actually deepens, and it runs entirely in the background.

The compiler's work divides into two kinds with different urgencies. On arrival of new material, it inserts that material into the store — a cheap, obligatory operation done promptly. When the machine is idle, it performs more expensive housekeeping: re-evaluating existing entries, promoting settled ones, and discarding entries that never prevail. The housekeeping is discretionary; if idle time never arrives, the obligatory operations still complete and only the store's tidiness suffers.

This design implies a staleness relation: the knowledge base lags reality by however far the compiler is backlogged. That lag is acceptable for durable facts about the user but unacceptable for the immediate past, which is precisely why the buffer exists — it covers the interval the knowledge base has not yet caught up to.

5The knowledge base as a selective store

5.1Pool and clean layer

For contested content — facts about the user or the world, where being correct matters — the system does not overwrite old notes with new ones. Beneath the clean knowledge base sits a pool (a gene pool) that may hold several competing versions (alleles) of the same element (a gene), each with a rank. The clean layer holds only the current top version per element — the fittest allele expressed at each gene — and that clean layer, the collected winners (a genome), is what the fast model reads. Variation lives in the pool; selection promotes winners into the clean layer.

5.2Ranking by pairwise comparison

Ranking uses comparison rather than absolute scoring. Asking a model to assign a numeric quality score yields poorly calibrated values that drift between evaluations; asking it which of two entries is better is a judgment models make far more reliably, and it emits only a single bit of output — valuable when output tokens are the scarce resource.

A new candidate is placed by incremental binary insertion into an already-sorted list, requiring on the order of log n comparisons rather than a full re-sort. The current clean-layer entry sits in that list as a permanent competitor, so the incumbent must defend its position on every insertion; promotion falls out of the same operation. An entry is promoted to the clean layer only once it has held the top rank across several passes — a stability threshold that trades freshness against reliability and can itself be exposed as a tunable parameter.

5.3Two passes

Because output generation is the expensive operation at low throughput, comparison and composition are separated. A comparison pass reads many candidates and emits only rankings — wide input, negligible output — and can run often, keeping the pool freshly ordered. A promotion pass composes a clean page from a settled winner — narrow input, careful output — and runs rarely, only on elements whose ordering has stabilised. Decoupling the two lets each run at its natural rate, and ensures the clean layer only ever exposes entries that have survived several rounds of scrutiny.

5.4Known problems in the ranking

Several failure modes are inherent to model-driven ranking and are treated as first-class design concerns rather than incidental bugs.

Comparison drift. The same model comparing the same candidates on different occasions may judge them differently, because surrounding context changes. Rank is therefore a moving estimate, not a fixed truth. Retaining timestamped rank history — rather than only the latest ordering — keeps this auditable and permits later smoothing.

Intransitivity. Model comparisons do not guarantee a total order; cycles (A over B, B over C, C over A) are possible, and a strict sort fed such judgments produces an order dependent on comparison sequence. Two mitigations apply: prefer sorting methods that degrade gracefully under noisy comparisons, and treat the ordering as approximate, since only the top position is load-bearing. Where intransitivity proves severe, a rating system of the Elo or tournament type absorbs contradictory judgments by averaging and returns a cardinal score derived only from the comparisons themselves.

Cold-start cost. At roughly a token per second, the dominant cost is paging model weights, not generating text. Many small comparison calls that each pay the paging cost can be slower in wall-clock terms than a single large read pass, even though each comparison is trivial in tokens. Comparisons for a given insertion must therefore be batched into one warm, resident session; if the model cannot be kept warm across a run, a single bulk-evaluation pass may outperform incremental pairwise comparison. This is an empirical question to be measured, not assumed.

Candidate identity. Two candidates should compete only if they concern the same element, so something must cluster candidates by what they are about. Mis-clustering is its own failure: distinct facts forced into one slot, or a single fact fragmented across several slots that never meet to compete. This is a retrieval-and-identity problem, and left unmanaged it is the most likely way for the store to rot over time. It is a core open problem (§9).

6Reliability and correctness

6.1The confident-wrong problem

A small model reading an incorrect page will repeat the error with full confidence. This is the primary user-facing risk of the architecture, and most of the design's correctness machinery exists to contain it.

6.2Stakes classification

Not all pages carry equal risk, and the machinery is applied selectively to control cost. Persona and style pages are low-stakes: if slightly wrong, the effect is a slightly-off voice, with no factual harm, and these carry no competition or heavy metadata. Pages recording facts about the user or the world are high-stakes and carry the full apparatus — provenance, timestamp, confidence tier, and pool competition. Restricting the expensive mechanisms to high-stakes pages is what keeps the scheme affordable at low throughput.

6.3Layered defences

High-stakes correctness rests on four layers, each catching what the previous one misses.

• Provenance and confidence tiers. Each fact records which turns it derived from, when, and whether it was stated directly or merely inferred. The runtime hedges on inferred or aging facts and asserts plainly only on direct, recent ones — converting confident-wrong into appropriately-tentative-wrong.
• Abstention. When retrieval surfaces nothing above a confidence threshold, the runtime is permitted to say it does not know rather than answer from a weak match. This is enforced at the retrieval layer by simply not surfacing low-confidence pages; a blank is preferable to a wrong page.
• Belief re-competition. Correction proceeds by out-competing, not overwriting. A wrong winner that is beaten is demoted to the pool rather than destroyed, so a belief that was wrong-then-right can prevail again and nothing correct is discarded to fix something incorrect.
• Human editability. The final layer is the user. Because the knowledge base is legible, any error surviving the automated defences remains correctable by hand.

The residual case that no automated layer closes is a confident wrong page the user never happens to correct. Provenance at least makes such a page auditable when the user does look; closing it entirely depends on the store being something a human can open.

7Scheduling

A resource governor decides when the slow-deep model may run, measured against the machine's compute, thermal, and battery budget. When the user is active or the device is constrained, the slow-deep model is throttled or paused; when the machine is idle, it runs at full throughput and performs its heaviest consolidation then, because that is when resources are free and — critically at low throughput — when the model can remain warm and resident across a run rather than repaying paging costs.

Idle housekeeping is the governor's low-priority lane and must be bounded, or it will consume every spare cycle re-evaluating entries that do not matter. It is ordered by stakes — high-stakes facts first, persona and style ignored — and stops at a queue threshold or when the user returns. The obligatory operations, responding and inserting new material, are never deferred; only discretionary tidiness is, which is what gives the system its graceful degradation under resource starvation.

8Properties

The architecture yields several properties as structural consequences rather than added features.

Privacy by locality. All computation and storage are local; no user data leaves the machine.

Auditability by construction. The knowledge base is legible, so the user can inspect, prune, and delete what the system believes about them. An opaque long-term profile would be a liability; a readable one is a control surface.

Depth without latency. Expensive cognition is removed from the interactive path entirely, so the system can be both deep and responsive.

Separation of concerns. The slow-deep compiler and the fast runtime are distinct components on distinct timescales, so neither is compromised to serve the other.

Graceful degradation. Under resource pressure the system sheds discretionary work first and preserves its obligations, so it fails soft rather than breaking.

9Limitations and open questions

Staleness. The knowledge base necessarily lags any content the compiler has not yet consolidated. The buffer covers the recent interval, but the lag is real for anything in between, and the acceptable bound depends on the application.

Consolidation triage. At the throughput ceiling the compiler is permanently behind, so its value is determined by what it chooses to consolidate next. The scheduler that makes this choice is therefore the system's hardest design problem, not an implementation detail; a poor scheduler consolidates trivia while missing what matters.

Candidate identity and clustering. Determining when two candidates concern the same element is unresolved and is the most likely long-term source of store degradation. It warrants dedicated treatment.

Ranking robustness. Whether incremental pairwise comparison, bulk scoring, or an Elo-style rating performs best under the warm/cold constraints of a token-per-second model is an empirical question to be measured on representative workloads.

Promotion tuning. The stability threshold governing promotion trades freshness against reliability and may need to vary per element or per user.

Evaluation. How to measure whether a knowledge base is good — accurate, well-organised, appropriately confident — is an open methodological question and a prerequisite for tuning the rest.

Schema. A concrete page-and-candidate schema — the exact fields for provenance, confidence, stakes, and rank history — is the natural next artifact, and would resolve several of the questions above by forcing them into a definite form.

10A note on terminology

The design borrows its vocabulary from genetics and evolution, because the mechanism genuinely matches: each concept is a gene; the rival versions of it are alleles; the store of all of them is a gene pool; ranking is selection; and the fittest allele expressed at each gene, collected together, is the genome the fast model reads. Consolidation is a kind of dreaming and idle maintenance a kind of sleep.

One caveat keeps the metaphor honest. Genes and evolution usually imply heredity — variation passed on and reproduced across generations — and that is the one part this design does not have; nothing is inherited between agents. What it takes is variation, selection, and expression within a single lifetime. The exact precedent is not Darwin across generations but the immune system across a lifetime: clonal selection, which maintains a population of antibody variants and amplifies whichever fit, with no inheritance involved.

These terms are used only where they name a real mechanism — each corresponds to a specific component above — and none is relied upon beyond the mechanism it labels. The metaphors are an aid to intuition, not a substitute for the engineering.

11Conclusion

The architecture trades the assumption of a single always-on model for a two-speed division of labour. A slow-deep model compiles experience into a legible knowledge base offline; a fast model reads that base to respond in real time. This buys deep, private, individual understanding and interactive latency at the same time, and yields auditability and graceful degradation as consequences of the structure rather than as bolted-on features. Its principal open problems — candidate identity, ranking robustness under a severe throughput ceiling, and consolidation triage — are tractable engineering problems with identifiable directions rather than fundamental obstacles.