HayloftResearch

Evaluating hybrid retrieval and durable memory in Hayloft

We measure source coverage, correction durability, and time to delivery in an agent with persistent memory, including a controlled comparison with Hindsight.

A personal assistant can retrieve an obsolete fact, send a fluent correction, and leave its persistent state unchanged. Answer accuracy alone misses the defect. We evaluated Hayloft at three boundaries where the outcomes can diverge. We checked which sources retrieval exposes, what the agent commits to storage, and what the user receives.

In a 175-trial release evaluation, joint success rose from 155/175 to 174/175 and source reads fell from 600 to 498 against previous production. Durable-state checks accounted for the clearest gain, from 12/30 to 30/30. In an earlier archive experiment, Hindsight retrieved all required evidence in its first five results for all 22 questions. Two Hayloft versions reached the same final answer accuracy through follow-up reads. The experiments probe different points in the execution path.

Revision semantics

Consider a Spanish-language correction that moves lunch from Wednesday to Friday. The observed timestamp tells us when the correction arrived. It supplies neither a calendar date for lunch nor a license to resolve “Friday” to the next Friday. Our test checks the successor fact, the link from its predecessor, and the temporal precision of the subsequent reply.

Hayloft applies a correction to a person fact in a SQLite write transaction. It resolves the active fact by ID, inserts a successor with owner-stated authority, and stores the successor’s ID in the predecessor’s superseded_by field. Active reads select rows with a null successor. The historical row remains addressable, and a second attempt to correct the retired ID is rejected. This gives the evaluator a concrete failure condition even when the reply uses the correct weekday.

Our durable-state checks cover fact replacement, preference persistence, and requested reminders. A saved preference for a particular update fails the rubric if the agent also creates a reminder the user never requested. The evaluator inspects the resulting records in addition to the delivery trace.

Hybrid retrieval and source provenance

Hayloft maintains a rebuildable index over source records. The lexical path ranks SQLite FTS5 matches with BM25. The dense path searches 1,024-dimensional embeddings produced by text-embedding-3-large. Each path can contribute up to 150 candidates. For passage d, reciprocal rank fusion sums 1 / (60 + rank) across lists containing d, using zero-based ranks. The system groups hits by source kind and title before model-based evidence selection. Each hit retains a passage ID and a reference to its underlying source.

The release changes what evidence survives that selection. Hayloft fetches selected archive emails in a batch and returns the recorded mailbox, addresses, body, and source reference. It distinguishes a complete body from a local excerpt or provider-truncated content. The commitment reader returns bounded excerpts from the messages establishing or revising a promise. This lets the agent resolve “Did I send it?” from a Sent record, while a draft with identical text still leaves the question open.

We instrumented a conservative measure of redundant retrieval. A raw open qualifies when an earlier read in the same turn already exposed the same source identifier and the entire returned text after whitespace normalization. These opens fell from 43 to 20 against the earlier release prototype, while total raw opens fell from 179 to 75. Metadata can still justify reopening a record. The counter identifies traces worth inspecting.

Of 66 recall trials, previous production used one, two, three, four, or five or more reads in 4, 5, 28, 13 and 16 trials respectively. The release candidate did so in 27, 15, 10, 6 and 8 trials.
Both versions answered all 66 ordinary recall trials correctly. The number they could answer in a single source read rose from 4 to 27.

An independent eight-trial check tested the stopping rule on short and paginated records. A receipt needed one read in each repetition. Recovering all three held inventory items from a long recording needed a search and two transcript pages in each repetition. All eight trials passed. The two amended-pickup trials still contained seven reads after the decisive instruction had arrived.

Cross-system archive retrieval

We ran four configurations against one synthetic account with 119 source records. The same answering model received the same 22 questions three times per configuration. Hayloft ran its then-production build and an earlier development revision (PR 461). Hindsight v1.2.3 ran with its CPU reranker enabled and disabled. Its retain pipeline produced 275 facts, with optional observation consolidation disabled. Hindsight exposes separate operations for ingestion, retrieval, and multi-step reflection. Tested release · Hindsight architecture paper.

First-search coverage measures whether the top five documents contain every required source for a question. A multi-source question earns zero if one required document is missing. We score coverage once per distinct question (22), and correct delivered answers once per execution (66). Reply latency ends when the local WhatsApp stand-in accepts the first reply and includes model and tool time.

CPU-reranked Hindsight covered 22 of 22 questions on the first search. Hayloft’s earlier revision had the lowest 95th-percentile delivery latency. Exact values follow in the table.
First-search coverage and first-delivery latency in the four-way archive experiment. Each line in panel B connects a configuration’s median and 95th percentile.
Archive experiment · 22 questions, 66 executions per configuration
SystemCorrect repliesCoverage @5p50p95
Hayloft · previous production66/6621/2210.5 s16.7 s
Hayloft · earlier revision66/6619/229.3 s15.2 s
Hindsight · CPU reranking66/6622/2231.0 s149.1 s
Hindsight · reranking off65/6614/226.8 s173.8 s

CPU-reranked Hindsight covered 22/22 on the first search. The earlier Hayloft revision covered 19/22, missing evidence on changed employers or residences, yet delivered 66/66 correct answers. Turning off Hindsight’s reranker lowered median delivery time to 6.8 seconds and coverage to 14/22. Some difficult queries then invoked reflection, and p95 reached 173.8 seconds. CPU placement, inference configuration, and the small synthetic corpus limit how far those timings generalize.

We also held Hindsight’s storage configuration fixed and changed the answering agent’s weekday instructions. Correct date precision rose from 5/12 to 10/12 in that crossover. It is possible to change a memory benchmark score substantially at the agent layer.

The four-way measurements come from the saved aggregate scorecard. The Hindsight bank and individual traces were subsequently removed. We did not rerun that system on the larger release battery. On ordinary archive recall in the later Hayloft validation, the release candidate delivered 66/66 correct answers at 8.33 seconds p50 and 14.09 seconds p95.

Release protocol and results

The release battery comprised 62 distinct synthetic scenarios and 175 executions. Each execution started from a fresh copy of its seed. We fixed the answering model (GPT-6 Sol), medium reasoning effort, source fixtures, and suite-specific rubrics. The development arms used the same seeded order and eight concurrent actor slots in successive time windows. The original-production column comes from an earlier matched replay.

Joint success is the conjunction of all applicable checks. We grade the delivered answer or deliberate silence, verify evidence support where requested, and inspect the durable effect of a write. Substantive reads count top-level attempts to retrieve records over every turn, including empty and failed lookups. Availability checks, writes, and delivery calls have separate counters. First-reply latency runs from the final user turn to the first accepted substantive delivery and excludes silent turns. We report the median and nearest-rank p95.

Release evaluation · 175 executions per version
MeasurePrevious productionEarlier prototypeRelease candidate
Tasks passed155 / 175173 / 175174 / 175
Source reads600718498
Stored-state checks passed12 / 3030 / 3030 / 30
Median first reply10.38 s9.90 s7.89 s
95th-percentile first reply26.14 s19.38 s17.48 s
Median trial completion14.15 s12.50 s11.31 s
95th-percentile trial completion44.09 s20.64 s20.02 s

The earlier prototype shows the tradeoff we saw during development. It already passed all 30 stored-state checks, but used 718 reads. Returning more of the relevant evidence in each result brought that down to 498, a 30.6% reduction against the prototype. On the 16 trials that specifically required source verification, all three versions passed.

Read reduction varied by scenario. Against the prototype, 39 of 62 scenarios used fewer reads, 13 tied, and 10 used more, summing repetitions within each scenario. Across six cross-person obligation trials, reads fell from 68 to 38 while the slowest reply increased from 22.15 to 45.33 seconds. One question about what the user was waiting for triggered a broader investigation even after the assistant had read the list of commitments. With six observations, nearest-rank p95 is the sample maximum. Repeated trials share scenarios, and sequential runs share neither provider load nor cache state. These measurements describe the observed workload. They provide no population-level latency guarantee.

We kept the original grade for the candidate’s one failed trial. Its calendar answer mentioned the date of the last refresh. The connector had supplied that date to the assistant, but the summary available to the grader omitted it. That discrepancy is why we report 174 out of 175.

Background execution and write amplification

A routine calendar sync and “Please check whether my appointment moved” can both turn up an unchanged appointment. Only the second one has a person waiting for an answer.

In three separate routine-check trials, the earlier prototype stayed silent but still performed 11 reads and made two successful writes to personal memory. The release candidate stayed silent with zero tool calls. When explicitly asked to check an appointment, it looked up the record and delivered the unchanged result in all three trials. The visible reply alone would have missed the biggest difference between those versions.

Deployment validation

These changes are live in hosted Hayloft. After integrating the rest of the release, we reran 87 memory trials on the exact source tree that shipped. All 66 ordinary recall questions, 12 corrections, six actionable background events and three quiet checks passed.

The release also changes how memory fits into daily use. A new user gets a temporary onboarding conversation that can lead to a first briefing from the sources they choose. Morning reviews can use open goals and stated preferences, and a finished goal leaves the active list. Calendar writes show overlapping events, browser sign-ins can survive an assistant restart, and a payment can proceed after the user approves it. These features shipped alongside the memory work and have their own checks.

We also tightened two boundaries around source access. A disconnected request now cancels its memory read and bounds the wait for the provider, and erasing WhatsApp reader history is scoped to the source the user selected, even when a live connection and an imported archive cover the same chat. The execution record captures tool outcomes and when the assistant received them, so we can reconstruct which evidence was available when it answered.

The question we’re still working on is when a second source is worth the wait. A corrected pickup instruction can settle where to go. A conversation about sending a package can leave open whether anyone actually sent it. A good assistant needs to recognize both situations.