A common agent problem isn’t “can the model think?”—it’s “can the agent remember?” Most systems rely on the model to re-read past context by replaying long chat histories or repeatedly summarizing prior work. That approach is expensive (tokens) and fragile (summaries drift). An offline-first memory design flips the equation: store knowledge locally, retrieve it locally, and only send relevant snippets back to the model.
This article outlines a practical offline-first architecture for agent memory using SQLite FTS5 for fast keyword recall, local embeddings for semantic recall, and hybrid ranking for robust results—so recall can happen with effectively zero token fees (no “check everything in the conversation history” behavior).
1) Why memory should be retrieved locally
Token-based recall usually fails in two ways:
- Cost: To find “that one detail,” systems burn tokens on long histories or repeated context stuffing.
- Latency and drift: Summaries and condensed memories can become outdated or incomplete, and the model can’t reliably “reconstruct” missing details without more prompts.
With offline-first memory, the agent never needs to ask the model to “remember everything.” Instead, it runs retrieval against a local datastore, then supplies only the small set of relevant results.
Key outcome: recall is driven by database queries and local vector search—no paid tokens required for the act of searching.
2) Fast keyword recall with SQLite FTS5
SQLite FTS5 is ideal for “exact-ish” memory: names, error messages, SKUs, commands, code fragments, and structured terms (“RFC”, “MCP”, “SQL”, “FTS5”, etc.).
Typical FTS5 flow
- Store documents (or passages) in a local SQLite table.
- Maintain an FTS5 virtual table that indexes the text.
- On each “memory recall” request:
- Extract keywords (or simply use the user/query text as-is).
- Run a fast
MATCHquery against FTS5. - Return top candidate passages.
Why FTS5 matters
- Speed: FTS is built for text search; keyword queries are typically very cheap.
- Determinism: Exact terms match reliably, which is valuable for technical and operational memory.
- Good first-pass recall: Even when semantic retrieval is available, keyword recall often catches high-precision matches.
However, FTS5 alone can miss paraphrases:
- “I stored the logs” vs “I archived the output”
- “bugfix” vs “transaction rollback”
That’s where embeddings come in.

3) Semantic recall with local embeddings
Keyword search fails when the user’s query doesn’t share surface words with the stored memory. Local embeddings solve that by mapping text into a vector space where similar meanings are neighbors.
Offline embedding design
- Use an embeddings model running locally (e.g., via a local model runner).
- For every memory chunk you store:
- Compute its embedding vector.
- Store the vector (and a pointer to the original text) in a local index.
- For recall:
- Embed the query text locally.
- Retrieve nearest neighbors from the vector store.
Benefits
- Paraphrase tolerance: “Find where we discussed token fees” can match memories that never used the exact phrase.
- Better coverage: Works across synonymous wording and natural language variation.
- True offline capability: No external API calls needed for embedding or retrieval.
Practical note
Because you’re storing vectors locally, you control:
- the embedding model version,
- embedding granularity (sentence vs paragraph chunks),
- and how often you compact/refresh indices.
4) Hybrid ranking: combine FTS5 + embeddings for stronger recall
Most high-quality offline memory systems use hybrid retrieval:
- FTS5 candidate set for precision (keyword relevance)
- Embedding candidate set for recall (semantic relevance)
Then combine results using a ranking strategy.
A simple hybrid strategy
- Run FTS5:
- Collect top N₁ passages.
- Score them by FTS relevance.
- Run embeddings:
- Collect top N₂ passages.
- Score them by vector similarity.
- Merge and rerank:
- If a passage appears in both sets, it’s a strong signal.
- Normalize scores (since FTS and vector scores aren’t directly comparable).
- Apply a weighted blend, e.g.:
final_score = w_keyword * score_fts + w_semantic * score_vec

Why hybrid works well
- FTS5 handles exact anchors: identifiers, names, and exact phrases.
- Embeddings handle meaning: when the user doesn’t know the exact words used earlier.
- Reranking improves robustness: each method covers the other’s blind spots.
5) Chunking and metadata: make retrieval dependable
To get good recall, you must decide what to store.
Chunking guidelines
- Split memory into manageable passages (e.g., paragraphs or ~200–800 tokens).
- Preserve enough context inside each chunk to be useful when presented to the model.
- Avoid overly small chunks that lose meaning, and overly large chunks that dilute relevance.
Metadata matters
Store fields like:
- source (message, file, log)
- timestamp or “session”
- tags (project, domain, tool)
- confidence or status (e.g., “verified”, “draft”)
Metadata can power more targeted searches:
- “Only recall from the last week”
- “Restrict to this project”
- “Search in code snippets only”
Even if the model never sees raw data, the retrieval layer can filter it before ranking.
6) How this removes token fees from recall
The cost saving comes from separating retrieval from generation.
Token-expensive behavior (typical)
- The agent stuffs large histories or repeatedly asks the LLM to “summarize and find.”
- To find a detail, it sends lots of text and pays for every input token.
Token-free (offline) behavior (hybrid memory)
- The agent queries local FTS5 and local embeddings without calling any paid service.
- It returns only the most relevant chunks to the model.
- The model generates answers using a small context window derived from local retrieval.
So even if the model itself is called for final generation, the search/recall step—where most wasted tokens occur—no longer burns tokens.
In practice, this can reduce costs dramatically, especially for:
- long-running agents,
- repeated “where did we store X?” queries,
- debugging workflows with many past logs,
- and “bring me the correct policy/decision from earlier” moments.
7) Brief context: keyless setup and self-hosting (supporting role)
Many agent frameworks now offer “instant” access via built-in providers (keyless onboarding), and some workflows support fully local model runners. In an offline-first memory design, those conveniences are secondary: the real advantage is that embedding computation and text retrieval happen locally once your memory store is built.
8) Putting it together: the offline-first recall loop
A typical offline-first loop looks like this:
- Ingest memory
- Chunk content
- Index text into SQLite FTS5
- Compute local embeddings
- Store vectors for semantic search
- Recall
- Query FTS5 for keyword candidates
- Query embeddings for semantic candidates
- Hybrid-merge and rerank
- Generate
- Send only the top relevant chunks to the LLM
- Produce the response
- Update
- Store new insights and outcomes back into the local memory
At no point does the system need to “search the entire conversation” inside a model prompt.
Conclusion
Offline-first agent memory is more than a storage detail—it’s an architectural choice that makes recall fast, private, and dramatically cheaper. By combining:
- SQLite FTS5 for deterministic keyword search,
- local embeddings for semantic matching,
- and hybrid ranking to merge both signals,
you can achieve “zero-token recall”: the agent retrieves relevant past knowledge locally and only uses tokens for the final generation step. The result is an agent that remembers accurately without paying to rediscover its own history.