Episodic memory: how the machine remembers your customers

Series: The Build Track · Post 4 of 7

The highest-leverage subsystem in a company AI, and the one everyone skips. How we built memory that answers "why did we email this person?" with dates.

By Ben Blair · 7 min read

Here's a test you can run on any "AI-powered outreach" system, including the one a vendor is demoing for you right now. Pick a contact and ask: why did we email this person, and what do we know about them?

If the answer comes back with dates and receipts (researched them on this day, found this, sent this, they replied with that) the system has memory. If the answer is a confidence score and a vibe, it has a database pretending to be memory. Most systems are the second kind, because memory is the subsystem everyone skips. It's invisible in a demo. It pays off in month three.

I think it's the single highest-leverage thing we built. This chapter is the design, the two failures that preceded it, and the honest part where I show you our own audit numbers, which are humbling.

One table, timestamped, append-only in spirit

The core is almost embarrassingly simple. A table called contact_episodes. Every row is one thing that happened, tied to one contact, at one moment:

CREATE TABLE contact_episodes (
    id            INTEGER PRIMARY KEY AUTOINCREMENT,
    contact_id    INTEGER NOT NULL,
    episode_type  TEXT NOT NULL,
    content       TEXT NOT NULL,
    source        TEXT,
    confidence    REAL DEFAULT 1.0,
    created_at    TEXT NOT NULL
);

Seven episode types, and the list is a worldview: research (what we found out about them), company (about their organization), outreach (what we sent them), response (how they responded), engagement (social and content signals), note (things I add by hand), scoring (why their score changed).

Look at what that last one implies. When a contact's warmth changes, the reason gets written down as an episode, same as everything else. The machine doesn't just hold a score. It holds the score's biography. That's the difference between a CRM field and a memory.

The confidence column is the other quiet load-bearer. A fact scraped by a local model is not as trustworthy as a reply the human actually sent us, and the row says so. When memory feeds generation later, provenance is the difference between personalization and fabrication. Chapter 6 is about what happens when you skip that.

Indexes on contact_id and episode_type, and that's the whole design. No graph database, no embeddings required for the core loop. The most valuable subsystem in the machine is a flat table with a timestamp discipline.

How it gets written, and how it gets read

The outreach engine runs daily at 7:55 and is deliberately slow. Up to 20 contacts researched and 10 drafts a day, hard caps, by design. Slow matters here. Each researched contact gets episodes written for findings and company intel. Each draft sent becomes an outreach episode plus a row in an outreach table that tracks subject, body, the angle we took, sent/opened/replied timestamps, and bounce status. Replies come back through inbox triage and land as response episodes that also bump warmth. Follow-up bumps are tracked per thread (we added explicit bump counting this month, with a politely-twice-and-done rule), so "did we already nudge this person" is a query, not a guess.

The read side is where the table earns its keep. Before drafting, the engine pulls the contact's last 30 episodes, flips them chronological, and builds a narrative. A dossier, assembled fresh from receipts at the moment of writing. No prior history returns the honest answer: "No prior research or interaction history." And I get the same view from a terminal. One command, one email address, the full timeline of everything the machine knows about a person and every word it's exchanged with them.

That command is the audit trail. When the machine emails someone, I can reconstruct exactly why, from rows with dates. Keep that in mind for the alternatives section, because it's the property every alternative gives up first.

One unglamorous guardrail worth copying. When research wants to improve the CRM record itself, fill in a city, a title, a company, the writeback goes through a hardcoded whitelist of allowed columns. The model proposes; the whitelist disposes. A local model with open-ended write access to your contact database is a bad day waiting for a cron slot.

What failed first

Two designs died before this one, and they're the two everybody tries.

Failure one: stuff everything into the prompt. Dump the contact's whole CRM record, prior emails, and notes into context and let the model sort it out. Works for the first demo. Then histories grow, the context fills with mostly-irrelevant text, the model starts attending to the wrong details, and costs and latency scale with history length. Worse, there's no provenance. The model sees a soup of text with no marker for what's verified versus what some earlier model guessed. The soup is how guesses get laundered into facts.

Failure two: rolling summaries. Keep a per-contact summary, update it after each event. Compact, cheap, and a one-way information shredder. Each rewrite is a chance for the summary to drift from the evidence. A date dropped here, a hedge dropped there. After ten rewrites you have a confident paragraph nobody can trace to anything. The receipts are gone, and "why did we email this person" is back to being unanswerable.

The episodic design keeps raw events forever and synthesizes fresh at read time. Synthesis is cheap and disposable. Evidence is permanent. If you remember one sentence from this chapter: store events, derive summaries, never the reverse.

The nightly synthesis, and where vectors actually fit

Contact memory is one scope. The machine also remembers itself, and that runs on the same store-events-derive-summaries principle, nightly.

At 11:30pm, a synthesizer reads the last 24 hours of operational activity: scheduler runs, outreach activity, engagement numbers, blog and social output. It scores each observation for importance and writes it into a brain database of facts, decisions, and a knowledge graph. At 11:45, an export mirrors that to plain markdown files, because a memory only a program can read is a memory with one point of failure and zero auditors. I can open last Tuesday as a text file.

Vector memory sits beside all this, not under it. A local ChromaDB store holds embeddings, generated locally so nothing leaves the box, in two collections: knowledge and episodes. Vectors answer the question SQL can't, which is "have we seen something like this before?" Fuzzy recall across thousands of entries. What vectors don't do is serve as the system of record, because similarity is not provenance.

And vector stores rot. Quietly. Left alone, everything becomes dimly similar to everything and recall quality sinks. So a nightly aging pass scores memories one-to-five on importance, ages them by tier, moves the losers to an archive collection, deletes archived entries past a 180-day TTL, and enforces hard size caps on both collections. Forgetting is a feature you have to build. Nobody puts that on the architecture diagram, and it's half of why recall still works.

The humbling part

Now the audit, because this track promised receipts even when they're unflattering.

Our CRM holds 156,925 contacts. When we audited it this month: 99.8% cold. About 280 contacts warm or hot. Roughly 18,000 records clean enough to act on, 8.1% with an email address at all. And the metric that stung, across the whole CRM, the engagement-events table held nine rows. Nine. The episodic machinery worked beautifully for the contacts the outreach engine touched, and meanwhile the other 150-odd-thousand records were a years-deep contact dump generating no signal, because nothing was wired to capture any.

The lesson I'd hand you: memory infrastructure without signal capture is a filing cabinet in an empty office. The table is necessary and nowhere near sufficient. Every channel a customer can touch, your site, your inbox, your social, your phone, either writes an episode somewhere or that touch never happened, as far as the machine knows. We built the cabinet first and the wiring second. Build them together.

The alternatives, fairly

RAG-only. Embed your documents, retrieve at generation time. Genuinely the right call for knowledge: your service docs, your pricing. But RAG over documents gives you recall without history. It can tell you what your refund policy says. It cannot tell you that this customer asked about refunds twice in March and got no reply. Use RAG for what you know. It does not cover who you know.

Fine-tuning. Train customer knowledge into the weights. Wrong tool, and it's worth being plain about why. It's expensive. It's stale the day after training. You can't delete one customer from a weight matrix, which becomes a real problem the first time someone invokes a privacy right. And there's no audit trail at all, because the knowledge is smeared across parameters. Fine-tuning teaches a model how to behave. Facts about customers belong in a database the model reads, not in the model.

Framework memory modules. LangChain and friends ship memory classes: conversation buffers, entity memory, vector-store wrappers. Fine for getting a prototype to remember the conversation it's currently in. The trouble starts at audit time. The storage schema is the framework's, the summarization policy is the framework's, and when you need to know why the AI claimed something about a customer, you're reverse-engineering an abstraction instead of querying your own table. Memory is precisely the layer where I'd tell even framework-friendly engineers: own the schema. It's one table. You just read the whole thing.

The test

Pick a real customer. Ask your system, or your vendor's demo: why did we last contact this person, what did we know when we did, and what happened next? The answer should have dates on every line, and you should be able to see the rows it came from.

Mine passes for the contacts the machine has touched, with a one-line command. It fails honestly for the rest of the database, and the audit told me so in numbers, which is itself the system working.

The scheduler is the heartbeat

Series: The Build Track · Post 3 of 7

A company AI isn't a chatbot. It's ~40 scheduled jobs that run while you sleep. How our scheduler handles missed-job recovery, timeouts, and silent failure.

By Ben Blair · 7 min read

Ask someone what a company AI is and they'll describe a chatbot. A box you type into. The box answers.

A chatbot is idle until you show up. A company AI is the opposite. It's the thing that worked all night and has something to show you. The difference between those two isn't the model. It's the scheduler. Strip everything else away and my system is a job table: 44 entries in the weekday schedule as I write this, a reduced set of about 22 on weekends, a few flagged Monday-only or Sunday-only. Email triage at 6:30. Contact sync at 7:30. Outreach research at 7:55. Post dispatch at 9:45 and 5:30. Engagement collection at 6:30pm. Memory synthesis at 11:30pm. The overnight pipeline at half past midnight. Whether or not anyone's awake.

This is the chapter on the part that sounds least like AI and matters most. The scheduler is maybe a thousand lines of Python, and it's the organ everything else hangs off. Here's how ours works, what it took three incidents to learn, and when you should just use cron instead.

One daemon, one table, one ledger

Ours is a single Python daemon under PM2. The schedule itself is a dictionary in the source file: time of day, job key, the command to run, a description, and a per-job timeout. Every 30 seconds it wakes, checks the clock against the table, and runs whatever's due. Every run gets a row in a SQLite ledger with scheduled time, start, end, duration, return code, status, and error.

That ledger is the design center, and I want to dwell on it because it's the thing cron doesn't give you. The ledger is how the system answers "did contact sync run today?" without anyone grepping logs. It's what the morning briefing reads. It's what recovery reasons from. A schedule without a ledger is a wish. The ledger is what makes it an operation.

Two details in the run path that earned their place.

Record the start before you execute. The daemon writes a "running" row to the ledger before the subprocess launches, then updates that same row on completion. Why: if the scheduler restarts mid-job, the naive version sees no completed run for that slot and fires the job again. Double-firing a job that sends email is not a theoretical concern. The "running" row makes the duplicate check see the first attempt immediately.

Per-job timeouts, sized from evidence. The Telegram digest gets 120 seconds. The deep CRM research job gets four hours. The outreach cycle used to get 30 minutes until we did the math. Up to 20 research passes and 10 drafts a day, each making local-model and web calls at about a minute apiece, and 30 minutes was exactly at the ceiling. It's 60 now, and the comment in the source shows the arithmetic. A timeout you can't justify with numbers is a timeout that will page you eventually, in one direction or the other.

One more cheap habit that pays daily: at boot, the daemon walks the entire job table and checks that every script exists and compiles. A syntax check, nothing imported or executed, plus a check for duplicate job keys. It costs a second at startup and converts "the 2am job has a typo" from a silent overnight failure into a loud boot error.

Missed jobs, and the bug that taught us recovery

A box that runs 24/7 still reboots. Windows updates itself. PM2 restarts things. The power blinks. The question isn't whether you'll miss scheduled slots. It's what happens to the missed ones.

Our first recovery design used a heartbeat file. The daemon writes a timestamp every cycle, and on startup, if the heartbeat is old, re-run what was missed in the gap. Reasonable. Wrong. We hit a failure mode where the scheduler was stuck, not executing jobs, but its main loop was alive and happily writing fresh heartbeats. The heartbeat said all good. The ledger said nothing ran for hours. Recovery looked at the heartbeat and concluded there was nothing to recover.

The fix was a rule I'd now apply anywhere: recover from the ledger, not the pulse. Current logic ignores the heartbeat entirely for this purpose. Any slot more than ten minutes overdue with no row in today's ledger is missed, and gets run. Because it's keyed on the ledger, it's idempotent. You can run recovery repeatedly and it won't double-fire anything.

Then recovery taught us a second lesson. This June, a recovery pass after downtime took two hours, since recovered jobs run one at a time and some are long. While it ground through the backlog, three new slots came due, and the single-pass design silently skipped them. We lost three jobs to the act of recovering others. So recovery is now multi-pass. After a pass completes, re-scan, because the world moved while you were catching up. If your recovery logic has never eaten a job, it just hasn't been measured yet.

The alarm is the silence

Failure alerting has a trap in it: the alert path depends on the thing that's failing. If the scheduler is down, who tells you the scheduler is down?

Our answer is a daily digest with the polarity flipped. Every morning at 9:05, a job reads the ledger and sends a Telegram message. What ran, what failed, what was missed. The content is useful. The existence of the message is the real signal. The description in the job table literally says "absence of digest = scheduler dark." If 9:05 passes and my phone is quiet, that silence is the alarm, and it's an alarm no failure can suppress, because it doesn't require anything to be working.

The same philosophy runs the morning brief at 6:00. The machine reads its own ledgers and databases and reports on what happened overnight, before I've had coffee. The jobs feeding it are sequenced for exactly that; the growth-analysis pass runs at 5:30 specifically so its findings are fresh for the briefing. Fifteen minutes later, another job exports key database state to flat JSON files so anything else, including the AI brain when a session wakes up, can read system state instantly without touching live SQLite.

The pattern underneath all of it: the system reports to you. You don't go spelunking. If checking on your AI requires opening a terminal, you'll stop checking around week three.

Resume cursors, or why every long job must checkpoint

Here's the constraint that shapes long jobs. The contact sync walks a CRM of 156,925 contacts, pulling from Google's API at a thousand contacts a page, and its slot has a ten-minute timeout. A full pass doesn't fit. The naive loop runs ten minutes, dies, and starts from page one tomorrow, making zero net progress forever while looking busy in the logs. We shipped exactly that bug.

The fix is a resume cursor. The job caps itself at 15 pages per run, saves the API's page token to a small JSON file, and the next run continues where it left off. Same idea inside the time budget: the job watches the clock against a 520-second budget, under the 600-second timeout so it stops gracefully rather than being killed, and persists the remainder. There's even handling for the unglamorous edge case where a saved page token expires between runs. Detect the rejection, clear the cursor, restart cleanly from page one.

The general rule: any job that can be interrupted will be, so every long job must be able to die at any moment and lose nothing but the current bite. Checkpoint to disk, not memory. Make the work idempotent so re-processing a page is harmless. This is the least exciting paragraph in this chapter and it's worth more than most of the others.

When you should not build this

Fair turns for the alternatives, because a custom daemon is not the default answer.

Cron (or Windows Task Scheduler) is the right call for fewer than ten independent jobs where a missed run doesn't matter. It is rock-solid at firing on time. What it doesn't do: recovery, a ledger, timeouts, or any answer to "what happened last night?" beyond grep. We started closer to cron, a pile of PM2 restart-on-schedule entries, and consolidated into one daemon precisely when "did it run?" became a question I was asking daily.

Airflow and Prefect are real orchestrators and they're better than my daemon at almost everything. DAGs, retries, backfills, a UI. They're built for data teams, and they cost like it. Not money, operational surface. Airflow is itself several services that need tending. I think they're right the day a second engineer joins, and a tax before that. My entire scheduler is one file I can read in fifteen minutes, which is a feature with no checkbox on a comparison chart.

n8n and the workflow tools are good glue for event-driven flows. Webhook in, actions out. Scheduling is something they do, not what they are. The weakness shows at the ledger: when the question is "which of my 44 jobs ran, failed, or got skipped this week, and why," you want a database you own, not a run history inside someone's UI.

The blunt summary: the scheduler itself is a solved problem and you should steal rather than innovate. What's not solved generically is the stuff bolted to it. Recovery semantics that match your jobs. The silence-is-the-alarm digest. Cursors in everything long. That's where the thousand lines went.

The test

Before you call your system autonomous, answer one question: if your machine was off from 2am to 6am last night, what happens, and how do you find out?

If the answer is "the missed work runs, once, in order, and a message on my phone tells me it happened," you have a heartbeat. If the answer involves you noticing something's stale two days later, you have a chatbot with appointments.

The tech stack, with receipts

Series: The Build Track · Post 2 of 7

The exact stack behind a marketing AI that runs 24/7: Python, SQLite, Ollama, PM2, Cloudflare. What each piece costs, and the alternative we rejected.

By Ben Blair · 7 min read

Every "how I built my AI" post eventually shows you a stack diagram with twelve logos on it. Mine has about six, and half of them are free.

That's not humility, it's the design. Chapter 1 made the argument: rent the brain, own the body. This chapter is the body's parts list. What we actually run, what each piece costs, and the alternative we rejected at every layer. The pattern repeats all the way down: boring infrastructure, interesting behavior. Every place I could choose between a clever tool and a dull one, I took dull, and I'll tell you what dull bought me.

The stack on one page

• Compute: one Windows box, running 24/7. Python 3.13.
• Process manager: PM2 running 6 daemons. Health monitor, context watchdog, the Telegram bot, the master scheduler, a read-only dashboard, and a Cloudflare tunnel.
• Data: SQLite. 21 separate databases, one per domain, plus a ChromaDB vector store.
• Local model: Ollama, serving qwen models for bulk work. Drafts, classification, research summaries.
• Brain: Claude, through a consumer subscription. Never the API.
• Web face: a FastAPI service on port 8801 behind a Cloudflare tunnel, and a Next.js 14 portal on Cloudflare Pages with Supabase handling auth.

That's the whole thing. No Kubernetes, no message queue, no microservices. Now the receipts, layer by layer.

A Windows box and PM2, which nobody recommends

The respectable answer is a Linux server with everything in Docker. I run Windows because the machine I own runs Windows, and the machine I own costs zero dollars a month.

The fair framing: Docker buys you reproducibility, and reproducibility matters when more than one person deploys the system, or when you deploy it more than once. I am one person deploying it once. What I actually need is "these six processes stay up and restart when they crash." PM2 is a process manager from the Node world that happily runs Python, and it does exactly that. The scheduler daemon, the health monitor, the Telegram bridge, the dashboard, the tunnel. PM2 keeps them alive and gives me one command to see all of them.

This choice cost me, and I'll itemize. Windows has sharp edges that Linux tutorials never mention. Subprocesses need special flags so they don't spawn visible windows. File paths fight you. The worst one took a while to even diagnose: when I started reading these databases from a Linux sandbox over a Windows file mount, the mount served stale, truncated views of large files that were actively growing. My 120 MB CRM database showed up corrupted on the other side. It wasn't corrupted. The mount was lying.

The fix was a tool that runs natively on the Windows box and uses SQLite's online backup API to write consistent point-in-time snapshots, plus a manifest with row counts so anything downstream can check freshness without opening the big file. That job now runs every morning at 08:15. Nobody's stack diagram includes "the job that exists because file mounts lie." Mine does.

If you're starting fresh and you're comfortable in Linux, use Linux. The architecture doesn't care. The principle is just: use the machine you already own before you rent one.

SQLite, which everybody outgrows except they don't

The CRM database holds 156,925 contacts in a single 120 MB SQLite file. Conventional wisdom says that should be Postgres by now. Conventional wisdom is off by about two orders of magnitude.

SQLite's real limit isn't size. It's concurrent writers. A web app with a thousand users hammering the same table needs Postgres. A company AI is the opposite shape: a handful of scheduled jobs, each writing to its own domain, mostly at different times of day. So we run 21 separate databases. One for the CRM, one for outreach history, one for social content, one for engagement telemetry, one for the scheduler's own run ledger, and so on. One database per domain means one writer per domain, which means SQLite's weakness never comes up. Each connection runs in WAL mode with a busy timeout, and that is the entire concurrency story.

Every database is a file. Backup is copying a file. Inspection is opening a file. The snapshot job backs up the important ones daily, and a monthly restore drill actually extracts a snapshot and checks integrity, because a backup you've never restored is a hypothesis.

When SQLite is actually wrong: multiple processes writing the same tables at the same moment, or you need someone else's machine to query your data over a network. The day I have employees doing concurrent writes, I move that one database to Postgres and nothing else changes. Files buy you that too. Migrations stay per-domain instead of big-bang.

Ollama and the local models, with the scar

Ollama serves local models on port 11434. The workhorses are qwen. qwen2.5-coder for bulk generation, and qwen3:8b for the outreach engine's research and drafting, which we switched to in May because it follows JSON instructions better. The marginal cost of a local model call is electricity. That's why the daily social cycle can afford roughly 96 sequential model calls, and the overnight content scan can push 18 accounts through a content filter without anyone watching a meter.

Here's the scar. Chapter 1 mentioned it and chapter 6 has the full story, but the stack lesson belongs here: a local model at this size will fabricate with total confidence. Ours invented client case studies, and two of them auto-published before the system caught it. The fix was not a better prompt. Prompts are requests. The fix was a mechanical gate: every number and named claim in generated content must exist in the source material the model was given, checked by code, not by asking nicely.

So the stack rule I'd give you is this. Local models are a labor layer, never a judgment layer, and between the labor layer and the public internet there has to be a gate made of code. If your stack has Ollama and no gate, you have a fabrication engine with a scheduler attached.

The alternative, doing bulk work through a paid API, is genuinely simpler and the models are better. But bulk work is where the volume is, and volume on a taxi meter is how hobby projects grow enterprise bills.

ChromaDB, locally, because vector stores rot quietly

Semantic memory lives in a local ChromaDB store with two collections, general knowledge and episodes, using embeddings generated by Ollama. Nothing leaves the box, even for indexing.

The hosted alternatives (Pinecone and friends) are good products solving a problem I don't have, which is serving vector search at scale to many users. My vector store has exactly one customer: the machine itself. What I do have, and the hosted pitch never mentions, is rot. A vector store that only grows becomes a memory where everything is dimly relevant to everything. Ours runs a nightly aging pass. Entries get scored by importance, archived entries get deleted after a 180-day TTL, and both collections have hard size caps. Chapter 4 goes deep on this. The stack-level point: wherever your vectors live, the compaction job is not optional, and you'll write it yourself either way.

The public face: tunnel, portal, Supabase

The website parts are the most conventional layer. A FastAPI service sits on port 8801 and a Cloudflare tunnel exposes it. Engine status is public; dashboard and CRM routes sit behind bearer-token auth. The client portal is Next.js 14 on Cloudflare Pages, with Supabase doing auth (Google, LinkedIn, email) and its database. A portal signup fires a webhook into the CRM, which is the moment a website visitor becomes a row the rest of the machine can act on.

The tunnel deserves a sentence of advocacy. It means the Windows box exposes no ports and I never touched a router config. The box makes an outbound connection to Cloudflare and traffic flows back through it. For a machine sitting in a house, I think this is flatly the right answer.

The bill, itemized

• Claude subscription: the one fixed cost that matters. Same consumer subscription anyone buys; current pricing is on Anthropic's site. This is the brain.
• Anthropic API: $0, by design. There's an API key in the environment file and a standing rule that nothing uses it.
• X API: about 35 cents a month. The other platform APIs we use are on free tiers.
• Hosting: roughly $0. Cloudflare Pages and the tunnel are free tier. Supabase is free tier at this size. The server is a PC I already owned, so the real cost is electricity.
• Ollama and every model on it: free software, electricity.

There are paid services wired in for specific jobs, an email-sending platform and a contact-enrichment API, that scale with usage and sit near zero when held. But the load-bearing monthly number is the Claude subscription plus pocket change. That's not an accident. It's what "rent the brain, own the body" looks like on a bank statement.

What this list is actually telling you

Notice what's absent. No agent framework, no orchestration platform, no managed anything in the hot path. Every component is either a file, a process, or a model server, and every one of them can be inspected with tools that existed fifteen years ago. Except the models.

That's the trade I keep making, and the one I'd push you toward: spend your novelty budget on behavior, not infrastructure. The interesting parts of this system live in the memory, the training loop, and the guardrails. All of it is interesting Python sitting on aggressively boring foundations. When something breaks at 2am the foundation is never the suspect, which keeps the search space small, which is why I sleep.

If you're an engineer, you could assemble this layer list in a weekend. The chapters ahead are why assembly was the easy part: the scheduler that keeps it honest, the memory that makes it useful, the guardrails we earned. If you're an owner and the words "WAL mode" made your eyes slide, that's useful data too.