Training modules: improving on a schedule, not by magic

Series: The Build Track · Post 5 of 7

Most "self-improving AI" claims are unfalsifiable. Here's a real training loop: telemetry, scored hypotheses, a weekly review, and a tuner with hard caps.

By Ben Blair · 7 min read

"Self-improving AI" is the most common claim in this industry and the least often checked. Ask a vendor how their system improves and you'll get the word "learns" doing a lot of unpaid labor. Learns how? From what data? Measured when? Improved compared to what?

Here's my position: a system improves if and only if you can write a SQL query that shows it improving. Everything else is vibes.

This chapter is the training loop we actually run. Telemetry in, hypotheses out, a weekly review that scores them, and a tuner that adjusts behavior inside hard limits. It's the least magical chapter in this track. That's deliberate. The loop is cron jobs and database joins. It is also the part that makes the whole machine worth owning, because a system that acts without measuring is just automation, and automation that never gets better is a depreciating asset.

The loop, end to end

Five stages, every one of them a scheduled job or a recurring session:

1. Telemetry. Every action the machine takes, every post and every send, gets a row in a telemetry database the moment it executes. Not a log line. A row, with a timestamp and a platform action id, in a table built to be joined against.
2. Responses. A collector runs every evening at 18:30 and fetches engagement on everything we've posted: likes, comments, impressions. Each pull is a row joined back to the original action.
3. Hypotheses. Every daily plan states, in writing, what it's testing. Not "post good content." Something falsifiable: operator-pain content beats case-study content on per-post engagement, same page, same slot, same count.
4. Review. Sunday, the system reviews its week the way a director would. It scores the open hypotheses against the response data and writes down what it believes now and why.
5. Tuning. A nightly tuner proposes envelope adjustments to posting counts and times, based on the evidence. It can move settings within hard caps. It cannot move the caps. That distinction is chapter 6's whole subject.

Notice what's missing: any step called "the model gets smarter." The model doesn't change. The settings change, the content choices change, and the written record of what works grows. The brain is rented, remember. What improves is the body and the playbook. The parts we own.

The embarrassing part first

For weeks, our responses table was empty.

I want to sit on that for a second, because it's the most common failure in this whole category and almost nobody admits it. We had telemetry on every action. We had daily plans with named hypotheses. We had a Sunday review. And the join that connects what we did to what happened didn't exist yet. The machine was running experiments and never collecting the results. Write-only science.

The daily planning log from that stretch says it plainly: "every lever experiment we run is write-only until outcomes join back." The plan for each day was disciplined. And unscoreable.

The fix was a response collector, and the design decisions in it are the generic playbook, so here they are.

Reuse the posting path for measurement. The same authenticated adapters that publish posts can fetch engagement on them. One auth per account per run, joined on the platform's action id, which the dispatcher logs on every executed post. Don't build a second integration to measure the first one.

Get longitudinal data from cadence, not complexity. The collector runs daily with a 20-hour dedupe gap, which automatically yields roughly 24, 48, and 72-hour engagement curves per post, out to a 14-day horizon. No special scheduling logic. Just a daily job and arithmetic.

Absence is signal. If the collector can't find a post it knows was published, it doesn't stay silent. It writes a row recording zero visibility and the failure reason. A post that disappeared is the platform telling you something. Silence in your own pipeline is the one thing you can't afford.

The first real collection run is worth reporting honestly, because this is what day one of measurement actually looks like. Facebook posts measured fine: older machine-generated posts sitting at 9 to 10 likes each with zero comments, and a newer hand-voiced post at 0 likes at the 15-hour mark, which is not yet a comparison because the ages don't match. X: 5 impressions, zero engagement, on a cold account. Expected. LinkedIn: both fetches failed outright, and the first version of the collector swallowed the error code, so we patched it the same day to record the HTTP status instead. Day one of measurement is mostly discovering what your measurement can't see yet.

That's the real texture of a training loop. Not a dashboard going up and to the right. A join that finally returns rows, half of them telling you about your own bugs.

Hypotheses you can score

The discipline that makes the loop work is older than AI: change one variable at a time.

Here is an actual day's experiment design from our planner, lightly compressed. Three pages, one post each. On LinkedIn, the theme changes from yesterday, operator-pain content instead of case-study content, while the slot and count stay fixed, so theme is the isolated variable. On Facebook, same midday slot as yesterday, theme rotated. On X, the content stays in the same class but the slot moves from midday to evening. The first deliberate point on that account's timing curve. Each hypothesis gets a name: H-0612a, H-0612b, H-0612c. Each pairs against a prior day's named hypothesis. All of them sit open until the response data is old enough to score.

The plan also held posting volume at one per page per day, at or below the soft targets, with the reasoning written down: with no outcome data yet, "volume buys risk, not learning."

One more honest detail. Before we had our own data, the system's "best posting times" were just platform priors. Generic recommendations, identical repeated values, the kind any blog post would give you. The machine knew nothing it hadn't been told. The entire point of the loop is to replace those priors with observed signal, one named hypothesis at a time. Any vendor whose system "knows the best time to post" on day one is showing you priors with a confident font.

The Sunday review and the tuner

Two different jobs close the loop, and the separation matters.

The director review is judgment. Once a week, the brain reads the week's telemetry, scores the open hypotheses, and writes conclusions in prose. What we now believe about themes, timing, and platforms, and what to test next. This is part of the roughly ninety minutes a day of actual reasoning I described in chapter 1, batched into its most valuable session.

The envelope tuner is mechanism. It runs on schedule, reads the same data, and proposes parameter changes within hard caps it cannot touch. Its early runs proposed nothing, and reported why: not enough days of data yet across the platforms. A tuner that says "no changes, insufficient evidence" is working correctly. A tuner that always finds something to change is fitting noise.

Judgment proposes strategy. Mechanism adjusts dials. Neither one gets to move the safety limits. If you collapse those three into one component, and most agent frameworks do, you can't audit which one made any given decision.

Decision journals are the actual training corpus

Every planning session writes a log: what was posted, why, what hypothesis it serves, and a lever-settings record covering counts, times, content sources, and budget, in a fixed format. Every review writes its scores and conclusions. These files pile up into the most valuable dataset we own, and it isn't engagement numbers. It's decisions with stated reasoning, which means any future brain, a new model, a better model, a model that doesn't exist yet, can read six months of judgment calls and inherit them.

One rule makes the corpus reusable: keep generic decision rules separate from business facts. "Isolate one variable per page per day" is a rule. It transfers to any client, any business. "This account's audience skews older and slower" is a fact about one account. We write them in separable form on purpose, because the rules half of the corpus is a product. It's the playbook a second deployment starts from. Mix them and you've got a diary. Separate them and you've got a methodology with local annotations.

What falsifiable looks like

So, the test I promised. A self-improvement claim is falsifiable when you can express it as a query: actions joined to responses, grouped by the variable you claimed to be testing, filtered by date range, compared across the change. Posts with theme X averaged this engagement at 48 hours; posts with theme Y at the same slot averaged that. If the system can't produce that table, it isn't learning. It's drifting, and someone is narrating the drift as progress.

That's also the question to ask any vendor selling adaptive anything: show me the join. Show me the table of actions, the table of outcomes, the key that connects them, and one decision the system made differently because of it. We can show ours, including the weeks the outcomes table was empty. The empty weeks are part of the receipt.

The loop only earns its keep if the machine acting on it can't hurt you while it learns. Which is why the next chapter exists, and why it's the one with our worst day in it.

Episodic memory: how the machine remembers your customers

Series: The Build Track · Post 4 of 7

The highest-leverage subsystem in a company AI, and the one everyone skips. How we built memory that answers "why did we email this person?" with dates.

By Ben Blair · 7 min read

Here's a test you can run on any "AI-powered outreach" system, including the one a vendor is demoing for you right now. Pick a contact and ask: why did we email this person, and what do we know about them?

If the answer comes back with dates and receipts (researched them on this day, found this, sent this, they replied with that) the system has memory. If the answer is a confidence score and a vibe, it has a database pretending to be memory. Most systems are the second kind, because memory is the subsystem everyone skips. It's invisible in a demo. It pays off in month three.

I think it's the single highest-leverage thing we built. This chapter is the design, the two failures that preceded it, and the honest part where I show you our own audit numbers, which are humbling.

One table, timestamped, append-only in spirit

The core is almost embarrassingly simple. A table called contact_episodes. Every row is one thing that happened, tied to one contact, at one moment:

CREATE TABLE contact_episodes (
    id            INTEGER PRIMARY KEY AUTOINCREMENT,
    contact_id    INTEGER NOT NULL,
    episode_type  TEXT NOT NULL,
    content       TEXT NOT NULL,
    source        TEXT,
    confidence    REAL DEFAULT 1.0,
    created_at    TEXT NOT NULL
);

Seven episode types, and the list is a worldview: research (what we found out about them), company (about their organization), outreach (what we sent them), response (how they responded), engagement (social and content signals), note (things I add by hand), scoring (why their score changed).

Look at what that last one implies. When a contact's warmth changes, the reason gets written down as an episode, same as everything else. The machine doesn't just hold a score. It holds the score's biography. That's the difference between a CRM field and a memory.

The confidence column is the other quiet load-bearer. A fact scraped by a local model is not as trustworthy as a reply the human actually sent us, and the row says so. When memory feeds generation later, provenance is the difference between personalization and fabrication. Chapter 6 is about what happens when you skip that.

Indexes on contact_id and episode_type, and that's the whole design. No graph database, no embeddings required for the core loop. The most valuable subsystem in the machine is a flat table with a timestamp discipline.

How it gets written, and how it gets read

The outreach engine runs daily at 7:55 and is deliberately slow. Up to 20 contacts researched and 10 drafts a day, hard caps, by design. Slow matters here. Each researched contact gets episodes written for findings and company intel. Each draft sent becomes an outreach episode plus a row in an outreach table that tracks subject, body, the angle we took, sent/opened/replied timestamps, and bounce status. Replies come back through inbox triage and land as response episodes that also bump warmth. Follow-up bumps are tracked per thread (we added explicit bump counting this month, with a politely-twice-and-done rule), so "did we already nudge this person" is a query, not a guess.

The read side is where the table earns its keep. Before drafting, the engine pulls the contact's last 30 episodes, flips them chronological, and builds a narrative. A dossier, assembled fresh from receipts at the moment of writing. No prior history returns the honest answer: "No prior research or interaction history." And I get the same view from a terminal. One command, one email address, the full timeline of everything the machine knows about a person and every word it's exchanged with them.

That command is the audit trail. When the machine emails someone, I can reconstruct exactly why, from rows with dates. Keep that in mind for the alternatives section, because it's the property every alternative gives up first.

One unglamorous guardrail worth copying. When research wants to improve the CRM record itself, fill in a city, a title, a company, the writeback goes through a hardcoded whitelist of allowed columns. The model proposes; the whitelist disposes. A local model with open-ended write access to your contact database is a bad day waiting for a cron slot.

What failed first

Two designs died before this one, and they're the two everybody tries.

Failure one: stuff everything into the prompt. Dump the contact's whole CRM record, prior emails, and notes into context and let the model sort it out. Works for the first demo. Then histories grow, the context fills with mostly-irrelevant text, the model starts attending to the wrong details, and costs and latency scale with history length. Worse, there's no provenance. The model sees a soup of text with no marker for what's verified versus what some earlier model guessed. The soup is how guesses get laundered into facts.

Failure two: rolling summaries. Keep a per-contact summary, update it after each event. Compact, cheap, and a one-way information shredder. Each rewrite is a chance for the summary to drift from the evidence. A date dropped here, a hedge dropped there. After ten rewrites you have a confident paragraph nobody can trace to anything. The receipts are gone, and "why did we email this person" is back to being unanswerable.

The episodic design keeps raw events forever and synthesizes fresh at read time. Synthesis is cheap and disposable. Evidence is permanent. If you remember one sentence from this chapter: store events, derive summaries, never the reverse.

The nightly synthesis, and where vectors actually fit

Contact memory is one scope. The machine also remembers itself, and that runs on the same store-events-derive-summaries principle, nightly.

At 11:30pm, a synthesizer reads the last 24 hours of operational activity: scheduler runs, outreach activity, engagement numbers, blog and social output. It scores each observation for importance and writes it into a brain database of facts, decisions, and a knowledge graph. At 11:45, an export mirrors that to plain markdown files, because a memory only a program can read is a memory with one point of failure and zero auditors. I can open last Tuesday as a text file.

Vector memory sits beside all this, not under it. A local ChromaDB store holds embeddings, generated locally so nothing leaves the box, in two collections: knowledge and episodes. Vectors answer the question SQL can't, which is "have we seen something like this before?" Fuzzy recall across thousands of entries. What vectors don't do is serve as the system of record, because similarity is not provenance.

And vector stores rot. Quietly. Left alone, everything becomes dimly similar to everything and recall quality sinks. So a nightly aging pass scores memories one-to-five on importance, ages them by tier, moves the losers to an archive collection, deletes archived entries past a 180-day TTL, and enforces hard size caps on both collections. Forgetting is a feature you have to build. Nobody puts that on the architecture diagram, and it's half of why recall still works.

The humbling part

Now the audit, because this track promised receipts even when they're unflattering.

Our CRM holds 156,925 contacts. When we audited it this month: 99.8% cold. About 280 contacts warm or hot. Roughly 18,000 records clean enough to act on, 8.1% with an email address at all. And the metric that stung, across the whole CRM, the engagement-events table held nine rows. Nine. The episodic machinery worked beautifully for the contacts the outreach engine touched, and meanwhile the other 150-odd-thousand records were a years-deep contact dump generating no signal, because nothing was wired to capture any.

The lesson I'd hand you: memory infrastructure without signal capture is a filing cabinet in an empty office. The table is necessary and nowhere near sufficient. Every channel a customer can touch, your site, your inbox, your social, your phone, either writes an episode somewhere or that touch never happened, as far as the machine knows. We built the cabinet first and the wiring second. Build them together.

The alternatives, fairly

RAG-only. Embed your documents, retrieve at generation time. Genuinely the right call for knowledge: your service docs, your pricing. But RAG over documents gives you recall without history. It can tell you what your refund policy says. It cannot tell you that this customer asked about refunds twice in March and got no reply. Use RAG for what you know. It does not cover who you know.

Fine-tuning. Train customer knowledge into the weights. Wrong tool, and it's worth being plain about why. It's expensive. It's stale the day after training. You can't delete one customer from a weight matrix, which becomes a real problem the first time someone invokes a privacy right. And there's no audit trail at all, because the knowledge is smeared across parameters. Fine-tuning teaches a model how to behave. Facts about customers belong in a database the model reads, not in the model.

Framework memory modules. LangChain and friends ship memory classes: conversation buffers, entity memory, vector-store wrappers. Fine for getting a prototype to remember the conversation it's currently in. The trouble starts at audit time. The storage schema is the framework's, the summarization policy is the framework's, and when you need to know why the AI claimed something about a customer, you're reverse-engineering an abstraction instead of querying your own table. Memory is precisely the layer where I'd tell even framework-friendly engineers: own the schema. It's one table. You just read the whole thing.

The test

Pick a real customer. Ask your system, or your vendor's demo: why did we last contact this person, what did we know when we did, and what happened next? The answer should have dates on every line, and you should be able to see the rows it came from.

Mine passes for the contacts the machine has touched, with a one-line command. It fails honestly for the rest of the database, and the audit told me so in numbers, which is itself the system working.

The scheduler is the heartbeat

Series: The Build Track · Post 3 of 7

A company AI isn't a chatbot. It's ~40 scheduled jobs that run while you sleep. How our scheduler handles missed-job recovery, timeouts, and silent failure.

By Ben Blair · 7 min read

Ask someone what a company AI is and they'll describe a chatbot. A box you type into. The box answers.

A chatbot is idle until you show up. A company AI is the opposite. It's the thing that worked all night and has something to show you. The difference between those two isn't the model. It's the scheduler. Strip everything else away and my system is a job table: 44 entries in the weekday schedule as I write this, a reduced set of about 22 on weekends, a few flagged Monday-only or Sunday-only. Email triage at 6:30. Contact sync at 7:30. Outreach research at 7:55. Post dispatch at 9:45 and 5:30. Engagement collection at 6:30pm. Memory synthesis at 11:30pm. The overnight pipeline at half past midnight. Whether or not anyone's awake.

This is the chapter on the part that sounds least like AI and matters most. The scheduler is maybe a thousand lines of Python, and it's the organ everything else hangs off. Here's how ours works, what it took three incidents to learn, and when you should just use cron instead.

One daemon, one table, one ledger

Ours is a single Python daemon under PM2. The schedule itself is a dictionary in the source file: time of day, job key, the command to run, a description, and a per-job timeout. Every 30 seconds it wakes, checks the clock against the table, and runs whatever's due. Every run gets a row in a SQLite ledger with scheduled time, start, end, duration, return code, status, and error.

That ledger is the design center, and I want to dwell on it because it's the thing cron doesn't give you. The ledger is how the system answers "did contact sync run today?" without anyone grepping logs. It's what the morning briefing reads. It's what recovery reasons from. A schedule without a ledger is a wish. The ledger is what makes it an operation.

Two details in the run path that earned their place.

Record the start before you execute. The daemon writes a "running" row to the ledger before the subprocess launches, then updates that same row on completion. Why: if the scheduler restarts mid-job, the naive version sees no completed run for that slot and fires the job again. Double-firing a job that sends email is not a theoretical concern. The "running" row makes the duplicate check see the first attempt immediately.

Per-job timeouts, sized from evidence. The Telegram digest gets 120 seconds. The deep CRM research job gets four hours. The outreach cycle used to get 30 minutes until we did the math. Up to 20 research passes and 10 drafts a day, each making local-model and web calls at about a minute apiece, and 30 minutes was exactly at the ceiling. It's 60 now, and the comment in the source shows the arithmetic. A timeout you can't justify with numbers is a timeout that will page you eventually, in one direction or the other.

One more cheap habit that pays daily: at boot, the daemon walks the entire job table and checks that every script exists and compiles. A syntax check, nothing imported or executed, plus a check for duplicate job keys. It costs a second at startup and converts "the 2am job has a typo" from a silent overnight failure into a loud boot error.

Missed jobs, and the bug that taught us recovery

A box that runs 24/7 still reboots. Windows updates itself. PM2 restarts things. The power blinks. The question isn't whether you'll miss scheduled slots. It's what happens to the missed ones.

Our first recovery design used a heartbeat file. The daemon writes a timestamp every cycle, and on startup, if the heartbeat is old, re-run what was missed in the gap. Reasonable. Wrong. We hit a failure mode where the scheduler was stuck, not executing jobs, but its main loop was alive and happily writing fresh heartbeats. The heartbeat said all good. The ledger said nothing ran for hours. Recovery looked at the heartbeat and concluded there was nothing to recover.

The fix was a rule I'd now apply anywhere: recover from the ledger, not the pulse. Current logic ignores the heartbeat entirely for this purpose. Any slot more than ten minutes overdue with no row in today's ledger is missed, and gets run. Because it's keyed on the ledger, it's idempotent. You can run recovery repeatedly and it won't double-fire anything.

Then recovery taught us a second lesson. This June, a recovery pass after downtime took two hours, since recovered jobs run one at a time and some are long. While it ground through the backlog, three new slots came due, and the single-pass design silently skipped them. We lost three jobs to the act of recovering others. So recovery is now multi-pass. After a pass completes, re-scan, because the world moved while you were catching up. If your recovery logic has never eaten a job, it just hasn't been measured yet.

The alarm is the silence

Failure alerting has a trap in it: the alert path depends on the thing that's failing. If the scheduler is down, who tells you the scheduler is down?

Our answer is a daily digest with the polarity flipped. Every morning at 9:05, a job reads the ledger and sends a Telegram message. What ran, what failed, what was missed. The content is useful. The existence of the message is the real signal. The description in the job table literally says "absence of digest = scheduler dark." If 9:05 passes and my phone is quiet, that silence is the alarm, and it's an alarm no failure can suppress, because it doesn't require anything to be working.

The same philosophy runs the morning brief at 6:00. The machine reads its own ledgers and databases and reports on what happened overnight, before I've had coffee. The jobs feeding it are sequenced for exactly that; the growth-analysis pass runs at 5:30 specifically so its findings are fresh for the briefing. Fifteen minutes later, another job exports key database state to flat JSON files so anything else, including the AI brain when a session wakes up, can read system state instantly without touching live SQLite.

The pattern underneath all of it: the system reports to you. You don't go spelunking. If checking on your AI requires opening a terminal, you'll stop checking around week three.

Resume cursors, or why every long job must checkpoint

Here's the constraint that shapes long jobs. The contact sync walks a CRM of 156,925 contacts, pulling from Google's API at a thousand contacts a page, and its slot has a ten-minute timeout. A full pass doesn't fit. The naive loop runs ten minutes, dies, and starts from page one tomorrow, making zero net progress forever while looking busy in the logs. We shipped exactly that bug.

The fix is a resume cursor. The job caps itself at 15 pages per run, saves the API's page token to a small JSON file, and the next run continues where it left off. Same idea inside the time budget: the job watches the clock against a 520-second budget, under the 600-second timeout so it stops gracefully rather than being killed, and persists the remainder. There's even handling for the unglamorous edge case where a saved page token expires between runs. Detect the rejection, clear the cursor, restart cleanly from page one.

The general rule: any job that can be interrupted will be, so every long job must be able to die at any moment and lose nothing but the current bite. Checkpoint to disk, not memory. Make the work idempotent so re-processing a page is harmless. This is the least exciting paragraph in this chapter and it's worth more than most of the others.

When you should not build this

Fair turns for the alternatives, because a custom daemon is not the default answer.

Cron (or Windows Task Scheduler) is the right call for fewer than ten independent jobs where a missed run doesn't matter. It is rock-solid at firing on time. What it doesn't do: recovery, a ledger, timeouts, or any answer to "what happened last night?" beyond grep. We started closer to cron, a pile of PM2 restart-on-schedule entries, and consolidated into one daemon precisely when "did it run?" became a question I was asking daily.

Airflow and Prefect are real orchestrators and they're better than my daemon at almost everything. DAGs, retries, backfills, a UI. They're built for data teams, and they cost like it. Not money, operational surface. Airflow is itself several services that need tending. I think they're right the day a second engineer joins, and a tax before that. My entire scheduler is one file I can read in fifteen minutes, which is a feature with no checkbox on a comparison chart.

n8n and the workflow tools are good glue for event-driven flows. Webhook in, actions out. Scheduling is something they do, not what they are. The weakness shows at the ledger: when the question is "which of my 44 jobs ran, failed, or got skipped this week, and why," you want a database you own, not a run history inside someone's UI.

The blunt summary: the scheduler itself is a solved problem and you should steal rather than innovate. What's not solved generically is the stuff bolted to it. Recovery semantics that match your jobs. The silence-is-the-alarm digest. Cursors in everything long. That's where the thousand lines went.

The test

Before you call your system autonomous, answer one question: if your machine was off from 2am to 6am last night, what happens, and how do you find out?

If the answer is "the missed work runs, once, in order, and a message on my phone tells me it happened," you have a heartbeat. If the answer involves you noticing something's stale two days later, you have a chatbot with appointments.