Build, buy, or hire: the honest math

Series: The Build Track · Post 7 of 7

When building a company AI yourself is the highest-leverage move you can make, and when it's a trap. The real skill stack, the hard parts, and the owner's math.

By Ben Blair · 7 min read

Six chapters ago I promised this track would be useful to the engineer and honest with the owner. This is the chapter where those two readers part ways, so let me say the conclusion up front and spend the rest earning it.

If you have an engineer with slack time and a long horizon, building this is one of the highest-leverage projects available to a small company. If you're the owner planning to build it yourself at 9pm after the kids are down, it's a trap. I say that as the person who built one.

What this actually took

Let me inventory what's running, because scope is the real measure of effort. Six background daemons under a process manager. A scheduler running roughly 40 jobs a day across weekday and weekend sets. Twenty-one SQLite databases plus a vector store. Fourteen tool subsystems: lead pipeline, social engine, local reasoning, memory, inbox, content, analytics, site optimization, ops. A CRM holding 156,925 contacts. Platform integrations for LinkedIn, Facebook, Instagram, and X, each with its own auth lifecycle. An envelope system, a fabrication gate, four kill-switch levels, telemetry, a response collector, a nightly tuner, and a weekly review. Plus a public website, a client portal, and a chat service behind a tunnel.

I'm not going to give you an hour count, because I didn't keep one and I refuse to invent a number in the chapter after the one about fabrication gates. What I can tell you honestly: this was months of evenings and full working days, not a weekend. The first working version of any one subsystem was usually days. The version that survives unattended operation was always weeks, and several pieces took three tries. Chapter 4's memory system replaced two failed designs. Chapter 6 documents the incident that built the fabrication gate. The gap between "demo" and "runs at 6am while I sleep without burning anything down" is where almost all the time went. I'd estimate the demo is a tenth of the work, and even that ratio might be generous.

And it isn't finished, because it's not a project. It's an operation. At one point our Google OAuth sat revoked for nine weeks because re-authing it kept losing to higher-priority fires. The maintenance load is real and permanent.

The skill stack, honestly

What you'd actually need to know. None of it is exotic. All of it is required at once, which is the real filter.

Python, comfortably. Not algorithms. Production habits. Logging, retries, timeouts, idempotency, resumable batch jobs. Our contact sync walks 157,000 contacts in small resumable bites because it will be interrupted, and code that can't resume is code that silently starts over.

SQL, genuinely. The training loop in chapter 5 is a join. If "actions joined to responses, grouped by theme, filtered by date" isn't something you can write without looking it up, the measurement layer won't exist, and you'll be running the unfalsifiable kind of "self-improving" system.

OAuth wrangling. The least glamorous and most relentlessly time-consuming skill on the list. Every platform has its own token lifetime, refresh quirks, scope vocabulary, and app-review gauntlet. We run a daily keepalive job whose entire purpose is making sure tokens don't quietly die. Plan for auth to be a permanent line item, not a setup step.

API quirks as a temperament. Documented limits that are wrong. Endpoints that return different shapes on Tuesdays. An engagement fetch that fails only for company pages because the token wants a scope nobody mentions. Our first LinkedIn measurement run failed exactly that way. The skill isn't knowing the quirks. It's patience plus diagnostics, recording the error code instead of swallowing it.

Prompt design, the boring kind. Not clever prompts. Constrained ones, with the discipline to never trust the constraint and always back it with a mechanical check. Chapters 5 and 6 are really one lesson: the model is labor, and the checking is yours.

Ops discipline. The rarest one. Reading your own morning brief every day. Working the approval queue instead of letting it hit 163. Running the restore drill. Writing the decision journal when nobody's grading you. Engineers underestimate this one most, because it isn't a skill you learn. It's a habit you keep.

What's hard versus what looks hard

The parts that look impressive are mostly easy. Posting to a social API is a solved problem, an afternoon per platform for the happy path. Generating content with an LLM is trivially easy, which is precisely the problem chapter 6 exists to handle. A dashboard is a weekend.

The genuinely hard parts are invisible in a demo.

Memory hygiene. Keeping an episodic record that's still trustworthy after six months of automated writes. Deduplication, compaction, TTLs, and the audit trail that answers "why did we email this person?" with dates. This is chapter 4, and it's the subsystem everyone skips because skipping it doesn't hurt until month three.

Guardrails. Not writing them. Believing in them early enough. The single choke point, fail-closed checks, the mechanical gate. All cheap to build on day one and expensive to retrofit after your own incident.

Auth lifetimes. See above. It never stops being on the list.

Measurement plumbing. The unglamorous join between actions and outcomes. We ran weeks of disciplined experiments that were unscoreable because the responses table was empty. Nobody demos this layer, and it's the difference between a system that improves and one that just runs.

Here's the tell I'd offer an owner evaluating any vendor or contractor: ask about the boring parts. If the pitch is all content generation and none of it is token refresh, queue discipline, and outcome joins, you're looking at the easy tenth.

When DIY is right

Real conditions, not flattery. You have an engineer, on staff or in the mirror, with actual slack, meaning this isn't competing with revenue work for the same hours. You have a long horizon, because the payoff compounds over quarters, not weeks. You have the temperament for ops, because you're not buying a tool, you're adopting an operation. And ideally the build itself has strategic value to you. For me, the system is also the product demonstration, so every hour did double duty. That's a luxury most businesses don't have, and I'd be lying if I pretended it didn't change my math.

Under those conditions, build. The architecture in this track works, the running costs are almost embarrassing (roughly 35 cents a month in API fees on our stack, by design), and the asset compounds. Everything you own keeps working when you swap the brain.

When it's a trap

You run a $2M–$20M company. You're technical enough to follow this track, which makes you exactly the person most likely to talk yourself into it. So run the owner's math: your evenings for two or three quarters, spent on OAuth scopes and resume cursors, versus spent on the selling and hiring only you can do. The system I've described mostly executes judgment that already exists in your business. Building the executor yourself, at your hourly value, to avoid paying someone whose rate is a fraction of yours, that's the trap, and the spreadsheet catches it in one row.

The failure mode isn't that you can't build it. It's that you half-build it. A system with generation but no guardrails, action but no measurement, memory but no hygiene, isn't a smaller version of this. It's chapter 6 waiting to happen, on your brand, while the unfinished parts age. Track 3 already walked the budget math for buying these outcomes instead. Set it side by side with this chapter's scope and the comparison mostly makes itself.

What about agencies? Most won't run this stack for you, and it's worth understanding why. The standard agency model sells deliverables. Posts, campaigns, reports, produced by people in their building, on their tools. What this track describes is infrastructure that lives in your business: your databases, your tokens, your memory, your guardrails. An agency that builds you that is engineering you out of their retainer, which is why almost none do. The few operators who will build and run owned systems are doing something closer to fractional engineering than marketing services. Ask whoever you talk to the chapter 6 question, "show me the one function every outbound action passes through," and you'll know within a minute which kind you've found.

The honest close

Seven chapters, one architecture: rent the brain, own the body. A scheduler for a heartbeat, episodic memory so the machine knows your customers, a training loop that improves on a schedule instead of by magic, and guardrails that were always going to be the longest chapter. None of it is beyond a good engineer. All of it together is months of disciplined work, and the discipline is the actual product.

If you've read this far and you're building one: genuinely, good luck, and steal anything here that helps. The failures are documented so yours can be different ones. If you've read this far and concluded that this is a lot, that's not a sales tactic, that's just what's true. Either way, you now know what's inside the box. Most people selling you one are counting on you not asking. Looking at these systems honestly, and telling owners which pieces would actually move their needle, is the work I do.

Guardrails before autonomy. Always before.

Series: The Build Track · Post 6 of 7

The chapter with our incident in it: a local model fabricated client case studies and two posts auto-published. The root cause, and the five layers that exist now.

By Ben Blair · 7 min read

One morning my system published two posts containing client case studies that do not exist.

One went to my LinkedIn business page, one to my Facebook page. A local model had generated them. Fake nonprofits, fake clients, fake percentage results, written with total confidence in my voice. Thirty-one more posts like them were sitting in the queue. Five were scheduled for that same day.

The system caught it the next morning. The daily planner read the queue, recognized the fabrications, froze both pages before the day's dispatch window, and wrote up the incident with the fix. Then the alerting layer failed too. The Telegram bot was unreachable from where the planner runs, so the alert that should have paged me never sent. I found out from the morning brief.

That's the honest version. Now the useful part: why it happened, why it was an architecture failure and not a bad prompt, and the layers that exist now. If you're building one of these, this chapter is the one to steal from. If you're evaluating someone else's, this chapter is the interrogation script.

The root cause was boring, which is the lesson

The fabricating model is the surface cause. The real failure was a config file.

Our operating envelopes, the per-platform caps I'll get to in a minute, are defined per account. The accounts in that config still had placeholder ids from initial setup. So when the dispatcher checked each queued post against its envelope, it found no matching account and silently skipped enforcement. Every gate downstream of that lookup quietly waved everything through. The same queue held three LinkedIn posts for a single day against a one-per-day cap, and nothing objected.

Read that failure chain again, because it generalizes. The unsafe content didn't beat the guardrails. The guardrails were configured off and nothing complained about it. A check that silently passes when its config is broken isn't a check. Fail-open is the default behavior of code nobody thought hard about, and fail-open is how two fabricated posts reach a real business page.

There was one more compounding finding, and it stung. While the local model was generating slop, 45 hand-written, voice-scored posts sat in a content library completely unused. The pipeline had been pointed at the wrong source the whole time. The immediate fix was one command that cancelled the generated queue, ingested real library content, and lifted the freezes, plus a standing morning step to cancel anything the generator queued overnight until the generation path itself was fixed or disabled. Interim guardrails are allowed to be a human running a script. They are not allowed to be hope.

So: the layers. Four of them, plus the architectural rule that makes them enforceable.

Layer 1: approval queues. Trust is earned per channel

Tier A policy, written in the envelope spec: 100% of public-posting actions go through an approval queue. A human sees it before the world does.

I think of approval queues as the autonomy throttle, not a permanent state. The progression is full review while a channel proves itself, then sampling, then spot-checks. Any incident resets the channel to full review. The mistake people make is binary thinking: either a human approves everything forever (so why build the machine), or the machine is autonomous on day one (see: the incident above). The queue is how you move between those states deliberately instead of by accident.

The honest cost: queues back up. Ours hit roughly 163 pending items during a stretch when I wasn't reviewing daily, which the system flagged, correctly, as the worst of both worlds. Generation on, review off. If you build a queue, you're committing to working it, or to explicitly lowering generation until you can. The queue's length is itself a metric the system should be watching.

Layer 2: operating envelopes. Caps with two ceilings

Every action type on every platform has an envelope with two numbers: a soft target the system hovers near on a normal day, and a soft max it never exceeds without an explicit waiver. Behind both sits a documented reference to the platform's actual ceiling, so the numbers trace to something.

Concretely, from our config. LinkedIn company posts: soft target 1 per day, soft max 1, because LinkedIn penalizes more than one post per 18 hours. Facebook page posts: target 1, max 3, with a minimum gap between posts. X: target 3, max 8. Around the counts sit shape constraints that keep behavior plausible rather than mechanical. Active hours 7am to 10pm, a silent overnight window, randomized gaps between actions, no duplicate text within 30 days.

The envelope spec also defines automatic freeze conditions. Two consecutive rate-limit responses, or any block screen, freezes the account entirely. The platforms' own immune systems are treated as tripwires, not obstacles.

Two ceilings matter because they separate the tuner from the limiter. Chapter 5's training loop is allowed to move behavior around below the soft max. Nothing automated moves the max. Optimization pressure and safety limits must not live in the same variable, or the optimizer will eventually eat the limit.

Layer 3: the fabrication gate. Never trust a prompt to enforce a rule

Our generation prompt contains a rule in plain language: only use numbers, client names, or results that appear verbatim in the provided source material; never invent a client or an outcome; if you have no facts, write opinion content with zero specific claims.

I want that rule in the prompt. I do not trust it, and neither should you. A prompt is a request. The incident is what a request is worth under pressure.

So there's a mechanical gate that doesn't ask the model anything. After generation, plain regex extracts every specific claim from the output. Percentages, dollar amounts, multipliers like "3x," count claims like "200 leads," client-result sentence patterns like "our client doubled..." Each one gets checked against the exact source text the model was given. Any claim not present in the input is flagged as invented. And the code comment states the policy outright: flagged posts can never auto-queue. They land in draft, flags attached, for human review. It's an absolute override. Even a post that passes every quality, safety, and voice check stays held if it contains one unsubstantiated number.

The principle generalizes past marketing: wherever an LLM's output crosses into the world, put a deterministic checker between them. The checker doesn't need to be smart. Regex against source text is almost insultingly dumb. It would have caught both of those posts, because a fabricated "grew 40%" contains a percentage that appears nowhere in the inputs. Dumb and mechanical beats smart and probabilistic at the boundary, every time.

Layer 4: kill switches. Four scopes, one semantics

When something goes wrong, you need stops at more than one radius. Ours are four flag files on disk:

• KS-1, global: one flag pauses every outbound action the system can take.
• KS-2, per-account: freeze one account. This is what the planner tripped for both business pages.
• KS-3, per-persona: freeze every account belonging to one identity.
• KS-4, network-wide: the everything-in-a-category stop for the experimental tier.

They're checked broadest to narrowest on every single action, and the decision, which switch, which flag file, what reason, is written into the action's log row, so a skipped action is auditable later.

Why flag files instead of database rows or API calls? Because a kill switch must work when everything else is broken. A file on disk can be created over SSH, by a cron job, by me in a panic, with no daemon healthy. The semantics are deliberately simple. File present and non-empty means active. Even clearing one has a fallback: if the file can't be deleted, truncating it to zero bytes counts as cleared. The off switch has its own off switch.

And the docstrings are honest about maturity. Two of the four levels are fully wired end-to-end; the other two are checked on every action but their automatic trip rules ship with a later phase. Guardrails have version numbers too. Pretending otherwise is how you get surprised.

The one choke point, and the question that exposes everyone

None of the four layers means anything without an architectural commitment: every outbound action passes through exactly one function.

In our dispatcher, directly in front of every platform adapter's post call, sits a single gate. It checks the kill switches in order, resolves the account's persona so a persona freeze binds even when the caller didn't pass one, and returns allowed-or-blocked with a named reason. The code's own comment says it plainly: every posting path must pass through this exactly once, at the lowest dispatch level. Not in the planner, where a clever path can route around it. At the last possible moment before the network call. Budget and envelope checks live at the same choke point, and the post-incident rule is that they fail closed. A check that errors blocks the action. It never waves it through.

The placeholder-id failure happened because a check effectively evaluated "account unknown" as "no limits apply." Fail-closed inverts that: unknown account, no action. You will occasionally block something legitimate. That's the correct trade. The alternative is what the incident cost.

So here's the test, and it's the single most useful question I can give you for evaluating any agent product, framework, or contractor's build: "Show me the one function every outbound action passes through." If the answer is a function, with a file and a line number, you're talking to someone with guardrails. If the answer is a diagram, a philosophy, or "each integration handles that," you're talking to someone who hasn't had their incident yet. Everyone gets one. The architecture decides whether it's a story you tell in a blog post or a lawsuit you settle quietly.

Training modules: improving on a schedule, not by magic

Series: The Build Track · Post 5 of 7

Most "self-improving AI" claims are unfalsifiable. Here's a real training loop: telemetry, scored hypotheses, a weekly review, and a tuner with hard caps.

By Ben Blair · 7 min read

"Self-improving AI" is the most common claim in this industry and the least often checked. Ask a vendor how their system improves and you'll get the word "learns" doing a lot of unpaid labor. Learns how? From what data? Measured when? Improved compared to what?

Here's my position: a system improves if and only if you can write a SQL query that shows it improving. Everything else is vibes.

This chapter is the training loop we actually run. Telemetry in, hypotheses out, a weekly review that scores them, and a tuner that adjusts behavior inside hard limits. It's the least magical chapter in this track. That's deliberate. The loop is cron jobs and database joins. It is also the part that makes the whole machine worth owning, because a system that acts without measuring is just automation, and automation that never gets better is a depreciating asset.

The loop, end to end

Five stages, every one of them a scheduled job or a recurring session:

1. Telemetry. Every action the machine takes, every post and every send, gets a row in a telemetry database the moment it executes. Not a log line. A row, with a timestamp and a platform action id, in a table built to be joined against.
2. Responses. A collector runs every evening at 18:30 and fetches engagement on everything we've posted: likes, comments, impressions. Each pull is a row joined back to the original action.
3. Hypotheses. Every daily plan states, in writing, what it's testing. Not "post good content." Something falsifiable: operator-pain content beats case-study content on per-post engagement, same page, same slot, same count.
4. Review. Sunday, the system reviews its week the way a director would. It scores the open hypotheses against the response data and writes down what it believes now and why.
5. Tuning. A nightly tuner proposes envelope adjustments to posting counts and times, based on the evidence. It can move settings within hard caps. It cannot move the caps. That distinction is chapter 6's whole subject.

Notice what's missing: any step called "the model gets smarter." The model doesn't change. The settings change, the content choices change, and the written record of what works grows. The brain is rented, remember. What improves is the body and the playbook. The parts we own.

The embarrassing part first

For weeks, our responses table was empty.

I want to sit on that for a second, because it's the most common failure in this whole category and almost nobody admits it. We had telemetry on every action. We had daily plans with named hypotheses. We had a Sunday review. And the join that connects what we did to what happened didn't exist yet. The machine was running experiments and never collecting the results. Write-only science.

The daily planning log from that stretch says it plainly: "every lever experiment we run is write-only until outcomes join back." The plan for each day was disciplined. And unscoreable.

The fix was a response collector, and the design decisions in it are the generic playbook, so here they are.

Reuse the posting path for measurement. The same authenticated adapters that publish posts can fetch engagement on them. One auth per account per run, joined on the platform's action id, which the dispatcher logs on every executed post. Don't build a second integration to measure the first one.

Get longitudinal data from cadence, not complexity. The collector runs daily with a 20-hour dedupe gap, which automatically yields roughly 24, 48, and 72-hour engagement curves per post, out to a 14-day horizon. No special scheduling logic. Just a daily job and arithmetic.

Absence is signal. If the collector can't find a post it knows was published, it doesn't stay silent. It writes a row recording zero visibility and the failure reason. A post that disappeared is the platform telling you something. Silence in your own pipeline is the one thing you can't afford.

The first real collection run is worth reporting honestly, because this is what day one of measurement actually looks like. Facebook posts measured fine: older machine-generated posts sitting at 9 to 10 likes each with zero comments, and a newer hand-voiced post at 0 likes at the 15-hour mark, which is not yet a comparison because the ages don't match. X: 5 impressions, zero engagement, on a cold account. Expected. LinkedIn: both fetches failed outright, and the first version of the collector swallowed the error code, so we patched it the same day to record the HTTP status instead. Day one of measurement is mostly discovering what your measurement can't see yet.

That's the real texture of a training loop. Not a dashboard going up and to the right. A join that finally returns rows, half of them telling you about your own bugs.

Hypotheses you can score

The discipline that makes the loop work is older than AI: change one variable at a time.

Here is an actual day's experiment design from our planner, lightly compressed. Three pages, one post each. On LinkedIn, the theme changes from yesterday, operator-pain content instead of case-study content, while the slot and count stay fixed, so theme is the isolated variable. On Facebook, same midday slot as yesterday, theme rotated. On X, the content stays in the same class but the slot moves from midday to evening. The first deliberate point on that account's timing curve. Each hypothesis gets a name: H-0612a, H-0612b, H-0612c. Each pairs against a prior day's named hypothesis. All of them sit open until the response data is old enough to score.

The plan also held posting volume at one per page per day, at or below the soft targets, with the reasoning written down: with no outcome data yet, "volume buys risk, not learning."

One more honest detail. Before we had our own data, the system's "best posting times" were just platform priors. Generic recommendations, identical repeated values, the kind any blog post would give you. The machine knew nothing it hadn't been told. The entire point of the loop is to replace those priors with observed signal, one named hypothesis at a time. Any vendor whose system "knows the best time to post" on day one is showing you priors with a confident font.

The Sunday review and the tuner

Two different jobs close the loop, and the separation matters.

The director review is judgment. Once a week, the brain reads the week's telemetry, scores the open hypotheses, and writes conclusions in prose. What we now believe about themes, timing, and platforms, and what to test next. This is part of the roughly ninety minutes a day of actual reasoning I described in chapter 1, batched into its most valuable session.

The envelope tuner is mechanism. It runs on schedule, reads the same data, and proposes parameter changes within hard caps it cannot touch. Its early runs proposed nothing, and reported why: not enough days of data yet across the platforms. A tuner that says "no changes, insufficient evidence" is working correctly. A tuner that always finds something to change is fitting noise.

Judgment proposes strategy. Mechanism adjusts dials. Neither one gets to move the safety limits. If you collapse those three into one component, and most agent frameworks do, you can't audit which one made any given decision.

Decision journals are the actual training corpus

Every planning session writes a log: what was posted, why, what hypothesis it serves, and a lever-settings record covering counts, times, content sources, and budget, in a fixed format. Every review writes its scores and conclusions. These files pile up into the most valuable dataset we own, and it isn't engagement numbers. It's decisions with stated reasoning, which means any future brain, a new model, a better model, a model that doesn't exist yet, can read six months of judgment calls and inherit them.

One rule makes the corpus reusable: keep generic decision rules separate from business facts. "Isolate one variable per page per day" is a rule. It transfers to any client, any business. "This account's audience skews older and slower" is a fact about one account. We write them in separable form on purpose, because the rules half of the corpus is a product. It's the playbook a second deployment starts from. Mix them and you've got a diary. Separate them and you've got a methodology with local annotations.

What falsifiable looks like

So, the test I promised. A self-improvement claim is falsifiable when you can express it as a query: actions joined to responses, grouped by the variable you claimed to be testing, filtered by date range, compared across the change. Posts with theme X averaged this engagement at 48 hours; posts with theme Y at the same slot averaged that. If the system can't produce that table, it isn't learning. It's drifting, and someone is narrating the drift as progress.

That's also the question to ask any vendor selling adaptive anything: show me the join. Show me the table of actions, the table of outcomes, the key that connects them, and one decision the system made differently because of it. We can show ours, including the weeks the outcomes table was empty. The empty weeks are part of the receipt.

The loop only earns its keep if the machine acting on it can't hurt you while it learns. Which is why the next chapter exists, and why it's the one with our worst day in it.