System ESM Reference Build 03

An AI lead response system, and the seven bugs I found before any client did

It catches a lead, calls them in seconds, qualifies them in conversation, books the appointment, and tells the owner. The hard part was never building it. The hard part was leaving it running.

Reference build. I built this as my agency's flagship offer and ran it on my own site first, on real infrastructure against real provider accounts. It is not paid client work, and no client has run it yet.

Retired 2026-09-01. It ran unattended for six weeks, then the subscription behind its host ended. Everything below happened; none of it is running today. The workflow definitions, the execution analysis and the incident write-ups are in the repo and still checkable.

16workflows live
7production bugs found
338probes audited
6 wksunattended, then retired

The problem

Whoever answers first usually wins

A lead fills in a form at 9pm. Most small businesses answer the next morning. The gap between those two facts is the entire business case.

So the system is obvious. Catch the lead, call them immediately, qualify them, book the appointment, update the records, tell the owner. Every part of that is a solved problem in isolation.

The part that isn't solved is leaving it running.

What I built

Sixteen workflows, and a larger operations layer than feature layer

A form submission hits a webhook, gets validated and deduplicated, and is written to the CRM. A provider router places an outbound AI voice call. The assistant qualifies the lead in conversation, scores them, and books a real calendar slot mid-call through tool calling. The lead row is updated, the transcript and an AI summary are stored, and the owner gets a notification.

FORM SUBMISSIONwebhook, validate, dedupe
SAVE TO CRME.164 phone as the row key
VOICE PROVIDER ROUTERreads config, one-shot fallback
AI QUALIFIES THE LEADin conversation, scored
BOOK THE SLOT MID-CALLlive calendar tool call
UPDATE ROW · STORE TRANSCRIPT · NOTIFY OWNER

deterministic workflow the judgement step

Around that sits the part that took longer: an error handler catching every workflow failure, an hourly health check probing each dependency, a nightly backup, daily and weekly digests, and a runbook that opens with a 2am decision tree.

Everything client-specific lives in a config file. The workflows read configuration and never hardcode a phone number, a calendar ID, or a company name. Voice providers go through an abstraction, so swapping one for another is a config change rather than a workflow change.

The hard part

Knowing whether an unattended system is actually working

A system like this fails in ways that look exactly like success, and it does it while you are asleep. Five of my seven production bugs were completely silent. Only two announced themselves.

None of those produce a red X anywhere. You find them by looking, or you find them from a customer.

The one that matters

The alarm that could fail silently

Two weeks in I was being paged for "Google Sheets CRM: Service unavailable." Seven times.

Before changing anything, I pulled the actual execution history out of the live database. 338 hourly probes over two weeks:

0Vapi failures
0Calendar failures
7Sheets failures (2.1%)
0consecutive

Not one of the seven was consecutive. Every single one self-healed on the next probe an hour later. There was no outage. My alerting was converting ordinary Google API jitter into seven pages.

Then the real find. The Sheets node had retry configured, three tries ten seconds apart. It had never once fired. retryOnFail is inert on any n8n node that has continueOnFail set, because those nodes handle the failure internally and push an error object rather than throwing, and the executor's retry wrapper only ever sees thrown errors. I confirmed it against the n8n source installed on the server rather than trusting the documentation.

The proof was arithmetic. One failed execution completed in 720 milliseconds. Three tries with ten second waits cannot finish in under twenty seconds. The retry was provably never running.

Then I found the worse one. My alert sender also had continueOnFail set. A failed alert was swallowed and the execution reported success. A monitoring system whose alarm could fail without telling anyone, while reporting green.

That is the failure mode that should scare anyone running unattended automation, and I only found it because I went looking for a different bug.

The fix, and how I knew it worked. Page only after two consecutive failures per dependency, re-page every six probes while still down, send an explicit recovery notice, and roll self-healing blips into a daily heartbeat instead of paging. The alert sender is now allowed to throw, so a genuinely undeliverable alert becomes a visible failed execution.

I replayed the new policy against the real 338 run history before deploying. Seven pages become zero, while an injected five-consecutive-failure outage still pages and still recovers.

Evidence

What you can check

ClaimArtifact
It ran live on real infrastructure16 workflows deployed, end to end since 2026-07-17
It places real callsReal outbound calls verified 2026-07-23 against the provider API, not just the workflow's own status
It is operable by someone elseA runbook opening with a 2am decision tree, plus 15 documentation files
It is deployable per clientConfig-driven workflows, a client template, a one-command deploy script
The monitoring is realHourly health checks, error handler, nightly backup, dedicated alert bot
The failures are documentedSeven production incidents, each with a root cause and a fix
The monitoring was audited against itself338 probes measured, two bugs found in the alerting layer, fix validated by replay before deploy

The last two rows are the point. Sixteen workflows is a weekend of work for anyone competent. Finding a bug in your own alarm is the thing that separates a demo from a system somebody can depend on.

Stack

A workflow engine, deliberately

n8n, not custom code. This is plumbing: webhooks, third-party API calls, retries, queues, scheduled probes, notification fan-out. That is exactly what a workflow engine is for, and writing it as a bespoke service would have meant hand-rolling the execution history and retry semantics that made the alerting bug findable in the first place.

Where I would not use it. If the core of the job were judgement over messy input, or writing prose, n8n would be the wrong tool and I would build an agent instead. My rule: deterministic where the input is structured and the decision is a rule you could write down, agentic where the input is messy and the output is a judgement. Lead routing is the first kind. The one genuinely agentic step, the qualification conversation, is delegated to a voice provider and comes back as structured data.

Self-hosted, on the client's own infrastructure. No hosting margin, no lock-in, and the client keeps their own data. It also means the operations discipline above is not optional, because there is no vendor dashboard to hide behind.

A provider abstraction from day one. Voice calls go through a router that reads config, so swapping providers is a config change. That was speculative when I built it and I would do it again, because voice pricing and quality move fast enough that being locked to one vendor is a real risk.

Cost

One VPS, and metered voice minutes

A single small server running the workflow engine and its database, free-tier Google APIs, and metered voice provider usage. Voice is the only meaningful variable cost and it scales with actual call volume.

I have not measured a real per-lead cost, because the lead volume has not been there. So I am not going to quote one.

For a real client

What I'd do differently

The limits

What this does not prove

The important one first.

  • This has processed almost no real leads.My own site is new and the leads sheet is approximately one row. The system is proven mechanically, end to end, with real calls and real bookings. It is not proven commercially. I have no conversion data, no close rate, no cost per booked appointment, and I am not going to imply otherwise.
  • Not paid client work. Nobody has run this on their business yet.
  • Eight of ten milestones done. CRM sync is partial, Google Sheets works and external CRMs are not started. Packaging is in progress.
  • The second voice provider adapter is written but untested. Only one provider has placed a real call.
  • A dedicated phone number was deliberately deferred, so the telephony surface is the provider's default rather than a client's own line.
  • Confirmation email and the retry workflows are not built.

What it does prove: I can build a multi-integration automation, deploy it to real infrastructure, operate it unattended, detect failures that do not announce themselves, root cause them against live data rather than guesses, fix them, verify the fix, and write it down so the next person can do it at 2am. Seven times.

The lead response product is one shape that capability takes. The capability is the thing worth hiring.