Converting raw source material — plain text, PDFs, Word documents — into polished, customer-facing documentation is repetitive skilled work. An LLM can do it well, but real-world inputs (60,000+ character PDFs) produce generations lasting 6–7 minutes: far past any serverless request timeout, so a naive implementation drops the connection mid-generation.
- Started with synchronous streaming; measured real workloads and found output length, not model choice, drives generation time — and no serverless tier survives the worst case
- Re-architected as an async job system: the serverless front end extracts content and images, writes a job row to Postgres, and hands off to a worker
- Ran generation on a long-lived container worker with no request timeout, writing progress and results back to the job store
- Had the browser poll a lightweight status endpoint — the user watches progress instead of a hanging request
Large documents that previously failed at the platform timeout now complete reliably, with live progress. The redesign cost nothing in the common case and made the worst case routine.
- Next.js · serverless front end
- Container worker (long-lived, no timeout)
- Supabase — Postgres job store with RLS
- Anthropic API
The architecture decision
Three options were on the table: stay serverless and cap input size, move to edge functions, or add a container worker. Measuring the real worst case — 400+ seconds of generation — eliminated the first two immediately: both platforms enforce wall-clock limits the workload simply exceeds. The container worker was the only design that survives the data, not the demo.
What this generalises to
Any LLM product with unbounded output length eventually meets this wall. "Be concise" prompting does not reliably bound output; model choice barely moves throughput. The durable answer is architectural: decouple the request from the generation, persist the job, and let the client poll. This pattern has since become my default for long-running AI work.