Agents as scaffolding for recurring tasks.
The author describes a pattern of prototyping workflows with AI agents then refactoring into code-driven processes, using agents only for tasks that require human-like judgment. A security vulnerability alert system illustrates how webhooks filter and route high-priority issues, delegating owner identification to an agent and formatting alerts via a second agent for reliable Slack notifications.
- Prototyping a workflow fully inside an agent prompt (filtering, assigning, formatting all in one) broke down under load, letting high/medium severity alerts slip through undetected.
- The fix was refactoring deterministic steps (filtering, batching, data extraction) into plain code, reserving agents only for tasks needing human-like judgment, like matching people to code ownership.
- The resulting hybrid pipeline (code filters/batches → agent assigns owners via CODEOWNERS/commits → second agent formats Slack messages) runs reliably without spamming non-critical alerts.
- The author's general pattern—prototype in pure agent, then refactor toward code with only a few targeted agent calls—is framed as producing faster, cheaper, more reliable automation than prompt tinkering or building strict evals.
I spent months turning our security‐patch workflow into a mix of code and AI agents. Every time Dependabot flags a vulnerability, a webhook launches our internal framework. A script first weeds out low‐priority or resolved issues, then bundles each remaining alert by repository. Those bundles go to an LLM that uses our “ownership” skill plus GitHub’s MCP logic—checking CODEOWNERS files, recent commits, and so on—to pick the right teams or individuals. A second agent then formats and posts a clean Slack notification.
At first I tried doing everything inside the agent prompt: filter by critical severity, assign owners, format messages. It broke down as soon as I pushed it harder—high‐ and medium‐severity alerts slipped through. Instead of endless prompt tinkering or building a hard eval to block wrong messages, I shifted most of the workflow back into plain code. Now software handles the deterministic steps—filtering, batching, data extraction—and only cedes control when human-style judgment is needed, like matching people to code areas.
That hybrid approach works flawlessly. We rolled out real-time critical‐issue alerts without worrying the bot will spam non-critical items. Next up is a weekly summary ping for open critical issues and eventually auto-generating the fixes themselves, leaving only a quick human review before deployment. Over time I’ve found this scaffold—prototype in pure agent, then refactor toward code with just a few agent calls—delivers faster, cheaper, more reliable automation.