Quick Overview
In this video, Nate Herk conducts a head-to-head build test comparing two versions of an autonomous software orchestrator, Fable 5.1 and Fable 5. He assigns both models an identical specification to build an incident-response workflow studio called OpsFlow to evaluate differences in UI design, runtime, and cost.
Key Points
- 1.Nate Herk tested Fable 5.1 and Fable 5 by giving both AI orchestrators an identical prompt to build an incident response application called OpsFlow.
- 2.Fable 5.1 generated an application that cost $1,220.24 and required 36 hours and 12 minutes of API time, relying heavily on Claude Opus for 57 percent of its workload.
- 3.Fable 5 completed its application build for $519.64 in 12 hours and 51 minutes of API time, delegating 80 percent of its tasks to Claude Sonnet.
- 4.In a blind evaluation conducted across six categories by Codex, Fable 5.1 achieved a weighted score of 9.1 out of 10, compared to 8.4 out of 10 for Fable 5.
- 5.The two generated applications implemented different visual layouts and incompatible JSON workflow schemas, with Fable 5 arranging nodes vertically and Fable 5.1 arranging them horizontally.
- 6.Despite Fable 5.1 scoring higher in visual design and dry run execution, Herk concluded that Fable 5 delivered better value given its substantially lower cost and runtime.
Summary
Nate Herk sets up an experiment comparing two versions of an AI orchestrator, Fable 5.1 and Fable 5, by tasking them with building the exact same web application from scratch. Both agents received an identical goal prompt directing them to act as lead orchestrators accountable for architecture, delegation, and final quality without directly writing code. The target application, named OpsFlow, was specified as a professional three-panel visual studio designed for modeling and simulating incident-response workflows, complete with custom triggers, conditions, human approvals, and execution timelines.
Upon reviewing the completed builds, Herk demonstrates how differently each agent interpreted the architecture. Fable 5 built a vertically aligned workflow editor on localhost port 4382. It features deep authoring controls, regex support, and extensive condition operators, but suffers from text scaling issues and cluttered card styling when zoomed out. Fable 5.1 created a horizontal canvas running on localhost port 5321. Its interface provides superior typography, clearer execution states, editable JSON payloads, and native light and dark modes. When testing file interchangeability, Herk discovers that the two agents structured their JSON workflow schemas differently, preventing workflows exported from one application from loading into the other.
To assess functional quality objectively, Herk had Codex perform a blind evaluation across six weighted categories. Fable 5.1 won five of the six categories, achieving a weighted overall score of 9.1 out of 10. It earned top marks for visual hierarchy (9.1), first-run ease of use (9.2), dry-run experience (9.3), safety validation (9.3), and accessibility (8.8). Fable 5 scored an overall 8.4 out of 10, winning only the workflow authoring category with a 9.1 compared to 5.1's 8.7, driven by its richer condition toolset and live preview features.
A substantial divide appears in the cost and execution profiles of the two builds. Fable 5.1 ran for 36 hours and 12 minutes of API time, consumed over 404,000 context tokens, and racked up a bill of $1,220.24 by routing 57 percent of tasks to Claude Opus and 40 percent to Sonnet. Fable 5 operated for 12 hours and 51 minutes, used 260,000 context tokens, and cost $519.64 while directing 80 percent of execution to Sonnet. Herk concludes that while Fable 5.1 produced a slightly more refined application, Fable 5 delivered far better economic value for autonomous project development.
Experiment Setup and Prompting Strategy
Nate Herk configured Fable 5.1 and Fable 5 with identical goal prompts inside their respective project directories. The prompt instructed each model to act exclusively as a lead orchestrator responsible for planning, delegation, sequencing, and quality standards, rather than writing code directly. Both agents were instructed to build OpsFlow, a local-first visual automation studio for designing and simulating incident-response workflows with node editing, dry runs, and local persistence.
Comparing the Generated Applications
Both versions produced functional interactive web applications running on local ports, but they made distinct design choices. Fable 5 produced a vertically oriented node layout with detailed authoring controls, regex condition operators, and embedded visual maps. Fable 5.1 constructed a horizontal workflow interface featuring cleaner card layouts, interactive payload inspectors, explicit approval steps, and downloadable run logs. The two applications also generated incompatible JSON schemas for importing and exporting workflow configurations.
Codex Benchmark and Scoring Breakdown
Herk used Codex to conduct an automated blind evaluation across six weighted criteria without disclosing which agent built which app. Fable 5.1 earned higher marks in visual design, first-run ease of use, dry-run simulation, validation safety, and accessibility, totaling a weighted score of 9.1 out of 10. Fable 5 scored 8.4 out of 10, outperforming Fable 5.1 only in the workflow authoring category due to its deeper condition operators and autorun tools.
Cost, Runtime, and Model Delegation Breakdown
The operational metrics revealed stark differences in efficiency between the two orchestrators. Fable 5.1 accumulated $1,220.24 across 36 hours and 12 minutes of API runtime, allocating 57 percent of its compute to Opus and 40 percent to Sonnet. Fable 5 finished in 12 hours and 51 minutes at a total cost of $519.64, routing 80 percent of its workload to Sonnet and only 12 percent to Opus. While Fable 5.1 produced a more polished user experience, Herk judged Fable 5 the practical winner because the quality difference did not justify an extra $700 in API expenditure.
The Bottom Line
The video establishes that while Fable 5.1 generates higher-scoring user interfaces and simulation flows through heavier reliance on Opus, it does so at more than double the financial and runtime cost of Fable 5. Fable 5 demonstrates effective task routing by using Sonnet for the vast majority of implementation work, producing comparable functionality at a fraction of the expense. The comparison leaves open how future prompt tuning or model routing rules might prevent newer orchestrators from overusing expensive reasoning models on standard development tasks.
FAQ
What is OpsFlow and what features did Fable build for it in the experiment?
OpsFlow is a local-first visual automation studio designed for building, editing, and simulating incident-response workflows. The application includes a node palette with triggers, conditions, actions, approvals, and resolutions, as well as an execution timeline and JSON import and export tools.
How did Nate Herk structure the goal prompt given to both Fable agents?
The prompt designated each agent as the lead orchestrator responsible for strategy, planning, delegation, and final quality review. It explicitly prohibited the orchestrator from writing code directly, instructing it to delegate architecture and review tasks to Opus and implementation, testing, and debugging tasks to Sonnet.
How did the total cost and API runtime compare between Fable 5.1 and Fable 5?
Fable 5.1 cost $1,220.24 across 36 hours and 12 minutes of API time. In contrast, Fable 5 cost $519.64 and finished in 12 hours and 51 minutes of API time.
What were the results of the blind evaluation conducted by Codex between both applications?
Codex awarded Fable 5.1 a weighted score of 9.1 out of 10 and Fable 5 a score of 8.4 out of 10. Fable 5.1 won in visual design, first-run usability, dry-run simulation, safety validation, and accessibility, while Fable 5 won in workflow authoring.
Why did Fable 5 cost significantly less to run than Fable 5.1 during the build?
Fable 5 delegated 80 percent of its execution workload to Claude Sonnet and only 12 percent to Opus. Fable 5.1 routed 57 percent of its total work to Opus and 40 percent to Sonnet, dramatically increasing token costs.
Worth watching for
Software engineers, AI agent developers, and engineering managers evaluating the practical costs, autonomous delegation patterns, and build quality of multi-agent software engineering systems.
- fable
- ai-agents
- software-engineering
- benchmarking
- workflow-automation
- opsflow