Skip to main content
AI Learning Resources

Quick Overview

This video is an AI development benchmark and review hosted by Nate Herk. In response to the release of Opus 5.5, Nate conducts an experiment comparing how the model performs across each of its effort settings when tasked with building the same complex web application.

Key Points

  • 1.Anthropic's Opus 5.5 model was tested across six effort levels ranging from Low to Ultracode using the exact same prompt to build a walkable 3D virtual conference world.
  • 2.Higher effort levels consistently consumed more tokens and cost more in API billing, scaling from $3.91 at Low effort to $50.38 at Max effort.
  • 3.Across all six runs, only the High effort level asked a clarifying question, while no runs used subagents during execution.
  • 4.The Extra effort tier delivered the most functional and interactive 3D environment, featuring interactive breakout rooms, working video playback, and seat selection.
  • 5.Max effort suffered from significant visual bugs, poor character movement, and non-interactive sessions despite running for nearly two and a half hours.

Summary

The video opens with Nate Herk introducing Opus 5.5 and setting up an experiment to evaluate how effort settings impact the final quality and cost of an AI-generated web application. Using the Claude Code environment, Nate submits the exact same prompt across six effort tiers: Low, Medium, High, Extra, Max, and Ultracode. The prompt directs the model to build a walkable third-person 3D tech conference application containing main stages, workshop tracks, an expo hall, and VIP lounges, using 105 gigabytes of video and asset recordings from the AIS Live virtual event.

Testing begins with the Low effort setting, which completed in 16 minutes and 43 seconds, consuming 191.3K tokens and costing $3.91 across 22 verification checks. While the space featured basic room geometry and an agenda board, it suffered from harsh lighting, static workshop screenshots instead of videos, and attendee avatars that vanished when approached. Nate then loads the Medium effort build, which took 1 hour and 13 minutes and cost $12.44 with 419.2K tokens. This version upgraded the lobby branding, introduced live video playback on workshop screens, added animated attendee avatars that wave, and built an interactive upstairs VIP lounge with working video feeds.

Moving to the High effort setting, the generation concluded in 1 hour and 7 minutes at a cost of $16.31 across 509.3K tokens and 22 checks. This build introduced an outdoor entrance plaza with speaker banners, automatic sliding doors, closed captions synchronized to main stage video playback, and an interactive VIP afterparty room with a DJ booth. Notably, High effort was the only run across all six tests that prompted Nate with a clarifying question during execution.

Nate then inspects the Extra effort level, which took 1 hour and 31 minutes, used 733.7K tokens, and cost $25.92 with 34 checks. Extra proved to be the most functional environment, providing an interactive attendee chat system, workable VIP pricing worksheets at conference tables, audience seating mechanics, and fully functioning presentation stages. When reviewing Max effort, Nate observes severe degradation despite it taking 2 hours and 28 minutes, consuming 1.18 million tokens, and costing $50.38 across 51 checks. Max featured awkward walking animations, rendering glitches when moving between hallways, broken stage video streams, and non-interactive seating.

The final test examines Ultracode, an extra-effort configuration that completed in 1 hour and 35 minutes for $18.69 across 606.2K tokens and 42 checks. Ultracode incorporated unique features like a playable basketball mini-game, a downloadable photo wall, and browsable PDF-style speaker slide decks, though it felt less cohesive than Extra. Reviewing the overall benchmark data, Nate points out that Max cost 12.9 times more than Low, but Extra delivered the best balance of stability, depth, and visual polish.

Experiment Setup and Prompt Design

Nate Herk tests the capabilities of Opus 5.5 across all available effort settings, which include Low, Medium, High, Extra, Max, and Ultracode. Each run receives identical instructions via a markdown prompt to construct a 3D walkable tech conference application based on 105 gigabytes of event assets from AIS Live. The evaluation criteria focus on visual design, physics implementation, stage presentations, interactivity, and runtime metrics.

Low to High Effort Comparison

The Low effort build completed in 16 minutes and 43 seconds for $3.91, producing a basic environment with static workshop slides and disappearing attendee avatars. Medium effort took 1 hour and 13 minutes, costing $12.44, and introduced live video feeds and attendee interactions. High effort finished in 1 hour and 7 minutes at $16.31, adding synced closed captions and speaker banner signage, while being the only setting to ask a question during development.

Extra, Max, and Ultracode Evaluations

Extra effort required 1 hour and 31 minutes and $25.92, creating the most complete venue with functioning worksheets, audience seating, and interactive breakout rooms. In contrast, Max effort took 2 hours and 28 minutes, cost $50.38, and resulted in broken walking physics, severe rendering glitches, and broken video players. Ultracode finished in 1 hour and 35 minutes for $18.69, adding photo wall downloads and slide deck viewers, but felt buggy compared to Extra.

The Bottom Line

The experiment demonstrates that increasing AI effort parameters does not guarantee linear improvements in application quality, as Max effort suffered from notable physics and rendering bugs despite generating the highest token and financial costs. The Extra effort setting emerged as the clear winner, delivering the most polished interactive venue at roughly half the cost and runtime of Max. The test establishes practical benchmarks for complex agentic web development tasks while leaving open whether model harness updates can resolve Ultracode's dynamic workflow inconsistencies.

FAQ

What is Opus 5.5 and what are Opus 5.5 effort levels used for?

Opus 5.5 is an AI model that supports selectable reasoning effort levels, such as Low, Medium, High, Extra, Max, and Ultracode. These effort levels determine the computational depth, reasoning time, verification checks, and token consumption the model applies to complex development tasks.

Which effort level produced the best overall 3D virtual conference application output?

The Extra effort level produced the best result, offering interactive chat, working VIP session worksheets, audience seating mechanics, and functional video playback without the severe bugs found in higher settings.

How did the cost and runtime scale between the Low and Max effort settings?

Low effort took 16 minutes and 43 seconds and cost $3.91, while Max effort took 2 hours and 28 minutes and cost $50.38, representing a 12.9 times increase in cost.

Did any of the Opus 5.5 effort runs use subagents during the builds?

None of the six effort runs spawned or used subagents, choosing instead to handle the entire build process within a single agent session.

Why did the Max effort build fail to outperform the Extra effort build?

The Max effort build suffered from visual glitching through walls, broken character walking animations, non-functioning stage videos, and disabled seating interactions despite running longer and consuming more tokens.

Worth watching for

Developers and AI engineers interested in evaluating how LLM effort parameters affect coding agent output quality, token usage, and API costs.

  • opus-5-5
  • claude
  • ai-coding
  • benchmarking
  • web-development
Watch on YouTubeMore videos from Nate Herk