Quick Overview
This panel discussion from OpenAI DevDay features Developer Experience lead Aarti Bagul alongside Applied AI engineers Mandeep Singh and Saptarshi Banerjee. The speakers address common missteps engineering teams make when managing LLM expenses in production environments. The talk examines technical levers and architectural patterns designed to maximize accuracy while minimizing API costs.
Key Points
- 1.Developers should optimize for cost per completed task rather than the list price of input and output tokens.
- 2.Small models like Luna paired with high reasoning effort can match or exceed the performance of larger previous-generation models at lower cost.
- 3.Programmatic tool calling allows external code to filter tool execution results, avoiding unnecessary token transmission back to the language model.
- 4.Prompt caching reduces cached input token costs by up to 90 percent and supports explicit break points in newer model families without cache invalidation.
- 5.The Batch API provides a 50 percent discount on token pricing for asynchronous workloads that can tolerate a 24-hour completion window.
- 6.Systematic evaluation sets and Pareto curves mapping accuracy against cost are essential for selecting the optimal model and configuration.
Summary
Aarti Bagul opens the panel at OpenAI DevDay alongside Applied AI engineers Mandeep Singh and Saptarshi Banerjee to examine methods for reducing AI spending. Singh explains that focusing solely on token list prices is misleading because lower-cost models often require more tokens to finish a prompt or fail outright, demanding costly human intervention. Using a personal experience with an airline chatbot that failed to rebook a flight, Singh illustrates why cost per task is the proper operational metric. He notes that customers like Perplexity achieved 9 percent higher accuracy at half the cost using 6-Astra compared to previous systems, while Notion reduced task costs by half after migrating from 5.5 to 5.6-Sol.
Singh outlines a framework for right-sizing models by defining task requirements and target accuracy thresholds before testing configurations along a Pareto curve of accuracy versus cost. While complex reasoning may warrant 6-Astra, simpler routing or extraction tasks can rely on Luna. Singh highlights Luna-maxing, a technique where running the smaller Luna model with high or extra-high reasoning effort allows it to outperform larger models from previous generations across 80 to 90 percent of developer workloads. Bagul also mentions the Decisions API for simpler decision-making workflows.
Banerjee presents four architectural levers outside model selection that control cost while maintaining quality: prompt caching, programmatic tool calling, reasoning effort calibration, and asynchronous batch processing. Prompt caching stores repetitive prefix text so shared context does not require redundant pre-processing. Programmatic tool calling runs small coordinator scripts to filter and aggregate multiple tool results before returning data to the language model, eliminating intermediate context calls. As demonstrated by legal tech company Clio, programmatic tool calling decreased prompt tokens by 38 percent in multi-step document analysis with zero quality loss.
Banerjee explains that reasoning effort functions as an adjustable parameter that developers should tune downward for routine tasks and increase only for complex problems, validating results through automated evaluations. He notes that OpenAI Batch API offers a 50 percent token discount for non-urgent requests processed within a 24-hour window. Addressing prompt caching optimization, Banerjee describes how developers previously relied on automatic token intervals but can now set explicit break points and explicit-only caching in GPT-5.6 and 6-Astra models. Software development firm Blitzy applied these caching techniques alongside Luna tool loops to increase cache reuse from 24 percent to 90 percent, reducing costs by 87 percent compared to GPT-5.4-mini.
The session concludes with Bagul highlighting developer observability tools, including the prompt caching admin dashboard and a diagnostics API that pinpoints cache misses. In their final advice, Singh and Banerjee urge engineering teams to construct rigorous evaluation datasets, adjust one variable at a time, and iteratively map solutions against the Pareto frontier to balance cost, latency, and task accuracy.
Measuring Cost per Task over Token Price
The panel explains that evaluating models purely on token pricing creates false economies. If a cheaper model uses excessive tokens or fails and requires human intervention, the overall task cost escalates. Mandeep Singh shares an example of an airline support bot failing to rebook a flight, illustrating how human handoffs inflate actual expenses. Benchmarks from Perplexity and Notion demonstrate that upgrading to more capable models like 6-Astra and 5.6-Sol cut task costs in half while improving overall accuracy.
Right-Sizing Models and the Pareto Frontier
Selecting the right model begins with defining the exact task and the required accuracy threshold. Engineering teams should plot candidate configurations along a Pareto curve balancing accuracy and cost. While complex orchestration suits frontier models like 6-Astra, simpler tasks benefit from smaller models such as Luna. Singh notes the practice of Luna-maxing, where smaller models with high reasoning effort can outperform older large models across 80 to 90 percent of practical developer workflows.
Architectural Levers for Cost Reduction
Saptarshi Banerjee outlines four engineering levers that reduce costs beyond basic model selection. Prompt caching reuses pre-processed context across requests, programmatic tool calling processes intermediate data in code before returning summaries to the model, calibrated reasoning effort limits unnecessary thinking tokens on routine queries, and the Batch API cuts token rates by 50 percent for requests with 24-hour turnaround times. Legal platform Clio achieved a 38 percent drop in prompt tokens by implementing programmatic tool calls.
Advanced Caching Strategies and Developer Tools
Developers can maximize prompt cache hit rates by stabilizing prompt prefixes and applying explicit break points available in GPT-5.6 and 6-Astra models. Software company Blitzy increased cache reuse from 24 percent to 90 percent and reduced costs by 87 percent versus GPT-5.4-mini by adopting Luna tool-calling loops. The panel also highlights the admin prompt caching dashboard and diagnostics API, which help developers track cache hit rates and identify exact points where cache misses happen.
The Bottom Line
The presentation establishes that AI cost efficiency depends on tracking cost per completed task and applying systematic architectural levers rather than simply choosing low-cost models. It outlines concrete technical strategies, including explicit prompt cache breakpoints, programmatic tool filtering, and calibrated reasoning effort. The session leaves open the exact financial thresholds and break-even points for individual corporate architectures, directing builders to evaluate their specific workloads with automated evaluation sets.
FAQ
What is cost per task and why is cost per task preferred over cost per token?
Cost per task measures the total financial expenditure required to successfully resolve an entire user objective, including tokens consumed across retries and the operational cost of human intervention. It is preferred over cost per token because cheaper models may fail to complete the task or generate excessive tokens, resulting in a higher total expenditure.
What is Luna-maxing in the context of OpenAI model right-sizing?
Luna-maxing refers to deploying the smaller, lower-cost Luna model configured with a high or extra-high reasoning effort parameter. This combination allows the model to outperform larger previous-generation models on complex reasoning while remaining significantly cheaper per task.
How does programmatic tool calling reduce prompt token consumption in AI applications?
Programmatic tool calling executes a local coordination script to run tools, aggregate responses, and filter raw data before sending results back to the language model. This prevents bulky intermediate data from entering the context window and consuming unnecessary prompt tokens.
How do explicit break points improve prompt caching in the OpenAI API?
Explicit break points let developers define exact boundaries where repetitive prompt content ends and variable input begins. This guarantees that static system instructions and tools remain cached even if user queries change frequently.
What discount does the OpenAI Batch API provide for non-urgent processing workloads?
The Batch API offers a 50 percent discount on token pricing compared to standard synchronous requests, provided the workload can accommodate a 24-hour completion window.
Worth watching for
Software engineers, system architects, and technical leaders building LLM-powered applications who need to lower inference costs without degrading output quality.
- openai
- prompt-caching
- cost-optimization
- api
- devday
- model-evaluation