Source details
- Original source
- Towards AI
- Published
- 2026-09-25
- Primary topic
- AI Policy
Why it matters
Regulation, copyright, courts, government action, and policy developments affecting AI deployment. Use the original source for the full report, then use the directory shortcuts below to compare the products and workflows the story points toward.
What happened
Last Updated on September 25, 2026 by Editorial Team Author(s): Ethan Mark Originally published on Towards AI. OpenAI Prompt Cache Diagnostics: Find the Prefix Drift Making Your AI App Expensive Your agent did not get smarter overnight. It may simply have gotten more expensive. A long-running AI workflow can look healthy while a tiny change quietly makes every turn reprocess thousands of tokens. A reordered tool, a request ID tucked into instructions, a changed schema, or a fallback model can turn a reusable prefix into fresh work. The response still arrives. Your logs still show success. The bill and first-token latency are the only clues. OpenAI Prompt Cache Diagnostics gives Responses API teams a way to stop guessing. Instead of treating a low cached-token count as a vague cost problem, you can compare one response with a recent baseline and get a classified explanation for the first detected mismatch. This guide shows how to turn that signal into a safe engineering loop: isolate the difference, make one reversible fix, and verify the result on representative traffic. The timing matters. OpenAI’s current API changelog lists the new GPT-6 Sol and GPT-6 Luna models, while Prompt Cache Diagnostics is available for GPT-5.6 and later supported models. A model rollout is a sensible time to confirm what your own application is actually reusing, rather than assume an upgraded model or a maintained session preserves every cost-saving property you expected. Why prompt-cache misses are a production problem Prompt caching reuses the work already done on an unchanged beginning of a request. OpenAI describes that reusable beginning as a prefix. When later calls share it, the platform can reuse cached computation rather than processing that material again. That can reduce input cost and shorten the time before a response begins. It is especially important for agent loops that carry forward instructions, a repository map, tool definitions, policy text, and previous messages. The trap is that cache reuse is not the same thing as “we kept the same conversation.” A conversation can be continuous while the request sent to the model has changed near its start. A framework may regenerate tools in a new order. An experiment flag may select another model. A logging helper may inject a timestamp into the instructions. A structured-output schema may gain an optional field. All are reasonable product changes. All can alter a prefix that you expected to reuse. That makes a cache miss a reliability signal as well as a finance signal. It tells you an implementation assumption changed between comparable calls. Sometimes that change is intentional and worthwhile. A safety fallback, context compaction, or a new tool may be exactly right. The goal is not a perfect hit rate. The goal is to know whether a costly change is deliberate, bounded, and paid for with eyes open. What OpenAI Prompt Cache Diagnostics actually tells you Prompt Cache Diagnostics is available in the Responses API for GPT-5.6 and later supported models. You choose a recent completed response from the same organization as a baseline, pass its ID through prompt_cache_options.comparison_response_id, then read prompt_cache_diagnostics on the current response. This comparison requests analysis; it does not load the earlier conversation or change the cache behavior of the new request. The result is deliberately narrower than a request-body diff. It may be a cache_hit, a cache_miss with a reason and affected-token estimate, comparison_response_not_found, or unavailable. A hit means the comparison found no cache miss. It does not mean every input token was reused, because new input still needs processing. For actual reuse and billing analysis, inspect usage.input_tokens_details.cached_tokens on the response. This distinction matters. Treat diagnostics as a high-quality hypothesis generator, not as a magic accounting system. OpenAI notes that diagnostics are best effort and report the first classified reason. Fix that one difference, repeat the same comparison, and you may expose the next difference behind it. The diagnostic loop to add to an agent harness Do not enable a comparison on every request forever without deciding how you will use it. Start with a small, representative route: an agent that does several tool calls, a support flow with a large policy prefix, or a coding task that repeatedly carries repository context. Save a response ID only after a successful, completed baseline. On its next comparable turn, compare against that baseline. from openai import OpenAI client = OpenAI()def run_agent_turn(instructions, user_input, tools, baseline_id=None): cache_options = {"mode": "implicit"} if baseline_id: cache_options["comparison_response_id"] = baseline_id response = client.responses.create( model="gpt-6-sol", instructions=instructions, input=user_input, tools=tools, prompt_cache_options=cache_options, ) cached = response.usage.input_tokens_details.cached_tokens diagnostic = response.prompt_cache_diagnostics record = { "response_id": response.id, "cached_tokens": cached, "diagnostic_type": diagnostic.type if diagnostic else None, "miss_reason": getattr(diagnostic, "reason", None), "missed_tokens": getattr(diagnostic, "cache_missed_tokens", None), } return response, record Keep that record small and non-sensitive. You want the response ID, route name, model, service tier, cache-read and cache-write token counts, latency, result status, and diagnostic classification. Do not copy raw prompts or customer content into a cost dashboard just because you are investigating cache reuse. OpenAI says the diagnostic records themselves contain configuration metadata, token estimates, and hashes rather than raw prompts or outputs, which makes them useful in stricter data environments too. Use a stable business key when grouping results: route plus customer plan, workflow version, or agent configuration version. That makes a spike actionable. “Cached tokens dropped” is hard to own. “The document-review route dropped after tool manifest version 14” gives an engineer a place to look. Read the miss reason like a change-review comment The fastest way to waste a week is to rewrite prompts after every miss. Instead, take the reason literally, form one narrow theory, and compare a controlled request again. The common cases are more mechanical than mysterious. tools_changed: stabilize the supplied tool contract Tool definitions are part of the rendered context. Adding, removing, reordering, or editing their names, descriptions, schemas, or configuration can prevent reuse. First, serialize the supplied tools in your own telemetry and compare a content hash, count, and order. Then decide whether you truly need to mutate the tool list. If […]
What to do next
Read the source, then use the company and guide links to understand which vendors, tools, or workflows are most exposed.
Last Updated on September 25, 2026 by Editorial Team Author(s): Ethan Mark Originally published on Towards AI. OpenAI Prompt Cache Diagnostics: Find the Prefix Drift Making Your AI App Expensive Your agent did not get smarter overnight. It may simply have gotten more expensive. A long-running AI workflow can look healthy while a tiny change quietly makes every turn reprocess thousands of tokens. A reordered tool, a request ID tucked into instructions, a changed schema, or a fallback model can turn a reusable prefix into fresh work. The response still arrives. Your logs still show success. The bill and first-token latency are the only clues. OpenAI Prompt Cache Diagnostics gives Responses API teams a way to stop guessing. Instead of treating a low cached-token count as a vague cost problem, you can compare one response with a recent baseline and get a classified explanation for the first detected mismatch. This guide shows how to turn that signal into a safe engineering loop: isolate the difference, make one reversible fix, and verify the result on representative traffic. The timing matters. OpenAI’s current API changelog lists the new GPT-6 Sol and GPT-6 Luna models, while Prompt Cache Diagnostics is available for GPT-5.6 and later supported models. A model rollout is a sensible time to confirm what your own application is actually reusing, rather than assume an upgraded model or a maintained session preserves every cost-saving property you expected. Why prompt-cache misses are a production problem Prompt caching reuses the work already done on an unchanged beginning of a request. OpenAI describes that reusable beginning as a prefix. When later calls share it, the platform can reuse cached computation rather than processing that material again. That can reduce input cost and shorten the time before a response begins. It is especially important for agent loops that carry forward instructions, a repository map, tool definitions, policy text, and previous messages. The trap is that cache reuse is not the same thing as “we kept the same conversation.” A conversation can be continuous while the request sent to the model has changed near its start. A framework may regenerate tools in a new order. An experiment flag may select another model. A logging helper may inject a timestamp into the instructions. A structured-output schema may gain an optional field. All are reasonable product changes. All can alter a prefix that you expected to reuse. That makes a cache miss a reliability signal as well as a finance signal. It tells you an implementation assumption changed between comparable calls. Sometimes that change is intentional and worthwhile. A safety fallback, context compaction, or a new tool may be exactly right. The goal is not a perfect hit rate. The goal is to know whether a costly change is deliberate, bounded, and paid for with eyes open. What OpenAI Prompt Cache Diagnostics actually tells you Prompt Cache Diagnostics is available in the Responses API for GPT-5.6 and later supported models. You choose a recent completed response from the same organization as a baseline, pass its ID through prompt_cache_options.comparison_response_id, then read prompt_cache_diagnostics on the current response. This comparison requests analysis; it does not load the earlier conversation or change the cache behavior of the new request. The result is deliberately narrower than a request-body diff. It may be a cache_hit, a cache_miss with a reason and affected-token estimate, comparison_response_not_found, or unavailable. A hit means the comparison found no cache miss. It does not mean every input token was reused, because new input still needs processing. For actual reuse and billing analysis, inspect usage.input_tokens_details.cached_tokens on the response. This distinction matters. Treat diagnostics as a high-quality hypothesis generator, not as a magic accounting system. OpenAI notes that diagnostics are best effort and report the first classified reason. Fix that one difference, repeat the same comparison, and you may expose the next difference behind it. The diagnostic loop to add to an agent harness Do not enable a comparison on every request forever without deciding how you will use it. Start with a small, representative route: an agent that does several tool calls, a support flow with a large policy prefix, or a coding task that repeatedly carries repository context. Save a response ID only after a successful, completed baseline. On its next comparable turn, compare against that baseline. from openai import OpenAI client = OpenAI()def run_agent_turn(instructions, user_input, tools, baseline_id=None): cache_options = {"mode": "implicit"} if baseline_id: cache_options["comparison_response_id"] = baseline_id response = client.responses.create( model="gpt-6-sol", instructions=instructions, input=user_input, tools=tools, prompt_cache_options=cache_options, ) cached = response.usage.input_tokens_details.cached_tokens diagnostic = response.prompt_cache_diagnostics record = { "response_id": response.id, "cached_tokens": cached, "diagnostic_type": diagnostic.type if diagnostic else None, "miss_reason": getattr(diagnostic, "reason", None), "missed_tokens": getattr(diagnostic, "cache_missed_tokens", None), } return response, record Keep that record small and non-sensitive. You want the response ID, route name, model, service tier, cache-read and cache-write token counts, latency, result status, and diagnostic classification. Do not copy raw prompts or customer content into a cost dashboard just because you are investigating cache reuse. OpenAI says the diagnostic records themselves contain configuration metadata, token estimates, and hashes rather than raw prompts or outputs, which makes them useful in stricter data environments too. Use a stable business key when grouping results: route plus customer plan, workflow version, or agent configuration version. That makes a spike actionable. “Cached tokens dropped” is hard to own. “The document-review route dropped after tool manifest version 14” gives an engineer a place to look. Read the miss reason like a change-review comment The fastest way to waste a week is to rewrite prompts after every miss. Instead, take the reason literally, form one narrow theory, and compare a controlled request again. The common cases are more mechanical than mysterious. tools_changed: stabilize the supplied tool contract Tool definitions are part of the rendered context. Adding, removing, reordering, or editing their names, descriptions, schemas, or configuration can prevent reuse. First, serialize the supplied tools in your own telemetry and compare a content hash, count, and order. Then decide whether you truly need to mutate the tool list. If […]
This AimostAll brief summarizes the linked source so readers can scan AI developments quickly and jump to the original reporting when needed.
