The tool definitions were the slow part
A 100% cache hit makes tokens cheaper, not invisible. Deferring the tool contracts halved our chat's time to first token.
Tobias Eichenwald · · 2 min read
This week I moved Roma's chat to Vercel AI SDK 7 and GPT-5.6. The migration itself was routine. What it unlocked wasn't: OpenAI can now defer tool definitions, loading one into context only when the model searches for it.
First attempt saved 3% of input tokens. Not a typo. Deferral drops a tool's parameter schema but keeps its description verbatim, because the description is the search index. And our descriptions ARE the contract: when to call, batching rules, argument docs, the works. Our fattest tool ran 2.6k tokens, almost all of it description. We were deferring the crumbs.
So we split every tool. A hand-written one-paragraph summary stays visible, carrying what routing needs: what the tool does and when to reach for it. The full contract moves into the parameter schema, which deferral actually drops. Everything is deferred except the clarifying-question tool (a model that must search before it can ask stops asking) and the search machinery itself. Measured at our production prompt shape: the tools block went from ~14k tokens to 2.2k, the request from 24.6k to 9.5k, and time to first token from 2.8s to 1.4s.
The part that surprised me: turns that USE a tool got faster too. I expected the search hop to cost us exactly there. Instead the tool call now starts at 1.7s instead of 3.0s.
The reason is that caching saves money, not attention. Even at a 100% cache hit, the model still reads every token before emitting its first one. We measured our full tool surface at roughly two seconds of latency while fully cached. The server-side tool search costs a few hundred milliseconds. Reading 14k tokens of contracts you won't use costs more.
Quality held: our tool-selection eval scored 84/84 trials, identical before and after. The one regression the end-to-end fixtures caught traced back to an old contradiction in our own system prompt, not to the deferral.
What I haven't watched yet: every loaded tool rides in the session's history afterwards, so a long session that touches ten tools creeps back toward a fat prompt. Cached, but I want a week of real sessions before this leaves the feature flag.