What actually moves time to first token

Our chat's time to first token was 2.8 seconds. It is around 1 second now, on the same model. Two months of work, and most of that work was finding out which of my ideas were wrong.

The tool contracts were most of the prompt. We run on GPT-5.6, which can defer tool loading: a tool's parameter schema stays out of the request until the model searches for that tool, while its description always ships, because the description is the search index. Our descriptions were the contract. When to call, batching rules, argument docs. The fattest one ran 2,656 tokens. So we split every tool: a hand-written one-paragraph summary stays in the description, and the full contract moves into the parameter schema, which deferral actually drops. The tools block went from about 14k tokens to 2.2k, time to first token halved, and tool selection held at 84 of 84 eval trials. I wrote that one up separately. It is the single biggest thing we did.

Tokens are not the unit of latency. Array entries are. I ran the same prompt at 0, 5, and 40 tool entries, holding input tokens fixed at about 30.5k, all of it roughly 100% cached. Warm medians: 690ms, 863ms, 1,383ms. The cost is real and it is sublinear. The first five entries run about 35ms each; by forty it averages 17ms. Caching never removes it, not even for an entry that is nothing but a name and a one-line summary. Then the opposite experiment: 12k extra tokens of prose in the system prompt cost 91ms.

Those two paragraphs look like they contradict each other, and the reconciliation is the whole point. The tokens we deleted from the tools block were not prose. They were schema living inside array entries, which the model reads and weighs before it emits anything. Prose is nearly free when cached. Array entries are not, ever.

A cache flag the API accepted and ignored. I shipped prompt_cache_retention: "24h" and believed it, because the docs said it existed. It is a legacy field for models up to 5.5, silently dropped by 5.6. Three probes came back cold before I stopped blaming my test setup, and the first of those probes was contaminated by its own design: my control run primed the very cache key it was measuring. A field that gets accepted and ignored looks exactly like a field that works. The real control is prompt_cache_options.ttl, and it caps at 30 minutes.

What prompt caching actually does, once I stopped guessing. Matching is exact prefix, up to the last message boundary. There is no longest-prefix fallback. Change one character early and you pay for everything after it, so anything volatile belongs at the end of a prompt and never near the top. Keys are per surface, and ours is per user on chat, which means a user's prefix gets warmed by that user or by nobody. Past the guaranteed window it is best effort. I measured hits at 25 minutes and misses by roughly two and a half hours.

So warm the cache instead of shrinking the prompt. Opening the app now fires a throwaway model call that builds the identical prefix the real turn will use, with tool calls disabled and output capped at 64 tokens. It costs one full prefill per app open, which is the honest price of the trick. In production the ping wrote 16,823 tokens with none cached, and the user's real turn eight seconds later read 12,804 of them from cache. The gap between those two numbers is the tail of the prefix that only exists once the user has actually typed something.

My first version of that ping built the prefix from a hardcoded reminder time and the user's stored timezone, while the real turn builds it from the device clock. Same intent, different bytes. For anyone whose phone disagreed with their profile, or who had changed their reminder hour, the ping would warm a prefix that no turn could ever read. It costs money, saves nothing, and reports success. The fix was to make both paths call one function, so the two prefixes are identical by construction rather than by my care.

Three ideas the measurements killed. A router in front of the main call to pick a reasoning budget per turn: I pulled two weeks of production turns first, and 52% of them spent zero reasoning tokens, average 50. The model was already throttling itself, so the router would have added a hop to save nothing. A hybrid tools array that drops rare tools out of the array and into a prose index: built, measured at 808ms against 1,200ms, then parked, because 390ms did not justify the machinery at forty tools. It gets revived at fifty. And the client rewrite, which is the one I would have gotten most wrong.

Instrument the client before you rewrite it. Our iOS issue described a second of send animation, a reducer rebuilding the whole message on every token, and the entire streaming pipeline on the main thread. All of it read off the code, none of it measured. I added timing spans first. The client adds about 57ms before the request leaves and roughly 170 to 290ms spread across the stream. The animation runs alongside the request instead of in front of it. Two of the three planned fixes are closed now, unbuilt. What the spans did find was a real round trip creating a session, 264 to 449ms, though only on the first message of a new chat, plus a pagination call that had been failing 100% of the time inside a retry loop nobody had noticed.

Where it lands. The floor for this model with no tools at all is about 650ms. We sit at roughly 1 second with a full array of forty entries, and the gap between those two numbers is mostly those entries and the network. Some turns come in at 800ms. Others are slower, which is what depending on someone else's inference capacity looks like. All of these are warm medians, and I have not broken out a p95, which is the number anyone actually shipping this should care about.

Three things I cannot close. I cannot prove the 30-minute cache flag does anything, because any gap long enough to kill the cache is longer than the guarantee it makes, so no experiment separates them. I have one live verification of the warm ping rather than a fleet-wide hit rate, so "cold morning turns are fixed" is still a claim and not a result. And every tool the model loads mid-session rides in the history for the rest of that session, so a long conversation creeps back toward a fat prompt. Cached, but never free, at somewhere between 17 and 35ms an entry.