How we cut response times by 80%
We rolled out speculative decoding across our inference cluster. Median time to first token dropped from 420 ms to 85 ms, and 88.4% of the draft tokens were accepted.
The change needed no new hardware: a small draft model proposes tokens, and the main model checks several of them at once.