Tag: inference
All the articles with the tag "inference".
Throughput per dollar, once the traffic is bursty
23 min readOne roofline number sets the speculative-decoding gate, the batch-size knee and the chunked-prefill floor. Why max-num-seqs is a latency control.