Latency

2 stories tagged Latency on Learning Fanatic.

Lesson 3.3: Streaming, latency and cost of a request Every model request has a price and a wait. Use the sliders to see how input size, output size and volume change your bill and what the user feels.
Lesson 8.2: Caching and batching to serve more requests Batching lets a model serve many requests at once, and caching avoids repeat work. Watch static, dynamic and continuous batching schedule the same six requests.