conftrace_

batch scheduling

4 papers

Explore in graph

Co-occurring keywords

large language model inference (5) throughput latency tradeoff (1) gpu compute utilization (1) inference scheduler (1) distributed serving (1) iteration-level scheduling (1) transformer model (1996) queuing theory (3) data flow graph (2) service-level objective (1)

Papers

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve OSDI 2024
$S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput NIPS 2023
Orca: A Distributed Serving System for Transformer-Based Generative Models OSDI 2022
Batchy: Batch-scheduling Data Flow Graphs with Service-level Objectives NSDI 2020