conftrace_

batch scheduling

5 papers

Explore in graph

Co-occurring keywords · papers mentioning both

throughput latency tradeoff (1) gpu compute utilization (1) inference scheduler (1) dataflow graph (1) distributed serving (1) iteration-level scheduling (1) large language model (1) transformer model (1) queuing theory (1) llm serving (1)

Papers

DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS ICLR 2025 Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve OSDI 2024 S^3: Increasing GPU Utilization during Generative Inference for Higher Throughput NeurIPS 2023 Orca: A Distributed Serving System for Transformer-Based Generative Models OSDI 2022 Batchy: Batch-scheduling Data Flow Graphs with Service-level Objectives NSDI 2020