wehub-resource-sync
e06fe8e8c6
Secret Leaks / trufflehog (push) Failing after 1s
Build documentation / build (push) Failing after 1s
Build documentation / build_other_lang (push) Failing after 0s
CodeQL Security Analysis / CodeQL Analysis (push) Failing after 0s
PR CI / pr-ci (push) Failing after 1s
Slow tests on important models (on Push - A10) / Get all modified files (push) Failing after 1s
Slow tests on important models (on Push - A10) / Model CI (push) Has been skipped
Self-hosted runner (benchmark) / Benchmark (aws-g5-4xlarge-cache) (push) Has been cancelled
New model PR merged notification / Notify new model (push) Has been cancelled
Update Transformers metadata / build_and_package (push) Has been cancelled
1.3 KiB
1.3 KiB
Continuous batching
This page documents the classes behind continuous batching inference: submitting prompts, configuring scheduling and memory limits, and retrieving results.
For usage examples, see the Continuous batching guide and for how scheduling and memory interact, see the Continuous batching architecture doc.
ContinuousMixin.generate_batch
autodoc ContinuousMixin.generate_batch
ContinuousBatchingManager
autodoc ContinuousBatchingManager
Continuous batching config
autodoc ContinuousBatchingConfig