wehub-resource-sync
59a0a3844c
PR Test AMD / cancel-on-close (push) Has been skipped
PR Test NVIDIA ARM / scan (push) Has been skipped
PR Test NVIDIA / cancel-on-close (push) Has been skipped
PR Test AMD / scan (push) Has been skipped
PR Test NVIDIA ARM / cancel-on-close (push) Has been skipped
PR Test NVIDIA / scan (push) Has been skipped
Release Docker Images / build (cu129-torch-2.11.0) (push) Has been skipped
Release Docker Images / build (cu130-torch-2.11.0) (push) Has been skipped
Release PyPI / publish (push) Has been skipped
Scheduler Python Test / test (push) Successful in 27m19s
Docs / build (push) Successful in 28m8s
Scheduler C++ Test / test (push) Successful in 28m19s
Scheduler C++ Test / test-flat (push) Successful in 28m18s
Docs / deploy (push) Has been cancelled
PR Test AMD / finish (push) Has been cancelled
PR Test NVIDIA / finish (push) Has been cancelled
PR Test NVIDIA ARM / finish (push) Has been cancelled
PR Test NVIDIA ARM / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
PR Test AMD / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
PR Test NVIDIA / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
1.4 KiB
1.4 KiB
Getting Started
This guide brings up a TokenSpeed development environment and verifies that the runtime can start.
Prerequisites
- NVIDIA GPU host
- Docker with GPU support
- enough shared memory for model serving
- access to the model checkpoints you plan to serve
Start a Runner Container
docker pull lightseekorg/tokenspeed-runner:latest
docker run -itd \
--shm-size 32g \
--gpus all \
-v /raid/cache:/home/runner/.cache \
--ipc=host \
--network=host \
--pid=host \
--privileged \
--name tokenspeed \
lightseekorg/tokenspeed-runner:latest \
/bin/bash
Inside the container:
git clone https://github.com/lightseekorg/tokenspeed.git
cd tokenspeed
Install Packages
Install the Python runtime:
export PIP_BREAK_SYSTEM_PACKAGES=1
pip install -e "./python" --no-build-isolation
Install the kernel package. Its Python package metadata installs the selected backend dependencies automatically.
pip install -e tokenspeed-kernel/python/ --no-build-isolation
Install the scheduler package:
pip install -e tokenspeed-scheduler/
Verify
tokenspeed env
tokenspeed serve --help
Launch
tokenspeed serve openai/gpt-oss-20b \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1
For model-specific examples, continue with Model Recipes.