wehub-resource-sync
59a0a3844c
PR Test AMD / cancel-on-close (push) Has been skipped
PR Test NVIDIA ARM / scan (push) Has been skipped
PR Test NVIDIA / cancel-on-close (push) Has been skipped
PR Test AMD / scan (push) Has been skipped
PR Test NVIDIA ARM / cancel-on-close (push) Has been skipped
PR Test NVIDIA / scan (push) Has been skipped
Release Docker Images / build (cu129-torch-2.11.0) (push) Has been skipped
Release Docker Images / build (cu130-torch-2.11.0) (push) Has been skipped
Release PyPI / publish (push) Has been skipped
Scheduler Python Test / test (push) Successful in 27m19s
Docs / build (push) Successful in 28m8s
Scheduler C++ Test / test (push) Successful in 28m19s
Scheduler C++ Test / test-flat (push) Successful in 28m18s
Docs / deploy (push) Has been cancelled
PR Test AMD / finish (push) Has been cancelled
PR Test NVIDIA / finish (push) Has been cancelled
PR Test NVIDIA ARM / finish (push) Has been cancelled
PR Test NVIDIA ARM / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
PR Test AMD / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
PR Test NVIDIA / ${{ matrix.name }} (${{ matrix.runner }}) (push) Has been cancelled
77 lines
1.4 KiB
Markdown
77 lines
1.4 KiB
Markdown
# Getting Started
|
|
|
|
This guide brings up a TokenSpeed development environment and verifies that the
|
|
runtime can start.
|
|
|
|
## Prerequisites
|
|
|
|
- NVIDIA GPU host
|
|
- Docker with GPU support
|
|
- enough shared memory for model serving
|
|
- access to the model checkpoints you plan to serve
|
|
|
|
## Start a Runner Container
|
|
|
|
```bash
|
|
docker pull lightseekorg/tokenspeed-runner:latest
|
|
|
|
docker run -itd \
|
|
--shm-size 32g \
|
|
--gpus all \
|
|
-v /raid/cache:/home/runner/.cache \
|
|
--ipc=host \
|
|
--network=host \
|
|
--pid=host \
|
|
--privileged \
|
|
--name tokenspeed \
|
|
lightseekorg/tokenspeed-runner:latest \
|
|
/bin/bash
|
|
```
|
|
|
|
Inside the container:
|
|
|
|
```bash
|
|
git clone https://github.com/lightseekorg/tokenspeed.git
|
|
cd tokenspeed
|
|
```
|
|
|
|
## Install Packages
|
|
|
|
Install the Python runtime:
|
|
|
|
```bash
|
|
export PIP_BREAK_SYSTEM_PACKAGES=1
|
|
pip install -e "./python" --no-build-isolation
|
|
```
|
|
|
|
Install the kernel package. Its Python package metadata installs the selected
|
|
backend dependencies automatically.
|
|
|
|
```bash
|
|
pip install -e tokenspeed-kernel/python/ --no-build-isolation
|
|
```
|
|
|
|
Install the scheduler package:
|
|
|
|
```bash
|
|
pip install -e tokenspeed-scheduler/
|
|
```
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
tokenspeed env
|
|
tokenspeed serve --help
|
|
```
|
|
|
|
## Launch
|
|
|
|
```bash
|
|
tokenspeed serve openai/gpt-oss-20b \
|
|
--host 0.0.0.0 \
|
|
--port 8000 \
|
|
--tensor-parallel-size 1
|
|
```
|
|
|
|
For model-specific examples, continue with [Model Recipes](../recipes/models.md).
|