chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,80 @@
|
||||
# Claude Code
|
||||
|
||||
[Claude Code](https://code.claude.com/docs/en/quickstart) is Anthropic's official agentic coding tool that lives in your terminal. It can understand your codebase, edit files, run commands, and help you write code more efficiently.
|
||||
|
||||
By pointing Claude Code at a vLLM server, you can use your own models as the backend instead of the Anthropic API. This is useful for:
|
||||
|
||||
- Running fully local/private coding assistance
|
||||
- Using open-weight models with tool calling capabilities
|
||||
- Testing and developing with custom models
|
||||
|
||||
## How It Works
|
||||
|
||||
vLLM implements the Anthropic Messages API, which is the same API that Claude Code uses to communicate with Anthropic's servers. By setting `ANTHROPIC_BASE_URL` to point at your vLLM server, Claude Code sends its requests to vLLM instead of Anthropic. vLLM then translates these requests to work with your local model and returns responses in the format Claude Code expects.
|
||||
|
||||
This means any model served by vLLM with proper tool calling support can act as a drop-in replacement for Claude models in Claude Code.
|
||||
|
||||
## Requirements
|
||||
|
||||
Claude Code requires a model with strong tool calling capabilities. The model must support the OpenAI-compatible tool calling API. See [Tool Calling](../../features/tool_calling.md) for details on enabling tool calling for your model.
|
||||
|
||||
## Installation
|
||||
|
||||
First, install Claude Code by following the [official installation guide](https://docs.anthropic.com/en/docs/claude-code/getting-started).
|
||||
|
||||
## Starting the vLLM Server
|
||||
|
||||
Start vLLM with a tool-calling capable model - here's an example using `openai/gpt-oss-120b`:
|
||||
|
||||
```bash
|
||||
vllm serve openai/gpt-oss-120b --served-model-name my-model --enable-auto-tool-choice --tool-call-parser openai
|
||||
```
|
||||
|
||||
For other models, you'll need to enable tool calling explicitly with `--enable-auto-tool-choice` and the right `--tool-call-parser`. Refer to the [Tool Calling documentation](../../features/tool_calling.md) for the correct flags for your model.
|
||||
|
||||
## Configuring Claude Code
|
||||
|
||||
Launch Claude Code with environment variables pointing to your vLLM server:
|
||||
|
||||
```bash
|
||||
ANTHROPIC_BASE_URL=http://localhost:8000 \
|
||||
ANTHROPIC_API_KEY=dummy \
|
||||
ANTHROPIC_AUTH_TOKEN=dummy \
|
||||
ANTHROPIC_DEFAULT_OPUS_MODEL=my-model \
|
||||
ANTHROPIC_DEFAULT_SONNET_MODEL=my-model \
|
||||
ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model \
|
||||
claude
|
||||
```
|
||||
|
||||
The environment variables:
|
||||
|
||||
| Variable | Description |
|
||||
| -------------------------------- | --------------------------------------------------------------------- |
|
||||
| `ANTHROPIC_BASE_URL` | Points to your vLLM server (default port is 8000) |
|
||||
| `ANTHROPIC_API_KEY` | Can be any value since vLLM doesn't require authentication by default |
|
||||
| `ANTHROPIC_AUTH_TOKEN` | Is required. Can be any value. |
|
||||
| `ANTHROPIC_DEFAULT_OPUS_MODEL` | Model name for Opus-tier requests |
|
||||
| `ANTHROPIC_DEFAULT_SONNET_MODEL` | Model name for Sonnet-tier requests |
|
||||
| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | Model name for Haiku-tier requests |
|
||||
|
||||
!!! tip
|
||||
You can add these environment variables to your shell profile (e.g., `.bashrc`, `.zshrc`), Claude Code configuration file (`~/.claude/settings.json`), or create a wrapper script for convenience.
|
||||
|
||||
!!! warning
|
||||
Claude Code recently started injecting a per-request hash in the system prompt, which can defeat [prefix caching](../../design/prefix_caching.md) because the prompt changes on every request, causing greatly reduced performance. This is addressed automatically in vLLM versions > 0.17.1 but for older versions `"CLAUDE_CODE_ATTRIBUTION_HEADER": "0"` should be added to the `"env"` section of `~/.claude/settings.json` (see this [blog post](https://unsloth.ai/docs/basics/claude-code#fixing-90-slower-inference-in-claude-code) from Unsloth).
|
||||
|
||||
## Testing the Setup
|
||||
|
||||
Once Claude Code launches, try a simple prompt to verify the connection:
|
||||
|
||||
|
||||
|
||||
If the model responds correctly, your setup is working. You can now use Claude Code with your vLLM-served model for coding tasks.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Connection refused**: Ensure vLLM is running and accessible at the specified URL. Check that the port matches.
|
||||
|
||||
**Tool calls not working**: Verify that your model supports tool calling and that you've enabled it with the correct `--tool-call-parser` flag. See [Tool Calling](../../features/tool_calling.md).
|
||||
|
||||
**Model not found**: Ensure the `--served-model-name` matches the model names in your environment variables. You cannot use model names with `/` in them, such as `openai/gpt-oss-120b` directly from Huggingface, so beware of that limitation with Claude Code.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Codex
|
||||
|
||||
[Codex](https://github.com/openai/codex) is OpenAI's official agentic coding tool that lives in your terminal. It can understand your codebase, edit files, run commands, and help you write code more efficiently.
|
||||
|
||||
By pointing Codex at a vLLM server, you can use your own models as the backend instead of the OpenAI API. This is useful for:
|
||||
|
||||
- Running fully local/private coding assistance
|
||||
- Using open-weight models with tool calling capabilities
|
||||
- Testing and developing with custom models
|
||||
|
||||
## How It Works
|
||||
|
||||
vLLM implements the OpenAI-Responses API, which is the same API that Codex uses to communicate with OpenAI's servers. By configuring Codex to point at your vLLM server, Codex sends its requests to vLLM instead of OpenAI. vLLM then translates these requests to work with your local model and returns responses in the format Codex expects.
|
||||
|
||||
This means any model served by vLLM with proper tool calling support can act as a drop-in replacement for OpenAI models in Codex.
|
||||
|
||||
## Requirements
|
||||
|
||||
Codex requires a model with strong tool calling capabilities. The model must support the OpenAI-Responses tool calling API. See [Tool Calling](../../features/tool_calling.md) for details on enabling tool calling for your model.
|
||||
|
||||
## Installation
|
||||
|
||||
First, install Codex by following the [official installation guide](https://github.com/openai/codex).
|
||||
|
||||
## Starting the vLLM Server
|
||||
|
||||
Start vLLM with a tool-calling capable model - here's an example using `Qwen/Qwen3-27B`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.6-27B --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
|
||||
|
||||
```
|
||||
|
||||
For other models, you'll need to enable tool calling explicitly with `--enable-auto-tool-choice` and the right `--tool-call-parser`. Refer to the [Tool Calling documentation](../../features/tool_calling.md) for the correct flags for your model.
|
||||
|
||||
## Configuring Codex
|
||||
|
||||
Codex is configured via a TOML file located at `~/.codex/config.toml`. Create or edit this file to point Codex at your vLLM server:
|
||||
|
||||
```toml
|
||||
model = "my-model"
|
||||
model_provider = "vllm"
|
||||
|
||||
[model_providers.vllm]
|
||||
name = "vLLM"
|
||||
env_key = "VLLM_API_KEY"
|
||||
base_url = "http://localhost:8000/v1"
|
||||
wire_api = "responses"
|
||||
```
|
||||
|
||||
The configuration fields:
|
||||
|
||||
| Field | Description |
|
||||
| ----- | ----------- |
|
||||
| `model` | The model name to use. Must match the `--served-model-name` you passed to vLLM. |
|
||||
| `model_provider` | Set to `"vllm"` to use your local vLLM server. |
|
||||
| `[model_providers.vllm]` | Configuration section for the vLLM provider. |
|
||||
| `name` | A display name for your vLLM provider. |
|
||||
| `env_key` | The name of an environment variable that Codex will read for the API key. vLLM does not require authentication by default, so this can be any value. |
|
||||
| `base_url` | The URL of your vLLM server's OpenAI-compatible API endpoint (default is `http://localhost:8000/v1`). |
|
||||
| `wire_api` | The API style to use. Set to `"responses"` for the OpenAI Responses API |
|
||||
|
||||
!!! tip
|
||||
You can set the `env_key` to any dummy environment variable since vLLM doesn't require authentication by default:
|
||||
```bash
|
||||
export VLLM_API_KEY=dummy
|
||||
```
|
||||
|
||||
!!! warning
|
||||
When using the `responses` API, ensure your vLLM version supports the OpenAI Responses API.
|
||||
|
||||
## Testing the Setup
|
||||
|
||||
Once Codex is configured, launch it in your project directory:
|
||||
|
||||
```bash
|
||||
codex
|
||||
```
|
||||
|
||||
Try a simple prompt to verify the connection, such as asking it to explain a file in your project. If the model responds correctly, your setup is working. You can now use Codex with your vLLM-served model for coding tasks.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Connection refused**: Ensure vLLM is running and accessible at the specified URL. Check that the port matches and that `base_url` includes the `/v1` path suffix.
|
||||
|
||||
**Tool calls not working**: Verify that your model supports tool calling and that you've enabled it with the correct `--tool-call-parser` flag. See [Tool Calling](../../features/tool_calling.md).
|
||||
|
||||
**Model not found**: Ensure the `model` field in `~/.codex/config.toml` matches the `--served-model-name` you passed to vLLM.
|
||||
@@ -0,0 +1,32 @@
|
||||
# LangChain
|
||||
|
||||
vLLM is also available via [LangChain](https://github.com/langchain-ai/langchain) .
|
||||
|
||||
To install LangChain, run
|
||||
|
||||
```bash
|
||||
pip install langchain langchain_community -q
|
||||
```
|
||||
|
||||
To run inference on a single or multiple GPUs, use `VLLM` class from `langchain`.
|
||||
|
||||
??? code
|
||||
|
||||
```python
|
||||
from langchain_community.llms import VLLM
|
||||
|
||||
llm = VLLM(
|
||||
model="Qwen/Qwen3-4B",
|
||||
trust_remote_code=True, # mandatory for hf models
|
||||
max_new_tokens=128,
|
||||
top_k=10,
|
||||
top_p=0.95,
|
||||
temperature=0.8,
|
||||
# for distributed inference
|
||||
# tensor_parallel_size=...,
|
||||
)
|
||||
|
||||
print(llm("What is the capital of France ?"))
|
||||
```
|
||||
|
||||
Please refer to this [Tutorial](https://python.langchain.com/docs/integrations/llms/vllm) for more details.
|
||||
@@ -0,0 +1,24 @@
|
||||
# LlamaIndex
|
||||
|
||||
vLLM is also available via [LlamaIndex](https://github.com/run-llama/llama_index) .
|
||||
|
||||
To install LlamaIndex, run
|
||||
|
||||
```bash
|
||||
pip install llama-index-llms-vllm -q
|
||||
```
|
||||
|
||||
To run inference on a single or multiple GPUs, use `Vllm` class from `llamaindex`.
|
||||
|
||||
```python
|
||||
from llama_index.llms.vllm import Vllm
|
||||
|
||||
llm = Vllm(
|
||||
model="microsoft/Orca-2-7b",
|
||||
tensor_parallel_size=4,
|
||||
max_new_tokens=100,
|
||||
vllm_kwargs={"gpu_memory_utilization": 0.5},
|
||||
)
|
||||
```
|
||||
|
||||
Please refer to this [Tutorial](https://docs.llamaindex.ai/en/latest/examples/llm/vllm/) for more details.
|
||||
Reference in New Issue
Block a user