chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,201 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -0,0 +1,416 @@
|
||||
# Step 3.7 Flash
|
||||
|
||||
- **[ModelPage]**: https://static.stepfun.com/blog/step-3.7-flash/
|
||||
- **[HuggingFace]**:
|
||||
- BF16: https://huggingface.co/stepfun-ai/Step-3.7-Flash/
|
||||
- FP8: https://huggingface.co/stepfun-ai/Step-3.7-Flash-FP8
|
||||
- NVFP4: https://huggingface.co/stepfun-ai/Step-3.7-Flash-NVFP4
|
||||
- GGUF: https://huggingface.co/stepfun-ai/Step-3.7-Flash-GGUF
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
Step 3.7 Flash is a 198B-parameter sparse Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding. Engineered for high-frequency production workloads, it activates approximately 11B parameters per token and delivers a throughput of up to 400 tokens per second. Step 3.7 Flash supports a 256k context window and offers three selectable reasoning levels (low, medium, and high) so developers can easily balance speed, cost, and cognitive depth.
|
||||
|
||||
We built Step 3.7 Flash for developers who need to scale agentic workflows that combine perception, search, and reasoning. It is designed to handle intensive tasks such as parsing massive financial reports in one pass, running multi-step search loops with cross-source verification, or operating concurrent coding agents in high-throughput pipelines.
|
||||
|
||||
## 2. Capabilities & Performance
|
||||
|
||||
### Multimodal Perception and Verification
|
||||
|
||||
The model delivers top-tier visual intelligence, securing first place on SimpleVQA (Search) with a 79.2 and achieving frontier parity on V* (Python) at 95.3. These metrics reflect strong visual grounding and retrieval-augmented reasoning beyond basic image description. The model accurately processes dense visual interfaces, such as UI wireframes, application GUIs, and data charts, to map them into structured code. When it encounters an incomplete visual asset, it can independently identify missing data and execute lookups to verify context before returning a factually verified conclusion.
|
||||
|
||||
### Workflow Integrity and Tool Orchestration
|
||||
|
||||
Execution reliability is critical for autonomous agents. Step 3.7 Flash leads the ClawEval-1.1 benchmark with a score of 67.1, which significantly outperforms the next closest competitor at 59.8. This performance demonstrates high resistance to adversarial traps and strict adherence to system policies during multi-turn orchestration. Backed by scores of 49.5 on Toolathlon and 48.1 on HLE w. Tool, this profile ensures high trajectory integrity. Step 3.7 Flash reliably interacts with external APIs and executes long-horizon workflows without drifting from instructions or violating system constraints.
|
||||
|
||||
### Code Engineering and Professional Baselines
|
||||
|
||||
Step 3.7 Flash is built for live engineering tasks and secured a definitive second-place finish on SWE-Bench PRO with a score of 56.3. It can independently trace multi-file repositories, isolate bugs from raw issue reports, and generate functional patches that pass automated unit tests. While evaluations like Terminal-Bench 2.1 (59.5) and GDPVal-AA (45.8) show clear areas for future optimization compared to the absolute peak of the cohort, they establish a dependable baseline for system interactions and structured professional deliverables.
|
||||
|
||||

|
||||
|
||||
### NVFP4 + MTP
|
||||
|
||||
Step 3.7 Flash is also available in an NVFP4-quantized variant for efficient deployment on NVIDIA GPUs. The latest [NVFP4 checkpoint](https://huggingface.co/stepfun-ai/Step-3.7-Flash-NVFP4) includes MTP draft layers and supports vLLM speculative decoding with:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
```
|
||||
|
||||
On GPQA Diamond avg@16, the NVFP4 + MTP checkpoint matches quality within statistical noise compared with the same NVFP4 checkpoint without MTP: **77.81% vs. 78.41%** item accuracy over 3168 records.
|
||||
|
||||
On a GB200 TP=4 vLLM setup with GPQA-style long-reasoning streaming prompts (~250 token prompt, ~1.6K token completion), NVFP4 + MTP improves aggregate decode throughput:
|
||||
|
||||
| Concurrency | NVFP4 + MTP | NVFP4 no-MTP | Speedup |
|
||||
|---:|---:|---:|---:|
|
||||
| 8 | 1309 tok/s | 1155 tok/s | 1.13x |
|
||||
| 32 | 4391 tok/s | 3480 tok/s | 1.26x |
|
||||
| **64** | **8229 tok/s** | 5667 tok/s | **1.45x** |
|
||||
|
||||
This makes the NVFP4 checkpoint a practical option for high-throughput long-reasoning workloads. This benchmark characterizes short-prompt, decode-heavy reasoning rather than long-context prefill performance.
|
||||
|
||||
## 3. Pricing
|
||||
|
||||
| Token Type | Price |
|
||||
|---|---|
|
||||
| Input (cache miss) | $0.20 / M tokens |
|
||||
| Input (cache hit) | $0.04 / M tokens |
|
||||
| Output | $1.15 / M tokens |
|
||||
|
||||
## 4. Availability, Deployment, and Ecosystem
|
||||
- Availability: Step 3.7 Flash is available on the StepFun Open Platform — [platform.stepfun.ai](https://platform.stepfun.ai) (Global) and [platform.stepfun.com](https://platform.stepfun.com) (China), OpenRouter, and NVIDIA NIM. StepFun is also partnering with DeepInfra, Fireworks AI, and Modal to expand availability soon.
|
||||
- Deployment: Step 3.7 Flash supports flexible deployment across cloud, data center, and local environments. For large-scale production and enterprise use cases, Step 3.7 Flash can be deployed on modern data center infrastructure. For local and workstation scenarios, it can also run on high-memory devices such as NVIDIA DGX Station, AMD Ryzen AI Max+ 395-based systems, and Mac Studio / Macbook Pro devices with at least 128GB unified memory.
|
||||
- Ecosystem: Step 3.7 Flash is supported across popular open-source infrastructure for both inference and model development. For inference and serving, developers can use vLLM, SGLang, Hugging Face Transformers, and llama.cpp. For model development & customization workflows, StepFun model support has landed in the NVIDIA Nemo ecosystem, including AutoModel, Megatron Core and Megatron Bridge. Step 3.7 Flash is also available as an NVIDIA NIM inference microservice for on-prem, cloud, or hybrid deployment.
|
||||
|
||||
## 5. Examples
|
||||
|
||||
You can get started with Step 3.7 Flash in minutes using StepFun's API or via other inference providers.
|
||||
|
||||
> Pick the right `base_url` for your region. StepFun operates two regional platforms with separate API hosts. The `base_url` you pass to the OpenAI client must match the platform where your API key was issued, otherwise requests will be rejected as unauthorized.
|
||||
>
|
||||
> - **Global**: [platform.stepfun.ai](https://platform.stepfun.ai) — `base_url=https://api.stepfun.ai/v1`
|
||||
> - **China**: [platform.stepfun.com](https://platform.stepfun.com) — `base_url=https://api.stepfun.com/v1`
|
||||
>
|
||||
> To avoid hard-coding the wrong region, the examples below read both the API key and base URL from environment variables. Export them once before running:
|
||||
>
|
||||
> ```bash
|
||||
> export STEP_API_KEY="sk-..."
|
||||
> export STEP_BASE_URL="https://api.stepfun.ai/v1" # use https://api.stepfun.com/v1 for the China platform
|
||||
> ```
|
||||
|
||||
### 5.1 Chat Example
|
||||
|
||||
```python
|
||||
import os
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key=os.environ["STEP_API_KEY"],
|
||||
base_url=os.environ["STEP_BASE_URL"],
|
||||
)
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model="step-3.7-flash",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are an AI assistant provided by StepFun. You are good at Chinese, English, and many other languages, and you can see, think, and act to help users get things done.",
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Introduce StepFun's artificial intelligence capabilities."
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
print(completion)
|
||||
```
|
||||
|
||||
### 5.2 Text and Image Input Example
|
||||
|
||||
```python
|
||||
import os
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key=os.environ["STEP_API_KEY"],
|
||||
base_url=os.environ["STEP_BASE_URL"],
|
||||
)
|
||||
|
||||
completion = client.chat.completions.create(
|
||||
model="step-3.7-flash",
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What is in this picture?"},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {"url": "https://example.com/photo.jpg"},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
print(completion)
|
||||
```
|
||||
|
||||
## 6. Local Deployment
|
||||
|
||||
Step 3.7 Flash is optimized for local inference and supports industry-standard backends including vLLM, SGLang, Hugging Face Transformers and llama.cpp.
|
||||
|
||||
### 6.1 vLLM
|
||||
|
||||
We recommend using StepFun's prebuilt vLLM Docker image with Step 3.7 support.
|
||||
|
||||
1. Install vLLM.
|
||||
|
||||
```bash
|
||||
# via Docker
|
||||
docker pull vllm/vllm-openai:stepfun37
|
||||
```
|
||||
|
||||
2. Launch the server.
|
||||
|
||||
- For FP8 model
|
||||
```bash
|
||||
vllm serve <MODEL_PATH_OR_HF_ID> \
|
||||
--served-model-name step3p7-flash \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--disable-cascade-attn \
|
||||
--reasoning-parser step3p5 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser step3p5 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
|
||||
--trust-remote-code
|
||||
```
|
||||
- For BF16 model
|
||||
```bash
|
||||
vllm serve <MODEL_PATH_OR_HF_ID> \
|
||||
--served-model-name step3p7-flash-bf16 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--disable-cascade-attn \
|
||||
--reasoning-parser step3p5 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser step3p5 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
- For NVFP4 model
|
||||
Compared to standard precisions, running the FP4 quantized version requires modelopt activation and FP8 KV Cache alignment.
|
||||
```bash
|
||||
python3 -m vllm.entrypoints.openai.api_server \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--model stepfun-ai/Step-3.7-Flash-NVFP4 \
|
||||
--served-model-name step3p7 \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--enable-expert-parallel \
|
||||
--trust-remote-code \
|
||||
--quantization modelopt \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len 8192 \
|
||||
--reasoning-parser step3p5 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser step3p5 \
|
||||
--async-scheduling \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
|
||||
```
|
||||
|
||||
### 6.2 SGLang
|
||||
|
||||
1. Install SGLang.
|
||||
|
||||
```bash
|
||||
# via Docker
|
||||
docker pull lmsysorg/sglang:dev-step-3.7-flash
|
||||
|
||||
# or from source (pip)
|
||||
pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git"
|
||||
```
|
||||
|
||||
2. Launch the server.
|
||||
|
||||
> **Note:** For Blackwell GPUs, `--mm-attention-backend fa4` may be used.
|
||||
|
||||
- For BF16 model
|
||||
|
||||
```bash
|
||||
sglang serve --model-path stepfun-ai/Step-3.7-Flash \
|
||||
--tp 8 \
|
||||
--reasoning-parser step3p5 \
|
||||
--tool-call-parser step3p5 \
|
||||
--enable-multimodal \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--enable-multi-layer-eagle \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- For FP8 model
|
||||
|
||||
```bash
|
||||
sglang serve --model-path stepfun-ai/Step-3.7-Flash-FP8 \
|
||||
--tp 8 \
|
||||
--ep 4 \
|
||||
--reasoning-parser step3p5 \
|
||||
--tool-call-parser step3p5 \
|
||||
--enable-multimodal \
|
||||
--speculative-algorithm EAGLE \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--enable-multi-layer-eagle \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
- For NVFP4 model
|
||||
|
||||
```bash
|
||||
sglang serve --model-path stepfun-ai/Step-3.7-Flash-NVFP4 \
|
||||
--tp 4 --ep 4 \
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--quantization modelopt_fp4 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser step3p5 \
|
||||
--tool-call-parser step3p5 \
|
||||
--attention-backend trtllm_mha
|
||||
```
|
||||
|
||||
### 6.3 Transformers (Debug / Verification)
|
||||
|
||||
Use this snippet for quick functional verification. For high-throughput serving, use vLLM or SGLang.
|
||||
|
||||
> **Note:** Deployment of this model requires `transformers` 5.0 or later.
|
||||
|
||||
```python
|
||||
from transformers import AutoProcessor, AutoModelForCausalLM
|
||||
|
||||
MODEL_PATH = "<MODEL_PATH_OR_HF_ID>"
|
||||
|
||||
# 1. Setup
|
||||
processor = AutoProcessor.from_pretrained(MODEL_PATH, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_PATH,
|
||||
device_map="auto",
|
||||
dtype="auto",
|
||||
trust_remote_code=True
|
||||
)
|
||||
|
||||
# 2. Prepare Input
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image", "url": "https://example.com/photo.jpg"},
|
||||
{"type": "text", "text": "What is in this picture?"}
|
||||
]
|
||||
},
|
||||
]
|
||||
inputs = processor.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
# 3. Generate
|
||||
generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
|
||||
output_text = processor.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
|
||||
|
||||
print(output_text)
|
||||
```
|
||||
|
||||
### 6.4 llama.cpp
|
||||
|
||||
**System Requirements**
|
||||
|
||||
GGUF Model Weights:
|
||||
|
||||
| Component | Quantization | File Size |
|
||||
|---|---|---|
|
||||
| Language Model | Q4_K_S | 111.5 GB |
|
||||
| Language Model | IQ4_XS | 104.99 GB |
|
||||
| Language Model | Q3_K_L | 102.5 GB |
|
||||
| Multimodal Projector | FP16 | 3.97 GB |
|
||||
|
||||
- **Runtime Overhead:** ~7 GB
|
||||
- **Minimum unified memory / VRAM:** 120 GB (e.g., Mac Studio, NVIDIA DGX Station, AMD Ryzen AI Max+ 395)
|
||||
- **Recommended:** 128 GB unified memory
|
||||
|
||||
**Steps**
|
||||
|
||||
1. Use llama.cpp:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/stepfun-ai/llama.cpp.git
|
||||
cd llama.cpp
|
||||
git checkout -b step3.7 origin/step3.7
|
||||
```
|
||||
|
||||
2. Build llama.cpp on Mac:
|
||||
|
||||
```bash
|
||||
cmake -B build-macos -S . \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DBUILD_SHARED_LIBS=ON \
|
||||
-DLLAMA_BUILD_SERVER=ON \
|
||||
-DLLAMA_BUILD_TESTS=ON \
|
||||
-DGGML_METAL=ON \
|
||||
-DGGML_METAL_EMBED_LIBRARY=ON \
|
||||
-DGGML_BLAS=ON \
|
||||
-DGGML_BLAS_VENDOR=Apple \
|
||||
-DGGML_ACCELERATE=ON \
|
||||
-DGGML_NATIVE=ON
|
||||
cmake --build build-macos -j8
|
||||
```
|
||||
|
||||
3. Build llama.cpp on DGX-Spark:
|
||||
|
||||
```bash
|
||||
cmake -S . -B build-cuda \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_GRAPHS=ON \
|
||||
-DGGML_CUDA_FORCE_MMQ=ON \
|
||||
-DLLAMA_OPENSSL=OFF \
|
||||
-DLLAMA_BUILD_COMMON=ON \
|
||||
-DLLAMA_BUILD_TOOLS=ON \
|
||||
-DLLAMA_BUILD_SERVER=ON \
|
||||
-DLLAMA_BUILD_EXAMPLES=OFF \
|
||||
-DLLAMA_BUILD_TESTS=OFF
|
||||
cmake --build build-cuda -j8
|
||||
```
|
||||
|
||||
4. Build llama.cpp on AMD Windows:
|
||||
|
||||
```bash
|
||||
cmake -S . -B build-vulkan \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_VULKAN=ON \
|
||||
-DGGML_NATIVE=ON \
|
||||
-DLLAMA_BUILD_SERVER=ON \
|
||||
-DLLAMA_BUILD_UI=OFF \
|
||||
-DLLAMA_BUILD_TOOLS=ON
|
||||
cmake --build build-vulkan -j8
|
||||
```
|
||||
|
||||
5. Run with `llama-cli`:
|
||||
|
||||
```bash
|
||||
./llama-cli -m Step3.7_Q4_K_S.gguf -b 2048 -ub 2048 -fa on --temp 1.0 -p "What's your name?"
|
||||
```
|
||||
|
||||
6. Test performance with `llama-batched-bench`:
|
||||
|
||||
```bash
|
||||
./llama-batched-bench -m step3.7_Q4_K_S.gguf -c 32768 -b 2048 -ub 2048 -npp 0,2048,8192,16384,32768 -ntg 128 -npl 1
|
||||
```
|
||||
|
||||
## 7. Using Step 3.7 Flash on Agent Platforms
|
||||
|
||||
You can use Step 3.7 Flash on Agent platforms such as Hermes Agent, OpenClaw, Kilo Code, and more.
|
||||
|
||||
## 8. Getting in Touch
|
||||
|
||||
As we work to shape the future of AGI by expanding broad model capabilities, we want to ensure we are solving the right problems. We invite you to be part of this continuous feedback loop — your insights directly influence our priorities.
|
||||
|
||||
- **Join the Conversation:** Our [Discord](https://discord.gg/RcMJhNVAQc) community is the primary hub for brainstorming future architectures, proposing capabilities, and getting early access updates 🚀
|
||||
- **Report Friction:** Encountering limitations? You can open an issue or start a discussion on GitHub / HuggingFace, or flag it directly in our Discord support channels.
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is open-sourced under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目:`stepfun-ai/Step-3.7-Flash`
|
||||
- 原始仓库:https://github.com/stepfun-ai/Step-3.7-Flash
|
||||
- 导入方式:上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 314 KiB |
Reference in New Issue
Block a user