.. _api-reference: WebLLM API Reference ==================== The ``MLCEngine`` class is the core interface of WebLLM. It enables model loading, chat completions, embeddings, and other operations. Below, we document its methods, along with the associated configuration interfaces. Interfaces ---------- The following interfaces are used as parameters or configurations within ``MLCEngine`` methods. They are linked to their respective methods for reference. MLCEngineConfig ^^^^^^^^^^^^^^^ Optional configurations for ``CreateMLCEngine()`` and ``CreateWebWorkerMLCEngine()``. - **Fields**: - ``appConfig``: Configure the app, including the list of models and whether to use IndexedDB cache. - ``initProgressCallback``: A callback for showing model loading progress. - ``logitProcessorRegistry``: A registry for stateful logit processors (see ``webllm.LogitProcessor``). - **Usage**: - ``appConfig``: Contains application-specific settings, including: - Model configurations. - IndexedDB caching preferences. - ``initProgressCallback``: Allows developers to visualize model loading progress by implementing a callback. - ``logitProcessorRegistry``: A ``Map`` object for registering custom logit processors. Only applies to ``MLCEngine``. .. note:: All fields are optional, and ``logitProcessorRegistry`` is only used in ``MLCEngine``. Example: .. code-block:: typescript const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct", { appConfig: { /* app-specific config */ }, initProgressCallback: (progress) => console.log(progress), }); GenerationConfig ^^^^^^^^^^^^^^^^ Configurations for a single generation task, primarily used in chat completions. - **Fields**: - ``repetition_penalty``, ``ignore_eos``: Parameters specific to MLC models. - ``top_p``, ``temperature``, ``max_tokens``, ``stop``: Common parameters shared with OpenAI APIs. - ``frequency_penalty``, ``presence_penalty``: Tune repetition behavior following OpenAI semantics. - ``logit_bias``, ``n``, ``logprobs``, ``top_logprobs``: Advanced sampling controls. - ``response_format``, ``enable_thinking``, ``enable_latency_breakdown``: Additional OpenAI-style request features. - **Usage**: - Fields like ``repetition_penalty`` and ``ignore_eos`` give explicit control over repetition handling and whether the model stops at the EOS token, respectively. - Common parameters shared with OpenAI APIs (e.g., ``temperature``, ``top_p``) ensure compatibility while still falling back to the values configured during ``MLCEngine.reload()`` when omitted. - ``frequency_penalty`` and ``presence_penalty`` mirror OpenAI's bounds ``[-2, 2]``; providing only one will default the other to ``0``. - ``response_format`` (for JSON or other schema outputs), ``enable_thinking``, and ``enable_latency_breakdown`` pass through directly to the engine and surface enhanced telemetry or structured responses when the underlying model supports them. Example: .. code-block:: typescript const messages = [ { role: "system", content: "You are a helpful assistant." }, { role: "user", content: "Explain WebLLM." }, ]; const response = await engine.chatCompletion({ messages, top_p: 0.9, temperature: 0.8, max_tokens: 150, }); ChatConfig ^^^^^^^^^^ Model's baseline configuration loaded from ``mlc-chat-config.json`` when ``MLCEngine.reload()`` runs. ``ChatOptions`` (and therefore the ``chatOpts`` argument to ``reload``) can override any subset of these fields. - **Fields** (subset): - ``tokenizer_files``, ``tokenizer_info``: Files and parameters required to initialize the tokenizer. - ``conv_template``, ``conv_config``: Conversation templates that define prompts, separators, and role formatting. - ``context_window_size``, ``sliding_window_size``, ``attention_sink_size``: KV-cache and memory settings. - Default generation knobs such as ``repetition_penalty``, ``frequency_penalty``, ``presence_penalty``, ``top_p``, and ``temperature``. - **Usage**: - Loaded automatically for each model; provides defaults that ``GenerationConfig`` falls back to when fields are omitted. - Override selected values per model load by supplying ``chatOpts`` (``Partial``) to ``MLCEngine.reload()``. Example: .. code-block:: typescript await engine.reload("Llama-3.1-8B-Instruct", { temperature: 0.7, repetition_penalty: 1.1, context_window_size: 4096, }); ChatCompletionRequest ^^^^^^^^^^^^^^^^^^^^^ Defines the structure for chat completion requests. - **Base Interface**: ``ChatCompletionRequestBase`` - Contains parameters such as ``messages``, ``stream``, ``frequency_penalty``, and ``presence_penalty``. - **Sub-interfaces**: - ``ChatCompletionRequestNonStreaming``: For non-streaming completions. - ``ChatCompletionRequestStreaming``: For streaming completions. - **Usage**: - Combines settings from ``GenerationConfig`` and ``ChatCompletionRequestBase`` to provide complete control over chat behavior. - The ``stream`` parameter enables streaming responses, improving interactivity in conversational agents. - The ``logit_bias`` feature allows controlling token generation probabilities, providing a mechanism to restrict or encourage specific outputs. Example: .. code-block:: typescript const response = await engine.chatCompletion({ messages: [ { role: "user", content: "Tell me about WebLLM." }, ], stream: true, }); Model Loading ------------- ``MLCEngine.reload(modelId: string | string[], chatOpts?: ChatOptions | ChatOptions[]): Promise`` Loads the specified model(s) into the engine. Uses ``MLCEngineConfig`` during initialization. - Parameters: - ``modelId``: Identifier(s) for the model(s) to load. - ``chatOpts``: Configuration for generation (see ``ChatConfig``). Example: .. code-block:: typescript await engine.reload(["Llama-3.1-8B", "Gemma-2B"], [ { temperature: 0.7 }, { top_p: 0.9 }, ]); ``MLCEngine.unload(): Promise`` Unloads all loaded models and clears their associated configurations. Example: .. code-block:: typescript await engine.unload(); --- Chat Completions ---------------- ``MLCEngine.chat.completions.create(request: ChatCompletionRequest): Promise>`` Generates chat-based completions using a specified request configuration. - Parameters: - ``request``: A ``ChatCompletionRequest`` instance. Example: .. code-block:: typescript const response = await engine.chat.completions.create({ messages: [ { role: "system", content: "You are a helpful AI assistant." }, { role: "user", content: "What is WebLLM?" }, ], temperature: 0.8, stream: false, }); --- Utility Methods ^^^^^^^^^^^^^^^ ``MLCEngine.getMessage(modelId?: string): Promise`` Retrieves the current output message from the specified model. - Parameters: - ``modelId``: (Optional) Identifier of model to query. Omitting modelId only works when the engine currently has a single model loaded. ``MLCEngine.resetChat(keepStats?: boolean, modelId?: string): Promise`` Resets the chat history and optionally retains usage statistics. - Parameters: - ``keepStats``: (Optional) If true, retains usage statistics. - ``modelId``: (Optional) Identifier of the model to reset. Omitting modelId only works when the engine currently has a single model loaded. GPU Information ---------------- The following methods provide detailed information about the GPU used for WebLLM computations. ``MLCEngine.getGPUVendor(): Promise`` Retrieves the vendor name of the GPU used for computations. This is useful for understanding hardware capabilities during inference. - **Returns**: A string indicating the GPU vendor (e.g., "Intel", "NVIDIA"). Example: .. code-block:: typescript const gpuVendor = await engine.getGPUVendor(); console.log(``GPU Vendor: ${gpuVendor}``); ``MLCEngine.getMaxStorageBufferBindingSize(): Promise`` Returns the maximum storage buffer size supported by the GPU. This is important when working with larger models that require significant memory for processing. - **Returns**: A number representing the maximum size in bytes. Example: .. code-block:: typescript const maxBufferSize = await engine.getMaxStorageBufferBindingSize(); console.log(``Max Storage Buffer Binding Size: ${maxBufferSize}``);