wehub/huggingface--transformers.js
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Build documentation / build (push) Failing after 0s
Details
Unit tests / build (18) (push) Has been cancelled
Details
Unit tests / build (20) (push) Has been cancelled
Details
Unit tests / build (22) (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:44:39 +08:00
commit 311a3666c4
703 changed files with 71017 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/transformers/docs/source/tutorials/browser-extension.md
+3
View File
@@ -0,0 +1,3 @@
# Building a browser extension
_Full tutorial coming soon..._ In the meantime, check out the example application: https://github.com/huggingface/transformers.js-examples/tree/main/browser-extension
packages/transformers/docs/source/tutorials/electron.md
+3
View File
@@ -0,0 +1,3 @@
# Building an Electron application
_Full tutorial coming soon..._ In the meantime, check out the example application: https://github.com/huggingface/transformers.js-examples/tree/main/electron
packages/transformers/docs/source/tutorials/next-ai-sdk.md
+437
View File
@@ -0,0 +1,437 @@
# Building a Next.js AI Chatbot with Vercel AI SDK
In this tutorial, we'll build an in-browser AI chatbot using Next.js, Transformers.js, and the Vercel AI SDK v6. The chatbot runs entirely client-side with WebGPU acceleration — and supports tool calling with human approval.
Useful links:
- [Source code](https://github.com/huggingface/transformers.js-examples/tree/main/next-vercel-ai-sdk-v6-tool-calling)
- [`@browser-ai/transformers-js` docs](https://www.browser-ai.dev/docs/ai-sdk-v6/transformers-js)
- [Vercel AI SDK docs](https://ai-sdk.dev/)
## Prerequisites
- [Node.js](https://nodejs.org/en/) version 18+
- [npm](https://www.npmjs.com/) version 9+
- A browser with WebGPU support (Chrome 113+, Edge 113+, or Firefox/Safari with flags enabled)
## Step 1: Create the project
Create a new Next.js application:
```bash
npx create-next-app@latest next-ai-chatbot
cd next-ai-chatbot
```
Install the AI and Transformers.js dependencies:
```bash
npm install ai @ai-sdk/react @browser-ai/transformers-js @huggingface/transformers zod
```
## Step 2: Configure Next.js for browser inference
Transformers.js uses ONNX Runtime under the hood for both browser and server-side (Node.js) inference. In our case we only need the browser runtime so we can tell Next.js to exclude the Node.js-specific packages when bundling for the browser. Update `next.config.ts`
```typescript
import type { NextConfig } from "next";
const nextConfig: NextConfig = {
output: "export", // optional: export as a static site
turbopack: {},
webpack: (config) => {
config.resolve.alias = {
...config.resolve.alias,
sharp$: false,
"onnxruntime-node$": false,
};
return config;
},
};
export default nextConfig;
```
## Step 3: Create the Web Worker
Running model inference on the main thread would block the UI. The `@browser-ai/transformers-js` package provides a ready-made worker handler that handles all the complexity for you.
Create `src/app/worker.ts`:
```typescript
import { TransformersJSWorkerHandler } from "@browser-ai/transformers-js";
const handler = new TransformersJSWorkerHandler();
self.onmessage = (msg: MessageEvent) => {
handler.onmessage(msg);
};
```
That's it — the handler takes care of model loading, inference, streaming, and communication with the main thread.
## Step 4: Define the model configuration
Create `src/app/models.ts` to define which models are available. These are ONNX-format models from Hugging Face:
```typescript
import { WorkerLoadOptions } from "@browser-ai/transformers-js";
export interface ModelConfig extends Omit<WorkerLoadOptions, "modelId"> {
id: string;
name: string;
supportsWorker?: boolean;
}
export const MODELS: ModelConfig[] = [
{
id: "onnx-community/Qwen3-0.6B-ONNX",
name: "Qwen3 0.6B",
device: "webgpu",
dtype: "q4f16",
supportsWorker: true,
},
{
id: "onnx-community/granite-4.0-350m-ONNX-web",
name: "Granite 4.0 350M",
device: "webgpu",
dtype: "fp16",
supportsWorker: true,
},
];
```
<Tip>
For tool calling, use reasoning models like Qwen3 which handle multi-step reasoning well, or fine-tuned model specifically for tool-calling capabilities. The `supportsWorker` flag controls whether the model is loaded in a Web Worker for better performance.
</Tip>
## Step 5: Define tools
Create `src/app/tools.ts` with tools the model can call. Each tool uses [Zod](https://zod.dev/) for input validation:
```typescript
import { tool } from "ai";
import z from "zod";
export const createTools = () => ({
getCurrentTime: tool({
description: "Get the current date and time.",
inputSchema: z.object({}),
execute: async () => {
const now = new Date();
return {
timestamp: now.toISOString(),
date: now.toLocaleDateString("en-US", {
weekday: "long", year: "numeric", month: "long", day: "numeric",
}),
time: now.toLocaleTimeString("en-US", {
hour: "2-digit", minute: "2-digit", second: "2-digit", hour12: true,
}),
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
};
},
}),
randomNumber: tool({
description: "Generate a random integer between min and max (inclusive).",
inputSchema: z.object({
min: z.number().describe("The minimum value (inclusive)"),
max: z.number().describe("The maximum value (inclusive)"),
}),
execute: async ({ min, max }) => {
return Math.floor(Math.random() * (Math.floor(max) - Math.ceil(min) + 1)) + Math.ceil(min);
},
}),
getLocation: tool({
description: "Get the user's current geographic location.",
inputSchema: z.object({}),
needsApproval: true, // requires user confirmation before executing
execute: async () => {
return new Promise((resolve, reject) => {
navigator.geolocation.getCurrentPosition(
(pos) => resolve({
latitude: pos.coords.latitude,
longitude: pos.coords.longitude,
}),
(err) => reject(err.message),
);
});
},
}),
});
```
The `getLocation` tool uses `needsApproval: true`, which means the AI SDK will pause execution and wait for the user to approve or reject the tool call before running it.
## Step 6: Create the chat transport
The Vercel AI SDK's `useChat` hook needs a [transport](https://ai-sdk.dev/docs/ai-sdk-ui/transport) that handles communication with the model. For client-side inference, we implement a custom `ChatTransport`.
Create `src/app/chat-transport.ts`:
```typescript
import {
ChatTransport, UIMessageChunk, streamText,
convertToModelMessages, ChatRequestOptions,
createUIMessageStream, stepCountIs,
} from "ai";
import {
TransformersJSLanguageModel,
TransformersUIMessage,
transformersJS,
} from "@browser-ai/transformers-js";
import { MODELS } from "./models";
import { createTools } from "./tools";
export class TransformersChatTransport
implements ChatTransport<TransformersUIMessage>
{
private model: TransformersJSLanguageModel;
private tools: ReturnType<typeof createTools>;
constructor() {
const config = MODELS[0];
this.model = transformersJS(config.id, {
device: config.device,
dtype: config.dtype,
...(config.supportsWorker
? {
worker: new Worker(new URL("./worker.ts", import.meta.url), {
type: "module",
}),
}
: {}),
});
this.tools = createTools();
}
async sendMessages(
options: {
chatId: string;
messages: TransformersUIMessage[];
abortSignal: AbortSignal | undefined;
} & {
trigger: "submit-message" | "submit-tool-result" | "regenerate-message";
messageId: string | undefined;
} & ChatRequestOptions,
): Promise<ReadableStream<UIMessageChunk>> {
const { messages, abortSignal } = options;
const prompt = await convertToModelMessages(messages);
return createUIMessageStream<TransformersUIMessage>({
execute: async ({ writer }) => {
// Track download progress if the model hasn't been downloaded yet
let downloadProgressId: string | undefined;
const availability = await this.model.availability();
if (availability !== "available") {
await this.model.createSessionWithProgress(
(progress: number) => {
const percent = Math.round(progress * 100);
if (progress >= 1) {
if (downloadProgressId) {
writer.write({
type: "data-modelDownloadProgress",
id: downloadProgressId,
data: {
status: "complete", progress: 100,
message: "Model ready!",
},
});
}
return;
}
if (!downloadProgressId) {
downloadProgressId = `download-${Date.now()}`;
}
writer.write({
type: "data-modelDownloadProgress",
id: downloadProgressId,
data: {
status: "downloading", progress: percent,
message: `Downloading model... ${percent}%`,
},
});
},
);
}
const result = streamText({
model: this.model,
tools: this.tools,
stopWhen: stepCountIs(5),
messages: prompt,
abortSignal,
});
writer.merge(result.toUIMessageStream({ sendStart: false }));
},
});
}
async reconnectToStream(): Promise<ReadableStream<UIMessageChunk> | null> {
return null;
}
}
```
Key parts of the transport:
- **Availability check**: Determines if the model needs downloading before inference.
- **Progress streaming**: Sends download progress as custom data parts (`data-modelDownloadProgress`) that the UI can render as a progress bar.
- **Tool support**: Passes the tools to `streamText()` so the model can call them.
- **Step limiting**: `stopWhen: stepCountIs(5)` prevents infinite tool-calling loops.
## Step 7: Build the chat UI
Now wire everything together in your page component. Create `src/app/page.tsx`:
```tsx
"use client";
import { useState } from "react";
import { useChat } from "@ai-sdk/react";
import { TransformersUIMessage } from "@browser-ai/transformers-js";
import { lastAssistantMessageIsCompleteWithApprovalResponses } from "ai";
import { TransformersChatTransport } from "./chat-transport";
export default function ChatPage() {
const [input, setInput] = useState("");
const {
messages,
sendMessage,
status,
stop,
addToolApprovalResponse,
} = useChat<TransformersUIMessage>({
transport: new TransformersChatTransport(),
experimental_throttle: 75,
// Automatically resumes after tool approval responses are submitted
sendAutomaticallyWhen: lastAssistantMessageIsCompleteWithApprovalResponses,
});
const handleSubmit = (e: React.FormEvent) => {
e.preventDefault();
if (input.trim() && status === "ready") {
sendMessage({ text: input });
setInput("");
}
};
return (
<div style={{ maxWidth: 600, margin: "0 auto", padding: 24 }}>
<h1>AI Chatbot</h1>
<div>
{messages.map((message) => (
<div key={message.id} style={{ marginBottom: 16 }}>
<strong>{message.role === "user" ? "You" : "Assistant"}:</strong>
{message.parts.map((part, i) => {
switch (part.type) {
case "text":
return <p key={i}>{part.text}</p>;
case "data-modelDownloadProgress":
if (!part.data.message) return null;
return (
<div key={i}>
<p>{part.data.message}</p>
{part.data.status === "downloading" && (
<progress value={part.data.progress} max={100} />
)}
</div>
);
default:
// Handle tool parts
if (part.type.startsWith("tool-") && "state" in part) {
if (
part.state === "approval-requested" &&
"approval" in part
) {
return (
<div key={i} style={{ border: "1px solid #ccc", padding: 8 }}>
<p>Tool <strong>{part.type.replace("tool-", "")}</strong> wants to run.</p>
<button onClick={() =>
addToolApprovalResponse({ id: part.approval!.id, approved: true })
}>
Approve
</button>
<button onClick={() =>
addToolApprovalResponse({
id: part.approval!.id, approved: false,
reason: "User denied",
})
}>
Deny
</button>
</div>
);
}
if ("output" in part && part.output) {
return (
<pre key={i} style={{ background: "#f5f5f5", padding: 8 }}>
{JSON.stringify(part.output, null, 2)}
</pre>
);
}
}
return null;
}
})}
</div>
))}
</div>
{status === "submitted" && <p><em>Thinking...</em></p>}
<form onSubmit={handleSubmit}>
<input
value={input}
onChange={(e) => setInput(e.target.value)}
placeholder="Ask something..."
style={{ width: "100%", padding: 8 }}
/>
<div style={{ marginTop: 8 }}>
{status === "streaming" ? (
<button type="button" onClick={stop}>Stop</button>
) : (
<button type="submit" disabled={!input.trim()}>Send</button>
)}
</div>
</form>
</div>
);
}
```
The component renders message parts based on their `type`:
- `text` &mdash; standard text output from the model.
- `data-modelDownloadProgress` &mdash; custom data parts sent by the transport during model download.
- `tool-*` &mdash; tool call parts with states like `approval-requested`, `output-available`, etc.
The `sendAutomaticallyWhen: lastAssistantMessageIsCompleteWithApprovalResponses` option tells `useChat` to automatically resume generation after the user responds to a tool approval request.
## Step 8: Run the application
Start the development server:
```bash
npm run dev
```
Open your browser and navigate to the URL shown in the terminal. The first time you send a message, the model will be downloaded and cached in the browser. Subsequent visits will load the cached model.
Try prompts like:
- "What time is it?"
- "Generate a random number between 1 and 100"
- "Where am I located?" (this will trigger a tool approval prompt)
## Next steps
- Add more models and a model selector &mdash; see the [full example source](https://github.com/huggingface/transformers.js-examples/tree/main/next-vercel-ai-sdk-v6-tool-calling) for a multi-model implementation with Zustand state management.
- Add a browser compatibility check with `doesBrowserSupportTransformersJS()` and fall back to a server-side route if WebGPU is unavailable.
- Explore the [Vercel AI SDK agents documentation](https://ai-sdk.dev/docs/agents/overview) for more complex agent patterns.
- See the [Vercel AI SDK guide](../integrations/vercel-ai-sdk) for a reference of all supported features (embeddings, vision, transcription, etc.).
packages/transformers/docs/source/tutorials/next.md
+432
View File
@@ -0,0 +1,432 @@
# Building a Next.js application
In this tutorial, we'll build a simple Next.js application that performs sentiment analysis using Transformers.js!
Since Transformers.js can run in the browser or in Node.js, you can choose whether you want to perform inference [client-side](#client-side-inference) or [server-side](#server-side-inference) (we'll show you how to do both). In either case, we will be developing with the new [App Router](https://nextjs.org/docs/app) paradigm.
The final product will look something like this:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/next-demo.gif)
Useful links:
- Demo site: [client-side](https://huggingface.co/spaces/Xenova/next-example-app) or [server-side](https://huggingface.co/spaces/Xenova/next-server-example-app)
- Source code: [client-side](https://github.com/huggingface/transformers.js/tree/main/examples/next-client) or [server-side](https://github.com/huggingface/transformers.js/tree/main/examples/next-server)
## Prerequisites
- [Node.js](https://nodejs.org/en/) version 18+
- [npm](https://www.npmjs.com/) version 9+
## Client-side inference
### Step 1: Initialise the project
Start by creating a new Next.js application using `create-next-app`:
```bash
npx create-next-app@latest
```
On installation, you'll see various prompts. For this demo, we'll be selecting those shown below in bold:
<pre>√ What is your project named? ... next
√ Would you like to use TypeScript? ... <b>No</b> / Yes
√ Would you like to use ESLint? ... No / <b>Yes</b>
√ Would you like to use Tailwind CSS? ... No / <b>Yes</b>
√ Would you like to use `src/` directory? ... No / <b>Yes</b>
√ Would you like to use App Router? (recommended) ... No / <b>Yes</b>
√ Would you like to customize the default import alias? ... <b>No</b> / Yes
</pre>
### Step 2: Install and configure Transformers.js
You can install Transformers.js from [NPM](https://www.npmjs.com/package/@huggingface/transformers) with the following command:
```bash
npm i @huggingface/transformers
```
We also need to update the `next.config.js` file to ignore node-specific modules when bundling for the browser:
```js
/** @type {import('next').NextConfig} */
const nextConfig = {
// (Optional) Export as a static site
// See https://nextjs.org/docs/pages/building-your-application/deploying/static-exports#configuration
output: "export", // Feel free to modify/remove this option
// Override the default webpack configuration
webpack: (config) => {
// See https://webpack.js.org/configuration/resolve/#resolvealias
config.resolve.alias = {
...config.resolve.alias,
sharp$: false,
"onnxruntime-node$": false,
};
return config;
},
};
module.exports = nextConfig;
```
Next, we'll create a new [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) script where we'll place all ML-related code. This is to ensure that the main thread is not blocked while the model is loading and performing inference. For this application, we'll be using [`Xenova/distilbert-base-uncased-finetuned-sst-2-english`](https://huggingface.co/Xenova/distilbert-base-uncased-finetuned-sst-2-english), a ~67M parameter model finetuned on the [Stanford Sentiment Treebank](https://huggingface.co/datasets/sst) dataset. Add the following code to `./src/app/worker.js`:
```js
import { pipeline, env } from "@huggingface/transformers";
// Skip local model check
env.allowLocalModels = false;
// Use the Singleton pattern to enable lazy construction of the pipeline.
class PipelineSingleton {
static task = "text-classification";
static model = "Xenova/distilbert-base-uncased-finetuned-sst-2-english";
static instance = null;
static async getInstance(progress_callback = null) {
if (this.instance === null) {
this.instance = pipeline(this.task, this.model, { progress_callback });
}
return this.instance;
}
}
// Listen for messages from the main thread
self.addEventListener("message", async (event) => {
// Retrieve the classification pipeline. When called for the first time,
// this will load the pipeline and save it for future use.
let classifier = await PipelineSingleton.getInstance((x) => {
// We also add a progress callback to the pipeline so that we can
// track model loading.
self.postMessage(x);
});
// Actually perform the classification
let output = await classifier(event.data.text);
// Send the output back to the main thread
self.postMessage({
status: "complete",
output: output,
});
});
```
### Step 3: Design the user interface
We'll now modify the default `./src/app/page.js` file so that it connects to our worker thread. Since we'll only be performing in-browser inference, we can opt-in to Client components using the [`'use client'` directive](https://nextjs.org/docs/getting-started/react-essentials#the-use-client-directive).
```jsx
'use client'
import { useState, useEffect, useRef, useCallback } from 'react'
export default function Home() {
/* TODO: Add state variables */
// Create a reference to the worker object.
const worker = useRef(null);
// We use the `useEffect` hook to set up the worker as soon as the `App` component is mounted.
useEffect(() => {
if (!worker.current) {
// Create the worker if it does not yet exist.
worker.current = new Worker(new URL('./worker.js', import.meta.url), {
type: 'module'
});
}
// Create a callback function for messages from the worker thread.
const onMessageReceived = (e) => { /* TODO: See below */};
// Attach the callback function as an event listener.
worker.current.addEventListener('message', onMessageReceived);
// Define a cleanup function for when the component is unmounted.
return () => worker.current.removeEventListener('message', onMessageReceived);
});
const classify = useCallback((text) => {
if (worker.current) {
worker.current.postMessage({ text });
}
}, []);
return ( /* TODO: See below */ )
}
```
Initialise the following state variables at the beginning of the `Home` component:
```jsx
// Keep track of the classification result and the model loading status.
const [result, setResult] = useState(null);
const [ready, setReady] = useState(null);
```
and fill in the `onMessageReceived` function to update these variables when the worker thread sends a message:
```js
const onMessageReceived = (e) => {
switch (e.data.status) {
case "initiate":
setReady(false);
break;
case "ready":
setReady(true);
break;
case "complete":
setResult(e.data.output[0]);
break;
}
};
```
Finally, we can add a simple UI to the `Home` component, consisting of an input textbox and a preformatted text element to display the classification result:
```jsx
<main className="flex min-h-screen flex-col items-center justify-center p-12">
<h1 className="text-5xl font-bold mb-2 text-center">Transformers.js</h1>
<h2 className="text-2xl mb-4 text-center">Next.js template</h2>
<input
className="w-full max-w-xs p-2 border border-gray-300 rounded mb-4"
type="text"
placeholder="Enter text here"
onInput={(e) => {
classify(e.target.value);
}}
/>
{ready !== null && (
<pre className="bg-gray-100 p-2 rounded">
{!ready || !result ? "Loading..." : JSON.stringify(result, null, 2)}
</pre>
)}
</main>
```
You can now run your application using the following command:
```bash
npm run dev
```
Visit the URL shown in the terminal (e.g., [http://localhost:3000/](http://localhost:3000/)) to see your application in action!
### (Optional) Step 4: Build and deploy
To build your application, simply run:
```bash
npm run build
```
This will bundle your application and output the static files to the `out` folder.
For this demo, we will deploy our application as a static [Hugging Face Space](https://huggingface.co/docs/hub/spaces), but you can deploy it anywhere you like! If you haven't already, you can create a free Hugging Face account [here](https://huggingface.co/join).
1. Visit [https://huggingface.co/new-space](https://huggingface.co/new-space) and fill in the form. Remember to select "Static" as the space type.
2. Click the "Create space" button at the bottom of the page.
3. Go to "Files" &rarr; "Add file" &rarr; "Upload files". Drag the files from the `out` folder into the upload box and click "Upload". After they have uploaded, scroll down to the button and click "Commit changes to main".
**That's it!** Your application should now be live at `https://huggingface.co/spaces/<your-username>/<your-space-name>`!
## Server-side inference
While there are many different ways to perform server-side inference, the simplest (which we will discuss in this tutorial) is using the new [Route Handlers](https://nextjs.org/docs/app/building-your-application/routing/router-handlers) feature.
### Step 1: Initialise the project
Start by creating a new Next.js application using `create-next-app`:
```bash
npx create-next-app@latest
```
On installation, you'll see various prompts. For this demo, we'll be selecting those shown below in bold:
<pre>√ What is your project named? ... next
√ Would you like to use TypeScript? ... <b>No</b> / Yes
√ Would you like to use ESLint? ... No / <b>Yes</b>
√ Would you like to use Tailwind CSS? ... No / <b>Yes</b>
√ Would you like to use `src/` directory? ... No / <b>Yes</b>
√ Would you like to use App Router? (recommended) ... No / <b>Yes</b>
√ Would you like to customize the default import alias? ... <b>No</b> / Yes
</pre>
### Step 2: Install and configure Transformers.js
You can install Transformers.js from [NPM](https://www.npmjs.com/package/@huggingface/transformers) with the following command:
```bash
npm i @huggingface/transformers
```
We also need to update the `next.config.js` file to prevent Webpack from bundling certain packages:
```js
/** @type {import('next').NextConfig} */
const nextConfig = {
// (Optional) Export as a standalone site
// See https://nextjs.org/docs/pages/api-reference/next-config-js/output#automatically-copying-traced-files
output: "standalone", // Feel free to modify/remove this option
// Indicate that these packages should not be bundled by webpack
experimental: {
serverComponentsExternalPackages: ["sharp", "onnxruntime-node"],
},
};
module.exports = nextConfig;
```
Next, let's set up our Route Handler. We can do this by creating two files in a new `./src/app/classify/` directory:
1. `pipeline.js` - to handle the construction of our pipeline.
```js
import { pipeline } from "@huggingface/transformers";
// Use the Singleton pattern to enable lazy construction of the pipeline.
// NOTE: We wrap the class in a function to prevent code duplication (see below).
const P = () =>
class PipelineSingleton {
static task = "text-classification";
static model = "Xenova/distilbert-base-uncased-finetuned-sst-2-english";
static instance = null;
static async getInstance(progress_callback = null) {
if (this.instance === null) {
this.instance = pipeline(this.task, this.model, {
progress_callback,
});
}
return this.instance;
}
};
let PipelineSingleton;
if (process.env.NODE_ENV !== "production") {
// When running in development mode, attach the pipeline to the
// global object so that it's preserved between hot reloads.
// For more information, see https://vercel.com/guides/nextjs-prisma-postgres
if (!global.PipelineSingleton) {
global.PipelineSingleton = P();
}
PipelineSingleton = global.PipelineSingleton;
} else {
PipelineSingleton = P();
}
export default PipelineSingleton;
```
2. `route.js` - to process requests made to the `/classify` route.
```js
import { NextResponse } from "next/server";
import PipelineSingleton from "./pipeline.js";
export async function GET(request) {
const text = request.nextUrl.searchParams.get("text");
if (!text) {
return NextResponse.json(
{
error: "Missing text parameter",
},
{ status: 400 },
);
}
// Get the classification pipeline. When called for the first time,
// this will load the pipeline and cache it for future use.
const classifier = await PipelineSingleton.getInstance();
// Actually perform the classification
const result = await classifier(text);
return NextResponse.json(result);
}
```
### Step 3: Design the user interface
We'll now modify the default `./src/app/page.js` file to make requests to our newly-created Route Handler.
```jsx
"use client";
import { useState } from "react";
export default function Home() {
// Keep track of the classification result and the model loading status.
const [result, setResult] = useState(null);
const [ready, setReady] = useState(null);
const classify = async (text) => {
if (!text) return;
if (ready === null) setReady(false);
// Make a request to the /classify route on the server.
const result = await fetch(`/classify?text=${encodeURIComponent(text)}`);
// If this is the first time we've made a request, set the ready flag.
if (!ready) setReady(true);
const json = await result.json();
setResult(json);
};
return (
<main className="flex min-h-screen flex-col items-center justify-center p-12">
<h1 className="text-5xl font-bold mb-2 text-center">Transformers.js</h1>
<h2 className="text-2xl mb-4 text-center">
Next.js template (server-side)
</h2>
<input
type="text"
className="w-full max-w-xs p-2 border border-gray-300 rounded mb-4"
placeholder="Enter text here"
onInput={(e) => {
classify(e.target.value);
}}
/>
{ready !== null && (
<pre className="bg-gray-100 p-2 rounded">
{!ready || !result ? "Loading..." : JSON.stringify(result, null, 2)}
</pre>
)}
</main>
);
}
```
You can now run your application using the following command:
```bash
npm run dev
```
Visit the URL shown in the terminal (e.g., [http://localhost:3000/](http://localhost:3000/)) to see your application in action!
### (Optional) Step 4: Build and deploy
For this demo, we will build and deploy our application to [Hugging Face Spaces](https://huggingface.co/docs/hub/spaces). If you haven't already, you can create a free Hugging Face account [here](https://huggingface.co/join).
1. Create a new `Dockerfile` in your project's root folder. You can use our [example Dockerfile](https://github.com/huggingface/transformers.js/blob/main/examples/next-server/Dockerfile) as a template.
2. Visit [https://huggingface.co/new-space](https://huggingface.co/new-space) and fill in the form. Remember to select "Docker" as the space type (you can choose the "Blank" Docker template).
3. Click the "Create space" button at the bottom of the page.
4. Go to "Files" &rarr; "Add file" &rarr; "Upload files". Drag the files from your project folder (excluding `node_modules` and `.next`, if present) into the upload box and click "Upload". After they have uploaded, scroll down to the button and click "Commit changes to main".
5. Add the following lines to the top of your `README.md`:
```
---
title: Next Server Example App
emoji: 🔥
colorFrom: yellow
colorTo: red
sdk: docker
pinned: false
app_port: 3000
---
```
**That's it!** Your application should now be live at `https://huggingface.co/spaces/<your-username>/<your-space-name>`!
packages/transformers/docs/source/tutorials/node.md
+214
View File
@@ -0,0 +1,214 @@
# Server-side Inference in Node.js
Although Transformers.js was originally designed to be used in the browser, it's also able to run inference on the server. In this tutorial, we will design a simple Node.js API that uses Transformers.js for sentiment analysis.
We'll also show you how to use the library in both CommonJS and ECMAScript modules, so you can choose the module system that works best for your project:
- [ECMAScript modules (ESM)](#ecmascript-modules-esm) - The official standard format
to package JavaScript code for reuse. It's the default module system in modern
browsers, with modules imported using `import` and exported using `export`.
Fortunately, starting with version 13.2.0, Node.js has stable support of ES modules.
- [CommonJS](#commonjs) - The default module system in Node.js. In this system,
modules are imported using `require()` and exported using `module.exports`.
<Tip>
Although you can always use the [Python library](https://github.com/huggingface/transformers) for server-side inference, using Transformers.js means that you can write all of your code in JavaScript (instead of having to set up and communicate with a separate Python process).
</Tip>
**Useful links:**
- Source code ([ESM](https://github.com/huggingface/transformers.js/tree/main/examples/node/esm/app.js) or [CommonJS](https://github.com/huggingface/transformers.js/tree/main/examples/node/commonjs/app.js))
- [Documentation](https://huggingface.co/docs/transformers.js)
## Prerequisites
- [Node.js](https://nodejs.org/en/) version 18+
- [npm](https://www.npmjs.com/) version 9+
## Getting started
Let's start by creating a new Node.js project and installing Transformers.js via [NPM](https://www.npmjs.com/package/@huggingface/transformers):
```bash
npm init -y
npm i @huggingface/transformers
```
Next, create a new file called `app.js`, which will be the entry point for our application. Depending on whether you're using [ECMAScript modules](#ecmascript-modules-esm) or [CommonJS](#commonjs), you will need to do some things differently (see below).
We'll also create a helper class called `MyClassificationPipeline` control the loading of the pipeline. It uses the [singleton pattern](https://en.wikipedia.org/wiki/Singleton_pattern) to lazily create a single instance of the pipeline when `getInstance` is first called, and uses this pipeline for all subsequent calls:
### ECMAScript modules (ESM)
To indicate that your project uses ECMAScript modules, you need to add `"type": "module"` to your `package.json`:
```json
{
...
"type": "module",
...
}
```
Next, you will need to add the following imports to the top of `app.js`:
```javascript
import http from "http";
import querystring from "querystring";
import url from "url";
```
Following that, let's import Transformers.js and define the `MyClassificationPipeline` class.
```javascript
import { pipeline, env } from "@huggingface/transformers";
class MyClassificationPipeline {
static task = "text-classification";
static model = "Xenova/distilbert-base-uncased-finetuned-sst-2-english";
static instance = null;
static async getInstance(progress_callback = null) {
if (this.instance === null) {
// NOTE: Uncomment this to change the cache directory
// env.cacheDir = './.cache';
this.instance = pipeline(this.task, this.model, { progress_callback });
}
return this.instance;
}
}
```
### CommonJS
Start by adding the following imports to the top of `app.js`:
```javascript
const http = require("http");
const querystring = require("querystring");
const url = require("url");
```
Following that, let's import Transformers.js and define the `MyClassificationPipeline` class. Since Transformers.js is an ESM module, we will need to dynamically import the library using the [`import()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import) function:
```javascript
class MyClassificationPipeline {
static task = "text-classification";
static model = "Xenova/distilbert-base-uncased-finetuned-sst-2-english";
static instance = null;
static async getInstance(progress_callback = null) {
if (this.instance === null) {
// Dynamically import the Transformers.js library
let { pipeline, env } = await import("@huggingface/transformers");
// NOTE: Uncomment this to change the cache directory
// env.cacheDir = './.cache';
this.instance = pipeline(this.task, this.model, { progress_callback });
}
return this.instance;
}
}
```
## Creating a basic HTTP server
Next, let's create a basic server with the built-in [HTTP](https://nodejs.org/api/http.html#http) module. We will listen for requests made to the server (using the `/classify` endpoint), extract the `text` query parameter, and run this through the pipeline.
```javascript
// Define the HTTP server
const server = http.createServer();
const hostname = "127.0.0.1";
const port = 3000;
// Listen for requests made to the server
server.on("request", async (req, res) => {
// Parse the request URL
const parsedUrl = url.parse(req.url);
// Extract the query parameters
const { text } = querystring.parse(parsedUrl.query);
// Set the response headers
res.setHeader("Content-Type", "application/json");
let response;
if (parsedUrl.pathname === "/classify" && text) {
const classifier = await MyClassificationPipeline.getInstance();
response = await classifier(text);
res.statusCode = 200;
} else {
response = { error: "Bad request" };
res.statusCode = 400;
}
// Send the JSON response
res.end(JSON.stringify(response));
});
server.listen(port, hostname, () => {
console.log(`Server running at http://${hostname}:${port}/`);
});
```
<Tip>
Since we use lazy loading, the first request made to the server will also be responsible for loading the pipeline. If you would like to begin loading the pipeline as soon as the server starts running, you can add the following line of code after defining `MyClassificationPipeline`:
```javascript
MyClassificationPipeline.getInstance();
```
</Tip>
To start the server, run the following command:
```bash
node app.js
```
The server should be live at http://127.0.0.1:3000/, which you can visit in your web browser. You should see the following message:
```json
{ "error": "Bad request" }
```
This is because we aren't targeting the `/classify` endpoint with a valid `text` query parameter. Let's try again, this time with a valid request. For example, you can visit http://127.0.0.1:3000/classify?text=I%20love%20Transformers.js and you should see:
```json
[{ "label": "POSITIVE", "score": 0.9996721148490906 }]
```
Great! We've successfully created a basic HTTP server that uses Transformers.js to classify text.
## (Optional) Customization
### Model caching
By default, the first time you run the application, it will download the model files and cache them on your file system (in `./node_modules/@huggingface/transformers/.cache/`). All subsequent requests will then use this model. You can change the location of the cache by setting `env.cacheDir`. For example, to cache the model in the `.cache` directory in the current working directory, you can add:
```javascript
env.cacheDir = "./.cache";
```
### Use local models
If you want to use local model files, you can set `env.localModelPath` as follows:
```javascript
// Specify a custom location for models (defaults to '/models/').
env.localModelPath = "/path/to/models/";
```
You can also disable loading of remote models by setting `env.allowRemoteModels` to `false`:
```javascript
// Disable the loading of remote models from the Hugging Face Hub:
env.allowRemoteModels = false;
```
packages/transformers/docs/source/tutorials/react.md
+524
View File
@@ -0,0 +1,524 @@
# Building a React application
In this tutorial, we'll be building a simple React application that performs multilingual translation using Transformers.js! The final product will look something like this:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/react-translator-demo.gif)
Useful links:
- [Demo site](https://huggingface.co/spaces/Xenova/react-translator)
- [Source code](https://github.com/huggingface/transformers.js-examples/tree/main/react-translator)
## Prerequisites
- [Node.js](https://nodejs.org/en/) version 18+
- [npm](https://www.npmjs.com/) version 9+
## Step 1: Initialise the project
For this tutorial, we will use [Vite](https://vitejs.dev/) to initialise our project. Vite is a build tool that allows us to quickly set up a React application with minimal configuration. Run the following command in your terminal:
```bash
npm create vite@latest react-translator -- --template react
```
If prompted to install `create-vite`, type <kbd>y</kbd> and press <kbd>Enter</kbd>.
Next, enter the project directory and install the necessary development dependencies:
```bash
cd react-translator
npm install
```
To test that our application is working, we can run the following command:
```bash
npm run dev
```
Visiting the URL shown in the terminal (e.g., [http://localhost:5173/](http://localhost:5173/)) should show the default "React + Vite" landing page.
You can stop the development server by pressing <kbd>Ctrl</kbd> + <kbd>C</kbd> in the terminal.
## Step 2: Install and configure Transformers.js
Now we get to the fun part: adding machine learning to our application! First, install Transformers.js from [NPM](https://www.npmjs.com/package/@huggingface/transformers) with the following command:
```bash
npm install @huggingface/transformers
```
For this application, we will use the [Xenova/nllb-200-distilled-600M](https://huggingface.co/Xenova/nllb-200-distilled-600M) model, which can perform multilingual translation among 200 languages. Before we start, there are 2 things we need to take note of:
1. ML inference can be quite computationally intensive, so it's better to load and run the models in a separate thread from the main (UI) thread.
2. Since the model is quite large (>1 GB), we don't want to download it until the user clicks the "Translate" button.
We can achieve both of these goals by using a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) and some [React hooks](https://react.dev/reference/react).
1. Create a file called `worker.js` in the `src` directory. This script will do all the heavy-lifting for us, including loading and running of the translation pipeline. To ensure the model is only loaded once, we will create the `MyTranslationPipeline` class which use the [singleton pattern](https://en.wikipedia.org/wiki/Singleton_pattern) to lazily create a single instance of the pipeline when `getInstance` is first called, and use this pipeline for all subsequent calls:
```javascript
import { pipeline, TextStreamer } from "@huggingface/transformers";
class MyTranslationPipeline {
static task = "translation";
static model = "Xenova/nllb-200-distilled-600M";
static instance = null;
static async getInstance(progress_callback = null) {
this.instance ??= pipeline(this.task, this.model, { progress_callback });
return this.instance;
}
}
```
2. Modify `App.jsx` in the `src` directory. This file is automatically created when initializing our React project, and will contain some boilerplate code. Inside the `App` function, let's create the web worker and store a reference to it using the `useRef` hook:
```jsx
// Remember to import the relevant hooks
import { useEffect, useRef, useState } from 'react'
import './App.css'
function App() {
// Create a reference to the worker object.
const worker = useRef(null);
// We use the `useEffect` hook to setup the worker as soon as the `App` component is mounted.
useEffect(() => {
// Create the worker if it does not yet exist.
worker.current ??= new Worker(new URL('./worker.js', import.meta.url), {
type: 'module'
});
// Create a callback function for messages from the worker thread.
const onMessageReceived = (e) => {
// TODO: Will fill in later
};
// Attach the callback function as an event listener.
worker.current.addEventListener('message', onMessageReceived);
// Define a cleanup function for when the component is unmounted.
return () => worker.current.removeEventListener('message', onMessageReceived);
});
return (
// TODO: Rest of our app goes here...
)
}
export default App
```
## Step 3: Design the user interface
<Tip>
We recommend starting the development server again with `npm run dev`
(if not already running) so that you can see your changes in real-time.
</Tip>
First, let's define our components. Create a folder called `components` in the `src` directory, and create the following files:
1. `LanguageSelector.jsx`: This component will allow the user to select the input and output languages. Check out the full list of languages [here](https://github.com/huggingface/transformers.js-examples/tree/main/react-translator/src/components/LanguageSelector.jsx).
```jsx
const LANGUAGES = {
"Acehnese (Arabic script)": "ace_Arab",
"Acehnese (Latin script)": "ace_Latn",
"Afrikaans": "afr_Latn",
...
"Zulu": "zul_Latn",
}
export default function LanguageSelector({ type, onChange, defaultLanguage }) {
return (
<div className='language-selector'>
<label>{type}: </label>
<select onChange={onChange} defaultValue={defaultLanguage}>
{Object.entries(LANGUAGES).map(([key, value]) => {
return <option key={key} value={value}>{key}</option>
})}
</select>
</div>
)
}
```
2. `Progress.jsx`: This component will display the progress for downloading each model file.
```jsx
export default function Progress({ text, percentage }) {
percentage = percentage ?? 0;
return (
<div className="progress-container">
<div className="progress-bar" style={{ width: `${percentage}%` }}>
{text} ({`${percentage.toFixed(2)}%`})
</div>
</div>
);
}
```
We can now use these components in `App.jsx` by adding these imports to the top of the file:
```jsx
import LanguageSelector from "./components/LanguageSelector";
import Progress from "./components/Progress";
```
Let's also add some state variables to keep track of a few things in our application, like model loading, languages, input text, and output text. Add the following code to the beginning of the `App` function in `src/App.jsx`:
```jsx
function App() {
// Model loading
const [ready, setReady] = useState(null);
const [disabled, setDisabled] = useState(false);
const [progressItems, setProgressItems] = useState([]);
// Inputs and outputs
const [input, setInput] = useState("I love walking my dog.");
const [sourceLanguage, setSourceLanguage] = useState("eng_Latn");
const [targetLanguage, setTargetLanguage] = useState("fra_Latn");
const [output, setOutput] = useState("");
// rest of the code...
}
```
Next, we can add our custom components to the main `App` component. We will also add two `textarea` elements for input and output text, and a `button` to trigger the translation. Modify the `return` statement to look like this:
```jsx
return (
<>
<h1>Transformers.js</h1>
<h2>ML-powered multilingual translation in React!</h2>
<div className="container">
<div className="language-container">
<LanguageSelector
type={"Source"}
defaultLanguage={"eng_Latn"}
onChange={(x) => setSourceLanguage(x.target.value)}
/>
<LanguageSelector
type={"Target"}
defaultLanguage={"fra_Latn"}
onChange={(x) => setTargetLanguage(x.target.value)}
/>
</div>
<div className="textbox-container">
<textarea
value={input}
rows={3}
onChange={(e) => setInput(e.target.value)}
></textarea>
<textarea value={output} rows={3} readOnly></textarea>
</div>
</div>
<button disabled={disabled} onClick={translate}>
Translate
</button>
<div className="progress-bars-container">
{ready === false && <label>Loading models... (only run once)</label>}
{progressItems.map((data) => (
<div key={data.file}>
<Progress text={data.file} percentage={data.progress} />
</div>
))}
</div>
</>
);
```
Don't worry about the `translate` function for now. We will define it in the next section.
Finally, we can add some CSS to make our app look a little nicer. Modify the following files in the `src` directory:
1. `index.css`:
<details>
<summary>View code</summary>
```css
:root {
font-family: Inter, system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
font-weight: 400;
color: #213547;
background-color: #ffffff;
font-synthesis: none;
text-rendering: optimizeLegibility;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
-webkit-text-size-adjust: 100%;
}
body {
margin: 0;
display: flex;
place-items: center;
min-width: 320px;
min-height: 100vh;
}
h1 {
font-size: 3.2em;
line-height: 1;
}
h1,
h2 {
margin: 8px;
}
select {
padding: 0.3em;
cursor: pointer;
}
textarea {
padding: 0.6em;
}
button {
padding: 0.6em 1.2em;
cursor: pointer;
font-weight: 500;
}
button[disabled] {
cursor: not-allowed;
}
select,
textarea,
button {
border-radius: 8px;
border: 1px solid transparent;
font-size: 1em;
font-family: inherit;
background-color: #f9f9f9;
transition: border-color 0.25s;
}
select:hover,
textarea:hover,
button:not([disabled]):hover {
border-color: #646cff;
}
select:focus,
select:focus-visible,
textarea:focus,
textarea:focus-visible,
button:focus,
button:focus-visible {
outline: 4px auto -webkit-focus-ring-color;
}
```
</details>
1. `App.css`
<details>
<summary>View code</summary>
```css
#root {
max-width: 1280px;
margin: 0 auto;
padding: 2rem;
text-align: center;
}
.language-container {
display: flex;
gap: 20px;
}
.textbox-container {
display: flex;
justify-content: center;
gap: 20px;
width: 800px;
}
.textbox-container > textarea,
.language-selector {
width: 50%;
}
.language-selector > select {
width: 150px;
}
.progress-container {
position: relative;
font-size: 14px;
color: white;
background-color: #e9ecef;
border: solid 1px;
border-radius: 8px;
text-align: left;
overflow: hidden;
}
.progress-bar {
padding: 0 4px;
z-index: 0;
top: 0;
width: 1%;
overflow: hidden;
background-color: #007bff;
white-space: nowrap;
}
.progress-text {
z-index: 2;
}
.selector-container {
display: flex;
gap: 20px;
}
.progress-bars-container {
padding: 8px;
height: 140px;
}
.container {
margin: 25px;
display: flex;
flex-direction: column;
gap: 10px;
}
```
</details>
## Step 4: Connecting everything together
Now that we have a basic user interface set up, we can finally connect everything together.
First, let's define the `translate` function, which will be called when the user clicks the `Translate` button. This sends a message (containing the input text, source language, and target language) to the worker thread for processing. We will also disable the button so the user doesn't click it multiple times. Add the following code just before the `return` statement in the `App` function:
```jsx
const translate = () => {
setDisabled(true);
setOutput("");
worker.current.postMessage({
text: input,
src_lang: sourceLanguage,
tgt_lang: targetLanguage,
});
};
```
Now, let's add an event listener in `src/worker.js` to listen for messages from the main thread. We will send back messages (e.g., for model loading progress and text streaming) to the main thread with `self.postMessage`.
```javascript
// Listen for messages from the main thread
self.addEventListener("message", async (event) => {
// Retrieve the translation pipeline. When called for the first time,
// this will load the pipeline and save it for future use.
const translator = await MyTranslationPipeline.getInstance((x) => {
// We also add a progress callback to the pipeline so that we can
// track model loading.
self.postMessage(x);
});
// Capture partial output as it streams from the pipeline
const streamer = new TextStreamer(translator.tokenizer, {
skip_prompt: true,
skip_special_tokens: true,
callback_function: function (text) {
self.postMessage({
status: "update",
output: text,
});
},
});
// Actually perform the translation
const output = await translator(event.data.text, {
tgt_lang: event.data.tgt_lang,
src_lang: event.data.src_lang,
// Allows for partial output to be captured
streamer,
});
// Send the output back to the main thread
self.postMessage({
status: "complete",
output,
});
});
```
Finally, let's fill in our `onMessageReceived` function in `src/App.jsx`, which will update the application state in response to messages from the worker thread. Add the following code inside the `useEffect` hook we defined earlier:
```jsx
const onMessageReceived = (e) => {
switch (e.data.status) {
case "initiate":
// Model file start load: add a new progress item to the list.
setReady(false);
setProgressItems((prev) => [...prev, e.data]);
break;
case "progress":
// Model file progress: update one of the progress items.
setProgressItems((prev) =>
prev.map((item) => {
if (item.file === e.data.file) {
return { ...item, progress: e.data.progress };
}
return item;
}),
);
break;
case "done":
// Model file loaded: remove the progress item from the list.
setProgressItems((prev) =>
prev.filter((item) => item.file !== e.data.file),
);
break;
case "ready":
// Pipeline ready: the worker is ready to accept messages.
setReady(true);
break;
case "update":
// Generation update: update the output text.
setOutput((o) => o + e.data.output);
break;
case "complete":
// Generation complete: re-enable the "Translate" button
setDisabled(false);
break;
}
};
```
You can now run the application with `npm run dev` and perform multilingual translation directly in your browser!
## (Optional) Step 5: Build and deploy
To build your application, simply run `npm run build`. This will bundle your application and output the static files to the `dist` folder.
For this demo, we will deploy our application as a static [Hugging Face Space](https://huggingface.co/docs/hub/spaces), but you can deploy it anywhere you like! If you haven't already, you can create a free Hugging Face account [here](https://huggingface.co/join).
1. Visit [https://huggingface.co/new-space](https://huggingface.co/new-space) and fill in the form. Remember to select "Static" as the space type.
2. Go to "Files" &rarr; "Add file" &rarr; "Upload files". Drag the `index.html` file and `public/` folder from the `dist` folder into the upload box and click "Upload". After they have uploaded, scroll down to the button and click "Commit changes to main".
**That's it!** Your application should now be live at `https://huggingface.co/spaces/<your-username>/<your-space-name>`!
packages/transformers/docs/source/tutorials/vanilla-js.md
+302
View File
@@ -0,0 +1,302 @@
# Building a Vanilla JavaScript Application
In this tutorial, you’ll build a simple web application that detects objects in images using Transformers.js! To follow along, all you need is a code editor, a browser, and a simple server (e.g., VS Code Live Server).
Here's how it works: the user clicks “Upload image” and selects an image using an input dialog. After analysing the image with an object detection model, the predicted bounding boxes are overlaid on top of the image, like this:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/js-detection-interence-zebra.png)
Useful links:
- [Demo site](https://huggingface.co/spaces/Scrimba/vanilla-js-object-detector)
- [Source code](https://github.com/huggingface/transformers.js-examples/tree/main/vanilla-js)
## Step 1: HTML and CSS setup
Before we start building with Transformers.js, we first need to lay the groundwork with some markup and styling. Create an `index.html` file with a basic HTML skeleton, and add the following `<main>` tag to the `<body>`:
```html
<main class="container">
<label class="custom-file-upload">
<input id="file-upload" type="file" accept="image/*" />
<img
class="upload-icon"
src="https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/upload-icon.png"
/>
Upload image
</label>
<div id="image-container"></div>
<p id="status"></p>
</main>
```
<details>
<summary>Click here to see a breakdown of this markup.</summary>
We’re adding an `<input>` element with `type="file"` that accepts images. This allows the user to select an image from their local file system using a popup dialog. The default styling for this element looks quite bad, so let's add some styling. The easiest way to achieve this is to wrap the `<input>` element in a `<label>`, hide the input, and then style the label as a button.
We’re also adding an empty `<div>` container for displaying the image, plus an empty `<p>` tag that we'll use to give status updates to the user while we download and run the model, since both of these operations take some time.
</details>
Next, add the following CSS rules in a `style.css` file and link it to the HTML:
```css
html,
body {
font-family: Arial, Helvetica, sans-serif;
}
.container {
margin: 40px auto;
width: max(50vw, 400px);
display: flex;
flex-direction: column;
align-items: center;
}
.custom-file-upload {
display: flex;
align-items: center;
gap: 10px;
border: 2px solid black;
padding: 8px 16px;
cursor: pointer;
border-radius: 6px;
}
#file-upload {
display: none;
}
.upload-icon {
width: 30px;
}
#image-container {
width: 100%;
margin-top: 20px;
position: relative;
}
#image-container > img {
width: 100%;
}
```
Here's how the UI looks at this point:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/js-detection-btn.png)
## Step 2: JavaScript setup
With the _boring_ part out of the way, let's start writing some JavaScript code! Create a file called `index.js` and link to it in `index.html` by adding the following to the end of the `<body>`:
```html
<script src="./index.js" type="module"></script>
```
<Tip>
The `type="module"` attribute is important, as it turns our file into a [JavaScript module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules), meaning that we’ll be able to use imports and exports.
</Tip>
Moving into `index.js`, let's import Transformers.js by adding the following line to the top of the file:
```js
import {
pipeline,
env,
} from "https://cdn.jsdelivr.net/npm/@huggingface/transformers";
```
Since we will be downloading the model from the Hugging Face Hub, we can skip the local model check by setting:
```js
env.allowLocalModels = false;
```
Next, let's create references to the various DOM elements we will access later:
```js
const fileUpload = document.getElementById("file-upload");
const imageContainer = document.getElementById("image-container");
const status = document.getElementById("status");
```
## Step 3: Create an object detection pipeline
We’re finally ready to create our object detection pipeline! As a reminder, a [pipeline](../pipelines). is a high-level interface provided by the library to perform a specific task. In our case, we will instantiate an object detection pipeline with the `pipeline()` helper function.
Since this can take some time (especially the first time when we have to download the ~40MB model), we first update the `status` paragraph so that the user knows that we’re about to load the model.
```js
status.textContent = "Loading model...";
```
<Tip>
To keep this tutorial simple, we'll be loading and running the model in the main (UI) thread. This is not recommended for production applications, since the UI will freeze when we're performing these actions. This is because JavaScript is a single-threaded language. To overcome this, you can use a [web worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) to download and run the model in the background. However, we’re not going to do cover that in this tutorial...
</Tip>
We can now call the `pipeline()` function that we imported at the top of our file, to create our object detection pipeline:
```js
const detector = await pipeline("object-detection", "Xenova/detr-resnet-50");
```
We’re passing two arguments into the `pipeline()` function: (1) task and (2) model.
1. The first tells Transformers.js what kind of task we want to perform. In our case, that is `object-detection`, but there are many other tasks that the library supports, including `text-generation`, `sentiment-analysis`, `summarization`, or `automatic-speech-recognition`. See [here](https://huggingface.co/docs/transformers.js/pipelines#tasks) for the full list.
2. The second argument specifies which model we would like to use to solve the given task. We will use [`Xenova/detr-resnet-50`](https://huggingface.co/Xenova/detr-resnet-50), as it is a relatively small (~40MB) but powerful model for detecting objects in an image.
Once the function returns, we’ll tell the user that the app is ready to be used.
```js
status.textContent = "Ready";
```
## Step 4: Create the image uploader
The next step is to support uploading/selection of images. To achieve this, we will listen for "change" events from the `fileUpload` element. In the callback function, we use a `FileReader()` to read the contents of the image if one is selected (and nothing otherwise).
```js
fileUpload.addEventListener("change", function (e) {
const file = e.target.files[0];
if (!file) {
return;
}
const reader = new FileReader();
// Set up a callback when the file is loaded
reader.onload = function (e2) {
imageContainer.innerHTML = "";
const image = document.createElement("img");
image.src = e2.target.result;
imageContainer.appendChild(image);
// detect(image); // Uncomment this line to run the model
};
reader.readAsDataURL(file);
});
```
Once the image has been loaded into the browser, the `reader.onload` callback function will be invoked. In it, we append the new `<img>` element to the `imageContainer` to be displayed to the user.
Don’t worry about the `detect(image)` function call (which is commented out) - we will explain it later! For now, try to run the app and upload an image to the browser. You should see your image displayed under the button like this:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/js-detection-btn-img.png)
## Step 5: Run the model
We’re finally ready to start interacting with Transformers.js! Let’s uncomment the `detect(image)` function call from the snippet above. Then we’ll define the function itself:
```js
async function detect(img) {
status.textContent = "Analysing...";
const output = await detector(img.src, {
threshold: 0.5,
percentage: true,
});
status.textContent = "";
console.log("output", output);
// ...
}
```
<Tip>
NOTE: The `detect` function needs to be asynchronous, since we’ll `await` the result of the the model.
</Tip>
Once we’ve updated the `status` to "Analysing", we’re ready to perform _inference_, which simply means to run the model with some data. This is done via the `detector()` function that was returned from `pipeline()`. The first argument we’re passing is the image data (`img.src`).
The second argument is an options object:
- We set the `threshold` property to `0.5`. This means that we want the model to be at least 50% confident before claiming it has detected an object in the image. The lower the threshold, the more objects it'll detect (but may misidentify objects); the higher the threshold, the fewer objects it'll detect (but may miss objects in the scene).
- We also specify `percentage: true`, which means that we want the bounding box for the objects to be returned as percentages (instead of pixels).
If you now try to run the app and upload an image, you should see the following output logged to the console:
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/js-detection-console.png)
In the example above, we uploaded an image of two elephants, so the `output` variable holds an array with two objects, each containing a `label` (the string “elephant”), a `score` (indicating the model's confidence in its prediction) and a `box` object (representing the bounding box of the detected entity).
## Step 6: Render the boxes
The final step is to display the `box` coordinates as rectangles around each of the elephants.
At the end of our `detect()` function, we’ll run the `renderBox` function on each object in the `output` array, using `.forEach()`.
```js
output.forEach(renderBox);
```
Here’s the code for the `renderBox()` function with comments to help you understand what’s going on:
```js
// Render a bounding box and label on the image
function renderBox({ box, label }) {
const { xmax, xmin, ymax, ymin } = box;
// Generate a random color for the box
const color =
"#" +
Math.floor(Math.random() * 0xffffff)
.toString(16)
.padStart(6, 0);
// Draw the box
const boxElement = document.createElement("div");
boxElement.className = "bounding-box";
Object.assign(boxElement.style, {
borderColor: color,
left: 100 * xmin + "%",
top: 100 * ymin + "%",
width: 100 * (xmax - xmin) + "%",
height: 100 * (ymax - ymin) + "%",
});
// Draw the label
const labelElement = document.createElement("span");
labelElement.textContent = label;
labelElement.className = "bounding-box-label";
labelElement.style.backgroundColor = color;
boxElement.appendChild(labelElement);
imageContainer.appendChild(boxElement);
}
```
The bounding box and label span also need some styling, so add the following to the `style.css` file:
```css
.bounding-box {
position: absolute;
box-sizing: border-box;
border-width: 2px;
border-style: solid;
}
.bounding-box-label {
color: white;
position: absolute;
font-size: 12px;
margin-top: -16px;
margin-left: -2px;
padding: 1px;
}
```
**And that’s it!**
You've now built your own fully-functional AI application that detects objects in images, which runs completely in your browser: no external server, APIs, or build tools. Pretty cool! 🥳
![Demo](https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/js-detection-inference-elephant.png)
The app is live at the following URL: [https://huggingface.co/spaces/Scrimba/vanilla-js-object-detector](https://huggingface.co/spaces/Scrimba/vanilla-js-object-detector)