mirrors/simonw--llm
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:48:46 +08:00
commit 6fdd9b4d7a
114 changed files with 40223 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/.gitignore
+1
View File
@@ -0,0 +1 @@
_build
docs/Makefile
+23
View File
@@ -0,0 +1,23 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = sqlite-utils
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
livehtml:
sphinx-autobuild -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0)
docs/_templates/base.html
Vendored
+16
View File
@@ -0,0 +1,16 @@
{%- extends "!base.html" %}
{%- block htmltitle -%}
{% if not docstitle %}
<title>{{ title|striptags|e }}</title>
{% elif pagename == master_doc %}
<title>LLM: A CLI utility and Python library for interacting with Large Language Models</title>
{% else %}
<title>{{ title|striptags|e }} - {{ docstitle|striptags|e }}</title>
{% endif %}
{%- endblock -%}
{% block site_meta %}
{{ super() }}
<script defer data-domain="llm.datasette.io" src="https://plausible.io/js/plausible.js"></script>
{% endblock %}
docs/aliases.md
+116
View File
@@ -0,0 +1,116 @@
(aliases)=
# Model aliases
LLM supports model aliases, which allow you to refer to a model by a short name instead of its full ID.
## Listing aliases
To list current aliases, run this:
```bash
llm aliases
```
Example output:
<!-- [[[cog
from click.testing import CliRunner
from llm.cli import cli
result = CliRunner().invoke(cli, ["aliases", "list"])
cog.out("```\n{}```".format(result.output))
]]] -->
```
4o : gpt-4o
chatgpt-4o : chatgpt-4o-latest
4o-mini : gpt-4o-mini
4.1 : gpt-4.1
4.1-mini : gpt-4.1-mini
4.1-nano : gpt-4.1-nano
3.5 : gpt-3.5-turbo
chatgpt : gpt-3.5-turbo
chatgpt-16k : gpt-3.5-turbo-16k
3.5-16k : gpt-3.5-turbo-16k
4 : gpt-4
gpt4 : gpt-4
4-32k : gpt-4-32k
gpt-4-turbo-preview : gpt-4-turbo
4-turbo : gpt-4-turbo
4t : gpt-4-turbo
gpt-4.5 : gpt-4.5-preview
3.5-instruct : gpt-3.5-turbo-instruct
chatgpt-instruct : gpt-3.5-turbo-instruct
ada : text-embedding-ada-002 (embedding)
ada-002 : text-embedding-ada-002 (embedding)
3-small : text-embedding-3-small (embedding)
3-large : text-embedding-3-large (embedding)
3-small-512 : text-embedding-3-small-512 (embedding)
3-large-256 : text-embedding-3-large-256 (embedding)
3-large-1024 : text-embedding-3-large-1024 (embedding)
```
<!-- [[[end]]] -->
Add `--json` to get that list back as JSON:
```bash
llm aliases list --json
```
Example output:
```json
{
"3.5": "gpt-3.5-turbo",
"chatgpt": "gpt-3.5-turbo",
"4": "gpt-4",
"gpt4": "gpt-4",
"ada": "ada-002"
}
```
## Adding a new alias
The `llm aliases set <alias> <model-id>` command can be used to add a new alias:
```bash
llm aliases set mini gpt-4o-mini
```
You can also pass one or more `-q search` options to set an alias on the first model matching those search terms:
```bash
llm aliases set mini -q 4o -q mini
```
Now you can run the `gpt-4o-mini` model using the `mini` alias like this:
```bash
llm -m mini 'An epic Greek-style saga about a cheesecake that builds a SQL database from scratch'
```
Aliases can be set for both regular models and {ref}`embedding models <embeddings>` using the same command. To set an alias of `oai` for the OpenAI `ada-002` embedding model use this:
```bash
llm aliases set oai ada-002
```
Now you can embed a string using that model like so:
```bash
llm embed -c 'hello world' -m oai
```
Output:
```
[-0.014945968054234982, 0.0014304015785455704, ...]
```
## Removing an alias
The `llm aliases remove <alias>` command will remove the specified alias:
```bash
llm aliases remove mini
```
## Viewing the aliases file
Aliases are stored in an `aliases.json` file in the LLM configuration directory.
To see the path to that file, run this:
```bash
llm aliases path
```
To view the content of that file, run this:
```bash
cat "$(llm aliases path)"
```
docs/changelog.md
+1029
View File
File diff suppressed because it is too large Load Diff
docs/conf.py
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from subprocess import PIPE, Popen
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"myst_parser",
"sphinx_copybutton",
"sphinx_markdown_builder",
"sphinx.ext.autodoc",
]
myst_enable_extensions = ["colon_fence"]
markdown_http_base = "https://llm.datasette.io/en/stable"
markdown_uri_doc_suffix = ".html"
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = ".rst"
# The master toctree document.
master_doc = "index"
# General information about the project.
project = "LLM"
copyright = "2025, Simon Willison"
author = "Simon Willison"
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
pipe = Popen("git describe --tags --always", stdout=PIPE, shell=True)
git_version = pipe.stdout.read().decode("utf8")
if git_version:
version = git_version.rsplit("-", 1)[0]
release = git_version
else:
version = ""
release = ""
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = "en"
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = "furo"
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {}
html_title = "LLM"
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
# -- Options for HTMLHelp output ------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = "llm-doc"
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(
master_doc,
"llm.tex",
"LLM documentation",
"Simon Willison",
"manual",
)
]
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(
master_doc,
"llm",
"LLM documentation",
[author],
1,
)
]
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(
master_doc,
"llm",
"LLM documentation",
author,
"llm",
" Access large language models from the command-line ",
"Miscellaneous",
)
]
docs/contributing.md
+66
View File
@@ -0,0 +1,66 @@
(contributing)=
# Contributing
To contribute to this tool, first checkout the code. Then run the tests with `uv run`:
```bash
cd llm
uv run pytest
```
You can run your development copy of `llm` using `uv run` as well:
```bash
uv run llm --help
```
## Updating recorded HTTP API interactions and associated snapshots
This project uses [pytest-recording](https://github.com/kiwicom/pytest-recording) to record OpenAI API responses for some of the tests, and [syrupy](https://github.com/syrupy-project/syrupy) to capture snapshots of their results.
If you add a new test that calls the API you can capture the API response and snapshot like this:
```bash
PYTEST_OPENAI_API_KEY="$(llm keys get openai)" uv run pytest --record-mode once --snapshot-update
```
Then review the new snapshots in `tests/__snapshots__/` to make sure they look correct.
## Debugging tricks
The default OpenAI plugin has a debugging mechanism for showing the exact requests and responses that were sent to the OpenAI API.
Set the `LLM_OPENAI_SHOW_RESPONSES` environment variable like this:
```bash
LLM_OPENAI_SHOW_RESPONSES=1 uv run llm -m chatgpt 'three word slogan for an otter-run bakery'
```
This will output details of the API requests and responses to the console.
Use `--no-stream` to see a more readable version of the body that avoids streaming the response:
```bash
LLM_OPENAI_SHOW_RESPONSES=1 uv run llm -m chatgpt --no-stream \
'three word slogan for an otter-run bakery'
```
## Documentation
Documentation for this project uses [MyST](https://myst-parser.readthedocs.io/) - it is written in Markdown and rendered using Sphinx.
To build the documentation locally, run the following:
```bash
just docs
```
This will start a live preview server, using [sphinx-autobuild](https://pypi.org/project/sphinx-autobuild/).
The CLI `--help` examples in the documentation are managed using [Cog](https://github.com/nedbat/cog). Update those files like this:
```bash
just cog
```
You'll need [Just](https://github.com/casey/just) installed to run these commands.
## Release process
To release a new version:
1. Update `docs/changelog.md` with the new changes.
2. Update the version number in `pyproject.toml`
3. Run `just cog` to update `docs/fragments.md` with the new version number.
4. [Create a GitHub release](https://github.com/simonw/llm/releases/new) for the new version.
5. Wait for the package to push to PyPI and then...
6. Run the [regenerate.yaml](https://github.com/simonw/homebrew-llm/actions/workflows/regenerate.yaml) workflow to update the Homebrew tap to the latest version.
docs/embeddings/cli.md
+443
View File
@@ -0,0 +1,443 @@
(embeddings-cli)=
# Embedding with the CLI
LLM provides command-line utilities for calculating and storing embeddings for pieces of content.
(embeddings-cli-embed)=
## llm embed
The `llm embed` command can be used to calculate embedding vectors for a string of content. These can be returned directly to the terminal, stored in a SQLite database, or both.
### Returning embeddings to the terminal
The simplest way to use this command is to pass content to it using the `-c/--content` option, like this:
```bash
llm embed -c 'This is some content' -m 3-small
```
`-m 3-small` specifies the OpenAI `text-embedding-3-small` model. You will need to have set an OpenAI API key using `llm keys set openai` for this to work.
You can install plugins to access other models. The [llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers) plugin can be used to run models on your own laptop, such as the [MiniLM-L6](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model:
```bash
llm install llm-sentence-transformers
llm embed -c 'This is some content' -m sentence-transformers/all-MiniLM-L6-v2
```
The `llm embed` command returns a JSON array of floating point numbers directly to the terminal:
```json
[0.123, 0.456, 0.789...]
```
You can omit the `-m/--model` option if you set a {ref}`default embedding model <embeddings-cli-embed-models-default>`.
You can also set the `LLM_EMBEDDING_MODEL` environment variable to set a default model for all `llm embed` commands in the current shell session:
```bash
export LLM_EMBEDDING_MODEL=3-small
llm embed -c 'This is some content'
```
LLM also offers a binary storage format for embeddings, described in {ref}`embeddings storage format <embeddings-storage>`.
You can output embeddings using that format as raw bytes using `--format blob`, or in hexadecimal using `--format hex`, or in Base64 using `--format base64`:
```bash
llm embed -c 'This is some content' -m 3-small --format base64
```
This outputs:
```
8NGzPFtdgTqHcZw7aUT6u+++WrwwpZo8XbSxv...
```
Some models such as [llm-clip](https://github.com/simonw/llm-clip) can run against binary data. You can pass in binary data using the `-i` and `--binary` options:
```bash
llm embed --binary -m clip -i image.jpg
```
Or from standard input like this:
```bash
cat image.jpg | llm embed --binary -m clip -i -
```
(embeddings-collections)=
### Storing embeddings in SQLite
Embeddings are much more useful if you store them somewhere, so you can calculate similarity scores between different embeddings later on.
LLM includes the concept of a **collection** of embeddings. A collection groups together a set of stored embeddings created using the same model, each with a unique ID within that collection.
Embeddings also store a hash of the content that was embedded. This hash is later used to avoid calculating duplicate embeddings for the same content.
First, we'll set a default model so we don't have to keep repeating it:
```bash
llm embed-models default 3-small
```
The `llm embed` command can store results directly in a named collection like this:
```bash
llm embed quotations philkarlton-1 -c \
'There are only two hard things in Computer Science: cache invalidation and naming things'
```
This stores the given text in the `quotations` collection under the key `philkarlton-1`.
You can also pipe content to standard input, like this:
```bash
cat one.txt | llm embed files one
```
This will store the embedding for the contents of `one.txt` in the `files` collection under the key `one`.
A collection will be created the first time you mention it.
Collections have a fixed embedding model, which is the model that was used for the first embedding stored in that collection.
In the above example this would have been the default embedding model at the time that the command was run.
The following example stores the embedding for the string "my happy hound" in a collection called `phrases` under the key `hound` and using the model `3-small`:
```bash
llm embed phrases hound -m 3-small -c 'my happy hound'
```
By default, the SQLite database used to store embeddings is the `embeddings.db` in the user content directory managed by LLM.
You can see the path to this directory by running `llm collections path`.
You can store embeddings in a different SQLite database by passing a path to it using the `-d/--database` option to `llm embed`. If this file does not exist yet the command will create it:
```bash
llm embed phrases hound -d my-embeddings.db -c 'my happy hound'
```
This creates a database file called `my-embeddings.db` in the current directory.
(embeddings-collections-content-metadata)=
#### Storing content and metadata
By default, only the entry ID and the embedding vector are stored in the database table.
You can store a copy of the original text in the `content` column by passing the `--store` option:
```bash
llm embed phrases hound -c 'my happy hound' --store
```
You can also store a JSON object containing arbitrary metadata in the `metadata` column by passing the `--metadata` option. This example uses both `--store` and `--metadata` options:
```bash
llm embed phrases hound \
-m 3-small \
-c 'my happy hound' \
--metadata '{"name": "Hound"}' \
--store
```
Data stored in this way will be returned by calls to `llm similar`, for example:
```bash
llm similar phrases -c 'hound'
```
```
{"id": "hound", "score": 0.8484683588631485, "content": "my happy hound", "metadata": {"name": "Hound"}}
```
(embeddings-cli-embed-multi)=
## llm embed-multi
The `llm embed` command embeds a single string at a time.
`llm embed-multi` can be used to embed multiple strings at once, taking advantage of any efficiencies that the embedding model may provide when processing multiple strings.
This command can be called in one of three ways:
1. With a CSV, TSV, JSON or newline-delimited JSON file
2. With a SQLite database and a SQL query
3. With one or more paths to directories, each accompanied by a glob pattern
All three mechanisms support these options:
- `-m model_id` to specify the embedding model to use
- `-d database.db` to specify a different database file to store the embeddings in
- `--store` to store the original content in the embeddings table in addition to the embedding vector
- `--prefix` to prepend a prefix to the stored ID of each item
- `--prepend` to prepend a string to the content before embedding
- `--batch-size SIZE` to process embeddings in batches of the specified size
The `--prepend` option is useful for embedding models that require you to prepend a special token to the content before embedding it. [nomic-embed-text-v2-moe](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe) for example requires documents to be prepended `'search_document: '` and search queries to be prepended `'search_query: '`.
(embeddings-cli-embed-multi-csv-etc)=
### Embedding data from a CSV, TSV or JSON file
You can embed data from a CSV, TSV or JSON file by passing that file to the command as the second option, after the collection name.
Your file must contain at least two columns. The first one is expected to contain the ID of the item, and any subsequent columns will be treated as containing content to be embedded.
An example CSV file might look like this:
```
id,content
one,This is the first item
two,This is the second item
```
TSV would use tabs instead of commas.
JSON files can be structured like this:
```json
[
{"id": "one", "content": "This is the first item"},
{"id": "two", "content": "This is the second item"}
]
```
Or as newline-delimited JSON like this:
```json
{"id": "one", "content": "This is the first item"}
{"id": "two", "content": "This is the second item"}
```
In each of these cases the file can be passed to `llm embed-multi` like this:
```bash
llm embed-multi items mydata.csv
```
The first argument is the name of the collection, the second is the filename.
You can also pipe content to standard input of the tool using `-`:
```bash
cat mydata.json | llm embed-multi items -
```
LLM will attempt to detect the format of your data automatically. If this doesn't work you can specify the format using the `--format` option. This is required if you are piping newline-delimited JSON to standard input.
```bash
cat mydata.json | llm embed-multi items - --format nl
```
Other supported `--format` options are `csv`, `tsv` and `json`.
This example embeds the data from a JSON file in a collection called `items` in database called `docs.db` using the `3-small` model and stores the original content in the `embeddings` table as well, adding a prefix of `my-items/` to each ID:
```bash
llm embed-multi items mydata.json \
-d docs.db \
-m 3-small \
--prefix my-items/ \
--store
```
(embeddings-cli-embed-multi-sqlite)=
### Embedding data from a SQLite database
You can embed data from a SQLite database using `--sql`, optionally combined with `--attach` to attach an additional database.
If you are storing embeddings in the same database as the source data, you can do this:
```bash
llm embed-multi docs \
-d docs.db \
--sql 'select id, title, content from documents' \
-m 3-small
```
The `docs.db` database here contains a `documents` table, and we want to embed the `title` and `content` columns from that table and store the results back in the same database.
To load content from a database other than the one you are using to store embeddings, attach it with the `--attach` option and use `alias.table` in your SQLite query:
```bash
llm embed-multi docs \
-d embeddings.db \
--attach other other.db \
--sql 'select id, title, content from other.documents' \
-m 3-small
```
(embeddings-cli-embed-multi-directories)=
### Embedding data from files in directories
LLM can embed the content of every text file in a specified directory, using the file's path and name as the ID.
Consider a directory structure like this:
```
docs/aliases.md
docs/contributing.md
docs/embeddings/binary.md
docs/embeddings/cli.md
docs/embeddings/index.md
docs/index.md
docs/logging.md
docs/plugins/directory.md
docs/plugins/index.md
```
To embed all of those documents, you can run the following:
```bash
llm embed-multi documentation \
-m 3-small \
--files docs '**/*.md' \
-d documentation.db \
--store
```
Here `--files docs '**/*.md'` specifies that the `docs` directory should be scanned for files matching the `**/*.md` glob pattern - which will match Markdown files in any nested directory.
The result of the above command is a `embeddings` table with the following IDs:
```
aliases.md
contributing.md
embeddings/binary.md
embeddings/cli.md
embeddings/index.md
index.md
logging.md
plugins/directory.md
plugins/index.md
```
Each corresponding to embedded content for the file in question.
The `--prefix` option can be used to add a prefix to each ID:
```bash
llm embed-multi documentation \
-m 3-small \
--files docs '**/*.md' \
-d documentation.db \
--store \
--prefix llm-docs/
```
This will result in the following IDs instead:
```
llm-docs/aliases.md
llm-docs/contributing.md
llm-docs/embeddings/binary.md
llm-docs/embeddings/cli.md
llm-docs/embeddings/index.md
llm-docs/index.md
llm-docs/logging.md
llm-docs/plugins/directory.md
llm-docs/plugins/index.md
```
Files are assumed to be `utf-8`, but LLM will fall back to `latin-1` if it encounters an encoding error. You can specify a different set of encodings using the `--encoding` option.
This example will try `utf-16` first and then `mac_roman` before falling back to `latin-1`:
```
llm embed-multi documentation \
-m 3-small \
--files docs '**/*.md' \
-d documentation.db \
--encoding utf-16 \
--encoding mac_roman \
--encoding latin-1
```
If a file cannot be read it will be logged to standard error but the script will keep on running.
If you are embedding binary content such as images for use with CLIP, add the `--binary` option:
```
llm embed-multi photos \
-m clip \
--files photos/ '*.jpeg' --binary
```
(embeddings-cli-similar)=
## llm similar
The `llm similar` command searches a collection of embeddings for the items that are most similar to a given or item ID, based on [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).
This currently uses a slow brute-force approach which does not scale well to large collections. See [issue 216](https://github.com/simonw/llm/issues/216) for plans to add a more scalable approach via vector indexes provided by plugins.
To search the `quotations` collection for items that are semantically similar to `'computer science'`:
```bash
llm similar quotations -c 'computer science'
```
This embeds the provided string and returns a newline-delimited list of JSON objects like this:
```json
{"id": "philkarlton-1", "score": 0.8323904531677017, "content": null, "metadata": null}
```
Use `-p/--plain` to get back results in plain text instead of JSON:
```bash
llm similar quotations -c 'computer science' -p
```
Example output:
```
philkarlton-1 (0.8323904531677017)
```
You can compare against text stored in a file using `-i filename`:
```bash
llm similar quotations -i one.txt
```
Or feed text to standard input using `-i -`:
```bash
echo 'computer science' | llm similar quotations -i -
```
When using a model like CLIP, you can find images similar to an input image using `-i filename` with `--binary`:
```bash
llm similar photos -i image.jpg --binary
```
You can filter results to only show IDs that begin with a specific prefix using --prefix:
```bash
llm similar quotations --prefix 'movies/' -c 'star wars'
```
(embeddings-cli-embed-models)=
## llm embed-models
To list all available embedding models, including those provided by plugins, run this command:
```bash
llm embed-models
```
The output should look something like this:
```
OpenAIEmbeddingModel: text-embedding-ada-002 (aliases: ada, ada-002)
OpenAIEmbeddingModel: text-embedding-3-small (aliases: 3-small)
OpenAIEmbeddingModel: text-embedding-3-large (aliases: 3-large)
...
```
Add `-q` one or more times to search for models matching those terms:
```bash
llm embed-models -q 3-small
```
(embeddings-cli-embed-models-default)=
### llm embed-models default
This command can be used to get and set the default embedding model.
This will return the name of the current default model:
```bash
llm embed-models default
```
You can set a different default like this:
```bash
llm embed-models default 3-small
```
This will set the default model to OpenAI's `3-small` model.
Any of the supported aliases for a model can be passed to this command.
You can unset the default model using `--remove-default`:
```bash
llm embed-models default --remove-default
```
When no default model is set, the `llm embed` and `llm embed-multi` commands will require that a model is specified using `-m/--model`.
## llm collections list
To list all of the collections in the embeddings database, run this command:
```bash
llm collections list
```
Add `--json` for JSON output:
```bash
llm collections list --json
```
Add `-d/--database` to specify a different database file:
```bash
llm collections list -d my-embeddings.db
```
## llm collections delete
To delete a collection from the database, run this:
```bash
llm collections delete collection-name
```
Pass `-d` to specify a different database file:
```bash
llm collections delete collection-name -d my-embeddings.db
```
docs/embeddings/index.md
+26
View File
@@ -0,0 +1,26 @@
(embeddings)=
# Embeddings
Embedding models allow you to take a piece of text - a word, sentence, paragraph or even a whole article, and convert that into an array of floating point numbers.
This floating point array is called an "embedding vector", and works as a numerical representation of the semantic meaning of the content in a many-multi-dimensional space.
By calculating the distance between embedding vectors, we can identify which content is semantically "nearest" to other content.
This can be used to build features like related article lookups. It can also be used to build semantic search, where a user can search for a phrase and get back results that are semantically similar to that phrase even if they do not share any exact keywords.
Some embedding models like [CLIP](https://github.com/simonw/llm-clip) can even work against binary files such as images. These can be used to search for images that are similar to other images, or to search for images that are semantically similar to a piece of text.
LLM supports multiple embedding models through {ref}`plugins <plugins>`. Once installed, an embedding model can be used on the command-line or via the Python API to calculate and store embeddings for content, and then to perform similarity searches against those embeddings.
See [LLM now provides tools for working with embeddings](https://simonwillison.net/2023/Sep/4/llm-embeddings/) for an extended explanation of embeddings, why they are useful and what you can do with them.
```{toctree}
---
maxdepth: 3
---
cli
python-api
writing-plugins
storage
```
docs/embeddings/python-api.md
+211
View File
@@ -0,0 +1,211 @@
(embeddings-python-api)=
# Using embeddings from Python
You can load an embedding model using its model ID or alias like this:
```python
import llm
embedding_model = llm.get_embedding_model("3-small")
```
To embed a string, returning a Python list of floating point numbers, use the `.embed()` method:
```python
vector = embedding_model.embed("my happy hound")
```
If the embedding model can handle binary input, you can call `.embed()` with a byte string instead. You can check the `supports_binary` property to see if this is supported:
```python
if embedding_model.supports_binary:
vector = embedding_model.embed(open("my-image.jpg", "rb").read())
```
The `embedding_model.supports_text` property indicates if the model supports text input.
Many embeddings models are more efficient when you embed multiple strings or binary strings at once. To embed multiple strings at once, use the `.embed_multi()` method:
```python
vectors = list(embedding_model.embed_multi(["my happy hound", "my dissatisfied cat"]))
```
This returns a generator that yields one embedding vector per string.
Embeddings are calculated in batches. By default all items will be processed in a single batch, unless the underlying embedding model has defined its own preferred batch size. You can pass a custom batch size using `batch_size=N`, for example:
```python
vectors = list(embedding_model.embed_multi(lines_from_file, batch_size=20))
```
(embeddings-python-collections)=
## Working with collections
The `llm.Collection` class can be used to work with **collections** of embeddings from Python code.
A collection is a named group of embedding vectors, each stored along with their IDs in a SQLite database table.
To work with embeddings in this way you will need an instance of a [sqlite-utils Database](https://sqlite-utils.datasette.io/en/stable/python-api.html#connecting-to-or-creating-a-database) object. You can then pass that to the `llm.Collection` constructor along with the unique string name of the collection and the ID of the embedding model you will be using with that collection:
```python
import sqlite_utils
import llm
# This collection will use an in-memory database that will be
# discarded when the Python process exits
collection = llm.Collection("entries", model_id="3-small")
# Or you can persist the database to disk like this:
db = sqlite_utils.Database("my-embeddings.db")
collection = llm.Collection("entries", db, model_id="3-small")
# You can pass a model directly using model= instead of model_id=
embedding_model = llm.get_embedding_model("3-small")
collection = llm.Collection("entries", db, model=embedding_model)
```
If the collection already exists in the database you can omit the `model` or `model_id` argument - the model ID will be read from the `collections` table.
To embed a single string and store it in the collection, use the `embed()` method:
```python
collection.embed("hound", "my happy hound")
```
This stores the embedding for the string "my happy hound" in the `entries` collection under the key `hound`.
Add `store=True` to store the text content itself in the database table along with the embedding vector.
To attach additional metadata to an item, pass a JSON-compatible dictionary as the `metadata=` argument:
```python
collection.embed("hound", "my happy hound", metadata={"name": "Hound"}, store=True)
```
This additional metadata will be stored as JSON in the `metadata` column of the embeddings database table.
(embeddings-python-bulk)=
### Storing embeddings in bulk
The `collection.embed_multi()` method can be used to store embeddings for multiple items at once. This can be more efficient for some embedding models.
```python
collection.embed_multi(
[
("hound", "my happy hound"),
("cat", "my dissatisfied cat"),
],
# Add this to store the strings in the content column:
store=True,
)
```
To include metadata to be stored with each item, call `embed_multi_with_metadata()`:
```python
collection.embed_multi_with_metadata(
[
("hound", "my happy hound", {"name": "Hound"}),
("cat", "my dissatisfied cat", {"name": "Cat"}),
],
# This can also take the store=True argument:
store=True,
)
```
The `batch_size=` argument defaults to 100, and will be used unless the embedding model itself defines a lower batch size. You can adjust this if you are having trouble with memory while embedding large collections:
```python
collection.embed_multi(
(
(i, line)
for i, line in enumerate(lines_in_file)
),
batch_size=10
)
```
(embeddings-python-collection-class)=
### Collection class reference
A collection instance has the following properties and methods:
- `id` - the integer ID of the collection in the database
- `name` - the string name of the collection (unique in the database)
- `model_id` - the string ID of the embedding model used for this collection
- `model()` - returns the `EmbeddingModel` instance, based on that `model_id`
- `count()` - returns the integer number of items in the collection
- `embed(id: str, text: str, metadata: dict=None, store: bool=False)` - embeds the given string and stores it in the collection under the given ID. Can optionally include metadata (stored as JSON) and store the text content itself in the database table.
- `embed_multi(entries: Iterable, store: bool=False, batch_size: int=100)` - see above
- `embed_multi_with_metadata(entries: Iterable, store: bool=False, batch_size: int=100)` - see above
- `similar(query: str, number: int=10)` - returns a list of entries that are most similar to the embedding of the given query string
- `similar_by_id(id: str, number: int=10)` - returns a list of entries that are most similar to the embedding of the item with the given ID
- `similar_by_vector(vector: List[float], number: int=10, skip_id: str=None)` - returns a list of entries that are most similar to the given embedding vector, optionally skipping the entry with the given ID
- `delete()` - deletes the collection and its embeddings from the database
There is also a `Collection.exists(db, name)` class method which returns a boolean value and can be used to determine if a collection exists or not in a database:
```python
if Collection.exists(db, "entries"):
print("The entries collection exists")
```
(embeddings-python-similar)=
## Retrieving similar items
Once you have populated a collection of embeddings you can retrieve the entries that are most similar to a given string using the `similar()` method.
This method uses a brute force approach, calculating distance scores against every document. This is fine for small collections, but will not scale to large collections. See [issue 216](https://github.com/simonw/llm/issues/216) for plans to add a more scalable approach via vector indexes provided by plugins.
```python
for entry in collection.similar("hound"):
print(entry.id, entry.score)
```
The string will first by embedded using the model for the collection.
The `entry` object returned is an object with the following properties:
- `id` - the string ID of the item
- `score` - the floating point similarity score between the item and the query string
- `content` - the string text content of the item, if it was stored - or `None`
- `metadata` - the dictionary (from JSON) metadata for the item, if it was stored - or `None`
This defaults to returning the 10 most similar items. You can change this by passing a different `number=` argument:
```python
for entry in collection.similar("hound", number=5):
print(entry.id, entry.score)
```
The `similar_by_id()` method takes the ID of another item in the collection and returns the most similar items to that one, based on the embedding that has already been stored for it:
```python
for entry in collection.similar_by_id("cat"):
print(entry.id, entry.score)
```
The item itself is excluded from the results.
(embeddings-sql-schema)=
## SQL schema
Here's the SQL schema used by the embeddings database:
<!-- [[[cog
import cog
from llm.embeddings_migrations import embeddings_migrations
import sqlite_utils
import re
db = sqlite_utils.Database(memory=True)
embeddings_migrations.apply(db)
cog.out("```sql\n")
for table in ("collections", "embeddings"):
schema = db[table].schema
cog.out(format(schema))
cog.out("\n")
cog.out("```\n")
]]] -->
```sql
CREATE TABLE "collections" (
"id" INTEGER PRIMARY KEY,
"name" TEXT,
"model" TEXT
)
CREATE TABLE "embeddings" (
"collection_id" INTEGER REFERENCES "collections"("id"),
"id" TEXT,
"embedding" BLOB,
"content" TEXT,
"content_blob" BLOB,
"content_hash" BLOB,
"metadata" TEXT,
"updated" INTEGER,
PRIMARY KEY ("collection_id", "id")
)
```
<!-- [[[end]]] -->
docs/embeddings/storage.md
+31
View File
@@ -0,0 +1,31 @@
(embeddings-storage)=
# Embedding storage format
The default output format of the `llm embed` command is a JSON array of floating point numbers.
LLM stores embeddings in space-efficient format: a little-endian binary sequences of 32-bit floating point numbers, each represented using 4 bytes.
These are stored in a `BLOB` column in a SQLite database.
The following Python functions can be used to convert between this format and an array of floating point numbers:
```python
import struct
def encode(values):
return struct.pack("<" + "f" * len(values), *values)
def decode(binary):
return struct.unpack("<" + "f" * (len(binary) // 4), binary)
```
These functions are available as `llm.encode()` and `llm.decode()`.
If you are using [NumPy](https://numpy.org/) you can decode one of these binary values like this:
```python
import numpy as np
numpy_array = np.frombuffer(value, "<f4")
```
The `<f4` format string here ensures NumPy will treat the data as a little-endian sequence of 32-bit floats.
docs/embeddings/writing-plugins.md
+73
View File
@@ -0,0 +1,73 @@
(embeddings-writing-plugins)=
# Writing plugins to add new embedding models
Read the {ref}`plugin tutorial <tutorial-model-plugin>` for details on how to develop and package a plugin.
This page shows an example plugin that implements and registers a new embedding model.
There are two components to an embedding model plugin:
1. An implementation of the `register_embedding_models()` hook, which takes a `register` callback function and calls it to register the new model with the LLM plugin system.
2. A class that extends the `llm.EmbeddingModel` abstract base class.
The only required method on this class is `embed_batch(texts)`, which takes an iterable of strings and returns an iterator over lists of floating point numbers.
The following example uses the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) package to provide access to the [MiniLM-L6](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) embedding model.
```{eval-rst}
.. autoclass:: llm.EmbeddingModel
:members: embed, embed_multi, embed_batch
```
```python
import llm
from sentence_transformers import SentenceTransformer
@llm.hookimpl
def register_embedding_models(register):
model_id = "sentence-transformers/all-MiniLM-L6-v2"
register(SentenceTransformerModel(model_id, model_id), aliases=("all-MiniLM-L6-v2",))
class SentenceTransformerModel(llm.EmbeddingModel):
def __init__(self, model_id, model_name):
self.model_id = model_id
self.model_name = model_name
self._model = None
def embed_batch(self, texts):
if self._model is None:
self._model = SentenceTransformer(self.model_name)
results = self._model.encode(texts)
return (list(map(float, result)) for result in results)
```
Once installed, the model provided by this plugin can be used with the {ref}`llm embed <embeddings-cli-embed>` command like this:
```bash
cat file.txt | llm embed -m sentence-transformers/all-MiniLM-L6-v2
```
Or via its registered alias like this:
```bash
cat file.txt | llm embed -m all-MiniLM-L6-v2
```
[llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers) is a complete example of a plugin that provides an embedding model.
[Execute Jina embeddings with a CLI using llm-embed-jina](https://simonwillison.net/2023/Oct/26/llm-embed-jina/#how-i-built-the-plugin) talks through a similar process to add support for the [Jina embeddings models](https://jina.ai/news/jina-ai-launches-worlds-first-open-source-8k-text-embedding-rivaling-openai/).
## Embedding binary content
If your model can embed binary content, use the `supports_binary` property to indicate that:
```python
class ClipEmbeddingModel(llm.EmbeddingModel):
model_id = "clip"
supports_binary = True
supports_text= True
```
`supports_text` defaults to `True` and so is not necessary here. You can set it to `False` if your model only supports binary data.
If your model accepts binary, your `.embed_batch()` model may be called with a list of Python bytestrings. These may be mixed with regular strings if the model accepts both types of input.
[llm-clip](https://github.com/simonw/llm-clip) is an example of a model that can embed both binary and text content.
docs/fragments.md
+217
View File
@@ -0,0 +1,217 @@
(fragments)=
# Fragments
LLM prompts can optionally be composed out of **fragments** - reusable pieces of text that are logged just once to the database and can then be attached to multiple prompts.
These are particularly useful when you are working with long context models, which support feeding large amounts of text in as part of your prompt.
Fragments primarily exist to save space in the database, but may be used to support other features such as vendor prompt caching as well.
Fragments can be specified using several different mechanisms:
- URLs to text files online
- Paths to text files on disk
- Aliases that have been attached to a specific fragment
- Hash IDs of stored fragments, where the ID is the SHA256 hash of the fragment content
- Fragments that are provided by custom plugins - these look like `plugin-name:argument`
(fragments-usage)=
## Using fragments in a prompt
Use the `-f/--fragment` option to specify one or more fragments to be used as part of your prompt:
```bash
llm -f https://llm.datasette.io/robots.txt "Explain this robots.txt file in detail"
```
Here we are specifying a fragment using a URL. The contents of that URL will be included in the prompt that is sent to the model, prepended prior to the prompt text.
<!--[[[cog
from importlib.metadata import version
llm_version = version("llm")
cog.out(f'The URL will be fetched with the user-agent `llm/{llm_version} (https://llm.datasette.io/)`.')
]]]-->
The URL will be fetched with the user-agent `llm/0.32a3 (https://llm.datasette.io/)`.
<!--[[[end]]]-->
The `-f` option can be used multiple times to combine together multiple fragments.
Fragments can also be files on disk, for example:
```bash
llm -f setup.py 'extract the metadata'
```
Use `-` to specify a fragment that is read from standard input:
```bash
llm -f - 'extract the metadata' < setup.py
```
This will read the contents of `setup.py` from standard input and use it as a fragment.
Fragments can also be used as part of your system prompt. Use `--sf value` or `--system-fragment value` instead of `-f`.
## Using fragments in chat
The `chat` command also supports the `-f` and `--sf` arguments to start a chat with fragments.
```bash
llm chat -f my_doc.txt
Chatting with gpt-4
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt.
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Explain this document to me
```
Fragments can also be added *during* a chat conversation using the `!fragment <my_fragment>` command.
```bash
Chatting with gpt-4
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt.
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> !fragment https://llm.datasette.io/en/stable/fragments.html
```
This can be combined with `!multi`:
```bash
> !multi
Explain the difference between fragments and templates to me
!fragment https://llm.datasette.io/en/stable/fragments.html https://llm.datasette.io/en/stable/templates.html
!end
```
Any `!fragment` lines found in a prompt created with `!edit` will not be parsed.
(fragments-browsing)=
## Browsing fragments
You can view a truncated version of the fragments you have previously stored in your database with the `llm fragments` command:
```bash
llm fragments
```
The output from that command looks like this:
```yaml
- hash: 0d6e368f9bc21f8db78c01e192ecf925841a957d8b991f5bf9f6239aa4d81815
aliases: []
datetime_utc: '2025-04-06 07:36:53'
source: https://raw.githubusercontent.com/simonw/llm-docs/refs/heads/main/llm/0.22.txt
content: |-
<documents>
<document index="1">
<source>docs/aliases.md</source>
<document_content>
(aliases)=
#...
- hash: 16b686067375182573e2aa16b5bfc1e64d48350232535d06444537e51f1fd60c
aliases: []
datetime_utc: '2025-04-06 23:03:47'
source: simonw/files-to-prompt/pyproject.toml
content: |-
[project]
name = "files-to-prompt"
version = "0.6"
description = "Concatenate a directory full of...
```
Those long `hash` values are IDs that can be used to reference a fragment in the future:
```bash
llm -f 16b686067375182573e2aa16b5bfc1e64d48350232535d06444537e51f1fd60c 'Extract metadata'
```
Use `-q searchterm` one or more times to search for fragments that match a specific set of search terms.
To view the full content of a fragment use `llm fragments show`:
```bash
llm fragments show 0d6e368f9bc21f8db78c01e192ecf925841a957d8b991f5bf9f6239aa4d81815
```
(fragments-aliases)=
## Setting aliases for fragments
You can assign aliases to fragments that you use often using the `llm fragments set` command:
```bash
llm fragments set mydocs ./docs.md
```
To remove an alias, use `llm fragments remove`:
```bash
llm fragments remove mydocs
```
You can then use that alias in place of the fragment hash ID:
```bash
llm -f mydocs 'How do I access metadata?'
```
Use `llm fragments --aliases` to see a full list of fragments that have been assigned aliases:
```bash
llm fragments --aliases
```
(fragments-logs)=
## Viewing fragments in your logs
The `llm logs` command lists the fragments that were used for a prompt. By default these are listed as fragment hash IDs, but you can use the `--expand` option to show the full content of each fragment.
This command will show the expanded fragments for your most recent conversation:
```bash
llm logs -c --expand
```
You can filter for logs that used a specific fragment using the `-f/--fragment` option:
```bash
llm logs -c -f 0d6e368f9bc21f8db78c01e192ecf925841a957d8b991f5bf9f6239aa4d81815
```
This accepts URLs, file paths, aliases, and hash IDs.
Multiple `-f` options will return responses that used **all** of the specified fragments.
Fragments are returned by `llm logs --json` as well. By default these are truncated but you can add the `-e/--expand` option to show the full content of each fragment.
```bash
llm logs -c --json --expand
```
(fragments-plugins)=
## Using fragments from plugins
LLM plugins can provide custom fragment loaders which do useful things.
One example is the [llm-fragments-github plugin](https://github.com/simonw/llm-fragments-github). This can convert the files from a public GitHub repository into a list of fragments, allowing you to ask questions about the full repository.
Here's how to try that out:
```bash
llm install llm-fragments-github
llm -f github:simonw/s3-credentials 'Suggest new features for this tool'
```
This plugin turns a single call to `-f github:simonw/s3-credentials` into multiple fragments, one for every text file in the [simonw/s3-credentials](https://github.com/simonw/s3-credentials) GitHub repository.
Running `llm logs -c` will show that this prompt incorporated 26 fragments, one for each file.
Running `llm logs -c --usage --expand` (shortcut: `llm logs -cue`) includes token usage information and turns each fragment ID into a full copy of that file. [Here's the output of that command](https://gist.github.com/simonw/c9bbbc5f6560b01f4b7882ac0194fb25).
Fragment plugins can return {ref}`attachments <usage-attachments>` (such as images) as well.
See the {ref}`register_fragment_loaders() plugin hook <plugin-hooks-register-fragment-loaders>` documentation for details on writing your own custom fragment plugin.
(fragments-loaders)=
## Listing available fragment prefixes
The `llm fragments loaders` command shows all prefixes that have been installed by plugins, along with their documentation:
```bash
llm install llm-fragments-github
llm fragments loaders
```
Example output:
```
github:
Load files from a GitHub repository as fragments
Argument is a GitHub repository URL or username/repository
issue:
Fetch GitHub issue and comments as Markdown
Argument is either "owner/repo/NUMBER"
or "https://github.com/owner/repo/issues/NUMBER"
```
docs/help.md
+1088
View File
File diff suppressed because it is too large Load Diff
docs/index.md
+133
View File
@@ -0,0 +1,133 @@
# LLM
[![GitHub repo](https://img.shields.io/badge/github-repo-green)](https://github.com/simonw/llm)
[![PyPI](https://img.shields.io/pypi/v/llm.svg)](https://pypi.org/project/llm/)
[![Changelog](https://img.shields.io/github/v/release/simonw/llm?include_prereleases&label=changelog)](https://llm.datasette.io/en/stable/changelog.html)
[![Tests](https://github.com/simonw/llm/workflows/Test/badge.svg)](https://github.com/simonw/llm/actions?query=workflow%3ATest)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm/blob/main/LICENSE)
[![Discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://datasette.io/discord-llm)
[![Homebrew](https://img.shields.io/homebrew/installs/dy/llm?color=yellow&label=homebrew&logo=homebrew)](https://formulae.brew.sh/formula/llm)
A CLI tool and Python library for interacting with **OpenAI**, **Anthropic's Claude**, **Google's Gemini**, **Meta's Llama** and dozens of other Large Language Models, both via remote APIs and with models that can be installed and run on your own machine.
Watch **[Language models on the command-line](https://www.youtube.com/watch?v=QUXQNi6jQ30)** on YouTube for a demo or [read the accompanying detailed notes](https://simonwillison.net/2024/Jun/17/cli-language-models/).
With LLM you can:
- {ref}`Run prompts from the command-line <usage-executing-prompts>`
- {ref}`Store prompts and responses in SQLite <logging>`
- {ref}`Generate and store embeddings <embeddings>`
- {ref}`Extract structured content from text and images <schemas>`
- {ref}`Grant models the ability to execute tools <tools>`
- ... and much, much more
## Quick start
First, install LLM using `pip` or Homebrew or `pipx` or `uv`:
```bash
pip install llm
```
Or with Homebrew (see {ref}`warning note <homebrew-warning>`):
```bash
brew install llm
```
Or with [pipx](https://pypa.github.io/pipx/):
```bash
pipx install llm
```
Or with [uv](https://docs.astral.sh/uv/guides/tools/)
```bash
uv tool install llm
```
If you have an [OpenAI API key](https://platform.openai.com/api-keys) key you can run this:
```bash
# Paste your OpenAI API key into this
llm keys set openai
# Run a prompt (with the default gpt-4o-mini model)
llm "Ten fun names for a pet pelican"
# Extract text from an image
llm "extract text" -a scanned-document.jpg
# Use a system prompt against a file
cat myfile.py | llm -s "Explain this code"
```
Run prompts against [Gemini](https://aistudio.google.com/apikey) or [Anthropic](https://console.anthropic.com/) with their respective plugins:
```bash
llm install llm-gemini
llm keys set gemini
# Paste Gemini API key here
llm -m gemini-2.0-flash 'Tell me fun facts about Mountain View'
llm install llm-anthropic
llm keys set anthropic
# Paste Anthropic API key here
llm -m claude-4-opus 'Impress me with wild facts about turnips'
```
You can also {ref}`install a plugin <installing-plugins>` to access models that can run on your local device. If you use [Ollama](https://ollama.com/):
```bash
# Install the plugin
llm install llm-ollama
# Download and run a prompt against the Orca Mini 7B model
ollama pull llama3.2:latest
llm -m llama3.2:latest 'What is the capital of France?'
```
To start {ref}`an interactive chat <usage-chat>` with a model, use `llm chat`:
```bash
llm chat -m gpt-4.1
```
```
Chatting with gpt-4.1
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt.
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Tell me a joke about a pelican
Why don't pelicans like to tip waiters?
Because they always have a big bill!
```
More background on this project:
- [llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/)
- [The LLM CLI tool now supports self-hosted language models via plugins](https://simonwillison.net/2023/Jul/12/llm/)
- [LLM now provides tools for working with embeddings](https://simonwillison.net/2023/Sep/4/llm-embeddings/)
- [Build an image search engine with llm-clip, chat with models with llm chat](https://simonwillison.net/2023/Sep/12/llm-clip-and-chat/)
- [You can now run prompts against images, audio and video in your terminal using LLM](https://simonwillison.net/2024/Oct/29/llm-multi-modal/)
- [Structured data extraction from unstructured content using LLM schemas](https://simonwillison.net/2025/Feb/28/llm-schemas/)
- [Long context support in LLM 0.24 using fragments and template plugins](https://simonwillison.net/2025/Apr/7/long-context-llm/)
See also [the llm tag](https://simonwillison.net/tags/llm/) on my blog.
## Contents
```{toctree}
---
maxdepth: 3
---
setup
usage
openai-models
other-models
tools
schemas
templates
fragments
aliases
embeddings/index
plugins/index
python-api
logging
related-tools
help
contributing
```
```{toctree}
---
maxdepth: 1
---
changelog
```
docs/logging.md
+425
View File
@@ -0,0 +1,425 @@
(logging)=
# Logging to SQLite
`llm` defaults to logging all prompts and responses to a SQLite database.
You can find the location of that database using the `llm logs path` command:
```bash
llm logs path
```
On my Mac that outputs:
```
/Users/simon/Library/Application Support/io.datasette.llm/logs.db
```
This will differ for other operating systems.
To avoid logging an individual prompt, pass `--no-log` or `-n` to the command:
```bash
llm 'Ten names for cheesecakes' -n
```
To turn logging by default off:
```bash
llm logs off
```
If you've turned off logging you can still log an individual prompt and response by adding `--log`:
```bash
llm 'Five ambitious names for a pet pterodactyl' --log
```
To turn logging by default back on again:
```bash
llm logs on
```
To see the status of the logs database, run this:
```bash
llm logs status
```
Example output:
```
Logging is ON for all prompts
Found log database at /Users/simon/Library/Application Support/io.datasette.llm/logs.db
Number of conversations logged: 33
Number of responses logged: 48
Database file size: 19.96MB
```
(logging-view)=
## Viewing the logs
You can view the logs using the `llm logs` command:
```bash
llm logs
```
This will output the three most recent logged items in Markdown format, showing both the prompt and the response formatted using Markdown.
To get back just the most recent prompt response as plain text, add `-r/--response`:
```bash
llm logs -r
```
Use `-x/--extract` to extract and return the first fenced code block from the selected log entries:
```bash
llm logs --extract
```
Or `--xl/--extract-last` for the last fenced code block:
```bash
llm logs --extract-last
```
Add `--json` to get the log messages in JSON instead:
```bash
llm logs --json
```
Add `-n 10` to see the ten most recent items:
```bash
llm logs -n 10
```
Or `-n 0` to see everything that has ever been logged:
```bash
llm logs -n 0
```
You can truncate the display of the prompts and responses using the `-t/--truncate` option. This can help make the JSON output more readable - though the `--short` option is usually better.
```bash
llm logs -n 1 -t --json
```
Example output:
```json
[
{
"id": "01jm8ec74wxsdatyn5pq1fp0s5",
"model": "anthropic/claude-3-haiku-20240307",
"prompt": "hi",
"system": null,
"prompt_json": null,
"response": "Hello! How can I assist you today?",
"conversation_id": "01jm8ec74taftdgj2t4zra9z0j",
"duration_ms": 560,
"datetime_utc": "2025-02-16T22:34:30.374882+00:00",
"input_tokens": 8,
"output_tokens": 12,
"token_details": null,
"conversation_name": "hi",
"conversation_model": "anthropic/claude-3-haiku-20240307",
"attachments": []
}
]
```
(logging-short)=
### -s/--short mode
Use `-s/--short` to see a shortened YAML log with truncated prompts and no responses:
```bash
llm logs -n 2 --short
```
Example output:
```yaml
- model: deepseek-reasoner
datetime: '2025-02-02T06:39:53'
conversation: 01jk2pk05xq3d0vgk0202zrsg1
prompt: H01 There are five huts. H02 The Scotsman lives in the purple hut. H03 The Welshman owns the parrot. H04 Kombucha is...
- model: o3-mini
datetime: '2025-02-02T19:03:05'
conversation: 01jk40qkxetedzpf1zd8k9bgww
system: Formatting re-enabled. Write a detailed README with extensive usage examples.
prompt: <documents> <document index="1"> <source>./Cargo.toml</source> <document_content> [package] name = "py-limbo" version...
```
Include `-u/--usage` to include token usage information:
```bash
llm logs -n 1 --short --usage
```
Example output:
```yaml
- model: o3-mini
datetime: '2025-02-16T23:00:56'
conversation: 01jm8fxxnef92n1663c6ays8xt
system: Produce Python code that demonstrates every possible usage of yaml.dump
with all of the arguments it can take, especi...
prompt: <documents> <document index="1"> <source>./setup.py</source> <document_content>
NAME = 'PyYAML' VERSION = '7.0.0.dev0...
usage:
input: 74793
output: 3550
details:
completion_tokens_details:
reasoning_tokens: 2240
```
(logging-conversation)=
### Logs for a conversation
To view the logs for the most recent {ref}`conversation <usage-conversation>` you have had with a model, use `-c`:
```bash
llm logs -c
```
To see logs for a specific conversation based on its ID, use `--cid ID` or `--conversation ID`:
```bash
llm logs --cid 01h82n0q9crqtnzmf13gkyxawg
```
(logging-search)=
### Searching the logs
You can search the logs for a search term in the `prompt` or the `response` columns.
```bash
llm logs -q 'cheesecake'
```
The most relevant results will be shown first.
To switch to sorting with most recent first, add `-l/--latest`. This can be combined with `-n` to limit the number of results shown:
```bash
llm logs -q 'cheesecake' -l -n 3
```
(logging-filter-id)=
### Filtering past a specific ID
If you want to retrieve all of the logs that were recorded since a specific response ID you can do so using these options:
- `--id-gt $ID` - every record with an ID greater than $ID
- `--id-gte $ID` - every record with an ID greater than or equal to $ID
IDs are always issued in ascending order by time, so this provides a useful way to see everything that has happened since a particular record.
This can be particularly useful when {ref}`working with schema data <schemas-logs>`, where you might want to access every record that you have created using a specific `--schema` but exclude records you have previously processed.
(logging-filter-model)=
### Filtering by model
You can filter to logs just for a specific model (or model alias) using `-m/--model`:
```bash
llm logs -m chatgpt
```
(logging-filter-fragments)=
### Filtering by prompts that used specific fragments
The `-f/--fragment X` option will filter for just responses that were created using the specified {ref}`fragment <usage-fragments>` hash or alias or URL or filename.
Fragments are displayed in the logs as their hash ID. Add `-e/--expand` to display fragments as their full content - this option works for both the default Markdown and the `--json` mode:
```bash
llm logs -f https://llm.datasette.io/robots.txt --expand
```
You can display just the content for a specific fragment hash ID (or alias) using the `llm fragments show` command:
```bash
llm fragments show 993fd38d898d2b59fd2d16c811da5bdac658faa34f0f4d411edde7c17ebb0680
```
If you provide multiple fragments you will get back responses that used _all_ of those fragments.
(logging-filter-tools)=
### Filtering by prompts that used specific tools
You can filter for responses that used tools from specific fragments with the `--tool/-T` option:
```bash
llm logs -T simple_eval
```
This will match responses that involved a _result_ from that tool. If the tool was not executed it will not be included in the filtered responses.
Pass `--tool/-T` multiple times for responses that used all of the specified tools.
Use the `llm logs --tools` flag to see _all_ responses that involved at least one tool result, including from `--functions`:
```bash
llm logs --tools
```
(logging-filter-schemas)=
### Browsing data collected using schemas
The `--schema X` option can be used to view responses that used the specified schema, using any of the {ref}`ways to specify a schema <schemas-specify>`:
```bash
llm logs --schema 'name, age int, bio'
```
This can be combined with `--data` and `--data-array` and `--data-key` to extract just the returned JSON data - consult the {ref}`schemas documentation <schemas-logs>` for details.
(logging-datasette)=
## Browsing logs using Datasette
You can also use [Datasette](https://datasette.io/) to browse your logs like this:
```bash
datasette "$(llm logs path)"
```
(logging-backup)=
## Backing up your database
You can backup your logs to another file using the `llm logs backup` command:
```bash
llm logs backup /tmp/backup.db
```
This uses SQLite [VACUUM INTO](https://sqlite.org/lang_vacuum.html#vacuum_with_an_into_clause) under the hood.
(logging-sql-schema)=
## SQL schema
Here's the SQL schema used by the `logs.db` database:
<!-- [[[cog
import cog
from llm.migrations import migrate
import sqlite_utils
import re
db = sqlite_utils.Database(memory=True)
migrate(db)
def cleanup_sql(sql):
first_line = sql.split('(')[0]
inner = re.search(r'\((.*)\)', sql, re.DOTALL).group(1)
columns = [l.strip() for l in inner.split(',')]
return first_line + '(\n ' + ',\n '.join(columns) + '\n);'
cog.out("```sql\n")
for table in (
"conversations", "schemas", "responses", "responses_fts", "attachments", "prompt_attachments",
"fragments", "fragment_aliases", "prompt_fragments", "system_fragments", "tools",
"tool_responses", "tool_calls", "tool_results", "tool_instances"
):
schema = db[table].schema
cog.out(format(cleanup_sql(schema)))
cog.out("\n")
cog.out("```\n")
]]] -->
```sql
CREATE TABLE "conversations" (
"id" TEXT PRIMARY KEY,
"name" TEXT,
"model" TEXT
);
CREATE TABLE "schemas" (
"id" TEXT PRIMARY KEY,
"content" TEXT
);
CREATE TABLE "responses" (
"id" TEXT PRIMARY KEY,
"model" TEXT,
"prompt" TEXT,
"system" TEXT,
"prompt_json" TEXT,
"options_json" TEXT,
"response" TEXT,
"response_json" TEXT,
"conversation_id" TEXT REFERENCES "conversations"("id"),
"duration_ms" INTEGER,
"datetime_utc" TEXT,
"input_tokens" INTEGER,
"output_tokens" INTEGER,
"token_details" TEXT,
"schema_id" TEXT REFERENCES "schemas"("id"),
"resolved_model" TEXT,
"reasoning" TEXT
);
CREATE VIRTUAL TABLE "responses_fts" USING FTS5 (
"prompt",
"response",
content="responses"
);
CREATE TABLE "attachments" (
"id" TEXT PRIMARY KEY,
"type" TEXT,
"path" TEXT,
"url" TEXT,
"content" BLOB
);
CREATE TABLE "prompt_attachments" (
"response_id" TEXT REFERENCES "responses"("id"),
"attachment_id" TEXT REFERENCES "attachments"("id"),
"order" INTEGER,
PRIMARY KEY ("response_id",
"attachment_id")
);
CREATE TABLE "fragments" (
"id" INTEGER PRIMARY KEY,
"hash" TEXT,
"content" TEXT,
"datetime_utc" TEXT,
"source" TEXT
);
CREATE TABLE "fragment_aliases" (
"alias" TEXT PRIMARY KEY,
"fragment_id" INTEGER REFERENCES "fragments"("id")
);
CREATE TABLE "prompt_fragments" (
"response_id" TEXT REFERENCES "responses"("id"),
"fragment_id" INTEGER REFERENCES "fragments"("id"),
"order" INTEGER,
PRIMARY KEY ("response_id",
"fragment_id",
"order")
);
CREATE TABLE "system_fragments" (
"response_id" TEXT REFERENCES "responses"("id"),
"fragment_id" INTEGER REFERENCES "fragments"("id"),
"order" INTEGER,
PRIMARY KEY ("response_id",
"fragment_id",
"order")
);
CREATE TABLE "tools" (
"id" INTEGER PRIMARY KEY,
"hash" TEXT,
"name" TEXT,
"description" TEXT,
"input_schema" TEXT,
"plugin" TEXT
);
CREATE TABLE "tool_responses" (
"tool_id" INTEGER REFERENCES "tools"("id"),
"response_id" TEXT REFERENCES "responses"("id"),
PRIMARY KEY ("tool_id",
"response_id")
);
CREATE TABLE "tool_calls" (
"id" INTEGER PRIMARY KEY,
"response_id" TEXT REFERENCES "responses"("id"),
"tool_id" INTEGER REFERENCES "tools"("id"),
"name" TEXT,
"arguments" TEXT,
"tool_call_id" TEXT
);
CREATE TABLE "tool_results" (
"id" INTEGER PRIMARY KEY,
"response_id" TEXT REFERENCES "responses"("id"),
"tool_id" INTEGER REFERENCES "tools"("id"),
"name" TEXT,
"output" TEXT,
"tool_call_id" TEXT,
"instance_id" INTEGER REFERENCES "tool_instances"("id"),
"exception" TEXT
);
CREATE TABLE "tool_instances" (
"id" INTEGER PRIMARY KEY,
"plugin" TEXT,
"name" TEXT,
"arguments" TEXT
);
```
<!-- [[[end]]] -->
`responses_fts` configures [SQLite full-text search](https://www.sqlite.org/fts5.html) against the `prompt` and `response` columns in the `responses` table.
docs/openai-models.md
+188
View File
@@ -0,0 +1,188 @@
(openai-models)=
# OpenAI models
LLM ships with a default plugin for talking to OpenAI's API. OpenAI offer both language models and embedding models, and LLM can access both types.
(openai-models-configuration)=
## Configuration
All OpenAI models are accessed using an API key. You can obtain one from [the API keys page](https://platform.openai.com/api-keys) on their site.
Once you have created a key, configure LLM to use it by running:
```bash
llm keys set openai
```
Then paste in the API key.
(openai-models-language)=
## OpenAI language models
Run `llm models` for a full list of available models. The OpenAI models supported by LLM are:
<!-- [[[cog
from click.testing import CliRunner
from llm.cli import cli
result = CliRunner().invoke(cli, ["models", "list"])
models = [line for line in result.output.split("\n") if line.startswith("OpenAI ")]
cog.out("```\n{}\n```".format("\n".join(models)))
]]] -->
```
OpenAI Chat: gpt-4o (aliases: 4o)
OpenAI Chat: chatgpt-4o-latest (aliases: chatgpt-4o)
OpenAI Chat: gpt-4o-mini (aliases: 4o-mini)
OpenAI Chat: gpt-4o-audio-preview
OpenAI Chat: gpt-4o-audio-preview-2024-12-17
OpenAI Chat: gpt-4o-audio-preview-2024-10-01
OpenAI Chat: gpt-4o-mini-audio-preview
OpenAI Chat: gpt-4o-mini-audio-preview-2024-12-17
OpenAI Chat: gpt-4.1 (aliases: 4.1)
OpenAI Chat: gpt-4.1-mini (aliases: 4.1-mini)
OpenAI Chat: gpt-4.1-nano (aliases: 4.1-nano)
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Chat: gpt-4-1106-preview
OpenAI Chat: gpt-4-0125-preview
OpenAI Chat: gpt-4-turbo-2024-04-09
OpenAI Chat: gpt-4-turbo (aliases: gpt-4-turbo-preview, 4-turbo, 4t)
OpenAI Chat: gpt-4.5-preview-2025-02-27
OpenAI Chat: gpt-4.5-preview (aliases: gpt-4.5)
OpenAI Responses: o1
OpenAI Responses: o1-2024-12-17
OpenAI Chat: o1-preview
OpenAI Chat: o1-mini
OpenAI Responses: o3-mini
OpenAI Responses: o3
OpenAI Responses: o4-mini
OpenAI Responses: gpt-5
OpenAI Responses: gpt-5-mini
OpenAI Responses: gpt-5-nano
OpenAI Responses: gpt-5-2025-08-07
OpenAI Responses: gpt-5-mini-2025-08-07
OpenAI Responses: gpt-5-nano-2025-08-07
OpenAI Responses: gpt-5.1
OpenAI Responses: gpt-5.1-chat-latest
OpenAI Responses: gpt-5.2
OpenAI Responses: gpt-5.2-chat-latest
OpenAI Responses: gpt-5.4
OpenAI Responses: gpt-5.4-2026-03-05
OpenAI Responses: gpt-5.4-mini
OpenAI Responses: gpt-5.4-mini-2026-03-17
OpenAI Responses: gpt-5.4-nano
OpenAI Responses: gpt-5.4-nano-2026-03-17
OpenAI Responses: gpt-5.5
OpenAI Responses: gpt-5.5-2026-04-23
OpenAI Completion: gpt-3.5-turbo-instruct (aliases: 3.5-instruct, chatgpt-instruct)
```
<!-- [[[end]]] -->
See [the OpenAI models documentation](https://platform.openai.com/docs/models) for details of each of these.
`gpt-4o-mini` (aliased to `4o-mini`) is the least expensive model, and is the default for if you don't specify a model at all. Consult [OpenAI's model documentation](https://platform.openai.com/docs/models) for details of the other models.
[o1-pro](https://platform.openai.com/docs/models/o1-pro) is not available through the Chat Completions API used by LLM's default OpenAI plugin. You can install the new [llm-openai-plugin](https://github.com/simonw/llm-openai-plugin) plugin to access that model.
## Model features
The following features work with OpenAI models:
- {ref}`System prompts <usage-system-prompts>` can be used to provide instructions that have a higher weight than the prompt itself.
- {ref}`Attachments <usage-attachments>`. Many OpenAI models support image inputs - check which ones using `llm models --options`. Any model that accepts images can also accept PDFs.
- {ref}`Schemas <usage-schemas>` can be used to influence the JSON structure of the model output.
- {ref}`Model options <usage-model-options>` can be used to set parameters like `temperature`. Use `llm models --options` for a full list of supported options.
(openai-models-embedding)=
## OpenAI embedding models
Run `llm embed-models` for a list of {ref}`embedding models <embeddings>`. The following OpenAI embedding models are supported by LLM:
```
ada-002 (aliases: ada, oai)
3-small
3-large
3-small-512
3-large-256
3-large-1024
```
The `3-small` model is currently the most inexpensive. `3-large` costs more but is more capable - see [New embedding models and API updates](https://openai.com/blog/new-embedding-models-and-api-updates) on the OpenAI blog for details and benchmarks.
An important characteristic of any embedding model is the size of the vector it returns. Smaller vectors cost less to store and query, but may be less accurate.
OpenAI `3-small` and `3-large` vectors can be safely truncated to lower dimensions without losing too much accuracy. The `-int` models provided by LLM are pre-configured to do this, so `3-large-256` is the `3-large` model truncated to 256 dimensions.
The vector size of the supported OpenAI embedding models are as follows:
| Model | Size |
| --- | --- |
| ada-002 | 1536 |
| 3-small | 1536 |
| 3-large | 3072 |
| 3-small-512 | 512 |
| 3-large-256 | 256 |
| 3-large-1024 | 1024 |
(openai-completion-models)=
## OpenAI completion models
The `gpt-3.5-turbo-instruct` model is a little different - it is a completion model rather than a chat model, described in [the OpenAI completions documentation](https://platform.openai.com/docs/api-reference/completions/create).
Completion models can be called with the `-o logprobs 3` option (not supported by chat models) which will cause LLM to store 3 log probabilities for each returned token in the SQLite database. Consult [this issue](https://github.com/simonw/llm/issues/284#issuecomment-1724772704) for details on how to read these values.
(openai-extra-models)=
## Adding more OpenAI models
OpenAI occasionally release new models with new names. LLM aims to ship new releases to support these, but you can also configure them directly, by adding them to a `extra-openai-models.yaml` configuration file.
Run this command to find the directory in which this file should be created:
```bash
dirname "$(llm logs path)"
```
On my Mac laptop I get this:
```
~/Library/Application Support/io.datasette.llm
```
Create a file in that directory called `extra-openai-models.yaml`.
Let's say OpenAI have just released the `gpt-3.5-turbo-0613` model and you want to use it, despite LLM not yet shipping support. You could configure that by adding this to the file:
```yaml
- model_id: gpt-3.5-turbo-0613
model_name: gpt-3.5-turbo-0613
aliases: ["0613"]
```
The `model_id` is the identifier that will be recorded in the LLM logs. You can use this to specify the model, or you can optionally include a list of aliases for that model. The `model_name` is the actual model identifier that will be passed to the API, which must match exactly what the API expects.
If the model is a completion model (such as `gpt-3.5-turbo-instruct`) add `completion: true` to the configuration.
If the model supports structured extraction using json_schema, add `supports_schema: true` to the configuration.
For reasoning models like `o1` or `o3-mini` add `reasoning: true`.
With this configuration in place, the following command should run a prompt against the new model:
```bash
llm -m 0613 'What is the capital of France?'
```
Run `llm models` to confirm that the new model is now available:
```bash
llm models
```
Example output:
```
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Chat: gpt-3.5-turbo-0613 (aliases: 0613)
```
Running `llm logs -n 1` should confirm that the prompt and response has been correctly logged to the database.
docs/other-models.md
+77
View File
@@ -0,0 +1,77 @@
(other-models)=
# Other models
LLM supports OpenAI models by default. You can install {ref}`plugins <plugins>` to add support for other models. You can also add additional OpenAI-API-compatible models {ref}`using a configuration file <openai-extra-models>`.
## Installing and using a local model
{ref}`LLM plugins <plugins>` can provide local models that run on your machine.
To install **[llm-gpt4all](https://github.com/simonw/llm-gpt4all)**, providing 17 models from the [GPT4All](https://gpt4all.io/) project, run this:
```bash
llm install llm-gpt4all
```
Run `llm models` to see the expanded list of available models.
To run a prompt through one of the models from GPT4All specify it using `-m/--model`:
```bash
llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
```
The model will be downloaded and cached the first time you use it.
Check the {ref}`plugin directory <plugin-directory>` for the latest list of available plugins for other models.
(openai-compatible-models)=
## OpenAI-compatible models
Projects such as [LocalAI](https://localai.io/) offer a REST API that imitates the OpenAI API but can be used to run other models, including models that can be installed on your own machine. These can be added using the same configuration mechanism.
The `model_id` is the name LLM will use for the model. The `model_name` is the name which needs to be passed to the API - this might differ from the `model_id`, especially if the `model_id` could potentially clash with other installed models.
The `api_base` key can be used to point the OpenAI client library at a different API endpoint.
To add the `orca-mini-3b` model hosted by a local installation of [LocalAI](https://localai.io/), add this to your `extra-openai-models.yaml` file:
```yaml
- model_id: orca-openai-compat
model_name: orca-mini-3b.ggmlv3
api_base: "http://localhost:8080"
```
If the `api_base` is set, the existing configured `openai` API key will not be sent by default.
You can set `api_key_name` to the name of a key stored using the {ref}`api-keys` feature.
Other keys you can use here:
- `completion: true` for completion models that should use the `/completion` endpoint as opposed to `/completion/chat`
- `supports_tools: true` for models that support tool calling
- `can_stream: false` to disable streaming mode for models that cannot stream
- `supports_schema: true` for models that support JSON structured schema output
- `vision: true` for models that can accept images as input
- `audio: true` for models that accept audio attachments
Having configured the model like this, run `llm models --options -m MODEL_ID` to check that it installed correctly. You can then run prompts against it like so:
```bash
llm -m orca-openai-compat 'What is the capital of France?'
```
And confirm they were logged correctly with:
```bash
llm logs -n 1
```
### Extra HTTP headers
Some providers such as [openrouter.ai](https://openrouter.ai/docs) may require the setting of additional HTTP headers. You can set those using the `headers:` key like this:
```yaml
- model_id: claude
model_name: anthropic/claude-2
api_base: "https://openrouter.ai/api/v1"
api_key_name: openrouter
headers:
HTTP-Referer: "https://llm.datasette.io/"
X-Title: LLM
```
docs/plugins/advanced-model-plugins.md
+614
View File
@@ -0,0 +1,614 @@
(advanced-model-plugins)=
# Advanced model plugins
The {ref}`model plugin tutorial <tutorial-model-plugin>` covers the basics of developing a plugin that adds support for a new model. This document covers more advanced topics.
Features to consider for your model plugin include:
- {ref}`Accepting API keys <advanced-model-plugins-api-keys>` using the standard mechanism that incorporates `llm keys set`, environment variables and support for passing an explicit key to the model.
- Including support for {ref}`Async models <advanced-model-plugins-async>` that can be used with Python's `asyncio` library.
- Support for {ref}`structured output <advanced-model-plugins-schemas>` using JSON schemas.
- Support for {ref}`tools <advanced-model-plugins-tools>`.
- Handling {ref}`attachments <advanced-model-plugins-attachments>` (images, audio and more) for multi-modal models.
- Tracking {ref}`token usage <advanced-model-plugins-usage>` for models that charge by the token.
(advanced-model-plugins-lazy)=
## Tip: lazily load expensive dependencies
If your plugin depends on an expensive library such as [PyTorch](https://pytorch.org/) you should avoid importing that dependency (or a dependency that uses that dependency) at the top level of your module. Expensive imports in plugins mean that even simple commands like `llm --help` can take a long time to run.
Instead, move those imports to inside the methods that need them. Here's an example [change to llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers/commit/f87df71e8a652a8cb05ad3836a79b815bcbfa64b) that shaved 1.8 seconds off the time it took to run `llm --help`!
(advanced-model-plugins-api-keys)=
## Models that accept API keys
Models that call out to API providers such as OpenAI, Anthropic or Google Gemini usually require an API key.
LLM's API key management mechanism {ref}`is described here <api-keys>`.
If your plugin requires an API key you should subclass the `llm.KeyModel` class instead of the `llm.Model` class. Start your model definition like this:
```python
import llm
class HostedModel(llm.KeyModel):
needs_key = "hosted" # Required
key_env_var = "HOSTED_API_KEY" # Optional
```
This tells LLM that your model requires an API key, which may be saved in the key registry under the key name `hosted` or might also be provided as the `HOSTED_API_KEY` environment variable.
Then when you define your `execute()` method it should take an extra `key=` parameter like this:
```python
def execute(self, prompt, stream, response, conversation, key=None):
# key= here will be the API key to use
```
LLM will pass in the key from the environment variable, key registry or that has been passed to LLM as the `--key` command-line option or the `model.prompt(..., key=)` parameter.
(advanced-model-plugins-async)=
## Async models
Plugins can optionally provide an asynchronous version of their model, suitable for use with Python [asyncio](https://docs.python.org/3/library/asyncio.html). This is particularly useful for remote models accessible by an HTTP API.
The async version of a model subclasses `llm.AsyncModel` instead of `llm.Model`. It must implement an `async def execute()` async generator method instead of `def execute()`.
This example shows a subset of the OpenAI default plugin illustrating how this method might work:
```python
from typing import AsyncGenerator
import llm
class MyAsyncModel(llm.AsyncModel):
# This can duplicate the model_id of the sync model:
model_id = "my-model-id"
async def execute(
self, prompt, stream, response, conversation=None
) -> AsyncGenerator[str, None]:
if stream:
completion = await client.chat.completions.create(
model=self.model_id,
messages=messages,
stream=True,
)
async for chunk in completion:
yield chunk.choices[0].delta.content
else:
completion = await client.chat.completions.create(
model=self.model_name or self.model_id,
messages=messages,
stream=False,
)
if completion.choices[0].message.content is not None:
yield completion.choices[0].message.content
```
If your model takes an API key you should instead subclass `llm.AsyncKeyModel` and have a `key=` parameter on your `.execute()` method:
```python
class MyAsyncModel(llm.AsyncKeyModel):
...
async def execute(
self, prompt, stream, response, conversation=None, key=None
) -> AsyncGenerator[str, None]:
```
This async model instance should then be passed to the `register()` method in the `register_models()` plugin hook:
```python
@hookimpl
def register_models(register):
register(
MyModel(), MyAsyncModel(), aliases=("my-model-aliases",)
)
```
The `prompt` object passed to your `execute()` method is an instance of {class}`~llm.Prompt`:
```{eval-rst}
.. autoclass:: llm.Prompt
:members: prompt, system
:exclude-members: model, options
```
(advanced-model-plugins-schemas)=
## Supporting schemas
If your model supports {ref}`structured output <schemas>` against a defined JSON schema you can implement support by first adding `supports_schema = True` to the class:
```python
class MyModel(llm.KeyModel):
...
support_schema = True
```
And then adding code to your `.execute()` method that checks for `prompt.schema` and, if it is present, uses that to prompt the model.
`prompt.schema` will always be a Python dictionary representing a JSON schema, even if the user passed in a Pydantic model class.
Check the [llm-gemini](https://github.com/simonw/llm-gemini) and [llm-anthropic](https://github.com/simonw/llm-anthropic) plugins for example of this pattern in action.
(advanced-model-plugins-tools)=
## Supporting tools
Adding {ref}`tools support <tools>` involves several steps:
1. Add `supports_tools = True` to your model class.
2. If `prompt.tools` is populated, turn that list of `llm.Tool` objects into the correct format for your model.
3. Look out for requests to call tools in the responses from your model. Call `response.add_tool_call(llm.ToolCall(...))` for each of those. This should work for streaming and non-streaming and async and non-async cases. Pass the provider's tool call ID as `tool_call_id=` if there is one; if you omit it LLM synthesizes a unique `tc_`-prefixed id, since consumers rely on every tool call having one.
4. If your prompt has a `prompt.tool_results` list, pass the information from those `llm.ToolResult` objects to your model.
5. Include `prompt.tools` and `prompt.tool_results` and tool calls from `response.tool_calls_or_raise()` in the conversation history constructed by your plugin.
6. Make sure your code is OK with prompts that do not have `prompt.prompt` set to a value, since they may be carrying exclusively the results of a tool call.
This [commit to llm-gemini](https://github.com/simonw/llm-gemini/commit/a7f1096cfbb733018eb41c29028a8cc6160be298) implementing tools helps demonstrate what this looks like for a real plugin.
Here are the relevant dataclasses:
```{eval-rst}
.. autoclass:: llm.Tool
.. autoclass:: llm.ToolCall
.. autoclass:: llm.ToolResult
```
(advanced-model-plugins-attachments)=
## Attachments for multi-modal models
Models such as GPT-4o, Claude 3.5 Sonnet and Google's Gemini 1.5 are multi-modal: they accept input in the form of images and maybe even audio, video and other formats.
LLM calls these **attachments**. Models can specify the types of attachments they accept and then implement special code in the `.execute()` method to handle them.
See {ref}`the Python attachments documentation <python-api-attachments>` for details on using attachments in the Python API.
### Specifying attachment types
A `Model` subclass can list the types of attachments it accepts by defining a `attachment_types` class attribute:
```python
class NewModel(llm.Model):
model_id = "new-model"
attachment_types = {
"image/png",
"image/jpeg",
"image/webp",
"image/gif",
}
```
These content types are detected when an attachment is passed to LLM using `llm -a filename`, or can be specified by the user using the `--attachment-type filename image/png` option.
**Note:** MP3 files will have their attachment type detected as `audio/mpeg`, not `audio/mp3`.
LLM will use the `attachment_types` attribute to validate that provided attachments should be accepted before passing them to the model.
### Handling attachments
The `prompt` object passed to the `execute()` method will have an `attachments` attribute containing a list of `Attachment` objects provided by the user.
An `Attachment` instance has the following properties:
- `url (str)`: The URL of the attachment, if it was provided as a URL
- `path (str)`: The resolved file path of the attachment, if it was provided as a file
- `type (str)`: The content type of the attachment, if it was provided
- `content (bytes)`: The binary content of the attachment, if it was provided
Generally only one of `url`, `path` or `content` will be set.
You should usually access the type and the content through one of these methods:
- `attachment.resolve_type() -> str`: Returns the `type` if it is available, otherwise attempts to guess the type by looking at the first few bytes of content
- `attachment.content_bytes() -> bytes`: Returns the binary content, which it may need to read from a file or fetch from a URL
- `attachment.base64_content() -> str`: Returns that content as a base64-encoded string
A `id()` method returns a database ID for this content, which is either a SHA256 hash of the binary content or, in the case of attachments hosted at an external URL, a hash of `{"url": url}` instead. This is an implementation detail which you should not need to access directly.
Note that it's possible for a prompt with an attachments to not include a text prompt at all, in which case `prompt.prompt` will be `None`.
Here's how the OpenAI plugin handles attachments, including the case where no `prompt.prompt` was provided:
```python
if not prompt.attachments:
messages.append({"role": "user", "content": prompt.prompt})
else:
attachment_message = []
if prompt.prompt:
attachment_message.append({"type": "text", "text": prompt.prompt})
for attachment in prompt.attachments:
attachment_message.append(_attachment(attachment))
messages.append({"role": "user", "content": attachment_message})
# And the code for creating the attachment message
def _attachment(attachment):
url = attachment.url
base64_content = ""
if not url or attachment.resolve_type().startswith("audio/"):
base64_content = attachment.base64_content()
url = f"data:{attachment.resolve_type()};base64,{base64_content}"
if attachment.resolve_type().startswith("image/"):
return {"type": "image_url", "image_url": {"url": url}}
else:
format_ = "wav" if attachment.resolve_type() == "audio/wav" else "mp3"
return {
"type": "input_audio",
"input_audio": {
"data": base64_content,
"format": format_,
},
}
```
As you can see, it uses `attachment.url` if that is available and otherwise falls back to using the `base64_content()` method to embed the image directly in the JSON sent to the API. For the OpenAI API audio attachments are always included as base64-encoded strings.
### Attachments from previous conversations
Conversation history — including attachments from prior turns — is available on the canonical `prompt.messages` list. See the [next section](#structured-messages-streaming) for how that works.
(structured-messages-streaming)=
## Structured messages and streaming events
The 0.32 alpha introduced a richer contract for plugins than "yield strings":
1. **`execute()` yields `StreamEvent` objects** (or plain `str`, still supported) so text, reasoning (thinking tokens), tool calls, and server-side tool results each surface as their own event type. The framework assembles these into typed `Part` objects.
2. **`build_messages` (or equivalent) reads `prompt.messages`** — a `list[llm.Message]` that is the complete input chain for this turn.
3. **Opaque provider tokens round-trip via `provider_metadata`** — Anthropic thinking signatures, Gemini thought signatures, OpenAI Responses API encrypted reasoning blobs. Plugins stash whatever the API returns, then echo it back on the next request.
**Older plugins still work.** A plugin that still yields plain `str` from `execute()` works unchanged — each string is wrapped as a `StreamEvent(type="text", chunk=...)` internally.
### Yielding StreamEvent from execute()
```python
from llm.parts import StreamEvent
def execute(self, prompt, stream, response, conversation, key=None):
messages = self.build_messages(prompt, conversation)
...
for chunk in provider_sdk.stream(...):
if chunk.type == "text":
yield StreamEvent(type="text", chunk=chunk.text)
elif chunk.type == "thinking":
yield StreamEvent(type="reasoning", chunk=chunk.text)
```
A `StreamEvent` has four frequently-used fields:
- **`type`** — one of `"text"`, `"reasoning"`, `"tool_call_name"`, `"tool_call_args"`, `"tool_result"`.
- **`chunk`** — the text fragment. For tool calls this is the tool name (for `tool_call_name`) or a partial JSON string (for `tool_call_args`).
- **`tool_call_id`** — the provider's id for the tool call, set on `tool_call_name` / `tool_call_args` / `tool_result` events. Also the signal the framework uses to group tool-call events into one `ToolCallPart`.
- **`provider_metadata`** — an optional `dict[str, dict]` namespaced by provider name. Carries opaque data (signatures, encrypted blobs) that must be echoed back on future requests.
Three additional fields exist for special cases:
- **`server_executed: bool`** — set `True` for server-side tool calls (for example, Anthropic web search) and their results. This means the model ran the tool internally as part of responding to the prompt.
- **`tool_name`** — set on `tool_result` events to identify which tool this result came from.
- **`part_index: int | None`** — defaults to `None`, which means "let the framework decide which Part this event belongs to." Pass an explicit integer only when you need to override the default grouping (see [below](#part-index-overrides)).
### How events group into Parts
When you leave `part_index` as `None` (the default), the framework groups events using these rules:
- **Consecutive same-family events concatenate.** Two `text` events in a row become one `TextPart`. Two `reasoning` events in a row become one `ReasoningPart`. A family transition (text → reasoning, or reasoning → text) starts a new Part.
- **Tool calls group by `tool_call_id`.** A `tool_call_name` and any number of `tool_call_args` events sharing a `tool_call_id` combine into one `ToolCallPart` — even if they're interleaved with other events (parallel tool calls).
- **`tool_result` is always its own Part**, paired to the originating call by `tool_call_id`.
| Stream | Resulting Parts |
|-------------------------------------------|----------------------------------------------------------|
| `text` × N | one `TextPart` |
| `reasoning` × N, then `text` × N | `ReasoningPart`, `TextPart` |
| `text`, `tool_call_name`+`args`, `text` | `TextPart`, `ToolCallPart`, `TextPart` |
| Parallel tool calls (interleaved by id) | one `ToolCallPart` per distinct `tool_call_id` |
| `reasoning`, tool call, `reasoning` | `ReasoningPart`, `ToolCallPart`, `ReasoningPart` |
(part-index-overrides)=
### Setting `part_index` explicitly
In rare cases you'll want to override the default grouping:
- **Forcing a single TextPart across non-adjacent text bursts.** If your provider interleaves text deltas with tool calls but you want all the text concatenated into one `TextPart`, pass `part_index=0` on every text event. (The default behavior produces separate `TextPart`s on each side of the tool calls — usually what you want, but not always.)
- **Tool-call args arriving before the id.** If your provider streams args before the `tool_call_id` is known, assign your own index per logical tool call and pass it on each event of that call.
You can mix explicit indices with `None` in the same stream — the framework reserves your explicit values and decides the rest.
(advanced-model-plugins-reasoning-tokens)=
### Reasoning tokens
For streamed reasoning text:
```python
yield StreamEvent(type="reasoning", chunk=text_chunk)
```
Reasoning events that appear before/after text events become distinct `ReasoningPart` and `TextPart` entries in `response.messages` automatically. If your provider emits two thinking blocks separated by a tool call, you'll get two `ReasoningPart`s.
Plugins should respect `prompt.hide_reasoning`. This is set when the caller passes `hide_reasoning=True` to `model.prompt()`, `conversation.prompt()`, `model.chain()`, `conversation.chain()`, or their async counterparts. It is also set by the CLI `-R/--hide-reasoning` option.
`prompt.hide_reasoning` means "hide visible reasoning output", not "disable model reasoning". If your provider requires an explicit request for visible reasoning summaries, do not request those summaries when `prompt.hide_reasoning` is true:
```python
kwargs = {}
if not prompt.hide_reasoning:
kwargs["reasoning"] = {"summary": "auto"}
```
If your provider emits reasoning blocks regardless of request parameters, keep yielding those reasoning events as usual:
```python
if chunk.type == "thinking":
yield StreamEvent(type="reasoning", chunk=chunk.text)
```
LLM's display layers use `prompt.hide_reasoning` to avoid showing those events to the user, while still allowing the framework to persist `ReasoningPart` objects and provider metadata for logs, serialization, and future turns.
### Tool calls
Each tool call emits two event types sharing a `tool_call_id`:
```python
yield StreamEvent(
type="tool_call_name",
chunk=tool_name,
tool_call_id=tool_call_id,
)
# then, as the provider streams JSON args:
yield StreamEvent(
type="tool_call_args",
chunk=partial_json_fragment,
tool_call_id=tool_call_id,
)
```
The framework groups them by `tool_call_id` — so parallel tool calls (where args for tool A and tool B interleave on the wire) work without any per-call index tracking. Some providers (Gemini) emit the complete tool call in one chunk — it's OK to emit both events back-to-back with the full name and full JSON.
For client-side tool calls — tools that LLM should execute locally in a chain — **also call `response.add_tool_call()`**. The chain-execution path (`response.tool_calls()` → `execute_tool_calls()`) reads from the explicitly-added list, not from the StreamEvent buffer.
```python
response.add_tool_call(
llm.ToolCall(
tool_call_id=tool_id,
name=tool_name,
arguments=parsed_args,
)
)
```
### Server-side tool calls
For tools the API executes internally, set `server_executed=True` on the events. Anthropic web search is an example: the API returns a `server_tool_use` block for the search request, followed by a `web_search_tool_result` block containing the result payload.
```python
yield StreamEvent(
type="tool_call_name",
chunk="web_search",
tool_call_id=tool_id,
server_executed=True,
)
yield StreamEvent(
type="tool_call_args",
chunk=json.dumps(query_args),
tool_call_id=tool_id,
server_executed=True,
)
```
The tool *result* (for example, the search hits) is also emitted as an event:
```python
yield StreamEvent(
type="tool_result",
chunk=human_readable_summary,
tool_call_id=tool_id,
server_executed=True,
tool_name="web_search",
provider_metadata={"myprovider": {"raw_content": full_payload}},
)
```
For providers that don't stream server-tool-result contents (Anthropic's `web_search_tool_result` blocks only arrive in the final message), emit those results as a post-stream step. After the main iteration loop completes, inspect the final message and emit tool_result events for any server-side results.
Do **not** call `response.add_tool_call()` for server-side tool calls. This method should only be used for tool calls that need to be executed locally by the framework.
### Opaque provider metadata
Some providers require you to echo back opaque fields on the next request for multi-turn continuity to work:
- **Anthropic** — `signature` on each thinking block; `encrypted_content` inside web_search_tool_result items.
- **Google Gemini** — `thoughtSignature` on `functionCall` parts when thinking is active.
- **OpenAI Responses API** — `encrypted_content` on reasoning items in stateless mode.
These values are attached to a `StreamEvent` via its `provider_metadata` field. The framework merges metadata across events that group into the same Part (last non-None wins per top-level key) and persists it on the finalized Part.
Namespace under your provider's name so transcripts that mix providers don't collide:
```python
# Anthropic signature arrives at the end of a thinking block.
yield StreamEvent(
type="reasoning",
chunk="",
provider_metadata={"anthropic": {"signature": sig}},
)
```
```python
# Gemini attaches thoughtSignature to a functionCall part.
yield StreamEvent(
type="tool_call_name",
chunk=name,
tool_call_id=tc_id,
provider_metadata={"gemini": {"thoughtSignature": sig}},
)
```
The framework round-trips the value verbatim via JSON, so use JSON-safe primitives (string, int, bool, dict, list) for provider metadata - use base64 encoding if you need to store binary data.
### Non-streaming path
When `stream=False` (or the provider returns a complete message at once), emit one event per content block.
```python
else:
completion = client.messages.create(**kwargs)
response.response_json = completion.model_dump()
for block in completion.content:
if block.type == "thinking":
yield StreamEvent(
type="reasoning",
chunk=block.thinking,
provider_metadata={"anthropic": {"signature": block.signature}},
)
elif block.type == "text":
yield StreamEvent(type="text", chunk=block.text)
elif block.type == "tool_use":
yield StreamEvent(
type="tool_call_name",
chunk=block.name,
tool_call_id=block.id,
)
yield StreamEvent(
type="tool_call_args",
chunk=json.dumps(block.input),
tool_call_id=block.id,
)
```
## Consuming prompt.messages in build_messages
`prompt.messages` is an `list[llm.Message]` that is always **the complete input chain for this turn** — whether the caller supplied it explicitly via `model.prompt(messages=[...])`, or it was synthesized from kwargs (`prompt=`, `system=`, `attachments=`, `tool_results=`), or it was pre-built by a `Conversation` or by `response.reply()`.
**Do not also walk `conversation.responses`.** History is already baked into `prompt.messages`; walking the conversation would double-emit.
A plugin's `build_messages` (or equivalent) iterates `prompt.messages` and dispatches per `Part` subtype:
```python
from llm.parts import (
TextPart,
ReasoningPart,
ToolCallPart,
ToolResultPart,
AttachmentPart,
)
def build_messages(self, prompt, conversation):
messages = []
for msg in prompt.messages:
if msg.role == "system":
# Some APIs put system on a separate kwarg (Anthropic, Gemini).
# OpenAI-style APIs emit it as a message; handle accordingly.
continue
self._append_message(messages, msg)
return messages
def _append_message(self, out, msg):
# Map llm's role to the provider's (assistant→model for Gemini,
# tool→user for Anthropic/Gemini tool_result convention, etc.)
role = self._provider_role(msg.role)
parts = []
for part in msg.parts:
if isinstance(part, TextPart):
parts.append({"type": "text", "text": part.text})
elif isinstance(part, ReasoningPart):
# Skip redacted reasoning (no content to echo back).
if part.redacted or not part.text:
continue
block = {"type": "thinking", "thinking": part.text}
# Restore the signature from provider_metadata.
sig = (part.provider_metadata or {}).get("anthropic", {}).get("signature")
if sig:
block["signature"] = sig
parts.append(block)
elif isinstance(part, ToolCallPart):
parts.append({
"type": "tool_use",
"id": part.tool_call_id,
"name": part.name,
"input": part.arguments,
})
elif isinstance(part, ToolResultPart):
parts.append({
"type": "tool_result",
"tool_use_id": part.tool_call_id,
"content": part.output,
})
elif isinstance(part, AttachmentPart) and part.attachment:
parts.append(self._attachment_block(part.attachment))
# Merge with the previous message if roles match (some providers
# require strict alternation between user and assistant).
if out and out[-1]["role"] == role:
out[-1]["content"].extend(parts)
else:
out.append({"role": role, "content": parts})
```
## Restoring opaque metadata on subsequent requests
When a conversation continues, your `build_messages` walks prior-turn Parts via `prompt.messages`. Each Part's `provider_metadata` is a `dict[str, dict]` keyed by provider name — extract your namespace and fold the fields back into the outgoing request body:
```python
if isinstance(part, ReasoningPart):
block = {"type": "thinking", "thinking": part.text}
pm = (part.provider_metadata or {}).get("anthropic", {})
if "signature" in pm:
block["signature"] = pm["signature"]
parts.append(block)
if isinstance(part, ToolCallPart):
fc_part = {"function_call": {"name": part.name, "args": part.arguments}}
pm = (part.provider_metadata or {}).get("gemini", {})
if "thoughtSignature" in pm:
# Gemini expects thoughtSignature beside function_call,
# not nested inside it.
fc_part["thoughtSignature"] = pm["thoughtSignature"]
parts.append(fc_part)
```
If the key is missing (an older transcript that pre-dates your plugin's support), fall through — don't fail. Treat other providers' entries as opaque; don't parse them.
(advanced-model-plugins-usage)=
## Tracking token usage
Models that charge by the token should track the number of tokens used by each prompt. The ``response.set_usage()`` method can be used to record the number of tokens used by a response - these will then be made available through the Python API and logged to the SQLite database for command-line users.
`response` here is the response object that is passed to `.execute()` as an argument.
Call ``response.set_usage()`` at the end of your `.execute()` method. It accepts keyword arguments `input=`, `output=` and `details=` - all three are optional. `input` and `output` should be integers, and `details` should be a dictionary that provides additional information beyond the input and output token counts.
This example logs 15 input tokens, 340 output tokens and notes that 37 tokens were cached:
```python
response.set_usage(input=15, output=340, details={"cached": 37})
```
(advanced-model-plugins-resolved-model)=
## Tracking resolved model names
In some cases the model ID that the user requested may not be the exact model that is executed. Many providers have a `model-latest` alias which may execute different models over time.
If those APIs return the _real_ model ID that was used, your plugin can record that in the `resources.resolved_model` column in the logs by calling this method and passing the string representing the resolved, final model ID:
```bash
response.set_resolved_model(resolved_model_id)
```
This string will be recorded in the database and shown in the output of `llm logs` and `llm logs --json`.
(tutorial-model-plugin-raise-errors)=
## LLM_RAISE_ERRORS
While working on a plugin it can be useful to request that errors are raised instead of being caught and logged, so you can access them from the Python debugger.
Set the `LLM_RAISE_ERRORS` environment variable to enable this behavior, then run `llm` like this:
```bash
LLM_RAISE_ERRORS=1 python -i -m llm ...
```
The `-i` option means Python will drop into an interactive shell if an error occurs. You can then open a debugger at the most recent error using:
```python
import pdb; pdb.pm()
```
docs/plugins/directory.md
+96
View File
@@ -0,0 +1,96 @@
(plugin-directory)=
# Plugin directory
The following plugins are available for LLM. Here's {ref}`how to install them <installing-plugins>`.
(plugin-directory-local-models)=
## Local models
These plugins all help you run LLMs directly on your own computer:
- **[llm-gguf](https://github.com/simonw/llm-gguf)** uses [llama.cpp](https://github.com/ggerganov/llama.cpp) to run models published in the GGUF format.
- **[llm-mlx](https://github.com/simonw/llm-mlx)** (Mac only) uses Apple's MLX framework to provide extremely high performance access to a large number of local models.
- **[llm-ollama](https://github.com/taketwo/llm-ollama)** adds support for local models run using [Ollama](https://ollama.ai/).
- **[llm-llamafile](https://github.com/simonw/llm-llamafile)** adds support for local models that are running locally using [llamafile](https://github.com/Mozilla-Ocho/llamafile).
- **[llm-mlc](https://github.com/simonw/llm-mlc)** can run local models released by the [MLC project](https://mlc.ai/mlc-llm/), including models that can take advantage of the GPU on Apple Silicon M1/M2 devices.
- **[llm-gpt4all](https://github.com/simonw/llm-gpt4all)** adds support for various models released by the [GPT4All](https://gpt4all.io/) project that are optimized to run locally on your own machine. These models include versions of Vicuna, Orca, Falcon and MPT - here's [a full list of models](https://observablehq.com/@simonw/gpt4all-models).
- **[llm-mpt30b](https://github.com/simonw/llm-mpt30b)** adds support for the [MPT-30B](https://huggingface.co/mosaicml/mpt-30b) local model.
(plugin-directory-remote-apis)=
## Remote APIs
These plugins can be used to interact with remotely hosted models via their API:
- **[llm-mistral](https://github.com/simonw/llm-mistral)** adds support for [Mistral AI](https://mistral.ai/)'s language and embedding models.
- **[llm-gemini](https://github.com/simonw/llm-gemini)** adds support for Google's [Gemini](https://ai.google.dev/docs) models.
- **[llm-anthropic](https://github.com/simonw/llm-anthropic)** supports Anthropic's [Claude 3 family](https://www.anthropic.com/news/claude-3-family), [3.5 Sonnet](https://www.anthropic.com/news/claude-3-5-sonnet) and beyond.
- **[llm-command-r](https://github.com/simonw/llm-command-r)** supports Cohere's Command R and [Command R Plus](https://txt.cohere.com/command-r-plus-microsoft-azure/) API models.
- **[llm-reka](https://github.com/simonw/llm-reka)** supports the [Reka](https://www.reka.ai/) family of models via their API.
- **[llm-perplexity](https://github.com/hex/llm-perplexity)** by Alexandru Geana supports the [Perplexity Labs](https://docs.perplexity.ai/) API models, including `llama-3-sonar-large-32k-online` which can search for things online and `llama-3-70b-instruct`.
- **[llm-groq](https://github.com/angerman/llm-groq)** by Moritz Angermann provides access to fast models hosted by [Groq](https://console.groq.com/docs/models).
- **[llm-grok](https://github.com/Hiepler/llm-grok)** by Benedikt Hiepler providing access to Grok model using the xAI API [Grok](https://x.ai/api).
- **[llm-anyscale-endpoints](https://github.com/simonw/llm-anyscale-endpoints)** supports models hosted on the [Anyscale Endpoints](https://app.endpoints.anyscale.com/) platform, including Llama 2 70B.
- **[llm-replicate](https://github.com/simonw/llm-replicate)** adds support for remote models hosted on [Replicate](https://replicate.com/), including Llama 2 from Meta AI.
- **[llm-fireworks](https://github.com/simonw/llm-fireworks)** supports models hosted by [Fireworks AI](https://fireworks.ai/).
- **[llm-openrouter](https://github.com/simonw/llm-openrouter)** provides access to models hosted on [OpenRouter](https://openrouter.ai/).
- **[llm-cohere](https://github.com/Accudio/llm-cohere)** by Alistair Shepherd provides `cohere-generate` and `cohere-summarize` API models, powered by [Cohere](https://cohere.com/).
- **[llm-bedrock](https://github.com/simonw/llm-bedrock)** adds support for Nova by Amazon via Amazon Bedrock.
- **[llm-bedrock-anthropic](https://github.com/sblakey/llm-bedrock-anthropic)** by Sean Blakey adds support for Claude and Claude Instant by Anthropic via Amazon Bedrock.
- **[llm-bedrock-meta](https://github.com/flabat/llm-bedrock-meta)** by Fabian Labat adds support for Llama 2 and Llama 3 by Meta via Amazon Bedrock.
- **[llm-together](https://github.com/wearedevx/llm-together)** adds support for the [Together AI](https://www.together.ai/) extensive family of hosted openly licensed models.
- **[llm-deepseek](https://github.com/abrasumente233/llm-deepseek)** adds support for the [DeepSeek](https://deepseek.com)'s DeepSeek-Chat and DeepSeek-Coder models.
- **[llm-lambda-labs](https://github.com/simonw/llm-lambda-labs)** provides access to models hosted by [Lambda Labs](https://docs.lambdalabs.com/public-cloud/lambda-chat-api/), including the Nous Hermes 3 series.
- **[llm-venice](https://github.com/ar-jan/llm-venice)** provides access to uncensored models hosted by privacy-focused [Venice AI](https://docs.venice.ai/), including Llama 3.1 405B.
If an API model host provides an OpenAI-compatible API you can also [configure LLM to talk to it](https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models) without needing an extra plugin.
(plugin-directory-tools)=
## Tools
The following plugins add new {ref}`tools <tools>` that can be used by models:
- **[llm-tools-simpleeval](https://github.com/simonw/llm-tools-simpleeval)** implements simple expression support for things like mathematics.
- **[llm-tools-quickjs](https://github.com/simonw/llm-tools-quickjs)** provides access to a sandboxed QuickJS JavaScript interpreter, allowing LLMs to run JavaScript code. The environment persists between calls so the model can set variables and build functions and reuse them later on.
- **[llm-tools-sqlite](https://github.com/simonw/llm-tools-sqlite)** can run read-only SQL queries against local SQLite databases.
- **[llm-tools-datasette](https://github.com/simonw/llm-tools-datasette)** can run SQL queries against a remote [Datasette](https://datasette.io/) instance.
- **[llm-tools-exa](https://github.com/daturkel/llm-tools-exa)** by Dan Turkel can perform web searches and question-answering using [exa.ai](https://exa.ai/).
- **[llm-tools-rag](https://github.com/daturkel/llm-tools-rag)** by Dan Turkel can perform searches over your LLM embedding collections for simple RAG.
(plugin-directory-loaders)=
## Fragments and template loaders
{ref}`LLM 0.24 <v0_24>` introduced support for plugins that define `-f prefix:value` or `-t prefix:value` custom loaders for fragments and templates.
- **[llm-video-frames](https://github.com/simonw/llm-video-frames)** uses `ffmpeg` to turn a video into a sequence of JPEG frames suitable for feeding into a vision model that doesn't support video inputs: `llm -f video-frames:video.mp4 'describe the key scenes in this video'`.
- **[llm-templates-github](https://github.com/simonw/llm-templates-github)** supports loading templates shared on GitHub, e.g. `llm -t gh:simonw/pelican-svg`.
- **[llm-templates-fabric](https://github.com/simonw/llm-templates-fabric)** provides access to the [Fabric](https://github.com/danielmiessler/fabric) collection of prompts: `cat setup.py | llm -t fabric:explain_code`.
- **[llm-fragments-github](https://github.com/simonw/llm-fragments-github)** can load entire GitHub repositories in a single operation: `llm -f github:simonw/files-to-prompt 'explain this code'`. It can also fetch issue threads as Markdown using `llm -f issue:https://github.com/simonw/llm-fragments-github/issues/3`.
- **[llm-hacker-news](https://github.com/simonw/llm-hacker-news)** imports conversations from Hacker News as fragments: `llm -f hn:43615912 'summary with illustrative direct quotes'`.
- **[llm-fragments-pypi](https://github.com/samueldg/llm-fragments-pypi)** loads [PyPI](https://pypi.org/) packages' description and metadata as fragments: `llm -f pypi:ruff "What flake8 plugins does ruff re-implement?"`.
- **[llm-fragments-pdf](https://github.com/daturkel/llm-fragments-pdf)** by Dan Turkel converts PDFs to markdown with [PyMuPDF4LLM](https://pymupdf.readthedocs.io/en/latest/pymupdf4llm/index.html) to use as fragments: `llm -f pdf:something.pdf "what's this about?"`.
- **[llm-fragments-site-text](https://github.com/daturkel/llm-fragments-site-text)** by Dan Turkel converts websites to markdown with [Trafilatura](https://trafilatura.readthedocs.io/en/latest/) to use as fragments: `llm -f site:https://example.com "summarize this"`.
- **[llm-fragments-reader](https://github.com/simonw/llm-fragments-reader)** runs a URL theough the Jina Reader API: `llm -f 'reader:https://simonwillison.net/tags/jina/' summary`.
(plugin-directory-embeddings)=
## Embedding models
{ref}`Embedding models <embeddings>` are models that can be used to generate and store embedding vectors for text.
- **[llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers)** adds support for embeddings using the [sentence-transformers](https://www.sbert.net/) library, which provides access to [a wide range](https://www.sbert.net/docs/pretrained_models.html) of embedding models.
- **[llm-clip](https://github.com/simonw/llm-clip)** provides the [CLIP](https://openai.com/research/clip) model, which can be used to embed images and text in the same vector space, enabling text search against images. See [Build an image search engine with llm-clip](https://simonwillison.net/2023/Sep/12/llm-clip-and-chat/) for more on this plugin.
- **[llm-embed-jina](https://github.com/simonw/llm-embed-jina)** provides Jina AI's [8K text embedding models](https://jina.ai/news/jina-ai-launches-worlds-first-open-source-8k-text-embedding-rivaling-openai/).
- **[llm-embed-onnx](https://github.com/simonw/llm-embed-onnx)** provides seven embedding models that can be executed using the ONNX model framework.
(plugin-directory-commands)=
## Extra commands
- **[llm-cmd](https://github.com/simonw/llm-cmd)** accepts a prompt for a shell command, runs that prompt and populates the result in your shell so you can review it, edit it and then hit `<enter>` to execute or `ctrl+c` to cancel.
- **[llm-cmd-comp](https://github.com/CGamesPlay/llm-cmd-comp)** provides a key binding for your shell that will launch a chat to build the command. When ready, hit `<enter>` and it will go right back into your shell command line, so you can run it.
- **[llm-python](https://github.com/simonw/llm-python)** adds a `llm python` command for running a Python interpreter in the same virtual environment as LLM. This is useful for debugging, and also provides a convenient way to interact with the LLM {ref}`python-api` if you installed LLM using Homebrew or `pipx`.
- **[llm-cluster](https://github.com/simonw/llm-cluster)** adds a `llm cluster` command for calculating clusters for a collection of embeddings. Calculated clusters can then be passed to a Large Language Model to generate a summary description.
- **[llm-jq](https://github.com/simonw/llm-jq)** lets you pipe in JSON data and a prompt describing a `jq` program, then executes the generated program against the JSON.
(plugin-directory-fun)=
## Just for fun
- **[llm-markov](https://github.com/simonw/llm-markov)** adds a simple model that generates output using a [Markov chain](https://en.wikipedia.org/wiki/Markov_chain). This example is used in the tutorial [Writing a plugin to support a new model](https://llm.datasette.io/en/latest/plugins/tutorial-model-plugin.html).
docs/plugins/index.md
+22
View File
@@ -0,0 +1,22 @@
(plugins)=
# Plugins
LLM plugins can enhance LLM by making alternative Large Language Models available, either via API or by running the models locally on your machine.
Plugins can also add new commands to the `llm` CLI tool.
The {ref}`plugin directory <plugin-directory>` lists available plugins that you can install and use.
{ref}`tutorial-model-plugin` describes how to build a new plugin in detail.
```{toctree}
---
maxdepth: 3
---
installing-plugins
directory
plugin-hooks
tutorial-model-plugin
advanced-model-plugins
plugin-utilities
```
docs/plugins/installing-plugins.md
+101
View File
@@ -0,0 +1,101 @@
(installing-plugins)=
# Installing plugins
Plugins must be installed in the same virtual environment as LLM itself.
You can find names of plugins to install in the {ref}`plugin directory <plugin-directory>`
Use the `llm install` command (a thin wrapper around `pip install`) to install plugins in the correct environment:
```bash
llm install llm-gpt4all
```
Plugins can be uninstalled with `llm uninstall`:
```bash
llm uninstall llm-gpt4all -y
```
The `-y` flag skips asking for confirmation.
You can see additional models that have been added by plugins by running:
```bash
llm models
```
Or add `--options` to include details of the options available for each model:
```bash
llm models --options
```
To run a prompt against a newly installed model, pass its name as the `-m/--model` option:
```bash
llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
```
## Listing installed plugins
Run `llm plugins` to list installed plugins:
```bash
llm plugins
```
```json
[
{
"name": "llm-anthropic",
"hooks": [
"register_models"
],
"version": "0.11"
},
{
"name": "llm-gguf",
"hooks": [
"register_commands",
"register_models"
],
"version": "0.1a0"
},
{
"name": "llm-clip",
"hooks": [
"register_commands",
"register_embedding_models"
],
"version": "0.1"
},
{
"name": "llm-cmd",
"hooks": [
"register_commands"
],
"version": "0.2a0"
},
{
"name": "llm-gemini",
"hooks": [
"register_embedding_models",
"register_models"
],
"version": "0.3"
}
]
```
(llm-load-plugins)=
## Running with a subset of plugins
By default, LLM will load all plugins that are installed in the same virtual environment as LLM itself.
You can control the set of plugins that is loaded using the `LLM_LOAD_PLUGINS` environment variable.
Set that to the empty string to disable all plugins:
```bash
LLM_LOAD_PLUGINS='' llm ...
```
Or to a comma-separated list of plugin names to load only those plugins:
```bash
LLM_LOAD_PLUGINS='llm-gpt4all,llm-cluster' llm ...
```
You can use the `llm plugins` command to check that it is working correctly:
```
LLM_LOAD_PLUGINS='' llm plugins
```
docs/plugins/llm-markov/llm_markov.py
+68
View File
@@ -0,0 +1,68 @@
import llm
import random
import time
from typing import Optional
from pydantic import field_validator, Field
@llm.hookimpl
def register_models(register):
register(Markov())
def build_markov_table(text):
words = text.split()
transitions = {}
# Loop through all but the last word
for i in range(len(words) - 1):
word = words[i]
next_word = words[i + 1]
transitions.setdefault(word, []).append(next_word)
return transitions
def generate(transitions, length, start_word=None):
all_words = list(transitions.keys())
next_word = start_word or random.choice(all_words)
for i in range(length):
yield next_word
options = transitions.get(next_word) or all_words
next_word = random.choice(options)
class Markov(llm.Model):
model_id = "markov"
can_stream = True
class Options(llm.Options):
length: Optional[int] = Field(
description="Number of words to generate", default=None
)
delay: Optional[float] = Field(
description="Seconds to delay between each token", default=None
)
@field_validator("length")
def validate_length(cls, length):
if length is None:
return None
if length < 2:
raise ValueError("length must be >= 2")
return length
@field_validator("delay")
def validate_delay(cls, delay):
if delay is None:
return None
if not 0 <= delay <= 10:
raise ValueError("delay must be between 0 and 10")
return delay
def execute(self, prompt, stream, response, conversation):
text = prompt.prompt
transitions = build_markov_table(text)
length = prompt.options.length or 20
for word in generate(transitions, length):
yield word + " "
if prompt.options.delay:
time.sleep(prompt.options.delay)
docs/plugins/llm-markov/pyproject.toml
+6
View File
@@ -0,0 +1,6 @@
[project]
name = "llm-markov"
version = "0.1"
[project.entry-points.llm]
markov = "llm_markov"
docs/plugins/plugin-hooks.md
+292
View File
@@ -0,0 +1,292 @@
(plugin-hooks)=
# Plugin hooks
Plugins use **plugin hooks** to customize LLM's behavior. These hooks are powered by the [Pluggy plugin system](https://pluggy.readthedocs.io/).
Each plugin can implement one or more hooks using the @hookimpl decorator against one of the hook function names described on this page.
LLM imitates the Datasette plugin system. The [Datasette plugin documentation](https://docs.datasette.io/en/stable/writing_plugins.html) describes how plugins work.
(plugin-hooks-register-commands)=
## register_commands(cli)
This hook adds new commands to the `llm` CLI tool - for example `llm extra-command`.
This example plugin adds a new `hello-world` command that prints "Hello world!":
```python
from llm import hookimpl
import click
@hookimpl
def register_commands(cli):
@cli.command(name="hello-world")
def hello_world():
"Print hello world"
click.echo("Hello world!")
```
This new command will be added to `llm --help` and can be run using `llm hello-world`.
(plugin-hooks-register-models)=
## register_models(register, model_aliases)
This hook can be used to register one or more additional models.
```python
import llm
@llm.hookimpl
def register_models(register):
register(HelloWorld())
class HelloWorld(llm.Model):
model_id = "helloworld"
def execute(self, prompt, stream, response):
return ["hello world"]
```
If your model includes an async version, you can register that too:
```python
class AsyncHelloWorld(llm.AsyncModel):
model_id = "helloworld"
async def execute(self, prompt, stream, response):
return ["hello world"]
@llm.hookimpl
def register_models(register):
register(HelloWorld(), AsyncHelloWorld(), aliases=("hw",))
```
This demonstrates how to register a model with both sync and async versions, and how to specify an alias for that model.
The `model_aliases` parameter is a list of {class}`~llm.ModelWithAliases` objects representing all models registered so far by other plugins. Plugins that use `@llm.hookimpl(trylast=True)` can use this to inspect or modify models registered by other plugins. Both parameters are optional - plugins can accept just `register`, just `model_aliases`, or both.
The {ref}`model plugin tutorial <tutorial-model-plugin>` describes how to use this hook in detail. Asynchronous models {ref}`are described here <advanced-model-plugins-async>`.
```{eval-rst}
.. autoclass:: llm.ModelWithAliases
:exclude-members: matches
```
(plugin-hooks-register-embedding-models)=
## register_embedding_models(register)
This hook can be used to register one or more additional embedding models, as described in {ref}`embeddings-writing-plugins`.
```python
import llm
@llm.hookimpl
def register_embedding_models(register):
register(HelloWorld())
class HelloWorld(llm.EmbeddingModel):
model_id = "helloworld"
def embed_batch(self, items):
return [[1, 2, 3], [4, 5, 6]]
```
(plugin-hooks-register-tools)=
## register_tools(register)
This hook can register one or more tool functions for use with LLM. See {ref}`the tools documentation <tools>` for more details.
This example registers two tools: `upper` and `count_character_in_word`.
```python
import llm
def upper(text: str) -> str:
"""Convert text to uppercase."""
return text.upper()
def count_char(text: str, character: str) -> int:
"""Count the number of occurrences of a character in a word."""
return text.count(character)
@llm.hookimpl
def register_tools(register):
register(upper)
# Here the name= argument is used to specify a different name for the tool:
register(count_char, name="count_character_in_word")
```
Tools can also be implemented as classes, as described in {ref}`Toolbox classes <python-api-toolbox>` in the Python API documentation.
You can register classes like the `Memory` example {ref}`from here <python-api-toolbox>` by passing the class (_not_ an instance of the class) to `register()`:
```python
import llm
class Memory(llm.Toolbox):
# Copy implementation from the Python API documentation
@llm.hookimpl
def register_tools(register):
register(Memory)
```
Once installed, this tool can be used like so:
```bash
llm chat -T Memory
```
If a tool name starts with a capital letter it is assumed to be a toolbox class, not a regular tool function.
Here's an example session with the Memory tool:
```
Chatting with gpt-4.1-mini
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> Remember my name is Henry
Tool call: Memory_set({'key': 'user_name', 'value': 'Henry'})
null
Got it, Henry! I'll remember your name. How can I assist you today?
> what keys are there?
Tool call: Memory_keys({})
[
"user_name"
]
Currently, there is one key stored: "user_name". Would you like to add or retrieve any information?
> read it
Tool call: Memory_get({'key': 'user_name'})
Henry
The value stored under the key "user_name" is Henry. Is there anything else you'd like to do?
> add Barrett to it
Tool call: Memory_append({'key': 'user_name', 'value': 'Barrett'})
null
I have added "Barrett" to the key "user_name". If you want, I can now show you the updated value.
> show value
Tool call: Memory_get({'key': 'user_name'})
Henry
Barrett
The value stored under the key "user_name" is now:
Henry
Barrett
Is there anything else you would like to do?
```
(plugin-hooks-register-template-loaders)=
## register_template_loaders(register)
Plugins can register new {ref}`template loaders <prompt-templates-loaders>` using the `register_template_loaders` hook.
Template loaders work with the `llm -t prefix:name` syntax. The prefix specifies the loader, then the registered loader function is called with the name as an argument. The loader function should return an `llm.Template()` object.
This example plugin registers `my-prefix` as a new template loader. Once installed it can be used like this:
```bash
llm -t my-prefix:my-template
```
Here's the Python code:
```python
import llm
@llm.hookimpl
def register_template_loaders(register):
register("my-prefix", my_template_loader)
def my_template_loader(template_path: str) -> llm.Template:
"""
Documentation for the template loader goes here. It will be displayed
when users run the 'llm templates loaders' command.
"""
try:
# Your logic to fetch the template content
# This is just an example:
prompt = "This is a sample prompt for {}".format(template_path)
system = "You are an assistant specialized in {}".format(template_path)
# Return a Template object with the required fields
return llm.Template(
name=template_path,
prompt=prompt,
system=system,
)
except Exception as e:
# Raise a ValueError with a clear message if the template cannot be found
raise ValueError(f"Template '{template_path}' could not be loaded: {str(e)}")
```
The `llm.Template` class has the following constructor:
```{eval-rst}
.. autoclass:: llm.Template
```
The loader function should raise a `ValueError` if the template cannot be found or loaded correctly, providing a clear error message.
Note that `functions:` provided by templates using this plugin hook will not be made available, to avoid the risk of plugin hooks that load templates from remote sources introducing arbitrary code execution vulnerabilities.
(plugin-hooks-register-fragment-loaders)=
## register_fragment_loaders(register)
Plugins can register new fragment loaders using the `register_fragment_loaders` hook. These can then be used with the `llm -f prefix:argument` syntax.
Fragment loader plugins differ from template loader plugins in that you can stack more than one fragment loader call together in the same prompt.
A fragment loader can return one or more string fragments or attachments, or a mixture of the two. The fragments will be concatenated together into the prompt string, while any attachments will be added to the list of attachments to be sent to the model.
The `prefix` specifies the loader. The `argument` will be passed to that registered callback..
The callback works in a very similar way to template loaders, but returns either a single `llm.Fragment`, a list of `llm.Fragment` objects, a single `llm.Attachment`, or a list that can mix `llm.Attachment` and `llm.Fragment` objects.
The `llm.Fragment` constructor takes a required string argument (the content of the fragment) and an optional second `source` argument, which is a string that may be displayed as debug information. For files this is a path and for URLs it is a URL. Your plugin can use anything you like for the `source` value.
See {ref}`the Python API documentation for attachments <python-api-attachments>` for details of the `llm.Attachment` class.
Here is some example code:
```python
import llm
@llm.hookimpl
def register_fragment_loaders(register):
register("my-fragments", my_fragment_loader)
def my_fragment_loader(argument: str) -> llm.Fragment:
"""
Documentation for the fragment loader goes here. It will be displayed
when users run the 'llm fragments loaders' command.
"""
try:
fragment = "Fragment content for {}".format(argument)
source = "my-fragments:{}".format(argument)
return llm.Fragment(fragment, source)
except Exception as ex:
# Raise a ValueError with a clear message if the fragment cannot be loaded
raise ValueError(
f"Fragment 'my-fragments:{argument}' could not be loaded: {str(ex)}"
)
# Or for the case where you want to return multiple fragments and attachments:
def my_fragment_loader(argument: str) -> list[llm.Fragment]:
"Docs go here."
return [
llm.Fragment("Fragment 1 content", "my-fragments:{argument}"),
llm.Fragment("Fragment 2 content", "my-fragments:{argument}"),
llm.Attachment(path="/path/to/image.png"),
]
```
A plugin like this one can be called like so:
```bash
llm -f my-fragments:argument
```
If multiple fragments are returned they will be used as if the user passed multiple `-f X` arguments to the command.
Multiple fragments are particularly useful for things like plugins that return every file in a directory. If these were concatenated together by the plugin, a change to a single file would invalidate the de-duplication cache for that whole fragment. Giving each file its own fragment means we can avoid storing multiple copies of that full collection if only a single file has changed.
docs/plugins/plugin-utilities.md
+92
View File
@@ -0,0 +1,92 @@
(plugin-utilities)=
# Utility functions for plugins
LLM provides some utility functions that may be useful to plugins.
(plugin-utilities-get-key)=
## llm.get_key()
This method can be used to look up secrets that users have stored using the {ref}`llm keys set <help-keys-set>` command. If your plugin needs to access an API key or other secret this can be a convenient way to provide that.
This returns either a string containing the key or `None` if the key could not be resolved.
Use the `alias="name"` option to retrieve the key set with that alias:
```python
github_key = llm.get_key(alias="github")
```
You can also add `env="ENV_VAR"` to fall back to looking in that environment variable if the key has not been configured:
```python
github_key = llm.get_key(alias="github", env="GITHUB_TOKEN")
```
In some cases you may allow users to provide a key as input, where they could input either the key itself or specify an alias to lookup in `keys.json`. Use the `input=` parameter for that:
```python
github_key = llm.get_key(input=input_from_user, alias="github", env="GITHUB_TOKEN")
```
An previous version of function used positional arguments in a confusing order. These are still supported but the new keyword arguments are recommended as a better way to use `llm.get_key()` going forward.
(plugin-utilities-user-dir)=
## llm.user_dir()
LLM stores various pieces of logging and configuration data in a directory on the user's machine.
On macOS this directory is `~/Library/Application Support/io.datasette.llm`, but this will differ on other operating systems.
The `llm.user_dir()` function returns the path to this directory as a `pathlib.Path` object, after creating that directory if it does not yet exist.
Plugins can use this to store their own data in a subdirectory of this directory.
```python
import llm
user_dir = llm.user_dir()
plugin_dir = data_path = user_dir / "my-plugin"
plugin_dir.mkdir(exist_ok=True)
data_path = plugin_dir / "plugin-data.db"
```
(plugin-utilities-modelerror)=
## llm.ModelError
If your model encounters an error that should be reported to the user you can raise this exception. For example:
```python
import llm
raise ModelError("MPT model not installed - try running 'llm mpt30b download'")
```
This will be caught by the CLI layer and displayed to the user as an error message.
(plugin-utilities-response-fake)=
## Response.fake()
When writing tests for a model it can be useful to generate fake response objects, for example in this test from [llm-mpt30b](https://github.com/simonw/llm-mpt30b):
```python
def test_build_prompt_conversation():
model = llm.get_model("mpt")
conversation = model.conversation()
conversation.responses = [
llm.Response.fake(model, "prompt 1", "system 1", "response 1"),
llm.Response.fake(model, "prompt 2", None, "response 2"),
llm.Response.fake(model, "prompt 3", None, "response 3"),
]
lines = model.build_prompt(llm.Prompt("prompt 4", model), conversation)
assert lines == [
"<|im_start|>system\system 1<|im_end|>\n",
"<|im_start|>user\nprompt 1<|im_end|>\n",
"<|im_start|>assistant\nresponse 1<|im_end|>\n",
"<|im_start|>user\nprompt 2<|im_end|>\n",
"<|im_start|>assistant\nresponse 2<|im_end|>\n",
"<|im_start|>user\nprompt 3<|im_end|>\n",
"<|im_start|>assistant\nresponse 3<|im_end|>\n",
"<|im_start|>user\nprompt 4<|im_end|>\n",
"<|im_start|>assistant\n",
]
```
The signature of `llm.Response.fake()` is:
```python
def fake(cls, model: Model, prompt: str, system: str, response: str):
```
docs/plugins/tutorial-model-plugin.md
+614
View File
@@ -0,0 +1,614 @@
(tutorial-model-plugin)=
# Developing a model plugin
This tutorial will walk you through developing a new plugin for LLM that adds support for a new Large Language Model.
We will be developing a plugin that implements a simple [Markov chain](https://en.wikipedia.org/wiki/Markov_chain) to generate words based on an input string. Markov chains are not technically large language models, but they provide a useful exercise for demonstrating how the LLM tool can be extended through plugins.
(tutorial-model-plugin-initial)=
## The initial structure of the plugin
First create a new directory with the name of your plugin - it should be called something like `llm-markov`.
```bash
mkdir llm-markov
cd llm-markov
```
In that directory create a file called `llm_markov.py` containing this:
```python
import llm
@llm.hookimpl
def register_models(register):
register(Markov())
class Markov(llm.Model):
model_id = "markov"
def execute(self, prompt, stream, response, conversation):
return ["hello world"]
```
The `def register_models()` function here is called by the plugin system (thanks to the `@hookimpl` decorator). It uses the `register()` function passed to it to register an instance of the new model.
The `Markov` class implements the model. It sets a `model_id` - an identifier that can be passed to `llm -m` in order to identify the model to be executed.
The logic for executing the model goes in the `execute()` method. We'll extend this to do something more useful in a later step.
Next, create a `pyproject.toml` file. This is necessary to tell LLM how to load your plugin:
```toml
[project]
name = "llm-markov"
version = "0.1"
[project.entry-points.llm]
markov = "llm_markov"
```
This is the simplest possible configuration. It defines a plugin name and provides an [entry point](https://setuptools.pypa.io/en/latest/userguide/entry_point.html) for `llm` telling it how to load the plugin.
If you are comfortable with Python virtual environments you can create one now for your project, activate it and run `pip install llm` before the next step.
If you aren't familiar with virtual environments, don't worry: you can develop plugins without them. You'll need to have LLM installed using Homebrew or `pipx` or one of the [other installation options](https://llm.datasette.io/en/latest/setup.html#installation).
(tutorial-model-plugin-installing)=
## Installing your plugin to try it out
Having created a directory with a `pyproject.toml` file and an `llm_markov.py` file, you can install your plugin into LLM by running this from inside your `llm-markov` directory:
```bash
llm install -e .
```
The `-e` stands for "editable" - it means you'll be able to make further changes to the `llm_markov.py` file that will be reflected without you having to reinstall the plugin.
The `.` means the current directory. You can also install editable plugins by passing a path to their directory this:
```bash
llm install -e path/to/llm-markov
```
To confirm that your plugin has installed correctly, run this command:
```bash
llm plugins
```
The output should look like this:
```json
[
{
"name": "llm-markov",
"hooks": [
"register_models"
],
"version": "0.1"
},
{
"name": "llm.default_plugins.openai_models",
"hooks": [
"register_commands",
"register_models"
]
}
]
```
This command lists default plugins that are included with LLM as well as new plugins that have been installed.
Now let's try the plugin by running a prompt through it:
```bash
llm -m markov "the cat sat on the mat"
```
It outputs:
```
hello world
```
Next, we'll make it execute and return the results of a Markov chain.
(tutorial-model-plugin-building)=
## Building the Markov chain
Markov chains can be thought of as the simplest possible example of a generative language model. They work by building an index of words that have been seen following other words.
Here's what that index looks like for the phrase "the cat sat on the mat"
```json
{
"the": ["cat", "mat"],
"cat": ["sat"],
"sat": ["on"],
"on": ["the"]
}
```
Here's a Python function that builds that data structure from a text input:
```python
def build_markov_table(text):
words = text.split()
transitions = {}
# Loop through all but the last word
for i in range(len(words) - 1):
word = words[i]
next_word = words[i + 1]
transitions.setdefault(word, []).append(next_word)
return transitions
```
We can try that out by pasting it into the interactive Python interpreter and running this:
```pycon
>>> transitions = build_markov_table("the cat sat on the mat")
>>> transitions
{'the': ['cat', 'mat'], 'cat': ['sat'], 'sat': ['on'], 'on': ['the']}
```
(tutorial-model-plugin-executing)=
## Executing the Markov chain
To execute the model, we start with a word. We look at the options for words that might come next and pick one of those at random. Then we repeat that process until we have produced the desired number of output words.
Some words might not have any following words from our training sentence. For our implementation we will fall back on picking a random word from our collection.
We will implement this as a [Python generator](https://realpython.com/introduction-to-python-generators/), using the yield keyword to produce each token:
```python
def generate(transitions, length, start_word=None):
all_words = list(transitions.keys())
next_word = start_word or random.choice(all_words)
for i in range(length):
yield next_word
options = transitions.get(next_word) or all_words
next_word = random.choice(options)
```
If you aren't familiar with generators, the above code could also be implemented like this - creating a Python list and returning it at the end of the function:
```python
def generate_list(transitions, length, start_word=None):
all_words = list(transitions.keys())
next_word = start_word or random.choice(all_words)
output = []
for i in range(length):
output.append(next_word)
options = transitions.get(next_word) or all_words
next_word = random.choice(options)
return output
```
You can try out the `generate()` function like this:
```python
lookup = build_markov_table("the cat sat on the mat")
for word in generate(transitions, 20):
print(word)
```
Or you can generate a full string sentence with it like this:
```python
sentence = " ".join(generate(transitions, 20))
```
(tutorial-model-plugin-register)=
## Adding that to the plugin
Our `execute()` method from earlier currently returns the list `["hello world"]`.
Update that to use our new Markov chain generator instead. Here's the full text of the new `llm_markov.py` file:
```python
import llm
import random
@llm.hookimpl
def register_models(register):
register(Markov())
def build_markov_table(text):
words = text.split()
transitions = {}
# Loop through all but the last word
for i in range(len(words) - 1):
word = words[i]
next_word = words[i + 1]
transitions.setdefault(word, []).append(next_word)
return transitions
def generate(transitions, length, start_word=None):
all_words = list(transitions.keys())
next_word = start_word or random.choice(all_words)
for i in range(length):
yield next_word
options = transitions.get(next_word) or all_words
next_word = random.choice(options)
class Markov(llm.Model):
model_id = "markov"
def execute(self, prompt, stream, response, conversation):
text = prompt.prompt
transitions = build_markov_table(text)
for word in generate(transitions, 20):
yield word + ' '
```
The `execute()` method can access the text prompt that the user provided using` prompt.prompt` - `prompt` is a `Prompt` object that might include other more advanced input details as well.
Now when you run this you should see the output of the Markov chain!
```bash
llm -m markov "the cat sat on the mat"
```
```
the mat the cat sat on the cat sat on the mat cat sat on the mat cat sat on
```
(tutorial-model-plugin-execute)=
## Understanding execute()
The full signature of the `execute()` method is:
```python
def execute(self, prompt, stream, response, conversation):
```
The `prompt` argument is a `Prompt` object that contains the text that the user provided, the system prompt and the provided options.
`stream` is a boolean that says if the model is being run in streaming mode.
`response` is the `Response` object that is being created by the model. This is provided so you can write additional information to `response.response_json`, which may be logged to the database.
`conversation` is the `Conversation` that the prompt is a part of - or `None` if no conversation was provided. Some models may use `conversation.responses` to access previous prompts and responses in the conversation and use them to construct a call to the LLM that includes previous context.
(tutorial-model-plugin-logging)=
## Prompts and responses are logged to the database
The prompt and the response will be logged to a SQLite database automatically by LLM. You can see the single most recent addition to the logs using:
```
llm logs -n 1
```
The output should look something like this:
```json
[
{
"id": "01h52s4yez2bd1qk2deq49wk8h",
"model": "markov",
"prompt": "the cat sat on the mat",
"system": null,
"prompt_json": null,
"options_json": {},
"response": "on the cat sat on the cat sat on the mat cat sat on the cat sat on the cat ",
"response_json": null,
"conversation_id": "01h52s4yey7zc5rjmczy3ft75g",
"duration_ms": 0,
"datetime_utc": "2023-07-11T15:29:34.685868",
"conversation_name": "the cat sat on the mat",
"conversation_model": "markov"
}
]
```
Plugins can log additional information to the database by assigning a dictionary to the `response.response_json` property during the `execute()` method.
Here's how to include that full `transitions` table in the `response_json` in the log:
```python
def execute(self, prompt, stream, response, conversation):
text = self.prompt.prompt
transitions = build_markov_table(text)
for word in generate(transitions, 20):
yield word + ' '
response.response_json = {"transitions": transitions}
```
Now when you run the logs command you'll see that too:
```bash
llm logs -n 1
```
```json
[
{
"id": 623,
"model": "markov",
"prompt": "the cat sat on the mat",
"system": null,
"prompt_json": null,
"options_json": {},
"response": "on the mat the cat sat on the cat sat on the mat sat on the cat sat on the ",
"response_json": {
"transitions": {
"the": [
"cat",
"mat"
],
"cat": [
"sat"
],
"sat": [
"on"
],
"on": [
"the"
]
}
},
"reply_to_id": null,
"chat_id": null,
"duration_ms": 0,
"datetime_utc": "2023-07-06T01:34:45.376637"
}
]
```
In this particular case this isn't a great idea here though: the `transitions` table is duplicate information, since it can be reproduced from the input data - and it can get really large for longer prompts.
(tutorial-model-plugin-options)=
## Adding options
LLM models can take options. For large language models these can be things like `temperature` or `top_k`.
Options are passed using the `-o/--option` command line parameters, for example:
```bash
llm -m gpt4 "ten pet pelican names" -o temperature 1.5
```
We're going to add two options to our Markov chain model:
- `length`: Number of words to generate
- `delay`: a floating point number of Delay in between output token
The `delay` token will let us simulate a streaming language model, where tokens take time to generate and are returned by the `execute()` function as they become ready.
Options are defined using an inner class on the model, called `Options`. It should extend the `llm.Options` class.
First, add this import to the top of your `llm_markov.py` file:
```python
from typing import Optional
```
Then add this `Options` class to your model:
```python
class Markov(Model):
model_id = "markov"
class Options(llm.Options):
length: Optional[int] = None
delay: Optional[float] = None
```
Let's add extra validation rules to our options. Length must be at least 2. Duration must be between 0 and 10.
The `Options` class uses [Pydantic 2](https://pydantic.dev/), which can support all sorts of advanced validation rules.
We can also add inline documentation, which can then be displayed by the `llm models --options` command.
Add these imports to the top of `llm_markov.py`:
```python
from pydantic import field_validator, Field
```
We can now add Pydantic field validators for our two new rules, plus inline documentation:
```python
class Options(llm.Options):
length: Optional[int] = Field(
description="Number of words to generate",
default=None
)
delay: Optional[float] = Field(
description="Seconds to delay between each token",
default=None
)
@field_validator("length")
def validate_length(cls, length):
if length is None:
return None
if length < 2:
raise ValueError("length must be >= 2")
return length
@field_validator("delay")
def validate_delay(cls, delay):
if delay is None:
return None
if not 0 <= delay <= 10:
raise ValueError("delay must be between 0 and 10")
return delay
```
Lets test our options validation:
```bash
llm -m markov "the cat sat on the mat" -o length -1
```
```
Error: length
Value error, length must be >= 2
```
Next, we will modify our `execute()` method to handle those options. Add this to the beginning of `llm_markov.py`:
```python
import time
```
Then replace the `execute()` method with this one:
```python
def execute(self, prompt, stream, response, conversation):
text = prompt.prompt
transitions = build_markov_table(text)
length = prompt.options.length or 20
for word in generate(transitions, length):
yield word + ' '
if prompt.options.delay:
time.sleep(prompt.options.delay)
```
Add `can_stream = True` to the top of the `Markov` model class, on the line below `model_id = "markov". This tells LLM that the model is able to stream content to the console.
The full `llm_markov.py` file should now look like this:
```{literalinclude} llm-markov/llm_markov.py
:language: python
```
Now we can request a 20 word completion with a 0.1s delay between tokens like this:
```bash
llm -m markov "the cat sat on the mat" \
-o length 20 -o delay 0.1
```
LLM provides a `--no-stream` option users can use to turn off streaming. Using that option causes LLM to gather the response from the stream and then return it to the console in one block. You can try that like this:
```bash
llm -m markov "the cat sat on the mat" \
-o length 20 -o delay 0.1 --no-stream
```
In this case it will still delay for 2s total while it gathers the tokens, then output them all at once.
That `--no-stream` option causes the `stream` argument passed to `execute()` to be false. Your `execute()` method can then behave differently depending on whether it is streaming or not.
Options are also logged to the database. You can see those here:
```bash
llm logs -n 1
```
```json
[
{
"id": 636,
"model": "markov",
"prompt": "the cat sat on the mat",
"system": null,
"prompt_json": null,
"options_json": {
"length": 20,
"delay": 0.1
},
"response": "the mat on the mat on the cat sat on the mat sat on the mat cat sat on the ",
"response_json": null,
"reply_to_id": null,
"chat_id": null,
"duration_ms": 2063,
"datetime_utc": "2023-07-07T03:02:28.232970"
}
]
```
(tutorial-model-plugin-distributing)=
## Distributing your plugin
There are many different options for distributing your new plugin so other people can try it out.
You can create a downloadable wheel or `.zip` or `.tar.gz` files, or share the plugin through GitHub Gists or repositories.
You can also publish your plugin to PyPI, the Python Package Index.
(tutorial-model-plugin-wheels)=
### Wheels and sdist packages
The easiest option is to produce a distributable package is to use the `build` command. First, install the `build` package by running this:
```bash
python -m pip install build
```
Then run `build` in your plugin directory to create the packages:
```bash
python -m build
```
This will create two files: `dist/llm-markov-0.1.tar.gz` and `dist/llm-markov-0.1-py3-none-any.whl`.
Either of these files can be used to install the plugin:
```bash
llm install dist/llm_markov-0.1-py3-none-any.whl
```
If you host this file somewhere online other people will be able to install it using `pip install` against the URL to your package:
```bash
llm install 'https://.../llm_markov-0.1-py3-none-any.whl'
```
You can run the following command at any time to uninstall your plugin, which is useful for testing out different installation methods:
```bash
llm uninstall llm-markov -y
```
(tutorial-model-plugin-gists)=
### GitHub Gists
A neat quick option for distributing a simple plugin is to host it in a GitHub Gist. These are available for free with a GitHub account, and can be public or private. Gists can contain multiple files but don't support directory structures - which is OK, because our plugin is just two files, `pyproject.toml` and `llm_markov.py`.
Here's an example Gist I created for this tutorial:
[https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d](https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d)
You can turn a Gist into an installable `.zip` URL by right-clicking on the "Download ZIP" button and selecting "Copy Link". Here's that link for my example Gist:
`https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d/archive/cc50c854414cb4deab3e3ab17e7e1e07d45cba0c.zip`
The plugin can be installed using the `llm install` command like this:
```bash
llm install 'https://gist.github.com/simonw/6e56d48dc2599bffba963cef0db27b6d/archive/cc50c854414cb4deab3e3ab17e7e1e07d45cba0c.zip'
```
(tutorial-model-plugin-github)=
## GitHub repositories
The same trick works for regular GitHub repositories as well: the "Download ZIP" button can be found by clicking the green "Code" button at the top of the repository. The URL which that provides can then be used to install the plugin that lives in that repository.
(tutorial-model-plugin-pypi)=
## Publishing plugins to PyPI
The [Python Package Index (PyPI)](https://pypi.org/) is the official repository for Python packages. You can upload your plugin to PyPI and reserve a name for it - once you have done that, anyone will be able to install your plugin using `llm install <name>`.
Follow [these instructions](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives) to publish a package to PyPI. The short version:
```bash
python -m pip install twine
python -m twine upload dist/*
```
You will need an account on PyPI, then you can enter your username and password - or create a token in the PyPI settings and use `__token__` as the username and the token as the password.
(tutorial-model-plugin-metadata)=
## Adding metadata
Before uploading a package to PyPI it's a good idea to add documentation and expand `pyproject.toml` with additional metadata.
Create a `README.md` file in the root of your plugin directory with instructions about how to install, configure and use your plugin.
You can then replace `pyproject.toml` with something like this:
```toml
[project]
name = "llm-markov"
version = "0.1"
description = "Plugin for LLM adding a Markov chain generating model"
readme = "README.md"
authors = [{name = "Simon Willison"}]
license = {text = "Apache-2.0"}
classifiers = [
"License :: OSI Approved :: Apache Software License"
]
dependencies = [
"llm"
]
requires-python = ">3.7"
[project.urls]
Homepage = "https://github.com/simonw/llm-markov"
Changelog = "https://github.com/simonw/llm-markov/releases"
Issues = "https://github.com/simonw/llm-markov/issues"
[project.entry-points.llm]
markov = "llm_markov"
```
This will pull in your README to be displayed as part of your project's listing page on PyPI.
It adds `llm` as a dependency, ensuring it will be installed if someone tries to install your plugin package without it.
It adds some links to useful pages (you can drop the `project.urls` section if those links are not useful for your project).
You should drop a `LICENSE` file into the GitHub repository for your package as well. I like to use the Apache 2 license [like this](https://github.com/simonw/llm/blob/main/LICENSE).
(tutorial-model-plugin-breaks)=
## What to do if it breaks
Sometimes you may make a change to your plugin that causes it to break, preventing `llm` from starting. For example you may see an error like this one:
```
$ llm 'hi'
Traceback (most recent call last):
...
File llm-markov/llm_markov.py", line 10
register(Markov()):
^
SyntaxError: invalid syntax
```
You may find that you are unable to uninstall the plugin using `llm uninstall llm-markov` because the command itself fails with the same error.
Should this happen, you can uninstall the plugin after first disabling it using the {ref}`LLM_LOAD_PLUGINS <llm-load-plugins>` environment variable like this:
```bash
LLM_LOAD_PLUGINS='' llm uninstall llm-markov
```
docs/python-api.md
+1006
View File
File diff suppressed because it is too large Load Diff
docs/related-tools.md
+58
View File
@@ -0,0 +1,58 @@
(related-tools)=
# Related tools
The following tools are designed to be used with LLM:
(related-tools-strip-tags)=
## strip-tags
[strip-tags](https://github.com/simonw/strip-tags) is a command for stripping tags from HTML. This is useful when working with LLMs because HTML tags can use up a lot of your token budget.
Here's how to summarize the front page of the New York Times, by both stripping tags and filtering to just the elements with `class="story-wrapper"`:
```bash
curl -s https://www.nytimes.com/ \
| strip-tags .story-wrapper \
| llm -s 'summarize the news'
```
[llm, ttok and strip-tags—CLI tools for working with ChatGPT and other LLMs](https://simonwillison.net/2023/May/18/cli-tools-for-llms/) describes ways to use `strip-tags` in more detail.
(related-tools-ttok)=
## ttok
[ttok](https://github.com/simonw/ttok) is a command-line tool for counting OpenAI tokens. You can use it to check if input is likely to fit in the token limit for GPT 3.5 or GPT4:
```bash
cat my-file.txt | ttok
```
```
125
```
It can also truncate input down to a desired number of tokens:
```bash
ttok This is too many tokens -t 3
```
```
This is too
```
This is useful for truncating a large document down to a size where it can be processed by an LLM.
(related-tools-symbex)=
## Symbex
[Symbex](https://github.com/simonw/symbex) is a tool for searching for symbols in Python codebases. It's useful for extracting just the code for a specific problem and then piping that into LLM for explanation, refactoring or other tasks.
Here's how to use it to find all functions that match `test*csv*` and use those to guess what the software under test does:
```bash
symbex 'test*csv*' | \
llm --system 'based on these tests guess what this tool does'
```
It can also be used to export symbols in a format that can be piped to {ref}`llm embed-multi <embeddings-cli-embed-multi>` in order to create embeddings:
```bash
symbex '*' '*:*' --nl | \
llm embed-multi symbols - \
--format nl --database embeddings.db --store
```
For more examples see [Symbex: search Python code for functions and classes, then pipe them into a LLM](https://simonwillison.net/2023/Jun/18/symbex/).
docs/requirements.txt
+7
View File
@@ -0,0 +1,7 @@
sphinx==7.2.6
furo==2023.9.10
sphinx-autobuild
sphinx-copybutton
sphinx-markdown-builder==0.6.8
myst-parser
cogapp
docs/schemas.md
+599
View File
@@ -0,0 +1,599 @@
(schemas)=
# Schemas
Large Language Models are very good at producing structured output as JSON or other formats. LLM's **schemas** feature allows you to define the exact structure of JSON data you want to receive from a model.
This feature is supported by models from OpenAI, Anthropic, Google Gemini and can be implemented for others {ref}`via plugins <advanced-model-plugins-schemas>`.
This page describes schemas used via the `llm` command-line tool. Schemas can also be used from the {ref}`Python API <python-api-schemas>`.
(schemas-tutorial)=
## Schemas tutorial
In this tutorial we're going to use schemas to analyze some news stories.
But first, let's invent some dogs!
### Getting started with dogs
LLMs are great at creating test data. Let's define a simple schema for a dog, using LLM's {ref}`concise schema syntax <schemas-dsl>`. We'll pass that to LLm with `llm --schema` and prompt it to "invent a cool dog":
```bash
llm --schema 'name, age int, one_sentence_bio' 'invent a cool dog'
```
I got back Ziggy:
```json
{
"name": "Ziggy",
"age": 4,
"one_sentence_bio": "Ziggy is a hyper-intelligent, bioluminescent dog who loves to perform tricks in the dark and guides his owner home using his glowing fur."
}
```
The response matched my schema, with `name` and `one_sentence_bio` string columns and an integer for `age`.
We're using the default LLM model here - `gpt-4o-mini`. Add `-m model` to use another model - for example use `-m o3-mini` to have O3 mini invent some dogs.
For a list of available models that support schemas, run this command:
```bash
llm models --schemas
```
Want several more dogs? You can pass in that same schema using `--schema-multi` and ask for several at once:
```bash
llm --schema-multi 'name, age int, one_sentence_bio' 'invent 3 really cool dogs'
```
Here's what I got:
```json
{
"items": [
{
"name": "Echo",
"age": 3,
"one_sentence_bio": "Echo is a sleek, silvery-blue Siberian Husky with mesmerizing blue eyes and a talent for mimicking sounds, making him a natural entertainer."
},
{
"name": "Nova",
"age": 2,
"one_sentence_bio": "Nova is a vibrant, spotted Dalmatian with an adventurous spirit and a knack for agility courses, always ready to leap into action."
},
{
"name": "Pixel",
"age": 4,
"one_sentence_bio": "Pixel is a playful, tech-savvy Poodle with a rainbow-colored coat, known for her ability to interact with smart devices and her love for puzzle toys."
}
]
}
```
So that's the basic idea: we can feed in a schema and LLM will pass it to the underlying model and (usually) get back JSON that conforms to that schema.
This stuff gets a _lot_ more useful when you start applying it to larger amounts of text, extracting structured details from unstructured content.
### Extracting people from a news articles
We are going to extract details of the people who are mentioned in different news stories, and then use those to compile a database.
Let's start by compiling a schema. For each person mentioned we want to extract the following details:
- Their name
- The organization they work for
- Their role
- What we learned about them from the story
We will also record the article headline and the publication date, to make things easier for us later on.
Using LLM's custom, concise schema language, this time with newlines separating the individual fields (for the dogs example we used commas):
```
name: the person's name
organization: who they represent
role: their job title or role
learned: what we learned about them from this story
article_headline: the headline of the story
article_date: the publication date in YYYY-MM-DD
```
As you can see, this schema definition is pretty simple - each line has the name of a property we want to capture, then an optional: followed by a description, which doubles as instructions for the model.
The full syntax is {ref}`described below <schemas-dsl>` - you can also include type information for things like numbers.
Let's run this against a news article.
Visit [AP News](https://apnews.com/) and grab the URL to an article. I'm using this one:
https://apnews.com/article/trump-federal-employees-firings-a85d1aaf1088e050d39dcf7e3664bb9f
There's quite a lot of HTML on that page, possibly even enough to exceed GPT-4o mini's 128,000 token input limit. We'll use another tool called [strip-tags](https://github.com/simonw/strip-tags) to reduce that. If you have [uv](https://docs.astral.sh/uv/) installed you can call it using `uvx strip-tags`, otherwise you'll need to install it first:
```
uv tool install strip-tags
# Or "pip install" or "pipx install"
```
Now we can run this command to extract the people from that article:
```bash
curl 'https://apnews.com/article/trump-federal-employees-firings-a85d1aaf1088e050d39dcf7e3664bb9f' | \
uvx strip-tags | \
llm --schema-multi "
name: the person's name
organization: who they represent
role: their job title or role
learned: what we learned about them from this story
article_headline: the headline of the story
article_date: the publication date in YYYY-MM-DD
" --system 'extract people mentioned in this article'
```
The output I got started like this:
```json
{
"items": [
{
"name": "William Alsup",
"organization": "U.S. District Court",
"role": "Judge",
"learned": "He ruled that the mass firings of probationary employees were likely unlawful and criticized the authority exercised by the Office of Personnel Management.",
"article_headline": "Judge finds mass firings of federal probationary workers were likely unlawful",
"article_date": "2025-02-26"
},
{
"name": "Everett Kelley",
"organization": "American Federation of Government Employees",
"role": "National President",
"learned": "He hailed the court's decision as a victory for employees who were illegally fired.",
"article_headline": "Judge finds mass firings of federal probationary workers were likely unlawful",
"article_date": "2025-02-26"
}
```
This data has been logged to LLM's {ref}`SQLite database <logging>`. We can retrieve the data back out again using the {ref}`llm logs <logging-view>` command like this:
```bash
llm logs -c --data
```
The `-c` flag means "use most recent conversation", and the `--data` flag outputs just the JSON data that was captured in the response.
We're going to want to use the same schema for other things. Schemas that we use are automatically logged to the database - we can view them using `llm schemas`:
```bash
llm schemas
```
Here's the output:
```
- id: 3b7702e71da3dd791d9e17b76c88730e
summary: |
{items: [{name, organization, role, learned, article_headline, article_date}]}
usage: |
1 time, most recently 2025-02-28T04:50:02.032081+00:00
```
To view the full schema, run that command with `--full`:
```bash
llm schemas --full
```
Which outputs:
```
- id: 3b7702e71da3dd791d9e17b76c88730e
schema: |
{
"type": "object",
"properties": {
"items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "the person's name"
},
...
```
That `3b7702e71da3dd791d9e17b76c88730e` ID can be used to run the same schema again. Let's try that now on a different URL:
```bash
curl 'https://apnews.com/article/bezos-katy-perry-blue-origin-launch-4a074e534baa664abfa6538159c12987' | \
uvx strip-tags | \
llm --schema 3b7702e71da3dd791d9e17b76c88730e \
--system 'extract people mentioned in this article'
```
Here we are using `--schema` because our schema ID already corresponds to an array of items.
The result starts like this:
```json
{
"items": [
{
"name": "Katy Perry",
"organization": "Blue Origin",
"role": "Singer",
"learned": "Katy Perry will join the all-female celebrity crew for a spaceflight organized by Blue Origin.",
"article_headline": "Katy Perry and Gayle King will join Jeff Bezos’ fiancee Lauren Sanchez on Blue Origin spaceflight",
"article_date": "2023-10-15"
},
```
One more trick: let's turn our schema and system prompt combination into a {ref}`template <prompt-templates>`.
```bash
llm --schema 3b7702e71da3dd791d9e17b76c88730e \
--system 'extract people mentioned in this article' \
--save people
```
This creates a new template called "people". We can confirm the template was created correctly using:
```bash
llm templates show people
```
Which will output the YAML version of the template looking like this:
```yaml
name: people
schema_object:
properties:
items:
items:
properties:
article_date:
description: the publication date in YYYY-MM-DD
type: string
article_headline:
description: the headline of the story
type: string
learned:
description: what we learned about them from this story
type: string
name:
description: the person's name
type: string
organization:
description: who they represent
type: string
role:
description: their job title or role
type: string
required:
- name
- organization
- role
- learned
- article_headline
- article_date
type: object
type: array
required:
- items
type: object
system: extract people mentioned in this article
```
We can now run our people extractor against another fresh URL. Let's use one from The Guardian:
```bash
curl https://www.theguardian.com/commentisfree/2025/feb/27/billy-mcfarland-new-fyre-festival-fantasist | \
strip-tags | llm -t people
```
Storing the schema in a template means we can just use `llm -t people` to run the prompt. Here's what I got back:
```json
{
"items": [
{
"name": "Billy McFarland",
"organization": "Fyre Festival",
"role": "Organiser",
"learned": "Billy McFarland is known for organizing the infamous Fyre Festival and was sentenced to six years in prison for wire fraud related to it. He is attempting to revive the festival with Fyre 2.",
"article_headline": "Welcome back Billy McFarland and a new Fyre festival. Shows you can’t keep a good fantasist down",
"article_date": "2025-02-27"
}
]
}
```
Depending on the model, schema extraction may work against images and PDF files as well.
I took a screenshot of part of [this story in the Onion](https://theonion.com/mark-zuckerberg-insists-anyone-with-same-skewed-values-1826829272/) and saved it to the following URL:
https://static.simonwillison.net/static/2025/onion-zuck.jpg
We can pass that as an {ref}`attachment <usage-attachments>` using the `-a` option. This time let's use GPT-4o:
```bash
llm -t people -a https://static.simonwillison.net/static/2025/onion-zuck.jpg -m gpt-4o
```
Which gave me back this:
```json
{
"items": [
{
"name": "Mark Zuckerberg",
"organization": "Facebook",
"role": "CEO",
"learned": "He addressed criticism by suggesting anyone with similar values and thirst for power could make the same mistakes.",
"article_headline": "Mark Zuckerberg Insists Anyone With Same Skewed Values And Unrelenting Thirst For Power Could Have Made Same Mistakes",
"article_date": "2018-06-14"
}
]
}
```
Now that we've extracted people from a number of different sources, let's load them into a database.
The {ref}`llm logs <logging-view>` command has several features for working with logged JSON objects. Since we've been recording multiple objects from each page in an `"items"` array using our `people` template we can access those using the following command:
```bash
llm logs --schema t:people --data-key items
```
In place of `t:people` we could use the `3b7702e71da3dd791d9e17b76c88730e` schema ID or even the original schema string instead, see {ref}`specifying a schema <schemas-specify>`.
This command outputs newline-delimited JSON for every item that has been captured using the specified schema:
```json
{"name": "Katy Perry", "organization": "Blue Origin", "role": "Singer", "learned": "She is one of the passengers on the upcoming spaceflight with Blue Origin."}
{"name": "Gayle King", "organization": "Blue Origin", "role": "TV Journalist", "learned": "She is participating in the upcoming Blue Origin spaceflight."}
{"name": "Lauren Sanchez", "organization": "Blue Origin", "role": "Helicopter Pilot and former TV Journalist", "learned": "She selected the crew for the Blue Origin spaceflight."}
{"name": "Aisha Bowe", "organization": "Engineering firm", "role": "Former NASA Rocket Scientist", "learned": "She is part of the crew for the spaceflight."}
{"name": "Amanda Nguyen", "organization": "Research Scientist", "role": "Activist and Scientist", "learned": "She is included in the crew for the upcoming Blue Origin flight."}
{"name": "Kerianne Flynn", "organization": "Movie Producer", "role": "Producer", "learned": "She will also be a passenger on the upcoming spaceflight."}
{"name": "Billy McFarland", "organization": "Fyre Festival", "role": "Organiser", "learned": "He was sentenced to six years in prison for wire fraud in 2018 and has launched a new festival called Fyre 2.", "article_headline": "Welcome back Billy McFarland and a new Fyre festival. Shows you can\u2019t keep a good fantasist down", "article_date": "2025-02-27"}
{"name": "Mark Zuckerberg", "organization": "Facebook", "role": "CEO", "learned": "He attempted to dismiss criticism by suggesting that anyone with similar values and thirst for power could have made the same mistakes.", "article_headline": "Mark Zuckerberg Insists Anyone With Same Skewed Values And Unrelenting Thirst For Power Could Have Made Same Mistakes", "article_date": "2018-06-14"}
```
If we add `--data-array` we'll get back a valid JSON array of objects instead:
```bash
llm logs --schema t:people --data-key items --data-array
```
Output starts:
```json
[{"name": "Katy Perry", "organization": "Blue Origin", "role": "Singer", "learned": "She is one of the passengers on the upcoming spaceflight with Blue Origin."},
{"name": "Gayle King", "organization": "Blue Origin", "role": "TV Journalist", "learned": "She is participating in the upcoming Blue Origin spaceflight."},
```
We can load this into a SQLite database using [sqlite-utils](https://sqlite-utils.datasette.io/), in particular the [sqlite-utils insert](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) command.
```bash
uv tool install sqlite-utils
# or pip install or pipx install
```
Now we can pipe the JSON into that tool to create a database with a `people` table:
```bash
llm logs --schema t:people --data-key items --data-array | \
sqlite-utils insert data.db people -
```
To see a table of the name, organization and role columns use [sqlite-utils rows](https://sqlite-utils.datasette.io/en/stable/cli.html#returning-all-rows-in-a-table):
```bash
sqlite-utils rows data.db people -t -c name -c organization -c role
```
Which produces:
```
name organization role
--------------- ------------------ -----------------------------------------
Katy Perry Blue Origin Singer
Gayle King Blue Origin TV Journalist
Lauren Sanchez Blue Origin Helicopter Pilot and former TV Journalist
Aisha Bowe Engineering firm Former NASA Rocket Scientist
Amanda Nguyen Research Scientist Activist and Scientist
Kerianne Flynn Movie Producer Producer
Billy McFarland Fyre Festival Organiser
Mark Zuckerberg Facebook CEO
```
We can also explore the database in a web interface using [Datasette](https://datasette.io/):
```bash
uvx datasette data.db
# Or install datasette first:
uv tool install datasette # or pip install or pipx install
datasette data.db
```
Visit `http://127.0.0.1:8001/data/people` to start navigating the data.
(schemas-json-schemas)=
## Using JSON schemas
The above examples have both used {ref}`concise schema syntax <schemas-dsl>`. LLM converts this format to [JSON schema](https://json-schema.org/), and you can use JSON schema directly yourself if you wish.
JSON schema covers the following:
- The data types of fields (string, number, array, object, etc.)
- Required vs. optional fields
- Nested data structures
- Constraints on values (minimum/maximum, patterns, etc.)
- Descriptions of those fields - these can be used to guide the language model
Different models may support different subsets of the overall JSON schema language. You should experiment to figure out what works for the model you are using.
LLM recommends that the top level of the schema is an object, not an array, for increased compatibility across multiple models. I suggest using `{"items": [array of objects]}` if you want to return an array.
The dogs schema above, `name, age int, one_sentence_bio`, would look like this as a full JSON schema:
```json
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"age": {
"type": "integer"
},
"one_sentence_bio": {
"type": "string"
}
},
"required": [
"name",
"age",
"one_sentence_bio"
]
}
```
This JSON can be passed directly to the `--schema` option, or saved in a file and passed as the filename.
```bash
llm --schema '{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"age": {
"type": "integer"
},
"one_sentence_bio": {
"type": "string"
}
},
"required": [
"name",
"age",
"one_sentence_bio"
]
}' 'a surprising dog'
```
Example output:
```json
{
"name": "Baxter",
"age": 3,
"one_sentence_bio": "Baxter is a rescue dog who learned to skateboard and now performs tricks at local parks, astonishing everyone with his skill!"
}
```
(schemas-specify)=
## Ways to specify a schema
LLM accepts schema definitions for both running prompts and exploring logged responses, using the `--schema` option.
This option can take multiple forms:
- A string providing a JSON schema: `--schema '{"type": "object", ...}'`
- A {ref}`condensed schema definition <schemas-dsl>`: `--schema 'name,age int'`
- The name or path of a file on disk containing a JSON schema: `--schema dogs.schema.json`
- The hexadecimal ID of a previously logged schema: `--schema 520f7aabb121afd14d0c6c237b39ba2d` - these IDs can be found using the `llm schemas` command.
- A schema that has been {ref}`saved in a template <prompt-templates-save>`: `--schema t:name-of-template`, see {ref}`schemas-reusable`.
(schemas-dsl)=
## Concise LLM schema syntax
JSON schema's can be time-consuming to construct by hand. LLM also supports a concise alternative syntax for specifying a schema.
A simple schema for an object with two string properties called `name` and `bio` looks like this:
name, bio
You can include type information by adding a type indicator after the property name, separated by a space.
name, bio, age int
Supported types are `int` for integers, `float` for floating point numbers, `str` for strings (the default) and `bool` for true/false booleans.
To include a description of the field to act as a hint to the model, add one after a colon:
name: the person's name, age int: their age, bio: a short bio
If your schema is getting long you can switch from comma-separated to newline-separated, which also allows you to use commas in those descriptions:
name: the person's name
age int: their age
bio: a short bio, no more than three sentences
You can experiment with the syntax using the `llm schemas dsl` command, which converts the input into a JSON schema:
```bash
llm schemas dsl 'name, age int'
```
Output:
```json
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"age": {
"type": "integer"
}
},
"required": [
"name",
"age"
]
}
```
The Python utility function `llm.schema_dsl(schema)` can be used to convert this syntax into the equivalent JSON schema dictionary when working with schemas {ref}`in the Python API <python-api-schemas>`.
(schemas-reusable)=
## Saving reusable schemas in templates
If you want to store a schema with a name so you can reuse it easily in the future, the easiest way to do so is to save it {ref}`in a template <prompt-templates-schemas>`.
The quickest way to do that is with the `llm --save` option:
```bash
llm --schema 'name, age int, one_sentence_bio' --save dog
```
Now you can use it like this:
```bash
llm --schema t:dog 'invent a dog'
```
Or:
```bash
llm --schema-multi t:dog 'invent three dogs'
```
(schemas-logs)=
## Browsing logged JSON objects created using schemas
By default, all JSON produced using schemas is logged to {ref}`a SQLite database <logging>`. You can use special options to the `llm logs` command to extract just those JSON objects in a useful format.
The `llm logs --schema X` filter option can be used to filter just for responses that were created using the specified schema. You can pass the full schema JSON, a path to the schema on disk or the schema ID.
The `--data` option causes just the JSON data collected by that schema to be outputted, as newline-delimited JSON.
If you instead want a JSON array of objects (with starting and ending square braces) you can use `--data-array` instead.
Let's invent some dogs:
```bash
llm --schema-multi 'name, ten_word_bio' 'invent 3 cool dogs'
llm --schema-multi 'name, ten_word_bio' 'invent 2 cool dogs'
```
Having logged these cool dogs, you can see just the data that was returned by those prompts like this:
```bash
llm logs --schema-multi 'name, ten_word_bio' --data
```
We need to use `--schema-multi` here because we used that when we first created these records. The `--schema` option is also supported, and can be passed a filename or JSON schema or schema ID as well.
Output:
```
{"items": [{"name": "Robo", "ten_word_bio": "A cybernetic dog with laser eyes and super intelligence."}, {"name": "Flamepaw", "ten_word_bio": "Fire-resistant dog with a talent for agility and tricks."}]}
{"items": [{"name": "Bolt", "ten_word_bio": "Lightning-fast border collie, loves frisbee and outdoor adventures."}, {"name": "Luna", "ten_word_bio": "Mystical husky with mesmerizing blue eyes, enjoys snow and play."}, {"name": "Ziggy", "ten_word_bio": "Quirky pug who loves belly rubs and quirky outfits."}]}
```
Note that the dogs are nested in that `"items"` key. To access the list of items from that key use `--data-key items`:
```bash
llm logs --schema-multi 'name, ten_word_bio' --data-key items
```
Output:
```
{"name": "Bolt", "ten_word_bio": "Lightning-fast border collie, loves frisbee and outdoor adventures."}
{"name": "Luna", "ten_word_bio": "Mystical husky with mesmerizing blue eyes, enjoys snow and play."}
{"name": "Ziggy", "ten_word_bio": "Quirky pug who loves belly rubs and quirky outfits."}
{"name": "Robo", "ten_word_bio": "A cybernetic dog with laser eyes and super intelligence."}
{"name": "Flamepaw", "ten_word_bio": "Fire-resistant dog with a talent for agility and tricks."}
```
Finally, to output a JSON array instead of newline-delimited JSON use `--data-array`:
```bash
llm logs --schema-multi 'name, ten_word_bio' --data-key items --data-array
```
Output:
```json
[{"name": "Bolt", "ten_word_bio": "Lightning-fast border collie, loves frisbee and outdoor adventures."},
{"name": "Luna", "ten_word_bio": "Mystical husky with mesmerizing blue eyes, enjoys snow and play."},
{"name": "Ziggy", "ten_word_bio": "Quirky pug who loves belly rubs and quirky outfits."},
{"name": "Robo", "ten_word_bio": "A cybernetic dog with laser eyes and super intelligence."},
{"name": "Flamepaw", "ten_word_bio": "Fire-resistant dog with a talent for agility and tricks."}]
```
Add `--data-ids` to include `"response_id"` and `"conversation_id"` fields in each of the returned objects reflecting the database IDs of the response and conversation they were a part of. This can be useful for tracking the source of each individual row.
```bash
llm logs --schema-multi 'name, ten_word_bio' --data-key items --data-ids
```
Output:
```json
{"name": "Nebula", "ten_word_bio": "A cosmic puppy with starry fur, loves adventures in space.", "response_id": "01jn4dawj8sq0c6t3emf4k5ryx", "conversation_id": "01jn4dawj8sq0c6t3emf4k5ryx"}
{"name": "Echo", "ten_word_bio": "A clever hound with extraordinary hearing, master of hide-and-seek.", "response_id": "01jn4dawj8sq0c6t3emf4k5ryx", "conversation_id": "01jn4dawj8sq0c6t3emf4k5ryx"}
{"name": "Biscuit", "ten_word_bio": "An adorable chef dog, bakes treats that everyone loves.", "response_id": "01jn4dawj8sq0c6t3emf4k5ryx", "conversation_id": "01jn4dawj8sq0c6t3emf4k5ryx"}
{"name": "Cosmo", "ten_word_bio": "Galactic explorer, loves adventures and chasing shooting stars.", "response_id": "01jn4daycb3svj0x7kvp7zrp4q", "conversation_id": "01jn4daycb3svj0x7kvp7zrp4q"}
{"name": "Pixel", "ten_word_bio": "Tech-savvy pup, builds gadgets and loves virtual playtime.", "response_id": "01jn4daycb3svj0x7kvp7zrp4q", "conversation_id": "01jn4daycb3svj0x7kvp7zrp4q"}
```
If a row already has a property called `"conversation_id"` or `"response_id"` additional underscores will be appended to the ID key until it no longer overlaps with the existing keys.
The `--id-gt $ID` and `--id-gte $ID` options can be useful for ignoring logged schema data prior to a certain point, see {ref}`logging-filter-id` for details.
docs/setup.md
+208
View File
@@ -0,0 +1,208 @@
# Setup
## Installation
Install this tool using `pip`:
```bash
pip install llm
```
Or using [pipx](https://pypa.github.io/pipx/):
```bash
pipx install llm
```
Or using [uv](https://docs.astral.sh/uv/guides/tools/) ({ref}`more tips below <setup-uvx>`):
```bash
uv tool install llm
```
Or using [Homebrew](https://brew.sh/) (see {ref}`warning note <homebrew-warning>`):
```bash
brew install llm
```
## Upgrading to the latest version
If you installed using `pip`:
```bash
pip install -U llm
```
For `pipx`:
```bash
pipx upgrade llm
```
For `uv`:
```bash
uv tool upgrade llm
```
For Homebrew:
```bash
brew upgrade llm
```
If the latest version is not yet available on Homebrew you can upgrade like this instead:
```bash
llm install -U llm
```
(setup-uvx)=
## Using uvx
If you have [uv](https://docs.astral.sh/uv/) installed you can also use the `uvx` command to try LLM without first installing it like this:
```bash
export OPENAI_API_KEY='sx-...'
uvx llm 'fun facts about skunks'
```
This will install and run LLM using a temporary virtual environment.
You can use the `--with` option to add extra plugins. To use Anthropic's models, for example:
```bash
export ANTHROPIC_API_KEY='...'
uvx --with llm-anthropic llm -m claude-3.5-haiku 'fun facts about skunks'
```
All of the usual LLM commands will work with `uvx llm`. Here's how to set your OpenAI key without needing an environment variable for example:
```bash
uvx llm keys set openai
# Paste key here
```
(homebrew-warning)=
## A note about Homebrew and PyTorch
The version of LLM packaged for Homebrew currently uses Python 3.12. The PyTorch project do not yet have a stable release of PyTorch for that version of Python.
This means that LLM plugins that depend on PyTorch such as [llm-sentence-transformers](https://github.com/simonw/llm-sentence-transformers) may not install cleanly with the Homebrew version of LLM.
You can workaround this by manually installing PyTorch before installing `llm-sentence-transformers`:
```bash
llm install llm-python
llm python -m pip install \
--pre torch torchvision \
--index-url https://download.pytorch.org/whl/nightly/cpu
llm install llm-sentence-transformers
```
This should produce a working installation of that plugin.
## Installing plugins
{ref}`plugins` can be used to add support for other language models, including models that can run on your own device.
For example, the [llm-gpt4all](https://github.com/simonw/llm-gpt4all) plugin adds support for 17 new models that can be installed on your own machine. You can install that like so:
```bash
llm install llm-gpt4all
```
(api-keys)=
## API key management
Many LLM models require an API key. These API keys can be provided to this tool using several different mechanisms.
You can obtain an API key for OpenAI's language models from [the API keys page](https://platform.openai.com/api-keys) on their site.
### Saving and using stored keys
The easiest way to store an API key is to use the `llm keys set` command:
```bash
llm keys set openai
```
You will be prompted to enter the key like this:
```
% llm keys set openai
Enter key:
```
Once stored, this key will be automatically used for subsequent calls to the API:
```bash
llm "Five ludicrous names for a pet lobster"
```
You can list the names of keys that have been set using this command:
```bash
llm keys
```
Keys that are stored in this way live in a file called `keys.json`. This file is located at the path shown when you run the following command:
```bash
llm keys path
```
On macOS this will be `~/Library/Application Support/io.datasette.llm/keys.json`. On Linux it may be something like `~/.config/io.datasette.llm/keys.json`.
### Passing keys using the --key option
Keys can be passed directly using the `--key` option, like this:
```bash
llm "Five names for pet weasels" --key sk-my-key-goes-here
```
You can also pass the alias of a key stored in the `keys.json` file. For example, if you want to maintain a personal API key you could add that like this:
```bash
llm keys set personal
```
And then use it for prompts like so:
```bash
llm "Five friendly names for a pet skunk" --key personal
```
### Keys in environment variables
Keys can also be set using an environment variable. These are different for different models.
For OpenAI models the key will be read from the `OPENAI_API_KEY` environment variable.
The environment variable will be used if no `--key` option is passed to the command and there is not a key configured in `keys.json`
To use an environment variable in place of the `keys.json` key run the prompt like this:
```bash
llm 'my prompt' --key $OPENAI_API_KEY
```
## Configuration
You can configure LLM in a number of different ways.
(setup-default-model)=
### Setting a custom default model
The model used when calling `llm` without the `-m/--model` option defaults to `gpt-4o-mini` - the fastest and least expensive OpenAI model.
You can use the `llm models default` command to set a different default model. For GPT-4o (slower and more expensive, but more capable) run this:
```bash
llm models default gpt-4o
```
You can view the current model by running this:
```
llm models default
```
Any of the supported aliases for a model can be passed to this command.
### Setting a custom directory location
This tool stores various files - prompt templates, stored keys, preferences, a database of logs - in a directory on your computer.
On macOS this is `~/Library/Application Support/io.datasette.llm/`.
On Linux it may be something like `~/.config/io.datasette.llm/`.
You can set a custom location for this directory by setting the `LLM_USER_PATH` environment variable:
```bash
export LLM_USER_PATH=/path/to/my/custom/directory
```
### Turning SQLite logging on and off
By default, LLM will log every prompt and response you make to a SQLite database - see {ref}`logging` for more details.
You can turn this behavior off by default by running:
```bash
llm logs off
```
Or turn it back on again with:
```
llm logs on
```
Run `llm logs status` to see the current states of the setting.
docs/templates.md
+387
View File
@@ -0,0 +1,387 @@
(prompt-templates)=
# Templates
A **template** can combine a prompt, system prompt, model, default model options, schema, and fragments into a single reusable unit.
Only one template can be used at a time. To compose multiple shorter pieces of prompts together consider using {ref}`fragments <fragments>` instead.
(prompt-templates-save)=
## Getting started with <code>--save</code>
The easiest way to create a template is using the `--save template_name` option.
Here's how to create a template for summarizing text:
```bash
llm '$input - summarize this' --save summarize
```
Put `$input` where you would like the user's input to be inserted. If you omit this their input will be added to the end of your regular prompt:
```bash
llm 'Summarize the following: ' --save summarize
```
You can also create templates using system prompts:
```bash
llm --system 'Summarize this' --save summarize
```
You can set the default model for a template using `--model`:
```bash
llm --system 'Summarize this' --model gpt-4o --save summarize
```
You can also save default options:
```bash
llm --system 'Speak in French' -o temperature 1.8 --save wild-french
```
If you want to include a literal `$` sign in your prompt, use `$$` instead:
```bash
llm --system 'Estimate the cost in $$ of this: $input' --save estimate
```
Use `--tool/-T` one or more times to add tools to the template:
```bash
llm -T llm_time --system 'Always include the current time in the answer' --save time
```
You can also use `--functions` to add Python function code directly to the template:
```bash
llm --functions 'def reverse_string(s): return s[::-1]' --system 'reverse any input' --save reverse
llm -t reverse 'Hello, world!'
```
Add `--schema` to bake a {ref}`schema <usage-schemas>` into your template:
```bash
llm --schema dog.schema.json 'invent a dog' --save dog
```
If you add `--extract` the setting to {ref}`extract the first fenced code block <usage-extract-fenced-code>` will be persisted in the template.
```bash
llm --system 'write a Python function' --extract --save python-function
llm -t python-function 'calculate haversine distance between two points'
```
In each of these cases the template will be saved in YAML format in a dedicated directory on disk.
(prompt-templates-using)=
## Using a template
You can execute a named template using the `-t/--template` option:
```bash
curl -s https://example.com/ | llm -t summarize
```
This can be combined with the `-m` option to specify a different model:
```bash
curl -s https://llm.datasette.io/en/latest/ | \
llm -t summarize -m gpt-3.5-turbo-16k
```
Templates can also be specified as a direct path to a YAML file on disk:
```bash
llm -t path/to/template.yaml 'extra prompt here'
```
Or as a URL to a YAML file hosted online:
```bash
llm -t https://raw.githubusercontent.com/simonw/llm-templates/refs/heads/main/python-app.yaml \
'Python app to pick a random line from a file'
```
Note that templates loaded via URLs will have any `functions:` keys ignored, to avoid accidentally executing arbitrary code. This restriction also applies to templates loaded via the {ref}`template loaders plugin mechanism <plugin-hooks-register-template-loaders>`.
(prompt-templates-list)=
## Listing available templates
This command lists all available templates:
```bash
llm templates
```
The output looks something like this:
```
cmd : system: reply with macos terminal commands only, no extra information
glados : system: You are GlaDOS prompt: Summarize this:
```
(prompt-templates-yaml)=
## Templates as YAML files
Templates are stored as YAML files on disk.
You can edit (or create) a YAML file for a template using the `llm templates edit` command:
```
llm templates edit summarize
```
This will open the system default editor.
:::{tip}
You can control which editor will be used here using the `EDITOR` environment variable - for example, to use VS Code:
```bash
export EDITOR="code -w"
```
Add that to your `~/.zshrc` or `~/.bashrc` file depending on which shell you use (`zsh` is the default on macOS since macOS Catalina in 2019).
:::
You can create or edit template files directly in the templates directory. The location of this directory is shown by the `llm templates path` command:
```bash
llm templates path
```
Example output:
```
/Users/simon/Library/Application Support/io.datasette.llm/templates
```
A basic YAML template looks like this:
```yaml
prompt: 'Summarize this: $input'
```
Or use YAML multi-line strings for longer inputs. I created this using `llm templates edit steampunk`:
```yaml
prompt: >
Summarize the following text.
Insert frequent satirical steampunk-themed illustrative anecdotes.
Really go wild with that.
Text to summarize: $input
```
The `prompt: >` causes the following indented text to be treated as a single string, with newlines collapsed to spaces. Use `prompt: |` to preserve newlines.
Running that with `llm -t steampunk` against GPT-4o (via [strip-tags](https://github.com/simonw/strip-tags) to remove HTML tags from the input and minify whitespace):
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t steampunk -m gpt-4o
```
Output:
> In a fantastical steampunk world, Simon Willison decided to merge an old MP3 recording with slides from the talk using iMovie. After exporting the slides as images and importing them into iMovie, he had to disable the default Ken Burns effect using the "Crop" tool. Then, Simon manually synchronized the audio by adjusting the duration of each image. Finally, he published the masterpiece to YouTube, with the whimsical magic of steampunk-infused illustrations leaving his viewers in awe.
(prompt-templates-system)=
### System prompts
When working with models that support system prompts you can set a system prompt using a `system:` key like so:
```yaml
system: Summarize this
```
If you specify only a system prompt you don't need to use the `$input` variable - `llm` will use the user's input as the whole of the regular prompt, which will then be processed using the instructions set in that system prompt.
You can combine system and regular prompts like so:
```yaml
system: You speak like an excitable Victorian adventurer
prompt: 'Summarize this: $input'
```
(prompt-templates-fragments)=
### Fragments
Templates can reference {ref}`Fragments <fragments>` using the `fragments:` and `system_fragments:` keys. These should be a list of fragment URLs, filepaths or hashes:
```yaml
fragments:
- https://example.com/robots.txt
- /path/to/file.txt
- 993fd38d898d2b59fd2d16c811da5bdac658faa34f0f4d411edde7c17ebb0680
system_fragments:
- https://example.com/systm-prompt.txt
```
(prompt-templates-options)=
### Options
Default options can be set using the `options:` key:
```yaml
name: wild-french
system: Speak in French
options:
temperature: 1.8
```
(prompt-templates-tools)=
### Tools
The `tools:` key can provide a list of tool names from other plugins - either function names or toolbox specifiers:
```yaml
name: time-plus
tools:
- llm_time
- Datasette("https://example.com/timezone-lookup")
```
The `functions:` key can provide a multi-line string of Python code defining additional functions:
```yaml
name: my-functions
functions: |
def reverse_string(s: str):
return s[::-1]
def greet(name: str):
return f"Hello, {name}!"
```
(prompt-templates-schemas)=
### Schemas
Use the `schema_object:` key to embed a JSON schema (as YAML) in your template. The easiest way to create these is with the `llm --schema ... --save name-of-template` command - the result should look something like this:
```yaml
name: dogs
schema_object:
properties:
dogs:
items:
properties:
bio:
type: string
name:
type: string
type: object
type: array
type: object
```
(prompt-templates-variables)=
### Additional template variables
Templates that work against the user's normal prompt input (content that is either piped to the tool via standard input or passed as a command-line argument) can use the `$input` variable.
You can use additional named variables. These will then need to be provided using the `-p/--param` option when executing the template.
Here's an example YAML template called `recipe`, which you can create using `llm templates edit recipe`:
```yaml
prompt: |
Suggest a recipe using ingredients: $ingredients
It should be based on cuisine from this country: $country
```
This can be executed like so:
```bash
llm -t recipe -p ingredients 'sausages, milk' -p country Germany
```
My output started like this:
> Recipe: German Sausage and Potato Soup
>
> Ingredients:
> - 4 German sausages
> - 2 cups whole milk
This example combines input piped to the tool with additional parameters. Call this `summarize`:
```yaml
system: Summarize this text in the voice of $voice
```
Then to run it:
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t summarize -p voice GlaDOS
```
I got this:
> My previous test subject seemed to have learned something new about iMovie. They exported keynote slides as individual images [...] Quite impressive for a human.
(prompt-default-parameters)=
### Specifying default parameters
When creating a template using the `--save` option you can pass `-p name value` to store the default values for parameters:
```bash
llm --system 'Summarize this text in the voice of $voice' \
--model gpt-4o -p voice GlaDOS --save summarize
```
You can specify default values for parameters in the YAML using the `defaults:` key.
```yaml
system: Summarize this text in the voice of $voice
defaults:
voice: GlaDOS
```
When running without `-p` it will choose the default:
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t summarize
```
But you can override the defaults with `-p`:
```bash
curl -s 'https://til.simonwillison.net/macos/imovie-slides-and-audio' | \
strip-tags -m | llm -t summarize -p voice Yoda
```
I got this:
> Text, summarize in Yoda's voice, I will: "Hmm, young padawan. Summary of this text, you seek. Hmmm. ...
(prompt-templates-extract)=
### Configuring code extraction
To configure the {ref}`extract first fenced code block <usage-extract-fenced-code>` setting for the template, add this:
```yaml
extract: true
```
(prompt-templates-default-model)=
### Setting a default model for a template
Templates executed using `llm -t template-name` will execute using the default model that the user has configured for the tool - or `gpt-3.5-turbo` if they have not configured their own default.
You can specify a new default model for a template using the `model:` key in the associated YAML. Here's a template called `roast`:
```yaml
model: gpt-4o
system: roast the user at every possible opportunity, be succinct
```
Example:
```bash
llm -t roast 'How are you today?'
```
> I'm doing great but with your boring questions, I must admit, I've seen more life in a cemetery.
(prompt-templates-loaders)=
## Template loaders from plugins
LLM plugins can {ref}`register prefixes <plugin-hooks-register-template-loaders>` that can be used to load templates from external sources.
[llm-templates-github](https://github.com/simonw/llm-templates-github) is an example which adds a `gh:` prefix which can be used to load templates from GitHub.
You can install that plugin like this:
```bash
llm install llm-templates-github
```
Use the `llm templates loaders` command to see details of the registered loaders.
```bash
llm templates loaders
```
Output:
```
gh:
Load a template from GitHub or local cache if available
Format: username/repo/template_name (without the .yaml extension)
or username/template_name which means username/llm-templates/template_name
```
Then you can then use it like this:
```bash
curl -sL 'https://llm.datasette.io/' | llm -t gh:simonw/summarize
```
The `-sL` flags to `curl` are used to follow redirects and suppress progress meters.
This command will fetch the content of the LLM index page and feed it to the template defined by [summarize.yaml](https://github.com/simonw/llm-templates/blob/main/summarize.yaml) in the [simonw/llm-templates](https://github.com/simonw/llm-templates) GitHub repository.
If two template loader plugins attempt to register the same prefix one of them will have `_1` added to the end of their prefix. Use `llm templates loaders` to check if this has occurred.
docs/tools.md
+102
View File
@@ -0,0 +1,102 @@
(tools)=
# Tools
Many Large Language Models have been trained to execute tools as part of responding to a prompt. LLM supports tool usage with both the command-line interface and the Python API.
Exposing tools to LLMs **carries risks**! Be sure to read the {ref}`warning below <tools-warning>`.
(tools-how-they-work)=
## How tools work
A tool is effectively a function that the model can request to be executed. Here's how that works:
1. The initial prompt to the model includes a list of available tools, containing their names, descriptions and parameters.
2. The model can choose to call one (or sometimes more than one) of those tools, returning a request for the tool to execute.
3. The code that calls the model - in this case LLM itself - then executes the specified tool with the provided arguments.
4. LLM prompts the model a second time, this time including the output of the tool execution.
5. The model can then use that output to generate its next response.
This sequence can run several times in a loop, allowing the LLM to access data, act on that data and then pass that data off to other tools for further processing.
:::{admonition} Tools can be dangerous
:class: danger
(tools-warning)=
## Warning: Tools can be dangerous
Applications built on top of LLMs suffer from a class of attacks called [prompt injection](https://simonwillison.net/tags/prompt-injection/) attacks. These occur when a malicious third party injects content into the LLM which causes it to take tool-based actions that act against the interests of the user of that application.
Be very careful about which tools you enable when you potentially might be exposed to untrusted sources of content - web pages, GitHub issues posted by other people, email and messages that have been sent to you that could come from an attacker.
Watch out for [the lethal trifecta](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) of prompt injection exfiltration attacks. If your tool-enabled LLM has the following:
- access to private data
- exposure to malicious instructions
- the ability to exfiltrate information
Anyone who can feed malicious instructions into your LLM - by leaving them on a web page it visits, or sending an email to an inbox that it monitors - could be able to trick your LLM into using other tools to access your private information and then exfiltrate (pass out) that data to somewhere the attacker can see it.
:::
(tools-trying-out)=
## Trying out tools
LLM comes with a default tool installed, called `llm_version`. You can try that out like this:
```bash
llm --tool llm_version "What version of LLM is this?" --td
```
You can also use `-T llm_version` as a shortcut for `--tool llm_version`.
The output should look like this:
```
Tool call: llm_version({})
0.26a0
The installed version of the LLM is 0.26a0.
```
Further tools can be installed using plugins, or you can use the `llm --functions` option to pass tools implemented as PYthon functions directly, as {ref}`described here <usage-tools>`.
(tools-implementation)=
## LLM's implementation of tools
In LLM every tool is defined as a Python function. The function can take any number of arguments and can return a string or an object that can be converted to a string.
Tool functions should include a docstring that describes what the function does. This docstring will become the description that is passed to the model.
Tools can also be defined as {ref}`toolbox classes <python-api-toolbox>`, a subclass of `llm.Toolbox` that allows multiple related tools to be bundled together. Toolbox classes can be configured when they are instantiated, and can also maintain state in between multiple tool calls.
The Python API can accept functions directly. The command-line interface has two ways for tools to be defined: via plugins that implement the {ref}`register_tools() plugin hook <plugin-hooks-register-tools>`, or directly on the command-line using the `--functions` argument to specify a block of Python code defining one or more functions - or a path to a Python file containing the same.
You can use tools {ref}`with the LLM command-line tool <usage-tools>` or {ref}`with the Python API <python-api-tools>`.
(tools-default)=
## Default tools
LLM includes some default tools for you to try out:
- `llm_version()` returns the current version of LLM
- `llm_time()` returns the current local and UTC time
Try them like this:
```bash
llm -T llm_version -T llm_time 'Give me the current time and LLM version' --td
```
(tools-tips)=
## Tips for implementing tools
Consult the {ref}`register_tools() plugin hook <plugin-hooks-register-tools>` documentation for examples of how to implement tools in plugins.
If your plugin needs access to API secrets I recommend storing those using `llm keys set api-name` and then reading them using the {ref}`plugin-utilities-get-key` utility function. This avoids secrets being logged to the database as part of tool calls.
If your tool implementation needs to know which tool call invoked it - for example to key state against the unique `tool_call_id` - add a parameter named `llm_tool_call` to your function. It will be passed the `llm.ToolCall` object for the current invocation, and is hidden from the schema the model sees. See {ref}`python-api-tools-llm-tool-call` for details.
<!-- Uncomment when this is true: The [llm-tools-datasette](https://github.com/simonw/llm-tools-datasette) plugin is a good example of this pattern in action. -->
docs/usage.md
+1666
View File
File diff suppressed because it is too large Load Diff