chore: import upstream snapshot with attribution
CI / changes (push) Has been cancelled
CI / cd libs/checkpoint (push) Has been cancelled
CI / cd libs/checkpoint-conformance (push) Has been cancelled
CI / cd libs/checkpoint-postgres (push) Has been cancelled
CI / cd libs/checkpoint-sqlite (push) Has been cancelled
CI / cd libs/cli (push) Has been cancelled
CI / cd libs/prebuilt (push) Has been cancelled
CI / cd libs/sdk-py (push) Has been cancelled
CI / cd libs/langgraph (push) Has been cancelled
CI / Check SDK methods matching (push) Has been cancelled
CI / Check CLI schema hasn't changed #3.13 (push) Has been cancelled
CI / CLI integration test (push) Has been cancelled
CI / sdk-py integration test (push) Has been cancelled
CI / CI Success (push) Has been cancelled
baseline / benchmark (push) Has been cancelled
Deploy Redirects to GitHub Pages / deploy (push) Has been cancelled
CI / changes (push) Has been cancelled
CI / cd libs/checkpoint (push) Has been cancelled
CI / cd libs/checkpoint-conformance (push) Has been cancelled
CI / cd libs/checkpoint-postgres (push) Has been cancelled
CI / cd libs/checkpoint-sqlite (push) Has been cancelled
CI / cd libs/cli (push) Has been cancelled
CI / cd libs/prebuilt (push) Has been cancelled
CI / cd libs/sdk-py (push) Has been cancelled
CI / cd libs/langgraph (push) Has been cancelled
CI / Check SDK methods matching (push) Has been cancelled
CI / Check CLI schema hasn't changed #3.13 (push) Has been cancelled
CI / CLI integration test (push) Has been cancelled
CI / sdk-py integration test (push) Has been cancelled
CI / CI Success (push) Has been cancelled
baseline / benchmark (push) Has been cancelled
Deploy Redirects to GitHub Pages / deploy (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 LangChain, Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,37 @@
|
||||
.PHONY: test test_watch lint type format
|
||||
|
||||
######################
|
||||
# TESTING AND COVERAGE
|
||||
######################
|
||||
|
||||
TEST ?= .
|
||||
|
||||
test:
|
||||
uv run pytest $(TEST)
|
||||
|
||||
test_watch:
|
||||
uv run ptw $(TEST)
|
||||
|
||||
######################
|
||||
# LINTING AND FORMATTING
|
||||
######################
|
||||
|
||||
# Define a variable for Python and notebook files.
|
||||
PYTHON_FILES=.
|
||||
lint format: PYTHON_FILES=.
|
||||
lint_diff format_diff: PYTHON_FILES=$(shell git diff --name-only --relative --diff-filter=d main . | grep -E '\.py$$|\.ipynb$$')
|
||||
lint_package: PYTHON_FILES=langgraph
|
||||
lint_tests: PYTHON_FILES=tests
|
||||
|
||||
lint lint_diff lint_package lint_tests:
|
||||
uv run ruff check .
|
||||
[ "$(PYTHON_FILES)" = "" ] || uv run ruff format $(PYTHON_FILES) --diff
|
||||
[ "$(PYTHON_FILES)" = "" ] || uv run ruff check --select I $(PYTHON_FILES)
|
||||
[ "$(PYTHON_FILES)" = "" ] || uv run ty check $(PYTHON_FILES)
|
||||
|
||||
type:
|
||||
uv run ty check $(PYTHON_FILES)
|
||||
|
||||
format format_diff:
|
||||
uv run ruff format $(PYTHON_FILES)
|
||||
uv run ruff check --select I --fix $(PYTHON_FILES)
|
||||
@@ -0,0 +1,123 @@
|
||||
# LangGraph SQLite Checkpoint
|
||||
|
||||
[](https://pypi.org/project/langgraph-checkpoint-sqlite/#history)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://pypistats.org/packages/langgraph-checkpoint-sqlite)
|
||||
[](https://x.com/langchain_oss)
|
||||
|
||||
To help you ship LangGraph apps to production faster, check out [LangSmith](https://www.langchain.com/langsmith).
|
||||
[LangSmith](https://www.langchain.com/langsmith) is a unified developer platform for building, testing, and monitoring LLM applications.
|
||||
|
||||
## Quick Install
|
||||
|
||||
```bash
|
||||
uv add langgraph-checkpoint-sqlite
|
||||
```
|
||||
|
||||
## 🤔 What is this?
|
||||
|
||||
This library provides a SQLite implementation of LangGraph's checkpoint saver, with both sync and async support via `aiosqlite`. Use it when you want LangGraph state persistence backed by SQLite for local development, testing, or lightweight deployments.
|
||||
|
||||
## 📖 Documentation
|
||||
|
||||
For full documentation, see the [API reference](https://reference.langchain.com/python/langgraph.checkpoint.sqlite). For conceptual guides on persistence and memory, see the [LangGraph Docs](https://docs.langchain.com/oss/python/langgraph/overview).
|
||||
|
||||
## Security
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Set `LANGGRAPH_STRICT_MSGPACK=true` or pass an explicit `allowed_msgpack_modules` list when creating your checkpointer. This restricts checkpoint deserialization to known-safe types, preventing code execution if the database is compromised. See the [langgraph-checkpoint README](https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint#serde) for details.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from langgraph.checkpoint.sqlite import SqliteSaver
|
||||
|
||||
write_config = {"configurable": {"thread_id": "1", "checkpoint_ns": ""}}
|
||||
read_config = {"configurable": {"thread_id": "1"}}
|
||||
|
||||
with SqliteSaver.from_conn_string(":memory:") as checkpointer:
|
||||
checkpoint = {
|
||||
"v": 4,
|
||||
"ts": "2024-07-31T20:14:19.804150+00:00",
|
||||
"id": "1ef4f797-8335-6428-8001-8a1503f9b875",
|
||||
"channel_values": {
|
||||
"my_key": "meow",
|
||||
"node": "node"
|
||||
},
|
||||
"channel_versions": {
|
||||
"__start__": 2,
|
||||
"my_key": 3,
|
||||
"start:node": 3,
|
||||
"node": 3
|
||||
},
|
||||
"versions_seen": {
|
||||
"__input__": {},
|
||||
"__start__": {
|
||||
"__start__": 1
|
||||
},
|
||||
"node": {
|
||||
"start:node": 2
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
# store checkpoint
|
||||
checkpointer.put(write_config, checkpoint, {}, {})
|
||||
|
||||
# load checkpoint
|
||||
checkpointer.get(read_config)
|
||||
|
||||
# list checkpoints
|
||||
list(checkpointer.list(read_config))
|
||||
```
|
||||
|
||||
### Async
|
||||
|
||||
```python
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as checkpointer:
|
||||
checkpoint = {
|
||||
"v": 4,
|
||||
"ts": "2024-07-31T20:14:19.804150+00:00",
|
||||
"id": "1ef4f797-8335-6428-8001-8a1503f9b875",
|
||||
"channel_values": {
|
||||
"my_key": "meow",
|
||||
"node": "node"
|
||||
},
|
||||
"channel_versions": {
|
||||
"__start__": 2,
|
||||
"my_key": 3,
|
||||
"start:node": 3,
|
||||
"node": 3
|
||||
},
|
||||
"versions_seen": {
|
||||
"__input__": {},
|
||||
"__start__": {
|
||||
"__start__": 1
|
||||
},
|
||||
"node": {
|
||||
"start:node": 2
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
# store checkpoint
|
||||
await checkpointer.aput(write_config, checkpoint, {}, {})
|
||||
|
||||
# load checkpoint
|
||||
await checkpointer.aget(read_config)
|
||||
|
||||
# list checkpoints
|
||||
[c async for c in checkpointer.alist(read_config)]
|
||||
```
|
||||
|
||||
## 📕 Releases & Versioning
|
||||
|
||||
See our [Releases](https://docs.langchain.com/oss/python/release-policy) and [Versioning](https://docs.langchain.com/oss/python/versioning) policies.
|
||||
|
||||
## 💁 Contributing
|
||||
|
||||
As an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.
|
||||
|
||||
For detailed information on how to contribute, see the [Contributing Guide](https://docs.langchain.com/oss/python/contributing/overview).
|
||||
@@ -0,0 +1,120 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import datetime
|
||||
import sqlite3
|
||||
import threading
|
||||
from collections.abc import Mapping, Sequence
|
||||
|
||||
from langgraph.cache.base import BaseCache, FullKey, Namespace, ValueT
|
||||
from langgraph.checkpoint.serde.base import SerializerProtocol
|
||||
|
||||
|
||||
class SqliteCache(BaseCache[ValueT]):
|
||||
"""File-based cache using SQLite."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
path: str,
|
||||
serde: SerializerProtocol | None = None,
|
||||
) -> None:
|
||||
"""Initialize the cache with a file path."""
|
||||
super().__init__(serde=serde)
|
||||
# SQLite backing store
|
||||
self._conn = sqlite3.connect(
|
||||
path,
|
||||
check_same_thread=False,
|
||||
)
|
||||
# Serialize access to the shared connection across threads
|
||||
self._lock = threading.RLock()
|
||||
# Better concurrency & atomicity
|
||||
self._conn.execute("PRAGMA journal_mode=WAL;")
|
||||
# Schema: key -> (expiry, encoding, value)
|
||||
self._conn.execute(
|
||||
"""CREATE TABLE IF NOT EXISTS cache (
|
||||
ns TEXT,
|
||||
key TEXT,
|
||||
expiry REAL,
|
||||
encoding TEXT NOT NULL,
|
||||
val BLOB NOT NULL,
|
||||
PRIMARY KEY (ns, key)
|
||||
)"""
|
||||
)
|
||||
self._conn.commit()
|
||||
|
||||
def get(self, keys: Sequence[FullKey]) -> dict[FullKey, ValueT]:
|
||||
"""Get the cached values for the given keys."""
|
||||
with self._lock, self._conn:
|
||||
now = datetime.datetime.now(datetime.timezone.utc).timestamp()
|
||||
if not keys:
|
||||
return {}
|
||||
placeholders = ",".join("(?, ?)" for _ in keys)
|
||||
params: list[str] = []
|
||||
for ns_tuple, key in keys:
|
||||
params.extend((",".join(ns_tuple), key))
|
||||
cursor = self._conn.execute(
|
||||
f"SELECT ns, key, expiry, encoding, val FROM cache WHERE (ns, key) IN ({placeholders})",
|
||||
tuple(params),
|
||||
)
|
||||
values: dict[FullKey, ValueT] = {}
|
||||
rows = cursor.fetchall()
|
||||
for ns, key, expiry, encoding, raw in rows:
|
||||
if expiry is not None and now > expiry:
|
||||
# purge expired entry
|
||||
self._conn.execute(
|
||||
"DELETE FROM cache WHERE (ns, key) = (?, ?)", (ns, key)
|
||||
)
|
||||
continue
|
||||
values[(tuple(ns.split(",")), key)] = self.serde.loads_typed(
|
||||
(encoding, raw)
|
||||
)
|
||||
return values
|
||||
|
||||
async def aget(self, keys: Sequence[FullKey]) -> dict[FullKey, ValueT]:
|
||||
"""Asynchronously get the cached values for the given keys."""
|
||||
return await asyncio.to_thread(self.get, keys)
|
||||
|
||||
def set(self, mapping: Mapping[FullKey, tuple[ValueT, int | None]]) -> None:
|
||||
"""Set the cached values for the given keys and TTLs."""
|
||||
with self._lock, self._conn:
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
for key, (value, ttl) in mapping.items():
|
||||
if ttl is not None:
|
||||
delta = datetime.timedelta(seconds=ttl)
|
||||
expiry: float | None = (now + delta).timestamp()
|
||||
else:
|
||||
expiry = None
|
||||
encoding, raw = self.serde.dumps_typed(value)
|
||||
self._conn.execute(
|
||||
"INSERT OR REPLACE INTO cache (ns, key, expiry, encoding, val) VALUES (?, ?, ?, ?, ?)",
|
||||
(",".join(key[0]), key[1], expiry, encoding, raw),
|
||||
)
|
||||
|
||||
async def aset(self, mapping: Mapping[FullKey, tuple[ValueT, int | None]]) -> None:
|
||||
"""Asynchronously set the cached values for the given keys and TTLs."""
|
||||
await asyncio.to_thread(self.set, mapping)
|
||||
|
||||
def clear(self, namespaces: Sequence[Namespace] | None = None) -> None:
|
||||
"""Delete the cached values for the given namespaces.
|
||||
If no namespaces are provided, clear all cached values."""
|
||||
with self._lock, self._conn:
|
||||
if namespaces is None:
|
||||
self._conn.execute("DELETE FROM cache")
|
||||
else:
|
||||
placeholders = ",".join("?" for _ in namespaces)
|
||||
self._conn.execute(
|
||||
f"DELETE FROM cache WHERE (ns) IN ({placeholders})",
|
||||
tuple(",".join(key) for key in namespaces),
|
||||
)
|
||||
|
||||
async def aclear(self, namespaces: Sequence[Namespace] | None = None) -> None:
|
||||
"""Asynchronously delete the cached values for the given namespaces.
|
||||
If no namespaces are provided, clear all cached values."""
|
||||
await asyncio.to_thread(self.clear, namespaces)
|
||||
|
||||
def __del__(self) -> None:
|
||||
try:
|
||||
self._conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,645 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import sqlite3
|
||||
import threading
|
||||
from collections.abc import AsyncIterator, Iterator, Mapping, Sequence
|
||||
from contextlib import closing, contextmanager
|
||||
from typing import Any, cast
|
||||
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from langgraph.checkpoint.base import (
|
||||
WRITES_IDX_MAP,
|
||||
BaseCheckpointSaver,
|
||||
ChannelVersions,
|
||||
Checkpoint,
|
||||
CheckpointMetadata,
|
||||
CheckpointTuple,
|
||||
DeltaChannelHistory,
|
||||
SerializerProtocol,
|
||||
get_checkpoint_id,
|
||||
get_checkpoint_metadata,
|
||||
)
|
||||
from langgraph.checkpoint.serde.jsonplus import JsonPlusSerializer
|
||||
|
||||
from langgraph.checkpoint.sqlite._delta import (
|
||||
DELTA_STAGE1_SQL,
|
||||
build_delta_channels_writes_history,
|
||||
build_delta_stage2_sql,
|
||||
step_walk_with_row,
|
||||
)
|
||||
from langgraph.checkpoint.sqlite.utils import search_where
|
||||
|
||||
_AIO_ERROR_MSG = (
|
||||
"The SqliteSaver does not support async methods. "
|
||||
"Consider using AsyncSqliteSaver instead.\n"
|
||||
"from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver\n"
|
||||
"Note: AsyncSqliteSaver requires the aiosqlite package to use.\n"
|
||||
"Install with:\n`pip install aiosqlite`\n"
|
||||
"See https://langchain-ai.github.io/langgraph/reference/checkpoints/#langgraph.checkpoint.sqlite.aio.AsyncSqliteSaver"
|
||||
"for more information."
|
||||
)
|
||||
|
||||
|
||||
class SqliteSaver(BaseCheckpointSaver[str]):
|
||||
"""A checkpoint saver that stores checkpoints in a SQLite database.
|
||||
|
||||
Note:
|
||||
This class is meant for lightweight, synchronous use cases
|
||||
(demos and small projects) and does not
|
||||
scale to multiple threads.
|
||||
For a similar sqlite saver with `async` support,
|
||||
consider using [AsyncSqliteSaver][langgraph.checkpoint.sqlite.aio.AsyncSqliteSaver].
|
||||
|
||||
Args:
|
||||
conn (sqlite3.Connection): The SQLite database connection.
|
||||
serde (Optional[SerializerProtocol]): The serializer to use for serializing and deserializing checkpoints. Defaults to JsonPlusSerializerCompat.
|
||||
|
||||
Examples:
|
||||
|
||||
>>> import sqlite3
|
||||
>>> from langgraph.checkpoint.sqlite import SqliteSaver
|
||||
>>> from langgraph.graph import StateGraph
|
||||
>>>
|
||||
>>> builder = StateGraph(int)
|
||||
>>> builder.add_node("add_one", lambda x: x + 1)
|
||||
>>> builder.set_entry_point("add_one")
|
||||
>>> builder.set_finish_point("add_one")
|
||||
>>> # Create a new SqliteSaver instance
|
||||
>>> # Note: check_same_thread=False is OK as the implementation uses a lock
|
||||
>>> # to ensure thread safety.
|
||||
>>> conn = sqlite3.connect("checkpoints.sqlite", check_same_thread=False)
|
||||
>>> memory = SqliteSaver(conn)
|
||||
>>> graph = builder.compile(checkpointer=memory)
|
||||
>>> config = {"configurable": {"thread_id": "1"}}
|
||||
>>> graph.get_state(config)
|
||||
>>> result = graph.invoke(3, config)
|
||||
>>> graph.get_state(config)
|
||||
StateSnapshot(values=4, next=(), config={'configurable': {'thread_id': '1', 'checkpoint_ns': '', 'checkpoint_id': '0c62ca34-ac19-445d-bbb0-5b4984975b2a'}}, parent_config=None)
|
||||
""" # noqa
|
||||
|
||||
conn: sqlite3.Connection
|
||||
is_setup: bool
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
conn: sqlite3.Connection,
|
||||
*,
|
||||
serde: SerializerProtocol | None = None,
|
||||
) -> None:
|
||||
super().__init__(serde=serde)
|
||||
self.jsonplus_serde = JsonPlusSerializer()
|
||||
self.conn = conn
|
||||
self.is_setup = False
|
||||
self.lock = threading.Lock()
|
||||
|
||||
@classmethod
|
||||
@contextmanager
|
||||
def from_conn_string(cls, conn_string: str) -> Iterator[SqliteSaver]:
|
||||
"""Create a new SqliteSaver instance from a connection string.
|
||||
|
||||
Args:
|
||||
conn_string: The SQLite connection string.
|
||||
|
||||
Yields:
|
||||
SqliteSaver: A new SqliteSaver instance.
|
||||
|
||||
Examples:
|
||||
|
||||
In memory:
|
||||
|
||||
with SqliteSaver.from_conn_string(":memory:") as memory:
|
||||
...
|
||||
|
||||
To disk:
|
||||
|
||||
with SqliteSaver.from_conn_string("checkpoints.sqlite") as memory:
|
||||
...
|
||||
"""
|
||||
with closing(
|
||||
sqlite3.connect(
|
||||
conn_string,
|
||||
# https://ricardoanderegg.com/posts/python-sqlite-thread-safety/
|
||||
check_same_thread=False,
|
||||
)
|
||||
) as conn:
|
||||
yield cls(conn)
|
||||
|
||||
def setup(self) -> None:
|
||||
"""Set up the checkpoint database.
|
||||
|
||||
This method creates the necessary tables in the SQLite database if they don't
|
||||
already exist. It is called automatically when needed and should not be called
|
||||
directly by the user.
|
||||
"""
|
||||
if self.is_setup:
|
||||
return
|
||||
|
||||
self.conn.executescript(
|
||||
"""
|
||||
PRAGMA journal_mode=WAL;
|
||||
CREATE TABLE IF NOT EXISTS checkpoints (
|
||||
thread_id TEXT NOT NULL,
|
||||
checkpoint_ns TEXT NOT NULL DEFAULT '',
|
||||
checkpoint_id TEXT NOT NULL,
|
||||
parent_checkpoint_id TEXT,
|
||||
type TEXT,
|
||||
checkpoint BLOB,
|
||||
metadata BLOB,
|
||||
PRIMARY KEY (thread_id, checkpoint_ns, checkpoint_id)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS writes (
|
||||
thread_id TEXT NOT NULL,
|
||||
checkpoint_ns TEXT NOT NULL DEFAULT '',
|
||||
checkpoint_id TEXT NOT NULL,
|
||||
task_id TEXT NOT NULL,
|
||||
idx INTEGER NOT NULL,
|
||||
channel TEXT NOT NULL,
|
||||
type TEXT,
|
||||
value BLOB,
|
||||
PRIMARY KEY (thread_id, checkpoint_ns, checkpoint_id, task_id, idx)
|
||||
);
|
||||
"""
|
||||
)
|
||||
|
||||
self.is_setup = True
|
||||
|
||||
@contextmanager
|
||||
def cursor(self, transaction: bool = True) -> Iterator[sqlite3.Cursor]:
|
||||
"""Get a cursor for the SQLite database.
|
||||
|
||||
This method returns a cursor for the SQLite database. It is used internally
|
||||
by the SqliteSaver and should not be called directly by the user.
|
||||
|
||||
Args:
|
||||
transaction (bool): Whether to commit the transaction when the cursor is closed. Defaults to True.
|
||||
|
||||
Yields:
|
||||
sqlite3.Cursor: A cursor for the SQLite database.
|
||||
"""
|
||||
with self.lock:
|
||||
self.setup()
|
||||
cur = self.conn.cursor()
|
||||
try:
|
||||
yield cur
|
||||
finally:
|
||||
if transaction:
|
||||
self.conn.commit()
|
||||
cur.close()
|
||||
|
||||
def get_tuple(self, config: RunnableConfig) -> CheckpointTuple | None:
|
||||
"""Get a checkpoint tuple from the database.
|
||||
|
||||
This method retrieves a checkpoint tuple from the SQLite database based on the
|
||||
provided config. If the config contains a `checkpoint_id` key, the checkpoint with
|
||||
the matching thread ID and checkpoint ID is retrieved. Otherwise, the latest checkpoint
|
||||
for the given thread ID is retrieved.
|
||||
|
||||
Args:
|
||||
config: The config to use for retrieving the checkpoint.
|
||||
|
||||
Returns:
|
||||
The retrieved checkpoint tuple, or None if no matching checkpoint was found.
|
||||
|
||||
Examples:
|
||||
|
||||
Basic:
|
||||
>>> config = {"configurable": {"thread_id": "1"}}
|
||||
>>> checkpoint_tuple = memory.get_tuple(config)
|
||||
>>> print(checkpoint_tuple)
|
||||
CheckpointTuple(...)
|
||||
|
||||
With checkpoint ID:
|
||||
|
||||
>>> config = {
|
||||
... "configurable": {
|
||||
... "thread_id": "1",
|
||||
... "checkpoint_ns": "",
|
||||
... "checkpoint_id": "1ef4f797-8335-6428-8001-8a1503f9b875",
|
||||
... }
|
||||
... }
|
||||
>>> checkpoint_tuple = memory.get_tuple(config)
|
||||
>>> print(checkpoint_tuple)
|
||||
CheckpointTuple(...)
|
||||
""" # noqa
|
||||
checkpoint_ns = config["configurable"].get("checkpoint_ns", "")
|
||||
with self.cursor(transaction=False) as cur:
|
||||
# find the latest checkpoint for the thread_id
|
||||
if checkpoint_id := get_checkpoint_id(config):
|
||||
cur.execute(
|
||||
"SELECT thread_id, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata FROM checkpoints WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ?",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
checkpoint_id,
|
||||
),
|
||||
)
|
||||
else:
|
||||
cur.execute(
|
||||
"SELECT thread_id, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata FROM checkpoints WHERE thread_id = ? AND checkpoint_ns = ? ORDER BY checkpoint_id DESC LIMIT 1",
|
||||
(str(config["configurable"]["thread_id"]), checkpoint_ns),
|
||||
)
|
||||
# if a checkpoint is found, return it
|
||||
if value := cur.fetchone():
|
||||
(
|
||||
thread_id,
|
||||
checkpoint_id,
|
||||
parent_checkpoint_id,
|
||||
type,
|
||||
checkpoint,
|
||||
metadata,
|
||||
) = value
|
||||
if not get_checkpoint_id(config):
|
||||
config = {
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint_id,
|
||||
}
|
||||
}
|
||||
# find any pending writes
|
||||
cur.execute(
|
||||
"SELECT task_id, channel, type, value FROM writes WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ? ORDER BY task_id, idx",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
str(config["configurable"]["checkpoint_id"]),
|
||||
),
|
||||
)
|
||||
# deserialize the checkpoint and metadata
|
||||
return CheckpointTuple(
|
||||
config,
|
||||
self.serde.loads_typed((type, checkpoint)),
|
||||
cast(
|
||||
CheckpointMetadata,
|
||||
json.loads(metadata) if metadata is not None else {},
|
||||
),
|
||||
(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": parent_checkpoint_id,
|
||||
}
|
||||
}
|
||||
if parent_checkpoint_id
|
||||
else None
|
||||
),
|
||||
[
|
||||
(task_id, channel, self.serde.loads_typed((type, value)))
|
||||
for task_id, channel, type, value in cur
|
||||
],
|
||||
)
|
||||
|
||||
def list(
|
||||
self,
|
||||
config: RunnableConfig | None,
|
||||
*,
|
||||
filter: dict[str, Any] | None = None,
|
||||
before: RunnableConfig | None = None,
|
||||
limit: int | None = None,
|
||||
) -> Iterator[CheckpointTuple]:
|
||||
"""List checkpoints from the database.
|
||||
|
||||
This method retrieves a list of checkpoint tuples from the SQLite database based
|
||||
on the provided config. The checkpoints are ordered by checkpoint ID in descending order (newest first).
|
||||
|
||||
Args:
|
||||
config: The config to use for listing the checkpoints.
|
||||
filter: Additional filtering criteria for metadata.
|
||||
before: If provided, only checkpoints before the specified checkpoint ID are returned.
|
||||
limit: The maximum number of checkpoints to return.
|
||||
|
||||
Yields:
|
||||
An iterator of checkpoint tuples.
|
||||
|
||||
Examples:
|
||||
>>> from langgraph.checkpoint.sqlite import SqliteSaver
|
||||
>>> with SqliteSaver.from_conn_string(":memory:") as memory:
|
||||
... # Run a graph, then list the checkpoints
|
||||
>>> config = {"configurable": {"thread_id": "1"}}
|
||||
>>> checkpoints = list(memory.list(config, limit=2))
|
||||
>>> print(checkpoints)
|
||||
[CheckpointTuple(...), CheckpointTuple(...)]
|
||||
|
||||
>>> config = {"configurable": {"thread_id": "1"}}
|
||||
>>> before = {"configurable": {"checkpoint_id": "1ef4f797-8335-6428-8001-8a1503f9b875"}}
|
||||
>>> with SqliteSaver.from_conn_string(":memory:") as memory:
|
||||
... # Run a graph, then list the checkpoints
|
||||
>>> checkpoints = list(memory.list(config, before=before))
|
||||
>>> print(checkpoints)
|
||||
[CheckpointTuple(...), ...]
|
||||
"""
|
||||
where, param_values = search_where(config, filter, before)
|
||||
query = f"""SELECT thread_id, checkpoint_ns, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata
|
||||
FROM checkpoints
|
||||
{where}
|
||||
ORDER BY checkpoint_id DESC"""
|
||||
if limit is not None:
|
||||
query += " LIMIT ?"
|
||||
param_values = (*param_values, limit)
|
||||
with self.cursor(transaction=False) as cur, closing(self.conn.cursor()) as wcur:
|
||||
cur.execute(query, param_values)
|
||||
for (
|
||||
thread_id,
|
||||
checkpoint_ns,
|
||||
checkpoint_id,
|
||||
parent_checkpoint_id,
|
||||
type,
|
||||
checkpoint,
|
||||
metadata,
|
||||
) in cur:
|
||||
wcur.execute(
|
||||
"SELECT task_id, channel, type, value FROM writes WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ? ORDER BY task_id, idx",
|
||||
(thread_id, checkpoint_ns, checkpoint_id),
|
||||
)
|
||||
yield CheckpointTuple(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint_id,
|
||||
}
|
||||
},
|
||||
self.serde.loads_typed((type, checkpoint)),
|
||||
cast(
|
||||
CheckpointMetadata,
|
||||
json.loads(metadata) if metadata is not None else {},
|
||||
),
|
||||
(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": parent_checkpoint_id,
|
||||
}
|
||||
}
|
||||
if parent_checkpoint_id
|
||||
else None
|
||||
),
|
||||
[
|
||||
(task_id, channel, self.serde.loads_typed((type, value)))
|
||||
for task_id, channel, type, value in wcur
|
||||
],
|
||||
)
|
||||
|
||||
def put(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
checkpoint: Checkpoint,
|
||||
metadata: CheckpointMetadata,
|
||||
new_versions: ChannelVersions,
|
||||
) -> RunnableConfig:
|
||||
"""Save a checkpoint to the database.
|
||||
|
||||
This method saves a checkpoint to the SQLite database. The checkpoint is associated
|
||||
with the provided config and its parent config (if any).
|
||||
|
||||
Args:
|
||||
config: The config to associate with the checkpoint.
|
||||
checkpoint: The checkpoint to save.
|
||||
metadata: Additional metadata to save with the checkpoint.
|
||||
new_versions: New channel versions as of this write.
|
||||
|
||||
Returns:
|
||||
RunnableConfig: Updated configuration after storing the checkpoint.
|
||||
|
||||
Examples:
|
||||
|
||||
>>> from langgraph.checkpoint.sqlite import SqliteSaver
|
||||
>>> with SqliteSaver.from_conn_string(":memory:") as memory:
|
||||
>>> config = {"configurable": {"thread_id": "1", "checkpoint_ns": ""}}
|
||||
>>> checkpoint = {"ts": "2024-05-04T06:32:42.235444+00:00", "id": "1ef4f797-8335-6428-8001-8a1503f9b875", "channel_values": {"key": "value"}}
|
||||
>>> saved_config = memory.put(config, checkpoint, {"source": "input", "step": 1, "writes": {"key": "value"}}, {})
|
||||
>>> print(saved_config)
|
||||
{'configurable': {'thread_id': '1', 'checkpoint_ns': '', 'checkpoint_id': '1ef4f797-8335-6428-8001-8a1503f9b875'}}
|
||||
"""
|
||||
thread_id = config["configurable"]["thread_id"]
|
||||
checkpoint_ns = config["configurable"]["checkpoint_ns"]
|
||||
type_, serialized_checkpoint = self.serde.dumps_typed(checkpoint)
|
||||
serialized_metadata = json.dumps(
|
||||
get_checkpoint_metadata(config, metadata), ensure_ascii=False
|
||||
).encode("utf-8", "ignore")
|
||||
with self.cursor() as cur:
|
||||
cur.execute(
|
||||
"INSERT OR REPLACE INTO checkpoints (thread_id, checkpoint_ns, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata) VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
checkpoint["id"],
|
||||
config["configurable"].get("checkpoint_id"),
|
||||
type_,
|
||||
serialized_checkpoint,
|
||||
serialized_metadata,
|
||||
),
|
||||
)
|
||||
return {
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint["id"],
|
||||
}
|
||||
}
|
||||
|
||||
def put_writes(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
writes: Sequence[tuple[str, Any]],
|
||||
task_id: str,
|
||||
task_path: str = "",
|
||||
) -> None:
|
||||
"""Store intermediate writes linked to a checkpoint.
|
||||
|
||||
This method saves intermediate writes associated with a checkpoint to the SQLite database.
|
||||
|
||||
Args:
|
||||
config: Configuration of the related checkpoint.
|
||||
writes: List of writes to store, each as (channel, value) pair.
|
||||
task_id: Identifier for the task creating the writes.
|
||||
task_path: Path of the task creating the writes.
|
||||
"""
|
||||
query = (
|
||||
"INSERT OR REPLACE INTO writes (thread_id, checkpoint_ns, checkpoint_id, task_id, idx, channel, type, value) VALUES (?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
if all(w[0] in WRITES_IDX_MAP for w in writes)
|
||||
else "INSERT OR IGNORE INTO writes (thread_id, checkpoint_ns, checkpoint_id, task_id, idx, channel, type, value) VALUES (?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
)
|
||||
with self.cursor() as cur:
|
||||
cur.executemany(
|
||||
query,
|
||||
[
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
str(config["configurable"]["checkpoint_ns"]),
|
||||
str(config["configurable"]["checkpoint_id"]),
|
||||
task_id,
|
||||
WRITES_IDX_MAP.get(channel, idx),
|
||||
channel,
|
||||
*self.serde.dumps_typed(value),
|
||||
)
|
||||
for idx, (channel, value) in enumerate(writes)
|
||||
],
|
||||
)
|
||||
|
||||
def delete_thread(self, thread_id: str) -> None:
|
||||
"""Delete all checkpoints and writes associated with a thread ID.
|
||||
|
||||
Args:
|
||||
thread_id: The thread ID to delete.
|
||||
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
with self.cursor() as cur:
|
||||
cur.execute(
|
||||
"DELETE FROM checkpoints WHERE thread_id = ?",
|
||||
(str(thread_id),),
|
||||
)
|
||||
cur.execute(
|
||||
"DELETE FROM writes WHERE thread_id = ?",
|
||||
(str(thread_id),),
|
||||
)
|
||||
|
||||
def get_delta_channel_history(
|
||||
self, *, config: RunnableConfig, channels: Sequence[str]
|
||||
) -> Mapping[str, DeltaChannelHistory]:
|
||||
"""Fast-path override of `BaseCheckpointSaver.get_delta_channel_history`.
|
||||
|
||||
Two-stage query:
|
||||
|
||||
* Stage 1 (paged): newest-first slice of `checkpoints` returning
|
||||
`(checkpoint_id, parent_checkpoint_id, type, checkpoint)` per
|
||||
ancestor. Sqlite has no JSONB, so we ship the full serialized
|
||||
checkpoint blob and inspect `channel_values` in Python. Pages
|
||||
newest-first by `checkpoint_id` with a `< cursor` predicate;
|
||||
page size is `DELTA_PAGE_SIZE`. Stops paging when every channel
|
||||
has found its seed or the chain is exhausted.
|
||||
|
||||
* Stage 2 (per-channel UNION ALL): one branch per channel reading
|
||||
`writes` filtered to that channel's specific `chain_cids`. No
|
||||
separate seed-blob fetch — sqlite stores `channel_values` inline
|
||||
in the checkpoint blob, so seeds come back from stage 1.
|
||||
"""
|
||||
if not channels:
|
||||
return {}
|
||||
channels = list(channels)
|
||||
thread_id = str(config["configurable"]["thread_id"])
|
||||
checkpoint_ns = config["configurable"].get("checkpoint_ns", "")
|
||||
checkpoint_id = get_checkpoint_id(config)
|
||||
if checkpoint_id is None:
|
||||
target = self.get_tuple(config)
|
||||
if target is None:
|
||||
return {ch: {"writes": []} for ch in channels}
|
||||
checkpoint_id = target.config["configurable"]["checkpoint_id"]
|
||||
|
||||
chain_by_ch: dict[str, list[str]] = {ch: [] for ch in channels}
|
||||
seed_val_by_ch: dict[str, Any] = {}
|
||||
walk_state: dict[str, Any] = {}
|
||||
seeded: set[str] = set()
|
||||
|
||||
with self.cursor(transaction=False) as cur:
|
||||
cur.execute(DELTA_STAGE1_SQL, (thread_id, checkpoint_ns, checkpoint_id))
|
||||
for row in cur:
|
||||
cid, parent_cid, type_tag, blob = row
|
||||
if step_walk_with_row(
|
||||
cid=cid,
|
||||
parent_cid=parent_cid,
|
||||
type_tag=type_tag,
|
||||
blob=blob,
|
||||
target_id=checkpoint_id,
|
||||
serde=self.serde,
|
||||
chain_by_ch=chain_by_ch,
|
||||
seed_val_by_ch=seed_val_by_ch,
|
||||
walk_state=walk_state,
|
||||
seeded=seeded,
|
||||
channels=channels,
|
||||
):
|
||||
break
|
||||
|
||||
channels_with_chain = [ch for ch in channels if chain_by_ch[ch]]
|
||||
stage2_sql = build_delta_stage2_sql(
|
||||
chain_lens=[len(chain_by_ch[ch]) for ch in channels_with_chain],
|
||||
)
|
||||
if stage2_sql:
|
||||
stage2_params: list[Any] = []
|
||||
for ch in channels_with_chain:
|
||||
stage2_params.extend(
|
||||
[thread_id, checkpoint_ns, ch, *chain_by_ch[ch]]
|
||||
)
|
||||
cur.execute(stage2_sql, stage2_params)
|
||||
stage2_rows = cast(
|
||||
"list[tuple[str, str, str, int, str, bytes]]", cur.fetchall()
|
||||
)
|
||||
else:
|
||||
stage2_rows = []
|
||||
|
||||
return build_delta_channels_writes_history(
|
||||
channels=channels,
|
||||
chain_by_ch=chain_by_ch,
|
||||
seed_val_by_ch=seed_val_by_ch,
|
||||
seeded=seeded,
|
||||
stage2_rows=stage2_rows,
|
||||
serde=self.serde,
|
||||
)
|
||||
|
||||
async def aget_tuple(self, config: RunnableConfig) -> CheckpointTuple | None:
|
||||
"""Get a checkpoint tuple from the database asynchronously.
|
||||
|
||||
Note:
|
||||
This async method is not supported by the SqliteSaver class.
|
||||
Use get_tuple() instead, or consider using [AsyncSqliteSaver][langgraph.checkpoint.sqlite.aio.AsyncSqliteSaver].
|
||||
"""
|
||||
raise NotImplementedError(_AIO_ERROR_MSG)
|
||||
|
||||
async def alist(
|
||||
self,
|
||||
config: RunnableConfig | None,
|
||||
*,
|
||||
filter: dict[str, Any] | None = None,
|
||||
before: RunnableConfig | None = None,
|
||||
limit: int | None = None,
|
||||
) -> AsyncIterator[CheckpointTuple]:
|
||||
"""List checkpoints from the database asynchronously.
|
||||
|
||||
Note:
|
||||
This async method is not supported by the SqliteSaver class.
|
||||
Use list() instead, or consider using [AsyncSqliteSaver][langgraph.checkpoint.sqlite.aio.AsyncSqliteSaver].
|
||||
"""
|
||||
raise NotImplementedError(_AIO_ERROR_MSG)
|
||||
yield
|
||||
|
||||
async def aput(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
checkpoint: Checkpoint,
|
||||
metadata: CheckpointMetadata,
|
||||
new_versions: ChannelVersions,
|
||||
) -> RunnableConfig:
|
||||
"""Save a checkpoint to the database asynchronously.
|
||||
|
||||
Note:
|
||||
This async method is not supported by the SqliteSaver class.
|
||||
Use put() instead, or consider using [AsyncSqliteSaver][langgraph.checkpoint.sqlite.aio.AsyncSqliteSaver].
|
||||
"""
|
||||
raise NotImplementedError(_AIO_ERROR_MSG)
|
||||
|
||||
def get_next_version(self, current: str | None, channel: None) -> str:
|
||||
"""Generate the next version ID for a channel.
|
||||
|
||||
This method creates a new version identifier for a channel based on its current version.
|
||||
|
||||
Args:
|
||||
current (Optional[str]): The current version identifier of the channel.
|
||||
|
||||
Returns:
|
||||
str: The next version identifier, which is guaranteed to be monotonically increasing.
|
||||
"""
|
||||
if current is None:
|
||||
current_v = 0
|
||||
elif isinstance(current, int):
|
||||
current_v = current
|
||||
else:
|
||||
current_v = int(current.split(".")[0])
|
||||
next_v = current_v + 1
|
||||
next_h = random.random()
|
||||
return f"{next_v:032}.{next_h:016}"
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Shared helpers for `get_delta_channel_history` on sqlite savers.
|
||||
|
||||
Mirrors the two-stage shape of `BasePostgresSaver` (ancestor walk +
|
||||
per-channel UNION ALL writes fetch), but adapted for sqlite's
|
||||
constraints. The structural differences:
|
||||
|
||||
* No JSONB — to inspect `channel_values` for a checkpoint we must
|
||||
deserialize the full blob. Stage 1 streams the cursor row-by-row and
|
||||
deserializes only the rows the merged walk visits, freeing each blob
|
||||
before advancing.
|
||||
* No separate blob table — `channel_values` lives inline in the
|
||||
checkpoint, so seeds come back from stage 1 with no second fetch.
|
||||
* Single merged walk (not K independent walks): each visited cid is
|
||||
deserialized exactly once, regardless of how many channels are still
|
||||
seeking their seed.
|
||||
|
||||
The streaming design keeps peak in-flight memory at roughly one
|
||||
deserialized checkpoint at a time, instead of holding the entire
|
||||
ancestor chain's worth of raw blobs as a `fetchall()`-materialized list.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from langgraph.checkpoint.base import DeltaChannelHistory, PendingWrite
|
||||
|
||||
# Stage 1 streams ancestors of `target_cid` newest-first. The `<=`
|
||||
# predicate keeps target itself in the stream so we can read its
|
||||
# `parent_checkpoint_id` from the first row without a separate lookup;
|
||||
# the caller skips target's own writes/seed (matches the
|
||||
# `BaseCheckpointSaver` contract).
|
||||
DELTA_STAGE1_SQL = (
|
||||
"SELECT checkpoint_id, parent_checkpoint_id, type, checkpoint "
|
||||
"FROM checkpoints "
|
||||
"WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id <= ? "
|
||||
"ORDER BY checkpoint_id DESC"
|
||||
)
|
||||
|
||||
|
||||
def build_delta_stage2_sql(*, chain_lens: Sequence[int]) -> str:
|
||||
"""Stage-2 per-channel UNION ALL fetching writes from `writes`.
|
||||
|
||||
One branch per channel with a non-empty chain. Each branch inlines its
|
||||
own `IN (?, ?, ...)` placeholder list because sqlite has no array-bind
|
||||
equivalent of postgres's `= ANY(%s)`. Caller passes parameters in
|
||||
matching order: `[thread_id, checkpoint_ns, channel, *chain_cids]` per
|
||||
branch.
|
||||
|
||||
Returns an empty string when no channel has a chain (caller skips
|
||||
executing in that case). Per-channel UNION ALL avoids the over-fetch
|
||||
of a single `channel = ANY(channels)` filter when channels have
|
||||
different chain depths — same rationale as postgres.
|
||||
"""
|
||||
branches: list[str] = []
|
||||
for n in chain_lens:
|
||||
cid_placeholders = ",".join("?" * n)
|
||||
branches.append(
|
||||
"SELECT checkpoint_id, channel, task_id, idx, type, value "
|
||||
"FROM writes "
|
||||
"WHERE thread_id = ? AND checkpoint_ns = ? AND channel = ? "
|
||||
f"AND checkpoint_id IN ({cid_placeholders})"
|
||||
)
|
||||
return " UNION ALL ".join(branches)
|
||||
|
||||
|
||||
def step_walk_with_row(
|
||||
*,
|
||||
cid: str,
|
||||
parent_cid: str | None,
|
||||
type_tag: str,
|
||||
blob: bytes,
|
||||
target_id: str,
|
||||
serde: Any,
|
||||
chain_by_ch: dict[str, list[str]],
|
||||
seed_val_by_ch: dict[str, Any],
|
||||
walk_state: dict[str, Any],
|
||||
seeded: set[str],
|
||||
channels: Sequence[str],
|
||||
) -> bool:
|
||||
"""Process one streamed stage-1 row in the merged ancestor walk.
|
||||
|
||||
The cursor returns (cid, parent_cid, type, blob) rows in
|
||||
`checkpoint_id` DESC order starting at target. The first row is
|
||||
target itself; we read its parent_cid to seed the walk and otherwise
|
||||
skip it (target's own writes/seed are not part of the contract).
|
||||
|
||||
For each subsequent row, if `cid` matches the walk's current
|
||||
position, we deserialize the blob, append the cid to every
|
||||
not-yet-seeded channel's chain, and check `channel_values` for
|
||||
seeds. The deserialized checkpoint is dropped before advancing — no
|
||||
cross-row cache, so peak in-flight is one deserialized checkpoint.
|
||||
|
||||
Off-path rows (different branch on the same thread) advance the
|
||||
cursor without doing any work.
|
||||
|
||||
Returns True when every requested channel is seeded — the caller
|
||||
can stop iterating and close the cursor.
|
||||
"""
|
||||
if "started" not in walk_state:
|
||||
if cid == target_id:
|
||||
walk_state["started"] = True
|
||||
walk_state["cur_cid"] = parent_cid
|
||||
walk_state["active"] = {ch for ch in channels if ch not in seeded}
|
||||
# Not target yet (or target not present): keep streaming.
|
||||
return False
|
||||
active: set[str] = walk_state["active"]
|
||||
if not active:
|
||||
return True
|
||||
if cid != walk_state["cur_cid"]:
|
||||
# Off-path row from a sibling branch — skip without deserializing.
|
||||
return False
|
||||
for ch in active:
|
||||
chain_by_ch[ch].append(cid)
|
||||
ckpt = serde.loads_typed((type_tag, blob))
|
||||
channel_values: Mapping[str, Any] = ckpt.get("channel_values") or {}
|
||||
for ch in [ch for ch in active if ch in channel_values]:
|
||||
seed_val_by_ch[ch] = channel_values[ch]
|
||||
seeded.add(ch)
|
||||
active.discard(ch)
|
||||
del ckpt, channel_values
|
||||
walk_state["cur_cid"] = parent_cid
|
||||
return not active
|
||||
|
||||
|
||||
def build_delta_channels_writes_history(
|
||||
*,
|
||||
channels: Sequence[str],
|
||||
chain_by_ch: Mapping[str, list[str]],
|
||||
seed_val_by_ch: Mapping[str, Any],
|
||||
seeded: set[str],
|
||||
stage2_rows: Sequence[tuple[str, str, str, int, str, bytes]],
|
||||
serde: Any,
|
||||
) -> dict[str, DeltaChannelHistory]:
|
||||
"""Demux stage-2 rows per channel; produce per-channel histories.
|
||||
|
||||
Stage-2 rows are `(checkpoint_id, channel, task_id, idx, type, value)`.
|
||||
Final write order is oldest→newest globally and `(task_id, idx)` within
|
||||
a checkpoint, matching the contract on `DeltaChannelHistory.writes`.
|
||||
|
||||
`seed` is omitted when the walk reached a true root with no snapshot
|
||||
found (channel never entered `seeded`); consumers treat absence as
|
||||
"start empty".
|
||||
"""
|
||||
writes_by_ch_by_cid: dict[str, dict[str, list[tuple[str, bytes, str, int]]]] = {
|
||||
ch: {} for ch in channels
|
||||
}
|
||||
for cid, ch, task_id, idx, type_tag, value_blob in stage2_rows:
|
||||
writes_by_ch_by_cid.setdefault(ch, {}).setdefault(cid, []).append(
|
||||
(type_tag, value_blob, task_id, idx)
|
||||
)
|
||||
for cid_map in writes_by_ch_by_cid.values():
|
||||
for ws in cid_map.values():
|
||||
ws.sort(key=lambda w: (w[2], w[3]))
|
||||
|
||||
result: dict[str, DeltaChannelHistory] = {}
|
||||
for ch in channels:
|
||||
chain_cids = chain_by_ch.get(ch, [])
|
||||
cid_writes = writes_by_ch_by_cid.get(ch, {})
|
||||
collected: list[PendingWrite] = []
|
||||
# Chain is newest-first; iterate oldest-first for the public order.
|
||||
for cid in reversed(chain_cids):
|
||||
for type_tag, value_blob, task_id, _idx in cid_writes.get(cid, []):
|
||||
collected.append(
|
||||
(task_id, ch, serde.loads_typed((type_tag, value_blob)))
|
||||
)
|
||||
entry: DeltaChannelHistory = {"writes": collected}
|
||||
if ch in seeded:
|
||||
entry["seed"] = seed_val_by_ch[ch]
|
||||
result[ch] = entry
|
||||
return result
|
||||
@@ -0,0 +1,738 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import random
|
||||
import threading
|
||||
from collections.abc import AsyncIterator, Callable, Iterator, Mapping, Sequence
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import Any, TypeVar, cast
|
||||
|
||||
import aiosqlite
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from langgraph.checkpoint.base import (
|
||||
WRITES_IDX_MAP,
|
||||
BaseCheckpointSaver,
|
||||
ChannelVersions,
|
||||
Checkpoint,
|
||||
CheckpointMetadata,
|
||||
CheckpointTuple,
|
||||
DeltaChannelHistory,
|
||||
SerializerProtocol,
|
||||
get_checkpoint_id,
|
||||
get_checkpoint_metadata,
|
||||
)
|
||||
from langgraph.checkpoint.serde.jsonplus import JsonPlusSerializer
|
||||
|
||||
from langgraph.checkpoint.sqlite._delta import (
|
||||
DELTA_STAGE1_SQL,
|
||||
build_delta_channels_writes_history,
|
||||
build_delta_stage2_sql,
|
||||
step_walk_with_row,
|
||||
)
|
||||
from langgraph.checkpoint.sqlite.utils import search_where
|
||||
|
||||
T = TypeVar("T", bound=Callable)
|
||||
|
||||
|
||||
class AsyncSqliteSaver(BaseCheckpointSaver[str]):
|
||||
"""An asynchronous checkpoint saver that stores checkpoints in a SQLite database.
|
||||
|
||||
This class provides an asynchronous interface for saving and retrieving checkpoints
|
||||
using a SQLite database. It's designed for use in asynchronous environments and
|
||||
offers better performance for I/O-bound operations compared to synchronous alternatives.
|
||||
|
||||
Attributes:
|
||||
conn (aiosqlite.Connection): The asynchronous SQLite database connection.
|
||||
serde (SerializerProtocol): The serializer used for encoding/decoding checkpoints.
|
||||
|
||||
Tip:
|
||||
Requires the [aiosqlite](https://pypi.org/project/aiosqlite/) package.
|
||||
Install it with `pip install aiosqlite`.
|
||||
|
||||
Warning:
|
||||
While this class supports asynchronous checkpointing, it is not recommended
|
||||
for production workloads due to limitations in SQLite's write performance.
|
||||
For production use, consider a more robust database like PostgreSQL.
|
||||
|
||||
Tip:
|
||||
Remember to **close the database connection** after executing your code,
|
||||
otherwise, you may see the graph "hang" after execution (since the program
|
||||
will not exit until the connection is closed).
|
||||
|
||||
The easiest way is to use the `async with` statement as shown in the examples.
|
||||
|
||||
```python
|
||||
async with AsyncSqliteSaver.from_conn_string("checkpoints.sqlite") as saver:
|
||||
# Your code here
|
||||
graph = builder.compile(checkpointer=saver)
|
||||
config = {"configurable": {"thread_id": "thread-1"}}
|
||||
async for event in graph.astream_events(..., config, version="v1"):
|
||||
print(event)
|
||||
```
|
||||
|
||||
Examples:
|
||||
Usage within StateGraph:
|
||||
|
||||
```pycon
|
||||
>>> import asyncio
|
||||
>>>
|
||||
>>> from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
>>> from langgraph.graph import StateGraph
|
||||
>>>
|
||||
>>> async def main():
|
||||
>>> builder = StateGraph(int)
|
||||
>>> builder.add_node("add_one", lambda x: x + 1)
|
||||
>>> builder.set_entry_point("add_one")
|
||||
>>> builder.set_finish_point("add_one")
|
||||
>>> async with AsyncSqliteSaver.from_conn_string("checkpoints.db") as memory:
|
||||
>>> graph = builder.compile(checkpointer=memory)
|
||||
>>> coro = graph.ainvoke(1, {"configurable": {"thread_id": "thread-1"}})
|
||||
>>> print(await asyncio.gather(coro))
|
||||
>>>
|
||||
>>> asyncio.run(main())
|
||||
Output: [2]
|
||||
```
|
||||
Raw usage:
|
||||
|
||||
```pycon
|
||||
>>> import asyncio
|
||||
>>> import aiosqlite
|
||||
>>> from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
>>>
|
||||
>>> async def main():
|
||||
>>> async with aiosqlite.connect("checkpoints.db") as conn:
|
||||
... saver = AsyncSqliteSaver(conn)
|
||||
... config = {"configurable": {"thread_id": "1", "checkpoint_ns": ""}}
|
||||
... checkpoint = {"ts": "2023-05-03T10:00:00Z", "data": {"key": "value"}, "id": "0c62ca34-ac19-445d-bbb0-5b4984975b2a"}
|
||||
... saved_config = await saver.aput(config, checkpoint, {}, {})
|
||||
... print(saved_config)
|
||||
>>> asyncio.run(main())
|
||||
{'configurable': {'thread_id': '1', 'checkpoint_ns': '', 'checkpoint_id': '0c62ca34-ac19-445d-bbb0-5b4984975b2a'}}
|
||||
```
|
||||
"""
|
||||
|
||||
lock: asyncio.Lock
|
||||
is_setup: bool
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
conn: aiosqlite.Connection,
|
||||
*,
|
||||
serde: SerializerProtocol | None = None,
|
||||
):
|
||||
super().__init__(serde=serde)
|
||||
self.jsonplus_serde = JsonPlusSerializer()
|
||||
self.conn = conn
|
||||
self.lock = asyncio.Lock()
|
||||
self.loop = asyncio.get_running_loop()
|
||||
self.is_setup = False
|
||||
|
||||
@classmethod
|
||||
@asynccontextmanager
|
||||
async def from_conn_string(
|
||||
cls, conn_string: str
|
||||
) -> AsyncIterator[AsyncSqliteSaver]:
|
||||
"""Create a new AsyncSqliteSaver instance from a connection string.
|
||||
|
||||
Args:
|
||||
conn_string: The SQLite connection string.
|
||||
|
||||
Yields:
|
||||
AsyncSqliteSaver: A new AsyncSqliteSaver instance.
|
||||
"""
|
||||
async with aiosqlite.connect(conn_string) as conn:
|
||||
yield cls(conn)
|
||||
|
||||
def get_tuple(self, config: RunnableConfig) -> CheckpointTuple | None:
|
||||
"""Get a checkpoint tuple from the database.
|
||||
|
||||
This method retrieves a checkpoint tuple from the SQLite database based on the
|
||||
provided config. If the config contains a `checkpoint_id` key, the checkpoint with
|
||||
the matching thread ID and checkpoint ID is retrieved. Otherwise, the latest checkpoint
|
||||
for the given thread ID is retrieved.
|
||||
|
||||
Args:
|
||||
config: The config to use for retrieving the checkpoint.
|
||||
|
||||
Returns:
|
||||
The retrieved checkpoint tuple, or None if no matching checkpoint was found.
|
||||
"""
|
||||
try:
|
||||
# check if we are in the main thread, only bg threads can block
|
||||
# we don't check in other methods to avoid the overhead
|
||||
if asyncio.get_running_loop() is self.loop:
|
||||
raise asyncio.InvalidStateError(
|
||||
"Synchronous calls to AsyncSqliteSaver are only allowed from a "
|
||||
"different thread. From the main thread, use the async interface. "
|
||||
"For example, use `await checkpointer.aget_tuple(...)` or `await "
|
||||
"graph.ainvoke(...)`."
|
||||
)
|
||||
except RuntimeError:
|
||||
pass
|
||||
return asyncio.run_coroutine_threadsafe(
|
||||
self.aget_tuple(config), self.loop
|
||||
).result()
|
||||
|
||||
def list(
|
||||
self,
|
||||
config: RunnableConfig | None,
|
||||
*,
|
||||
filter: dict[str, Any] | None = None,
|
||||
before: RunnableConfig | None = None,
|
||||
limit: int | None = None,
|
||||
) -> Iterator[CheckpointTuple]:
|
||||
"""List checkpoints from the database asynchronously.
|
||||
|
||||
This method retrieves a list of checkpoint tuples from the SQLite database based
|
||||
on the provided config. The checkpoints are ordered by checkpoint ID in descending order (newest first).
|
||||
|
||||
Args:
|
||||
config: Base configuration for filtering checkpoints.
|
||||
filter: Additional filtering criteria for metadata.
|
||||
before: If provided, only checkpoints before the specified checkpoint ID are returned.
|
||||
limit: Maximum number of checkpoints to return.
|
||||
|
||||
Yields:
|
||||
An iterator of matching checkpoint tuples.
|
||||
"""
|
||||
try:
|
||||
# check if we are in the main thread, only bg threads can block
|
||||
# we don't check in other methods to avoid the overhead
|
||||
if asyncio.get_running_loop() is self.loop:
|
||||
raise asyncio.InvalidStateError(
|
||||
"Synchronous calls to AsyncSqliteSaver are only allowed from a "
|
||||
"different thread. From the main thread, use the async interface. "
|
||||
"For example, use `checkpointer.alist(...)` or `await "
|
||||
"graph.ainvoke(...)`."
|
||||
)
|
||||
except RuntimeError:
|
||||
pass
|
||||
aiter_ = self.alist(config, filter=filter, before=before, limit=limit)
|
||||
while True:
|
||||
try:
|
||||
yield asyncio.run_coroutine_threadsafe(
|
||||
anext(aiter_), # type: ignore[arg-type] # noqa: F821
|
||||
self.loop,
|
||||
).result()
|
||||
except StopAsyncIteration:
|
||||
break
|
||||
|
||||
def put(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
checkpoint: Checkpoint,
|
||||
metadata: CheckpointMetadata,
|
||||
new_versions: ChannelVersions,
|
||||
) -> RunnableConfig:
|
||||
"""Save a checkpoint to the database.
|
||||
|
||||
This method saves a checkpoint to the SQLite database. The checkpoint is associated
|
||||
with the provided config and its parent config (if any).
|
||||
|
||||
Args:
|
||||
config: The config to associate with the checkpoint.
|
||||
checkpoint: The checkpoint to save.
|
||||
metadata: Additional metadata to save with the checkpoint.
|
||||
new_versions: New channel versions as of this write.
|
||||
|
||||
Returns:
|
||||
RunnableConfig: Updated configuration after storing the checkpoint.
|
||||
"""
|
||||
return asyncio.run_coroutine_threadsafe(
|
||||
self.aput(config, checkpoint, metadata, new_versions), self.loop
|
||||
).result()
|
||||
|
||||
def put_writes(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
writes: Sequence[tuple[str, Any]],
|
||||
task_id: str,
|
||||
task_path: str = "",
|
||||
) -> None:
|
||||
return asyncio.run_coroutine_threadsafe(
|
||||
self.aput_writes(config, writes, task_id, task_path), self.loop
|
||||
).result()
|
||||
|
||||
def delete_thread(self, thread_id: str) -> None:
|
||||
"""Delete all checkpoints and writes associated with a thread ID.
|
||||
|
||||
Args:
|
||||
thread_id: The thread ID to delete.
|
||||
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
try:
|
||||
# check if we are in the main thread, only bg threads can block
|
||||
# we don't check in other methods to avoid the overhead
|
||||
if asyncio.get_running_loop() is self.loop:
|
||||
raise asyncio.InvalidStateError(
|
||||
"Synchronous calls to AsyncSqliteSaver are only allowed from a "
|
||||
"different thread. From the main thread, use the async interface. "
|
||||
"For example, use `checkpointer.alist(...)` or `await "
|
||||
"graph.ainvoke(...)`."
|
||||
)
|
||||
except RuntimeError:
|
||||
pass
|
||||
return asyncio.run_coroutine_threadsafe(
|
||||
self.adelete_thread(thread_id), self.loop
|
||||
).result()
|
||||
|
||||
def get_delta_channel_history(
|
||||
self, *, config: RunnableConfig, channels: Sequence[str]
|
||||
) -> Mapping[str, DeltaChannelHistory]:
|
||||
"""Sync bridge to `aget_delta_channel_history`.
|
||||
|
||||
Mirrors the same cross-thread guard as `get_tuple` /
|
||||
`delete_thread` — calling from the loop thread raises rather than
|
||||
deadlocking.
|
||||
"""
|
||||
try:
|
||||
if asyncio.get_running_loop() is self.loop:
|
||||
raise asyncio.InvalidStateError(
|
||||
"Synchronous calls to AsyncSqliteSaver are only allowed from a "
|
||||
"different thread. From the main thread, use the async interface. "
|
||||
"For example, use `await checkpointer.aget_delta_channel_history(...)`."
|
||||
)
|
||||
except RuntimeError:
|
||||
pass
|
||||
return asyncio.run_coroutine_threadsafe(
|
||||
self.aget_delta_channel_history(config=config, channels=channels),
|
||||
self.loop,
|
||||
).result()
|
||||
|
||||
async def setup(self) -> None:
|
||||
"""Set up the checkpoint database asynchronously.
|
||||
|
||||
This method creates the necessary tables in the SQLite database if they don't
|
||||
already exist. It is called automatically when needed and should not be called
|
||||
directly by the user.
|
||||
"""
|
||||
async with self.lock:
|
||||
if self.is_setup:
|
||||
return
|
||||
await _ensure_connected(self.conn)
|
||||
async with self.conn.executescript(
|
||||
"""
|
||||
PRAGMA journal_mode=WAL;
|
||||
CREATE TABLE IF NOT EXISTS checkpoints (
|
||||
thread_id TEXT NOT NULL,
|
||||
checkpoint_ns TEXT NOT NULL DEFAULT '',
|
||||
checkpoint_id TEXT NOT NULL,
|
||||
parent_checkpoint_id TEXT,
|
||||
type TEXT,
|
||||
checkpoint BLOB,
|
||||
metadata BLOB,
|
||||
PRIMARY KEY (thread_id, checkpoint_ns, checkpoint_id)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS writes (
|
||||
thread_id TEXT NOT NULL,
|
||||
checkpoint_ns TEXT NOT NULL DEFAULT '',
|
||||
checkpoint_id TEXT NOT NULL,
|
||||
task_id TEXT NOT NULL,
|
||||
idx INTEGER NOT NULL,
|
||||
channel TEXT NOT NULL,
|
||||
type TEXT,
|
||||
value BLOB,
|
||||
PRIMARY KEY (thread_id, checkpoint_ns, checkpoint_id, task_id, idx)
|
||||
);
|
||||
"""
|
||||
):
|
||||
await self.conn.commit()
|
||||
|
||||
self.is_setup = True
|
||||
|
||||
async def aget_tuple(self, config: RunnableConfig) -> CheckpointTuple | None:
|
||||
"""Get a checkpoint tuple from the database asynchronously.
|
||||
|
||||
This method retrieves a checkpoint tuple from the SQLite database based on the
|
||||
provided config. If the config contains a `checkpoint_id` key, the checkpoint with
|
||||
the matching thread ID and checkpoint ID is retrieved. Otherwise, the latest checkpoint
|
||||
for the given thread ID is retrieved.
|
||||
|
||||
Args:
|
||||
config: The config to use for retrieving the checkpoint.
|
||||
|
||||
Returns:
|
||||
The retrieved checkpoint tuple, or None if no matching checkpoint was found.
|
||||
"""
|
||||
await self.setup()
|
||||
checkpoint_ns = config["configurable"].get("checkpoint_ns", "")
|
||||
async with self.lock, self.conn.cursor() as cur:
|
||||
# find the latest checkpoint for the thread_id
|
||||
if checkpoint_id := get_checkpoint_id(config):
|
||||
await cur.execute(
|
||||
"SELECT thread_id, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata FROM checkpoints WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ?",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
checkpoint_id,
|
||||
),
|
||||
)
|
||||
else:
|
||||
await cur.execute(
|
||||
"SELECT thread_id, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata FROM checkpoints WHERE thread_id = ? AND checkpoint_ns = ? ORDER BY checkpoint_id DESC LIMIT 1",
|
||||
(str(config["configurable"]["thread_id"]), checkpoint_ns),
|
||||
)
|
||||
# if a checkpoint is found, return it
|
||||
if value := await cur.fetchone():
|
||||
(
|
||||
thread_id,
|
||||
checkpoint_id,
|
||||
parent_checkpoint_id,
|
||||
type,
|
||||
checkpoint,
|
||||
metadata,
|
||||
) = value
|
||||
if not get_checkpoint_id(config):
|
||||
config = {
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint_id,
|
||||
}
|
||||
}
|
||||
# find any pending writes
|
||||
await cur.execute(
|
||||
"SELECT task_id, channel, type, value FROM writes WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ? ORDER BY task_id, idx",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
str(config["configurable"]["checkpoint_id"]),
|
||||
),
|
||||
)
|
||||
# deserialize the checkpoint and metadata
|
||||
return CheckpointTuple(
|
||||
config,
|
||||
self.serde.loads_typed((type, checkpoint)),
|
||||
cast(
|
||||
CheckpointMetadata,
|
||||
(json.loads(metadata) if metadata is not None else {}),
|
||||
),
|
||||
(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": parent_checkpoint_id,
|
||||
}
|
||||
}
|
||||
if parent_checkpoint_id
|
||||
else None
|
||||
),
|
||||
[
|
||||
(task_id, channel, self.serde.loads_typed((type, value)))
|
||||
async for task_id, channel, type, value in cur
|
||||
],
|
||||
)
|
||||
|
||||
async def alist(
|
||||
self,
|
||||
config: RunnableConfig | None,
|
||||
*,
|
||||
filter: dict[str, Any] | None = None,
|
||||
before: RunnableConfig | None = None,
|
||||
limit: int | None = None,
|
||||
) -> AsyncIterator[CheckpointTuple]:
|
||||
"""List checkpoints from the database asynchronously.
|
||||
|
||||
This method retrieves a list of checkpoint tuples from the SQLite database based
|
||||
on the provided config. The checkpoints are ordered by checkpoint ID in descending order (newest first).
|
||||
|
||||
Args:
|
||||
config: Base configuration for filtering checkpoints.
|
||||
filter: Additional filtering criteria for metadata.
|
||||
before: If provided, only checkpoints before the specified checkpoint ID are returned.
|
||||
limit: Maximum number of checkpoints to return.
|
||||
|
||||
Yields:
|
||||
An asynchronous iterator of matching checkpoint tuples.
|
||||
"""
|
||||
await self.setup()
|
||||
where, params = search_where(config, filter, before)
|
||||
query = f"""SELECT thread_id, checkpoint_ns, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata
|
||||
FROM checkpoints
|
||||
{where}
|
||||
ORDER BY checkpoint_id DESC"""
|
||||
if limit is not None:
|
||||
query += " LIMIT ?"
|
||||
params = (*params, limit)
|
||||
async with (
|
||||
self.lock,
|
||||
self.conn.execute(query, params) as cur,
|
||||
self.conn.cursor() as wcur,
|
||||
):
|
||||
async for (
|
||||
thread_id,
|
||||
checkpoint_ns,
|
||||
checkpoint_id,
|
||||
parent_checkpoint_id,
|
||||
type,
|
||||
checkpoint,
|
||||
metadata,
|
||||
) in cur:
|
||||
await wcur.execute(
|
||||
"SELECT task_id, channel, type, value FROM writes WHERE thread_id = ? AND checkpoint_ns = ? AND checkpoint_id = ? ORDER BY task_id, idx",
|
||||
(thread_id, checkpoint_ns, checkpoint_id),
|
||||
)
|
||||
yield CheckpointTuple(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint_id,
|
||||
}
|
||||
},
|
||||
self.serde.loads_typed((type, checkpoint)),
|
||||
cast(
|
||||
CheckpointMetadata,
|
||||
(json.loads(metadata) if metadata is not None else {}),
|
||||
),
|
||||
(
|
||||
{
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": parent_checkpoint_id,
|
||||
}
|
||||
}
|
||||
if parent_checkpoint_id
|
||||
else None
|
||||
),
|
||||
[
|
||||
(task_id, channel, self.serde.loads_typed((type, value)))
|
||||
async for task_id, channel, type, value in wcur
|
||||
],
|
||||
)
|
||||
|
||||
async def aput(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
checkpoint: Checkpoint,
|
||||
metadata: CheckpointMetadata,
|
||||
new_versions: ChannelVersions,
|
||||
) -> RunnableConfig:
|
||||
"""Save a checkpoint to the database asynchronously.
|
||||
|
||||
This method saves a checkpoint to the SQLite database. The checkpoint is associated
|
||||
with the provided config and its parent config (if any).
|
||||
|
||||
Args:
|
||||
config: The config to associate with the checkpoint.
|
||||
checkpoint: The checkpoint to save.
|
||||
metadata: Additional metadata to save with the checkpoint.
|
||||
new_versions: New channel versions as of this write.
|
||||
|
||||
Returns:
|
||||
RunnableConfig: Updated configuration after storing the checkpoint.
|
||||
"""
|
||||
await self.setup()
|
||||
thread_id = config["configurable"]["thread_id"]
|
||||
checkpoint_ns = config["configurable"]["checkpoint_ns"]
|
||||
type_, serialized_checkpoint = self.serde.dumps_typed(checkpoint)
|
||||
serialized_metadata = json.dumps(
|
||||
get_checkpoint_metadata(config, metadata), ensure_ascii=False
|
||||
).encode("utf-8", "ignore")
|
||||
async with (
|
||||
self.lock,
|
||||
self.conn.execute(
|
||||
"INSERT OR REPLACE INTO checkpoints (thread_id, checkpoint_ns, checkpoint_id, parent_checkpoint_id, type, checkpoint, metadata) VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
checkpoint_ns,
|
||||
checkpoint["id"],
|
||||
config["configurable"].get("checkpoint_id"),
|
||||
type_,
|
||||
serialized_checkpoint,
|
||||
serialized_metadata,
|
||||
),
|
||||
),
|
||||
):
|
||||
await self.conn.commit()
|
||||
return {
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
"checkpoint_ns": checkpoint_ns,
|
||||
"checkpoint_id": checkpoint["id"],
|
||||
}
|
||||
}
|
||||
|
||||
async def aput_writes(
|
||||
self,
|
||||
config: RunnableConfig,
|
||||
writes: Sequence[tuple[str, Any]],
|
||||
task_id: str,
|
||||
task_path: str = "",
|
||||
) -> None:
|
||||
"""Store intermediate writes linked to a checkpoint asynchronously.
|
||||
|
||||
This method saves intermediate writes associated with a checkpoint to the database.
|
||||
|
||||
Args:
|
||||
config: Configuration of the related checkpoint.
|
||||
writes: List of writes to store, each as (channel, value) pair.
|
||||
task_id: Identifier for the task creating the writes.
|
||||
task_path: Path of the task creating the writes.
|
||||
"""
|
||||
query = (
|
||||
"INSERT OR REPLACE INTO writes (thread_id, checkpoint_ns, checkpoint_id, task_id, idx, channel, type, value) VALUES (?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
if all(w[0] in WRITES_IDX_MAP for w in writes)
|
||||
else "INSERT OR IGNORE INTO writes (thread_id, checkpoint_ns, checkpoint_id, task_id, idx, channel, type, value) VALUES (?, ?, ?, ?, ?, ?, ?, ?)"
|
||||
)
|
||||
await self.setup()
|
||||
async with self.lock, self.conn.cursor() as cur:
|
||||
await cur.executemany(
|
||||
query,
|
||||
[
|
||||
(
|
||||
str(config["configurable"]["thread_id"]),
|
||||
str(config["configurable"]["checkpoint_ns"]),
|
||||
str(config["configurable"]["checkpoint_id"]),
|
||||
task_id,
|
||||
WRITES_IDX_MAP.get(channel, idx),
|
||||
channel,
|
||||
*self.serde.dumps_typed(value),
|
||||
)
|
||||
for idx, (channel, value) in enumerate(writes)
|
||||
],
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def adelete_thread(self, thread_id: str) -> None:
|
||||
"""Delete all checkpoints and writes associated with a thread ID.
|
||||
|
||||
Args:
|
||||
thread_id: The thread ID to delete.
|
||||
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
async with self.lock, self.conn.cursor() as cur:
|
||||
await cur.execute(
|
||||
"DELETE FROM checkpoints WHERE thread_id = ?",
|
||||
(str(thread_id),),
|
||||
)
|
||||
await cur.execute(
|
||||
"DELETE FROM writes WHERE thread_id = ?",
|
||||
(str(thread_id),),
|
||||
)
|
||||
await self.conn.commit()
|
||||
|
||||
async def aget_delta_channel_history(
|
||||
self, *, config: RunnableConfig, channels: Sequence[str]
|
||||
) -> Mapping[str, DeltaChannelHistory]:
|
||||
"""Fast-path override of `BaseCheckpointSaver.aget_delta_channel_history`.
|
||||
|
||||
See `SqliteSaver.get_delta_channel_history` for design notes; this
|
||||
is the async equivalent using `aiosqlite` cursors. Stage 1 pages
|
||||
the parent chain newest-first and Python-deserializes each
|
||||
checkpoint blob to find per-channel snapshots; stage 2 fetches
|
||||
only the relevant writes via per-channel UNION ALL.
|
||||
"""
|
||||
if not channels:
|
||||
return {}
|
||||
channels = list(channels)
|
||||
await self.setup()
|
||||
thread_id = str(config["configurable"]["thread_id"])
|
||||
checkpoint_ns = config["configurable"].get("checkpoint_ns", "")
|
||||
checkpoint_id = get_checkpoint_id(config)
|
||||
if checkpoint_id is None:
|
||||
target = await self.aget_tuple(config)
|
||||
if target is None:
|
||||
return {ch: {"writes": []} for ch in channels}
|
||||
checkpoint_id = target.config["configurable"]["checkpoint_id"]
|
||||
|
||||
chain_by_ch: dict[str, list[str]] = {ch: [] for ch in channels}
|
||||
seed_val_by_ch: dict[str, Any] = {}
|
||||
walk_state: dict[str, Any] = {}
|
||||
seeded: set[str] = set()
|
||||
|
||||
async with self.lock, self.conn.cursor() as cur:
|
||||
await cur.execute(
|
||||
DELTA_STAGE1_SQL, (thread_id, checkpoint_ns, checkpoint_id)
|
||||
)
|
||||
async for row in cur:
|
||||
cid, parent_cid, type_tag, blob = row
|
||||
if step_walk_with_row(
|
||||
cid=cid,
|
||||
parent_cid=parent_cid,
|
||||
type_tag=type_tag,
|
||||
blob=blob,
|
||||
target_id=checkpoint_id,
|
||||
serde=self.serde,
|
||||
chain_by_ch=chain_by_ch,
|
||||
seed_val_by_ch=seed_val_by_ch,
|
||||
walk_state=walk_state,
|
||||
seeded=seeded,
|
||||
channels=channels,
|
||||
):
|
||||
break
|
||||
|
||||
channels_with_chain = [ch for ch in channels if chain_by_ch[ch]]
|
||||
stage2_sql = build_delta_stage2_sql(
|
||||
chain_lens=[len(chain_by_ch[ch]) for ch in channels_with_chain],
|
||||
)
|
||||
if stage2_sql:
|
||||
stage2_params: list[Any] = []
|
||||
for ch in channels_with_chain:
|
||||
stage2_params.extend(
|
||||
[thread_id, checkpoint_ns, ch, *chain_by_ch[ch]]
|
||||
)
|
||||
await cur.execute(stage2_sql, stage2_params)
|
||||
stage2_rows = cast(
|
||||
"list[tuple[str, str, str, int, str, bytes]]",
|
||||
await cur.fetchall(),
|
||||
)
|
||||
else:
|
||||
stage2_rows = []
|
||||
|
||||
return build_delta_channels_writes_history(
|
||||
channels=channels,
|
||||
chain_by_ch=chain_by_ch,
|
||||
seed_val_by_ch=seed_val_by_ch,
|
||||
seeded=seeded,
|
||||
stage2_rows=stage2_rows,
|
||||
serde=self.serde,
|
||||
)
|
||||
|
||||
def get_next_version(self, current: str | None, channel: None) -> str:
|
||||
"""Generate the next version ID for a channel.
|
||||
|
||||
This method creates a new version identifier for a channel based on its current version.
|
||||
|
||||
Args:
|
||||
current (Optional[str]): The current version identifier of the channel.
|
||||
|
||||
Returns:
|
||||
str: The next version identifier, which is guaranteed to be monotonically increasing.
|
||||
"""
|
||||
if current is None:
|
||||
current_v = 0
|
||||
elif isinstance(current, int):
|
||||
current_v = current
|
||||
else:
|
||||
current_v = int(current.split(".")[0])
|
||||
next_v = current_v + 1
|
||||
next_h = random.random()
|
||||
return f"{next_v:032}.{next_h:016}"
|
||||
|
||||
|
||||
async def _ensure_connected(conn: aiosqlite.Connection) -> None:
|
||||
if not _CONN_STARTED_CHECK(conn):
|
||||
await conn
|
||||
|
||||
|
||||
def _build_conn_started_check() -> Callable[[aiosqlite.Connection], bool]:
|
||||
is_alive = getattr(aiosqlite.Connection, "is_alive", None)
|
||||
if callable(is_alive):
|
||||
return lambda conn: conn.is_alive() # type: ignore[attr-defined]
|
||||
|
||||
def _started(conn: aiosqlite.Connection) -> bool:
|
||||
thread: threading.Thread | None = getattr(conn, "_thread", None)
|
||||
return False if thread is None else thread.is_alive()
|
||||
|
||||
return _started
|
||||
|
||||
|
||||
_CONN_STARTED_CHECK = _build_conn_started_check()
|
||||
@@ -0,0 +1,116 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from langgraph.checkpoint.base import get_checkpoint_id
|
||||
|
||||
_FILTER_PATTERN = re.compile(r"^[a-zA-Z0-9_.-]+$")
|
||||
|
||||
|
||||
def _validate_filter_key(key: str) -> None:
|
||||
"""Validate that a filter key is safe for use in SQL queries.
|
||||
|
||||
Args:
|
||||
key: The filter key to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If the key contains invalid characters that could enable SQL injection
|
||||
"""
|
||||
# Allow alphanumeric characters, underscores, dots, and hyphens
|
||||
# This covers typical JSON property names while preventing SQL injection
|
||||
if not _FILTER_PATTERN.match(key):
|
||||
raise ValueError(
|
||||
f"Invalid filter key: '{key}'. Filter keys must contain only alphanumeric characters, underscores, dots, and hyphens."
|
||||
)
|
||||
|
||||
|
||||
def _metadata_predicate(
|
||||
metadata_filter: dict[str, Any],
|
||||
) -> tuple[Sequence[str], Sequence[Any]]:
|
||||
"""Return WHERE clause predicates for (a)search() given metadata filter.
|
||||
|
||||
This method returns a tuple of a string and a tuple of values. The string
|
||||
is the parametered WHERE clause predicate (excluding the WHERE keyword):
|
||||
"column1 = ? AND column2 IS ?". The tuple of values contains the values
|
||||
for each of the corresponding parameters.
|
||||
"""
|
||||
|
||||
def _where_value(query_value: Any) -> tuple[str, Any]:
|
||||
"""Return tuple of operator and value for WHERE clause predicate."""
|
||||
if query_value is None:
|
||||
return ("IS ?", None)
|
||||
elif (
|
||||
isinstance(query_value, str)
|
||||
or isinstance(query_value, int)
|
||||
or isinstance(query_value, float)
|
||||
):
|
||||
return ("= ?", query_value)
|
||||
elif isinstance(query_value, bool):
|
||||
return ("= ?", 1 if query_value else 0)
|
||||
elif isinstance(query_value, dict) or isinstance(query_value, list):
|
||||
# query value for JSON object cannot have trailing space after separators (, :)
|
||||
# SQLite json_extract() returns JSON string without whitespace
|
||||
return ("= ?", json.dumps(query_value, separators=(",", ":")))
|
||||
else:
|
||||
return ("= ?", str(query_value))
|
||||
|
||||
predicates = []
|
||||
param_values = []
|
||||
|
||||
# process metadata query
|
||||
for query_key, query_value in metadata_filter.items():
|
||||
_validate_filter_key(query_key)
|
||||
operator, param_value = _where_value(query_value)
|
||||
predicates.append(
|
||||
f"json_extract(CAST(metadata AS TEXT), '$.{query_key}') {operator}"
|
||||
)
|
||||
param_values.append(param_value)
|
||||
|
||||
return (predicates, param_values)
|
||||
|
||||
|
||||
def search_where(
|
||||
config: RunnableConfig | None,
|
||||
filter: dict[str, Any] | None,
|
||||
before: RunnableConfig | None = None,
|
||||
) -> tuple[str, Sequence[Any]]:
|
||||
"""Return WHERE clause predicates for (a)search() given metadata filter
|
||||
and `before` config.
|
||||
|
||||
This method returns a tuple of a string and a tuple of values. The string
|
||||
is the parametered WHERE clause predicate (including the WHERE keyword):
|
||||
"WHERE column1 = ? AND column2 IS ?". The tuple of values contains the
|
||||
values for each of the corresponding parameters.
|
||||
"""
|
||||
wheres = []
|
||||
param_values = []
|
||||
|
||||
# construct predicate for config filter
|
||||
if config is not None:
|
||||
wheres.append("thread_id = ?")
|
||||
param_values.append(config["configurable"]["thread_id"])
|
||||
checkpoint_ns = config["configurable"].get("checkpoint_ns")
|
||||
if checkpoint_ns is not None:
|
||||
wheres.append("checkpoint_ns = ?")
|
||||
param_values.append(checkpoint_ns)
|
||||
|
||||
if checkpoint_id := get_checkpoint_id(config):
|
||||
wheres.append("checkpoint_id = ?")
|
||||
param_values.append(checkpoint_id)
|
||||
|
||||
# construct predicate for metadata filter
|
||||
if filter:
|
||||
metadata_predicates, metadata_values = _metadata_predicate(filter)
|
||||
wheres.extend(metadata_predicates)
|
||||
param_values.extend(metadata_values)
|
||||
|
||||
# construct predicate for `before`
|
||||
if before is not None:
|
||||
wheres.append("checkpoint_id < ?")
|
||||
param_values.append(get_checkpoint_id(before))
|
||||
|
||||
return ("WHERE " + " AND ".join(wheres) if wheres else "", param_values)
|
||||
@@ -0,0 +1,4 @@
|
||||
from langgraph.store.sqlite.aio import AsyncSqliteStore
|
||||
from langgraph.store.sqlite.base import SqliteStore
|
||||
|
||||
__all__ = ["AsyncSqliteStore", "SqliteStore"]
|
||||
@@ -0,0 +1,623 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections import defaultdict
|
||||
from collections.abc import AsyncIterator, Callable, Iterable, Sequence
|
||||
from contextlib import asynccontextmanager
|
||||
from types import TracebackType
|
||||
from typing import Any, cast
|
||||
|
||||
import aiosqlite
|
||||
import orjson
|
||||
import sqlite_vec # type: ignore[import-untyped]
|
||||
from langgraph.store.base import (
|
||||
GetOp,
|
||||
ListNamespacesOp,
|
||||
Op,
|
||||
PutOp,
|
||||
Result,
|
||||
SearchOp,
|
||||
TTLConfig,
|
||||
)
|
||||
from langgraph.store.base.batch import AsyncBatchedBaseStore
|
||||
|
||||
from langgraph.store.sqlite.base import (
|
||||
_PLACEHOLDER,
|
||||
BaseSqliteStore,
|
||||
SqliteIndexConfig,
|
||||
_decode_ns_text,
|
||||
_ensure_index_config,
|
||||
_group_ops,
|
||||
_row_to_item,
|
||||
_row_to_search_item,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AsyncSqliteStore(AsyncBatchedBaseStore, BaseSqliteStore):
|
||||
"""Asynchronous SQLite-backed store with optional vector search.
|
||||
|
||||
This class provides an asynchronous interface for storing and retrieving data
|
||||
using a SQLite database with support for vector search capabilities.
|
||||
|
||||
Examples:
|
||||
Basic setup and usage:
|
||||
```python
|
||||
from langgraph.store.sqlite import AsyncSqliteStore
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(":memory:") as store:
|
||||
await store.setup() # Run migrations
|
||||
|
||||
# Store and retrieve data
|
||||
await store.aput(("users", "123"), "prefs", {"theme": "dark"})
|
||||
item = await store.aget(("users", "123"), "prefs")
|
||||
```
|
||||
|
||||
Vector search using LangChain embeddings:
|
||||
```python
|
||||
from langchain_openai import OpenAIEmbeddings
|
||||
from langgraph.store.sqlite import AsyncSqliteStore
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
":memory:",
|
||||
index={
|
||||
"dims": 1536,
|
||||
"embed": OpenAIEmbeddings(),
|
||||
"fields": ["text"] # specify which fields to embed
|
||||
}
|
||||
) as store:
|
||||
await store.setup() # Run migrations once
|
||||
|
||||
# Store documents
|
||||
await store.aput(("docs",), "doc1", {"text": "Python tutorial"})
|
||||
await store.aput(("docs",), "doc2", {"text": "TypeScript guide"})
|
||||
await store.aput(("docs",), "doc3", {"text": "Other guide"}, index=False) # don't index
|
||||
|
||||
# Search by similarity
|
||||
results = await store.asearch(("docs",), query="programming guides", limit=2)
|
||||
```
|
||||
|
||||
Warning:
|
||||
Make sure to call `setup()` before first use to create necessary tables and indexes.
|
||||
|
||||
Note:
|
||||
This class requires the aiosqlite package. Install with `pip install aiosqlite`.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
conn: aiosqlite.Connection,
|
||||
*,
|
||||
deserializer: Callable[[bytes | str | orjson.Fragment], dict[str, Any]]
|
||||
| None = None,
|
||||
index: SqliteIndexConfig | None = None,
|
||||
ttl: TTLConfig | None = None,
|
||||
):
|
||||
"""Initialize the async SQLite store.
|
||||
|
||||
Args:
|
||||
conn: The SQLite database connection.
|
||||
deserializer: Optional custom deserializer function for values.
|
||||
index: Optional vector search configuration.
|
||||
ttl: Optional time-to-live configuration.
|
||||
"""
|
||||
super().__init__()
|
||||
self._deserializer = deserializer
|
||||
self.conn = conn
|
||||
self.lock = asyncio.Lock()
|
||||
self.loop = asyncio.get_running_loop()
|
||||
self.is_setup = False
|
||||
self.index_config = index
|
||||
if self.index_config:
|
||||
self.embeddings, self.index_config = _ensure_index_config(self.index_config)
|
||||
else:
|
||||
self.embeddings = None
|
||||
self.ttl_config = ttl
|
||||
self._ttl_sweeper_task: asyncio.Task[None] | None = None
|
||||
self._ttl_stop_event = asyncio.Event()
|
||||
|
||||
@classmethod
|
||||
@asynccontextmanager
|
||||
async def from_conn_string(
|
||||
cls,
|
||||
conn_string: str,
|
||||
*,
|
||||
index: SqliteIndexConfig | None = None,
|
||||
ttl: TTLConfig | None = None,
|
||||
) -> AsyncIterator[AsyncSqliteStore]:
|
||||
"""Create a new AsyncSqliteStore instance from a connection string.
|
||||
|
||||
Args:
|
||||
conn_string: The SQLite connection string.
|
||||
index: Optional vector search configuration.
|
||||
ttl: Optional time-to-live configuration.
|
||||
|
||||
Returns:
|
||||
An AsyncSqliteStore instance wrapped in an async context manager.
|
||||
"""
|
||||
async with aiosqlite.connect(conn_string, isolation_level=None) as conn:
|
||||
yield cls(conn, index=index, ttl=ttl)
|
||||
|
||||
async def setup(self) -> None:
|
||||
"""Set up the store database.
|
||||
|
||||
This method creates the necessary tables in the SQLite database if they don't
|
||||
already exist and runs database migrations. It should be called before first use.
|
||||
"""
|
||||
async with self.lock:
|
||||
if self.is_setup:
|
||||
return
|
||||
|
||||
# Create migrations table if it doesn't exist
|
||||
await self.conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS store_migrations (
|
||||
v INTEGER PRIMARY KEY
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
# Check current migration version
|
||||
async with self.conn.execute(
|
||||
"SELECT v FROM store_migrations ORDER BY v DESC LIMIT 1"
|
||||
) as cur:
|
||||
row = await cur.fetchone()
|
||||
if row is None:
|
||||
version = -1
|
||||
else:
|
||||
version = row[0]
|
||||
|
||||
# Apply migrations
|
||||
for v, sql in enumerate(self.MIGRATIONS[version + 1 :], start=version + 1):
|
||||
await self.conn.executescript(sql)
|
||||
await self.conn.execute(
|
||||
"INSERT INTO store_migrations (v) VALUES (?)", (v,)
|
||||
)
|
||||
|
||||
# Apply vector migrations if index config is provided
|
||||
if self.index_config:
|
||||
# Create vector migrations table if it doesn't exist
|
||||
await self.conn.enable_load_extension(True)
|
||||
await self.conn.load_extension(sqlite_vec.loadable_path())
|
||||
await self.conn.enable_load_extension(False)
|
||||
await self.conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS vector_migrations (
|
||||
v INTEGER PRIMARY KEY
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
# Check current vector migration version
|
||||
async with self.conn.execute(
|
||||
"SELECT v FROM vector_migrations ORDER BY v DESC LIMIT 1"
|
||||
) as cur:
|
||||
row = await cur.fetchone()
|
||||
if row is None:
|
||||
version = -1
|
||||
else:
|
||||
version = row[0]
|
||||
|
||||
# Apply vector migrations
|
||||
for v, sql in enumerate(
|
||||
self.VECTOR_MIGRATIONS[version + 1 :], start=version + 1
|
||||
):
|
||||
await self.conn.executescript(sql)
|
||||
await self.conn.execute(
|
||||
"INSERT INTO vector_migrations (v) VALUES (?)", (v,)
|
||||
)
|
||||
|
||||
self.is_setup = True
|
||||
|
||||
@asynccontextmanager
|
||||
async def _cursor(
|
||||
self, *, transaction: bool = True
|
||||
) -> AsyncIterator[aiosqlite.Cursor]:
|
||||
"""Get a cursor for the SQLite database.
|
||||
|
||||
Args:
|
||||
transaction: Whether to use a transaction for database operations.
|
||||
|
||||
Yields:
|
||||
An SQLite cursor object.
|
||||
"""
|
||||
if not self.is_setup:
|
||||
await self.setup()
|
||||
async with self.lock:
|
||||
if transaction:
|
||||
await self.conn.execute("BEGIN")
|
||||
|
||||
async with self.conn.cursor() as cur:
|
||||
try:
|
||||
yield cur
|
||||
finally:
|
||||
if transaction:
|
||||
await self.conn.execute("COMMIT")
|
||||
|
||||
async def sweep_ttl(self) -> int:
|
||||
"""Delete expired store items based on TTL.
|
||||
|
||||
Returns:
|
||||
int: The number of deleted items.
|
||||
"""
|
||||
async with self._cursor() as cur:
|
||||
await cur.execute(
|
||||
"""
|
||||
DELETE FROM store
|
||||
WHERE expires_at IS NOT NULL AND expires_at < CURRENT_TIMESTAMP
|
||||
"""
|
||||
)
|
||||
deleted_count = cur.rowcount
|
||||
return deleted_count
|
||||
|
||||
async def start_ttl_sweeper(
|
||||
self, sweep_interval_minutes: int | None = None
|
||||
) -> asyncio.Task[None]:
|
||||
"""Periodically delete expired store items based on TTL.
|
||||
|
||||
Returns:
|
||||
Task that can be awaited or cancelled.
|
||||
"""
|
||||
if not self.ttl_config:
|
||||
return asyncio.create_task(asyncio.sleep(0))
|
||||
|
||||
if self._ttl_sweeper_task is not None and not self._ttl_sweeper_task.done():
|
||||
return self._ttl_sweeper_task
|
||||
|
||||
self._ttl_stop_event.clear()
|
||||
|
||||
interval = float(
|
||||
sweep_interval_minutes or self.ttl_config.get("sweep_interval_minutes") or 5
|
||||
)
|
||||
logger.info(f"Starting store TTL sweeper with interval {interval} minutes")
|
||||
|
||||
async def _sweep_loop() -> None:
|
||||
while not self._ttl_stop_event.is_set():
|
||||
try:
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
self._ttl_stop_event.wait(),
|
||||
timeout=interval * 60,
|
||||
)
|
||||
break
|
||||
except asyncio.TimeoutError:
|
||||
pass
|
||||
|
||||
expired_items = await self.sweep_ttl()
|
||||
if expired_items > 0:
|
||||
logger.info(f"Store swept {expired_items} expired items")
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as exc:
|
||||
logger.exception("Store TTL sweep iteration failed", exc_info=exc)
|
||||
|
||||
task = asyncio.create_task(_sweep_loop())
|
||||
task.set_name("ttl_sweeper")
|
||||
self._ttl_sweeper_task = task
|
||||
return task
|
||||
|
||||
async def stop_ttl_sweeper(self, timeout: float | None = None) -> bool:
|
||||
"""Stop the TTL sweeper task if it's running.
|
||||
|
||||
Args:
|
||||
timeout: Maximum time to wait for the task to stop, in seconds.
|
||||
If `None`, wait indefinitely.
|
||||
|
||||
Returns:
|
||||
bool: True if the task was successfully stopped or wasn't running,
|
||||
False if the timeout was reached before the task stopped.
|
||||
"""
|
||||
if self._ttl_sweeper_task is None or self._ttl_sweeper_task.done():
|
||||
return True
|
||||
|
||||
logger.info("Stopping TTL sweeper task")
|
||||
self._ttl_stop_event.set()
|
||||
|
||||
if timeout is not None:
|
||||
try:
|
||||
await asyncio.wait_for(self._ttl_sweeper_task, timeout=timeout)
|
||||
success = True
|
||||
except asyncio.TimeoutError:
|
||||
success = False
|
||||
else:
|
||||
await self._ttl_sweeper_task
|
||||
success = True
|
||||
|
||||
if success:
|
||||
self._ttl_sweeper_task = None
|
||||
logger.info("TTL sweeper task stopped")
|
||||
else:
|
||||
logger.warning("Timed out waiting for TTL sweeper task to stop")
|
||||
|
||||
return success
|
||||
|
||||
async def __aenter__(self) -> AsyncSqliteStore:
|
||||
return self
|
||||
|
||||
async def __aexit__(
|
||||
self,
|
||||
exc_type: type[BaseException] | None,
|
||||
exc_val: BaseException | None,
|
||||
exc_tb: TracebackType | None,
|
||||
) -> None:
|
||||
# Ensure the TTL sweeper task is stopped when exiting the context
|
||||
if hasattr(self, "_ttl_sweeper_task") and self._ttl_sweeper_task is not None:
|
||||
# Set the event to signal the task to stop
|
||||
self._ttl_stop_event.set()
|
||||
# We don't wait for the task to complete here to avoid blocking
|
||||
# The task will clean up itself gracefully
|
||||
|
||||
async def abatch(self, ops: Iterable[Op]) -> list[Result]:
|
||||
"""Execute a batch of operations asynchronously.
|
||||
|
||||
Args:
|
||||
ops: Iterable of operations to execute.
|
||||
|
||||
Returns:
|
||||
List of operation results.
|
||||
"""
|
||||
grouped_ops, num_ops = _group_ops(ops)
|
||||
results: list[Result] = [None] * num_ops
|
||||
|
||||
async with self._cursor(transaction=True) as cur:
|
||||
if GetOp in grouped_ops:
|
||||
await self._batch_get_ops(
|
||||
cast(Sequence[tuple[int, GetOp]], grouped_ops[GetOp]), results, cur
|
||||
)
|
||||
|
||||
if SearchOp in grouped_ops:
|
||||
await self._batch_search_ops(
|
||||
cast(Sequence[tuple[int, SearchOp]], grouped_ops[SearchOp]),
|
||||
results,
|
||||
cur,
|
||||
)
|
||||
|
||||
if ListNamespacesOp in grouped_ops:
|
||||
await self._batch_list_namespaces_ops(
|
||||
cast(
|
||||
Sequence[tuple[int, ListNamespacesOp]],
|
||||
grouped_ops[ListNamespacesOp],
|
||||
),
|
||||
results,
|
||||
cur,
|
||||
)
|
||||
|
||||
if PutOp in grouped_ops:
|
||||
await self._batch_put_ops(
|
||||
cast(Sequence[tuple[int, PutOp]], grouped_ops[PutOp]), cur
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
async def _batch_get_ops(
|
||||
self,
|
||||
get_ops: Sequence[tuple[int, GetOp]],
|
||||
results: list[Result],
|
||||
cur: aiosqlite.Cursor,
|
||||
) -> None:
|
||||
"""Process batch GET operations.
|
||||
|
||||
Args:
|
||||
get_ops: Sequence of GET operations.
|
||||
results: List to store results in.
|
||||
cur: Database cursor.
|
||||
"""
|
||||
# Group all queries by namespace to execute all operations for each namespace together
|
||||
namespace_queries = defaultdict(list)
|
||||
for prepared_query in self._get_batch_GET_ops_queries(get_ops):
|
||||
namespace_queries[prepared_query.namespace].append(prepared_query)
|
||||
|
||||
# Process each namespace's operations
|
||||
for namespace, queries in namespace_queries.items():
|
||||
# Execute TTL refresh queries first
|
||||
for query in queries:
|
||||
if query.kind == "refresh":
|
||||
try:
|
||||
await cur.execute(query.query, query.params)
|
||||
except Exception as e:
|
||||
raise ValueError(
|
||||
f"Error executing TTL refresh: \n{query.query}\n{query.params}\n{e}"
|
||||
) from e
|
||||
|
||||
# Then execute GET queries and process results
|
||||
for query in queries:
|
||||
if query.kind == "get":
|
||||
try:
|
||||
await cur.execute(query.query, query.params)
|
||||
except Exception as e:
|
||||
raise ValueError(
|
||||
f"Error executing GET query: \n{query.query}\n{query.params}\n{e}"
|
||||
) from e
|
||||
|
||||
rows = await cur.fetchall()
|
||||
key_to_row = {
|
||||
row[0]: {
|
||||
"key": row[0],
|
||||
"value": row[1],
|
||||
"created_at": row[2],
|
||||
"updated_at": row[3],
|
||||
"expires_at": row[4] if len(row) > 4 else None,
|
||||
"ttl_minutes": row[5] if len(row) > 5 else None,
|
||||
}
|
||||
for row in rows
|
||||
}
|
||||
|
||||
# Process results for this query
|
||||
for idx, key in query.items:
|
||||
row = key_to_row.get(key)
|
||||
if row:
|
||||
results[idx] = _row_to_item(
|
||||
namespace, row, loader=self._deserializer
|
||||
)
|
||||
else:
|
||||
results[idx] = None
|
||||
|
||||
async def _batch_put_ops(
|
||||
self,
|
||||
put_ops: Sequence[tuple[int, PutOp]],
|
||||
cur: aiosqlite.Cursor,
|
||||
) -> None:
|
||||
"""Process batch PUT operations.
|
||||
|
||||
Args:
|
||||
put_ops: Sequence of PUT operations.
|
||||
cur: Database cursor.
|
||||
"""
|
||||
queries, embedding_request = self._prepare_batch_PUT_queries(put_ops)
|
||||
if embedding_request:
|
||||
if self.embeddings is None:
|
||||
# Should not get here since the embedding config is required
|
||||
# to return an embedding_request above
|
||||
raise ValueError(
|
||||
"Embedding configuration is required for vector operations "
|
||||
f"(for semantic search). "
|
||||
f"Please provide an Embeddings when initializing the {self.__class__.__name__}."
|
||||
)
|
||||
|
||||
query, txt_params = embedding_request
|
||||
# Update the params to replace the raw text with the vectors
|
||||
vectors = await self.embeddings.aembed_documents(
|
||||
[param[-1] for param in txt_params]
|
||||
)
|
||||
|
||||
# Convert vectors to SQLite-friendly format
|
||||
vector_params = []
|
||||
for (ns, k, pathname, _), vector in zip(txt_params, vectors, strict=False):
|
||||
vector_params.extend(
|
||||
[ns, k, pathname, sqlite_vec.serialize_float32(vector)]
|
||||
)
|
||||
|
||||
queries.append((query, vector_params))
|
||||
|
||||
for query, params in queries:
|
||||
await cur.execute(query, params)
|
||||
|
||||
async def _batch_search_ops(
|
||||
self,
|
||||
search_ops: Sequence[tuple[int, SearchOp]],
|
||||
results: list[Result],
|
||||
cur: aiosqlite.Cursor,
|
||||
) -> None:
|
||||
"""Process batch SEARCH operations.
|
||||
|
||||
Args:
|
||||
search_ops: Sequence of SEARCH operations.
|
||||
results: List to store results in.
|
||||
cur: Database cursor.
|
||||
"""
|
||||
prepared_queries, embedding_requests = self._prepare_batch_search_queries(
|
||||
search_ops
|
||||
)
|
||||
|
||||
# Setup dot_product function if it doesn't exist
|
||||
if embedding_requests and self.embeddings:
|
||||
vectors = await self.embeddings.aembed_documents(
|
||||
[query for _, query in embedding_requests]
|
||||
)
|
||||
|
||||
for (embed_req_idx, _), embedding in zip(
|
||||
embedding_requests, vectors, strict=False
|
||||
):
|
||||
# Find the corresponding query in prepared_queries
|
||||
# The embed_req_idx is the original index in search_ops, which should map to prepared_queries
|
||||
if embed_req_idx < len(prepared_queries):
|
||||
_params_list: list = prepared_queries[embed_req_idx][1]
|
||||
for i, param in enumerate(_params_list):
|
||||
if param is _PLACEHOLDER:
|
||||
_params_list[i] = sqlite_vec.serialize_float32(embedding)
|
||||
else:
|
||||
logger.warning(
|
||||
f"Embedding request index {embed_req_idx} out of bounds for prepared_queries."
|
||||
)
|
||||
|
||||
for (original_op_idx, _), (query, params, needs_refresh) in zip(
|
||||
search_ops, prepared_queries, strict=False
|
||||
):
|
||||
await cur.execute(query, params)
|
||||
rows = await cur.fetchall()
|
||||
|
||||
if needs_refresh and rows and self.ttl_config:
|
||||
keys_to_refresh = []
|
||||
for row_data in rows:
|
||||
# Assuming row_data[0] is prefix (text), row_data[1] is key (text)
|
||||
# These are raw text values directly from the DB.
|
||||
keys_to_refresh.append((row_data[0], row_data[1]))
|
||||
|
||||
if keys_to_refresh:
|
||||
updates_by_prefix = defaultdict(list)
|
||||
for prefix_text, key_text in keys_to_refresh:
|
||||
updates_by_prefix[prefix_text].append(key_text)
|
||||
|
||||
for prefix_text, key_list in updates_by_prefix.items():
|
||||
placeholders = ",".join(["?"] * len(key_list))
|
||||
update_query = f"""
|
||||
UPDATE store
|
||||
SET expires_at = DATETIME(CURRENT_TIMESTAMP, '+' || ttl_minutes || ' minutes')
|
||||
WHERE prefix = ? AND key IN ({placeholders}) AND ttl_minutes IS NOT NULL
|
||||
"""
|
||||
update_params = (prefix_text, *key_list)
|
||||
try:
|
||||
await cur.execute(update_query, update_params)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error during TTL refresh update for search: {e}"
|
||||
)
|
||||
|
||||
# Process rows into items
|
||||
if "score" in query: # Vector search query
|
||||
items = [
|
||||
_row_to_search_item(
|
||||
_decode_ns_text(row[0]), # prefix
|
||||
{
|
||||
"key": row[1], # key
|
||||
"value": row[2], # value
|
||||
"created_at": row[3],
|
||||
"updated_at": row[4],
|
||||
"expires_at": row[5] if len(row) > 5 else None,
|
||||
"ttl_minutes": row[6] if len(row) > 6 else None,
|
||||
"score": row[7] if len(row) > 7 else None,
|
||||
},
|
||||
loader=self._deserializer,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
else: # Regular search query
|
||||
items = [
|
||||
_row_to_search_item(
|
||||
_decode_ns_text(row[0]), # prefix
|
||||
{
|
||||
"key": row[1], # key
|
||||
"value": row[2], # value
|
||||
"created_at": row[3],
|
||||
"updated_at": row[4],
|
||||
"expires_at": row[5] if len(row) > 5 else None,
|
||||
"ttl_minutes": row[6] if len(row) > 6 else None,
|
||||
},
|
||||
loader=self._deserializer,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
results[original_op_idx] = items
|
||||
|
||||
async def _batch_list_namespaces_ops(
|
||||
self,
|
||||
list_ops: Sequence[tuple[int, ListNamespacesOp]],
|
||||
results: list[Result],
|
||||
cur: aiosqlite.Cursor,
|
||||
) -> None:
|
||||
"""Process batch LIST NAMESPACES operations.
|
||||
|
||||
Args:
|
||||
list_ops: Sequence of LIST NAMESPACES operations.
|
||||
results: List to store results in.
|
||||
cur: Database cursor.
|
||||
"""
|
||||
queries = self._get_batch_list_namespaces_queries(list_ops)
|
||||
for (query, params), (idx, _) in zip(queries, list_ops, strict=False):
|
||||
await cur.execute(query, params)
|
||||
|
||||
rows = await cur.fetchall()
|
||||
results[idx] = [_decode_ns_text(row[0]) for row in rows]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,88 @@
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "langgraph-checkpoint-sqlite"
|
||||
version = "3.1.0"
|
||||
description = "Library with a SQLite implementation of LangGraph checkpoint saver."
|
||||
authors = []
|
||||
requires-python = ">=3.10"
|
||||
readme = "README.md"
|
||||
license = "MIT"
|
||||
license-files = ['LICENSE']
|
||||
dependencies = [
|
||||
"langgraph-checkpoint>=4.1.0,<5.0.0",
|
||||
"aiosqlite>=0.20",
|
||||
"sqlite-vec>=0.1.6",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Source = "https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint-sqlite"
|
||||
Twitter = "https://x.com/langchain_oss"
|
||||
Slack = "https://www.langchain.com/join-community"
|
||||
Reddit = "https://www.reddit.com/r/LangChain/"
|
||||
|
||||
[dependency-groups]
|
||||
test = [
|
||||
"pytest",
|
||||
"pytest-asyncio",
|
||||
"pytest-mock",
|
||||
"pytest-watcher",
|
||||
"langgraph-checkpoint",
|
||||
"pytest-retry>=1.7.0",
|
||||
]
|
||||
lint = [
|
||||
"ruff",
|
||||
"codespell",
|
||||
"ty",
|
||||
]
|
||||
dev = [
|
||||
{include-group = "test"},
|
||||
{include-group = "lint"},
|
||||
]
|
||||
|
||||
[tool.uv]
|
||||
default-groups = ['dev']
|
||||
|
||||
[tool.uv.sources]
|
||||
langgraph-checkpoint = { path = "../checkpoint", editable = true }
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
include = ["langgraph"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
addopts = "--strict-markers --strict-config --durations=5 -vv"
|
||||
asyncio_mode = "auto"
|
||||
|
||||
[tool.ruff]
|
||||
lint.select = [
|
||||
"E", # pycodestyle
|
||||
"F", # Pyflakes
|
||||
"UP", # pyupgrade
|
||||
"B", # flake8-bugbear
|
||||
"I", # isort
|
||||
"UP", # pyupgrade
|
||||
]
|
||||
lint.ignore = ["E501", "B008"]
|
||||
target-version = "py310"
|
||||
|
||||
[tool.ty.rules]
|
||||
invalid-argument-type = "ignore"
|
||||
invalid-assignment = "ignore"
|
||||
invalid-key = "ignore"
|
||||
invalid-method-override = "ignore"
|
||||
invalid-return-type = "ignore"
|
||||
invalid-yield = "ignore"
|
||||
not-iterable = "ignore"
|
||||
not-subscriptable = "ignore"
|
||||
unused-type-ignore-comment = "ignore"
|
||||
unresolved-attribute = "ignore"
|
||||
unresolved-import = "ignore"
|
||||
unsupported-operator = "ignore"
|
||||
|
||||
[tool.pytest-watcher]
|
||||
now = true
|
||||
delay = 0.1
|
||||
runner_args = ["--ff", "-v", "--tb", "short"]
|
||||
patterns = ["*.py"]
|
||||
@@ -0,0 +1,190 @@
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from langgraph.checkpoint.base import (
|
||||
Checkpoint,
|
||||
CheckpointMetadata,
|
||||
create_checkpoint,
|
||||
empty_checkpoint,
|
||||
)
|
||||
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
|
||||
|
||||
class TestAsyncSqliteSaver:
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self) -> None:
|
||||
# objects for test setup
|
||||
self.config_1: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-1",
|
||||
"checkpoint_id": "1",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
self.config_2: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_id": "2",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
self.config_3: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_id": "2-inner",
|
||||
"checkpoint_ns": "inner",
|
||||
}
|
||||
}
|
||||
|
||||
self.chkpnt_1: Checkpoint = empty_checkpoint()
|
||||
self.chkpnt_2: Checkpoint = create_checkpoint(self.chkpnt_1, {}, 1)
|
||||
self.chkpnt_3: Checkpoint = empty_checkpoint()
|
||||
|
||||
self.metadata_1: CheckpointMetadata = {
|
||||
"source": "input",
|
||||
"step": 2,
|
||||
"writes": {},
|
||||
"score": 1,
|
||||
}
|
||||
self.metadata_2: CheckpointMetadata = {
|
||||
"source": "loop",
|
||||
"step": 1,
|
||||
"writes": {"foo": "bar"},
|
||||
"score": None,
|
||||
}
|
||||
self.metadata_3: CheckpointMetadata = {}
|
||||
|
||||
async def test_combined_metadata(self) -> None:
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_ns": "",
|
||||
"__super_private_key": "super_private_value",
|
||||
},
|
||||
"metadata": {"run_id": "my_run_id"},
|
||||
}
|
||||
await saver.aput(config, self.chkpnt_2, self.metadata_2, {})
|
||||
checkpoint = await saver.aget_tuple(config)
|
||||
assert checkpoint is not None and checkpoint.metadata == {
|
||||
**self.metadata_2,
|
||||
"run_id": "my_run_id",
|
||||
}
|
||||
|
||||
async def test_asearch(self) -> None:
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
await saver.aput(self.config_1, self.chkpnt_1, self.metadata_1, {})
|
||||
await saver.aput(self.config_2, self.chkpnt_2, self.metadata_2, {})
|
||||
await saver.aput(self.config_3, self.chkpnt_3, self.metadata_3, {})
|
||||
|
||||
# call method / assertions
|
||||
query_1 = {"source": "input"} # search by 1 key
|
||||
query_2 = {
|
||||
"step": 1,
|
||||
"writes": {"foo": "bar"},
|
||||
} # search by multiple keys
|
||||
query_3: dict[str, Any] = {} # search by no keys, return all checkpoints
|
||||
query_4 = {"source": "update", "step": 1} # no match
|
||||
|
||||
search_results_1 = [c async for c in saver.alist(None, filter=query_1)]
|
||||
assert len(search_results_1) == 1
|
||||
assert search_results_1[0].metadata == self.metadata_1
|
||||
|
||||
search_results_2 = [c async for c in saver.alist(None, filter=query_2)]
|
||||
assert len(search_results_2) == 1
|
||||
assert search_results_2[0].metadata == self.metadata_2
|
||||
|
||||
search_results_3 = [c async for c in saver.alist(None, filter=query_3)]
|
||||
assert len(search_results_3) == 3
|
||||
|
||||
search_results_4 = [c async for c in saver.alist(None, filter=query_4)]
|
||||
assert len(search_results_4) == 0
|
||||
|
||||
# search by config (defaults to checkpoints across all namespaces)
|
||||
search_results_5 = [
|
||||
c
|
||||
async for c in saver.alist({"configurable": {"thread_id": "thread-2"}})
|
||||
]
|
||||
assert len(search_results_5) == 2
|
||||
assert {
|
||||
search_results_5[0].config["configurable"]["checkpoint_ns"],
|
||||
search_results_5[1].config["configurable"]["checkpoint_ns"],
|
||||
} == {"", "inner"}
|
||||
|
||||
# Test limit param
|
||||
search_results_6 = [
|
||||
c
|
||||
async for c in saver.alist(
|
||||
{"configurable": {"thread_id": "thread-2"}}, limit=1
|
||||
)
|
||||
]
|
||||
assert len(search_results_6) == 1
|
||||
assert search_results_6[0].config["configurable"]["thread_id"] == "thread-2"
|
||||
|
||||
# Test before param
|
||||
search_results_7 = [
|
||||
c async for c in saver.alist(None, before=search_results_5[1].config)
|
||||
]
|
||||
assert len(search_results_7) == 1
|
||||
assert search_results_7[0].config["configurable"]["thread_id"] == "thread-1"
|
||||
|
||||
async def test_limit_parameter_sql_injection_prevention(self) -> None:
|
||||
"""Test that the limit parameter properly uses parameterized queries to prevent SQL injection."""
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
# Setup: Create multiple checkpoints
|
||||
for i in range(5):
|
||||
config: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": f"thread-{i}",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
checkpoint = empty_checkpoint()
|
||||
metadata: CheckpointMetadata = {"index": i}
|
||||
await saver.aput(config, checkpoint, metadata, {})
|
||||
|
||||
# Test that limit works correctly with valid integer
|
||||
results = [c async for c in saver.alist(None, limit=2)]
|
||||
assert len(results) == 2
|
||||
|
||||
# Test that limit=0 returns no results
|
||||
results = [c async for c in saver.alist(None, limit=0)]
|
||||
assert len(results) == 0
|
||||
|
||||
# Test that limit=None returns all results
|
||||
results = [c async for c in saver.alist(None, limit=None)]
|
||||
assert len(results) == 5
|
||||
|
||||
# Test explicit SQL injection attempt via limit parameter
|
||||
# Even if type checking is bypassed and a malicious string is passed,
|
||||
# the parameterized query will treat it as a value, not SQL code
|
||||
# This would cause an error (can't convert string to int for LIMIT),
|
||||
# which is the correct secure behavior
|
||||
malicious_limits = [
|
||||
"1; DROP TABLE checkpoints; --",
|
||||
"1 OR 1=1",
|
||||
"999999 UNION SELECT * FROM checkpoints",
|
||||
]
|
||||
|
||||
for malicious_limit in malicious_limits:
|
||||
# The parameterized query should safely reject non-integer limits
|
||||
# or convert them in a way that prevents SQL injection
|
||||
try:
|
||||
# Bypass type checking by casting
|
||||
results = [
|
||||
c
|
||||
async for c in saver.alist(None, limit=malicious_limit) # type: ignore
|
||||
]
|
||||
# If it doesn't raise an error, it should at least not execute the injection
|
||||
# SQLite's parameter binding will try to convert the string to an integer
|
||||
# which will either fail or treat it as 0
|
||||
except Exception:
|
||||
# Expected: SQLite should reject invalid limit values
|
||||
pass
|
||||
|
||||
# Verify the checkpoints table still exists and has all data
|
||||
# (would have been dropped if injection succeeded)
|
||||
results = [c async for c in saver.alist(None, limit=None)]
|
||||
assert len(results) == 5
|
||||
@@ -0,0 +1,718 @@
|
||||
import asyncio
|
||||
import os
|
||||
import tempfile
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator, Generator, Iterable
|
||||
from contextlib import asynccontextmanager
|
||||
from typing import cast
|
||||
|
||||
import pytest
|
||||
from langgraph.store.base import (
|
||||
GetOp,
|
||||
Item,
|
||||
ListNamespacesOp,
|
||||
PutOp,
|
||||
SearchOp,
|
||||
)
|
||||
|
||||
from langgraph.store.sqlite import AsyncSqliteStore
|
||||
from langgraph.store.sqlite.base import SqliteIndexConfig
|
||||
from tests.test_store import CharacterEmbeddings
|
||||
|
||||
|
||||
@pytest.fixture(scope="function", params=["memory", "file"])
|
||||
async def store(request: pytest.FixtureRequest) -> AsyncIterator[AsyncSqliteStore]:
|
||||
"""Create an AsyncSqliteStore for testing."""
|
||||
if request.param == "memory":
|
||||
# In-memory store
|
||||
async with AsyncSqliteStore.from_conn_string(":memory:") as store:
|
||||
await store.setup()
|
||||
yield store
|
||||
else:
|
||||
# Temporary file store
|
||||
temp_file = tempfile.NamedTemporaryFile(delete=False)
|
||||
temp_file.close()
|
||||
try:
|
||||
async with AsyncSqliteStore.from_conn_string(temp_file.name) as store:
|
||||
await store.setup()
|
||||
yield store
|
||||
finally:
|
||||
os.unlink(temp_file.name)
|
||||
|
||||
|
||||
@pytest.fixture(scope="function")
|
||||
def fake_embeddings() -> CharacterEmbeddings:
|
||||
"""Create fake embeddings for testing."""
|
||||
return CharacterEmbeddings(dims=500)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def create_vector_store(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
conn_string: str = ":memory:",
|
||||
text_fields: list[str] | None = None,
|
||||
) -> AsyncIterator[AsyncSqliteStore]:
|
||||
"""Create an AsyncSqliteStore with vector search capabilities."""
|
||||
index_config: SqliteIndexConfig = {
|
||||
"dims": fake_embeddings.dims,
|
||||
"embed": fake_embeddings,
|
||||
"text_fields": text_fields,
|
||||
}
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
conn_string, index=index_config
|
||||
) as store:
|
||||
await store.setup()
|
||||
yield store
|
||||
|
||||
|
||||
@pytest.fixture(scope="function", params=["memory", "file"])
|
||||
def conn_string(request: pytest.FixtureRequest) -> Generator[str, None, None]:
|
||||
if request.param == "memory":
|
||||
yield ":memory:"
|
||||
else:
|
||||
temp_file = tempfile.NamedTemporaryFile(delete=False)
|
||||
temp_file.close()
|
||||
try:
|
||||
yield temp_file.name
|
||||
finally:
|
||||
os.unlink(temp_file.name)
|
||||
|
||||
|
||||
async def test_no_running_loop(store: AsyncSqliteStore) -> None:
|
||||
"""Test that sync methods raise proper errors in the main thread."""
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.put(("foo", "bar"), "baz", {"val": "baz"})
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.get(("foo", "bar"), "baz")
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.delete(("foo", "bar"), "baz")
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.search(("foo", "bar"))
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.list_namespaces(prefix=("foo",))
|
||||
with pytest.raises(asyncio.InvalidStateError):
|
||||
store.batch([PutOp(namespace=("foo", "bar"), key="baz", value={"val": "baz"})])
|
||||
|
||||
|
||||
async def test_large_batches_async(store: AsyncSqliteStore) -> None:
|
||||
"""Test processing large batch operations asynchronously."""
|
||||
N = 100
|
||||
M = 10
|
||||
coros = []
|
||||
for m in range(M):
|
||||
for i in range(N):
|
||||
coros.append(
|
||||
store.aput(
|
||||
("test", "foo", "bar", "baz", str(m % 2)),
|
||||
f"key{i}",
|
||||
value={"foo": "bar" + str(i)},
|
||||
)
|
||||
)
|
||||
coros.append(
|
||||
asyncio.create_task(
|
||||
store.aget(
|
||||
("test", "foo", "bar", "baz", str(m % 2)),
|
||||
f"key{i}",
|
||||
)
|
||||
)
|
||||
)
|
||||
coros.append(
|
||||
asyncio.create_task(
|
||||
store.alist_namespaces(
|
||||
prefix=None,
|
||||
max_depth=m + 1,
|
||||
)
|
||||
)
|
||||
)
|
||||
coros.append(
|
||||
asyncio.create_task(
|
||||
store.asearch(
|
||||
("test",),
|
||||
)
|
||||
)
|
||||
)
|
||||
coros.append(
|
||||
store.aput(
|
||||
("test", "foo", "bar", "baz", str(m % 2)),
|
||||
f"key{i}",
|
||||
value={"foo": "bar" + str(i)},
|
||||
)
|
||||
)
|
||||
coros.append(
|
||||
store.adelete(
|
||||
("test", "foo", "bar", "baz", str(m % 2)),
|
||||
f"key{i}",
|
||||
)
|
||||
)
|
||||
|
||||
results = await asyncio.gather(*coros)
|
||||
assert len(results) == M * N * 6
|
||||
|
||||
|
||||
async def test_abatch_order(store: AsyncSqliteStore) -> None:
|
||||
"""Test ordering of batch operations in async context."""
|
||||
# Setup test data
|
||||
await store.aput(("test", "foo"), "key1", {"data": "value1"})
|
||||
await store.aput(("test", "bar"), "key2", {"data": "value2"})
|
||||
|
||||
ops = [
|
||||
GetOp(namespace=("test", "foo"), key="key1"),
|
||||
PutOp(namespace=("test", "bar"), key="key2", value={"data": "value2"}),
|
||||
SearchOp(
|
||||
namespace_prefix=("test",), filter={"data": "value1"}, limit=10, offset=0
|
||||
),
|
||||
ListNamespacesOp(match_conditions=None, max_depth=None, limit=10, offset=0),
|
||||
GetOp(namespace=("test",), key="key3"),
|
||||
]
|
||||
|
||||
results = await store.abatch(
|
||||
cast(Iterable[GetOp | PutOp | SearchOp | ListNamespacesOp], ops)
|
||||
)
|
||||
assert len(results) == 5
|
||||
assert isinstance(results[0], Item)
|
||||
assert isinstance(results[0].value, dict)
|
||||
assert results[0].value == {"data": "value1"}
|
||||
assert results[0].key == "key1"
|
||||
assert results[1] is None # Put operation returns None
|
||||
assert isinstance(results[2], list)
|
||||
# SQLite query implementation might return different results
|
||||
# Just check that we get a list back and don't check the exact content
|
||||
assert isinstance(results[3], list)
|
||||
assert len(results[3]) > 0
|
||||
assert results[4] is None # Non-existent key returns None
|
||||
|
||||
# Test reordered operations
|
||||
ops_reordered = [
|
||||
SearchOp(namespace_prefix=("test",), filter=None, limit=5, offset=0),
|
||||
GetOp(namespace=("test", "bar"), key="key2"),
|
||||
ListNamespacesOp(match_conditions=None, max_depth=None, limit=5, offset=0),
|
||||
PutOp(namespace=("test",), key="key3", value={"data": "value3"}),
|
||||
GetOp(namespace=("test", "foo"), key="key1"),
|
||||
]
|
||||
|
||||
results_reordered = await store.abatch(
|
||||
cast(Iterable[GetOp | PutOp | SearchOp | ListNamespacesOp], ops_reordered)
|
||||
)
|
||||
assert len(results_reordered) == 5
|
||||
assert isinstance(results_reordered[0], list)
|
||||
assert len(results_reordered[0]) >= 2 # Should find at least our two test items
|
||||
assert isinstance(results_reordered[1], Item)
|
||||
assert results_reordered[1].value == {"data": "value2"}
|
||||
assert results_reordered[1].key == "key2"
|
||||
assert isinstance(results_reordered[2], list)
|
||||
assert len(results_reordered[2]) > 0
|
||||
assert results_reordered[3] is None # Put operation returns None
|
||||
assert isinstance(results_reordered[4], Item)
|
||||
assert results_reordered[4].value == {"data": "value1"}
|
||||
assert results_reordered[4].key == "key1"
|
||||
|
||||
|
||||
async def test_batch_get_ops(store: AsyncSqliteStore) -> None:
|
||||
"""Test GET operations in batch context."""
|
||||
# Setup test data
|
||||
await store.aput(("test",), "key1", {"data": "value1"})
|
||||
await store.aput(("test",), "key2", {"data": "value2"})
|
||||
|
||||
ops = [
|
||||
GetOp(namespace=("test",), key="key1"),
|
||||
GetOp(namespace=("test",), key="key2"),
|
||||
GetOp(namespace=("test",), key="key3"), # Non-existent key
|
||||
]
|
||||
|
||||
results = await store.abatch(ops)
|
||||
|
||||
assert len(results) == 3
|
||||
assert results[0] is not None
|
||||
assert results[1] is not None
|
||||
assert results[2] is None
|
||||
if results[0] is not None:
|
||||
assert results[0].key == "key1"
|
||||
if results[1] is not None:
|
||||
assert results[1].key == "key2"
|
||||
|
||||
|
||||
async def test_batch_put_ops(store: AsyncSqliteStore) -> None:
|
||||
"""Test PUT operations in batch context."""
|
||||
ops = [
|
||||
PutOp(namespace=("test",), key="key1", value={"data": "value1"}),
|
||||
PutOp(namespace=("test",), key="key2", value={"data": "value2"}),
|
||||
PutOp(namespace=("test",), key="key3", value=None), # Delete operation
|
||||
]
|
||||
|
||||
results = await store.abatch(ops)
|
||||
assert len(results) == 3
|
||||
assert all(result is None for result in results)
|
||||
|
||||
# Verify the puts worked
|
||||
items = await store.asearch(("test",), limit=10)
|
||||
assert len(items) == 2 # key3 had None value so wasn't stored
|
||||
|
||||
|
||||
async def test_batch_search_ops(store: AsyncSqliteStore) -> None:
|
||||
"""Test SEARCH operations in batch context."""
|
||||
# Setup test data
|
||||
await store.aput(("test", "foo"), "key1", {"data": "value1"})
|
||||
await store.aput(("test", "bar"), "key2", {"data": "value2"})
|
||||
|
||||
ops = [
|
||||
SearchOp(
|
||||
namespace_prefix=("test",), filter={"data": "value1"}, limit=10, offset=0
|
||||
),
|
||||
SearchOp(namespace_prefix=("test",), filter=None, limit=5, offset=0),
|
||||
]
|
||||
|
||||
results = await store.abatch(ops)
|
||||
|
||||
assert len(results) == 2
|
||||
# SQLite query implementation might return different results
|
||||
# Just check that we get lists back and don't check the exact content
|
||||
assert isinstance(results[0], list)
|
||||
assert isinstance(results[1], list)
|
||||
assert len(results[1]) >= 1 # We should at least find some results
|
||||
|
||||
|
||||
async def test_batch_list_namespaces_ops(store: AsyncSqliteStore) -> None:
|
||||
"""Test LIST NAMESPACES operations in batch context."""
|
||||
# Setup test data
|
||||
await store.aput(("test", "namespace1"), "key1", {"data": "value1"})
|
||||
await store.aput(("test", "namespace2"), "key2", {"data": "value2"})
|
||||
|
||||
ops = [ListNamespacesOp(match_conditions=None, max_depth=None, limit=10, offset=0)]
|
||||
|
||||
results = await store.abatch(ops)
|
||||
|
||||
assert len(results) == 1
|
||||
if isinstance(results[0], list):
|
||||
assert len(results[0]) == 2
|
||||
assert ("test", "namespace1") in results[0]
|
||||
assert ("test", "namespace2") in results[0]
|
||||
|
||||
|
||||
async def test_vector_store_initialization(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
) -> None:
|
||||
"""Test store initialization with embedding config."""
|
||||
async with create_vector_store(fake_embeddings) as store:
|
||||
assert store.index_config is not None
|
||||
assert store.index_config["dims"] == fake_embeddings.dims
|
||||
if hasattr(store.index_config.get("embed"), "embed_documents"):
|
||||
assert store.index_config["embed"] == fake_embeddings
|
||||
|
||||
|
||||
async def test_vector_insert_with_auto_embedding(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
conn_string: str,
|
||||
) -> None:
|
||||
"""Test inserting items that get auto-embedded."""
|
||||
async with create_vector_store(fake_embeddings, conn_string=conn_string) as store:
|
||||
docs = [
|
||||
("doc1", {"text": "short text"}),
|
||||
("doc2", {"text": "longer text document"}),
|
||||
("doc3", {"text": "longest text document here"}),
|
||||
("doc4", {"description": "text in description field"}),
|
||||
("doc5", {"content": "text in content field"}),
|
||||
("doc6", {"body": "text in body field"}),
|
||||
]
|
||||
|
||||
for key, value in docs:
|
||||
await store.aput(("test",), key, value)
|
||||
|
||||
results = await store.asearch(("test",), query="long text")
|
||||
assert len(results) > 0
|
||||
|
||||
doc_order = [r.key for r in results]
|
||||
assert "doc2" in doc_order
|
||||
assert "doc3" in doc_order
|
||||
|
||||
|
||||
async def test_vector_update_with_embedding(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
conn_string: str,
|
||||
) -> None:
|
||||
"""Test that updating items properly updates their embeddings."""
|
||||
async with create_vector_store(fake_embeddings, conn_string=conn_string) as store:
|
||||
await store.aput(("test",), "doc1", {"text": "zany zebra Xerxes"})
|
||||
await store.aput(("test",), "doc2", {"text": "something about dogs"})
|
||||
await store.aput(("test",), "doc3", {"text": "text about birds"})
|
||||
|
||||
results_initial = await store.asearch(("test",), query="Zany Xerxes")
|
||||
assert len(results_initial) > 0
|
||||
assert results_initial[0].score is not None
|
||||
assert results_initial[0].key == "doc1"
|
||||
initial_score = results_initial[0].score
|
||||
|
||||
await store.aput(("test",), "doc1", {"text": "new text about dogs"})
|
||||
|
||||
results_after = await store.asearch(("test",), query="Zany Xerxes")
|
||||
after_score = next((r.score for r in results_after if r.key == "doc1"), 0.0)
|
||||
assert (
|
||||
after_score is not None
|
||||
and initial_score is not None
|
||||
and after_score < initial_score
|
||||
)
|
||||
|
||||
results_new = await store.asearch(("test",), query="new text about dogs")
|
||||
for r in results_new:
|
||||
if r.key == "doc1":
|
||||
assert (
|
||||
r.score is not None
|
||||
and after_score is not None
|
||||
and r.score > after_score
|
||||
)
|
||||
|
||||
# Don't index this one
|
||||
await store.aput(
|
||||
("test",), "doc4", {"text": "new text about dogs"}, index=False
|
||||
)
|
||||
results_new = await store.asearch(
|
||||
("test",), query="new text about dogs", limit=3
|
||||
)
|
||||
assert not any(r.key == "doc4" for r in results_new)
|
||||
|
||||
|
||||
async def test_vector_search_with_filters(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
conn_string: str,
|
||||
) -> None:
|
||||
"""Test combining vector search with filters."""
|
||||
async with create_vector_store(fake_embeddings, conn_string=conn_string) as store:
|
||||
docs = [
|
||||
("doc1", {"text": "red apple", "color": "red", "score": 4.5}),
|
||||
("doc2", {"text": "red car", "color": "red", "score": 3.0}),
|
||||
("doc3", {"text": "green apple", "color": "green", "score": 4.0}),
|
||||
("doc4", {"text": "blue car", "color": "blue", "score": 3.5}),
|
||||
]
|
||||
|
||||
for key, value in docs:
|
||||
await store.aput(("test",), key, value)
|
||||
|
||||
# Vector search with filters can be inconsistent in test environments
|
||||
# Skip asserting exact results as we've already validated the functionality
|
||||
# in the synchronous tests
|
||||
_ = await store.asearch(("test",), query="apple", filter={"color": "red"})
|
||||
|
||||
# Skip asserting exact results as we've already validated the functionality
|
||||
# in the synchronous tests
|
||||
_ = await store.asearch(("test",), query="car", filter={"color": "red"})
|
||||
|
||||
# Skip asserting exact results as we've already validated the functionality
|
||||
# in the synchronous tests
|
||||
_ = await store.asearch(
|
||||
("test",), query="bbbbluuu", filter={"score": {"$gt": 3.2}}
|
||||
)
|
||||
|
||||
# Skip asserting exact results as we've already validated the functionality
|
||||
# in the synchronous tests
|
||||
_ = await store.asearch(
|
||||
("test",), query="apple", filter={"score": {"$gte": 4.0}, "color": "green"}
|
||||
)
|
||||
|
||||
|
||||
async def test_vector_search_pagination(fake_embeddings: CharacterEmbeddings) -> None:
|
||||
"""Test pagination with vector search."""
|
||||
async with create_vector_store(fake_embeddings) as store:
|
||||
for i in range(5):
|
||||
await store.aput(
|
||||
("test",), f"doc{i}", {"text": f"test document number {i}"}
|
||||
)
|
||||
|
||||
results_page1 = await store.asearch(("test",), query="test", limit=2)
|
||||
results_page2 = await store.asearch(("test",), query="test", limit=2, offset=2)
|
||||
|
||||
assert len(results_page1) == 2
|
||||
assert len(results_page2) == 2
|
||||
assert results_page1[0].key != results_page2[0].key
|
||||
|
||||
all_results = await store.asearch(("test",), query="test", limit=10)
|
||||
assert len(all_results) == 5
|
||||
|
||||
|
||||
async def test_vector_search_edge_cases(fake_embeddings: CharacterEmbeddings) -> None:
|
||||
"""Test edge cases in vector search."""
|
||||
async with create_vector_store(fake_embeddings) as store:
|
||||
await store.aput(("test",), "doc1", {"text": "test document"})
|
||||
|
||||
results = await store.asearch(("test",), query="")
|
||||
assert len(results) == 1
|
||||
|
||||
results = await store.asearch(("test",), query=None)
|
||||
assert len(results) == 1
|
||||
|
||||
long_query = "test " * 100
|
||||
results = await store.asearch(("test",), query=long_query)
|
||||
assert len(results) == 1
|
||||
|
||||
special_query = "test!@#$%^&*()"
|
||||
results = await store.asearch(("test",), query=special_query)
|
||||
assert len(results) == 1
|
||||
|
||||
|
||||
async def test_embed_with_path(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
) -> None:
|
||||
"""Test vector search with specific text fields in SQLite store."""
|
||||
async with create_vector_store(
|
||||
fake_embeddings, text_fields=["key0", "key1", "key3"]
|
||||
) as store:
|
||||
# This will have 2 vectors representing it
|
||||
doc1 = {
|
||||
# Omit key0 - check it doesn't raise an error
|
||||
"key1": "xxx",
|
||||
"key2": "yyy",
|
||||
"key3": "zzz",
|
||||
}
|
||||
# This will have 3 vectors representing it
|
||||
doc2 = {
|
||||
"key0": "uuu",
|
||||
"key1": "vvv",
|
||||
"key2": "www",
|
||||
"key3": "xxx",
|
||||
}
|
||||
await store.aput(("test",), "doc1", doc1)
|
||||
await store.aput(("test",), "doc2", doc2)
|
||||
|
||||
# doc2.key3 and doc1.key1 both would have the highest score
|
||||
results = await store.asearch(("test",), query="xxx")
|
||||
assert len(results) == 2
|
||||
assert results[0].key != results[1].key
|
||||
assert results[0].score > 0.9
|
||||
assert results[1].score > 0.9
|
||||
|
||||
# ~Only match doc2
|
||||
results = await store.asearch(("test",), query="uuu")
|
||||
assert len(results) == 2
|
||||
assert results[0].key != results[1].key
|
||||
assert results[0].key == "doc2"
|
||||
assert results[0].score > results[1].score
|
||||
|
||||
# Un-indexed - will have low results for both. Not zero (because we're projecting)
|
||||
# but less than the above.
|
||||
results = await store.asearch(("test",), query="www")
|
||||
assert len(results) == 2
|
||||
assert results[0].score < 0.9
|
||||
assert results[1].score < 0.9
|
||||
|
||||
|
||||
async def test_basic_store_ops(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
) -> None:
|
||||
"""Test vector search with specific text fields in SQLite store."""
|
||||
async with create_vector_store(
|
||||
fake_embeddings, text_fields=["key0", "key1", "key3"]
|
||||
) as store:
|
||||
uid = uuid.uuid4().hex
|
||||
namespace = (uid, "test", "documents")
|
||||
item_id = "doc1"
|
||||
item_value = {"title": "Test Document", "content": "Hello, World!"}
|
||||
results = await store.asearch((uid,))
|
||||
assert len(results) == 0
|
||||
|
||||
await store.aput(namespace, item_id, item_value)
|
||||
item = await store.aget(namespace, item_id)
|
||||
|
||||
assert item is not None
|
||||
assert item.namespace == namespace
|
||||
assert item.key == item_id
|
||||
assert item.value == item_value
|
||||
assert item.created_at is not None
|
||||
assert item.updated_at is not None
|
||||
|
||||
updated_value = {
|
||||
"title": "Updated Test Document",
|
||||
"content": "Hello, LangGraph!",
|
||||
}
|
||||
await asyncio.sleep(1.01)
|
||||
await store.aput(namespace, item_id, updated_value)
|
||||
updated_item = await store.aget(namespace, item_id)
|
||||
assert updated_item is not None
|
||||
|
||||
assert updated_item.value == updated_value
|
||||
assert updated_item.updated_at > item.updated_at
|
||||
different_namespace = (uid, "test", "other_documents")
|
||||
item_in_different_namespace = await store.aget(different_namespace, item_id)
|
||||
assert item_in_different_namespace is None
|
||||
|
||||
new_item_id = "doc2"
|
||||
new_item_value = {"title": "Another Document", "content": "Greetings!"}
|
||||
await store.aput(namespace, new_item_id, new_item_value)
|
||||
|
||||
items = await store.asearch((uid, "test"), limit=10)
|
||||
assert len(items) == 2
|
||||
assert any(item.key == item_id for item in items)
|
||||
assert any(item.key == new_item_id for item in items)
|
||||
|
||||
namespaces = await store.alist_namespaces(prefix=(uid, "test"))
|
||||
assert (uid, "test", "documents") in namespaces
|
||||
|
||||
await store.adelete(namespace, item_id)
|
||||
await store.adelete(namespace, new_item_id)
|
||||
deleted_item = await store.aget(namespace, item_id)
|
||||
assert deleted_item is None
|
||||
|
||||
deleted_item = await store.aget(namespace, new_item_id)
|
||||
assert deleted_item is None
|
||||
|
||||
empty_search_results = await store.asearch((uid, "test"), limit=10)
|
||||
assert len(empty_search_results) == 0
|
||||
|
||||
|
||||
async def test_list_namespaces(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
) -> None:
|
||||
"""Test list namespaces functionality with various filters."""
|
||||
async with create_vector_store(
|
||||
fake_embeddings, text_fields=["key0", "key1", "key3"]
|
||||
) as store:
|
||||
test_pref = str(uuid.uuid4())
|
||||
test_namespaces = [
|
||||
(test_pref, "test", "documents", "public", test_pref),
|
||||
(test_pref, "test", "documents", "private", test_pref),
|
||||
(test_pref, "test", "images", "public", test_pref),
|
||||
(test_pref, "test", "images", "private", test_pref),
|
||||
(test_pref, "prod", "documents", "public", test_pref),
|
||||
(test_pref, "prod", "documents", "some", "nesting", "public", test_pref),
|
||||
(test_pref, "prod", "documents", "private", test_pref),
|
||||
]
|
||||
|
||||
# Add test data
|
||||
for namespace in test_namespaces:
|
||||
await store.aput(namespace, "dummy", {"content": "dummy"})
|
||||
|
||||
# Test prefix filtering
|
||||
prefix_result = await store.alist_namespaces(prefix=(test_pref, "test"))
|
||||
assert len(prefix_result) == 4
|
||||
assert all(ns[1] == "test" for ns in prefix_result)
|
||||
|
||||
# Test specific prefix
|
||||
specific_prefix_result = await store.alist_namespaces(
|
||||
prefix=(test_pref, "test", "documents")
|
||||
)
|
||||
assert len(specific_prefix_result) == 2
|
||||
assert all(ns[1:3] == ("test", "documents") for ns in specific_prefix_result)
|
||||
|
||||
# Test suffix filtering
|
||||
suffix_result = await store.alist_namespaces(suffix=("public", test_pref))
|
||||
assert len(suffix_result) == 4
|
||||
assert all(ns[-2] == "public" for ns in suffix_result)
|
||||
|
||||
# Test combined prefix and suffix
|
||||
prefix_suffix_result = await store.alist_namespaces(
|
||||
prefix=(test_pref, "test"), suffix=("public", test_pref)
|
||||
)
|
||||
assert len(prefix_suffix_result) == 2
|
||||
assert all(
|
||||
ns[1] == "test" and ns[-2] == "public" for ns in prefix_suffix_result
|
||||
)
|
||||
|
||||
# Test wildcard in prefix
|
||||
wildcard_prefix_result = await store.alist_namespaces(
|
||||
prefix=(test_pref, "*", "documents")
|
||||
)
|
||||
assert len(wildcard_prefix_result) == 5
|
||||
assert all(ns[2] == "documents" for ns in wildcard_prefix_result)
|
||||
|
||||
# Test wildcard in suffix
|
||||
wildcard_suffix_result = await store.alist_namespaces(
|
||||
suffix=("*", "public", test_pref)
|
||||
)
|
||||
assert len(wildcard_suffix_result) == 4
|
||||
assert all(ns[-2] == "public" for ns in wildcard_suffix_result)
|
||||
|
||||
wildcard_single = await store.alist_namespaces(
|
||||
suffix=("some", "*", "public", test_pref)
|
||||
)
|
||||
assert len(wildcard_single) == 1
|
||||
assert wildcard_single[0] == (
|
||||
test_pref,
|
||||
"prod",
|
||||
"documents",
|
||||
"some",
|
||||
"nesting",
|
||||
"public",
|
||||
test_pref,
|
||||
)
|
||||
|
||||
# Test max depth
|
||||
max_depth_result = await store.alist_namespaces(max_depth=3)
|
||||
assert all(len(ns) <= 3 for ns in max_depth_result)
|
||||
|
||||
max_depth_result = await store.alist_namespaces(
|
||||
max_depth=4, prefix=(test_pref, "*", "documents")
|
||||
)
|
||||
assert len(set(res for res in max_depth_result)) == len(max_depth_result) == 5
|
||||
|
||||
# Test pagination
|
||||
limit_result = await store.alist_namespaces(prefix=(test_pref,), limit=3)
|
||||
assert len(limit_result) == 3
|
||||
|
||||
offset_result = await store.alist_namespaces(prefix=(test_pref,), offset=3)
|
||||
assert len(offset_result) == len(test_namespaces) - 3
|
||||
|
||||
empty_prefix_result = await store.alist_namespaces(prefix=(test_pref,))
|
||||
assert len(empty_prefix_result) == len(test_namespaces)
|
||||
assert set(empty_prefix_result) == set(test_namespaces)
|
||||
|
||||
# Clean up
|
||||
for namespace in test_namespaces:
|
||||
await store.adelete(namespace, "dummy")
|
||||
|
||||
|
||||
async def test_search_items(
|
||||
fake_embeddings: CharacterEmbeddings,
|
||||
) -> None:
|
||||
"""Test search_items functionality by calling store methods directly."""
|
||||
base = "test_search_items"
|
||||
test_namespaces = [
|
||||
(base, "documents", "user1"),
|
||||
(base, "documents", "user2"),
|
||||
(base, "reports", "department1"),
|
||||
(base, "reports", "department2"),
|
||||
]
|
||||
test_items = [
|
||||
{"title": "Doc 1", "author": "John Doe", "tags": ["important"]},
|
||||
{"title": "Doc 2", "author": "Jane Smith", "tags": ["draft"]},
|
||||
{"title": "Report A", "author": "John Doe", "tags": ["final"]},
|
||||
{"title": "Report B", "author": "Alice Johnson", "tags": ["draft"]},
|
||||
]
|
||||
|
||||
async with create_vector_store(
|
||||
fake_embeddings, text_fields=["key0", "key1", "key3"]
|
||||
) as store:
|
||||
# Insert test data
|
||||
for ns, item in zip(test_namespaces, test_items, strict=False):
|
||||
key = f"item_{ns[-1]}"
|
||||
await store.aput(ns, key, item)
|
||||
|
||||
# 1. Search documents
|
||||
docs = await store.asearch((base, "documents"))
|
||||
assert len(docs) == 2
|
||||
assert all(item.namespace[1] == "documents" for item in docs)
|
||||
|
||||
# 2. Search reports
|
||||
reports = await store.asearch((base, "reports"))
|
||||
assert len(reports) == 2
|
||||
assert all(item.namespace[1] == "reports" for item in reports)
|
||||
|
||||
# 3. Pagination
|
||||
first_page = await store.asearch((base,), limit=2, offset=0)
|
||||
second_page = await store.asearch((base,), limit=2, offset=2)
|
||||
assert len(first_page) == 2
|
||||
assert len(second_page) == 2
|
||||
keys_page1 = {item.key for item in first_page}
|
||||
keys_page2 = {item.key for item in second_page}
|
||||
assert keys_page1.isdisjoint(keys_page2)
|
||||
all_items = await store.asearch((base,))
|
||||
assert len(all_items) == 4
|
||||
|
||||
john_items = await store.asearch((base,), filter={"author": "John Doe"})
|
||||
assert len(john_items) == 2
|
||||
assert all(item.value["author"] == "John Doe" for item in john_items)
|
||||
|
||||
draft_items = await store.asearch((base,), filter={"tags": ["draft"]})
|
||||
assert len(draft_items) == 2
|
||||
assert all("draft" in item.value["tags"] for item in draft_items)
|
||||
|
||||
for ns in test_namespaces:
|
||||
key = f"item_{ns[-1]}"
|
||||
await store.adelete(ns, key)
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Run delta-channel conformance capabilities against AsyncSqliteSaver."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
pytest.importorskip(
|
||||
"langgraph.checkpoint.conformance",
|
||||
reason="langgraph-checkpoint-conformance not installed",
|
||||
)
|
||||
pytest.importorskip("aiosqlite", reason="aiosqlite not installed")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delta_channel_conformance():
|
||||
from langgraph.checkpoint.conformance import validate
|
||||
from langgraph.checkpoint.conformance.initializer import checkpointer_test
|
||||
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
|
||||
|
||||
@checkpointer_test(name="AsyncSqliteSaver")
|
||||
async def sqlite_saver():
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
yield saver
|
||||
|
||||
report = await validate(
|
||||
sqlite_saver,
|
||||
capabilities={
|
||||
"delta_channel_history",
|
||||
},
|
||||
)
|
||||
for cap, result in report.results.items():
|
||||
if result.passed is False:
|
||||
details = "\n".join(result.failures or [])
|
||||
pytest.fail(f"Capability {cap} failed:\n{details}")
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Sqlite-specific migration smoke tests: BinaryOperatorAggregate -> DeltaChannel.
|
||||
|
||||
Mirrors `libs/langgraph/tests/test_delta_channel_migration.py` (which
|
||||
covers `InMemorySaver` + a third-party fallback to the base default
|
||||
impl). This file exercises the same migration scenario through the
|
||||
sqlite-specific `SqliteSaver.get_delta_channel_history` override —
|
||||
specifically that the streaming ancestor walk finds a pre-migration
|
||||
plain `channel_values[ch]` entry and surfaces it as the `seed`, with
|
||||
post-migration writes folding on top through the reducer.
|
||||
|
||||
Pre-migration checkpoints under `BinaryOperatorAggregate` carry the
|
||||
full accumulated value at every settled super-step boundary. The
|
||||
override has to identify those as "real" seeds (not `_DeltaSnapshot`
|
||||
sentinels) — the saver layer is intentionally delta-agnostic and just
|
||||
returns whatever is stored in `channel_values[ch]`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import operator
|
||||
from typing import Annotated, Any
|
||||
|
||||
import pytest
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
# `langgraph` core isn't a dep of `langgraph-checkpoint-sqlite`. Skip the
|
||||
# whole module rather than importerror-ing in the standalone CI shape.
|
||||
pytest.importorskip("langgraph.channels.delta", reason="langgraph core not installed")
|
||||
pytest.importorskip("langgraph.channels.binop", reason="langgraph core not installed")
|
||||
pytest.importorskip("langgraph.graph", reason="langgraph core not installed")
|
||||
|
||||
from langgraph.channels.binop import BinaryOperatorAggregate # type: ignore[import-untyped] # noqa: E402,I001
|
||||
from langgraph.channels.delta import DeltaChannel # type: ignore[import-untyped] # noqa: E402
|
||||
from langgraph.graph import END, START, StateGraph # type: ignore[import-untyped] # noqa: E402
|
||||
from typing_extensions import TypedDict # noqa: E402
|
||||
|
||||
from langgraph.checkpoint.sqlite import SqliteSaver # noqa: E402
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver # noqa: E402
|
||||
|
||||
pytestmark = pytest.mark.anyio
|
||||
|
||||
|
||||
def _noop(_state: Any) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _list_concat(state: list, writes: list) -> list:
|
||||
result = list(state)
|
||||
for w in writes:
|
||||
result.extend(w if isinstance(w, list) else [w])
|
||||
return result
|
||||
|
||||
|
||||
def _binop_graph(checkpointer: Any) -> Any:
|
||||
class BinopState(TypedDict):
|
||||
items: Annotated[list, BinaryOperatorAggregate(list, operator.add)]
|
||||
|
||||
return (
|
||||
StateGraph(BinopState)
|
||||
.add_node("noop", _noop)
|
||||
.add_edge(START, "noop")
|
||||
.add_edge("noop", END)
|
||||
.compile(checkpointer=checkpointer)
|
||||
)
|
||||
|
||||
|
||||
def _delta_graph(checkpointer: Any) -> Any:
|
||||
class DeltaState(TypedDict):
|
||||
items: Annotated[list, DeltaChannel(_list_concat)]
|
||||
|
||||
return (
|
||||
StateGraph(DeltaState)
|
||||
.add_node("noop", _noop)
|
||||
.add_edge(START, "noop")
|
||||
.add_edge("noop", END)
|
||||
.compile(checkpointer=checkpointer)
|
||||
)
|
||||
|
||||
|
||||
def _drive(graph: Any, config: RunnableConfig, tag: str, n: int) -> None:
|
||||
for i in range(n):
|
||||
graph.invoke({"items": [f"{tag}{i}"]}, config)
|
||||
|
||||
|
||||
async def _adrive(graph: Any, config: RunnableConfig, tag: str, n: int) -> None:
|
||||
for i in range(n):
|
||||
await graph.ainvoke({"items": [f"{tag}{i}"]}, config)
|
||||
|
||||
|
||||
def _settled_boundaries(history: list) -> list[tuple[RunnableConfig, list]]:
|
||||
"""`(config, items)` for every checkpoint with `next == ('__start__',)`
|
||||
— the stable inter-invoke boundaries that round-trip predictably.
|
||||
"""
|
||||
return [
|
||||
(s.config, list(s.values.get("items", [])))
|
||||
for s in history
|
||||
if s.next == ("__start__",)
|
||||
]
|
||||
|
||||
|
||||
def test_migration_preserves_pre_migration_state_sync() -> None:
|
||||
"""Drive 3 invokes under `BinaryOperatorAggregate`, swap the
|
||||
annotation to `DeltaChannel` on the same sqlite-backed thread, and
|
||||
verify every settled pre-migration boundary round-trips exactly.
|
||||
|
||||
The override's streaming walk must identify the plain accumulated
|
||||
list at each pre-migration ancestor as a valid `seed` even though
|
||||
no `_DeltaSnapshot` was ever written there.
|
||||
"""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "mig-sync"}}
|
||||
|
||||
binop = _binop_graph(saver)
|
||||
_drive(binop, config, "u", 3)
|
||||
|
||||
pre_boundaries = _settled_boundaries(list(binop.get_state_history(config)))
|
||||
assert len(pre_boundaries) >= 2, "expected multiple settled boundaries"
|
||||
|
||||
delta = _delta_graph(saver)
|
||||
for cfg, items in pre_boundaries:
|
||||
snap = delta.get_state(cfg)
|
||||
assert list(snap.values.get("items", [])) == items, (
|
||||
f"snapshot mismatch at {cfg['configurable']['checkpoint_id']}: "
|
||||
f"expected {items}, got {snap.values.get('items', [])}"
|
||||
)
|
||||
|
||||
|
||||
def test_migration_continued_thread_folds_deltas_on_seed_sync() -> None:
|
||||
"""After migration, driving one more super-step extends the
|
||||
pre-migration accumulated state via the delta reducer — the seed
|
||||
plus a single new write.
|
||||
"""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "mig-continue-sync"}}
|
||||
|
||||
binop = _binop_graph(saver)
|
||||
_drive(binop, config, "u", 3)
|
||||
|
||||
pre_history = list(binop.get_state_history(config))
|
||||
pre_boundaries = _settled_boundaries(pre_history)
|
||||
# Latest settled boundary — the leaf pre-migration state.
|
||||
leaf_cfg, leaf_items = pre_boundaries[0]
|
||||
assert leaf_items, "expected non-empty pre-migration leaf"
|
||||
|
||||
delta = _delta_graph(saver)
|
||||
delta.invoke({"items": ["after-migration"]}, leaf_cfg)
|
||||
new_state = delta.get_state(config).values["items"]
|
||||
assert new_state[: len(leaf_items)] == leaf_items
|
||||
assert "after-migration" in new_state
|
||||
|
||||
|
||||
async def test_migration_preserves_pre_migration_state_async() -> None:
|
||||
"""Async equivalent of the basic-migration round-trip check on
|
||||
`AsyncSqliteSaver`."""
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "mig-async"}}
|
||||
|
||||
binop = _binop_graph(saver)
|
||||
await _adrive(binop, config, "u", 3)
|
||||
|
||||
pre_history = [s async for s in binop.aget_state_history(config)]
|
||||
pre_boundaries = _settled_boundaries(pre_history)
|
||||
assert len(pre_boundaries) >= 2
|
||||
|
||||
delta = _delta_graph(saver)
|
||||
for cfg, items in pre_boundaries:
|
||||
snap = await delta.aget_state(cfg)
|
||||
assert list(snap.values.get("items", [])) == items, (
|
||||
f"async snapshot mismatch at {cfg['configurable']['checkpoint_id']}"
|
||||
)
|
||||
@@ -0,0 +1,271 @@
|
||||
"""Smoke tests for `BaseCheckpointSaver.get_delta_channel_history` on sqlite.
|
||||
|
||||
`SqliteSaver` (and `AsyncSqliteSaver`) deliberately don't override the
|
||||
default implementation in `BaseCheckpointSaver` — these tests pin the
|
||||
default impl to behave correctly end-to-end against a real persistent
|
||||
saver and a real `DeltaChannel`-backed graph.
|
||||
|
||||
Scenarios covered:
|
||||
|
||||
* Empty `channels` argument returns an empty mapping (no I/O).
|
||||
* On a non-trivial multi-checkpoint thread, per-channel writes come back
|
||||
oldest→newest.
|
||||
* When the walk reaches the root without ever finding a stored value,
|
||||
`seed` is omitted from the entry (consumer treats absence as "start
|
||||
empty").
|
||||
* When a `_DeltaSnapshot` blob is present at an ancestor, it is returned
|
||||
as the `seed`.
|
||||
* The async saver returns the same shape via `aget_delta_channel_history`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import operator
|
||||
from typing import Annotated, Any
|
||||
|
||||
import pytest
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
# `langgraph` is not a dep of `langgraph-checkpoint-sqlite`. When tests run
|
||||
# in the sqlite lib's standalone CI environment without it installed, skip
|
||||
# the whole module rather than failing at import.
|
||||
pytest.importorskip("langgraph.channels.delta", reason="langgraph core not installed")
|
||||
pytest.importorskip("langgraph.graph", reason="langgraph core not installed")
|
||||
|
||||
from langgraph.channels.delta import DeltaChannel # type: ignore[import-untyped] # noqa: E402,I001
|
||||
from langgraph.checkpoint.serde.types import _DeltaSnapshot # noqa: E402
|
||||
from langgraph.graph import END, START, StateGraph # type: ignore[import-untyped] # noqa: E402
|
||||
from typing_extensions import TypedDict # noqa: E402
|
||||
|
||||
from langgraph.checkpoint.sqlite import SqliteSaver # noqa: E402
|
||||
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver # noqa: E402
|
||||
|
||||
pytestmark = pytest.mark.anyio
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Graph helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _noop(_state: Any) -> dict[str, Any]:
|
||||
return {}
|
||||
|
||||
|
||||
class _DeltaState(TypedDict):
|
||||
items: Annotated[list, DeltaChannel(operator.add)]
|
||||
|
||||
|
||||
def _delta_graph(checkpointer: Any) -> Any:
|
||||
return (
|
||||
StateGraph(_DeltaState)
|
||||
.add_node("noop", _noop)
|
||||
.add_edge(START, "noop")
|
||||
.add_edge("noop", END)
|
||||
.compile(checkpointer=checkpointer)
|
||||
)
|
||||
|
||||
|
||||
def _drive(graph: Any, config: RunnableConfig, n: int) -> None:
|
||||
for i in range(n):
|
||||
graph.invoke({"items": [f"v{i}"]}, config)
|
||||
|
||||
|
||||
async def _adrive(graph: Any, config: RunnableConfig, n: int) -> None:
|
||||
for i in range(n):
|
||||
await graph.ainvoke({"items": [f"v{i}"]}, config)
|
||||
|
||||
|
||||
def _pick_non_root(saver: Any, config: RunnableConfig) -> RunnableConfig:
|
||||
"""Return a config pointing at a checkpoint that has at least one ancestor.
|
||||
|
||||
`get_delta_channel_history` walks the parent chain — calling it on the root
|
||||
checkpoint produces `writes=[]` and no `seed`, which is uninteresting
|
||||
for the multi-step assertions below.
|
||||
"""
|
||||
history = list(saver.list(config))
|
||||
assert history, "expected non-empty history"
|
||||
# `list` yields newest→oldest; the second entry has the first entry
|
||||
# as its parent, so its parent_config is non-None.
|
||||
for tup in history:
|
||||
if tup.parent_config is not None:
|
||||
return tup.config
|
||||
raise AssertionError("no checkpoint with a parent in history")
|
||||
|
||||
|
||||
async def _apick_non_root(saver: Any, config: RunnableConfig) -> RunnableConfig:
|
||||
history = [tup async for tup in saver.alist(config)]
|
||||
assert history, "expected non-empty history"
|
||||
for tup in history:
|
||||
if tup.parent_config is not None:
|
||||
return tup.config
|
||||
raise AssertionError("no checkpoint with a parent in history")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sync: SqliteSaver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_empty_channels_returns_empty_mapping_sync() -> None:
|
||||
"""Empty `channels` short-circuits to `{}` without touching storage."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "empty"}}
|
||||
assert saver.get_delta_channel_history(config=config, channels=[]) == {}
|
||||
|
||||
|
||||
def test_writes_history_oldest_to_newest_sync() -> None:
|
||||
"""Per-channel writes accumulated across the walk come back oldest→newest."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "history-sync"}}
|
||||
graph = _delta_graph(saver)
|
||||
_drive(graph, config, 3)
|
||||
|
||||
target_cfg = _pick_non_root(saver, config)
|
||||
result = saver.get_delta_channel_history(config=target_cfg, channels=["items"])
|
||||
|
||||
assert "items" in result
|
||||
entry = result["items"]
|
||||
assert isinstance(entry["writes"], list)
|
||||
|
||||
# If any writes were collected, their values should be in oldest→newest
|
||||
# order — i.e. tagged 'v0', 'v1', ... matching invoke order.
|
||||
write_values: list[Any] = []
|
||||
for _task_id, channel, value in entry["writes"]:
|
||||
assert channel == "items"
|
||||
write_values.extend(value if isinstance(value, list) else [value])
|
||||
|
||||
# `_drive` invokes with payloads ['v0'], ['v1'], ['v2']. Whatever
|
||||
# subset shows up in the chain must be a contiguous prefix in order.
|
||||
for idx, val in enumerate(write_values):
|
||||
assert val == f"v{idx}", (
|
||||
f"writes not in oldest→newest order: {write_values}"
|
||||
)
|
||||
|
||||
|
||||
def test_seed_present_when_snapshot_in_ancestor_sync() -> None:
|
||||
"""Inserting a `_DeltaSnapshot` blob at an ancestor → walk returns it as `seed`."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "seed-sync"}}
|
||||
graph = _delta_graph(saver)
|
||||
_drive(graph, config, 2)
|
||||
|
||||
# Find the oldest non-root checkpoint, then walk to its parent and
|
||||
# rewrite that parent's `channel_values["items"]` to a real
|
||||
# `_DeltaSnapshot`. After this surgery, calling `get_delta_channel_history`
|
||||
# at the leaf must return the snapshot value as `seed`.
|
||||
history = list(saver.list(config))
|
||||
assert len(history) >= 2
|
||||
leaf_tup = history[0]
|
||||
# Walk to an ancestor with a parent_config (any non-root will do).
|
||||
ancestor_tup = next(
|
||||
(tup for tup in history if tup.parent_config is not None), None
|
||||
)
|
||||
assert ancestor_tup is not None
|
||||
parent_cfg = ancestor_tup.parent_config
|
||||
assert parent_cfg is not None
|
||||
parent_tup = saver.get_tuple(parent_cfg)
|
||||
assert parent_tup is not None
|
||||
|
||||
snapshot_value = ["seeded", "items"]
|
||||
parent_tup.checkpoint["channel_values"]["items"] = _DeltaSnapshot(
|
||||
snapshot_value
|
||||
)
|
||||
# Make sure the channel has a version so the optimized blob lookup
|
||||
# in any future override has something to hit.
|
||||
parent_tup.checkpoint["channel_versions"].setdefault("items", 1)
|
||||
saver.put(
|
||||
parent_tup.parent_config or {"configurable": parent_cfg["configurable"]},
|
||||
parent_tup.checkpoint,
|
||||
parent_tup.metadata,
|
||||
{},
|
||||
)
|
||||
|
||||
result = saver.get_delta_channel_history(
|
||||
config=leaf_tup.config, channels=["items"]
|
||||
)
|
||||
entry = result["items"]
|
||||
assert "seed" in entry, f"expected seed to be present, got {entry}"
|
||||
seed = entry["seed"]
|
||||
assert isinstance(seed, _DeltaSnapshot), (
|
||||
f"expected _DeltaSnapshot, got {seed!r}"
|
||||
)
|
||||
assert seed.value == snapshot_value
|
||||
|
||||
|
||||
def test_seed_omitted_when_walk_reaches_root_sync() -> None:
|
||||
"""`get_delta_channel_history` on the root checkpoint → no `seed` key, no writes."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "root-sync"}}
|
||||
graph = _delta_graph(saver)
|
||||
_drive(graph, config, 1)
|
||||
|
||||
history = list(saver.list(config))
|
||||
# Root is the oldest checkpoint (no parent_config).
|
||||
root_tup = history[-1]
|
||||
assert root_tup.parent_config is None
|
||||
|
||||
result = saver.get_delta_channel_history(
|
||||
config=root_tup.config, channels=["items"]
|
||||
)
|
||||
entry = result["items"]
|
||||
assert "seed" not in entry, f"root-walk should have no seed, got {entry}"
|
||||
assert entry["writes"] == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async: AsyncSqliteSaver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_empty_channels_returns_empty_mapping_async() -> None:
|
||||
"""Async equivalent of the empty-channels short-circuit."""
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "empty-async"}}
|
||||
assert await saver.aget_delta_channel_history(config=config, channels=[]) == {}
|
||||
|
||||
|
||||
async def test_writes_history_oldest_to_newest_async() -> None:
|
||||
"""Async equivalent of the oldest→newest ordering check."""
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "history-async"}}
|
||||
graph = _delta_graph(saver)
|
||||
await _adrive(graph, config, 3)
|
||||
|
||||
target_cfg = await _apick_non_root(saver, config)
|
||||
result = await saver.aget_delta_channel_history(
|
||||
config=target_cfg, channels=["items"]
|
||||
)
|
||||
|
||||
assert "items" in result
|
||||
entry = result["items"]
|
||||
assert isinstance(entry["writes"], list)
|
||||
|
||||
write_values: list[Any] = []
|
||||
for _task_id, channel, value in entry["writes"]:
|
||||
assert channel == "items"
|
||||
write_values.extend(value if isinstance(value, list) else [value])
|
||||
|
||||
for idx, val in enumerate(write_values):
|
||||
assert val == f"v{idx}", (
|
||||
f"writes not in oldest→newest order: {write_values}"
|
||||
)
|
||||
|
||||
|
||||
async def test_seed_omitted_when_walk_reaches_root_async() -> None:
|
||||
"""Async equivalent of the root-walk seed-absence check."""
|
||||
async with AsyncSqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {"configurable": {"thread_id": "root-async"}}
|
||||
graph = _delta_graph(saver)
|
||||
await _adrive(graph, config, 1)
|
||||
|
||||
history = [tup async for tup in saver.alist(config)]
|
||||
root_tup = history[-1]
|
||||
assert root_tup.parent_config is None
|
||||
|
||||
result = await saver.aget_delta_channel_history(
|
||||
config=root_tup.config, channels=["items"]
|
||||
)
|
||||
entry = result["items"]
|
||||
assert "seed" not in entry, f"root-walk should have no seed, got {entry}"
|
||||
assert entry["writes"] == []
|
||||
@@ -0,0 +1,309 @@
|
||||
from typing import Any, cast
|
||||
|
||||
import pytest
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
from langgraph.checkpoint.base import (
|
||||
Checkpoint,
|
||||
CheckpointMetadata,
|
||||
create_checkpoint,
|
||||
empty_checkpoint,
|
||||
)
|
||||
|
||||
from langgraph.checkpoint.sqlite import SqliteSaver
|
||||
from langgraph.checkpoint.sqlite.utils import _metadata_predicate, search_where
|
||||
|
||||
|
||||
class TestSqliteSaver:
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup(self) -> None:
|
||||
# objects for test setup
|
||||
self.config_1: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-1",
|
||||
# for backwards compatibility testing
|
||||
"checkpoint_id": "1",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
self.config_2: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_id": "2",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
self.config_3: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_id": "2-inner",
|
||||
"checkpoint_ns": "inner",
|
||||
}
|
||||
}
|
||||
|
||||
self.chkpnt_1: Checkpoint = empty_checkpoint()
|
||||
self.chkpnt_2: Checkpoint = create_checkpoint(self.chkpnt_1, {}, 1)
|
||||
self.chkpnt_3: Checkpoint = empty_checkpoint()
|
||||
|
||||
self.metadata_1: CheckpointMetadata = {
|
||||
"source": "input",
|
||||
"step": 2,
|
||||
"writes": {},
|
||||
"score": 1,
|
||||
}
|
||||
self.metadata_2: CheckpointMetadata = {
|
||||
"source": "loop",
|
||||
"step": 1,
|
||||
"writes": {"foo": "bar"},
|
||||
"score": None,
|
||||
}
|
||||
self.metadata_3: CheckpointMetadata = {}
|
||||
|
||||
def test_combined_metadata(self) -> None:
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-2",
|
||||
"checkpoint_ns": "",
|
||||
"__super_private_key": "super_private_value",
|
||||
},
|
||||
"metadata": {"run_id": "my_run_id"},
|
||||
}
|
||||
saver.put(config, self.chkpnt_2, self.metadata_2, {})
|
||||
checkpoint = saver.get_tuple(config)
|
||||
assert checkpoint is not None and checkpoint.metadata == {
|
||||
**self.metadata_2,
|
||||
"run_id": "my_run_id",
|
||||
}
|
||||
|
||||
def test_search(self) -> None:
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
# set up test
|
||||
# save checkpoints
|
||||
saver.put(self.config_1, self.chkpnt_1, self.metadata_1, {})
|
||||
saver.put(self.config_2, self.chkpnt_2, self.metadata_2, {})
|
||||
saver.put(self.config_3, self.chkpnt_3, self.metadata_3, {})
|
||||
|
||||
# call method / assertions
|
||||
query_1 = {"source": "input"} # search by 1 key
|
||||
query_2 = {
|
||||
"step": 1,
|
||||
"writes": {"foo": "bar"},
|
||||
} # search by multiple keys
|
||||
query_3: dict[str, Any] = {} # search by no keys, return all checkpoints
|
||||
query_4 = {"source": "update", "step": 1} # no match
|
||||
|
||||
search_results_1 = list(saver.list(None, filter=query_1))
|
||||
assert len(search_results_1) == 1
|
||||
assert search_results_1[0].metadata == self.metadata_1
|
||||
|
||||
search_results_2 = list(saver.list(None, filter=query_2))
|
||||
assert len(search_results_2) == 1
|
||||
assert search_results_2[0].metadata == self.metadata_2
|
||||
|
||||
search_results_3 = list(saver.list(None, filter=query_3))
|
||||
assert len(search_results_3) == 3
|
||||
|
||||
search_results_4 = list(saver.list(None, filter=query_4))
|
||||
assert len(search_results_4) == 0
|
||||
|
||||
# search by config (defaults to checkpoints across all namespaces)
|
||||
search_results_5 = list(
|
||||
saver.list({"configurable": {"thread_id": "thread-2"}})
|
||||
)
|
||||
assert len(search_results_5) == 2
|
||||
assert {
|
||||
search_results_5[0].config["configurable"]["checkpoint_ns"],
|
||||
search_results_5[1].config["configurable"]["checkpoint_ns"],
|
||||
} == {"", "inner"}
|
||||
|
||||
# search with before param
|
||||
search_results_6 = list(saver.list(None, before=search_results_5[1].config))
|
||||
assert len(search_results_6) == 1
|
||||
assert search_results_6[0].config["configurable"]["thread_id"] == "thread-1"
|
||||
|
||||
# search with limit param
|
||||
search_results_7 = list(
|
||||
saver.list({"configurable": {"thread_id": "thread-2"}}, limit=1)
|
||||
)
|
||||
assert len(search_results_7) == 1
|
||||
assert search_results_7[0].config["configurable"]["thread_id"] == "thread-2"
|
||||
|
||||
def test_search_where(self) -> None:
|
||||
# call method / assertions
|
||||
expected_predicate_1 = "WHERE json_extract(CAST(metadata AS TEXT), '$.source') = ? AND json_extract(CAST(metadata AS TEXT), '$.step') = ? AND json_extract(CAST(metadata AS TEXT), '$.writes') = ? AND json_extract(CAST(metadata AS TEXT), '$.score') = ? AND checkpoint_id < ?"
|
||||
expected_param_values_1 = ["input", 2, "{}", 1, "1"]
|
||||
assert search_where(
|
||||
None, cast(dict[str, Any], self.metadata_1), self.config_1
|
||||
) == (
|
||||
expected_predicate_1,
|
||||
expected_param_values_1,
|
||||
)
|
||||
|
||||
def test_metadata_predicate(self) -> None:
|
||||
# call method / assertions
|
||||
expected_predicate_1 = [
|
||||
"json_extract(CAST(metadata AS TEXT), '$.source') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.step') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.writes') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.score') = ?",
|
||||
]
|
||||
expected_predicate_2 = [
|
||||
"json_extract(CAST(metadata AS TEXT), '$.source') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.step') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.writes') = ?",
|
||||
"json_extract(CAST(metadata AS TEXT), '$.score') IS ?",
|
||||
]
|
||||
expected_predicate_3: list[str] = []
|
||||
|
||||
expected_param_values_1 = ["input", 2, "{}", 1]
|
||||
expected_param_values_2 = ["loop", 1, '{"foo":"bar"}', None]
|
||||
expected_param_values_3: list[Any] = []
|
||||
|
||||
assert _metadata_predicate(cast(dict[str, Any], self.metadata_1)) == (
|
||||
expected_predicate_1,
|
||||
expected_param_values_1,
|
||||
)
|
||||
assert _metadata_predicate(cast(dict[str, Any], self.metadata_2)) == (
|
||||
expected_predicate_2,
|
||||
expected_param_values_2,
|
||||
)
|
||||
assert _metadata_predicate(cast(dict[str, Any], self.metadata_3)) == (
|
||||
expected_predicate_3,
|
||||
expected_param_values_3,
|
||||
)
|
||||
|
||||
async def test_informative_async_errors(self) -> None:
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
# call method / assertions
|
||||
with pytest.raises(NotImplementedError, match="AsyncSqliteSaver"):
|
||||
await saver.aget(self.config_1)
|
||||
with pytest.raises(NotImplementedError, match="AsyncSqliteSaver"):
|
||||
await saver.aget_tuple(self.config_1)
|
||||
with pytest.raises(NotImplementedError, match="AsyncSqliteSaver"):
|
||||
async for _ in saver.alist(self.config_1):
|
||||
pass
|
||||
|
||||
def test_metadata_predicate_sql_injection_prevention(self) -> None:
|
||||
"""Test that _metadata_predicate rejects malicious filter keys."""
|
||||
# Test various SQL injection payloads
|
||||
malicious_keys = [
|
||||
"x') OR '1'='1", # Boolean-based injection
|
||||
"x') OR 1=1 --", # Comment-based injection
|
||||
"x') UNION SELECT 1,2,3,4,5,6,7 --", # UNION-based injection
|
||||
"access') = 'public' OR '1'='1' OR json_extract(value, '$.", # Complex injection
|
||||
"'; DROP TABLE checkpoints; --", # Destructive injection
|
||||
]
|
||||
|
||||
for malicious_key in malicious_keys:
|
||||
with pytest.raises(ValueError, match="Invalid filter key"):
|
||||
_metadata_predicate({malicious_key: "dummy"})
|
||||
|
||||
def test_checkpoint_search_sql_injection_prevention(self) -> None:
|
||||
"""Test that SQL injection via malicious filter keys is prevented in checkpoint search."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
# Setup: Create checkpoints with different metadata
|
||||
config_public: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-public",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
config_private: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-private",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
|
||||
checkpoint_public = empty_checkpoint()
|
||||
checkpoint_private = empty_checkpoint()
|
||||
|
||||
metadata_public: CheckpointMetadata = {
|
||||
"access": "public",
|
||||
"data": "public information",
|
||||
}
|
||||
metadata_private: CheckpointMetadata = {
|
||||
"access": "private",
|
||||
"data": "secret information",
|
||||
"password": "secret123",
|
||||
}
|
||||
|
||||
saver.put(config_public, checkpoint_public, metadata_public, {})
|
||||
saver.put(config_private, checkpoint_private, metadata_private, {})
|
||||
|
||||
# Normal query - should return only public checkpoint
|
||||
normal_results = list(saver.list(None, filter={"access": "public"}))
|
||||
assert len(normal_results) == 1
|
||||
assert normal_results[0].metadata["access"] == "public"
|
||||
|
||||
# SQL injection attempt should raise ValueError
|
||||
malicious_key = (
|
||||
"access') = 'public' OR '1'='1' OR json_extract(metadata, '$."
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="Invalid filter key"):
|
||||
list(saver.list(None, filter={malicious_key: "dummy"}))
|
||||
|
||||
def test_limit_parameter_sql_injection_prevention(self) -> None:
|
||||
"""Test that the limit parameter properly uses parameterized queries to prevent SQL injection."""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
# Setup: Create multiple checkpoints
|
||||
for i in range(5):
|
||||
config: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": f"thread-{i}",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
checkpoint = empty_checkpoint()
|
||||
metadata: CheckpointMetadata = {"index": i}
|
||||
saver.put(config, checkpoint, metadata, {})
|
||||
|
||||
# Test that limit works correctly with valid integer
|
||||
results = list(saver.list(None, limit=2))
|
||||
assert len(results) == 2
|
||||
|
||||
# Test that limit=0 returns no results
|
||||
results = list(saver.list(None, limit=0))
|
||||
assert len(results) == 0
|
||||
|
||||
# Test that limit=None returns all results
|
||||
results = list(saver.list(None, limit=None))
|
||||
assert len(results) == 5
|
||||
|
||||
def test_metadata_filter_keys_with_hyphens_and_digits(self) -> None:
|
||||
"""Metadata keys with hyphens and digit-start should be filterable.
|
||||
|
||||
This exposes incorrect JSON path handling (unquoted segments) by asserting
|
||||
that such filters successfully match saved checkpoints.
|
||||
"""
|
||||
with SqliteSaver.from_conn_string(":memory:") as saver:
|
||||
config: RunnableConfig = {
|
||||
"configurable": {
|
||||
"thread_id": "thread-hyphen-digit",
|
||||
"checkpoint_ns": "",
|
||||
}
|
||||
}
|
||||
checkpoint = empty_checkpoint()
|
||||
metadata: CheckpointMetadata = {
|
||||
"access-level": "public",
|
||||
"user": {"access-level": "nested", "123abc": "ok2"},
|
||||
"123abc": "ok",
|
||||
}
|
||||
saver.put(config, checkpoint, metadata, {})
|
||||
|
||||
# Top-level hyphenated key
|
||||
results = list(saver.list(None, filter={"access-level": "public"}))
|
||||
assert len(results) == 1
|
||||
|
||||
# Nested hyphenated key via dotted path
|
||||
results = list(saver.list(None, filter={"user.access-level": "nested"}))
|
||||
assert len(results) == 1
|
||||
|
||||
# Top-level digit-starting key
|
||||
results = list(saver.list(None, filter={"123abc": "ok"}))
|
||||
assert len(results) == 1
|
||||
|
||||
# Nested digit-starting key via dotted path
|
||||
results = list(saver.list(None, filter={"user.123abc": "ok2"}))
|
||||
assert len(results) == 1
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,429 @@
|
||||
"""Test SQLite store Time-To-Live (TTL) functionality."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
from collections.abc import Generator
|
||||
|
||||
import pytest
|
||||
from langgraph.store.base import TTLConfig
|
||||
|
||||
from langgraph.store.sqlite import SqliteStore
|
||||
from langgraph.store.sqlite.aio import AsyncSqliteStore
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_db_file() -> Generator[str, None, None]:
|
||||
"""Create a temporary database file for testing."""
|
||||
fd, path = tempfile.mkstemp()
|
||||
os.close(fd)
|
||||
yield path
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
def test_ttl_basic(temp_db_file: str) -> None:
|
||||
"""Test basic TTL functionality with synchronous API."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
with SqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes}
|
||||
) as store:
|
||||
store.setup()
|
||||
|
||||
store.put(("test",), "item1", {"value": "test"})
|
||||
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is not None
|
||||
assert item.value["value"] == "test"
|
||||
|
||||
time.sleep(ttl_seconds + 1.0)
|
||||
|
||||
store.sweep_ttl()
|
||||
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
|
||||
@pytest.mark.flaky(retries=3)
|
||||
def test_ttl_refresh(temp_db_file: str) -> None:
|
||||
"""Test TTL refresh on read."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
with SqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes, "refresh_on_read": True}
|
||||
) as store:
|
||||
store.setup()
|
||||
|
||||
# Store an item with TTL
|
||||
store.put(("test",), "item1", {"value": "test"})
|
||||
|
||||
# Sleep almost to expiration
|
||||
time.sleep(ttl_seconds - 0.5)
|
||||
swept = store.sweep_ttl()
|
||||
assert swept == 0
|
||||
|
||||
# Get the item and refresh TTL
|
||||
item = store.get(("test",), "item1", refresh_ttl=True)
|
||||
assert item is not None
|
||||
|
||||
time.sleep(ttl_seconds - 0.5)
|
||||
swept = store.sweep_ttl()
|
||||
assert swept == 0
|
||||
|
||||
# Get the item, should still be there
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is not None
|
||||
assert item.value["value"] == "test"
|
||||
|
||||
# Sleep again but don't refresh this time
|
||||
time.sleep(ttl_seconds + 0.75)
|
||||
|
||||
swept = store.sweep_ttl()
|
||||
assert swept == 1
|
||||
|
||||
# Item should be gone now
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
|
||||
def test_ttl_sweeper(temp_db_file: str) -> None:
|
||||
"""Test TTL sweeper thread."""
|
||||
ttl_seconds = 2
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
ttl_config: TTLConfig = {
|
||||
"default_ttl": ttl_minutes,
|
||||
"sweep_interval_minutes": ttl_minutes / 2,
|
||||
}
|
||||
with SqliteStore.from_conn_string(
|
||||
temp_db_file,
|
||||
ttl=ttl_config,
|
||||
) as store:
|
||||
store.setup()
|
||||
|
||||
# Start the TTL sweeper
|
||||
store.start_ttl_sweeper()
|
||||
|
||||
# Store an item with TTL
|
||||
store.put(("test",), "item1", {"value": "test"})
|
||||
|
||||
# Item should be there initially
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is not None
|
||||
|
||||
# Wait for TTL to expire and the sweeper to run
|
||||
time.sleep(ttl_seconds + (ttl_seconds / 2) + 0.5)
|
||||
|
||||
# Item should be gone now (swept automatically)
|
||||
item = store.get(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
# Stop the sweeper
|
||||
store.stop_ttl_sweeper()
|
||||
|
||||
|
||||
@pytest.mark.flaky(retries=3)
|
||||
def test_ttl_custom_value(temp_db_file: str) -> None:
|
||||
"""Test TTL with custom value per item."""
|
||||
with SqliteStore.from_conn_string(temp_db_file) as store:
|
||||
store.setup()
|
||||
|
||||
# Store items with different TTLs
|
||||
store.put(("test",), "item1", {"value": "short"}, ttl=1 / 60) # 1 second
|
||||
store.put(("test",), "item2", {"value": "long"}, ttl=3 / 60) # 3 seconds
|
||||
|
||||
# Item with short TTL
|
||||
time.sleep(2) # Wait for short TTL
|
||||
store.sweep_ttl()
|
||||
|
||||
# Short TTL item should be gone, long TTL item should remain
|
||||
item1 = store.get(("test",), "item1")
|
||||
item2 = store.get(("test",), "item2")
|
||||
assert item1 is None
|
||||
assert item2 is not None
|
||||
|
||||
# Wait for the second item's TTL
|
||||
time.sleep(4)
|
||||
store.sweep_ttl()
|
||||
|
||||
# Now both should be gone
|
||||
item2 = store.get(("test",), "item2")
|
||||
assert item2 is None
|
||||
|
||||
|
||||
@pytest.mark.flaky(retries=3)
|
||||
def test_ttl_override_default(temp_db_file: str) -> None:
|
||||
"""Test overriding default TTL at the item level."""
|
||||
with SqliteStore.from_conn_string(
|
||||
temp_db_file,
|
||||
ttl={"default_ttl": 5 / 60}, # 5 seconds default
|
||||
) as store:
|
||||
store.setup()
|
||||
|
||||
# Store an item with shorter than default TTL
|
||||
store.put(("test",), "item1", {"value": "override"}, ttl=1 / 60) # 1 second
|
||||
|
||||
# Store an item with default TTL
|
||||
store.put(("test",), "item2", {"value": "default"}) # Uses default 5 seconds
|
||||
|
||||
# Store an item with no TTL
|
||||
store.put(("test",), "item3", {"value": "permanent"}, ttl=None)
|
||||
|
||||
# Wait for the override TTL to expire
|
||||
time.sleep(2)
|
||||
store.sweep_ttl()
|
||||
|
||||
# Check results
|
||||
item1 = store.get(("test",), "item1")
|
||||
item2 = store.get(("test",), "item2")
|
||||
item3 = store.get(("test",), "item3")
|
||||
|
||||
assert item1 is None # Should be expired
|
||||
assert item2 is not None # Default TTL, should still be there
|
||||
assert item3 is not None # No TTL, should still be there
|
||||
|
||||
# Wait for default TTL to expire
|
||||
time.sleep(4)
|
||||
store.sweep_ttl()
|
||||
|
||||
# Check results again
|
||||
item2 = store.get(("test",), "item2")
|
||||
item3 = store.get(("test",), "item3")
|
||||
|
||||
assert item2 is None # Default TTL item should be gone
|
||||
assert item3 is not None # No TTL item should still be there
|
||||
|
||||
|
||||
@pytest.mark.flaky(retries=3)
|
||||
def test_search_with_ttl(temp_db_file: str) -> None:
|
||||
"""Test TTL with search operations."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
with SqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes}
|
||||
) as store:
|
||||
store.setup()
|
||||
|
||||
# Store items
|
||||
store.put(("test",), "item1", {"value": "apple"})
|
||||
store.put(("test",), "item2", {"value": "banana"})
|
||||
|
||||
# Search before expiration
|
||||
results = store.search(("test",), filter={"value": "apple"})
|
||||
assert len(results) == 1
|
||||
assert results[0].key == "item1"
|
||||
|
||||
# Wait for TTL to expire
|
||||
time.sleep(ttl_seconds + 1)
|
||||
store.sweep_ttl()
|
||||
|
||||
# Search after expiration
|
||||
results = store.search(("test",), filter={"value": "apple"})
|
||||
assert len(results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_ttl_basic(temp_db_file: str) -> None:
|
||||
"""Test basic TTL functionality with asynchronous API."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes}
|
||||
) as store:
|
||||
await store.setup()
|
||||
|
||||
# Store an item with TTL
|
||||
await store.aput(("test",), "item1", {"value": "test"})
|
||||
|
||||
# Get the item before expiration
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is not None
|
||||
assert item.value["value"] == "test"
|
||||
|
||||
# Wait for TTL to expire
|
||||
await asyncio.sleep(ttl_seconds + 1.0)
|
||||
|
||||
# Manual sweep needed without the sweeper thread
|
||||
await store.sweep_ttl()
|
||||
|
||||
# Item should be gone now
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.flaky(retries=3)
|
||||
async def test_async_ttl_refresh(temp_db_file: str) -> None:
|
||||
"""Test TTL refresh on read with async API."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes, "refresh_on_read": True}
|
||||
) as store:
|
||||
await store.setup()
|
||||
|
||||
# Store an item with TTL
|
||||
await store.aput(("test",), "item1", {"value": "test"})
|
||||
|
||||
# Sleep almost to expiration
|
||||
await asyncio.sleep(ttl_seconds - 0.5)
|
||||
|
||||
# Get the item and refresh TTL
|
||||
item = await store.aget(("test",), "item1", refresh_ttl=True)
|
||||
assert item is not None
|
||||
|
||||
# Sleep again - without refresh, would have expired by now
|
||||
await asyncio.sleep(ttl_seconds - 0.5)
|
||||
|
||||
# Get the item, should still be there
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is not None
|
||||
assert item.value["value"] == "test"
|
||||
|
||||
# Sleep again but don't refresh this time
|
||||
await asyncio.sleep(ttl_seconds + 1.0)
|
||||
|
||||
# Manual sweep
|
||||
await store.sweep_ttl()
|
||||
|
||||
# Item should be gone now
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_async_ttl_sweeper(temp_db_file: str) -> None:
|
||||
"""Test TTL sweeper thread with async API."""
|
||||
ttl_seconds = 2
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
ttl_config: TTLConfig = {
|
||||
"default_ttl": ttl_minutes,
|
||||
"sweep_interval_minutes": ttl_minutes / 2,
|
||||
}
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
temp_db_file,
|
||||
ttl=ttl_config,
|
||||
) as store:
|
||||
await store.setup()
|
||||
|
||||
# Start the TTL sweeper
|
||||
await store.start_ttl_sweeper()
|
||||
|
||||
# Store an item with TTL
|
||||
await store.aput(("test",), "item1", {"value": "test"})
|
||||
|
||||
# Item should be there initially
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is not None
|
||||
|
||||
# Wait for TTL to expire and the sweeper to run
|
||||
await asyncio.sleep(ttl_seconds + (ttl_seconds / 2) + 0.5)
|
||||
|
||||
# Item should be gone now (swept automatically)
|
||||
item = await store.aget(("test",), "item1")
|
||||
assert item is None
|
||||
|
||||
# Stop the sweeper
|
||||
await store.stop_ttl_sweeper()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.flaky(retries=3)
|
||||
async def test_async_search_with_ttl(temp_db_file: str) -> None:
|
||||
"""Test TTL with search operations using async API."""
|
||||
ttl_seconds = 1
|
||||
ttl_minutes = ttl_seconds / 60
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes}
|
||||
) as store:
|
||||
await store.setup()
|
||||
|
||||
# Store items
|
||||
await store.aput(("test",), "item1", {"value": "apple"})
|
||||
await store.aput(("test",), "item2", {"value": "banana"})
|
||||
|
||||
# Search before expiration
|
||||
results = await store.asearch(("test",), filter={"value": "apple"})
|
||||
assert len(results) == 1
|
||||
assert results[0].key == "item1"
|
||||
|
||||
# Wait for TTL to expire
|
||||
await asyncio.sleep(ttl_seconds + 1)
|
||||
await store.sweep_ttl()
|
||||
|
||||
# Search after expiration
|
||||
results = await store.asearch(("test",), filter={"value": "apple"})
|
||||
assert len(results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.flaky(retries=3)
|
||||
async def test_async_asearch_refresh_ttl(temp_db_file: str) -> None:
|
||||
"""Test TTL refresh on asearch with async API."""
|
||||
ttl_seconds = 4.0 # Increased TTL for less sensitivity to timing
|
||||
ttl_minutes = ttl_seconds / 60.0
|
||||
|
||||
async with AsyncSqliteStore.from_conn_string(
|
||||
temp_db_file, ttl={"default_ttl": ttl_minutes, "refresh_on_read": True}
|
||||
) as store:
|
||||
await store.setup()
|
||||
|
||||
namespace = ("docs", "user1")
|
||||
# t=0: items put, expire at t=4.0s
|
||||
await store.aput(namespace, "item1", {"text": "content1", "id": 1})
|
||||
await store.aput(namespace, "item2", {"text": "content2", "id": 2})
|
||||
|
||||
# t=3.0s: (after sleep ttl_seconds * 0.75 = 3s)
|
||||
await asyncio.sleep(ttl_seconds * 0.75)
|
||||
|
||||
# Perform asearch with refresh_ttl=True for item1.
|
||||
# item1's TTL should be refreshed. New expiry: t=3.0s + 4.0s = t=7.0s.
|
||||
# item2's TTL is not affected. Expires at t=4.0s.
|
||||
searched_items = await store.asearch(
|
||||
namespace, filter={"id": 1}, refresh_ttl=True
|
||||
)
|
||||
assert len(searched_items) == 1
|
||||
assert searched_items[0].key == "item1"
|
||||
|
||||
# t=5.0s: (after sleep ttl_seconds * 0.5 = 2s more. Total elapsed: 3s + 2s = 5s)
|
||||
await asyncio.sleep(ttl_seconds * 0.5)
|
||||
# At this point:
|
||||
# - item1 (refreshed by asearch) should expire at t=7.0s. Should be ALIVE.
|
||||
# - item2 (original TTL) should have expired at t=4.0s. Should be GONE after sweep.
|
||||
|
||||
await store.sweep_ttl()
|
||||
|
||||
# Check item1 (should exist due to asearch refresh)
|
||||
item1_check1 = await store.aget(namespace, "item1", refresh_ttl=False)
|
||||
assert item1_check1 is not None, (
|
||||
"Item1 should exist after asearch refresh and first sweep"
|
||||
)
|
||||
assert item1_check1.value["text"] == "content1"
|
||||
|
||||
# Check item2 (should be gone)
|
||||
item2_check1 = await store.aget(namespace, "item2", refresh_ttl=False)
|
||||
assert item2_check1 is None, (
|
||||
"Item2 should be gone after its original TTL expired"
|
||||
)
|
||||
|
||||
# t=7.5s: (after sleep ttl_seconds * 0.625 = 2.5s more. Total elapsed: 5s + 2.5s = 7.5s)
|
||||
await asyncio.sleep(ttl_seconds * 0.625)
|
||||
# At this point:
|
||||
# - item1 (refreshed by asearch, expired at t=7.0s) should be GONE after sweep.
|
||||
|
||||
await store.sweep_ttl()
|
||||
|
||||
# Check item1 again (should be gone now)
|
||||
item1_final_check = await store.aget(namespace, "item1", refresh_ttl=False)
|
||||
assert item1_final_check is None, (
|
||||
"Item1 should be gone after its refreshed TTL expired"
|
||||
)
|
||||
Generated
+1371
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user