chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,141 @@
|
||||
---
|
||||
name: blocking-io-guard
|
||||
description: Ensure async-path backend code that could block the asyncio event loop is protected by a teeth-verified runtime anchor in tests/blocking_io/. Use when changing backend Python under app/, packages/harness/deerflow/, or scripts/, when running a blocking-IO triage round over the whole repo, or when a reviewer/CI asks for blocking-IO coverage. Runs a deterministic scan (changed-lines or full-repo), routes each candidate, drafts/extends an anchor, and proves it fails when the blocking IO regresses.
|
||||
---
|
||||
|
||||
# Blocking-IO Guard Skill
|
||||
|
||||
Help a contributor ship backend async changes together with the runtime anchor
|
||||
that lets DeerFlow's blocking-IO CI gate actually see the new code. The dynamic
|
||||
detector only catches blocking IO on paths a test executes — this skill closes
|
||||
that gap, either for your own diff or for a repo-wide triage round.
|
||||
|
||||
Read `references/good-anchor-rules.md` before writing any anchor.
|
||||
Only read `references/sop-skeleton.md` when generalizing this SOP to another
|
||||
detector domain — it is not needed to execute the steps below.
|
||||
|
||||
## When to use
|
||||
|
||||
- Your change touches Python under `backend/app/`,
|
||||
`backend/packages/harness/deerflow/`, or `backend/scripts/` and may run on
|
||||
the async event loop (Mode A). If unsure, run Step 0 — it answers
|
||||
deterministically.
|
||||
- You are doing a maintenance triage round over the existing codebase
|
||||
(Mode B).
|
||||
|
||||
## SOP (router)
|
||||
|
||||
### Step 0 — Scope (deterministic)
|
||||
|
||||
**Mode A — your own diff** (default, pre-PR). From repo root:
|
||||
|
||||
```bash
|
||||
uv run --project backend python scripts/scan_changed_blocking_io.py --base origin/main
|
||||
```
|
||||
|
||||
Lists blocking-IO candidates your change introduces: findings on lines the
|
||||
diff added, **plus** findings that are new versus the merge base — the latter
|
||||
catches a new async caller exposing an old sync helper whose blocking line is
|
||||
not in the diff. The diff is `<base>...HEAD`, so **commit your work first** —
|
||||
uncommitted lines are not selected.
|
||||
|
||||
If the list is empty, this change introduces no blocking-IO surface *that the
|
||||
static detector can see in the changed files*. One residual blind spot
|
||||
remains: reachability is same-file only, so a new async caller of a sync
|
||||
helper **defined in another file** is invisible to both selections. If your
|
||||
diff adds an async call into a helper that lives elsewhere, check that helper
|
||||
manually (codegraph or `git grep`) before stopping.
|
||||
|
||||
**Mode B — full-repo triage round.** From repo root:
|
||||
|
||||
```bash
|
||||
make detect-blocking-io
|
||||
```
|
||||
|
||||
Prints a summary and writes the complete structured finding list to
|
||||
`.deer-flow/blocking-io-findings.json`. Work HIGH priority first; do not start
|
||||
MEDIUM until every HIGH is dispositioned (fixed, guarded, or recorded
|
||||
NO-ACTION).
|
||||
|
||||
**Batching policy (PR sizing).** One **fix unit** per PR while any HIGH
|
||||
remains: a fix unit is one root cause — usually a single HIGH, but two HIGHs
|
||||
resolved by the same one-place fix belong together. Once no HIGH remains,
|
||||
MEDIUM/LOW may be batched (about five per round, grouped by module or by
|
||||
disposition) so each PR stays reviewable. A new Blockbuster rule is never
|
||||
batched with anything — it always ships alone (see Step 5).
|
||||
|
||||
Both modes emit the same JSON shape per finding: `priority`, `location`
|
||||
(path/line/function), `blocking_call` (category/operation/symbol),
|
||||
`event_loop_exposure`, `reason`, `code`. Priority is a deterministic review
|
||||
ordering, not proof of a bug — Step 1 makes the actual call.
|
||||
|
||||
### Step 1 — Judge each candidate (router)
|
||||
|
||||
Read the code around each candidate and route it:
|
||||
|
||||
- **Already offloaded** (`asyncio.to_thread`, `run_in_executor`, async client) →
|
||||
**GUARD**: add/extend an anchor that locks the offload so a future edit cannot
|
||||
move it back onto the loop.
|
||||
- **On the loop, not offloaded** → **FIX+ANCHOR**: offload the production code
|
||||
(your fix), then add an anchor that guards it.
|
||||
- **Not actually exposed / acceptable** (rare: scanner false positive,
|
||||
startup-only code) → **NO-ACTION**: record one line of why.
|
||||
- **Cross-file caveat**: the scanner's async reachability is same-file only
|
||||
(`ASYNC_REACHABLE_SAME_FILE`). If the candidate is a *sync helper*, check for
|
||||
async callers in other files (codegraph or `git grep`) before deciding
|
||||
NO-ACTION.
|
||||
|
||||
### Step 2 — Apply the fix, then re-scan (FIX+ANCHOR only)
|
||||
|
||||
Offload the blocking call in production code, then re-run the Step 0 scan and
|
||||
confirm the candidate no longer appears. If the offloaded call sits in a
|
||||
`finally` / cleanup path, keep it best-effort and bounded (swallow-and-log,
|
||||
`asyncio.wait_for`) so a failing or hung cleanup cannot mask the primary
|
||||
exception. Match by the stable key
|
||||
**(path, function, symbol)** — line numbers shift after edits, so never
|
||||
compare by line.
|
||||
|
||||
- The finding must disappear. If it still shows, the fix did not remove the
|
||||
blocking pattern (e.g. the call is still a direct call, not offloaded) —
|
||||
go back before touching any test.
|
||||
- GUARD / NO-ACTION routes skip this step: a residual finding there is
|
||||
*expected* (the raw call still exists inside a sync helper with the offload
|
||||
at the caller, or the exposure was judged acceptable).
|
||||
|
||||
This is pattern-level feedback in seconds; it complements but never replaces
|
||||
Step 5 — only the runtime gate proves the event loop is actually protected.
|
||||
|
||||
### Step 3 — Check existing anchors
|
||||
|
||||
Look in `backend/tests/blocking_io/` for a test that drives the production async
|
||||
entry point reaching this candidate's branch.
|
||||
|
||||
- Covers this branch already → go to Step 5 (re-verify teeth).
|
||||
- Covers the entry point but not this branch (e.g. happy path covered,
|
||||
cleanup/404/409 not) → **extend** that anchor.
|
||||
- None → create one from `templates/anchor.template.py`.
|
||||
|
||||
### Step 4 — Generate / extend the anchor
|
||||
|
||||
Follow `references/good-anchor-rules.md`. Drive the *specific* branch (e.g. force
|
||||
the create failure that hits the cleanup `shutil.rmtree`). Never bypass the
|
||||
blocking surface with a test-only `asyncio.to_thread` wrapper.
|
||||
|
||||
### Step 5 — Verify teeth (mandatory; also the anchor-vs-rule discriminator)
|
||||
|
||||
1. Reintroduce the block (GUARD: temporarily revert the offload; FIX+ANCHOR: run
|
||||
against the pre-fix code).
|
||||
2. Run `cd backend && make test-blocking-io` (or target the one test). It **must
|
||||
go RED**.
|
||||
3. Restore the fix. It **must go GREEN**.
|
||||
|
||||
A real block that stays GREEN means Blockbuster has no rule for that
|
||||
primitive — that is the **RULE** route; see `references/good-anchor-rules.md`
|
||||
for the admission criteria before adding one.
|
||||
|
||||
### Step 6 — Deliver
|
||||
|
||||
Commit the anchor(s) with your change; `make test-blocking-io` green. In the PR,
|
||||
note: candidates found, each disposition, the re-scan result (Step 2), and
|
||||
the teeth evidence (red→green). Include the reason for any NO-ACTION. A new
|
||||
Blockbuster rule, if any, goes in its own commit with the evidence from Step 5.
|
||||
@@ -0,0 +1,65 @@
|
||||
# Good anchor rules + teeth (blocking-IO fill)
|
||||
|
||||
Distilled from `backend/docs/BLOCKING_IO_DETECTION.md`. An anchor lives in
|
||||
`backend/tests/blocking_io/`; the suite's conftest runs each test under the
|
||||
strict Blockbuster gate scoped to `app.*` / `deerflow.*`.
|
||||
|
||||
The examples in this file and in `templates/` are all filesystem-flavored.
|
||||
They demonstrate how to *write* the test, not what the SOP covers: the same
|
||||
rules apply to every category the detector reports (FILE_IO, HTTP,
|
||||
SUBPROCESS, SLEEP), and the acceptance criterion is always the teeth check
|
||||
below — never similarity to an example.
|
||||
|
||||
## A good anchor
|
||||
|
||||
- Calls the **real production async entry point** — not a low-level helper,
|
||||
unless that helper *is* the entry point production executes.
|
||||
- Does **not** bypass the blocking surface with a test-only
|
||||
`asyncio.to_thread` / `run_in_executor` wrapper.
|
||||
- Uses **real local filesystem** inputs when the bug shape is filesystem IO.
|
||||
- Mocks **only** the external dependency boundary (network service, third-party
|
||||
saver), never the offload being guarded.
|
||||
- Drives the **specific branch** you are protecting (error / cleanup / 404 /
|
||||
409), not just the happy path.
|
||||
|
||||
## Teeth (the acceptance test)
|
||||
|
||||
An anchor only counts if the gate actually fires when the code blocks:
|
||||
|
||||
1. Reintroduce the block (revert the offload, or run pre-fix code).
|
||||
2. `cd backend && make test-blocking-io` → the anchor **must fail** (RED).
|
||||
3. Restore the fix → the anchor **must pass** (GREEN).
|
||||
|
||||
A green-on-happy-path anchor with no proven red is fake coverage. Don't ship it.
|
||||
|
||||
## The RULE route (rare; strict admission criteria)
|
||||
|
||||
Blockbuster's built-in rules cover the common blocking primitives well. The
|
||||
two deliberate openings in this SOP are:
|
||||
|
||||
1. **Coverage opening** (the normal case): the rules already see the
|
||||
primitive — you only need an anchor so runtime detection executes the real
|
||||
business path and CI prevents regression.
|
||||
2. **Rule opening** (rare): you reintroduced a *real* block and the gate
|
||||
stayed GREEN — Blockbuster has no rule for that primitive.
|
||||
|
||||
A project rule lives in `_PROJECT_BLOCKING_RULES` inside
|
||||
`backend/tests/support/detectors/blocking_io_runtime.py` and changes detection
|
||||
for the **entire** blocking-IO suite — global blast radius. Admission criteria
|
||||
for adding one:
|
||||
|
||||
- You have the **fails-to-fail anchor** as evidence: a good anchor (per the
|
||||
rules above) that drives a genuinely blocking path and stays green. No
|
||||
evidence, no rule.
|
||||
- The primitive is a real blocking call (verified against its implementation
|
||||
or docs), not a false positive of the static detector.
|
||||
- The rule ships in its **own commit**, naming the primitive, the anchor that
|
||||
exposed the gap, and the suite-wide impact. Run the full
|
||||
`make test-blocking-io` suite after adding it — a new rule can turn other
|
||||
previously-green tests red, and each such red is either a real latent bug
|
||||
(fix it) or rule overreach (narrow the rule).
|
||||
- If you are not in a position to own that blast radius (e.g. external
|
||||
contributor), escalate to a maintainer with the evidence instead.
|
||||
|
||||
**Never add a runtime rule just because a path is untested** — that case needs
|
||||
an anchor, not a rule.
|
||||
@@ -0,0 +1,34 @@
|
||||
# SOP skeleton (generic shape — extraction seam)
|
||||
|
||||
This is the domain-agnostic shape the blocking-IO skill instantiates. It exists
|
||||
so a second detector/gate domain can reuse the flow without copying it. Do not
|
||||
add machinery for that until a second domain actually appears (YAGNI).
|
||||
|
||||
A domain provides:
|
||||
- a **static detector** that can scan a diff (or the whole tree) and emit
|
||||
located candidates,
|
||||
- a **CI gate** that fails when the bad pattern executes,
|
||||
- a **test location** for guard tests,
|
||||
- **good-test rules** for that gate,
|
||||
- a **teeth definition** (how to make the gate fire on purpose).
|
||||
|
||||
Steps:
|
||||
1. **Scope (deterministic):** intersect the diff's added lines with the
|
||||
detector's findings → candidates this change introduced/touched. (Or, in
|
||||
triage mode, take the full finding list ordered by priority.)
|
||||
2. **Judge (router):** per candidate — guard existing fix / fix + guard /
|
||||
no-action / rule (the gate cannot see the primitive).
|
||||
3. **Fix + re-scope (fixes only):** apply the fix, re-run the detector; the
|
||||
fixed candidate must vanish from the findings (match by a stable key, not
|
||||
line numbers). Pattern-level feedback in seconds — complements, never
|
||||
replaces, step 5.
|
||||
4. **Generate:** draft or extend a guard test per the good-test rules, driving
|
||||
the specific branch.
|
||||
5. **Verify teeth:** make the bad pattern happen → gate must fail; restore →
|
||||
gate must pass. A pattern that stays green while genuinely bad is the
|
||||
"rule" signal, not a coverage success.
|
||||
6. **Deliver:** commit the verified guard test; any gate-rule change ships in
|
||||
its own commit with the fails-to-fail evidence attached.
|
||||
|
||||
To add a domain: supply a new fill doc (like `good-anchor-rules.md`) + detector,
|
||||
and promote this file into a parent skill the instances point at.
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Template: a tests/blocking_io/ runtime anchor.
|
||||
|
||||
Copy into backend/tests/blocking_io/test_<area>.py and adapt. The suite's
|
||||
conftest already wraps every test here in the strict Blockbuster gate, so you do
|
||||
NOT import or activate the detector — just drive the real async entry point.
|
||||
|
||||
Teeth check before you commit (see references/good-anchor-rules.md):
|
||||
1. reintroduce the block -> `cd backend && make test-blocking-io` must FAIL
|
||||
2. restore the fix -> it must PASS
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# from app.<module> import <real_async_entry_point>
|
||||
|
||||
pytestmark = pytest.mark.asyncio
|
||||
|
||||
|
||||
async def test_<entry_point>_offloads_blocking_io_on_<branch>(tmp_path: Path) -> None:
|
||||
# Arrange: real inputs at the boundary the code blocks on (FS -> tmp_path;
|
||||
# HTTP/subprocess -> stub the external service). Mock ONLY the external
|
||||
# boundary, never the offload under test.
|
||||
|
||||
# Act + Assert: call the REAL production async entry point and drive the
|
||||
# specific branch you are guarding (e.g. force a failure to hit the cleanup
|
||||
# path). If the entry point performs blocking IO on the loop, the gate fails.
|
||||
# await <real_async_entry_point>(...)
|
||||
raise NotImplementedError("Replace with the real async entry point call.")
|
||||
@@ -0,0 +1,293 @@
|
||||
---
|
||||
name: deerflow-maintainer-orchestrator
|
||||
description: "Use when a DeerFlow maintainer needs comment-only GitHub issue or PR handling: resolve issue/PR scopes with gh, analyze issues, post or draft issue comments, perform PR review comments, review PR or issue batches, compare competing PRs that target the same issue, give fix strategy, risk classification, and validation guidance. Intended for maintainers and trusted local agents, not general contributors."
|
||||
---
|
||||
|
||||
# DeerFlow Maintainer Orchestrator
|
||||
|
||||
## Core Rule
|
||||
|
||||
This is a comment-plane skill: resolve GitHub scope, inspect evidence, and prepare or post DeerFlow issue comments and PR review comments. Keep the work comment-scoped; do not turn it into coding, branch management, release work, artifact closure, or other maintainer operations.
|
||||
|
||||
When the maintainer asks to process, handle, comment on, or review a bounded set of issues or PRs, proceed without asking follow-up questions. Treat that request as authorization for one public issue comment per selected non-skipped issue and one PR review comment per selected PR with high-confidence findings. If a PR has no high-confidence findings, do not post a public comment; report that result to the maintainer only. If the maintainer explicitly asks for analysis only, return comment-ready drafts without posting.
|
||||
|
||||
The maintainer's normal interaction should be: provide scope; receive posted comment URLs, PR review URLs, clean results, skipped items, failures, or drafts. Do not offload technical analysis to the maintainer. Make the best evidence-backed recommendation in the comment itself: describe the risk, impact, likely fix, and validation path. Ask the reporter or PR author for missing evidence only when the artifact lacks enough data to diagnose.
|
||||
|
||||
Output only the maintainer run result or comment draft. Do not announce the skill name, mode, or that no code was edited unless the user asks for process details.
|
||||
|
||||
Match the dominant language of the issue or PR unless the maintainer asks for another language. Chinese issue or PR text gets Chinese output; English issue or PR text gets English output. For mixed artifacts, use the body language, not logs or code.
|
||||
|
||||
## Artifact Resolution
|
||||
|
||||
Use GitHub tooling to resolve artifact type and scope. Do not ask the maintainer to clarify when `gh` or GitHub API can determine the answer.
|
||||
|
||||
1. Default repository is `bytedance/deer-flow` unless a URL or explicit repo says otherwise.
|
||||
2. For URLs, route `/issues/<number>` to Issue Flow and `/pull/<number>` to PR Review Flow.
|
||||
3. For typed numbers, use the typed command:
|
||||
- Issue: `gh issue view <number> --repo <repo> --json number,title,url,state,body,labels,author,comments`
|
||||
- PR: `gh pr view <number> --repo <repo> --json number,title,url,state,body,author,files,comments,reviews,statusCheckRollup,baseRefName,headRefName`
|
||||
4. Normalize multiple explicit references such as `#123`, `# 123`, and bare `123` into a number list, preserving order and de-duplicating exact repeats.
|
||||
5. For untyped numbers, try `gh pr view <number> --repo <repo> --json number,url` first. If it fails, use `gh issue view <number> --repo <repo> --json number,url`. Do not ask which type it is.
|
||||
6. For issue batches, use `gh issue list`, not the mixed GitHub issues endpoint. For PR batches, use `gh pr list`.
|
||||
7. Respect maintainer-provided count or time window. There is no hard five-item cap. If the scope is broad and underspecified, choose a practical recent slice, state the slice used, prioritize newest and highest-risk items, and report any unprocessed remainder.
|
||||
8. For "recent/latest" wording without a count, use a small default recent slice. For "recent hours" wording without a number, use six hours. Do not ask.
|
||||
9. Use `gh api` when `gh issue/pr view/list` lacks required fields such as timeline events, review threads, or precise search filters.
|
||||
10. Use GitHub search only as a fallback for natural-language filters that cannot be represented by view/list/API calls. Do not use web search for artifact routing unless GitHub tooling is unavailable.
|
||||
11. When an issue has more than one candidate resolving PR, gather them all before reviewing: the issue's linked/Development PRs, closing keywords (`Closes/Fixes #<issue>`) found via `gh api` timeline cross-reference events, and PRs that mention the issue. Route them into Competing PR Comparison.
|
||||
12. If no artifact type, number, URL, count, time window, or searchable GitHub scope can be resolved, stop with a compact "scope unresolved" report. Do not ask a follow-up question.
|
||||
|
||||
Use concise repo-local references such as `#123` and `PR #123` in maintainer reports and comments. Include full GitHub URLs only for posted comment/review links returned by GitHub or when the maintainer supplied an explicit URL.
|
||||
|
||||
## Existing Coverage and Re-Runs
|
||||
|
||||
Existing comments suppress duplicate **posting**, not **analysis**. Always analyze the artifact in full, then post only the net-new delta over what is already covered.
|
||||
|
||||
1. Read existing maintainer/trusted-agent comments and reviews as prior coverage.
|
||||
2. Analyze the artifact fully regardless of what already exists. A prior comment may be partial — catching A while missing B.
|
||||
3. Keep only net-new, high-confidence items not already materially covered.
|
||||
4. Non-empty delta: post one comment that explicitly builds on the prior coverage (for example `Adding to @reviewer's review:`) and states only the new items. Do not restate covered points.
|
||||
5. Empty delta: post nothing public; report `Already covered` to the maintainer with the existing comment/review URL.
|
||||
6. Idempotency: treat your own earlier skill-authored comments as already-covered. On a re-run, never stack a second comment that repeats an earlier one — post only genuinely new delta, or nothing.
|
||||
|
||||
RFC issues are the one hard skip: no analysis and no post unless the maintainer overrides.
|
||||
|
||||
## Issue Flow
|
||||
|
||||
Use Issue Flow for GitHub issues, bug reports, feature requests, support questions, and issue batches.
|
||||
|
||||
Start every issue with a cheap precheck:
|
||||
|
||||
1. Fetch issue metadata, labels, author, body, and existing comments.
|
||||
2. If labels, title, or body mark the issue as RFC (`rfc`, `[RFC]`, `RFC:`, or `Request for Comments`), classify it as `rfc-no-comment`, skip deep analysis, and do not post anything public unless the maintainer explicitly overrides the RFC skip for that item.
|
||||
3. Existing maintainer or trusted-agent comments are prior coverage, not an automatic skip. Analyze fully and post only the net-new delta (see Existing Coverage and Re-Runs).
|
||||
4. Treat ordinary reporter replies, thanks, unrelated discussion, or incomplete guesses as non-blocking.
|
||||
5. Report already-covered or skipped issues to the maintainer only as compact identifiers plus the reason or existing comment URL when available.
|
||||
|
||||
For non-skipped issues:
|
||||
|
||||
1. Read enough context to avoid guessing: issue body, comments, screenshots, logs, reproduction details, linked artifacts, and relevant DeerFlow code/docs.
|
||||
2. Classify the surface:
|
||||
- Frontend UI
|
||||
- Backend API
|
||||
- Agents / LangGraph
|
||||
- Sandbox
|
||||
- Skills
|
||||
- MCP
|
||||
- Dependencies
|
||||
- Default behavior
|
||||
- Docs / tests / CI only
|
||||
3. Classify actionability:
|
||||
- `ready-to-fix`: bounded, evidence sufficient, validation path clear.
|
||||
- `needs-more-evidence`: repro, logs, environment, screenshots, exact expected behavior, or failing case missing.
|
||||
- `defer-or-close`: duplicate, stale, unsupported, unactionable, or out of scope.
|
||||
- `rfc-no-comment`: RFC issue; skip public comments by default.
|
||||
4. Produce a public-safe comment from the analysis, not the analysis labels:
|
||||
- Start with one natural opener that connects to the issue context. Prefer `Thanks @author.` for reporter-authored issues when it reads naturally; omit the mention for bots, maintainer-authored tracking issues, or cases where it would add noise.
|
||||
- The opener must say something specific about the next step or boundary, not a generic assessment. Do not use generic phrases such as "This is actionable", "I would treat this as", "ready to fix", or surface/actionability/risk labels.
|
||||
- Use the smallest stable template that fits:
|
||||
|
||||
```text
|
||||
Thanks @author. <one specific sentence that frames the fix, investigation, or missing evidence.>
|
||||
|
||||
Recommended solution:
|
||||
- ...
|
||||
|
||||
Validation:
|
||||
- ...
|
||||
```
|
||||
|
||||
- Add `Evidence:` only when citing concrete code, logs, reproduction details, or other proof helps the author act.
|
||||
- Add `Risk:` only when architecture, security, public API, default behavior, or compatibility impact must be called out explicitly; make the risk specific.
|
||||
- Add `Missing info:` only when the issue cannot be diagnosed without more evidence; ask for the smallest useful data.
|
||||
- Put relevant files/components inside `Evidence:` or `Recommended solution:` bullets instead of separate metadata fields.
|
||||
- Every posted issue comment should contain concrete modification guidance and validation guidance unless the only useful response is `Missing info:`.
|
||||
5. Immediately before posting, refresh comments; fold any equivalent comment that appeared during analysis into prior coverage and post only the remaining delta.
|
||||
6. Post one issue comment when posting is authorized; otherwise return the same text as `Reply draft`.
|
||||
|
||||
Do not expose private reasoning, credentials, internal-only context, or unsupported promises. Do not say a fix was made unless a separate coding workflow actually changed code.
|
||||
|
||||
## PR Review Flow
|
||||
|
||||
Use PR Review Flow for GitHub pull requests and PR batches.
|
||||
|
||||
Start every PR with a cheap precheck:
|
||||
|
||||
1. Fetch PR metadata, changed file list, checks summary, existing PR reviews, existing PR comments, and review threads when available.
|
||||
2. Existing maintainer or trusted-agent reviews are prior coverage, not an automatic skip. Review fully and post only the net-new delta (see Existing Coverage and Re-Runs).
|
||||
3. Read `statusCheckRollup` as signal, not verdict. Failing required checks are themselves a reportable finding (build failure = P0; failing tests or lint = P1/P2 by impact). Green checks lower risk but never excuse reading the actual changed code path — confirm suspect logic by reading the source, not by trusting green CI. Tests passing does not prove the changed branch is exercised.
|
||||
4. Treat author replies, thanks, unrelated discussion, or incomplete guesses as non-blocking.
|
||||
5. Report already-covered or clean PRs to the maintainer only, with the existing review/comment URL when available.
|
||||
|
||||
### Diff Base Rule
|
||||
|
||||
Before reviewing a local PR branch or local diff, fetch the base repository's target branch and compare against that fresh remote-tracking ref, not a possibly stale local `main`.
|
||||
|
||||
- For fork checkouts, prefer `upstream/<base-branch>` when `upstream` points to the base repository.
|
||||
- For direct upstream checkouts, use the base remote's fetched branch, usually `origin/<base-branch>`.
|
||||
- Prefer GitHub PR base metadata for the target branch. For non-PR local diffs, use the base repository default branch. If metadata is unavailable, default to `main` only after fetching the base remote.
|
||||
- Refresh the comparison ref explicitly, for example `git fetch <base-remote> +refs/heads/<base-branch>:refs/remotes/<base-remote>/<base-branch>`, then inspect `BASE=$(git merge-base HEAD <base-remote>/<base-branch>)` and `git diff "$BASE"...HEAD`.
|
||||
- If using `FETCH_HEAD` from a single-branch fetch instead, diff against that verified `FETCH_HEAD` immediately and do not later substitute a possibly stale remote-tracking ref.
|
||||
- Resolve the PR head explicitly. For fork PRs whose head branch is not on the base repo, fetch the PR ref: `git fetch <base-remote> pull/<n>/head:pr-<n>`. The fork's own branch ref and `gh api .../contents?ref=<fork-branch>` will 404 against the base repo. Record the head SHA you reviewed.
|
||||
- Re-check the head SHA immediately before posting. If the PR head moved during analysis, re-review the new diff or abort — never post a review against a diff the PR no longer has.
|
||||
- For uncommitted local changes, review committed branch changes against the fresh base first, then include working-tree changes separately.
|
||||
- If the base remote or base branch cannot be established, use the GitHub PR files/diff as the source of truth. If neither local nor GitHub diff can be read, return a compact failure report and do not post a review.
|
||||
|
||||
Before posting a PR review comment:
|
||||
|
||||
1. Review only the current diff against the fresh base and changed files. Do not comment on unrelated pre-existing code unless the diff makes it newly risky.
|
||||
2. Do not report low-confidence guesses. If evidence is insufficient, omit the finding.
|
||||
3. Prioritize correctness, safety, maintainability, production risk, compatibility, and missing critical tests over style.
|
||||
4. Report concrete architecture, security, public API, default-behavior, and compatibility problems as findings when the diff causes or exposes them.
|
||||
5. Check changed behavior, edge cases, error paths, state mutation, transactions, locks, cache invalidation, cleanup, security boundaries, missing tests, performance/reliability, and API compatibility.
|
||||
6. Immediately before posting, refresh reviews/comments and fold any equivalent review that appeared during analysis into prior coverage; post only the remaining delta.
|
||||
7. Apply the Posting Gate. If the gate yields public findings, post one PR review comment in the PR language. Otherwise post nothing public and report the result (`No high-confidence review findings.` or `Already covered`) plus any sub-threshold items as `Maintainer notes`.
|
||||
|
||||
For public PR reviews with findings, start with one short opener that fits the review context and matches the finding count. Use singular wording only for exactly one finding, for example `Thanks @author. I found one issue that should be addressed before this is ready.` Use plural wording for multiple findings, for example `Thanks @author. I found a few issues that should be addressed before this is ready.` Omit the mention for bots or when it adds noise.
|
||||
|
||||
For each finding, use:
|
||||
|
||||
```text
|
||||
[P0/P1/P2] Title
|
||||
|
||||
- Location: file and line/range
|
||||
- Problem: what can go wrong
|
||||
- Evidence: why the diff causes it
|
||||
- Suggested fix: concrete minimal fix
|
||||
- Test: what test should cover it
|
||||
```
|
||||
|
||||
Severity guide:
|
||||
|
||||
- `P0`: causes outage, data loss, security breach, or build failure.
|
||||
- `P1`: likely production bug, serious regression, broken compatibility, or high-risk security/architecture issue.
|
||||
- `P2`: correctness, maintainability, or test concern with lower risk.
|
||||
|
||||
### Posting Gate
|
||||
|
||||
Posting depends on BOTH confidence (is the problem real?) and severity (how bad if real). They are independent axes — "no high-confidence findings" means none across P0/P1/P2, not merely "no P0".
|
||||
|
||||
- Post publicly only items that are high-confidence AND at least P2.
|
||||
- For a public P2, additionally require that the diff itself introduces or worsens the issue. Do not raise a public P2 for pre-existing behavior the diff only touches, or for a change that is a net improvement over the prior state.
|
||||
- A high-confidence P0/P1 is always worth posting. A low-confidence P1 is not — omit it, or route it to `Maintainer notes` framed as a hypothesis to verify.
|
||||
- Sub-threshold but real observations (net-improvement nits, bounded or low-risk concerns, pre-existing issues, low-confidence hypotheses) go to the `Maintainer notes` channel in the run result, never to a public comment.
|
||||
|
||||
Do not produce compliments, summaries, or general advice. For sensitive security issues, describe impact and remediation without exploit instructions.
|
||||
|
||||
## Batch Handling
|
||||
|
||||
When the scope has multiple artifacts, cluster before reviewing and synthesize after.
|
||||
|
||||
Cluster by relatedness, not by type. Group artifacts that share files, interfaces, or the same issue/feature into one cluster; same-type artifacts that touch disjoint files are independent.
|
||||
|
||||
- Related cluster: review in ONE shared context so cross-artifact reasoning is possible — parallel agents cannot see each other's findings. If it cannot fit one context, fan out per sub-group and reconcile in the synthesis pass; never split it blind, without that re-aggregation.
|
||||
- Independent clusters: may run in parallel. Offloading a large or independent batch to one subagent per cluster keeps the main context clean — consider it for big batches, prefer offering it to the maintainer over silently spawning, and do not spawn for two or three related items or when the cold-start cost is not earned.
|
||||
|
||||
After per-artifact review, run one synthesis pass over the whole batch and report it to the maintainer (decision-support, not a public comment):
|
||||
|
||||
- Overlapping files and merge-order/conflict surface — which PRs touch the same files and will conflict pairwise.
|
||||
- Duplicate or competing solutions to the same problem.
|
||||
- Composition risk — changes each safe alone but interacting (for example, two PRs editing the same module or table).
|
||||
|
||||
## Competing PR Comparison
|
||||
|
||||
When several PRs target the same issue, compare them instead of reviewing each in isolation.
|
||||
|
||||
1. Pull the issue's acceptance criteria (reported problem and expected behavior); that is the rubric anchor.
|
||||
2. Score each PR on: does it actually resolve the issue's ask; correctness and edge/error-path coverage; test quality; blast radius and compatibility; maintainability. Use the same DeerFlow Review Heuristics and Posting Gate as a single review.
|
||||
3. Report a maintainer-facing comparison — strongest PR and why, what each is missing — in the run result.
|
||||
4. Keep the public surface constructive and per-PR: post each PR's own gate-passing findings normally. Do not publicly rank PRs against each other or tell an author their PR is worse than a competitor's; winner selection stays in the maintainer report.
|
||||
|
||||
## No-Question Policy
|
||||
|
||||
Do not ask the maintainer routine clarification questions. The skill should save maintainer time by turning scope into comments through a fixed workflow.
|
||||
|
||||
Stop without asking only when:
|
||||
|
||||
- no issue/PR scope can be resolved through URLs, numbers, `gh` view/list, `gh api`, or GitHub search fallback;
|
||||
- GitHub authentication, repository access, or comment posting fails;
|
||||
- the requested action is outside comment-only scope;
|
||||
- posting would require private credentials, private security details, or non-public context.
|
||||
|
||||
In these cases, return a compact failure report with the attempted command path and the smallest next action. Do not phrase it as a question unless the maintainer explicitly asked to be prompted.
|
||||
|
||||
## DeerFlow Review Heuristics
|
||||
|
||||
Treat these as high-signal areas for issue comments and PR findings:
|
||||
|
||||
- `backend/packages/harness/deerflow/` must not import `app.*`.
|
||||
- App may depend on harness; harness must stay publishable and app-agnostic.
|
||||
- Frontend thread/message behavior and Gateway/LangGraph-compatible SSE are contract surfaces.
|
||||
- Sandbox permissions, bash/file-write tools, skill installation, and remote execution are security-sensitive.
|
||||
- Default model/provider behavior, config migration, persistence schema, public API/SSE, and LangGraph thread/run lifecycle are compatibility-sensitive.
|
||||
- Runtime docs should track user-facing or developer-facing behavior changes.
|
||||
- Security-sensitive comments should provide proof and remediation, not vague assertions.
|
||||
|
||||
## Validation Guidance
|
||||
|
||||
Recommend the checks matching the touched surface:
|
||||
|
||||
| Surface | Suggested validation |
|
||||
| --- | --- |
|
||||
| Backend API / harness / agents / MCP / skills runtime | `cd backend && make lint && make test` |
|
||||
| Blocking IO or async file/network work | `cd backend && make test-blocking-io` or a focused blocking-IO regression |
|
||||
| Harness/app boundary | `cd backend && uv run pytest tests/test_harness_boundary.py` |
|
||||
| Frontend UI/core | `cd frontend && pnpm format && pnpm lint && pnpm typecheck && BETTER_AUTH_SECRET=local-dev-secret pnpm build && make test` |
|
||||
| Front/back thread or SSE contract | backend replay golden and full-stack replay render where feasible |
|
||||
| Frontend user workflow | Playwright E2E or browser proof with screenshot/DOM assertion |
|
||||
| Docker/sandbox/provisioner | focused backend tests plus Docker/provisioner smoke when feasible |
|
||||
| Docs-only | targeted markdown review |
|
||||
|
||||
## Output
|
||||
|
||||
For Issue Flow:
|
||||
|
||||
```text
|
||||
Run result:
|
||||
Posted:
|
||||
Skipped:
|
||||
Already covered:
|
||||
Failed:
|
||||
Maintainer notes:
|
||||
Per issue:
|
||||
Issue:
|
||||
Surface:
|
||||
Actionability:
|
||||
Risk:
|
||||
Comment:
|
||||
Validation:
|
||||
Comment status:
|
||||
```
|
||||
|
||||
For PR Review Flow:
|
||||
|
||||
```text
|
||||
Run result:
|
||||
Reviewed:
|
||||
Skipped:
|
||||
Clean:
|
||||
Already covered:
|
||||
Failed:
|
||||
Maintainer notes:
|
||||
Per PR:
|
||||
PR:
|
||||
Public review:
|
||||
Findings:
|
||||
Review status:
|
||||
```
|
||||
|
||||
For analysis-only requests, replace `Posted`/`Reviewed` with `Drafted` and include the comment/review text without posting.
|
||||
|
||||
For batches, prefer a compact maintainer-facing table after the headline counts:
|
||||
|
||||
```text
|
||||
| Artifact | Status | Public action | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| #123 | posted | comment URL | short reason |
|
||||
| PR #456 | reviewed | review URL | P1: finding title |
|
||||
| PR #789 | clean | none | No high-confidence review findings. |
|
||||
| #321 | already covered | none | existing maintainer comment |
|
||||
```
|
||||
|
||||
For multi-artifact batches, follow the table with a `Batch synthesis` block (overlapping files, merge-order/conflict surface, duplicate or competing solutions, composition risk) and, when issues had competing PRs, a `Competing PR comparison` block. Both are maintainer-only.
|
||||
|
||||
Omit empty categories, no-op fields, routine command output, and raw logs. Report meaningful changes, evidence, and options.
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
name: smoke-test
|
||||
description: End-to-end smoke test skill for DeerFlow. Guides through: 1) Pulling latest code, 2) Docker OR Local installation and deployment (user preference, default to Local if Docker network issues), 3) Service availability verification, 4) Health check, 5) Final test report. Use when the user says "run smoke test", "smoke test deployment", "verify installation", "test service availability", "end-to-end test", or similar.
|
||||
---
|
||||
|
||||
# DeerFlow Smoke Test Skill
|
||||
|
||||
This skill guides the Agent through DeerFlow's full end-to-end smoke test workflow, including code updates, deployment (supporting both Docker and local installation modes), service availability verification, and health checks.
|
||||
|
||||
## Deployment Mode Selection
|
||||
|
||||
This skill supports two deployment modes:
|
||||
- **Local installation mode** (recommended, especially when network issues occur) - Run all services directly on the local machine
|
||||
- **Docker mode** - Run all services inside Docker containers
|
||||
|
||||
**Selection strategy**:
|
||||
- If the user explicitly asks for Docker mode, use Docker
|
||||
- If network issues occur (such as slow image pulls), automatically switch to local mode
|
||||
- Default to local mode whenever possible
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
smoke-test/
|
||||
├── SKILL.md ← You are here - core workflow and logic
|
||||
├── scripts/
|
||||
│ ├── check_docker.sh ← Check the Docker environment
|
||||
│ ├── check_local_env.sh ← Check local environment dependencies
|
||||
│ ├── frontend_check.sh ← Frontend page smoke check
|
||||
│ ├── pull_code.sh ← Pull the latest code
|
||||
│ ├── deploy_docker.sh ← Docker deployment
|
||||
│ ├── deploy_local.sh ← Local deployment
|
||||
│ └── health_check.sh ← Service health check
|
||||
├── references/
|
||||
│ ├── SOP.md ← Standard operating procedure
|
||||
│ └── troubleshooting.md ← Troubleshooting guide
|
||||
└── templates/
|
||||
├── report.local.template.md ← Local mode smoke test report template
|
||||
└── report.docker.template.md ← Docker mode smoke test report template
|
||||
```
|
||||
|
||||
## Standard Operating Procedure (SOP)
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
1. **Confirm current directory** - Verify that the current working directory is the DeerFlow project root
|
||||
2. **Check Git status** - See whether there are uncommitted changes
|
||||
3. **Pull the latest code** - Use `git pull origin main` to get the latest updates
|
||||
4. **Confirm code update** - Verify that the latest code was pulled successfully
|
||||
|
||||
### Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
**Choose deployment mode**:
|
||||
- Ask for user preference, or choose automatically based on network conditions
|
||||
- Default to local installation mode
|
||||
|
||||
**Local mode environment check**:
|
||||
1. **Check Node.js version** - Requires 22+
|
||||
2. **Check pnpm** - Package manager
|
||||
3. **Check uv** - Python package manager
|
||||
4. **Check nginx** - Reverse proxy
|
||||
5. **Check required ports** - Confirm that ports 2026, 3000, and 8001 are not occupied
|
||||
|
||||
**Docker mode environment check** (if Docker is selected):
|
||||
1. **Check whether Docker is installed** - Run `docker --version`
|
||||
2. **Check Docker daemon status** - Run `docker info`
|
||||
3. **Check Docker Compose availability** - Run `docker compose version`
|
||||
4. **Check required ports** - Confirm that port 2026 is not occupied
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
1. **Check whether config.yaml exists**
|
||||
- If it does not exist, run `make config` to generate it
|
||||
- If it already exists, check whether it needs an upgrade with `make config-upgrade`
|
||||
2. **Check the .env file**
|
||||
- Verify that required environment variables are configured
|
||||
- Especially model API keys such as `OPENAI_API_KEY`
|
||||
|
||||
### Phase 4: Deployment Execution
|
||||
|
||||
**Local mode deployment**:
|
||||
1. **Check dependencies** - Run `make check`
|
||||
2. **Install dependencies** - Run `make install`
|
||||
3. **(Optional) Pre-pull the sandbox image** - If needed, run `make setup-sandbox`
|
||||
4. **Start services** - Run `make start` (production mode)
|
||||
5. **Wait for startup** - Give all services enough time to start completely (90-120 seconds recommended)
|
||||
|
||||
**Docker mode deployment** (if Docker is selected):
|
||||
1. **Initialize Docker environment** - Run `make docker-init`
|
||||
2. **Start Docker services** - Run `make up` (production mode)
|
||||
3. **Wait for startup** - Give all containers enough time to start completely (60 seconds recommended)
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
**Local mode health check**:
|
||||
1. **Check process status** - Confirm that Gateway, Frontend, and Nginx processes are all running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph-compatible API** - Verify the `/api/langgraph/*` route exposed by Gateway
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`. The script auto-detects whether authentication is enabled and, if so, registers / logs in a smoke-test user so the real pages are verified rather than the login redirect.
|
||||
|
||||
**Docker mode health check** (when using Docker):
|
||||
1. **Check container status** - Run `docker ps` and confirm that all containers are running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph-compatible API** - Verify the `/api/langgraph/*` route exposed by Gateway
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`. The script auto-detects whether authentication is enabled and, if so, registers / logs in a smoke-test user so the real pages are verified rather than the login redirect.
|
||||
|
||||
### Optional Functional Verification
|
||||
|
||||
1. **List available models** - Verify that model configuration loads correctly
|
||||
2. **List available skills** - Verify that the skill directory is mounted correctly
|
||||
3. **Simple chat test** - Send a simple message to verify the end-to-end flow
|
||||
|
||||
### Phase 6: Generate Test Report
|
||||
|
||||
1. **Collect all test results** - Summarize execution status for each phase
|
||||
2. **Record encountered issues** - If anything fails, record the error details
|
||||
3. **Generate the final report** - Use the template that matches the selected deployment mode to create the complete test report, including overall conclusion, detailed key test cases, and explicit frontend page / route results
|
||||
4. **Provide follow-up recommendations** - Offer suggestions based on the test results
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- **Follow the sequence** - Execute strictly in the order described above
|
||||
- **Idempotency** - Every step should be safe to repeat
|
||||
- **Error handling** - If a step fails, stop and report the issue, then provide troubleshooting suggestions
|
||||
- **Detailed logging** - Record the execution result and status of each step
|
||||
- **User confirmation** - Ask for confirmation before potentially risky operations such as overwriting config
|
||||
- **Mode preference** - Prefer local mode to avoid network-related issues
|
||||
- **Template requirement** - The final report must use the matching template under `templates/`; do not output a free-form summary instead of the template-based report
|
||||
- **Report clarity** - The execution summary must include the overall pass/fail conclusion plus per-case result explanations, and frontend smoke check results must be listed explicitly in the report
|
||||
- **Optional phase handling** - If functional verification is not executed, do not present it as a separate skipped phase in the final report
|
||||
|
||||
## Known Acceptable Warnings
|
||||
|
||||
The following warnings can appear during smoke testing and do not block a successful result:
|
||||
- Feishu/Lark SSL errors in Gateway logs (certificate verification failure) can be ignored if that channel is not enabled
|
||||
- Warnings in Gateway logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
|
||||
- The `frontend_check.sh` script automatically handles authentication. When auth is enabled it registers / logs in a smoke-test user (`smoke-test@deerflow.dev` by default) to verify the real `/workspace/*` pages. The registration may produce a log entry from the auth provider, which is expected and harmless.
|
||||
|
||||
## Key Tools
|
||||
|
||||
Use the following tools during execution:
|
||||
|
||||
1. **bash** - Run shell commands
|
||||
2. **present_file** - Show generated reports and important files
|
||||
3. **task_tool** - Organize complex steps with subtasks when needed
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Smoke test pass criteria (local mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Local environment check passes (Node.js 22+, pnpm, uv, nginx)
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make check` passes
|
||||
- [x] `make install` completes successfully
|
||||
- [x] `make dev` starts successfully
|
||||
- [x] All service processes run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
Smoke test pass criteria (Docker mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Docker environment check passes
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make docker-init` completes successfully
|
||||
- [x] `make up` completes successfully
|
||||
- [x] All Docker containers run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
## Read Reference Files
|
||||
|
||||
Before starting execution, read the following reference files:
|
||||
1. `references/SOP.md` - Detailed step-by-step operating instructions
|
||||
2. `references/troubleshooting.md` - Common issues and solutions
|
||||
3. `templates/report.local.template.md` - Local mode test report template
|
||||
4. `templates/report.docker.template.md` - Docker mode test report template
|
||||
@@ -0,0 +1,482 @@
|
||||
# DeerFlow Smoke Test Standard Operating Procedure (SOP)
|
||||
|
||||
This document describes the detailed operating steps for each phase of the DeerFlow smoke test.
|
||||
|
||||
## Phase 1: Code Update Check
|
||||
|
||||
### 1.1 Confirm Current Directory
|
||||
|
||||
**Objective**: Verify that the current working directory is the DeerFlow project root.
|
||||
|
||||
**Steps**:
|
||||
1. Run `pwd` to view the current working directory
|
||||
2. Check whether the directory contains the following files/directories:
|
||||
- `Makefile`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `config.example.yaml`
|
||||
|
||||
**Success Criteria**: The current directory contains all of the files/directories listed above.
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Check Git Status
|
||||
|
||||
**Objective**: Check whether there are uncommitted changes.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git status`
|
||||
2. Check whether the output includes "Changes not staged for commit" or "Untracked files"
|
||||
|
||||
**Notes**:
|
||||
- If there are uncommitted changes, recommend that the user commit or stash them first to avoid conflicts while pulling
|
||||
- If the user confirms that they want to continue, this step can be skipped
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Pull the Latest Code
|
||||
|
||||
**Objective**: Fetch the latest code updates.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git fetch origin main`
|
||||
2. Run `git pull origin main`
|
||||
|
||||
**Success Criteria**:
|
||||
- The commands succeed without errors
|
||||
- The output shows "Already up to date" or indicates that new commits were pulled successfully
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Confirm Code Update
|
||||
|
||||
**Objective**: Verify that the latest code was pulled successfully.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git log -1 --oneline` to view the latest commit
|
||||
2. Record the commit hash and message
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
### 2.1 Choose Deployment Mode
|
||||
|
||||
**Objective**: Decide whether to use local mode or Docker mode.
|
||||
|
||||
**Decision Flow**:
|
||||
1. Prefer local mode first to avoid network-related issues
|
||||
2. If the user explicitly requests Docker, use Docker
|
||||
3. If Docker network issues occur, switch to local mode automatically
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Local Mode Environment Check
|
||||
|
||||
**Objective**: Verify that local development environment dependencies are satisfied.
|
||||
|
||||
#### 2.2.1 Check Node.js Version
|
||||
|
||||
**Steps**:
|
||||
1. If nvm is used, run `nvm use 22` to switch to Node 22+
|
||||
2. Run `node --version`
|
||||
|
||||
**Success Criteria**: Version >= 22.x
|
||||
|
||||
**Failure Handling**:
|
||||
- If the version is too low, ask the user to install/switch Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
- Or install it from the official website: https://nodejs.org/
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.2 Check pnpm
|
||||
|
||||
**Steps**:
|
||||
1. Run `pnpm --version`
|
||||
|
||||
**Success Criteria**: The command returns pnpm version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If pnpm is not installed, ask the user to install it with `npm install -g pnpm`
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.3 Check uv
|
||||
|
||||
**Steps**:
|
||||
1. Run `uv --version`
|
||||
|
||||
**Success Criteria**: The command returns uv version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If uv is not installed, ask the user to install uv
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.4 Check nginx
|
||||
|
||||
**Steps**:
|
||||
1. Run `nginx -v`
|
||||
|
||||
**Success Criteria**: The command returns nginx version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- macOS: install with Homebrew using `brew install nginx`
|
||||
- Linux: install using the system package manager
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.5 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run the following commands to check ports:
|
||||
```bash
|
||||
lsof -i :2026 # Main port
|
||||
lsof -i :3000 # Frontend
|
||||
lsof -i :8001 # Gateway
|
||||
```
|
||||
|
||||
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
|
||||
|
||||
**Failure Handling**:
|
||||
- If a port is occupied, ask the user to stop the related process
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Docker Mode Environment Check (If Docker Is Selected)
|
||||
|
||||
#### 2.3.1 Check Whether Docker Is Installed
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker --version`
|
||||
|
||||
**Success Criteria**: The command returns Docker version information, such as "Docker version 24.x.x".
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.2 Check Docker Daemon Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker info`
|
||||
|
||||
**Success Criteria**: The command runs successfully and shows Docker system information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If it fails, ask the user to start Docker Desktop or the Docker service
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.3 Check Docker Compose Availability
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker compose version`
|
||||
|
||||
**Success Criteria**: The command returns Docker Compose version information.
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.4 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run `lsof -i :2026` (macOS/Linux) or `netstat -ano | findstr :2026` (Windows)
|
||||
|
||||
**Success Criteria**: Port 2026 is free, or it is occupied only by a DeerFlow-related process.
|
||||
|
||||
**Failure Handling**:
|
||||
- If the port is occupied by another process, ask the user to stop that process or change the configuration
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Configuration Preparation
|
||||
|
||||
### 3.1 Check config.yaml
|
||||
|
||||
**Steps**:
|
||||
1. Check whether `config.yaml` exists
|
||||
2. If it does not exist, run `make config`
|
||||
3. If it already exists, consider running `make config-upgrade` to merge new fields
|
||||
|
||||
**Validation**:
|
||||
- Check whether at least one model is configured in config.yaml
|
||||
- Check whether the model configuration references the correct environment variables
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Check the .env File
|
||||
|
||||
**Steps**:
|
||||
1. Check whether the `.env` file exists
|
||||
2. If it does not exist, copy it from `.env.example`
|
||||
3. Check whether the following environment variables are configured:
|
||||
- `OPENAI_API_KEY` (or other model API keys)
|
||||
- Other required settings
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Deployment Execution
|
||||
|
||||
### 4.1 Local Mode Deployment
|
||||
|
||||
#### 4.1.1 Check Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make check`
|
||||
|
||||
**Description**: This command validates all required tools (Node.js 22+, pnpm, uv, nginx).
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.2 Install Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make install`
|
||||
|
||||
**Description**: This command installs both backend and frontend dependencies.
|
||||
|
||||
**Notes**:
|
||||
- This step may take some time
|
||||
- If network issues cause failures, try using a closer or mirrored package registry
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.3 (Optional) Pre-pull the Sandbox Image
|
||||
|
||||
**Steps**:
|
||||
1. If Docker / Container sandbox is used, run `make setup-sandbox`
|
||||
|
||||
**Description**: This step is optional and not needed for local sandbox mode.
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.4 Start Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make dev-daemon` (background mode)
|
||||
|
||||
**Description**: This command starts all services (Gateway embedded runtime, Frontend, Nginx).
|
||||
|
||||
**Notes**:
|
||||
- `make dev` runs in the foreground and stops with Ctrl+C
|
||||
- `make dev-daemon` runs in the background
|
||||
- Use `make stop` to stop services
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.5 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 90-120 seconds for all services to start completely
|
||||
2. You can monitor startup progress by checking these log files:
|
||||
- `logs/gateway.log`
|
||||
- `logs/frontend.log`
|
||||
- `logs/nginx.log`
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Docker Mode Deployment (If Docker Is Selected)
|
||||
|
||||
#### 4.2.1 Initialize the Docker Environment
|
||||
|
||||
**Steps**:
|
||||
1. Run `make docker-init`
|
||||
|
||||
**Description**: This command pulls the sandbox image if needed.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.2 Start Docker Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make up`
|
||||
|
||||
**Description**: This command builds and starts all required Docker containers in production.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.3 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 60-90 seconds for all services to start completely
|
||||
2. You can run `make docker-logs` to monitor startup progress
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Service Health Check
|
||||
|
||||
### 5.1 Local Mode Health Check
|
||||
|
||||
#### 5.1.1 Check Process Status
|
||||
|
||||
**Steps**:
|
||||
1. Run the following command to check processes:
|
||||
```bash
|
||||
ps aux | grep -E "(uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
**Success Criteria**: Confirm that the following processes are running:
|
||||
- Gateway (`uvicorn app.gateway.app:app`)
|
||||
- Frontend (`next dev` or `next start`)
|
||||
- Nginx (`nginx`)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Check LangGraph-compatible API
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/api/langgraph/assistants/lead_agent` to verify Gateway's LangGraph-compatible API route is reachable.
|
||||
2. A `401` response is acceptable when authentication is enabled and no session cookie is provided.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.5 Frontend Route Smoke Check
|
||||
|
||||
**Objective**: Verify key `/workspace/*` frontend routes render correctly.
|
||||
|
||||
**Steps**:
|
||||
1. Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh`.
|
||||
2. The script auto-detects whether authentication (`DEER_FLOW_AUTH_DISABLED`) is enabled.
|
||||
3. When auth is on, the script registers / logs in a smoke-test user (`smoke-test@deerflow.dev` by default) and passes the session cookie so the real `/workspace/*` pages are verified — not the login redirect.
|
||||
4. When auth is off, the routes are checked anonymously as before.
|
||||
|
||||
**Customisation**:
|
||||
- `SMOKE_TEST_EMAIL` — email for the test account (default: `smoke-test@deerflow.dev`).
|
||||
- `SMOKE_TEST_PASSWORD` — password for the test account (default: `SmokeTest123!`).
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Docker Mode Health Check (When Using Docker)
|
||||
|
||||
#### 5.2.1 Check Container Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker ps`
|
||||
2. Confirm that the following containers are running:
|
||||
- `deer-flow-nginx`
|
||||
- `deer-flow-frontend`
|
||||
- `deer-flow-gateway`
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.4 Check LangGraph-compatible API
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/api/langgraph/assistants/lead_agent` to verify Gateway's LangGraph-compatible API route is reachable.
|
||||
2. A `401` response is acceptable when authentication is enabled and no session cookie is provided.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.5 Frontend Route Smoke Check
|
||||
|
||||
**Objective**: Verify key `/workspace/*` frontend routes render correctly.
|
||||
|
||||
**Steps**:
|
||||
1. Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh`.
|
||||
2. The script auto-detects whether authentication (`DEER_FLOW_AUTH_DISABLED`) is enabled.
|
||||
3. When auth is on, the script registers / logs in a smoke-test user (`smoke-test@deerflow.dev` by default) and passes the session cookie so the real `/workspace/*` pages are verified — not the login redirect.
|
||||
4. When auth is off, the routes are checked anonymously as before.
|
||||
|
||||
**Customisation**:
|
||||
- `SMOKE_TEST_EMAIL` — email for the test account (default: `smoke-test@deerflow.dev`).
|
||||
- `SMOKE_TEST_PASSWORD` — password for the test account (default: `SmokeTest123!`).
|
||||
|
||||
---
|
||||
|
||||
## Optional Functional Verification
|
||||
|
||||
### 6.1 List Available Models
|
||||
|
||||
**Steps**: Verify the model list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 List Available Skills
|
||||
|
||||
**Steps**: Verify the skill list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.3 Simple Chat Test
|
||||
|
||||
**Steps**: Send a simple message to test the complete workflow.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Generate the Test Report
|
||||
|
||||
### 6.1 Collect Test Results
|
||||
|
||||
Summarize the execution status of each phase and record successful and failed items.
|
||||
|
||||
### 6.2 Record Issues
|
||||
|
||||
If anything fails, record detailed error information.
|
||||
|
||||
### 6.3 Generate the Report
|
||||
|
||||
Use the template to create a complete test report.
|
||||
|
||||
### 6.4 Provide Recommendations
|
||||
|
||||
Provide follow-up recommendations based on the test results.
|
||||
@@ -0,0 +1,593 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
This document lists common issues encountered during DeerFlow smoke testing and how to resolve them.
|
||||
|
||||
## Code Update Issues
|
||||
|
||||
### Issue: `git pull` Fails with a Merge Conflict Warning
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
error: Your local changes to the following files would be overwritten by merge
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Option A: Commit local changes first
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "Save local changes"
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
2. Option B: Stash local changes
|
||||
```bash
|
||||
git stash
|
||||
git pull origin main
|
||||
git stash pop # Restore changes later if needed
|
||||
```
|
||||
|
||||
3. Option C: Discard local changes (use with caution)
|
||||
```bash
|
||||
git reset --hard HEAD
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Environment Issues
|
||||
|
||||
### Issue: Node.js Version Is Too Old
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Node.js version is too old. Requires 22+, got x.x.x
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install or upgrade Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
|
||||
2. Or download and install it from the official website: https://nodejs.org/
|
||||
|
||||
3. Verify the version:
|
||||
```bash
|
||||
node --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: pnpm Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: pnpm
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install pnpm with npm:
|
||||
```bash
|
||||
npm install -g pnpm
|
||||
```
|
||||
|
||||
2. Or use the official installation script:
|
||||
```bash
|
||||
curl -fsSL https://get.pnpm.io/install.sh | sh -
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
pnpm --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: uv Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: uv
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Use the official installation script:
|
||||
```bash
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
2. macOS users can also install it with Homebrew:
|
||||
```bash
|
||||
brew install uv
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
uv --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: nginx Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: nginx
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. macOS (Homebrew):
|
||||
```bash
|
||||
brew install nginx
|
||||
```
|
||||
|
||||
2. Ubuntu/Debian:
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install nginx
|
||||
```
|
||||
|
||||
3. CentOS/RHEL:
|
||||
```bash
|
||||
sudo yum install nginx
|
||||
```
|
||||
|
||||
4. Verify the installation:
|
||||
```bash
|
||||
nginx -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Port Is Already in Use
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: listen EADDRINUSE: address already in use :::2026
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Find the process using the port:
|
||||
```bash
|
||||
lsof -i :2026 # macOS/Linux
|
||||
netstat -ano | findstr :2026 # Windows
|
||||
```
|
||||
|
||||
2. Stop that process:
|
||||
```bash
|
||||
kill -9 <PID> # macOS/Linux
|
||||
taskkill /PID <PID> /F # Windows
|
||||
```
|
||||
|
||||
3. Or stop DeerFlow services first:
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Dependency Installation Issues
|
||||
|
||||
### Issue: `make install` Fails Due to Network Timeout
|
||||
|
||||
**Symptoms**:
|
||||
Network timeouts or connection failures occur during dependency installation.
|
||||
|
||||
**Solutions**:
|
||||
1. Configure pnpm to use a mirror registry:
|
||||
```bash
|
||||
pnpm config set registry https://registry.npmmirror.com
|
||||
```
|
||||
|
||||
2. Configure uv to use a mirror registry:
|
||||
```bash
|
||||
uv pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
```
|
||||
|
||||
3. Retry the installation:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Python Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `uv sync`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the uv cache:
|
||||
```bash
|
||||
cd backend
|
||||
uv cache clean
|
||||
```
|
||||
|
||||
2. Resync dependencies:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
3. View detailed error logs:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync --verbose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `pnpm install`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the pnpm cache:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm store prune
|
||||
```
|
||||
|
||||
2. Remove node_modules and the lock file:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules pnpm-lock.yaml
|
||||
```
|
||||
|
||||
3. Reinstall:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Service Startup Issues
|
||||
|
||||
### Issue: Services Exit Immediately After Startup
|
||||
|
||||
**Symptoms**:
|
||||
Processes exit quickly after running `make dev-daemon`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check log files:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml is configured correctly
|
||||
3. Check environment variables in the .env file
|
||||
4. Confirm that required ports are not occupied
|
||||
5. Stop all services and restart:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start Because Temp Directories Do Not Exist
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
nginx: [emerg] mkdir() "/opt/homebrew/var/run/nginx/client_body_temp" failed (2: No such file or directory)
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
Add local temp directory configuration to `docker/nginx/nginx.local.conf` so nginx uses the repository's temp directory.
|
||||
|
||||
Add the following at the beginning of the `http` block:
|
||||
```nginx
|
||||
client_body_temp_path temp/client_body_temp;
|
||||
proxy_temp_path temp/proxy_temp;
|
||||
fastcgi_temp_path temp/fastcgi_temp;
|
||||
uwsgi_temp_path temp/uwsgi_temp;
|
||||
scgi_temp_path temp/scgi_temp;
|
||||
```
|
||||
|
||||
Note: The `temp/` directory under the repository root is created automatically by `make dev` or `make dev-daemon`.
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start (General)
|
||||
|
||||
**Symptoms**:
|
||||
The nginx process fails to start or reports an error.
|
||||
|
||||
**Solutions**:
|
||||
1. Check the nginx configuration:
|
||||
```bash
|
||||
nginx -t -c docker/nginx/nginx.local.conf -p .
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Ensure no other nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
4. If needed, stop existing nginx processes:
|
||||
```bash
|
||||
pkill -9 nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Compilation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Compilation errors appear in `frontend.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check frontend logs:
|
||||
```bash
|
||||
tail -f logs/frontend.log
|
||||
```
|
||||
|
||||
2. Check whether Node.js version is 22+
|
||||
3. Reinstall frontend dependencies:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules .next
|
||||
pnpm install
|
||||
```
|
||||
|
||||
4. Restart services:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Gateway Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `gateway.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
|
||||
---
|
||||
|
||||
## Docker-Related Issues
|
||||
|
||||
### Issue: Docker Commands Cannot Run
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Cannot connect to the Docker daemon
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Confirm that Docker Desktop is running
|
||||
2. macOS: check whether the Docker icon appears in the top menu bar
|
||||
3. Linux: run `sudo systemctl start docker`
|
||||
4. Run `docker info` again to verify
|
||||
|
||||
---
|
||||
|
||||
### Issue: `make docker-init` Fails to Pull the Image
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error pulling image: connection refused
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Check network connectivity
|
||||
2. Configure a Docker image mirror if needed
|
||||
3. Check whether a proxy is required
|
||||
4. Switch to local installation mode if necessary (recommended)
|
||||
|
||||
---
|
||||
|
||||
## Configuration File Issues
|
||||
|
||||
### Issue: config.yaml Is Missing or Invalid
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: could not read config.yaml
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Regenerate the configuration file:
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
2. Check YAML syntax:
|
||||
- Make sure indentation is correct (use 2 spaces)
|
||||
- Make sure there are no tab characters
|
||||
- Check that there is a space after each colon
|
||||
|
||||
3. Use a YAML validation tool to check the format
|
||||
|
||||
---
|
||||
|
||||
### Issue: Model API Key Is Not Configured
|
||||
|
||||
**Symptoms**:
|
||||
After services start, API requests fail with authentication errors.
|
||||
|
||||
**Solutions**:
|
||||
1. Edit the .env file and add the API key:
|
||||
```bash
|
||||
OPENAI_API_KEY=your-actual-api-key-here
|
||||
```
|
||||
|
||||
2. Restart services (local mode):
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
3. Restart services (Docker mode):
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
4. Confirm that the model configuration in config.yaml references the environment variable correctly
|
||||
|
||||
---
|
||||
|
||||
## Service Health Check Issues
|
||||
|
||||
### Issue: Frontend Page Is Not Accessible
|
||||
|
||||
**Symptoms**:
|
||||
The browser shows a connection failure when visiting http://localhost:2026.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Confirm that the nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Confirm that the nginx container is running:
|
||||
```bash
|
||||
docker ps | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml logs nginx
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
---
|
||||
|
||||
### Issue: API Gateway Health Check Fails
|
||||
|
||||
**Symptoms**:
|
||||
Accessing `/health` returns an error or times out.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Check gateway container logs:
|
||||
```bash
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml is mounted correctly
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
|
||||
---
|
||||
|
||||
## Common Diagnostic Commands
|
||||
|
||||
### Local Mode Diagnostics
|
||||
|
||||
#### View All Service Processes
|
||||
```bash
|
||||
ps aux | grep -E "(uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
#### View Service Logs
|
||||
```bash
|
||||
# View all logs
|
||||
tail -f logs/*.log
|
||||
|
||||
# View specific service logs
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
#### Stop All Services
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
#### Fully Reset the Local Environment
|
||||
```bash
|
||||
make stop
|
||||
make clean
|
||||
make config
|
||||
make install
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Docker Mode Diagnostics
|
||||
|
||||
#### View All Container Status
|
||||
```bash
|
||||
docker ps -a
|
||||
```
|
||||
|
||||
#### View Container Resource Usage
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
#### Enter a Container for Debugging
|
||||
```bash
|
||||
docker exec -it deer-flow-gateway sh
|
||||
```
|
||||
|
||||
#### Clean Up All DeerFlow-Related Containers and Images
|
||||
```bash
|
||||
make docker-stop
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml down -v
|
||||
```
|
||||
|
||||
#### Fully Reset the Docker Environment
|
||||
```bash
|
||||
make docker-stop
|
||||
make clean
|
||||
make config
|
||||
make docker-init
|
||||
make docker-start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Get More Help
|
||||
|
||||
If the solutions above do not resolve the issue:
|
||||
1. Check the GitHub issues for the project: https://github.com/bytedance/deer-flow/issues
|
||||
2. Review the project documentation: README.md and the `backend/docs/` directory
|
||||
3. Open a new issue and include detailed error logs
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Docker Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether Docker is installed
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
echo "✓ Docker is installed"
|
||||
docker --version
|
||||
else
|
||||
echo "✗ Docker is not installed"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the Docker daemon
|
||||
if docker info >/dev/null 2>&1; then
|
||||
echo "✓ Docker daemon is running normally"
|
||||
else
|
||||
echo "✗ Docker daemon is not running"
|
||||
echo " Please start Docker Desktop or the Docker service"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check Docker Compose
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
echo "✓ Docker Compose is available"
|
||||
docker compose version
|
||||
else
|
||||
echo "✗ Docker Compose is not available"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check port 2026
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is required to check whether port 2026 is available"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
port_2026_usage="$(lsof -nP -iTCP:2026 -sTCP:LISTEN 2>/dev/null || true)"
|
||||
if [ -n "$port_2026_usage" ]; then
|
||||
echo "⚠ Port 2026 is already in use"
|
||||
echo " Occupying process:"
|
||||
echo "$port_2026_usage"
|
||||
|
||||
deerflow_process_found=0
|
||||
while IFS= read -r pid; do
|
||||
if [ -z "$pid" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
process_command="$(ps -p "$pid" -o command= 2>/dev/null || true)"
|
||||
case "$process_command" in
|
||||
*[Dd]eer[Ff]low*|*[Dd]eerflow*|*[Nn]ginx*deerflow*|*deerflow/*[Nn]ginx*)
|
||||
deerflow_process_found=1
|
||||
;;
|
||||
esac
|
||||
done <<EOF
|
||||
$(printf '%s\n' "$port_2026_usage" | awk 'NR > 1 {print $2}')
|
||||
EOF
|
||||
|
||||
if [ "$deerflow_process_found" -eq 1 ]; then
|
||||
echo "✓ Port 2026 is occupied by DeerFlow"
|
||||
else
|
||||
echo "✗ Port 2026 must be free before starting DeerFlow"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "✓ Port 2026 is available"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Environment Check Complete"
|
||||
echo "=========================================="
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Local Development Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
|
||||
# Check Node.js
|
||||
echo "1. Checking Node.js..."
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
NODE_VERSION=$(node --version | sed 's/v//')
|
||||
NODE_MAJOR=$(echo "$NODE_VERSION" | cut -d. -f1)
|
||||
if [ "$NODE_MAJOR" -ge 22 ]; then
|
||||
echo "✓ Node.js is installed (version: $NODE_VERSION)"
|
||||
else
|
||||
echo "✗ Node.js version is too old (current: $NODE_VERSION, required: 22+)"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
echo "✗ Node.js is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check pnpm
|
||||
echo "2. Checking pnpm..."
|
||||
if command -v pnpm >/dev/null 2>&1; then
|
||||
echo "✓ pnpm is installed (version: $(pnpm --version))"
|
||||
else
|
||||
echo "✗ pnpm is not installed"
|
||||
echo " Install command: npm install -g pnpm"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check uv
|
||||
echo "3. Checking uv..."
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
echo "✓ uv is installed (version: $(uv --version))"
|
||||
else
|
||||
echo "✗ uv is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check nginx
|
||||
echo "4. Checking nginx..."
|
||||
if command -v nginx >/dev/null 2>&1; then
|
||||
echo "✓ nginx is installed (version: $(nginx -v 2>&1))"
|
||||
else
|
||||
echo "✗ nginx is not installed"
|
||||
echo " macOS: brew install nginx"
|
||||
echo " Linux: install it with the system package manager"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check ports
|
||||
echo "5. Checking ports..."
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is not installed, so port availability cannot be verified"
|
||||
echo " Install lsof and rerun this check"
|
||||
all_passed=false
|
||||
else
|
||||
for port in 2026 3000 8001; do
|
||||
if lsof -i :$port >/dev/null 2>&1; then
|
||||
echo "⚠ Port $port is already in use:"
|
||||
lsof -i :$port | head -2
|
||||
all_passed=false
|
||||
else
|
||||
echo "✓ Port $port is available"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Summary
|
||||
echo "=========================================="
|
||||
echo " Environment Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All environment checks passed!"
|
||||
echo ""
|
||||
echo "Next step: run make install to install dependencies"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed. Please fix the issues above first"
|
||||
exit 1
|
||||
fi
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the frontend .env file
|
||||
if [ ! -f "frontend/.env" ]; then
|
||||
echo "frontend/.env does not exist. Copying it from the example..."
|
||||
if [ -f "frontend/.env.example" ]; then
|
||||
cp frontend/.env.example frontend/.env
|
||||
echo "✓ Created the frontend/.env file"
|
||||
else
|
||||
echo "⚠ frontend/.env.example does not exist. Please create frontend/.env manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ frontend/.env file exists"
|
||||
fi
|
||||
echo ""
|
||||
# Initialize the Docker environment
|
||||
echo "Initializing the Docker environment..."
|
||||
make docker-init
|
||||
echo ""
|
||||
|
||||
# Start Docker services
|
||||
echo "Starting Docker services..."
|
||||
make docker-start
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs: make docker-logs"
|
||||
echo "🛑 Stop services: make docker-stop"
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Local Mode Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check dependencies
|
||||
echo "Checking dependencies..."
|
||||
make check
|
||||
echo ""
|
||||
|
||||
# Install dependencies
|
||||
echo "Installing dependencies..."
|
||||
make install
|
||||
echo ""
|
||||
|
||||
# Start services
|
||||
echo "Starting services (background mode)..."
|
||||
make dev-daemon
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs:"
|
||||
echo " - logs/gateway.log"
|
||||
echo " - logs/frontend.log"
|
||||
echo " - logs/nginx.log"
|
||||
echo "🛑 Stop services: make stop"
|
||||
echo ""
|
||||
echo "Please wait 90-120 seconds for all services to start completely, then run the health check"
|
||||
@@ -0,0 +1,170 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Page Smoke Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:2026}"
|
||||
DOC_PATH="${DOC_PATH:-/en/docs}"
|
||||
|
||||
# When the gateway has authentication enabled (DEER_FLOW_AUTH_DISABLED != 1),
|
||||
# protected /workspace/* routes redirect anonymous requests to /login.
|
||||
# We detect auth, register / log in a smoke-test user, and pass the session
|
||||
# cookie to all curl calls so the real pages are verified, not the login form.
|
||||
SMOKE_TEST_EMAIL="${SMOKE_TEST_EMAIL:-smoke-test@deerflow.dev}"
|
||||
SMOKE_TEST_PASSWORD="${SMOKE_TEST_PASSWORD:-SmokeTest123!}"
|
||||
COOKIE_JAR=$(mktemp /tmp/deerflow-smoke-cookies.XXXXXX)
|
||||
trap 'rm -f "$COOKIE_JAR"' EXIT
|
||||
CURL_AUTH_OPTS=""
|
||||
|
||||
authenticate() {
|
||||
# Make sure the service is reachable before detecting auth
|
||||
local health
|
||||
health=$(curl -s -o /dev/null -w "%{http_code}" "${BASE_URL}/" 2>/dev/null)
|
||||
if [ "$health" = "000" ]; then
|
||||
echo "✗ Cannot reach ${BASE_URL} — is the service running?"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# A protected endpoint returns 401 when auth is on
|
||||
local auth_check
|
||||
auth_check=$(curl -s -o /dev/null -w "%{http_code}" "${BASE_URL}/api/models" 2>/dev/null)
|
||||
if [ "$auth_check" != "401" ]; then
|
||||
echo "ℹ Auth is disabled — no login needed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "🔐 Auth is enabled — setting up smoke test session..."
|
||||
|
||||
# Check whether the system needs first-boot initialization
|
||||
local needs_setup
|
||||
needs_setup=$(curl -s "${BASE_URL}/api/v1/auth/setup-status" \
|
||||
| grep -o '"needs_setup":[^,}]*' | grep -o 'true\|false')
|
||||
if [ "$needs_setup" = "true" ]; then
|
||||
echo " Initializing system with smoke test admin..."
|
||||
local init_code
|
||||
init_code=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-X POST "${BASE_URL}/api/v1/auth/initialize" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"email\":\"${SMOKE_TEST_EMAIL}\",\"password\":\"${SMOKE_TEST_PASSWORD}\"}" \
|
||||
-c "$COOKIE_JAR" 2>/dev/null)
|
||||
if [ "$init_code" != "201" ]; then
|
||||
echo "✗ Initialize failed (HTTP $init_code)"
|
||||
return 1
|
||||
fi
|
||||
echo "✓ Admin initialized & logged in"
|
||||
elif [ -z "$needs_setup" ]; then
|
||||
echo "⚠ Could not determine setup status — skipping initialize, trying register/login"
|
||||
else
|
||||
# Register first — on success (201) it also auto-logs-in via the cookie.
|
||||
# This avoids a wasted login attempt that counts toward rate-limiting.
|
||||
local auth_code
|
||||
auth_code=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-X POST "${BASE_URL}/api/v1/auth/register" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"email\":\"${SMOKE_TEST_EMAIL}\",\"password\":\"${SMOKE_TEST_PASSWORD}\"}" \
|
||||
-c "$COOKIE_JAR" 2>/dev/null)
|
||||
|
||||
if [ "$auth_code" = "201" ]; then
|
||||
echo "✓ Registered as ${SMOKE_TEST_EMAIL}"
|
||||
else
|
||||
# User already exists — clear stale cookies from register attempt, then log in
|
||||
: > "$COOKIE_JAR"
|
||||
local login_code
|
||||
login_code=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-X POST "${BASE_URL}/api/v1/auth/login/local" \
|
||||
--data-urlencode "username=${SMOKE_TEST_EMAIL}" \
|
||||
--data-urlencode "password=${SMOKE_TEST_PASSWORD}" \
|
||||
-c "$COOKIE_JAR" 2>/dev/null)
|
||||
if [ "$login_code" != "200" ]; then
|
||||
echo "✗ Login failed (HTTP $login_code)"
|
||||
return 1
|
||||
fi
|
||||
echo "✓ Logged in as ${SMOKE_TEST_EMAIL}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Verify the session cookie works across all branches
|
||||
local me_code
|
||||
me_code=$(curl -s -o /dev/null -w "%{http_code}" \
|
||||
-b "$COOKIE_JAR" "${BASE_URL}/api/v1/auth/me" 2>/dev/null)
|
||||
|
||||
if [ "$me_code" = "200" ]; then
|
||||
echo "✓ Session verified for ${SMOKE_TEST_EMAIL}"
|
||||
CURL_AUTH_OPTS="-b $COOKIE_JAR"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo "✗ Auth failed — session cookie not accepted (HTTP $me_code)"
|
||||
echo " Set SMOKE_TEST_EMAIL / SMOKE_TEST_PASSWORD or check the existing account"
|
||||
return 1
|
||||
}
|
||||
|
||||
all_passed=true
|
||||
|
||||
check_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
|
||||
local status
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" -L ${CURL_AUTH_OPTS} "$url")"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name ($url) -> $status"
|
||||
else
|
||||
echo "✗ $name ($url) -> $status (expected: $expected_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_final_url() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_path_re="$3"
|
||||
|
||||
local effective
|
||||
effective="$(curl -s -o /dev/null -w "%{url_effective}" -L ${CURL_AUTH_OPTS} "$url")"
|
||||
if echo "$effective" | grep -Eq "$expected_path_re"; then
|
||||
echo "✓ $name redirect target -> $effective"
|
||||
else
|
||||
echo "✗ $name redirect target -> $effective (expected path: $expected_path_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
# Authenticate before checking protected routes
|
||||
authenticate || exit 1
|
||||
|
||||
echo ""
|
||||
echo "1. Checking entry pages..."
|
||||
check_status "Landing page" "${BASE_URL}/" "200"
|
||||
check_status "Workspace redirect" "${BASE_URL}/workspace" "200|301|302|307|308"
|
||||
check_final_url "Workspace redirect" "${BASE_URL}/workspace" "/workspace/chats/"
|
||||
echo ""
|
||||
|
||||
echo "2. Checking key workspace routes..."
|
||||
check_status "New chat page" "${BASE_URL}/workspace/chats/new" "200"
|
||||
check_final_url "New chat page" "${BASE_URL}/workspace/chats/new" "/workspace/"
|
||||
check_status "Chats list page" "${BASE_URL}/workspace/chats" "200"
|
||||
check_final_url "Chats list page" "${BASE_URL}/workspace/chats" "/workspace/"
|
||||
check_status "Agents gallery page" "${BASE_URL}/workspace/agents" "200"
|
||||
check_final_url "Agents gallery page" "${BASE_URL}/workspace/agents" "/workspace/agents"
|
||||
echo ""
|
||||
|
||||
echo "3. Checking docs route (optional)..."
|
||||
check_status "Docs page" "${BASE_URL}${DOC_PATH}" "200|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Smoke Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ Frontend smoke checks passed!"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Frontend smoke checks failed"
|
||||
exit 1
|
||||
fi
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Service Health Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
mode="${SMOKE_TEST_MODE:-auto}"
|
||||
summary_hint="make logs"
|
||||
|
||||
print_step() {
|
||||
echo "$1"
|
||||
}
|
||||
|
||||
check_http_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
local status
|
||||
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" "$url" 2>/dev/null)"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name is accessible ($url -> $status)"
|
||||
else
|
||||
echo "✗ $name is not accessible ($url -> ${status:-000})"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_listen_port() {
|
||||
local name="$1"
|
||||
local port="$2"
|
||||
|
||||
if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "✓ $name is listening on port $port"
|
||||
else
|
||||
echo "✗ $name is not listening on port $port"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
docker_available() {
|
||||
command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1
|
||||
}
|
||||
|
||||
detect_mode() {
|
||||
case "$mode" in
|
||||
local|docker)
|
||||
echo "$mode"
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
if docker_available && docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "docker"
|
||||
else
|
||||
echo "local"
|
||||
fi
|
||||
}
|
||||
|
||||
mode="$(detect_mode)"
|
||||
|
||||
echo "Deployment mode: $mode"
|
||||
echo ""
|
||||
|
||||
if [ "$mode" = "docker" ]; then
|
||||
summary_hint="make docker-logs"
|
||||
print_step "1. Checking container status..."
|
||||
if docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "✓ Containers are running:"
|
||||
docker ps --format " - {{.Names}} ({{.Status}})"
|
||||
else
|
||||
echo "✗ No DeerFlow-related containers are running"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
summary_hint="logs/{gateway,frontend,nginx}.log"
|
||||
print_step "1. Checking local service ports..."
|
||||
check_listen_port "Nginx" 2026
|
||||
check_listen_port "Frontend" 3000
|
||||
check_listen_port "Gateway" 8001
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "2. Waiting for services to fully start (30 seconds)..."
|
||||
sleep 30
|
||||
echo ""
|
||||
|
||||
echo "3. Checking frontend service..."
|
||||
check_http_status "Frontend service" "http://localhost:2026" "200|301|302|307|308"
|
||||
echo ""
|
||||
|
||||
echo "4. Checking API Gateway..."
|
||||
health_response=$(curl -s http://localhost:2026/health 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$health_response" ]; then
|
||||
echo "✓ API Gateway health check passed"
|
||||
echo " Response: $health_response"
|
||||
else
|
||||
echo "✗ API Gateway health check failed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "5. Checking LangGraph-compatible Gateway API..."
|
||||
check_http_status "LangGraph-compatible Gateway API" "http://localhost:2026/api/langgraph/assistants/lead_agent" "200|401"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Health Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All checks passed!"
|
||||
echo ""
|
||||
echo "🌐 Application URL: http://localhost:2026"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed"
|
||||
echo ""
|
||||
echo "Please review: $summary_hint"
|
||||
exit 1
|
||||
fi
|
||||
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Pulling the Latest Code"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether the current directory is a Git repository
|
||||
if [ ! -d ".git" ]; then
|
||||
echo "✗ The current directory is not a Git repository"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check Git status
|
||||
echo "Checking Git status..."
|
||||
if git status --porcelain | grep -q .; then
|
||||
echo "⚠ Uncommitted changes detected:"
|
||||
git status --short
|
||||
echo ""
|
||||
echo "Please commit or stash your changes before continuing"
|
||||
echo "Options:"
|
||||
echo " 1. git add . && git commit -m 'Save changes'"
|
||||
echo " 2. git stash (stash changes and restore them later)"
|
||||
echo " 3. git reset --hard HEAD (discard local changes - use with caution)"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ Working tree is clean"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Fetch remote updates
|
||||
echo "Fetching remote updates..."
|
||||
git fetch origin main
|
||||
echo ""
|
||||
|
||||
# Pull the latest code
|
||||
echo "Pulling the latest code..."
|
||||
git pull origin main
|
||||
echo ""
|
||||
|
||||
# Show the latest commit
|
||||
echo "Latest commit:"
|
||||
git log -1 --oneline
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Code Update Complete"
|
||||
echo "=========================================="
|
||||
@@ -0,0 +1,179 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Docker
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Docker Environment Check
|
||||
|
||||
- [x] Docker version - {{status_docker_version}}
|
||||
- [x] Docker daemon - {{status_docker_daemon}}
|
||||
- [x] Docker Compose - {{status_docker_compose}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Docker Deployment
|
||||
|
||||
- [x] docker-init - {{status_docker_init}}
|
||||
- [x] docker-start - {{status_docker_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Container status - {{status_containers}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph-compatible Gateway API - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Docker Version
|
||||
```text
|
||||
{{docker_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Container Status
|
||||
|
||||
| Container Name | Status | Uptime |
|
||||
|----------|------|----------|
|
||||
| deer-flow-nginx | {{nginx_status}} | {{nginx_uptime}} |
|
||||
| deer-flow-frontend | {{frontend_status}} | {{frontend_uptime}} |
|
||||
| deer-flow-gateway | {{gateway_status}} | {{gateway_uptime}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check Docker logs: `make docker-logs`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
@@ -0,0 +1,185 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Local
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Local Environment Check
|
||||
|
||||
- [x] Node.js version - {{status_node_version}}
|
||||
- [x] pnpm - {{status_pnpm}}
|
||||
- [x] uv - {{status_uv}}
|
||||
- [x] nginx - {{status_nginx}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Local Deployment
|
||||
|
||||
- [x] make check - {{status_make_check}}
|
||||
- [x] make install - {{status_make_install}}
|
||||
- [x] make dev-daemon / make dev - {{status_local_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Process status - {{status_processes}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph-compatible Gateway API - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Local Dependency Versions
|
||||
```text
|
||||
Node.js: {{node_version_output}}
|
||||
pnpm: {{pnpm_version_output}}
|
||||
uv: {{uv_version_output}}
|
||||
nginx: {{nginx_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Local Service Status
|
||||
|
||||
| Service | Status | Endpoint |
|
||||
|---------|--------|----------|
|
||||
| Nginx | {{nginx_status}} | {{nginx_endpoint}} |
|
||||
| Frontend | {{frontend_status}} | {{frontend_endpoint}} |
|
||||
| Gateway | {{gateway_status}} | {{gateway_endpoint}} |
|
||||
| Gateway LangGraph API | {{langgraph_status}} | {{langgraph_endpoint}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check local logs: `logs/{gateway,frontend,nginx}.log`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make stop && make clean && make install && make dev-daemon`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
@@ -0,0 +1,71 @@
|
||||
.env
|
||||
Dockerfile
|
||||
.dockerignore
|
||||
.git
|
||||
.gitignore
|
||||
docker/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
.venv/
|
||||
|
||||
# Web
|
||||
node_modules
|
||||
npm-debug.log
|
||||
.next
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Project specific
|
||||
conf.yaml
|
||||
web/
|
||||
docs/
|
||||
examples/
|
||||
assets/
|
||||
tests/
|
||||
*.log
|
||||
|
||||
# Exclude directories not needed in Docker context
|
||||
# Frontend build only needs frontend/
|
||||
# Backend build only needs backend/
|
||||
scripts/
|
||||
logs/
|
||||
docker/
|
||||
skills/
|
||||
frontend/.next
|
||||
frontend/node_modules
|
||||
backend/.venv
|
||||
backend/htmlcov
|
||||
backend/.coverage
|
||||
*.md
|
||||
!README.md
|
||||
!frontend/README.md
|
||||
!backend/README.md
|
||||
@@ -0,0 +1,87 @@
|
||||
# Serper API Key (Google Search) - https://serper.dev
|
||||
SERPER_API_KEY=your-serper-api-key
|
||||
|
||||
# TAVILY API Key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
|
||||
# Jina API Key
|
||||
JINA_API_KEY=your-jina-api-key
|
||||
|
||||
# InfoQuest API Key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# Browser CORS allowlist for split-origin or port-forwarded deployments (comma-separated exact origins).
|
||||
# Leave unset when using the unified nginx endpoint, e.g. http://localhost:2026.
|
||||
# GATEWAY_CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
|
||||
|
||||
# Optional:
|
||||
# FIRECRAWL_API_KEY=your-firecrawl-api-key
|
||||
# VOLCENGINE_API_KEY=your-volcengine-api-key
|
||||
# OPENAI_API_KEY=your-openai-api-key
|
||||
# GEMINI_API_KEY=your-gemini-api-key
|
||||
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
||||
# STEPFUN_API_KEY=your-stepfun-api-key # OpenAI-compatible, see https://platform.stepfun.com
|
||||
# VLLM_API_KEY=your-vllm-api-key # OpenAI-compatible
|
||||
|
||||
# E2B cloud sandbox API key — required only when using E2BSandboxProvider.
|
||||
# Sign up at https://e2b.dev/dashboard
|
||||
# E2B_API_KEY=your-e2b-api-key
|
||||
# FEISHU_APP_ID=your-feishu-app-id
|
||||
# FEISHU_APP_SECRET=your-feishu-app-secret
|
||||
|
||||
# SLACK_BOT_TOKEN=your-slack-bot-token
|
||||
# SLACK_APP_TOKEN=your-slack-app-token
|
||||
# TELEGRAM_BOT_TOKEN=your-telegram-bot-token
|
||||
# DISCORD_BOT_TOKEN=your-discord-bot-token
|
||||
|
||||
# Enable LangSmith to monitor and debug your LLM calls, agent runs, and tool executions.
|
||||
# LANGSMITH_TRACING=true
|
||||
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
# LANGSMITH_API_KEY=your-langsmith-api-key
|
||||
# LANGSMITH_PROJECT=your-langsmith-project
|
||||
|
||||
# GitHub API Token
|
||||
# GITHUB_TOKEN=your-github-token
|
||||
|
||||
# Database (only needed when config.yaml has database.backend: postgres)
|
||||
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
|
||||
#
|
||||
# WECOM_BOT_ID=your-wecom-bot-id
|
||||
# WECOM_BOT_SECRET=your-wecom-bot-secret
|
||||
# DINGTALK_CLIENT_ID=your-dingtalk-client-id
|
||||
# DINGTALK_CLIENT_SECRET=your-dingtalk-client-secret
|
||||
|
||||
# Set to "false" to disable Swagger UI, ReDoc, and OpenAPI schema in production
|
||||
# GATEWAY_ENABLE_DOCS=false
|
||||
|
||||
# Shared internal Gateway auth token for multi-worker deployments.
|
||||
# `make up` generates and persists this automatically; set it manually only
|
||||
# when you run Gateway workers outside the bundled deploy script.
|
||||
# DEER_FLOW_INTERNAL_AUTH_TOKEN=your-shared-internal-token
|
||||
|
||||
# ── Frontend SSR → Gateway wiring ─────────────────────────────────────────────
|
||||
# The Next.js server uses these to reach the Gateway during SSR (auth checks,
|
||||
# /api/* rewrites). They default to localhost values that match `make dev` and
|
||||
# `make start`, so most local users do not need to set them.
|
||||
#
|
||||
# Override only when the Gateway is not on localhost:8001 (e.g. when the
|
||||
# frontend and gateway run on different hosts, in containers with a service
|
||||
# alias, or behind a different port). docker-compose already sets these.
|
||||
# DEER_FLOW_INTERNAL_GATEWAY_BASE_URL=http://localhost:8001
|
||||
# DEER_FLOW_TRUSTED_ORIGINS=http://localhost:3000,http://localhost:2026
|
||||
|
||||
# ── Claude Code / Codex CLI subscription as a model provider (optional) ───────
|
||||
# If you configure a ClaudeChatModel / Codex model provider (or an ACP agent)
|
||||
# that reuses your CLI subscription login, prefer passing a token via env over
|
||||
# bind-mounting your whole ~/.claude / ~/.codex into the container. The Gateway
|
||||
# credential loader reads these first, so no directory mount is needed.
|
||||
# CLAUDE_CODE_CREDENTIALS_PATH points at a single .credentials.json (Claude)
|
||||
# rather than the whole dir. docker-compose.cli-auth.yaml is the opt-in
|
||||
# directory-mount fallback for adapters that need the full CLI config.
|
||||
# ACP adapters often take their own env API key (e.g. ANTHROPIC_API_KEY) and
|
||||
# need no mount at all — check the adapter's docs. See SECURITY.md.
|
||||
# CLAUDE_CODE_OAUTH_TOKEN=your-claude-code-oauth-token
|
||||
# ANTHROPIC_AUTH_TOKEN=your-anthropic-auth-token
|
||||
# CLAUDE_CODE_CREDENTIALS_PATH=/path/to/.claude/.credentials.json
|
||||
# CODEX_AUTH_PATH=/path/to/codex/auth.json
|
||||
@@ -0,0 +1,43 @@
|
||||
# Normalize line endings to LF for all text files
|
||||
* text=auto eol=lf
|
||||
|
||||
# Shell scripts and makefiles must always use LF
|
||||
*.sh text eol=lf
|
||||
Makefile text eol=lf
|
||||
**/Makefile text eol=lf
|
||||
|
||||
# Common config/source files
|
||||
*.yml text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.toml text eol=lf
|
||||
*.json text eol=lf
|
||||
*.md text eol=lf
|
||||
*.py text eol=lf
|
||||
*.ts text eol=lf
|
||||
*.tsx text eol=lf
|
||||
*.js text eol=lf
|
||||
*.jsx text eol=lf
|
||||
*.css text eol=lf
|
||||
*.scss text eol=lf
|
||||
*.html text eol=lf
|
||||
*.env text eol=lf
|
||||
|
||||
# Windows scripts
|
||||
*.bat text eol=crlf
|
||||
*.cmd text eol=crlf
|
||||
|
||||
# Binary assets
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.jpeg binary
|
||||
*.gif binary
|
||||
*.webp binary
|
||||
*.ico binary
|
||||
*.pdf binary
|
||||
*.zip binary
|
||||
*.tar binary
|
||||
*.gz binary
|
||||
*.mp4 binary
|
||||
*.mov binary
|
||||
*.woff binary
|
||||
*.woff2 binary
|
||||
@@ -0,0 +1,170 @@
|
||||
name: 🐛 Bug report
|
||||
description: Report something that isn't working so maintainers can reproduce and fix it.
|
||||
title: "[bug] "
|
||||
labels: ["bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for taking the time to file a bug. A clear, reproducible report is the
|
||||
single biggest factor in how fast it gets fixed.
|
||||
|
||||
Please fill in every required field — especially **reproduction steps** and **logs**.
|
||||
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before you start
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/bytedance/deer-flow/issues?q=is%3Aissue) and this is not a duplicate.
|
||||
required: true
|
||||
- label: I can reproduce this on the latest `main`.
|
||||
required: false
|
||||
|
||||
- type: input
|
||||
id: summary
|
||||
attributes:
|
||||
label: Problem summary
|
||||
description: One sentence describing the bug.
|
||||
placeholder: e.g. make dev fails to start the gateway service
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: areas
|
||||
attributes:
|
||||
label: Affected area(s)
|
||||
description: Which part of DeerFlow does this touch? Select all that apply.
|
||||
multiple: true
|
||||
options:
|
||||
- Frontend (UI / Next.js)
|
||||
- Backend API (gateway / endpoints / SSE)
|
||||
- Agents / LangGraph (graph, prompts, langgraph.json)
|
||||
- Sandbox / Docker
|
||||
- Skills
|
||||
- MCP
|
||||
- Config / setup (make, config.yaml, env)
|
||||
- Docs
|
||||
- CI infra
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: The actual behavior. Include the key error lines verbatim.
|
||||
placeholder: When I do X, I expected Y but I got Z.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
placeholder: What did you expect to happen instead?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
description: Exact commands and sequence. Minimal steps that reliably reproduce the problem.
|
||||
placeholder: |
|
||||
1. make check
|
||||
2. make install
|
||||
3. make dev
|
||||
4. ...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant logs
|
||||
description: Paste key lines from logs (for example `logs/gateway.log`, `logs/frontend.log`). Redact secrets.
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: run_mode
|
||||
attributes:
|
||||
label: How are you running DeerFlow?
|
||||
options:
|
||||
- Local (make dev)
|
||||
- Docker (make docker-start)
|
||||
- CI
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS
|
||||
- Linux
|
||||
- Windows
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: platform_details
|
||||
attributes:
|
||||
label: Platform details
|
||||
description: Architecture and shell, if relevant.
|
||||
placeholder: e.g. arm64, zsh
|
||||
|
||||
- type: input
|
||||
id: python_version
|
||||
attributes:
|
||||
label: Python version
|
||||
placeholder: e.g. Python 3.12.9
|
||||
|
||||
- type: input
|
||||
id: node_version
|
||||
attributes:
|
||||
label: Node.js version
|
||||
placeholder: e.g. v22.11.0
|
||||
|
||||
- type: input
|
||||
id: pnpm_version
|
||||
attributes:
|
||||
label: pnpm version
|
||||
placeholder: e.g. 10.26.2
|
||||
|
||||
- type: input
|
||||
id: uv_version
|
||||
attributes:
|
||||
label: uv version
|
||||
placeholder: e.g. 0.7.20
|
||||
|
||||
- type: textarea
|
||||
id: git_info
|
||||
attributes:
|
||||
label: Git state
|
||||
description: Output of `git branch --show-current` and the latest commit SHA.
|
||||
placeholder: |
|
||||
branch: feature/my-branch
|
||||
commit: abcdef1
|
||||
|
||||
- type: textarea
|
||||
id: support_bundle
|
||||
attributes:
|
||||
label: Support bundle summary
|
||||
description: For local setup, configuration, sandbox, or runtime issues, run `make support-bundle` and paste the generated `*-issue-summary.md` here. AI-assisted reports can start from `*-issue-draft.md`, but every REQUIRED placeholder must be replaced before filing. Attach the zip only if a maintainer asks for the evidence bundle or the summary is not enough.
|
||||
placeholder: |
|
||||
## DeerFlow support bundle summary
|
||||
- Triage status: ...
|
||||
- Active signals: ...
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Screenshots, related issues, config snippets (redacted), or anything else that helps triage.
|
||||
@@ -0,0 +1,11 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: 💬 Questions & usage help
|
||||
url: https://github.com/bytedance/deer-flow/discussions/categories/q-a
|
||||
about: "How do I use X? Why does Y behave like that? Ask in Discussions — it gets answered faster and stays searchable."
|
||||
- name: 💡 Ideas & proposals
|
||||
url: https://github.com/bytedance/deer-flow/discussions/categories/ideas
|
||||
about: Have a half-formed idea? Float it in Discussions before opening a formal feature request.
|
||||
- name: 🔒 Report a security vulnerability
|
||||
url: https://github.com/bytedance/deer-flow/security/policy
|
||||
about: Do not open a public issue for security problems. Follow the security policy instead.
|
||||
@@ -0,0 +1,67 @@
|
||||
name: 💡 Feature request
|
||||
description: Propose a new capability or an improvement to an existing one.
|
||||
title: "[feat] "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for the suggestion. For non-trivial features, please open a
|
||||
[Discussion](https://github.com/bytedance/deer-flow/discussions/categories/ideas)
|
||||
first to align on scope before writing code.
|
||||
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before you start
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/bytedance/deer-flow/issues?q=is%3Aissue) and this is not a duplicate.
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Problem / motivation
|
||||
description: What problem does this solve? What is painful today, or what does it unblock?
|
||||
placeholder: "I'm always frustrated when ..."
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
description: Describe the change from a user's / caller's perspective.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: areas
|
||||
attributes:
|
||||
label: Affected area(s)
|
||||
description: Which part of DeerFlow would this touch? Select all that apply.
|
||||
multiple: true
|
||||
options:
|
||||
- Frontend (UI / Next.js)
|
||||
- Backend API (gateway / endpoints / SSE)
|
||||
- Agents / LangGraph (graph, prompts, langgraph.json)
|
||||
- Sandbox / Docker
|
||||
- Skills
|
||||
- MCP
|
||||
- Config / setup
|
||||
- Docs
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
description: Other approaches you weighed and why you discarded them.
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Mockups, links, related issues, or anything else that helps.
|
||||
@@ -0,0 +1,213 @@
|
||||
# Copilot Onboarding Instructions for DeerFlow
|
||||
|
||||
Use this file as the default operating guide for this repository. Follow it first, and only search the codebase when this file is incomplete or incorrect.
|
||||
|
||||
## 1) Repository Summary
|
||||
|
||||
DeerFlow is a full-stack "super agent harness".
|
||||
|
||||
- Backend: Python 3.12, LangGraph + FastAPI gateway, sandbox/tool system, memory, MCP integration.
|
||||
- Frontend: Next.js 16 + React 19 + TypeScript + pnpm.
|
||||
- Local dev entrypoint: root `Makefile` starts backend + frontend + nginx on `http://localhost:2026`.
|
||||
- Docker dev entrypoint: `make docker-*` (mode-aware provisioner startup from `config.yaml`).
|
||||
|
||||
Current repo footprint is medium-large (backend service, frontend app, docker stack, skills library, docs).
|
||||
|
||||
## 2) Runtime and Toolchain Requirements
|
||||
|
||||
Validated in this repo on macOS:
|
||||
|
||||
- Node.js `>=22` (validated with Node `23.11.0`)
|
||||
- pnpm (repo expects lockfile generated by pnpm 10; validated with pnpm `10.26.2` and `10.15.0`)
|
||||
- Python `>=3.12` (CI uses `3.12`)
|
||||
- `uv` (validated with `0.7.20`)
|
||||
- `nginx` (required for `make dev` unified local endpoint)
|
||||
|
||||
Always run from repo root unless a command explicitly says otherwise.
|
||||
|
||||
## 3) Build/Test/Lint/Run - Verified Command Sequences
|
||||
|
||||
These were executed and validated in this repository.
|
||||
|
||||
### A. Bootstrap and install
|
||||
|
||||
1. Check prerequisites:
|
||||
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
Observed: passes when required tools are installed.
|
||||
|
||||
2. Install dependencies (recommended order: backend then frontend, as implemented by `make install`):
|
||||
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
### B. Backend CI-equivalent validation
|
||||
|
||||
Run from `backend/`:
|
||||
|
||||
```bash
|
||||
make lint
|
||||
make test
|
||||
```
|
||||
|
||||
Validated results:
|
||||
|
||||
- `make lint`: pass (`ruff check .`)
|
||||
- `make test`: pass (`277 passed, 15 warnings in ~76.6s`)
|
||||
|
||||
CI parity:
|
||||
|
||||
- `.github/workflows/backend-unit-tests.yml` runs on pull requests.
|
||||
- CI executes `uv sync --group dev`, then `make lint`, then `make test` in `backend/`.
|
||||
|
||||
### C. Frontend validation
|
||||
|
||||
Run from `frontend/`.
|
||||
|
||||
Recommended reliable sequence:
|
||||
|
||||
```bash
|
||||
pnpm lint
|
||||
pnpm typecheck
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
```
|
||||
|
||||
Observed failure modes and workarounds:
|
||||
|
||||
- `pnpm build` fails without `BETTER_AUTH_SECRET` in production-mode env validation.
|
||||
- Workaround: set `BETTER_AUTH_SECRET` (best) or set `SKIP_ENV_VALIDATION=1`.
|
||||
- Even with `SKIP_ENV_VALIDATION=1`, Better Auth can still warn/error in logs about default secret; prefer setting a real non-default secret.
|
||||
- `pnpm check` currently fails (`next lint` invocation is incompatible here and resolves to an invalid directory). Do not rely on `pnpm check`; run `pnpm lint` and `pnpm typecheck` explicitly.
|
||||
|
||||
### D. Run locally (all services)
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- Stops existing local services first.
|
||||
- Starts Gateway (`8001`, with the embedded LangGraph-compatible runtime), Frontend (`3000`), nginx (`2026`). There is no standalone LangGraph service.
|
||||
- Unified app endpoint: `http://localhost:2026`.
|
||||
- Logs: `logs/gateway.log`, `logs/frontend.log`, `logs/nginx.log`.
|
||||
|
||||
Stop services:
|
||||
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
If tool sessions/timeouts interrupt `make dev`, run `make stop` again to ensure cleanup.
|
||||
|
||||
### E. Config bootstrap
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Important behavior:
|
||||
|
||||
- This intentionally aborts if `config.yaml` (or `config.yml`/`configure.yml`) already exists.
|
||||
- Use `make config` only for first-time setup in a clean clone.
|
||||
|
||||
## 4) Command Order That Minimizes Failures
|
||||
|
||||
Use this exact order for local code changes:
|
||||
|
||||
1. `make check`
|
||||
2. `make install` (if frontend fails with proxy errors, rerun frontend install with proxy vars unset)
|
||||
3. Backend checks: `cd backend && make lint && make test`
|
||||
4. Frontend checks: `cd frontend && pnpm lint && pnpm typecheck`
|
||||
5. Frontend build (if UI changes or release-sensitive changes): `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
Always run backend lint/tests before opening PRs because that is what CI enforces.
|
||||
|
||||
## 5) Project Layout and Architecture (High-Value Paths)
|
||||
|
||||
Root-level orchestration and config:
|
||||
|
||||
- `Makefile` - main local/dev/docker command entrypoints
|
||||
- `config.example.yaml` - primary app config template
|
||||
- `config.yaml` - local active config (gitignored)
|
||||
- `docker/docker-compose-dev.yaml` - Docker dev topology
|
||||
- `.github/workflows/backend-unit-tests.yml` - PR validation workflow
|
||||
|
||||
Backend core:
|
||||
|
||||
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
|
||||
- `backend/app/gateway/` - FastAPI gateway API
|
||||
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
|
||||
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
|
||||
- `backend/packages/harness/deerflow/mcp/` - MCP integration
|
||||
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
|
||||
- `backend/pyproject.toml` - Python deps and `requires-python`
|
||||
- `backend/ruff.toml` - lint/format policy
|
||||
- `backend/tests/` - backend unit and integration-like tests
|
||||
|
||||
Frontend core:
|
||||
|
||||
- `frontend/src/app/` - Next.js routes/pages
|
||||
- `frontend/src/components/` - UI components
|
||||
- `frontend/src/core/` - app logic (threads, tools, API, models)
|
||||
- `frontend/src/env.js` - env schema/validation (critical for build behavior)
|
||||
- `frontend/package.json` - scripts/deps
|
||||
- `frontend/eslint.config.js` - lint rules
|
||||
- `frontend/tsconfig.json` - TS config
|
||||
|
||||
Skills and assets:
|
||||
|
||||
- `skills/public/` - built-in skill packs loaded by agent runtime
|
||||
|
||||
## 6) Pre-Checkin / Validation Expectations
|
||||
|
||||
Before submitting changes, run at minimum:
|
||||
|
||||
- Backend: `cd backend && make lint && make test`
|
||||
- Frontend (if touched): `cd frontend && pnpm lint && pnpm typecheck`
|
||||
- Frontend build when changing env/auth/routing/build-sensitive files: `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
If touching orchestration/config (`Makefile`, `docker/*`, `config*.yaml`), also run `make dev` and verify the four services start.
|
||||
|
||||
## 7) Non-Obvious Dependencies and Gotchas
|
||||
|
||||
- Proxy env vars can silently break frontend network operations (`pnpm install`/registry access).
|
||||
- `BETTER_AUTH_SECRET` is effectively required for reliable frontend production build validation.
|
||||
- Next.js may warn about multiple lockfiles and workspace root inference; this is currently a warning, not a build blocker.
|
||||
- `make config` is non-idempotent by design when config already exists.
|
||||
- `make dev` includes process cleanup and can emit shutdown logs/noise if interrupted; this is expected.
|
||||
|
||||
## 8) Root Inventory (quick reference)
|
||||
|
||||
Important root entries:
|
||||
|
||||
- `.github/`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `docker/`
|
||||
- `skills/`
|
||||
- `scripts/`
|
||||
- `docs/`
|
||||
- `README.md`
|
||||
- `CONTRIBUTING.md`
|
||||
- `Makefile`
|
||||
- `config.example.yaml`
|
||||
- `extensions_config.example.json`
|
||||
|
||||
## 9) Instruction Priority
|
||||
|
||||
Trust this onboarding guide first.
|
||||
|
||||
Only do broad repo searches (`grep/find/code search`) when:
|
||||
|
||||
- you need file-level implementation details not listed here,
|
||||
- a command here fails and you need updated replacement behavior,
|
||||
- or CI/workflow definitions have changed since this file was written.
|
||||
@@ -0,0 +1,119 @@
|
||||
# Declarative label source of truth for DeerFlow.
|
||||
#
|
||||
# This file is the single source of truth for repository labels used by the
|
||||
# auto-labeling workflows (.github/workflows/pr-labeler.yml, pr-triage.yml,
|
||||
# issue-triage.yml). Auto-labelers can only apply labels that already exist,
|
||||
# so every label referenced by a workflow MUST be declared here.
|
||||
#
|
||||
# Apply with: uv run --with pyyaml python scripts/sync_labels.py [--repo OWNER/NAME]
|
||||
# CI keeps it in sync via .github/workflows/label-sync.yml (runs on changes here).
|
||||
#
|
||||
# Sync is additive/update-only: it creates or updates the labels listed below
|
||||
# and never deletes labels that are not listed.
|
||||
#
|
||||
# Color = 6-digit hex without the leading '#'.
|
||||
|
||||
labels:
|
||||
# ── Type ─────────────────────────────────────────────────────────────────
|
||||
# Mostly GitHub defaults; declared here so colors/descriptions stay stable
|
||||
# and so issue templates can rely on them existing.
|
||||
- name: bug
|
||||
color: d73a4a
|
||||
description: Something isn't working
|
||||
- name: enhancement
|
||||
color: a2eeef
|
||||
description: New feature or request
|
||||
- name: documentation
|
||||
color: 0075ca
|
||||
description: Improvements or additions to documentation
|
||||
- name: question
|
||||
color: d876e3
|
||||
description: Further information is requested
|
||||
|
||||
# ── Area (auto, by changed paths — see .github/labeler.yml) ───────────────
|
||||
# Mirrors the "Surface area" section of the pull request template.
|
||||
- name: "area:frontend"
|
||||
color: c5def5
|
||||
description: Next.js frontend under frontend/
|
||||
- name: "area:backend"
|
||||
color: c5def5
|
||||
description: Gateway / runtime / core backend under backend/
|
||||
- name: "area:agents"
|
||||
color: c5def5
|
||||
description: Agents, subagents, graph wiring, prompts, langgraph.json
|
||||
- name: "area:sandbox"
|
||||
color: c5def5
|
||||
description: Sandboxed execution and docker/
|
||||
- name: "area:skills"
|
||||
color: c5def5
|
||||
description: Skills under skills/ or the skills harness
|
||||
- name: "area:mcp"
|
||||
color: c5def5
|
||||
description: Model Context Protocol integration
|
||||
- name: "area:ci"
|
||||
color: c5def5
|
||||
description: GitHub Actions, CI config, repo tooling
|
||||
- name: "area:docs"
|
||||
color: c5def5
|
||||
description: Documentation and Markdown only
|
||||
- name: "area:deps"
|
||||
color: c5def5
|
||||
description: Dependency manifests / lockfiles
|
||||
|
||||
# ── Size (auto, by additions + deletions — see pr-triage.yml) ─────────────
|
||||
- name: "size/XS"
|
||||
color: "009900"
|
||||
description: PR changes < 20 lines
|
||||
- name: "size/S"
|
||||
color: 77bb00
|
||||
description: PR changes 20-100 lines
|
||||
- name: "size/M"
|
||||
color: eebb00
|
||||
description: PR changes 100-300 lines
|
||||
- name: "size/L"
|
||||
color: ee9900
|
||||
description: PR changes 300-700 lines
|
||||
- name: "size/XL"
|
||||
color: ee5500
|
||||
description: PR changes 700+ lines
|
||||
|
||||
# ── Risk (auto, by changed paths — see pr-triage.yml) ─────────────────────
|
||||
- name: "risk:low"
|
||||
color: 0e8a16
|
||||
description: "Low risk: docs / i18n / assets only"
|
||||
- name: "risk:medium"
|
||||
color: fbca04
|
||||
description: "Medium risk: regular code changes"
|
||||
- name: "risk:high"
|
||||
color: b60205
|
||||
description: "High risk: backend API, agents, sandbox, auth, deps, CI"
|
||||
|
||||
# ── Priority (manual) ─────────────────────────────────────────────────────
|
||||
- name: P0
|
||||
color: b60205
|
||||
description: Critical priority
|
||||
- name: P1
|
||||
color: d93f0b
|
||||
description: Major priority
|
||||
- name: P2
|
||||
color: e99695
|
||||
description: Normal priority
|
||||
|
||||
# ── Status (auto + manual) ────────────────────────────────────────────────
|
||||
- name: needs-triage
|
||||
color: fef2c0
|
||||
description: Awaiting maintainer triage
|
||||
- name: needs-validation
|
||||
color: d4c5f9
|
||||
description: Touches front/back contract surface; needs real-path validation
|
||||
- name: skip-validation
|
||||
color: cccccc
|
||||
description: "Maintainer override: do not auto-add needs-validation on this PR"
|
||||
- name: reviewing
|
||||
color: 5319e7
|
||||
description: A maintainer is reviewing this PR
|
||||
|
||||
# ── Contributor ───────────────────────────────────────────────────────────
|
||||
- name: first-time-contributor
|
||||
color: c2e0c6
|
||||
description: First contribution to this repository — be welcoming
|
||||
@@ -0,0 +1,75 @@
|
||||
<!-- Reference a related issue with #123. Use Fixes / Closes / Resolves to
|
||||
auto-close it on merge. Delete this line if the PR doesn't reference an issue. -->
|
||||
Fixes #
|
||||
|
||||
## Why
|
||||
|
||||
<!-- Why are you opening this PR? Cover two things:
|
||||
- The trigger — what made you write this? A bug you hit, a feature you need,
|
||||
tech debt, or a prod issue?
|
||||
- The pain being addressed — user-facing problem, or what it unblocks.
|
||||
For non-trivial features, please open an issue/discussion first to align on
|
||||
scope before writing code. -->
|
||||
|
||||
|
||||
## What changed
|
||||
|
||||
<!-- Describe the change from a user's / caller's perspective, not as a code diff. e.g.:
|
||||
- "Settings now has a 'Custom endpoint' field, off by default"
|
||||
- "Backend /api/chat gains a `stream` flag, defaults to false"
|
||||
- "Default model changed from X to Y — existing users notice on first run" -->
|
||||
|
||||
|
||||
## Surface area
|
||||
|
||||
<!-- Check every box that applies. Reviewers use this to scope the review. -->
|
||||
|
||||
- [ ] **Frontend UI** — page / component / setting / interaction under `frontend/`
|
||||
- [ ] **Backend API** — endpoint / SSE event / request-response shape under `backend/app`
|
||||
- [ ] **Agents / LangGraph** — agent node, graph wiring, `langgraph.json`, or prompt change
|
||||
- [ ] **Sandbox** — `docker/` or sandboxed execution
|
||||
- [ ] **Skills** — change under `skills/`
|
||||
- [ ] **Dependencies** — new/upgraded entry in `backend/pyproject.toml` or `frontend/package.json` (say what it buys us)
|
||||
- [ ] **Default behavior change** — changes existing behavior without the user opting in (default model, default setting, data shape)
|
||||
- [ ] **Docs / tests / CI only** — no runtime behavior change
|
||||
|
||||
|
||||
## Screenshots / Recording
|
||||
|
||||
<!-- If you checked "Frontend UI", attach screenshots showing the entry point —
|
||||
where users discover the change — not just the feature in isolation.
|
||||
Before/after is best for behavior changes. Short GIFs welcome. -->
|
||||
|
||||
|
||||
## Bug fix verification
|
||||
|
||||
<!-- Skip (delete) this section if this PR is not a bug fix.
|
||||
|
||||
Bugs should be encoded as a failing test that goes red before the fix.
|
||||
Confirm:
|
||||
- Test path that reproduces the bug:
|
||||
- Did it go red on `main` and green on this branch? (yes / no)
|
||||
- If a red test wasn't cheap to write, explain why and what you did instead. -->
|
||||
|
||||
|
||||
## Validation
|
||||
|
||||
<!-- What you actually ran. Run at least the checks for the area you changed:
|
||||
Backend: cd backend && make lint && make test
|
||||
Frontend: cd frontend && pnpm format && pnpm lint && pnpm typecheck && BETTER_AUTH_SECRET=local-dev-secret pnpm build && make test
|
||||
Frontend E2E (if you touched frontend/): cd frontend && make test-e2e -->
|
||||
|
||||
|
||||
## AI assistance
|
||||
|
||||
<!-- DeerFlow is an AI project — most PRs here use AI coding tools, and that's
|
||||
welcome. Disclosing it just helps reviewers calibrate how closely to read the
|
||||
diff. Please fill all three; don't delete the section. -->
|
||||
|
||||
**Tool(s) used:** <!-- e.g. Claude Code, Cursor, GitHub Copilot, Codex, Windsurf, or "none" -->
|
||||
|
||||
**How you used it:** <!-- e.g. "generated the module from a spec", "autocomplete only",
|
||||
"AI wrote tests, I wrote the impl". A prompt or conversation link is great too. -->
|
||||
|
||||
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
name: Backend Blocking IO
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main", "2.0.x-dev"]
|
||||
paths:
|
||||
- "backend/**"
|
||||
- ".github/workflows/backend-blocking-io-tests.yml"
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- "backend/**"
|
||||
- ".github/workflows/backend-blocking-io-tests.yml"
|
||||
|
||||
concurrency:
|
||||
group: blocking-io-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-blocking-io:
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Run blocking IO regression tests
|
||||
working-directory: backend
|
||||
run: make test-blocking-io
|
||||
@@ -0,0 +1,40 @@
|
||||
name: Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main', '2.0.x-dev' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
concurrency:
|
||||
group: unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Run unit tests of backend
|
||||
working-directory: backend
|
||||
run: make test
|
||||
@@ -0,0 +1,87 @@
|
||||
name: Publish Helm Chart
|
||||
|
||||
# Publishes the DeerFlow Helm chart as an OCI artifact to GHCR alongside the
|
||||
# container images (see container.yaml). Triggers on the same `v*` tags.
|
||||
#
|
||||
# On pull requests touching the chart or config.example.yaml, `validate-chart`
|
||||
# runs lint + template render + a config_version drift check (the chart's
|
||||
# embedded config_version must not lag config.example.yaml) so a broken or
|
||||
# stale chart fails the PR, not the release.
|
||||
#
|
||||
# Users then install with:
|
||||
# helm install deer-flow oci://ghcr.io/${{ owner }}/deer-flow --version <ver>
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
pull_request:
|
||||
paths:
|
||||
- "deploy/helm/deer-flow/**"
|
||||
- "config.example.yaml"
|
||||
- ".github/workflows/chart.yaml"
|
||||
- "scripts/check_config_version.sh"
|
||||
|
||||
jobs:
|
||||
validate-chart:
|
||||
# Runs on PRs and release tags: catch a broken render or a stale
|
||||
# config_version before merge / publish. A broken chart published under a
|
||||
# vX.Y.Z tag is an immutable OCI artifact (GHCR won't let you overwrite
|
||||
# --version), so a regression must fail here, not on install.
|
||||
if: github.event_name == 'pull_request' || startsWith(github.ref, 'refs/tags/v')
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
|
||||
# ubuntu-latest ships with helm 3 preinstalled - no setup-helm action needed.
|
||||
- name: Lint chart
|
||||
run: helm lint deploy/helm/deer-flow
|
||||
- name: Validate templates render
|
||||
run: helm template deer-flow deploy/helm/deer-flow --include-crds >/dev/null
|
||||
|
||||
# The chart's `config:` block embeds a config_version that must not fall
|
||||
# behind config.example.yaml. A stale version is silent in-cluster (the
|
||||
# image ships no example to compare against, so _check_config_version
|
||||
# never warns) but means the chart's config is authored against an older
|
||||
# schema. Bump it in values.yaml and the README example. config_version
|
||||
# gates no runtime behavior - it only drives the outdated-warning - so a
|
||||
# bare version bump needs no field changes. Logic lives in
|
||||
# scripts/check_config_version.sh (shared with nightly.yaml).
|
||||
- name: config_version drift check
|
||||
run: bash scripts/check_config_version.sh
|
||||
|
||||
verify-versions:
|
||||
# Gate the release: every version source must match the v* tag. A forgotten
|
||||
# bump in Chart.yaml, pyproject.toml, or package.json fails here and skips
|
||||
# the publish. See scripts/verify_versions.sh.
|
||||
if: startsWith(github.ref, 'refs/tags/v')
|
||||
uses: ./.github/workflows/verify-versions.yml
|
||||
|
||||
publish-chart:
|
||||
if: startsWith(github.ref, 'refs/tags/v')
|
||||
needs: [verify-versions, validate-chart]
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
|
||||
- name: Log in to GHCR
|
||||
run: |
|
||||
echo "${{ secrets.GITHUB_TOKEN }}" | \
|
||||
helm registry login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
|
||||
- name: Package chart
|
||||
run: helm package deploy/helm/deer-flow --destination ./packages
|
||||
|
||||
- name: Push chart to GHCR
|
||||
run: |
|
||||
for pkg in ./packages/*.tgz; do
|
||||
echo "--- pushing $pkg"
|
||||
helm push "$pkg" oci://ghcr.io/${{ github.repository_owner }}
|
||||
done
|
||||
@@ -0,0 +1,162 @@
|
||||
name: Publish Containers
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
verify-versions:
|
||||
# Gate the release: every version source must match the v* tag. A forgotten
|
||||
# bump in Chart.yaml, pyproject.toml, or package.json fails here and skips
|
||||
# all image builds. See scripts/verify_versions.sh.
|
||||
uses: ./.github/workflows/verify-versions.yml
|
||||
|
||||
backend-container:
|
||||
needs: verify-versions
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-backend
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
# Bake the `postgres` extra into the published image so multi-replica
|
||||
# deployments (K8s/Helm) can use shared Postgres persistence instead of
|
||||
# file-based SQLite. sqlite/redis-only single-replica setups still work;
|
||||
# this only adds the Postgres driver. See backend/Dockerfile `UV_EXTRAS`.
|
||||
build-args: |
|
||||
UV_EXTRAS=postgres
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be #v2.4.0
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
frontend-container:
|
||||
needs: verify-versions
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-frontend
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: frontend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be #v2.4.0
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
provisioner-container:
|
||||
needs: verify-versions
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-provisioner
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: docker/provisioner
|
||||
file: docker/provisioner/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be #v2.4.0
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
@@ -0,0 +1,63 @@
|
||||
name: E2E Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main', '2.0.x-dev' ]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
|
||||
concurrency:
|
||||
group: e2e-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-tests:
|
||||
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
working-directory: frontend
|
||||
run: npx playwright install chromium --with-deps
|
||||
|
||||
- name: Run E2E tests
|
||||
working-directory: frontend
|
||||
run: pnpm exec playwright test
|
||||
env:
|
||||
SKIP_ENV_VALIDATION: '1'
|
||||
|
||||
- name: Upload Playwright report
|
||||
uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
with:
|
||||
name: playwright-report
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
@@ -0,0 +1,43 @@
|
||||
name: Frontend Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main', '2.0.x-dev' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
concurrency:
|
||||
group: frontend-unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
frontend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Run unit tests of frontend
|
||||
working-directory: frontend
|
||||
run: make test
|
||||
@@ -0,0 +1,38 @@
|
||||
name: Label Sync
|
||||
|
||||
# Keeps repository labels in sync with the declarative source of truth
|
||||
# (.github/labels.yml). Runs whenever that file changes on main, and can be
|
||||
# triggered manually. Additive/update-only — never deletes labels.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- ".github/labels.yml"
|
||||
- "scripts/sync_labels.py"
|
||||
- ".github/workflows/label-sync.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
concurrency:
|
||||
group: label-sync
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
sync:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Sync labels
|
||||
run: uv run --with pyyaml python scripts/sync_labels.py
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
@@ -0,0 +1,78 @@
|
||||
name: Lint Check
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main', '2.0.x-dev' ]
|
||||
pull_request:
|
||||
branches: [ '*' ]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
lint-backend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Check uv.lock is in sync
|
||||
working-directory: backend
|
||||
run: uv lock --check
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: backend
|
||||
run: |
|
||||
uv sync --group dev
|
||||
|
||||
- name: Lint backend
|
||||
working-directory: backend
|
||||
run: make lint
|
||||
|
||||
lint-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm install --frozen-lockfile
|
||||
|
||||
- name: Check frontend formatting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm format
|
||||
|
||||
- name: Run frontend linting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm lint
|
||||
|
||||
- name: Check TypeScript types
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm typecheck
|
||||
|
||||
- name: Build frontend
|
||||
run: |
|
||||
cd frontend
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
@@ -0,0 +1,234 @@
|
||||
name: Nightly Build
|
||||
|
||||
# Nightly build of the three DeerFlow container images (backend, frontend,
|
||||
# provisioner) and the Helm chart, published to GHCR from the default branch.
|
||||
#
|
||||
# This mirrors the tag-driven release workflows (container.yaml + chart.yaml)
|
||||
# but trades the `v*` tag for date-based tags, since nightly ships unreleased
|
||||
# `main`. The `verify-versions` gate is intentionally skipped - there is no tag
|
||||
# to match - and `latest` is left untouched so it keeps tracking the last `v*`
|
||||
# release. Images are amd64-only to match the release builds.
|
||||
#
|
||||
# Artifacts (under the running repo's owner):
|
||||
# ghcr.io/<owner>/deer-flow-{backend,frontend,provisioner}:nightly
|
||||
# ghcr.io/<owner>/deer-flow-{backend,frontend,provisioner}:nightly-YYYYMMDD
|
||||
# oci://ghcr.io/<owner>/deer-flow chart version <base>-nightly.YYYYMMDD-<sha>
|
||||
#
|
||||
# The nightly chart defaults image.tag=nightly and image.registry=ghcr.io/<owner>
|
||||
# (patched in-workflow, never committed), so installing it pulls the matching
|
||||
# nightly images with no values overrides.
|
||||
#
|
||||
# Restricted to the upstream repo: every job is gated on
|
||||
# `github.repository == 'bytedance/deer-flow'`, so a scheduled run or manual
|
||||
# dispatch on a fork skips all jobs rather than pushing to the fork's own GHCR
|
||||
# namespace. Scheduled workflows also only fire on the default branch.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 16 * * *" # 16:00 UTC daily (adjust as needed)
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: nightly
|
||||
# Don't cancel a running nightly - a mid-cancel leaves a half-pushed set.
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
|
||||
jobs:
|
||||
prepare:
|
||||
if: github.repository == 'bytedance/deer-flow'
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
date: ${{ steps.date.outputs.date }}
|
||||
nightly_version: ${{ steps.nightly_version.outputs.nightly_version }}
|
||||
steps:
|
||||
- name: Set nightly date (UTC)
|
||||
id: date
|
||||
# Shared by build-images (image tag) and publish-chart (chart version)
|
||||
# so the two stay in sync for a given run.
|
||||
run: echo "date=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"
|
||||
- name: Checkout repository
|
||||
# Needed to read Chart.yaml's base version for the nightly version
|
||||
# string computed below.
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
- name: Compute nightly version
|
||||
id: nightly_version
|
||||
# Single source of truth for the nightly version string:
|
||||
# <base>-nightly.<YYYYMMDD>-<short_sha> (mirrors the chart's nightly
|
||||
# scheme). build-images injects it into the frontend image (About-page
|
||||
# version) and publish-chart stamps it on Chart.yaml, so the version
|
||||
# users see and the chart version can't drift apart.
|
||||
run: |
|
||||
set -euo pipefail
|
||||
BASE=$(grep -m1 '^version:' deploy/helm/deer-flow/Chart.yaml | awk '{print $2}')
|
||||
# A malformed/missing Chart.yaml version would yield an empty BASE ->
|
||||
# `-nightly.<date>-<sha>` (invalid semver) that now flows into both
|
||||
# the chart publish and the frontend About-page version. Fail loudly.
|
||||
# `pipefail` matters: without it the pipeline's exit code is awk's,
|
||||
# so `set -e` alone wouldn't catch a grep miss.
|
||||
test -n "$BASE" || { echo "::error::empty base version from Chart.yaml"; exit 1; }
|
||||
SHORT_SHA="${GITHUB_SHA::7}"
|
||||
NIGHTLY="${BASE}-nightly.${{ steps.date.outputs.date }}-${SHORT_SHA}"
|
||||
echo "nightly_version=${NIGHTLY}" >> "$GITHUB_OUTPUT"
|
||||
echo "Nightly version: ${NIGHTLY}"
|
||||
|
||||
validate-chart:
|
||||
if: github.repository == 'bytedance/deer-flow'
|
||||
# Catch a broken render or a stale config_version before publish. A broken
|
||||
# chart published under an OCI version is immutable (GHCR won't let you
|
||||
# overwrite --version), so a regression must fail here, not on install.
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
|
||||
- name: Lint chart
|
||||
run: helm lint deploy/helm/deer-flow
|
||||
- name: Validate templates render
|
||||
run: helm template deer-flow deploy/helm/deer-flow --include-crds >/dev/null
|
||||
|
||||
# The chart's `config:` block embeds a config_version that must not fall
|
||||
# behind config.example.yaml. Shared with chart.yaml via
|
||||
# scripts/check_config_version.sh so the two workflows can't drift.
|
||||
- name: config_version drift check
|
||||
run: bash scripts/check_config_version.sh
|
||||
|
||||
build-images:
|
||||
needs: prepare
|
||||
if: github.repository == 'bytedance/deer-flow'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- component: backend
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
# Bake the `postgres` extra so multi-replica (K8s/Helm) deployments
|
||||
# can use shared Postgres. Matches container.yaml's release image.
|
||||
build-args: "UV_EXTRAS=postgres"
|
||||
- component: frontend
|
||||
context: .
|
||||
file: frontend/Dockerfile
|
||||
build-args: ""
|
||||
- component: provisioner
|
||||
context: docker/provisioner
|
||||
file: docker/provisioner/Dockerfile
|
||||
build-args: ""
|
||||
env:
|
||||
IMAGE_NAME: ${{ github.repository }}-${{ matrix.component }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
# `nightly` rolls forward each run; `nightly-YYYYMMDD` is pinned to a
|
||||
# day but mutable within it (a same-day re-dispatch overwrites it);
|
||||
# `sha-<short>` is the only truly immutable tag. No `latest` (that
|
||||
# stays on v* releases) and no branch/tag refs.
|
||||
tags: |
|
||||
type=raw,value=nightly
|
||||
type=raw,value=nightly-${{ needs.prepare.outputs.date }}
|
||||
type=sha
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: ${{ matrix.context }}
|
||||
file: ${{ matrix.file }}
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
# APP_VERSION is consumed only by the frontend Dockerfile (it stamps
|
||||
# the About-page version); backend/provisioner Dockerfiles don't
|
||||
# declare it. Passed via build-arg because build-push-action doesn't
|
||||
# forward host env into the BuildKit build.
|
||||
build-args: |
|
||||
${{ matrix.build-args }}
|
||||
${{ matrix.component == 'frontend' && format('APP_VERSION={0}', needs.prepare.outputs.nightly_version) || '' }}
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@e8998f949152b193b063cb0ec769d69d929409be #v2.4.0
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
publish-chart:
|
||||
# Ship the chart only after images build - it references all three, so a
|
||||
# failed image build withholds the chart rather than publishing one that
|
||||
# would fail to pull in-cluster.
|
||||
needs: [prepare, validate-chart, build-images]
|
||||
if: github.repository == 'bytedance/deer-flow'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
NIGHTLY: ${{ needs.prepare.outputs.nightly_version }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
|
||||
- name: Patch chart to a nightly version + nightly image defaults
|
||||
# Bumps Chart.yaml version/appVersion to the nightly version computed in
|
||||
# the prepare job (<base>-nightly.<date>-<sha>, a valid semver
|
||||
# prerelease; the short SHA makes each dispatch's version unique, so a
|
||||
# same-day re-dispatch re-publishes cleanly - OCI chart versions are
|
||||
# immutable and otherwise can't be overwritten). The same string is
|
||||
# injected into the frontend image, so the About-page version and the
|
||||
# chart version match. Repoints the chart's default image registry/tag
|
||||
# at the nightly build. Patches are in-workflow only - nothing is
|
||||
# committed back.
|
||||
run: |
|
||||
set -eu
|
||||
CHART=deploy/helm/deer-flow
|
||||
echo "Nightly chart version: ${NIGHTLY}"
|
||||
sed -i "s|^version:.*|version: ${NIGHTLY}|" "$CHART/Chart.yaml"
|
||||
sed -i "s|^appVersion:.*|appVersion: \"${NIGHTLY}\"|" "$CHART/Chart.yaml"
|
||||
sed -i "s|^ registry: \"\".*| registry: \"ghcr.io/${OWNER}\"|" "$CHART/values.yaml"
|
||||
sed -i 's|^ tag: "latest"| tag: "nightly"|' "$CHART/values.yaml"
|
||||
# Gate the patches: sed exits 0 even on zero matches, so a drifted
|
||||
# Chart.yaml/values.yaml would otherwise ship a chart that silently
|
||||
# pulls the wrong (release `latest`) images. Fail loudly if any patch
|
||||
# missed.
|
||||
grep -q "^version: ${NIGHTLY}$" "$CHART/Chart.yaml" || { echo "::error::Chart.yaml version sed did not apply"; exit 1; }
|
||||
grep -q "^appVersion: \"${NIGHTLY}\"$" "$CHART/Chart.yaml" || { echo "::error::Chart.yaml appVersion sed did not apply"; exit 1; }
|
||||
grep -q '^ registry: "ghcr.io/' "$CHART/values.yaml" || { echo "::error::values.yaml registry sed did not apply"; exit 1; }
|
||||
grep -q '^ tag: "nightly"' "$CHART/values.yaml" || { echo "::error::values.yaml tag sed did not apply"; exit 1; }
|
||||
echo "--- Chart.yaml (head) ---"; sed -n '1,7p' "$CHART/Chart.yaml"
|
||||
echo "--- image block ---"; sed -n '11,14p' "$CHART/values.yaml"
|
||||
|
||||
- name: Lint patched chart
|
||||
run: helm lint deploy/helm/deer-flow
|
||||
|
||||
- name: Log in to GHCR
|
||||
run: |
|
||||
echo "${{ secrets.GITHUB_TOKEN }}" | \
|
||||
helm registry login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
|
||||
- name: Package and push chart
|
||||
run: |
|
||||
helm package deploy/helm/deer-flow --destination ./packages
|
||||
for pkg in ./packages/*.tgz; do
|
||||
echo "--- pushing $pkg"
|
||||
helm push "$pkg" oci://ghcr.io/${{ github.repository_owner }}
|
||||
done
|
||||
@@ -0,0 +1,108 @@
|
||||
name: Replay E2E (front-back contract)
|
||||
|
||||
# Guards the front-back contract via record/replay (no API key in CI):
|
||||
# Layer 1 — backend golden: replay a recorded trace through the real gateway,
|
||||
# assert the SSE event sequence matches the committed golden.
|
||||
# Layer 2 — full-stack render: real Next.js frontend + real gateway (replay
|
||||
# model) + Chromium; assert the replayed turns render in the browser.
|
||||
# Triggered by changes on EITHER side of the contract so a backend change can no
|
||||
# longer pass without the frontend-facing checks running.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main", "2.0.x-dev"]
|
||||
paths:
|
||||
- "frontend/**"
|
||||
- "backend/app/gateway/**"
|
||||
- "backend/packages/harness/**"
|
||||
- "backend/tests/fixtures/replay/**"
|
||||
- "backend/tests/replay_provider.py"
|
||||
- "backend/tests/_replay_fixture.py"
|
||||
- "backend/tests/seed_runs_router.py"
|
||||
- "backend/tests/test_replay_golden.py"
|
||||
- "backend/scripts/run_replay_gateway.py"
|
||||
- ".github/workflows/replay-e2e.yml"
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- "frontend/**"
|
||||
- "backend/app/gateway/**"
|
||||
- "backend/packages/harness/**"
|
||||
- "backend/tests/fixtures/replay/**"
|
||||
- "backend/tests/replay_provider.py"
|
||||
- "backend/tests/_replay_fixture.py"
|
||||
- "backend/tests/seed_runs_router.py"
|
||||
- "backend/tests/test_replay_golden.py"
|
||||
- "backend/scripts/run_replay_gateway.py"
|
||||
- ".github/workflows/replay-e2e.yml"
|
||||
|
||||
concurrency:
|
||||
group: replay-e2e-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-replay-golden:
|
||||
name: Layer 1 — backend golden (no API key)
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
- name: Replay golden (backend SSE contract)
|
||||
working-directory: backend
|
||||
run: PYTHONPATH=. uv run pytest tests/test_replay_golden.py -v
|
||||
|
||||
fullstack-replay-render:
|
||||
name: Layer 2 — full-stack render (no API key)
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
- name: Install backend dependencies (replay gateway)
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "22"
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
- name: Install Playwright Chromium
|
||||
working-directory: frontend
|
||||
run: npx playwright install chromium --with-deps
|
||||
- name: Full-stack replay render (DOM assertions are the gate)
|
||||
working-directory: frontend
|
||||
run: pnpm exec playwright test -c playwright.real-backend.config.ts
|
||||
- name: Upload report + render artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
with:
|
||||
name: replay-render
|
||||
path: |
|
||||
frontend/playwright-report/
|
||||
frontend/test-results/
|
||||
retention-days: 7
|
||||
@@ -0,0 +1,70 @@
|
||||
name: Skill Review CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main", "2.0.x-dev"]
|
||||
paths:
|
||||
- "skills/public/**"
|
||||
- "backend/packages/harness/deerflow/skills/review/**"
|
||||
- "contracts/skill_review/**"
|
||||
- "scripts/review_changed_public_skills.py"
|
||||
- "backend/pyproject.toml"
|
||||
- "backend/uv.lock"
|
||||
- ".github/workflows/skill-review-ci.yml"
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- "skills/public/**"
|
||||
- "backend/packages/harness/deerflow/skills/review/**"
|
||||
- "contracts/skill_review/**"
|
||||
- "scripts/review_changed_public_skills.py"
|
||||
- "backend/pyproject.toml"
|
||||
- "backend/uv.lock"
|
||||
- ".github/workflows/skill-review-ci.yml"
|
||||
|
||||
concurrency:
|
||||
group: skill-review-ci-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
skill-review:
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Review changed public skills (pull request)
|
||||
if: github.event_name == 'pull_request'
|
||||
working-directory: backend
|
||||
run: |
|
||||
uv run python ../scripts/review_changed_public_skills.py \
|
||||
--base-ref "${{ github.event.pull_request.base.sha }}" \
|
||||
--head-ref "${{ github.event.pull_request.head.sha }}"
|
||||
|
||||
- name: Review changed public skills (push)
|
||||
if: github.event_name == 'push'
|
||||
working-directory: backend
|
||||
run: |
|
||||
uv run python ../scripts/review_changed_public_skills.py \
|
||||
--before "${{ github.event.before }}" \
|
||||
--after "${{ github.event.after }}"
|
||||
@@ -0,0 +1,223 @@
|
||||
name: Triage
|
||||
|
||||
# One workflow for all event-driven PR/issue labeling. Replaces the former
|
||||
# pr-labeler / pr-triage / issue-triage workflows (and drops actions/labeler).
|
||||
#
|
||||
# Design notes:
|
||||
# * All jobs are pure-metadata: they read changed-file lists / PR fields / the
|
||||
# review payload via the API and write labels. PR code is NEVER checked out
|
||||
# or executed, so pull_request_target is safe here.
|
||||
# * Each job only reconciles labels in namespaces IT owns
|
||||
# (area:* / size/* / risk:* / needs-validation). It never touches labels
|
||||
# applied by maintainers or other tools (bug, priority, etc.). first-time-
|
||||
# contributor and reviewing are add-only.
|
||||
# * State is read LIVE (listFiles + listLabelsOnIssue) at run time, not from
|
||||
# the (stale) event payload, so rapid synchronize events converge instead
|
||||
# of thrashing.
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
issues:
|
||||
types: [opened]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
# ── PR: area / size / risk / needs-validation / first-time ─────────────────
|
||||
pr-labels:
|
||||
if: github.event_name == 'pull_request_target' && github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-pr-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- name: Apply PR labels from live state
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const pr = context.payload.pull_request;
|
||||
const { owner, repo } = context.repo;
|
||||
const num = pr.number;
|
||||
|
||||
// ---- live changed files ----
|
||||
const files = await github.paginate(github.rest.pulls.listFiles, {
|
||||
owner, repo, pull_number: num, per_page: 100,
|
||||
});
|
||||
const paths = files.map(f => f.filename);
|
||||
const m = (re) => paths.some(p => re.test(p));
|
||||
|
||||
// ---- area: replaces .github/labeler.yml (path -> area) ----
|
||||
const AREA_RULES = [
|
||||
['area:frontend', [/^frontend\//]],
|
||||
['area:backend', [/^backend\/app\//, /^backend\/packages\/harness\/deerflow\/(runtime|persistence|config|tools|guardrails|tracing|models|utils|uploads)\//]],
|
||||
['area:agents', [/^backend\/packages\/harness\/deerflow\/(agents|subagents|reflection)\//, /(^|\/)langgraph\.json$/, /^backend\/.*\/prompts\//]],
|
||||
['area:sandbox', [/^docker\//, /^backend\/packages\/harness\/deerflow\/sandbox\//, /(^|\/)Dockerfile$/]],
|
||||
['area:skills', [/^skills\//, /^backend\/packages\/harness\/deerflow\/skills\//, /^frontend\/src\/core\/skills\//]],
|
||||
['area:mcp', [/^backend\/packages\/harness\/deerflow\/mcp\//, /^frontend\/src\/core\/mcp\//]],
|
||||
['area:ci', [/^\.github\//, /^scripts\//]],
|
||||
['area:docs', [/^docs\//, /\.mdx?$/]],
|
||||
['area:deps', [/(^|\/)(pyproject\.toml|uv\.lock|package\.json|pnpm-lock\.yaml)$/]],
|
||||
];
|
||||
const areaLabels = AREA_RULES
|
||||
.filter(([, res]) => res.some(re => m(re)))
|
||||
.map(([label]) => label);
|
||||
|
||||
// ---- size: additions+deletions, excluding lockfiles/snapshots ----
|
||||
const EXCLUDE_SIZE = /(^|\/)(uv\.lock|pnpm-lock\.yaml|package-lock\.json)$|\.snap$/;
|
||||
const churn = files
|
||||
.filter(f => !EXCLUDE_SIZE.test(f.filename))
|
||||
.reduce((s, f) => s + (f.additions || 0) + (f.deletions || 0), 0);
|
||||
const sizeLabel =
|
||||
churn < 20 ? 'size/XS' :
|
||||
churn < 100 ? 'size/S' :
|
||||
churn < 300 ? 'size/M' :
|
||||
churn < 700 ? 'size/L' : 'size/XL';
|
||||
|
||||
// ---- risk ----
|
||||
const docsOnly = paths.length > 0 && paths.every(p =>
|
||||
/\.(md|mdx|txt)$/i.test(p) || p.startsWith('docs/') ||
|
||||
/\.(png|jpe?g|gif|svg|webp|ico)$/i.test(p));
|
||||
const highRisk =
|
||||
m(/^backend\/app\/gateway\//) ||
|
||||
m(/^backend\/packages\/harness\/deerflow\/(agents|subagents|sandbox)\//) ||
|
||||
m(/(^|\/)langgraph\.json$/) ||
|
||||
m(/(^|\/)(auth|authz|security)/i) ||
|
||||
m(/(pyproject\.toml|uv\.lock|package\.json|pnpm-lock\.yaml)$/) ||
|
||||
m(/^docker\//) ||
|
||||
m(/^\.github\/workflows\//);
|
||||
const riskLabel = docsOnly ? 'risk:low' : (highRisk ? 'risk:high' : 'risk:medium');
|
||||
|
||||
// ---- needs-validation: front/back contract surface ----
|
||||
const contract =
|
||||
m(/^backend\/app\/gateway\//) ||
|
||||
m(/^backend\/packages\/harness\/deerflow\/(agents|subagents)\//) ||
|
||||
m(/(^|\/)langgraph\.json$/) ||
|
||||
m(/^frontend\/src\/core\/(api|threads|messages)\//);
|
||||
|
||||
// ---- live current labels (NOT the stale event payload) ----
|
||||
const current = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number: num, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
const hasSkip = current.includes('skip-validation');
|
||||
|
||||
// Reconcile ONLY namespaces we own; never touch others.
|
||||
const owned = (n) =>
|
||||
n.startsWith('area:') || n.startsWith('size/') ||
|
||||
n.startsWith('risk:') || n === 'needs-validation';
|
||||
const desired = new Set([...areaLabels, sizeLabel, riskLabel]);
|
||||
if (contract && !hasSkip) desired.add('needs-validation');
|
||||
|
||||
const toRemove = current.filter(n => owned(n) && !desired.has(n));
|
||||
const toAdd = [...desired].filter(n => !current.includes(n));
|
||||
|
||||
// first-time-contributor: add-only, on opened, real users only.
|
||||
if (context.payload.action === 'opened' &&
|
||||
pr.user.type === 'User' &&
|
||||
['FIRST_TIME_CONTRIBUTOR', 'FIRST_TIMER'].includes(pr.author_association) &&
|
||||
!current.includes('first-time-contributor')) {
|
||||
toAdd.push('first-time-contributor');
|
||||
}
|
||||
|
||||
for (const name of toRemove) {
|
||||
try {
|
||||
await github.rest.issues.removeLabel({ owner, repo, issue_number: num, name });
|
||||
} catch (e) {
|
||||
if (e.status !== 404) throw e;
|
||||
}
|
||||
}
|
||||
if (toAdd.length) {
|
||||
await github.rest.issues.addLabels({ owner, repo, issue_number: num, labels: toAdd });
|
||||
}
|
||||
core.info(`area=[${areaLabels.join(',')}] ${sizeLabel} ${riskLabel} churn=${churn} ` +
|
||||
`validation=${desired.has('needs-validation')} ` +
|
||||
`(+${toAdd.join(',') || '-'} / -${toRemove.join(',') || '-'})`);
|
||||
|
||||
# ── PR: reviewing label on a maintainer's human review ─────────────────────
|
||||
reviewing:
|
||||
if: github.event_name == 'pull_request_review'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-review-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- name: Add reviewing label for maintainer reviews
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const { owner, repo } = context.repo;
|
||||
const num = context.payload.pull_request.number;
|
||||
const review = context.payload.review;
|
||||
const assoc = review.author_association; // payload field; no API call
|
||||
const type = review.user && review.user.type;
|
||||
|
||||
// author_association is NONE for every automated reviewer
|
||||
// (Copilot, CodeRabbit, Codex, Sourcery, ...), so this allowlist
|
||||
// drops them all without a denylist — and never calls the
|
||||
// collaborators API that 404s on "Copilot is not a user".
|
||||
// user.type === 'User' guards the rare bot-added-as-collaborator case.
|
||||
if (!['OWNER', 'MEMBER', 'COLLABORATOR'].includes(assoc) || type !== 'User') {
|
||||
core.info(`reviewer ${review.user && review.user.login} assoc=${assoc} type=${type}; skipping.`);
|
||||
return;
|
||||
}
|
||||
|
||||
const labels = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number: num, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
if (labels.includes('reviewing')) {
|
||||
core.info('Already labeled reviewing; skipping.');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await github.rest.issues.addLabels({
|
||||
owner, repo, issue_number: num, labels: ['reviewing'],
|
||||
});
|
||||
core.info('Added "reviewing".');
|
||||
} catch (e) {
|
||||
if (e.status === 403) core.info('No permission to label (expected on some fork PRs).');
|
||||
else throw e;
|
||||
}
|
||||
|
||||
# ── Issue: needs-triage on every new issue ────────────────────────────────
|
||||
issue-triage:
|
||||
if: github.event_name == 'issues'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-issue-${{ github.event.issue.number }}
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- name: Add needs-triage label
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const { owner, repo } = context.repo;
|
||||
const issue_number = context.payload.issue.number;
|
||||
|
||||
// Read live labels (not the event payload) so labels added at creation
|
||||
// time via the API or by another automation are seen — consistent with
|
||||
// the live-state reads in the PR jobs above.
|
||||
const current = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
if (current.includes('needs-triage')) {
|
||||
core.info('Issue already has needs-triage; nothing to do.');
|
||||
return;
|
||||
}
|
||||
// Self-heal: create the label if it does not exist yet.
|
||||
try {
|
||||
await github.rest.issues.createLabel({
|
||||
owner, repo, name: 'needs-triage', color: 'fef2c0',
|
||||
description: 'Awaiting maintainer triage',
|
||||
});
|
||||
} catch (e) {
|
||||
if (e.status !== 422) throw e; // 422 = already exists
|
||||
}
|
||||
await github.rest.issues.addLabels({
|
||||
owner, repo, issue_number, labels: ['needs-triage'],
|
||||
});
|
||||
core.info(`Added needs-triage to #${issue_number}.`);
|
||||
@@ -0,0 +1,27 @@
|
||||
name: Verify Versions
|
||||
|
||||
# Reusable workflow: checks that every project version source agrees with the
|
||||
# git tag that triggered the release. Called by chart.yaml and container.yaml
|
||||
# on v* tags so a forgotten version bump blocks the entire release (container
|
||||
# images + chart), not just the chart.
|
||||
#
|
||||
# Sources verified: deploy/helm/deer-flow/Chart.yaml (version + appVersion),
|
||||
# backend/pyproject.toml, frontend/package.json. Logic lives in
|
||||
# scripts/verify_versions.sh so it can also be run locally.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
verify-versions:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 #v6.0.3
|
||||
|
||||
- name: Verify all version sources match tag
|
||||
run: |
|
||||
TAG_VERSION="${GITHUB_REF_NAME#v}"
|
||||
bash scripts/verify_versions.sh "$TAG_VERSION"
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
# DeerFlow docker image cache
|
||||
docker/.cache/
|
||||
# oh-my-claudecode state
|
||||
.omc/
|
||||
# OS generated files
|
||||
.DS_Store
|
||||
*.local
|
||||
._*
|
||||
.Spotlight-V100
|
||||
.Trashes
|
||||
ehthumbs.db
|
||||
Thumbs.db
|
||||
|
||||
# Python cache
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
venv/
|
||||
|
||||
# Benchmark outputs
|
||||
bench_results.jsonl
|
||||
bench_optimized.jsonl
|
||||
results.jsonl
|
||||
backend/scripts/benchmark/*.jsonl
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
|
||||
# Configuration files
|
||||
config.yaml
|
||||
mcp_config.json
|
||||
extensions_config.json
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Coverage report
|
||||
coverage.xml
|
||||
coverage/
|
||||
.deer-flow/
|
||||
.claude/
|
||||
skills/custom/*
|
||||
logs/
|
||||
log/
|
||||
debug.log
|
||||
|
||||
# Local git hooks (keep only on this machine, do not push)
|
||||
.githooks/
|
||||
|
||||
# pnpm
|
||||
.pnpm-store
|
||||
sandbox_image_cache.tar
|
||||
|
||||
# ignore the legacy `web` folder
|
||||
web/
|
||||
|
||||
# Deployment artifacts
|
||||
backend/Dockerfile.langgraph
|
||||
config.yaml.bak
|
||||
.playwright-mcp
|
||||
/frontend/test-results/
|
||||
/frontend/playwright-report/
|
||||
.gstack/
|
||||
.worktrees
|
||||
@@ -0,0 +1,39 @@
|
||||
repos:
|
||||
# Backend: ruff lint + format via uv (uses the same ruff version as backend deps)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ruff
|
||||
name: ruff lint
|
||||
entry: bash -c 'cd backend && uv run ruff check --fix "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
- id: ruff-format
|
||||
name: ruff format
|
||||
entry: bash -c 'cd backend && uv run ruff format "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
- id: uv-lock-check
|
||||
name: uv lock check
|
||||
entry: bash -c 'cd backend && uv lock --check'
|
||||
language: system
|
||||
pass_filenames: false
|
||||
files: ^backend/(pyproject\.toml|uv\.lock|packages/harness/pyproject\.toml)$
|
||||
|
||||
# Frontend: eslint + prettier (must run from frontend/ for node_modules resolution)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: frontend-eslint
|
||||
name: eslint (frontend)
|
||||
entry: bash -c 'cd frontend && npx eslint --fix "${@/#frontend\//}"' --
|
||||
language: system
|
||||
types_or: [javascript, tsx, ts]
|
||||
files: ^frontend/
|
||||
|
||||
- id: frontend-prettier
|
||||
name: prettier (frontend)
|
||||
entry: bash -c 'cd frontend && npx prettier --write "${@/#frontend\//}"' --
|
||||
language: system
|
||||
files: ^frontend/
|
||||
types_or: [javascript, tsx, ts, json, css]
|
||||
@@ -0,0 +1,137 @@
|
||||
# AGENTS.md
|
||||
|
||||
This file provides guidance to AI coding agents (Claude Code, Codex, and others) when working with code in this repository. It is the source of truth; the sibling `CLAUDE.md` imports it via `@AGENTS.md`.
|
||||
|
||||
It is the **monorepo orientation layer**: it maps the whole repo and points to the
|
||||
module guides that own the depth. For anything inside a module, read that module's
|
||||
guide rather than expecting full detail here:
|
||||
|
||||
- **[backend/AGENTS.md](backend/AGENTS.md)** — backend depth: harness/app split, agent &
|
||||
middleware chain, sandbox, MCP, skills, memory, IM channels, persistence/migrations,
|
||||
config system, test layout.
|
||||
- **[frontend/AGENTS.md](frontend/AGENTS.md)** — frontend depth: Next.js App Router layout,
|
||||
thread/streaming data flow, code style, commands.
|
||||
|
||||
## What is DeerFlow
|
||||
|
||||
DeerFlow is a LangGraph-based AI super-agent system with a full-stack architecture. The
|
||||
backend runs a "super agent" with sandboxed execution, persistent memory, subagent
|
||||
delegation, and extensible tools (built-in, MCP, community), all per-thread isolated. The
|
||||
frontend is a Next.js chat UI. External IM platforms (Feishu, Slack, Telegram, Discord,
|
||||
DingTalk) bridge into the same agent through the Gateway.
|
||||
|
||||
## Service Topology
|
||||
|
||||
A single `make dev` / Docker stack runs four cooperating services:
|
||||
|
||||
| Service | Port | Role |
|
||||
| --------------- | ------ | ------------------------------------------------------------------- |
|
||||
| **Nginx** | `2026` | Unified reverse-proxy entry point — open this in the browser |
|
||||
| **Gateway API** | `8001` | FastAPI REST API + embedded LangGraph-compatible agent runtime |
|
||||
| **Frontend** | `3000` | Next.js web interface |
|
||||
| **Provisioner** | `8002` | Optional — only when sandbox is configured for provisioner/K8s mode |
|
||||
|
||||
Nginx is the single public entry: it serves the frontend and proxies `/api/langgraph/*`
|
||||
to the Gateway's LangGraph runtime, rewriting it to Gateway's native `/api/*` routes; all
|
||||
other `/api/*` go straight to the Gateway REST routers. See
|
||||
[backend/AGENTS.md](backend/AGENTS.md) for the runtime and router detail.
|
||||
|
||||
## Repository Map
|
||||
|
||||
```
|
||||
deer-flow/
|
||||
├── Makefile # Root orchestration: drives the full stack (dev/start/stop, docker, setup)
|
||||
├── config.example.yaml # Template → copy to config.yaml (gitignored) at repo root
|
||||
├── extensions_config.example.json # Template → copy to extensions_config.json (gitignored): MCP servers + skills
|
||||
├── backend/ # Python backend — see backend/AGENTS.md
|
||||
│ ├── Makefile # Per-module backend commands (dev, gateway, test, lint, migrate-rev)
|
||||
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*) — agent framework
|
||||
│ └── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||
├── frontend/ # Next.js frontend (pnpm) — see frontend/AGENTS.md
|
||||
├── docker/ # docker-compose files, nginx config, provisioner
|
||||
├── skills/ # Agent skills: public/ (committed), custom/ (gitignored)
|
||||
├── contracts/ # Cross-component JSON contracts (e.g. subagent status, skill review)
|
||||
├── scripts/ # Root orchestration scripts invoked by the Makefile (check, configure, doctor, support_bundle, serve, nginx, docker, deploy, setup_wizard)
|
||||
├── tests/ # Root-level tests (currently tests/skills/ — public skill tests)
|
||||
└── docs/ # Cross-cutting docs, plans, and design notes
|
||||
```
|
||||
|
||||
Runtime config lives at the **repo root**: copy `config.example.yaml` → `config.yaml`
|
||||
(main app config) and `extensions_config.example.json` → `extensions_config.json` (MCP
|
||||
servers + skills). Both real files are gitignored and may be edited at runtime via the
|
||||
Gateway API. Config schema and resolution order are documented in
|
||||
[backend/AGENTS.md](backend/AGENTS.md).
|
||||
|
||||
Skill quality review note:
|
||||
- `skills/public/skill-reviewer/` is the built-in read-only skill quality reviewer.
|
||||
It uses the harness-layer `review_skill_package` tool and contracts in
|
||||
`contracts/skill_review/`. Model-visible review data is compact and
|
||||
tag-neutralized; full raw payloads stay in tool artifacts. See
|
||||
[backend/AGENTS.md](backend/AGENTS.md) for the non-activation, SkillScan, and
|
||||
`skill-creator` ownership boundaries.
|
||||
|
||||
Scheduled-task note:
|
||||
- The scheduled-task MVP adds a workspace page at `/workspace/scheduled-tasks` plus a background scheduler service gated by `config.yaml -> scheduler.enabled`.
|
||||
- Scheduled background runs are intentionally non-interactive: they execute through the normal run lifecycle, but the lead-agent toolset excludes `ask_clarification` when `context.non_interactive=true`. The key is honored only for internally-authenticated callers (the scheduler launch path); client-supplied `context.non_interactive` is dropped.
|
||||
|
||||
## Commands: Root vs. Module
|
||||
|
||||
**Root `make` targets drive the whole stack** (run from the repo root):
|
||||
|
||||
```bash
|
||||
make setup # Interactive setup wizard (recommended for new users)
|
||||
make doctor # Check configuration and system requirements
|
||||
make support-bundle # Generate redacted troubleshooting summary, AI issue draft, and optional zip
|
||||
make config # Generate local config files from the examples
|
||||
make check # Check that required tools are installed
|
||||
make install # Install all dependencies (frontend + backend + pre-commit hooks)
|
||||
make dev # Start all services with hot-reload (Gateway + Frontend + Nginx)
|
||||
make start # Start all services in production mode (local, optimized)
|
||||
make stop # Stop all running services
|
||||
make up / down # Build/stop the production Docker stack (browser at localhost:2026)
|
||||
make docker-start / docker-stop / docker-logs # Docker development environment
|
||||
```
|
||||
|
||||
Run `make help` for the full list.
|
||||
|
||||
**Per-module commands drive a single module** (run inside that module):
|
||||
|
||||
```bash
|
||||
# Backend (see backend/AGENTS.md for the full set)
|
||||
cd backend && make dev # Gateway API with reload (port 8001)
|
||||
cd backend && make test # Backend test suite
|
||||
cd backend && make lint # ruff check
|
||||
cd backend && make format # ruff format
|
||||
|
||||
# Frontend (see frontend/AGENTS.md for the full set)
|
||||
cd frontend && pnpm dev # Dev server with Turbopack (port 3000)
|
||||
cd frontend && pnpm check # Lint + type check (run before committing)
|
||||
cd frontend && pnpm test # Unit tests
|
||||
```
|
||||
|
||||
Rule of thumb: **root `make` = the full application**; **`backend/Makefile` and `frontend/`
|
||||
(`pnpm`) = per-module work.**
|
||||
|
||||
## Where to Go Next
|
||||
|
||||
- Backend work → **[backend/AGENTS.md](backend/AGENTS.md)**
|
||||
- Frontend work → **[frontend/AGENTS.md](frontend/AGENTS.md)**
|
||||
- Setup & install → **[Install.md](Install.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**
|
||||
- Project overview & usage → **[README.md](README.md)** (translations: `README_zh.md`,
|
||||
`README_ja.md`, `README_fr.md`, `README_ru.md`)
|
||||
- Security policy → **[SECURITY.md](SECURITY.md)**
|
||||
- Changes → **[CHANGELOG.md](CHANGELOG.md)**
|
||||
- Cutting a release → **[RELEASING.md](RELEASING.md)**
|
||||
|
||||
## Cross-Cutting Conventions
|
||||
|
||||
These apply repo-wide; module guides own the module-specific detail.
|
||||
|
||||
- **Documentation update policy** — keep docs in sync with code: update `README.md` for
|
||||
user-facing changes and the relevant `AGENTS.md` for development/architecture changes in
|
||||
the same change set.
|
||||
- **Test-driven development** — features and bug fixes ship with tests. Backend tests live
|
||||
in `backend/tests/` (TDD is mandatory there; see [backend/AGENTS.md](backend/AGENTS.md));
|
||||
frontend tests live in `frontend/tests/`.
|
||||
- **Format before pushing** — run `make format` (backend) / `pnpm check` (frontend). Backend
|
||||
CI enforces `ruff format --check`, so formatting must be clean before a push.
|
||||
+520
@@ -0,0 +1,520 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to DeerFlow are documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [2.0.0] — 2026-06-15
|
||||
|
||||
DeerFlow 2.0 is a ground-up rewrite around a "super agent" harness with
|
||||
sub-agents, persistent memory, sandbox execution, and an extensible
|
||||
skills/tools system. It shares no code with the 1.x line, which now lives on
|
||||
the [`main-1.x` branch](https://github.com/bytedance/deer-flow/tree/main-1.x).
|
||||
|
||||
This release closes [milestone 2.0.0](https://github.com/bytedance/deer-flow/milestone/1)
|
||||
with **180 merged pull requests** since the first 2.0 milestone tag.
|
||||
|
||||
### ⚠ Breaking changes
|
||||
|
||||
- **harness:** Hydrate runs from `RunStore` and persist interrupted status. Run
|
||||
cancellation/multitask semantics now require a working RunStore on the
|
||||
worker that owns the run; cross-worker cancels return 409 instead of
|
||||
silently appearing successful. ([#2932])
|
||||
|
||||
### Added
|
||||
|
||||
#### Agents & runtime
|
||||
- **agent:** Custom-agent self-updates with user isolation — agents can persist
|
||||
edits to their own `SOUL.md` / `config.yaml` from inside a normal chat.
|
||||
([#2713])
|
||||
- **loop-detection:** Make loop detection configurable with per-tool frequency
|
||||
overrides; keep configurable on/off switch. ([#2586], [#2711])
|
||||
- **loop-detection:** Defer warning injection so detector pairs cleanly with
|
||||
tool-call lifecycle. ([#2752])
|
||||
- **run:** Propagate `model_name` from the gateway request through the runtime
|
||||
and persistence stack into the SQLite-backed store. ([#2775])
|
||||
- **subagents:** Stream subagent token usage to the header via terminal task
|
||||
events. ([#2882])
|
||||
- **memory:** Add `memory.token_counting` config to opt out of tiktoken for
|
||||
network-restricted deployments. ([#3465])
|
||||
- **suggest:** Make AI follow-up question suggestions optional. ([#3591])
|
||||
|
||||
#### Models & integrations
|
||||
- **models:** Add StepFun reasoning model adapter. ([#3461])
|
||||
- **community:** Add Brave Search web search tool. ([#3528])
|
||||
- **channels:** Enhance Discord with mention-only mode, thread routing, and
|
||||
typing indicators. ([#2842])
|
||||
- **im:** Add user-owned IM channel connections — users can bind their own
|
||||
Slack/Telegram/Discord/Feishu/DingTalk/WeChat/WeCom accounts on top of the
|
||||
operator-configured bots. ([#3487])
|
||||
- **models:** Add patched MiMo reasoning content support. ([#3298])
|
||||
- **models:** Add MiniMax provider for image/video/podcast skills plus a new
|
||||
music-generation skill. ([#3437])
|
||||
- **community:** Add SearXNG and Browserless web search/fetch tools. ([#3451])
|
||||
- **community:** Add Serper Google Images provider for `image_search`. ([#3575])
|
||||
- **channels:** Stream Telegram agent replies by editing the placeholder
|
||||
message in place. ([#3534])
|
||||
|
||||
#### Observability
|
||||
- **trace:** Set the LangGraph trace name to `lead_agent` (or the custom
|
||||
agent's `agent_name`) for cleaner Langfuse/LangSmith traces. ([#3101])
|
||||
- **frontend:** Refine token usage display modes. ([#2329])
|
||||
- **defaults:** Enable token usage tracking by default. ([#2841])
|
||||
- **defaults:** Raise default summarization trigger threshold. ([#3174])
|
||||
- **trace:** Attribute subagent spans to the parent thread's Langfuse trace.
|
||||
([#3611])
|
||||
|
||||
#### Skills
|
||||
- **skill:** Add `blocking-io-guard` skill for blocking-IO triage and runtime
|
||||
anchors. ([#3503])
|
||||
- **skill:** Add maintainer issue and PR workflow skill. ([#3554])
|
||||
- **skill:** Strengthen the maintainer orchestrator review workflow. ([#3606])
|
||||
|
||||
### Performance
|
||||
|
||||
- **harness:** Push thread metadata filters into SQL instead of post-filtering
|
||||
in Python. ([#2865])
|
||||
- **runtime:** Index runs by `thread_id` to avoid O(n) scans in `RunManager`.
|
||||
([#3499])
|
||||
- **runtime:** Index messages in `MemoryRunEventStore` to avoid O(n) scans.
|
||||
([#3531])
|
||||
- **persistence:** Cache `Base.to_dict` column reflection per class. ([#3654])
|
||||
- **sandbox:** Speed up `should_ignore_name` in glob/grep walks. ([#3657])
|
||||
|
||||
### Security
|
||||
|
||||
- **upload:** Reject symlinked upload destinations. ([#2623])
|
||||
- **uploads:** Add Windows support for safe symlink-protected uploads.
|
||||
([#2794])
|
||||
- **mcp:** Mask sensitive values in MCP config API responses. ([#2667])
|
||||
- **mcp:** Harden the MCP config endpoint against malformed input. ([#3425])
|
||||
- **auth:** Reject cross-site auth POSTs. ([#2740])
|
||||
- **gateway:** Cap skill artifact preview decompression to prevent
|
||||
zip-bomb-style abuse. ([#2963])
|
||||
- **sandbox:** Mount the host Docker socket only in aio (DooD) sandbox mode.
|
||||
([#3517])
|
||||
- **sandbox:** Do not bind-mount host CLI auth dirs by default. ([#3521])
|
||||
|
||||
### Fixed
|
||||
|
||||
#### Runtime, gateway & persistence
|
||||
- **runtime:** Rollback restore checkpoint now supersedes newer checkpoints.
|
||||
([#2582])
|
||||
- **runtime:** Persist run message summaries. ([#2850])
|
||||
- **runtime:** Bound `write_file` execution-failure observations to keep
|
||||
failure traces from blowing out the context. ([#3133])
|
||||
- **runtime:** Protect the sync singleton's init and reset paths. ([#3413])
|
||||
- **runtime:** Avoid PostgreSQL aggregate `FOR UPDATE` on run events.
|
||||
([#2962])
|
||||
- **runs:** Restore historical runs from persistent store after a gateway
|
||||
restart. ([#2989])
|
||||
- **gateway:** Return ISO 8601 timestamps from threads endpoints. ([#2599])
|
||||
- **gateway:** Make cancel idempotent for already-interrupted runs. ([#3058])
|
||||
- **gateway:** Split `stream_existing_run` into per-method routes for unique
|
||||
OpenAPI `operationId`s. ([#3228])
|
||||
- **events:** Serialize structured DB event content. ([#2762])
|
||||
- **persistence:** Emit timezone-aware timestamps from SQLite-backed stores.
|
||||
([#3130])
|
||||
- **persistence:** Reuse token usage model grouping expression. ([#2910])
|
||||
- **runs:** Ignore stale run reconnect conflicts. ([#3284])
|
||||
- **nginx:** Defer CORS to the gateway allowlist instead of double-applying it.
|
||||
([#2861])
|
||||
- **persistence:** Fix runtime journal run lifecycle events. ([#3470])
|
||||
- **gateway:** Enforce thread ownership on stateless run endpoints. ([#3473])
|
||||
- **runtime:** Propagate interrupt through SSE values events for the LangGraph
|
||||
SDK. ([#3605])
|
||||
- **serialization:** Strip base64 image data from streamed values events.
|
||||
([#3631])
|
||||
- **history:** Strip base64 image data from REST endpoint responses. ([#3535])
|
||||
- **gateway:** Attribute token usage to the actual models. ([#3658])
|
||||
|
||||
#### Agents, subagents & middleware
|
||||
- **subagents:** Make subagent timeout terminal state atomic. ([#2583])
|
||||
- **subagents:** Use model override for tools and middleware. ([#2641])
|
||||
- **subagents:** Consolidate `system_prompt` and skills into a single
|
||||
`SystemMessage`. ([#2701])
|
||||
- **subagent:** Isolate subagents from the parent run's checkpointer.
|
||||
([#3559])
|
||||
- **agents:** Make `update_agent` honor `runtime.context` `user_id` like
|
||||
`setup_agent` does. ([#2867])
|
||||
- **agents:** Resolve duplicate `todos` channel type conflict in
|
||||
`TodoMiddleware`. ([#3200])
|
||||
- **agents:** Offload blocking filesystem IO in the custom-agent router off
|
||||
the event loop. ([#3457])
|
||||
- **agents:** Keep new agent bootstrap in user scope. ([#2784])
|
||||
- **loop-detection:** Keep tool-call pairing on warn injection. ([#2725])
|
||||
- **middleware:** Sync raw tool-call metadata. ([#2757])
|
||||
- **middleware:** Handle invalid tool calls in dangling pairing middleware.
|
||||
([#2891])
|
||||
- **middleware:** Prevent todo completion reminder IM-message leak. ([#2907])
|
||||
- **middleware:** Normalize tool result adjacency before model calls.
|
||||
([#2939])
|
||||
- **agents:** Require `config.yaml` in `resolve_agent_dir` to skip memory-only
|
||||
directories. ([#3481])
|
||||
- **agents:** Sync `agent_name` across context/configurable and reject empty
|
||||
soul. ([#3553])
|
||||
- **middleware:** Offload the uploads scan in `UploadsMiddleware` off the event
|
||||
loop. ([#3311])
|
||||
- **middleware:** Offload memory injection off the event loop to prevent
|
||||
tiktoken blocking. ([#3411])
|
||||
- **middleware:** Externalize oversized tool output into the sandbox for
|
||||
non-mounted sandboxes. ([#3417])
|
||||
- **middleware:** Preserve the sandbox reducer in middleware state. ([#3629])
|
||||
- **subagents:** Raise general-purpose `max_turns` to 150 and default timeout to
|
||||
30 min. ([#3610])
|
||||
|
||||
#### Memory & tracing
|
||||
- **memory:** Replace short-lived `asyncio.run()` with a persistent event
|
||||
loop. ([#2627])
|
||||
- **memory:** Isolate queued memory updates by agent. ([#2941])
|
||||
- **memory:** Parse wrapped memory-update JSON responses. ([#3252])
|
||||
- **tracing:** Propagate `session_id` and `user_id` into Langfuse traces.
|
||||
([#2944])
|
||||
- **trace:** Decode unicode escape sequences in non-ASCII memory trace info.
|
||||
([#3104])
|
||||
|
||||
#### Tools, sandbox & MCP
|
||||
- **mcp:** Fix env resolution in MCP config lists. ([#2556])
|
||||
- **models:** Record Codex token usage in `usage_metadata`. ([#2585])
|
||||
- **sandbox:** Supplement `list_running` in `RemoteSandboxBackend`. ([#2716])
|
||||
- **sandbox:** Disable MSYS path conversion for Git Bash on Windows.
|
||||
([#2766])
|
||||
- **sandbox:** Avoid blocking sandbox readiness polling. ([#2822])
|
||||
- **sandbox:** Uphold the `/mnt/user-data` contract at the `Sandbox` API
|
||||
boundary. ([#2881])
|
||||
- **sandbox:** Scope provisioner PVC data by user. ([#2973])
|
||||
- **sandbox:** Merge idempotent sandbox state updates. ([#3518])
|
||||
- **tools:** Introduce `Runtime` type alias to eliminate Pydantic serialization
|
||||
warnings. ([#2774])
|
||||
- **tools:** Preserve `tool_search` promotions across re-entrant
|
||||
`get_available_tools`. ([#2885])
|
||||
- **harness:** Wrap async-only config tools for sync client execution.
|
||||
([#2878])
|
||||
- **harness:** Wrap all async-only tools for sync clients. ([#2935])
|
||||
- **tool-search:** Reliably hide deferred MCP schemas by removing the
|
||||
ContextVar. ([#3342])
|
||||
- **search:** Fix DDGS Wikipedia region handling. ([#3423])
|
||||
- **web_fetch:** Support a proxy for the Jina reader in restricted networks.
|
||||
([#3430])
|
||||
- **sandbox:** Persist lazily-acquired sandbox state via `Command`. ([#3464])
|
||||
- **sandbox:** Fix stale AIO sandbox cache reuse. ([#3494])
|
||||
- **sandbox:** Create a shell session before retrying on a fresh id. ([#3577])
|
||||
- **sandbox:** Stop flagging string-literal path fragments as unsafe absolute
|
||||
paths. ([#3623])
|
||||
- **sandbox:** Return an actionable hint when `read_file` hits a binary file.
|
||||
([#3624])
|
||||
- **mcp:** Make stdio MCP-produced files resolvable via virtual sandbox paths.
|
||||
([#3600])
|
||||
- **mcp:** Surface admin-required state on the settings tools page. ([#3533])
|
||||
- **mcp:** Add a tools cache reset endpoint. ([#3602])
|
||||
- **uploads:** Fix the upload file size contract. ([#3408])
|
||||
|
||||
#### Skills & channels
|
||||
- **skills:** Enforce `allowed-tools` metadata. ([#2626])
|
||||
- **skills:** Harden slash skill activation across chat channels. ([#3466])
|
||||
- **skills:** Fix custom skill install permissions. ([#3241])
|
||||
- **channels:** Authenticate gateway command requests. ([#2742])
|
||||
- **skills:** Surface the offending line and a quoting hint on SKILL.md YAML
|
||||
errors. ([#3335])
|
||||
- **skills:** Keep skill archive installation off the event loop. ([#3505])
|
||||
- **channels:** Ignore hidden control messages when extracting replies.
|
||||
([#3270])
|
||||
- **channels:** Reload config on channel restart. ([#3514])
|
||||
- **channels:** Surface WeCom WebSocket connection failures. ([#3526])
|
||||
- **channels:** Close the Discord file handle after upload. ([#3561])
|
||||
- **channels:** Require a bound identity for user-owned IM messages. ([#3578])
|
||||
- **channels:** Scope IM files and helper commands to the owner. ([#3579])
|
||||
- **channels:** Make runtime provider state authoritative. ([#3580])
|
||||
- **channels:** Harden runtime credential management APIs. ([#3581])
|
||||
- **channels:** Make the channel connect flow deterministic. ([#3582])
|
||||
- **channels:** Centralize shared channel retry helpers. ([#3583])
|
||||
- **channels:** Add operational guardrails. ([#3584])
|
||||
- **channels:** Unsubscribe channel listeners by equality. ([#3608])
|
||||
|
||||
#### Auth
|
||||
- **auth:** Replace setup-status 429 rate limit with a cached response.
|
||||
([#2915])
|
||||
- **auth:** Persist auto-generated JWT secret so it survives restarts.
|
||||
([#2933])
|
||||
- **auth:** Align auth-disabled mode with mock history loading. ([#3471])
|
||||
|
||||
#### Frontend
|
||||
- **frontend:** Restore `localhost` fallback for `getGatewayConfig` in prod
|
||||
mode. ([#2718])
|
||||
- **chat:** Prevent the first user message from being swallowed in new
|
||||
conversations. ([#2731])
|
||||
- **frontend:** Use backend thread token usage for the header total. ([#2800])
|
||||
- **frontend:** Wait for async chat submit before clearing the input.
|
||||
([#2940])
|
||||
- **frontend:** Resolve login page flickering and the resize-observer loop.
|
||||
([#2954])
|
||||
- **frontend:** Deduplicate restored thread messages. ([#2958])
|
||||
- **frontend:** Avoid duplicate optimistic user message. ([#3002])
|
||||
- **frontend:** Hide the copy button for streaming assistant messages.
|
||||
([#3176])
|
||||
- **frontend:** Show a new thread in the sidebar immediately on creation.
|
||||
([#3283])
|
||||
- **frontend:** Isolate new chat thread messages. ([#3508])
|
||||
- **frontend:** Cap deeply nested list indentation to prevent render crashes.
|
||||
([#3393], [#3570])
|
||||
- **token-usage:** Dedupe token usage aggregation by message id. ([#2770])
|
||||
- **frontend:** Fall back to Streamdown clipboard copy. ([#3397])
|
||||
- **frontend:** Remove the Backspace shortcut for deleting prompt attachments.
|
||||
([#3410])
|
||||
- **frontend:** Restructure the Memory settings toolbar into two rows. ([#3433])
|
||||
- **suggestions:** Strip inline `<think>` reasoning before parsing follow-up
|
||||
questions. ([#3435])
|
||||
- **frontend:** Stop fetching follow-up suggestions when they are disabled.
|
||||
([#3599])
|
||||
- **frontend:** Paginate the workspace chat list beyond 50 threads. ([#3485])
|
||||
- **frontend:** Prevent user message bubble overflow with long unbreakable
|
||||
strings. ([#3488])
|
||||
- **frontend:** Keep the workspace interactive when the SSR auth probe cannot
|
||||
reach the gateway. ([#3495])
|
||||
- **frontend:** Render user messages as plain text and cap blockquote nesting.
|
||||
([#3502])
|
||||
- **frontend:** Reset the active chat after deletion. ([#3519])
|
||||
- **frontend:** Improve the mobile workspace layout. ([#3646])
|
||||
- **frontend:** Render full content for multi-part AI messages. ([#3649])
|
||||
|
||||
#### Build, deploy, scripts & config
|
||||
- **packaging:** Add `postgres` extra for store/checkpointer support; clarify
|
||||
install guidance. ([#2584])
|
||||
- **harness:** Resolve runtime paths from the project root. ([#2642])
|
||||
- **docker:** Force nginx to resolve upstream names at request time.
|
||||
([#2717])
|
||||
- **docker:** Default Gateway to a single worker to prevent multi-worker
|
||||
breakage. ([#3475])
|
||||
- **scripts:** Preserve `uv` extras across `make dev` restarts. ([#2767],
|
||||
[#2754])
|
||||
- **scripts:** Clean up local nginx on stop. ([#3005])
|
||||
- **deploy:** Fall back to `python` / `openssl` when `python3` is absent for
|
||||
secret generation. ([#3074])
|
||||
- **config:** Make the reload boundary discoverable from code. ([#3144],
|
||||
[#3153])
|
||||
- **replay-e2e:** Key replay fixtures by caller and conversation. ([#3453])
|
||||
- **setup:** Refresh LLM provider wizard defaults. ([#3421])
|
||||
- **config:** Coerce null `config.yaml` list sections to an empty list. ([#3434])
|
||||
- **scripts:** Exclude runtime state from gateway reload. ([#3426])
|
||||
- **scripts:** Create the backend/sandbox dir before the uvicorn reload-exclude.
|
||||
([#3460])
|
||||
- **scripts:** Stop next-server correctly after `make start-daemon`. ([#3498])
|
||||
- **makefile:** Fix per-commit hooks installation. ([#3569])
|
||||
- **replay-e2e:** Match replay by conversation, not the living system prompt.
|
||||
([#3436])
|
||||
|
||||
### Changed
|
||||
|
||||
- **provider (refactor):** Share assistant payload replay matching across
|
||||
providers. ([#3307])
|
||||
- **lead-agent (refactor):** Make `build_middlewares` public to drop the last
|
||||
cross-module private import. ([#3458])
|
||||
- **todo (refactor):** Remove the unused completion reminder counter. ([#3530])
|
||||
|
||||
### Documentation
|
||||
|
||||
- Document blocking-IO detection usage and maintenance. ([#3233])
|
||||
- Clean standalone LangGraph server remnants from docs. ([#3301])
|
||||
- Add AI assistance disclosure to the PR template and CONTRIBUTING. ([#3398])
|
||||
- Document custom AIO sandbox images. ([#3548])
|
||||
|
||||
### Internal
|
||||
|
||||
- **dev:** Add async/thread boundary detector. ([#2936])
|
||||
- **runtime:** Add lifecycle end-to-end coverage. ([#2946])
|
||||
- **windows:** Add `PYTHONIOENCODING` and `PYTHONUTF8` to backend Makefile
|
||||
targets. ([#3069])
|
||||
- **blocking-io:** Fail-loud repo-root resolution and shared detector CLI
|
||||
shim. ([#3512])
|
||||
- **runtime:** Add a Blockbuster runtime anchor for `JsonlRunEventStore` async
|
||||
IO. ([#3313])
|
||||
- **ci:** Consolidate PR/issue labeling and fix the reviewing-job crash and
|
||||
label thrash. ([#3455])
|
||||
|
||||
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
||||
[#2329]: https://github.com/bytedance/deer-flow/pull/2329
|
||||
[#2556]: https://github.com/bytedance/deer-flow/pull/2556
|
||||
[#2582]: https://github.com/bytedance/deer-flow/pull/2582
|
||||
[#2583]: https://github.com/bytedance/deer-flow/pull/2583
|
||||
[#2584]: https://github.com/bytedance/deer-flow/pull/2584
|
||||
[#2585]: https://github.com/bytedance/deer-flow/pull/2585
|
||||
[#2586]: https://github.com/bytedance/deer-flow/pull/2586
|
||||
[#2599]: https://github.com/bytedance/deer-flow/pull/2599
|
||||
[#2623]: https://github.com/bytedance/deer-flow/pull/2623
|
||||
[#2626]: https://github.com/bytedance/deer-flow/pull/2626
|
||||
[#2627]: https://github.com/bytedance/deer-flow/pull/2627
|
||||
[#2641]: https://github.com/bytedance/deer-flow/pull/2641
|
||||
[#2642]: https://github.com/bytedance/deer-flow/pull/2642
|
||||
[#2667]: https://github.com/bytedance/deer-flow/pull/2667
|
||||
[#2701]: https://github.com/bytedance/deer-flow/pull/2701
|
||||
[#2711]: https://github.com/bytedance/deer-flow/pull/2711
|
||||
[#2713]: https://github.com/bytedance/deer-flow/pull/2713
|
||||
[#2716]: https://github.com/bytedance/deer-flow/pull/2716
|
||||
[#2717]: https://github.com/bytedance/deer-flow/pull/2717
|
||||
[#2718]: https://github.com/bytedance/deer-flow/pull/2718
|
||||
[#2725]: https://github.com/bytedance/deer-flow/pull/2725
|
||||
[#2731]: https://github.com/bytedance/deer-flow/pull/2731
|
||||
[#2740]: https://github.com/bytedance/deer-flow/pull/2740
|
||||
[#2742]: https://github.com/bytedance/deer-flow/pull/2742
|
||||
[#2752]: https://github.com/bytedance/deer-flow/pull/2752
|
||||
[#2754]: https://github.com/bytedance/deer-flow/pull/2754
|
||||
[#2757]: https://github.com/bytedance/deer-flow/pull/2757
|
||||
[#2762]: https://github.com/bytedance/deer-flow/pull/2762
|
||||
[#2766]: https://github.com/bytedance/deer-flow/pull/2766
|
||||
[#2767]: https://github.com/bytedance/deer-flow/pull/2767
|
||||
[#2770]: https://github.com/bytedance/deer-flow/pull/2770
|
||||
[#2774]: https://github.com/bytedance/deer-flow/pull/2774
|
||||
[#2775]: https://github.com/bytedance/deer-flow/pull/2775
|
||||
[#2784]: https://github.com/bytedance/deer-flow/pull/2784
|
||||
[#2794]: https://github.com/bytedance/deer-flow/pull/2794
|
||||
[#2800]: https://github.com/bytedance/deer-flow/pull/2800
|
||||
[#2822]: https://github.com/bytedance/deer-flow/pull/2822
|
||||
[#2841]: https://github.com/bytedance/deer-flow/pull/2841
|
||||
[#2842]: https://github.com/bytedance/deer-flow/pull/2842
|
||||
[#2850]: https://github.com/bytedance/deer-flow/pull/2850
|
||||
[#2861]: https://github.com/bytedance/deer-flow/pull/2861
|
||||
[#2865]: https://github.com/bytedance/deer-flow/pull/2865
|
||||
[#2867]: https://github.com/bytedance/deer-flow/pull/2867
|
||||
[#2878]: https://github.com/bytedance/deer-flow/pull/2878
|
||||
[#2881]: https://github.com/bytedance/deer-flow/pull/2881
|
||||
[#2882]: https://github.com/bytedance/deer-flow/pull/2882
|
||||
[#2885]: https://github.com/bytedance/deer-flow/pull/2885
|
||||
[#2891]: https://github.com/bytedance/deer-flow/pull/2891
|
||||
[#2907]: https://github.com/bytedance/deer-flow/pull/2907
|
||||
[#2910]: https://github.com/bytedance/deer-flow/pull/2910
|
||||
[#2915]: https://github.com/bytedance/deer-flow/pull/2915
|
||||
[#2932]: https://github.com/bytedance/deer-flow/pull/2932
|
||||
[#2933]: https://github.com/bytedance/deer-flow/pull/2933
|
||||
[#2935]: https://github.com/bytedance/deer-flow/pull/2935
|
||||
[#2936]: https://github.com/bytedance/deer-flow/pull/2936
|
||||
[#2939]: https://github.com/bytedance/deer-flow/pull/2939
|
||||
[#2940]: https://github.com/bytedance/deer-flow/pull/2940
|
||||
[#2941]: https://github.com/bytedance/deer-flow/pull/2941
|
||||
[#2944]: https://github.com/bytedance/deer-flow/pull/2944
|
||||
[#2946]: https://github.com/bytedance/deer-flow/pull/2946
|
||||
[#2954]: https://github.com/bytedance/deer-flow/pull/2954
|
||||
[#2958]: https://github.com/bytedance/deer-flow/pull/2958
|
||||
[#2962]: https://github.com/bytedance/deer-flow/pull/2962
|
||||
[#2963]: https://github.com/bytedance/deer-flow/pull/2963
|
||||
[#2973]: https://github.com/bytedance/deer-flow/pull/2973
|
||||
[#2989]: https://github.com/bytedance/deer-flow/pull/2989
|
||||
[#3002]: https://github.com/bytedance/deer-flow/pull/3002
|
||||
[#3005]: https://github.com/bytedance/deer-flow/pull/3005
|
||||
[#3058]: https://github.com/bytedance/deer-flow/pull/3058
|
||||
[#3069]: https://github.com/bytedance/deer-flow/pull/3069
|
||||
[#3074]: https://github.com/bytedance/deer-flow/pull/3074
|
||||
[#3101]: https://github.com/bytedance/deer-flow/pull/3101
|
||||
[#3104]: https://github.com/bytedance/deer-flow/pull/3104
|
||||
[#3130]: https://github.com/bytedance/deer-flow/pull/3130
|
||||
[#3133]: https://github.com/bytedance/deer-flow/pull/3133
|
||||
[#3144]: https://github.com/bytedance/deer-flow/pull/3144
|
||||
[#3153]: https://github.com/bytedance/deer-flow/pull/3153
|
||||
[#3174]: https://github.com/bytedance/deer-flow/pull/3174
|
||||
[#3176]: https://github.com/bytedance/deer-flow/pull/3176
|
||||
[#3200]: https://github.com/bytedance/deer-flow/pull/3200
|
||||
[#3228]: https://github.com/bytedance/deer-flow/pull/3228
|
||||
[#3233]: https://github.com/bytedance/deer-flow/pull/3233
|
||||
[#3241]: https://github.com/bytedance/deer-flow/pull/3241
|
||||
[#3252]: https://github.com/bytedance/deer-flow/pull/3252
|
||||
[#3270]: https://github.com/bytedance/deer-flow/pull/3270
|
||||
[#3283]: https://github.com/bytedance/deer-flow/pull/3283
|
||||
[#3284]: https://github.com/bytedance/deer-flow/pull/3284
|
||||
[#3298]: https://github.com/bytedance/deer-flow/pull/3298
|
||||
[#3301]: https://github.com/bytedance/deer-flow/pull/3301
|
||||
[#3307]: https://github.com/bytedance/deer-flow/pull/3307
|
||||
[#3311]: https://github.com/bytedance/deer-flow/pull/3311
|
||||
[#3313]: https://github.com/bytedance/deer-flow/pull/3313
|
||||
[#3335]: https://github.com/bytedance/deer-flow/pull/3335
|
||||
[#3342]: https://github.com/bytedance/deer-flow/pull/3342
|
||||
[#3393]: https://github.com/bytedance/deer-flow/pull/3393
|
||||
[#3397]: https://github.com/bytedance/deer-flow/pull/3397
|
||||
[#3398]: https://github.com/bytedance/deer-flow/pull/3398
|
||||
[#3408]: https://github.com/bytedance/deer-flow/pull/3408
|
||||
[#3410]: https://github.com/bytedance/deer-flow/pull/3410
|
||||
[#3411]: https://github.com/bytedance/deer-flow/pull/3411
|
||||
[#3413]: https://github.com/bytedance/deer-flow/pull/3413
|
||||
[#3417]: https://github.com/bytedance/deer-flow/pull/3417
|
||||
[#3421]: https://github.com/bytedance/deer-flow/pull/3421
|
||||
[#3423]: https://github.com/bytedance/deer-flow/pull/3423
|
||||
[#3425]: https://github.com/bytedance/deer-flow/pull/3425
|
||||
[#3426]: https://github.com/bytedance/deer-flow/pull/3426
|
||||
[#3430]: https://github.com/bytedance/deer-flow/pull/3430
|
||||
[#3433]: https://github.com/bytedance/deer-flow/pull/3433
|
||||
[#3434]: https://github.com/bytedance/deer-flow/pull/3434
|
||||
[#3435]: https://github.com/bytedance/deer-flow/pull/3435
|
||||
[#3436]: https://github.com/bytedance/deer-flow/pull/3436
|
||||
[#3437]: https://github.com/bytedance/deer-flow/pull/3437
|
||||
[#3451]: https://github.com/bytedance/deer-flow/pull/3451
|
||||
[#3453]: https://github.com/bytedance/deer-flow/pull/3453
|
||||
[#3455]: https://github.com/bytedance/deer-flow/pull/3455
|
||||
[#3457]: https://github.com/bytedance/deer-flow/pull/3457
|
||||
[#3458]: https://github.com/bytedance/deer-flow/pull/3458
|
||||
[#3460]: https://github.com/bytedance/deer-flow/pull/3460
|
||||
[#3461]: https://github.com/bytedance/deer-flow/pull/3461
|
||||
[#3464]: https://github.com/bytedance/deer-flow/pull/3464
|
||||
[#3465]: https://github.com/bytedance/deer-flow/pull/3465
|
||||
[#3466]: https://github.com/bytedance/deer-flow/pull/3466
|
||||
[#3470]: https://github.com/bytedance/deer-flow/pull/3470
|
||||
[#3471]: https://github.com/bytedance/deer-flow/pull/3471
|
||||
[#3473]: https://github.com/bytedance/deer-flow/pull/3473
|
||||
[#3475]: https://github.com/bytedance/deer-flow/pull/3475
|
||||
[#3481]: https://github.com/bytedance/deer-flow/pull/3481
|
||||
[#3485]: https://github.com/bytedance/deer-flow/pull/3485
|
||||
[#3487]: https://github.com/bytedance/deer-flow/pull/3487
|
||||
[#3488]: https://github.com/bytedance/deer-flow/pull/3488
|
||||
[#3494]: https://github.com/bytedance/deer-flow/pull/3494
|
||||
[#3495]: https://github.com/bytedance/deer-flow/pull/3495
|
||||
[#3498]: https://github.com/bytedance/deer-flow/pull/3498
|
||||
[#3499]: https://github.com/bytedance/deer-flow/pull/3499
|
||||
[#3502]: https://github.com/bytedance/deer-flow/pull/3502
|
||||
[#3503]: https://github.com/bytedance/deer-flow/pull/3503
|
||||
[#3505]: https://github.com/bytedance/deer-flow/pull/3505
|
||||
[#3508]: https://github.com/bytedance/deer-flow/pull/3508
|
||||
[#3512]: https://github.com/bytedance/deer-flow/pull/3512
|
||||
[#3514]: https://github.com/bytedance/deer-flow/pull/3514
|
||||
[#3517]: https://github.com/bytedance/deer-flow/pull/3517
|
||||
[#3518]: https://github.com/bytedance/deer-flow/pull/3518
|
||||
[#3519]: https://github.com/bytedance/deer-flow/pull/3519
|
||||
[#3521]: https://github.com/bytedance/deer-flow/pull/3521
|
||||
[#3526]: https://github.com/bytedance/deer-flow/pull/3526
|
||||
[#3528]: https://github.com/bytedance/deer-flow/pull/3528
|
||||
[#3530]: https://github.com/bytedance/deer-flow/pull/3530
|
||||
[#3531]: https://github.com/bytedance/deer-flow/pull/3531
|
||||
[#3533]: https://github.com/bytedance/deer-flow/pull/3533
|
||||
[#3534]: https://github.com/bytedance/deer-flow/pull/3534
|
||||
[#3535]: https://github.com/bytedance/deer-flow/pull/3535
|
||||
[#3548]: https://github.com/bytedance/deer-flow/pull/3548
|
||||
[#3553]: https://github.com/bytedance/deer-flow/pull/3553
|
||||
[#3554]: https://github.com/bytedance/deer-flow/pull/3554
|
||||
[#3559]: https://github.com/bytedance/deer-flow/pull/3559
|
||||
[#3561]: https://github.com/bytedance/deer-flow/pull/3561
|
||||
[#3569]: https://github.com/bytedance/deer-flow/pull/3569
|
||||
[#3570]: https://github.com/bytedance/deer-flow/pull/3570
|
||||
[#3575]: https://github.com/bytedance/deer-flow/pull/3575
|
||||
[#3577]: https://github.com/bytedance/deer-flow/pull/3577
|
||||
[#3578]: https://github.com/bytedance/deer-flow/pull/3578
|
||||
[#3579]: https://github.com/bytedance/deer-flow/pull/3579
|
||||
[#3580]: https://github.com/bytedance/deer-flow/pull/3580
|
||||
[#3581]: https://github.com/bytedance/deer-flow/pull/3581
|
||||
[#3582]: https://github.com/bytedance/deer-flow/pull/3582
|
||||
[#3583]: https://github.com/bytedance/deer-flow/pull/3583
|
||||
[#3584]: https://github.com/bytedance/deer-flow/pull/3584
|
||||
[#3591]: https://github.com/bytedance/deer-flow/pull/3591
|
||||
[#3599]: https://github.com/bytedance/deer-flow/pull/3599
|
||||
[#3600]: https://github.com/bytedance/deer-flow/pull/3600
|
||||
[#3602]: https://github.com/bytedance/deer-flow/pull/3602
|
||||
[#3605]: https://github.com/bytedance/deer-flow/pull/3605
|
||||
[#3606]: https://github.com/bytedance/deer-flow/pull/3606
|
||||
[#3608]: https://github.com/bytedance/deer-flow/pull/3608
|
||||
[#3610]: https://github.com/bytedance/deer-flow/pull/3610
|
||||
[#3611]: https://github.com/bytedance/deer-flow/pull/3611
|
||||
[#3623]: https://github.com/bytedance/deer-flow/pull/3623
|
||||
[#3624]: https://github.com/bytedance/deer-flow/pull/3624
|
||||
[#3629]: https://github.com/bytedance/deer-flow/pull/3629
|
||||
[#3631]: https://github.com/bytedance/deer-flow/pull/3631
|
||||
[#3646]: https://github.com/bytedance/deer-flow/pull/3646
|
||||
[#3649]: https://github.com/bytedance/deer-flow/pull/3649
|
||||
[#3654]: https://github.com/bytedance/deer-flow/pull/3654
|
||||
[#3657]: https://github.com/bytedance/deer-flow/pull/3657
|
||||
[#3658]: https://github.com/bytedance/deer-flow/pull/3658
|
||||
+477
@@ -0,0 +1,477 @@
|
||||
# 更新日志
|
||||
|
||||
本文件记录 DeerFlow 的所有重要变更。
|
||||
|
||||
格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)，
|
||||
版本号遵循 [语义化版本规范](https://semver.org/lang/zh-CN/)。
|
||||
|
||||
[English](./CHANGELOG.md) | 中文
|
||||
|
||||
## [2.0.0] — 2026-06-15
|
||||
|
||||
DeerFlow 2.0 是围绕"超级智能体"框架的彻底重写，核心包含子智能体、持久化记忆、
|
||||
沙箱执行以及可扩展的技能（Skill）/工具系统。本版本与 1.x 系列**没有共享代码**，
|
||||
原有的 Deep Research 框架仍在
|
||||
[`main-1.x` 分支](https://github.com/bytedance/deer-flow/tree/main-1.x)上维护。
|
||||
|
||||
本次发布关闭了
|
||||
[2.0.0 里程碑](https://github.com/bytedance/deer-flow/milestone/1)，
|
||||
自首个 2.0 里程碑标签以来累计合并 **180 个 Pull Request**。
|
||||
|
||||
### ⚠ 不兼容变更（Breaking Changes）
|
||||
|
||||
- **harness：** 从 `RunStore` 重新加载历史 run，并持久化"已中断（interrupted）"
|
||||
状态。run 的取消 / 多任务调度语义现在要求拥有该 run 的 worker 上具备可用的
|
||||
RunStore；跨 worker 的取消请求将返回 `409`，不再静默"伪成功"。([#2932])
|
||||
|
||||
### 新增
|
||||
|
||||
#### 智能体与运行时
|
||||
- **智能体：** 自定义智能体支持自我更新，并按用户隔离 —— 智能体可在普通对话中
|
||||
持久化对自身 `SOUL.md` / `config.yaml` 的修改。([#2713])
|
||||
- **循环检测：** 支持配置开关，并可按工具维度覆盖触发频率。([#2586]、[#2711])
|
||||
- **循环检测：** 警告注入延后执行，避免与工具调用生命周期错位。([#2752])
|
||||
- **运行：** 将 `model_name` 从 gateway 请求一直透传到运行时与持久化层（SQLite
|
||||
存储）。([#2775])
|
||||
- **子智能体：** 通过终态任务事件，把子智能体的 token 用量流式上报到 header。
|
||||
([#2882])
|
||||
- **记忆：** 新增 `memory.token_counting` 配置，支持在受限网络环境下禁用
|
||||
tiktoken。([#3465])
|
||||
- **建议：** AI 追问（follow-up）建议改为可选。([#3591])
|
||||
|
||||
#### 模型与集成
|
||||
- **模型：** 新增 StepFun 推理模型适配器。([#3461])
|
||||
- **社区工具：** 新增 Brave Search 网络检索工具。([#3528])
|
||||
- **渠道：** Discord 增加"仅响应 mention"模式、话题路由（thread routing）以及
|
||||
正在输入指示。([#2842])
|
||||
- **IM：** 新增"用户自有 IM 渠道连接"——用户可以在运营方配置的 bot 之上，绑定
|
||||
自己的 Slack / Telegram / Discord / 飞书 / 钉钉 / 微信 / 企业微信 账号。
|
||||
([#3487])
|
||||
- **模型：** 新增 MiMo 推理内容（reasoning content）的补丁支持。([#3298])
|
||||
- **模型：** 新增 MiniMax provider，用于图像 / 视频 / 播客类技能，并新增"音乐
|
||||
生成"技能。([#3437])
|
||||
- **社区工具：** 新增 SearXNG 与 Browserless 的网络检索 / 抓取工具。([#3451])
|
||||
- **社区工具：** 为 `image_search` 新增 Serper Google 图片 provider。([#3575])
|
||||
- **渠道：** Telegram 智能体回复改为就地编辑占位消息的方式流式输出。([#3534])
|
||||
|
||||
#### 可观测性
|
||||
- **追踪：** LangGraph 追踪名设置为 `lead_agent`（自定义智能体则使用其
|
||||
`agent_name`），让 Langfuse / LangSmith 中的 trace 更清晰。([#3101])
|
||||
- **前端：** 优化 token 用量的展示模式。([#2329])
|
||||
- **默认值：** 默认开启 token 用量统计。([#2841])
|
||||
- **默认值：** 提高默认的上下文摘要触发阈值。([#3174])
|
||||
- **追踪：** 把子智能体的 span 归属到父线程的 Langfuse trace 上。([#3611])
|
||||
|
||||
#### 技能
|
||||
- **技能：** 新增 `blocking-io-guard` 技能，用于阻塞 IO 排查与运行时锚点。
|
||||
([#3503])
|
||||
- **技能：** 新增面向维护者的 issue 与 PR 工作流技能。([#3554])
|
||||
- **技能：** 增强维护者编排（orchestrator）的评审工作流。([#3606])
|
||||
|
||||
### 性能优化
|
||||
|
||||
- **harness：** 把 thread 元数据过滤下推到 SQL，不再在 Python 侧后过滤。
|
||||
([#2865])
|
||||
- **运行时：** 为 run 增加 `thread_id` 索引，避免 `RunManager` 中的 O(n) 扫描。
|
||||
([#3499])
|
||||
- **运行时：** 为 `MemoryRunEventStore` 中的消息建立索引，避免 O(n) 扫描。
|
||||
([#3531])
|
||||
- **持久化：** 按类缓存 `Base.to_dict` 的列反射结果。([#3654])
|
||||
- **沙箱：** 加快 glob/grep 遍历中的 `should_ignore_name` 判断。([#3657])
|
||||
|
||||
### 安全
|
||||
|
||||
- **上传：** 拒绝指向符号链接的上传目标。([#2623])
|
||||
- **上传：** 在 Windows 上支持基于符号链接保护的安全上传。([#2794])
|
||||
- **MCP：** 在 MCP 配置接口的响应中对敏感字段进行脱敏。([#2667])
|
||||
- **MCP：** 加固 MCP 配置接口对异常输入的处理。([#3425])
|
||||
- **认证：** 拒绝跨站点（cross-site）的认证 POST 请求。([#2740])
|
||||
- **网关：** 限制 skill artifact 预览的解压上限，避免被 zip-bomb 类构造滥用。
|
||||
([#2963])
|
||||
- **沙箱：** 仅在 aio（DooD）沙箱模式下挂载宿主机的 Docker socket。([#3517])
|
||||
- **沙箱：** 默认不再 bind-mount 宿主机的 CLI 认证目录。([#3521])
|
||||
|
||||
### 修复
|
||||
|
||||
#### 运行时、网关与持久化
|
||||
- **运行时：** rollback 恢复的 checkpoint 现在能够覆盖更新的 checkpoint。
|
||||
([#2582])
|
||||
- **运行时：** 持久化 run 的消息摘要。([#2850])
|
||||
- **运行时：** 限制 `write_file` 执行失败时上报的观测信息长度，避免失败 trace
|
||||
撑爆上下文。([#3133])
|
||||
- **运行时：** 加锁保护同步单例的初始化与 reset 路径。([#3413])
|
||||
- **运行时：** 为 run events 移除 PostgreSQL 上不必要的聚合
|
||||
`FOR UPDATE`。([#2962])
|
||||
- **运行：** gateway 重启后从持久化存储中恢复历史 run。([#2989])
|
||||
- **网关：** threads 接口返回 ISO 8601 格式的时间戳。([#2599])
|
||||
- **网关：** 对已经处于 interrupted 状态的 run，cancel 接口幂等返回。
|
||||
([#3058])
|
||||
- **网关：** 将 `stream_existing_run` 拆分为按 HTTP 方法区分的多个路由，确保
|
||||
OpenAPI `operationId` 唯一。([#3228])
|
||||
- **事件：** 序列化结构化的 DB event 内容。([#2762])
|
||||
- **持久化：** SQLite 后端的存储统一返回带时区的时间戳。([#3130])
|
||||
- **持久化：** 复用 token 用量按模型分组的 SQL 表达式。([#2910])
|
||||
- **运行：** 忽略已过期的 run reconnect 冲突。([#3284])
|
||||
- **nginx：** 把 CORS 策略下放到 gateway 的 allowlist，避免双重应用。
|
||||
([#2861])
|
||||
- **持久化：** 修复运行时 journal 中 run 生命周期事件的记录。([#3470])
|
||||
- **网关：** 在无状态（stateless）run 接口上强制校验 thread 归属。([#3473])
|
||||
- **运行时：** 通过 SSE values 事件把 interrupt 透传给 LangGraph SDK。([#3605])
|
||||
- **序列化：** 从流式 values 事件中剥离 base64 图片数据。([#3631])
|
||||
- **历史：** 从 REST 接口响应中剥离 base64 图片数据。([#3535])
|
||||
- **网关：** token 用量归因到实际使用的模型。([#3658])
|
||||
|
||||
#### 智能体、子智能体与中间件
|
||||
- **子智能体：** 让"子智能体超时"成为原子化的终态。([#2583])
|
||||
- **子智能体：** 工具与中间件按 model 覆盖（model override）来构造。([#2641])
|
||||
- **子智能体：** 把 `system_prompt` 与 skills 合并到单条 `SystemMessage`。
|
||||
([#2701])
|
||||
- **子智能体：** 子智能体与父 run 的 checkpointer 隔离。([#3559])
|
||||
- **智能体：** `update_agent` 与 `setup_agent` 一致，遵循 `runtime.context` 的
|
||||
`user_id`。([#2867])
|
||||
- **智能体：** 解决 `TodoMiddleware` 中 `todos` 通道的类型冲突。([#3200])
|
||||
- **智能体：** 把自定义智能体路由中的阻塞文件 IO 移出事件循环。([#3457])
|
||||
- **智能体：** 新智能体的 bootstrap 流程保持在用户作用域内。([#2784])
|
||||
- **循环检测：** 注入 warn 时仍保持 tool-call 配对。([#2725])
|
||||
- **中间件：** 同步原始 tool-call 元数据。([#2757])
|
||||
- **中间件：** dangling 配对中间件正确处理非法 tool call。([#2891])
|
||||
- **中间件：** 防止 todo 完成提醒消息泄漏到 IM 渠道。([#2907])
|
||||
- **中间件：** 调用模型前先把 tool result 的相邻关系规范化。([#2939])
|
||||
- **智能体：** `resolve_agent_dir` 要求存在 `config.yaml`，从而跳过仅含记忆的
|
||||
目录。([#3481])
|
||||
- **智能体：** 在 context / configurable 间同步 `agent_name`，并拒绝空的
|
||||
soul。([#3553])
|
||||
- **中间件：** 把 `UploadsMiddleware` 中的上传扫描移出事件循环。([#3311])
|
||||
- **中间件：** 把记忆注入移出事件循环，避免 tiktoken 造成阻塞。([#3411])
|
||||
- **中间件：** 针对非挂载型沙箱，把超限的工具输出外置到沙箱中。([#3417])
|
||||
- **中间件：** 在中间件 state 中保留 sandbox reducer。([#3629])
|
||||
- **子智能体：** general-purpose 的 `max_turns` 提升到 150，默认超时提升到 30
|
||||
分钟。([#3610])
|
||||
|
||||
#### 记忆与追踪
|
||||
- **记忆：** 用常驻事件循环替换短生命周期的 `asyncio.run()`。([#2627])
|
||||
- **记忆：** 队列化的 memory 更新按智能体维度隔离。([#2941])
|
||||
- **记忆：** 解析被外层包裹的 memory 更新 JSON 响应。([#3252])
|
||||
- **追踪：** 把 `session_id` 与 `user_id` 透传到 Langfuse trace。([#2944])
|
||||
- **追踪：** 修复中文 memory trace 信息显示为 unicode 转义序列的问题。
|
||||
([#3104])
|
||||
|
||||
#### 工具、沙箱与 MCP
|
||||
- **MCP：** 修复 MCP 配置中列表型变量的环境变量解析。([#2556])
|
||||
- **模型：** Codex 的 token 用量记录到 `usage_metadata`。([#2585])
|
||||
- **沙箱：** 在 `RemoteSandboxBackend` 中补上 `list_running`。([#2716])
|
||||
- **沙箱：** Windows / Git Bash 下关闭 MSYS 路径转换。([#2766])
|
||||
- **沙箱：** 沙箱就绪轮询不再阻塞事件循环。([#2822])
|
||||
- **沙箱：** `Sandbox` API 边界统一遵守 `/mnt/user-data` 契约。([#2881])
|
||||
- **沙箱：** Provisioner 的 PVC 数据按用户隔离。([#2973])
|
||||
- **沙箱：** 合并幂等的沙箱状态更新。([#3518])
|
||||
- **工具：** 引入 `Runtime` 类型别名，消除 Pydantic 序列化告警。([#2774])
|
||||
- **工具：** 在重入式 `get_available_tools` 调用之间保留 `tool_search` 的提升
|
||||
状态。([#2885])
|
||||
- **harness：** 为同步客户端封装仅异步可用的 config 工具。([#2878])
|
||||
- **harness：** 同步客户端可用所有原本仅异步的工具（统一封装）。([#2935])
|
||||
- **工具检索：** 通过移除 ContextVar，可靠地隐藏延迟加载的 MCP schema。
|
||||
([#3342])
|
||||
- **检索：** 修复 DDGS 的维基百科区域处理。([#3423])
|
||||
- **web_fetch：** 在受限网络环境下为 Jina reader 支持代理。([#3430])
|
||||
- **沙箱：** 通过 `Command` 持久化懒加载获取到的沙箱状态。([#3464])
|
||||
- **沙箱：** 修复 AIO 沙箱缓存被陈旧复用的问题。([#3494])
|
||||
- **沙箱：** 在使用全新 id 重试前先创建 shell session。([#3577])
|
||||
- **沙箱：** 不再把字符串字面量路径片段误判为不安全的绝对路径。([#3623])
|
||||
- **沙箱：** `read_file` 命中二进制文件时返回可操作的提示信息。([#3624])
|
||||
- **MCP：** 让 stdio MCP 产出的文件可通过虚拟沙箱路径解析。([#3600])
|
||||
- **MCP：** 在设置页的工具列表上展示"需要管理员权限"的状态。([#3533])
|
||||
- **MCP：** 新增工具缓存重置接口。([#3602])
|
||||
- **上传：** 修复上传文件大小的接口契约。([#3408])
|
||||
|
||||
#### 技能与渠道
|
||||
- **技能：** 强制校验 `allowed-tools` 元数据。([#2626])
|
||||
- **技能：** 在各聊天渠道下加固 `/skill` 斜杠激活。([#3466])
|
||||
- **技能：** 修复自定义 skill 安装时的权限问题。([#3241])
|
||||
- **渠道：** Gateway 的命令请求需要鉴权。([#2742])
|
||||
- **技能：** SKILL.md 出现 YAML 错误时，展示出错行号与引号提示。([#3335])
|
||||
- **技能：** 把技能压缩包的安装移出事件循环。([#3505])
|
||||
- **渠道：** 提取回复时忽略隐藏的控制消息。([#3270])
|
||||
- **渠道：** 渠道重启时重新加载配置。([#3514])
|
||||
- **渠道：** 暴露企业微信（WeCom）WebSocket 连接失败的信息。([#3526])
|
||||
- **渠道：** Discord 上传完成后关闭文件句柄。([#3561])
|
||||
- **渠道：** 用户自有 IM 消息要求绑定身份。([#3578])
|
||||
- **渠道：** IM 文件与辅助命令限定在 owner 作用域内。([#3579])
|
||||
- **渠道：** 以运行时的 provider 状态为权威来源。([#3580])
|
||||
- **渠道：** 加固运行时凭证管理接口。([#3581])
|
||||
- **渠道：** 让渠道连接流程可确定（deterministic）。([#3582])
|
||||
- **渠道：** 集中各渠道共享的重试辅助逻辑。([#3583])
|
||||
- **渠道：** 增加运行态的防护约束（operational guardrails）。([#3584])
|
||||
- **渠道：** 按相等性（equality）退订渠道监听器。([#3608])
|
||||
|
||||
#### 认证
|
||||
- **认证：** 用缓存响应替换 setup-status 接口的 429 限流。([#2915])
|
||||
- **认证：** 自动生成的 JWT secret 持久化保存，重启后仍可用。([#2933])
|
||||
- **认证：** 对齐"认证禁用（auth-disabled）"模式与 mock 历史加载行为。
|
||||
([#3471])
|
||||
|
||||
#### 前端
|
||||
- **前端：** 在 prod 模式下恢复 `getGatewayConfig` 的 `localhost` 兜底。
|
||||
([#2718])
|
||||
- **聊天：** 修复新会话第一条用户消息被吞掉的问题。([#2731])
|
||||
- **前端：** header 总计 token 数采用后端线程级 token 用量。([#2800])
|
||||
- **前端：** 异步 chat submit 完成后再清空输入框。([#2940])
|
||||
- **前端：** 修复登录页闪烁与 ResizeObserver 死循环。([#2954])
|
||||
- **前端：** 对恢复出的会话消息去重。([#2958])
|
||||
- **前端：** 避免乐观渲染产生重复的用户消息。([#3002])
|
||||
- **前端：** 流式中的 assistant 消息不再展示复制按钮。([#3176])
|
||||
- **前端：** 新建 thread 后立即在侧边栏显示。([#3283])
|
||||
- **前端：** 新建会话的 thread 消息相互隔离。([#3508])
|
||||
- **前端：** 限制深层嵌套列表的缩进，避免渲染崩溃。([#3393]、[#3570])
|
||||
- **token 用量：** token 用量按 message id 去重聚合。([#2770])
|
||||
- **前端：** 剪贴板复制回退到 Streamdown。([#3397])
|
||||
- **前端：** 移除用 Backspace 删除 prompt 附件的快捷键。([#3410])
|
||||
- **前端：** 把记忆设置的工具栏重排为两行。([#3433])
|
||||
- **建议：** 解析追问问题前先剥离内联的 `<think>` 推理内容。([#3435])
|
||||
- **前端：** 追问建议被禁用时不再发起请求。([#3599])
|
||||
- **前端：** 工作区会话列表在超过 50 个 thread 后分页加载。([#3485])
|
||||
- **前端：** 避免长串不可断行文本导致用户消息气泡溢出。([#3488])
|
||||
- **前端：** SSR 鉴权探测无法连通 gateway 时仍保持工作区可交互。([#3495])
|
||||
- **前端：** 用户消息按纯文本渲染，并限制引用（blockquote）嵌套层级。
|
||||
([#3502])
|
||||
- **前端：** 删除后重置当前激活的会话。([#3519])
|
||||
- **前端：** 优化移动端工作区布局。([#3646])
|
||||
- **前端：** 多段（multi-part）AI 消息渲染完整内容。([#3649])
|
||||
|
||||
#### 构建、部署、脚本与配置
|
||||
- **打包：** 新增 `postgres` extra 以支持 store / checkpointer，并完善安装
|
||||
说明。([#2584])
|
||||
- **harness：** 运行时路径以项目根目录为基准解析。([#2642])
|
||||
- **Docker：** 让 nginx 在每次请求时再解析 upstream 名称。([#2717])
|
||||
- **Docker：** 把 Gateway 默认改为单 worker，避免多 worker 模式下出现异常。
|
||||
([#3475])
|
||||
- **脚本：** `make dev` 重启时保留 `uv` extras。([#2767]、[#2754])
|
||||
- **脚本：** 停止时清理本地 nginx。([#3005])
|
||||
- **部署：** 没有 `python3` 时，secret 生成回退到 `python` / `openssl`。
|
||||
([#3074])
|
||||
- **配置：** 让 reload boundary 在代码层面可发现。([#3144]、[#3153])
|
||||
- **replay-e2e：** 重放 fixture 按调用方与会话作为 key。([#3453])
|
||||
- **安装向导：** 更新 LLM provider 向导的默认值。([#3421])
|
||||
- **配置：** 把 `config.yaml` 中为 null 的列表字段归一为空列表。([#3434])
|
||||
- **脚本：** gateway reload 时排除运行时状态目录。([#3426])
|
||||
- **脚本：** 在 uvicorn 的 reload-exclude 生效前先创建 backend/sandbox 目录。
|
||||
([#3460])
|
||||
- **脚本：** 修复 `make start-daemon` 之后无法用 `make stop` 正确停止
|
||||
next-server 的问题。([#3498])
|
||||
- **Makefile：** 修复 per-commit hooks 的安装。([#3569])
|
||||
- **replay-e2e：** 重放匹配改为按会话，而非使用当前系统 prompt。([#3436])
|
||||
|
||||
### 变更
|
||||
|
||||
- **provider（重构）：** 各 provider 间共享 assistant payload 的回放匹配逻辑。
|
||||
([#3307])
|
||||
- **lead-agent（重构）：** 把 `build_middlewares` 改为 public，去掉最后一个跨
|
||||
模块的私有导入。([#3458])
|
||||
- **todo（重构）：** 移除未使用的完成提醒计数器。([#3530])
|
||||
|
||||
### 文档
|
||||
|
||||
- 补充 blocking-IO 检测的使用与维护说明。([#3233])
|
||||
- 清理文档中残留的"独立 LangGraph 服务器"相关内容。([#3301])
|
||||
- 在 PR 模板与 CONTRIBUTING 中补充 AI 辅助声明。([#3398])
|
||||
- 补充自定义 AIO 沙箱镜像的文档。([#3548])
|
||||
|
||||
### 内部改进
|
||||
|
||||
- **开发：** 新增 async / thread 边界检测器。([#2936])
|
||||
- **运行时：** 增加 lifecycle 端到端测试覆盖。([#2946])
|
||||
- **Windows：** 后端 Makefile 各 target 加入 `PYTHONIOENCODING` 与
|
||||
`PYTHONUTF8`。([#3069])
|
||||
- **blocking-io：** 检测器以"显式失败（fail-loud）"方式解析仓库根目录，并提供
|
||||
共享 CLI 入口。([#3512])
|
||||
- **运行时：** 为 `JsonlRunEventStore` 的异步 IO 增加 Blockbuster 运行时锚点。
|
||||
([#3313])
|
||||
- **CI：** 统一 PR / issue 打标签逻辑，修复 reviewing 任务的崩溃与标签抖动。
|
||||
([#3455])
|
||||
|
||||
[2.0.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.0.0
|
||||
[#2329]: https://github.com/bytedance/deer-flow/pull/2329
|
||||
[#2556]: https://github.com/bytedance/deer-flow/pull/2556
|
||||
[#2582]: https://github.com/bytedance/deer-flow/pull/2582
|
||||
[#2583]: https://github.com/bytedance/deer-flow/pull/2583
|
||||
[#2584]: https://github.com/bytedance/deer-flow/pull/2584
|
||||
[#2585]: https://github.com/bytedance/deer-flow/pull/2585
|
||||
[#2586]: https://github.com/bytedance/deer-flow/pull/2586
|
||||
[#2599]: https://github.com/bytedance/deer-flow/pull/2599
|
||||
[#2623]: https://github.com/bytedance/deer-flow/pull/2623
|
||||
[#2626]: https://github.com/bytedance/deer-flow/pull/2626
|
||||
[#2627]: https://github.com/bytedance/deer-flow/pull/2627
|
||||
[#2641]: https://github.com/bytedance/deer-flow/pull/2641
|
||||
[#2642]: https://github.com/bytedance/deer-flow/pull/2642
|
||||
[#2667]: https://github.com/bytedance/deer-flow/pull/2667
|
||||
[#2701]: https://github.com/bytedance/deer-flow/pull/2701
|
||||
[#2711]: https://github.com/bytedance/deer-flow/pull/2711
|
||||
[#2713]: https://github.com/bytedance/deer-flow/pull/2713
|
||||
[#2716]: https://github.com/bytedance/deer-flow/pull/2716
|
||||
[#2717]: https://github.com/bytedance/deer-flow/pull/2717
|
||||
[#2718]: https://github.com/bytedance/deer-flow/pull/2718
|
||||
[#2725]: https://github.com/bytedance/deer-flow/pull/2725
|
||||
[#2731]: https://github.com/bytedance/deer-flow/pull/2731
|
||||
[#2740]: https://github.com/bytedance/deer-flow/pull/2740
|
||||
[#2742]: https://github.com/bytedance/deer-flow/pull/2742
|
||||
[#2752]: https://github.com/bytedance/deer-flow/pull/2752
|
||||
[#2754]: https://github.com/bytedance/deer-flow/pull/2754
|
||||
[#2757]: https://github.com/bytedance/deer-flow/pull/2757
|
||||
[#2762]: https://github.com/bytedance/deer-flow/pull/2762
|
||||
[#2766]: https://github.com/bytedance/deer-flow/pull/2766
|
||||
[#2767]: https://github.com/bytedance/deer-flow/pull/2767
|
||||
[#2770]: https://github.com/bytedance/deer-flow/pull/2770
|
||||
[#2774]: https://github.com/bytedance/deer-flow/pull/2774
|
||||
[#2775]: https://github.com/bytedance/deer-flow/pull/2775
|
||||
[#2784]: https://github.com/bytedance/deer-flow/pull/2784
|
||||
[#2794]: https://github.com/bytedance/deer-flow/pull/2794
|
||||
[#2800]: https://github.com/bytedance/deer-flow/pull/2800
|
||||
[#2822]: https://github.com/bytedance/deer-flow/pull/2822
|
||||
[#2841]: https://github.com/bytedance/deer-flow/pull/2841
|
||||
[#2842]: https://github.com/bytedance/deer-flow/pull/2842
|
||||
[#2850]: https://github.com/bytedance/deer-flow/pull/2850
|
||||
[#2861]: https://github.com/bytedance/deer-flow/pull/2861
|
||||
[#2865]: https://github.com/bytedance/deer-flow/pull/2865
|
||||
[#2867]: https://github.com/bytedance/deer-flow/pull/2867
|
||||
[#2878]: https://github.com/bytedance/deer-flow/pull/2878
|
||||
[#2881]: https://github.com/bytedance/deer-flow/pull/2881
|
||||
[#2882]: https://github.com/bytedance/deer-flow/pull/2882
|
||||
[#2885]: https://github.com/bytedance/deer-flow/pull/2885
|
||||
[#2891]: https://github.com/bytedance/deer-flow/pull/2891
|
||||
[#2907]: https://github.com/bytedance/deer-flow/pull/2907
|
||||
[#2910]: https://github.com/bytedance/deer-flow/pull/2910
|
||||
[#2915]: https://github.com/bytedance/deer-flow/pull/2915
|
||||
[#2932]: https://github.com/bytedance/deer-flow/pull/2932
|
||||
[#2933]: https://github.com/bytedance/deer-flow/pull/2933
|
||||
[#2935]: https://github.com/bytedance/deer-flow/pull/2935
|
||||
[#2936]: https://github.com/bytedance/deer-flow/pull/2936
|
||||
[#2939]: https://github.com/bytedance/deer-flow/pull/2939
|
||||
[#2940]: https://github.com/bytedance/deer-flow/pull/2940
|
||||
[#2941]: https://github.com/bytedance/deer-flow/pull/2941
|
||||
[#2944]: https://github.com/bytedance/deer-flow/pull/2944
|
||||
[#2946]: https://github.com/bytedance/deer-flow/pull/2946
|
||||
[#2954]: https://github.com/bytedance/deer-flow/pull/2954
|
||||
[#2958]: https://github.com/bytedance/deer-flow/pull/2958
|
||||
[#2962]: https://github.com/bytedance/deer-flow/pull/2962
|
||||
[#2963]: https://github.com/bytedance/deer-flow/pull/2963
|
||||
[#2973]: https://github.com/bytedance/deer-flow/pull/2973
|
||||
[#2989]: https://github.com/bytedance/deer-flow/pull/2989
|
||||
[#3002]: https://github.com/bytedance/deer-flow/pull/3002
|
||||
[#3005]: https://github.com/bytedance/deer-flow/pull/3005
|
||||
[#3058]: https://github.com/bytedance/deer-flow/pull/3058
|
||||
[#3069]: https://github.com/bytedance/deer-flow/pull/3069
|
||||
[#3074]: https://github.com/bytedance/deer-flow/pull/3074
|
||||
[#3101]: https://github.com/bytedance/deer-flow/pull/3101
|
||||
[#3104]: https://github.com/bytedance/deer-flow/pull/3104
|
||||
[#3130]: https://github.com/bytedance/deer-flow/pull/3130
|
||||
[#3133]: https://github.com/bytedance/deer-flow/pull/3133
|
||||
[#3144]: https://github.com/bytedance/deer-flow/pull/3144
|
||||
[#3153]: https://github.com/bytedance/deer-flow/pull/3153
|
||||
[#3174]: https://github.com/bytedance/deer-flow/pull/3174
|
||||
[#3176]: https://github.com/bytedance/deer-flow/pull/3176
|
||||
[#3200]: https://github.com/bytedance/deer-flow/pull/3200
|
||||
[#3228]: https://github.com/bytedance/deer-flow/pull/3228
|
||||
[#3233]: https://github.com/bytedance/deer-flow/pull/3233
|
||||
[#3241]: https://github.com/bytedance/deer-flow/pull/3241
|
||||
[#3252]: https://github.com/bytedance/deer-flow/pull/3252
|
||||
[#3270]: https://github.com/bytedance/deer-flow/pull/3270
|
||||
[#3283]: https://github.com/bytedance/deer-flow/pull/3283
|
||||
[#3284]: https://github.com/bytedance/deer-flow/pull/3284
|
||||
[#3298]: https://github.com/bytedance/deer-flow/pull/3298
|
||||
[#3301]: https://github.com/bytedance/deer-flow/pull/3301
|
||||
[#3307]: https://github.com/bytedance/deer-flow/pull/3307
|
||||
[#3311]: https://github.com/bytedance/deer-flow/pull/3311
|
||||
[#3313]: https://github.com/bytedance/deer-flow/pull/3313
|
||||
[#3335]: https://github.com/bytedance/deer-flow/pull/3335
|
||||
[#3342]: https://github.com/bytedance/deer-flow/pull/3342
|
||||
[#3393]: https://github.com/bytedance/deer-flow/pull/3393
|
||||
[#3397]: https://github.com/bytedance/deer-flow/pull/3397
|
||||
[#3398]: https://github.com/bytedance/deer-flow/pull/3398
|
||||
[#3408]: https://github.com/bytedance/deer-flow/pull/3408
|
||||
[#3410]: https://github.com/bytedance/deer-flow/pull/3410
|
||||
[#3411]: https://github.com/bytedance/deer-flow/pull/3411
|
||||
[#3413]: https://github.com/bytedance/deer-flow/pull/3413
|
||||
[#3417]: https://github.com/bytedance/deer-flow/pull/3417
|
||||
[#3421]: https://github.com/bytedance/deer-flow/pull/3421
|
||||
[#3423]: https://github.com/bytedance/deer-flow/pull/3423
|
||||
[#3425]: https://github.com/bytedance/deer-flow/pull/3425
|
||||
[#3426]: https://github.com/bytedance/deer-flow/pull/3426
|
||||
[#3430]: https://github.com/bytedance/deer-flow/pull/3430
|
||||
[#3433]: https://github.com/bytedance/deer-flow/pull/3433
|
||||
[#3434]: https://github.com/bytedance/deer-flow/pull/3434
|
||||
[#3435]: https://github.com/bytedance/deer-flow/pull/3435
|
||||
[#3436]: https://github.com/bytedance/deer-flow/pull/3436
|
||||
[#3437]: https://github.com/bytedance/deer-flow/pull/3437
|
||||
[#3451]: https://github.com/bytedance/deer-flow/pull/3451
|
||||
[#3453]: https://github.com/bytedance/deer-flow/pull/3453
|
||||
[#3455]: https://github.com/bytedance/deer-flow/pull/3455
|
||||
[#3457]: https://github.com/bytedance/deer-flow/pull/3457
|
||||
[#3458]: https://github.com/bytedance/deer-flow/pull/3458
|
||||
[#3460]: https://github.com/bytedance/deer-flow/pull/3460
|
||||
[#3461]: https://github.com/bytedance/deer-flow/pull/3461
|
||||
[#3464]: https://github.com/bytedance/deer-flow/pull/3464
|
||||
[#3465]: https://github.com/bytedance/deer-flow/pull/3465
|
||||
[#3466]: https://github.com/bytedance/deer-flow/pull/3466
|
||||
[#3470]: https://github.com/bytedance/deer-flow/pull/3470
|
||||
[#3471]: https://github.com/bytedance/deer-flow/pull/3471
|
||||
[#3473]: https://github.com/bytedance/deer-flow/pull/3473
|
||||
[#3475]: https://github.com/bytedance/deer-flow/pull/3475
|
||||
[#3481]: https://github.com/bytedance/deer-flow/pull/3481
|
||||
[#3485]: https://github.com/bytedance/deer-flow/pull/3485
|
||||
[#3487]: https://github.com/bytedance/deer-flow/pull/3487
|
||||
[#3488]: https://github.com/bytedance/deer-flow/pull/3488
|
||||
[#3494]: https://github.com/bytedance/deer-flow/pull/3494
|
||||
[#3495]: https://github.com/bytedance/deer-flow/pull/3495
|
||||
[#3498]: https://github.com/bytedance/deer-flow/pull/3498
|
||||
[#3499]: https://github.com/bytedance/deer-flow/pull/3499
|
||||
[#3502]: https://github.com/bytedance/deer-flow/pull/3502
|
||||
[#3503]: https://github.com/bytedance/deer-flow/pull/3503
|
||||
[#3505]: https://github.com/bytedance/deer-flow/pull/3505
|
||||
[#3508]: https://github.com/bytedance/deer-flow/pull/3508
|
||||
[#3512]: https://github.com/bytedance/deer-flow/pull/3512
|
||||
[#3514]: https://github.com/bytedance/deer-flow/pull/3514
|
||||
[#3517]: https://github.com/bytedance/deer-flow/pull/3517
|
||||
[#3518]: https://github.com/bytedance/deer-flow/pull/3518
|
||||
[#3519]: https://github.com/bytedance/deer-flow/pull/3519
|
||||
[#3521]: https://github.com/bytedance/deer-flow/pull/3521
|
||||
[#3526]: https://github.com/bytedance/deer-flow/pull/3526
|
||||
[#3528]: https://github.com/bytedance/deer-flow/pull/3528
|
||||
[#3530]: https://github.com/bytedance/deer-flow/pull/3530
|
||||
[#3531]: https://github.com/bytedance/deer-flow/pull/3531
|
||||
[#3533]: https://github.com/bytedance/deer-flow/pull/3533
|
||||
[#3534]: https://github.com/bytedance/deer-flow/pull/3534
|
||||
[#3535]: https://github.com/bytedance/deer-flow/pull/3535
|
||||
[#3548]: https://github.com/bytedance/deer-flow/pull/3548
|
||||
[#3553]: https://github.com/bytedance/deer-flow/pull/3553
|
||||
[#3554]: https://github.com/bytedance/deer-flow/pull/3554
|
||||
[#3559]: https://github.com/bytedance/deer-flow/pull/3559
|
||||
[#3561]: https://github.com/bytedance/deer-flow/pull/3561
|
||||
[#3569]: https://github.com/bytedance/deer-flow/pull/3569
|
||||
[#3570]: https://github.com/bytedance/deer-flow/pull/3570
|
||||
[#3575]: https://github.com/bytedance/deer-flow/pull/3575
|
||||
[#3577]: https://github.com/bytedance/deer-flow/pull/3577
|
||||
[#3578]: https://github.com/bytedance/deer-flow/pull/3578
|
||||
[#3579]: https://github.com/bytedance/deer-flow/pull/3579
|
||||
[#3580]: https://github.com/bytedance/deer-flow/pull/3580
|
||||
[#3581]: https://github.com/bytedance/deer-flow/pull/3581
|
||||
[#3582]: https://github.com/bytedance/deer-flow/pull/3582
|
||||
[#3583]: https://github.com/bytedance/deer-flow/pull/3583
|
||||
[#3584]: https://github.com/bytedance/deer-flow/pull/3584
|
||||
[#3591]: https://github.com/bytedance/deer-flow/pull/3591
|
||||
[#3599]: https://github.com/bytedance/deer-flow/pull/3599
|
||||
[#3600]: https://github.com/bytedance/deer-flow/pull/3600
|
||||
[#3602]: https://github.com/bytedance/deer-flow/pull/3602
|
||||
[#3605]: https://github.com/bytedance/deer-flow/pull/3605
|
||||
[#3606]: https://github.com/bytedance/deer-flow/pull/3606
|
||||
[#3608]: https://github.com/bytedance/deer-flow/pull/3608
|
||||
[#3610]: https://github.com/bytedance/deer-flow/pull/3610
|
||||
[#3611]: https://github.com/bytedance/deer-flow/pull/3611
|
||||
[#3623]: https://github.com/bytedance/deer-flow/pull/3623
|
||||
[#3624]: https://github.com/bytedance/deer-flow/pull/3624
|
||||
[#3629]: https://github.com/bytedance/deer-flow/pull/3629
|
||||
[#3631]: https://github.com/bytedance/deer-flow/pull/3631
|
||||
[#3646]: https://github.com/bytedance/deer-flow/pull/3646
|
||||
[#3649]: https://github.com/bytedance/deer-flow/pull/3649
|
||||
[#3654]: https://github.com/bytedance/deer-flow/pull/3654
|
||||
[#3657]: https://github.com/bytedance/deer-flow/pull/3657
|
||||
[#3658]: https://github.com/bytedance/deer-flow/pull/3658
|
||||
@@ -0,0 +1,5 @@
|
||||
# CLAUDE.md
|
||||
|
||||
The repo's agent guidance lives in [AGENTS.md](AGENTS.md) so it is shared across coding agents (Claude Code, Codex, and others). Claude Code imports it below.
|
||||
|
||||
@AGENTS.md
|
||||
@@ -0,0 +1,128 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, religion, or sexual identity
|
||||
and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the
|
||||
overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or
|
||||
advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email
|
||||
address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official e-mail address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the community leaders responsible for enforcement at
|
||||
willem.jiang@gmail.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series
|
||||
of actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or
|
||||
permanent ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within
|
||||
the community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.0, available at
|
||||
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
|
||||
|
||||
Community Impact Guidelines were inspired by [Mozilla's code of conduct
|
||||
enforcement ladder](https://github.com/mozilla/diversity).
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
https://www.contributor-covenant.org/faq. Translations are available at
|
||||
https://www.contributor-covenant.org/translations.
|
||||
+381
@@ -0,0 +1,381 @@
|
||||
# Contributing to DeerFlow
|
||||
|
||||
Thank you for your interest in contributing to DeerFlow! This guide will help you set up your development environment and understand our development workflow.
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
We offer two development environments. **Docker is recommended** for the most consistent and hassle-free experience.
|
||||
|
||||
### Option 1: Docker Development (Recommended)
|
||||
|
||||
Docker provides a consistent, isolated environment with all dependencies pre-configured. No need to install Node.js, Python, or nginx on your local machine.
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
- Docker Desktop or Docker Engine
|
||||
- pnpm (for caching optimization)
|
||||
|
||||
#### Setup Steps
|
||||
|
||||
1. **Configure the application**:
|
||||
```bash
|
||||
# Copy example configuration
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Set your API keys
|
||||
export OPENAI_API_KEY="your-key-here"
|
||||
# or edit config.yaml directly
|
||||
```
|
||||
|
||||
2. **Initialize Docker environment** (first time only):
|
||||
```bash
|
||||
make docker-init
|
||||
```
|
||||
This will:
|
||||
- Build Docker images
|
||||
- Install frontend dependencies (pnpm)
|
||||
- Install backend dependencies (uv)
|
||||
- Share pnpm cache with host for faster builds
|
||||
|
||||
3. **Start development services**:
|
||||
```bash
|
||||
make docker-start
|
||||
```
|
||||
`make docker-start` reads `config.yaml` and starts `provisioner` only for provisioner/Kubernetes sandbox mode.
|
||||
|
||||
All services will start with hot-reload enabled:
|
||||
- Frontend changes are automatically reloaded
|
||||
- Backend changes trigger automatic restart
|
||||
- Gateway-hosted LangGraph-compatible runtime supports hot-reload
|
||||
|
||||
4. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
- API Gateway: http://localhost:2026/api/*
|
||||
- LangGraph-compatible API: http://localhost:2026/api/langgraph/*
|
||||
|
||||
#### Docker Commands
|
||||
|
||||
```bash
|
||||
# Build the custom k3s image (with pre-cached sandbox image)
|
||||
make docker-init
|
||||
# Start Docker services (mode-aware, localhost:2026)
|
||||
make docker-start
|
||||
# Stop Docker development services
|
||||
make docker-stop
|
||||
# View Docker development logs
|
||||
make docker-logs
|
||||
# View Docker frontend logs
|
||||
make docker-logs-frontend
|
||||
# View Docker gateway logs
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
If Docker builds are slow in your network, you can override the default package registries before running `make docker-init` or `make docker-start`:
|
||||
|
||||
```bash
|
||||
export UV_INDEX_URL=https://pypi.org/simple
|
||||
export NPM_REGISTRY=https://registry.npmjs.org
|
||||
```
|
||||
|
||||
#### Recommended host resources
|
||||
|
||||
Use these as practical starting points for development and review environments:
|
||||
|
||||
| Scenario | Starting point | Recommended | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| `make dev` on one machine | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Best when DeerFlow uses hosted model APIs. |
|
||||
| `make docker-start` review environment | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Docker image builds and sandbox containers need extra headroom. |
|
||||
| Shared Linux test server | 8 vCPU, 16 GB RAM | 16 vCPU, 32 GB RAM | Prefer this for heavier multi-agent runs or multiple reviewers. |
|
||||
|
||||
`2 vCPU / 4 GB` environments often fail to start reliably or become unresponsive under normal DeerFlow workloads.
|
||||
|
||||
#### Linux: Docker daemon permission denied
|
||||
|
||||
If `make docker-init`, `make docker-start`, or `make docker-stop` fails on Linux with an error like below, your current user likely does not have permission to access the Docker daemon socket:
|
||||
|
||||
```text
|
||||
unable to get image 'deer-flow-gateway': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
|
||||
```
|
||||
|
||||
Recommended fix: add your current user to the `docker` group so Docker commands work without `sudo`.
|
||||
|
||||
1. Confirm the `docker` group exists:
|
||||
```bash
|
||||
getent group docker
|
||||
```
|
||||
2. Add your current user to the `docker` group:
|
||||
```bash
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
3. Apply the new group membership. The most reliable option is to log out completely and then log back in. If you want to refresh the current shell session instead, run:
|
||||
```bash
|
||||
newgrp docker
|
||||
```
|
||||
4. Verify Docker access:
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
5. Retry the DeerFlow command:
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
If `docker ps` still reports a permission error after `usermod`, fully log out and log back in before retrying.
|
||||
|
||||
#### Docker Architecture
|
||||
|
||||
```
|
||||
Host Machine
|
||||
↓
|
||||
Docker Compose (deer-flow-dev)
|
||||
├→ nginx (port 2026) ← Reverse proxy
|
||||
├→ web (port 3000) ← Frontend with hot-reload
|
||||
├→ gateway (port 8001) ← Gateway API + LangGraph-compatible runtime with hot-reload
|
||||
└→ provisioner (optional, port 8002) ← Started only in provisioner/K8s sandbox mode
|
||||
```
|
||||
|
||||
**Benefits of Docker Development**:
|
||||
- ✅ Consistent environment across different machines
|
||||
- ✅ No need to install Node.js, Python, or nginx locally
|
||||
- ✅ Isolated dependencies and services
|
||||
- ✅ Easy cleanup and reset
|
||||
- ✅ Hot-reload for all services
|
||||
- ✅ Production-like environment
|
||||
|
||||
### Option 2: Local Development
|
||||
|
||||
If you prefer to run services directly on your machine:
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
Check that you have all required tools installed:
|
||||
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
Required tools:
|
||||
- Node.js 22+
|
||||
- pnpm
|
||||
- uv (Python package manager)
|
||||
- nginx
|
||||
|
||||
#### Setup Steps
|
||||
|
||||
1. **Configure the application** (same as Docker setup above)
|
||||
|
||||
2. **Install dependencies** (this also sets up pre-commit hooks):
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
3. **Run development server** (starts all services with nginx):
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
4. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
- All API requests are automatically proxied through nginx
|
||||
|
||||
#### Manual Service Control
|
||||
|
||||
If you need to start services individually:
|
||||
|
||||
1. **Start backend service**:
|
||||
```bash
|
||||
# Terminal 1: Start Gateway API + embedded agent runtime (port 8001)
|
||||
cd backend
|
||||
make dev
|
||||
|
||||
# Terminal 2: Start Frontend (port 3000)
|
||||
cd frontend
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
2. **Start nginx** (run from the repo root):
|
||||
```bash
|
||||
make nginx
|
||||
```
|
||||
|
||||
This runs `scripts/nginx.sh`, which launches nginx in the foreground the same way `scripts/serve.sh` (used by `make dev` / `make start`) does: it pre-creates the `logs/` and `temp/` directories and uses the local dev config at `docker/nginx/nginx.local.conf`.
|
||||
|
||||
3. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
|
||||
#### Nginx Configuration
|
||||
|
||||
The nginx configuration provides:
|
||||
- Unified entry point on port 2026
|
||||
- Rewrites `/api/langgraph/*` to Gateway's LangGraph-compatible API (8001)
|
||||
- Routes other `/api/*` endpoints to Gateway API (8001)
|
||||
- Routes non-API requests to Frontend (3000)
|
||||
- Same-origin API routing; split-origin or port-forwarded browser clients should use the Gateway `GATEWAY_CORS_ORIGINS` allowlist
|
||||
- SSE/streaming support for real-time agent responses
|
||||
- Optimized timeouts for long-running operations
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
deer-flow/
|
||||
├── config.example.yaml # Configuration template
|
||||
├── extensions_config.example.json # MCP and Skills configuration template
|
||||
├── Makefile # Build and development commands
|
||||
├── scripts/
|
||||
│ └── docker.sh # Docker management script
|
||||
├── docker/
|
||||
│ ├── docker-compose-dev.yaml # Docker Compose configuration
|
||||
│ └── nginx/
|
||||
│ ├── nginx.conf # Nginx config for Docker
|
||||
│ └── nginx.local.conf # Nginx config for local dev
|
||||
├── backend/ # Backend application
|
||||
│ ├── packages/harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ └── deerflow/ # Agents, tools, sandbox, MCP, skills, config
|
||||
│ ├── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||
│ │ ├── gateway/ # Gateway API and LangGraph-compatible runtime (port 8001)
|
||||
│ │ └── channels/ # IM channel integrations
|
||||
│ ├── docs/ # Backend documentation
|
||||
│ └── Makefile # Backend commands
|
||||
├── frontend/ # Frontend application
|
||||
│ └── Makefile # Frontend commands
|
||||
└── skills/ # Agent skills
|
||||
├── public/ # Public skills
|
||||
└── custom/ # Custom skills
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser
|
||||
↓
|
||||
Nginx (port 2026) ← Unified entry point
|
||||
├→ Frontend (port 3000) ← / (non-API requests)
|
||||
└→ Gateway API (port 8001) ← /api/* and /api/langgraph/* (LangGraph-compatible agent interactions)
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. **Create a feature branch**:
|
||||
```bash
|
||||
git checkout -b feature/your-feature-name
|
||||
```
|
||||
|
||||
2. **Make your changes** with hot-reload enabled
|
||||
|
||||
3. **Format and lint your code** (CI will reject unformatted code):
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
make format # ruff check --fix + ruff format
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
pnpm format:write # Prettier
|
||||
```
|
||||
|
||||
4. **Test your changes** thoroughly
|
||||
|
||||
5. **Commit your changes**:
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "feat: description of your changes"
|
||||
```
|
||||
|
||||
6. **Push and create a Pull Request**:
|
||||
```bash
|
||||
git push origin feature/your-feature-name
|
||||
```
|
||||
|
||||
## AI assistance disclosure
|
||||
|
||||
DeerFlow is an AI project and we welcome AI-assisted contributions. To help
|
||||
reviewers calibrate how closely to read a change, **every pull request must
|
||||
complete the "AI assistance" section of the
|
||||
[PR template](.github/pull_request_template.md)**:
|
||||
|
||||
- which tool(s) you used (or `none`),
|
||||
- how you used them, and
|
||||
- a confirmation that a human has read, understands, and takes responsibility
|
||||
for the change.
|
||||
|
||||
Please don't delete the section. PRs that ignore it may be asked to fill it in
|
||||
before review.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Backend tests
|
||||
cd backend
|
||||
make test
|
||||
|
||||
# Frontend unit tests
|
||||
cd frontend
|
||||
make test
|
||||
|
||||
# Frontend E2E tests (requires Chromium; builds and auto-starts the Next.js production server)
|
||||
cd frontend
|
||||
make test-e2e
|
||||
```
|
||||
|
||||
### PR Regression Checks
|
||||
|
||||
Every pull request triggers the following CI workflows:
|
||||
|
||||
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
|
||||
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
|
||||
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
|
||||
|
||||
## Code Style
|
||||
|
||||
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
|
||||
- **Frontend (TypeScript)**: We use ESLint and Prettier. Run `pnpm format:write` before committing.
|
||||
- CI enforces formatting — PRs with unformatted code will fail the lint check.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration Guide](backend/docs/CONFIGURATION.md) - Setup and configuration
|
||||
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture
|
||||
- [MCP Setup Guide](backend/docs/MCP_SERVER.md) - Model Context Protocol configuration
|
||||
|
||||
## Troubleshooting Bundle
|
||||
|
||||
For setup, configuration, sandbox, or runtime issues, generate a redacted support
|
||||
summary before filing:
|
||||
|
||||
```bash
|
||||
make support-bundle
|
||||
```
|
||||
|
||||
The command prints reporter next steps, writes a `*-issue-summary.md` file that
|
||||
you can paste into the issue, writes a `*-issue-draft.md` file for AI-assisted
|
||||
issue filing, and writes an optional evidence zip under
|
||||
`.deer-flow/support-bundles/`. The zip includes toolchain versions, sanitized
|
||||
`config.yaml` and `extensions_config.json` summaries, enabled tool/skill/MCP
|
||||
structure, git metadata, and redacted `make doctor` output.
|
||||
|
||||
When filing the issue, paste the generated `*-issue-summary.md` into the issue
|
||||
body. If an AI assistant files the issue, start from `*-issue-draft.md` and
|
||||
replace every REQUIRED placeholder before filing; the draft intentionally does
|
||||
not invent reproduction steps, expected behavior, or a problem summary. Attach
|
||||
the zip only if a maintainer asks for the evidence bundle, or if the summary
|
||||
alone is not enough to diagnose the issue. Maintainers and AI-assisted triage
|
||||
should start with `triage.json`, which contains stable signals such as
|
||||
`config_missing`, `node_version_too_old`, `doctor_failed`, and suggested next
|
||||
steps. The other JSON files are evidence for follow-up inspection.
|
||||
|
||||
It intentionally does **not** include `.env`, raw conversation messages, or the
|
||||
contents of files in thread workspaces/uploads/outputs. If you need to include a
|
||||
thread, run `cd backend && uv run python ../scripts/support_bundle.py --thread-id
|
||||
<thread-id> --include-doctor`; this adds file manifests only. Please still review
|
||||
the generated zip before attaching it to a public issue.
|
||||
|
||||
## Need Help?
|
||||
|
||||
- Check existing [Issues](https://github.com/bytedance/deer-flow/issues)
|
||||
- Read the [Documentation](backend/docs/)
|
||||
- Ask questions in [Discussions](https://github.com/bytedance/deer-flow/discussions)
|
||||
|
||||
## License
|
||||
|
||||
By contributing to DeerFlow, you agree that your contributions will be licensed under the [MIT License](./LICENSE).
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
# DeerFlow Install
|
||||
|
||||
This file is for coding agents. If the DeerFlow repository is not already cloned and open, clone `https://github.com/bytedance/deer-flow.git` first, then continue from the repository root.
|
||||
|
||||
## Goal
|
||||
|
||||
Bootstrap a DeerFlow local development workspace on the user's machine with the least risky path available.
|
||||
|
||||
Default preference:
|
||||
|
||||
1. Docker development environment
|
||||
2. Local development environment
|
||||
|
||||
Do not assume API keys or model credentials exist. Set up everything that can be prepared safely, then stop with a concise summary of what the user still needs to provide.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Be idempotent. Re-running this document should not damage an existing setup.
|
||||
- Prefer existing repo commands over ad hoc shell commands.
|
||||
- Do not use `sudo` or install system packages without explicit user approval.
|
||||
- Do not overwrite existing user config values unless the user asks.
|
||||
- If a step fails, stop, explain the blocker, and provide the smallest next action.
|
||||
- If multiple setup paths are possible, prefer Docker when Docker is already available.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Consider the setup successful when all of the following are true:
|
||||
|
||||
- The DeerFlow repository is cloned and the current working directory is the repo root.
|
||||
- `config.yaml` exists.
|
||||
- For Docker setup, `make docker-init` completed successfully and Docker prerequisites are prepared, but services are not assumed to be running yet.
|
||||
- For local setup, `make check` passed or reported no missing prerequisites, and `make install` completed successfully.
|
||||
- The user receives the exact next command to launch DeerFlow.
|
||||
- The user also receives any missing model configuration or referenced environment variable names from `config.yaml`, without inspecting secret-bearing files for actual values.
|
||||
|
||||
## Steps
|
||||
|
||||
- If the current directory is not the DeerFlow repository root, clone `https://github.com/bytedance/deer-flow.git` if needed, then change into the repository root.
|
||||
- Confirm the current directory is the DeerFlow repository root by checking that `Makefile`, `backend/`, `frontend/`, and `config.example.yaml` exist.
|
||||
- Detect whether `config.yaml` already exists.
|
||||
- If `config.yaml` does not exist, run `make config`.
|
||||
- Detect whether Docker is available and the daemon is reachable with `docker info`.
|
||||
- If Docker is available:
|
||||
- Run `make docker-init`.
|
||||
- Treat this as Docker prerequisite preparation only. Do not claim that app services, compose validation, or image builds have already succeeded.
|
||||
- Do not start long-running services unless the user explicitly asks or this setup request clearly includes launch verification.
|
||||
- Tell the user the recommended next command is `make docker-start`.
|
||||
- If Docker is not available:
|
||||
- Run `make check`.
|
||||
- If `make check` reports missing system dependencies such as `node`, `pnpm`, `uv`, or `nginx`, stop and report the missing tools instead of attempting privileged installs.
|
||||
- If prerequisites are satisfied, run `make install`.
|
||||
- Tell the user the recommended next command is `make dev`.
|
||||
- Inspect `config.yaml` only for missing model entries or referenced environment variable placeholders. Do not read `.env`, `frontend/.env`, or other secret-bearing files.
|
||||
- If no model is configured, tell the user they must add at least one entry under `models` in `config.yaml`.
|
||||
- If `config.yaml` references variables such as `$OPENAI_API_KEY`, tell the user which variable names still need real values, but do not verify them by opening secret-bearing files.
|
||||
- If the repository already appears configured, avoid repeating expensive work unless it is necessary to verify the environment.
|
||||
|
||||
## Verification
|
||||
|
||||
Use the lightest verification that matches the chosen setup path.
|
||||
|
||||
For Docker setup:
|
||||
|
||||
- Confirm `make docker-init` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- State explicitly that Docker services were not started and `make docker-start` is still the first real launch step.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
For local setup:
|
||||
|
||||
- Confirm `make install` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
## Final Response Format
|
||||
|
||||
Return a short status report with:
|
||||
|
||||
1. Setup path used: Docker or local
|
||||
2. Setup level reached: Docker prerequisites prepared or local dependencies installed
|
||||
3. Files created or detected: for example `config.yaml`
|
||||
4. Remaining user action: model config, env var values, auth files, or nothing
|
||||
5. Exact next command to start DeerFlow
|
||||
|
||||
## EXECUTE NOW
|
||||
|
||||
Complete the steps above. When finished, stop after the setup boundary and report status instead of continuing into unrelated project work.
|
||||
@@ -0,0 +1,22 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
|
||||
Copyright (c) 2025-2026 DeerFlow Authors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,174 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
|
||||
.PHONY: help config config-upgrade check install setup doctor support-bundle detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon nginx stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway docker-logs-redis
|
||||
|
||||
BASH ?= bash
|
||||
BACKEND_UV_RUN = cd backend && uv run
|
||||
|
||||
# Detect OS for Windows compatibility
|
||||
ifeq ($(OS),Windows_NT)
|
||||
SHELL := cmd.exe
|
||||
PYTHON ?= python
|
||||
# Run repo shell scripts through Git Bash when Make is launched from cmd.exe / PowerShell.
|
||||
RUN_WITH_GIT_BASH = call scripts\run-with-git-bash.cmd
|
||||
else
|
||||
PYTHON ?= python3
|
||||
RUN_WITH_GIT_BASH =
|
||||
endif
|
||||
|
||||
help:
|
||||
@echo "DeerFlow Development Commands:"
|
||||
@echo " make setup - Interactive setup wizard (recommended for new users)"
|
||||
@echo " make doctor - Check configuration and system requirements"
|
||||
@echo " make support-bundle - Create a redacted issue summary, AI draft, and evidence bundle"
|
||||
@echo " make config - Generate local config files (aborts if config already exists)"
|
||||
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
|
||||
@echo " make check - Check if all required tools are installed"
|
||||
@echo " make detect-thread-boundaries - Inventory async/thread boundary points"
|
||||
@echo " make detect-blocking-io - Inventory blocking IO that may block the backend event loop"
|
||||
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@echo " make dev - Start all services in development mode (with hot-reloading)"
|
||||
@echo " make dev-daemon - Start dev services in background (daemon mode)"
|
||||
@echo " make start - Start all services in production mode (optimized, no hot-reloading)"
|
||||
@echo " make start-daemon - Start prod services in background (daemon mode)"
|
||||
@echo " make nginx - Start nginx alone in the foreground (local dev config)"
|
||||
@echo " make stop - Stop all running services"
|
||||
@echo " make clean - Clean up processes and temporary files"
|
||||
@echo ""
|
||||
@echo "Docker Production Commands:"
|
||||
@echo " make up - Build and start production Docker services (localhost:2026)"
|
||||
@echo " make down - Stop and remove production Docker containers"
|
||||
@echo ""
|
||||
@echo "Docker Development Commands:"
|
||||
@echo " make docker-init - Pull the sandbox image"
|
||||
@echo " make docker-start - Start Docker services (mode-aware from config.yaml, localhost:2026)"
|
||||
@echo " make docker-stop - Stop Docker development services"
|
||||
@echo " make docker-logs - View Docker development logs"
|
||||
@echo " make docker-logs-frontend - View Docker frontend logs"
|
||||
@echo " make docker-logs-gateway - View Docker gateway logs"
|
||||
@echo " make docker-logs-redis - View Docker Redis logs"
|
||||
|
||||
## Setup & Diagnosis
|
||||
setup:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/setup_wizard.py
|
||||
|
||||
doctor:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/doctor.py
|
||||
|
||||
support-bundle:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/support_bundle.py --include-doctor
|
||||
|
||||
detect-thread-boundaries:
|
||||
@$(PYTHON) ./scripts/detect_thread_boundaries.py
|
||||
|
||||
detect-blocking-io:
|
||||
@$(MAKE) -C backend detect-blocking-io
|
||||
|
||||
config:
|
||||
@$(PYTHON) ./scripts/configure.py
|
||||
|
||||
config-upgrade:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/config-upgrade.sh
|
||||
|
||||
# Check required tools
|
||||
check:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
|
||||
# Install all dependencies
|
||||
install:
|
||||
@echo "Installing backend dependencies..."
|
||||
@cd backend && uv sync
|
||||
@echo "Installing frontend dependencies..."
|
||||
@cd frontend && pnpm install
|
||||
@echo "Installing pre-commit hooks..."
|
||||
@uv tool install pre-commit
|
||||
@pre-commit install --overwrite
|
||||
@echo "✓ All dependencies installed"
|
||||
@echo ""
|
||||
@echo "=========================================="
|
||||
@echo " Optional: Pre-pull Sandbox Image"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@echo "If you plan to use Docker/Container-based sandbox, you can pre-pull the image:"
|
||||
@echo " make setup-sandbox"
|
||||
@echo ""
|
||||
|
||||
# Pre-pull sandbox Docker image (optional but recommended)
|
||||
setup-sandbox:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/setup-sandbox.sh
|
||||
|
||||
# Start all services in development mode (with hot-reloading)
|
||||
dev:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev
|
||||
|
||||
# Start all services in production mode (with optimizations)
|
||||
start:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod
|
||||
|
||||
# Start all services in daemon mode (background)
|
||||
dev-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --daemon
|
||||
|
||||
# Start prod services in daemon mode (background)
|
||||
start-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --daemon
|
||||
|
||||
# Start nginx alone in the foreground with the local dev config
|
||||
nginx:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/nginx.sh
|
||||
|
||||
# Stop all services
|
||||
stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --stop
|
||||
|
||||
# Clean up
|
||||
clean: stop
|
||||
@echo "Cleaning up..."
|
||||
@-rm -rf backend/.deer-flow 2>/dev/null || true
|
||||
@-rm -rf logs/*.log 2>/dev/null || true
|
||||
@echo "✓ Cleanup complete"
|
||||
|
||||
# ==========================================
|
||||
# Docker Development Commands
|
||||
# ==========================================
|
||||
|
||||
# Initialize Docker containers and install dependencies
|
||||
docker-init:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh init
|
||||
|
||||
# Start Docker development environment
|
||||
docker-start:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start
|
||||
|
||||
# Stop Docker development environment
|
||||
docker-stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh stop
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs-frontend:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --frontend
|
||||
docker-logs-gateway:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --gateway
|
||||
docker-logs-redis:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --redis
|
||||
|
||||
# ==========================================
|
||||
# Production Docker Commands
|
||||
# ==========================================
|
||||
|
||||
# Build and start production services
|
||||
up:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh
|
||||
|
||||
# Stop and remove production containers
|
||||
down:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh down
|
||||
@@ -0,0 +1,904 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
English | [中文](./README_zh.md) | [日本語](./README_ja.md) | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> On February 28th, 2026, DeerFlow claimed the 🏆 #1 spot on GitHub Trending following the launch of version 2. Thanks a million to our incredible community — you made this happen! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) is an open-source **super agent harness** that orchestrates **sub-agents**, **memory**, and **sandboxes** to do almost anything — powered by **extensible skills**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 is a ground-up rewrite.** It shares no code with v1. If you're looking for the original Deep Research framework, it's maintained on the [`1.x` branch](https://github.com/bytedance/deer-flow/tree/main-1.x) — contributions there are still welcome. Active development has moved to 2.0.
|
||||
|
||||
## Official Website
|
||||
|
||||
Learn more and see **real demos** on our [**official website**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan from ByteDance Volcengine
|
||||
|
||||
- We strongly recommend using Doubao-Seed-2.0-Code, DeepSeek v3.2 and Kimi 2.5 to run DeerFlow
|
||||
- [Learn more](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [中国大陆地区的开发者请点击这里](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow has newly integrated the intelligent search and crawling toolset independently developed by BytePlus--[InfoQuest (supports free online experience)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Official Website](#official-website)
|
||||
- [Coding Plan from ByteDance Volcengine](#coding-plan-from-bytedance-volcengine)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [One-Line Agent Setup](#one-line-agent-setup)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Configuration](#configuration)
|
||||
- [Running the Application](#running-the-application)
|
||||
- [Deployment Sizing](#deployment-sizing)
|
||||
- [Option 1: Docker (Recommended)](#option-1-docker-recommended)
|
||||
- [Option 2: Local Development](#option-2-local-development)
|
||||
- [Advanced](#advanced)
|
||||
- [Sandbox Mode](#sandbox-mode)
|
||||
- [MCP Server](#mcp-server)
|
||||
- [IM Channels](#im-channels)
|
||||
- [LangSmith Tracing](#langsmith-tracing)
|
||||
- [Langfuse Tracing](#langfuse-tracing)
|
||||
- [Using Both Providers](#using-both-providers)
|
||||
- [From Deep Research to Super Agent Harness](#from-deep-research-to-super-agent-harness)
|
||||
- [Core Features](#core-features)
|
||||
- [Skills \& Tools](#skills--tools)
|
||||
- [Claude Code Integration](#claude-code-integration)
|
||||
- [Session Goals](#session-goals)
|
||||
- [Manual Context Compaction](#manual-context-compaction)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox \& File System](#sandbox--file-system)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Long-Term Memory](#long-term-memory)
|
||||
- [Recommended Models](#recommended-models)
|
||||
- [Embedded Python Client](#embedded-python-client)
|
||||
- [Scheduled Tasks](#scheduled-tasks)
|
||||
- [Terminal Workbench (TUI)](#terminal-workbench-tui)
|
||||
- [Documentation](#documentation)
|
||||
- [⚠️ Security Notice](#️-security-notice)
|
||||
- [Improper Deployment May Introduce Security Risks](#improper-deployment-may-introduce-security-risks)
|
||||
- [Security Recommendations](#security-recommendations)
|
||||
- [Contributing](#contributing)
|
||||
- [License](#license)
|
||||
- [Acknowledgments](#acknowledgments)
|
||||
- [Key Contributors](#key-contributors)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## One-Line Agent Setup
|
||||
|
||||
If you use Claude Code, Codex, Cursor, Windsurf, or another coding agent, you can hand it the setup instructions in one sentence:
|
||||
|
||||
```text
|
||||
Help me clone DeerFlow if needed, then bootstrap it for local development by following https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
That prompt is intended for coding agents. It tells the agent to clone the repo if needed, choose Docker when available, and stop with the exact next command plus any missing config the user still needs to provide.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Configuration
|
||||
|
||||
1. **Clone the DeerFlow repository**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Run the setup wizard**
|
||||
|
||||
From the project root directory (`deer-flow/`), run:
|
||||
|
||||
```bash
|
||||
make setup
|
||||
```
|
||||
|
||||
This launches an interactive wizard that guides you through choosing an LLM provider, optional web search, and execution/safety preferences such as sandbox mode, bash access, and file-write tools. It generates a minimal `config.yaml` and writes your keys to `.env`. Takes about 2 minutes.
|
||||
|
||||
The wizard also lets you configure an optional web search provider, or skip it for now.
|
||||
|
||||
Run `make doctor` at any time to verify your setup and get actionable fix hints.
|
||||
If you are opening a GitHub issue about a local setup or runtime problem, run
|
||||
`make support-bundle`. The command prints reporter next steps, writes a
|
||||
`*-issue-summary.md` file to paste into the issue, a `*-issue-draft.md` file
|
||||
for AI-assisted issue filing, and an optional evidence zip under
|
||||
`.deer-flow/support-bundles/`. If an AI assistant files the issue, start from
|
||||
the draft and replace every REQUIRED placeholder instead of inventing missing
|
||||
facts. Attach the zip only if a maintainer asks for it, or if the summary
|
||||
alone is not enough. Maintainers and AI triage tools can start with
|
||||
`triage.json`; the bundle includes redacted diagnostics and file manifests
|
||||
only, and does not include `.env`, raw conversation messages, or user file
|
||||
contents.
|
||||
|
||||
> **Advanced / manual configuration**: If you prefer to edit `config.yaml` directly, run `make config` instead to copy the full template. See `config.example.yaml` for the complete reference including CLI-backed providers (Codex CLI, Claude Code OAuth), OpenRouter, Responses API, and more.
|
||||
|
||||
<details>
|
||||
<summary>Manual model configuration examples</summary>
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
```
|
||||
|
||||
OpenRouter and similar OpenAI-compatible gateways should be configured with `langchain_openai:ChatOpenAI` plus `base_url`. If you prefer a provider-specific environment variable name, point `api_key` at that variable explicitly (for example `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
To route OpenAI models through `/v1/responses`, keep using `langchain_openai:ChatOpenAI` and set `use_responses_api: true` with `output_version: responses/v1`.
|
||||
|
||||
For vLLM 0.19.0, use `deerflow.models.vllm_provider:VllmChatModel`. For Qwen-style reasoning models, DeerFlow toggles reasoning with `extra_body.chat_template_kwargs.enable_thinking` and preserves vLLM's non-standard `reasoning` field across multi-turn tool-call conversations. Legacy `thinking` configs are normalized automatically for backward compatibility. Reasoning models may also require the server to be started with `--reasoning-parser ...`. If your local vLLM deployment accepts any non-empty API key, you can still set `VLLM_API_KEY` to a placeholder value.
|
||||
|
||||
CLI-backed provider examples:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI reads `~/.codex/auth.json`
|
||||
- Claude Code accepts `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_CREDENTIALS_PATH`, or `~/.claude/.credentials.json`
|
||||
- ACP agent entries are separate from model providers — if you configure `acp_agents.codex`, point it at a Codex ACP adapter such as `npx -y @zed-industries/codex-acp`
|
||||
- On macOS, export Claude Code auth explicitly if needed:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
API keys can also be set manually in `.env` (recommended) or exported in your shell:
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Running the Application
|
||||
|
||||
#### Deployment Sizing
|
||||
|
||||
Use the table below as a practical starting point when choosing how to run DeerFlow:
|
||||
|
||||
| Deployment target | Starting point | Recommended | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| Local evaluation / `make dev` | 4 vCPU, 8 GB RAM, 20 GB free SSD | 8 vCPU, 16 GB RAM | Good for one developer or one light session with hosted model APIs. `2 vCPU / 4 GB` is usually not enough. |
|
||||
| Docker development / `make docker-start` | 4 vCPU, 8 GB RAM, 25 GB free SSD | 8 vCPU, 16 GB RAM | Image builds, bind mounts, and sandbox containers need more headroom than pure local dev. |
|
||||
| Long-running server / `make up` | 8 vCPU, 16 GB RAM, 40 GB free SSD | 16 vCPU, 32 GB RAM | Preferred for shared use, multi-agent runs, report generation, or heavier sandbox workloads. |
|
||||
|
||||
- These numbers cover DeerFlow itself. If you also host a local LLM, size that service separately.
|
||||
- Linux plus Docker is the recommended deployment target for a persistent server. macOS and Windows are best treated as development or evaluation environments.
|
||||
- If CPU or memory usage stays pinned, reduce concurrent runs first, then move to the next sizing tier.
|
||||
|
||||
#### Option 1: Docker (Recommended)
|
||||
|
||||
**Development** (hot-reload, source mounts):
|
||||
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` starts `provisioner` only when `config.yaml` uses provisioner mode (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` with `provisioner_url`).
|
||||
|
||||
Docker builds use the upstream `uv` registry by default. If you need faster mirrors in restricted networks, export `UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple` and `NPM_REGISTRY=https://registry.npmmirror.com` before running `make docker-init` or `make docker-start`.
|
||||
|
||||
Backend processes automatically pick up `config.yaml` changes on the next config access, so model metadata updates do not require a manual restart during development.
|
||||
|
||||
> [!TIP]
|
||||
> On Linux, if Docker-based commands fail with `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, add your user to the `docker` group and re-login before retrying. See [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) for the full fix.
|
||||
|
||||
**Production** (builds images locally, mounts runtime config and data):
|
||||
|
||||
```bash
|
||||
make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
Access: http://localhost:2026
|
||||
|
||||
For persistent deployments, configure `database.backend` as `sqlite` or
|
||||
`postgres`. The selected backend is shared by the LangGraph checkpointer,
|
||||
LangGraph Store, and DeerFlow application data. The deprecated `checkpointer`
|
||||
section, when present, overrides the first two for backward compatibility.
|
||||
|
||||
The unified nginx endpoint is same-origin by default and does not emit browser CORS headers. If you run a split-origin or port-forwarded browser client, set `GATEWAY_CORS_ORIGINS` to comma-separated exact origins such as `http://localhost:3000`; the Gateway then applies the CORS allowlist and matching CSRF origin checks.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The Gateway still owns active run tasks in process, so production defaults to a single Gateway worker (`GATEWAY_WORKERS=1`). The Redis stream bridge (`stream_bridge.type: redis`) shares SSE delivery and `Last-Event-ID` replay across workers, with a rolling retained-buffer TTL (`stream_ttl_seconds`) as a cleanup safety net. Malformed reconnect IDs live-tail new events instead of replaying the retained buffer. It does not make run cancellation, request de-duplication, or IM channel state fully cross-worker by itself; use single-worker Gateway or explicit sticky routing/ownership before raising `GATEWAY_WORKERS`.
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
|
||||
|
||||
#### Option 2: Local Development
|
||||
|
||||
If you prefer running services locally:
|
||||
|
||||
Prerequisite: complete the "Configuration" steps above first (`make setup`). `make dev` requires a valid `config.yaml` in the project root. Set `DEER_FLOW_PROJECT_ROOT` to define that root explicitly, or `DEER_FLOW_CONFIG_PATH` to point at a specific config file. Runtime state defaults to `.deer-flow` under the project root and can be moved with `DEER_FLOW_HOME`; skills default to `skills/` under the project root and can be moved with `DEER_FLOW_SKILLS_PATH`. Run `make doctor` to verify your setup before starting.
|
||||
On Windows, run the local development flow from Git Bash. Native `cmd.exe` and PowerShell shells are not supported for the bash-based service scripts, and WSL is not guaranteed because some scripts rely on Git for Windows utilities such as `cygpath`.
|
||||
|
||||
1. **Check prerequisites**:
|
||||
```bash
|
||||
make check # Verifies Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Install dependencies**:
|
||||
```bash
|
||||
make install # Install backend + frontend dependencies + pre-commit hooks
|
||||
```
|
||||
|
||||
3. **(Optional) Pre-pull sandbox image**:
|
||||
```bash
|
||||
# Recommended if using Docker/Container-based sandbox
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **(Optional) Load sample memory data for local review**:
|
||||
```bash
|
||||
python scripts/load_memory_sample.py
|
||||
```
|
||||
This copies the sample fixture into the default local runtime memory file so reviewers can immediately test `Settings > Memory`.
|
||||
See [backend/docs/MEMORY_SETTINGS_REVIEW.md](backend/docs/MEMORY_SETTINGS_REVIEW.md) for the shortest review flow.
|
||||
|
||||
5. **Start services**:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
6. **Access**: http://localhost:2026
|
||||
|
||||
#### Startup Modes
|
||||
|
||||
DeerFlow runs the agent runtime inside the Gateway API. Development mode enables hot-reload; production mode uses a pre-built frontend.
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
Gateway owns `/api/langgraph/*` and translates those public LangGraph-compatible paths to its native `/api/*` routers behind nginx.
|
||||
|
||||
#### Docker Production Deployment
|
||||
|
||||
`deploy.sh` supports building and starting separately:
|
||||
|
||||
```bash
|
||||
# One-step (build + start)
|
||||
deploy.sh
|
||||
|
||||
# Two-step (build once, start later)
|
||||
deploy.sh build # build all images
|
||||
deploy.sh start # start pre-built images
|
||||
|
||||
# Stop
|
||||
deploy.sh down
|
||||
```
|
||||
|
||||
### Advanced
|
||||
#### Sandbox Mode
|
||||
|
||||
DeerFlow supports multiple sandbox execution modes:
|
||||
- **Local Execution** (runs sandbox code directly on the host machine)
|
||||
- **Docker Execution** (runs sandbox code in isolated Docker containers)
|
||||
- **Docker Execution with Kubernetes** (runs sandbox code in Kubernetes pods via provisioner service)
|
||||
|
||||
For Docker development, service startup follows `config.yaml` sandbox mode. In Local/Docker modes, `provisioner` is not started.
|
||||
|
||||
See the [Sandbox Configuration Guide](backend/docs/CONFIGURATION.md#sandbox) to configure your preferred mode.
|
||||
|
||||
#### MCP Server
|
||||
|
||||
DeerFlow supports configurable MCP servers and skills to extend its capabilities.
|
||||
For HTTP/SSE MCP servers, OAuth token flows are supported (`client_credentials`, `refresh_token`).
|
||||
For stdio MCP servers, per-tool call timeouts can be configured with `tool_call_timeout`.
|
||||
MCP routing hints can also prefer a specific MCP tool for matching requests without forbidding other tools. When `tool_search` defers MCP schemas, matching routing metadata can auto-promote up to `tool_search.auto_promote_top_k` deferred schemas before the model call.
|
||||
See the [MCP Server Guide](backend/docs/MCP_SERVER.md) for detailed instructions.
|
||||
|
||||
#### IM Channels
|
||||
|
||||
DeerFlow supports receiving tasks from messaging apps. Channels auto-start when configured — no public IP required for any of them.
|
||||
|
||||
DeerFlow can also expose user-owned IM channel connections in the workspace UI. When `channel_connections` is enabled, logged-in users can bind Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, or WeCom from the sidebar / Settings > Channels. It reuses the existing outbound `channels.*` transports, so no public IP or provider callback URL is required. Incoming IM messages then run under the connected DeerFlow user account. See [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md) for setup and security notes.
|
||||
|
||||
| Channel | Transport | Difficulty |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API (long-polling) | Easy |
|
||||
| Slack | Socket Mode | Moderate |
|
||||
| Feishu / Lark | WebSocket | Moderate |
|
||||
| WeChat | Tencent iLink (long-polling) | Moderate |
|
||||
| WeCom | WebSocket | Moderate |
|
||||
| DingTalk | Stream Push (WebSocket) | Moderate |
|
||||
|
||||
**Configuration in `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# Optional: global session defaults for all mobile channels
|
||||
session:
|
||||
assistant_id: lead_agent # or a custom agent name; custom agents are routed via lead_agent + agent_name
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # optional: allow first-time QR bootstrap when bot_token is absent
|
||||
allowed_users: [] # empty = allow all
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
# Optional: per-channel / per-user session settings
|
||||
session:
|
||||
assistant_id: mobile-agent # custom agent names are also supported here
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip-agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # Client ID of your DingTalk application
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # Client Secret of your DingTalk application
|
||||
allowed_users: [] # empty = allow all
|
||||
card_template_id: "" # Optional: AI Card template ID for streaming typewriter effect
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `assistant_id: lead_agent` calls the default LangGraph assistant directly.
|
||||
- If `assistant_id` is set to a custom agent name, DeerFlow still routes through `lead_agent` and injects that value as `agent_name`, so the custom agent's SOUL/config takes effect for IM channels.
|
||||
- IM channel workers call Gateway's LangGraph-compatible API internally and automatically attach process-local internal auth plus the CSRF cookie/header pair required for thread and run creation.
|
||||
- Feishu/Lark now queues rapid follow-up messages per mapped DeerFlow `thread_id` instead of immediately surfacing the generic busy reply, and topic replies keep a per-message card with a compact source-message preview across queued/running/final patches.
|
||||
|
||||
Set the corresponding API keys in your `.env` file:
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegram Setup**
|
||||
|
||||
1. Chat with [@BotFather](https://t.me/BotFather), send `/newbot`, and copy the HTTP API token.
|
||||
2. Set `TELEGRAM_BOT_TOKEN` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**Slack Setup**
|
||||
|
||||
1. Create a Slack App at [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
|
||||
2. Under **OAuth & Permissions**, add Bot Token Scopes: `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
|
||||
3. Enable **Socket Mode** → generate an App-Level Token (`xapp-…`) with `connections:write` scope.
|
||||
4. Under **Event Subscriptions**, subscribe to bot events: `app_mention`, `message.im`.
|
||||
5. Set `SLACK_BOT_TOKEN` and `SLACK_APP_TOKEN` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**Feishu / Lark Setup**
|
||||
|
||||
1. Create an app on [Feishu Open Platform](https://open.feishu.cn/) → enable **Bot** capability.
|
||||
2. Add permissions: `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
|
||||
3. Under **Events**, subscribe to `im.message.receive_v1` and select **Long Connection** mode.
|
||||
4. Copy the App ID and App Secret. Set `FEISHU_APP_ID` and `FEISHU_APP_SECRET` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**WeChat Setup**
|
||||
|
||||
1. Enable the `wechat` channel in `config.yaml`.
|
||||
2. Either set `WECHAT_BOT_TOKEN` in `.env`, or set `qrcode_login_enabled: true` for first-time QR bootstrap.
|
||||
3. When `bot_token` is absent and QR bootstrap is enabled, watch backend logs for the QR content returned by iLink and complete the binding flow.
|
||||
4. After the QR flow succeeds, DeerFlow persists the acquired token under `state_dir` for later restarts.
|
||||
5. For Docker Compose deployments, keep `state_dir` on a persistent volume so the `get_updates_buf` cursor and saved auth state survive restarts.
|
||||
|
||||
**WeCom Setup**
|
||||
|
||||
1. Create a bot on the WeCom AI Bot platform and obtain the `bot_id` and `bot_secret`.
|
||||
2. Enable `channels.wecom` in `config.yaml` and fill in `bot_id` / `bot_secret`.
|
||||
3. Set `WECOM_BOT_ID` and `WECOM_BOT_SECRET` in `.env`.
|
||||
4. Make sure backend dependencies include `wecom-aibot-python-sdk`. The channel uses a WebSocket long connection and does not require a public callback URL.
|
||||
5. The current integration supports inbound text, image, and file messages. Final images/files generated by the agent are also sent back to the WeCom conversation.
|
||||
|
||||
**DingTalk Setup**
|
||||
|
||||
1. Create a DingTalk application in the [DingTalk Developer Console](https://open.dingtalk.com/) and enable **Robot** capability.
|
||||
2. Set the message receiving mode to **Stream Mode** in the robot configuration page.
|
||||
3. Copy the `Client ID` and `Client Secret`, set `DINGTALK_CLIENT_ID` and `DINGTALK_CLIENT_SECRET` in `.env`, and enable the channel in `config.yaml`.
|
||||
4. *(Optional)* To enable streaming AI Card replies (typewriter effect), create an **AI Card** template on the [DingTalk Card Platform](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), then set `card_template_id` in `config.yaml` to the template ID. You also need to apply for the `Card.Streaming.Write` and `Card.Instance.Write` permissions.
|
||||
|
||||
|
||||
When DeerFlow runs in Docker Compose, IM channels execute inside the `gateway` container. In that case, do not point `channels.langgraph_url` or `channels.gateway_url` at `localhost`; use container service names such as `http://gateway:8001/api` and `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` and `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
|
||||
**Commands**
|
||||
|
||||
Once a channel is connected, you can interact with DeerFlow directly from the chat:
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/new` | Start a new conversation |
|
||||
| `/status` | Show current thread info |
|
||||
| `/models` | List available models |
|
||||
| `/memory` | View memory |
|
||||
| `/help` | Show help |
|
||||
|
||||
> Messages without a command prefix are treated as regular chat — DeerFlow creates a thread and responds conversationally.
|
||||
|
||||
#### Request Trace Correlation
|
||||
|
||||
Gateway request trace correlation is disabled by default so existing HTTP responses and log formats stay unchanged. To enable it, set:
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
enhance:
|
||||
enabled: true
|
||||
format: text
|
||||
```
|
||||
|
||||
When enabled, every Gateway HTTP response includes `X-Trace-Id`, logs include `trace_id`, and Langfuse traces created by that request include `metadata.deerflow_trace_id` with the same value.
|
||||
|
||||
#### LangSmith Tracing
|
||||
|
||||
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, and tool executions are traced and visible in the LangSmith dashboard.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
#### Langfuse Tracing
|
||||
|
||||
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
If you are using a self-hosted Langfuse instance, set `LANGFUSE_BASE_URL` to your deployment URL.
|
||||
|
||||
**Trace correlation fields.** Every agent run is annotated with Langfuse's reserved trace attributes so the Sessions and Users pages light up automatically:
|
||||
|
||||
- `session_id` = LangGraph `thread_id` — groups every trace of the same conversation
|
||||
- `user_id` = effective user from `get_effective_user_id()` (falls back to `default` in no-auth mode)
|
||||
- `trace_name` = assistant id (defaults to `lead-agent`)
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]` (omitted when not set)
|
||||
- `metadata.deerflow_trace_id` = DeerFlow request correlation id, matching `X-Trace-Id` when request trace correlation is enabled
|
||||
|
||||
These are injected into `RunnableConfig.metadata` at the graph invocation root for both the gateway path (`runtime/runs/worker.py::run_agent`) and the embedded path (`client.py::DeerFlowClient.stream`), so any LangChain-compatible callback can read them. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment.
|
||||
|
||||
#### Using Both Providers
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow attaches both tracing callbacks and reports the same model activity to both systems.
|
||||
|
||||
If a provider is explicitly enabled but missing required credentials, or if its callback fails to initialize, DeerFlow fails fast when tracing is initialized during model creation and the error message names the provider that caused the failure.
|
||||
|
||||
For Docker deployments, tracing is disabled by default. Set `LANGSMITH_TRACING=true` and `LANGSMITH_API_KEY` in your `.env` to enable it.
|
||||
|
||||
## From Deep Research to Super Agent Harness
|
||||
|
||||
DeerFlow started as a Deep Research framework — and the community ran with it. Since launch, developers have pushed it far beyond research: building data pipelines, generating slide decks, spinning up dashboards, automating content workflows. Things we never anticipated.
|
||||
|
||||
That told us something important: DeerFlow wasn't just a research tool. It was a **harness** — a runtime that gives agents the infrastructure to actually get work done.
|
||||
|
||||
So we rebuilt it from scratch.
|
||||
|
||||
DeerFlow 2.0 is no longer a framework you wire together. It's a super agent harness — batteries included, fully extensible. Built on LangGraph and LangChain, it ships with everything an agent needs out of the box: a filesystem, memory, skills, sandbox-aware execution, and the ability to plan and spawn sub-agents for complex, multi-step tasks.
|
||||
|
||||
Use it as-is. Or tear it apart and make it yours.
|
||||
|
||||
## Core Features
|
||||
|
||||
### Skills & Tools
|
||||
|
||||
Skills are what make DeerFlow do *almost anything*.
|
||||
|
||||
A standard Agent Skill is a structured capability module — a Markdown file that defines a workflow, best practices, and references to supporting resources. DeerFlow ships with built-in skills for research, report generation, slide creation, web pages, image and video generation, and more. But the real power is extensibility: add your own skills, replace the built-in ones, or combine them into compound workflows.
|
||||
|
||||
Skills are loaded progressively — only when the task needs them, not all at once. This keeps the context window lean and makes DeerFlow work well even with token-sensitive models.
|
||||
|
||||
Users can explicitly activate an enabled skill for a single turn by starting the request with `/skill-name`, for example `/data-analysis analyze uploads/foo.csv`. DeerFlow loads that skill's `SKILL.md` as hidden current-turn context while leaving the base prompt limited to skill metadata. Slash activation respects disabled skills, custom-agent skill whitelists, and existing channel commands such as `/new` and `/help`.
|
||||
|
||||
When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
|
||||
|
||||
Skill installs and agent-managed skill edits run through **SkillScan**, a native deterministic safety scanner before the LLM-based skill scanner. Phase 1 runs offline with no Semgrep/OpenGrep dependency, blocks high-confidence `CRITICAL` findings such as private keys or shell execution, and passes warning findings to the LLM scanner for contextual review. Set `skill_scan.enabled: false` in `config.yaml` to disable only the deterministic analyzers; safe archive extraction and the LLM scanner still run.
|
||||
|
||||
DeerFlow also ships with **skill-reviewer**, a public skill for read-only skill quality review. It uses the built-in `review_skill_package` tool to inspect installed skills, local packages, archives, or pasted `SKILL.md` content without activating the target skill, binding its secrets, executing its scripts, or installing it. The tool returns a compact, tag-neutralized JSON payload to the model context and keeps the full raw review payload in the tool artifact for programmatic consumers. The deterministic review core reuses DeerFlow parsing and SkillScan facts, emits versioned JSON contracts under `contracts/skill_review/`, and can be run from the backend CLI:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
uv run python -m deerflow.skills.review.cli ../skills/public/data-analysis --format text --fail-on error --fail-on-incomplete
|
||||
```
|
||||
|
||||
Tools follow the same philosophy. DeerFlow comes with a core toolset — web search, web fetch, rendered web capture, file operations, bash execution — and supports custom tools via MCP servers and Python functions. Swap anything. Add anything.
|
||||
|
||||
Gateway-generated follow-up suggestions now normalize both plain-string model output and block/list-style rich content before parsing the JSON array response, so provider-specific content wrappers do not silently drop suggestions.
|
||||
|
||||
The Web UI composer can polish draft input before sending. The rewrite runs as a short Gateway LLM request using the `input_polish` model configuration, keeps slash skill prefixes such as `/data-analysis`, and only replaces the local draft after the user clicks the polish button; it does not create a thread run or persist a message.
|
||||
|
||||
The Web UI composer also supports browser-based voice dictation when the browser exposes the Web Speech API. The microphone button transcribes speech into the local draft only; DeerFlow receives only the transcribed text, while audio handling is delegated to the browser or operating system speech-recognition service according to that environment's policy. Users can review or edit the text before sending.
|
||||
|
||||
Interrupted first-turn runs still persist a fallback conversation title, so stopping a streaming response does not leave the thread as "Untitled" after refresh.
|
||||
|
||||
In the Web UI, completed assistant turns can be branched into a new main conversation. The new thread starts from that turn's checkpoint. Because workspace files are not checkpointed, the branch only receives a best-effort copy of the current workspace when you branch from the latest turn; branching from an older turn keeps just the restored message history so the branch never inherits files that were created in a later part of the conversation.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← yours
|
||||
```
|
||||
|
||||
#### Claude Code Integration
|
||||
|
||||
The `claude-to-deerflow` skill lets you interact with a running DeerFlow instance directly from [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Send research tasks, check status, manage threads — all without leaving the terminal.
|
||||
|
||||
**Install the skill**:
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
Then make sure DeerFlow is running (default at `http://localhost:2026`) and use the `/claude-to-deerflow` command in Claude Code.
|
||||
|
||||
**What you can do**:
|
||||
- Send messages to DeerFlow and get streaming responses
|
||||
- Choose execution modes: flash (fast), standard, pro (planning), ultra (sub-agents)
|
||||
- Check DeerFlow health, list models/skills/agents
|
||||
- Manage threads and conversation history
|
||||
- Upload files for analysis
|
||||
|
||||
**Environment variables** (optional, for custom endpoints):
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
See [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) for the full API reference.
|
||||
|
||||
### Session Goals
|
||||
|
||||
Use `/goal <completion condition>` to attach one active completion condition to the current thread. The goal is thread-scoped state, not a skill activation, so it stays active across turns until DeerFlow determines it has been satisfied or you clear it.
|
||||
|
||||
Supported commands:
|
||||
|
||||
```text
|
||||
/goal finish the implementation and make all tests pass
|
||||
/goal # show the active goal
|
||||
/goal clear # clear it
|
||||
```
|
||||
|
||||
After each Gateway-backed run, DeerFlow evaluates the visible conversation against the active goal with a non-thinking evaluator model. The evaluator must return a typed blocker (`missing_evidence`, `needs_user_input`, `run_failed`, `external_wait`, or `goal_not_met_yet`) plus visible evidence. DeerFlow only injects a hidden continuation when the latest assistant turn is durably checkpointed, the blocker is `goal_not_met_yet`, the thread did not change during evaluation, and the no-progress breaker has not fired. The safety cap defaults to 8 hidden continuations, and repeated identical non-progress evaluations stop after 2 attempts. `/goal clear` and any user-authored new input win over queued continuations. When the goal is satisfied, DeerFlow clears it automatically and publishes the updated thread state.
|
||||
|
||||
The Web UI shows the active goal above the composer. The same command is available from the TUI and supported IM channels. In the Web UI and supported IM channels, setting `/goal <completion condition>` also starts a run with the condition as the task; status and clear commands only manage goal state.
|
||||
|
||||
### Manual Context Compaction
|
||||
|
||||
Use `/compact` in the Web UI composer to summarize older context for the current thread. DeerFlow keeps the full chat visible, but future model calls use the compacted summary plus recent messages. The command is ignored when there is not enough history to compact, and it is blocked while the thread has a run in flight.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Complex tasks rarely fit in a single pass. DeerFlow decomposes them.
|
||||
|
||||
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions. Sub-agents run in parallel when possible, report back structured results, and the lead agent synthesizes everything into a coherent output. Long-running sub-agents compact older history when summarization is enabled and re-inject the summary as guarded, hidden durable context before continuing, so recent assistant/tool activity remains grounded in the task. Provider/model request failures are reported as failed sub-agent tasks rather than successful results, so the lead agent and Web UI can react to them correctly. Collapsed sub-agent cards show the effective model and, when the provider returns usage metadata, a cumulative token total that updates after each completed sub-agent LLM call and persists after a reload. When token usage tracking is enabled, completed sub-agent usage is also attributed back to the dispatching step.
|
||||
|
||||
This is how DeerFlow handles tasks that take minutes to hours: a research task might fan out into a dozen sub-agents, each exploring a different angle, then converge into a single report — or a website — or a slide deck with generated visuals. One harness, many hands.
|
||||
|
||||
### Sandbox & File System
|
||||
|
||||
DeerFlow doesn't just *talk* about doing things. It has its own computer.
|
||||
|
||||
Each task gets its own execution environment with a full filesystem view — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It can view images and, when configured safely, execute shell commands.
|
||||
|
||||
After each run, DeerFlow records a workspace change summary for the run-owned `workspace` and `outputs` directories. The Web UI shows a compact "files changed" badge on the assistant turn; opening it reveals created, modified, and deleted files with text diffs when safe to display. Uploads are excluded because they are user inputs, not agent-generated changes. Large, binary, or sensitive-looking files are shown as metadata only.
|
||||
|
||||
With `AioSandboxProvider`, shell execution runs inside isolated containers. With `LocalSandboxProvider`, file tools still map to per-thread directories on the host, but host `bash` is disabled by default because it is not a secure isolation boundary. Re-enable host bash only for fully trusted local workflows. Host bash commands have a wall-clock timeout, and long-lived processes should be started in the background with output redirected to a workspace log.
|
||||
|
||||
This is the difference between a chatbot with tool access and an agent with an actual execution environment.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← your files
|
||||
├── workspace/ ← agents' working directory
|
||||
└── outputs/ ← final deliverables
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Isolated Sub-Agent Context**: Each sub-agent runs in its own isolated context. This means that the sub-agent will not be able to see the context of the main agent or other sub-agents. This is important to ensure that the sub-agent is able to focus on the task at hand and not be distracted by the context of the main agent or other sub-agents.
|
||||
|
||||
**Summarization**: Within a session, DeerFlow manages context aggressively — summarizing completed sub-tasks, offloading intermediate results to the filesystem, compressing what's no longer immediately relevant. This lets it stay sharp across long, multi-step tasks without blowing the context window.
|
||||
|
||||
**Strict Tool-Call Recovery**: When a provider or middleware interrupts a tool-call loop, DeerFlow now strips provider-level raw tool-call metadata on forced-stop assistant messages and injects placeholder tool results for dangling calls before the next model invocation. This keeps OpenAI-compatible reasoning models that strictly validate `tool_call_id` sequences from failing with malformed history errors.
|
||||
|
||||
**Visible Tool-Run Completion**: For interactive turns, DeerFlow retries an empty post-tool final response once, then surfaces a visible error instead of reporting a silent successful run.
|
||||
|
||||
### Long-Term Memory
|
||||
|
||||
Most agents forget everything the moment a conversation ends. DeerFlow remembers.
|
||||
|
||||
Across sessions, DeerFlow builds a persistent memory of your profile, preferences, and accumulated knowledge. The more you use it, the better it knows you — your writing style, your technical stack, your recurring workflows. Memory is stored locally and stays under your control.
|
||||
|
||||
Memory updates now skip duplicate fact entries at apply time, so repeated preferences and context do not accumulate endlessly across sessions.
|
||||
|
||||
## Recommended Models
|
||||
|
||||
DeerFlow is model-agnostic — it works with any LLM that implements the OpenAI-compatible API. That said, it performs best with models that support:
|
||||
|
||||
- **Long context windows** (100k+ tokens) for deep research and multi-step tasks
|
||||
- **Reasoning capabilities** for adaptive planning and complex decomposition
|
||||
- **Multimodal inputs** for image understanding and video comprehension
|
||||
- **Strong tool-use** for reliable function calling and structured outputs
|
||||
|
||||
## Embedded Python Client
|
||||
|
||||
DeerFlow can be used as an embedded Python library without running the full HTTP services. The `DeerFlowClient` provides direct in-process access to all agent and Gateway capabilities, returning the same response schemas as the HTTP Gateway API. The HTTP Gateway also exposes `DELETE /api/threads/{thread_id}` to remove DeerFlow-managed local thread data after the LangGraph thread itself has been deleted:
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
client.set_goal("thread-1", "finish the implementation and make all tests pass")
|
||||
client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
|
||||
client.clear_goal("thread-1")
|
||||
```
|
||||
|
||||
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/packages/harness/deerflow/client.py` for full API documentation.
|
||||
|
||||
## Scheduled Tasks
|
||||
|
||||
DeerFlow now includes a first-class scheduled-task MVP in the workspace.
|
||||
|
||||
Current MVP capabilities:
|
||||
|
||||
- Manage tasks at `/workspace/scheduled-tasks`
|
||||
- Choose whether each scheduled task reuses a thread or creates a fresh thread per run
|
||||
- Support `once` and `cron` schedules
|
||||
- Run background scheduled executions as non-interactive DeerFlow runs (`ask_clarification` is not exposed there)
|
||||
- Use `skip` overlap behavior for due cron executions that collide with an active run on the same reused thread
|
||||
- Pause, resume, trigger, inspect history, and delete tasks
|
||||
- Execute scheduled work through the normal DeerFlow run lifecycle
|
||||
|
||||
Current MVP limits:
|
||||
|
||||
- No conversation-created `schedule_task` tool yet
|
||||
- No text-only notification jobs
|
||||
- No channel or GitHub dispatch targets
|
||||
- No `interval` schedule type in this first cut
|
||||
|
||||
Enable background polling with `config.yaml -> scheduler.enabled`. Manual trigger uses the same scheduled-task resource and execution path.
|
||||
|
||||
## Terminal Workbench (TUI)
|
||||
|
||||
`deerflow` is a terminal-native workbench for people who live in the shell. It runs **embedded** over `DeerFlowClient` — no Gateway, frontend, nginx, or Docker required — while honoring the same `config.yaml`, checkpointer, skills, memory, MCP, and sandbox settings as the rest of DeerFlow.
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # optional 'textual' dependency
|
||||
|
||||
deerflow # launch the terminal UI (TTY required)
|
||||
deerflow --continue # resume the most recent thread
|
||||
deerflow --resume THREAD # resume a thread by id
|
||||
deerflow --print "summarize this repo" # headless one-shot answer to stdout
|
||||
deerflow --json "hello" # headless newline-delimited StreamEvents
|
||||
```
|
||||
|
||||
A keyboard-driven chat surface with a streaming transcript (Markdown-rendered answers), compact tool-activity cards, a `/` slash-command palette, `/goal` goal management, `/model` and `/threads` pickers, input history, and `Esc` / `Ctrl+C` interrupt. Sessions opened in the TUI also appear in the Web UI sidebar — it writes the shared thread store under the local default user, so terminal and web stay in sync **without running the Gateway**.
|
||||
|
||||
See [backend/docs/TUI.md](backend/docs/TUI.md) for the full guide.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Contributing Guide](CONTRIBUTING.md) - Development environment setup and workflow
|
||||
- [Configuration Guide](backend/docs/CONFIGURATION.md) - Setup and configuration instructions
|
||||
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture details
|
||||
- [Backend Architecture](backend/README.md) - Backend architecture and API reference
|
||||
|
||||
## ⚠️ Security Notice
|
||||
|
||||
### Improper Deployment May Introduce Security Risks
|
||||
|
||||
DeerFlow has key high-privilege capabilities including **system command execution, resource operations, and business logic invocation**, and is designed by default to be **deployed in a local trusted environment (accessible only via the 127.0.0.1 loopback interface)**. If you deploy the agent in untrusted environments — such as LAN networks, public cloud servers, or other multi-endpoint accessible environments — without strict security measures, it may introduce security risks, including:
|
||||
|
||||
- **Unauthorized illegal invocation**: Agent functionality could be discovered by unauthorized third parties or malicious internet scanners, triggering bulk unauthorized requests that execute high-risk operations such as system commands and file read/write, potentially causing serious security consequences.
|
||||
- **Compliance and legal risks**: If the agent is illegally invoked to conduct cyberattacks, data theft, or other illegal activities, it may result in legal liability and compliance risks.
|
||||
|
||||
### Security Recommendations
|
||||
|
||||
**Note: We strongly recommend deploying DeerFlow in a local trusted network environment.** If you need cross-device or cross-network deployment, you must implement strict security measures, such as:
|
||||
|
||||
- **IP allowlist**: Use `iptables`, or deploy hardware firewalls / switches with Access Control Lists (ACL), to **configure IP allowlist rules** and deny access from all other IP addresses.
|
||||
- **Authentication gateway**: Configure a reverse proxy (e.g., nginx) and **enable strong pre-authentication**, blocking any unauthenticated access.
|
||||
- **Network isolation**: Where possible, place the agent and trusted devices in the **same dedicated VLAN**, isolated from other network devices.
|
||||
- **Stay updated**: Continue to follow DeerFlow's security feature updates.
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, workflow, and guidelines.
|
||||
|
||||
Regression coverage includes Docker sandbox mode detection and provisioner kubeconfig-path handling tests in `backend/tests/`.
|
||||
Backend blocking-IO diagnostics are available from the repository root with
|
||||
`make detect-blocking-io`: it statically scans backend business code for
|
||||
blocking IO that may run on the backend event loop, prints a concise summary,
|
||||
and writes complete JSON findings to `.deer-flow/blocking-io-findings.json`.
|
||||
The JSON includes compact review records with `priority`, `location`,
|
||||
`blocking_call`, `event_loop_exposure`, `reason`, and `code`.
|
||||
Gateway artifact serving now forces active web content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) to download as attachments instead of inline rendering, reducing XSS risk for generated artifacts.
|
||||
|
||||
## License
|
||||
|
||||
This project is open source and available under the [MIT License](./LICENSE).
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
DeerFlow is built upon the incredible work of the open-source community. We are deeply grateful to all the projects and contributors whose efforts have made DeerFlow possible. Truly, we stand on the shoulders of giants.
|
||||
|
||||
We would like to extend our sincere appreciation to the following projects for their invaluable contributions:
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**: Their exceptional framework powers our LLM interactions and chains, enabling seamless integration and functionality.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**: Their innovative approach to multi-agent orchestration has been instrumental in enabling DeerFlow's sophisticated workflows.
|
||||
|
||||
These projects exemplify the transformative power of open-source collaboration, and we are proud to build upon their foundations.
|
||||
|
||||
### Key Contributors
|
||||
|
||||
A heartfelt thank you goes out to the core authors of `DeerFlow`, whose vision, passion, and dedication have brought this project to life:
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
Your unwavering commitment and expertise have been the driving force behind DeerFlow's success. We are honored to have you at the helm of this journey.
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目：`bytedance/deer-flow`
|
||||
- 原始仓库：https://github.com/bytedance/deer-flow
|
||||
- 导入方式：上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
|
||||
+772
@@ -0,0 +1,772 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | Français | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> Le 28 février 2026, DeerFlow a décroché la 🏆 1re place sur GitHub Trending suite au lancement de la version 2. Un immense merci à notre incroyable communauté — c'est grâce à vous ! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) est un **super agent harness** open source qui orchestre des **sub-agents**, de la **mémoire** et des **sandboxes** pour accomplir pratiquement n'importe quelle tâche — le tout propulsé par des **skills extensibles**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 est une réécriture complète.** Il ne partage aucun code avec la v1. Si vous cherchez le framework Deep Research original, il est maintenu sur la [branche `1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x) — les contributions y sont toujours les bienvenues. Le développement actif a migré vers la 2.0.
|
||||
|
||||
## Site officiel
|
||||
|
||||
Découvrez-en plus et regardez des **démos réelles** sur notre [**site officiel**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan de ByteDance Volcengine
|
||||
|
||||
- Nous recommandons fortement d'utiliser Doubao-Seed-2.0-Code, DeepSeek v3.2 et Kimi 2.5 pour exécuter DeerFlow
|
||||
- [En savoir plus](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Développeurs en Chine continentale, cliquez ici](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow intègre désormais le toolkit de recherche et de crawling intelligent développé par BytePlus — [InfoQuest (essai gratuit en ligne)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Table des matières
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Site officiel](#site-officiel)
|
||||
- [Coding Plan de ByteDance Volcengine](#coding-plan-de-bytedance-volcengine)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Table des matières](#table-des-matières)
|
||||
- [Installation en une phrase pour un coding agent](#installation-en-une-phrase-pour-un-coding-agent)
|
||||
- [Démarrage rapide](#démarrage-rapide)
|
||||
- [Configuration](#configuration)
|
||||
- [Lancer l'application](#lancer-lapplication)
|
||||
- [Option 1 : Docker (recommandé)](#option-1--docker-recommandé)
|
||||
- [Option 2 : Développement local](#option-2--développement-local)
|
||||
- [Avancé](#avancé)
|
||||
- [Mode Sandbox](#mode-sandbox)
|
||||
- [Serveur MCP](#serveur-mcp)
|
||||
- [Canaux de messagerie](#canaux-de-messagerie)
|
||||
- [Traçage LangSmith](#traçage-langsmith)
|
||||
- [Traçage Langfuse](#traçage-langfuse)
|
||||
- [Utiliser les deux fournisseurs](#utiliser-les-deux-fournisseurs)
|
||||
- [Du Deep Research au Super Agent Harness](#du-deep-research-au-super-agent-harness)
|
||||
- [Fonctionnalités principales](#fonctionnalités-principales)
|
||||
- [Skills et outils](#skills-et-outils)
|
||||
- [Intégration Claude Code](#intégration-claude-code)
|
||||
- [Objectifs de session (Session Goals)](#objectifs-de-session-session-goals)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox et système de fichiers](#sandbox-et-système-de-fichiers)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Mémoire à long terme](#mémoire-à-long-terme)
|
||||
- [Modèles recommandés](#modèles-recommandés)
|
||||
- [Client Python intégré](#client-python-intégré)
|
||||
- [Tâches planifiées (Scheduled Tasks)](#tâches-planifiées-scheduled-tasks)
|
||||
- [Atelier terminal (TUI)](#atelier-terminal-tui)
|
||||
- [Documentation](#documentation)
|
||||
- [⚠️ Avertissement de sécurité](#️-avertissement-de-sécurité)
|
||||
- [Contribuer](#contribuer)
|
||||
- [Licence](#licence)
|
||||
- [Remerciements](#remerciements)
|
||||
- [Contributeurs principaux](#contributeurs-principaux)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## Installation en une phrase pour un coding agent
|
||||
|
||||
Si vous utilisez Claude Code, Codex, Cursor, Windsurf ou un autre coding agent, vous pouvez simplement lui envoyer cette phrase :
|
||||
|
||||
```text
|
||||
Aide-moi à cloner DeerFlow si nécessaire, puis à initialiser son environnement de développement local en suivant https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Ce prompt est destiné aux coding agents. Il leur demande de cloner le dépôt si nécessaire, de privilégier Docker quand il est disponible, puis de s'arrêter avec la commande exacte pour lancer DeerFlow et la liste des configurations encore manquantes.
|
||||
|
||||
## Démarrage rapide
|
||||
|
||||
### Configuration
|
||||
|
||||
1. **Cloner le dépôt DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Lancer l'assistant de configuration (recommandé)**
|
||||
|
||||
Depuis le répertoire racine du projet (`deer-flow/`), exécutez :
|
||||
|
||||
```bash
|
||||
make setup
|
||||
```
|
||||
|
||||
Cette commande lance un assistant interactif qui vous guide dans le choix d'un fournisseur LLM, d'une recherche web optionnelle et des préférences d'exécution/sécurité (mode sandbox, accès bash, outils d'écriture de fichiers). Il génère un `config.yaml` minimal et écrit vos clés dans `.env`. Comptez environ 2 minutes.
|
||||
|
||||
Exécutez `make doctor` à tout moment pour vérifier votre configuration et obtenir des pistes de correction concrètes.
|
||||
Si vous ouvrez une issue GitHub à propos d'un problème de configuration ou d'exécution en local, exécutez
|
||||
`make support-bundle`. La commande affiche les prochaines étapes pour le rapporteur, écrit un fichier
|
||||
`*-issue-summary.md` à coller dans l'issue, un fichier `*-issue-draft.md` destiné au dépôt d'issue
|
||||
assisté par IA, ainsi qu'un zip de preuves optionnel sous
|
||||
`.deer-flow/support-bundles/`. Si un assistant IA dépose l'issue, partez du brouillon et remplacez
|
||||
chaque placeholder REQUIRED au lieu d'inventer les informations manquantes. N'attachez le zip que si
|
||||
un mainteneur le demande, ou si le résumé seul ne suffit pas. Les mainteneurs et les outils de triage
|
||||
IA peuvent commencer par `triage.json` ; le bundle ne contient que des diagnostics expurgés et des
|
||||
manifestes de fichiers, et n'inclut ni `.env`, ni les messages bruts des conversations, ni le contenu
|
||||
des fichiers de l'utilisateur.
|
||||
|
||||
> **Configuration avancée / manuelle** : si vous préférez éditer `config.yaml` directement, exécutez plutôt `make config` pour copier le template complet. Voir `config.example.yaml` pour la référence complète, y compris les providers basés sur un CLI (Codex CLI, Claude Code OAuth), OpenRouter, l'API Responses, et plus encore.
|
||||
|
||||
<details>
|
||||
<summary>Exemples de configuration manuelle des modèles</summary>
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
```
|
||||
|
||||
OpenRouter et les passerelles compatibles OpenAI similaires doivent être configurés avec `langchain_openai:ChatOpenAI` et `base_url`. Si vous préférez utiliser un nom de variable d'environnement propre au fournisseur, pointez `api_key` vers cette variable explicitement (par exemple `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
Pour router les modèles OpenAI via `/v1/responses`, continuez d'utiliser `langchain_openai:ChatOpenAI` et définissez `use_responses_api: true` avec `output_version: responses/v1`.
|
||||
|
||||
Pour vLLM 0.19.0, utilisez `deerflow.models.vllm_provider:VllmChatModel`. Pour les modèles de raisonnement de type Qwen, DeerFlow active le raisonnement via `extra_body.chat_template_kwargs.enable_thinking` et préserve le champ non standard `reasoning` de vLLM au fil des conversations multi-tours avec appels d'outils. Les anciennes configurations `thinking` sont normalisées automatiquement pour assurer la rétrocompatibilité. Les modèles de raisonnement peuvent aussi exiger que le serveur soit démarré avec `--reasoning-parser ...`. Si votre déploiement vLLM local accepte n'importe quelle clé API non vide, vous pouvez tout de même définir `VLLM_API_KEY` avec une valeur factice.
|
||||
|
||||
Exemples de providers basés sur un CLI :
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI lit `~/.codex/auth.json`
|
||||
- Claude Code accepte `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_CREDENTIALS_PATH`, ou `~/.claude/.credentials.json`
|
||||
- Les entrées d'agents ACP sont distinctes des providers de modèles — si vous configurez `acp_agents.codex`, pointez-le vers un adaptateur Codex ACP tel que `npx -y @zed-industries/codex-acp`
|
||||
- Sur macOS, exportez l'auth Claude Code explicitement si nécessaire :
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
Les clés API peuvent aussi être définies manuellement dans `.env` (recommandé) ou exportées dans votre shell :
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Lancer l'application
|
||||
|
||||
#### Option 1 : Docker (recommandé)
|
||||
|
||||
**Développement** (hot-reload, montage des sources) :
|
||||
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` ne lance `provisioner` que si `config.yaml` utilise le mode provisioner (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` avec `provisioner_url`).
|
||||
Les processus backend récupèrent automatiquement les changements dans `config.yaml` au prochain accès à la configuration, donc les mises à jour de métadonnées des modèles ne nécessitent pas de redémarrage manuel en développement.
|
||||
|
||||
> [!TIP]
|
||||
> Sous Linux, si les commandes Docker échouent avec `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, ajoutez votre utilisateur au groupe `docker` et reconnectez-vous avant de réessayer. Voir [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) pour la solution complète.
|
||||
|
||||
**Production** (build des images en local, montage de la config et des données) :
|
||||
|
||||
```bash
|
||||
make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Le runtime d'agent s'exécute actuellement dans la Gateway. nginx réécrit `/api/langgraph/*` vers l'API compatible LangGraph servie par la Gateway.
|
||||
|
||||
Accès : http://localhost:2026
|
||||
|
||||
Voir [CONTRIBUTING.md](CONTRIBUTING.md) pour le guide complet de développement avec Docker.
|
||||
|
||||
#### Option 2 : Développement local
|
||||
|
||||
Si vous préférez lancer les services en local :
|
||||
|
||||
Prérequis : complétez d'abord les étapes de « Configuration » ci-dessus (`make setup`). `make dev` nécessite un fichier `config.yaml` valide à la racine du projet. Définissez `DEER_FLOW_PROJECT_ROOT` pour indiquer explicitement cette racine, ou `DEER_FLOW_CONFIG_PATH` pour pointer vers un fichier de configuration précis. L'état d'exécution est écrit par défaut dans `.deer-flow` sous la racine du projet et peut être déplacé avec `DEER_FLOW_HOME` ; les skills sont lus par défaut depuis `skills/` sous la racine du projet et peuvent être déplacés avec `DEER_FLOW_SKILLS_PATH`. Exécutez `make doctor` pour vérifier votre configuration avant de démarrer.
|
||||
Sous Windows, exécutez le flux de développement local depuis Git Bash. Les shells natifs `cmd.exe` et PowerShell ne sont pas pris en charge pour les scripts de service basés sur bash, et WSL n'est pas garanti car certains scripts dépendent d'utilitaires de Git for Windows comme `cygpath`.
|
||||
|
||||
1. **Vérifier les prérequis** :
|
||||
```bash
|
||||
make check # Verifies Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Installer les dépendances** :
|
||||
```bash
|
||||
make install # Install backend + frontend dependencies
|
||||
```
|
||||
|
||||
3. **(Optionnel) Pré-télécharger l'image sandbox** :
|
||||
```bash
|
||||
# Recommended if using Docker/Container-based sandbox
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Démarrer les services** :
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Accès** : http://localhost:2026
|
||||
|
||||
### Avancé
|
||||
#### Mode Sandbox
|
||||
|
||||
DeerFlow supporte plusieurs modes d'exécution sandbox :
|
||||
- **Exécution locale** (exécute le code sandbox directement sur la machine hôte)
|
||||
- **Exécution Docker** (exécute le code sandbox dans des conteneurs Docker isolés)
|
||||
- **Exécution Docker avec Kubernetes** (exécute le code sandbox dans des pods Kubernetes via le service provisioner)
|
||||
|
||||
En développement Docker, le démarrage des services suit le mode sandbox défini dans `config.yaml`. En mode Local/Docker, `provisioner` n'est pas démarré.
|
||||
|
||||
Voir le [Guide de configuration Sandbox](backend/docs/CONFIGURATION.md#sandbox) pour configurer le mode de votre choix.
|
||||
|
||||
#### Serveur MCP
|
||||
|
||||
DeerFlow supporte des serveurs MCP et des skills configurables pour étendre ses capacités.
|
||||
Pour les serveurs MCP HTTP/SSE, les flux de tokens OAuth sont supportés (`client_credentials`, `refresh_token`).
|
||||
Voir le [Guide MCP Server](backend/docs/MCP_SERVER.md) pour les instructions détaillées.
|
||||
|
||||
#### Canaux de messagerie
|
||||
|
||||
DeerFlow peut recevoir des tâches depuis des applications de messagerie. Les canaux démarrent automatiquement une fois configurés — aucune IP publique n'est requise.
|
||||
|
||||
DeerFlow peut aussi exposer des connexions de canaux IM appartenant à l'utilisateur dans l'UI du workspace. Quand `channel_connections` est activé, les utilisateurs connectés peuvent lier Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat ou WeCom depuis la barre latérale / Settings > Channels. Cela réutilise les transports sortants `channels.*` existants, donc aucune IP publique ni URL de callback provider n'est requise. Les messages IM entrants s'exécutent ensuite sous le compte utilisateur DeerFlow connecté. Voir [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md) pour la configuration et les notes de sécurité.
|
||||
|
||||
| Canal | Transport | Difficulté |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API (long-polling) | Facile |
|
||||
| Slack | Socket Mode | Modérée |
|
||||
| Feishu / Lark | WebSocket | Modérée |
|
||||
| WeChat | Tencent iLink (long-polling) | Modérée |
|
||||
| WeCom | WebSocket | Modérée |
|
||||
| DingTalk | Stream Push (WebSocket) | Modérée |
|
||||
|
||||
**Configuration dans `config.yaml` :**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# Optional: global session defaults for all mobile channels
|
||||
session:
|
||||
assistant_id: lead_agent
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
# Optional: per-channel / per-user session settings
|
||||
session:
|
||||
assistant_id: mobile_agent
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip_agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # optionnel : autorise le bootstrap QR à la première utilisation quand bot_token est absent
|
||||
allowed_users: [] # vide = tout le monde autorisé
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId depuis DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret depuis DingTalk Open Platform
|
||||
allowed_users: [] # vide = tout le monde autorisé
|
||||
card_template_id: "" # Optionnel : ID de modèle AI Card pour l'effet machine à écrire en streaming
|
||||
```
|
||||
|
||||
Définissez les clés API correspondantes dans votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Configuration Telegram**
|
||||
|
||||
1. Ouvrez une conversation avec [@BotFather](https://t.me/BotFather), envoyez `/newbot`, et copiez le token HTTP API.
|
||||
2. Définissez `TELEGRAM_BOT_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Slack**
|
||||
|
||||
1. Créez une Slack App sur [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
|
||||
2. Dans **OAuth & Permissions**, ajoutez les Bot Token Scopes : `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
|
||||
3. Activez le **Socket Mode** → générez un App-Level Token (`xapp-…`) avec le scope `connections:write`.
|
||||
4. Dans **Event Subscriptions**, abonnez-vous aux bot events : `app_mention`, `message.im`.
|
||||
5. Définissez `SLACK_BOT_TOKEN` et `SLACK_APP_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Feishu / Lark**
|
||||
|
||||
1. Créez une application sur [Feishu Open Platform](https://open.feishu.cn/) → activez la capacité **Bot**.
|
||||
2. Ajoutez les permissions : `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
|
||||
3. Dans **Events**, abonnez-vous à `im.message.receive_v1` et sélectionnez le mode **Long Connection**.
|
||||
4. Copiez l'App ID et l'App Secret. Définissez `FEISHU_APP_ID` et `FEISHU_APP_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration WeChat**
|
||||
|
||||
1. Activez le canal `wechat` dans `config.yaml`.
|
||||
2. Soit définissez `WECHAT_BOT_TOKEN` dans `.env`, soit mettez `qrcode_login_enabled: true` pour le bootstrap QR à la première utilisation.
|
||||
3. Quand `bot_token` est absent et que le bootstrap QR est activé, surveillez les logs du backend pour le contenu du QR renvoyé par iLink et complétez le flux de binding.
|
||||
4. Une fois le flux QR réussi, DeerFlow persiste le token acquis sous `state_dir` pour les redémarrages ultérieurs.
|
||||
5. Pour les déploiements Docker Compose, gardez `state_dir` sur un volume persistant afin que le curseur `get_updates_buf` et l'état d'auth sauvegardé survivent aux redémarrages.
|
||||
|
||||
**Configuration WeCom**
|
||||
|
||||
1. Créez un bot sur la plateforme WeCom AI Bot et obtenez le `bot_id` et le `bot_secret`.
|
||||
2. Activez `channels.wecom` dans `config.yaml` et renseignez `bot_id` / `bot_secret`.
|
||||
3. Définissez `WECOM_BOT_ID` et `WECOM_BOT_SECRET` dans `.env`.
|
||||
4. Assurez-vous que les dépendances du backend incluent `wecom-aibot-python-sdk`. Le canal utilise une connexion longue WebSocket et ne nécessite pas d'URL de callback publique.
|
||||
5. L'intégration actuelle prend en charge les messages entrants texte, image et fichier. Les images/fichiers finaux générés par l'agent sont aussi renvoyés dans la conversation WeCom.
|
||||
|
||||
**Configuration DingTalk**
|
||||
|
||||
1. Créez une application sur [DingTalk Open Platform](https://open.dingtalk.com/) et activez la capacité **Robot**.
|
||||
2. Dans la page de configuration du robot, définissez le mode de réception des messages sur **Stream**.
|
||||
3. Copiez le `Client ID` et le `Client Secret`. Définissez `DINGTALK_CLIENT_ID` et `DINGTALK_CLIENT_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
4. *(Optionnel)* Pour activer les réponses en streaming AI Card (effet machine à écrire), créez un modèle **AI Card** sur la [plateforme de cartes DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), puis définissez `card_template_id` dans `config.yaml` avec l'ID du modèle. Vous devez également demander les permissions `Card.Streaming.Write` et `Card.Instance.Write`.
|
||||
|
||||
**Commandes**
|
||||
|
||||
Une fois un canal connecté, vous pouvez interagir avec DeerFlow directement depuis le chat :
|
||||
|
||||
| Commande | Description |
|
||||
|---------|-------------|
|
||||
| `/new` | Démarrer une nouvelle conversation |
|
||||
| `/status` | Afficher les infos du thread en cours |
|
||||
| `/models` | Lister les modèles disponibles |
|
||||
| `/memory` | Consulter la mémoire |
|
||||
| `/help` | Afficher l'aide |
|
||||
|
||||
> Les messages sans préfixe de commande sont traités comme du chat classique — DeerFlow crée un thread et répond de manière conversationnelle.
|
||||
|
||||
#### Traçage LangSmith
|
||||
|
||||
DeerFlow intègre nativement [LangSmith](https://smith.langchain.com) pour l'observabilité. Une fois activé, tous les appels LLM, les exécutions d'agents et les exécutions d'outils sont tracés et visibles dans le tableau de bord LangSmith.
|
||||
|
||||
Ajoutez les lignes suivantes à votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
#### Traçage Langfuse
|
||||
|
||||
DeerFlow prend également en charge l'observabilité via [Langfuse](https://langfuse.com) pour les exécutions compatibles LangChain.
|
||||
|
||||
Ajoutez les lignes suivantes à votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
Si vous utilisez une instance Langfuse auto-hébergée, définissez `LANGFUSE_BASE_URL` sur l'URL de votre déploiement.
|
||||
|
||||
**Champs de corrélation des traces.** Chaque exécution d'agent est annotée avec les attributs de trace réservés de Langfuse afin que les pages Sessions et Users se remplissent automatiquement :
|
||||
|
||||
- `session_id` = `thread_id` de LangGraph — regroupe toutes les traces d'une même conversation
|
||||
- `user_id` = utilisateur effectif issu de `get_effective_user_id()` (revient à `default` en mode sans authentification)
|
||||
- `trace_name` = assistant id (par défaut `lead-agent`)
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]` (omis lorsqu'ils ne sont pas définis)
|
||||
- `metadata.deerflow_trace_id` = id de corrélation de requête DeerFlow, identique à `X-Trace-Id` lorsque la corrélation de trace des requêtes est activée
|
||||
|
||||
Ces champs sont injectés dans `RunnableConfig.metadata` à la racine de l'invocation du graphe, à la fois pour le chemin gateway (`runtime/runs/worker.py::run_agent`) et le chemin embarqué (`client.py::DeerFlowClient.stream`), de sorte que tout callback compatible LangChain puisse les lire. Définissez `DEER_FLOW_ENV` (ou `ENVIRONMENT`) pour étiqueter les traces par environnement de déploiement.
|
||||
|
||||
#### Utiliser les deux fournisseurs
|
||||
|
||||
Si LangSmith et Langfuse sont tous deux activés, DeerFlow attache les deux callbacks de traçage et rapporte la même activité de modèle aux deux systèmes.
|
||||
|
||||
Si un fournisseur est explicitement activé mais qu'il manque les identifiants requis, ou si son callback échoue à s'initialiser, DeerFlow échoue immédiatement (fail fast) lors de l'initialisation du traçage à la création du modèle, et le message d'erreur indique le fournisseur à l'origine de l'échec.
|
||||
|
||||
Pour les déploiements Docker, le traçage est désactivé par défaut. Définissez `LANGSMITH_TRACING=true` et `LANGSMITH_API_KEY` dans votre `.env` pour l'activer.
|
||||
|
||||
## Du Deep Research au Super Agent Harness
|
||||
|
||||
DeerFlow a démarré comme un framework de Deep Research — et la communauté s'en est emparée. Depuis le lancement, les développeurs l'ont poussé bien au-delà de la recherche : construction de pipelines de données, génération de présentations, mise en place de dashboards, automatisation de workflows de contenu. Des usages qu'on n'avait jamais anticipés.
|
||||
|
||||
Ça nous a révélé quelque chose d'important : DeerFlow n'était pas qu'un simple outil de recherche. C'était un **harness** — un runtime qui donne aux agents l'infrastructure nécessaire pour vraiment accomplir du travail.
|
||||
|
||||
On l'a donc reconstruit de zéro.
|
||||
|
||||
DeerFlow 2.0 n'est plus un framework à assembler soi-même. C'est un super agent harness — clé en main et entièrement extensible. Construit sur LangGraph et LangChain, il embarque tout ce dont un agent a besoin out of the box : un système de fichiers, de la mémoire, des skills, une exécution sandboxée, et la capacité de planifier et de lancer des sub-agents pour les tâches complexes et multi-étapes.
|
||||
|
||||
Utilisez-le tel quel. Ou démontez-le et faites-en le vôtre.
|
||||
|
||||
## Fonctionnalités principales
|
||||
|
||||
### Skills et outils
|
||||
|
||||
Les skills sont ce qui permet à DeerFlow de faire *pratiquement n'importe quoi*.
|
||||
|
||||
Un Agent Skill standard est un module de capacité structuré — un fichier Markdown qui définit un workflow, des bonnes pratiques et des références vers des ressources associées. DeerFlow est livré avec des skills intégrés pour la recherche, la génération de rapports, la création de présentations, les pages web, la génération d'images et de vidéos, et bien plus. Mais la vraie force réside dans l'extensibilité : ajoutez vos propres skills, remplacez ceux fournis, ou combinez-les en workflows composites.
|
||||
|
||||
Les skills sont chargés progressivement — uniquement quand la tâche le nécessite, pas tous en même temps. Ça permet de garder la fenêtre de contexte légère et de bien fonctionner même avec des modèles sensibles au nombre de tokens.
|
||||
|
||||
Quand vous installez des archives `.skill` via le Gateway, DeerFlow accepte les métadonnées frontmatter optionnelles standard comme `version`, `author` et `compatibility`, plutôt que de rejeter des skills externes par ailleurs valides.
|
||||
|
||||
Les outils suivent la même philosophie. DeerFlow est livré avec un ensemble d'outils de base — recherche web, fetch de pages web, opérations sur les fichiers, exécution bash — et supporte les outils custom via des serveurs MCP et des fonctions Python. Remplacez n'importe quoi. Ajoutez n'importe quoi.
|
||||
|
||||
Les suggestions de suivi générées par le Gateway normalisent désormais aussi bien la sortie texte brut du modèle que le contenu riche au format bloc/liste avant de parser la réponse en tableau JSON, de sorte que les wrappers de contenu propres à chaque provider ne suppriment plus silencieusement les suggestions.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← yours
|
||||
```
|
||||
|
||||
#### Intégration Claude Code
|
||||
|
||||
Le skill `claude-to-deerflow` vous permet d'interagir avec une instance DeerFlow en cours d'exécution directement depuis [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Envoyez des tâches de recherche, vérifiez le statut, gérez les threads — le tout sans quitter le terminal.
|
||||
|
||||
**Installer le skill** :
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
Assurez-vous ensuite que DeerFlow tourne (par défaut sur `http://localhost:2026`) et utilisez la commande `/claude-to-deerflow` dans Claude Code.
|
||||
|
||||
**Ce que vous pouvez faire** :
|
||||
- Envoyer des messages à DeerFlow et recevoir des réponses en streaming
|
||||
- Choisir le mode d'exécution : flash (rapide), standard, pro (planification), ultra (sub-agents)
|
||||
- Vérifier la santé de DeerFlow, lister les modèles/skills/agents
|
||||
- Gérer les threads et l'historique des conversations
|
||||
- Upload des fichiers pour analyse
|
||||
|
||||
**Variables d'environnement** (optionnel, pour des endpoints custom) :
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
Voir [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) pour la référence API complète.
|
||||
|
||||
### Objectifs de session (Session Goals)
|
||||
|
||||
Utilisez `/goal <condition de complétion>` pour attacher une condition de complétion active au thread courant. Le goal est un état de portée thread, pas une activation de skill — il reste actif entre les tours jusqu'à ce que DeerFlow détermine qu'il a été satisfait, ou jusqu'à ce que vous le supprimiez.
|
||||
|
||||
Commandes prises en charge :
|
||||
|
||||
```text
|
||||
/goal finish the implementation and make all tests pass
|
||||
/goal # afficher le goal actif
|
||||
/goal clear # le supprimer
|
||||
```
|
||||
|
||||
Après chaque exécution menée par la Gateway, DeerFlow évalue la conversation visible par rapport au goal actif à l'aide d'un modèle évaluateur non-thinking. L'évaluateur doit renvoyer un blocker typé (`missing_evidence`, `needs_user_input`, `run_failed`, `external_wait` ou `goal_not_met_yet`) accompagné de preuves visibles. DeerFlow n'injecte une hidden continuation que si le dernier tour assistant est durablement checkpointé, que le blocker est `goal_not_met_yet`, que le thread n'a pas changé durant l'évaluation et que le disjoncteur de non-progression n'a pas déclenché. Le plafond de sécurité est de 8 hidden continuations par défaut, et les évaluations identiques de non-progression s'arrêtent après 2 tentatives répétées. `/goal clear` ainsi que toute nouvelle saisie utilisateur ont priorité sur les continuations en file d'attente. Lorsque le goal est satisfait, DeerFlow le supprime automatiquement et publie l'état mis à jour du thread.
|
||||
|
||||
Le Web UI affiche le goal actif au-dessus de la zone de saisie. La même commande est disponible depuis le TUI et les canaux IM pris en charge. Dans le Web UI et les canaux IM pris en charge, définir `/goal <condition de complétion>` lance aussi une exécution avec la condition comme tâche ; les commandes de statut et de suppression ne gèrent que l'état du goal lui-même.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Les tâches complexes tiennent rarement en une seule passe. DeerFlow les décompose.
|
||||
|
||||
L'agent principal peut lancer des sub-agents à la volée — chacun avec son propre contexte délimité, ses outils et ses conditions d'arrêt. Les sub-agents s'exécutent en parallèle quand c'est possible, remontent des résultats structurés, et l'agent principal synthétise le tout en une sortie cohérente.
|
||||
|
||||
C'est comme ça que DeerFlow gère les tâches qui prennent de quelques minutes à plusieurs heures : une tâche de recherche peut se déployer en une dizaine de sub-agents, chacun explorant un angle différent, puis converger vers un seul rapport — ou un site web — ou un jeu de slides avec des visuels générés. Un seul harness, de nombreuses mains.
|
||||
|
||||
### Sandbox et système de fichiers
|
||||
|
||||
DeerFlow ne se contente pas de *parler* de faire les choses. Il dispose de son propre ordinateur.
|
||||
|
||||
Chaque tâche s'exécute dans un conteneur Docker isolé avec un système de fichiers complet — skills, workspace, uploads, outputs. L'agent lit, écrit et édite des fichiers. Il exécute des commandes bash et du code. Il visualise des images. Le tout sandboxé, le tout auditable, zéro contamination entre les sessions.
|
||||
|
||||
C'est la différence entre un chatbot avec accès à des outils et un agent doté d'un véritable environnement d'exécution.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← your files
|
||||
├── workspace/ ← agents' working directory
|
||||
└── outputs/ ← final deliverables
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Contexte isolé des Sub-Agents** : chaque sub-agent s'exécute dans son propre contexte isolé. Il ne peut voir ni le contexte de l'agent principal, ni celui des autres sub-agents. L'objectif est de garantir que chaque sub-agent reste concentré sur sa tâche sans être parasité par des informations non pertinentes.
|
||||
|
||||
**Résumé** : au sein d'une session, DeerFlow gère le contexte de manière agressive — en résumant les sous-tâches terminées, en déchargeant les résultats intermédiaires vers le système de fichiers, en compressant ce qui n'est plus immédiatement pertinent. Ça lui permet de rester efficace sur des tâches longues et multi-étapes sans faire exploser la fenêtre de contexte.
|
||||
|
||||
### Mémoire à long terme
|
||||
|
||||
La plupart des agents oublient tout dès qu'une conversation se termine. DeerFlow, lui, se souvient.
|
||||
|
||||
D'une session à l'autre, DeerFlow construit une mémoire persistante de votre profil, de vos préférences et de vos connaissances accumulées. Plus vous l'utilisez, mieux il vous connaît — votre style d'écriture, votre stack technique, vos workflows récurrents. La mémoire est stockée localement et reste sous votre contrôle.
|
||||
|
||||
Les mises à jour de la mémoire ignorent désormais les entrées de faits en double au moment de l'application, de sorte que les préférences et le contexte répétés ne s'accumulent plus indéfiniment entre les sessions.
|
||||
|
||||
## Modèles recommandés
|
||||
|
||||
DeerFlow est agnostique en termes de modèle — il fonctionne avec n'importe quel LLM implémentant l'API compatible OpenAI. Cela dit, il offre de meilleures performances avec des modèles qui supportent :
|
||||
|
||||
- **De longues fenêtres de contexte** (100k+ tokens) pour la recherche approfondie et les tâches multi-étapes
|
||||
- **Des capacités de raisonnement** pour la planification adaptative et la décomposition de tâches complexes
|
||||
- **Des entrées multimodales** pour la compréhension d'images et de vidéos
|
||||
- **Un usage fiable des outils (tool use)** pour des appels de fonctions et des sorties structurées fiables
|
||||
|
||||
## Client Python intégré
|
||||
|
||||
DeerFlow peut être utilisé comme bibliothèque Python intégrée sans lancer l'ensemble des services HTTP. Le `DeerFlowClient` fournit un accès direct in-process à toutes les capacités d'agent et de Gateway, en retournant les mêmes schémas de réponse que l'API HTTP Gateway. Le HTTP Gateway expose également `DELETE /api/threads/{thread_id}` pour supprimer les données de thread locales gérées par DeerFlow après la suppression du thread LangGraph :
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
client.set_goal("thread-1", "finish the implementation and make all tests pass")
|
||||
client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
|
||||
client.clear_goal("thread-1")
|
||||
```
|
||||
|
||||
Toutes les méthodes retournant des dicts sont validées en CI contre les modèles de réponse Pydantic du Gateway (`TestGatewayConformance`), garantissant que le client intégré reste synchronisé avec les schémas de l'API HTTP. Voir `backend/packages/harness/deerflow/client.py` pour la documentation API complète.
|
||||
|
||||
## Tâches planifiées (Scheduled Tasks)
|
||||
|
||||
DeerFlow inclut désormais un MVP de tâches planifiées (scheduled-task) de premier niveau dans le workspace.
|
||||
|
||||
Capacités actuelles du MVP :
|
||||
|
||||
- Gérer les tâches depuis `/workspace/scheduled-tasks`
|
||||
- Choisir si chaque tâche planifiée réutilise un thread ou crée un nouveau thread à chaque exécution
|
||||
- Prendre en charge les planifications `once` et `cron`
|
||||
- Exécuter les tâches planifiées en arrière-plan comme des exécutions DeerFlow non interactives (`ask_clarification` n'y est pas exposé)
|
||||
- Utiliser le comportement de chevauchement `skip` pour les exécutions cron dues qui entrent en collision avec une exécution active sur le même thread réutilisé
|
||||
- Mettre en pause, reprendre, déclencher, inspecter l'historique et supprimer les tâches
|
||||
- Exécuter le travail planifié via le cycle de vie d'exécution normal de DeerFlow
|
||||
|
||||
Limites actuelles du MVP :
|
||||
|
||||
- Pas encore d'outil `schedule_task` créable depuis la conversation
|
||||
- Pas de tâches de notification en texte seul
|
||||
- Pas de cibles de dispatch canal ou GitHub
|
||||
- Pas de type de planification `interval` dans cette première version
|
||||
|
||||
Activez le polling en arrière-plan avec `config.yaml -> scheduler.enabled`. Le déclenchement manuel utilise la même ressource scheduled-task et le même chemin d'exécution.
|
||||
|
||||
## Atelier terminal (TUI)
|
||||
|
||||
`deerflow` est un atelier natif terminal pour ceux qui vivent dans le shell. Il s'exécute de manière **intégrée** sur `DeerFlowClient` — pas besoin de Gateway, de frontend, de nginx ou de Docker — tout en honorant les mêmes `config.yaml`, checkpointer, skills, mémoire, MCP et sandbox que le reste de DeerFlow.
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # dépendance optionnelle 'textual'
|
||||
|
||||
deerflow # lancer l'UI terminal (TTY requis)
|
||||
deerflow --continue # reprendre le thread le plus récent
|
||||
deerflow --resume THREAD # reprendre un thread par id
|
||||
deerflow --print "summarize this repo" # réponse one-shot headless vers stdout
|
||||
deerflow --json "hello" # StreamEvents séparés par saut de ligne en mode headless
|
||||
```
|
||||
|
||||
Une surface de chat pilotée au clavier avec un transcript en streaming (réponses rendues en Markdown), des cartes d'activité d'outils compactes, une palette de commandes slash `/`, la gestion des goal via `/goal`, des sélecteurs `/model` et `/threads`, l'historique de saisie, et l'interruption via `Esc` / `Ctrl+C`. Les sessions ouvertes dans le TUI apparaissent aussi dans la barre latérale du Web UI — elles écrivent dans le magasin de threads partagé sous l'utilisateur local par défaut, donc le terminal et le web restent synchronisés **sans lancer la Gateway**.
|
||||
|
||||
Voir [backend/docs/TUI.md](backend/docs/TUI.md) pour le guide complet.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Guide de contribution](CONTRIBUTING.md) - Mise en place de l'environnement de développement et workflow
|
||||
- [Guide de configuration](backend/docs/CONFIGURATION.md) - Instructions d'installation et de configuration
|
||||
- [Vue d'ensemble de l'architecture](backend/CLAUDE.md) - Détails de l'architecture technique
|
||||
- [Architecture backend](backend/README.md) - Architecture backend et référence API
|
||||
|
||||
## ⚠️ Avertissement de sécurité
|
||||
|
||||
### Un déploiement inapproprié peut introduire des risques de sécurité
|
||||
|
||||
DeerFlow dispose de capacités clés à hauts privilèges, notamment **l'exécution de commandes système, les opérations sur les ressources et l'invocation de logique métier**. Il est conçu par défaut pour être **déployé dans un environnement local de confiance (accessible uniquement via l'interface de loopback 127.0.0.1)**. Si vous déployez l'agent dans des environnements non fiables — tels que des réseaux LAN, des serveurs cloud publics ou d'autres environnements accessibles depuis plusieurs terminaux — sans mesures de sécurité strictes, cela peut introduire des risques, notamment :
|
||||
|
||||
- **Invocation non autorisée** : les fonctionnalités de l'agent pourraient être découvertes par des tiers non autorisés ou des scanners malveillants, déclenchant des requêtes non autorisées en masse qui exécutent des opérations à haut risque (commandes système, lecture/écriture de fichiers), pouvant causer de graves conséquences.
|
||||
- **Risques juridiques et de conformité** : si l'agent est utilisé illégalement pour mener des cyberattaques, du vol de données ou d'autres activités illicites, cela peut entraîner des responsabilités juridiques et des risques de conformité.
|
||||
|
||||
### Recommandations de sécurité
|
||||
|
||||
**Note : nous recommandons fortement de déployer DeerFlow dans un environnement réseau local de confiance.** Si vous avez besoin d'un déploiement multi-appareils ou multi-réseaux, vous devez mettre en place des mesures de sécurité strictes, par exemple :
|
||||
|
||||
- **Liste blanche d'IP** : utilisez `iptables`, ou déployez des pare-feux matériels / commutateurs avec ACL, pour **configurer des règles de liste blanche d'IP** et refuser l'accès à toutes les autres adresses IP.
|
||||
- **Passerelle d'authentification** : configurez un proxy inverse (ex. nginx) et **activez une authentification forte en amont**, bloquant tout accès non authentifié.
|
||||
- **Isolation réseau** : si possible, placez l'agent et les appareils de confiance dans le **même VLAN dédié**, isolé des autres équipements réseau.
|
||||
- **Restez informé** : continuez à suivre les mises à jour de sécurité du projet DeerFlow.
|
||||
|
||||
## Contribuer
|
||||
|
||||
Les contributions sont les bienvenues ! Consultez [CONTRIBUTING.md](CONTRIBUTING.md) pour la mise en place de l'environnement de développement, le workflow et les conventions.
|
||||
|
||||
La couverture de tests de régression inclut la détection du mode sandbox Docker et les tests de gestion du kubeconfig-path du provisioner dans `backend/tests/`.
|
||||
|
||||
## Licence
|
||||
|
||||
Ce projet est open source et disponible sous la [Licence MIT](./LICENSE).
|
||||
|
||||
## Remerciements
|
||||
|
||||
DeerFlow est construit sur le travail remarquable de la communauté open source. Nous sommes profondément reconnaissants envers tous les projets et contributeurs dont les efforts ont rendu DeerFlow possible. Nous nous tenons véritablement sur les épaules de géants.
|
||||
|
||||
Nous tenons à exprimer notre sincère gratitude aux projets suivants pour leurs contributions inestimables :
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** : leur excellent framework propulse nos interactions LLM et nos chaînes, permettant une intégration et des fonctionnalités fluides.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** : leur approche innovante de l'orchestration multi-agents a été déterminante pour les workflows sophistiqués de DeerFlow.
|
||||
|
||||
Ces projets illustrent le pouvoir transformateur de la collaboration open source, et nous sommes fiers de bâtir sur leurs fondations.
|
||||
|
||||
### Contributeurs principaux
|
||||
|
||||
Un grand merci aux auteurs principaux de `DeerFlow`, dont la vision, la passion et le dévouement ont donné vie à ce projet :
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
Votre engagement sans faille et votre expertise sont le moteur du succès de DeerFlow. Nous sommes honorés de vous avoir à la barre de cette aventure.
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+759
@@ -0,0 +1,759 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | 日本語 | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> 2026年2月28日、バージョン2のリリースに伴い、DeerFlowはGitHub Trendingで🏆 第1位を獲得しました。素晴らしいコミュニティの皆さん、ありがとうございます！💪🔥
|
||||
|
||||
DeerFlow（**D**eep **E**xploration and **E**fficient **R**esearch **Flow**）は、**サブエージェント**、**メモリ**、**サンドボックス**を統合し、**拡張可能なスキル**によってあらゆるタスクを実行できるオープンソースの**スーパーエージェントハーネス**です。
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0はゼロからの完全な書き直しです。** v1とコードを共有していません。オリジナルのDeep Researchフレームワークをお探しの場合は、[`1.x`ブランチ](https://github.com/bytedance/deer-flow/tree/main-1.x)で引き続きメンテナンスされています。現在の開発は2.0に移行しています。
|
||||
|
||||
## 公式ウェブサイト
|
||||
|
||||
**実際のデモ**は[**公式ウェブサイト**](https://deerflow.tech)でご覧いただけます。
|
||||
|
||||
## ByteDance Volcengine のコーディングプラン
|
||||
|
||||
- DeerFlowの実行には、Doubao-Seed-2.0-Code、DeepSeek v3.2、Kimi 2.5の使用を強く推奨します
|
||||
- [詳細はこちら](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [中国大陸の開発者はこちらをクリック](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlowは、BytePlusが独自に開発したインテリジェント検索・クローリングツールセット「[InfoQuest（無料オンライン体験対応）](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)」を新たに統合しました。
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## 目次
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [公式ウェブサイト](#公式ウェブサイト)
|
||||
- [ByteDance Volcengine のコーディングプラン](#bytedance-volcengine-のコーディングプラン)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [目次](#目次)
|
||||
- [Coding Agent に一文でセットアップを依頼](#coding-agent-に一文でセットアップを依頼)
|
||||
- [クイックスタート](#クイックスタート)
|
||||
- [設定](#設定)
|
||||
- [アプリケーションの実行](#アプリケーションの実行)
|
||||
- [オプション1: Docker（推奨）](#オプション1-docker推奨)
|
||||
- [オプション2: ローカル開発](#オプション2-ローカル開発)
|
||||
- [詳細設定](#詳細設定)
|
||||
- [サンドボックスモード](#サンドボックスモード)
|
||||
- [MCPサーバー](#mcpサーバー)
|
||||
- [IMチャネル](#imチャネル)
|
||||
- [LangSmithトレーシング](#langsmithトレーシング)
|
||||
- [Langfuseトレーシング](#langfuseトレーシング)
|
||||
- [両方のプロバイダーを使用する](#両方のプロバイダーを使用する)
|
||||
- [Deep Researchからスーパーエージェントハーネスへ](#deep-researchからスーパーエージェントハーネスへ)
|
||||
- [コア機能](#コア機能)
|
||||
- [スキルとツール](#スキルとツール)
|
||||
- [Claude Code連携](#claude-code連携)
|
||||
- [セッションゴール (Session Goals)](#セッションゴール-session-goals)
|
||||
- [サブエージェント](#サブエージェント)
|
||||
- [サンドボックスとファイルシステム](#サンドボックスとファイルシステム)
|
||||
- [コンテキストエンジニアリング](#コンテキストエンジニアリング)
|
||||
- [長期メモリ](#長期メモリ)
|
||||
- [推奨モデル](#推奨モデル)
|
||||
- [組み込みPythonクライアント](#組み込みpythonクライアント)
|
||||
- [スケジュールタスク (Scheduled Tasks)](#スケジュールタスク-scheduled-tasks)
|
||||
- [ターミナルワークベンチ (TUI)](#ターミナルワークベンチ-tui)
|
||||
- [ドキュメント](#ドキュメント)
|
||||
- [⚠️ セキュリティに関する注意](#️-セキュリティに関する注意)
|
||||
- [コントリビュート](#コントリビュート)
|
||||
- [ライセンス](#ライセンス)
|
||||
- [謝辞](#謝辞)
|
||||
- [主要コントリビューター](#主要コントリビューター)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## Coding Agent に一文でセットアップを依頼
|
||||
|
||||
Claude Code、Codex、Cursor、Windsurf などの coding agent を使っているなら、次の一文をそのまま渡せます。
|
||||
|
||||
```text
|
||||
DeerFlow がまだ clone されていなければ先に clone してから、https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md に従ってローカル開発環境を初期化してください
|
||||
```
|
||||
|
||||
このプロンプトは coding agent 向けです。必要なら先にリポジトリを clone し、Docker が使える場合は Docker を優先して初期セットアップを行い、最後に次の起動コマンドと不足している設定項目だけを返します。
|
||||
|
||||
## クイックスタート
|
||||
|
||||
### 設定
|
||||
|
||||
1. **DeerFlowリポジトリをクローン**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **セットアップウィザードの実行（推奨）**
|
||||
|
||||
プロジェクトルートディレクトリ（`deer-flow/`）から以下を実行します：
|
||||
|
||||
```bash
|
||||
make setup
|
||||
```
|
||||
|
||||
対話式ウィザードが起動し、LLMプロバイダーの選択、オプションのWeb検索、そしてサンドボックスモード・bash権限・ファイル書き込みツールなどの実行/安全設定を順に案内します。最小構成の`config.yaml`を生成し、APIキーを`.env`に書き込みます。所要時間は約2分です。
|
||||
|
||||
いつでも`make doctor`を実行して、設定を確認し、具体的な修正ヒントを得られます。
|
||||
ローカルセットアップや実行時の問題についてGitHub issueを起票する場合は、`make support-bundle`を実行してください。このコマンドは報告者向けの次のステップを表示し、issueに貼り付けるための`*-issue-summary.md`ファイルと、AI支援でissueを起票するための`*-issue-draft.md`ファイルを書き出し、オプションで証跡zipを`.deer-flow/support-bundles/`以下に作成します。AIアシスタントがissueを起票する場合は、ドラフトを起点にして、不足している事実を創作するのではなく、すべてのREQUIREDプレースホルダーを置き換えてください。zipは、メンテナーから求められた場合、またはサマリーだけでは不十分な場合にのみ添付してください。メンテナーやAIトリアージツールは`triage.json`から確認を始められます。バンドルに含まれるのはリダクト済みの診断情報とファイルマニフェストのみで、`.env`、生の会話メッセージ、ユーザーファイルの内容は含まれません。
|
||||
|
||||
> **上級者向け / 手動設定**：`config.yaml`を直接編集したい場合は、代わりに`make config`を実行して完全なテンプレートをコピーしてください。CLI連携プロバイダー（Codex CLI、Claude Code OAuth）、OpenRouter、Responses APIなどを含む完全なリファレンスは`config.example.yaml`を参照してください。
|
||||
|
||||
<details>
|
||||
<summary>手動モデル設定の例</summary>
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
```
|
||||
|
||||
OpenRouterやOpenAI互換のゲートウェイは、`langchain_openai:ChatOpenAI`と`base_url`で設定します。プロバイダー固有の環境変数名を使用したい場合は、`api_key`でその変数を明示的に指定してください（例：`api_key: $OPENROUTER_API_KEY`）。
|
||||
|
||||
OpenAIモデルを`/v1/responses`経由でルーティングするには、引き続き`langchain_openai:ChatOpenAI`を使用し、`use_responses_api: true`と`output_version: responses/v1`を設定してください。
|
||||
|
||||
vLLM 0.19.0では`deerflow.models.vllm_provider:VllmChatModel`を使用してください。Qwen系のreasoningモデルでは、DeerFlowは`extra_body.chat_template_kwargs.enable_thinking`でreasoningを切り替え、マルチターンのツールコール会話にわたってvLLM独自の非標準`reasoning`フィールドを保持します。従来の`thinking`設定は後方互換性のため自動的に正規化されます。reasoningモデルはサーバー側で`--reasoning-parser ...`を付けて起動する必要がある場合もあります。ローカルのvLLMデプロイメントが空でない任意のAPIキーを受け付ける場合でも、`VLLM_API_KEY`にはプレースホルダー値を設定しておけます。
|
||||
|
||||
CLI連携プロバイダーの例：
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLIは`~/.codex/auth.json`を読み取ります
|
||||
- Claude Codeは`CLAUDE_CODE_OAUTH_TOKEN`、`ANTHROPIC_AUTH_TOKEN`、`CLAUDE_CODE_CREDENTIALS_PATH`、または`~/.claude/.credentials.json`を受け付けます
|
||||
- ACPエージェントのエントリはモデルプロバイダーとは別物です。`acp_agents.codex`を設定する場合は、`npx -y @zed-industries/codex-acp`のようなCodex ACPアダプターを指定してください
|
||||
- macOSでは、必要に応じてClaude Codeの認証情報を明示的にエクスポートしてください：
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
APIキーは`.env`で手動設定する（推奨）ことも、シェルでエクスポートすることもできます：
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### アプリケーションの実行
|
||||
|
||||
#### オプション1: Docker（推奨）
|
||||
|
||||
**開発環境**（ホットリロード、ソースマウント）：
|
||||
|
||||
```bash
|
||||
make docker-init # サンドボックスイメージをプル（初回またはイメージ更新時のみ）
|
||||
make docker-start # サービスを開始（config.yamlからサンドボックスモードを自動検出）
|
||||
```
|
||||
|
||||
`make docker-start`は、`config.yaml`がプロビジョナーモード（`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider`と`provisioner_url`）を使用している場合にのみ`provisioner`を起動します。
|
||||
|
||||
**本番環境**（ローカルでイメージをビルドし、ランタイム設定とデータをマウント）：
|
||||
|
||||
```bash
|
||||
make up # イメージをビルドして全本番サービスを開始
|
||||
make down # コンテナを停止して削除
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Agentランタイムは現在Gateway内で実行されます。`/api/langgraph/*`はnginxによってGatewayのLangGraph-compatible APIへ書き換えられます。
|
||||
|
||||
アクセス: http://localhost:2026
|
||||
|
||||
詳細なDocker開発ガイドは[CONTRIBUTING.md](CONTRIBUTING.md)をご覧ください。
|
||||
|
||||
#### オプション2: ローカル開発
|
||||
|
||||
サービスをローカルで実行する場合：
|
||||
|
||||
前提条件：上記の「設定」手順を先に完了してください（`make setup`）。`make dev`にはプロジェクトルートに有効な`config.yaml`が必要です。`DEER_FLOW_PROJECT_ROOT`でプロジェクトルートを明示的に指定するか、`DEER_FLOW_CONFIG_PATH`で特定の設定ファイルを指定できます。実行時の状態はデフォルトでプロジェクトルート直下の`.deer-flow`に書き込まれ、`DEER_FLOW_HOME`で移動できます。skillsはデフォルトでプロジェクトルート直下の`skills/`から読み込まれ、`DEER_FLOW_SKILLS_PATH`で移動できます。起動前に`make doctor`を実行して設定を確認してください。
|
||||
Windowsでは、ローカル開発フローはGit Bashから実行してください。bashベースのサービススクリプトはネイティブの`cmd.exe`やPowerShellではサポートされておらず、一部のスクリプトがGit for Windowsの`cygpath`などのユーティリティに依存しているため、WSLでの動作も保証されません。
|
||||
|
||||
1. **前提条件の確認**：
|
||||
```bash
|
||||
make check # Node.js 22+、pnpm、uv、nginxを検証
|
||||
```
|
||||
|
||||
2. **依存関係のインストール**：
|
||||
```bash
|
||||
make install # バックエンド＋フロントエンドの依存関係をインストール
|
||||
```
|
||||
|
||||
3. **（オプション）サンドボックスイメージの事前プル**：
|
||||
```bash
|
||||
# Docker/コンテナベースのサンドボックス使用時に推奨
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **サービスの開始**：
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **アクセス**: http://localhost:2026
|
||||
|
||||
### 詳細設定
|
||||
#### サンドボックスモード
|
||||
|
||||
DeerFlowは複数のサンドボックス実行モードをサポートしています：
|
||||
- **ローカル実行**（ホストマシン上で直接サンドボックスコードを実行）
|
||||
- **Docker実行**（分離されたDockerコンテナ内でサンドボックスコードを実行）
|
||||
- **KubernetesによるDocker実行**（プロビジョナーサービス経由でKubernetesポッドでサンドボックスコードを実行）
|
||||
|
||||
Docker開発では、サービスの起動は`config.yaml`のサンドボックスモードに従います。ローカル/Dockerモードでは`provisioner`は起動されません。
|
||||
|
||||
お好みのモードの設定については[サンドボックス設定ガイド](backend/docs/CONFIGURATION.md#sandbox)をご覧ください。
|
||||
|
||||
#### MCPサーバー
|
||||
|
||||
DeerFlowは、機能を拡張するための設定可能なMCPサーバーとスキルをサポートしています。
|
||||
HTTP/SSE MCPサーバーでは、OAuthトークンフロー（`client_credentials`、`refresh_token`）がサポートされています。
|
||||
詳細な手順は[MCPサーバーガイド](backend/docs/MCP_SERVER.md)をご覧ください。
|
||||
|
||||
#### IMチャネル
|
||||
|
||||
DeerFlowはメッセージングアプリからのタスク受信をサポートしています。チャネルは設定時に自動的に開始されます。いずれもパブリックIPは不要です。
|
||||
|
||||
DeerFlowはワークスペースUIでユーザー所有のIMチャネル接続を公開することもできます。`channel_connections`を有効にすると、ログイン済みユーザーはサイドバー / Settings > ChannelsからTelegram、Slack、Discord、Feishu/Lark、DingTalk、WeChat、WeComをバインドできます。これは既存の`channels.*`送信トランスポートを再利用するため、パブリックIPやプロバイダーのコールバックURLは不要です。受信したIMメッセージは接続したDeerFlowユーザーアカウントの下で実行されます。セットアップとセキュリティ上の注意は[IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md)をご覧ください。
|
||||
|
||||
| チャネル | トランスポート | 難易度 |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API（ロングポーリング） | 簡単 |
|
||||
| Slack | Socket Mode | 中程度 |
|
||||
| Feishu / Lark | WebSocket | 中程度 |
|
||||
| WeChat | Tencent iLink（ロングポーリング） | 中程度 |
|
||||
| WeCom | WebSocket | 中程度 |
|
||||
| DingTalk | Stream Push（WebSocket） | 中程度 |
|
||||
|
||||
**`config.yaml`での設定：**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL（デフォルト: http://localhost:8001/api）
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# Gateway API URL（デフォルト: http://localhost:8001）
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# オプション: 全モバイルチャネルのグローバルセッションデフォルト
|
||||
session:
|
||||
assistant_id: lead_agent
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-...（Socket Mode）
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
|
||||
# オプション: チャネル/ユーザーごとのセッション設定
|
||||
session:
|
||||
assistant_id: mobile_agent
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip_agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # オプション：bot_tokenがない場合に初回のQRブートストラップを許可
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # DingTalk Open PlatformのClientId
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # DingTalk Open PlatformのClientSecret
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
card_template_id: "" # オプション：ストリーミングタイプライター効果用のAIカードテンプレートID
|
||||
```
|
||||
|
||||
対応するAPIキーを`.env`ファイルに設定します：
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegramのセットアップ**
|
||||
|
||||
1. [@BotFather](https://t.me/BotFather)とチャットし、`/newbot`を送信してHTTP APIトークンをコピーします。
|
||||
2. `.env`に`TELEGRAM_BOT_TOKEN`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**Slackのセットアップ**
|
||||
|
||||
1. [api.slack.com/apps](https://api.slack.com/apps)でSlackアプリを作成 → 新規アプリ作成 → 最初から作成。
|
||||
2. **OAuth & Permissions**で、Botトークンスコープを追加：`app_mentions:read`、`chat:write`、`im:history`、`im:read`、`im:write`、`files:write`。
|
||||
3. **Socket Mode**を有効化 → `connections:write`スコープのApp-Levelトークン（`xapp-…`）を生成。
|
||||
4. **Event Subscriptions**で、ボットイベントを購読：`app_mention`、`message.im`。
|
||||
5. `.env`に`SLACK_BOT_TOKEN`と`SLACK_APP_TOKEN`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**Feishu / Larkのセットアップ**
|
||||
|
||||
1. [Feishu Open Platform](https://open.feishu.cn/)でアプリを作成 → **ボット**機能を有効化。
|
||||
2. 権限を追加：`im:message`、`im:message.p2p_msg:readonly`、`im:resource`。
|
||||
3. **イベント**で`im.message.receive_v1`を購読し、**ロングコネクション**モードを選択。
|
||||
4. App IDとApp Secretをコピー。`.env`に`FEISHU_APP_ID`と`FEISHU_APP_SECRET`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**WeChatのセットアップ**
|
||||
|
||||
1. `config.yaml`で`wechat`チャネルを有効にします。
|
||||
2. `.env`に`WECHAT_BOT_TOKEN`を設定するか、初回のQRブートストラップのために`qrcode_login_enabled: true`を設定します。
|
||||
3. `bot_token`がなくQRブートストラップが有効な場合は、バックエンドログでiLinkが返したQRコンテンツを監視し、バインドフローを完了します。
|
||||
4. QRフローが成功した後、DeerFlowは取得したトークンを`state_dir`に永続化し、以降の再起動で再利用します。
|
||||
5. Docker Composeデプロイでは、`state_dir`を永続ボリュームに置き、`get_updates_buf`カーソルと保存済みの認証ステートが再起動後も保持されるようにしてください。
|
||||
|
||||
**WeComのセットアップ**
|
||||
|
||||
1. WeCom AI Botプラットフォームでボットを作成し、`bot_id`と`bot_secret`を取得します。
|
||||
2. `config.yaml`で`channels.wecom`を有効にし、`bot_id` / `bot_secret`を入力します。
|
||||
3. `.env`に`WECOM_BOT_ID`と`WECOM_BOT_SECRET`を設定します。
|
||||
4. バックエンドの依存関係に`wecom-aibot-python-sdk`が含まれていることを確認してください。このチャネルはWebSocketロングコネクションを使用し、パブリックなコールバックURLは不要です。
|
||||
5. 現在の統合では、受信テキスト、画像、ファイルメッセージをサポートしています。エージェントが生成した最終的な画像/ファイルもWeComの会話に送り返されます。
|
||||
|
||||
**DingTalkのセットアップ**
|
||||
|
||||
1. [DingTalk Open Platform](https://open.dingtalk.com/)でアプリを作成し、**ロボット**機能を有効化します。
|
||||
2. ロボット設定ページでメッセージ受信モードを**Streamモード**に設定します。
|
||||
3. `Client ID`と`Client Secret`をコピー。`.env`に`DINGTALK_CLIENT_ID`と`DINGTALK_CLIENT_SECRET`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
4. *（オプション）* ストリーミングAIカード返信（タイプライター効果）を有効にするには、[DingTalkカードプラットフォーム](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card)で**AIカード**テンプレートを作成し、`config.yaml`の`card_template_id`にテンプレートIDを設定します。`Card.Streaming.Write` および `Card.Instance.Write` 権限の申請も必要です。
|
||||
|
||||
**コマンド**
|
||||
|
||||
チャネル接続後、チャットから直接DeerFlowと対話できます：
|
||||
|
||||
| コマンド | 説明 |
|
||||
|---------|-------------|
|
||||
| `/new` | 新しい会話を開始 |
|
||||
| `/status` | 現在のスレッド情報を表示 |
|
||||
| `/models` | 利用可能なモデルを一覧表示 |
|
||||
| `/memory` | メモリを表示 |
|
||||
| `/help` | ヘルプを表示 |
|
||||
|
||||
> コマンドプレフィックスのないメッセージは通常のチャットとして扱われ、DeerFlowがスレッドを作成して会話形式で応答します。
|
||||
|
||||
#### LangSmithトレーシング
|
||||
|
||||
DeerFlowには[LangSmith](https://smith.langchain.com)による可観測性が組み込まれています。有効にすると、すべてのLLM呼び出し、エージェント実行、ツール実行がトレースされ、LangSmithダッシュボードで確認できます。
|
||||
|
||||
`.env`ファイルに以下を追加します：
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
#### Langfuseトレーシング
|
||||
|
||||
DeerFlowは、LangChain互換の実行に対して[Langfuse](https://langfuse.com)による可観測性もサポートしています。
|
||||
|
||||
`.env`ファイルに以下を追加します：
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
セルフホストのLangfuseインスタンスを使用している場合は、`LANGFUSE_BASE_URL`をデプロイ先のURLに設定します。
|
||||
|
||||
**トレース関連付けフィールド。** 各エージェント実行には、Langfuseの予約済みトレース属性が付与されるため、SessionsページとUsersページが自動的に表示されます：
|
||||
|
||||
- `session_id` = LangGraphの`thread_id`——同一会話のすべてのトレースをグループ化します
|
||||
- `user_id` = `get_effective_user_id()`から取得した有効なユーザー（認証なしモードでは`default`にフォールバック）
|
||||
- `trace_name` = assistant id（デフォルトは`lead-agent`）
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]`（未設定の場合は省略）
|
||||
- `metadata.deerflow_trace_id` = DeerFlowのリクエスト関連付けid。リクエストトレース関連付けが有効な場合は`X-Trace-Id`と一致します
|
||||
|
||||
これらは、gatewayパス（`runtime/runs/worker.py::run_agent`）と埋め込みパス（`client.py::DeerFlowClient.stream`）の両方で、グラフ呼び出しのルートで`RunnableConfig.metadata`に注入されるため、LangChain互換の任意のcallbackから読み取れます。`DEER_FLOW_ENV`（または`ENVIRONMENT`）を設定すると、デプロイ環境ごとにトレースにタグを付けられます。
|
||||
|
||||
#### 両方のプロバイダーを使用する
|
||||
|
||||
LangSmithとLangfuseの両方を有効にすると、DeerFlowは両方のトレーシングcallbackを取り付け、同じモデルアクティビティを両方のシステムに報告します。
|
||||
|
||||
あるプロバイダーが明示的に有効化されているにもかかわらず必要な認証情報が欠けている場合、またはそのcallbackの初期化に失敗した場合、DeerFlowはモデル作成時のトレーシング初期化中に早期に失敗（fail fast）し、エラーメッセージには失敗の原因となったプロバイダー名が示されます。
|
||||
|
||||
Dockerデプロイでは、トレーシングはデフォルトで無効です。`.env`で`LANGSMITH_TRACING=true`と`LANGSMITH_API_KEY`を設定して有効にします。
|
||||
|
||||
## Deep Researchからスーパーエージェントハーネスへ
|
||||
|
||||
DeerFlowはDeep Researchフレームワークとして始まり、コミュニティがそれを大きく発展させました。リリース以来、開発者たちはリサーチを超えて活用してきました：データパイプラインの構築、スライドデッキの生成、ダッシュボードの立ち上げ、コンテンツワークフローの自動化。私たちが予想もしなかったことです。
|
||||
|
||||
これは重要なことを示していました：DeerFlowは単なるリサーチツールではなかったのです。それは**ハーネス**——エージェントが実際に仕事をこなすためのインフラを提供するランタイムでした。
|
||||
|
||||
そこで、ゼロから再構築しました。
|
||||
|
||||
DeerFlow 2.0は、もはやつなぎ合わせるフレームワークではありません。バッテリー同梱、完全に拡張可能なスーパーエージェントハーネスです。LangGraphとLangChainの上に構築され、エージェントが必要とするすべてを標準搭載しています：ファイルシステム、メモリ、スキル、サンドボックス実行、そして複雑なマルチステップタスクのためのプランニングとサブエージェントの生成機能。
|
||||
|
||||
そのまま使うもよし。分解して自分のものにするもよし。
|
||||
|
||||
## コア機能
|
||||
|
||||
### スキルとツール
|
||||
|
||||
スキルこそが、DeerFlowを*ほぼ何でもできる*ものにしています。
|
||||
|
||||
標準的なエージェントスキルは構造化された機能モジュールです——ワークフロー、ベストプラクティス、サポートリソースへの参照を定義するMarkdownファイルです。DeerFlowにはリサーチ、レポート生成、スライド作成、Webページ、画像・動画生成などの組み込みスキルが付属しています。しかし、真の力は拡張性にあります：独自のスキルを追加し、組み込みスキルを置き換え、複合ワークフローに組み合わせることができます。
|
||||
|
||||
スキルはプログレッシブに読み込まれます——タスクが必要とする時にのみ、一度にすべてではありません。これによりコンテキストウィンドウを軽量に保ち、トークンに敏感なモデルでもDeerFlowがうまく動作します。
|
||||
|
||||
Gateway経由で`.skill`アーカイブをインストールする際、DeerFlowは`version`、`author`、`compatibility`などの標準的なオプショナルフロントマターメタデータを受け入れ、有効な外部スキルを拒否しません。
|
||||
|
||||
ツールも同じ哲学に従います。DeerFlowにはコアツールセット——Web検索、Webフェッチ、ファイル操作、bash実行——が付属し、MCPサーバーやPython関数によるカスタムツールをサポートしています。何でも入れ替え可能、何でも追加可能です。
|
||||
|
||||
Gatewayが生成するフォローアップ提案は、プレーン文字列のモデル出力とブロック/リスト形式のリッチコンテンツの両方をJSON配列レスポンスの解析前に正規化するため、プロバイダー固有のコンテンツラッパーが提案をサイレントにドロップすることはありません。
|
||||
|
||||
```
|
||||
# サンドボックスコンテナ内のパス
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← あなたのカスタムスキル
|
||||
```
|
||||
|
||||
#### Claude Code連携
|
||||
|
||||
`claude-to-deerflow`スキルを使えば、[Claude Code](https://docs.anthropic.com/en/docs/claude-code)から直接、実行中のDeerFlowインスタンスと対話できます。リサーチタスクの送信、ステータスの確認、スレッドの管理——すべてターミナルから離れずに実行できます。
|
||||
|
||||
**スキルのインストール**：
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
DeerFlowが実行中であることを確認し（デフォルトは`http://localhost:2026`）、Claude Codeで`/claude-to-deerflow`コマンドを使用します。
|
||||
|
||||
**できること**：
|
||||
- DeerFlowにメッセージを送信してストリーミングレスポンスを取得
|
||||
- 実行モードの選択：flash（高速）、standard、pro（プランニング）、ultra（サブエージェント）
|
||||
- DeerFlowのヘルスチェック、モデル/スキル/エージェントの一覧表示
|
||||
- スレッドと会話履歴の管理
|
||||
- 分析用ファイルのアップロード
|
||||
|
||||
**環境変数**（オプション、カスタムエンドポイント用）：
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # 統合プロキシベースURL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
完全なAPIリファレンスは[`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md)をご覧ください。
|
||||
|
||||
### セッションゴール (Session Goals)
|
||||
|
||||
`/goal <完了条件>`を使うと、現在のスレッドに1つのアクティブな完了条件を紐付けられます。このゴールはスレッドスコープのステートであり、スキルの有効化ではないため、DeerFlowが満たされたと判定するか、あなたがクリアするまでターンをまたいで有効なまま維持されます。
|
||||
|
||||
対応するコマンド：
|
||||
|
||||
```text
|
||||
/goal finish the implementation and make all tests pass
|
||||
/goal # アクティブなゴールを表示
|
||||
/goal clear # クリアする
|
||||
```
|
||||
|
||||
各Gateway駆動のrunの後に、DeerFlowはnon-thinkingな評価モデルを使って、可視の会話をアクティブなゴールと照らし合わせます。評価モデルは型付きblocker（`missing_evidence`、`needs_user_input`、`run_failed`、`external_wait`、`goal_not_met_yet`）と可視の証拠を返さなければなりません。DeerFlowがhidden continuationを注入するのは、直近のassistantターンが耐久性のあるチェックポイントに保存され、blockerが`goal_not_met_yet`であり、評価中にスレッドが変化せず、no-progressブレーカーが発火していない場合のみです。安全上限はデフォルトで8回のhidden continuationで、同一の非進行評価が繰り返されると2回で停止します。`/goal clear`と、ユーザーが手書きした新規入力はすべて、キュー内のcontinuationより優先されます。ゴールが満たされると、DeerFlowは自動的にクリアし、更新されたスレッドステートを公開します。
|
||||
|
||||
Web UIは入力欄の上にアクティブなゴールを表示します。同じコマンドはTUIとサポート対象のIMチャネルからも利用できます。Web UIとサポート対象のIMチャネルでは、`/goal <完了条件>`を設定するとその条件をタスクとしてrunを開始します。ステータス確認やクリアのコマンドはゴールステートの管理のみを行います。
|
||||
|
||||
### サブエージェント
|
||||
|
||||
複雑なタスクは単一のパスに収まりません。DeerFlowはそれを分解します。
|
||||
|
||||
リードエージェントはオンザフライでサブエージェントを生成できます——それぞれ独自のスコープ付きコンテキスト、ツール、終了条件を持ちます。サブエージェントは可能な限り並列で実行され、構造化された結果を報告し、リードエージェントがすべてを一貫した出力に統合します。
|
||||
|
||||
これがDeerFlowが数分から数時間かかるタスクを処理する方法です：リサーチタスクが十数のサブエージェントに展開され、それぞれが異なる角度を探索し、1つのレポート——またはWebサイト——または生成されたビジュアル付きのスライドデッキに収束します。1つのハーネス、多くの手。
|
||||
|
||||
### サンドボックスとファイルシステム
|
||||
|
||||
DeerFlowは物事を*語る*だけではありません。自分のコンピューターを持っています。
|
||||
|
||||
各タスクは、完全なファイルシステムを持つ分離されたDockerコンテナ内で実行されます——スキル、ワークスペース、アップロード、出力。エージェントはファイルの読み書き・編集を行います。bashコマンドを実行し、コーディングを行います。画像を表示します。すべてサンドボックス化され、すべて監査可能で、セッション間の汚染はゼロです。
|
||||
|
||||
これが、ツールアクセスのあるチャットボットと、実際の実行環境を持つエージェントの違いです。
|
||||
|
||||
```
|
||||
# サンドボックスコンテナ内のパス
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← あなたのファイル
|
||||
├── workspace/ ← エージェントの作業ディレクトリ
|
||||
└── outputs/ ← 最終成果物
|
||||
```
|
||||
|
||||
### コンテキストエンジニアリング
|
||||
|
||||
**分離されたサブエージェントコンテキスト**：各サブエージェントは独自の分離されたコンテキストで実行されます。これにより、サブエージェントはメインエージェントや他のサブエージェントのコンテキストを見ることができません。これは、サブエージェントが目の前のタスクに集中し、メインエージェントや他のサブエージェントのコンテキストに気を取られないようにするために重要です。
|
||||
|
||||
**要約化**：セッション内で、DeerFlowはコンテキストを積極的に管理します——完了したサブタスクの要約、中間結果のファイルシステムへのオフロード、もはや直接関係のないものの圧縮。これにより、コンテキストウィンドウを超えることなく、長いマルチステップタスク全体を通じてシャープさを維持します。
|
||||
|
||||
### 長期メモリ
|
||||
|
||||
ほとんどのエージェントは、会話が終わるとすべてを忘れます。DeerFlowは記憶します。
|
||||
|
||||
セッションをまたいで、DeerFlowはあなたのプロフィール、好み、蓄積された知識の永続的なメモリを構築します。使えば使うほど、あなたのことをよく知るようになります——あなたの文体、技術スタック、繰り返されるワークフロー。メモリはローカルに保存され、あなたの管理下にあります。
|
||||
|
||||
メモリ更新は適用時に重複するファクトエントリをスキップするようになり、繰り返される好みやコンテキストがセッションをまたいで際限なく蓄積されることはありません。
|
||||
|
||||
## 推奨モデル
|
||||
|
||||
DeerFlowはモデルに依存しません——OpenAI互換APIを実装する任意のLLMで動作します。とはいえ、以下をサポートするモデルで最高のパフォーマンスを発揮します：
|
||||
|
||||
- **長いコンテキストウィンドウ**（10万トークン以上）：深いリサーチとマルチステップタスク向け
|
||||
- **推論能力**：適応的なプランニングと複雑な分解向け
|
||||
- **マルチモーダル入力**：画像理解と動画理解向け
|
||||
- **強力なツール使用**：信頼性の高いファンクションコーリングと構造化された出力向け
|
||||
|
||||
## 組み込みPythonクライアント
|
||||
|
||||
DeerFlowは、完全なHTTPサービスを実行せずに組み込みPythonライブラリとして使用できます。`DeerFlowClient`は、すべてのエージェントとGateway機能へのプロセス内直接アクセスを提供し、HTTP Gateway APIと同じレスポンススキーマを返します。HTTP Gatewayは、LangGraphスレッド自体が削除された後にDeerFlow管理下のローカルスレッドデータを削除するための`DELETE /api/threads/{thread_id}`も公開しています：
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# チャット
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# ストリーミング（LangGraph SSEプロトコル：values、messages-tuple、end）
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# 設定＆管理 — Gateway準拠のdictを返す
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
client.set_goal("thread-1", "finish the implementation and make all tests pass")
|
||||
client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
|
||||
client.clear_goal("thread-1")
|
||||
```
|
||||
|
||||
すべてのdict返却メソッドはCIでGateway Pydanticレスポンスモデルに対して検証されており（`TestGatewayConformance`）、組み込みクライアントがHTTP APIスキーマと同期していることを保証します。完全なAPIドキュメントは`backend/packages/harness/deerflow/client.py`をご覧ください。
|
||||
|
||||
## スケジュールタスク (Scheduled Tasks)
|
||||
|
||||
DeerFlowには現在、ワークスペース内でファーストクラスのスケジュールタスクMVPが組み込まれています。
|
||||
|
||||
現在のMVPの機能：
|
||||
|
||||
- `/workspace/scheduled-tasks`でタスクを管理
|
||||
- 各スケジュールタスクがスレッドを再利用するか、実行ごとに新しいスレッドを作成するかを選択可能
|
||||
- `once`と`cron`のスケジュールをサポート
|
||||
- バックグラウンドのスケジュール実行を非対話型のDeerFlow runとして実行（`ask_clarification`はここでは公開されません）
|
||||
- 再利用された同じスレッド上でアクティブなrunと衝突する期限到来のcron実行に対して`skip`オーバーラップ挙動を使用
|
||||
- タスクの一時停止、再開、トリガー、履歴確認、削除
|
||||
- スケジュールされた作業を通常のDeerFlow runライフサイクルを通じて実行
|
||||
|
||||
現在のMVPの制限：
|
||||
|
||||
- 会話で`schedule_task`ツールを作成する機能はまだありません
|
||||
- テキストのみの通知ジョブはありません
|
||||
- チャネルやGitHubのディスパッチターゲットはありません
|
||||
- この最初のバージョンでは`interval`スケジュールタイプはありません
|
||||
|
||||
`config.yaml -> scheduler.enabled`でバックグラウンドポーリングを有効にします。手動トリガーは同じスケジュールタスクリソースと実行パスを使用します。
|
||||
|
||||
## ターミナルワークベンチ (TUI)
|
||||
|
||||
`deerflow`は、シェルに暮らす人々のためのターミナルネイティブなワークベンチです。**組み込み**で`DeerFlowClient`上で実行され、Gateway、フロントエンド、nginx、Dockerは不要ですが、DeerFlowの他の部分と同じ`config.yaml`、checkpointer、スキル、メモリ、MCP、サンドボックス設定を尊重します。
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # オプションの'textual'依存関係
|
||||
|
||||
deerflow # ターミナルUIを起動（TTYが必要）
|
||||
deerflow --continue # 直近のスレッドを再開
|
||||
deerflow --resume THREAD # IDでスレッドを再開
|
||||
deerflow --print "summarize this repo" # ヘッドレスでstdoutにワンショットの回答を出力
|
||||
deerflow --json "hello" # ヘッドレスで改行区切りのStreamEventを出力
|
||||
```
|
||||
|
||||
ストリーミング文字起こし（Markdownでレンダリングされた回答）、コンパクトなツールアクティビティカード、`/`スラッシュコマンドパレット、`/goal`ゴール管理、`/model`と`/threads`ピッカー、入力履歴、`Esc` / `Ctrl+C`割り込みを備えた、キーボード駆動のチャット画面。TUIで開いたセッションはWeb UIのサイドバーにも表示されます。ローカルのデフォルトユーザーの下で共有スレッドストアに書き込むため、**Gatewayを実行せずに**ターミナルとウェブが同期します。
|
||||
|
||||
完全なガイドは[backend/docs/TUI.md](backend/docs/TUI.md)をご覧ください。
|
||||
|
||||
## ドキュメント
|
||||
|
||||
- [コントリビュートガイド](CONTRIBUTING.md) - 開発環境のセットアップとワークフロー
|
||||
- [設定ガイド](backend/docs/CONFIGURATION.md) - セットアップと設定の手順
|
||||
- [アーキテクチャ概要](backend/CLAUDE.md) - 技術的なアーキテクチャの詳細
|
||||
- [バックエンドアーキテクチャ](backend/README.md) - バックエンドアーキテクチャとAPIリファレンス
|
||||
|
||||
## ⚠️ セキュリティに関する注意
|
||||
|
||||
### 不適切なデプロイはセキュリティリスクを引き起こす可能性があります
|
||||
|
||||
DeerFlowは**システムコマンドの実行、リソース操作、ビジネスロジックの呼び出し**などの重要な高権限機能を備えており、デフォルトでは**ローカルの信頼できる環境（127.0.0.1のループバックアクセスのみ）にデプロイされる設計**になっています。信頼できないLAN、公開クラウドサーバー、または複数のエンドポイントからアクセス可能なネットワーク環境にエージェントをデプロイし、厳格なセキュリティ対策を講じない場合、以下のようなセキュリティリスクが生じる可能性があります：
|
||||
|
||||
- **不正な違法呼び出し**：エージェントの機能が権限のない第三者や悪意のあるインターネットスキャナーに発見され、システムコマンドやファイル読み書きなどの高リスク操作を実行する不正な一括リクエストが引き起こされ、重大なセキュリティ上の問題が発生する可能性があります。
|
||||
- **コンプライアンスおよび法的リスク**：エージェントがサイバー攻撃やデータ窃取などの違法行為に不正使用された場合、法的責任やコンプライアンス上のリスクが生じる可能性があります。
|
||||
|
||||
### セキュリティ推奨事項
|
||||
|
||||
**注意：DeerFlowはローカルの信頼できるネットワーク環境にデプロイすることを強く推奨します。** クロスデバイス・クロスネットワークのデプロイが必要な場合は、以下のような厳格なセキュリティ対策を実装する必要があります：
|
||||
|
||||
- **IPホワイトリストの設定**：`iptables`を使用するか、ハードウェアファイアウォール / ACL機能付きスイッチをデプロイして**IPホワイトリストルールを設定**し、他のすべてのIPアドレスからのアクセスを拒否します。
|
||||
- **前置認証**：リバースプロキシ（nginxなど）を設定し、**強力な前置認証を有効化**して、認証なしのアクセスをブロックします。
|
||||
- **ネットワーク分離**：可能であれば、エージェントと信頼できるデバイスを**同一の専用VLAN**に配置し、他のネットワークデバイスから隔離します。
|
||||
- **アップデートを継続的に確認**：DeerFlowのセキュリティ機能のアップデートを継続的にフォローしてください。
|
||||
|
||||
## コントリビュート
|
||||
|
||||
コントリビューションを歓迎します！開発環境のセットアップ、ワークフロー、ガイドラインについては[CONTRIBUTING.md](CONTRIBUTING.md)をご覧ください。
|
||||
|
||||
回帰テストのカバレッジには、`backend/tests/`でのDockerサンドボックスモード検出とプロビジョナーkubeconfig-pathハンドリングテストが含まれます。
|
||||
|
||||
## ライセンス
|
||||
|
||||
このプロジェクトはオープンソースであり、[MITライセンス](./LICENSE)の下で提供されています。
|
||||
|
||||
## 謝辞
|
||||
|
||||
DeerFlowはオープンソースコミュニティの素晴らしい成果の上に構築されています。DeerFlowを可能にしてくれたすべてのプロジェクトとコントリビューターに深く感謝いたします。まさに、巨人の肩の上に立っています。
|
||||
|
||||
以下のプロジェクトの貴重な貢献に心からの感謝を申し上げます：
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**：その優れたフレームワークがLLMのインタラクションとチェーンを支え、シームレスな統合と機能を実現しています。
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**：マルチエージェントオーケストレーションへの革新的なアプローチが、DeerFlowの洗練されたワークフローの実現に大きく貢献しています。
|
||||
|
||||
これらのプロジェクトはオープンソースコラボレーションの変革的な力を体現しており、その基盤の上に構築できることを誇りに思います。
|
||||
|
||||
### 主要コントリビューター
|
||||
|
||||
`DeerFlow`のコア著者に心からの感謝を捧げます。そのビジョン、情熱、献身がこのプロジェクトに命を吹き込みました：
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
揺るぎないコミットメントと専門知識が、DeerFlowの成功の原動力です。この旅の先頭に立ってくださっていることを光栄に思います。
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+690
@@ -0,0 +1,690 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | [Français](./README_fr.md) | Русский
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
|
||||
> 28 февраля 2026 года DeerFlow занял 🏆 #1 в GitHub Trending после релиза версии 2. Спасибо огромное нашему сообществу — всё благодаря вам! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) — open-source **Super Agent Harness**, который управляет **Sub-Agents**, **Memory** и **Sandbox** для решения почти любой задачи. Всё на основе расширяемых **Skills**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 — проект переписан с нуля.** Общего кода с v1 нет. Если нужен оригинальный Deep Research фреймворк — он живёт в ветке [`1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x), туда тоже принимают контрибьюты. Активная разработка идёт в 2.0.
|
||||
|
||||
## Официальный сайт
|
||||
|
||||
Больше информации и живые демо на [**официальном сайте**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan от ByteDance Volcengine
|
||||
|
||||
- Рекомендуем Doubao-Seed-2.0-Code, DeepSeek v3.2 и Kimi 2.5 для запуска DeerFlow
|
||||
- [Подробнее](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Для разработчиков из материкового Китая](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow интегрирован с инструментарием для умного поиска и краулинга от BytePlus — [InfoQuest (есть бесплатный онлайн-доступ)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png"
|
||||
alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Содержание
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Официальный сайт](#официальный-сайт)
|
||||
- [Coding Plan от ByteDance Volcengine](#coding-plan-от-bytedance-volcengine)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Содержание](#содержание)
|
||||
- [Установка одной фразой для coding agent](#установка-одной-фразой-для-coding-agent)
|
||||
- [Быстрый старт](#быстрый-старт)
|
||||
- [Конфигурация](#конфигурация)
|
||||
- [Запуск](#запуск)
|
||||
- [Вариант 1: Docker (рекомендуется)](#вариант-1-docker-рекомендуется)
|
||||
- [Вариант 2: Локальная разработка](#вариант-2-локальная-разработка)
|
||||
- [Дополнительно](#дополнительно)
|
||||
- [Режим Sandbox](#режим-sandbox)
|
||||
- [MCP-сервер](#mcp-сервер)
|
||||
- [Мессенджеры](#мессенджеры)
|
||||
- [Трассировка LangSmith](#трассировка-langsmith)
|
||||
- [Трассировка Langfuse](#трассировка-langfuse)
|
||||
- [Использование обоих провайдеров](#использование-обоих-провайдеров)
|
||||
- [От Deep Research к Super Agent Harness](#от-deep-research-к-super-agent-harness)
|
||||
- [Core Features](#core-features)
|
||||
- [Skills & Tools](#skills--tools)
|
||||
- [Интеграция с Claude Code](#интеграция-с-claude-code)
|
||||
- [Цели сессии (Session Goals)](#цели-сессии-session-goals)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox & файловая система](#sandbox--файловая-система)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Long-Term Memory](#long-term-memory)
|
||||
- [Рекомендуемые модели](#рекомендуемые-модели)
|
||||
- [Встроенный Python-клиент](#встроенный-python-клиент)
|
||||
- [Запланированные задачи (Scheduled Tasks)](#запланированные-задачи-scheduled-tasks)
|
||||
- [Терминальная панель (TUI)](#терминальная-панель-tui)
|
||||
- [Документация](#документация)
|
||||
- [⚠️ Безопасность](#️-безопасность)
|
||||
- [Участие в разработке](#участие-в-разработке)
|
||||
- [Лицензия](#лицензия)
|
||||
- [Благодарности](#благодарности)
|
||||
- [Ключевые контрибьюторы](#ключевые-контрибьюторы)
|
||||
- [История звёзд](#история-звёзд)
|
||||
|
||||
## Установка одной фразой для coding agent
|
||||
|
||||
Если вы используете Claude Code, Codex, Cursor, Windsurf или другой coding agent, просто отправьте ему эту фразу:
|
||||
|
||||
```text
|
||||
Если DeerFlow еще не клонирован, сначала клонируй его, а затем подготовь локальное окружение разработки по инструкции https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Этот prompt предназначен для coding agent. Он просит агента при необходимости сначала клонировать репозиторий, предпочесть Docker, если он доступен, и в конце вернуть точную команду запуска и список недостающих настроек.
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
### Конфигурация
|
||||
|
||||
1. **Склонировать репозиторий DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Запустить мастер настройки (рекомендуется)**
|
||||
|
||||
Из корня проекта (`deer-flow/`) запустите:
|
||||
|
||||
```bash
|
||||
make setup
|
||||
```
|
||||
|
||||
Запустится интерактивный мастер, который поможет выбрать LLM-провайдера, опциональный веб-поиск и настройки выполнения/безопасности (режим sandbox, доступ к bash, инструменты записи файлов). Он сгенерирует минимальный `config.yaml` и запишет ключи в `.env`. Это занимает около 2 минут.
|
||||
|
||||
В любой момент запускайте `make doctor`, чтобы проверить конфигурацию и получить конкретные подсказки по исправлению.
|
||||
Если вы открываете GitHub issue о проблеме с локальной установкой или работой системы, выполните
|
||||
`make support-bundle`. Команда выводит дальнейшие шаги для автора отчёта, создаёт файл
|
||||
`*-issue-summary.md`, который нужно вставить в issue, файл `*-issue-draft.md`
|
||||
для оформления issue с помощью AI и, опционально, zip-архив с диагностикой в
|
||||
`.deer-flow/support-bundles/`. Если issue оформляет AI-ассистент, он должен начать
|
||||
с черновика и заменить каждый плейсхолдер REQUIRED, а не выдумывать недостающие
|
||||
факты. Прикладывайте zip-архив только если его запросит мейнтейнер или если одной
|
||||
сводки недостаточно. Мейнтейнеры и AI-инструменты триажа могут начинать с
|
||||
`triage.json`; архив содержит только очищенную от чувствительных данных диагностику
|
||||
и манифесты файлов и не включает `.env`, исходные сообщения диалогов или содержимое
|
||||
пользовательских файлов.
|
||||
|
||||
> **Продвинутая / ручная настройка**: если вы предпочитаете редактировать `config.yaml` напрямую, выполните вместо этого `make config`, чтобы скопировать полный шаблон. Полный справочник — `config.example.yaml`, включая CLI-провайдеров (Codex CLI, Claude Code OAuth), OpenRouter, Responses API и многое другое.
|
||||
|
||||
<details>
|
||||
<summary>Примеры ручной настройки моделей</summary>
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
```
|
||||
|
||||
OpenRouter и аналогичные OpenAI-совместимые шлюзы настраиваются через `langchain_openai:ChatOpenAI` с параметром `base_url`. Если вы предпочитаете имя переменной окружения, специфичное для провайдера, укажите его в `api_key` явно (например, `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
Чтобы направить модели OpenAI через `/v1/responses`, продолжайте использовать `langchain_openai:ChatOpenAI` и задайте `use_responses_api: true` вместе с `output_version: responses/v1`.
|
||||
|
||||
Для vLLM 0.19.0 используйте `deerflow.models.vllm_provider:VllmChatModel`. Для reasoning-моделей в стиле Qwen DeerFlow переключает режим рассуждений через `extra_body.chat_template_kwargs.enable_thinking` и сохраняет нестандартное поле `reasoning` vLLM в многоходовых диалогах с вызовами инструментов. Устаревшие конфигурации `thinking` автоматически нормализуются для обратной совместимости. Reasoning-моделям также может потребоваться запуск сервера с флагом `--reasoning-parser ...`. Если ваш локальный vLLM принимает любой непустой API-ключ, всё равно задайте `VLLM_API_KEY` со значением-заглушкой.
|
||||
|
||||
Примеры CLI-провайдеров:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI читает `~/.codex/auth.json`
|
||||
- Claude Code принимает `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_CREDENTIALS_PATH` или `~/.claude/.credentials.json`
|
||||
- Записи ACP-агентов настраиваются отдельно от провайдеров моделей — если вы настраиваете `acp_agents.codex`, укажите в нём Codex ACP-адаптер, например `npx -y @zed-industries/codex-acp`
|
||||
- На macOS при необходимости экспортируйте аутентификацию Claude Code явно:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
API-ключи также можно задать вручную в `.env` (рекомендуется) или экспортировать в оболочке:
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Запуск
|
||||
|
||||
#### Вариант 1: Docker (рекомендуется)
|
||||
|
||||
**Разработка** (hot-reload, монтирование исходников):
|
||||
|
||||
```bash
|
||||
make docker-init # Загрузить образ Sandbox (один раз или при обновлении)
|
||||
make docker-start # Запустить сервисы
|
||||
```
|
||||
|
||||
**Продакшен** (собирает образы локально):
|
||||
|
||||
```bash
|
||||
make up # Собрать образы и запустить все сервисы
|
||||
make down # Остановить и удалить контейнеры
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> На Linux при ошибке `permission denied` для Docker daemon добавьте пользователя в группу `docker` и перелогиньтесь. Подробнее в [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied).
|
||||
|
||||
Адрес: http://localhost:2026
|
||||
|
||||
#### Вариант 2: Локальная разработка
|
||||
|
||||
Предварительное условие: сначала выполните шаги раздела «Конфигурация» выше (`make setup`). Для `make dev` нужен корректный `config.yaml` в корне проекта. Задайте `DEER_FLOW_PROJECT_ROOT`, чтобы явно указать корень проекта, или `DEER_FLOW_CONFIG_PATH`, чтобы указать конкретный файл конфигурации. Состояние времени выполнения по умолчанию записывается в `.deer-flow` в корне проекта и может быть перенесено через `DEER_FLOW_HOME`; skills по умолчанию читаются из `skills/` в корне проекта, путь можно переопределить через `DEER_FLOW_SKILLS_PATH`. Перед запуском выполните `make doctor`, чтобы проверить настройку.
|
||||
В Windows запускайте локальный процесс разработки из Git Bash. Нативные оболочки `cmd.exe` и PowerShell не поддерживаются для сервисных скриптов на bash, а работа в WSL не гарантируется, поскольку некоторые скрипты зависят от утилит Git for Windows, таких как `cygpath`.
|
||||
|
||||
1. **Проверить зависимости**:
|
||||
```bash
|
||||
make check # Проверяет Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Установить зависимости**:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
3. **(Опционально) Загрузить образ Sandbox заранее**:
|
||||
```bash
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Запустить сервисы**:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Адрес**: http://localhost:2026
|
||||
|
||||
### Дополнительно
|
||||
|
||||
#### Режим Sandbox
|
||||
|
||||
DeerFlow поддерживает несколько режимов выполнения:
|
||||
- **Локальное выполнение** — код запускается прямо на хосте
|
||||
- **Docker** — код выполняется в изолированных Docker-контейнерах
|
||||
- **Docker + Kubernetes** — выполнение в Kubernetes-подах через provisioner
|
||||
|
||||
Подробнее в [руководстве по конфигурации Sandbox](backend/docs/CONFIGURATION.md#sandbox).
|
||||
|
||||
#### MCP-сервер
|
||||
|
||||
DeerFlow поддерживает настраиваемые MCP-серверы для расширения возможностей. Для HTTP/SSE MCP-серверов поддерживаются OAuth-токены (`client_credentials`, `refresh_token`). Подробнее в [руководстве по MCP-серверу](backend/docs/MCP_SERVER.md).
|
||||
|
||||
#### Мессенджеры
|
||||
|
||||
DeerFlow принимает задачи прямо из мессенджеров. Каналы запускаются автоматически при настройке, публичный IP не нужен.
|
||||
|
||||
DeerFlow может также предоставлять в workspace UI пользовательские подключения IM-каналов. Когда включён `channel_connections`, вошедшие в систему пользователи могут привязать Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat или WeCom из боковой панели / Settings > Channels. Это переиспользует существующие исходящие транспорты `channels.*`, поэтому публичный IP или URL обратного вызова провайдера не требуются. Входящие IM-сообщения выполняются от имени подключённого пользователя DeerFlow. Настройки и вопросы безопасности описаны в [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md).
|
||||
|
||||
| Канал | Транспорт | Сложность |
|
||||
|-------|-----------|-----------|
|
||||
| Telegram | Bot API (long-polling) | Просто |
|
||||
| Slack | Socket Mode | Средне |
|
||||
| Feishu / Lark | WebSocket | Средне |
|
||||
| WeChat | Tencent iLink (long-polling) | Средне |
|
||||
| WeCom | WebSocket | Средне |
|
||||
| DingTalk | Stream Push (WebSocket) | Средне |
|
||||
|
||||
**Конфигурация в `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN
|
||||
app_token: $SLACK_APP_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # опционально: разрешить первичную загрузку через QR-код при отсутствии bot_token
|
||||
allowed_users: [] # пусто = разрешить всем
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId с DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret с DingTalk Open Platform
|
||||
allowed_users: [] # пусто = разрешить всем
|
||||
card_template_id: "" # Опционально: ID шаблона AI Card для потокового эффекта печатной машинки
|
||||
```
|
||||
|
||||
**Ключи API в `.env`:**
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Настройка Telegram**
|
||||
|
||||
1. Напишите [@BotFather](https://t.me/BotFather), отправьте `/newbot` и скопируйте HTTP API-токен.
|
||||
2. Укажите `TELEGRAM_BOT_TOKEN` в `.env` и включите канал в `config.yaml`.
|
||||
|
||||
**Настройка WeChat**
|
||||
|
||||
1. Включите канал `wechat` в `config.yaml`.
|
||||
2. Либо задайте `WECHAT_BOT_TOKEN` в `.env`, либо установите `qrcode_login_enabled: true` для первичной загрузки через QR-код.
|
||||
3. Когда `bot_token` отсутствует и загрузка через QR включена, следите за логами бэкенда — там появится QR-контент, возвращённый iLink, — и завершите процесс привязки.
|
||||
4. После успешного прохождения QR-процесса DeerFlow сохраняет полученный токен в `state_dir` для последующих перезапусков.
|
||||
5. Для развёртываний Docker Compose держите `state_dir` на постоянном томе, чтобы курсор `get_updates_buf` и сохранённое состояние аутентификации переживали перезапуски.
|
||||
|
||||
**Настройка WeCom**
|
||||
|
||||
1. Создайте бота на платформе WeCom AI Bot и получите `bot_id` и `bot_secret`.
|
||||
2. Включите `channels.wecom` в `config.yaml` и заполните `bot_id` / `bot_secret`.
|
||||
3. Задайте `WECOM_BOT_ID` и `WECOM_BOT_SECRET` в `.env`.
|
||||
4. Убедитесь, что зависимости бэкенда включают `wecom-aibot-python-sdk`. Канал использует долговременное WebSocket-соединение и не требует публичного URL обратного вызова.
|
||||
5. Текущая интеграция поддерживает входящие текстовые сообщения, изображения и файлы. Итоговые изображения/файлы, сгенерированные агентом, также отправляются обратно в диалог WeCom.
|
||||
|
||||
**Настройка DingTalk**
|
||||
|
||||
1. Создайте приложение на [DingTalk Open Platform](https://open.dingtalk.com/) и включите возможность **Робот**.
|
||||
2. На странице настроек робота установите режим приёма сообщений на **Stream**.
|
||||
3. Скопируйте `Client ID` и `Client Secret`. Укажите `DINGTALK_CLIENT_ID` и `DINGTALK_CLIENT_SECRET` в `.env` и включите канал в `config.yaml`.
|
||||
4. *(Опционально)* Для включения потоковых ответов AI Card (эффект печатной машинки) создайте шаблон **AI Card** на [платформе карточек DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), затем укажите `card_template_id` в `config.yaml` с ID шаблона. Также необходимо запросить разрешения `Card.Streaming.Write` и `Card.Instance.Write`.
|
||||
|
||||
**Доступные команды**
|
||||
|
||||
| Команда | Описание |
|
||||
|---------|----------|
|
||||
| `/new` | Начать новый диалог |
|
||||
| `/status` | Показать информацию о текущем треде |
|
||||
| `/models` | Список доступных моделей |
|
||||
| `/memory` | Просмотреть память |
|
||||
| `/help` | Показать справку |
|
||||
|
||||
> Сообщения без команды воспринимаются как обычный чат — DeerFlow создаёт тред и отвечает.
|
||||
|
||||
#### Трассировка LangSmith
|
||||
|
||||
DeerFlow имеет встроенную интеграцию с [LangSmith](https://smith.langchain.com) для наблюдаемости. При включении все вызовы LLM, запуски агентов и выполнения инструментов отслеживаются и отображаются в дашборде LangSmith.
|
||||
|
||||
Добавьте в файл `.env` в корне проекта:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=deer-flow
|
||||
```
|
||||
|
||||
`LANGSMITH_ENDPOINT` по умолчанию `https://api.smith.langchain.com` и может быть переопределён при необходимости. Устаревшие переменные `LANGCHAIN_*` (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY` и т.д.) также поддерживаются для обратной совместимости; `LANGSMITH_*` имеет приоритет, когда заданы обе.
|
||||
|
||||
#### Трассировка Langfuse
|
||||
|
||||
DeerFlow также поддерживает наблюдаемость через [Langfuse](https://langfuse.com) для запусков, совместимых с LangChain.
|
||||
|
||||
Добавьте в файл `.env`:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
Если вы используете собственный экземпляр Langfuse, укажите `LANGFUSE_BASE_URL` в качестве URL вашего развёртывания.
|
||||
|
||||
**Поля корреляции трасс.** Каждый запуск агента аннотируется зарезервированными атрибутами трассировки Langfuse, поэтому страницы Sessions и Users заполняются автоматически:
|
||||
|
||||
- `session_id` = `thread_id` LangGraph — группирует все трассы одного диалога
|
||||
- `user_id` = эффективный пользователь из `get_effective_user_id()` (возвращается к `default` в режиме без аутентификации)
|
||||
- `trace_name` = assistant id (по умолчанию `lead-agent`)
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]` (опускается, если не заданы)
|
||||
- `metadata.deerflow_trace_id` = идентификатор корреляции запросов DeerFlow, совпадающий с `X-Trace-Id`, когда корреляция трассировки запросов включена
|
||||
|
||||
Эти поля внедряются в `RunnableConfig.metadata` в корне вызова графа как для gateway-пути (`runtime/runs/worker.py::run_agent`), так и для встроенного пути (`client.py::DeerFlowClient.stream`), поэтому любой LangChain-совместимый callback может их прочитать. Установите `DEER_FLOW_ENV` (или `ENVIRONMENT`) для тегирования трасс по среде развёртывания.
|
||||
|
||||
#### Использование обоих провайдеров
|
||||
|
||||
Если и LangSmith, и Langfuse включены, DeerFlow подключает оба callback'а трассировки и отправляет одну и ту же активность модели в обе системы.
|
||||
|
||||
Если провайдер явно включён, но отсутствуют необходимые учётные данные, или если его callback не может инициализироваться, DeerFlow завершает работу с ошибкой (fail fast) при инициализации трассировки во время создания модели, а сообщение об ошибке указывает провайдера, вызвавшего сбой.
|
||||
|
||||
В Docker-развёртываниях трассировка отключена по умолчанию. Установите `LANGSMITH_TRACING=true` и `LANGSMITH_API_KEY` в `.env` для включения.
|
||||
|
||||
## От Deep Research к Super Agent Harness
|
||||
|
||||
DeerFlow начинался как фреймворк для Deep Research, и сообщество вышло далеко за эти рамки. После запуска разработчики строили пайплайны, генерировали презентации, поднимали дашборды, автоматизировали контент. То, чего мы не ожидали.
|
||||
|
||||
Стало понятно: DeerFlow не просто research-инструмент. Это **harness**: runtime, который даёт агентам необходимую инфраструктуру.
|
||||
|
||||
Поэтому мы переписали всё с нуля.
|
||||
|
||||
DeerFlow 2.0 — это Super Agent Harness «из коробки». Batteries included, полностью расширяемый. Построен на LangGraph и LangChain. По умолчанию есть всё, что нужно агенту: файловая система, memory, skills, sandbox-выполнение и возможность планировать и запускать sub-agents для сложных многошаговых задач.
|
||||
|
||||
Используйте как есть. Или разберите и переделайте под себя.
|
||||
|
||||
## Core Features
|
||||
|
||||
### Skills & Tools
|
||||
|
||||
Skills — это то, что позволяет DeerFlow делать почти что угодно.
|
||||
|
||||
Agent Skill — это структурированный модуль: Markdown-файл с описанием воркфлоу, лучших практик и ссылок на ресурсы. DeerFlow поставляется со встроенными skills для ресёрча, генерации отчётов, слайдов, веб-страниц, изображений и видео. Но главное — расширяемость: добавляйте свои skills, заменяйте встроенные или собирайте из них составные воркфлоу.
|
||||
|
||||
Skills загружаются по мере необходимости, только когда задача их требует. Это держит контекстное окно чистым.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← ваш skill
|
||||
```
|
||||
|
||||
#### Интеграция с Claude Code
|
||||
|
||||
Skill `claude-to-deerflow` позволяет работать с DeerFlow прямо из [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Отправляйте задачи, проверяйте статус, управляйте тредами, не выходя из терминала.
|
||||
|
||||
**Установка скилла**:
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
**Что можно делать**:
|
||||
- Отправлять сообщения в DeerFlow и получать потоковые ответы
|
||||
- Выбирать режимы выполнения: flash (быстро), standard, pro (planning), ultra (sub-agents)
|
||||
- Проверять статус DeerFlow, просматривать модели, скиллы, агентов
|
||||
- Управлять тредами и историей диалога
|
||||
- Загружать файлы для анализа
|
||||
|
||||
Полный справочник API в [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md).
|
||||
|
||||
### Цели сессии (Session Goals)
|
||||
|
||||
Используйте `/goal <условие завершения>`, чтобы привязать к текущему треду одно активное условие завершения. Цель — это состояние уровня треда, а не активация навыка, поэтому она остаётся активной между ходами, пока DeerFlow не сочтёт её выполненной или пока вы её не очистите.
|
||||
|
||||
Поддерживаемые команды:
|
||||
|
||||
```text
|
||||
/goal finish the implementation and make all tests pass
|
||||
/goal # показать активную цель
|
||||
/goal clear # очистить её
|
||||
```
|
||||
|
||||
После каждого запуска, выполненного через Gateway, DeerFlow оценивает видимый диалог относительно активной цели с помощью non-thinking модели-оценщика. Оценщик должен вернуть типизированный блокер (`missing_evidence`, `needs_user_input`, `run_failed`, `external_wait` или `goal_not_met_yet`) с видимыми доказательствами. DeerFlow добавляет hidden continuation только тогда, когда последний ход assistant сохранён в чекпоинте, блокер имеет тип `goal_not_met_yet`, тред не изменился во время оценки и счётчик отсутствия прогресса не сработал. Предел безопасности по умолчанию — 8 hidden continuation, а повторяющиеся одинаковые оценки без прогресса останавливаются после 2 попыток. `/goal clear` и любой новый ввод от пользователя имеют приоритет над continuation в очереди. Когда цель выполнена, DeerFlow очищает её автоматически и публикует обновлённое состояние треда.
|
||||
|
||||
Веб-интерфейс показывает активную цель над полем ввода. Та же команда доступна из TUI и поддерживаемых IM-каналов. В веб-интерфейсе и поддерживаемых IM-каналах установка `/goal <условие завершения>` также запускает выполнение с условием в качестве задачи; команды статуса и очистки только управляют состоянием цели.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Сложные задачи редко решаются за один проход. DeerFlow их декомпозирует.
|
||||
|
||||
Lead agent запускает sub-agents на лету, каждый со своим изолированным контекстом, инструментами и условиями завершения. Sub-agents работают параллельно, возвращают структурированные результаты, а lead agent собирает всё в единый итог.
|
||||
|
||||
Вот как DeerFlow справляется с задачами на минуты и часы: research-задача разветвляется в дюжину sub-agents, каждый копает свой угол, потом всё сходится в один отчёт, или сайт, или слайддек со сгенерированными визуалами. Один harness, много рук.
|
||||
|
||||
### Sandbox & файловая система
|
||||
|
||||
DeerFlow не просто *говорит* о том, что умеет что-то делать. У него есть собственный компьютер.
|
||||
|
||||
Каждая задача выполняется внутри изолированного Docker-контейнера с полной файловой системой: skills, workspace, uploads, outputs. Агент читает, пишет и редактирует файлы. Выполняет bash-команды и пишет код. Смотрит на изображения. Всё изолировано, всё прозрачно, никакого пересечения между сессиями.
|
||||
|
||||
Это разница между чатботом с доступом к инструментам и агентом с реальной средой выполнения.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← ваши файлы
|
||||
├── workspace/ ← рабочая директория агентов
|
||||
└── outputs/ ← результаты
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Изолированный контекст**: каждый sub-agent работает в своём контексте и не видит контекст главного агента или других sub-agents. Агент фокусируется на своей задаче.
|
||||
|
||||
**Управление контекстом**: внутри сессии DeerFlow агрессивно сжимает контекст и суммирует завершённые подзадачи, выгружает промежуточные результаты в файловую систему, сжимает то, что уже не актуально. На длинных многошаговых задачах контекстное окно не переполняется.
|
||||
|
||||
### Long-Term Memory
|
||||
|
||||
Большинство агентов забывают всё, когда диалог заканчивается. DeerFlow помнит.
|
||||
|
||||
DeerFlow сохраняет ваш профиль, предпочтения и накопленные знания между сессиями. Чем больше используете, тем лучше он вас знает: стиль, технологический стек, повторяющиеся воркфлоу. Всё хранится локально и остаётся под вашим контролем.
|
||||
|
||||
## Рекомендуемые модели
|
||||
|
||||
DeerFlow работает с любым LLM через OpenAI-совместимый API. Лучше всего — с моделями, которые поддерживают:
|
||||
|
||||
- **Большое контекстное окно** (100k+ токенов) — для deep research и многошаговых задач
|
||||
- **Reasoning capabilities** — для адаптивного планирования и сложной декомпозиции
|
||||
- **Multimodal inputs** — для работы с изображениями и видео
|
||||
- **Strong tool-use** — для надёжного вызова функций и структурированных ответов
|
||||
|
||||
## Встроенный Python-клиент
|
||||
|
||||
DeerFlow можно использовать как Python-библиотеку прямо в коде — без запуска HTTP-сервисов. `DeerFlowClient` даёт доступ ко всем возможностям агента и Gateway, возвращает те же схемы ответов, что и HTTP Gateway API. HTTP Gateway также предоставляет `DELETE /api/threads/{thread_id}` для удаления локальных данных треда, управляемых DeerFlow, после того как сам LangGraph thread был удалён:
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
client.set_goal("thread-1", "finish the implementation and make all tests pass")
|
||||
client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
|
||||
client.clear_goal("thread-1")
|
||||
```
|
||||
|
||||
## Запланированные задачи (Scheduled Tasks)
|
||||
|
||||
Теперь в DeerFlow есть первоклассный MVP запланированных задач (scheduled-task) в workspace.
|
||||
|
||||
Текущие возможности MVP:
|
||||
|
||||
- Управление задачами на `/workspace/scheduled-tasks`
|
||||
- Выбор: каждая запланированная задача переиспользует тред или создаёт новый тред для каждого запуска
|
||||
- Поддержка расписаний `once` и `cron`
|
||||
- Фоновые запланированные запуски выполняются как неинтерактивные запуски DeerFlow (`ask_clarification` там не предоставляется)
|
||||
- При совпадении наступившего cron-запуска с активным запуском на том же переиспользуемом треде применяется поведение перекрытия `skip`
|
||||
- Приостановка, возобновление, ручной запуск, просмотр истории и удаление задач
|
||||
- Запланированные задачи выполняются через стандартный жизненный цикл запуска DeerFlow
|
||||
|
||||
Текущие ограничения MVP:
|
||||
|
||||
- Пока нет инструмента `schedule_task`, создающего задачи в диалоге
|
||||
- Нет заданий с текстовыми уведомлениями
|
||||
- Нет каналов или целей отправки GitHub
|
||||
- В этой первой версии нет типа расписания `interval`
|
||||
|
||||
Включите фоновый опрос через `config.yaml -> scheduler.enabled`. Ручной запуск использует тот же ресурс и путь выполнения scheduled-task.
|
||||
|
||||
## Терминальная панель (TUI)
|
||||
|
||||
`deerflow` — это нативная терминальная панель для тех, кто живёт в шелле. Она работает **встроенной** поверх `DeerFlowClient` — без Gateway, фронтенда, nginx или Docker — и при этом учитывает те же настройки `config.yaml`, checkpointer, skills, memory, MCP и sandbox, что и остальной DeerFlow.
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # опциональная зависимость 'textual'
|
||||
|
||||
deerflow # запустить терминальный UI (требуется TTY)
|
||||
deerflow --continue # возобновить последний тред
|
||||
deerflow --resume THREAD # возобновить тред по id
|
||||
deerflow --print "summarize this repo" # автономный разовый ответ в stdout
|
||||
deerflow --json "hello" # автономный режим, StreamEvents с разделением новой строкой
|
||||
```
|
||||
|
||||
Интерфейс чата с управлением с клавиатуры: потоковый транскрипт (ответы рендерятся в Markdown), компактные карточки активности инструментов, палитра слэш-команд `/`, управление целями `/goal`, селекторы `/model` и `/threads`, история ввода, а также прерывание через `Esc` / `Ctrl+C`. Сессии, открытые в TUI, также появляются в боковой панели веб-интерфейса — TUI пишет в общее хранилище тредов под локальным пользователем по умолчанию, поэтому терминал и веб остаются синхронизированными **без запуска Gateway**.
|
||||
|
||||
Полное руководство — в [backend/docs/TUI.md](backend/docs/TUI.md).
|
||||
|
||||
## Документация
|
||||
|
||||
- [Руководство по участию](CONTRIBUTING.md) — настройка среды разработки, воркфлоу и гайдлайны
|
||||
- [Руководство по конфигурации](backend/docs/CONFIGURATION.md) — инструкции по настройке
|
||||
- [Обзор архитектуры](backend/CLAUDE.md) — технические детали
|
||||
- [Архитектура бэкенда](backend/README.md) — бэкенд и справочник API
|
||||
|
||||
## ⚠️ Безопасность
|
||||
|
||||
### Неправильное развёртывание может привести к угрозам безопасности
|
||||
|
||||
DeerFlow обладает ключевыми высокопривилегированными возможностями, включая **выполнение системных команд, операции с ресурсами и вызов бизнес-логики**. По умолчанию он рассчитан на **развёртывание в локальной доверенной среде (доступ только через loopback-адрес 127.0.0.1)**. Если вы разворачиваете агент в недоверенных средах — локальных сетях, публичных облачных серверах или других окружениях, доступных с нескольких устройств — без строгих мер безопасности, это может привести к следующим угрозам:
|
||||
|
||||
- **Несанкционированные вызовы**: функциональность агента может быть обнаружена неавторизованными третьими лицами или вредоносными сканерами, что приведёт к массовым несанкционированным запросам с выполнением высокорисковых операций (системные команды, чтение/запись файлов) и серьёзным последствиям для безопасности.
|
||||
- **Юридические и compliance-риски**: если агент будет незаконно использован для кибератак, кражи данных или других противоправных действий, это может повлечь юридическую ответственность и compliance-риски.
|
||||
|
||||
### Рекомендации по безопасности
|
||||
|
||||
**Примечание: настоятельно рекомендуем развёртывать DeerFlow только в локальной доверенной сети.** Если вам необходимо развёртывание через несколько устройств или сетей, обязательно реализуйте строгие меры безопасности, например:
|
||||
|
||||
- **Белый список IP-адресов**: используйте `iptables` или аппаратные межсетевые экраны / коммутаторы с ACL, чтобы **настроить правила белого списка IP** и заблокировать доступ со всех остальных адресов.
|
||||
- **Шлюз аутентификации**: настройте обратный прокси (nginx и др.) и **включите строгую предварительную аутентификацию**, запрещающую любой доступ без авторизации.
|
||||
- **Сетевая изоляция**: по возможности разместите агент и доверенные устройства в **одном выделенном VLAN**, изолированном от остальной сети.
|
||||
- **Следите за обновлениями**: регулярно отслеживайте обновления безопасности проекта DeerFlow.
|
||||
|
||||
## Участие в разработке
|
||||
|
||||
Приветствуем контрибьюторов! Настройка среды разработки, воркфлоу и гайдлайны — в [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## Лицензия
|
||||
|
||||
Проект распространяется под [лицензией MIT](./LICENSE).
|
||||
|
||||
## Благодарности
|
||||
|
||||
DeerFlow стоит на плечах open-source сообщества. Спасибо всем проектам и разработчикам, чья работа сделала его возможным.
|
||||
|
||||
Отдельная благодарность:
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** — фреймворк для взаимодействия с LLM и построения цепочек.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** — многоагентная оркестрация, на которой держатся сложные воркфлоу DeerFlow.
|
||||
|
||||
### Ключевые контрибьюторы
|
||||
|
||||
Авторы DeerFlow, без которых проекта бы не было:
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
## История звёзд
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+789
@@ -0,0 +1,789 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | 中文 | [日本語](./README_ja.md) | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> 2026 年 2 月 28 日，DeerFlow 2 发布后登上 GitHub Trending 第 1 名。非常感谢社区的支持，这是大家一起做到的。
|
||||
|
||||
DeerFlow（**D**eep **E**xploration and **E**fficient **R**esearch **Flow**）是一个开源的 **super agent harness**。它把 **sub-agents**、**memory** 和 **sandbox** 组织在一起，再配合可扩展的 **skills**，让 agent 可以完成几乎任何事情。
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 是一次彻底重写。** 它和 v1 没有共用代码。如果你要找的是最初的 Deep Research 框架，可以前往 [`1.x` 分支](https://github.com/bytedance/deer-flow/tree/main-1.x)。那里仍然欢迎贡献；当前的主要开发已经转向 2.0。
|
||||
|
||||
## 官网
|
||||
|
||||
想了解更多，或者直接看**真实演示**，可以访问[**官网**](https://deerflow.tech)。
|
||||
|
||||
## 字节跳动火山引擎方舟 Coding Plan
|
||||
|
||||
- 我们推荐使用 Doubao-Seed-2.0-Code、DeepSeek v3.2 和 Kimi 2.5 运行 DeerFlow
|
||||
- [现在就加入 Coding Plan](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [海外地区的开发者请点击这里](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow 新近集成了 BytePlus 自研的智能搜索与抓取工具集——[InfoQuest（支持免费在线体验）](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
## 目录
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [官网](#官网)
|
||||
- [字节跳动火山引擎方舟 Coding Plan](#字节跳动火山引擎方舟-coding-plan)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [目录](#目录)
|
||||
- [一句话交给 Coding Agent 安装](#一句话交给-coding-agent-安装)
|
||||
- [快速开始](#快速开始)
|
||||
- [配置](#配置)
|
||||
- [运行应用](#运行应用)
|
||||
- [部署建议与资源规划](#部署建议与资源规划)
|
||||
- [方式一：Docker（推荐）](#方式一docker推荐)
|
||||
- [方式二：本地开发](#方式二本地开发)
|
||||
- [进阶配置](#进阶配置)
|
||||
- [Sandbox 模式](#sandbox-模式)
|
||||
- [MCP Server](#mcp-server)
|
||||
- [IM 渠道](#im-渠道)
|
||||
- [LangSmith 链路追踪](#langsmith-链路追踪)
|
||||
- [Langfuse 链路追踪](#langfuse-链路追踪)
|
||||
- [同时使用两种追踪服务](#同时使用两种追踪服务)
|
||||
- [从 Deep Research 到 Super Agent Harness](#从-deep-research-到-super-agent-harness)
|
||||
- [核心特性](#核心特性)
|
||||
- [Skills 与 Tools](#skills-与-tools)
|
||||
- [Claude Code 集成](#claude-code-集成)
|
||||
- [Session Goals](#session-goals)
|
||||
- [手动上下文压缩](#手动上下文压缩)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox 与文件系统](#sandbox-与文件系统)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [长期记忆](#长期记忆)
|
||||
- [推荐模型](#推荐模型)
|
||||
- [内嵌 Python Client](#内嵌-python-client)
|
||||
- [定时任务 (Scheduled Tasks)](#定时任务-scheduled-tasks)
|
||||
- [终端工作台 (TUI)](#终端工作台-tui)
|
||||
- [文档](#文档)
|
||||
- [⚠️ 安全使用](#️-安全使用)
|
||||
- [参与贡献](#参与贡献)
|
||||
- [许可证](#许可证)
|
||||
- [致谢](#致谢)
|
||||
- [核心贡献者](#核心贡献者)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## 一句话交给 Coding Agent 安装
|
||||
|
||||
如果你在用 Claude Code、Codex、Cursor、Windsurf 或其他 coding agent，可以直接把下面这句话发给它：
|
||||
|
||||
```text
|
||||
如果还没 clone DeerFlow，就先 clone，然后按照 https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md 把它的本地开发环境初始化好
|
||||
```
|
||||
|
||||
这条提示词是给 coding agent 用的。它会在需要时先 clone 仓库，优先选择 Docker，完成初始化，并在结束时告诉你下一条启动命令，以及还缺哪些配置需要你补充。
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 配置
|
||||
|
||||
1. **克隆 DeerFlow 仓库**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **运行安装向导（推荐）**
|
||||
|
||||
在项目根目录（`deer-flow/`）执行：
|
||||
|
||||
```bash
|
||||
make setup
|
||||
```
|
||||
|
||||
这会启动一个交互式向导，引导你选择 LLM provider、可选的 web 搜索工具，以及 sandbox 模式、bash 权限、文件写入等执行/安全偏好。它会生成一份最小化的 `config.yaml`，并把 API key 写入 `.env`，大约 2 分钟完成。
|
||||
|
||||
随时可以运行 `make doctor` 检查配置和系统环境，并获得可执行的修复建议。
|
||||
如果你要提交本地安装、配置或运行问题，可以执行 `make support-bundle`。
|
||||
命令会直接打印 reporter 下一步建议，并在 `.deer-flow/support-bundles/` 下生成
|
||||
`*-issue-summary.md`、面向 AI 辅助提 issue 的 `*-issue-draft.md`，以及可选证据
|
||||
zip。提交 GitHub issue 时，先把 `*-issue-summary.md` 粘贴到 issue 正文；如果由
|
||||
AI 助手代填 issue，就从 `*-issue-draft.md` 开始，并先替换所有 REQUIRED 占位符，
|
||||
不要编造未知事实。只有维护者要求证据包，或摘要不足以诊断时，再附上 zip。维护者
|
||||
或 AI 辅助 triage 可以优先读取 `triage.json`；bundle 只包含脱敏后的诊断信息和
|
||||
文件 manifest，不包含 `.env`、原始对话消息或用户文件内容；提交前仍建议自己快速
|
||||
检查一遍。
|
||||
|
||||
> **进阶 / 手动配置**：如果你更想直接编辑 `config.yaml`，可以改用 `make config` 复制完整的示例模板。完整参考见 `config.example.yaml`，其中包含 CLI-backed provider（Codex CLI、Claude Code OAuth）、OpenRouter、Responses API 等更多配置。
|
||||
|
||||
<details>
|
||||
<summary>手动模型配置示例</summary>
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
```
|
||||
|
||||
OpenRouter 以及类似的 OpenAI 兼容网关，建议通过 `langchain_openai:ChatOpenAI` 配合 `base_url` 来配置。如果你更想用 provider 自己的环境变量名，也可以直接把 `api_key` 指向对应变量，例如 `api_key: $OPENROUTER_API_KEY`。
|
||||
|
||||
如果要让 OpenAI 模型走 `/v1/responses`，继续使用 `langchain_openai:ChatOpenAI`，并设置 `use_responses_api: true` 和 `output_version: responses/v1`。
|
||||
|
||||
对于 vLLM 0.19.0，请使用 `deerflow.models.vllm_provider:VllmChatModel`。对于 Qwen 风格的推理模型，DeerFlow 通过 `extra_body.chat_template_kwargs.enable_thinking` 开关推理，并在多轮 tool-call 对话中保留 vLLM 非标准的 `reasoning` 字段。旧版 `thinking` 配置会自动规范化以保持向后兼容。推理模型可能还需要在启动 vLLM 服务时加上 `--reasoning-parser ...` 参数。如果你的本地 vLLM 部署接受任意非空 API key，可以把 `VLLM_API_KEY` 设为一个占位值。
|
||||
|
||||
CLI-backed provider 配置示例：
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI 会读取 `~/.codex/auth.json`
|
||||
- Claude Code 支持 `CLAUDE_CODE_OAUTH_TOKEN`、`ANTHROPIC_AUTH_TOKEN`、`CLAUDE_CODE_CREDENTIALS_PATH`，或 `~/.claude/.credentials.json`
|
||||
- ACP agent 条目与 model provider 是分开配置的——如果你配置了 `acp_agents.codex`，请把它指向一个 Codex ACP 适配器，例如 `npx -y @zed-industries/codex-acp`
|
||||
- 在 macOS 上，如有需要可显式导出 Claude Code 的认证信息：
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
API key 也可以手动写入 `.env` 文件（推荐）或在 shell 中导出：
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### 运行应用
|
||||
|
||||
#### 部署建议与资源规划
|
||||
|
||||
可以先按下面的资源档位来选择 DeerFlow 的运行方式：
|
||||
|
||||
| 部署场景 | 起步配置 | 推荐配置 | 说明 |
|
||||
|---------|-----------|------------|-------|
|
||||
| 本地体验 / `make dev` | 4 vCPU、8 GB 内存、20 GB SSD 可用空间 | 8 vCPU、16 GB 内存 | 适合单个开发者或单个轻量会话，且模型走外部 API。`2 核 / 4 GB` 通常跑不稳。 |
|
||||
| Docker 开发 / `make docker-start` | 4 vCPU、8 GB 内存、25 GB SSD 可用空间 | 8 vCPU、16 GB 内存 | 镜像构建、源码挂载和 sandbox 容器都会比纯本地模式更吃资源。 |
|
||||
| 长期运行服务 / `make up` | 8 vCPU、16 GB 内存、40 GB SSD 可用空间 | 16 vCPU、32 GB 内存 | 更适合共享环境、多 agent 任务、报告生成或更重的 sandbox 负载。 |
|
||||
|
||||
- 上面的配置只覆盖 DeerFlow 本身；如果你还要本机部署本地大模型，请单独为模型服务预留资源。
|
||||
- 持续运行的服务更推荐使用 Linux + Docker。macOS 和 Windows 更适合作为开发机或体验环境。
|
||||
- 如果 CPU 或内存长期打满，先降低并发会话或重任务数量，再考虑升级到更高一档配置。
|
||||
|
||||
#### 方式一：Docker（推荐）
|
||||
|
||||
**开发模式**（支持热更新，挂载源码）：
|
||||
|
||||
```bash
|
||||
make docker-init # 拉取 sandbox 镜像（首次运行或镜像更新时执行）
|
||||
make docker-start # 启动服务（会根据 config.yaml 自动判断 sandbox 模式）
|
||||
```
|
||||
|
||||
如果 `config.yaml` 使用的是 provisioner 模式（`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` 且配置了 `provisioner_url`），`make docker-start` 才会启动 `provisioner`。
|
||||
|
||||
**生产模式**（本地构建镜像，并挂载运行期配置与数据）：
|
||||
|
||||
```bash
|
||||
make up # 构建镜像并启动全部生产服务
|
||||
make down # 停止并移除容器
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 当前 Agent 运行时嵌入在 Gateway 中运行，`/api/langgraph/*` 会由 nginx 重写到 Gateway 的 LangGraph-compatible API。
|
||||
|
||||
访问地址：http://localhost:2026
|
||||
|
||||
更完整的 Docker 开发说明见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
||||
|
||||
#### 方式二：本地开发
|
||||
|
||||
如果你更希望直接在本地启动各个服务：
|
||||
|
||||
前提：先完成上面的“配置”步骤（`make setup`）。`make dev` 需要有效配置文件，默认读取项目根目录下的 `config.yaml`。可以用 `DEER_FLOW_PROJECT_ROOT` 显式指定项目根目录，也可以用 `DEER_FLOW_CONFIG_PATH` 指向某个具体配置文件。运行期状态默认写到项目根目录下的 `.deer-flow`，可用 `DEER_FLOW_HOME` 覆盖；skills 默认读取项目根目录下的 `skills/`，可用 `DEER_FLOW_SKILLS_PATH` 覆盖。启动前先运行 `make doctor` 校验配置。
|
||||
在 Windows 上，请使用 Git Bash 运行本地开发流程。基于 bash 的服务脚本不支持直接在原生 `cmd.exe` 或 PowerShell 中执行，且 WSL 也不保证可用，因为部分脚本依赖 Git for Windows 的 `cygpath` 等工具。
|
||||
|
||||
1. **检查依赖环境**：
|
||||
```bash
|
||||
make check # 校验 Node.js 22+、pnpm、uv、nginx
|
||||
```
|
||||
|
||||
2. **安装依赖**：
|
||||
```bash
|
||||
make install # 安装 backend + frontend 依赖
|
||||
```
|
||||
|
||||
3. **（可选）预拉取 sandbox 镜像**：
|
||||
```bash
|
||||
# 如果使用 Docker / Container sandbox，建议先执行
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **启动服务**：
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **访问地址**：http://localhost:2026
|
||||
|
||||
### 进阶配置
|
||||
#### Sandbox 模式
|
||||
|
||||
DeerFlow 支持多种 sandbox 执行方式：
|
||||
- **本地执行**（直接在宿主机上运行 sandbox 代码）
|
||||
- **Docker 执行**（在隔离的 Docker 容器里运行 sandbox 代码）
|
||||
- **Docker + Kubernetes 执行**（通过 provisioner 服务在 Kubernetes Pod 中运行 sandbox 代码）
|
||||
|
||||
Docker 开发时，服务启动行为会遵循 `config.yaml` 里的 sandbox 模式。在 Local / Docker 模式下，不会启动 `provisioner`。
|
||||
|
||||
如果要配置你自己的模式，参见 [Sandbox 配置指南](backend/docs/CONFIGURATION.md#sandbox)。
|
||||
|
||||
#### MCP Server
|
||||
|
||||
DeerFlow 支持可配置的 MCP Server 和 skills，用来扩展能力。
|
||||
对于 HTTP/SSE MCP Server，还支持 OAuth token 流程（`client_credentials`、`refresh_token`）。
|
||||
详细说明见 [MCP Server 指南](backend/docs/MCP_SERVER.md)。
|
||||
|
||||
#### IM 渠道
|
||||
|
||||
DeerFlow 支持从即时通讯应用接收任务。只要配置完成，对应渠道会自动启动，而且都不需要公网 IP。
|
||||
|
||||
DeerFlow 还可以在 workspace UI 里暴露用户自有的 IM 渠道连接。启用 `channel_connections` 后，已登录用户可以从侧边栏 / Settings > Channels 绑定 Telegram、Slack、Discord、Feishu/Lark、DingTalk、WeChat 或 WeCom。它复用现有的 `channels.*` 出站传输，因此不需要公网 IP 或 provider 回调地址。入站 IM 消息会以所连接的 DeerFlow 用户身份运行。设置和安全注意事项参见 [IM Channel Connections](backend/docs/IM_CHANNEL_CONNECTIONS.md)。
|
||||
|
||||
| 渠道 | 传输方式 | 上手难度 |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API（long-polling） | 简单 |
|
||||
| Slack | Socket Mode | 中等 |
|
||||
| Feishu / Lark | WebSocket | 中等 |
|
||||
| WeChat | Tencent iLink（long-polling） | 中等 |
|
||||
| 企业微信智能机器人 | WebSocket | 中等 |
|
||||
| 钉钉 | Stream Push（WebSocket） | 中等 |
|
||||
|
||||
**`config.yaml` 中的配置示例：**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL（默认：http://localhost:8001/api）
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# Gateway API URL（默认：http://localhost:8001）
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# 可选：所有移动端渠道共用的全局 session 默认值
|
||||
session:
|
||||
assistant_id: lead_agent # 也可以填自定义 agent 名；渠道层会自动转换为 lead_agent + agent_name
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # 国内版（默认）
|
||||
# domain: https://open.larksuite.com # 国际版
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-...（Socket Mode）
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
|
||||
# 可选：按渠道 / 按用户单独覆盖 session 配置
|
||||
session:
|
||||
assistant_id: mobile-agent # 这里同样支持自定义 agent 名
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip-agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # 可选：bot_token 缺失时允许首次扫码登录引导
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # 钉钉开放平台 ClientId
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # 钉钉开放平台 ClientSecret
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
card_template_id: "" # 可选：AI 卡片模板 ID，用于流式打字机效果
|
||||
```
|
||||
|
||||
说明：
|
||||
- `assistant_id: lead_agent` 会直接调用默认的 LangGraph assistant。
|
||||
- 如果 `assistant_id` 填的是自定义 agent 名，DeerFlow 仍然会走 `lead_agent`，同时把该值注入为 `agent_name`，这样 IM 渠道也会生效对应 agent 的 SOUL 和配置。
|
||||
|
||||
在 `.env` 里设置对应的 API key：
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# 企业微信智能机器人
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# 钉钉
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegram 配置**
|
||||
|
||||
1. 打开 [@BotFather](https://t.me/BotFather)，发送 `/newbot`，复制生成的 HTTP API token。
|
||||
2. 在 `.env` 中设置 `TELEGRAM_BOT_TOKEN`，并在 `config.yaml` 里启用该渠道。
|
||||
|
||||
**Slack 配置**
|
||||
|
||||
1. 前往 [api.slack.com/apps](https://api.slack.com/apps) 创建 Slack App：Create New App → From scratch。
|
||||
2. 在 **OAuth & Permissions** 中添加 Bot Token Scopes：`app_mentions:read`、`chat:write`、`im:history`、`im:read`、`im:write`、`files:write`。
|
||||
3. 启用 **Socket Mode**，生成带 `connections:write` 权限的 App-Level Token（`xapp-...`）。
|
||||
4. 在 **Event Subscriptions** 中订阅 bot events：`app_mention`、`message.im`。
|
||||
5. 在 `.env` 中设置 `SLACK_BOT_TOKEN` 和 `SLACK_APP_TOKEN`，并在 `config.yaml` 中启用该渠道。
|
||||
|
||||
**Feishu / Lark 配置**
|
||||
|
||||
1. 在 [飞书开放平台](https://open.feishu.cn/) 创建应用，并启用 **Bot** 能力。
|
||||
2. 添加权限：`im:message`、`im:message.p2p_msg:readonly`、`im:resource`。
|
||||
3. 在 **事件订阅** 中订阅 `im.message.receive_v1`，连接方式选择 **长连接**。
|
||||
4. 复制 App ID 和 App Secret，在 `.env` 中设置 `FEISHU_APP_ID` 和 `FEISHU_APP_SECRET`，并在 `config.yaml` 中启用该渠道。
|
||||
|
||||
**WeChat 配置**
|
||||
|
||||
1. 在 `config.yaml` 中启用 `wechat` 渠道。
|
||||
2. 在 `.env` 中设置 `WECHAT_BOT_TOKEN`，或者把 `qrcode_login_enabled` 设为 `true` 以便首次扫码登录引导。
|
||||
3. 当 `bot_token` 缺失且启用了扫码引导时，留意后端日志里 iLink 返回的二维码内容，并完成绑定流程。
|
||||
4. 扫码流程成功后，DeerFlow 会把获取到的 token 持久化到 `state_dir`，便于后续重启复用。
|
||||
5. Docker Compose 部署时，请把 `state_dir` 放在持久化卷上，这样 `get_updates_buf` 游标和已保存的登录状态才能在重启后保留。
|
||||
|
||||
**企业微信智能机器人配置**
|
||||
|
||||
1. 在企业微信智能机器人平台创建机器人，获取 `bot_id` 和 `bot_secret`。
|
||||
2. 在 `config.yaml` 中启用 `channels.wecom`，并填入 `bot_id` / `bot_secret`。
|
||||
3. 在 `.env` 中设置 `WECOM_BOT_ID` 和 `WECOM_BOT_SECRET`。
|
||||
4. 安装后端依赖时确保包含 `wecom-aibot-python-sdk`，渠道会通过 WebSocket 长连接接收消息，无需公网回调地址。
|
||||
5. 当前支持文本、图片和文件入站消息；agent 生成的最终图片/文件也会回传到企业微信会话中。
|
||||
|
||||
**钉钉配置**
|
||||
|
||||
1. 在 [钉钉开放平台](https://open.dingtalk.com/) 创建应用，并启用 **机器人** 能力。
|
||||
2. 在机器人配置页面设置消息接收模式为 **Stream模式**。
|
||||
3. 复制 `Client ID` 和 `Client Secret`，在 `.env` 中设置 `DINGTALK_CLIENT_ID` 和 `DINGTALK_CLIENT_SECRET`，并在 `config.yaml` 中启用该渠道。
|
||||
4. *（可选）* 如需开启流式 AI 卡片回复（打字机效果），请在[钉钉卡片平台](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card)创建 **AI 卡片**模板，然后在 `config.yaml` 中将 `card_template_id` 设为该模板 ID。同时需要申请 `Card.Streaming.Write` 和 `Card.Instance.Write` 权限。
|
||||
|
||||
**命令**
|
||||
|
||||
渠道连接完成后，你可以直接在聊天窗口里和 DeerFlow 交互：
|
||||
|
||||
| 命令 | 说明 |
|
||||
|---------|-------------|
|
||||
| `/new` | 开启新对话 |
|
||||
| `/status` | 查看当前 thread 信息 |
|
||||
| `/models` | 列出可用模型 |
|
||||
| `/memory` | 查看 memory |
|
||||
| `/help` | 查看帮助 |
|
||||
|
||||
> 没有命令前缀的消息会被当作普通聊天处理。DeerFlow 会自动创建 thread，并以对话方式回复。
|
||||
|
||||
#### LangSmith 链路追踪
|
||||
|
||||
DeerFlow 内置了 [LangSmith](https://smith.langchain.com) 集成，用于可观测性。启用后，所有 LLM 调用、agent 运行和工具执行都会被追踪，并在 LangSmith 仪表盘中展示。
|
||||
|
||||
在 `.env` 文件中添加以下配置：
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
#### Langfuse 链路追踪
|
||||
|
||||
DeerFlow 同样支持 [Langfuse](https://langfuse.com) 可观测性，适用于兼容 LangChain 的运行。
|
||||
|
||||
在 `.env` 文件中添加以下配置：
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
如果你使用自托管的 Langfuse 实例，请将 `LANGFUSE_BASE_URL` 设置为你的部署地址。
|
||||
|
||||
**链路关联字段。** 每次 agent 运行都会标注 Langfuse 的保留追踪属性，这样 Sessions 和 Users 页面就能自动填充数据：
|
||||
|
||||
- `session_id` = LangGraph 的 `thread_id`——将同一会话的所有 trace 归为一组
|
||||
- `user_id` = 来自 `get_effective_user_id()` 的有效用户（在无鉴权模式下回退为 `default`）
|
||||
- `trace_name` = assistant id（默认为 `lead-agent`）
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]`（未设置时省略）
|
||||
- `metadata.deerflow_trace_id` = DeerFlow 的请求关联 id，当启用请求链路关联（request trace correlation）时与 `X-Trace-Id` 一致
|
||||
|
||||
这些字段会在图（graph）调用的根部注入到 `RunnableConfig.metadata`，同时覆盖 gateway 路径（`runtime/runs/worker.py::run_agent`）和内嵌路径（`client.py::DeerFlowClient.stream`），因此任何兼容 LangChain 的 callback 都能读取到它们。设置 `DEER_FLOW_ENV`（或 `ENVIRONMENT`）可按部署环境为 trace 打标签。
|
||||
|
||||
#### 同时使用两种追踪服务
|
||||
|
||||
如果同时启用 LangSmith 和 Langfuse，DeerFlow 会挂载两个追踪 callback，并将相同的模型活动上报到两个系统。
|
||||
|
||||
如果某个 provider 被显式启用但缺少必要的凭据，或其 callback 初始化失败，DeerFlow 会在创建模型、初始化追踪时快速失败（fail fast），错误信息会指明导致失败的 provider。
|
||||
|
||||
Docker 部署时，追踪默认关闭。在 `.env` 中设置 `LANGSMITH_TRACING=true` 和 `LANGSMITH_API_KEY` 即可启用。
|
||||
|
||||
## 从 Deep Research 到 Super Agent Harness
|
||||
|
||||
DeerFlow 最初是一个 Deep Research 框架，后来社区把它一路推到了更远的地方。上线之后，开发者拿它去做的事情早就不止研究：搭数据流水线、生成演示文稿、快速起 dashboard、自动化内容流程，很多方向一开始连我们自己都没想到。
|
||||
|
||||
这让我们意识到一件事：DeerFlow 不只是一个研究工具。它更像一个 **harness**，一个真正让 agents 把事情做完的运行时基础设施。
|
||||
|
||||
所以我们把它从头重做了一遍。
|
||||
|
||||
DeerFlow 2.0 不再是一个需要你自己拼装的 framework。它是一个开箱即用、同时又足够可扩展的 super agent harness。基于 LangGraph 和 LangChain 构建，默认就带上了 agent 真正会用到的关键能力：文件系统、memory、skills、sandbox 执行环境，以及为复杂多步骤任务做规划、拉起 sub-agents 的能力。
|
||||
|
||||
你可以直接拿来用，也可以拆开重组，改成你自己的样子。
|
||||
|
||||
## 核心特性
|
||||
|
||||
### Skills 与 Tools
|
||||
|
||||
Skills 是 DeerFlow 能做“几乎任何事”的关键。
|
||||
|
||||
标准的 Agent Skill 是一种结构化能力模块，通常就是一个 Markdown 文件，里面定义了工作流、最佳实践，以及相关的参考资源。DeerFlow 自带一批内置 skills，覆盖研究、报告生成、演示文稿制作、网页生成、图像和视频生成等场景。真正有意思的地方在于它的扩展性：你可以加自己的 skills，替换内置 skills，或者把多个 skills 组合成复合工作流。
|
||||
|
||||
Skills 采用按需渐进加载，不会一次性把所有内容都塞进上下文。只有任务确实需要时才加载，这样能把上下文窗口控制得更干净，也更适合对 token 比较敏感的模型。
|
||||
|
||||
通过 Gateway 安装 `.skill` 压缩包时，DeerFlow 会接受标准的可选 frontmatter 元数据，比如 `version`、`author`、`compatibility`，不会把本来合法的外部 skill 拒之门外。
|
||||
|
||||
Tools 也是同样的思路。DeerFlow 自带一组核心工具：网页搜索、网页抓取、网页渲染截图、文件操作、bash 执行；同时也支持通过 MCP Server 和 Python 函数扩展自定义工具。你可以替换任何一项，也可以继续往里加。
|
||||
|
||||
Gateway 生成后续建议时，现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化，再去解析 JSON 数组响应，因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。
|
||||
|
||||
Web UI 支持从已完成的 assistant 回复分叉出一个新的主对话。新 thread 会从该回复对应的 checkpoint 开始，并尽力复制当前 thread 的工作区文件。
|
||||
|
||||
```text
|
||||
# sandbox 容器内的路径
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← 你的 skill
|
||||
```
|
||||
|
||||
#### Claude Code 集成
|
||||
|
||||
借助 `claude-to-deerflow` skill，你可以直接在 [Claude Code](https://docs.anthropic.com/en/docs/claude-code) 里和正在运行的 DeerFlow 实例交互。不用离开终端，就能下发研究任务、查看状态、管理 threads。
|
||||
|
||||
**安装这个 skill：**
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
然后确认 DeerFlow 已经启动（默认地址是 `http://localhost:2026`），在 Claude Code 里使用 `/claude-to-deerflow` 命令即可。
|
||||
|
||||
**你可以做的事情包括：**
|
||||
- 给 DeerFlow 发送消息，并接收流式响应
|
||||
- 选择执行模式：flash（更快）、standard、pro（规划模式）、ultra（sub-agents 模式）
|
||||
- 检查 DeerFlow 健康状态，列出 models / skills / agents
|
||||
- 管理 threads 和会话历史
|
||||
- 上传文件做分析
|
||||
|
||||
**环境变量**（可选，用于自定义端点）：
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # 统一代理基地址
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
完整 API 说明见 [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md)。
|
||||
|
||||
Web UI 输入框支持浏览器侧语音听写。浏览器提供 Web Speech API 时，麦克风按钮会把语音转写为本地草稿；DeerFlow 只接收转写后的文本，音频处理交由浏览器或操作系统语音识别服务按其环境策略完成。用户可以在发送前继续检查和编辑文本。
|
||||
|
||||
### Session Goals
|
||||
|
||||
用 `/goal <完成条件>` 为当前 thread 绑定一个激活态的完成条件。这个 goal 是 thread 维度的状态，而不是技能激活，所以它会跨轮次持续生效，直到 DeerFlow 判定它已被满足、或者你手动清除它。
|
||||
|
||||
支持的命令：
|
||||
|
||||
```text
|
||||
/goal finish the implementation and make all tests pass
|
||||
/goal # 查看当前激活的 goal
|
||||
/goal clear # 清除它
|
||||
```
|
||||
|
||||
每次 Gateway 驱动的 run 结束后，DeerFlow 会用一个 non-thinking 的评估模型，把可见的对话内容拿去和激活的 goal 比对。评估模型必须返回一个带类型的 blocker（`missing_evidence`、`needs_user_input`、`run_failed`、`external_wait` 或 `goal_not_met_yet`），并附上可见证据。只有在最近一轮 assistant 回复已被持久化 checkpoint、blocker 是 `goal_not_met_yet`、评估期间 thread 没有变化、且无进展熔断器没有触发时，DeerFlow 才会注入一次 hidden continuation。安全上限默认是 8 次 hidden continuation；连续两次相同的无进展评估后就会停止。`/goal clear` 以及任何用户手动输入的新内容，优先级都高于排队中的 continuation。当 goal 被满足时，DeerFlow 会自动清除它，并发布更新后的 thread 状态。
|
||||
|
||||
Web UI 会在输入框上方展示当前激活的 goal。同样的命令在 TUI 和受支持的 IM 渠道里也可用。在 Web UI 和受支持的 IM 渠道里，设置 `/goal <完成条件>` 还会以该条件作为任务启动一次 run；状态查询和清除命令则只管理 goal 状态本身。
|
||||
|
||||
### 手动上下文压缩
|
||||
|
||||
在 Web UI 输入框中使用 `/compact`，可以把当前 thread 的早期上下文压缩成摘要。完整聊天记录仍会保留在界面上，但后续模型调用会基于压缩摘要和最近消息继续。当前历史不足时不会压缩；thread 正在运行任务时会阻止压缩。
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
复杂任务通常不可能一次完成，DeerFlow 会先拆解，再执行。
|
||||
|
||||
lead agent 可以按需动态拉起 sub-agents。每个 sub-agent 都有自己独立的上下文、工具和终止条件。只要条件允许，它们就会并行运行，返回结构化结果，最后再由 lead agent 汇总成一份完整输出。
|
||||
|
||||
这也是 DeerFlow 能处理从几分钟到几小时任务的原因。比如一个研究任务，可以拆成十几个 sub-agents，分别探索不同方向，最后合并成一份报告，或者一个网站，或者一套带生成视觉内容的演示文稿。一个 harness，多路并行。
|
||||
|
||||
### Sandbox 与文件系统
|
||||
|
||||
DeerFlow 不只是“会说它能做”，它是真的有一台自己的“电脑”。
|
||||
|
||||
每个任务都运行在隔离的 Docker 容器里，里面有完整的文件系统，包括 skills、workspace、uploads、outputs。agent 可以读写和编辑文件，可以执行 bash 命令和代码，也可以查看图片。整个过程都在 sandbox 内完成，可审计、会隔离，不会在不同 session 之间互相污染。
|
||||
|
||||
这就是“带工具的聊天机器人”和“真正有执行环境的 agent”之间的差别。
|
||||
|
||||
```text
|
||||
# sandbox 容器内的路径
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← 你的文件
|
||||
├── workspace/ ← agents 的工作目录
|
||||
└── outputs/ ← 最终交付物
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**隔离的 Sub-Agent Context**：每个 sub-agent 都在自己独立的上下文里运行。它看不到主 agent 的上下文，也看不到其他 sub-agents 的上下文。这样做的目的很直接，就是让它只聚焦当前任务，不被无关信息干扰。
|
||||
|
||||
**摘要压缩**：在单个 session 内，DeerFlow 会比较积极地管理上下文，包括总结已完成的子任务、把中间结果转存到文件系统、压缩暂时不重要的信息。这样在长链路、多步骤任务里，它也能保持聚焦，而不会轻易把上下文窗口打爆。
|
||||
|
||||
### 长期记忆
|
||||
|
||||
大多数 agents 会在对话结束后把一切都忘掉，DeerFlow 不一样。
|
||||
|
||||
跨 session 使用时，DeerFlow 会逐步积累关于你的持久 memory，包括你的个人偏好、知识背景，以及长期沉淀下来的工作习惯。你用得越多，它越了解你的写作风格、技术栈和重复出现的工作流。memory 保存在本地，控制权也始终在你手里。
|
||||
|
||||
## 推荐模型
|
||||
|
||||
DeerFlow 对模型没有强绑定，只要实现了 OpenAI 兼容 API 的 LLM，理论上都可以接入。不过在下面这些能力上表现更强的模型，通常会更适合 DeerFlow：
|
||||
|
||||
- **长上下文窗口**（100k+ tokens），适合深度研究和多步骤任务
|
||||
- **推理能力**，适合自适应规划和复杂拆解
|
||||
- **多模态输入**，适合理解图片和视频
|
||||
- **稳定的 tool use 能力**，适合可靠的函数调用和结构化输出
|
||||
|
||||
## 内嵌 Python Client
|
||||
|
||||
DeerFlow 也可以作为内嵌的 Python 库使用，不必启动完整的 HTTP 服务。`DeerFlowClient` 提供了进程内的直接访问方式，覆盖所有 agent 和 Gateway 能力，返回的数据结构与 HTTP Gateway API 保持一致。HTTP Gateway 还提供 `DELETE /api/threads/{thread_id}`，用于在 LangGraph thread 本身被删除之后，清理 DeerFlow 托管的本地 thread 数据：
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming（LangGraph SSE 协议：values、messages-tuple、end）
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# 配置与管理：返回值与 Gateway 对齐的 dict
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
client.set_goal("thread-1", "finish the implementation and make all tests pass")
|
||||
client.get_goal("thread-1") # {"goal": {...}} or {"goal": None}
|
||||
client.clear_goal("thread-1")
|
||||
```
|
||||
|
||||
所有返回 dict 的方法都会在 CI 中通过 Gateway 的 Pydantic 响应模型校验（`TestGatewayConformance`），以确保内嵌 client 始终和 HTTP API schema 保持同步。完整 API 说明见 `backend/packages/harness/deerflow/client.py`。
|
||||
|
||||
## 定时任务 (Scheduled Tasks)
|
||||
|
||||
DeerFlow 现在在 workspace 里内置了一个一等的定时任务（scheduled-task）MVP。
|
||||
|
||||
当前 MVP 能力：
|
||||
|
||||
- 在 `/workspace/scheduled-tasks` 管理任务
|
||||
- 每个定时任务可以选择复用同一个 thread，也可以选择每次运行新建一个 thread
|
||||
- 支持 `once` 和 `cron` 两种调度方式
|
||||
- 后台定时执行以非交互式 DeerFlow run 运行（那里不会暴露 `ask_clarification`）
|
||||
- 当到期的 cron 执行与同一复用 thread 上的活跃 run 冲突时，采用 `skip` 的重叠处理策略
|
||||
- 支持暂停、恢复、手动触发、查看历史和删除任务
|
||||
- 定时任务通过正常的 DeerFlow run 生命周期执行
|
||||
|
||||
当前 MVP 限制：
|
||||
|
||||
- 暂时还没有可在对话中创建任务的 `schedule_task` 工具
|
||||
- 没有纯文本通知任务
|
||||
- 没有渠道或 GitHub 分发目标
|
||||
- 第一版没有 `interval` 调度类型
|
||||
|
||||
通过 `config.yaml -> scheduler.enabled` 开启后台轮询。手动触发使用同样的 scheduled-task 资源和执行路径。
|
||||
|
||||
## 终端工作台 (TUI)
|
||||
|
||||
`deerflow` 是一个面向终端用户的工作台，**内嵌**运行在 `DeerFlowClient` 之上——无需启动 Gateway、前端、nginx 或 Docker，同时沿用与 DeerFlow 其它部分相同的 `config.yaml`、checkpointer、技能、记忆、MCP 和沙箱配置。
|
||||
|
||||
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # 可选的 'textual' 依赖
|
||||
|
||||
deerflow # 启动终端 UI（需要 TTY）
|
||||
deerflow --continue # 恢复最近一次会话
|
||||
deerflow --resume THREAD # 按 id 恢复指定会话
|
||||
deerflow --print "总结一下这个仓库" # 无头模式，结果打印到 stdout
|
||||
deerflow --json "hello" # 无头模式，输出按行分隔的 StreamEvent
|
||||
```
|
||||
|
||||
键盘驱动的对话界面：流式渲染的对话区（回答按 Markdown 渲染）、紧凑的工具活动卡片、`/` 斜杠命令面板、`/model` 与 `/threads` 选择器、输入历史，以及 `Esc` / `Ctrl+C` 打断。在 TUI 里开启的会话也会出现在 Web UI 侧边栏——它会以本地默认用户身份写入共享的会话存储，因此终端与网页保持同步，**无需运行 Gateway**。
|
||||
|
||||
完整说明见 [backend/docs/TUI.md](backend/docs/TUI.md)。
|
||||
|
||||
## 文档
|
||||
|
||||
- [贡献指南](CONTRIBUTING.md) - 开发环境搭建与协作流程
|
||||
- [配置指南](backend/docs/CONFIGURATION.md) - 安装与配置说明
|
||||
- [架构概览](backend/CLAUDE.md) - 技术架构说明
|
||||
- [后端架构](backend/README.md) - 后端架构与 API 参考
|
||||
|
||||
## ⚠️ 安全使用
|
||||
|
||||
### 不恰当的部署可能导致安全风险
|
||||
|
||||
DeerFlow 具备**系统指令执行、资源操作、业务逻辑调用**等关键高权限能力，默认设计为**部署在本地可信环境（仅本机 127.0.0.1 回环访问）**。若您将 agent 部署至不可信局域网、公网云服务器等可被多终端访问的网络环境，且未采取严格的安全防护措施，可能导致安全风险，例如：
|
||||
|
||||
- **未授权的非法调用**：agent 功能被未授权的第三方、公网恶意扫描程序探测到，进而发起批量非法调用请求，执行系统命令、文件读写等高危操作，可能导致安全后果。
|
||||
- **合规与法律风险**：若 agent 被非法调用用于实施网络攻击、信息窃取等违法违规行为，可能产生法律责任与合规风险。
|
||||
|
||||
### 安全使用建议
|
||||
|
||||
**注意：建议您将 DeerFlow 部署在本地可信的网络环境下。** 若您有跨设备、跨网络的部署需求，必须加入严格的安全措施。例如，采取如下手段：
|
||||
|
||||
- **设置访问 IP 白名单**：使用 `iptables`，或部署硬件防火墙 / 带访问控制（ACL）功能的交换机等，**配置规则设置 IP 白名单**，拒绝其他所有 IP 进行访问。
|
||||
- **前置身份验证**：配置反向代理（nginx 等），并**开启高强度的前置身份验证功能**，禁止无任何身份验证的访问。
|
||||
- **网络隔离**：若有可能，建议将 agent 和可信设备划分到**同一个专用 VLAN**，与其他网络设备做隔离。
|
||||
- **持续关注项目更新**：请持续关注 DeerFlow 项目的安全功能更新。
|
||||
|
||||
## 参与贡献
|
||||
|
||||
欢迎参与贡献。开发环境、工作流和相关规范见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
||||
|
||||
目前回归测试已经覆盖 Docker sandbox 模式识别，以及 `backend/tests/` 中 provisioner kubeconfig-path 处理相关测试。
|
||||
|
||||
## 许可证
|
||||
|
||||
本项目采用 [MIT License](./LICENSE) 开源发布。
|
||||
|
||||
## 致谢
|
||||
|
||||
DeerFlow 建立在开源社区大量优秀工作的基础上。所有让 DeerFlow 成为可能的项目和贡献者，我们都心怀感谢。毫不夸张地说，我们是站在巨人的肩膀上继续往前走。
|
||||
|
||||
特别感谢以下项目带来的关键支持：
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**：它们提供的优秀框架支撑了我们的 LLM 交互与 chains，让整体集成和能力编排顺畅可用。
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**：它们在多 agent 编排上的创新方式，是 DeerFlow 复杂工作流得以成立的重要基础。
|
||||
|
||||
这些项目体现了开源协作真正的力量，我们也很高兴能继续建立在这些基础之上。
|
||||
|
||||
### 核心贡献者
|
||||
|
||||
感谢 `DeerFlow` 的核心作者，是他们的判断、投入和持续推进，才让这个项目真正落地：
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
# Releasing DeerFlow
|
||||
|
||||
DeerFlow releases are **tag-driven**: pushing a `v*` git tag triggers the
|
||||
publishing workflows. There is no separate release script that bumps versions —
|
||||
the maintainer bumps the version sources, updates the changelog, commits, and
|
||||
tags. The helper scripts below keep the version sources in lockstep, and CI
|
||||
gates the release on them agreeing with the tag.
|
||||
|
||||
## Version sources
|
||||
|
||||
A release version must appear, identically, in four places:
|
||||
|
||||
| File | Field |
|
||||
| -------------------------------------- | -------------------- |
|
||||
| `backend/pyproject.toml` | `version = "X.Y.Z"` |
|
||||
| `frontend/package.json` | `"version": "X.Y.Z"` |
|
||||
| `deploy/helm/deer-flow/Chart.yaml` | `version: X.Y.Z` |
|
||||
| `deploy/helm/deer-flow/Chart.yaml` | `appVersion: "X.Y.Z"`|
|
||||
|
||||
Plus the git tag `vX.Y.Z` itself, which is the canonical release identifier.
|
||||
|
||||
Container images are tagged from the git tag (not from these files), and the
|
||||
Helm chart version is validated against the tag — so if any source lags the
|
||||
tag, the release is blocked (see [Version gate](#version-gate)).
|
||||
|
||||
The frontend's in-app About page (Settings ▸ About) is a *derived* consumer, not
|
||||
a fifth source: it reads `frontend/package.json`'s version at build time, so it
|
||||
tracks the table above automatically with no bump needed. Nightly builds override
|
||||
it with the chart's nightly string (`<base>-nightly.<YYYYMMDD>-<short_sha>`) via
|
||||
the `APP_VERSION` build-arg in `nightly.yaml`, so a nightly image's About page
|
||||
distinguishes it from a release.
|
||||
|
||||
## Helper scripts
|
||||
|
||||
- `scripts/bump_version.sh <version>` — set all four fields at once, then
|
||||
self-verify. Tolerates a leading `v` (e.g. `v2.2.0`).
|
||||
```bash
|
||||
scripts/bump_version.sh 2.2.0
|
||||
```
|
||||
- `scripts/verify_versions.sh [version]` — check that all sources agree. With
|
||||
no argument it requires mutual equality; with an argument it requires every
|
||||
source to equal it. Exits non-zero on mismatch. Run it locally before tagging
|
||||
to catch drift early:
|
||||
```bash
|
||||
scripts/verify_versions.sh 2.2.0
|
||||
```
|
||||
|
||||
## Release procedure
|
||||
|
||||
1. **Bump the version** across all sources:
|
||||
```bash
|
||||
scripts/bump_version.sh 2.2.0
|
||||
```
|
||||
2. **Update `CHANGELOG.md`**: rename the `## [Unreleased]` section to
|
||||
`## [2.2.0] — YYYY-MM-DD` (note the em dash `—`), and add a link reference
|
||||
at the bottom of the file:
|
||||
```
|
||||
[2.2.0]: https://github.com/bytedance/deer-flow/releases/tag/v2.2.0
|
||||
```
|
||||
Start a fresh `## [Unreleased]` section above it for the next cycle.
|
||||
3. **Commit** the version + changelog changes:
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "release: v2.2.0"
|
||||
```
|
||||
4. **Tag and push**:
|
||||
```bash
|
||||
git tag v2.2.0
|
||||
git push origin v2.2.0
|
||||
```
|
||||
Pushing the tag triggers the publishing workflows (below).
|
||||
|
||||
## What CI publishes on a `v*` tag
|
||||
|
||||
- `.github/workflows/container.yaml` — builds and pushes `backend`,
|
||||
`frontend`, and `provisioner` images to `ghcr.io`, tagged with the release
|
||||
version (and `latest` on the default branch).
|
||||
- `.github/workflows/chart.yaml` — packages the Helm chart and pushes it as an
|
||||
OCI artifact to `ghcr.io`. Users install with:
|
||||
```bash
|
||||
helm install deer-flow oci://ghcr.io/<owner>/deer-flow --version 2.2.0
|
||||
```
|
||||
|
||||
## Nightly builds
|
||||
|
||||
`.github/workflows/nightly.yaml` runs on a schedule (and `workflow_dispatch`)
|
||||
to publish the same three images plus the chart from unreleased `main`. It is
|
||||
**not** gated by the version check (there is no `v*` tag) and it does **not**
|
||||
touch the `latest` tag, which stays pinned to the last `v*` release. Every job
|
||||
is gated on `github.repository == 'bytedance/deer-flow'`, so it only runs on
|
||||
the upstream repo - a scheduled run or manual dispatch on a fork skips all jobs.
|
||||
|
||||
Artifacts (under the running repo's owner, where `<date>` is `YYYYMMDD`):
|
||||
|
||||
- Images: `ghcr.io/<owner>/deer-flow-{backend,frontend,provisioner}:nightly`
|
||||
(rolling, overwritten each run) and `:nightly-<date>` (pinned to a day, but
|
||||
mutable within it - a same-day re-dispatch overwrites it). For a truly
|
||||
immutable pin, use `:sha-<short>`.
|
||||
- Chart: `oci://ghcr.io/<owner>/deer-flow`, version `<base>-nightly.<date>-<sha>`
|
||||
(e.g. `2.2.0-nightly.20260710-77a3652`). The short SHA makes each dispatch's
|
||||
chart version unique, so a same-day re-dispatch re-publishes cleanly (OCI
|
||||
chart versions are immutable and otherwise can't be overwritten). The
|
||||
packaged chart defaults `image.registry=ghcr.io/<owner>` and
|
||||
`image.tag=nightly`, so installing it pulls the matching nightly images with
|
||||
no values overrides:
|
||||
```bash
|
||||
helm install deer-flow oci://ghcr.io/<owner>/deer-flow \
|
||||
--version 2.2.0-nightly.20260710-77a3652
|
||||
```
|
||||
|
||||
The chart version is patched in-workflow only - `Chart.yaml` and `values.yaml`
|
||||
in the repo are never modified.
|
||||
|
||||
## Version gate
|
||||
|
||||
Both publishing workflows call `.github/workflows/verify-versions.yml` as their
|
||||
first job. It runs `scripts/verify_versions.sh` against the tag (minus the
|
||||
`v`). If any of the four version sources doesn't match the tag, the verify job
|
||||
fails and **all** publish jobs are skipped — no images, no chart.
|
||||
|
||||
When it fails, the job annotation names the offending file and suggests the
|
||||
fix:
|
||||
|
||||
```
|
||||
::error::frontend/package.json is '2.1.0' but expected '2.2.0'.
|
||||
Tip: run scripts/bump_version.sh 2.2.0 to align all sources.
|
||||
```
|
||||
|
||||
## Pre-releases (RCs)
|
||||
|
||||
Pre-release tags like `v2.2.0-rc1` are valid `v*` tags and trigger the same
|
||||
workflows. The version sources must equal the full pre-release string
|
||||
(`2.2.0-rc1`) — the gate compares exact strings. Use the same procedure with
|
||||
the rc version:
|
||||
|
||||
```bash
|
||||
scripts/bump_version.sh 2.2.0-rc1
|
||||
# update CHANGELOG, commit, tag v2.2.0-rc1, push
|
||||
```
|
||||
|
||||
## Recovering from a failed gate
|
||||
|
||||
If the gate failed because a source was forgotten:
|
||||
|
||||
1. Run `scripts/bump_version.sh <version>` to align the sources.
|
||||
2. Amend or add a follow-up commit.
|
||||
3. Delete and re-create the tag, then push it:
|
||||
```bash
|
||||
git tag -d v2.2.0
|
||||
git tag v2.2.0
|
||||
git push origin :refs/tags/v2.2.0
|
||||
git push origin v2.2.0
|
||||
```
|
||||
|
||||
Re-pushing the tag re-triggers the workflows. Because the gate blocks **all**
|
||||
artifacts when it fails, nothing was published under the bad tag, so re-tagging
|
||||
is safe — no images or chart were pushed to overwrite.
|
||||
|
||||
## Post-release
|
||||
|
||||
Optionally draft a **GitHub Release** from the tag, pasting the corresponding
|
||||
`CHANGELOG.md` section as the release notes. The changelog link references
|
||||
point at these release URLs.
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
As deer-flow doesn't provide an official release yet, please use the latest version to receive security updates.
|
||||
Currently, we have two branches to maintain:
|
||||
* main branch for deer-flow 2.x
|
||||
* main-1.x branch for deer-flow 1.x
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
Please go to https://github.com/bytedance/deer-flow/security to report the vulnerability you find.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Python-generated files
|
||||
__pycache__/
|
||||
*.py[oc]
|
||||
build/
|
||||
dist/
|
||||
wheels/
|
||||
*.egg-info
|
||||
.coverage
|
||||
.coverage.*
|
||||
.ruff_cache
|
||||
agent_history.gif
|
||||
static/browser_history/*.gif
|
||||
|
||||
log/
|
||||
log/*
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
venv/
|
||||
|
||||
# User config file
|
||||
config.yaml
|
||||
|
||||
# Langgraph
|
||||
.langgraph_api
|
||||
|
||||
# Sandbox runtime working dir — pre-created and excluded from uvicorn reload
|
||||
# (scripts/serve.sh, docker/dev-entrypoint.sh). Anchored so it does not match
|
||||
# the source package backend/packages/harness/deerflow/sandbox/.
|
||||
/sandbox/
|
||||
|
||||
# Claude Code settings
|
||||
.claude/settings.local.json
|
||||
@@ -0,0 +1 @@
|
||||
3.12
|
||||
@@ -0,0 +1,899 @@
|
||||
# AGENTS.md
|
||||
|
||||
This file provides guidance to AI coding agents (Claude Code, Codex, and others) when working with code in this repository. It is the source of truth; the sibling `CLAUDE.md` imports it via `@AGENTS.md`.
|
||||
|
||||
## Project Overview
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
|
||||
|
||||
**Architecture**:
|
||||
- **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime
|
||||
- **Frontend** (port 3000): Next.js web interface
|
||||
- **Nginx** (port 2026): Unified reverse proxy entry point
|
||||
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
|
||||
|
||||
**Runtime**:
|
||||
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
|
||||
- Scheduled-task executions must reuse that same Gateway run lifecycle. The scheduler may decide *when* work runs, but it must dispatch through the existing run path rather than introducing a parallel execution stack.
|
||||
|
||||
**Project Structure**:
|
||||
```
|
||||
deer-flow/
|
||||
├── Makefile # Root commands (check, install, dev, stop)
|
||||
├── config.yaml # Main application configuration
|
||||
├── extensions_config.json # MCP servers and skills configuration
|
||||
├── backend/ # Backend application (this directory)
|
||||
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
|
||||
│ ├── langgraph.json # LangGraph Studio graph configuration
|
||||
│ ├── packages/
|
||||
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── deerflow/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # middleware components (see Middleware Chain section)
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image, review_skill_package)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (search/fetch/scrape, image search, AIO sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ ├── app/ # Application layer (import: app.*)
|
||||
│ │ ├── gateway/ # FastAPI Gateway API
|
||||
│ │ │ ├── app.py # FastAPI application
|
||||
│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
|
||||
│ │ └── channels/ # IM platform integrations
|
||||
│ ├── tests/ # Test suite
|
||||
│ └── docs/ # Documentation
|
||||
├── frontend/ # Next.js frontend application
|
||||
└── skills/ # Agent skills directory
|
||||
├── public/ # Public skills (committed)
|
||||
└── custom/ # Custom skills (gitignored)
|
||||
```
|
||||
|
||||
## Important Development Guidelines
|
||||
|
||||
### Documentation Update Policy
|
||||
**CRITICAL: Always update README.md and AGENTS.md after every code change**
|
||||
|
||||
When making code changes, you MUST update the relevant documentation:
|
||||
- Update `README.md` for user-facing changes (features, setup, usage instructions)
|
||||
- Update `AGENTS.md` for development changes (architecture, commands, workflows, internal systems). `CLAUDE.md` imports it via `@AGENTS.md`, so editing `AGENTS.md` updates both.
|
||||
- Keep documentation synchronized with the codebase at all times
|
||||
- Ensure accuracy and timeliness of all documentation
|
||||
|
||||
## Commands
|
||||
|
||||
**Root directory** (for full application):
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make start # Start production services locally
|
||||
make stop # Stop all services
|
||||
```
|
||||
|
||||
**Backend directory** (for backend development only):
|
||||
```bash
|
||||
make install # Install backend dependencies
|
||||
make dev # Run Gateway API with reload (port 8001)
|
||||
make gateway # Run Gateway API only (port 8001)
|
||||
make test # Run all backend tests
|
||||
make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/
|
||||
make lint # Lint with ruff
|
||||
make format # Format code with ruff
|
||||
make migrate-rev MSG="..." # Autogenerate a new alembic revision (see Schema Migrations section)
|
||||
```
|
||||
|
||||
The `detect-blocking-io` target parses `app/`, `packages/harness/deerflow/`,
|
||||
and `scripts/` with AST. By default it reports only blocking IO candidates that
|
||||
are inside async code, reachable from async code in the same file, or reachable
|
||||
from sync-only `AgentMiddleware` before/after hooks that LangGraph can execute
|
||||
on the async graph path. It prints a concise summary and writes complete JSON
|
||||
findings to `.deer-flow/blocking-io-findings.json` at the repository root
|
||||
(both `make detect-blocking-io` from the repo root and `cd backend && make
|
||||
detect-blocking-io` resolve to the same repo-root path). JSON findings include
|
||||
`priority`, `location`, `blocking_call`, `event_loop_exposure`, `reason`, and
|
||||
`code` for model-assisted or manual review. `priority` is a deterministic
|
||||
review ordering from operation type, not proof of a bug. Bare-name same-file
|
||||
calls are resolved by function name, so duplicate helper names in one file can
|
||||
conservatively over-report async reachability. It is intentionally
|
||||
informational and is not run from CI in this round.
|
||||
|
||||
For a diff-scoped view of the same findings, `scripts/scan_changed_blocking_io.py`
|
||||
(repo root) reports findings on the added lines of `git diff <base>...HEAD`
|
||||
plus findings new versus the merge base (so a new async caller exposing an
|
||||
untouched sync helper in the same file is still reported) — used by the
|
||||
`blocking-io-guard` skill (`.agent/skills/blocking-io-guard/`) as the
|
||||
deterministic scope step before routing each candidate to a fix and/or a
|
||||
`tests/blocking_io/` runtime anchor.
|
||||
|
||||
Regression tests related to Docker/provisioner behavior:
|
||||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||||
- `tests/test_provisioner_request_threading.py` (keeps provisioner sandbox CRUD
|
||||
endpoints as sync FastAPI handlers so synchronous K8s client calls run in the
|
||||
Starlette worker pool instead of on the ASGI event loop)
|
||||
|
||||
Blocking-IO runtime gate (`tests/blocking_io/`):
|
||||
- Wraps every item under `tests/blocking_io/` with a strict Blockbuster
|
||||
context scoped to `app.*` and `deerflow.*` (see
|
||||
`tests/support/detectors/blocking_io_runtime.py`). Any sync blocking IO
|
||||
call whose stack passes through DeerFlow business code while running on
|
||||
the asyncio event loop raises `BlockingError` and fails the test.
|
||||
- Regression anchors live there: `test_skills_load.py` (locks the
|
||||
`asyncio.to_thread` offload around `LocalSkillStorage.load_skills`, fix
|
||||
for #1917); `test_sqlite_lifespan.py` (locks the offload around
|
||||
SQLite path resolution plus `ensure_sqlite_parent_dir`, fix for #1912);
|
||||
`test_jsonl_run_event_store.py` (locks `JsonlRunEventStore`'s async
|
||||
API offloading its file IO via `asyncio.to_thread`);
|
||||
`test_uploads_middleware.py` (locks `UploadsMiddleware.abefore_agent`
|
||||
offloading the uploads-directory scan off the event loop); and
|
||||
`test_uploads_router.py` (locks Gateway upload/list/delete endpoints
|
||||
offloading upload directory creation, staged writes, chmod/cleanup,
|
||||
directory scans/deletes, and remote sandbox sync off the event loop).
|
||||
- `test_gate_smoke.py` is a meta-test asserting the gate actually catches
|
||||
unoffloaded blocking IO and that the `@pytest.mark.allow_blocking_io`
|
||||
opt-out works.
|
||||
- Coverage boundary: the gate only sees code that test execution actually
|
||||
touches. Static AST coverage is a separate concern (out of scope for
|
||||
this PR).
|
||||
- CI: runs on every PR via `.github/workflows/backend-blocking-io-tests.yml`,
|
||||
hard-fail.
|
||||
|
||||
Boundary check (harness → app import firewall):
|
||||
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
|
||||
|
||||
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Harness / App Split
|
||||
|
||||
The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
**Import conventions**:
|
||||
```python
|
||||
# Harness internal
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
# App internal
|
||||
from app.gateway.app import app
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# App → Harness (allowed)
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
|
||||
# from app.gateway.routers.uploads import ... # ← will fail CI
|
||||
```
|
||||
|
||||
Package import hygiene: the `deerflow.agents` and `deerflow.subagents` package
|
||||
roots expose heavyweight graph/executor entrypoints lazily. Internal modules
|
||||
that only need lightweight types, config, or registries should import the
|
||||
concrete submodule instead of adding eager package-root imports that pull in the
|
||||
tool graph or subagent executor during state/schema imports.
|
||||
|
||||
### Agent System
|
||||
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`, `goal`, `promoted`, `delegations`, `skill_context`, `summary_text`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear), `merge_goal` (preserve the active goal across ordinary state updates unless the goal writer replaces it), `merge_promoted` (catalog-hash-scoped deferred tool promotions), `merge_delegations` (append task delegation entries, same id latest wins, terminal status never downgraded, capped to the most recent entries), and `merge_skill_context` (dedupe active-skill references by path, keep the most recently read entries; entries store a name/path/description reference, not the SKILL.md body). `summary_text` is a LastValue channel updated by summarization and projected into model requests as durable context data instead of being stored as a `messages` item.
|
||||
|
||||
**Runtime Configuration** (via `config.configurable`):
|
||||
- `thinking_enabled` - Enable model's extended thinking
|
||||
- `model_name` - Select specific LLM model
|
||||
- `is_plan_mode` - Enable TodoList middleware
|
||||
- `subagent_enabled` - Enable task delegation tool
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Lead-agent middlewares are assembled in strict order across three functions: the shared base in `packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py` (`_build_runtime_middlewares`, exposed via `build_lead_runtime_middlewares`), then the lead-only middlewares appended in `packages/harness/deerflow/agents/lead_agent/agent.py` (`build_middlewares`). Items marked *(optional)* are appended only when their config/runtime condition holds, so the live chain length varies.
|
||||
|
||||
**Shared runtime base** (`build_lead_runtime_middlewares`; subagents reuse most of this via `build_subagent_runtime_middlewares`):
|
||||
|
||||
1. **InputSanitizationMiddleware** - First, so it is the outermost `wrap_model_call` wrapper; every inner middleware (including LLM retries) sees sanitized messages
|
||||
2. **ToolOutputBudgetMiddleware** - Caps tool output size (per app config) before it re-enters the model context
|
||||
3. **ToolResultSanitizationMiddleware** - Neutralizes framework/injection tags (e.g. `<system-reminder>`) and boundary markers in *remote-content* tool results (`web_fetch`/`web_search`/`image_search`/`web_capture`) so attacker-controlled fetched pages cannot forge trusted framework context. Mirrors `InputSanitizationMiddleware`'s user-input guardrail for the other untrusted-content entry point; sits inner of `ToolOutputBudgetMiddleware` (neutralizes the raw output, then the budget truncates). Local tool output (bash/read_file) is left untouched. Scope is a name-based allowlist, so MCP remote-content tools registered under other names (e.g. `fetch_url`) are not yet covered — a metadata-tagging follow-up is tracked in the middleware source
|
||||
4. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves `user_id` via `get_effective_user_id()` (falls back to `"default"` in no-auth mode)
|
||||
5. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation (lead agent only)
|
||||
6. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
|
||||
7. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., user interruption), preserving raw provider tool-call payloads in `additional_kwargs["tool_calls"]`; malformed empty tool-call names are normalized to a recoverable error so strict OpenAI-compatible providers do not reject the next request
|
||||
8. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later stages run
|
||||
9. **GuardrailMiddleware** - *(optional, if `guardrails.enabled`)* Pre-tool-call authorization via pluggable `GuardrailProvider`; returns an error ToolMessage on deny. Providers: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md)
|
||||
10. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution
|
||||
11. **ReadBeforeWriteMiddleware** - *(optional, if `read_before_write.enabled`, default on)* Outermost write gate (issue #3857): `read_file` stamps a content hash onto its ToolMessage; `write_file` (append/overwrite-existing) and `str_replace` are blocked unless the newest mark for that path matches the file's current hash. Sits outside ToolProgressMiddleware and ToolErrorHandlingMiddleware so a blocked write returns immediately without consuming a ToolProgress slot. Blocked results call `normalize_tool_result` directly to stamp `deerflow_tool_meta` (`recoverable_by_model=True`) before returning, keeping the result well-formed for any outer consumer. Marks live on messages, so summarization dropping the read result invalidates the gate automatically; writes never refresh marks, forcing a re-read between consecutive edits. Gate check + tool execution are serialized per (thread, path) so same-turn parallel writes cannot reuse one stale mark; on sandboxes whose `read_file` reports failures as `"Error: ..."` strings instead of raising (AIO/E2B), uninspectable targets fail open (creation proceeds, no mark stamped)
|
||||
12. **ToolProgressMiddleware** - *(optional, if `tool_progress.enabled`)* State-machine-based stagnation guard (RFC #3177). Outer wrapper around ToolErrorHandlingMiddleware so its `wrap_tool_call` receives results already stamped with `deerflow_tool_meta`. Tracks per-(thread, tool) consecutive "no-new-info" calls across three error categories: (a) `recoverable_by_model=True` (no_results, not_found, permission, Jaccard-duplicate success): ACTIVE → WARNED (terminal — hint re-injected on each subsequent problem); (b) `recoverable_by_model=False, action≠stop` (rate_limited, transient): ACTIVE → WARNED → BLOCKED after `warn_escalation_count` more problems; (c) `recoverable_by_model=False, action=stop` (auth, config, internal): immediately BLOCKED on first occurrence. **Division of labor with LoopDetectionMiddleware:** ToolProgressMiddleware is a result-quality guard — fires after tool execution and blocks specific tools that stop producing new information; LoopDetectionMiddleware is a call-pattern guard — fires after the model responds and hard-stops the whole turn when the model repeatedly issues identical tool_calls. Both can inject HumanMessage hints in the same model call without conflict; neither reads the other's internal state.
|
||||
13. **ToolErrorHandlingMiddleware** - Receives `AppConfig`, converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting, stamps every result with `deerflow_tool_meta` (status / error_type / recoverable_by_model / recommended_next_action / source) via `tool_result_meta.normalize_tool_result`, stamps structured metadata for task exception wrappers, and stamps skill-read metadata for downstream durable-context capture. Task tool result text is generated from the same status/result/error inputs as the structured metadata so callers do not hand-write a second protocol string.
|
||||
|
||||
**Lead-only middlewares** (`build_middlewares`, appended after the base):
|
||||
|
||||
14. **DynamicContextMiddleware** - Injects the current date (and optionally memory) as a `<system-reminder>` into the first HumanMessage, keeping the base system prompt fully static for prefix-cache reuse
|
||||
15. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, injects the `SKILL.md` body as hidden current-turn context, and records a `middleware:skill_activation` audit event
|
||||
16. **DurableContextMiddleware** - Captures `task` delegations into `ThreadState.delegations` (including in-progress dispatches and terminal result summaries) and loaded skill-file references (name/path/description, parsed in-memory - not the body) into `ThreadState.skill_context` before summarization can compact the paired tool-call/result messages, then projects durable context into each model request. Static authority rules are injected as a `SystemMessage`; untrusted field values (`summary_text`, delegation results, skill descriptions) are injected separately as a hidden `HumanMessage` data block so compressed history, delegated work, and which skills are active stay visible without being stored as `messages` or promoted to system-role instructions. `build_subagent_runtime_middlewares` also attaches this middleware immediately before subagent summarization so a compacted `summary_text` is projected ahead of a preserved assistant/tool tail instead of leaving strict providers with an assistant-first request.
|
||||
17. **SummarizationMiddleware** - *(optional, if enabled)* Context reduction when approaching token limits
|
||||
18. **TodoListMiddleware** - *(optional, if `is_plan_mode`)* Task tracking with the `write_todos` tool
|
||||
19. **TokenUsageMiddleware** - *(optional, if `token_usage.enabled`)* Records token usage metrics; subagent usage is merged back into the dispatching AIMessage by message position
|
||||
20. **TitleMiddleware** - Auto-generates the thread title after the first complete exchange and normalizes structured message content before prompting the title model. If a first-turn run is interrupted before this middleware can write a title, `runtime/runs/worker.py` keeps the run in a finalizing state, persists a local fallback title from the latest checkpoint or original run input, and then syncs it to `threads_meta.display_name`. Replacement runs admitted by `multitask_strategy="interrupt"` / `"rollback"` wait for older same-thread finalization before entering the graph; the interrupted run only skips the fallback title write once a later run has started and may have advanced the checkpoint.
|
||||
21. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
22. **ViewImageMiddleware** - *(optional, if the model supports vision)* Injects base64 image data before the LLM call
|
||||
23. **McpRoutingMiddleware** - *(optional, if `tool_search.enabled` and PR1 MCP routing metadata produce a routing index)* Auto-promotes matching deferred MCP tool schemas before the model call by writing a minimal `promoted` state update. It matches only the latest real `HumanMessage`, uses the global `tool_search.auto_promote_top_k` limit (default 3, clamped to 1..5), never executes tools, and must be installed before `DeferredToolFilterMiddleware`
|
||||
24. **DeferredToolFilterMiddleware** - *(optional, if `tool_search.enabled`)* Hides deferred (MCP) tool schemas from the bound model until `tool_search` or `McpRoutingMiddleware` promotes them (reads per-thread promotions from `ThreadState.promoted`, hash-scoped)
|
||||
25. **SystemMessageCoalescingMiddleware** - Merges every SystemMessage into a single leading SystemMessage per request; provider-agnostic fix for strict backends (vLLM/SGLang/Qwen/Anthropic) that reject non-leading system messages. Touches the per-request payload only (checkpoint state unchanged); on midnight crossings only the latest `dynamic_context_reminder` SystemMessage survives
|
||||
26. **SubagentLimitMiddleware** - *(optional, if `subagent_enabled`)* Truncates excess `task` tool calls to enforce the `MAX_CONCURRENT_SUBAGENTS` limit
|
||||
27. **LoopDetectionMiddleware** - *(optional, if `loop_detection.enabled`)* Detects repeated tool-call loops; hard-stop clears both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer; stamps `loop_capped` via `consume_stop_reason` (#3875 Phase 2), symmetric to `TokenBudgetMiddleware`
|
||||
28. **TokenBudgetMiddleware** - *(optional, if `token_budget.enabled`)* Enforces per-run token limits
|
||||
29. **Custom middlewares** - *(optional)* Any `custom_middlewares` passed to `build_middlewares` are injected here, before the terminal-response/safety/clarification tail
|
||||
30. **TerminalResponseMiddleware** - When a provider returns an empty terminal `AIMessage` after tool execution, injects a hidden recovery prompt and retries the model once; a second empty response is replaced in checkpoint state by a visible error fallback marked for the run worker, so the run finishes as an error instead of a silent success
|
||||
31. **SafetyFinishReasonMiddleware** - *(optional, if `safety_finish_reason.enabled`)* Suppresses tool execution when the provider safety-terminated the response (e.g. `finish_reason=content_filter`); registered after terminal-response/custom middlewares so LangChain's reverse-order `after_model` dispatch runs it first
|
||||
32. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, writes a readable `ToolMessage.content` fallback plus structured `ToolMessage.artifact.human_input` request payload, and interrupts via `Command(goto=END)` (must be last). Because this middleware can short-circuit tool execution before LangChain emits `on_tool_end`, `RunJournal` performs a root-run final reconciliation for allowlisted clarification `ToolMessage`s whose `tool_call_id` was produced by the current run, so human-input request cards remain recoverable from `run_events` after checkpoint compaction. Human Input Card replies are submitted as `hide_from_ui` `HumanMessage`s with `additional_kwargs.human_input_response`; `RunJournal` persists only allowlisted hidden response sources (currently `ask_clarification`) as `llm.human.input`, which preserves answered-card state after compaction without exposing generic internal hidden context.
|
||||
|
||||
### Configuration System
|
||||
|
||||
**Main Configuration** (`config.yaml`):
|
||||
|
||||
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
|
||||
|
||||
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||||
|
||||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path or file content signature changes. The signature includes file metadata and a content digest, so Gateway and LangGraph reads stay aligned with `config.yaml` edits even on object-store or network mounts where mtime can remain stale.
|
||||
|
||||
**Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`.
|
||||
|
||||
Infrastructure fields are **restart-required**. The authoritative list lives in `packages/harness/deerflow/config/reload_boundary.py::STARTUP_ONLY_FIELDS` and is mirrored by the standardised `"startup-only:"` prefix on the corresponding `Field(description=...)` in `AppConfig`, so IDE hover on those fields surfaces the reason inline (no need to context-switch into this table). Currently registered: `database`, `checkpointer`, `run_events`, `stream_bridge`, `sandbox`, `log_level`, `logging`, `channels`, `channel_connections`. Adding a new restart-required field requires updating the registry; drift is pinned by `tests/test_reload_boundary.py`.
|
||||
|
||||
**Persistence backend resolution**: the unified `database` section selects the
|
||||
Gateway's LangGraph checkpointer, LangGraph Store, and DeerFlow SQL repositories.
|
||||
The deprecated `checkpointer` section remains backward compatible and, when
|
||||
present, overrides `database` for the LangGraph checkpointer and Store only;
|
||||
application repositories continue to use `database`.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
3. `config.yaml` in current directory (backend/)
|
||||
4. `config.yaml` in parent directory (project root - **recommended location**)
|
||||
|
||||
Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`).
|
||||
`ModelConfig` also declares `use_responses_api` and `output_version` so OpenAI `/v1/responses` can be enabled explicitly while still using `langchain_openai:ChatOpenAI`.
|
||||
|
||||
**Extensions Configuration** (`extensions_config.json`):
|
||||
|
||||
MCP servers and skills are configured together in `extensions_config.json` in project root:
|
||||
|
||||
Docker development mounts the project directory at `/app/project` and points
|
||||
`DEER_FLOW_CONFIG_PATH` / `DEER_FLOW_EXTENSIONS_CONFIG_PATH` into that directory.
|
||||
Keep mutable config files behind a directory bind mount: single-file bind mounts
|
||||
can become stale or inaccessible when a host editor replaces a file on save.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable
|
||||
3. `extensions_config.json` in current directory (backend/)
|
||||
4. `extensions_config.json` in parent directory (project root - **recommended location**)
|
||||
|
||||
### Gateway API (`app/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled).
|
||||
|
||||
CORS is same-origin by default when requests enter through nginx on port 2026. Split-origin or port-forwarded browser clients must opt in with `GATEWAY_CORS_ORIGINS` (comma-separated exact origins); Gateway `CORSMiddleware` and `CSRFMiddleware` both read that variable so browser CORS and auth-origin checks stay aligned.
|
||||
|
||||
**Routers**:
|
||||
|
||||
| Router | Endpoints |
|
||||
|--------|-----------|
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **Features** (`/api/features`) | `GET /` - report config-gated feature availability (currently `agents_api.enabled`) for frontend UI gating |
|
||||
| **Console** (`/api/console`) | Read-only cross-thread observability for the current user (the data layer for an operations dashboard or external monitoring): `GET /stats` - headline counters (runs/threads/agents/tokens/cost); `GET /runs` - paginated run history joined with thread titles (per-run cost); `GET /usage` - zero-filled daily token series + per-model breakdown with spend. Queries `runs`/`threads_meta` directly as a reporting layer (no new `RunStore` methods); requires a SQL database backend — returns 503 on `database.backend: memory`. Real-cost estimation reads optional `models[*].pricing` (`currency`, `input_per_million`, `output_per_million`, `input_cache_hit_per_million`; `ModelConfig` is `extra="allow"`, so no schema change) and prices each run from its `token_usage_by_model` input/output split. Pricing is **cache-aware**: `RunJournal` accumulates prompt-cache hits from `usage_metadata.input_token_details.cache_read` into a sparse `cache_read_tokens` bucket key (also threaded through `SubagentTokenCollector` → `record_external_llm_usage_records`), and cache-hit input tokens are billed at `input_cache_hit_per_million` (omitted → billed at the miss price, a conservative upper bound). Legacy rows fall back to run-level totals at `model_name`; unpriced models yield `cost: null` and cost fields are null when no pricing is configured |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; `POST /branches` - create a new main-thread branch from a completed assistant turn checkpoint. Workspace files are not checkpointed, so the branch only best-effort copies the current workspace when branching from the **latest** turn (`workspace_clone_mode="current_thread_best_effort"`); branching from an older/historical turn skips the copy (`workspace_clone_mode="skipped_historical_turn"`) so the branch never inherits files that only exist in a later timeline; `GET /goal`, `PUT /goal`, `DELETE /goal` - read, set, and clear the active thread goal; `POST /compact` - manually summarize older active context into `summary_text` and retain the recent message window, blocked while a run is in flight; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Suggestions** (`/api/suggestions`) | `GET /config` - returns global suggestions config boolean; `POST /threads/{id}/suggestions` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
|
||||
| **Input Polish** (`/api/input-polish`) | `POST /` - rewrite a composer draft before it is sent. This is a short authenticated `runs:create` LLM request using `input_polish` config; it does not create a LangGraph run, persist a message, or modify thread state. Shares the non-graph one-shot LLM path (`deerflow.utils.oneshot_llm.run_oneshot_llm`) with the suggestions route so model build + Langfuse metadata + invoke stay in one place; validates the same stripped view of the draft it sends to the model, and preserves literal `<think>` substrings in the rewrite (`strip_think_blocks(truncate_unclosed=False)`) |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `POST /regenerate/prepare` - prepare clean input + checkpoint metadata for regenerating the latest assistant answer; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /{rid}/workspace-changes` - workspace/output file change summary and optional diffs; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **GitHub Webhooks** (`/api/webhooks/github`) | `POST /` - receive GitHub App / repo webhook deliveries. Verifies `X-Hub-Signature-256` against `GITHUB_WEBHOOK_SECRET`; exempt from auth + CSRF because authenticity is enforced by HMAC. The route is fail-closed: mounted only when `GITHUB_WEBHOOK_SECRET` is set, or when explicit dev opt-in `DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1` is set. Recognized events include `ping`, `issues`, `issue_comment`, `pull_request`, `pull_request_review`, and `pull_request_review_comment`; unknown events return 200 with `handled=false`. Fan-out runtime failures return 503 so GitHub retries; permanent/non-retryable conditions such as `channels.github.enabled: false`, unknown events, malformed payloads, or unavailable channel service return 200 with a skipped/handled response. |
|
||||
| **GitHub Event-Driven Agents** | Custom agents can declare a `github:` block in their `config.yaml` to bind to repos and event triggers. Webhook fan-out publishes one `InboundMessage` per matching binding to the channel bus; `GitHubChannel` routes those messages through `ChannelManager`. The response `dispatch` summarizes matched/fired/skipped agents. |
|
||||
|
||||
**Workspace change review**: `packages/harness/deerflow/workspace_changes/`
|
||||
captures a pre-run and post-run snapshot of the thread-owned `workspace` and
|
||||
`outputs` directories. `runtime/runs/worker.py` performs the filesystem scan via
|
||||
`asyncio.to_thread` and writes a `workspace_changes` event with category
|
||||
`workspace` when changes exist. Uploads are intentionally excluded. Text diffs
|
||||
are size-limited; binary, large, and sensitive-looking paths are persisted as
|
||||
metadata only.
|
||||
|
||||
**RunManager / RunStore contract**:
|
||||
- `RunManager.get()` is async; direct callers must `await` it.
|
||||
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
|
||||
- `cancel()` and `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persist interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
|
||||
- Store-only hydrated runs are readable history. If the current worker has no in-memory task/control state for that run, cancellation APIs can return 409 because this worker cannot stop the task.
|
||||
- `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
|
||||
- Redis `StreamBridge` keys use a rolling retained-buffer TTL (`stream_bridge.stream_ttl_seconds`, refreshed on `publish()` / `publish_end()`) as a leak safety net, not as a run timeout. Startup orphan recovery publishes `END_SENTINEL` and schedules stream cleanup for recovered runs; malformed `Last-Event-ID` reconnect values live-tail new Redis events rather than replaying the retained buffer. Do not broaden this into a shared-database multi-pod reaper without adding worker ownership/liveness first.
|
||||
- Thread-scoped run creation accepts `checkpoint` / `checkpoint_id`; Gateway validates the checkpoint belongs to the request thread before writing `checkpoint_id` / `checkpoint_ns` into `config.configurable` for LangGraph branching.
|
||||
- Thread-scoped Gateway runs evaluate an active `ThreadState.goal` after the visible turn completes. `runtime/goal.py` asks a non-thinking evaluator model to judge only visible conversation evidence and return a typed blocker; the evaluator model is created once per run and reused across hidden continuation checks. Satisfied goals are cleared; every non-satisfied evaluation — continuable or stand-down — is persisted with `last_evaluation` (the blocker, reason, and evidence summary; outcomes that stop the loop additionally record a `stand_down_reason` for observability), but only `goal_not_met_yet` evaluations are streamed as hidden `HumanMessage` continuations, and only when a durable assistant end-of-turn checkpoint exists, the run has not been aborted, the thread did not change during evaluation, and the no-progress breaker has not fired. The continuation cap is 8 — a hard maximum in the `0`–`8` range; callers requesting more are clamped (`set_goal`/TUI) or rejected with 422 (`PUT /goal`). The no-progress breaker keys on the latest visible assistant evidence (not the evaluator's free-text reason, which an LLM rewords every turn), so two consecutive continuations that add no new visible assistant output stop the loop after 2 attempts. Model-response cleanup helpers such as think-block stripping and code-fence stripping live in `deerflow.utils.llm_text` so `runtime/goal.py` and Gateway suggestion parsing share the same JSON-prep behavior.
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runtime, all other `/api/*` → Gateway REST APIs.
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command(command, env=None)`, `read_file`, `write_file`, `list_dir`. The optional `env` injects per-call environment variables (request-scoped secrets — see Request-Scoped Secrets below); `LocalSandbox` merges it via `subprocess.run(env=...)` and `AioSandbox` routes env-bearing commands through the `bash.exec(env=...)` API on a fresh session.
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop.
|
||||
**Environment policy** (`sandbox/env_policy.py`): `execute_command` no longer inherits the full `os.environ`. `build_sandbox_env()` scrubs secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`) from the inherited environment before layering injected request secrets on top, so platform credentials (e.g. `OPENAI_API_KEY`) never leak into skill subprocesses. Benign vars (`PATH`, `HOME`, `LANG`, `VIRTUAL_ENV`, ...) are preserved.
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`. Legacy global-custom mounts are gated by the same user-scoped skill discovery rule used for prompt/list visibility; providers must not infer visibility from raw directory presence alone.
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation. Active-cache and warm-pool entries are checked with the backend during acquire/reuse; definitively dead containers are dropped from all in-process maps so the thread can discover or create a fresh sandbox instead of reusing a stale client. Backend health-check failures are treated as unknown, not dead; local discovery likewise treats an unverifiable container as not adoptable and falls through to create rather than failing acquire. `get()` remains an in-memory lookup for event-loop-safe tool paths. Legacy global-custom mounts follow the same shared visibility helper as local and remote providers.
|
||||
- `BoxliteProvider` (`packages/harness/deerflow/community/boxlite/`) - BoxLite micro-VM isolation. The `boxlite` runtime is optional (`deerflow-harness[boxlite]`) and lazy-imported only when this provider is selected. The provider owns one private asyncio event loop on a daemon thread because BoxLite handles are loop-affine; sync `Sandbox` calls marshal onto that loop with `run_coroutine_threadsafe`.
|
||||
Boxes are named deterministically from `user_id:thread_id`, released into an in-process warm pool after each agent turn, and reclaimed only by the same user/thread. Warm-pool health checks use a short explicit timeout and forward that timeout through both BoxLite `exec(timeout=...)` and the private-loop `.result(timeout)` bridge so a hung VM cannot pin the per-thread acquire lock indefinitely.
|
||||
`sandbox.replicas` caps active + warm VMs per gateway process; if capacity is exhausted, only warm-pool VMs are evicted. `sandbox.idle_timeout` stops idle warm VMs after the configured seconds. `reset()` is intentionally a lightweight registry clear for `reset_sandbox_provider()` and does not close boxes, stop the idle reaper, or close the private loop; full teardown remains `shutdown()`.
|
||||
|
||||
|
||||
**Shared warm-pool lifecycle:** community sandbox providers that keep released sandboxes alive for fast reuse share `deerflow.community.warm_pool_lifecycle.WarmPoolLifecycleMixin`. The mixin owns the common `DEFAULT_IDLE_TIMEOUT=600`, `IDLE_CHECK_INTERVAL=60`, `DEFAULT_REPLICAS=3`, idle-checker loop, warm-pool expiry, oldest-warm eviction, replica counting, and soft-cap logging. Providers remain responsible for their own active registries, creation/discovery, health checks, and destroy hook (`_destroy_warm_entry`): AIO destroys `SandboxInfo` through its backend; Boxlite closes loop-affine `BoxliteBox` handles. AIO keeps active-idle cleanup outside the mixin and delegates only warm-pool expiry to the shared helper.
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `LocalSandboxProvider` builds per-thread `PathMapping`s for the user-data prefixes at acquire time; `tools.py` keeps `replace_virtual_path()` / `replace_virtual_paths_in_command()` as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept `/mnt/user-data/...` natively.
|
||||
- Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
|
||||
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling. For `LocalSandbox` (host bash), POSIX output is captured through bounded pipe-drain threads and stdin is `/dev/null`, so a backgrounded long-lived process (`server &`) returns immediately instead of blocking the turn on an inherited pipe, while unredirected background output is drained without growing anonymous temp files. Commands that read stdin get immediate EOF. The command runs in its own process group with a wall-clock timeout (`sandbox.bash_command_timeout`, default 600s); on timeout the whole group is killed and the agent gets a notice telling it to background long-lived processes. The bash tool description itself also instructs the model to background long-lived processes (e.g. servers) up front so it doesn't waste the turn waiting on a foreground server. See `LocalSandbox.execute_command` / `_run_posix_command` and `bash_tool`'s docstring.
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `write_file` - Write/append to files, creates directories; overwrites by default and exposes the `append` argument in the model-facing schema for end-of-file writes; subject to the read-before-write gate when `read_before_write.enabled` (see Middleware Chain)
|
||||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process; subject to the read-before-write gate when `read_before_write.enabled` (see Middleware Chain)
|
||||
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
|
||||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||||
**Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`); default subagent timeout `subagents.timeout_seconds=1800` (30 min) and built-in `general-purpose` `max_turns=150` (raised from 100/15-min so deep-research subtasks stop hitting `GraphRecursionError` out of the box)
|
||||
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result. `task_started` carries the resolved effective model name. The per-subagent `SubagentTokenCollector` publishes a cumulative usage snapshot to the shared `SubagentResult` after every completed LLM response; the next `task_running` event carries that snapshot, so collapsed workspace cards can update without re-accounting parent-run totals. Terminal ToolMessage metadata (`subagent_model_name`, `subagent_token_usage`) and the persisted `subagent.end` event retain the model/usage after reload; absent provider usage stays absent rather than being estimated as zero.
|
||||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||||
**Handled LLM failures**: `LLMErrorHandlingMiddleware` deliberately converts provider/model exceptions into an `AIMessage` so the graph can end cleanly, stamping `additional_kwargs.deerflow_error_fallback=true` plus error metadata. Clean graph termination does not imply subagent success: `SubagentExecutor` inspects the last assistant message at terminalization and maps a marked fallback to `SubagentStatus.FAILED`, which then emits `task_failed` and the existing structured `subagent_error`. Only the marker is authoritative — error-looking assistant prose without it remains a normal completed result, so neither the executor nor frontend parses display text as a status protocol.
|
||||
**Guardrail caps & `stop_reason` (#3875 Phase 2)**: three independent axes can end a subagent run early, and all now surface *why* through one additive field rather than a new status enum. **Turn axis**: `recursion_limit` on the subagent `run_config` equals `max_turns`, so exhausting the turn budget raises `GraphRecursionError` from `agent.astream`; `executor.py::_aexecute` catches it specifically (before the generic `except Exception`). **Token axis**: `TokenBudgetMiddleware` is attached per-agent via `build_subagent_runtime_middlewares` from `subagents.token_budget` (default `max_tokens` **coupled to `summarization.enabled`** — 1,000,000 when subagent summarization is on, 2,000,000 when off, warn at 0.7, hard-stop at 1.0; a user-set budget always wins regardless of the switch — #3875 Phase 3; a backstop against a subagent that burns tokens on trivial work). It does *not* raise: at the hard-stop threshold it strips the in-flight turn's tool calls, forces `finish_reason="stop"`, and lets the run complete naturally with a final answer. **Loop axis**: `LoopDetectionMiddleware` (attached at the same point) catches repeated identical tool-call sets — or one tool *type* called many times with varying args — and its hard-stop likewise strips `tool_calls` and forces a final answer without raising, recording `loop_capped`. Each guard exposes its cap on a per-`run_id` `consume_stop_reason(run_id)` accessor; `_aexecute` collects **every** middleware with that method (duck-typed via `hasattr`, so the executor has no import coupling to the guard classes) and surfaces the first non-`None` reason — adding a future guard needs no executor change. **Surfacing**: whichever axis fired, `_aexecute` stamps a normal status plus an additive reason — `completed` + `stop_reason=token_capped|turn_capped|loop_capped` when a usable final answer (or partial recovered from the last streamed chunk via `_extract_final_result` → `utils/messages.py::message_content_to_text`, returning a `"No response Generated"` sentinel when no text survived) was produced; `failed` + `stop_reason=turn_capped` when nothing usable survived. `SubagentResult.stop_reason` flows through `task_tool.py::_task_result_command` → `format_subagent_result_message` (renders `Task Succeeded (capped: ...)` / `Task failed (capped: ...)`) and `make_subagent_additional_kwargs`, which stamps the additive `subagent_stop_reason` key alongside the normal `subagent_status`. **Why additive, not an enum**: a new status value would break v1 consumers; an optional field is ignored by older frontends and ledger readers, so the cross-language contract (`contracts/subagent_status_contract.json` v2 + `subagents/status_contract.py` + `frontend/.../subtask-result.ts`, pinned by `test_status_values_match_contract` / `test_stop_reason_values_match_contract`) stays backward-compatible. The durable delegation ledger captures `stop_reason` onto the entry and renders model-facing guidance ("hit a guardrail cap with a partial result; reuse it, retry tighter, or raise the per-agent budget (`max_turns` / `token_budget`)") so the lead reuses a capped completion knowingly instead of mistaking it for a clean one. (Phase 1 shipped this surfacing as a `MAX_TURNS_REACHED` status enum in #3949; Phase 2 replaced that enum with the additive `stop_reason` field per the agreed design — the `max_turns_reached` status value and `SubagentStatus.MAX_TURNS_REACHED` are gone.)
|
||||
**Context compaction (#3875 Phase 3, #4039)**: subagents inherit `DeerFlowSummarizationMiddleware` via `build_subagent_runtime_middlewares`, gated on the **same** `summarization.enabled` switch the lead reads (one config covers both chains; trigger/keep/model/prompt come from the shared `summarization` config so they cannot drift). The subagent builder attaches `DurableContextMiddleware` immediately before summarization, using the same skills path/read-tool settings as the lead chain. Compaction stores the generated summary in `ThreadState.summary_text` rather than as a `messages` item; the durable-context wrapper therefore projects it into the next model request as guarded hidden human data. This is required when a message-count keep policy preserves only an assistant tool-call plus its tool results: without the injected summary the next request begins with assistant/tool history and strict OpenAI-compatible providers can reject it. Because `DurableContextMiddleware` inserts a second `SystemMessage(authority_contract)` after the subagent's leading system prompt, the builder also appends `SystemMessageCoalescingMiddleware` innermost (mirroring the lead chain, appended after the optional summarization middleware so it is unconditionally last) to merge every `SystemMessage` into one leading `system_message` — otherwise the durable fix would trade #4039's assistant-first HTTP 400 for a duplicate-system 400 on the same strict backends (#4040). The factory is called with `skip_memory_flush=True` on the subagent path: the lead's `memory_flush_hook` (attached when `memory.enabled`) flushes pre-compaction messages into durable memory keyed by `thread_id`, and subagents share the parent's `thread_id`, so without skipping the hook a subagent's internal turns would pollute the **parent** thread's durable memory. Placement differs from the lead chain (lead appends summarization *before* the guard trio; subagent appends it *after*) — benign because the middleware implements only `before_model` (compaction) with no `after_model`/`consume_stop_reason`, so it cannot disturb the Phase 2 guard-cap stop-reason channel. Compaction rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, which shrinks `len(messages)` below the step-capture cursor mid-run; `capture_new_step_messages` (see Step capture below) resets the cursor to the new tail on contraction so steps appended after the compaction point are not silently dropped.
|
||||
**Step capture & persistence (#3779)**: `executor.py` captures both assistant turns (`AIMessage`) **and** tool outputs (`ToolMessage`) via `subagents/step_events.py::capture_new_step_messages`, which walks the *newly-appended tail* of each `stream_mode="values"` chunk (not just `messages[-1]`) so a multi-tool-call turn — where LangGraph's `ToolNode` appends several `ToolMessage`s in one super-step — keeps every tool output instead of dropping all but the last. `runtime/runs/worker.py::_SubagentEventBuffer` additionally persists these `task_*` custom events to the `RunEventStore` as `subagent.start`/`subagent.step`/`subagent.end` (`category="subagent"`, `task_id` in `metadata`). It **batches** writes via `put_batch` (flushing on a terminal `subagent.end`, at `FLUSH_THRESHOLD` events, and in the worker's `finally`) rather than one `put()` per step, since `put()` is a documented low-frequency path (per-thread advisory lock per call) and a deep subagent (`max_turns=150`) emits hundreds of steps on the hot stream loop. `build_subagent_step` caps both the per-step `text` and each tool call's serialized `args` at `SUBAGENT_STEP_MAX_CHARS` (flagged `truncated` / `args_truncated`) so a large `write_file`/`bash` payload can't produce an unbounded row. The dedicated category keeps them out of `list_messages` (the thread feed) while `list_events` returns them for the frontend's fetch-on-expand backfill. `list_events` accepts `task_id` (filters on `metadata["task_id"]` — SQL-side in `DbRunEventStore` via `event_metadata["task_id"].as_string()`, in-memory in the JSONL/memory stores) plus an `after_seq` forward cursor, so the card pages through one subagent's steps without the run-wide `limit` truncating the tail (no schema migration: the filter rides the existing run-scoped index). `step_events.py` is a pure, unit-tested layer (`build_subagent_step` / `subagent_run_event`). **History contraction (#3875 Phase 3)**: `capture_new_step_messages` assumes append-only growth, but `DeerFlowSummarizationMiddleware` rewrites the messages channel via `RemoveMessage(id=REMOVE_ALL_MESSAGES)`, shrinking `len(messages)` below the cursor mid-run. On contraction (`total < processed_count`) the cursor resets to the new tail; `capture_step_message`'s id/content dedup prevents re-emitting pre-compaction steps, so steps appended after the compaction point are still captured instead of being dropped until `total` overtakes the stale cursor.
|
||||
**Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` assembles deferral after policy filtering via the shared `assemble_deferred_tools` (fail-closed), appends the `tool_search` tool, injects the `<available-deferred-tools>` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `McpRoutingMiddleware` (when PR1 routing metadata matches deferred tools) before `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(...)`. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run
|
||||
**Checkpointer isolation**: Subagent graphs are compiled with `checkpointer=False` to avoid inheriting the parent run's checkpointer, since subagents are one-shot and never resume.
|
||||
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
|
||||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||||
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
|
||||
2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
|
||||
3. **Built-in tools**:
|
||||
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
|
||||
- `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware, which preserves text fallback and adds `artifact.human_input` for Web UI Human Input Cards)
|
||||
- `view_image` - Read image as base64 (added only if model supports vision)
|
||||
- `setup_agent` - Bootstrap-only: persist a brand-new custom agent's `SOUL.md` and `config.yaml`. Bound only when `is_bootstrap=True`.
|
||||
- `update_agent` - Custom-agent-only: persist self-updates to the current agent's `SOUL.md` / `config.yaml` from inside a normal chat (partial update + atomic write). Bound when `agent_name` is set and `is_bootstrap=False`.
|
||||
4. **Subagent tool** (if enabled):
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type)
|
||||
|
||||
Scheduled-task runtime note:
|
||||
- Scheduled background runs set `context.non_interactive=true` and therefore exclude `ask_clarification` from the lead-agent tool list. This keeps scheduler-triggered runs from stalling on human confirmation mid-execution. `non_interactive` is an internal-only context key: it is merged from `body.context` only when the request authenticated as the process-internal user (the scheduler path), never from arbitrary HTTP/IM clients.
|
||||
|
||||
**Community tools** (`packages/harness/deerflow/community/`): optional integrations, each in its own subpackage and wired through `config.yaml`. Documented examples:
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||||
- `firecrawl/` - Web scraping via Firecrawl API
|
||||
- `image_search/` - Image search via DuckDuckGo
|
||||
- `aio_sandbox/` - Docker-based isolation (`AioSandboxProvider`)
|
||||
|
||||
Additional providers also live here (`boxlite`, `brave`, `browserless`, `crawl4ai`, `ddg_search`, `e2b_sandbox`, `exa`, `fastcrw`, `groundroute`, `infoquest`, `searxng`, `serper`); see each subpackage for specifics.
|
||||
|
||||
**ACP agent tools**:
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
|
||||
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
|
||||
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||||
|
||||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||||
|
||||
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
|
||||
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
|
||||
- **Cache invalidation**: Detects config file changes via mtime comparison
|
||||
- **Transports**: stdio (command-based), SSE, HTTP
|
||||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||||
- **Routing hints**: `extensions_config.json -> mcpServers.<server>.routing` and
|
||||
`tools.<original_tool_name>.routing` are soft preference metadata. The effective
|
||||
routing is resolved while `mcp/tools.py::get_mcp_tools()` still has both
|
||||
`source_name` and the original MCP tool name, then stored on `tool.metadata`
|
||||
under `deerflow_mcp_routing`. Prompt rendering uses
|
||||
`tools/builtins/tool_search.py::get_mcp_routing_hints_prompt_section`, which
|
||||
references `tool_search` when a hinted MCP tool is currently deferred; do not
|
||||
add a parallel routing middleware for PR1-style preference hints.
|
||||
- **Stdio file outputs**: Persistent stdio sessions are scoped by `user_id:thread_id`. For stdio transports only, DeerFlow pins the subprocess default `cwd` to the thread workspace and `TMPDIR`/`TMP`/`TEMP` to `workspace/.mcp/tmp/`, unless the operator explicitly configured `cwd` or temp env values. SSE/HTTP transports skip this filesystem prep entirely.
|
||||
- **Stdio path translation**: MCP-returned local file references are not copied. If a `ResourceLink` or conservative free-text path resolves to an existing file inside the thread's mounted user-data tree, it is translated deterministically to `/mnt/user-data/...`; paths outside that tree remain unchanged.
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via mtime
|
||||
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
|
||||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools, required-secrets)
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json
|
||||
- **Injection (legacy / default)**: Enabled skills are listed in the agent system prompt with full metadata and container paths (`<available_skills>` block). Controlled by `skills.deferred_discovery: false` (default).
|
||||
- **Deferred discovery** (`skills.deferred_discovery: true`): Skills are listed by name only in a compact `<skill_index>` block, keeping the system prompt prefix-cache friendly. The agent calls the `describe_skill` tool at runtime to fetch full metadata for skills it wants to use, then loads the SKILL.md via `read_file`. Two new modules support this path:
|
||||
- `skills/catalog.py` — `SkillCatalog` (immutable, searchable; query forms: `select:a,b`, `+prefix`, free-text regex); `select:` returns all requested skills without a result cap; other modes cap at `MAX_RESULTS=5`.
|
||||
- `skills/describe.py` — `build_describe_skill_tool(catalog)` builds the `describe_skill` tool as a closure; `build_skill_search_setup(skills, enabled, ...)` produces a `SkillSearchSetup(describe_skill_tool, skill_names)` that is wired into both the LangGraph agent factory (`agent.py`) and the embedded client (`client.py`).
|
||||
- **Slash activation**: `/skill-name task` loads that enabled skill's `SKILL.md` for the current model call only. The resolver rejects leading whitespace, missing separators, reserved channel commands (`/new`, `/help`, `/bootstrap`, `/status`, `/models`, `/memory`, `/goal`), disabled skills, and skills outside a custom agent's whitelist.
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
- **SkillScan**: `packages/harness/deerflow/skills/skillscan/` is the native deterministic scanner for `.skill` archives and agent-managed skill writes. It runs offline before the LLM scanner, emits structured findings (`rule_id`, `severity`, `file`, `line`, `message`, `remediation`, redacted `evidence` — category/analyzer are encoded in the `rule_id` prefix), blocks `CRITICAL`, and passes warning findings into `scan_skill_content()`. `scan_archive_preflight()` / `scan_skill_dir()` are pure sync functions (dispatch off the event loop); `enforce_static_scan()` applies the blocking policy and the `skill_scan.enabled` kill switch. Do not add Semgrep/OpenGrep or YAML rule-engine dependencies to the core path; Phase 1 rule specs live in Python constants next to their analyzers in `skillscan/orchestrator.py`.
|
||||
- **Skill Review Core**: `packages/harness/deerflow/skills/review/` provides read-only package snapshots, deterministic facts, resource/eval analysis, report rendering, and the CLI (`python -m deerflow.skills.review.cli`). It reuses the shared frontmatter helper and SkillScan; it must not import `app.*`, execute target scripts, install dependencies, or call networks. JSON contracts live in `contracts/skill_review/`. The `review_skill_package` built-in tool labels results with `review_subject_entry` and never `skill_context_entry`, so reviewing a target does not activate it, bind its `required-secrets`, or apply its `allowed-tools`. Its model-visible `ToolMessage.content` is a compact JSON payload with untrusted control tags neutralized; the full raw review payload, including Markdown renders, stays in `ToolMessage.artifact`. CI should run the CLI with `--fail-on error --fail-on-incomplete` so blocker/error findings and truncated/not-assessed packages fail the gate. The public `skills/public/skill-reviewer` skill owns semantic readiness review and suggestions only; mutation and runtime experiments remain owned by `skill-creator`.
|
||||
|
||||
#### Request-Scoped Secrets (`required-secrets`)
|
||||
|
||||
Lets a caller pass per-request, short-lived end-user credentials (e.g. an ERP token) to a skill's sandbox scripts without the value entering the prompt, tool arguments, the executed command string, or traces (issue #3861).
|
||||
|
||||
- **Declare**: a skill lists the secrets it needs in `SKILL.md` frontmatter — `required-secrets:` as a string list or `{name, optional}` mappings. `name` is both the lookup key and the env var name exposed to scripts. Parsed by `skills/parser.py::parse_required_secrets` into `Skill.required_secrets` (`SecretRequirement`); malformed entries are dropped with a warning.
|
||||
- **Carry**: the caller sends values out-of-band in the run request's `context.secrets` mapping (never a message). `runtime/secret_context.py` owns the contract (`SECRETS_CONTEXT_KEY`, `extract_request_secrets`). The existing `context` passthrough carries it to `runtime.context` without mirroring into `configurable`. `build_run_config` still sets `configurable.thread_id` on the context path — the checkpointer requires it.
|
||||
- **Bind (point A+)**: `SkillActivationMiddleware._resolve_secret_bindings` recomputes the injection set (`runtime.context[__active_skill_secrets]`) on every model call from two unioned sources, then REPLACES the key. (1) *Slash*: the run's most recent `/skill` activation, persisted as a source on the run context (only the activated skill's **canonical container path**, never its declared secrets) so the whole tool loop after the activation call keeps the binding; a new activation replaces it. Slash reads the genuine user text via `get_original_user_content_text`; `InputSanitizationMiddleware` preserves it (`ORIGINAL_USER_CONTENT_KEY`), so activation fires even after sanitization. (2) *In-context* (autonomous invocation): skills the model actually loaded in this thread — `ThreadState.skill_context` entries. **Both sources resolve the live registry skill by normalized container path on every call** (`_resolve_registry_skill`) and bind only that skill's own declared secrets — enabled + allowlist checked for both; the `secrets-autonomous: false` opt-out (malformed values fail closed to `false`) additionally gates the in-context path but exempts explicit slash. Resolving by registry — not by trusting the source's stored data — is what makes a caller-forged `__slash_skill_secret_source` harmless (`runtime.context` is caller-mergeable; the gateway also strips caller `__`-keys in `build_run_config`), #3938. Authorization is three-gated regardless of activation style: skill **enabled** by the operator × values **supplied per-request** by the caller (`context.secrets`) × names **declared** in frontmatter (∩ semantics). Because the set is recomputed per call, a skill evicted from `skill_context` (capacity) or a caller that stops supplying a value loses injection on the next call. The injected value always comes from the caller's request, never the host environment (scrubbed first — see below), so a declared name that also exists in the host env is safe: the caller's value wins and the host value is dropped (the #3861 per-user-key-overrides-shared-key case). Missing required secrets are logged once per binding change, not injected; binding changes are recorded as a `middleware:skill_secrets` journal event (skill and secret names only, never values).
|
||||
- **Inject**: `bash_tool` reads the injection set and passes it as `execute_command(env=...)`. Scope is the activation turn/run only — a run without `/skill` activation injects nothing.
|
||||
- **AIO image requirement**: on `AioSandbox` the env path uses the `bash.exec` API (`POST /v1/bash/exec`), which upstream all-in-one-sandbox only ships since `1.9.3` — older images (including a `latest` tag frozen on the `1.0.0.x` line) 404 the whole `/v1/bash/*` namespace. `AioSandbox` detects the 404, remembers the capability gap on the instance, and fails fast with an actionable upgrade error instead of letting the model retry raw 404s; there is deliberately **no** fallback through the legacy shell path because none keeps the secret values out of the command string (#3921). Regression tests: `tests/test_aio_sandbox.py::TestBashExecUnsupportedFailFast`.
|
||||
- **Inherited-env scrub**: `execute_command` no longer leaks the Gateway's `os.environ` to skill subprocesses — `env_policy.build_sandbox_env` drops secret-looking names (`*KEY*`/`*SECRET*`/`*TOKEN*`/`*PASS*`/`*CREDENTIAL*`/`*DSN*` + a connection-string denylist like `DATABASE_URL`/`REDIS_URL`/`GH_PAT`, plus no-flag credential sources like `MYSQL_PWD`/`REDISCLI_AUTH`/`PGPASSFILE`/`PGSERVICEFILE`) so platform credentials never reach a skill; a skill that needs one must declare it.
|
||||
- **Leak surfaces sealed** (verified by a real-gateway e2e run — secret reaches the sandbox but none of these): prompt (value never in a message), trace (`tracing/metadata.py` never copies `context`), checkpoint (secrets live on `runtime.context`, not graph state), audit (journal records names only), stdout (`tools.py::mask_secret_values` redacts injected values from bash output), and **run-record persistence + run API** (`services.py::start_run` stores `redact_config_secrets(body.config)` so `runs.kwargs_json` and `RunResponse.kwargs` never carry the secret).
|
||||
- **Scope / non-goals**: no persistence/vaulting — values are request-scoped and never stored server-side, so long-lived use means the caller re-supplies `context.secrets` on each request while the skill stays in `skill_context`; subagents do not inherit the injection set; the MCP per-user-credential gap (#3322) is a sibling, not covered here. Tests: `tests/test_skill_request_scoped_secrets.py`.
|
||||
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||||
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
|
||||
- Supports `supports_vision` flag for image understanding models
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||||
|
||||
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
|
||||
|
||||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints
|
||||
- Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns
|
||||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias
|
||||
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram, Discord, DingTalk, GitHub) to the DeerFlow agent via Gateway's LangGraph-compatible API.
|
||||
|
||||
**Architecture**: Channels communicate with Gateway through the `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
|
||||
|
||||
**Components**:
|
||||
- `message_bus.py` - Async pub/sub hub (`InboundMessage` → queue → dispatcher; `OutboundMessage` → callbacks → channels)
|
||||
- `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]` → `thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations)
|
||||
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands including `/goal` (setting a goal persists it through Gateway and then routes the objective as a chat turn), keeps Slack/Discord on `client.runs.wait()`, uses `client.runs.stream(["messages-tuple", "values"])` for Feishu/Telegram incremental outbound updates, serializes same-thread Feishu turns in-manager when the channel's `ChannelRunPolicy.serialize_thread_runs=True` so rapid follow-ups queue instead of tripping the runtime busy reply, and switches to `client.runs.create()` (fire-and-forget, returns once the run is `pending`) for channels whose `ChannelRunPolicy.fire_and_forget=True` so long autonomous runs do not hit the SDK default 300s `httpx.ReadTimeout`
|
||||
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
|
||||
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
|
||||
- `slack.py` / `feishu.py` / `telegram.py` / `discord.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `telegram.py` registers the "Working on it..." placeholder as the stream target and edits it in place via `editMessageText`; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
|
||||
- `github.py` - Webhook-driven GitHub channel. Inbound messages come from `POST /api/webhooks/github`; outbound is log-only because GitHub agents post explicitly with `gh` from their sandbox when they choose to comment or create a PR
|
||||
- `app/gateway/routers/channel_connections.py` - Browser-facing user connection and disconnect APIs
|
||||
- `deerflow.persistence.channel_connections` - SQL-backed user-owned connection, optional credential, connect state, and conversation store
|
||||
|
||||
**Message Flow**:
|
||||
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
|
||||
- For GitHub, the webhook router verifies the delivery then calls `fanout_event(bus, ...)`; matching agent bindings publish one `InboundMessage` each instead of a long-polling channel worker.
|
||||
2. `ChannelManager._dispatch_loop()` consumes from queue
|
||||
3. For user-owned channel connections, incoming messages carry `connection_id`, `owner_user_id`, and `workspace_id`; `owner_user_id` becomes the DeerFlow run `user_id`, while the raw platform user id remains `channel_user_id`. The Gateway forwards `channel_user_id` from `body.context` into the runtime context only (never `configurable`, which is checkpointed), and `bash_tool` exposes it to sandbox commands as the fixed env var `DEERFLOW_CHANNEL_USER_ID` — via a shell-quoted command-string prefix, NOT the `execute_command(env=...)` channel, which is reserved for request-scoped secrets and would switch `AioSandbox` onto the `bash.exec` path (image >= 1.9.3, fresh session per call). Per-call injection keeps group-chat identity correct (one thread/sandbox, many senders) **without depending on the AIO shell's session semantics**: every IM-channel command carries an explicit `export VAR=<id>; ` (valid id) or `unset VAR; ` (empty / non-str / over the 256-char cap, since `body.context` is client-writable). The AIO no-env path reuses a persistent shell session (the reason for the class lock, #1433), so a bare command could otherwise resolve a stale id an earlier sender exported; the `unset` closes the window the length/type guard would open (a dropped id would inherit the previous sender's value). Non-IM runs (no `channel_user_id` in context) are left untouched. Not injected on the Windows local sandbox (its PowerShell/cmd.exe fallback has no `export`/`unset`). Propagates across `task` delegation: `task_tool` captures the dispatching turn's id and the subagent executor forwards it into the subagent's runtime context, same as the guardrail attribution fields. The var is informational, never authorization-grade: any bash command can overwrite it (and web clients can set `body.context.channel_user_id`), so skills must not treat it as authenticated identity. Tests: `tests/test_channel_user_id_env.py`
|
||||
4. For chat: look up/create thread through Gateway's LangGraph-compatible API
|
||||
5. Feishu/Telegram chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
|
||||
6. Slack/Discord chat: `runs.wait()` → extract final response → publish outbound
|
||||
6b. GitHub chat (`ChannelRunPolicy.fire_and_forget=True`): `runs.create()` returns once the run is `pending`; the manager does not wait for the final state and does not publish an outbound. The agent posts its own reply mid-run via `gh` from the sandbox. `ConflictError` on a busy thread still trips the standard `THREAD_BUSY_MESSAGE` path (log-only on GitHub).
|
||||
7. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement). Messages already sent inside an existing Feishu topic carry a compact source-message preview in that card, and queued same-thread follow-ups patch their own source message's card from queued → running → final without falling back to the generic busy reply.
|
||||
8. Telegram streaming: the "Working on it..." placeholder message is registered as the stream target; non-final updates `editMessageText` it in place (channel-side throttle: 1s in private chats, 3s in groups due to Telegram's 20 msg/min group cap; 4096-char truncation; rate-limited updates dropped); the final update performs the last edit and splits >4096 texts into follow-up messages
|
||||
9. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` → create card with initial text → stream updates via `PUT /v1.0/card/streaming` → finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
|
||||
10. For commands (`/new`, `/status`, `/models`, `/memory`, `/goal`, `/help`): handle locally or query Gateway API
|
||||
11. Outbound → channel callbacks → platform reply
|
||||
- GitHub is the exception: the channel logs the final assistant message and does **not** auto-post it to GitHub. Agents use the sandbox `gh` CLI (`gh issue comment`, `gh pr comment`, `gh pr create`, etc.) for intentional writeback, so silence is cheap when several agents fan out on the same event.
|
||||
|
||||
**Owner-scoped file storage**: inbound files, uploads, and output artifacts are staged under the DeerFlow owner's bucket so they land where the agent run reads/writes (`users/{user_id}/threads/{thread_id}/user-data/{uploads,outputs}`). `ChannelManager._handle_chat` resolves the storage owner once via `_channel_storage_user_id(msg)` (sanitized owner id, falling back to `safe(msg.user_id)` for unbound auth-enabled channels — mirroring `_resolve_run_params`'s run identity; `None` only when no identity is available) and threads it as the `user_id=` kwarg through the file pipeline:
|
||||
- `Channel.receive_file(msg, thread_id, user_id=...)` — owner-bound channels persist downloaded files under the owner's bucket instead of the default bucket
|
||||
- `_ingest_inbound_files(...)` and the underlying `ensure_uploads_dir` / `get_uploads_dir` — owner-scoped via the same kwarg
|
||||
- `_resolve_attachments` / `_prepare_artifact_delivery` — resolve output artifacts from the bound owner's bucket
|
||||
The cached value is reused for both the blocking (`runs.wait`) and streaming (`_handle_streaming_chat`) paths, so uploads and artifact delivery always target the same bucket even if a channel returns a rewritten `InboundMessage` from `receive_file`. The bucket id matches the memory bucket resolved by `_resolve_memory_user_id` (both normalize through `make_safe_user_id`).
|
||||
|
||||
**Configuration** (`config.yaml` -> `channels`):
|
||||
- `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`)
|
||||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming), `github` (operator kill-switch `enabled`, plus `default_mention_login` for mention-required GitHub triggers)
|
||||
|
||||
**User-owned channel connections** (`config.yaml` -> `channel_connections`):
|
||||
- Disabled by default. It is a user-binding layer on top of the existing `channels.*` runtime config, not a replacement for provider bot credentials.
|
||||
- No public IP, OAuth callback URL, or provider webhook route is required by the current implementation.
|
||||
- Telegram uses a deep-link `/start <code>` flow over the existing long-polling worker. Slack, Discord, Feishu/Lark, DingTalk, WeChat, and WeCom use `/connect <code>` over their existing outbound channel workers.
|
||||
- Frontend APIs: `GET /api/channels/providers`, `GET /api/channels/connections`, `POST /api/channels/{provider}/connect`, and `DELETE /api/channels/connections/{connection_id}`.
|
||||
- Browser APIs remain protected by normal Gateway auth/CSRF. Provider messages arrive through the already-configured channel workers.
|
||||
- Provider-level `connection_status` reflects the user's newest connection row. With no binding it is `not_connected`, except in auth-disabled local mode where a configured running channel reports `connected` because all channel messages already route to the default user.
|
||||
- Slack replies use the configured operator bot token from `channels.slack` unless per-connection credentials are present; unreadable or corrupt stored credentials are treated as unavailable.
|
||||
- Telegram, Slack, Discord, Feishu/Lark, DingTalk, WeChat, and WeCom workers resolve incoming platform identities to connection records before reaching `ChannelManager`.
|
||||
- **Connect-code ordering vs `allowed_users`**: inbound workers consume a valid `/connect <code>` (or Telegram `/start <code>`) **before** applying the `allowed_users` filter, so a newly allowlisted-but-unbound user can bootstrap their first bind via the browser flow. Consequence: `allowed_users` is **not** a bind-time defense — any sender who possesses a valid code can consume it (not only allowlisted users). The bind security model rests on the code's confidentiality: `secrets.token_urlsafe(16)`, 600 s TTL, one-time `consume_oauth_state`, and codes surfaced only in the initiating browser (never echoed to chat). `allowed_users` still gates ordinary (non-bind) messages.
|
||||
- **Single-active-owner transfer semantics**: an external identity is keyed by `(provider, external_account_id, workspace_id)`. The latest successful bind wins — `upsert_connection` revokes other owners' active rows for the same identity (ownership transfer). This invariant is enforced at the DB layer by the partial unique index `uq_channel_connection_active_identity` (`WHERE status != 'revoked'`), so concurrent connects from different owners cannot both end `connected`; the losing writer retries against the now-visible state. `find_connection_by_external_identity` therefore resolves deterministically.
|
||||
- See `backend/docs/IM_CHANNEL_CONNECTIONS.md` for provider setup, operational notes, and the architecture diagrams (connect-code flow, single-active-owner transfer, sync vs streaming dispatch, owner-scoped file storage pipeline).
|
||||
|
||||
**GitHub event-driven agents** (webhook-driven IM channel):
|
||||
- Custom agents declare a `github:` block in their `config.yaml` to bind to repos and event triggers; the webhook route is fail-closed by default (mounted only when `GITHUB_WEBHOOK_SECRET` is set) and exempt from auth/CSRF because authenticity is enforced by HMAC.
|
||||
- Outbound is **log-only** by design: each agent posts its own reply mid-run via the `gh` CLI from its sandbox, so the manager uses `fire_and_forget=True` and `runs.create()` returns once pending.
|
||||
- See [backend/docs/GITHUB_AGENTS.md](docs/GITHUB_AGENTS.md) for the architecture diagrams: webhook → fan-out → `InboundMessage` dispatch, `preferred_thread_id = UUID5(repo, number, agent_name)` thread determinism, mention-handle precedence chain, GH token lifecycle via `GH_TOKEN`/`GITHUB_TOKEN` per-call `extra_env`, and the narrow `ConflictError` (HTTP 409) thread-create race recovery.
|
||||
|
||||
|
||||
### Memory System (`packages/harness/deerflow/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary
|
||||
- `prompt.py` - Prompt templates for memory updates
|
||||
- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple
|
||||
- `tools.py` - Tool-driven memory mode (`memory_search`, `memory_add`, `memory_update`, `memory_delete`) using the same storage/update primitives
|
||||
|
||||
**Per-User Isolation**:
|
||||
- Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json`
|
||||
- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json`
|
||||
- Custom agent definitions (`SOUL.md` + `config.yaml`) are also per-user at `{base_dir}/users/{user_id}/agents/{agent_name}/`. The legacy shared layout `{base_dir}/agents/{agent_name}/` remains read-only fallback for unmigrated installations
|
||||
- Middleware mode captures `user_id` via `get_effective_user_id()` at enqueue time; tool mode resolves `user_id` and `agent_name` from `ToolRuntime.context` via `resolve_runtime_user_id(runtime)` so tool calls stay scoped to the authenticated user and active custom agent
|
||||
- The `/api/memory*` endpoints resolve the owner through `_resolve_memory_user_id(request)`: trusted internal callers (IM channel workers carrying the `X-DeerFlow-Owner-User-Id` header, e.g. a bound `/memory` command) act for the connection owner; browser/API callers fall back to `get_effective_user_id()`. The header is only honored after `AuthMiddleware` validated the internal token, mirroring `get_trusted_internal_owner_user_id` used by the threads router
|
||||
- In no-auth mode, `user_id` defaults to `"default"` (constant `DEFAULT_USER_ID`)
|
||||
- Absolute `storage_path` in config opts out of per-user isolation
|
||||
- **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json`, `threads/`, and `agents/` into per-user layout. Supports `--dry-run` (preview changes) and `--user-id USER_ID` (assign unowned legacy data to a user, defaults to `default`).
|
||||
|
||||
**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`):
|
||||
- **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries)
|
||||
- **History**: `recentMonths`, `earlierContext`, `longTermBackground`
|
||||
- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source`
|
||||
|
||||
**Workflow**:
|
||||
- `memory.mode: middleware` (default) keeps the passive path: `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, queues conversation with the captured `user_id`, and the debounced background thread invokes the LLM to extract context updates and facts using the stored `user_id`.
|
||||
- `memory.mode: tool` skips `MemoryMiddleware` and registers `memory_search`, `memory_add`, `memory_update`, and `memory_delete` on the agent. The model decides when to search, add, update, or delete facts; this is opt-in/experimental and should not be described as better than middleware mode without eval evidence.
|
||||
- Both modes share `FileMemoryStorage`, per-user/per-agent isolation, prompt injection, manual CRUD primitives, and the updater backend.
|
||||
- Middleware mode queue debounces (30s default), batches updates, deduplicates per-thread, applies updates atomically (temp file + rename) with cache invalidation, and skips duplicate fact content before append.
|
||||
- Staleness pass (same LLM invocation as the regular updater, no extra API call): when `staleness_review_enabled` is `true` and at least `staleness_min_candidates` aged facts exist, `_select_stale_candidates` selects facts older than `staleness_age_days` that are not in `staleness_protected_categories` (default: `correction`), surfaces them in the prompt, and the LLM judges each as KEEP or REMOVE. `_apply_updates` enforces the guardrail unconditionally at apply time: it intersects the LLM-returned removal set with `_select_stale_candidates` output before applying the per-cycle cap (`staleness_max_removals_per_cycle`), so protected and non-aged facts can never be deleted regardless of model behavior or the feature flag setting.
|
||||
- Consolidation pass (same LLM invocation as the regular updater, no extra API call): when `consolidation_enabled` is `true` and at least one category holds `consolidation_min_facts` or more facts, `_select_consolidation_candidates` identifies fragmented categories and surfaces at most `consolidation_max_groups_per_cycle` of them (largest first) in the prompt. The LLM decides which groups to merge and proposes a synthesised fact per group. `_apply_updates` enforces guardrails: source IDs must exist and must not overlap across groups, group size is capped at `consolidation_max_sources`, the merged fact's confidence cannot exceed the source maximum, and facts below `fact_confidence_threshold` are not written.
|
||||
- Next interaction injects selected facts + context into `<memory>` tags in the system prompt when `injection_enabled` is true.
|
||||
|
||||
**Token counting** (`packages/harness/deerflow/agents/memory/prompt.py`):
|
||||
- `_count_tokens` budgets the injection. In default `tiktoken` mode, the encoding is loaded lazily and cached.
|
||||
- Failed tiktoken loads are cached with a timestamp. During the fixed cooldown (`_TIKTOKEN_RETRY_COOLDOWN_S`, 600s), callers fall back to char estimation immediately instead of re-triggering the blocking BPE download; after the cooldown, transient outages can self-heal without a restart.
|
||||
- In-flight loads are cached as a LOADING sentinel so concurrent callers fall back instead of spawning more blocking threads.
|
||||
- Set `memory.token_counting: char` to skip tiktoken entirely and use the network-free CJK-aware char estimate.
|
||||
|
||||
Focused regression coverage for the updater lives in `backend/tests/test_memory_updater.py`.
|
||||
|
||||
**Configuration** (`config.yaml` → `memory`):
|
||||
- `enabled` / `injection_enabled` - Master switches
|
||||
- `mode` - Operation mode: `middleware` (default passive background extraction) or `tool` (experimental model-driven memory tools). Modes are mutually exclusive.
|
||||
- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation)
|
||||
- `debounce_seconds` - Wait time before processing (default: 30)
|
||||
- `model_name` - LLM for updates (null = default model)
|
||||
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
|
||||
- `max_injection_tokens` - Token limit for prompt injection (2000)
|
||||
- `token_counting` - Token counting strategy for the injection budget: `tiktoken` (default, accurate but may download BPE data from a public endpoint on first use — can block for a long time in network-restricted environments, see issues #3402/#3429) or `char` (network-free CJK-aware char estimate, never touches tiktoken)
|
||||
- `staleness_review_enabled` - Enable proactive staleness pruning of aged facts (default: `true`; only triggers when aged candidates exist)
|
||||
- `staleness_age_days` - Age in days before a fact becomes a staleness candidate (default: 90; range: 30–365)
|
||||
- `staleness_min_candidates` - Minimum aged candidates required to trigger a review cycle (default: 3; range: 1–50)
|
||||
- `staleness_max_removals_per_cycle` - Maximum facts removed in a single cycle; lowest-confidence entries are kept when the LLM requests more (default: 10; range: 1–50)
|
||||
- `staleness_protected_categories` - Fact categories that are never pruned by staleness review (default: `["correction"]`)
|
||||
- `consolidation_enabled` - Enable memory consolidation (default: `true`; no extra API call — runs in the same LLM invocation as the normal memory update)
|
||||
- `consolidation_min_facts` - Minimum facts in a category to trigger consolidation review (default: 8; range: 3–30)
|
||||
- `consolidation_max_groups_per_cycle` - Maximum categories the LLM can merge in one cycle (default: 3; range: 1–10; also controls the LLM's prompt instruction)
|
||||
- `consolidation_max_sources` - Maximum source facts per merge group; prevents over-merging (default: 8; range: 2–20)
|
||||
|
||||
### Reflection System (`packages/harness/deerflow/reflection/`)
|
||||
|
||||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||||
|
||||
### Schema Migrations (`packages/harness/deerflow/persistence/migrations/`)
|
||||
|
||||
DeerFlow's application tables (`runs`, `threads_meta`, `feedback`, `users`, `run_events`, plus the four `channel_*` tables) are owned by alembic via a **hybrid bootstrap** strategy. LangGraph's checkpointer tables (`checkpoints`, `checkpoint_blobs`, `checkpoint_writes`, `checkpoint_migrations`) live in the same database but are owned by LangGraph and excluded from alembic's view via `migrations/_env_filters.py::include_object`.
|
||||
|
||||
**Convention**: every ORM model change (new column, new table, new index) MUST ship as an alembic revision under `migrations/versions/`. The Gateway runs `alembic upgrade head` automatically on startup; users do not run `alembic` manually in production.
|
||||
|
||||
**Hybrid bootstrap** (`persistence/bootstrap.py::bootstrap_schema`, invoked from `persistence/engine.py::init_engine`):
|
||||
|
||||
| DB state | Action |
|
||||
|-------------------------------------------|-----------------------------------------|
|
||||
| empty (no DeerFlow tables) | `create_all` + `alembic stamp head` |
|
||||
| legacy (DeerFlow tables, no `alembic_version`) | `create_all` (baseline tables only, backfill) + `alembic stamp 0001_baseline` + `upgrade head` |
|
||||
| versioned (`alembic_version` row exists) | `alembic upgrade head` |
|
||||
|
||||
The legacy branch handles pre-alembic databases that already have at least one DeerFlow-owned table. `create_all` runs first because stamping at `0001_baseline` makes alembic skip the baseline's own `create_table` DDL on the subsequent upgrade — so any baseline table introduced into `Base.metadata` after the user's DB was first provisioned (e.g. the `channel_*` tables from PR #1930 for users upgrading across multiple releases) would otherwise never be created, and the first request hitting that table would 500 with `no such table`. The backfill is **restricted to `_BASELINE_TABLE_NAMES`** so it does not also create tables that future revisions introduce — those revisions' own `op.create_table` would otherwise fail with `relation already exists`. A guard test pins `_BASELINE_TABLE_NAMES` against `0001_baseline.upgrade()`'s actual output, so editing 0001 to add or remove a table forces a matching update to the constant. Column-level shape (pre-#3658 vs post-#3658 vs manual-ALTER for `token_usage_by_model`) is answered by each `versions/*.py` revision via the idempotent helpers in `migrations/_helpers.py` (`safe_add_column` / `safe_drop_column`) which no-op when the change is already present and `logger.warning` on shape drift. **Adding a new ORM column / table only requires a new revision file — no edit to `bootstrap.py` is needed** *unless* the new revision adds a new baseline table (rare; only happens when a new model is part of the baseline rather than introduced by its own revision).
|
||||
|
||||
The empty-DB path keeps using `create_all` because `Base.metadata` is the only authoritative schema source — `create_all` renders both SQLite (JSON, type affinity) and Postgres (JSONB, partial indexes) correctly without anyone having to keep a hand-written baseline in lockstep. `0001_baseline.upgrade()` is therefore almost never executed in practice; it exists as a stamp target + chain root.
|
||||
|
||||
**Concurrency safety**: Postgres uses `pg_advisory_lock` to serialise concurrent Gateway instances. SQLite uses a per-engine `asyncio.Lock` for same-process startup and is best-effort across processes via SQLite's file-level write lock + `PRAGMA busy_timeout`; multi-instance deployments should use Postgres. Column revisions in `versions/` additionally use idempotent helpers (`_helpers.py::safe_add_column`, `safe_drop_column`) so repeated post-baseline changes and retries are no-ops when the change is already present.
|
||||
|
||||
**Authoring a new revision**:
|
||||
```bash
|
||||
cd backend && make migrate-rev MSG="add foo column to runs"
|
||||
```
|
||||
This invokes `alembic revision --autogenerate` against the live ORM models. Review the generated file under `migrations/versions/` and switch raw `op.add_column` / `op.drop_column` calls to the idempotent helpers from `_helpers.py` before committing. There is no `make migrate` / `make migrate-stamp` target on purpose — the only execution path is Gateway startup, which keeps operational mistakes off the table.
|
||||
|
||||
**Where things live**:
|
||||
- `migrations/env.py` — alembic env, delegates filter to `_env_filters.py`, sets `render_as_batch=True` for SQLite ALTER support
|
||||
- `migrations/_env_filters.py::include_object` — drops LangGraph checkpointer tables from alembic's view
|
||||
- `migrations/_helpers.py` — `safe_add_column` / `safe_drop_column`
|
||||
- `migrations/versions/0001_baseline.py` — chain root, matches the schema `create_all` produces from `Base.metadata`
|
||||
- `migrations/versions/0002_runs_token_usage.py` — fixes issue #3682
|
||||
- `persistence/bootstrap.py` — `bootstrap_schema(engine, backend=...)`, the three-branch decision + locking
|
||||
- Tests: `tests/test_persistence_bootstrap.py` (branches), `tests/test_persistence_bootstrap_concurrency.py` (concurrency), `tests/test_persistence_bootstrap_regression.py` (issue #3682), `tests/test_persistence_migrations_env.py` (filter), `tests/blocking_io/test_persistence_bootstrap.py` (asyncio.to_thread anchor)
|
||||
|
||||
### Terminal Workbench / TUI (`packages/harness/deerflow/tui/`)
|
||||
|
||||
A terminal-native UI over the embedded harness, exposed as the `deerflow` console script (`[project.scripts]` in `packages/harness/pyproject.toml`). It is a UI shell over `DeerFlowClient` and does **not** fork agent behavior. `textual` is an optional dependency (`deerflow-harness[tui]`; also in the backend dev group); the console script degrades to headless help when it is absent. Full guide: [docs/TUI.md](docs/TUI.md).
|
||||
|
||||
**Module layout** (all layers except `app.py` are pure / Textual-free and unit-tested directly):
|
||||
- `cli.py` — `plan_launch()` (pure launch-mode decision) + headless `--print` / `--json` + `main()` entry point. TTY → TUI, else headless help. Uses an **absolute** `from deerflow.tui.app import run_tui` so the `app.py` module name doesn't trip `test_harness_boundary.py` (which records relative import module names verbatim).
|
||||
- `view_state.py` — `ViewState` + `reduce(state, action)`, the testable heart. Rows: user / assistant / tool / system. Title captured from `values` events.
|
||||
- `runtime.py` — `translate(StreamEvent) -> [Action]` (pure) + `stream_actions()` which brackets a run with `RunStarted`/`RunEnded` and turns model errors into an `AssistantError` row.
|
||||
- `message_format.py` / `command_registry.py` / `input_history.py` / `render.py` / `theme.py` — pure helpers (tool summaries, slash registry + `resolve()`, ↑/↓ history, Rich renderers).
|
||||
- `app.py` — Textual `App`. Runs `DeerFlowClient.stream()` (sync) on a worker thread and marshals actions to the UI thread via `call_from_thread`. Slash palette with `/goal` management + model/thread modal pickers; priority key bindings gated by `check_action` so they never steal keys from overlays or the composer.
|
||||
- `session.py` / `persistence.py` — builds the client + checkpointer and the `ThreadMetaWriter`.
|
||||
|
||||
**Web UI visibility**: the Web UI lists threads from the `threads_meta` SQL table (user-scoped), not the checkpointer. `persistence.py` writes a `threads_meta` row under the default user (`"default"`) into the same DB the Gateway reads — via the harness-only `deerflow.persistence.engine.init_engine_from_config()` — so TUI sessions appear in the Web UI sidebar **without** running the Gateway. Best-effort: a no-op on the `memory` backend. All DB work runs on one long-lived background event loop (a SQLAlchemy async engine is bound to its creating loop).
|
||||
|
||||
**Tests**: `tests/test_tui_*.py` — pure layers via plain pytest, the app/palette/overlays via Textual's pilot harness with a fake in-process session, and `test_tui_persistence.py` for the `threads_meta` round-trip.
|
||||
|
||||
### Request Trace Context (`packages/harness/deerflow/trace_context.py`)
|
||||
|
||||
Request trace correlation is controlled by `logging.enhance.enabled` at **both** entry points, gated through the shared helper `deerflow.config.app_config.is_trace_correlation_enabled` so the Gateway and embedded paths cannot drift:
|
||||
|
||||
- **Gateway HTTP**: `app.gateway.trace_middleware.TraceMiddleware` binds one request-level trace id per HTTP request, inheriting inbound `X-Trace-Id` when present or generating a new id otherwise. The middleware writes the final value to every HTTP response at `http.response.start`, which covers SSE / streaming responses without consuming the body.
|
||||
- **Embedded / TUI / CLI**: `DeerFlowClient.stream()` mints (or inherits) a request-level trace id per turn only when the flag is on. When it is off, no fresh id is minted — a caller that explicitly wraps `stream()` in `request_trace_context(...)` still opts in, because the downstream `get_current_trace_id()` read propagates that value into Langfuse metadata regardless of the flag. Because `stream()` is a sync generator (which shares the caller's context), the id binding is set/reset around each `next()` step rather than around `yield from`: this keeps LangGraph node execution and its log records inside the binding, while returning control to the caller with the ContextVar restored — avoids cross-request leak between yields and `ValueError: <Token> was created in a different Context` on GC-driven close of an abandoned generator (regression pinned by `tests/test_client_langfuse_metadata.py::test_stream_does_not_leak_trace_id_to_caller_context_between_yields` and `::test_stream_abandoned_generator_close_does_not_raise_cross_context`).
|
||||
|
||||
The same ContextVar value is injected into enhanced log records as `trace_id` and into Langfuse metadata as `deerflow_trace_id`.
|
||||
|
||||
`logging` is registered as a **restart-required** field
|
||||
(`STARTUP_ONLY_FIELDS["logging"]`): `configure_logging()` installs the trace-context
|
||||
filter and enhanced formatter on root handlers only during app.py lifespan startup,
|
||||
and `TraceMiddleware` captures `logging.enhance.enabled` once when the FastAPI app
|
||||
is constructed (via `resolve_trace_enabled(get_app_config())` in `create_app()`,
|
||||
itself a thin alias for `is_trace_correlation_enabled`). This keeps the response
|
||||
`X-Trace-Id` header, log `trace_id` fields, and Langfuse `deerflow_trace_id`
|
||||
coherent — a runtime `config.yaml` edit to `logging.enhance.*` needs a Gateway
|
||||
restart to take effect. The `deerflow_trace_id` chain inherits this guarantee
|
||||
transitively because every injection point ultimately reads the same
|
||||
`trace_context` ContextVar that the middleware alone populates. `DeerFlowClient`
|
||||
reads its own `self._app_config` snapshot (captured at `__init__`) through the
|
||||
same helper for the embedded gate.
|
||||
|
||||
`deerflow_trace_id` is a DeerFlow correlation metadata key, not Langfuse's native
|
||||
trace id and not a DeerFlow `run_id`. Keep the existing subagent `trace_id` field
|
||||
separate: that short id is still only for subagent execution logs/status.
|
||||
|
||||
### Tracing System (`packages/harness/deerflow/tracing/`)
|
||||
|
||||
LangSmith and Langfuse are both supported. The wiring lives in two layers:
|
||||
|
||||
- `factory.py::build_tracing_callbacks()` — returns the LangChain `CallbackHandler` list for the providers currently enabled via env vars (`LANGSMITH_TRACING`, `LANGFUSE_TRACING`, etc.). The handlers are attached at the **graph invocation root** for in-graph runs (`make_lead_agent` and `DeerFlowClient.stream` both append them to `config["callbacks"]` before invoking the graph) so a single run produces one trace with all node / LLM / tool calls as child spans. Standalone callers — anything that invokes a model outside such a graph (e.g. `MemoryUpdater`) — keep `create_chat_model`'s default `attach_tracing=True`, which falls back to model-level callback attachment.
|
||||
- `metadata.py::build_langfuse_trace_metadata()` — builds the Langfuse-reserved trace attributes for `RunnableConfig.metadata`. The Langfuse v4 `langchain.CallbackHandler` lifts these onto the root trace (see its `_parse_langfuse_trace_attributes`), but only when it sees `on_chain_start(parent_run_id=None)` — which is why the callbacks have to live at the graph root, not the model.
|
||||
|
||||
**Trace-attribute injection points**: both `runtime/runs/worker.py::run_agent` (gateway path) and `client.py::DeerFlowClient.stream` (embedded path) merge the metadata into `config["metadata"]` right before constructing the graph. `subagents/executor.py::_aexecute` does the same for every subagent run so subagent traces group under the parent thread's session card (carrying the parent `thread_id` → `langfuse_session_id`, the user_id captured at `task_tool` → `langfuse_user_id`, and a `subagent:<normalized-name>` trace name). Caller-supplied keys win via `setdefault`, so an external `session_id` override is preserved. Field mapping:
|
||||
|
||||
| Langfuse field | Source |
|
||||
|-----------------------|----------------------------------------------|
|
||||
| `langfuse_session_id` | LangGraph `thread_id` |
|
||||
| `langfuse_user_id` | `get_effective_user_id()` (`default` in no-auth); for subagents, captured from `runtime.context` at `task_tool` time via `resolve_runtime_user_id()` |
|
||||
| `langfuse_trace_name` | `RunRecord.assistant_id` / client `agent_name` (defaults to `lead-agent`); for subagents, `subagent:<name>` (lowercased, `_` → `-`) |
|
||||
| `langfuse_tags` | `env:<DEER_FLOW_ENV>` + `model:<model_name>` |
|
||||
| `deerflow_trace_id` | Current request/entry trace id from `deerflow.trace_context`; matches `X-Trace-Id` for enhanced Gateway HTTP requests. Gated by `logging.enhance.enabled` in both gateway and embedded paths via `is_trace_correlation_enabled` — off by default; embedded callers can still opt in per-turn by wrapping `stream()` in `request_trace_context(...)` |
|
||||
|
||||
Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only deployments are unaffected. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment. Tests live in `tests/test_tracing_factory.py`, `tests/test_tracing_metadata.py`, `tests/test_worker_langfuse_metadata.py`, `tests/test_client_langfuse_metadata.py`, and `tests/test_subagent_executor.py::TestSubagentTracingWiring`.
|
||||
|
||||
### Config Schema
|
||||
|
||||
**`config.yaml`** key sections:
|
||||
- `models[]` - LLM configs with `use` class path, `supports_thinking`, `supports_vision`, provider-specific fields
|
||||
- `logging.enhance` - Optional request trace correlation (`enabled`, `format`) for Gateway `X-Trace-Id`, log `trace_id`, and Langfuse `deerflow_trace_id`
|
||||
- vLLM reasoning models should use `deerflow.models.vllm_provider:VllmChatModel`; for Qwen-style parsers prefer `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking`, and DeerFlow will also normalize the older `thinking` alias
|
||||
- `tools[]` - Tool configs with `use` variable path and `group`
|
||||
- `tool_groups[]` - Logical groupings for tools
|
||||
- `sandbox.use` - Sandbox provider class path
|
||||
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
|
||||
- `skills.deferred_discovery` - When `true`, replaces the full-metadata `<available_skills>` prompt block with a compact `<skill_index>` (names only) and registers the `describe_skill` tool so the agent fetches metadata on demand. Defaults to `false` (legacy full-metadata injection)
|
||||
- `title` - Auto-title generation (enabled, max_words, max_chars, model_name; null model_name uses fast local fallback, explicit model_name uses the prompt_template LLM path)
|
||||
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
|
||||
- `subagents.enabled` - Master switch for subagent delegation
|
||||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens, staleness_review_enabled, staleness_age_days, staleness_min_candidates, staleness_max_removals_per_cycle, staleness_protected_categories)
|
||||
|
||||
**`extensions_config.json`**:
|
||||
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description, `routing`, `tools`, `tool_call_timeout`). `routing.mode="prefer"` emits `<mcp_routing_hints>` prompt guidance; if `tool_search` defers the hinted tool, `McpRoutingMiddleware` can also auto-promote matching deferred schemas before the model call. It does not hard-disable other tools.
|
||||
- `tool_search.auto_promote_top_k` - Global MCP routing auto-promote breadth. Default `3`, clamped to `1..5`; applies only when `tool_search.enabled=true` and only to policy-filtered deferred MCP tools with `routing.mode="prefer"` and non-empty keywords.
|
||||
- `skills` - Map of skill name → state (enabled)
|
||||
|
||||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||||
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
|
||||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||||
|
||||
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
|
||||
|
||||
**Agent Conversation**:
|
||||
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
|
||||
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
|
||||
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
|
||||
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
|
||||
- `"custom"` — forwarded from `StreamWriter`
|
||||
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
|
||||
- Agent created lazily via `create_agent()` + `build_middlewares()`, same as `make_lead_agent`
|
||||
- Supports `checkpointer` parameter for state persistence across turns
|
||||
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
|
||||
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
|
||||
|
||||
**Gateway Equivalent Methods** (replaces Gateway API):
|
||||
|
||||
| Category | Methods | Return format |
|
||||
|----------|---------|---------------|
|
||||
| Models | `list_models()`, `get_model(name)` | `{"models": [...]}`, `{name, display_name, ...}` |
|
||||
| MCP | `get_mcp_config()`, `update_mcp_config(servers)` | `{"mcp_servers": {...}}` |
|
||||
| Skills | `list_skills()`, `get_skill(name)`, `update_skill(name, enabled)`, `install_skill(path)` | `{"skills": [...]}` |
|
||||
| Goals | `get_goal(thread_id)`, `set_goal(thread_id, objective, max_continuations=8)`, `clear_goal(thread_id)` | `{"goal": {...}}` or `{"goal": None}` |
|
||||
| Memory | `get_memory()`, `reload_memory()`, `get_memory_config()`, `get_memory_status()` | dict |
|
||||
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
|
||||
| Artifacts | `get_artifact(thread_id, path)` → `(bytes, mime_type)` | tuple |
|
||||
|
||||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||||
|
||||
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
|
||||
|
||||
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Test-Driven Development (TDD) — MANDATORY
|
||||
|
||||
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
|
||||
|
||||
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
|
||||
- Run the full suite before and after your change: `make test`
|
||||
- Tests must pass before a feature is considered complete
|
||||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Run a specific test file
|
||||
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v
|
||||
```
|
||||
|
||||
### Running the Full Application
|
||||
|
||||
From the **project root** directory:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
This starts all services and makes the application available at `http://localhost:2026`.
|
||||
|
||||
**All startup modes:**
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
**Nginx routing**:
|
||||
- `/api/langgraph/*` → Gateway embedded runtime (8001), rewritten to `/api/*`
|
||||
- `/api/*` (other) → Gateway API (8001)
|
||||
- `/` (non-API) → Frontend (3000)
|
||||
|
||||
### Running Backend Services Separately
|
||||
|
||||
From the **backend** directory:
|
||||
|
||||
```bash
|
||||
# Gateway API
|
||||
make gateway
|
||||
```
|
||||
|
||||
Direct access (without nginx):
|
||||
- Gateway: `http://localhost:8001`
|
||||
|
||||
### Frontend Configuration
|
||||
|
||||
The frontend uses environment variables to connect to backend services:
|
||||
- `NEXT_PUBLIC_LANGGRAPH_BASE_URL` - Defaults to `/api/langgraph` (through nginx)
|
||||
- `NEXT_PUBLIC_BACKEND_BASE_URL` - Defaults to empty string (through nginx)
|
||||
|
||||
When using `make dev` from root, the frontend automatically connects through nginx.
|
||||
|
||||
## Key Features
|
||||
|
||||
### File Upload
|
||||
|
||||
Multi-file upload with automatic document conversion:
|
||||
- Endpoint: `POST /api/threads/{thread_id}/uploads`
|
||||
- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`)
|
||||
- Rejects directory inputs before copying so uploads stay all-or-nothing
|
||||
- Reuses one conversion worker per request when called from an active event loop
|
||||
- Files stored in thread-isolated directories under the resolving user's bucket (`users/{user_id}/threads/{thread_id}/user-data/uploads`). For IM channels the owner is threaded explicitly via the `user_id=` kwarg (see IM Channels → Owner-scoped file storage); HTTP/embedded callers resolve it from `get_effective_user_id()`
|
||||
- Duplicate filenames in a single upload request are auto-renamed with `_N` suffixes so later files do not truncate earlier files
|
||||
- Gateway HTTP uploads stage bytes as `.upload-*.part` files and atomically replace the destination only after size validation. These staging files are hidden from upload listings, agent upload context, and sandbox listing/search tools, and swept on Gateway startup if a hard crash leaves one behind.
|
||||
- Gateway HTTP upload/list/delete handlers offload filesystem work through `deerflow.utils.file_io.run_file_io`, a dedicated ContextVar-preserving file IO executor. Non-mounted sandbox uploads acquire sandboxes with `SandboxProvider.acquire_async()` and offload `read_bytes()` plus `sandbox.update_file()` together.
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
|
||||
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||||
|
||||
### Plan Mode
|
||||
|
||||
TodoList middleware for complex multi-step tasks:
|
||||
- Controlled via runtime config: `config.configurable.is_plan_mode = True`
|
||||
- Provides `write_todos` tool for task tracking
|
||||
- One task in_progress at a time, real-time updates
|
||||
|
||||
See [docs/plan_mode_usage.md](docs/plan_mode_usage.md) for details.
|
||||
|
||||
### Context Summarization
|
||||
|
||||
Automatic conversation summarization when approaching token limits:
|
||||
- Configured in `config.yaml` under `summarization` key
|
||||
- Trigger types: tokens, messages, or fraction of max input
|
||||
- Keeps recent messages while summarizing older ones
|
||||
- Manual compaction uses `POST /api/threads/{id}/compact`, reuses the same
|
||||
`DeerFlowSummarizationMiddleware`, writes a new checkpoint with updated
|
||||
`messages` and `summary_text`, and bumps only those channel versions.
|
||||
The route shares the per-thread serialization gate used by `/goal` writes
|
||||
and run admission so compaction cannot race with goal updates or runs that
|
||||
read/write checkpoints.
|
||||
|
||||
See [docs/summarization.md](docs/summarization.md) for details.
|
||||
|
||||
### Vision Support
|
||||
|
||||
For models with `supports_vision: true`:
|
||||
- `ViewImageMiddleware` processes images in conversation
|
||||
- `view_image_tool` added to agent's toolset
|
||||
- Images automatically converted to base64 and injected into state
|
||||
|
||||
## Code Style
|
||||
|
||||
- Uses `ruff` for linting and formatting
|
||||
- Line length: 240 characters
|
||||
- Python 3.12+ with type hints
|
||||
- Double quotes, space indentation
|
||||
|
||||
## Documentation
|
||||
|
||||
See `docs/` directory for detailed documentation:
|
||||
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
|
||||
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
|
||||
- [API.md](docs/API.md) - API reference
|
||||
- [SETUP.md](docs/SETUP.md) - Setup guide
|
||||
- [FILE_UPLOAD.md](docs/FILE_UPLOAD.md) - File upload feature
|
||||
- [PATH_EXAMPLES.md](docs/PATH_EXAMPLES.md) - Path types and usage
|
||||
- [summarization.md](docs/summarization.md) - Context summarization
|
||||
- [plan_mode_usage.md](docs/plan_mode_usage.md) - Plan mode with TodoList
|
||||
@@ -0,0 +1,5 @@
|
||||
# CLAUDE.md
|
||||
|
||||
The backend agent guidance lives in [AGENTS.md](AGENTS.md) so it is shared across coding agents (Claude Code, Codex, and others). Claude Code imports it below.
|
||||
|
||||
@AGENTS.md
|
||||
@@ -0,0 +1,399 @@
|
||||
# Contributing to DeerFlow Backend
|
||||
|
||||
Thank you for your interest in contributing to DeerFlow! This document provides guidelines and instructions for contributing to the backend codebase.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Getting Started](#getting-started)
|
||||
- [Development Setup](#development-setup)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Code Style](#code-style)
|
||||
- [Making Changes](#making-changes)
|
||||
- [Testing](#testing)
|
||||
- [Pull Request Process](#pull-request-process)
|
||||
- [Architecture Guidelines](#architecture-guidelines)
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12 or higher
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
- Git
|
||||
- Docker (optional, for Docker sandbox testing)
|
||||
|
||||
### Fork and Clone
|
||||
|
||||
1. Fork the repository on GitHub
|
||||
2. Clone your fork locally:
|
||||
```bash
|
||||
git clone https://github.com/YOUR_USERNAME/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Install backend dependencies
|
||||
cd backend
|
||||
make install
|
||||
```
|
||||
|
||||
### Configure Environment
|
||||
|
||||
Set up your API keys for testing:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="your-api-key"
|
||||
# Add other keys as needed
|
||||
```
|
||||
|
||||
### Run the Development Server
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
make dev
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
backend/
|
||||
├── packages/harness/deerflow/ # deerflow-harness package (import: deerflow.*)
|
||||
│ ├── agents/ # Agent system
|
||||
│ │ ├── lead_agent/ # Main agent (agent.py factory, prompt.py)
|
||||
│ │ ├── middlewares/ # Agent middleware chain
|
||||
│ │ ├── memory/ # Memory extraction & storage
|
||||
│ │ └── thread_state.py # Thread state definition
|
||||
│ ├── sandbox/ # Sandbox execution
|
||||
│ │ ├── local/ # Local sandbox provider
|
||||
│ │ ├── sandbox.py # Abstract interface
|
||||
│ │ ├── tools.py # Sandbox tools (bash, file ops)
|
||||
│ │ └── middleware.py # Sandbox lifecycle
|
||||
│ ├── subagents/ # Subagent delegation
|
||||
│ ├── tools/builtins/ # Built-in tools
|
||||
│ ├── mcp/ # MCP integration
|
||||
│ ├── models/ # Model factory
|
||||
│ ├── skills/ # Skills system
|
||||
│ ├── config/ # Configuration system
|
||||
│ ├── runtime/ # Embedded run execution (RunManager, StreamBridge)
|
||||
│ ├── persistence/ # Checkpointer/store engines & schema migrations
|
||||
│ ├── guardrails/ # Pre-tool-call authorization providers
|
||||
│ ├── tracing/ # Tracer factory & trace metadata
|
||||
│ ├── uploads/ # Uploads manager
|
||||
│ ├── tui/ # Terminal UI (`deerflow` console script)
|
||||
│ ├── community/ # Community tools (tavily/, jina_ai/, firecrawl/, …)
|
||||
│ ├── reflection/ # Dynamic module loading
|
||||
│ └── utils/ # Utilities
|
||||
└── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||
├── gateway/ # Gateway API
|
||||
│ ├── app.py # FastAPI application
|
||||
│ └── routers/ # Route handlers (threads, models, mcp, skills, uploads, …)
|
||||
└── channels/ # IM channel integrations (Feishu, Slack, Telegram, …)
|
||||
```
|
||||
|
||||
See [AGENTS.md](AGENTS.md) for the full module-by-module breakdown.
|
||||
|
||||
## Code Style
|
||||
|
||||
### Linting and Formatting
|
||||
|
||||
We use `ruff` for both linting and formatting:
|
||||
|
||||
```bash
|
||||
# Check for issues
|
||||
make lint
|
||||
|
||||
# Auto-fix and format
|
||||
make format
|
||||
```
|
||||
|
||||
### Style Guidelines
|
||||
|
||||
- **Line length**: 240 characters maximum
|
||||
- **Python version**: 3.12+ features allowed
|
||||
- **Type hints**: Use type hints for function signatures
|
||||
- **Quotes**: Double quotes for strings
|
||||
- **Indentation**: 4 spaces (no tabs)
|
||||
- **Imports**: Group by standard library, third-party, local
|
||||
|
||||
### Docstrings
|
||||
|
||||
Use docstrings for public functions and classes:
|
||||
|
||||
```python
|
||||
def create_chat_model(name: str, thinking_enabled: bool = False) -> BaseChatModel:
|
||||
"""Create a chat model instance from configuration.
|
||||
|
||||
Args:
|
||||
name: The model name as defined in config.yaml
|
||||
thinking_enabled: Whether to enable extended thinking
|
||||
|
||||
Returns:
|
||||
A configured LangChain chat model instance
|
||||
|
||||
Raises:
|
||||
ValueError: If the model name is not found in configuration
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Branch Naming
|
||||
|
||||
Use descriptive branch names:
|
||||
|
||||
- `feature/add-new-tool` - New features
|
||||
- `fix/sandbox-timeout` - Bug fixes
|
||||
- `docs/update-readme` - Documentation
|
||||
- `refactor/config-system` - Code refactoring
|
||||
|
||||
### Commit Messages
|
||||
|
||||
Write clear, concise commit messages:
|
||||
|
||||
```
|
||||
feat: add support for Claude 3.5 model
|
||||
|
||||
- Add model configuration in config.yaml
|
||||
- Update model factory to handle Claude-specific settings
|
||||
- Add tests for new model
|
||||
```
|
||||
|
||||
Prefix types:
|
||||
- `feat:` - New feature
|
||||
- `fix:` - Bug fix
|
||||
- `docs:` - Documentation
|
||||
- `refactor:` - Code refactoring
|
||||
- `test:` - Tests
|
||||
- `chore:` - Build/config changes
|
||||
|
||||
## Testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
### Writing Tests
|
||||
|
||||
Place tests in the `tests/` directory mirroring the source structure:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── test_models/
|
||||
│ └── test_factory.py
|
||||
├── test_sandbox/
|
||||
│ └── test_local.py
|
||||
└── test_gateway/
|
||||
└── test_models_router.py
|
||||
```
|
||||
|
||||
Example test:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from deerflow.models.factory import create_chat_model
|
||||
|
||||
def test_create_chat_model_with_valid_name():
|
||||
"""Test that a valid model name creates a model instance."""
|
||||
model = create_chat_model("gpt-4")
|
||||
assert model is not None
|
||||
|
||||
def test_create_chat_model_with_invalid_name():
|
||||
"""Test that an invalid model name raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
create_chat_model("nonexistent-model")
|
||||
```
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
### Before Submitting
|
||||
|
||||
1. **Ensure tests pass**: `uv run pytest`
|
||||
2. **Run linter**: `make lint`
|
||||
3. **Format code**: `make format`
|
||||
4. **Update documentation** if needed
|
||||
|
||||
### PR Description
|
||||
|
||||
Include in your PR description:
|
||||
|
||||
- **What**: Brief description of changes
|
||||
- **Why**: Motivation for the change
|
||||
- **How**: Implementation approach
|
||||
- **Testing**: How you tested the changes
|
||||
|
||||
### Review Process
|
||||
|
||||
1. Submit PR with clear description
|
||||
2. Address review feedback
|
||||
3. Ensure CI passes
|
||||
4. Maintainer will merge when approved
|
||||
|
||||
## Architecture Guidelines
|
||||
|
||||
### Adding New Tools
|
||||
|
||||
1. Create tool in `packages/harness/deerflow/tools/builtins/` or `packages/harness/deerflow/community/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/tools/builtins/my_tool.py
|
||||
from langchain_core.tools import tool
|
||||
|
||||
@tool
|
||||
def my_tool(param: str) -> str:
|
||||
"""Tool description for the agent.
|
||||
|
||||
Args:
|
||||
param: Description of the parameter
|
||||
|
||||
Returns:
|
||||
Description of return value
|
||||
"""
|
||||
return f"Result: {param}"
|
||||
```
|
||||
|
||||
2. Register in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
tools:
|
||||
- name: my_tool
|
||||
group: my_group
|
||||
use: deerflow.tools.builtins.my_tool:my_tool
|
||||
```
|
||||
|
||||
### Adding New Middleware
|
||||
|
||||
1. Create middleware in `packages/harness/deerflow/agents/middlewares/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/agents/middlewares/my_middleware.py
|
||||
from langchain.agents.middleware import BaseMiddleware
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
class MyMiddleware(BaseMiddleware):
|
||||
"""Middleware description."""
|
||||
|
||||
def transform_state(self, state: dict, config: RunnableConfig) -> dict:
|
||||
"""Transform the state before agent execution."""
|
||||
# Modify state as needed
|
||||
return state
|
||||
```
|
||||
|
||||
2. Register in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
|
||||
```python
|
||||
middlewares = [
|
||||
ThreadDataMiddleware(),
|
||||
SandboxMiddleware(),
|
||||
MyMiddleware(), # Add your middleware
|
||||
TitleMiddleware(),
|
||||
ClarificationMiddleware(),
|
||||
]
|
||||
```
|
||||
|
||||
### Adding New API Endpoints
|
||||
|
||||
1. Create router in `app/gateway/routers/`:
|
||||
|
||||
```python
|
||||
# app/gateway/routers/my_router.py
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/my-endpoint", tags=["my-endpoint"])
|
||||
|
||||
@router.get("/")
|
||||
async def get_items():
|
||||
"""Get all items."""
|
||||
return {"items": []}
|
||||
|
||||
@router.post("/")
|
||||
async def create_item(data: dict):
|
||||
"""Create a new item."""
|
||||
return {"created": data}
|
||||
```
|
||||
|
||||
2. Register in `app/gateway/app.py`:
|
||||
|
||||
```python
|
||||
from app.gateway.routers import my_router
|
||||
|
||||
app.include_router(my_router.router)
|
||||
```
|
||||
|
||||
### Configuration Changes
|
||||
|
||||
When adding new configuration options:
|
||||
|
||||
1. Update `packages/harness/deerflow/config/app_config.py` with new fields
|
||||
2. Add default values in `config.example.yaml`
|
||||
3. Document in `docs/CONFIGURATION.md`
|
||||
|
||||
### MCP Server Integration
|
||||
|
||||
To add support for a new MCP server:
|
||||
|
||||
1. Add configuration in `extensions_config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"my-server": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@my-org/mcp-server"],
|
||||
"description": "My MCP Server"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Update `extensions_config.example.json` with the new server
|
||||
|
||||
### Skills Development
|
||||
|
||||
To create a new skill:
|
||||
|
||||
1. Create directory in `skills/public/` or `skills/custom/`:
|
||||
|
||||
```
|
||||
skills/public/my-skill/
|
||||
└── SKILL.md
|
||||
```
|
||||
|
||||
2. Write `SKILL.md` with YAML front matter:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: My Skill
|
||||
description: What this skill does
|
||||
license: MIT
|
||||
allowed-tools:
|
||||
- read_file
|
||||
- write_file
|
||||
- bash
|
||||
---
|
||||
|
||||
# My Skill
|
||||
|
||||
Instructions for the agent when this skill is enabled...
|
||||
```
|
||||
|
||||
## Questions?
|
||||
|
||||
If you have questions about contributing:
|
||||
|
||||
1. Check existing documentation in `docs/`
|
||||
2. Look for similar issues or PRs on GitHub
|
||||
3. Open a discussion or issue on GitHub
|
||||
|
||||
Thank you for contributing to DeerFlow!
|
||||
@@ -0,0 +1,116 @@
|
||||
# Backend Dockerfile — multi-stage build
|
||||
# Stage 1 (builder): compiles native Python extensions with build-essential
|
||||
# Stage 2 (dev): retains toolchain for dev containers (uv sync at startup)
|
||||
# Stage 3 (runtime): clean image without compiler toolchain for production
|
||||
|
||||
# UV source image (override for restricted networks that cannot reach ghcr.io)
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
|
||||
FROM ${UV_IMAGE} AS uv-source
|
||||
|
||||
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
|
||||
FROM python:3.12-slim-bookworm AS builder
|
||||
|
||||
ARG NODE_MAJOR=22
|
||||
ARG APT_MIRROR
|
||||
ARG UV_INDEX_URL
|
||||
# Optional extras to install (e.g. "postgres" for PostgreSQL support)
|
||||
# Usage: docker build --build-arg UV_EXTRAS=postgres,discord ...
|
||||
ARG UV_EXTRAS
|
||||
|
||||
# Optionally override apt mirror for restricted networks (e.g. APT_MIRROR=mirrors.aliyun.com)
|
||||
RUN if [ -n "${APT_MIRROR}" ]; then \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list.d/debian.sources 2>/dev/null || true; \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list 2>/dev/null || true; \
|
||||
fi
|
||||
|
||||
# Install build tools + Node.js (build-essential needed for native Python extensions)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
build-essential \
|
||||
gnupg \
|
||||
ca-certificates \
|
||||
&& mkdir -p /etc/apt/keyrings \
|
||||
&& curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
|
||||
&& echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend source code
|
||||
COPY backend ./backend
|
||||
|
||||
# Install dependencies with cache mount
|
||||
# `redis` is always installed because Docker Compose defaults the stream bridge
|
||||
# to Redis (DEER_FLOW_STREAM_BRIDGE_REDIS_URL); it is an optional extra elsewhere
|
||||
# so slim non-Docker installs do not pull it.
|
||||
# When UV_EXTRAS is set (comma- or whitespace-separated), installs optional dependencies.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
sh -c 'cd backend && \
|
||||
set -f; \
|
||||
extras_flags=""; \
|
||||
for extra in $(printf "%s" "$UV_EXTRAS" | tr "," " "); do \
|
||||
case "$extra" in \
|
||||
[!A-Za-z]* | *[!A-Za-z0-9_-]*) \
|
||||
echo "UV_EXTRAS entry $extra is invalid (must match [A-Za-z][A-Za-z0-9_-]*)" >&2; \
|
||||
exit 1; \
|
||||
;; \
|
||||
esac; \
|
||||
extras_flags="$extras_flags --extra $extra"; \
|
||||
done; \
|
||||
UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync --extra redis $extras_flags'
|
||||
|
||||
# UTF-8 locale prevents UnicodeEncodeError on Chinese/emoji content in minimal
|
||||
# containers where locale configuration may be missing and the default encoding is not UTF-8.
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
|
||||
# ── Stage 2: Dev ──────────────────────────────────────────────────────────────
|
||||
# Retains compiler toolchain from builder so startup-time `uv sync` can build
|
||||
# source distributions in development containers.
|
||||
FROM builder AS dev
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
EXPOSE 8001
|
||||
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
|
||||
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
|
||||
# Copy Node.js runtime from builder (provides npx for MCP servers)
|
||||
COPY --from=builder /usr/bin/node /usr/bin/node
|
||||
COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
|
||||
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
|
||||
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend with pre-built virtualenv from builder
|
||||
COPY --from=builder /app/backend ./backend
|
||||
|
||||
# Expose Gateway API port.
|
||||
EXPOSE 8001
|
||||
|
||||
# Default command (can be overridden in docker-compose)
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
@@ -0,0 +1,35 @@
|
||||
install:
|
||||
uv sync
|
||||
|
||||
dev:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload
|
||||
|
||||
gateway:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/ -v
|
||||
|
||||
test-blocking-io:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/blocking_io -q --tb=short
|
||||
|
||||
lint:
|
||||
uvx ruff check .
|
||||
uvx ruff format --check .
|
||||
|
||||
format:
|
||||
uvx ruff check . --fix && uvx ruff format .
|
||||
|
||||
detect-blocking-io:
|
||||
@PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run python ../scripts/detect_blocking_io_static.py --output ../.deer-flow/blocking-io-findings.json
|
||||
|
||||
# Generate a new alembic revision by autogenerating the diff against the live
|
||||
# ORM models. Usage: make migrate-rev MSG="add foo column to runs"
|
||||
# The Gateway runs `alembic upgrade head` automatically at startup via
|
||||
# `bootstrap_schema`, so there is intentionally no `migrate` / `migrate-stamp`
|
||||
# target -- the single execution path keeps ops mistakes off the table.
|
||||
# The script builds a fresh temp SQLite at head, then diffs the live models
|
||||
# against it, so a clean checkout does NOT need a pre-existing ./data/deerflow.db.
|
||||
migrate-rev:
|
||||
@if [ -z "$(MSG)" ]; then echo 'usage: make migrate-rev MSG="describe the change"'; exit 1; fi
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run python scripts/_autogen_revision.py "$(MSG)"
|
||||
@@ -0,0 +1,503 @@
|
||||
# DeerFlow Backend
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────┐
|
||||
│ Nginx (Port 2026) │
|
||||
│ Unified reverse proxy │
|
||||
└───────┬──────────────────┬───────────┘
|
||||
│
|
||||
/api/langgraph/* │ /api/* (other)
|
||||
rewritten to /api/* │
|
||||
▼
|
||||
┌────────────────────────────────────────┐
|
||||
│ Gateway API (8001) │
|
||||
│ FastAPI REST + agent runtime │
|
||||
│ │
|
||||
│ Models, MCP, Skills, Memory, Uploads, │
|
||||
│ Artifacts, Threads, Runs, Streaming │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ Lead Agent │ │
|
||||
│ │ Middleware Chain, Tools, Subagents │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Request Routing** (via Nginx):
|
||||
- `/api/langgraph/*` → Gateway LangGraph-compatible API - agent interactions, threads, streaming
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
|
||||
- `/` (non-API) → Frontend - Next.js web interface
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
|
||||
### Lead Agent
|
||||
|
||||
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
|
||||
|
||||
- **Dynamic model selection** with thinking and vision support
|
||||
- **Middleware chain** for cross-cutting concerns (9 middlewares)
|
||||
- **Tool system** with sandbox, MCP, community, and built-in tools
|
||||
- **Subagent delegation** for parallel task execution
|
||||
- **System prompt** with skills injection, memory context, and working directory guidance
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Middlewares execute in strict order, each handling a specific concern:
|
||||
|
||||
| # | Middleware | Purpose |
|
||||
|---|-----------|---------|
|
||||
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
|
||||
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
|
||||
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
|
||||
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
|
||||
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
|
||||
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
|
||||
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
|
||||
| 8 | **ViewImageMiddleware** | Injects image data for vision-capable models (conditional) |
|
||||
| 9 | **ClarificationMiddleware** | Intercepts clarification requests and interrupts execution (must be last) |
|
||||
|
||||
### Sandbox System
|
||||
|
||||
Per-thread isolated execution with virtual path translation:
|
||||
|
||||
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/). Async runtime paths use async sandbox lifecycle hooks so startup, readiness polling, and release do not block the event loop. `AioSandboxProvider` validates active-cache and warm-pool containers during acquire/reuse, dropping definitively dead entries so a thread can provision a fresh sandbox after an unexpected container exit while keeping `get()` as an in-memory lookup. Backend health-check failures are treated as unknown, not dead, and a container that cannot be verified during discovery is simply not adopted (acquire falls through to create instead of failing).
|
||||
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **SkillScan**: Native offline deterministic scanning runs before the LLM skill scanner on installs and agent-managed skill writes; `CRITICAL` findings block and warning findings become LLM context
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`write_file` overwrites by default and exposes `append` for end-of-file writes; `bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
|
||||
### Subagent System
|
||||
|
||||
Async task delegation with concurrent execution:
|
||||
|
||||
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
|
||||
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
|
||||
- **Execution**: Background thread pools with status tracking and SSE events
|
||||
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
|
||||
|
||||
### Memory System
|
||||
|
||||
LLM-powered persistent context retention across conversations:
|
||||
|
||||
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
|
||||
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
|
||||
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
|
||||
- **System prompt injection**: Top facts + context injected into agent prompts
|
||||
- **Storage**: JSON file with mtime-based cache invalidation
|
||||
|
||||
### Tool Ecosystem
|
||||
|
||||
| Category | Tools |
|
||||
|----------|-------|
|
||||
| **Sandbox** | `bash`, `ls`, `read_file`, `write_file`, `str_replace` |
|
||||
| **Built-in** | `present_files`, `ask_clarification`, `view_image`, `task` (subagent) |
|
||||
| **Community** | Tavily (web search), Jina AI (web fetch), Crawl4AI (web fetch), Firecrawl (scraping), fastCRW (scraping), DuckDuckGo (image search) |
|
||||
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
|
||||
| **Skills** | Domain-specific workflows injected via system prompt |
|
||||
|
||||
### Gateway API
|
||||
|
||||
FastAPI application providing REST endpoints for frontend integration:
|
||||
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `GET /api/models` | List available LLM models |
|
||||
| `GET/PUT /api/mcp/config` | Manage MCP server configurations |
|
||||
| `POST /api/mcp/cache/reset` | Reset cached MCP tools so they reload on next use |
|
||||
| `GET/PUT /api/skills` | List and manage skills |
|
||||
| `POST /api/skills/install` | Install skill from `.skill` archive |
|
||||
| `GET /api/memory` | Retrieve memory data |
|
||||
| `POST /api/memory/reload` | Force memory reload |
|
||||
| `GET /api/memory/config` | Memory configuration |
|
||||
| `GET /api/memory/status` | Combined config + data |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths, auto-renames duplicate filenames in one request) |
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
|
||||
### IM Channels
|
||||
|
||||
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])`, serializes rapid same-thread turns inside the channel manager, and updates a single in-thread card per source message in place.
|
||||
|
||||
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow. When a follow-up arrives inside an existing Feishu topic while another turn is still running, the later message now waits on the mapped DeerFlow `thread_id`, receives a queued/running card on that exact source message, and keeps a compact source-message blockquote in subsequent patches so rapid consecutive questions remain distinguishable.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12+
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
- API keys for your chosen LLM provider
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
cd deer-flow
|
||||
|
||||
# Copy configuration files
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Install backend dependencies
|
||||
cd backend
|
||||
make install
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Edit `config.yaml` in the project root:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
supports_thinking: false
|
||||
supports_vision: true
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
supports_vision: true
|
||||
```
|
||||
|
||||
Set your API keys:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="your-api-key-here"
|
||||
```
|
||||
|
||||
### Running
|
||||
|
||||
**Full Application** (from project root):
|
||||
|
||||
```bash
|
||||
make dev # Starts Gateway + Frontend + Nginx
|
||||
```
|
||||
|
||||
Access at: http://localhost:2026
|
||||
|
||||
**Backend Only** (from backend directory):
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
make dev
|
||||
```
|
||||
|
||||
Direct access: Gateway at http://localhost:8001
|
||||
|
||||
**Terminal Workbench (TUI)** — a terminal-native UI over the embedded harness,
|
||||
no services required:
|
||||
|
||||
```bash
|
||||
uv pip install 'deerflow-harness[tui]' # optional 'textual' dependency
|
||||
deerflow # launch the TUI
|
||||
deerflow --print "summarize this repo" # headless one-shot
|
||||
```
|
||||
|
||||
Sessions opened in the TUI appear in the Web UI sidebar (it writes the shared
|
||||
`threads_meta` store under the local default user). See [docs/TUI.md](docs/TUI.md).
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
backend/
|
||||
├── packages/harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ └── deerflow/
|
||||
│ ├── agents/ # Agent system
|
||||
│ │ ├── lead_agent/ # Main agent (factory, prompts)
|
||||
│ │ ├── middlewares/ # Middleware components
|
||||
│ │ ├── memory/ # Memory extraction & storage
|
||||
│ │ └── thread_state.py # ThreadState schema
|
||||
│ ├── sandbox/ # Sandbox execution
|
||||
│ │ ├── local/ # Local filesystem provider
|
||||
│ │ ├── sandbox.py # Abstract interface
|
||||
│ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ └── middleware.py # Sandbox lifecycle
|
||||
│ ├── subagents/ # Subagent delegation
|
||||
│ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ ├── executor.py # Background execution engine
|
||||
│ │ └── registry.py # Agent registry
|
||||
│ ├── tools/builtins/ # Built-in tools
|
||||
│ ├── mcp/ # MCP protocol integration
|
||||
│ ├── models/ # Model factory
|
||||
│ ├── skills/ # Skill discovery & loading
|
||||
│ ├── config/ # Configuration system
|
||||
│ ├── runtime/ # Embedded run execution (RunManager, StreamBridge)
|
||||
│ ├── persistence/ # Checkpointer/store engines & schema migrations
|
||||
│ ├── guardrails/ # Pre-tool-call authorization providers
|
||||
│ ├── tracing/ # Tracer factory & trace metadata
|
||||
│ ├── uploads/ # Uploads manager
|
||||
│ ├── tui/ # Terminal UI (`deerflow` console script)
|
||||
│ ├── community/ # Community tools & providers
|
||||
│ ├── reflection/ # Dynamic module loading
|
||||
│ └── utils/ # Utilities
|
||||
├── app/ # FastAPI Gateway + IM channels (import: app.*)
|
||||
│ ├── gateway/ # Gateway API
|
||||
│ │ ├── app.py # Application setup
|
||||
│ │ └── routers/ # Route modules
|
||||
│ └── channels/ # IM channel integrations
|
||||
├── docs/ # Documentation
|
||||
├── tests/ # Test suite
|
||||
├── langgraph.json # LangGraph graph registry for tooling/Studio compatibility
|
||||
├── pyproject.toml # Python dependencies
|
||||
├── Makefile # Development commands
|
||||
└── Dockerfile # Container build
|
||||
```
|
||||
|
||||
`langgraph.json` is not the default service entrypoint. The scripts and Docker
|
||||
deployments run the Gateway embedded runtime; the file is kept for LangGraph
|
||||
tooling, Studio, or direct LangGraph Server compatibility.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Main Configuration (`config.yaml`)
|
||||
|
||||
Place in project root. Config values starting with `$` resolve as environment variables.
|
||||
|
||||
Key sections:
|
||||
- `models` - LLM configurations with class paths, API keys, thinking/vision flags
|
||||
- `tools` - Tool definitions with module paths and groups
|
||||
- `tool_groups` - Logical tool groupings
|
||||
- `sandbox` - Execution environment provider
|
||||
- `skills` - Skills directory paths
|
||||
- `title` - Auto-title generation settings
|
||||
- `summarization` - Context summarization settings
|
||||
- `subagents` - Subagent system (enabled/disabled)
|
||||
- `memory` - Memory system settings (enabled, storage, debounce, facts limits)
|
||||
|
||||
Provider note:
|
||||
- `models[*].use` references provider classes by module path (for example `langchain_openai:ChatOpenAI`).
|
||||
- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example `uv add langchain-google-genai`).
|
||||
|
||||
### Extensions Configuration (`extensions_config.json`)
|
||||
|
||||
MCP servers and skill states in a single file:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
|
||||
},
|
||||
"secure-http": {
|
||||
"enabled": true,
|
||||
"type": "http",
|
||||
"url": "https://api.example.com/mcp",
|
||||
"oauth": {
|
||||
"enabled": true,
|
||||
"token_url": "https://auth.example.com/oauth/token",
|
||||
"grant_type": "client_credentials",
|
||||
"client_id": "$MCP_OAUTH_CLIENT_ID",
|
||||
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
|
||||
}
|
||||
},
|
||||
"postgres": {
|
||||
"enabled": false,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-postgres", "postgresql://localhost/mydb"],
|
||||
"description": "PostgreSQL database access",
|
||||
"routing": {
|
||||
"mode": "prefer",
|
||||
"priority": 50,
|
||||
"keywords": ["orders", "users", "SQL", "database", "table"]
|
||||
},
|
||||
"tools": {
|
||||
"query": {
|
||||
"routing": {
|
||||
"priority": 100,
|
||||
"keywords": ["query database", "orders table", "metrics"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"skills": {
|
||||
"pdf-processing": {"enabled": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`routing` adds soft MCP preference hints to the agent prompt. It helps the
|
||||
model prefer a configured MCP tool for matching requests without forbidding
|
||||
other tools. When `tool_search.enabled=true` defers MCP schemas, matching
|
||||
routing metadata can auto-promote up to `tool_search.auto_promote_top_k`
|
||||
deferred schemas before the model call.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
- `DEER_FLOW_CONFIG_PATH` - Override config.yaml location
|
||||
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Override extensions_config.json location
|
||||
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
|
||||
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
|
||||
|
||||
### LangSmith Tracing
|
||||
|
||||
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
|
||||
2. Add the following to your `.env` file in the project root:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
|
||||
|
||||
### Langfuse Tracing
|
||||
|
||||
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
|
||||
|
||||
### Dual Provider Behavior
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
|
||||
|
||||
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
|
||||
|
||||
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
### Commands
|
||||
|
||||
```bash
|
||||
make install # Install dependencies
|
||||
make dev # Run Gateway API + embedded agent runtime (port 8001)
|
||||
make gateway # Run Gateway API without reload (port 8001)
|
||||
make lint # Run linter (ruff)
|
||||
make format # Format code (ruff)
|
||||
make detect-blocking-io # Inventory blocking IO that may block the backend event loop
|
||||
make migrate-rev MSG="..." # Autogenerate a new alembic revision against the live ORM models
|
||||
```
|
||||
|
||||
### Schema Migrations
|
||||
|
||||
DeerFlow's application tables (`runs`, `threads_meta`, `feedback`, `users`,
|
||||
`run_events`, and the `channel_*` tables) are owned by alembic. The Gateway
|
||||
runs `alembic upgrade head` automatically on startup via
|
||||
`bootstrap_schema(engine, backend=...)`, so operators do not run `alembic`
|
||||
manually in production. Bootstrap is concurrency-safe (Postgres advisory lock
|
||||
across processes; per-engine `asyncio.Lock` inside one SQLite process) and
|
||||
idempotent against pre-existing schemas (empty / legacy / versioned).
|
||||
|
||||
When you add or change an ORM model, ship the change as a new revision under
|
||||
`packages/harness/deerflow/persistence/migrations/versions/`:
|
||||
|
||||
```bash
|
||||
make migrate-rev MSG="add foo column to runs"
|
||||
```
|
||||
|
||||
The target invokes `scripts/_autogen_revision.py`, which builds a fresh temp
|
||||
SQLite at `head` and diffs the live models against it — so a clean checkout
|
||||
does not need a pre-existing `./data/deerflow.db`. Review the generated file
|
||||
and switch raw `op.add_column` / `op.drop_column` calls to the idempotent
|
||||
helpers in `migrations/_helpers.py` before committing. There is no
|
||||
`make migrate` / `make migrate-stamp` target on purpose — Gateway startup is
|
||||
the only execution path, which keeps operational mistakes off the table. See
|
||||
`backend/CLAUDE.md` (Schema Migrations) for the full design.
|
||||
|
||||
### Code Style
|
||||
|
||||
- **Linter/Formatter**: `ruff`
|
||||
- **Line length**: 240 characters
|
||||
- **Python**: 3.12+ with type hints
|
||||
- **Quotes**: Double quotes
|
||||
- **Indentation**: 4 spaces
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
`make detect-blocking-io` statically scans backend business code for blocking
|
||||
IO that may run on the backend event loop and is not test-coverage-bound. It
|
||||
prints a concise summary for human review and writes complete JSON findings to
|
||||
`.deer-flow/blocking-io-findings.json` at the repository root (regardless of
|
||||
whether the target is invoked from the repo root or from `backend/`). JSON
|
||||
findings include both broad IO category and review-oriented fields such as
|
||||
`priority`, `location`, `blocking_call`, `event_loop_exposure`, `reason`, and
|
||||
`code`. `priority` is a deterministic review ordering from the operation type,
|
||||
not proof of a bug. Bare-name same-file calls are resolved by function name,
|
||||
so duplicate helper names in one file can conservatively over-report async
|
||||
reachability.
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
|
||||
- **LangGraph** (1.0.6+) - Agent framework and multi-agent orchestration
|
||||
- **LangChain** (1.2.3+) - LLM abstractions and tool system
|
||||
- **FastAPI** (0.115.0+) - Gateway REST API
|
||||
- **langchain-mcp-adapters** - Model Context Protocol support
|
||||
- **agent-sandbox** - Sandboxed code execution
|
||||
- **markitdown** - Multi-format document conversion
|
||||
- **tavily-python** / **firecrawl-py** - Web search and scraping
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration Guide](docs/CONFIGURATION.md)
|
||||
- [Architecture Details](docs/ARCHITECTURE.md)
|
||||
- [API Reference](docs/API.md)
|
||||
- [File Upload](docs/FILE_UPLOAD.md)
|
||||
- [Path Examples](docs/PATH_EXAMPLES.md)
|
||||
- [Context Summarization](docs/summarization.md)
|
||||
- [Plan Mode](docs/plan_mode_usage.md)
|
||||
- [Setup Guide](docs/SETUP.md)
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
See the [LICENSE](../LICENSE) file in the project root.
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
|
||||
@@ -0,0 +1,16 @@
|
||||
"""IM Channel integration for DeerFlow.
|
||||
|
||||
Provides a pluggable channel system that connects external messaging platforms
|
||||
(Feishu/Lark, Slack, Telegram) to the DeerFlow agent via the ChannelManager,
|
||||
which uses ``langgraph-sdk`` to communicate with Gateway's LangGraph-compatible API.
|
||||
"""
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
|
||||
|
||||
__all__ = [
|
||||
"Channel",
|
||||
"InboundMessage",
|
||||
"MessageBus",
|
||||
"OutboundMessage",
|
||||
]
|
||||
@@ -0,0 +1,199 @@
|
||||
"""Abstract base class for IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Awaitable, Callable
|
||||
from concurrent.futures import CancelledError as FutureCancelledError
|
||||
from typing import Any, TypeVar
|
||||
|
||||
from app.channels.commands import extract_connect_code
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class Channel(ABC):
|
||||
"""Base class for all IM channel implementations.
|
||||
|
||||
Each channel connects to an external messaging platform and:
|
||||
1. Receives messages, wraps them as InboundMessage, publishes to the bus.
|
||||
2. Subscribes to outbound messages and sends replies back to the platform.
|
||||
|
||||
Subclasses must implement ``start``, ``stop``, and ``send``.
|
||||
"""
|
||||
|
||||
def __init__(self, name: str, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
self.name = name
|
||||
self.bus = bus
|
||||
self.config = config
|
||||
self._running = False
|
||||
self._connection_repo: Any = config.get("connection_repo")
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return False
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def start(self) -> None:
|
||||
"""Start listening for messages from the external platform."""
|
||||
|
||||
@abstractmethod
|
||||
async def stop(self) -> None:
|
||||
"""Gracefully stop the channel."""
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
"""Send a message back to the external platform.
|
||||
|
||||
The implementation should use ``msg.chat_id`` and ``msg.thread_ts``
|
||||
to route the reply to the correct conversation/thread.
|
||||
"""
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
"""Upload a single file attachment to the platform.
|
||||
|
||||
Returns True if the upload succeeded, False otherwise.
|
||||
Default implementation returns False (no file upload support).
|
||||
"""
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
async def _send_with_retry(
|
||||
self,
|
||||
operation: Callable[[], Awaitable[T]],
|
||||
*,
|
||||
max_retries: int,
|
||||
log_prefix: str | None = None,
|
||||
operation_name: str = "send",
|
||||
) -> T:
|
||||
"""Run an outbound send operation with the shared channel retry policy."""
|
||||
prefix = log_prefix or f"[{self.name}]"
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
return await operation()
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < max_retries - 1:
|
||||
delay = 2**attempt
|
||||
logger.warning(
|
||||
"%s %s failed (attempt %d/%d), retrying in %ds: %s",
|
||||
prefix,
|
||||
operation_name,
|
||||
attempt + 1,
|
||||
max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("%s %s failed after %d attempts: %s", prefix, operation_name, max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError(f"{self.name} {operation_name} failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
def _log_future_error(self, fut: Any, name: str, msg_id: Any) -> None:
|
||||
"""Callback for concurrent futures scheduled from channel worker threads."""
|
||||
try:
|
||||
exc = fut.exception()
|
||||
except (asyncio.CancelledError, FutureCancelledError, asyncio.InvalidStateError):
|
||||
return
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to inspect future for %s (msg_id=%s)", self.name, name, msg_id)
|
||||
return
|
||||
|
||||
if exc:
|
||||
logger.error("[%s] %s failed for msg_id=%s: %s", self.name, name, msg_id, exc)
|
||||
|
||||
def _pending_connect_code(self, text: str) -> str | None:
|
||||
"""Return the one-time bind code if *text* is a ``/connect <code>`` command
|
||||
and channel connections are configured, else ``None``.
|
||||
|
||||
Adapters MUST consult this **before** applying their ``allowed_users`` /
|
||||
``_check_user`` gate, so a browser-initiated bind can bootstrap an external
|
||||
identity that the platform bot has never seen and is therefore not yet
|
||||
authorized. (Telegram uses its deep-link ``/start <token>`` flow instead.)
|
||||
"""
|
||||
if self._connection_repo is None:
|
||||
return None
|
||||
return extract_connect_code(text)
|
||||
|
||||
def _make_inbound(
|
||||
self,
|
||||
chat_id: str,
|
||||
user_id: str,
|
||||
text: str,
|
||||
*,
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT,
|
||||
thread_ts: str | None = None,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> InboundMessage:
|
||||
"""Convenience factory for creating InboundMessage instances."""
|
||||
return InboundMessage(
|
||||
channel_name=self.name,
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
files=files or [],
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Outbound callback registered with the bus.
|
||||
|
||||
Only forwards messages targeted at this channel.
|
||||
Sends the text message first, then uploads any file attachments.
|
||||
File uploads are skipped entirely when the text send fails to avoid
|
||||
partial deliveries (files without accompanying text).
|
||||
"""
|
||||
if msg.channel_name == self.name:
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
return # Do not attempt file uploads when the text message failed
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
async def receive_file(self, msg: InboundMessage, thread_id: str, *, user_id: str | None = None) -> InboundMessage:
|
||||
"""
|
||||
Optionally process and materialize inbound file attachments for this channel.
|
||||
|
||||
By default, this method does nothing and simply returns the original message.
|
||||
Subclasses (e.g. FeishuChannel) may override this to download files (images, documents, etc)
|
||||
referenced in msg.files, save them to the sandbox, and update msg.text to include
|
||||
the sandbox file paths for downstream model consumption.
|
||||
|
||||
Args:
|
||||
msg: The inbound message, possibly containing file metadata in msg.files.
|
||||
thread_id: The resolved DeerFlow thread ID for sandbox path context.
|
||||
user_id: Optional DeerFlow storage user ID for user-scoped channel workers.
|
||||
|
||||
Returns:
|
||||
The (possibly modified) InboundMessage, with text and/or files updated as needed.
|
||||
"""
|
||||
del user_id
|
||||
return msg
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Shared command definitions used by all channel implementations.
|
||||
|
||||
Keeping the authoritative command set in one place ensures that channel
|
||||
parsers (e.g. Feishu) and the ChannelManager dispatcher stay in sync
|
||||
automatically — adding or removing a command here is the single edit
|
||||
required.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
KNOWN_CHANNEL_COMMANDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/bootstrap",
|
||||
"/goal",
|
||||
"/new",
|
||||
"/status",
|
||||
"/models",
|
||||
"/memory",
|
||||
"/help",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def extract_connect_code(text: str) -> str | None:
|
||||
"""Extract the one-time channel binding code from a connect command."""
|
||||
parts = text.strip().split()
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
command = parts[0].lower()
|
||||
if command in {"/connect", "connect"}:
|
||||
return parts[1]
|
||||
return None
|
||||
|
||||
|
||||
def is_known_channel_command(text: str) -> bool:
|
||||
"""Return whether text starts with a registered channel control command."""
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Helpers for attaching persisted channel connection ownership to inbound messages."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from app.channels.message_bus import InboundMessage
|
||||
|
||||
|
||||
async def attach_connection_identity(
|
||||
inbound: InboundMessage,
|
||||
*,
|
||||
repo: Any,
|
||||
provider: str,
|
||||
workspace_id: str | None,
|
||||
fallback_without_workspace: bool = False,
|
||||
) -> InboundMessage:
|
||||
"""Attach connection metadata to an inbound message when a persisted binding exists."""
|
||||
if repo is None:
|
||||
return inbound
|
||||
|
||||
workspace_candidates: list[str | None] = []
|
||||
if workspace_id:
|
||||
workspace_candidates.append(workspace_id)
|
||||
if fallback_without_workspace:
|
||||
workspace_candidates.append(None)
|
||||
if not workspace_candidates:
|
||||
return inbound
|
||||
|
||||
for candidate in workspace_candidates:
|
||||
connection = await repo.find_connection_by_external_identity(
|
||||
provider=provider,
|
||||
external_account_id=inbound.user_id,
|
||||
workspace_id=candidate,
|
||||
)
|
||||
if connection is None:
|
||||
continue
|
||||
|
||||
inbound.connection_id = connection["id"]
|
||||
inbound.owner_user_id = connection["owner_user_id"]
|
||||
inbound.workspace_id = connection.get("workspace_id")
|
||||
return inbound
|
||||
|
||||
return inbound
|
||||
@@ -0,0 +1,822 @@
|
||||
"""DingTalk channel implementation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.connection_identity import attach_connection_identity
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DINGTALK_API_BASE = "https://api.dingtalk.com"
|
||||
|
||||
_TOKEN_REFRESH_MARGIN_SECONDS = 300
|
||||
|
||||
_CONVERSATION_TYPE_P2P = "1"
|
||||
_CONVERSATION_TYPE_GROUP = "2"
|
||||
|
||||
_MAX_UPLOAD_SIZE_BYTES = 20 * 1024 * 1024
|
||||
|
||||
|
||||
def _normalize_conversation_type(raw: Any) -> str:
|
||||
"""Normalize ``conversationType`` to ``"1"`` (P2P) or ``"2"`` (group).
|
||||
|
||||
Stream payloads may send int or string values.
|
||||
"""
|
||||
if raw is None:
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
s = str(raw).strip()
|
||||
if s == _CONVERSATION_TYPE_GROUP:
|
||||
return _CONVERSATION_TYPE_GROUP
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, (list, tuple, set)):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"DingTalk allowed_users should be a list of user IDs; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(uid) for uid in values if str(uid)}
|
||||
|
||||
|
||||
def _is_dingtalk_command(text: str) -> bool:
|
||||
return is_known_channel_command(text)
|
||||
|
||||
|
||||
def _extract_text_from_rich_text(rich_text_list: list) -> str:
|
||||
parts: list[str] = []
|
||||
for item in rich_text_list:
|
||||
if isinstance(item, dict) and "text" in item:
|
||||
parts.append(item["text"])
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
_FENCED_CODE_BLOCK_RE = re.compile(r"```(\w*)\n(.*?)```", re.DOTALL)
|
||||
_INLINE_CODE_RE = re.compile(r"`([^`\n]+)`")
|
||||
_HORIZONTAL_RULE_RE = re.compile(r"^-{3,}$", re.MULTILINE)
|
||||
_TABLE_SEPARATOR_RE = re.compile(r"^\|[-:| ]+\|$", re.MULTILINE)
|
||||
|
||||
|
||||
def _convert_markdown_table(text: str) -> str:
|
||||
# DingTalk sampleMarkdown does not render pipe-delimited tables.
|
||||
lines = text.split("\n")
|
||||
result: list[str] = []
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
# Detect table: header row followed by separator row
|
||||
if i + 1 < len(lines) and line.strip().startswith("|") and _TABLE_SEPARATOR_RE.match(lines[i + 1].strip()):
|
||||
headers = [h.strip() for h in line.strip().strip("|").split("|")]
|
||||
i += 2 # skip header + separator
|
||||
while i < len(lines) and lines[i].strip().startswith("|"):
|
||||
cells = [c.strip() for c in lines[i].strip().strip("|").split("|")]
|
||||
for h, c in zip(headers, cells):
|
||||
result.append(f"> **{h}**: {c}")
|
||||
result.append("")
|
||||
i += 1
|
||||
else:
|
||||
result.append(line)
|
||||
i += 1
|
||||
return "\n".join(result)
|
||||
|
||||
|
||||
def _adapt_markdown_for_dingtalk(text: str) -> str:
|
||||
"""Adapt markdown for DingTalk's limited sampleMarkdown renderer."""
|
||||
|
||||
def _code_block_to_quote(match: re.Match) -> str:
|
||||
lang = match.group(1)
|
||||
code = match.group(2).rstrip("\n")
|
||||
prefix = f"> **{lang}**\n" if lang else ""
|
||||
quoted_lines = "\n".join(f"> {line}" for line in code.split("\n"))
|
||||
return f"{prefix}{quoted_lines}\n"
|
||||
|
||||
text = _FENCED_CODE_BLOCK_RE.sub(_code_block_to_quote, text)
|
||||
text = _INLINE_CODE_RE.sub(r"**\1**", text)
|
||||
text = _convert_markdown_table(text)
|
||||
text = _HORIZONTAL_RULE_RE.sub("───────────", text)
|
||||
return text
|
||||
|
||||
|
||||
class DingTalkChannel(Channel):
|
||||
"""DingTalk IM channel using Stream Push (WebSocket, no public IP needed)."""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="dingtalk", bus=bus, config=config)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._client_id: str = ""
|
||||
self._client_secret: str = ""
|
||||
self._allowed_users: set[str] = _normalize_allowed_users(config.get("allowed_users"))
|
||||
self._cached_token: str = ""
|
||||
self._token_expires_at: float = 0.0
|
||||
self._token_lock = asyncio.Lock()
|
||||
self._card_template_id: str = config.get("card_template_id", "")
|
||||
self._card_track_ids: dict[str, str] = {}
|
||||
self._dingtalk_client: Any = None
|
||||
self._stream_client: Any = None
|
||||
self._incoming_messages: dict[str, Any] = {}
|
||||
self._incoming_messages_lock = threading.Lock()
|
||||
self._card_repliers: dict[str, Any] = {}
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return bool(self._card_template_id)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import dingtalk_stream # noqa: F401
|
||||
except ImportError:
|
||||
logger.error("dingtalk-stream is not installed. Install it with: uv add dingtalk-stream")
|
||||
return
|
||||
|
||||
client_id = self.config.get("client_id", "")
|
||||
client_secret = self.config.get("client_secret", "")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error("DingTalk channel requires client_id and client_secret")
|
||||
return
|
||||
|
||||
self._client_id = client_id
|
||||
self._client_secret = client_secret
|
||||
self._main_loop = asyncio.get_running_loop()
|
||||
|
||||
if self._card_template_id:
|
||||
logger.info("[DingTalk] AI Card mode enabled (template=%s)", self._card_template_id)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_stream,
|
||||
args=(client_id, client_secret),
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
logger.info("DingTalk channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
stream_client = self._stream_client
|
||||
if stream_client is not None:
|
||||
try:
|
||||
if hasattr(stream_client, "disconnect"):
|
||||
stream_client.disconnect()
|
||||
except Exception:
|
||||
logger.debug("[DingTalk] error disconnecting stream client", exc_info=True)
|
||||
|
||||
self._dingtalk_client = None
|
||||
self._stream_client = None
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages.clear()
|
||||
self._card_repliers.clear()
|
||||
self._card_track_ids.clear()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
logger.info("DingTalk channel stopped")
|
||||
|
||||
def _resolve_routing(self, msg: OutboundMessage) -> tuple[str, str, str]:
|
||||
"""Return (conversation_type, sender_staff_id, conversation_id).
|
||||
|
||||
Uses msg.chat_id as the primary routing key; metadata as fallback.
|
||||
"""
|
||||
conversation_type = _normalize_conversation_type(msg.metadata.get("conversation_type"))
|
||||
sender_staff_id = msg.metadata.get("sender_staff_id", "")
|
||||
conversation_id = msg.metadata.get("conversation_id", "")
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
conversation_id = msg.chat_id or conversation_id
|
||||
else:
|
||||
sender_staff_id = msg.chat_id or sender_staff_id
|
||||
return conversation_type, sender_staff_id, conversation_id
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
# Card mode: stream update to existing AI card
|
||||
source_key = self._make_card_source_key_from_outbound(msg)
|
||||
out_track_id = self._card_track_ids.get(source_key)
|
||||
|
||||
# ``card_template_id`` enables ``runs.stream`` (non-final + final outbounds).
|
||||
# If card creation failed, skip non-final chunks to avoid duplicate messages.
|
||||
if self._card_template_id and not out_track_id and not msg.is_final:
|
||||
return
|
||||
|
||||
if out_track_id:
|
||||
try:
|
||||
await self._stream_update_card(
|
||||
out_track_id,
|
||||
msg.text,
|
||||
is_finalize=msg.is_final,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
|
||||
return
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
return
|
||||
|
||||
async def send_markdown() -> None:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, msg.text, at_user_ids=[sender_staff_id] if sender_staff_id else None)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, msg.text)
|
||||
|
||||
# Non-card mode: send sampleMarkdown with retry
|
||||
await self._send_with_retry(
|
||||
send_markdown,
|
||||
max_retries=_max_retries,
|
||||
log_prefix="[DingTalk]",
|
||||
)
|
||||
return
|
||||
|
||||
async def _send_markdown_fallback(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_type: str,
|
||||
sender_staff_id: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
try:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, text)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] markdown fallback also failed")
|
||||
raise
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if attachment.size > _MAX_UPLOAD_SIZE_BYTES:
|
||||
logger.warning("[DingTalk] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media(attachment.actual_path, "image" if attachment.is_image else "file")
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
if attachment.is_image:
|
||||
msg_key = "sampleImageMsg"
|
||||
msg_param = json.dumps({"photoURL": media_id})
|
||||
else:
|
||||
msg_key = "sampleFile"
|
||||
msg_param = json.dumps(
|
||||
{
|
||||
"fileUrl": media_id,
|
||||
"fileName": attachment.filename,
|
||||
"fileSize": str(attachment.size),
|
||||
}
|
||||
)
|
||||
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
else:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"userIds": [sender_staff_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
logger.info("[DingTalk] file sent: %s", attachment.filename)
|
||||
return True
|
||||
except (httpx.HTTPError, OSError, ValueError, TypeError, AttributeError):
|
||||
logger.exception("[DingTalk] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- stream client (runs in dedicated thread) --------------------------
|
||||
|
||||
def _run_stream(self, client_id: str, client_secret: str) -> None:
|
||||
try:
|
||||
import dingtalk_stream
|
||||
|
||||
credential = dingtalk_stream.Credential(client_id, client_secret)
|
||||
client = dingtalk_stream.DingTalkStreamClient(credential)
|
||||
self._stream_client = client
|
||||
client.register_callback_handler(
|
||||
dingtalk_stream.chatbot.ChatbotMessage.TOPIC,
|
||||
_DingTalkMessageHandler(self),
|
||||
)
|
||||
client.start_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("DingTalk Stream Push error")
|
||||
finally:
|
||||
self._stream_client = None
|
||||
|
||||
def _on_chatbot_message(self, message: Any) -> None:
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
sender_staff_id = message.sender_staff_id or ""
|
||||
conversation_type = _normalize_conversation_type(message.conversation_type)
|
||||
conversation_id = message.conversation_id or ""
|
||||
msg_id = message.message_id or ""
|
||||
sender_nick = message.sender_nick or ""
|
||||
|
||||
text = self._extract_text(message)
|
||||
if not text:
|
||||
logger.info("[DingTalk] empty text, ignoring message")
|
||||
return
|
||||
|
||||
connect_code = self._pending_connect_code(text)
|
||||
if connect_code:
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(
|
||||
self._bind_connection_from_connect_code(
|
||||
conversation_type=conversation_type,
|
||||
sender_staff_id=sender_staff_id,
|
||||
sender_nick=sender_nick,
|
||||
conversation_id=conversation_id,
|
||||
code=connect_code,
|
||||
),
|
||||
self._main_loop,
|
||||
)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "bind_connection", mid))
|
||||
else:
|
||||
logger.warning("[DingTalk] main loop not running, cannot bind channel connection")
|
||||
return
|
||||
|
||||
if self._allowed_users and sender_staff_id not in self._allowed_users:
|
||||
logger.debug("[DingTalk] ignoring message from non-allowed user: %s", sender_staff_id)
|
||||
return
|
||||
|
||||
# Log only metadata (length, not content) so message text never reaches
|
||||
# INFO logs, and only after the allowed_users gate so blocked senders are
|
||||
# not logged at all.
|
||||
logger.info(
|
||||
"[DingTalk] parsed message: conv_type=%s, msg_id=%s, sender=%s(%s), text_len=%d",
|
||||
conversation_type,
|
||||
msg_id,
|
||||
sender_staff_id,
|
||||
sender_nick,
|
||||
len(text or ""),
|
||||
)
|
||||
|
||||
if _is_dingtalk_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# P2P: topic_id=None (single thread per user, like Telegram private chat)
|
||||
# Group: topic_id=msg_id (each new message starts a new topic, like Feishu)
|
||||
topic_id: str | None = msg_id if conversation_type == _CONVERSATION_TYPE_GROUP else None
|
||||
|
||||
# chat_id uses conversation_id for groups, sender_staff_id for P2P
|
||||
chat_id = conversation_id if conversation_type == _CONVERSATION_TYPE_GROUP else sender_staff_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=sender_staff_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
metadata={
|
||||
"conversation_type": conversation_type,
|
||||
"conversation_id": conversation_id,
|
||||
"sender_staff_id": sender_staff_id,
|
||||
"sender_nick": sender_nick,
|
||||
"message_id": msg_id,
|
||||
},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages[source_key] = message
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
logger.info("[DingTalk] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
|
||||
fut = asyncio.run_coroutine_threadsafe(
|
||||
self._prepare_inbound(chat_id, inbound),
|
||||
self._main_loop,
|
||||
)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
|
||||
else:
|
||||
logger.warning("[DingTalk] main loop not running, cannot publish inbound message")
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] error processing chatbot message")
|
||||
|
||||
@staticmethod
|
||||
def _extract_text(message: Any) -> str:
|
||||
msg_type = message.message_type
|
||||
if msg_type == "text" and message.text:
|
||||
return message.text.content.strip()
|
||||
if msg_type == "richText" and message.rich_text_content:
|
||||
return _extract_text_from_rich_text(message.rich_text_content.rich_text_list).strip()
|
||||
return ""
|
||||
|
||||
async def _prepare_inbound(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
inbound = await self._attach_connection_identity(inbound)
|
||||
# Running reply must finish before publish_inbound so AI card tracks are
|
||||
# registered before the manager emits streaming outbounds.
|
||||
await self._send_running_reply(chat_id, inbound)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
@staticmethod
|
||||
def _connection_workspace_id(conversation_type: str, conversation_id: str) -> str | None:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP and conversation_id:
|
||||
return conversation_id
|
||||
return None
|
||||
|
||||
async def _attach_connection_identity(self, inbound: InboundMessage) -> InboundMessage:
|
||||
conversation_type = str(inbound.metadata.get("conversation_type") or _CONVERSATION_TYPE_P2P)
|
||||
conversation_id = str(inbound.metadata.get("conversation_id") or "")
|
||||
return await attach_connection_identity(
|
||||
inbound,
|
||||
repo=self._connection_repo,
|
||||
provider="dingtalk",
|
||||
workspace_id=self._connection_workspace_id(conversation_type, conversation_id),
|
||||
fallback_without_workspace=True,
|
||||
)
|
||||
|
||||
async def _bind_connection_from_connect_code(
|
||||
self,
|
||||
*,
|
||||
conversation_type: str,
|
||||
sender_staff_id: str,
|
||||
sender_nick: str,
|
||||
conversation_id: str,
|
||||
code: str,
|
||||
) -> bool:
|
||||
if self._connection_repo is None or not code:
|
||||
return False
|
||||
|
||||
state = await self._connection_repo.consume_oauth_state(provider="dingtalk", state=code)
|
||||
if state is None:
|
||||
await self._send_connection_reply(
|
||||
conversation_type,
|
||||
sender_staff_id,
|
||||
conversation_id,
|
||||
"DingTalk connection code is invalid or expired.",
|
||||
)
|
||||
return True
|
||||
|
||||
if not sender_staff_id:
|
||||
await self._send_connection_reply(
|
||||
conversation_type,
|
||||
sender_staff_id,
|
||||
conversation_id,
|
||||
"DingTalk connection could not be completed from this message.",
|
||||
)
|
||||
return True
|
||||
|
||||
await self._connection_repo.upsert_connection(
|
||||
owner_user_id=state["owner_user_id"],
|
||||
provider="dingtalk",
|
||||
external_account_id=sender_staff_id,
|
||||
external_account_name=sender_nick or None,
|
||||
workspace_id=self._connection_workspace_id(conversation_type, conversation_id),
|
||||
metadata={
|
||||
"conversation_type": conversation_type,
|
||||
"conversation_id": conversation_id,
|
||||
},
|
||||
status="connected",
|
||||
)
|
||||
await self._send_connection_reply(
|
||||
conversation_type,
|
||||
sender_staff_id,
|
||||
conversation_id,
|
||||
"DingTalk connected to DeerFlow.",
|
||||
)
|
||||
return True
|
||||
|
||||
async def _send_connection_reply(
|
||||
self,
|
||||
conversation_type: str,
|
||||
sender_staff_id: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
robot_code = self._client_id
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
if conversation_id:
|
||||
await self._send_text_message_to_group(robot_code, conversation_id, text)
|
||||
return
|
||||
if sender_staff_id:
|
||||
await self._send_text_message_to_user(robot_code, sender_staff_id, text)
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
conversation_type = inbound.metadata.get("conversation_type", _CONVERSATION_TYPE_P2P)
|
||||
sender_staff_id = inbound.metadata.get("sender_staff_id", "")
|
||||
conversation_id = inbound.metadata.get("conversation_id", "")
|
||||
text = "\u23f3 Working on it..."
|
||||
|
||||
try:
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
chatbot_message = self._incoming_messages.pop(source_key, None)
|
||||
out_track_id = await self._create_and_deliver_card(
|
||||
text,
|
||||
chatbot_message=chatbot_message,
|
||||
)
|
||||
if out_track_id:
|
||||
self._card_track_ids[source_key] = out_track_id
|
||||
logger.info("[DingTalk] AI card running reply sent for chat=%s", chat_id)
|
||||
return
|
||||
|
||||
robot_code = self._client_id
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_text_message_to_group(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_text_message_to_user(robot_code, sender_staff_id, text)
|
||||
logger.info("[DingTalk] 'Working on it...' reply sent for chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to send running reply for chat=%s", chat_id)
|
||||
|
||||
# -- DingTalk API helpers ----------------------------------------------
|
||||
|
||||
async def _get_access_token(self) -> str:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with self._token_lock:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/oauth2/accessToken",
|
||||
json={"appKey": self._client_id, "appSecret": self._client_secret}, # DingTalk API field names
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"DingTalk access token response must be a JSON object, got {type(data).__name__}")
|
||||
|
||||
access_token = data.get("accessToken")
|
||||
if not isinstance(access_token, str) or not access_token.strip():
|
||||
raise ValueError("DingTalk access token response did not contain a usable accessToken")
|
||||
|
||||
raw_expires_in = data.get("expireIn", 7200)
|
||||
try:
|
||||
expires_in = int(raw_expires_in)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning("[DingTalk] invalid expireIn value %r, using default 7200s", raw_expires_in)
|
||||
expires_in = 7200
|
||||
|
||||
self._cached_token = access_token.strip()
|
||||
self._token_expires_at = time.monotonic() + expires_in - _TOKEN_REFRESH_MARGIN_SECONDS
|
||||
return self._cached_token
|
||||
|
||||
@staticmethod
|
||||
def _api_headers(token: str) -> dict[str, str]:
|
||||
return {
|
||||
"x-acs-dingtalk-access-token": token,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def _send_text_message_to_user(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_text_message_to_group(self, robot_code: str, conversation_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_p2p_message(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] P2P message sent to user=%s", user_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] P2P send response: %s", data)
|
||||
|
||||
async def _send_group_message(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
*,
|
||||
at_user_ids: list[str] | None = None, # noqa: ARG002
|
||||
) -> None:
|
||||
# at_user_ids accepted for call-site compatibility but not passed to the API
|
||||
# (sampleMarkdown does not support @mentions).
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] group message sent to conversation=%s", conversation_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] group send response: %s", data)
|
||||
|
||||
# -- AI Card streaming helpers -------------------------------------------
|
||||
|
||||
def _make_card_source_key(self, inbound: InboundMessage) -> str:
|
||||
m = inbound.metadata
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{m.get('message_id', '')}"
|
||||
|
||||
def _make_card_source_key_from_outbound(self, msg: OutboundMessage) -> str:
|
||||
m = msg.metadata
|
||||
correlation_id = m.get("message_id") or msg.thread_ts or ""
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{correlation_id}"
|
||||
|
||||
async def _create_and_deliver_card(
|
||||
self,
|
||||
initial_text: str,
|
||||
*,
|
||||
chatbot_message: Any = None,
|
||||
) -> str | None:
|
||||
if self._dingtalk_client is None or chatbot_message is None:
|
||||
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
|
||||
return None
|
||||
|
||||
try:
|
||||
from dingtalk_stream.card_replier import AICardReplier
|
||||
except ImportError:
|
||||
logger.warning("[DingTalk] dingtalk-stream card_replier not available")
|
||||
return None
|
||||
|
||||
try:
|
||||
replier = AICardReplier(self._dingtalk_client, chatbot_message)
|
||||
card_instance_id = await replier.async_create_and_deliver_card(
|
||||
card_template_id=self._card_template_id,
|
||||
card_data={"content": initial_text},
|
||||
)
|
||||
if not card_instance_id:
|
||||
return None
|
||||
|
||||
self._card_repliers[card_instance_id] = replier
|
||||
logger.info("[DingTalk] AI card created: outTrackId=%s", card_instance_id)
|
||||
return card_instance_id
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to create AI card")
|
||||
return None
|
||||
|
||||
async def _stream_update_card(
|
||||
self,
|
||||
out_track_id: str,
|
||||
content: str,
|
||||
*,
|
||||
is_finalize: bool = False,
|
||||
is_error: bool = False,
|
||||
) -> None:
|
||||
replier = self._card_repliers.get(out_track_id)
|
||||
if not replier:
|
||||
raise RuntimeError(f"No AICardReplier found for track ID {out_track_id}")
|
||||
|
||||
await replier.async_streaming(
|
||||
card_instance_id=out_track_id,
|
||||
content_key="content",
|
||||
content_value=content,
|
||||
append=False,
|
||||
finished=is_finalize,
|
||||
failed=is_error,
|
||||
)
|
||||
|
||||
# -- media upload --------------------------------------------------------
|
||||
|
||||
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
|
||||
try:
|
||||
file_bytes = await asyncio.to_thread(Path(file_path).read_bytes)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/files/upload",
|
||||
headers={"x-acs-dingtalk-access-token": token},
|
||||
files={"file": ("upload", file_bytes)},
|
||||
data={"type": media_type},
|
||||
)
|
||||
response.raise_for_status()
|
||||
try:
|
||||
payload = response.json()
|
||||
except json.JSONDecodeError:
|
||||
logger.exception("[DingTalk] failed to decode upload response JSON: %s", file_path)
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
logger.warning("[DingTalk] unexpected upload response type %s for %s", type(payload).__name__, file_path)
|
||||
return None
|
||||
return payload.get("mediaId")
|
||||
except (httpx.HTTPError, OSError):
|
||||
logger.exception("[DingTalk] failed to upload media: %s", file_path)
|
||||
return None
|
||||
|
||||
|
||||
class _DingTalkMessageHandler:
|
||||
"""Callback handler registered with dingtalk-stream."""
|
||||
|
||||
def __init__(self, channel: DingTalkChannel) -> None:
|
||||
self._channel = channel
|
||||
|
||||
def pre_start(self) -> None:
|
||||
if hasattr(self, "dingtalk_client") and self.dingtalk_client is not None:
|
||||
self._channel._dingtalk_client = self.dingtalk_client
|
||||
|
||||
async def raw_process(self, callback_message: Any) -> Any:
|
||||
import dingtalk_stream
|
||||
from dingtalk_stream.frames import Headers
|
||||
|
||||
code, message = await self.process(callback_message)
|
||||
ack_message = dingtalk_stream.AckMessage()
|
||||
ack_message.code = code
|
||||
ack_message.headers.message_id = callback_message.headers.message_id
|
||||
ack_message.headers.content_type = Headers.CONTENT_TYPE_APPLICATION_JSON
|
||||
ack_message.data = {"response": message}
|
||||
return ack_message
|
||||
|
||||
async def process(self, callback: Any) -> tuple[int, str]:
|
||||
import dingtalk_stream
|
||||
|
||||
incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
|
||||
self._channel._on_chatbot_message(incoming_message)
|
||||
return dingtalk_stream.AckMessage.STATUS_OK, "OK"
|
||||
@@ -0,0 +1,643 @@
|
||||
"""Discord channel integration using discord.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.connection_identity import attach_connection_identity
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DISCORD_MAX_MESSAGE_LEN = 2000
|
||||
|
||||
|
||||
class DiscordChannel(Channel):
|
||||
"""Discord bot channel.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.discord``):
|
||||
- ``bot_token``: Discord Bot token.
|
||||
- ``allowed_guilds``: (optional) List of allowed Discord guild IDs. Empty = allow all.
|
||||
- ``mention_only``: (optional) If true, only respond when the bot is mentioned.
|
||||
- ``allowed_channels``: (optional) List of channel IDs where messages are always accepted
|
||||
(even when mention_only is true). Use for channels where you want the bot to respond
|
||||
without mentions. Empty = mention_only applies everywhere.
|
||||
- ``thread_mode``: (optional) If true, group a channel conversation into a thread.
|
||||
Default: same as ``mention_only``.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="discord", bus=bus, config=config)
|
||||
self._bot_token = str(config.get("bot_token", "")).strip()
|
||||
self._allowed_guilds: set[int] = set()
|
||||
for guild_id in config.get("allowed_guilds", []):
|
||||
try:
|
||||
self._allowed_guilds.add(int(guild_id))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
self._mention_only: bool = bool(config.get("mention_only", False))
|
||||
self._thread_mode: bool = config.get("thread_mode", self._mention_only)
|
||||
self._allowed_channels: set[str] = set()
|
||||
for channel_id in config.get("allowed_channels", []):
|
||||
self._allowed_channels.add(str(channel_id))
|
||||
|
||||
# Session tracking: channel_id -> Discord thread_id (in-memory, persisted to JSON).
|
||||
# Uses a dedicated JSON file separate from ChannelStore, which maps IM
|
||||
# conversations to DeerFlow thread IDs — a different concern.
|
||||
self._active_threads: dict[str, str] = {}
|
||||
# Reverse-lookup set for O(1) thread ID checks (avoids O(n) scan of _active_threads.values()).
|
||||
self._active_thread_ids: set[str] = set()
|
||||
# Lock protecting _active_threads and the JSON file from concurrent access.
|
||||
# _run_client (Discord loop thread) and the main thread both read/write.
|
||||
self._thread_store_lock = threading.Lock()
|
||||
store = config.get("channel_store")
|
||||
if store is not None:
|
||||
self._thread_store_path = store._path.parent / "discord_threads.json"
|
||||
else:
|
||||
self._thread_store_path = Path.home() / ".deer-flow" / "channels" / "discord_threads.json"
|
||||
|
||||
# Typing indicator management
|
||||
self._typing_tasks: dict[str, asyncio.Task] = {}
|
||||
|
||||
self._client = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._discord_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._discord_module = None
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import discord
|
||||
except ImportError:
|
||||
logger.error("discord.py is not installed. Install it with: uv add discord.py")
|
||||
return
|
||||
|
||||
if not self._bot_token:
|
||||
logger.error("Discord channel requires bot_token")
|
||||
return
|
||||
|
||||
intents = discord.Intents.default()
|
||||
intents.messages = True
|
||||
intents.guilds = True
|
||||
intents.message_content = True
|
||||
|
||||
client = discord.Client(
|
||||
intents=intents,
|
||||
allowed_mentions=discord.AllowedMentions.none(),
|
||||
)
|
||||
self._client = client
|
||||
self._discord_module = discord
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
|
||||
@client.event
|
||||
async def on_message(message) -> None:
|
||||
await self._on_message(message)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(target=self._run_client, daemon=True)
|
||||
self._thread.start()
|
||||
await asyncio.to_thread(self._load_active_threads)
|
||||
logger.info("Discord channel started")
|
||||
|
||||
def _load_active_threads(self) -> None:
|
||||
"""Restore Discord thread mappings from the dedicated JSON file on startup."""
|
||||
with self._thread_store_lock:
|
||||
try:
|
||||
if not self._thread_store_path.exists():
|
||||
logger.debug("[Discord] no thread mappings file at %s", self._thread_store_path)
|
||||
return
|
||||
data = json.loads(self._thread_store_path.read_text())
|
||||
self._active_threads.clear()
|
||||
self._active_thread_ids.clear()
|
||||
for channel_id, thread_id in data.items():
|
||||
self._active_threads[channel_id] = thread_id
|
||||
self._active_thread_ids.add(thread_id)
|
||||
if self._active_threads:
|
||||
logger.info("[Discord] restored %d thread mappings from %s", len(self._active_threads), self._thread_store_path)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to load thread mappings")
|
||||
|
||||
def _record_thread_mapping(self, channel_id: str, thread_id: str) -> None:
|
||||
"""Synchronously update the in-memory channel->thread mapping and its reverse-lookup set.
|
||||
|
||||
Runs on the event loop (no IO, no await) so a follow-up message in the
|
||||
newly created thread is recognized immediately, before the offloaded
|
||||
persistence write completes. Deferring this update into the worker
|
||||
thread opened a window where ``_active_thread_ids`` had not yet been
|
||||
updated and an inbound message was misclassified as orphaned (see the
|
||||
#3927 review). Persistence is handled separately by
|
||||
``_persist_thread_mappings``.
|
||||
"""
|
||||
old_id = self._active_threads.get(channel_id)
|
||||
self._active_threads[channel_id] = thread_id
|
||||
if old_id:
|
||||
self._active_thread_ids.discard(old_id)
|
||||
self._active_thread_ids.add(thread_id)
|
||||
|
||||
def _persist_thread_mappings(self) -> None:
|
||||
"""Flush the current in-memory thread mappings to disk.
|
||||
|
||||
Intended for ``asyncio.to_thread``: this is pure filesystem IO. The
|
||||
in-memory state is updated synchronously by ``_record_thread_mapping``,
|
||||
so persistence latency never delays visibility of a new mapping to
|
||||
inbound-message handling. The mapping is snapshotted under the store
|
||||
lock so a concurrent record cannot mutate the dict mid-serialization.
|
||||
"""
|
||||
with self._thread_store_lock:
|
||||
try:
|
||||
snapshot = dict(self._active_threads)
|
||||
self._thread_store_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._thread_store_path.write_text(json.dumps(snapshot, indent=2))
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to persist thread mappings")
|
||||
|
||||
@staticmethod
|
||||
def _read_attachment_bytes(path: str) -> bytes:
|
||||
"""Read an attachment file synchronously (intended for ``asyncio.to_thread``)."""
|
||||
with open(path, "rb") as fp:
|
||||
return fp.read()
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
# Cancel all active typing indicator tasks
|
||||
for target_id, task in list(self._typing_tasks.items()):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
logger.debug("[Discord] cancelled typing task for target %s", target_id)
|
||||
self._typing_tasks.clear()
|
||||
|
||||
if self._client and self._discord_loop and self._discord_loop.is_running():
|
||||
close_future = asyncio.run_coroutine_threadsafe(self._client.close(), self._discord_loop)
|
||||
try:
|
||||
await asyncio.wait_for(asyncio.wrap_future(close_future), timeout=10)
|
||||
except TimeoutError:
|
||||
logger.warning("[Discord] client close timed out after 10s")
|
||||
except Exception:
|
||||
logger.exception("[Discord] error while closing client")
|
||||
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
|
||||
self._client = None
|
||||
self._discord_loop = None
|
||||
self._discord_module = None
|
||||
logger.info("Discord channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
# Stop typing indicator once we're sending the response
|
||||
stop_future = asyncio.run_coroutine_threadsafe(self._stop_typing(msg.chat_id, msg.thread_ts), self._discord_loop)
|
||||
await asyncio.wrap_future(stop_future)
|
||||
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return
|
||||
|
||||
text = msg.text or ""
|
||||
for chunk in self._split_text(text):
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(chunk), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
stop_future = asyncio.run_coroutine_threadsafe(self._stop_typing(msg.chat_id, msg.thread_ts), self._discord_loop)
|
||||
await asyncio.wrap_future(stop_future)
|
||||
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for file upload chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return False
|
||||
|
||||
if self._discord_module is None:
|
||||
return False
|
||||
|
||||
try:
|
||||
# Read the attachment off the event loop (open + read are blocking IO),
|
||||
# then hand discord.py an in-memory buffer. The bytes are consumed while
|
||||
# ``target.send`` runs on ``_discord_loop``; once that future resolves the
|
||||
# buffer can be reclaimed, so this avoids leaking a file handle on both the
|
||||
# success and failure paths.
|
||||
data = await asyncio.to_thread(self._read_attachment_bytes, str(attachment.actual_path))
|
||||
file = self._discord_module.File(io.BytesIO(data), filename=attachment.filename)
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(file=file), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
logger.info("[Discord] file uploaded: %s", attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _start_typing(self, channel, chat_id: str, thread_ts: str | None = None) -> None:
|
||||
"""Starts a loop to send periodic typing indicators."""
|
||||
target_id = thread_ts or chat_id
|
||||
if target_id in self._typing_tasks:
|
||||
return # Already typing for this target
|
||||
|
||||
async def _typing_loop():
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
await channel.trigger_typing()
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(10)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
task = asyncio.create_task(_typing_loop())
|
||||
self._typing_tasks[target_id] = task
|
||||
|
||||
async def _stop_typing(self, chat_id: str, thread_ts: str | None = None) -> None:
|
||||
"""Stops the typing loop for a specific target."""
|
||||
target_id = thread_ts or chat_id
|
||||
task = self._typing_tasks.pop(target_id, None)
|
||||
if task and not task.done():
|
||||
task.cancel()
|
||||
logger.debug("[Discord] stopped typing indicator for target %s", target_id)
|
||||
|
||||
async def _add_reaction(self, message) -> None:
|
||||
"""Add a checkmark reaction to acknowledge the message was received."""
|
||||
try:
|
||||
await message.add_reaction("✅")
|
||||
except Exception:
|
||||
logger.debug("[Discord] failed to add reaction to message %s", message.id, exc_info=True)
|
||||
|
||||
async def _on_message(self, message) -> None:
|
||||
if not self._running or not self._client:
|
||||
return
|
||||
|
||||
if message.author.bot:
|
||||
return
|
||||
|
||||
if self._client.user and message.author.id == self._client.user.id:
|
||||
return
|
||||
|
||||
guild = message.guild
|
||||
if self._allowed_guilds:
|
||||
if guild is None or guild.id not in self._allowed_guilds:
|
||||
return
|
||||
|
||||
text = (message.content or "").strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
if self._discord_module is None:
|
||||
return
|
||||
|
||||
# Determine whether the bot is mentioned in this message
|
||||
user = self._client.user if self._client else None
|
||||
if user:
|
||||
bot_mention = user.mention # <@ID>
|
||||
alt_mention = f"<@!{user.id}>" # <@!ID> (ping variant)
|
||||
standard_mention = f"<@{user.id}>"
|
||||
else:
|
||||
bot_mention = None
|
||||
alt_mention = None
|
||||
standard_mention = ""
|
||||
has_mention = (bot_mention and bot_mention in message.content) or (alt_mention and alt_mention in message.content) or (standard_mention and standard_mention in message.content)
|
||||
|
||||
# Strip mention from text for processing
|
||||
if has_mention:
|
||||
text = text.replace(bot_mention or "", "").replace(alt_mention or "", "").replace(standard_mention or "", "").strip()
|
||||
# Don't return early if text is empty — still process the mention (e.g., create thread)
|
||||
|
||||
connect_code = self._pending_connect_code(text)
|
||||
if connect_code and await self._bind_connection_from_connect_code(message, connect_code):
|
||||
return
|
||||
|
||||
# --- Determine thread/channel routing and typing target ---
|
||||
thread_id = None
|
||||
chat_id = None
|
||||
typing_target = None # The Discord object to type into
|
||||
|
||||
if isinstance(message.channel, self._discord_module.Thread):
|
||||
# --- Message already inside a thread ---
|
||||
thread_obj = message.channel
|
||||
thread_id = str(thread_obj.id)
|
||||
chat_id = str(thread_obj.parent_id or thread_obj.id)
|
||||
typing_target = thread_obj
|
||||
|
||||
# If this is a known active thread, process normally
|
||||
if thread_id in self._active_thread_ids:
|
||||
msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=str(message.author.id),
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_id,
|
||||
metadata={
|
||||
"guild_id": str(guild.id) if guild else None,
|
||||
"channel_id": str(message.channel.id),
|
||||
"message_id": str(message.id),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_id
|
||||
inbound = await self._attach_connection_identity(inbound, guild_id=str(guild.id) if guild else None)
|
||||
self._publish(inbound)
|
||||
# Start typing indicator in the thread
|
||||
if typing_target:
|
||||
asyncio.create_task(self._start_typing(typing_target, chat_id, thread_id))
|
||||
asyncio.create_task(self._add_reaction(message))
|
||||
return
|
||||
|
||||
# Thread not tracked (orphaned) — create new thread and handle below
|
||||
logger.debug("[Discord] message in orphaned thread %s, will create new thread", thread_id)
|
||||
thread_id = None
|
||||
typing_target = None
|
||||
|
||||
# At this point we're guaranteed to be in a channel, not a thread
|
||||
# (the Thread case is handled above). Apply mention_only for all
|
||||
# non-thread messages — no special case needed.
|
||||
channel_id = str(message.channel.id)
|
||||
|
||||
# Check if there's an active thread for this channel
|
||||
if channel_id in self._active_threads:
|
||||
# respect mention_only: if enabled, only process messages that mention the bot
|
||||
# (unless the channel is in allowed_channels)
|
||||
# Messages within a thread are always allowed through (continuation).
|
||||
# At this code point we know the message is in a channel, not a thread
|
||||
# (Thread case handled above), so always apply the check.
|
||||
if self._mention_only and not has_mention and channel_id not in self._allowed_channels:
|
||||
logger.debug("[Discord] skipping no-@ message in channel %s (not in thread)", channel_id)
|
||||
return
|
||||
# mention_only + fresh @ → create new thread instead of routing to existing one
|
||||
if self._mention_only and has_mention:
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is not None:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._record_thread_mapping(channel_id, target_thread_id)
|
||||
await asyncio.to_thread(self._persist_thread_mappings)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj
|
||||
logger.info("[Discord] created new thread %s in channel %s on mention (replacing existing thread)", target_thread_id, channel_id)
|
||||
else:
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel
|
||||
else:
|
||||
# Existing session → route to the existing thread
|
||||
target_thread_id = self._active_threads[channel_id]
|
||||
logger.debug("[Discord] routing message in channel %s to existing thread %s", channel_id, target_thread_id)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = await self._get_channel_or_thread(target_thread_id)
|
||||
elif self._mention_only and not has_mention and channel_id not in self._allowed_channels:
|
||||
# Not mentioned and not in an allowed channel → skip
|
||||
logger.debug("[Discord] skipping message without mention in channel %s", channel_id)
|
||||
return
|
||||
elif self._mention_only and has_mention:
|
||||
# First mention in this channel → create thread
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is not None:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._record_thread_mapping(channel_id, target_thread_id)
|
||||
await asyncio.to_thread(self._persist_thread_mappings)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj # Type into the new thread
|
||||
logger.info("[Discord] created thread %s in channel %s for user %s", target_thread_id, channel_id, message.author.display_name)
|
||||
else:
|
||||
# Fallback: thread creation failed (disabled/permissions), reply in channel
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
elif self._thread_mode:
|
||||
# thread_mode but mention_only is False → create thread anyway for conversation grouping
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is None:
|
||||
# Thread creation failed (disabled/permissions), fall back to channel replies
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
else:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._record_thread_mapping(channel_id, target_thread_id)
|
||||
await asyncio.to_thread(self._persist_thread_mappings)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj # Type into the new thread
|
||||
else:
|
||||
# No threading — reply directly in channel
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
|
||||
msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=str(message.author.id),
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_id,
|
||||
metadata={
|
||||
"guild_id": str(guild.id) if guild else None,
|
||||
"channel_id": str(message.channel.id),
|
||||
"message_id": str(message.id),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_id
|
||||
inbound = await self._attach_connection_identity(inbound, guild_id=str(guild.id) if guild else None)
|
||||
|
||||
# Start typing indicator in the correct target (thread or channel)
|
||||
if typing_target:
|
||||
asyncio.create_task(self._start_typing(typing_target, chat_id, thread_id))
|
||||
|
||||
self._publish(inbound)
|
||||
asyncio.create_task(self._add_reaction(message))
|
||||
|
||||
def _publish(self, inbound) -> None:
|
||||
"""Publish an inbound message to the main event loop."""
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
future = asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._main_loop)
|
||||
future.add_done_callback(lambda f: logger.exception("[Discord] publish_inbound failed", exc_info=f.exception()) if f.exception() else None)
|
||||
|
||||
async def _attach_connection_identity(self, inbound: InboundMessage, guild_id: str | None = None) -> InboundMessage:
|
||||
return await attach_connection_identity(
|
||||
inbound,
|
||||
repo=self._connection_repo,
|
||||
provider="discord",
|
||||
workspace_id=guild_id,
|
||||
fallback_without_workspace=True,
|
||||
)
|
||||
|
||||
async def _bind_connection_from_connect_code(self, message, code: str) -> bool:
|
||||
if self._connection_repo is None or not code:
|
||||
return False
|
||||
|
||||
state = await self._connection_repo.consume_oauth_state(provider="discord", state=code)
|
||||
if state is None:
|
||||
await self._send_connection_reply(message, "Discord connection code is invalid or expired.")
|
||||
return True
|
||||
|
||||
guild = getattr(message, "guild", None)
|
||||
channel = getattr(message, "channel", None)
|
||||
author = getattr(message, "author", None)
|
||||
user_id = str(getattr(author, "id", "") or "")
|
||||
if not user_id:
|
||||
await self._send_connection_reply(message, "Discord connection could not be completed from this message.")
|
||||
return True
|
||||
|
||||
guild_id = str(getattr(guild, "id", "") or "") or None
|
||||
await self._connection_repo.upsert_connection(
|
||||
owner_user_id=state["owner_user_id"],
|
||||
provider="discord",
|
||||
external_account_id=user_id,
|
||||
external_account_name=getattr(author, "display_name", None) or getattr(author, "name", None),
|
||||
workspace_id=guild_id,
|
||||
workspace_name=getattr(guild, "name", None) if guild is not None else None,
|
||||
metadata={
|
||||
"guild_id": guild_id,
|
||||
"channel_id": str(getattr(channel, "id", "") or ""),
|
||||
},
|
||||
status="connected",
|
||||
)
|
||||
await self._send_connection_reply(message, "Discord connected to DeerFlow.")
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
async def _send_connection_reply(message, text: str) -> None:
|
||||
channel = getattr(message, "channel", None)
|
||||
send = getattr(channel, "send", None)
|
||||
if send is None:
|
||||
return
|
||||
try:
|
||||
await send(text)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to send connection reply")
|
||||
|
||||
def _run_client(self) -> None:
|
||||
self._discord_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._discord_loop)
|
||||
try:
|
||||
self._discord_loop.run_until_complete(self._client.start(self._bot_token))
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Discord client error")
|
||||
finally:
|
||||
try:
|
||||
if self._client and not self._client.is_closed():
|
||||
self._discord_loop.run_until_complete(self._client.close())
|
||||
except Exception:
|
||||
logger.exception("Error during Discord shutdown")
|
||||
|
||||
async def _create_thread(self, message):
|
||||
try:
|
||||
if self._discord_module is None:
|
||||
return None
|
||||
|
||||
# Only TextChannel (type 0) and NewsChannel (type 10) support threads
|
||||
channel_type = message.channel.type
|
||||
if channel_type not in (
|
||||
self._discord_module.ChannelType.text,
|
||||
self._discord_module.ChannelType.news,
|
||||
):
|
||||
logger.info(
|
||||
"[Discord] channel type %s (%s) does not support threads",
|
||||
channel_type.value,
|
||||
channel_type.name,
|
||||
)
|
||||
return None
|
||||
|
||||
thread_name = f"deerflow-{message.author.display_name}-{message.id}"[:100]
|
||||
return await message.create_thread(name=thread_name)
|
||||
except self._discord_module.errors.HTTPException as exc:
|
||||
if exc.code == 50024:
|
||||
logger.info(
|
||||
"[Discord] cannot create thread in channel %s (error code 50024): %s",
|
||||
message.channel.id,
|
||||
channel_type.name if (channel_type := message.channel.type) else "unknown",
|
||||
)
|
||||
else:
|
||||
logger.exception(
|
||||
"[Discord] failed to create thread for message=%s (HTTPException %s)",
|
||||
message.id,
|
||||
exc.code,
|
||||
)
|
||||
return None
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to create thread for message=%s (threads may be disabled or missing permissions)", message.id)
|
||||
return None
|
||||
|
||||
async def _resolve_target(self, msg: OutboundMessage):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
target_ids: list[str] = []
|
||||
if msg.thread_ts:
|
||||
target_ids.append(msg.thread_ts)
|
||||
if msg.chat_id and msg.chat_id not in target_ids:
|
||||
target_ids.append(msg.chat_id)
|
||||
|
||||
for raw_id in target_ids:
|
||||
target = await self._get_channel_or_thread(raw_id)
|
||||
if target is not None:
|
||||
return target
|
||||
return None
|
||||
|
||||
async def _get_channel_or_thread(self, raw_id: str):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
try:
|
||||
target_id = int(raw_id)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
get_future = asyncio.run_coroutine_threadsafe(self._fetch_channel(target_id), self._discord_loop)
|
||||
try:
|
||||
return await asyncio.wrap_future(get_future)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to resolve target id=%s", raw_id)
|
||||
return None
|
||||
|
||||
async def _fetch_channel(self, target_id: int):
|
||||
if not self._client:
|
||||
return None
|
||||
|
||||
channel = self._client.get_channel(target_id)
|
||||
if channel is not None:
|
||||
return channel
|
||||
|
||||
try:
|
||||
return await self._client.fetch_channel(target_id)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _split_text(text: str) -> list[str]:
|
||||
if not text:
|
||||
return [""]
|
||||
|
||||
chunks: list[str] = []
|
||||
remaining = text
|
||||
while len(remaining) > _DISCORD_MAX_MESSAGE_LEN:
|
||||
split_at = remaining.rfind("\n", 0, _DISCORD_MAX_MESSAGE_LEN)
|
||||
if split_at <= 0:
|
||||
split_at = _DISCORD_MAX_MESSAGE_LEN
|
||||
chunks.append(remaining[:split_at])
|
||||
remaining = remaining[split_at:].lstrip("\n")
|
||||
|
||||
if remaining:
|
||||
chunks.append(remaining)
|
||||
|
||||
return chunks
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,15 @@
|
||||
"""Per-run policy registration for the Feishu channel."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from app.channels.run_policy import CHANNEL_RUN_POLICY, ChannelRunPolicy
|
||||
|
||||
|
||||
def register_policy() -> None:
|
||||
"""Register Feishu's queue-same-thread behavior in the shared policy map."""
|
||||
CHANNEL_RUN_POLICY["feishu"] = ChannelRunPolicy(
|
||||
serialize_thread_runs=True,
|
||||
)
|
||||
|
||||
|
||||
register_policy()
|
||||
@@ -0,0 +1,115 @@
|
||||
"""GitHub channel — webhook-driven IM channel for PR/issue comments.
|
||||
|
||||
Unlike other IM channels (Feishu, Slack, Telegram) which long-poll or use
|
||||
WebSockets, GitHub delivers messages via HTTP push webhooks. This channel
|
||||
therefore has a no-op ``start``/``stop`` — inbound messages arrive through
|
||||
``POST /api/webhooks/github`` and are published to the bus by the webhook
|
||||
route handler.
|
||||
|
||||
**The channel does not auto-post the agent's final response.** Each GitHub
|
||||
agent (coder, reviewer, …) has the `gh` CLI in its sandbox and is expected to
|
||||
decide for itself what — if anything — to post on the issue or PR, and to use
|
||||
``gh issue comment`` / ``gh pr comment`` / ``gh pr create`` during the run.
|
||||
The agent's final assistant message is logged at INFO for visibility in
|
||||
``gateway.log`` but is **not** sent to GitHub.
|
||||
|
||||
Why log-only rather than auto-post:
|
||||
|
||||
- Two agents can bind the same event (e.g. coder + reviewer on a mention).
|
||||
If both auto-posted their final messages, the user would see two replies
|
||||
for every mention even when only one had useful work to do. Letting the
|
||||
LLM call ``gh`` mid-run means silence is just "the LLM did not call gh."
|
||||
- The agent often wants to post *intermediate* updates (an issue comment
|
||||
linking the PR, a separate comment on a new sub-issue, …) — the
|
||||
auto-post-the-final-message contract didn't model that and forced the
|
||||
final message to play double duty.
|
||||
- The dispatcher's per-agent ``_is_self_event`` gate already prevents the
|
||||
comments the LLM posts via ``gh`` from looping the webhook back into a
|
||||
new run for the same agent.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import MessageBus, OutboundMessage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GitHubChannel(Channel):
|
||||
"""Webhook-driven GitHub channel.
|
||||
|
||||
Inbound: ``POST /api/webhooks/github`` publishes ``InboundMessage`` to
|
||||
the bus. Outbound: ``send`` is log-only (see module docstring) — agents
|
||||
post to GitHub themselves via the ``gh`` CLI in their sandbox.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.github``):
|
||||
|
||||
- ``enabled`` (bool): set to ``true`` to activate.
|
||||
- ``default_mention_login`` (str, optional): bot handle used by
|
||||
``require_mention`` when the agent binding does not set one.
|
||||
Falls back to ``"deerflow-bot"``.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="github", bus=bus, config=config)
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Register the outbound callback.
|
||||
|
||||
GitHub is push-based (webhooks), so no long-poll or socket
|
||||
listener is needed. We only register for outbound replies so the
|
||||
agent's final message gets logged.
|
||||
"""
|
||||
if self._running:
|
||||
return
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
self._running = True
|
||||
logger.info("GitHubChannel started (webhook-driven, no polling)")
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Unregister the outbound callback."""
|
||||
if not self._running:
|
||||
return
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
self._running = False
|
||||
logger.info("GitHubChannel stopped")
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
"""Log the agent's final message — do NOT post it to GitHub.
|
||||
|
||||
GitHub agents post to issues/PRs themselves via ``gh`` mid-run; the
|
||||
final assistant message is logged for ``gateway.log`` visibility but
|
||||
is not delivered to the platform. See the module docstring for why.
|
||||
|
||||
Metadata layout (read for logging context only):
|
||||
- ``repo`` (str, e.g. ``"owner/name"``) — falls back to ``chat_id``
|
||||
- ``number`` (int, issue or PR number)
|
||||
- ``installation_id`` (int)
|
||||
"""
|
||||
gh = msg.metadata.get("github", {}) if isinstance(msg.metadata, dict) else {}
|
||||
if not isinstance(gh, dict):
|
||||
gh = {}
|
||||
|
||||
repo = gh.get("repo") or msg.chat_id
|
||||
number = gh.get("number")
|
||||
|
||||
body = msg.text or ""
|
||||
|
||||
logger.info(
|
||||
"[GitHubChannel] final message from agent for %s#%s (text_len=%d) — not posted; agents use `gh` directly",
|
||||
repo,
|
||||
number,
|
||||
len(body),
|
||||
)
|
||||
# Mirror the body itself at DEBUG so operators can correlate without
|
||||
# spamming INFO. Truncate to keep log lines bounded.
|
||||
if body:
|
||||
logger.debug("[GitHubChannel] final body (truncated to 2000 chars): %s", body[:2000])
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,190 @@
|
||||
"""MessageBus — async pub/sub hub that decouples channels from the agent dispatcher."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Callable, Coroutine
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PENDING_CLARIFICATION_METADATA_KEY = "pending_clarification"
|
||||
RESOLVED_FROM_PENDING_CLARIFICATION_METADATA_KEY = "resolved_from_pending_clarification"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class InboundMessageType(StrEnum):
|
||||
"""Types of messages arriving from IM channels."""
|
||||
|
||||
CHAT = "chat"
|
||||
COMMAND = "command"
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundMessage:
|
||||
"""A message arriving from an IM channel toward the agent dispatcher.
|
||||
|
||||
Attributes:
|
||||
channel_name: Name of the source channel (e.g. "feishu", "slack").
|
||||
chat_id: Platform-specific chat/conversation identifier.
|
||||
user_id: Platform-specific user identifier.
|
||||
text: The message text.
|
||||
msg_type: Whether this is a regular chat message or a command.
|
||||
thread_ts: Optional platform thread identifier (for threaded replies).
|
||||
topic_id: Conversation topic identifier used to map to a DeerFlow thread.
|
||||
Messages sharing the same ``topic_id`` within a ``chat_id`` will
|
||||
reuse the same DeerFlow thread. When ``None``, each message
|
||||
creates a new thread (one-shot Q&A).
|
||||
connection_id: Optional DeerFlow channel connection id. When present,
|
||||
conversation mapping is scoped by the connection instead of the
|
||||
legacy global ``channel_name:chat_id[:topic_id]`` key.
|
||||
owner_user_id: DeerFlow user id that owns the channel connection.
|
||||
Platform user ids stay in ``user_id``.
|
||||
workspace_id: Optional external workspace/guild/team id.
|
||||
files: Optional list of file attachments (platform-specific dicts).
|
||||
metadata: Arbitrary extra data from the channel.
|
||||
created_at: Unix timestamp when the message was created.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
user_id: str
|
||||
text: str
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT
|
||||
thread_ts: str | None = None
|
||||
topic_id: str | None = None
|
||||
connection_id: str | None = None
|
||||
owner_user_id: str | None = None
|
||||
workspace_id: str | None = None
|
||||
files: list[dict[str, Any]] = field(default_factory=list)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResolvedAttachment:
|
||||
"""A file attachment resolved to a host filesystem path, ready for upload.
|
||||
|
||||
Attributes:
|
||||
virtual_path: Original virtual path (e.g. /mnt/user-data/outputs/report.pdf).
|
||||
actual_path: Resolved host filesystem path.
|
||||
filename: Basename of the file.
|
||||
mime_type: MIME type (e.g. "application/pdf").
|
||||
size: File size in bytes.
|
||||
is_image: True for image/* MIME types (platforms may handle images differently).
|
||||
"""
|
||||
|
||||
virtual_path: str
|
||||
actual_path: Path
|
||||
filename: str
|
||||
mime_type: str
|
||||
size: int
|
||||
is_image: bool
|
||||
|
||||
|
||||
@dataclass
|
||||
class OutboundMessage:
|
||||
"""A message from the agent dispatcher back to a channel.
|
||||
|
||||
Attributes:
|
||||
channel_name: Target channel name (used for routing).
|
||||
chat_id: Target chat/conversation identifier.
|
||||
thread_id: DeerFlow thread ID that produced this response.
|
||||
text: The response text.
|
||||
artifacts: List of artifact paths produced by the agent.
|
||||
is_final: Whether this is the final message in the response stream.
|
||||
thread_ts: Optional platform thread identifier for threaded replies.
|
||||
metadata: Arbitrary extra data.
|
||||
connection_id: Optional DeerFlow channel connection id used for
|
||||
connection-specific outbound credentials.
|
||||
owner_user_id: DeerFlow user id that owns the channel connection.
|
||||
created_at: Unix timestamp.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
thread_id: str
|
||||
text: str
|
||||
artifacts: list[str] = field(default_factory=list)
|
||||
attachments: list[ResolvedAttachment] = field(default_factory=list)
|
||||
is_final: bool = True
|
||||
thread_ts: str | None = None
|
||||
connection_id: str | None = None
|
||||
owner_user_id: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MessageBus
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
OutboundCallback = Callable[[OutboundMessage], Coroutine[Any, Any, None]]
|
||||
|
||||
|
||||
class MessageBus:
|
||||
"""Async pub/sub hub connecting channels and the agent dispatcher.
|
||||
|
||||
Channels publish inbound messages; the dispatcher consumes them.
|
||||
The dispatcher publishes outbound messages; channels receive them
|
||||
via registered callbacks.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._inbound_queue: asyncio.Queue[InboundMessage] = asyncio.Queue()
|
||||
self._outbound_listeners: list[OutboundCallback] = []
|
||||
|
||||
# -- inbound -----------------------------------------------------------
|
||||
|
||||
async def publish_inbound(self, msg: InboundMessage) -> None:
|
||||
"""Enqueue an inbound message from a channel."""
|
||||
await self._inbound_queue.put(msg)
|
||||
logger.info(
|
||||
"[Bus] inbound enqueued: channel=%s, chat_id=%s, type=%s, queue_size=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
msg.msg_type.value,
|
||||
self._inbound_queue.qsize(),
|
||||
)
|
||||
|
||||
async def get_inbound(self) -> InboundMessage:
|
||||
"""Block until the next inbound message is available."""
|
||||
return await self._inbound_queue.get()
|
||||
|
||||
@property
|
||||
def inbound_queue(self) -> asyncio.Queue[InboundMessage]:
|
||||
return self._inbound_queue
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
def subscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Register an async callback for outbound messages."""
|
||||
self._outbound_listeners.append(callback)
|
||||
|
||||
def unsubscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Remove a previously registered outbound callback."""
|
||||
self._outbound_listeners = [cb for cb in self._outbound_listeners if cb != callback]
|
||||
|
||||
async def publish_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Dispatch an outbound message to all registered listeners."""
|
||||
logger.info(
|
||||
"[Bus] outbound dispatching: channel=%s, chat_id=%s, listeners=%d, text_len=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
len(self._outbound_listeners),
|
||||
len(msg.text),
|
||||
)
|
||||
for callback in self._outbound_listeners:
|
||||
try:
|
||||
await callback(msg)
|
||||
except Exception:
|
||||
logger.exception("Error in outbound callback for channel=%s", msg.channel_name)
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Per-channel run policy registry.
|
||||
|
||||
Holds the global ``CHANNEL_RUN_POLICY`` map and its :class:`ChannelRunPolicy`
|
||||
descriptor. Split into its own module so channels can register their own
|
||||
policy entries (typically as a side-effect of importing their package)
|
||||
without creating a circular dependency on :mod:`app.channels.manager`.
|
||||
|
||||
The dispatch path in :class:`app.channels.manager.ChannelManager` looks
|
||||
up policy entries by ``msg.channel_name`` and applies them after
|
||||
``_resolve_run_params``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.channels.message_bus import InboundMessage
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ChannelRunPolicy:
|
||||
"""Per-channel knobs applied by :meth:`ChannelManager._apply_channel_policy`.
|
||||
|
||||
Webhook-driven channels (GitHub today; others later) need four
|
||||
things the generic interactive-chat path does not: a higher
|
||||
``recursion_limit`` for autonomous long runs, suppression of
|
||||
``ask_clarification`` (no human is synchronously present), a
|
||||
credentials provider that mints platform tokens for the agent, and
|
||||
an opt-out from the per-sender bound-identity gate (authenticity is
|
||||
enforced at the webhook route by HMAC, and there is no equivalent
|
||||
of a per-user ``/connect`` handshake to perform).
|
||||
|
||||
Declaring all four on one dataclass keeps the channel's run
|
||||
behavior in a single discoverable place and turns "add a new
|
||||
webhook channel" into a one-row registration instead of touching
|
||||
multiple separate methods on the manager.
|
||||
|
||||
Attributes:
|
||||
is_interactive: When False, the manager sets
|
||||
``run_context["disable_clarification"] = True`` so
|
||||
``ClarificationMiddleware`` returns a "proceed with best
|
||||
judgment" ToolMessage instead of interrupting via
|
||||
``Command(goto=END)``. Defaults to True (the safe default
|
||||
for an IM channel).
|
||||
default_recursion_limit: When set, the manager raises
|
||||
``run_config["recursion_limit"]`` to ``max(existing,
|
||||
limit)``. None leaves the global default (100) untouched —
|
||||
interactive chat turns don't need 250 super-steps.
|
||||
credentials_provider: Optional async hook that mutates
|
||||
``run_context`` with platform-specific credentials. Called
|
||||
after ``_resolve_run_params``. Exceptions are caught and
|
||||
logged so a credential failure degrades gracefully (agent
|
||||
runs read-only) instead of dropping the delivery.
|
||||
requires_bound_identity: When False, the manager skips the
|
||||
per-sender bound-identity gate (``_get_bound_identity_rejection``)
|
||||
for this channel even when ``channel_connections.enabled`` is
|
||||
on. Webhook-authenticated channels (GitHub) have no
|
||||
per-sender ``/connect`` handshake — authenticity is enforced
|
||||
by HMAC at the webhook route, and the binding from "sender"
|
||||
to DeerFlow user is encoded in the agent's ``config.yaml``
|
||||
ownership, not in the channel-connections table. Defaults to
|
||||
True (the safe default for an interactive IM channel).
|
||||
fire_and_forget: When True, the manager schedules the run with
|
||||
``runs.create`` (returns immediately once the run is
|
||||
``pending``) instead of ``runs.wait`` (which keeps an HTTP
|
||||
stream open for the entire run lifetime). Channels that do
|
||||
their own outbound during the run — e.g. GitHub, where the
|
||||
agent posts to the issue/PR via the ``gh`` CLI in its
|
||||
sandbox — don't need the manager to ferry a final state
|
||||
back. Eliminates the SDK's 300s ``httpx.ReadTimeout`` on
|
||||
runs that legitimately take more than 5 minutes, and the
|
||||
false "internal error" outbound that follows when it
|
||||
fires. Defaults to False (the safe default for an
|
||||
interactive IM channel that depends on the manager to
|
||||
publish the agent's reply).
|
||||
serialize_thread_runs: When True, the manager serializes
|
||||
same-thread inbound turns for this channel instead of
|
||||
surfacing the runtime's generic busy-thread error. This is
|
||||
useful for chat surfaces like Feishu topics where rapid
|
||||
follow-up messages should queue behind the active turn while
|
||||
unrelated DeerFlow threads continue concurrently. Defaults
|
||||
to False so existing channels keep the runtime's native
|
||||
multitask behavior unless they opt in explicitly.
|
||||
"""
|
||||
|
||||
is_interactive: bool = True
|
||||
default_recursion_limit: int | None = None
|
||||
credentials_provider: Callable[[InboundMessage, dict[str, Any]], Awaitable[None]] | None = None
|
||||
requires_bound_identity: bool = True
|
||||
fire_and_forget: bool = False
|
||||
serialize_thread_runs: bool = False
|
||||
|
||||
|
||||
# Channel name → policy. Channels absent from this map fall through to
|
||||
# the policy default (an interactive IM channel with no credential
|
||||
# plumbing) — which is what every IM channel had before GitHub. Webhook
|
||||
# channels register their entry at package-import time (see
|
||||
# ``app.gateway.github.run_policy``).
|
||||
CHANNEL_RUN_POLICY: dict[str, ChannelRunPolicy] = {}
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Local persistence for runtime IM channel configuration."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
RUNTIME_CHANNEL_DISABLED_FLAG = "_runtime_disabled"
|
||||
|
||||
|
||||
class ChannelRuntimeConfigStore:
|
||||
"""JSON-backed store for channel credentials entered from the UI.
|
||||
|
||||
This intentionally mirrors ``ChannelStore``: local/private deployments get
|
||||
durable runtime configuration without needing a public callback URL or a
|
||||
config.yaml edit.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
if path is None:
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
path = Path(get_paths().base_dir) / "channels" / "runtime-config.json"
|
||||
self._path = Path(path)
|
||||
self._path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._data: dict[str, dict[str, Any]] = self._load()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def _load(self) -> dict[str, dict[str, Any]]:
|
||||
if self._path.exists():
|
||||
try:
|
||||
raw = json.loads(self._path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
logger.warning("Corrupt channel runtime config store at %s, starting fresh", self._path)
|
||||
return {}
|
||||
if isinstance(raw, dict):
|
||||
return {str(name): dict(value) for name, value in raw.items() if isinstance(value, dict)}
|
||||
return {}
|
||||
|
||||
def _save(self) -> None:
|
||||
fd = tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
dir=self._path.parent,
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
)
|
||||
try:
|
||||
try:
|
||||
Path(fd.name).chmod(0o600)
|
||||
except OSError:
|
||||
logger.debug("Unable to chmod temporary channel runtime config store at %s", fd.name, exc_info=True)
|
||||
json.dump(self._data, fd, indent=2, ensure_ascii=False)
|
||||
fd.close()
|
||||
Path(fd.name).replace(self._path)
|
||||
try:
|
||||
self._path.chmod(0o600)
|
||||
except OSError:
|
||||
logger.debug("Unable to chmod channel runtime config store at %s", self._path, exc_info=True)
|
||||
except BaseException:
|
||||
fd.close()
|
||||
Path(fd.name).unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
def load_all(self) -> dict[str, dict[str, Any]]:
|
||||
with self._lock:
|
||||
return {name: dict(config) for name, config in self._data.items()}
|
||||
|
||||
def get_provider_config(self, provider: str) -> dict[str, Any] | None:
|
||||
with self._lock:
|
||||
config = self._data.get(provider)
|
||||
return dict(config) if isinstance(config, dict) else None
|
||||
|
||||
def set_provider_config(self, provider: str, config: dict[str, Any]) -> None:
|
||||
with self._lock:
|
||||
self._data[provider] = dict(config)
|
||||
self._save()
|
||||
|
||||
def set_provider_disconnected(self, provider: str) -> None:
|
||||
with self._lock:
|
||||
self._data[provider] = {
|
||||
"enabled": False,
|
||||
RUNTIME_CHANNEL_DISABLED_FLAG: True,
|
||||
}
|
||||
self._save()
|
||||
|
||||
def remove_provider_config(self, provider: str) -> bool:
|
||||
with self._lock:
|
||||
if provider not in self._data:
|
||||
return False
|
||||
del self._data[provider]
|
||||
self._save()
|
||||
return True
|
||||
|
||||
|
||||
def _provider_enabled(channel_connections_config: Any, provider: str) -> bool:
|
||||
provider_config = getattr(channel_connections_config, provider, None)
|
||||
return bool(getattr(provider_config, "enabled", False))
|
||||
|
||||
|
||||
def _runtime_channel_disconnected(runtime_config: dict[str, Any]) -> bool:
|
||||
return runtime_config.get(RUNTIME_CHANNEL_DISABLED_FLAG) is True and runtime_config.get("enabled") is False
|
||||
|
||||
|
||||
def merge_runtime_channel_configs(
|
||||
channels_config: dict[str, Any],
|
||||
channel_connections_config: Any,
|
||||
*,
|
||||
store: ChannelRuntimeConfigStore | None = None,
|
||||
) -> None:
|
||||
"""Merge persisted runtime provider config into ``channels_config`` in-place."""
|
||||
if channel_connections_config is None or not getattr(channel_connections_config, "enabled", False):
|
||||
return
|
||||
|
||||
runtime_store = store or ChannelRuntimeConfigStore()
|
||||
for provider, runtime_config in runtime_store.load_all().items():
|
||||
if not _provider_enabled(channel_connections_config, provider):
|
||||
continue
|
||||
if _runtime_channel_disconnected(runtime_config):
|
||||
channels_config.pop(provider, None)
|
||||
continue
|
||||
existing = channels_config.get(provider)
|
||||
merged = dict(existing) if isinstance(existing, dict) else {}
|
||||
merged.update(runtime_config)
|
||||
channels_config[provider] = merged
|
||||
|
||||
|
||||
def apply_runtime_connection_config(
|
||||
channel_connections_config: Any,
|
||||
*,
|
||||
store: ChannelRuntimeConfigStore | None = None,
|
||||
) -> Any:
|
||||
"""Apply persisted connection metadata that lives outside ``channels``.
|
||||
|
||||
Telegram uses a bot username for deep links; UI-entered values are stored
|
||||
with the runtime channel config so local restarts keep the provider
|
||||
configured.
|
||||
"""
|
||||
if channel_connections_config is None or not getattr(channel_connections_config, "enabled", False):
|
||||
return channel_connections_config
|
||||
|
||||
runtime_store = store or ChannelRuntimeConfigStore()
|
||||
telegram_runtime_config = runtime_store.get_provider_config("telegram")
|
||||
bot_username = ""
|
||||
if isinstance(telegram_runtime_config, dict):
|
||||
bot_username = str(telegram_runtime_config.get("bot_username") or "").strip()
|
||||
if not bot_username or not _provider_enabled(channel_connections_config, "telegram"):
|
||||
return channel_connections_config
|
||||
|
||||
config = channel_connections_config.model_copy(deep=True)
|
||||
config.telegram.bot_username = bot_username
|
||||
return config
|
||||
@@ -0,0 +1,427 @@
|
||||
"""ChannelService — manages the lifecycle of all IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.manager import DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.runtime_config_store import merge_runtime_channel_configs
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.channel_connections_config import ChannelConnectionsConfig
|
||||
|
||||
# Channel name → import path for lazy loading
|
||||
_CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"dingtalk": "app.channels.dingtalk:DingTalkChannel",
|
||||
"discord": "app.channels.discord:DiscordChannel",
|
||||
"feishu": "app.channels.feishu:FeishuChannel",
|
||||
"github": "app.channels.github:GitHubChannel",
|
||||
"slack": "app.channels.slack:SlackChannel",
|
||||
"telegram": "app.channels.telegram:TelegramChannel",
|
||||
"wechat": "app.channels.wechat:WechatChannel",
|
||||
"wecom": "app.channels.wecom:WeComChannel",
|
||||
}
|
||||
|
||||
# Keys that indicate a user has configured credentials for a channel.
|
||||
_CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = {
|
||||
"dingtalk": ["client_id", "client_secret"],
|
||||
"discord": ["bot_token"],
|
||||
"feishu": ["app_id", "app_secret"],
|
||||
"slack": ["bot_token", "app_token"],
|
||||
"telegram": ["bot_token"],
|
||||
"wecom": ["bot_id", "bot_secret"],
|
||||
"wechat": ["bot_token"],
|
||||
}
|
||||
|
||||
_CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
|
||||
_CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
|
||||
|
||||
|
||||
def _channel_has_credentials(name: str, channel_config: dict[str, Any]) -> bool:
|
||||
cred_keys = _CHANNEL_CREDENTIAL_KEYS.get(name, [])
|
||||
return any(not isinstance(channel_config.get(key), bool) and channel_config.get(key) is not None and str(channel_config[key]).strip() for key in cred_keys)
|
||||
|
||||
|
||||
def _resolve_service_url(config: dict[str, Any], config_key: str, env_key: str, default: str) -> str:
|
||||
value = config.pop(config_key, None)
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value
|
||||
env_value = os.getenv(env_key, "").strip()
|
||||
if env_value:
|
||||
return env_value
|
||||
return default
|
||||
|
||||
|
||||
def _merge_channel_connection_runtime_config(channels_config: dict[str, Any], app_config: AppConfig) -> None:
|
||||
connection_config = getattr(app_config, "channel_connections", None)
|
||||
merge_runtime_channel_configs(channels_config, connection_config)
|
||||
|
||||
|
||||
def _make_connection_repo(connection_config: ChannelConnectionsConfig | None):
|
||||
if connection_config is None or not getattr(connection_config, "enabled", False):
|
||||
return None
|
||||
|
||||
try:
|
||||
from deerflow.persistence.channel_connections import ChannelConnectionRepository
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
except Exception:
|
||||
logger.exception("Failed to import channel connection repository")
|
||||
return None
|
||||
|
||||
session_factory = get_session_factory()
|
||||
if session_factory is None:
|
||||
logger.warning("Channel connections are enabled but database persistence is not available")
|
||||
return None
|
||||
return ChannelConnectionRepository(session_factory)
|
||||
|
||||
|
||||
class ChannelService:
|
||||
"""Manages the lifecycle of all configured IM channels.
|
||||
|
||||
Reads configuration from ``config.yaml`` under the ``channels`` key,
|
||||
instantiates enabled channels, and starts the ChannelManager dispatcher.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
channels_config: dict[str, Any] | None = None,
|
||||
*,
|
||||
connection_repo: Any | None = None,
|
||||
require_bound_identity: bool = False,
|
||||
) -> None:
|
||||
self.bus = MessageBus()
|
||||
self.store = ChannelStore()
|
||||
self._connection_repo = connection_repo
|
||||
config = dict(channels_config or {})
|
||||
langgraph_url = _resolve_service_url(config, "langgraph_url", _CHANNELS_LANGGRAPH_URL_ENV, DEFAULT_LANGGRAPH_URL)
|
||||
gateway_url = _resolve_service_url(config, "gateway_url", _CHANNELS_GATEWAY_URL_ENV, DEFAULT_GATEWAY_URL)
|
||||
default_session = config.pop("session", None)
|
||||
channel_sessions = {name: channel_config.get("session") for name, channel_config in config.items() if isinstance(channel_config, dict)}
|
||||
self.manager = ChannelManager(
|
||||
bus=self.bus,
|
||||
store=self.store,
|
||||
langgraph_url=langgraph_url,
|
||||
gateway_url=gateway_url,
|
||||
default_session=default_session if isinstance(default_session, dict) else None,
|
||||
channel_sessions=channel_sessions,
|
||||
connection_repo=connection_repo,
|
||||
require_bound_identity=require_bound_identity,
|
||||
)
|
||||
self._channels: dict[str, Any] = {} # name -> Channel instance
|
||||
self._config = config
|
||||
self._running = False
|
||||
self._readiness_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
@classmethod
|
||||
def from_app_config(cls, app_config: AppConfig | None = None) -> ChannelService:
|
||||
"""Create a ChannelService from the application config."""
|
||||
if app_config is None:
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
app_config = get_app_config()
|
||||
channels_config = {}
|
||||
# extra fields are allowed by AppConfig (extra="allow")
|
||||
extra = app_config.model_extra or {}
|
||||
if "channels" in extra:
|
||||
channels_config = dict(extra["channels"] or {})
|
||||
_merge_channel_connection_runtime_config(channels_config, app_config)
|
||||
connection_config = getattr(app_config, "channel_connections", None)
|
||||
connections_enabled = connection_config is not None and getattr(connection_config, "enabled", False)
|
||||
require_bound_identity = bool(connections_enabled and getattr(connection_config, "require_bound_identity", True))
|
||||
return cls(
|
||||
channels_config=channels_config,
|
||||
connection_repo=_make_connection_repo(connection_config),
|
||||
require_bound_identity=require_bound_identity,
|
||||
)
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the manager and all enabled channels."""
|
||||
if self._running:
|
||||
return
|
||||
|
||||
await self.manager.start()
|
||||
self._running = True
|
||||
|
||||
ready_status = await self.ensure_ready_channels(attempts=2)
|
||||
ready_count = sum(1 for ready in ready_status.values() if ready)
|
||||
logger.info("ChannelService started with %d/%d ready channels", ready_count, len(ready_status))
|
||||
|
||||
async def ensure_ready_channels(self, *, attempts: int = 1) -> dict[str, bool]:
|
||||
"""Start or restart enabled configured channels that are not ready."""
|
||||
ready_status: dict[str, bool] = {}
|
||||
for name, channel_config in self._config.items():
|
||||
if not isinstance(channel_config, dict):
|
||||
continue
|
||||
if not channel_config.get("enabled", False):
|
||||
if _channel_has_credentials(name, channel_config):
|
||||
logger.warning(
|
||||
"A configured channel has credentials configured but is disabled. Set enabled: true under its channels entry in config.yaml to activate it.",
|
||||
)
|
||||
else:
|
||||
logger.info("A configured channel is disabled, skipping")
|
||||
continue
|
||||
|
||||
ready_status[name] = await self.ensure_channel_ready(name, attempts=attempts)
|
||||
return ready_status
|
||||
|
||||
async def ensure_channel_ready(
|
||||
self,
|
||||
name: str,
|
||||
config: dict[str, Any] | None = None,
|
||||
*,
|
||||
attempts: int = 1,
|
||||
) -> bool:
|
||||
"""Ensure a single enabled channel is running using its current config."""
|
||||
if not self._running:
|
||||
logger.warning("ChannelService is not running; cannot ensure channel readiness")
|
||||
return False
|
||||
|
||||
if config is not None:
|
||||
self._config[name] = dict(config)
|
||||
|
||||
# Serialize per channel: readiness is polled from request handlers, so
|
||||
# concurrent calls must not stop/start the same channel worker twice.
|
||||
lock = self._readiness_locks.setdefault(name, asyncio.Lock())
|
||||
async with lock:
|
||||
channel_config = self._config.get(name)
|
||||
if not channel_config or not isinstance(channel_config, dict):
|
||||
logger.warning("No config for requested channel")
|
||||
return False
|
||||
if not channel_config.get("enabled", False):
|
||||
return False
|
||||
|
||||
channel = self._channels.get(name)
|
||||
if channel is not None and channel.is_running:
|
||||
return True
|
||||
|
||||
if channel is not None:
|
||||
try:
|
||||
await channel.stop()
|
||||
except Exception:
|
||||
logger.exception("Error stopping non-running channel before readiness retry")
|
||||
self._channels.pop(name, None)
|
||||
|
||||
max_attempts = max(1, attempts)
|
||||
for attempt in range(max_attempts):
|
||||
if attempt > 0:
|
||||
logger.info("Retrying channel startup after readiness check")
|
||||
if await self._start_channel(name, channel_config):
|
||||
return True
|
||||
return False
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop all channels and the manager."""
|
||||
for name, channel in list(self._channels.items()):
|
||||
try:
|
||||
await channel.stop()
|
||||
logger.info("Channel stopped")
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel")
|
||||
self._channels.clear()
|
||||
|
||||
await self.manager.stop()
|
||||
self._running = False
|
||||
logger.info("ChannelService stopped")
|
||||
|
||||
def _load_channel_config(self, name: str) -> dict[str, Any] | None:
|
||||
"""Load the latest config for a specific channel from disk.
|
||||
|
||||
Uses ``get_app_config()`` which detects file changes via config
|
||||
signature, so edits to ``config.yaml`` are picked up without a process
|
||||
restart.
|
||||
The UI runtime-config overlay applied at startup is re-applied here
|
||||
so a file-driven reload neither drops credentials entered from the
|
||||
browser nor resurrects a channel disconnected from it.
|
||||
Falls back to the cached ``self._config`` when config loading fails.
|
||||
"""
|
||||
try:
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
app_config = get_app_config()
|
||||
extra = app_config.model_extra or {}
|
||||
channels_config = dict(extra.get("channels") or {})
|
||||
_merge_channel_connection_runtime_config(channels_config, app_config)
|
||||
channel_config = channels_config.get(name)
|
||||
if isinstance(channel_config, dict):
|
||||
# Update the cached config so get_status() stays consistent.
|
||||
self._config[name] = channel_config
|
||||
return channel_config
|
||||
except Exception:
|
||||
logger.exception("Failed to reload config for channel %s, using cached version", name)
|
||||
return self._config.get(name)
|
||||
|
||||
async def restart_channel(self, name: str, *, reload_config: bool = True) -> bool:
|
||||
"""Restart a specific channel. Returns True if successful."""
|
||||
if name in self._channels:
|
||||
try:
|
||||
await self._channels[name].stop()
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel for restart")
|
||||
del self._channels[name]
|
||||
|
||||
if reload_config:
|
||||
# Reading config.yaml and the runtime store is disk IO; keep it
|
||||
# off the event loop.
|
||||
config = await asyncio.to_thread(self._load_channel_config, name)
|
||||
else:
|
||||
config = self._config.get(name)
|
||||
if not config or not isinstance(config, dict):
|
||||
logger.warning("No config for requested channel")
|
||||
return False
|
||||
|
||||
if not config.get("enabled", False):
|
||||
logger.info("Channel %s is disabled, skipping restart", name)
|
||||
return True
|
||||
|
||||
return await self._start_channel(name, config)
|
||||
|
||||
async def configure_channel(self, name: str, config: dict[str, Any]) -> bool:
|
||||
"""Apply runtime config for a channel and restart it if the service is running."""
|
||||
self._config[name] = dict(config)
|
||||
if not self._running:
|
||||
return True
|
||||
# The caller just supplied the authoritative config (e.g. credentials
|
||||
# entered in the browser that are never written to config.yaml) — a
|
||||
# file reload here would clobber it with the stale on-disk entry.
|
||||
return await self.restart_channel(name, reload_config=False)
|
||||
|
||||
async def remove_channel(self, name: str) -> bool:
|
||||
"""Remove runtime config for a channel and stop it if currently running."""
|
||||
self._config.pop(name, None)
|
||||
channel = self._channels.pop(name, None)
|
||||
if channel is None:
|
||||
return True
|
||||
try:
|
||||
await channel.stop()
|
||||
logger.info("Channel stopped and removed")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel for removal")
|
||||
return False
|
||||
|
||||
async def _start_channel(self, name: str, config: dict[str, Any]) -> bool:
|
||||
"""Instantiate and start a single channel."""
|
||||
import_path = _CHANNEL_REGISTRY.get(name)
|
||||
if not import_path:
|
||||
logger.warning("Unknown channel type")
|
||||
return False
|
||||
|
||||
try:
|
||||
from deerflow.reflection import resolve_class
|
||||
|
||||
channel_cls = resolve_class(import_path, base_class=None)
|
||||
except Exception:
|
||||
logger.exception("Failed to import channel class")
|
||||
return False
|
||||
|
||||
try:
|
||||
config = dict(config)
|
||||
config["channel_store"] = self.store
|
||||
if self._connection_repo is not None:
|
||||
config["connection_repo"] = self._connection_repo
|
||||
channel = channel_cls(bus=self.bus, config=config)
|
||||
self._channels[name] = channel
|
||||
await channel.start()
|
||||
if not channel.is_running:
|
||||
self._channels.pop(name, None)
|
||||
logger.error("Channel did not enter a running state after start()")
|
||||
return False
|
||||
logger.info("Channel started")
|
||||
return True
|
||||
except Exception:
|
||||
self._channels.pop(name, None)
|
||||
logger.exception("Failed to start channel")
|
||||
return False
|
||||
|
||||
def get_status(self) -> dict[str, Any]:
|
||||
"""Return status information for all channels."""
|
||||
channels_status = {}
|
||||
for name in _CHANNEL_REGISTRY:
|
||||
config = self._config.get(name, {})
|
||||
enabled = isinstance(config, dict) and config.get("enabled", False)
|
||||
running = name in self._channels and self._channels[name].is_running
|
||||
channels_status[name] = {
|
||||
"enabled": enabled,
|
||||
"running": running,
|
||||
}
|
||||
return {
|
||||
"service_running": self._running,
|
||||
"channels": channels_status,
|
||||
}
|
||||
|
||||
def get_channel(self, name: str) -> Channel | None:
|
||||
"""Return a running channel instance by name when available."""
|
||||
return self._channels.get(name)
|
||||
|
||||
def is_channel_enabled(self, name: str) -> bool:
|
||||
"""Return whether ``channels.<name>.enabled`` is truthy in the live config.
|
||||
|
||||
Tracks the runtime-authoritative ``_config`` dict, which
|
||||
:meth:`configure_channel` updates when the UI flips the
|
||||
enabled flag — so callers that read this between requests get
|
||||
the current effective setting without re-reading config.yaml.
|
||||
Used by the GitHub webhook router as a fan-out kill-switch:
|
||||
``channels.github.enabled: false`` skips dispatch even though
|
||||
the webhook route itself remains mounted (which is governed by
|
||||
``GITHUB_WEBHOOK_SECRET``, not this flag).
|
||||
"""
|
||||
config = self._config.get(name)
|
||||
if not isinstance(config, dict):
|
||||
return False
|
||||
return bool(config.get("enabled", False))
|
||||
|
||||
def get_channel_config(self, name: str) -> dict[str, Any] | None:
|
||||
"""Return a shallow copy of the live ``channels.<name>`` block, or None.
|
||||
|
||||
Mirrors :meth:`is_channel_enabled` in tracking the runtime-
|
||||
authoritative ``_config`` dict, so callers see the same effective
|
||||
configuration the manager sees — including any updates pushed via
|
||||
:meth:`configure_channel` from the UI. Returns ``None`` when no
|
||||
config exists for ``name`` (rather than an empty dict) so callers
|
||||
can distinguish "not configured" from "configured with defaults".
|
||||
The shallow copy keeps callers from accidentally mutating live
|
||||
config state.
|
||||
"""
|
||||
config = self._config.get(name)
|
||||
if not isinstance(config, dict):
|
||||
return None
|
||||
return dict(config)
|
||||
|
||||
|
||||
# -- singleton access -------------------------------------------------------
|
||||
|
||||
_channel_service: ChannelService | None = None
|
||||
|
||||
|
||||
def get_channel_service() -> ChannelService | None:
|
||||
"""Get the singleton ChannelService instance (if started)."""
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def start_channel_service(app_config: AppConfig | None = None) -> ChannelService:
|
||||
"""Create and start the global ChannelService from app config."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
return _channel_service
|
||||
# from_app_config reads the JSON channel store and runtime config files;
|
||||
# keep that disk IO off the event loop.
|
||||
_channel_service = await asyncio.to_thread(ChannelService.from_app_config, app_config)
|
||||
await _channel_service.start()
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def stop_channel_service() -> None:
|
||||
"""Stop the global ChannelService."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
await _channel_service.stop()
|
||||
_channel_service = None
|
||||
@@ -0,0 +1,410 @@
|
||||
"""Slack channel — connects via Socket Mode (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from markdown_to_mrkdwn import SlackMarkdownConverter
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.connection_identity import attach_connection_identity
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_slack_md_converter = SlackMarkdownConverter()
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, list | tuple | set):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"Slack allowed_users should be a list of Slack user IDs or a single Slack user ID string; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(user_id) for user_id in values if str(user_id)}
|
||||
|
||||
|
||||
def _strip_leading_slack_bot_mention(text: str, bot_user_id: str | None) -> str:
|
||||
if not bot_user_id:
|
||||
return text
|
||||
if not text.startswith("<@"):
|
||||
return text
|
||||
end = text.find(">")
|
||||
if end <= 2:
|
||||
return text
|
||||
mentioned_user_id = text[2:end].split("|", 1)[0].lstrip("!")
|
||||
if mentioned_user_id != bot_user_id:
|
||||
return text
|
||||
return text[end + 1 :].lstrip()
|
||||
|
||||
|
||||
class SlackChannel(Channel):
|
||||
"""Slack IM channel using Socket Mode (WebSocket, no public IP).
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.slack``):
|
||||
- ``bot_token``: Slack Bot User OAuth Token (xoxb-...).
|
||||
- ``app_token``: Slack App-Level Token (xapp-...) for Socket Mode.
|
||||
- ``allowed_users``: (optional) List of allowed Slack user IDs, or a
|
||||
single Slack user ID string as shorthand. Empty = allow all. Other
|
||||
scalar values are treated as a single string with a warning.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="slack", bus=bus, config=config)
|
||||
self._socket_client = None
|
||||
self._web_client = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users = _normalize_allowed_users(config.get("allowed_users", []))
|
||||
self._web_client_factory = config.get("web_client_factory")
|
||||
self._connection_web_clients: dict[str, tuple[str, Any]] = {}
|
||||
configured_bot_user_id = config.get("bot_user_id")
|
||||
self._bot_user_id = str(configured_bot_user_id).lstrip("@") if configured_bot_user_id else None
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from slack_sdk import WebClient
|
||||
from slack_sdk.socket_mode import SocketModeClient
|
||||
from slack_sdk.socket_mode.response import SocketModeResponse
|
||||
except ImportError:
|
||||
logger.error("slack-sdk is not installed. Install it with: uv add slack-sdk")
|
||||
return
|
||||
|
||||
self._SocketModeResponse = SocketModeResponse
|
||||
if self._web_client_factory is None:
|
||||
self._web_client_factory = WebClient
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
app_token = self.config.get("app_token", "")
|
||||
|
||||
if self.config.get("event_delivery") == "http":
|
||||
logger.error("Slack HTTP Events mode is not supported by this channel adapter; use Socket Mode with app_token")
|
||||
return
|
||||
|
||||
if not bot_token or not app_token:
|
||||
logger.error("Slack channel requires bot_token and app_token")
|
||||
return
|
||||
|
||||
await self._initialize_operator_web_client(str(bot_token))
|
||||
self._socket_client = SocketModeClient(
|
||||
app_token=app_token,
|
||||
web_client=self._web_client,
|
||||
)
|
||||
self._loop = asyncio.get_event_loop()
|
||||
|
||||
self._socket_client.socket_mode_request_listeners.append(self._on_socket_event)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Start socket mode in background thread
|
||||
asyncio.get_event_loop().run_in_executor(None, self._socket_client.connect)
|
||||
logger.info("Slack channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._socket_client:
|
||||
self._socket_client.close()
|
||||
self._socket_client = None
|
||||
logger.info("Slack channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
web_client = await self._get_web_client_for_message(msg)
|
||||
if not web_client:
|
||||
return
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"text": _slack_md_converter.convert(msg.text),
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
async def post_message() -> None:
|
||||
await asyncio.to_thread(web_client.chat_postMessage, **kwargs)
|
||||
# Add a completion reaction to the thread root
|
||||
if msg.thread_ts:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction_with_client,
|
||||
web_client,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"white_check_mark",
|
||||
)
|
||||
|
||||
try:
|
||||
await self._send_with_retry(
|
||||
post_message,
|
||||
max_retries=_max_retries,
|
||||
log_prefix="[Slack]",
|
||||
)
|
||||
except Exception:
|
||||
# Add failure reaction on error
|
||||
if msg.thread_ts:
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction_with_client,
|
||||
web_client,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"x",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
web_client = await self._get_web_client_for_message(msg)
|
||||
if not web_client:
|
||||
return False
|
||||
|
||||
try:
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"file": str(attachment.actual_path),
|
||||
"filename": attachment.filename,
|
||||
"title": attachment.filename,
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
await asyncio.to_thread(web_client.files_upload_v2, **kwargs)
|
||||
logger.info("[Slack] file uploaded: %s to channel=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
async def _initialize_operator_web_client(self, bot_token: str) -> None:
|
||||
self._web_client = self._web_client_factory(token=bot_token)
|
||||
if self._bot_user_id is not None:
|
||||
return
|
||||
try:
|
||||
auth_info = await asyncio.to_thread(self._web_client.auth_test)
|
||||
user_id = auth_info.get("user_id") if isinstance(auth_info, dict) else None
|
||||
if user_id is None:
|
||||
auth_get = getattr(auth_info, "get", None)
|
||||
user_id = auth_get("user_id") if callable(auth_get) else None
|
||||
if isinstance(user_id, str) and user_id:
|
||||
self._bot_user_id = user_id
|
||||
except Exception:
|
||||
logger.warning("[Slack] failed to resolve bot user id; app mention text may include the bot mention", exc_info=True)
|
||||
|
||||
async def _get_web_client_for_message(self, msg: OutboundMessage):
|
||||
if msg.connection_id and self._connection_repo is not None:
|
||||
credentials = await self._connection_repo.get_credentials(msg.connection_id)
|
||||
access_token = credentials.get("access_token") if credentials else None
|
||||
if not access_token:
|
||||
return self._web_client
|
||||
# WebClient keeps its own HTTP session and rate-limit state, so
|
||||
# reuse one per connection until its token changes.
|
||||
cached = self._connection_web_clients.get(msg.connection_id)
|
||||
if cached is not None and cached[0] == access_token:
|
||||
return cached[1]
|
||||
if self._web_client_factory is None:
|
||||
from slack_sdk import WebClient
|
||||
|
||||
self._web_client_factory = WebClient
|
||||
web_client = self._web_client_factory(token=access_token)
|
||||
self._connection_web_clients[msg.connection_id] = (access_token, web_client)
|
||||
return web_client
|
||||
return self._web_client
|
||||
|
||||
@staticmethod
|
||||
def _add_reaction_with_client(web_client, channel_id: str, timestamp: str, emoji: str) -> None:
|
||||
try:
|
||||
web_client.reactions_add(
|
||||
channel=channel_id,
|
||||
timestamp=timestamp,
|
||||
name=emoji,
|
||||
)
|
||||
except Exception as exc:
|
||||
if "already_reacted" not in str(exc):
|
||||
logger.warning("[Slack] failed to add reaction %s: %s", emoji, exc)
|
||||
|
||||
def _add_reaction(self, channel_id: str, timestamp: str, emoji: str) -> None:
|
||||
"""Add an emoji reaction to a message (best-effort, non-blocking)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
self._add_reaction_with_client(self._web_client, channel_id, timestamp, emoji)
|
||||
|
||||
def _send_running_reply(self, channel_id: str, thread_ts: str) -> None:
|
||||
"""Send a 'Working on it......' reply in the thread (called from SDK thread)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
self._web_client.chat_postMessage(
|
||||
channel=channel_id,
|
||||
text=":hourglass_flowing_sand: Working on it...",
|
||||
thread_ts=thread_ts,
|
||||
)
|
||||
logger.info("[Slack] 'Working on it...' reply sent in channel=%s, thread_ts=%s", channel_id, thread_ts)
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to send running reply in channel=%s", channel_id)
|
||||
|
||||
def _on_socket_event(self, client, req) -> None:
|
||||
"""Called by slack-sdk for each Socket Mode event."""
|
||||
try:
|
||||
# Acknowledge the event
|
||||
response = self._SocketModeResponse(envelope_id=req.envelope_id)
|
||||
client.send_socket_mode_response(response)
|
||||
|
||||
event_type = req.type
|
||||
if event_type != "events_api":
|
||||
return
|
||||
|
||||
if self._bot_user_id is None:
|
||||
authorization = next((item for item in req.payload.get("authorizations", []) if isinstance(item, dict)), None)
|
||||
user_id = authorization.get("user_id") if authorization else None
|
||||
if isinstance(user_id, str) and user_id:
|
||||
self._bot_user_id = user_id
|
||||
|
||||
event = req.payload.get("event", {})
|
||||
etype = event.get("type", "")
|
||||
|
||||
# Handle message events (DM or @mention)
|
||||
if etype in ("message", "app_mention"):
|
||||
self._handle_message_event(
|
||||
event,
|
||||
team_id=req.payload.get("team_id") or req.payload.get("team") or event.get("team"),
|
||||
)
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error processing Slack event")
|
||||
|
||||
def _handle_message_event(self, event: dict, *, team_id: str | None = None) -> None:
|
||||
# Ignore bot messages
|
||||
if event.get("bot_id") or event.get("subtype"):
|
||||
return
|
||||
|
||||
user_id = event.get("user", "")
|
||||
|
||||
text = event.get("text", "").strip()
|
||||
if event.get("type") == "app_mention":
|
||||
text = _strip_leading_slack_bot_mention(text, self._bot_user_id)
|
||||
if not text:
|
||||
return
|
||||
|
||||
connect_code = self._pending_connect_code(text)
|
||||
if connect_code:
|
||||
if self._loop and self._loop.is_running():
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self._bind_connection_from_connect_code(
|
||||
event=event,
|
||||
team_id=str(team_id or ""),
|
||||
code=connect_code,
|
||||
),
|
||||
self._loop,
|
||||
)
|
||||
return
|
||||
|
||||
# Check allowed users after connect-code handling so browser-initiated
|
||||
# binding can bootstrap a new external identity.
|
||||
if self._allowed_users and user_id not in self._allowed_users:
|
||||
logger.debug("Ignoring message from non-allowed user: %s", user_id)
|
||||
return
|
||||
|
||||
channel_id = event.get("channel", "")
|
||||
thread_ts = event.get("thread_ts") or event.get("ts", "")
|
||||
|
||||
if is_known_channel_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# topic_id: use thread_ts as the topic identifier.
|
||||
# For threaded messages, thread_ts is the root message ts (shared topic).
|
||||
# For non-threaded messages, thread_ts is the message's own ts (new topic).
|
||||
inbound = self._make_inbound(
|
||||
chat_id=channel_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
metadata={
|
||||
# team_id is already resolved (payload team_id/team, else event team) by the caller.
|
||||
"team_id": team_id,
|
||||
"message_id": event.get("ts"),
|
||||
"client_msg_id": event.get("client_msg_id"),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_ts
|
||||
|
||||
if self._loop and self._loop.is_running():
|
||||
# Acknowledge with an eyes reaction
|
||||
self._add_reaction(channel_id, event.get("ts", thread_ts), "eyes")
|
||||
# Send "running" reply first (fire-and-forget from SDK thread)
|
||||
self._send_running_reply(channel_id, thread_ts)
|
||||
if self._connection_repo is None:
|
||||
asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._loop)
|
||||
else:
|
||||
asyncio.run_coroutine_threadsafe(self._publish_inbound_with_connection(inbound, team_id=team_id), self._loop)
|
||||
|
||||
async def _publish_inbound_with_connection(self, inbound, *, team_id: str | None = None) -> None:
|
||||
inbound = await self._attach_connection_identity(inbound, team_id=team_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _attach_connection_identity(self, inbound, *, team_id: str | None = None):
|
||||
workspace_id = str(team_id or inbound.metadata.get("team_id") or "")
|
||||
return await attach_connection_identity(
|
||||
inbound,
|
||||
repo=self._connection_repo,
|
||||
provider="slack",
|
||||
workspace_id=workspace_id,
|
||||
)
|
||||
|
||||
async def _bind_connection_from_connect_code(self, *, event: dict, team_id: str, code: str) -> bool:
|
||||
if self._connection_repo is None or not code:
|
||||
return False
|
||||
|
||||
channel_id = str(event.get("channel") or "")
|
||||
thread_ts = str(event.get("thread_ts") or event.get("ts") or "")
|
||||
state = await self._connection_repo.consume_oauth_state(provider="slack", state=code)
|
||||
if state is None:
|
||||
await self._post_connection_reply(channel_id, "Slack connection code is invalid or expired.", thread_ts)
|
||||
return True
|
||||
|
||||
user_id = str(event.get("user") or "")
|
||||
if not user_id or not team_id:
|
||||
await self._post_connection_reply(channel_id, "Slack connection could not be completed from this message.", thread_ts)
|
||||
return True
|
||||
|
||||
await self._connection_repo.upsert_connection(
|
||||
owner_user_id=state["owner_user_id"],
|
||||
provider="slack",
|
||||
external_account_id=user_id,
|
||||
workspace_id=team_id,
|
||||
metadata={
|
||||
"team_id": team_id,
|
||||
"channel_id": channel_id,
|
||||
},
|
||||
status="connected",
|
||||
)
|
||||
await self._post_connection_reply(channel_id, "Slack connected to DeerFlow.", thread_ts)
|
||||
return True
|
||||
|
||||
async def _post_connection_reply(self, channel_id: str, text: str, thread_ts: str | None = None) -> None:
|
||||
if not self._web_client or not channel_id:
|
||||
return
|
||||
kwargs: dict[str, Any] = {"channel": channel_id, "text": text}
|
||||
if thread_ts:
|
||||
kwargs["thread_ts"] = thread_ts
|
||||
try:
|
||||
await asyncio.to_thread(self._web_client.chat_postMessage, **kwargs)
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to send connection reply in channel=%s", channel_id)
|
||||
@@ -0,0 +1,153 @@
|
||||
"""ChannelStore — persists IM chat-to-DeerFlow thread mappings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ChannelStore:
|
||||
"""JSON-file-backed store that maps IM conversations to DeerFlow threads.
|
||||
|
||||
Data layout (on disk)::
|
||||
|
||||
{
|
||||
"<channel_name>:<chat_id>": {
|
||||
"thread_id": "<uuid>",
|
||||
"user_id": "<platform_user>",
|
||||
"created_at": 1700000000.0,
|
||||
"updated_at": 1700000000.0
|
||||
},
|
||||
...
|
||||
}
|
||||
|
||||
The store is intentionally simple — a single JSON file that is atomically
|
||||
rewritten on every mutation. For production workloads with high concurrency,
|
||||
this can be swapped for a proper database backend.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
if path is None:
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
path = Path(get_paths().base_dir) / "channels" / "store.json"
|
||||
self._path = Path(path)
|
||||
self._path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._data: dict[str, dict[str, Any]] = self._load()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# -- persistence -------------------------------------------------------
|
||||
|
||||
def _load(self) -> dict[str, dict[str, Any]]:
|
||||
if self._path.exists():
|
||||
try:
|
||||
return json.loads(self._path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
logger.warning("Corrupt channel store at %s, starting fresh", self._path)
|
||||
return {}
|
||||
|
||||
def _save(self) -> None:
|
||||
fd = tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
dir=self._path.parent,
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
)
|
||||
try:
|
||||
json.dump(self._data, fd, indent=2)
|
||||
fd.close()
|
||||
Path(fd.name).replace(self._path)
|
||||
except BaseException:
|
||||
fd.close()
|
||||
Path(fd.name).unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
# -- key helpers -------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _key(channel_name: str, chat_id: str, topic_id: str | None = None) -> str:
|
||||
if topic_id:
|
||||
return f"{channel_name}:{chat_id}:{topic_id}"
|
||||
return f"{channel_name}:{chat_id}"
|
||||
|
||||
# -- public API --------------------------------------------------------
|
||||
|
||||
def get_thread_id(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> str | None:
|
||||
"""Look up the DeerFlow thread_id for a given IM conversation/topic."""
|
||||
entry = self._data.get(self._key(channel_name, chat_id, topic_id))
|
||||
return entry["thread_id"] if entry else None
|
||||
|
||||
def set_thread_id(
|
||||
self,
|
||||
channel_name: str,
|
||||
chat_id: str,
|
||||
thread_id: str,
|
||||
*,
|
||||
topic_id: str | None = None,
|
||||
user_id: str = "",
|
||||
) -> None:
|
||||
"""Create or update the mapping for an IM conversation/topic."""
|
||||
with self._lock:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
now = time.time()
|
||||
existing = self._data.get(key)
|
||||
self._data[key] = {
|
||||
"thread_id": thread_id,
|
||||
"user_id": user_id,
|
||||
"created_at": existing["created_at"] if existing else now,
|
||||
"updated_at": now,
|
||||
}
|
||||
self._save()
|
||||
|
||||
def remove(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> bool:
|
||||
"""Remove a mapping.
|
||||
|
||||
If ``topic_id`` is provided, only that specific conversation/topic mapping is removed.
|
||||
If ``topic_id`` is omitted, all mappings whose key starts with
|
||||
``"<channel_name>:<chat_id>"`` (including topic-specific ones) are removed.
|
||||
|
||||
Returns True if at least one mapping was removed.
|
||||
"""
|
||||
with self._lock:
|
||||
# Remove a specific conversation/topic mapping.
|
||||
if topic_id is not None:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
if key in self._data:
|
||||
del self._data[key]
|
||||
self._save()
|
||||
return True
|
||||
return False
|
||||
|
||||
# Remove all mappings for this channel/chat_id (base and any topic-specific keys).
|
||||
prefix = self._key(channel_name, chat_id)
|
||||
keys_to_delete = [k for k in self._data if k == prefix or k.startswith(prefix + ":")]
|
||||
if not keys_to_delete:
|
||||
return False
|
||||
|
||||
for k in keys_to_delete:
|
||||
del self._data[k]
|
||||
self._save()
|
||||
return True
|
||||
|
||||
def list_entries(self, channel_name: str | None = None) -> list[dict[str, Any]]:
|
||||
"""List all stored mappings, optionally filtered by channel."""
|
||||
results = []
|
||||
for key, entry in self._data.items():
|
||||
parts = key.split(":", 2)
|
||||
ch = parts[0]
|
||||
chat = parts[1] if len(parts) > 1 else ""
|
||||
topic = parts[2] if len(parts) > 2 else None
|
||||
if channel_name and ch != channel_name:
|
||||
continue
|
||||
item: dict[str, Any] = {"channel_name": ch, "chat_id": chat, **entry}
|
||||
if topic is not None:
|
||||
item["topic_id"] = topic
|
||||
results.append(item)
|
||||
return results
|
||||
@@ -0,0 +1,561 @@
|
||||
"""Telegram channel — connects via long-polling (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.connection_identity import attach_connection_identity
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TELEGRAM_MAX_MESSAGE_LENGTH = 4096
|
||||
STREAM_EDIT_MIN_INTERVAL_SECONDS = 1.0
|
||||
# Groups (negative chat_id) are capped at 20 messages/minute by Telegram,
|
||||
# so stream edits there must pace well below the private-chat 1 msg/s guideline.
|
||||
STREAM_EDIT_GROUP_MIN_INTERVAL_SECONDS = 3.0
|
||||
# Bound on tracked in-flight streamed messages; entries normally clear on the
|
||||
# final update, this only guards against leaks when a final never arrives.
|
||||
MAX_TRACKED_STREAM_MESSAGES = 256
|
||||
|
||||
# Indirection so tests can patch the clock without touching the global time module.
|
||||
_monotonic = time.monotonic
|
||||
|
||||
|
||||
class TelegramChannel(Channel):
|
||||
"""Telegram bot channel using long-polling.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.telegram``):
|
||||
- ``bot_token``: Telegram Bot API token (from @BotFather).
|
||||
- ``allowed_users``: (optional) List of allowed Telegram user IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="telegram", bus=bus, config=config)
|
||||
self._application = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._tg_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users: set[int] = set()
|
||||
for uid in config.get("allowed_users", []):
|
||||
try:
|
||||
self._allowed_users.add(int(uid))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
# chat_id -> last sent message_id for threaded replies
|
||||
self._last_bot_message: dict[str, int] = {}
|
||||
# stream_key ("chat_id:thread_ts") -> state of the in-flight streamed
|
||||
# bot message being edited in place: {"message_id", "last_edit_at", "last_text"}
|
||||
self._stream_messages: dict[str, dict[str, Any]] = {}
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from telegram.ext import ApplicationBuilder, CommandHandler, MessageHandler, filters
|
||||
except ImportError:
|
||||
logger.error("python-telegram-bot is not installed. Install it with: uv add python-telegram-bot")
|
||||
return
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
if not bot_token:
|
||||
logger.error("Telegram channel requires bot_token")
|
||||
return
|
||||
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Build the application
|
||||
app = ApplicationBuilder().token(bot_token).build()
|
||||
|
||||
# Command handlers
|
||||
app.add_handler(CommandHandler("start", self._cmd_start))
|
||||
app.add_handler(CommandHandler("bootstrap", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("new", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("status", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("models", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("memory", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("goal", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("help", self._cmd_generic))
|
||||
|
||||
# Slash skill commands are dynamic and cannot all be pre-registered
|
||||
# with Telegram, so route unknown slash commands through chat handling.
|
||||
app.add_handler(MessageHandler(filters.TEXT & filters.COMMAND, self._on_text))
|
||||
|
||||
# General message handler
|
||||
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
|
||||
|
||||
self._application = app
|
||||
|
||||
# Run polling in a dedicated thread with its own event loop
|
||||
self._thread = threading.Thread(target=self._run_polling, daemon=True)
|
||||
self._thread.start()
|
||||
logger.info("Telegram channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._tg_loop and self._tg_loop.is_running():
|
||||
self._tg_loop.call_soon_threadsafe(self._tg_loop.stop)
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
self._application = None
|
||||
logger.info("Telegram channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._application:
|
||||
return
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("Invalid Telegram chat_id: %s", msg.chat_id)
|
||||
return
|
||||
|
||||
key = self._stream_key(msg.chat_id, msg.thread_ts)
|
||||
|
||||
if not msg.is_final:
|
||||
await self._send_stream_update(chat_id, key, msg.text, reply_to=self._parse_message_id(msg.thread_ts))
|
||||
return
|
||||
|
||||
state = self._stream_messages.pop(key, None)
|
||||
if state is not None:
|
||||
await self._finalize_stream_message(chat_id, msg.chat_id, state, msg.text)
|
||||
return
|
||||
|
||||
await self._send_new_message(chat_id, msg.chat_id, msg.text, _max_retries=_max_retries)
|
||||
|
||||
async def _send_stream_update(self, chat_id: int, key: str, text: str, reply_to: int | None = None) -> None:
|
||||
"""Edit the in-flight streamed message with accumulated text.
|
||||
|
||||
Updates are best-effort: throttled, rate-limit drops are silent. The
|
||||
manager always publishes a final message afterwards, which guarantees
|
||||
delivery of the complete text.
|
||||
"""
|
||||
if not text:
|
||||
return
|
||||
|
||||
display = text
|
||||
if len(display) > TELEGRAM_MAX_MESSAGE_LENGTH:
|
||||
display = display[: TELEGRAM_MAX_MESSAGE_LENGTH - 1] + "…"
|
||||
|
||||
bot = self._application.bot
|
||||
state = self._stream_messages.get(key)
|
||||
|
||||
send_kwargs: dict[str, Any] = {"chat_id": chat_id, "text": display}
|
||||
if reply_to:
|
||||
send_kwargs["reply_to_message_id"] = reply_to
|
||||
|
||||
if state is None:
|
||||
try:
|
||||
sent = await bot.send_message(**send_kwargs)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to start stream message in chat=%s", chat_id)
|
||||
return
|
||||
self._register_stream_message(key, message_id=sent.message_id, last_text=display, last_edit_at=_monotonic())
|
||||
return
|
||||
|
||||
now = _monotonic()
|
||||
min_interval = STREAM_EDIT_GROUP_MIN_INTERVAL_SECONDS if chat_id < 0 else STREAM_EDIT_MIN_INTERVAL_SECONDS
|
||||
if now - state["last_edit_at"] < min_interval:
|
||||
return
|
||||
if display == state["last_text"]:
|
||||
return
|
||||
|
||||
try:
|
||||
await bot.edit_message_text(chat_id=chat_id, message_id=state["message_id"], text=display)
|
||||
except Exception as exc:
|
||||
if self._is_not_modified(exc):
|
||||
state["last_text"] = display
|
||||
return
|
||||
if self._is_retry_after(exc):
|
||||
logger.debug("[Telegram] stream edit rate-limited in chat=%s, dropping update", chat_id)
|
||||
return
|
||||
logger.warning("[Telegram] stream edit failed in chat=%s, sending new message: %s", chat_id, exc)
|
||||
try:
|
||||
sent = await bot.send_message(**send_kwargs)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send fallback stream message in chat=%s", chat_id)
|
||||
return
|
||||
state["message_id"] = sent.message_id
|
||||
|
||||
state["last_edit_at"] = _monotonic()
|
||||
state["last_text"] = display
|
||||
|
||||
async def _finalize_stream_message(self, chat_id: int, chat_key: str, state: dict[str, Any], text: str) -> None:
|
||||
"""Apply the final text: edit the streamed message, splitting overflow into follow-ups."""
|
||||
bot = self._application.bot
|
||||
chunks = self._split_message(text or "")
|
||||
|
||||
edited = True
|
||||
if chunks[0] != state["last_text"]:
|
||||
edited = await self._edit_final_chunk(bot, chat_id, state["message_id"], chunks[0])
|
||||
|
||||
if edited:
|
||||
self._last_bot_message[chat_key] = state["message_id"]
|
||||
else:
|
||||
# Edit could not be applied (e.g. message deleted) — deliver the
|
||||
# first chunk as a fresh message with the standard retry policy.
|
||||
await self._send_new_message(chat_id, chat_key, chunks[0])
|
||||
|
||||
for chunk in chunks[1:]:
|
||||
await self._send_new_message(chat_id, chat_key, chunk)
|
||||
|
||||
async def _edit_final_chunk(self, bot, chat_id: int, message_id: int, text: str) -> bool:
|
||||
"""Edit with one rate-limit retry. Returns False if the edit could not be applied."""
|
||||
for attempt in range(2):
|
||||
try:
|
||||
await bot.edit_message_text(chat_id=chat_id, message_id=message_id, text=text)
|
||||
return True
|
||||
except Exception as exc:
|
||||
if self._is_not_modified(exc):
|
||||
return True
|
||||
if self._is_retry_after(exc) and attempt == 0:
|
||||
await asyncio.sleep(self._retry_after_seconds(exc))
|
||||
continue
|
||||
logger.warning("[Telegram] final edit failed in chat=%s: %s", chat_id, exc)
|
||||
return False
|
||||
return False
|
||||
|
||||
async def _send_new_message(self, chat_id: int, chat_key: str, text: str, *, _max_retries: int = 3) -> int | None:
|
||||
"""Send a fresh message with retry/backoff. Returns the sent message_id."""
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "text": text}
|
||||
|
||||
# Reply to the last bot message in this chat for threading
|
||||
reply_to = self._last_bot_message.get(chat_key)
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
|
||||
bot = self._application.bot
|
||||
|
||||
async def send_message() -> int:
|
||||
sent = await bot.send_message(**kwargs)
|
||||
self._last_bot_message[chat_key] = sent.message_id
|
||||
return sent.message_id
|
||||
|
||||
return await self._send_with_retry(
|
||||
send_message,
|
||||
max_retries=_max_retries,
|
||||
log_prefix="[Telegram]",
|
||||
)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._application:
|
||||
return False
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("[Telegram] Invalid chat_id: %s", msg.chat_id)
|
||||
return False
|
||||
|
||||
# Telegram limits: 10MB for photos, 50MB for documents
|
||||
if attachment.size > 50 * 1024 * 1024:
|
||||
logger.warning("[Telegram] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
bot = self._application.bot
|
||||
reply_to = self._last_bot_message.get(msg.chat_id)
|
||||
|
||||
try:
|
||||
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": f}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_photo(**kwargs)
|
||||
else:
|
||||
from telegram import InputFile
|
||||
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
input_file = InputFile(f, filename=attachment.filename)
|
||||
kwargs = {"chat_id": chat_id, "document": input_file}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_document(**kwargs)
|
||||
|
||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _stream_key(chat_id: str, thread_ts: str | None) -> str:
|
||||
return f"{chat_id}:{thread_ts or ''}"
|
||||
|
||||
@staticmethod
|
||||
def _parse_message_id(value: str | None) -> int | None:
|
||||
try:
|
||||
return int(value) if value else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
def _register_stream_message(self, key: str, *, message_id: int, last_text: str, last_edit_at: float) -> None:
|
||||
self._stream_messages.pop(key, None)
|
||||
while len(self._stream_messages) >= MAX_TRACKED_STREAM_MESSAGES:
|
||||
self._stream_messages.pop(next(iter(self._stream_messages)))
|
||||
self._stream_messages[key] = {
|
||||
"message_id": message_id,
|
||||
"last_edit_at": last_edit_at,
|
||||
"last_text": last_text,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _is_retry_after(exc: Exception) -> bool:
|
||||
return getattr(exc, "retry_after", None) is not None
|
||||
|
||||
@staticmethod
|
||||
def _retry_after_seconds(exc: Exception) -> float:
|
||||
value = getattr(exc, "retry_after", 0)
|
||||
if hasattr(value, "total_seconds"):
|
||||
return float(value.total_seconds())
|
||||
return float(value)
|
||||
|
||||
@staticmethod
|
||||
def _is_not_modified(exc: Exception) -> bool:
|
||||
return "message is not modified" in str(exc).lower()
|
||||
|
||||
@staticmethod
|
||||
def _split_message(text: str) -> list[str]:
|
||||
return [text[i : i + TELEGRAM_MAX_MESSAGE_LENGTH] for i in range(0, len(text), TELEGRAM_MAX_MESSAGE_LENGTH)] or [text]
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
|
||||
"""Send a 'Working on it...' reply and register it as the stream target."""
|
||||
if not self._application:
|
||||
return
|
||||
try:
|
||||
bot = self._application.bot
|
||||
sent = await bot.send_message(
|
||||
chat_id=int(chat_id),
|
||||
text="Working on it...",
|
||||
reply_to_message_id=reply_to_message_id,
|
||||
)
|
||||
self._register_stream_message(
|
||||
self._stream_key(chat_id, str(reply_to_message_id)),
|
||||
message_id=sent.message_id,
|
||||
last_text="Working on it...",
|
||||
last_edit_at=0.0,
|
||||
)
|
||||
logger.info("[Telegram] 'Working on it...' reply sent in chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send running reply in chat=%s", chat_id)
|
||||
|
||||
def _run_polling(self) -> None:
|
||||
"""Run telegram polling in a dedicated thread."""
|
||||
self._tg_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._tg_loop)
|
||||
try:
|
||||
# Cannot use run_polling() because it calls add_signal_handler(),
|
||||
# which only works in the main thread. Instead, manually
|
||||
# initialize the application and start the updater.
|
||||
self._tg_loop.run_until_complete(self._application.initialize())
|
||||
self._tg_loop.run_until_complete(self._application.start())
|
||||
self._tg_loop.run_until_complete(self._application.updater.start_polling())
|
||||
self._tg_loop.run_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Telegram polling error")
|
||||
finally:
|
||||
# Graceful shutdown
|
||||
try:
|
||||
if self._application.updater.running:
|
||||
self._tg_loop.run_until_complete(self._application.updater.stop())
|
||||
self._tg_loop.run_until_complete(self._application.stop())
|
||||
self._tg_loop.run_until_complete(self._application.shutdown())
|
||||
except Exception:
|
||||
logger.exception("Error during Telegram shutdown")
|
||||
|
||||
def _check_user(self, user_id: int) -> bool:
|
||||
if not self._allowed_users:
|
||||
return True
|
||||
return user_id in self._allowed_users
|
||||
|
||||
@staticmethod
|
||||
def _telegram_display_name(user) -> str:
|
||||
full_name = getattr(user, "full_name", None)
|
||||
if isinstance(full_name, str) and full_name:
|
||||
return full_name
|
||||
username = getattr(user, "username", None)
|
||||
if isinstance(username, str) and username:
|
||||
return username
|
||||
return str(getattr(user, "id", ""))
|
||||
|
||||
async def _bind_connection_from_start_token(self, update, state_token: str) -> bool:
|
||||
if self._connection_repo is None or not state_token:
|
||||
return False
|
||||
|
||||
state = await self._connection_repo.consume_oauth_state(provider="telegram", state=state_token)
|
||||
if state is None:
|
||||
await update.message.reply_text("Telegram connection link is invalid or expired.")
|
||||
return True
|
||||
|
||||
owner_user_id = state["owner_user_id"]
|
||||
user_id = str(update.effective_user.id)
|
||||
chat_id = str(update.effective_chat.id)
|
||||
connection = await self._connection_repo.upsert_connection(
|
||||
owner_user_id=owner_user_id,
|
||||
provider="telegram",
|
||||
external_account_id=user_id,
|
||||
external_account_name=self._telegram_display_name(update.effective_user),
|
||||
workspace_id=chat_id,
|
||||
workspace_name=None,
|
||||
metadata={
|
||||
"chat_id": chat_id,
|
||||
"chat_type": update.effective_chat.type,
|
||||
"telegram_username": getattr(update.effective_user, "username", None),
|
||||
},
|
||||
status="connected",
|
||||
)
|
||||
logger.info("[Telegram] bound chat=%s user=%s to DeerFlow user=%s connection=%s", chat_id, user_id, owner_user_id, connection["id"])
|
||||
await update.message.reply_text("Telegram connected to DeerFlow.")
|
||||
return True
|
||||
|
||||
async def _attach_connection_identity(self, inbound: InboundMessage) -> InboundMessage:
|
||||
return await attach_connection_identity(
|
||||
inbound,
|
||||
repo=self._connection_repo,
|
||||
provider="telegram",
|
||||
workspace_id=inbound.chat_id,
|
||||
)
|
||||
|
||||
def _get_bot_username(self, context) -> str | None:
|
||||
bot = getattr(context, "bot", None)
|
||||
username = getattr(bot, "username", None)
|
||||
if not username and self._application is not None:
|
||||
username = getattr(getattr(self._application, "bot", None), "username", None)
|
||||
return str(username) if username else None
|
||||
|
||||
@staticmethod
|
||||
def _strip_bot_username_from_leading_command(text: str, bot_username: str | None) -> str:
|
||||
username = (bot_username or "").lstrip("@").lower()
|
||||
if not username or not text.startswith("/"):
|
||||
return text
|
||||
|
||||
parts = text.split(maxsplit=1)
|
||||
command_token = parts[0]
|
||||
if "@" not in command_token:
|
||||
return text
|
||||
|
||||
command_name, addressed_username = command_token[1:].rsplit("@", 1)
|
||||
if not command_name or addressed_username.lower() != username:
|
||||
return text
|
||||
|
||||
normalized = f"/{command_name}"
|
||||
if len(parts) > 1:
|
||||
normalized = f"{normalized} {parts[1]}"
|
||||
return normalized
|
||||
|
||||
async def _cmd_start(self, update, context) -> None:
|
||||
"""Handle /start command."""
|
||||
args = getattr(context, "args", []) if context is not None else []
|
||||
if args:
|
||||
# Handle the deep-link bind token before applying allowed_users so a
|
||||
# browser-initiated bind can bootstrap a new external identity.
|
||||
handled = await self._bind_connection_from_start_token(update, str(args[0]))
|
||||
if handled:
|
||||
return
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
await update.message.reply_text("Welcome to DeerFlow! Send me a message to start a conversation.\nType /help for available commands.")
|
||||
|
||||
async def _process_incoming_with_reply(self, chat_id: str, msg_id: int, inbound: InboundMessage) -> None:
|
||||
await self._send_running_reply(chat_id, msg_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _cmd_generic(self, update, context) -> None:
|
||||
"""Forward slash commands to the channel manager."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = self._strip_bot_username_from_leading_command(update.message.text.strip(), self._get_bot_username(context))
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# Use the same topic_id logic as _on_text so that commands
|
||||
# like /new target the correct thread mapping.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.COMMAND,
|
||||
thread_ts=msg_id,
|
||||
metadata={"message_id": msg_id},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
inbound = await self._attach_connection_identity(inbound)
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
|
||||
async def _on_text(self, update, context) -> None:
|
||||
"""Handle regular text messages."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = self._strip_bot_username_from_leading_command(update.message.text.strip(), self._get_bot_username(context))
|
||||
if not text:
|
||||
return
|
||||
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# topic_id determines which DeerFlow thread the message maps to.
|
||||
# In private chats, use None so that all messages share a single
|
||||
# thread (the store key becomes "channel:chat_id").
|
||||
# In group chats, use the reply-to message id or the current
|
||||
# message id to keep separate conversation threads.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.CHAT,
|
||||
thread_ts=msg_id,
|
||||
metadata={"message_id": msg_id},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
inbound = await self._attach_connection_identity(inbound)
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,470 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any, cast
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.connection_identity import attach_connection_identity
|
||||
from app.channels.message_bus import (
|
||||
InboundMessage,
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
OutboundMessage,
|
||||
ResolvedAttachment,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WeComChannel(Channel):
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="wecom", bus=bus, config=config)
|
||||
self._bot_id: str | None = None
|
||||
self._bot_secret: str | None = None
|
||||
self._ws_client = None
|
||||
self._ws_task: asyncio.Task | None = None
|
||||
self._ws_frames: dict[str, dict[str, Any]] = {}
|
||||
self._ws_stream_ids: dict[str, str] = {}
|
||||
self._working_message = "Working on it..."
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
def _clear_ws_context(self, thread_ts: str | None) -> None:
|
||||
if not thread_ts:
|
||||
return
|
||||
self._ws_frames.pop(thread_ts, None)
|
||||
self._ws_stream_ids.pop(thread_ts, None)
|
||||
|
||||
async def _send_ws_upload_command(self, req_id: str, body: dict[str, Any], cmd: str) -> dict[str, Any]:
|
||||
if not self._ws_client:
|
||||
raise RuntimeError("WeCom WebSocket client is not available")
|
||||
|
||||
ws_manager = getattr(self._ws_client, "_ws_manager", None)
|
||||
send_reply = getattr(ws_manager, "send_reply", None)
|
||||
if not callable(send_reply):
|
||||
raise RuntimeError("Installed wecom-aibot-python-sdk does not expose the WebSocket media upload API expected by DeerFlow. Use wecom-aibot-python-sdk==0.1.6 or update the adapter.")
|
||||
|
||||
send_reply_async = cast(Callable[[str, dict[str, Any], str], Awaitable[dict[str, Any]]], send_reply)
|
||||
return await send_reply_async(req_id, body, cmd)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
bot_id = self.config.get("bot_id")
|
||||
bot_secret = self.config.get("bot_secret")
|
||||
working_message = self.config.get("working_message")
|
||||
|
||||
self._bot_id = bot_id if isinstance(bot_id, str) and bot_id else None
|
||||
self._bot_secret = bot_secret if isinstance(bot_secret, str) and bot_secret else None
|
||||
self._working_message = working_message if isinstance(working_message, str) and working_message else "Working on it..."
|
||||
|
||||
if not self._bot_id or not self._bot_secret:
|
||||
logger.error("WeCom channel requires bot_id and bot_secret")
|
||||
return
|
||||
|
||||
try:
|
||||
from aibot import WSClient, WSClientOptions
|
||||
except ImportError:
|
||||
logger.error("wecom-aibot-python-sdk is not installed. Install it with: uv add wecom-aibot-python-sdk")
|
||||
return
|
||||
else:
|
||||
self._ws_client = WSClient(WSClientOptions(bot_id=self._bot_id, secret=self._bot_secret, logger=logger))
|
||||
self._ws_client.on("message.text", self._on_ws_text)
|
||||
self._ws_client.on("message.mixed", self._on_ws_mixed)
|
||||
self._ws_client.on("message.image", self._on_ws_image)
|
||||
self._ws_client.on("message.file", self._on_ws_file)
|
||||
self._ws_client.on("error", self._on_ws_error)
|
||||
self._ws_client.on("disconnected", self._on_ws_disconnected)
|
||||
self._ws_task = asyncio.create_task(self._ws_client.connect())
|
||||
self._ws_task.add_done_callback(self._on_ws_task_done)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
logger.info("WeCom channel started")
|
||||
|
||||
def _on_ws_task_done(self, task: asyncio.Task) -> None:
|
||||
if task.cancelled():
|
||||
return
|
||||
exc = task.exception()
|
||||
if exc is None:
|
||||
return
|
||||
logger.error(
|
||||
"WeCom WebSocket connection task failed: %s. Check that the network/proxy allows wss://openws.work.weixin.qq.com and that bot_id/bot_secret are valid.",
|
||||
exc,
|
||||
)
|
||||
|
||||
def _on_ws_error(self, error: Any) -> None:
|
||||
logger.error("WeCom WebSocket error: %s", error)
|
||||
|
||||
def _on_ws_disconnected(self, *args: Any) -> None:
|
||||
detail = f" ({args[0]})" if args else ""
|
||||
logger.warning("WeCom WebSocket disconnected%s; SDK will attempt to reconnect", detail)
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._ws_task:
|
||||
try:
|
||||
self._ws_task.cancel()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_task = None
|
||||
if self._ws_client:
|
||||
try:
|
||||
self._ws_client.disconnect()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_client = None
|
||||
self._ws_frames.clear()
|
||||
self._ws_stream_ids.clear()
|
||||
logger.info("WeCom channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if self._ws_client:
|
||||
await self._send_ws(msg, _max_retries=_max_retries)
|
||||
return
|
||||
logger.warning("[WeCom] send called but WebSocket client is not available")
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
if msg.channel_name != self.name:
|
||||
return
|
||||
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
return
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not msg.is_final:
|
||||
return True
|
||||
if not self._ws_client:
|
||||
return False
|
||||
if not msg.thread_ts:
|
||||
return False
|
||||
frame = self._ws_frames.get(msg.thread_ts)
|
||||
if not frame:
|
||||
return False
|
||||
|
||||
media_type = "image" if attachment.is_image else "file"
|
||||
size_limit = 2 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
|
||||
if attachment.size > size_limit:
|
||||
logger.warning(
|
||||
"[WeCom] %s too large (%d bytes), skipping: %s",
|
||||
media_type,
|
||||
attachment.size,
|
||||
attachment.filename,
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media_ws(
|
||||
media_type=media_type,
|
||||
filename=attachment.filename,
|
||||
path=str(attachment.actual_path),
|
||||
size=attachment.size,
|
||||
)
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
body = {media_type: {"media_id": media_id}, "msgtype": media_type}
|
||||
await self._ws_client.reply(frame, body)
|
||||
logger.debug("[WeCom] %s sent via ws: %s", media_type, attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[WeCom] failed to upload/send file via ws: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _on_ws_text(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
text = ((body.get("text") or {}).get("content") or "").strip()
|
||||
quote = (((body.get("quote") or {}).get("text") or {}).get("content") or "").strip()
|
||||
if not text and not quote:
|
||||
return
|
||||
await self._publish_ws_inbound(frame, text + (f"\nQuote message: {quote}" if quote else ""))
|
||||
|
||||
async def _on_ws_mixed(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
mixed = body.get("mixed") or {}
|
||||
items = mixed.get("msg_item") or []
|
||||
parts: list[str] = []
|
||||
files: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
item_type = (item or {}).get("msgtype")
|
||||
if item_type == "text":
|
||||
content = (((item or {}).get("text") or {}).get("content") or "").strip()
|
||||
if content:
|
||||
parts.append(content)
|
||||
elif item_type in ("image", "file"):
|
||||
payload = (item or {}).get(item_type) or {}
|
||||
url = payload.get("url")
|
||||
aeskey = payload.get("aeskey")
|
||||
if isinstance(url, str) and url:
|
||||
files.append(
|
||||
{
|
||||
"type": item_type,
|
||||
"url": url,
|
||||
"aeskey": (aeskey if isinstance(aeskey, str) and aeskey else None),
|
||||
}
|
||||
)
|
||||
text = "\n\n".join(parts).strip()
|
||||
if not text and not files:
|
||||
return
|
||||
if not text:
|
||||
text = "（receive image/file）"
|
||||
await self._publish_ws_inbound(frame, text, files=files)
|
||||
|
||||
async def _on_ws_image(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
image = body.get("image") or {}
|
||||
url = image.get("url")
|
||||
aeskey = image.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive image ）",
|
||||
files=[
|
||||
{
|
||||
"type": "image",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _on_ws_file(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
file_obj = body.get("file") or {}
|
||||
url = file_obj.get("url")
|
||||
aeskey = file_obj.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive file）",
|
||||
files=[
|
||||
{
|
||||
"type": "file",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _publish_ws_inbound(
|
||||
self,
|
||||
frame: dict[str, Any],
|
||||
text: str,
|
||||
*,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return
|
||||
|
||||
body = frame.get("body", {}) or {}
|
||||
msg_id = body.get("msgid")
|
||||
if not msg_id:
|
||||
return
|
||||
|
||||
user_id = (body.get("from") or {}).get("userid")
|
||||
|
||||
connect_code = self._pending_connect_code(text)
|
||||
if connect_code:
|
||||
handled = await self._bind_connection_from_connect_code(
|
||||
frame=frame,
|
||||
user_id=str(user_id or ""),
|
||||
code=connect_code,
|
||||
)
|
||||
if handled:
|
||||
return
|
||||
|
||||
inbound_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=user_id, # keep user's conversation in memory
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=inbound_type,
|
||||
thread_ts=msg_id,
|
||||
files=files or [],
|
||||
metadata={
|
||||
"aibotid": body.get("aibotid"),
|
||||
"chattype": body.get("chattype"),
|
||||
"message_id": msg_id,
|
||||
},
|
||||
)
|
||||
inbound.topic_id = user_id # keep the same thread
|
||||
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_frames[msg_id] = frame
|
||||
self._ws_stream_ids[msg_id] = stream_id
|
||||
|
||||
try:
|
||||
await self._ws_client.reply_stream(frame, stream_id, self._working_message, False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
inbound = await self._attach_connection_identity(inbound)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _attach_connection_identity(self, inbound: InboundMessage) -> InboundMessage:
|
||||
return await attach_connection_identity(
|
||||
inbound,
|
||||
repo=self._connection_repo,
|
||||
provider="wecom",
|
||||
workspace_id=str(inbound.metadata.get("aibotid") or "") or None,
|
||||
fallback_without_workspace=True,
|
||||
)
|
||||
|
||||
async def _bind_connection_from_connect_code(self, *, frame: dict[str, Any], user_id: str, code: str) -> bool:
|
||||
if self._connection_repo is None or not code:
|
||||
return False
|
||||
|
||||
state = await self._connection_repo.consume_oauth_state(provider="wecom", state=code)
|
||||
if state is None:
|
||||
await self._send_connection_reply(frame, "WeCom connection code is invalid or expired.")
|
||||
return True
|
||||
|
||||
if not user_id:
|
||||
await self._send_connection_reply(frame, "WeCom connection could not be completed from this message.")
|
||||
return True
|
||||
|
||||
body = frame.get("body", {}) or {}
|
||||
workspace_id = str(body.get("aibotid") or "") or None
|
||||
await self._connection_repo.upsert_connection(
|
||||
owner_user_id=state["owner_user_id"],
|
||||
provider="wecom",
|
||||
external_account_id=user_id,
|
||||
workspace_id=workspace_id,
|
||||
metadata={
|
||||
"aibotid": workspace_id,
|
||||
"chattype": body.get("chattype"),
|
||||
},
|
||||
status="connected",
|
||||
)
|
||||
await self._send_connection_reply(frame, "WeCom connected to DeerFlow.")
|
||||
return True
|
||||
|
||||
async def _send_connection_reply(self, frame: dict[str, Any], text: str) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
await self._ws_client.reply(frame, {"msgtype": "text", "text": {"content": text}})
|
||||
|
||||
async def _send_ws(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
generate_req_id = None
|
||||
|
||||
if msg.thread_ts and msg.thread_ts in self._ws_frames:
|
||||
frame = self._ws_frames[msg.thread_ts]
|
||||
stream_id = self._ws_stream_ids.get(msg.thread_ts)
|
||||
if not stream_id and generate_req_id:
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_stream_ids[msg.thread_ts] = stream_id
|
||||
if not stream_id:
|
||||
return
|
||||
|
||||
await self._send_with_retry(
|
||||
lambda: self._ws_client.reply_stream(frame, stream_id, msg.text, bool(msg.is_final)),
|
||||
max_retries=_max_retries,
|
||||
log_prefix="[WeCom]",
|
||||
operation_name="stream send",
|
||||
)
|
||||
return
|
||||
|
||||
body = {"msgtype": "markdown", "markdown": {"content": msg.text}}
|
||||
await self._send_with_retry(
|
||||
lambda: self._ws_client.send_message(msg.chat_id, body),
|
||||
max_retries=_max_retries,
|
||||
log_prefix="[WeCom]",
|
||||
)
|
||||
|
||||
async def _upload_media_ws(
|
||||
self,
|
||||
*,
|
||||
media_type: str,
|
||||
filename: str,
|
||||
path: str,
|
||||
size: int,
|
||||
) -> str | None:
|
||||
if not self._ws_client:
|
||||
return None
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
chunk_size = 512 * 1024
|
||||
total_chunks = (size + chunk_size - 1) // chunk_size
|
||||
if total_chunks < 1 or total_chunks > 100:
|
||||
logger.warning("[WeCom] invalid total_chunks=%d for %s", total_chunks, filename)
|
||||
return None
|
||||
|
||||
md5_hasher = hashlib.md5()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
||||
md5_hasher.update(chunk)
|
||||
md5 = md5_hasher.hexdigest()
|
||||
|
||||
init_req_id = generate_req_id("aibot_upload_media_init")
|
||||
init_body = {
|
||||
"type": media_type,
|
||||
"filename": filename,
|
||||
"total_size": int(size),
|
||||
"total_chunks": int(total_chunks),
|
||||
"md5": md5,
|
||||
}
|
||||
init_ack = await self._send_ws_upload_command(init_req_id, init_body, "aibot_upload_media_init")
|
||||
upload_id = (init_ack.get("body") or {}).get("upload_id")
|
||||
if not upload_id:
|
||||
logger.warning("[WeCom] upload init returned no upload_id: %s", init_ack)
|
||||
return None
|
||||
|
||||
with open(path, "rb") as f:
|
||||
for idx in range(total_chunks):
|
||||
data = f.read(chunk_size)
|
||||
if not data:
|
||||
break
|
||||
chunk_req_id = generate_req_id("aibot_upload_media_chunk")
|
||||
chunk_body = {
|
||||
"upload_id": upload_id,
|
||||
"chunk_index": int(idx),
|
||||
"base64_data": base64.b64encode(data).decode("utf-8"),
|
||||
}
|
||||
await self._send_ws_upload_command(chunk_req_id, chunk_body, "aibot_upload_media_chunk")
|
||||
|
||||
finish_req_id = generate_req_id("aibot_upload_media_finish")
|
||||
finish_ack = await self._send_ws_upload_command(finish_req_id, {"upload_id": upload_id}, "aibot_upload_media_finish")
|
||||
media_id = (finish_ack.get("body") or {}).get("media_id")
|
||||
if not media_id:
|
||||
logger.warning("[WeCom] upload finish returned no media_id: %s", finish_ack)
|
||||
return None
|
||||
return media_id
|
||||
@@ -0,0 +1,12 @@
|
||||
from .config import GatewayConfig, get_gateway_config
|
||||
|
||||
__all__ = ["app", "create_app", "GatewayConfig", "get_gateway_config"]
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazily expose the FastAPI app without initializing it on package import."""
|
||||
if name in {"app", "create_app"}:
|
||||
from .app import app, create_app
|
||||
|
||||
return app if name == "app" else create_app
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
@@ -0,0 +1,518 @@
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.gateway.auth_disabled import warn_if_auth_disabled_enabled
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware, get_configured_cors_origins
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.routers import (
|
||||
agents,
|
||||
artifacts,
|
||||
assistants_compat,
|
||||
auth,
|
||||
channel_connections,
|
||||
channels,
|
||||
console,
|
||||
features,
|
||||
feedback,
|
||||
github_webhooks,
|
||||
input_polish,
|
||||
mcp,
|
||||
memory,
|
||||
models,
|
||||
runs,
|
||||
scheduled_tasks,
|
||||
skills,
|
||||
suggestions,
|
||||
thread_runs,
|
||||
threads,
|
||||
uploads,
|
||||
)
|
||||
from app.gateway.trace_middleware import TraceMiddleware, resolve_trace_enabled
|
||||
from deerflow.config import app_config as deerflow_app_config
|
||||
from deerflow.logging_config import DEFAULT_LOG_DATE_FORMAT, DEFAULT_LOG_FORMAT, configure_logging
|
||||
from deerflow.uploads.manager import cleanup_stale_upload_staging_files
|
||||
|
||||
AppConfig = deerflow_app_config.AppConfig
|
||||
get_app_config = deerflow_app_config.get_app_config
|
||||
|
||||
# Default logging; lifespan overrides from config.yaml log_level.
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format=DEFAULT_LOG_FORMAT,
|
||||
datefmt=DEFAULT_LOG_DATE_FORMAT,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Upper bound (seconds) each lifespan shutdown hook is allowed to run.
|
||||
# Bounds worker exit time so uvicorn's reload supervisor does not keep
|
||||
# firing signals into a worker that is stuck waiting for shutdown cleanup.
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
async def _ensure_admin_user(app: FastAPI) -> None:
|
||||
"""Startup hook: handle first boot and migrate orphan threads otherwise.
|
||||
|
||||
After admin creation, migrate orphan threads from the LangGraph
|
||||
store (metadata.user_id unset) to the admin account. This is the
|
||||
"no-auth → with-auth" upgrade path: users who ran DeerFlow without
|
||||
authentication have existing LangGraph thread data that needs an
|
||||
owner assigned.
|
||||
First boot (no admin exists):
|
||||
- Does NOT create any user accounts automatically.
|
||||
- The operator must visit ``/setup`` to create the first admin.
|
||||
|
||||
Subsequent boots (admin already exists):
|
||||
- Runs the one-time "no-auth → with-auth" orphan thread migration for
|
||||
existing LangGraph thread metadata that has no user_id.
|
||||
|
||||
No SQL persistence migration is needed: the four user_id columns
|
||||
(threads_meta, runs, run_events, feedback) only come into existence
|
||||
alongside the auth module via create_all, so freshly created tables
|
||||
never contain NULL-owner rows.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.gateway.deps import get_local_provider
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
except RuntimeError:
|
||||
# Auth persistence may not be initialized in some test/boot paths.
|
||||
# Skip admin migration work rather than failing gateway startup.
|
||||
logger.warning("Auth persistence not ready; skipping admin bootstrap check")
|
||||
return
|
||||
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
return
|
||||
|
||||
admin_count = await provider.count_admin_users()
|
||||
|
||||
if admin_count == 0:
|
||||
logger.info("=" * 60)
|
||||
logger.info(" First boot detected — no admin account exists.")
|
||||
logger.info(" Visit /setup to complete admin account creation.")
|
||||
logger.info("=" * 60)
|
||||
return
|
||||
|
||||
# Admin already exists — run orphan thread migration for any
|
||||
# LangGraph thread metadata that pre-dates the auth module.
|
||||
async with sf() as session:
|
||||
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
|
||||
if row is None:
|
||||
return # Should not happen (admin_count > 0 above), but be safe.
|
||||
|
||||
admin_id = str(row.id)
|
||||
|
||||
# LangGraph store orphan migration — non-fatal.
|
||||
# This covers the "no-auth → with-auth" upgrade path for users
|
||||
# whose existing LangGraph thread metadata has no user_id set.
|
||||
store = getattr(app.state, "store", None)
|
||||
if store is not None:
|
||||
try:
|
||||
migrated = await _migrate_orphaned_threads(store, admin_id)
|
||||
if migrated:
|
||||
logger.info("Migrated %d orphan LangGraph thread(s) to admin", migrated)
|
||||
except Exception:
|
||||
logger.exception("LangGraph thread migration failed (non-fatal)")
|
||||
|
||||
|
||||
async def _iter_store_items(store, namespace, *, page_size: int = 500):
|
||||
"""Paginated async iterator over a LangGraph store namespace.
|
||||
|
||||
Replaces the old hardcoded ``limit=1000`` call with a cursor-style
|
||||
loop so that environments with more than one page of orphans do
|
||||
not silently lose data. Terminates when a page is empty OR when a
|
||||
short page arrives (indicating the last page).
|
||||
"""
|
||||
offset = 0
|
||||
while True:
|
||||
batch = await store.asearch(namespace, limit=page_size, offset=offset)
|
||||
if not batch:
|
||||
return
|
||||
for item in batch:
|
||||
yield item
|
||||
if len(batch) < page_size:
|
||||
return
|
||||
offset += page_size
|
||||
|
||||
|
||||
async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
|
||||
"""Migrate LangGraph store threads with no user_id to the given admin.
|
||||
|
||||
Uses cursor pagination so all orphans are migrated regardless of
|
||||
count. Returns the number of rows migrated.
|
||||
"""
|
||||
migrated = 0
|
||||
async for item in _iter_store_items(store, ("threads",)):
|
||||
metadata = item.value.get("metadata", {})
|
||||
if not metadata.get("user_id"):
|
||||
metadata["user_id"] = admin_user_id
|
||||
item.value["metadata"] = metadata
|
||||
await store.aput(("threads",), item.key, item.value)
|
||||
migrated += 1
|
||||
return migrated
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Application lifespan handler."""
|
||||
|
||||
# Load config and check necessary environment variables at startup.
|
||||
# `startup_config` is a local snapshot used only for one-shot bootstrap
|
||||
# work (logging level, langgraph_runtime engines, channels). Request-time
|
||||
# config resolution always routes through `get_app_config()` in
|
||||
# `app/gateway/deps.py::get_config()` so `config.yaml` edits become
|
||||
# visible without a process restart. We deliberately do NOT cache this
|
||||
# snapshot on `app.state` to keep that contract enforceable.
|
||||
try:
|
||||
startup_config = get_app_config()
|
||||
configure_logging(startup_config)
|
||||
logger.info("Configuration loaded successfully")
|
||||
warn_if_auth_disabled_enabled()
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to load configuration during gateway startup: {e}"
|
||||
logger.exception(error_msg)
|
||||
raise RuntimeError(error_msg) from e
|
||||
config = get_gateway_config()
|
||||
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
|
||||
|
||||
# Pre-warm tiktoken encoding cache so the first memory-injection request
|
||||
# never blocks on the BPE data download (which hits an OpenAI/Azure URL
|
||||
# that may be unreachable in restricted networks — see issue #3402).
|
||||
# When memory.token_counting is "char", token counting never touches
|
||||
# tiktoken, so skip the warm-up entirely (avoids even the 5s probe in
|
||||
# network-restricted deployments — see issue #3429).
|
||||
if startup_config.memory.token_counting == "char":
|
||||
logger.info("memory.token_counting='char'; skipping tiktoken warm-up (network-free token estimation)")
|
||||
else:
|
||||
try:
|
||||
from deerflow.agents.memory.prompt import warm_tiktoken_cache
|
||||
|
||||
warmed = await asyncio.wait_for(
|
||||
asyncio.to_thread(warm_tiktoken_cache),
|
||||
timeout=5,
|
||||
)
|
||||
if warmed:
|
||||
logger.info("tiktoken encoding cache warmed successfully")
|
||||
else:
|
||||
logger.warning("tiktoken encoding cache warm-up failed; token counting will use character-based fallback until tiktoken loads successfully")
|
||||
except TimeoutError:
|
||||
logger.warning("tiktoken encoding cache warm-up timed out; token counting will use character-based fallback until tiktoken loads successfully")
|
||||
except Exception:
|
||||
logger.warning("tiktoken warm-up skipped", exc_info=True)
|
||||
|
||||
try:
|
||||
removed_upload_staging_files = await asyncio.to_thread(cleanup_stale_upload_staging_files)
|
||||
if removed_upload_staging_files:
|
||||
logger.info("Removed %d stale upload staging file(s)", removed_upload_staging_files)
|
||||
except Exception:
|
||||
logger.warning("Upload staging file cleanup skipped", exc_info=True)
|
||||
|
||||
# Initialize LangGraph runtime components (StreamBridge, RunManager, checkpointer, store)
|
||||
async with langgraph_runtime(app, startup_config):
|
||||
logger.info("LangGraph runtime initialised")
|
||||
|
||||
# Check admin bootstrap state and migrate orphan threads after admin exists.
|
||||
# Must run AFTER langgraph_runtime so app.state.store is available for thread migration
|
||||
await _ensure_admin_user(app)
|
||||
|
||||
# Start IM channel service if any channels are configured
|
||||
try:
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
channel_service = await start_channel_service(startup_config)
|
||||
logger.info("Channel service started: %s", channel_service.get_status())
|
||||
except Exception:
|
||||
logger.exception("No IM channels configured or channel service failed to start")
|
||||
|
||||
try:
|
||||
from app.gateway.services import launch_scheduled_thread_run
|
||||
from app.scheduler import ScheduledTaskService
|
||||
|
||||
if getattr(app.state, "scheduled_task_repo", None) is not None and getattr(app.state, "scheduled_task_run_repo", None) is not None:
|
||||
scheduled_task_service = ScheduledTaskService(
|
||||
task_repo=app.state.scheduled_task_repo,
|
||||
task_run_repo=app.state.scheduled_task_run_repo,
|
||||
launch_run=lambda **kwargs: launch_scheduled_thread_run(app=app, **kwargs),
|
||||
poll_interval_seconds=startup_config.scheduler.poll_interval_seconds,
|
||||
lease_seconds=startup_config.scheduler.lease_seconds,
|
||||
max_concurrent_runs=startup_config.scheduler.max_concurrent_runs,
|
||||
)
|
||||
app.state.scheduled_task_service = scheduled_task_service
|
||||
if startup_config.scheduler.enabled:
|
||||
await scheduled_task_service.start()
|
||||
except Exception:
|
||||
logger.exception("Failed to initialize scheduled task service")
|
||||
|
||||
yield
|
||||
|
||||
try:
|
||||
await auth.close_oidc_service()
|
||||
except Exception:
|
||||
logger.exception("Failed to close OIDC service")
|
||||
|
||||
# Stop channel service on shutdown (bounded to prevent worker hang)
|
||||
try:
|
||||
from app.channels.service import stop_channel_service
|
||||
|
||||
await asyncio.wait_for(
|
||||
stop_channel_service(),
|
||||
timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
except TimeoutError:
|
||||
logger.warning(
|
||||
"Channel service shutdown exceeded %.1fs; proceeding with worker exit.",
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to stop channel service")
|
||||
|
||||
if getattr(app.state, "scheduled_task_service", None) is not None:
|
||||
try:
|
||||
await app.state.scheduled_task_service.stop()
|
||||
except Exception:
|
||||
logger.exception("Failed to stop scheduled task service")
|
||||
|
||||
logger.info("Shutting down API Gateway")
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application.
|
||||
|
||||
Returns:
|
||||
Configured FastAPI application instance.
|
||||
"""
|
||||
config = get_gateway_config()
|
||||
docs_url = "/docs" if config.enable_docs else None
|
||||
redoc_url = "/redoc" if config.enable_docs else None
|
||||
openapi_url = "/openapi.json" if config.enable_docs else None
|
||||
|
||||
app = FastAPI(
|
||||
title="DeerFlow API Gateway",
|
||||
description="""
|
||||
## DeerFlow API Gateway
|
||||
|
||||
API Gateway for DeerFlow - A LangGraph-based AI agent backend with sandbox execution capabilities.
|
||||
|
||||
### Features
|
||||
|
||||
- **Models Management**: Query and retrieve available AI models
|
||||
- **MCP Configuration**: Manage Model Context Protocol (MCP) server configurations
|
||||
- **Memory Management**: Access and manage global memory data for personalized conversations
|
||||
- **Skills Management**: Query and manage skills and their enabled status
|
||||
- **Artifacts**: Access thread artifacts and generated files
|
||||
- **Health Monitoring**: System health check endpoints
|
||||
|
||||
### Architecture
|
||||
|
||||
LangGraph-compatible requests are routed through nginx to this gateway.
|
||||
This gateway provides runtime endpoints for agent runs plus custom endpoints for models, MCP configuration, skills, and artifacts.
|
||||
""",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
docs_url=docs_url,
|
||||
redoc_url=redoc_url,
|
||||
openapi_url=openapi_url,
|
||||
openapi_tags=[
|
||||
{
|
||||
"name": "models",
|
||||
"description": "Operations for querying available AI models and their configurations",
|
||||
},
|
||||
{
|
||||
"name": "mcp",
|
||||
"description": "Manage Model Context Protocol (MCP) server configurations",
|
||||
},
|
||||
{
|
||||
"name": "memory",
|
||||
"description": "Access and manage global memory data for personalized conversations",
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"description": "Manage skills and their configurations",
|
||||
},
|
||||
{
|
||||
"name": "artifacts",
|
||||
"description": "Access and download thread artifacts and generated files",
|
||||
},
|
||||
{
|
||||
"name": "uploads",
|
||||
"description": "Upload and manage user files for threads",
|
||||
},
|
||||
{
|
||||
"name": "threads",
|
||||
"description": "Manage DeerFlow thread-local filesystem data",
|
||||
},
|
||||
{
|
||||
"name": "agents",
|
||||
"description": "Create and manage custom agents with per-agent config and prompts",
|
||||
},
|
||||
{
|
||||
"name": "suggestions",
|
||||
"description": "Generate follow-up question suggestions for conversations",
|
||||
},
|
||||
{
|
||||
"name": "input-polish",
|
||||
"description": "Polish composer draft input before sending",
|
||||
},
|
||||
{
|
||||
"name": "channels",
|
||||
"description": "Manage IM channel integrations (Feishu, Slack, Telegram)",
|
||||
},
|
||||
{
|
||||
"name": "assistants-compat",
|
||||
"description": "LangGraph Platform-compatible assistants API (stub)",
|
||||
},
|
||||
{
|
||||
"name": "runs",
|
||||
"description": "LangGraph Platform-compatible runs lifecycle (create, stream, cancel)",
|
||||
},
|
||||
{
|
||||
"name": "health",
|
||||
"description": "Health check and system status endpoints",
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
# Auth: reject unauthenticated requests to non-public paths (fail-closed safety net)
|
||||
app.add_middleware(AuthMiddleware)
|
||||
|
||||
# CSRF: Double Submit Cookie pattern for state-changing requests
|
||||
app.add_middleware(CSRFMiddleware)
|
||||
|
||||
# CORS: the unified nginx endpoint is same-origin by default. Split-origin
|
||||
# browser clients must opt in with this explicit Gateway allowlist so CORS
|
||||
# and CSRF origin checks share the same source of truth.
|
||||
cors_origins = sorted(get_configured_cors_origins())
|
||||
if cors_origins:
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=cors_origins,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Request trace correlation: when logging.enhance.enabled=true, bind one
|
||||
# trace id per Gateway HTTP request and write it to response start headers.
|
||||
# `logging` is registered as restart-required (see reload_boundary.py) so we
|
||||
# snapshot the flag from the startup AppConfig instead of reading live; a
|
||||
# runtime toggle would otherwise leave the log formatter (installed once by
|
||||
# configure_logging() at lifespan startup) out of sync with the middleware.
|
||||
app.add_middleware(TraceMiddleware, enabled=_resolve_trace_enabled_for_app_construction())
|
||||
|
||||
# Include routers
|
||||
# Models API is mounted at /api/models
|
||||
app.include_router(models.router)
|
||||
|
||||
# Features API is mounted at /api/features
|
||||
app.include_router(features.router)
|
||||
|
||||
# Console API (cross-thread observability) is mounted at /api/console
|
||||
app.include_router(console.router)
|
||||
|
||||
# MCP API is mounted at /api/mcp
|
||||
app.include_router(mcp.router)
|
||||
|
||||
# Memory API is mounted at /api/memory
|
||||
app.include_router(memory.router)
|
||||
|
||||
# Skills API is mounted at /api/skills
|
||||
app.include_router(skills.router)
|
||||
|
||||
# Artifacts API is mounted at /api/threads/{thread_id}/artifacts
|
||||
app.include_router(artifacts.router)
|
||||
|
||||
# Uploads API is mounted at /api/threads/{thread_id}/uploads
|
||||
app.include_router(uploads.router)
|
||||
|
||||
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
||||
app.include_router(threads.router)
|
||||
|
||||
# Scheduled tasks API is mounted at /api/scheduled-tasks
|
||||
app.include_router(scheduled_tasks.router)
|
||||
|
||||
# Agents API is mounted at /api/agents
|
||||
app.include_router(agents.router)
|
||||
|
||||
# Suggestions API is mounted at /api/threads/{thread_id}/suggestions
|
||||
app.include_router(suggestions.router)
|
||||
|
||||
# Input polishing API is mounted at /api/input-polish
|
||||
app.include_router(input_polish.router)
|
||||
|
||||
# User-facing IM channel connection API is mounted at /api/channels
|
||||
app.include_router(channel_connections.router)
|
||||
|
||||
# Channels API is mounted at /api/channels
|
||||
app.include_router(channels.router)
|
||||
|
||||
# Assistants compatibility API (LangGraph Platform stub)
|
||||
app.include_router(assistants_compat.router)
|
||||
|
||||
# Auth API is mounted at /api/v1/auth
|
||||
app.include_router(auth.router)
|
||||
|
||||
# Feedback API is mounted at /api/threads/{thread_id}/runs/{run_id}/feedback
|
||||
app.include_router(feedback.router)
|
||||
|
||||
# Thread Runs API (LangGraph Platform-compatible runs lifecycle)
|
||||
app.include_router(thread_runs.router)
|
||||
|
||||
# Stateless Runs API (stream/wait without a pre-existing thread)
|
||||
app.include_router(runs.router)
|
||||
|
||||
# GitHub webhooks API is mounted at /api/webhooks/github
|
||||
# Exempt from auth and CSRF middleware (see auth_middleware._PUBLIC_PATH_PREFIXES
|
||||
# and csrf_middleware.should_check_csrf); authenticity is enforced via the
|
||||
# X-Hub-Signature-256 HMAC against GITHUB_WEBHOOK_SECRET.
|
||||
# Including this router transitively imports app.gateway.github, which
|
||||
# registers the GitHub channel's ChannelRunPolicy as an import side-effect.
|
||||
#
|
||||
# Fail-closed: only mount the route when a webhook secret is configured
|
||||
# (or when the explicit DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS=1
|
||||
# dev opt-in is set). A misconfigured deployment without a secret cannot
|
||||
# serve forged deliveries because the URL responds 404 — there is no
|
||||
# handler to reach.
|
||||
if github_webhooks.is_route_enabled():
|
||||
app.include_router(github_webhooks.router)
|
||||
logger.info("GitHub webhooks route mounted at /api/webhooks/github")
|
||||
else:
|
||||
logger.warning("GitHub webhooks route NOT mounted: GITHUB_WEBHOOK_SECRET unset and DEER_FLOW_ALLOW_UNVERIFIED_GITHUB_WEBHOOKS not set. /api/webhooks/github will respond 404. Configure either env var to enable the route.")
|
||||
|
||||
@app.get("/health", tags=["health"])
|
||||
async def health_check() -> dict[str, str]:
|
||||
"""Health check endpoint.
|
||||
|
||||
Returns:
|
||||
Service health status information.
|
||||
"""
|
||||
return {"status": "healthy", "service": "deer-flow-gateway"}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
def _resolve_trace_enabled_for_app_construction() -> bool:
|
||||
"""Resolve the trace middleware flag without making imports require config.yaml."""
|
||||
try:
|
||||
return resolve_trace_enabled(get_app_config())
|
||||
except FileNotFoundError:
|
||||
# Startup lifespan still performs strict config loading before serving.
|
||||
logger.debug("config.yaml not found while constructing Gateway app; TraceMiddleware disabled for this app instance")
|
||||
return False
|
||||
|
||||
|
||||
# Create app instance for uvicorn
|
||||
app = create_app()
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Authentication module for DeerFlow.
|
||||
|
||||
This module provides:
|
||||
- JWT-based authentication
|
||||
- Provider Factory pattern for extensible auth methods
|
||||
- UserRepository interface for storage backends (SQLite)
|
||||
"""
|
||||
|
||||
from app.gateway.auth.config import AuthConfig, get_auth_config, set_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError
|
||||
from app.gateway.auth.jwt import TokenPayload, create_access_token, decode_token
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
from app.gateway.auth.models import User, UserResponse
|
||||
from app.gateway.auth.password import hash_password, verify_password
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
__all__ = [
|
||||
# Config
|
||||
"AuthConfig",
|
||||
"get_auth_config",
|
||||
"set_auth_config",
|
||||
# Errors
|
||||
"AuthErrorCode",
|
||||
"AuthErrorResponse",
|
||||
"TokenError",
|
||||
# JWT
|
||||
"TokenPayload",
|
||||
"create_access_token",
|
||||
"decode_token",
|
||||
# Password
|
||||
"hash_password",
|
||||
"verify_password",
|
||||
# Models
|
||||
"User",
|
||||
"UserResponse",
|
||||
# Providers
|
||||
"AuthProvider",
|
||||
"LocalAuthProvider",
|
||||
# Repository
|
||||
"UserRepository",
|
||||
]
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Authentication configuration for DeerFlow."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SECRET_FILE = ".jwt_secret"
|
||||
|
||||
|
||||
class AuthConfig(BaseModel):
|
||||
"""JWT and auth-related configuration. Parsed once at startup.
|
||||
|
||||
Note: the ``users`` table now lives in the shared persistence
|
||||
database managed by ``deerflow.persistence.engine``. The old
|
||||
``users_db_path`` config key has been removed — user storage is
|
||||
configured through ``config.database`` like every other table.
|
||||
"""
|
||||
|
||||
jwt_secret: str = Field(
|
||||
...,
|
||||
description="Secret key for JWT signing. MUST be set via AUTH_JWT_SECRET.",
|
||||
)
|
||||
token_expiry_days: int = Field(default=7, ge=1, le=30)
|
||||
oauth_github_client_id: str | None = Field(default=None)
|
||||
oauth_github_client_secret: str | None = Field(default=None)
|
||||
|
||||
|
||||
_auth_config: AuthConfig | None = None
|
||||
|
||||
|
||||
def _load_or_create_secret() -> str:
|
||||
"""Load persisted JWT secret from ``{base_dir}/.jwt_secret``, or generate and persist a new one."""
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
paths = get_paths()
|
||||
secret_file = paths.base_dir / _SECRET_FILE
|
||||
|
||||
try:
|
||||
if secret_file.exists():
|
||||
secret = secret_file.read_text(encoding="utf-8").strip()
|
||||
if secret:
|
||||
return secret
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Failed to read JWT secret from {secret_file}. Set AUTH_JWT_SECRET explicitly or fix DEER_FLOW_HOME/base directory permissions so DeerFlow can read its persisted auth secret.") from exc
|
||||
|
||||
secret = secrets.token_urlsafe(32)
|
||||
try:
|
||||
secret_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd = os.open(secret_file, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(secret)
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Failed to persist JWT secret to {secret_file}. Set AUTH_JWT_SECRET explicitly or fix DEER_FLOW_HOME/base directory permissions so DeerFlow can store a stable auth secret.") from exc
|
||||
return secret
|
||||
|
||||
|
||||
def get_auth_config() -> AuthConfig:
|
||||
"""Get the global AuthConfig instance. Parses from env on first call."""
|
||||
global _auth_config
|
||||
if _auth_config is None:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
jwt_secret = os.environ.get("AUTH_JWT_SECRET")
|
||||
if not jwt_secret:
|
||||
jwt_secret = _load_or_create_secret()
|
||||
os.environ["AUTH_JWT_SECRET"] = jwt_secret
|
||||
logger.warning(
|
||||
"⚠ AUTH_JWT_SECRET is not set — using an auto-generated secret "
|
||||
"persisted to .jwt_secret. Sessions will survive restarts. "
|
||||
"For production, add AUTH_JWT_SECRET to your .env file: "
|
||||
'python -c "import secrets; print(secrets.token_urlsafe(32))"'
|
||||
)
|
||||
_auth_config = AuthConfig(jwt_secret=jwt_secret)
|
||||
return _auth_config
|
||||
|
||||
|
||||
def set_auth_config(config: AuthConfig) -> None:
|
||||
"""Set the global AuthConfig instance (for testing)."""
|
||||
global _auth_config
|
||||
_auth_config = config
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Write initial admin credentials to a restricted file instead of logs.
|
||||
|
||||
Logging secrets to stdout/stderr is a well-known CodeQL finding
|
||||
(py/clear-text-logging-sensitive-data) — in production those logs
|
||||
get collected into ELK/Splunk/etc and become a secret sprawl
|
||||
source. This helper writes the credential to a 0600 file that only
|
||||
the process user can read, and returns the path so the caller can
|
||||
log **the path** (not the password) for the operator to pick up.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
_CREDENTIAL_FILENAME = "admin_initial_credentials.txt"
|
||||
|
||||
|
||||
def write_initial_credentials(email: str, password: str, *, label: str = "initial") -> Path:
|
||||
"""Write the admin email + password to ``{base_dir}/admin_initial_credentials.txt``.
|
||||
|
||||
The file is created **atomically** with mode 0600 via ``os.open``
|
||||
so the password is never world-readable, even for the single syscall
|
||||
window between ``write_text`` and ``chmod``.
|
||||
|
||||
``label`` distinguishes "initial" (fresh creation) from "reset"
|
||||
(password reset) in the file header so an operator picking up the
|
||||
file after a restart can tell which event produced it.
|
||||
|
||||
Returns the absolute :class:`Path` to the file.
|
||||
"""
|
||||
target = get_paths().base_dir / _CREDENTIAL_FILENAME
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
content = (
|
||||
f"# DeerFlow admin {label} credentials\n# This file is generated on first boot or password reset.\n# Change the password after login via Settings -> Account,\n# then delete this file.\n#\nemail: {email}\npassword: {password}\n"
|
||||
)
|
||||
|
||||
# Atomic 0600 create-or-truncate. O_TRUNC (not O_EXCL) so the
|
||||
# reset-password path can rewrite an existing file without a
|
||||
# separate unlink-then-create dance.
|
||||
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(content)
|
||||
|
||||
return target.resolve()
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Typed error definitions for auth module.
|
||||
|
||||
AuthErrorCode: exhaustive enum of all auth failure conditions.
|
||||
TokenError: exhaustive enum of JWT decode failures.
|
||||
AuthErrorResponse: structured error payload for HTTP responses.
|
||||
"""
|
||||
|
||||
from enum import StrEnum
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class AuthErrorCode(StrEnum):
|
||||
"""Exhaustive list of auth error conditions."""
|
||||
|
||||
INVALID_CREDENTIALS = "invalid_credentials"
|
||||
TOKEN_EXPIRED = "token_expired"
|
||||
TOKEN_INVALID = "token_invalid"
|
||||
USER_NOT_FOUND = "user_not_found"
|
||||
EMAIL_ALREADY_EXISTS = "email_already_exists"
|
||||
PROVIDER_NOT_FOUND = "provider_not_found"
|
||||
NOT_AUTHENTICATED = "not_authenticated"
|
||||
SYSTEM_ALREADY_INITIALIZED = "system_already_initialized"
|
||||
|
||||
|
||||
class TokenError(StrEnum):
|
||||
"""Exhaustive list of JWT decode failure reasons."""
|
||||
|
||||
EXPIRED = "expired"
|
||||
INVALID_SIGNATURE = "invalid_signature"
|
||||
MALFORMED = "malformed"
|
||||
|
||||
|
||||
class AuthErrorResponse(BaseModel):
|
||||
"""Structured error response — replaces bare `detail` strings."""
|
||||
|
||||
code: AuthErrorCode
|
||||
message: str
|
||||
|
||||
|
||||
def token_error_to_code(err: TokenError) -> AuthErrorCode:
|
||||
"""Map TokenError to AuthErrorCode — single source of truth."""
|
||||
if err == TokenError.EXPIRED:
|
||||
return AuthErrorCode.TOKEN_EXPIRED
|
||||
return AuthErrorCode.TOKEN_INVALID
|
||||
@@ -0,0 +1,55 @@
|
||||
"""JWT token creation and verification."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import jwt
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import TokenError
|
||||
|
||||
|
||||
class TokenPayload(BaseModel):
|
||||
"""JWT token payload."""
|
||||
|
||||
sub: str # user_id
|
||||
exp: datetime
|
||||
iat: datetime | None = None
|
||||
ver: int = 0 # token_version — must match User.token_version
|
||||
|
||||
|
||||
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0) -> str:
|
||||
"""Create a JWT access token.
|
||||
|
||||
Args:
|
||||
user_id: The user's UUID as string
|
||||
expires_delta: Optional custom expiry, defaults to 7 days
|
||||
token_version: User's current token_version for invalidation
|
||||
|
||||
Returns:
|
||||
Encoded JWT string
|
||||
"""
|
||||
config = get_auth_config()
|
||||
expiry = expires_delta or timedelta(days=config.token_expiry_days)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
|
||||
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
|
||||
|
||||
|
||||
def decode_token(token: str) -> TokenPayload | TokenError:
|
||||
"""Decode and validate a JWT token.
|
||||
|
||||
Returns:
|
||||
TokenPayload if valid, or a specific TokenError variant.
|
||||
"""
|
||||
config = get_auth_config()
|
||||
try:
|
||||
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
|
||||
return TokenPayload(**payload)
|
||||
except jwt.ExpiredSignatureError:
|
||||
return TokenError.EXPIRED
|
||||
except jwt.InvalidSignatureError:
|
||||
return TokenError.INVALID_SIGNATURE
|
||||
except jwt.PyJWTError:
|
||||
return TokenError.MALFORMED
|
||||
@@ -0,0 +1,132 @@
|
||||
"""Local email/password authentication provider."""
|
||||
|
||||
import logging
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth.password import hash_password_async, needs_rehash, verify_password_async
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LocalAuthProvider(AuthProvider):
|
||||
"""Email/password authentication provider using local database."""
|
||||
|
||||
def __init__(self, repository: UserRepository):
|
||||
"""Initialize with a UserRepository.
|
||||
|
||||
Args:
|
||||
repository: UserRepository implementation (SQLite)
|
||||
"""
|
||||
self._repo = repository
|
||||
|
||||
async def authenticate(self, credentials: dict) -> User | None:
|
||||
"""Authenticate with email and password.
|
||||
|
||||
Args:
|
||||
credentials: dict with 'email' and 'password' keys
|
||||
|
||||
Returns:
|
||||
User if authentication succeeds, None otherwise
|
||||
"""
|
||||
email = credentials.get("email")
|
||||
password = credentials.get("password")
|
||||
|
||||
if not email or not password:
|
||||
return None
|
||||
|
||||
user = await self._repo.get_user_by_email(email)
|
||||
if user is None:
|
||||
return None
|
||||
|
||||
if user.password_hash is None:
|
||||
# OAuth user without local password
|
||||
return None
|
||||
|
||||
if not await verify_password_async(password, user.password_hash):
|
||||
return None
|
||||
|
||||
if needs_rehash(user.password_hash):
|
||||
try:
|
||||
user.password_hash = await hash_password_async(password)
|
||||
await self._repo.update_user(user)
|
||||
except Exception:
|
||||
# Rehash is an opportunistic upgrade; a transient DB error must not
|
||||
# prevent an otherwise-valid login from succeeding.
|
||||
logger.warning("Failed to rehash password for user %s; login will still succeed", user.email, exc_info=True)
|
||||
|
||||
return user
|
||||
|
||||
async def get_user(self, user_id: str) -> User | None:
|
||||
"""Get user by ID."""
|
||||
return await self._repo.get_user_by_id(user_id)
|
||||
|
||||
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
|
||||
"""Create a new local user.
|
||||
|
||||
Args:
|
||||
email: User email address
|
||||
password: Plain text password (will be hashed)
|
||||
system_role: Role to assign ("admin" or "user")
|
||||
needs_setup: If True, user must complete setup on first login
|
||||
|
||||
Returns:
|
||||
Created User instance
|
||||
"""
|
||||
password_hash = await hash_password_async(password) if password else None
|
||||
user = User(
|
||||
email=email,
|
||||
password_hash=password_hash,
|
||||
system_role=system_role,
|
||||
needs_setup=needs_setup,
|
||||
)
|
||||
return await self._repo.create_user(user)
|
||||
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
"""Get user by OAuth provider and ID."""
|
||||
return await self._repo.get_user_by_oauth(provider, oauth_id)
|
||||
|
||||
async def count_users(self) -> int:
|
||||
"""Return total number of registered users."""
|
||||
return await self._repo.count_users()
|
||||
|
||||
async def count_admin_users(self) -> int:
|
||||
"""Return number of admin users."""
|
||||
return await self._repo.count_admin_users()
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user."""
|
||||
return await self._repo.update_user(user)
|
||||
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
"""Get user by email."""
|
||||
return await self._repo.get_user_by_email(email)
|
||||
|
||||
async def create_oauth_user(
|
||||
self,
|
||||
email: str,
|
||||
oauth_provider: str,
|
||||
oauth_id: str,
|
||||
system_role: str = "user",
|
||||
) -> User:
|
||||
"""Create a new user from an OAuth/OIDC login.
|
||||
|
||||
Args:
|
||||
email: Verified email from the OIDC provider
|
||||
oauth_provider: Provider ID (e.g. 'keycloak', 'google')
|
||||
oauth_id: User's subject claim from the ID token
|
||||
system_role: Role to assign ("admin" or "user")
|
||||
|
||||
Returns:
|
||||
Created User instance
|
||||
"""
|
||||
user = User(
|
||||
email=email,
|
||||
password_hash=None,
|
||||
system_role=system_role,
|
||||
needs_setup=False,
|
||||
oauth_provider=oauth_provider,
|
||||
oauth_id=oauth_id,
|
||||
)
|
||||
return await self._repo.create_user(user)
|
||||
@@ -0,0 +1,42 @@
|
||||
"""User Pydantic models for authentication."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, EmailStr, Field
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
"""Return current UTC time (timezone-aware)."""
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""Internal user representation."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: UUID = Field(default_factory=uuid4, description="Primary key")
|
||||
email: EmailStr = Field(..., description="Unique email address")
|
||||
password_hash: str | None = Field(None, description="bcrypt hash, nullable for OAuth users")
|
||||
system_role: Literal["admin", "user"] = Field(default="user")
|
||||
created_at: datetime = Field(default_factory=_utc_now)
|
||||
|
||||
# OAuth linkage (optional)
|
||||
oauth_provider: str | None = Field(None, description="e.g. 'github', 'google'")
|
||||
oauth_id: str | None = Field(None, description="User ID from OAuth provider")
|
||||
|
||||
# Auth lifecycle
|
||||
needs_setup: bool = Field(default=False, description="True when a reset account must complete setup")
|
||||
token_version: int = Field(default=0, description="Incremented on password change to invalidate old JWTs")
|
||||
|
||||
|
||||
class UserResponse(BaseModel):
|
||||
"""Response model for user info endpoint."""
|
||||
|
||||
id: str
|
||||
email: str
|
||||
system_role: Literal["admin", "user"]
|
||||
needs_setup: bool = False
|
||||
oauth_provider: str | None = Field(None, description="OAuth/SSO provider ID if the user logged in via SSO (e.g. 'keycloak')")
|
||||
@@ -0,0 +1,433 @@
|
||||
"""OIDC (OpenID Connect) authentication service.
|
||||
|
||||
Provides provider-agnostic OIDC operations: discovery, authorization URL
|
||||
generation, token exchange, ID token validation, and userinfo retrieval.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import secrets
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import httpx
|
||||
import jwt
|
||||
from jwt import PyJWK
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Data types ────────────────────────────────────────────────────────────
|
||||
|
||||
OIDC_DISCOVERY_PATH = "/.well-known/openid-configuration"
|
||||
METADATA_CACHE_TTL = 300 # 5 minutes
|
||||
JWKS_CACHE_TTL = 300
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OIDCMetadata:
|
||||
"""Resolved OIDC provider metadata after discovery."""
|
||||
|
||||
issuer: str
|
||||
authorization_endpoint: str
|
||||
token_endpoint: str
|
||||
userinfo_endpoint: str | None
|
||||
jwks_uri: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OIDCIdentity:
|
||||
"""Normalized identity extracted from an OIDC provider response."""
|
||||
|
||||
provider: str
|
||||
subject: str
|
||||
email: str
|
||||
email_verified: bool
|
||||
name: str | None
|
||||
claims: dict[str, Any]
|
||||
|
||||
|
||||
class OIDCError(Exception):
|
||||
"""Base error for OIDC operations. Message is safe for API responses."""
|
||||
|
||||
|
||||
class OIDCProviderError(OIDCError):
|
||||
"""The OIDC provider returned an error (e.g. access_denied)."""
|
||||
|
||||
|
||||
class OIDCValidationError(OIDCError):
|
||||
"""ID token validation failed."""
|
||||
|
||||
|
||||
class OIDCUserInfoMismatch(OIDCError):
|
||||
"""UserInfo sub does not match ID token sub."""
|
||||
|
||||
|
||||
# ── Service ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class OIDCService:
|
||||
"""OIDC authentication service.
|
||||
|
||||
Uses in-process caching for provider metadata and JWKS. The cache is
|
||||
keyed by the provider's ``issuer`` — different providers get separate
|
||||
entries. TTLs are configurable via constructor arguments.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
metadata_cache_ttl: float = METADATA_CACHE_TTL,
|
||||
jwks_cache_ttl: float = JWKS_CACHE_TTL,
|
||||
) -> None:
|
||||
self._metadata_cache: dict[str, tuple[float, dict[str, Any]]] = {}
|
||||
self._jwks_cache: dict[str, tuple[float, dict[str, Any]]] = {}
|
||||
self._metadata_ttl = metadata_cache_ttl
|
||||
self._jwks_ttl = jwks_cache_ttl
|
||||
self._http = httpx.AsyncClient(timeout=httpx.Timeout(15.0))
|
||||
|
||||
async def close(self) -> None:
|
||||
"""Close the underlying HTTP client."""
|
||||
await self._http.aclose()
|
||||
|
||||
# ── Discovery ──────────────────────────────────────────────────────────
|
||||
|
||||
async def discover(self, issuer: str, overrides: dict[str, str | None] | None = None) -> OIDCMetadata:
|
||||
"""Fetch and cache OIDC discovery metadata from the issuer.
|
||||
|
||||
``overrides`` may contain endpoint URIs to override discovery values
|
||||
(e.g. for providers with non-standard endpoints).
|
||||
"""
|
||||
now = time.time()
|
||||
cached = self._metadata_cache.get(issuer)
|
||||
if cached and now - cached[0] < self._metadata_ttl:
|
||||
return self._metadata_from_dict(cached[1], overrides)
|
||||
|
||||
discovery_url = issuer.rstrip("/") + OIDC_DISCOVERY_PATH
|
||||
try:
|
||||
resp = await self._http.get(discovery_url)
|
||||
resp.raise_for_status()
|
||||
data: dict[str, Any] = resp.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise OIDCError(f"OIDC discovery failed for issuer {issuer}: HTTP {exc.response.status_code}") from exc
|
||||
except httpx.RequestError as exc:
|
||||
raise OIDCError(f"OIDC discovery failed for issuer {issuer}: {exc}") from exc
|
||||
|
||||
discovered_issuer = data.get("issuer")
|
||||
if not discovered_issuer:
|
||||
raise OIDCError(f"OIDC discovery response from {issuer} is missing the issuer field")
|
||||
|
||||
# RFC 8414 §4: the metadata issuer must equal the configured issuer.
|
||||
# Pinning it prevents a tampered/rogue discovery document from steering
|
||||
# the accepted `iss` (and thus the ID-token forgery surface) to an
|
||||
# attacker-chosen value.
|
||||
if discovered_issuer.rstrip("/") != issuer.rstrip("/"):
|
||||
raise OIDCError(f"OIDC discovered issuer '{discovered_issuer}' does not match configured issuer '{issuer}'")
|
||||
|
||||
self._metadata_cache[issuer] = (now, data)
|
||||
return self._metadata_from_dict(data, overrides)
|
||||
|
||||
def _metadata_from_dict(self, data: dict[str, Any], overrides: dict[str, str | None] | None) -> OIDCMetadata:
|
||||
"""Build OIDCMetadata from a discovery dict, applying endpoint overrides."""
|
||||
overrides = overrides or {}
|
||||
return OIDCMetadata(
|
||||
issuer=data["issuer"],
|
||||
authorization_endpoint=overrides.get("authorization_endpoint") or data["authorization_endpoint"],
|
||||
token_endpoint=overrides.get("token_endpoint") or data["token_endpoint"],
|
||||
userinfo_endpoint=overrides.get("userinfo_endpoint") or data.get("userinfo_endpoint"),
|
||||
jwks_uri=overrides.get("jwks_uri") or data["jwks_uri"],
|
||||
)
|
||||
|
||||
# ── Authorization URL ──────────────────────────────────────────────────
|
||||
|
||||
def build_authorization_url(
|
||||
self,
|
||||
metadata: OIDCMetadata,
|
||||
client_id: str,
|
||||
redirect_uri: str,
|
||||
scopes: list[str],
|
||||
state: str,
|
||||
nonce: str | None = None,
|
||||
code_challenge: str | None = None,
|
||||
) -> str:
|
||||
"""Build the OIDC authorization URL for the provider.
|
||||
|
||||
Returns a URL the browser should be redirected to.
|
||||
"""
|
||||
params: dict[str, str] = {
|
||||
"response_type": "code",
|
||||
"client_id": client_id,
|
||||
"redirect_uri": redirect_uri,
|
||||
"scope": " ".join(scopes),
|
||||
"state": state,
|
||||
}
|
||||
if nonce:
|
||||
params["nonce"] = nonce
|
||||
if code_challenge:
|
||||
params["code_challenge"] = code_challenge
|
||||
params["code_challenge_method"] = "S256"
|
||||
|
||||
return f"{metadata.authorization_endpoint}?{urlencode(params)}"
|
||||
|
||||
# ── Token exchange ─────────────────────────────────────────────────────
|
||||
|
||||
async def exchange_code(
|
||||
self,
|
||||
metadata: OIDCMetadata,
|
||||
client_id: str,
|
||||
client_secret: str | None,
|
||||
code: str,
|
||||
redirect_uri: str,
|
||||
code_verifier: str | None = None,
|
||||
auth_method: str = "client_secret_post",
|
||||
) -> dict[str, Any]:
|
||||
"""Exchange the authorization code for tokens at the token endpoint."""
|
||||
data: dict[str, str] = {
|
||||
"grant_type": "authorization_code",
|
||||
"code": code,
|
||||
"redirect_uri": redirect_uri,
|
||||
"client_id": client_id,
|
||||
}
|
||||
if code_verifier:
|
||||
data["code_verifier"] = code_verifier
|
||||
|
||||
headers: dict[str, str] = {"Accept": "application/json"}
|
||||
|
||||
if auth_method == "client_secret_basic" and client_secret:
|
||||
import base64
|
||||
|
||||
creds = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode("ascii")
|
||||
headers["Authorization"] = f"Basic {creds}"
|
||||
elif auth_method == "client_secret_post" and client_secret:
|
||||
data["client_secret"] = client_secret
|
||||
|
||||
try:
|
||||
resp = await self._http.post(metadata.token_endpoint, data=data, headers=headers)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
body = "unknown"
|
||||
try:
|
||||
body = exc.response.text[:200]
|
||||
except Exception:
|
||||
pass
|
||||
raise OIDCError(f"Token exchange failed: HTTP {exc.response.status_code} — {body}") from exc
|
||||
except httpx.RequestError as exc:
|
||||
raise OIDCError(f"Token exchange failed: {exc}") from exc
|
||||
|
||||
# ── JWKS loading ───────────────────────────────────────────────────────
|
||||
|
||||
async def _load_jwks(self, jwks_uri: str, force_refresh: bool = False) -> dict[str, Any]:
|
||||
"""Load (and cache) JWKS from the provider.
|
||||
|
||||
Set ``force_refresh=True`` to bypass the cache (e.g. on a kid miss).
|
||||
"""
|
||||
now = time.time()
|
||||
cached = self._jwks_cache.get(jwks_uri)
|
||||
if not force_refresh and cached and now - cached[0] < self._jwks_ttl:
|
||||
return cached[1]
|
||||
|
||||
try:
|
||||
resp = await self._http.get(jwks_uri)
|
||||
resp.raise_for_status()
|
||||
data: dict[str, Any] = resp.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise OIDCError(f"JWKS fetch failed: HTTP {exc.response.status_code}") from exc
|
||||
except httpx.RequestError as exc:
|
||||
raise OIDCError(f"JWKS fetch failed: {exc}") from exc
|
||||
|
||||
self._jwks_cache[jwks_uri] = (now, data)
|
||||
return data
|
||||
|
||||
async def _resolve_signing_key(
|
||||
self,
|
||||
jwks_data: dict[str, Any],
|
||||
kid: str | None,
|
||||
algorithm: str,
|
||||
jwks_uri: str,
|
||||
) -> Any | None:
|
||||
"""Find the signing key matching ``kid`` in the JWKS.
|
||||
|
||||
Returns the key object or ``None`` if no match is found. Catches
|
||||
invalid JWK entries (e.g. wrong key type for the algorithm) and
|
||||
logs a warning so a single bad entry does not crash validation.
|
||||
"""
|
||||
for jwk_dict in jwks_data.get("keys", []):
|
||||
if kid and jwk_dict.get("kid") != kid:
|
||||
continue
|
||||
try:
|
||||
jwk = PyJWK(jwk_dict, algorithm=algorithm)
|
||||
return jwk.key
|
||||
except jwt.PyJWTError as exc:
|
||||
logger.warning("Skipping invalid JWK (kid=%s) from %s: %s", kid, jwks_uri, exc)
|
||||
if not kid:
|
||||
# No kid in token — try next key
|
||||
continue
|
||||
# kid was specified and this key is the one — fail fast
|
||||
raise OIDCValidationError(f"JWK for kid={kid} is invalid: {exc}") from exc
|
||||
return None
|
||||
|
||||
# ── ID token validation ────────────────────────────────────────────────
|
||||
|
||||
async def validate_id_token(
|
||||
self,
|
||||
metadata: OIDCMetadata,
|
||||
client_id: str,
|
||||
id_token: str,
|
||||
nonce: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Validate the ID token and return its claims.
|
||||
|
||||
Validates: signature (via JWKS), issuer, audience, expiration,
|
||||
issued-at, and nonce (if provided).
|
||||
"""
|
||||
jwks_data = await self._load_jwks(metadata.jwks_uri)
|
||||
|
||||
# Resolve the signing key from the JWKS using the token's kid header
|
||||
jwt_header = jwt.get_unverified_header(id_token)
|
||||
kid = jwt_header.get("kid")
|
||||
alg = jwt_header.get("alg", "RS256")
|
||||
|
||||
allowed_algorithms = ["RS256", "RS384", "RS512", "ES256", "ES384", "ES512"]
|
||||
if alg not in allowed_algorithms:
|
||||
raise OIDCValidationError(f"ID token uses unsupported algorithm '{alg}'")
|
||||
|
||||
# Resolve signing key, refetching JWKS once on kid miss for key rotation
|
||||
signing_key = await self._resolve_signing_key(jwks_data, kid, alg, metadata.jwks_uri)
|
||||
if signing_key is None:
|
||||
jwks_data = await self._load_jwks(metadata.jwks_uri, force_refresh=True)
|
||||
signing_key = await self._resolve_signing_key(jwks_data, kid, alg, metadata.jwks_uri)
|
||||
if signing_key is None:
|
||||
raise OIDCValidationError(f"No matching JWK found for kid={kid} after JWKS refresh")
|
||||
|
||||
try:
|
||||
claims = jwt.decode(
|
||||
id_token,
|
||||
key=signing_key,
|
||||
algorithms=allowed_algorithms,
|
||||
audience=client_id,
|
||||
issuer=metadata.issuer,
|
||||
options={
|
||||
"verify_exp": True,
|
||||
"verify_iat": True,
|
||||
"require": ["exp", "iss", "sub", "aud"],
|
||||
},
|
||||
)
|
||||
except jwt.ExpiredSignatureError:
|
||||
raise OIDCValidationError("ID token has expired")
|
||||
except jwt.InvalidIssuerError:
|
||||
raise OIDCValidationError("ID token has an invalid issuer")
|
||||
except jwt.InvalidAudienceError:
|
||||
raise OIDCValidationError("ID token has an invalid audience")
|
||||
except jwt.PyJWTError as exc:
|
||||
raise OIDCValidationError(f"ID token validation failed: {exc}") from exc
|
||||
|
||||
# Validate nonce if expected
|
||||
if nonce is not None:
|
||||
token_nonce = claims.get("nonce")
|
||||
if not token_nonce:
|
||||
raise OIDCValidationError("ID token is missing the nonce claim")
|
||||
if not _constant_time_compare(nonce, token_nonce):
|
||||
raise OIDCValidationError("ID token nonce does not match")
|
||||
|
||||
return claims
|
||||
|
||||
# ── UserInfo ────────────────────────────────────────────────────────────
|
||||
|
||||
async def fetch_userinfo(self, metadata: OIDCMetadata, access_token: str, expected_sub: str) -> dict[str, Any]:
|
||||
"""Fetch userinfo from the UserInfo endpoint.
|
||||
|
||||
Validates that the ``sub`` claim matches ``expected_sub``
|
||||
(from the ID token) to prevent userinfo injection.
|
||||
"""
|
||||
if not metadata.userinfo_endpoint:
|
||||
return {}
|
||||
|
||||
headers = {"Authorization": f"Bearer {access_token}"}
|
||||
try:
|
||||
resp = await self._http.get(metadata.userinfo_endpoint, headers=headers)
|
||||
resp.raise_for_status()
|
||||
userinfo: dict[str, Any] = resp.json()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
raise OIDCError(f"UserInfo fetch failed: HTTP {exc.response.status_code}") from exc
|
||||
except httpx.RequestError as exc:
|
||||
raise OIDCError(f"UserInfo fetch failed: {exc}") from exc
|
||||
|
||||
if userinfo.get("sub") and userinfo["sub"] != expected_sub:
|
||||
raise OIDCUserInfoMismatch("UserInfo sub does not match ID token sub")
|
||||
|
||||
return userinfo
|
||||
|
||||
# ── Orchestrated callback ──────────────────────────────────────────────
|
||||
|
||||
async def authenticate_callback(
|
||||
self,
|
||||
provider_id: str,
|
||||
metadata: OIDCMetadata,
|
||||
client_id: str,
|
||||
client_secret: str | None,
|
||||
code: str,
|
||||
redirect_uri: str,
|
||||
code_verifier: str | None = None,
|
||||
nonce: str | None = None,
|
||||
auth_method: str = "client_secret_post",
|
||||
) -> OIDCIdentity:
|
||||
"""Orchestrate the full OIDC callback: token exchange, ID token validation, userinfo.
|
||||
|
||||
Returns a normalized ``OIDCIdentity``.
|
||||
"""
|
||||
token_response = await self.exchange_code(
|
||||
metadata=metadata,
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
code=code,
|
||||
redirect_uri=redirect_uri,
|
||||
code_verifier=code_verifier,
|
||||
auth_method=auth_method,
|
||||
)
|
||||
|
||||
id_token = token_response.get("id_token")
|
||||
if not id_token:
|
||||
raise OIDCError("Token response is missing id_token")
|
||||
|
||||
access_token = token_response.get("access_token", "")
|
||||
|
||||
claims = await self.validate_id_token(
|
||||
metadata=metadata,
|
||||
client_id=client_id,
|
||||
id_token=id_token,
|
||||
nonce=nonce,
|
||||
)
|
||||
|
||||
# Fetch userinfo for email/name if not present in ID token
|
||||
userinfo: dict[str, Any] = {}
|
||||
if metadata.userinfo_endpoint and access_token:
|
||||
try:
|
||||
userinfo = await self.fetch_userinfo(
|
||||
metadata=metadata,
|
||||
access_token=access_token,
|
||||
expected_sub=claims["sub"],
|
||||
)
|
||||
except OIDCError as exc:
|
||||
logger.warning("OIDC userinfo fetch failed (continuing with ID token): %s", exc)
|
||||
|
||||
# Merge userinfo into claims (userinfo takes precedence for email)
|
||||
merged = {**claims, **userinfo}
|
||||
|
||||
email = merged.get("email") or ""
|
||||
email_verified = merged.get("email_verified") is True
|
||||
|
||||
return OIDCIdentity(
|
||||
provider=provider_id,
|
||||
subject=claims["sub"],
|
||||
email=email,
|
||||
email_verified=email_verified,
|
||||
name=merged.get("name"),
|
||||
claims=merged,
|
||||
)
|
||||
|
||||
|
||||
def _constant_time_compare(a: str, b: str) -> bool:
|
||||
"""Constant-time string comparison."""
|
||||
return secrets.compare_digest(a, b)
|
||||
@@ -0,0 +1,121 @@
|
||||
"""OIDC state management via signed HttpOnly cookies.
|
||||
|
||||
Stores OIDC state, nonce, and PKCE verifier in a short-lived signed cookie
|
||||
instead of server-side storage. This keeps the implementation stateless and
|
||||
compatible with multi-worker deployments without Redis.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
import time
|
||||
|
||||
import jwt
|
||||
from fastapi import Request, Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.csrf_middleware import is_secure_request
|
||||
|
||||
OIDC_STATE_COOKIE_PREFIX = "df_oidc_state_"
|
||||
OIDC_STATE_MAX_AGE = 300 # 5 minutes
|
||||
OIDC_STATE_BYTES = 32
|
||||
OIDC_NONCE_BYTES = 16
|
||||
OIDC_CODE_VERIFIER_BYTES = 32
|
||||
|
||||
|
||||
class OIDCStatePayload(BaseModel):
|
||||
"""Payload stored inside the signed OIDC state cookie."""
|
||||
|
||||
provider: str = Field(description="OIDC provider ID (must match the state cookie)") # noqa: E501
|
||||
state: str = Field(description="Cryptographically random state value — compared in constant time with the query param") # noqa: E501
|
||||
nonce: str | None = Field(default=None, description="OIDC nonce, verified against the ID token nonce claim")
|
||||
code_verifier: str | None = Field(default=None, description="PKCE code verifier, sent during token exchange")
|
||||
next_path: str = Field(default="/workspace", description="Redirect target after successful auth")
|
||||
issued_at: float = Field(default_factory=time.time, description="Unix timestamp of cookie creation")
|
||||
|
||||
|
||||
def _sign_state_payload(payload: OIDCStatePayload) -> str:
|
||||
"""Sign the state payload with the JWT secret to prevent tampering."""
|
||||
secret = get_auth_config().jwt_secret
|
||||
return jwt.encode(payload.model_dump(), secret, algorithm="HS256")
|
||||
|
||||
|
||||
def _verify_state_signed(signed: str, max_age: int = OIDC_STATE_MAX_AGE) -> OIDCStatePayload | None:
|
||||
"""Verify a signed state payload and return it, or None if invalid/expired."""
|
||||
secret = get_auth_config().jwt_secret
|
||||
try:
|
||||
decoded = jwt.decode(signed, secret, algorithms=["HS256"])
|
||||
payload = OIDCStatePayload(**decoded)
|
||||
if time.time() - payload.issued_at > max_age:
|
||||
return None
|
||||
return payload
|
||||
except jwt.PyJWTError:
|
||||
return None
|
||||
|
||||
|
||||
def generate_oidc_state() -> str:
|
||||
"""Generate a cryptographically random state string."""
|
||||
return secrets.token_urlsafe(OIDC_STATE_BYTES)
|
||||
|
||||
|
||||
def generate_nonce() -> str:
|
||||
"""Generate a cryptographically random nonce for ID token validation."""
|
||||
return secrets.token_urlsafe(OIDC_NONCE_BYTES)
|
||||
|
||||
|
||||
def generate_code_verifier() -> str:
|
||||
"""Generate a PKCE code verifier (plain random string)."""
|
||||
return secrets.token_urlsafe(OIDC_CODE_VERIFIER_BYTES)
|
||||
|
||||
|
||||
def compute_code_challenge(verifier: str) -> str:
|
||||
"""Compute the S256 PKCE code challenge from a verifier."""
|
||||
import hashlib
|
||||
|
||||
return _base64url_encode(hashlib.sha256(verifier.encode("ascii")).digest())
|
||||
|
||||
|
||||
def _base64url_encode(data: bytes) -> str:
|
||||
"""Base64url-encode without padding, as required by RFC 7636 and OIDC."""
|
||||
import base64
|
||||
|
||||
return base64.urlsafe_b64encode(data).rstrip(b"=").decode("ascii")
|
||||
|
||||
|
||||
def _cookie_name(provider: str) -> str:
|
||||
return f"{OIDC_STATE_COOKIE_PREFIX}{provider}"
|
||||
|
||||
|
||||
def set_state_cookie(response: Response, request: Request, payload: OIDCStatePayload) -> None:
|
||||
"""Set the signed OIDC state cookie on the response."""
|
||||
signed = _sign_state_payload(payload)
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key=_cookie_name(payload.provider),
|
||||
value=signed,
|
||||
httponly=True,
|
||||
secure=is_https,
|
||||
samesite="lax",
|
||||
max_age=OIDC_STATE_MAX_AGE,
|
||||
path=f"/api/v1/auth/callback/{payload.provider}",
|
||||
)
|
||||
|
||||
|
||||
def get_state_cookie(request: Request, provider: str) -> OIDCStatePayload | None:
|
||||
"""Read and verify the signed OIDC state cookie for the given provider."""
|
||||
signed = request.cookies.get(_cookie_name(provider))
|
||||
if not signed:
|
||||
return None
|
||||
return _verify_state_signed(signed)
|
||||
|
||||
|
||||
def delete_state_cookie(response: Response, request: Request, provider: str) -> None:
|
||||
"""Delete the OIDC state cookie."""
|
||||
is_https = is_secure_request(request)
|
||||
response.delete_cookie(
|
||||
key=_cookie_name(provider),
|
||||
secure=is_https,
|
||||
samesite="lax",
|
||||
path=f"/api/v1/auth/callback/{provider}",
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Password hashing utilities with versioned hash format.
|
||||
|
||||
Hash format: ``$dfv<N>$<bcrypt_hash>`` where ``<N>`` is the version.
|
||||
|
||||
- **v1** (legacy): ``bcrypt(password)`` — plain bcrypt, susceptible to
|
||||
72-byte silent truncation.
|
||||
- **v2** (current): ``bcrypt(b64(sha256(password)))`` — SHA-256 pre-hash
|
||||
avoids the 72-byte truncation limit so the full password contributes
|
||||
to the hash.
|
||||
|
||||
Verification auto-detects the version and falls back to v1 for hashes
|
||||
without a prefix, so existing deployments upgrade transparently on next
|
||||
login.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
|
||||
import bcrypt
|
||||
|
||||
_CURRENT_VERSION = 2
|
||||
_PREFIX_V2 = "$dfv2$"
|
||||
_PREFIX_V1 = "$dfv1$"
|
||||
|
||||
|
||||
def _pre_hash_v2(password: str) -> bytes:
|
||||
"""SHA-256 pre-hash to bypass bcrypt's 72-byte limit."""
|
||||
return base64.b64encode(hashlib.sha256(password.encode("utf-8")).digest())
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
"""Hash a password (current version: v2 — SHA-256 + bcrypt)."""
|
||||
raw = bcrypt.hashpw(_pre_hash_v2(password), bcrypt.gensalt()).decode("utf-8")
|
||||
return f"{_PREFIX_V2}{raw}"
|
||||
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password, auto-detecting the hash version.
|
||||
|
||||
Accepts v2 (``$dfv2$…``), v1 (``$dfv1$…``), and bare bcrypt hashes
|
||||
(treated as v1 for backward compatibility with pre-versioning data).
|
||||
"""
|
||||
try:
|
||||
if hashed_password.startswith(_PREFIX_V2):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V2) :]
|
||||
return bcrypt.checkpw(_pre_hash_v2(plain_password), bcrypt_hash.encode("utf-8"))
|
||||
|
||||
if hashed_password.startswith(_PREFIX_V1):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V1) :]
|
||||
else:
|
||||
bcrypt_hash = hashed_password
|
||||
|
||||
return bcrypt.checkpw(plain_password.encode("utf-8"), bcrypt_hash.encode("utf-8"))
|
||||
except ValueError:
|
||||
# bcrypt raises ValueError for malformed or corrupt hashes (e.g., invalid salt).
|
||||
# Fail closed rather than crashing the request.
|
||||
return False
|
||||
|
||||
|
||||
def needs_rehash(hashed_password: str) -> bool:
|
||||
"""Return True if the hash uses an older version and should be rehashed."""
|
||||
return not hashed_password.startswith(_PREFIX_V2)
|
||||
|
||||
|
||||
async def hash_password_async(password: str) -> str:
|
||||
"""Hash a password using bcrypt (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password hashing.
|
||||
"""
|
||||
return await asyncio.to_thread(hash_password, password)
|
||||
|
||||
|
||||
async def verify_password_async(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password against its hash (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password verification.
|
||||
"""
|
||||
return await asyncio.to_thread(verify_password, plain_password, hashed_password)
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Auth provider abstraction."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class AuthProvider(ABC):
|
||||
"""Abstract base class for authentication providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def authenticate(self, credentials: dict) -> "User | None":
|
||||
"""Authenticate user with given credentials.
|
||||
|
||||
Returns User if authentication succeeds, None otherwise.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user(self, user_id: str) -> "User | None":
|
||||
"""Retrieve user by ID."""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
# Import User at runtime to avoid circular imports
|
||||
from app.gateway.auth.models import User # noqa: E402
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user