commit 75f3dd141c97ed6bc4aae4340b24e725829175c1 Author: wehub-resource-sync Date: Mon Jul 13 13:11:07 2026 +0800 chore: import upstream snapshot with attribution diff --git a/.claude/agents/claude-md-reviewer.md b/.claude/agents/claude-md-reviewer.md new file mode 100644 index 0000000..1eadbee --- /dev/null +++ b/.claude/agents/claude-md-reviewer.md @@ -0,0 +1,104 @@ +--- +name: claude-md-reviewer +description: Audits a plan or a working-tree diff against the repo's CLAUDE.md standards. Invoke after producing an implementation plan and again after finishing an implementation. +tools: Read, Grep, Glob, Bash +--- + +You audit work against the Databasus `CLAUDE.md` standards. You report violations; you never fix them. You have no `Write` or `Edit` tool — the calling agent applies every fix. + +Your caller tells you the mode: **plan** or **implementation**. If the mode is absent, infer it: a plan file path means plan mode, otherwise implementation mode. + +## Step 1 — Resolve scope + +**Implementation mode.** Run `git status --porcelain` and `git diff HEAD` to get the changed files and their contents. Untracked files do not appear in `git diff` — read them with `Read`. + +**Plan mode.** Read the plan file the caller names. Extract the files it proposes to touch, the names it proposes to introduce, and any behavior it proposes to preserve. + +## Step 2 — Read the governing docs + +Every file is governed by **two** documents at once, and you audit it against both. Neither replaces the other: the root doc carries the project-wide philosophy, and the module doc carries the stack-specific rules on top of it. + +1. **Always** read `CLAUDE.md` at the repo root. It governs every file in every module, with no exceptions. +2. **Also** read the module doc for each area the change touches: + + | Touched path | Module doc | + | --- | --- | + | `backend/**` | `backend/CLAUDE.md` | + | `agent/**` | `agent/verification/CLAUDE.md` | + | `frontend/**` | `frontend/CLAUDE.md` | + +A change spanning several modules is audited against the root doc plus *every* module doc it touches. A change under `backend/` is audited against the root doc **and** `backend/CLAUDE.md` — never the module doc alone, and never the root doc alone. + +Skip only the module docs that govern nothing in scope. Never skip the root doc. + +## Step 3 — Audit + +Audit against the root doc and the module doc together. Run the root-level pass over *every* changed file, including files under `backend/`, `agent/` and `frontend/` — a module doc's silence on naming or comments does not exempt that module from the root rules. + +These are the rules that get violated most. They are not the whole of the docs — the docs you read in step 2 are authoritative, and this list is a prompt for where to look first. + +### From the root `CLAUDE.md` — applies to every changed file + +**Naming.** Names state intent, not mechanism. No `data`, `handle`, `process`, `tmp`, `helper`, `manager`. No type-suffix noise (`nameStr`, `agentList`, `tokenObj`). Booleans and predicate methods take `is` / `can` / `has` / `should` — `IsAborted(id)`, not `AbortedContains(id)`. State that holds the entity being acted on names the entity — `deletingAgentId`, not `deletingId`. Getters take a `Get` prefix and name the entity — `GetRunningVerificationIDs()`, not `Active()`; this is house style and deliberately departs from vanilla Go. A getter returning a bool stays a predicate (`Is...`/`Has...`). A name that hides a second effect is a lie — a function that records *and* cancels is `recordAndCancelAborts`, or it is two functions. Tests use domain nouns, never `got` / `want` / `expected`. + +**Comments.** The default is no comment. A comment that restates what the code does is a naming bug: if `// Foo does X` sits above `func Foo`, the fix is to rename `Foo` until the comment is redundant. Only a *why* justifies a comment — a business rule, a cross-system constraint, a non-obvious optimisation. No "how it was" comments (`used to be X`, `renamed from Y`, `kept for legacy callers`); history lives in git. + +**Backward compatibility.** Never preserved unless the user asked for it. Flag every deprecation shim, alias, and fallback for the old shape. + +**Language.** English only in code, comments, identifiers, log messages, API strings, test assertions, and commit messages — including user-facing fallback copy. + +**Types and signatures.** No generic type names (`Manager`, `Provisioner`, `Handler`). No package stutter (`container.ContainerManager`) — `revive` fails CI on it; put the noun on the variable instead. Roughly four or more positional parameters, or two adjacent same-typed parameters, become a named struct. + +**Security.** No disabled or weakened security checks to make a build pass. Every GitHub Action pinned to a full commit SHA with a `# vX.Y.Z` comment — no floating `@v4` or `@main`. Workflows default to top-level `permissions: contents: read`. + +### From the module docs — applies on top of the root pass + +**`backend/CLAUDE.md`.** Dependency injection through `SetupDependencies()`; never inject another feature's repository. Controller tests preferred over unit tests; `features/tests/` is reserved for backup→restore cycle tests; clean up test data. Logging: values in the message, IDs and errors as key-value pairs, scope IDs via `logger.With(...)`; never log secrets, tokens, or credentials — redact at the logger layer, not at call sites. File organization, spacing between logical statements, time handling, and modern Go (`slices`, `context` helpers, `omitzero` over `omitempty`, `new(val)`). + +**`agent/verification/CLAUDE.md`.** The same spacing, file organization, background-service, testing, time-handling, logging and modern-Go rules as the backend doc, in their agent-specific form. Read it rather than assuming it matches the backend doc. + +**`frontend/CLAUDE.md`.** Feature-Sliced Design: import direction, correct slice placement, no cross-imports between same-layer slices. React component structure order, vertical spacing, UI kit and icons, forms and progressive disclosure, user-facing copy. + +## Step 4 — Lint (implementation mode only) + +Run the linter for each directory the diff actually touches, and no others: + +- `backend/**` → `make lint` in `backend/` +- `agent/**` → `make lint` in `agent/verification/` +- `frontend/**` → `pnpm lint` in `frontend/` + +Report each failure as a finding. Skip this step entirely in plan mode. + +## Step 5 — Report + +Open with a verdict line, alone, exactly one of: + +``` +PASS +CHANGES REQUIRED +``` + +On the line below the verdict, name the docs you audited against, so the caller can see the root doc was not skipped: + +``` +Audited against: CLAUDE.md, backend/CLAUDE.md +``` + +On `CHANGES REQUIRED`, list findings beneath that, most severe first, one per line. Every finding names the doc the rule comes from: + +``` +: — [] — +``` + +For example: + +``` +backend/internal/features/system/agent/service.go:42 — [CLAUDE.md] getters take a Get prefix and name the entity — rename Active() to GetRunningVerificationIDs() +backend/internal/features/system/agent/di.go:17 — [backend/CLAUDE.md] never inject another feature's repository — depend on the audit log service, not AuditLogRepository +``` + +In plan mode a finding cites the plan's section instead of a line number. + +On `PASS`, list nothing below the `Audited against:` line. + +Report only violations of a rule written in one of the docs you read. Do not invent style preferences the docs do not state. Do not restate what the code does. Do not praise. If a rule is ambiguous as applied to this code, say so in the finding and name the reading you took, rather than silently choosing one. diff --git a/.claude/hooks/require-implementation-review.sh b/.claude/hooks/require-implementation-review.sh new file mode 100755 index 0000000..a1b012d --- /dev/null +++ b/.claude/hooks/require-implementation-review.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Stop hook: blocks once per distinct working-tree state until claude-md-reviewer has audited it. +# +# Blocking is guarded by a marker file keyed on session id + a hash of the diff. Claude Code does +# not document a `stop_hook_active` field, so the marker is the only thing standing between this +# hook and an infinite stop loop. Any git failure exits 0 — never block on a broken repo. + +set -uo pipefail + +REVIEWED_SOURCE_DIRS=(backend agent frontend) + +hook_input=$(cat) + +project_dir=${CLAUDE_PROJECT_DIR:-$(git rev-parse --show-toplevel 2>/dev/null)} || exit 0 +[ -n "$project_dir" ] || exit 0 +cd "$project_dir" 2>/dev/null || exit 0 + +git rev-parse --verify HEAD >/dev/null 2>&1 || exit 0 + +changed_paths=$(git status --porcelain -- "${REVIEWED_SOURCE_DIRS[@]}" 2>/dev/null) || exit 0 +[ -n "$changed_paths" ] || exit 0 + +# `git diff` omits untracked files, so hash their contents separately. Without this an edit to a +# newly created file would not change the hash, and its review round would be silently skipped. +tree_hash=$( + { + git diff HEAD -- "${REVIEWED_SOURCE_DIRS[@]}" 2>/dev/null + git ls-files --others --exclude-standard -z -- "${REVIEWED_SOURCE_DIRS[@]}" 2>/dev/null | + xargs -0 -r sha256sum 2>/dev/null + } | sha256sum | cut -d' ' -f1 +) || exit 0 +[ -n "$tree_hash" ] || exit 0 + +session_id=$(printf '%s' "$hook_input" | jq -r '.session_id // "unknown"' 2>/dev/null) +session_id=${session_id//[^a-zA-Z0-9_-]/} +[ -n "$session_id" ] || session_id=unknown + +marker_dir="${TMPDIR:-/tmp}/databasus-claude-review" +mkdir -p "$marker_dir" 2>/dev/null || exit 0 +marker="$marker_dir/$session_id-$tree_hash" + +[ -e "$marker" ] && exit 0 +: >"$marker" 2>/dev/null || exit 0 + +jq -n '{ + decision: "block", + reason: "The working tree has unreviewed changes under backend/, agent/ or frontend/. Invoke the claude-md-reviewer subagent in implementation mode, then resolve every CHANGES REQUIRED finding before stopping. If it returns PASS, stop as normal." +}' diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..ff97ff2 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,25 @@ +{ + "hooks": { + "PostToolUse": [ + { + "matcher": "ExitPlanMode", + "hooks": [ + { + "type": "command", + "command": "jq -n '{hookSpecificOutput:{hookEventName:\"PostToolUse\",additionalContext:\"Before writing any code, invoke the claude-md-reviewer subagent in plan mode on the approved plan file. Resolve every CHANGES REQUIRED finding before implementing.\"}}'" + } + ] + } + ], + "Stop": [ + { + "hooks": [ + { + "type": "command", + "command": "\"$CLAUDE_PROJECT_DIR/.claude/hooks/require-implementation-review.sh\"" + } + ] + } + ] + } +} diff --git a/.coderabbit.yaml b/.coderabbit.yaml new file mode 100644 index 0000000..b42ae7d --- /dev/null +++ b/.coderabbit.yaml @@ -0,0 +1,76 @@ +language: en-US + +reviews: + profile: chill + request_changes_workflow: false + high_level_summary: false + poem: false + review_status: false + commit_status: false + collapse_walkthrough: true + changed_files_summary: false + sequence_diagrams: false + estimate_code_review_effort: false + assess_linked_issues: false + related_issues: false + related_prs: false + suggested_labels: false + auto_apply_labels: false + suggested_reviewers: false + auto_assign_reviewers: false + + auto_review: + enabled: true + auto_incremental_review: true + drafts: false + + finishing_touches: + docstrings: + enabled: false + + path_filters: + - "!**/*.lock" + - "!**/node_modules/**" + - "!**/dist/**" + - "!**/build/**" + - "!**/*.pb.go" + - "!**/*_gen.go" + - "!backend/docs/**" + + path_instructions: + - path: ".github/workflows/**" + instructions: | + Security-critical workflow review: + - All third-party actions must be pinned to a full commit SHA, with a "# vX.Y.Z" tag comment. Flag floating tags (@v4, @main, @master). + - Top-level "permissions:" must exist and default to least privilege (contents: read). Flag missing or overly broad top-level permissions. + - Jobs that don't write should not have write scope. Flag contents: write, packages: write, id-token: write that aren't justified by the job's purpose. + - Untrusted PR input (github.event.pull_request.title, body, head.ref) must not be substituted directly into "run:" scripts (workflow injection). Flag and recommend env-var indirection. + - "pull_request_target" with secrets exposed to fork PRs is a serious risk. Flag and require justification. + + - path: "**/Dockerfile*" + instructions: | + Docker security: + - Pin base images to a specific version tag and ideally a digest. Flag floating tags like ":latest" or bare ":alpine" without a version. + - Avoid running as root in the final image; flag missing USER directive in production stages. + - Don't embed secrets in build args or COPY. Flag suspicious literal-looking secrets, keys, or tokens. + + tools: + gitleaks: + enabled: true + semgrep: + enabled: true + ast-grep: + essential_rules: false + actionlint: + enabled: false + hadolint: + enabled: false + shellcheck: + enabled: false + languagetool: + enabled: false + markdownlint: + enabled: false + +chat: + auto_reply: true diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile new file mode 100644 index 0000000..a3cf6a2 --- /dev/null +++ b/.devcontainer/Dockerfile @@ -0,0 +1,56 @@ +FROM golang:1.26-bookworm + +ARG NODE_MAJOR=24 +ARG PNPM_VERSION=9.15.9 +ARG GOOSE_VERSION=v3.27.1 +ARG SWAG_VERSION=v1.16.4 +ARG GOLANGCI_LINT_VERSION=v2.11.3 +ARG USERNAME=vscode +ARG USER_UID=1000 +ARG USER_GID=1000 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + ca-certificates curl gnupg sudo git \ + postgresql-client \ + zstd \ + python3 python3-pip pre-commit \ + libncurses5 libncurses6 libtinfo5 libtinfo6 \ + libmariadb3 libgnutls30 \ + && mkdir -p /etc/apt/keyrings \ + && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key \ + | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \ + && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" \ + > /etc/apt/sources.list.d/nodesource.list \ + && curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg \ + | gpg --dearmor -o /etc/apt/keyrings/githubcli-archive-keyring.gpg \ + && chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" \ + > /etc/apt/sources.list.d/github-cli.list \ + && apt-get update \ + && apt-get install -y --no-install-recommends nodejs gh \ + && rm -rf /var/lib/apt/lists/* + +RUN corepack enable + +RUN groupadd --gid ${USER_GID} ${USERNAME} \ + && useradd --uid ${USER_UID} --gid ${USER_GID} --create-home --shell /bin/bash ${USERNAME} \ + && echo "${USERNAME} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/${USERNAME} \ + && chmod 0440 /etc/sudoers.d/${USERNAME} \ + && mkdir -p /home/${USERNAME}/go /home/${USERNAME}/.cache /home/${USERNAME}/.local/share/pnpm \ + && chown -R ${USERNAME}:${USERNAME} /home/${USERNAME} + +ENV GOPATH=/home/${USERNAME}/go +ENV PATH=${GOPATH}/bin:${PATH} + +USER ${USERNAME} + +RUN corepack prepare pnpm@${PNPM_VERSION} --activate + +RUN go install github.com/pressly/goose/v3/cmd/goose@${GOOSE_VERSION} \ + && go install github.com/swaggo/swag/cmd/swag@${SWAG_VERSION} + +RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh \ + | sh -s -- -b $(go env GOPATH)/bin ${GOLANGCI_LINT_VERSION} + +USER root diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json new file mode 100644 index 0000000..97ee9e1 --- /dev/null +++ b/.devcontainer/devcontainer.json @@ -0,0 +1,54 @@ +{ + "name": "Databasus", + "build": { + "dockerfile": "Dockerfile", + "context": "." + }, + "workspaceFolder": "/workspaces/databasus", + "workspaceMount": "source=databasus-workspace,target=/workspaces/databasus,type=volume", + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": { + "version": "latest", + "moby": true, + "dockerDashComposeVersion": "latest" + } + }, + "mounts": [ + "source=databasus-go-mod-cache,target=/home/vscode/go/pkg/mod,type=volume", + "source=databasus-go-build-cache,target=/home/vscode/.cache/go-build,type=volume", + "source=databasus-golangci-cache,target=/home/vscode/.cache/golangci-lint,type=volume", + "source=databasus-pre-commit-cache,target=/home/vscode/.cache/pre-commit,type=volume", + "source=databasus-pnpm-store,target=/home/vscode/.local/share/pnpm,type=volume" + ], + "forwardPorts": [4005, 5173], + "portsAttributes": { + "4005": { "label": "Backend (Gin)" }, + "5173": { "label": "Vite dev server", "onAutoForward": "openBrowser" } + }, + "containerEnv": { + "GOCACHE": "/home/vscode/.cache/go-build", + "GOMODCACHE": "/home/vscode/go/pkg/mod" + }, + "customizations": { + "vscode": { + "extensions": [ + "golang.go", + "ms-azuretools.vscode-docker", + "redhat.vscode-yaml", + "dbaeumer.vscode-eslint", + "esbenp.prettier-vscode", + "bradlc.vscode-tailwindcss" + ], + "settings": { + "go.lintTool": "golangci-lint", + "go.useLanguageServer": true, + "go.testTimeout": "15m", + "[go]": { "editor.formatOnSave": true }, + "eslint.workingDirectories": [{ "directory": "frontend", "changeProcessCWD": true }] + } + } + }, + "postCreateCommand": "bash .devcontainer/post-create.sh", + "containerUser": "root", + "remoteUser": "vscode" +} diff --git a/.devcontainer/post-create.sh b/.devcontainer/post-create.sh new file mode 100755 index 0000000..1703a62 --- /dev/null +++ b/.devcontainer/post-create.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +# The docker-in-docker feature pins iptables to the legacy backend, which needs the +# ip_tables/iptable_nat kernel modules. Hosts running an nft-only firewall stack never +# load them, so dockerd dies initializing the bridge driver. Re-point iptables at the +# nft backend and restart the daemon the entrypoint already gave up on. +if ! docker info >/dev/null 2>&1; then + sudo update-alternatives --set iptables /usr/sbin/iptables-nft + sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-nft + sudo pkill dockerd || true + sudo pkill containerd || true + sudo /usr/local/share/docker-init.sh +fi + +sudo chown -R vscode:vscode \ + /workspaces/databasus \ + /home/vscode/go \ + /home/vscode/.cache \ + /home/vscode/.local/share/pnpm + +cd /workspaces/databasus + +cd backend +go mod download +cd .. + +cd frontend +pnpm install --frozen-lockfile +cd .. + +pre-commit install --install-hooks diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..4411c43 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,72 @@ +# Git and GitHub +.git +.gitignore +.github + +# Node modules everywhere +node_modules +**/node_modules + +# Backend - exclude everything except what's needed for build +backend/mysqldata +backend/pgdata +backend/mariadbdata +backend/mongodbdata +backend/temp +backend/images +backend/bin +backend/*.exe + +# Scripts and data directories +scripts +databasus-data + +# IDE and editor files +.idea +.vscode +.cursor +**/*.swp +**/*.swo + +# Documentation and articles (not needed for build) +articles +docs +pages + +# Notifiers not needed in container +notifiers + +# Dist (will be built fresh) +frontend/dist + +# Environment files (handled separately) +.env.local +.env.development + +# Logs and temp files +**/*.log +tmp +temp + +# OS files +.DS_Store +Thumbs.db + +# Helm charts and deployment configs +deploy + +# License and other root files +LICENSE +CITATION.cff +*.md + +# Assets - exclude SVGs and Windows-only tool binaries (used only for native Windows dev, +# never copied into the Docker image — see Dockerfile) +assets/*.svg +assets/tools/win-x64 + +# Python cache +**/__pycache__ + +# Pre-commit config +.pre-commit-config.yaml diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..8a841b1 --- /dev/null +++ b/.env.example @@ -0,0 +1,69 @@ +# app +ENV_MODE=development +SHOW_DB_INSTALLATION_VERIFICATION_LOGS=true + +# dev DB (used by docker-compose.yml and backend) +DEV_DB_NAME=databasus +DEV_DB_USERNAME=postgres +DEV_DB_PASSWORD=Q1234567 +DATABASE_DSN="host=localhost user=postgres password=Q1234567 dbname=databasus port=5437 sslmode=disable" +DATABASE_URL=postgres://postgres:Q1234567@localhost:5437/databasus?sslmode=disable + +# test DB (used by backend when IsTesting=true so tests don't pollute the dev DB) +TEST_DATABASE_DSN="host=localhost user=postgres password=Q1234567 dbname=databasus port=5438 sslmode=disable" + +# migrations +GOOSE_DRIVER=postgres +GOOSE_DBSTRING=postgres://postgres:Q1234567@localhost:5437/databasus?sslmode=disable +GOOSE_TEST_DBSTRING=postgres://postgres:Q1234567@localhost:5438/databasus?sslmode=disable + +# valkey +VALKEY_HOST=127.0.0.1 +VALKEY_PORT=6379 +VALKEY_USERNAME= +VALKEY_PASSWORD= +VALKEY_IS_SSL=false + +# logging +VICTORIA_LOGS_URL= +VICTORIA_LOGS_PASSWORD= + +# tests +TEST_PARALLEL_WORKERS=8 +TEST_LOCALHOST=localhost +TEST_LOGICAL_POSTGRES_16_PORT=5004 +TEST_PHYSICAL_POSTGRES_17_PORT=5007 +TEST_PHYSICAL_POSTGRES_18_PORT=5008 + +# wal-backup (live remote-WAL test rig) +WAL_BACKUP_POSTGRES_18_PORT=5018 +WAL_BACKUP_DB_NAME=testdb +WAL_BACKUP_DB_USER=testuser +WAL_BACKUP_DB_PASSWORD=testpassword + +# oauth +GITHUB_CLIENT_ID= +GITHUB_CLIENT_SECRET= +GOOGLE_CLIENT_ID= +GOOGLE_CLIENT_SECRET= + +# cloudflare turnstile +CLOUDFLARE_TURNSTILE_SITE_KEY= +CLOUDFLARE_TURNSTILE_SECRET_KEY= + +# smtp +SMTP_HOST=test-mailpit +SMTP_PORT=1025 +SMTP_USER=info@databasus.com +SMTP_PASSWORD=password +SMTP_FROM=info@databasus.com +SMTP_INSECURE_SKIP_VERIFY=false + +# domain (used in email links) +DATABASUS_URL= + +# frontend (vite) +VITE_GITHUB_CLIENT_ID= +VITE_GOOGLE_CLIENT_ID= +VITE_IS_EMAIL_CONFIGURED=false +VITE_CLOUDFLARE_TURNSTILE_SITE_KEY= diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..d6e4885 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,18 @@ +# Normalize all text files to LF in the repo, regardless of host OS. +# Prevents CRLF noise when checking out on Windows and mounting into the Linux devcontainer. +* text=auto eol=lf + +# Keep genuinely binary assets untouched. +*.png binary +*.jpg binary +*.jpeg binary +*.gif binary +*.svg text eol=lf +*.ico binary +*.woff binary +*.woff2 binary +*.pdf binary + +# Windows-only scripts must keep CRLF. +*.bat text eol=crlf +*.cmd text eol=crlf diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..5753abd --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,102 @@ +# Code of Conduct + +## Our Pledge + +We as members, contributors and maintainers pledge to make participation in the Databasus community a friendly and welcoming experience for everyone, regardless of background, experience level or personal circumstances. + +We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive and healthy community. + +## Our Standards + +### Examples of behavior that contributes to a positive environment + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members +- Helping newcomers get started with contributions +- Providing clear and constructive feedback on pull requests +- Celebrating successes and acknowledging contributions + +### Examples of unacceptable behavior + +- Trolling, insulting or derogatory comments, and personal or political attacks +- Publishing others' private information, such as physical or email addresses, without their explicit permission +- Spam, self-promotion or off-topic content in project spaces +- Other conduct which could reasonably be considered inappropriate in a professional setting + +## Scope + +This Code of Conduct applies within all community spaces, including: + +- GitHub repositories (issues, pull requests, discussions, comments) +- Telegram channels and direct messages related to Databasus +- Social media interactions when representing the project +- Community forums and online discussions +- Any other spaces where Databasus community members interact + +This Code of Conduct also applies when an individual is officially representing the community in public spaces, such as using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. + +## Enforcement + +Instances of abusive or unacceptable behavior may be reported to the community leaders responsible for enforcement: + +- **Email**: [info@databasus.com](mailto:info@databasus.com) +- **Telegram**: [@rostislav_dugin](https://t.me/rostislav_dugin) + +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of actions. + +**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the community. + +## Contributing with Respect + +When contributing to Databasus, please: + +- Be patient with maintainers and other contributors +- Understand that everyone has different levels of experience +- Ask questions in a respectful manner +- Accept that your contribution may not be accepted, and be open to feedback +- Follow the [contribution guidelines](https://databasus.com/contribute) + +For code contributions, remember to: + +- Discuss significant changes before implementing them +- Be open to code review feedback +- Help review others' contributions when possible + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html](https://www.contributor-covenant.org/version/2/1/code_of_conduct.html). + +Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity). + +For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq). diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 0000000..8a9ef97 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,44 @@ +--- +name: Bug Report +about: Report a bug or unexpected behavior in Databasus +labels: bug +--- + +## Databasus version (screenshot) + +It is displayed in the bottom left corner of the Databasus UI. Please attach screenshot, not just version text + + + +## Operating system and architecture + + + +## Database type and version (optional, for DB-related bugs) + + + +## Describe the bug (please write manually, do not ask AI to summarize) + +**What happened:** + +**What I expected:** + +## Steps to reproduce + +1. +2. +3. + +## Have you asked AI how to solve the issue? + + + +- [ ] Claude Sonnet 4.6 or newer +- [ ] ChatGPT 5.2 or newer +- [ ] No + + +## Additional context / logs + + diff --git a/.github/SECURITY.md b/.github/SECURITY.md new file mode 100644 index 0000000..e02a831 --- /dev/null +++ b/.github/SECURITY.md @@ -0,0 +1,66 @@ +# Security Policy + +## Reporting a Vulnerability + +If you discover a security vulnerability in Databasus, please report it responsibly. **Do not create a public GitHub issue for security vulnerabilities.** + +### How to Report + +1. **Email** (preferred): Send details to [info@databasus.com](mailto:info@databasus.com) +2. **Telegram**: Contact [@rostislav_dugin](https://t.me/rostislav_dugin) +3. **GitHub Security Advisories**: Use the [private vulnerability reporting](https://github.com/databasus/databasus/security/advisories/new) feature + +### What to Include + +- Description of the vulnerability +- Steps to reproduce the issue +- Potential impact and severity assessment +- Any suggested fixes (optional) + +## Supported Versions + +| Version | Supported | +| ------- | --------- | +| Latest | Yes | + +We recommend always using the latest version of Databasus. Security patches are applied to the most recent release. + +### PostgreSQL Compatibility + +Databasus supports PostgreSQL versions 12, 13, 14, 15, 16, 17 and 18. + +### MySQL Compatibility + +Databasus supports MySQL versions 5.7, 8 and 9. + +### MariaDB Compatibility + +Databasus supports MariaDB versions 10 and 11. + +### MongoDB Compatibility + +Databasus supports MongoDB versions 4, 5, 6, 7 and 8. + +## Response Timeline + +- **Acknowledgment**: Within 48-72 hours +- **Initial Assessment**: Within 1 week +- **Fix Timeline**: Depends on severity, but we aim to address critical issues as quickly as possible + +We follow a coordinated disclosure policy. We ask that you give us reasonable time to address the vulnerability before any public disclosure. + +## Security Features + +Databasus is designed with security in mind. For full details, see our [security documentation](https://databasus.com/security). + +Key features include: + +- **AES-256-GCM Encryption**: Enterprise-grade encryption for backup files and sensitive data +- **Read-Only Database Access**: Databasus uses read-only access by default and warns if write permissions are detected +- **Role-Based Access Control**: Assign viewer, member, admin or owner roles within workspaces +- **Audit Logging**: Track all system activities and changes made by users +- **Zero-Trust Storage**: Encrypted backups are safe even in shared cloud storage + +## License + +Databasus is licensed under [Apache 2.0](../LICENSE). \ No newline at end of file diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..a708c51 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,71 @@ +version: 2 + +updates: + - package-ecosystem: "npm" + directory: "/frontend" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 10 + cooldown: + default-days: 7 + semver-major-days: 30 + semver-minor-days: 7 + semver-patch-days: 3 + groups: + frontend-minor-and-patch: + update-types: ["minor", "patch"] + labels: ["dependencies", "frontend"] + commit-message: + prefix: "REFACTOR (deps)" + + - package-ecosystem: "gomod" + directory: "/backend" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 10 + cooldown: + default-days: 7 + semver-major-days: 30 + semver-minor-days: 7 + semver-patch-days: 3 + groups: + backend-minor-and-patch: + update-types: ["minor", "patch"] + labels: ["dependencies", "backend"] + commit-message: + prefix: "REFACTOR (deps)" + + - package-ecosystem: "docker" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + cooldown: + default-days: 14 + semver-major-days: 30 + semver-minor-days: 14 + semver-patch-days: 7 + labels: ["dependencies", "docker"] + commit-message: + prefix: "REFACTOR (deps)" + + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + cooldown: + default-days: 7 + semver-major-days: 30 + semver-minor-days: 7 + semver-patch-days: 3 + groups: + actions-minor-and-patch: + update-types: ["minor", "patch"] + labels: ["dependencies", "ci"] + commit-message: + prefix: "REFACTOR (deps)" diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.yml new file mode 100644 index 0000000..51442c8 --- /dev/null +++ b/.github/workflows/ci-release.yml @@ -0,0 +1,1047 @@ +name: CI and Release + +on: + push: + branches: ["**"] + pull_request: + branches: ["**"] + workflow_dispatch: + +permissions: + contents: read + +jobs: + lint-backend: + if: github.ref != 'refs/heads/develop' + runs-on: self-hosted + container: + image: golang:1.26.3 + volumes: + - /runner-cache/go-pkg:/go/pkg/mod + - /runner-cache/go-build:/root/.cache/go-build + - /runner-cache/golangci-lint:/root/.cache/golangci-lint + - /runner-cache/apt-archives:/var/cache/apt/archives + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Configure Git for container + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Download Go modules + run: | + cd backend + go mod download + + - name: Install golangci-lint + run: | + go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.11.3 + echo "$(go env GOPATH)/bin" >> $GITHUB_PATH + + - name: Install swag for swagger generation + run: go install github.com/swaggo/swag/cmd/swag@v1.16.4 + + - name: Generate swagger docs + run: | + cd backend + swag init -d . -g cmd/main.go -o swagger + + - name: Run golangci-lint + run: | + cd backend + golangci-lint run + + - name: Verify go mod tidy + run: | + cd backend + go mod tidy + git diff --exit-code go.mod go.sum || (echo "go mod tidy made changes, please run 'go mod tidy' and commit the changes" && exit 1) + + lint-frontend: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up pnpm + uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + with: + package_json_file: frontend/package.json + + - name: Set up Node.js + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: "20" + cache: "pnpm" + cache-dependency-path: frontend/pnpm-lock.yaml + + - name: Install dependencies + run: | + cd frontend + pnpm install --frozen-lockfile + + - name: Check if prettier was run + run: | + cd frontend + pnpm format + git diff --exit-code || (echo "Prettier made changes, please run 'pnpm format' and commit the changes" && exit 1) + + - name: Check if linter was run + run: | + cd frontend + pnpm lint + + - name: Build frontend + run: | + cd frontend + pnpm build + + dockerfile-scan: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Install Trivy + run: | + sudo apt-get update + sudo apt-get install -y wget apt-transport-https gnupg + wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --batch --no-tty --yes --dearmor -o /usr/share/keyrings/trivy.gpg + echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list + sudo apt-get update + sudo apt-get install -y trivy + + - name: Trivy Dockerfile misconfiguration scan + run: | + trivy config --severity HIGH,CRITICAL --exit-code 1 Dockerfile + trivy config --severity HIGH,CRITICAL --exit-code 1 agent/verification/images/postgres/Dockerfile + + test-frontend: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + needs: [lint-frontend] + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up pnpm + uses: pnpm/action-setup@fc06bc1257f339d1d5d8b3a19a8cae5388b55320 # v4.4.0 + with: + package_json_file: frontend/package.json + + - name: Set up Node.js + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: "20" + cache: "pnpm" + cache-dependency-path: frontend/pnpm-lock.yaml + + - name: Install dependencies + run: | + cd frontend + pnpm install --frozen-lockfile + + - name: Run frontend tests + run: | + cd frontend + pnpm test + + lint-verification-agent: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Go + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 + with: + go-version: "1.26.3" + cache-dependency-path: agent/verification/go.sum + + - name: Download Go modules + run: | + cd agent/verification + go mod download + + - name: Install golangci-lint + run: | + go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.11.3 + echo "$(go env GOPATH)/bin" >> $GITHUB_PATH + + - name: Run golangci-lint + run: | + cd agent/verification + golangci-lint run + + - name: Verify go mod tidy + run: | + cd agent/verification + go mod tidy + git diff --exit-code go.mod go.sum || (echo "go mod tidy made changes, please run 'go mod tidy' and commit the changes" && exit 1) + + test-verification-agent: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + needs: [lint-verification-agent] + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Go + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 + with: + go-version: "1.26.3" + cache-dependency-path: agent/verification/go.sum + + - name: Download Go modules + run: | + cd agent/verification + go mod download + + - name: Run Go tests + run: | + cd agent/verification + go test -race -count=1 -failfast ./internal/... + + e2e-verification-agent: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + needs: [lint-verification-agent] + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Run e2e tests + run: | + cd agent/verification + make e2e + + - name: Cleanup + if: always() + run: | + cd agent/verification/e2e + docker compose down -v --rmi local || true + rm -rf artifacts || true + + # Self-hosted: performant high-frequency CPU is used to start many containers and run tests fast. Tests + # step is bottle-neck, because we need a lot of containers and cannot parallelize tests due to shared resources + test-backend: + if: github.ref != 'refs/heads/develop' + runs-on: self-hosted + needs: [lint-backend] + container: + image: golang:1.26.3 + options: --privileged -v /var/run/docker.sock:/var/run/docker.sock --add-host=host.docker.internal:host-gateway + volumes: + - /runner-cache/go-pkg:/go/pkg/mod + - /runner-cache/go-build:/root/.cache/go-build + - /runner-cache/apt-archives:/var/cache/apt/archives + # Identity-mapped scratch dir: physical restore/boot tests bind their t.TempDir() into + # sibling containers (recovery script's pg_combinebackup, bound-volume boot). The host + # daemon resolves bind sources, so the path must exist identically inside and on the host. + - /runner-cache/test-tmp:/runner-cache/test-tmp + steps: + - name: Install Docker CLI + run: | + apt-get update -qq + apt-get install -y -qq docker.io docker-compose netcat-openbsd wget + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Configure Git for container + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Download Go modules + run: | + cd backend + go mod download + + - name: Create .env file for testing + run: | + cp .env.example .env + cat >> .env << EOF + # CI overrides — host=172.17.0.1 reaches host from container + DATABASE_DSN="host=172.17.0.1 user=postgres password=Q1234567 dbname=databasus port=5437 sslmode=disable" + DATABASE_URL=postgres://postgres:Q1234567@172.17.0.1:5437/databasus?sslmode=disable + GOOSE_DBSTRING=postgres://postgres:Q1234567@172.17.0.1:5437/databasus?sslmode=disable + TEST_DATABASE_DSN="host=172.17.0.1 user=postgres password=Q1234567 dbname=databasus port=5438 sslmode=disable" + GOOSE_TEST_DBSTRING=postgres://postgres:Q1234567@172.17.0.1:5438/databasus?sslmode=disable + TEST_LOCALHOST=172.17.0.1 + TEST_DB_HOST=172.17.0.1 + VALKEY_HOST=172.17.0.1 + EOF + + - name: Remove leftover testcontainers from previous runs + run: | + # Self-hosted runners are not ephemeral, so a cancelled job or crashed runner + # leaves Ryuk-less testcontainers behind (Ryuk is disabled below). Clear them + # before starting so a stale container can't collide with this run. Only + # org.testcontainers-labelled ones match — compose services are untouched. + docker rm -f $(docker ps -aq --filter label=org.testcontainers=true) 2>/dev/null || true + # Same non-ephemeral-runner reasoning: clear stale per-run reconstruction dirs from the + # identity-mapped scratch volume so a crashed run can't collide with this one. + rm -rf /runner-cache/test-tmp/* 2>/dev/null || true + + - name: Start test containers + run: | + docker compose -f docker-compose.yml up -d + + - name: Wait for containers to be ready + run: | + # Wait for main dev database (postgres listens on 5432 inside the container) + timeout 60 bash -c 'until docker exec backend-db pg_isready -U postgres; do sleep 2; done' + + # Wait for test database (postgres listens on 5432 inside the container) + timeout 60 bash -c 'until docker exec backend-db-test pg_isready -U postgres; do sleep 2; done' + + # Wait for Valkey (cache) + echo "Waiting for Valkey..." + timeout 60 bash -c 'until docker exec backend-valkey valkey-cli ping 2>/dev/null | grep -q PONG; do sleep 2; done' + echo "Valkey is ready!" + + # Wait for test databases (using 172.17.0.1 from container). + # 5004 = shared logical-postgres-16 fixture; 5007-5008 = shared physical-postgres 17/18 + # primary sources. The per-version, no-summary, tablespace, mTLS and restore-target + # physical containers — like the per-version logical PG tests — run on testcontainers and + # need no fixed port. + timeout 60 bash -c 'until nc -z 172.17.0.1 5004; do sleep 2; done' + timeout 60 bash -c 'until nc -z 172.17.0.1 5007; do sleep 2; done' + timeout 60 bash -c 'until nc -z 172.17.0.1 5008; do sleep 2; done' + + - name: Create data and temp directories + run: | + # Create directories that are used for backups and restore + # These paths match what's configured in config.go + mkdir -p databasus-data/backups + mkdir -p databasus-data/temp + # Scratch dir for $TMPDIR below; identity-mapped so sibling-container binds resolve. + mkdir -p /runner-cache/test-tmp + + - name: Install database client dependencies + run: | + apt-get update -qq + apt-get install -y -qq libncurses6 libpq5 zstd + ln -sf /usr/lib/x86_64-linux-gnu/libncurses.so.6 /usr/lib/x86_64-linux-gnu/libncurses.so.5 || true + ln -sf /usr/lib/x86_64-linux-gnu/libtinfo.so.6 /usr/lib/x86_64-linux-gnu/libtinfo.so.5 || true + + - name: Provision per-worker test databases and run migrations + env: + TEST_PARALLEL_WORKERS: "8" + run: | + cd backend + go install github.com/pressly/goose/v3/cmd/goose@v3.27.1 + set -a && . ../.env && set +a + # cleanup_test_db creates databasus_w0..w{workers-1}, flushes Valkey, and + # resets the test PG containers. Then migrate each slot DB so every + # parallel `go test` worker connects to a migrated metadata DB. + go run ./cmd/cleanup_test_db + for i in $(seq 0 $((TEST_PARALLEL_WORKERS - 1))); do + dbstring=$(echo "$GOOSE_TEST_DBSTRING" | sed -E "s#/([^/?]+)\?#/\1_w$i?#") + echo "migrating slot $i" + GOOSE_DBSTRING="$dbstring" goose -dir ./migrations up + done + + - name: Pull testcontainer images + run: make -C backend pull-testcontainers + + - name: Run Go tests + env: + # MongoDB test containers are started in-process via testcontainers-go. + # Their published ports live on the Docker host bridge (reachable at the same + # address compose used), so override the host testcontainers reports. Ryuk is + # disabled because its callback port is unreliable from this sibling-container + # runner; cleanup is handled by t.Cleanup plus the label-filter step below. + TESTCONTAINERS_HOST_OVERRIDE: 172.17.0.1 + TESTCONTAINERS_RYUK_DISABLED: "true" + # t.TempDir() honors $TMPDIR. Point it at the identity-mapped scratch volume so the + # physical restore/boot tests' bind sources exist at the same path on the host daemon. + TMPDIR: /runner-cache/test-tmp + # Each package runs in its own metadata DB + Valkey logical DB, so + # packages run in parallel; -p must equal the provisioned worker count. + TEST_PARALLEL_WORKERS: "8" + run: | + cd backend + go test -p=8 -count=1 -failfast -timeout 15m ./internal/... + + - name: Remove leftover testcontainers + if: always() + run: | + # Safety net: t.Cleanup terminates MongoDB testcontainers, but a panicking + # test can skip it. Remove anything testcontainers-go labelled. + docker rm -f $(docker ps -aq --filter label=org.testcontainers=true) 2>/dev/null || true + # Safety net for the scratch volume: t.TempDir() self-cleans, but a panic can skip it. + rm -rf /runner-cache/test-tmp/* 2>/dev/null || true + + - name: Stop test containers + if: always() + run: | + # Stop and remove containers (keeping images for next run) + docker compose -f docker-compose.yml down -v + + # Clean up all data directories created by docker-compose + echo "Cleaning up data directories..." + rm -rf pgdata || true + rm -rf pgdata-test || true + rm -rf valkey-data || true + rm -rf temp/nas || true + rm -rf databasus-data || true + + echo "Cleanup complete" + + build-dev-image: + runs-on: self-hosted + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/develop' }} + steps: + - name: Clean workspace + run: | + sudo rm -rf "$GITHUB_WORKSPACE"/* || true + sudo rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up QEMU (enables multi-arch emulation) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - name: Build amd64 dev image and load locally + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: . + push: false + load: true + platforms: linux/amd64 + cache-to: type=gha,mode=min,scope=dev-build + cache-from: type=gha,scope=dev-build + build-args: | + APP_VERSION=dev-${{ github.sha }} + tags: | + databasus-dev:smoke-${{ github.sha }} + + - name: Run smoke pings against dev image + run: | + mkdir -p /tmp/databasus-dev-smoke-data + docker run -d --name databasus-dev-smoke \ + -p 4005:4005 \ + -v /tmp/databasus-dev-smoke-data:/databasus-data \ + databasus-dev:smoke-${{ github.sha }} + + for i in $(seq 1 60); do + health_body=$(mktemp) + code=$(curl -s -o "$health_body" -w '%{http_code}' http://localhost:4005/api/v1/system/health || true) + if [ "$code" = "200" ] && grep -q '"status"' "$health_body"; then + echo "healthcheck OK after ${i} attempt(s)" + rm -f "$health_body" + break + fi + rm -f "$health_body" + if [ "$i" -eq 60 ]; then + echo "healthcheck never returned valid JSON (last code: ${code})" + docker logs databasus-dev-smoke + exit 1 + fi + sleep 2 + done + + body=$(curl -sf http://localhost:4005/) + if ! echo "$body" | grep -q 'id="root"'; then + echo "index.html did not contain React root div" + echo "$body" + exit 1 + fi + echo "index.html OK" + + for arch in amd64 arm64; do + echo "Checking verification agent download for $arch..." + out=$(mktemp) + http=$(curl -sS -o "$out" -w '%{http_code}' "http://localhost:4005/api/v1/system/verification-agent?arch=$arch") + if [ "$http" != "200" ]; then + echo "verification agent ($arch) download failed: HTTP $http" + head -c 200 "$out" || true + docker logs databasus-dev-smoke + rm -f "$out" + exit 1 + fi + magic=$(head -c 4 "$out" | od -An -t x1 | tr -d ' ') + if [ "$magic" != "7f454c46" ]; then + echo "verification agent ($arch) is not an ELF binary (first 4 bytes: $magic)" + head -c 200 "$out" || true + docker logs databasus-dev-smoke + rm -f "$out" + exit 1 + fi + size=$(stat -c%s "$out") + echo "verification agent ($arch) OK (${size} bytes)" + rm -f "$out" + done + + - name: Install Trivy + run: | + sudo apt-get update + sudo apt-get install -y wget apt-transport-https gnupg + wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --batch --no-tty --yes --dearmor -o /usr/share/keyrings/trivy.gpg + echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list + sudo apt-get update + sudo apt-get install -y trivy + + - name: Trivy scan dev image (CRITICAL, blocking) + run: trivy image --severity CRITICAL --ignore-unfixed --pkg-types os,library --scanners vuln --exit-code 1 databasus-dev:smoke-${{ github.sha }} + + - name: Trivy scan dev image (HIGH, advisory) + if: always() + run: trivy image --severity HIGH --ignore-unfixed --pkg-types os,library --scanners vuln --exit-code 0 databasus-dev:smoke-${{ github.sha }} + + - name: Cleanup smoke container + if: always() + run: | + docker rm -f databasus-dev-smoke || true + docker rmi databasus-dev:smoke-${{ github.sha }} || true + rm -rf /tmp/databasus-dev-smoke-data || true + + push-dev-image: + runs-on: self-hosted + needs: [build-dev-image] + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/develop' }} + steps: + - name: Clean workspace + run: | + sudo rm -rf "$GITHUB_WORKSPACE"/* || true + sudo rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up QEMU (enables multi-arch emulation) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - name: Log in to Docker Hub + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Build and push dev image + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: . + push: true + platforms: linux/amd64,linux/arm64 + cache-from: type=gha,scope=dev-build + build-args: | + APP_VERSION=dev-${{ github.sha }} + tags: | + databasus/databasus-dev:latest + databasus/databasus-dev:${{ github.sha }} + + build-image: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + needs: [lint-backend, lint-frontend, lint-verification-agent, dockerfile-scan] + steps: + - name: Clean workspace + run: | + sudo rm -rf "$GITHUB_WORKSPACE"/* || true + sudo rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up QEMU (enables multi-arch emulation) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - name: Build amd64 image and load locally + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: . + push: false + load: true + platforms: linux/amd64 + cache-to: type=gha,mode=max,scope=release-build + cache-from: type=gha,scope=release-build + build-args: | + APP_VERSION=ci-${{ github.sha }} + tags: | + databasus:smoke-${{ github.sha }} + + - name: Run smoke pings against built image + run: | + mkdir -p /tmp/databasus-smoke-data + docker run -d --name databasus-smoke \ + -p 4005:4005 \ + -v /tmp/databasus-smoke-data:/databasus-data \ + databasus:smoke-${{ github.sha }} + + for i in $(seq 1 60); do + health_body=$(mktemp) + code=$(curl -s -o "$health_body" -w '%{http_code}' http://localhost:4005/api/v1/system/health || true) + if [ "$code" = "200" ] && grep -q '"status"' "$health_body"; then + echo "healthcheck OK after ${i} attempt(s)" + rm -f "$health_body" + break + fi + rm -f "$health_body" + if [ "$i" -eq 60 ]; then + echo "healthcheck never returned valid JSON (last code: ${code})" + docker logs databasus-smoke + exit 1 + fi + sleep 2 + done + + body=$(curl -sf http://localhost:4005/) + if ! echo "$body" | grep -q 'id="root"'; then + echo "index.html did not contain React root div" + echo "$body" + exit 1 + fi + echo "index.html OK" + + for arch in amd64 arm64; do + echo "Checking verification agent download for $arch..." + out=$(mktemp) + http=$(curl -sS -o "$out" -w '%{http_code}' "http://localhost:4005/api/v1/system/verification-agent?arch=$arch") + if [ "$http" != "200" ]; then + echo "verification agent ($arch) download failed: HTTP $http" + head -c 200 "$out" || true + docker logs databasus-smoke + rm -f "$out" + exit 1 + fi + magic=$(head -c 4 "$out" | od -An -t x1 | tr -d ' ') + if [ "$magic" != "7f454c46" ]; then + echo "verification agent ($arch) is not an ELF binary (first 4 bytes: $magic)" + head -c 200 "$out" || true + docker logs databasus-smoke + rm -f "$out" + exit 1 + fi + size=$(stat -c%s "$out") + echo "verification agent ($arch) OK (${size} bytes)" + rm -f "$out" + done + + - name: Install Trivy + run: | + sudo apt-get update + sudo apt-get install -y wget apt-transport-https gnupg + wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --batch --no-tty --yes --dearmor -o /usr/share/keyrings/trivy.gpg + echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | sudo tee /etc/apt/sources.list.d/trivy.list + sudo apt-get update + sudo apt-get install -y trivy + + - name: Trivy scan release image (CRITICAL, blocking) + run: trivy image --severity CRITICAL --ignore-unfixed --pkg-types os,library --scanners vuln --exit-code 1 databasus:smoke-${{ github.sha }} + + - name: Trivy scan release image (HIGH, advisory) + if: always() + run: trivy image --severity HIGH --ignore-unfixed --pkg-types os,library --scanners vuln --exit-code 0 databasus:smoke-${{ github.sha }} + + - name: Cleanup smoke container + if: always() + run: | + docker rm -f databasus-smoke || true + docker rmi databasus:smoke-${{ github.sha }} || true + rm -rf /tmp/databasus-smoke-data || true + + build-verification-image: + if: github.ref != 'refs/heads/develop' + runs-on: ubuntu-latest + needs: [lint-verification-agent, dockerfile-scan] + permissions: + contents: read + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + # Validate the extension Dockerfile builds (amd64, representative major) on + # every PR; the full multi-arch matrix is built at release time below. + - name: Build verification image (amd64, no push) + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: agent/verification/images/postgres + push: false + platforms: linux/amd64 + build-args: | + PG_MAJOR=16 + cache-from: type=gha,scope=verification-postgres-16 + cache-to: type=gha,mode=min,scope=verification-postgres-16 + + determine-version: + runs-on: self-hosted + container: + image: node:20 + needs: [test-backend, test-frontend, test-verification-agent, e2e-verification-agent] + if: ${{ github.ref == 'refs/heads/main' && !contains(github.event.head_commit.message, '[skip-release]') }} + outputs: + should_release: ${{ steps.version_bump.outputs.should_release }} + new_version: ${{ steps.version_bump.outputs.new_version }} + bump_type: ${{ steps.version_bump.outputs.bump_type }} + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + fetch-depth: 0 + + - name: Configure Git for container + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Install semver + run: npm install -g semver + + - name: Get current version + id: current_version + run: | + LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0") + echo "current_version=${LATEST_TAG#v}" >> $GITHUB_OUTPUT + echo "Current version: ${LATEST_TAG#v}" + + - name: Analyze commits and determine version bump + id: version_bump + shell: bash + run: | + CURRENT_VERSION="${{ steps.current_version.outputs.current_version }}" + LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0") + + # Get commits since last tag + if [ "$LATEST_TAG" = "v0.0.0" ]; then + COMMITS=$(git log --pretty=format:"%s" --no-merges) + else + COMMITS=$(git log ${LATEST_TAG}..HEAD --pretty=format:"%s" --no-merges) + fi + + echo "Analyzing commits:" + echo "$COMMITS" + + # Initialize flags + HAS_FEATURE=false + HAS_FIX=false + HAS_BREAKING=false + + # Analyze each commit - USE PROCESS SUBSTITUTION to avoid subshell variable scope issues + while IFS= read -r commit; do + if [[ "$commit" =~ ^FEATURE ]]; then + HAS_FEATURE=true + echo "Found FEATURE commit: $commit" + elif [[ "$commit" =~ ^FIX ]]; then + HAS_FIX=true + echo "Found FIX commit: $commit" + elif [[ "$commit" =~ ^REFACTOR ]]; then + HAS_FIX=true # Treat refactor as patch + echo "Found REFACTOR commit: $commit" + fi + + # Check for breaking changes + if [[ "$commit" =~ BREAKING[[:space:]]CHANGE ]] || [[ "$commit" =~ "!" ]]; then + HAS_BREAKING=true + echo "Found BREAKING CHANGE: $commit" + fi + done < <(printf '%s\n' "$COMMITS") + + # Determine version bump + if [ "$HAS_BREAKING" = true ]; then + BUMP_TYPE="major" + elif [ "$HAS_FEATURE" = true ]; then + BUMP_TYPE="minor" + elif [ "$HAS_FIX" = true ]; then + BUMP_TYPE="patch" + else + BUMP_TYPE="none" + fi + + echo "bump_type=$BUMP_TYPE" >> $GITHUB_OUTPUT + + if [ "$BUMP_TYPE" != "none" ]; then + NEW_VERSION=$(npx semver -i $BUMP_TYPE $CURRENT_VERSION) + echo "new_version=$NEW_VERSION" >> $GITHUB_OUTPUT + echo "should_release=true" >> $GITHUB_OUTPUT + echo "New version will be: $NEW_VERSION" + else + echo "should_release=false" >> $GITHUB_OUTPUT + echo "No version bump needed" + fi + + push-image: + runs-on: ubuntu-latest + needs: [determine-version, build-image] + if: ${{ needs.determine-version.outputs.should_release == 'true' }} + permissions: + contents: write + steps: + - name: Clean workspace + run: | + sudo rm -rf "$GITHUB_WORKSPACE"/* || true + sudo rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up QEMU (enables multi-arch emulation) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - name: Log in to Docker Hub + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Build and push with version tags + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: . + push: true + platforms: linux/amd64,linux/arm64 + cache-from: type=gha,scope=release-build + build-args: | + APP_VERSION=${{ needs.determine-version.outputs.new_version }} + tags: | + databasus/databasus:latest + databasus/databasus:v${{ needs.determine-version.outputs.new_version }} + databasus/databasus:${{ github.sha }} + + push-verification-image: + runs-on: ubuntu-latest + needs: [determine-version, build-verification-image] + if: ${{ needs.determine-version.outputs.should_release == 'true' }} + permissions: + contents: read + strategy: + fail-fast: false + matrix: + major: ["12", "13", "14", "15", "16", "17", "18"] + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up QEMU (enables multi-arch emulation) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0 + + - name: Log in to Docker Hub + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + # Pin the base to its current manifest digest so the published image is + # reproducible rather than tracking a floating postgres: tag. + - name: Resolve digest-pinned base image + id: base + run: | + DIGEST=$(docker buildx imagetools inspect postgres:${{ matrix.major }} --format '{{.Manifest.Digest}}') + echo "image=postgres:${{ matrix.major }}@${DIGEST}" >> "$GITHUB_OUTPUT" + + - name: Build and push verification image + uses: docker/build-push-action@ca052bb54ab0790a636c9b5f226502c73d547a25 # v5.4.0 + with: + context: agent/verification/images/postgres + push: true + platforms: linux/amd64,linux/arm64 + build-args: | + PG_MAJOR=${{ matrix.major }} + BASE_IMAGE=${{ steps.base.outputs.image }} + tags: | + databasus/verification-postgres:${{ matrix.major }} + databasus/verification-postgres:${{ matrix.major }}-v${{ needs.determine-version.outputs.new_version }} + + release: + runs-on: self-hosted + container: + image: node:20 + needs: [determine-version, push-image] + if: ${{ needs.determine-version.outputs.should_release == 'true' }} + permissions: + contents: write + pull-requests: write + steps: + - name: Clean workspace + run: | + rm -rf "$GITHUB_WORKSPACE"/* || true + rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Configure Git for container + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Generate changelog + id: changelog + shell: bash + run: | + NEW_VERSION="${{ needs.determine-version.outputs.new_version }}" + LATEST_TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "v0.0.0") + + # Get commits since last tag + if [ "$LATEST_TAG" = "v0.0.0" ]; then + COMMITS=$(git log --pretty=format:"%s|%H|%an|%ad" --date=short --no-merges) + else + COMMITS=$(git log ${LATEST_TAG}..HEAD --pretty=format:"%s|%H|%an|%ad" --date=short --no-merges) + fi + + # Build the changelog with real newlines (NL) instead of \n literals + # so we don't have to round-trip through `echo -e`. echo -e interprets + # backslash sequences inside commit subjects (e.g. "\c" suppresses + # further output) which can silently truncate the changelog and break + # the GitHub Actions multiline-output heredoc parser. + NL=$'\n' + CHANGELOG="# Changelog${NL}${NL}## [${NEW_VERSION}] - $(date +%Y-%m-%d)${NL}${NL}" + + # Group commits by type and area + FEATURES="" + FIXES="" + REFACTORS="" + + # USE PROCESS SUBSTITUTION to avoid subshell variable scope issues + while IFS= read -r line; do + if [ -n "$line" ]; then + COMMIT_MSG=$(echo "$line" | cut -d'|' -f1) + COMMIT_HASH=$(echo "$line" | cut -d'|' -f2) + SHORT_HASH=${COMMIT_HASH:0:7} + + # Parse commit message format: TYPE (area): description + if [[ "$COMMIT_MSG" == FEATURE* ]]; then + TEMP="${COMMIT_MSG#FEATURE}" + TEMP="${TEMP#"${TEMP%%[![:space:]]*}"}" + if [[ "$TEMP" == \(* ]]; then + AREA=$(echo "$TEMP" | sed 's/^(\([^)]*\)).*/\1/') + DESC=$(echo "$TEMP" | sed 's/^([^)]*):[[:space:]]*//') + FEATURES="${FEATURES}- **${AREA}**: ${DESC} ([${SHORT_HASH}](https://github.com/${{ github.repository }}/commit/${COMMIT_HASH}))${NL}" + fi + elif [[ "$COMMIT_MSG" == FIX* ]]; then + TEMP="${COMMIT_MSG#FIX}" + TEMP="${TEMP#"${TEMP%%[![:space:]]*}"}" + if [[ "$TEMP" == \(* ]]; then + AREA=$(echo "$TEMP" | sed 's/^(\([^)]*\)).*/\1/') + DESC=$(echo "$TEMP" | sed 's/^([^)]*):[[:space:]]*//') + FIXES="${FIXES}- **${AREA}**: ${DESC} ([${SHORT_HASH}](https://github.com/${{ github.repository }}/commit/${COMMIT_HASH}))${NL}" + fi + elif [[ "$COMMIT_MSG" == REFACTOR* ]]; then + TEMP="${COMMIT_MSG#REFACTOR}" + TEMP="${TEMP#"${TEMP%%[![:space:]]*}"}" + if [[ "$TEMP" == \(* ]]; then + AREA=$(echo "$TEMP" | sed 's/^(\([^)]*\)).*/\1/') + DESC=$(echo "$TEMP" | sed 's/^([^)]*):[[:space:]]*//') + REFACTORS="${REFACTORS}- **${AREA}**: ${DESC} ([${SHORT_HASH}](https://github.com/${{ github.repository }}/commit/${COMMIT_HASH}))${NL}" + fi + fi + fi + done < <(printf '%s\n' "$COMMITS") + + # Build changelog sections + if [ -n "$FEATURES" ]; then + CHANGELOG="${CHANGELOG}### ✨ Features${NL}${FEATURES}${NL}" + fi + + if [ -n "$FIXES" ]; then + CHANGELOG="${CHANGELOG}### 🐛 Bug Fixes${NL}${FIXES}${NL}" + fi + + if [ -n "$REFACTORS" ]; then + CHANGELOG="${CHANGELOG}### 🔨 Refactoring${NL}${REFACTORS}${NL}" + fi + + # Add Docker image info + CHANGELOG="${CHANGELOG}### 🐳 Docker${NL}" + CHANGELOG="${CHANGELOG}- **Image**: \`databasus/databasus:v${NEW_VERSION}\`${NL}" + CHANGELOG="${CHANGELOG}- **Platforms**: linux/amd64, linux/arm64${NL}" + + # Use a random delimiter so any literal "EOF" line inside the + # changelog (e.g. inside a commit subject) cannot prematurely + # terminate the multiline output heredoc. + DELIMITER="EOF_$(openssl rand -hex 16)" + { + printf '%s<<%s\n' 'changelog' "$DELIMITER" + printf '%s\n' "$CHANGELOG" + printf '%s\n' "$DELIMITER" + } >> "$GITHUB_OUTPUT" + + - name: Create GitHub Release + uses: actions/create-release@0cb9c9b65d5d1901c1f53e5e66eaf4afd303e70e # v1.1.4 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + with: + tag_name: v${{ needs.determine-version.outputs.new_version }} + release_name: Release v${{ needs.determine-version.outputs.new_version }} + body: ${{ steps.changelog.outputs.changelog }} + draft: false + prerelease: false + + publish-helm-chart: + runs-on: self-hosted + container: + image: alpine:3.19 + volumes: + - /runner-cache/apk-cache:/etc/apk/cache + needs: [determine-version, push-image] + if: ${{ needs.determine-version.outputs.should_release == 'true' }} + permissions: + contents: read + packages: write + steps: + - name: Clean workspace + run: | + rm -rf "$GITHUB_WORKSPACE"/* || true + rm -rf "$GITHUB_WORKSPACE"/.* || true + + - name: Install dependencies + run: | + apk add --no-cache git bash curl + + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Configure Git for container + run: | + git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Set up Helm + uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1 + with: + version: v3.14.0 + + - name: Log in to GHCR + run: echo "${{ secrets.GITHUB_TOKEN }}" | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin + + - name: Update Chart.yaml with release version + run: | + VERSION="${{ needs.determine-version.outputs.new_version }}" + sed -i "s/^version: .*/version: ${VERSION}/" deploy/helm/Chart.yaml + sed -i "s/^appVersion: .*/appVersion: \"v${VERSION}\"/" deploy/helm/Chart.yaml + cat deploy/helm/Chart.yaml + + - name: Package Helm chart + run: helm package deploy/helm --destination . + + - name: Push Helm chart to GHCR + run: | + VERSION="${{ needs.determine-version.outputs.new_version }}" + helm push databasus-${VERSION}.tgz oci://ghcr.io/databasus/charts diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..b3e674e --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,60 @@ +name: CodeQL + +on: + push: + branches: [main] + pull_request: + branches: ["**"] + schedule: + # CodeQL query packs are updated by GitHub over time; a scheduled scan + # surfaces new findings even when no code has changed. + - cron: "0 6 * * 1" + +permissions: + contents: read + +concurrency: + group: codeql-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + analyze: + name: Analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + permissions: + security-events: write + actions: read + contents: read + + strategy: + fail-fast: false + matrix: + include: + - language: go + build-mode: autobuild + - language: javascript-typescript + build-mode: none + - language: actions + build-mode: none + + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Go + if: matrix.language == 'go' + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 + with: + go-version: "1.26.3" + + - name: Initialize CodeQL + uses: github/codeql-action/init@68bde559dea0fdcac2102bfdf6230c5f70eb485e # v4.35.4 + with: + languages: ${{ matrix.language }} + build-mode: ${{ matrix.build-mode }} + queries: security-extended + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@68bde559dea0fdcac2102bfdf6230c5f70eb485e # v4.35.4 + with: + category: "/language:${{ matrix.language }}" diff --git a/.github/workflows/dependency-review.yml b/.github/workflows/dependency-review.yml new file mode 100644 index 0000000..88f5602 --- /dev/null +++ b/.github/workflows/dependency-review.yml @@ -0,0 +1,22 @@ +name: Dependency Review + +on: + pull_request: + branches: ["**"] + +permissions: + contents: read + pull-requests: write + +jobs: + dependency-review: + runs-on: ubuntu-latest + steps: + - name: Check out code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Dependency Review + uses: actions/dependency-review-action@da24556b548a50705dd671f47852072ea4c105d9 # v4.7.1 + with: + fail-on-severity: high + comment-summary-in-pr: on-failure diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6500501 --- /dev/null +++ b/.gitignore @@ -0,0 +1,26 @@ +ansible/ +databasus-data/ +.env +pgdata/ +pgdata-test/ +mysqldata/ +mariadbdata/ +valkey-data/ +victoria-logs-data/ +wal-queue/ +temp/ +node_modules/ +.idea +/articles + +.DS_Store +/scripts +.vscode/settings.json +.claude/* +!.claude/agents/ +!.claude/hooks/ +!.claude/settings.json +backend/cleanup_test_db.exe +plans/ + +.pnpm-store diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..da9ab96 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,44 @@ +# Pre-commit configuration +# See https://pre-commit.com for more information +repos: + # Frontend checks + - repo: local + hooks: + - id: frontend-format + name: Frontend Format (Prettier) + entry: bash -c "cd frontend && npm run format" + language: system + files: ^frontend/.*\.(ts|tsx|js|jsx|json|css|md)$ + pass_filenames: false + + - id: frontend-lint + name: Frontend Lint (ESLint) + entry: bash -c "cd frontend && npm run lint" + language: system + files: ^frontend/.*\.(ts|tsx|js|jsx)$ + pass_filenames: false + + - id: frontend-build + name: Frontend Build + entry: bash -c "cd frontend && npm run build" + language: system + files: ^frontend/.*\.(ts|tsx|js|jsx|json|css)$ + pass_filenames: false + + # Backend checks + - repo: local + hooks: + - id: backend-format-and-lint + name: Backend Format & Lint (golangci-lint) + entry: bash -c "cd backend && golangci-lint fmt ./internal/... ./cmd/... && golangci-lint run ./internal/... ./cmd/..." + language: system + files: ^backend/.*\.go$ + pass_filenames: false + + - id: backend-go-mod-tidy + name: Backend Go Mod Tidy + entry: bash -c "cd backend && go mod tidy" + language: system + files: ^backend/.*\.go$ + pass_filenames: false + diff --git a/.trivyignore b/.trivyignore new file mode 100644 index 0000000..5a30622 --- /dev/null +++ b/.trivyignore @@ -0,0 +1,16 @@ +# Trivy misconfiguration findings ignored for this repo. +# See: https://aquasecurity.github.io/trivy/latest/docs/configuration/filtering/ +# +# DS-0002 — "Specify at least 1 USER command in Dockerfile with non-root user as argument" +# +# Suppressed for `Dockerfile` because the structural check is a false +# positive for this image: +# - The final stage creates two non-root users (postgres UID 999, +# databasus UID 65532) and the main app process drops to non-root +# via `exec gosu databasus ./main` at the end of start.sh. +# - The entrypoint must start as root to: +# * Remap PUID/PGID at runtime (NAS / Synology / Unraid use case). +# * chown the mounted /databasus-data volume on first boot. +# * Run `initdb` on the embedded PostgreSQL data dir on first boot. +# - Adding a real `USER databasus` directive would break boot. +DS-0002 diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000..239df4d --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,36 @@ +cff-version: 1.2.0 +title: Databasus +message: "If you use this software, please cite it as below." +type: software +authors: + - family-names: Dugin + given-names: Rostislav +repository-code: https://github.com/databasus/databasus +url: https://databasus.com +abstract: "Free, open source and self-hosted solution for automated databases backups with multiple storage options and notifications." +keywords: + - docker + - kubernetes + - golang + - backups + - postgres + - devops + - backup + - database + - tools + - monitoring + - ftp + - postgresql + - s3 + - psql + - web-ui + - self-hosted + - pg + - system-administration + - database-backup + - mysql + - mongodb + - mariadb +license: Apache-2.0 +version: 2.21.0 +date-released: "2026-01-05" diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..1de75ed --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,147 @@ +# Databasus — Agent Rules and Guidelines + +This document contains project-wide coding standards and best practices for Databasus. +This is NOT a strict set of rules — it is a set of recommendations to help write better, more consistent code. + +Per-folder rules live next to the code they govern: + +- [`backend/CLAUDE.md`](backend/CLAUDE.md) — Go + Gin + GORM + PostgreSQL backend (controllers, migrations, CRUD, DI, testing, logging) +- [`agent/verification/CLAUDE.md`](agent/verification/CLAUDE.md) — Go verification agent CLI (self-update + capacity heartbeat; restore logic deferred) +- [`frontend/CLAUDE.md`](frontend/CLAUDE.md) — React 19 + TypeScript + Vite + Ant Design + Tailwind + +This root file holds the engineering philosophy that applies everywhere. + +--- + +## Development environment + +Work happens inside the repo's [Dev Container](.devcontainer/devcontainer.json). The container ships Go, Node.js + pnpm, Docker-in-Docker, linters and matching VS Code extensions, so the toolchain is identical for every contributor. Ports `4005` (backend) and `5173` (Vite) are forwarded automatically. Don't install or rely on host-level SDKs — run `make`, `pnpm` and `docker` commands from inside the container. + +--- + +## Language in code + +**English only in code, comments, identifiers, log messages, API strings, test assertions, and commit messages.** No other language inside `backend/`, `agent/`, or `frontend/src/` — even for user-facing fallback copy or error messages. + +--- + +## Engineering philosophy + +**Think like a skeptical senior engineer and code reviewer. Don't just do what was asked — also think about what should have been asked. Catch real issues, not theoretical ones.** + +### Task tiers (scale your response to the task) + +- **Trivial** (typos, formatting, single-field adds): apply directly. Steps 5 only. +- **Standard** (CRUD, typical features): steps 1, 5. +- **Complex** (architecture, security, performance-critical): all steps. +- **Unclear** (ambiguous requirements): steps 1 and 4 are mandatory. + +### Steps for non-trivial tasks + +1. **Restate the objective**, list explicit + inferred assumptions, flag shaky ones. +2. **Propose solutions** — for complex tasks, 2–3 approaches including a simpler baseline; recommend one with tradeoffs (complexity, maintainability, performance, extensibility). +3. **Identify risks** — edge cases, security/privacy, performance, operational concerns (deployment, observability, rollback). Before finalizing, ask "what could go wrong?" and patch. +4. **Handle ambiguity** — pick a reasonable default, label it, note what changes under alternative assumptions. +5. **Deliver quality** — correct, testable, maintainable code with minimal tests/validation. Prefer controller tests over unit tests. +6. **Fix root causes, not symptoms** — ask "why did this happen?" and address the underlying issue. + +### After each run: suggest refactorings + +Reread the diff with fresh eyes and **list** (don't silently apply) refactor suggestions: unclear names, duplication, dead code, deep nesting, misplaced responsibilities, leaky abstractions. Keep suggestions concrete (file + lines), behavior-preserving, and scoped to the current change. If the diff is already clean, say so in one line. + +### Mandatory compliance review + +Every non-trivial change is audited twice by the [`claude-md-reviewer`](.claude/agents/claude-md-reviewer.md) subagent — against this document **and** the module doc for each area it touches (`backend/CLAUDE.md`, `agent/verification/CLAUDE.md`, `frontend/CLAUDE.md`). The module docs add stack-specific rules on top of this one; they never replace it, so a change under `backend/` answers to both. + +1. **After planning, before writing code** — it checks the proposed names, file placement, and any planned backward-compat shims while they're still cheap to change. +2. **After implementing, before finishing the turn** — it checks the working-tree diff and runs the linter for each directory the diff touches. + +The reviewer is read-only: it reports findings, you apply the fixes. Resolve every `CHANGES REQUIRED` finding before moving on. Hooks in [`.claude/settings.json`](.claude/settings.json) prompt for both checkpoints, but the obligation is this rule, not the hook — honour it if hooks are disabled. + +### Naming + +Name variables and functions for **intent**, not mechanism. Naming is the biggest readability lever — avoid generic placeholders (`data`, `handle`, `process`, `tmp`, `helper`, `manager`), type-suffix noise (`nameStr`, `agentList`, `tokenObj`), and mechanism-flavored names (`tickNow`, `hbResp`, `dataObj`). + +Booleans take an `is` / `can` / `has` / `should` prefix (`isAllowed`, `canAccess`, `hasItems`, `shouldRetry`) — never bare nouns/verbs like `allowed` or `touches`. + +State that holds "the entity currently being X-ed" must include the entity: `createdAgent`, not `created`; `listedAgents`, not `listed`; `deletingAgentId`, not `deletingId`. Pair related state explicitly: `revealedToken` + `revealedTokenAgentName`, not `tokenAgentName`. Loop variables get the domain word — `for _, agent := range listedAgents` (Go) — single letters only inside trivial one-line lambdas (`.map((a) => a.id)`). + +Test-mechanism names (`got`, `want`, `expected`) are not acceptable — use the domain noun: `persistedAgent`, `firstRotation`, `secondRotation`. Match domain language across the wire — if the API says `agent`, don't rename to `worker` or `node` in client code. + +Idiomatic short names stay where the convention is well-established: Go receivers (`s *AgentService`, `r *AgentRepository`, `ctx *gin.Context`) and JS/TS error catches (`} catch (e) {`). When no good name exists, the abstraction is wrong — extract or rename it, don't reach for `helper` / `tmp` / `manager`. + +Examples: + +```go +// bad +var hbResp HeartbeatResponse +var got *Agent +for _, a := range listed { ... } + +// good +var heartbeatResponse HeartbeatResponse +var persistedAgent *Agent +for _, agent := range listedAgents { ... } +``` + +```ts +// bad +const [tickNow, setTickNow] = useState(Date.now()); +const [deletingId, setDeletingId] = useState(null); +const [tokenAgentName, setTokenAgentName] = useState(''); + +// good +const [currentTimeMs, setCurrentTimeMs] = useState(Date.now()); +const [deletingAgentId, setDeletingAgentId] = useState(null); +const [revealedTokenAgentName, setRevealedTokenAgentName] = useState(''); +``` + +### Functions, methods and types + +The intent rule applies to callables and types too — this is where it's violated most: + +- **A name that needs a "what" comment is a naming bug.** Doc comments are for *why*, ordering, and hidden constraints — never to restate behavior. If you wrote `// Foo does X` above `func Foo`, rename `Foo` until that comment is redundant, then delete it. Keep only the sentence a name *cannot* carry (e.g. "must be called before the container exists"). +- **Predicate methods read as a question**, same as boolean vars: `IsAborted(id)`, not `AbortedContains(id)`; `HasCapacity()`, not `CapacityCheck()`. +- **The name states everything the unit does.** A function that records *and* cancels is `recordAndCancelAborts` — or it's two functions. A name that hides a second effect is a lie. +- **Getters take a `Get` prefix and name the entity.** `GetRunningVerificationIDs()`, not `Active()`, `RunningVerificationIDs()`, or `DiskUsageBytes()`. This is a deliberate house style: prefer the explicit `Get` *even though vanilla Go idiom omits it* — it keeps accessors visually distinct from actions and matches the backend (`GetAuditLogService`, `GetGlobalAuditLogs`). Constructors stay `New...`; a getter that returns a bool is still a predicate, so it stays `Is/Has...`, not `Get...`. +- **Long positional parameter lists become a struct.** ~4+ params, or two same-typed params adjacent → a named parameter/DTO struct (the input counterpart to the result type, e.g. `spawnPlan` → `SpawnSpec`). Kills call-site ordering bugs and the comment explaining argument order. +- **Type names: avoid generic *and* avoid package stutter.** Not `Manager` / `Provisioner` / `Handler` (generic), not `container.ContainerManager` (stutters — `revive` fails CI). If the only honest name stutters, put the noun on the *variable*, not the type: keep the type `container.Manager`, name the variable `containerManager`. If no precise type name exists, the abstraction is wrong — split it. + +```go +// bad — comment restates the name; name hides the second effect; predicate isn't a question +// applyAborts records the abort set and cancels each registered job. +func (h *Heartbeater) applyAborts(ids []uuid.UUID) { ... } +func (h *Heartbeater) AbortedContains(id uuid.UUID) bool { ... } + +// good — the names carry it; no comment needed +func (h *Heartbeater) recordAndCancelAborts(ids []uuid.UUID) { ... } +func (h *Heartbeater) IsAborted(id uuid.UUID) bool { ... } +``` + +### Linting and formatting + +After each change run linting and formatting depending on folder you are working it. +- backend and agent has `make lint` commands +- frontend has `pnpm lint` and `pnpm format` commands + +### Comments are a last resort + +The default is **no comment**. Before writing one, reach for a clearer name or a smaller function first — self-explanatory code beats commented code every time. A comment is justified only when it carries something the code cannot: a *why* (business rule, hidden cross-system constraint, non-obvious algorithm or optimisation, ADR reference). Never write a comment that restates *what* the code does or narrates the obvious — if a `// Foo does X` comment sits above the code, that's a naming bug: rename until the comment is redundant, then delete it. This applies everywhere, tests included — a well-named test needs no header comment explaining what it checks. + +### No "how it was" comments, no unrequested backward compatibility + +Don't write comments that explain previous behavior ("used to be X", "was renamed from Y", "kept for legacy callers"). Code shows the current state; history lives in git. + +Don't preserve backward compatibility unless the user asks for it. No deprecation shims, no aliases, no fallbacks for the old shape. When planning a change that would break existing callers, schemas, configs, or APIs, call out the break explicitly in the plan. If the user approves it, delete the old code outright — do not leave a transition layer behind. + +### Security + +Databasus handles sensitive data, so security is a layered defence. CodeQL, CodeRabbit, Codex Security, Trivy, Dependabot, gitleaks and semgrep all run on every PR. When working in the repo: + +- Don't disable or weaken security checks to make a build pass. For genuine false positives, suppress explicitly with a documented reason (e.g. `.trivyignore`) and explain why in the PR. +- Pin every new GitHub Action to a full commit SHA with a `# vX.Y.Z` tag comment. No floating tags like `@v4` or `@main`. +- Workflows default to top-level `permissions: contents: read`; elevate per-job only when justified. +- Keep Dockerfiles free of secrets, floating base-image tags and unjustified root. If root-at-start is required (PUID/PGID remap, volume chown, initdb), drop privileges with `gosu` before `exec`-ing the app. +- Never log secrets, tokens or credentials. Redact at the logger layer, not at call sites. + +The README's `🛡️ Security & reliability engineering` section is the public-facing version of these practices — keep both consistent if substance changes. \ No newline at end of file diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..3a62d49 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,443 @@ +# ========= BUILD FRONTEND ========= +FROM --platform=$BUILDPLATFORM node:24-alpine AS frontend-build + +WORKDIR /frontend + +# Add version for the frontend build +ARG APP_VERSION=dev +ENV VITE_APP_VERSION=$APP_VERSION + +COPY frontend/package.json frontend/pnpm-lock.yaml ./ +RUN corepack enable && pnpm install --frozen-lockfile +COPY frontend/ ./ + +# Copy .env file (with fallback to .env.production.example) +RUN if [ ! -f .env ]; then \ + if [ -f .env.production.example ]; then \ + cp .env.production.example .env; \ + fi; \ + fi + +RUN pnpm build + +# ========= BUILD BACKEND ========= +# Backend build stage +FROM --platform=$BUILDPLATFORM golang:1.26.3 AS backend-build + +# Make TARGET args available early so tools built here match the final image arch +ARG TARGETOS +ARG TARGETARCH + +# Install Go public tools needed in runtime. Use `go build` for goose so the +# binary is compiled for the target architecture instead of downloading a +# prebuilt binary which may have the wrong architecture (causes exec format +# errors on ARM). +RUN git clone --depth 1 --branch v3.27.1 https://github.com/pressly/goose.git /tmp/goose && \ + cd /tmp/goose/cmd/goose && \ + GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH:-amd64} \ + go build -o /usr/local/bin/goose . && \ + rm -rf /tmp/goose +RUN go install github.com/swaggo/swag/cmd/swag@v1.16.4 + +# Set working directory +WORKDIR /app + +# Install Go dependencies +COPY backend/go.mod backend/go.sum ./ +RUN go mod download + +# Create required directories for embedding +RUN mkdir -p /app/ui/build + +# Copy frontend build output for embedding +COPY --from=frontend-build /frontend/dist /app/ui/build + +# Generate Swagger documentation +COPY backend/ ./ +RUN swag init -d . -g cmd/main.go -o swagger + +# Compile the backend +ARG TARGETOS +ARG TARGETARCH +ARG TARGETVARIANT +RUN CGO_ENABLED=0 \ + GOOS=$TARGETOS \ + GOARCH=$TARGETARCH \ + go build -o /app/main ./cmd + + +# ========= BUILD VERIFICATION AGENT ========= +FROM --platform=$BUILDPLATFORM golang:1.26.3 AS verification-agent-build + +ARG APP_VERSION=dev + +WORKDIR /agent + +COPY agent/verification/go.mod agent/verification/go.sum ./ +RUN go mod download + +COPY agent/verification/ ./ + +RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \ + go build -ldflags "-X main.Version=${APP_VERSION}" \ + -o /verification-agent-binaries/databasus-verification-agent-linux-amd64 ./cmd + +RUN CGO_ENABLED=0 GOOS=linux GOARCH=arm64 \ + go build -ldflags "-X main.Version=${APP_VERSION}" \ + -o /verification-agent-binaries/databasus-verification-agent-linux-arm64 ./cmd + + +# ========= RUNTIME ========= +# In this final image we ship only what the running app actually needs: +# - Build-only tooling (compilers, codegen, key fetchers) stays in the earlier +# stages above and never reaches here. +# - Anything pulled in for a single build step is purged within the same layer. +# - We keep this tight because every extra binary widens the attack surface and +# adds another CVE to track. +FROM debian:bookworm-slim + +# Add version metadata to runtime image +ARG APP_VERSION=dev +ARG TARGETARCH +LABEL org.opencontainers.image.version=$APP_VERSION +ENV APP_VERSION=$APP_VERSION +ENV CONTAINER_ARCH=$TARGETARCH + +# Set production mode for Docker containers +ENV ENV_MODE=production + +# ========= Install all apt packages in a single layer ========= +# Base packages + PostgreSQL 17 (pgdg repo) + Valkey (greensec repo) + rclone, in +# one RUN to minimise layer count and cache-export overhead. +# +# - wget: build-only — fetches the repo signing keys, then purged at the end of +# this RUN (see the "minimal attack surface" note on the runtime stage above). +# - Repo keys: scoped signed-by keyrings, not the deprecated global apt-key trust +# store, so a compromised repo key cannot vouch for any other repository. +# - Codename: hardcoded "bookworm" (base image is pinned), so no lsb-release. +# - Valkey: bound to localhost only — never exposed outside the container. +RUN set -eux; \ + apt-get update; \ + apt-get install -y --no-install-recommends \ + ca-certificates gosu rclone \ + libncurses5 libncurses6 libmariadb3 libgnutls30 \ + wget; \ + wget -qO /usr/share/keyrings/pgdg.asc https://www.postgresql.org/media/keys/ACCC4CF8.asc; \ + echo "deb [signed-by=/usr/share/keyrings/pgdg.asc] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" \ + > /etc/apt/sources.list.d/pgdg.list; \ + wget -qO /usr/share/keyrings/greensec.github.io-valkey-debian.key \ + https://greensec.github.io/valkey-debian/public.key; \ + echo "deb [signed-by=/usr/share/keyrings/greensec.github.io-valkey-debian.key] https://greensec.github.io/valkey-debian/repo bookworm main" \ + > /etc/apt/sources.list.d/valkey-debian.list; \ + apt-get update; \ + apt-get install -y --no-install-recommends postgresql-17 valkey; \ + apt-get purge -y --auto-remove wget; \ + rm -rf /var/lib/apt/lists/* + +# ========= Pre-built DB client binaries (PG, MySQL, MariaDB, MongoDB) ========= +# All client tools live under /app/assets/tools// — the backend resolves +# them at runtime via runtime.GOARCH. Use a bind mount so only the tree matching +# $TARGETARCH ends up in an image layer (the unused arch never materialises). +ARG TARGETARCH +RUN --mount=type=bind,source=assets/tools,target=/ctx/tools,readonly \ + mkdir -p /app/assets/tools && \ + if [ "$TARGETARCH" = "amd64" ]; then \ + cp -r /ctx/tools/x64 /app/assets/tools/x64; \ + elif [ "$TARGETARCH" = "arm64" ]; then \ + cp -r /ctx/tools/arm /app/assets/tools/arm; \ + fi && \ + chmod +x /app/assets/tools/*/postgresql/*/bin/* \ + /app/assets/tools/*/mysql/*/bin/* \ + /app/assets/tools/*/mariadb/*/bin/* \ + /app/assets/tools/*/mongodb/bin/* + +# Create postgres user and set up directories +RUN groupadd -g 999 postgres || true && \ + useradd -m -s /bin/bash -u 999 -g 999 postgres || true && \ + mkdir -p /databasus-data/pgdata && \ + chown -R postgres:postgres /databasus-data/pgdata + +# Create non-root user for the main application process +RUN useradd -r -s /usr/sbin/nologin -u 65532 databasus + +WORKDIR /app + +# Copy Goose from build stage +COPY --from=backend-build /usr/local/bin/goose /usr/local/bin/goose + +# Copy app binary +COPY --from=backend-build /app/main . + +# Expose the binary as the `databasus` command on PATH (e.g. `databasus healthcheck`) +RUN ln -s /app/main /usr/local/bin/databasus + +# Copy migrations directory +COPY backend/migrations ./migrations + +# Copy UI files +COPY --from=backend-build /app/ui/build ./ui/build + +# Copy verification agent binaries (both architectures) — served by the backend +# at GET /api/v1/system/verification-agent?arch=amd64|arm64 +RUN mkdir -p ./agent-binaries +COPY --from=verification-agent-build /verification-agent-binaries/* ./agent-binaries/ + +# Bake .env.example as /.env so the binary has defaults when no env file is +# mounted. The backend looks for .env at the parent of cwd (= /app), i.e. /. +# Real env vars (-e, compose, k8s) take precedence — godotenv.Load does not +# overwrite already-set variables. +COPY .env.example /.env + +# Create startup script +COPY </dev/null)" ]; then + echo "" + echo "==========================================" + echo "ERROR: Legacy volume detected!" + echo "==========================================" + echo "" + echo "You are using the \`postgresus-data\` folder. It seems you changed the image name from Postgresus to Databasus without changing the volume." + echo "" + echo "Please either:" + echo " 1. Switch back to image rostislavdugin/postgresus:latest (supported until ~Dec 2026)" + echo " 2. Read the migration guide: https://databasus.com/installation/#postgresus-migration" + echo "" + echo "==========================================" + exit 1 +fi + +# ========= Adjust postgres user UID/GID ========= +PUID=\${PUID:-999} +PGID=\${PGID:-999} + +CURRENT_UID=\$(id -u postgres) +CURRENT_GID=\$(id -g postgres) + +if [ "\$CURRENT_GID" != "\$PGID" ]; then + echo "Adjusting postgres group GID from \$CURRENT_GID to \$PGID..." + groupmod -o -g "\$PGID" postgres +fi + +if [ "\$CURRENT_UID" != "\$PUID" ]; then + echo "Adjusting postgres user UID from \$CURRENT_UID to \$PUID..." + usermod -o -u "\$PUID" postgres +fi + +# PostgreSQL 17 binary paths +PG_BIN="/usr/lib/postgresql/17/bin" + +# Generate runtime configuration for frontend +echo "Generating runtime configuration..." + +# Detect if email is configured (both SMTP_HOST and DATABASUS_URL must be set) +if [ -n "\${SMTP_HOST:-}" ] && [ -n "\${DATABASUS_URL:-}" ]; then + IS_EMAIL_CONFIGURED="true" +else + IS_EMAIL_CONFIGURED="false" +fi + +cat > /app/ui/build/runtime-config.js </dev/null; then + echo "Injecting analytics script..." + sed -i "s## \${ANALYTICS_SCRIPT}\\ + #" /app/ui/build/index.html + fi +fi + +# Ensure proper ownership of data directory +echo "Setting up data directory permissions..." +mkdir -p /databasus-data/pgdata +mkdir -p /databasus-data/temp +mkdir -p /databasus-data/backups +chown databasus:databasus /databasus-data +chown -R postgres:postgres /databasus-data/pgdata +chown -R databasus:databasus /databasus-data/temp /databasus-data/backups +# Upgrade path: secret.key and instance.json may be owned by root or postgres +# from older images. Re-own them so the non-root main process can read/write. +chown databasus:databasus /databasus-data/secret.key /databasus-data/instance.json 2>/dev/null || true +chmod 700 /databasus-data/temp + +# ========= Start Valkey (internal cache) ========= +echo "Configuring Valkey cache..." +cat > /tmp/valkey.conf << 'VALKEY_CONFIG' +port 6379 +bind 127.0.0.1 +protected-mode yes +save "" +maxmemory 256mb +maxmemory-policy allkeys-lru +VALKEY_CONFIG + +echo "Starting Valkey..." +valkey-server /tmp/valkey.conf & +VALKEY_PID=\$! + +echo "Waiting for Valkey to be ready..." +for i in {1..30}; do + if valkey-cli ping >/dev/null 2>&1; then + echo "Valkey is ready!" + break + fi + sleep 1 +done + +# Initialize PostgreSQL if not already initialized +if [ ! -s "/databasus-data/pgdata/PG_VERSION" ]; then + echo "Initializing PostgreSQL database..." + gosu postgres \$PG_BIN/initdb -D /databasus-data/pgdata --encoding=UTF8 --locale=C.UTF-8 + + # Configure PostgreSQL + echo "host all all 127.0.0.1/32 md5" >> /databasus-data/pgdata/pg_hba.conf + echo "local all all trust" >> /databasus-data/pgdata/pg_hba.conf + echo "port = 5437" >> /databasus-data/pgdata/postgresql.conf + echo "listen_addresses = 'localhost'" >> /databasus-data/pgdata/postgresql.conf + echo "shared_buffers = 256MB" >> /databasus-data/pgdata/postgresql.conf + echo "max_connections = 100" >> /databasus-data/pgdata/postgresql.conf +fi + +# Function to start PostgreSQL and wait for it to be ready +start_postgres() { + echo "Starting PostgreSQL..." + # -k /tmp: create Unix socket and lock file in /tmp instead of /var/run/postgresql/. + # On NAS systems (e.g. TrueNAS Scale), the ZFS-backed Docker overlay filesystem + # ignores chown/chmod on directories from image layers, so PostgreSQL gets + # "Permission denied" when creating .s.PGSQL.5437.lock in /var/run/postgresql/. + # All internal connections use TCP (-h localhost), so the socket location does not matter. + gosu postgres \$PG_BIN/postgres -D /databasus-data/pgdata -p 5437 -k /tmp & + POSTGRES_PID=\$! + + echo "Waiting for PostgreSQL to be ready..." + for i in {1..30}; do + if gosu postgres \$PG_BIN/pg_isready -p 5437 -h localhost >/dev/null 2>&1; then + echo "PostgreSQL is ready!" + return 0 + fi + sleep 1 + done + return 1 +} + +# Try to start PostgreSQL +if ! start_postgres; then + echo "" + echo "==========================================" + echo "PostgreSQL failed to start. Attempting WAL reset recovery..." + echo "==========================================" + echo "" + + # Kill any remaining postgres processes + pkill -9 postgres 2>/dev/null || true + sleep 2 + + # Attempt pg_resetwal to recover from WAL corruption + echo "Running pg_resetwal to reset WAL..." + if gosu postgres \$PG_BIN/pg_resetwal -f /databasus-data/pgdata; then + echo "WAL reset successful. Restarting PostgreSQL..." + + # Try starting PostgreSQL again after WAL reset + if start_postgres; then + echo "PostgreSQL recovered successfully after WAL reset!" + else + echo "" + echo "==========================================" + echo "ERROR: PostgreSQL failed to start even after WAL reset." + echo "The database may be severely corrupted." + echo "" + echo "Options:" + echo " 1. Delete the volume and start fresh (data loss)" + echo " 2. Manually inspect /databasus-data/pgdata for issues" + echo "==========================================" + exit 1 + fi + else + echo "" + echo "==========================================" + echo "ERROR: pg_resetwal failed." + echo "The database may be severely corrupted." + echo "" + echo "Options:" + echo " 1. Delete the volume and start fresh (data loss)" + echo " 2. Manually inspect /databasus-data/pgdata for issues" + echo "==========================================" + exit 1 + fi +fi + +# Create database and set password for postgres user +echo "Setting up database and user..." +gosu postgres \$PG_BIN/psql -p 5437 -h localhost -d postgres << 'SQL' + +-- We use stub password, because internal DB is not exposed outside container +ALTER USER postgres WITH PASSWORD 'Q1234567'; +SELECT 'CREATE DATABASE databasus OWNER postgres' +WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'databasus') +\\gexec +\\q +SQL + +# ========= Refuse to start if legacy WAL backup data exists ========= +# The agent-based WAL_V1 backup type was removed. Existing installs with +# WAL-mode databases must downgrade, manually remove them and then upgrade. +echo "Checking for legacy WAL backup configuration..." +WAL_CHECK_DSN="postgres://postgres:Q1234567@localhost:5437/databasus" + +WAL_CHECK_COL=\$(gosu databasus \$PG_BIN/psql "\$WAL_CHECK_DSN" -tA -c "SELECT 1 FROM information_schema.columns WHERE table_name='postgresql_databases' AND column_name='backup_type' LIMIT 1" 2>/dev/null || true) + +if [ "\$WAL_CHECK_COL" = "1" ]; then + WAL_CHECK_ROW=\$(gosu databasus \$PG_BIN/psql "\$WAL_CHECK_DSN" -tA -c "SELECT 1 FROM postgresql_databases WHERE backup_type='WAL_V1' LIMIT 1" 2>/dev/null || true) + if [ "\$WAL_CHECK_ROW" = "1" ]; then + echo "" + echo "==========================================" + echo "ERROR: Agent (WAL_V1) backup approach is no longer supported." + echo "==========================================" + echo "" + echo "Please downgrade to version 3.42.0, remove all WAL-mode databases" + echo "manually and then upgrade again. This safeguard exists to avoid" + echo "corrupting already-set-up agents." + echo "" + echo "==========================================" + exit 1 + fi +fi +echo "No legacy WAL backup data detected." + +# Start the main application +echo "Starting Databasus application..." + +exec gosu databasus ./main +EOF + +LABEL org.opencontainers.image.source="https://github.com/databasus/databasus" + +RUN chmod +x /app/start.sh + +EXPOSE 4005 + +# Liveness probe: the runtime image ships no wget/curl, so the binary checks +# itself. Targets the dependency-free /system/version endpoint (not the deep +# /system/health) so a degraded-but-serving instance is never restarted. +HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \ + CMD ["databasus", "healthcheck"] + +# Volume for PostgreSQL data +VOLUME ["/databasus-data"] + +ENTRYPOINT ["/app/start.sh"] +CMD [] diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..aa10267 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (which shall not include communications that are solely written + by You). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based upon (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and derivative works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control + systems, and issue tracking systems that are managed by, or on behalf + of, the Licensor for the purpose of discussing and improving the Work, + but excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to use, reproduce, modify, distribute, prepare + Derivative Works of, and publicly display, publicly perform, + sublicense, and distribute the Work and such Derivative Works in + Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright notice to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. When redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "license" line as the copyright notice for easier + identification within third-party archives. + + Copyright 2026 Databasus + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/ + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/NOTICE.md b/NOTICE.md new file mode 100644 index 0000000..4fcacd6 --- /dev/null +++ b/NOTICE.md @@ -0,0 +1,22 @@ +Copyright © 2025–2026 Rostislav Dugin and contributors. + +“Databasus” is a trademark of Rostislav Dugin. + +The source code in this repository is licensed under the Apache License, Version 2.0. +That license applies to the code only and does not grant any right to use the +Databasus name, logo, or branding, except for reasonable and customary referential +use in describing the origin of the software and reproducing the content of this NOTICE. + +Permitted referential use includes truthful use of the name “Databasus” to identify +the original Databasus project in software catalogs, deployment templates, hosting +panels, package indexes, compatibility pages, integrations, tutorials, reviews, and +similar informational materials, including phrases such as “Databasus”, +“Deploy Databasus”, “Databasus on Coolify”, and “Compatible with Databasus”. + +You may not use “Databasus” as the name or primary branding of a competing product, +service, fork, distribution, or hosted offering, or in any manner likely to cause +confusion as to source, affiliation, sponsorship, or endorsement. + +Nothing in this repository transfers, waives, limits, or estops any rights in the +Databasus mark. All trademark rights are reserved except for the limited referential +use stated above. \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..eb35d0d --- /dev/null +++ b/README.md @@ -0,0 +1,317 @@ +
+ Databasus Logo + +

PostgreSQL backup tool

+

Databasus is a free, open source and self-hosted tool to backup PostgreSQL. Make backups with different storages (S3, Google Drive, FTP, etc.) and notifications about progress (Slack, Discord, Telegram, etc.). With a focus on Point-in-Time Recovery at low RPO/RTO

+ + + [![PostgreSQL](https://img.shields.io/badge/PostgreSQL-336791?logo=postgresql&logoColor=white)](https://www.postgresql.org/) + [![MySQL](https://img.shields.io/badge/MySQL-4479A1?logo=mysql&logoColor=white)](https://www.mysql.com/) + [![MariaDB](https://img.shields.io/badge/MariaDB-003545?logo=mariadb&logoColor=white)](https://mariadb.org/) + [![MongoDB](https://img.shields.io/badge/MongoDB-47A248?logo=mongodb&logoColor=white)](https://www.mongodb.com/) +
+ [![Apache 2.0 License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE) + [![Docker Pulls](https://img.shields.io/docker/pulls/databasus/databasus?color=brightgreen)](https://hub.docker.com/r/databasus/databasus) + [![Platform](https://img.shields.io/badge/platform-linux%20%7C%20macos%20%7C%20windows-lightgrey)](https://github.com/databasus/databasus) + [![Self Hosted](https://img.shields.io/badge/self--hosted-yes-brightgreen)](https://github.com/databasus/databasus) + [![Open Source](https://img.shields.io/badge/open%20source-❤️-red)](https://github.com/databasus/databasus) + +

+ Features • + Installation • + Usage • + License • + Contributing +

+ +

+ 🌐 Databasus website +

+ + Databasus Dark Dashboard + + Databasus Dashboard +
+ +--- + +## ✨ Features + +### 📦 **Backup types** + +- **Physical**: file-level copy of the entire database cluster over PostgreSQL native incremental backups mechanism (read more) + - **Full**: a complete, self-contained copy of the cluster + - **Incremental**: stores only what changed since the previous full backup, so backups stay small and fast + - **WAL streaming**: continuously captures the database write stream, enabling Point-in-time recovery (PITR). Designed for disaster recovery and near-zero data loss +- **Logical**: native dump of the database in its engine-specific binary format (compressed, suitable for parallel restore) + +### 🔄 **Scheduled backups** + +- **Flexible scheduling**: hourly, daily, weekly, monthly or cron +- **Precise timing**: run backups at specific times (e.g., 4 AM during low traffic) +- **Smart compression**: 4-8x space savings with balanced compression (~20% overhead) + +### 🧪 **Restore verification** (docs) + +Databasus performs a real restore to confirm backups are usable, not just intact on disk or checksum check. + +- **Triggers**: after each backup or on a flexible schedule (hourly, daily, weekly, monthly or cron) +- **Real restore**: spins up a database container, runs the restore and checks the restored size against the backup +- **Report**: lists every table with its row count +- **Optional notifications**: send the report or failure-only alerts through any configured notifier + +### 🗑️ **Retention policies** + +- **Time period**: Keep backups for a fixed duration (e.g., 7 days, 3 months, 1 year) +- **Count**: Keep a fixed number of the most recent backups (e.g., last 30) +- **GFS (Grandfather-Father-Son)**: Layered retention — keep hourly, daily, weekly, monthly and yearly backups independently for fine-grained long-term history (enterprises requirement) +- **Size limits**: Set per-backup and total storage size caps to control storage usage + +### 🗄️ **Multiple storage destinations** (view supported) + +- **Local storage**: Keep backups on your VPS/server +- **Cloud storage**: S3, Cloudflare R2, Google Drive, NAS, Dropbox, SFTP, Rclone and more +- **Secure**: All data stays under your control + +### 📱 **Notifications** (view supported) + +- **Multiple channels**: Email, Telegram, Slack, Discord, webhooks +- **Real-time updates**: Success and failure notifications +- **Team integration**: Perfect for DevOps workflows + +### 🔒 **Enterprise-grade security** (docs) + +- **AES-256-GCM encryption**: Enterprise-grade protection for backup files +- **Zero-trust storage**: Backups are encrypted and remain useless to attackers, so you can safely store them in shared storage like S3, Azure Blob Storage, etc. +- **Encryption for secrets**: Any sensitive data is encrypted and never exposed, even in logs or error messages +- **Read-only user**: Databasus uses a read-only user by default for backups and never stores anything that can modify your data + +### 👥 **Suitable for teams** (docs) + +- **Workspaces**: Group databases, notifiers and storages for different projects or teams +- **Access management**: Control who can view or manage specific databases with role-based permissions +- **Audit logs**: Track all system activities and changes made by users +- **User roles**: Assign viewer, member, admin or owner roles within workspaces + +### 🎨 **UX-Friendly** + +- **Designer-polished UI**: Clean, intuitive interface crafted with attention to detail +- **Dark & light themes**: Choose the look that suits your workflow +- **Mobile adaptive**: Check your backups from anywhere on any device + +### 💾 **Supported databases** + +- **PostgreSQL**: 14, 15, 16, 17 and 18 (physical and logical) +- **MySQL**: 5.7 and 8 (logical only) +- **MariaDB**: 10, 11 and 12 (logical only) +- **MongoDB**: 4.2+, 5, 6, 7 and 8 (logical only) +- +### 🐳 **Self-hosted & secure** + +- **Docker-based**: Easy deployment and management +- **Privacy-first**: All your data stays on your infrastructure +- **Open source**: Apache 2.0 licensed, inspect every line of code + +### 📦 Installation (docs) + +You have four ways to install Databasus: + +- Automated script (recommended) +- Simple Docker run +- Docker Compose setup +- Kubernetes with Helm + +Databasus Dashboard + +--- + +## 📦 Installation + +You have four ways to install Databasus: automated script (recommended), simple Docker run, or Docker Compose setup. + +### Option 1: Automated installation script (recommended, Linux only) + +The installation script will: + +- ✅ Install Docker with Docker Compose (if not already installed) +- ✅ Set up Databasus +- ✅ Configure automatic startup on system reboot + +```bash +sudo apt-get install -y curl && \ +sudo curl -sSL https://raw.githubusercontent.com/databasus/databasus/refs/heads/main/install-databasus.sh \ +| sudo bash +``` + +### Option 2: Simple Docker run + +The easiest way to run Databasus: + +```bash +docker run -d \ + --name databasus \ + -p 4005:4005 \ + -v ./databasus-data:/databasus-data \ + --restart unless-stopped \ + databasus/databasus:latest +``` + +This single command will: + +- ✅ Start Databasus +- ✅ Store all data in `./databasus-data` directory +- ✅ Automatically restart on system reboot + +### Option 3: Docker Compose setup + +Create a `docker-compose.yml` file with the following configuration: + +```yaml +services: + databasus: + container_name: databasus + image: databasus/databasus:latest + ports: + - "4005:4005" + volumes: + - ./databasus-data:/databasus-data + restart: unless-stopped + healthcheck: + test: ["CMD", "databasus", "healthcheck"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 60s +``` + +Then run: + +```bash +docker compose up -d +``` + +### Option 4: Kubernetes with Helm + +For Kubernetes deployments, install directly from the OCI registry. + +**With ClusterIP + port-forward (development/testing):** + +```bash +helm install databasus oci://ghcr.io/databasus/charts/databasus \ + -n databasus --create-namespace +``` + +```bash +kubectl port-forward svc/databasus-service 4005:4005 -n databasus +# Access at http://localhost:4005 +``` + +**With LoadBalancer (cloud environments):** + +```bash +helm install databasus oci://ghcr.io/databasus/charts/databasus \ + -n databasus --create-namespace \ + --set service.type=LoadBalancer +``` + +```bash +kubectl get svc databasus-service -n databasus +# Access at http://:4005 +``` + +**With Ingress (domain-based access):** + +```bash +helm install databasus oci://ghcr.io/databasus/charts/databasus \ + -n databasus --create-namespace \ + --set ingress.enabled=true \ + --set ingress.hosts[0].host=backup.example.com +``` + +For more options (NodePort, TLS, HTTPRoute for Gateway API), see the [Helm chart README](deploy/helm/README.md). + +--- + +## 🚀 Usage + +1. **Access the dashboard**: Navigate to `http://localhost:4005` +2. **Add your first database for backup**: Click "New Database" and follow the setup wizard +3. **Configure schedule**: Choose from hourly, daily, weekly, monthly or cron intervals +4. **Set database connection**: Enter your database credentials and connection details +5. **Choose storage**: Select where to store your backups (local, S3, Google Drive, etc.) +6. **Configure retention policy**: Choose time period, count or GFS to control how long backups are kept +7. **Add notifications** (optional): Configure email, Telegram, Slack, or webhook notifications +8. **Save and start**: Databasus will validate settings and begin the backup schedule + +### 🔑 Resetting password (docs) + +If you need to reset the password, you can use the built-in password reset command: + +```bash +docker exec -it databasus ./main --new-password="YourNewSecurePassword123" --email="admin" +``` + +Replace `admin` with the actual email address of the user whose password you want to reset. + +### 💾 Backuping Databasus itself + +After installation, it is also recommended to backup your Databasus itself or, at least, to copy secret key used for encryption (30 seconds is needed). So you are able to restore from your encrypted backups if you lose access to the server with Databasus or it is corrupted. + +--- + +## 🛡️ Security & reliability engineering + +Databasus works with sensitive data, so preventing vulnerabilities, unauthorised access and data leaks is a primary concern. We invest in this on both sides of the system: in the code itself (permission checks, encryption, careful handling of secrets) and in the infrastructure around it (dependency analysis, CVE response, DevSecOps best practices). The pipeline below runs automatically on every commit and PR. No single layer is enough on its own, but together they reduce the chance of vulnerable code, unsafe dependencies, broken images, or non-restorable backups reaching a release. + +For static analysis we combine several independent passes. CodeQL scans the full codebase for security issues. CodeRabbit reviews every PR and runs gitleaks for secret scanning and semgrep for security rules inline. Dockerfiles and CI workflows get extra rules of their own (pinned action references, least-privilege permissions, suspicious base images), so insecure patterns are flagged before they ever merge. On top of these per-PR checks, Codex Security from OpenAI runs regular, deeper audits of the whole codebase. It's a separate program that catches architectural and cross-cutting issues narrow PR-time scans can miss. + +On the dependency side, Dependabot watches all of our dependencies against the GitHub Advisory Database and surfaces CVEs within minutes of publication. Updates run through a cooldown so newly-published versions get a chance to mature before we adopt them. This is a deliberate defence against compromised-package incidents like supply-chain attack. The Dependency Review Action blocks any PR that introduces a new HIGH or CRITICAL CVE outright. + +Container images are scanned with Trivy on every build. A separate Trivy pass on the Dockerfile catches misconfigurations before they make it into an image. All GitHub Actions are pinned to full commit SHAs rather than floating tags like `@v4` or `@main`, which have been an active attack vector in 2025. Workflows default to least-privilege permissions and only elevate per-job when genuinely needed. + +Critical paths are covered by both unit and integration tests, run against real database containers for every supported engine and major version. Restore is the path that matters most for a backup tool, so we test it explicitly: every PR runs full backup-then-restore cycles against those same real containers, verifying that backups can actually be restored end-to-end, not just written successfully. The rest of the CI/CD pipeline runs lint, type-check, the full test suite, image smoke tests and multi-architecture builds on every PR. A release only ships if all of it passes. + +Found a vulnerability? Report it via the GitHub Security tab. See [SECURITY.md](https://github.com/databasus/databasus?tab=security-ov-file#readme). Security reports are the highest-priority work queue. For runtime application security (AES-256-GCM at rest, zero-trust storage, encrypted secrets, read-only DB user by default) see [Enterprise-grade security](#-enterprise-grade-security) in the Features section above. + +--- + +## 📝 License + +This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details + +## 🤝 Contributing + +Contributions are welcome! Read the contributing guide for more details, priorities and rules. If you want to contribute but don't know where to start, message me on Telegram [@rostislav_dugin](https://t.me/rostislav_dugin) + +Also you can join our large community of developers, DBAs and DevOps engineers on Telegram [@databasus_community](https://t.me/databasus_community). + +## AI disclaimer + +There have been questions about AI usage in project development in issues and discussions. As the project focuses on security, reliability and production usage, it's important to explain how AI is used in the development process. + +First of all, we are proud to say that Databasus has been accepted into both [Claude for Open Source](https://claude.com/contact-sales/claude-for-oss) by Anthropic and [Codex for Open Source](https://developers.openai.com/codex/community/codex-for-oss/) by OpenAI in March 2026. For us it is one more signal that the project was recognized as important open-source software and was as critical infrastructure worth supporting independently by two of the world's leading AI companies. Read more at [databasus.com/faq](https://databasus.com/faq#oss-programs). + +Despite of this, we have the following rules how AI is used in the development process: + +AI is used as a helper for: + +- verification of code quality and searching for vulnerabilities +- cleaning up and improving documentation, comments and code +- assistance during development +- double-checking PRs and commits after human review +- additional security analysis of PRs via Codex Security + +AI is not used for: + +- writing entire code +- "vibe code" approach +- code without line-by-line verification by a human +- code without tests + +So AI is just an assistant and a tool for developers to increase productivity and ensure code quality. The work is done by developers. + +Moreover, it's important to note that we do not differentiate between bad human code and AI vibe code. There are strict requirements for any code to be merged to keep the codebase maintainable. + +Even if code is written manually by a human, it's not guaranteed to be merged. Vibe code is not allowed at all and all such PRs are rejected by default (see [contributing guide](https://databasus.com/contribute)). + +The engineering safeguards behind these rules (CI, static analysis, dependency scanning, test coverage and vulnerability response) are documented in [Security & reliability engineering](#️-security--reliability-engineering) above. diff --git a/README.wehub.md b/README.wehub.md new file mode 100644 index 0000000..c9d5d9a --- /dev/null +++ b/README.wehub.md @@ -0,0 +1,7 @@ +# WeHub 来源说明 + +- 原始项目：`databasus/databasus` +- 原始仓库：https://github.com/databasus/databasus +- 导入方式：上游默认分支的最新快照 +- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准 +- 本文件仅用于记录来源，不代表 WeHub 是原项目作者 diff --git a/adr/0000-adr-template.md b/adr/0000-adr-template.md new file mode 100644 index 0000000..88db076 --- /dev/null +++ b/adr/0000-adr-template.md @@ -0,0 +1,47 @@ +# ADR-NNNN: + +- **Status:** Proposed | Accepted | Deprecated | Superseded by [ADR-XXXX](./XXXX-...) +- **Date:** YYYY-MM-DD +- **Deciders:** +- **Tags:** + +## Context + +What problem are we solving and why now? Describe the forces at play: business goals, +technical constraints, prior decisions, and anything an outside reader needs to +understand the trade-offs. State assumptions explicitly and flag the shaky ones. + +## Decision + +The decision in one or two sentences, written as a present-tense statement of fact +("We use X to do Y"). Follow with the specifics: what changes, what stays, scope +boundaries, and the contract this establishes for callers. + +## Alternatives considered + +For each option, give the shape and the reason it lost. Be concrete — a rejected +alternative without a reason is just noise. + +- **Option A — .** Brief description. Rejected because . +- **Option B — .** Brief description. Rejected because . +- **Do nothing.** What breaks or stays painful if we keep the status quo. + +## Consequences + +### Positive + +- What gets easier, faster, safer, or cheaper. + +### Negative + +- New constraints, costs, complexity, or risks we're accepting. + +### Neutral / follow-ups + +- Work this unlocks or forces later (migrations, deprecations, doc updates, + monitoring, rollout plan). + +## References + +- Related ADRs: [ADR-XXXX](./XXXX-...) +- Issues / PRs / design docs / external links diff --git a/adr/0001-use-docker-as-base.md b/adr/0001-use-docker-as-base.md new file mode 100644 index 0000000..9b98bed --- /dev/null +++ b/adr/0001-use-docker-as-base.md @@ -0,0 +1,73 @@ +# ADR-0001: Ship as a Docker image + +- **Status:** Accepted +- **Date:** ~2024-06-01 +- **Tags:** deployment, packaging, ops + +## Context + +Postgresus is a system for backup and restore operations. + +It require for work: +- a Go backend (Gin + GORM) with an embedded scheduler; +- a React frontend, served as static assets; +- PostgreSQL tools (`pg_dump`, `pg_restore`, `pg_receivewal`) required for backup and restore operations. + +We need to decide what the deploy unit is. + +## Decision + +We ship as a published `docker-compose.yml` with two services: the Postgresus +app image (backend + frontend + every supported `pg_dump`/`pg_restore` +version) and a PostgreSQL service for internal state. The full install +is always: + +```bash +docker compose up -d +``` + +against our compose file. We don't publish standalone binaries, OS packages, +a single-image `docker run` flow or hosted SaaS. + +## Alternatives considered + +- **A single static binary** (Go binary with the frontend embedded via + `embed.FS`) plus PostgreSQL available on the host. + +## Consequences + +### Positive + +- **Backend and frontend in one unit.** The frontend ships as static files + served by the Go process. No reverse proxy needed, no version skew between + API and UI. +- **One deploy unit, one upgrade path.** Migrations, config validation and + the embedded scheduler all live in the app container; internal state lives + in the sibling PostgreSQL service. `docker compose pull && docker compose + up -d` is the entire upgrade procedure; state lives in named volumes so + rollback is a tag change. +- **Trivial install.** `docker compose up -d` works on any host with Docker. + No language toolchains, no `apt install postgresql-client-16`, no PATH + games. +- **Cross-OS.** Same image works on Linux, macOS and Windows. +- **Predictable runtime.** The exact `pg_dump` used for a backup is the one + installed during image build when we install all PostgreSQL versions. +- **Subjectively, it's just nice to work with.** This isn't a serious argument + on its own, but it matters in a self-hosted product where the install + experience *is* the first-impression UX. + +### Negative + +- **Hard dependency on Docker.** Users who can't or won't run Docker (some + shared-hosting environments, philosophical preferences) aren't served. We + accept this — the target user already has Docker. +- **Big image, slow first pull.** Carrying multiple PostgreSQL versions makes + the image meaningfully larger than a Go binary would be. Multi-stage builds + and careful layer ordering help, but the first pull is still slower than + downloading a binary. Subsequent upgrades benefit from layer reuse. +- **Slow CI builds.** Multi-arch builds with several engines and several + major versions per engine take longer than `go build`. Acceptable for the + team; visible on release day. +- **"I just want a binary" friction.** Some users prefer a systemd unit or a + native package. We're explicitly not catering to that — the maintenance + cost of a second distribution channel outweighs the UX win. \ No newline at end of file diff --git a/adr/0002-build-in-postgresql-instead-of-sqlite.md b/adr/0002-build-in-postgresql-instead-of-sqlite.md new file mode 100644 index 0000000..4b0704e --- /dev/null +++ b/adr/0002-build-in-postgresql-instead-of-sqlite.md @@ -0,0 +1,53 @@ +# ADR-0002: Built-in PostgreSQL instead of SQLite + +- **Status:** Accepted +- **Date:** ~2024-06-01 +- **Tags:** storage, database, ops + +## Context + +Postgresus needs an internal datastore for its own state (schedules, backup +metadata, users). We need to decide what that datastore is and how we ship +it. + +## Decision + +We use PostgreSQL as the internal datastore and ship it as a second service +in the `docker-compose.yml` alongside the Postgresus app. + +## Alternatives considered + +- **SQLite** as an embedded datastore — no separate service, single file on + disk. +- **Full PostgreSQL as a sibling compose service** (chosen). +- **Letting the user choose between SQLite and PostgreSQL.** Rejected: a + single supported storage engine keeps the test matrix and support burden + small. + +## Consequences + +### Positive + +- **Remote access if needed in the future.** PostgreSQL can be exposed over + the network without changing the storage engine. +- **Heavy writing if needed in the future.** PostgreSQL handles concurrent + writers without the single-writer limitation of SQLite. +- **More complex toolchain if needed in the future.** Extensions, replication + and richer query features are available when we need them. +- **We can connect the system to an external PostgreSQL.** Same engine means + swapping the bundled instance for a managed one is a config change, not a + rewrite. +- **Subjective experience.** SQLite tends to cause problems as a project + grows. Personally I (Rostislav) have had many issues with remote access, + backups and multiple writers. SQLite is fine for the current scope, but + choosing the heavier option up front avoids a forced migration later. + +### Negative + +- **Separate service to run and operate.** SQLite would have been a single + file inside the app container; PostgreSQL is a second compose service with + its own image, volume and lifecycle. Acceptable cost for the upside above. +- **More complicated to back up right now.** Backing up a live PostgreSQL + instance is more involved than copying a single SQLite file. +- **Extra CPU and RAM usage.** Running a PostgreSQL process adds overhead, + but it is small enough to be affordable on the target hosts. diff --git a/adr/0003-build-in-postgresql-instead-of-separate-service.md b/adr/0003-build-in-postgresql-instead-of-separate-service.md new file mode 100644 index 0000000..2037ad4 --- /dev/null +++ b/adr/0003-build-in-postgresql-instead-of-separate-service.md @@ -0,0 +1,51 @@ +# ADR-0003: Embed PostgreSQL in the image instead of a separate service + +- **Status:** Accepted +- **Date:** 2025-07-21 +- **Tags:** storage, database, packaging, ops + +## Context + +[ADR-0002](0002-build-in-postgresql-instead-of-sqlite.md) chose PostgreSQL as +the internal datastore and shipped it as a second `docker-compose` service +next to the app. In practice the two-service layout turned out to be heavier +to operate than expected: users have to manage a compose file, two images, +two volumes and the dependency between them, and the project can no longer +be installed with a plain `docker run`. Moreover, users do not upgrade +their PostgreSQL with Postgresus upgrade. + +We need to decide whether to keep the sibling service or fold PostgreSQL into +the app image. + +## Decision + +We embed PostgreSQL directly into the Postgresus image and run it as a +process inside the same container as the app. The install flow goes back to +a single `docker run` (or a one-service compose file). + +## Alternatives considered + +- **Keep PostgreSQL as a sibling compose service** (previous design). +- **Embed PostgreSQL in the app image** (chosen). + +## Consequences + +### Positive + +- **Single-container install.** `docker run` is enough — no compose + file required, no service ordering, no inter-service networking to + explain. +- **Easier to manage.** One image, one volume, one lifecycle. Upgrades, + backups of the install itself and rollbacks all act on a single unit. +- **More predictable environment.** The exact PostgreSQL version, config and + data directory layout are pinned by the image — users can't accidentally + point the app at an incompatible external instance or skew versions + between the two services. + +### Negative + +- **No ability to connect to a remote PostgreSQL.** The internal datastore + is now tied to the image; pointing Postgresus at a managed or external + PostgreSQL for its own state is no longer supported. Backups still target + arbitrary external PostgreSQL instances — this only affects Postgresus's + own state store. diff --git a/adr/0004-focus-on-streaming-chunk-by-chunk-with-backpressure.md b/adr/0004-focus-on-streaming-chunk-by-chunk-with-backpressure.md new file mode 100644 index 0000000..5589ee4 --- /dev/null +++ b/adr/0004-focus-on-streaming-chunk-by-chunk-with-backpressure.md @@ -0,0 +1,76 @@ +# ADR-0004: Stream backup and restore data chunk-by-chunk with backpressure + +- **Status:** Accepted +- **Date:** 2025-12-05 +- **Tags:** backups, storage, performance, reliability + +## Context + +The original backup pipeline ran `pg_dump` and pushed its output to the +configured storage backend (S3 or local) as a single in-process flow, +accumulating the unsent tail of the dump in memory. With a single backup +running at a time this was fine. + +Once real users started backing up 5–10 databases in parallel, the host ran +out of memory. The cause was the same on every install: the producer +(`pg_dump`) was consistently faster than the consumer (upload to storage), +so the difference accumulated in RAM. A slow storage backend, a transient +network error or an S3 throughput cap made the situation strictly worse — +the pipeline kept reading from `pg_dump` with nowhere to send the bytes, +and a single stalled backup could OOM-kill the whole container along with +every other backup running next to it. + +The same shape applies to restore: reading a backup faster than the target +PostgreSQL can ingest it would let the buffered tail grow without bound. + +## Decision + +All backup and restore data moves through the pipeline as a stream of +fixed-size chunks (illustrative ~8 MB; the exact size depends on the +operation and the storage backend). The producer (e.g. `pg_dump` stdout, +`pg_basebackup` stream or the storage downloader during restore) and the +consumer (the storage uploader during backup, `pg_restore` or PostgreSQL +during restore) are coupled by **backpressure**: the producer cannot +enqueue chunk *N+2* until chunk *N* has been accepted by the consumer. A +small bounded in-flight window (illustrative ~16 MB per pipeline) caps +per-process memory regardless of any upstream/downstream speed mismatch. + +**Backup and restore data does not land on the Postgresus host's disk.** +Bytes stream directly between PostgreSQL and the storage backend in both +directions. The Postgresus container's local disk is not used as a staging +area for backup payloads. + +**Exception — operations that genuinely require a materialised file.** +`pg_restore -j >1` cannot consume a pipe; it needs a real file on disk to +parallelise across workers. In that case Postgresus explicitly pre-checks +that the local disk has enough free space to hold the full backup and +refuses the operation with a clear error if it does not. The streaming +model remains the default; materialisation is an opt-in fallback gated by +a precondition check. + +## Consequences + +### Positive + +- **Bounded per-pipeline memory.** Memory usage is capped by the in-flight + window, independent of the size of the dump or the speed of the storage + backend. A 500 GB database costs the same RAM as a 500 MB one. +- **Predictable parallelism.** N parallel backups consume N × window-size + of memory, not N × dump-size. Capacity planning becomes a straight-line + function of how many concurrent jobs the operator allows. +- **Slow storage degrades gracefully.** A slow, throttled or stalled + storage backend slows the producer instead of crashing the host. The + worst case is a long-running backup, not an OOM. +- **No local disk requirement for normal backup/restore.** Operators do + not need to provision Postgresus-host disk equal to their largest + backup, and S3-as-primary-storage deployments work without local scratch + space. +- **Backups larger than the host's free disk are supported.** Database + size is bounded by the storage backend's capacity, not by the + Postgresus container's filesystem. + +## References + +- [ADR-0001](./0001-use-docker-as-base.md) — Docker base image (defines + the single-container environment whose memory limits this decision is + written against). diff --git a/adr/0005-why-assets-binaries-instead-of-dbs-installation.md b/adr/0005-why-assets-binaries-instead-of-dbs-installation.md new file mode 100644 index 0000000..28bdbf1 --- /dev/null +++ b/adr/0005-why-assets-binaries-instead-of-dbs-installation.md @@ -0,0 +1,76 @@ +# ADR-0005: Vendor PostgreSQL client binaries under `assets/` instead of installing them in the Dockerfile + +- **Status:** Accepted +- **Date:** 2026-05-24 +- **Tags:** packaging, ci, supply-chain, dockerfile + +## Context + +Postgresus needs the per-version client tools for every database engine it +supports — `pg_dump` / `pg_restore` / `pg_basebackup` for PostgreSQL, +`mysqldump` / `mysql` for MySQL and MariaDB, `mongodump` / `mongorestore` for +MongoDB. The original approach installed all of them at image-build time +straight from upstream apt repositories (PGDG, the MySQL/MariaDB apt repos, +MongoDB's apt repo), one `apt-get install` invocation per supported major. + +This worked when only a couple of PostgreSQL versions were supported. Then +MySQL, MariaDB and MongoDB landed on top of PostgreSQL, and the supported +matrix grew to **18 engine/major combinations**: + +- PostgreSQL: 12, 13, 14, 15, 16, 17, 18 (7 majors) +- MySQL: 5.7, 8, 9 (3 majors) +- MariaDB: 10, 11, 12 (3 majors) +- MongoDB: 4.2+, 5, 6, 7, 8 (5 majors) + +At that size the build-time install model fell over: + +- **CI time crossed 30 minutes** and kept climbing. Most of that time was + spent re-downloading the same `.deb` packages on every build because the + apt layers invalidated easily. +- **CI failed often** for reasons that had nothing to do with our code — + PGDG / MySQL / MariaDB / MongoDB mirror timeouts, transient apt errors, + DNS hiccups, GPG key churn. Each engine added its own apt repo, its own + signing key and its own way to flake. Every external dependency in the + build path is another way the build can fail. +- **The Dockerfile carried a lot of accidental complexity** — four sets of + repo setup, four sets of key imports, version pinning per engine, + multi-major installs and apt-cache cleanup. + +We need to decide whether to keep installing clients from upstream apt +repositories at build time or vendor the binaries we actually use. + +## Decision + +We download the client binaries we need once across all supported engines, +commit them under `assets/` keyed by engine, major version and architecture +and `COPY` them into the image. The Dockerfile no longer talks to PGDG, the +MySQL / MariaDB repos, the MongoDB repo or any other external package +repository for database client tools. + +## Consequences + +### Positive + +- **CI is dramatically faster and more stable.** The `COPY` of vendored + binaries is a cached layer that costs near-zero on rebuilds. The build no + longer downloads hundreds of megabytes of `.deb` packages on every run. +- **No external dependency in the build path.** PGDG outages, mirror + flakiness and key-rotation events stop breaking our builds. Builds are + reproducible from a clean checkout without network access to third-party + repositories. +- **Smaller supply-chain attack surface.** The build no longer fetches and + executes arbitrary versions of third-party packages resolved at build + time. The exact bytes that ship in the image are the exact bytes + committed to the repo and reviewable in a PR. +- **Smaller, simpler Dockerfile.** Repo setup, GPG key handling, pinning, + multi-version installs and apt-cache cleanup all go away. What's left is + a handful of `COPY` lines that say plainly what is in the image. +- **Version pinning is explicit and auditable.** Upgrading a client is a + visible commit that swaps the binary under `assets/`, not an opaque + `apt-get install` that resolves to whatever PGDG happens to serve that + day. + +## References + +- [ADR-0001](./0001-use-docker-as-base.md) — Docker base image (the + single-container environment whose Dockerfile this decision simplifies). diff --git a/adr/0006-cloud-load-and-traffic-distribution-between-nodes.md b/adr/0006-cloud-load-and-traffic-distribution-between-nodes.md new file mode 100644 index 0000000..da92922 --- /dev/null +++ b/adr/0006-cloud-load-and-traffic-distribution-between-nodes.md @@ -0,0 +1,63 @@ +# ADR-0006: Distribute cloud load across primary and processing nodes + +- **Status:** Deprecated (cloud edition discontinued, 2026-06; multi-node deployment vars `IS_PRIMARY_NODE` / `IS_PROCESSING_NODE` remain valid) +- **Date:** 2026-02-01 +- **Tags:** cloud, deployment, networking, multi-node + +## Context + +The project required funding so we need to launch Cloud to invest in development. + +In cloud mode Databasus faces two pressures that a single-node deployment cannot answer. + +**Source-IP exposure.** Every backup opens an outbound connection from the Databasus host to the customer's external database. A single-node cloud install would expose one fixed IP to every tenant — a public target for attack. + +**Bandwidth contention.** Backups can be arbitrarily large. One tenant pulling 20 backups of 1 TB each will saturate even a 10 Gbit/s uplink for a significant amount of time and starve every other tenant's backup or download down to a crawl. The instance-mode design assumes a single operator with predictable load; the cloud has many tenants with adversarial timing. + +The instance (self-hosted) deployment does not have these pressures: one operator runs one node against their own databases on their own network. + +## Decision + +Cloud deployments run as a cluster of nodes built from the **same binary**, differentiated by two orthogonal axes. + +### Application roles + +Roles are controlled by environment flags: + +- `IS_PRIMARY_NODE=true` — runs schedulers, migrations, the backup and restore registries that distribute work over Valkey pub/sub. One node per Databasus installation. +- `IS_PROCESSING_NODE=true` — runs the backup and restore worker loops that pick up jobs from Valkey and stream bytes between customer databases and storage. N nodes per Databasus installation. + +Both flags can be set on the same node. Instance mode is exactly this case: `IS_MANY_NODES_MODE=false` defaults both to `true` so one container handles everything. + +### Network exposure + +Network exposure is controlled by infrastructure, not by the binary: + +- **API-facing nodes** sit behind CloudFlare with port 4005 published through a reverse proxy. CloudFlare absorbs DDoS on the API surface. +- **Processing nodes** do not publish port 4005 in their Docker config and are not added to any reverse-proxy upstream. They reach the rest of the cluster only through the internal Valkey channel. They are not reachable from the public internet so they cannot be DDoSed on the API surface. + +Inter-node coordination flows through Valkey pub/sub — primary publishes assignments on `backup:submit` and `restore:submit`; processing nodes subscribe and report completions on the corresponding completion channels. Nodes never call each other over HTTP. + +### Bandwidth fairness + +Each node has a configured `NODE_NETWORK_THROUGHPUT_MBPS` budget (default 125 MB/s ≈ 1 Gbit/s). The `BandwidthManager` reserves 75% of the budget for data-moving operations and splits it equally across the active ones in real time. Ten parallel operations on a 1 Gbit/s node get ~100 Mbit/s each. Starting or finishing an operation recomputes every active rate so no single user can hog the pipe. + +A per-user lock caps each user to one in-flight operation of each kind so one tenant cannot fan out an unbounded number of parallel jobs against the node. + +## Consequences + +### Positive + +- **Many source IPs instead of one.** Processing nodes are deployed across multiple hosts and regions. A customer sees one of N IPs per backup so no single IP carries the whole fleet's reputation. +- **API and backup capacity scale independently.** Adding API capacity costs a small node behind CloudFlare. Adding backup capacity costs a gigabit-class processing node with no inbound exposure. +- **Fair sharing under load.** No tenant can starve another by starting a huge download. The split is computed per active session so the worst case for any user is a slow download, not a failed one. +- **One binary, one image.** Roles are environment configuration so deploys, tests and rollbacks operate on a single artifact. + +### Negative + +- **The deployment contract lives outside the code.** "Processing node has no port mapping and no proxy upstream" is enforced by ops, not by the binary. A misconfigured deploy can expose a processing node's API. The contract belongs in the deployment runbook and infra-as-code. +- **CloudFlare protects only inbound API traffic.** Backup throughput is outbound from processing nodes directly to customer databases and storage so it never traverses CloudFlare. Flooding a processing node's uplink is still possible; the mitigation is gigabit-class nodes and the per-node bandwidth budget. + +## References + +- [ADR-0004](./0004-focus-on-streaming-chunk-by-chunk-with-backpressure.md) — chunk-streaming with backpressure (the memory-safety foundation this scheduling sits on top of). diff --git a/adr/0007-cloud-readonly-encypted-backups.md b/adr/0007-cloud-readonly-encypted-backups.md new file mode 100644 index 0000000..3acb14b --- /dev/null +++ b/adr/0007-cloud-readonly-encypted-backups.md @@ -0,0 +1,41 @@ +# ADR-0007: Enforce read-only database access and encrypted backups in cloud + +- **Status:** Deprecated (cloud edition discontinued, 2026-06) +- **Date:** 2026-02-01 +- **Tags:** cloud, security, backups, access-control + +## Context + +Databasus is a backup tool — by definition it holds the keys to the most valuable data a customer has. We treat that responsibility seriously and invest in security at every layer. + +- **Application.** Secret encryption at rest, scoped credentials per tenant, audit logging on every privileged action. Every PR runs through CodeQL, Trivy, Dependabot, gitleaks, semgrep, CodeRabbit and Codex Security. Security review is mandatory on changed code. +- **Infrastructure.** Network isolation per [ADR-0006](./0006-cloud-load-and-traffic-distribution-between-nodes.md), processing nodes with no public ingress, CloudFlare in front of the API, hardened Docker images with non-root runtime, GitHub Actions pinned to commit SHAs, regular base-image refresh. +- **Access.** Least-privilege IAM on storage backends, short-lived tokens, MFA on operator accounts, no shared production credentials. + +That stack is genuinely strong and we keep raising it. The chance of a breach against any one layer is small. + +It is not zero. A kernel 0-day, a TLS-library CVE, etc. — any of these can land on a Monday morning and cut through layers we trusted on Sunday night. AI offensive tooling like Claude Mythos now chews through systems that were considered hardened a year ago, compressing the gap between disclosure and weaponisation to hours. On a long enough timeline some external dependency we do not own fails. + +We add one more layer that does not depend on any of the above holding: we structurally remove anything Databasus could do that would harm a customer. Even in the scenario where every other defence is bypassed, the attacker finds nothing useful at the end of the chain. + +## Decision + +Cloud mode enforces two hard rules at the application level. + +**Read-only database access.** Cloud customers can only connect Databasus to a database role with the read privileges sufficient to run the backup tool for that engine — `SELECT` plus the engine-specific reads needed by `pg_dump` / `mysqldump` / `mongodump`. Roles with write, DDL or superuser privileges are rejected at connection-setup time with a clear error. The check runs before the first backup is scheduled and is repeated on every credential rotation. + +**Encrypted backups only.** Every backup artifact produced in cloud mode is encrypted on the processing node before it leaves the process. "Backup without encryption" is not a supported option in cloud — the toggle does not exist in the cloud UI or API. + +Self-hosted instance mode keeps both options open. The operator owns the security perimeter and may choose write access or plain backups for their own valid reasons. + +## Consequences + +### Positive + +- **A Databasus compromise cannot corrupt customer data.** Even with full credentials in hand the attacker can only read. There is no write path through Databasus to customer production — the worst case is structurally bounded, not policy-bounded. +- **The 0.0001% case has a defined ceiling.** No single defence has to be perfect because each layer caps what the next failure can do. Weak independent guarantees compose into a strong one. +- **One answer to "can I give a write user?"** Cleaner support story, no edge-case configuration to debug. + +## References + +- [ADR-0006](./0006-cloud-load-and-traffic-distribution-between-nodes.md) — cluster topology and network isolation for cloud nodes (the perimeter this decision layers on top of). diff --git a/adr/0008-why-pg17-native-backups-with-mandatory-wal-summary.md b/adr/0008-why-pg17-native-backups-with-mandatory-wal-summary.md new file mode 100644 index 0000000..11cad9a --- /dev/null +++ b/adr/0008-why-pg17-native-backups-with-mandatory-wal-summary.md @@ -0,0 +1,109 @@ +# ADR-0008: Use PostgreSQL 17 native physical backups with mandatory WAL summary + +- **Status:** Accepted +- **Date:** 2026-05-26 +- **Tags:** backups, postgresql, physical, pitr + +## Context + +Databasus needs physical backups with incremental support and WAL streaming +to deliver PITR. Three implementation paths exist for that pipeline today: + +- **Custom block-level incremental** — read `$PGDATA` directly, hash blocks, + store block-level diffs in a proprietary format. This is what pgBackRest + (`--repo-block` since 2.46) and WAL-G do. The cost is a full backup format + with bundle / pagefile / manifest code, block-checksum tracking on the + client side and per-engine retention machinery. +- **Pre-PG-17 incrementals via WAL replay** — take periodic FULLs and treat + `FULL + raw WAL replay since stop_lsn` as a substitute for incremental. + The "incrementals" don't exist as artifacts; the user pays for them with + replay time on every restore. +- **PG 17 native incremental** — `pg_basebackup --incremental=` against the + previous backup's manifest, server-side WAL summarizer tracking changed + blocks, `pg_combinebackup` to materialise a chain into a runnable PGDATA. + Block-level granularity is computed by PostgreSQL itself from WAL summary + files in `$PGDATA/pg_wal/summaries/`. + +## Decision + +We use **PostgreSQL 17's native physical backup stack** for every physical +flow Databasus runs: `pg_basebackup` for FULLs, `pg_basebackup --incremental=` +for incrementals against the parent backup's manifest, `pg_receivewal` for +WAL streaming and `pg_combinebackup` on restore. No custom block-level +format, no pgBackRest-style block hashing, no WAL-G-style bundle code. + +The decision has two hard rules attached: + +**Physical backups refuse PG < 17.** The required binaries +(`pg_basebackup --incremental`, `pg_combinebackup`) and the required GUC +(`summarize_wal`) do not exist before PG 17. Connection setup refuses +non-17+ clusters with a clear error pointing at the logical path. + +**`summarize_wal = on` is mandatory whenever incrementals are selected.** +For backup types `FULL_AND_INCREMENTAL` and `FULL_INCREMENTAL_AND_WAL_STREAM`, +the database-setup flow refuses to save the config until the upstream reports +`summarize_wal = on`. The UI surfaces the fix-it command +(`ALTER SYSTEM SET summarize_wal = on; SELECT pg_reload_conf();`) and a +"try again" button. `FULL_ONLY` skips the check — no INCRs ever spawn so +no summaries are needed. + +We do not support `FULL + raw WAL replay` as a substitute for incrementals. +If the user picked incrementals the system delivers incrementals — refusing +to start without summaries is the gate that keeps the runtime in a single +reliable shape. + +## Alternatives considered + +- **Implement our own block-level incremental** (pgBackRest / WAL-G shape). + Rejected because PostgreSQL 17 ships the same capability natively. Writing + our own would mean owning bundle code, pagefile code, manifest code and a + separate backup format — large surface area to maintain and a separate + class of bugs (delta math, block-checksum drift, manifest format + versioning) — for no functional gain over what `pg_basebackup --incremental` + already delivers. +- **Accept incrementals without `summarize_wal = on` by falling back to + full WAL replay between FULLs.** Rejected because the fallback is + unreliable (no per-cycle manifest-level integrity check) and catastrophic + for RTO (replay of every WAL segment since the last FULL on every + restore). The user asked for incrementals — silently delivering "FULL + every cycle with slow restore" is worse than refusing the config. +- **Support physical backups against PG ≤ 16 via the legacy + `pg_basebackup` + WAL-replay path.** Rejected for the same RTO and + reliability reasons above, plus the per-version branching cost across the + whole physical pipeline. PG ≤ 16 users stay on logical backups, which + cover the same data with a different operational profile. + +## Consequences + +### Positive + +- **Smaller code surface.** No bundle / pagefile / block-checksum + machinery to maintain. The physical pipeline orchestrates PG-native + binaries (`pg_basebackup`, `pg_receivewal`, `pg_combinebackup`) and + records artifacts in storage; the format itself is PG's responsibility. +- **Battle-tested incremental math.** Block-level change tracking lives in + PostgreSQL's WAL summarizer — a server-side feature exercised by + upstream's test suite and the broader PG ecosystem. We inherit that + testing without paying for it. +- **Forward compatibility.** Improvements to `pg_basebackup` / + `pg_combinebackup` / summarizer in PG 18+ accrue to us automatically as + the bundled binaries roll forward. +- **Single runtime shape.** Mandatory `summarize_wal = on` collapses + "what happens when summaries are missing on the source" into one + failure mode (`SUMMARIZER_OFF` ⇒ refuse) instead of a fallback + matrix. + +### Negative + +- **PG 17+ requirement for physical.** Operators on PG 12–16 cannot use + physical backups in Databasus and stay on `pg_dump`. This is a real + exclusion — PG 16 was a current major as of late 2024 — but the trade-off + is deliberate: we built the physical pipeline on a feature set that did + not exist before PG 17, and back-porting it is the work we explicitly + chose not to do. +- **Managed PG that forbids `ALTER SYSTEM`.** On a managed PG where + `summarize_wal` cannot be flipped via SQL (RDS / Cloud SQL / Azure DB + parameter-group flows), the user must change the GUC through the + provider's UI before Databasus will save an incremental config. We + surface the command for self-managed clusters and the parameter name for + managed ones; we do not silently downgrade to FULL-only. \ No newline at end of file diff --git a/adr/0009-why-remote-physical-backups-instead-of-agents.md b/adr/0009-why-remote-physical-backups-instead-of-agents.md new file mode 100644 index 0000000..acc7ba9 --- /dev/null +++ b/adr/0009-why-remote-physical-backups-instead-of-agents.md @@ -0,0 +1,96 @@ +# ADR-0009: Run physical backups remotely instead of via an agent on the database host + +- **Status:** Accepted +- **Date:** 2026-05-26 +- **Tags:** backups, postgresql, physical, architecture, deployment + +## Context + +Physical backups built on PG 17 native (per +[ADR-0008](./0008-why-pg17-native-backups-with-mandatory-wal-summary.md)) +can be driven in two shapes: + +- **Agent-mode.** Install a Databasus-shipped binary on the same host as the + user's PostgreSQL. The agent reads `$PGDATA` / `pg_wal` locally, runs + `pg_basebackup` against `localhost`, ships artifacts to storage on the + user's behalf and coordinates with the central Databasus over an outbound + channel. This is the shape pgBackRest / Barman use, and the shape WAL_V1 + used inside Databasus before it was deleted. +- **Remote-mode.** The Databasus host itself runs `pg_basebackup` and + `pg_receivewal` against the user's PG over the wire, using the standard + replication protocol. The user's host runs nothing of ours. Setup on the + user's side is one role with `REPLICATION` plus a `replication`-line in + `pg_hba.conf`. + +We tried agent-mode in WAL_V1 and removed it. + +## Decision + +All Databasus physical backup flows run **remotely** from the Databasus +host against the user's PostgreSQL via the replication protocol. Databasus +never ships a per-host agent. For users whose PG is not reachable from the +Databasus host directly, the supported escape hatch is an **SSH tunnel** +from Databasus to a jump host inside the user's perimeter — same backup +code path, transport wrapped. + +Three reasons drive the choice: + +**1. Simpler setup for the user.** No binary install on the database host, +no init script, no agent upgrade cycle, no host-level capacity reservation. +The end-to-end setup is: create a role with `REPLICATION` privilege, add +one `replication` line to `pg_hba.conf` (or grant `rds_replication` on +managed PG), reload, paste creds into Databasus. Most managed PG providers +forbid running anything on the DB host anyway, so an agent shape would +either exclude managed PG entirely or fork into a remote-mode branch for +that case — which is the branch we now use universally. + +**2. Less code and a cleaner test surface.** A single execution path — +"Databasus host runs `pg_basebackup` / `pg_receivewal` against a remote +endpoint" — keeps the physical pipeline identical to how logical backups +already work. No second binary to build, version, sign and ship; no agent +self-update protocol; no E2E matrix that stands up a real agent next to +a real PG and exercises the cross-process protocol in CI. The CI matrix for +physical backups is the same matrix we already run for logical: spawn a PG +container, point Databasus at it, run the flow. + +**3. Privacy and closed-network deployments are already covered.** +Databasus is normally deployed inside the user's own perimeter — the +remote connection from Databasus to PG crosses a private network the user +controls, not the public internet. For the minority of deployments where +PG sits behind a stricter inner perimeter than Databasus itself, an SSH +tunnel from Databasus to a jump host inside the PG perimeter delivers the +same security profile as an agent would (no PG port exposed beyond the +inner perimeter, traffic encrypted, authentication via the user's own SSH +infrastructure) while keeping the backup code path unchanged. This covers +the long tail without forking the pipeline. + +## Alternatives considered + +- **Agent on the user's database host.** Rejected for the reasons above: + doubles the binary / release / CI surface, excludes managed PG by + construction (or forks into remote-mode anyway), and offers no + functional capability that remote-mode + SSH tunnel does not already + deliver. The historical WAL_V1 implementation is the concrete evidence — + it carried agent-token plumbing, a separate `agent/backup/` package, a + scheduler bail-out branch and per-method `if BackupType == WAL_V1 + { not supported }` shims through the codebase, all of which dissolved + cleanly when remote-mode replaced it. + +## Consequences + +### Positive + +- **One code path for physical backups.** The remote-execution shape + matches logical backups and the verification agent's restore flow, so + shared infrastructure (node assignment, dead-node detection, bandwidth + fairness, encryption, storage abstraction) carries over without + per-shape branching. +- **No agent release pipeline.** No second artifact to version, sign, + publish, document or auto-update across the installed base. The + Databasus binary is the only thing the user runs. +- **Managed PG works out of the box.** RDS / Cloud SQL / Azure DB ship + with the replication protocol exposed and host-side installs forbidden. + Remote-mode fits that shape natively; agent-mode would not. +- **Single test surface.** Physical CI exercises the same + "spawn PG container, point Databasus at it" matrix as logical, plus an + SSH-tunnel variant. No cross-process agent protocol to fuzz. \ No newline at end of file diff --git a/adr/0010-no-support-for-customer-tablespaces.md b/adr/0010-no-support-for-customer-tablespaces.md new file mode 100644 index 0000000..4f0a133 --- /dev/null +++ b/adr/0010-no-support-for-customer-tablespaces.md @@ -0,0 +1,37 @@ +# ADR-0010: No support for custom tablespaces in physical backups + +- **Status:** Accepted +- **Date:** 2026-05-26 +- **Tags:** backups, postgresql, physical, tablespaces + +## Context + +Physical backups stream `pg_basebackup --pgdata=- --format=tar` directly into storage with zero local disk (see [ADR-0004](./0004-focus-on-streaming-chunk-by-chunk-with-backpressure.md)). `pg_basebackup` writes one tar to stdout for the main data directory plus one tar per custom tablespace. Since stdout is a single stream `pg_basebackup` refuses `-D -` whenever any tablespace other than `pg_default` and `pg_global` exists. + +Two paths could preserve physical backups for those clusters: a bounded local staging buffer that catches the multi-file tar output and multipart-uploads each file in flight; or a custom implementation of the PG BASE_BACKUP replication protocol that bypasses `pg_basebackup` entirely (the path pgBackRest and WAL-G take). Both require significant new architecture. + +Custom tablespaces are a minority configuration even among production PG operators we target. Managed PG (RDS, Aurora, Cloud SQL, Azure DB) restricts or forbids them. Kubernetes operators (CloudNativePG, Zalando, Crunchy) deploy one PVC per instance. Modern self-hosted PG runs on a single large NVMe where ZFS or LVM has replaced the I/O-tiering motivation tablespaces historically served. + +The remaining slice is either legacy installations carrying a tablespace decision made years ago, or specialty workloads like Greenplum (which ships its own `gpbackup`) and TimescaleDB chunk-tiering (which sits on pgBackRest / WAL-G already and is not part of our target audience). + +## Decision + +Physical backups refuse clusters with any tablespace outside `pg_default` and `pg_global`. The refusal is a pre-flight check (`CheckNoCustomTablespaces` in `timeline.go`) that runs before any backup row is created. The scheduler logs `physical_backup_rejected_unsupported_tablespaces` with the offending spcnames and sends one notification per database per 24 hours. The user keeps the logical backup path which handles tablespaces transparently through `pg_dump`'s SQL emission. + +This is a permanent decision not a roadmap item. We will not revisit it without a concrete user request carrying real numbers — cluster size, tablespace count and willingness to provision staging disk. + +## Alternatives considered + +- **Bounded local staging.** Reserve a local volume on the Databasus host, let `pg_basebackup -D ` write multiple tars there, watch with inotify, multipart-upload to storage as files grow and delete after upload. Rejected because it doubles pipeline complexity (loses the kernel-pipe back pressure that comes free with stdout streaming), introduces an ENOSPC failure mode that depends on relative speed of write and upload and serves an audience that has not asked for it. +- **Own PG replication protocol (pgBackRest / WAL-G shape).** Parse `BASE_BACKUP` replication messages directly through `pglogrepl`, emit our own multi-tablespace archive format. Rejected as the explicit anti-direction in [ADR-0008](./0008-why-pg17-native-backups-with-mandatory-wal-summary.md) — we delegate the backup format to PG precisely so we do not own a parallel implementation of PG-internal structure that has to track every major release. + +## Consequences + +### Positive + +- Streaming with kernel-pipe back pressure stays the universal physical-backup pipeline. No staging-disk mode to test, document or fail at runtime. +- The metadata sidecar drops `Tablespaces` and `TablespaceLocation`. Restore-time tablespace mapping code is never written. A class of restore failures ("the original mount path no longer exists on the target host") is removed by construction. + +### Negative + +- Users with custom tablespaces who want physical / PITR get a clean refusal and must either drop the tablespaces or stay on logical backups. We estimate this affects at most 5–15% production setups. diff --git a/adr/0011-no-partial-wal-uploads.md b/adr/0011-no-partial-wal-uploads.md new file mode 100644 index 0000000..aca74e6 --- /dev/null +++ b/adr/0011-no-partial-wal-uploads.md @@ -0,0 +1,38 @@ +# ADR-0011: No partial WAL uploads to cloud storage + +- **Status:** Accepted +- **Date:** 2026-05-27 +- **Tags:** backups, postgresql, physical, wal, pitr + +## Context + +`pg_receivewal` streams WAL from the source cluster to Databasus in real time, writing into a `.partial` file that grows as the source writes. When the segment fills (default 16 MB) PostgreSQL rotates it and `pg_receivewal` renames the file to its final name. The plan's invariant — `.partial never uploaded` — means we only ship fully-rotated segments to cloud storage. + +That bounds cloud-side RPO by `wal_segment_size`. On an active OLTP cluster a segment fills in seconds and the gap is invisible. On an idle cluster the same 16 MB might accumulate over hours or days, and a source crash in that window can lose everything since the last completed rotation. + +PostgreSQL has a native knob for this exact problem: `archive_timeout`. Setting it on the source forces a WAL switch every N seconds regardless of fill, capping the size of any unrotated tail. Every mainstream physical backup tool (pgBackRest, WAL-G, Barman) takes the same shape we do — upload completed segments only, document `archive_timeout` as the RPO control. + +## Decision + +We upload completed WAL segments and nothing else. `.partial` stays a transient file on the Databasus host and never reaches storage. + +Customers who need a tighter RPO than `wal_segment_size` set `archive_timeout` on their source cluster (or the parameter-group equivalent on managed PG). The UI surfaces this as a one-line hint next to the physical-backup config: the SQL command for self-managed, the parameter-group key for RDS / Cloud SQL / Azure. + +Databasus does not call `pg_switch_wal()` on the source. RPO control lives where the data lives — on the customer's cluster. + +## Alternatives considered + +- **Partial-tip channel.** Snapshot the current `.partial` every N seconds, compress, encrypt, upload under a tip key that gets overwritten on each snapshot. Delete the tip when the underlying segment rotates and uploads normally. Rejected because it doubles the WAL pipeline: a new `physical_wal_partial_tips` table, a snapshotter goroutine alongside the existing streamer, lifecycle handover between tip-deleter and snapshotter, restore-side merge logic, plus ongoing storage PUT cost (~$8/day per 100 DBs at a 5-second interval on S3). None of pgBackRest, WAL-G or Barman ship this — we'd be inventing a non-standard mechanism for an effect `archive_timeout` already delivers. +- **Databasus-driven `pg_switch_wal()` on a timer.** Force rotation from Databasus every N seconds without touching the customer's GUC. Rejected because it mutates source-cluster state beyond what `REPLICATION` privilege normally implies, masks the real WAL behaviour from the operator (they see "RPO 5s" without understanding why, then get a surprise after failover or when Databasus is paused) and creates two competing rotation cadences (ours plus whatever the operator may set later). + +## Consequences + +### Positive + +- The WAL pipeline stays a one-direction conveyor: `pg_receivewal` produces completed segments, we ship them. No mutable artifacts in storage, no tip lifecycle, no merge step at restore time. Every row in `physical_wal_segments` is immutable once `file_name` is non-NULL. +- RPO control sits with the customer, where the storage cost of tighter RPO (more segments per day, each mostly zero-padded) is also paid. We don't make budget decisions for them. + +### Negative + +- Cloud-side RPO on an idle cluster with default settings can be arbitrarily large. Users who don't read the help text and run an idle DB will be unpleasantly surprised after a source-side disaster. +- Managed PG users who can't set `archive_timeout` through their provider's parameter group have no escape hatch. In practice every major managed PG (RDS, Cloud SQL, Azure DB) exposes the parameter, but we accept being blocked on the rare provider that doesn't. diff --git a/adr/0012-pg-basebackup-tar-fetch-and-manifest-reconstruction.md b/adr/0012-pg-basebackup-tar-fetch-and-manifest-reconstruction.md new file mode 100644 index 0000000..9fc9567 --- /dev/null +++ b/adr/0012-pg-basebackup-tar-fetch-and-manifest-reconstruction.md @@ -0,0 +1,146 @@ +# ADR-0012: pg_basebackup streams a single tar to stdout; backup_manifest is reconstructed on Databasus + +- **Status:** Accepted +- **Date:** 2026-05-27 +- **Tags:** backups, postgresql, physical, pg_basebackup, compression, manifest + +## Context + +[ADR-0008](./0008-why-pg17-native-backups-with-mandatory-wal-summary.md) +commits Databasus to driving PG's native physical-backup binaries; +[ADR-0009](./0009-why-remote-physical-backups-instead-of-agents.md) +commits to running them remotely from the Databasus host. Within those +constraints the FULL/INCR executor must satisfy three things at once: + +1. **No local staging.** The host serves many sources; staging a 3 TB + database per concurrent backup would need tens of TB of disk that + sits cold between backups. +2. **One self-contained artifact** per backup — base data plus the WAL + needed to make it consistent — ready for one-shot restore. +3. **A `backup_manifest` alongside every artifact.** + `pg_basebackup --incremental=` needs the parent + manifest to find changed blocks; no manifest means no incrementals, + which ADR-0008 forbids. + +`pg_basebackup` exposes its features through CLI flags, and the wrong +combination silently breaks one of these. This ADR records the chosen +flag set — and the one conflict (server-side compression vs. embedded +manifest) that forces us to rebuild the manifest ourselves. + +## Decision + +Invoke `pg_basebackup` with +`--pgdata=- --format=tar --wal-method=fetch --compress=server-zstd:5 --no-manifest` +(INCR adds `--incremental=`). The compressed stdout is +teed on Databasus into two independent branches: one uploads the bytes +to storage as-is, the other decompresses and walks the tar to +reconstruct `backup_manifest`. Each flag earns its place: + +**`--format=tar --pgdata=-`** streams one tar to stdout, piped straight +into the storage `SaveFile(io.Reader)`. `--format=plain` would need a +real disk path (breaks no-staging) or ~1 TB of uncompressed staging. +ADR-0010 already forbids custom tablespaces, so there is exactly one +tar — no concatenation on our side. + +**`--wal-method=fetch`** collects WAL through the same replication +connection at end-of-backup and inlines it as `pg_wal/...` tar entries, +keeping the single-stream invariant. `--wal-method=stream` needs a +second output channel that stdout-tar mode refuses. Fetch's one +weakness — the source must retain WAL until end-of-backup — is covered +by the per-backup replication slot (`backup_slot.go`), which pins WAL +for the backup's duration. + +**`--compress=server-zstd:5`** — on TB-scale databases the +PG→Databasus link dominates, and compressing on the source sends ~1/3 +of the bytes: + +| Link | client-zstd | server-zstd | +|----------------|-------------|-------------| +| 1 Gbit/s LAN | ~7 hours | ~2.3 hours | +| 100 Mbit/s WAN | ~3 days | ~22 hours | + +More PG-side CPU for far less wire time is the right trade on +multi-hour windows, so server-side compression is mandatory here. +Because it depends on the source build, the codec degrades +`server-zstd:5 → server-gzip:6 → none`. + +**`--no-manifest` + reconstruction.** Server compression plus +tar-to-stdout makes pg_basebackup refuse to embed the manifest +(`cannot inject manifest into a compressed tar file`): the stream is +already compressed when bytes leave PG, so it cannot re-open it to +append the trailing manifest entry, and there is no side-channel flag +under `--pgdata=-`. `--no-manifest` removes the conflict but takes +manifests away, which breaks incrementals — so we rebuild it. Every +manifest field is derivable on Databasus from the tar stream plus a +couple of values already on hand (the cluster's system identifier and +the backup's LSN range), targeting PG's version-2 format. +(`--manifest-checksums` is itself incompatible with `--no-manifest`, so +it is omitted; our serializer fixes the per-file checksum at SHA-256.) + +We don't aim for byte-identity with PG's own manifest — only a valid, +self-consistent one that truthfully describes the artifact. +`pg_verifybackup` validates that directly: pointed at the produced +compressed-tar artifact plus its reconstructed sidecar (`-m`) it +recomputes every checksum and size, the file set and the manifest's +own SHA-256, so it passes only if the reconstruction is correct — and +catches PG format-drift in CI before production. Reading a compressed +tar needs **PG 18**'s `pg_verifybackup` (tar-format support landed in +PG 18), a test-only tool dependency even though the executor drives +PG 17. + +The reconstructed manifest is a separate sidecar object alongside the +artifact, encrypted independently with its own key material. INCR +downloads the sidecar, materialises it as a temp file and passes it to +`--incremental=`. + +## Alternatives considered + +- **Stay on `--compress=client-zstd:5`.** Zero new code. Rejected: + 7+ hours per FULL on gigabit LAN is not viable as fleets grow into + TB-scale databases. WAN is unusable. + +- **Local staging (`--pgdata= --compress=server-zstd:5`).** PG + emits its own manifest as a side file `backup_manifest`; no + reconstruction needed. Rejected because it requires ~1 TB of + staging disk per concurrent backup — exactly the constraint the + streaming pipeline was designed to avoid. A backup host serving 10 + sources would need tens of TB of staging-only disk sitting cold + between backups. + +- **Speak the replication protocol directly (pgMoneta-style).** Send + `BASE_BACKUP COMPRESSION 'server-zstd:5' MANIFEST 'yes'` over a + libpq replication connection; manifest arrives as a separate + CopyData stream byte-exact from PG. Rejected because it requires + re-implementing pg_basebackup's full surface — multi-tablespace + handling, WAL fetch trailer, error recovery, incremental protocol — + and maintaining that against every PG release. The single benefit + (no reconstruction) costs 2-3 KLOC of replication-protocol code + carried for our use only. The spirit of ADR-0008 — use PG's native + binaries, don't fork them — applies; if we ever need features + pg_basebackup cannot provide (mid-stream heartbeats for WAN, + structured progress events) this is the path we revisit. + +## Consequences + +### Positive + +- PG→Databasus traffic compressed ~3x. 1-3 TB backups complete in + hours, not days; WAN deployments become viable. +- Self-contained tar artifact preserved (base + WAL + reconstructed + manifest sidecar). +- Incremental chain unblocked — the manifest sidecar is consumed by + `--incremental=` exactly as PG's own would have been. +- No new deployment requirement. pg_basebackup remains the workhorse + and keeps owning retry, error mapping and multi-tablespace handling + for free. + +### Negative + +- Manifest-reconstruction code (codec reader + tar walker + PG-format + serialiser) to maintain. +- Reconstruction-correctness risk against PG's manifest format, + mitigated by the `pg_verifybackup` gate (PG 18): a drifted + reconstruction fails in CI, not production. +- Net CPU on Databasus to decompress and SHA-256 the full backup + contents. Acceptable on a dedicated backup host; the levers are the + server-side zstd level and the manifest-path worker count. diff --git a/adr/0013-reuse-shared-in-memory-test-databases.md b/adr/0013-reuse-shared-in-memory-test-databases.md new file mode 100644 index 0000000..9e11801 --- /dev/null +++ b/adr/0013-reuse-shared-in-memory-test-databases.md @@ -0,0 +1,131 @@ +# ADR-0013: Reuse one in-memory test database per version instead of one container per test + +- **Status:** Accepted +- **Date:** 2026-06-08 +- **Deciders:** Backend team +- **Tags:** backend, testing, ci + +## Context + +Our database tests run the same checks against many versions of each engine (PostgreSQL, MySQL, +MariaDB, MongoDB). We tried two setups before this one and both broke. + +**Before — one big docker-compose.** We started every version of every engine up front and kept them +all running. That is ~50 containers alive at the same time and a 16 GB CI machine ran out of memory: + +``` +docker-compose up (everything, all the time) + postgres:12 .. postgres:18 + mysql:5.7 .. mysql:8.4 + mariadb:10.6 .. mariadb:12.0 + mongo:4.0 .. mongo:8.0 + ────────────────────────────── + ~50 containers alive at once → out of RAM +``` + +**Naive fix — a fresh container per test.** Every single test booted its own database and killed it +afterwards. A database cold-starts in 20–60s and one package has 40+ tests, so the suite hit the +15-minute timeout: + +``` +test 1 → boot DB → run → kill DB +test 2 → boot DB → run → kill DB +... (40+ times per package) → too slow +``` + +We want both: fast tests and bounded memory, without tests stepping on each other. + +## Decision + +For each database version we boot one container, run all the test functions against it, then shut it +down before the next version: + +``` +boot postgres:16 (once) + ├─ run test 1 + ├─ run test 2 + └─ run test 3 +shut down postgres:16 +→ boot postgres:17 ... +``` + +Two more things make this fast and safe: the database files live in **RAM** (so each boot and restore +is quick) and the test packages run **in parallel** (so we get the speed back) while each one stays +isolated (so peak memory stays bounded). + +- **One container per version, reused across the version's tests.** Each matrix package declares its + version list once and loops it with an outer `t.Run`. Inside that subtest it boots the server with + `StartPostgres` / `StartMysql` / … (`containers/{postgres,mysql,mariadb,mongodb}.go`), then runs the + former test functions as inner subtests. `StartXxx` registers `t.Cleanup`, which Go runs when the + version subtest returns — so the container is torn down before the next version boots and **only one + matrix container is alive per package at a time**. Each `go test` package is its own process, so its + containers belong only to it. The one rule for test authors: when you create a fixed-name object (a + table or user without a random suffix), start with `DROP ... IF EXISTS` — within a version the server + is reused and the subtests run one after another. + +- **Database files in RAM.** Each engine's data dir is mounted on tmpfs `rw,size=512m` (size is pinned — + otherwise Docker reserves half the host RAM per container). Crash-safety is off because we throw these + servers away: Postgres `fsync=off / full_page_writes=off / synchronous_commit=off`; MySQL & MariaDB + `innodb-flush-log-at-trx-commit=0 / innodb-doublewrite=0 / sync-binlog=0 / skip-log-bin`. Files: + `containers/{postgres,mysql,mariadb,mongodb}.go`. + +- **Run packages in parallel.** `go test -p=8` (`TEST_PARALLEL_WORKERS` in `backend/Makefile`). One + matrix container is alive per package, so peak memory stays around 10–11 GB instead of the old + all-at-once that ran out of RAM. + +### How parallel workers stay isolated + +Eight packages run at the same time and share the same Postgres, Valkey and backup infrastructure, so +each one grabs its own **worker slot** using a Postgres advisory lock +(`config.go` → `applyTestWorkerSlot` / `claimTestWorkerSlot`). The slot gives it a private copy of +everything shared: + +- its own **metadata database**, named `…_w{slot}`; +- its own **Valkey/Redis database** (the slot number); +- its own **cache prefix** `"w{slot}:"`, which also tags the **backup-node registry** + (`backups/backups/backuping/nodes/registry.go`) — every Redis key and pub/sub channel. + +So two packages running side by side never touch each other's databases, cache entries, backup nodes or +pub/sub channels. + +## Alternatives considered + +- **docker-compose, all versions up at once.** Rejected: ~50 containers exhaust RAM on 16 GB CI and the + packages share one set of databases with no isolation. +- **A fresh container per test.** Rejected: 40+ boots per package × 20–60s cold start → 15-minute + timeouts. +- **Database files on disk instead of RAM.** Rejected: the slow part is fsync on cold start and on the + write-heavy restore; RAM removes it for servers we throw away anyway. + +## Consequences + +### Positive + +- The two packages that used to time out at 900s now finish in ~30s; the whole suite went from timing + out to ~1.5–4 minutes. +- Peak memory is bounded and predictable — one matrix container per package — and parallel packages are + fully isolated. + +### Negative + +- Because a server is reused across a version's subtests, authors must `DROP ... IF EXISTS` for + fixed-name objects and must not run those subtests with `t.Parallel`. +- Data in RAM is volatile — fine for throwaway test servers, but there is no crash safety. +- If a CI runner is short on memory, lower `TEST_PARALLEL_WORKERS` (e.g. to 6). + +### Neutral / follow-ups + +- On a hard `-timeout` or `os.Exit`, `t.Cleanup` is skipped; the Makefile/CI label-sweep cleans up any + leftover containers. + +## References + +- `backend/internal/util/testing/containers/{postgres,mysql,mariadb,mongodb}.go` (the `StartXxx` helpers) +- `backend/internal/features/tests/logical/postgresql/backup_restore_test.go` — the per-version + orchestrator pattern +- `backend/internal/features/tests/physical/postgresql/{pg17,pg18}` — the physical backup→restore + E2E split one package per PostgreSQL major, so the two versions run as isolated, parallel test + binaries (each with its own control plane and throwaway source + restore-target containers) +- `backend/internal/config/config.go` (`applyTestWorkerSlot` / `claimTestWorkerSlot`) +- `backend/internal/features/backups/backups/backuping/nodes/registry.go` +- `backend/Makefile` diff --git a/agent/verification/.dockerignore b/agent/verification/.dockerignore new file mode 100644 index 0000000..c2cb149 --- /dev/null +++ b/agent/verification/.dockerignore @@ -0,0 +1,23 @@ +# Local dev state — regenerated at runtime, never belongs in a build context +databasus-verification.log +databasus-verification.log.old +databasus-verification.daemon.log +databasus-verification.lock +databasus-verification.json + +# Local build artifacts (host binaries) +databasus-verification-agent + +# E2E working directories +e2e/artifacts/ + +# VCS and editor +.git/ +.gitignore +.idea/ +.vscode/ + +# Docs / env +*.md +.env +.env.example diff --git a/agent/verification/.env.example b/agent/verification/.env.example new file mode 100644 index 0000000..63f07af --- /dev/null +++ b/agent/verification/.env.example @@ -0,0 +1,3 @@ +DATABASUS_URL= +AGENT_ID= +AGENT_TOKEN= diff --git a/agent/verification/.gitignore b/agent/verification/.gitignore new file mode 100644 index 0000000..5758a36 --- /dev/null +++ b/agent/verification/.gitignore @@ -0,0 +1,18 @@ +# Runtime state — regenerated at runtime, never committed +databasus-verification.json +databasus-verification.log +databasus-verification.log.old +databasus-verification.daemon.log +databasus-verification.lock +.env +.test-tmp/ + +# Local build artifacts (host binaries) +databasus-verification-agent + +# E2E working directories +e2e/artifacts/ + +# Editor / VCS +.idea/ +.vscode/ diff --git a/agent/verification/.golangci.yml b/agent/verification/.golangci.yml new file mode 100644 index 0000000..6889079 --- /dev/null +++ b/agent/verification/.golangci.yml @@ -0,0 +1,41 @@ +version: "2" + +run: + timeout: 5m + tests: false + concurrency: 4 + +linters: + default: standard + enable: + - funcorder + - bodyclose + - errorlint + - gocritic + - unconvert + - misspell + - errname + - noctx + - modernize + + settings: + errcheck: + check-type-assertions: true + +formatters: + enable: + - gofumpt + - golines + - gci + + settings: + golines: + max-len: 120 + gofumpt: + module-path: databasus-verification-agent + extra-rules: true + gci: + sections: + - standard + - default + - localmodule diff --git a/agent/verification/CLAUDE.md b/agent/verification/CLAUDE.md new file mode 100644 index 0000000..96bb464 --- /dev/null +++ b/agent/verification/CLAUDE.md @@ -0,0 +1,289 @@ +# Verification agent guidelines (Go CLI) + +Coding standards for the Databasus verification agent — a Go CLI worker that runs on a cloud-managed Linux VM, reports its capacity to the backend over HTTP, and (once the restore phase lands) claims and verifies backups. It has no Gin HTTP server and owns no database schema. The long-running goroutine in this phase is the **capacity heartbeat loop**; the claim/report runner arrives with the restore phase. There is no Windows daemon — the agent runs in the foreground under systemd or as a container. + +For project-wide engineering philosophy, naming, and lint/format commands, see the root `CLAUDE.md`. For the backend (Gin/GORM/Swagger) ruleset, see `backend/CLAUDE.md`. + +--- + +## Table of Contents + +- [Spacing between logical statements](#spacing-between-logical-statements) +- [Comments](#comments) +- [File organization](#file-organization) +- [Background services](#background-services) +- [Testing](#testing) +- [Time handling](#time-handling) +- [Logging](#logging) +- [Modern Go](#modern-go) + +--- + +## Spacing between logical statements + +Add blank lines between logical blocks so the flow is visible at a glance: + +- before the final `return` +- after variable declarations, before they're used +- between error handling and subsequent logic +- between distinct logical operations + +Bad: + +```go +func encodeMessages(messages []Message) (string, error) { + if len(messages) > 0 { + messagesBytes, err := json.Marshal(messages) + if err != nil { + return "", err + } + return string(messagesBytes), nil + } + return "", nil +} +``` + +Good: + +```go +func encodeMessages(messages []Message) (string, error) { + if len(messages) > 0 { + messagesBytes, err := json.Marshal(messages) + if err != nil { + return "", err + } + + return string(messagesBytes), nil + } + + return "", nil +} +``` + +--- + +## Comments + +- **No obvious comments** — don't restate what the code already shows. +- **Explain *why*, not *what*** — code shows what happens; comments explain business rules, hidden constraints, or non-obvious optimizations. +- **Prefer refactoring over commenting** — better names or smaller functions usually beat a comment. +- **Complex algorithms deserve comments** — formulas, business rules, non-obvious optimizations. +- **No "Summary" / "Conclusion" sections in `.md` files** unless explicitly requested. + +Bad (each comment restates the function name): + +```go +// Send heartbeat +sendHeartbeat(request) + +// CreateValidLogItems creates valid log items for testing +func CreateValidLogItems(count int, uniqueID string) []LogItemRequestDTO { +``` + +--- + +## File organization + +One responsibility per file. Don't dump a whole package into one file — split +by role so a reader can find a type by its filename. Conventional names within +a feature package: + +- `doc.go` — package doc comment, once the package spans more than one file +- `.go` — the core type and its methods (the orchestrator/executor) +- `dto.go` — request/response and cross-package data + interface seams +- `errors.go` — sentinel errors (`var Err... = errors.New(...)`) +- `enums.go` — typed-constant groups (`type Status string` + its values) +- `constants.go` — package-level constants that aren't an enum +- background loops, reapers, and pools get their own file (`reaper.go`, `pool.go`) + +Only create a file when there is real content for it — an empty `enums.go` or +`constants.go` is noise, not structure. Test files mirror the source split: +`restorer.go` → `restorer_test.go`, `diskexhaustion.go` → +`diskexhaustion_test.go`. + +--- + +## Background services + +The agent ships at least one long-running goroutine (the capacity `Heartbeater`; also `BackgroundUpgrader`). Calling `Run()` twice on the same instance is always a bug — duplicate goroutines leak resources and corrupt state. **Always panic; never just log a warning.** + +```go +type Heartbeater struct { + // ... + hasRun atomic.Bool +} + +func (h *Heartbeater) Run(ctx context.Context) { + if h.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", h)) + } + + ticker := time.NewTicker(heartbeatInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + h.beat(ctx, logger) + } + } +} +``` + +`atomic.Bool.Swap(true)` does the check-and-set atomically — no `sync.Once` needed. + +--- + +## Testing + +**Always run tests after writing them and verify they pass.** + +### Naming + +- `Test_WhatWeDo_WhatWeExpect` +- `Test_WhatWeDo_WhichConditions_WhatWeExpect` + +Examples: `Test_DeriveCapacity_WhenConcurrentJobsExceedCPU_ReturnsError`, `Test_ValidateTransport_WhenHttpAndNotTTYWithoutFlag_FailsFast`, `Test_Heartbeat_WhenCalled_SendsFlatEnvelopeWithBearerAndAgentPath`, `Test_BackgroundUpgrader_WhenRunCalledTwice_Panics`. + +### Where tests live + +- **Unit / package tests**: alongside the code, named `*_test.go` (e.g. `internal/features/heartbeat/heartbeat_test.go`). +- **End-to-end tests**: under `agent/verification/e2e/`. Run via `make e2e` / `make e2e-clean`. + +The agent has no HTTP API of its own, so there are no controller tests — test exported functions of the packages, the CLI commands, and the HTTP-client/IO logic directly. + +### Refactor tests as you touch them + +When editing existing tests, look for repetitive setup that should become a helper, oversized tests that should be split, and similar patterns across files that should be consolidated. Helpers live in a `testing.go` file inside the package. + +### Clean up test data + +If a test creates files, processes, or external state, clean it up via `t.Cleanup(...)` or `defer`. Skip cleanup only when the test uses an isolated tempdir the OS reclaims, or when it explicitly validates a failure path where cleanup isn't possible. + +--- + +## Time handling + +Always use `time.Now().UTC()` instead of `time.Now()` to keep timezones consistent across the application. + +--- + +## Logging + +We use `log/slog`. Three rules. + +### 1. Scope IDs early via `logger.With(...)` + +Attach `agent_id`, `verification_id`, `upgrade_target`, etc. as soon as you know them so every downstream line carries them automatically. For background services, also scope `job_id` (fresh UUID per execution) and `job_name` (stable snake_case const) in `Run()`. + +```go +const jobName = "verification_heartbeat" + +func (h *Heartbeater) Run(ctx context.Context) { + logger := h.log.With("job_id", uuid.New(), "job_name", jobName) + + logger.Info("heartbeat loop started") + // every subsequent log carries job_id + job_name +} +``` + +### 2. Values in message, IDs and errors as kv pairs + +Sizes, counts, and status transitions go into the message via `fmt.Sprintf`. IDs and errors stay as structured kv pairs so they're searchable in log aggregation tools. + +```go +// good +logger.Info(fmt.Sprintf("heartbeat ok: last_seen_at=%s", seenAt)) +logger.Info("agent registered", "agent_id", agentID) +logger.Error("heartbeat failed", "error", err) + +// bad — ID buried in the message string, error formatted instead of attached +logger.Info(fmt.Sprintf("agent registered %s", agentID)) +logger.Error(fmt.Sprintf("heartbeat failed: %v", err)) +``` + +### 3. Style and level + +- All keys `snake_case` (`agent_id`, `verification_id`) — never camelCase. +- Messages start lowercase, no trailing period. +- **Debug**: routine ops, function entry, query result counts. +- **Info**: significant state changes, completed actions (`"agent started"`, `"heartbeat loop stopped"`). +- **Warn**: degraded but recoverable (`"heartbeat failed, will retry next tick"`, `"upgrade skipped: same version"`). +- **Error**: failures that need attention (`"auto-update failed"`, `"failed to re-exec after upgrade"`). + +### What never goes into logs + +- **Passwords, tokens, API keys, full `Authorization` headers** — centralize redaction; don't redact ad-hoc at call sites (config logging masks the token via `maskSensitive`). +- **Full request / response bodies** — log only the fields you actually need. + +--- + +## Modern Go + +Prefer modern stdlib idioms over manual equivalents. + +### `slices` package — avoid manual loops + +```go +slices.Contains(items, x) +slices.Index(items, x) // returns index or -1 +slices.IndexFunc(items, func(item T) bool { return item.ID == id }) +slices.SortFunc(items, func(a, b T) int { return cmp.Compare(a.X, b.X) }) +slices.Sort(items) +slices.Max(items) / slices.Min(items) +slices.Reverse(items) // in-place +slices.Compact(items) // remove consecutive duplicates +slices.Clone(s) +slices.Clip(s) +``` + +### Quick wins + +- `any` instead of `interface{}`. +- `for i := range len(items)` instead of `for i := 0; i < len(items); i++`. +- `sync.OnceFunc(fn)` / `sync.OnceValue(fn)` instead of `sync.Once` + wrapper. +- `t.Context()` in tests instead of `context.WithCancel(context.Background())` + `defer cancel()` — auto-cancels at test end. +- `wg.Go(fn)` instead of `wg.Add(1)` + `go func() { defer wg.Done(); fn() }()`. +- `slog.DiscardHandler` for a no-op logger in tests. + +### `context` helpers + +```go +stop := context.AfterFunc(ctx, cleanup) // run cleanup on cancellation +ctx, cancel := context.WithTimeoutCause(parent, d, ErrTimeout) // timeout with cause +ctx, cancel := context.WithDeadlineCause(parent, deadline, ErrDeadline) // deadline with cause +``` + +### `omitzero` instead of `omitempty` for non-nullable types + +`omitempty` is broken for `time.Duration`, `time.Time`, structs, slices, and maps — it doesn't omit a zero value. Use `omitzero`: + +```go +// good +type Config struct { + Timeout time.Duration `json:"timeout,omitzero"` + CreatedAt time.Time `json:"createdAt,omitzero"` +} + +// bad +type Config struct { + Timeout time.Duration `json:"timeout,omitempty"` // broken for Duration! + CreatedAt time.Time `json:"createdAt,omitempty"` +} +``` + +### `new(val)` for pointer literals (Go 1.26+) + +`new()` accepts expressions, eliminating the temp-variable pattern: + +```go +// good +cfg := Config{Timeout: new(30), Debug: new(true)} + +// bad +timeout := 30 +debug := true +cfg := Config{Timeout: &timeout, Debug: &debug} +``` diff --git a/agent/verification/Makefile b/agent/verification/Makefile new file mode 100644 index 0000000..04aff9a --- /dev/null +++ b/agent/verification/Makefile @@ -0,0 +1,44 @@ +.PHONY: run build test lint e2e e2e-clean + +-include .env +export + +CLEAN_ARTIFACTS := rm -rf e2e/artifacts +DOWN_QUIET := cd e2e && docker compose down -v --remove-orphans 2>/dev/null +DOWN_CLEAN_QUIET := cd e2e && docker compose down -v --rmi local 2>/dev/null + +run: + go run ./cmd run \ + --databasus-host $(DATABASUS_URL) \ + --agent-id $(AGENT_ID) \ + --token $(AGENT_TOKEN) \ + --max-cpu 4 \ + --max-ram-mb 4096 \ + --max-disk-gb 1000 \ + --max-concurrent-jobs 2 \ + --allow-insecure-http \ + --skip-update + +build: + CGO_ENABLED=0 go build -ldflags "-X main.Version=$(VERSION)" -o databasus-verification-agent ./cmd + +test: + go test -race -count=1 -failfast ./internal/... + +lint: + golangci-lint fmt ./cmd/... ./internal/... ./e2e/... && golangci-lint run ./cmd/... ./internal/... ./e2e/... + +e2e: + -$(DOWN_QUIET) + -$(CLEAN_ARTIFACTS) + cd e2e && docker compose build --no-cache e2e-mock-server + cd e2e && docker compose build + cd e2e && docker compose run --rm e2e-agent-builder + cd e2e && docker compose run --rm e2e-fixture-gen + cd e2e && docker compose up -d e2e-mock-server + cd e2e && docker compose run --rm e2e-agent-runner + cd e2e && docker compose down -v + +e2e-clean: + -$(DOWN_CLEAN_QUIET) + -$(CLEAN_ARTIFACTS) diff --git a/agent/verification/cmd/main.go b/agent/verification/cmd/main.go new file mode 100644 index 0000000..2a0c7be --- /dev/null +++ b/agent/verification/cmd/main.go @@ -0,0 +1,209 @@ +package main + +import ( + "errors" + "flag" + "fmt" + "log/slog" + "os" + "path/filepath" + "strings" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/start" + "databasus-verification-agent/internal/features/upgrade" + "databasus-verification-agent/internal/logger" +) + +var Version = "dev" + +func main() { + if len(os.Args) < 2 { + printUsage() + os.Exit(1) + } + + switch os.Args[1] { + case "start": + runStart(os.Args[2:]) + case "run": + runAgent(os.Args[2:]) + case "_run": + runDaemon(os.Args[2:]) + case "stop": + runStop() + case "status": + runStatus() + case "version": + fmt.Println(Version) + default: + fmt.Fprintf(os.Stderr, "unknown command: %s\n", os.Args[1]) + printUsage() + os.Exit(1) + } +} + +func runStart(args []string) { + fs := flag.NewFlagSet("start", flag.ExitOnError) + + isSkipUpdate := fs.Bool("skip-update", false, "Skip auto-update check") + + cfg := &config.Config{} + cfg.LoadFromJSONAndArgs(fs, args) + + if err := cfg.SaveToJSON(); err != nil { + fmt.Fprintf(os.Stderr, "Failed to save config: %v\n", err) + } + + log := logger.GetLogger() + + isDev := checkIsDevelopment() + runUpdateCheck(cfg.DatabasusHost, *isSkipUpdate, isDev, log) + + if err := start.Start(cfg, Version, isDev, log); err != nil { + if errors.Is(err, upgrade.ErrUpgradeRestart) { + reexecAfterUpgrade(log) + } + + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } +} + +func runAgent(args []string) { + fs := flag.NewFlagSet("run", flag.ExitOnError) + + isSkipUpdate := fs.Bool("skip-update", false, "Skip auto-update check") + + cfg := &config.Config{} + cfg.LoadFromJSONAndArgs(fs, args) + + if err := cfg.SaveToJSON(); err != nil { + fmt.Fprintf(os.Stderr, "Failed to save config: %v\n", err) + } + + log := logger.GetLogger() + + isDev := checkIsDevelopment() + runUpdateCheck(cfg.DatabasusHost, *isSkipUpdate, isDev, log) + + if err := start.RunDaemon(cfg, Version, isDev, log); err != nil { + if errors.Is(err, upgrade.ErrUpgradeRestart) { + reexecAfterUpgrade(log) + } + + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } +} + +func runDaemon(args []string) { + fs := flag.NewFlagSet("_run", flag.ExitOnError) + + if err := fs.Parse(args); err != nil { + os.Exit(1) + } + + log := logger.GetLogger() + + cfg := &config.Config{} + cfg.LoadFromJSON() + + if err := start.RunDaemon(cfg, Version, checkIsDevelopment(), log); err != nil { + if errors.Is(err, upgrade.ErrUpgradeRestart) { + reexecAfterUpgrade(log) + } + + log.Error("Agent exited with error", "error", err) + os.Exit(1) + } +} + +func runStop() { + log := logger.GetLogger() + + if err := start.Stop(log); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } +} + +func runStatus() { + log := logger.GetLogger() + + if err := start.Status(log); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } +} + +func printUsage() { + fmt.Fprintln(os.Stderr, "Usage: databasus-verification-agent [flags]") + fmt.Fprintln(os.Stderr, "") + fmt.Fprintln(os.Stderr, "Commands:") + fmt.Fprintln(os.Stderr, " start Start the verification agent in the background (daemon)") + fmt.Fprintln(os.Stderr, " run Run the verification agent in the foreground (systemd / container)") + fmt.Fprintln(os.Stderr, " stop Stop a running agent") + fmt.Fprintln(os.Stderr, " status Show agent status") + fmt.Fprintln(os.Stderr, " version Print agent version") +} + +func runUpdateCheck(host string, isSkipUpdate, isDev bool, log *slog.Logger) { + if isSkipUpdate { + return + } + + if host == "" { + return + } + + apiClient := api.NewClient(host, "", "", log) + + isUpgraded, err := upgrade.CheckAndUpdate(apiClient, Version, isDev, log) + if err != nil { + log.Error("Auto-update failed", "error", err) + os.Exit(1) + } + + if isUpgraded { + reexecAfterUpgrade(log) + } +} + +func checkIsDevelopment() bool { + dir, err := os.Getwd() + if err != nil { + return false + } + + for range 3 { + if data, err := os.ReadFile(filepath.Join(dir, ".env")); err == nil { + return parseEnvMode(data) + } + + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err == nil { + return false + } + + dir = filepath.Dir(dir) + } + + return false +} + +func parseEnvMode(data []byte) bool { + for line := range strings.SplitSeq(string(data), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + + parts := strings.SplitN(line, "=", 2) + if len(parts) == 2 && strings.TrimSpace(parts[0]) == "ENV_MODE" { + return strings.TrimSpace(parts[1]) == "development" + } + } + + return false +} diff --git a/agent/verification/cmd/reexec.go b/agent/verification/cmd/reexec.go new file mode 100644 index 0000000..4cc49c3 --- /dev/null +++ b/agent/verification/cmd/reexec.go @@ -0,0 +1,22 @@ +package main + +import ( + "log/slog" + "os" + "syscall" +) + +func reexecAfterUpgrade(log *slog.Logger) { + selfPath, err := os.Executable() + if err != nil { + log.Error("Failed to resolve executable for re-exec", "error", err) + os.Exit(1) + } + + log.Info("Re-executing after upgrade...") + + if err := syscall.Exec(selfPath, os.Args, os.Environ()); err != nil { + log.Error("Failed to re-exec after upgrade", "error", err) + os.Exit(1) + } +} diff --git a/agent/verification/e2e/.env.example b/agent/verification/e2e/.env.example new file mode 100644 index 0000000..872b2a6 --- /dev/null +++ b/agent/verification/e2e/.env.example @@ -0,0 +1,4 @@ +# Host port the e2e mock-server publishes. The agent-runner reaches it via +# 127.0.0.1: (network_mode: host). Defaults to 4050 in docker-compose.yml; +# override here only if 4050 is already taken on the host. +E2E_MOCK_SERVER_PORT=4050 diff --git a/agent/verification/e2e/Dockerfile.agent-builder b/agent/verification/e2e/Dockerfile.agent-builder new file mode 100644 index 0000000..d636c74 --- /dev/null +++ b/agent/verification/e2e/Dockerfile.agent-builder @@ -0,0 +1,14 @@ +# Builds verification agent binaries with different versions so the upgrade +# behaviour (v1 -> v2) can be tested. Built from ./cmd (not ./cmd/main.go) +# because the package has platform-split files. +FROM golang:1.26.3-alpine AS build +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN CGO_ENABLED=0 go build -ldflags "-X main.Version=v1.0.0" -o /out/agent-v1 ./cmd +RUN CGO_ENABLED=0 go build -ldflags "-X main.Version=v2.0.0" -o /out/agent-v2 ./cmd + +FROM alpine:3.21 +COPY --from=build /out/ /out/ +CMD ["cp", "-v", "/out/agent-v1", "/out/agent-v2", "/artifacts/"] diff --git a/agent/verification/e2e/Dockerfile.agent-runner b/agent/verification/e2e/Dockerfile.agent-runner new file mode 100644 index 0000000..9a47325 --- /dev/null +++ b/agent/verification/e2e/Dockerfile.agent-runner @@ -0,0 +1,13 @@ +# Runs the verification agent against the mock server. The agent spawns +# ephemeral Postgres containers on the host daemon (docker socket mounted by +# compose); docker-cli here is for the e2e scripts themselves (leak_check +# inspects containers/networks/volumes by label after each terminal outcome). +FROM debian:bookworm-slim + +RUN apt-get update && \ + apt-get install -y --no-install-recommends \ + ca-certificates curl docker.io jq && \ + rm -rf /var/lib/apt/lists/* + +WORKDIR /tmp +ENTRYPOINT [] diff --git a/agent/verification/e2e/Dockerfile.fixture-gen b/agent/verification/e2e/Dockerfile.fixture-gen new file mode 100644 index 0000000..98adb9a --- /dev/null +++ b/agent/verification/e2e/Dockerfile.fixture-gen @@ -0,0 +1,16 @@ +# Generates one pg_dump fixture per supported major (good-pg12.dump .. +# good-pg18.dump) plus a single broken.dump sentinel. The entrypoint shells +# out to docker against the host daemon (mounted /var/run/docker.sock) and +# spawns a sibling postgres:N container per iteration so each dump is produced +# by the matching server. No in-container postgres binaries needed. +FROM debian:bookworm-slim + +RUN apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates docker.io && \ + rm -rf /var/lib/apt/lists/* + +COPY fixture-gen/entrypoint.sh /usr/local/bin/fixture-gen +RUN chmod +x /usr/local/bin/fixture-gen + +ENTRYPOINT [] +CMD ["/usr/local/bin/fixture-gen"] diff --git a/agent/verification/e2e/Dockerfile.mock-server b/agent/verification/e2e/Dockerfile.mock-server new file mode 100644 index 0000000..1bd16f1 --- /dev/null +++ b/agent/verification/e2e/Dockerfile.mock-server @@ -0,0 +1,10 @@ +# Mock Databasus API server for version checks, verification-agent binary +# downloads and heartbeat capture. Serves static responses and files. +FROM golang:1.26.3-alpine AS build +WORKDIR /app +COPY mock-server/main.go . +RUN CGO_ENABLED=0 go build -o mock-server main.go + +FROM alpine:3.21 +COPY --from=build /app/mock-server /usr/local/bin/mock-server +ENTRYPOINT ["mock-server"] diff --git a/agent/verification/e2e/docker-compose.yml b/agent/verification/e2e/docker-compose.yml new file mode 100644 index 0000000..fe9c1e8 --- /dev/null +++ b/agent/verification/e2e/docker-compose.yml @@ -0,0 +1,65 @@ +services: + e2e-agent-builder: + build: + context: .. + dockerfile: e2e/Dockerfile.agent-builder + volumes: + - ./artifacts:/artifacts + container_name: e2e-verification-agent-builder + + e2e-fixture-gen: + build: + context: . + dockerfile: Dockerfile.fixture-gen + volumes: + - ./artifacts:/artifacts + # The entrypoint spawns one sibling postgres:N container per supported + # major (12..18) and pipes pg_dump out via docker exec, so it talks to + # the host daemon the same way e2e-agent-runner does. + - /var/run/docker.sock:/var/run/docker.sock + container_name: e2e-verification-fixture-gen + + e2e-mock-server: + build: + context: . + dockerfile: Dockerfile.mock-server + volumes: + - ./artifacts:/artifacts:ro + container_name: e2e-verification-mock-server + # Published to host so e2e-agent-runner (network_mode: host) can reach it + # via 127.0.0.1: — same path the spawned Postgres ports use. Host + # port defaults to 4050; override via E2E_MOCK_SERVER_PORT in + # agent/verification/e2e/.env if 4050 is already taken. + ports: + - "${E2E_MOCK_SERVER_PORT:-4050}:4050" + healthcheck: + test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:4050/health"] + interval: 2s + timeout: 5s + retries: 10 + + e2e-agent-runner: + build: + context: . + dockerfile: Dockerfile.agent-runner + volumes: + - ./artifacts:/opt/agent/artifacts:ro + - ./scripts:/opt/agent/scripts:ro + # The agent's Docker SDK client spawns the ephemeral Postgres containers + # on the host daemon; pg_restore runs inside those containers (no host + # client, no assets bundle). + - /var/run/docker.sock:/var/run/docker.sock + # Mirrors production: the agent runs directly on the VM, so 127.0.0.1 IS + # the host. Spawned Postgres containers publish 5432 to a random host port + # bound to the host's 127.0.0.1 (dockerengine.go binds {HostIP:"127.0.0.1", + # HostPort:"0"}); the agent-process verifier reaches them on that loopback. + # A bridge-network runner cannot see those ports — its 127.0.0.1 is its + # own. Linux-only; document accordingly. + network_mode: host + environment: + - E2E_MOCK_SERVER_PORT=${E2E_MOCK_SERVER_PORT:-4050} + depends_on: + e2e-mock-server: + condition: service_healthy + container_name: e2e-verification-agent-runner + command: ["bash", "/opt/agent/scripts/run-all.sh"] diff --git a/agent/verification/e2e/fixture-gen/entrypoint.sh b/agent/verification/e2e/fixture-gen/entrypoint.sh new file mode 100644 index 0000000..3878f31 --- /dev/null +++ b/agent/verification/e2e/fixture-gen/entrypoint.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# Produces /artifacts/good-pgN.dump for each supported Postgres major, one +# /artifacts/good-timescale-pg17.dump (a hypertable dump), and a single +# /artifacts/broken.dump sentinel. For each fixture: spawn a sibling container +# against the host daemon, wait for pg_isready, seed the schema, pg_dump -Fc to +# the artifacts dir, and remove the container. Producer and consumer (pg_restore +# in the agent's spawned target) share the image so the archive header and the +# extension catalog version match. +set -euo pipefail + +VERSIONS=(12 13 14 15 16 17 18) +ARTIFACTS=/artifacts +BROKEN="$ARTIFACTS/broken.dump" + +mkdir -p "$ARTIFACTS" + +CID="" +cleanup() { + if [ -n "${CID:-}" ]; then + docker rm -f "$CID" >/dev/null 2>&1 || true + fi +} +trap cleanup EXIT + +for V in "${VERSIONS[@]}"; do + CID="$(docker run -d -e POSTGRES_PASSWORD=test "postgres:$V")" + + READY=0 + for _ in $(seq 1 30); do + if docker exec "$CID" pg_isready -U postgres -h 127.0.0.1 >/dev/null 2>&1; then + READY=1 + break + fi + sleep 1 + done + + if [ "$READY" -ne 1 ]; then + echo "FAIL: postgres:$V never became ready within 30s" + docker logs "$CID" 2>&1 | tail -30 + exit 1 + fi + + docker exec -i "$CID" psql -U postgres -d postgres -v ON_ERROR_STOP=1 <<'SQL' +CREATE TABLE public.t_a (id int PRIMARY KEY, name text); +CREATE TABLE public.t_b ( + id int PRIMARY KEY, + a_id int NOT NULL REFERENCES public.t_a(id), + value text +); +INSERT INTO public.t_a (id, name) VALUES (1, 'alpha'), (2, 'beta'), (3, 'gamma'); +INSERT INTO public.t_b (id, a_id, value) VALUES + (1, 1, 'x'), (2, 1, 'y'), (3, 2, 'z'), (4, 3, 'w'), (5, 3, 'v'); +SQL + + OUT="$ARTIFACTS/good-pg${V}.dump" + docker exec "$CID" pg_dump -Fc -U postgres postgres > "$OUT" + + docker rm -f "$CID" >/dev/null + CID="" + + echo "fixture: good-pg${V}.dump=$(stat -c%s "$OUT")B" +done + +# TimescaleDB fixture: dumped from the exact image the agent restores into +# (timescale/timescaledb:2.17.0-pg17), so the archive and the extension catalog +# version match. It carries a hypertable spanning many chunks; restoring it +# needs the agent's timescaledb_pre_restore / timescaledb_post_restore wrapping +# and single-threaded -j, or the _timescaledb_catalog restore fails. +TS_IMAGE="timescale/timescaledb:2.17.0-pg17" +TS_OUT="$ARTIFACTS/good-timescale-pg17.dump" + +CID="$(docker run -d -e POSTGRES_PASSWORD=test "$TS_IMAGE")" + +READY=0 +for _ in $(seq 1 30); do + if docker exec "$CID" pg_isready -U postgres -h 127.0.0.1 >/dev/null 2>&1; then + READY=1 + break + fi + sleep 1 +done + +if [ "$READY" -ne 1 ]; then + echo "FAIL: timescaledb never became ready within 30s" + docker logs "$CID" 2>&1 | tail -30 + exit 1 +fi + +docker exec -i "$CID" psql -U postgres -d postgres -v ON_ERROR_STOP=1 <<'SQL' +CREATE EXTENSION IF NOT EXISTS timescaledb; +CREATE TABLE public.sensor_data ( + time TIMESTAMPTZ NOT NULL, + sensor_id int NOT NULL, + temperature double precision NOT NULL +); +SELECT create_hypertable('public.sensor_data', 'time'); +INSERT INTO public.sensor_data (time, sensor_id, temperature) +SELECT ts, (random() * 10)::int, random() * 100 +FROM generate_series('2024-01-01'::timestamptz, '2024-03-01'::timestamptz, interval '1 hour') AS ts; +SQL + +docker exec "$CID" pg_dump -Fc -U postgres postgres > "$TS_OUT" + +docker rm -f "$CID" >/dev/null +CID="" + +echo "fixture: good-timescale-pg17.dump=$(stat -c%s "$TS_OUT")B" + +printf 'not-a-valid-pg-dump-custom-format-archive\n' > "$BROKEN" +chmod 644 "$ARTIFACTS"/good-*.dump "$BROKEN" +echo "fixture: broken.dump=$(stat -c%s "$BROKEN")B" diff --git a/agent/verification/e2e/mock-server/main.go b/agent/verification/e2e/mock-server/main.go new file mode 100644 index 0000000..9e9c5fb --- /dev/null +++ b/agent/verification/e2e/mock-server/main.go @@ -0,0 +1,368 @@ +// Mock Databasus API server for the verification agent e2e: serves version +// checks, the verification-agent binary download, the capacity heartbeat, and +// the job protocol (claim / backup-stream / report) with fault-injection +// controls. Stdlib only — built standalone via `go build main.go`. +package main + +import ( + "encoding/json" + "log" + "net/http" + "os" + "sync" + "time" +) + +type server struct { + mu sync.RWMutex + version string + binaryPath string + heartbeats int + lastBody map[string]any + + // job protocol state + claimJob map[string]any + backupFixturePath string + reports []map[string]any + abortIDs []string + streamFailRemain int + tearStreamOnce bool + reportGone bool +} + +func main() { + s := &server{version: "v1.0.0", binaryPath: "/artifacts/agent-v1"} + + mux := http.NewServeMux() + mux.HandleFunc("GET /api/v1/system/version", s.handleVersion) + mux.HandleFunc("GET /api/v1/system/verification-agent", s.handleDownload) + mux.HandleFunc("POST /api/v1/agent/verification/{agentId}/heartbeat", s.handleHeartbeat) + + mux.HandleFunc("POST /api/v1/agent/verifications/{agentId}/claim", s.handleClaim) + mux.HandleFunc("GET /api/v1/agent/verifications/{agentId}/{id}/backup-stream", s.handleBackupStream) + mux.HandleFunc("POST /api/v1/agent/verifications/{agentId}/{id}/report", s.handleReport) + + mux.HandleFunc("POST /mock/set-version", s.handleSetVersion) + mux.HandleFunc("POST /mock/set-binary-path", s.handleSetBinaryPath) + mux.HandleFunc("GET /mock/heartbeats", s.handleHeartbeatStatus) + mux.HandleFunc("POST /mock/set-claim", s.handleSetClaim) + mux.HandleFunc("POST /mock/set-backup-fixture", s.handleSetBackupFixture) + mux.HandleFunc("POST /mock/set-abort", s.handleSetAbort) + mux.HandleFunc("POST /mock/set-stream-fail", s.handleSetStreamFail) + mux.HandleFunc("POST /mock/set-report-gone", s.handleSetReportGone) + mux.HandleFunc("GET /mock/reports", s.handleGetReports) + mux.HandleFunc("POST /mock/reset", s.handleReset) + mux.HandleFunc("GET /health", s.handleHealth) + + addr := ":4050" + log.Printf("mock server starting on %s", addr) + + if err := http.ListenAndServe(addr, mux); err != nil { + log.Fatalf("server failed: %v", err) + } +} + +func (s *server) handleVersion(w http.ResponseWriter, _ *http.Request) { + s.mu.RLock() + version := s.version + s.mu.RUnlock() + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"version": version}) +} + +func (s *server) handleDownload(w http.ResponseWriter, r *http.Request) { + s.mu.RLock() + path := s.binaryPath + version := s.version + s.mu.RUnlock() + + log.Printf("GET verification-agent (arch=%s) -> %s", r.URL.Query().Get("arch"), path) + w.Header().Set("X-Databasus-Version", version) + http.ServeFile(w, r, path) +} + +func (s *server) handleHeartbeat(w http.ResponseWriter, r *http.Request) { + var body map[string]any + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.heartbeats++ + s.lastBody = body + count := s.heartbeats + abortIDs := append([]string{}, s.abortIDs...) + s.mu.Unlock() + + log.Printf("POST heartbeat (agent=%s) -> count=%d", r.PathValue("agentId"), count) + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": abortIDs, + }) +} + +func (s *server) handleClaim(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + job := s.claimJob + s.claimJob = nil // one-shot: a single verification per test run + s.mu.Unlock() + + if job == nil { + w.WriteHeader(http.StatusNoContent) + return + } + + log.Printf("POST claim (agent=%s) -> assigning %v", r.PathValue("agentId"), job["verificationId"]) + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(job) +} + +func (s *server) handleBackupStream(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + failRemain := s.streamFailRemain + if failRemain > 0 { + s.streamFailRemain-- + } + tear := s.tearStreamOnce + s.tearStreamOnce = false + fixture := s.backupFixturePath + s.mu.Unlock() + + if failRemain > 0 { + log.Printf("GET backup-stream -> injected 503 (remaining=%d)", failRemain-1) + w.WriteHeader(http.StatusServiceUnavailable) + return + } + + if tear { + log.Printf("GET backup-stream -> tearing connection mid-body") + w.Header().Set("Content-Type", "application/octet-stream") + _, _ = w.Write([]byte("PARTIAL")) + + if flusher, ok := w.(http.Flusher); ok { + flusher.Flush() + } + + if hj, ok := w.(http.Hijacker); ok { + conn, _, err := hj.Hijack() + if err == nil { + _ = conn.Close() + } + } + + return + } + + log.Printf("GET backup-stream (id=%s) -> %s", r.PathValue("id"), fixture) + w.Header().Set("Content-Type", "application/octet-stream") + http.ServeFile(w, r, fixture) +} + +func (s *server) handleReport(w http.ResponseWriter, r *http.Request) { + var body map[string]any + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + body["verificationId"] = r.PathValue("id") + + s.mu.Lock() + gone := s.reportGone + if !gone { + s.reports = append(s.reports, body) + } + s.mu.Unlock() + + if gone { + log.Printf("POST report (id=%s) -> 410 gone", r.PathValue("id")) + w.WriteHeader(http.StatusGone) + _, _ = w.Write([]byte(`{"reason":"gone"}`)) + return + } + + log.Printf("POST report (id=%s) -> status=%v exit=%v", r.PathValue("id"), body["status"], body["pgRestoreExitCode"]) + w.WriteHeader(http.StatusNoContent) +} + +func (s *server) handleSetVersion(w http.ResponseWriter, r *http.Request) { + var body struct { + Version string `json:"version"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.version = body.Version + s.mu.Unlock() + + log.Printf("POST /mock/set-version -> %s", body.Version) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetBinaryPath(w http.ResponseWriter, r *http.Request) { + var body struct { + BinaryPath string `json:"binaryPath"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.binaryPath = body.BinaryPath + s.mu.Unlock() + + log.Printf("POST /mock/set-binary-path -> %s", body.BinaryPath) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetClaim(w http.ResponseWriter, r *http.Request) { + var job map[string]any + if err := json.NewDecoder(r.Body).Decode(&job); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.claimJob = job + s.mu.Unlock() + + log.Printf("POST /mock/set-claim -> %v", job["verificationId"]) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetBackupFixture(w http.ResponseWriter, r *http.Request) { + var body struct { + Path string `json:"path"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + if _, err := os.Stat(body.Path); err != nil { + http.Error(w, "fixture not found: "+err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.backupFixturePath = body.Path + s.mu.Unlock() + + log.Printf("POST /mock/set-backup-fixture -> %s", body.Path) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetAbort(w http.ResponseWriter, r *http.Request) { + var body struct { + IDs []string `json:"ids"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.abortIDs = body.IDs + s.mu.Unlock() + + log.Printf("POST /mock/set-abort -> %v", body.IDs) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetStreamFail(w http.ResponseWriter, r *http.Request) { + var body struct { + Count int `json:"count"` + TearStreamOnce bool `json:"tearStreamOnce"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.streamFailRemain = body.Count + s.tearStreamOnce = body.TearStreamOnce + s.mu.Unlock() + + log.Printf("POST /mock/set-stream-fail -> count=%d tear=%v", body.Count, body.TearStreamOnce) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleSetReportGone(w http.ResponseWriter, r *http.Request) { + var body struct { + Gone bool `json:"gone"` + } + + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + s.reportGone = body.Gone + s.mu.Unlock() + + log.Printf("POST /mock/set-report-gone -> %v", body.Gone) + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleGetReports(w http.ResponseWriter, _ *http.Request) { + s.mu.RLock() + defer s.mu.RUnlock() + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "count": len(s.reports), + "reports": s.reports, + }) +} + +func (s *server) handleHeartbeatStatus(w http.ResponseWriter, _ *http.Request) { + s.mu.RLock() + defer s.mu.RUnlock() + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "count": s.heartbeats, + "last": s.lastBody, + }) +} + +// handleReset wipes per-test state (reports, claim, abort/stream/report-gone +// flags, fixture path). Version + binary path are left alone — pin them via +// /mock/set-version + /mock/set-binary-path. Each restore-style e2e test +// should call this at the top to avoid seeing prior-test artifacts. +func (s *server) handleReset(w http.ResponseWriter, _ *http.Request) { + s.mu.Lock() + s.reports = nil + s.claimJob = nil + s.backupFixturePath = "" + s.abortIDs = nil + s.streamFailRemain = 0 + s.tearStreamOnce = false + s.reportGone = false + s.heartbeats = 0 + s.lastBody = nil + s.mu.Unlock() + + log.Printf("POST /mock/reset -> state wiped") + w.WriteHeader(http.StatusOK) +} + +func (s *server) handleHealth(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ok")) +} diff --git a/agent/verification/e2e/scripts/lib.sh b/agent/verification/e2e/scripts/lib.sh new file mode 100644 index 0000000..566f061 --- /dev/null +++ b/agent/verification/e2e/scripts/lib.sh @@ -0,0 +1,137 @@ +#!/bin/bash +# Shared helpers for verification-agent e2e scripts (sourced by each +# scripts/test-*.sh). The runner uses network_mode: host so 127.0.0.1:4050 +# reaches the published mock-server port AND the random host ports the +# spawned Postgres containers bind 5432 to — same setup as production +# (agent on the VM, sibling containers). + +MOCK="http://127.0.0.1:${E2E_MOCK_SERVER_PORT:-4050}" +ARTIFACTS="/opt/agent/artifacts" + +hb_count() { + curl -sf "$MOCK/mock/heartbeats" | sed -n 's/.*"count":\([0-9]*\).*/\1/p' +} + +reports_json() { + curl -sf "$MOCK/mock/reports" +} + +# reset_mock_version pins the mock to v1.0.0 / agent-v1 so the agent's +# background upgrader finds no newer version and the process stays up for the +# whole test. +reset_mock_version() { + curl -sf -X POST "$MOCK/mock/set-version" \ + -H 'Content-Type: application/json' -d '{"version":"v1.0.0"}' + curl -sf -X POST "$MOCK/mock/set-binary-path" \ + -H 'Content-Type: application/json' -d '{"binaryPath":"/artifacts/agent-v1"}' +} + +# reset_mock_state wipes per-test state (reports, claim, fixture, abort/stream +# flags, heartbeat counter). The mock is shared across all e2e scripts in one +# run; without this, a later script sees an earlier script's report and bails. +reset_mock_state() { + curl -sf -X POST "$MOCK/mock/reset" >/dev/null +} + +# leak_check is non-zero if any container/network labeled with the given agent +# ID survives. The anonymous PGDATA volume carries no label and is removed +# alongside the container by RemoveOptions{RemoveVolumes:true}, so a leaked +# container is the canonical signal of a leaked PGDATA. +leak_check() { + local agent_id="$1" + local label="databasus.verification.agent_id=${agent_id}" + local rc=0 + + local containers + containers="$(docker ps -a --filter "label=${label}" --format '{{.ID}} {{.Names}}')" + if [ -n "$containers" ]; then + echo "FAIL: leaked containers labeled ${label}:" + echo "$containers" + rc=1 + fi + + local networks + networks="$(docker network ls --filter "label=${label}" --format '{{.ID}} {{.Name}}')" + if [ -n "$networks" ]; then + echo "FAIL: leaked networks labeled ${label}:" + echo "$networks" + rc=1 + fi + + return $rc +} + +# start_agent [extra_flag...] — runs `databasus-verification-agent run` +# with the standard launch block in the current dir, redirects stdout+stderr to +# ./agent.out, and sets AGENT_PID. Extra flags forwarded to the binary. +start_agent() { + local agent_id="$1" + shift + ./databasus-verification-agent run \ + --databasus-host "$MOCK" \ + --agent-id "$agent_id" \ + --token test-token \ + --max-cpu 4 \ + --max-ram-mb 2048 \ + --max-disk-gb 20 \ + --max-concurrent-jobs 2 \ + --allow-insecure-http \ + --skip-update \ + "$@" > agent.out 2>&1 & + AGENT_PID=$! +} + +stop_agent() { + kill "$AGENT_PID" 2>/dev/null || true + wait "$AGENT_PID" 2>/dev/null || true +} + +dump_diagnostics() { + echo "---- agent.out (last 30) ----" + tail -30 agent.out 2>/dev/null || echo "(no agent.out)" + echo "---- databasus-verification.log (last 60) ----" + tail -60 databasus-verification.log 2>/dev/null || echo "(no log file)" +} + +# wait_for_report [fatal_pattern] — polls +# /mock/reports every 2s until grep -q "$pattern" matches. If fatal_pattern is +# set and matches first, exits 1 with diagnostics. On timeout, exits 1 with +# diagnostics. On success, REPORTS holds the body for follow-up asserts. +wait_for_report() { + local want="$1" + local seconds="$2" + local fatal="${3:-}" + local deadline=$((SECONDS + seconds)) + REPORTS="" + while [ $SECONDS -lt $deadline ]; do + REPORTS="$(reports_json)" + if echo "$REPORTS" | grep -q "$want"; then + return 0 + fi + if [ -n "$fatal" ] && echo "$REPORTS" | grep -q "$fatal"; then + echo "FAIL: saw fatal pattern $fatal before expected $want" + echo "reports: $REPORTS" + dump_diagnostics + exit 1 + fi + sleep 2 + done + echo "FAIL: no report matching $want within ${seconds}s" + echo "reports: $REPORTS" + dump_diagnostics + exit 1 +} + +# assert_report — selects the report whose +# verificationId matches and asserts jq_expr is truthy. On failure prints the +# matching report pretty-printed. +assert_report() { + local vid="$1" + local expr="$2" + if ! reports_json | jq -e --arg vid "$vid" \ + "(.reports[] | select(.verificationId == \$vid)) | $expr" >/dev/null; then + echo "FAIL: report assertion failed for $vid: $expr" + reports_json | jq --arg vid "$vid" '.reports[] | select(.verificationId == $vid)' + exit 1 + fi +} diff --git a/agent/verification/e2e/scripts/run-all.sh b/agent/verification/e2e/scripts/run-all.sh new file mode 100644 index 0000000..d200aef --- /dev/null +++ b/agent/verification/e2e/scripts/run-all.sh @@ -0,0 +1,51 @@ +#!/bin/bash +set -uo pipefail + +SCRIPT_DIR="$(dirname "$0")" + +TESTS=( + "test-heartbeat-and-upgrade.sh" + "test-start-stop-status.sh" + "test-startup-purge.sh" + "test-restore-broken.sh" + "test-restore-disk-budget.sh" + "test-restore-timescale.sh" +) + +PG_VERSIONS=(12 13 14 15 16 17 18) + +echo "========================================" +echo " Verification agent e2e" +echo "========================================" + +PASSED=0 +FAILED=0 + +for test in "${TESTS[@]}"; do + echo "" + echo "---- $test ----" + if bash "$SCRIPT_DIR/$test"; then + PASSED=$((PASSED + 1)) + else + FAILED=$((FAILED + 1)) + fi +done + +for V in "${PG_VERSIONS[@]}"; do + echo "" + echo "---- test-restore-success.sh $V ----" + if bash "$SCRIPT_DIR/test-restore-success.sh" "$V"; then + PASSED=$((PASSED + 1)) + else + FAILED=$((FAILED + 1)) + fi +done + +echo "" +echo "========================================" +echo " Results: $PASSED passed, $FAILED failed" +echo "========================================" + +if [ "$FAILED" -ne 0 ]; then + exit 1 +fi diff --git a/agent/verification/e2e/scripts/test-heartbeat-and-upgrade.sh b/agent/verification/e2e/scripts/test-heartbeat-and-upgrade.sh new file mode 100644 index 0000000..e8e79f7 --- /dev/null +++ b/agent/verification/e2e/scripts/test-heartbeat-and-upgrade.sh @@ -0,0 +1,122 @@ +#!/bin/bash +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +WORK="/tmp/agent-work" +AGENT_ID="11111111-1111-1111-1111-111111111111" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +INITIAL="$(./databasus-verification-agent version)" +if [ "$INITIAL" != "v1.0.0" ]; then + echo "FAIL: expected initial version v1.0.0, got $INITIAL" + exit 1 +fi +echo "Initial version: $INITIAL" + +./databasus-verification-agent run \ + --databasus-host "$MOCK" \ + --agent-id "$AGENT_ID" \ + --token test-token \ + --max-cpu 4 \ + --max-ram-mb 2048 \ + --max-disk-gb 20 \ + --max-concurrent-jobs 2 \ + --allow-insecure-http > agent.out 2>&1 & +AGENT_PID=$! + +# 1) Capacity heartbeat is received (Run sends one immediately). +COUNT=0 +for _ in $(seq 1 20); do + COUNT="$(hb_count || echo 0)" + [ "${COUNT:-0}" -ge 1 ] && break + sleep 1 +done +if [ "${COUNT:-0}" -lt 1 ]; then + echo "FAIL: no heartbeat received" + cat agent.out + exit 1 +fi +echo "Heartbeat received (count=$COUNT)" + +# 2) Heartbeat wire shape: flat maxRamGb (2048 MB -> 2 GB), empty job list. +BODY="$(curl -sf "$MOCK/mock/heartbeats")" +if ! echo "$BODY" | grep -q '"maxRamGb":2'; then + echo "FAIL: heartbeat missing maxRamGb:2" + echo "$BODY" + exit 1 +fi +if ! echo "$BODY" | grep -q '"currentVerificationIds":\[\]'; then + echo "FAIL: heartbeat currentVerificationIds not []" + echo "$BODY" + exit 1 +fi +echo "Heartbeat wire shape OK" + +# 3) Bump version -> background self-update downloads v2 and re-execs. +curl -sf -X POST "$MOCK/mock/set-binary-path" \ + -H 'Content-Type: application/json' -d '{"binaryPath":"/artifacts/agent-v2"}' +curl -sf -X POST "$MOCK/mock/set-version" \ + -H 'Content-Type: application/json' -d '{"version":"v2.0.0"}' +echo "Mock bumped to v2.0.0, waiting for background upgrade..." + +DEADLINE=$((SECONDS + 90)) +while [ $SECONDS -lt $DEADLINE ]; do + [ "$(./databasus-verification-agent version)" = "v2.0.0" ] && break + sleep 3 +done + +FINAL="$(./databasus-verification-agent version)" +if [ "$FINAL" != "v2.0.0" ]; then + echo "FAIL: expected v2.0.0 after background upgrade, got $FINAL" + cat agent.out + exit 1 +fi +echo "Binary upgraded to $FINAL" + +if ! grep -q "Agent binary updated" agent.out; then + echo "FAIL: upgrade log line missing" + cat agent.out + exit 1 +fi +if ! grep -q "Re-executing after upgrade" agent.out; then + echo "FAIL: re-exec log line missing" + cat agent.out + exit 1 +fi + +# 4) Same PID is alive after re-exec (syscall.Exec preserves the PID). +if ! kill -0 "$AGENT_PID" 2>/dev/null; then + echo "FAIL: agent process died during self-update" + cat agent.out + exit 1 +fi + +# 5) Heartbeats resume after re-exec. +BEFORE="$(hb_count || echo 0)" +RESUMED=0 +for _ in $(seq 1 45); do + AFTER="$(hb_count || echo 0)" + if [ "${AFTER:-0}" -gt "${BEFORE:-0}" ]; then + RESUMED=1 + break + fi + sleep 1 +done +if [ "$RESUMED" -ne 1 ]; then + echo "FAIL: heartbeats did not resume after re-exec ($BEFORE -> ${AFTER:-0})" + cat agent.out + exit 1 +fi +echo "Heartbeats resumed after re-exec ($BEFORE -> $AFTER)" + +kill "$AGENT_PID" 2>/dev/null || true +echo "Verification agent e2e passed" diff --git a/agent/verification/e2e/scripts/test-restore-broken.sh b/agent/verification/e2e/scripts/test-restore-broken.sh new file mode 100644 index 0000000..15996d7 --- /dev/null +++ b/agent/verification/e2e/scripts/test-restore-broken.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# Restore e2e (broken dump): mock serves a file that is NOT a valid -Fc +# archive; pg_restore -Fc inside the spawned Postgres container must reject +# it and exit non-zero, and the agent must POST FAILED with a real +# pgRestoreExitCode. Independent of the verifier-conn fix — the failure +# happens at restore, never reaches the verifier. +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +WORK="/tmp/agent-work-restore-broken" +AGENT_ID="33333333-3333-3333-3333-333333333333" +VERIFICATION_ID="33333333-aaaa-aaaa-aaaa-333333333333" +BACKUP_ID="33333333-bbbb-bbbb-bbbb-333333333333" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_state +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +start_agent "$AGENT_ID" + +curl -sf -X POST "$MOCK/mock/set-backup-fixture" \ + -H 'Content-Type: application/json' \ + -d '{"path":"/artifacts/broken.dump"}' + +curl -sf -X POST "$MOCK/mock/set-claim" \ + -H 'Content-Type: application/json' \ + -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":2048,\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"16\"}}}" + +# Spawn + image pull (first run) + restore + report. Generous budget on cold cache. +wait_for_report '"status":"FAILED"' 240 + +assert_report "$VERIFICATION_ID" '.pgRestoreExitCode >= 1' + +echo "Broken-dump report OK: FAILED with non-zero pgRestoreExitCode" + +stop_agent + +if ! leak_check "$AGENT_ID"; then + echo "---- agent.out ----" + cat agent.out + exit 1 +fi + +echo "Verification agent restore-broken e2e passed" diff --git a/agent/verification/e2e/scripts/test-restore-disk-budget.sh b/agent/verification/e2e/scripts/test-restore-disk-budget.sh new file mode 100644 index 0000000..0297f67 --- /dev/null +++ b/agent/verification/e2e/scripts/test-restore-disk-budget.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# Disk-budget enforcement (real Docker): tiny maxContainerDiskMb (1 MB) against +# the standard postgres:16 init footprint (~50 MB) makes the disk watcher trip +# on its first poll. The agent must POST FAILED with failureKind: +# DISK_LIMIT_EXCEEDED and the container/anonymous PGDATA volume must be torn +# down. The watcher polls every 3 s so the verdict lands within ~10 s of spawn. +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +WORK="/tmp/agent-work-disk-budget" +AGENT_ID="55555555-5555-5555-5555-555555555555" +VERIFICATION_ID="55555555-aaaa-aaaa-aaaa-555555555555" +BACKUP_ID="55555555-bbbb-bbbb-bbbb-555555555555" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_state +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +start_agent "$AGENT_ID" + +curl -sf -X POST "$MOCK/mock/set-backup-fixture" \ + -H 'Content-Type: application/json' \ + -d '{"path":"/artifacts/good-pg16.dump"}' + +curl -sf -X POST "$MOCK/mock/set-claim" \ + -H 'Content-Type: application/json' \ + -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":1,\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"16\"}}}" + +wait_for_report '"failureKind":"DISK_LIMIT_EXCEEDED"' 180 '"status":"COMPLETED"' + +assert_report "$VERIFICATION_ID" '.status == "FAILED"' + +echo "Disk-budget verdict OK: FAILED with failureKind:DISK_LIMIT_EXCEEDED" + +stop_agent + +if ! leak_check "$AGENT_ID"; then + exit 1 +fi + +echo "Verification agent restore-disk-budget e2e passed" diff --git a/agent/verification/e2e/scripts/test-restore-success.sh b/agent/verification/e2e/scripts/test-restore-success.sh new file mode 100644 index 0000000..ef3823d --- /dev/null +++ b/agent/verification/e2e/scripts/test-restore-success.sh @@ -0,0 +1,58 @@ +#!/bin/bash +# Restore e2e (correct dump) for one Postgres major: mock serves the matching +# -Fc archive of a tiny seeded schema; the agent must run pg_restore inside +# the spawned postgres:N container, then the verifier (out-of-container pgx +# via the container's published host port) must collect tier stats, and the +# agent must POST COMPLETED with non-null dbSizeBytesAfterRestore / +# tableCount / pgRestoreExitCode==0 / non-empty tableStats. The PG major is +# the first positional arg (default 16); run-all.sh loops it over 12..18. +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +PG_VERSION="${1:-16}" + +WORK="/tmp/agent-work-restore-success-pg${PG_VERSION}" +AGENT_ID="44444444-4444-4444-4444-444444444444" +VERIFICATION_ID="44444444-aaaa-aaaa-aaaa-4444444444${PG_VERSION}" +BACKUP_ID="44444444-bbbb-bbbb-bbbb-4444444444${PG_VERSION}" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_state +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +start_agent "$AGENT_ID" + +curl -sf -X POST "$MOCK/mock/set-backup-fixture" \ + -H 'Content-Type: application/json' \ + -d "{\"path\":\"/artifacts/good-pg${PG_VERSION}.dump\"}" + +curl -sf -X POST "$MOCK/mock/set-claim" \ + -H 'Content-Type: application/json' \ + -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":2048,\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"${PG_VERSION}\"}}}" + +wait_for_report '"status":"COMPLETED"' 240 '"status":"FAILED"' + +assert_report "$VERIFICATION_ID" '.pgRestoreExitCode == 0' +assert_report "$VERIFICATION_ID" '.dbSizeBytesAfterRestore > 0' +assert_report "$VERIFICATION_ID" '.tableCount == 2' +assert_report "$VERIFICATION_ID" '.schemaCount == 1' +assert_report "$VERIFICATION_ID" '(.tableStats | map(.name) | sort) == ["t_a","t_b"]' + +echo "Success report OK (pg${PG_VERSION}): COMPLETED with tableCount=2 schemaCount=1 t_a+t_b present" + +stop_agent + +if ! leak_check "$AGENT_ID"; then + echo "---- agent.out ----" + cat agent.out + exit 1 +fi + +echo "Verification agent restore-success e2e (pg${PG_VERSION}) passed" diff --git a/agent/verification/e2e/scripts/test-restore-timescale.sh b/agent/verification/e2e/scripts/test-restore-timescale.sh new file mode 100644 index 0000000..e7ab282 --- /dev/null +++ b/agent/verification/e2e/scripts/test-restore-timescale.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# TimescaleDB restore e2e: the mock serves a hypertable -Fc dump and a claim +# whose timescaledbVersion is set, so the agent must spawn a version-matched +# timescale/timescaledb:-pg container, run timescaledb_pre_restore / +# timescaledb_post_restore around a single-threaded pg_restore, then the +# verifier must collect stats and the agent must POST COMPLETED with exit 0. +# Without the hooks (or with parallel -j) the _timescaledb_catalog restore fails +# and the agent would report FAILED instead. +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +PG_VERSION="17" +TS_VERSION="2.17.0" + +WORK="/tmp/agent-work-restore-timescale" +AGENT_ID="55555555-5555-5555-5555-555555555555" +VERIFICATION_ID="55555555-aaaa-aaaa-aaaa-555555555555" +BACKUP_ID="55555555-bbbb-bbbb-bbbb-555555555555" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_state +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +start_agent "$AGENT_ID" + +curl -sf -X POST "$MOCK/mock/set-backup-fixture" \ + -H 'Content-Type: application/json' \ + -d '{"path":"/artifacts/good-timescale-pg17.dump"}' + +curl -sf -X POST "$MOCK/mock/set-claim" \ + -H 'Content-Type: application/json' \ + -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":4096,\"timescaledbVersion\":\"${TS_VERSION}\",\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"${PG_VERSION}\"}}}" + +wait_for_report '"status":"COMPLETED"' 300 '"status":"FAILED"' + +assert_report "$VERIFICATION_ID" '.pgRestoreExitCode == 0' +assert_report "$VERIFICATION_ID" '.dbSizeBytesAfterRestore > 0' +# The hypertable's parent table is restored into public... +assert_report "$VERIFICATION_ID" '(.tableStats | map(.name) | any(. == "sensor_data"))' +# ...and its data lands in _timescaledb_internal chunks, proving the catalog + +# chunk restore (the part that fails without the pre/post hooks) succeeded. +assert_report "$VERIFICATION_ID" '(.tableStats | map(select(.name | startswith("_hyper_")) | .rowCount) | add) > 0' + +echo "TimescaleDB restore report OK: COMPLETED with sensor_data hypertable + chunk rows restored" + +stop_agent + +if ! leak_check "$AGENT_ID"; then + echo "---- agent.out ----" + cat agent.out + exit 1 +fi + +echo "Verification agent restore-timescale e2e passed" diff --git a/agent/verification/e2e/scripts/test-start-stop-status.sh b/agent/verification/e2e/scripts/test-start-stop-status.sh new file mode 100644 index 0000000..6112ae1 --- /dev/null +++ b/agent/verification/e2e/scripts/test-start-stop-status.sh @@ -0,0 +1,131 @@ +#!/bin/bash +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +WORK="/tmp/agent-work-startstop" +AGENT_ID="22222222-2222-2222-2222-222222222222" + +dump_daemon_log() { + echo "---- databasus-verification.daemon.log ----" + cat databasus-verification.daemon.log 2>/dev/null || echo "(no daemon log)" +} + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_version + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +LAUNCH_FLAGS=( + --databasus-host "$MOCK" + --agent-id "$AGENT_ID" + --token test-token + --max-cpu 4 + --max-ram-mb 2048 + --max-disk-gb 20 + --max-concurrent-jobs 2 + --allow-insecure-http + --skip-update +) + +# 1) status before start -> not running. +STATUS_OUT="$(./databasus-verification-agent status)" +echo "status (before): $STATUS_OUT" +if ! echo "$STATUS_OUT" | grep -q "Agent is not running"; then + echo "FAIL: expected 'Agent is not running' before start" + exit 1 +fi + +# 2) start detaches, returns promptly, prints the daemon PID. +START_OUT="$(./databasus-verification-agent start "${LAUNCH_FLAGS[@]}")" +echo "start: $START_OUT" +if ! echo "$START_OUT" | grep -q "Agent started in background (PID "; then + echo "FAIL: start did not report a background PID" + dump_daemon_log + exit 1 +fi + +# 3) start created the single-instance lock file. +if [ ! -f databasus-verification.lock ]; then + echo "FAIL: databasus-verification.lock not created" + dump_daemon_log + exit 1 +fi + +# 4) the detached daemon heartbeats. +RESUMED=0 +for _ in $(seq 1 20); do + COUNT="$(hb_count || echo 0)" + if [ "${COUNT:-0}" -ge 1 ]; then + RESUMED=1 + break + fi + sleep 1 +done +if [ "$RESUMED" -ne 1 ]; then + echo "FAIL: detached daemon did not heartbeat after start" + dump_daemon_log + exit 1 +fi +echo "Daemon heartbeating" + +# 5) status -> running with a live PID. +STATUS_OUT="$(./databasus-verification-agent status)" +echo "status (running): $STATUS_OUT" +if ! echo "$STATUS_OUT" | grep -q "Agent is running (PID "; then + echo "FAIL: status did not report running" + dump_daemon_log + exit 1 +fi +PID="$(echo "$STATUS_OUT" | sed -n 's/.*PID \([0-9]*\).*/\1/p')" +if ! kill -0 "$PID" 2>/dev/null; then + echo "FAIL: reported PID $PID is not alive" + exit 1 +fi + +# 6) a second agent in the same directory is rejected by the lock. `run` is the +# foreground command, so the lock error surfaces directly on its stderr. +set +e +SECOND_OUT="$(./databasus-verification-agent run "${LAUNCH_FLAGS[@]}" 2>&1)" +SECOND_RC=$? +set -e +echo "second run: $SECOND_OUT" +if [ "$SECOND_RC" -eq 0 ]; then + echo "FAIL: a second agent started while one was already running" + exit 1 +fi +if ! echo "$SECOND_OUT" | grep -q "another instance is already running"; then + echo "FAIL: second agent did not fail with the single-instance error" + exit 1 +fi +echo "Second instance rejected" + +# 7) stop terminates the daemon within the stop timeout. +./databasus-verification-agent stop +GONE=0 +for _ in $(seq 1 35); do + if ! kill -0 "$PID" 2>/dev/null; then + GONE=1 + break + fi + sleep 1 +done +if [ "$GONE" -ne 1 ]; then + echo "FAIL: agent (PID $PID) still alive after stop" + exit 1 +fi +echo "Agent stopped" + +# 8) status after stop -> not running again. +STATUS_OUT="$(./databasus-verification-agent status)" +echo "status (after): $STATUS_OUT" +if ! echo "$STATUS_OUT" | grep -q "Agent is not running"; then + echo "FAIL: expected 'Agent is not running' after stop" + exit 1 +fi + +echo "Verification agent start/stop/status e2e passed" diff --git a/agent/verification/e2e/scripts/test-startup-purge.sh b/agent/verification/e2e/scripts/test-startup-purge.sh new file mode 100644 index 0000000..055c083 --- /dev/null +++ b/agent/verification/e2e/scripts/test-startup-purge.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# Startup purge: at boot the agent removes every container labeled with its +# agent_id — the single-agent invariant means no other process owns containers +# with the same agent_id, so blanket removal is safe. Pre-seed a stray, start +# the agent, assert it's gone. +set -euo pipefail + +source "$(dirname "$0")/lib.sh" + +WORK="/tmp/agent-work-purge" +AGENT_ID="66666666-6666-6666-6666-666666666666" +STRAY_NAME="databasus-verif-purge-stray" +STRAY_LABEL="databasus.verification.agent_id=${AGENT_ID}" + +rm -rf "$WORK" +mkdir -p "$WORK" +cd "$WORK" + +reset_mock_state +reset_mock_version + +docker rm -f "$STRAY_NAME" >/dev/null 2>&1 || true +docker run -d --name "$STRAY_NAME" --label "$STRAY_LABEL" alpine sleep 600 >/dev/null + +if ! docker ps -a --filter "label=$STRAY_LABEL" --format '{{.Names}}' | grep -q "$STRAY_NAME"; then + echo "FAIL: pre-seed stray container not present" + exit 1 +fi +echo "Pre-seeded stray container $STRAY_NAME with label $STRAY_LABEL" + +cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent +chmod +x ./databasus-verification-agent + +start_agent "$AGENT_ID" + +GONE=0 +for _ in $(seq 1 30); do + if [ -z "$(docker ps -a --filter "label=$STRAY_LABEL" --format '{{.Names}}')" ]; then + GONE=1 + break + fi + sleep 1 +done + +if [ "$GONE" -ne 1 ]; then + echo "FAIL: stray container still present after agent start; PurgeContainers did not run" + docker ps -a --filter "label=$STRAY_LABEL" + echo "---- agent.out (last 30) ----" + tail -30 agent.out + stop_agent + exit 1 +fi +echo "Stray container purged by agent startup" + +stop_agent + +if ! leak_check "$AGENT_ID"; then + exit 1 +fi + +echo "Verification agent startup-purge e2e passed" diff --git a/agent/verification/go.mod b/agent/verification/go.mod new file mode 100644 index 0000000..c137abf --- /dev/null +++ b/agent/verification/go.mod @@ -0,0 +1,41 @@ +module databasus-verification-agent + +go 1.26.3 + +require ( + github.com/go-resty/resty/v2 v2.17.2 + github.com/google/uuid v1.6.0 + github.com/jackc/pgx/v5 v5.9.2 + github.com/moby/moby/api v1.54.2 + github.com/moby/moby/client v0.4.1 + github.com/stretchr/testify v1.11.1 +) + +require ( + github.com/Microsoft/go-winio v0.6.2 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/containerd/errdefs v1.0.0 // indirect + github.com/containerd/errdefs/pkg v0.3.0 // indirect + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/distribution/reference v0.6.0 // indirect + github.com/docker/go-connections v0.7.0 // indirect + github.com/docker/go-units v0.5.0 // indirect + github.com/felixge/httpsnoop v1.0.4 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/moby/docker-image-spec v1.3.1 // indirect + github.com/opencontainers/go-digest v1.0.0 // indirect + github.com/opencontainers/image-spec v1.1.1 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 // indirect + go.opentelemetry.io/otel v1.43.0 // indirect + go.opentelemetry.io/otel/metric v1.43.0 // indirect + go.opentelemetry.io/otel/trace v1.43.0 // indirect + golang.org/x/net v0.52.0 // indirect + golang.org/x/sys v0.42.0 // indirect + golang.org/x/text v0.35.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) diff --git a/agent/verification/go.sum b/agent/verification/go.sum new file mode 100644 index 0000000..2fafd06 --- /dev/null +++ b/agent/verification/go.sum @@ -0,0 +1,95 @@ +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= +github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= +github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= +github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= +github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= +github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c= +github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q= +github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= +github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-resty/resty/v2 v2.17.2 h1:FQW5oHYcIlkCNrMD2lloGScxcHJ0gkjshV3qcQAyHQk= +github.com/go-resty/resty/v2 v2.17.2/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw= +github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= +github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo= +github.com/moby/moby/api v1.54.2 h1:wiat9QAhnDQjA7wk1kh/TqHz2I1uUA7M7t9SAl/JNXg= +github.com/moby/moby/api v1.54.2/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs= +github.com/moby/moby/client v0.4.1 h1:DMQgisVoMkmMs7fp3ROSdiBnoAu8+vo3GggFl06M/wY= +github.com/moby/moby/client v0.4.1/go.mod h1:z52C9O2POPOsnxZAy//WtKcQ32P+jT/NGeXu/7nfjGQ= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= +github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0 h1:CqXxU8VOmDefoh0+ztfGaymYbhdB/tT3zs79QaZTNGY= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.68.0/go.mod h1:BuhAPThV8PBHBvg8ZzZ/Ok3idOdhWIodywz2xEcRbJo= +go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I= +go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0= +go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM= +go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY= +go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg= +go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg= +go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw= +go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A= +go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A= +go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0= +golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0= +golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo= +golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8= +golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA= +golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE= +golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q= +gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA= +pgregory.net/rapid v1.2.0 h1:keKAYRcjm+e1F0oAuU5F5+YPAWcyxNNRK2wud503Gnk= +pgregory.net/rapid v1.2.0/go.mod h1:PY5XlDGj0+V1FCq0o192FdRhpKHGTRIWBgqjDBTrq04= diff --git a/agent/verification/images/postgres/Dockerfile b/agent/verification/images/postgres/Dockerfile new file mode 100644 index 0000000..bc9eb09 --- /dev/null +++ b/agent/verification/images/postgres/Dockerfile @@ -0,0 +1,71 @@ +# syntax=docker/dockerfile:1 + +# Databasus verification image: the official Postgres image plus a curated set of +# common extensions, so a dump's CREATE EXTENSION statements succeed when the +# agent restores a backup for verification (issue #619 PostGIS, plus auxiliary +# extensions like pg_stat_kcache / set_user). This mirrors how the agent already +# uses timescale/timescaledb for TimescaleDB. Built per PG major and published as +# databasus/verification-postgres:. +# +# BASE_IMAGE is digest-pinned by CI (postgres:@sha256:...). The tag-only +# default exists solely for local `docker build --build-arg PG_MAJOR=16` runs. +ARG PG_MAJOR=16 +ARG BASE_IMAGE=postgres:${PG_MAJOR} + +FROM ${BASE_IMAGE} + +# ARGs declared before FROM are not visible in the build stage — redeclare it. +ARG PG_MAJOR + +# Extensions whose CREATE EXTENSION works once the files are present, plus the +# Tier-2 monitoring extensions that additionally need shared_preload_libraries +# (set below). The official image already configures the apt.postgresql.org repo +# these packages come from, so every entry must be installable from PGDG apt for +# all majors 12-18 (verified). A missing package fails the build loudly rather +# than silently shipping a half-populated image — important because the agent +# tolerates missing extensions at restore time, so a thinned image would mark +# such backups "verified" without actually exercising the extension. +# +# Intentionally NOT bundled: topn (Citus), pg_hashids, anonymizer +# (postgresql_anonymizer) and pg_stat_monitor (Percona). PGDG apt does not +# package these — they need source builds or Percona's own repo. Add them via +# that route (not this list) if verification coverage for them is ever required. +ARG EXTENSION_PACKAGES="\ +postgresql-${PG_MAJOR}-postgis-3 \ +postgresql-${PG_MAJOR}-pgrouting \ +postgresql-${PG_MAJOR}-h3 \ +postgresql-${PG_MAJOR}-pgvector \ +postgresql-${PG_MAJOR}-rum \ +postgresql-${PG_MAJOR}-partman \ +postgresql-${PG_MAJOR}-repack \ +postgresql-${PG_MAJOR}-hypopg \ +postgresql-${PG_MAJOR}-hll \ +postgresql-${PG_MAJOR}-set-user \ +postgresql-${PG_MAJOR}-pgaudit \ +postgresql-${PG_MAJOR}-ip4r \ +postgresql-${PG_MAJOR}-semver \ +postgresql-${PG_MAJOR}-http \ +postgresql-${PG_MAJOR}-mysql-fdw \ +postgresql-${PG_MAJOR}-tds-fdw \ +postgresql-${PG_MAJOR}-pg-stat-kcache \ +postgresql-${PG_MAJOR}-pg-qualstats \ +postgresql-${PG_MAJOR}-pg-wait-sampling \ +postgresql-plpython3-${PG_MAJOR} \ +postgresql-plperl-${PG_MAJOR} \ +postgresql-pltcl-${PG_MAJOR}" + +# Tier-2 extensions refuse to CREATE unless their library is preloaded. This list +# must match the packages installed above: a name whose library is absent makes +# Postgres refuse to start. Appending to the initdb sample means every fresh +# PGDATA (the verification container is ephemeral) inherits it; the agent does not +# pass -c shared_preload_libraries, so this value stands at runtime. +ARG PRELOAD_LIBRARIES="pg_stat_statements,pg_stat_kcache,pg_qualstats,pg_wait_sampling,pgaudit,set_user" + +# Root is the base image's user; the official entrypoint still gosu-drops to the +# postgres user before exec, so the agent's hardened runtime is unchanged. +RUN set -eux; \ + apt-get update; \ + apt-get install -y --no-install-recommends ${EXTENSION_PACKAGES}; \ + rm -rf /var/lib/apt/lists/*; \ + printf "shared_preload_libraries = '%s'\n" "${PRELOAD_LIBRARIES}" \ + >> "/usr/share/postgresql/${PG_MAJOR}/postgresql.conf.sample" diff --git a/agent/verification/internal/config/config.go b/agent/verification/internal/config/config.go new file mode 100644 index 0000000..1be13ad --- /dev/null +++ b/agent/verification/internal/config/config.go @@ -0,0 +1,244 @@ +package config + +import ( + "bufio" + "encoding/json" + "flag" + "fmt" + "net/url" + "os" + "strings" + + "databasus-verification-agent/internal/logger" +) + +var log = logger.GetLogger() + +const configFileName = "databasus-verification.json" + +const minRAMMbPerJob = 512 + +// DefaultVerificationPgImageRepo bundles common extensions so a dump's CREATE +// EXTENSION statements succeed during verification; override via +// verificationPgImageRepo (e.g. a private registry, or "postgres" for stock). +const DefaultVerificationPgImageRepo = "databasus/verification-postgres" + +type Config struct { + DatabasusHost string `json:"databasusHost"` + AgentID string `json:"agentId"` + Token string `json:"token"` + MaxCPU int `json:"maxCpu"` + MaxRAMMb int `json:"maxRamMb"` + MaxDiskGb int `json:"maxDiskGb"` + MaxConcurrentJobs int `json:"maxConcurrentJobs"` + AllowInsecureHTTP bool `json:"allowInsecureHttp"` + VerificationPgImageRepo string `json:"verificationPgImageRepo"` + + flagSources map[string]string +} + +func (c *Config) GetVerificationPgImageRepo() string { + if c.VerificationPgImageRepo == "" { + return DefaultVerificationPgImageRepo + } + + return c.VerificationPgImageRepo +} + +type Capacity struct { + MaxCPU int + MaxRAMMb int + MaxDiskGb int + MaxConcurrentJobs int + + CPUPerJob int + RAMMbPerJob int +} + +// LoadFromJSONAndArgs reads databasus-verification.json into the struct +// and overrides JSON values with any explicitly provided CLI flags. +func (c *Config) LoadFromJSONAndArgs(fs *flag.FlagSet, args []string) { + c.loadFromJSON() + + bindings := c.flagBindings() + c.flagSources = sourcesFromConfig(bindings, c) + + appliers := make([]func(*Config, map[string]string), 0, len(bindings)) + for _, binding := range bindings { + appliers = append(appliers, binding.register(fs)) + } + + if err := fs.Parse(args); err != nil { + os.Exit(1) + } + + for _, apply := range appliers { + apply(c, c.flagSources) + } + + log.Info("========= Loading config ============") + c.logConfigSources(bindings) + log.Info("========= Config has been loaded ====") +} + +// SaveToJSON writes the current struct to databasus-verification.json. +func (c *Config) SaveToJSON() error { + data, err := json.MarshalIndent(c, "", " ") + if err != nil { + return err + } + + return os.WriteFile(configFileName, data, 0o644) +} + +// LoadFromJSON reads databasus-verification.json without applying CLI flags. +// The detached _run daemon uses this: the parent `start` already validated and +// persisted the config, so the child trusts the file as-is. +func (c *Config) LoadFromJSON() { + c.loadFromJSON() +} + +func (c *Config) Validate() (Capacity, error) { + if c.DatabasusHost == "" { + return Capacity{}, fmt.Errorf("databasus-host is required") + } + + if c.AgentID == "" { + return Capacity{}, fmt.Errorf("agent-id is required") + } + + if c.Token == "" { + return Capacity{}, fmt.Errorf("token is required") + } + + return c.DeriveCapacity() +} + +func (c *Config) DeriveCapacity() (Capacity, error) { + if c.MaxConcurrentJobs < 1 { + return Capacity{}, fmt.Errorf( + "max-concurrent-jobs must be >= 1 (got %d)", c.MaxConcurrentJobs) + } + + if c.MaxCPU < 1 { + return Capacity{}, fmt.Errorf("max-cpu must be >= 1 (got %d)", c.MaxCPU) + } + + if c.MaxDiskGb < 1 { + return Capacity{}, fmt.Errorf("max-disk-gb must be >= 1 (got %d)", c.MaxDiskGb) + } + + if c.MaxRAMMb < minRAMMbPerJob { + return Capacity{}, fmt.Errorf( + "max-ram-mb must be >= %d (got %d)", minRAMMbPerJob, c.MaxRAMMb) + } + + cpuPerJob := c.MaxCPU / c.MaxConcurrentJobs + ramMbPerJob := c.MaxRAMMb / c.MaxConcurrentJobs + + if cpuPerJob < 1 { + return Capacity{}, fmt.Errorf( + "max-cpu (%d) split across max-concurrent-jobs (%d) yields < 1 CPU per job; "+ + "lower max-concurrent-jobs or raise max-cpu", + c.MaxCPU, c.MaxConcurrentJobs) + } + + if ramMbPerJob < minRAMMbPerJob { + return Capacity{}, fmt.Errorf( + "max-ram-mb (%d) split across max-concurrent-jobs (%d) yields %d MB per job, "+ + "below the %d MB floor; lower max-concurrent-jobs or raise max-ram-mb", + c.MaxRAMMb, c.MaxConcurrentJobs, ramMbPerJob, minRAMMbPerJob) + } + + return Capacity{ + MaxCPU: c.MaxCPU, + MaxRAMMb: c.MaxRAMMb, + MaxDiskGb: c.MaxDiskGb, + MaxConcurrentJobs: c.MaxConcurrentJobs, + CPUPerJob: cpuPerJob, + RAMMbPerJob: ramMbPerJob, + }, nil +} + +// ValidateTransport enforces the http/https gate before any goroutine starts. +// The per-agent token and the decrypted backup stream both cross this link, so +// plain HTTP is allowed only with explicit operator consent. +func (c *Config) ValidateTransport(isStdinTTY bool, in *bufio.Reader) error { + parsed, err := url.Parse(c.DatabasusHost) + if err != nil { + return fmt.Errorf("databasus-host is not a valid URL: %w", err) + } + + switch parsed.Scheme { + case "https": + return nil + + case "http": + return c.consentToInsecureHTTP(isStdinTTY, in) + + default: + return fmt.Errorf( + "databasus-host must start with https:// or http:// (got scheme %q)", parsed.Scheme) + } +} + +func (c *Config) loadFromJSON() { + data, err := os.ReadFile(configFileName) + if err != nil { + if os.IsNotExist(err) { + log.Info("No databasus-verification.json found, will create on save") + return + } + + log.Warn("Failed to read databasus-verification.json", "error", err) + + return + } + + if err := json.Unmarshal(data, c); err != nil { + log.Warn("Failed to parse databasus-verification.json", "error", err) + + return + } + + log.Info("Configuration loaded from " + configFileName) +} + +func (c *Config) consentToInsecureHTTP(isStdinTTY bool, in *bufio.Reader) error { + if c.AllowInsecureHTTP { + log.Warn("databasus-host is plain HTTP; transport is unencrypted") + + return nil + } + + if !isStdinTTY { + return fmt.Errorf( + "refusing to use plain HTTP over a non-interactive connection: " + + "switch --databasus-host to https:// or pass --allow-insecure-http " + + "to accept unencrypted transport") + } + + fmt.Fprint(os.Stderr, + "WARNING: connecting to the primary over plain HTTP, not HTTPS. "+ + "The agent token and decrypted backup data are sent unencrypted. "+ + "Continue? [y/N] ") + + answer, _ := in.ReadString('\n') + answer = strings.ToLower(strings.TrimSpace(answer)) + + if answer != "y" && answer != "yes" { + return fmt.Errorf("aborted: plain HTTP transport declined by operator") + } + + return nil +} + +func maskSensitive(value string) string { + if value == "" { + return "(not set)" + } + + visibleLen := max(len(value)/4, 1) + + return value[:visibleLen] + "***" +} diff --git a/agent/verification/internal/config/config_test.go b/agent/verification/internal/config/config_test.go new file mode 100644 index 0000000..074dac8 --- /dev/null +++ b/agent/verification/internal/config/config_test.go @@ -0,0 +1,290 @@ +package config + +import ( + "bufio" + "flag" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func loadConfig(t *testing.T, args ...string) *Config { + t.Helper() + + cfg := &Config{} + cfg.LoadFromJSONAndArgs(flag.NewFlagSet("test", flag.ContinueOnError), args) + + return cfg +} + +var allFlagNames = []string{ + "databasus-host", "agent-id", "token", "max-cpu", "max-ram-mb", + "max-disk-gb", "max-concurrent-jobs", "allow-insecure-http", "verification-pg-image-repo", +} + +func emptyReader() *bufio.Reader { + return bufio.NewReader(strings.NewReader("")) +} + +func Test_Validate_WhenConfigComplete_ReturnsCapacity(t *testing.T) { + cfg := &Config{ + DatabasusHost: "https://primary.example:4005", + AgentID: "agent-1", + Token: "secret", + MaxCPU: 8, + MaxRAMMb: 4096, + MaxDiskGb: 100, + MaxConcurrentJobs: 4, + } + + capacity, err := cfg.Validate() + + require.NoError(t, err) + assert.Equal(t, 2, capacity.CPUPerJob) + assert.Equal(t, 1024, capacity.RAMMbPerJob) +} + +func Test_Validate_WhenRequiredStringMissing_ReturnsError(t *testing.T) { + for name, cfg := range map[string]*Config{ + "no databasus-host": {AgentID: "a", Token: "t", MaxCPU: 4, MaxRAMMb: 2048, MaxDiskGb: 10, MaxConcurrentJobs: 1}, + "no agent-id": {DatabasusHost: "https://x:4005", Token: "t", MaxCPU: 4, MaxRAMMb: 2048, MaxDiskGb: 10, MaxConcurrentJobs: 1}, + "no token": {DatabasusHost: "https://x:4005", AgentID: "a", MaxCPU: 4, MaxRAMMb: 2048, MaxDiskGb: 10, MaxConcurrentJobs: 1}, + } { + t.Run(name, func(t *testing.T) { + _, err := cfg.Validate() + require.Error(t, err) + }) + } +} + +func Test_Validate_WhenStringsSetButCapacityInvalid_ReturnsCapacityError(t *testing.T) { + cfg := &Config{ + DatabasusHost: "https://primary.example:4005", + AgentID: "agent-1", + Token: "secret", + MaxCPU: 2, + MaxRAMMb: 4096, + MaxDiskGb: 100, + MaxConcurrentJobs: 4, + } + + _, err := cfg.Validate() + + require.Error(t, err) + assert.Contains(t, err.Error(), "CPU per job") +} + +func Test_DeriveCapacity_WhenConfigValid_DerivesPerJobSplit(t *testing.T) { + cfg := &Config{MaxCPU: 8, MaxRAMMb: 4096, MaxDiskGb: 100, MaxConcurrentJobs: 4} + + capacity, err := cfg.DeriveCapacity() + + require.NoError(t, err) + assert.Equal(t, 2, capacity.CPUPerJob) + assert.Equal(t, 1024, capacity.RAMMbPerJob) + assert.Equal(t, 8, capacity.MaxCPU) + assert.Equal(t, 4096, capacity.MaxRAMMb) + assert.Equal(t, 100, capacity.MaxDiskGb) + assert.Equal(t, 4, capacity.MaxConcurrentJobs) +} + +func Test_DeriveCapacity_WhenConcurrentJobsExceedCPU_ReturnsError(t *testing.T) { + cfg := &Config{MaxCPU: 2, MaxRAMMb: 4096, MaxDiskGb: 100, MaxConcurrentJobs: 4} + + _, err := cfg.DeriveCapacity() + + require.Error(t, err) + assert.Contains(t, err.Error(), "CPU per job") +} + +func Test_DeriveCapacity_WhenRAMBelowFloor_ReturnsError(t *testing.T) { + cfg := &Config{MaxCPU: 4, MaxRAMMb: 256, MaxDiskGb: 100, MaxConcurrentJobs: 1} + + _, err := cfg.DeriveCapacity() + + require.Error(t, err) + assert.Contains(t, err.Error(), "max-ram-mb") +} + +func Test_DeriveCapacity_WhenRAMPerJobBelowFloor_ReturnsError(t *testing.T) { + cfg := &Config{MaxCPU: 8, MaxRAMMb: 1024, MaxDiskGb: 100, MaxConcurrentJobs: 8} + + _, err := cfg.DeriveCapacity() + + require.Error(t, err) + assert.Contains(t, err.Error(), "per job") +} + +func Test_DeriveCapacity_WhenZeroFields_ReturnsError(t *testing.T) { + for name, cfg := range map[string]*Config{ + "no concurrent jobs": {MaxCPU: 4, MaxRAMMb: 2048, MaxDiskGb: 10, MaxConcurrentJobs: 0}, + "no cpu": {MaxCPU: 0, MaxRAMMb: 2048, MaxDiskGb: 10, MaxConcurrentJobs: 1}, + "no disk": {MaxCPU: 4, MaxRAMMb: 2048, MaxDiskGb: 0, MaxConcurrentJobs: 1}, + } { + t.Run(name, func(t *testing.T) { + _, err := cfg.DeriveCapacity() + require.Error(t, err) + }) + } +} + +func Test_ValidateTransport_WhenHTTPS_PassesWithoutPrompt(t *testing.T) { + cfg := &Config{DatabasusHost: "https://primary.example:4005"} + + err := cfg.ValidateTransport(false, emptyReader()) + + require.NoError(t, err) +} + +func Test_ValidateTransport_WhenHTTPNonTTYWithoutFlag_FailsNamingBothFixes(t *testing.T) { + cfg := &Config{DatabasusHost: "http://primary.example:4005"} + + err := cfg.ValidateTransport(false, emptyReader()) + + require.Error(t, err) + assert.Contains(t, err.Error(), "https://") + assert.Contains(t, err.Error(), "--allow-insecure-http") +} + +func Test_ValidateTransport_WhenHTTPWithAllowFlag_PassesWithWarn(t *testing.T) { + cfg := &Config{DatabasusHost: "http://primary.example:4005", AllowInsecureHTTP: true} + + err := cfg.ValidateTransport(false, emptyReader()) + + require.NoError(t, err) +} + +func Test_ValidateTransport_WhenHTTPTTYAndOperatorConsents_Passes(t *testing.T) { + cfg := &Config{DatabasusHost: "http://primary.example:4005"} + + err := cfg.ValidateTransport(true, bufio.NewReader(strings.NewReader("y\n"))) + + require.NoError(t, err) +} + +func Test_ValidateTransport_WhenHTTPTTYAndOperatorDeclines_ReturnsError(t *testing.T) { + cfg := &Config{DatabasusHost: "http://primary.example:4005"} + + err := cfg.ValidateTransport(true, bufio.NewReader(strings.NewReader("n\n"))) + + require.Error(t, err) +} + +func Test_ValidateTransport_WhenSchemeUnsupported_ReturnsError(t *testing.T) { + cfg := &Config{DatabasusHost: "ftp://primary.example:4005"} + + err := cfg.ValidateTransport(true, emptyReader()) + + require.Error(t, err) + assert.Contains(t, err.Error(), "https://") +} + +func Test_MaskSensitive_WhenEmpty_ReturnsNotSet(t *testing.T) { + assert.Equal(t, "(not set)", maskSensitive("")) +} + +func Test_MaskSensitive_WhenValue_RevealsQuarterThenMasks(t *testing.T) { + assert.Equal(t, "ab***", maskSensitive("abcdefgh")) +} + +func Test_GetVerificationPgImageRepo_WhenUnset_ReturnsBundledDefault(t *testing.T) { + cfg := &Config{} + + assert.Equal(t, DefaultVerificationPgImageRepo, cfg.GetVerificationPgImageRepo()) +} + +func Test_GetVerificationPgImageRepo_WhenSet_ReturnsConfiguredValue(t *testing.T) { + cfg := &Config{VerificationPgImageRepo: "registry.internal/pg-verify"} + + assert.Equal(t, "registry.internal/pg-verify", cfg.GetVerificationPgImageRepo()) +} + +func Test_LoadFromJSONAndArgs_WhenFlagsProvided_OverrideEveryValue(t *testing.T) { + t.Chdir(t.TempDir()) + + cfg := loadConfig(t, + "--databasus-host", "https://flag:4005", + "--agent-id", "agent-x", + "--token", "tok", + "--max-cpu", "8", + "--max-ram-mb", "4096", + "--max-disk-gb", "100", + "--max-concurrent-jobs", "4", + "--allow-insecure-http", + "--verification-pg-image-repo", "registry/pg", + ) + + assert.Equal(t, "https://flag:4005", cfg.DatabasusHost) + assert.Equal(t, "agent-x", cfg.AgentID) + assert.Equal(t, "tok", cfg.Token) + assert.Equal(t, 8, cfg.MaxCPU) + assert.Equal(t, 4096, cfg.MaxRAMMb) + assert.Equal(t, 100, cfg.MaxDiskGb) + assert.Equal(t, 4, cfg.MaxConcurrentJobs) + assert.True(t, cfg.AllowInsecureHTTP) + assert.Equal(t, "registry/pg", cfg.VerificationPgImageRepo) + + for _, name := range allFlagNames { + assert.Equal(t, "command line args", cfg.flagSources[name], name) + } +} + +func Test_LoadFromJSONAndArgs_WhenOnlyFilePresent_LoadsFileAndMarksSource(t *testing.T) { + t.Chdir(t.TempDir()) + source := &Config{DatabasusHost: "https://file:4005", AgentID: "agent-file", MaxCPU: 6} + require.NoError(t, source.SaveToJSON()) + + cfg := loadConfig(t) + + assert.Equal(t, "https://file:4005", cfg.DatabasusHost) + assert.Equal(t, "agent-file", cfg.AgentID) + assert.Equal(t, 6, cfg.MaxCPU) + assert.Equal(t, configFileName, cfg.flagSources["databasus-host"]) +} + +func Test_LoadFromJSONAndArgs_WhenFlagAndFile_FlagOverridesFile(t *testing.T) { + t.Chdir(t.TempDir()) + source := &Config{DatabasusHost: "https://file:4005", AgentID: "agent-file"} + require.NoError(t, source.SaveToJSON()) + + cfg := loadConfig(t, "--databasus-host", "https://flag:4005") + + assert.Equal(t, "https://flag:4005", cfg.DatabasusHost, "flag overrides file") + assert.Equal(t, "agent-file", cfg.AgentID, "untouched field keeps the file value") + assert.Equal(t, "command line args", cfg.flagSources["databasus-host"]) + assert.Equal(t, configFileName, cfg.flagSources["agent-id"]) +} + +func Test_LoadFromJSONAndArgs_WhenZeroValuedFlags_DoNotOverrideFile(t *testing.T) { + t.Chdir(t.TempDir()) + source := &Config{MaxCPU: 8, VerificationPgImageRepo: "registry/pg"} + require.NoError(t, source.SaveToJSON()) + + cfg := loadConfig(t, "--max-cpu", "0", "--verification-pg-image-repo", "") + + assert.Equal(t, 8, cfg.MaxCPU, "an explicit zero flag cannot unset a file value") + assert.Equal(t, "registry/pg", cfg.VerificationPgImageRepo) +} + +func Test_LoadFromJSONAndArgs_WhenAllowInsecureHTTPInFile_StaysTrueWithoutFlag(t *testing.T) { + t.Chdir(t.TempDir()) + source := &Config{AllowInsecureHTTP: true} + require.NoError(t, source.SaveToJSON()) + + cfg := loadConfig(t) + + assert.True(t, cfg.AllowInsecureHTTP) + assert.Equal(t, configFileName, cfg.flagSources["allow-insecure-http"]) +} + +func Test_LoadFromJSONAndArgs_WhenNothingProvided_AllSourcesNotConfigured(t *testing.T) { + t.Chdir(t.TempDir()) + + cfg := loadConfig(t) + + for _, name := range allFlagNames { + assert.Equal(t, "not configured", cfg.flagSources[name], name) + } +} diff --git a/agent/verification/internal/config/flags.go b/agent/verification/internal/config/flags.go new file mode 100644 index 0000000..0424ed3 --- /dev/null +++ b/agent/verification/internal/config/flags.go @@ -0,0 +1,166 @@ +package config + +import "flag" + +const ( + sourceNotConfigured = "not configured" + sourceCommandLine = "command line args" +) + +// flagBinding ties one CLI flag to its Config field for the whole lifecycle — +// registration, applying an explicitly-set value, source attribution, and +// logging. Collecting these per flag keeps adding a flag a one-entry change +// instead of edits across four parallel blocks that silently drift apart. +type flagBinding struct { + name string + usage string + + // register declares the flag on fs and returns an applier that copies an + // explicitly-set value onto c and records the command-line source. A + // zero/empty flag is treated as "not provided" — it never overrides nor + // unsets a value already loaded from JSON. + register func(fs *flag.FlagSet) func(c *Config, sources map[string]string) + + // isSet reports whether c already carries a value (loaded from JSON). + isSet func(c *Config) bool + + // value is the loggable value, masked for secrets. + value func(c *Config) any +} + +func stringBinding( + name, usage string, + get func(*Config) string, + set func(*Config, string), +) flagBinding { + return flagBinding{ + name: name, + usage: usage, + register: func(fs *flag.FlagSet) func(*Config, map[string]string) { + value := fs.String(name, "", usage) + + return func(c *Config, sources map[string]string) { + if *value != "" { + set(c, *value) + sources[name] = sourceCommandLine + } + } + }, + isSet: func(c *Config) bool { return get(c) != "" }, + value: func(c *Config) any { return get(c) }, + } +} + +func secretBinding( + name, usage string, + get func(*Config) string, + set func(*Config, string), +) flagBinding { + binding := stringBinding(name, usage, get, set) + binding.value = func(c *Config) any { return maskSensitive(get(c)) } + + return binding +} + +func intBinding( + name, usage string, + get func(*Config) int, + set func(*Config, int), +) flagBinding { + return flagBinding{ + name: name, + usage: usage, + register: func(fs *flag.FlagSet) func(*Config, map[string]string) { + value := fs.Int(name, 0, usage) + + return func(c *Config, sources map[string]string) { + if *value != 0 { + set(c, *value) + sources[name] = sourceCommandLine + } + } + }, + isSet: func(c *Config) bool { return get(c) != 0 }, + value: func(c *Config) any { return get(c) }, + } +} + +// boolBinding latches: the flag only ever turns the value on, so a persisted +// true is never flipped back off by its absence on the command line. +func boolBinding( + name, usage string, + get func(*Config) bool, + set func(*Config, bool), +) flagBinding { + return flagBinding{ + name: name, + usage: usage, + register: func(fs *flag.FlagSet) func(*Config, map[string]string) { + value := fs.Bool(name, false, usage) + + return func(c *Config, sources map[string]string) { + if *value { + set(c, true) + sources[name] = sourceCommandLine + } + } + }, + isSet: func(c *Config) bool { return get(c) }, + value: func(c *Config) any { return get(c) }, + } +} + +func (c *Config) flagBindings() []flagBinding { + return []flagBinding{ + stringBinding("databasus-host", "Databasus server URL (e.g. https://your-server:4005)", + func(c *Config) string { return c.DatabasusHost }, + func(c *Config, v string) { c.DatabasusHost = v }), + stringBinding("agent-id", "Verification agent ID", + func(c *Config) string { return c.AgentID }, + func(c *Config, v string) { c.AgentID = v }), + secretBinding("token", "Verification agent token", + func(c *Config) string { return c.Token }, + func(c *Config, v string) { c.Token = v }), + intBinding("max-cpu", "Total CPU cores available to the agent", + func(c *Config) int { return c.MaxCPU }, + func(c *Config, v int) { c.MaxCPU = v }), + intBinding("max-ram-mb", "Total RAM in MB available to the agent", + func(c *Config) int { return c.MaxRAMMb }, + func(c *Config, v int) { c.MaxRAMMb = v }), + intBinding("max-disk-gb", "Total scratch disk in GB available to the agent", + func(c *Config) int { return c.MaxDiskGb }, + func(c *Config, v int) { c.MaxDiskGb = v }), + intBinding("max-concurrent-jobs", "Number of verifications to run in parallel", + func(c *Config) int { return c.MaxConcurrentJobs }, + func(c *Config, v int) { c.MaxConcurrentJobs = v }), + boolBinding("allow-insecure-http", + "Permit a plain http:// databasus-host (token and backup data sent unencrypted)", + func(c *Config) bool { return c.AllowInsecureHTTP }, + func(c *Config, v bool) { c.AllowInsecureHTTP = v }), + stringBinding("verification-pg-image-repo", + "Container image repo bundling PostgreSQL extensions for verification "+ + "(default "+DefaultVerificationPgImageRepo+"; set to 'postgres' for the stock image)", + func(c *Config) string { return c.VerificationPgImageRepo }, + func(c *Config, v string) { c.VerificationPgImageRepo = v }), + } +} + +func sourcesFromConfig(bindings []flagBinding, c *Config) map[string]string { + sources := make(map[string]string, len(bindings)) + + for _, binding := range bindings { + if binding.isSet(c) { + sources[binding.name] = configFileName + } else { + sources[binding.name] = sourceNotConfigured + } + } + + return sources +} + +func (c *Config) logConfigSources(bindings []flagBinding) { + for _, binding := range bindings { + log.Info(binding.name, "value", binding.value(c), "source", c.flagSources[binding.name]) + } +} diff --git a/agent/verification/internal/features/api/api.go b/agent/verification/internal/features/api/api.go new file mode 100644 index 0000000..f0ac368 --- /dev/null +++ b/agent/verification/internal/features/api/api.go @@ -0,0 +1,315 @@ +package api + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log/slog" + "net/http" + "net/url" + "os" + "strings" + "time" + + "github.com/go-resty/resty/v2" + "github.com/google/uuid" +) + +// reportRetryDelay returns the exponential backoff for report attempt n +// (1-based): 1s,2s,4s,8s,16s,32s, capped at maxBackoff. The exponent is +// clamped before the shift so a large attempt count cannot overflow. +func reportRetryDelay(attempt int) time.Duration { + return expBackoff(attempt - 1) +} + +func expBackoff(shift int) time.Duration { + const maxShift = 5 // 2^5 s = 32s = maxBackoff + + if shift < 0 { + shift = 0 + } + + if shift >= maxShift { + return maxBackoff + } + + d := time.Duration(1< maxBackoff { + return maxBackoff + } + + return d +} + +type Client struct { + json *resty.Client + noRetry *resty.Client + streamHTTP *http.Client + host string + token string + agentID string + log *slog.Logger +} + +func NewClient(host, token, agentID string, log *slog.Logger) *Client { + setAuth := func(_ *resty.Client, req *resty.Request) error { + if h := bearerHeader(token); h != "" { + req.SetHeader("Authorization", h) + } + + return nil + } + + jsonClient := resty.New(). + SetTimeout(apiCallTimeout). + SetRetryCount(maxRetryAttempts - 1). + SetRetryWaitTime(retryBaseDelay). + SetRetryMaxWaitTime(4 * retryBaseDelay). + AddRetryCondition(func(resp *resty.Response, err error) bool { + return err != nil || resp.StatusCode() >= 500 + }). + SetHeaders(noCacheHeaders). + OnBeforeRequest(setAuth) + + // Claim and report own their retry policy explicitly (claim: loop retries + // forever; report: bounded retry-with-deadline). A retrying client here + // would make "exactly N retries / give up at the budget" untestable. + noRetryClient := resty.New(). + SetTimeout(apiCallTimeout). + SetHeaders(noCacheHeaders). + OnBeforeRequest(setAuth) + + return &Client{ + json: jsonClient, + noRetry: noRetryClient, + streamHTTP: &http.Client{}, + host: host, + token: token, + agentID: agentID, + log: log, + } +} + +func (c *Client) FetchServerVersion(ctx context.Context) (string, error) { + var ver versionResponse + + httpResp, err := c.json.R(). + SetContext(ctx). + SetResult(&ver). + Get(c.buildURL(versionPath)) + if err != nil { + return "", err + } + + if err := c.checkResponse(httpResp, "fetch server version"); err != nil { + return "", err + } + + return ver.Version, nil +} + +func (c *Client) DownloadVerificationAgentBinary( + ctx context.Context, + arch, destPath, fallbackVersion string, +) (string, error) { + requestURL := c.buildURL(verificationAgentBinaryPath) + "?" + url.Values{"arch": {arch}}.Encode() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, requestURL, nil) + if err != nil { + return "", fmt.Errorf("create agent download request: %w", err) + } + + c.setStreamHeaders(req) + + resp, err := c.streamHTTP.Do(req) + if err != nil { + return "", err + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("server returned %d for verification agent download", resp.StatusCode) + } + + file, err := os.Create(destPath) + if err != nil { + return "", err + } + defer func() { _ = file.Close() }() + + if _, err := io.Copy(file, resp.Body); err != nil { + return "", err + } + + downloadedVersion := strings.TrimSpace(resp.Header.Get(agentVersionHeader)) + if downloadedVersion == "" { + return fallbackVersion, nil + } + + return downloadedVersion, nil +} + +func (c *Client) Heartbeat( + ctx context.Context, + request HeartbeatRequest, +) (*HeartbeatResponse, error) { + var response HeartbeatResponse + + httpResp, err := c.json.R(). + SetContext(ctx). + SetBody(request). + SetResult(&response). + Post(c.buildURL(fmt.Sprintf(heartbeatPathFmt, c.agentID))) + if err != nil { + return nil, err + } + + if err := c.checkResponse(httpResp, "heartbeat"); err != nil { + return nil, err + } + + return &response, nil +} + +// ClaimVerification performs a single claim attempt. 200 returns the assignment; +// 204 returns (nil, nil) meaning nothing fits; any other status is an error. +// The runner loop owns retry/backoff — this never loops. +func (c *Client) ClaimVerification( + ctx context.Context, + capacity AgentCapacity, +) (*JobAssignment, error) { + resp, err := c.noRetry.R(). + SetContext(ctx). + SetBody(ClaimRequest{Capacity: capacity}). + Post(c.buildURL(fmt.Sprintf(claimPathFmt, c.agentID))) + if err != nil { + return nil, err + } + + if resp.StatusCode() == http.StatusNoContent { + return nil, nil + } + + if resp.StatusCode() != http.StatusOK { + return nil, &ResponseError{Op: "claim", StatusCode: resp.StatusCode(), Body: resp.String()} + } + + var assignment JobAssignment + if err := json.Unmarshal(resp.Body(), &assignment); err != nil { + return nil, fmt.Errorf("decode job assignment: %w", err) + } + + return &assignment, nil +} + +// DownloadBackup opens the plaintext backup stream. On 200 it returns the +// response body for the caller to stream into the container. A non-2xx status +// is returned as *ResponseError so the caller can tell 410 (drop, no report) +// from other 4xx (agent-setup failure) from 5xx (retryable). A transport error +// is returned as-is (retryable). +func (c *Client) DownloadBackup( + ctx context.Context, + verificationID uuid.UUID, +) (io.ReadCloser, error) { + requestURL := c.buildURL(fmt.Sprintf(backupStreamPathFmt, c.agentID, verificationID)) + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, requestURL, nil) + if err != nil { + return nil, fmt.Errorf("create backup stream request: %w", err) + } + + c.setStreamHeaders(req) + + resp, err := c.streamHTTP.Do(req) + if err != nil { + return nil, err + } + + if resp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) + _ = resp.Body.Close() + + return nil, &ResponseError{ + Op: "backup stream", + StatusCode: resp.StatusCode, + Body: string(body), + } + } + + return resp.Body, nil +} + +// Report submits the terminal result with bounded retry-with-deadline. Transport +// errors and 5xx are retried with exponential backoff under reportRetryBudget; a +// 410 returns ErrReportGone immediately (zero retries); any other 4xx is a +// non-retryable error; budget exhaustion returns ErrReportBudgetExhausted so the +// run is reclaimed by the backend on the agent's next heartbeat. +func (c *Client) Report( + ctx context.Context, + verificationID uuid.UUID, + req ReportRequest, +) error { + ctx, cancel := context.WithTimeoutCause(ctx, reportRetryBudget, ErrReportBudgetExhausted) + defer cancel() + + requestURL := c.buildURL(fmt.Sprintf(reportPathFmt, c.agentID, verificationID)) + + for attempt := 1; ; attempt++ { + resp, err := c.noRetry.R(). + SetContext(ctx). + SetBody(req). + Post(requestURL) + + if err == nil { + switch { + case resp.StatusCode() == http.StatusNoContent: + return nil + case resp.StatusCode() == http.StatusGone: + return ErrReportGone + case resp.StatusCode() < 500: + return &ResponseError{ + Op: "report", + StatusCode: resp.StatusCode(), + Body: resp.String(), + } + } + } + + select { + case <-ctx.Done(): + return context.Cause(ctx) + case <-timeAfterFn(reportRetryDelay(attempt)): + } + } +} + +func (c *Client) buildURL(path string) string { + return c.host + path +} + +func (c *Client) checkResponse(resp *resty.Response, method string) error { + if resp.StatusCode() >= 400 { + return fmt.Errorf("%s: server returned status %d: %s", method, resp.StatusCode(), resp.String()) + } + + return nil +} + +func (c *Client) setStreamHeaders(req *http.Request) { + if h := bearerHeader(c.token); h != "" { + req.Header.Set("Authorization", h) + } + + for name, value := range noCacheHeaders { + req.Header.Set(name, value) + } +} + +func bearerHeader(token string) string { + if token == "" { + return "" + } + + return "Bearer " + token +} diff --git a/agent/verification/internal/features/api/api_test.go b/agent/verification/internal/features/api/api_test.go new file mode 100644 index 0000000..9d61e54 --- /dev/null +++ b/agent/verification/internal/features/api/api_test.go @@ -0,0 +1,429 @@ +package api + +import ( + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sync/atomic" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/testutil" +) + +func instantAfter(t *testing.T) { + t.Helper() + + original := timeAfterFn + timeAfterFn = func(time.Duration) <-chan time.Time { + ch := make(chan time.Time, 1) + ch <- time.Now() + + return ch + } + + t.Cleanup(func() { timeAfterFn = original }) +} + +func Test_ClaimVerification_WhenServer200_SendsNestedMaxRamMbAndDeserializes(t *testing.T) { + var capturedPath, capturedAuth string + var capturedBody map[string]any + + verificationID := uuid.New() + backupID := uuid.New() + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + capturedPath = r.URL.Path + capturedAuth = r.Header.Get("Authorization") + require.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody)) + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "verificationId": verificationID.String(), + "backupId": backupID.String(), + "backupSizeMb": 120.5, + "maxContainerDiskMb": 800.25, + "database": map[string]any{ + "type": "POSTGRES_LOGICAL", + "postgresqlLogical": map[string]any{"version": "16"}, + }, + }) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "tok123", "agent-9", testutil.DiscardLogger()) + + assignment, err := client.ClaimVerification(t.Context(), AgentCapacity{ + MaxCPU: 8, MaxRAMMb: 4096, MaxDiskGb: 100, MaxConcurrentJobs: 4, + }) + + require.NoError(t, err) + require.NotNil(t, assignment) + + assert.Equal(t, "/api/v1/agent/verifications/agent-9/claim", capturedPath) + assert.Equal(t, "Bearer tok123", capturedAuth) + + capacity, ok := capturedBody["capacity"].(map[string]any) + require.True(t, ok, "claim body must nest capacity under \"capacity\"") + assert.Equal(t, float64(4096), capacity["maxRamMb"]) + _, hasRamGb := capacity["maxRamGb"] + assert.False(t, hasRamGb, "claim envelope must use maxRamMb, never maxRamGb") + + assert.Equal(t, verificationID, assignment.VerificationID) + assert.Equal(t, backupID, assignment.BackupID) + assert.InDelta(t, 120.5, assignment.BackupSizeMb, 0.001) + assert.InDelta(t, 800.25, assignment.MaxContainerDiskMb, 0.001) + assert.Equal(t, "POSTGRES_LOGICAL", assignment.Database.Type) + require.NotNil(t, assignment.Database.PostgresqlLogical) + assert.Equal(t, "16", assignment.Database.PostgresqlLogical.Version) +} + +func Test_ClaimVerification_WhenServer204_ReturnsNilNoError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "t", "a", testutil.DiscardLogger()) + + assignment, err := client.ClaimVerification(t.Context(), AgentCapacity{}) + + require.NoError(t, err) + assert.Nil(t, assignment) +} + +func Test_ClaimVerification_WhenServer500_ReturnsRetryableResponseError(t *testing.T) { + var calls atomic.Int32 + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls.Add(1) + w.WriteHeader(http.StatusInternalServerError) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "t", "a", testutil.DiscardLogger()) + + _, err := client.ClaimVerification(t.Context(), AgentCapacity{}) + + require.Error(t, err) + assert.Equal(t, int32(1), calls.Load(), "claim must not retry internally — the loop owns retry") + + var respErr *ResponseError + require.True(t, errors.As(err, &respErr)) + assert.True(t, respErr.Retryable()) +} + +func Test_DownloadBackup_WhenServer200_ReturnsBody(t *testing.T) { + payload := []byte("PGDMP fake custom-format archive") + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, + "/api/v1/agent/verifications/agent-x/"+ + "22222222-2222-2222-2222-222222222222/backup-stream", + r.URL.Path) + _, _ = w.Write(payload) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "agent-x", testutil.DiscardLogger()) + + body, err := client.DownloadBackup( + t.Context(), uuid.MustParse("22222222-2222-2222-2222-222222222222")) + require.NoError(t, err) + t.Cleanup(func() { _ = body.Close() }) + + read, err := io.ReadAll(body) + require.NoError(t, err) + assert.Equal(t, payload, read) +} + +func Test_DownloadBackup_WhenServer410_ReturnsGoneResponseError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusGone) + _, _ = w.Write([]byte(`{"reason":"gone"}`)) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "a", testutil.DiscardLogger()) + + _, err := client.DownloadBackup(t.Context(), uuid.New()) + + require.Error(t, err) + var respErr *ResponseError + require.True(t, errors.As(err, &respErr)) + assert.True(t, respErr.IsGone()) + assert.False(t, respErr.Retryable()) +} + +func Test_Report_WhenServer204_Succeeds(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, "/api/v1/agent/verifications/a/33333333-3333-3333-3333-333333333333/report", + r.URL.Path) + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "a", testutil.DiscardLogger()) + + err := client.Report(t.Context(), + uuid.MustParse("33333333-3333-3333-3333-333333333333"), + ReportRequest{Status: VerificationStatusCompleted}) + + require.NoError(t, err) +} + +func Test_Report_WhenServer410_ReturnsErrReportGoneWithoutRetry(t *testing.T) { + var calls atomic.Int32 + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls.Add(1) + w.WriteHeader(http.StatusGone) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "a", testutil.DiscardLogger()) + + err := client.Report(t.Context(), uuid.New(), ReportRequest{Status: VerificationStatusFailed}) + + require.ErrorIs(t, err, ErrReportGone) + assert.Equal(t, int32(1), calls.Load(), "410 must not be retried") +} + +func Test_Report_When5xxThenRecovers_RetriesThenSucceeds(t *testing.T) { + instantAfter(t) + + var calls atomic.Int32 + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) < 3 { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + + w.WriteHeader(http.StatusNoContent) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "a", testutil.DiscardLogger()) + + err := client.Report(t.Context(), uuid.New(), ReportRequest{Status: VerificationStatusCompleted}) + + require.NoError(t, err) + assert.Equal(t, int32(3), calls.Load()) +} + +func Test_Report_When5xxAndBudgetExhausted_ReturnsBudgetError(t *testing.T) { + originalBudget := reportRetryBudget + reportRetryBudget = 25 * time.Millisecond + t.Cleanup(func() { reportRetryBudget = originalBudget }) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusBadGateway) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "a", testutil.DiscardLogger()) + + err := client.Report(t.Context(), uuid.New(), ReportRequest{Status: VerificationStatusFailed}) + + require.ErrorIs(t, err, ErrReportBudgetExhausted) +} + +func Test_Constants_SatisfyInvariants(t *testing.T) { + // reportRetryBudget must stay a short, bounded window: once it is exhausted + // the backend reclaims the run on the agent's next heartbeat, so retrying + // for minutes would be pointless. 5m is a generous sanity ceiling. + assert.Less(t, reportRetryBudget, 5*time.Minute) + + assert.Equal(t, maxBackoff, reportRetryDelay(99), "report backoff must cap at maxBackoff") + assert.Equal(t, 1*time.Second, reportRetryDelay(1)) + assert.Equal(t, 2*time.Second, reportRetryDelay(2)) +} + +func Test_Heartbeat_WhenCalled_SendsFlatEnvelopeWithBearerAndAgentPath(t *testing.T) { + var capturedPath, capturedMethod, capturedAuth string + var capturedBody map[string]any + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + capturedPath = r.URL.Path + capturedMethod = r.Method + capturedAuth = r.Header.Get("Authorization") + + require.NoError(t, json.NewDecoder(r.Body).Decode(&capturedBody)) + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": []string{}, + }) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "tok123", "11111111-1111-1111-1111-111111111111", testutil.DiscardLogger()) + + response, err := client.Heartbeat(t.Context(), HeartbeatRequest{ + MaxCPU: 8, + MaxRAMGb: 4, + MaxDiskGb: 100, + MaxConcurrentJobs: 4, + CurrentVerificationIDs: []uuid.UUID{}, + }) + + require.NoError(t, err) + require.NotNil(t, response) + + assert.Equal(t, http.MethodPost, capturedMethod) + assert.Equal(t, + "/api/v1/agent/verification/11111111-1111-1111-1111-111111111111/heartbeat", + capturedPath) + assert.Equal(t, "Bearer tok123", capturedAuth) + + assert.Equal(t, float64(8), capturedBody["maxCpu"]) + assert.Equal(t, float64(4), capturedBody["maxRamGb"]) + assert.Equal(t, float64(100), capturedBody["maxDiskGb"]) + assert.Equal(t, float64(4), capturedBody["maxConcurrentJobs"]) + + _, hasRamMb := capturedBody["maxRamMb"] + assert.False(t, hasRamMb, "heartbeat envelope must use maxRamGb, never maxRamMb") + + require.Contains(t, capturedBody, "currentVerificationIds") + assert.Equal(t, []any{}, capturedBody["currentVerificationIds"], + "currentVerificationIds must serialize as [] not null") +} + +func Test_Heartbeat_WhenServerReturns4xx_ReturnsErrorWithoutRetry(t *testing.T) { + var calls int + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + calls++ + w.WriteHeader(http.StatusUnauthorized) + _, _ = w.Write([]byte(`{"error":"invalid agent credentials"}`)) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "bad", "agent-1", testutil.DiscardLogger()) + + _, err := client.Heartbeat(t.Context(), HeartbeatRequest{CurrentVerificationIDs: []uuid.UUID{}}) + + require.Error(t, err) + assert.Equal(t, 1, calls, "4xx must not be retried") +} + +func Test_FetchServerVersion_WhenServerResponds_ReturnsVersion(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, "/api/v1/system/version", r.URL.Path) + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"version": "v3.27.0"}) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + + version, err := client.FetchServerVersion(t.Context()) + + require.NoError(t, err) + assert.Equal(t, "v3.27.0", version) +} + +func Test_DownloadVerificationAgentBinary_WhenServerServesFile_WritesItToDisk(t *testing.T) { + payload := []byte("\x7fELF fake verification agent") + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, "/api/v1/system/verification-agent", r.URL.Path) + assert.Equal(t, "arm64", r.URL.Query().Get("arch")) + w.Header().Set(agentVersionHeader, "v3.45.0") + _, _ = w.Write(payload) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + destPath := filepath.Join(t.TempDir(), "agent") + + downloadedVersion, err := client.DownloadVerificationAgentBinary( + t.Context(), "arm64", destPath, "v3.43.0", + ) + require.NoError(t, err) + assert.Equal(t, "v3.45.0", downloadedVersion) + + written, err := os.ReadFile(destPath) + require.NoError(t, err) + assert.Equal(t, payload, written) +} + +func Test_DownloadVerificationAgentBinary_WhenServerOmitsVersionHeader_ReturnsFallbackVersion(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte("binary")) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + + downloadedVersion, err := client.DownloadVerificationAgentBinary( + t.Context(), "amd64", filepath.Join(t.TempDir(), "agent"), "v3.43.0", + ) + + require.NoError(t, err) + assert.Equal(t, "v3.43.0", downloadedVersion) +} + +func Test_DownloadVerificationAgentBinary_WhenCalled_SendsNoCacheRequestHeaders(t *testing.T) { + var requestHeaders http.Header + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requestHeaders = r.Header.Clone() + _, _ = w.Write([]byte("binary")) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + + _, err := client.DownloadVerificationAgentBinary( + t.Context(), "amd64", filepath.Join(t.TempDir(), "agent"), "v1.0.0", + ) + require.NoError(t, err) + + assert.Equal(t, "no-cache", requestHeaders.Get("Cache-Control")) + assert.Equal(t, "no-cache", requestHeaders.Get("Pragma")) +} + +func Test_FetchServerVersion_WhenCalled_SendsNoCacheRequestHeaders(t *testing.T) { + var requestHeaders http.Header + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requestHeaders = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"version":"v3.45.0"}`)) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + + _, err := client.FetchServerVersion(t.Context()) + require.NoError(t, err) + + assert.Equal(t, "no-cache", requestHeaders.Get("Cache-Control")) + assert.Equal(t, "no-cache", requestHeaders.Get("Pragma")) +} + +func Test_DownloadVerificationAgentBinary_WhenServer404_ReturnsError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNotFound) + })) + t.Cleanup(server.Close) + + client := NewClient(server.URL, "", "", testutil.DiscardLogger()) + + _, err := client.DownloadVerificationAgentBinary( + t.Context(), "amd64", filepath.Join(t.TempDir(), "a"), "v1.0.0", + ) + + require.Error(t, err) +} diff --git a/agent/verification/internal/features/api/constants.go b/agent/verification/internal/features/api/constants.go new file mode 100644 index 0000000..233f59a --- /dev/null +++ b/agent/verification/internal/features/api/constants.go @@ -0,0 +1,37 @@ +package api + +import "time" + +const ( + versionPath = "/api/v1/system/version" + verificationAgentBinaryPath = "/api/v1/system/verification-agent" + heartbeatPathFmt = "/api/v1/agent/verification/%s/heartbeat" + + claimPathFmt = "/api/v1/agent/verifications/%s/claim" + backupStreamPathFmt = "/api/v1/agent/verifications/%s/%s/backup-stream" + reportPathFmt = "/api/v1/agent/verifications/%s/%s/report" + + apiCallTimeout = 30 * time.Second + maxRetryAttempts = 3 + retryBaseDelay = 1 * time.Second + + maxBackoff = 32 * time.Second + + agentVersionHeader = "X-Databasus-Version" +) + +// Forces revalidation at any intermediary already holding a stale entry — +// response-side no-store only stops a proxy from caching from now on. +var noCacheHeaders = map[string]string{ + "Cache-Control": "no-cache", + "Pragma": "no-cache", +} + +// timeAfterFn is time.After in production; tests swap it to avoid real sleeps in +// the report retry loop. reportRetryBudget is the report retry deadline (60s in +// production); tests shrink it to exercise budget exhaustion without a 60s +// wall-clock wait. +var ( + timeAfterFn = time.After + reportRetryBudget = 60 * time.Second +) diff --git a/agent/verification/internal/features/api/dto.go b/agent/verification/internal/features/api/dto.go new file mode 100644 index 0000000..760f283 --- /dev/null +++ b/agent/verification/internal/features/api/dto.go @@ -0,0 +1,88 @@ +package api + +import ( + "fmt" + "net/http" + "time" + + "github.com/google/uuid" +) + +type HeartbeatRequest struct { + MaxCPU int `json:"maxCpu"` + MaxRAMGb int `json:"maxRamGb"` + MaxDiskGb int `json:"maxDiskGb"` + MaxConcurrentJobs int `json:"maxConcurrentJobs"` + CurrentVerificationIDs []uuid.UUID `json:"currentVerificationIds"` +} + +type HeartbeatResponse struct { + LastSeenAt time.Time `json:"lastSeenAt"` + AbortVerificationIDs []uuid.UUID `json:"abortVerificationIds"` +} + +type versionResponse struct { + Version string `json:"version"` +} + +type AgentCapacity struct { + MaxCPU int `json:"maxCpu"` + MaxRAMMb int `json:"maxRamMb"` + MaxDiskGb int `json:"maxDiskGb"` + MaxConcurrentJobs int `json:"maxConcurrentJobs"` +} + +type ClaimRequest struct { + Capacity AgentCapacity `json:"capacity"` +} + +type AssignedPostgresql struct { + Version string `json:"version"` +} + +type AssignedDatabase struct { + Type string `json:"type"` + PostgresqlLogical *AssignedPostgresql `json:"postgresqlLogical"` +} + +type JobAssignment struct { + VerificationID uuid.UUID `json:"verificationId"` + BackupID uuid.UUID `json:"backupId"` + BackupSizeMb float64 `json:"backupSizeMb"` + MaxContainerDiskMb float64 `json:"maxContainerDiskMb"` + Database AssignedDatabase `json:"database"` + TimescaledbVersion string `json:"timescaledbVersion"` +} + +type ReportTableStat struct { + SchemaName string `json:"schemaName"` + Name string `json:"name"` + RowCount int64 `json:"rowCount"` +} + +type ReportRequest struct { + Status VerificationStatus `json:"status"` + PgRestoreExitCode *int `json:"pgRestoreExitCode"` + FailureKind *FailureType `json:"failureKind"` + RestoreDurationMs *int64 `json:"restoreDurationMs"` + VerifyDurationMs *int64 `json:"verifyDurationMs"` + DBSizeBytesAfterRestore *int64 `json:"dbSizeBytesAfterRestore"` + TableCount *int `json:"tableCount"` + SchemaCount *int `json:"schemaCount"` + TableStats []ReportTableStat `json:"tableStats"` + FailMessage *string `json:"failMessage"` +} + +type ResponseError struct { + Op string + StatusCode int + Body string +} + +func (e *ResponseError) Error() string { + return fmt.Sprintf("%s: server returned status %d: %s", e.Op, e.StatusCode, e.Body) +} + +func (e *ResponseError) Retryable() bool { return e.StatusCode >= 500 } + +func (e *ResponseError) IsGone() bool { return e.StatusCode == http.StatusGone } diff --git a/agent/verification/internal/features/api/enums.go b/agent/verification/internal/features/api/enums.go new file mode 100644 index 0000000..fe31ed6 --- /dev/null +++ b/agent/verification/internal/features/api/enums.go @@ -0,0 +1,15 @@ +package api + +type VerificationStatus string + +const ( + VerificationStatusCompleted VerificationStatus = "COMPLETED" + VerificationStatusFailed VerificationStatus = "FAILED" +) + +type FailureType string + +// FailureKindDiskLimitExceeded tells the backend the restore exceeded its +// server-computed per-job disk budget — a terminal verdict, distinct from the +// retryable nil-exit-code path. +const FailureKindDiskLimitExceeded FailureType = "DISK_LIMIT_EXCEEDED" diff --git a/agent/verification/internal/features/api/errors.go b/agent/verification/internal/features/api/errors.go new file mode 100644 index 0000000..9c3f4b8 --- /dev/null +++ b/agent/verification/internal/features/api/errors.go @@ -0,0 +1,12 @@ +package api + +import "errors" + +// ErrReportGone signals the report route returned 410 — the row is no longer +// this agent's. The caller logs at info and drops; it is never retried. +var ErrReportGone = errors.New("verification no longer owned by this agent") + +// ErrReportBudgetExhausted is the cause attached when report retries exceed +// reportRetryBudget; the run is then reclaimed by the backend on the agent's +// next heartbeat. +var ErrReportBudgetExhausted = errors.New("report retry budget exhausted") diff --git a/agent/verification/internal/features/api/idle_timeout_reader.go b/agent/verification/internal/features/api/idle_timeout_reader.go new file mode 100644 index 0000000..5f4c914 --- /dev/null +++ b/agent/verification/internal/features/api/idle_timeout_reader.go @@ -0,0 +1,60 @@ +package api + +import ( + "context" + "fmt" + "io" + "time" +) + +// IdleTimeoutReader wraps an io.Reader and cancels the associated context +// if no bytes are successfully read within the specified timeout duration. +// This detects stalled uploads where the network or source stops transmitting data. +// +// When the idle timeout fires, the reader is also closed (if it implements io.Closer) +// to unblock any goroutine blocked on the underlying Read. +type IdleTimeoutReader struct { + reader io.Reader + timeout time.Duration + cancel context.CancelCauseFunc + timer *time.Timer +} + +// NewIdleTimeoutReader creates a reader that cancels the context via cancel +// if Read does not return any bytes for the given timeout duration. +func NewIdleTimeoutReader(reader io.Reader, timeout time.Duration, cancel context.CancelCauseFunc) *IdleTimeoutReader { + r := &IdleTimeoutReader{ + reader: reader, + timeout: timeout, + cancel: cancel, + } + + r.timer = time.AfterFunc(timeout, func() { + cancel(fmt.Errorf("upload idle timeout: no bytes transmitted for %v", timeout)) + + if closer, ok := reader.(io.Closer); ok { + _ = closer.Close() + } + }) + + return r +} + +func (r *IdleTimeoutReader) Read(p []byte) (int, error) { + n, err := r.reader.Read(p) + + if n > 0 { + r.timer.Reset(r.timeout) + } + + if err != nil && err != io.EOF { + r.Stop() + } + + return n, err +} + +// Stop cancels the idle timer. Must be called when the reader is no longer needed. +func (r *IdleTimeoutReader) Stop() { + r.timer.Stop() +} diff --git a/agent/verification/internal/features/api/idle_timeout_reader_test.go b/agent/verification/internal/features/api/idle_timeout_reader_test.go new file mode 100644 index 0000000..17d52ff --- /dev/null +++ b/agent/verification/internal/features/api/idle_timeout_reader_test.go @@ -0,0 +1,112 @@ +package api + +import ( + "context" + "fmt" + "io" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func Test_ReadThroughIdleTimeoutReader_WhenBytesFlowContinuously_DoesNotCancelContext(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + + pr, pw := io.Pipe() + + idleReader := NewIdleTimeoutReader(pr, 200*time.Millisecond, cancel) + defer idleReader.Stop() + + go func() { + for range 5 { + _, _ = pw.Write([]byte("data")) + time.Sleep(50 * time.Millisecond) + } + + _ = pw.Close() + }() + + data, err := io.ReadAll(idleReader) + + require.NoError(t, err) + assert.Equal(t, "datadatadatadatadata", string(data)) + assert.NoError(t, ctx.Err(), "context should not be cancelled when bytes flow continuously") +} + +func Test_ReadThroughIdleTimeoutReader_WhenNoBytesTransmitted_CancelsContext(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + + pr, _ := io.Pipe() + + idleReader := NewIdleTimeoutReader(pr, 100*time.Millisecond, cancel) + defer idleReader.Stop() + + time.Sleep(200 * time.Millisecond) + + assert.Error(t, ctx.Err(), "context should be cancelled when no bytes are transmitted") + assert.Contains(t, context.Cause(ctx).Error(), "upload idle timeout") +} + +func Test_ReadThroughIdleTimeoutReader_WhenBytesStopMidStream_CancelsContext(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + + pr, pw := io.Pipe() + + idleReader := NewIdleTimeoutReader(pr, 100*time.Millisecond, cancel) + defer idleReader.Stop() + + go func() { + _, _ = pw.Write([]byte("initial")) + // Stop writing — simulate stalled source + }() + + buf := make([]byte, 1024) + n, _ := idleReader.Read(buf) + assert.Equal(t, "initial", string(buf[:n])) + + time.Sleep(200 * time.Millisecond) + + assert.Error(t, ctx.Err(), "context should be cancelled when bytes stop mid-stream") + assert.Contains(t, context.Cause(ctx).Error(), "upload idle timeout") +} + +func Test_StopIdleTimeoutReader_WhenCalledBeforeTimeout_DoesNotCancelContext(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + + pr, _ := io.Pipe() + + idleReader := NewIdleTimeoutReader(pr, 100*time.Millisecond, cancel) + idleReader.Stop() + + time.Sleep(200 * time.Millisecond) + + assert.NoError(t, ctx.Err(), "context should not be cancelled when reader is stopped before timeout") +} + +func Test_ReadThroughIdleTimeoutReader_WhenReaderReturnsError_PropagatesError(t *testing.T) { + ctx, cancel := context.WithCancelCause(t.Context()) + defer cancel(nil) + + pr, pw := io.Pipe() + + idleReader := NewIdleTimeoutReader(pr, 5*time.Second, cancel) + defer idleReader.Stop() + + expectedErr := fmt.Errorf("test read error") + _ = pw.CloseWithError(expectedErr) + + buf := make([]byte, 1024) + _, err := idleReader.Read(buf) + + assert.ErrorIs(t, err, expectedErr) + + // Timer should be stopped after error — context should not be cancelled + time.Sleep(100 * time.Millisecond) + assert.NoError(t, ctx.Err(), "context should not be cancelled after reader error stops the timer") +} diff --git a/agent/verification/internal/features/container/container.go b/agent/verification/internal/features/container/container.go new file mode 100644 index 0000000..bcdf85c --- /dev/null +++ b/agent/verification/internal/features/container/container.go @@ -0,0 +1,108 @@ +package container + +import ( + "context" + "fmt" + "io" + "log/slog" + "strconv" + "strings" + "time" + + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/restore" +) + +const ( + containerTerminateTimeout = 30 * time.Second + + restoreUser = "postgres" + restoreDB = "postgres" +) + +// PostgresContainer exposes two distinct Postgres conns. pg_restore runs +// INSIDE the container and reaches the DB on the container's own loopback +// (GetInContainerConn). The agent-process verifier runs OUTSIDE the container +// and reaches the DB via the random 127.0.0.1 host port the container's 5432 +// was published to (GetVerifierConn) — populated by Manager.Spawn after the +// container starts. +type PostgresContainer struct { + engine *dockerEngine + id string + networkID string + hostPort int + password string + log *slog.Logger +} + +func (c *PostgresContainer) Exec( + ctx context.Context, cmd []string, stdin io.Reader, env []string, +) (restore.ExecResult, error) { + return c.engine.Exec(ctx, c.id, cmd, stdin, env) +} + +func (c *PostgresContainer) GetInContainerConn() dbconn.Conn { + return dbconn.Conn{ + Host: "127.0.0.1", Port: pgInternalPort, + User: restoreUser, Password: c.password, Database: restoreDB, + } +} + +func (c *PostgresContainer) GetVerifierConn() dbconn.Conn { + return dbconn.Conn{ + Host: "127.0.0.1", Port: c.hostPort, + User: restoreUser, Password: c.password, Database: restoreDB, + } +} + +// GetDiskUsageBytes sums the byte footprint of the dirs the restore writes +// into: PGDATA (an anonymous volume declared by the postgres image, NOT in +// SizeRw) plus /tmp (where the agent stages the dump). Daemon-reported SizeRw +// alone misses PGDATA entirely; an in-container `du -sb` does not. +func (c *PostgresContainer) GetDiskUsageBytes(ctx context.Context) (int64, error) { + res, err := c.engine.Exec(ctx, c.id, + []string{"du", "-sb", "/var/lib/postgresql/data", "/tmp"}, nil, nil) + if err != nil { + return 0, fmt.Errorf("exec du: %w", err) + } + + if res.ExitCode != 0 { + return 0, fmt.Errorf("du exited %d: %s", res.ExitCode, strings.TrimSpace(res.Stderr)) + } + + var total int64 + for line := range strings.SplitSeq(strings.TrimRight(res.Stdout, "\n"), "\n") { + fields := strings.Fields(line) + if len(fields) == 0 { + continue + } + + n, parseErr := strconv.ParseInt(fields[0], 10, 64) + if parseErr != nil { + return 0, fmt.Errorf("parse du line %q: %w", line, parseErr) + } + + total += n + } + + return total, nil +} + +// Terminate removes the container and its network. The caller's context is +// intentionally ignored: teardown runs during job shutdown when that context is +// usually already cancelled, so the removal deadline is rooted in a fresh +// background context. The parameter is kept to satisfy runner.JobContainer. +func (c *PostgresContainer) Terminate(context.Context) error { + ctx, cancel := context.WithTimeout(context.Background(), containerTerminateTimeout) + defer cancel() + + rmErr := c.engine.RemoveContainer(ctx, c.id) + + if c.networkID != "" { + if netErr := c.engine.RemoveNetwork(ctx, c.networkID); netErr != nil && rmErr == nil { + rmErr = netErr + } + } + + return rmErr +} diff --git a/agent/verification/internal/features/container/dockerengine.go b/agent/verification/internal/features/container/dockerengine.go new file mode 100644 index 0000000..c535495 --- /dev/null +++ b/agent/verification/internal/features/container/dockerengine.go @@ -0,0 +1,339 @@ +// Package container owns the ephemeral Postgres container lifecycle. Every +// Docker-SDK call is confined to this file; the Manager builds a hardened +// SpawnSpec (unit-tested) that this file translates 1:1 to Docker. +package container + +import ( + "bytes" + "context" + "fmt" + "io" + "net/netip" + "strconv" + "strings" + + "github.com/moby/moby/api/pkg/stdcopy" + "github.com/moby/moby/api/types/container" + "github.com/moby/moby/api/types/network" + "github.com/moby/moby/client" + + "databasus-verification-agent/internal/features/restore" +) + +const ( + // LabelVerificationID / LabelAgentID tag every container and network so the + // startup purge can find and remove this agent's leftovers by agent ID + LabelVerificationID = "databasus.verification.verification_id" + LabelAgentID = "databasus.verification.agent_id" + + containerPidsLimit = 512 + + // pgInternalPort is the in-container Postgres port; it is published to an + // ephemeral 127.0.0.1 host port for the verifier. + pgInternalPort = 5432 +) + +// pgPublishedPort is the parsed network.Port key reused for ExposedPorts and +// PortBindings; ParsePort is the only constructor in the new SDK. +var pgPublishedPort = mustParsePort(strconv.Itoa(pgInternalPort) + "/tcp") + +func mustParsePort(s string) network.Port { + p, err := network.ParsePort(s) + if err != nil { + panic(fmt.Sprintf("parse port %q: %v", s, err)) + } + + return p +} + +type dockerEngine struct { + cli *client.Client +} + +func NewDockerEngine() (*dockerEngine, error) { + cli, err := client.New(client.FromEnv) + if err != nil { + return nil, fmt.Errorf("create docker client: %w", err) + } + + return &dockerEngine{cli: cli}, nil +} + +func (e *dockerEngine) Ping(ctx context.Context) error { + _, err := e.cli.Ping(ctx, client.PingOptions{}) + return err +} + +func (e *dockerEngine) UserNSRemapEnabled(ctx context.Context) (bool, error) { + info, err := e.cli.Info(ctx, client.InfoOptions{}) + if err != nil { + return false, err + } + + for _, opt := range info.Info.SecurityOptions { + if strings.Contains(opt, "name=userns") { + return true, nil + } + } + + return false, nil +} + +func (e *dockerEngine) EnsureImage(ctx context.Context, ref string) error { + if _, err := e.cli.ImageInspect(ctx, ref); err == nil { + return nil + } + + reader, err := e.cli.ImagePull(ctx, ref, client.ImagePullOptions{}) + if err != nil { + return fmt.Errorf("image pull: %w", err) + } + defer func() { _ = reader.Close() }() + + if _, err := io.Copy(io.Discard, reader); err != nil { + return fmt.Errorf("drain image pull: %w", err) + } + + return nil +} + +func (e *dockerEngine) CreateNetwork( + ctx context.Context, name string, labels map[string]string, +) (string, error) { + // Each job gets its own user-defined bridge — that isolates jobs from each + // other. We deliberately do NOT set Internal:true: Docker silently disables + // port publishing on internal networks, and the agent-process verifier can + // only reach the restored DB via the published host port (verifier package + // doc). Outbound from PG to the host network is therefore not blocked at + // the docker layer; rely on the host firewall + the other hardening + // controls (cap-drop, no-new-privs, pids/memory, ephemeral, userns). + resp, err := e.cli.NetworkCreate(ctx, name, client.NetworkCreateOptions{ + Driver: "bridge", + Labels: labels, + }) + if err != nil { + return "", err + } + + return resp.ID, nil +} + +func (e *dockerEngine) RemoveNetwork(ctx context.Context, networkID string) error { + _, err := e.cli.NetworkRemove(ctx, networkID, client.NetworkRemoveOptions{}) + return err +} + +func (e *dockerEngine) CreateContainer(ctx context.Context, spec SpawnSpec) (string, error) { + securityOpt := []string{} + if spec.NoNewPrivileges { + securityOpt = append(securityOpt, "no-new-privileges") + } + + capDrop := []string{} + if spec.CapDropAll { + capDrop = append(capDrop, "ALL") + } + + var pidsLimit *int64 + if spec.PidsLimit > 0 { + limit := spec.PidsLimit + pidsLimit = &limit + } + + hostConfig := &container.HostConfig{ + SecurityOpt: securityOpt, + CapDrop: capDrop, + CapAdd: spec.CapAdd, + ReadonlyRootfs: spec.ReadonlyRootfs, + NetworkMode: container.NetworkMode(spec.NetworkID), + PortBindings: network.PortMap{ + pgPublishedPort: []network.PortBinding{ + {HostIP: netip.MustParseAddr("127.0.0.1"), HostPort: "0"}, + }, + }, + Resources: container.Resources{ + NanoCPUs: spec.NanoCPUs, + Memory: spec.MemoryBytes, + PidsLimit: pidsLimit, + }, + } + + resp, err := e.cli.ContainerCreate(ctx, client.ContainerCreateOptions{ + Config: &container.Config{ + Image: spec.Image, + Cmd: spec.Cmd, + Env: spec.Env, + Labels: spec.Labels, + ExposedPorts: network.PortSet{pgPublishedPort: struct{}{}}, + }, + HostConfig: hostConfig, + Name: spec.Name, + }) + if err != nil { + return "", err + } + + return resp.ID, nil +} + +func (e *dockerEngine) StartContainer(ctx context.Context, containerID string) error { + _, err := e.cli.ContainerStart(ctx, containerID, client.ContainerStartOptions{}) + return err +} + +func (e *dockerEngine) HostPort( + ctx context.Context, containerID, containerPort string, +) (int, error) { + insp, err := e.cli.ContainerInspect(ctx, containerID, client.ContainerInspectOptions{}) + if err != nil { + return 0, err + } + + requestedPort, err := network.ParsePort(containerPort) + if err != nil { + return 0, fmt.Errorf("parse container port %q: %w", containerPort, err) + } + + bindings, ok := insp.Container.NetworkSettings.Ports[requestedPort] + if !ok || len(bindings) == 0 { + return 0, fmt.Errorf("container port %s is not published", containerPort) + } + + return strconv.Atoi(bindings[0].HostPort) +} + +func (e *dockerEngine) InspectSecurity( + ctx context.Context, containerID string, +) (SecurityState, error) { + insp, err := e.cli.ContainerInspect(ctx, containerID, client.ContainerInspectOptions{}) + if err != nil { + return SecurityState{}, err + } + + state := SecurityState{ + ReadonlyRootfs: insp.Container.HostConfig.ReadonlyRootfs, + // Only flag mounts the AGENT requested. insp.Mounts (runtime) includes + // image-declared anonymous volumes (postgres:16 declares VOLUME + // /var/lib/postgresql/data for PGDATA) that we explicitly want; only + // host bind mounts from our spec are a hardening regression. + HasHostBinds: len(insp.Container.HostConfig.Binds) > 0 || len(insp.Container.HostConfig.Mounts) > 0, + } + + for _, opt := range insp.Container.HostConfig.SecurityOpt { + if strings.Contains(opt, "no-new-privileges") { + state.NoNewPrivileges = true + } + } + + for _, cap := range insp.Container.HostConfig.CapDrop { + if cap == "ALL" { + state.CapDropAll = true + } + } + + return state, nil +} + +func (e *dockerEngine) Exec( + ctx context.Context, + containerID string, + cmd []string, + stdin io.Reader, + env []string, +) (restore.ExecResult, error) { + created, err := e.cli.ExecCreate(ctx, containerID, client.ExecCreateOptions{ + Cmd: cmd, + Env: env, + AttachStdin: stdin != nil, + AttachStdout: true, + AttachStderr: true, + }) + if err != nil { + return restore.ExecResult{}, fmt.Errorf("exec create: %w", err) + } + + attached, err := e.cli.ExecAttach(ctx, created.ID, client.ExecAttachOptions{}) + if err != nil { + return restore.ExecResult{}, fmt.Errorf("exec attach: %w", err) + } + defer attached.Close() + + // The hijacked conn's Read/Write are not ctx-aware: a container that wedges + // while consuming stdin (or producing output) would otherwise hang Exec + // indefinitely, defeating jobCtx/heartbeat-abort. Closing it on cancellation + // unblocks the in-flight copy; the deferred Close above is then a harmless + // second close. + stopOnCancel := context.AfterFunc(ctx, func() { attached.Close() }) + defer stopOnCancel() + + if stdin != nil { + copyErr := make(chan error, 1) + go func() { + _, cErr := io.Copy(attached.Conn, stdin) + _ = attached.CloseWrite() + copyErr <- cErr + }() + + if cErr := <-copyErr; cErr != nil { + return restore.ExecResult{}, fmt.Errorf("exec stdin copy: %w", cErr) + } + } + + var stdout, stderr bytes.Buffer + if _, err := stdcopy.StdCopy(&stdout, &stderr, attached.Reader); err != nil { + return restore.ExecResult{}, fmt.Errorf("exec stream copy: %w", err) + } + + inspect, err := e.cli.ExecInspect(ctx, created.ID, client.ExecInspectOptions{}) + if err != nil { + return restore.ExecResult{}, fmt.Errorf("exec inspect: %w", err) + } + + return restore.ExecResult{ + ExitCode: inspect.ExitCode, + Stdout: stdout.String(), + Stderr: stderr.String(), + }, nil +} + +func (e *dockerEngine) RemoveContainer(ctx context.Context, containerID string) error { + _, err := e.cli.ContainerRemove(ctx, containerID, client.ContainerRemoveOptions{ + Force: true, + RemoveVolumes: true, + }) + return err +} + +func (e *dockerEngine) ListManaged( + ctx context.Context, agentID string, +) ([]ManagedContainer, error) { + listed, err := e.cli.ContainerList(ctx, client.ContainerListOptions{ + All: true, + Filters: make(client.Filters).Add("label", LabelAgentID+"="+agentID), + }) + if err != nil { + return nil, err + } + + managed := make([]ManagedContainer, 0, len(listed.Items)) + for _, s := range listed.Items { + mc := ManagedContainer{ + ID: s.ID, + VerificationID: s.Labels[LabelVerificationID], + CreatedUnix: s.Created, + } + + if s.NetworkSettings != nil { + for _, endpoint := range s.NetworkSettings.Networks { + if endpoint.NetworkID != "" { + mc.NetworkID = endpoint.NetworkID + break + } + } + } + + managed = append(managed, mc) + } + + return managed, nil +} diff --git a/agent/verification/internal/features/container/fake_engine_test.go b/agent/verification/internal/features/container/fake_engine_test.go new file mode 100644 index 0000000..abddc92 --- /dev/null +++ b/agent/verification/internal/features/container/fake_engine_test.go @@ -0,0 +1,31 @@ +package container + +import "context" + +type fakePurgeEngine struct { + managed []ManagedContainer + removedContainer []string + removedNetwork []string + listErr error + removeContainerErr error +} + +func (f *fakePurgeEngine) ListManaged(context.Context, string) ([]ManagedContainer, error) { + return f.managed, f.listErr +} + +func (f *fakePurgeEngine) RemoveContainer(_ context.Context, id string) error { + if f.removeContainerErr != nil { + return f.removeContainerErr + } + + f.removedContainer = append(f.removedContainer, id) + + return nil +} + +func (f *fakePurgeEngine) RemoveNetwork(_ context.Context, id string) error { + f.removedNetwork = append(f.removedNetwork, id) + + return nil +} diff --git a/agent/verification/internal/features/container/managedcontainer.go b/agent/verification/internal/features/container/managedcontainer.go new file mode 100644 index 0000000..83a2a6a --- /dev/null +++ b/agent/verification/internal/features/container/managedcontainer.go @@ -0,0 +1,8 @@ +package container + +type ManagedContainer struct { + ID string + NetworkID string + VerificationID string + CreatedUnix int64 +} diff --git a/agent/verification/internal/features/container/manager.go b/agent/verification/internal/features/container/manager.go new file mode 100644 index 0000000..52f0225 --- /dev/null +++ b/agent/verification/internal/features/container/manager.go @@ -0,0 +1,302 @@ +package container + +import ( + "context" + "crypto/rand" + "encoding/hex" + "fmt" + "log/slog" + "strconv" + "time" + + "github.com/jackc/pgx/v5" + + "databasus-verification-agent/internal/features/dbconn" +) + +const ( + containerReadyTimeout = 60 * time.Second + readyPollInterval = 1 * time.Second + + // stockEngineImageRepo is the official Postgres image; the job's major version + // (from the backend assignment) is appended as the tag. It is the fallback + // when the configured (extension-bundling) repo can't be pulled. + stockEngineImageRepo = "postgres" + + // timescaleImageRepo carries the timescaledb extension; the tag is + // "-pg" so the engine matches the dump exactly + // (pg_restore cannot cross extension versions). + timescaleImageRepo = "timescale/timescaledb" +) + +// minimalCaps are the only capabilities added back after CapDrop ALL — exactly +// what the official entrypoint needs for initdb + the gosu privilege drop, not +// a blanket keep. +var minimalCaps = []string{"CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"} + +// restoreTunedPostgresCmd runs the restore container's Postgres with crash +// safety traded for a smaller on-disk footprint. The container is destroyed +// after verification, so durability is irrelevant: fsync=off and +// full_page_writes=off cut WAL volume, faster checkpoints recycle pg_wal +// sooner. max_wal_senders=0 is mandatory under wal_level=minimal — Postgres +// refuses to start otherwise. The first arg stays "postgres" so the official +// image entrypoint still runs initdb. +var restoreTunedPostgresCmd = []string{ + "postgres", + "-c", "fsync=off", + "-c", "full_page_writes=off", + "-c", "synchronous_commit=off", + "-c", "wal_level=minimal", + "-c", "max_wal_senders=0", + "-c", "max_wal_size=1GB", +} + +type Manager struct { + engine *dockerEngine + agentID string + engineImageRepo string + log *slog.Logger +} + +func NewManager( + engine *dockerEngine, + agentID string, + engineImageRepo string, + log *slog.Logger, +) *Manager { + return &Manager{ + engine: engine, + agentID: agentID, + engineImageRepo: engineImageRepo, + log: log, + } +} + +func (m *Manager) StartupSelfCheck(ctx context.Context) error { + if err := m.engine.Ping(ctx); err != nil { + return fmt.Errorf("docker daemon unreachable: %w", err) + } + + m.log.Info("docker daemon is reachable") + + remapOn, err := m.engine.UserNSRemapEnabled(ctx) + if err != nil { + m.log.Warn("could not determine user-namespace remap state", "error", err) + } else if !remapOn { + m.log.Warn( + "docker user-namespace remapping is OFF — the strongest control " + + "against a container escape is absent; enable userns-remap on the " + + "verification host") + } + + return nil +} + +// PurgeContainers removes every container+network this agent owns so each +// process starts from a clean slate. Call once at startup, after +// StartupSelfCheck (Docker must be reachable) and before any job is claimed. +// See purgeAgentContainers for why blanket removal is safe under one agent. +func (m *Manager) PurgeContainers(ctx context.Context) { + purgeAgentContainers(ctx, m.engine, m.agentID, m.log) +} + +// Spawn's every failure is pre-pg_restore — the runner reports it with no exit +// code (retryable AgentSetupFailed), never BackupRejected. +func (m *Manager) Spawn(jobCtx context.Context, req SpawnRequest) (*PostgresContainer, error) { + image := imageForJob(req, m.engineImageRepo) + + if err := m.engine.EnsureImage(jobCtx, image); err != nil { + fallbackImage, canFallBack := stockFallbackImage(req, image) + if !canFallBack { + return nil, fmt.Errorf("ensure image: %w", err) + } + + // The extension-bundling image is unreachable (registry down, air-gapped, + // not yet published). Fall back to the stock Postgres image; missing + // extensions then surface as ignored pg_restore errors the runner tolerates. + m.log.Warn("bundled verification image unavailable; falling back to stock postgres image", + "bundled_image", image, "stock_image", fallbackImage, "error", err) + + image = fallbackImage + + if err := m.engine.EnsureImage(jobCtx, image); err != nil { + return nil, fmt.Errorf("ensure stock image: %w", err) + } + } + + password, err := randomPassword() + if err != nil { + return nil, fmt.Errorf("generate password: %w", err) + } + + // LabelAgentID is load-bearing for the startup purge: ListManaged filters + // solely on it, so a container spawned without this label can never be + // purged and leaks until the host is cleaned by hand. + labels := map[string]string{ + LabelAgentID: m.agentID, + LabelVerificationID: req.VerificationID.String(), + } + + networkID, err := m.engine.CreateNetwork( + jobCtx, "databasus-verif-"+req.VerificationID.String(), labels) + if err != nil { + return nil, fmt.Errorf("create isolated network: %w", err) + } + + spec := m.buildSpec(spawnPlan{ + verificationID: req.VerificationID, + image: image, + password: password, + cpuPerJob: req.CPUPerJob, + ramMbPerJob: req.RAMMbPerJob, + networkID: networkID, + labels: labels, + }) + + containerID, err := m.engine.CreateContainer(jobCtx, spec) + if err != nil { + _ = m.engine.RemoveNetwork(jobCtx, networkID) + return nil, fmt.Errorf("create container: %w", err) + } + + c := &PostgresContainer{engine: m.engine, id: containerID, networkID: networkID, password: password, log: m.log} + + if err := m.engine.StartContainer(jobCtx, containerID); err != nil { + _ = c.Terminate(jobCtx) + return nil, fmt.Errorf("start container: %w", err) + } + + hostPort, err := m.engine.HostPort(jobCtx, containerID, strconv.Itoa(pgInternalPort)+"/tcp") + if err != nil { + _ = c.Terminate(jobCtx) + return nil, fmt.Errorf("resolve published port: %w", err) + } + + c.hostPort = hostPort + + if err := waitForReady(jobCtx, c.GetVerifierConn()); err != nil { + _ = c.Terminate(jobCtx) + return nil, fmt.Errorf("container never became ready: %w", err) + } + + if err := m.assertSecurity(jobCtx, containerID); err != nil { + _ = c.Terminate(jobCtx) + return nil, err + } + + return c, nil +} + +func (m *Manager) buildSpec(plan spawnPlan) SpawnSpec { + return SpawnSpec{ + Name: "databasus-verif-" + plan.verificationID.String(), + Image: plan.image, + Cmd: restoreTunedPostgresCmd, + Env: []string{"POSTGRES_PASSWORD=" + plan.password}, + Labels: plan.labels, + NanoCPUs: int64(plan.cpuPerJob) * 1_000_000_000, + MemoryBytes: int64(plan.ramMbPerJob) * 1024 * 1024, + PidsLimit: containerPidsLimit, + NetworkID: plan.networkID, + + NoNewPrivileges: true, + CapDropAll: true, + CapAdd: minimalCaps, + // rootfs is writable because the official image needs a writable PGDATA; + // the per-job disk budget is enforced by the agent's disk watcher, not a + // Docker cap. The dominant controls (cap-drop, no-new-privs, pids, + // memory, userns, ephemeral lifecycle) remain in force. The job network + // is a per-job user-defined bridge (job isolation) but NOT --internal: + // see CreateNetwork in dockerengine.go for why. + ReadonlyRootfs: false, + } +} + +func (m *Manager) assertSecurity(ctx context.Context, containerID string) error { + state, err := m.engine.InspectSecurity(ctx, containerID) + if err != nil { + return fmt.Errorf("inspect security state: %w", err) + } + + switch { + case !state.NoNewPrivileges: + return fmt.Errorf("hardening regression: no-new-privileges not applied") + case !state.CapDropAll: + return fmt.Errorf("hardening regression: CapDrop ALL not applied") + case state.HasHostBinds: + return fmt.Errorf("hardening regression: container has host bind mounts") + } + + return nil +} + +func waitForReady(ctx context.Context, conn dbconn.Conn) error { + readyCtx, cancel := context.WithTimeout(ctx, containerReadyTimeout) + defer cancel() + + ticker := time.NewTicker(readyPollInterval) + defer ticker.Stop() + + for { + if pingOnce(readyCtx, conn) { + return nil + } + + select { + case <-readyCtx.Done(): + return readyCtx.Err() + case <-ticker.C: + } + } +} + +func pingOnce(ctx context.Context, conn dbconn.Conn) bool { + pingCtx, cancel := context.WithTimeout(ctx, readyPollInterval) + defer cancel() + + pgConn, err := pgx.Connect(pingCtx, conn.DSN()) + if err != nil { + return false + } + defer func() { _ = pgConn.Close(pingCtx) }() + + return pgConn.Ping(pingCtx) == nil +} + +func imageForJob(req SpawnRequest, engineImageRepo string) string { + if req.TimescaledbVersion != "" { + return fmt.Sprintf("%s:%s-pg%s", timescaleImageRepo, req.TimescaledbVersion, req.PgMajor) + } + + return imageForMajor(req.PgMajor, engineImageRepo) +} + +func imageForMajor(pgMajor, engineImageRepo string) string { + return engineImageRepo + ":" + pgMajor +} + +// stockFallbackImage decides whether a failed pull of the configured image may +// retry against the official Postgres image. Timescale jobs need their exact +// version-matched image (pg_restore cannot cross extension versions), so they +// never fall back; nor does a job already targeting the stock repo. +func stockFallbackImage(req SpawnRequest, attemptedImage string) (string, bool) { + if req.TimescaledbVersion != "" { + return "", false + } + + stockImage := imageForMajor(req.PgMajor, stockEngineImageRepo) + if stockImage == attemptedImage { + return "", false + } + + return stockImage, true +} + +func randomPassword() (string, error) { + buf := make([]byte, 24) + if _, err := rand.Read(buf); err != nil { + return "", err + } + + return hex.EncodeToString(buf), nil +} diff --git a/agent/verification/internal/features/container/postgres_test.go b/agent/verification/internal/features/container/postgres_test.go new file mode 100644 index 0000000..ac7ea55 --- /dev/null +++ b/agent/verification/internal/features/container/postgres_test.go @@ -0,0 +1,129 @@ +package container + +import ( + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + "databasus-verification-agent/internal/testutil" +) + +func Test_BuildSpec_AppliesHardeningControls(t *testing.T) { + // nil engine: buildSpec is pure and never calls the engine. + containerManager := NewManager(nil, "agent-1", "databasus/verification-postgres", testutil.DiscardLogger()) + + labels := map[string]string{LabelAgentID: "agent-1"} + spec := containerManager.buildSpec(spawnPlan{ + verificationID: uuid.New(), + image: "postgres@sha256:x", + password: "pw", + cpuPerJob: 2, + ramMbPerJob: 1024, + networkID: "net-id", + labels: labels, + }) + + assert.True(t, spec.NoNewPrivileges) + assert.True(t, spec.CapDropAll) + assert.Equal(t, minimalCaps, spec.CapAdd) + assert.EqualValues(t, containerPidsLimit, spec.PidsLimit) + assert.Equal(t, "net-id", spec.NetworkID) + assert.Equal(t, int64(2)*1_000_000_000, spec.NanoCPUs) + assert.Equal(t, int64(1024)*1024*1024, spec.MemoryBytes) + assert.Equal(t, []string{"POSTGRES_PASSWORD=pw"}, spec.Env) +} + +func Test_BuildSpec_SetsRestoreTunedPostgresCmd(t *testing.T) { + containerManager := NewManager(nil, "agent-1", "databasus/verification-postgres", testutil.DiscardLogger()) + + spec := containerManager.buildSpec(spawnPlan{ + verificationID: uuid.New(), + image: "postgres@sha256:x", + password: "pw", + cpuPerJob: 2, + ramMbPerJob: 1024, + networkID: "net-id", + labels: map[string]string{LabelAgentID: "agent-1"}, + }) + + assert.Equal(t, restoreTunedPostgresCmd, spec.Cmd) + assert.Equal(t, "postgres", spec.Cmd[0]) +} + +func Test_GetInContainerConn_UsesInternalPort(t *testing.T) { + c := &PostgresContainer{password: "pw"} + + conn := c.GetInContainerConn() + + assert.Equal(t, "127.0.0.1", conn.Host) + assert.Equal(t, pgInternalPort, conn.Port) + assert.Equal(t, restoreUser, conn.User) + assert.Equal(t, restoreDB, conn.Database) + assert.Equal(t, "pw", conn.Password) +} + +func Test_GetVerifierConn_WhenSpawned_UsesResolvedHostPort(t *testing.T) { + c := &PostgresContainer{password: "pw", hostPort: 54321} + + conn := c.GetVerifierConn() + + assert.Equal(t, "127.0.0.1", conn.Host) + assert.Equal(t, 54321, conn.Port) + assert.Equal(t, restoreUser, conn.User) + assert.Equal(t, restoreDB, conn.Database) + assert.Equal(t, "pw", conn.Password) +} + +func Test_ImageForMajor_AppendsMajorTagToConfiguredRepo(t *testing.T) { + cases := []struct { + major string + expected string + }{ + {"12", "databasus/verification-postgres:12"}, + {"16", "databasus/verification-postgres:16"}, + {"18", "databasus/verification-postgres:18"}, + } + + for _, tc := range cases { + t.Run(tc.major, func(t *testing.T) { + assert.Equal(t, tc.expected, imageForMajor(tc.major, "databasus/verification-postgres")) + }) + } +} + +func Test_ImageForJob_WhenTimescaleVersionSet_ReturnsVersionMatchedTimescaleImage(t *testing.T) { + image := imageForJob( + SpawnRequest{PgMajor: "17", TimescaledbVersion: "2.17.0"}, "databasus/verification-postgres") + + assert.Equal(t, "timescale/timescaledb:2.17.0-pg17", image, + "timescale jobs ignore the engine repo and use the version-matched timescale image") +} + +func Test_ImageForJob_WhenNoTimescaleVersion_UsesConfiguredEngineRepo(t *testing.T) { + image := imageForJob(SpawnRequest{PgMajor: "17"}, "databasus/verification-postgres") + + assert.Equal(t, "databasus/verification-postgres:17", image) +} + +func Test_StockFallbackImage_WhenBundledEngineRepo_FallsBackToStockPostgres(t *testing.T) { + fallbackImage, canFallBack := stockFallbackImage( + SpawnRequest{PgMajor: "16"}, "databasus/verification-postgres:16") + + assert.True(t, canFallBack) + assert.Equal(t, "postgres:16", fallbackImage) +} + +func Test_StockFallbackImage_WhenAlreadyStockRepo_DoesNotFallBack(t *testing.T) { + _, canFallBack := stockFallbackImage(SpawnRequest{PgMajor: "16"}, "postgres:16") + + assert.False(t, canFallBack, "a job already on the stock image has nothing to fall back to") +} + +func Test_StockFallbackImage_WhenTimescaleJob_DoesNotFallBack(t *testing.T) { + _, canFallBack := stockFallbackImage( + SpawnRequest{PgMajor: "16", TimescaledbVersion: "2.17.0"}, "timescale/timescaledb:2.17.0-pg16") + + assert.False(t, canFallBack, + "timescale needs its exact version-matched image; stock postgres cannot restore it") +} diff --git a/agent/verification/internal/features/container/purge.go b/agent/verification/internal/features/container/purge.go new file mode 100644 index 0000000..009a296 --- /dev/null +++ b/agent/verification/internal/features/container/purge.go @@ -0,0 +1,58 @@ +package container + +import ( + "context" + "fmt" + "log/slog" +) + +// purgeEngine is the Docker seam for the startup purge — the same three methods +// the unit tests fake, rather than the whole Docker surface. +type purgeEngine interface { + ListManaged(ctx context.Context, agentID string) ([]ManagedContainer, error) + RemoveContainer(ctx context.Context, containerID string) error + RemoveNetwork(ctx context.Context, networkID string) error +} + +// purgeAgentContainers runs once at startup, with no age grace and no +// active-set check, so each agent process begins from a clean slate +// ("always fresh"). +// +// Best-effort by design: a per-item failure is logged and skipped, and a +// failed listing is logged and ignored, never blocking startup. Container and +// network names are per-UUID (databasus-verif-), so a leftover never +// collides with a fresh run; degraded (a few stale containers until the next +// restart) beats refusing to start on a transient Docker hiccup. +func purgeAgentContainers(ctx context.Context, engine purgeEngine, agentID string, log *slog.Logger) { + managed, err := engine.ListManaged(ctx, agentID) + if err != nil { + log.Warn("startup purge could not list managed containers", "error", err) + + return + } + + if len(managed) == 0 { + return + } + + removed := 0 + for _, mc := range managed { + if err := engine.RemoveContainer(ctx, mc.ID); err != nil { + log.Warn("startup purge failed to remove container", + "verification_id", mc.VerificationID, "error", err) + + continue + } + + if mc.NetworkID != "" { + if err := engine.RemoveNetwork(ctx, mc.NetworkID); err != nil { + log.Warn("startup purge failed to remove network", + "verification_id", mc.VerificationID, "error", err) + } + } + + removed++ + } + + log.Info(fmt.Sprintf("startup purge removed %d stale container(s)", removed)) +} diff --git a/agent/verification/internal/features/container/purge_test.go b/agent/verification/internal/features/container/purge_test.go new file mode 100644 index 0000000..7f394c1 --- /dev/null +++ b/agent/verification/internal/features/container/purge_test.go @@ -0,0 +1,74 @@ +package container + +import ( + "errors" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + "databasus-verification-agent/internal/testutil" +) + +func Test_PurgeAgentContainers_RemovesEveryContainerAndNetwork(t *testing.T) { + engine := &fakePurgeEngine{ + managed: []ManagedContainer{ + {ID: "c1", NetworkID: "n1", VerificationID: uuid.New().String()}, + {ID: "c2", NetworkID: "n2", VerificationID: uuid.New().String()}, + }, + } + + purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger()) + + assert.ElementsMatch(t, []string{"c1", "c2"}, engine.removedContainer) + assert.ElementsMatch(t, []string{"n1", "n2"}, engine.removedNetwork) +} + +func Test_PurgeAgentContainers_WhenNetworkIDEmpty_RemovesContainerOnly(t *testing.T) { + engine := &fakePurgeEngine{ + managed: []ManagedContainer{{ID: "c1", VerificationID: uuid.New().String()}}, + } + + purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger()) + + assert.Equal(t, []string{"c1"}, engine.removedContainer) + assert.Empty(t, engine.removedNetwork) +} + +func Test_PurgeAgentContainers_WhenListFails_RemovesNothing(t *testing.T) { + engine := &fakePurgeEngine{ + managed: []ManagedContainer{{ID: "c1", NetworkID: "n1"}}, + listErr: errors.New("docker unreachable"), + } + + purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger()) + + assert.Empty(t, engine.removedContainer) + assert.Empty(t, engine.removedNetwork) +} + +func Test_PurgeAgentContainers_WhenListEmpty_DoesNothing(t *testing.T) { + engine := &fakePurgeEngine{} + + purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger()) + + assert.Empty(t, engine.removedContainer) + assert.Empty(t, engine.removedNetwork) +} + +func Test_PurgeAgentContainers_WhenContainerRemovalFails_SkipsNetworkAndContinues(t *testing.T) { + engine := &fakePurgeEngine{ + managed: []ManagedContainer{ + {ID: "c1", NetworkID: "n1", VerificationID: uuid.New().String()}, + {ID: "c2", NetworkID: "n2", VerificationID: uuid.New().String()}, + }, + removeContainerErr: errors.New("container in use"), + } + + assert.NotPanics(t, func() { + purgeAgentContainers(t.Context(), engine, "agent-1", testutil.DiscardLogger()) + }) + + assert.Empty(t, engine.removedContainer) + assert.Empty(t, engine.removedNetwork) +} diff --git a/agent/verification/internal/features/container/securitystate.go b/agent/verification/internal/features/container/securitystate.go new file mode 100644 index 0000000..e56bba0 --- /dev/null +++ b/agent/verification/internal/features/container/securitystate.go @@ -0,0 +1,10 @@ +package container + +// SecurityState is read back from the live container so each job re-asserts its +// own hardening landed (a daemon glitch fails that job, never the process). +type SecurityState struct { + NoNewPrivileges bool + CapDropAll bool + ReadonlyRootfs bool + HasHostBinds bool +} diff --git a/agent/verification/internal/features/container/spawnspec.go b/agent/verification/internal/features/container/spawnspec.go new file mode 100644 index 0000000..f5ba02f --- /dev/null +++ b/agent/verification/internal/features/container/spawnspec.go @@ -0,0 +1,44 @@ +package container + +import "github.com/google/uuid" + +type SpawnRequest struct { + PgMajor string + CPUPerJob int + RAMMbPerJob int + VerificationID uuid.UUID + TimescaledbVersion string +} + +type spawnPlan struct { + verificationID uuid.UUID + image string + password string + cpuPerJob int + ramMbPerJob int + networkID string + labels map[string]string +} + +// SpawnSpec is the security-hardened container request. Every clause is a +// load-bearing control: the container is the only boundary around a restore +// that runs attacker-controlled code as the DB superuser. Constructed by the +// Manager (unit-testable) and translated 1:1 to Docker by dockerEngine. +type SpawnSpec struct { + Name string + Image string + Cmd []string + Env []string + Labels map[string]string + NanoCPUs int64 + MemoryBytes int64 + PidsLimit int64 + NetworkID string + + // The hardening invariants, explicit so a unit test asserts them and the + // engine translation cannot silently drop one. + NoNewPrivileges bool + CapDropAll bool + CapAdd []string + ReadonlyRootfs bool +} diff --git a/agent/verification/internal/features/dbconn/dbconn.go b/agent/verification/internal/features/dbconn/dbconn.go new file mode 100644 index 0000000..6640772 --- /dev/null +++ b/agent/verification/internal/features/dbconn/dbconn.go @@ -0,0 +1,18 @@ +package dbconn + +import "fmt" + +type Conn struct { + Host string + Port int + User string + Password string + Database string +} + +func (c Conn) DSN() string { + return fmt.Sprintf( + "host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", + c.Host, c.Port, c.User, c.Password, c.Database, + ) +} diff --git a/agent/verification/internal/features/heartbeat/heartbeat.go b/agent/verification/internal/features/heartbeat/heartbeat.go new file mode 100644 index 0000000..e26ceee --- /dev/null +++ b/agent/verification/internal/features/heartbeat/heartbeat.go @@ -0,0 +1,150 @@ +package heartbeat + +import ( + "context" + "fmt" + "log/slog" + "sync" + "sync/atomic" + "time" + + "github.com/google/uuid" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" +) + +const ( + jobName = "verification_heartbeat" + heartbeatInterval = 30 * time.Second +) + +// Heartbeater periodically reports the agent's capacity and the set of +// in-flight verification IDs (the backend reclaims any RUNNING job absent from +// that set), and acts on the abort list in the response. It is the single +// source of truth for which verifications are active and records the last +// abort set so the runner can re-check it immediately before any FAILED report +// (the abort/report-race mitigation). +type Heartbeater struct { + api *api.Client + capacity config.Capacity + hasRun atomic.Bool + + mu sync.Mutex + registry map[uuid.UUID]context.CancelFunc + lastAbort map[uuid.UUID]struct{} + + log *slog.Logger +} + +func NewHeartbeater(apiClient *api.Client, capacity config.Capacity, log *slog.Logger) *Heartbeater { + return &Heartbeater{ + api: apiClient, + capacity: capacity, + registry: make(map[uuid.UUID]context.CancelFunc), + lastAbort: make(map[uuid.UUID]struct{}), + log: log, + } +} + +func (h *Heartbeater) Run(ctx context.Context) { + if h.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", h)) + } + + logger := h.log.With("job_id", uuid.New(), "job_name", jobName) + logger.Info("heartbeat loop started") + + ticker := time.NewTicker(heartbeatInterval) + defer ticker.Stop() + + if ctx.Err() == nil { + h.sendHeartbeat(ctx, logger) + } + + for { + select { + case <-ctx.Done(): + logger.Info("heartbeat loop stopped") + + return + case <-ticker.C: + h.sendHeartbeat(ctx, logger) + } + } +} + +// TrackVerification must be called before the container/network exists so the +// ID rides the heartbeat envelope and is reachable by an abort the instant it +// has any artifact. +func (h *Heartbeater) TrackVerification(id uuid.UUID, cancel context.CancelFunc) { + h.mu.Lock() + defer h.mu.Unlock() + + h.registry[id] = cancel +} + +func (h *Heartbeater) UntrackVerification(id uuid.UUID) { + h.mu.Lock() + defer h.mu.Unlock() + + delete(h.registry, id) +} + +func (h *Heartbeater) GetRunningVerificationIDs() []uuid.UUID { + h.mu.Lock() + defer h.mu.Unlock() + + ids := make([]uuid.UUID, 0, len(h.registry)) + for id := range h.registry { + ids = append(ids, id) + } + + return ids +} + +func (h *Heartbeater) IsAborted(id uuid.UUID) bool { + h.mu.Lock() + defer h.mu.Unlock() + + _, aborted := h.lastAbort[id] + + return aborted +} + +func (h *Heartbeater) sendHeartbeat(ctx context.Context, logger *slog.Logger) { + request := api.HeartbeatRequest{ + MaxCPU: h.capacity.MaxCPU, + MaxRAMGb: h.capacity.MaxRAMMb / 1024, + MaxDiskGb: h.capacity.MaxDiskGb, + MaxConcurrentJobs: h.capacity.MaxConcurrentJobs, + CurrentVerificationIDs: h.GetRunningVerificationIDs(), + } + + response, err := h.api.Heartbeat(ctx, request) + if err != nil { + logger.Warn("heartbeat failed, will retry next tick", "error", err) + + return + } + + h.abortVerifications(response.AbortVerificationIDs, logger) + + logger.Debug(fmt.Sprintf( + "heartbeat ok: last_seen_at=%s", response.LastSeenAt.UTC().Format(time.RFC3339))) +} + +func (h *Heartbeater) abortVerifications(abortIDs []uuid.UUID, logger *slog.Logger) { + h.mu.Lock() + defer h.mu.Unlock() + + h.lastAbort = make(map[uuid.UUID]struct{}, len(abortIDs)) + for _, id := range abortIDs { + h.lastAbort[id] = struct{}{} + + if cancel, registered := h.registry[id]; registered { + logger.Info(fmt.Sprintf("aborting verification %s on server request", id)) + cancel() + } + } +} diff --git a/agent/verification/internal/features/heartbeat/heartbeat_test.go b/agent/verification/internal/features/heartbeat/heartbeat_test.go new file mode 100644 index 0000000..fa2484d --- /dev/null +++ b/agent/verification/internal/features/heartbeat/heartbeat_test.go @@ -0,0 +1,144 @@ +package heartbeat + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/testutil" +) + +func Test_Beat_WhenNoJobsRegistered_ReportsCapacityWithRamConvertedToGbAndNoVerificationIDs(t *testing.T) { + var body map[string]any + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + require.NoError(t, json.NewDecoder(r.Body).Decode(&body)) + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": []string{}, + }) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "tok", "agent-1", testutil.DiscardLogger()) + capacity := config.Capacity{ + MaxCPU: 8, + MaxRAMMb: 4096, + MaxDiskGb: 100, + MaxConcurrentJobs: 4, + } + + NewHeartbeater(client, capacity, testutil.DiscardLogger()).sendHeartbeat(t.Context(), testutil.DiscardLogger()) + + assert.Equal(t, float64(8), body["maxCpu"]) + assert.Equal(t, float64(4), body["maxRamGb"], "4096 MB must be reported as 4 GB") + assert.Equal(t, float64(100), body["maxDiskGb"]) + assert.Equal(t, float64(4), body["maxConcurrentJobs"]) + assert.Equal(t, []any{}, body["currentVerificationIds"]) +} + +func Test_Beat_WhenJobRegistered_ReportsItsIDAsCurrentVerificationID(t *testing.T) { + var body map[string]any + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + require.NoError(t, json.NewDecoder(r.Body).Decode(&body)) + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": []string{}, + }) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "tok", "agent-1", testutil.DiscardLogger()) + hb := NewHeartbeater(client, config.Capacity{}, testutil.DiscardLogger()) + + jobID := uuid.New() + hb.TrackVerification(jobID, func() {}) + + hb.sendHeartbeat(t.Context(), testutil.DiscardLogger()) + + ids, ok := body["currentVerificationIds"].([]any) + require.True(t, ok) + require.Len(t, ids, 1) + assert.Equal(t, jobID.String(), ids[0]) +} + +func Test_Beat_WhenResponseListsAbortID_CancelsMatchingJobAndRecordsAbort(t *testing.T) { + abortID := uuid.New() + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": []string{abortID.String()}, + }) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "tok", "agent-1", testutil.DiscardLogger()) + hb := NewHeartbeater(client, config.Capacity{}, testutil.DiscardLogger()) + + canceled := make(chan struct{}) + hb.TrackVerification(abortID, func() { close(canceled) }) + + hb.sendHeartbeat(t.Context(), testutil.DiscardLogger()) + + select { + case <-canceled: + case <-time.After(2 * time.Second): + t.Fatal("registered job's context was not cancelled on server abort") + } + + assert.True(t, hb.IsAborted(abortID), + "the abort set must be recorded for the runner's pre-FAILED re-check") +} + +func Test_Heartbeater_Run_WhenCalledTwice_Panics(t *testing.T) { + client := api.NewClient("http://127.0.0.1:0", "tok", "agent-1", testutil.DiscardLogger()) + heartbeater := NewHeartbeater(client, config.Capacity{}, testutil.DiscardLogger()) + + ctx, cancel := context.WithCancel(t.Context()) + cancel() + heartbeater.Run(ctx) + + assert.Panics(t, func() { heartbeater.Run(ctx) }) +} + +func Test_Heartbeater_Run_WhenContextCanceled_StopsPromptly(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC().Format(time.RFC3339), + "abortVerificationIds": []string{}, + }) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "tok", "agent-1", testutil.DiscardLogger()) + heartbeater := NewHeartbeater(client, config.Capacity{MaxConcurrentJobs: 1}, testutil.DiscardLogger()) + + ctx, cancel := context.WithCancel(t.Context()) + stopped := make(chan struct{}) + + go func() { + heartbeater.Run(ctx) + close(stopped) + }() + + cancel() + + select { + case <-stopped: + case <-time.After(5 * time.Second): + t.Fatal("Run did not return after context cancellation") + } +} diff --git a/agent/verification/internal/features/restore/diskexhaustion.go b/agent/verification/internal/features/restore/diskexhaustion.go new file mode 100644 index 0000000..cd61ef0 --- /dev/null +++ b/agent/verification/internal/features/restore/diskexhaustion.go @@ -0,0 +1,23 @@ +package restore + +import "strings" + +// The runner uses this to reclassify a non-zero pg_restore exit away from +// BackupRejected: exceeding an estimate-derived bound is agent infra, not +// proof the backup is corrupt. +func IsDiskExhausted(stderrTail string) bool { + lowered := strings.ToLower(stderrTail) + + for _, sig := range []string{ + "no space left on device", + "could not extend", + "disk full", + "wrote only", + } { + if strings.Contains(lowered, sig) { + return true + } + } + + return false +} diff --git a/agent/verification/internal/features/restore/diskexhaustion_test.go b/agent/verification/internal/features/restore/diskexhaustion_test.go new file mode 100644 index 0000000..b66ade1 --- /dev/null +++ b/agent/verification/internal/features/restore/diskexhaustion_test.go @@ -0,0 +1,14 @@ +package restore + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func Test_IsDiskExhausted_DetectsENOSPCSignatures(t *testing.T) { + assert.True(t, IsDiskExhausted("pg_restore: error: could not write to output file: No space left on device")) + assert.True(t, IsDiskExhausted("ERROR: could not extend file")) + assert.False(t, IsDiskExhausted("pg_restore: error: relation already exists")) + assert.False(t, IsDiskExhausted("")) +} diff --git a/agent/verification/internal/features/restore/doc.go b/agent/verification/internal/features/restore/doc.go new file mode 100644 index 0000000..192a755 --- /dev/null +++ b/agent/verification/internal/features/restore/doc.go @@ -0,0 +1,4 @@ +// Package restore drives pg_restore inside the job's ephemeral container. It +// never imports the Docker SDK: it works through the ExecRunner interface so +// the runner's exit-code contract is unit-testable with a fake. +package restore diff --git a/agent/verification/internal/features/restore/dto.go b/agent/verification/internal/features/restore/dto.go new file mode 100644 index 0000000..fe13add --- /dev/null +++ b/agent/verification/internal/features/restore/dto.go @@ -0,0 +1,16 @@ +package restore + +// A non-zero ExitCode is not a Go error here — only failing to create, start, +// or attach the exec is. +type ExecResult struct { + ExitCode int + Stdout string + Stderr string +} + +// Result is populated even on the error path (see ErrRestoreFailed). +type Result struct { + PgRestoreExitCode int + DurationMs int64 + StderrTail string +} diff --git a/agent/verification/internal/features/restore/errors.go b/agent/verification/internal/features/restore/errors.go new file mode 100644 index 0000000..d57eb48 --- /dev/null +++ b/agent/verification/internal/features/restore/errors.go @@ -0,0 +1,9 @@ +package restore + +import "errors" + +// ErrRestoreFailed is the contract boundary the runner keys on: a non-zero +// pg_restore exit is terminal "backup rejected" (modulo the disk-ceiling +// probe), whereas any other RunPgRestore error is an exec-infrastructure +// failure with no usable exit code — a retryable agent-setup failure. +var ErrRestoreFailed = errors.New("pg_restore exited non-zero") diff --git a/agent/verification/internal/features/restore/interfaces.go b/agent/verification/internal/features/restore/interfaces.go new file mode 100644 index 0000000..7ad600d --- /dev/null +++ b/agent/verification/internal/features/restore/interfaces.go @@ -0,0 +1,10 @@ +package restore + +import ( + "context" + "io" +) + +type ExecRunner interface { + Exec(ctx context.Context, cmd []string, stdin io.Reader, env []string) (ExecResult, error) +} diff --git a/agent/verification/internal/features/restore/missingextensions.go b/agent/verification/internal/features/restore/missingextensions.go new file mode 100644 index 0000000..07cbbbe --- /dev/null +++ b/agent/verification/internal/features/restore/missingextensions.go @@ -0,0 +1,112 @@ +package restore + +import ( + "regexp" + "slices" + "strconv" + "strings" +) + +const ignoredErrorsMarker = "errors ignored on restore:" + +var extensionNamePattern = regexp.MustCompile(`extension "([^"]+)"`) + +// IsMissingExtensionOnly reports whether a non-zero pg_restore exit is explained +// solely by extensions absent from this environment — i.e. the data itself +// restored. pg_restore runs without --exit-on-error, so it tails "errors ignored +// on restore: N"; this returns true only when every visible item error is +// extension-related and their count equals N. The count guard matters because +// StderrTail is capped at 8192 bytes: a truncated tail could hide a non-extension +// error, so when N can't be fully accounted for we refuse to tolerate. +func IsMissingExtensionOnly(stderrTail string) bool { + ignoredCount, hasMarker := parseIgnoredErrorCount(stderrTail) + if !hasMarker || ignoredCount < 1 { + return false + } + + itemErrors := queryErrorLines(stderrTail) + if len(itemErrors) != ignoredCount { + return false + } + + for _, line := range itemErrors { + if !isExtensionErrorLine(line) { + return false + } + } + + return true +} + +// ExtractUnavailableExtensions returns the de-duplicated, sorted names of the +// extensions pg_restore could not create, for an operator-facing log line. +func ExtractUnavailableExtensions(stderrTail string) []string { + seen := map[string]struct{}{} + + var names []string + + for _, line := range queryErrorLines(stderrTail) { + if !isExtensionErrorLine(line) { + continue + } + + for _, match := range extensionNamePattern.FindAllStringSubmatch(line, -1) { + name := match[1] + if _, isDup := seen[name]; isDup { + continue + } + + seen[name] = struct{}{} + names = append(names, name) + } + } + + slices.Sort(names) + + return names +} + +func parseIgnoredErrorCount(stderrTail string) (int, bool) { + idx := strings.LastIndex(strings.ToLower(stderrTail), ignoredErrorsMarker) + if idx < 0 { + return 0, false + } + + fields := strings.Fields(stderrTail[idx+len(ignoredErrorsMarker):]) + if len(fields) == 0 { + return 0, false + } + + count, err := strconv.Atoi(fields[0]) + if err != nil { + return 0, false + } + + return count, true +} + +func queryErrorLines(stderrTail string) []string { + var lines []string + + for line := range strings.SplitSeq(stderrTail, "\n") { + if strings.Contains(strings.ToLower(line), "could not execute query: error:") { + lines = append(lines, line) + } + } + + return lines +} + +// isExtensionErrorLine matches the phrasings a missing extension produces — the +// failed CREATE EXTENSION and its cascading COMMENT ON EXTENSION. The "extension" +// guard keeps unrelated cascades (`type "..." does not exist`) out. +func isExtensionErrorLine(line string) bool { + lowered := strings.ToLower(line) + if !strings.Contains(lowered, "extension") { + return false + } + + return strings.Contains(lowered, "is not available") || + strings.Contains(lowered, "could not open extension control file") || + strings.Contains(lowered, "does not exist") +} diff --git a/agent/verification/internal/features/restore/missingextensions_test.go b/agent/verification/internal/features/restore/missingextensions_test.go new file mode 100644 index 0000000..932cd28 --- /dev/null +++ b/agent/verification/internal/features/restore/missingextensions_test.go @@ -0,0 +1,96 @@ +package restore + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +const missingAuxExtensionsStderr = `pg_restore: error: could not execute query: ERROR: extension "pg_stat_kcache" is not available +DETAIL: Could not open extension control file "/usr/share/postgresql/16/extension/pg_stat_kcache.control": No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. +Command was: CREATE EXTENSION IF NOT EXISTS pg_stat_kcache WITH SCHEMA public; + + +pg_restore: error: could not execute query: ERROR: extension "pg_stat_kcache" does not exist +Command was: COMMENT ON EXTENSION pg_stat_kcache IS 'Kernel statistics gathering'; + + +pg_restore: error: could not execute query: ERROR: extension "set_user" is not available +DETAIL: Could not open extension control file "/usr/share/postgresql/16/extension/set_user.control": No such file or directory. +HINT: The extension must first be installed on the system where PostgreSQL is running. +Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public; + + +pg_restore: error: could not execute query: ERROR: extension "set_user" does not exist +Command was: COMMENT ON EXTENSION set_user IS 'similar to SET ROLE but with added logging'; + + +pg_restore: warning: errors ignored on restore: 4 +` + +// A missing data-bearing extension (PostGIS) cascades into type/relation errors +// that are NOT extension-related — the data did not restore, so this must not be +// tolerated. +const postgisCascadeStderr = `pg_restore: error: could not execute query: ERROR: extension "postgis" is not available +DETAIL: Could not open extension control file "/usr/share/postgresql/13/extension/postgis.control": No such file or directory. +Command was: CREATE EXTENSION IF NOT EXISTS postgis WITH SCHEMA public; + + +pg_restore: error: could not execute query: ERROR: type "public.geometry" does not exist +Command was: CREATE TABLE public.places (geom public.geometry); + + +pg_restore: error: could not execute query: ERROR: relation "public.places" does not exist +Command was: COPY public.places (geom) FROM stdin; + + +pg_restore: warning: errors ignored on restore: 3 +` + +func Test_IsMissingExtensionOnly_WhenAllIgnoredErrorsAreMissingExtensions_ReturnsTrue(t *testing.T) { + assert.True(t, IsMissingExtensionOnly(missingAuxExtensionsStderr)) +} + +func Test_IsMissingExtensionOnly_WhenCascadeDataErrorsPresent_ReturnsFalse(t *testing.T) { + assert.False(t, IsMissingExtensionOnly(postgisCascadeStderr)) +} + +func Test_IsMissingExtensionOnly_WhenNoIgnoredErrorsMarker_ReturnsFalse(t *testing.T) { + fatal := `pg_restore: error: could not execute query: ERROR: extension "set_user" is not available +Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public; +pg_restore: error: aborting because of errors` + + assert.False(t, IsMissingExtensionOnly(fatal)) +} + +func Test_IsMissingExtensionOnly_WhenVisibleErrorCountBelowMarker_ReturnsFalse(t *testing.T) { + truncated := `pg_restore: error: could not execute query: ERROR: extension "set_user" is not available +Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public; +pg_restore: error: could not execute query: ERROR: extension "set_user" does not exist +Command was: COMMENT ON EXTENSION set_user IS 'x'; +pg_restore: warning: errors ignored on restore: 5` + + assert.False(t, IsMissingExtensionOnly(truncated), + "a truncated tail that cannot account for all N errors must not be tolerated") +} + +func Test_IsMissingExtensionOnly_WhenNonExtensionErrorMixedIn_ReturnsFalse(t *testing.T) { + mixed := `pg_restore: error: could not execute query: ERROR: extension "set_user" is not available +Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public; +pg_restore: error: could not execute query: ERROR: duplicate key value violates unique constraint "places_pkey" +Command was: COPY public.places (id) FROM stdin; +pg_restore: warning: errors ignored on restore: 2` + + assert.False(t, IsMissingExtensionOnly(mixed)) +} + +func Test_IsMissingExtensionOnly_WhenEmpty_ReturnsFalse(t *testing.T) { + assert.False(t, IsMissingExtensionOnly("")) +} + +func Test_ExtractUnavailableExtensions_ReturnsSortedDedupedNames(t *testing.T) { + assert.Equal(t, + []string{"pg_stat_kcache", "set_user"}, + ExtractUnavailableExtensions(missingAuxExtensionsStderr)) +} diff --git a/agent/verification/internal/features/restore/restorer.go b/agent/verification/internal/features/restore/restorer.go new file mode 100644 index 0000000..0c66686 --- /dev/null +++ b/agent/verification/internal/features/restore/restorer.go @@ -0,0 +1,86 @@ +package restore + +import ( + "context" + "fmt" + "io" + "log/slog" + "strconv" + "time" + + "databasus-verification-agent/internal/features/dbconn" +) + +type Restorer struct { + log *slog.Logger +} + +func NewRestorer(log *slog.Logger) *Restorer { + return &Restorer{log: log} +} + +// No host bind mount: the decrypted dump never touches the agent host. +func (r *Restorer) StageBackupViaExec( + ctx context.Context, + exec ExecRunner, + body io.Reader, + destPath string, +) error { + result, err := exec.Exec(ctx, []string{"dd", "of=" + destPath, "bs=4M"}, body, nil) + if err != nil { + return fmt.Errorf("stage exec: %w", err) + } + + if result.ExitCode != 0 { + return fmt.Errorf( + "stage dd exited %d: %s", result.ExitCode, tailString(result.Stderr, 2048)) + } + + return nil +} + +func (r *Restorer) RunPgRestore( + ctx context.Context, + exec ExecRunner, + archivePath string, + conn dbconn.Conn, + parallelJobs int, +) (Result, error) { + started := time.Now().UTC() + + cmd := []string{ + "pg_restore", "-Fc", "--no-password", + "--no-owner", "--no-acl", + "-h", conn.Host, "-p", strconv.Itoa(conn.Port), + "-U", conn.User, "-d", conn.Database, + "-j", strconv.Itoa(parallelJobs), + archivePath, + } + env := []string{"PGPASSWORD=" + conn.Password} + + execResult, err := exec.Exec(ctx, cmd, nil, env) + + result := Result{ + PgRestoreExitCode: execResult.ExitCode, + DurationMs: time.Since(started).Milliseconds(), + StderrTail: tailString(execResult.Stderr, 8192), + } + + if err != nil { + return result, fmt.Errorf("pg_restore exec: %w", err) + } + + if execResult.ExitCode != 0 { + return result, fmt.Errorf("%w (code %d)", ErrRestoreFailed, execResult.ExitCode) + } + + return result, nil +} + +func tailString(s string, maxBytes int) string { + if len(s) <= maxBytes { + return s + } + + return s[len(s)-maxBytes:] +} diff --git a/agent/verification/internal/features/restore/restorer_test.go b/agent/verification/internal/features/restore/restorer_test.go new file mode 100644 index 0000000..9b8893b --- /dev/null +++ b/agent/verification/internal/features/restore/restorer_test.go @@ -0,0 +1,139 @@ +package restore + +import ( + "context" + "errors" + "io" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/testutil" +) + +type recordedExec struct { + cmd []string + env []string + stdin []byte +} + +type fakeExecRunner struct { + recorded []recordedExec + result ExecResult + err error +} + +func (f *fakeExecRunner) Exec( + _ context.Context, cmd []string, stdin io.Reader, env []string, +) (ExecResult, error) { + rec := recordedExec{cmd: cmd, env: env} + if stdin != nil { + rec.stdin, _ = io.ReadAll(stdin) + } + + f.recorded = append(f.recorded, rec) + + return f.result, f.err +} + +func testConn() dbconn.Conn { + return dbconn.Conn{ + Host: "127.0.0.1", Port: 5432, + User: "postgres", Password: "deadbeef", Database: "verifydb", + } +} + +func Test_StageBackupViaExec_WhenDdSucceeds_StreamsBodyToDest(t *testing.T) { + exec := &fakeExecRunner{result: ExecResult{ExitCode: 0}} + r := NewRestorer(testutil.DiscardLogger()) + + err := r.StageBackupViaExec( + t.Context(), exec, strings.NewReader("ARCHIVE BYTES"), "/restore/x.dump") + + require.NoError(t, err) + require.Len(t, exec.recorded, 1) + assert.Equal(t, []string{"dd", "of=/restore/x.dump", "bs=4M"}, exec.recorded[0].cmd) + assert.Equal(t, "ARCHIVE BYTES", string(exec.recorded[0].stdin)) +} + +func Test_StageBackupViaExec_WhenDdExitsNonZero_ReturnsError(t *testing.T) { + exec := &fakeExecRunner{result: ExecResult{ExitCode: 1, Stderr: "No space left on device"}} + r := NewRestorer(testutil.DiscardLogger()) + + err := r.StageBackupViaExec(t.Context(), exec, strings.NewReader("x"), "/restore/x.dump") + + require.Error(t, err) + assert.Contains(t, err.Error(), "stage dd exited 1") + assert.Contains(t, err.Error(), "No space left") +} + +func Test_StageBackupViaExec_WhenExecInfraFails_ReturnsError(t *testing.T) { + exec := &fakeExecRunner{err: errors.New("docker daemon unreachable")} + r := NewRestorer(testutil.DiscardLogger()) + + err := r.StageBackupViaExec(t.Context(), exec, strings.NewReader("x"), "/restore/x.dump") + + require.Error(t, err) + assert.Contains(t, err.Error(), "stage exec") +} + +func Test_RunPgRestore_WhenPgRestoreSucceeds_ReturnsResultWithoutError(t *testing.T) { + exec := &fakeExecRunner{result: ExecResult{ExitCode: 0}} + r := NewRestorer(testutil.DiscardLogger()) + + result, err := r.RunPgRestore(t.Context(), exec, "/restore/x.dump", testConn(), 4) + + require.NoError(t, err) + assert.Equal(t, 0, result.PgRestoreExitCode) + + require.Len(t, exec.recorded, 1) + cmd := exec.recorded[0].cmd + assert.Equal(t, "pg_restore", cmd[0]) + assert.Contains(t, cmd, "-Fc") + assert.Contains(t, cmd, "--no-password") + assert.Contains(t, cmd, "--no-owner") + assert.Contains(t, cmd, "--no-acl") + assert.Contains(t, cmd, "-j") + assert.Contains(t, cmd, "4") + assert.Contains(t, cmd, "verifydb") + assert.Equal(t, "/restore/x.dump", cmd[len(cmd)-1]) + assert.Equal(t, []string{"PGPASSWORD=deadbeef"}, exec.recorded[0].env) +} + +func Test_RunPgRestore_WhenPgRestoreExitsNonZero_ReturnsErrRestoreFailedWithPopulatedResult(t *testing.T) { + exec := &fakeExecRunner{result: ExecResult{ExitCode: 1, Stderr: "could not execute query"}} + r := NewRestorer(testutil.DiscardLogger()) + + result, err := r.RunPgRestore(t.Context(), exec, "/restore/x.dump", testConn(), 2) + + require.Error(t, err) + assert.ErrorIs(t, err, ErrRestoreFailed) + assert.Equal(t, 1, result.PgRestoreExitCode) + assert.Contains(t, result.StderrTail, "could not execute query") +} + +func Test_RunPgRestore_WhenExecInfraFails_ReturnsNonRestoreError(t *testing.T) { + exec := &fakeExecRunner{err: errors.New("exec create failed")} + r := NewRestorer(testutil.DiscardLogger()) + + _, err := r.RunPgRestore(t.Context(), exec, "/restore/x.dump", testConn(), 1) + + require.Error(t, err) + assert.NotErrorIs(t, err, ErrRestoreFailed, + "an exec-infrastructure failure must not look like a non-zero pg_restore exit") +} + +func Test_RunPgRestore_WhenStderrHuge_TailIsTruncatedSuffix(t *testing.T) { + huge := strings.Repeat("A", 9000) + "TAIL-MARKER" + exec := &fakeExecRunner{result: ExecResult{ExitCode: 0, Stderr: huge}} + r := NewRestorer(testutil.DiscardLogger()) + + result, err := r.RunPgRestore(t.Context(), exec, "/restore/x.dump", testConn(), 1) + + require.NoError(t, err) + assert.Len(t, result.StderrTail, 8192) + assert.True(t, strings.HasSuffix(result.StderrTail, "TAIL-MARKER")) +} diff --git a/agent/verification/internal/features/restore/timescaledb.go b/agent/verification/internal/features/restore/timescaledb.go new file mode 100644 index 0000000..4a670bb --- /dev/null +++ b/agent/verification/internal/features/restore/timescaledb.go @@ -0,0 +1,49 @@ +package restore + +import ( + "context" + "fmt" + + "github.com/jackc/pgx/v5" + + "databasus-verification-agent/internal/features/dbconn" +) + +// TimescaleDB restore procedure run against the throwaway restore target. +// +// timescaledb_pre_restore() sets a database-level GUC (ALTER DATABASE ... SET +// timescaledb.restoring='on') that the separate in-container pg_restore connection inherits, and +// stops background workers. timescaledb_post_restore() resets it and restarts the workers, returning +// the database to normal mode — it must run before the verifier collects stats. The connection used +// here is the agent-host verifier conn (published port); pg_restore uses the in-container conn, but +// both reach the same database, so the GUC carries across. + +// RunTimescalePreRestore installs the extension (a freshly initialised target may lack it) and +// enters restoring mode. +func (r *Restorer) RunTimescalePreRestore(ctx context.Context, conn dbconn.Conn) error { + return execStatements(ctx, conn, + "CREATE EXTENSION IF NOT EXISTS timescaledb", + "SELECT timescaledb_pre_restore()", + ) +} + +// RunTimescalePostRestore leaves restoring mode and restarts background workers. +func (r *Restorer) RunTimescalePostRestore(ctx context.Context, conn dbconn.Conn) error { + return execStatements(ctx, conn, "SELECT timescaledb_post_restore()") +} + +func execStatements(ctx context.Context, conn dbconn.Conn, statements ...string) error { + pgConn, err := pgx.Connect(ctx, conn.DSN()) + if err != nil { + return fmt.Errorf("connect restore target: %w", err) + } + defer func() { _ = pgConn.Close(ctx) }() + + for _, statement := range statements { + if _, err := pgConn.Exec(ctx, statement); err != nil { + return fmt.Errorf("run %q: %w", statement, err) + } + } + + return nil +} diff --git a/agent/verification/internal/features/runner/abort_midrestore_integration_test.go b/agent/verification/internal/features/runner/abort_midrestore_integration_test.go new file mode 100644 index 0000000..295c1be --- /dev/null +++ b/agent/verification/internal/features/runner/abort_midrestore_integration_test.go @@ -0,0 +1,131 @@ +package runner + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "slices" + "sync/atomic" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/heartbeat" + "databasus-verification-agent/internal/testutil" +) + +type abortMidRestoreBackend struct { + abortID uuid.UUID + reportHits atomic.Int32 +} + +func newAbortMidRestoreBackend(abortID uuid.UUID) (*abortMidRestoreBackend, *httptest.Server) { + backend := &abortMidRestoreBackend{abortID: abortID} + + mux := http.NewServeMux() + + mux.HandleFunc( + "GET /api/v1/agent/verifications/{agentId}/{id}/backup-stream", + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ARCHIVE")) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verifications/{agentId}/{id}/report", + func(w http.ResponseWriter, _ *http.Request) { + backend.reportHits.Add(1) + + w.WriteHeader(http.StatusOK) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verification/{agentId}/heartbeat", + func(w http.ResponseWriter, req *http.Request) { + var body struct { + CurrentVerificationIDs []uuid.UUID `json:"currentVerificationIds"` + } + _ = json.NewDecoder(req.Body).Decode(&body) + + abortIDs := []uuid.UUID{} + if slices.Contains(body.CurrentVerificationIDs, backend.abortID) { + abortIDs = []uuid.UUID{backend.abortID} + } + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC(), + "abortVerificationIds": abortIDs, + }) + }, + ) + + return backend, httptest.NewServer(mux) +} + +func Test_ExecuteJob_WhenServerAbortsMidRestore_HardStopsRestoreCleansContainerAndDoesNotReport( + t *testing.T, +) { + job := postgresJob() + + backend, server := newAbortMidRestoreBackend(job.VerificationID) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "", uuid.NewString(), testutil.DiscardLogger()) + heartbeater := heartbeat.NewHeartbeater(client, testCapacity(), testutil.DiscardLogger()) + + jobContainer := fakeContainerWith(testConn()) + restorer := &fakeRestorer{runBlocks: true} + + runnerUnderTest := NewRunner( + client, testCapacity(), NewPool(2), + &fakeSpawner{container: jobContainer}, + restorer, + &fakeStats{}, + heartbeater, + testutil.DiscardLogger(), + ) + + execDone := make(chan struct{}) + go func() { + runnerUnderTest.executeJob(t.Context(), job) + close(execDone) + }() + + require.Eventually(t, func() bool { return restorer.runEntered.Load() }, + 2*time.Second, 5*time.Millisecond, + "executeJob must reach pg_restore before the server delivers the abort") + + beatCtx, beatCancel := context.WithCancel(t.Context()) + beatDone := make(chan struct{}) + go func() { + heartbeater.Run(beatCtx) + close(beatDone) + }() + + select { + case <-execDone: + case <-time.After(5 * time.Second): + t.Fatal("executeJob did not return after the server abort; restore was not hard-stopped") + } + + beatCancel() + <-beatDone + + assert.True(t, restorer.runCtxCancelled.Load(), + "pg_restore must be hard-stopped by jobCtx cancellation, not run to completion") + assert.True(t, jobContainer.terminated, + "the container and its anonymous PGDATA volume must be torn down on a server abort, "+ + "even though the same cancellation triggered the teardown") + assert.Equal(t, int32(0), backend.reportHits.Load(), + "a server-aborted verification must be dropped silently with no FAILED report") + assert.True(t, heartbeater.IsAborted(job.VerificationID), + "the abort set must be recorded for the runner's pre-FAILED re-check") +} diff --git a/agent/verification/internal/features/runner/concurrent_restores_integration_test.go b/agent/verification/internal/features/runner/concurrent_restores_integration_test.go new file mode 100644 index 0000000..8a15440 --- /dev/null +++ b/agent/verification/internal/features/runner/concurrent_restores_integration_test.go @@ -0,0 +1,275 @@ +package runner + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/heartbeat" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" + "databasus-verification-agent/internal/testutil" +) + +type concurrentRestoreBackend struct { + claims []*api.JobAssignment + claimIdx atomic.Int32 + + mu sync.Mutex + reports map[uuid.UUID]api.VerificationStatus +} + +func newConcurrentRestoreBackend(claims []*api.JobAssignment) (*concurrentRestoreBackend, *httptest.Server) { + backend := &concurrentRestoreBackend{ + claims: claims, + reports: make(map[uuid.UUID]api.VerificationStatus), + } + + mux := http.NewServeMux() + + mux.HandleFunc( + "POST /api/v1/agent/verifications/{agentId}/claim", + func(w http.ResponseWriter, _ *http.Request) { + i := int(backend.claimIdx.Add(1)) - 1 + if i >= len(backend.claims) { + w.WriteHeader(http.StatusNoContent) + return + } + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(backend.claims[i]) + }, + ) + + mux.HandleFunc( + "GET /api/v1/agent/verifications/{agentId}/{id}/backup-stream", + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ARCHIVE")) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verifications/{agentId}/{id}/report", + func(w http.ResponseWriter, req *http.Request) { + verificationID, err := uuid.Parse(req.PathValue("id")) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + var body struct { + Status api.VerificationStatus `json:"status"` + } + _ = json.NewDecoder(req.Body).Decode(&body) + + backend.mu.Lock() + backend.reports[verificationID] = body.Status + backend.mu.Unlock() + + w.WriteHeader(http.StatusNoContent) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verification/{agentId}/heartbeat", + func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC(), + "abortVerificationIds": []uuid.UUID{}, + }) + }, + ) + + return backend, httptest.NewServer(mux) +} + +func (b *concurrentRestoreBackend) reportsSnapshot() map[uuid.UUID]api.VerificationStatus { + b.mu.Lock() + defer b.mu.Unlock() + + out := make(map[uuid.UUID]api.VerificationStatus, len(b.reports)) + for id, status := range b.reports { + out[id] = status + } + + return out +} + +type recordingContainer struct { + id int + inContainerConn dbconn.Conn + verifierConn dbconn.Conn + terminated atomic.Bool +} + +func (c *recordingContainer) Exec( + _ context.Context, _ []string, stdin io.Reader, _ []string, +) (restore.ExecResult, error) { + if stdin != nil { + _, _ = io.Copy(io.Discard, stdin) + } + + return restore.ExecResult{}, nil +} + +func (c *recordingContainer) GetInContainerConn() dbconn.Conn { return c.inContainerConn } +func (c *recordingContainer) GetVerifierConn() dbconn.Conn { return c.verifierConn } + +func (c *recordingContainer) GetDiskUsageBytes(context.Context) (int64, error) { + return 0, nil +} + +func (c *recordingContainer) Terminate(context.Context) error { + c.terminated.Store(true) + return nil +} + +type recordingSpawner struct { + nextID atomic.Int32 + + mu sync.Mutex + containers []*recordingContainer +} + +func (s *recordingSpawner) Spawn(_ context.Context, _ SpawnRequest) (JobContainer, error) { + c := &recordingContainer{ + id: int(s.nextID.Add(1)), + inContainerConn: testConn(), + verifierConn: testConn(), + } + + s.mu.Lock() + s.containers = append(s.containers, c) + s.mu.Unlock() + + return c, nil +} + +func (s *recordingSpawner) snapshot() []*recordingContainer { + s.mu.Lock() + defer s.mu.Unlock() + + out := make([]*recordingContainer, len(s.containers)) + copy(out, s.containers) + + return out +} + +type barrierRestorer struct { + barrier *sync.WaitGroup +} + +func (r *barrierRestorer) StageBackupViaExec( + _ context.Context, _ restore.ExecRunner, body io.Reader, _ string, +) error { + if body != nil { + _, _ = io.Copy(io.Discard, body) + } + + return nil +} + +func (r *barrierRestorer) RunPgRestore( + _ context.Context, _ restore.ExecRunner, _ string, _ dbconn.Conn, _ int, +) (restore.Result, error) { + r.barrier.Done() + r.barrier.Wait() + + return restore.Result{PgRestoreExitCode: 0, DurationMs: 1}, nil +} + +func (r *barrierRestorer) RunTimescalePreRestore(context.Context, dbconn.Conn) error { return nil } +func (r *barrierRestorer) RunTimescalePostRestore(context.Context, dbconn.Conn) error { return nil } + +func makeConcurrentAssignment(id uuid.UUID) *api.JobAssignment { + return &api.JobAssignment{ + VerificationID: id, + BackupID: uuid.New(), + BackupSizeMb: 1, + MaxContainerDiskMb: 0, + Database: api.AssignedDatabase{ + Type: "POSTGRES_LOGICAL", + PostgresqlLogical: &api.AssignedPostgresql{Version: "16"}, + }, + } +} + +func Test_Run_WhenTwoVerificationsClaimed_EachGetsIsolatedContainerAndBothComplete( + t *testing.T, +) { + id1, id2 := uuid.New(), uuid.New() + claims := []*api.JobAssignment{ + makeConcurrentAssignment(id1), + makeConcurrentAssignment(id2), + } + + backend, server := newConcurrentRestoreBackend(claims) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "", uuid.NewString(), testutil.DiscardLogger()) + heartbeater := heartbeat.NewHeartbeater(client, testCapacity(), testutil.DiscardLogger()) + + spawner := &recordingSpawner{} + barrier := &sync.WaitGroup{} + barrier.Add(2) + + runnerUnderTest := NewRunner( + client, testCapacity(), NewPool(2), + spawner, + &barrierRestorer{barrier: barrier}, + &fakeStats{stats: verifier.Stats{DBSizeBytes: 1, SchemaCount: 1, TableCount: 1}}, + heartbeater, + testutil.DiscardLogger(), + ) + runnerUnderTest.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + runCtx, runCancel := context.WithCancel(t.Context()) + runDone := make(chan struct{}) + go func() { + runnerUnderTest.Run(runCtx) + close(runDone) + }() + + require.Eventually(t, func() bool { + snapshot := backend.reportsSnapshot() + return snapshot[id1] == api.VerificationStatusCompleted && + snapshot[id2] == api.VerificationStatusCompleted + }, 10*time.Second, 20*time.Millisecond, + "both verifications must complete via the real claim loop + NewPool(2) fan-out; "+ + "the barrier in pg_restore would deadlock unless both jobs were truly in flight at once") + + runCancel() + select { + case <-runDone: + case <-time.After(5 * time.Second): + t.Fatal("Run did not return after context cancellation") + } + + spawned := spawner.snapshot() + require.Len(t, spawned, 2, + "each claimed verification must spawn its own container — proves isolation, not reuse") + assert.NotEqual(t, spawned[0].id, spawned[1].id, + "the two spawned containers must be distinct instances") + for _, c := range spawned { + assert.True(t, c.terminated.Load(), + "every spawned container must be torn down (deferred Terminate in executeJob)") + } + + reports := backend.reportsSnapshot() + assert.Equal(t, api.VerificationStatusCompleted, reports[id1]) + assert.Equal(t, api.VerificationStatusCompleted, reports[id2]) +} diff --git a/agent/verification/internal/features/runner/diskwatch.go b/agent/verification/internal/features/runner/diskwatch.go new file mode 100644 index 0000000..0fd5f12 --- /dev/null +++ b/agent/verification/internal/features/runner/diskwatch.go @@ -0,0 +1,60 @@ +package runner + +import ( + "context" + "fmt" + "log/slog" + "time" +) + +var diskWatchInterval = 3 * time.Second + +// diskWatcher polls the job's written bytes and fires onLimitExceeded once when +// the footprint reaches the server-computed ceiling. It never trips on a probe +// error (a transient daemon hiccup must not fail a healthy restore). The caller +// owns the reaction: onLimitExceeded records the trip and cancels the job. +type diskWatcher struct { + probe diskUsageProber + ceilingBytes int64 + interval time.Duration + onLimitExceeded func() + log *slog.Logger +} + +func newDiskWatcher( + probe diskUsageProber, ceilingBytes int64, onLimitExceeded func(), log *slog.Logger, +) *diskWatcher { + return &diskWatcher{ + probe: probe, + ceilingBytes: ceilingBytes, + interval: diskWatchInterval, + onLimitExceeded: onLimitExceeded, + log: log, + } +} + +func (w *diskWatcher) run(ctx context.Context) { + ticker := time.NewTicker(w.interval) + defer ticker.Stop() + + for { + used, err := w.probe.GetDiskUsageBytes(ctx) + switch { + case err != nil: + w.log.Debug("disk watch probe failed, will retry next tick", "error", err) + case used >= w.ceilingBytes: + w.log.Warn(fmt.Sprintf( + "job disk usage %d bytes reached ceiling %d bytes; aborting restore", + used, w.ceilingBytes)) + w.onLimitExceeded() + + return + } + + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } +} diff --git a/agent/verification/internal/features/runner/diskwatch_test.go b/agent/verification/internal/features/runner/diskwatch_test.go new file mode 100644 index 0000000..d7f700b --- /dev/null +++ b/agent/verification/internal/features/runner/diskwatch_test.go @@ -0,0 +1,139 @@ +package runner + +import ( + "context" + "errors" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/testutil" +) + +type fakeDiskProbe struct { + mu sync.Mutex + values []int64 + errs []error + calls int +} + +func (p *fakeDiskProbe) GetDiskUsageBytes(context.Context) (int64, error) { + p.mu.Lock() + defer p.mu.Unlock() + + i := p.calls + if i >= len(p.values) { + i = len(p.values) - 1 + } + + p.calls++ + + var err error + if p.calls-1 < len(p.errs) { + err = p.errs[p.calls-1] + } + + return p.values[i], err +} + +func newTestDiskWatcher(probe diskUsageProber, ceilingBytes int64, onLimitExceeded func()) *diskWatcher { + w := newDiskWatcher(probe, ceilingBytes, onLimitExceeded, testutil.DiscardLogger()) + w.interval = time.Millisecond + + return w +} + +func Test_DiskWatcher_WhenUsageReachesCeiling_FiresOnce(t *testing.T) { + var fired atomic.Int32 + done := make(chan struct{}) + + w := newTestDiskWatcher( + &fakeDiskProbe{values: []int64{100}}, 100, + func() { fired.Add(1); close(done) }, + ) + + go w.run(t.Context()) + + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("watcher never fired despite usage at ceiling") + } + + time.Sleep(20 * time.Millisecond) + assert.Equal(t, int32(1), fired.Load(), "onLimitExceeded must fire exactly once") +} + +func Test_DiskWatcher_WhenUsageBelowCeiling_NeverFires(t *testing.T) { + var fired atomic.Int32 + ctx, cancel := context.WithCancel(t.Context()) + + w := newTestDiskWatcher( + &fakeDiskProbe{values: []int64{50}}, 100, + func() { fired.Add(1) }, + ) + + stopped := make(chan struct{}) + go func() { w.run(ctx); close(stopped) }() + + time.Sleep(30 * time.Millisecond) + cancel() + + select { + case <-stopped: + case <-time.After(2 * time.Second): + t.Fatal("watcher did not stop on context cancel") + } + + assert.Equal(t, int32(0), fired.Load(), "must not fire while under the ceiling") +} + +func Test_DiskWatcher_WhenProbeErrorsThenRecovers_DoesNotTripOnError(t *testing.T) { + var fired atomic.Int32 + done := make(chan struct{}) + + probe := &fakeDiskProbe{ + values: []int64{0, 0, 200}, + errs: []error{errors.New("daemon busy"), errors.New("daemon busy"), nil}, + } + + w := newTestDiskWatcher(probe, 100, func() { fired.Add(1); close(done) }) + + go w.run(t.Context()) + + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("watcher never fired after probe recovered above ceiling") + } + + assert.Equal(t, int32(1), fired.Load(), + "a probe error must not trip the watcher nor stop the loop") +} + +func Test_DiskWatcher_WhenContextCancelled_StopsWithoutFiring(t *testing.T) { + var fired atomic.Int32 + ctx, cancel := context.WithCancel(t.Context()) + + w := newTestDiskWatcher( + &fakeDiskProbe{values: []int64{10}}, 100, + func() { fired.Add(1) }, + ) + + stopped := make(chan struct{}) + go func() { w.run(ctx); close(stopped) }() + + cancel() + + select { + case <-stopped: + case <-time.After(2 * time.Second): + t.Fatal("watcher did not return after context cancel") + } + + require.Equal(t, int32(0), fired.Load()) +} diff --git a/agent/verification/internal/features/runner/fakes_test.go b/agent/verification/internal/features/runner/fakes_test.go new file mode 100644 index 0000000..95b67c7 --- /dev/null +++ b/agent/verification/internal/features/runner/fakes_test.go @@ -0,0 +1,223 @@ +package runner + +import ( + "bytes" + "context" + "io" + "sync" + "sync/atomic" + + "github.com/google/uuid" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" +) + +type fakeAPI struct { + mu sync.Mutex + + claims []*api.JobAssignment + claimErrs []error + claimIdx int + + downloadBody []byte + downloadErr error + + reportErr error + reportedReqs []api.ReportRequest + reportedIDs []uuid.UUID + reportCalled int +} + +func (f *fakeAPI) ClaimVerification( + _ context.Context, _ api.AgentCapacity, +) (*api.JobAssignment, error) { + f.mu.Lock() + defer f.mu.Unlock() + + i := f.claimIdx + f.claimIdx++ + + if i < len(f.claimErrs) && f.claimErrs[i] != nil { + return nil, f.claimErrs[i] + } + + if i < len(f.claims) { + return f.claims[i], nil + } + + return nil, nil +} + +func (f *fakeAPI) DownloadBackup( + _ context.Context, _ uuid.UUID, +) (io.ReadCloser, error) { + if f.downloadErr != nil { + return nil, f.downloadErr + } + + return io.NopCloser(bytes.NewReader(f.downloadBody)), nil +} + +func (f *fakeAPI) Report( + _ context.Context, verificationID uuid.UUID, req api.ReportRequest, +) error { + f.mu.Lock() + defer f.mu.Unlock() + + f.reportCalled++ + f.reportedReqs = append(f.reportedReqs, req) + f.reportedIDs = append(f.reportedIDs, verificationID) + + return f.reportErr +} + +func (f *fakeAPI) lastReport() (api.ReportRequest, bool) { + f.mu.Lock() + defer f.mu.Unlock() + + if len(f.reportedReqs) == 0 { + return api.ReportRequest{}, false + } + + return f.reportedReqs[len(f.reportedReqs)-1], true +} + +type fakeContainer struct { + inContainerConn dbconn.Conn + verifierConn dbconn.Conn + terminated bool + diskUsageBytes int64 + diskUsageErr error +} + +func (c *fakeContainer) Exec( + _ context.Context, _ []string, stdin io.Reader, _ []string, +) (restore.ExecResult, error) { + if stdin != nil { + _, _ = io.Copy(io.Discard, stdin) + } + + return restore.ExecResult{}, nil +} + +func (c *fakeContainer) GetInContainerConn() dbconn.Conn { return c.inContainerConn } +func (c *fakeContainer) GetVerifierConn() dbconn.Conn { return c.verifierConn } + +func (c *fakeContainer) GetDiskUsageBytes(context.Context) (int64, error) { + return c.diskUsageBytes, c.diskUsageErr +} + +func (c *fakeContainer) Terminate(context.Context) error { + c.terminated = true + return nil +} + +type fakeSpawner struct { + container JobContainer + err error +} + +func (s *fakeSpawner) Spawn(_ context.Context, _ SpawnRequest) (JobContainer, error) { + if s.err != nil { + return nil, s.err + } + + return s.container, nil +} + +type fakeRestorer struct { + stageErr error + runResult restore.Result + runErr error + runBlocks bool + + runEntered atomic.Bool + runCtxCancelled atomic.Bool + + // calls records the restore-related call order ("pre", "restore", "post") for a single job. + calls []string + // runParallelJobs records the -j value RunPgRestore was invoked with. + runParallelJobs int +} + +func (r *fakeRestorer) StageBackupViaExec( + _ context.Context, _ restore.ExecRunner, body io.Reader, _ string, +) error { + if body != nil { + _, _ = io.Copy(io.Discard, body) + } + + return r.stageErr +} + +func (r *fakeRestorer) RunPgRestore( + ctx context.Context, _ restore.ExecRunner, _ string, _ dbconn.Conn, parallelJobs int, +) (restore.Result, error) { + r.calls = append(r.calls, "restore") + r.runParallelJobs = parallelJobs + + if r.runBlocks { + r.runEntered.Store(true) + <-ctx.Done() + r.runCtxCancelled.Store(true) + + return restore.Result{}, ctx.Err() + } + + return r.runResult, r.runErr +} + +func (r *fakeRestorer) RunTimescalePreRestore(context.Context, dbconn.Conn) error { + r.calls = append(r.calls, "pre") + + return nil +} + +func (r *fakeRestorer) RunTimescalePostRestore(context.Context, dbconn.Conn) error { + r.calls = append(r.calls, "post") + + return nil +} + +type fakeStats struct { + stats verifier.Stats + err error +} + +func (s *fakeStats) CollectStats(context.Context, dbconn.Conn) (verifier.Stats, error) { + return s.stats, s.err +} + +type fakeRegistrar struct { + mu sync.Mutex + registered map[uuid.UUID]struct{} + aborted bool +} + +func newFakeRegistrar() *fakeRegistrar { + return &fakeRegistrar{registered: make(map[uuid.UUID]struct{})} +} + +func (r *fakeRegistrar) TrackVerification(id uuid.UUID, _ context.CancelFunc) { + r.mu.Lock() + defer r.mu.Unlock() + + r.registered[id] = struct{}{} +} + +func (r *fakeRegistrar) UntrackVerification(id uuid.UUID) { + r.mu.Lock() + defer r.mu.Unlock() + + delete(r.registered, id) +} + +func (r *fakeRegistrar) IsAborted(uuid.UUID) bool { + r.mu.Lock() + defer r.mu.Unlock() + + return r.aborted +} diff --git a/agent/verification/internal/features/runner/interfaces.go b/agent/verification/internal/features/runner/interfaces.go new file mode 100644 index 0000000..f78de08 --- /dev/null +++ b/agent/verification/internal/features/runner/interfaces.go @@ -0,0 +1,61 @@ +package runner + +import ( + "context" + "io" + + "github.com/google/uuid" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" +) + +type JobContainer interface { + Exec(ctx context.Context, cmd []string, stdin io.Reader, env []string) (restore.ExecResult, error) + GetInContainerConn() dbconn.Conn + GetVerifierConn() dbconn.Conn + GetDiskUsageBytes(ctx context.Context) (int64, error) + Terminate(ctx context.Context) error +} + +type Spawner interface { + Spawn(ctx context.Context, req SpawnRequest) (JobContainer, error) +} + +type APIClient interface { + ClaimVerification(ctx context.Context, capacity api.AgentCapacity) (*api.JobAssignment, error) + DownloadBackup(ctx context.Context, verificationID uuid.UUID) (io.ReadCloser, error) + Report(ctx context.Context, verificationID uuid.UUID, req api.ReportRequest) error +} + +type Restorer interface { + StageBackupViaExec(ctx context.Context, exec restore.ExecRunner, body io.Reader, destPath string) error + RunPgRestore( + ctx context.Context, + exec restore.ExecRunner, + archivePath string, + conn dbconn.Conn, + parallelJobs int, + ) (restore.Result, error) + RunTimescalePreRestore(ctx context.Context, conn dbconn.Conn) error + RunTimescalePostRestore(ctx context.Context, conn dbconn.Conn) error +} + +type StatsCollector interface { + CollectStats(ctx context.Context, conn dbconn.Conn) (verifier.Stats, error) +} + +// Registrar is the heartbeat registry seam: register before any container +// exists so the ID rides every heartbeat, and re-check the recorded abort set +// before any FAILED POST. +type Registrar interface { + TrackVerification(id uuid.UUID, cancel context.CancelFunc) + UntrackVerification(id uuid.UUID) + IsAborted(id uuid.UUID) bool +} + +type diskUsageProber interface { + GetDiskUsageBytes(ctx context.Context) (int64, error) +} diff --git a/agent/verification/internal/features/runner/pool.go b/agent/verification/internal/features/runner/pool.go new file mode 100644 index 0000000..8dfedd4 --- /dev/null +++ b/agent/verification/internal/features/runner/pool.go @@ -0,0 +1,45 @@ +package runner + +import "sync" + +// Pool is a plain bounded-concurrency semaphore. There is deliberately no +// disk math here — the server is the sole disk-admission authority; the pool +// only caps how many verifications run at once. +type Pool struct { + slots chan struct{} + wg sync.WaitGroup +} + +func NewPool(maxConcurrentJobs int) *Pool { + return &Pool{slots: make(chan struct{}, maxConcurrentJobs)} +} + +func (p *Pool) Saturated() bool { + return len(p.slots) == cap(p.slots) +} + +// Go acquires a slot and runs fn in a goroutine, releasing the slot when fn +// returns. The runner loop is the only producer and only calls Go after +// Saturated() == false, so the send never blocks in practice. +// +// There is deliberately no recover() here. Container freshness has no periodic +// reaper — it relies on the invariant that any lost container is followed by a +// process restart (then the startup purge cleans up). A panicking job must +// therefore crash the process. Adding recovery would let a long-lived process +// leak a container with no safety net; restore a sweep before doing so. +func (p *Pool) Go(fn func()) { + p.slots <- struct{}{} + + p.wg.Go(func() { + defer func() { <-p.slots }() + + fn() + }) +} + +// Wait blocks until every in-flight job has returned. The runner calls it on +// shutdown so deferred container teardown runs before the process exits or +// re-execs on self-update. +func (p *Pool) Wait() { + p.wg.Wait() +} diff --git a/agent/verification/internal/features/runner/pool_test.go b/agent/verification/internal/features/runner/pool_test.go new file mode 100644 index 0000000..f84aa22 --- /dev/null +++ b/agent/verification/internal/features/runner/pool_test.go @@ -0,0 +1,59 @@ +package runner + +import ( + "sync" + "sync/atomic" + "testing" + + "github.com/stretchr/testify/assert" +) + +func Test_Pool_Saturated_ReflectsSlotOccupancyAsJobsStartAndFinish(t *testing.T) { + pool := NewPool(2) + assert.False(t, pool.Saturated()) + + release := make(chan struct{}) + var started sync.WaitGroup + started.Add(2) + + for range 2 { + pool.Go(func() { + started.Done() + <-release + }) + } + + started.Wait() + assert.True(t, pool.Saturated(), "pool must report saturated at MaxConcurrentJobs") + + close(release) + pool.Wait() + assert.False(t, pool.Saturated(), "slots must be released when jobs finish") +} + +func Test_Pool_WhenFloodedWithJobs_NeverExceedsCapAndWaitDrainsAll(t *testing.T) { + const capLimit = 4 + pool := NewPool(capLimit) + + var concurrent, maxConcurrent atomic.Int32 + + for range 100 { + pool.Go(func() { + cur := concurrent.Add(1) + for { + observed := maxConcurrent.Load() + if cur <= observed || maxConcurrent.CompareAndSwap(observed, cur) { + break + } + } + + concurrent.Add(-1) + }) + } + + pool.Wait() + + assert.LessOrEqual(t, int(maxConcurrent.Load()), capLimit, + "concurrency must never exceed the slot count") + assert.Equal(t, int32(0), concurrent.Load(), "Wait must drain every job") +} diff --git a/agent/verification/internal/features/runner/report_giveup_integration_test.go b/agent/verification/internal/features/runner/report_giveup_integration_test.go new file mode 100644 index 0000000..5c4aca7 --- /dev/null +++ b/agent/verification/internal/features/runner/report_giveup_integration_test.go @@ -0,0 +1,155 @@ +package runner + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/heartbeat" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" + "databasus-verification-agent/internal/testutil" +) + +type giveupMockBackend struct { + mu sync.Mutex + heartbeatCount int + lastHeartbeatIDs []uuid.UUID + + reportHits atomic.Int32 + secondReport chan struct{} + signalSecondOnce sync.Once +} + +func newGiveupMockBackend() (*giveupMockBackend, *httptest.Server) { + backend := &giveupMockBackend{secondReport: make(chan struct{})} + + mux := http.NewServeMux() + + mux.HandleFunc( + "GET /api/v1/agent/verifications/{agentId}/{id}/backup-stream", + func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ARCHIVE")) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verifications/{agentId}/{id}/report", + func(w http.ResponseWriter, _ *http.Request) { + if backend.reportHits.Add(1) == 2 { + backend.signalSecondOnce.Do(func() { close(backend.secondReport) }) + } + + w.WriteHeader(http.StatusServiceUnavailable) + }, + ) + + mux.HandleFunc( + "POST /api/v1/agent/verification/{agentId}/heartbeat", + func(w http.ResponseWriter, req *http.Request) { + var body struct { + CurrentVerificationIDs []uuid.UUID `json:"currentVerificationIds"` + } + _ = json.NewDecoder(req.Body).Decode(&body) + + backend.mu.Lock() + backend.heartbeatCount++ + backend.lastHeartbeatIDs = body.CurrentVerificationIDs + backend.mu.Unlock() + + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "lastSeenAt": time.Now().UTC(), + "abortVerificationIds": []uuid.UUID{}, + }) + }, + ) + + return backend, httptest.NewServer(mux) +} + +func (b *giveupMockBackend) heartbeatsSeen() int { + b.mu.Lock() + defer b.mu.Unlock() + + return b.heartbeatCount +} + +func (b *giveupMockBackend) lastHeartbeatVerificationIDs() []uuid.UUID { + b.mu.Lock() + defer b.mu.Unlock() + + return b.lastHeartbeatIDs +} + +func Test_ExecuteJob_WhenReportGivenUp_UnregistersAndHeartbeatOmitsJob(t *testing.T) { + backend, server := newGiveupMockBackend() + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "", uuid.NewString(), testutil.DiscardLogger()) + heartbeater := heartbeat.NewHeartbeater(client, testCapacity(), testutil.DiscardLogger()) + + runnerUnderTest := NewRunner( + client, testCapacity(), NewPool(2), + okSpawner(), + &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0}}, + &fakeStats{stats: verifier.Stats{ + DBSizeBytes: 9_000_000, + SchemaCount: 2, + TableCount: 3, + TableStats: []verifier.TableStat{{SchemaName: "public", Name: "t1", RowCount: 10}}, + }}, + heartbeater, + testutil.DiscardLogger(), + ) + runnerUnderTest.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + job := postgresJob() + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + go func() { + select { + case <-backend.secondReport: + cancel() + case <-ctx.Done(): + } + }() + + runnerUnderTest.executeJob(ctx, job) + + assert.GreaterOrEqual(t, backend.reportHits.Load(), int32(2), + "the real report retry loop must have actually retried against the unresponsive server") + assert.NotContains(t, heartbeater.GetRunningVerificationIDs(), job.VerificationID, + "report give-up must untrack the job from the real heartbeat registry") + + beatCtx, beatCancel := context.WithCancel(t.Context()) + beatDone := make(chan struct{}) + go func() { + heartbeater.Run(beatCtx) + close(beatDone) + }() + + require.Eventually(t, func() bool { return backend.heartbeatsSeen() > 0 }, + 2*time.Second, 10*time.Millisecond, + "the real Heartbeater must send at least one heartbeat") + + beatCancel() + <-beatDone + + assert.NotContains(t, backend.lastHeartbeatVerificationIDs(), job.VerificationID, + "after give-up the real heartbeat wire must no longer carry the dropped job") +} diff --git a/agent/verification/internal/features/runner/runner.go b/agent/verification/internal/features/runner/runner.go new file mode 100644 index 0000000..7e3ab3e --- /dev/null +++ b/agent/verification/internal/features/runner/runner.go @@ -0,0 +1,583 @@ +package runner + +import ( + "context" + "errors" + "fmt" + "log/slog" + "slices" + "strings" + "sync/atomic" + "time" + + "github.com/google/uuid" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" +) + +const ( + jobName = "verification_runner" + + claimErrorBackoff = 5 * time.Second + idleClaimBackoff = 15 * time.Second + maxBackupDownloadAttempts = 5 + streamIdleTimeout = 60 * time.Second + maxBackupDownloadBackoff = 32 * time.Second + + // archive layout inside the per-job container's writable volume. + inContainerArchiveDir = "/tmp" + + maxParallelRestoreJobs = 8 +) + +var supportedMajors = []string{"12", "13", "14", "15", "16", "17", "18"} + +// errAbortNoReport signals the job was aborted (jobCtx cancelled, or the +// stream returned 410) — executeJob returns without POSTing a report. +var errAbortNoReport = errors.New("verification aborted; no report") + +// backupDownloadBackoffFn is the download retry delay; tests swap it to avoid real sleeps. +var backupDownloadBackoffFn = backupDownloadBackoff + +type SpawnRequest struct { + PgMajor string + CPUPerJob int + RAMMbPerJob int + VerificationID uuid.UUID + TimescaledbVersion string +} + +type Runner struct { + api APIClient + capacity config.Capacity + pool *Pool + spawner Spawner + restorer Restorer + verifier StatsCollector + heartbeat Registrar + connAlive func(ctx context.Context, conn dbconn.Conn) bool + hasRun atomic.Bool + log *slog.Logger +} + +func NewRunner( + apiClient APIClient, + capacity config.Capacity, + pool *Pool, + spawner Spawner, + restorer Restorer, + statsCollector StatsCollector, + heartbeat Registrar, + log *slog.Logger, +) *Runner { + return &Runner{ + api: apiClient, + capacity: capacity, + pool: pool, + spawner: spawner, + restorer: restorer, + verifier: statsCollector, + heartbeat: heartbeat, + connAlive: verifier.ProbeConnAlive, + log: log, + } +} + +func (r *Runner) Run(ctx context.Context) { + if r.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", r)) + } + + logger := r.log.With("job_id", uuid.New(), "job_name", jobName) + logger.Info("runner loop started") + + for ctx.Err() == nil { + // Gate claiming on local capacity so we never hold a server-claimed + // job we cannot start immediately. + if r.pool.Saturated() { + if !sleepOrDone(ctx, 1*time.Second) { + break + } + + continue + } + + capacity := api.AgentCapacity{ + MaxCPU: r.capacity.MaxCPU, + MaxRAMMb: r.capacity.MaxRAMMb, + MaxDiskGb: r.capacity.MaxDiskGb, + MaxConcurrentJobs: r.capacity.MaxConcurrentJobs, + } + + assignment, err := r.api.ClaimVerification(ctx, capacity) + if err != nil { + logger.Warn("claim failed", "error", err) + + if !sleepOrDone(ctx, claimErrorBackoff) { + break + } + + continue + } + + if assignment == nil { + if !sleepOrDone(ctx, idleClaimBackoff) { + break + } + + continue + } + + r.pool.Go(func() { r.executeJob(ctx, assignment) }) + } + + logger.Info("runner loop draining in-flight jobs") + r.pool.Wait() + logger.Info("runner loop stopped") +} + +func (r *Runner) executeJob(ctx context.Context, job *api.JobAssignment) { + runLogger := r.log.With( + "job_id", uuid.New(), + "job_name", jobName, + "verification_id", job.VerificationID, + ) + + jobCtx, cancel := context.WithCancel(ctx) + defer cancel() + + // Register FIRST so the ID is in the heartbeat envelope before any + // container exists: the backend must never see an artifact-bearing job + // that is absent from this agent's reported in-flight set. + r.heartbeat.TrackVerification(job.VerificationID, cancel) + defer r.heartbeat.UntrackVerification(job.VerificationID) + + pgMajor, err := pgMajorFromDatabase(job.Database) + if err != nil { + r.reportFailure(ctx, job.VerificationID, nil, fmt.Sprintf("resolve pg major: %v", err), runLogger) + + return + } + + jobContainer, err := r.spawner.Spawn(jobCtx, SpawnRequest{ + PgMajor: pgMajor, + CPUPerJob: r.capacity.CPUPerJob, + RAMMbPerJob: r.capacity.RAMMbPerJob, + VerificationID: job.VerificationID, + TimescaledbVersion: job.TimescaledbVersion, + }) + if err != nil { + if jobCtx.Err() != nil { + return + } + + r.reportFailure(ctx, job.VerificationID, nil, fmt.Sprintf("spawn container: %v", err), runLogger) + + return + } + + defer func() { + if termErr := jobContainer.Terminate(ctx); termErr != nil { + runLogger.Error("failed to terminate container", "error", termErr) + } + }() + + // Server-authoritative per-job disk budget (downloaded archive + restored + // DB + safe gap). The agent enforces it by watching the container's written + // bytes and aborting the instant the budget is reached — uniform on every + // storage driver, unlike a Docker --storage-opt cap. + diskBudgetMb := int64(job.MaxContainerDiskMb) + diskBudgetFailMessage := fmt.Sprintf("restore exceeded the per-job disk budget of %d MB", diskBudgetMb) + + var isDiskLimitHit atomic.Bool + + // A zero/absent budget (older backend) means no enforceable ceiling — skip + // the watcher rather than trip instantly on used >= 0. + if diskBudgetMb > 0 { + watchCtx, watchStop := context.WithCancel(jobCtx) + defer watchStop() + + go newDiskWatcher( + jobContainer, diskBudgetMb*1024*1024, + func() { isDiskLimitHit.Store(true); cancel() }, + runLogger, + ).run(watchCtx) + } else { + runLogger.Warn("no per-job disk budget from server; disk watcher disabled for this job") + } + + archivePath := fmt.Sprintf("%s/%s.dump", inContainerArchiveDir, job.VerificationID) + + if err := r.downloadBackupIntoContainer( + jobCtx, + jobContainer, + job.VerificationID, + archivePath, + runLogger, + ); err != nil { + if isDiskLimitHit.Load() { + r.reportDiskLimitExceeded(ctx, job.VerificationID, diskBudgetFailMessage, runLogger) + + return + } + + if errors.Is(err, errAbortNoReport) { + return + } + + r.reportFailure(ctx, job.VerificationID, nil, fmt.Sprintf("download backup: %v", err), runLogger) + + return + } + + parallelJobs := min(maxParallelRestoreJobs, r.capacity.CPUPerJob) + + // TimescaleDB restores single-threaded and in restoring mode. Single-threaded because parallel + // pg_restore loads dependent _timescaledb_catalog rows before the rows they reference, tripping + // the catalog's own foreign keys. post_restore runs on the success path below, before stats; on + // failure the throwaway container is discarded, so nothing is left in restoring mode to repair + // (no defer, unlike the backend's user-facing restore). The hooks use the verifier (host) conn + // while pg_restore uses the in-container conn, but the database-level restoring GUC carries across. + if job.TimescaledbVersion != "" { + parallelJobs = 1 + + if err := r.restorer.RunTimescalePreRestore(jobCtx, jobContainer.GetVerifierConn()); err != nil { + if jobCtx.Err() != nil { + return + } + + r.reportFailure(ctx, job.VerificationID, nil, + fmt.Sprintf("timescaledb_pre_restore: %v", err), runLogger) + + return + } + } + + restoreResult, err := r.restorer.RunPgRestore( + jobCtx, jobContainer, archivePath, jobContainer.GetInContainerConn(), parallelJobs) + if err != nil { + if isDiskLimitHit.Load() { + r.reportDiskLimitExceeded(ctx, job.VerificationID, diskBudgetFailMessage, runLogger) + + return + } + + if jobCtx.Err() != nil { + return + } + + if !errors.Is(err, restore.ErrRestoreFailed) { + // Exec-infrastructure failure (no usable exit code). + r.reportFailure(ctx, job.VerificationID, nil, + fmt.Sprintf("pg_restore exec: %v", err), runLogger) + + return + } + + if restore.IsDiskExhausted(restoreResult.StderrTail) { + // Hitting the estimate-derived ceiling is agent infra, not proof the + // backup is corrupt — omit the exit code so the backend retries + // (AgentSetupFailed), never BackupRejected. + r.reportFailure(ctx, job.VerificationID, nil, + fmt.Sprintf("pg_restore hit job disk ceiling: %v", err), runLogger) + + return + } + + if !restore.IsMissingExtensionOnly(restoreResult.StderrTail) { + runLogger.Error("pg_restore failed", + "exit_code", restoreResult.PgRestoreExitCode, + "stderr_tail", restoreResult.StderrTail) + + r.reportFailure(ctx, job.VerificationID, &restoreResult, + fmt.Sprintf("pg_restore failed: %v", err), runLogger) + + return + } + + // The only items pg_restore skipped are CREATE/COMMENT EXTENSION for + // extensions this verification environment lacks — the data restored. + // Treat the backup as restorable and fall through to stats; the backend's + // restored-size check is the remaining safety net. + runLogger.Warn( + "pg_restore completed with only missing-extension errors; treating backup as restorable", + "exit_code", restoreResult.PgRestoreExitCode, + "skipped_extensions", + strings.Join(restore.ExtractUnavailableExtensions(restoreResult.StderrTail), ","), + "stderr_tail", restoreResult.StderrTail) + } + + if job.TimescaledbVersion != "" { + if err := r.restorer.RunTimescalePostRestore(jobCtx, jobContainer.GetVerifierConn()); err != nil { + if jobCtx.Err() != nil { + return + } + + r.reportFailure(ctx, job.VerificationID, &restoreResult, + fmt.Sprintf("timescaledb_post_restore: %v", err), runLogger) + + return + } + } + + verifierConn := jobContainer.GetVerifierConn() + + stats, err := r.verifier.CollectStats(jobCtx, verifierConn) + if err != nil { + if isDiskLimitHit.Load() { + r.reportDiskLimitExceeded(ctx, job.VerificationID, diskBudgetFailMessage, runLogger) + + return + } + + if jobCtx.Err() != nil { + return + } + + if !r.connAlive(jobCtx, verifierConn) { + r.reportFailure(ctx, job.VerificationID, nil, + fmt.Sprintf("verify (conn dead): %v", err), runLogger) + + return + } + + r.reportFailure(ctx, job.VerificationID, &restoreResult, + fmt.Sprintf("verify: %v", err), runLogger) + + return + } + + if isDiskLimitHit.Load() { + r.reportDiskLimitExceeded(ctx, job.VerificationID, diskBudgetFailMessage, runLogger) + + return + } + + r.reportSuccess(ctx, job.VerificationID, restoreResult, stats, runLogger) +} + +func (r *Runner) downloadBackupIntoContainer( + jobCtx context.Context, + jobContainer JobContainer, + verificationID uuid.UUID, + archivePath string, + logger *slog.Logger, +) error { + var lastErr error + + for attempt := 1; attempt <= maxBackupDownloadAttempts; attempt++ { + if jobCtx.Err() != nil { + return errAbortNoReport + } + + err := r.downloadBackupIntoContainerOnce(jobCtx, jobContainer, verificationID, archivePath) + if err == nil { + return nil + } + + var respErr *api.ResponseError + if errors.As(err, &respErr) { + if respErr.IsGone() { + logger.Info("backup stream gone (410); dropping without report") + + return errAbortNoReport + } + + if !respErr.Retryable() { + return fmt.Errorf("backup stream non-retryable: %w", err) + } + } + + lastErr = err + logger.Warn("backup stream attempt failed", + "attempt", attempt, "max_attempts", maxBackupDownloadAttempts, "error", err) + + if !sleepOrDone(jobCtx, backupDownloadBackoffFn(attempt)) { + return errAbortNoReport + } + } + + return fmt.Errorf("backup stream failed after %d attempts: %w", maxBackupDownloadAttempts, lastErr) +} + +func (r *Runner) downloadBackupIntoContainerOnce( + jobCtx context.Context, + jobContainer JobContainer, + verificationID uuid.UUID, + archivePath string, +) error { + body, err := r.api.DownloadBackup(jobCtx, verificationID) + if err != nil { + return err + } + defer func() { _ = body.Close() }() + + dlCtx, dlCancel := context.WithCancelCause(jobCtx) + defer dlCancel(nil) + + idleReader := api.NewIdleTimeoutReader(body, streamIdleTimeout, dlCancel) + defer idleReader.Stop() + + return r.restorer.StageBackupViaExec(dlCtx, jobContainer, idleReader, archivePath) +} + +func (r *Runner) reportSuccess( + ctx context.Context, + verificationID uuid.UUID, + restoreResult restore.Result, + stats verifier.Stats, + logger *slog.Logger, +) { + tableStats := make([]api.ReportTableStat, 0, len(stats.TableStats)) + for _, ts := range stats.TableStats { + tableStats = append(tableStats, api.ReportTableStat{ + SchemaName: ts.SchemaName, + Name: ts.Name, + RowCount: ts.RowCount, + }) + } + + req := api.ReportRequest{ + Status: api.VerificationStatusCompleted, + PgRestoreExitCode: &restoreResult.PgRestoreExitCode, + RestoreDurationMs: &restoreResult.DurationMs, + DBSizeBytesAfterRestore: &stats.DBSizeBytes, + TableCount: &stats.TableCount, + SchemaCount: &stats.SchemaCount, + TableStats: tableStats, + } + + r.sendReport(ctx, verificationID, req, logger) +} + +func (r *Runner) reportFailure( + ctx context.Context, + verificationID uuid.UUID, + restoreResult *restore.Result, + failMessage string, + logger *slog.Logger, +) { + if r.shouldSkipFailedReport(ctx, verificationID, logger) { + return + } + + message := failMessage + req := api.ReportRequest{ + Status: api.VerificationStatusFailed, + FailMessage: &message, + } + + if restoreResult != nil { + req.PgRestoreExitCode = &restoreResult.PgRestoreExitCode + req.RestoreDurationMs = &restoreResult.DurationMs + } + + r.sendReport(ctx, verificationID, req, logger) +} + +// reportDiskLimitExceeded posts the terminal disk-budget verdict. FailureKind +// makes the backend classify it DISK_LIMIT_EXCEEDED (terminal) instead of the +// retryable nil-exit-code path: the budget is server-computed, so retrying the +// same job against the same budget would fail identically. +func (r *Runner) reportDiskLimitExceeded( + ctx context.Context, + verificationID uuid.UUID, + failMessage string, + logger *slog.Logger, +) { + if r.shouldSkipFailedReport(ctx, verificationID, logger) { + return + } + + message := failMessage + kind := api.FailureKindDiskLimitExceeded + r.sendReport(ctx, verificationID, api.ReportRequest{ + Status: api.VerificationStatusFailed, + FailMessage: &message, + FailureKind: &kind, + }, logger) +} + +// shouldSkipFailedReport is the abort/report-race guard (the only agent-side +// lever with a frozen FAILED path): never POST FAILED for an aborted/cancelled +// ID — a spurious FAILED would flip CANCELED→FAILED or resurrect a cancelled +// row. +func (r *Runner) shouldSkipFailedReport( + ctx context.Context, verificationID uuid.UUID, logger *slog.Logger, +) bool { + if ctx.Err() != nil || r.heartbeat.IsAborted(verificationID) { + logger.Info("skipping FAILED report: verification aborted/cancelled") + + return true + } + + return false +} + +func (r *Runner) sendReport( + ctx context.Context, + verificationID uuid.UUID, + req api.ReportRequest, + logger *slog.Logger, +) { + err := r.api.Report(ctx, verificationID, req) + if err == nil { + logger.Info(fmt.Sprintf("report accepted: status=%s", req.Status)) + + return + } + + switch { + case errors.Is(err, api.ErrReportGone): + logger.Info("report dropped: verification no longer owned by this agent (410)") + case errors.Is(err, api.ErrReportBudgetExhausted): + logger.Warn("report abandoned: retry budget exhausted; backend will reclaim on next heartbeat") + default: + logger.Warn("report failed", "error", err) + } +} + +func pgMajorFromDatabase(db api.AssignedDatabase) (string, error) { + if db.Type != "POSTGRES_LOGICAL" { + return "", fmt.Errorf("unsupported database type %q (v1 is Postgres only)", db.Type) + } + + if db.PostgresqlLogical == nil || db.PostgresqlLogical.Version == "" { + return "", errors.New("assignment missing postgresql version") + } + + if !slices.Contains(supportedMajors, db.PostgresqlLogical.Version) { + return "", fmt.Errorf("unsupported postgres major %q", db.PostgresqlLogical.Version) + } + + return db.PostgresqlLogical.Version, nil +} + +func backupDownloadBackoff(attempt int) time.Duration { + if attempt < 1 { + attempt = 1 + } + + d := time.Duration(1< maxBackupDownloadBackoff { + return maxBackupDownloadBackoff + } + + return d +} + +func sleepOrDone(ctx context.Context, d time.Duration) bool { + timer := time.NewTimer(d) + defer timer.Stop() + + select { + case <-ctx.Done(): + return false + case <-timer.C: + return true + } +} diff --git a/agent/verification/internal/features/runner/runner_test.go b/agent/verification/internal/features/runner/runner_test.go new file mode 100644 index 0000000..5596c4b --- /dev/null +++ b/agent/verification/internal/features/runner/runner_test.go @@ -0,0 +1,382 @@ +package runner + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/dbconn" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/verifier" + "databasus-verification-agent/internal/testutil" +) + +func testCapacity() config.Capacity { + return config.Capacity{ + MaxCPU: 4, MaxRAMMb: 4096, MaxDiskGb: 50, MaxConcurrentJobs: 2, + CPUPerJob: 2, RAMMbPerJob: 2048, + } +} + +func postgresJob() *api.JobAssignment { + return &api.JobAssignment{ + VerificationID: uuid.New(), + BackupID: uuid.New(), + BackupSizeMb: 50, + MaxContainerDiskMb: 200, + Database: api.AssignedDatabase{ + Type: "POSTGRES_LOGICAL", + PostgresqlLogical: &api.AssignedPostgresql{Version: "16"}, + }, + } +} + +func newTestRunner( + apiClient APIClient, spawner Spawner, restorer Restorer, + stats StatsCollector, reg Registrar, +) *Runner { + return NewRunner( + apiClient, testCapacity(), NewPool(2), + spawner, restorer, stats, reg, testutil.DiscardLogger(), + ) +} + +func testConn() dbconn.Conn { + return dbconn.Conn{ + Host: "127.0.0.1", Port: 5432, + User: "postgres", Password: "pw", Database: "postgres", + } +} + +func fakeContainerWith(conn dbconn.Conn) *fakeContainer { + return &fakeContainer{inContainerConn: conn, verifierConn: conn} +} + +func okSpawner() *fakeSpawner { + return &fakeSpawner{container: fakeContainerWith(testConn())} +} + +func Test_ExecuteJob_WhenPgMajorUnsupported_ReportsFailedWithoutExitCode(t *testing.T) { + apiClient := &fakeAPI{} + r := newTestRunner(apiClient, okSpawner(), &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + job := postgresJob() + job.Database.Type = "MYSQL" + + r.executeJob(t.Context(), job) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode) +} + +func Test_ExecuteJob_WhenSpawnFails_ReportsFailedWithoutExitCode(t *testing.T) { + apiClient := &fakeAPI{} + spawner := &fakeSpawner{err: errors.New("image pull failed")} + r := newTestRunner(apiClient, spawner, &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode) +} + +func Test_ExecuteJob_WhenStreamReturns410_DoesNotReport(t *testing.T) { + apiClient := &fakeAPI{downloadErr: &api.ResponseError{Op: "backup stream", StatusCode: 410}} + r := newTestRunner(apiClient, okSpawner(), &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + assert.Equal(t, 0, apiClient.reportCalled, "a 410 on the stream is an abort — no report") +} + +func Test_ExecuteJob_WhenStreamExhaustsRetries_ReportsFailedWithoutExitCode(t *testing.T) { + original := backupDownloadBackoffFn + backupDownloadBackoffFn = func(int) time.Duration { return 0 } + t.Cleanup(func() { backupDownloadBackoffFn = original }) + + apiClient := &fakeAPI{downloadErr: errors.New("connection refused")} + r := newTestRunner(apiClient, okSpawner(), &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode) +} + +func Test_ExecuteJob_WhenRestoreFailsAndDiskExhausted_ReportsFailedWithoutExitCode(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{ + runErr: fmt.Errorf("restore: %w", restore.ErrRestoreFailed), + runResult: restore.Result{PgRestoreExitCode: 1, StderrTail: "could not write: No space left on device"}, + } + r := newTestRunner(apiClient, okSpawner(), restorer, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode, + "hitting the estimate-derived disk ceiling is agent infra, not BackupRejected") +} + +func Test_ExecuteJob_WhenDiskWatcherTrips_ReportsTerminalDiskLimitExceeded(t *testing.T) { + originalInterval := diskWatchInterval + diskWatchInterval = time.Millisecond + t.Cleanup(func() { diskWatchInterval = originalInterval }) + + originalBackoff := backupDownloadBackoffFn + backupDownloadBackoffFn = func(int) time.Duration { return 0 } + t.Cleanup(func() { backupDownloadBackoffFn = originalBackoff }) + + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + jobContainer := fakeContainerWith(testConn()) + jobContainer.diskUsageBytes = 1 << 40 + restorer := &fakeRestorer{runBlocks: true} + + r := newTestRunner( + apiClient, &fakeSpawner{container: jobContainer}, restorer, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + require.NotNil(t, report.FailureKind) + assert.Equal(t, api.FailureKindDiskLimitExceeded, *report.FailureKind) + assert.Nil(t, report.PgRestoreExitCode, "a disk-limit verdict carries no pg_restore exit code") + assert.True(t, jobContainer.terminated, "the container must be torn down on disk-limit abort") +} + +func Test_ExecuteJob_WhenServerSendsNoDiskBudget_DoesNotFalselyTrip(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + jobContainer := fakeContainerWith(testConn()) + jobContainer.diskUsageBytes = 1 << 40 + + job := postgresJob() + job.MaxContainerDiskMb = 0 + + r := newTestRunner( + apiClient, &fakeSpawner{container: jobContainer}, &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), job) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusCompleted, report.Status, + "a zero budget disables the watcher; the job must not be falsely failed") + assert.Nil(t, report.FailureKind) +} + +func Test_ExecuteJob_WhenRestoreFailsNormally_ReportsFailedWithExitCode(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{ + runErr: fmt.Errorf("restore: %w", restore.ErrRestoreFailed), + runResult: restore.Result{PgRestoreExitCode: 1, DurationMs: 42, StderrTail: "syntax error in dump"}, + } + r := newTestRunner(apiClient, okSpawner(), restorer, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + require.NotNil(t, report.PgRestoreExitCode) + assert.Equal(t, 1, *report.PgRestoreExitCode) +} + +func Test_ExecuteJob_WhenRestoreFailsWithOnlyMissingExtensions_ReportsCompleted(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + stderr := "pg_restore: error: could not execute query: ERROR: extension \"set_user\" is not available\n" + + "Command was: CREATE EXTENSION IF NOT EXISTS set_user WITH SCHEMA public;\n" + + "pg_restore: error: could not execute query: ERROR: extension \"set_user\" does not exist\n" + + "Command was: COMMENT ON EXTENSION set_user IS 'x';\n" + + "pg_restore: warning: errors ignored on restore: 2\n" + restorer := &fakeRestorer{ + runErr: fmt.Errorf("restore: %w", restore.ErrRestoreFailed), + runResult: restore.Result{PgRestoreExitCode: 1, DurationMs: 10, StderrTail: stderr}, + } + stats := &fakeStats{stats: verifier.Stats{DBSizeBytes: 9_000_000, TableCount: 3}} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusCompleted, report.Status, + "a restore whose only failures are missing extensions is restorable") + require.NotNil(t, report.PgRestoreExitCode) + assert.Equal(t, 1, *report.PgRestoreExitCode, + "the real non-zero exit code is preserved on the success report") + require.NotNil(t, report.DBSizeBytesAfterRestore) + assert.Equal(t, int64(9_000_000), *report.DBSizeBytesAfterRestore) +} + +func Test_ExecuteJob_WhenRestoreExecInfraFails_ReportsFailedWithoutExitCode(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runErr: errors.New("exec create failed")} + r := newTestRunner(apiClient, okSpawner(), restorer, &fakeStats{}, newFakeRegistrar()) + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode) +} + +func Test_ExecuteJob_WhenVerifyFailsAndConnDead_ReportsFailedWithoutExitCode(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0}} + stats := &fakeStats{err: errors.New("tier 1 failed")} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return false } + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + assert.Nil(t, report.PgRestoreExitCode, "a dead connection is agent infra (retryable)") +} + +func Test_ExecuteJob_WhenVerifyFailsAndConnAlive_ReportsFailedWithExitCodeZero(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0}} + stats := &fakeStats{err: errors.New("tier 1 impossible result")} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusFailed, report.Status) + require.NotNil(t, report.PgRestoreExitCode) + assert.Equal(t, 0, *report.PgRestoreExitCode, + "a live conn with a broken tier-1 result is BackupRejected with exit 0") +} + +func Test_ExecuteJob_WhenAllSucceeds_ReportsCompletedWithStats(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0, DurationMs: 1234}} + stats := &fakeStats{stats: verifier.Stats{ + DBSizeBytes: 9_000_000, + SchemaCount: 2, + TableCount: 3, + TableStats: []verifier.TableStat{{SchemaName: "public", Name: "t1", RowCount: 10}}, + }} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + r.executeJob(t.Context(), postgresJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusCompleted, report.Status) + require.NotNil(t, report.DBSizeBytesAfterRestore) + assert.Equal(t, int64(9_000_000), *report.DBSizeBytesAfterRestore) + require.NotNil(t, report.TableCount) + assert.Equal(t, 3, *report.TableCount) + require.Len(t, report.TableStats, 1) + assert.Equal(t, "public", report.TableStats[0].SchemaName) +} + +func timescaleJob() *api.JobAssignment { + job := postgresJob() + job.TimescaledbVersion = "2.17.0" + + return job +} + +func Test_ExecuteJob_WhenTimescaleJobSucceeds_RunsPreRestoreThenRestoreThenPostRestore(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0}} + stats := &fakeStats{stats: verifier.Stats{TableCount: 1}} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + r.executeJob(t.Context(), timescaleJob()) + + report, ok := apiClient.lastReport() + require.True(t, ok) + assert.Equal(t, api.VerificationStatusCompleted, report.Status) + assert.Equal(t, []string{"pre", "restore", "post"}, restorer.calls, + "timescaledb restore must enter restoring mode, restore, then leave it") + assert.Equal(t, 1, restorer.runParallelJobs, + "timescaledb restore must be single-threaded to keep _timescaledb_catalog FK order") +} + +func Test_ExecuteJob_WhenNonTimescaleJob_DoesNotRunTimescaleHooks(t *testing.T) { + apiClient := &fakeAPI{downloadBody: []byte("ARCHIVE")} + restorer := &fakeRestorer{runResult: restore.Result{PgRestoreExitCode: 0}} + stats := &fakeStats{stats: verifier.Stats{TableCount: 1}} + r := newTestRunner(apiClient, okSpawner(), restorer, stats, newFakeRegistrar()) + r.connAlive = func(context.Context, dbconn.Conn) bool { return true } + + r.executeJob(t.Context(), postgresJob()) + + assert.Equal(t, []string{"restore"}, restorer.calls, + "a non-timescaledb restore must not run the timescaledb hooks") + assert.Equal(t, 2, restorer.runParallelJobs, + "a non-timescaledb restore keeps parallel jobs (CPUPerJob)") +} + +func Test_ExecuteJob_WhenAbortedBeforeFailedReport_DoesNotReport(t *testing.T) { + apiClient := &fakeAPI{} + spawner := &fakeSpawner{err: errors.New("spawn failed")} + reg := newFakeRegistrar() + reg.aborted = true + r := newTestRunner(apiClient, spawner, &fakeRestorer{}, &fakeStats{}, reg) + + r.executeJob(t.Context(), postgresJob()) + + assert.Equal(t, 0, apiClient.reportCalled, + "an aborted/cancelled verification must not get a spurious FAILED report") +} + +func Test_Run_WhenContextCancelled_DrainsAndStops(t *testing.T) { + apiClient := &fakeAPI{} + r := newTestRunner(apiClient, okSpawner(), &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + ctx, cancel := context.WithCancel(t.Context()) + + done := make(chan struct{}) + go func() { + r.Run(ctx) + close(done) + }() + + cancel() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("Run did not return after context cancellation") + } +} + +func Test_Run_WhenCalledTwice_Panics(t *testing.T) { + r := newTestRunner(&fakeAPI{}, okSpawner(), &fakeRestorer{}, &fakeStats{}, newFakeRegistrar()) + + ctx, cancel := context.WithCancel(t.Context()) + cancel() + r.Run(ctx) + + assert.Panics(t, func() { r.Run(ctx) }) +} diff --git a/agent/verification/internal/features/start/daemon.go b/agent/verification/internal/features/start/daemon.go new file mode 100644 index 0000000..c34d478 --- /dev/null +++ b/agent/verification/internal/features/start/daemon.go @@ -0,0 +1,119 @@ +package start + +import ( + "context" + "errors" + "fmt" + "log/slog" + "os" + "os/exec" + "syscall" + "time" +) + +const ( + logFileName = "databasus-verification.daemon.log" + stopTimeout = 30 * time.Second + stopPollInterval = 500 * time.Millisecond + daemonStartupDelay = 500 * time.Millisecond +) + +func Stop(log *slog.Logger) error { + pid, err := ReadLockFilePID() + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return errors.New("agent is not running (no lock file found)") + } + + return fmt.Errorf("failed to read lock file: %w", err) + } + + if !isProcessAlive(pid) { + _ = os.Remove(lockFileName) + return fmt.Errorf("agent is not running (stale lock file removed, PID %d)", pid) + } + + log.Info("Sending SIGTERM to agent", "pid", pid) + + if err := syscall.Kill(pid, syscall.SIGTERM); err != nil { + return fmt.Errorf("failed to send SIGTERM to PID %d: %w", pid, err) + } + + deadline := time.Now().Add(stopTimeout) + for time.Now().Before(deadline) { + if !isProcessAlive(pid) { + log.Info("Agent stopped", "pid", pid) + return nil + } + + time.Sleep(stopPollInterval) + } + + return fmt.Errorf("agent (PID %d) did not stop within %s — process may be stuck", pid, stopTimeout) +} + +func Status(log *slog.Logger) error { + pid, err := ReadLockFilePID() + if err != nil { + if errors.Is(err, os.ErrNotExist) { + fmt.Println("Agent is not running") + return nil + } + + return fmt.Errorf("failed to read lock file: %w", err) + } + + if isProcessAlive(pid) { + fmt.Printf("Agent is running (PID %d)\n", pid) + } else { + fmt.Println("Agent is not running (stale lock file)") + _ = os.Remove(lockFileName) + } + + return nil +} + +func spawnDaemon(log *slog.Logger) (int, error) { + execPath, err := os.Executable() + if err != nil { + return 0, fmt.Errorf("failed to resolve executable path: %w", err) + } + + args := []string{"_run"} + + logFile, err := os.OpenFile(logFileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return 0, fmt.Errorf("failed to open log file %s: %w", logFileName, err) + } + + cwd, err := os.Getwd() + if err != nil { + _ = logFile.Close() + return 0, fmt.Errorf("failed to get working directory: %w", err) + } + + cmd := exec.CommandContext(context.Background(), execPath, args...) + cmd.Dir = cwd + cmd.Stderr = logFile + cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true} + + if err := cmd.Start(); err != nil { + _ = logFile.Close() + return 0, fmt.Errorf("failed to start daemon process: %w", err) + } + + pid := cmd.Process.Pid + + // Detach — we don't wait for the child + _ = logFile.Close() + + time.Sleep(daemonStartupDelay) + + if !isProcessAlive(pid) { + return 0, fmt.Errorf("daemon process (PID %d) exited immediately — check %s for details", pid, logFileName) + } + + log.Info("Daemon spawned", "pid", pid, "log", logFileName) + + return pid, nil +} diff --git a/agent/verification/internal/features/start/doc.go b/agent/verification/internal/features/start/doc.go new file mode 100644 index 0000000..ddf7755 --- /dev/null +++ b/agent/verification/internal/features/start/doc.go @@ -0,0 +1,6 @@ +// Package start runs the verification agent as a single-instance background +// daemon: it spawns a detached process, holds an exclusive lock so a second +// agent cannot run from the same working directory, and supervises the +// capacity heartbeat and verification runner until a signal or a self-upgrade +// restart. +package start diff --git a/agent/verification/internal/features/start/lock.go b/agent/verification/internal/features/start/lock.go new file mode 100644 index 0000000..2844346 --- /dev/null +++ b/agent/verification/internal/features/start/lock.go @@ -0,0 +1,130 @@ +package start + +import ( + "errors" + "fmt" + "io" + "log/slog" + "os" + "strconv" + "strings" + "syscall" +) + +const lockFileName = "databasus-verification.lock" + +func AcquireLock(log *slog.Logger) (*os.File, error) { + f, err := os.OpenFile(lockFileName, os.O_CREATE|os.O_RDWR, 0o644) + if err != nil { + return nil, fmt.Errorf("failed to open lock file: %w", err) + } + + err = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) + if err == nil { + if err := writePID(f); err != nil { + _ = f.Close() + return nil, err + } + + log.Info("Process lock acquired", "pid", os.Getpid(), "lockFile", lockFileName) + + return f, nil + } + + if !errors.Is(err, syscall.EWOULDBLOCK) { + _ = f.Close() + return nil, fmt.Errorf("failed to acquire lock: %w", err) + } + + pid, pidErr := readLockPID(f) + _ = f.Close() + + if pidErr != nil { + return nil, fmt.Errorf("another instance is already running") + } + + return nil, fmt.Errorf("another instance is already running (PID %d)", pid) +} + +func ReleaseLock(f *os.File) { + _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN) + + lockedStat, lockedErr := f.Stat() + _ = f.Close() + + if lockedErr != nil { + _ = os.Remove(lockFileName) + return + } + + diskStat, diskErr := os.Stat(lockFileName) + if diskErr != nil { + return + } + + if os.SameFile(lockedStat, diskStat) { + _ = os.Remove(lockFileName) + } +} + +func ReadLockFilePID() (int, error) { + f, err := os.Open(lockFileName) + if err != nil { + return 0, err + } + defer func() { _ = f.Close() }() + + return readLockPID(f) +} + +func writePID(f *os.File) error { + if err := f.Truncate(0); err != nil { + return fmt.Errorf("failed to truncate lock file: %w", err) + } + + if _, err := f.Seek(0, io.SeekStart); err != nil { + return fmt.Errorf("failed to seek lock file: %w", err) + } + + if _, err := fmt.Fprintf(f, "%d\n", os.Getpid()); err != nil { + return fmt.Errorf("failed to write PID to lock file: %w", err) + } + + return f.Sync() +} + +func readLockPID(f *os.File) (int, error) { + if _, err := f.Seek(0, io.SeekStart); err != nil { + return 0, err + } + + data, err := io.ReadAll(f) + if err != nil { + return 0, err + } + + s := strings.TrimSpace(string(data)) + if s == "" { + return 0, errors.New("lock file is empty") + } + + pid, err := strconv.Atoi(s) + if err != nil { + return 0, fmt.Errorf("invalid PID in lock file: %w", err) + } + + return pid, nil +} + +func isProcessAlive(pid int) bool { + err := syscall.Kill(pid, 0) + if err == nil { + return true + } + + if errors.Is(err, syscall.EPERM) { + return true + } + + return false +} diff --git a/agent/verification/internal/features/start/lock_test.go b/agent/verification/internal/features/start/lock_test.go new file mode 100644 index 0000000..97f0e3f --- /dev/null +++ b/agent/verification/internal/features/start/lock_test.go @@ -0,0 +1,148 @@ +//go:build !windows + +package start + +import ( + "fmt" + "os" + "strconv" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/logger" +) + +func Test_AcquireLock_LockFileCreatedWithPID(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + data, err := os.ReadFile(lockFileName) + require.NoError(t, err) + + pid, err := strconv.Atoi(strings.TrimSpace(string(data))) + require.NoError(t, err) + assert.Equal(t, os.Getpid(), pid) +} + +func Test_AcquireLock_SecondAcquireFails_WhenFirstHeld(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + first, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(first) + + second, err := AcquireLock(log) + assert.Nil(t, second) + require.Error(t, err) + assert.Contains(t, err.Error(), "another instance is already running") + assert.Contains(t, err.Error(), fmt.Sprintf("PID %d", os.Getpid())) +} + +func Test_AcquireLock_StaleLockReacquired_WhenProcessDead(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + err := os.WriteFile(lockFileName, []byte("999999999\n"), 0o644) + require.NoError(t, err) + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + data, err := os.ReadFile(lockFileName) + require.NoError(t, err) + + pid, err := strconv.Atoi(strings.TrimSpace(string(data))) + require.NoError(t, err) + assert.Equal(t, os.Getpid(), pid) +} + +func Test_ReleaseLock_LockFileRemoved(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + + ReleaseLock(lockFile) + + _, err = os.Stat(lockFileName) + assert.True(t, os.IsNotExist(err)) +} + +func Test_AcquireLock_ReacquiredAfterRelease(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + first, err := AcquireLock(log) + require.NoError(t, err) + ReleaseLock(first) + + second, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(second) + + data, err := os.ReadFile(lockFileName) + require.NoError(t, err) + + pid, err := strconv.Atoi(strings.TrimSpace(string(data))) + require.NoError(t, err) + assert.Equal(t, os.Getpid(), pid) +} + +func Test_isProcessAlive_ReturnsTrueForSelf(t *testing.T) { + assert.True(t, isProcessAlive(os.Getpid())) +} + +func Test_isProcessAlive_ReturnsFalseForNonExistentPID(t *testing.T) { + assert.False(t, isProcessAlive(999999999)) +} + +func Test_readLockPID_ParsesValidPID(t *testing.T) { + setupTempDir(t) + + f, err := os.CreateTemp("", "lock-test-*") + require.NoError(t, err) + defer os.Remove(f.Name()) + + _, err = f.WriteString("12345\n") + require.NoError(t, err) + + pid, err := readLockPID(f) + require.NoError(t, err) + assert.Equal(t, 12345, pid) +} + +func Test_readLockPID_ReturnsErrorForEmptyFile(t *testing.T) { + setupTempDir(t) + + f, err := os.CreateTemp("", "lock-test-*") + require.NoError(t, err) + defer os.Remove(f.Name()) + + _, err = readLockPID(f) + require.Error(t, err) + assert.Contains(t, err.Error(), "lock file is empty") +} + +func setupTempDir(t *testing.T) string { + t.Helper() + + origDir, err := os.Getwd() + require.NoError(t, err) + + dir := t.TempDir() + require.NoError(t, os.Chdir(dir)) + + t.Cleanup(func() { _ = os.Chdir(origDir) }) + + return dir +} diff --git a/agent/verification/internal/features/start/lock_watcher.go b/agent/verification/internal/features/start/lock_watcher.go new file mode 100644 index 0000000..a7d90e4 --- /dev/null +++ b/agent/verification/internal/features/start/lock_watcher.go @@ -0,0 +1,88 @@ +package start + +import ( + "context" + "log/slog" + "os" + "syscall" + "time" +) + +const lockWatchInterval = 5 * time.Second + +type LockWatcher struct { + originalInode uint64 + cancel context.CancelFunc + log *slog.Logger +} + +func NewLockWatcher(lockFile *os.File, cancel context.CancelFunc, log *slog.Logger) (*LockWatcher, error) { + inode, err := getFileInode(lockFile) + if err != nil { + return nil, err + } + + return &LockWatcher{ + originalInode: inode, + cancel: cancel, + log: log, + }, nil +} + +func (w *LockWatcher) Run(ctx context.Context) { + ticker := time.NewTicker(lockWatchInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + w.check() + } + } +} + +func (w *LockWatcher) check() { + info, err := os.Stat(lockFileName) + if err != nil { + w.log.Error("Lock file disappeared, shutting down", "file", lockFileName, "error", err) + w.cancel() + + return + } + + currentInode, err := getStatInode(info) + if err != nil { + w.log.Error("Failed to read lock file inode, shutting down", "error", err) + w.cancel() + + return + } + + if currentInode != w.originalInode { + w.log.Error("Lock file was replaced (inode changed), shutting down", + "originalInode", w.originalInode, + "currentInode", currentInode, + ) + w.cancel() + } +} + +func getFileInode(f *os.File) (uint64, error) { + info, err := f.Stat() + if err != nil { + return 0, err + } + + return getStatInode(info) +} + +func getStatInode(info os.FileInfo) (uint64, error) { + stat, ok := info.Sys().(*syscall.Stat_t) + if !ok { + return 0, os.ErrInvalid + } + + return stat.Ino, nil +} diff --git a/agent/verification/internal/features/start/lock_watcher_test.go b/agent/verification/internal/features/start/lock_watcher_test.go new file mode 100644 index 0000000..cc5ef1d --- /dev/null +++ b/agent/verification/internal/features/start/lock_watcher_test.go @@ -0,0 +1,110 @@ +//go:build !windows + +package start + +import ( + "context" + "os" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/logger" +) + +func Test_NewLockWatcher_CapturesInode(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + _, cancel := context.WithCancel(t.Context()) + defer cancel() + + watcher, err := NewLockWatcher(lockFile, cancel, log) + require.NoError(t, err) + assert.NotZero(t, watcher.originalInode) +} + +func Test_LockWatcher_FileUnchanged_ContextNotCancelled(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + watcher, err := NewLockWatcher(lockFile, cancel, log) + require.NoError(t, err) + + watcher.check() + watcher.check() + watcher.check() + + select { + case <-ctx.Done(): + t.Fatal("context should not be cancelled when lock file is unchanged") + default: + } +} + +func Test_LockWatcher_FileDeleted_CancelsContext(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + watcher, err := NewLockWatcher(lockFile, cancel, log) + require.NoError(t, err) + + err = os.Remove(lockFileName) + require.NoError(t, err) + + watcher.check() + + select { + case <-ctx.Done(): + default: + t.Fatal("context should be cancelled when lock file is deleted") + } +} + +func Test_LockWatcher_FileReplacedWithDifferentInode_CancelsContext(t *testing.T) { + setupTempDir(t) + log := logger.GetLogger() + + lockFile, err := AcquireLock(log) + require.NoError(t, err) + defer ReleaseLock(lockFile) + + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + + watcher, err := NewLockWatcher(lockFile, cancel, log) + require.NoError(t, err) + + err = os.Remove(lockFileName) + require.NoError(t, err) + + err = os.WriteFile(lockFileName, []byte("99999\n"), 0o644) + require.NoError(t, err) + + watcher.check() + + select { + case <-ctx.Done(): + default: + t.Fatal("context should be cancelled when lock file inode changes") + } +} diff --git a/agent/verification/internal/features/start/lock_windows.go b/agent/verification/internal/features/start/lock_windows.go new file mode 100644 index 0000000..c8fa44e --- /dev/null +++ b/agent/verification/internal/features/start/lock_windows.go @@ -0,0 +1,18 @@ +package start + +import ( + "log/slog" + "os" +) + +func AcquireLock(log *slog.Logger) (*os.File, error) { + log.Warn("Process locking is not supported on Windows, skipping") + + return nil, nil +} + +func ReleaseLock(f *os.File) { + if f != nil { + _ = f.Close() + } +} diff --git a/agent/verification/internal/features/start/spawner.go b/agent/verification/internal/features/start/spawner.go new file mode 100644 index 0000000..8a07697 --- /dev/null +++ b/agent/verification/internal/features/start/spawner.go @@ -0,0 +1,29 @@ +package start + +import ( + "context" + + "databasus-verification-agent/internal/features/container" + "databasus-verification-agent/internal/features/runner" +) + +type containerManagerSpawner struct { + containerManager *container.Manager +} + +func (s containerManagerSpawner) Spawn( + ctx context.Context, req runner.SpawnRequest, +) (runner.JobContainer, error) { + jobContainer, err := s.containerManager.Spawn(ctx, container.SpawnRequest{ + PgMajor: req.PgMajor, + CPUPerJob: req.CPUPerJob, + RAMMbPerJob: req.RAMMbPerJob, + VerificationID: req.VerificationID, + TimescaledbVersion: req.TimescaledbVersion, + }) + if err != nil { + return nil, err + } + + return jobContainer, nil +} diff --git a/agent/verification/internal/features/start/start.go b/agent/verification/internal/features/start/start.go new file mode 100644 index 0000000..c407045 --- /dev/null +++ b/agent/verification/internal/features/start/start.go @@ -0,0 +1,142 @@ +package start + +import ( + "bufio" + "context" + "fmt" + "log/slog" + "os" + "os/signal" + "runtime" + "sync" + "syscall" + "time" + + "databasus-verification-agent/internal/config" + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/features/container" + "databasus-verification-agent/internal/features/heartbeat" + "databasus-verification-agent/internal/features/restore" + "databasus-verification-agent/internal/features/runner" + "databasus-verification-agent/internal/features/upgrade" + "databasus-verification-agent/internal/features/verifier" +) + +func Start(cfg *config.Config, agentVersion string, isDev bool, log *slog.Logger) error { + if _, err := cfg.Validate(); err != nil { + return err + } + + if runtime.GOOS == "windows" { + return RunDaemon(cfg, agentVersion, isDev, log) + } + + pid, err := spawnDaemon(log) + if err != nil { + return err + } + + fmt.Printf("Agent started in background (PID %d)\n", pid) + + return nil +} + +func RunDaemon(cfg *config.Config, agentVersion string, isDev bool, log *slog.Logger) error { + capacity, err := cfg.Validate() + if err != nil { + return err + } + + if err := cfg.ValidateTransport(isStdinTTY(), bufio.NewReader(os.Stdin)); err != nil { + return err + } + + lockFile, err := AcquireLock(log) + if err != nil { + return err + } + defer ReleaseLock(lockFile) + + ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer cancel() + + watcher, err := NewLockWatcher(lockFile, cancel, log) + if err != nil { + return fmt.Errorf("failed to initialize lock watcher: %w", err) + } + go watcher.Run(ctx) + + apiClient := api.NewClient(cfg.DatabasusHost, cfg.Token, cfg.AgentID, log) + + engine, err := container.NewDockerEngine() + if err != nil { + return err + } + + containerManager := container.NewManager(engine, cfg.AgentID, cfg.GetVerificationPgImageRepo(), log) + if err := containerManager.StartupSelfCheck(ctx); err != nil { + return err + } + + // Start fresh: remove any container leaked by a prior ungraceful death + // before claiming the first job. Safe to wipe unconditionally because the + // lock guarantees a single agent owns no live container the instant it + // starts. + containerManager.PurgeContainers(ctx) + + pool := runner.NewPool(capacity.MaxConcurrentJobs) + heartbeater := heartbeat.NewHeartbeater(apiClient, capacity, log) + verificationRunner := runner.NewRunner( + apiClient, capacity, pool, + containerManagerSpawner{containerManager: containerManager}, + restore.NewRestorer(log), + verifier.NewVerifier(log), + heartbeater, + log, + ) + + var backgroundUpgrader *upgrade.BackgroundUpgrader + if agentVersion != "dev" && !isDev { + backgroundUpgrader = upgrade.NewBackgroundUpgrader(apiClient, agentVersion, isDev, cancel, log) + go backgroundUpgrader.Run(ctx) + } + + log.Info("Agent started") + + var wg sync.WaitGroup + wg.Go(func() { heartbeater.Run(ctx) }) + wg.Go(func() { verificationRunner.Run(ctx) }) + + <-ctx.Done() + + // verificationRunner.Run drains in-flight jobs (pool.Wait) before it + // returns, so wg.Wait only completes once every container is torn down — + // strictly before any re-exec on self-update. + wg.Wait() + + if backgroundUpgrader != nil { + backgroundUpgrader.WaitForCompletion(30 * time.Second) + + if backgroundUpgrader.IsUpgraded() { + // The deferred ReleaseLock runs before this returns to the caller, + // freeing the flock and removing the lock file. The caller then + // syscall.Exec's the same PID with os.Args unchanged, so the new + // image re-enters via the same subcommand (`run` or `_run`) and + // re-acquires the lock cleanly with no contender. + return upgrade.ErrUpgradeRestart + } + } + + log.Info("Agent stopped") + + return nil +} + +func isStdinTTY() bool { + info, err := os.Stdin.Stat() + if err != nil { + return false + } + + return info.Mode()&os.ModeCharDevice != 0 +} diff --git a/agent/verification/internal/features/upgrade/background_upgrader.go b/agent/verification/internal/features/upgrade/background_upgrader.go new file mode 100644 index 0000000..7131342 --- /dev/null +++ b/agent/verification/internal/features/upgrade/background_upgrader.go @@ -0,0 +1,95 @@ +package upgrade + +import ( + "context" + "fmt" + "log/slog" + "sync/atomic" + "time" + + "databasus-verification-agent/internal/features/api" +) + +const backgroundCheckInterval = 10 * time.Second + +type BackgroundUpgrader struct { + apiClient *api.Client + currentVersion string + isDev bool + cancel context.CancelFunc + hasRun atomic.Bool + isUpgraded atomic.Bool + log *slog.Logger + done chan struct{} +} + +func NewBackgroundUpgrader( + apiClient *api.Client, + currentVersion string, + isDev bool, + cancel context.CancelFunc, + log *slog.Logger, +) *BackgroundUpgrader { + return &BackgroundUpgrader{ + apiClient, + currentVersion, + isDev, + cancel, + atomic.Bool{}, + atomic.Bool{}, + log, + make(chan struct{}), + } +} + +func (u *BackgroundUpgrader) Run(ctx context.Context) { + if u.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", u)) + } + + defer close(u.done) + + ticker := time.NewTicker(backgroundCheckInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if u.checkAndUpgrade() { + return + } + } + } +} + +func (u *BackgroundUpgrader) IsUpgraded() bool { + return u.isUpgraded.Load() +} + +func (u *BackgroundUpgrader) WaitForCompletion(timeout time.Duration) { + select { + case <-u.done: + case <-time.After(timeout): + } +} + +func (u *BackgroundUpgrader) checkAndUpgrade() bool { + isUpgraded, err := CheckAndUpdate(u.apiClient, u.currentVersion, u.isDev, u.log) + if err != nil { + u.log.Warn("Background update check failed", "error", err) + + return false + } + + if !isUpgraded { + return false + } + + u.log.Info("Background upgrade complete, restarting...") + u.isUpgraded.Store(true) + u.cancel() + + return true +} diff --git a/agent/verification/internal/features/upgrade/errors.go b/agent/verification/internal/features/upgrade/errors.go new file mode 100644 index 0000000..ddecc64 --- /dev/null +++ b/agent/verification/internal/features/upgrade/errors.go @@ -0,0 +1,5 @@ +package upgrade + +import "errors" + +var ErrUpgradeRestart = errors.New("agent upgraded, restart required") diff --git a/agent/verification/internal/features/upgrade/upgrader.go b/agent/verification/internal/features/upgrade/upgrader.go new file mode 100644 index 0000000..a4eccc0 --- /dev/null +++ b/agent/verification/internal/features/upgrade/upgrader.go @@ -0,0 +1,114 @@ +package upgrade + +import ( + "context" + "fmt" + "log/slog" + "os" + "os/exec" + "runtime" + "strings" + "time" + + "databasus-verification-agent/internal/features/api" +) + +// verifyBinaryTimeout bounds the freshly-downloaded binary's `version` probe so +// a corrupt or wrong-arch binary that hangs cannot wedge startup or the +// background-upgrade goroutine. A var so tests can shrink it. +var verifyBinaryTimeout = 30 * time.Second + +// binaryDownloadTimeout bounds the agent self-download. The streaming HTTP +// client has no client-level timeout (it is shared with the long-lived backup +// stream), so the deadline is scoped per call here. +const binaryDownloadTimeout = 10 * time.Minute + +// CheckAndUpdate checks if a new version is available and upgrades the binary on disk. +// Returns (true, nil) if the binary was upgraded, (false, nil) if already up to date, +// or (false, err) on failure. Callers are responsible for re-exec or restart signaling. +func CheckAndUpdate(apiClient *api.Client, currentVersion string, isDev bool, log *slog.Logger) (bool, error) { + if isDev { + log.Info("Skipping update check (development mode)") + + return false, nil + } + + serverVersion, err := apiClient.FetchServerVersion(context.Background()) + if err != nil { + log.Warn("Could not reach server for update check", "error", err) + + return false, fmt.Errorf( + "unable to check version, please verify Databasus server is available: %w", + err, + ) + } + + if serverVersion == currentVersion { + log.Info("Agent version is up to date", "version", currentVersion) + + return false, nil + } + + log.Info("Updating agent...", "current", currentVersion, "target", serverVersion) + + selfPath, err := os.Executable() + if err != nil { + return false, fmt.Errorf("failed to determine executable path: %w", err) + } + + tempPath := selfPath + ".update" + + defer func() { + _ = os.Remove(tempPath) + }() + + downloadCtx, cancelDownload := context.WithTimeout(context.Background(), binaryDownloadTimeout) + defer cancelDownload() + + downloadedVersion, err := apiClient.DownloadVerificationAgentBinary( + downloadCtx, runtime.GOARCH, tempPath, serverVersion, + ) + if err != nil { + return false, fmt.Errorf("failed to download update: %w", err) + } + + if err := os.Chmod(tempPath, 0o755); err != nil { + return false, fmt.Errorf("failed to set permissions on update: %w", err) + } + + if err := verifyBinary(tempPath, downloadedVersion); err != nil { + return false, fmt.Errorf("update verification failed: %w", err) + } + + if downloadedVersion == currentVersion { + log.Info("Agent version is up to date", "version", currentVersion) + return false, nil + } + + if err := os.Rename(tempPath, selfPath); err != nil { + return false, fmt.Errorf("failed to replace binary (try --skip-update if this persists): %w", err) + } + + log.Info("Agent binary updated", "version", downloadedVersion) + + return true, nil +} + +func verifyBinary(binaryPath, expectedVersion string) error { + ctx, cancel := context.WithTimeout(context.Background(), verifyBinaryTimeout) + defer cancel() + + cmd := exec.CommandContext(ctx, binaryPath, "version") + + output, err := cmd.Output() + if err != nil { + return fmt.Errorf("binary failed to execute: %w", err) + } + + reportedVersion := strings.TrimSpace(string(output)) + if reportedVersion != expectedVersion { + return fmt.Errorf("version mismatch: expected %q, got %q", expectedVersion, reportedVersion) + } + + return nil +} diff --git a/agent/verification/internal/features/upgrade/upgrader_test.go b/agent/verification/internal/features/upgrade/upgrader_test.go new file mode 100644 index 0000000..69f0cc7 --- /dev/null +++ b/agent/verification/internal/features/upgrade/upgrader_test.go @@ -0,0 +1,221 @@ +package upgrade + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-verification-agent/internal/features/api" + "databasus-verification-agent/internal/testutil" +) + +func goBuildHelper(t *testing.T, mainSrc string) string { + t.Helper() + + dir := t.TempDir() + src := filepath.Join(dir, "main.go") + require.NoError(t, os.WriteFile(src, []byte(mainSrc), 0o644)) + + bin := filepath.Join(dir, "helper") + + out, err := exec.Command("go", "build", "-o", bin, src).CombinedOutput() + require.NoError(t, err, "build helper: %s", out) + + return bin +} + +func buildVersionHelper(t *testing.T, version string) string { + t.Helper() + + return goBuildHelper(t, "package main\n"+ + "import (\"fmt\"; \"os\")\n"+ + "func main(){ if len(os.Args)>1 && os.Args[1]==\"version\" { fmt.Println(\""+ + version+"\"); return }; os.Exit(2) }\n") +} + +func buildHangingHelper(t *testing.T) string { + t.Helper() + + return goBuildHelper(t, "package main\n"+ + "import \"time\"\n"+ + "func main(){ time.Sleep(30 * time.Second) }\n") +} + +func Test_VerifyBinary_WhenVersionMatches_ReturnsNil(t *testing.T) { + helper := buildVersionHelper(t, "v2.0.0") + + require.NoError(t, verifyBinary(helper, "v2.0.0")) +} + +func Test_VerifyBinary_WhenVersionMismatch_ReturnsError(t *testing.T) { + helper := buildVersionHelper(t, "v2.0.0") + + err := verifyBinary(helper, "v9.9.9") + + require.Error(t, err) + assert.Contains(t, err.Error(), "version mismatch") +} + +func Test_VerifyBinary_WhenBinaryNotExecutable_ReturnsError(t *testing.T) { + notBinary := filepath.Join(t.TempDir(), "not-a-binary") + require.NoError(t, os.WriteFile(notBinary, []byte("plain text"), 0o644)) + + require.Error(t, verifyBinary(notBinary, "v1.0.0")) +} + +func Test_VerifyBinary_WhenBinaryHangs_TimesOutInsteadOfBlocking(t *testing.T) { + original := verifyBinaryTimeout + verifyBinaryTimeout = 200 * time.Millisecond + t.Cleanup(func() { verifyBinaryTimeout = original }) + + hanging := buildHangingHelper(t) + + start := time.Now() + err := verifyBinary(hanging, "v1.0.0") + elapsed := time.Since(start) + + require.Error(t, err) + assert.Less(t, elapsed, 10*time.Second, "verifyBinary must abort on timeout, not block") + assert.NotContains(t, err.Error(), "version mismatch") +} + +func Test_CheckAndUpdate_WhenDevelopmentMode_DoesNotContactServer(t *testing.T) { + var hits atomic.Int32 + + server := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { + hits.Add(1) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "", "", testutil.DiscardLogger()) + + upgraded, err := CheckAndUpdate(client, "v1.0.0", true, testutil.DiscardLogger()) + + require.NoError(t, err) + assert.False(t, upgraded) + assert.Equal(t, int32(0), hits.Load()) +} + +func Test_CheckAndUpdate_WhenServerVersionMatches_DoesNotDownload(t *testing.T) { + var downloadHits atomic.Int32 + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/system/version" { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"version": "v1.0.0"}) + return + } + + downloadHits.Add(1) + })) + t.Cleanup(server.Close) + + client := api.NewClient(server.URL, "", "", testutil.DiscardLogger()) + + upgraded, err := CheckAndUpdate(client, "v1.0.0", false, testutil.DiscardLogger()) + + require.NoError(t, err) + assert.False(t, upgraded) + assert.Equal(t, int32(0), downloadHits.Load()) +} + +type fakeBackend struct { + versionEndpointReports string + downloadHeaderReports string + servedBinaryPrints string +} + +// Both callers below stop before CheckAndUpdate's os.Rename, which would +// otherwise overwrite the running test binary. +func startFakeBackend(t *testing.T, backend fakeBackend) *httptest.Server { + t.Helper() + + binary, err := os.ReadFile(buildVersionHelper(t, backend.servedBinaryPrints)) + require.NoError(t, err) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/system/version" { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"version": backend.versionEndpointReports}) + + return + } + + w.Header().Set("X-Databasus-Version", backend.downloadHeaderReports) + _, _ = w.Write(binary) + })) + t.Cleanup(server.Close) + + return server +} + +func Test_CheckAndUpdate_WhenServerVersionIsStaleButBinaryMatchesCurrent_SkipsUpgrade(t *testing.T) { + server := startFakeBackend(t, fakeBackend{ + versionEndpointReports: "v3.43.0", + downloadHeaderReports: "v3.45.0", + servedBinaryPrints: "v3.45.0", + }) + client := api.NewClient(server.URL, "", "", testutil.DiscardLogger()) + + upgraded, err := CheckAndUpdate(client, "v3.45.0", false, testutil.DiscardLogger()) + + require.NoError(t, err) + assert.False(t, upgraded, "a stale /system/version must not make the agent re-exec into itself") +} + +func Test_CheckAndUpdate_WhenDownloadedBinaryDisagreesWithVersionHeader_ReturnsMismatchError(t *testing.T) { + server := startFakeBackend(t, fakeBackend{ + versionEndpointReports: "v3.46.0", + downloadHeaderReports: "v3.46.0", + servedBinaryPrints: "v3.44.0", + }) + client := api.NewClient(server.URL, "", "", testutil.DiscardLogger()) + + upgraded, err := CheckAndUpdate(client, "v3.44.0", false, testutil.DiscardLogger()) + + require.Error(t, err) + assert.False(t, upgraded) + assert.Contains(t, err.Error(), "version mismatch") +} + +func Test_CheckAndUpdate_WhenServerUnreachable_ReturnsError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) {})) + server.Close() + + client := api.NewClient(server.URL, "", "", testutil.DiscardLogger()) + + upgraded, err := CheckAndUpdate(client, "v1.0.0", false, testutil.DiscardLogger()) + + require.Error(t, err) + assert.False(t, upgraded) + assert.Contains(t, err.Error(), "unable to check version") +} + +func Test_BackgroundUpgrader_WhenDevelopmentMode_NeverUpgrades(t *testing.T) { + client := api.NewClient("http://127.0.0.1:0", "", "", testutil.DiscardLogger()) + upgrader := NewBackgroundUpgrader(client, "v1.0.0", true, func() {}, testutil.DiscardLogger()) + + assert.False(t, upgrader.checkAndUpgrade()) + assert.False(t, upgrader.IsUpgraded()) +} + +func Test_BackgroundUpgrader_WhenRunCalledTwice_Panics(t *testing.T) { + client := api.NewClient("http://127.0.0.1:0", "", "", testutil.DiscardLogger()) + ctx, cancel := context.WithCancel(t.Context()) + upgrader := NewBackgroundUpgrader(client, "v1.0.0", true, cancel, testutil.DiscardLogger()) + + cancel() + upgrader.Run(ctx) + + assert.Panics(t, func() { upgrader.Run(ctx) }) +} diff --git a/agent/verification/internal/features/verifier/verifier.go b/agent/verification/internal/features/verifier/verifier.go new file mode 100644 index 0000000..8ed62d4 --- /dev/null +++ b/agent/verification/internal/features/verifier/verifier.go @@ -0,0 +1,194 @@ +// Package verifier runs the post-restore check tiers against the restored +// container over the host-published port. It returns stats and an error; it +// does NOT classify the failure — the runner decides (a dead connection is +// agent infra, a reachable DB with an impossible tier-1 result is a bad +// backup). +package verifier + +import ( + "context" + "fmt" + "log/slog" + "time" + + "github.com/jackc/pgx/v5" + + "databasus-verification-agent/internal/features/dbconn" +) + +const ( + queryTimeout = 30 * time.Second + probeTimeout = 5 * time.Second + analyzeTimeoutMin = 60 * time.Second + analyzeTimeoutMax = 10 * time.Minute + + excludedSchemas = "'pg_catalog','information_schema','pg_toast'" + // information_schema.schemata on PG 12 exposes per-session pg_temp_N and + // pg_toast_temp_N namespaces that PG 13+ hides; the regex filters them so + // SchemaCount / TableCount / tableStats stay consistent across majors. + excludedSchemaRegex = `^pg_(temp|toast_temp)_` + + dbSizeSQL = `SELECT pg_database_size(current_database())` + schemaCountSQL = `SELECT count(*) FROM information_schema.schemata ` + + `WHERE schema_name NOT IN (` + excludedSchemas + `) ` + + `AND schema_name !~ '` + excludedSchemaRegex + `'` + tableCountSQL = `SELECT count(*) FROM information_schema.tables ` + + `WHERE table_schema NOT IN (` + excludedSchemas + `) ` + + `AND table_schema !~ '` + excludedSchemaRegex + `' ` + + `AND table_type = 'BASE TABLE'` + tableStatsSQL = `SELECT n.nspname, c.relname, c.reltuples::bigint ` + + `FROM pg_class c JOIN pg_namespace n ON c.relnamespace = n.oid ` + + `WHERE c.relkind = 'r' ` + + `AND n.nspname NOT IN (` + excludedSchemas + `) ` + + `AND n.nspname !~ '` + excludedSchemaRegex + `' ` + + `ORDER BY c.reltuples DESC LIMIT 100` + fkCountSQL = `SELECT count(*) FROM information_schema.referential_constraints` +) + +type TableStat struct { + SchemaName string + Name string + RowCount int64 +} + +type Stats struct { + DBSizeBytes int64 + SchemaCount int + TableCount int + TableStats []TableStat + FKCount int +} + +type Verifier struct { + analyzeTimeoutMax time.Duration + log *slog.Logger +} + +func NewVerifier(log *slog.Logger) *Verifier { + return &Verifier{analyzeTimeoutMax: analyzeTimeoutMax, log: log} +} + +func (v *Verifier) CollectStats(ctx context.Context, conn dbconn.Conn) (Stats, error) { + pgConn, err := pgx.Connect(ctx, conn.DSN()) + if err != nil { + return Stats{}, fmt.Errorf("connect restored db: %w", err) + } + defer func() { _ = pgConn.Close(ctx) }() + + var stats Stats + + if err := collectRequiredStat(ctx, pgConn, dbSizeSQL, &stats.DBSizeBytes); err != nil { + return Stats{}, fmt.Errorf("tier 1 database size: %w", err) + } + + collectOptionalStat(ctx, v.log, pgConn, schemaCountSQL, &stats.SchemaCount, "tier 2 schema count unavailable") + collectOptionalStat(ctx, v.log, pgConn, tableCountSQL, &stats.TableCount, "tier 3 table count unavailable") + + stats.TableStats = v.collectTableStats(ctx, pgConn, stats.DBSizeBytes) + + collectOptionalStat(ctx, v.log, pgConn, fkCountSQL, &stats.FKCount, "tier 5 fk metadata unavailable") + + return stats, nil +} + +// Best-effort: any failure yields nil, never an error — a large DB whose +// ANALYZE exceeds the budget is expected, not a verification failure. +func (v *Verifier) collectTableStats( + ctx context.Context, + pgConn *pgx.Conn, + dbSizeBytes int64, +) []TableStat { + analyzeCtx, cancel := context.WithTimeout( + ctx, computeAnalyzeTimeout(dbSizeBytes, v.analyzeTimeoutMax)) + defer cancel() + + if _, err := pgConn.Exec(analyzeCtx, "ANALYZE"); err != nil { + v.log.Warn("tier 4 skipped: ANALYZE failed or timed out", "error", err) + return nil + } + + queryCtx, cancel := context.WithTimeout(ctx, queryTimeout) + defer cancel() + + rows, err := pgConn.Query(queryCtx, tableStatsSQL) + if err != nil { + v.log.Warn("tier 4 table stats unavailable", "error", err) + return nil + } + defer rows.Close() + + var tableStats []TableStat + for rows.Next() { + var stat TableStat + if err := rows.Scan(&stat.SchemaName, &stat.Name, &stat.RowCount); err != nil { + v.log.Warn("tier 4 row scan failed", "error", err) + return nil + } + + tableStats = append(tableStats, stat) + } + + if err := rows.Err(); err != nil { + v.log.Warn("tier 4 row iteration failed", "error", err) + return nil + } + + return tableStats +} + +// ProbeConnAlive is the runner's disambiguator: a failed tier with a dead +// connection is agent infra (retry), a failed tier on a live connection is a +// broken restored DB (terminal). +func ProbeConnAlive(ctx context.Context, conn dbconn.Conn) bool { + probeCtx, cancel := context.WithTimeout(ctx, probeTimeout) + defer cancel() + + pgConn, err := pgx.Connect(probeCtx, conn.DSN()) + if err != nil { + return false + } + defer func() { _ = pgConn.Close(probeCtx) }() + + return pgConn.Ping(probeCtx) == nil +} + +func collectRequiredStat[T any](ctx context.Context, pgConn *pgx.Conn, sql string, dest *T) error { + queryCtx, cancel := context.WithTimeout(ctx, queryTimeout) + defer cancel() + + return pgConn.QueryRow(queryCtx, sql).Scan(dest) +} + +func collectOptionalStat[T any]( + ctx context.Context, + log *slog.Logger, + pgConn *pgx.Conn, + sql string, + dest *T, + label string, +) { + if err := collectRequiredStat(ctx, pgConn, sql, dest); err != nil { + log.Warn(label, "error", err) + } +} + +// computeAnalyzeTimeout scales the ANALYZE budget with database size: a flat +// 30s makes tier-4 stats reliably absent on exactly the large databases they +// matter for. +func computeAnalyzeTimeout(dbSizeBytes int64, maxTimeout time.Duration) time.Duration { + const bytesPerGB = 1024 * 1024 * 1024 + const perGB = 30 * time.Second + + gb := float64(dbSizeBytes) / bytesPerGB + + scaled := analyzeTimeoutMin + time.Duration(gb*float64(perGB)) + if scaled < analyzeTimeoutMin { + return analyzeTimeoutMin + } + + if scaled > maxTimeout { + return maxTimeout + } + + return scaled +} diff --git a/agent/verification/internal/features/verifier/verifier_test.go b/agent/verification/internal/features/verifier/verifier_test.go new file mode 100644 index 0000000..60a466a --- /dev/null +++ b/agent/verification/internal/features/verifier/verifier_test.go @@ -0,0 +1,38 @@ +package verifier + +import ( + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +func Test_ComputeAnalyzeTimeout_ScalesWithSizeWithinFloorAndCap(t *testing.T) { + maxTimeout := 10 * time.Minute + + // Empty DB → exactly the 60s floor; a tiny DB stays close to it. + assert.Equal(t, analyzeTimeoutMin, computeAnalyzeTimeout(0, maxTimeout)) + tiny := computeAnalyzeTimeout(50*1024*1024, maxTimeout) + assert.GreaterOrEqual(t, tiny, analyzeTimeoutMin) + assert.Less(t, tiny, analyzeTimeoutMin+2*time.Second) + + // 10 GB → 60s + 10*30s = 360s, below the cap. + assert.Equal(t, 360*time.Second, computeAnalyzeTimeout(10*1024*1024*1024, maxTimeout)) + + // Huge DB → capped at maxTimeout, never unbounded. + assert.Equal(t, maxTimeout, computeAnalyzeTimeout(500*1024*1024*1024, maxTimeout)) +} + +func Test_TierQueries_ExcludeSystemSchemas(t *testing.T) { + for _, sql := range []string{schemaCountSQL, tableCountSQL, tableStatsSQL} { + assert.Contains(t, sql, "pg_catalog") + assert.Contains(t, sql, "information_schema") + assert.Contains(t, sql, "pg_toast") + assert.Contains(t, sql, excludedSchemaRegex) + } + + assert.True(t, strings.HasPrefix(dbSizeSQL, "SELECT pg_database_size")) + assert.Contains(t, tableCountSQL, "BASE TABLE") + assert.Contains(t, tableStatsSQL, "LIMIT 100") +} diff --git a/agent/verification/internal/logger/logger.go b/agent/verification/internal/logger/logger.go new file mode 100644 index 0000000..253fcc5 --- /dev/null +++ b/agent/verification/internal/logger/logger.go @@ -0,0 +1,115 @@ +package logger + +import ( + "fmt" + "io" + "log/slog" + "os" + "sync" + "time" +) + +const ( + logFileName = "databasus-verification.log" + oldLogFileName = "databasus-verification.log.old" + maxLogFileSize = 5 * 1024 * 1024 // 5MB +) + +type rotatingWriter struct { + mu sync.Mutex + file *os.File + currentSize int64 + maxSize int64 + logPath string + oldLogPath string +} + +func (w *rotatingWriter) Write(p []byte) (int, error) { + w.mu.Lock() + defer w.mu.Unlock() + + if w.currentSize+int64(len(p)) > w.maxSize { + if err := w.rotate(); err != nil { + return 0, fmt.Errorf("failed to rotate log file: %w", err) + } + } + + n, err := w.file.Write(p) + w.currentSize += int64(n) + + return n, err +} + +func (w *rotatingWriter) rotate() error { + if err := w.file.Close(); err != nil { + return fmt.Errorf("failed to close %s: %w", w.logPath, err) + } + + if err := os.Remove(w.oldLogPath); err != nil && !os.IsNotExist(err) { + return fmt.Errorf("failed to remove %s: %w", w.oldLogPath, err) + } + + if err := os.Rename(w.logPath, w.oldLogPath); err != nil { + return fmt.Errorf("failed to rename %s to %s: %w", w.logPath, w.oldLogPath, err) + } + + f, err := os.OpenFile(w.logPath, os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return fmt.Errorf("failed to create new %s: %w", w.logPath, err) + } + + w.file = f + w.currentSize = 0 + + return nil +} + +var loggerInstance *slog.Logger + +var initLogger = sync.OnceFunc(initialize) + +func GetLogger() *slog.Logger { + initLogger() + return loggerInstance +} + +func initialize() { + writer := buildWriter() + + loggerInstance = slog.New(slog.NewTextHandler(writer, &slog.HandlerOptions{ + Level: slog.LevelInfo, + ReplaceAttr: func(groups []string, a slog.Attr) slog.Attr { + if a.Key == slog.TimeKey { + a.Value = slog.StringValue(time.Now().Format("2006/01/02 15:04:05")) + } + if a.Key == slog.LevelKey { + return slog.Attr{} + } + + return a + }, + })) +} + +func buildWriter() io.Writer { + f, err := os.OpenFile(logFileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to open %s for logging: %v\n", logFileName, err) + return os.Stdout + } + + var currentSize int64 + if info, err := f.Stat(); err == nil { + currentSize = info.Size() + } + + rw := &rotatingWriter{ + file: f, + currentSize: currentSize, + maxSize: maxLogFileSize, + logPath: logFileName, + oldLogPath: oldLogFileName, + } + + return io.MultiWriter(os.Stdout, rw) +} diff --git a/agent/verification/internal/testutil/testutil.go b/agent/verification/internal/testutil/testutil.go new file mode 100644 index 0000000..0a9b20b --- /dev/null +++ b/agent/verification/internal/testutil/testutil.go @@ -0,0 +1,11 @@ +// Package testutil holds helpers shared across the verification agent's +// package tests. +package testutil + +import "log/slog" + +// DiscardLogger returns a no-op logger so tests don't write to the rotating +// log file or stdout. +func DiscardLogger() *slog.Logger { + return slog.New(slog.DiscardHandler) +} diff --git a/agent/verification/mock-server.exe b/agent/verification/mock-server.exe new file mode 100644 index 0000000..e9ad0d4 Binary files /dev/null and b/agent/verification/mock-server.exe differ diff --git a/assets/dashboard-dark.svg b/assets/dashboard-dark.svg new file mode 100644 index 0000000..90abdff --- /dev/null +++ b/assets/dashboard-dark.svg @@ -0,0 +1,768 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/dashboard.svg b/assets/dashboard.svg new file mode 100644 index 0000000..14c5d84 --- /dev/null +++ b/assets/dashboard.svg @@ -0,0 +1,834 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/healthchecks.svg b/assets/healthchecks.svg new file mode 100644 index 0000000..6dd3037 --- /dev/null +++ b/assets/healthchecks.svg @@ -0,0 +1,335 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/assets/logo-square.png b/assets/logo-square.png new file mode 100644 index 0000000..adf63ba Binary files /dev/null and b/assets/logo-square.png differ diff --git a/assets/logo-square.svg b/assets/logo-square.svg new file mode 100644 index 0000000..b41cdd4 --- /dev/null +++ b/assets/logo-square.svg @@ -0,0 +1,12 @@ + + + + + + + + + + + + diff --git a/assets/logo.svg b/assets/logo.svg new file mode 100644 index 0000000..40e1b17 --- /dev/null +++ b/assets/logo.svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/assets/tools/README.md b/assets/tools/README.md new file mode 100644 index 0000000..3daf0a7 --- /dev/null +++ b/assets/tools/README.md @@ -0,0 +1,34 @@ +Pre-built DB client binaries committed to the repo so that local dev, +CI, and the Docker image all read from one place. The Go backend +resolves them at runtime via `runtime.GOOS`+`runtime.GOARCH` → +`assets/tools///-/bin/`. + +Layout (one subtree per arch, identical shape): + +``` +assets/tools// + postgresql/postgresql-{12,13,14,15,16,17,18}/bin/ + pg_dump, pg_restore, psql + mysql/mysql-{5.7,8.0,8.4,9}/bin/ + mysql, mysqldump + mariadb/mariadb-{10.6,12.1}/bin/ + mariadb, mariadb-dump + mongodb/bin/ + mongodump, mongorestore +``` + +`` keys (mapping in `backend/internal/util/tools/paths.go`): + +| GOOS / GOARCH | key | size | +|-------------------|-------|---------| +| `linux` / `amd64` | `x64` | ~160 MB | +| `linux` / `arm64` | `arm` | ~125 MB | + +Notes: +- MySQL `5.7` is amd64-only — `arm/mysql/mysql-5.7/` is intentionally absent. +- MariaDB ships two client versions: legacy `10.6` (for MariaDB servers 5.5 / 10.1) and modern `12.1` (for 10.2+). The mapping lives in `tools/mariadb.go`. +- MongoDB Database Tools are version 100.16.1 across all arches and are backward-compatible with all supported server versions (4.2 – 8.2). MongoDB 4.0 is not supported (wire version 7, requires older mongodump). + +To refresh a tool set, drop the corresponding `bin/` contents in place +and commit. There are no install scripts in this directory; binaries +are sourced from the upstream vendor downloads. diff --git a/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb b/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb new file mode 100755 index 0000000..eefb2bf Binary files /dev/null and b/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb differ diff --git a/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb-dump b/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb-dump new file mode 100755 index 0000000..6d8f4f6 Binary files /dev/null and b/assets/tools/arm/mariadb/mariadb-10.6/bin/mariadb-dump differ diff --git a/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb b/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb new file mode 100755 index 0000000..eef87e9 Binary files /dev/null and b/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb differ diff --git a/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb-dump b/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb-dump new file mode 100755 index 0000000..5d36ed5 Binary files /dev/null and b/assets/tools/arm/mariadb/mariadb-12.1/bin/mariadb-dump differ diff --git a/assets/tools/arm/mongodb/bin/mongodump b/assets/tools/arm/mongodb/bin/mongodump new file mode 100755 index 0000000..b0bb79d Binary files /dev/null and b/assets/tools/arm/mongodb/bin/mongodump differ diff --git a/assets/tools/arm/mongodb/bin/mongorestore b/assets/tools/arm/mongodb/bin/mongorestore new file mode 100755 index 0000000..6e9e859 Binary files /dev/null and b/assets/tools/arm/mongodb/bin/mongorestore differ diff --git a/assets/tools/arm/mysql/mysql-8.0/bin/mysql b/assets/tools/arm/mysql/mysql-8.0/bin/mysql new file mode 100755 index 0000000..9b3ace0 Binary files /dev/null and b/assets/tools/arm/mysql/mysql-8.0/bin/mysql differ diff --git a/assets/tools/arm/mysql/mysql-8.0/bin/mysqldump b/assets/tools/arm/mysql/mysql-8.0/bin/mysqldump new file mode 100755 index 0000000..c14203b Binary files /dev/null and b/assets/tools/arm/mysql/mysql-8.0/bin/mysqldump differ diff --git a/assets/tools/arm/mysql/mysql-8.4/bin/mysql b/assets/tools/arm/mysql/mysql-8.4/bin/mysql new file mode 100755 index 0000000..ce6bdfd Binary files /dev/null and b/assets/tools/arm/mysql/mysql-8.4/bin/mysql differ diff --git a/assets/tools/arm/mysql/mysql-8.4/bin/mysqldump b/assets/tools/arm/mysql/mysql-8.4/bin/mysqldump new file mode 100755 index 0000000..f947a62 Binary files /dev/null and b/assets/tools/arm/mysql/mysql-8.4/bin/mysqldump differ diff --git a/assets/tools/arm/mysql/mysql-9/bin/mysql b/assets/tools/arm/mysql/mysql-9/bin/mysql new file mode 100755 index 0000000..e99d3d3 Binary files /dev/null and b/assets/tools/arm/mysql/mysql-9/bin/mysql differ diff --git a/assets/tools/arm/mysql/mysql-9/bin/mysqldump b/assets/tools/arm/mysql/mysql-9/bin/mysqldump new file mode 100755 index 0000000..5cea950 Binary files /dev/null and b/assets/tools/arm/mysql/mysql-9/bin/mysqldump differ diff --git a/assets/tools/arm/postgresql/postgresql-12/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-12/bin/pg_dump new file mode 100755 index 0000000..c4951e2 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-12/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-12/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-12/bin/pg_restore new file mode 100755 index 0000000..d792101 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-12/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-12/bin/psql b/assets/tools/arm/postgresql/postgresql-12/bin/psql new file mode 100755 index 0000000..bbf28d4 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-12/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-13/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-13/bin/pg_dump new file mode 100755 index 0000000..60118c7 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-13/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-13/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-13/bin/pg_restore new file mode 100755 index 0000000..3b4fb8c Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-13/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-13/bin/psql b/assets/tools/arm/postgresql/postgresql-13/bin/psql new file mode 100755 index 0000000..377be0a Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-13/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-14/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-14/bin/pg_dump new file mode 100755 index 0000000..7c6bcad Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-14/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-14/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-14/bin/pg_restore new file mode 100755 index 0000000..6df17ae Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-14/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-14/bin/psql b/assets/tools/arm/postgresql/postgresql-14/bin/psql new file mode 100755 index 0000000..8616815 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-14/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-15/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-15/bin/pg_dump new file mode 100755 index 0000000..8eecace Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-15/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-15/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-15/bin/pg_restore new file mode 100755 index 0000000..6ad5e4b Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-15/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-15/bin/psql b/assets/tools/arm/postgresql/postgresql-15/bin/psql new file mode 100755 index 0000000..ce11939 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-15/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-16/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-16/bin/pg_dump new file mode 100755 index 0000000..88ebd97 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-16/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-16/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-16/bin/pg_restore new file mode 100755 index 0000000..eee24bb Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-16/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-16/bin/psql b/assets/tools/arm/postgresql/postgresql-16/bin/psql new file mode 100755 index 0000000..e96c287 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-16/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/pg_basebackup b/assets/tools/arm/postgresql/postgresql-17/bin/pg_basebackup new file mode 100755 index 0000000..dba5c47 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/pg_basebackup differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/pg_combinebackup b/assets/tools/arm/postgresql/postgresql-17/bin/pg_combinebackup new file mode 100755 index 0000000..bbf9994 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/pg_combinebackup differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-17/bin/pg_dump new file mode 100755 index 0000000..2eb8f0e Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/pg_receivewal b/assets/tools/arm/postgresql/postgresql-17/bin/pg_receivewal new file mode 100755 index 0000000..f252b36 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/pg_receivewal differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-17/bin/pg_restore new file mode 100755 index 0000000..87a71cc Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-17/bin/psql b/assets/tools/arm/postgresql/postgresql-17/bin/psql new file mode 100755 index 0000000..d981a9c Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-17/bin/psql differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/pg_basebackup b/assets/tools/arm/postgresql/postgresql-18/bin/pg_basebackup new file mode 100755 index 0000000..89352fa Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/pg_basebackup differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/pg_combinebackup b/assets/tools/arm/postgresql/postgresql-18/bin/pg_combinebackup new file mode 100755 index 0000000..275bc13 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/pg_combinebackup differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/pg_dump b/assets/tools/arm/postgresql/postgresql-18/bin/pg_dump new file mode 100755 index 0000000..e770af7 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/pg_dump differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/pg_receivewal b/assets/tools/arm/postgresql/postgresql-18/bin/pg_receivewal new file mode 100755 index 0000000..2cf7a4b Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/pg_receivewal differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/pg_restore b/assets/tools/arm/postgresql/postgresql-18/bin/pg_restore new file mode 100755 index 0000000..d4a58fe Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/pg_restore differ diff --git a/assets/tools/arm/postgresql/postgresql-18/bin/psql b/assets/tools/arm/postgresql/postgresql-18/bin/psql new file mode 100755 index 0000000..db4c8e2 Binary files /dev/null and b/assets/tools/arm/postgresql/postgresql-18/bin/psql differ diff --git a/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb b/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb new file mode 100755 index 0000000..b790a7c Binary files /dev/null and b/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb differ diff --git a/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb-dump b/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb-dump new file mode 100755 index 0000000..f73438d Binary files /dev/null and b/assets/tools/x64/mariadb/mariadb-10.6/bin/mariadb-dump differ diff --git a/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb b/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb new file mode 100755 index 0000000..1eeaf6c Binary files /dev/null and b/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb differ diff --git a/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb-dump b/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb-dump new file mode 100755 index 0000000..c815a22 Binary files /dev/null and b/assets/tools/x64/mariadb/mariadb-12.1/bin/mariadb-dump differ diff --git a/assets/tools/x64/mongodb/bin/mongodump b/assets/tools/x64/mongodb/bin/mongodump new file mode 100755 index 0000000..9ffddbc Binary files /dev/null and b/assets/tools/x64/mongodb/bin/mongodump differ diff --git a/assets/tools/x64/mongodb/bin/mongorestore b/assets/tools/x64/mongodb/bin/mongorestore new file mode 100755 index 0000000..e9a8ddc Binary files /dev/null and b/assets/tools/x64/mongodb/bin/mongorestore differ diff --git a/assets/tools/x64/mysql/mysql-5.7/bin/mysql b/assets/tools/x64/mysql/mysql-5.7/bin/mysql new file mode 100755 index 0000000..86a7185 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-5.7/bin/mysql differ diff --git a/assets/tools/x64/mysql/mysql-5.7/bin/mysqldump b/assets/tools/x64/mysql/mysql-5.7/bin/mysqldump new file mode 100755 index 0000000..7a728b5 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-5.7/bin/mysqldump differ diff --git a/assets/tools/x64/mysql/mysql-8.0/bin/mysql b/assets/tools/x64/mysql/mysql-8.0/bin/mysql new file mode 100755 index 0000000..57c5a77 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-8.0/bin/mysql differ diff --git a/assets/tools/x64/mysql/mysql-8.0/bin/mysqldump b/assets/tools/x64/mysql/mysql-8.0/bin/mysqldump new file mode 100755 index 0000000..f193675 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-8.0/bin/mysqldump differ diff --git a/assets/tools/x64/mysql/mysql-8.4/bin/mysql b/assets/tools/x64/mysql/mysql-8.4/bin/mysql new file mode 100755 index 0000000..b8d0bcd Binary files /dev/null and b/assets/tools/x64/mysql/mysql-8.4/bin/mysql differ diff --git a/assets/tools/x64/mysql/mysql-8.4/bin/mysqldump b/assets/tools/x64/mysql/mysql-8.4/bin/mysqldump new file mode 100755 index 0000000..211fb2d Binary files /dev/null and b/assets/tools/x64/mysql/mysql-8.4/bin/mysqldump differ diff --git a/assets/tools/x64/mysql/mysql-9/bin/mysql b/assets/tools/x64/mysql/mysql-9/bin/mysql new file mode 100755 index 0000000..3afd871 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-9/bin/mysql differ diff --git a/assets/tools/x64/mysql/mysql-9/bin/mysqldump b/assets/tools/x64/mysql/mysql-9/bin/mysqldump new file mode 100755 index 0000000..3891918 Binary files /dev/null and b/assets/tools/x64/mysql/mysql-9/bin/mysqldump differ diff --git a/assets/tools/x64/postgresql/postgresql-12/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-12/bin/pg_dump new file mode 100755 index 0000000..d2016e3 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-12/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-12/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-12/bin/pg_restore new file mode 100755 index 0000000..1084f53 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-12/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-12/bin/psql b/assets/tools/x64/postgresql/postgresql-12/bin/psql new file mode 100755 index 0000000..42218f9 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-12/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-13/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-13/bin/pg_dump new file mode 100755 index 0000000..60690d5 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-13/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-13/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-13/bin/pg_restore new file mode 100755 index 0000000..f13321a Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-13/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-13/bin/psql b/assets/tools/x64/postgresql/postgresql-13/bin/psql new file mode 100755 index 0000000..c96b260 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-13/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-14/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-14/bin/pg_dump new file mode 100755 index 0000000..955b243 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-14/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-14/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-14/bin/pg_restore new file mode 100755 index 0000000..355c315 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-14/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-14/bin/psql b/assets/tools/x64/postgresql/postgresql-14/bin/psql new file mode 100755 index 0000000..4819235 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-14/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-15/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-15/bin/pg_dump new file mode 100755 index 0000000..8881dab Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-15/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-15/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-15/bin/pg_restore new file mode 100755 index 0000000..cbd44ba Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-15/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-15/bin/psql b/assets/tools/x64/postgresql/postgresql-15/bin/psql new file mode 100755 index 0000000..260408d Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-15/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-16/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-16/bin/pg_dump new file mode 100755 index 0000000..8d20464 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-16/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-16/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-16/bin/pg_restore new file mode 100755 index 0000000..929e15a Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-16/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-16/bin/psql b/assets/tools/x64/postgresql/postgresql-16/bin/psql new file mode 100755 index 0000000..3855edd Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-16/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/pg_basebackup b/assets/tools/x64/postgresql/postgresql-17/bin/pg_basebackup new file mode 100755 index 0000000..ad5a8bf Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/pg_basebackup differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/pg_combinebackup b/assets/tools/x64/postgresql/postgresql-17/bin/pg_combinebackup new file mode 100755 index 0000000..f6f8b72 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/pg_combinebackup differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-17/bin/pg_dump new file mode 100755 index 0000000..51a235d Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/pg_receivewal b/assets/tools/x64/postgresql/postgresql-17/bin/pg_receivewal new file mode 100755 index 0000000..fcfefbd Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/pg_receivewal differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-17/bin/pg_restore new file mode 100755 index 0000000..c7c75c3 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-17/bin/psql b/assets/tools/x64/postgresql/postgresql-17/bin/psql new file mode 100755 index 0000000..01b099e Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-17/bin/psql differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/pg_basebackup b/assets/tools/x64/postgresql/postgresql-18/bin/pg_basebackup new file mode 100755 index 0000000..56d2c91 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/pg_basebackup differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/pg_combinebackup b/assets/tools/x64/postgresql/postgresql-18/bin/pg_combinebackup new file mode 100755 index 0000000..ca3df21 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/pg_combinebackup differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/pg_dump b/assets/tools/x64/postgresql/postgresql-18/bin/pg_dump new file mode 100755 index 0000000..0fa75a3 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/pg_dump differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/pg_receivewal b/assets/tools/x64/postgresql/postgresql-18/bin/pg_receivewal new file mode 100755 index 0000000..2d20243 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/pg_receivewal differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/pg_restore b/assets/tools/x64/postgresql/postgresql-18/bin/pg_restore new file mode 100755 index 0000000..6e041e7 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/pg_restore differ diff --git a/assets/tools/x64/postgresql/postgresql-18/bin/psql b/assets/tools/x64/postgresql/postgresql-18/bin/psql new file mode 100755 index 0000000..8460c00 Binary files /dev/null and b/assets/tools/x64/postgresql/postgresql-18/bin/psql differ diff --git a/backend/.gitignore b/backend/.gitignore new file mode 100644 index 0000000..1537030 --- /dev/null +++ b/backend/.gitignore @@ -0,0 +1,12 @@ +main +.env +pgdata_test/ +swagger/ +swagger/* +swagger/docs.go +swagger/swagger.json +swagger/swagger.yaml +ui/build/* +pgdata-for-restore/ +temp/ +temp/ diff --git a/backend/.golangci.yml b/backend/.golangci.yml new file mode 100644 index 0000000..b248eeb --- /dev/null +++ b/backend/.golangci.yml @@ -0,0 +1,41 @@ +version: "2" + +run: + timeout: 5m + tests: false + concurrency: 4 + +linters: + default: standard + enable: + - funcorder + - bodyclose + - errorlint + - gocritic + - unconvert + - misspell + - errname + - noctx + - modernize + + settings: + errcheck: + check-type-assertions: true + +formatters: + enable: + - gofumpt + - golines + - gci + + settings: + golines: + max-len: 120 + gofumpt: + module-path: databasus-backend + extra-rules: true + gci: + sections: + - standard + - default + - localmodule diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md new file mode 100644 index 0000000..b48fc77 --- /dev/null +++ b/backend/CLAUDE.md @@ -0,0 +1,678 @@ +# Backend guidelines (Go + Gin + GORM) + +Coding standards for the Databasus backend (Go + Gin + GORM + PostgreSQL). +For project-wide engineering philosophy, naming, and lint/format commands, see the root `CLAUDE.md`. + +--- + +## Table of Contents + +- [Spacing between logical statements](#spacing-between-logical-statements) +- [Comments](#comments) +- [File organization](#file-organization) +- [Controllers](#controllers) +- [Dependency injection (DI)](#dependency-injection-di) +- [Background services](#background-services) +- [Migrations](#migrations) +- [Testing](#testing) +- [Time handling](#time-handling) +- [Logging](#logging) +- [CRUD examples](#crud-examples) +- [Modern Go](#modern-go) + +--- + +## Spacing between logical statements + +Add blank lines between logical blocks so the flow is visible at a glance: + +- before the final `return` +- after variable declarations, before they're used +- between error handling and subsequent logic +- between distinct logical operations + +Bad: + +```go +func (r *Repository) FindById(id uuid.UUID) (*models.Task, error) { + var task models.Task + result := storage.GetDb().Where("id = ?", id).First(&task) + if result.Error != nil { + if errors.Is(result.Error, gorm.ErrRecordNotFound) { + return nil, errors.New("task not found") + } + return nil, result.Error + } + return &task, nil +} +``` + +Good: + +```go +func (r *Repository) FindById(id uuid.UUID) (*models.Task, error) { + var task models.Task + + result := storage.GetDb().Where("id = ?", id).First(&task) + if result.Error != nil { + if errors.Is(result.Error, gorm.ErrRecordNotFound) { + return nil, errors.New("task not found") + } + + return nil, result.Error + } + + return &task, nil +} +``` + +--- + +## Comments + +**Comment only in the rare, genuinely non-obvious case.** The default is no comment. A +comment must earn its place by carrying something the code itself cannot — never by +restating, narrating, or labelling what is already visible. + +- **Almost never comment.** If you're about to write one, first try a clearer name or a + smaller function. Self-explanatory code with no comment beats commented code. +- **A comment may only explain *why*** — a business rule, a hidden cross-system constraint, + a non-obvious algorithm or optimisation, an ADR reference. Never *what* the code does. +- **Don't narrate the obvious.** No "first failing check", no "the detected platform", no + "re-open the connection", no step-by-step play-by-play. If the reader can see it, delete it. +- **A name that needs a "what" comment is a naming bug** — rename until the comment is + redundant, then delete it. +- **Swagger comments are mandatory** for every HTTP endpoint. +- **No "Summary" / "Conclusion" sections in `.md` files** unless explicitly requested. + +Bad (comments restate the code / narrate the obvious): + +```go +// Create test project +project := CreateTestProject(projectName, user, router) + +// CreateValidLogItems creates valid log items for testing +func CreateValidLogItems(count int, uniqueID string) []logs_receiving.LogItemRequestDTO { + +// the first failing check, plus the detected platform +type ConnectionDiagnostics struct { +``` + +--- + +## File organization + +One responsibility per file. Don't dump a whole package into one file — split +by role so a reader can find a type by its filename. Conventional names within +a feature package: + +- `doc.go` — package doc comment, once the package spans more than one file +- `.go` — the core type and its methods (the orchestrator/executor) +- `dto.go` — request/response and cross-package data + interface seams +- `errors.go` — sentinel errors (`var Err... = errors.New(...)`) +- `enums.go` — typed-constant groups (`type Status string` + its values) +- `constants.go` — package-level constants that aren't an enum +- background loops, reapers, and pools get their own file (`reaper.go`, `pool.go`) + +Only create a file when there is real content for it — an empty `enums.go` or +`constants.go` is noise, not structure. Test files mirror the source split: +`restorer.go` → `restorer_test.go`, `diskexhaustion.go` → +`diskexhaustion_test.go`. The CRUD layout — `controller.go`, `service.go`, +`repository.go`, `model.go`, `dto.go`, `di.go` — is the same principle applied +to a Gin/GORM feature. + +--- + +## Controllers + +- One controller per feature, combining all routes for that feature. +- Method names express **what we do** (`GetAvailableTasks`), not the suffix `Handler`. +- All routes use Gin and take `*gin.Context`. +- Every route is documented with Swagger annotations — see [CRUD examples → controller.go](#controllergo) for the canonical annotation format. + +```go +type AuditLogController struct { + auditLogService *AuditLogService +} + +func (c *AuditLogController) RegisterRoutes(router *gin.RouterGroup) { + auditRoutes := router.Group("/audit-logs") + + auditRoutes.GET("/global", c.GetGlobalAuditLogs) + auditRoutes.GET("/users/:userId", c.GetUserAuditLogs) +} +``` + +--- + +## Dependency injection (DI) + +For DI files (controllers, services, repositories, use cases — not plain data structs), use **implicit positional field declaration**. Adding a new dependency must force every callsite to update; named-field syntax silently accepts a missing field as zero. + +```go +// good — positional, fails to compile if a field is added or reordered +var orderController = &OrderController{ + orderService, + bot_users.GetBotUserService(), + bots.GetBotService(), + users.GetUserService(), +} + +// bad — named fields, silently accept new fields with zero values +var orderController = &OrderController{ + orderService: orderService, + botUserService: bot_users.GetBotUserService(), + botService: bots.GetBotService(), + userService: users.GetUserService(), +} +``` + +Apply this anywhere you see the `var fooController = &FooController{...}` + `GetFooController()` getter shape. + +### `SetupDependencies()` pattern + +`SetupDependencies()` runs cross-feature wiring (e.g. registering audit-log writers on user services) and **must be idempotent** — tests call it many times. Use `sync.OnceFunc`: + +```go +var SetupDependencies = sync.OnceFunc(func() { + users_services.GetUserService().SetAuditLogWriter(auditLogService) + users_services.GetSettingsService().SetAuditLogWriter(auditLogService) +}) +``` + +`sync.OnceFunc` (Go 1.21+) is concise and thread-safe — don't reinvent it with `sync.Once` + `atomic.Bool`. + +### Never inject another feature's repository + +A feature's repositories are its **private implementation detail**. When feature A needs data that lives under feature B, extend B's *service* with a public method and inject **that service** — not B's repository. + +The seam must live at the service layer so authorization, logging, and invariants stay auditable in one place, and so schema changes don't ripple across features. A feature injecting its **own** repository is unchanged. + +```go +// good — cross-feature data access goes through the owning service +var backupController = &BackupController{ + backupService, + databases.GetDatabaseService(), // service, not repository +} + +// bad — reaches into another feature's private layer +var backupController = &BackupController{ + backupService, + databases.GetDatabaseRepository(), // bypasses authz / logging / invariants +} +``` + +--- + +## Background services + +A background service runs an infinite loop in a goroutine. Calling `Run()` twice on the same instance is always a bug — duplicate goroutines leak resources and corrupt state. **Always panic; never just log a warning.** + +```go +type BackgroundService struct { + // ... + hasRun atomic.Bool +} + +func (s *BackgroundService) Run(ctx context.Context) { + if s.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", s)) + } + + ticker := time.NewTicker(1 * time.Minute) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + s.doWork() + } + } +} +``` + +`atomic.Bool.Swap(true)` does the check-and-set atomically — no `sync.Once` needed. Applies to schedulers, registries, worker nodes, and cleanup services. + +--- + +## Migrations + +- PostgreSQL only. +- Primary UUID keys → `gen_random_uuid()`. +- Time columns → `TIMESTAMPTZ` (timestamp with zone), never `TIMESTAMP`. +- Tables, constraints, and indexes go in **separate** declarations (table first, then each constraint / index on its own statement). +- Format SQL: align column types, put each constraint clause on its own line. + +```sql +CREATE TABLE marketplace_info ( + bot_id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + title TEXT NOT NULL, + description TEXT NOT NULL, + short_description TEXT NOT NULL, + tutorial_url TEXT, + info_order BIGINT NOT NULL DEFAULT 0, + is_published BOOLEAN NOT NULL DEFAULT FALSE +); + +ALTER TABLE marketplace_info_images + ADD CONSTRAINT fk_marketplace_info_images_bot_id + FOREIGN KEY (bot_id) + REFERENCES marketplace_info (bot_id); +``` + +For migrations stub generation use Makefile and only then fill manually. + +--- + +## Testing + +**Always run tests after writing them and verify they pass.** + +### Naming + +- `Test_WhatWeDo_WhatWeExpect` +- `Test_WhatWeDo_WhichConditions_WhatWeExpect` + +Examples: `Test_CreateApiKey_WhenUserIsProjectOwner_ApiKeyCreated`, `Test_DeleteApiKey_WhenUserIsProjectMember_ReturnsForbidden`, `Test_GetProjectAuditLogs_WithDifferentUserRoles_EnforcesPermissionsCorrectly`, `Test_ProjectLifecycleE2E_CompletesSuccessfully`. + +This convention applies to `t.Run` subtest names too. When a per-version orchestrator runs former +top-level test functions as subtests (see ADR-0013), name each subtest the full former function +name verbatim — `t.Run("Test_CreateReadOnlyUser_UserCanReadButNotWrite", ...)`, not an abbreviation. +Reserve non-`Test_` subtest names for grouping labels only: the engine-version group (`"PostgreSQL 16"`) +and per-case variants (`"CPU=4 directory"`). + +### Prefer controller tests over unit tests + +- Test through HTTP endpoints whenever possible — that's the contract real callers see. +- Avoid testing repositories or services in isolation; cover them via the API. +- Use unit tests only for complex model logic with no API surface. +- File names: `controller_test.go` or `service_test.go` — never `integration_test.go`. + +### `features/tests/` is for backup → restore cycle tests only + +The `internal/features/tests/` package is reserved for **end-to-end tests that exercise a full backup → restore cycle** against a real database container (today: `tests/logical/`; physical restore tests land here once the restore path ships). These tests cross multiple feature packages and have no single "owner" file, so they live in this shared package. + +**Everything else** — controller tests, service tests, executor / backuper tests, slot-lifecycle tests, helpers — lives next to its source per the source-split rule (`backup_slot.go` → `backup_slot_test.go`, `full.go` → `full_test.go`). Shared per-package test fixtures go in that package's `testing.go`. + +### Shared testing utilities + +Each feature creates a `testing.go` (or `testing/testing.go`) with router builders, model creation helpers, and request helpers used by other features' tests. Build creation helpers via the API (`POST /...`) — not direct DB inserts — so the helpers double as a sanity check on the create endpoint. + +### Refactor tests as you touch them + +When editing existing tests, look for: repetitive setup that should become a helper, oversized tests that should be split, inline test data that should reuse a helper, and similar patterns across files that should be consolidated. + +### Clean up test data + +If the feature has a DELETE endpoint or cleanup method, **use it** in the test (`defer` or `t.Cleanup`). Prefer API cleanup over direct DB delete. Skip cleanup only when the test runs in an auto-rollback transaction, the cleanup endpoint doesn't exist yet, or the test explicitly validates a failure path where cleanup isn't possible. + +### Canonical controller test + +```go +func Test_CreateApiKey_WhenUserIsProjectOwner_ApiKeyCreated(t *testing.T) { + router := CreateApiKeyTestRouter(GetProjectController(), GetMembershipController()) + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + project, _ := projects_testing.CreateTestProjectViaAPI("Test Project", owner, router) + + request := CreateApiKeyRequestDTO{Name: "Test API Key"} + var response ApiKey + test_utils.MakePostRequestAndUnmarshal(t, router, + "/api/v1/projects/api-keys/"+project.ID.String(), + "Bearer "+owner.Token, request, http.StatusOK, &response) + + assert.Equal(t, "Test API Key", response.Name) + assert.NotEmpty(t, response.Token) +} +``` + +The same shape extends to: +- **Permission-failure tests** — assert `http.StatusForbidden` plus the error message body. +- **Cross-tenant isolation tests** — create a second user/project, attempt to access the first via the second's endpoint, assert `http.StatusBadRequest`. +- **E2E lifecycle tests** — chain create → mutate → transfer → verify, all via API. + +--- + +## Time handling + +Always use `time.Now().UTC()` instead of `time.Now()` to keep timezones consistent across the application. + +--- + +## Logging + +We use `log/slog`. Three rules. + +### 1. Scope IDs early via `logger.With(...)` + +Attach `database_id`, `backup_id`, `verification_id`, etc. as soon as you know them so every downstream line carries them automatically. + +```go +func (s *BackupService) CreateBackup(logger *slog.Logger, databaseID uuid.UUID) { + logger = logger.With("database_id", databaseID) + + logger.Debug("creating backup") + // every subsequent log automatically carries database_id +} +``` + +For background services, also scope `task_name` per subtask in `Run()`. Inside loops, scope further with the loop's identifying ID. + +### 2. Values in message, IDs and errors as kv pairs + +Sizes, counts, and status transitions go into the message via `fmt.Sprintf`. IDs and errors stay as structured kv pairs so they're searchable in log aggregation tools. + +```go +// good +logger.Info(fmt.Sprintf("subscription renewed: %s -> %s, %d GB", oldStatus, newStatus, sub.StorageGB)) +logger.Info("deleted old backup", "backup_id", backup.ID) +logger.Error("failed to save subscription", "error", err) + +// bad — ID buried in the message string, error formatted instead of attached +logger.Info(fmt.Sprintf("deleted old backup %s", backup.ID)) +logger.Error(fmt.Sprintf("failed to save subscription: %v", err)) +``` + +### 3. Style and level + +- All keys `snake_case` (`database_id`, `total_size_mb`) — never camelCase. +- Messages start lowercase, no trailing period. +- **Debug**: routine ops, function entry, query result counts. +- **Info**: significant state changes, completed actions. +- **Warn**: degraded but recoverable. +- **Error**: failures that need attention. + +### Required contextual fields + +Every log line must carry enough to reconstruct one entity's history: + +- **`request_id`** — auto-generated per HTTP request by Gin middleware and echoed back in the response. Attached to every log inside an HTTP request; do not add it manually. +- **`user_id`, `database_id`, `backup_id`, `verification_id`, etc.** — attach via `logger.With(...)` at the boundary where the entity becomes known so downstream logs inherit it. +- **Background jobs / schedulers** (no HTTP request, no `request_id`): pass these inline on every log call: + - **`job_id`** — fresh UUID per execution; the correlation ID for one run. + - **`job_name`** — stable snake_case identifier of which job this is (e.g. `"backup_retention_cleanup"`, `"audit_log_cleanup"`). Define it as a `const` next to the service — never the struct's type name (renames break log queries). Lets you query "all runs of job X" or "all errors from job Y" across history. + +```go +const jobName = "backup_retention_cleanup" + +func (c *BackupCleaner) Run(ctx context.Context) { + jobID := uuid.New() + logger := c.logger.With("job_id", jobID, "job_name", jobName) + + logger.Info("job started") + // ... +} +``` + +### What never goes into logs + +- **Passwords, tokens, API keys, full `Authorization` / `Cookie` headers** — centralize redaction at the logger config; don't redact ad-hoc at call sites. +- **PII (email, phone)** — mask before logging (`r***@example.com`, `+7***1234`). +- **Full request / response bodies** — log only the fields you actually need, never whole payloads. + +--- + +## CRUD examples + +A complete feature has these files. The controller carries the canonical Swagger annotation format — that's the part to memorize. + +### controller.go + +```go +package audit_logs + +type AuditLogController struct { + auditLogService *AuditLogService +} + +func (c *AuditLogController) RegisterRoutes(router *gin.RouterGroup) { + auditRoutes := router.Group("/audit-logs") + + auditRoutes.GET("/global", c.GetGlobalAuditLogs) + auditRoutes.GET("/users/:userId", c.GetUserAuditLogs) +} + +// GetGlobalAuditLogs +// @Summary Get global audit logs (ADMIN only) +// @Description Retrieve all audit logs across the system +// @Tags audit-logs +// @Accept json +// @Produce json +// @Security BearerAuth +// @Param limit query int false "Limit number of results" default(100) +// @Param offset query int false "Offset for pagination" default(0) +// @Param beforeDate query string false "Filter logs created before this date (RFC3339 format)" format(date-time) +// @Success 200 {object} GetAuditLogsResponse +// @Failure 401 {object} map[string]string +// @Failure 403 {object} map[string]string +// @Router /audit-logs/global [get] +func (c *AuditLogController) GetGlobalAuditLogs(ctx *gin.Context) { + user, isOk := ctx.MustGet("user").(*user_models.User) + if !isOk { + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid user type in context"}) + return + } + + request := &GetAuditLogsRequest{} + if err := ctx.ShouldBindQuery(request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid query parameters"}) + return + } + + response, err := c.auditLogService.GetGlobalAuditLogs(user, request) + if err != nil { + if err.Error() == "only administrators can view global audit logs" { + ctx.JSON(http.StatusForbidden, gin.H{"error": err.Error()}) + return + } + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to retrieve audit logs"}) + return + } + + ctx.JSON(http.StatusOK, response) +} +``` + +### di.go + +```go +var auditLogRepository = &AuditLogRepository{} +var auditLogService = &AuditLogService{ + auditLogRepository, + logger.GetLogger(), +} +var auditLogController = &AuditLogController{auditLogService} + +func GetAuditLogService() *AuditLogService { return auditLogService } +func GetAuditLogController() *AuditLogController { return auditLogController } + +var SetupDependencies = sync.OnceFunc(func() { + users_services.GetUserService().SetAuditLogWriter(auditLogService) + users_services.GetSettingsService().SetAuditLogWriter(auditLogService) +}) +``` + +### dto.go + +```go +type GetAuditLogsRequest struct { + Limit int `form:"limit" json:"limit"` + Offset int `form:"offset" json:"offset"` + BeforeDate *time.Time `form:"beforeDate" json:"beforeDate"` +} + +type GetAuditLogsResponse struct { + AuditLogs []*AuditLog `json:"auditLogs"` + Total int64 `json:"total"` + Limit int `json:"limit"` + Offset int `json:"offset"` +} +``` + +### model.go + +```go +type AuditLog struct { + ID uuid.UUID `json:"id" gorm:"column:id"` + UserID *uuid.UUID `json:"userId" gorm:"column:user_id"` + ProjectID *uuid.UUID `json:"projectId" gorm:"column:project_id"` + Message string `json:"message" gorm:"column:message"` + CreatedAt time.Time `json:"createdAt" gorm:"column:created_at"` +} + +func (AuditLog) TableName() string { return "audit_logs" } +``` + +### repository.go + +```go +type AuditLogRepository struct{} + +func (r *AuditLogRepository) Create(auditLog *AuditLog) error { + if auditLog.ID == uuid.Nil { + auditLog.ID = uuid.New() + } + + return storage.GetDb().Create(auditLog).Error +} + +func (r *AuditLogRepository) GetGlobal(limit, offset int, beforeDate *time.Time) ([]*AuditLog, error) { + var auditLogs []*AuditLog + + query := storage.GetDb().Order("created_at DESC") + if beforeDate != nil { + query = query.Where("created_at < ?", *beforeDate) + } + + err := query.Limit(limit).Offset(offset).Find(&auditLogs).Error + + return auditLogs, err +} +``` + +### service.go + +```go +type AuditLogService struct { + auditLogRepository *AuditLogRepository + logger *slog.Logger +} + +func (s *AuditLogService) WriteAuditLog(message string, userID, projectID *uuid.UUID) { + auditLog := &AuditLog{ + UserID: userID, + ProjectID: projectID, + Message: message, + CreatedAt: time.Now().UTC(), + } + + if err := s.auditLogRepository.Create(auditLog); err != nil { + s.logger.Error("failed to create audit log", "error", err) + } +} + +func (s *AuditLogService) GetGlobalAuditLogs( + user *user_models.User, + request *GetAuditLogsRequest, +) (*GetAuditLogsResponse, error) { + if user.Role != user_enums.UserRoleAdmin { + return nil, errors.New("only administrators can view global audit logs") + } + + limit := request.Limit + if limit <= 0 || limit > 1000 { + limit = 100 + } + + offset := max(request.Offset, 0) + + auditLogs, err := s.auditLogRepository.GetGlobal(limit, offset, request.BeforeDate) + if err != nil { + return nil, err + } + + total, err := s.auditLogRepository.CountGlobal(request.BeforeDate) + if err != nil { + return nil, err + } + + return &GetAuditLogsResponse{ + AuditLogs: auditLogs, + Total: total, + Limit: limit, + Offset: offset, + }, nil +} +``` + +For `controller_test.go`, see [Testing → Canonical controller test](#canonical-controller-test) above. + +--- + +## Modern Go + +Prefer modern stdlib idioms over manual equivalents. + +### `slices` package — avoid manual loops + +```go +slices.Contains(items, x) +slices.Index(items, x) // returns index or -1 +slices.IndexFunc(items, func(item T) bool { return item.ID == id }) +slices.SortFunc(items, func(a, b T) int { return cmp.Compare(a.X, b.X) }) +slices.Sort(items) // ordered types +slices.Max(items) / slices.Min(items) +slices.Reverse(items) // in-place +slices.Compact(items) // remove consecutive duplicates +slices.Clone(s) // shallow copy +slices.Clip(s) // trim unused capacity +``` + +### Quick wins + +- `any` instead of `interface{}`. +- `for i := range len(items)` instead of `for i := 0; i < len(items); i++`. +- `sync.OnceFunc(fn)` / `sync.OnceValue(fn)` instead of `sync.Once` + wrapper. +- `t.Context()` in tests instead of `context.WithCancel(context.Background())` + `defer cancel()` — auto-cancels at test end. +- `wg.Go(fn)` instead of `wg.Add(1)` + `go func() { defer wg.Done(); fn() }()`. + +### `context` helpers + +```go +stop := context.AfterFunc(ctx, cleanup) // run cleanup on cancellation +ctx, cancel := context.WithTimeoutCause(parent, d, ErrTimeout) // timeout with cause +ctx, cancel := context.WithDeadlineCause(parent, deadline, ErrDeadline) // deadline with cause +``` + +### `omitzero` instead of `omitempty` for non-nullable types + +`omitempty` is broken for `time.Duration`, `time.Time`, structs, slices, and maps — it doesn't omit a zero value. Use `omitzero`: + +```go +// good +type Config struct { + Timeout time.Duration `json:"timeout,omitzero"` + CreatedAt time.Time `json:"createdAt,omitzero"` +} + +// bad +type Config struct { + Timeout time.Duration `json:"timeout,omitempty"` // broken for Duration! + CreatedAt time.Time `json:"createdAt,omitempty"` +} +``` + +### `new(val)` for pointer literals (Go 1.26+) + +`new()` accepts expressions, eliminating the temp-variable pattern: + +```go +// good +cfg := Config{Timeout: new(30), Debug: new(true)} + +// bad +timeout := 30 +debug := true +cfg := Config{Timeout: &timeout, Debug: &debug} +``` diff --git a/backend/Makefile b/backend/Makefile new file mode 100644 index 0000000..1161ae1 --- /dev/null +++ b/backend/Makefile @@ -0,0 +1,61 @@ +-include ../.env + +PORT ?= 4005 +TEST_PARALLEL_WORKERS ?= 8 + +run: kill-stale + go run ./cmd + +test: clean-testcontainers pull-testcontainers + TEST_PARALLEL_WORKERS=$(TEST_PARALLEL_WORKERS) go run ./cmd/cleanup_test_db + @for i in $$(seq 0 $$(( $(TEST_PARALLEL_WORKERS) - 1 ))); do \ + dbstring=$$(echo "$(GOOSE_TEST_DBSTRING)" | sed -E "s#/([^/?]+)\?#/\1_w$$i?#"); \ + echo "migrating slot $$i"; \ + goose -dir ./migrations postgres "$$dbstring" up || exit 1; \ + done + TESTCONTAINERS_RYUK_DISABLED=true TEST_PARALLEL_WORKERS=$(TEST_PARALLEL_WORKERS) go test -p=$(TEST_PARALLEL_WORKERS) -count=1 -failfast -timeout 15m ./internal/... + +clean-testcontainers: + @ids=$$(docker ps -aq --filter "label=org.testcontainers"); \ + if [ -n "$$ids" ]; then docker rm -f $$ids; else echo "no leftover testcontainers"; fi + +pull-testcontainers: + # Tag must start with a digit so a quoted "postgres:postgres" chown arg in physical + # tests isn't mistaken for an image tag; every real image tag here is version-like. + @grep -rhoE '"(mysql|mariadb|mongo|postgres):[0-9][a-zA-Z0-9._-]*"' ./internal \ + | tr -d '"' | sort -u \ + | xargs -P 4 -I {} sh -c 'for n in 1 2 3; do docker pull {} && exit 0; echo "retry {} ($$n)"; sleep 3; done; echo "FAILED to pull {} after 3 attempts"; exit 1' + +lint: + golangci-lint fmt ./cmd/... ./internal/... && golangci-lint run ./cmd/... ./internal/... + +migration-create: + goose -dir ./migrations create $(name) sql + +migration-up: + goose -dir ./migrations postgres "$(GOOSE_DBSTRING)" up + +migration-up-test: + goose -dir ./migrations postgres "$(GOOSE_TEST_DBSTRING)" up + +migration-down: + goose -dir ./migrations postgres "$(GOOSE_DBSTRING)" down + +swagger: + swag init -g ./cmd/main.go -o swagger + +# A stale `make run` keeps holding $(PORT) and answers with the previously built +# code; a second run that fails to bind only logs the error and serves nothing, +# so requests silently hit the old binary. Free the port first — killing the +# listener also makes its parent `go run` exit. The dev container has no +# lsof/fuser, so resolve the listener through /proc. +kill-stale: + @port_hex=$$(printf ':%04X' $(PORT)); \ + inode=$$(awk -v p="$$port_hex" '$$4 == "0A" && $$2 ~ p"$$" {print $$10}' /proc/net/tcp /proc/net/tcp6 | head -1); \ + [ -n "$$inode" ] || exit 0; \ + for pid in $$(ls /proc | grep -E '^[0-9]+$$'); do \ + if ls -l /proc/$$pid/fd 2>/dev/null | grep -q "socket:\[$$inode\]"; then \ + echo "freeing port $(PORT): killing stale backend pid $$pid"; \ + kill -9 $$pid 2>/dev/null || true; \ + fi; \ + done diff --git a/backend/README.md b/backend/README.md new file mode 100644 index 0000000..25309a0 --- /dev/null +++ b/backend/README.md @@ -0,0 +1,43 @@ +# Before run + +> Copy .env.example to .env in the repo root (single source for backend, frontend, and docker compose) + +# Run + +To run: + +> make run + +To run tests: + +> make test + +Before commit (make sure `golangci-lint` is installed): + +> make lint + +# Migrations + +To create migration: + +> make migration-create name=MIGRATION_NAME + +To run migrations: + +> make migration-up + +If latest migration failed: + +To rollback on migration: + +> make migration-down + +# Swagger + +To generate swagger docs: + +> make swagger + +Swagger URL is: + +> http://localhost:4005/api/v1/docs/swagger/index.html#/ \ No newline at end of file diff --git a/backend/cleanup_test_db b/backend/cleanup_test_db new file mode 100644 index 0000000..f801398 Binary files /dev/null and b/backend/cleanup_test_db differ diff --git a/backend/cmd/cleanup_test_db/main.go b/backend/cmd/cleanup_test_db/main.go new file mode 100644 index 0000000..4822229 --- /dev/null +++ b/backend/cmd/cleanup_test_db/main.go @@ -0,0 +1,320 @@ +// Resets state before `make test` runs. Two distinct resets: +// +// 1. The Databasus metadata DB: drop and recreate the test schema referenced +// by TEST_DATABASE_DSN so Goose migrations apply against a clean slate. +// +// 2. Each test PostgreSQL container (versions 12-18): drop every non-system +// database (leftover backups, restores, and `testdb` itself), recreate +// `testdb`, and drop every non-system role. This is what makes containers +// usable across runs - tests like CreateReadOnlyUser issue +// `REVOKE CREATE ON SCHEMA public FROM PUBLIC` and create per-test roles +// that otherwise persist across runs and break subsequent pg_dump calls +// with "permission denied for schema public". +// +// Reads the test DSN through config (config.GetEnv() auto-swaps DatabaseDsn to +// TestDatabaseDsn when IsTesting is true). IsTesting is detected from os.Args +// containing the substring "test" - the binary path "cleanup_test_db" satisfies +// that. Renaming the binary or its directory will break detection. +package main + +import ( + "context" + "database/sql" + "fmt" + "log/slog" + "os" + "strings" + + _ "github.com/lib/pq" + "gorm.io/driver/postgres" + "gorm.io/gorm" + gormLogger "gorm.io/gorm/logger" + + "databasus-backend/internal/config" + cache_utils "databasus-backend/internal/util/cache" + "databasus-backend/internal/util/logger" +) + +const ( + systemDb = "postgres" + testPgContainerUser = "testuser" + testPgContainerPassword = "testpassword" + testPgContainerDb = "testdb" +) + +func main() { + log := logger.GetLogger() + ctx := context.Background() + + env := config.GetEnv() + if !env.IsTesting { + log.Error("cleanup_test_db must run with IsTesting=true (binary name must contain 'test')") + os.Exit(1) + } + + if err := resetValkey(log); err != nil { + log.Error("failed to reset Valkey", "error", err) + os.Exit(1) + } + + if err := resetMetadataDb(log, env); err != nil { + log.Error("failed to reset metadata DB", "error", err) + os.Exit(1) + } + + if err := resetTestPostgresContainers(ctx, log, env); err != nil { + log.Error("failed to reset test PG containers", "error", err) + os.Exit(1) + } +} + +// resetValkey wipes every key so each `make test` starts from a clean cache. +// Without this, a -failfast'd previous run can leave in-flight backup claims and +// other state that breaks the next run's scheduler assumptions. +func resetValkey(log *slog.Logger) error { + log.Info("resetting Valkey") + + // FLUSHALL, not ClearAllCache: parallel test workers each use their own Valkey + // logical DB (0..pool-1), so a per-DB clear would leave the others dirty. + if err := cache_utils.FlushAll(); err != nil { + return fmt.Errorf("flush cache: %w", err) + } + + log.Info("Valkey reset complete") + return nil +} + +// resetMetadataDb drops and recreates the metadata DB for every test worker slot +// (databasus_test_w0 .. databasus_test_w{pool-1}). Tests run with `go test -p`, +// each binary claiming one slot DB; provisioning all of them up front lets the +// per-slot Goose migrations apply against a clean slate. +func resetMetadataDb(log *slog.Logger, env *config.EnvVariables) error { + baseDbName, systemDsn, err := config.RewriteDbName(env.DatabaseDsn, systemDb) + if err != nil { + return fmt.Errorf("parse TEST_DATABASE_DSN: %w", err) + } + + db, err := gorm.Open(postgres.Open(systemDsn), &gorm.Config{ + Logger: gormLogger.Default.LogMode(gormLogger.Silent), + }) + if err != nil { + return fmt.Errorf("connect to system postgres DB: %w", err) + } + + for slot := range env.TestParallelWorkers { + slotDbName := fmt.Sprintf("%s_w%d", baseDbName, slot) + quoted := quoteIdentifier(slotDbName) + + if err := db.Exec(fmt.Sprintf("DROP DATABASE IF EXISTS %s WITH (FORCE)", quoted)).Error; err != nil { + return fmt.Errorf("drop database %s: %w", slotDbName, err) + } + + if err := db.Exec(fmt.Sprintf("CREATE DATABASE %s", quoted)).Error; err != nil { + return fmt.Errorf("create database %s: %w", slotDbName, err) + } + + log.Info("metadata DB reset complete", "db", slotDbName) + } + + return nil +} + +func resetTestPostgresContainers(ctx context.Context, log *slog.Logger, env *config.EnvVariables) error { + containers := []struct { + label string + port string + }{ + {"logical-16", env.TestLogicalPostgres16Port}, + {"physical-17", env.TestPhysicalPostgres17Port}, + {"physical-18", env.TestPhysicalPostgres18Port}, + } + + for _, c := range containers { + if c.port == "" { + continue + } + + if err := resetOnePostgresContainer(ctx, log, env.TestLocalhost, c.label, c.port); err != nil { + return fmt.Errorf("PG %s on %s:%s: %w", c.label, env.TestLocalhost, c.port, err) + } + } + + return nil +} + +func resetOnePostgresContainer(ctx context.Context, log *slog.Logger, host, label, port string) error { + log = log.With("pg_label", label, "port", port) + + systemDsn := fmt.Sprintf( + "host=%s port=%s user=%s password=%s dbname=%s sslmode=disable", + host, port, testPgContainerUser, testPgContainerPassword, systemDb, + ) + + db, err := sql.Open("postgres", systemDsn) + if err != nil { + return fmt.Errorf("open: %w", err) + } + defer func() { _ = db.Close() }() + + if err := db.PingContext(ctx); err != nil { + return fmt.Errorf("ping: %w", err) + } + + if err := dropDatabasusReplicationSlots(ctx, log, db); err != nil { + return fmt.Errorf("drop databasus replication slots: %w", err) + } + + userDbs, err := listUserDatabases(ctx, db) + if err != nil { + return fmt.Errorf("list databases: %w", err) + } + + for _, name := range userDbs { + if _, err := db.ExecContext(ctx, fmt.Sprintf( + "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = %s AND pid <> pg_backend_pid()", + quoteLiteral(name), + )); err != nil { + log.Warn("could not terminate connections to DB", "db", name, "error", err) + } + + // No WITH (FORCE) - that's PG 13+ only and we support PG 12. + // pg_terminate_backend above already kicked off other sessions. + if _, err := db.ExecContext( + ctx, + fmt.Sprintf("DROP DATABASE IF EXISTS %s", quoteIdentifier(name)), + ); err != nil { + return fmt.Errorf("drop database %s: %w", name, err) + } + } + + if _, err := db.ExecContext( + ctx, + fmt.Sprintf("CREATE DATABASE %s", quoteIdentifier(testPgContainerDb)), + ); err != nil { + return fmt.Errorf("create database %s: %w", testPgContainerDb, err) + } + + userRoles, err := listUserRoles(ctx, db) + if err != nil { + return fmt.Errorf("list roles: %w", err) + } + + owner := quoteIdentifier(testPgContainerUser) + for _, role := range userRoles { + quotedRole := quoteIdentifier(role) + // REASSIGN/DROP OWNED release any objects still pinned to the role so + // DROP ROLE can succeed. Failures here are best-effort - the role may + // own nothing, in which case these are no-ops. + _, _ = db.ExecContext(ctx, fmt.Sprintf("REASSIGN OWNED BY %s TO %s", quotedRole, owner)) + _, _ = db.ExecContext(ctx, fmt.Sprintf("DROP OWNED BY %s", quotedRole)) + + if _, err := db.ExecContext(ctx, fmt.Sprintf("DROP ROLE IF EXISTS %s", quotedRole)); err != nil { + log.Warn("could not drop role", "role", role, "error", err) + } + } + + log.Info("test PG container reset complete") + return nil +} + +// dropDatabasusReplicationSlots removes every replication slot whose name +// starts with the databasus prefixes. Idempotent: a missing slot returns +// pgx.ErrNoRows from the subquery and the SELECT yields no rows. +func dropDatabasusReplicationSlots(ctx context.Context, log *slog.Logger, db *sql.DB) error { + rows, err := db.QueryContext(ctx, ` + SELECT slot_name FROM pg_replication_slots + WHERE slot_name LIKE 'databasus_basebackup_%' + OR slot_name LIKE 'databasus_slot_%' + `) + if err != nil { + return fmt.Errorf("list slots: %w", err) + } + defer func() { _ = rows.Close() }() + + var slotNames []string + for rows.Next() { + var name string + if err := rows.Scan(&name); err != nil { + return fmt.Errorf("scan slot name: %w", err) + } + + slotNames = append(slotNames, name) + } + + if err := rows.Err(); err != nil { + return err + } + + for _, name := range slotNames { + // pg_drop_replication_slot fails if the slot is "active". For tests + // we want a hard reset, so terminate any active session on the slot + // first; ignore errors because slot may have just been released. + _, _ = db.ExecContext(ctx, + "SELECT pg_terminate_backend(active_pid) FROM pg_replication_slots WHERE slot_name = $1 AND active", + name, + ) + + if _, err := db.ExecContext(ctx, "SELECT pg_drop_replication_slot($1)", name); err != nil { + log.Warn("could not drop replication slot", "slot_name", name, "error", err) + continue + } + + log.Info("dropped leftover replication slot", "slot_name", name) + } + + return nil +} + +func listUserDatabases(ctx context.Context, db *sql.DB) ([]string, error) { + rows, err := db.QueryContext(ctx, ` + SELECT datname FROM pg_database + WHERE NOT datistemplate + AND datname <> $1 + `, systemDb) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + + var names []string + for rows.Next() { + var name string + if err := rows.Scan(&name); err != nil { + return nil, err + } + names = append(names, name) + } + + return names, rows.Err() +} + +func listUserRoles(ctx context.Context, db *sql.DB) ([]string, error) { + rows, err := db.QueryContext(ctx, ` + SELECT rolname FROM pg_roles + WHERE rolname NOT IN ($1, $2) + AND rolname NOT LIKE 'pg\_%' ESCAPE '\' + `, systemDb, testPgContainerUser) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + + var names []string + for rows.Next() { + var name string + if err := rows.Scan(&name); err != nil { + return nil, err + } + names = append(names, name) + } + + return names, rows.Err() +} + +func quoteIdentifier(s string) string { + return `"` + strings.ReplaceAll(s, `"`, `""`) + `"` +} + +func quoteLiteral(s string) string { + return "'" + strings.ReplaceAll(s, "'", "''") + "'" +} diff --git a/backend/cmd/healthcheck.go b/backend/cmd/healthcheck.go new file mode 100644 index 0000000..91859d0 --- /dev/null +++ b/backend/cmd/healthcheck.go @@ -0,0 +1,47 @@ +package main + +import ( + "context" + "fmt" + "io" + "net/http" + "os" + "time" +) + +const healthcheckTimeout = 4 * time.Second + +func checkServerAlive(baseURL string) error { + ctx, cancel := context.WithTimeout(context.Background(), healthcheckTimeout) + defer cancel() + + request, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/api/v1/system/version", nil) + if err != nil { + return err + } + + response, err := http.DefaultClient.Do(request) + if err != nil { + return err + } + defer func() { _ = response.Body.Close() }() + + _, _ = io.Copy(io.Discard, response.Body) + + if response.StatusCode != http.StatusOK { + return fmt.Errorf("unexpected status code: %d", response.StatusCode) + } + + return nil +} + +func runHealthcheckCommand() { + err := checkServerAlive("http://localhost" + serverAddr) + if err != nil { + fmt.Fprintf(os.Stderr, "healthcheck failed: %v\n", err) + os.Exit(1) + } + + fmt.Println("healthcheck ok") + os.Exit(0) +} diff --git a/backend/cmd/healthcheck_test.go b/backend/cmd/healthcheck_test.go new file mode 100644 index 0000000..e9b4859 --- /dev/null +++ b/backend/cmd/healthcheck_test.go @@ -0,0 +1,43 @@ +package main + +import ( + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/assert" +) + +func Test_CheckServerAlive_WhenServerReturns200_ReturnsNil(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + assert.Equal(t, "/api/v1/system/version", r.URL.Path) + w.WriteHeader(http.StatusOK) + })) + defer server.Close() + + err := checkServerAlive(server.URL) + + assert.NoError(t, err) +} + +func Test_CheckServerAlive_WhenServerReturnsNon200_ReturnsErrorWithStatus(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + })) + defer server.Close() + + err := checkServerAlive(server.URL) + + assert.Error(t, err) + assert.Contains(t, err.Error(), "500") +} + +func Test_CheckServerAlive_WhenServerUnreachable_ReturnsError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) {})) + unreachableURL := server.URL + server.Close() + + err := checkServerAlive(unreachableURL) + + assert.Error(t, err) +} diff --git a/backend/cmd/main.go b/backend/cmd/main.go new file mode 100644 index 0000000..7a53468 --- /dev/null +++ b/backend/cmd/main.go @@ -0,0 +1,472 @@ +package main + +import ( + "context" + "flag" + "fmt" + "log/slog" + "net/http" + "os" + "os/exec" + "os/signal" + "path/filepath" + "runtime/debug" + "syscall" + "time" + + "github.com/gin-contrib/cors" + "github.com/gin-contrib/gzip" + "github.com/gin-gonic/gin" + swaggerFiles "github.com/swaggo/files" + ginSwagger "github.com/swaggo/gin-swagger" + + "databasus-backend/internal/config" + "databasus-backend/internal/features/audit_logs" + "databasus-backend/internal/features/backups/backups/backuping/logical" + backuping_physical "databasus-backend/internal/features/backups/backups/backuping/physical" + backups_controllers_logical "databasus-backend/internal/features/backups/backups/controllers/logical" + backups_controllers_physical "databasus-backend/internal/features/backups/backups/controllers/physical" + backups_download "databasus-backend/internal/features/backups/backups/download" + backups_services "databasus-backend/internal/features/backups/backups/services" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/disk" + "databasus-backend/internal/features/encryption/secrets" + healthcheck_attempt "databasus-backend/internal/features/healthcheck/attempt" + healthcheck_config "databasus-backend/internal/features/healthcheck/config" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/restores" + "databasus-backend/internal/features/restores/restoring" + "databasus-backend/internal/features/storages" + system_agent "databasus-backend/internal/features/system/agent" + system_healthcheck "databasus-backend/internal/features/system/healthcheck" + system_version "databasus-backend/internal/features/system/version" + task_cancellation "databasus-backend/internal/features/tasks/cancellation" + "databasus-backend/internal/features/telemetry" + users_controllers "databasus-backend/internal/features/users/controllers" + users_middleware "databasus-backend/internal/features/users/middleware" + users_services "databasus-backend/internal/features/users/services" + verification_agents "databasus-backend/internal/features/verification/agents" + verification_config "databasus-backend/internal/features/verification/config" + verification_runs "databasus-backend/internal/features/verification/runs" + workspaces_controllers "databasus-backend/internal/features/workspaces/controllers" + "databasus-backend/internal/middleware" + cache_utils "databasus-backend/internal/util/cache" + env_utils "databasus-backend/internal/util/env" + files_utils "databasus-backend/internal/util/files" + "databasus-backend/internal/util/logger" + _ "databasus-backend/swagger" // swagger docs +) + +// @title Databasus Backend API +// @version 1.0 +// @description API for Databasus +// @termsOfService http://swagger.io/terms/ + +// @host localhost:4005 +// @BasePath /api/v1 +// @schemes http + +const serverAddr = ":4005" + +func main() { + if len(os.Args) > 1 && os.Args[1] == "healthcheck" { + runHealthcheckCommand() // exits the process + return + } + + log := logger.GetLogger() + + cache_utils.TestCacheConnection() + + log.Info("Clearing cache...") + + err := cache_utils.ClearAllCache() + if err != nil { + log.Error("Failed to clear cache", "error", err) + os.Exit(1) + } + + runMigrations(log) + + // create directories that used for backups and restore + err = files_utils.EnsureDirectories([]string{ + config.GetEnv().TempFolder, + config.GetEnv().DataFolder, + }) + if err != nil { + log.Error("Failed to ensure directories", "error", err) + os.Exit(1) + } + + err = secrets.GetSecretKeyService().MigrateKeyFromDbToFileIfExist() + if err != nil { + log.Error("Failed to migrate secret key from database to file", "error", err) + os.Exit(1) + } + + err = users_services.GetUserService().CreateInitialAdmin() + if err != nil { + log.Error("Failed to create initial admin", "error", err) + os.Exit(1) + } + + handlePasswordReset(log) + + go generateSwaggerDocs(log) + + gin.SetMode(gin.ReleaseMode) + ginApp := gin.New() + ginApp.Use(gin.Logger()) + ginApp.Use(ginRecoveryWithLogger(log)) + ginApp.Use(middleware.NoStoreCacheControl()) + + // Add GZIP compression middleware + ginApp.Use(gzip.Gzip( + gzip.DefaultCompression, + // Don't compress already compressed files + gzip.WithExcludedExtensions( + []string{".png", ".gif", ".jpeg", ".jpg", ".ico", ".svg", ".pdf", ".mp4"}, + ), + )) + + enableCors(ginApp) + setUpRoutes(ginApp) + setUpDependencies() + + announceTelemetry() + + runBackgroundTasks(log) + + mountFrontend(ginApp) + + startServerWithGracefulShutdown(log, ginApp) +} + +func handlePasswordReset(log *slog.Logger) { + audit_logs.SetupDependencies() + + newPassword := flag.String("new-password", "", "Set a new password for the user") + email := flag.String("email", "", "Email of the user to reset password") + + flag.Parse() + + if *newPassword == "" { + return + } + + log.Info("Found reset password command - reseting password...") + + if *email == "" { + log.Info("No email provided, please provide an email via --email=\"some@email.com\" flag") + os.Exit(1) + } + + resetPassword(*email, *newPassword, log) +} + +func resetPassword(email, newPassword string, log *slog.Logger) { + log.Info("Resetting password...") + + userService := users_services.GetUserService() + err := userService.ChangeUserPasswordByEmail(email, newPassword) + if err != nil { + log.Error("Failed to reset password", "error", err) + os.Exit(1) + } + + log.Info("Password reset successfully") + os.Exit(0) +} + +func startServerWithGracefulShutdown(log *slog.Logger, app *gin.Engine) { + srv := &http.Server{ + Addr: serverAddr, + Handler: app, + } + + go func() { + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + log.Error("listen:", "error", err) + } + }() + + quit := make(chan os.Signal, 1) + signal.Notify(quit, os.Interrupt, syscall.SIGTERM) + <-quit + log.Info("Shutdown signal received") + + // Gracefully shutdown VictoriaLogs writer + logger.ShutdownVictoriaLogs() + + // The context is used to inform the server it has 10 seconds to finish + // the request it is currently handling + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := srv.Shutdown(ctx); err != nil { + log.Error("Server forced to shutdown:", "error", err) + } + + log.Info("Server gracefully stopped") +} + +func setUpRoutes(r *gin.Engine) { + v1 := r.Group("/api/v1") + + // Mount Swagger UI + v1.GET("/docs/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler)) + + // Public routes (only user auth routes and healthcheck should be public) + userController := users_controllers.GetUserController() + userController.RegisterRoutes(v1) + system_healthcheck.GetHealthcheckController().RegisterRoutes(v1) + system_version.GetVersionController().RegisterRoutes(v1) + system_agent.GetAgentController().RegisterRoutes(v1) + backups_controllers_logical.GetBackupController().RegisterPublicRoutes(v1) + backups_controllers_physical.GetPhysicalBackupController().RegisterPublicRoutes(v1) + databases.GetDatabaseController().RegisterPublicRoutes(v1) + verification_agents.GetAgentFacingController().RegisterRoutes(v1) + verification_runs.GetVerificationAgentController().RegisterRoutes(v1) + + // Setup auth middleware + userService := users_services.GetUserService() + authMiddleware := users_middleware.AuthMiddleware(userService) + + // Protected routes + protected := v1.Group("") + protected.Use(authMiddleware) + + userController.RegisterProtectedRoutes(protected) + workspaces_controllers.GetWorkspaceController().RegisterRoutes(protected) + workspaces_controllers.GetMembershipController().RegisterRoutes(protected) + disk.GetDiskController().RegisterRoutes(protected) + notifiers.GetNotifierController().RegisterRoutes(protected) + storages.GetStorageController().RegisterRoutes(protected) + databases.GetDatabaseController().RegisterRoutes(protected) + backups_controllers_logical.GetBackupController().RegisterRoutes(protected) + backups_controllers_physical.GetPhysicalBackupController().RegisterRoutes(protected) + restores.GetRestoreController().RegisterRoutes(protected) + healthcheck_config.GetHealthcheckConfigController().RegisterRoutes(protected) + healthcheck_attempt.GetHealthcheckAttemptController().RegisterRoutes(protected) + backups_config_logical.GetBackupConfigController().RegisterRoutes(protected) + backups_config_physical.GetBackupConfigController().RegisterRoutes(protected) + audit_logs.GetAuditLogController().RegisterRoutes(protected) + users_controllers.GetManagementController().RegisterRoutes(protected) + users_controllers.GetSettingsController().RegisterRoutes(protected) + verification_agents.GetAgentController().RegisterRoutes(protected) + verification_config.GetVerificationConfigController().RegisterRoutes(protected) + verification_runs.GetVerificationController().RegisterRoutes(protected) +} + +func setUpDependencies() { + databases.SetupDependencies() + backups_services.SetupDependencies() + restores.SetupDependencies() + healthcheck_config.SetupDependencies() + audit_logs.SetupDependencies() + notifiers.SetupDependencies() + storages.SetupDependencies() + backups_config_logical.SetupDependencies() + backups_config_physical.SetupDependencies() + backuping_physical.SetupDependencies() + verification_config.SetupDependencies() + verification_runs.SetupDependencies() + task_cancellation.SetupDependencies() + + telemetry.SetupDependencies() +} + +func announceTelemetry() { + if config.GetEnv().IsDisableAnonymousTelemetry { + return + } + + fmt.Println( + "Anonymous telemetry collected (Databasus version, OS/arch, etc.). No DB contents, no user data. " + + "To disable, set IS_DISABLE_ANONYMOUS_TELEMETRY=true in your .env", + ) +} + +func runBackgroundTasks(log *slog.Logger) { + log.Info("Preparing to run background tasks...") + + // Create context that will be cancelled on shutdown + ctx, cancel := context.WithCancel(context.Background()) + + // Set up signal handling for graceful shutdown + quit := make(chan os.Signal, 1) + signal.Notify(quit, os.Interrupt, syscall.SIGTERM) + go func() { + <-quit + log.Info("Shutdown signal received, cancelling all background tasks") + cancel() + }() + + err := files_utils.CleanFolder(config.GetEnv().TempFolder) + if err != nil { + log.Error("Failed to clean temp folder", "error", err) + } + + log.Info("Starting background tasks...") + + go runWithPanicLogging(log, "backup background service", func() { + backuping_logical.GetBackupsScheduler().Run(ctx) + }) + + go runWithPanicLogging(log, "verification scheduler", func() { + verification_runs.GetVerificationScheduler().Run(ctx) + }) + + go runWithPanicLogging(log, "backup cleaner background service", func() { + backuping_logical.GetBackupCleaner().Run(ctx) + }) + + go runWithPanicLogging(log, "physical backup scheduler background service", func() { + backuping_physical.GetPhysicalBackupsScheduler().Run(ctx) + }) + + go runWithPanicLogging(log, "physical backup cleaner background service", func() { + backuping_physical.GetPhysicalBackupCleaner().Run(ctx) + }) + + go runWithPanicLogging(log, "restore background service", func() { + restoring.GetRestoresScheduler().Run(ctx) + }) + + go runWithPanicLogging(log, "healthcheck attempt background service", func() { + healthcheck_attempt.GetHealthcheckAttemptBackgroundService().Run(ctx) + }) + + go runWithPanicLogging(log, "audit log cleanup background service", func() { + audit_logs.GetAuditLogBackgroundService().Run(ctx) + }) + + go runWithPanicLogging(log, "download token cleanup background service", func() { + backups_download.GetDownloadTokenBackgroundService().Run(ctx) + }) + + go runWithPanicLogging(log, "physical wal stream supervisor background service", func() { + backuping_physical.GetPhysicalWalStreamSupervisor().Run(ctx) + }) + + go runWithPanicLogging(log, "telemetry background service", func() { + telemetry.GetTelemetryBackgroundService().Run(ctx) + }) +} + +func runWithPanicLogging(log *slog.Logger, serviceName string, fn func()) { + defer func() { + if r := recover(); r != nil { + log.Error("Panic in "+serviceName, "error", r, "stacktrace", string(debug.Stack())) + } + }() + fn() +} + +// Keep in mind: docs appear after second launch, because Swagger +// is generated into Go files. So if we changed files, we generate +// new docs, but still need to restart the server to see them. +func generateSwaggerDocs(log *slog.Logger) { + if config.GetEnv().EnvMode == env_utils.EnvModeProduction { + return + } + + // Run swag from the current directory instead of parent + // Use the current directory as the base for swag init + // This ensures swag can find the files regardless of where the command is run from + currentDir, err := os.Getwd() + if err != nil { + log.Error("Failed to get current directory", "error", err) + return + } + + cmd := exec.CommandContext( + context.Background(), "swag", "init", "-d", currentDir, "-g", "cmd/main.go", "-o", "swagger", + ) + + output, err := cmd.CombinedOutput() + if err != nil { + log.Error("Failed to generate Swagger docs", "error", err, "output", string(output)) + return + } + + log.Info("Swagger documentation generated successfully") +} + +func runMigrations(log *slog.Logger) { + log.Info("Running database migrations...") + + cmd := exec.CommandContext(context.Background(), "goose", "-dir", "./migrations", "up") + cmd.Env = append( + os.Environ(), + "GOOSE_DRIVER=postgres", + "GOOSE_DBSTRING="+config.GetEnv().DatabaseDsn, + ) + + output, err := cmd.CombinedOutput() + if err != nil { + log.Error("Failed to run migrations", "error", err, "output", string(output)) + os.Exit(1) + } + + log.Info("Database migrations completed successfully", "output", string(output)) +} + +func enableCors(ginApp *gin.Engine) { + if config.GetEnv().EnvMode == env_utils.EnvModeDevelopment { + // Setup CORS + ginApp.Use(cors.New(cors.Config{ + AllowOrigins: []string{"*"}, + AllowMethods: []string{"GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS"}, + AllowHeaders: []string{ + "Origin", + "Content-Length", + "Content-Type", + "Authorization", + "Accept", + "Accept-Language", + "Accept-Encoding", + "Access-Control-Request-Method", + "Access-Control-Request-Headers", + "Access-Control-Allow-Methods", + "Access-Control-Allow-Headers", + "Access-Control-Allow-Origin", + }, + AllowCredentials: true, + })) + } +} + +func ginRecoveryWithLogger(log *slog.Logger) gin.HandlerFunc { + return func(ctx *gin.Context) { + defer func() { + if r := recover(); r != nil { + log.Error("Panic recovered in HTTP handler", + "error", r, + "stacktrace", string(debug.Stack()), + "method", ctx.Request.Method, + "path", ctx.Request.URL.Path, + ) + + ctx.AbortWithStatus(http.StatusInternalServerError) + } + }() + + ctx.Next() + } +} + +func mountFrontend(ginApp *gin.Engine) { + staticDir := "./ui/build" + ginApp.NoRoute(func(c *gin.Context) { + path := filepath.Join(staticDir, c.Request.URL.Path) + + if info, err := os.Stat(path); err == nil && !info.IsDir() { + c.File(path) + return + } + + c.File(filepath.Join(staticDir, "index.html")) + }) +} diff --git a/backend/go.mod b/backend/go.mod new file mode 100644 index 0000000..295dd72 --- /dev/null +++ b/backend/go.mod @@ -0,0 +1,315 @@ +module databasus-backend + +go 1.26.3 + +require ( + github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 + github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 + github.com/gin-contrib/cors v1.7.5 + github.com/gin-contrib/gzip v1.2.3 + github.com/gin-gonic/gin v1.10.0 + github.com/golang-jwt/jwt/v4 v4.5.2 + github.com/google/uuid v1.6.0 + github.com/ilyakaznacheev/cleanenv v1.5.0 + github.com/jackc/pgx/v5 v5.9.2 + github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb + github.com/jmoiron/sqlx v1.4.0 + github.com/joho/godotenv v1.5.1 + github.com/lib/pq v1.10.9 + github.com/minio/minio-go/v7 v7.0.100 + github.com/moby/moby/api v1.54.1 + github.com/pkg/sftp v1.13.10 + github.com/rclone/rclone v1.74.3 + github.com/robfig/cron/v3 v3.0.1 + github.com/shirou/gopsutil/v4 v4.26.3 + github.com/stretchr/testify v1.11.1 + github.com/swaggo/files v1.0.1 + github.com/swaggo/gin-swagger v1.6.0 + github.com/swaggo/swag v1.16.4 + github.com/testcontainers/testcontainers-go v0.42.0 + github.com/valkey-io/valkey-go v1.0.70 + go.mongodb.org/mongo-driver v1.17.9 + golang.org/x/crypto v0.52.0 + gorm.io/driver/postgres v1.5.11 + gorm.io/gorm v1.26.1 +) + +require ( + dario.cat/mergo v1.0.2 // indirect + filippo.io/edwards25519 v1.1.0 // indirect + github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 // indirect + github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 // indirect + github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.4 // indirect + github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect + github.com/Azure/go-ntlmssp v0.1.1 // indirect + github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 // indirect + github.com/FilenCloudDienste/filen-sdk-go v0.0.39 // indirect + github.com/Files-com/files-sdk-go/v3 v3.3.82 // indirect + github.com/IBM/go-sdk-core/v5 v5.21.2 // indirect + github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd // indirect + github.com/Microsoft/go-winio v0.6.2 // indirect + github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf // indirect + github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e // indirect + github.com/ProtonMail/go-crypto v1.4.1 // indirect + github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f // indirect + github.com/ProtonMail/go-srp v0.0.7 // indirect + github.com/ProtonMail/gopenpgp/v2 v2.9.0 // indirect + github.com/PuerkitoBio/goquery v1.11.0 // indirect + github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 // indirect + github.com/abbot/go-http-auth v0.4.0 // indirect + github.com/adrg/xdg v0.5.3 // indirect + github.com/anchore/go-lzo v0.1.0 // indirect + github.com/andybalholm/cascadia v1.3.3 // indirect + github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc // indirect + github.com/aws/aws-sdk-go-v2 v1.41.5 // indirect + github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect + github.com/aws/aws-sdk-go-v2/config v1.32.14 // indirect + github.com/aws/aws-sdk-go-v2/credentials v1.19.14 // indirect + github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.13 // indirect + github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect + github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 // indirect + github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect + github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 // indirect + github.com/aws/aws-sdk-go-v2/service/s3 v1.99.0 // indirect + github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 // indirect + github.com/aws/aws-sdk-go-v2/service/sso v1.30.15 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.19 // indirect + github.com/aws/aws-sdk-go-v2/service/sts v1.41.10 // indirect + github.com/aws/smithy-go v1.24.3 // indirect + github.com/bahlo/generic-list-go v0.2.0 // indirect + github.com/beorn7/perks v1.0.1 // indirect + github.com/boombuler/barcode v1.1.0 // indirect + github.com/bradenaw/juniper v0.15.3 // indirect + github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect + github.com/buengese/sgzip v0.1.1 // indirect + github.com/buger/jsonparser v1.1.2 // indirect + github.com/calebcase/tmpfile v1.0.3 // indirect + github.com/cenkalti/backoff/v4 v4.3.0 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect + github.com/clipperhouse/uax29/v2 v2.7.0 // indirect + github.com/cloudflare/circl v1.6.3 // indirect + github.com/cloudinary/cloudinary-go/v2 v2.15.0 // indirect + github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc // indirect + github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect + github.com/colinmarc/hdfs/v2 v2.4.0 // indirect + github.com/containerd/errdefs v1.0.0 // indirect + github.com/containerd/errdefs/pkg v0.3.0 // indirect + github.com/containerd/log v0.1.0 // indirect + github.com/containerd/platforms v0.2.1 // indirect + github.com/coreos/go-semver v0.3.1 // indirect + github.com/coreos/go-systemd/v22 v22.6.0 // indirect + github.com/cpuguy83/dockercfg v0.3.2 // indirect + github.com/creasty/defaults v1.8.0 // indirect + github.com/cronokirby/saferith v0.33.1-0.20250226174546-1f11f94ce488 // indirect + github.com/diskfs/go-diskfs v1.7.0 // indirect + github.com/distribution/reference v0.6.0 // indirect + github.com/docker/go-connections v0.6.0 // indirect + github.com/docker/go-units v0.5.0 // indirect + github.com/dromara/dongle v1.0.1 // indirect + github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 // indirect + github.com/emersion/go-message v0.18.2 // indirect + github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff // indirect + github.com/flynn/noise v1.1.0 // indirect + github.com/go-chi/chi/v5 v5.2.5 // indirect + github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 // indirect + github.com/go-git/go-billy/v5 v5.9.0 // indirect + github.com/go-openapi/errors v0.22.6 // indirect + github.com/go-openapi/strfmt v0.25.0 // indirect + github.com/go-resty/resty/v2 v2.17.2 // indirect + github.com/go-viper/mapstructure/v2 v2.5.0 // indirect + github.com/gofrs/flock v0.13.0 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang-jwt/jwt/v5 v5.3.1 // indirect + github.com/golang/snappy v0.0.4 // indirect + github.com/google/btree v1.1.3 // indirect + github.com/gorilla/schema v1.4.1 // indirect + github.com/hashicorp/errwrap v1.1.0 // indirect + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect + github.com/hashicorp/go-multierror v1.1.1 // indirect + github.com/hashicorp/go-retryablehttp v0.7.8 // indirect + github.com/hashicorp/go-uuid v1.0.3 // indirect + github.com/internxt/rclone-adapter v0.0.0-20260331173834-036f908d0160 // indirect + github.com/jcmturner/aescts/v2 v2.0.0 // indirect + github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect + github.com/jcmturner/gofork v1.7.6 // indirect + github.com/jcmturner/goidentity/v6 v6.0.1 // indirect + github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect + github.com/jcmturner/rpc/v2 v2.0.3 // indirect + github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect + github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect + github.com/klauspost/crc32 v1.3.0 // indirect + github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 // indirect + github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 // indirect + github.com/kr/fs v0.1.0 // indirect + github.com/kylelemons/godebug v1.1.0 // indirect + github.com/lanrat/extsort v1.4.2 // indirect + github.com/lpar/date v1.0.0 // indirect + github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 // indirect + github.com/magiconair/properties v1.8.10 // indirect + github.com/mattn/go-colorable v0.1.14 // indirect + github.com/mattn/go-runewidth v0.0.22 // indirect + github.com/mitchellh/go-homedir v1.1.0 // indirect + github.com/moby/docker-image-spec v1.3.1 // indirect + github.com/moby/go-archive v0.2.0 // indirect + github.com/moby/moby/client v0.4.0 // indirect + github.com/moby/patternmatcher v0.6.1 // indirect + github.com/moby/sys/sequential v0.6.0 // indirect + github.com/moby/sys/user v0.4.0 // indirect + github.com/moby/sys/userns v0.1.0 // indirect + github.com/moby/term v0.5.2 // indirect + github.com/montanaflynn/stats v0.7.1 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/ncw/swift/v2 v2.0.5 // indirect + github.com/oklog/ulid v1.3.1 // indirect + github.com/opencontainers/go-digest v1.0.0 // indirect + github.com/opencontainers/image-spec v1.1.1 // indirect + github.com/oracle/oci-go-sdk/v65 v65.111.0 // indirect + github.com/panjf2000/ants/v2 v2.11.5 // indirect + github.com/patrickmn/go-cache v2.1.0+incompatible // indirect + github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 // indirect + github.com/peterh/liner v1.2.2 // indirect + github.com/pierrec/lz4/v4 v4.1.25 // indirect + github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/pkg/xattr v0.4.12 // indirect + github.com/pquerna/otp v1.5.0 // indirect + github.com/prometheus/client_golang v1.23.2 // indirect + github.com/prometheus/client_model v0.6.2 // indirect + github.com/prometheus/common v0.67.5 // indirect + github.com/prometheus/procfs v0.19.2 // indirect + github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 // indirect + github.com/rclone/Proton-API-Bridge v1.0.3 // indirect + github.com/rclone/go-proton-api v1.0.2 // indirect + github.com/relvacode/iso8601 v1.7.0 // indirect + github.com/rfjakob/eme v1.2.0 // indirect + github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 // indirect + github.com/samber/lo v1.52.0 // indirect + github.com/sirupsen/logrus v1.9.4 // indirect + github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 // indirect + github.com/sony/gobreaker v1.0.0 // indirect + github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 // indirect + github.com/spf13/pflag v1.0.10 // indirect + github.com/t3rm1n4l/go-mega v0.0.0-20251120131202-6845944c051c // indirect + github.com/tklauser/go-sysconf v0.3.16 // indirect + github.com/tklauser/numcpus v0.11.0 // indirect + github.com/tyler-smith/go-bip39 v1.1.0 // indirect + github.com/ulikunitz/xz v0.5.15 // indirect + github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect + github.com/xanzy/ssh-agent v0.3.3 // indirect + github.com/xdg-go/pbkdf2 v1.0.0 // indirect + github.com/xdg-go/scram v1.1.2 // indirect + github.com/xdg-go/stringprep v1.0.4 // indirect + github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect + github.com/yunify/qingstor-sdk-go/v3 v3.2.0 // indirect + github.com/zeebo/blake3 v0.2.4 // indirect + github.com/zeebo/errs v1.4.0 // indirect + github.com/zeebo/xxh3 v1.1.0 // indirect + go.etcd.io/bbolt v1.4.3 // indirect + go.yaml.in/yaml/v2 v2.4.3 // indirect + go.yaml.in/yaml/v3 v3.0.4 // indirect + golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect + golang.org/x/image v0.41.0 // indirect + golang.org/x/term v0.43.0 // indirect + golang.org/x/time v0.15.0 // indirect + gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect + gopkg.in/validator.v2 v2.0.1 // indirect + moul.io/http2curl/v2 v2.3.0 // indirect + sigs.k8s.io/yaml v1.6.0 // indirect + storj.io/common v0.0.0-20260225132117-99155641c30a // indirect + storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 // indirect + storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 // indirect + storj.io/infectious v0.0.2 // indirect + storj.io/picobuf v0.0.4 // indirect + storj.io/uplink v1.14.0 // indirect +) + +require ( + cloud.google.com/go/auth v0.20.0 // indirect + cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect + cloud.google.com/go/compute/metadata v0.9.0 // indirect + github.com/geoffgarside/ber v1.2.0 // indirect + github.com/google/s2a-go v0.1.9 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect + github.com/googleapis/gax-go/v2 v2.21.0 // indirect + github.com/hirochachacha/go-smb2 v1.1.0 + google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect + google.golang.org/grpc v1.80.0 // indirect +) + +require ( + github.com/BurntSushi/toml v1.5.0 // indirect + github.com/KyleBanks/depth v1.2.1 // indirect + github.com/PuerkitoBio/purell v1.1.1 // indirect + github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect + github.com/bytedance/sonic v1.13.2 // indirect + github.com/bytedance/sonic/loader v0.2.4 // indirect + github.com/cloudwego/base64x v0.1.5 // indirect + github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect + github.com/dustin/go-humanize v1.0.1 // indirect + github.com/ebitengine/purego v0.10.0 // indirect + github.com/felixge/httpsnoop v1.0.4 // indirect + github.com/gabriel-vasile/mimetype v1.4.13 // indirect + github.com/gin-contrib/sse v1.1.0 // indirect + github.com/go-ini/ini v1.67.0 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/go-ole/go-ole v1.3.0 // indirect + github.com/go-openapi/jsonpointer v0.19.5 // indirect + github.com/go-openapi/jsonreference v0.19.6 // indirect + github.com/go-openapi/spec v0.20.4 // indirect + github.com/go-openapi/swag v0.19.15 // indirect + github.com/go-playground/locales v0.14.1 // indirect + github.com/go-playground/universal-translator v0.18.1 // indirect + github.com/go-playground/validator/v10 v10.30.1 // indirect + github.com/go-sql-driver/mysql v1.9.2 + github.com/goccy/go-json v0.10.5 // indirect + github.com/jackc/pgpassfile v1.0.0 // indirect + github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect + github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/jinzhu/inflection v1.0.0 // indirect + github.com/jinzhu/now v1.1.5 // indirect + github.com/josharian/intern v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/klauspost/compress v1.18.5 + github.com/klauspost/cpuid/v2 v2.3.0 // indirect + github.com/leodido/go-urn v1.4.0 // indirect + github.com/mailru/easyjson v0.9.1 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect + github.com/minio/crc64nvme v1.1.1 // indirect + github.com/minio/md5-simd v1.1.2 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/pelletier/go-toml/v2 v2.2.4 // indirect + github.com/philhofer/fwd v1.2.0 // indirect + github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect + github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect + github.com/rs/xid v1.6.0 // indirect + github.com/stretchr/objx v0.5.3 // indirect + github.com/tinylib/msgp v1.6.3 // indirect + github.com/twitchyliquid64/golang-asm v0.15.1 // indirect + github.com/ugorji/go/codec v1.2.12 // indirect + github.com/yusufpapurcu/wmi v1.2.4 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect + go.opentelemetry.io/otel v1.43.0 // indirect + go.opentelemetry.io/otel/metric v1.43.0 // indirect + go.opentelemetry.io/otel/trace v1.43.0 // indirect + golang.org/x/arch v0.17.0 // indirect + golang.org/x/net v0.55.0 // indirect + golang.org/x/oauth2 v0.36.0 + golang.org/x/sync v0.20.0 // indirect + golang.org/x/sys v0.45.0 // indirect + golang.org/x/text v0.37.0 // indirect + golang.org/x/tools v0.44.0 // indirect + google.golang.org/api v0.275.0 + google.golang.org/protobuf v1.36.11 // indirect + gopkg.in/yaml.v2 v2.4.0 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 // indirect +) diff --git a/backend/go.sum b/backend/go.sum new file mode 100644 index 0000000..521d0fa --- /dev/null +++ b/backend/go.sum @@ -0,0 +1,1204 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU= +cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU= +cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY= +cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc= +cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0= +cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To= +cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4= +cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M= +cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc= +cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk= +cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs= +cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc= +cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY= +cloud.google.com/go/auth v0.20.0 h1:kXTssoVb4azsVDoUiF8KvxAqrsQcQtB53DcSgta74CA= +cloud.google.com/go/auth v0.20.0/go.mod h1:942/yi/itH1SsmpyrbnTMDgGfdy2BUqIKyd0cyYLc5Q= +cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc= +cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c= +cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= +cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= +cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= +cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg= +cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= +cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= +cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs= +cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10= +cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= +cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= +cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= +cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= +cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= +cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU= +cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= +cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= +cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= +cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs= +cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= +dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8= +dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA= +dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= +filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= +filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= +github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8afgbRMd7mFxO99hRNu+6tazq8nFF9lIwo9JFroBk= +github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8= +github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28= +github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA= +github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1 h1:Hk5QBxZQC1jb2Fwj6mpzme37xbCDdNTxU7O9eb5+LB4= +github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1/go.mod h1:IYus9qsFobWIc2YVwe/WPjcnyCkPKtnHAqUYeebc8z0= +github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.3.2 h1:yz1bePFlP5Vws5+8ez6T3HWXPmwOK7Yvq8QxDBD3SKY= +github.com/Azure/azure-sdk-for-go/sdk/azidentity/cache v0.3.2/go.mod h1:Pa9ZNPuoNu/GztvBSKk9J1cDJW6vk/n0zLtV4mgd8N8= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 h1:9iefClla7iYpfYWdzPCRDozdmndjTm8DXdpCzPajMgA= +github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2/go.mod h1:XtLgD3ZD34DAaVIIAyG3objl5DynM3CQ/vMcbBNJZGI= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig= +github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA= +github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 h1:jWQK1GI+LeGGUKBADtcH2rRqPxYB1Ljwms5gFA2LqrM= +github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4/go.mod h1:8mwH4klAm9DUgR2EEHyEEAQlRDvLPyg5fQry3y+cDew= +github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.4 h1:tZh20RjgfMxKBxJiIS75iTVAKIUxrST5X2dVHMTptL4= +github.com/Azure/azure-sdk-for-go/sdk/storage/azfile v1.5.4/go.mod h1:vGYAk36rhMVCfTP7v+RVruCR0zmPe6S+36KRpDCLySw= +github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg= +github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= +github.com/Azure/go-ntlmssp v0.1.1 h1:l+FM/EEMb0U9QZE7mKNEDw5Mu3mFiaa2GKOoTSsNDPw= +github.com/Azure/go-ntlmssp v0.1.1/go.mod h1:NYqdhxd/8aAct/s4qSYZEerdPuH1liG2/X9DiVTbhpk= +github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1 h1:WJTmL004Abzc5wDB5VtZG2PJk5ndYDgVacGqfirKxjM= +github.com/AzureAD/microsoft-authentication-extensions-for-go/cache v0.1.1/go.mod h1:tCcJZ0uHAmvjsVYzEFivsRTN00oz5BEsRgQHu5JZ9WE= +github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0 h1:XRzhVemXdgvJqCH0sFfrBUTnUJSBrBf7++ypk+twtRs= +github.com/AzureAD/microsoft-authentication-library-for-go v1.6.0/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/BurntSushi/toml v1.2.1/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ= +github.com/BurntSushi/toml v1.5.0 h1:W5quZX/G/csjUnuI8SUYlsHs9M38FC7znL0lIO+DvMg= +github.com/BurntSushi/toml v1.5.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho= +github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= +github.com/FilenCloudDienste/filen-sdk-go v0.0.39 h1:tgV5jYL6dsXop9TpDTIQU6UwJjws122HrwskaEE/igY= +github.com/FilenCloudDienste/filen-sdk-go v0.0.39/go.mod h1:0cBhKXQg49XbKZZfk5TCDa3sVLP+xMxZTWL+7KY0XR0= +github.com/Files-com/files-sdk-go/v3 v3.3.82 h1:2RfP0d2QgkFH64BjZSWd59aMsc28IyWsUuHqU0txBtY= +github.com/Files-com/files-sdk-go/v3 v3.3.82/go.mod h1:IPk80dOmc7VFC0DJ85xMTPmre+8xoXX6kGHAkf5jRRw= +github.com/IBM/go-sdk-core/v5 v5.21.2 h1:mJ5QbLPOm4g5qhZiVB6wbSllfpeUExftGoyPek2hk4M= +github.com/IBM/go-sdk-core/v5 v5.21.2/go.mod h1:ngpMgwkjur1VNUjqn11LPk3o5eCyOCRbcfg/0YAY7Hc= +github.com/KyleBanks/depth v1.2.1 h1:5h8fQADFrWtarTdtDudMmGsC7GPbOAu6RVB3ffsVFHc= +github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6XgeJcm8brE= +github.com/Masterminds/semver/v3 v3.2.0 h1:3MEsd0SM6jqZojhjLWWeBY+Kcjy9i6MQAeY7YgDP83g= +github.com/Masterminds/semver/v3 v3.2.0/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ= +github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd h1:nzE1YQBdx1bq9IlZinHa+HVffy+NmVRoKr+wHN8fpLE= +github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd/go.mod h1:C8yoIfvESpM3GD07OCHU7fqI7lhwyZ2Td1rbNbTAhnc= +github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY= +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/ProtonMail/bcrypt v0.0.0-20210511135022-227b4adcab57/go.mod h1:HecWFHognK8GfRDGnFQbW/LiV7A3MX3gZVs45vk5h8I= +github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf h1:yc9daCCYUefEs69zUkSzubzjBbL+cmOXgnmt9Fyd9ug= +github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf/go.mod h1:o0ESU9p83twszAU8LBeJKFAAMX14tISa0yk4Oo5TOqo= +github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e h1:lCsqUUACrcMC83lg5rTo9Y0PnPItE61JSfvMyIcANwk= +github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e/go.mod h1:Og5/Dz1MiGpCJn51XujZwxiLG7WzvvjE5PRpZBQmAHo= +github.com/ProtonMail/go-crypto v0.0.0-20230321155629-9a39f2531310/go.mod h1:8TI4H3IbrackdNgv+92dI+rhpCaLqM0IfpgCgenFvRE= +github.com/ProtonMail/go-crypto v1.4.1 h1:9RfcZHqEQUvP8RzecWEUafnZVtEvrBVL9BiF67IQOfM= +github.com/ProtonMail/go-crypto v1.4.1/go.mod h1:e1OaTyu5SYVrO9gKOEhTc+5UcXtTUa+P3uLudwcgPqo= +github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f h1:tCbYj7/299ekTTXpdwKYF8eBlsYsDVoggDAuAjoK66k= +github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f/go.mod h1:gcr0kNtGBqin9zDW9GOHcVntrwnjrK+qdJ06mWYBybw= +github.com/ProtonMail/go-srp v0.0.7 h1:Sos3Qk+th4tQR64vsxGIxYpN3rdnG9Wf9K4ZloC1JrI= +github.com/ProtonMail/go-srp v0.0.7/go.mod h1:giCp+7qRnMIcCvI6V6U3S1lDDXDQYx2ewJ6F/9wdlJk= +github.com/ProtonMail/gopenpgp/v2 v2.9.0 h1:ruLzBmwe4dR1hdnrsEJ/S7psSBmV15gFttFUPP/+/kE= +github.com/ProtonMail/gopenpgp/v2 v2.9.0/go.mod h1:IldDyh9Hv1ZCCYatTuuEt1XZJ0OPjxLpTarDfglih7s= +github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw= +github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ= +github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI= +github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0= +github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M= +github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE= +github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0 h1:iLDOF0rdGTrol/q8OfPIIs5kLD8XvA2q75o6Uq/tgak= +github.com/a1ex3/zstd-seekable-format-go/pkg v0.10.0/go.mod h1:DrEWcQJjz7t5iF2duaiyhg4jyoF0kxOD6LtECNGkZ/Q= +github.com/aalpar/deheap v1.1.2 h1:MABHLcnjqsffb8GLkUFDigqpBBxOMz0DoKM9QfELeTw= +github.com/aalpar/deheap v1.1.2/go.mod h1:A+nfkD4JbS05sewV0he/MYgR/90vfqyMoNNROgs+rmA= +github.com/abbot/go-http-auth v0.4.0 h1:QjmvZ5gSC7jm3Zg54DqWE/T5m1t2AfDu6QlXJT0EVT0= +github.com/abbot/go-http-auth v0.4.0/go.mod h1:Cz6ARTIzApMJDzh5bRMSUou6UMSp0IEXg9km/ci7TJM= +github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78= +github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ= +github.com/anchore/go-lzo v0.1.0 h1:NgAacnzqPeGH49Ky19QKLBZEuFRqtTG9cdaucc3Vncs= +github.com/anchore/go-lzo v0.1.0/go.mod h1:3kLx0bve2oN1iDwgM1U5zGku1Tfbdb0No5qp1eL1fIk= +github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM= +github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA= +github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc h1:LoL75er+LKDHDUfU5tRvFwxH0LjPpZN8OoG8Ll+liGU= +github.com/appscode/go-querystring v0.0.0-20170504095604-0126cfb3f1dc/go.mod h1:w648aMHEgFYS6xb0KVMMtZ2uMeemhiKCuD2vj6gY52A= +github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY= +github.com/aws/aws-sdk-go-v2 v1.41.5/go.mod h1:mwsPRE8ceUUpiTgF7QmQIJ7lgsKUPQOUl3o72QBrE1o= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 h1:eBMB84YGghSocM7PsjmmPffTa+1FBUeNvGvFou6V/4o= +github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8/go.mod h1:lyw7GFp3qENLh7kwzf7iMzAxDn+NzjXEAGjKS2UOKqI= +github.com/aws/aws-sdk-go-v2/config v1.32.14 h1:opVIRo/ZbbI8OIqSOKmpFaY7IwfFUOCCXBsUpJOwDdI= +github.com/aws/aws-sdk-go-v2/config v1.32.14/go.mod h1:U4/V0uKxh0Tl5sxmCBZ3AecYny4UNlVmObYjKuuaiOo= +github.com/aws/aws-sdk-go-v2/credentials v1.19.14 h1:n+UcGWAIZHkXzYt87uMFBv/l8THYELoX6gVcUvgl6fI= +github.com/aws/aws-sdk-go-v2/credentials v1.19.14/go.mod h1:cJKuyWB59Mqi0jM3nFYQRmnHVQIcgoxjEMAbLkpr62w= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 h1:NUS3K4BTDArQqNu2ih7yeDLaS3bmHD0YndtA6UP884g= +github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21/go.mod h1:YWNWJQNjKigKY1RHVJCuupeWDrrHjRqHm0N9rdrWzYI= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.13 h1:uMC4oL6G3MNhodo358QEqSDjrgvzV3TUQ58nyQSGq2E= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.13/go.mod h1:Cer86AE2686DvVUe57LPve3jUBmbujuaonSX8pNzGgw= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 h1:Rgg6wvjjtX8bNHcvi9OnXWwcE0a2vGpbwmtICOsvcf4= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21/go.mod h1:A/kJFst/nm//cyqonihbdpQZwiUhhzpqTsdbhDdRF9c= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 h1:PEgGVtPoB6NTpPrBgqSE5hE/o47Ij9qk/SEZFbUOe9A= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21/go.mod h1:p+hz+PRAYlY3zcpJhPwXlLC4C+kqn70WIHwnzAfs6ps= +github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6 h1:qYQ4pzQ2Oz6WpQ8T3HvGHnZydA72MnLuFK9tJwmrbHw= +github.com/aws/aws-sdk-go-v2/internal/ini v1.8.6/go.mod h1:O3h0IK87yXci+kg6flUKzJnWeziQUKciKrLjcatSNcY= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 h1:rWyie/PxDRIdhNf4DzRk0lvjVOqFJuNnO8WwaIRVxzQ= +github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22/go.mod h1:zd/JsJ4P7oGfUhXn1VyLqaRZwPmZwg44Jf2dS84Dm3Y= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 h1:5EniKhLZe4xzL7a+fU3C2tfUN4nWIqlLesfrjkuPFTY= +github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7/go.mod h1:x0nZssQ3qZSnIcePWLvcoFisRXJzcTVvYpAAdYX8+GI= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 h1:JRaIgADQS/U6uXDqlPiefP32yXTda7Kqfx+LgspooZM= +github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13/go.mod h1:CEuVn5WqOMilYl+tbccq8+N2ieCy0gVn3OtRb0vBNNM= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 h1:c31//R3xgIJMSC8S6hEVq+38DcvUlgFY0FM6mSI5oto= +github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21/go.mod h1:r6+pf23ouCB718FUxaqzZdbpYFyDtehyZcmP5KL9FkA= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 h1:ZlvrNcHSFFWURB8avufQq9gFsheUgjVD9536obIknfM= +github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21/go.mod h1:cv3TNhVrssKR0O/xxLJVRfd2oazSnZnkUeTf6ctUwfQ= +github.com/aws/aws-sdk-go-v2/service/s3 v1.99.0 h1:hlSuz394kV0vhv9drL5lhuEFbEOEP1VyQpy15qWh1Pk= +github.com/aws/aws-sdk-go-v2/service/s3 v1.99.0/go.mod h1:uoA43SdFwacedBfSgfFSjjCvYe8aYBS7EnU5GZ/YKMM= +github.com/aws/aws-sdk-go-v2/service/signin v1.0.9 h1:QKZH0S178gCmFEgst8hN0mCX1KxLgHBKKY/CLqwP8lg= +github.com/aws/aws-sdk-go-v2/service/signin v1.0.9/go.mod h1:7yuQJoT+OoH8aqIxw9vwF+8KpvLZ8AWmvmUWHsGQZvI= +github.com/aws/aws-sdk-go-v2/service/sso v1.30.15 h1:lFd1+ZSEYJZYvv9d6kXzhkZu07si3f+GQ1AaYwa2LUM= +github.com/aws/aws-sdk-go-v2/service/sso v1.30.15/go.mod h1:WSvS1NLr7JaPunCXqpJnWk1Bjo7IxzZXrZi1QQCkuqM= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.19 h1:dzztQ1YmfPrxdrOiuZRMF6fuOwWlWpD2StNLTceKpys= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.19/go.mod h1:YO8TrYtFdl5w/4vmjL8zaBSsiNp3w0L1FfKVKenZT7w= +github.com/aws/aws-sdk-go-v2/service/sts v1.41.10 h1:p8ogvvLugcR/zLBXTXrTkj0RYBUdErbMnAFFp12Lm/U= +github.com/aws/aws-sdk-go-v2/service/sts v1.41.10/go.mod h1:60dv0eZJfeVXfbT1tFJinbHrDfSJ2GZl4Q//OSSNAVw= +github.com/aws/smithy-go v1.24.3 h1:XgOAaUgx+HhVBoP4v8n6HCQoTRDhoMghKqw4LNHsDNg= +github.com/aws/smithy-go v1.24.3/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc= +github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk= +github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg= +github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= +github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= +github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= +github.com/boombuler/barcode v1.1.0 h1:ChaYjBR63fr4LFyGn8E8nt7dBSt3MiU3zMOZqFvVkHo= +github.com/boombuler/barcode v1.1.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8= +github.com/bradenaw/juniper v0.15.3 h1:RHIAMEDTpvmzV1wg1jMAHGOoI2oJUSPx3lxRldXnFGo= +github.com/bradenaw/juniper v0.15.3/go.mod h1:UX4FX57kVSaDp4TPqvSjkAAewmRFAfXf27BOs5z9dq8= +github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 h1:GKTyiRCL6zVf5wWaqKnf+7Qs6GbEPfd4iMOitWzXJx8= +github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8/go.mod h1:spo1JLcs67NmW1aVLEgtA8Yy1elc+X8y5SRW1sFW4Og= +github.com/buengese/sgzip v0.1.1 h1:ry+T8l1mlmiWEsDrH/YHZnCVWD2S3im1KLsyO+8ZmTU= +github.com/buengese/sgzip v0.1.1/go.mod h1:i5ZiXGF3fhV7gL1xaRRL1nDnmpNj0X061FQzOS8VMas= +github.com/buger/jsonparser v1.1.2 h1:frqHqw7otoVbk5M8LlE/L7HTnIq2v9RX6EJ48i9AxJk= +github.com/buger/jsonparser v1.1.2/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0= +github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0= +github.com/bytedance/sonic v1.13.2 h1:8/H1FempDZqC4VqjptGo14QQlJx8VdZJegxs6wwfqpQ= +github.com/bytedance/sonic v1.13.2/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4= +github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU= +github.com/bytedance/sonic/loader v0.2.4 h1:ZWCw4stuXUsn1/+zQDqeE7JKP+QO47tz7QCNan80NzY= +github.com/bytedance/sonic/loader v0.2.4/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI= +github.com/calebcase/tmpfile v1.0.3 h1:BZrOWZ79gJqQ3XbAQlihYZf/YCV0H4KPIdM5K5oMpJo= +github.com/calebcase/tmpfile v1.0.3/go.mod h1:UAUc01aHeC+pudPagY/lWvt2qS9ZO5Zzof6/tIUzqeI= +github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8= +github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 h1:z0uK8UQqjMVYzvk4tiiu3obv2B44+XBsvgEJREQfnO8= +github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9/go.mod h1:Jl2neWsQaDanWORdqZ4emBl50J4/aRBBS4FyyG9/PFo= +github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI= +github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI= +github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= +github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= +github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I= +github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8= +github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4= +github.com/cloudinary/cloudinary-go/v2 v2.15.0 h1:iLoIwb7BJECHTbNcmIhYDsQhoZiACWGNvEpyqQy97Dk= +github.com/cloudinary/cloudinary-go/v2 v2.15.0/go.mod h1:ireC4gqVetsjVhYlwjUJwKTbZuWjEIynbR9zQTlqsvo= +github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc h1:t8YjNUCt1DimB4HCIXBztwWMhgxr5yG5/YaRl9Afdfg= +github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc/go.mod h1:CgWpFCFWzzEA5hVkhAc6DZZzGd3czx+BblvOzjmg6KA= +github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc h1:0xCWmFKBmarCqqqLeM7jFBSw/Or81UEElFqO8MY+GDs= +github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc/go.mod h1:uvR42Hb/t52HQd7x5/ZLzZEK8oihrFpgnodIJ1vte2E= +github.com/cloudwego/base64x v0.1.5 h1:XPciSp1xaq2VCSt6lF0phncD4koWyULpl5bUxbfCyP4= +github.com/cloudwego/base64x v0.1.5/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w= +github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= +github.com/colinmarc/hdfs/v2 v2.4.0 h1:v6R8oBx/Wu9fHpdPoJJjpGSUxo8NhHIwrwsfhFvU9W0= +github.com/colinmarc/hdfs/v2 v2.4.0/go.mod h1:0NAO+/3knbMx6+5pCv+Hcbaz4xn/Zzbn9+WIib2rKVI= +github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= +github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M= +github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE= +github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk= +github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I= +github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo= +github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A= +github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw= +github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4= +github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec= +github.com/coreos/go-systemd/v22 v22.6.0 h1:aGVa/v8B7hpb0TKl0MWoAavPDmHvobFe5R5zn0bCJWo= +github.com/coreos/go-systemd/v22 v22.6.0/go.mod h1:iG+pp635Fo7ZmV/j14KUcmEyWF+0X7Lua8rrTWzYgWU= +github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA= +github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc= +github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= +github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s= +github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE= +github.com/creasty/defaults v1.8.0 h1:z27FJxCAa0JKt3utc0sCImAEb+spPucmKoOdLHvHYKk= +github.com/creasty/defaults v1.8.0/go.mod h1:iGzKe6pbEHnpMPtfDXZEr0NVxWnPTjb1bbDy08fPzYM= +github.com/cronokirby/saferith v0.33.0/go.mod h1:QKJhjoqUtBsXCAVEjw38mFqoi7DebT7kthcD7UzbnoA= +github.com/cronokirby/saferith v0.33.1-0.20250226174546-1f11f94ce488 h1:tLWBZgPg6TV67oe76W4p+aUQEWIa52wbcuiz8GFd3vo= +github.com/cronokirby/saferith v0.33.1-0.20250226174546-1f11f94ce488/go.mod h1:QKJhjoqUtBsXCAVEjw38mFqoi7DebT7kthcD7UzbnoA= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/diskfs/go-diskfs v1.7.0 h1:vonWmt5CMowXwUc79jWyGrf2DIMeoOjkLlMnQYGVOs8= +github.com/diskfs/go-diskfs v1.7.0/go.mod h1:LhQyXqOugWFRahYUSw47NyZJPezFzB9UELwhpszLP/k= +github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= +github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= +github.com/djherbis/times v1.6.0 h1:w2ctJ92J8fBvWPxugmXIv7Nz7Q3iDMKNx9v5ocVH20c= +github.com/djherbis/times v1.6.0/go.mod h1:gOHeRAz2h+VJNZ5Gmc/o7iD9k4wW7NMVqieYCY99oc0= +github.com/dnaeon/go-vcr v1.2.0 h1:zHCHvJYTMh1N7xnV7zf1m1GPBF9Ad0Jk/whtQ1663qI= +github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ= +github.com/docker/go-connections v0.6.0 h1:LlMG9azAe1TqfR7sO+NJttz1gy6KO7VJBh+pMmjSD94= +github.com/docker/go-connections v0.6.0/go.mod h1:AahvXYshr6JgfUJGdDCs2b5EZG/vmaMAntpSFH5BFKE= +github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= +github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/dromara/dongle v1.0.1 h1:si/7UP/EXxnFVZok1cNos70GiMGxInAYMilHQFP5dJs= +github.com/dromara/dongle v1.0.1/go.mod h1:ebFhTaDgxaDIKppycENTWlBsxz8mWCPWOLnsEgDpMv4= +github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5 h1:FT+t0UEDykcor4y3dMVKXIiWJETBpRgERYTGlmMd7HU= +github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5/go.mod h1:rSS3kM9XMzSQ6pw91Qgd6yB5jdt70N4OdtrAf74As5M= +github.com/dsnet/try v0.0.3 h1:ptR59SsrcFUYbT/FhAbKTV6iLkeD6O18qfIWRml2fqI= +github.com/dsnet/try v0.0.3/go.mod h1:WBM8tRpUmnXXhY1U6/S8dt6UWdHTQ7y8A5YSkRCkq40= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/ebitengine/purego v0.10.0 h1:QIw4xfpWT6GWTzaW5XEKy3HXoqrJGx1ijYHzTF0/ISU= +github.com/ebitengine/purego v0.10.0/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ= +github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab h1:h1UgjJdAAhj+uPL68n7XASS6bU+07ZX1WJvVS2eyoeY= +github.com/elliotwutingfeng/asciiset v0.0.0-20230602022725-51bbb787efab/go.mod h1:GLo/8fDswSAniFG+BFIaiSPcK610jyzgEhWYPQwuQdw= +github.com/emersion/go-message v0.18.2 h1:rl55SQdjd9oJcIoQNhubD2Acs1E6IzlZISRTK7x/Lpg= +github.com/emersion/go-message v0.18.2/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA= +github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff h1:4N8wnS3f1hNHSmFD5zgFkWCyA4L1kCDkImPAtK7D6tg= +github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff/go.mod h1:HMJKR5wlh/ziNp+sHEDV2ltblO4JD2+IdDOWtGcQBTM= +github.com/emmansun/gmsm v0.15.5/go.mod h1:2m4jygryohSWkaSduFErgCwQKab5BNjURoFrn2DNwyU= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/fatih/color v1.16.0 h1:zmkK9Ngbjj+K0yRhTVONQh1p/HknKYSlNT+vZCzyokM= +github.com/fatih/color v1.16.0/go.mod h1:fL2Sau1YI5c0pdGEVCbKQbLXB6edEj1ZgiY4NijnWvE= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= +github.com/flynn/noise v1.1.0 h1:KjPQoQCEFdZDiP03phOvGi11+SVVhBG2wOWAorLsstg= +github.com/flynn/noise v1.1.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag= +github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= +github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= +github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM= +github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s= +github.com/geoffgarside/ber v1.1.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc= +github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64= +github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc= +github.com/gin-contrib/cors v1.7.5 h1:cXC9SmofOrRg0w9PigwGlHG3ztswH6bqq4vJVXnvYMk= +github.com/gin-contrib/cors v1.7.5/go.mod h1:4q3yi7xBEDDWKapjT2o1V7mScKDDr8k+jZ0fSquGoy0= +github.com/gin-contrib/gzip v1.2.3 h1:dAhT722RuEG330ce2agAs75z7yB+NKvX/ZM1r8w0u2U= +github.com/gin-contrib/gzip v1.2.3/go.mod h1:ad72i4Bzmaypk8M762gNXa2wkxxjbz0icRNnuLJ9a/c= +github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w= +github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM= +github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU= +github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y= +github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug= +github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0= +github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68= +github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo= +github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA= +github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw= +github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU= +github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= +github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= +github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A= +github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= +github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE= +github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78= +github.com/go-openapi/errors v0.22.6 h1:eDxcf89O8odEnohIXwEjY1IB4ph5vmbUsBMsFNwXWPo= +github.com/go-openapi/errors v0.22.6/go.mod h1:z9S8ASTUqx7+CP1Q8dD8ewGH/1JWFFLX/2PmAYNQLgk= +github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonpointer v0.19.5 h1:gZr+CIYByUqjcgeLXnQu2gHYQC9o73G2XUeOFYEICuY= +github.com/go-openapi/jsonpointer v0.19.5/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg= +github.com/go-openapi/jsonreference v0.19.6 h1:UBIxjkht+AWIgYzCDSv2GN+E/togfwXUJFRTWhl2Jjs= +github.com/go-openapi/jsonreference v0.19.6/go.mod h1:diGHMEHg2IqXZGKxqyvWdfWU/aim5Dprw5bqpKkTvns= +github.com/go-openapi/spec v0.20.4 h1:O8hJrt0UMnhHcluhIdUgCLRWyM2x7QkBXRvOs7m+O1M= +github.com/go-openapi/spec v0.20.4/go.mod h1:faYFR1CvsJZ0mNsmsphTMSoRrNV3TEDoAM7FOEWeq8I= +github.com/go-openapi/strfmt v0.25.0 h1:7R0RX7mbKLa9EYCTHRcCuIPcaqlyQiWNPTXwClK0saQ= +github.com/go-openapi/strfmt v0.25.0/go.mod h1:nNXct7OzbwrMY9+5tLX4I21pzcmE6ccMGXl3jFdPfn8= +github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk= +github.com/go-openapi/swag v0.19.15 h1:D2NRCBzS9/pEY3gP9Nl8aDqGUcPFrwG2p+CNFrLyrCM= +github.com/go-openapi/swag v0.19.15/go.mod h1:QYRuS/SOXUCsnplDa677K7+DxSOj6IPNl/eQntq43wQ= +github.com/go-openapi/testify/v2 v2.0.2 h1:X999g3jeLcoY8qctY/c/Z8iBHTbwLz7R2WXd6Ub6wls= +github.com/go-openapi/testify/v2 v2.0.2/go.mod h1:HCPmvFFnheKK2BuwSA0TbbdxJ3I16pjwMkYkP4Ywn54= +github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s= +github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4= +github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA= +github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY= +github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY= +github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY= +github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy06ntQJp0BBvFG0w= +github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM= +github.com/go-resty/resty/v2 v2.17.2 h1:FQW5oHYcIlkCNrMD2lloGScxcHJ0gkjshV3qcQAyHQk= +github.com/go-resty/resty/v2 v2.17.2/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA= +github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg= +github.com/go-sql-driver/mysql v1.9.2 h1:4cNKDYQ1I84SXslGddlsrMhc8k4LeDVj6Ad6WRjiHuU= +github.com/go-sql-driver/mysql v1.9.2/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU= +github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI= +github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8= +github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro= +github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= +github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= +github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= +github.com/gofrs/flock v0.13.0 h1:95JolYOvGMqeH31+FC7D2+uULf6mG61mEZ/A8dRYMzw= +github.com/gofrs/flock v0.13.0/go.mod h1:jxeyy9R1auM5S6JYDBhDt+E2TCo7DkratH4Pgi8P+Z0= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI= +github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= +github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY= +github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y= +github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= +github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= +github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw= +github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= +github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= +github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg= +github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= +github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= +github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= +github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= +github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= +github.com/google/pprof v0.0.0-20240509144519-723abb6459b7 h1:velgFPYr1X9TDwLIfkV7fWqsFlf7TeP11M/7kPd/dVI= +github.com/google/pprof v0.0.0-20240509144519-723abb6459b7/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw= +github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= +github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0= +github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.3.14 h1:yh8ncqsbUY4shRD5dA6RlzjJaT4hi3kII+zYw8wmLb8= +github.com/googleapis/enterprise-certificate-proxy v0.3.14/go.mod h1:vqVt9yG9480NtzREnTlmGSBmFrA+bzb0yl0TxoBQXOg= +github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= +github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= +github.com/googleapis/gax-go/v2 v2.21.0 h1:h45NjjzEO3faG9Lg/cFrBh2PgegVVgzqKzuZl/wMbiI= +github.com/googleapis/gax-go/v2 v2.21.0/go.mod h1:But/NJU6TnZsrLai/xBAQLLz+Hc7fHZJt/hsCz3Fih4= +github.com/gorilla/schema v1.4.1 h1:jUg5hUjCSDZpNGLuXQOgIWGdlgrIdYvgQ0wZtdK1M3E= +github.com/gorilla/schema v1.4.1/go.mod h1:Dg5SSm5PV60mhF2NFaTV1xuYYj8tV8NOPRo4FggUMnM= +github.com/gorilla/securecookie v1.1.1 h1:miw7JPhV+b/lAHSXz4qd/nN9jRiAFV5FwjeKyCS8BvQ= +github.com/gorilla/securecookie v1.1.1/go.mod h1:ra0sb63/xPlUeL+yeDciTfxMRAA+MP+HVt/4epWDjd4= +github.com/gorilla/sessions v1.2.1 h1:DHd3rPN5lE3Ts3D8rKkQ8x/0kqfeNmBAaiSi+o7FsgI= +github.com/gorilla/sessions v1.2.1/go.mod h1:dk2InVEVJ0sfLlnXv9EAgkf6ecYs/i80K/zI+bUmuGM= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= +github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= +github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= +github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k= +github.com/hashicorp/go-hclog v1.6.3/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M= +github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= +github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= +github.com/hashicorp/go-retryablehttp v0.7.8 h1:ylXZWnqa7Lhqpk0L1P1LzDtGcCR0rPVUrx/c8Unxc48= +github.com/hashicorp/go-retryablehttp v0.7.8/go.mod h1:rjiScheydd+CxvumBsIrFKlx3iS0jrZ7LvzFGFmuKbw= +github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/go-uuid v1.0.3 h1:2gKiV6YVmrJ1i2CKKa9obLvRieoRGviZFL26PcT/Co8= +github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= +github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hirochachacha/go-smb2 v1.1.0 h1:b6hs9qKIql9eVXAiN0M2wSFY5xnhbHAQoCwRKbaRTZI= +github.com/hirochachacha/go-smb2 v1.1.0/go.mod h1:8F1A4d5EZzrGu5R7PU163UcMRDJQl4FtcxjBfsY8TZE= +github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= +github.com/ilyakaznacheev/cleanenv v1.5.0 h1:0VNZXggJE2OYdXE87bfSSwGxeiGt9moSR2lOrsHHvr4= +github.com/ilyakaznacheev/cleanenv v1.5.0/go.mod h1:a5aDzaJrLCQZsazHol1w8InnDcOX0OColm64SlIi6gk= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/internxt/rclone-adapter v0.0.0-20260331173834-036f908d0160 h1:PV6ipOJN0JIek4dqPqsTtenQmjI1SZntCUgPzhfk9gM= +github.com/internxt/rclone-adapter v0.0.0-20260331173834-036f908d0160/go.mod h1:yRuPDnHgGmsbvopF0amMqXxr4n32GynzX5GTwFYDHaw= +github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= +github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= +github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= +github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw= +github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= +github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= +github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8= +github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs= +github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo= +github.com/jcmturner/dnsutils/v2 v2.0.0/go.mod h1:b0TnjGOvI/n42bZa+hmXL+kFJZsFT7G4t3HTlQ184QM= +github.com/jcmturner/gofork v1.7.6 h1:QH0l3hzAU1tfT3rZCnW5zXl+orbkNMMRGJfdJjHVETg= +github.com/jcmturner/gofork v1.7.6/go.mod h1:1622LH6i/EZqLloHfE7IeZ0uEJwMSUyQ/nDd82IeqRo= +github.com/jcmturner/goidentity/v6 v6.0.1 h1:VKnZd2oEIMorCTsFBnJWbExfNN7yZr3EhJAxwOkZg6o= +github.com/jcmturner/goidentity/v6 v6.0.1/go.mod h1:X1YW3bgtvwAXju7V3LCIMpY0Gbxyjn/mY9zx4tFonSg= +github.com/jcmturner/gokrb5/v8 v8.4.4 h1:x1Sv4HaTpepFkXbt2IkL29DXRf8sOfZXo8eRKh687T8= +github.com/jcmturner/gokrb5/v8 v8.4.4/go.mod h1:1btQEpgT6k+unzCwX1KdWMEwPPkkgBtP+F6aCACiMrs= +github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY= +github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc= +github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E= +github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc= +github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ= +github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8= +github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb h1:6vkM8gO+zFV2m21QzGYyUSq5TP0VQgP2Xz3UQyCN2kI= +github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb/go.mod h1:H1+whwD0Qe3YOunlXIWhh3rlvzW5cZfkMDYGQPg+KAM= +github.com/jmoiron/sqlx v1.4.0 h1:1PLqN7S1UYp5t4SrVVnt4nUVNemrDAtxlulVe+Qgm3o= +github.com/jmoiron/sqlx v1.4.0/go.mod h1:ZrZ7UsYB/weZdl2Bxg6jCRO9c3YHl8r3ahlKmRT4JLY= +github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0= +github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4= +github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= +github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= +github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= +github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= +github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= +github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk= +github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 h1:JcltaO1HXM5S2KYOYcKgAV7slU0xPy1OcvrVgn98sRQ= +github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7/go.mod h1:MEkhEPFwP3yudWO0lj6vfYpLIB+3eIcuIW+e0AZzUQk= +github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 h1:G+9t9cEtnC9jFiTxyptEKuNIAbiN5ZCQzX2a74lj3xg= +github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004/go.mod h1:KmHnJWQrgEvbuy0vcvj00gtMqbvNn1L+3YUZLK/B92c= +github.com/keybase/go-keychain v0.0.1 h1:way+bWYa6lDppZoZcgMbYsvC7GxljxrskdNInRtuthU= +github.com/keybase/go-keychain v0.0.1/go.mod h1:PdEILRW3i9D8JcdM+FmY6RwkHGnhHxXwkPPMeUgOK1k= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE= +github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= +github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM= +github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw= +github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M= +github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988 h1:CjEMN21Xkr9+zwPmZPaJJw+apzVbjGL5uK/6g9Q2jGU= +github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988/go.mod h1:/agobYum3uo/8V6yPVnq+R82pyVGCeuWW5arT4Txn8A= +github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6 h1:FHVoZMOVRA+6/y4yRlbiR3WvsrOcKBd/f64H7YiWR2U= +github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6/go.mod h1:MRAz4Gsxd+OzrZ0owwrUHc0zLESL+1Y5syqK/sJxK2A= +github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8= +github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= +github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= +github.com/lanrat/extsort v1.4.2 h1:akbLIdo4PhNZtvjpaWnbXtGMmLtnGzXplkzfgl+XTTY= +github.com/lanrat/extsort v1.4.2/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4= +github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ= +github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI= +github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= +github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= +github.com/lpar/date v1.0.0 h1:bq/zVqFTUmsxvd/CylidY4Udqpr9BOFrParoP6p0x/I= +github.com/lpar/date v1.0.0/go.mod h1:KjYe0dDyMQTgpqcUz4LEIeM5VZwhggjVx/V2dtc8NSo= +github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 h1:PTw+yKnXcOFCR6+8hHTyWBeQ/P4Nb7dd4/0ohEcWQuM= +github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= +github.com/magiconair/properties v1.8.10 h1:s31yESBquKXCV9a/ScB3ESkOjUYYv+X0rg8SYxI99mE= +github.com/magiconair/properties v1.8.10/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0= +github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc= +github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= +github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8= +github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU= +github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= +github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU= +github.com/mattn/go-runewidth v0.0.22 h1:76lXsPn6FyHtTY+jt2fTTvsMUCZq1k0qwRsAMuxzKAk= +github.com/mattn/go-runewidth v0.0.22/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs= +github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU= +github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y= +github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI= +github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg= +github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= +github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= +github.com/minio/minio-go/v7 v7.0.100 h1:ShkWi8Tyj9RtU57OQB2HIXKz4bFgtVib0bbT1sbtLI8= +github.com/minio/minio-go/v7 v7.0.100/go.mod h1:EtGNKtlX20iL2yaYnxEigaIvj0G0GwSDnifnG8ClIdw= +github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= +github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo= +github.com/moby/go-archive v0.2.0 h1:zg5QDUM2mi0JIM9fdQZWC7U8+2ZfixfTYoHL7rWUcP8= +github.com/moby/go-archive v0.2.0/go.mod h1:mNeivT14o8xU+5q1YnNrkQVpK+dnNe/K6fHqnTg4qPU= +github.com/moby/moby/api v1.54.1 h1:TqVzuJkOLsgLDDwNLmYqACUuTehOHRGKiPhvH8V3Nn4= +github.com/moby/moby/api v1.54.1/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs= +github.com/moby/moby/client v0.4.0 h1:S+2XegzHQrrvTCvF6s5HFzcrywWQmuVnhOXe2kiWjIw= +github.com/moby/moby/client v0.4.0/go.mod h1:QWPbvWchQbxBNdaLSpoKpCdf5E+WxFAgNHogCWDoa7g= +github.com/moby/patternmatcher v0.6.1 h1:qlhtafmr6kgMIJjKJMDmMWq7WLkKIo23hsrpR3x084U= +github.com/moby/patternmatcher v0.6.1/go.mod h1:hDPoyOpDY7OrrMDLaYoY3hf52gNCR/YOUYxkhApJIxc= +github.com/moby/sys/mountinfo v0.7.2 h1:1shs6aH5s4o5H2zQLn796ADW1wMrIwHsyJ2v9KouLrg= +github.com/moby/sys/mountinfo v0.7.2/go.mod h1:1YOa8w8Ih7uW0wALDUgT1dTTSBrZ+HiBLGws92L2RU4= +github.com/moby/sys/sequential v0.6.0 h1:qrx7XFUd/5DxtqcoH1h438hF5TmOvzC/lspjy7zgvCU= +github.com/moby/sys/sequential v0.6.0/go.mod h1:uyv8EUTrca5PnDsdMGXhZe6CCe8U/UiTWd+lL+7b/Ko= +github.com/moby/sys/user v0.4.0 h1:jhcMKit7SA80hivmFJcbB1vqmw//wU61Zdui2eQXuMs= +github.com/moby/sys/user v0.4.0/go.mod h1:bG+tYYYJgaMtRKgEmuueC0hJEAZWwtIbZTB+85uoHjs= +github.com/moby/sys/userns v0.1.0 h1:tVLXkFOxVu9A64/yh59slHVv9ahO9UIev4JZusOLG/g= +github.com/moby/sys/userns v0.1.0/go.mod h1:IHUYgu/kao6N8YZlp9Cf444ySSvCmDlmzUcYfDHOl28= +github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ= +github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc= +github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M= +github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk= +github.com/montanaflynn/stats v0.7.1 h1:etflOAAHORrCC44V+aR6Ftzort912ZU+YLiSTuV8eaE= +github.com/montanaflynn/stats v0.7.1/go.mod h1:etXPPgVO6n31NxCd9KQUMvCM+ve0ruNzt6R8Bnaayow= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= +github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= +github.com/ncw/swift/v2 v2.0.5 h1:9o5Gsd7bInAFEqsGPcaUdsboMbqf8lnNtxqWKFT9iz8= +github.com/ncw/swift/v2 v2.0.5/go.mod h1:cbAO76/ZwcFrFlHdXPjaqWZ9R7Hdar7HpjRXBfbjigk= +github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= +github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= +github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= +github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4= +github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U= +github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= +github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= +github.com/onsi/ginkgo/v2 v2.17.3 h1:oJcvKpIb7/8uLpDDtnQuf18xVnwKp8DTD7DQ6gTd/MU= +github.com/onsi/ginkgo/v2 v2.17.3/go.mod h1:nP2DPOQoNsQmsVyv5rDA8JkXQoCs6goXIvr/PRJ1eCc= +github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM= +github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= +github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M= +github.com/oracle/oci-go-sdk/v65 v65.111.0 h1:eDkWg6ZN0uKwWzSekoFcQJhR+C+F/aVdTwr+lGHU9Qk= +github.com/oracle/oci-go-sdk/v65 v65.111.0/go.mod h1:8ZzvzuEG/cFLFZhxg/Mg1w19KqyXBKO3c17QIc5PkGs= +github.com/panjf2000/ants/v2 v2.11.5 h1:a7LMnMEeux/ebqTux140tRiaqcFTV0q2bEHF03nl6Rg= +github.com/panjf2000/ants/v2 v2.11.5/go.mod h1:8u92CYMUc6gyvTIw8Ru7Mt7+/ESnJahz5EVtqfrilek= +github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= +github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ= +github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4= +github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY= +github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14 h1:XeOYlK9W1uCmhjJSsY78Mcuh7MVkNjTzmHx1yBzizSU= +github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14/go.mod h1:jVblp62SafmidSkvWrXyxAme3gaTfEtWwRPGz5cpvHg= +github.com/peterh/liner v1.2.2 h1:aJ4AOodmL+JxOZZEL2u9iJf8omNRpqHc/EbrK+3mAXw= +github.com/peterh/liner v1.2.2/go.mod h1:xFwJyiKIXJZUKItq5dGHZSTBRAuG/CpeNpWLyiNRNwI= +github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= +github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= +github.com/pierrec/lz4/v4 v4.1.25 h1:kocOqRffaIbU5djlIBr7Wh+cx82C0vtFb0fOurZHqD0= +github.com/pierrec/lz4/v4 v4.1.25/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= +github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= +github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU= +github.com/pkg/diff v0.0.0-20200914180035-5b29258ca4f7/go.mod h1:zO8QMzTeZd5cpnIkz/Gn6iK0jDfGicM1nynOkkPIl28= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/sftp v1.13.10 h1:+5FbKNTe5Z9aspU88DPIKJ9z2KZoaGCu6Sr6kKR/5mU= +github.com/pkg/sftp v1.13.10/go.mod h1:bJ1a7uDhrX/4OII+agvy28lzRvQrmIQuaHrcI1HbeGA= +github.com/pkg/xattr v0.4.12 h1:rRTkSyFNTRElv6pkA3zpjHpQ90p/OdHQC1GmGh1aTjM= +github.com/pkg/xattr v0.4.12/go.mod h1:di8WF84zAKk8jzR1UBTEWh9AUlIZZ7M/JNt8e9B6ktU= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU= +github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE= +github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs= +github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg= +github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= +github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= +github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= +github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4= +github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw= +github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws= +github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw= +github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8 h1:Y258uzXU/potCYnQd1r6wlAnoMB68BiCkCcCnKx1SH8= +github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8/go.mod h1:bSJjRokAHHOhA+XFxplld8w2R/dXLH7Z3BZ532vhFwU= +github.com/quic-go/quic-go v0.59.0 h1:OLJkp1Mlm/aS7dpKgTc6cnpynnD2Xg7C1pwL6vy/SAw= +github.com/quic-go/quic-go v0.59.0/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU= +github.com/rclone/Proton-API-Bridge v1.0.3 h1:Bs7RC4xCFSN0BPIYVda/BNxp0qo3NV0gB2VZqx2KIew= +github.com/rclone/Proton-API-Bridge v1.0.3/go.mod h1:26RAest751Ofk+F/d8xtl4UyWXrZvMQwn39U8rm/WKM= +github.com/rclone/go-proton-api v1.0.2 h1:cJtJUab0MGJ3C6q5kiEJs3pbyhSLnOKMyYOQehA0PBc= +github.com/rclone/go-proton-api v1.0.2/go.mod h1:LB2kCEaZMzNn3ocdz+qYfxXmuLxxN0ka62KJd2x53Bc= +github.com/rclone/rclone v1.74.3 h1:a2wln7pvEa0tS1WIZJKulEkVjxgC1DkCoyxYydkdiSY= +github.com/rclone/rclone v1.74.3/go.mod h1:t5Mh86PO49DD7xlPt0trnK/aNf2Z3M0uip4l1Jqwiv8= +github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo= +github.com/relvacode/iso8601 v1.7.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I= +github.com/rfjakob/eme v1.2.0 h1:8dAHL+WVAw06+7DkRKnRiFp1JL3QjcJEZFqDnndUaSI= +github.com/rfjakob/eme v1.2.0/go.mod h1:cVvpasglm/G3ngEfcfT/Wt0GwhkuO32pf/poW6Nyk1k= +github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= +github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= +github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= +github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ= +github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc= +github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU= +github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= +github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06 h1:OkMGxebDjyw0ULyrTYWeN0UNCCkmCWfjPnIA2W6oviI= +github.com/sabhiram/go-gitignore v0.0.0-20210923224102-525f6e181f06/go.mod h1:+ePHsJ1keEjQtpvf9HHw0f4ZeJ0TLRsxhunSI2hYJSs= +github.com/samber/lo v1.52.0 h1:Rvi+3BFHES3A8meP33VPAxiBZX/Aws5RxrschYGjomw= +github.com/samber/lo v1.52.0/go.mod h1:4+MXEGsJzbKGaUEQFKBq2xtfuznW9oz/WrgyzMzRoM0= +github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo= +github.com/shirou/gopsutil/v4 v4.26.3 h1:2ESdQt90yU3oXF/CdOlRCJxrP+Am1aBYubTMTfxJ1qc= +github.com/shirou/gopsutil/v4 v4.26.3/go.mod h1:LZ6ewCSkBqUpvSOf+LsTGnRinC6iaNUNMGBtDkJBaLQ= +github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0= +github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w= +github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g= +github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 h1:JIAuq3EEf9cgbU6AtGPK4CTG3Zf6CKMNqf0MHTggAUA= +github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966/go.mod h1:sUM3LWHvSMaG192sy56D9F7CNvL7jUJVXoqM1QKLnog= +github.com/snabb/httpreaderat v1.0.1 h1:whlb+vuZmyjqVop8x1EKOg05l2NE4z9lsMMXjmSUCnY= +github.com/snabb/httpreaderat v1.0.1/go.mod h1:lpbGrKDWF37yvRbtRvQsbesS6Ty5c83t8ztannPoMsA= +github.com/sony/gobreaker v1.0.0 h1:feX5fGGXSl3dYd4aHZItw+FpHLvvoaqkawKjVNiFMNQ= +github.com/sony/gobreaker v1.0.0/go.mod h1:ZKptC7FHNvhBz7dN2LGjPVBz2sZJmc0/PkyDJOjmxWY= +github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368 h1:GyYC5Ntqk/yy9lEIGE7chdIvt4zP44taycwd9YDSGdc= +github.com/spacemonkeygo/monkit/v3 v3.0.25-0.20251022131615-eb24eb109368/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA= +github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= +github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= +github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk= +github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/objx v0.5.3 h1:jmXUvGomnU1o3W/V5h2VEradbpJDwGrzugQQvL0POH4= +github.com/stretchr/objx v0.5.3/go.mod h1:rDQraq+vQZU7Fde9LOZLr8Tax6zZvy4kuNKF+QYS+U0= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.3.1-0.20190311161405-34c6fa2dc709/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/swaggo/files v1.0.1 h1:J1bVJ4XHZNq0I46UU90611i9/YzdrF7x92oX1ig5IdE= +github.com/swaggo/files v1.0.1/go.mod h1:0qXmMNH6sXNf+73t65aKeB+ApmgxdnkQzVTAj2uaMUg= +github.com/swaggo/gin-swagger v1.6.0 h1:y8sxvQ3E20/RCyrXeFfg60r6H0Z+SwpTjMYsMm+zy8M= +github.com/swaggo/gin-swagger v1.6.0/go.mod h1:BG00cCEy294xtVpyIAHG6+e2Qzj/xKlRdOqDkvq0uzo= +github.com/swaggo/swag v1.16.4 h1:clWJtd9LStiG3VeijiCfOVODP6VpHtKdQy9ELFG3s1A= +github.com/swaggo/swag v1.16.4/go.mod h1:VBsHJRsDvfYvqoiMKnsdwhNV9LEMHgEDZcyVYX0sxPg= +github.com/t3rm1n4l/go-mega v0.0.0-20251120131202-6845944c051c h1:dtcOwRimeiBFrlutmF6K94l0rxYFARNFMA+lSQ41C+M= +github.com/t3rm1n4l/go-mega v0.0.0-20251120131202-6845944c051c/go.mod h1:BF/l2jNyK+2h/BJZ7VLMAz6m/IWjA2F67gTjV1C/+Bo= +github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502/go.mod h1:p9lPsd+cx33L3H9nNoecRRxPssFKUwwI50I3pZ0yT+8= +github.com/testcontainers/testcontainers-go v0.42.0 h1:He3IhTzTZOygSXLJPMX7n44XtK+qhjat1nI9cneBbUY= +github.com/testcontainers/testcontainers-go v0.42.0/go.mod h1:vZjdY1YmUA1qEForxOIOazfsrdyORJAbhi0bp8plN30= +github.com/tinylib/msgp v1.6.3 h1:bCSxiTz386UTgyT1i0MSCvdbWjVW+8sG3PjkGsZQt4s= +github.com/tinylib/msgp v1.6.3/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA= +github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA= +github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI= +github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw= +github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ= +github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI= +github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08= +github.com/tyler-smith/go-bip39 v1.1.0 h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8= +github.com/tyler-smith/go-bip39 v1.1.0/go.mod h1:gUYDtqQw1JS3ZJ8UWVcGTGqqr6YIN3CWg+kkNaLt55U= +github.com/ugorji/go/codec v1.2.12 h1:9LC83zGrHhuUA9l16C9AHXAqEV/2wBQ4nkvumAE65EE= +github.com/ugorji/go/codec v1.2.12/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg= +github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY= +github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/unknwon/goconfig v1.0.0 h1:rS7O+CmUdli1T+oDm7fYj1MwqNWtEJfNj+FqcUHML8U= +github.com/unknwon/goconfig v1.0.0/go.mod h1:qu2ZQ/wcC/if2u32263HTVC39PeOQRSmidQk3DuDFQ8= +github.com/valkey-io/valkey-go v1.0.70 h1:mjYNT8qiazxDAJ0QNQ8twWT/YFOkOoRd40ERV2mB49Y= +github.com/valkey-io/valkey-go v1.0.70/go.mod h1:VGhZ6fs68Qrn2+OhH+6waZH27bjpgQOiLyUQyXuYK5k= +github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc= +github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw= +github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM= +github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw= +github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c= +github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI= +github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY= +github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4= +github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8= +github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM= +github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM= +github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfSfmXjznFBSZNN13rSJjlIOI1fUNAtF7rmI= +github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= +github.com/yunify/qingstor-sdk-go/v3 v3.2.0 h1:9sB2WZMgjwSUNZhrgvaNGazVltoFUUfuS9f0uCWtTr8= +github.com/yunify/qingstor-sdk-go/v3 v3.2.0/go.mod h1:KciFNuMu6F4WLk9nGwwK69sCGKLCdd9f97ac/wfumS4= +github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0= +github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= +github.com/zeebo/assert v1.3.1 h1:vukIABvugfNMZMQO1ABsyQDJDTVQbn+LWSMy1ol1h6A= +github.com/zeebo/assert v1.3.1/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0= +github.com/zeebo/blake3 v0.2.4 h1:KYQPkhpRtcqh0ssGYcKLG1JYvddkEA8QwCM/yBqhaZI= +github.com/zeebo/blake3 v0.2.4/go.mod h1:7eeQ6d2iXWRGF6npfaxl2CU+xy2Fjo2gxeyZGCRUjcE= +github.com/zeebo/errs v1.4.0 h1:XNdoD/RRMKP7HD0UhJnIzUy74ISdGGxURlYG8HSWSfM= +github.com/zeebo/errs v1.4.0/go.mod h1:sgbWHsvVuTPHcqJJGQ1WhI5KbWlHYz+2+2C/LSEtCw4= +github.com/zeebo/mwc v0.0.7 h1:0NerGhCww6ZQx+/xCx5iwznftveokvto1KILpYfENZk= +github.com/zeebo/mwc v0.0.7/go.mod h1:0B32or6moOig1YGuqMoimBpU9QK9uYaGG2bBOuddqtE= +github.com/zeebo/pcg v1.0.1 h1:lyqfGeWiv4ahac6ttHs+I5hwtH/+1mrhlCtVNQM2kHo= +github.com/zeebo/pcg v1.0.1/go.mod h1:09F0S9iiKrwn9rlI5yjLkmrug154/YRW6KnnXVDM/l4= +github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs= +github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s= +go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo= +go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E= +go.mongodb.org/mongo-driver v1.17.9 h1:IexDdCuuNJ3BHrELgBlyaH9p60JXAvdzWR128q+U5tU= +go.mongodb.org/mongo-driver v1.17.9/go.mod h1:LlOhpH5NUEfhxcAwG0UEkMqwYcc4JU18gtCdGudk/tQ= +go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= +go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 h1:OyrsyzuttWTSur2qN/Lm0m2a8yqyIjUVBZcxFPuXq2o= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0/go.mod h1:C2NGBr+kAB4bk3xtMXfZ94gqFDtg/GkI7e9zqGh5Beg= +go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I= +go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0= +go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM= +go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY= +go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg= +go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg= +go.opentelemetry.io/otel/sdk/metric v1.42.0 h1:D/1QR46Clz6ajyZ3G8SgNlTJKBdGp84q9RKCAZ3YGuA= +go.opentelemetry.io/otel/sdk/metric v1.42.0/go.mod h1:Ua6AAlDKdZ7tdvaQKfSmnFTdHx37+J4ba8MwVCYM5hc= +go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A= +go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0= +go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8= +go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= +go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= +golang.org/x/arch v0.17.0 h1:4O3dfLzd+lQewptAHqjewQZQDyEdejz3VwgeYwkZneU= +golang.org/x/arch v0.17.0/go.mod h1:bdwinDaKcfZUGpH09BB7ZmOfhalA8lQdzl62l8gGWsk= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.0.0-20200728195943-123391ffb6de/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= +golang.org/x/crypto v0.4.0/go.mod h1:3quD/ATkf6oY+rnes5c3ExXTbLc8mueNue5/DoinL80= +golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58= +golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU= +golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc= +golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU= +golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8= +golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= +golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= +golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= +golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek= +golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY= +golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4= +golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= +golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= +golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM= +golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80= +golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= +golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= +golang.org/x/image v0.41.0 h1:8wS72eGJMJaBxK6okTzd4WaXumUlTVlb753MlsSvTCo= +golang.org/x/image v0.41.0/go.mod h1:uIc348UZMSvS5Z65CVZ7iDPaNobNFEPeJ4kbqTOszmA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs= +golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= +golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= +golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= +golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= +golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= +golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= +golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= +golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= +golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM= +golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.3.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc= +golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= +golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk= +golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= +golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= +golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4= +golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= +golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= +golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= +golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y= +golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= +golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220408201424-a24fb2fb8a0f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= +golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= +golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= +golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU= +golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk= +golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY= +golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM= +golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= +golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= +golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ= +golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= +golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= +golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= +golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= +golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= +golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= +golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw= +golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= +golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= +golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= +golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA= +golang.org/x/tools v0.0.0-20201211185031-d93e913c1a58/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58= +golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk= +golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= +golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE= +google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M= +google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= +google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg= +google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI= +google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE= +google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= +google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE= +google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM= +google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc= +google.golang.org/api v0.275.0 h1:vfY5d9vFVJeWEZT65QDd9hbndr7FyZ2+6mIzGAh71NI= +google.golang.org/api v0.275.0/go.mod h1:Fnag/EWUPIcJXuIkP1pjoTgS5vdxlk3eeemL7Do6bvw= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0= +google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8= +google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc= +google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA= +google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= +google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= +google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA= +google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= +google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= +google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= +google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 h1:XzmzkmB14QhVhgnawEVsOn6OFsnpyxNPRY9QV01dNB0= +google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:L43LFes82YgSonw6iTXTxXUX1OlULt4AQtkik4ULL/I= +google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 h1:41r6JMbpzBMen0R/4TZeeAmGXSJC7DftGINUodzTkPI= +google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:EIQZ5bFCfRQDV4MhRle7+OgjNtZ6P1PiZBgAKuxXu/Y= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 h1:m8qni9SQFH0tJc1X0vmnpw/0t+AImlSvp30sEupozUg= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= +google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk= +google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= +google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak= +google.golang.org/grpc v1.80.0 h1:Xr6m2WmWZLETvUNvIUmeD5OAagMw3FiKmMlTdViWsHM= +google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= +google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/validator.v2 v2.0.1 h1:xF0KWyGWXm/LM2G1TrEjqOu4pa6coO9AlWSf3msVfDY= +gopkg.in/validator.v2 v2.0.1/go.mod h1:lIUZBlB3Im4s/eYp39Ry/wkR02yOPhZ9IwIRBjuPuG8= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= +gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +gorm.io/driver/postgres v1.5.11 h1:ubBVAfbKEUld/twyKZ0IYn9rSQh448EdelLYk9Mv314= +gorm.io/driver/postgres v1.5.11/go.mod h1:DX3GReXH+3FPWGrrgffdvCk3DQ1dwDPdmbenSkweRGI= +gorm.io/gorm v1.26.1 h1:ghB2gUI9FkS46luZtn6DLZ0f6ooBJ5IbVej2ENFDjRw= +gorm.io/gorm v1.26.1/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE= +gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q= +gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= +honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= +honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= +moul.io/http2curl/v2 v2.3.0 h1:9r3JfDzWPcbIklMOs2TnIFzDYvfAZvjeavG6EzP7jYs= +moul.io/http2curl/v2 v2.3.0/go.mod h1:RW4hyBjTWSYDOxapodpNEtX0g5Eb16sxklBqmd2RHcE= +nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50= +olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 h1:slmdOY3vp8a7KQbHkL+FLbvbkgMqmXojpFUO/jENuqQ= +olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3/go.mod h1:oVgVk4OWVDi43qWBEyGhXgYxt7+ED4iYNpTngSLX2Iw= +pgregory.net/rapid v1.2.0 h1:keKAYRcjm+e1F0oAuU5F5+YPAWcyxNNRK2wud503Gnk= +pgregory.net/rapid v1.2.0/go.mod h1:PY5XlDGj0+V1FCq0o192FdRhpKHGTRIWBgqjDBTrq04= +rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= +rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= +rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= +sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs= +sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4= +storj.io/common v0.0.0-20260225132117-99155641c30a h1:7gSBQY3vhQMIqi3vfaEXR7mreRjcBLfVsYY0rHIN7P0= +storj.io/common v0.0.0-20260225132117-99155641c30a/go.mod h1:kyxwKwlfH4paBZCZt/szvRB770ieAIJF73iDy2DpEHw= +storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55 h1:8OE12DvUnB9lfZcHe7IDGsuhjrY9GBAr964PVHmhsro= +storj.io/drpc v0.0.35-0.20250513201419-f7819ea69b55/go.mod h1:Y9LZaa8esL1PW2IDMqJE7CFSNq7d5bQ3RI7mGPtmKMg= +storj.io/eventkit v0.0.0-20250410172343-61f26d3de156 h1:5MZ0CyMbG6Pi0rRzUWVG6dvpXjbBYEX2oyXuj+tT+sk= +storj.io/eventkit v0.0.0-20250410172343-61f26d3de156/go.mod h1:CpnM6kfZV58dcq3lpbo/IQ4/KoutarnTSHY0GYVwnYw= +storj.io/infectious v0.0.2 h1:rGIdDC/6gNYAStsxsZU79D/MqFjNyJc1tsyyj9sTl7Q= +storj.io/infectious v0.0.2/go.mod h1:QEjKKww28Sjl1x8iDsjBpOM4r1Yp8RsowNcItsZJ1Vs= +storj.io/picobuf v0.0.4 h1:qswHDla+YZ2TovGtMnU4astjvrADSIz84FXRn0qgP6o= +storj.io/picobuf v0.0.4/go.mod h1:hSMxmZc58MS/2qSLy1I0idovlO7+6K47wIGUyRZa6mg= +storj.io/uplink v1.14.0 h1:J1yXlt0aRr6kgLTHWXOWosNCFVfbamlcyd+CSxyIczo= +storj.io/uplink v1.14.0/go.mod h1:2ysmjzd/1Xtz4VKoErNcSqBQz3UC9WKTVuLMV1cNu6E= diff --git a/backend/internal/config/config.go b/backend/internal/config/config.go new file mode 100644 index 0000000..c891b1e --- /dev/null +++ b/backend/internal/config/config.go @@ -0,0 +1,393 @@ +package config + +import ( + "context" + "database/sql" + "fmt" + "os" + "path/filepath" + "strings" + "sync" + "time" + + "github.com/ilyakaznacheev/cleanenv" + _ "github.com/jackc/pgx/v5/stdlib" + "github.com/joho/godotenv" + + env_utils "databasus-backend/internal/util/env" + "databasus-backend/internal/util/logger" + "databasus-backend/internal/util/tools" +) + +var log = logger.GetLogger() + +const ( + AppModeWeb = "web" + AppModeBackground = "background" +) + +const ( + // defaultTestParallelWorkers is the number of parallel test workers, used + // when TEST_PARALLEL_WORKERS is unset. Each worker gets its own metadata DB + // and Valkey logical DB, so this must stay <= 16 (Valkey's default logical DB + // count) and equal to the `go test -p` value. + defaultTestParallelWorkers = 8 + + // testSlotAdvisoryLockBase is the first pg_advisory_lock key reserved for + // test-worker slot claims; slot N uses base+N. + testSlotAdvisoryLockBase = 945_000_000 + + // testSlotClaimTimeout bounds the wait for a free slot, absorbing the brief + // window where `go test -p` starts the next package before the previous + // process has exited and released its advisory lock. + testSlotClaimTimeout = 60 * time.Second +) + +type EnvVariables struct { + IsTesting bool + EnvMode env_utils.EnvMode `env:"ENV_MODE" required:"true"` + + // Internal database + DatabaseDsn string `env:"DATABASE_DSN" required:"true"` + TestDatabaseDsn string `env:"TEST_DATABASE_DSN"` + TestParallelWorkers int `env:"TEST_PARALLEL_WORKERS"` + // Internal Valkey + ValkeyHost string `env:"VALKEY_HOST" required:"true"` + ValkeyPort string `env:"VALKEY_PORT" required:"true"` + ValkeyUsername string `env:"VALKEY_USERNAME"` + ValkeyPassword string `env:"VALKEY_PASSWORD"` + ValkeyIsSsl bool `env:"VALKEY_IS_SSL" required:"true"` + + // Per-worker test isolation (computed, only set under `go test`): each test + // binary claims a slot 0..TestParallelWorkers-1 that selects its own metadata DB + // (DatabaseDsn is rewritten to dbname=_w{slot}), its own Valkey logical + // DB (ValkeySelectDB), and a cache-key namespace ("w{slot}:"). All zero/empty in + // production. + ValkeySelectDB int + CacheNamespace string + + TestLocalhost string `env:"TEST_LOCALHOST"` + + ShowDbInstallationVerificationLogs bool `env:"SHOW_DB_INSTALLATION_VERIFICATION_LOGS"` + + DataFolder string + TempFolder string + SecretKeyPath string + TelemetryInstancePath string + + IsDisableAnonymousTelemetry bool `env:"IS_DISABLE_ANONYMOUS_TELEMETRY"` + + // TestLogicalPostgres16Port is the shared always-on logical-Postgres fixture used by + // GetTestPostgresConfig / CreateTestDatabase across the test suite. The per-version PG + // logical tests (14-18 + SSL + mTLS) run on testcontainers and need no fixed port. + TestLogicalPostgres16Port string `env:"TEST_LOGICAL_POSTGRES_16_PORT"` + + // The physical primary sources are the shared always-on fixture (like the logical PG-16 + // fixture); the per-version, no-summary, tablespace, mTLS and restore-target containers all run + // on testcontainers and need no fixed port. + TestPhysicalPostgres17Port string `env:"TEST_PHYSICAL_POSTGRES_17_PORT"` + TestPhysicalPostgres18Port string `env:"TEST_PHYSICAL_POSTGRES_18_PORT"` + + // oauth + GitHubClientID string `env:"GITHUB_CLIENT_ID"` + GitHubClientSecret string `env:"GITHUB_CLIENT_SECRET"` + GoogleClientID string `env:"GOOGLE_CLIENT_ID"` + GoogleClientSecret string `env:"GOOGLE_CLIENT_SECRET"` + + // Cloudflare Turnstile + CloudflareTurnstileSecretKey string `env:"CLOUDFLARE_TURNSTILE_SECRET_KEY"` + CloudflareTurnstileSiteKey string `env:"CLOUDFLARE_TURNSTILE_SITE_KEY"` + + // SMTP configuration (optional) + SMTPHost string `env:"SMTP_HOST"` + SMTPPort int `env:"SMTP_PORT"` + SMTPUser string `env:"SMTP_USER"` + SMTPPassword string `env:"SMTP_PASSWORD"` + SMTPFrom string `env:"SMTP_FROM"` + SMTPInsecureSkipVerify bool `env:"SMTP_INSECURE_SKIP_VERIFY"` + + // Application URL (optional) - used for email links + DatabasusURL string `env:"DATABASUS_URL"` +} + +var env EnvVariables + +var initEnv = sync.OnceFunc(loadEnvVariables) + +func GetEnv() *EnvVariables { + initEnv() + return &env +} + +func loadEnvVariables() { + cwd, err := os.Getwd() + if err != nil { + log.Warn("could not get current working directory", "error", err) + cwd = "." + } + + backendRoot := cwd + for { + if _, err := os.Stat(filepath.Join(backendRoot, "go.mod")); err == nil { + break + } + + parent := filepath.Dir(backendRoot) + if parent == backendRoot { + break + } + + backendRoot = parent + } + + envPath := filepath.Join(filepath.Dir(backendRoot), ".env") + + log.Info("Trying to load .env", "path", envPath) + if err := godotenv.Load(envPath); err != nil { + log.Error("Error loading .env file from repo root", "path", envPath, "error", err) + os.Exit(1) + } + log.Info("Successfully loaded .env", "path", envPath) + + // Empty values for non-string fields (e.g. SMTP_PORT=) crash cleanenv's + // strconv parsing. Drop them so cleanenv falls back to the Go zero value. + unsetEmptyEnvVars() + + err = cleanenv.ReadEnv(&env) + if err != nil { + log.Error("Configuration could not be loaded", "error", err) + os.Exit(1) + } + + if env.SMTPHost != "" && env.SMTPPort <= 0 { + log.Error("SMTP_PORT must be a positive integer when SMTP_HOST is set", "value", env.SMTPPort) + os.Exit(1) + } + + // Set default value for ShowDbInstallationVerificationLogs if not defined + if os.Getenv("SHOW_DB_INSTALLATION_VERIFICATION_LOGS") == "" { + env.ShowDbInstallationVerificationLogs = true + } + + for _, arg := range os.Args { + if strings.Contains(arg, "test") { + env.IsTesting = true + break + } + } + + if env.IsTesting { + if env.TestDatabaseDsn == "" { + log.Error("TEST_DATABASE_DSN is empty") + os.Exit(1) + } + + env.DatabaseDsn = env.TestDatabaseDsn + + if env.TestParallelWorkers <= 0 { + env.TestParallelWorkers = defaultTestParallelWorkers + } + + // Only a real `go test` binary claims a per-worker slot; the cleanup_test_db + // command and any other tool run with IsTesting=true must operate on all + // slots, so they keep the base test DSN and default Valkey DB. + if strings.Contains(os.Args[0], ".test") { + applyTestWorkerSlot() + } + } + + if env.DatabaseDsn == "" { + log.Error("DATABASE_DSN is empty") + os.Exit(1) + } + + if env.EnvMode == "" { + log.Error("ENV_MODE is empty") + os.Exit(1) + } + if env.EnvMode != "development" && env.EnvMode != "production" { + log.Error("ENV_MODE is invalid", "mode", env.EnvMode) + os.Exit(1) + } + log.Info("ENV_MODE loaded", "mode", env.EnvMode) + + tools.LogAndExitIfClientToolsBroken(log, env.ShowDbInstallationVerificationLogs) + + if env.TestLocalhost == "" { + env.TestLocalhost = "localhost" + } + + // Valkey + if env.ValkeyHost == "" { + log.Error("VALKEY_HOST is empty") + os.Exit(1) + } + if env.ValkeyPort == "" { + log.Error("VALKEY_PORT is empty") + os.Exit(1) + } + + // Store the data and temp folders one level below the root + // (projectRoot/databasus-data -> /databasus-data) + env.DataFolder = filepath.Join(filepath.Dir(backendRoot), "databasus-data", "backups") + env.TempFolder = filepath.Join(filepath.Dir(backendRoot), "databasus-data", "temp") + env.SecretKeyPath = filepath.Join(filepath.Dir(backendRoot), "databasus-data", "secret.key") + env.TelemetryInstancePath = filepath.Join( + filepath.Dir(backendRoot), "databasus-data", "instance.json", + ) + + if env.IsTesting { + if env.TestLogicalPostgres16Port == "" { + log.Error("TEST_LOGICAL_POSTGRES_16_PORT is empty") + os.Exit(1) + } + if env.TestPhysicalPostgres17Port == "" { + log.Error("TEST_PHYSICAL_POSTGRES_17_PORT is empty") + os.Exit(1) + } + if env.TestPhysicalPostgres18Port == "" { + log.Error("TEST_PHYSICAL_POSTGRES_18_PORT is empty") + os.Exit(1) + } + } + + log.Info("Environment variables loaded successfully!") +} + +func unsetEmptyEnvVars() { + for _, kv := range os.Environ() { + key, value, ok := strings.Cut(kv, "=") + if !ok { + continue + } + + if value == "" { + _ = os.Unsetenv(key) + } + } +} + +// slotLockConn holds the system-DB connection whose session owns this worker's +// advisory lock. It must live for the whole process: closing it (or letting it +// be garbage-collected) releases the lock and frees the slot for another worker +// mid-run. It is assigned-only by design — the reference itself is the point, so +// it anchors the connection lifetime as a GC root. +// +//nolint:unused // assigned-only: keeps the advisory-lock connection alive for the process lifetime +var slotLockConn *sql.Conn + +// applyTestWorkerSlot claims a free slot for this test binary and rewrites the +// env so the worker runs fully isolated: its own metadata DB, Valkey logical DB, +// and registry namespace. +func applyTestWorkerSlot() { + baseDbName, _, err := RewriteDbName(env.TestDatabaseDsn, systemDbName) + if err != nil { + log.Error("could not parse TEST_DATABASE_DSN for slot isolation", "error", err) + os.Exit(1) + } + + slot := claimTestWorkerSlot(env.TestDatabaseDsn, env.TestParallelWorkers) + + slotDbName := fmt.Sprintf("%s_w%d", baseDbName, slot) + _, slotDsn, err := RewriteDbName(env.TestDatabaseDsn, slotDbName) + if err != nil { + log.Error("could not build per-slot DSN", "error", err) + os.Exit(1) + } + + env.DatabaseDsn = slotDsn + env.ValkeySelectDB = slot + env.CacheNamespace = fmt.Sprintf("w%d:", slot) + + log.Info("claimed test worker slot", "slot", slot, "db", slotDbName) +} + +// systemDbName is the always-present database used for slot coordination and as +// the connection target when (re)creating per-slot databases. +const systemDbName = "postgres" + +// claimTestWorkerSlot acquires a session-level advisory lock for the first free +// slot in [0,pool) on the system Postgres DB and returns it. The lock is held by +// slotLockConn until the process exits. It retries until testSlotClaimTimeout to +// absorb the handoff window where `go test -p` overlaps a finishing worker. +func claimTestWorkerSlot(testDsn string, pool int) int { + _, systemDsn, err := RewriteDbName(testDsn, systemDbName) + if err != nil { + log.Error("could not build system DSN for slot claim", "error", err) + os.Exit(1) + } + + db, err := sql.Open("pgx", systemDsn) + if err != nil { + log.Error("could not open system DB for slot claim", "error", err) + os.Exit(1) + } + + ctx := context.Background() + conn, err := db.Conn(ctx) + if err != nil { + log.Error("could not get system DB connection for slot claim", "error", err) + os.Exit(1) + } + + deadline := time.Now().Add(testSlotClaimTimeout) + for { + for slot := range pool { + var locked bool + lockErr := conn.QueryRowContext( + ctx, + "SELECT pg_try_advisory_lock($1)", + testSlotAdvisoryLockBase+int64(slot), + ).Scan(&locked) + if lockErr != nil { + log.Error("advisory lock query failed during slot claim", "error", lockErr) + os.Exit(1) + } + + if locked { + slotLockConn = conn + return slot + } + } + + if time.Now().After(deadline) { + log.Error( + "no free test DB slot", + "pool", pool, + "hint", "TEST_PARALLEL_WORKERS must be >= the `go test -p` value", + ) + os.Exit(1) + } + + time.Sleep(100 * time.Millisecond) + } +} + +// RewriteDbName replaces the dbname= token in a keyword/value Postgres DSN, +// returning the original dbname and the rewritten DSN. Used both to derive the +// system DSN and to build per-slot test DSNs. +func RewriteDbName(dsn, newDbName string) (origDbName, rewritten string, err error) { + parts := strings.Fields(dsn) + out := make([]string, 0, len(parts)) + + for _, p := range parts { + k, v, ok := strings.Cut(p, "=") + if !ok { + return "", "", fmt.Errorf("invalid DSN token: %q", p) + } + + if k == "dbname" { + origDbName = v + out = append(out, "dbname="+newDbName) + continue + } + + out = append(out, p) + } + + if origDbName == "" { + return "", "", fmt.Errorf("DSN missing dbname: %q", dsn) + } + + return origDbName, strings.Join(out, " "), nil +} diff --git a/backend/internal/config/signals.go b/backend/internal/config/signals.go new file mode 100644 index 0000000..64b89ea --- /dev/null +++ b/backend/internal/config/signals.go @@ -0,0 +1,23 @@ +package config + +import ( + "os" + "os/signal" + "syscall" +) + +var isShutDownSignalReceived = false + +func StartListeningForShutdownSignal() { + quit := make(chan os.Signal, 1) + signal.Notify(quit, os.Interrupt, syscall.SIGTERM) + + go func() { + <-quit + isShutDownSignalReceived = true + }() +} + +func IsShouldShutdown() bool { + return isShutDownSignalReceived +} diff --git a/backend/internal/features/audit_logs/background_service.go b/backend/internal/features/audit_logs/background_service.go new file mode 100644 index 0000000..ababa02 --- /dev/null +++ b/backend/internal/features/audit_logs/background_service.go @@ -0,0 +1,46 @@ +package audit_logs + +import ( + "context" + "fmt" + "log/slog" + "sync/atomic" + "time" +) + +type AuditLogBackgroundService struct { + auditLogService *AuditLogService + logger *slog.Logger + + hasRun atomic.Bool +} + +func (s *AuditLogBackgroundService) Run(ctx context.Context) { + if s.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", s)) + } + + s.logger.Info("Starting audit log cleanup background service") + + if ctx.Err() != nil { + return + } + + ticker := time.NewTicker(1 * time.Hour) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if err := s.cleanOldAuditLogs(); err != nil { + s.logger.Error("Failed to clean old audit logs", "error", err) + } + } + } +} + +func (s *AuditLogBackgroundService) cleanOldAuditLogs() error { + return s.auditLogService.CleanOldAuditLogs() +} diff --git a/backend/internal/features/audit_logs/background_service_test.go b/backend/internal/features/audit_logs/background_service_test.go new file mode 100644 index 0000000..9e6a046 --- /dev/null +++ b/backend/internal/features/audit_logs/background_service_test.go @@ -0,0 +1,141 @@ +package audit_logs + +import ( + "fmt" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "gorm.io/gorm" + + user_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + "databasus-backend/internal/storage" +) + +func Test_CleanOldAuditLogs_DeletesLogsOlderThanOneYear(t *testing.T) { + service := GetAuditLogService() + user := users_testing.CreateTestUser(user_enums.UserRoleMember) + db := storage.GetDb() + baseTime := time.Now().UTC() + + // Create old logs (more than 1 year old) + createTimedAuditLog(db, &user.UserID, "Old log 1", baseTime.Add(-400*24*time.Hour)) + createTimedAuditLog(db, &user.UserID, "Old log 2", baseTime.Add(-370*24*time.Hour)) + + // Create recent logs (less than 1 year old) + createAuditLog(service, "Recent log 1", &user.UserID, nil) + createAuditLog(service, "Recent log 2", &user.UserID, nil) + + // Run cleanup + err := service.CleanOldAuditLogs() + assert.NoError(t, err) + + // Verify old logs were deleted + oneYearAgo := baseTime.Add(-365 * 24 * time.Hour) + var oldLogs []*AuditLog + db.Where("created_at < ?", oneYearAgo).Find(&oldLogs) + assert.Equal(t, 0, len(oldLogs), "All logs older than 1 year should be deleted") + + // Verify recent logs still exist + var recentLogs []*AuditLog + db.Where("created_at >= ?", oneYearAgo).Find(&recentLogs) + assert.GreaterOrEqual(t, len(recentLogs), 2, "Recent logs should not be deleted") +} + +func Test_CleanOldAuditLogs_PreservesLogsNewerThanOneYear(t *testing.T) { + service := GetAuditLogService() + user := users_testing.CreateTestUser(user_enums.UserRoleMember) + db := storage.GetDb() + baseTime := time.Now().UTC() + + // Create logs exactly at boundary (1 year old) + boundaryTime := baseTime.Add(-365 * 24 * time.Hour) + createTimedAuditLog(db, &user.UserID, "Boundary log", boundaryTime) + + // Create recent logs + createTimedAuditLog(db, &user.UserID, "Recent log 1", baseTime.Add(-364*24*time.Hour)) + createTimedAuditLog(db, &user.UserID, "Recent log 2", baseTime.Add(-100*24*time.Hour)) + createAuditLog(service, "Current log", &user.UserID, nil) + + // Get count before cleanup + var countBefore int64 + db.Model(&AuditLog{}).Count(&countBefore) + + // Run cleanup + err := service.CleanOldAuditLogs() + assert.NoError(t, err) + + // Get count after cleanup + var countAfter int64 + db.Model(&AuditLog{}).Count(&countAfter) + + // Verify logs newer than 1 year are preserved + oneYearAgo := baseTime.Add(-365 * 24 * time.Hour) + var recentLogs []*AuditLog + db.Where("created_at >= ?", oneYearAgo).Find(&recentLogs) + + messages := make([]string, len(recentLogs)) + for i, log := range recentLogs { + messages[i] = log.Message + } + + assert.Contains(t, messages, "Recent log 1") + assert.Contains(t, messages, "Recent log 2") + assert.Contains(t, messages, "Current log") +} + +func Test_CleanOldAuditLogs_HandlesEmptyDatabase(t *testing.T) { + service := GetAuditLogService() + + // Run cleanup on database that may have no old logs + err := service.CleanOldAuditLogs() + assert.NoError(t, err) +} + +func Test_CleanOldAuditLogs_DeletesMultipleOldLogs(t *testing.T) { + service := GetAuditLogService() + user := users_testing.CreateTestUser(user_enums.UserRoleMember) + db := storage.GetDb() + baseTime := time.Now().UTC() + + // Create many old logs with specific UUIDs to track them + testLogIDs := make([]uuid.UUID, 5) + for i := range 5 { + testLogIDs[i] = uuid.New() + daysAgo := 400 + (i * 10) + log := &AuditLog{ + ID: testLogIDs[i], + UserID: &user.UserID, + Message: fmt.Sprintf("Test old log %d", i), + CreatedAt: baseTime.Add(-time.Duration(daysAgo) * 24 * time.Hour), + } + result := db.Create(log) + assert.NoError(t, result.Error) + } + + // Verify logs exist before cleanup + var logsBeforeCleanup []*AuditLog + db.Where("id IN ?", testLogIDs).Find(&logsBeforeCleanup) + assert.Equal(t, 5, len(logsBeforeCleanup), "All test logs should exist before cleanup") + + // Run cleanup + err := service.CleanOldAuditLogs() + assert.NoError(t, err) + + // Verify test logs were deleted + var logsAfterCleanup []*AuditLog + db.Where("id IN ?", testLogIDs).Find(&logsAfterCleanup) + assert.Equal(t, 0, len(logsAfterCleanup), "All old test logs should be deleted") +} + +func createTimedAuditLog(db *gorm.DB, userID *uuid.UUID, message string, createdAt time.Time) { + log := &AuditLog{ + ID: uuid.New(), + UserID: userID, + Message: message, + CreatedAt: createdAt, + } + db.Create(log) +} diff --git a/backend/internal/features/audit_logs/controller.go b/backend/internal/features/audit_logs/controller.go new file mode 100644 index 0000000..f4961a9 --- /dev/null +++ b/backend/internal/features/audit_logs/controller.go @@ -0,0 +1,112 @@ +package audit_logs + +import ( + "errors" + "net/http" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + user_models "databasus-backend/internal/features/users/models" +) + +type AuditLogController struct { + auditLogService *AuditLogService +} + +func (c *AuditLogController) RegisterRoutes(router *gin.RouterGroup) { + // All audit log endpoints require authentication (handled in main.go) + auditRoutes := router.Group("/audit-logs") + + auditRoutes.GET("/global", c.GetGlobalAuditLogs) + auditRoutes.GET("/users/:userId", c.GetUserAuditLogs) +} + +// GetGlobalAuditLogs +// @Summary Get global audit logs (ADMIN only) +// @Description Retrieve all audit logs across the system +// @Tags audit-logs +// @Accept json +// @Produce json +// @Security BearerAuth +// @Param limit query int false "Limit number of results" default(100) +// @Param offset query int false "Offset for pagination" default(0) +// @Param beforeDate query string false "Filter logs created before this date (RFC3339 format)" format(date-time) +// @Success 200 {object} GetAuditLogsResponse +// @Failure 401 {object} map[string]string +// @Failure 403 {object} map[string]string +// @Router /audit-logs/global [get] +func (c *AuditLogController) GetGlobalAuditLogs(ctx *gin.Context) { + user, isOk := ctx.MustGet("user").(*user_models.User) + if !isOk { + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid user type in context"}) + return + } + + request := &GetAuditLogsRequest{} + if err := ctx.ShouldBindQuery(request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid query parameters"}) + return + } + + response, err := c.auditLogService.GetGlobalAuditLogs(user, request) + if err != nil { + if errors.Is(err, ErrOnlyAdminsCanViewGlobalLogs) { + ctx.JSON(http.StatusForbidden, gin.H{"error": err.Error()}) + return + } + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to retrieve audit logs"}) + return + } + + ctx.JSON(http.StatusOK, response) +} + +// GetUserAuditLogs +// @Summary Get user audit logs +// @Description Retrieve audit logs for a specific user +// @Tags audit-logs +// @Accept json +// @Produce json +// @Security BearerAuth +// @Param userId path string true "User ID" +// @Param limit query int false "Limit number of results" default(100) +// @Param offset query int false "Offset for pagination" default(0) +// @Param beforeDate query string false "Filter logs created before this date (RFC3339 format)" format(date-time) +// @Success 200 {object} GetAuditLogsResponse +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 403 {object} map[string]string +// @Router /audit-logs/users/{userId} [get] +func (c *AuditLogController) GetUserAuditLogs(ctx *gin.Context) { + user, isOk := ctx.MustGet("user").(*user_models.User) + if !isOk { + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Invalid user type in context"}) + return + } + + userIDStr := ctx.Param("userId") + targetUserID, err := uuid.Parse(userIDStr) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid user ID"}) + return + } + + request := &GetAuditLogsRequest{} + if err := ctx.ShouldBindQuery(request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "Invalid query parameters"}) + return + } + + response, err := c.auditLogService.GetUserAuditLogs(targetUserID, user, request) + if err != nil { + if errors.Is(err, ErrInsufficientPermissionsToViewLogs) { + ctx.JSON(http.StatusForbidden, gin.H{"error": err.Error()}) + return + } + ctx.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to retrieve audit logs"}) + return + } + + ctx.JSON(http.StatusOK, response) +} diff --git a/backend/internal/features/audit_logs/controller_test.go b/backend/internal/features/audit_logs/controller_test.go new file mode 100644 index 0000000..89a0a10 --- /dev/null +++ b/backend/internal/features/audit_logs/controller_test.go @@ -0,0 +1,154 @@ +package audit_logs + +import ( + "fmt" + "net/http" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + user_enums "databasus-backend/internal/features/users/enums" + users_middleware "databasus-backend/internal/features/users/middleware" + users_services "databasus-backend/internal/features/users/services" + users_testing "databasus-backend/internal/features/users/testing" + test_utils "databasus-backend/internal/util/testing" +) + +func Test_GetGlobalAuditLogs_WithDifferentUserRoles_EnforcesPermissionsCorrectly(t *testing.T) { + adminUser := users_testing.CreateTestUser(user_enums.UserRoleAdmin) + memberUser := users_testing.CreateTestUser(user_enums.UserRoleMember) + router := createRouter() + service := GetAuditLogService() + workspaceID := uuid.New() + testID := uuid.New().String() + + // Create test logs with unique identifiers + userLogMessage := fmt.Sprintf("Test log with user %s", testID) + workspaceLogMessage := fmt.Sprintf("Test log with workspace %s", testID) + standaloneLogMessage := fmt.Sprintf("Test log standalone %s", testID) + + createAuditLog(service, userLogMessage, &adminUser.UserID, nil) + createAuditLog(service, workspaceLogMessage, nil, &workspaceID) + createAuditLog(service, standaloneLogMessage, nil, nil) + + // Test ADMIN can access global logs + var response GetAuditLogsResponse + test_utils.MakeGetRequestAndUnmarshal(t, router, + "/api/v1/audit-logs/global?limit=100", "Bearer "+adminUser.Token, http.StatusOK, &response) + + // Verify our specific test logs are present + messages := extractMessages(response.AuditLogs) + assert.Contains(t, messages, userLogMessage) + assert.Contains(t, messages, workspaceLogMessage) + assert.Contains(t, messages, standaloneLogMessage) + + // Test MEMBER cannot access global logs + resp := test_utils.MakeGetRequest(t, router, "/api/v1/audit-logs/global", + "Bearer "+memberUser.Token, http.StatusForbidden) + assert.Contains(t, string(resp.Body), "only administrators can view global audit logs") +} + +func Test_GetUserAuditLogs_WithDifferentUserRoles_EnforcesPermissionsCorrectly(t *testing.T) { + adminUser := users_testing.CreateTestUser(user_enums.UserRoleAdmin) + user1 := users_testing.CreateTestUser(user_enums.UserRoleMember) + user2 := users_testing.CreateTestUser(user_enums.UserRoleMember) + router := createRouter() + service := GetAuditLogService() + workspaceID := uuid.New() + testID := uuid.New().String() + + // Create test logs for different users with unique identifiers + user1FirstMessage := fmt.Sprintf("Test log user1 first %s", testID) + user1SecondMessage := fmt.Sprintf("Test log user1 second %s", testID) + user2FirstMessage := fmt.Sprintf("Test log user2 first %s", testID) + user2SecondMessage := fmt.Sprintf("Test log user2 second %s", testID) + workspaceLogMessage := fmt.Sprintf("Test workspace log %s", testID) + + createAuditLog(service, user1FirstMessage, &user1.UserID, nil) + createAuditLog(service, user1SecondMessage, &user1.UserID, &workspaceID) + createAuditLog(service, user2FirstMessage, &user2.UserID, nil) + createAuditLog(service, user2SecondMessage, &user2.UserID, &workspaceID) + createAuditLog(service, workspaceLogMessage, nil, &workspaceID) + + // Test ADMIN can view any user's logs + var user1Response GetAuditLogsResponse + test_utils.MakeGetRequestAndUnmarshal(t, router, + fmt.Sprintf("/api/v1/audit-logs/users/%s?limit=100", user1.UserID.String()), + "Bearer "+adminUser.Token, http.StatusOK, &user1Response) + + // Verify user1's specific logs are present + messages := extractMessages(user1Response.AuditLogs) + assert.Contains(t, messages, user1FirstMessage) + assert.Contains(t, messages, user1SecondMessage) + + // Count only our test logs for user1 + testLogsCount := 0 + for _, message := range messages { + if message == user1FirstMessage || message == user1SecondMessage { + testLogsCount++ + } + } + assert.Equal(t, 2, testLogsCount) + + // Test user can view own logs + var ownLogsResponse GetAuditLogsResponse + test_utils.MakeGetRequestAndUnmarshal(t, router, + fmt.Sprintf("/api/v1/audit-logs/users/%s?limit=100", user2.UserID.String()), + "Bearer "+user2.Token, http.StatusOK, &ownLogsResponse) + + // Verify user2's specific logs are present + ownMessages := extractMessages(ownLogsResponse.AuditLogs) + assert.Contains(t, ownMessages, user2FirstMessage) + assert.Contains(t, ownMessages, user2SecondMessage) + + // Test user cannot view other user's logs + resp := test_utils.MakeGetRequest(t, router, + fmt.Sprintf("/api/v1/audit-logs/users/%s", user1.UserID.String()), + "Bearer "+user2.Token, http.StatusForbidden) + + assert.Contains(t, string(resp.Body), "insufficient permissions") +} + +func Test_GetGlobalAuditLogs_WithBeforeDateFilter_ReturnsFilteredLogs(t *testing.T) { + adminUser := users_testing.CreateTestUser(user_enums.UserRoleAdmin) + router := createRouter() + baseTime := time.Now().UTC() + + // Set filter time to 30 minutes ago + beforeTime := baseTime.Add(-30 * time.Minute) + + var filteredResponse GetAuditLogsResponse + test_utils.MakeGetRequestAndUnmarshal( + t, + router, + fmt.Sprintf( + "/api/v1/audit-logs/global?beforeDate=%s&limit=1000", + beforeTime.Format(time.RFC3339), + ), + "Bearer "+adminUser.Token, + http.StatusOK, + &filteredResponse, + ) + + // Verify ALL returned logs are older than the filter time + for _, log := range filteredResponse.AuditLogs { + assert.True(t, log.CreatedAt.Before(beforeTime), + fmt.Sprintf("Log created at %s should be before filter time %s", + log.CreatedAt.Format(time.RFC3339), beforeTime.Format(time.RFC3339))) + } +} + +func createRouter() *gin.Engine { + gin.SetMode(gin.TestMode) + router := gin.New() + SetupDependencies() + + v1 := router.Group("/api/v1") + protected := v1.Group("").Use(users_middleware.AuthMiddleware(users_services.GetUserService())) + GetAuditLogController().RegisterRoutes(protected.(*gin.RouterGroup)) + + return router +} diff --git a/backend/internal/features/audit_logs/di.go b/backend/internal/features/audit_logs/di.go new file mode 100644 index 0000000..a0640db --- /dev/null +++ b/backend/internal/features/audit_logs/di.go @@ -0,0 +1,43 @@ +package audit_logs + +import ( + "sync" + + users_services "databasus-backend/internal/features/users/services" + "databasus-backend/internal/util/logger" +) + +var ( + auditLogRepository = &AuditLogRepository{} + auditLogService = &AuditLogService{ + auditLogRepository, + logger.GetLogger(), + } +) + +var auditLogController = &AuditLogController{ + auditLogService, +} + +var auditLogBackgroundService = &AuditLogBackgroundService{ + auditLogService: auditLogService, + logger: logger.GetLogger(), +} + +func GetAuditLogService() *AuditLogService { + return auditLogService +} + +func GetAuditLogController() *AuditLogController { + return auditLogController +} + +func GetAuditLogBackgroundService() *AuditLogBackgroundService { + return auditLogBackgroundService +} + +var SetupDependencies = sync.OnceFunc(func() { + users_services.GetUserService().SetAuditLogWriter(auditLogService) + users_services.GetSettingsService().SetAuditLogWriter(auditLogService) + users_services.GetManagementService().SetAuditLogWriter(auditLogService) +}) diff --git a/backend/internal/features/audit_logs/dto.go b/backend/internal/features/audit_logs/dto.go new file mode 100644 index 0000000..6f2773e --- /dev/null +++ b/backend/internal/features/audit_logs/dto.go @@ -0,0 +1,31 @@ +package audit_logs + +import ( + "time" + + "github.com/google/uuid" +) + +type GetAuditLogsRequest struct { + Limit int `form:"limit" json:"limit"` + Offset int `form:"offset" json:"offset"` + BeforeDate *time.Time `form:"beforeDate" json:"beforeDate"` +} + +type GetAuditLogsResponse struct { + AuditLogs []*AuditLogDTO `json:"auditLogs"` + Total int64 `json:"total"` + Limit int `json:"limit"` + Offset int `json:"offset"` +} + +type AuditLogDTO struct { + ID uuid.UUID `json:"id" gorm:"column:id"` + UserID *uuid.UUID `json:"userId" gorm:"column:user_id"` + WorkspaceID *uuid.UUID `json:"workspaceId" gorm:"column:workspace_id"` + Message string `json:"message" gorm:"column:message"` + CreatedAt time.Time `json:"createdAt" gorm:"column:created_at"` + UserEmail *string `json:"userEmail" gorm:"column:user_email"` + UserName *string `json:"userName" gorm:"column:user_name"` + WorkspaceName *string `json:"workspaceName" gorm:"column:workspace_name"` +} diff --git a/backend/internal/features/audit_logs/errors.go b/backend/internal/features/audit_logs/errors.go new file mode 100644 index 0000000..0214ac5 --- /dev/null +++ b/backend/internal/features/audit_logs/errors.go @@ -0,0 +1,12 @@ +package audit_logs + +import "errors" + +var ( + ErrOnlyAdminsCanViewGlobalLogs = errors.New( + "only administrators can view global audit logs", + ) + ErrInsufficientPermissionsToViewLogs = errors.New( + "insufficient permissions to view user audit logs", + ) +) diff --git a/backend/internal/features/audit_logs/models.go b/backend/internal/features/audit_logs/models.go new file mode 100644 index 0000000..cbff174 --- /dev/null +++ b/backend/internal/features/audit_logs/models.go @@ -0,0 +1,19 @@ +package audit_logs + +import ( + "time" + + "github.com/google/uuid" +) + +type AuditLog struct { + ID uuid.UUID `json:"id" gorm:"column:id"` + UserID *uuid.UUID `json:"userId" gorm:"column:user_id"` + WorkspaceID *uuid.UUID `json:"workspaceId" gorm:"column:workspace_id"` + Message string `json:"message" gorm:"column:message"` + CreatedAt time.Time `json:"createdAt" gorm:"column:created_at"` +} + +func (AuditLog) TableName() string { + return "audit_logs" +} diff --git a/backend/internal/features/audit_logs/repository.go b/backend/internal/features/audit_logs/repository.go new file mode 100644 index 0000000..9d1d38e --- /dev/null +++ b/backend/internal/features/audit_logs/repository.go @@ -0,0 +1,152 @@ +package audit_logs + +import ( + "time" + + "github.com/google/uuid" + + "databasus-backend/internal/storage" +) + +type AuditLogRepository struct{} + +func (r *AuditLogRepository) Create(auditLog *AuditLog) error { + if auditLog.ID == uuid.Nil { + auditLog.ID = uuid.New() + } + + return storage.GetDb().Create(auditLog).Error +} + +func (r *AuditLogRepository) GetGlobal( + limit, offset int, + beforeDate *time.Time, +) ([]*AuditLogDTO, error) { + auditLogs := make([]*AuditLogDTO, 0) + + sql := ` + SELECT + al.id, + al.user_id, + al.workspace_id, + al.message, + al.created_at, + u.email as user_email, + u.name as user_name, + w.name as workspace_name + FROM audit_logs al + LEFT JOIN users u ON al.user_id = u.id + LEFT JOIN workspaces w ON al.workspace_id = w.id` + + args := []any{} + + if beforeDate != nil { + sql += " WHERE al.created_at < ?" + args = append(args, *beforeDate) + } + + sql += " ORDER BY al.created_at DESC LIMIT ? OFFSET ?" + args = append(args, limit, offset) + + err := storage.GetDb().Raw(sql, args...).Scan(&auditLogs).Error + + return auditLogs, err +} + +func (r *AuditLogRepository) GetByUser( + userID uuid.UUID, + limit, offset int, + beforeDate *time.Time, +) ([]*AuditLogDTO, error) { + auditLogs := make([]*AuditLogDTO, 0) + + sql := ` + SELECT + al.id, + al.user_id, + al.workspace_id, + al.message, + al.created_at, + u.email as user_email, + u.name as user_name, + w.name as workspace_name + FROM audit_logs al + LEFT JOIN users u ON al.user_id = u.id + LEFT JOIN workspaces w ON al.workspace_id = w.id + WHERE al.user_id = ?` + + args := []any{userID} + + if beforeDate != nil { + sql += " AND al.created_at < ?" + args = append(args, *beforeDate) + } + + sql += " ORDER BY al.created_at DESC LIMIT ? OFFSET ?" + args = append(args, limit, offset) + + err := storage.GetDb().Raw(sql, args...).Scan(&auditLogs).Error + + return auditLogs, err +} + +func (r *AuditLogRepository) GetByWorkspace( + workspaceID uuid.UUID, + limit, offset int, + beforeDate *time.Time, +) ([]*AuditLogDTO, error) { + auditLogs := make([]*AuditLogDTO, 0) + + sql := ` + SELECT + al.id, + al.user_id, + al.workspace_id, + al.message, + al.created_at, + u.email as user_email, + u.name as user_name, + w.name as workspace_name + FROM audit_logs al + LEFT JOIN users u ON al.user_id = u.id + LEFT JOIN workspaces w ON al.workspace_id = w.id + WHERE al.workspace_id = ?` + + args := []any{workspaceID} + + if beforeDate != nil { + sql += " AND al.created_at < ?" + args = append(args, *beforeDate) + } + + sql += " ORDER BY al.created_at DESC LIMIT ? OFFSET ?" + args = append(args, limit, offset) + + err := storage.GetDb().Raw(sql, args...).Scan(&auditLogs).Error + + return auditLogs, err +} + +func (r *AuditLogRepository) CountGlobal(beforeDate *time.Time) (int64, error) { + var count int64 + query := storage.GetDb().Model(&AuditLog{}) + + if beforeDate != nil { + query = query.Where("created_at < ?", *beforeDate) + } + + err := query.Count(&count).Error + return count, err +} + +func (r *AuditLogRepository) DeleteOlderThan(beforeDate time.Time) (int64, error) { + result := storage.GetDb(). + Where("created_at < ?", beforeDate). + Delete(&AuditLog{}) + + if result.Error != nil { + return 0, result.Error + } + + return result.RowsAffected, nil +} diff --git a/backend/internal/features/audit_logs/service.go b/backend/internal/features/audit_logs/service.go new file mode 100644 index 0000000..87df9f9 --- /dev/null +++ b/backend/internal/features/audit_logs/service.go @@ -0,0 +1,152 @@ +package audit_logs + +import ( + "log/slog" + "time" + + "github.com/google/uuid" + + user_enums "databasus-backend/internal/features/users/enums" + user_models "databasus-backend/internal/features/users/models" +) + +type AuditLogService struct { + auditLogRepository *AuditLogRepository + logger *slog.Logger +} + +func (s *AuditLogService) WriteAuditLog( + message string, + userID *uuid.UUID, + workspaceID *uuid.UUID, +) { + auditLog := &AuditLog{ + UserID: userID, + WorkspaceID: workspaceID, + Message: message, + CreatedAt: time.Now().UTC(), + } + + err := s.auditLogRepository.Create(auditLog) + if err != nil { + s.logger.Error("failed to create audit log", "error", err) + return + } +} + +func (s *AuditLogService) CreateAuditLog(auditLog *AuditLog) error { + return s.auditLogRepository.Create(auditLog) +} + +func (s *AuditLogService) GetGlobalAuditLogs( + user *user_models.User, + request *GetAuditLogsRequest, +) (*GetAuditLogsResponse, error) { + if user.Role != user_enums.UserRoleAdmin { + return nil, ErrOnlyAdminsCanViewGlobalLogs + } + + limit := request.Limit + if limit <= 0 || limit > 1000 { + limit = 100 + } + + offset := max(request.Offset, 0) + + auditLogs, err := s.auditLogRepository.GetGlobal(limit, offset, request.BeforeDate) + if err != nil { + return nil, err + } + + total, err := s.auditLogRepository.CountGlobal(request.BeforeDate) + if err != nil { + return nil, err + } + + return &GetAuditLogsResponse{ + AuditLogs: auditLogs, + Total: total, + Limit: limit, + Offset: offset, + }, nil +} + +func (s *AuditLogService) GetUserAuditLogs( + targetUserID uuid.UUID, + user *user_models.User, + request *GetAuditLogsRequest, +) (*GetAuditLogsResponse, error) { + // Users can view their own logs, ADMIN can view any user's logs + if user.Role != user_enums.UserRoleAdmin && user.ID != targetUserID { + return nil, ErrInsufficientPermissionsToViewLogs + } + + limit := request.Limit + if limit <= 0 || limit > 1000 { + limit = 100 + } + + offset := max(request.Offset, 0) + + auditLogs, err := s.auditLogRepository.GetByUser( + targetUserID, + limit, + offset, + request.BeforeDate, + ) + if err != nil { + return nil, err + } + + return &GetAuditLogsResponse{ + AuditLogs: auditLogs, + Total: int64(len(auditLogs)), + Limit: limit, + Offset: offset, + }, nil +} + +func (s *AuditLogService) GetWorkspaceAuditLogs( + workspaceID uuid.UUID, + request *GetAuditLogsRequest, +) (*GetAuditLogsResponse, error) { + limit := request.Limit + if limit <= 0 || limit > 1000 { + limit = 100 + } + + offset := max(request.Offset, 0) + + auditLogs, err := s.auditLogRepository.GetByWorkspace( + workspaceID, + limit, + offset, + request.BeforeDate, + ) + if err != nil { + return nil, err + } + + return &GetAuditLogsResponse{ + AuditLogs: auditLogs, + Total: int64(len(auditLogs)), + Limit: limit, + Offset: offset, + }, nil +} + +func (s *AuditLogService) CleanOldAuditLogs() error { + oneYearAgo := time.Now().UTC().Add(-365 * 24 * time.Hour) + + deletedCount, err := s.auditLogRepository.DeleteOlderThan(oneYearAgo) + if err != nil { + s.logger.Error("Failed to delete old audit logs", "error", err) + return err + } + + if deletedCount > 0 { + s.logger.Info("Deleted old audit logs", "count", deletedCount, "olderThan", oneYearAgo) + } + + return nil +} diff --git a/backend/internal/features/audit_logs/service_test.go b/backend/internal/features/audit_logs/service_test.go new file mode 100644 index 0000000..764f48c --- /dev/null +++ b/backend/internal/features/audit_logs/service_test.go @@ -0,0 +1,83 @@ +package audit_logs + +import ( + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + user_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" +) + +func Test_AuditLogs_WorkspaceSpecificLogs(t *testing.T) { + service := GetAuditLogService() + user1 := users_testing.CreateTestUser(user_enums.UserRoleMember) + user2 := users_testing.CreateTestUser(user_enums.UserRoleMember) + workspace1ID, workspace2ID := uuid.New(), uuid.New() + + // Create test logs for workspaces + createAuditLog(service, "Test workspace1 log first", &user1.UserID, &workspace1ID) + createAuditLog(service, "Test workspace1 log second", &user2.UserID, &workspace1ID) + createAuditLog(service, "Test workspace2 log first", &user1.UserID, &workspace2ID) + createAuditLog(service, "Test workspace2 log second", &user2.UserID, &workspace2ID) + createAuditLog(service, "Test no workspace log", &user1.UserID, nil) + + request := &GetAuditLogsRequest{Limit: 10, Offset: 0} + + // Test workspace 1 logs + workspace1Response, err := service.GetWorkspaceAuditLogs(workspace1ID, request) + assert.NoError(t, err) + assert.Equal(t, 2, len(workspace1Response.AuditLogs)) + + messages := extractMessages(workspace1Response.AuditLogs) + assert.Contains(t, messages, "Test workspace1 log first") + assert.Contains(t, messages, "Test workspace1 log second") + for _, log := range workspace1Response.AuditLogs { + assert.Equal(t, &workspace1ID, log.WorkspaceID) + } + + // Test workspace 2 logs + workspace2Response, err := service.GetWorkspaceAuditLogs(workspace2ID, request) + assert.NoError(t, err) + assert.Equal(t, 2, len(workspace2Response.AuditLogs)) + + messages2 := extractMessages(workspace2Response.AuditLogs) + assert.Contains(t, messages2, "Test workspace2 log first") + assert.Contains(t, messages2, "Test workspace2 log second") + + // Test pagination + limitedResponse, err := service.GetWorkspaceAuditLogs(workspace1ID, + &GetAuditLogsRequest{Limit: 1, Offset: 0}) + assert.NoError(t, err) + assert.Equal(t, 1, len(limitedResponse.AuditLogs)) + assert.Equal(t, 1, limitedResponse.Limit) + + // Test beforeDate filter + beforeTime := time.Now().UTC().Add(-1 * time.Minute) + filteredResponse, err := service.GetWorkspaceAuditLogs(workspace1ID, + &GetAuditLogsRequest{Limit: 10, BeforeDate: &beforeTime}) + assert.NoError(t, err) + for _, log := range filteredResponse.AuditLogs { + assert.True(t, log.CreatedAt.Before(beforeTime)) + assert.NotNil(t, log.UserEmail, "User email should be present for logs with user_id") + assert.NotNil( + t, + log.WorkspaceName, + "Workspace name should be present for logs with workspace_id", + ) + } +} + +func createAuditLog(service *AuditLogService, message string, userID, workspaceID *uuid.UUID) { + service.WriteAuditLog(message, userID, workspaceID) +} + +func extractMessages(logs []*AuditLogDTO) []string { + messages := make([]string, len(logs)) + for i, log := range logs { + messages[i] = log.Message + } + return messages +} diff --git a/backend/internal/features/backups/backups/backuping/logical/backuper.go b/backend/internal/features/backups/backups/backuping/logical/backuper.go new file mode 100644 index 0000000..1c3df46 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/backuper.go @@ -0,0 +1,350 @@ +package backuping_logical + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "slices" + "strings" + "time" + + "github.com/google/uuid" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + workspaces_services "databasus-backend/internal/features/workspaces/services" + util_encryption "databasus-backend/internal/util/encryption" +) + +type Backuper struct { + databaseService *databases.DatabaseService + fieldEncryptor util_encryption.FieldEncryptor + workspaceService *workspaces_services.WorkspaceService + backupRepository *backups_core_logical.BackupRepository + backupConfigService *backups_config_logical.BackupConfigService + storageService *storages.StorageService + notificationSender backups_core_logical.NotificationSender + backupCancelManager *tasks_cancellation.TaskCancelManager + logger *slog.Logger + createBackupUseCase backups_core_logical.CreateBackupUsecase +} + +func (b *Backuper) MakeBackup(backupID uuid.UUID, isCallNotifier bool) { + backup, err := b.backupRepository.FindByID(backupID) + if err != nil { + b.logger.Error("Failed to get backup by ID", "backupId", backupID, "error", err) + return + } + + databaseID := backup.DatabaseID + + database, err := b.databaseService.GetDatabaseByID(databaseID) + if err != nil { + b.logger.Error("Failed to get database by ID", "databaseId", databaseID, "error", err) + return + } + + backupConfig, err := b.backupConfigService.GetBackupConfigByDbId(databaseID) + if err != nil { + b.logger.Error("Failed to get backup config by database ID", "error", err) + return + } + + if backupConfig.StorageID == nil { + b.logger.Error("Backup config storage ID is not defined") + return + } + + storage, err := b.storageService.GetStorageByID(*backupConfig.StorageID) + if err != nil { + b.logger.Error("Failed to get storage by ID", "error", err) + return + } + + start := time.Now().UTC() + + ctx, cancel := context.WithCancel(context.Background()) + b.backupCancelManager.RegisterTask(backup.ID, cancel) + defer b.backupCancelManager.UnregisterTask(backup.ID) + + backupProgressListener := func( + completedMBs float64, + ) { + backup.BackupSizeMb = completedMBs + backup.BackupDurationMs = time.Since(start).Milliseconds() + + if err := b.backupRepository.Save(backup); err != nil { + b.logger.Error("Failed to update backup progress", "error", err) + } + } + + backupMetadata, err := b.createBackupUseCase.Execute( + ctx, + backup, + backupConfig, + database, + storage, + backupProgressListener, + ) + if err != nil { + // Check if backup was already marked as failed by progress listener (e.g., size limit exceeded) + // If so, skip error handling to avoid overwriting the status + currentBackup, fetchErr := b.backupRepository.FindByID(backup.ID) + if fetchErr == nil && currentBackup.Status == backups_core_logical.BackupStatusFailed { + b.logger.Warn( + "Backup already marked as failed by progress listener, skipping error handling", + "backupId", + backup.ID, + "failMessage", + *currentBackup.FailMessage, + ) + + // Still call notification for size limit failures + b.SendBackupNotification( + backupConfig, + currentBackup, + backups_config_logical.NotificationBackupFailed, + currentBackup.FailMessage, + ) + + return + } + + errMsg := err.Error() + + // Log detailed error information for debugging + b.logger.Error("Backup execution failed", + "backupId", backup.ID, + "databaseId", databaseID, + "databaseType", database.Type, + "storageId", storage.ID, + "storageType", storage.Type, + "error", err, + "errorMessage", errMsg, + ) + + // Check if backup was cancelled (not due to shutdown) + isCancelled := strings.Contains(errMsg, "backup cancelled") || + strings.Contains(errMsg, "context canceled") || + errors.Is(err, context.Canceled) + isShutdown := strings.Contains(errMsg, "shutdown") + + if isCancelled && !isShutdown { + b.logger.Warn("Backup was cancelled by user or system", + "backupId", backup.ID, + "isCancelled", isCancelled, + "isShutdown", isShutdown, + ) + + backup.Status = backups_core_logical.BackupStatusCanceled + backup.BackupDurationMs = time.Since(start).Milliseconds() + backup.BackupSizeMb = 0 + + if err := b.backupRepository.Save(backup); err != nil { + b.logger.Error("Failed to save cancelled backup", "error", err) + } + + // Delete partial backup from storage + storage, storageErr := b.storageService.GetStorageByID(backup.StorageID) + if storageErr == nil { + if deleteErr := storage.DeleteFile(b.fieldEncryptor, backup.FileName); deleteErr != nil { + b.logger.Error( + "Failed to delete partial backup file", + "backupId", + backup.ID, + "error", + deleteErr, + ) + } + } + + return + } + + backup.FailMessage = &errMsg + backup.Status = backups_core_logical.BackupStatusFailed + backup.BackupDurationMs = time.Since(start).Milliseconds() + backup.BackupSizeMb = 0 + + if updateErr := b.databaseService.SetBackupError(databaseID, errMsg); updateErr != nil { + b.logger.Error( + "Failed to update database last backup time", + "databaseId", + databaseID, + "error", + updateErr, + ) + } + + if err := b.backupRepository.Save(backup); err != nil { + b.logger.Error("Failed to save backup", "error", err) + } + + b.SendBackupNotification( + backupConfig, + backup, + backups_config_logical.NotificationBackupFailed, + &errMsg, + ) + + return + } + + backup.BackupDurationMs = time.Since(start).Milliseconds() + + // Update backup with encryption metadata if provided + if backupMetadata != nil { + backupMetadata.BackupID = backup.ID + + if err := backupMetadata.Validate(); err != nil { + b.logger.Error("Failed to validate backup metadata", "error", err) + return + } + + backup.EncryptionSalt = backupMetadata.EncryptionSalt + backup.EncryptionIV = backupMetadata.EncryptionIV + backup.Encryption = backupMetadata.Encryption + } + + if backupMetadata != nil { + metadataJSON, err := json.Marshal(backupMetadata) + if err != nil { + b.logger.Error("Failed to marshal backup metadata to JSON", + "backupId", backup.ID, + "error", err, + ) + } else { + metadataReader := bytes.NewReader(metadataJSON) + metadataFileName := backup.FileName + ".metadata" + + if err := storage.SaveFile( + context.Background(), + b.fieldEncryptor, + b.logger, + metadataFileName, + metadataReader, + ); err != nil { + b.logger.Error("Failed to save backup metadata file to storage", + "backupId", backup.ID, + "fileName", metadataFileName, + "error", err, + ) + } else { + b.logger.Info("Backup metadata file saved successfully", + "backupId", backup.ID, + "fileName", metadataFileName, + ) + } + } + } + + backup.Status = backups_core_logical.BackupStatusCompleted + + if err := b.backupRepository.Save(backup); err != nil { + b.logger.Error("Failed to save backup", "error", err) + return + } + + // Update database last backup time + now := time.Now().UTC() + if updateErr := b.databaseService.SetLastBackupTime(databaseID, now); updateErr != nil { + b.logger.Error( + "Failed to update database last backup time", + "databaseId", + databaseID, + "error", + updateErr, + ) + } + + if backup.Status != backups_core_logical.BackupStatusCompleted && !isCallNotifier { + return + } + + b.SendBackupNotification( + backupConfig, + backup, + backups_config_logical.NotificationBackupSuccess, + nil, + ) +} + +func (b *Backuper) SendBackupNotification( + backupConfig *backups_config_logical.LogicalBackupConfig, + backup *backups_core_logical.LogicalBackup, + notificationType backups_config_logical.BackupNotificationType, + errorMessage *string, +) { + database, err := b.databaseService.GetDatabaseByID(backupConfig.DatabaseID) + if err != nil { + return + } + + workspace, err := b.workspaceService.GetWorkspaceByID(*database.WorkspaceID) + if err != nil { + return + } + + for _, notifier := range database.Notifiers { + if !slices.Contains( + backupConfig.SendNotificationsOn, + notificationType, + ) { + continue + } + + title := "" + switch notificationType { + case backups_config_logical.NotificationBackupFailed: + title = fmt.Sprintf( + "❌ Backup failed for database \"%s\" (workspace \"%s\")", + database.Name, + workspace.Name, + ) + case backups_config_logical.NotificationBackupSuccess: + title = fmt.Sprintf( + "✅ Backup completed for database \"%s\" (workspace \"%s\")", + database.Name, + workspace.Name, + ) + } + + message := "" + if errorMessage != nil { + message = *errorMessage + } else { + // Format size conditionally + var sizeStr string + if backup.BackupSizeMb < 1024 { + sizeStr = fmt.Sprintf("%.2f MB", backup.BackupSizeMb) + } else { + sizeGB := backup.BackupSizeMb / 1024 + sizeStr = fmt.Sprintf("%.2f GB", sizeGB) + } + + // Format duration as "0m 0s 0ms" + totalMs := backup.BackupDurationMs + minutes := totalMs / (1000 * 60) + seconds := (totalMs % (1000 * 60)) / 1000 + durationStr := fmt.Sprintf("%dm %ds", minutes, seconds) + + message = fmt.Sprintf( + "Backup completed successfully in %s.\nCompressed backup size: %s", + durationStr, + sizeStr, + ) + } + + b.notificationSender.SendNotification( + ¬ifier, + title, + message, + ) + } +} diff --git a/backend/internal/features/backups/backups/backuping/logical/backuper_test.go b/backend/internal/features/backups/backups/backuping/logical/backuper_test.go new file mode 100644 index 0000000..d34f69b --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/backuper_test.go @@ -0,0 +1,155 @@ +package backuping_logical + +import ( + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + cache_utils "databasus-backend/internal/util/cache" +) + +func Test_BackupExecuted_NotificationSent(t *testing.T) { + cache_utils.ClearAllCache() + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + backups_config_logical.EnableBackupsForTestDatabase(database.ID, storage) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) // Wait for cascading deletes + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + t.Run("BackupFailed_FailNotificationSent", func(t *testing.T) { + mockNotificationSender := &MockNotificationSender{} + backuper := CreateTestBackuper() + backuper.notificationSender = mockNotificationSender + backuper.createBackupUseCase = &CreateFailedBackupUsecase{} + + // Create a backup record directly that will be looked up by MakeBackup + backup := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + CreatedAt: time.Now().UTC(), + } + err := backupRepository.Save(backup) + assert.NoError(t, err) + + // Set up expectations + mockNotificationSender.On("SendNotification", + mock.Anything, + mock.MatchedBy(func(title string) bool { + return strings.Contains(title, "❌ Backup failed") + }), + mock.MatchedBy(func(message string) bool { + return strings.Contains(message, "backup failed") + }), + ).Once() + + backuper.MakeBackup(backup.ID, true) + + // Verify all expectations were met + mockNotificationSender.AssertExpectations(t) + }) + + t.Run("BackupSuccess_SuccessNotificationSent", func(t *testing.T) { + mockNotificationSender := &MockNotificationSender{} + backuper := CreateTestBackuper() + backuper.notificationSender = mockNotificationSender + backuper.createBackupUseCase = &CreateSuccessBackupUsecase{} + + // Create a backup record directly that will be looked up by MakeBackup + backup := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + CreatedAt: time.Now().UTC(), + } + err := backupRepository.Save(backup) + assert.NoError(t, err) + + // Set up expectations + mockNotificationSender.On("SendNotification", + mock.Anything, + mock.MatchedBy(func(title string) bool { + return strings.Contains(title, "✅ Backup completed") + }), + mock.MatchedBy(func(message string) bool { + return strings.Contains(message, "Backup completed successfully") + }), + ).Once() + + backuper.MakeBackup(backup.ID, true) + + // Verify all expectations were met + mockNotificationSender.AssertExpectations(t) + }) + + t.Run("BackupSuccess_VerifyNotificationContent", func(t *testing.T) { + mockNotificationSender := &MockNotificationSender{} + backuper := CreateTestBackuper() + backuper.notificationSender = mockNotificationSender + backuper.createBackupUseCase = &CreateSuccessBackupUsecase{} + + // Create a backup record directly that will be looked up by MakeBackup + backup := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + CreatedAt: time.Now().UTC(), + } + err := backupRepository.Save(backup) + assert.NoError(t, err) + + // capture arguments + var capturedNotifier *notifiers.Notifier + var capturedTitle string + var capturedMessage string + + mockNotificationSender.On("SendNotification", + mock.Anything, + mock.AnythingOfType("string"), + mock.AnythingOfType("string"), + ).Run(func(args mock.Arguments) { + capturedNotifier = args.Get(0).(*notifiers.Notifier) + capturedTitle = args.Get(1).(string) + capturedMessage = args.Get(2).(string) + }).Once() + + backuper.MakeBackup(backup.ID, true) + + // Verify expectations were met + mockNotificationSender.AssertExpectations(t) + + // Additional detailed assertions + assert.Contains(t, capturedTitle, "✅ Backup completed") + assert.Contains(t, capturedTitle, database.Name) + assert.Contains(t, capturedMessage, "Backup completed successfully") + assert.Contains(t, capturedMessage, "10.00 MB") + assert.Equal(t, notifier.ID, capturedNotifier.ID) + }) +} diff --git a/backend/internal/features/backups/backups/backuping/logical/cleaner.go b/backend/internal/features/backups/backups/backuping/logical/cleaner.go new file mode 100644 index 0000000..e44d1d0 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/cleaner.go @@ -0,0 +1,263 @@ +package backuping_logical + +import ( + "context" + "fmt" + "log/slog" + "sync/atomic" + "time" + + "github.com/google/uuid" + + "databasus-backend/internal/features/backups/backups/backuping/shared/gfs" + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/storages" + util_encryption "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/period" +) + +const ( + cleanerTickerInterval = 3 * time.Second + recentBackupGracePeriod = 60 * time.Minute +) + +type BackupCleaner struct { + backupRepository *backups_core_logical.BackupRepository + storageService *storages.StorageService + backupConfigService *backups_config_logical.BackupConfigService + fieldEncryptor util_encryption.FieldEncryptor + logger *slog.Logger + backupRemoveListeners []backups_core_logical.BackupRemoveListener + + hasRun atomic.Bool +} + +func (c *BackupCleaner) Run(ctx context.Context) { + if c.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", c)) + } + + if ctx.Err() != nil { + return + } + + retentionLog := c.logger.With("task_name", "clean_by_retention_policy") + + ticker := time.NewTicker(cleanerTickerInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if err := c.cleanByRetentionPolicy(retentionLog); err != nil { + retentionLog.Error("failed to clean backups by retention policy", "error", err) + } + } + } +} + +func (c *BackupCleaner) DeleteBackup(backup *backups_core_logical.LogicalBackup) error { + for _, listener := range c.backupRemoveListeners { + if err := listener.OnBeforeBackupRemove(backup); err != nil { + return err + } + } + + storage, err := c.storageService.GetStorageByID(backup.StorageID) + if err != nil { + return err + } + + if err := storage.DeleteFile(c.fieldEncryptor, backup.FileName); err != nil { + // we do not return error here, because sometimes clean up performed + // before unavailable storage removal or change - therefore we should + // proceed even in case of error. It's possible that some S3 or + // storage is not available yet, it should not block us + c.logger.Error("Failed to delete backup file", "error", err) + } + + metadataFileName := backup.FileName + ".metadata" + if err := storage.DeleteFile(c.fieldEncryptor, metadataFileName); err != nil { + c.logger.Error("Failed to delete backup metadata file", "error", err) + } + + return c.backupRepository.DeleteByID(backup.ID) +} + +func (c *BackupCleaner) AddBackupRemoveListener(listener backups_core_logical.BackupRemoveListener) { + c.backupRemoveListeners = append(c.backupRemoveListeners, listener) +} + +func (c *BackupCleaner) cleanByRetentionPolicy(logger *slog.Logger) error { + enabledBackupConfigs, err := c.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + return err + } + + for _, backupConfig := range enabledBackupConfigs { + dbLog := logger.With("database_id", backupConfig.DatabaseID, "policy", backupConfig.RetentionPolicyType) + + var cleanErr error + + switch backupConfig.RetentionPolicyType { + case backups_config_logical.RetentionPolicyTypeCount: + cleanErr = c.cleanByCount(dbLog, backupConfig) + case backups_config_logical.RetentionPolicyTypeGFS: + cleanErr = c.cleanByGFS(dbLog, backupConfig) + default: + cleanErr = c.cleanByTimePeriod(dbLog, backupConfig) + } + + if cleanErr != nil { + dbLog.Error("failed to clean backups by retention policy", "error", cleanErr) + } + } + + return nil +} + +func (c *BackupCleaner) cleanByTimePeriod( + logger *slog.Logger, + backupConfig *backups_config_logical.LogicalBackupConfig, +) error { + if backupConfig.RetentionTimePeriod == "" { + return nil + } + + if backupConfig.RetentionTimePeriod == period.PeriodForever { + return nil + } + + cutoff := time.Now().UTC().Add(-backupConfig.RetentionTimePeriod.ToDuration()) + + oldBackups, err := c.backupRepository.FindBackupsBeforeDate(backupConfig.DatabaseID, cutoff) + if err != nil { + return fmt.Errorf("failed to find old backups for database %s: %w", backupConfig.DatabaseID, err) + } + + for _, backup := range oldBackups { + if isRecentBackup(backup) { + continue + } + + if err := c.DeleteBackup(backup); err != nil { + logger.Error("failed to delete backup", "backup_id", backup.ID, "error", err) + continue + } + + logger.Info("deleted old backup", "backup_id", backup.ID) + } + + return nil +} + +func (c *BackupCleaner) cleanByCount( + logger *slog.Logger, + backupConfig *backups_config_logical.LogicalBackupConfig, +) error { + if backupConfig.RetentionCount <= 0 { + return nil + } + + completedBackups, err := c.findCompletedBackups(backupConfig.DatabaseID) + if err != nil { + return err + } + + if len(completedBackups) <= backupConfig.RetentionCount { + return nil + } + + successMsg := fmt.Sprintf("deleted backup by count policy: retention count is %d", backupConfig.RetentionCount) + for _, backup := range completedBackups[backupConfig.RetentionCount:] { + if isRecentBackup(backup) { + continue + } + + if err := c.DeleteBackup(backup); err != nil { + logger.Error("failed to delete backup", "backup_id", backup.ID, "error", err) + continue + } + + logger.Info(successMsg, "backup_id", backup.ID) + } + + return nil +} + +func (c *BackupCleaner) cleanByGFS( + logger *slog.Logger, + backupConfig *backups_config_logical.LogicalBackupConfig, +) error { + if backupConfig.RetentionGfsHours <= 0 && backupConfig.RetentionGfsDays <= 0 && + backupConfig.RetentionGfsWeeks <= 0 && backupConfig.RetentionGfsMonths <= 0 && + backupConfig.RetentionGfsYears <= 0 { + return nil + } + + completedBackups, err := c.findCompletedBackups(backupConfig.DatabaseID) + if err != nil { + return err + } + + keepSet := buildGFSKeepSet( + completedBackups, + backupConfig.RetentionGfsHours, + backupConfig.RetentionGfsDays, + backupConfig.RetentionGfsWeeks, + backupConfig.RetentionGfsMonths, + backupConfig.RetentionGfsYears, + ) + + for _, backup := range completedBackups { + if keepSet[backup.ID] { + continue + } + + if isRecentBackup(backup) { + continue + } + + if err := c.DeleteBackup(backup); err != nil { + logger.Error("failed to delete backup", "backup_id", backup.ID, "error", err) + continue + } + + logger.Info("deleted backup by GFS policy", "backup_id", backup.ID) + } + + return nil +} + +func (c *BackupCleaner) findCompletedBackups(databaseID uuid.UUID) ([]*backups_core_logical.LogicalBackup, error) { + completed, err := c.backupRepository.FindByDatabaseIdAndStatus( + databaseID, + backups_core_logical.BackupStatusCompleted, + ) + if err != nil { + return nil, fmt.Errorf("failed to find completed backups for database %s: %w", databaseID, err) + } + + return completed, nil +} + +func isRecentBackup(backup *backups_core_logical.LogicalBackup) bool { + return time.Since(backup.CreatedAt) < recentBackupGracePeriod +} + +// buildGFSKeepSet projects logical backups onto the shared GFS keep-set +// algorithm. Backups must be sorted newest-first. +func buildGFSKeepSet( + backups []*backups_core_logical.LogicalBackup, + hours, days, weeks, months, years int, +) map[uuid.UUID]bool { + items := make([]gfs.Item, len(backups)) + for i, backup := range backups { + items[i] = gfs.Item{ID: backup.ID, CreatedAt: backup.CreatedAt} + } + + return gfs.GetItemsToRetain(items, hours, days, weeks, months, years) +} diff --git a/backend/internal/features/backups/backups/backuping/logical/cleaner_gfs_test.go b/backend/internal/features/backups/backups/backuping/logical/cleaner_gfs_test.go new file mode 100644 index 0000000..57ec19b --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/cleaner_gfs_test.go @@ -0,0 +1,1165 @@ +package backuping_logical + +import ( + "fmt" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" +) + +func Test_BuildGFSKeepSet(t *testing.T) { + // Fixed reference time: a Wednesday mid-month to avoid boundary edge cases in the default tests. + // Use time.Date for determinism across test runs. + ref := time.Date(2025, 6, 18, 12, 0, 0, 0, time.UTC) // Wednesday, 2025-06-18 + + day := 24 * time.Hour + week := 7 * day + + newBackup := func(createdAt time.Time) *backups_core_logical.LogicalBackup { + return &backups_core_logical.LogicalBackup{ID: uuid.New(), CreatedAt: createdAt} + } + + // backupsEveryDay returns n backups, newest-first, each 1 day apart. + backupsEveryDay := func(n int) []*backups_core_logical.LogicalBackup { + bs := make([]*backups_core_logical.LogicalBackup, n) + for i := range n { + bs[i] = newBackup(ref.Add(-time.Duration(i) * day)) + } + return bs + } + + // backupsEveryWeek returns n backups, newest-first, each 7 days apart. + backupsEveryWeek := func(n int) []*backups_core_logical.LogicalBackup { + bs := make([]*backups_core_logical.LogicalBackup, n) + for i := range n { + bs[i] = newBackup(ref.Add(-time.Duration(i) * week)) + } + return bs + } + + hour := time.Hour + + // backupsEveryHour returns n backups, newest-first, each 1 hour apart. + backupsEveryHour := func(n int) []*backups_core_logical.LogicalBackup { + bs := make([]*backups_core_logical.LogicalBackup, n) + for i := range n { + bs[i] = newBackup(ref.Add(-time.Duration(i) * hour)) + } + return bs + } + + // backupsEveryMonth returns n backups, newest-first, each ~1 month apart. + backupsEveryMonth := func(n int) []*backups_core_logical.LogicalBackup { + bs := make([]*backups_core_logical.LogicalBackup, n) + for i := range n { + bs[i] = newBackup(ref.AddDate(0, -i, 0)) + } + return bs + } + + // backupsEveryYear returns n backups, newest-first, each 1 year apart. + backupsEveryYear := func(n int) []*backups_core_logical.LogicalBackup { + bs := make([]*backups_core_logical.LogicalBackup, n) + for i := range n { + bs[i] = newBackup(ref.AddDate(-i, 0, 0)) + } + return bs + } + + tests := []struct { + name string + backups []*backups_core_logical.LogicalBackup + hours int + days int + weeks int + months int + years int + keptIndices []int // which indices in backups should be kept + deletedRange *[2]int // optional: all indices in [from, to) must be deleted + }{ + { + name: "OnlyHourlySlots_KeepsNewest3Of5", + backups: backupsEveryHour(5), + hours: 3, + keptIndices: []int{0, 1, 2}, + }, + { + name: "SameHourDedup_OnlyNewestKeptForHourlySlot", + backups: []*backups_core_logical.LogicalBackup{ + newBackup(ref.Truncate(hour).Add(45 * time.Minute)), + newBackup(ref.Truncate(hour).Add(10 * time.Minute)), + }, + hours: 1, + keptIndices: []int{0}, + }, + { + name: "OnlyDailySlots_KeepsNewest3Of5", + backups: backupsEveryDay(5), + days: 3, + keptIndices: []int{0, 1, 2}, + }, + { + name: "OnlyDailySlots_FewerBackupsThanSlots_KeepsAll", + backups: backupsEveryDay(2), + days: 5, + keptIndices: []int{0, 1}, + }, + { + name: "OnlyWeeklySlots_KeepsNewest2Weeks", + backups: backupsEveryWeek(4), + weeks: 2, + keptIndices: []int{0, 1}, + }, + { + name: "OnlyMonthlySlots_KeepsNewest2Months", + backups: []*backups_core_logical.LogicalBackup{ + newBackup(time.Date(2025, 6, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2025, 5, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2025, 4, 1, 12, 0, 0, 0, time.UTC)), + }, + months: 2, + keptIndices: []int{0, 1}, + }, + { + name: "OnlyYearlySlots_KeepsNewest2Years", + backups: []*backups_core_logical.LogicalBackup{ + newBackup(time.Date(2025, 6, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2024, 6, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2023, 6, 1, 12, 0, 0, 0, time.UTC)), + }, + years: 2, + keptIndices: []int{0, 1}, + }, + { + name: "SameDayDedup_OnlyNewestKeptForDailySlot", + backups: []*backups_core_logical.LogicalBackup{ + // Two backups on the same day; newest-first order + newBackup(ref.Truncate(day).Add(10 * time.Hour)), + newBackup(ref.Truncate(day).Add(2 * time.Hour)), + }, + days: 1, + keptIndices: []int{0}, + }, + { + name: "SameWeekDedup_OnlyNewestKeptForWeeklySlot", + backups: []*backups_core_logical.LogicalBackup{ + // ref is Wednesday; add Thursday of same week + newBackup(ref.Add(1 * day)), // Thursday same week + newBackup(ref), // Wednesday same week + }, + weeks: 1, + keptIndices: []int{0}, + }, + { + name: "AdditiveSlots_NewestFillsDailyAndWeeklyAndMonthly", + // Newest backup fills daily + weekly + monthly simultaneously + backups: []*backups_core_logical.LogicalBackup{ + newBackup(time.Date(2025, 6, 18, 12, 0, 0, 0, time.UTC)), // newest + newBackup(time.Date(2025, 6, 11, 12, 0, 0, 0, time.UTC)), // 1 week ago + newBackup(time.Date(2025, 5, 18, 12, 0, 0, 0, time.UTC)), // 1 month ago + newBackup(time.Date(2025, 4, 18, 12, 0, 0, 0, time.UTC)), // 2 months ago + }, + days: 1, + weeks: 2, + months: 2, + keptIndices: []int{0, 1, 2}, + }, + { + name: "YearBoundary_CorrectlySplitsAcrossYears", + backups: []*backups_core_logical.LogicalBackup{ + newBackup(time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2024, 12, 31, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2024, 6, 1, 12, 0, 0, 0, time.UTC)), + newBackup(time.Date(2023, 6, 1, 12, 0, 0, 0, time.UTC)), + }, + years: 2, + keptIndices: []int{0, 1}, // 2025 and 2024 kept; 2024-06 and 2023 deleted + }, + { + name: "ISOWeekBoundary_Jan1UsesCorrectISOWeek", + // 2025-01-01 is ISO week 1 of 2025; 2024-12-28 is ISO week 52 of 2024 + backups: []*backups_core_logical.LogicalBackup{ + newBackup(time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC)), // ISO week 2025-W01 + newBackup(time.Date(2024, 12, 28, 12, 0, 0, 0, time.UTC)), // ISO week 2024-W52 + }, + weeks: 2, + keptIndices: []int{0, 1}, // different ISO weeks → both kept + }, + { + name: "EmptyBackups_ReturnsEmptyKeepSet", + backups: []*backups_core_logical.LogicalBackup{}, + hours: 3, + days: 3, + weeks: 2, + months: 1, + years: 1, + keptIndices: []int{}, + }, + { + name: "AllZeroSlots_KeepsNothing", + backups: backupsEveryDay(5), + hours: 0, + days: 0, + weeks: 0, + months: 0, + years: 0, + keptIndices: []int{}, + }, + { + name: "AllSlotsActive_FullCombination", + backups: backupsEveryWeek(12), + days: 2, + weeks: 3, + months: 2, + years: 1, + // 2 daily (indices 0,1) + 3rd weekly slot (index 2) + 2nd monthly slot (index 3 or later). + // Additive slots: newest fills daily+weekly+monthly+yearly; each subsequent week fills another weekly, + // and a backup ~4 weeks later fills the 2nd monthly slot. + keptIndices: []int{0, 1, 2, 3}, + }, + { + name: "RealisticGFS_20DailyBackups_7Days4Weeks1Month", + backups: backupsEveryDay(20), + days: 7, + weeks: 4, + months: 1, + // 7 daily: indices 0-6 (Jun 18-12). Index 0 also fills week 25 + month 2025-06. + // Index 3 (Jun 15, Sun) fills week 24. Indices 7-9 are week 24 (seen) → not kept. + // Index 10 (Jun 8, Sun) fills week 23. Indices 11-16 are week 23 (seen) → not kept. + // Index 17 (Jun 1, Sun) fills week 22. Index 18-19 are week 22 (seen) → not kept. + // Monthly slot 1 filled by index 0 (2025-06). Total kept: 9. + keptIndices: []int{0, 1, 2, 3, 4, 5, 6, 10, 17}, + }, + + // Cross-level absorption tests: when backup frequency is lower than slot granularity, + // each backup gets a unique key at the higher-frequency level, filling slots that + // should only cover recent time periods. This causes excess backups to be retained. + + // Adjacent-level absorption: + { + // Daily backups with hourly slots: each daily backup has a unique hour key + // (different date-hour combo), so 23 backups fill 23/24 hourly slots. + // Hourly slots should only cover the most recent 24 hours, not span weeks. + name: "HourlyAbsorbsDaily_DailyBackupsWithHourlySlots_KeepsTooMany", + backups: backupsEveryDay(23), + hours: 24, + days: 7, + weeks: 4, + months: 12, + years: 3, + // Correct behavior: 10 kept (7 daily + 2 extra weekly + 1 monthly for 2025-05). + // Index 18 (May 31) is the first backup in month "2025-05" within the 12-month window. + // Bug: all 23 kept because each daily backup fills a unique hourly slot. + keptIndices: []int{0, 1, 2, 3, 4, 5, 6, 10, 17, 18}, + }, + { + // Weekly backups with daily slots: each weekly backup has a unique day key, + // so 10 weekly backups fill 7/7 daily slots + 3 weekly slots = all 10 kept. + // Daily slots should only cover the most recent 7 days, not span months. + name: "DailyAbsorbsWeekly_WeeklyBackupsWithDailySlots_KeepsTooMany", + backups: backupsEveryWeek(10), + days: 7, + weeks: 4, + // Correct behavior: ~8 kept (4 weekly, some overlap with daily slots that + // should only cover recent 7 days). Extra daily slots shouldn't retain + // backups older than 7 days. + // Bug: all 10 kept because each weekly backup fills a unique day slot. + keptIndices: []int{0, 1, 2, 3}, + }, + { + // Monthly backups with many weekly slots: each monthly backup has a unique week key, + // so 8 monthly backups fill 8/52 weekly slots, all getting kept. + // Weekly slots should only cover the most recent weeks, not span years. + name: "WeeklyAbsorbsMonthly_MonthlyBackupsWithWeeklySlots_KeepsTooMany", + backups: backupsEveryMonth(8), + weeks: 52, + months: 3, + // Correct behavior: 3 kept (3 monthly slots, weekly should only cover recent 52 weeks + // but not artificially retain monthly backups). + // Bug: all 8 kept because each monthly backup fills a unique week slot. + keptIndices: []int{0, 1, 2}, + }, + { + // Yearly backups with monthly slots: each yearly backup (on different month of year) + // has a unique month key, so 5 yearly backups fill 5/12 monthly slots = all 5 kept. + // Monthly slots should only cover the most recent 12 months, not span decades. + name: "MonthlyAbsorbsYearly_YearlyBackupsWithMonthlySlots_KeepsTooMany", + backups: backupsEveryYear(5), + months: 12, + years: 3, + // Correct behavior: 3 kept (3 yearly slots). + // Bug: all 5 kept because each yearly backup fills a unique month slot. + keptIndices: []int{0, 1, 2}, + }, + + // Non-adjacent (skip-level) absorption: + { + // Weekly backups with hourly slots: each weekly backup has a unique hour key, + // so 8 weekly backups fill 8/24 hourly slots = all 8 kept. + name: "HourlyAbsorbsWeekly_WeeklyBackupsWithHourlySlots_KeepsTooMany", + backups: backupsEveryWeek(8), + hours: 24, + weeks: 4, + // Correct behavior: 4 kept (4 weekly slots, hourly should only cover recent 24h). + // Bug: all 8 kept because each weekly backup fills a unique hourly slot. + keptIndices: []int{0, 1, 2, 3}, + }, + { + // Monthly backups with daily slots: each monthly backup has a unique day key, + // so 6 monthly backups fill 6/7 daily slots = all 6 kept. + name: "DailyAbsorbsMonthly_MonthlyBackupsWithDailySlots_KeepsTooMany", + backups: backupsEveryMonth(6), + days: 7, + months: 3, + // Correct behavior: 3 kept (3 monthly slots, daily should only cover recent 7 days). + // Bug: all 6 kept because each monthly backup fills a unique day slot. + keptIndices: []int{0, 1, 2}, + }, + + // Full-stack (production-like config): + { + // Production config with all levels active. Daily backups span 30 days. + // Hourly slots absorb everything because each daily backup has a unique hour key. + name: "FullStack_AllLevelsAbsorb_DailyBackupsWithFullConfig_KeepsTooMany", + backups: backupsEveryDay(30), + hours: 24, + days: 7, + weeks: 4, + months: 1, + years: 1, + // Correct behavior: ~9 kept (7 daily + ~2 extra weekly, month+year overlap). + // Bug: 24+ kept because hourly absorbs 24 unique date-hour combos. + keptIndices: []int{0, 1, 2, 3, 4, 5, 6, 10, 17}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + keepSet := buildGFSKeepSet(tc.backups, tc.hours, tc.days, tc.weeks, tc.months, tc.years) + + keptIndexSet := make(map[int]bool, len(tc.keptIndices)) + for _, idx := range tc.keptIndices { + keptIndexSet[idx] = true + } + + for i, backup := range tc.backups { + if keptIndexSet[i] { + assert.True(t, keepSet[backup.ID], + "backup at index %d (date=%s) should be kept", + i, backup.CreatedAt.Format("2006-01-02 15:04")) + } else { + assert.False(t, keepSet[backup.ID], + "backup at index %d (date=%s) should be deleted", + i, backup.CreatedAt.Format("2006-01-02 15:04")) + } + } + }) + } +} + +func Test_CleanByGFS_KeepsCorrectBackupsPerSlot(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 3, + RetentionGfsWeeks: 0, + RetentionGfsMonths: 0, + RetentionGfsYears: 0, + StorageID: &storage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 5 backups on 5 different days; only the 3 newest days should be kept + var backupIDs []uuid.UUID + for i := range 5 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-time.Duration(4-i) * 24 * time.Hour).Truncate(24 * time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + backupIDs = append(backupIDs, backup.ID) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 3, len(remainingBackups)) + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + assert.False(t, remainingIDs[backupIDs[0]], "Oldest daily backup should be deleted") + assert.False(t, remainingIDs[backupIDs[1]], "2nd oldest daily backup should be deleted") + assert.True(t, remainingIDs[backupIDs[2]], "3rd backup should remain") + assert.True(t, remainingIDs[backupIDs[3]], "4th backup should remain") + assert.True(t, remainingIDs[backupIDs[4]], "Newest backup should remain") +} + +func Test_CleanByGFS_WithWeeklyAndMonthlySlots_KeepsWiderSpread(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 2, + RetentionGfsWeeks: 2, + RetentionGfsMonths: 1, + RetentionGfsYears: 0, + StorageID: &storage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create one backup per week for 6 weeks (each on Monday of that week) + // GFS should keep: 2 daily (most recent 2 unique days) + 2 weekly + 1 monthly = up to 5 unique + var createdIDs []uuid.UUID + for i := range 6 { + weekOffset := time.Duration(5-i) * 7 * 24 * time.Hour + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-weekOffset).Truncate(24 * time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + createdIDs = append(createdIDs, backup.ID) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + // We should have at most 5 backups kept (2 daily + 2 weekly + 1 monthly, but with overlap possible) + // The exact count depends on how many unique periods are covered + assert.LessOrEqual(t, len(remainingBackups), 5) + assert.GreaterOrEqual(t, len(remainingBackups), 1) + + // The two most recent backups should always be retained (daily slots) + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + assert.True(t, remainingIDs[createdIDs[4]], "Second newest backup should be retained (daily)") + assert.True(t, remainingIDs[createdIDs[5]], "Newest backup should be retained (daily)") +} + +func Test_CleanByGFS_WithHourlySlots_KeepsCorrectBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsHours: 3, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 5 backups spaced 1 hour apart; only the 3 newest hours should be kept + var backupIDs []uuid.UUID + for i := range 5 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-time.Duration(4-i) * time.Hour).Truncate(time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + backupIDs = append(backupIDs, backup.ID) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 3, len(remainingBackups)) + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + assert.False(t, remainingIDs[backupIDs[0]], "Oldest hourly backup should be deleted") + assert.False(t, remainingIDs[backupIDs[1]], "2nd oldest hourly backup should be deleted") + assert.True(t, remainingIDs[backupIDs[2]], "3rd backup should remain") + assert.True(t, remainingIDs[backupIDs[3]], "4th backup should remain") + assert.True(t, remainingIDs[backupIDs[4]], "Newest backup should remain") +} + +func Test_CleanByGFS_SkipsRecentBackup_WhenNotInKeepSet(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + // Keep only 1 daily slot. We create 2 old backups plus two recent backups on today. + // Backups are ordered newest-first, so the 15-min-old backup fills the single daily slot. + // The 30-min-old backup is the same day → not in the GFS keep-set, but it is still recent + // (within grace period) and must be preserved. + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 1, + StorageID: &storage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + oldBackup1 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-3 * 24 * time.Hour).Truncate(24 * time.Hour), + } + oldBackup2 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-2 * 24 * time.Hour).Truncate(24 * time.Hour), + } + // Newest backup today — will fill the single GFS daily slot. + newestTodayBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-15 * time.Minute), + } + // Slightly older backup, also today — NOT in GFS keep-set (duplicate day), + // but within the 60-minute grace period so it must survive. + recentNotInKeepSet := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-30 * time.Minute), + } + + for _, b := range []*backups_core_logical.LogicalBackup{oldBackup1, oldBackup2, newestTodayBackup, recentNotInKeepSet} { + err = backupRepository.Save(b) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + + assert.False(t, remainingIDs[oldBackup1.ID], "Old backup 1 should be deleted by GFS") + assert.False(t, remainingIDs[oldBackup2.ID], "Old backup 2 should be deleted by GFS") + assert.True( + t, + remainingIDs[newestTodayBackup.ID], + "Newest backup fills GFS daily slot and must remain", + ) + assert.True( + t, + remainingIDs[recentNotInKeepSet.ID], + "Recent backup not in keep-set must be preserved by grace period", + ) +} + +func Test_CleanByGFS_With20DailyBackups_KeepsOnlyExpectedCount(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 7, + RetentionGfsWeeks: 4, + RetentionGfsMonths: 1, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 20 daily backups, all older than the 60-minute grace period. + // backupIDs[0] is oldest (19 days ago), backupIDs[19] is newest (~2h ago). + var backupIDs []uuid.UUID + for i := range 20 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-time.Duration(19-i)*24*time.Hour - 2*time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + backupIDs = append(backupIDs, backup.ID) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + // GFS 7d/4w/1m: 7 daily slots + up to 4 weekly + 1 monthly (with overlap). + // Theoretical max unique kept: 12. Realistic: ~9 depending on ISO week layout. + assert.LessOrEqual(t, len(remainingBackups), 12, + "At most 12 backups should be retained (7d+4w+1m)") + assert.GreaterOrEqual(t, len(remainingBackups), 7, + "At least 7 backups should be retained for daily slots") + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + + // The newest 7 backups (indices 13-19) must always be retained (daily slots) + for i := 13; i < 20; i++ { + assert.True(t, remainingIDs[backupIDs[i]], + "Backup at position %d (one of 7 newest) should be retained", i) + } +} + +func Test_CleanByGFS_WithMultipleBackupsPerDay_KeepsOnlyOnePerDailySlot(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 7, + RetentionGfsWeeks: 4, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 3 backups per day for 10 days = 30 total, all beyond grace period. + // Each day gets backups at base+0h, base+6h, base+12h. + for day := range 10 { + for sub := range 3 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add( + -time.Duration(9-day)*24*time.Hour - + 2*time.Hour - + time.Duration(2-sub)*6*time.Hour, + ), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + // 10 days spans ~2 ISO weeks. GFS 7d/4w: weekly slots overlap with daily. + // Expected: ~7-9 backups (7 daily + possibly extra weekly for uncovered weeks). + assert.LessOrEqual(t, len(remainingBackups), 11, + "Should keep at most 11 backups (7d+4w theoretical max)") + assert.GreaterOrEqual(t, len(remainingBackups), 7, + "Should keep at least 7 backups for daily slots") + + // Only 1 backup per calendar day should be retained + dayCount := make(map[string]int) + for _, backup := range remainingBackups { + dayKey := backup.CreatedAt.Format("2006-01-02") + dayCount[dayKey]++ + } + + for dayKey, count := range dayCount { + assert.Equal(t, 1, count, + "Day %s should have exactly 1 retained backup, got %d", dayKey, count) + } +} + +// When backups are created once per day but hourly slots are configured (e.g. 24h), +// each daily backup lands in a unique hour key (different date-hour combo), filling +// hourly slots with old backups that shouldn't be retained by hourly rotation. +// Hourly slots should only cover the most recent hours, not span across weeks. +func Test_CleanByGFS_With24HourlySlotsAnd23DailyBackups_DeletesExcessBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsHours: 24, + RetentionGfsDays: 7, + RetentionGfsWeeks: 4, + RetentionGfsMonths: 12, + RetentionGfsYears: 3, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + for i := range 23 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 0.02, + CreatedAt: now.Add(-time.Duration(22-i)*24*time.Hour - 2*time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + assert.Less(t, len(remainingBackups), 20, + "GFS with 24h/7d/4w/12m/3y should not retain all 23 daily backups") + assert.GreaterOrEqual(t, len(remainingBackups), 7, + "At least 7 backups should be retained for daily slots") +} + +// Same scenario as above but with hourly slots disabled (0h). This verifies +// that the daily/weekly/monthly/yearly rotation correctly prunes excess backups +// when hourly slots are not involved. +func Test_CleanByGFS_WithDisabledHourlySlotsAnd23DailyBackups_DeletesExcessBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsHours: 0, + RetentionGfsDays: 7, + RetentionGfsWeeks: 4, + RetentionGfsMonths: 12, + RetentionGfsYears: 3, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + for i := range 23 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 0.02, + CreatedAt: now.Add(-time.Duration(22-i)*24*time.Hour - 2*time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + assert.Less(t, len(remainingBackups), 15, + "GFS with 0h/7d/4w/12m/3y should not retain all 23 daily backups") + assert.GreaterOrEqual(t, len(remainingBackups), 7, + "At least 7 backups should be retained for daily slots") +} + +// When weekly backups exist but daily slots are configured, each weekly backup +// has a unique day key, filling daily slots with backups that are weeks apart. +// Daily slots should only cover the most recent days, not span months. +func Test_CleanByGFS_WithDailySlotsAndWeeklyBackups_DeletesExcessBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsDays: 7, + RetentionGfsWeeks: 4, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 10 weekly backups (1 per week, all >2h old past grace period). + // With 7d/4w config, correct behavior: ~8 kept (4 weekly + overlap with daily for recent ones). + // Daily slots should NOT absorb weekly backups that are older than 7 days. + for i := range 10 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 0.02, + CreatedAt: now.Add(-time.Duration(9-i)*7*24*time.Hour - 2*time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + fmt.Printf( + "[TEST] WithDailySlotsAndWeeklyBackups: %d remaining out of 10\n", + len(remainingBackups), + ) + + // With weekly backups and 7d/4w config, daily slots should only cover the most recent + // 7 days. Since backups are 1 week apart, at most 1 backup falls within the daily window. + // Correct: ~4-5 kept (4 weekly + possibly 1 overlapping daily). Not 7+. + assert.LessOrEqual( + t, + len(remainingBackups), + 5, + "GFS with 7d/4w should keep at most ~5 weekly backups — daily slots should not absorb weekly backups older than 7 days", + ) + assert.GreaterOrEqual(t, len(remainingBackups), 4, + "At least 4 backups should be retained for weekly slots") +} + +// When monthly backups exist but weekly slots are configured, each monthly backup +// has a unique week key, filling weekly slots with backups that are months apart. +// Weekly slots should only cover the most recent weeks, not span years. +func Test_CleanByGFS_WithWeeklySlotsAndMonthlyBackups_DeletesExcessBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeGFS, + RetentionGfsWeeks: 52, + RetentionGfsMonths: 3, + StorageID: &testStorage.ID, + BackupInterval: interval, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + // Create 8 monthly backups (1 per month, all >2h old past grace period). + // With 52w/3m config, correct behavior: 3 kept (3 monthly slots; weekly should only + // cover recent 52 weeks but not artificially retain old monthly backups). + // Bug: all 8 kept because each monthly backup fills a unique weekly slot. + for i := range 8 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 0.02, + CreatedAt: now.AddDate(0, -(7 - i), 0).Add(-2 * time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + fmt.Printf( + "[TEST] WithWeeklySlotsAndMonthlyBackups: %d remaining out of 8\n", + len(remainingBackups), + ) + + // Weekly slots should not absorb monthly backups. Correct: ~3 kept (3 monthly). + assert.LessOrEqual( + t, + len(remainingBackups), + 5, + "GFS with 52w/3m should keep at most ~5 backups — weekly slots should not absorb monthly backups", + ) + assert.GreaterOrEqual(t, len(remainingBackups), 3, + "At least 3 backups should be retained for monthly slots") +} diff --git a/backend/internal/features/backups/backups/backuping/logical/cleaner_test.go b/backend/internal/features/backups/backups/backuping/logical/cleaner_test.go new file mode 100644 index 0000000..73264bb --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/cleaner_test.go @@ -0,0 +1,606 @@ +package backuping_logical + +import ( + "log/slog" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/intervals" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + "databasus-backend/internal/util/logger" + "databasus-backend/internal/util/period" +) + +func Test_CleanOldBackups_DeletesBackupsOlderThanRetentionTimePeriod(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeTimePeriod, + RetentionTimePeriod: period.PeriodWeek, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + oldBackup1 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-10 * 24 * time.Hour), + } + oldBackup2 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-8 * 24 * time.Hour), + } + recentBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-3 * 24 * time.Hour), + } + + err = backupRepository.Save(oldBackup1) + assert.NoError(t, err) + err = backupRepository.Save(oldBackup2) + assert.NoError(t, err) + err = backupRepository.Save(recentBackup) + assert.NoError(t, err) + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 1, len(remainingBackups)) + assert.Equal(t, recentBackup.ID, remainingBackups[0].ID) +} + +func Test_CleanOldBackups_SkipsDatabaseWithForeverRetentionPeriod(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeTimePeriod, + RetentionTimePeriod: period.PeriodForever, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + oldBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: time.Now().UTC().Add(-365 * 24 * time.Hour), + } + err = backupRepository.Save(oldBackup) + assert.NoError(t, err) + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 1, len(remainingBackups)) + assert.Equal(t, oldBackup.ID, remainingBackups[0].ID) +} + +func Test_CleanByCount_KeepsNewestNBackups_DeletesOlder(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeCount, + RetentionCount: 3, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + var backupIDs []uuid.UUID + for i := range 5 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add( + -time.Duration(4-i) * time.Hour, + ), // oldest first in loop, newest = i=4 + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + backupIDs = append(backupIDs, backup.ID) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 3, len(remainingBackups)) + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + assert.False(t, remainingIDs[backupIDs[0]], "Oldest backup should be deleted") + assert.False(t, remainingIDs[backupIDs[1]], "2nd oldest backup should be deleted") + assert.True(t, remainingIDs[backupIDs[2]], "3rd backup should remain") + assert.True(t, remainingIDs[backupIDs[3]], "4th backup should remain") + assert.True(t, remainingIDs[backupIDs[4]], "Newest backup should remain") +} + +func Test_CleanByCount_WhenUnderLimit_NoBackupsDeleted(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeCount, + RetentionCount: 10, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + for i := range 5 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: time.Now().UTC().Add(-time.Duration(i) * time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 5, len(remainingBackups)) +} + +func Test_CleanByCount_DoesNotDeleteInProgressBackups(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeCount, + RetentionCount: 2, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + for i := range 3 { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-time.Duration(3-i) * time.Hour), + } + err = backupRepository.Save(backup) + assert.NoError(t, err) + } + + inProgressBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 5, + CreatedAt: now, + } + err = backupRepository.Save(inProgressBackup) + assert.NoError(t, err) + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + var inProgressFound bool + for _, backup := range remainingBackups { + if backup.ID == inProgressBackup.ID { + inProgressFound = true + } + } + assert.True(t, inProgressFound, "In-progress backup should not be deleted by count policy") +} + +// Test_DeleteBackup_WhenStorageDeleteFails_BackupStillRemovedFromDatabase verifies resilience +// when storage becomes unavailable. Even if storage.DeleteFile fails (e.g., storage is offline, +// credentials changed, or storage was deleted), the backup record should still be removed from +// the database. This prevents orphaned backup records when storage is no longer accessible. +func Test_DeleteBackup_WhenStorageDeleteFails_BackupStillRemovedFromDatabase(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + testStorage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, testStorage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(testStorage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: testStorage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: time.Now().UTC(), + } + err := backupRepository.Save(backup) + assert.NoError(t, err) + + cleaner := GetBackupCleaner() + + err = cleaner.DeleteBackup(backup) + assert.NoError(t, err, "DeleteBackup should succeed even when storage file doesn't exist") + + deletedBackup, err := backupRepository.FindByID(backup.ID) + assert.Error(t, err, "Backup should not exist in database") + assert.Nil(t, deletedBackup) +} + +func Test_CleanByTimePeriod_SkipsRecentBackup_EvenIfOlderThanRetention(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + // Retention period is 1 day — any backup older than 1 day should be deleted. + // But the recent backup was created only 30 minutes ago and must be preserved. + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeTimePeriod, + RetentionTimePeriod: period.PeriodDay, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + oldBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-2 * 24 * time.Hour), + } + recentBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-30 * time.Minute), + } + + err = backupRepository.Save(oldBackup) + assert.NoError(t, err) + err = backupRepository.Save(recentBackup) + assert.NoError(t, err) + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Equal(t, 1, len(remainingBackups)) + assert.Equal(t, recentBackup.ID, remainingBackups[0].ID) +} + +func Test_CleanByCount_SkipsRecentBackup_EvenIfOverLimit(t *testing.T) { + router := CreateTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + interval := createTestInterval() + + // Retention count is 2 — 4 backups exist so 2 should be deleted. + // The oldest backup in the "excess" tail was made 30 min ago — it must be preserved. + backupConfig := &backups_config_logical.LogicalBackupConfig{ + DatabaseID: database.ID, + IsBackupsEnabled: true, + RetentionPolicyType: backups_config_logical.RetentionPolicyTypeCount, + RetentionCount: 2, + StorageID: &storage.ID, + BackupInterval: interval, + Encryption: backups_core_enums.BackupEncryptionEncrypted, + } + _, err := backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + now := time.Now().UTC() + + oldBackup1 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-5 * time.Hour), + } + oldBackup2 := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-3 * time.Hour), + } + // This backup is 3rd newest and would normally be deleted — but it is recent. + recentExcessBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-30 * time.Minute), + } + newestBackup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10, + CreatedAt: now.Add(-10 * time.Minute), + } + + for _, b := range []*backups_core_logical.LogicalBackup{oldBackup1, oldBackup2, recentExcessBackup, newestBackup} { + err = backupRepository.Save(b) + assert.NoError(t, err) + } + + cleaner := GetBackupCleaner() + err = cleaner.cleanByRetentionPolicy(testLogger()) + assert.NoError(t, err) + + remainingBackups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + remainingIDs := make(map[uuid.UUID]bool) + for _, backup := range remainingBackups { + remainingIDs[backup.ID] = true + } + + assert.False(t, remainingIDs[oldBackup1.ID], "Oldest non-recent backup should be deleted") + assert.False(t, remainingIDs[oldBackup2.ID], "2nd oldest non-recent backup should be deleted") + assert.True( + t, + remainingIDs[recentExcessBackup.ID], + "Recent backup must be preserved despite being over limit", + ) + assert.True(t, remainingIDs[newestBackup.ID], "Newest backup should be preserved") +} + +// Mock listener for testing +type mockBackupRemoveListener struct { + onBeforeBackupRemove func(*backups_core_logical.LogicalBackup) error +} + +func (m *mockBackupRemoveListener) OnBeforeBackupRemove(backup *backups_core_logical.LogicalBackup) error { + if m.onBeforeBackupRemove != nil { + return m.onBeforeBackupRemove(backup) + } + + return nil +} + +func testLogger() *slog.Logger { + return logger.GetLogger().With("task_name", "test") +} + +func createTestInterval() intervals.Interval { + timeOfDay := "04:00" + + return intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } +} diff --git a/backend/internal/features/backups/backups/backuping/logical/di.go b/backend/internal/features/backups/backups/backuping/logical/di.go new file mode 100644 index 0000000..eeff84a --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/di.go @@ -0,0 +1,69 @@ +package backuping_logical + +import ( + "sync/atomic" + "time" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + usecases_logical "databasus-backend/internal/features/backups/backups/usecases/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + workspaces_services "databasus-backend/internal/features/workspaces/services" + "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/logger" +) + +var backupRepository = &backups_core_logical.BackupRepository{} + +var taskCancelManager = tasks_cancellation.GetTaskCancelManager() + +var backupCleaner = &BackupCleaner{ + backupRepository, + storages.GetStorageService(), + backups_config_logical.GetBackupConfigService(), + encryption.GetFieldEncryptor(), + logger.GetLogger(), + []backups_core_logical.BackupRemoveListener{}, + atomic.Bool{}, +} + +var backuper = &Backuper{ + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + workspaces_services.GetWorkspaceService(), + backupRepository, + backups_config_logical.GetBackupConfigService(), + storages.GetStorageService(), + notifiers.GetNotifierService(), + taskCancelManager, + logger.GetLogger(), + usecases_logical.GetCreateBackupUsecase(), +} + +var backupsScheduler = &BackupsScheduler{ + backupRepository, + backups_config_logical.GetBackupConfigService(), + taskCancelManager, + databases.GetDatabaseService(), + time.Now().UTC(), + logger.GetLogger(), + backuper, + []backups_core_logical.BackupCompletionListener{}, + atomic.Bool{}, + atomic.Bool{}, +} + +func GetBackupsScheduler() *BackupsScheduler { + return backupsScheduler +} + +func GetBackuper() *Backuper { + return backuper +} + +func GetBackupCleaner() *BackupCleaner { + return backupCleaner +} diff --git a/backend/internal/features/backups/backups/backuping/logical/mocks.go b/backend/internal/features/backups/backups/backuping/logical/mocks.go new file mode 100644 index 0000000..1d91087 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/mocks.go @@ -0,0 +1,195 @@ +package backuping_logical + +import ( + "context" + "errors" + "sync/atomic" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/mock" + + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" +) + +type MockNotificationSender struct { + mock.Mock +} + +func (m *MockNotificationSender) SendNotification( + notifier *notifiers.Notifier, + title string, + message string, +) { + m.Called(notifier, title, message) +} + +type CreateFailedBackupUsecase struct{} + +func (uc *CreateFailedBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + backupProgressListener(10) + return nil, errors.New("backup failed") +} + +type CreateSuccessBackupUsecase struct{} + +func (uc *CreateSuccessBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + backupProgressListener(10) + return &backups_core_logical.BackupMetadata{ + EncryptionSalt: nil, + EncryptionIV: nil, + Encryption: backups_core_enums.BackupEncryptionNone, + }, nil +} + +// CreateLargeBackupUsecase simulates a large backup (10000 MB) +type CreateLargeBackupUsecase struct{} + +func (uc *CreateLargeBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + backupProgressListener(10000) + return &backups_core_logical.BackupMetadata{ + EncryptionSalt: nil, + EncryptionIV: nil, + Encryption: backups_core_enums.BackupEncryptionNone, + }, nil +} + +// CreateProgressiveBackupUsecase simulates progressive size updates that exceed limit +type CreateProgressiveBackupUsecase struct{} + +func (uc *CreateProgressiveBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + // Simulate progressive backup that grows beyond limit + backupProgressListener(1) + if ctx.Err() != nil { + return nil, ctx.Err() + } + + backupProgressListener(3) + if ctx.Err() != nil { + return nil, ctx.Err() + } + + backupProgressListener(5) + if ctx.Err() != nil { + return nil, ctx.Err() + } + + backupProgressListener(10) // This exceeds the 5 MB limit + if ctx.Err() != nil { + return nil, ctx.Err() + } + + // Should not reach here due to cancellation + return &backups_core_logical.BackupMetadata{ + EncryptionSalt: nil, + EncryptionIV: nil, + Encryption: backups_core_enums.BackupEncryptionNone, + }, nil +} + +// CreateMediumBackupUsecase simulates a 50 MB backup +type CreateMediumBackupUsecase struct{} + +func (uc *CreateMediumBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + backupProgressListener(50) + return &backups_core_logical.BackupMetadata{ + EncryptionSalt: nil, + EncryptionIV: nil, + Encryption: backups_core_enums.BackupEncryptionNone, + }, nil +} + +// MockTrackingBackupUsecase tracks backup use case calls for testing parallel execution +type MockTrackingBackupUsecase struct { + callCount atomic.Int32 + calledBackupIDs chan uuid.UUID +} + +func NewMockTrackingBackupUsecase() *MockTrackingBackupUsecase { + return &MockTrackingBackupUsecase{ + calledBackupIDs: make(chan uuid.UUID, 10), + } +} + +func (m *MockTrackingBackupUsecase) Execute( + ctx context.Context, + backup *backups_core_logical.LogicalBackup, + backupConfig *backups_config_logical.LogicalBackupConfig, + database *databases.Database, + storage *storages.Storage, + backupProgressListener func(completedMBs float64), +) (*backups_core_logical.BackupMetadata, error) { + m.callCount.Add(1) + + // Send backup ID to channel (non-blocking) + select { + case m.calledBackupIDs <- backup.ID: + default: + } + + // Simulate backup work + time.Sleep(100 * time.Millisecond) + backupProgressListener(10) + + return &backups_core_logical.BackupMetadata{ + EncryptionSalt: nil, + EncryptionIV: nil, + Encryption: backups_core_enums.BackupEncryptionNone, + }, nil +} + +func (m *MockTrackingBackupUsecase) GetCallCount() int32 { + return m.callCount.Load() +} + +func (m *MockTrackingBackupUsecase) GetCalledBackupIDs() []uuid.UUID { + ids := []uuid.UUID{} + for { + select { + case id := <-m.calledBackupIDs: + ids = append(ids, id) + default: + return ids + } + } +} diff --git a/backend/internal/features/backups/backups/backuping/logical/scheduler.go b/backend/internal/features/backups/backups/backuping/logical/scheduler.go new file mode 100644 index 0000000..a921fe1 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/scheduler.go @@ -0,0 +1,317 @@ +package backuping_logical + +import ( + "context" + "fmt" + "log/slog" + "sync/atomic" + "time" + + "github.com/google/uuid" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + task_cancellation "databasus-backend/internal/features/tasks/cancellation" +) + +const ( + schedulerTickerInterval = 15 * time.Second + schedulerHealthcheckThreshold = 5 * time.Minute +) + +type BackupsScheduler struct { + backupRepository *backups_core_logical.BackupRepository + backupConfigService *backups_config_logical.BackupConfigService + taskCancelManager *task_cancellation.TaskCancelManager + databaseService *databases.DatabaseService + + lastBackupTime time.Time + logger *slog.Logger + + backuper *Backuper + + backupCompletionListeners []backups_core_logical.BackupCompletionListener + + hasRun atomic.Bool + isReady atomic.Bool +} + +// IsRunning reports whether the scheduler has finished startup recovery and is +// ready to dispatch work. Tests use it to gate the start of work. +func (s *BackupsScheduler) IsRunning() bool { + return s.isReady.Load() +} + +func (s *BackupsScheduler) AddBackupCompletionListener( + listener backups_core_logical.BackupCompletionListener, +) { + s.backupCompletionListeners = append(s.backupCompletionListeners, listener) +} + +func (s *BackupsScheduler) Run(ctx context.Context) { + if s.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", s)) + } + + s.lastBackupTime = time.Now().UTC() + + if err := s.failBackupsInProgress(); err != nil { + s.logger.Error("Failed to fail backups in progress", "error", err) + panic(err) + } + + s.isReady.Store(true) + defer s.isReady.Store(false) + + if ctx.Err() != nil { + return + } + + ticker := time.NewTicker(schedulerTickerInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if err := s.runPendingBackups(); err != nil { + s.logger.Error("Failed to run pending backups", "error", err) + } + + s.lastBackupTime = time.Now().UTC() + } + } +} + +func (s *BackupsScheduler) IsSchedulerRunning() bool { + // if last backup time is more than 5 minutes ago, return false + return s.lastBackupTime.After(time.Now().UTC().Add(-schedulerHealthcheckThreshold)) +} + +func (s *BackupsScheduler) StartBackup(database *databases.Database, isCallNotifier bool) { + backupConfig, err := s.backupConfigService.GetBackupConfigByDbId(database.ID) + if err != nil { + s.logger.Error("Failed to get backup config by database ID", "error", err) + return + } + + if backupConfig.StorageID == nil { + s.logger.Error("Backup config storage ID is nil", "databaseId", database.ID) + return + } + + // Check for existing in-progress backups + inProgressBackups, err := s.backupRepository.FindByDatabaseIdAndStatus( + database.ID, + backups_core_logical.BackupStatusInProgress, + ) + if err != nil { + s.logger.Error( + "Failed to check for in-progress backups", + "databaseId", + database.ID, + "error", + err, + ) + return + } + + if len(inProgressBackups) > 0 { + s.logger.Warn( + "Backup already in progress for database, skipping new backup", + "databaseId", + database.ID, + "existingBackupId", + inProgressBackups[0].ID, + ) + return + } + + backupID := uuid.New() + timestamp := time.Now().UTC() + + backup := &backups_core_logical.LogicalBackup{ + ID: backupID, + DatabaseID: backupConfig.DatabaseID, + StorageID: *backupConfig.StorageID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 0, + CreatedAt: timestamp, + } + + backup.GenerateFilename(database.Name) + + if err := s.backupRepository.Save(backup); err != nil { + s.logger.Error( + "Failed to save backup", + "databaseId", + backupConfig.DatabaseID, + "error", + err, + ) + return + } + + go func() { + s.backuper.MakeBackup(backup.ID, isCallNotifier) + s.onBackupCompleted(backup.ID) + }() + + s.logger.Info( + "Successfully triggered scheduled backup", + "databaseId", + backupConfig.DatabaseID, + "backupId", + backup.ID, + ) +} + +// GetRemainedBackupTryCount returns the number of remaining backup tries for a given backup. +// If the backup is not failed or the backup config does not allow retries, it returns 0. +// If the backup is failed and the backup config allows retries, it returns the number of remaining tries. +// If the backup is failed and the backup config does not allow retries, it returns 0. +func (s *BackupsScheduler) GetRemainedBackupTryCount(lastBackup *backups_core_logical.LogicalBackup) int { + if lastBackup == nil { + return 0 + } + + if lastBackup.Status != backups_core_logical.BackupStatusFailed { + return 0 + } + + if lastBackup.IsSkipRetry { + return 0 + } + + backupConfig, err := s.backupConfigService.GetBackupConfigByDbId(lastBackup.DatabaseID) + if err != nil { + s.logger.Error("Failed to get backup config by database ID", "error", err) + return 0 + } + + if !backupConfig.IsRetryIfFailed { + return 0 + } + + maxFailedTriesCount := backupConfig.MaxFailedTriesCount + + lastBackups, err := s.backupRepository.FindByDatabaseIDWithLimit( + lastBackup.DatabaseID, + maxFailedTriesCount, + ) + if err != nil { + s.logger.Error("Failed to find last backups by database ID", "error", err) + return 0 + } + + lastFailedBackups := make([]*backups_core_logical.LogicalBackup, 0) + + for _, backup := range lastBackups { + if backup.Status == backups_core_logical.BackupStatusFailed { + lastFailedBackups = append(lastFailedBackups, backup) + } + } + + return maxFailedTriesCount - len(lastFailedBackups) +} + +func (s *BackupsScheduler) runPendingBackups() error { + enabledBackupConfigs, err := s.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + return err + } + + for _, backupConfig := range enabledBackupConfigs { + lastBackup, err := s.backupRepository.FindLastByDatabaseID(backupConfig.DatabaseID) + if err != nil { + s.logger.Error( + "Failed to get last backup for database", + "databaseId", + backupConfig.DatabaseID, + "error", + err, + ) + continue + } + + var lastBackupTime *time.Time + if lastBackup != nil { + lastBackupTime = &lastBackup.CreatedAt + } + + remainedBackupTryCount := s.GetRemainedBackupTryCount(lastBackup) + + if backupConfig.BackupInterval.ShouldTriggerBackup(time.Now().UTC(), lastBackupTime) || + remainedBackupTryCount > 0 { + s.logger.Info( + "Triggering scheduled backup", + "databaseId", + backupConfig.DatabaseID, + "intervalType", + backupConfig.BackupInterval.Type, + ) + + database, err := s.databaseService.GetDatabaseByID(backupConfig.DatabaseID) + if err != nil { + s.logger.Error("Failed to get database by ID", "error", err) + continue + } + + s.StartBackup(database, remainedBackupTryCount == 1) + continue + } + } + + return nil +} + +func (s *BackupsScheduler) failBackupsInProgress() error { + backupsInProgress, err := s.backupRepository.FindByStatus(backups_core_logical.BackupStatusInProgress) + if err != nil { + return err + } + + for _, backup := range backupsInProgress { + if err := s.taskCancelManager.CancelTask(backup.ID); err != nil { + s.logger.Error( + "Failed to cancel backup via task cancel manager", + "backupId", + backup.ID, + "error", + err, + ) + } + + backupConfig, err := s.backupConfigService.GetBackupConfigByDbId(backup.DatabaseID) + if err != nil { + s.logger.Error("Failed to get backup config by database ID", "error", err) + continue + } + + failMessage := "Backup failed due to application restart" + backup.FailMessage = &failMessage + backup.Status = backups_core_logical.BackupStatusFailed + backup.BackupSizeMb = 0 + + s.backuper.SendBackupNotification( + backupConfig, + backup, + backups_config_logical.NotificationBackupFailed, + &failMessage, + ) + + if err := s.backupRepository.Save(backup); err != nil { + return err + } + } + + return nil +} + +func (s *BackupsScheduler) onBackupCompleted(backupID uuid.UUID) { + for _, listener := range s.backupCompletionListeners { + go listener.OnBackupCompleted(backupID) + } +} diff --git a/backend/internal/features/backups/backups/backuping/logical/scheduler_test.go b/backend/internal/features/backups/backups/backuping/logical/scheduler_test.go new file mode 100644 index 0000000..8cc523a --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/scheduler_test.go @@ -0,0 +1,926 @@ +package backuping_logical + +import ( + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/intervals" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + cache_utils "databasus-backend/internal/util/cache" + "databasus-backend/internal/util/period" + "databasus-backend/internal/util/testing/containers" +) + +func Test_RunPendingBackups_ByDatabaseType_OnlySchedulesNonAgentManagedBackups(t *testing.T) { + type testCase struct { + name string + createDatabase func(t *testing.T, workspaceID uuid.UUID, storage *storages.Storage, notifier *notifiers.Notifier) *databases.Database + isBackupExpected bool + } + + testCases := []testCase{ + { + name: "PostgreSQL PG_DUMP - backup runs", + createDatabase: func(_ *testing.T, workspaceID uuid.UUID, storage *storages.Storage, notifier *notifiers.Notifier) *databases.Database { + return databases.CreateTestDatabase(workspaceID, storage, notifier) + }, + isBackupExpected: true, + }, + { + name: "MariaDB - backup runs", + createDatabase: func(t *testing.T, workspaceID uuid.UUID, _ *storages.Storage, notifier *notifiers.Notifier) *databases.Database { + endpoint := containers.StartMariadb(t, "mariadb:10.11") + return databases.CreateTestMariadbDatabase(endpoint.Host, endpoint.Port, workspaceID, notifier) + }, + isBackupExpected: true, + }, + { + name: "MongoDB - backup runs", + createDatabase: func(t *testing.T, workspaceID uuid.UUID, _ *storages.Storage, notifier *notifiers.Notifier) *databases.Database { + endpoint := containers.StartMongodb(t, "mongo:7.0") + return databases.CreateTestMongodbDatabase(endpoint.Host, endpoint.Port, workspaceID, notifier) + }, + isBackupExpected: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + cache_utils.ClearAllCache() + + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := tc.createDatabase(t, workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // add old backup (24h ago) + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: time.Now().UTC().Add(-24 * time.Hour), + }) + + GetBackupsScheduler().runPendingBackups() + + if tc.isBackupExpected { + WaitForBackupCompletion(t, database.ID, 1, 10*time.Second) + + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 2) + } else { + time.Sleep(100 * time.Millisecond) + + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1) + } + + time.Sleep(200 * time.Millisecond) + }) + } +} + +func Test_RunPendingBackups_WhenLastBackupWasYesterday_CreatesNewBackup(t *testing.T) { + cache_utils.ClearAllCache() + // setup data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for the database + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // add old backup + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusCompleted, + + CreatedAt: time.Now().UTC().Add(-24 * time.Hour), + }) + + GetBackupsScheduler().runPendingBackups() + + // Wait for backup to complete (runs in goroutine) + WaitForBackupCompletion(t, database.ID, 1, 10*time.Second) + + // assertions + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 2) + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenLastBackupWasRecentlyCompleted_SkipsBackup(t *testing.T) { + cache_utils.ClearAllCache() + // setup data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for the database + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusCompleted, + + CreatedAt: time.Now().UTC(), + }) + + GetBackupsScheduler().runPendingBackups() + + time.Sleep(100 * time.Millisecond) + + // assertions + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1) // Should still be 1 backup, no new backup created + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenLastBackupFailedAndRetriesDisabled_SkipsBackup(t *testing.T) { + cache_utils.ClearAllCache() + // setup data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for the database with retries disabled + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + backupConfig.IsRetryIfFailed = false + backupConfig.MaxFailedTriesCount = 0 + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + failMessage := "backup failed" + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusFailed, + FailMessage: &failMessage, + + CreatedAt: time.Now().UTC(), + }) + + GetBackupsScheduler().runPendingBackups() + + time.Sleep(100 * time.Millisecond) + + // assertions + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1) // Should still be 1 backup, no retry attempted + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenLastBackupFailedAndRetriesEnabled_CreatesNewBackup(t *testing.T) { + cache_utils.ClearAllCache() + // setup data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for the database with retries enabled + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + backupConfig.IsRetryIfFailed = true + backupConfig.MaxFailedTriesCount = 3 + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // add failed backup + failMessage := "backup failed" + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusFailed, + FailMessage: &failMessage, + + CreatedAt: time.Now().UTC().Add(-1 * time.Hour), + }) + + GetBackupsScheduler().runPendingBackups() + + // Wait for backup to complete (runs in goroutine) + WaitForBackupCompletion(t, database.ID, 1, 10*time.Second) + + // assertions + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 2) // Should have 2 backups, retry was attempted + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenFailedBackupsExceedMaxRetries_SkipsBackup(t *testing.T) { + cache_utils.ClearAllCache() + // setup data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // cleanup backups first + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for the database with retries enabled + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + backupConfig.IsRetryIfFailed = true + backupConfig.MaxFailedTriesCount = 3 + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + failMessage := "backup failed" + + for range 3 { + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusFailed, + FailMessage: &failMessage, + + CreatedAt: time.Now().UTC(), + }) + } + + GetBackupsScheduler().runPendingBackups() + + time.Sleep(100 * time.Millisecond) + + // assertions + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 3) // Should have 3 backups, not more than max + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenBackupsDisabled_SkipsBackup(t *testing.T) { + cache_utils.ClearAllCache() + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = false + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // add old backup that would trigger new backup if enabled + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusCompleted, + + CreatedAt: time.Now().UTC().Add(-24 * time.Hour), + }) + + GetBackupsScheduler().runPendingBackups() + + time.Sleep(100 * time.Millisecond) + + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1) + + // Wait for any cleanup operations to complete before defer cleanup runs + time.Sleep(200 * time.Millisecond) +} + +func Test_FailBackupsInProgress_WhenSchedulerStarts_CancelsBackupsAndUpdatesStatus(t *testing.T) { + cache_utils.ClearAllCache() + + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + + cache_utils.ClearAllCache() + }() + + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // Create two in-progress backups that should be failed on scheduler restart + backup1 := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 10.5, + CreatedAt: time.Now().UTC().Add(-30 * time.Minute), + } + err = backupRepository.Save(backup1) + assert.NoError(t, err) + + backup2 := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 5.2, + CreatedAt: time.Now().UTC().Add(-15 * time.Minute), + } + err = backupRepository.Save(backup2) + assert.NoError(t, err) + + // Create a completed backup to verify it's not affected by failBackupsInProgress + completedBackup := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 20.0, + CreatedAt: time.Now().UTC().Add(-1 * time.Hour), + } + err = backupRepository.Save(completedBackup) + assert.NoError(t, err) + + // Trigger the scheduler's failBackupsInProgress logic + // This should cancel in-progress backups and mark them as failed + err = GetBackupsScheduler().failBackupsInProgress() + assert.NoError(t, err) + + // Verify all backups exist and were processed correctly + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 3) + + var failedCount int + var completedCount int + for _, backup := range backups { + switch backup.Status { + case backups_core_logical.BackupStatusFailed: + failedCount++ + // Verify fail message indicates application restart + assert.NotNil(t, backup.FailMessage) + assert.Equal(t, "Backup failed due to application restart", *backup.FailMessage) + // Verify backup size was reset to 0 + assert.Equal(t, float64(0), backup.BackupSizeMb) + case backups_core_logical.BackupStatusCompleted: + completedCount++ + } + } + + // Verify correct number of backups in each state + assert.Equal(t, 2, failedCount) + assert.Equal(t, 1, completedCount) + + time.Sleep(200 * time.Millisecond) +} + +func Test_StartBackup_WhenBackupAlreadyInProgress_SkipsNewBackup(t *testing.T) { + cache_utils.ClearAllCache() + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // Create an in-progress backup manually + inProgressBackup := &backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 0, + CreatedAt: time.Now().UTC(), + } + err = backupRepository.Save(inProgressBackup) + assert.NoError(t, err) + + // Try to start a new backup - should be skipped + GetBackupsScheduler().StartBackup(database, false) + + time.Sleep(200 * time.Millisecond) + + // Verify only 1 backup exists (the original in-progress one) + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1) + assert.Equal(t, backups_core_logical.BackupStatusInProgress, backups[0].Status) + assert.Equal(t, inProgressBackup.ID, backups[0].ID) + + time.Sleep(200 * time.Millisecond) +} + +func Test_RunPendingBackups_WhenLastBackupFailedWithIsSkipRetry_SkipsBackupEvenWithRetriesEnabled( + t *testing.T, +) { + cache_utils.ClearAllCache() + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + backups, _ := backupRepository.FindByDatabaseID(database.ID) + for _, backup := range backups { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups with retries enabled and high retry count + backupConfig, err := backups_config_logical.GetBackupConfigService().GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig.IsBackupsEnabled = true + backupConfig.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig.RetentionTimePeriod = period.PeriodWeek + backupConfig.Storage = storage + backupConfig.StorageID = &storage.ID + backupConfig.IsRetryIfFailed = true + backupConfig.MaxFailedTriesCount = 5 + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + // Create a failed backup with IsSkipRetry set to true + failMessage := "backup failed due to size limit exceeded" + backupRepository.Save(&backups_core_logical.LogicalBackup{ + DatabaseID: database.ID, + StorageID: storage.ID, + + Status: backups_core_logical.BackupStatusFailed, + FailMessage: &failMessage, + IsSkipRetry: true, + + CreatedAt: time.Now().UTC(), + }) + + // Verify GetRemainedBackupTryCount returns 0 even though retries are enabled + lastBackup, err := backupRepository.FindLastByDatabaseID(database.ID) + assert.NoError(t, err) + assert.NotNil(t, lastBackup) + + remainedTries := GetBackupsScheduler().GetRemainedBackupTryCount(lastBackup) + assert.Equal(t, 0, remainedTries, "Should return 0 tries when IsSkipRetry is true") + + // Run the scheduler + GetBackupsScheduler().runPendingBackups() + + time.Sleep(100 * time.Millisecond) + + // Verify no new backup was created (still only 1 backup exists) + backups, err := backupRepository.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Len(t, backups, 1, "No retry should be attempted when IsSkipRetry is true") + + time.Sleep(200 * time.Millisecond) +} + +func Test_StartBackup_When2BackupsStartedForDifferentDatabases_BothUseCasesAreCalled(t *testing.T) { + cache_utils.ClearAllCache() + + // Create mock tracking use case + mockUseCase := NewMockTrackingBackupUsecase() + + // Create Backuper with mock use case and wire it into the scheduler so + // StartBackup dispatches to the mock in-process. + backuper := CreateTestBackuperWithUseCase(mockUseCase) + + // Create scheduler + scheduler := CreateTestSchedulerWithBackuper(backuper) + schedulerCancel := StartSchedulerForTest(t, scheduler) + defer schedulerCancel() + + // Setup test data + user := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + router := CreateTestRouter() + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + + // Create 2 separate databases + database1 := databases.CreateTestDatabase(workspace.ID, storage, notifier) + database2 := databases.CreateTestDatabase(workspace.ID, storage, notifier) + + defer func() { + // Cleanup backups for database1 + backups1, _ := backupRepository.FindByDatabaseID(database1.ID) + for _, backup := range backups1 { + backupRepository.DeleteByID(backup.ID) + } + + // Cleanup backups for database2 + backups2, _ := backupRepository.FindByDatabaseID(database2.ID) + for _, backup := range backups2 { + backupRepository.DeleteByID(backup.ID) + } + + databases.RemoveTestDatabase(database1) + databases.RemoveTestDatabase(database2) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + notifiers.RemoveTestNotifier(notifier) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + // Enable backups for database1 + backupConfig1, err := backups_config_logical.GetBackupConfigService(). + GetBackupConfigByDbId(database1.ID) + assert.NoError(t, err) + + timeOfDay := "04:00" + backupConfig1.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig1.IsBackupsEnabled = true + backupConfig1.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig1.RetentionTimePeriod = period.PeriodWeek + backupConfig1.Storage = storage + backupConfig1.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig1) + assert.NoError(t, err) + + // Enable backups for database2 + backupConfig2, err := backups_config_logical.GetBackupConfigService(). + GetBackupConfigByDbId(database2.ID) + assert.NoError(t, err) + + backupConfig2.BackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + backupConfig2.IsBackupsEnabled = true + backupConfig2.RetentionPolicyType = backups_config_logical.RetentionPolicyTypeTimePeriod + backupConfig2.RetentionTimePeriod = period.PeriodWeek + backupConfig2.Storage = storage + backupConfig2.StorageID = &storage.ID + + _, err = backups_config_logical.GetBackupConfigService().SaveBackupConfig(backupConfig2) + assert.NoError(t, err) + + // Start 2 backups simultaneously + t.Log("Starting backup for database1") + scheduler.StartBackup(database1, false) + + t.Log("Starting backup for database2") + scheduler.StartBackup(database2, false) + + // Wait up to 10 seconds for both backups to complete + t.Log("Waiting for both backups to complete...") + + success := assert.Eventually(t, func() bool { + callCount := mockUseCase.GetCallCount() + t.Logf("Current call count: %d/2", callCount) + return callCount == 2 + }, 10*time.Second, 200*time.Millisecond, "Both use cases should be called within 10 seconds") + + if !success { + t.Logf("Test failed: Only %d out of 2 use cases were called", mockUseCase.GetCallCount()) + } + + // Verify both backup IDs were received + calledBackupIDs := mockUseCase.GetCalledBackupIDs() + t.Logf("Called backup IDs: %v", calledBackupIDs) + assert.Len(t, calledBackupIDs, 2, "Both backup IDs should be tracked") + + // The mock records the call at use-case entry, before MakeBackup persists the + // terminal status, so wait for both dispatch goroutines to finish flipping the + // backups to COMPLETED before asserting on their status. + assert.Eventually(t, func() bool { + dbBackups1, _ := backupRepository.FindByDatabaseID(database1.ID) + dbBackups2, _ := backupRepository.FindByDatabaseID(database2.ID) + + return len(dbBackups1) == 1 && + dbBackups1[0].Status == backups_core_logical.BackupStatusCompleted && + len(dbBackups2) == 1 && + dbBackups2[0].Status == backups_core_logical.BackupStatusCompleted + }, 10*time.Second, 100*time.Millisecond, "both backups should reach COMPLETED") + + // Verify both backups exist in repository and are completed + backups1, err := backupRepository.FindByDatabaseID(database1.ID) + assert.NoError(t, err) + assert.Len(t, backups1, 1, "Database1 should have 1 backup") + if len(backups1) > 0 { + t.Logf("Database1 backup status: %s", backups1[0].Status) + } + + backups2, err := backupRepository.FindByDatabaseID(database2.ID) + assert.NoError(t, err) + assert.Len(t, backups2, 1, "Database2 should have 1 backup") + if len(backups2) > 0 { + t.Logf("Database2 backup status: %s", backups2[0].Status) + } + + // Verify both backups completed successfully + if len(backups1) > 0 { + assert.Equal(t, backups_core_logical.BackupStatusCompleted, backups1[0].Status, + "Database1 backup should be completed") + } + + if len(backups2) > 0 { + assert.Equal(t, backups_core_logical.BackupStatusCompleted, backups2[0].Status, + "Database2 backup should be completed") + } + + time.Sleep(200 * time.Millisecond) +} diff --git a/backend/internal/features/backups/backups/backuping/logical/setup_test.go b/backend/internal/features/backups/backups/backuping/logical/setup_test.go new file mode 100644 index 0000000..d174230 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/setup_test.go @@ -0,0 +1,20 @@ +package backuping_logical + +import ( + "os" + "testing" + + cache_utils "databasus-backend/internal/util/cache" +) + +// TestMain wipes Valkey at the start of this package's test binary so the +// scheduler/backuper subscriptions don't inherit leftover state from a +// previous run that died before its deferred Unsubscribe fired (e.g. a +// -failfast'd suite that exited mid-test). +func TestMain(m *testing.M) { + if err := cache_utils.ClearAllCache(); err != nil { + panic(err) + } + + os.Exit(m.Run()) +} diff --git a/backend/internal/features/backups/backups/backuping/logical/testing.go b/backend/internal/features/backups/backups/backuping/logical/testing.go new file mode 100644 index 0000000..74d4e55 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/logical/testing.go @@ -0,0 +1,245 @@ +package backuping_logical + +import ( + "context" + "sync/atomic" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + usecases_logical "databasus-backend/internal/features/backups/backups/usecases/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + workspaces_controllers "databasus-backend/internal/features/workspaces/controllers" + workspaces_services "databasus-backend/internal/features/workspaces/services" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + "databasus-backend/internal/storage" + "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/logger" +) + +func seedBackup( + t *testing.T, + label string, + backup *backups_core_logical.LogicalBackup, +) *backups_core_logical.LogicalBackup { + t.Helper() + + if err := storage.GetDb().Create(backup).Error; err != nil { + t.Fatalf("seed %s backup: %v", label, err) + } + + return backup +} + +func SeedTestBackup( + t *testing.T, + databaseID, storageID uuid.UUID, + sizeMb float64, +) *backups_core_logical.LogicalBackup { + t.Helper() + + return seedBackup(t, "completed", &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + FileName: "test-backup-" + uuid.New().String(), + DatabaseID: databaseID, + StorageID: storageID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: sizeMb, + BackupRawDbSizeMb: sizeMb, + CreatedAt: time.Now().UTC(), + }) +} + +// SeedInProgressTestBackup inserts an IN_PROGRESS backup row so MakeBackup can +// pick it up and drive it through to completion (mirrors backuper_test.go's +// manual setup, but reusable across packages). +func SeedInProgressTestBackup( + t *testing.T, + databaseID, storageID uuid.UUID, +) *backups_core_logical.LogicalBackup { + t.Helper() + + return seedBackup(t, "in-progress", &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: databaseID, + StorageID: storageID, + Status: backups_core_logical.BackupStatusInProgress, + CreatedAt: time.Now().UTC(), + }) +} + +func CreateTestRouter() *gin.Engine { + router := workspaces_testing.CreateTestRouter( + workspaces_controllers.GetWorkspaceController(), + workspaces_controllers.GetMembershipController(), + databases.GetDatabaseController(), + backups_config_logical.GetBackupConfigController(), + ) + + return router +} + +func CreateTestBackupCleaner() *BackupCleaner { + return &BackupCleaner{ + backupRepository, + storages.GetStorageService(), + backups_config_logical.GetBackupConfigService(), + encryption.GetFieldEncryptor(), + logger.GetLogger(), + []backups_core_logical.BackupRemoveListener{}, + atomic.Bool{}, + } +} + +func CreateTestBackuper() *Backuper { + return &Backuper{ + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + workspaces_services.GetWorkspaceService(), + backupRepository, + backups_config_logical.GetBackupConfigService(), + storages.GetStorageService(), + notifiers.GetNotifierService(), + taskCancelManager, + logger.GetLogger(), + usecases_logical.GetCreateBackupUsecase(), + } +} + +func CreateTestBackuperWithUseCase(useCase backups_core_logical.CreateBackupUsecase) *Backuper { + return &Backuper{ + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + workspaces_services.GetWorkspaceService(), + backupRepository, + backups_config_logical.GetBackupConfigService(), + storages.GetStorageService(), + notifiers.GetNotifierService(), + taskCancelManager, + logger.GetLogger(), + useCase, + } +} + +func CreateTestScheduler() *BackupsScheduler { + return CreateTestSchedulerWithBackuper(CreateTestBackuper()) +} + +// CreateTestSchedulerWithBackuper wires the scheduler to a specific backuper, so +// a test can inject a backuper with a mock use case and have StartBackup dispatch +// to it in-process. +func CreateTestSchedulerWithBackuper(backuper *Backuper) *BackupsScheduler { + return &BackupsScheduler{ + backupRepository, + backups_config_logical.GetBackupConfigService(), + taskCancelManager, + databases.GetDatabaseService(), + time.Now().UTC(), + logger.GetLogger(), + backuper, + []backups_core_logical.BackupCompletionListener{}, + atomic.Bool{}, + atomic.Bool{}, + } +} + +// WaitForBackupCompletion waits for a new backup to be created and completed (or failed) +// for the given database. It checks for backups with count greater than expectedInitialCount. +func WaitForBackupCompletion( + t *testing.T, + databaseID uuid.UUID, + expectedInitialCount int, + timeout time.Duration, +) { + deadline := time.Now().UTC().Add(timeout) + + for time.Now().UTC().Before(deadline) { + backups, err := backupRepository.FindByDatabaseID(databaseID) + if err != nil { + t.Logf("WaitForBackupCompletion: error finding backups: %v", err) + time.Sleep(50 * time.Millisecond) + continue + } + + t.Logf( + "WaitForBackupCompletion: found %d backups (expected > %d)", + len(backups), + expectedInitialCount, + ) + + if len(backups) > expectedInitialCount { + // Check if the newest backup has completed or failed + newestBackup := backups[0] + t.Logf("WaitForBackupCompletion: newest backup status: %s", newestBackup.Status) + + if newestBackup.Status == backups_core_logical.BackupStatusCompleted || + newestBackup.Status == backups_core_logical.BackupStatusFailed || + newestBackup.Status == backups_core_logical.BackupStatusCanceled { + t.Logf( + "WaitForBackupCompletion: backup finished with status %s", + newestBackup.Status, + ) + return + } + } + + time.Sleep(50 * time.Millisecond) + } + + t.Logf("WaitForBackupCompletion: timeout waiting for backup to complete") +} + +// StartSchedulerForTest starts the BackupsScheduler in a goroutine for testing. +// The scheduler subscribes to task completions and manages backup lifecycle. +// Returns a context cancel function that should be deferred to stop the scheduler. +// +// PubSubManager.Subscribe handshakes with Valkey before returning, so we +// don't need to sleep here waiting for the subscription to register. Poll +// the scheduler's hasRun flag instead to be sure Run() has entered its +// loop before the caller proceeds. +func StartSchedulerForTest(t *testing.T, scheduler *BackupsScheduler) context.CancelFunc { + ctx, cancel := context.WithCancel(context.Background()) + + done := make(chan struct{}) + + go func() { + scheduler.Run(ctx) + close(done) + }() + + deadline := time.Now().UTC().Add(5 * time.Second) + for time.Now().UTC().Before(deadline) { + if scheduler.IsRunning() { + t.Log("BackupsScheduler started") + + return func() { + cancel() + select { + case <-done: + t.Log("BackupsScheduler stopped gracefully") + case <-time.After(2 * time.Second): + t.Log("BackupsScheduler stop timeout") + } + } + } + + time.Sleep(25 * time.Millisecond) + } + + t.Fatal("BackupsScheduler failed to start within timeout") + + return nil +} + +// TriggerBackupCompletedForTest fires the DI scheduler's backup-completion +// listeners for a backup, standing in for the in-process completion hook that +// StartBackup's dispatch goroutine calls after MakeBackup returns. +func TriggerBackupCompletedForTest(backupID uuid.UUID) { + backupsScheduler.onBackupCompleted(backupID) +} diff --git a/backend/internal/features/backups/backups/backuping/physical/backuper.go b/backend/internal/features/backups/backups/backuping/physical/backuper.go new file mode 100644 index 0000000..236a67e --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/backuper.go @@ -0,0 +1,668 @@ +package backuping_physical + +import ( + "context" + "errors" + "fmt" + "log/slog" + "slices" + "time" + + "github.com/google/uuid" + "gorm.io/gorm" + "gorm.io/gorm/clause" + + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + encryption_secrets "databasus-backend/internal/features/encryption/secrets" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + workspaces_services "databasus-backend/internal/features/workspaces/services" + "databasus-backend/internal/storage" + util_encryption "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/walmath" +) + +// PhysicalBackuper drives a FULL or INCR through the postgresql executor. +// The scheduler invokes MakeBackup directly in a goroutine. +type PhysicalBackuper struct { + databaseService *databases.DatabaseService + fieldEncryptor util_encryption.FieldEncryptor + workspaceService *workspaces_services.WorkspaceService + fullRepo *physical_repositories.PhysicalFullBackupRepository + incrRepo *physical_repositories.PhysicalIncrementalBackupRepository + inFlightRepo *physical_repositories.PhysicalInFlightBackupRepository + historyRepo *physical_repositories.PhysicalWalHistoryRepository + backupConfigService *backups_config_physical.BackupConfigService + storageService *storages.StorageService + notificationSender NotificationSender + taskCancelManager *tasks_cancellation.TaskCancelManager + secretKeyService *encryption_secrets.SecretKeyService + logger *slog.Logger + fullExecutor FullBackupExecutor + incrExecutor IncrementalBackupExecutor +} + +func (b *PhysicalBackuper) MakeBackup(backupID uuid.UUID, isCallNotifier bool) { + logger := b.logger.With("backup_id", backupID) + + fullBackup, err := b.fullRepo.FindByID(backupID) + if err != nil { + logger.Error("failed to look up full backup row", "error", err) + return + } + + if fullBackup != nil { + b.runFullBackup(logger, fullBackup, isCallNotifier) + return + } + + incrBackup, err := b.incrRepo.FindByID(backupID) + if err != nil { + logger.Error("failed to look up incremental backup row", "error", err) + + return + } + + if incrBackup != nil { + b.runIncrementalBackup(logger, incrBackup, isCallNotifier) + return + } + + logger.Warn("backup not found in either typed table; ignoring assignment") +} + +func (b *PhysicalBackuper) runFullBackup( + logger *slog.Logger, + fullBackup *physical_models.PhysicalFullBackup, + isCallNotifier bool, +) { + backupCtx, ok := b.loadBackupContext(logger, fullBackup.DatabaseID) + if !ok { + b.finalizeFullAsError(fullBackup, physical_enums.PhysicalBackupErrorPgBasebackupFailed, + "failed to load backup context") + + return + } + + ctx, cancel := context.WithCancel(context.Background()) + b.taskCancelManager.RegisterTask(fullBackup.ID, cancel) + defer b.taskCancelManager.UnregisterTask(fullBackup.ID) + + fullBackupSpec := postgresql_executor.FullBackupSpec{ + CommonBackupSpec: postgresql_executor.CommonBackupSpec{ + SourceDB: backupCtx.Database.PostgresqlPhysical, + DatabaseName: backupCtx.Database.Name, + StorageID: backupCtx.Storage.ID, + Storage: backupCtx.Storage, + Encryption: backupCtx.Config.Encryption, + MasterKey: backupCtx.MasterKey, + FieldEncryptor: b.fieldEncryptor, + FullRepo: b.fullRepo, + HistoryRepo: b.historyRepo, + Logger: logger, + ProgressListener: func(completedMb float64, elapsedMs int64) { + if err := b.fullRepo.UpdateProgress(fullBackup.ID, completedMb, elapsedMs); err != nil { + logger.Error("failed to update full backup progress", "error", err) + } + }, + }, + Backup: fullBackup, + } + + backupResult, err := b.fullExecutor.Execute(ctx, fullBackupSpec) + if err != nil { + logger.Error("full executor returned error", "error", err) + + b.finalizeFullAsError(fullBackup, physical_enums.PhysicalBackupErrorPgBasebackupFailed, err.Error()) + + return + } + + if backupResult.Status != physical_enums.PhysicalBackupStatusCompleted { + logger.Warn("full executor returned non-COMPLETED result", + "status", backupResult.Status, + "reason", reasonOrEmpty(backupResult.ErrorReason), + "message", backupResult.ErrorMessage) + } + + if err := b.persistFullResult(fullBackup, backupResult); err != nil { + logger.Error("failed to persist full result", "error", err) + + return + } + + if isCallNotifier { + b.sendFullBackupNotification(backupCtx.Config, backupCtx.Database, fullBackup, backupResult) + } +} + +func (b *PhysicalBackuper) runIncrementalBackup( + logger *slog.Logger, + incrBackup *physical_models.PhysicalIncrementalBackup, + isCallNotifier bool, +) { + backupCtx, ok := b.loadBackupContext(logger, incrBackup.DatabaseID) + if !ok { + b.finalizeIncrAsError(incrBackup, physical_enums.PhysicalBackupErrorPgBasebackupFailed, + "failed to load backup context") + + return + } + + parentRef, err := b.resolveParentManifest(incrBackup) + if err != nil { + logger.Error("failed to resolve parent manifest", "error", err) + + b.finalizeIncrAsChainBroken(incrBackup, + physical_enums.PhysicalBackupErrorParentManifestMissing, err.Error()) + + return + } + + ctx, cancel := context.WithCancel(context.Background()) + b.taskCancelManager.RegisterTask(incrBackup.ID, cancel) + defer b.taskCancelManager.UnregisterTask(incrBackup.ID) + + incrBackupSpec := postgresql_executor.IncrementalBackupSpec{ + CommonBackupSpec: postgresql_executor.CommonBackupSpec{ + SourceDB: backupCtx.Database.PostgresqlPhysical, + DatabaseName: backupCtx.Database.Name, + StorageID: backupCtx.Storage.ID, + Storage: backupCtx.Storage, + Encryption: backupCtx.Config.Encryption, + MasterKey: backupCtx.MasterKey, + FieldEncryptor: b.fieldEncryptor, + FullRepo: b.fullRepo, + HistoryRepo: b.historyRepo, + Logger: logger, + ProgressListener: func(completedMb float64, elapsedMs int64) { + if err := b.incrRepo.UpdateProgress(incrBackup.ID, completedMb, elapsedMs); err != nil { + logger.Error("failed to update incremental backup progress", "error", err) + } + }, + }, + Backup: incrBackup, + ParentManifest: parentRef, + IncrRepo: b.incrRepo, + IncrementalCadence: backupCtx.Config.IncrementalBackupInterval.ApproxPeriod(), + } + + backupResult, err := b.incrExecutor.Execute(ctx, incrBackupSpec) + if err != nil { + logger.Error("incremental executor returned error", "error", err) + + b.finalizeIncrAsError(incrBackup, physical_enums.PhysicalBackupErrorPgBasebackupFailed, err.Error()) + + return + } + + if backupResult.Status != physical_enums.PhysicalBackupStatusCompleted { + logger.Warn("incremental executor returned non-COMPLETED result", + "status", backupResult.Status, + "reason", reasonOrEmpty(backupResult.ErrorReason), + "message", backupResult.ErrorMessage) + } + + if err := b.persistIncrResult(incrBackup, backupResult); err != nil { + logger.Error("failed to persist incremental result", "error", err) + + return + } + + if isCallNotifier { + b.sendIncrBackupNotification(backupCtx.Config, backupCtx.Database, incrBackup, backupResult) + } +} + +func (b *PhysicalBackuper) loadBackupContext( + logger *slog.Logger, + databaseID uuid.UUID, +) (*backupContext, bool) { + cfg, err := b.backupConfigService.GetBackupConfigByDbId(databaseID) + if err != nil { + logger.Error("failed to fetch physical backup config", "error", err) + + return nil, false + } + + if cfg.StorageID == nil { + logger.Error("physical backup config has no storage id") + + return nil, false + } + + db, err := b.databaseService.GetDatabaseByID(databaseID) + if err != nil { + logger.Error("failed to fetch database by id", "error", err) + + return nil, false + } + + if db.PostgresqlPhysical == nil { + logger.Error("database is not a physical postgres database") + + return nil, false + } + + storage, err := b.storageService.GetStorageByID(*cfg.StorageID) + if err != nil { + logger.Error("failed to fetch storage", "error", err) + + return nil, false + } + + masterKey := "" + if cfg.Encryption == backups_core_enums.BackupEncryptionEncrypted { + key, secretErr := b.secretKeyService.GetSecretKey() + if secretErr != nil { + logger.Error("failed to fetch master key", "error", secretErr) + + return nil, false + } + + masterKey = key + } + + return &backupContext{cfg, db, storage, masterKey}, true +} + +func (b *PhysicalBackuper) resolveParentManifest( + incrBackup *physical_models.PhysicalIncrementalBackup, +) (postgresql_executor.ParentManifestRef, error) { + if incrBackup.ParentIncrementalBackupID != nil { + parent, lookupErr := b.incrRepo.FindByID(*incrBackup.ParentIncrementalBackupID) + if lookupErr != nil { + return postgresql_executor.ParentManifestRef{}, fmt.Errorf("look up parent incr: %w", lookupErr) + } + + if parent == nil || parent.ManifestFileName == nil || parent.StopLSN == nil { + return postgresql_executor.ParentManifestRef{}, errors.New( + "parent incremental row missing or has no manifest_file_name / stop_lsn", + ) + } + + return postgresql_executor.ParentManifestRef{ + BackupID: parent.ID, + FileName: *parent.ManifestFileName, + Encryption: parent.Encryption, + Salt: derefString(parent.ManifestEncryptionSalt), + IV: derefString(parent.ManifestEncryptionIV), + StopLSN: *parent.StopLSN, + }, nil + } + + parent, lookupErr := b.fullRepo.FindByID(incrBackup.RootFullBackupID) + if lookupErr != nil { + return postgresql_executor.ParentManifestRef{}, fmt.Errorf("look up root full: %w", lookupErr) + } + + if parent == nil || parent.ManifestFileName == nil || parent.StopLSN == nil { + return postgresql_executor.ParentManifestRef{}, errors.New( + "root full row missing or has no manifest_file_name / stop_lsn") + } + + return postgresql_executor.ParentManifestRef{ + BackupID: parent.ID, + FileName: *parent.ManifestFileName, + Encryption: parent.Encryption, + Salt: derefString(parent.ManifestEncryptionSalt), + IV: derefString(parent.ManifestEncryptionIV), + StopLSN: *parent.StopLSN, + }, nil +} + +func (b *PhysicalBackuper) persistFullResult( + fullBackup *physical_models.PhysicalFullBackup, + backupResult postgresql_executor.PhysicalBackupResult, +) error { + fullBackup.Status = backupResult.Status + fullBackup.ErrorReason = backupResult.ErrorReason + fullBackup.FailMessage = nilOrPtr(backupResult.ErrorMessage) + + if backupResult.Status == physical_enums.PhysicalBackupStatusCompleted { + fullBackup.TimelineID = backupResult.TimelineID + fullBackup.StartLSN = lsnPtr(backupResult.StartLSN) + fullBackup.StopLSN = lsnPtr(backupResult.StopLSN) + fullBackup.BackupSizeMb = &backupResult.BackupSizeMb + fullBackup.BackupDurationMs = &backupResult.BackupDurationMs + fullBackup.Encryption = backupResult.EncryptionAlgo + fullBackup.EncryptionSalt = nilOrPtr(backupResult.EncryptionSalt) + fullBackup.EncryptionIV = nilOrPtr(backupResult.EncryptionIV) + + fullBackup.Compression = backupResult.Compression + fullBackup.ManifestFileName = nilOrPtr(backupResult.ManifestFileName) + fullBackup.ManifestEncryptionSalt = nilOrPtr(backupResult.ManifestEncryptionSalt) + fullBackup.ManifestEncryptionIV = nilOrPtr(backupResult.ManifestEncryptionIV) + + completed := backupResult.CompletedAt + if completed.IsZero() { + completed = time.Now().UTC() + } + + fullBackup.CompletedAt = &completed + } + + return b.saveTerminalResultIfInProgress( + fullBackup.DatabaseID, + fullBackup.ID, + func(tx *gorm.DB) (physical_enums.PhysicalBackupStatus, error) { + var current physical_models.PhysicalFullBackup + + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Select("status"). + Where("id = ?", fullBackup.ID). + First(¤t).Error; err != nil { + return "", err + } + + return current.Status, nil + }, + func(tx *gorm.DB) error { return tx.Save(fullBackup).Error }, + ) +} + +func (b *PhysicalBackuper) persistIncrResult( + incrBackup *physical_models.PhysicalIncrementalBackup, + backupResult postgresql_executor.PhysicalBackupResult, +) error { + incrBackup.Status = backupResult.Status + incrBackup.ErrorReason = backupResult.ErrorReason + incrBackup.FailMessage = nilOrPtr(backupResult.ErrorMessage) + + if backupResult.Status == physical_enums.PhysicalBackupStatusCompleted { + incrBackup.TimelineID = backupResult.TimelineID + incrBackup.StartLSN = lsnPtr(backupResult.StartLSN) + incrBackup.StopLSN = lsnPtr(backupResult.StopLSN) + incrBackup.BackupSizeMb = &backupResult.BackupSizeMb + incrBackup.BackupDurationMs = &backupResult.BackupDurationMs + incrBackup.Encryption = backupResult.EncryptionAlgo + incrBackup.EncryptionSalt = nilOrPtr(backupResult.EncryptionSalt) + incrBackup.EncryptionIV = nilOrPtr(backupResult.EncryptionIV) + + incrBackup.Compression = backupResult.Compression + incrBackup.ManifestFileName = nilOrPtr(backupResult.ManifestFileName) + incrBackup.ManifestEncryptionSalt = nilOrPtr(backupResult.ManifestEncryptionSalt) + incrBackup.ManifestEncryptionIV = nilOrPtr(backupResult.ManifestEncryptionIV) + + completed := backupResult.CompletedAt + if completed.IsZero() { + completed = time.Now().UTC() + } + + incrBackup.CompletedAt = &completed + } + + return b.saveTerminalResultIfInProgress( + incrBackup.DatabaseID, + incrBackup.ID, + func(tx *gorm.DB) (physical_enums.PhysicalBackupStatus, error) { + var current physical_models.PhysicalIncrementalBackup + + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}). + Select("status"). + Where("id = ?", incrBackup.ID). + First(¤t).Error; err != nil { + return "", err + } + + return current.Status, nil + }, + func(tx *gorm.DB) error { return tx.Save(incrBackup).Error }, + ) +} + +// saveTerminalResultIfInProgress writes the mutated backup row and releases its +// in-flight claim in one transaction, but only while the row is still +// IN_PROGRESS. A backuper can return after a restart-recovery sweep already +// moved the row to a terminal status — and possibly handed the database to a +// fresh backup. Persisting then would resurrect a superseded backup, so the +// guarded read (locked FOR UPDATE to serialize against the sweep's conditional +// update) skips the write instead. The claim delete is +// scoped to backupID so it can never remove the newer backup's claim. +func (b *PhysicalBackuper) saveTerminalResultIfInProgress( + databaseID, backupID uuid.UUID, + loadStatus func(tx *gorm.DB) (physical_enums.PhysicalBackupStatus, error), + save func(tx *gorm.DB) error, +) error { + superseded := false + + err := storage.GetDb().Transaction(func(tx *gorm.DB) error { + status, err := loadStatus(tx) + if err != nil { + return err + } + + if status != physical_enums.PhysicalBackupStatusInProgress { + superseded = true + + return nil + } + + if err := save(tx); err != nil { + return err + } + + return tx.Delete( + &physical_models.PhysicalInFlightBackup{}, + "database_id = ? AND backup_id = ?", + databaseID, + backupID, + ).Error + }) + if err != nil { + return err + } + + if superseded { + b.logger.Warn("backup row no longer in progress; skipping terminal persist", "backup_id", backupID) + } + + return nil +} + +func (b *PhysicalBackuper) finalizeFullAsError( + fullBackup *physical_models.PhysicalFullBackup, + reason physical_enums.PhysicalBackupErrorReason, + message string, +) { + r := reason + + fullBackup.Status = physical_enums.PhysicalBackupStatusError + fullBackup.ErrorReason = &r + fullBackup.FailMessage = nilOrPtr(message) + + if err := b.fullRepo.Save(fullBackup); err != nil { + b.logger.Error("failed to flip full row to ERROR", "backup_id", fullBackup.ID, "error", err) + } + + _ = b.inFlightRepo.ReleaseOwned(fullBackup.DatabaseID, fullBackup.ID) +} + +func (b *PhysicalBackuper) finalizeIncrAsError( + incrBackup *physical_models.PhysicalIncrementalBackup, + reason physical_enums.PhysicalBackupErrorReason, + message string, +) { + b.finalizeIncrWithStatus(incrBackup, physical_enums.PhysicalBackupStatusError, reason, message) +} + +// finalizeIncrAsChainBroken closes the chain instead of marking a transient +// failure. Use it only for irreversible conditions (a missing parent manifest, +// expired summaries) where retrying the same INCR is futile: CHAIN_BROKEN forces +// the next scheduler tick to open a fresh FULL, whereas ERROR would keep the +// chain extendable and retry the doomed INCR forever. +func (b *PhysicalBackuper) finalizeIncrAsChainBroken( + incrBackup *physical_models.PhysicalIncrementalBackup, + reason physical_enums.PhysicalBackupErrorReason, + message string, +) { + b.finalizeIncrWithStatus(incrBackup, physical_enums.PhysicalBackupStatusChainBroken, reason, message) +} + +func (b *PhysicalBackuper) finalizeIncrWithStatus( + incrBackup *physical_models.PhysicalIncrementalBackup, + status physical_enums.PhysicalBackupStatus, + reason physical_enums.PhysicalBackupErrorReason, + message string, +) { + r := reason + + incrBackup.Status = status + incrBackup.ErrorReason = &r + incrBackup.FailMessage = nilOrPtr(message) + + if err := b.incrRepo.Save(incrBackup); err != nil { + b.logger.Error("failed to flip incr row to terminal status", + "status", status, "backup_id", incrBackup.ID, "error", err) + } + + _ = b.inFlightRepo.ReleaseOwned(incrBackup.DatabaseID, incrBackup.ID) +} + +func (b *PhysicalBackuper) sendFullBackupNotification( + cfg *backups_config_physical.PhysicalBackupConfig, + db *databases.Database, + fullBackup *physical_models.PhysicalFullBackup, + backupResult postgresql_executor.PhysicalBackupResult, +) { + notificationType, title, message := classifyFullBackupNotification(db, fullBackup, backupResult, b.workspaceService) + if notificationType == "" { + return + } + + if !slices.Contains(cfg.SendNotificationsOn, notificationType) { + return + } + + for _, notifier := range db.Notifiers { + b.notificationSender.SendNotification(¬ifier, title, message) + } +} + +func (b *PhysicalBackuper) sendIncrBackupNotification( + cfg *backups_config_physical.PhysicalBackupConfig, + db *databases.Database, + incrBackup *physical_models.PhysicalIncrementalBackup, + backupResult postgresql_executor.PhysicalBackupResult, +) { + notificationType, title, message := classifyIncrBackupNotification(db, incrBackup, backupResult, b.workspaceService) + if notificationType == "" { + return + } + + if !slices.Contains(cfg.SendNotificationsOn, notificationType) { + return + } + + for _, notifier := range db.Notifiers { + b.notificationSender.SendNotification(¬ifier, title, message) + } +} + +func classifyFullBackupNotification( + db *databases.Database, + fullBackup *physical_models.PhysicalFullBackup, + backupResult postgresql_executor.PhysicalBackupResult, + workspaceService *workspaces_services.WorkspaceService, +) (backups_config_physical.BackupNotificationType, string, string) { + workspaceName := "unknown" + if db.WorkspaceID != nil { + if ws, err := workspaceService.GetWorkspaceByID(*db.WorkspaceID); err == nil { + workspaceName = ws.Name + } + } + + switch fullBackup.Status { + case physical_enums.PhysicalBackupStatusCompleted: + return backups_config_physical.NotificationBackupSuccess, + fmt.Sprintf("Physical FULL completed for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s size=%.2f MB duration=%dms", + fullBackup.ID, backupResult.BackupSizeMb, backupResult.BackupDurationMs) + + case physical_enums.PhysicalBackupStatusError: + return backups_config_physical.NotificationBackupFailed, + fmt.Sprintf("Physical FULL failed for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s reason=%s message=%s", + fullBackup.ID, reasonOrEmpty(fullBackup.ErrorReason), backupResult.ErrorMessage) + + case physical_enums.PhysicalBackupStatusChainBroken: + return backups_config_physical.NotificationChainBroken, + fmt.Sprintf("Physical FULL chain-broken for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s reason=%s message=%s", + fullBackup.ID, reasonOrEmpty(fullBackup.ErrorReason), backupResult.ErrorMessage) + } + + return "", "", "" +} + +func classifyIncrBackupNotification( + db *databases.Database, + incrBackup *physical_models.PhysicalIncrementalBackup, + backupResult postgresql_executor.PhysicalBackupResult, + workspaceService *workspaces_services.WorkspaceService, +) (backups_config_physical.BackupNotificationType, string, string) { + workspaceName := "unknown" + if db.WorkspaceID != nil { + if ws, err := workspaceService.GetWorkspaceByID(*db.WorkspaceID); err == nil { + workspaceName = ws.Name + } + } + + switch incrBackup.Status { + case physical_enums.PhysicalBackupStatusCompleted: + return backups_config_physical.NotificationBackupSuccess, + fmt.Sprintf("Physical INCR completed for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s size=%.2f MB duration=%dms", + incrBackup.ID, backupResult.BackupSizeMb, backupResult.BackupDurationMs) + + case physical_enums.PhysicalBackupStatusError: + return backups_config_physical.NotificationBackupFailed, + fmt.Sprintf("Physical INCR failed for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s reason=%s message=%s", + incrBackup.ID, reasonOrEmpty(incrBackup.ErrorReason), backupResult.ErrorMessage) + + case physical_enums.PhysicalBackupStatusChainBroken: + return backups_config_physical.NotificationChainBroken, + fmt.Sprintf("Physical INCR chain-broken for %q (workspace %q)", db.Name, workspaceName), + fmt.Sprintf("backup_id=%s reason=%s message=%s", + incrBackup.ID, reasonOrEmpty(incrBackup.ErrorReason), backupResult.ErrorMessage) + } + + return "", "", "" +} + +func reasonOrEmpty(r *physical_enums.PhysicalBackupErrorReason) string { + if r == nil { + return "" + } + + return string(*r) +} + +func derefString(s *string) string { + if s == nil { + return "" + } + + return *s +} + +func nilOrPtr(s string) *string { + if s == "" { + return nil + } + + return &s +} + +func lsnPtr(v walmath.LSN) *walmath.LSN { + out := v + + return &out +} diff --git a/backend/internal/features/backups/backups/backuping/physical/backuper_test.go b/backend/internal/features/backups/backups/backuping/physical/backuper_test.go new file mode 100644 index 0000000..9b0fc89 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/backuper_test.go @@ -0,0 +1,763 @@ +package backuping_physical + +import ( + "errors" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_testing "databasus-backend/internal/features/backups/backups/core/physical/testing" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + workspaces_services "databasus-backend/internal/features/workspaces/services" + "databasus-backend/internal/storage" + "databasus-backend/internal/util/logger" + "databasus-backend/internal/util/walmath" +) + +func Test_ResolveParentManifest_WhenChainRootFull_ReturnsRootRef(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, walmath.LSN(0x1000000), walmath.LSN(0x2000000)) + rootFull.ManifestFileName = new("root.manifest") + rootFull.Encryption = backups_core_enums.BackupEncryptionEncrypted + rootFull.ManifestEncryptionSalt = new("root-salt") + rootFull.ManifestEncryptionIV = new("root-iv") + physical_testing.CreateTestFullBackup(t, rootFull) + + incrBackup := &physical_models.PhysicalIncrementalBackup{ + RootFullBackupID: rootFull.ID, + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + } + + parentRef, err := backuper.resolveParentManifest(incrBackup) + require.NoError(t, err) + assert.Equal(t, rootFull.ID, parentRef.BackupID) + assert.Equal(t, "root.manifest", parentRef.FileName) + assert.Equal(t, backups_core_enums.BackupEncryptionEncrypted, parentRef.Encryption) + assert.Equal(t, "root-salt", parentRef.Salt) + assert.Equal(t, "root-iv", parentRef.IV) +} + +func Test_ResolveParentManifest_WhenParentIncrementalPresent_ReturnsParentRef(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, walmath.LSN(0x1000000), walmath.LSN(0x2000000)) + rootFull.ManifestFileName = new("root.manifest") + physical_testing.CreateTestFullBackup(t, rootFull) + + parentIncr := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.DB.ID, prereqs.Storage.ID, rootFull.ID, nil, 1, + walmath.LSN(0x2000000), walmath.LSN(0x3000000)) + parentIncr.ManifestFileName = new("parent.manifest") + parentIncr.Encryption = backups_core_enums.BackupEncryptionNone + physical_testing.CreateTestIncrementalBackup(t, parentIncr) + + incrBackup := &physical_models.PhysicalIncrementalBackup{ + RootFullBackupID: rootFull.ID, + ParentIncrementalBackupID: new(parentIncr.ID), + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + } + + parentRef, err := backuper.resolveParentManifest(incrBackup) + require.NoError(t, err) + assert.Equal(t, parentIncr.ID, parentRef.BackupID) + assert.Equal(t, "parent.manifest", parentRef.FileName) +} + +func Test_ResolveParentManifest_WhenParentMissingManifestFileName_ReturnsError(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, walmath.LSN(0x1000000), walmath.LSN(0x2000000)) + rootFull.ManifestFileName = new("root.manifest") + physical_testing.CreateTestFullBackup(t, rootFull) + + // Parent incremental with no manifest_file_name — the chain reference is unusable. + parentIncr := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.DB.ID, prereqs.Storage.ID, rootFull.ID, nil, 1, + walmath.LSN(0x2000000), walmath.LSN(0x3000000)) + physical_testing.CreateTestIncrementalBackup(t, parentIncr) + + incrBackup := &physical_models.PhysicalIncrementalBackup{ + RootFullBackupID: rootFull.ID, + ParentIncrementalBackupID: new(parentIncr.ID), + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + } + + _, err := backuper.resolveParentManifest(incrBackup) + require.Error(t, err) +} + +func Test_ResolveParentManifest_WhenRootFullMissing_ReturnsError(t *testing.T) { + backuper := CreateTestPhysicalBackuper(nil) + + incrBackup := &physical_models.PhysicalIncrementalBackup{ + RootFullBackupID: uuid.New(), + } + + _, err := backuper.resolveParentManifest(incrBackup) + require.Error(t, err) +} + +func Test_PersistFullResult_WhenCompleted_CopiesCompressionManifestFieldsAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + fullBackup := seedInProgressFull(t, prereqs) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, fullBackup.ID) + + result := postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusCompleted, + TimelineID: 7, + StartLSN: walmath.LSN(0x100), + StopLSN: walmath.LSN(0x200), + BackupSizeMb: 42.5, + BackupDurationMs: 1234, + EncryptionAlgo: backups_core_enums.BackupEncryptionNone, + Compression: physical_enums.PhysicalBackupCompressionZstd, + ManifestFileName: "artifact.manifest", + ManifestEncryptionSalt: "manifest-salt", + ManifestEncryptionIV: "manifest-iv", + CompletedAt: time.Now().UTC(), + } + + require.NoError(t, backuper.persistFullResult(fullBackup, result)) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusCompleted, persisted.Status) + assert.Equal(t, physical_enums.PhysicalBackupCompressionZstd, persisted.Compression) + require.NotNil(t, persisted.ManifestFileName) + assert.Equal(t, "artifact.manifest", *persisted.ManifestFileName) + require.NotNil(t, persisted.ManifestEncryptionSalt) + assert.Equal(t, "manifest-salt", *persisted.ManifestEncryptionSalt) + require.NotNil(t, persisted.ManifestEncryptionIV) + assert.Equal(t, "manifest-iv", *persisted.ManifestEncryptionIV) + require.NotNil(t, persisted.CompletedAt) + require.NotNil(t, persisted.BackupSizeMb) + assert.InDelta(t, 42.5, *persisted.BackupSizeMb, 0.001) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_PersistFullResult_WhenErrorStatus_SkipsCompletionFieldsAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + fullBackup := seedInProgressFull(t, prereqs) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, fullBackup.ID) + + result := postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusError, + ErrorReason: new(physical_enums.PhysicalBackupErrorPgBasebackupFailed), + ManifestFileName: "should-not-be-copied", + } + + require.NoError(t, backuper.persistFullResult(fullBackup, result)) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorPgBasebackupFailed, *persisted.ErrorReason) + assert.Nil(t, persisted.ManifestFileName, "completion-only fields must not be copied on a non-COMPLETED result") + assert.Nil(t, persisted.CompletedAt) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_PersistIncrementalResult_WhenCompleted_CopiesFieldsAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := seedCompletedRootFull(t, prereqs) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeIncremental, incrBackup.ID) + + result := postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusCompleted, + TimelineID: 3, + StartLSN: walmath.LSN(0x300), + StopLSN: walmath.LSN(0x400), + BackupSizeMb: 7.25, + BackupDurationMs: 555, + EncryptionAlgo: backups_core_enums.BackupEncryptionNone, + Compression: physical_enums.PhysicalBackupCompressionGzip, + ManifestFileName: "incr.manifest", + CompletedAt: time.Now().UTC(), + } + + require.NoError(t, backuper.persistIncrResult(incrBackup, result)) + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusCompleted, persisted.Status) + assert.Equal(t, physical_enums.PhysicalBackupCompressionGzip, persisted.Compression) + require.NotNil(t, persisted.ManifestFileName) + assert.Equal(t, "incr.manifest", *persisted.ManifestFileName) + require.NotNil(t, persisted.CompletedAt) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_FinalizeFullAsError_SetsErrorStatusReasonAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + fullBackup := seedInProgressFull(t, prereqs) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, fullBackup.ID) + + backuper.finalizeFullAsError( + fullBackup, + physical_enums.PhysicalBackupErrorPgBasebackupFailed, + "failed to load backup context", + ) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorPgBasebackupFailed, *persisted.ErrorReason) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_FinalizeIncrementalAsError_SetsErrorStatusReasonAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := seedCompletedRootFull(t, prereqs) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeIncremental, incrBackup.ID) + + backuper.finalizeIncrAsError( + incrBackup, + physical_enums.PhysicalBackupErrorPgBasebackupFailed, + "transient executor failure", + ) + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorPgBasebackupFailed, *persisted.ErrorReason) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_FinalizeIncrementalAsChainBroken_SetsChainBrokenStatusReasonAndReleasesInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + rootFull := seedCompletedRootFull(t, prereqs) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeIncremental, incrBackup.ID) + + backuper.finalizeIncrAsChainBroken( + incrBackup, + physical_enums.PhysicalBackupErrorParentManifestMissing, + "parent manifest missing", + ) + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + require.NotNil(t, persisted) + + assert.Equal(t, physical_enums.PhysicalBackupStatusChainBroken, persisted.Status) + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorParentManifestMissing, *persisted.ErrorReason) + + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_ClassifyFullBackupNotification_WhenCompleted_ReturnsSuccessNotification(t *testing.T) { + database := &databases.Database{Name: "mydb"} + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusCompleted, + } + result := postgresql_executor.PhysicalBackupResult{BackupSizeMb: 12.5, BackupDurationMs: 999} + + notificationType, title, message := classifyFullBackupNotification( + database, fullBackup, result, workspaces_services.GetWorkspaceService()) + + assert.Equal(t, backups_config_physical.NotificationBackupSuccess, notificationType) + assert.Contains(t, title, "completed") + assert.Contains(t, title, "mydb") + assert.Contains(t, message, fullBackup.ID.String()) +} + +func Test_ClassifyFullBackupNotification_WhenErrorStatus_ReturnsFailedNotification(t *testing.T) { + database := &databases.Database{Name: "mydb"} + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusError, + ErrorReason: new(physical_enums.PhysicalBackupErrorPgBasebackupFailed), + } + result := postgresql_executor.PhysicalBackupResult{ErrorMessage: "disk full"} + + notificationType, title, _ := classifyFullBackupNotification( + database, fullBackup, result, workspaces_services.GetWorkspaceService()) + + assert.Equal(t, backups_config_physical.NotificationBackupFailed, notificationType) + assert.Contains(t, title, "failed") +} + +func Test_ClassifyFullBackupNotification_WhenChainBroken_ReturnsChainBrokenNotification(t *testing.T) { + database := &databases.Database{Name: "mydb"} + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusChainBroken, + } + + notificationType, title, _ := classifyFullBackupNotification( + database, fullBackup, postgresql_executor.PhysicalBackupResult{}, workspaces_services.GetWorkspaceService()) + + assert.Equal(t, backups_config_physical.NotificationChainBroken, notificationType) + assert.Contains(t, title, "chain-broken") +} + +func Test_ClassifyFullBackupNotification_WhenInProgress_ReturnsEmpty(t *testing.T) { + database := &databases.Database{Name: "mydb"} + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusInProgress, + } + + notificationType, _, _ := classifyFullBackupNotification( + database, fullBackup, postgresql_executor.PhysicalBackupResult{}, workspaces_services.GetWorkspaceService()) + + assert.Empty(t, notificationType) +} + +func Test_SendFullBackupNotification_WhenTypeNotInSendOn_DoesNotSend(t *testing.T) { + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + + // Completed maps to BACKUP_SUCCESS, which is NOT in SendNotificationsOn. + cfg := &backups_config_physical.PhysicalBackupConfig{ + SendNotificationsOn: []backups_config_physical.BackupNotificationType{ + backups_config_physical.NotificationChainBroken, + }, + } + database := &databases.Database{ + Name: "mydb", + Notifiers: []notifiers.Notifier{{ID: uuid.New(), Name: "n1"}}, + } + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusCompleted, + } + + backuper.sendFullBackupNotification(cfg, database, fullBackup, postgresql_executor.PhysicalBackupResult{}) + + assert.Empty(t, sender.sentNotifications) +} + +func Test_SendFullBackupNotification_WhenEnabled_FansOutToAllNotifiers(t *testing.T) { + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + + cfg := &backups_config_physical.PhysicalBackupConfig{ + SendNotificationsOn: []backups_config_physical.BackupNotificationType{ + backups_config_physical.NotificationBackupSuccess, + }, + } + firstNotifierID, secondNotifierID := uuid.New(), uuid.New() + database := &databases.Database{ + Name: "mydb", + Notifiers: []notifiers.Notifier{ + {ID: firstNotifierID, Name: "first"}, + {ID: secondNotifierID, Name: "second"}, + }, + } + fullBackup := &physical_models.PhysicalFullBackup{ + ID: uuid.New(), + Status: physical_enums.PhysicalBackupStatusCompleted, + } + + backuper.sendFullBackupNotification(cfg, database, fullBackup, postgresql_executor.PhysicalBackupResult{}) + + require.Len(t, sender.sentNotifications, 2) + notifiedIDs := []uuid.UUID{ + sender.sentNotifications[0].Notifier.ID, + sender.sentNotifications[1].Notifier.ID, + } + assert.Contains(t, notifiedIDs, firstNotifierID) + assert.Contains(t, notifiedIDs, secondNotifierID) + assert.Contains(t, sender.sentNotifications[0].Title, "completed") +} + +func Test_LoadBackupContext_WhenNotEncrypted_LeavesMasterKeyEmpty(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + backupCtx, ok := backuper.loadBackupContext(logger.GetLogger(), prereqs.DB.ID) + require.True(t, ok) + require.NotNil(t, backupCtx) + + assert.Empty(t, backupCtx.MasterKey) + assert.NotNil(t, backupCtx.Config) + assert.NotNil(t, backupCtx.Database) + assert.NotNil(t, backupCtx.Storage) +} + +func Test_LoadBackupContext_WhenEncrypted_PopulatesMasterKey(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + prereqs.Config.Encryption = backups_core_enums.BackupEncryptionEncrypted + _, err := backups_config_physical.GetBackupConfigService().SaveBackupConfig(prereqs.Config) + require.NoError(t, err) + + backupCtx, ok := backuper.loadBackupContext(logger.GetLogger(), prereqs.DB.ID) + require.True(t, ok) + require.NotNil(t, backupCtx) + + assert.NotEmpty(t, backupCtx.MasterKey) +} + +func Test_LoadBackupContext_WhenConfigHasNoStorageID_ReturnsNotOk(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + + prereqs.Config.IsBackupsEnabled = false + prereqs.Config.StorageID = nil + prereqs.Config.Storage = nil + _, err := backups_config_physical.GetBackupConfigService().SaveBackupConfig(prereqs.Config) + require.NoError(t, err) + + _, ok := backuper.loadBackupContext(logger.GetLogger(), prereqs.DB.ID) + assert.False(t, ok) +} + +func Test_MakeBackup_WhenRowAbsentInBothTables_InvokesNoExecutor(t *testing.T) { + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, incrExecutor := installFakeExecutors(backuper) + + backuper.MakeBackup(uuid.New(), false) + + assert.Equal(t, 0, fullExecutor.callCount) + assert.Equal(t, 0, incrExecutor.callCount) +} + +func Test_MakeBackup_WhenFullRowExists_InvokesFullExecutorOnly(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, incrExecutor := installFakeExecutors(backuper) + fullExecutor.result = erroredResult() + + fullBackup := seedInProgressFull(t, prereqs) + + backuper.MakeBackup(fullBackup.ID, false) + + assert.Equal(t, 1, fullExecutor.callCount) + assert.Equal(t, 0, incrExecutor.callCount) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) +} + +func Test_MakeBackup_WhenIncrementalRowExists_InvokesIncrementalExecutorOnly(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, incrExecutor := installFakeExecutors(backuper) + incrExecutor.result = erroredResult() + + rootFull := seedCompletedRootFull(t, prereqs) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + + backuper.MakeBackup(incrBackup.ID, false) + + assert.Equal(t, 0, fullExecutor.callCount) + assert.Equal(t, 1, incrExecutor.callCount) + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) +} + +func Test_RunFullBackup_WhenExecutorReturnsGoError_FlipsToErrorAndSkipsNotification(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, _ := installFakeExecutors(backuper) + fullExecutor.err = errors.New("executor blew up") + + fullBackup := seedInProgressFull(t, prereqs) + + backuper.MakeBackup(fullBackup.ID, true) + + assert.Equal(t, 1, fullExecutor.callCount) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorPgBasebackupFailed, *persisted.ErrorReason) + + assert.Empty(t, sender.sentNotifications, "a Go error returns before the notification block") +} + +func Test_RunFullBackup_WhenExecutorResultErrorStatus_PersistsErrorAndSendsFailedNotification(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, _ := installFakeExecutors(backuper) + fullExecutor.result = postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusError, + ErrorReason: new(physical_enums.PhysicalBackupErrorPgBasebackupFailed), + ErrorMessage: "disk full", + } + + fullBackup := seedInProgressFull(t, prereqs) + + backuper.MakeBackup(fullBackup.ID, true) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + + require.Len(t, sender.sentNotifications, 1) + assert.Contains(t, sender.sentNotifications[0].Title, "failed") +} + +func Test_RunFullBackup_WhenExecutorResultChainBroken_PersistsChainBrokenAndSendsChainBrokenNotification(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + fullExecutor, _ := installFakeExecutors(backuper) + fullExecutor.result = postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusChainBroken, + ErrorReason: new(physical_enums.PhysicalBackupErrorManifestCorrupted), + } + + fullBackup := seedInProgressFull(t, prereqs) + + backuper.MakeBackup(fullBackup.ID, true) + + persisted, err := physical_repositories.GetFullBackupRepository().FindByID(fullBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusChainBroken, persisted.Status) + + require.Len(t, sender.sentNotifications, 1) + assert.Contains(t, sender.sentNotifications[0].Title, "chain-broken") +} + +func Test_RunIncrementalBackup_WhenParentManifestMissing_FlipsToChainBrokenBeforeExecutor(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + _, incrExecutor := installFakeExecutors(backuper) + + // Root full with no manifest_file_name: resolveParentManifest fails before the executor runs. + rootFull := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, walmath.LSN(0x1000000), walmath.LSN(0x2000000)) + physical_testing.CreateTestFullBackup(t, rootFull) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + + backuper.MakeBackup(incrBackup.ID, true) + + assert.Equal(t, 0, incrExecutor.callCount, "executor must not run when the parent manifest is unresolved") + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusChainBroken, persisted.Status, + "a missing parent manifest is irreversible, so the chain must break (not retry as ERROR)") + require.NotNil(t, persisted.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorParentManifestMissing, *persisted.ErrorReason) + + assert.Empty(t, sender.sentNotifications, + "pre-executor finalize paths are silent, matching the full-backup path") +} + +func Test_RunIncrementalBackup_WhenExecutorResultErrorStatus_PersistsErrorAndSendsFailedNotification(t *testing.T) { + prereqs := seedBackupPrereqs(t) + sender := &recordingNotificationSender{} + backuper := CreateTestPhysicalBackuper(sender) + _, incrExecutor := installFakeExecutors(backuper) + incrExecutor.result = erroredResult() + + rootFull := seedCompletedRootFull(t, prereqs) + incrBackup := seedInProgressIncr(t, prereqs, rootFull.ID) + + backuper.MakeBackup(incrBackup.ID, true) + + assert.Equal(t, 1, incrExecutor.callCount) + + persisted, err := physical_repositories.GetIncrementalBackupRepository().FindByID(incrBackup.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, persisted.Status) + + require.Len(t, sender.sentNotifications, 1) + assert.Contains(t, sender.sentNotifications[0].Title, "INCR failed") +} + +func Test_ReleaseOwned_WhenForeignBackupHoldsClaim_LeavesItIntact(t *testing.T) { + prereqs := seedBackupPrereqs(t) + inFlightRepo := physical_repositories.GetInFlightBackupRepository() + + liveBackupID := uuid.New() + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, liveBackupID) + + staleBackupID := uuid.New() + require.NoError(t, inFlightRepo.ReleaseOwned(prereqs.DB.ID, staleBackupID)) + + claim, err := inFlightRepo.FindByDatabaseID(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, claim, "a stale backup's release must not delete the live claim") + assert.Equal(t, liveBackupID, claim.BackupID) + + require.NoError(t, inFlightRepo.ReleaseOwned(prereqs.DB.ID, liveBackupID)) + assertInFlightReleased(t, prereqs.DB.ID) +} + +func Test_PersistFullResult_WhenRowNoLongerInProgress_DoesNotResurrect(t *testing.T) { + prereqs := seedBackupPrereqs(t) + backuper := CreateTestPhysicalBackuper(nil) + fullRepo := physical_repositories.GetFullBackupRepository() + inFlightRepo := physical_repositories.GetInFlightBackupRepository() + + full := seedInProgressFull(t, prereqs) + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, full.ID) + + // A restart recovery already failed this backup, released its claim, and a + // fresh backup took the database's in-flight slot. + require.NoError(t, fullRepo.UpdateStatus(full.ID, physical_enums.PhysicalBackupStatusError, nil)) + require.NoError(t, inFlightRepo.ReleaseOwned(prereqs.DB.ID, full.ID)) + + newBackupID := uuid.New() + claimInFlight(t, prereqs.DB.ID, physical_enums.PhysicalBackupTypeFull, newBackupID) + + err := backuper.persistFullResult(full, postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusCompleted, + }) + require.NoError(t, err) + + reloaded, err := fullRepo.FindByID(full.ID) + require.NoError(t, err) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, reloaded.Status, + "a superseded backup must not be resurrected to COMPLETED") + + claim, err := inFlightRepo.FindByDatabaseID(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, claim, "the newer backup's claim must survive") + assert.Equal(t, newBackupID, claim.BackupID) +} + +func installFakeExecutors(backuper *PhysicalBackuper) (*fakeFullExecutor, *fakeIncrementalExecutor) { + fullExecutor := &fakeFullExecutor{} + incrExecutor := &fakeIncrementalExecutor{} + backuper.fullExecutor = fullExecutor + backuper.incrExecutor = incrExecutor + + return fullExecutor, incrExecutor +} + +// erroredResult is the canned ERROR outcome the fake executors return when a +// test only cares that a non-COMPLETED result is persisted and routed. +func erroredResult() postgresql_executor.PhysicalBackupResult { + return postgresql_executor.PhysicalBackupResult{ + Status: physical_enums.PhysicalBackupStatusError, + ErrorReason: new(physical_enums.PhysicalBackupErrorPgBasebackupFailed), + } +} + +func seedInProgressFull(t *testing.T, prereqs *backupPrereqs) *physical_models.PhysicalFullBackup { + t.Helper() + + return physical_testing.CreateTestFullBackup(t, &physical_models.PhysicalFullBackup{ + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + TimelineID: 1, + Status: physical_enums.PhysicalBackupStatusInProgress, + Encryption: backups_core_enums.BackupEncryptionNone, + CreatedAt: time.Now().UTC(), + }) +} + +func seedCompletedRootFull(t *testing.T, prereqs *backupPrereqs) *physical_models.PhysicalFullBackup { + t.Helper() + + rootFull := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, walmath.LSN(0x1000000), walmath.LSN(0x2000000)) + rootFull.ManifestFileName = new("root.manifest") + + return physical_testing.CreateTestFullBackup(t, rootFull) +} + +func seedInProgressIncr( + t *testing.T, + prereqs *backupPrereqs, + rootFullBackupID uuid.UUID, +) *physical_models.PhysicalIncrementalBackup { + t.Helper() + + return physical_testing.CreateTestIncrementalBackup(t, &physical_models.PhysicalIncrementalBackup{ + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + RootFullBackupID: rootFullBackupID, + TimelineID: 1, + Status: physical_enums.PhysicalBackupStatusInProgress, + Encryption: backups_core_enums.BackupEncryptionNone, + CreatedAt: time.Now().UTC(), + }) +} + +func claimInFlight( + t *testing.T, + databaseID uuid.UUID, + backupType physical_enums.PhysicalBackupType, + backupID uuid.UUID, +) { + t.Helper() + + claimed, err := physical_repositories.GetInFlightBackupRepository().Claim( + storage.GetDb(), physical_repositories.ClaimSpec{ + DatabaseID: databaseID, + BackupType: backupType, + BackupID: backupID, + }) + require.NoError(t, err) + require.True(t, claimed) +} + +func assertInFlightReleased(t *testing.T, databaseID uuid.UUID) { + t.Helper() + + inFlight, err := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(databaseID) + require.NoError(t, err) + assert.Nil(t, inFlight, "in-flight claim must be released") +} diff --git a/backend/internal/features/backups/backups/backuping/physical/canceller.go b/backend/internal/features/backups/backups/backuping/physical/canceller.go new file mode 100644 index 0000000..7374a7c --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/canceller.go @@ -0,0 +1,80 @@ +package backuping_physical + +import ( + "log/slog" + + "github.com/google/uuid" + + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" +) + +// PhysicalBackupCanceller stands a database's in-flight physical backup down: it +// cancels the running backup task (the executor unwinds on the cancelled +// context) and releases the cross-table single-in-flight claim. It is the one +// place that knows how to stop a running FULL/INCR, shared by the config-change +// listener, the database-remove listener, and the user-facing cancel/delete +// endpoints. +type PhysicalBackupCanceller struct { + inFlightRepo *physical_repositories.PhysicalInFlightBackupRepository + taskCancelManager *tasks_cancellation.TaskCancelManager + logger *slog.Logger +} + +func NewPhysicalBackupCanceller( + inFlightRepo *physical_repositories.PhysicalInFlightBackupRepository, + taskCancelManager *tasks_cancellation.TaskCancelManager, + logger *slog.Logger, +) *PhysicalBackupCanceller { + return &PhysicalBackupCanceller{inFlightRepo, taskCancelManager, logger} +} + +// CancelInFlightForDatabase cancels whatever backup the database currently holds +// in flight, whichever it is. Use it for teardown (config disable, db removal) +// where any running backup must stop. A no claim is a no-op. +func (c *PhysicalBackupCanceller) CancelInFlightForDatabase(databaseID uuid.UUID) { + logger := c.logger.With("database_id", databaseID) + + claim, err := c.inFlightRepo.FindByDatabaseID(databaseID) + if err != nil { + logger.Error("failed to look up in-flight backup for cancel", "error", err) + + return + } + + if claim == nil { + return + } + + c.cancelClaim(logger, databaseID, claim.BackupID) +} + +// CancelInFlightBackup cancels the database's in-flight backup only when the +// claim still names backupID. It returns whether a matching claim was found and +// cancelled, so a delete path can tell "I stopped the running backup" from +// "nothing was running for this row". Scoping by backupID avoids stopping a +// newer backup that took the claim after the targeted one finished. +func (c *PhysicalBackupCanceller) CancelInFlightBackup(databaseID, backupID uuid.UUID) (bool, error) { + claim, err := c.inFlightRepo.FindByDatabaseID(databaseID) + if err != nil { + return false, err + } + + if claim == nil || claim.BackupID != backupID { + return false, nil + } + + c.cancelClaim(c.logger.With("database_id", databaseID), databaseID, backupID) + + return true, nil +} + +func (c *PhysicalBackupCanceller) cancelClaim(logger *slog.Logger, databaseID, backupID uuid.UUID) { + if err := c.taskCancelManager.CancelTask(backupID); err != nil { + logger.Error("failed to cancel in-flight backup task", "backup_id", backupID, "error", err) + } + + if err := c.inFlightRepo.ReleaseOwned(databaseID, backupID); err != nil { + logger.Error("failed to release in-flight claim", "backup_id", backupID, "error", err) + } +} diff --git a/backend/internal/features/backups/backups/backuping/physical/cleaner.go b/backend/internal/features/backups/backups/backuping/physical/cleaner.go new file mode 100644 index 0000000..ad40270 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/cleaner.go @@ -0,0 +1,482 @@ +package backuping_physical + +import ( + "context" + "fmt" + "log/slog" + "math" + "slices" + "sync/atomic" + "time" + + "github.com/google/uuid" + + "databasus-backend/internal/features/backups/backups/backuping/shared/gfs" + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_service "databasus-backend/internal/features/backups/backups/core/physical/service" + usecases_physical_postgresql "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/intervals" +) + +// PhysicalBackupCleaner trims physical backups every tick through two passes: +// retention policy and orphan sweep. It never issues raw DELETEs — every removal +// goes through PhysicalBackupService — and it never touches the active +// (extendable) chain: it operates only on the non-extendable set that chain_view +// derives. +type PhysicalBackupCleaner struct { + physicalBackupService *physical_service.PhysicalBackupService + chainViewService *chain_view.ChainViewService + backupConfigService *backups_config_physical.BackupConfigService + fullRepo *physical_repositories.PhysicalFullBackupRepository + walSegmentRepo *physical_repositories.PhysicalWalSegmentRepository + logger *slog.Logger + + hasRun atomic.Bool +} + +func (c *PhysicalBackupCleaner) Run(ctx context.Context) { + if c.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", c)) + } + + if ctx.Err() != nil { + return + } + + c.runStartupSlotCleanup(ctx) + + ticker := time.NewTicker(cleanerTickInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + logger := c.logger.With("job_id", uuid.New(), "job_name", cleanerJobName) + + if err := c.cleanByRetentionPolicy(ctx, logger); err != nil { + logger.Error("failed to clean by retention policy", "error", err) + } + + if err := c.cleanOrphans(ctx, logger); err != nil { + logger.Error("failed to clean orphans", "error", err) + } + } + } +} + +// runStartupSlotCleanup drops orphan per-backup replication slots once at boot. +// In single-instance mode any backup that was running died with the previous +// process, so no slot is protected — the scheduler's restart recovery fails and +// releases every in-flight claim, and the matching slots are reclaimable orphans. +func (c *PhysicalBackupCleaner) runStartupSlotCleanup(ctx context.Context) { + isSlotProtected := func(uuid.UUID) (bool, error) { return false, nil } + + if err := usecases_physical_postgresql.RunStartupCleanup(ctx, c.logger, isSlotProtected); err != nil { + c.logger.Error("physical startup slot cleanup reported failures", "error", err) + } +} + +func (c *PhysicalBackupCleaner) cleanByRetentionPolicy(ctx context.Context, logger *slog.Logger) error { + enabledConfigs, err := c.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + return err + } + + for _, backupConfig := range enabledConfigs { + dbLog := logger.With("database_id", backupConfig.DatabaseID, "retention", backupConfig.Retention) + + var cleanErr error + + switch backupConfig.Retention { + case backups_config_physical.RetentionChains: + cleanErr = c.cleanByChains(ctx, dbLog, backupConfig) + case backups_config_physical.RetentionFullBackups: + cleanErr = c.cleanByFulls(ctx, dbLog, backupConfig) + case backups_config_physical.RetentionChainsAndFullBackups: + cleanErr = c.cleanByCombined(ctx, dbLog, backupConfig) + } + + if cleanErr != nil { + dbLog.Error("failed to clean database by retention policy", "error", cleanErr) + } + } + + return nil +} + +// nonExtendableChainsNewestEndFirst returns every non-extendable chain for a DB +// sorted by chain-end timestamp, newest first. The active (extendable) head is +// already excluded by FindNonExtendableChainsByDatabase, so nothing here can +// ever delete it. +func (c *PhysicalBackupCleaner) nonExtendableChainsNewestEndFirst( + databaseID uuid.UUID, +) ([]chainCandidate, error) { + views, err := c.chainViewService.FindNonExtendableChainsByDatabase(databaseID) + if err != nil { + return nil, err + } + + candidates := make([]chainCandidate, 0, len(views)) + + for _, view := range views { + endTs, err := c.chainViewService.GetChainEndTimestamp(view.RootFull.ID) + if err != nil { + c.logger.Error("failed to get chain end timestamp", + "root_full_backup_id", view.RootFull.ID, "error", err) + + continue + } + + candidates = append(candidates, chainCandidate{view: view, endTs: endTs}) + } + + slices.SortFunc(candidates, func(a, b chainCandidate) int { + return b.endTs.Compare(a.endTs) + }) + + return candidates, nil +} + +// deleteChainThroughService removes a whole chain via the service, bounded by +// the per-tick WAL byte budget, logging progress. +func (c *PhysicalBackupCleaner) deleteChainThroughService( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, + rootFullBackupID uuid.UUID, +) { + summary, err := c.physicalBackupService.DeleteFull( + ctx, rootFullBackupID, c.walDeleteBudgetMB(backupConfig.DatabaseID), + ) + if err != nil { + logger.Error("failed to delete chain", "root_full_backup_id", rootFullBackupID, "error", err) + + return + } + + logger.Info(fmt.Sprintf( + "chain cleanup progress: %d wal, %d incr, %d history, %.1f MB, complete=%v", + summary.WalSegments, summary.Incrementals, summary.HistoryFiles, + summary.BytesDeletedMB, summary.ChainFullyDeleted, + ), "root_full_backup_id", rootFullBackupID) +} + +// deleteChainDependentsThroughService drops a chain's INCRs and WAL but keeps +// the FULL — used by FULL_BACKUPS policies that retain a kept FULL as a +// standalone restore point. +func (c *PhysicalBackupCleaner) deleteChainDependentsThroughService( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, + rootFullBackupID uuid.UUID, +) { + summary, err := c.physicalBackupService.DeleteChainDependentsKeepFull( + ctx, rootFullBackupID, c.walDeleteBudgetMB(backupConfig.DatabaseID), + ) + if err != nil { + logger.Error("failed to delete chain dependents", "root_full_backup_id", rootFullBackupID, "error", err) + + return + } + + logger.Info(fmt.Sprintf( + "chain dependents cleanup: %d wal, %d incr, %.1f MB", + summary.WalSegments, summary.Incrementals, summary.BytesDeletedMB, + ), "root_full_backup_id", rootFullBackupID) +} + +// walDeleteBudgetMB anchors the per-tick WAL byte budget to the latest COMPLETED +// FULL's size, floored at minWalDeleteBudgetMB. +func (c *PhysicalBackupCleaner) walDeleteBudgetMB(databaseID uuid.UUID) float64 { + fulls, err := c.fullRepo.FindCompletedNewestFirstByDatabase(databaseID) + if err == nil && len(fulls) > 0 && fulls[0].BackupSizeMb != nil { + return math.Max(*fulls[0].BackupSizeMb, minWalDeleteBudgetMB) + } + + return minWalDeleteBudgetMB +} + +// isChainWithinGrace reports whether a chain is too young to evict: its end +// timestamp is within max(full, incr cadence) × 2 (floored at the 60-minute +// per-backup grace). A failure to read the timestamp is treated as within grace +// so a transient error never causes a premature delete. +func (c *PhysicalBackupCleaner) isChainWithinGrace( + backupConfig *backups_config_physical.PhysicalBackupConfig, + rootFullBackupID uuid.UUID, +) bool { + endTs, err := c.chainViewService.GetChainEndTimestamp(rootFullBackupID) + if err != nil { + c.logger.Error("failed to get chain end timestamp; treating as within grace", + "root_full_backup_id", rootFullBackupID, "error", err) + + return true + } + + grace := max( + approxIntervalDuration(backupConfig.FullBackupInterval), + approxIntervalDuration(backupConfig.IncrementalBackupInterval), + ) * chainGraceIntervalMultiplier + + grace = max(grace, recentBackupGracePeriod) + + return time.Since(endTs) < grace +} + +// cleanByChains keeps the N newest non-extendable chains (by chain-end +// timestamp) and deletes the rest, honoring the per-chain grace period. The +// active chain is never a candidate. +func (c *PhysicalBackupCleaner) cleanByChains( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) error { + keepCount := backupConfig.ChainsRetention.Count + if keepCount <= 0 { + return nil + } + + candidates, err := c.nonExtendableChainsNewestEndFirst(backupConfig.DatabaseID) + if err != nil { + return err + } + + if len(candidates) <= keepCount { + return nil + } + + for _, candidate := range candidates[keepCount:] { + if c.isChainWithinGrace(backupConfig, candidate.view.RootFull.ID) { + continue + } + + c.deleteChainThroughService(ctx, logger, backupConfig, candidate.view.RootFull.ID) + } + + return nil +} + +// cleanByFulls dispatches the FULL_BACKUPS retention policy to its LAST_N or GFS +// variant. Both keep a set of FULLs; non-extendable chains rooted at a kept FULL +// shed their INCR + WAL (the FULL stays a standalone restore point), and chains +// rooted at a non-kept FULL are deleted whole. +func (c *PhysicalBackupCleaner) cleanByFulls( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) error { + completedFulls, err := c.fullRepo.FindCompletedNewestFirstByDatabase(backupConfig.DatabaseID) + if err != nil { + return err + } + + keepFullIDs := c.fullBackupsKeepSet(backupConfig, completedFulls) + if keepFullIDs == nil { + return nil + } + + candidates, err := c.nonExtendableChainsNewestEndFirst(backupConfig.DatabaseID) + if err != nil { + return err + } + + for _, candidate := range candidates { + rootFullBackupID := candidate.view.RootFull.ID + + if c.isChainWithinGrace(backupConfig, rootFullBackupID) { + continue + } + + if keepFullIDs[rootFullBackupID] { + c.deleteChainDependentsThroughService(ctx, logger, backupConfig, rootFullBackupID) + + continue + } + + c.deleteChainThroughService(ctx, logger, backupConfig, rootFullBackupID) + } + + return nil +} + +// fullBackupsKeepSet computes the kept-FULL id set for the FULL_BACKUPS policy. +// Returns nil when the policy has no effective configuration (so the caller +// no-ops rather than treating an empty set as "delete everything"). +func (c *PhysicalBackupCleaner) fullBackupsKeepSet( + backupConfig *backups_config_physical.PhysicalBackupConfig, + completedFullsNewestFirst []*physical_models.PhysicalFullBackup, +) map[uuid.UUID]bool { + retention := backupConfig.FullBackupsRetention + + switch retention.Policy { + case backups_config_physical.FullBackupsRetentionPolicyLastN: + if retention.Count <= 0 { + return nil + } + + keep := make(map[uuid.UUID]bool) + for i, full := range completedFullsNewestFirst { + if i < retention.Count { + keep[full.ID] = true + } + } + + return keep + + case backups_config_physical.FullBackupsRetentionPolicyGfs: + if retention.GfsHours <= 0 && retention.GfsDays <= 0 && retention.GfsWeeks <= 0 && + retention.GfsMonths <= 0 && retention.GfsYears <= 0 { + return nil + } + + items := make([]gfs.Item, len(completedFullsNewestFirst)) + for i, full := range completedFullsNewestFirst { + items[i] = gfs.Item{ID: full.ID, CreatedAt: full.CreatedAt} + } + + return gfs.GetItemsToRetain( + items, retention.GfsHours, retention.GfsDays, retention.GfsWeeks, + retention.GfsMonths, retention.GfsYears, + ) + } + + return nil +} + +// cleanByCombined keeps the UNION of the CHAINS keep-set (N newest +// non-extendable chains) and the FULL_BACKUPS keep-set (LAST_N or GFS over +// FULLs), deleting every other non-extendable chain whole. A chain in either +// keep-set is preserved entirely (INCR + WAL included). +func (c *PhysicalBackupCleaner) cleanByCombined( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) error { + candidates, err := c.nonExtendableChainsNewestEndFirst(backupConfig.DatabaseID) + if err != nil { + return err + } + + keepRoots := make(map[uuid.UUID]bool) + + for i, candidate := range candidates { + if i < backupConfig.ChainsRetention.Count { + keepRoots[candidate.view.RootFull.ID] = true + } + } + + completedFulls, err := c.fullRepo.FindCompletedNewestFirstByDatabase(backupConfig.DatabaseID) + if err != nil { + return err + } + + for fullID := range c.fullBackupsKeepSet(backupConfig, completedFulls) { + keepRoots[fullID] = true + } + + for _, candidate := range candidates { + rootFullBackupID := candidate.view.RootFull.ID + + if keepRoots[rootFullBackupID] { + continue + } + + if c.isChainWithinGrace(backupConfig, rootFullBackupID) { + continue + } + + c.deleteChainThroughService(ctx, logger, backupConfig, rootFullBackupID) + } + + return nil +} + +// cleanOrphans removes WAL that no longer belongs to any chain and reaps +// abandoned insert-first claims, per enabled database. +func (c *PhysicalBackupCleaner) cleanOrphans(ctx context.Context, logger *slog.Logger) error { + enabledConfigs, err := c.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + return err + } + + for _, backupConfig := range enabledConfigs { + dbLog := logger.With("database_id", backupConfig.DatabaseID) + + c.cleanOrphanWalForDatabase(ctx, dbLog, backupConfig.DatabaseID) + c.reapAbandonedWalClaims(dbLog, backupConfig.DatabaseID) + } + + return nil +} + +func (c *PhysicalBackupCleaner) cleanOrphanWalForDatabase( + ctx context.Context, + logger *slog.Logger, + databaseID uuid.UUID, +) { + orphans, err := c.chainViewService.FindWalOrphansByDatabase(databaseID) + if err != nil { + logger.Error("failed to find orphan wal", "error", err) + + return + } + + budget := c.walDeleteBudgetMB(databaseID) + + for _, orphan := range orphans { + segment := orphan.WalSegment + + // A one-segment span [start, end) uniquely matches this orphan — segments + // never overlap, so no chain-covered WAL is ever caught. + span := chain_view.LSNRange{Start: segment.StartLSN, End: segment.EndLSN} + + rows, mb, err := c.physicalBackupService.DeleteWalSegmentsInSpan( + ctx, databaseID, segment.TimelineID, span, budget, + ) + if err != nil { + logger.Error("failed to delete orphan wal segment", "wal_filename", segment.WalFilename, "error", err) + + continue + } + + if rows > 0 { + logger.Info(fmt.Sprintf("deleted orphan wal segment (%.2f MB)", mb), "wal_filename", segment.WalFilename) + } + } +} + +func (c *PhysicalBackupCleaner) reapAbandonedWalClaims(logger *slog.Logger, databaseID uuid.UUID) { + cutoff := time.Now().UTC().Add(-walClaimGracePeriod) + + deleted, err := c.walSegmentRepo.DeleteAbandonedClaims(databaseID, cutoff) + if err != nil { + logger.Error("failed to reap abandoned wal claims", "error", err) + + return + } + + if deleted > 0 { + logger.Info(fmt.Sprintf("reaped %d abandoned wal claims", deleted)) + } +} + +// approxIntervalDuration maps a schedule to an approximate period for grace +// math. Cron and unset schedules are treated as daily. +func approxIntervalDuration(interval intervals.Interval) time.Duration { + switch interval.Type { + case intervals.IntervalHourly: + return time.Hour + case intervals.IntervalDaily: + return 24 * time.Hour + case intervals.IntervalWeekly: + return 7 * 24 * time.Hour + case intervals.IntervalMonthly: + return 30 * 24 * time.Hour + default: + return 24 * time.Hour + } +} diff --git a/backend/internal/features/backups/backups/backuping/physical/cleaner_test.go b/backend/internal/features/backups/backups/backuping/physical/cleaner_test.go new file mode 100644 index 0000000..5677c8b --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/cleaner_test.go @@ -0,0 +1,368 @@ +package backuping_physical + +import ( + "context" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_testing "databasus-backend/internal/features/backups/backups/core/physical/testing" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/intervals" + "databasus-backend/internal/storage" + "databasus-backend/internal/util/logger" +) + +// Test_RunStartupSlotCleanup_DropsOrphanSlot is the end-to-end proof of the +// "Databasus crashed mid-backup" path: the source keeps the per-backup slot and +// the in-flight claim survives the crash. In single-instance mode the backup that +// held the slot died with the process, so the startup sweep treats every claimed +// slot as a reclaimable orphan and drops it against the real source PG. +func Test_RunStartupSlotCleanup_DropsOrphanSlot(t *testing.T) { + fixture := postgresql_executor.SetupPhysicalDBForBackup(t) + cleaner := CreateTestPhysicalCleaner() + + slotName := postgresql_executor.SlotName(fixture.DB.PostgresqlPhysical.ID) + adminConn := postgresql_executor.OpenAdminConn(t, fixture) + + _, err := adminConn.Exec(context.Background(), + "SELECT pg_create_physical_replication_slot($1, true)", slotName) + require.NoError(t, err, "pre-create the orphan slot a crashed backup left behind") + t.Cleanup(func() { + _, _ = adminConn.Exec(context.Background(), + `SELECT pg_drop_replication_slot(slot_name) + FROM pg_replication_slots WHERE slot_name = $1`, slotName) + }) + + require.NoError(t, physical_repositories.GetInFlightBackupRepository().Release(fixture.DB.ID)) + _, err = physical_repositories.GetInFlightBackupRepository().Claim( + storage.GetDb(), physical_repositories.ClaimSpec{ + DatabaseID: fixture.DB.ID, + BackupType: physical_enums.PhysicalBackupTypeFull, + BackupID: fixture.BackupID, + }) + require.NoError(t, err) + + cleaner.runStartupSlotCleanup(t.Context()) + + assert.False(t, postgresql_executor.SlotExists(t, adminConn, slotName), + "a per-backup orphan slot must be dropped at startup in single-instance mode") +} + +// seedChainFull seeds a COMPLETED FULL at a start segment and age so tests can +// build multiple distinct chains with deterministic ordering (higher segment = +// newer = the active head). +func seedChainFull( + t *testing.T, + prereqs *backupPrereqs, + startSegment, ageHours int, +) *physical_models.PhysicalFullBackup { + t.Helper() + + full := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, testLSN(startSegment), testLSN(startSegment+1)) + + at := time.Now().UTC().Add(-time.Duration(ageHours) * time.Hour) + full.CreatedAt = at + full.CompletedAt = &at + full.BackupSizeMb = new(1000.0) + + return physical_testing.CreateTestFullBackup(t, full) +} + +// shortGraceConfig sets hourly cadences so the per-chain grace is 2 h — chains +// older than 2 h become evictable, which keeps retention tests deterministic. +func shortGraceConfig(backupConfig *backups_config_physical.PhysicalBackupConfig) { + backupConfig.FullBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} + backupConfig.IncrementalBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} +} + +func fullExists(t *testing.T, id uuid.UUID) bool { + t.Helper() + + full, err := physical_repositories.GetFullBackupRepository().FindByID(id) + require.NoError(t, err) + + return full != nil +} + +func incrementalExists(t *testing.T, id uuid.UUID) bool { + t.Helper() + + incremental, err := physical_repositories.GetIncrementalBackupRepository().FindByID(id) + require.NoError(t, err) + + return incremental != nil +} + +func walExists(t *testing.T, id uuid.UUID) bool { + t.Helper() + + segment, err := physical_repositories.GetWalSegmentRepository().FindByID(id) + require.NoError(t, err) + + return segment != nil +} + +func Test_CleanByChains_KeepsNLatestNonExtendableChains(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.ChainsRetention = backups_config_physical.ChainsRetention{Count: 2} + + active := seedChainFull(t, prereqs, 8, 0) // newest → extendable head + keptA := seedChainFull(t, prereqs, 6, 3) // non-extendable, kept by count + keptB := seedChainFull(t, prereqs, 4, 6) // non-extendable, kept by count + oldA := seedChainFull(t, prereqs, 2, 9) // non-extendable, evicted + oldB := seedChainFull(t, prereqs, 0, 12) // non-extendable, evicted + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByChains(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID), "active chain is never a candidate") + assert.True(t, fullExists(t, keptA.ID)) + assert.True(t, fullExists(t, keptB.ID)) + assert.False(t, fullExists(t, oldA.ID)) + assert.False(t, fullExists(t, oldB.ID)) +} + +func Test_CleanByChains_GracePeriodProtectsRecentChainBeyondKeepCount(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) // 2 h grace + prereqs.Config.ChainsRetention = backups_config_physical.ChainsRetention{Count: 1} + + active := seedChainFull(t, prereqs, 6, 0) + recentKept := seedChainFull(t, prereqs, 4, 0) // within grace, kept by count anyway + recentBeyond := seedChainFull(t, prereqs, 2, 1) // within grace, beyond count → grace saves it + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByChains(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, recentKept.ID)) + assert.True(t, fullExists(t, recentBeyond.ID), "a chain younger than the grace period is never evicted") +} + +func Test_CleanByFullsLastN_KeepsNewestFullsAndDropsTheirIncrAndWal(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.FullBackupsRetention = backups_config_physical.FullBackupsRetention{ + Policy: backups_config_physical.FullBackupsRetentionPolicyLastN, + Count: 2, + } + + active := seedChainFull(t, prereqs, 9, 0) + keptFull := seedChainFull(t, prereqs, 6, 3) // non-extendable but in newest-2 fulls + droppedFull := seedChainFull(t, prereqs, 3, 6) + + // keptFull's chain gets an INCR + WAL that LAST_N must shed while keeping the + // FULL. Age their timestamps to 3 h so the chain-end stays outside the grace + // window (a fresh INCR/WAL would make the whole chain grace-protected). + threeHoursAgo := time.Now().UTC().Add(-3 * time.Hour) + + incrModel := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.DB.ID, prereqs.Storage.ID, keptFull.ID, nil, 1, testLSN(7), testLSN(8)) + incrModel.CreatedAt = threeHoursAgo + incrModel.CompletedAt = &threeHoursAgo + keptIncr := physical_testing.CreateTestIncrementalBackup(t, incrModel) + + walModel := physical_testing.NewTestWalSegment( + prereqs.DB.ID, prereqs.Storage.ID, 1, "000000010000000000000007", testLSN(7), testLSN(8)) + walModel.ReceivedAt = threeHoursAgo + keptWal := physical_testing.CreateTestWalSegment(t, walModel) + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByFulls(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, keptFull.ID), "newest-N full kept") + assert.False(t, incrementalExists(t, keptIncr.ID), "kept full's incr dropped") + assert.False(t, walExists(t, keptWal.ID), "kept full's wal dropped") + assert.False(t, fullExists(t, droppedFull.ID), "non-kept chain deleted entirely") +} + +func Test_CleanByFullsGfs_KeepsBucketRepresentativesDropsExtras(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.FullBackupsRetention = backups_config_physical.FullBackupsRetention{ + Policy: backups_config_physical.FullBackupsRetentionPolicyGfs, + GfsHours: 2, + } + + // 4 distinct hourly buckets; GFS keeps the 2 newest (active + age2). + active := seedChainFull(t, prereqs, 9, 0) + keptHour := seedChainFull(t, prereqs, 6, 2) + droppedA := seedChainFull(t, prereqs, 3, 3) + droppedB := seedChainFull(t, prereqs, 0, 4) + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByFulls(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, keptHour.ID), "GFS keeps the 2nd-newest hourly bucket") + assert.False(t, fullExists(t, droppedA.ID)) + assert.False(t, fullExists(t, droppedB.ID)) +} + +func Test_CleanByCombined_KeepsUnionOfChainsAndFulls(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.ChainsRetention = backups_config_physical.ChainsRetention{Count: 1} + prereqs.Config.FullBackupsRetention = backups_config_physical.FullBackupsRetention{ + Policy: backups_config_physical.FullBackupsRetentionPolicyLastN, + Count: 3, + } + + active := seedChainFull(t, prereqs, 9, 0) + keptByChains := seedChainFull(t, prereqs, 6, 3) // newest non-ext → CHAINS keeps it + keptByFulls := seedChainFull(t, prereqs, 3, 6) // 3rd-newest full → FULL_BACKUPS keeps it + dropped := seedChainFull(t, prereqs, 0, 9) // in neither keep-set + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByCombined(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, keptByChains.ID), "kept by CHAINS policy") + assert.True(t, fullExists(t, keptByFulls.ID), "kept by FULL_BACKUPS policy even though CHAINS count=1") + assert.False(t, fullExists(t, dropped.ID)) +} + +func Test_CleanOrphanWalForDatabase_WhenWalOutsideAllChains_DeletesOrphan(t *testing.T) { + prereqs := seedBackupPrereqs(t) + + // No FULL covers this segment, so it is orphan WAL the cleaner must reclaim. + orphan := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.DB.ID, prereqs.Storage.ID, 1, "000000010000000000000005", testLSN(5), testLSN(6))) + + cleaner := CreateTestPhysicalCleaner() + cleaner.cleanOrphanWalForDatabase(context.Background(), logger.GetLogger(), prereqs.DB.ID) + + assert.False(t, walExists(t, orphan.ID), "WAL not covered by any chain is deleted") +} + +func Test_CleanOrphanWalForDatabase_WhenWalCoveredByChain_KeepsIt(t *testing.T) { + prereqs := seedBackupPrereqs(t) + + // A COMPLETED FULL at segment 1 covers everything from its start_lsn onward on + // the same timeline, so the segment at 2 is in-chain, not an orphan. + seedChainFull(t, prereqs, 1, 0) + covered := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.DB.ID, prereqs.Storage.ID, 1, "000000010000000000000002", testLSN(2), testLSN(3))) + + cleaner := CreateTestPhysicalCleaner() + cleaner.cleanOrphanWalForDatabase(context.Background(), logger.GetLogger(), prereqs.DB.ID) + + assert.True(t, walExists(t, covered.ID), "chain-covered WAL must never be caught by the orphan pass") +} + +func Test_ReapAbandonedWalClaims_WhenClaimOlderThanGrace_DeletesIt(t *testing.T) { + prereqs := seedBackupPrereqs(t) + + // An insert-first claim (file_name NULL) whose upload never finished and that + // has aged past WAL_CLAIM_GRACE (1h). NULL file_name ⇒ no storage object exists. + twoHoursAgo := time.Now().UTC().Add(-2 * time.Hour) + abandoned := &physical_models.PhysicalWalSegment{ + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + TimelineID: 1, + FileName: nil, + WalFilename: "000000010000000000000009", + StartLSN: testLSN(9), + EndLSN: testLSN(10), + ReceivedAt: twoHoursAgo, + ClaimedAt: twoHoursAgo, + } + require.NoError(t, physical_repositories.GetWalSegmentRepository().Insert(abandoned)) + + cleaner := CreateTestPhysicalCleaner() + cleaner.reapAbandonedWalClaims(logger.GetLogger(), prereqs.DB.ID) + + assert.False( + t, + walExists(t, abandoned.ID), + "an abandoned NULL-file_name claim past grace is reaped (no storage I/O)", + ) +} + +func Test_ReapAbandonedWalClaims_WhenClaimWithinGrace_KeepsIt(t *testing.T) { + prereqs := seedBackupPrereqs(t) + + thirtyMinutesAgo := time.Now().UTC().Add(-30 * time.Minute) + freshClaim := &physical_models.PhysicalWalSegment{ + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + TimelineID: 1, + FileName: nil, + WalFilename: "000000010000000000000009", + StartLSN: testLSN(9), + EndLSN: testLSN(10), + ReceivedAt: thirtyMinutesAgo, + ClaimedAt: thirtyMinutesAgo, + } + require.NoError(t, physical_repositories.GetWalSegmentRepository().Insert(freshClaim)) + + cleaner := CreateTestPhysicalCleaner() + cleaner.reapAbandonedWalClaims(logger.GetLogger(), prereqs.DB.ID) + + assert.True(t, walExists(t, freshClaim.ID), "a live in-flight claim within grace must survive") +} + +func Test_CleanByChains_WhenKeepCountZero_DeletesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.ChainsRetention = backups_config_physical.ChainsRetention{Count: 0} + + active := seedChainFull(t, prereqs, 4, 0) + old := seedChainFull(t, prereqs, 2, 9) + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByChains(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, old.ID), "keep-count 0 is treated as keep-all, not delete-all") +} + +func Test_CleanByFulls_WhenNoEffectiveConfig_DeletesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + // LAST_N with count 0 yields no keep-set; the policy must no-op rather than + // interpret an empty keep-set as "delete every chain". + prereqs.Config.FullBackupsRetention = backups_config_physical.FullBackupsRetention{ + Policy: backups_config_physical.FullBackupsRetentionPolicyLastN, + Count: 0, + } + + active := seedChainFull(t, prereqs, 4, 0) + old := seedChainFull(t, prereqs, 2, 9) + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByFulls(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, old.ID), "an empty keep-set must never delete everything") +} + +func Test_CleanByChains_WhenInProgressFullExists_NeverDeletesIt(t *testing.T) { + prereqs := seedBackupPrereqs(t) + shortGraceConfig(prereqs.Config) + prereqs.Config.ChainsRetention = backups_config_physical.ChainsRetention{Count: 1} + + inProgress := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusInProgress, 9, 1) + active := seedChainFull(t, prereqs, 6, 3) // newest COMPLETED → extendable head + keptByCount := seedChainFull(t, prereqs, 4, 6) + evicted := seedChainFull(t, prereqs, 2, 9) + + cleaner := CreateTestPhysicalCleaner() + require.NoError(t, cleaner.cleanByChains(context.Background(), logger.GetLogger(), prereqs.Config)) + + assert.True(t, fullExists(t, inProgress.ID), "an IN_PROGRESS full is never a retention candidate") + assert.True(t, fullExists(t, active.ID)) + assert.True(t, fullExists(t, keptByCount.ID)) + assert.False(t, fullExists(t, evicted.ID), "completed chains beyond the keep-count are still evicted") +} diff --git a/backend/internal/features/backups/backups/backuping/physical/config_change_listener.go b/backend/internal/features/backups/backups/backuping/physical/config_change_listener.go new file mode 100644 index 0000000..1cf3c01 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/config_change_listener.go @@ -0,0 +1,66 @@ +package backuping_physical + +import ( + "log/slog" + + "github.com/google/uuid" + + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" +) + +// PhysicalBackupCancellationListener stands physical backup work down on the two +// events that require it: a config change (backups disabled or BackupType +// demoted from WAL_STREAM) and a database removal. It implements both the +// config-change and db-remove listener seams. +type PhysicalBackupCancellationListener struct { + canceller *PhysicalBackupCanceller + walStreamerRepo *physical_repositories.PhysicalWalStreamerRepository + taskCancelManager *tasks_cancellation.TaskCancelManager + logger *slog.Logger +} + +// OnBackupConfigChanged reacts to a config transition. Disabling backups cancels +// any in-flight FULL/INCR; both disabling and demoting BackupType away from +// WAL_STREAM delete the streamer row so the scheduler cannot silently re-spawn a +// streamer the user turned off. The source-PG replication slot is intentionally +// preserved (re-enabling would reuse it). +func (l *PhysicalBackupCancellationListener) OnBackupConfigChanged( + oldConfig, newConfig *backups_config_physical.PhysicalBackupConfig, +) { + databaseID := newConfig.DatabaseID + logger := l.logger.With("database_id", databaseID) + + if oldConfig.IsBackupsEnabled && !newConfig.IsBackupsEnabled { + l.canceller.CancelInFlightForDatabase(databaseID) + } + + l.deleteStreamerRow(logger, databaseID) +} + +// OnBeforeDatabaseRemove cancels any in-flight backup and removes the streamer +// row before the database (and its cascade-deleted catalog rows) goes away. +func (l *PhysicalBackupCancellationListener) OnBeforeDatabaseRemove(databaseID uuid.UUID) error { + logger := l.logger.With("database_id", databaseID) + + l.canceller.CancelInFlightForDatabase(databaseID) + l.deleteStreamerRow(logger, databaseID) + + return nil +} + +// deleteStreamerRow tears the WAL streamer down: it first cancels the +// long-running streamer task (registered in TaskCancelManager keyed by +// database_id) so the local pg_receivewal goroutine stops, then deletes the +// heartbeat row so the supervisor cannot re-spawn it. Order matters — dropping +// the row without cancelling would leave an orphaned receiver holding the slot. +func (l *PhysicalBackupCancellationListener) deleteStreamerRow(logger *slog.Logger, databaseID uuid.UUID) { + if err := l.taskCancelManager.CancelTask(databaseID); err != nil { + logger.Error("failed to cancel wal streamer task", "error", err) + } + + if err := l.walStreamerRepo.DeleteByDatabaseID(databaseID); err != nil { + logger.Error("failed to delete wal streamer row", "error", err) + } +} diff --git a/backend/internal/features/backups/backups/backuping/physical/config_change_listener_test.go b/backend/internal/features/backups/backups/backuping/physical/config_change_listener_test.go new file mode 100644 index 0000000..4a1ee51 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/config_change_listener_test.go @@ -0,0 +1,105 @@ +package backuping_physical + +import ( + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + postgresql_physical "databasus-backend/internal/features/databases/databases/postgresql/physical" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + "databasus-backend/internal/storage" + "databasus-backend/internal/util/logger" +) + +func newTestCancellationListener() *PhysicalBackupCancellationListener { + return &PhysicalBackupCancellationListener{ + NewPhysicalBackupCanceller( + physical_repositories.GetInFlightBackupRepository(), + tasks_cancellation.GetTaskCancelManager(), + logger.GetLogger(), + ), + physical_repositories.GetWalStreamerRepository(), + tasks_cancellation.GetTaskCancelManager(), + logger.GetLogger(), + } +} + +func seedClaimAndStreamer(t *testing.T, databaseID uuid.UUID) { + t.Helper() + + ok, err := physical_repositories.GetInFlightBackupRepository().Claim( + storage.GetDb(), physical_repositories.ClaimSpec{ + DatabaseID: databaseID, + BackupType: physical_enums.PhysicalBackupTypeFull, + BackupID: uuid.New(), + }) + require.NoError(t, err) + require.True(t, ok) + + require.NoError(t, physical_repositories.GetWalStreamerRepository().Claim(databaseID)) +} + +func configWithBackupType( + databaseID uuid.UUID, + enabled bool, + backupType postgresql_physical.BackupType, +) *backups_config_physical.PhysicalBackupConfig { + return &backups_config_physical.PhysicalBackupConfig{ + DatabaseID: databaseID, + IsBackupsEnabled: enabled, + PostgresqlPhysical: &postgresql_physical.PostgresqlPhysicalDatabase{BackupType: backupType}, + } +} + +func Test_OnBackupConfigChanged_WhenBackupsDisabled_CancelsInFlightAndDeletesStreamer(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedClaimAndStreamer(t, prereqs.DB.ID) + listener := newTestCancellationListener() + + oldConfig := configWithBackupType(prereqs.DB.ID, true, postgresql_physical.BackupTypeFullIncrementalAndWalStream) + newConfig := configWithBackupType(prereqs.DB.ID, false, postgresql_physical.BackupTypeFullIncrementalAndWalStream) + + listener.OnBackupConfigChanged(oldConfig, newConfig) + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, claim, "disabling backups must cancel + release the in-flight claim") + + streamer, _ := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, streamer, "disabling backups must delete the streamer row") +} + +func Test_OnBackupConfigChanged_WhenDemotedFromWalStream_DeletesStreamerKeepsInFlight(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedClaimAndStreamer(t, prereqs.DB.ID) + listener := newTestCancellationListener() + + oldConfig := configWithBackupType(prereqs.DB.ID, true, postgresql_physical.BackupTypeFullIncrementalAndWalStream) + newConfig := configWithBackupType(prereqs.DB.ID, true, postgresql_physical.BackupTypeFullAndIncremental) + + listener.OnBackupConfigChanged(oldConfig, newConfig) + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.NotNil(t, claim, "demoting BackupType must leave in-flight FULL/INCR running") + + streamer, _ := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, streamer, "demoting BackupType must delete the streamer row") +} + +func Test_OnBeforeDatabaseRemove_CancelsInFlightAndDeletesStreamer(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedClaimAndStreamer(t, prereqs.DB.ID) + listener := newTestCancellationListener() + + require.NoError(t, listener.OnBeforeDatabaseRemove(prereqs.DB.ID)) + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, claim) + + streamer, _ := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, streamer) +} diff --git a/backend/internal/features/backups/backups/backuping/physical/constants.go b/backend/internal/features/backups/backups/backuping/physical/constants.go new file mode 100644 index 0000000..3ec15f9 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/constants.go @@ -0,0 +1,65 @@ +package backuping_physical + +import "time" + +const ( + // Scheduler tick and recovery sweep share this cadence. Each tick is a handful + // of cheap indexed queries per enabled DB, so a 1 s cadence keeps out-of-cadence + // backup triggers and crash recovery near-immediate without meaningful load + // (user-configured FULL/INCR intervals still span hours to days — the tick only + // decides whether one is due). + schedulerTickInterval = 1 * time.Second + + // IsSchedulerRunning() reports unhealthy when no tick completed within this + // window. Mirrors logical's healthcheck threshold. + schedulerHealthcheckThreshold = 5 * time.Minute + + // Stable log job_name for the scheduler; never the struct type name, since a + // rename would silently break log queries. + schedulerJobName = "physical_backup_scheduler" + + // Cleaner ticks every 3 s so retention decisions become visible almost + // immediately, while the per-tick WAL byte budget keeps a single tick bounded + // even at this cadence. Mirrors logical. + cleanerTickInterval = 3 * time.Second + + // Never delete a chain whose end timestamp is younger than + // max(full, incr cadence) × 2 — protects a chain that just completed or is + // still being extended from premature retention eviction. + chainGraceIntervalMultiplier = 2 + + // A WAL row with file_name still NULL past 1 h is an abandoned insert-first + // claim (the live owner finishes in seconds); reap it so the + // (database_id, timeline_id, start_lsn) slot is free to re-receive. + walClaimGracePeriod = 1 * time.Hour + + cleanerJobName = "physical_backup_retention_cleanup" + + // The WAL-stream supervisor reconciles owned streamers on the same 15 s + // cadence as the scheduler — cadence-driven work is cheap, and the tighter + // bound comes from streamer reclaim, where 15 s + streamerHeartbeatStaleness + // keeps detect-to-reclaim under ~2 min after a crash. + walStreamSupervisorTickInterval = 15 * time.Second + + // A streamer row whose heartbeat is older than this is assumed dead and + // reclaimable by another process. Heartbeats fire every tick (15 s), so 90 s = + // 6 missed beats — tolerant of GC pauses without leaving a dead streamer + // unclaimed for long. + streamerHeartbeatStaleness = 90 * time.Second + + // Bounds how long a reconcile waits for one streamer goroutine to drain on + // stop so a stuck pg_receivewal teardown can't stall the whole supervisor. + streamerStopTimeout = 30 * time.Second + + walStreamSupervisorJobName = "physical_wal_stream_supervisor" +) + +// Per-tick WAL deletion budget anchors to the latest FULL's size (a cluster +// producing a 10 GB FULL produces O(10 GB) of WAL between FULLs, so "one FULL's +// worth" is a self-scaling chunk), with a 256 MB floor so clusters with tiny +// FULLs but heavy WAL don't crawl. +const minWalDeleteBudgetMB float64 = 256 + +// Conservative fallback floor in cleaner grace logic, mirroring logical's +// 60-minute floor on individual backups. +const recentBackupGracePeriod = 60 * time.Minute diff --git a/backend/internal/features/backups/backups/backuping/physical/di.go b/backend/internal/features/backups/backups/backuping/physical/di.go new file mode 100644 index 0000000..e9d3132 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/di.go @@ -0,0 +1,124 @@ +package backuping_physical + +import ( + "sync" + "sync/atomic" + + "github.com/google/uuid" + + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_service "databasus-backend/internal/features/backups/backups/core/physical/service" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + encryption_secrets "databasus-backend/internal/features/encryption/secrets" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + workspaces_services "databasus-backend/internal/features/workspaces/services" + "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/logger" +) + +var physicalBackuper = &PhysicalBackuper{ + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + workspaces_services.GetWorkspaceService(), + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetIncrementalBackupRepository(), + physical_repositories.GetInFlightBackupRepository(), + physical_repositories.GetWalHistoryRepository(), + backups_config_physical.GetBackupConfigService(), + storages.GetStorageService(), + notifiers.GetNotifierService(), + tasks_cancellation.GetTaskCancelManager(), + encryption_secrets.GetSecretKeyService(), + logger.GetLogger(), + postgresql_executor.NewCreateFullBackupUsecase(), + postgresql_executor.NewCreateIncrementalBackupUsecase(), +} + +func GetPhysicalBackuper() *PhysicalBackuper { return physicalBackuper } + +var physicalBackupsScheduler = &PhysicalBackupsScheduler{ + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetIncrementalBackupRepository(), + physical_repositories.GetInFlightBackupRepository(), + backups_config_physical.GetBackupConfigService(), + chain_view.GetChainViewService(), + tasks_cancellation.GetTaskCancelManager(), + physicalBackuper, + atomicTime{}, + logger.GetLogger(), + atomic.Bool{}, + atomic.Bool{}, +} + +func GetPhysicalBackupsScheduler() *PhysicalBackupsScheduler { return physicalBackupsScheduler } + +var physicalBackupCleaner = &PhysicalBackupCleaner{ + physical_service.GetPhysicalBackupService(), + chain_view.GetChainViewService(), + backups_config_physical.GetBackupConfigService(), + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetWalSegmentRepository(), + logger.GetLogger(), + atomic.Bool{}, +} + +func GetPhysicalBackupCleaner() *PhysicalBackupCleaner { return physicalBackupCleaner } + +var physicalWalStreamSupervisor = &PhysicalWalStreamSupervisor{ + databases.GetDatabaseService(), + backups_config_physical.GetBackupConfigService(), + storages.GetStorageService(), + physical_repositories.GetWalSegmentRepository(), + physical_repositories.GetWalHistoryRepository(), + physical_repositories.GetWalStreamerRepository(), + notifiers.GetNotifierService(), + tasks_cancellation.GetTaskCancelManager(), + encryption_secrets.GetSecretKeyService(), + encryption.GetFieldEncryptor(), + logger.GetLogger(), + sync.Mutex{}, + make(map[uuid.UUID]*runningStreamer), + atomicTime{}, + atomic.Bool{}, + atomic.Bool{}, +} + +func GetPhysicalWalStreamSupervisor() *PhysicalWalStreamSupervisor { + return physicalWalStreamSupervisor +} + +var physicalSlotCleanupListener = postgresql_executor.NewPhysicalSlotCleanupListener( + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + logger.GetLogger(), +) + +var physicalBackupCanceller = NewPhysicalBackupCanceller( + physical_repositories.GetInFlightBackupRepository(), + tasks_cancellation.GetTaskCancelManager(), + logger.GetLogger(), +) + +func GetPhysicalBackupCanceller() *PhysicalBackupCanceller { return physicalBackupCanceller } + +var physicalBackupCancellationListener = &PhysicalBackupCancellationListener{ + physicalBackupCanceller, + physical_repositories.GetWalStreamerRepository(), + tasks_cancellation.GetTaskCancelManager(), + logger.GetLogger(), +} + +var SetupDependencies = sync.OnceFunc(func() { + // Order matters: the cancellation listener stops the local pg_receivewal and + // deletes the streamer row first, so the slot cleanup listener that runs next + // can drop the (now detaching) WAL slot instead of refusing it as active and + // leaving it to pin WAL forever. + databases.GetDatabaseService().AddDbRemoveListener(physicalBackupCancellationListener) + databases.GetDatabaseService().AddDbRemoveListener(physicalSlotCleanupListener) + backups_config_physical.GetBackupConfigService().SetBackupConfigChangeListener(physicalBackupCancellationListener) +}) diff --git a/backend/internal/features/backups/backups/backuping/physical/dto.go b/backend/internal/features/backups/backups/backuping/physical/dto.go new file mode 100644 index 0000000..be50f13 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/dto.go @@ -0,0 +1,24 @@ +package backuping_physical + +import ( + "time" + + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/storages" +) + +type backupContext struct { + Config *backups_config_physical.PhysicalBackupConfig + Database *databases.Database + Storage *storages.Storage + MasterKey string +} + +// chainCandidate pairs a non-extendable chain with its end timestamp so passes +// can order by recency without recomputing it. +type chainCandidate struct { + view *chain_view.ChainView + endTs time.Time +} diff --git a/backend/internal/features/backups/backups/backuping/physical/health.go b/backend/internal/features/backups/backups/backuping/physical/health.go new file mode 100644 index 0000000..869ebee --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/health.go @@ -0,0 +1,25 @@ +package backuping_physical + +import ( + "sync/atomic" + "time" +) + +// atomicTime is a race-free holder for a time.Time that one background loop +// writes (on every tick) and a health-check goroutine reads concurrently. A +// bare time.Time field is a multi-word struct, so the unsynchronized +// write/read of lastTickTime is a data race; storing the value as unix-nanos in +// an atomic.Int64 makes both sides race-free. The zero value reads back as the +// Unix epoch, so a never-ticked holder reports "unhealthy" — matching the old +// zero-value time.Time semantics. +type atomicTime struct { + unixNano atomic.Int64 +} + +func (a *atomicTime) Store(t time.Time) { + a.unixNano.Store(t.UnixNano()) +} + +func (a *atomicTime) Load() time.Time { + return time.Unix(0, a.unixNano.Load()).UTC() +} diff --git a/backend/internal/features/backups/backups/backuping/physical/interfaces.go b/backend/internal/features/backups/backups/backuping/physical/interfaces.go new file mode 100644 index 0000000..af1ccac --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/interfaces.go @@ -0,0 +1,26 @@ +package backuping_physical + +import ( + "context" + + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + "databasus-backend/internal/features/notifiers" +) + +type NotificationSender interface { + SendNotification(notifier *notifiers.Notifier, title, message string) +} + +type FullBackupExecutor interface { + Execute( + ctx context.Context, + spec postgresql_executor.FullBackupSpec, + ) (postgresql_executor.PhysicalBackupResult, error) +} + +type IncrementalBackupExecutor interface { + Execute( + ctx context.Context, + spec postgresql_executor.IncrementalBackupSpec, + ) (postgresql_executor.PhysicalBackupResult, error) +} diff --git a/backend/internal/features/backups/backups/backuping/physical/mocks_test.go b/backend/internal/features/backups/backups/backuping/physical/mocks_test.go new file mode 100644 index 0000000..516b145 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/mocks_test.go @@ -0,0 +1,146 @@ +package backuping_physical + +import ( + "context" + "testing" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" + + backuping_logical "databasus-backend/internal/features/backups/backups/backuping/logical" + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + physical_testing "databasus-backend/internal/features/backups/backups/core/physical/testing" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/intervals" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_enums "databasus-backend/internal/features/users/enums" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_models "databasus-backend/internal/features/workspaces/models" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" +) + +// fakeFullExecutor stands in for the real pg_basebackup-driven CreateFullBackupUsecase so +// orchestrator tests can pin the executor's outcome without a live cluster or +// object storage. The fake never touches storage, so these tests stay hermetic. +type fakeFullExecutor struct { + result postgresql_executor.PhysicalBackupResult + err error + callCount int +} + +func (e *fakeFullExecutor) Execute( + _ context.Context, + _ postgresql_executor.FullBackupSpec, +) (postgresql_executor.PhysicalBackupResult, error) { + e.callCount++ + + return e.result, e.err +} + +type fakeIncrementalExecutor struct { + result postgresql_executor.PhysicalBackupResult + err error + callCount int +} + +func (e *fakeIncrementalExecutor) Execute( + _ context.Context, + _ postgresql_executor.IncrementalBackupSpec, +) (postgresql_executor.PhysicalBackupResult, error) { + e.callCount++ + + return e.result, e.err +} + +type sentNotification struct { + Notifier *notifiers.Notifier + Title string + Message string +} + +// recordingNotificationSender captures every dispatched notification so tests +// can assert on count (fan-out), routing (which notifier), and content. +type recordingNotificationSender struct { + sentNotifications []sentNotification +} + +func (s *recordingNotificationSender) SendNotification( + notifier *notifiers.Notifier, + title string, + message string, +) { + s.sentNotifications = append(s.sentNotifications, sentNotification{ + Notifier: notifier, + Title: title, + Message: message, + }) +} + +// backupPrereqs is everything loadBackupContext needs to resolve: a workspace, +// a storage, a notifier, a POSTGRES_PHYSICAL database, and a saved backup +// config pointing at that storage. Tests seed their own backup rows against +// DB.ID / Storage.ID. +type backupPrereqs struct { + Workspace *workspaces_models.Workspace + Storage *storages.Storage + Notifier *notifiers.Notifier + DB *databases.Database + Config *backups_config_physical.PhysicalBackupConfig +} + +// seedBackupPrereqs builds the loadBackupContext prerequisites hermetically. +// It mirrors postgresql.SetupPhysicalDBForBackupVersion but deliberately skips +// PopulateDbData (no live PG) and the backup row / in-flight claim (tests own +// those). Cleanup deletes the physical catalog before the database is removed. +func seedBackupPrereqs(t *testing.T) *backupPrereqs { + t.Helper() + + router := backuping_logical.CreateTestRouter() + + owner := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + + workspace := workspaces_testing.CreateTestWorkspace("ws "+uuid.New().String(), owner, router) + t.Cleanup(func() { workspaces_testing.RemoveTestWorkspace(workspace, router) }) + + testStorage := storages.CreateTestStorage(workspace.ID) + t.Cleanup(func() { storages.RemoveTestStorage(testStorage.ID) }) + + notifier := notifiers.CreateTestNotifier(workspace.ID) + t.Cleanup(func() { notifiers.RemoveTestNotifier(notifier) }) + + db := databases.CreateTestPhysicalPostgresDatabase(workspace.ID, notifier, "17") + t.Cleanup(func() { databases.RemoveTestDatabase(db) }) + + cfgService := backups_config_physical.GetBackupConfigService() + + cfg, err := cfgService.GetBackupConfigByDbId(db.ID) + require.NoError(t, err) + + timeOfDay := "04:00" + + cfg.IsBackupsEnabled = true + cfg.StorageID = &testStorage.ID + cfg.Storage = testStorage + cfg.Encryption = backups_core_enums.BackupEncryptionNone + cfg.PostgresqlPhysical = db.PostgresqlPhysical + cfg.FullBackupInterval = intervals.Interval{ + Type: intervals.IntervalDaily, + TimeOfDay: &timeOfDay, + } + + savedConfig, err := cfgService.SaveBackupConfig(cfg) + require.NoError(t, err) + + t.Cleanup(func() { physical_testing.DeleteAllPhysicalCatalogForDatabase(t, db.ID) }) + + return &backupPrereqs{ + Workspace: workspace, + Storage: testStorage, + Notifier: notifier, + DB: db, + Config: savedConfig, + } +} diff --git a/backend/internal/features/backups/backups/backuping/physical/scheduler.go b/backend/internal/features/backups/backups/backuping/physical/scheduler.go new file mode 100644 index 0000000..f618265 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/scheduler.go @@ -0,0 +1,553 @@ +package backuping_physical + +import ( + "context" + "fmt" + "log/slog" + "sync/atomic" + "time" + + "github.com/google/uuid" + "gorm.io/gorm" + + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + postgresql_physical "databasus-backend/internal/features/databases/databases/postgresql/physical" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + "databasus-backend/internal/storage" +) + +// PhysicalBackupsScheduler drives the FULL/INCR decision, the cross-table +// single-in-flight claim + typed-row INSERT, and the in-process hand-off to the +// PhysicalBackuper. It is the "front half"; the backuper writes terminal +// status and releases the claim. +// +// Retry policy (no IsRetryIfFailed/MaxFailedTriesCount columns exist on the +// physical config, so retry is derived from chain state, not a counter): +// - ERROR: the chain stays extendable, so the next cadence-due tick re-attempts +// the SAME kind. A freshly-failed attempt is the newest row of its kind, so +// the cadence check on its created_at prevents a tight retry loop. +// - CHAIN_BROKEN: no extendable chain remains, so the next tick opens a new FULL. +// - CANCELED: neither COMPLETED nor CHAIN_BROKEN; the chain it belonged to stays +// extendable and resumes on cadence — no immediate auto-retry. +type PhysicalBackupsScheduler struct { + fullRepo *physical_repositories.PhysicalFullBackupRepository + incrRepo *physical_repositories.PhysicalIncrementalBackupRepository + inFlightRepo *physical_repositories.PhysicalInFlightBackupRepository + backupConfigService *backups_config_physical.BackupConfigService + chainViewService *chain_view.ChainViewService + taskCancelManager *tasks_cancellation.TaskCancelManager + backuper *PhysicalBackuper + + lastTickTime atomicTime + logger *slog.Logger + + hasRun atomic.Bool + isReady atomic.Bool +} + +// IsRunning reports whether the scheduler has finished startup recovery and is +// ready to dispatch work. Tests gate on it. +func (s *PhysicalBackupsScheduler) IsRunning() bool { + return s.isReady.Load() +} + +func (s *PhysicalBackupsScheduler) IsSchedulerRunning() bool { + return s.lastTickTime.Load().After(time.Now().UTC().Add(-schedulerHealthcheckThreshold)) +} + +func (s *PhysicalBackupsScheduler) Run(ctx context.Context) { + if s.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", s)) + } + + s.logger = s.logger.With("job_id", uuid.New(), "job_name", schedulerJobName) + + s.lastTickTime.Store(time.Now().UTC()) + + if err := s.recoverInFlightBackupsOnRestart(); err != nil { + s.logger.Error("failed to recover in-flight physical backups on restart", "error", err) + + panic(err) + } + + s.isReady.Store(true) + defer s.isReady.Store(false) + + if ctx.Err() != nil { + return + } + + ticker := time.NewTicker(schedulerTickInterval) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + if err := s.runPendingBackups(); err != nil { + s.logger.Error("failed to run pending physical backups", "error", err) + } + + s.lastTickTime.Store(time.Now().UTC()) + } + } +} + +func (s *PhysicalBackupsScheduler) runPendingBackups() error { + enabledConfigs, err := s.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + return err + } + + now := time.Now().UTC() + + for _, backupConfig := range enabledConfigs { + s.evaluateConfig(now, backupConfig) + } + + return nil +} + +func (s *PhysicalBackupsScheduler) evaluateConfig( + now time.Time, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) { + logger := s.logger.With("database_id", backupConfig.DatabaseID, "job_name", schedulerJobName) + + if backupConfig.StorageID == nil { + logger.Error("physical backup config has no storage id; skipping") + + return + } + + decision, ok := s.decideBackupKind(logger, now, backupConfig) + if !ok { + return + } + + s.scheduleBackup(logger, backupConfig, decision) +} + +// backupDecision is the outcome of the per-tick FULL-vs-INCR decision. +type backupDecision struct { + kind physical_enums.PhysicalBackupType + incrRootFullBackupID uuid.UUID + incrParentIncrID *uuid.UUID + forceFullRequestedAt *time.Time + forceIncrementalRequestedAt *time.Time +} + +// decideBackupKind picks FULL or INCR purely from catalog state + cadence (no +// source-PG connection): FULL when its interval is due (covers bootstrap, chain +// rotation, and post-CHAIN_BROKEN re-anchor since no extendable chain exists); +// INCR when incrementals are enabled, an extendable chain exists, and the INCR +// interval is due. The shipped executors handle summarizer/timeline reality at +// run time, returning CHAIN_BROKEN when an INCR cannot actually proceed. +func (s *PhysicalBackupsScheduler) decideBackupKind( + logger *slog.Logger, + now time.Time, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) (backupDecision, bool) { + lastFull, err := s.fullRepo.FindLastFullAnyStatusByDatabase(backupConfig.DatabaseID) + if err != nil { + logger.Error("failed to find last full backup", "error", err) + + return backupDecision{}, false + } + + if backupConfig.ForceFullRequestedAt != nil { + return backupDecision{ + kind: physical_enums.PhysicalBackupTypeFull, + forceFullRequestedAt: backupConfig.ForceFullRequestedAt, + }, true + } + + if backupConfig.ForceIncrementalRequestedAt != nil { + if decision, ok := s.decideForcedIncremental(logger, backupConfig); ok { + return decision, true + } + } + + lastIncr, err := s.incrRepo.FindLastByDatabase(backupConfig.DatabaseID) + if err != nil { + logger.Error("failed to find last incremental backup", "error", err) + + return backupDecision{}, false + } + + if backupConfig.FullBackupInterval.ShouldTriggerBackup(now, createdAtOrNil(lastFull)) { + return backupDecision{kind: physical_enums.PhysicalBackupTypeFull}, true + } + + if !isIncrementalEnabled(backupConfig) { + return backupDecision{}, false + } + + extendableChain, err := s.chainViewService.FindLastExtendableChainByDatabase(backupConfig.DatabaseID) + if err != nil { + logger.Error("failed to find extendable chain", "error", err) + + return backupDecision{}, false + } + + lastBackupTime := newestCreatedAt(lastFull, lastIncr) + + if extendableChain == nil || + !backupConfig.IncrementalBackupInterval.ShouldTriggerBackup(now, lastBackupTime) { + return backupDecision{}, false + } + + parentIncrID, err := s.resolveIncrParent(extendableChain.RootFull.ID) + if err != nil { + logger.Error("failed to resolve incremental parent", "error", err) + + return backupDecision{}, false + } + + return backupDecision{ + kind: physical_enums.PhysicalBackupTypeIncremental, + incrRootFullBackupID: extendableChain.RootFull.ID, + incrParentIncrID: parentIncrID, + }, true +} + +// decideForcedIncremental honors an out-of-cadence incremental request. It can +// only be satisfied when incrementals are enabled and an extendable chain +// exists; otherwise the stale request is cleared (so it can't loop forever) and +// ok=false lets the caller fall through to normal cadence evaluation. The +// controller already rejects "incremental with no chain" up front, but the +// chain can disappear between request and tick, so the scheduler must cope. +func (s *PhysicalBackupsScheduler) decideForcedIncremental( + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) (backupDecision, bool) { + if !isIncrementalEnabled(backupConfig) { + s.clearIncrementalRequest(logger, backupConfig) + + return backupDecision{}, false + } + + extendableChain, err := s.chainViewService.FindLastExtendableChainByDatabase(backupConfig.DatabaseID) + if err != nil { + logger.Error("failed to find extendable chain for forced incremental", "error", err) + + return backupDecision{}, false + } + + if extendableChain == nil { + logger.Warn("forced incremental requested but no extendable chain exists; clearing request") + s.clearIncrementalRequest(logger, backupConfig) + + return backupDecision{}, false + } + + parentIncrID, err := s.resolveIncrParent(extendableChain.RootFull.ID) + if err != nil { + logger.Error("failed to resolve incremental parent for forced incremental", "error", err) + + return backupDecision{}, false + } + + return backupDecision{ + kind: physical_enums.PhysicalBackupTypeIncremental, + incrRootFullBackupID: extendableChain.RootFull.ID, + incrParentIncrID: parentIncrID, + forceIncrementalRequestedAt: backupConfig.ForceIncrementalRequestedAt, + }, true +} + +func (s *PhysicalBackupsScheduler) clearIncrementalRequest( + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) { + if err := s.backupConfigService.ClearIncrementalBackupRequest( + backupConfig.DatabaseID, backupConfig.ForceIncrementalRequestedAt, + ); err != nil { + logger.Error("failed to clear incremental backup request", "error", err) + } +} + +// resolveIncrParent returns the latest COMPLETED INCR in the chain as the new +// INCR's parent, or nil when the chain has none yet (then the parent is the +// FULL, resolved from root_full_backup_id at read time). +func (s *PhysicalBackupsScheduler) resolveIncrParent(rootFullBackupID uuid.UUID) (*uuid.UUID, error) { + parent, err := s.incrRepo.FindLatestCompletedByRootFull(rootFullBackupID) + if err != nil { + return nil, err + } + + if parent == nil { + return nil, nil + } + + return &parent.ID, nil +} + +func (s *PhysicalBackupsScheduler) scheduleBackup( + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, + decision backupDecision, +) { + backupID := uuid.New() + logger = logger.With("backup_id", backupID, "backup_kind", decision.kind) + + claimed, err := s.claimAndInsert(backupConfig, backupID, decision) + if err != nil { + logger.Error("failed to claim and insert backup row", "error", err) + + return + } + + if !claimed { + logger.Debug("in-flight slot already claimed by another instance; skipping") + + return + } + + if decision.forceFullRequestedAt != nil { + if err := s.backupConfigService.ClearFullBackupRequest( + backupConfig.DatabaseID, + decision.forceFullRequestedAt, + ); err != nil { + logger.Error("failed to clear forced full request", "error", err) + } + } + + if decision.forceIncrementalRequestedAt != nil { + if err := s.backupConfigService.ClearIncrementalBackupRequest( + backupConfig.DatabaseID, + decision.forceIncrementalRequestedAt, + ); err != nil { + logger.Error("failed to clear forced incremental request", "error", err) + } + } + + go s.backuper.MakeBackup(backupID, true) + + logger.Info("scheduled physical backup") +} + +// claimAndInsert reserves the cross-table in-flight slot and inserts the typed +// IN_PROGRESS row in ONE transaction. Both use the tx handle so they commit +// together; a lost claim (another instance won) commits nothing and returns +// claimed=false. The typed INSERT goes through tx.Create, NOT repo.Save (which +// uses the global DB) — that is what makes the claim and the row atomic. +func (s *PhysicalBackupsScheduler) claimAndInsert( + backupConfig *backups_config_physical.PhysicalBackupConfig, + backupID uuid.UUID, + decision backupDecision, +) (bool, error) { + now := time.Now().UTC() + claimed := false + + txErr := storage.GetDb().Transaction(func(tx *gorm.DB) error { + ok, claimErr := s.inFlightRepo.Claim(tx, physical_repositories.ClaimSpec{ + DatabaseID: backupConfig.DatabaseID, + BackupType: decision.kind, + BackupID: backupID, + }) + if claimErr != nil { + return claimErr + } + + if !ok { + return nil + } + + claimed = true + + if decision.kind == physical_enums.PhysicalBackupTypeFull { + return tx.Create(&physical_models.PhysicalFullBackup{ + ID: backupID, + DatabaseID: backupConfig.DatabaseID, + StorageID: *backupConfig.StorageID, + Status: physical_enums.PhysicalBackupStatusInProgress, + CreatedAt: now, + }).Error + } + + return tx.Create(&physical_models.PhysicalIncrementalBackup{ + ID: backupID, + DatabaseID: backupConfig.DatabaseID, + StorageID: *backupConfig.StorageID, + RootFullBackupID: decision.incrRootFullBackupID, + ParentIncrementalBackupID: decision.incrParentIncrID, + Status: physical_enums.PhysicalBackupStatusInProgress, + CreatedAt: now, + }).Error + }) + if txErr != nil { + return false, txErr + } + + return claimed, nil +} + +// recoverInFlightBackupsOnRestart reconciles the IN_PROGRESS backups the DB still +// holds when this instance (re)started. In single-instance mode the goroutine that +// was running each backup died with the previous process, so every in-flight claim +// is orphaned: fail the row and release the claim, freeing the database's +// single-in-flight slot for a fresh attempt. +// +// Runs once at Run() entry. The claim is the source of truth for in-flight state — +// claimAndInsert writes the claim and the typed row in one transaction — so +// iterating claims covers every backup that was running. +func (s *PhysicalBackupsScheduler) recoverInFlightBackupsOnRestart() error { + claims, err := s.inFlightRepo.FindAll() + if err != nil { + return err + } + + claimedBackupIDs := make(map[uuid.UUID]struct{}, len(claims)) + + for _, claim := range claims { + claimedBackupIDs[claim.BackupID] = struct{}{} + + s.failOrphanedBackup(claim.BackupType, claim.BackupID, claim.DatabaseID) + } + + return s.failClaimlessInProgressBackups(claimedBackupIDs) +} + +// failClaimlessInProgressBackups fails any IN_PROGRESS row that has no matching +// in-flight claim. The atomic claim+insert should never leave such a row, so this +// is a defensive sweep: without it a stray row could sit IN_PROGRESS forever after +// a restart, since claim-driven recovery would never see it. +func (s *PhysicalBackupsScheduler) failClaimlessInProgressBackups(claimedBackupIDs map[uuid.UUID]struct{}) error { + fulls, err := s.fullRepo.FindAllInProgress() + if err != nil { + return err + } + + for _, full := range fulls { + if _, ok := claimedBackupIDs[full.ID]; ok { + continue + } + + s.failOrphanedBackup(physical_enums.PhysicalBackupTypeFull, full.ID, full.DatabaseID) + } + + incrementals, err := s.incrRepo.FindAllInProgress() + if err != nil { + return err + } + + for _, incremental := range incrementals { + if _, ok := claimedBackupIDs[incremental.ID]; ok { + continue + } + + s.failOrphanedBackup(physical_enums.PhysicalBackupTypeIncremental, incremental.ID, incremental.DatabaseID) + } + + return nil +} + +func (s *PhysicalBackupsScheduler) failOrphanedBackup( + kind physical_enums.PhysicalBackupType, + backupID, databaseID uuid.UUID, +) { + // Best-effort cancel of any locally-registered task; harmless if none + // (a fresh process holds no registrations). + if err := s.taskCancelManager.CancelTask(backupID); err != nil { + s.logger.Error("failed to cancel orphaned backup task", "backup_id", backupID, "error", err) + } + + if err := s.failBackupAndReleaseClaim( + kind, backupID, databaseID, physical_enums.PhysicalBackupErrorApplicationRestart, + "Backup was interrupted by an application restart and marked failed. Trigger a new backup.", + ); err != nil { + s.logger.Error("failed to fail orphaned backup on restart", "backup_id", backupID, "error", err) + } +} + +// failBackupAndReleaseClaim flips one typed row to ERROR and deletes the +// matching in-flight claim in the SAME transaction. Pairing the two is +// essential: an orphan claim left behind would block every future tick from +// acquiring the cross-table single-in-flight slot for that DB, freezing it. +// +// It deliberately does NOT touch storage. A row reaches this path only while +// IN_PROGRESS, and claimAndInsert inserts FULL/INCR rows with file_name = NULL — +// a name is written only at COMPLETED. A NULL file_name is proof no object was +// ever uploaded under any name, so there is nothing to delete (the same reasoning +// as PhysicalWalSegmentRepository.DeleteAbandonedClaims). This MUST be revisited +// if physical FULL ever starts writing file_name at upload start: at that point a +// rolled-back row could reference a partial object that needs storage cleanup +// before the status flip. +func (s *PhysicalBackupsScheduler) failBackupAndReleaseClaim( + kind physical_enums.PhysicalBackupType, + backupID, databaseID uuid.UUID, + reason physical_enums.PhysicalBackupErrorReason, + message string, +) error { + var typedModel any = &physical_models.PhysicalFullBackup{} + if kind == physical_enums.PhysicalBackupTypeIncremental { + typedModel = &physical_models.PhysicalIncrementalBackup{} + } + + return storage.GetDb().Transaction(func(tx *gorm.DB) error { + // Guard on IN_PROGRESS so a late completion that already wrote a terminal + // status wins the race — the backuper's persist and this sweep both + // conditionally transition the row, so at most one of them takes effect. + if err := tx.Model(typedModel). + Where("id = ? AND status = ?", backupID, physical_enums.PhysicalBackupStatusInProgress). + Updates(map[string]any{ + "status": physical_enums.PhysicalBackupStatusError, + "error_reason": reason, + "fail_message": message, + }).Error; err != nil { + return err + } + + // Scope the claim delete to backup_id so failing a stale backup cannot + // remove a newer backup's claim for the same database. + return tx.Delete( + &physical_models.PhysicalInFlightBackup{}, + "database_id = ? AND backup_id = ?", + databaseID, + backupID, + ).Error + }) +} + +func createdAtOrNil(full *physical_models.PhysicalFullBackup) *time.Time { + if full == nil { + return nil + } + + return &full.CreatedAt +} + +func newestCreatedAt( + full *physical_models.PhysicalFullBackup, + incr *physical_models.PhysicalIncrementalBackup, +) *time.Time { + switch { + case full == nil && incr == nil: + return nil + case incr == nil: + return &full.CreatedAt + case full == nil: + return &incr.CreatedAt + case incr.CreatedAt.After(full.CreatedAt): + return &incr.CreatedAt + default: + return &full.CreatedAt + } +} + +func isIncrementalEnabled(backupConfig *backups_config_physical.PhysicalBackupConfig) bool { + if backupConfig.PostgresqlPhysical == nil { + return false + } + + backupType := backupConfig.PostgresqlPhysical.BackupType + + return backupType == postgresql_physical.BackupTypeFullAndIncremental || + backupType == postgresql_physical.BackupTypeFullIncrementalAndWalStream +} diff --git a/backend/internal/features/backups/backups/backuping/physical/scheduler_test.go b/backend/internal/features/backups/backups/backuping/physical/scheduler_test.go new file mode 100644 index 0000000..deed0bc --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/scheduler_test.go @@ -0,0 +1,583 @@ +package backuping_physical + +import ( + "context" + "sync" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_testing "databasus-backend/internal/features/backups/backups/core/physical/testing" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + postgresql_physical "databasus-backend/internal/features/databases/databases/postgresql/physical" + "databasus-backend/internal/features/intervals" + "databasus-backend/internal/storage" + "databasus-backend/internal/util/logger" + "databasus-backend/internal/util/walmath" +) + +const testSegmentMB = 16 + +func testLSN(segments int) walmath.LSN { + return walmath.LSN(segments * testSegmentMB * 1024 * 1024) +} + +// makeIncrementalDue mutates the in-memory config so the FULL interval is not +// due (generic weekly) but the INCR interval is (hourly) and enables INCRs. +func makeIncrementalDue(prereqs *backupPrereqs) { + prereqs.Config.PostgresqlPhysical.BackupType = postgresql_physical.BackupTypeFullAndIncremental + prereqs.Config.FullBackupInterval = intervals.Interval{Type: intervals.IntervalWeekly} + prereqs.Config.IncrementalBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} +} + +// seedFullWithStatusAndAge persists a FULL of the given status, aged by ageHours, +// so the cadence / chain-state decision tests are deterministic. Non-COMPLETED +// fulls clear CompletedAt — only COMPLETED fulls anchor an extendable chain. +func seedFullWithStatusAndAge( + t *testing.T, + prereqs *backupPrereqs, + status physical_enums.PhysicalBackupStatus, + startSegment, ageHours int, +) *physical_models.PhysicalFullBackup { + t.Helper() + + full := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, prereqs.Storage.ID, 1, testLSN(startSegment), testLSN(startSegment+1)) + + full.CreatedAt = time.Now().UTC().Add(-time.Duration(ageHours) * time.Hour) + full.Status = status + + if status != physical_enums.PhysicalBackupStatusCompleted { + full.CompletedAt = nil + } + + return physical_testing.CreateTestFullBackup(t, full) +} + +// seedIncrWithStatusAndAge persists an INCR of the given status on a chain root, +// aged by ageHours. Used to assert how a terminal incr status steers the next +// decision (ERROR/CANCELED keep the chain extendable; CHAIN_BROKEN does not). +func seedIncrWithStatusAndAge( + t *testing.T, + prereqs *backupPrereqs, + rootFullBackupID uuid.UUID, + status physical_enums.PhysicalBackupStatus, + startSegment, ageHours int, +) *physical_models.PhysicalIncrementalBackup { + t.Helper() + + incr := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.DB.ID, prereqs.Storage.ID, rootFullBackupID, nil, 1, testLSN(startSegment), testLSN(startSegment+1)) + + incr.CreatedAt = time.Now().UTC().Add(-time.Duration(ageHours) * time.Hour) + incr.Status = status + + if status != physical_enums.PhysicalBackupStatusCompleted { + incr.CompletedAt = nil + } + + return physical_testing.CreateTestIncrementalBackup(t, incr) +} + +func Test_DecideBackupKind_WhenRecentlyCompletedFullAndIncrNotDue_SchedulesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) // full weekly, incr hourly, INCR enabled + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 0) + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "a just-completed chain with neither cadence due schedules nothing") +} + +func Test_DecideBackupKind_WhenLastFullErroredAndFullCadenceNotDue_SchedulesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + prereqs.Config.FullBackupInterval = intervals.Interval{Type: intervals.IntervalWeekly} + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusError, 1, 2) + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "a freshly errored full is not retried before its cadence — no tight retry loop") +} + +func Test_DecideBackupKind_WhenLastFullErroredAndFullCadenceDue_RetriesFull(t *testing.T) { + prereqs := seedBackupPrereqs(t) + prereqs.Config.FullBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusError, 1, 2) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind, + "ERROR is transient: the next cadence-due tick retries the same kind") +} + +func Test_DecideBackupKind_WhenLastIncrErroredAndChainExtendable_RetriesIncremental(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + full := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 3) + seedIncrWithStatusAndAge(t, prereqs, full.ID, physical_enums.PhysicalBackupStatusError, 2, 2) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeIncremental, decision.kind, + "an ERROR incr keeps the chain extendable, so the retry stays an INCR") + assert.Equal(t, full.ID, decision.incrRootFullBackupID) + assert.Nil(t, decision.incrParentIncrID, "an errored incr is not a COMPLETED parent") +} + +func Test_DecideBackupKind_WhenChainBrokenAndFullCadenceDue_ReAnchorsWithFull(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + prereqs.Config.FullBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} + full := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 2) + seedIncrWithStatusAndAge(t, prereqs, full.ID, physical_enums.PhysicalBackupStatusChainBroken, 2, 1) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind, + "a CHAIN_BROKEN chain re-anchors with a new FULL, never an INCR") +} + +func Test_DecideBackupKind_WhenChainBrokenAndIncrDueButFullNotDue_SchedulesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) // full weekly (not due), incr hourly + full := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 3) + seedIncrWithStatusAndAge(t, prereqs, full.ID, physical_enums.PhysicalBackupStatusChainBroken, 2, 2) + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "a broken chain never spawns an INCR; it waits for the FULL cadence") +} + +func Test_DecideBackupKind_WhenCanceledIncrRecent_SchedulesNothing(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + full := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 3) + seedIncrWithStatusAndAge(t, prereqs, full.ID, physical_enums.PhysicalBackupStatusCanceled, 2, 0) + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "a just-canceled incr is not auto-retried; the chain resumes on cadence") +} + +func Test_DecideBackupKind_WhenFullAndIncrBothDue_ChoosesFull(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) // full weekly, incr hourly + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 24*8) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind, + "when both cadences are due the same tick, FULL takes precedence") +} + +func Test_DecideBackupKind_WhenNoFullExists_ChoosesFull(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind) +} + +func Test_DecideBackupKind_WhenExtendableChainAndIncrDue_ChoosesIncrementalWithFullParent(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + + fullModel := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, + prereqs.Storage.ID, + 1, + testLSN(0), + testLSN(1), + ) + fullModel.CreatedAt = time.Now().UTC().Add(-2 * time.Hour) + full := physical_testing.CreateTestFullBackup(t, fullModel) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeIncremental, decision.kind) + assert.Equal(t, full.ID, decision.incrRootFullBackupID) + assert.Nil(t, decision.incrParentIncrID, "first INCR has no INCR parent — resolves to the FULL at read time") +} + +func Test_DecideBackupKind_WhenChainHasCompletedIncr_ChoosesIncrementalWithIncrParent(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + + fullModel := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, + prereqs.Storage.ID, + 1, + testLSN(0), + testLSN(1), + ) + fullModel.CreatedAt = time.Now().UTC().Add(-3 * time.Hour) + full := physical_testing.CreateTestFullBackup(t, fullModel) + + incrModel := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.DB.ID, prereqs.Storage.ID, full.ID, nil, 1, testLSN(1), testLSN(2)) + incrModel.CreatedAt = time.Now().UTC().Add(-2 * time.Hour) + incr := physical_testing.CreateTestIncrementalBackup(t, incrModel) + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeIncremental, decision.kind) + assert.Equal(t, full.ID, decision.incrRootFullBackupID) + require.NotNil(t, decision.incrParentIncrID) + assert.Equal(t, incr.ID, *decision.incrParentIncrID) +} + +func Test_DecideBackupKind_WhenFullOnlyConfig_NeverChoosesIncremental(t *testing.T) { + prereqs := seedBackupPrereqs(t) + // FULL_ONLY (default), FULL not due (generic weekly), INCR interval is set + // but must be ignored. + prereqs.Config.FullBackupInterval = intervals.Interval{Type: intervals.IntervalWeekly} + prereqs.Config.IncrementalBackupInterval = intervals.Interval{Type: intervals.IntervalHourly} + + fullModel := physical_testing.NewTestCompletedFullBackup( + prereqs.DB.ID, + prereqs.Storage.ID, + 1, + testLSN(0), + testLSN(1), + ) + fullModel.CreatedAt = time.Now().UTC().Add(-2 * time.Hour) + physical_testing.CreateTestFullBackup(t, fullModel) + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "FULL_ONLY config must never schedule an incremental") +} + +func Test_ClaimAndInsert_CreatesInProgressRowAndClaim(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + backupID := uuid.New() + + claimed, err := scheduler.claimAndInsert(prereqs.Config, backupID, + backupDecision{kind: physical_enums.PhysicalBackupTypeFull}) + + require.NoError(t, err) + assert.True(t, claimed) + + full, _ := physical_repositories.GetFullBackupRepository().FindByID(backupID) + require.NotNil(t, full) + assert.Equal(t, physical_enums.PhysicalBackupStatusInProgress, full.Status) + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + require.NotNil(t, claim) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, claim.BackupType) + assert.Equal(t, backupID, claim.BackupID) +} + +func Test_ClaimAndInsert_WhenConcurrentClaimSameDatabase_OnlyOneSucceeds(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + + results := make([]bool, 2) + backupIDs := []uuid.UUID{uuid.New(), uuid.New()} + + var waitGroup sync.WaitGroup + for i := range 2 { + waitGroup.Go(func() { + claimed, _ := scheduler.claimAndInsert(prereqs.Config, backupIDs[i], + backupDecision{kind: physical_enums.PhysicalBackupTypeFull}) + results[i] = claimed + }) + } + waitGroup.Wait() + + assert.True(t, results[0] != results[1], "exactly one of two concurrent claims must win") + + fulls, err := physical_repositories.GetFullBackupRepository().FindAllInProgress() + require.NoError(t, err) + + inProgressForDB := 0 + for _, full := range fulls { + if full.DatabaseID == prereqs.DB.ID { + inProgressForDB++ + } + } + assert.Equal(t, 1, inProgressForDB, "only the winning claim inserts a typed row") +} + +func Test_EvaluateConfig_WhenInFlightClaimExists_SkipsWithoutNewRow(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + + ok, err := physical_repositories.GetInFlightBackupRepository().Claim( + storage.GetDb(), physical_repositories.ClaimSpec{ + DatabaseID: prereqs.DB.ID, + BackupType: physical_enums.PhysicalBackupTypeFull, + BackupID: uuid.New(), + }) + require.NoError(t, err) + require.True(t, ok) + + scheduler.evaluateConfig(time.Now().UTC(), prereqs.Config) + + lastFull, _ := physical_repositories.GetFullBackupRepository().FindLastFullAnyStatusByDatabase(prereqs.DB.ID) + assert.Nil(t, lastFull, "the existing claim must block a new typed row this tick") +} + +func Test_DecideBackupKind_WhenFullRequested_ChoosesFullBeforeCadence(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 0) + + requestedAt := time.Now().UTC() + prereqs.Config.ForceFullRequestedAt = &requestedAt + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind) + require.NotNil(t, decision.forceFullRequestedAt) + assert.Equal(t, requestedAt, *decision.forceFullRequestedAt) +} + +func Test_DecideBackupKind_WhenIncrementalRequested_ChoosesIncrementalOutOfCadence(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) + // A just-completed full means the INCR cadence is NOT due; the forced request + // must override cadence and still pick an incremental on the existing chain. + full := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 0) + + requestedAt := time.Now().UTC() + prereqs.Config.ForceIncrementalRequestedAt = &requestedAt + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeIncremental, decision.kind) + assert.Equal(t, full.ID, decision.incrRootFullBackupID) + require.NotNil(t, decision.forceIncrementalRequestedAt) + assert.Equal(t, requestedAt, *decision.forceIncrementalRequestedAt) +} + +func Test_DecideBackupKind_WhenIncrementalRequestedButNoExtendableChain_ClearsRequestAndFallsThrough(t *testing.T) { + prereqs := seedBackupPrereqs(t) + makeIncrementalDue(prereqs) // incrementals enabled, but no full backup exists yet + + require.NoError(t, backups_config_physical.GetBackupConfigService().RequestIncrementalBackupNow(prereqs.DB.ID)) + requested, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, requested.ForceIncrementalRequestedAt) + prereqs.Config.ForceIncrementalRequestedAt = requested.ForceIncrementalRequestedAt + + scheduler := CreateTestPhysicalScheduler() + + decision, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + require.True(t, ok) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, decision.kind, + "with no chain the forced incremental can't run, so the decision falls through to a bootstrap full") + + cleared, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.DB.ID) + require.NoError(t, err) + assert.Nil(t, cleared.ForceIncrementalRequestedAt, + "an unsatisfiable forced incremental must be cleared so it can't loop forever") +} + +func Test_DecideBackupKind_WhenIncrementalRequestedButIncrementalsDisabled_ClearsRequest(t *testing.T) { + prereqs := seedBackupPrereqs(t) + // Default config is FULL_ONLY (incrementals disabled). A completed full means a + // chain exists, isolating the "disabled" clear branch from the "no chain" one. + seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 1, 0) + + require.NoError(t, backups_config_physical.GetBackupConfigService().RequestIncrementalBackupNow(prereqs.DB.ID)) + requested, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, requested.ForceIncrementalRequestedAt) + prereqs.Config.ForceIncrementalRequestedAt = requested.ForceIncrementalRequestedAt + + scheduler := CreateTestPhysicalScheduler() + + _, ok := scheduler.decideBackupKind(logger.GetLogger(), time.Now().UTC(), prereqs.Config) + + assert.False(t, ok, "a forced incremental on a FULL_ONLY config schedules nothing this tick") + + cleared, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.DB.ID) + require.NoError(t, err) + assert.Nil(t, cleared.ForceIncrementalRequestedAt, + "a forced incremental on a FULL_ONLY config must be cleared") +} + +func seedInProgressFullWithClaim(t *testing.T, prereqs *backupPrereqs) uuid.UUID { + t.Helper() + + backupID := uuid.New() + full := &physical_models.PhysicalFullBackup{ + ID: backupID, + DatabaseID: prereqs.DB.ID, + StorageID: prereqs.Storage.ID, + Status: physical_enums.PhysicalBackupStatusInProgress, + } + require.NoError(t, physical_repositories.GetFullBackupRepository().Save(full)) + + ok, err := physical_repositories.GetInFlightBackupRepository().Claim( + storage.GetDb(), physical_repositories.ClaimSpec{ + DatabaseID: prereqs.DB.ID, + BackupType: physical_enums.PhysicalBackupTypeFull, + BackupID: backupID, + }) + require.NoError(t, err) + require.True(t, ok) + + return backupID +} + +func Test_RecoverInFlightOnRestart_FailsAndReleasesClaim(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + backupID := seedInProgressFullWithClaim(t, prereqs) + + require.NoError(t, scheduler.recoverInFlightBackupsOnRestart()) + + got, _ := physical_repositories.GetFullBackupRepository().FindByID(backupID) + require.NotNil(t, got) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, got.Status) + require.NotNil(t, got.ErrorReason) + assert.Equal(t, physical_enums.PhysicalBackupErrorApplicationRestart, *got.ErrorReason) + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, claim, "a backup orphaned by restart must be failed and released") +} + +func Test_ClaimAndInsert_WhenConcurrentFullAndIncrSameDatabase_OnlyOneSucceeds(t *testing.T) { + prereqs := seedBackupPrereqs(t) + // The INCR claimer needs a real chain root to satisfy the FK; the FULL claimer + // needs nothing. The race is on the single cross-table in-flight slot. + rootFull := seedFullWithStatusAndAge(t, prereqs, physical_enums.PhysicalBackupStatusCompleted, 0, 1) + scheduler := CreateTestPhysicalScheduler() + + results := make([]bool, 2) + fullID := uuid.New() + incrID := uuid.New() + + var waitGroup sync.WaitGroup + waitGroup.Go(func() { + claimed, _ := scheduler.claimAndInsert(prereqs.Config, fullID, + backupDecision{kind: physical_enums.PhysicalBackupTypeFull}) + results[0] = claimed + }) + waitGroup.Go(func() { + claimed, _ := scheduler.claimAndInsert( + prereqs.Config, + incrID, + backupDecision{ + kind: physical_enums.PhysicalBackupTypeIncremental, + incrRootFullBackupID: rootFull.ID, + }, + ) + results[1] = claimed + }) + waitGroup.Wait() + + assert.True(t, results[0] != results[1], "exactly one of a concurrent FULL/INCR claim wins the slot") + + inProgressFulls, err := physical_repositories.GetFullBackupRepository().FindAllInProgress() + require.NoError(t, err) + inProgressIncrs, err := physical_repositories.GetIncrementalBackupRepository().FindAllInProgress() + require.NoError(t, err) + + inProgressForDB := 0 + for _, full := range inProgressFulls { + if full.DatabaseID == prereqs.DB.ID { + inProgressForDB++ + } + } + for _, incr := range inProgressIncrs { + if incr.DatabaseID == prereqs.DB.ID { + inProgressForDB++ + } + } + assert.Equal(t, 1, inProgressForDB, "the loser inserts no row in either typed table") +} + +func Test_EvaluateConfig_WhenStorageIDNil_SkipsWithoutScheduling(t *testing.T) { + prereqs := seedBackupPrereqs(t) + prereqs.Config.StorageID = nil + scheduler := CreateTestPhysicalScheduler() + + scheduler.evaluateConfig(time.Now().UTC(), prereqs.Config) + + lastFull, _ := physical_repositories.GetFullBackupRepository().FindLastFullAnyStatusByDatabase(prereqs.DB.ID) + assert.Nil(t, lastFull, "a config without a storage id must schedule nothing") + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, claim) +} + +func Test_SchedulerRun_WhenCalledTwice_Panics(t *testing.T) { + scheduler := CreateTestPhysicalScheduler() + + // An already-cancelled context lets the first Run run startup recovery, flip + // hasRun, then return at the ctx.Err() guard without entering the ticker loop. + cancelledCtx, cancel := context.WithCancel(context.Background()) + cancel() + scheduler.Run(cancelledCtx) + + assert.Panics(t, func() { scheduler.Run(context.Background()) }, + "a second Run on the same scheduler must panic via the hasRun guard") +} + +func Test_FailBackupAndReleaseClaim_WhenFullRolledBack_LeavesNoNamedArtifact(t *testing.T) { + prereqs := seedBackupPrereqs(t) + scheduler := CreateTestPhysicalScheduler() + backupID := seedInProgressFullWithClaim(t, prereqs) + + require.NoError(t, scheduler.failBackupAndReleaseClaim( + physical_enums.PhysicalBackupTypeFull, backupID, prereqs.DB.ID, + physical_enums.PhysicalBackupErrorApplicationRestart, "restart interrupted backup")) + + full, _ := physical_repositories.GetFullBackupRepository().FindByID(backupID) + require.NotNil(t, full) + assert.Equal(t, physical_enums.PhysicalBackupStatusError, full.Status) + require.NotNil(t, full.FailMessage) + assert.Equal(t, "restart interrupted backup", *full.FailMessage) + assert.Nil(t, full.FileName, + "a rolled-back FULL carries no file_name — the invariant that makes skipping storage cleanup safe") + + claim, _ := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.DB.ID) + assert.Nil(t, claim, "the in-flight claim must be released in the same tx as the status flip") +} diff --git a/backend/internal/features/backups/backups/backuping/physical/setup_test.go b/backend/internal/features/backups/backups/backuping/physical/setup_test.go new file mode 100644 index 0000000..c67adef --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/setup_test.go @@ -0,0 +1,16 @@ +package backuping_physical + +import ( + "os" + "testing" + + cache_utils "databasus-backend/internal/util/cache" +) + +func TestMain(m *testing.M) { + if err := cache_utils.ClearAllCache(); err != nil { + panic(err) + } + + os.Exit(m.Run()) +} diff --git a/backend/internal/features/backups/backups/backuping/physical/testing.go b/backend/internal/features/backups/backups/backuping/physical/testing.go new file mode 100644 index 0000000..d4c2caa --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/testing.go @@ -0,0 +1,196 @@ +package backuping_physical + +import ( + "context" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/google/uuid" + + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_service "databasus-backend/internal/features/backups/backups/core/physical/service" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + encryption_secrets "databasus-backend/internal/features/encryption/secrets" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + workspaces_services "databasus-backend/internal/features/workspaces/services" + "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/logger" +) + +// CreateTestPhysicalBackuper returns a fully wired PhysicalBackuper for +// tests. The notification sender is parameterized so tests that don't want +// to exercise the notifier stack can inject a counting / no-op stub. +// Pass nil to use the production notifier service. +func CreateTestPhysicalBackuper(notificationSender NotificationSender) *PhysicalBackuper { + sender := notificationSender + if sender == nil { + sender = notifiers.GetNotifierService() + } + + return &PhysicalBackuper{ + databases.GetDatabaseService(), + encryption.GetFieldEncryptor(), + workspaces_services.GetWorkspaceService(), + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetIncrementalBackupRepository(), + physical_repositories.GetInFlightBackupRepository(), + physical_repositories.GetWalHistoryRepository(), + backups_config_physical.GetBackupConfigService(), + storages.GetStorageService(), + sender, + tasks_cancellation.GetTaskCancelManager(), + encryption_secrets.GetSecretKeyService(), + logger.GetLogger(), + postgresql_executor.NewCreateFullBackupUsecase(), + postgresql_executor.NewCreateIncrementalBackupUsecase(), + } +} + +// CreateTestPhysicalScheduler returns a scheduler wired to the production repos +// and its own in-process backuper. +func CreateTestPhysicalScheduler() *PhysicalBackupsScheduler { + return &PhysicalBackupsScheduler{ + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetIncrementalBackupRepository(), + physical_repositories.GetInFlightBackupRepository(), + backups_config_physical.GetBackupConfigService(), + chain_view.GetChainViewService(), + tasks_cancellation.GetTaskCancelManager(), + CreateTestPhysicalBackuper(nil), + atomicTime{}, + logger.GetLogger(), + atomic.Bool{}, + atomic.Bool{}, + } +} + +// CreateTestWalStreamSupervisor returns a fresh WAL stream supervisor wired to +// the production repos and services. A fresh instance (not a copy of the DI +// singleton) keeps each test's hasRun/running state isolated and avoids copying +// the embedded mutex. +func CreateTestWalStreamSupervisor() *PhysicalWalStreamSupervisor { + return &PhysicalWalStreamSupervisor{ + databases.GetDatabaseService(), + backups_config_physical.GetBackupConfigService(), + storages.GetStorageService(), + physical_repositories.GetWalSegmentRepository(), + physical_repositories.GetWalHistoryRepository(), + physical_repositories.GetWalStreamerRepository(), + notifiers.GetNotifierService(), + tasks_cancellation.GetTaskCancelManager(), + encryption_secrets.GetSecretKeyService(), + encryption.GetFieldEncryptor(), + logger.GetLogger(), + sync.Mutex{}, + make(map[uuid.UUID]*runningStreamer), + atomicTime{}, + atomic.Bool{}, + atomic.Bool{}, + } +} + +// CreateTestPhysicalCleaner returns a cleaner wired to the production service + +// repos. +func CreateTestPhysicalCleaner() *PhysicalBackupCleaner { + return &PhysicalBackupCleaner{ + physical_service.GetPhysicalBackupService(), + chain_view.GetChainViewService(), + backups_config_physical.GetBackupConfigService(), + physical_repositories.GetFullBackupRepository(), + physical_repositories.GetWalSegmentRepository(), + logger.GetLogger(), + atomic.Bool{}, + } +} + +// StartPhysicalSchedulerForTest starts a fresh scheduler's Run loop in a +// goroutine and waits until it is ready (IsRunning). The scheduler invokes its +// own in-process backuper directly, so a backup requested over HTTP (config-enable +// bootstrap for the FULL, the trigger endpoint for incrementals) is picked up on +// the next tick and run to a terminal state. Returns a cancel func the caller +// defers; it cancels and waits for the goroutine to drain. +func StartPhysicalSchedulerForTest(t *testing.T) context.CancelFunc { + t.Helper() + + scheduler := CreateTestPhysicalScheduler() + + ctx, cancel := context.WithCancel(context.Background()) + + done := make(chan struct{}) + + go func() { + scheduler.Run(ctx) + close(done) + }() + + deadline := time.Now().UTC().Add(5 * time.Second) + for time.Now().UTC().Before(deadline) { + if scheduler.IsRunning() { + return func() { + cancel() + + select { + case <-done: + case <-time.After(2 * time.Second): + t.Log("physical scheduler stop timeout") + } + } + } + + time.Sleep(50 * time.Millisecond) + } + + t.Fatalf("physical scheduler failed to start within timeout") + + return nil +} + +// StartPhysicalWalStreamSupervisorForTest starts a fresh WAL stream supervisor's +// Run loop in a goroutine and waits until it is ready (IsRunning). It mirrors the +// supervisor wiring in cmd/main.go: with the supervisor up, enabling +// WAL-stream backups over the HTTP config API causes it to claim the database and +// create its replication slot on the next reconcile tick — so tests drive the slot +// lifecycle through the API instead of starting a streamer by hand. Returns a +// cancel func the caller defers; it cancels and waits for the goroutine to drain. +func StartPhysicalWalStreamSupervisorForTest(t *testing.T) context.CancelFunc { + t.Helper() + + supervisor := CreateTestWalStreamSupervisor() + + ctx, cancel := context.WithCancel(context.Background()) + + done := make(chan struct{}) + + go func() { + supervisor.Run(ctx) + close(done) + }() + + deadline := time.Now().UTC().Add(5 * time.Second) + for time.Now().UTC().Before(deadline) { + if supervisor.IsRunning() { + return func() { + cancel() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Log("physical wal stream supervisor stop timeout") + } + } + } + + time.Sleep(50 * time.Millisecond) + } + + t.Fatalf("physical wal stream supervisor failed to start within timeout") + + return nil +} diff --git a/backend/internal/features/backups/backups/backuping/physical/wal_supervisor.go b/backend/internal/features/backups/backups/backuping/physical/wal_supervisor.go new file mode 100644 index 0000000..def0bf2 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/wal_supervisor.go @@ -0,0 +1,451 @@ +package backuping_physical + +import ( + "context" + "fmt" + "log/slog" + "os" + "path/filepath" + "slices" + "sync" + "sync/atomic" + "time" + + "github.com/google/uuid" + + "databasus-backend/internal/config" + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + postgresql_executor "databasus-backend/internal/features/backups/backups/usecases/physical/postgresql" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + postgresql_physical "databasus-backend/internal/features/databases/databases/postgresql/physical" + encryption_secrets "databasus-backend/internal/features/encryption/secrets" + "databasus-backend/internal/features/storages" + tasks_cancellation "databasus-backend/internal/features/tasks/cancellation" + util_encryption "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/walmath" +) + +// PhysicalWalStreamSupervisor is the background service that owns long-running +// pg_receivewal streamers. Ownership and restart recovery run through the +// physical_wal_streamers heartbeat table: every tick it discovers WAL_STREAM +// databases, CAS-claims any that are unclaimed / FAILED / stale, runs one +// WalStreamSupervisor goroutine per owned DB, and heartbeats the rows it owns. +type PhysicalWalStreamSupervisor struct { + databaseService *databases.DatabaseService + backupConfigService *backups_config_physical.BackupConfigService + storageService *storages.StorageService + walSegmentRepo *physical_repositories.PhysicalWalSegmentRepository + historyRepo *physical_repositories.PhysicalWalHistoryRepository + walStreamerRepo *physical_repositories.PhysicalWalStreamerRepository + notificationSender NotificationSender + taskCancelManager *tasks_cancellation.TaskCancelManager + secretKeyService *encryption_secrets.SecretKeyService + fieldEncryptor util_encryption.FieldEncryptor + logger *slog.Logger + + mu sync.Mutex + running map[uuid.UUID]*runningStreamer + + lastTickTime atomicTime + + hasRun atomic.Bool + isReady atomic.Bool +} + +// runningStreamer is the handle to one locally-owned streamer goroutine. +type runningStreamer struct { + cancel context.CancelFunc + done chan struct{} + watchDir string + shouldRemoveWatchDir atomic.Bool +} + +func (s *PhysicalWalStreamSupervisor) IsRunning() bool { + return s.isReady.Load() +} + +func (s *PhysicalWalStreamSupervisor) IsSupervisorHealthy() bool { + return s.lastTickTime.Load().After(time.Now().UTC().Add(-schedulerHealthcheckThreshold)) +} + +func (s *PhysicalWalStreamSupervisor) Run(ctx context.Context) { + if s.hasRun.Swap(true) { + panic(fmt.Sprintf("%T.Run() called multiple times", s)) + } + + s.logger = s.logger.With("job_id", uuid.New(), "job_name", walStreamSupervisorJobName) + + s.lastTickTime.Store(time.Now().UTC()) + + if err := s.recoverStreamersOnStartup(); err != nil { + s.logger.Error( + "failed to recover wal streamers on startup", + "error", + err, + "job_name", + walStreamSupervisorJobName, + ) + + panic(err) + } + + s.isReady.Store(true) + + defer func() { + s.isReady.Store(false) + s.stopAllStreamers() + }() + + ticker := time.NewTicker(walStreamSupervisorTickInterval) + defer ticker.Stop() + + s.reconcile(ctx) + + for { + select { + case <-ctx.Done(): + return + + case <-ticker.C: + s.reconcile(ctx) + + s.lastTickTime.Store(time.Now().UTC()) + } + } +} + +func (s *PhysicalWalStreamSupervisor) recoverStreamersOnStartup() error { + failed, err := s.walStreamerRepo.MarkStaleRunningFailed(int(streamerHeartbeatStaleness.Seconds())) + if err != nil { + return err + } + + if failed > 0 { + s.logger.Warn( + fmt.Sprintf("recovered %d stale wal streamers on startup", failed), + "job_name", walStreamSupervisorJobName, + ) + } + + return nil +} + +// reconcile starts streamers for newly-claimable WAL_STREAM databases, stops +// local streamers whose database is no longer a candidate (disabled or demoted), +// and heartbeats the rows we own. +func (s *PhysicalWalStreamSupervisor) reconcile(ctx context.Context) { + configs, err := s.backupConfigService.GetBackupConfigsWithEnabledBackups() + if err != nil { + s.logger.Error( + "wal supervisor: failed to load enabled configs", + "error", + err, + "job_name", + walStreamSupervisorJobName, + ) + + return + } + + candidates := make(map[uuid.UUID]bool) + + for _, backupConfig := range configs { + if !isWalStreamCandidate(backupConfig) { + continue + } + + logger := s.logger.With("database_id", backupConfig.DatabaseID, "job_name", walStreamSupervisorJobName) + + candidates[backupConfig.DatabaseID] = true + + s.ensureStreamerRunning(ctx, logger, backupConfig) + } + + s.stopNonCandidates(candidates) + s.heartbeatOwnedStreamers() +} + +// isWalStreamCandidate reports whether a config should have a running streamer: +// WAL_STREAM backup type with storage configured. +func isWalStreamCandidate(backupConfig *backups_config_physical.PhysicalBackupConfig) bool { + if backupConfig.PostgresqlPhysical == nil || + backupConfig.PostgresqlPhysical.BackupType != postgresql_physical.BackupTypeFullIncrementalAndWalStream { + return false + } + + return backupConfig.StorageID != nil +} + +// ensureStreamerRunning starts a streamer for backupConfig when we don't already +// run one locally and we can win the heartbeat-table claim. +func (s *PhysicalWalStreamSupervisor) ensureStreamerRunning( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) { + s.mu.Lock() + _, alreadyRunning := s.running[backupConfig.DatabaseID] + s.mu.Unlock() + + if alreadyRunning { + return + } + + claimed, err := s.walStreamerRepo.ClaimIfClaimable( + backupConfig.DatabaseID, int(streamerHeartbeatStaleness.Seconds()), + ) + if err != nil { + logger.Error("wal supervisor: claim failed", "error", err) + + return + } + + if !claimed { + return + } + + s.startStreamer(ctx, logger, backupConfig) +} + +func (s *PhysicalWalStreamSupervisor) startStreamer( + ctx context.Context, + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) { + db, err := s.databaseService.GetDatabaseByID(backupConfig.DatabaseID) + if err != nil || db.PostgresqlPhysical == nil { + logger.Error("wal supervisor: failed to load database for streamer", "error", err) + s.releaseClaim(logger, backupConfig.DatabaseID) + + return + } + + storage, err := s.storageService.GetStorageByID(*backupConfig.StorageID) + if err != nil { + logger.Error("wal supervisor: failed to load storage for streamer", "error", err) + s.releaseClaim(logger, backupConfig.DatabaseID) + + return + } + + masterKey, ok := s.resolveMasterKey(logger, backupConfig) + if !ok { + s.releaseClaim(logger, backupConfig.DatabaseID) + + return + } + + // Derive from the supervisor's run ctx so a process shutdown cancels every + // streamer; the per-DB cancel (registered with TaskCancelManager + stored on + // runningStreamer) handles targeted teardown on disable / demote / db-remove. + streamerCtx, cancel := context.WithCancel(ctx) + done := make(chan struct{}) + streamer := &runningStreamer{ + cancel: cancel, + done: done, + watchDir: filepath.Join(config.GetEnv().DataFolder, "wal-queue", db.ID.String()), + } + + s.taskCancelManager.RegisterTask(db.ID, func() { + streamer.shouldRemoveWatchDir.Store(true) + cancel() + }) + + supervisor := postgresql_executor.NewWalStreamSupervisor(postgresql_executor.WalStreamSpec{ + DatabaseID: db.ID, + SourceDB: db.PostgresqlPhysical, + StorageID: storage.ID, + Storage: storage, + Encryption: backupConfig.Encryption, + MasterKey: masterKey, + FieldEncryptor: s.fieldEncryptor, + WalSegmentRepo: s.walSegmentRepo, + HistoryRepo: s.historyRepo, + WatchDirRoot: config.GetEnv().DataFolder, + WalLagThresholdBytes: backupConfig.WalLagThresholdBytes, + OnGapDetected: s.gapNotifier(db, backupConfig), + OnSlotRebuilt: s.slotRebuildFullRequester(logger, backupConfig), + Logger: s.logger, + }) + + s.mu.Lock() + s.running[db.ID] = streamer + s.mu.Unlock() + + logger.Info("wal stream supervisor started for database") + + go func() { + defer close(done) + defer s.taskCancelManager.UnregisterTask(db.ID) + defer s.removeWatchDirIfRequested(logger, streamer) + + if err := supervisor.Run(streamerCtx); err != nil { + logger.Error("wal stream supervisor exited with error", "error", err) + + // Mark FAILED so a later tick can reclaim it. A clean + // ctx-cancel (shutdown / lifecycle stop) does not reach here with an + // error, so the row is left for the cancelling path to handle. + if markErr := s.walStreamerRepo.MarkFailed(db.ID); markErr != nil { + logger.Error("wal supervisor: failed to mark streamer failed", "error", markErr) + } + } + + s.mu.Lock() + delete(s.running, db.ID) + s.mu.Unlock() + }() +} + +func (s *PhysicalWalStreamSupervisor) resolveMasterKey( + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) (string, bool) { + if backupConfig.Encryption != backups_core_enums.BackupEncryptionEncrypted { + return "", true + } + + key, err := s.secretKeyService.GetSecretKey() + if err != nil { + logger.Error("wal supervisor: failed to fetch master key", "error", err) + + return "", false + } + + return key, true +} + +// gapNotifier builds the post-upload gap-probe callback: it sends a BackupFailed +// notification (when the config opts in) to each of the database's notifiers. +func (s *PhysicalWalStreamSupervisor) gapNotifier( + db *databases.Database, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) func(gapStart, gapEnd walmath.LSN) { + return func(gapStart, gapEnd walmath.LSN) { + if !slices.Contains(backupConfig.SendNotificationsOn, backups_config_physical.NotificationWalGap) { + return + } + + title := fmt.Sprintf("Physical WAL gap detected for %q", db.Name) + message := fmt.Sprintf("database_id=%s gap=[%s, %s)", db.ID, gapStart.String(), gapEnd.String()) + + for _, notifier := range db.Notifiers { + s.notificationSender.SendNotification(¬ifier, title, message) + } + } +} + +func (s *PhysicalWalStreamSupervisor) slotRebuildFullRequester( + logger *slog.Logger, + backupConfig *backups_config_physical.PhysicalBackupConfig, +) func(context.Context, string) error { + return func(_ context.Context, reason string) error { + if err := s.backupConfigService.RequestFullBackupNow(backupConfig.DatabaseID); err != nil { + return err + } + + logger.Warn("requested out-of-cadence full backup after wal slot rebuild", "reason", reason) + + return nil + } +} + +func (s *PhysicalWalStreamSupervisor) stopNonCandidates(candidates map[uuid.UUID]bool) { + s.mu.Lock() + var toStop []uuid.UUID + + for databaseID := range s.running { + if !candidates[databaseID] { + toStop = append(toStop, databaseID) + } + } + s.mu.Unlock() + + for _, databaseID := range toStop { + s.stopStreamer(databaseID, true) + } +} + +func (s *PhysicalWalStreamSupervisor) stopStreamer(databaseID uuid.UUID, shouldRemoveWatchDir bool) { + s.mu.Lock() + streamer := s.running[databaseID] + s.mu.Unlock() + + if streamer == nil { + return + } + + if shouldRemoveWatchDir { + streamer.shouldRemoveWatchDir.Store(true) + } + + streamer.cancel() + + select { + case <-streamer.done: + case <-time.After(streamerStopTimeout): + s.logger.Warn("wal supervisor: streamer stop timed out", "database_id", databaseID) + } + + s.mu.Lock() + delete(s.running, databaseID) + s.mu.Unlock() + + if err := s.walStreamerRepo.MarkFailed(databaseID); err != nil { + s.logger.Error( + "wal supervisor: failed to mark stopped streamer failed", + "database_id", + databaseID, + "error", + err, + ) + } +} + +func (s *PhysicalWalStreamSupervisor) heartbeatOwnedStreamers() { + s.mu.Lock() + owned := make([]uuid.UUID, 0, len(s.running)) + for databaseID := range s.running { + owned = append(owned, databaseID) + } + s.mu.Unlock() + + for _, databaseID := range owned { + if err := s.walStreamerRepo.Heartbeat(databaseID); err != nil { + s.logger.Error("wal supervisor: heartbeat failed", "database_id", databaseID, "error", err) + } + } +} + +func (s *PhysicalWalStreamSupervisor) stopAllStreamers() { + s.mu.Lock() + owned := make([]uuid.UUID, 0, len(s.running)) + for databaseID := range s.running { + owned = append(owned, databaseID) + } + s.mu.Unlock() + + for _, databaseID := range owned { + s.stopStreamer(databaseID, false) + } +} + +func (s *PhysicalWalStreamSupervisor) removeWatchDirIfRequested(logger *slog.Logger, streamer *runningStreamer) { + if !streamer.shouldRemoveWatchDir.Load() { + return + } + + if err := os.RemoveAll(streamer.watchDir); err != nil { + logger.Warn("failed to remove wal queue directory", "watch_dir", streamer.watchDir, "error", err) + } +} + +// releaseClaim marks a just-won streamer row FAILED when we could not actually +// start the streamer, so the slot is immediately reclaimable rather than looking +// alive until the heartbeat goes stale. +func (s *PhysicalWalStreamSupervisor) releaseClaim(logger *slog.Logger, databaseID uuid.UUID) { + if err := s.walStreamerRepo.MarkFailed(databaseID); err != nil { + logger.Error("wal supervisor: failed to release streamer claim", "error", err) + } +} diff --git a/backend/internal/features/backups/backups/backuping/physical/wal_supervisor_test.go b/backend/internal/features/backups/backups/backuping/physical/wal_supervisor_test.go new file mode 100644 index 0000000..e866fa0 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/physical/wal_supervisor_test.go @@ -0,0 +1,207 @@ +package backuping_physical + +import ( + "os" + "path/filepath" + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/require" + + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + "databasus-backend/internal/storage" +) + +// seedStreamerRow inserts a physical_wal_streamers row with an explicit +// last_heartbeat_at and status, for ClaimIfClaimable tests. database_id must FK +// to a real databases row. +func seedStreamerRow( + t *testing.T, + databaseID uuid.UUID, + heartbeatAt time.Time, + status physical_enums.PhysicalWalStreamerStatus, +) { + t.Helper() + + require.NoError(t, storage.GetDb().Exec(` + INSERT INTO physical_wal_streamers (database_id, started_at, last_heartbeat_at, status) + VALUES (?, NOW(), ?, ?) + ON CONFLICT (database_id) DO UPDATE + SET last_heartbeat_at = EXCLUDED.last_heartbeat_at, status = EXCLUDED.status + `, databaseID, heartbeatAt, status).Error) + + t.Cleanup(func() { + _ = physical_repositories.GetWalStreamerRepository().DeleteByDatabaseID(databaseID) + }) +} + +func Test_ClaimIfClaimable_WhenNoRowExists_InsertsAndClaims(t *testing.T) { + prereqs := seedBackupPrereqs(t) + + t.Cleanup(func() { + _ = physical_repositories.GetWalStreamerRepository().DeleteByDatabaseID(prereqs.DB.ID) + }) + + claimed, err := physical_repositories.GetWalStreamerRepository().ClaimIfClaimable(prereqs.DB.ID, 90) + require.NoError(t, err) + require.True(t, claimed, "a missing row must be insert-claimed") + + row, err := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, row) + require.Equal(t, physical_enums.PhysicalWalStreamerStatusRunning, row.Status) +} + +func Test_ClaimIfClaimable_WhenFreshRunningRow_NotClaimable(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedStreamerRow(t, prereqs.DB.ID, time.Now().UTC(), physical_enums.PhysicalWalStreamerStatusRunning) + + claimed, err := physical_repositories.GetWalStreamerRepository().ClaimIfClaimable(prereqs.DB.ID, 90) + require.NoError(t, err) + require.False(t, claimed, "a fresh RUNNING streamer owned by a live process must not be reclaimed") +} + +func Test_ClaimIfClaimable_WhenHeartbeatStale_Claimable(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedStreamerRow( + t, + prereqs.DB.ID, + time.Now().UTC().Add(-5*time.Minute), + physical_enums.PhysicalWalStreamerStatusRunning, + ) + + claimed, err := physical_repositories.GetWalStreamerRepository().ClaimIfClaimable(prereqs.DB.ID, 90) + require.NoError(t, err) + require.True(t, claimed, "a streamer whose heartbeat aged past staleness must be reclaimable") +} + +func Test_ClaimIfClaimable_WhenFailedRow_Claimable(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedStreamerRow(t, prereqs.DB.ID, time.Now().UTC(), physical_enums.PhysicalWalStreamerStatusFailed) + + claimed, err := physical_repositories.GetWalStreamerRepository().ClaimIfClaimable(prereqs.DB.ID, 90) + require.NoError(t, err) + require.True(t, claimed, "a FAILED streamer must be reclaimable even with a fresh heartbeat") +} + +func Test_ClaimIfClaimable_WhenConcurrentClaims_ExactlyOneWins(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedStreamerRow( + t, + prereqs.DB.ID, + time.Now().UTC().Add(-5*time.Minute), + physical_enums.PhysicalWalStreamerStatusRunning, + ) + + const racers = 8 + + type claimOutcome struct { + claimed bool + err error + } + + results := make(chan claimOutcome, racers) + start := make(chan struct{}) + + for range racers { + go func() { + <-start + + // Don't call require.* off the test goroutine — collect and assert below. + claimed, err := physical_repositories.GetWalStreamerRepository().ClaimIfClaimable(prereqs.DB.ID, 90) + results <- claimOutcome{claimed: claimed, err: err} + }() + } + + close(start) + + wins := 0 + for range racers { + outcome := <-results + require.NoError(t, outcome.err) + + if outcome.claimed { + wins++ + } + } + + require.Equal(t, 1, wins, "exactly one concurrent claimant may win the stale streamer") +} + +func Test_PhysicalWalStreamSupervisorRun_WhenCalledTwice_Panics(t *testing.T) { + supervisor := CreateTestWalStreamSupervisor() + + ctx := t.Context() + + go supervisor.Run(ctx) + + deadline := time.Now().UTC().Add(5 * time.Second) + for time.Now().UTC().Before(deadline) { + if supervisor.IsRunning() { + break + } + + time.Sleep(50 * time.Millisecond) + } + + require.True(t, supervisor.IsRunning(), "supervisor must report running before the double-run check") + + require.Panics(t, func() { supervisor.Run(ctx) }, + "a second Run() on the same instance must panic") +} + +func Test_StopStreamer_WhenOwnedStreamerStops_MarksRowFailed(t *testing.T) { + prereqs := seedBackupPrereqs(t) + supervisor := CreateTestWalStreamSupervisor() + require.NoError(t, physical_repositories.GetWalStreamerRepository().Claim(prereqs.DB.ID)) + + done := make(chan struct{}) + close(done) + + supervisor.running[prereqs.DB.ID] = &runningStreamer{ + cancel: func() {}, + done: done, + } + + supervisor.stopStreamer(prereqs.DB.ID, false) + + streamer, err := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, streamer) + require.Equal(t, physical_enums.PhysicalWalStreamerStatusFailed, streamer.Status) +} + +func Test_RecoverStreamersOnStartup_WhenRunningRowStale_MarksFailed(t *testing.T) { + prereqs := seedBackupPrereqs(t) + seedStreamerRow( + t, + prereqs.DB.ID, + time.Now().UTC().Add(-2*streamerHeartbeatStaleness), + physical_enums.PhysicalWalStreamerStatusRunning, + ) + + supervisor := CreateTestWalStreamSupervisor() + + require.NoError(t, supervisor.recoverStreamersOnStartup()) + + streamer, err := physical_repositories.GetWalStreamerRepository().FindByDatabaseID(prereqs.DB.ID) + require.NoError(t, err) + require.NotNil(t, streamer) + require.Equal(t, physical_enums.PhysicalWalStreamerStatusFailed, streamer.Status) +} + +func Test_RemoveWatchDirIfRequested_WhenCleanupRequested_RemovesQueue(t *testing.T) { + supervisor := CreateTestWalStreamSupervisor() + watchDir := filepath.Join(t.TempDir(), "wal-queue") + require.NoError(t, os.MkdirAll(watchDir, 0o700)) + require.NoError(t, os.WriteFile(filepath.Join(watchDir, "segment"), []byte("wal"), 0o600)) + + streamer := &runningStreamer{watchDir: watchDir} + streamer.shouldRemoveWatchDir.Store(true) + + supervisor.removeWatchDirIfRequested(supervisor.logger, streamer) + + require.NoDirExists(t, watchDir) +} diff --git a/backend/internal/features/backups/backups/backuping/shared/gfs/gfs.go b/backend/internal/features/backups/backups/backuping/shared/gfs/gfs.go new file mode 100644 index 0000000..2ff7bf7 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/shared/gfs/gfs.go @@ -0,0 +1,152 @@ +package gfs + +import ( + "cmp" + "fmt" + "slices" + "time" + + "github.com/google/uuid" +) + +type Item struct { + ID uuid.UUID + CreatedAt time.Time +} + +func GetItemsToRetain( + items []Item, + hours, days, weeks, months, years int, +) map[uuid.UUID]bool { + keep := make(map[uuid.UUID]bool) + + if len(items) == 0 { + return keep + } + + itemsNewestFirst := sortNewestFirst(items) + + hoursSeen := make(map[string]bool) + daysSeen := make(map[string]bool) + weeksSeen := make(map[string]bool) + monthsSeen := make(map[string]bool) + yearsSeen := make(map[string]bool) + + hoursKept, daysKept, weeksKept, monthsKept, yearsKept := 0, 0, 0, 0, 0 + + // Compute per-level time-window cutoffs so higher-frequency slots + // cannot absorb items that belong to lower-frequency levels. + ref := itemsNewestFirst[0].CreatedAt + + rawHourlyCutoff := ref.Add(-time.Duration(hours) * time.Hour) + rawDailyCutoff := ref.Add(-time.Duration(days) * 24 * time.Hour) + rawWeeklyCutoff := ref.Add(-time.Duration(weeks) * 7 * 24 * time.Hour) + rawMonthlyCutoff := ref.AddDate(0, -months, 0) + rawYearlyCutoff := ref.AddDate(-years, 0, 0) + + // Hierarchical capping: each level's window cannot extend further back + // than the nearest active lower-frequency level's window. + yearlyCutoff := rawYearlyCutoff + + monthlyCutoff := rawMonthlyCutoff + if years > 0 { + monthlyCutoff = laterOf(monthlyCutoff, yearlyCutoff) + } + + weeklyCutoff := rawWeeklyCutoff + if months > 0 { + weeklyCutoff = laterOf(weeklyCutoff, monthlyCutoff) + } else if years > 0 { + weeklyCutoff = laterOf(weeklyCutoff, yearlyCutoff) + } + + dailyCutoff := rawDailyCutoff + switch { + case weeks > 0: + dailyCutoff = laterOf(dailyCutoff, weeklyCutoff) + case months > 0: + dailyCutoff = laterOf(dailyCutoff, monthlyCutoff) + case years > 0: + dailyCutoff = laterOf(dailyCutoff, yearlyCutoff) + } + + hourlyCutoff := rawHourlyCutoff + switch { + case days > 0: + hourlyCutoff = laterOf(hourlyCutoff, dailyCutoff) + case weeks > 0: + hourlyCutoff = laterOf(hourlyCutoff, weeklyCutoff) + case months > 0: + hourlyCutoff = laterOf(hourlyCutoff, monthlyCutoff) + case years > 0: + hourlyCutoff = laterOf(hourlyCutoff, yearlyCutoff) + } + + for _, item := range itemsNewestFirst { + t := item.CreatedAt + + hourKey := t.Format("2006-01-02-15") + dayKey := t.Format("2006-01-02") + weekYear, week := t.ISOWeek() + weekKey := fmt.Sprintf("%d-%02d", weekYear, week) + monthKey := t.Format("2006-01") + yearKey := t.Format("2006") + + if hours > 0 && hoursKept < hours && !hoursSeen[hourKey] && t.After(hourlyCutoff) { + keep[item.ID] = true + hoursSeen[hourKey] = true + hoursKept++ + } + + if days > 0 && daysKept < days && !daysSeen[dayKey] && t.After(dailyCutoff) { + keep[item.ID] = true + daysSeen[dayKey] = true + daysKept++ + } + + if weeks > 0 && weeksKept < weeks && !weeksSeen[weekKey] && t.After(weeklyCutoff) { + keep[item.ID] = true + weeksSeen[weekKey] = true + weeksKept++ + } + + if months > 0 && monthsKept < months && !monthsSeen[monthKey] && t.After(monthlyCutoff) { + keep[item.ID] = true + monthsSeen[monthKey] = true + monthsKept++ + } + + if years > 0 && yearsKept < years && !yearsSeen[yearKey] && t.After(yearlyCutoff) { + keep[item.ID] = true + yearsSeen[yearKey] = true + yearsKept++ + } + } + + return keep +} + +func laterOf(a, b time.Time) time.Time { + if a.After(b) { + return a + } + + return b +} + +// sortNewestFirst returns a copy of items ordered from newest to oldest, +// leaving the caller's slice untouched. Ties are broken by ID so the result +// is deterministic when several items share the exact same timestamp. +func sortNewestFirst(items []Item) []Item { + sorted := slices.Clone(items) + + slices.SortFunc(sorted, func(a, b Item) int { + if byTime := b.CreatedAt.Compare(a.CreatedAt); byTime != 0 { + return byTime + } + + return cmp.Compare(a.ID.String(), b.ID.String()) + }) + + return sorted +} diff --git a/backend/internal/features/backups/backups/backuping/shared/gfs/gfs_test.go b/backend/internal/features/backups/backups/backuping/shared/gfs/gfs_test.go new file mode 100644 index 0000000..852f1a6 --- /dev/null +++ b/backend/internal/features/backups/backups/backuping/shared/gfs/gfs_test.go @@ -0,0 +1,184 @@ +package gfs + +import ( + "testing" + "time" + + "github.com/google/uuid" + "github.com/stretchr/testify/assert" +) + +// itemAt builds an Item with a deterministic ID derived from the offset so a +// test can refer to the same instant twice and still get a stable identity. +func itemAt(reference time.Time, offset time.Duration) Item { + at := reference.Add(offset) + + return Item{ + ID: uuid.NewSHA1(uuid.Nil, []byte(at.Format(time.RFC3339Nano))), + CreatedAt: at, + } +} + +func Test_GetItemsToRetain_WhenNoItems_ReturnsEmptyKeepSet(t *testing.T) { + keep := GetItemsToRetain(nil, 1, 1, 1, 1, 1) + + assert.Empty(t, keep) +} + +func Test_GetItemsToRetain_WhenAllRetentionsZero_KeepsNothing(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + items := []Item{ + itemAt(reference, 0), + itemAt(reference, -1*time.Hour), + itemAt(reference, -24*time.Hour), + } + + keep := GetItemsToRetain(items, 0, 0, 0, 0, 0) + + assert.Empty(t, keep) +} + +func Test_GetItemsToRetain_WhenItemFillsEveryLevel_KeptOnce(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + newest := itemAt(reference, 0) + older := itemAt(reference, -90*24*time.Hour) + + keep := GetItemsToRetain([]Item{newest, older}, 1, 1, 1, 1, 1) + + // The single newest item simultaneously fills the hourly, daily, weekly, + // monthly, and yearly slot, so each level is satisfied by it alone. + assert.True(t, keep[newest.ID]) + assert.Len(t, keep, 1) +} + +func Test_GetItemsToRetain_WhenInputUnsorted_SortsBeforeApplyingScheme(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + newest := itemAt(reference, 0) + middle := itemAt(reference, -1*time.Hour) + oldest := itemAt(reference, -2*time.Hour) + + // Deliberately shuffled: the function must sort newest-first itself. + unsorted := []Item{oldest, newest, middle} + + keep := GetItemsToRetain(unsorted, 2, 0, 0, 0, 0) + + assert.True(t, keep[newest.ID]) + assert.True(t, keep[middle.ID]) + assert.False(t, keep[oldest.ID]) + assert.Len(t, keep, 2) +} + +func Test_GetItemsToRetain_DoesNotMutateCallerSlice(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + oldest := itemAt(reference, -2*time.Hour) + newest := itemAt(reference, 0) + items := []Item{oldest, newest} + + GetItemsToRetain(items, 5, 0, 0, 0, 0) + + assert.Equal(t, oldest.ID, items[0].ID) + assert.Equal(t, newest.ID, items[1].ID) +} + +func Test_GetItemsToRetain_WhenMultipleItemsShareOneHour_KeepsOnlyNewestOfThatHour(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 30, 0, 0, time.UTC) + hourNewest := itemAt(reference, 0) + hourMiddle := itemAt(reference, -10*time.Minute) + hourOldest := itemAt(reference, -20*time.Minute) + previousHour := itemAt(reference, -1*time.Hour) + + keep := GetItemsToRetain( + []Item{hourMiddle, hourOldest, hourNewest, previousHour}, 2, 0, 0, 0, 0, + ) + + // All three of the first three share the same hour bucket, so only the + // newest of them is kept; the previous hour fills the second hourly slot. + assert.True(t, keep[hourNewest.ID]) + assert.True(t, keep[previousHour.ID]) + assert.False(t, keep[hourMiddle.ID]) + assert.False(t, keep[hourOldest.ID]) + assert.Len(t, keep, 2) +} + +func Test_GetItemsToRetain_WhenHourlyCountExceeded_StopsKeeping(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + items := []Item{ + itemAt(reference, 0), + itemAt(reference, -1*time.Hour), + itemAt(reference, -2*time.Hour), + itemAt(reference, -3*time.Hour), + } + + keep := GetItemsToRetain(items, 2, 0, 0, 0, 0) + + assert.True(t, keep[items[0].ID]) + assert.True(t, keep[items[1].ID]) + assert.False(t, keep[items[2].ID]) + assert.False(t, keep[items[3].ID]) + assert.Len(t, keep, 2) +} + +func Test_GetItemsToRetain_WithDailyRetention_KeepsOnePerDay(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + today := itemAt(reference, 0) + todayEarlier := itemAt(reference, -3*time.Hour) + yesterday := itemAt(reference, -24*time.Hour) + twoDaysAgo := itemAt(reference, -48*time.Hour) + + keep := GetItemsToRetain( + []Item{today, todayEarlier, yesterday, twoDaysAgo}, 0, 3, 0, 0, 0, + ) + + assert.True(t, keep[today.ID]) + assert.True(t, keep[yesterday.ID]) + assert.True(t, keep[twoDaysAgo.ID]) + assert.False(t, keep[todayEarlier.ID]) + assert.Len(t, keep, 3) +} + +func Test_GetItemsToRetain_WhenItemsShareExactTimestamp_IsDeterministic(t *testing.T) { + at := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + first := Item{ID: uuid.New(), CreatedAt: at} + second := Item{ID: uuid.New(), CreatedAt: at} + + // Two items in the same hour bucket: exactly one survives, regardless of + // the input order, and the choice is stable across repeated runs. + keepAsGiven := GetItemsToRetain([]Item{first, second}, 1, 0, 0, 0, 0) + keepReversed := GetItemsToRetain([]Item{second, first}, 1, 0, 0, 0, 0) + + assert.Len(t, keepAsGiven, 1) + assert.Equal(t, keepAsGiven, keepReversed) +} + +func Test_GetItemsToRetain_WithMixedLevels_KeepsExpectedItems(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + now := itemAt(reference, 0) + oneHourAgo := itemAt(reference, -1*time.Hour) + yesterday := itemAt(reference, -24*time.Hour) + lastWeek := itemAt(reference, -8*24*time.Hour) + lastMonth := itemAt(reference, -40*24*time.Hour) + lastYear := itemAt(reference, -300*24*time.Hour) + + items := []Item{now, oneHourAgo, yesterday, lastWeek, lastMonth, lastYear} + + keep := GetItemsToRetain(items, 2, 2, 2, 2, 2) + + assert.True(t, keep[now.ID]) + assert.True(t, keep[oneHourAgo.ID]) + assert.True(t, keep[yesterday.ID]) + assert.True(t, keep[lastWeek.ID]) + assert.True(t, keep[lastMonth.ID]) + assert.True(t, keep[lastYear.ID]) +} + +func Test_GetItemsToRetain_DropsItemsOlderThanEveryWindow(t *testing.T) { + reference := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC) + recent := itemAt(reference, 0) + ancient := itemAt(reference, -10*365*24*time.Hour) + + keep := GetItemsToRetain([]Item{recent, ancient}, 1, 1, 1, 1, 1) + + assert.True(t, keep[recent.ID]) + assert.False(t, keep[ancient.ID]) + assert.Len(t, keep, 1) +} diff --git a/backend/internal/features/backups/backups/controllers/logical/controller.go b/backend/internal/features/backups/backups/controllers/logical/controller.go new file mode 100644 index 0000000..3acf724 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/logical/controller.go @@ -0,0 +1,388 @@ +package backups_controllers_logical + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + "databasus-backend/internal/features/backups/backups/download/stream_guard" + backups_dto_logical "databasus-backend/internal/features/backups/backups/dto/logical" + backups_services "databasus-backend/internal/features/backups/backups/services" + "databasus-backend/internal/features/databases" + users_middleware "databasus-backend/internal/features/users/middleware" + files_utils "databasus-backend/internal/util/files" +) + +type BackupController struct { + backupService *backups_services.LogicalBackupService +} + +func (c *BackupController) RegisterRoutes(router *gin.RouterGroup) { + router.GET("/backups", c.GetBackups) + router.POST("/backups", c.MakeBackup) + router.POST("/backups/:id/download-token", c.GenerateDownloadToken) + router.DELETE("/backups/:id", c.DeleteBackup) + router.POST("/backups/:id/cancel", c.CancelBackup) +} + +// RegisterPublicRoutes registers routes that don't require Bearer authentication +// (they have their own authentication mechanisms like download tokens) +func (c *BackupController) RegisterPublicRoutes(router *gin.RouterGroup) { + router.GET("/backups/:id/file", c.GetFile) +} + +// GetBackups +// @Summary Get backups for a database +// @Description Get paginated backups for the specified database with optional filters +// @Tags backups +// @Produce json +// @Param database_id query string true "Database ID" +// @Param limit query int false "Number of items per page" default(10) +// @Param offset query int false "Offset for pagination" default(0) +// @Param status query []string false "Filter by backup status (can be repeated)" Enums(IN_PROGRESS, COMPLETED, FAILED, CANCELED) +// @Param beforeDate query string false "Filter backups created before this date (RFC3339)" format(date-time) +// @Param pgWalBackupType query string false "Filter by WAL backup type" Enums(PG_FULL_BACKUP, PG_WAL_SEGMENT) +// @Success 200 {object} backups_dto_logical.GetBackupsResponse +// @Failure 400 +// @Failure 401 +// @Failure 500 +// @Router /backups [get] +func (c *BackupController) GetBackups(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + var request backups_dto_logical.GetBackupsRequest + if err := ctx.ShouldBindQuery(&request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + databaseID, err := uuid.Parse(request.DatabaseID) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid database_id"}) + return + } + + filters := c.buildBackupFilters(&request) + + response, err := c.backupService.GetBackups(user, databaseID, request.Limit, request.Offset, filters) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, response) +} + +// MakeBackup +// @Summary Create a backup +// @Description Create a new backup for the specified database +// @Tags backups +// @Accept json +// @Produce json +// @Param request body backups_dto_logical.MakeBackupRequest true "Backup creation data" +// @Success 200 {object} map[string]string +// @Failure 400 +// @Failure 401 +// @Failure 500 +// @Router /backups [post] +func (c *BackupController) MakeBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + var request backups_dto_logical.MakeBackupRequest + if err := ctx.ShouldBindJSON(&request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + if err := c.backupService.MakeBackupWithAuth(user, request.DatabaseID); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, gin.H{"message": "backup started successfully"}) +} + +// DeleteBackup +// @Summary Delete a backup +// @Description Delete an existing backup +// @Tags backups +// @Param id path string true "Backup ID" +// @Success 204 +// @Failure 400 +// @Failure 401 +// @Failure 500 +// @Router /backups/{id} [delete] +func (c *BackupController) DeleteBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + id, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + if err := c.backupService.DeleteBackup(user, id); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.Status(http.StatusNoContent) +} + +// CancelBackup +// @Summary Cancel an in-progress backup +// @Description Cancel a backup that is currently in progress +// @Tags backups +// @Param id path string true "Backup ID" +// @Success 204 +// @Failure 400 +// @Failure 401 +// @Failure 500 +// @Router /backups/{id}/cancel [post] +func (c *BackupController) CancelBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + id, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + if err := c.backupService.CancelBackup(user, id); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.Status(http.StatusNoContent) +} + +// GenerateDownloadToken +// @Summary Generate short-lived download token +// @Description Generate a token for downloading a backup file (valid for 5 minutes) +// @Tags backups +// @Param id path string true "Backup ID" +// @Success 200 {object} download_token.GenerateTokenResponse +// @Failure 400 +// @Failure 401 +// @Failure 409 {object} map[string]string "Download already in progress" +// @Router /backups/{id}/download-token [post] +func (c *BackupController) GenerateDownloadToken(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + id, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + response, err := c.backupService.GenerateDownloadToken(user, id) + if err != nil { + if errors.Is(err, stream_guard.ErrDownloadAlreadyInProgress) { + ctx.JSON( + http.StatusConflict, + gin.H{ + "error": "Download already in progress for some of backups. Please wait until previous download completed or cancel it", + }, + ) + return + } + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, response) +} + +// GetFile +// @Summary Download a backup file +// @Description Download the backup file for the specified backup using a download token. +// @Description +// @Description **Download Concurrency Control:** +// @Description - Only one download per user is allowed at a time +// @Description - If a download is already in progress, returns 409 Conflict +// @Description - Downloads are tracked using cache with 5-second TTL and 3-second heartbeat +// @Description - Browser cancellations automatically release the download lock +// @Description - Server crashes are handled via automatic cache expiry (5 seconds) +// @Tags backups +// @Param id path string true "Backup ID" +// @Param token query string true "Download token" +// @Success 200 {file} file +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 409 {object} map[string]string "Download already in progress" +// @Failure 500 {object} map[string]string +// @Router /backups/{id}/file [get] +func (c *BackupController) GetFile(ctx *gin.Context) { + token := ctx.Query("token") + if token == "" { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "download token is required"}) + return + } + + backupIDParam := ctx.Param("id") + backupID, err := uuid.Parse(backupIDParam) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + downloadToken, err := c.backupService.ValidateDownloadToken(token) + if err != nil { + if errors.Is(err, stream_guard.ErrDownloadAlreadyInProgress) { + ctx.JSON( + http.StatusConflict, + gin.H{ + "error": "download already in progress for this user. Please wait until previous download completed or cancel it", + }, + ) + return + } + + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "invalid or expired download token"}) + return + } + + if downloadToken.BackupID != backupID { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "invalid or expired download token"}) + return + } + + fileReader, backup, database, err := c.backupService.GetBackupFileWithoutAuth( + downloadToken.BackupID, + ) + if err != nil { + c.backupService.ReleaseDownloadLock(downloadToken.UserID) + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + defer func() { + cancelHeartbeat() + c.backupService.ReleaseDownloadLock(downloadToken.UserID) + if err := fileReader.Close(); err != nil { + fmt.Printf("Error closing file reader: %v\n", err) + } + }() + + go c.startDownloadHeartbeat(heartbeatCtx, downloadToken.UserID) + + filename := c.generateBackupFilename(backup, database) + + if backup.BackupSizeMb > 0 { + sizeBytes := int64(backup.BackupSizeMb * 1024 * 1024) + ctx.Header("Content-Length", fmt.Sprintf("%d", sizeBytes)) + } + + ctx.Header("Content-Type", "application/octet-stream") + ctx.Header( + "Content-Disposition", + fmt.Sprintf("attachment; filename=\"%s\"", filename), + ) + + _, err = io.Copy(ctx.Writer, fileReader) + if err != nil { + fmt.Printf("Error streaming file: %v\n", err) + } + + c.backupService.WriteAuditLogForDownload(downloadToken.UserID, backup, database) +} + +func (c *BackupController) generateBackupFilename( + backup *backups_core_logical.LogicalBackup, + database *databases.Database, +) string { + // Format timestamp as YYYY-MM-DD_HH-mm-ss + timestamp := backup.CreatedAt.Format("2006-01-02_15-04-05") + + // Sanitize database name for filename (replace spaces and special chars) + safeName := files_utils.SanitizeFilename(database.Name) + + // Determine extension based on database type + extension := c.getBackupExtension(database.Type) + + return fmt.Sprintf("%s_backup_%s%s", safeName, timestamp, extension) +} + +func (c *BackupController) getBackupExtension( + dbType databases.DatabaseType, +) string { + switch dbType { + case databases.DatabaseTypeMysql, databases.DatabaseTypeMariadb: + return ".sql.zst" + case databases.DatabaseTypePostgresLogical: + // PostgreSQL custom format + return ".dump" + case databases.DatabaseTypeMongodb: + return ".archive" + default: + return ".backup" + } +} + +func (c *BackupController) startDownloadHeartbeat(ctx context.Context, userID uuid.UUID) { + ticker := time.NewTicker(stream_guard.GetDownloadHeartbeatInterval()) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + c.backupService.RefreshDownloadLock(userID) + } + } +} + +func (c *BackupController) buildBackupFilters( + request *backups_dto_logical.GetBackupsRequest, +) *backups_core_logical.BackupFilters { + isHasFilters := len(request.Statuses) > 0 || + request.BeforeDate != nil + + if !isHasFilters { + return nil + } + + filters := &backups_core_logical.BackupFilters{} + + if len(request.Statuses) > 0 { + statuses := make([]backups_core_logical.BackupStatus, 0, len(request.Statuses)) + for _, statusStr := range request.Statuses { + statuses = append(statuses, backups_core_logical.BackupStatus(statusStr)) + } + + filters.Statuses = statuses + } + + filters.BeforeDate = request.BeforeDate + + return filters +} diff --git a/backend/internal/features/backups/backups/controllers/logical/controller_test.go b/backend/internal/features/backups/backups/controllers/logical/controller_test.go new file mode 100644 index 0000000..6898bd8 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/logical/controller_test.go @@ -0,0 +1,1674 @@ +package backups_controllers_logical + +import ( + "context" + "encoding/json" + "fmt" + "io" + "log/slog" + "net/http" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + "databasus-backend/internal/config" + audit_logs "databasus-backend/internal/features/audit_logs" + backuping_logical "databasus-backend/internal/features/backups/backups/backuping/logical" + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_download "databasus-backend/internal/features/backups/backups/download" + "databasus-backend/internal/features/backups/backups/download/download_token" + backups_dto_logical "databasus-backend/internal/features/backups/backups/dto/logical" + backups_services "databasus-backend/internal/features/backups/backups/services" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + postgresql_logical "databasus-backend/internal/features/databases/databases/postgresql/logical" + "databasus-backend/internal/features/storages" + local_storage "databasus-backend/internal/features/storages/models/local" + task_cancellation "databasus-backend/internal/features/tasks/cancellation" + users_dto "databasus-backend/internal/features/users/dto" + users_enums "databasus-backend/internal/features/users/enums" + users_services "databasus-backend/internal/features/users/services" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_models "databasus-backend/internal/features/workspaces/models" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + "databasus-backend/internal/util/encryption" + files_utils "databasus-backend/internal/util/files" + test_utils "databasus-backend/internal/util/testing" + "databasus-backend/internal/util/tools" +) + +func Test_GetBackups_PermissionsEnforced(t *testing.T) { + tests := []struct { + name string + workspaceRole *users_enums.WorkspaceRole + isGlobalAdmin bool + expectSuccess bool + expectedStatusCode int + }{ + { + name: "workspace viewer can get backups", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleViewer; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "workspace member can get backups", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleMember; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "non-member cannot get backups", + workspaceRole: nil, + isGlobalAdmin: false, + expectSuccess: false, + expectedStatusCode: http.StatusBadRequest, + }, + { + name: "global admin can get backups", + workspaceRole: nil, + isGlobalAdmin: true, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, _, storage := createTestDatabaseWithBackups(workspace, owner, router) + + var testUserToken string + if tt.isGlobalAdmin { + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + testUserToken = admin.Token + } else if tt.workspaceRole != nil { + if *tt.workspaceRole == users_enums.WorkspaceRoleOwner { + testUserToken = owner.Token + } else { + member := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + workspace, + member, + *tt.workspaceRole, + owner.Token, + router, + ) + testUserToken = member.Token + } + } else { + nonMember := users_testing.CreateTestUser(users_enums.UserRoleMember) + testUserToken = nonMember.Token + } + + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups?database_id=%s", database.ID.String()), + "Bearer "+testUserToken, + tt.expectedStatusCode, + ) + + if tt.expectSuccess { + var response backups_dto_logical.GetBackupsResponse + err := json.Unmarshal(testResp.Body, &response) + assert.NoError(t, err) + assert.GreaterOrEqual(t, len(response.Backups), 1) + assert.GreaterOrEqual(t, response.Total, int64(1)) + } else { + assert.Contains(t, string(testResp.Body), "insufficient permissions") + } + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }) + } +} + +func Test_GetBackups_WithStatusFilter_ReturnsFilteredBackups(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + defer func() { + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + now := time.Now().UTC() + + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: now.Add(-3 * time.Hour), + }) + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusFailed, + CreatedAt: now.Add(-2 * time.Hour), + }) + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCanceled, + CreatedAt: now.Add(-1 * time.Hour), + }) + + // Single status filter + var singleResponse backups_dto_logical.GetBackupsResponse + test_utils.MakeGetRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups?database_id=%s&status=COMPLETED", database.ID.String()), + "Bearer "+owner.Token, + http.StatusOK, + &singleResponse, + ) + + assert.Equal(t, int64(1), singleResponse.Total) + assert.Len(t, singleResponse.Backups, 1) + assert.Equal(t, backups_core_logical.BackupStatusCompleted, singleResponse.Backups[0].Status) + + // Multiple status filter + var multiResponse backups_dto_logical.GetBackupsResponse + test_utils.MakeGetRequestAndUnmarshal( + t, + router, + fmt.Sprintf( + "/api/v1/backups?database_id=%s&status=COMPLETED&status=FAILED", + database.ID.String(), + ), + "Bearer "+owner.Token, + http.StatusOK, + &multiResponse, + ) + + assert.Equal(t, int64(2), multiResponse.Total) + assert.Len(t, multiResponse.Backups, 2) + + for _, backup := range multiResponse.Backups { + assert.True( + t, + backup.Status == backups_core_logical.BackupStatusCompleted || + backup.Status == backups_core_logical.BackupStatusFailed, + "expected COMPLETED or FAILED, got %s", backup.Status, + ) + } +} + +func Test_GetBackups_WithBeforeDateFilter_ReturnsFilteredBackups(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + defer func() { + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + now := time.Now().UTC() + cutoff := now.Add(-1 * time.Hour) + + olderBackup := CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: now.Add(-3 * time.Hour), + }) + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: now, + }) + + var response backups_dto_logical.GetBackupsResponse + test_utils.MakeGetRequestAndUnmarshal( + t, + router, + fmt.Sprintf( + "/api/v1/backups?database_id=%s&beforeDate=%s", + database.ID.String(), + cutoff.Format(time.RFC3339), + ), + "Bearer "+owner.Token, + http.StatusOK, + &response, + ) + + assert.Equal(t, int64(1), response.Total) + assert.Len(t, response.Backups, 1) + assert.Equal(t, olderBackup.ID, response.Backups[0].ID) +} + +func Test_GetBackups_WithCombinedFilters_ReturnsFilteredBackups(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + defer func() { + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + now := time.Now().UTC() + cutoff := now.Add(-1 * time.Hour) + + // Old completed — should match + oldCompleted := CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: now.Add(-3 * time.Hour), + }) + // Old failed — should NOT match (wrong status) + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusFailed, + CreatedAt: now.Add(-2 * time.Hour), + }) + // New completed — should NOT match (too recent) + CreateTestBackupWithOptions(database.ID, storage.ID, TestBackupOptions{ + Status: backups_core_logical.BackupStatusCompleted, + CreatedAt: now, + }) + + var response backups_dto_logical.GetBackupsResponse + test_utils.MakeGetRequestAndUnmarshal( + t, + router, + fmt.Sprintf( + "/api/v1/backups?database_id=%s&status=COMPLETED&beforeDate=%s", + database.ID.String(), + cutoff.Format(time.RFC3339), + ), + "Bearer "+owner.Token, + http.StatusOK, + &response, + ) + + assert.Equal(t, int64(1), response.Total) + assert.Len(t, response.Backups, 1) + assert.Equal(t, oldCompleted.ID, response.Backups[0].ID) +} + +func Test_CreateBackup_PermissionsEnforced(t *testing.T) { + tests := []struct { + name string + workspaceRole *users_enums.WorkspaceRole + isGlobalAdmin bool + expectSuccess bool + expectedStatusCode int + }{ + { + name: "workspace owner can create backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleOwner; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "workspace member can create backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleMember; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "workspace viewer can create backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleViewer; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "non-member cannot create backup", + workspaceRole: nil, + isGlobalAdmin: false, + expectSuccess: false, + expectedStatusCode: http.StatusBadRequest, + }, + { + name: "global admin can create backup", + workspaceRole: nil, + isGlobalAdmin: true, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + enableBackupForDatabase(database.ID) + + var testUserToken string + if tt.isGlobalAdmin { + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + testUserToken = admin.Token + } else if tt.workspaceRole != nil { + if *tt.workspaceRole == users_enums.WorkspaceRoleOwner { + testUserToken = owner.Token + } else { + member := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + workspace, + member, + *tt.workspaceRole, + owner.Token, + router, + ) + testUserToken = member.Token + } + } else { + nonMember := users_testing.CreateTestUser(users_enums.UserRoleMember) + testUserToken = nonMember.Token + } + + request := backups_dto_logical.MakeBackupRequest{DatabaseID: database.ID} + testResp := test_utils.MakePostRequest( + t, + router, + "/api/v1/backups", + "Bearer "+testUserToken, + request, + tt.expectedStatusCode, + ) + + if tt.expectSuccess { + assert.Contains(t, string(testResp.Body), "backup started successfully") + } else { + assert.Contains(t, string(testResp.Body), "insufficient permissions") + } + + // Cleanup + databases.RemoveTestDatabase(database) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }) + } +} + +func Test_CreateBackup_AuditLogWritten(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + enableBackupForDatabase(database.ID) + + request := backups_dto_logical.MakeBackupRequest{DatabaseID: database.ID} + test_utils.MakePostRequest( + t, + router, + "/api/v1/backups", + "Bearer "+owner.Token, + request, + http.StatusOK, + ) + + time.Sleep(100 * time.Millisecond) + + auditLogService := audit_logs.GetAuditLogService() + auditLogs, err := auditLogService.GetWorkspaceAuditLogs( + workspace.ID, + &audit_logs.GetAuditLogsRequest{ + Limit: 100, + Offset: 0, + }, + ) + assert.NoError(t, err) + + found := false + for _, log := range auditLogs.AuditLogs { + if strings.Contains(log.Message, "Backup manually initiated") && + strings.Contains(log.Message, database.Name) { + found = true + break + } + } + assert.True(t, found, "Audit log for backup creation not found") + + // Cleanup + databases.RemoveTestDatabase(database) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DeleteBackup_PermissionsEnforced(t *testing.T) { + tests := []struct { + name string + workspaceRole *users_enums.WorkspaceRole + isGlobalAdmin bool + expectSuccess bool + expectedStatusCode int + }{ + { + name: "workspace owner can delete backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleOwner; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusNoContent, + }, + { + name: "workspace member can delete backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleMember; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusNoContent, + }, + { + name: "workspace viewer cannot delete backup", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleViewer; return &r }(), + isGlobalAdmin: false, + expectSuccess: false, + expectedStatusCode: http.StatusBadRequest, + }, + { + name: "non-member cannot delete backup", + workspaceRole: nil, + isGlobalAdmin: false, + expectSuccess: false, + expectedStatusCode: http.StatusBadRequest, + }, + { + name: "global admin can delete backup", + workspaceRole: nil, + isGlobalAdmin: true, + expectSuccess: true, + expectedStatusCode: http.StatusNoContent, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + var testUserToken string + if tt.isGlobalAdmin { + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + testUserToken = admin.Token + } else if tt.workspaceRole != nil { + if *tt.workspaceRole == users_enums.WorkspaceRoleOwner { + testUserToken = owner.Token + } else { + member := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + workspace, + member, + *tt.workspaceRole, + owner.Token, + router, + ) + testUserToken = member.Token + } + } else { + nonMember := users_testing.CreateTestUser(users_enums.UserRoleMember) + testUserToken = nonMember.Token + } + + testResp := test_utils.MakeDeleteRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s", backup.ID.String()), + "Bearer "+testUserToken, + tt.expectedStatusCode, + ) + + if !tt.expectSuccess { + assert.Contains(t, string(testResp.Body), "insufficient permissions") + } else { + userService := users_services.GetUserService() + ownerUser, err := userService.GetUserFromToken(owner.Token) + assert.NoError(t, err) + + response, err := backups_services.GetBackupService().GetBackups(ownerUser, database.ID, 10, 0, nil) + assert.NoError(t, err) + assert.Equal(t, 0, len(response.Backups)) + } + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }) + } +} + +func Test_DeleteBackup_AuditLogWritten(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + test_utils.MakeDeleteRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s", backup.ID.String()), + "Bearer "+owner.Token, + http.StatusNoContent, + ) + + time.Sleep(100 * time.Millisecond) + + auditLogService := audit_logs.GetAuditLogService() + auditLogs, err := auditLogService.GetWorkspaceAuditLogs( + workspace.ID, + &audit_logs.GetAuditLogsRequest{ + Limit: 100, + Offset: 0, + }, + ) + assert.NoError(t, err) + + found := false + for _, log := range auditLogs.AuditLogs { + if strings.Contains(log.Message, "Backup deleted") && + strings.Contains(log.Message, database.Name) { + found = true + break + } + } + assert.True(t, found, "Audit log for backup deletion not found") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_GenerateDownloadToken_PermissionsEnforced(t *testing.T) { + tests := []struct { + name string + workspaceRole *users_enums.WorkspaceRole + isGlobalAdmin bool + expectSuccess bool + expectedStatusCode int + }{ + { + name: "workspace viewer can generate token", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleViewer; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "workspace member can generate token", + workspaceRole: func() *users_enums.WorkspaceRole { r := users_enums.WorkspaceRoleMember; return &r }(), + isGlobalAdmin: false, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + { + name: "non-member cannot generate token", + workspaceRole: nil, + isGlobalAdmin: false, + expectSuccess: false, + expectedStatusCode: http.StatusBadRequest, + }, + { + name: "global admin can generate token", + workspaceRole: nil, + isGlobalAdmin: true, + expectSuccess: true, + expectedStatusCode: http.StatusOK, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + var testUserToken string + if tt.isGlobalAdmin { + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + testUserToken = admin.Token + } else if tt.workspaceRole != nil { + if *tt.workspaceRole == users_enums.WorkspaceRoleOwner { + testUserToken = owner.Token + } else { + member := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + workspace, + member, + *tt.workspaceRole, + owner.Token, + router, + ) + testUserToken = member.Token + } + } else { + nonMember := users_testing.CreateTestUser(users_enums.UserRoleMember) + testUserToken = nonMember.Token + } + + testResp := test_utils.MakePostRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+testUserToken, + nil, + tt.expectedStatusCode, + ) + + if tt.expectSuccess { + var response download_token.GenerateTokenResponse + err := json.Unmarshal(testResp.Body, &response) + assert.NoError(t, err) + assert.NotEmpty(t, response.Token) + assert.NotEmpty(t, response.Filename) + assert.Equal(t, backup.ID, response.BackupID) + } else { + assert.Contains(t, string(testResp.Body), "insufficient permissions") + } + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }) + } +} + +func Test_DownloadBackup_WithValidToken_Success(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Generate download token + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + // Download with token + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), tokenResponse.Token), + "", + http.StatusOK, + ) + + // Verify response + contentDisposition := testResp.Headers.Get("Content-Disposition") + assert.Contains(t, contentDisposition, "attachment") + assert.Contains(t, contentDisposition, tokenResponse.Filename) + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_WithoutToken_Unauthorized(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Try to download without token + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file", backup.ID.String()), + "", + http.StatusUnauthorized, + ) + + assert.Contains(t, string(testResp.Body), "download token is required") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_WithInvalidToken_Unauthorized(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Try to download with invalid token + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), "invalid-token-xyz"), + "", + http.StatusUnauthorized, + ) + + assert.Contains(t, string(testResp.Body), "invalid or expired download token") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_WithExpiredToken_Unauthorized(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Get user for token generation + userService := users_services.GetUserService() + user, err := userService.GetUserFromToken(owner.Token) + assert.NoError(t, err) + + // Create an expired token directly in the database + expiredToken := createExpiredDownloadToken(backup.ID, user.ID) + + // Try to download with expired token + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), expiredToken), + "", + http.StatusUnauthorized, + ) + + assert.Contains(t, string(testResp.Body), "invalid or expired download token") + + // Verify audit log was NOT created for failed download + time.Sleep(100 * time.Millisecond) + auditLogService := audit_logs.GetAuditLogService() + auditLogs, err := auditLogService.GetWorkspaceAuditLogs( + workspace.ID, + &audit_logs.GetAuditLogsRequest{ + Limit: 100, + Offset: 0, + }, + ) + assert.NoError(t, err) + + found := false + for _, log := range auditLogs.AuditLogs { + if strings.Contains(log.Message, "Backup file downloaded") && + strings.Contains(log.Message, database.Name) { + found = true + break + } + } + assert.False(t, found, "Audit log should NOT be created for failed download with expired token") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_TokenUsedOnce_CannotReuseToken(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Generate download token + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + // Download with token (first time - should succeed) + test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), tokenResponse.Token), + "", + http.StatusOK, + ) + + // Try to download again with same token (should fail) + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), tokenResponse.Token), + "", + http.StatusUnauthorized, + ) + + assert.Contains(t, string(testResp.Body), "invalid or expired download token") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_WithDifferentBackupToken_Unauthorized(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database1 := createTestDatabase("Database 1", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + configService := backups_config_logical.GetBackupConfigService() + config1, err := configService.GetBackupConfigByDbId(database1.ID) + assert.NoError(t, err) + config1.IsBackupsEnabled = true + config1.StorageID = &storage.ID + config1.Storage = storage + _, err = configService.SaveBackupConfig(config1) + assert.NoError(t, err) + + backup1 := createTestBackup(database1, owner) + + database2 := createTestDatabase("Database 2", workspace.ID, owner.Token, router) + config2, err := configService.GetBackupConfigByDbId(database2.ID) + assert.NoError(t, err) + config2.IsBackupsEnabled = true + config2.StorageID = &storage.ID + config2.Storage = storage + _, err = configService.SaveBackupConfig(config2) + assert.NoError(t, err) + + backup2 := createTestBackup(database2, owner) + + // Generate token for backup1 + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup1.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + // Try to use backup1's token to download backup2 (should fail) + testResp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup2.ID.String(), tokenResponse.Token), + "", + http.StatusUnauthorized, + ) + + assert.Contains(t, string(testResp.Body), "invalid or expired download token") + + // Cleanup + databases.RemoveTestDatabase(database1) + databases.RemoveTestDatabase(database2) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_AuditLogWritten(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + // Generate download token + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + // Download with token + test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), tokenResponse.Token), + "", + http.StatusOK, + ) + + time.Sleep(100 * time.Millisecond) + + auditLogService := audit_logs.GetAuditLogService() + auditLogs, err := auditLogService.GetWorkspaceAuditLogs( + workspace.ID, + &audit_logs.GetAuditLogsRequest{ + Limit: 100, + Offset: 0, + }, + ) + assert.NoError(t, err) + + found := false + for _, log := range auditLogs.AuditLogs { + if strings.Contains(log.Message, "Backup file downloaded") && + strings.Contains(log.Message, database.Name) { + found = true + break + } + } + assert.True(t, found, "Audit log for backup download not found") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_DownloadBackup_ProperFilenameForPostgreSQL(t *testing.T) { + tests := []struct { + name string + databaseName string + expectedExt string + expectedInName string + }{ + { + name: "PostgreSQL database", + databaseName: "my_postgres_db", + expectedExt: ".dump", + expectedInName: "my_postgres_db_backup_", + }, + { + name: "Database name with spaces", + databaseName: "my test db", + expectedExt: ".dump", + expectedInName: "my_test_db_backup_", + }, + { + name: "Database name with special characters", + databaseName: "my:db/test", + expectedExt: ".dump", + expectedInName: "my-db-test_backup_", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase(tt.databaseName, workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + configService := backups_config_logical.GetBackupConfigService() + config, err := configService.GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + config.IsBackupsEnabled = true + config.StorageID = &storage.ID + config.Storage = storage + _, err = configService.SaveBackupConfig(config) + assert.NoError(t, err) + + backup := createTestBackup(database, owner) + + // Generate download token + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + // Download with token + resp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf( + "/api/v1/backups/%s/file?token=%s", + backup.ID.String(), + tokenResponse.Token, + ), + "", + http.StatusOK, + ) + + contentDisposition := resp.Headers.Get("Content-Disposition") + assert.NotEmpty(t, contentDisposition, "Content-Disposition header should be present") + + // Verify the filename contains expected parts + assert.Contains( + t, + contentDisposition, + tt.expectedInName, + "Filename should contain sanitized database name", + ) + assert.Contains( + t, + contentDisposition, + tt.expectedExt, + "Filename should have correct extension", + ) + assert.Contains(t, contentDisposition, "attachment", "Should be an attachment") + + // Verify timestamp format (YYYY-MM-DD_HH-mm-ss) + assert.Regexp( + t, + `\d{4}-\d{2}-\d{2}_\d{2}-\d{2}-\d{2}`, + contentDisposition, + "Filename should contain timestamp", + ) + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }) + } +} + +func Test_SanitizeFilename(t *testing.T) { + tests := []struct { + input string + expected string + }{ + {input: "simple_name", expected: "simple_name"}, + {input: "name with spaces", expected: "name_with_spaces"}, + {input: "name/with\\slashes", expected: "name-with-slashes"}, + {input: "name:with*special?chars", expected: "name-with-special-chars"}, + {input: "namepipes|", expected: "name-with-pipes-"}, + {input: `name"with"quotes`, expected: "name-with-quotes"}, + } + + for _, tt := range tests { + t.Run(tt.input, func(t *testing.T) { + result := files_utils.SanitizeFilename(tt.input) + assert.Equal(t, tt.expected, result) + }) + } +} + +func Test_CancelBackup_InProgressBackup_SuccessfullyCancelled(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + configService := backups_config_logical.GetBackupConfigService() + config, err := configService.GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + config.IsBackupsEnabled = true + config.StorageID = &storage.ID + config.Storage = storage + _, err = configService.SaveBackupConfig(config) + assert.NoError(t, err) + + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: storage.ID, + Status: backups_core_logical.BackupStatusInProgress, + BackupSizeMb: 0, + BackupDurationMs: 0, + CreatedAt: time.Now().UTC(), + } + + repo := &backups_core_logical.BackupRepository{} + err = repo.Save(backup) + assert.NoError(t, err) + + // Register a cancellable context for the backup + task_cancellation.GetTaskCancelManager().RegisterTask(backup.ID, func() {}) + + resp := test_utils.MakePostRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/cancel", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusNoContent, + ) + + assert.Equal(t, http.StatusNoContent, resp.StatusCode) + + // Verify audit log was created + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + userService := users_services.GetUserService() + adminUser, err := userService.GetUserFromToken(admin.Token) + assert.NoError(t, err) + + auditLogService := audit_logs.GetAuditLogService() + auditLogs, err := auditLogService.GetGlobalAuditLogs( + adminUser, + &audit_logs.GetAuditLogsRequest{Limit: 100, Offset: 0}, + ) + assert.NoError(t, err) + + foundCancelLog := false + for _, log := range auditLogs.AuditLogs { + if strings.Contains(log.Message, "Backup cancelled") && + strings.Contains(log.Message, database.Name) { + foundCancelLog = true + break + } + } + assert.True(t, foundCancelLog, "Cancel audit log should be created") + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_ConcurrentDownloadPrevention(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + var token1Response download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &token1Response, + ) + + var token2Response download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &token2Response, + ) + + service := backups_services.GetBackupService() + + // Hold the per-user stream lock to simulate a download in progress, deterministically — + // without racing a real download goroutine, which finishes too fast to observe under load. + service.RefreshDownloadLock(owner.UserID) + assert.True(t, service.IsDownloadInProgress(owner.UserID)) + + resp := test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), token1Response.Token), + "", + http.StatusConflict, + ) + + var errorResponse map[string]string + err := json.Unmarshal(resp.Body, &errorResponse) + assert.NoError(t, err) + assert.Contains(t, errorResponse["error"], "download already in progress") + + // After the lock is released, a download proceeds normally. + service.ReleaseDownloadLock(owner.UserID) + assert.False(t, service.IsDownloadInProgress(owner.UserID)) + + test_utils.MakeGetRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/file?token=%s", backup.ID.String(), token2Response.Token), + "", + http.StatusOK, + ) + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_GenerateDownloadToken_BlockedWhenDownloadInProgress(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, backup, storage := createTestDatabaseWithBackups(workspace, owner, router) + + service := backups_services.GetBackupService() + + // Mark a download in progress for this user deterministically. Taking the per-user stream + // lock is exactly what a real download does; setting it directly avoids racing a real + // download, which completes too fast to observe reliably under parallel test load. + service.RefreshDownloadLock(owner.UserID) + assert.True(t, service.IsDownloadInProgress(owner.UserID)) + + resp := test_utils.MakePostRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusConflict, + ) + + var errorResponse map[string]string + err := json.Unmarshal(resp.Body, &errorResponse) + assert.NoError(t, err) + assert.Contains(t, errorResponse["error"], "Download already in progress") + + // Releasing the lock allows token generation again. + service.ReleaseDownloadLock(owner.UserID) + assert.False(t, service.IsDownloadInProgress(owner.UserID)) + + var tokenResponse download_token.GenerateTokenResponse + test_utils.MakePostRequestAndUnmarshal( + t, + router, + fmt.Sprintf("/api/v1/backups/%s/download-token", backup.ID.String()), + "Bearer "+owner.Token, + nil, + http.StatusOK, + &tokenResponse, + ) + + assert.NotEmpty(t, tokenResponse.Token) + + // Cleanup + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func Test_MakeBackup_VerifyBackupAndMetadataFilesExistInStorage(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database, _, storage := createTestDatabaseWithBackups(workspace, owner, router) + + scheduler := backuping_logical.CreateTestScheduler() + schedulerCancel := backuping_logical.StartSchedulerForTest(t, scheduler) + defer schedulerCancel() + + backupRepo := &backups_core_logical.BackupRepository{} + initialBackups, err := backupRepo.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + request := backups_dto_logical.MakeBackupRequest{DatabaseID: database.ID} + test_utils.MakePostRequest( + t, + router, + "/api/v1/backups", + "Bearer "+owner.Token, + request, + http.StatusOK, + ) + + backuping_logical.WaitForBackupCompletion(t, database.ID, len(initialBackups), 30*time.Second) + + backups, err := backupRepo.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Greater(t, len(backups), len(initialBackups)) + + backup := backups[0] + assert.Equal(t, backups_core_logical.BackupStatusCompleted, backup.Status) + + storageService := storages.GetStorageService() + backupStorage, err := storageService.GetStorageByID(backup.StorageID) + assert.NoError(t, err) + + encryptor := encryption.GetFieldEncryptor() + + backupFile, err := backupStorage.GetFile(encryptor, backup.FileName) + require.NoError(t, err) + backupFile.Close() + + metadataFile, err := backupStorage.GetFile(encryptor, backup.FileName+".metadata") + require.NoError(t, err) + + metadataContent, err := io.ReadAll(metadataFile) + require.NoError(t, err) + metadataFile.Close() + + var storageMetadata backups_core_logical.BackupMetadata + err = json.Unmarshal(metadataContent, &storageMetadata) + assert.NoError(t, err) + + assert.Equal(t, backup.ID, storageMetadata.BackupID) + + if backup.EncryptionSalt != nil && storageMetadata.EncryptionSalt != nil { + assert.Equal(t, *backup.EncryptionSalt, *storageMetadata.EncryptionSalt) + } + + if backup.EncryptionIV != nil && storageMetadata.EncryptionIV != nil { + assert.Equal(t, *backup.EncryptionIV, *storageMetadata.EncryptionIV) + } + + assert.Equal(t, backup.Encryption, storageMetadata.Encryption) + + err = backupRepo.DeleteByID(backup.ID) + assert.NoError(t, err) + + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) +} + +func createTestRouter() *gin.Engine { + return CreateTestRouter() +} + +func createTestDatabase( + name string, + workspaceID uuid.UUID, + token string, + router *gin.Engine, +) *databases.Database { + env := config.GetEnv() + port, err := strconv.Atoi(env.TestLogicalPostgres16Port) + if err != nil { + panic(fmt.Sprintf("Failed to parse TEST_LOGICAL_POSTGRES_16_PORT: %v", err)) + } + + testDbName := "testdb" + request := databases.Database{ + Name: name, + WorkspaceID: &workspaceID, + Type: databases.DatabaseTypePostgresLogical, + PostgresqlLogical: &postgresql_logical.PostgresqlLogicalDatabase{ + Version: tools.PostgresqlVersion16, + Host: config.GetEnv().TestLocalhost, + Port: port, + Username: "testuser", + Password: "testpassword", + Database: &testDbName, + CpuCount: 1, + }, + } + + w := workspaces_testing.MakeAPIRequest( + router, + "POST", + "/api/v1/databases/create", + "Bearer "+token, + request, + ) + + if w.Code != http.StatusCreated { + panic( + fmt.Sprintf("Failed to create database. Status: %d, Body: %s", w.Code, w.Body.String()), + ) + } + + var database databases.Database + if err := json.Unmarshal(w.Body.Bytes(), &database); err != nil { + panic(err) + } + + return &database +} + +func createTestStorage(workspaceID uuid.UUID) *storages.Storage { + storage := &storages.Storage{ + WorkspaceID: workspaceID, + Type: storages.StorageTypeLocal, + Name: "Test Storage " + uuid.New().String(), + LocalStorage: &local_storage.LocalStorage{}, + } + + repo := &storages.StorageRepository{} + storage, err := repo.Save(storage) + if err != nil { + panic(err) + } + + return storage +} + +func enableBackupForDatabase(databaseID uuid.UUID) { + configService := backups_config_logical.GetBackupConfigService() + config, err := configService.GetBackupConfigByDbId(databaseID) + if err != nil { + panic(err) + } + + config.IsBackupsEnabled = true + _, err = configService.SaveBackupConfig(config) + if err != nil { + panic(err) + } +} + +func createTestDatabaseWithBackups( + workspace *workspaces_models.Workspace, + owner *users_dto.SignInResponseDTO, + router *gin.Engine, +) (*databases.Database, *backups_core_logical.LogicalBackup, *storages.Storage) { + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + configService := backups_config_logical.GetBackupConfigService() + config, err := configService.GetBackupConfigByDbId(database.ID) + if err != nil { + panic(err) + } + + config.IsBackupsEnabled = true + config.StorageID = &storage.ID + config.Storage = storage + _, err = configService.SaveBackupConfig(config) + if err != nil { + panic(err) + } + + backup := createTestBackup(database, owner) + + return database, backup, storage +} + +func createTestBackup( + database *databases.Database, + owner *users_dto.SignInResponseDTO, +) *backups_core_logical.LogicalBackup { + userService := users_services.GetUserService() + user, err := userService.GetUserFromToken(owner.Token) + if err != nil { + panic(err) + } + + loadedStorages, err := storages.GetStorageService().GetStorages(user, *database.WorkspaceID) + if err != nil || len(loadedStorages) == 0 { + panic("No storage found for workspace") + } + + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: database.ID, + StorageID: loadedStorages[0].ID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10.5, + BackupDurationMs: 1000, + CreatedAt: time.Now().UTC(), + } + + repo := &backups_core_logical.BackupRepository{} + if err := repo.Save(backup); err != nil { + panic(err) + } + + // Create a dummy backup file for testing download functionality + dummyContent := []byte("dummy backup content for testing") + reader := strings.NewReader(string(dummyContent)) + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + if err := loadedStorages[0].SaveFile( + context.Background(), + encryption.GetFieldEncryptor(), + logger, + backup.ID.String(), + reader, + ); err != nil { + panic(fmt.Sprintf("Failed to create test backup file: %v", err)) + } + + return backup +} + +func createExpiredDownloadToken(backupID, userID uuid.UUID) string { + tokenService := backups_download.GetDownloadTokenService() + token, err := tokenService.Generate(backupID, userID) + if err != nil { + panic(fmt.Sprintf("Failed to generate download token: %v", err)) + } + + // Manually update the token to be expired + repo := &download_token.Repository{} + downloadToken, err := repo.FindByToken(token) + if err != nil || downloadToken == nil { + panic(fmt.Sprintf("Failed to find generated token: %v", err)) + } + + // Set expiration to 10 minutes ago + downloadToken.ExpiresAt = time.Now().UTC().Add(-10 * time.Minute) + if err := repo.Update(downloadToken); err != nil { + panic(fmt.Sprintf("Failed to update token expiration: %v", err)) + } + + return token +} + +func Test_DeleteBackup_RemovesBackupAndMetadataFilesFromDisk(t *testing.T) { + router := createTestRouter() + owner := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Test Workspace", owner, router) + + database := createTestDatabase("Test Database", workspace.ID, owner.Token, router) + storage := createTestStorage(workspace.ID) + + configService := backups_config_logical.GetBackupConfigService() + backupConfig, err := configService.GetBackupConfigByDbId(database.ID) + assert.NoError(t, err) + + backupConfig.IsBackupsEnabled = true + backupConfig.StorageID = &storage.ID + backupConfig.Storage = storage + _, err = configService.SaveBackupConfig(backupConfig) + assert.NoError(t, err) + + defer func() { + databases.RemoveTestDatabase(database) + time.Sleep(50 * time.Millisecond) + storages.RemoveTestStorage(storage.ID) + workspaces_testing.RemoveTestWorkspace(workspace, router) + }() + + scheduler := backuping_logical.CreateTestScheduler() + schedulerCancel := backuping_logical.StartSchedulerForTest(t, scheduler) + defer schedulerCancel() + + backupRepo := &backups_core_logical.BackupRepository{} + initialBackups, err := backupRepo.FindByDatabaseID(database.ID) + assert.NoError(t, err) + + request := backups_dto_logical.MakeBackupRequest{DatabaseID: database.ID} + test_utils.MakePostRequest( + t, + router, + "/api/v1/backups", + "Bearer "+owner.Token, + request, + http.StatusOK, + ) + + backuping_logical.WaitForBackupCompletion(t, database.ID, len(initialBackups), 30*time.Second) + + backups, err := backupRepo.FindByDatabaseID(database.ID) + assert.NoError(t, err) + assert.Greater(t, len(backups), len(initialBackups)) + + backup := backups[0] + assert.Equal(t, backups_core_logical.BackupStatusCompleted, backup.Status) + + dataFolder := config.GetEnv().DataFolder + backupFilePath := filepath.Join(dataFolder, backup.FileName) + metadataFilePath := filepath.Join(dataFolder, backup.FileName+".metadata") + + _, err = os.Stat(backupFilePath) + assert.NoError(t, err, "backup file should exist on disk before deletion") + + _, err = os.Stat(metadataFilePath) + assert.NoError(t, err, "metadata file should exist on disk before deletion") + + test_utils.MakeDeleteRequest( + t, + router, + fmt.Sprintf("/api/v1/backups/%s", backup.ID.String()), + "Bearer "+owner.Token, + http.StatusNoContent, + ) + + _, err = os.Stat(backupFilePath) + assert.True(t, os.IsNotExist(err), "backup file should be removed from disk after deletion") + + _, err = os.Stat(metadataFilePath) + assert.True(t, os.IsNotExist(err), "metadata file should be removed from disk after deletion") +} diff --git a/backend/internal/features/backups/backups/controllers/logical/di.go b/backend/internal/features/backups/backups/controllers/logical/di.go new file mode 100644 index 0000000..79129a8 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/logical/di.go @@ -0,0 +1,13 @@ +package backups_controllers_logical + +import ( + backups_services "databasus-backend/internal/features/backups/backups/services" +) + +var backupController = &BackupController{ + backups_services.GetBackupService(), +} + +func GetBackupController() *BackupController { + return backupController +} diff --git a/backend/internal/features/backups/backups/controllers/logical/testing.go b/backend/internal/features/backups/backups/controllers/logical/testing.go new file mode 100644 index 0000000..e53cb3a --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/logical/testing.go @@ -0,0 +1,124 @@ +package backups_controllers_logical + +import ( + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + backups_core_logical "databasus-backend/internal/features/backups/backups/core/logical" + backups_config_logical "databasus-backend/internal/features/backups/config/logical" + "databasus-backend/internal/features/databases" + workspaces_controllers "databasus-backend/internal/features/workspaces/controllers" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" +) + +func CreateTestRouter() *gin.Engine { + router := workspaces_testing.CreateTestRouter( + workspaces_controllers.GetWorkspaceController(), + workspaces_controllers.GetMembershipController(), + databases.GetDatabaseController(), + backups_config_logical.GetBackupConfigController(), + GetBackupController(), + ) + + // Register public routes (no auth required - token-based) + v1 := router.Group("/api/v1") + GetBackupController().RegisterPublicRoutes(v1) + + return router +} + +// WaitForBackupCompletion waits for a new backup to be created and completed (or failed) +// for the given database. It checks for backups with count greater than expectedInitialCount. +func WaitForBackupCompletion( + t *testing.T, + databaseID uuid.UUID, + expectedInitialCount int, + timeout time.Duration, +) { + deadline := time.Now().UTC().Add(timeout) + + for time.Now().UTC().Before(deadline) { + backups, err := backups_core_logical.GetBackupRepository().FindByDatabaseID(databaseID) + if err != nil { + t.Logf("WaitForBackupCompletion: error finding backups: %v", err) + time.Sleep(50 * time.Millisecond) + continue + } + + t.Logf( + "WaitForBackupCompletion: found %d backups (expected > %d)", + len(backups), + expectedInitialCount, + ) + + if len(backups) > expectedInitialCount { + // Check if the newest backup has completed or failed + newestBackup := backups[0] + t.Logf("WaitForBackupCompletion: newest backup status: %s", newestBackup.Status) + + if newestBackup.Status == backups_core_logical.BackupStatusCompleted || + newestBackup.Status == backups_core_logical.BackupStatusFailed || + newestBackup.Status == backups_core_logical.BackupStatusCanceled { + t.Logf( + "WaitForBackupCompletion: backup finished with status %s", + newestBackup.Status, + ) + return + } + } + + time.Sleep(50 * time.Millisecond) + } + + t.Logf("WaitForBackupCompletion: timeout waiting for backup to complete") +} + +// CreateTestBackup creates a simple test backup record for testing purposes +func CreateTestBackup(databaseID, storageID uuid.UUID) *backups_core_logical.LogicalBackup { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: databaseID, + StorageID: storageID, + Status: backups_core_logical.BackupStatusCompleted, + BackupSizeMb: 10.5, + BackupDurationMs: 1000, + CreatedAt: time.Now().UTC(), + } + + repo := &backups_core_logical.BackupRepository{} + if err := repo.Save(backup); err != nil { + panic(err) + } + + return backup +} + +type TestBackupOptions struct { + Status backups_core_logical.BackupStatus + CreatedAt time.Time +} + +func CreateTestBackupWithOptions( + databaseID, storageID uuid.UUID, + opts TestBackupOptions, +) *backups_core_logical.LogicalBackup { + backup := &backups_core_logical.LogicalBackup{ + ID: uuid.New(), + DatabaseID: databaseID, + StorageID: storageID, + Status: opts.Status, + BackupSizeMb: 10.5, + BackupDurationMs: 1000, + CreatedAt: opts.CreatedAt, + } + + repo := &backups_core_logical.BackupRepository{} + if err := repo.Save(backup); err != nil { + panic(err) + } + + return backup +} diff --git a/backend/internal/features/backups/backups/controllers/physical/controller.go b/backend/internal/features/backups/backups/controllers/physical/controller.go new file mode 100644 index 0000000..e0ad4d5 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/physical/controller.go @@ -0,0 +1,461 @@ +package backups_controllers_physical + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + "databasus-backend/internal/features/backups/backups/core/physical/chain_view" + "databasus-backend/internal/features/backups/backups/download/restore_token" + "databasus-backend/internal/features/backups/backups/download/stream_guard" + backups_dto_physical "databasus-backend/internal/features/backups/backups/dto/physical" + backups_services "databasus-backend/internal/features/backups/backups/services" + users_middleware "databasus-backend/internal/features/users/middleware" + users_models "databasus-backend/internal/features/users/models" +) + +type PhysicalBackupController struct { + physicalBackupService *backups_services.PhysicalBackupService + restoreTokenService *restore_token.Service +} + +func (c *PhysicalBackupController) RegisterRoutes(router *gin.RouterGroup) { + router.GET("/backups/physical/database/:id/backups", c.GetBackups) + router.POST("/backups/physical/database/:id/trigger", c.TriggerBackup) + router.POST("/backups/physical/database/:id/restore-token", c.GenerateRestoreToken) + + router.POST("/backups/physical/backups/:backupId/restore-token", c.GenerateBackupRestoreToken) + router.POST("/backups/physical/backups/:backupId/cancel", c.CancelBackup) + router.DELETE("/backups/physical/backups/:backupId", c.DeleteBackup) +} + +func (c *PhysicalBackupController) RegisterPublicRoutes(router *gin.RouterGroup) { + router.GET("/backups/physical/restore-stream", c.GetRestoreStream) + router.GET("/backups/physical/recovery-script", c.GetRecoveryScript) +} + +// GetRecoveryScript +// @Summary Download the physical-restore helper script +// @Description Returns a POSIX sh script that takes a restore-stream URL (with its token) or a local bundle path, decompresses the backup blobs and WAL with the host's zstd up front, runs pg_combinebackup and wires up recovery with a plain-cp restore_command so the started cluster needs no extra tools. Pass --combine-image to run pg_combinebackup in a container when the host lacks the PostgreSQL tools. It carries no secrets, so it needs no auth. +// @Tags backups-physical +// @Produce plain +// @Success 200 {string} string "shell script" +// @Router /backups/physical/recovery-script [get] +func (c *PhysicalBackupController) GetRecoveryScript(ctx *gin.Context) { + ctx.Header("Content-Type", "text/x-shellscript; charset=utf-8") + ctx.Header("Content-Disposition", `inline; filename="databasus-recovery.sh"`) + ctx.String(http.StatusOK, recoveryScript) +} + +// GetBackups +// @Summary List physical backups for a database +// @Description Returns a page of physical backups — FULLs, incrementals and committed WAL segments — as one flat list newest-first, with the database's total on-disk usage and the total count for pagination. The frontend filters by type. +// @Tags backups-physical +// @Produce json +// @Security BearerAuth +// @Param id path string true "Database ID" +// @Param limit query int false "Page size (default 50, max 1000)" +// @Param offset query int false "Offset for pagination" default(0) +// @Param type query []string false "Filter by backup type - repeatable, matches any" Enums(FULL, INCREMENTAL, WAL) collectionFormat(multi) +// @Param status query []string false "Filter by status - repeatable, matches any (e.g. COMPLETED, IN_PROGRESS)" collectionFormat(multi) +// @Param beforeDate query string false "Keep only backups created before this date (RFC3339)" format(date-time) +// @Success 200 {object} backups_dto_physical.GetPhysicalBackupsResponse +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Router /backups/physical/database/{id}/backups [get] +func (c *PhysicalBackupController) GetBackups(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + databaseID, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid database ID"}) + return + } + + var request backups_dto_physical.GetPhysicalBackupsRequest + if err := ctx.ShouldBindQuery(&request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + response, err := c.physicalBackupService.GetBackups(user, databaseID, &request) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, response) +} + +// TriggerBackup +// @Summary Trigger a physical backup +// @Description Requests an out-of-cadence backup, honored on the scheduler's next tick. type=full forces a new full; type=incremental extends the current chain (409 if none); type=auto lets the scheduler pick. +// @Tags backups-physical +// @Accept json +// @Produce json +// @Security BearerAuth +// @Param id path string true "Database ID" +// @Param request body backups_dto_physical.TriggerBackupRequest true "Backup type to trigger" +// @Success 202 {object} map[string]string +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 409 {object} map[string]string +// @Router /backups/physical/database/{id}/trigger [post] +func (c *PhysicalBackupController) TriggerBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + databaseID, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid database ID"}) + return + } + + var request backups_dto_physical.TriggerBackupRequest + if err := ctx.ShouldBindJSON(&request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + if err := c.requestBackupOfType(user, databaseID, request.Type); err != nil { + if errors.Is(err, backups_services.ErrNoExtendableChain) { + ctx.JSON(http.StatusConflict, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusAccepted, gin.H{"message": "backup requested"}) +} + +// GenerateRestoreToken +// @Summary Issue a restore-stream token +// @Description Issues a single-use, short-TTL token that authorizes the agent-less restore stream. Omit targetTime to restore to the latest available point. +// @Tags backups-physical +// @Accept json +// @Produce json +// @Security BearerAuth +// @Param id path string true "Database ID" +// @Param request body backups_dto_physical.GenerateRestoreTokenRequest true "Restore target" +// @Success 200 {object} backups_dto_physical.GenerateRestoreTokenResponse +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 409 {object} map[string]string +// @Router /backups/physical/database/{id}/restore-token [post] +func (c *PhysicalBackupController) GenerateRestoreToken(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + databaseID, err := uuid.Parse(ctx.Param("id")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid database ID"}) + return + } + + var request backups_dto_physical.GenerateRestoreTokenRequest + if err := ctx.ShouldBindJSON(&request); err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + token, err := c.physicalBackupService.GenerateRestoreToken(user, databaseID, &request) + if err != nil { + if errors.Is(err, stream_guard.ErrDownloadAlreadyInProgress) { + ctx.JSON(http.StatusConflict, gin.H{"error": err.Error()}) + return + } + + // An unreachable target (WAL gap, no chain, target before earliest) is + // resolved at issue time, so surface it here with the same status the + // stream endpoint would have returned. + if status, message, isResolverError := classifyRestoreStreamError(err); isResolverError { + ctx.JSON(status, gin.H{"error": message}) + return + } + + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, backups_dto_physical.GenerateRestoreTokenResponse{ + Token: token, + URL: "/api/v1/backups/physical/restore-stream?token=" + token, + }) +} + +// GenerateBackupRestoreToken +// @Summary Issue a restore-stream token for a specific backup +// @Description Issues a single-use token authorizing a per-backup restore stream: a FULL restores just itself; an incremental restores its FULL plus its incremental ancestors. No WAL replay is involved. +// @Tags backups-physical +// @Produce json +// @Security BearerAuth +// @Param backupId path string true "Backup ID (FULL or incremental)" +// @Success 200 {object} backups_dto_physical.GenerateRestoreTokenResponse +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 404 {object} map[string]string +// @Failure 409 {object} map[string]string +// @Router /backups/physical/backups/{backupId}/restore-token [post] +func (c *PhysicalBackupController) GenerateBackupRestoreToken(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + backupID, err := uuid.Parse(ctx.Param("backupId")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + token, err := c.physicalBackupService.GenerateBackupRestoreToken(user, backupID) + if err != nil { + if errors.Is(err, backups_services.ErrBackupNotFound) { + ctx.JSON(http.StatusNotFound, gin.H{"error": err.Error()}) + return + } + + if errors.Is(err, stream_guard.ErrDownloadAlreadyInProgress) { + ctx.JSON(http.StatusConflict, gin.H{"error": err.Error()}) + return + } + + // A non-restorable backup (missing chain, not COMPLETED) is resolved up + // front; surface it with the same status the stream endpoint would. + if status, message, isResolverError := classifyRestoreStreamError(err); isResolverError { + ctx.JSON(status, gin.H{"error": message}) + return + } + + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusOK, backups_dto_physical.GenerateRestoreTokenResponse{ + Token: token, + URL: "/api/v1/backups/physical/restore-stream?token=" + token, + }) +} + +// CancelBackup +// @Summary Cancel an in-progress physical backup +// @Description Stops a FULL or incremental backup that is currently running and releases its in-flight claim. Returns 400 if the backup is not in progress. +// @Tags backups-physical +// @Security BearerAuth +// @Param backupId path string true "Backup ID" +// @Success 204 +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 404 {object} map[string]string +// @Router /backups/physical/backups/{backupId}/cancel [post] +func (c *PhysicalBackupController) CancelBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + backupID, err := uuid.Parse(ctx.Param("backupId")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + if err := c.physicalBackupService.CancelBackup(user, backupID); err != nil { + if errors.Is(err, backups_services.ErrBackupNotFound) { + ctx.JSON(http.StatusNotFound, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.Status(http.StatusNoContent) +} + +// DeleteBackup +// @Summary Delete a physical backup and its dependent cascade +// @Description Deletes a backup. A FULL removes its whole chain (incrementals + WAL); an incremental removes it and its descendant incrementals (WAL kept); a WAL segment removes it and the later WAL up to the next FULL/incremental. A running backup in the removed set is cancelled first. +// @Tags backups-physical +// @Security BearerAuth +// @Param backupId path string true "Backup ID (FULL, incremental or WAL segment)" +// @Success 204 +// @Failure 400 {object} map[string]string +// @Failure 401 {object} map[string]string +// @Failure 404 {object} map[string]string +// @Router /backups/physical/backups/{backupId} [delete] +func (c *PhysicalBackupController) DeleteBackup(ctx *gin.Context) { + user, ok := users_middleware.GetUserFromContext(ctx) + if !ok { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "User not authenticated"}) + return + } + + backupID, err := uuid.Parse(ctx.Param("backupId")) + if err != nil { + ctx.JSON(http.StatusBadRequest, gin.H{"error": "invalid backup ID"}) + return + } + + if err := c.physicalBackupService.DeleteBackup(user, backupID); err != nil { + if errors.Is(err, backups_services.ErrBackupNotFound) { + ctx.JSON(http.StatusNotFound, gin.H{"error": err.Error()}) + return + } + + ctx.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + + ctx.Status(http.StatusNoContent) +} + +// GetRestoreStream +// @Summary Stream a ready-to-restore directory as a tar +// @Description Resolves the restore set for the token's database + target and streams an artifact-only tar of the stored, still-compressed bytes (full/base.tar.zst, incr-N/base.tar.zst, each with a backup_manifest, plus wal/ and MANIFEST.sha256). curl '' | tar -x, then run the reconstruction command shown in the UI (decompress the blobs, pg_combinebackup + recovery settings). Public: authorized by the ?token= query param, not Bearer. +// @Tags backups-physical +// @Produce application/x-tar +// @Param token query string true "Restore token" +// @Success 200 +// @Failure 401 {object} map[string]string +// @Failure 404 {object} map[string]string +// @Failure 409 {object} map[string]string +// @Failure 422 {object} map[string]string +// @Router /backups/physical/restore-stream [get] +func (c *PhysicalBackupController) GetRestoreStream(ctx *gin.Context) { + token := ctx.Query("token") + if token == "" { + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "restore token is required"}) + return + } + + restoreToken, err := c.restoreTokenService.ValidateAndConsumeRestoreToken(token) + if err != nil { + if errors.Is(err, stream_guard.ErrDownloadAlreadyInProgress) { + ctx.JSON(http.StatusConflict, gin.H{ + "error": "a download or restore is already in progress for this user", + }) + return + } + + ctx.JSON(http.StatusUnauthorized, gin.H{"error": "invalid or expired restore token"}) + return + } + + heartbeatCtx, cancelHeartbeat := context.WithCancel(context.Background()) + defer func() { + cancelHeartbeat() + c.restoreTokenService.ReleaseDownloadLock(restoreToken.UserID) + }() + + go c.startStreamHeartbeat(heartbeatCtx, restoreToken.UserID) + + ctx.Header("Content-Type", "application/x-tar") + ctx.Header("Content-Disposition", + fmt.Sprintf(`attachment; filename="restore-%s.tar"`, restoreToken.DatabaseID)) + + if err := c.openRestoreStream(restoreToken, ctx.Writer); err != nil { + // Resolution runs before any bytes are written, so if nothing has been + // flushed yet we can still answer with a proper status. Once the tar has + // started, the status is already 200 and we can only log. + if !ctx.Writer.Written() { + status, message := mapRestoreStreamError(err) + ctx.JSON(status, gin.H{"error": message}) + } + + return + } +} + +// mapRestoreStreamError translates resolver failures into HTTP statuses for the +// stream endpoint, where an unrecognized error is an internal failure (500). +func mapRestoreStreamError(err error) (int, string) { + if status, message, isResolverError := classifyRestoreStreamError(err); isResolverError { + return status, message + } + + return http.StatusInternalServerError, err.Error() +} + +// classifyRestoreStreamError maps the resolver's typed failures to HTTP statuses +// and reports whether err was one of them. A WAL gap or out-of-range target is +// the caller's mistake to correct (422); no chain is a not-found (404). The bool +// lets callers fall through to their own default for everything else. +func classifyRestoreStreamError(err error) (int, string, bool) { + var gapErr chain_view.WalGapBeforeTargetError + if errors.As(err, &gapErr) { + return http.StatusUnprocessableEntity, gapErr.Error(), true + } + + if errors.Is(err, chain_view.ErrTargetBeforeEarliest) { + return http.StatusUnprocessableEntity, err.Error(), true + } + + if errors.Is(err, chain_view.ErrNoChainForRestore) { + return http.StatusNotFound, err.Error(), true + } + + return 0, "", false +} + +// openRestoreStream dispatches by what the token carries: a BackupID streams a +// per-backup restore (FULL + incremental ancestors, no WAL); otherwise +// TargetTime drives a point-in-time restore. +func (c *PhysicalBackupController) openRestoreStream(token *restore_token.Token, w io.Writer) error { + if token.BackupID != nil { + return c.physicalBackupService.OpenRestoreStreamForBackup(token.DatabaseID, *token.BackupID, w) + } + + return c.physicalBackupService.OpenRestoreStream(token.DatabaseID, token.TargetTime, w) +} + +func (c *PhysicalBackupController) requestBackupOfType( + user *users_models.User, + databaseID uuid.UUID, + backupType backups_dto_physical.TriggerBackupType, +) error { + switch backupType { + case backups_dto_physical.TriggerBackupTypeFull: + return c.physicalBackupService.RequestFullBackup(user, databaseID) + + case backups_dto_physical.TriggerBackupTypeIncremental: + return c.physicalBackupService.RequestIncrementalBackup(user, databaseID) + + default: + return c.physicalBackupService.RequestBackup(user, databaseID) + } +} + +func (c *PhysicalBackupController) startStreamHeartbeat(ctx context.Context, userID uuid.UUID) { + ticker := time.NewTicker(stream_guard.GetDownloadHeartbeatInterval()) + defer ticker.Stop() + + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + c.restoreTokenService.RefreshDownloadLock(userID) + } + } +} diff --git a/backend/internal/features/backups/backups/controllers/physical/controller_test.go b/backend/internal/features/backups/backups/controllers/physical/controller_test.go new file mode 100644 index 0000000..a5fabc8 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/physical/controller_test.go @@ -0,0 +1,1209 @@ +package backups_controllers_physical_test + +import ( + "archive/tar" + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "strings" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + "github.com/klauspost/compress/zstd" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + + audit_logs "databasus-backend/internal/features/audit_logs" + backuping_physical "databasus-backend/internal/features/backups/backups/backuping/physical" + backups_controllers_physical "databasus-backend/internal/features/backups/backups/controllers/physical" + backups_core_enums "databasus-backend/internal/features/backups/backups/core/enums" + physical_enums "databasus-backend/internal/features/backups/backups/core/physical/enums" + physical_models "databasus-backend/internal/features/backups/backups/core/physical/models" + physical_repositories "databasus-backend/internal/features/backups/backups/core/physical/repositories" + physical_testing "databasus-backend/internal/features/backups/backups/core/physical/testing" + backups_download "databasus-backend/internal/features/backups/backups/download" + backups_dto_physical "databasus-backend/internal/features/backups/backups/dto/physical" + backups_config_physical "databasus-backend/internal/features/backups/config/physical" + "databasus-backend/internal/features/databases" + "databasus-backend/internal/features/notifiers" + "databasus-backend/internal/features/storages" + users_dto "databasus-backend/internal/features/users/dto" + users_enums "databasus-backend/internal/features/users/enums" + users_middleware "databasus-backend/internal/features/users/middleware" + users_services "databasus-backend/internal/features/users/services" + users_testing "databasus-backend/internal/features/users/testing" + workspaces_controllers "databasus-backend/internal/features/workspaces/controllers" + workspaces_models "databasus-backend/internal/features/workspaces/models" + workspaces_testing "databasus-backend/internal/features/workspaces/testing" + "databasus-backend/internal/util/encryption" + "databasus-backend/internal/util/logger" + test_utils "databasus-backend/internal/util/testing" + "databasus-backend/internal/util/walmath" +) + +const segmentBytes = 16 * 1024 * 1024 + +type physicalControllerPrereqs struct { + user *users_dto.SignInResponseDTO + workspace *workspaces_models.Workspace + storage *storages.Storage + notifier *notifiers.Notifier + database *databases.Database + router *gin.Engine +} + +func createPhysicalControllerPrereqs(t *testing.T) *physicalControllerPrereqs { + t.Helper() + + router := newPhysicalControllerRouter() + user := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspace := workspaces_testing.CreateTestWorkspace("Physical Ctrl "+uuid.NewString(), user, router) + storage := storages.CreateTestStorage(workspace.ID) + notifier := notifiers.CreateTestNotifier(workspace.ID) + database := databases.CreateTestPhysicalPostgresDatabase(workspace.ID, notifier, "17") + + t.Cleanup(func() { + physical_testing.DeleteAllPhysicalCatalogForDatabase(t, database.ID) + databases.RemoveTestDatabase(database) + notifiers.RemoveTestNotifier(notifier) + storages.RemoveTestStorage(storage.ID) + }) + + return &physicalControllerPrereqs{ + user: user, + workspace: workspace, + storage: storage, + notifier: notifier, + database: database, + router: router, + } +} + +// newPhysicalControllerRouter wires the physical controller's protected and +// public routes (CreateTestRouter only registers protected ones) plus the +// supporting controllers and feature dependencies. +func newPhysicalControllerRouter() *gin.Engine { + gin.SetMode(gin.TestMode) + router := gin.New() + + v1 := router.Group("/api/v1") + + backups_controllers_physical.GetPhysicalBackupController().RegisterPublicRoutes(v1) + + protected := v1.Group("").Use(users_middleware.AuthMiddleware(users_services.GetUserService())) + if routerGroup, ok := protected.(*gin.RouterGroup); ok { + workspaces_controllers.GetWorkspaceController().RegisterRoutes(routerGroup) + workspaces_controllers.GetMembershipController().RegisterRoutes(routerGroup) + databases.GetDatabaseController().RegisterRoutes(routerGroup) + backups_controllers_physical.GetPhysicalBackupController().RegisterRoutes(routerGroup) + } + + storages.SetupDependencies() + databases.SetupDependencies() + notifiers.SetupDependencies() + backups_config_physical.SetupDependencies() + backuping_physical.SetupDependencies() + + return router +} + +func Test_GetBackups_WhenOwner_ReturnsFlatListSortedByCreatedAt(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + base := time.Now().UTC().Add(-1 * time.Hour) + + fullModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + fullModel.CreatedAt = base + full := physical_testing.CreateTestFullBackup(t, fullModel) + + incrModel := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes)) + incrModel.CreatedAt = base.Add(10 * time.Minute) + incr := physical_testing.CreateTestIncrementalBackup(t, incrModel) + + walModel := physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + walModel.ReceivedAt = base.Add(20 * time.Minute) + walSegment := physical_testing.CreateTestWalSegment(t, walModel) + + var response backups_dto_physical.GetPhysicalBackupsResponse + test_utils.MakeGetRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/backups", + "Bearer "+prereqs.user.Token, http.StatusOK, &response) + + require.Len(t, response.Backups, 3) + assert.EqualValues(t, 3, response.Total) + + // Newest-first: WAL (received last), then incremental, then full. + assert.Equal(t, walSegment.ID, response.Backups[0].ID) + assert.Equal(t, physical_enums.PhysicalBackupTypeWal, response.Backups[0].Type) + assert.Equal(t, incr.ID, response.Backups[1].ID) + assert.Equal(t, physical_enums.PhysicalBackupTypeIncremental, response.Backups[1].Type) + assert.Equal(t, full.ID, response.Backups[2].ID) + assert.Equal(t, physical_enums.PhysicalBackupTypeFull, response.Backups[2].Type) + + assert.Equal(t, full.ID, *response.Backups[1].RootFullBackupID) + assert.NotNil(t, response.Backups[0].WalFilename) + assert.Positive(t, response.TotalUsageMb) +} + +func Test_GetBackups_WhenBackupFailed_ReturnsFailMessage(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + base := time.Now().UTC().Add(-1 * time.Hour) + + completedModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + completedModel.CreatedAt = base + completed := physical_testing.CreateTestFullBackup(t, completedModel) + + failReason := physical_enums.PhysicalBackupErrorPgBasebackupFailed + failedModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + failedModel.CreatedAt = base.Add(10 * time.Minute) + failedModel.Status = physical_enums.PhysicalBackupStatusError + failedModel.ErrorReason = &failReason + failedModel.FailMessage = new("pg_basebackup failed; stderr: connection refused") + failed := physical_testing.CreateTestFullBackup(t, failedModel) + + var response backups_dto_physical.GetPhysicalBackupsResponse + test_utils.MakeGetRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/backups", + "Bearer "+prereqs.user.Token, http.StatusOK, &response) + + require.Len(t, response.Backups, 2) + + // Newest-first: the failed backup, then the completed one. + assert.Equal(t, failed.ID, response.Backups[0].ID) + require.NotNil(t, response.Backups[0].FailMessage) + assert.Equal(t, "pg_basebackup failed; stderr: connection refused", *response.Backups[0].FailMessage) + + assert.Equal(t, completed.ID, response.Backups[1].ID) + assert.Nil(t, response.Backups[1].FailMessage, + "a completed backup carries no failure message") +} + +func Test_GetBackups_Paginated_ReturnsRequestedPage(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + base := time.Now().UTC().Add(-1 * time.Hour) + + fullModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + fullModel.CreatedAt = base + full := physical_testing.CreateTestFullBackup(t, fullModel) + + incrModel := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes)) + incrModel.CreatedAt = base.Add(10 * time.Minute) + incr := physical_testing.CreateTestIncrementalBackup(t, incrModel) + + walModel := physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + walModel.ReceivedAt = base.Add(20 * time.Minute) + walSegment := physical_testing.CreateTestWalSegment(t, walModel) + + listURL := "/api/v1/backups/physical/database/" + prereqs.database.ID.String() + "/backups" + + var firstPage backups_dto_physical.GetPhysicalBackupsResponse + test_utils.MakeGetRequestAndUnmarshal(t, prereqs.router, + listURL+"?limit=2&offset=0", "Bearer "+prereqs.user.Token, http.StatusOK, &firstPage) + + require.Len(t, firstPage.Backups, 2) + assert.EqualValues(t, 3, firstPage.Total) + assert.Equal(t, 2, firstPage.Limit) + assert.Equal(t, 0, firstPage.Offset) + assert.Equal(t, walSegment.ID, firstPage.Backups[0].ID) + assert.Equal(t, incr.ID, firstPage.Backups[1].ID) + + var secondPage backups_dto_physical.GetPhysicalBackupsResponse + test_utils.MakeGetRequestAndUnmarshal(t, prereqs.router, + listURL+"?limit=2&offset=2", "Bearer "+prereqs.user.Token, http.StatusOK, &secondPage) + + require.Len(t, secondPage.Backups, 1) + assert.EqualValues(t, 3, secondPage.Total) + assert.Equal(t, full.ID, secondPage.Backups[0].ID) +} + +func Test_GetBackups_WhenNonMember_ReturnsError(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + outsider := users_testing.CreateTestUser(users_enums.UserRoleMember) + + test_utils.MakeGetRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/backups", + "Bearer "+outsider.Token, http.StatusBadRequest) +} + +func Test_TriggerBackup_Full_SetsForceFullFlag(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + // Touch the config so a row exists for the request flag to land on. + _, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.database.ID) + require.NoError(t, err) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/trigger", + "Bearer "+prereqs.user.Token, + backups_dto_physical.TriggerBackupRequest{Type: backups_dto_physical.TriggerBackupTypeFull}, + http.StatusAccepted) + + config, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.database.ID) + require.NoError(t, err) + assert.NotNil(t, config.ForceFullRequestedAt, "trigger full must set the force-full request flag") +} + +func Test_TriggerBackup_IncrementalWithoutExtendableChain_Returns409(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/trigger", + "Bearer "+prereqs.user.Token, + backups_dto_physical.TriggerBackupRequest{Type: backups_dto_physical.TriggerBackupTypeIncremental}, + http.StatusConflict) +} + +func Test_CancelBackup_WhenInProgress_CancelsAndReleasesClaim(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + physical_testing.CreateTestInFlightClaim(t, prereqs.database.ID, full.ID, physical_enums.PhysicalBackupTypeFull) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/cancel", + "Bearer "+prereqs.user.Token, nil, http.StatusNoContent) + + claim, err := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.database.ID) + require.NoError(t, err) + assert.Nil(t, claim, "cancel must release the in-flight claim") +} + +func Test_CancelBackup_WhenNotInProgress_Returns400(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/cancel", + "Bearer "+prereqs.user.Token, nil, http.StatusBadRequest) +} + +func Test_DeleteBackup_Full_CascadesChain(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + incr := physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + walSegment := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000001", + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String(), + "Bearer "+prereqs.user.Token, http.StatusNoContent) + + assertFullGone(t, full.ID) + assertIncrementalGone(t, incr.ID) + assertWalGone(t, walSegment.ID) +} + +func Test_DeleteBackup_Incremental_RemovesDescendantsKeepsWalAndFull(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + firstIncr := physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + secondIncr := physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, &firstIncr.ID, 1, + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes))) + walSegment := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000001", + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+firstIncr.ID.String(), + "Bearer "+prereqs.user.Token, http.StatusNoContent) + + assertIncrementalGone(t, firstIncr.ID) + assertIncrementalGone(t, secondIncr.ID) + assertFullPresent(t, full.ID) + assertWalPresent(t, walSegment.ID) +} + +func Test_DeleteBackup_Wal_RemovesFollowingWalUpToNextAnchor(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + firstWal := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000001", + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + secondWal := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes))) + thirdWal := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000003", + walmath.LSN(3*segmentBytes), walmath.LSN(4*segmentBytes))) + + // An incremental anchored at LSN 3 bounds the WAL delete: segments at or + // after it survive. + anchorIncr := physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(3*segmentBytes), walmath.LSN(4*segmentBytes))) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+firstWal.ID.String(), + "Bearer "+prereqs.user.Token, http.StatusNoContent) + + assertWalGone(t, firstWal.ID) + assertWalGone(t, secondWal.ID) + assertWalPresent(t, thirdWal.ID) + assertFullPresent(t, full.ID) + assertIncrementalPresent(t, anchorIncr.ID) +} + +func Test_GenerateRestoreToken_ReturnsTokenAndURL(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + // A token is only issued for a resolvable restore set, so the chain must + // exist before requesting one. + physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + var response backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, + backups_dto_physical.GenerateRestoreTokenRequest{}, + http.StatusOK, &response) + + assert.NotEmpty(t, response.Token) + assert.Contains(t, response.URL, response.Token) +} + +func Test_GenerateRestoreToken_WhenNoChain_Returns404(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, + backups_dto_physical.GenerateRestoreTokenRequest{}, + http.StatusNotFound) +} + +// Test_GenerateRestoreToken_WhenTargetReachable_ReturnsToken seeds a contiguous +// WAL run past the chain and asks for a point that the run actually reaches, so +// the restore set resolves and a token is minted. +func Test_GenerateRestoreToken_WhenTargetReachable_ReturnsToken(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + base := time.Now().UTC().Add(-2 * time.Hour) + + full := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + full.CompletedAt = ptrTime(base) + physical_testing.CreateTestFullBackup(t, full) + + incr := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes)) + incr.CompletedAt = ptrTime(base.Add(10 * time.Minute)) + physical_testing.CreateTestIncrementalBackup(t, incr) + + // A WAL segment contiguous with the incremental, received after it. + walSegment := physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + walSegment.ReceivedAt = base.Add(12 * time.Minute) + physical_testing.CreateTestWalSegment(t, walSegment) + + targetTime := base.Add(11 * time.Minute) + + var response backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, + backups_dto_physical.GenerateRestoreTokenRequest{TargetTime: &targetTime}, + http.StatusOK, &response) + + assert.NotEmpty(t, response.Token) +} + +// Test_GenerateRestoreToken_WhenTargetPastWalGap_Returns422 verifies the gap is +// caught at token-issue time (the restore set is resolved before a token is +// minted), so the user never burns a token on an unreachable target. +func Test_GenerateRestoreToken_WhenTargetPastWalGap_Returns422(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + base := time.Now().UTC().Add(-2 * time.Hour) + + full := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + full.CompletedAt = ptrTime(base) + physical_testing.CreateTestFullBackup(t, full) + + incr := physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes)) + incr.CompletedAt = ptrTime(base.Add(10 * time.Minute)) + physical_testing.CreateTestIncrementalBackup(t, incr) + + // One WAL segment then a gap (no segment covering [3,4)). + w2 := physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + w2.ReceivedAt = base.Add(12 * time.Minute) + physical_testing.CreateTestWalSegment(t, w2) + + targetTime := base.Add(60 * time.Minute) + + response := test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, + backups_dto_physical.GenerateRestoreTokenRequest{TargetTime: &targetTime}, + http.StatusUnprocessableEntity) + + var body map[string]string + require.NoError(t, json.Unmarshal(response.Body, &body)) + assert.Contains(t, body["error"], "wal gap") +} + +func Test_GenerateBackupRestoreToken_ForFull_ReturnsTokenAndURL(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + var response backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusOK, &response) + + assert.NotEmpty(t, response.Token) + assert.Contains(t, response.URL, response.Token) +} + +func Test_GenerateBackupRestoreToken_WhenBackupMissing_Returns404(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+uuid.NewString()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusNotFound) +} + +func Test_GetRestoreStream_WhenTokenInvalid_Returns401(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakeGetRequest(t, prereqs.router, + "/api/v1/backups/physical/restore-stream?token=not-a-real-token", + "", http.StatusUnauthorized) +} + +func assertFullGone(t *testing.T, id uuid.UUID) { + t.Helper() + + full, err := physical_repositories.GetFullBackupRepository().FindByID(id) + require.NoError(t, err) + assert.Nil(t, full, "full row must be gone") +} + +func assertFullPresent(t *testing.T, id uuid.UUID) { + t.Helper() + + full, err := physical_repositories.GetFullBackupRepository().FindByID(id) + require.NoError(t, err) + assert.NotNil(t, full, "full row must survive") +} + +func assertIncrementalGone(t *testing.T, id uuid.UUID) { + t.Helper() + + incremental, err := physical_repositories.GetIncrementalBackupRepository().FindByID(id) + require.NoError(t, err) + assert.Nil(t, incremental, "incremental row must be gone") +} + +func assertIncrementalPresent(t *testing.T, id uuid.UUID) { + t.Helper() + + incremental, err := physical_repositories.GetIncrementalBackupRepository().FindByID(id) + require.NoError(t, err) + assert.NotNil(t, incremental, "incremental row must survive") +} + +func assertWalGone(t *testing.T, id uuid.UUID) { + t.Helper() + + walSegment, err := physical_repositories.GetWalSegmentRepository().FindByID(id) + require.NoError(t, err) + assert.Nil(t, walSegment, "wal row must be gone") +} + +func assertWalPresent(t *testing.T, id uuid.UUID) { + t.Helper() + + walSegment, err := physical_repositories.GetWalSegmentRepository().FindByID(id) + require.NoError(t, err) + assert.NotNil(t, walSegment, "wal row must survive") +} + +func ptrTime(t time.Time) *time.Time { + return &t +} + +// --- Permission matrix ------------------------------------------------------- + +func Test_GetBackups_PermissionsEnforced(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + listURL := "/api/v1/backups/physical/database/" + prereqs.database.ID.String() + "/backups" + + // Listing is a read: a viewer is allowed, only a non-member is rejected. + runRolePermissionMatrix(t, prereqs, false, http.StatusOK, func(authHeader string) int { + return workspaces_testing.MakeAPIRequest(prereqs.router, "GET", listURL, authHeader, nil).Code + }) +} + +func Test_TriggerBackup_PermissionsEnforced(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + _, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.database.ID) + require.NoError(t, err) + + triggerURL := "/api/v1/backups/physical/database/" + prereqs.database.ID.String() + "/trigger" + request := backups_dto_physical.TriggerBackupRequest{Type: backups_dto_physical.TriggerBackupTypeFull} + + // Triggering changes state: a viewer is rejected. + runRolePermissionMatrix(t, prereqs, true, http.StatusAccepted, func(authHeader string) int { + return workspaces_testing.MakeAPIRequest(prereqs.router, "POST", triggerURL, authHeader, request).Code + }) +} + +// A viewer could delete physical backups before authorization was tightened to +// manage-DBs; this pins the fix. +func Test_DeleteBackup_WhenViewer_Returns400AndKeepsBackup(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + viewer := addWorkspaceViewer(t, prereqs) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String(), + "Bearer "+viewer.Token, http.StatusBadRequest) + + assertFullPresent(t, full.ID) +} + +func Test_CancelBackup_WhenViewer_Returns400AndKeepsClaim(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + physical_testing.CreateTestInFlightClaim(t, prereqs.database.ID, full.ID, physical_enums.PhysicalBackupTypeFull) + + viewer := addWorkspaceViewer(t, prereqs) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/cancel", + "Bearer "+viewer.Token, nil, http.StatusBadRequest) + + claim, err := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.database.ID) + require.NoError(t, err) + assert.NotNil(t, claim, "a viewer's rejected cancel must not release the claim") +} + +// --- Audit logging ----------------------------------------------------------- + +func Test_DeleteBackup_WritesAuditLog(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String(), + "Bearer "+prereqs.user.Token, http.StatusNoContent) + + assertAuditLogContains(t, prereqs.workspace.ID, "Physical full backup deleted", prereqs.database.Name) +} + +func Test_CancelBackup_WritesAuditLog(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + physical_testing.CreateTestInFlightClaim(t, prereqs.database.ID, full.ID, physical_enums.PhysicalBackupTypeFull) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/cancel", + "Bearer "+prereqs.user.Token, nil, http.StatusNoContent) + + assertAuditLogContains(t, prereqs.workspace.ID, "Physical backup cancelled", prereqs.database.Name) +} + +// --- Filters ----------------------------------------------------------------- + +func Test_GetBackups_FilterByType_ReturnsOnlyThatType(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full, incr, walSegment := seedOneOfEachType(t, prereqs) + + fullPage := getBackups(t, prereqs, "?type=FULL") + require.Len(t, fullPage.Backups, 1) + assert.Equal(t, full.ID, fullPage.Backups[0].ID) + assert.EqualValues(t, 1, fullPage.Total) + + incrPage := getBackups(t, prereqs, "?type=INCREMENTAL") + require.Len(t, incrPage.Backups, 1) + assert.Equal(t, incr.ID, incrPage.Backups[0].ID) + + walPage := getBackups(t, prereqs, "?type=WAL") + require.Len(t, walPage.Backups, 1) + assert.Equal(t, walSegment.ID, walPage.Backups[0].ID) +} + +func Test_GetBackups_FilterByStatus_ReturnsOnlyThatStatus(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + completed := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + inProgress := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + + completedPage := getBackups(t, prereqs, "?status=COMPLETED") + require.Len(t, completedPage.Backups, 1) + assert.Equal(t, completed.ID, completedPage.Backups[0].ID) + + inProgressPage := getBackups(t, prereqs, "?status=IN_PROGRESS") + require.Len(t, inProgressPage.Backups, 1) + assert.Equal(t, inProgress.ID, inProgressPage.Backups[0].ID) +} + +func Test_GetBackups_FilterByBeforeDate_ExcludesNewer(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + now := time.Now().UTC() + + olderModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + olderModel.CreatedAt = now.Add(-3 * time.Hour) + older := physical_testing.CreateTestFullBackup(t, olderModel) + + newerModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + newerModel.CreatedAt = now + physical_testing.CreateTestFullBackup(t, newerModel) + + cutoff := now.Add(-1 * time.Hour).Format(time.RFC3339) + page := getBackups(t, prereqs, "?beforeDate="+cutoff) + + require.Len(t, page.Backups, 1) + assert.Equal(t, older.ID, page.Backups[0].ID) + assert.EqualValues(t, 1, page.Total) +} + +func Test_GetBackups_FilterByMultipleTypes_ReturnsThoseTypes(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full, _, walSegment := seedOneOfEachType(t, prereqs) + + page := getBackups(t, prereqs, "?type=FULL&type=WAL") + + require.Len(t, page.Backups, 2) + assert.EqualValues(t, 2, page.Total) + + returnedIDs := []uuid.UUID{page.Backups[0].ID, page.Backups[1].ID} + assert.Contains(t, returnedIDs, full.ID) + assert.Contains(t, returnedIDs, walSegment.ID) +} + +func Test_GetBackups_FilterByMultipleStatuses_ReturnsThoseStatuses(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + completed := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + inProgress := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + + erroredModel := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes)) + erroredModel.Status = physical_enums.PhysicalBackupStatusError + physical_testing.CreateTestFullBackup(t, erroredModel) + + page := getBackups(t, prereqs, "?status=COMPLETED&status=IN_PROGRESS") + + require.Len(t, page.Backups, 2) + assert.EqualValues(t, 2, page.Total) + + returnedIDs := []uuid.UUID{page.Backups[0].ID, page.Backups[1].ID} + assert.Contains(t, returnedIDs, completed.ID) + assert.Contains(t, returnedIDs, inProgress.ID) +} + +func Test_GetBackups_FilterByTypesAndStatuses_AppliesAndAcrossDimensions(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + completedFull := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + // Excluded by the status dimension. + physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + + // Excluded by the type dimension (COMPLETED but INCREMENTAL). + physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, completedFull.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + + page := getBackups(t, prereqs, "?type=FULL&status=COMPLETED") + + require.Len(t, page.Backups, 1) + assert.Equal(t, completedFull.ID, page.Backups[0].ID) + assert.EqualValues(t, 1, page.Total) +} + +// --- List edges -------------------------------------------------------------- + +func Test_GetBackups_WhenEmpty_ReturnsEmptyPage(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + page := getBackups(t, prereqs, "") + + assert.Empty(t, page.Backups) + assert.EqualValues(t, 0, page.Total) + assert.Zero(t, page.TotalUsageMb) +} + +func Test_GetBackups_IsolatesByDatabase(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + ownFull := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + other := createSecondPhysicalDatabase(t, prereqs) + physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + other.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + page := getBackups(t, prereqs, "") + + require.Len(t, page.Backups, 1) + assert.Equal(t, ownFull.ID, page.Backups[0].ID) +} + +func Test_GetBackups_DefaultsAndCapsPageLimit(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + unset := getBackups(t, prereqs, "") + assert.Equal(t, 50, unset.Limit) + + capped := getBackups(t, prereqs, "?limit=5000") + assert.Equal(t, 1000, capped.Limit) +} + +func Test_GetBackups_OffsetPastEnd_ReturnsEmptyPageWithTotal(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + page := getBackups(t, prereqs, "?offset=10") + + assert.Empty(t, page.Backups) + assert.EqualValues(t, 1, page.Total) + assert.Equal(t, 10, page.Offset) +} + +// --- Cancel / delete edge cases --------------------------------------------- + +func Test_DeleteBackup_WhenUnknownId_Returns404(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+uuid.NewString(), + "Bearer "+prereqs.user.Token, http.StatusNotFound) +} + +func Test_CancelBackup_WhenUnknownId_Returns404(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+uuid.NewString()+"/cancel", + "Bearer "+prereqs.user.Token, nil, http.StatusNotFound) +} + +// A WAL segment has no trigger/cancel identity, so its id is not cancellable. +func Test_CancelBackup_WhenWalSegment_Returns404(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + walSegment := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000001", + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+walSegment.ID.String()+"/cancel", + "Bearer "+prereqs.user.Token, nil, http.StatusNotFound) +} + +func Test_DeleteBackup_WhenInProgress_CancelsThenDeletes(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, + physical_testing.NewTestInProgressFullBackup(prereqs.database.ID, prereqs.storage.ID, 1)) + physical_testing.CreateTestInFlightClaim(t, prereqs.database.ID, full.ID, physical_enums.PhysicalBackupTypeFull) + + test_utils.MakeDeleteRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String(), + "Bearer "+prereqs.user.Token, http.StatusNoContent) + + assertFullGone(t, full.ID) + + claim, err := physical_repositories.GetInFlightBackupRepository().FindByDatabaseID(prereqs.database.ID) + require.NoError(t, err) + assert.Nil(t, claim, "deleting an in-progress backup must release its claim") +} + +func Test_TriggerBackup_Auto_Returns202(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + _, err := backups_config_physical.GetBackupConfigService().GetBackupConfigByDbId(prereqs.database.ID) + require.NoError(t, err) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/trigger", + "Bearer "+prereqs.user.Token, + backups_dto_physical.TriggerBackupRequest{Type: backups_dto_physical.TriggerBackupTypeAuto}, + http.StatusAccepted) +} + +// --- Restore stream happy path ---------------------------------------------- + +// Deep tar layout is covered by restore_stream's writer_test; this asserts the +// controller path end to end (token → resolve → stream → headers) and the +// per-backup invariant that no WAL rides along. +func Test_GetRestoreStream_WithBackupToken_StreamsTarWithoutWal(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := createStreamableFullBackup(t, prereqs) + + var tokenResponse backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusOK, &tokenResponse) + + response := test_utils.MakeGetRequest(t, prereqs.router, + "/api/v1/backups/physical/restore-stream?token="+tokenResponse.Token, + "", http.StatusOK) + + assert.Equal(t, "application/x-tar", response.Headers.Get("Content-Type")) + assert.Contains(t, response.Headers.Get("Content-Disposition"), "restore-") + + names := readTarEntryNames(t, response.Body) + assert.Contains(t, names, "full/base.tar.zst") + assert.Contains(t, names, "full/backup_manifest") + for _, name := range names { + assert.False(t, strings.HasPrefix(name, "wal/"), + "a per-backup restore must carry no WAL, got %q", name) + } +} + +// The recovery script is a static, secret-free helper served without auth; the UI +// tells the user to curl it. Lock its content-type and that it is a runnable POSIX +// script that drives pg_combinebackup. +func Test_GetRecoveryScript_ReturnsRunnableShellScript(t *testing.T) { + router := newPhysicalControllerRouter() + + response := test_utils.MakeGetRequest(t, router, + "/api/v1/backups/physical/recovery-script", "", http.StatusOK) + + assert.Contains(t, response.Headers.Get("Content-Type"), "text/x-shellscript") + assert.True(t, strings.HasPrefix(string(response.Body), "#!/bin/sh"), + "the recovery script must be a runnable POSIX sh script") + assert.Contains(t, string(response.Body), "pg_combinebackup", + "the recovery script must reconstruct the cluster") + assert.Contains(t, string(response.Body), "--pg-bin", + "the recovery script must accept a PostgreSQL bin path") + assert.Contains(t, string(response.Body), "--target-time", + "the recovery script must accept the PITR target time as an argument") + assert.NotContains(t, string(response.Body), "ZSTD_BIN", + "WAL is decompressed on the host up front; the script must not wire zstd into restore_command") +} + +func Test_GetRestoreStream_TokenIsSingleUse(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := createStreamableFullBackup(t, prereqs) + + var tokenResponse backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusOK, &tokenResponse) + + streamURL := "/api/v1/backups/physical/restore-stream?token=" + tokenResponse.Token + + test_utils.MakeGetRequest(t, prereqs.router, streamURL, "", http.StatusOK) + test_utils.MakeGetRequest(t, prereqs.router, streamURL, "", http.StatusUnauthorized) +} + +// A second heavy stream while the user already has one in progress is rejected +// with 409: downloads and restores share one per-user slot in the stream guard. +// The slot is occupied directly so the race is deterministic, not timing-based. +func Test_GetRestoreStream_WhenRestoreInProgress_Returns409(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + // Issue the token while idle (issuance is gated on the same slot), then occupy + // the slot before consuming it — the stream's slot acquisition fails with 409. + var tokenResponse backups_dto_physical.GenerateRestoreTokenResponse + test_utils.MakePostRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusOK, &tokenResponse) + + occupyUserStreamSlot(t, prereqs.user.UserID) + + test_utils.MakeGetRequest(t, prereqs.router, + "/api/v1/backups/physical/restore-stream?token="+tokenResponse.Token, + "", http.StatusConflict) +} + +// Issuing a restore token while the user already has a stream in progress is +// rejected up front with 409, so they never mint a token they can't use. +func Test_GenerateBackupRestoreToken_WhenRestoreInProgress_Returns409(t *testing.T) { + prereqs := createPhysicalControllerPrereqs(t) + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + + occupyUserStreamSlot(t, prereqs.user.UserID) + + test_utils.MakePostRequest(t, prereqs.router, + "/api/v1/backups/physical/backups/"+full.ID.String()+"/restore-token", + "Bearer "+prereqs.user.Token, nil, http.StatusConflict) +} + +// --- Helpers ----------------------------------------------------------------- + +// runRolePermissionMatrix exercises one endpoint across workspace roles via +// doRequest, which returns the HTTP status for the given Authorization header. +// successCode is what authorized roles receive; managersOnly=true means a viewer +// is rejected (state-changing ops), false means a viewer is allowed (reads). A +// non-member is always rejected; a global admin is always allowed. +func runRolePermissionMatrix( + t *testing.T, + prereqs *physicalControllerPrereqs, + managersOnly bool, + successCode int, + doRequest func(authHeader string) int, +) { + t.Helper() + + member := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + prereqs.workspace, member, users_enums.WorkspaceRoleMember, prereqs.user.Token, prereqs.router) + + viewer := addWorkspaceViewer(t, prereqs) + nonMember := users_testing.CreateTestUser(users_enums.UserRoleMember) + admin := users_testing.CreateTestUser(users_enums.UserRoleAdmin) + + viewerCode := successCode + if managersOnly { + viewerCode = http.StatusBadRequest + } + + assert.Equal(t, successCode, doRequest("Bearer "+prereqs.user.Token), "owner") + assert.Equal(t, successCode, doRequest("Bearer "+member.Token), "member") + assert.Equal(t, viewerCode, doRequest("Bearer "+viewer.Token), "viewer") + assert.Equal(t, http.StatusBadRequest, doRequest("Bearer "+nonMember.Token), "non-member") + assert.Equal(t, successCode, doRequest("Bearer "+admin.Token), "global admin") +} + +// occupyUserStreamSlot takes the user's shared download/restore slot to stand in +// for an in-flight heavy stream, releasing it at test end. Both token services +// embed the same guard, so this blocks token issuance and a second stream alike — +// letting the 409 path be asserted deterministically instead of via a timing race. +func occupyUserStreamSlot(t *testing.T, userID uuid.UUID) { + t.Helper() + + restoreTokenService := backups_download.GetRestoreTokenService() + if err := restoreTokenService.AcquireSlot(userID); err != nil { + t.Fatalf("occupy stream slot: %v", err) + } + + t.Cleanup(func() { + restoreTokenService.ReleaseDownloadLock(userID) + }) +} + +func addWorkspaceViewer(t *testing.T, prereqs *physicalControllerPrereqs) *users_dto.SignInResponseDTO { + t.Helper() + + viewer := users_testing.CreateTestUser(users_enums.UserRoleMember) + workspaces_testing.AddMemberToWorkspace( + prereqs.workspace, viewer, users_enums.WorkspaceRoleViewer, prereqs.user.Token, prereqs.router) + + return viewer +} + +func getBackups( + t *testing.T, + prereqs *physicalControllerPrereqs, + query string, +) backups_dto_physical.GetPhysicalBackupsResponse { + t.Helper() + + var response backups_dto_physical.GetPhysicalBackupsResponse + test_utils.MakeGetRequestAndUnmarshal(t, prereqs.router, + "/api/v1/backups/physical/database/"+prereqs.database.ID.String()+"/backups"+query, + "Bearer "+prereqs.user.Token, http.StatusOK, &response) + + return response +} + +// seedOneOfEachType creates one COMPLETED full, one incremental on it, and one +// committed WAL segment for the prereqs database — the minimal mix for filter and +// listing assertions. +func seedOneOfEachType(t *testing.T, prereqs *physicalControllerPrereqs) ( + *physical_models.PhysicalFullBackup, + *physical_models.PhysicalIncrementalBackup, + *physical_models.PhysicalWalSegment, +) { + t.Helper() + + full := physical_testing.CreateTestFullBackup(t, physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes))) + incr := physical_testing.CreateTestIncrementalBackup(t, physical_testing.NewTestCompletedIncrementalBackup( + prereqs.database.ID, prereqs.storage.ID, full.ID, nil, 1, + walmath.LSN(segmentBytes), walmath.LSN(2*segmentBytes))) + walSegment := physical_testing.CreateTestWalSegment(t, physical_testing.NewTestWalSegment( + prereqs.database.ID, prereqs.storage.ID, 1, "000000010000000000000002", + walmath.LSN(2*segmentBytes), walmath.LSN(3*segmentBytes))) + + return full, incr, walSegment +} + +func createSecondPhysicalDatabase(t *testing.T, prereqs *physicalControllerPrereqs) *databases.Database { + t.Helper() + + database := databases.CreateTestPhysicalPostgresDatabase(prereqs.workspace.ID, prereqs.notifier, "17") + t.Cleanup(func() { + physical_testing.DeleteAllPhysicalCatalogForDatabase(t, database.ID) + databases.RemoveTestDatabase(database) + }) + + return database +} + +func assertAuditLogContains(t *testing.T, workspaceID uuid.UUID, messageFragment, databaseName string) { + t.Helper() + + logs, err := audit_logs.GetAuditLogService().GetWorkspaceAuditLogs( + workspaceID, &audit_logs.GetAuditLogsRequest{Limit: 100, Offset: 0}) + require.NoError(t, err) + + for _, entry := range logs.AuditLogs { + if strings.Contains(entry.Message, messageFragment) && strings.Contains(entry.Message, databaseName) { + return + } + } + + t.Fatalf("audit log %q for database %q not found", messageFragment, databaseName) +} + +// createStreamableFullBackup seeds a COMPLETED full whose stored artifact and +// manifest sidecar actually exist in storage, so the restore stream can assemble a +// real tar. NewTestCompletedFullBackup only writes a catalog row with no bytes +// behind it, which cannot be streamed. The artifact is a minimal zstd-compressed +// PGDATA tar; the manifest is stored raw, matching the executor's convention. +func createStreamableFullBackup(t *testing.T, prereqs *physicalControllerPrereqs) *physical_models.PhysicalFullBackup { + t.Helper() + + full := physical_testing.NewTestCompletedFullBackup( + prereqs.database.ID, prereqs.storage.ID, 1, walmath.LSN(0), walmath.LSN(segmentBytes)) + full.Encryption = backups_core_enums.BackupEncryptionNone + full.Compression = physical_enums.PhysicalBackupCompressionZstd + full.ManifestFileName = new(*full.FileName + ".manifest") + + blob := zstdTar(t, map[string]string{ + "PG_VERSION": "17\n", + "base/1/1259": "heap-bytes", + }) + full.BackupSizeMb = new(float64(len(blob)) / (1024 * 1024)) + + saveStorageObject(t, prereqs.storage, *full.FileName, blob) + saveStorageObject(t, prereqs.storage, *full.ManifestFileName, + []byte(`{ "PostgreSQL-Backup-Manifest-Version": 2 }`)) + + return physical_testing.CreateTestFullBackup(t, full) +} + +func saveStorageObject(t *testing.T, storage *storages.Storage, name string, body []byte) { + t.Helper() + + encryptor := encryption.GetFieldEncryptor() + if err := storage.SaveFile( + context.Background(), encryptor, logger.GetLogger(), name, bytes.NewReader(body)); err != nil { + t.Fatalf("save storage object %s: %v", name, err) + } + + t.Cleanup(func() { _ = storage.DeleteFile(encryptor, name) }) +} + +func zstdTar(t *testing.T, files map[string]string) []byte { + t.Helper() + + var rawTar bytes.Buffer + tarWriter := tar.NewWriter(&rawTar) + for name, content := range files { + require.NoError(t, tarWriter.WriteHeader(&tar.Header{ + Name: name, Mode: 0o600, Size: int64(len(content)), Typeflag: tar.TypeReg, + })) + _, err := tarWriter.Write([]byte(content)) + require.NoError(t, err) + } + require.NoError(t, tarWriter.Close()) + + var compressed bytes.Buffer + encoder, err := zstd.NewWriter(&compressed) + require.NoError(t, err) + _, err = encoder.Write(rawTar.Bytes()) + require.NoError(t, err) + require.NoError(t, encoder.Close()) + + return compressed.Bytes() +} + +func readTarEntryNames(t *testing.T, body []byte) []string { + t.Helper() + + var names []string + tarReader := tar.NewReader(bytes.NewReader(body)) + for { + header, err := tarReader.Next() + if err == io.EOF { + break + } + require.NoError(t, err) + names = append(names, header.Name) + } + + return names +} diff --git a/backend/internal/features/backups/backups/controllers/physical/di.go b/backend/internal/features/backups/backups/controllers/physical/di.go new file mode 100644 index 0000000..d8ddfa8 --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/physical/di.go @@ -0,0 +1,15 @@ +package backups_controllers_physical + +import ( + backups_download "databasus-backend/internal/features/backups/backups/download" + backups_services "databasus-backend/internal/features/backups/backups/services" +) + +var physicalBackupController = &PhysicalBackupController{ + backups_services.GetPhysicalBackupService(), + backups_download.GetRestoreTokenService(), +} + +func GetPhysicalBackupController() *PhysicalBackupController { + return physicalBackupController +} diff --git a/backend/internal/features/backups/backups/controllers/physical/recovery_script.go b/backend/internal/features/backups/backups/controllers/physical/recovery_script.go new file mode 100644 index 0000000..2b9eb3b --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/physical/recovery_script.go @@ -0,0 +1,6 @@ +package backups_controllers_physical + +import _ "embed" + +//go:embed recovery_script.sh +var recoveryScript string diff --git a/backend/internal/features/backups/backups/controllers/physical/recovery_script.sh b/backend/internal/features/backups/backups/controllers/physical/recovery_script.sh new file mode 100644 index 0000000..b35e63b --- /dev/null +++ b/backend/internal/features/backups/backups/controllers/physical/recovery_script.sh @@ -0,0 +1,427 @@ +#!/bin/sh +# Databasus physical-restore helper. +# +# Takes a restore bundle produced by the Databasus restore stream, decompresses +# what needs decompressing, runs pg_combinebackup over the full + incremental +# chain, and wires up WAL replay / point-in-time recovery. +# +# Usage: +# sh databasus-recovery.sh [--pg-bin ] [--combine-image ] [--target-time ] [output-dir] +# +# --pg-bin directory holding the PostgreSQL tools (pg_combinebackup, +# pg_ctl) when they are not on PATH. +# --combine-image run pg_combinebackup inside this Docker image instead of on +# the host - for hosts that have zstd but no PostgreSQL tools. +# The host still decompresses the backup blobs and WAL with its +# zstd; only the version-specific pg_combinebackup runs in the +# container, and the started cluster needs no extra tools. +# --target-time point-in-time recovery target, a PostgreSQL-parseable UTC +# timestamp (e.g. '2026-06-12 14:30:00+00:00'). Replays WAL +# up to this point and promotes; omit to replay all shipped +# WAL (latest). Ignored when the bundle ships no WAL. +# an http(s):// restore-stream URL (with its token) or a +# path to an already-downloaded .tar bundle. +# [output-dir] where to build the restore (default: databasus-restore). + +set -eu + +usage() { + cat >&2 <<'USAGE' +usage: sh databasus-recovery.sh [--pg-bin ] [--combine-image ] [--target-time ] [output-dir] + + --pg-bin directory holding the PostgreSQL tools (pg_combinebackup, + pg_ctl) when they are not on PATH + --combine-image run pg_combinebackup inside this Docker image instead of on + the host (decompression still uses the host's zstd) + --target-time point-in-time recovery target, a PostgreSQL-parseable UTC + timestamp (e.g. '2026-06-12 14:30:00+00:00'); omit to + replay all shipped WAL. Ignored when the bundle ships no WAL + an http(s):// restore-stream URL (with its token) or a + path to an already-downloaded .tar bundle + [output-dir] where to build the restore (default: databasus-restore) +USAGE +} + +PG_BIN="" +COMBINE_IMAGE="" +TARGET_TIME="" +while [ $# -gt 0 ]; do + case "$1" in + --pg-bin) + [ $# -ge 2 ] || { + echo "error: --pg-bin needs a directory" >&2 + exit 2 + } + PG_BIN="$2" + shift 2 + ;; + --pg-bin=*) + PG_BIN="${1#*=}" + shift + ;; + --combine-image) + [ $# -ge 2 ] || { + echo "error: --combine-image needs an image reference" >&2 + exit 2 + } + COMBINE_IMAGE="$2" + shift 2 + ;; + --combine-image=*) + COMBINE_IMAGE="${1#*=}" + shift + ;; + --target-time) + [ $# -ge 2 ] || { + echo "error: --target-time needs a timestamp" >&2 + exit 2 + } + TARGET_TIME="$2" + shift 2 + ;; + --target-time=*) + TARGET_TIME="${1#*=}" + shift + ;; + -h | --help) + usage + exit 0 + ;; + --) + shift + break + ;; + -*) + echo "error: unknown option '$1'" >&2 + usage + exit 2 + ;; + *) + break + ;; + esac +done + +SOURCE="${1:-}" +OUT_DIR="${2:-databasus-restore}" + +if [ -z "$SOURCE" ]; then + usage + exit 2 +fi + +require_tool() { + if ! command -v "$1" >/dev/null 2>&1; then + echo "error: '$1' is required but was not found on PATH" >&2 + [ -n "${2:-}" ] && echo " $2" >&2 + exit 1 + fi +} + +# The restore builds the cluster inside . If itself is already a live PostgreSQL +# data or install directory (PG_VERSION, pg_hba.conf, the binaries, ...), the user almost +# certainly aimed the restore at the wrong place: we would write the cluster a level deeper +# and the existing one would keep being served. Refuse up front rather than clobber or +# silently restore into the wrong layout. +require_empty_target() { + target="$1" + [ -d "$target" ] || return 0 + + for marker in PG_VERSION postgresql.conf pg_hba.conf base global postmaster.pid \ + bin/postgres bin/pg_ctl; do + if [ -e "$target/$marker" ]; then + echo "error: output directory '$target' already looks like a PostgreSQL data or install directory (found '$marker')" >&2 + echo " Point the restore at an empty directory instead." >&2 + exit 1 + fi + done +} + +# Major version out of a `... (PostgreSQL) 18.1` banner. +pg_major_from_banner() { + echo "$1" | sed -n 's/.*PostgreSQL) \([0-9][0-9]*\).*/\1/p' +} + +# Where the official postgres image expects PGDATA: PG 18 moved it under a version-specific +# path, PG <=17 keeps the flat one. The restored cluster must be bind-mounted here exactly. +container_pgdata() { + if [ "${1:-0}" -ge 18 ] 2>/dev/null; then + echo "/var/lib/postgresql/$1/docker" + else + echo "/var/lib/postgresql/data" + fi +} + +# Where to build the cluster under , mirroring the image's PGDATA layout so a volume-root +# mount (-v :/var/lib/postgresql on PG 18) finds it. PG 18 nests /docker; PG <=17 +# keeps /data, which the user mounts straight at /var/lib/postgresql/data. +host_cluster_dir() { + if [ "${2:-0}" -ge 18 ] 2>/dev/null; then + echo "$1/$2/docker" + else + echo "$1/data" + fi +} + +# Dump pg_controldata for the reconstructed cluster in the C locale so its labels stay English +# (they are localized otherwise and the parse below would break). Runs the same way as +# pg_combinebackup: inside --combine-image, or off --pg-bin / PATH on the host. +control_data() { + if [ -n "$COMBINE_IMAGE" ]; then + docker run --rm --user "$(id -u):$(id -g)" -e LC_ALL=C -v "$OUT_ABS:$OUT_ABS" \ + "$COMBINE_IMAGE" pg_controldata "$DATA_DIR" + else + LC_ALL=C "${PG_BIN:+$PG_BIN/}pg_controldata" "$DATA_DIR" + fi +} + +# Read one "