chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,44 @@
|
||||
# Docs audit — 2026-04-18
|
||||
|
||||
Audit of every `.mdx` file under `docs/` (excluding `docs/deprecated/` —
|
||||
those are intentional preservation) against the rules in
|
||||
`.claude/skills/doc-author/SKILL.md` (upstream Mintlify, MIT) and the
|
||||
InsForge overlay at `.claude/skills/doc-author/INSFORGE.md`.
|
||||
|
||||
Methodology: targeted `grep` passes for each rule category plus manual
|
||||
cross-check of `docs/docs.json` navigation entries against the filesystem.
|
||||
Fixed in this PR: the top five high-severity rows. Everything else is
|
||||
filed as follow-up tickets per #1117.
|
||||
|
||||
**Key:** severity is the operational impact if the issue ships.
|
||||
`high` = broken nav or broken link a reader will hit; `med` = visible
|
||||
style inconsistency with the overlay or an orphaned page; `low` = minor
|
||||
copy nit.
|
||||
|
||||
## Findings
|
||||
|
||||
| # | Page | Issue | Severity | Recommended fix |
|
||||
|---|------|-------|----------|-----------------|
|
||||
| 1 | `docs/docs.json` (lines 125-127) | Nav points at `sdks/typescript/ui-components/react`, `sdks/typescript/ui-components/nextjs`, `sdks/typescript/ui-components/customization` — none of these `.mdx` files exist on disk, so the entire "UI Components" group breaks Mintlify nav rendering. | high | Remove the three dead entries (and the surrounding `{ "group": "UI Components", ... }` wrapper). Re-add when pages land via a future ticket. |
|
||||
| 2 | `docs/sdks/typescript/overview.mdx` (lines 38-39) | Same dead targets as #1: body prose links to `/sdks/typescript/ui-components/react` and `/sdks/typescript/ui-components/nextjs`. Readers clicking from the SDK landing page hit 404. | high | Delete the two bullet links (or point them at the npm package pages). Skill §"Internal links": root-relative paths, and only to real pages. |
|
||||
| 3 | `docs/changelog.mdx` (lines 41, 64, 80, 91, 150, 188, 204) | Seven broken internal links using the obsolete `/core-concepts/{ai,email,authentication,database}/sdk` and `/core-concepts/authentication/ui-components/react` routes. The SDK reference pages live at `/sdks/typescript/*` now. | high | Repoint each link to the current `/sdks/typescript/<module>` page. For the `ui-components/react` link (line 150), drop it — no replacement page exists yet. |
|
||||
| 4 | `docs/sdks/flutter/*.mdx` (all 6 pages: `overview`, `auth`, `database`, `storage`, `realtime`, `ai`) | Flutter SDK has six reference pages on disk but zero entries in `docs/docs.json` navigation → readers cannot discover them from the site nav. Skill §"What requires escalation": changes that affect navigation structure. Orphaned pages are worse than missing pages. | high | Add a `"Flutter"` group under the `"SDK & Examples"` tab in `docs/docs.json`, mirroring the Swift/Kotlin groups. Six pages in the order `overview → database → auth → storage → realtime → ai` (Flutter has no `functions.mdx`). |
|
||||
| 5 | `docs/sdks/typescript/email.mdx` (line 6) | Experimental-feature callout uses `<Note>` — overlay INSFORGE.md §"Experimental features get `<Warning>` at top" requires `<Warning>` for experimental/private-preview status. `<Note>` is for neutral context; readers don't register it as a stability signal. Consistent with how `docs/core-concepts/email/architecture.mdx:6-8` and `docs/core-concepts/deployments/architecture.mdx:6-8` already treat experimental features. | high | Change the `<Note>` tag to `<Warning>`, keeping body text identical. |
|
||||
| 6 | `docs/partnership.mdx` (lines 76-91) | 16 lines begin with `- ✅` emoji-prefixed bullets. Skill §"What to avoid / Never use: Emoji in documentation." | med | Strip the ✅ emoji; plain bullets read as checklist items already. One-commit, zero structural change. |
|
||||
| 7 | `docs/core-concepts/realtime/sdk.mdx` and `docs/core-concepts/storage/sdk.mdx` | Both pages exist on disk but are NOT in `docs/docs.json` navigation — they're orphaned. They also duplicate content that lives at `/sdks/typescript/realtime` and `/sdks/typescript/storage` (the canonical SDK-tab pages). Deprecated-API smell: the SDK-tab pages are canonical, the core-concepts pages are the older location. | med | Follow-up ticket: either delete both (canonical SDK docs live under `/sdks/typescript/*` per docs.json:110-141) or add them to nav. Audit recommends deletion; `changelog.mdx:106` still links to `/core-concepts/realtime/sdk` so a link rewrite is part of the same ticket. |
|
||||
| 8 | `docs/sdks/rest/storage.mdx` (sections at lines 154, 196, 289) | Three H2 sections titled "Upload Object (Deprecated)", "Upload with Auto-Generated Key (Deprecated)", "Download Object (Deprecated)". The parenthetical label works, but a `<Warning>` callout inside each section would make the status machine-readable (skill §"Callouts / `<Warning>` — something potentially destructive"). | med | Follow-up ticket: prepend a `<Warning>These endpoints are deprecated. Use [...] instead.</Warning>` callout at the top of each of the three sections; do not remove the content (REST reference is intentionally complete). |
|
||||
| 9 | `docs/sdks/typescript/database.mdx` (line 379) | Example imports `@insforge/react` with an inline comment `// or '@insforge/react-router'`. No `/sdks/typescript/ui-components/*` page exists (see #1) to explain when you'd pick which — the reader has nowhere to go. | med | Follow-up ticket: after UI-components pages are restored (#1), add a cross-link here. Standalone fix not useful. |
|
||||
| 10 | `docs/quickstart.mdx` (lines 13-15) | Uses markdown numbered list `1. 2. 3.` for a three-action sequence inside "Step 1" (which is itself an H2). Skill §"Steps for sequential procedures: use the `<Steps>` component". Impact is small — the page is already readable. | low | Deferred. Minimal-diff conversion is ~20 lines for a 5-line list; not worth this PR. Folded into catch-all follow-up. |
|
||||
| 11 | `docs/sdks/typescript/email.mdx` (line 2) | Title is `"Emails SDK Reference"` (plural) but description says `"Send custom transactional emails with the InsForge SDK"` and the module name elsewhere is "Email" (singular; matches `docs.json:218` `"group": "Email"` and the architecture page `docs/core-concepts/email/architecture.mdx`). | low | Rename title to `"Email SDK Reference"` for consistency. Folded into catch-all follow-up. |
|
||||
| 12 | Many pages | Most H2 headings use Title Case (`## Quick Start`, `## Framework-Specific Packages`, `## Learn More`). Skill §"Voice and structure / Sentence case for headings". This is a systemic repo convention, not a one-page bug. | low | No action. Folded into catch-all follow-up to consider as a separate discussion. Overlay does not override on this, but repo-wide changes are out of scope per ticket #1117. |
|
||||
|
||||
## Summary
|
||||
|
||||
- **High-severity rows:** 5 (fixed in this PR).
|
||||
- **Medium-severity rows:** 4 (one follow-up ticket each).
|
||||
- **Low-severity rows:** 3 (folded into a single catch-all ticket).
|
||||
|
||||
## Fix commits in this PR
|
||||
|
||||
One commit per fixed row, each citing the skill/overlay section it enforces.
|
||||
See the `insforge/commander/1117` branch.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,794 @@
|
||||
# Compute Dashboard UX Fixes Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make Compute Services usable from the dashboard — add action buttons, create dialog, logs panel, and fix broken endpoint URLs.
|
||||
|
||||
**Architecture:** Incremental additions to the existing ComputePage + ServiceCard components. All API methods and mutations already exist in hooks/services. We're just wiring UI to existing plumbing. No backend changes.
|
||||
|
||||
**Tech Stack:** React, TypeScript, @insforge/ui (Radix-based), @tanstack/react-query, lucide-react icons
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-04-07-compute-dashboard-ux-design.md`
|
||||
|
||||
---
|
||||
|
||||
### File Map
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `packages/dashboard/src/features/compute/constants.ts` | Modify | Add `getReachableUrl` helper, region labels |
|
||||
| `packages/dashboard/src/features/compute/hooks/useComputeServices.ts` | Modify | Add `useLogs` hook export |
|
||||
| `packages/dashboard/src/features/compute/components/ServiceLogs.tsx` | Create | Log entries display with refresh |
|
||||
| `packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx` | Create | Create service form dialog |
|
||||
| `packages/dashboard/src/features/compute/components/ServiceCard.tsx` | Modify | Add dropdown menu with actions |
|
||||
| `packages/dashboard/src/features/compute/pages/ComputePage.tsx` | Modify | Add create button, action buttons, logs, enhanced detail view |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add helper utilities to constants.ts
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/dashboard/src/features/compute/constants.ts`
|
||||
|
||||
- [ ] **Step 1: Add `getReachableUrl` helper and region labels**
|
||||
|
||||
```ts
|
||||
import type { ServiceSchema, ServiceStatus } from '@insforge/shared-schemas';
|
||||
|
||||
export const statusColors: Record<ServiceStatus, string> = {
|
||||
running: 'bg-green-500',
|
||||
deploying: 'bg-yellow-500',
|
||||
creating: 'bg-yellow-500',
|
||||
stopped: 'bg-gray-400',
|
||||
failed: 'bg-red-500',
|
||||
destroying: 'bg-orange-500',
|
||||
};
|
||||
|
||||
export const CPU_TIERS = [
|
||||
{ value: 'shared-1x', label: 'Shared 1x' },
|
||||
{ value: 'shared-2x', label: 'Shared 2x' },
|
||||
{ value: 'performance-1x', label: 'Performance 1x' },
|
||||
{ value: 'performance-2x', label: 'Performance 2x' },
|
||||
{ value: 'performance-4x', label: 'Performance 4x' },
|
||||
] as const;
|
||||
|
||||
export const MEMORY_OPTIONS = [256, 512, 1024, 2048, 4096, 8192] as const;
|
||||
|
||||
export const REGIONS = [
|
||||
{ value: 'iad', label: 'Ashburn, VA (iad)' },
|
||||
{ value: 'sin', label: 'Singapore (sin)' },
|
||||
{ value: 'lax', label: 'Los Angeles (lax)' },
|
||||
{ value: 'lhr', label: 'London (lhr)' },
|
||||
{ value: 'nrt', label: 'Tokyo (nrt)' },
|
||||
{ value: 'ams', label: 'Amsterdam (ams)' },
|
||||
{ value: 'syd', label: 'Sydney (syd)' },
|
||||
] as const;
|
||||
|
||||
/** Return the actual reachable .fly.dev URL instead of a custom domain that may not resolve */
|
||||
export function getReachableUrl(service: ServiceSchema): string | null {
|
||||
if (service.flyAppId) {
|
||||
return `https://${service.flyAppId}.fly.dev`;
|
||||
}
|
||||
return service.endpointUrl;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/constants.ts
|
||||
git commit -m "feat(compute/ui): add helper utilities for create form and endpoint URL"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Add useLogs hook to useComputeServices
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/dashboard/src/features/compute/hooks/useComputeServices.ts`
|
||||
|
||||
- [ ] **Step 1: Add useLogs query hook**
|
||||
|
||||
Add this new exported hook after the existing `useComputeServices` function in the same file:
|
||||
|
||||
```ts
|
||||
export function useServiceLogs(serviceId: string | null) {
|
||||
return useQuery({
|
||||
queryKey: ['compute', 'services', serviceId, 'logs'],
|
||||
queryFn: () => computeServicesApi.logs(serviceId!, 50),
|
||||
enabled: !!serviceId,
|
||||
staleTime: 0, // always refetch on mount for logs
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
This requires adding `useQuery` to the imports (it's already imported, just need to use it in the new hook).
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/hooks/useComputeServices.ts
|
||||
git commit -m "feat(compute/ui): add useServiceLogs hook"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Create ServiceLogs component
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/dashboard/src/features/compute/components/ServiceLogs.tsx`
|
||||
|
||||
- [ ] **Step 1: Create the logs display component**
|
||||
|
||||
```tsx
|
||||
import { RefreshCw } from 'lucide-react';
|
||||
import { Button } from '@insforge/ui';
|
||||
import { useServiceLogs } from '../hooks/useComputeServices';
|
||||
|
||||
interface ServiceLogsProps {
|
||||
serviceId: string;
|
||||
}
|
||||
|
||||
export function ServiceLogs({ serviceId }: ServiceLogsProps) {
|
||||
const { data: logs = [], isLoading, refetch, isFetching } = useServiceLogs(serviceId);
|
||||
|
||||
return (
|
||||
<div className="bg-card border border-[var(--alpha-8)] rounded-lg overflow-hidden">
|
||||
<div className="flex items-center justify-between px-4 py-3 border-b border-[var(--alpha-8)]">
|
||||
<h3 className="text-sm font-medium text-foreground">Logs</h3>
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
onClick={() => void refetch()}
|
||||
disabled={isFetching}
|
||||
>
|
||||
<RefreshCw className={`h-3.5 w-3.5 ${isFetching ? 'animate-spin' : ''}`} />
|
||||
Refresh
|
||||
</Button>
|
||||
</div>
|
||||
<div className="max-h-[300px] overflow-y-auto p-4">
|
||||
{isLoading ? (
|
||||
<p className="text-xs text-muted-foreground">Loading logs...</p>
|
||||
) : logs.length === 0 ? (
|
||||
<p className="text-xs text-muted-foreground">No logs available.</p>
|
||||
) : (
|
||||
<pre className="text-xs font-mono text-muted-foreground space-y-0.5">
|
||||
{logs.map((entry, i) => (
|
||||
<div key={i}>
|
||||
<span className="text-foreground/60">
|
||||
{new Date(entry.timestamp).toISOString().replace('T', ' ').slice(0, 19)}
|
||||
</span>
|
||||
{' '}
|
||||
{entry.message}
|
||||
</div>
|
||||
))}
|
||||
</pre>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/components/ServiceLogs.tsx
|
||||
git commit -m "feat(compute/ui): add ServiceLogs component"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Create CreateServiceDialog component
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx`
|
||||
|
||||
- [ ] **Step 1: Create the dialog component**
|
||||
|
||||
```tsx
|
||||
import { useState } from 'react';
|
||||
import {
|
||||
Button,
|
||||
Dialog,
|
||||
DialogContent,
|
||||
DialogHeader,
|
||||
DialogTitle,
|
||||
DialogDescription,
|
||||
DialogBody,
|
||||
DialogFooter,
|
||||
Input,
|
||||
Select,
|
||||
SelectTrigger,
|
||||
SelectValue,
|
||||
SelectContent,
|
||||
SelectItem,
|
||||
} from '@insforge/ui';
|
||||
import { CPU_TIERS, MEMORY_OPTIONS, REGIONS } from '../constants';
|
||||
import type { CreateServiceRequest } from '@insforge/shared-schemas';
|
||||
|
||||
interface CreateServiceDialogProps {
|
||||
open: boolean;
|
||||
onOpenChange: (open: boolean) => void;
|
||||
onCreate: (data: CreateServiceRequest) => Promise<unknown>;
|
||||
isCreating: boolean;
|
||||
}
|
||||
|
||||
export function CreateServiceDialog({
|
||||
open,
|
||||
onOpenChange,
|
||||
onCreate,
|
||||
isCreating,
|
||||
}: CreateServiceDialogProps) {
|
||||
const [name, setName] = useState('');
|
||||
const [imageUrl, setImageUrl] = useState('');
|
||||
const [port, setPort] = useState('8080');
|
||||
const [cpu, setCpu] = useState('shared-1x');
|
||||
const [memory, setMemory] = useState('512');
|
||||
const [region, setRegion] = useState('iad');
|
||||
|
||||
const resetForm = () => {
|
||||
setName('');
|
||||
setImageUrl('');
|
||||
setPort('8080');
|
||||
setCpu('shared-1x');
|
||||
setMemory('512');
|
||||
setRegion('iad');
|
||||
};
|
||||
|
||||
const handleSubmit = async () => {
|
||||
await onCreate({
|
||||
name,
|
||||
imageUrl,
|
||||
port: Number(port),
|
||||
cpu: cpu as CreateServiceRequest['cpu'],
|
||||
memory: Number(memory),
|
||||
region,
|
||||
});
|
||||
resetForm();
|
||||
onOpenChange(false);
|
||||
};
|
||||
|
||||
const isValid = name.length > 0 && imageUrl.length > 0 && Number(port) > 0;
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={onOpenChange}>
|
||||
<DialogContent className="max-w-[520px]">
|
||||
<DialogHeader>
|
||||
<DialogTitle>Create Service</DialogTitle>
|
||||
<DialogDescription>Deploy a Docker container as a compute service.</DialogDescription>
|
||||
</DialogHeader>
|
||||
|
||||
<DialogBody>
|
||||
<div className="flex flex-col gap-4">
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">Name</label>
|
||||
<Input
|
||||
placeholder="my-api"
|
||||
value={name}
|
||||
onChange={(e) => setName(e.target.value)}
|
||||
/>
|
||||
<p className="text-xs text-muted-foreground">DNS-safe: lowercase, numbers, dashes</p>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">Image URL</label>
|
||||
<Input
|
||||
placeholder="nginx:alpine"
|
||||
value={imageUrl}
|
||||
onChange={(e) => setImageUrl(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">Port</label>
|
||||
<Input
|
||||
type="number"
|
||||
value={port}
|
||||
onChange={(e) => setPort(e.target.value)}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">Region</label>
|
||||
<Select value={region} onValueChange={setRegion}>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{REGIONS.map((r) => (
|
||||
<SelectItem key={r.value} value={r.value}>
|
||||
{r.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-2 gap-4">
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">CPU</label>
|
||||
<Select value={cpu} onValueChange={setCpu}>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{CPU_TIERS.map((t) => (
|
||||
<SelectItem key={t.value} value={t.value}>
|
||||
{t.label}
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
|
||||
<div className="flex flex-col gap-1.5">
|
||||
<label className="text-sm font-medium text-foreground">Memory</label>
|
||||
<Select value={memory} onValueChange={setMemory}>
|
||||
<SelectTrigger>
|
||||
<SelectValue />
|
||||
</SelectTrigger>
|
||||
<SelectContent>
|
||||
{MEMORY_OPTIONS.map((m) => (
|
||||
<SelectItem key={m} value={String(m)}>
|
||||
{m} MB
|
||||
</SelectItem>
|
||||
))}
|
||||
</SelectContent>
|
||||
</Select>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</DialogBody>
|
||||
|
||||
<DialogFooter>
|
||||
<Button
|
||||
variant="secondary"
|
||||
size="lg"
|
||||
disabled={isCreating}
|
||||
onClick={() => onOpenChange(false)}
|
||||
>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
variant="primary"
|
||||
size="lg"
|
||||
disabled={!isValid || isCreating}
|
||||
onClick={() => void handleSubmit()}
|
||||
>
|
||||
{isCreating ? 'Creating...' : 'Create Service'}
|
||||
</Button>
|
||||
</DialogFooter>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx
|
||||
git commit -m "feat(compute/ui): add CreateServiceDialog component"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Add dropdown menu to ServiceCard
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/dashboard/src/features/compute/components/ServiceCard.tsx`
|
||||
|
||||
- [ ] **Step 1: Replace ServiceCard with version that has dropdown actions**
|
||||
|
||||
```tsx
|
||||
import { ExternalLink, MoreVertical, Play, Square, Trash2 } from 'lucide-react';
|
||||
import {
|
||||
DropdownMenu,
|
||||
DropdownMenuTrigger,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuSeparator,
|
||||
} from '@insforge/ui';
|
||||
import type { ServiceSchema } from '@insforge/shared-schemas';
|
||||
import { statusColors, getReachableUrl } from '../constants';
|
||||
|
||||
interface ServiceCardProps {
|
||||
service: ServiceSchema;
|
||||
onClick: () => void;
|
||||
onStop: (id: string) => void;
|
||||
onStart: (id: string) => void;
|
||||
onDelete: (id: string) => void;
|
||||
}
|
||||
|
||||
export function ServiceCard({ service, onClick, onStop, onStart, onDelete }: ServiceCardProps) {
|
||||
const reachableUrl = getReachableUrl(service);
|
||||
|
||||
return (
|
||||
<div
|
||||
role="button"
|
||||
tabIndex={0}
|
||||
onClick={onClick}
|
||||
onKeyDown={(e) => { if (e.key === 'Enter') onClick(); }}
|
||||
className="w-full text-left bg-card border border-[var(--alpha-8)] rounded-lg p-4 hover:border-foreground/20 transition-colors cursor-pointer"
|
||||
>
|
||||
<div className="flex items-center justify-between mb-3">
|
||||
<h3 className="text-sm font-medium text-foreground truncate">{service.name}</h3>
|
||||
<div className="flex items-center gap-2">
|
||||
<span className="flex items-center gap-1.5 text-xs text-muted-foreground">
|
||||
<span className={`inline-block h-2 w-2 rounded-full ${statusColors[service.status]}`} />
|
||||
{service.status}
|
||||
</span>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
className="inline-flex h-6 w-6 items-center justify-center rounded text-muted-foreground hover:bg-[var(--alpha-8)] hover:text-foreground"
|
||||
>
|
||||
<MoreVertical className="h-3.5 w-3.5" />
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align="end" onClick={(e) => e.stopPropagation()}>
|
||||
{service.status === 'running' && (
|
||||
<DropdownMenuItem onClick={() => onStop(service.id)}>
|
||||
<Square className="h-3.5 w-3.5" />
|
||||
Stop
|
||||
</DropdownMenuItem>
|
||||
)}
|
||||
{service.status === 'stopped' && (
|
||||
<DropdownMenuItem onClick={() => onStart(service.id)}>
|
||||
<Play className="h-3.5 w-3.5" />
|
||||
Start
|
||||
</DropdownMenuItem>
|
||||
)}
|
||||
{(service.status === 'running' || service.status === 'stopped') && (
|
||||
<DropdownMenuSeparator />
|
||||
)}
|
||||
<DropdownMenuItem
|
||||
onClick={() => onDelete(service.id)}
|
||||
className="text-destructive"
|
||||
>
|
||||
<Trash2 className="h-3.5 w-3.5" />
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p className="text-xs text-muted-foreground truncate mb-3" title={service.imageUrl}>
|
||||
{service.imageUrl === 'dockerfile' ? 'Built from Dockerfile' : service.imageUrl}
|
||||
</p>
|
||||
|
||||
{reachableUrl && (
|
||||
<a
|
||||
href={reachableUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
onClick={(e) => e.stopPropagation()}
|
||||
className="inline-flex items-center gap-1 text-xs text-primary hover:underline mb-3"
|
||||
>
|
||||
<ExternalLink className="h-3 w-3" />
|
||||
{reachableUrl}
|
||||
</a>
|
||||
)}
|
||||
|
||||
<div className="flex items-center gap-3 text-xs text-muted-foreground pt-2 border-t border-[var(--alpha-8)]">
|
||||
<span>CPU: {service.cpu}</span>
|
||||
<span>Memory: {service.memory} MB</span>
|
||||
<span>{service.region}</span>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/components/ServiceCard.tsx
|
||||
git commit -m "feat(compute/ui): add dropdown actions and fix endpoint URL on ServiceCard"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Rebuild ComputePage with actions, create button, logs, and enhanced detail view
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/dashboard/src/features/compute/pages/ComputePage.tsx`
|
||||
|
||||
- [ ] **Step 1: Replace ComputePage with the full implementation**
|
||||
|
||||
```tsx
|
||||
import { useState } from 'react';
|
||||
import { Loader2, ArrowLeft, Plus, Play, Square, Trash2, AlertTriangle } from 'lucide-react';
|
||||
import { Button, ConfirmDialog } from '@insforge/ui';
|
||||
import { useComputeServices } from '../hooks/useComputeServices';
|
||||
import { ServiceCard } from '../components/ServiceCard';
|
||||
import { ServiceLogs } from '../components/ServiceLogs';
|
||||
import { CreateServiceDialog } from '../components/CreateServiceDialog';
|
||||
import { statusColors, getReachableUrl } from '../constants';
|
||||
import type { ServiceSchema } from '@insforge/shared-schemas';
|
||||
|
||||
export default function ComputePage() {
|
||||
const { services, isLoading, create, remove, stop, start, isCreating } = useComputeServices();
|
||||
const [selectedService, setSelectedService] = useState<ServiceSchema | null>(null);
|
||||
const [createOpen, setCreateOpen] = useState(false);
|
||||
const [deleteTarget, setDeleteTarget] = useState<string | null>(null);
|
||||
|
||||
// Keep selected service in sync with latest data
|
||||
const currentService = selectedService
|
||||
? services.find((s) => s.id === selectedService.id) ?? selectedService
|
||||
: null;
|
||||
|
||||
const handleDelete = async (id: string) => {
|
||||
await remove(id);
|
||||
if (selectedService?.id === id) {
|
||||
setSelectedService(null);
|
||||
}
|
||||
setDeleteTarget(null);
|
||||
};
|
||||
|
||||
if (isLoading) {
|
||||
return (
|
||||
<div className="flex items-center justify-center h-64">
|
||||
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (currentService) {
|
||||
const reachableUrl = getReachableUrl(currentService);
|
||||
|
||||
return (
|
||||
<div className="h-full flex flex-col bg-[rgb(var(--semantic-0))]">
|
||||
<div className="flex-1 min-h-0 overflow-y-auto px-10">
|
||||
<div className="max-w-[1024px] w-full mx-auto flex flex-col gap-6 pt-10 pb-6">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setSelectedService(null)}
|
||||
className="flex items-center gap-1.5 text-sm text-muted-foreground hover:text-foreground transition-colors self-start"
|
||||
>
|
||||
<ArrowLeft className="h-4 w-4" />
|
||||
Back to services
|
||||
</button>
|
||||
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex items-center gap-3">
|
||||
<h1 className="text-2xl font-medium text-foreground leading-8">
|
||||
{currentService.name}
|
||||
</h1>
|
||||
<span className="flex items-center gap-1.5 text-sm text-muted-foreground">
|
||||
<span
|
||||
className={`inline-block h-2 w-2 rounded-full ${statusColors[currentService.status]}`}
|
||||
/>
|
||||
{currentService.status}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<div className="flex items-center gap-2">
|
||||
{currentService.status === 'running' && (
|
||||
<Button
|
||||
variant="secondary"
|
||||
size="sm"
|
||||
onClick={() => void stop(currentService.id)}
|
||||
>
|
||||
<Square className="h-3.5 w-3.5" />
|
||||
Stop
|
||||
</Button>
|
||||
)}
|
||||
{currentService.status === 'stopped' && (
|
||||
<Button
|
||||
variant="secondary"
|
||||
size="sm"
|
||||
onClick={() => void start(currentService.id)}
|
||||
>
|
||||
<Play className="h-3.5 w-3.5" />
|
||||
Start
|
||||
</Button>
|
||||
)}
|
||||
<Button
|
||||
variant="destructive"
|
||||
size="sm"
|
||||
onClick={() => setDeleteTarget(currentService.id)}
|
||||
>
|
||||
<Trash2 className="h-3.5 w-3.5" />
|
||||
Delete
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{currentService.status === 'failed' && (
|
||||
<div className="flex items-center gap-2 px-4 py-3 bg-destructive/10 border border-destructive/20 rounded-lg text-sm text-destructive">
|
||||
<AlertTriangle className="h-4 w-4 shrink-0" />
|
||||
This service failed to deploy. You can delete it and try again.
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-6">
|
||||
<dl className="grid grid-cols-2 gap-4 text-sm">
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">Image</dt>
|
||||
<dd className="text-foreground break-all">
|
||||
{currentService.imageUrl === 'dockerfile'
|
||||
? 'Built from Dockerfile'
|
||||
: currentService.imageUrl}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">Port</dt>
|
||||
<dd className="text-foreground">{currentService.port}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">CPU</dt>
|
||||
<dd className="text-foreground">{currentService.cpu}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">Memory</dt>
|
||||
<dd className="text-foreground">{currentService.memory} MB</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">Region</dt>
|
||||
<dd className="text-foreground">{currentService.region}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt className="text-muted-foreground mb-1">Created</dt>
|
||||
<dd className="text-foreground">
|
||||
{new Date(currentService.createdAt).toLocaleDateString(undefined, {
|
||||
year: 'numeric',
|
||||
month: 'short',
|
||||
day: 'numeric',
|
||||
hour: '2-digit',
|
||||
minute: '2-digit',
|
||||
})}
|
||||
</dd>
|
||||
</div>
|
||||
<div className="col-span-2">
|
||||
<dt className="text-muted-foreground mb-1">Endpoint URL</dt>
|
||||
<dd className="text-foreground">
|
||||
{reachableUrl ? (
|
||||
<a
|
||||
href={reachableUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
className="text-primary hover:underline"
|
||||
>
|
||||
{reachableUrl}
|
||||
</a>
|
||||
) : (
|
||||
<span className="text-muted-foreground">Not available</span>
|
||||
)}
|
||||
</dd>
|
||||
</div>
|
||||
</dl>
|
||||
</div>
|
||||
|
||||
{currentService.flyMachineId && (
|
||||
<ServiceLogs serviceId={currentService.id} />
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<ConfirmDialog
|
||||
open={deleteTarget !== null}
|
||||
onOpenChange={(open) => { if (!open) setDeleteTarget(null); }}
|
||||
title="Delete service"
|
||||
description={`This will permanently delete "${currentService.name}" and destroy its Fly.io resources. This cannot be undone.`}
|
||||
confirmText="Delete"
|
||||
destructive
|
||||
onConfirm={() => handleDelete(deleteTarget!)}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="h-full flex flex-col bg-[rgb(var(--semantic-0))]">
|
||||
<div className="flex-1 min-h-0 overflow-y-auto px-10">
|
||||
<div className="max-w-[1024px] w-full mx-auto flex flex-col gap-8 pt-10 pb-6">
|
||||
{/* Services Section */}
|
||||
<div className="flex flex-col gap-4">
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="flex flex-col gap-1">
|
||||
<h1 className="text-2xl font-medium text-foreground leading-8">Services</h1>
|
||||
<p className="text-sm leading-5 text-muted-foreground">
|
||||
Deploy and manage long-running containers on your infrastructure.
|
||||
</p>
|
||||
</div>
|
||||
<Button variant="primary" size="sm" onClick={() => setCreateOpen(true)}>
|
||||
<Plus className="h-4 w-4" />
|
||||
Create Service
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{services.length === 0 ? (
|
||||
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-8 text-center">
|
||||
<p className="text-sm text-muted-foreground mb-2">No services deployed yet.</p>
|
||||
<p className="text-xs text-muted-foreground mb-4">
|
||||
Create a service using the button above or the CLI:{' '}
|
||||
<code className="px-1.5 py-0.5 bg-muted rounded text-xs">
|
||||
insforge compute create --name my-api --image nginx:alpine
|
||||
</code>
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
|
||||
{services.map((service) => (
|
||||
<ServiceCard
|
||||
key={service.id}
|
||||
service={service}
|
||||
onClick={() => setSelectedService(service)}
|
||||
onStop={(id) => void stop(id)}
|
||||
onStart={(id) => void start(id)}
|
||||
onDelete={setDeleteTarget}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Jobs Section Placeholder */}
|
||||
<div className="flex flex-col gap-2">
|
||||
<h2 className="text-lg font-medium text-foreground">Jobs</h2>
|
||||
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-6 text-center">
|
||||
<p className="text-sm text-muted-foreground">Coming soon</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<CreateServiceDialog
|
||||
open={createOpen}
|
||||
onOpenChange={setCreateOpen}
|
||||
onCreate={create}
|
||||
isCreating={isCreating}
|
||||
/>
|
||||
|
||||
<ConfirmDialog
|
||||
open={deleteTarget !== null}
|
||||
onOpenChange={(open) => { if (!open) setDeleteTarget(null); }}
|
||||
title="Delete service"
|
||||
description="This will permanently delete this service and destroy its Fly.io resources. This cannot be undone."
|
||||
confirmText="Delete"
|
||||
destructive
|
||||
onConfirm={() => {
|
||||
if (deleteTarget) return handleDelete(deleteTarget);
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify the app builds**
|
||||
|
||||
Run: `cd packages/dashboard && npx tsc --noEmit`
|
||||
Expected: No type errors
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add packages/dashboard/src/features/compute/pages/ComputePage.tsx
|
||||
git commit -m "feat(compute/ui): add create button, action buttons, logs panel, and enhanced detail view"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Verify everything works end-to-end
|
||||
|
||||
- [ ] **Step 1: Run type check across the whole project**
|
||||
|
||||
Run: `npm run typecheck`
|
||||
Expected: No errors
|
||||
|
||||
- [ ] **Step 2: Run all tests**
|
||||
|
||||
Run: `npm test`
|
||||
Expected: All 471+ tests pass
|
||||
|
||||
- [ ] **Step 3: Final commit if any fixes were needed**
|
||||
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix(compute/ui): address typecheck and test issues"
|
||||
```
|
||||
@@ -0,0 +1,811 @@
|
||||
# Direct Deploy Flow Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement and ship a manifest-based direct deployment flow that streams source files through InsForge to Vercel without exposing Vercel credentials, while preserving the legacy zip/S3 deployment path for backward compatibility.
|
||||
|
||||
**Architecture:** The backend exposes a direct deployment session endpoint that stores deployment rows in `deployments.runs`, stores manifest/upload state in `deployments.files`, and returns a `fileId` per entry. Clients then stream each file to `PUT /api/deployments/:id/files/:fileId/content`; InsForge validates the byte stream against the registered SHA/size and proxies it to Vercel's file upload API. Once every manifest row is marked uploaded, the client calls `POST /api/deployments/:id/start`, and InsForge creates the Vercel deployment from uploaded file SHAs. MCP uses the direct flow when the backend version supports it, and falls back to the legacy zip upload flow when the backend is older or returns `404`.
|
||||
|
||||
**Tech Stack:** Express, PostgreSQL, Zod shared schemas, Axios + node-fetch streaming, Vercel file upload API, React service layer, Docker Compose, Vitest, MCP server tooling
|
||||
|
||||
**Reference:** `docs/core-concepts/deployments/architecture.mdx`
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### InsForge repo
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `packages/shared-schemas/src/deployments-api.schema.ts` | Shared request/response contracts for direct deployment manifest, file upload response, and start payload |
|
||||
| `backend/src/infra/database/migrations/031_create-deployment-files.sql` | Moves `system.deployments` to `deployments.runs` and creates `deployments.files` for direct upload tracking |
|
||||
| `backend/src/types/deployments.ts` | Backend deployment status semantics used by both direct and legacy flows |
|
||||
| `backend/src/api/routes/deployments/index.routes.ts` | HTTP entrypoints for legacy create, direct create, file upload, start, metadata, slug, and custom domains |
|
||||
| `backend/src/services/deployments/deployment.service.ts` | Core orchestration for manifest validation, file upload proxying, direct-vs-legacy branching, and self-hosted Vercel support |
|
||||
| `backend/src/providers/deployments/vercel.provider.ts` | Streaming file upload implementation against Vercel's `/v2/files` API |
|
||||
| `backend/src/server.ts` | Global rate-limit bypass for the direct file upload endpoint |
|
||||
| `backend/tests/unit/deployment-direct-flow.test.ts` | Service-level coverage for direct deployment session creation, upload validation, and self-hosted config behavior |
|
||||
| `backend/tests/unit/vercel-upload-batching.test.ts` | Provider-level coverage for streaming uploads, 409 dedupe handling, and 429 retry semantics |
|
||||
| `packages/dashboard/src/features/deployments/services/deployments.service.ts` | Client methods mirroring the direct deployment endpoints for future UI or SDK consumers |
|
||||
| `.env.example` | Root sample env showing Vercel credentials required for self-hosted deployments |
|
||||
| `docker-compose.yml` | Dev compose pass-through for Vercel credentials |
|
||||
| `docker-compose.prod.yml` | Prod compose pass-through for Vercel credentials |
|
||||
| `deploy/docker-compose/.env.example` | Packaged deployment sample env |
|
||||
| `deploy/docker-compose/docker-compose.yml` | Packaged deployment compose pass-through |
|
||||
| `docs/agent-docs/deployment.md` | Agent-facing deployment instructions |
|
||||
| `docs/core-concepts/deployments/architecture.mdx` | Product/architecture documentation for both direct and legacy flows |
|
||||
|
||||
### MCP repo
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `../insforge-mcp/src/shared/tools/types.ts` | Register context carrying backend version into tool registration |
|
||||
| `../insforge-mcp/src/shared/tools/index.ts` | Backend version discovery and tool registration |
|
||||
| `../insforge-mcp/src/shared/tools/deployment.ts` | Direct deployment manifest collection, bounded parallel uploads, remote shell instructions, and legacy fallback |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Lock the shared contract and manifest persistence layer
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/shared-schemas/src/deployments-api.schema.ts`
|
||||
- Modify: `backend/src/infra/database/migrations/031_create-deployment-files.sql`
|
||||
- Modify: `backend/src/types/deployments.ts`
|
||||
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
|
||||
- Test: `backend/tests/unit/deployment-schema-migration.test.ts`
|
||||
|
||||
- [ ] **Step 1: Add the direct deployment request/response schemas**
|
||||
|
||||
```ts
|
||||
export const deploymentManifestFileEntrySchema = z.object({
|
||||
path: deploymentFilePathSchema,
|
||||
sha: z.string().regex(/^[a-f0-9]{40}$/i, 'sha must be a SHA-1 hex digest'),
|
||||
size: z.number().int().nonnegative(),
|
||||
});
|
||||
|
||||
export const deploymentManifestFileSchema = deploymentManifestFileEntrySchema.extend({
|
||||
fileId: z.string().uuid(),
|
||||
uploadedAt: z.string().datetime().nullable(),
|
||||
});
|
||||
|
||||
export const createDirectDeploymentRequestSchema = z
|
||||
.object({
|
||||
files: z.array(deploymentManifestFileEntrySchema).min(1),
|
||||
})
|
||||
.superRefine(({ files }, ctx) => {
|
||||
const firstSeenByPath = new Map<string, number>();
|
||||
|
||||
files.forEach((file, index) => {
|
||||
const existingIndex = firstSeenByPath.get(file.path);
|
||||
if (existingIndex !== undefined) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
message: 'duplicate file path',
|
||||
path: ['files', index, 'path'],
|
||||
});
|
||||
return;
|
||||
}
|
||||
firstSeenByPath.set(file.path, index);
|
||||
});
|
||||
});
|
||||
|
||||
export const createDirectDeploymentResponseSchema = z.object({
|
||||
id: z.string().uuid(),
|
||||
status: deploymentSchema.shape.status,
|
||||
files: z.array(deploymentManifestFileSchema),
|
||||
});
|
||||
|
||||
export const uploadDeploymentFileResponseSchema = deploymentManifestFileSchema.extend({
|
||||
uploadedAt: z.string().datetime(),
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Ensure migration 031 moves the published deployments table and creates resumable file tracking**
|
||||
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS deployments;
|
||||
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (
|
||||
SELECT 1
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'system' AND table_name = 'deployments'
|
||||
) THEN
|
||||
ALTER TABLE system.deployments SET SCHEMA deployments;
|
||||
END IF;
|
||||
|
||||
IF EXISTS (
|
||||
SELECT 1
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'deployments' AND table_name = 'deployments'
|
||||
) AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'deployments' AND table_name = 'runs'
|
||||
) THEN
|
||||
ALTER TABLE deployments.deployments RENAME TO runs;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS deployments.files (
|
||||
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||
deployment_id UUID NOT NULL REFERENCES deployments.runs(id) ON DELETE CASCADE,
|
||||
file_path TEXT NOT NULL,
|
||||
sha TEXT NOT NULL CHECK (sha ~ '^[a-f0-9]{40}$'),
|
||||
size_bytes INTEGER NOT NULL CHECK (size_bytes >= 0),
|
||||
uploaded_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
UNIQUE (deployment_id, file_path)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_deployment_files_deployment_id
|
||||
ON deployments.files(deployment_id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_deployment_files_uploaded_at
|
||||
ON deployments.files(deployment_id, uploaded_at);
|
||||
|
||||
DROP TRIGGER IF EXISTS update_system_deployments_updated_at ON deployments.runs;
|
||||
DROP TRIGGER IF EXISTS update_deployments_updated_at ON deployments.runs;
|
||||
DROP TRIGGER IF EXISTS update_runs_updated_at ON deployments.runs;
|
||||
CREATE TRIGGER update_runs_updated_at BEFORE UPDATE ON deployments.runs
|
||||
FOR EACH ROW EXECUTE FUNCTION system.update_updated_at();
|
||||
|
||||
DROP TRIGGER IF EXISTS update_system_deployment_files_updated_at ON deployments.files;
|
||||
DROP TRIGGER IF EXISTS update_deployment_files_updated_at ON deployments.files;
|
||||
DROP TRIGGER IF EXISTS update_files_updated_at ON deployments.files;
|
||||
CREATE TRIGGER update_files_updated_at BEFORE UPDATE ON deployments.files
|
||||
FOR EACH ROW EXECUTE FUNCTION system.update_updated_at();
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add failing service tests that pin the contract**
|
||||
|
||||
```ts
|
||||
it('creates a direct deployment and stores its manifest in one transaction', async () => {
|
||||
const service = DeploymentService.getInstance();
|
||||
|
||||
const result = await service.createDirectDeployment({
|
||||
files: [{ path: 'src/index.ts', sha: 'A'.repeat(40), size: 12 }],
|
||||
});
|
||||
|
||||
expect(result.status).toBe(DeploymentStatus.WAITING);
|
||||
expect(result.files[0]).toMatchObject({
|
||||
path: 'src/index.ts',
|
||||
sha: 'a'.repeat(40),
|
||||
size: 12,
|
||||
uploadedAt: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('rejects duplicate manifest paths before opening a transaction', async () => {
|
||||
const service = DeploymentService.getInstance();
|
||||
|
||||
await expect(
|
||||
service.createDirectDeployment({
|
||||
files: [
|
||||
{ path: 'src/index.ts', sha: 'a'.repeat(40), size: 12 },
|
||||
{ path: 'src/index.ts', sha: 'b'.repeat(40), size: 13 },
|
||||
],
|
||||
})
|
||||
).rejects.toMatchObject({
|
||||
statusCode: 400,
|
||||
code: 'INVALID_INPUT',
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted backend test to verify the contract is red first, then green after implementation**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/backend
|
||||
npm test -- deployment-direct-flow.test.ts
|
||||
```
|
||||
|
||||
Expected before backend implementation: failure around missing direct deployment behavior or schema mismatch
|
||||
Expected after Task 2: `Test Files 1 passed`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge
|
||||
git add \
|
||||
packages/shared-schemas/src/deployments-api.schema.ts \
|
||||
backend/src/infra/database/migrations/031_create-deployment-files.sql \
|
||||
backend/src/types/deployments.ts \
|
||||
backend/tests/unit/deployment-direct-flow.test.ts
|
||||
git commit -m "feat: add direct deployment manifest contracts"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Implement the backend direct upload proxy
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/api/routes/deployments/index.routes.ts`
|
||||
- Modify: `backend/src/services/deployments/deployment.service.ts`
|
||||
- Modify: `backend/src/providers/deployments/vercel.provider.ts`
|
||||
- Modify: `backend/src/server.ts`
|
||||
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
|
||||
- Test: `backend/tests/unit/vercel-upload-batching.test.ts`
|
||||
|
||||
- [ ] **Step 1: Add the direct deployment HTTP routes**
|
||||
|
||||
```ts
|
||||
router.post('/direct', verifyAdmin, async (req, res, next) => {
|
||||
try {
|
||||
const validationResult = createDirectDeploymentRequestSchema.safeParse(req.body);
|
||||
if (!validationResult.success) {
|
||||
throw new AppError(
|
||||
validationResult.error.issues.map((e) => `${e.path.join('.')}: ${e.message}`).join(', '),
|
||||
400,
|
||||
ERROR_CODES.INVALID_INPUT
|
||||
);
|
||||
}
|
||||
|
||||
const response = await deploymentService.createDirectDeployment(validationResult.data);
|
||||
successResponse(res, response, 201);
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
});
|
||||
|
||||
router.put('/:id/files/:fileId/content', verifyAdmin, async (req, res, next) => {
|
||||
try {
|
||||
const abortController = new AbortController();
|
||||
req.on('aborted', () => abortController.abort());
|
||||
res.on('close', () => {
|
||||
if (!res.writableEnded) abortController.abort();
|
||||
});
|
||||
|
||||
const response = await deploymentService.uploadDeploymentFileContent(
|
||||
req.params.id,
|
||||
req.params.fileId,
|
||||
req,
|
||||
{ signal: abortController.signal }
|
||||
);
|
||||
|
||||
successResponse(res, response);
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Implement manifest registration, byte validation, and direct-vs-legacy start branching in the service**
|
||||
|
||||
```ts
|
||||
async createDirectDeployment(
|
||||
input: CreateDirectDeploymentRequest
|
||||
): Promise<CreateDirectDeploymentResponse> {
|
||||
this.assertDeploymentServiceConfigured();
|
||||
|
||||
const files = this.validateDeploymentManifest(input.files);
|
||||
const totalSizeBytes = files.reduce((sum, file) => sum + file.size, 0);
|
||||
const client = await this.getPool().connect();
|
||||
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const deployment = await client.query(
|
||||
`INSERT INTO deployments.runs (provider, status, metadata)
|
||||
VALUES ($1, $2, $3)
|
||||
RETURNING
|
||||
id,
|
||||
provider_deployment_id as "providerDeploymentId",
|
||||
provider,
|
||||
status,
|
||||
url,
|
||||
metadata,
|
||||
created_at as "createdAt",
|
||||
updated_at as "updatedAt"`,
|
||||
[
|
||||
'vercel',
|
||||
DeploymentStatus.WAITING,
|
||||
JSON.stringify({
|
||||
uploadMode: 'direct',
|
||||
fileCount: files.length,
|
||||
totalSizeBytes,
|
||||
manifestCreatedAt: new Date().toISOString(),
|
||||
}),
|
||||
]
|
||||
);
|
||||
const insertedFiles = await this.insertDeploymentFiles(client, deployment.rows[0].id, files);
|
||||
await client.query('COMMIT');
|
||||
|
||||
return {
|
||||
id: deployment.rows[0].id,
|
||||
status: deployment.rows[0].status,
|
||||
files: insertedFiles.map((row) => this.toDeploymentFileResponse(row)),
|
||||
};
|
||||
} catch (error) {
|
||||
await client.query('ROLLBACK').catch(() => {});
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
async uploadDeploymentFileContent(
|
||||
id: string,
|
||||
fileId: string,
|
||||
content: Readable,
|
||||
options: { signal?: AbortSignal } = {}
|
||||
): Promise<UploadDeploymentFileResponse> {
|
||||
const deployment = await this.getDeploymentById(id);
|
||||
const file = await this.getDeploymentFileById(id, fileId);
|
||||
|
||||
await this.vercelProvider.uploadFileStream({
|
||||
content: this.createValidatedFileStream(content, file.sha, file.size),
|
||||
sha: file.sha,
|
||||
size: file.size,
|
||||
signal: options.signal,
|
||||
});
|
||||
|
||||
const uploadedFile = await this.markDeploymentFileUploaded(id, fileId);
|
||||
return {
|
||||
...this.toDeploymentFileResponse(uploadedFile),
|
||||
uploadedAt: uploadedFile.uploadedAt.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
async startDeployment(id: string, input: StartDeploymentRequest = {}): Promise<DeploymentRecord> {
|
||||
const deployment = await this.getDeploymentById(id);
|
||||
const files = await this.getDeploymentFiles(id);
|
||||
const uploadMode = this.getUploadMode(deployment, files.length);
|
||||
|
||||
if (uploadMode === 'direct') {
|
||||
return this.startDirectDeployment(id, input, files);
|
||||
}
|
||||
return this.startLegacyDeployment(id, input);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Stream bytes to Vercel without buffering or automatic retry**
|
||||
|
||||
```ts
|
||||
async uploadFileStream(input: {
|
||||
content: Readable;
|
||||
sha: string;
|
||||
size: number;
|
||||
signal?: AbortSignal;
|
||||
}): Promise<string> {
|
||||
const credentials = await this.getCredentials();
|
||||
|
||||
try {
|
||||
await axios.post(`https://api.vercel.com/v2/files?teamId=${credentials.teamId}`, input.content, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${credentials.token}`,
|
||||
'Content-Type': 'application/octet-stream',
|
||||
'Content-Length': input.size.toString(),
|
||||
'x-vercel-digest': input.sha,
|
||||
},
|
||||
maxBodyLength: Infinity,
|
||||
maxContentLength: Infinity,
|
||||
signal: input.signal,
|
||||
});
|
||||
|
||||
return input.sha;
|
||||
} catch (error) {
|
||||
if (axios.isAxiosError(error) && error.response?.status === 409) return input.sha;
|
||||
if (axios.isAxiosError(error) && error.response?.status === 429) {
|
||||
throw new AppError(
|
||||
'Vercel rate limit exceeded for file upload. Wait a moment and retry the file upload.',
|
||||
429,
|
||||
ERROR_CODES.RATE_LIMITED
|
||||
);
|
||||
}
|
||||
if (axios.isAxiosError(error) && error.code === 'ERR_CANCELED') {
|
||||
throw new AppError('Vercel file upload was interrupted.', 499, ERROR_CODES.DEPLOYMENT_UPLOAD_CANCELED);
|
||||
}
|
||||
throw new AppError('Failed to upload file to Vercel', 500, ERROR_CODES.INTERNAL_ERROR);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Bypass the global Express rate limiter for per-file direct uploads**
|
||||
|
||||
```ts
|
||||
function shouldSkipGlobalRateLimit(req: Request): boolean {
|
||||
if (req.path === '/api/health') {
|
||||
return true;
|
||||
}
|
||||
|
||||
return (
|
||||
req.method === 'PUT' && /^\/api\/deployments\/[^/]+\/files\/[^/]+\/content$/.test(req.path)
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run focused backend verification**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/backend
|
||||
npm test -- deployment-direct-flow.test.ts
|
||||
npm test -- vercel-upload-batching.test.ts
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- `deployment-direct-flow.test.ts` passes
|
||||
- `vercel-upload-batching.test.ts` passes
|
||||
- `tsup` build succeeds
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge
|
||||
git add \
|
||||
backend/src/api/routes/deployments/index.routes.ts \
|
||||
backend/src/services/deployments/deployment.service.ts \
|
||||
backend/src/providers/deployments/vercel.provider.ts \
|
||||
backend/src/server.ts \
|
||||
backend/tests/unit/deployment-direct-flow.test.ts \
|
||||
backend/tests/unit/vercel-upload-batching.test.ts
|
||||
git commit -m "feat: add direct deployment upload proxy"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Wire MCP local and remote deployment flows with compatibility fallback
|
||||
|
||||
**Files:**
|
||||
- Modify: `../insforge-mcp/src/shared/tools/types.ts`
|
||||
- Modify: `../insforge-mcp/src/shared/tools/index.ts`
|
||||
- Modify: `../insforge-mcp/src/shared/tools/deployment.ts`
|
||||
|
||||
- [ ] **Step 1: Carry backend version through MCP tool registration context**
|
||||
|
||||
```ts
|
||||
export interface RegisterContext {
|
||||
API_BASE_URL: string;
|
||||
backendVersion: string;
|
||||
isRemote: boolean;
|
||||
registerTool: RegisterToolFn;
|
||||
withUsageTracking: WithUsageTracking;
|
||||
getApiKey: GetApiKey;
|
||||
addBackgroundContext: AddBackgroundContext;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add direct deployment helpers with bounded parallel file uploads**
|
||||
|
||||
```ts
|
||||
const DIRECT_DEPLOYMENT_MIN_VERSION = '2.0.4';
|
||||
const DEFAULT_DIRECT_UPLOAD_CONCURRENCY = 8;
|
||||
const MAX_DIRECT_UPLOAD_CONCURRENCY = 32;
|
||||
|
||||
async function deployDirect(
|
||||
API_BASE_URL: string,
|
||||
apiKey: string,
|
||||
sourceDirectory: string,
|
||||
startBody: StartDeploymentRequest
|
||||
) {
|
||||
const files = await collectDeploymentFiles(sourceDirectory);
|
||||
const manifestFiles = files.map(({ path, sha, size }) => ({ path, sha, size }));
|
||||
const createResult = await createDirectDeploymentSession(API_BASE_URL, apiKey, manifestFiles);
|
||||
const localFileByPath = new Map(files.map((file) => [file.path, file] as const));
|
||||
const uploadConcurrency = getDirectUploadConcurrency();
|
||||
|
||||
await runWithConcurrency(createResult.files, uploadConcurrency, async (file) => {
|
||||
const localFile = localFileByPath.get(file.path);
|
||||
await uploadDeploymentFileContent(API_BASE_URL, apiKey, createResult.id, file, localFile);
|
||||
});
|
||||
|
||||
const startResult = await startDeployment(API_BASE_URL, apiKey, createResult.id, startBody);
|
||||
return { deploymentId: createResult.id, fileCount: files.length, uploadConcurrency, startResult };
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Prefer the direct flow when supported, but fall back to legacy zip on older backends or `404`**
|
||||
|
||||
```ts
|
||||
const supportsDirectDeployment = supportsDirectDeploymentVersion(backendVersion);
|
||||
|
||||
if (supportsDirectDeployment) {
|
||||
try {
|
||||
const { fileCount, uploadConcurrency, startResult } = await deployDirect(
|
||||
API_BASE_URL,
|
||||
getApiKey(),
|
||||
resolvedSourceDir,
|
||||
startBody
|
||||
);
|
||||
|
||||
return await addBackgroundContext({
|
||||
content: [
|
||||
{
|
||||
type: 'text',
|
||||
text:
|
||||
formatSuccessMessage('Deployment started', startResult) +
|
||||
`\n\nUploaded ${fileCount} files through direct deployment proxy with concurrency ${uploadConcurrency}.`,
|
||||
},
|
||||
],
|
||||
});
|
||||
} catch (error) {
|
||||
if (!(error instanceof DirectDeploymentUnsupportedError)) {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback to the legacy zip + S3 path.
|
||||
const createResponse = await fetch(`${API_BASE_URL}/api/deployments`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'x-api-key': getApiKey(),
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Return remote shell instructions that match the same direct flow**
|
||||
|
||||
```ts
|
||||
return `Direct deployment upload is available for this backend.
|
||||
|
||||
Please execute the following command locally from a shell that has INSFORGE_API_KEY set, then call the \`start-deployment\` tool with the deployment ID printed by the script. Set \`INSFORGE_DEPLOY_UPLOAD_CONCURRENCY\` if you want to tune parallel uploads; the default is 8 and the maximum is 32.
|
||||
|
||||
\`\`\`bash
|
||||
cd ${escapedDir}
|
||||
INSFORGE_API_KEY="\${INSFORGE_API_KEY:?Set INSFORGE_API_KEY to your InsForge API key}" node --input-type=module <<'NODE'
|
||||
const createResult = await api('/api/deployments/direct', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
files: localFiles.map(({ path, sha, size }) => ({ path, sha, size })),
|
||||
}),
|
||||
});
|
||||
|
||||
await runWithConcurrency(createResult.files, uploadConcurrency, async (manifestFile) => {
|
||||
const localFile = localFileByPath.get(manifestFile.path);
|
||||
await uploadFile(createResult.id, manifestFile, localFile);
|
||||
});
|
||||
|
||||
console.log('Deployment files uploaded. Deployment ID: ' + createResult.id);
|
||||
NODE
|
||||
\`\`\`
|
||||
|
||||
If the upload is interrupted after the deployment ID is printed, query \`deployments.files\` with the raw SQL tool for that \`deployment_id\` to inspect \`uploaded_at\`.`;
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Verify MCP compilation and smoke-test tool registration**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/insforge-mcp
|
||||
npm test
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- `vitest run --passWithNoTests` exits successfully
|
||||
- `tsup` build succeeds
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/insforge-mcp
|
||||
git add \
|
||||
src/shared/tools/types.ts \
|
||||
src/shared/tools/index.ts \
|
||||
src/shared/tools/deployment.ts
|
||||
git commit -m "feat: add direct deployment support to mcp"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Surface self-hosted configuration and document the new flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/dashboard/src/features/deployments/services/deployments.service.ts`
|
||||
- Modify: `.env.example`
|
||||
- Modify: `docker-compose.yml`
|
||||
- Modify: `docker-compose.prod.yml`
|
||||
- Modify: `deploy/docker-compose/.env.example`
|
||||
- Modify: `deploy/docker-compose/docker-compose.yml`
|
||||
- Modify: `docs/agent-docs/deployment.md`
|
||||
- Modify: `docs/core-concepts/deployments/architecture.mdx`
|
||||
|
||||
- [ ] **Step 1: Mirror the direct deployment endpoints in the dashboard service client**
|
||||
|
||||
```ts
|
||||
async createDirectDeployment(
|
||||
data: CreateDirectDeploymentRequest
|
||||
): Promise<CreateDirectDeploymentResponse> {
|
||||
return apiClient.request('/deployments/direct', {
|
||||
method: 'POST',
|
||||
headers: apiClient.withAccessToken(),
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
}
|
||||
|
||||
async uploadDeploymentFileContent(
|
||||
id: string,
|
||||
fileId: string,
|
||||
content: Blob | ArrayBuffer
|
||||
): Promise<UploadDeploymentFileResponse> {
|
||||
return apiClient.request(`/deployments/${id}/files/${fileId}/content`, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
...apiClient.withAccessToken(),
|
||||
'Content-Type': 'application/octet-stream',
|
||||
},
|
||||
body: content,
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add self-hosted Vercel credentials to the sample env and compose files**
|
||||
|
||||
```env
|
||||
# Deployment Configuration (Optional)
|
||||
# Required for self-hosted site deployments and custom domains.
|
||||
VERCEL_TOKEN=
|
||||
VERCEL_TEAM_ID=
|
||||
VERCEL_PROJECT_ID=
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Deployment Configuration
|
||||
- VERCEL_TOKEN=${VERCEL_TOKEN:-}
|
||||
- VERCEL_TEAM_ID=${VERCEL_TEAM_ID:-}
|
||||
- VERCEL_PROJECT_ID=${VERCEL_PROJECT_ID:-}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Document the direct flow and the legacy fallback explicitly**
|
||||
|
||||
```md
|
||||
## Overview
|
||||
|
||||
Deploy frontend applications to InsForge using the `create-deployment` MCP tool. The tool handles uploading source files, building, and deploying automatically.
|
||||
Source files are uploaded individually through InsForge's deployment proxy; do not zip the project or upload deployment artifacts to storage yourself.
|
||||
The REST API still supports the legacy zip upload flow for backward compatibility.
|
||||
```
|
||||
|
||||
```mdx
|
||||
### Direct Upload Flow
|
||||
|
||||
1. **Create Deployment**: Agent calls `POST /api/deployments/direct` with each relative file path, SHA-1 digest, and byte size
|
||||
2. **Upload Files**: Agent streams each file to `PUT /api/deployments/:id/files/:fileId/content`
|
||||
3. **Retry If Needed**: Inspect `deployments.files` for `uploaded_at`
|
||||
4. **Start Build**: Agent calls `POST /api/deployments/:id/start`
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify the sample config and dashboard package build**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge
|
||||
docker compose -f docker-compose.yml config --quiet
|
||||
docker compose -f docker-compose.prod.yml config --quiet
|
||||
docker compose -f deploy/docker-compose/docker-compose.yml config --quiet
|
||||
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard
|
||||
npm run typecheck
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- all three `docker compose ... config --quiet` commands exit 0
|
||||
- dashboard typecheck succeeds
|
||||
- dashboard build succeeds
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge
|
||||
git add \
|
||||
packages/dashboard/src/features/deployments/services/deployments.service.ts \
|
||||
.env.example \
|
||||
docker-compose.yml \
|
||||
docker-compose.prod.yml \
|
||||
deploy/docker-compose/.env.example \
|
||||
deploy/docker-compose/docker-compose.yml \
|
||||
docs/agent-docs/deployment.md \
|
||||
docs/core-concepts/deployments/architecture.mdx
|
||||
git commit -m "docs: document direct deploy flow and self-host config"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Run the full cross-repo verification pass
|
||||
|
||||
**Files:**
|
||||
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
|
||||
- Test: `backend/tests/unit/vercel-upload-batching.test.ts`
|
||||
- Test: `../insforge-mcp/src/shared/tools/deployment.ts`
|
||||
- Test: compose manifests and docs touched above
|
||||
|
||||
- [ ] **Step 1: Run the full backend verification suite**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/backend
|
||||
npm test
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- `vitest run` passes
|
||||
- backend `tsup` build succeeds
|
||||
|
||||
- [ ] **Step 2: Run shared schema and dashboard verification**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas
|
||||
npm run build
|
||||
|
||||
cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard
|
||||
npm run typecheck
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- shared schemas `tsc` build succeeds
|
||||
- dashboard typecheck and build both succeed
|
||||
|
||||
- [ ] **Step 3: Run MCP verification**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/insforge-mcp
|
||||
npm test
|
||||
npm run build
|
||||
```
|
||||
|
||||
Expected:
|
||||
- MCP tests succeed
|
||||
- MCP build succeeds
|
||||
|
||||
- [ ] **Step 4: Perform the real smoke tests**
|
||||
|
||||
```text
|
||||
1. Legacy backend smoke test:
|
||||
- Call POST /api/deployments
|
||||
- Upload a zip to the returned presigned URL
|
||||
- Call POST /api/deployments/:id/start
|
||||
- Confirm the flow still works unchanged
|
||||
|
||||
2. Direct backend smoke test:
|
||||
- Call POST /api/deployments/direct with a small manifest
|
||||
- Upload at least one file to PUT /api/deployments/:id/files/:fileId/content
|
||||
- Call POST /api/deployments/:id/start
|
||||
- Confirm deployments.runs transitions to READY
|
||||
|
||||
3. MCP smoke test:
|
||||
- Against a direct-capable backend, run create-deployment from a real source directory
|
||||
- Confirm direct file uploads happen with bounded concurrency
|
||||
- Against an older backend or forced 404, confirm fallback to legacy zip upload
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Final commit or tag-ready checkpoint**
|
||||
|
||||
```bash
|
||||
cd /Users/lyu/Documents/GitHub/InsForge
|
||||
git status
|
||||
git log --oneline -n 5
|
||||
```
|
||||
|
||||
Expected:
|
||||
- working tree clean or only intentionally staged release changes
|
||||
- recent commits show the contract, backend, MCP, and docs/config slices in order
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
### Spec coverage
|
||||
|
||||
- Shared schemas cover manifest registration, returned `fileId`s, upload response, and start payloads.
|
||||
- Backend plan covers `deployments.runs` / `deployments.files` persistence, streaming proxy, rate-limit bypass, and direct-vs-legacy branching.
|
||||
- MCP plan covers backend version gating, direct-first behavior, and `404`-based fallback.
|
||||
- Self-host coverage includes `.env.example`, root compose, packaged compose, and docs.
|
||||
- Validation covers backend, shared schemas, dashboard, MCP, compose config parsing, and both legacy + direct smoke tests.
|
||||
|
||||
### Placeholder scan
|
||||
|
||||
- No `TODO`, `TBD`, or "implement later" placeholders remain.
|
||||
- Every code-changing step includes a concrete code block.
|
||||
- Every verification step includes a concrete command or explicit manual smoke-test checklist.
|
||||
|
||||
### Type consistency
|
||||
|
||||
- `CreateDirectDeploymentRequest`, `CreateDirectDeploymentResponse`, `DeploymentManifestFile`, and `StartDeploymentRequest` are used consistently between shared schemas, backend, dashboard service client, and MCP.
|
||||
- Backend routes and MCP use the same endpoint names: `POST /api/deployments/direct`, `PUT /api/deployments/:id/files/:fileId/content`, and `POST /api/deployments/:id/start`.
|
||||
- The plan keeps the legacy entrypoint as `POST /api/deployments`.
|
||||
@@ -0,0 +1,669 @@
|
||||
# Custom Database Migrations Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add developer-facing database migrations that execute immediately against the `public` schema and record only successful runs in `system.custom_migrations`.
|
||||
|
||||
**Architecture:** The backend owns the migration workflow through a dedicated service and admin routes under `/api/database/migrations`. The service parses and validates SQL, runs it inside a single transaction with a transaction-scoped advisory lock, and inserts a history row into `system.custom_migrations` only after the SQL succeeds. Shared schemas define the request/response contract, and the dashboard consumes that contract through a `service -> hook -> UI` flow with a new read-only Database Studio page.
|
||||
|
||||
**Tech Stack:** PostgreSQL 15, Express, TypeScript, Zod, React, React Query, CodeMirror, Vitest
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
| Path | Responsibility |
|
||||
| --- | --- |
|
||||
| `backend/src/infra/database/migrations/032_create-custom-migrations.sql` | Create `system.custom_migrations` with the minimal applied-history shape |
|
||||
| `backend/tests/unit/custom-migrations-table-migration.test.ts` | Validate SQL migration structure and idempotency guards |
|
||||
| `backend/src/services/database/database-migration.service.ts` | List migrations and run new migrations transactionally |
|
||||
| `backend/tests/unit/database-migration.service.test.ts` | Service-level validation and transactional behavior |
|
||||
| `backend/src/api/routes/database/migrations.routes.ts` | Admin routes for list/create |
|
||||
| `backend/src/api/routes/database/index.routes.ts` | Mount new migrations router |
|
||||
| `backend/src/utils/sql-parser.ts` | Reuse or extend helpers for schema restrictions and socket invalidation |
|
||||
| `packages/shared-schemas/src/database.schema.ts` | `migrationSchema` domain shape |
|
||||
| `packages/shared-schemas/src/database-api.schema.ts` | Request/response schemas for migrations API |
|
||||
| `packages/shared-schemas/src/index.ts` | Export shared migration contracts |
|
||||
| `packages/dashboard/src/features/database/services/migration.service.ts` | Dashboard API client for migrations |
|
||||
| `packages/dashboard/src/features/database/hooks/useMigrations.ts` | React Query hooks for list/create |
|
||||
| `packages/dashboard/src/features/database/components/DatabaseSidebar.tsx` | Add `Migrations` to the Database Studio sidebar |
|
||||
| `packages/dashboard/src/features/database/components/MigrationFormDialog.tsx` | Run-migration dialog with SQL editor |
|
||||
| `packages/dashboard/src/features/database/pages/MigrationsPage.tsx` | Read-only migration history page |
|
||||
| `packages/dashboard/src/router/AppRoutes.tsx` | Register `/dashboard/database/migrations` |
|
||||
|
||||
### Task 1: Create the Applied-Migrations Table
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/infra/database/migrations/032_create-custom-migrations.sql`
|
||||
- Test: `backend/tests/unit/custom-migrations-table-migration.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing migration SQL test**
|
||||
|
||||
```ts
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
|
||||
const currentDir = path.dirname(fileURLToPath(import.meta.url));
|
||||
const migrationPath = path.resolve(
|
||||
currentDir,
|
||||
'../../src/infra/database/migrations/032_create-custom-migrations.sql'
|
||||
);
|
||||
|
||||
describe('032_create-custom-migrations migration', () => {
|
||||
const sql = fs.readFileSync(migrationPath, 'utf8');
|
||||
|
||||
it('creates system.custom_migrations with the expected columns', () => {
|
||||
expect(sql).toMatch(/CREATE TABLE IF NOT EXISTS system\.custom_migrations/i);
|
||||
expect(sql).toMatch(/sequence_number INTEGER PRIMARY KEY/i);
|
||||
expect(sql).toMatch(/name TEXT NOT NULL UNIQUE/i);
|
||||
expect(sql).toMatch(/statements TEXT\[\] NOT NULL/i);
|
||||
expect(sql).toMatch(/created_at TIMESTAMPTZ NOT NULL DEFAULT now\(\)/i);
|
||||
});
|
||||
|
||||
it('does not expose a surrogate id column', () => {
|
||||
expect(sql).not.toMatch(/\bid\b/i);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the migration SQL test to verify it fails**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- custom-migrations-table-migration.test.ts`
|
||||
Expected: FAIL with a missing file error for `032_create-custom-migrations.sql`
|
||||
|
||||
- [ ] **Step 3: Write the migration SQL file**
|
||||
|
||||
```sql
|
||||
CREATE SCHEMA IF NOT EXISTS system;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS system.custom_migrations (
|
||||
sequence_number INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
statements TEXT[] NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add idempotent guards for re-runs**
|
||||
|
||||
```sql
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'system'
|
||||
AND table_name = 'custom_migrations'
|
||||
) THEN
|
||||
CREATE TABLE system.custom_migrations (
|
||||
sequence_number INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
statements TEXT[] NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
END IF;
|
||||
END $$;
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Re-run the migration SQL test**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- custom-migrations-table-migration.test.ts`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
backend/src/infra/database/migrations/032_create-custom-migrations.sql \
|
||||
backend/tests/unit/custom-migrations-table-migration.test.ts
|
||||
git commit -m "feat: add custom migrations history table"
|
||||
```
|
||||
|
||||
### Task 2: Implement Backend Migration Service
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/services/database/database-migration.service.ts`
|
||||
- Test: `backend/tests/unit/database-migration.service.test.ts`
|
||||
- Modify: `backend/src/utils/sql-parser.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing service tests**
|
||||
|
||||
```ts
|
||||
describe('DatabaseMigrationService.createMigration', () => {
|
||||
it('rejects statements that reference a non-public schema', async () => {
|
||||
await expect(
|
||||
service.createMigration({
|
||||
name: 'break-auth',
|
||||
sql: 'CREATE TABLE auth.test_users (id uuid);',
|
||||
actor: 'local:admin',
|
||||
})
|
||||
).rejects.toThrow(/public schema/i);
|
||||
});
|
||||
|
||||
it('inserts history only after SQL succeeds', async () => {
|
||||
mockClient.query
|
||||
.mockResolvedValueOnce(undefined) // BEGIN
|
||||
.mockResolvedValueOnce({ rows: [{ next_sequence_number: 1 }] })
|
||||
.mockResolvedValueOnce(undefined) // statement
|
||||
.mockResolvedValueOnce(undefined) // insert history
|
||||
.mockResolvedValueOnce(undefined); // COMMIT
|
||||
|
||||
await service.createMigration({
|
||||
name: 'create-posts',
|
||||
sql: 'CREATE TABLE posts (id uuid primary key);',
|
||||
actor: 'local:admin',
|
||||
});
|
||||
|
||||
expect(mockClient.query).toHaveBeenCalledWith(
|
||||
expect.stringMatching(/INSERT INTO system\.custom_migrations/i),
|
||||
expect.any(Array)
|
||||
);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the service tests to verify they fail**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration.service.test.ts`
|
||||
Expected: FAIL because `database-migration.service.ts` does not exist yet
|
||||
|
||||
- [ ] **Step 3: Create the service skeleton**
|
||||
|
||||
```ts
|
||||
export class DatabaseMigrationService {
|
||||
private static instance: DatabaseMigrationService;
|
||||
private dbManager = DatabaseManager.getInstance();
|
||||
|
||||
public static getInstance(): DatabaseMigrationService {
|
||||
if (!DatabaseMigrationService.instance) {
|
||||
DatabaseMigrationService.instance = new DatabaseMigrationService();
|
||||
}
|
||||
return DatabaseMigrationService.instance;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Implement list and create methods**
|
||||
|
||||
```ts
|
||||
async listMigrations(): Promise<DatabaseMigrationsResponse> {
|
||||
const result = await this.dbManager.getPool().query(`
|
||||
SELECT
|
||||
sequence_number AS "sequenceNumber",
|
||||
name,
|
||||
statements,
|
||||
created_at AS "createdAt"
|
||||
FROM system.custom_migrations
|
||||
ORDER BY sequence_number DESC
|
||||
`);
|
||||
|
||||
return { migrations: result.rows };
|
||||
}
|
||||
|
||||
async createMigration(input: CreateMigrationRequest & { actor: string }): Promise<CreateMigrationResponse> {
|
||||
const statements = parseSQLStatements(input.sql);
|
||||
this.assertPublicSchemaOnly(statements);
|
||||
return this.runMigrationTransaction(input.name, statements);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Enforce the public-schema-only rules**
|
||||
|
||||
```ts
|
||||
private assertPublicSchemaOnly(statements: string[]): void {
|
||||
for (const statement of statements) {
|
||||
const parsed = statement.toLowerCase();
|
||||
if (/\b(auth|system|storage|ai|functions|realtime|schedules|pg_catalog|information_schema)\b/.test(parsed)) {
|
||||
throw new AppError(
|
||||
'Custom migrations may only target the public schema.',
|
||||
400,
|
||||
ERROR_CODES.DATABASE_FORBIDDEN
|
||||
);
|
||||
}
|
||||
|
||||
if (/\b(set\s+search_path|begin\b|commit\b|rollback\b|create\s+schema|drop\s+schema)\b/.test(parsed)) {
|
||||
throw new AppError(
|
||||
'Custom migrations cannot modify schema routing or manage their own transactions.',
|
||||
400,
|
||||
ERROR_CODES.DATABASE_FORBIDDEN
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Execute the migration transactionally and insert history on success**
|
||||
|
||||
```ts
|
||||
await client.query('BEGIN');
|
||||
await client.query("SELECT pg_advisory_xact_lock(hashtext('system.custom_migrations'))");
|
||||
await client.query('SET LOCAL search_path TO public');
|
||||
|
||||
const sequenceResult = await client.query(`
|
||||
SELECT COALESCE(MAX(sequence_number), 0) + 1 AS next_sequence_number
|
||||
FROM system.custom_migrations
|
||||
`);
|
||||
|
||||
for (const statement of statements) {
|
||||
await client.query(statement);
|
||||
}
|
||||
|
||||
await client.query(
|
||||
`
|
||||
INSERT INTO system.custom_migrations (sequence_number, name, statements)
|
||||
VALUES ($1, $2, $3)
|
||||
`,
|
||||
[sequenceNumber, name, statements]
|
||||
);
|
||||
|
||||
await client.query(`NOTIFY pgrst, 'reload schema';`);
|
||||
await client.query('COMMIT');
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Extend the SQL change typing so migration creates invalidate the new page**
|
||||
|
||||
```ts
|
||||
export interface DatabaseResourceUpdate {
|
||||
type: 'tables' | 'table' | 'records' | 'index' | 'trigger' | 'policy' | 'function' | 'extension' | 'migration';
|
||||
name?: string;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Re-run the service tests**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration.service.test.ts`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 9: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
backend/src/services/database/database-migration.service.ts \
|
||||
backend/src/utils/sql-parser.ts \
|
||||
backend/tests/unit/database-migration.service.test.ts
|
||||
git commit -m "feat: add custom migration backend service"
|
||||
```
|
||||
|
||||
### Task 3: Expose Admin Routes for Migrations
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/src/api/routes/database/migrations.routes.ts`
|
||||
- Modify: `backend/src/api/routes/database/index.routes.ts`
|
||||
|
||||
- [ ] **Step 1: Write the failing route-level behavior test or request fixture**
|
||||
|
||||
```ts
|
||||
it('mounts GET /api/database/migrations', async () => {
|
||||
const response = await request(app)
|
||||
.get('/api/database/migrations')
|
||||
.set('Authorization', `Bearer ${adminToken}`);
|
||||
|
||||
expect(response.status).toBe(200);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Create the migrations router**
|
||||
|
||||
```ts
|
||||
const router = Router();
|
||||
const migrationService = DatabaseMigrationService.getInstance();
|
||||
|
||||
router.get('/', verifyAdmin, async (_req, res, next) => {
|
||||
try {
|
||||
const response = await migrationService.listMigrations();
|
||||
successResponse(res, response);
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add the create-and-run endpoint**
|
||||
|
||||
```ts
|
||||
router.post('/', verifyAdmin, async (req: AuthRequest, res, next) => {
|
||||
try {
|
||||
const validation = createMigrationRequestSchema.safeParse(req.body);
|
||||
if (!validation.success) {
|
||||
throw new AppError('Invalid migration payload', 400, ERROR_CODES.INVALID_INPUT);
|
||||
}
|
||||
|
||||
const result = await migrationService.createMigration({
|
||||
...validation.data,
|
||||
actor: req.user?.email || 'api-key',
|
||||
});
|
||||
|
||||
await auditService.log({
|
||||
actor: req.user?.email || 'api-key',
|
||||
action: 'CREATE_CUSTOM_MIGRATION',
|
||||
module: 'DATABASE',
|
||||
details: { name: validation.data.name, statementCount: result.statements.length },
|
||||
ip_address: req.ip,
|
||||
});
|
||||
|
||||
successResponse(res, result, 201);
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Mount the router in the database route index**
|
||||
|
||||
```ts
|
||||
import { databaseMigrationsRouter } from './migrations.routes.js';
|
||||
|
||||
router.use('/migrations', databaseMigrationsRouter);
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the focused backend tests**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
backend/src/api/routes/database/migrations.routes.ts \
|
||||
backend/src/api/routes/database/index.routes.ts
|
||||
git commit -m "feat: expose custom migration admin routes"
|
||||
```
|
||||
|
||||
### Task 4: Add Shared Migration Contracts
|
||||
|
||||
**Files:**
|
||||
- Modify: `packages/shared-schemas/src/database.schema.ts`
|
||||
- Modify: `packages/shared-schemas/src/database-api.schema.ts`
|
||||
- Modify: `packages/shared-schemas/src/index.ts`
|
||||
|
||||
- [ ] **Step 1: Add the migration domain schema**
|
||||
|
||||
```ts
|
||||
export const migrationSchema = z.object({
|
||||
sequenceNumber: z.number().int().positive(),
|
||||
name: z.string().min(1),
|
||||
statements: z.array(z.string()).min(1),
|
||||
createdAt: z.string(),
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add request/response schemas**
|
||||
|
||||
```ts
|
||||
export const createMigrationRequestSchema = z.object({
|
||||
name: z.string().min(1, 'Migration name is required'),
|
||||
sql: z.string().min(1, 'Migration SQL is required'),
|
||||
});
|
||||
|
||||
export const createMigrationResponseSchema = migrationSchema.extend({
|
||||
message: z.string(),
|
||||
});
|
||||
|
||||
export const databaseMigrationsResponseSchema = z.object({
|
||||
migrations: z.array(migrationSchema),
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Export the new contracts**
|
||||
|
||||
```ts
|
||||
export type Migration = z.infer<typeof migrationSchema>;
|
||||
export type CreateMigrationRequest = z.infer<typeof createMigrationRequestSchema>;
|
||||
export type CreateMigrationResponse = z.infer<typeof createMigrationResponseSchema>;
|
||||
export type DatabaseMigrationsResponse = z.infer<typeof databaseMigrationsResponseSchema>;
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Build shared schemas**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas && npm run build`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
packages/shared-schemas/src/database.schema.ts \
|
||||
packages/shared-schemas/src/database-api.schema.ts \
|
||||
packages/shared-schemas/src/index.ts
|
||||
git commit -m "feat: add shared migration api contracts"
|
||||
```
|
||||
|
||||
### Task 5: Add Dashboard Service, Hook, Route, and Sidebar Entry
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/dashboard/src/features/database/services/migration.service.ts`
|
||||
- Create: `packages/dashboard/src/features/database/hooks/useMigrations.ts`
|
||||
- Modify: `packages/dashboard/src/features/database/components/DatabaseSidebar.tsx`
|
||||
- Modify: `packages/dashboard/src/router/AppRoutes.tsx`
|
||||
- Modify: `packages/dashboard/src/lib/contexts/SocketContext.tsx`
|
||||
|
||||
- [ ] **Step 1: Create the dashboard migration service**
|
||||
|
||||
```ts
|
||||
export class MigrationService {
|
||||
async listMigrations(): Promise<DatabaseMigrationsResponse> {
|
||||
return apiClient.request('/database/migrations', {
|
||||
method: 'GET',
|
||||
headers: apiClient.withAccessToken({}),
|
||||
});
|
||||
}
|
||||
|
||||
async createMigration(body: CreateMigrationRequest): Promise<CreateMigrationResponse> {
|
||||
return apiClient.request('/database/migrations', {
|
||||
method: 'POST',
|
||||
headers: apiClient.withAccessToken({ 'Content-Type': 'application/json' }),
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the React Query hook**
|
||||
|
||||
```ts
|
||||
export function useMigrations(enabled = false) {
|
||||
const queryClient = useQueryClient();
|
||||
const { showToast } = useToast();
|
||||
|
||||
const query = useQuery({
|
||||
queryKey: ['database', 'migrations'],
|
||||
queryFn: () => migrationService.listMigrations(),
|
||||
enabled,
|
||||
});
|
||||
|
||||
const createMutation = useMutation({
|
||||
mutationFn: (body: CreateMigrationRequest) => migrationService.createMigration(body),
|
||||
onSuccess: async () => {
|
||||
await queryClient.invalidateQueries({ queryKey: ['database', 'migrations'] });
|
||||
await queryClient.invalidateQueries({ queryKey: ['tables'] });
|
||||
showToast('Migration executed successfully', 'success');
|
||||
},
|
||||
});
|
||||
|
||||
return { ...query, createMigration: createMutation.mutateAsync, isCreating: createMutation.isPending };
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Register the route and sidebar item**
|
||||
|
||||
```tsx
|
||||
{
|
||||
id: 'migrations',
|
||||
label: 'Migrations',
|
||||
href: '/dashboard/database/migrations',
|
||||
},
|
||||
```
|
||||
|
||||
```tsx
|
||||
<Route path="migrations" element={<MigrationsPage />} />
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Invalidate the migrations query on socket data updates**
|
||||
|
||||
```ts
|
||||
case 'migration':
|
||||
void queryClient.invalidateQueries({ queryKey: ['database', 'migrations'] });
|
||||
break;
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run dashboard typecheck**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run typecheck`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
packages/dashboard/src/features/database/services/migration.service.ts \
|
||||
packages/dashboard/src/features/database/hooks/useMigrations.ts \
|
||||
packages/dashboard/src/features/database/components/DatabaseSidebar.tsx \
|
||||
packages/dashboard/src/router/AppRoutes.tsx \
|
||||
packages/dashboard/src/lib/contexts/SocketContext.tsx
|
||||
git commit -m "feat: wire dashboard migration data flow"
|
||||
```
|
||||
|
||||
### Task 6: Build the Migrations Page and Run Dialog
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/dashboard/src/features/database/pages/MigrationsPage.tsx`
|
||||
- Create: `packages/dashboard/src/features/database/components/MigrationFormDialog.tsx`
|
||||
- Reuse: `packages/dashboard/src/features/database/components/SQLModal.tsx`
|
||||
- Reuse: `packages/dashboard/src/components/CodeEditor.tsx`
|
||||
|
||||
- [ ] **Step 1: Create the migration run dialog**
|
||||
|
||||
```tsx
|
||||
export function MigrationFormDialog({ open, onOpenChange, onSubmit, isSubmitting }: Props) {
|
||||
const [name, setName] = useState('');
|
||||
const [sql, setSql] = useState('');
|
||||
|
||||
return (
|
||||
<Dialog open={open} onOpenChange={onOpenChange}>
|
||||
<DialogContent className="max-w-3xl">
|
||||
<DialogHeader>
|
||||
<DialogTitle>Run Migration</DialogTitle>
|
||||
<DialogDescription>
|
||||
This runs immediately against the public schema and is recorded only if it succeeds.
|
||||
</DialogDescription>
|
||||
</DialogHeader>
|
||||
<Input value={name} onChange={(e) => setName(e.target.value)} placeholder="add_posts_table" />
|
||||
<div className="h-80 rounded-lg border border-border">
|
||||
<CodeEditor value={sql} onChange={setSql} editable language="sql" />
|
||||
</div>
|
||||
<Button onClick={() => void onSubmit({ name, sql })} disabled={isSubmitting}>
|
||||
Run Migration
|
||||
</Button>
|
||||
</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Create the history page**
|
||||
|
||||
```tsx
|
||||
export default function MigrationsPage() {
|
||||
const navigate = useNavigate();
|
||||
const { data, isLoading, error, refetch, createMigration, isCreating } = useMigrations(true);
|
||||
const [open, setOpen] = useState(false);
|
||||
const [sqlModal, setSqlModal] = useState({ open: false, title: '', value: '' });
|
||||
|
||||
return (
|
||||
<div className="flex h-full min-h-0 overflow-hidden bg-[rgb(var(--semantic-1))]">
|
||||
<DatabaseStudioSidebarPanel onBack={() => void navigate('/dashboard/database/tables', { state: { slideFromStudio: true } })} />
|
||||
<div className="min-w-0 flex-1 flex flex-col overflow-hidden bg-[rgb(var(--semantic-1))]">
|
||||
<TableHeader title="Database Migrations" showDividerAfterTitle />
|
||||
<DataGrid data={rows} columns={columns} showSelection={false} showPagination={false} noPadding className="h-full" />
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Make the SQL column read-only through the existing SQL modal**
|
||||
|
||||
```tsx
|
||||
renderCell: ({ row }) => (
|
||||
<SQLCellButton
|
||||
value={row.statements.join('\n\n')}
|
||||
onClick={() =>
|
||||
setSqlModal({
|
||||
open: true,
|
||||
title: `Migration ${row.sequenceNumber}: ${row.name}`,
|
||||
value: row.statements.join('\n\n'),
|
||||
})
|
||||
}
|
||||
/>
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Handle loading, empty, and error states using the existing database page patterns**
|
||||
|
||||
```tsx
|
||||
if (error) {
|
||||
return <EmptyState title="Failed to load migrations" description={error.message} />;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Build the dashboard package**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run build`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add \
|
||||
packages/dashboard/src/features/database/pages/MigrationsPage.tsx \
|
||||
packages/dashboard/src/features/database/components/MigrationFormDialog.tsx
|
||||
git commit -m "feat: add database migrations studio page"
|
||||
```
|
||||
|
||||
### Task 7: Validate the Full Feature
|
||||
|
||||
**Files:**
|
||||
- Validate: `backend/**`
|
||||
- Validate: `packages/shared-schemas/**`
|
||||
- Validate: `packages/dashboard/**`
|
||||
|
||||
- [ ] **Step 1: Run backend tests**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 2: Run backend build**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm run build`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Run shared schema build**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas && npm run build`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 4: Run dashboard typecheck and build**
|
||||
|
||||
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run typecheck && npm run build`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Smoke-test the feature manually**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:7130/api/database/migrations \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer <admin-token>" \
|
||||
-d '{"name":"create_posts","sql":"CREATE TABLE posts (id UUID PRIMARY KEY DEFAULT gen_random_uuid());"}'
|
||||
```
|
||||
|
||||
Expected: `201` with the new migration row, and `GET /api/database/migrations` returns it.
|
||||
|
||||
- [ ] **Step 6: Commit final validation fixes**
|
||||
|
||||
```bash
|
||||
git add backend packages/shared-schemas packages/dashboard
|
||||
git commit -m "feat: add custom database migrations"
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,410 @@
|
||||
# Backend Branching (OSS) Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** Add read-only fallback from a branch project's storage to its parent project's S3 directory, gated by a new `PARENT_APP_KEY` env var. Read-only, transparent, no HTTP API changes.
|
||||
|
||||
**Architecture:** Extend `S3StorageProvider` with an optional `parentAppKey`. Read methods (`getObject`, `headObject`, `getObjectStream`, `getDownloadStrategy`) attempt the branch's S3 path first; on 404, retry the parent's path. Writes always target the branch path. Service layer and HTTP routes unchanged.
|
||||
|
||||
**Tech Stack:** TypeScript, Node.js, Express, AWS SDK v3 (`@aws-sdk/client-s3`), Vitest.
|
||||
|
||||
**Spec:** [docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md](../specs/2026-04-29-backend-branching-oss-design.md)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
**Modify:**
|
||||
- `backend/src/providers/storage/s3.provider.ts` — accept `parentAppKey`, add fallback in read methods
|
||||
- `backend/src/services/storage/storage.service.ts` — pass `PARENT_APP_KEY` env var into provider constructor
|
||||
- `backend/src/providers/storage/base.provider.ts` (interface, if any) — no new methods, just a fallback-aware contract
|
||||
|
||||
**Create:**
|
||||
- `backend/tests/unit/storage-s3-fallback.test.ts` — fallback behavior unit tests
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Extend S3StorageProvider Constructor
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/providers/storage/s3.provider.ts:49-56`
|
||||
|
||||
- [ ] **Step 1: Accept `parentAppKey` in constructor**
|
||||
|
||||
```typescript
|
||||
export class S3StorageProvider implements StorageProvider {
|
||||
private s3Client: S3Client | null = null;
|
||||
|
||||
constructor(
|
||||
private s3Bucket: string,
|
||||
private appKey: string,
|
||||
private region: string = 'us-east-2',
|
||||
private parentAppKey?: string,
|
||||
) {
|
||||
// ... existing init ...
|
||||
}
|
||||
|
||||
private getS3Key(bucket: string, key: string): string {
|
||||
return `${this.appKey}/${bucket}/${key}`;
|
||||
}
|
||||
|
||||
private getParentS3Key(bucket: string, key: string): string | null {
|
||||
return this.parentAppKey ? `${this.parentAppKey}/${bucket}/${key}` : null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run typecheck**
|
||||
|
||||
Run: `cd backend && npx tsc --noEmit`
|
||||
Expected: no errors.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/providers/storage/s3.provider.ts
|
||||
git commit -m "feat(branching): S3StorageProvider accepts optional parentAppKey"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Fallback in Read Methods (TDD)
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/tests/unit/storage-s3-fallback.test.ts`
|
||||
- Modify: `backend/src/providers/storage/s3.provider.ts` (`getObject`, `headObject`, `getObjectStream`)
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
```typescript
|
||||
// backend/tests/unit/storage-s3-fallback.test.ts
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
import { S3StorageProvider } from '@/providers/storage/s3.provider.js';
|
||||
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
|
||||
|
||||
describe('S3StorageProvider fallback to parent', () => {
|
||||
let sendMock: ReturnType<typeof vi.fn>;
|
||||
beforeEach(() => {
|
||||
sendMock = vi.fn();
|
||||
vi.spyOn(S3Client.prototype, 'send').mockImplementation(sendMock as any);
|
||||
});
|
||||
|
||||
it('returns branch object when present (no fallback call)', async () => {
|
||||
sendMock.mockResolvedValueOnce({ Body: streamOf('hello') });
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
|
||||
(p as any).s3Client = new S3Client({}); // bypass async init for unit
|
||||
const out = await p.getObject('foo', 'a.txt');
|
||||
expect(out?.toString()).toBe('hello');
|
||||
expect(sendMock).toHaveBeenCalledTimes(1);
|
||||
const cmd: any = sendMock.mock.calls[0][0];
|
||||
expect(cmd.input.Key).toBe('branchkey/foo/a.txt');
|
||||
});
|
||||
|
||||
it('falls back to parent when branch returns NoSuchKey', async () => {
|
||||
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
|
||||
sendMock.mockRejectedValueOnce(noSuchKey).mockResolvedValueOnce({ Body: streamOf('parent-data') });
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
|
||||
(p as any).s3Client = new S3Client({});
|
||||
const out = await p.getObject('foo', 'a.txt');
|
||||
expect(out?.toString()).toBe('parent-data');
|
||||
const k1 = (sendMock.mock.calls[0][0] as any).input.Key;
|
||||
const k2 = (sendMock.mock.calls[1][0] as any).input.Key;
|
||||
expect(k1).toBe('branchkey/foo/a.txt');
|
||||
expect(k2).toBe('parentkey/foo/a.txt');
|
||||
});
|
||||
|
||||
it('returns null when both branch and parent are missing', async () => {
|
||||
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
|
||||
sendMock.mockRejectedValueOnce(noSuchKey).mockRejectedValueOnce(noSuchKey);
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
|
||||
(p as any).s3Client = new S3Client({});
|
||||
const out = await p.getObject('foo', 'a.txt');
|
||||
expect(out).toBeNull();
|
||||
});
|
||||
|
||||
it('does NOT fall back when parentAppKey is unset', async () => {
|
||||
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
|
||||
sendMock.mockRejectedValueOnce(noSuchKey);
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2'); // no parentAppKey
|
||||
(p as any).s3Client = new S3Client({});
|
||||
const out = await p.getObject('foo', 'a.txt');
|
||||
expect(out).toBeNull();
|
||||
expect(sendMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
function streamOf(s: string) {
|
||||
const { Readable } = require('node:stream');
|
||||
return Readable.from([Buffer.from(s)]);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
|
||||
Expected: FAIL — fallback not yet implemented.
|
||||
|
||||
- [ ] **Step 3: Implement fallback in `getObject`**
|
||||
|
||||
```typescript
|
||||
async getObject(bucket: string, key: string): Promise<Buffer | null> {
|
||||
if (!this.s3Client) throw new Error('S3 client not initialized');
|
||||
const primary = await this.tryGet(this.getS3Key(bucket, key));
|
||||
if (primary !== null) return primary;
|
||||
const parentKey = this.getParentS3Key(bucket, key);
|
||||
if (!parentKey) return null;
|
||||
return this.tryGet(parentKey);
|
||||
}
|
||||
|
||||
private async tryGet(s3Key: string): Promise<Buffer | null> {
|
||||
try {
|
||||
const cmd = new GetObjectCommand({ Bucket: this.s3Bucket, Key: s3Key });
|
||||
const res = await this.s3Client!.send(cmd);
|
||||
if (!res.Body) return null;
|
||||
const chunks: Buffer[] = [];
|
||||
// @ts-ignore SDK v3 stream
|
||||
for await (const c of res.Body) chunks.push(Buffer.isBuffer(c) ? c : Buffer.from(c));
|
||||
return Buffer.concat(chunks);
|
||||
} catch (err: any) {
|
||||
if (err?.name === 'NoSuchKey' || err?.$metadata?.httpStatusCode === 404) return null;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Implement fallback in `headObject` and `getObjectStream`**
|
||||
|
||||
Apply the same primary-then-parent pattern:
|
||||
|
||||
```typescript
|
||||
async headObject(bucket: string, key: string): Promise<{ size: number; mimeType: string } | null> {
|
||||
const tryHead = async (s3Key: string) => {
|
||||
try {
|
||||
const res = await this.s3Client!.send(new HeadObjectCommand({ Bucket: this.s3Bucket, Key: s3Key }));
|
||||
return { size: res.ContentLength ?? 0, mimeType: res.ContentType ?? 'application/octet-stream' };
|
||||
} catch (err: any) {
|
||||
if (err?.name === 'NotFound' || err?.$metadata?.httpStatusCode === 404) return null;
|
||||
throw err;
|
||||
}
|
||||
};
|
||||
const primary = await tryHead(this.getS3Key(bucket, key));
|
||||
if (primary) return primary;
|
||||
const parentKey = this.getParentS3Key(bucket, key);
|
||||
return parentKey ? tryHead(parentKey) : null;
|
||||
}
|
||||
```
|
||||
|
||||
Mirror the pattern in `getObjectStream` if present.
|
||||
|
||||
- [ ] **Step 5: Run tests**
|
||||
|
||||
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
|
||||
Expected: all four cases pass.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/providers/storage/s3.provider.ts backend/tests/unit/storage-s3-fallback.test.ts
|
||||
git commit -m "feat(branching): read-only S3 fallback to parent appkey on 404"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Fallback in Presigned URL Generation (`getDownloadStrategy`)
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/providers/storage/s3.provider.ts` (`getDownloadStrategy` method)
|
||||
|
||||
- [ ] **Step 1: Write failing test**
|
||||
|
||||
Add to `backend/tests/unit/storage-s3-fallback.test.ts`:
|
||||
|
||||
```typescript
|
||||
it('presigned URL: signs branch key when present', async () => {
|
||||
// Mock HeadObject to succeed for branch path
|
||||
sendMock.mockResolvedValueOnce({ ContentLength: 5 });
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
|
||||
(p as unknown as { s3Client: S3Client }).s3Client = new S3Client({
|
||||
region: 'us-east-2',
|
||||
credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
|
||||
});
|
||||
const strategy = await p.getDownloadStrategy('foo', 'a.txt');
|
||||
expect(strategy.url).toContain('branchkey/foo/a.txt');
|
||||
});
|
||||
|
||||
it('presigned URL: signs parent key when branch HEAD returns 404', async () => {
|
||||
const notFound = Object.assign(new Error('NotFound'), { name: 'NotFound' });
|
||||
sendMock.mockRejectedValueOnce(notFound);
|
||||
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
|
||||
(p as unknown as { s3Client: S3Client }).s3Client = new S3Client({
|
||||
region: 'us-east-2',
|
||||
credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
|
||||
});
|
||||
const strategy = await p.getDownloadStrategy('foo', 'a.txt');
|
||||
expect(strategy.url).toContain('parentkey/foo/a.txt');
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
|
||||
Expected: FAIL.
|
||||
|
||||
- [ ] **Step 3: Implement fallback in `getDownloadStrategy`**
|
||||
|
||||
`getDownloadStrategy` already constructs a presigned URL via `getSignedUrl`. To support branching, do a HEAD on the branch path first; if missing (and a parent is configured) sign the parent path. HEAD failures other than 404 are caught and logged — URL generation defaults to the branch key rather than aborting.
|
||||
|
||||
```typescript
|
||||
const branchKey = this.getS3Key(bucket, key);
|
||||
const parentKey = this.getParentS3Key(bucket, key);
|
||||
let s3Key = branchKey;
|
||||
if (parentKey) {
|
||||
try {
|
||||
const branchExists = await this.tryHeadObject(branchKey);
|
||||
if (!branchExists) {
|
||||
s3Key = parentKey;
|
||||
}
|
||||
} catch (headErr) {
|
||||
// HEAD failures shouldn't break URL generation. Default to the branch
|
||||
// key; if the object truly only lives on the parent, the signed URL
|
||||
// will 404 at download time — degraded but recoverable.
|
||||
logger.warn('Branch HEAD check failed in getDownloadStrategy; signing branch key', {
|
||||
bucket, key,
|
||||
error: headErr instanceof Error ? headErr.message : String(headErr),
|
||||
});
|
||||
}
|
||||
}
|
||||
// ...existing CloudFront / getSignedUrl code uses s3Key.
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests**
|
||||
|
||||
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/providers/storage/s3.provider.ts backend/tests/unit/storage-s3-fallback.test.ts
|
||||
git commit -m "feat(branching): presigned URL fallback to parent path"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire `PARENT_APP_KEY` Env Var Through StorageService
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/services/storage/storage.service.ts:29-45`
|
||||
|
||||
- [ ] **Step 1: Pass `PARENT_APP_KEY` to S3StorageProvider**
|
||||
|
||||
```typescript
|
||||
private constructor() {
|
||||
const s3Bucket = process.env.AWS_S3_BUCKET;
|
||||
const appKey = process.env.APP_KEY || 'local';
|
||||
const parentAppKey = process.env.PARENT_APP_KEY; // <-- new
|
||||
|
||||
if (s3Bucket) {
|
||||
this.provider = new S3StorageProvider(
|
||||
s3Bucket,
|
||||
appKey,
|
||||
process.env.AWS_REGION || 'us-east-2',
|
||||
parentAppKey,
|
||||
);
|
||||
if (parentAppKey) {
|
||||
logger.info('Storage initialized in branch mode', { appKey, parentAppKey });
|
||||
}
|
||||
} else {
|
||||
// local storage — fallback unsupported in v1, ignore
|
||||
const baseDir = process.env.STORAGE_DIR || path.resolve(process.cwd(), 'insforge-storage');
|
||||
this.provider = new LocalStorageProvider(baseDir);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify boot log**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd backend
|
||||
APP_KEY=branchkey PARENT_APP_KEY=parentkey AWS_S3_BUCKET=test-bucket npm run dev 2>&1 | head -30
|
||||
```
|
||||
Expected: log line `Storage initialized in branch mode { appKey: 'branchkey', parentAppKey: 'parentkey' }`. Stop with Ctrl-C.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add backend/src/services/storage/storage.service.ts
|
||||
git commit -m "feat(branching): wire PARENT_APP_KEY through to S3 provider"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Manual Smoke Test
|
||||
|
||||
- [ ] **Step 1: Two-namespace smoke against real S3**
|
||||
|
||||
Pre-requisites: AWS creds + a test bucket with two prefixes:
|
||||
- `parentkey/photos/cat.jpg` (some real file)
|
||||
- `branchkey/photos/dog.jpg` (some other real file)
|
||||
|
||||
```bash
|
||||
# Branch mode boot
|
||||
APP_KEY=branchkey PARENT_APP_KEY=parentkey AWS_S3_BUCKET=insforge-storage-test \
|
||||
AWS_REGION=us-east-2 npm run dev:backend
|
||||
```
|
||||
|
||||
Then in another terminal (after seeding `storage.objects` rows so RLS allows access):
|
||||
|
||||
```bash
|
||||
# Should hit branch (own file)
|
||||
curl -i http://localhost:7130/api/storage/buckets/photos/objects/dog.jpg
|
||||
|
||||
# Should hit parent (fallback)
|
||||
curl -i http://localhost:7130/api/storage/buckets/photos/objects/cat.jpg
|
||||
|
||||
# Write goes to branch only
|
||||
curl -i -X PUT http://localhost:7130/api/storage/buckets/photos/objects/new.txt \
|
||||
-H "Authorization: Bearer $TOKEN" --data-binary "branch-write"
|
||||
|
||||
# Verify on S3 directly
|
||||
aws s3 ls s3://insforge-storage-test/branchkey/photos/ # contains new.txt
|
||||
aws s3 ls s3://insforge-storage-test/parentkey/photos/ # unchanged
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Open PR**
|
||||
|
||||
```bash
|
||||
git push -u origin feat/branching
|
||||
gh pr create --title "feat: storage fallback to parent for branch projects" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
- Adds optional `PARENT_APP_KEY` env var that triggers read-only fallback in S3 storage provider.
|
||||
- All read methods (getObject, headObject, getObjectStream, getDownloadStrategy) try branch path first, fall back to parent path on 404.
|
||||
- Writes are unchanged — they always target the branch's own appkey path.
|
||||
- Local storage provider is unaffected (no fallback).
|
||||
|
||||
## Spec
|
||||
- [docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md](docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md)
|
||||
|
||||
## Test plan
|
||||
- [ ] Vitest suite green
|
||||
- [ ] Manual: GET on branch object hits branch
|
||||
- [ ] Manual: GET on parent-only object falls back to parent
|
||||
- [ ] Manual: GET on missing object returns 404
|
||||
- [ ] Manual: PUT on branch creates only at branch path
|
||||
- [ ] Manual: existing non-branch projects (no PARENT_APP_KEY) are unaffected
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Notes
|
||||
|
||||
- **Fallback is read-only**: writes (PUT/DELETE/multipart) always go to branch path. Verified by not modifying any write methods.
|
||||
- **Local provider unchanged**: only S3 provider gets the fallback. Local installs aren't branched.
|
||||
- **No HTTP API changes**: routes and middleware unchanged. RLS / bucket-visibility checks remain authoritative.
|
||||
- **No new IAM permissions**: existing role can read all prefixes in `AWS_S3_BUCKET`.
|
||||
- **Schema-only mode**: branches with `mode='schema-only'` truncate `storage.objects` in the DB → RLS lookup returns 404 before reaching the provider → fallback never runs. Documented in spec as expected.
|
||||
@@ -0,0 +1,231 @@
|
||||
# Stripe Payments Current Implementation Spec
|
||||
|
||||
**Status:** Current source-of-truth implementation for the Stripe payments foundation.
|
||||
|
||||
**Audience:** InsForge engineers, dashboard maintainers, CLI/SDK authors, OpenAPI maintainers, and agents that need to understand how to build on the payment feature.
|
||||
|
||||
**Goal:** Let a developer configure their own Stripe test/live secret keys, let agents configure Stripe catalog objects, and let InsForge support complete runtime payment flows for one-time purchases, subscriptions, webhook projections, and customer billing portal sessions.
|
||||
|
||||
## Product Direction
|
||||
|
||||
InsForge uses the developer-owned Stripe account model for this phase. Developers configure `STRIPE_TEST_SECRET_KEY` and/or `STRIPE_LIVE_SECRET_KEY` through the dashboard, or seed them from environment variables into the secret store. InsForge does not use Connected Accounts, claimable sandboxes, or test-to-live publishing in this version.
|
||||
|
||||
Stripe is the source of truth. InsForge stores a local mirror and runtime projection so agents and the dashboard can reason about the state of payments without re-querying Stripe for every action. Mutations still go to Stripe first; InsForge only updates its mirror after Stripe succeeds.
|
||||
|
||||
The system is agent-first. The dashboard provides visibility and controls, but the backend API and shared schemas are the primary surface that agents, CLI commands, SDKs, and generated apps should build against.
|
||||
|
||||
## Current Capabilities
|
||||
|
||||
This implementation supports test and live Stripe environments as independent targets. Every payment table includes `environment = 'test' | 'live'` instead of using duplicated live/test table sets.
|
||||
|
||||
Developers and agents can manage Stripe products and prices in either environment. Product and price create calls use caller-stable idempotency keys when provided. Updates and archives are applied directly to Stripe, then mirrored locally. Product deletion hard-deletes the local mirror only after Stripe confirms the product was deleted.
|
||||
|
||||
Developers can run a unified sync. Sync pulls products, prices, customers, and subscriptions from every configured environment, skips unconfigured environments, and records the latest sync status on the environment connection row. Manual sync also checks the Stripe account id before writing, so switching a key to a different account clears stale mirrored payment data before importing the new account's data.
|
||||
|
||||
Generated apps can create Checkout Sessions at runtime. Anonymous one-time checkout is allowed. Identified one-time checkout is allowed and can reuse an existing Stripe customer mapping. Subscription checkout requires a billing subject because subscriptions represent ongoing entitlement.
|
||||
|
||||
Generated apps can create Stripe Billing Portal Sessions for authenticated users. This is intentionally mediated through `payments.customer_portal_sessions`, where developers or agents can define custom RLS policies for their app's subject model.
|
||||
|
||||
Managed Stripe webhooks are used to keep runtime projections current. InsForge stores webhook signing secrets in the secret store, not in environment variables. Webhook setup is best-effort during new account configuration and can also be retried from the dashboard Webhooks tab.
|
||||
|
||||
## Data Model
|
||||
|
||||
The payments schema is created by `backend/src/infra/database/migrations/038_create-payments-schema.sql`.
|
||||
|
||||
`payments.stripe_connections` stores one row per environment. It tracks Stripe account identity, key/configured status, managed webhook endpoint metadata, latest sync status, sync errors, and sync counts.
|
||||
|
||||
`payments.products` and `payments.prices` mirror Stripe catalog state. Sync overwrites local drift with Stripe data and removes local rows that no longer exist in Stripe for that environment.
|
||||
|
||||
`payments.checkout_sessions` stores short-lived checkout attempts. It starts as `initialized`, moves to `open` after Stripe session creation, and is updated to `completed`, `expired`, or `failed` through webhooks or backend errors. It intentionally does not enable RLS by default to reduce DX friction. Developers can ask agents to add RLS later if their app needs stricter checkout-attempt policies.
|
||||
|
||||
`payments.customer_portal_sessions` stores customer portal creation attempts. It enables RLS by default because portal creation must be gated by the application's subject ownership model.
|
||||
|
||||
`payments.stripe_customer_mappings` maps arbitrary app billing subjects to Stripe customers. The subject is intentionally generic: `subject_type` and `subject_id` can represent a user, team, organization, tenant, group, workspace, or another app-specific billing owner.
|
||||
|
||||
`payments.customers` mirrors Stripe customer rows for admin visibility and debugging. It is intentionally read-only from the app's perspective and does not replace `stripe_customer_mappings` as the operational subject-to-customer bridge.
|
||||
|
||||
`payments.subscriptions` and `payments.subscription_items` mirror current subscription state. Sync and webhooks fetch full subscription item lists when Stripe pagination requires it, then delete local subscription items not present in Stripe.
|
||||
|
||||
`payments.payment_history` records one-time payments, subscription invoices, failed payments, refunds, and refund state on original payments. It is webhook-driven and designed to tolerate out-of-order Stripe events.
|
||||
|
||||
`payments.webhook_events` records Stripe webhook processing state. It deduplicates events by `(environment, stripe_event_id)`, retries failed or pending events, and ignores already processed events.
|
||||
|
||||
## Backend Surface
|
||||
|
||||
Runtime routes use `verifyUser` so generated apps can call them with InsForge user tokens, including anon tokens for anonymous one-time checkout.
|
||||
|
||||
- `POST /api/payments/:environment/checkout-sessions`: creates a local checkout attempt, then creates a Stripe Checkout Session. Subscription mode requires `subject`.
|
||||
- `POST /api/payments/:environment/customer-portal-sessions`: creates a local portal attempt under the caller context, checks `stripe_customer_mappings`, then creates a Stripe Billing Portal Session. Anonymous users are rejected.
|
||||
- `POST /api/webhooks/stripe/:environment`: receives Stripe webhooks with a raw body and verifies the Stripe signature.
|
||||
|
||||
Admin routes use `verifyAdmin` and are intended for dashboard, agents, CLI, and SDK admin surfaces.
|
||||
|
||||
- `GET /api/payments/status`: returns environment connection, sync, and webhook status.
|
||||
- `GET /api/payments/config`: returns Stripe key availability and masked key info.
|
||||
- `PUT /api/payments/:environment/config`: stores a Stripe secret key in the secret store. New or different accounts trigger managed webhook setup and unified sync.
|
||||
- `DELETE /api/payments/:environment/config`: disables the secret key and best-effort deletes managed Stripe webhook endpoints for that environment.
|
||||
- `POST /api/payments/sync`: syncs products, prices, customers, and subscriptions for all configured environments.
|
||||
- `POST /api/payments/:environment/sync`: syncs products, prices, customers, and subscriptions for one environment.
|
||||
- `GET /api/payments/:environment/customers`: lists mirrored Stripe customers for one environment.
|
||||
- `POST /api/payments/:environment/webhook`: recreates the InsForge-managed Stripe webhook endpoint for the environment.
|
||||
- `GET /api/payments/:environment/catalog`: reads mirrored products and prices.
|
||||
- `GET|POST|PATCH|DELETE /api/payments/:environment/catalog/products...`: manages Stripe products.
|
||||
- `GET|POST|PATCH|DELETE /api/payments/:environment/catalog/prices...`: manages Stripe prices, where delete archives the Stripe price.
|
||||
- `GET /api/payments/:environment/subscriptions`: reads mirrored subscriptions for dashboard/admin use.
|
||||
- `GET /api/payments/:environment/payment-history`: reads payment history for dashboard/admin use.
|
||||
|
||||
## Key Management and Sync Semantics
|
||||
|
||||
The secret store is the canonical runtime source for Stripe keys. Environment variables are only seed inputs.
|
||||
|
||||
- Test key secret name: `STRIPE_TEST_SECRET_KEY`
|
||||
- Live key secret name: `STRIPE_LIVE_SECRET_KEY`
|
||||
- Test webhook secret store key: `STRIPE_TEST_WEBHOOK_SECRET`
|
||||
- Live webhook secret store key: `STRIPE_LIVE_WEBHOOK_SECRET`
|
||||
|
||||
When a Stripe key is saved, InsForge validates the key prefix, retrieves the Stripe account id, and compares it with the existing connection row.
|
||||
|
||||
If the exact key is already configured and the connection has an account id, the save is a no-op.
|
||||
|
||||
If the key is different but points to the same Stripe account, InsForge updates the secret and connection metadata but skips webhook recreation and sync.
|
||||
|
||||
If the key points to a different Stripe account, InsForge clears all mirrored payment data for that environment, best-effort recreates the managed webhook, persists the new key and account metadata, then runs unified sync.
|
||||
|
||||
If webhook creation fails because the backend URL is not publicly accessible, key configuration still succeeds and sync still runs. Developers can retry webhook setup later from the Webhooks tab or use Stripe CLI for local webhook testing.
|
||||
|
||||
Manual sync does not touch webhook setup. It only pulls Stripe data and updates local mirrors/projections.
|
||||
|
||||
## Checkout Flow
|
||||
|
||||
Checkout creation is a two-step local-plus-Stripe process.
|
||||
|
||||
First, InsForge inserts `payments.checkout_sessions` using the caller's Postgres role and JWT context. This lets future developer-defined policies work without changing the route. The current migration does not enable RLS by default.
|
||||
|
||||
Second, if the insert succeeds, InsForge creates the Stripe Checkout Session. If Stripe creation succeeds, the local row becomes `open` with Stripe ids and URL. If Stripe creation fails, the local row becomes `failed`.
|
||||
|
||||
Idempotency is handled at both layers. The local table has a unique partial index on `(environment, idempotency_key)`. If a caller retries the same request and the existing row has a usable Stripe URL, InsForge returns it. If the existing row is incomplete, InsForge retries Stripe creation using the same local row.
|
||||
|
||||
Caller-provided metadata cannot use keys that start with `insforge_`. InsForge owns these reserved keys because webhooks trust them to recover checkout mode, checkout session id, and billing subject.
|
||||
|
||||
When an identified one-time checkout has no existing customer mapping, InsForge asks Stripe Checkout to create a customer with `customer_creation = always`. When the checkout completes and Stripe returns a customer id, InsForge creates or updates `payments.stripe_customer_mappings`.
|
||||
|
||||
One-time checkout does not currently create a separate checkout item projection. The checkout attempt stores line items, while durable fulfillment state is represented by webhook-driven payment history and app-specific business tables.
|
||||
|
||||
## Customer Portal Flow
|
||||
|
||||
Customer portal sessions are authenticated-only. Anonymous users cannot create them.
|
||||
|
||||
The request includes a billing subject and optional return URL/configuration id. InsForge inserts `payments.customer_portal_sessions` using the caller's Postgres role and JWT context. Developers or agents can add app-specific RLS policies on this table to decide who may create portal sessions for which subject.
|
||||
|
||||
After the local insert succeeds, InsForge looks up the subject in `payments.stripe_customer_mappings`. If there is no mapped Stripe customer, the request fails with `404`. If there is a mapping, InsForge creates a Stripe Billing Portal Session and stores the returned portal URL.
|
||||
|
||||
## Webhook Projection Flow
|
||||
|
||||
Managed webhooks listen for the events needed to maintain checkout, subscription, payment history, and refund projections:
|
||||
|
||||
- `checkout.session.completed`
|
||||
- `checkout.session.async_payment_succeeded`
|
||||
- `checkout.session.async_payment_failed`
|
||||
- `checkout.session.expired`
|
||||
- `invoice.paid`
|
||||
- `invoice.payment_failed`
|
||||
- `payment_intent.succeeded`
|
||||
- `payment_intent.payment_failed`
|
||||
- `charge.refunded`
|
||||
- `refund.created`
|
||||
- `refund.updated`
|
||||
- `refund.failed`
|
||||
- `customer.subscription.created`
|
||||
- `customer.subscription.updated`
|
||||
- `customer.subscription.deleted`
|
||||
- `customer.subscription.paused`
|
||||
- `customer.subscription.resumed`
|
||||
|
||||
Webhook processing is idempotent. A duplicate processed or ignored event is returned as already handled. Failed events can be retried and will increment `attempt_count`.
|
||||
|
||||
Payment history supports out-of-order refund events. When refund context is missing locally, InsForge retrieves the PaymentIntent, Charge, and Invoice Payments context from Stripe, hydrates the original payment/invoice row where possible, and preserves previously known context with `COALESCE` rather than overwriting it with null fields.
|
||||
|
||||
Subscription projections support existing Stripe accounts. If a synced subscription has no InsForge billing subject mapping, it is still imported with nullable subject fields and counted as unmapped.
|
||||
|
||||
## Dashboard Surface
|
||||
|
||||
The Payments feature has a secondary menu with Products and Subscriptions.
|
||||
|
||||
Products shows test/live tabs, environment-specific empty states when a key is missing, product rows aligned with the Realtime Messages visual style, and product detail rows for associated prices.
|
||||
|
||||
Subscriptions shows test/live tabs and subscription detail rows for subscription items. Existing Stripe subscriptions can appear without InsForge subject mapping.
|
||||
|
||||
The Payments Settings dialog has three tabs:
|
||||
|
||||
- Stripe Keys: configure or remove test/live secret keys.
|
||||
- Sync: run unified sync across configured environments.
|
||||
- Webhooks: view managed webhook state and retry automatic webhook configuration.
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
`PaymentService` is the main orchestrator. It delegates focused work to smaller payment services and keeps cross-service orchestration in one place.
|
||||
|
||||
`PaymentConfigService` owns key storage, account status, managed webhook configuration, mirror clearing on account changes, and catalog snapshot writes.
|
||||
|
||||
`PaymentProductService` owns product list/get/create/update/delete and local product mirror writes.
|
||||
|
||||
`PaymentPriceService` owns price list/get/create/update/archive and local price mirror writes.
|
||||
|
||||
`PaymentCheckoutService` owns local checkout session insertion, retry lookup, and checkout row status updates.
|
||||
|
||||
`PaymentCustomerPortalService` owns local customer portal session insertion and status updates.
|
||||
|
||||
`PaymentHistoryService` owns payment history projection from checkout, invoice, payment intent, charge, and refund events.
|
||||
|
||||
`PaymentSubscriptionService` owns subscription and subscription item projection from sync and webhooks.
|
||||
|
||||
`PaymentWebhookService` owns webhook event deduplication and processing status records.
|
||||
|
||||
`StripeProvider` is the only wrapper around the official Stripe SDK. It owns Stripe API calls, pagination, webhook signature construction, and optional idempotency request options.
|
||||
|
||||
Helpers are pure utility functions only. They do not query Postgres or call Stripe.
|
||||
|
||||
## Explicit Non-Goals for This Phase
|
||||
|
||||
This phase does not implement Stripe Connected Accounts, Express/Custom account onboarding, claimable sandboxes, or platform-managed merchant accounts.
|
||||
|
||||
This phase does not implement test-to-live publishing or catalog diff application. Agents can target `test` or `live` explicitly in product and price APIs.
|
||||
|
||||
This phase does not expose runtime-safe read APIs for end-user subscription/payment state. Admin reads exist today. End-user reads are deferred because permission semantics depend on each app's subject model.
|
||||
|
||||
This phase does not mirror invoices, charges, payment methods, or checkout session line items beyond the customer projection added for admin visibility.
|
||||
|
||||
This phase does not define default app-specific RLS policies for payment history, subscriptions, customer mappings, or customer portal sessions. Agents should generate policies based on the developer's app schema.
|
||||
|
||||
## CLI, SDK, Docs, and OpenAPI Surfaces
|
||||
|
||||
CLI and SDK work should expose the runtime route pair first: create Checkout Session and create Customer Portal Session. These are the APIs generated apps need to collect money and let customers manage subscriptions.
|
||||
|
||||
Admin SDK and CLI work should then expose key configuration, status, unified sync, webhook configuration, catalog reads, product CRUD, price CRUD, subscription reads, and payment history reads.
|
||||
|
||||
OpenAPI documents the current `/api/payments` and `/api/webhooks/stripe/:environment` surfaces in `openapi/payments.yaml`, including environment targeting and the distinction between runtime routes and admin routes.
|
||||
|
||||
Agent docs should focus on workflows:
|
||||
|
||||
- Configure Stripe test/live keys.
|
||||
- Sync Stripe state.
|
||||
- Create products and one-time/recurring prices.
|
||||
- Build checkout flows with success and cancel URLs.
|
||||
- Build subscription checkout with a billing subject.
|
||||
- Add customer portal access with app-specific RLS on `payments.customer_portal_sessions`.
|
||||
- Use webhooks and payment projections to update app-specific entitlement tables.
|
||||
|
||||
Public docs should explain the developer-owned Stripe account model, local development webhook limitations, Stripe CLI testing, and the fact that Stripe remains the source of truth.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Use this checklist when changing the payment implementation:
|
||||
|
||||
- Backend unit tests cover config, sync, catalog CRUD, checkout, portal sessions, webhooks, refunds, subscriptions, and migration idempotency.
|
||||
- Backend lint, typecheck, and build pass.
|
||||
- Shared schema lint, typecheck, and build pass.
|
||||
- Dashboard lint, typecheck, and build pass when dashboard payment UI changes.
|
||||
- Payments migrations remain idempotent: every create/index/trigger/grant/alter operation in migrations 039 and 040 is safe to re-run.
|
||||
- Stripe webhook secrets are not documented or required as environment variables.
|
||||
- Product and price mutations call Stripe first and update the local mirror only after Stripe succeeds.
|
||||
- Sync treats Stripe as the source of truth and clears mirrors only when the Stripe account id changes.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Payments Customers Mirror Implementation Plan
|
||||
|
||||
> **For agentic workers:** Use the repo payment patterns as the source of truth. Keep `payments.customers` read-only and admin-facing. Do not make runtime checkout or portal flows depend on it.
|
||||
|
||||
**Goal:** Add a `payments.customers` table that mirrors Stripe customers for each Stripe environment and expose it in the admin payments dashboard as a display-only surface.
|
||||
|
||||
**Architecture:** Keep `payments.stripe_customer_mappings` as the operational subject-to-customer bridge. Add a separate customer mirror that is populated from Stripe sync plus customer webhooks, and expose it through an admin read route plus dashboard page.
|
||||
|
||||
**Files expected to change:**
|
||||
- Create: `backend/src/infra/database/migrations/040_create-payments-customers-table.sql`
|
||||
- Create: `backend/src/services/payments/payment-customer.service.ts`
|
||||
- Modify: `backend/src/types/payments.ts`
|
||||
- Modify: `backend/src/providers/payments/stripe.provider.ts`
|
||||
- Modify: `backend/src/services/payments/constants.ts`
|
||||
- Modify: `backend/src/services/payments/payment-config.service.ts`
|
||||
- Modify: `backend/src/services/payments/payment.service.ts`
|
||||
- Modify: `backend/src/api/routes/payments/index.routes.ts`
|
||||
- Modify: `packages/shared-schemas/src/payments.schema.ts`
|
||||
- Modify: `packages/shared-schemas/src/payments-api.schema.ts`
|
||||
- Modify: `openapi/payments.yaml`
|
||||
- Modify: `packages/dashboard/src/router/AppRoutes.tsx`
|
||||
- Modify: `packages/dashboard/src/features/payments/components/PaymentsSidebar.tsx`
|
||||
- Create: `packages/dashboard/src/features/payments/hooks/usePaymentCustomers.ts`
|
||||
- Modify: `packages/dashboard/src/features/payments/services/payments.service.ts`
|
||||
- Create: `packages/dashboard/src/features/payments/pages/CustomersPage.tsx`
|
||||
- Modify: `backend/tests/unit/payments-schema-migration.test.ts`
|
||||
- Modify: `backend/tests/unit/stripe-provider.test.ts`
|
||||
- Modify: `backend/tests/unit/payment.service.test.ts`
|
||||
|
||||
**Implementation order:**
|
||||
1. Add the database table and migration coverage.
|
||||
2. Add backend Stripe customer types/provider methods and mirror service.
|
||||
3. Hook customer sync and customer webhook projection into the payment orchestration layer.
|
||||
4. Expose admin read APIs and shared schemas.
|
||||
5. Add the dashboard Customers page.
|
||||
6. Update docs/OpenAPI and run verification.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,226 @@
|
||||
# Custom SMTP & Email Templates
|
||||
|
||||
## Overview
|
||||
|
||||
Allow project admins to configure their own SMTP server for email delivery instead of relying on InsForge's cloud backend. Includes customizable email templates with a Source/Preview editor.
|
||||
|
||||
## Motivation
|
||||
|
||||
Currently all emails (verification, password reset, raw) route through InsForge's cloud API. Self-hosted users and cloud users who prefer their own mail delivery have no alternative. This feature adds a pluggable SMTP provider and a template editor, following the same UX pattern as Supabase.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Provider Selection Flow
|
||||
|
||||
```
|
||||
EmailService.sendWithTemplate() / sendRaw()
|
||||
-> Check email.config (enabled?)
|
||||
-> YES: SmtpEmailProvider (nodemailer) + local template rendering
|
||||
-> NO: CloudEmailProvider (current behavior, unchanged)
|
||||
```
|
||||
|
||||
No breaking changes. The `EmailProvider` interface (`sendWithTemplate`, `sendRaw`, `supportsTemplates`) remains unchanged. Callers are unaffected.
|
||||
|
||||
## Database
|
||||
|
||||
### Table: `email.config` (singleton)
|
||||
|
||||
| Column | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| id | UUID | gen_random_uuid() | Primary key |
|
||||
| enabled | BOOLEAN | FALSE | Toggle SMTP on/off without deleting config |
|
||||
| host | TEXT | NOT NULL | SMTP server hostname or IP |
|
||||
| port | INTEGER | 465 | SMTP port (465 for TLS, 587 for STARTTLS) |
|
||||
| username | TEXT | NOT NULL | SMTP auth username |
|
||||
| password_encrypted | TEXT | NOT NULL | Encrypted SMTP password |
|
||||
| sender_email | TEXT | NOT NULL | From address (e.g. noreply@yourdomain.com) |
|
||||
| sender_name | TEXT | NOT NULL | Display name in recipient inbox |
|
||||
| min_interval_seconds | INTEGER | 60 | Minimum seconds between emails to same user |
|
||||
| created_at | TIMESTAMPTZ | NOW() | |
|
||||
| updated_at | TIMESTAMPTZ | NOW() | |
|
||||
|
||||
Singleton enforced via unique index on constant expression (same pattern as `auth.configs`).
|
||||
|
||||
### Table: `email.templates`
|
||||
|
||||
| Column | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| id | UUID | gen_random_uuid() | Primary key |
|
||||
| template_type | TEXT | NOT NULL, UNIQUE | Template identifier |
|
||||
| subject | TEXT | NOT NULL | Email subject line |
|
||||
| body_html | TEXT | NOT NULL | Raw HTML with placeholder variables |
|
||||
| created_at | TIMESTAMPTZ | NOW() | |
|
||||
| updated_at | TIMESTAMPTZ | NOW() | |
|
||||
|
||||
### Template Types
|
||||
|
||||
- `email-verification-code`
|
||||
- `email-verification-link`
|
||||
- `reset-password-code`
|
||||
- `reset-password-link`
|
||||
|
||||
### Template Placeholders
|
||||
|
||||
| Placeholder | Description | Available In |
|
||||
|-------------|-------------|--------------|
|
||||
| `{{ code }}` | 6-digit OTP code | verification-code, reset-password-code |
|
||||
| `{{ link }}` | Verification/reset URL | verification-link, reset-password-link |
|
||||
| `{{ email }}` | Recipient email address | All templates |
|
||||
|
||||
### Default Templates
|
||||
|
||||
Seeded on migration. Clean, minimal HTML with inline CSS. Example for `email-verification-code`:
|
||||
|
||||
- **Subject:** "Verify your email"
|
||||
- **Body:** Simple centered card with "Your verification code is: {{ code }}" and expiry notice
|
||||
|
||||
## Backend
|
||||
|
||||
### New: `SmtpEmailProvider`
|
||||
|
||||
**Location:** `backend/src/providers/email/smtp.provider.ts`
|
||||
|
||||
- Implements `EmailProvider` interface
|
||||
- Uses `nodemailer` to send emails
|
||||
- `sendWithTemplate()`: queries `email.templates WHERE template_type = $1` using the `EmailTemplate` string value, replaces placeholders via string interpolation (all placeholder values are HTML-escaped before interpolation to prevent XSS), sends via SMTP
|
||||
- `sendRaw()`: sends directly with provided to/subject/html/cc/bcc. The `from` field is always overridden with `sender_email`/`sender_name` from SMTP config (prevents spoofing)
|
||||
- `supportsTemplates()`: returns `true`
|
||||
- TLS auto-detected by nodemailer based on port (465 = implicit TLS, 587 = STARTTLS)
|
||||
- Transporter created on-demand from DB config (config changes take effect without restart)
|
||||
|
||||
**Note:** Enabling SMTP automatically switches from cloud-rendered templates to locally-rendered templates from `email.templates`, even if the user has not customized them. The seeded defaults ensure this works out of the box.
|
||||
|
||||
### Modified: `EmailService`
|
||||
|
||||
**Location:** `backend/src/services/email/email.service.ts`
|
||||
|
||||
- On each send call, check `email.config` for an enabled config
|
||||
- If enabled: delegate to `SmtpEmailProvider`
|
||||
- If not enabled: delegate to `CloudEmailProvider` (current default)
|
||||
- Provider is resolved per-call, not cached at startup
|
||||
|
||||
### New: SMTP Config Service
|
||||
|
||||
**Location:** `backend/src/services/email/smtp-config.service.ts`
|
||||
|
||||
- `getSmtpConfig()`: returns config with password masked
|
||||
- `upsertSmtpConfig(input)`: creates or updates config, encrypts password using `EncryptionManager` (AES-256-GCM). On save, validates SMTP connection via `transporter.verify()` — rejects invalid credentials before persisting
|
||||
- Singleton pattern (consistent with other services)
|
||||
|
||||
### New: Email Template Service
|
||||
|
||||
**Location:** `backend/src/services/email/email-template.service.ts`
|
||||
|
||||
- `getTemplates()`: returns all templates
|
||||
- `getTemplate(type)`: returns single template
|
||||
- `updateTemplate(type, subject, bodyHtml)`: updates a template
|
||||
|
||||
### Validation Schemas
|
||||
|
||||
New Zod schemas in `@insforge/shared-schemas` (consistent with existing patterns like `updateAuthConfigSchema`):
|
||||
|
||||
- `upsertSmtpConfigRequestSchema` — validates host, port (number), username, password (optional on update), sender_email (email format), sender_name, min_interval_seconds (positive integer), enabled (boolean)
|
||||
- `updateEmailTemplateRequestSchema` — validates subject (non-empty string), body_html (non-empty string)
|
||||
|
||||
### API Endpoints
|
||||
|
||||
All endpoints require `verifyAdmin` middleware (same as `GET/PUT /api/auth/config`).
|
||||
All PUT endpoints emit audit log entries via `auditService.log()` (consistent with existing `PUT /api/auth/config` pattern):
|
||||
- `PUT /api/auth/smtp-config` → action: `'UPDATE_SMTP_CONFIG'`
|
||||
- `PUT /api/auth/email-templates/:type` → action: `'UPDATE_EMAIL_TEMPLATE'`
|
||||
|
||||
#### SMTP Config
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/auth/smtp-config` | Get SMTP config (password masked as boolean) |
|
||||
| PUT | `/api/auth/smtp-config` | Create or update SMTP config |
|
||||
|
||||
#### Email Templates
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/auth/email-templates` | Get all email templates |
|
||||
| PUT | `/api/auth/email-templates/:type` | Update a template's subject and body |
|
||||
|
||||
### Password Handling
|
||||
|
||||
- Password encrypted before storage (never stored in plaintext)
|
||||
- GET endpoint returns `has_password: true/false` instead of the actual value
|
||||
- Password can only be overwritten, never viewed after saving
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
- `min_interval_seconds` from SMTP config enforced at the service level
|
||||
- Tracked via an in-memory `Map<string, number>` per recipient in `EmailService` (lightweight, no DB overhead; does not survive restarts or span multiple instances)
|
||||
- Before sending, check if enough time has elapsed since the last email to the same recipient; reject with 429 if not
|
||||
- All recipients are rate-limited (not just the primary), enforced for both `sendWithTemplate` and `sendRaw`
|
||||
- This is a service-level check (not Express middleware), since it depends on per-recipient state rather than per-IP
|
||||
|
||||
## Frontend
|
||||
|
||||
### Auth Settings Dialog — New "Email" Tab
|
||||
|
||||
**Location:** `frontend/src/features/auth/components/AuthSettingsMenuDialog.tsx`
|
||||
|
||||
A new tab added alongside existing tabs (General, Email Verification, Password).
|
||||
|
||||
#### Card 1: SMTP Provider Settings
|
||||
|
||||
- **Toggle:** "Enable Custom SMTP" (switch)
|
||||
- When enabled, show fields:
|
||||
- Sender email (text input)
|
||||
- Sender name (text input)
|
||||
- Host (text input)
|
||||
- Port (number input, default 465)
|
||||
- Minimum interval (number input, seconds, default 60)
|
||||
- Username (text input)
|
||||
- Password (password input, masked, cannot be viewed once saved)
|
||||
- Save button
|
||||
- Helper text: "Your SMTP credentials will always be encrypted in our database."
|
||||
|
||||
#### Card 2: Email Templates
|
||||
|
||||
- Dropdown: select template type
|
||||
- Subject line (text input)
|
||||
- Two sub-tabs: **Source** | **Preview**
|
||||
- Source: HTML textarea for editing raw HTML
|
||||
- Preview: rendered HTML in sandboxed iframe
|
||||
- Helper text showing available placeholders for selected template type
|
||||
- Save button (per template)
|
||||
|
||||
### UI Behavior
|
||||
|
||||
- Fields disabled when SMTP toggle is off
|
||||
- Password field shows placeholder dots when a password exists, empty when not set
|
||||
- Template preview updates live as user edits source HTML
|
||||
- Follows existing dialog styling and component patterns
|
||||
- The "Email" tab uses its own independent form state and API calls (separate from the existing auth config form), since it talks to different endpoints (`/api/auth/smtp-config` and `/api/auth/email-templates`)
|
||||
- The `AuthSettingsSection` type union is extended with `'email'`
|
||||
|
||||
## Migration
|
||||
|
||||
**New migration file:** `backend/src/infra/database/migrations/0XX_create-smtp-config-and-email-templates.sql`
|
||||
|
||||
1. Create `email.config` table with singleton constraint
|
||||
2. Create `email.templates` table
|
||||
3. Seed 4 default templates with subjects and HTML bodies
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `nodemailer` — new npm dependency for SMTP transport
|
||||
- `@types/nodemailer` — dev dependency for TypeScript types
|
||||
- No other new dependencies required
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Template engine (Handlebars, EJS, etc.) — simple string interpolation is sufficient
|
||||
- Env-var-based SMTP config — can be added later as an enhancement
|
||||
- Send Test Email button — not in Supabase's UX, not included here
|
||||
- Additional template types beyond the existing 4 auth flows
|
||||
- "Reset to Default" template button (known limitation — users must manually revert templates)
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- No "Reset to Default" for templates — if a user edits a template and wants to revert, they must manually restore the original HTML
|
||||
- Self-signed TLS certificates on SMTP servers are not supported (nodemailer rejects by default) — can be added later via an optional `tls_reject_unauthorized` flag
|
||||
@@ -0,0 +1,132 @@
|
||||
# Compute Dashboard UX Fixes
|
||||
|
||||
**Date:** 2026-04-07
|
||||
**Status:** Draft
|
||||
**Branch:** feat/compute-services
|
||||
|
||||
## Context
|
||||
|
||||
The Compute Services feature has a complete backend API and CLI, but the dashboard UI is read-only — users can see services but can't manage them. Primary users are agents (CLI/API); the dashboard serves as a monitoring/visibility layer.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Add action buttons (stop/start/delete) to detail view and card dropdown menus
|
||||
2. Add a "Create Service" dialog for basic service creation from the UI
|
||||
3. Add a logs panel in the detail view
|
||||
4. Show error state info on failed services
|
||||
5. Fix endpoint URL to show the real `.fly.dev` URL
|
||||
6. Add missing metadata (region, timestamps) to detail view
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Full redesign of page layout (keeping existing card grid + detail view)
|
||||
- Env var editing UI (use CLI for that)
|
||||
- Real-time log streaming (polling/manual refresh is fine)
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Detail View — Action Buttons
|
||||
|
||||
Add a button row next to the service name heading:
|
||||
|
||||
- **Stop** button (visible when status is `running`) — calls `stop` mutation
|
||||
- **Start** button (visible when status is `stopped`) — calls `start` mutation
|
||||
- **Delete** button (always visible) — opens ConfirmDialog, calls `remove` mutation
|
||||
- Buttons are disabled during pending mutations (loading spinners)
|
||||
- After delete succeeds, navigate back to list view
|
||||
|
||||
Uses: `Button` and `ConfirmDialog` from `@insforge/ui`.
|
||||
|
||||
### 2. Detail View — Events Panel
|
||||
|
||||
Below the specs card, add an events section:
|
||||
|
||||
- Fetches events via `computeServicesApi.events(id, 50)` using a `useQuery` with manual refetch
|
||||
- Displays as a scrollable monospace block with timestamps
|
||||
- "Refresh" button to re-fetch
|
||||
- Empty state: "No events available"
|
||||
- Only shown when `flyMachineId` exists (no events for services that never deployed)
|
||||
|
||||
New component: `ServiceEvents.tsx`
|
||||
|
||||
> **Why `/events` and not `/logs`:** the endpoint surfaces Fly machine
|
||||
> **lifecycle events** from `/apps/:app/machines/:id/events` (start/stop/restart/
|
||||
> exit), **not container stdout/stderr**. The honest name is "events"; calling
|
||||
> it `/logs` was misleading — it's enough to spot crash loops via exit-stopped
|
||||
> events but not enough to debug what the app actually printed.
|
||||
>
|
||||
> **Real container stdout/stderr is roadmap work**, and when it lands it
|
||||
> reuses the freshly-vacated `/logs` URL. Three viable paths, evaluated
|
||||
> [in this thread](https://github.com/InsForge/InsForge/pull/1062#discussion):
|
||||
>
|
||||
> | Path | Where flyctl spawns | Token source | Notes |
|
||||
> |---|---|---|---|
|
||||
> | A. Cloud-backend spawns `flyctl logs --no-tail -j` | cloud-backend host | org token in cloud's `process.env` | Token never leaves cloud. ~150 lines + tests. Mirrors existing `flyctl tokens attenuate` pattern in `defaultAttenuator`. |
|
||||
> | B. Per-app read-logs macaroon, OSS spawns | user's host | short-lived macaroon minted by cloud (mirrors `compute deploy-token`) | Adds flyctl dep on user laptops + token storage/rotation. ~230 lines + ops burden. |
|
||||
> | C. NATS streaming directly | OSS or cloud-backend | org token | What `flyctl logs` uses internally. Most "correct" but largest surface. |
|
||||
>
|
||||
> Recommendation when this lands: start with A (smallest delta, reuses
|
||||
> existing flyctl shell-out infra).
|
||||
|
||||
### 3. Detail View — Enhanced Specs
|
||||
|
||||
Add to the existing specs grid:
|
||||
- Region
|
||||
- Created at (formatted timestamp)
|
||||
- Updated at (formatted timestamp)
|
||||
|
||||
For failed services: show a red alert banner above the specs card saying "This service failed to deploy" with a Delete button.
|
||||
|
||||
### 4. Service Cards — Dropdown Menu
|
||||
|
||||
Add a three-dot `DropdownMenu` (from `@insforge/ui`) in the top-right of each card:
|
||||
|
||||
- Stop (when running)
|
||||
- Start (when stopped)
|
||||
- Delete (always, with confirm)
|
||||
|
||||
`onClick` stops propagation so the card click (navigate to detail) still works.
|
||||
|
||||
### 5. List Header — Create Button
|
||||
|
||||
Add a `+ Create Service` button next to the "Services" heading.
|
||||
|
||||
Opens a `Dialog` with form fields:
|
||||
- Name (text input, required)
|
||||
- Image URL (text input, required)
|
||||
- Port (number input, default 8080)
|
||||
- CPU tier (Select: shared-1x, shared-2x, performance-1x, etc.)
|
||||
- Memory (Select: 256, 512, 1024, 2048, 4096, 8192)
|
||||
- Region (Select: iad, sin, lax, etc.)
|
||||
|
||||
Calls `create` mutation on submit. Dialog closes on success.
|
||||
|
||||
New component: `CreateServiceDialog.tsx`
|
||||
|
||||
### 6. Endpoint URL Fix
|
||||
|
||||
In both `ServiceCard` and `ComputePage` detail view:
|
||||
- If `endpointUrl` contains a custom domain but `flyAppId` exists, show `https://{flyAppId}.fly.dev` instead
|
||||
- Helper function: `getReachableUrl(service)` in `constants.ts`
|
||||
|
||||
## Files
|
||||
|
||||
| File | Action | What |
|
||||
|------|--------|------|
|
||||
| `ComputePage.tsx` | Modify | Add create button, action buttons, events, enhanced specs, failed state |
|
||||
| `ServiceCard.tsx` | Modify | Add dropdown menu, use `getReachableUrl` |
|
||||
| `useComputeServices.ts` | Modify | Add `useServiceEvents` query hook export |
|
||||
| `constants.ts` | Modify | Add `getReachableUrl` helper |
|
||||
| `CreateServiceDialog.tsx` | New | Create service form dialog |
|
||||
| `ServiceEvents.tsx` | New | Events display component |
|
||||
|
||||
## Dependencies
|
||||
|
||||
All UI components needed already exist in `@insforge/ui`:
|
||||
- Button, Dialog, DialogContent, DialogTitle, DialogDescription
|
||||
- ConfirmDialog
|
||||
- DropdownMenu
|
||||
- Input, Select
|
||||
|
||||
All API methods already exist in `compute.service.ts`.
|
||||
All mutations already exist in `useComputeServices` hook.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Spec: docs/ audit + minimal high-severity fixes (closes #1117)
|
||||
|
||||
**Date:** 2026-04-18
|
||||
**Author:** Commander worker (insforge/commander/1117)
|
||||
**Tier:** T2
|
||||
|
||||
## Problem
|
||||
|
||||
InsForge's `docs/` tree has ~65 non-deprecated `.mdx` pages. Ticket #1117 asks
|
||||
for a systematic audit of these pages against the newly-vendored Mintlify
|
||||
`doc-author` skill (at `.claude/skills/doc-author/`) and the InsForge overlay
|
||||
at `.claude/skills/doc-author/INSFORGE.md`. Known-smell categories the ticket
|
||||
calls out:
|
||||
|
||||
- Broken internal links to renamed/removed pages
|
||||
- Navigation pointing at files that no longer exist
|
||||
- Stale code examples diverging from the current SDK surface
|
||||
- Deprecated APIs referenced from non-deprecated pages
|
||||
- Vanilla-markdown where Mintlify components would be idiomatic
|
||||
(`> ⚠️` → `<Warning>`, fence groups → `<CodeGroup>`, `1. 2. 3.` → `<Steps>`)
|
||||
- Frontmatter gaps per the InsForge overlay (`title`+`description` only;
|
||||
no `<ParamField>`; experimental features get `<Warning>` at top)
|
||||
|
||||
## Goals
|
||||
|
||||
1. Produce a single source-of-truth audit table at
|
||||
`docs/_audit-2026-04-18.md` listing every issue found, one row per
|
||||
`(page, issue)`, with severity and a recommended fix.
|
||||
2. Apply **minimal-diff** fixes to the top five high-severity pages in the
|
||||
same PR. Every commit cites the anti-pattern from the skill.
|
||||
3. File one follow-up GitHub issue per remaining high-severity page
|
||||
(assignee `tonychang04`, label `commander-ready`), and one catch-all
|
||||
issue for the combined medium/low-severity list.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Rewriting pages for style or tone. The skill explicitly favors
|
||||
minimal-diff.
|
||||
- Touching anything under `docs/deprecated/`. Those are intentionally
|
||||
preserved.
|
||||
- Source code changes. Audit is docs-only (`*.mdx`, `docs.json`, the new
|
||||
audit file itself).
|
||||
- Restructuring navigation beyond removing dead entries or fixing typos
|
||||
— structure work is its own follow-up ticket.
|
||||
|
||||
## Proposed approach
|
||||
|
||||
1. **Read** every `.mdx` outside `docs/deprecated/` (found 65 files).
|
||||
2. **Check** each against the categories above using targeted grep
|
||||
passes: link resolution, frontmatter shape, `<ParamField>` usage,
|
||||
markdown-warning patterns, emoji usage, fence groups, deprecated-API
|
||||
mentions.
|
||||
3. **Cross-check** `docs/docs.json` navigation entries against the
|
||||
filesystem — missing target files are high-severity (they break
|
||||
navigation rendering).
|
||||
4. **Cross-check** existing files against navigation — pages that exist
|
||||
but aren't reachable from nav are medium-severity orphans.
|
||||
5. **Write** the audit table.
|
||||
6. **Fix** the top five high-severity items with one commit per page,
|
||||
each commit message citing the skill section it enforces.
|
||||
7. **File** follow-up issues via `gh issue create` per the ticket.
|
||||
|
||||
### Alternatives considered
|
||||
|
||||
- **Rewrite pages wholesale to match skill voice/tone.** Rejected: the
|
||||
skill and ticket both explicitly forbid this.
|
||||
- **Skip the audit, just fix the obvious broken links.** Rejected: the
|
||||
ticket asks for the audit as the primary deliverable — the fixes are
|
||||
secondary. The audit becomes the roadmap for follow-up tickets.
|
||||
- **Include medium/low fixes in this PR.** Rejected: would blow past the
|
||||
25-minute budget and dilute review focus. Catch-all follow-up issue
|
||||
keeps them tracked without blocking this PR.
|
||||
|
||||
## Test plan
|
||||
|
||||
- Manual: every link modified in this PR resolves to an existing file
|
||||
on disk (`test -f docs/<target>.mdx`).
|
||||
- Manual: every page modified still parses as MDX (run `grep -c '^---$'`
|
||||
to verify frontmatter fences are balanced).
|
||||
- Manual: `docs.json` is valid JSON after any edits (`jq empty`).
|
||||
- Follow-up: a Mintlify CI check or local preview would catch any
|
||||
remaining nav/link regressions — out of scope for this PR.
|
||||
|
||||
## Risks / rollback
|
||||
|
||||
- **Risk:** removing a broken link from `changelog.mdx` might change
|
||||
rendered output in a way a reader already bookmarked. **Mitigation:**
|
||||
we repoint rather than delete where a valid destination exists (the
|
||||
SDK pages under `/sdks/typescript/*`).
|
||||
- **Risk:** fixing `docs.json` might drop an intentional placeholder
|
||||
for future content. **Mitigation:** each nav entry we remove is
|
||||
confirmed to have zero matching `.mdx` in the tree; if the author
|
||||
intended a placeholder, they can re-add on the follow-up ticket.
|
||||
- **Rollback:** revert the PR. All changes are docs-only; no runtime
|
||||
impact.
|
||||
@@ -0,0 +1,408 @@
|
||||
# D Test Onboarding Design
|
||||
|
||||
**Status:** Implemented
|
||||
**Owner:** @CarmenDou
|
||||
**Date:** 2026-04-21 (originally) · Updated 2026-04-25
|
||||
**Branch:** `feat/support-dtest-onboarding` · **PR:** [#1142](https://github.com/InsForge/insforge-cloud-backend/pull/1142)
|
||||
|
||||
## Context
|
||||
|
||||
Dashboard home is gated by a single PostHog feature flag `dashboard-v4-experiment` with two variants:
|
||||
|
||||
- default / `control` (`DashboardPage`) — baseline, unchanged
|
||||
- `d_test` — new **install-first** onboarding introduced in this spec
|
||||
|
||||
(An earlier `c_test` variant was retired during d_test development; the `CTestDashboardPage` and `ConnectDialogV2` files were deleted, with the prompt stepper carried forward into the d_test connected dashboard.)
|
||||
|
||||
D test ships a reworked "Install InsForge" client picker as the pre-connection view, and a connected dashboard (header + 4 metric cards + prompt stepper). On d_test the top-nav Connect button does **not** open any dialog — it switches the page back to the Install view so users can re-visit setup at any time. When dashboard runs inside the InsForge cloud control plane (`insforge.dev`) iframe, the parent's Connect button mirrors this behaviour through a `D_TEST_VIEW_CHANGED` postMessage.
|
||||
|
||||
Figma references:
|
||||
- Install InsForge (client picker): `2194:75236`
|
||||
- Client detail page (Claude Code example): `2226:78350`
|
||||
- Connection String detail: `2226:79152`
|
||||
- Connected dashboard: `2380:89947`
|
||||
|
||||
## Goals
|
||||
|
||||
1. Let users connect any coding agent (OpenClaw, Claude Code, Codex, Antigravity, Cursor, OpenCode, Copilot, Cline, "Other") or connect directly via DB connection string / API keys, from a single discoverable page.
|
||||
2. On the connected dashboard show: project header + 4 metric cards (User / Database / Storage / Edge Functions) + a "Your Agent can now do the work for you" prompt stepper to guide further exploration.
|
||||
3. Use d_test–owned install components (`DTestCLISection`, `DTestMCPSection`, `QuickStartPromptCard`) that can iterate independently of the legacy connect UI; keep the shared `ConnectionStringSectionV2` / `APIKeysSectionV2` for direct-connect tiles.
|
||||
4. Allow users to toggle between Install view and Dashboard view freely after first connection, including from the InsForge cloud control plane's top-bar Connect button (cross-frame postMessage).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing the install commands themselves at the CLI / MCP level (the CLI prompt is a copy-paste recipe; MCP JSON / install commands are the existing ones). The d_test prompt does inject a fresh user API key into the CLI command and substitutes the real DB password into the connection-string prompt — both are display-only changes.
|
||||
- Changing onboarding detection logic beyond what `useMcpUsage().hasCompletedOnboarding` already provides.
|
||||
- Replacing the `ConnectDialog` for the `control` variant — it keeps the existing modal.
|
||||
|
||||
## Connected-State Detection
|
||||
|
||||
D test treats a user as **connected** when `useMcpUsage().hasCompletedOnboarding` is true. That hook resolves to `!!records.length` where `records` comes from `/mcp-usage?success=true&limit=200`, i.e. the agent has successfully invoked ≥ 1 MCP tool.
|
||||
|
||||
This is the same signal C test uses today. No new backend work.
|
||||
|
||||
## View Model
|
||||
|
||||
Two top-level views, both mounted at `/dashboard` (the existing Dashboard home route), switched by an in-page state `view: 'install' | 'dashboard'`. View is **session-local React state** (not URL-backed, not persisted) — simpler than the earlier design and still covers every user-facing transition.
|
||||
|
||||
### View resolution
|
||||
|
||||
On mount, once `useMcpUsage()` finishes loading, the initial view is:
|
||||
|
||||
```text
|
||||
hasCompletedOnboarding ? 'dashboard' : 'install'
|
||||
```
|
||||
|
||||
Thereafter, the view only changes on three events:
|
||||
|
||||
1. **Onboarding completes** (`hasCompletedOnboarding` flips false → true): auto-switch to `'dashboard'`. This is the "MCP call succeeds → jump to dashboard" UX.
|
||||
2. **Connect clicked** (in d_test) — either our top-nav Connect button (`AppHeader`) when the dashboard renders standalone, or the **InsForge cloud control plane's** top-bar Connect button via `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` postMessage (the iframe scenario). Both route to `setView('install')`. While view is `'install'`, the Connect button is rendered as **disabled** so the user doesn't loop on it.
|
||||
3. **`[X]` clicked on Install page**: switch to `'dashboard'`.
|
||||
|
||||
On refresh the session state resets. The initial-view rule re-runs, so a connected user lands back on dashboard and an unconnected user lands on install — both are the correct defaults. The transient "I just clicked Connect to peek at install" intent is not persisted; if the user wants Install again, they click Connect again.
|
||||
|
||||
Within the Install view there is a sub-state `selectedClient`:
|
||||
|
||||
```text
|
||||
view = 'install'
|
||||
├── selectedClient === null → InstallInsForgePage (All Clients)
|
||||
└── selectedClient !== null → ClientDetailPage for that client
|
||||
```
|
||||
|
||||
`selectedClient` is session-local and resets when switching to dashboard.
|
||||
|
||||
## Navigation Map
|
||||
|
||||
```text
|
||||
┌────────────────────────────┐ ┌────────────────────────────┐
|
||||
│ InstallInsForgePage │ [X] close │ DTestConnectedDashboard │
|
||||
│ (All Clients) │────────────────────▶│ (header + 4 metrics) │
|
||||
│ │ │ │
|
||||
│ │◀────────────────────│ │
|
||||
└────┬───────────────────────┘ TopNav Connect └────────────────────────────┘
|
||||
│
|
||||
│ click tile
|
||||
▼
|
||||
┌────────────────────────────┐
|
||||
│ ClientDetailPage │
|
||||
│ (← All Clients) │─── CLI tab ──▶ <NewCLISection />
|
||||
│ │
|
||||
│ │─── MCP tab ──▶ <MCPSection initialAgentId={id} />
|
||||
│ │
|
||||
│ │ (or ConnectionStringSectionV2 / APIKeysSectionV2
|
||||
│ │ for Direct Connect tiles, no CLI/MCP toggle)
|
||||
└────────────────────────────┘
|
||||
```
|
||||
|
||||
## Install Page Layout
|
||||
|
||||
Three stacked sections, max-width 640 px, top-padding 64 px, centered:
|
||||
|
||||
1. **"Setup In OpenClaw"** — single tile for OpenClaw with `Install` button. (OpenClaw is a distinct agent, not a Figma typo for Claude Code; it is registered as its own `MCPAgent` with `id='openclaw'`, uses `@insforge/install --client openclaw`, and is the `FEATURED_OPENCLAW_ID` in `clientRegistry.tsx`.)
|
||||
2. **"Install in Coding Agent"** — 2-column × 4-row grid of tiles. Tiles in display order:
|
||||
1. Claude Code | Codex
|
||||
2. Antigravity | Cursor
|
||||
3. OpenCode | Copilot
|
||||
4. Cline | Other Agents
|
||||
3. **"Direct Connect"** — 2 tab-style tiles side by side: Connection String | API Keys. These are visually similar to agent tiles but open different detail content.
|
||||
|
||||
Top-right of the page header row (same row as the title, within the max-w-640 column): `[X]` close button → switches view to `'dashboard'` (clears `?view` param) and sets `installDismissed = true` in localStorage.
|
||||
|
||||
Title text: "Install InsForge".
|
||||
|
||||
## Client Detail Page Layout
|
||||
|
||||
Top: `← All Clients` text button (always the same label, regardless of client) → clears `selectedClient`.
|
||||
|
||||
Below: 32 px client icon + client display name (h2, 28 px medium).
|
||||
|
||||
Content changes per client type:
|
||||
|
||||
### Coding agents (OpenClaw, Claude Code, Codex, Antigravity, Cursor, OpenCode, Copilot, Cline, Other Agents)
|
||||
|
||||
- CLI / MCP toggle (`toggle nav` pattern from Figma) — only rendered when the entry's `tabs` field exposes more than one tab. **OpenClaw** has `tabs: ['cli']` (CLI only, no toggle); **Other Agents** has `tabs: ['mcp']` (MCP only, jumps directly into the MCP JSON config); the rest default to both.
|
||||
- **CLI tab** → `<DTestCLISection agentName={...} />`. The prompt embeds a real `uak_…` user API key minted by the cloud control plane (`onRequestUserApiKey` callback) on every section mount, with a 3-month TTL — falls back to `<placeholder>` when the host doesn't provide the callback (self-hosted preview).
|
||||
- **MCP tab** → `<DTestMCPSection agentId={id} apiKey={...} appUrl={...} />`.
|
||||
- For specific agents, `agentId` matches the tile id (`openclaw`, `claude-code`, `codex`, `cursor`, `antigravity`, `opencode`, `copilot`, `cline`).
|
||||
- For "Other Agents", the entry sets `mcpAgentId: 'mcp'` which jumps directly to the MCP JSON config (no agent dropdown needed).
|
||||
- For Cursor and Qoder (deeplink-capable), Step 1 shows an "Install to <agent>" button that opens the MCP-install deeplink and Step 2 shows a "Paste Prompt to <agent>" button that opens the agent's chat-with-prompt deeplink (`cursor://anysphere.cursor-deeplink/prompt?text=...` or `qoder://aicoding.aicoding-deeplink/chat?text=...&mode=agent`). Falls back to clipboard copy if the prompt exceeds Cursor's 8000-char URL limit. Other agents show the terminal command + prompt code blocks.
|
||||
|
||||
### Connection String tile
|
||||
|
||||
- No CLI/MCP toggle.
|
||||
- Wrapped in a `<QuickStartPromptCard />` whose prompt embeds the real DB connection string (parent's API returns it with the password masked as `********`; we substitute the real password in via `useDatabasePassword()` so the prompt is paste-ready).
|
||||
- Below the prompt: `<ConnectionStringSectionV2 variant="vertical" />` with a Show/Hide toggle on the password field. The "copy parameters" button always copies the real password regardless of reveal state, matching the connection-string copy behavior.
|
||||
- Title: "Connection String", icon: database.
|
||||
|
||||
### API Keys tile
|
||||
|
||||
- No CLI/MCP toggle.
|
||||
- Content: `<APIKeysSectionV2 apiKey={...} anonKey={...} appUrl={...} />`.
|
||||
- Title: "API Keys", icon: key.
|
||||
|
||||
## Connected Dashboard Layout
|
||||
|
||||
Matches Figma node `2380:89947`, with the prompt stepper carried over from the (now-deleted) c_test design.
|
||||
|
||||
```text
|
||||
<h1> My Project </h1> [INSTANCE BADGE] ● Healthy
|
||||
|
||||
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
|
||||
│ User │ │ DB │ │ Stor │ │ Fns │
|
||||
└──────┘ └──────┘ └──────┘ └──────┘
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Your Agent can now do the work for you [Dismiss] │
|
||||
│ Open your coding agent and start building your │
|
||||
│ project with prompts │
|
||||
│ │
|
||||
│ ┌────────────────┬──────────────────────────────────┐ │
|
||||
│ │ Database │ Step content (icon, title, │ │
|
||||
│ │ Authentication │ prompt body, Copy / Go-to) │ │
|
||||
│ │ Storage │ │ │
|
||||
│ │ Model Gateway │ │ │
|
||||
│ │ Deployment │ │ │
|
||||
│ └────────────────┴──────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Project title, instance-type badge, and health badge use `useCloudProjectInfo` + `useMetadata`. Each metric card uses the shared `MetricCard` component. The stepper is the self-contained `DashboardPromptStepper` — see the next section.
|
||||
|
||||
## Prompt Stepper (`DashboardPromptStepper`)
|
||||
|
||||
A 5-step "Start building" stepper rendered below the metric cards. Self-contained: each instance manages its own dismiss flag and step-completion derivation.
|
||||
|
||||
- **5 steps** (database → auth → storage → model gateway → deployment), each with a copy-pastable prompt and a "Go to <area>" button.
|
||||
- **Live completion detection** from existing hooks (intentionally schema-agnostic so prompts work for any project, not just a fixed demo):
|
||||
- `database` → `tables.some(t => t.recordCount > 0)` (any user table has rows)
|
||||
- `auth` → `totalUsers >= 1`
|
||||
- `storage` → `storage.buckets.length > 0` (any bucket exists)
|
||||
- `ai` → `aiUsageSummary.totalRequests > 0`
|
||||
- `deployment` → `currentDeploymentId` exists
|
||||
- **Sticky completion**: once a step is detected complete, it stays complete in localStorage (`insforge-ctest-step-<key>-done-<projectId>`) even if the agent later removes the source data (e.g. via newly added RLS policies). Key prefix is `insforge-ctest-` for backwards compatibility with users who progressed through c_test.
|
||||
- **Dismiss**: persists `insforge-prompt-stepper-dismissed-<projectId>` in localStorage. Once dismissed for a project, the stepper never shows for that project again.
|
||||
|
||||
## File Plan
|
||||
|
||||
### New files
|
||||
|
||||
```text
|
||||
packages/dashboard/src/features/dashboard/
|
||||
├── pages/
|
||||
│ └── DTestDashboardPage.tsx # entry; reads view state, dispatches
|
||||
└── components/
|
||||
└── dtest/
|
||||
├── InstallInsForgePage.tsx # All Clients view (3 sections)
|
||||
├── ClientDetailPage.tsx # detail shell: back + title + slot
|
||||
├── ClientTile.tsx # reusable tile for agents & direct-connect
|
||||
├── DTestConnectedDashboard.tsx # header + 4 metric cards + prompt stepper
|
||||
├── DTestCLISection.tsx # CLI tab content (prompt with real uak_ key)
|
||||
├── DTestMCPSection.tsx # MCP tab content (deeplinks for Cursor/Qoder, terminal/JSON for others)
|
||||
├── DTestConnectTip.tsx # fixed-position "you can re-connect" tip overlay (cloud-hosting only)
|
||||
├── DTestViewContext.tsx # React context: view + selectedClient + cross-frame postMessage
|
||||
├── DashboardPromptStepper.tsx # self-contained 5-step stepper for connected dashboard
|
||||
├── QuickStartPromptCard.tsx # generic "Paste this into your agent" prompt card
|
||||
└── clientRegistry.tsx # tile metadata (id, label, icon, kind, mcpAgentId, tabs)
|
||||
```
|
||||
|
||||
Plus a logo asset for the Other Agents tile (`assets/logos/other_agents.svg`) and updated logos for Claude Code (PNG) and Codex (SVG).
|
||||
|
||||
### Shared components extracted
|
||||
|
||||
```text
|
||||
packages/dashboard/src/features/dashboard/components/
|
||||
└── MetricCard.tsx # lifted from CTestDashboardPage.tsx (currently an inner function)
|
||||
```
|
||||
|
||||
`CTestDashboardPage.tsx` loses its inner `MetricCard` definition and imports the shared one.
|
||||
|
||||
### Modified files
|
||||
|
||||
- `packages/dashboard/src/router/AppRoutes.tsx`
|
||||
- Pick `DTestDashboardPage` when `dashboardVariant === 'd_test'`, otherwise `DashboardPage`.
|
||||
- `packages/dashboard/src/layout/AppLayout.tsx`
|
||||
- Always render the `ConnectDialog` (v1); the `c_test`-branched V2 dialog was removed.
|
||||
- Wrap the layout tree in `DTestViewProvider` so `AppHeader`, `DTestDashboardPage`, `DTestConnectTip`, and `AppSidebar` all share view state.
|
||||
- Add `ConnectOverlayBridge` (rendered inside the provider) — listens for `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` postMessages from the parent window. In d_test, routes the signal to `setView('install')` instead of opening the dialog.
|
||||
- `packages/dashboard/src/layout/AppHeader.tsx`
|
||||
- On d_test: Connect onClick calls `setView('install')` from `DTestViewContext`. Disabled when already on the Install view.
|
||||
- Tip JSX/state was extracted to `DTestConnectTip` (fixed-position overlay, see below).
|
||||
- `packages/dashboard/src/layout/AppSidebar.tsx`
|
||||
- Don't highlight the Dashboard nav item while the user is on the d_test Install view.
|
||||
- `packages/dashboard/src/lib/config/DashboardHostContext.tsx`
|
||||
- Add `onRequestUserApiKey?: () => Promise<string>` to the host contract, plumbed through `InsforgeDashboard` props.
|
||||
- `packages/dashboard/src/lib/analytics/posthog.tsx`
|
||||
- Restore `session_recording: { recordCrossOriginIframes: true }` so PostHog session replay doesn't choke on the cross-origin iframe boundary (was dropped in an earlier refactor).
|
||||
- `packages/dashboard/src/lib/contexts/SocketContext.tsx`
|
||||
- Rename the `experiment_variant` tag on `onboarding_completed` analytics to `dashboard-v4-experiment`.
|
||||
|
||||
#### Frontend bridge (`frontend/src/cloud-hosting/`)
|
||||
|
||||
- `useCloudHosting.ts`: adds `requestUserApiKey()` (REQUEST_USER_API_KEY postMessage with USER_API_KEY / USER_API_KEY_ERROR response).
|
||||
- `CloudHostingDashboard.tsx`: passes `onRequestUserApiKey={requestUserApiKey}` through to `InsForgeDashboard`.
|
||||
|
||||
### Deleted files
|
||||
|
||||
- `packages/dashboard/src/features/dashboard/pages/CTestDashboardPage.tsx` — c_test variant retired; the prompt stepper was extracted into `DashboardPromptStepper.tsx`.
|
||||
- `packages/dashboard/src/features/dashboard/components/connect/ConnectDialogV2.tsx` — only used by c_test.
|
||||
|
||||
## Client Registry
|
||||
|
||||
`clientRegistry.ts` centralizes the tile data so both the grid and the detail routing can look up by id:
|
||||
|
||||
```ts
|
||||
type ClientId =
|
||||
| 'openclaw'
|
||||
| 'claude-code'
|
||||
| 'codex'
|
||||
| 'antigravity'
|
||||
| 'cursor'
|
||||
| 'opencode'
|
||||
| 'copilot'
|
||||
| 'cline'
|
||||
| 'other'
|
||||
| 'connection-string'
|
||||
| 'api-keys';
|
||||
|
||||
type ClientEntry = {
|
||||
id: ClientId;
|
||||
label: string;
|
||||
icon: ReactNode;
|
||||
detailIcon: ReactNode;
|
||||
kind: 'agent' | 'direct-connect';
|
||||
/** MCP detail preselection. Use 'mcp' for "Other Agents"; omit for direct-connect. */
|
||||
mcpAgentId?: string;
|
||||
/**
|
||||
* Tabs available on the detail page for `kind: 'agent'`. Omit = both CLI and
|
||||
* MCP. Use ['cli'] for OpenClaw (install flow only), ['mcp'] for "Other
|
||||
* Agents" (drops straight into the MCP JSON config).
|
||||
*/
|
||||
tabs?: ReadonlyArray<'cli' | 'mcp'>;
|
||||
};
|
||||
```
|
||||
|
||||
`FEATURED_OPENCLAW_ID = 'openclaw'` is the featured tile in Section 1; `CODING_AGENT_GRID_IDS` renders the Section 2 grid starting with `'claude-code'`. The `other` entry sets `mcpAgentId: 'mcp'` and `tabs: ['mcp']`; OpenClaw sets `tabs: ['cli']`.
|
||||
|
||||
The "featured" section ("Setup In OpenClaw") and grid consume the same entries; only the section they render in differs.
|
||||
|
||||
## State Management
|
||||
|
||||
A React context (`DTestViewContext`) provided at `AppLayout` level owns `view` + `selectedClient` and exposes a `useDTestView` hook for both `AppHeader` and `DTestDashboardPage`:
|
||||
|
||||
```tsx
|
||||
export function DTestViewProvider({ children }: { children: ReactNode }) {
|
||||
const { hasCompletedOnboarding, isLoading } = useMcpUsage();
|
||||
const [selectedClient, setSelectedClient] = useState<ClientId | null>(null);
|
||||
const [view, setViewState] = useState<DTestView>('install');
|
||||
|
||||
// Initialise from onboarding state once loading finishes; thereafter
|
||||
// auto-flip to dashboard on every false → true transition.
|
||||
const didInit = useRef(false);
|
||||
const prevOnboarding = useRef(hasCompletedOnboarding);
|
||||
useEffect(() => {
|
||||
if (isLoading) return;
|
||||
if (!didInit.current) {
|
||||
setViewState(hasCompletedOnboarding ? 'dashboard' : 'install');
|
||||
didInit.current = true;
|
||||
} else if (!prevOnboarding.current && hasCompletedOnboarding) {
|
||||
setViewState('dashboard');
|
||||
}
|
||||
prevOnboarding.current = hasCompletedOnboarding;
|
||||
}, [hasCompletedOnboarding, isLoading]);
|
||||
|
||||
const setView = useCallback((next: DTestView) => {
|
||||
setViewState(next);
|
||||
if (next === 'dashboard') setSelectedClient(null);
|
||||
}, []);
|
||||
|
||||
// ...provider returned here
|
||||
}
|
||||
```
|
||||
|
||||
Key points:
|
||||
|
||||
- **No URL param, no localStorage.** View is pure session state. Refresh recomputes from `hasCompletedOnboarding`.
|
||||
- **Single source of truth for view.** `AppHeader.showConnectTip` and `DTestDashboardPage` both read `view` from the same context, so the Connect tip correctly hides while the user is on the Install view.
|
||||
- **Provider is mounted for every user**, not just d_test. Non-d_test components don't consume it, and `useMcpUsage()` is already React-Query-cached so the extra call is free.
|
||||
- **`[X]` on Install** calls `setView('dashboard')` — no dismissal flag, no persistence.
|
||||
- **Top-nav Connect on d_test** (only while on `/dashboard`) calls `setView('install')`.
|
||||
- **MCP call success** (the `hasCompletedOnboarding` false → true transition) auto-switches to `'dashboard'` so users see their connected state immediately.
|
||||
- `selectedClient` is session-local; switching to dashboard clears it.
|
||||
|
||||
## Cross-frame postMessage protocol
|
||||
|
||||
When the dashboard runs inside the InsForge cloud control plane (`insforge.dev`) via iframe, it coordinates with the parent through several postMessage events:
|
||||
|
||||
| Direction | Type | Purpose |
|
||||
|---|---|---|
|
||||
| Parent → iframe | `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` | Parent's top-bar Connect button click. iframe handles in `ConnectOverlayBridge`: in d_test → `setView('install')`; otherwise → opens the v1 ConnectDialog. |
|
||||
| iframe → Parent | `D_TEST_VIEW_CHANGED { view: 'install' \| 'dashboard' }` | View state mirror. Parent's `ConnectButton` reads this and disables itself while view is `'install'`. Only sent when the variant is `d_test`. |
|
||||
| iframe → Parent | `REQUEST_USER_API_KEY` | Iframe wants a fresh `uak_…` PAT for the CLI install prompt. |
|
||||
| Parent → iframe | `USER_API_KEY { apiKey }` / `USER_API_KEY_ERROR { error }` | Response to the above. Parent's `userApiKeyService` calls `POST /account/v1/api-keys` with a 90-day TTL. |
|
||||
|
||||
The `useCloudHosting` hook (in `frontend/src/cloud-hosting/`) owns the iframe-side request/response bookkeeping; the parent side lives in `insforge-cloud/src/app/dashboard/project/[projectId]/page.tsx` (existing handler) and `insforge-cloud/src/features/project/components/ConnectButton.tsx` (new disable-state subscriber).
|
||||
|
||||
## Feature Flag
|
||||
|
||||
Dashboard variant is gated by a single PostHog flag, `dashboard-v4-experiment`. Resolved values:
|
||||
|
||||
- `'d_test'` → `DTestDashboardPage`
|
||||
- anything else → `DashboardPage` (the legacy default)
|
||||
|
||||
```ts
|
||||
// AppRoutes.tsx
|
||||
const dashboardVariant = getFeatureFlag('dashboard-v4-experiment');
|
||||
const DashboardHomePage = dashboardVariant === 'd_test' ? DTestDashboardPage : DashboardPage;
|
||||
```
|
||||
|
||||
PostHog flag configuration is dashboard-side, out of scope for the code PR. The `dashboard-v3-experiment` flag is no longer referenced in code; SocketContext analytics report `dashboard-v4-experiment` instead.
|
||||
|
||||
## Testing
|
||||
|
||||
This is a UI-only change; verification is primarily manual through the dev server (PostHog override) and the staging cloud control plane (real iframe).
|
||||
|
||||
- For each variant (`control`, `d_test`):
|
||||
- Load `/dashboard` with an account that has **no** MCP usage → correct "unconnected" view renders.
|
||||
- Load `/dashboard` with an account that has MCP usage → correct "connected" view renders.
|
||||
- D-test-specific flows (self-hosted preview):
|
||||
- Click each agent tile → detail page renders with the right icon/title. OpenClaw shows CLI only. Other Agents shows MCP only. Other agents show CLI/MCP toggle, default to CLI.
|
||||
- Click Connection String tile → prompt + `ConnectionStringSectionV2` rendered inside detail shell. Real DB password substituted in the prompt and copy.
|
||||
- Click API Keys tile → `APIKeysSectionV2` renders inside the detail shell.
|
||||
- `← All Clients` from any detail → back to grid.
|
||||
- `[X]` on Install page → lands on dashboard view.
|
||||
- Connect button in top nav (on d_test) → routes to Install page. Becomes disabled while on Install.
|
||||
- MCP tool succeeds while on Install → view auto-flips to dashboard.
|
||||
- Cursor / Qoder "Paste Prompt to" button opens deeplink (URL bar shows `cursor://` or `qoder://`); for other agents, copies to clipboard.
|
||||
- D-test-specific flows (cloud iframe — staging):
|
||||
- Connect button in InsForge cloud's top-bar disables while iframe view = `'install'`.
|
||||
- CLI install prompt embeds a real `uak_…` key (each tab mount mints a new one).
|
||||
- DTestConnectTip overlay appears in cloud-hosting on dashboard view; dismiss persists per project.
|
||||
- Sidebar Dashboard nav item not highlighted while on Install view.
|
||||
- Cross-variant regression:
|
||||
- On `control`, Connect button still opens `ConnectDialog` modal, not Install page.
|
||||
|
||||
## Risk & Rollback
|
||||
|
||||
- Feature-flagged end-to-end; rollback is a PostHog flag change (set to `control` or remove).
|
||||
- `DTestViewProvider` is mounted for all users, not just d_test. It calls `useMcpUsage()` at layout level, but that hook is already invoked by `AppHeader` and is React-Query-cached, so the provider does not add a new request.
|
||||
- Cross-frame postMessage requires both halves (iframe-side `D_TEST_VIEW_CHANGED` emit + parent-side `ConnectButton` listener) to be deployed. Either half landing alone is harmless: the parent's Connect button just defaults to enabled, and the iframe's bridge silently no-ops if no listener exists.
|
||||
- User API key minting flow gates on `host.onRequestUserApiKey` being defined. Self-hosted installs (no host callback) fall back to `<placeholder>` in the CLI prompt — visible but obviously placeholder, copy disabled.
|
||||
- Backend-side: the cloud control plane's `userApiKeyService` calls `POST /account/v1/api-keys` with a 90-day TTL. Backend has a soft `MAX_ACTIVE_KEYS_PER_USER = 500` cap (in `appConfig.limits.maxActiveApiKeysPerUser`); on overflow returns 409 which surfaces as "Could not generate API key" in the UI without crashing.
|
||||
|
||||
## Connect Tip (`DTestConnectTip`)
|
||||
|
||||
Floating "You can always click here to re-connect" hint that appears on the connected dashboard view in cloud-hosting only. Rendered at `AppLayout` level (NOT inside `AppHeader`, because `showNavbar={false}` hides our `AppHeader` when the dashboard runs inside the cloud iframe — the tip needs to live outside it).
|
||||
|
||||
Display conditions (all must be true):
|
||||
- `host.mode === 'cloud-hosting'`
|
||||
- `dashboardVariant === 'd_test'`
|
||||
- Current view = `'dashboard'`
|
||||
- Not dismissed (per-project localStorage flag `insforge-dtest-connect-tip-dismissed-<projectId>`)
|
||||
|
||||
Position: `fixed right-4`. Top offset depends on `host.showNavbar`: `top-2` when our AppHeader is hidden (cloud-hosting iframe — sits just below the parent's top bar) or `top-14` when it shows (self-hosted preview — clears our 48px AppHeader). Dismissed state persists per-project; once dismissed, the tip never reappears for that project.
|
||||
|
||||
The arrow on the tip card points up at the (parent's) Connect button via offset `right-[72px]` within the 220px-wide card.
|
||||
|
||||
## Open Items
|
||||
|
||||
- None at the time of merge — d_test variant configuration in PostHog is set up; backend's `MAX_ACTIVE_KEYS_PER_USER = 500` cap is in place.
|
||||
@@ -0,0 +1,545 @@
|
||||
# S3-Compatible Storage Gateway
|
||||
|
||||
## Overview
|
||||
|
||||
Add an S3-protocol HTTP gateway in front of InsForge's Storage module so that any AWS S3-compatible client (`aws` CLI, `rclone`, AWS SDKs, Terraform, backup tools) can read and write InsForge buckets with no code changes.
|
||||
|
||||
The gateway sits at `/storage/v1/s3` on each project's backend host (e.g. `{appkey}.{region}.insforge.app/storage/v1/s3`). It verifies AWS SigV4 signatures against project-scoped access keys, dispatches to a small set of S3 operation handlers, and delegates physical IO to the existing `S3StorageProvider`. Object and bucket metadata stays consistent with the REST API by sharing the `storage.buckets` and `storage.objects` tables.
|
||||
|
||||
## Motivation
|
||||
|
||||
InsForge today exposes only a REST API for storage. Developers who want to migrate existing S3-based workloads (CI upload steps, `aws s3 sync`, backup scripts, Terraform `aws_s3_object` resources) have to rewrite their integration. Supabase shipped an S3-compatible gateway for the same reason; open-sourcing that design ([supabase/storage](https://github.com/supabase/storage)) means a well-trodden implementation path exists. This feature closes the compatibility gap so that InsForge Storage becomes drop-in usable from any S3 toolchain.
|
||||
|
||||
## Goals & Non-Goals
|
||||
|
||||
### Goals (v1)
|
||||
|
||||
1. `aws s3 cp <file> s3://<bucket>/<key>` and the reverse work zero-config (including files >200 MB via automatic multipart).
|
||||
2. `aws s3 sync` and `rclone sync` work bidirectionally.
|
||||
3. Objects uploaded via S3 protocol appear immediately in the InsForge Dashboard and REST API `GET /api/storage/buckets/:bucket/objects`. Reverse direction works too.
|
||||
4. Secret access keys are never exposed after creation; the DB stores only encrypted ciphertext.
|
||||
5. Large uploads stream end-to-end; memory usage does not scale with object size.
|
||||
|
||||
### Non-Goals (v1)
|
||||
|
||||
- Presigned URLs (query-string auth) for GET or PUT. Users wanting browser direct uploads use the existing REST `POST /api/storage/buckets/:bucket/upload-strategy`.
|
||||
- Session-token auth (user-JWT-scoped S3 access, Supabase-style `sessionToken` via `X-Amz-Security-Token`). v1 only supports the project-admin-level `storage.s3_access_keys` credentials. See Open Question 7.
|
||||
- Support for the `LocalStorageProvider` backend. The gateway refuses to mount if the backend is local; self-hosted users who want S3 protocol run MinIO and point `AWS_S3_BUCKET` + `S3_ENDPOINT_URL` at it.
|
||||
- Virtual-hosted-style URLs (`{bucket}.endpoint/...`). Only path-style (`endpoint/{bucket}/{key}`).
|
||||
- Signature V2.
|
||||
- S3 governance features: versioning, SSE-C / SSE-KMS, ACLs, bucket policy, object lock, tagging, lifecycle, replication, inventory, analytics, CORS config.
|
||||
- S3 event notifications.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Endpoint & Routing
|
||||
|
||||
- External endpoint: `https://{appkey}.{region}.insforge.app/storage/v1/s3`
|
||||
- SDK configuration: `{ endpoint, region: 'us-east-2', forcePathStyle: true, credentials }`
|
||||
- Signature region defaults to `us-east-2` to match the region our `S3StorageProvider` uses by default (see `s3.provider.ts`), so requests forwarded to the underlying S3 don't need a separate region translation step. The validated region comes from `AWS_REGION` — the same env var the S3 provider already reads — so clients sign with the same region the backing bucket lives in, and the Dashboard's S3 Config page (`GET /api/storage/s3/config`) surfaces exactly what the middleware will accept.
|
||||
- Mount path is `/storage/v1/s3` with **no `/api` prefix**. The `/api` prefix would force clients to configure `endpoint=<host>/api`, breaking S3 tooling conventions.
|
||||
|
||||
### Request Lifecycle
|
||||
|
||||
```text
|
||||
Client (aws CLI / SDK / rclone)
|
||||
│ PUT /storage/v1/s3/my-bucket/photo.jpg
|
||||
│ Authorization: AWS4-HMAC-SHA256 Credential=AK.../us-east-2/s3/aws4_request ...
|
||||
│ x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
|
||||
▼
|
||||
[1] Express app — `/storage/v1/s3/*` mounted BEFORE express.json() body parser.
|
||||
▼
|
||||
[2] SigV4 Middleware
|
||||
│ - Parse Authorization header → AccessKeyId
|
||||
│ - Lookup via LRU cache → storage.s3_access_keys row → decrypt secret
|
||||
│ - Verify header signature (clock skew, canonical request, string-to-sign, HMAC chain)
|
||||
│ - For STREAMING-* requests, only header verification happens here; body verification
|
||||
│ happens chunk-by-chunk as the stream is consumed downstream.
|
||||
│ - Asynchronously update last_used_at (fire and forget).
|
||||
▼
|
||||
[3] S3 Router — dispatch by (method, path shape, query string, headers).
|
||||
│ Examples: PUT /{bucket}/{key}?partNumber=N&uploadId=X → UploadPart;
|
||||
│ POST /{bucket}?delete → DeleteObjects.
|
||||
▼
|
||||
[4] Operation Handler (one file per op)
|
||||
│ - Delegates physical IO to S3StorageProvider (extended).
|
||||
│ - Reads/writes metadata via StorageService against storage.objects / storage.buckets.
|
||||
▼
|
||||
[5] Response Serializer
|
||||
│ - 2xx: status + S3-style headers (ETag, Content-Length, ...); ListXxx responds with XML.
|
||||
│ - Error: <Error><Code>...</Code>...</Error> XML via shared error helper.
|
||||
```
|
||||
|
||||
### Module Layout
|
||||
|
||||
```text
|
||||
backend/src/
|
||||
├── api/
|
||||
│ ├── middlewares/
|
||||
│ │ └── s3-sigv4.ts # SigV4 verification middleware + LRU cache
|
||||
│ └── routes/
|
||||
│ ├── s3-gateway/ # NEW
|
||||
│ │ ├── index.routes.ts # mount + method/path/query dispatch
|
||||
│ │ ├── commands/ # one file per S3 op (14 ops + 2 stubs)
|
||||
│ │ ├── xml.ts # XML serialization via xml2js
|
||||
│ │ └── errors.ts # S3 error code → XML response
|
||||
│ └── storage/
|
||||
│ └── index.routes.ts # EXTENDED with /s3/access-keys CRUD subroutes
|
||||
├── services/
|
||||
│ └── storage/
|
||||
│ ├── s3-access-key.service.ts # NEW — key CRUD, encryption, LRU cache
|
||||
│ ├── s3-signature.ts # NEW — SigV4 algorithm (header + chunked stream)
|
||||
│ └── storage.service.ts # EXTENDED — multipart-aware methods
|
||||
└── providers/
|
||||
└── storage/
|
||||
├── base.provider.ts # EXTENDED interface
|
||||
└── s3.provider.ts # implements new methods
|
||||
```
|
||||
|
||||
### Responsibility Boundaries
|
||||
|
||||
| Concern | Owner | Rationale |
|
||||
|---|---|---|
|
||||
| SigV4 verification and op dispatch | `s3-sigv4` middleware + router | Isolated from business logic, unit-testable. |
|
||||
| S3 op semantics (Put/Get/List/…) | `commands/*.ts` (one per op) | Keeps each file small and single-purpose. |
|
||||
| Physical object IO | `S3StorageProvider` | Only place that instantiates `S3Client`. |
|
||||
| Object metadata read/write | `StorageService` | Shared with REST path; prevents format drift. |
|
||||
| XML serialization | `xml.ts` | Consistent formatting across operations. |
|
||||
|
||||
### Body Parser Ordering
|
||||
|
||||
`server.ts` currently does:
|
||||
|
||||
```ts
|
||||
app.use(express.json({ limit: '100mb' }));
|
||||
app.use('/api', apiRouter);
|
||||
```
|
||||
|
||||
This must change to mount the S3 router **before** any body-consuming middleware:
|
||||
|
||||
```ts
|
||||
app.use('/storage/v1/s3', s3GatewayRouter); // streaming-aware, never calls express.json()
|
||||
|
||||
app.use(express.json({ limit: '100mb' }));
|
||||
app.use('/api', apiRouter);
|
||||
```
|
||||
|
||||
The S3 router handles `req` as a Readable stream directly and never consumes the body via `bodyParser` / `multer`.
|
||||
|
||||
## Access Key Model
|
||||
|
||||
### Database
|
||||
|
||||
Migration `033_create-s3-access-keys.sql`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS storage.s3_access_keys (
|
||||
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||
access_key_id TEXT NOT NULL UNIQUE,
|
||||
secret_access_key_encrypted TEXT NOT NULL,
|
||||
description TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_used_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_s3_access_keys_last_used_at
|
||||
ON storage.s3_access_keys (last_used_at);
|
||||
```
|
||||
|
||||
Table lives in the `storage` schema alongside `storage.buckets`, `storage.objects`, `storage.config`. No `app_key` column (one project per process). No `updated_at` (keys are immutable — modify by destroy + recreate). No foreign key to users (keys are project-scoped, not user-scoped).
|
||||
|
||||
### Credential Format
|
||||
|
||||
| Field | Format | Length | Example |
|
||||
|---|---|---|---|
|
||||
| `access_key_id` | `INSF` prefix + 16 upper-case alphanumeric | 20 | `INSFABC123DEF456GH78` |
|
||||
| `secret_access_key` | 40 base64url chars (from 30 random bytes, stripped of padding) | 40 | `x7K2-a_pL9qRs4N8vYzWcE1fH5gJ3mUtBoD6ViXk` |
|
||||
|
||||
Lengths match AWS conventions (20 / 40) to avoid SDK validation errors. Both fields are generated with `crypto.randomBytes` and then formatted. Base64url alphabet (`A–Z a–z 0–9 - _`) is chosen because AWS SigV4 puts the access key id into the canonical request as-is; any character outside `unreserved` would need URI-encoding handling in our own verifier.
|
||||
|
||||
### Secret Storage
|
||||
|
||||
The `secret_access_key_encrypted` column stores `EncryptionManager.encrypt()` output (AES-256-GCM), reversible because SigV4 verification requires the raw secret to recompute HMAC signatures. The encryption key comes from the existing `ENCRYPTION_KEY` env var. Rotating `ENCRYPTION_KEY` invalidates all stored secrets and requires users to recreate keys — documented behaviour.
|
||||
|
||||
### Constraints
|
||||
|
||||
- Hard cap of **50 keys per project**. `S3AccessKeyService.create` performs the count check and the insert inside a single SERIALIZABLE transaction, so concurrent creations cannot both pass the check and overshoot the cap. Over-limit returns `400 S3_ACCESS_KEY_LIMIT_EXCEEDED`.
|
||||
- Keys are immutable. No update endpoint.
|
||||
- Plaintext secret is returned **only once** in the creation response. Subsequent `GET` calls never return the secret.
|
||||
- `last_used_at` updated asynchronously on each successful SigV4 verification via `setImmediate` (fire-and-forget, errors swallowed to avoid blocking the request).
|
||||
|
||||
### Management API
|
||||
|
||||
Mounted under the existing `storageRouter`, protected by `verifyAdmin`:
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/storage/s3/access-keys` | Create; body `{ description? }`; response contains plaintext secret (once). |
|
||||
| `GET` | `/api/storage/s3/access-keys` | List (no secrets). |
|
||||
| `DELETE` | `/api/storage/s3/access-keys/:id` | Delete; LRU cache invalidated synchronously. |
|
||||
|
||||
Audit events: `CREATE_S3_ACCESS_KEY`, `DELETE_S3_ACCESS_KEY`.
|
||||
|
||||
### Runtime Cache
|
||||
|
||||
SigV4 verification is on the hot path (every S3 request). Pure DB lookup would be a bottleneck.
|
||||
|
||||
- In-process LRU keyed by `access_key_id`, value `{ secret_plaintext, id }`.
|
||||
- Size 1024 entries (well above the 50-key cap, so effectively never evicts) with 5-minute TTL (bounds staleness after delete).
|
||||
- Synchronous invalidation on delete.
|
||||
- Implementation: `lru-cache` npm package. (Already a transitive dep of AWS SDK; if not in lockfile, added directly.)
|
||||
|
||||
### Authorization Semantics
|
||||
|
||||
A valid S3 credential in its project grants:
|
||||
- Read and write on **all** buckets, ignoring `public`/`private` flags (those apply to anonymous access; credential holders are not anonymous).
|
||||
- `CreateBucket` / `DeleteBucket`.
|
||||
- No cross-project access possible — physical process isolation enforces this.
|
||||
|
||||
S3-protocol uploads record the originating access key via the new `s3_access_key_id` column on `storage.objects` (see Schema Extensions below). `uploaded_by` is `NULL` on S3-protocol uploads — we do not overload that UUID column with a string marker.
|
||||
|
||||
## Request Handling Pipeline
|
||||
|
||||
### Express Mount Order
|
||||
|
||||
Already covered above: `/storage/v1/s3` router mounts before `express.json()`.
|
||||
|
||||
### SigV4 Header-Signed Requests
|
||||
|
||||
AWS SigV4 (the short form):
|
||||
|
||||
1. Parse `Authorization: AWS4-HMAC-SHA256 Credential=<ak>/<date>/<region>/s3/aws4_request, SignedHeaders=<sorted;list>, Signature=<sig>`.
|
||||
2. Look up credential (cache, then DB) → plaintext secret.
|
||||
3. Build **Canonical Request**. The `<URI-encoded path>` MUST be derived from the raw, percent-encoded request path as the client sent it (e.g. `req.originalUrl` in Express), **not** a URL-decoded representation — otherwise object keys containing percent-encoded characters produce signature mismatches.
|
||||
```text
|
||||
<METHOD>\n
|
||||
<URI-encoded path>\n
|
||||
<canonical query>\n
|
||||
<canonical headers>\n
|
||||
\n
|
||||
<signed headers list>\n
|
||||
<x-amz-content-sha256 value>
|
||||
```
|
||||
4. Build **String-to-Sign**: `AWS4-HMAC-SHA256\n<datetime>\n<scope>\nSHA256(canonical request)`.
|
||||
5. Derive signing key: `HMAC(HMAC(HMAC(HMAC("AWS4"+secret, date), region), "s3"), "aws4_request")`.
|
||||
6. Compute `HMAC(signingKey, stringToSign)`, compare to client signature with `crypto.timingSafeEqual`. Length mismatch → return `SignatureDoesNotMatch` without invoking timing-safe comparison.
|
||||
7. For requests with a hashed body (not `UNSIGNED-PAYLOAD`, not `STREAMING-*`), additionally verify `SHA-256(body) === x-amz-content-sha256` header.
|
||||
|
||||
### SigV4 Streaming (Chunked) Requests
|
||||
|
||||
Triggered by `x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD`. Body structure:
|
||||
|
||||
```text
|
||||
<chunk1-size-hex>;chunk-signature=<sig1>\r\n
|
||||
<chunk1-payload>\r\n
|
||||
<chunk2-size-hex>;chunk-signature=<sig2>\r\n
|
||||
<chunk2-payload>\r\n
|
||||
...
|
||||
0;chunk-signature=<final-sig>\r\n
|
||||
\r\n
|
||||
```
|
||||
|
||||
Verification is streamed, not buffered:
|
||||
|
||||
1. Verify header signature as above; the canonical-request body hash is the literal `STREAMING-AWS4-HMAC-SHA256-PAYLOAD`. The computed header signature is the `seed_signature` for chunk verification.
|
||||
2. Pipe `req` through a `ChunkSignatureV4Parser` Transform with states `HEADER → DATA → CRLF → HEADER → …`:
|
||||
- Parse chunk header; record declared chunk size and signature.
|
||||
- Stream incoming bytes into a running `crypto.createHash('sha256')` while pushing them downstream.
|
||||
- At end of DATA: compose chunk string-to-sign = `AWS4-HMAC-SHA256-PAYLOAD\n<datetime>\n<scope>\n<previous_signature>\n<SHA256("")>\n<SHA256(chunk_payload)>`, HMAC-verify, compare to declared signature.
|
||||
- Chain: the first chunk's `previous_signature` is the header's `seed_signature`; each subsequent chunk uses the previous chunk's signature.
|
||||
3. Downstream (the `Body` parameter of `PutObjectCommand` / `UploadPartCommand`) receives only verified payload bytes.
|
||||
4. Any chunk signature mismatch: destroy the stream, return `SignatureDoesNotMatch` (403) XML error.
|
||||
|
||||
A `SegmentedBufferQueue` keeps header parsing memory bounded; chunk header length is capped at 128 bytes to prevent pathological allocations.
|
||||
|
||||
### Operation Dispatch
|
||||
|
||||
S3 keys off (method, path shape, query string, and sometimes headers). The dispatcher maps to command handlers:
|
||||
|
||||
| Method | Path | Query / Header | Op |
|
||||
|---|---|---|---|
|
||||
| `PUT` | `/{bucket}/{key}` | — | PutObject |
|
||||
| `PUT` | `/{bucket}/{key}` | `?partNumber=N&uploadId=X` | UploadPart |
|
||||
| `PUT` | `/{bucket}/{key}` | header `x-amz-copy-source` | CopyObject |
|
||||
| `POST` | `/{bucket}/{key}` | `?uploads` | CreateMultipartUpload |
|
||||
| `POST` | `/{bucket}/{key}` | `?uploadId=X` | CompleteMultipartUpload |
|
||||
| `POST` | `/{bucket}` | `?delete` | DeleteObjects |
|
||||
| `DELETE` | `/{bucket}/{key}` | — | DeleteObject |
|
||||
| `DELETE` | `/{bucket}/{key}` | `?uploadId=X` | AbortMultipartUpload |
|
||||
| `GET` | `/{bucket}/{key}` | — | GetObject |
|
||||
| `GET` | `/{bucket}/{key}` | `?uploadId=X` | ListParts |
|
||||
| `GET` | `/{bucket}` | — / `?list-type=2` | ListObjectsV2 |
|
||||
| `GET` | `/` | — | ListBuckets |
|
||||
| `HEAD` | `/{bucket}/{key}` | — | HeadObject |
|
||||
| `HEAD` | `/{bucket}` | — | HeadBucket |
|
||||
| `PUT` | `/{bucket}` | — | CreateBucket |
|
||||
| `DELETE` | `/{bucket}` | — | DeleteBucket |
|
||||
|
||||
Implementation is a single `dispatch(req)` function, not Express's per-verb router, because the path shapes conflict (`/bucket/key` vs `/bucket` vs `/`).
|
||||
|
||||
### Path Parsing
|
||||
|
||||
- Express mounts at `/storage/v1/s3`, so `req.path` is `/{bucket}/{key}` or `/{bucket}/` or `/`.
|
||||
- First segment → bucket name; remainder → object key.
|
||||
- Bucket name validation reuses the existing regex `^[a-zA-Z0-9_-]+$`.
|
||||
- Key validation rejects `..` and leading `/` (same rules as the REST layer).
|
||||
|
||||
### Clock Skew Protection
|
||||
|
||||
`X-Amz-Date` is compared to server time with a tolerance of **15 minutes** (AWS standard). Out-of-range returns `RequestTimeTooSkewed` before signature comparison.
|
||||
|
||||
## Provider Extensions & Metadata Sync
|
||||
|
||||
### `StorageProvider` Interface Additions
|
||||
|
||||
```ts
|
||||
interface StorageProvider {
|
||||
// ...existing methods kept
|
||||
|
||||
putObjectStream(
|
||||
bucket: string,
|
||||
key: string,
|
||||
body: Readable,
|
||||
opts: { contentType?: string; contentLength?: number }
|
||||
): Promise<{ etag: string; size: number }>;
|
||||
|
||||
headObject(bucket: string, key: string): Promise<{
|
||||
size: number;
|
||||
etag: string;
|
||||
contentType?: string;
|
||||
lastModified: Date;
|
||||
} | null>;
|
||||
|
||||
copyObject(
|
||||
srcBucket: string, srcKey: string,
|
||||
dstBucket: string, dstKey: string
|
||||
): Promise<{ etag: string; lastModified: Date }>;
|
||||
|
||||
getObjectStream(
|
||||
bucket: string,
|
||||
key: string,
|
||||
opts?: { range?: string }
|
||||
): Promise<{
|
||||
body: Readable;
|
||||
size: number;
|
||||
etag: string;
|
||||
contentType?: string;
|
||||
lastModified: Date;
|
||||
}>;
|
||||
|
||||
createMultipartUpload(
|
||||
bucket: string, key: string, opts: { contentType?: string }
|
||||
): Promise<{ uploadId: string }>;
|
||||
|
||||
uploadPart(
|
||||
bucket: string, key: string, uploadId: string,
|
||||
partNumber: number, body: Readable, contentLength: number
|
||||
): Promise<{ etag: string }>;
|
||||
|
||||
completeMultipartUpload(
|
||||
bucket: string, key: string, uploadId: string,
|
||||
parts: Array<{ partNumber: number; etag: string }>
|
||||
): Promise<{ etag: string; size: number }>;
|
||||
|
||||
abortMultipartUpload(bucket: string, key: string, uploadId: string): Promise<void>;
|
||||
|
||||
listParts(
|
||||
bucket: string, key: string, uploadId: string,
|
||||
opts: { maxParts?: number; partNumberMarker?: number }
|
||||
): Promise<{
|
||||
parts: Array<{ partNumber: number; etag: string; size: number; lastModified: Date }>;
|
||||
isTruncated: boolean;
|
||||
nextPartNumberMarker?: number;
|
||||
}>;
|
||||
}
|
||||
```
|
||||
|
||||
### LocalStorageProvider Behavior
|
||||
|
||||
All new methods throw:
|
||||
|
||||
```ts
|
||||
throw new AppError(
|
||||
'S3 protocol requires an S3 storage backend. Set AWS_S3_BUCKET (and optionally S3_ENDPOINT_URL for MinIO).',
|
||||
501, ERROR_CODES.NOT_IMPLEMENTED
|
||||
);
|
||||
```
|
||||
|
||||
At startup, the gateway detects the active provider. If not `S3StorageProvider`, mount a stub router at `/storage/v1/s3` that returns a 501 S3 XML error for every request.
|
||||
|
||||
### S3StorageProvider Implementation
|
||||
|
||||
Each new method is a thin wrapper over an AWS SDK v3 command, reusing the existing `getS3Key(bucket, key)` helper that applies the `{appKey}/{bucket}/{key}` prefix. Multipart operations use real S3 multipart (uploadId is the real S3 uploadId); InsForge stores no multipart state in its own DB.
|
||||
|
||||
### Metadata Synchronization Table
|
||||
|
||||
| Op | S3 action | DB action |
|
||||
|---|---|---|
|
||||
| `PutObject` | `PutObjectCommand` (streaming) | `INSERT ... ON CONFLICT (bucket, key) DO UPDATE` |
|
||||
| `GetObject` | `GetObjectCommand` stream | Read metadata from `storage.objects` |
|
||||
| `HeadObject` | — | Read from `storage.objects` |
|
||||
| `DeleteObject` | `DeleteObjectCommand` | `DELETE FROM storage.objects WHERE bucket=$1 AND key=$2` |
|
||||
| `DeleteObjects` | Parallel `DeleteObjectCommand`s | Single `DELETE ... WHERE (bucket, key) IN (...)` |
|
||||
| `CopyObject` | `CopyObjectCommand` (server-side) | `INSERT` destination row |
|
||||
| `CreateMultipartUpload` | `CreateMultipartUploadCommand` | None |
|
||||
| `UploadPart` | `UploadPartCommand` (streaming) | None |
|
||||
| `CompleteMultipartUpload` | `CompleteMultipartUploadCommand` | `INSERT ... ON CONFLICT DO UPDATE` with final size + ETag |
|
||||
| `AbortMultipartUpload` | `AbortMultipartUploadCommand` | None |
|
||||
| `CreateBucket` | `S3StorageProvider.createBucket` | `INSERT INTO storage.buckets (name, public) VALUES ($1, false)` |
|
||||
| `DeleteBucket` | `S3StorageProvider.deleteBucket` | Check empty first; `DELETE FROM storage.buckets ...` |
|
||||
| `ListObjectsV2` | — | Query `storage.objects` |
|
||||
| `ListBuckets` | — | Query `storage.buckets` |
|
||||
| `HeadBucket` | — | Query `storage.buckets` |
|
||||
|
||||
List operations read from the DB, not live S3, because the DB is the source of truth for object metadata.
|
||||
|
||||
### Schema Extensions for `storage.objects`
|
||||
|
||||
Migration `034_extend-storage-objects-for-s3-protocol.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE storage.objects
|
||||
ADD COLUMN IF NOT EXISTS uploaded_via TEXT NOT NULL DEFAULT 'rest'
|
||||
CHECK (uploaded_via IN ('rest', 's3', 'dashboard')),
|
||||
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT,
|
||||
ADD COLUMN IF NOT EXISTS etag TEXT;
|
||||
```
|
||||
|
||||
- Existing REST/Dashboard upload paths write `uploaded_via='rest'` or `'dashboard'`, leave `s3_access_key_id` NULL.
|
||||
- S3 gateway writes `uploaded_via='s3'`, `s3_access_key_id=<ak>`, `uploaded_by=NULL`.
|
||||
- `etag` populated for all future uploads so HeadObject does not need to fall back to live S3.
|
||||
- `uploaded_by` column stays; its type is unchanged. S3 writes use NULL there.
|
||||
|
||||
### ListObjectsV2 Implementation Notes
|
||||
|
||||
- Query params parsed: `prefix`, `delimiter`, `continuation-token`, `start-after`, `max-keys` (capped at 1000; default 1000).
|
||||
- Base query: `SELECT key, size, mime_type, etag, uploaded_at FROM storage.objects WHERE bucket=$1 AND key LIKE $prefix||'%' [AND key > $start_after] ORDER BY key LIMIT $max_keys+1`.
|
||||
- The `+1` detects truncation without a second query.
|
||||
- `delimiter='/'`: common-prefix rollup done in application code after SELECT (SQL approach is brittle across edge cases).
|
||||
- `continuation-token`: base64-encoded last returned key.
|
||||
|
||||
## Operation Scope (v1)
|
||||
|
||||
### Implemented
|
||||
|
||||
**Bucket-level (5):** `ListBuckets`, `CreateBucket`, `DeleteBucket`, `HeadBucket`, `ListObjectsV2`.
|
||||
|
||||
**Object-level (11):** `PutObject`, `GetObject`, `HeadObject`, `DeleteObject`, `DeleteObjects` (batch), `CopyObject`, `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListParts`.
|
||||
|
||||
**Stubs (2):** `GetBucketLocation` returns `<LocationConstraint>us-east-2</LocationConstraint>`; `GetBucketVersioning` returns `<VersioningConfiguration><Status>Disabled</Status></VersioningConfiguration>`. Both are commonly probed by SDKs on client init.
|
||||
|
||||
### Explicitly Rejected (return `NotImplemented` 501)
|
||||
|
||||
`GetBucketAcl`, `PutBucketAcl`, `GetBucketCors`, `PutBucketCors`, `GetObjectTagging`, `PutObjectTagging`, `GetObjectAcl`, `PutObjectAcl`, `UploadPartCopy`, and all versioning / lifecycle / replication / inventory endpoints.
|
||||
|
||||
### Bucket Name Rules
|
||||
|
||||
`CreateBucket` applies the existing InsForge regex `^[a-zA-Z0-9_-]+$`. This is looser than AWS's DNS-compatible rules (lowercase only, 3–63 chars, no underscores). Tradeoff: REST and S3 see the same rules, at the cost of occasional SDK-side warnings. Documented.
|
||||
|
||||
### Size Limits
|
||||
|
||||
- Single `PutObject`: 5 GB (AWS cap). Enforced via `Content-Length`; over-limit returns `EntityTooLarge` 400.
|
||||
- `UploadPart`: 5 MB min (except last part), 5 GB max.
|
||||
- Total multipart object: 5 TB (enforced by real S3, we pass-through the error).
|
||||
- Optional env var `S3_PROTOCOL_MAX_OBJECT_SIZE_GB` can **lower** the per-object cap below 5 GB for a given deployment (abuse protection). It cannot raise the ceiling above the AWS single-PutObject max of 5 GB. Default: unset (i.e. the 5 GB cap applies). Does not relate to REST-layer `storage.config.max_file_size_mb`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
All non-2xx responses return S3-format XML:
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Error>
|
||||
<Code>NoSuchKey</Code>
|
||||
<Message>The specified key does not exist.</Message>
|
||||
<Resource>/my-bucket/photo.jpg</Resource>
|
||||
<RequestId>...</RequestId>
|
||||
</Error>
|
||||
```
|
||||
|
||||
### Error Code Map
|
||||
|
||||
| Internal condition | S3 code | HTTP |
|
||||
|---|---|---|
|
||||
| Signature mismatch | `SignatureDoesNotMatch` | 403 |
|
||||
| Access key missing or deleted | `InvalidAccessKeyId` | 403 |
|
||||
| Clock skew > 15 min | `RequestTimeTooSkewed` | 403 |
|
||||
| Malformed Authorization header | `AuthorizationHeaderMalformed` | 400 |
|
||||
| Bucket missing | `NoSuchBucket` | 404 |
|
||||
| Object missing | `NoSuchKey` | 404 |
|
||||
| Bucket already exists | `BucketAlreadyOwnedByYou` | 409 |
|
||||
| Non-empty on DeleteBucket | `BucketNotEmpty` | 409 |
|
||||
| Invalid bucket name | `InvalidBucketName` | 400 |
|
||||
| Body exceeds single-object limit | `EntityTooLarge` | 400 |
|
||||
| Multipart part too small | `EntityTooSmall` | 400 |
|
||||
| Unsupported operation | `NotImplemented` | 501 |
|
||||
| Anything else | `InternalError` | 500 |
|
||||
|
||||
Shared helper `sendS3Error(res, code, message, requestId)` in `s3-gateway/errors.ts`. `RequestId` = `crypto.randomUUID()` generated per request and echoed into audit / access logs.
|
||||
|
||||
## Security
|
||||
|
||||
- Secret exposure surface is exactly one response (create). Secrets are never logged; request/response logging scrubs the `secret_access_key_encrypted` column. LRU cache lives only in memory; process restart clears it.
|
||||
- Rate limiting: the S3 path is excluded from `express-rate-limit` for request-level throttling (because streaming uploads would misfire). Instead, per-access-key throttling is applied in the SigV4 middleware.
|
||||
- Audit: `CREATE_S3_ACCESS_KEY` / `DELETE_S3_ACCESS_KEY` go to `AuditService`. Data-plane S3 operations are not audited (volume is prohibitive); access logs are sufficient.
|
||||
- Encryption key rotation: rotating `ENCRYPTION_KEY` invalidates stored secrets. Users must recreate keys. Behaviour documented.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Testing is co-equal with implementation — the quality gate for "S3-compatible".
|
||||
|
||||
1. **SigV4 unit tests** using AWS's published `aws4_testsuite` fixtures (canonical request / string-to-sign / signature triples hardcoded, compared to our implementation).
|
||||
2. **Integration tests (CI)**: run the backend against MinIO via docker-compose. Exercise the full op surface via `@aws-sdk/client-s3`: Put/Get/Head small + large + multipart, ListObjectsV2 with pagination / prefix / delimiter, Copy, Delete, DeleteObjects.
|
||||
3. **CLI black-box tests (manual at minimum, scripted in CI where feasible)**:
|
||||
- `aws s3 cp` — 100 KB, 100 MB, 1 GB (forces multipart).
|
||||
- `aws s3 sync` — both directions.
|
||||
- `aws s3 rm --recursive`.
|
||||
- `rclone copy` and `rclone sync`.
|
||||
4. **Negative tests**: wrong signature, expired date, deleted access key, non-existent bucket/key, DeleteBucket on non-empty bucket.
|
||||
|
||||
Tests live under `backend/tests/s3-gateway/`.
|
||||
|
||||
## Risk Register
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| SigV4 implementation bug breaks some SDKs | High | Port-by-line from Supabase's `signature-v4.ts`; compliance test suite covers aws-cli / aws-sdk-js / boto3 / rclone. |
|
||||
| Streaming chunk parser state-machine bug corrupts uploads or leaks memory | High | Unit tests cover chunk-size boundaries (exact, multi-packet, equal to highWaterMark); `byte-limit-stream` caps per-request size. |
|
||||
| `storage.objects` and real S3 drift (S3 write succeeds, DB write fails) | Medium | Write S3 first, then DB; on DB failure, compensating S3 delete; periodic reconcile job. |
|
||||
| Mount order regression swallows raw body via `express.json()` | Medium | Integration test that streams a 10 MB body through to verify chunks arrive; code-review checklist. |
|
||||
| Hot access key causes DB contention | Medium | LRU cache + single-flight mutex on cache miss. |
|
||||
| `crypto.timingSafeEqual` throws on length mismatch | Low | Length check before call; mismatched length returns `SignatureDoesNotMatch`. |
|
||||
| Client with wrong region signature slips through | Low | Validate `Credential` scope's region field equals `us-east-2`. |
|
||||
| Noise from unsupported presigned-URL failures | Low | Docs explicitly redirect users to REST `upload-strategy`. |
|
||||
| Credential enumeration / DoS | Low | 50-key hard cap; per-access-key rate limit. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
These are parked; they don't block the design but should be resolved during or before implementation.
|
||||
|
||||
1. **Dashboard UI** for S3 access key management. Scope-excluded from this design; tracked as a separate spec / PR.
|
||||
2. **`storage.objects.etag` backfill.** Existing rows need `etag` populated (either lazily on first HeadObject or via migration backfill). Lazy is simpler, migration is more consistent — decide during implementation.
|
||||
3. **MinIO self-hosted support.** The design supports it by construction; should we document and recommend it as the self-hosted S3-protocol path? (Recommend: yes.)
|
||||
4. **Updating existing REST/Dashboard upload paths** to write `uploaded_via='rest' | 'dashboard'`. Required to keep the new column meaningful. Non-breaking (has DEFAULT).
|
||||
5. **Cross-project isolation** rests on the "one process = one app_key" deployment assumption. If the platform ever consolidates processes, this design needs revisiting. Add a code comment flagging the assumption.
|
||||
6. **Host-based routing** (`{appkey}.{region}.insforge.app`) is an infrastructure-layer concern (ingress / DNS / ALB), not backend code. The design assumes it works; ingress config is out of scope for this spec.
|
||||
7. **Session-token auth (user-JWT-scoped S3 access)** — Supabase supports a third credential shape `{accessKeyId: project_ref, secretAccessKey: anonKey, sessionToken: <user JWT>}` that lets S3 operations respect per-user permissions via Postgres RLS. Deliberately cut from v1:
|
||||
- The `storage.s3_access_keys` path covers the main server-side use cases (CI, scripts, backup tooling, rclone).
|
||||
- InsForge does not use Postgres RLS today; replicating Supabase's "DB filters by JWT" model would mean building an application-layer user-scoping policy on every S3 handler — a separate design problem with its own scope.
|
||||
- Browser direct uploads are already served by `POST /api/storage/buckets/:bucket/upload-strategy`.
|
||||
|
||||
If added later, the shape should be: detect `X-Amz-Security-Token` in the SigV4 middleware, verify via `TokenManager.verifyToken()`, attach the user identity to `S3AuthContext`, and introduce a handler-layer policy (or bucket-visibility + ownership check) gating every operation. Needs its own spec.
|
||||
|
||||
## Tradeoffs Summary
|
||||
|
||||
**Pros**
|
||||
- Aligned with Supabase's open-source protocol path; documentation and community experience transfer.
|
||||
- Shared namespace means Dashboard and S3 protocol see the same objects — minimal user cognitive load.
|
||||
- Single-tenant per process avoids multi-tenant routing work.
|
||||
- Reuses existing infrastructure (`EncryptionManager`, `StorageService`, `S3StorageProvider`, migration framework).
|
||||
|
||||
**Cons**
|
||||
- Implementation size is non-trivial (~2500–3500 lines of new code + tests).
|
||||
- SigV4 chunked-payload parser is precision protocol work — bugs show up as "one SDK works, another doesn't".
|
||||
- Presigned URLs cut from v1 for scope; adding them later touches the data plane.
|
||||
- LocalStorageProvider users don't get S3 protocol; they must run MinIO.
|
||||
@@ -0,0 +1,121 @@
|
||||
# Backend Branching (OSS) — Design
|
||||
|
||||
**Date:** 2026-04-29
|
||||
**Status:** Draft
|
||||
**Source PRD:** [insforge-cloud-backend Backend-Branching.md](../../../../insforge-cloud-backend/Backend-Branching.md)
|
||||
**Cloud-backend spec:** [2026-04-29-backend-branching-cloud-design.md](../../../../insforge-cloud-backend/docs/superpowers/specs/2026-04-29-backend-branching-cloud-design.md)
|
||||
|
||||
This spec covers the **OSS slice** of backend branching v1: read-only fallback from a branch project's storage to its parent project's S3 directory.
|
||||
|
||||
## Problem
|
||||
|
||||
When a branch project is created from a parent, copying every storage object would be slow and wasteful (parent may have GB of files). The cloud-backend spec dictates: branch's S3 directory starts empty; reads of parent objects must transparently succeed via fallback. Branch writes go to branch's directory only; parent files are never modified from a branch.
|
||||
|
||||
## Goals (v1)
|
||||
|
||||
1. The OSS storage server, when started in "branch mode" (with an injected `PARENT_APP_KEY` env var), reads object data from parent's S3 path when the branch's path returns nothing.
|
||||
2. Fallback is **read-only**. Writes (PUT/DELETE/multipart-upload) always target the branch's own appkey path.
|
||||
3. Existing RLS / bucket-visibility checks continue to apply unchanged.
|
||||
4. No HTTP API additions. Behavior change is server-internal.
|
||||
5. Local-storage provider does **not** support fallback (cloud-only). Local installs are not branched.
|
||||
|
||||
## Non-Goals (v1)
|
||||
|
||||
- Cross-project fallback for non-branch projects.
|
||||
- Write-through: copying a parent object into branch when first written to (CoW). Branches always start with the empty file set in their own namespace; modifications create new objects.
|
||||
- Deletion markers: branch cannot "hide" a parent file. Deleting from a branch only deletes the branch's own copy (which doesn't exist for inherited files), so the parent file remains visible. Documented as v1 limitation.
|
||||
- Schema-only mode files: when the branch was created with `mode='schema-only'`, the cloud-backend truncates `storage.objects`, so RLS/metadata returns 404 for parent files. This matches "fresh start" semantics — fallback is only effective for `mode='full'` branches.
|
||||
|
||||
## Current State Summary
|
||||
|
||||
- OSS storage server: TypeScript / Express monorepo at `backend/`. See [backend/src/server.ts:1-80](../../../backend/src/server.ts).
|
||||
- S3 path layout: `{appKey}/{bucketName}/{key}` in a single shared bucket (`AWS_S3_BUCKET`). Code in [backend/src/providers/storage/s3.provider.ts:91-93](../../../backend/src/providers/storage/s3.provider.ts).
|
||||
- `appKey` loaded from `APP_KEY` env var at server startup. Singleton pattern in [backend/src/services/storage/storage.service.ts:29-45](../../../backend/src/services/storage/storage.service.ts).
|
||||
- HTTP read endpoint: `GET /api/storage/buckets/:bucketName/objects/*` ([backend/src/api/routes/storage/index.routes.ts:398-459](../../../backend/src/api/routes/storage/index.routes.ts)).
|
||||
- `StorageService.getObject` returns `null` if metadata absent or S3 returns nothing — these are the natural interception points for fallback ([backend/src/services/storage/storage.service.ts:201-238](../../../backend/src/services/storage/storage.service.ts)).
|
||||
- Presigned URL flow: `getDownloadStrategy` calls `getSignedUrl` from `@aws-sdk/s3-request-presigner` (S3 sigV4 GET against `{appKey}/{bucket}/{key}`).
|
||||
|
||||
## Design
|
||||
|
||||
### Configuration
|
||||
|
||||
New env var:
|
||||
- `PARENT_APP_KEY` (optional). When set, the server runs in "branch mode" and falls back to this appkey for object reads.
|
||||
|
||||
The cloud-backend injects this at branch container startup (alongside the existing `APP_KEY`).
|
||||
|
||||
### S3 Provider Changes
|
||||
|
||||
Extend `S3StorageProvider` to optionally hold a `parentAppKey`:
|
||||
|
||||
```typescript
|
||||
constructor(
|
||||
private s3Bucket: string,
|
||||
private appKey: string,
|
||||
private region: string = 'us-east-2',
|
||||
private parentAppKey?: string,
|
||||
) { ... }
|
||||
|
||||
private getS3Key(bucket: string, key: string): string {
|
||||
return `${this.appKey}/${bucket}/${key}`;
|
||||
}
|
||||
private getParentS3Key(bucket: string, key: string): string | null {
|
||||
return this.parentAppKey ? `${this.parentAppKey}/${bucket}/${key}` : null;
|
||||
}
|
||||
```
|
||||
|
||||
Read methods (`getObject`, `headObject`, `getObjectStream`, `getDownloadStrategy`) get a single new helper:
|
||||
|
||||
```typescript
|
||||
private async withFallback<T>(primary: () => Promise<T | null>, parent: () => Promise<T | null>): Promise<T | null> {
|
||||
const a = await primary();
|
||||
if (a !== null) return a;
|
||||
if (!this.parentAppKey) return null;
|
||||
return parent();
|
||||
}
|
||||
```
|
||||
|
||||
Each read becomes:
|
||||
- Attempt with `getS3Key(bucket, key)`. On null/404 → if `parentAppKey` set, attempt with `getParentS3Key(bucket, key)`.
|
||||
|
||||
Write methods (`putObject`, `deleteObject`, `createMultipartUpload`, etc.) do **not** call the fallback — they always target `getS3Key`.
|
||||
|
||||
### Presigned URLs
|
||||
|
||||
`getDownloadStrategy(bucket, key)` is the only read path that doesn't actually fetch the object — it just constructs a signed URL. For fallback to work for presigned URLs, we must do a HEAD against the branch path first; if 404, sign against the parent path. This adds a HEAD round-trip but is required for correctness. Non-404 HEAD failures (network, IAM, throttling) are caught and logged: URL generation falls back to the branch key rather than aborting the whole call. If the object truly only lives on the parent path, the signed URL will 404 at download time — degraded but recoverable, and a strict improvement over failing the entire request.
|
||||
|
||||
### Service Layer
|
||||
|
||||
`StorageService.getObject` and `objectIsVisible` rely on a metadata row in `storage.objects`. For `mode='full'` branches, that row exists (copied via pg_dump). The provider does the actual S3 fallback; service layer needs no change.
|
||||
|
||||
For `mode='schema-only'` branches, `storage.objects` is truncated by the cloud-backend after restore. Result: branch users get 404 from RLS lookup, never reaching the provider. This is the expected behavior — the storage feature simply doesn't apply.
|
||||
|
||||
### Failure Modes
|
||||
|
||||
- `parentAppKey` set but parent's directory was deleted (branch outlived parent — shouldn't happen given lifecycle cascade): primary 404 + parent 404 → final 404. Same as today.
|
||||
- Parent exists but specific key missing on parent too: same 404. No new error path.
|
||||
- IAM error reading parent path (cross-prefix permission denied): non-404 errors are propagated by the read helpers (`tryHeadObject`, `tryGetObjectStream`, `tryGetObject`); only true 404s are treated as not-found and trigger parent fallback. The public `getObject` wraps `withFallback` and surfaces any error as `null` to preserve the prior service-layer contract, but parent fallback is no longer triggered by transient/IAM failures. The IAM role for the EC2 already has access to the entire `insforge-storage` bucket per the existing single-bucket model; no permissions change required.
|
||||
|
||||
### Compatibility
|
||||
|
||||
- Local storage provider: ignored. `LocalStorageProvider` never receives `parentAppKey`. The constructor signature can be extended for symmetry but the fallback path no-ops.
|
||||
- Existing non-branch projects: `PARENT_APP_KEY` is unset → fallback path never executes → zero behavior change.
|
||||
|
||||
## API
|
||||
|
||||
No new HTTP endpoints. Existing endpoints' behavior changes only for branches with `PARENT_APP_KEY` set, and only for read paths.
|
||||
|
||||
## Acceptance
|
||||
|
||||
A branch container started with `APP_KEY=branchkey` and `PARENT_APP_KEY=parentkey`:
|
||||
1. `GET /api/storage/buckets/foo/objects/path/to/file.jpg` returns the file from `parentkey/foo/path/to/file.jpg` if `branchkey/foo/path/to/file.jpg` doesn't exist (assuming branch DB has the metadata row).
|
||||
2. `PUT /api/storage/buckets/foo/objects/path/to/file.jpg` writes to `branchkey/foo/path/to/file.jpg` only.
|
||||
3. After (2), step (1) returns the new branch-local file (branch path takes priority).
|
||||
4. `DELETE /api/storage/buckets/foo/objects/path/to/file.jpg` (after 2) removes the branch's copy. Subsequent GET falls back to parent file again.
|
||||
5. RLS still gates: a user without read access to the bucket gets 404, regardless of fallback.
|
||||
|
||||
## Open Questions / TBD
|
||||
|
||||
1. **Deletion markers (post-v1).** A branch may want to "hide" a parent file. Requires a sentinel object or a row in branch's metadata. Not in v1.
|
||||
2. **CoW write-through.** When a branch writes to `path/to/file.jpg` after reading the parent's version, should we copy on first write? Today writing creates a fresh file at the branch path; reads naturally see the new file. No copy needed unless we want diff/merge of file contents (not in product scope).
|
||||
3. **Schema-only branch + storage**. Confirm with cloud-backend team: should schema-only mode skip truncating `storage.objects`, so storage fallback works in that mode too? Current spec assumes truncate (matching "fresh start"). Worth a one-line check.
|
||||
@@ -0,0 +1,359 @@
|
||||
# PostHog Analytics — Layout Redesign Design
|
||||
|
||||
**Status**: Draft
|
||||
**Date**: 2026-05-24
|
||||
**Repo**: InsForge (OSS dashboard)
|
||||
**Scope**: `packages/dashboard/src/features/analytics/*` + sidebar nav + routing
|
||||
**Design refs** (`Figma`):
|
||||
- Disconnected overview — `3177-62515`
|
||||
- Traffic — `3174-54062`, `3177-59015`, `3177-59464`
|
||||
- User Retention — `3177-54743`
|
||||
- Session Replay — `3181-10216`
|
||||
- Settings (Analytics Config modal) — `3190-68392`, `3190-68947`
|
||||
|
||||
---
|
||||
|
||||
## Problem
|
||||
|
||||
`AnalyticsPage` is a single stacked page. Disconnected → centered `EmptyConnectPanel`. Connected → top action bar + ConnectStatusBar + Setup-with-Prompt + ApiKeyCard + KPI + 3 Breakdowns + Retention card + Recent Replays card, all in one vertical scroll.
|
||||
|
||||
The redesign splits the feature into a **secondary-sidebar pattern** matching how Authentication / Payments / Realtime / Deployments are already organized in the dashboard:
|
||||
|
||||
- Left secondary sidebar titled **Analytics** with sub-items **Traffic / User Retention / Session Replay**
|
||||
- A gear icon in the sidebar header opens an **Analytics Config** modal (connection info + setup prompt + disconnect)
|
||||
- When PostHog is **not connected**: sub-items are disabled, the gear is disabled, the main area shows a single empty-connect CTA panel
|
||||
- When connected: each sub-item routes to a focused dashboard page
|
||||
|
||||
Goal: implement that layout using existing InsForge components (`FeatureSidebar`, `MenuDialog`, `Button`, etc.) and existing analytics building blocks (`KpiSectionWithTrend`, `BreakdownPanel`, `RetentionCard`, `RecentReplaysCard`). No hard-coded colors / spacing — Tailwind semantic classes + tokens only.
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- New data fetching / new API endpoints — sub-pages reuse existing hooks (`useWebOverview`, `useTrend`, `useRetention`, `useRecordings`, `usePosthogConnection`).
|
||||
- Changing the Connect / OAuth flow itself — `onConnectPosthog` from `useDashboardHost` stays the contract.
|
||||
- Self-host parity — `isCloudHosting` gate in `AppRoutes` stays as-is; this feature is cloud-only.
|
||||
- Feature flagging / phased roll-out — staging handled via OSS test tags.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Routes (`router/AppRoutes.tsx`)
|
||||
|
||||
Today (single route):
|
||||
```tsx
|
||||
{isCloudHosting && <Route path="/dashboard/analytics" element={<AnalyticsPage />} />}
|
||||
```
|
||||
|
||||
After (layout + nested routes, mirrors `/dashboard/authentication` and `/dashboard/payments`):
|
||||
```tsx
|
||||
{isCloudHosting && (
|
||||
<Route path="/dashboard/analytics" element={<AnalyticsLayout />}>
|
||||
<Route index element={<Navigate to="traffic" replace />} />
|
||||
<Route path="traffic" element={<TrafficPage />} />
|
||||
<Route path="retention" element={<RetentionPage />} />
|
||||
<Route path="session-replay" element={<SessionReplayPage />} />
|
||||
</Route>
|
||||
)}
|
||||
```
|
||||
|
||||
The bare `/dashboard/analytics` always redirects to `/traffic` (matches `payments → catalog`, `realtime → channels`). Disconnected state is handled inside `AnalyticsLayout`, not by the router — see below.
|
||||
|
||||
### Components
|
||||
|
||||
Mirrors `features/auth`, `features/payments`, `features/realtime`, `features/deployments` — Layout / Sidebar / feature-level dialog modals all live under `components/`, sub-pages under `pages/`, default-exported Layout imported by `AppRoutes`:
|
||||
|
||||
```
|
||||
features/analytics/
|
||||
├── components/
|
||||
│ ├── AnalyticsLayout.tsx NEW — outer shell (sidebar + Outlet | EmptyConnectPanel). DEFAULT EXPORT.
|
||||
│ ├── AnalyticsSidebar.tsx NEW — wraps FeatureSidebar, owns Settings dialog open state. Named export.
|
||||
│ ├── AnalyticsConfigDialog.tsx NEW — MenuDialog: connection info + setup prompt + disconnect. Named export.
|
||||
│ └── posthog/ (existing — kept)
|
||||
│ ├── EmptyConnectPanel.tsx (reused as-is for disconnected main area)
|
||||
│ ├── ApiKeyCard.tsx (content folded into AnalyticsConfigDialog; file deleted)
|
||||
│ ├── ConnectStatusBar.tsx (content folded into AnalyticsConfigDialog; file deleted)
|
||||
│ ├── DisconnectDialog.tsx (kept — triggered from inside AnalyticsConfigDialog)
|
||||
│ ├── KpiSectionWithTrend.tsx (used by TrafficPage)
|
||||
│ ├── BreakdownPanel.tsx (used by TrafficPage)
|
||||
│ ├── RetentionCard.tsx (used by RetentionPage, full-width)
|
||||
│ ├── RecentReplaysCard.tsx (used by SessionReplayPage)
|
||||
│ ├── ReplayModal.tsx (unchanged — opened from SessionReplayPage)
|
||||
│ └── TimeRangeSelector.tsx (rendered in each sub-page's header)
|
||||
├── pages/ NEW directory (matches features/auth/pages)
|
||||
│ ├── TrafficPage.tsx NEW — named export
|
||||
│ ├── RetentionPage.tsx NEW — named export
|
||||
│ └── SessionReplayPage.tsx NEW — named export
|
||||
├── AnalyticsPage.tsx DELETE (logic split across Layout + pages)
|
||||
├── index.ts UPDATE — drop `export { AnalyticsPage }` (router imports layout directly, matching realtime/payments which have no barrel)
|
||||
├── context/TimeRangeContext.tsx (kept — provider moves into AnalyticsLayout)
|
||||
├── hooks/ (kept — no changes; useRecordings may grow limit/offset parameters)
|
||||
├── lib/ (kept)
|
||||
└── services/ (kept)
|
||||
```
|
||||
|
||||
`AppRoutes.tsx` import follows the existing convention:
|
||||
```ts
|
||||
import AnalyticsLayout from '#features/analytics/components/AnalyticsLayout';
|
||||
import { TrafficPage } from '#features/analytics/pages/TrafficPage';
|
||||
import { RetentionPage } from '#features/analytics/pages/RetentionPage';
|
||||
import { SessionReplayPage } from '#features/analytics/pages/SessionReplayPage';
|
||||
```
|
||||
|
||||
### Shared component change
|
||||
|
||||
`components/FeatureSidebar.tsx` needs a `disabled` flag on items. Today `FeatureSidebarListItem` only has `id / label / href / sectionEnd / onClick`. Add:
|
||||
|
||||
```ts
|
||||
export interface FeatureSidebarListItem {
|
||||
id: string;
|
||||
label: string;
|
||||
href?: string;
|
||||
sectionEnd?: boolean;
|
||||
onClick?: () => void;
|
||||
disabled?: boolean; // NEW
|
||||
}
|
||||
```
|
||||
|
||||
When `disabled === true`:
|
||||
- The row renders as a `<div>` (not `<Link>`), so clicking is a no-op and there's no router navigation.
|
||||
- `aria-disabled="true"` on the container.
|
||||
- Visual: `text-muted-foreground/50 cursor-not-allowed` and no hover treatment (no `hover:bg-alpha-4` / `hover:text-foreground`).
|
||||
- The hidden `useMatch` result is ignored — disabled items never appear "selected".
|
||||
|
||||
`headerButtons` already supports `disabled`, which is what we use for the gear-icon Settings button.
|
||||
|
||||
Auth / Payments / Realtime / Deployments sidebars never pass `disabled` today, so adding the optional field is backwards-compatible.
|
||||
|
||||
---
|
||||
|
||||
## Behavior
|
||||
|
||||
### `AnalyticsLayout`
|
||||
|
||||
```tsx
|
||||
export default function AnalyticsLayout() {
|
||||
const conn = usePosthogConnection();
|
||||
const { projectId, isLoading: projectIdLoading, error: projectIdError } = useProjectId();
|
||||
|
||||
const connected = !conn.isLoading && !conn.isError && !!conn.data;
|
||||
|
||||
return (
|
||||
<TimeRangeProvider>
|
||||
<div className="flex h-full min-h-0 overflow-hidden bg-[rgb(var(--semantic-1))]">
|
||||
<AnalyticsSidebar connected={connected} />
|
||||
<div className="min-w-0 flex-1 overflow-hidden">
|
||||
{conn.isLoading || projectIdLoading ? (
|
||||
<LoadingState />
|
||||
) : conn.isError ? (
|
||||
<ErrorState message="Failed to load PostHog connection." />
|
||||
) : !conn.data ? (
|
||||
projectIdError || !projectId
|
||||
? <ErrorState message="Failed to load project ID." />
|
||||
: <DisconnectedMain projectId={projectId} />
|
||||
) : (
|
||||
<Outlet />
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</TimeRangeProvider>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- `LoadingState` / `ErrorState` are existing `#components` exports.
|
||||
- `DisconnectedMain` is a thin wrapper that centers `EmptyConnectPanel` with appropriate page padding — same CTA, same copy as today (`Connect PostHog`, `One-click setup of a PostHog project for product analytics.`).
|
||||
- When disconnected, `<Outlet />` is not rendered — so even direct navigation to `/dashboard/analytics/retention` still shows the empty-connect panel. URL stays correct; on connect, React-Query invalidation flips `connected` to `true` and the sub-page mounts.
|
||||
|
||||
### `AnalyticsSidebar`
|
||||
|
||||
```tsx
|
||||
const ITEMS = (connected: boolean): FeatureSidebarListItem[] => [
|
||||
{ id: 'traffic', label: 'Traffic', href: '/dashboard/analytics/traffic', disabled: !connected },
|
||||
{ id: 'retention', label: 'User Retention', href: '/dashboard/analytics/retention', disabled: !connected },
|
||||
{ id: 'session-replay', label: 'Session Replay', href: '/dashboard/analytics/session-replay', disabled: !connected },
|
||||
];
|
||||
|
||||
export function AnalyticsSidebar({ connected }: { connected: boolean }) {
|
||||
const [settingsOpen, setSettingsOpen] = useState(false);
|
||||
return (
|
||||
<>
|
||||
<FeatureSidebar
|
||||
title="Analytics"
|
||||
items={ITEMS(connected)}
|
||||
headerButtons={[{
|
||||
id: 'analytics-settings',
|
||||
label: 'Analytics Config',
|
||||
icon: Settings,
|
||||
onClick: () => setSettingsOpen(true),
|
||||
disabled: !connected,
|
||||
}]}
|
||||
/>
|
||||
<AnalyticsConfigDialog open={settingsOpen} onOpenChange={setSettingsOpen} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
Pattern is identical to `AuthenticationSidebar`. No new sidebar styling — disabled visual lives inside `FeatureSidebar`.
|
||||
|
||||
### `AnalyticsConfigDialog`
|
||||
|
||||
Built on the same `MenuDialog` primitives `AuthSettingsMenuDialog` uses. design node `3190-68392` shows the modal opened against the disconnected layout (rare — gear is disabled when disconnected, so this is more of a design reference), and `3190-68947` shows it against the connected layout. Both share the same modal structure — one side-nav section, `Connection` content on the right — so a single `AnalyticsConfigDialog` covers both nodes. We keep `MenuDialogSideNav` for visual parity even though there's only one section today:
|
||||
|
||||
```tsx
|
||||
<MenuDialog open={open} onOpenChange={onOpenChange}>
|
||||
<MenuDialogContent>
|
||||
<MenuDialogSideNav>
|
||||
<MenuDialogSideNavHeader>
|
||||
<MenuDialogSideNavTitle>Analytics Config</MenuDialogSideNavTitle>
|
||||
</MenuDialogSideNavHeader>
|
||||
<MenuDialogNav>
|
||||
<MenuDialogNavList>
|
||||
<MenuDialogNavItem active>Connection</MenuDialogNavItem>
|
||||
</MenuDialogNavList>
|
||||
</MenuDialogNav>
|
||||
</MenuDialogSideNav>
|
||||
<MenuDialogMain>
|
||||
<MenuDialogHeader>
|
||||
<MenuDialogTitle>Connection</MenuDialogTitle>
|
||||
<MenuDialogCloseButton />
|
||||
</MenuDialogHeader>
|
||||
<MenuDialogBody>
|
||||
{/* 1. ConnectStatusBar (compact form) */}
|
||||
{/* 2. Read-only Input fields: API host, Project ID, API key (with show/hide) */}
|
||||
{/* 3. Setup-with-Prompt block (existing inline copy, no new shared component per memory) */}
|
||||
</MenuDialogBody>
|
||||
<MenuDialogFooter>
|
||||
<Button variant="ghost" onClick={() => setDisconnecting(true)}>Disconnect</Button>
|
||||
</MenuDialogFooter>
|
||||
</MenuDialogMain>
|
||||
</MenuDialogContent>
|
||||
</MenuDialog>
|
||||
```
|
||||
|
||||
`Disconnect` opens the existing `DisconnectDialog` (modal stacking is fine — `AuthSettingsMenuDialog` already nests confirm modals the same way).
|
||||
|
||||
Fields:
|
||||
- **API host** — `Input` (read-only), copy button
|
||||
- **Project ID** — `Input` (read-only), copy button
|
||||
- **API key** — `Input` (read-only, type=password by default), show/hide toggle, copy button. This is the existing `ApiKeyCard` content lifted into the dialog row layout.
|
||||
|
||||
The "Setup with Prompt" block in current `AnalyticsPage.tsx` (lines 121–137) is placed inline verbatim into `MenuDialogBody` — per `[[feedback_inline_prompt_block_pattern]]`, do not extract it into a shared component.
|
||||
|
||||
### Sub-pages
|
||||
|
||||
Each sub-page renders a page header (title + `TimeRangeSelector` where the underlying data hook respects the range) + its content. No outer `<h1>Analytics</h1>` (the sidebar title already says it).
|
||||
|
||||
**`TrafficPage`** — design node `3174-54062 / 3177-59015 / 3177-59464`:
|
||||
```
|
||||
[ header: Traffic [ TimeRangeSelector ] ]
|
||||
[ KpiSectionWithTrend enabled ]
|
||||
[ grid: BreakdownPanel(Page) | BreakdownPanel(Country) | BreakdownPanel(DeviceType) ]
|
||||
```
|
||||
Keep the existing `grid grid-cols-1 md:grid-cols-3 gap-3` from today's `AnalyticsPage` for the 3 breakdown panels — the design shows the row of 3 side-by-side at standard widths. KPI section spans the full width above.
|
||||
|
||||
Node `3174-54062` is the connect-success toast state (`Analytics setup succeeded`) — this surface is rendered by the global toast (`useToast`), already wired in `AnalyticsPage` today. We move that `subscribePosthogConnectionStatus` `useEffect` into `AnalyticsLayout` so the toast still appears when the user is on any sub-route during connect completion.
|
||||
|
||||
**`RetentionPage`** — design node `3177-54743`:
|
||||
```
|
||||
[ header: User Retention [ TimeRangeSelector ] ]
|
||||
[ RetentionCard enabled, full-width ]
|
||||
```
|
||||
`RetentionCard` currently sits inside a multi-section scroll — moving it to its own page gives it the full content width per the design.
|
||||
|
||||
**`SessionReplayPage`** — design node `3181-10216`:
|
||||
```
|
||||
[ header: Session Replay ]
|
||||
[ RecentReplaysCard enabled, full-width, paginated ]
|
||||
```
|
||||
|
||||
Note: no `TimeRangeSelector` here — `useRecordings` only takes `limit` and doesn't read the time-range context, so showing the selector would be misleading. If the recordings backend later grows time-range filtering, wire `useRecordings` to `TimeRangeContext` and add the selector back.
|
||||
The design shows pagination at the bottom of the list. `RecentReplaysCard` today returns ~N recent replays; this redesign asks it to paginate. Use `Pagination` from `@insforge/ui`. Pagination logic is a small extension to `useRecordings` (`limit` + `offset`) — covered in the implementation plan.
|
||||
|
||||
### `Info` notice about Web Analytics lag
|
||||
|
||||
Currently rendered between `ApiKeyCard` and `KpiSectionWithTrend` in `AnalyticsPage`. Keep it visible to users — move it to the **top of TrafficPage** (where the KPI data lives), since that's the surface where the lag would be observed. Same `Info` icon + copy.
|
||||
|
||||
---
|
||||
|
||||
## Styling (tokens only)
|
||||
|
||||
Everything maps to existing Tailwind semantic classes already used by Auth/Payments/Realtime:
|
||||
|
||||
| Design element | Token / class |
|
||||
|---------------------|---------------------------------------------------------------------------|
|
||||
| Sidebar background | `bg-semantic-1` (already on `FeatureSidebar`) |
|
||||
| Sidebar border | `border-[var(--alpha-8)]` (already on `FeatureSidebar`) |
|
||||
| Active item | `bg-alpha-8 text-foreground` (already in `FeatureSidebarItemRow`) |
|
||||
| Hover | `hover:bg-alpha-4 hover:text-foreground` (already) |
|
||||
| Disabled item | `text-muted-foreground/50 cursor-not-allowed` (NEW branch in `FeatureSidebar`) |
|
||||
| Page background | `bg-[rgb(var(--semantic-1))]` (matches `AuthenticationLayout`) |
|
||||
| Card background | `bg-card` |
|
||||
| Card border | `border-[var(--alpha-8)]` |
|
||||
| Foreground text | `text-foreground` / `text-muted-foreground` |
|
||||
| Modal overlay/chrome | inherited from `MenuDialog` primitives — nothing to set per-instance |
|
||||
|
||||
No hex values, no `bg-[#1B1B1B]`, no inline `rgba()`.
|
||||
|
||||
---
|
||||
|
||||
## Risks / Edge Cases
|
||||
|
||||
1. **Direct deep-link to a sub-route while disconnected.** Handled — `AnalyticsLayout` swaps in `EmptyConnectPanel` regardless of which child route matches. URL keeps the user's intended destination so post-connect they see the right page after a single React-Query re-fetch.
|
||||
|
||||
2. **Connection state flipping mid-session** (cloud completes OAuth in another tab). `usePosthogConnection` already wires React-Query invalidation via `subscribePosthogConnectionStatus`. We move that subscription from `AnalyticsPage` into `AnalyticsLayout` so it's active on every sub-route.
|
||||
|
||||
3. **`projectId` resolves after `conn.data`.** Current code holds the connected view until `projectId` is available. Layout preserves that — it renders `LoadingState` until both resolve.
|
||||
|
||||
4. **`onConnectPosthog` undefined** (host doesn't provide it). `EmptyConnectPanel` already disables the button in that case; no change needed.
|
||||
|
||||
5. **Pagination behavior on Session Replay.** New `limit/offset` parameters on `useRecordings`. If the current backend endpoint only returns a fixed page, the implementation plan should confirm pagination parameters are supported before wiring `Pagination`. Fallback: keep showing the top-N list without pagination, hide the control.
|
||||
|
||||
6. **`MenuDialog` z-index vs `DisconnectDialog`.** Auth already stacks `Dialog`-on-`MenuDialog` (Mail provider config + confirms). The same Radix portal stacking applies — no new work.
|
||||
|
||||
---
|
||||
|
||||
## File-Level Change Summary
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `packages/dashboard/src/components/FeatureSidebar.tsx` | Add `disabled?: boolean` to `FeatureSidebarListItem`; render disabled variant in `FeatureSidebarItemRow` (no `<Link>`, `aria-disabled`, muted tokens, no hover). |
|
||||
| `packages/dashboard/src/features/analytics/components/AnalyticsLayout.tsx` | NEW (default export) — sidebar + Outlet, owns `usePosthogConnection` + `TimeRangeProvider` + connect-status `useEffect`, falls back to `EmptyConnectPanel` when disconnected. |
|
||||
| `packages/dashboard/src/features/analytics/components/AnalyticsSidebar.tsx` | NEW (named export) — wraps `FeatureSidebar` with title `Analytics`, 3 sub-items, Settings header button. |
|
||||
| `packages/dashboard/src/features/analytics/components/AnalyticsConfigDialog.tsx` | NEW (named export) — `MenuDialog` with `Connection` section: read-only host / project ID / API key inputs, Setup-with-Prompt block, Disconnect button. |
|
||||
| `packages/dashboard/src/features/analytics/pages/TrafficPage.tsx` | NEW — header (`Traffic` + TimeRangeSelector) + lag `Info` + `KpiSectionWithTrend` + 3 `BreakdownPanel`s. |
|
||||
| `packages/dashboard/src/features/analytics/pages/RetentionPage.tsx` | NEW — header + `RetentionCard`. |
|
||||
| `packages/dashboard/src/features/analytics/pages/SessionReplayPage.tsx` | NEW — header + `RecentReplaysCard` + `Pagination`. |
|
||||
| `packages/dashboard/src/features/analytics/AnalyticsPage.tsx` | DELETE — superseded. |
|
||||
| `packages/dashboard/src/features/analytics/components/posthog/ApiKeyCard.tsx` | DELETE — content lifted into `AnalyticsConfigDialog`. |
|
||||
| `packages/dashboard/src/features/analytics/components/posthog/ConnectStatusBar.tsx` | DELETE — content lifted into `AnalyticsConfigDialog`. |
|
||||
| `packages/dashboard/src/features/analytics/index.ts` | Replace `export { AnalyticsPage }` with no-op (delete file) or empty — router imports Layout directly. Match `realtime` / `payments` which have no barrel. |
|
||||
| `packages/dashboard/src/router/AppRoutes.tsx` | Replace single `<Route path="/dashboard/analytics" element={<AnalyticsPage />} />` with `<Route path="/dashboard/analytics" element={<AnalyticsLayout />}>` + nested `index → Navigate to traffic`, `/traffic`, `/retention`, `/session-replay`. Drop the `import { AnalyticsPage } from '#features/analytics'` line, add new imports per the new convention. |
|
||||
| `packages/dashboard/src/features/analytics/hooks/useRecordings.ts` | Extend with `limit` / `offset` parameters for pagination (verify backend support in implementation plan; fallback to top-N list if not). |
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
1. `/dashboard/analytics` (cloud) — disconnected:
|
||||
- Left sidebar titled **Analytics** with 3 disabled sub-items and a disabled gear icon
|
||||
- Main area shows the `Connect PostHog` empty state (centered, single CTA)
|
||||
- Clicking a disabled sub-item does nothing; URL does not change
|
||||
2. `/dashboard/analytics` — connected:
|
||||
- Auto-redirects to `/dashboard/analytics/traffic`
|
||||
- All 3 sub-items enabled, gear icon enabled
|
||||
3. `/dashboard/analytics/traffic` — connected:
|
||||
- Page header with title + time range selector
|
||||
- Web Analytics lag `Info` notice above the KPI row
|
||||
- KPI row + 3 breakdown sections
|
||||
4. `/dashboard/analytics/retention` — `RetentionCard` full-width
|
||||
5. `/dashboard/analytics/session-replay` — `RecentReplaysCard` full-width + bottom `Pagination`
|
||||
6. Gear icon opens **Analytics Config** modal:
|
||||
- Read-only host / project ID / API key inputs (key hidden by default with show toggle)
|
||||
- Setup-with-Prompt copyable block
|
||||
- Disconnect button → existing `DisconnectDialog` confirm flow
|
||||
7. Disconnecting from inside the modal returns the user to the disconnected layout (sub-items disabled, empty CTA).
|
||||
8. No hex values / inline `rgba()` added anywhere — every color/border/spacing is a Tailwind semantic class or existing CSS var.
|
||||
9. `isCloudHosting` route gate behavior unchanged — self-hosted dashboards still don't show Analytics.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Compute Container Logs — Design
|
||||
|
||||
**Date:** 2026-06-04
|
||||
**Priority:** Dashboard-first — click into a compute service and see its container stdout/stderr.
|
||||
|
||||
## Problem
|
||||
|
||||
Compute services (containers on Fly.io) expose only **machine lifecycle events**
|
||||
(`ServiceEvents` panel / `compute events` — start/stop/exit/restart). There is no way to
|
||||
see container **stdout/stderr** ("application logs") from the dashboard or CLI. This adds it.
|
||||
|
||||
## Verified foundation
|
||||
|
||||
Fly exposes an HTTP REST logs endpoint (the same one `flyctl logs` and Fly's dashboard use).
|
||||
Tested live against a throwaway Fly container on 2026-06-04:
|
||||
|
||||
```text
|
||||
GET https://api.fly.io/api/v1/apps/{app}/logs
|
||||
Auth: Authorization: FlyV1 <org-macaroon> # NOT "Bearer"
|
||||
Query: instance=<machineId> next_token=<ns-unix-cursor>
|
||||
Returns: HTTP 200, data[].attributes { timestamp(RFC3339), message, instance, region }
|
||||
+ meta.next_token cursor
|
||||
History: ~7-day retention (populates the view on open); re-poll with next_token to tail
|
||||
```
|
||||
|
||||
### Gotchas (both load-bearing, both hit during implementation)
|
||||
|
||||
1. **Auth scheme.** The existing providers call `api.machines.dev` with `Bearer`. The logs
|
||||
endpoint is on `api.fly.io` and **rejects `Bearer` with 401** — it requires the Fly
|
||||
macaroon scheme `Authorization: FlyV1 <token>`. Same token, different host + prefix.
|
||||
2. **`next_token` precision.** It is a nanosecond Unix timestamp that exceeds
|
||||
`Number.MAX_SAFE_INTEGER`. `JSON.parse()` silently rounds it and corrupts the cursor, so
|
||||
the provider extracts it from the raw response text to preserve every digit.
|
||||
|
||||
## Auth boundary (org token never leaves the backend)
|
||||
|
||||
```text
|
||||
CLI ──Bearer ik_<project key>──┐
|
||||
Dashboard ──user access token────────┤──▶ Backend ──FlyV1 <org token>──▶ Fly
|
||||
(browser) (withAccessToken) ┘ (sole holder of the org token)
|
||||
```
|
||||
|
||||
Neither the CLI nor the dashboard ever holds the Fly org token. The backend authorizes the
|
||||
request (project ownership), then calls Fly. Identical boundary to the existing `events` route.
|
||||
|
||||
## What shipped in this PR (monorepo: `insforge-current-main`)
|
||||
|
||||
Mirrors the `events` feature end to end. New surface is one provider method (`getLogs`) feeding
|
||||
a route, a service method, and the dashboard panel.
|
||||
|
||||
- `shared-schemas`: `computeLogLineSchema` / `computeLogsResponseSchema` (+ types).
|
||||
- `compute.provider.ts`: `ComputeLogLine` / `ComputeLogsResult` + `getLogs()` on the interface.
|
||||
- `fly.provider.ts`: `getLogs()` — `api.fly.io` + `FlyV1`, `instance` scoping, RFC3339→epoch-ms
|
||||
normalization, precision-safe `next_token`. (Self-host path; unit-tested.)
|
||||
- `cloud.provider.ts`: `getLogs()` — delegates `GET /machines/:id/logs` to the control plane.
|
||||
- `services.service.ts`: `getServiceLogs()` (ownership guard, mirrors `getServiceEvents`).
|
||||
- `services.routes.ts`: `GET /:id/logs` (`verifyAdmin`, project-ownership, `limit`, `next_token`).
|
||||
- Dashboard: `computeServicesApi.logs()`, `useServiceLogs` hook, `ServiceLogs.tsx` (recent on
|
||||
open + **Live** toggle re-polling every 2s), mounted in the service detail view beside Events.
|
||||
|
||||
Live tail in v1 = re-fetch the recent window on an interval (stateless; no cursor
|
||||
accumulation/dedup to get wrong). SSE is a possible later upgrade.
|
||||
|
||||
## Companion PR (separate repo: `insforge-cloud-backend`)
|
||||
|
||||
For InsForge Cloud, the dashboard runs through `CloudComputeProvider`, which delegates to the
|
||||
cloud control plane. That backend needs the matching `GET …/machines/:id/logs` route (its
|
||||
own `fly-client.getLogs` with the same `api.fly.io` + `FlyV1` call). **Self-hosted works
|
||||
fully from this PR alone** (FlyProvider path); cloud needs the companion route to light up.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No Vector / log-shipper / NATS / WireGuard. The REST endpoint is sufficient.
|
||||
- No durable log store / new DB table — relies on Fly's ~7-day retention. A persisted,
|
||||
searchable store is a deferred future phase behind the same surfaces, only if needed.
|
||||
- No CLI in this PR — Phase 2 (`compute logs <id> --follow`), reusing the same route.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Unofficial endpoint.** Fly documents the logs REST endpoint as "stable but not officially
|
||||
supported — flyctl depends on it." Acceptable; the provider degrades to a thrown error (not
|
||||
a crash) on non-200, and the durable-store path is the long-term fallback.
|
||||
Reference in New Issue
Block a user