wehub/insforge--insforge
1
0
Fork 0
Code Issues Pull Requests Actions 6 Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Lint and Format Check / lint-and-format (push) Failing after 0s
Details
Check Migrations / Check for duplicate migration numbers (push) Failing after 1s
Details
CI Pre-merge Check / CI Pre-merge Check (push) Failing after 2m17s
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:23:40 +08:00
commit 3a28426bf4
1399 changed files with 257375 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/docs/deployment.md
+169
View File
@@ -0,0 +1,169 @@
# InsForge Deployment - Agent Documentation
## Overview
Deploy frontend applications to InsForge with the `create-deployment` MCP tool. The tool uploads the source directory through InsForge's direct deployment flow, starts a Vercel-backed production build, and records the result in `deployments.runs`.
Use the source directory, not a pre-built artifact directory. The backend validates the manifest, streams each file to the provider, verifies file size and SHA-1 digest, then starts the deployment after all files are uploaded.
The REST API still supports the legacy source-zip flow for older clients. Do not use it from agents unless the direct MCP tool is unavailable.
## Deploy With MCP
Call `create-deployment` with:
| Parameter | Required | Description |
|-----------|----------|-------------|
| `sourceDirectory` | Yes | Absolute path to the app source directory, for example `/Users/me/project/frontend`. |
| `projectSettings.buildCommand` | No | Build command, for example `npm run build`. Omit to use provider defaults. |
| `projectSettings.outputDirectory` | No | Static output directory, for example `dist` or `build`. |
| `projectSettings.installCommand` | No | Install command, for example `npm install` or `pnpm install`. |
| `projectSettings.devCommand` | No | Development command metadata for framework-aware builds. |
| `projectSettings.rootDirectory` | No | Root directory inside the uploaded source tree. |
| `envVars` | No | Array of `{ "key": "...", "value": "..." }` variables to create or update before build. |
| `meta` | No | String key-value metadata for provider deployment creation. |
Example:
```json
{
"sourceDirectory": "/Users/me/project/frontend",
"projectSettings": {
"buildCommand": "npm run build",
"outputDirectory": "dist"
},
"envVars": [
{
"key": "VITE_INSFORGE_BASE_URL",
"value": "https://your-project.region.insforge.app"
},
{
"key": "VITE_INSFORGE_ANON_KEY",
"value": "your-anon-key"
}
],
"meta": {
"source": "agent"
}
}
```
Important:
- `sourceDirectory` must be an absolute path.
- Upload app source files, not only `dist`, unless the project is intentionally a plain static site.
- Include framework files needed by Vercel, such as `package.json`, lock file, framework config, and `vercel.json` when needed.
- Prefix browser-exposed variables correctly, for example `VITE_` for Vite and `NEXT_PUBLIC_` for Next.js.
- Do not put service-role keys, admin tokens, or private provider keys in browser-exposed variables.
- Tailwind projects should stay on Tailwind CSS 3.4 unless the app already supports v4.
## What The Tool Does
The current deployment path maps to these backend steps:
1. Register a direct upload manifest in `POST /api/deployments/direct`.
2. Upload each file with `PUT /api/deployments/:id/files/:fileId/content`.
3. Start the provider build with `POST /api/deployments/:id/start`.
4. Poll or sync status until the run reaches `READY`, `ERROR`, or `CANCELED`.
The direct file upload endpoint requires `Content-Type: application/octet-stream`. The backend rejects missing files, size mismatches, SHA mismatches, invalid relative paths, and attempts to start before every file has `uploaded_at`.
## Check Deployment Status
Use the Dashboard, MCP SQL tools, or the REST API. For SQL:
```sql
SELECT id, provider_deployment_id, status, url, metadata, created_at, updated_at
FROM deployments.runs
ORDER BY created_at DESC
LIMIT 5;
```
To inspect an interrupted direct upload:
```sql
SELECT file_path, size_bytes, uploaded_at
FROM deployments.files
WHERE deployment_id = '<deployment-id>'
ORDER BY file_path;
```
If a deployment has a provider ID but the status looks stale, sync it through the API:
```http
POST /api/deployments/<deployment-id>/sync
```
## Status Values
| Status | Description |
|--------|-------------|
| `WAITING` | Run exists and is waiting for source uploads. |
| `UPLOADING` | Source files are uploading or provider deployment creation is in progress. |
| `QUEUED` | Provider accepted the deployment and queued the build. |
| `BUILDING` | Provider is building the app. |
| `READY` | Deployment completed. The `url` column contains the live URL. |
| `ERROR` | Upload, build, provider, or webhook processing failed. |
| `CANCELED` | Deployment was canceled. |
## Get The Live URL
After status is `READY`, read the URL:
```sql
SELECT url
FROM deployments.runs
WHERE id = '<deployment-id>';
```
The Dashboard may show a friendlier domain when a custom `.insforge.site` slug or user-owned domain is configured.
## Environment Variables
You can pass `envVars` during deployment or manage them from the Dashboard. Deployment env vars are Vercel encrypted environment variables, not database rows.
Rules:
- Lists show keys and metadata only. Values are hidden unless a single variable is fetched for editing.
- Duplicate keys in one create-or-update request are rejected.
- Values are applied to `production`, `preview`, and `development` targets.
- Use public prefixes only for values that browser code may read.
## SPA Routing
For React, Vue, Svelte, or other single-page apps that need client-side routing, add `vercel.json` at the app root:
```json
{
"rewrites": [
{
"source": "/(.*)",
"destination": "/index.html"
}
]
}
```
Do not add this rewrite to frameworks that already own routing, such as Next.js.
## Common Failures
| Symptom | Check |
|---------|-------|
| Deployment service is not configured | Self-hosted backend needs `VERCEL_TOKEN`, `VERCEL_TEAM_ID`, and `VERCEL_PROJECT_ID`. |
| Upload fails with content type error | Direct file upload must use `application/octet-stream`. |
| Upload fails with size or SHA mismatch | Recompute the manifest from the exact bytes being uploaded. |
| Start fails before provider build | Query `deployments.files` and confirm every row has `uploaded_at`. |
| Build cannot find scripts | Confirm `package.json`, lock file, root directory, and build command. |
| SPA route returns 404 | Add the `vercel.json` rewrite for static SPA apps. |
| Provider rate limit | Retry the failed upload/start step with a fresh request. |
## Quick Reference
| Task | Preferred tool |
|------|----------------|
| Deploy app | `create-deployment` MCP tool |
| Check latest runs | Dashboard or `SELECT * FROM deployments.runs ORDER BY created_at DESC` |
| Inspect direct file upload state | `SELECT * FROM deployments.files WHERE deployment_id = '...'` |
| Manage env vars | Dashboard Sites -> Environment Variables |
| Manage domains | Dashboard Sites -> Domains |
.agents/docs/insforge-instructions-sdk.md
+131
View File
@@ -0,0 +1,131 @@
# InsForge SDK Documentation - Overview
## What is InsForge?
Backend-as-a-service (BaaS) platform providing:
- **Database**: PostgreSQL with PostgREST API
- **Authentication**: Email/password + OAuth (Google, GitHub)
- **Storage**: File upload/download
- **AI**: OpenRouter key provisioning and model catalog for direct OpenAI-compatible integrations
- **Functions**: Serverless function deployment
- **Realtime**: WebSocket pub/sub (database + client events)
## Installation
The following is a step-by-step guide to installing and using the InsForge TypeScript SDK for Web applications. If you are building other types of applications, please refer to:
- [Swift SDK documentation](/sdks/swift/overview) for iOS, macOS, tvOS, and watchOS applications.
- [Kotlin SDK documentation](/sdks/kotlin/overview) for Android applications.
- [REST API documentation](/sdks/rest/overview) for direct HTTP API access.
### 🚨 CRITICAL: Follow these steps in order
### Step 1: Download Template
Use the `download-template` MCP tool to create a new project with your backend URL and anon key pre-configured.
### Step 2: Install SDK
```bash
npm install @insforge/sdk@latest
```
### Step 3: Create SDK Client
You must create a client instance using `createClient()` with your base URL and anon key:
```javascript
import { createClient } from '@insforge/sdk';
const client = createClient({
baseUrl: 'https://your-app.region.insforge.app', // Your InsForge backend URL
anonKey: 'your-anon-key-here' // Get this from backend metadata
});
```
**API BASE URL**: Your API base URL is `https://your-app.region.insforge.app`.
## Getting Detailed Documentation
### 🚨 CRITICAL: Always Fetch Documentation Before Writing Code
InsForge provides official SDKs and REST APIs, use them to interact with InsForge services from your application code.
- [TypeScript SDK](/sdks/typescript/overview) - JavaScript/TypeScript
- [Swift SDK](/sdks/swift/overview) - iOS, macOS, tvOS, and watchOS
- [Kotlin SDK](/sdks/kotlin/overview) - Android and Kotlin Multiplatform
- [REST API](/sdks/rest/overview) - Direct HTTP API access
Before writing or editing any InsForge integration code, you **MUST** call the `fetch-docs` or `fetch-sdk-docs` MCP tool to get the latest SDK documentation. This ensures you have accurate, up-to-date implementation patterns.
### Use the InsForge `fetch-docs` MCP tool to get specific SDK documentation:
Available documentation types:
- `"instructions"` - Essential backend setup (START HERE)
- `"real-time"` - Real-time pub/sub (database + client events) via WebSockets
- `"db-sdk-typescript"` - Database operations with TypeScript SDK
- **Authentication** - Choose based on implementation:
- `"auth-sdk-typescript"` - TypeScript SDK methods for custom auth flows
- `"auth-components-react"` - Pre-built auth UI for React+Vite (single-page app)
- `"auth-components-react-router"` - Pre-built auth UI for React(Vite+React Router) (multi-page app)
- `"auth-components-nextjs"` - Pre-built auth UI for Next.js (SSR app)
- `"storage-sdk"` - File storage operations
- `"functions-sdk"` - Serverless functions invocation
- `"ai-integration-sdk"` - AI integration with the provisioned OpenRouter key and OpenAI SDK
- `"deployment"` - Deploy frontend applications via MCP tool
- `"payments"` - Stripe Checkout, Billing Portal, webhook projections, and fulfillment patterns
These docs are mostly for the TypeScript SDK. For other languages, you can also use the `fetch-sdk-docs` MCP tool to get specific documentation.
### Use the InsForge `fetch-sdk-docs` MCP tool to get specific SDK documentation
You can fetch SDK documentation using the `fetch-sdk-docs` MCP tool with a specific feature type and language.
Available feature types:
- `db` - Database operations
- `storage` - File storage operations
- `functions` - Serverless functions invocation
- `auth` - User authentication
- `ai` - AI integration with the provisioned OpenRouter key and OpenAI SDK
- `realtime` - Real-time pub/sub (database + client events) via WebSockets
- `payments` - Stripe Checkout and Billing Portal with webhook-based fulfillment
Available languages:
- `typescript` - JavaScript/TypeScript SDK
- `swift` - Swift SDK (for iOS, macOS, tvOS, and watchOS)
- `kotlin` - Kotlin SDK (for Android and JVM applications)
- `rest-api` - REST API
Payments currently has TypeScript SDK docs only. Use the Payments API reference for non-TypeScript clients.
## When to Use SDK vs MCP Tools
### Always SDK for Application Logic:
- Authentication (register, login, logout, profiles)
- Database CRUD (select, insert, update, delete)
- Storage operations (upload, download files)
- AI integration via the provisioned OpenRouter key with the OpenAI SDK or OpenRouter HTTP API
- Serverless function invocation
- Payments checkout and customer portal session creation
### Use MCP Tools for Infrastructure:
- Project scaffolding (`download-template`) - Download starter templates with InsForge integration
- Backend setup and metadata (`get-backend-metadata`)
- Database schema management (`run-raw-sql`, `get-table-schema`)
- Storage bucket creation (`create-bucket`, `list-buckets`, `delete-bucket`)
- Serverless function deployment (`create-function`, `update-function`, `delete-function`)
- Frontend deployment (`create-deployment`) - Deploy frontend apps to InsForge hosting
## Important Notes
- For auth: use `auth-sdk` for custom UI, or framework-specific components for pre-built UI
- SDK returns `{data, error}` structure for all operations
- Database inserts require array format: `[{...}]`
- Serverless functions have one endpoint and do not support nested route paths
- Storage: Upload files to buckets, store URLs in database
- AI integrations should call OpenRouter directly with `baseURL: "https://openrouter.ai/api/v1"` and a server-side `OPENROUTER_API_KEY`
- **EXTRA IMPORTANT**: Use Tailwind CSS 3.4 (do not upgrade to v4). Lock these dependencies in `package.json`
.agents/docs/payments-razorpay.md
+297
View File
@@ -0,0 +1,297 @@
# InsForge Razorpay Payments - Agent Documentation
## Use Razorpay For
- Razorpay Orders with Razorpay Checkout for one-time payments.
- Razorpay Subscriptions with Razorpay Checkout authorization.
- Razorpay Items and Plans.
- Manual Razorpay webhook setup.
- Cancel, pause, and resume subscription management through backend routes.
Do not use Stripe Checkout, Stripe Prices, or Billing Portal concepts in a Razorpay flow. Razorpay Checkout runs in the app with `checkout.js`; it does not return a hosted Checkout URL.
## Before Coding
1. Use `environment: "test"` unless the user explicitly approves live Razorpay changes.
2. Confirm both Key ID and Key Secret are configured for the target environment.
3. Confirm Items and Plans exist in that same environment for subscription flows.
4. Confirm the Razorpay webhook is manually configured in the Razorpay Dashboard with a public HTTPS URL.
5. Treat Checkout callback verification as immediate authenticity only. Durable fulfillment must come from webhooks.
6. For one-time products, prefer creating Razorpay Items even though Orders can be amount-only. Items keep the catalog visible after sync; Orders are payment attempts.
## Webhook Setup
Razorpay webhooks are manual. Generate or view the webhook URL and secret in Dashboard -> Payments -> Settings -> Webhooks.
Use these backend routes for admin tooling:
```http
GET /api/payments/razorpay/test/webhook
POST /api/payments/razorpay/test/webhook/rotate-secret
```
Recommended active events are the events InsForge handles:
- `payment.authorized`
- `payment.captured`
- `payment.failed`
- `order.paid`
- `invoice.paid`
- `invoice.expired`
- `refund.created`
- `refund.processed`
- `refund.failed`
- `subscription.created`
- `subscription.activated`
- `subscription.charged`
- `subscription.updated`
- `subscription.cancelled`
- `subscription.paused`
- `subscription.resumed`
- `subscription.halted`
- `subscription.completed`
- `subscription.expired`
Razorpay can only deliver webhooks to a public HTTPS URL.
## One-Time Order
Create an app-owned pending order first. Then create the Razorpay Order with the provider-scoped SDK:
```typescript
const { data, error } = await insforge.payments.razorpay.createOrder('test', {
amount: 50000,
currency: 'INR',
receipt: 'order_123',
subject: { type: 'team', id: 'team_123' },
customerEmail: 'buyer@example.com',
notes: { order_id: 'order_123' }
});
if (error) throw error;
```
If a fulfillment trigger reads `notes.order_id`, pass `notes: { order_id: ... }` when creating the Order or Subscription.
Open Razorpay Checkout in the frontend with `data.checkoutOptions`. After Checkout returns `razorpay_order_id`, `razorpay_payment_id`, and `razorpay_signature`, verify through the SDK:
```typescript
await insforge.payments.razorpay.verifyOrder('test', {
orderId: response.razorpay_order_id,
paymentId: response.razorpay_payment_id,
signature: response.razorpay_signature
});
```
Only treat verification as proof that the client return was authentic. Fulfillment should still come from verified Razorpay webhook events.
## Subscription
Razorpay subscriptions use Plans, not Stripe Prices. A Plan is a recurring definition around a Razorpay Item.
Create or sync the plan first, then create the subscription through the provider-scoped SDK:
```typescript
const { data, error } = await insforge.payments.razorpay.createSubscription('test', {
planId: 'plan_123',
totalCount: 12,
subject: { type: 'team', id: 'team_123' },
customerEmail: 'buyer@example.com'
});
if (error) throw error;
```
Open Razorpay Checkout with `data.checkoutOptions.subscription_id`. After Checkout returns the subscription payment signature values, verify through the SDK:
```typescript
await insforge.payments.razorpay.verifySubscription('test', {
subscriptionId: response.razorpay_subscription_id,
paymentId: response.razorpay_payment_id,
signature: response.razorpay_signature
});
```
Manage the subscription through provider-scoped SDK helpers:
```typescript
await insforge.payments.razorpay.cancelSubscription('test', 'sub_123', {
cancelAtCycleEnd: false
});
await insforge.payments.razorpay.pauseSubscription('test', 'sub_123');
await insforge.payments.razorpay.resumeSubscription('test', 'sub_123');
```
Razorpay subscription creation evaluates the caller's `INSERT` policy on `payments.razorpay_subscriptions`. Cancel, pause, and resume evaluate `UPDATE` policies. PostgreSQL also applies `SELECT` policies to rows returned by `INSERT/UPDATE ... RETURNING`, so add matching `SELECT` visibility for the same billing subject when a policy probe needs to return the row. Do not let users submit arbitrary subjects unless the app checks that they can manage that billing subject.
## Fulfillment
Do not mark orders paid or grant subscription access from Checkout callback verification alone. Use verified Razorpay webhook events for durable fulfillment. Do not attach fulfillment triggers to provider mirror tables such as `payments.razorpay_subscriptions`.
```sql
CREATE OR REPLACE FUNCTION public.fulfill_razorpay_order()
RETURNS TRIGGER AS $$
BEGIN
IF NEW.provider = 'razorpay'
AND NEW.event_type IN ('payment.captured', 'order.paid', 'invoice.paid')
AND NEW.processing_status = 'processed'
AND COALESCE(
NEW.payload -> 'payload' -> 'payment' -> 'entity' -> 'notes' ->> 'order_id',
NEW.payload -> 'payload' -> 'invoice' -> 'entity' -> 'notes' ->> 'order_id'
) IS NOT NULL THEN
UPDATE public.orders
SET status = 'paid',
paid_at = COALESCE(NEW.processed_at, NOW())
WHERE id::text = COALESCE(
NEW.payload -> 'payload' -> 'payment' -> 'entity' -> 'notes' ->> 'order_id',
NEW.payload -> 'payload' -> 'invoice' -> 'entity' -> 'notes' ->> 'order_id'
)
AND status = 'pending';
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE TRIGGER fulfill_razorpay_order_from_webhook
AFTER INSERT OR UPDATE ON payments.webhook_events
FOR EACH ROW
EXECUTE FUNCTION public.fulfill_razorpay_order();
```
For subscriptions, resolve the billing subject from the subscription entity's `notes` in the event payload — InsForge stamps `insforge_subject_type` and `insforge_subject_id` into notes at subscription creation (and creates the `payments.customer_mappings` row at the same time, so the mapping is also a safe fallback):
```sql
CREATE OR REPLACE FUNCTION public.grant_razorpay_subscription_access()
RETURNS TRIGGER AS $$
DECLARE
v_subject_type TEXT;
v_subject_id TEXT;
BEGIN
IF NEW.provider = 'razorpay'
AND NEW.event_type = 'subscription.charged'
AND NEW.processing_status = 'processed' THEN
v_subject_type := NEW.payload -> 'payload' -> 'subscription' -> 'entity'
-> 'notes' ->> 'insforge_subject_type';
v_subject_id := NEW.payload -> 'payload' -> 'subscription' -> 'entity'
-> 'notes' ->> 'insforge_subject_id';
IF v_subject_id IS NULL THEN
SELECT m.subject_type, m.subject_id
INTO v_subject_type, v_subject_id
FROM payments.customer_mappings m
WHERE m.provider = NEW.provider
AND m.environment = NEW.environment
AND m.provider_customer_id = NEW.payload -> 'payload' -> 'subscription'
-> 'entity' ->> 'customer_id';
END IF;
IF v_subject_id IS NULL THEN
RAISE WARNING 'Razorpay event % has no resolvable billing subject', NEW.provider_event_id;
RETURN NEW;
END IF;
-- Branch on the subject type sent at creation; team_id is a UUID here,
-- so the type check also guards the cast.
IF v_subject_type = 'team' THEN
INSERT INTO public.team_entitlements (team_id, plan, active, updated_at)
VALUES (v_subject_id::uuid, 'pro', true, NOW())
ON CONFLICT (team_id) DO UPDATE SET
plan = EXCLUDED.plan,
active = true,
updated_at = NOW();
END IF;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE TRIGGER grant_razorpay_subscription_access_from_webhook
AFTER INSERT OR UPDATE ON payments.webhook_events
FOR EACH ROW
EXECUTE FUNCTION public.grant_razorpay_subscription_access();
```
Handle revocation the same way from `subscription.cancelled`, `subscription.halted`, and `subscription.expired`.
Adapt the payload lookup to the app schema and event shape. Protect app-owned billing tables with RLS. Use `payments.transactions` for dashboard/reporting only.
## Security
- Add RLS or server-side membership checks before exposing order or subscription flows for shared subjects.
- Consider RLS on `payments.razorpay_orders` and `payments.razorpay_subscriptions`.
- Do not expose `payments.customers`, `payments.transactions`, or `payments.razorpay_subscriptions` directly to end users.
- Do not write provider-managed payments tables directly. Use the Payments API, Razorpay webhooks, or app-owned trigger targets.
- Notes keys starting with `insforge_` are reserved.
## Debugging
Check recent Razorpay order attempts:
```sql
SELECT id, environment, status, subject_type, subject_id,
order_id, receipt, amount, currency, verified_payment_id,
last_error, created_at, updated_at
FROM payments.razorpay_orders
ORDER BY created_at DESC
LIMIT 20;
```
Check recent Razorpay subscriptions:
```sql
SELECT environment, subscription_id, plan_id, customer_id, status,
subject_type, subject_id, authorization_payment_id,
current_start, current_end, created_at, updated_at
FROM payments.razorpay_subscriptions
ORDER BY created_at DESC
LIMIT 20;
```
Check customer mappings:
```sql
SELECT provider, environment, subject_type, subject_id, provider_customer_id, created_at, updated_at
FROM payments.customer_mappings
WHERE provider = 'razorpay'
ORDER BY updated_at DESC
LIMIT 20;
```
Check Razorpay transactions:
```sql
SELECT provider, environment, type, status, subject_type, subject_id,
provider_object_type, provider_object_id, amount, currency,
paid_at, failed_at, refunded_at, created_at
FROM payments.transactions
WHERE provider = 'razorpay'
ORDER BY created_at DESC
LIMIT 20;
```
Check webhook failures:
```sql
SELECT provider, environment, provider_event_id, event_type, processing_status,
attempt_count, last_error, received_at, processed_at
FROM payments.webhook_events
WHERE provider = 'razorpay'
AND processing_status IN ('failed', 'pending')
ORDER BY received_at DESC
LIMIT 20;
```
## Common Failures
| Symptom | Check |
|---------|-------|
| Order creation fails | Verify Key ID and Key Secret for the environment and confirm amount is in the smallest currency unit. |
| Checkout does not open | Confirm `https://checkout.razorpay.com/v1/checkout.js` is loaded and `checkoutOptions.key` is present. |
| Signature verification fails | Pass the exact order or subscription ID, payment ID, and signature returned by Razorpay Checkout. |
| Webhook signature is invalid | Confirm the Razorpay Dashboard webhook secret matches Dashboard -> Payments -> Settings -> Webhooks for the same `test` or `live` environment, and that the URL ends with `/api/webhooks/razorpay/{environment}`. |
| Subscription creation fails | Verify the Plan exists in the same Razorpay environment and has a valid Item. |
| Payment shows in Razorpay but not InsForge | Check manual webhook setup and `payments.webhook_events`. |
| User can start a subscription for another team | Add RLS or server-side membership checks for the billing subject. |
.agents/docs/payments-stripe.md
+287
View File
@@ -0,0 +1,287 @@
# InsForge Stripe Payments - Agent Documentation
## Use Stripe For
- Stripe Checkout for one-time payments.
- Stripe Checkout for subscriptions.
- Stripe Billing Portal links for existing customers.
- Stripe Products and Prices.
- Stripe-managed subscription and invoice lifecycle.
Do not build raw card collection UI. Do not use Razorpay concepts such as Orders, Items, or Plans in a Stripe flow.
## Before Coding
1. Use `environment: "test"` unless the user explicitly approves live Stripe changes.
2. Confirm the Stripe secret key is configured for the target environment.
3. Confirm Product and Price IDs exist in that same environment.
4. Confirm the backend can receive Stripe webhooks. InsForge manages the Stripe webhook endpoint when the backend is reachable.
5. Treat Checkout success URLs as UX redirects only. Fulfillment must come from webhooks.
Project admins configure Stripe in Dashboard -> Payments -> Settings or with the CLI:
```bash
npx @insforge/cli payments stripe status
npx @insforge/cli payments stripe config set --environment test sk_test_xxx
npx @insforge/cli payments stripe sync --environment test
npx @insforge/cli payments stripe webhooks configure --environment test
```
## Runtime Setup
Use the TypeScript SDK from application code:
```typescript
import { createClient } from '@insforge/sdk';
const insforge = createClient({
baseUrl: 'https://your-project.insforge.app',
anonKey: 'your-anon-key'
});
```
Checkout requires an InsForge user token. Guest one-time checkout can use an anonymous InsForge token. API keys are not a replacement because the backend needs user context for `payments.stripe_checkout_sessions`.
## One-Time Checkout
Create an app-owned pending order first, then start Checkout:
```typescript
const { data: order, error: orderError } = await insforge
.from('orders')
.insert([{ user_id: user.id, status: 'pending' }])
.select()
.single();
if (orderError) throw orderError;
const { data, error } = await insforge.payments.stripe.createCheckoutSession('test', {
mode: 'payment',
lineItems: [{ priceId: 'price_123', quantity: 1 }],
successUrl: `${window.location.origin}/orders/${order.id}`,
cancelUrl: `${window.location.origin}/pricing`,
customerEmail: user.email,
metadata: { order_id: order.id },
idempotencyKey: `order:${order.id}`
});
if (error) throw error;
if (data?.checkoutSession.url) {
window.location.assign(data.checkoutSession.url);
}
```
For anonymous one-time purchases, omit `subject` and pass `customerEmail` when available.
## Subscription Checkout
Subscriptions require a billing subject. Pick a stable app owner such as user, team, organization, workspace, tenant, or group.
```typescript
const { data, error } = await insforge.payments.stripe.createCheckoutSession('test', {
mode: 'subscription',
subject: { type: 'team', id: teamId },
lineItems: [{ priceId: 'price_monthly_123', quantity: 1 }],
successUrl: `${window.location.origin}/billing/success`,
cancelUrl: `${window.location.origin}/billing`,
customerEmail: user.email,
idempotencyKey: `team:${teamId}:pro-monthly`
});
if (error) throw error;
if (data?.checkoutSession.url) {
window.location.assign(data.checkoutSession.url);
}
```
Do not let users submit arbitrary `subject.type` and `subject.id` values unless the app checks they can manage that billing subject.
## Customer Portal
Use Billing Portal after Checkout has created a customer mapping for the subject.
```typescript
const { data, error } = await insforge.payments.stripe.createCustomerPortalSession('test', {
subject: { type: 'team', id: teamId },
returnUrl: `${window.location.origin}/billing`
});
if (error) {
if ('statusCode' in error && error.statusCode === 404) {
return;
}
throw error;
}
if (data?.customerPortalSession.url) {
window.location.assign(data.customerPortalSession.url);
}
```
Portal creation requires an authenticated user and an existing `payments.customer_mappings` row for the subject.
## Fulfillment
Create triggers from verified Stripe webhook events into app-owned tables.
Webhook events are verified and processed independently. InsForge commits all rows derived from an event before marking that event `processed`, but there is no ordering guarantee across events: Stripe can deliver `invoice.paid` before `checkout.session.completed`, so rows created by another event (such as `payments.customer_mappings`) may not exist yet when your trigger fires. Resolve the billing subject from the event payload first and use `payments.customer_mappings` as a fallback, exactly as the examples below do. Never let fulfillment skip silently: log or dead-letter events you cannot resolve.
### One-time payments
```sql
CREATE OR REPLACE FUNCTION public.fulfill_paid_order()
RETURNS TRIGGER AS $$
BEGIN
IF NEW.provider = 'stripe'
AND NEW.event_type = 'checkout.session.completed'
AND NEW.processing_status = 'processed'
AND (NEW.payload -> 'data' -> 'object' -> 'metadata' ->> 'order_id') IS NOT NULL THEN
UPDATE public.orders
SET status = 'paid',
paid_at = COALESCE(NEW.processed_at, NOW())
WHERE id::text = NEW.payload -> 'data' -> 'object' -> 'metadata' ->> 'order_id'
AND status = 'pending';
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE TRIGGER fulfill_paid_order_from_stripe_webhook
AFTER INSERT OR UPDATE ON payments.webhook_events
FOR EACH ROW
EXECUTE FUNCTION public.fulfill_paid_order();
```
### Subscriptions
Subscription events do not carry your app's `metadata`. Resolve the billing subject from the subscription metadata embedded in the event payload — InsForge stamps `insforge_subject_type` and `insforge_subject_id` at checkout, and Stripe snapshots it onto subscription-generated invoices as `parent.subscription_details.metadata`. Check `invoice.metadata` next, then fall back to `payments.customer_mappings` (the same order InsForge uses internally):
```sql
CREATE OR REPLACE FUNCTION public.grant_subscription_access()
RETURNS TRIGGER AS $$
DECLARE
v_subject_type TEXT;
v_subject_id TEXT;
BEGIN
IF NEW.provider = 'stripe'
AND NEW.event_type = 'invoice.paid'
AND NEW.processing_status = 'processed' THEN
v_subject_type := COALESCE(
NEW.payload -> 'data' -> 'object' -> 'parent'
-> 'subscription_details' -> 'metadata' ->> 'insforge_subject_type',
NEW.payload -> 'data' -> 'object' -> 'metadata' ->> 'insforge_subject_type'
);
v_subject_id := COALESCE(
NEW.payload -> 'data' -> 'object' -> 'parent'
-> 'subscription_details' -> 'metadata' ->> 'insforge_subject_id',
NEW.payload -> 'data' -> 'object' -> 'metadata' ->> 'insforge_subject_id'
);
IF v_subject_id IS NULL THEN
SELECT m.subject_type, m.subject_id
INTO v_subject_type, v_subject_id
FROM payments.customer_mappings m
WHERE m.provider = NEW.provider
AND m.environment = NEW.environment
AND m.provider_customer_id = NEW.payload -> 'data' -> 'object' ->> 'customer';
END IF;
IF v_subject_id IS NULL THEN
RAISE WARNING 'Stripe event % has no resolvable billing subject', NEW.provider_event_id;
RETURN NEW;
END IF;
-- Branch on the subject type sent at checkout; team_id is a UUID here,
-- so the type check also guards the cast.
IF v_subject_type = 'team' THEN
INSERT INTO public.team_entitlements (team_id, plan, active, updated_at)
VALUES (v_subject_id::uuid, 'pro', true, NOW())
ON CONFLICT (team_id) DO UPDATE SET
plan = EXCLUDED.plan,
active = true,
updated_at = NOW();
END IF;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE TRIGGER grant_subscription_access_from_stripe_webhook
AFTER INSERT OR UPDATE ON payments.webhook_events
FOR EACH ROW
EXECUTE FUNCTION public.grant_subscription_access();
```
Adjust the trigger target to the app-owned entitlement table for the billing subject type used at checkout. Handle revocation the same way from `customer.subscription.deleted` and `customer.subscription.updated` events (`payload -> 'data' -> 'object' -> 'metadata'` holds the subject keys on subscription events).
Use `payments.transactions` for dashboard/reporting only.
## Security
- Add RLS or server-side membership checks before exposing checkout or portal flows for shared subjects.
- Consider RLS on `payments.stripe_checkout_sessions` and `payments.stripe_customer_portal_sessions`.
- PostgreSQL applies `SELECT` policies to rows returned by `INSERT ... RETURNING` and idempotent retry lookups. If checkout creation is denied even though an `INSERT` policy exists, add matching `SELECT` visibility for the same billing subject and idempotency key.
- Do not expose `payments.customers`, `payments.transactions`, `payments.stripe_subscriptions`, or `payments.stripe_subscription_items` directly to end users.
- Do not write Stripe-managed payments tables directly. Use the Payments API, Stripe webhooks, or app-owned trigger targets.
- Metadata keys starting with `insforge_` are reserved.
## Debugging
Check recent checkout attempts:
```sql
SELECT id, environment, mode, status, payment_status, subject_type, subject_id,
checkout_session_id, customer_id, subscription_id,
last_error, created_at, updated_at
FROM payments.stripe_checkout_sessions
ORDER BY created_at DESC
LIMIT 20;
```
Check customer mappings:
```sql
SELECT provider, environment, subject_type, subject_id, provider_customer_id, created_at, updated_at
FROM payments.customer_mappings
WHERE provider = 'stripe'
ORDER BY updated_at DESC
LIMIT 20;
```
Check Stripe transactions:
```sql
SELECT provider, environment, type, status, subject_type, subject_id,
provider_object_type, provider_object_id, amount, currency,
paid_at, failed_at, refunded_at, created_at
FROM payments.transactions
WHERE provider = 'stripe'
ORDER BY created_at DESC
LIMIT 20;
```
Check webhook failures:
```sql
SELECT provider, environment, provider_event_id, event_type, processing_status,
attempt_count, last_error, received_at, processed_at
FROM payments.webhook_events
WHERE provider = 'stripe'
AND processing_status IN ('failed', 'pending')
ORDER BY received_at DESC
LIMIT 20;
```
## Common Failures
| Symptom | Check |
|---------|-------|
| Checkout returns Stripe key not configured | Configure the correct `test` or `live` Stripe key. |
| Checkout uses the wrong price | Verify the Price ID belongs to the selected environment. |
| Duplicate checkout attempts | Use a stable `idempotencyKey` based on the order, cart, or billing subject. |
| Portal returns not found | The subject has no Stripe customer mapping yet. Have the customer complete Checkout first. |
| Payment shows in Stripe but not InsForge | Check Stripe webhook configuration and `payments.webhook_events`. |
| User can start checkout for another team | Add RLS or server-side membership checks for the billing subject. |
.agents/docs/payments.md
+30
View File
@@ -0,0 +1,30 @@
# InsForge Payments - Agent Documentation
Use a provider-specific guide before coding. Stripe and Razorpay have different payment models, runtime APIs, table names, and webhook setup requirements.
## Provider guides
- [Stripe payments](./payments-stripe.md): Stripe Checkout, Billing Portal, Products, Prices, Subscriptions, managed webhooks.
- [Razorpay payments](./payments-razorpay.md): Razorpay Orders, Checkout script, Items, Plans, Subscriptions, manual webhooks.
## Shared rules
1. Use `environment: "test"` unless the user explicitly approves live payment changes.
2. Never put provider secret keys in frontend code or browser-exposed deployment variables.
3. Do not fulfill from Stripe success URLs or Razorpay Checkout callbacks alone.
4. Fulfillment should run from verified rows in `payments.webhook_events`.
5. Keep user-facing order, credit, and entitlement state in app-owned tables with app-owned RLS.
6. Use `payments.transactions` for dashboard and reporting queries, not as the primary fulfillment contract.
7. Webhook events are processed independently with no cross-event ordering guarantee. Rows derived from an event are committed before that event is marked `processed`, but rows derived from other events (such as `payments.customer_mappings` or provider mirrors) may not exist yet. Fulfillment triggers must resolve billing subjects from the event payload first and treat lookups into rows owned by other events as fallbacks.
## Shared tables
| Table | Use |
|-------|-----|
| `payments.provider_connections` | Provider/environment connection, sync, and webhook setup status. |
| `payments.customer_mappings` | Billing subject to provider customer ID mapping. |
| `payments.customers` | Admin/customer mirror for dashboard visibility. |
| `payments.webhook_events` | Verified provider event ledger. Build durable fulfillment triggers here. |
| `payments.transactions` | Dashboard/reporting projection for successful, failed, pending, and refunded transactions. |
Legacy `payments.payment_history` rows are migrated into `payments.transactions`, but triggers on `payment_history` are not migrated automatically. Recreate fulfillment triggers on `payments.webhook_events`.
.agents/docs/real-time.md
+268
View File
@@ -0,0 +1,268 @@
# InsForge Realtime - Agent Documentation
## Use Realtime For
- Live client updates from database changes.
- Client-to-client broadcasts on named channels.
- Ephemeral presence on subscribed channels.
- Webhook fan-out for messages published to a channel.
Realtime delivers events to subscribed clients and configured webhook URLs. If server-side code must perform work after a database change, such as sending email, writing another table, or calling an API, put that work in an Edge Function and invoke it from a database trigger.
## Mental Model
1. Create channel patterns in `realtime.channels`.
2. Publish messages into `realtime.messages`.
3. Postgres triggers `pg_notify('realtime_message', message_id)`.
4. The backend loads the message, checks the channel is enabled, and delivers it to Socket.IO subscribers plus channel webhooks.
5. Delivery stats are written back to the message row.
Channel patterns use SQL `LIKE`: `order:%` matches `order:123`. Use `:` as the separator and `%` as the wildcard. Do not use `_`.
## Backend Setup
### 1. Create Channel Patterns
```sql
INSERT INTO realtime.channels (pattern, description, enabled)
VALUES
('orders', 'Global order events', true),
('order:%', 'Per-order events', true),
('chat:%', 'Chat room events', true);
```
You can also create channels in the Dashboard Realtime page.
### 2. Publish Database Changes
Create a trigger on the app table you want to watch. In the trigger function, call `realtime.publish(channel, event, payload)` to choose the channel, event name, and payload.
```sql
CREATE OR REPLACE FUNCTION notify_order_status()
RETURNS TRIGGER AS $$
BEGIN
PERFORM realtime.publish(
'order:' || NEW.id::text,
'status_changed',
jsonb_build_object(
'id', NEW.id,
'status', NEW.status,
'updatedAt', NEW.updated_at
)
);
RETURN NEW;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE TRIGGER order_status_realtime
AFTER UPDATE OF status ON orders
FOR EACH ROW
WHEN (OLD.status IS DISTINCT FROM NEW.status)
EXECUTE FUNCTION notify_order_status();
```
For delete events, use `OLD` instead of `NEW`.
### 3. Add Access Control When Needed
Realtime is open by default. `anon` and `authenticated` can subscribe to enabled channels and publish to channels they joined.
To restrict access:
```sql
ALTER TABLE realtime.channels ENABLE ROW LEVEL SECURITY;
ALTER TABLE realtime.messages ENABLE ROW LEVEL SECURITY;
```
Subscribe is controlled by `SELECT` policies on `realtime.channels`:
```sql
CREATE POLICY "users_subscribe_own_orders"
ON realtime.channels
FOR SELECT
TO authenticated
USING (
pattern = 'order:%'
AND EXISTS (
SELECT 1
FROM orders
WHERE id = NULLIF(split_part(realtime.channel_name(), ':', 2), '')::uuid
AND user_id = auth.uid()
)
);
```
Publish is controlled by `INSERT` policies on `realtime.messages`:
```sql
CREATE POLICY "members_publish_chat"
ON realtime.messages
FOR INSERT
TO authenticated
WITH CHECK (
channel_name LIKE 'chat:%'
AND EXISTS (
SELECT 1
FROM chat_members
WHERE room_id = NULLIF(split_part(channel_name, ':', 2), '')::uuid
AND user_id = auth.uid()
)
);
```
Use `realtime.channel_name()` in subscribe policies because `realtime.channels` stores patterns, while the client requests a resolved channel such as `order:123`.
## Frontend SDK Pattern
```typescript
import { createClient } from '@insforge/sdk';
const insforge = createClient({
baseUrl: 'https://your-project.insforge.app',
anonKey: 'your-anon-key'
});
insforge.realtime.on('error', ({ channel, code, message }) => {
console.error(channel, code, message);
});
await insforge.realtime.connect();
const response = await insforge.realtime.subscribe(`order:${orderId}`);
if (!response.ok) {
throw new Error(response.error.message);
}
insforge.realtime.on('status_changed', (payload) => {
console.log(payload.status);
console.log(payload.meta.messageId);
});
```
### SDK Methods
| Task | Method |
|------|--------|
| Connect | `await insforge.realtime.connect()` |
| Subscribe | `await insforge.realtime.subscribe(channel)` |
| Publish | `await insforge.realtime.publish(channel, event, payload)` |
| Listen | `insforge.realtime.on(event, callback)` |
| Listen once | `insforge.realtime.once(event, callback)` |
| Remove listener | `insforge.realtime.off(event, callback)` |
| Unsubscribe | `insforge.realtime.unsubscribe(channel)` |
| Disconnect | `insforge.realtime.disconnect()` |
| List local subscriptions | `insforge.realtime.getSubscribedChannels()` |
Client publish requires a successful subscription to the same channel first.
## Raw Socket.IO Contract
Use this only when the SDK is not available.
```typescript
import { io } from 'socket.io-client';
const socket = io('https://your-project.insforge.app', {
auth: {
token: '<user-jwt-or-anon-token>'
}
});
socket.emit('realtime:subscribe', { channel: 'chat:room-1' }, (response) => {
if (!response.ok) {
console.error(response.error);
}
});
socket.on('new_message', (message) => {
console.log(message);
});
socket.emit('realtime:publish', {
channel: 'chat:room-1',
event: 'new_message',
payload: { text: 'Hello' }
});
```
Events:
| Event | Direction |
|-------|-----------|
| `realtime:subscribe` | Client to server |
| `realtime:unsubscribe` | Client to server |
| `realtime:publish` | Client to server |
| Custom event name | Server to client |
| `presence:join` | Server to client |
| `presence:leave` | Server to client |
| `realtime:error` | Server to client |
## Presence
`subscribe()` returns:
```json
{
"ok": true,
"channel": "chat:room-1",
"presence": {
"members": [
{
"type": "user",
"presenceId": "user-id",
"joinedAt": "2026-04-25T17:00:00.000Z"
}
]
}
}
```
Listen for `presence:join` and `presence:leave` to keep local online state current. Presence is in-memory online state, not durable membership.
## Webhooks
Channel `webhookUrls` receive the message payload as the request body.
Headers:
| Header | Meaning |
|--------|---------|
| `X-InsForge-Event` | Event name |
| `X-InsForge-Channel` | Resolved channel name |
| `X-InsForge-Message-Id` | Message UUID |
Webhook delivery counts appear in message history as `whAudienceCount` and `whDeliveredCount`.
## Message Retention
`realtime.config.retention_days` controls cleanup:
- `NULL`: keep messages indefinitely.
- Positive integer: delete messages older than that many days.
The cleanup job runs daily through pg_cron.
## Dashboard And REST Checks
Use Dashboard Realtime pages to verify:
- Channels: patterns, enabled state, webhooks.
- Messages: payloads, sender type, WebSocket audience, webhook delivery.
- Permissions: RLS policies for subscribe and publish.
- Settings: message retention.
REST endpoints:
| Endpoint | Purpose |
|----------|---------|
| `GET /api/realtime/channels` | List channels |
| `POST /api/realtime/channels` | Create channel |
| `PUT /api/realtime/channels/{id}` | Update channel |
| `DELETE /api/realtime/channels/{id}` | Delete channel |
| `GET /api/realtime/messages` | List message history |
| `DELETE /api/realtime/messages` | Clear all message history |
| `GET /api/realtime/messages/stats` | Message stats |
| `GET /api/realtime/permissions` | Realtime RLS policies |
| `GET /api/realtime/config` | Retention config |
| `PATCH /api/realtime/config` | Update retention config |
.agents/skills/insforge-dev/SKILL.md
+73
View File
@@ -0,0 +1,73 @@
---
name: insforge-dev
description: Use this skill set when contributing to the InsForge monorepo itself. This is for InsForge maintainers and contributors editing the platform, the shared dashboard package, the self-hosting shell, the UI library, shared schemas, tests, or docs.
---
# InsForge Dev
Use this skill set for work inside the InsForge repository.
Then use the narrowest package skill that matches the task:
- `backend`
- `dashboard`
- `ui`
- `shared-schemas`
- `docs`
Use the cross-repo release workflow skill when preparing an OSS PR:
- `e2e-testing`
## Core Rules
1. Identify the package boundary before editing.
- `backend/`: API, auth, database, providers, realtime, schedules
- `packages/dashboard/`: publishable dashboard package that supports `self-hosting` and `cloud-hosting` modes
- `frontend/`: local React + Vite shell that mounts `packages/dashboard/` in `self-hosting` mode
- `packages/shared-schemas/`: cross-package contracts
- `packages/ui/`: reusable design-system primitives
- `docs/`: product and agent-facing documentation
2. Put code in the narrowest correct layer.
- Contract change: `packages/shared-schemas/` first, then consumers.
- Backend behavior: route -> service -> provider/infra.
- Shared dashboard behavior, routes, features, exports, and host contracts: `packages/dashboard/`.
- Self-hosting-only bootstrap, env wiring, and shell styling: `frontend/`.
- Reusable primitive: `packages/ui/` first.
3. Preserve repo conventions.
- Backend TS source uses ESM-style `.js` import specifiers.
- Backend success responses usually return raw JSON, not `{ data }`.
- Backend validation commonly uses shared Zod schemas plus `AppError`.
- Dashboard data access goes through `apiClient` and React Query.
- Dashboard frontend tests are split into Vitest unit tests, Vitest component tests, and Playwright UI smoke tests.
- Shared payloads belong in `@insforge/shared-schemas`.
- Never use the TypeScript `any` type. Prefer precise types, schema-derived types, `unknown`, or generics.
4. Do not confuse repo development with app development on InsForge.
- This repo contains the platform, the publishable dashboard package, and a local shell for self-hosting mode.
- Keep guidance focused on maintaining InsForge itself.
## Finish Rules
- Run the smallest validation that gives confidence for the change.
- Use repo-level checks like `npm run lint`, `npm run build`, `npm run typecheck`, and `npm test` when the change crosses package boundaries — each is wired through Turborepo (`turbo run <task>`) and covers every workspace, including `packages/dashboard/` and `packages/shared-schemas/`.
- For dashboard UI behavior changes, choose the lowest useful frontend test layer from the `dashboard` skill and run that command before reporting back.
- Use the package-specific validation steps in the child skill when the work is isolated to one package.
- When reporting back, state what changed, what you validated, and what you could not validate.
## Pre-PR Checklist (Mandatory Before Pushing to a PR)
Before opening a PR or pushing new commits to an existing PR branch, run **all** of the following from the repo root and do not proceed while any of them fail on files your change touches:
1. `npx turbo run typecheck` — must pass across all packages.
2. `npx turbo run lint` — must pass. If the failure is pre-existing in `main` and unrelated to your change, scope it:
- Run `npx eslint <your-changed-files>` and confirm your files are clean.
- Call out the pre-existing debt in the PR body so reviewers know it is not yours.
- Auto-fixable prettier/eslint errors in your own diff must be fixed (`npx eslint --fix <file>` or `npm run format`).
3. `npx turbo run test` (or the package-specific test command) — all tests must pass, including any new tests you added for the change.
4. `npx turbo run build` if routing, config, schemas, or cross-package exports changed.
5. Use `e2e-testing` to run the deterministic cross-repo E2E gate before opening, updating, or submitting the InsForge OSS PR.
Never push with failing checks on files you touched, even if CI would catch them later. CI failures slow reviewers down and the lint fix almost always takes less than a minute locally.
.agents/skills/insforge-dev/backend/SKILL.md
+74
View File
@@ -0,0 +1,74 @@
---
name: backend
description: Use this skill when contributing to InsForge's backend package. This is for maintainers editing backend routes, services, providers, auth, database logic (including RLS-enforced surfaces like storage and realtime), schedules, or backend tests in the InsForge monorepo.
---
# InsForge Dev Backend
Use this skill for `backend/` work in the InsForge repository.
## Scope
- `backend/src/api/**`
- `backend/src/services/**`
- `backend/src/providers/**`
- `backend/src/infra/**`
- `backend/tests/**`
## Working Rules
1. Keep the route -> service -> provider/infra split intact.
- Routes handle auth, parsing, validation, and delegation.
- Services own business logic and orchestration.
- Providers and infra wrap external systems or lower-level integrations.
- Service layer code should be the only layer that interacts with the core PostgreSQL database.
- Do not put direct database access in routes.
- Do not bypass services when reading from or writing to Postgres.
2. Follow backend conventions.
- Use ESM-style `.js` import specifiers in TypeScript source.
- InsForge's core database is PostgreSQL.
- InsForge currently runs as a single-instance server, so be careful about introducing logic that assumes distributed coordination, cross-instance locking, or background worker separation.
- Reuse shared schemas from `@insforge/shared-schemas` when contracts cross packages.
- Use `safeParse` plus `AppError` for invalid input.
- Return successful results through `successResponse`.
- Preserve existing auth middleware patterns such as `verifyAdmin`, `verifyUser`, and `verifyApiKey`.
- Never use the TypeScript `any` type. Prefer precise interfaces, schema-derived types, `unknown`, or constrained generics.
- For schema changes, write a new migration file instead of editing database structure manually.
- Put schema changes under `backend/src/infra/database/migrations/`.
3. Write idempotent migrations. Every SQL migration must be safe to re-run.
- Use `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`.
- Never use bare `ALTER TABLE ... RENAME TO` — it fails if the target name already exists. Wrap renames in a `DO` block that checks `information_schema.tables` for both source and target.
- Always `DROP TRIGGER IF EXISTS` before `CREATE TRIGGER`.
- Guard data migrations and `DROP COLUMN` behind `information_schema.columns` checks when the column may already be gone.
- Use `ON CONFLICT` or `WHERE NOT EXISTS` for seed `INSERT` statements.
4. Preserve existing behavior around mutation flows.
- Keep audit logging when surrounding routes already log state changes.
- Keep error handling flowing through shared middleware.
- Do not introduce a new response envelope unless the existing feature already uses one.
- For critical flows with multiple dependent database writes, use an explicit transactional process so the whole operation succeeds or fails together.
- Be especially careful with transactions around auth, secrets, billing-like usage updates, schema changes, and any flow that would leave the system inconsistent if partially applied.
5. Use Postgres Row Level Security, not app-side filters, for tables accessed via authenticated end-user routes (anything where `req.user` reaches the service layer). RLS-enforced services such as storage, realtime, and payments should use `withUserContext`. Tables accessed only by admin or service-internal paths (audit logs, billing aggregations) don't need RLS. Do not write `WHERE user_id = $1` filters in services; let RLS evaluate `auth.jwt() ->> 'sub'` against the row.
- Plumb identity through `withUserContext(pool, ctx, fn, settings?)` from `services/database/user-context.service.ts`. It opens a transaction, sets `SET LOCAL ROLE` plus the canonical `request.jwt.claims` JSON GUC via `set_config`, applies optional transaction-local settings such as `realtime.channel_name`, runs `fn`, commits on success or rolls back on error, and resets role in `finally` so policies see the calling user via `auth.jwt() ->> 'sub'`.
- Keep `UserContext` user-only and defined in `api/middlewares/auth.ts`: `{ id, role, email? }` (`id` is always present at the API level). API keys and admin bypass flags do not belong inside `UserContext`.
- Routes that issue out-of-band URLs (S3 presigned redirects, signed download links, anything the client redeems against a service that won't re-evaluate RLS) must do an explicit RLS-scoped existence check before handing the URL out — RLS does not fire when the client redeems the URL directly. See `StorageService.objectIsVisible` as the template.
- Migrations that enable RLS on an existing populated table must auto-install a sensible default policy set so the upgrade does not silently break existing rows. See migration 036's `IF EXISTS (SELECT 1 FROM <table>) THEN <create policies> END IF` pattern.
- When adding a new RLS-enforced table: enable RLS, `GRANT` table-level CRUD to `authenticated`, and write per-operation policies (SELECT, INSERT, UPDATE, DELETE). Public-bucket-style anonymous bypasses live at the route layer before calling the RLS helper, not in policies.
- Normal raw SQL and custom migrations execute as `project_admin`. It has service-key row visibility, but PostgreSQL grants and ownership still limit object access and DDL.
6. Always write unit tests for new code.
- Every new feature, migration, service, or bug fix should have accompanying unit tests.
- For migrations, write tests that validate SQL structure and idempotency guards (see `tests/unit/redirect-url-whitelist-migration.test.ts` for the pattern).
- For services, test business logic and error cases.
- For RLS-gated services, mock the pool/client and pin the SQL sequence (see `tests/unit/user-context.service.test.ts` and `tests/unit/storage-object-is-visible.test.ts`).
- Run the full test suite before submitting work: `cd backend && npm test`.
## Validation
- `cd backend && npm test`
- `cd backend && npm run build`
For contract changes, also validate `packages/shared-schemas/` and any affected dashboard consumers.
.agents/skills/insforge-dev/dashboard/SKILL.md
+107
View File
@@ -0,0 +1,107 @@
---
name: dashboard
description: Use this skill when contributing to InsForge's shared dashboard package. This is for maintainers editing `packages/dashboard`, which ships in `self-hosting` and `cloud-hosting` modes, and the local `frontend/` shell used for `self-hosting` in this repo.
---
# InsForge Dev Dashboard
Use this skill for dashboard work in the InsForge repository.
## Scope
- `packages/dashboard/src/**`
- `packages/dashboard/package.json`
- `packages/dashboard/README.md`
- `packages/dashboard/*.config.*`
- `frontend/src/**`
- `frontend/package.json`
## Working Rules
1. Respect the shared-package versus host-app boundary.
- This dashboard is built with React and TypeScript.
- `packages/dashboard/` is the source of truth for the dashboard product.
- The package must support both `self-hosting` and `cloud-hosting` modes.
- Keep self-hosting-only bootstrap, local env defaults, and shell styling in `frontend/`.
- Do not let `packages/dashboard/` depend on `frontend/`.
- If both modes need a capability, define it in the package API first.
2. Preserve dashboard data-flow conventions.
- Follow the flow `service -> hook -> UI`.
- Use `apiClient` for HTTP calls so auth refresh and error handling stay consistent.
- Put request logic in services, data fetching and mutation state in hooks, and rendering/orchestration in UI components and pages.
- Reuse existing contexts, host abstractions, and hooks before creating new global state.
3. Reuse the existing component layers.
- Use `@insforge/ui` for generic primitives.
- Use shared dashboard components when the pattern is already present.
- Keep reusable dashboard UI in `packages/dashboard/`.
- Only add UI to `frontend/` when it is specific to the local self-hosting shell.
- Keep package styles scoped to the dashboard container.
4. Keep the package surface aligned with shared contracts.
- Import cross-package types and Zod-derived shapes from `@insforge/shared-schemas`.
- When backend payloads change, update the related services, hooks, UI, and exported types together.
- Keep `packages/dashboard/src/index.ts` and `packages/dashboard/src/types` aligned with the public package API.
- Never use the TypeScript `any` type. Prefer precise prop, state, API, and hook result types.
## Frontend Testing
Use the lowest test layer that covers the risk. Add one focused regression test for bug fixes when practical.
| Change type | Test layer | Location | Command |
| --- | --- | --- | --- |
| Pure helper, parser, formatter, state reducer | Unit | `packages/dashboard/src/**/__tests__/*.test.ts` | `npm --workspace @insforge/dashboard run test:unit` |
| React component behavior, forms, dialogs, conditional rendering | Component | `packages/dashboard/src/**/__tests__/*.test.tsx` | `npm --workspace @insforge/dashboard run test:component` |
| Routing, auth redirects, host-mode integration, browser-only behavior | UI smoke | `packages/dashboard/tests/ui/*.spec.ts` | `npm --workspace @insforge/dashboard run test:ui` |
Conventions:
- `npm --workspace @insforge/dashboard run test` runs the full Vitest suite: unit plus component.
- GitHub Actions splits dashboard checks into unit, component, and UI jobs in `.github/workflows/frontend-tests.yml`. Update the workflow when adding or renaming test scripts.
- Unit tests should avoid React rendering and network mocking. Test data transformations directly.
- Component tests use Testing Library with `packages/dashboard/src/test/setup.ts`. Mock hooks, services, and host context at the package boundary; assert user-visible behavior and callback effects, not implementation details or CSS class strings.
- UI smoke tests use Playwright against the local `frontend/` shell. Mock backend API routes in `packages/dashboard/tests/ui/fixtures/`; every mocked route must either fulfill, fallback, or abort. Do not leave requests pending after a test branch.
- Prefer one clear test per user-observable behavior over broad snapshot tests. Avoid brittle assertions tied to copy that is not part of the behavior being protected.
## Local debug: viewing cloud-hosting-only UI in self-hosting
**Use when** previewing UI gated on `useIsCloudHostingMode()`, `isInsForgeCloudProject()`, or a PostHog feature flag (e.g. the CTest dashboard variant, `dashboard-v3-experiment === 'c_test'`, the CLI connect panel) while running the local `frontend/` self-hosting shell.
The lowest-friction approach is to **temporarily hardcode** the three gates below to `true`/the new branch, then restart the Vite dev server. These edits bypass real host/project detection and MUST be fully reverted before committing — landing them breaks both self-hosting and cloud-hosting users.
### Hardcodes
1. `packages/dashboard/src/lib/config/DashboardHostContext.tsx` — `useIsCloudHostingMode()` → `return true;` (was `useDashboardHost().mode === 'cloud-hosting'`).
2. `packages/dashboard/src/lib/utils/utils.ts` — `isInsForgeCloudProject()` → `return true;` (was the `.insforge.app` hostname check).
3. If the UI is also feature-flag-gated, hardcode the consumer. For CTest: `AppRoutes.tsx` → `const DashboardHomePage = CTestDashboardPage;` and, if relevant, the matching branch in `AppLayout.tsx` for `<ConnectDialogV2>`.
Mark every hardcode with a trailing `// LOCAL DEBUG: <original expression>` comment so revert is a mechanical search.
### Revert checklist — run all before committing
1. `git grep -n "LOCAL DEBUG" packages/dashboard/src/` returns zero matches.
2. Each gate is restored to its **original expression**, not just an equivalent value (the `mode === 'cloud-hosting'` comparison, the hostname check, the `getFeatureFlag(...)` call must all be back).
3. Any imports deleted during debug (commonly `DashboardPage`, `getFeatureFlag`, `ConnectDialog`) are restored.
4. `cd packages/dashboard && npm run lint && npm run typecheck` both pass.
5. `git diff` of the four files above shows only intended changes — no `return true;`, no missing imports.
### Rationalizations to reject
| Excuse | Reality |
|--------|---------|
| "I'll revert in a follow-up PR." | Follow-up = a window where prod is broken. Revert now. |
| "The original check was effectively the same." | If it were, you wouldn't have needed the hardcode. Restore the expression, not a value-equivalent. |
| "Lint passed, so the deleted import doesn't matter." | Lint passed because the import was deleted; on revert the original code needs it back. |
| "I'll ship the env-var override instead." | No env-var override is wired in the code. Don't invent one on the commit path — restore the original. |
## Validation
- `cd packages/dashboard && npm run test:unit` when changing pure helpers or reducers
- `cd packages/dashboard && npm run test:component` when changing React component behavior
- `cd packages/dashboard && npm run test:ui` when changing routing, auth, host-mode integration, or browser-only behavior
- `cd packages/dashboard && npm run typecheck`
- `cd packages/dashboard && npm run build`
- `cd frontend && npm run build` when the local self-hosting shell changes
For shared contract changes, also validate `packages/shared-schemas/` and the affected backend surface.
.agents/skills/insforge-dev/docs/SKILL.md
+47
View File
@@ -0,0 +1,47 @@
---
name: docs
description: Use this skill when contributing to InsForge's product documentation in this repository. This is for maintainers editing public docs in `docs/core-concepts`, agent docs in `.agents/docs`, SDK integration guides in `docs/sdks`, and OpenAPI specs in `openapi`.
---
# InsForge Dev Docs
Use this skill for `docs/core-concepts/`, `.agents/docs/`, `docs/sdks/`, and `openapi/` in the InsForge repository.
The documentation in this repo is primarily product documentation for InsForge users and agents integrating with InsForge. This skill is for InsForge engineers maintaining that public documentation surface.
## Scope
- `docs/core-concepts/**`
- `.agents/docs/**`
- `docs/sdks/**`
- `openapi/**`
## Working Rules
1. Put each document in the correct documentation surface.
- Human-friendly docs published on the public doc site belong in `docs/core-concepts/` and related public doc folders.
- For implementation-heavy public docs, prefer an `architecture.md` file inside the relevant `docs/core-concepts/<domain>/` folder.
- Agent-only instructions belong in `.agents/docs/`.
- SDK integration guides for each framework belong in `docs/sdks/`.
- OpenAPI contract changes belong in the matching files under `openapi/`.
2. Match the writing style to the audience.
- Public docs should be human-friendly and explain the implementation clearly.
- Keep public docs human-sounding: avoid AI-writing tells such as em dashes, rule-of-three lists, "not just X but Y" parallelism, inflated significance, vague attribution, and AI vocabulary (delve, leverage, underscore, seamless, robust). See the `doc-author` skill's `INSFORGE.md` overlay, "Sound human, not AI-generated".
- `architecture.md` pages in `docs/core-concepts/` should explain how the feature works in detail.
- Agent docs in `.agents/docs/` should be instruction-first and execution-oriented.
- Agent docs should avoid explanatory filler and focus on the exact steps an agent should follow to complete the work.
3. Prevent documentation drift on every implementation change.
- Before changing implementation, check the current user-facing docs for that feature.
- After changing implementation, update the relevant Markdown docs and the relevant OpenAPI YAML files in the same pass.
- Do not treat OpenAPI and Markdown as separate optional follow-ups when the feature contract or behavior changed.
- If a change affects agent workflows, update the corresponding file in `.agents/docs/`.
- If a change affects public product understanding, update the corresponding file in `docs/core-concepts/`, including `architecture.md` when implementation details changed.
- If a change affects SDK integration guidance, update the corresponding framework guide in `docs/sdks/`.
## Validation
- Re-read every documented command, path, route, and payload for correctness.
- Cross-check OpenAPI YAML and Markdown docs against the implemented behavior before finishing.
- Mention anything you could not verify directly.
.agents/skills/insforge-dev/e2e-testing/SKILL.md
+142
View File
@@ -0,0 +1,142 @@
---
name: e2e-testing
description: Use this skill when an InsForge maintainer has finished an OSS repo change and is ready to open, update, or submit the InsForge PR. Runs the release-quality deterministic E2E gate by building a package.json-derived InsForge test image tag, deciding whether sibling agent-e2e fixture coverage must change, dispatching the Deterministic Fixture E2E workflow, waiting for results, and triaging failures before PR submission.
---
# InsForge E2E Testing Gate
Use this skill after local implementation and normal InsForge pre-PR checks pass, and before opening, updating, or submitting the InsForge OSS PR.
This is an additional release-quality gate. It does not replace local `typecheck`, `lint`, `test`, or `build` validation from the parent `insforge-dev` skill.
## Repositories
- InsForge OSS repo: current workspace.
- E2E repo: remote GitHub repository `InsForge/agent-e2e`.
Use the remote `InsForge/agent-e2e` repository for workflow dispatch and read-only workflow checks. Do not rely on a developer-specific local checkout path. Create or use a local checkout only when the deterministic fixture tests must be edited.
## Build The Test Tag
1. Read the root InsForge `package.json` version.
2. Increment only the patch number.
3. Build a tag in this form:
```text
v<major>.<minor>.<patch+1>-<feature-or-issue-slug>
```
Example: root version `2.2.3` and feature `storage returning rls` becomes `v2.2.4-storage-returning-rls`.
Use the `package.json` version as the only source of truth for the base version. Ignore higher existing test tags when calculating the base version.
Slug rules:
- Prefer the issue key, PR topic, or branch topic.
- Lowercase all letters.
- Replace runs of non-alphanumeric characters with one hyphen.
- Trim leading and trailing hyphens.
- Keep it short enough to scan in GitHub Actions and image tags.
## Build The InsForge Image
Dispatch the InsForge `Build and Push Docker Image` workflow with the test tag:
```bash
gh workflow run "Build and Push Docker Image" --repo InsForge/InsForge --ref <insforge-feature-branch> -f test_tag=<test-tag>
```
Then wait for the matching run to complete:
```bash
gh run list --repo InsForge/InsForge --workflow "Build and Push Docker Image" --limit 10
gh run watch --repo InsForge/InsForge <run-id>
```
Do not start the cross-repo E2E workflow until the image build succeeds.
## Decide Whether `agent-e2e` Must Change
Inspect the InsForge diff and compare it with deterministic fixture coverage in `InsForge/agent-e2e`.
For read-only checks, prefer remote GitHub access such as `gh api`, `gh repo view`, or remote file reads. Use a local checkout only when editing fixture files or when remote inspection is not enough to understand coverage.
Update `agent-e2e` when the InsForge change adds, removes, or changes behavior that the deterministic fixture should assert, including:
- API contract or validation behavior.
- Auth, permissions, RLS, storage, realtime, functions, schedules, AI, SDK, or CLI-visible behavior.
- Any regression that local tests cover but the release gate should also protect across the deployed runtime.
Do not update `agent-e2e` for internal refactors, docs-only changes, local test-only changes, or behavior already covered by the deterministic fixture with no assertion change needed.
Ignore `Support Desk Agent E2E (Exploratory)`. It is not part of this gate.
## If No E2E Test Update Is Needed
Run the deterministic fixture workflow from `agent-e2e` main:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref main -f insforge_tag=<test-tag>
```
Find and watch the run:
```bash
gh run list --repo InsForge/agent-e2e --workflow "Deterministic Fixture E2E" --limit 10
gh run watch --repo InsForge/agent-e2e <run-id>
```
## If E2E Tests Need An Update
Work in a local checkout of the remote `InsForge/agent-e2e` repo only for the fixture update.
1. Use an existing clean checkout or clone `https://github.com/InsForge/agent-e2e.git` into an isolated workspace.
2. Fetch `origin` and start from `origin/main`.
3. Create a branch named `codex/<short-topic>`.
4. Update only the deterministic fixture validators, fixtures, app assertions, and docs needed for the InsForge behavior change.
5. Run the smallest local validation that gives confidence:
```bash
npm run typecheck
npm run lint
npm run fixture:e2e:dry
```
Run broader validation when the changed fixture area supports it.
6. Open an `agent-e2e` PR for the fixture update.
7. Dispatch the deterministic fixture workflow from the `agent-e2e` branch that contains the fixture update:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref <agent-e2e-branch> -f insforge_tag=<test-tag>
```
8. Watch the run before proceeding with the InsForge PR.
## Interpret Results
If the deterministic fixture workflow passes:
- Link the run in the InsForge PR body or final PR notes.
- If an `agent-e2e` PR was required, link that PR too.
- Proceed with opening, updating, or submitting the InsForge OSS PR.
If the deterministic fixture workflow fails:
1. Inspect the failed job logs and uploaded artifact.
2. Identify whether the failure is caused by the InsForge implementation, the new or existing deterministic fixture, a transient infrastructure problem, or an unrelated existing failure.
3. Fix the correct branch:
- InsForge implementation bug: update the InsForge branch, rebuild the test image with the same tag or a new short retry tag, then rerun E2E.
- Fixture bug or missing assertion update: update the `agent-e2e` branch, rerun local validation, then rerun E2E from that branch.
- Transient infrastructure issue: rerun once after noting the evidence.
4. Do not submit the InsForge PR until the deterministic result is clear or the user explicitly accepts the risk.
## Report Back
When finished, report:
- Test tag used.
- InsForge image workflow run result.
- Whether `agent-e2e` changed.
- Deterministic Fixture E2E run result.
- Links to the InsForge PR, `agent-e2e` PR if any, and workflow runs.
.agents/skills/insforge-dev/shared-schemas/SKILL.md
+43
View File
@@ -0,0 +1,43 @@
---
name: shared-schemas
description: Use this skill when contributing to InsForge's shared schema package. This is for maintainers editing published Zod contracts, exported types, and shared API payload definitions consumed by InsForge packages in this repo and other InsForge tooling.
---
# InsForge Dev Shared Schemas
Use this skill for `packages/shared-schemas/` work in the InsForge repository.
## Scope
- `packages/shared-schemas/src/**`
- any code in this repo that consumes the changed contracts
- downstream InsForge tooling that depends on the published `@insforge/shared-schemas` package, including consumers not present in this repository
## Working Rules
1. Treat `packages/shared-schemas/` as the source of truth for cross-package payloads.
- If a request, response, or domain shape is shared across InsForge surfaces, define it here.
- Do not duplicate the same contract in package-local files.
- Remember that this package is not only for the repo's backend and dashboard packages. It is also consumed by other InsForge tooling, including MCP and SDK code that may live outside this repository.
2. Keep schemas organized by domain.
- Follow the existing `*.schema.ts` and `*-api.schema.ts` split when it fits the current package pattern.
- Keep `packages/shared-schemas/src/index.ts` aligned with the intended public API.
- Treat exported names and schema shapes as a public contract surface, not just an internal refactor target.
3. Use schema changes as a synchronization trigger.
- Update backend validation and response usage.
- Update shared dashboard services, hooks, and UI assumptions in `packages/dashboard/`.
- Check for import sites across `packages/*`, `frontend/`, and `backend/` before finishing.
- Call out likely downstream impact on MCP, SDK, or other external InsForge consumers when a change alters exported names, schema semantics, or payload shape.
- Be conservative with breaking changes. If a breaking contract change is necessary, make it explicit in the handoff.
- Never use the TypeScript `any` type. Shared contracts should stay explicit and trustworthy across package boundaries.
## Validation
- `cd packages/shared-schemas && npm run build`
- `cd backend && npx tsc --noEmit`
- `cd packages/dashboard && npm run typecheck`
Run package-specific tests as needed where behavior changed.
If external consumers such as MCP or SDK cannot be validated from this repo, say that clearly instead of implying they were covered.
.agents/skills/insforge-dev/ui/SKILL.md
+42
View File
@@ -0,0 +1,42 @@
---
name: ui
description: Use this skill when contributing to InsForge's reusable UI package. This is for maintainers editing design-system primitives, exports, styles, and package-level component behavior in the InsForge monorepo.
---
# InsForge Dev UI
Use this skill for `packages/ui/` work in the InsForge repository.
## Scope
- `packages/ui/src/components/**`
- `packages/ui/src/lib/**`
- `packages/ui/src/index.ts`
- `packages/ui/src/styles.css`
## Working Rules
1. Put only reusable primitives here.
- If the component is generic across dashboard features or other InsForge apps, it belongs in `packages/ui/`.
- If it is tightly coupled to one dashboard workflow but should ship to both OSS and cloud hosts, keep it in `packages/dashboard/`.
- If it is only for the self-hosting host app, keep it in `frontend/`.
2. Preserve the package's implementation style.
- Use `class-variance-authority` for variants when appropriate.
- Use the shared `cn()` helper for class merging.
- Follow the existing Radix-wrapper and typed-export patterns.
3. Keep the public surface in sync.
- Export new public components from `packages/ui/src/index.ts`.
- Avoid adding internal-only abstractions to the package surface unless they are meant to be consumed.
- Never use the TypeScript `any` type. Keep component props and exported helpers strictly typed.
4. Validate downstream impact.
- The shared dashboard package consumes this package directly, so UI changes can break `packages/dashboard/` even if `packages/ui/` itself builds cleanly.
## Validation
- `cd packages/ui && npm run build`
- `cd packages/ui && npm run typecheck`
Also validate `packages/dashboard/` when the changed component is used in the dashboard, and validate `frontend/` if the host app integration or CSS entrypoints changed.
.archive/docs/deprecated/insforge-auth-api.md
+213
View File
@@ -0,0 +1,213 @@
# Insforge OSS Authentication API Documentation
## Overview
Insforge uses JWT tokens and API keys for authentication. Store tokens in localStorage after login.
**All requests**: Use `Authorization: Bearer <token>` header (for both JWT tokens and API keys)
## Base URL
`http://localhost:7130`
## User Authentication
### Register New User
**POST** `/api/auth/users`
Body: `{"email": "user@example.com", "password": "password", "name": "User Name"}`
Returns: `{"accessToken": "...", "user": {"id": "...", "email": "...", "name": "...", "emailVerified": false, "createdAt": "...", "updatedAt": "..."}}`
**Note:** This creates an entry in the `users` table with the same `id` for profile data
### Login User
**POST** `/api/auth/sessions`
Body: `{"email": "user@example.com", "password": "password"}`
Returns: `{"accessToken": "...", "user": {"id": "...", "email": "...", "name": "...", "emailVerified": false, "createdAt": "...", "updatedAt": "..."}}`
### Get Current User
**GET** `/api/auth/sessions/current`
Headers: `Authorization: Bearer <accessToken>`
Returns: `{"user": {"id": "...", "email": "...", "role": "authenticated"}}`
**Note**: Returns LIMITED fields (id, email, role). For user profile data (nickname, avatar, bio, etc.), query `/api/database/records/users?id=eq.<user_id>`
**Common errors:**
- `401` with `"code": "MISSING_AUTHORIZATION_HEADER"` → No token provided
- `401` with `"code": "INVALID_TOKEN"` → Token expired or invalid
## Admin Authentication
### Admin Login
**POST** `/api/auth/admin/sessions`
Request:
```json
{
"username": "admin",
"password": "change-this-password"
}
```
Response:
```json
{
"accessToken": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
"projectAdmin": {
"subject": "local:admin",
"username": "admin"
}
}
```
```bash
# Mac/Linux
curl -X POST http://localhost:7130/api/auth/admin/sessions \
-H 'Content-Type: application/json' \
-d '{"username":"admin","password":"change-this-password"}'
# Windows PowerShell (use curl.exe)
curl.exe -X POST http://localhost:7130/api/auth/admin/sessions \
-H "Content-Type: application/json" \
-d '{\"username\":\"admin\",\"password\":\"change-this-password\"}'
```
## Error Response Format
All error responses follow this format:
```json
{
"code": "ERROR_CODE",
"message": "Human-readable error message"
}
```
Example error:
```json
{
"code": "INVALID_EMAIL",
"message": "Please provide a valid email"
}
```
## OAuth Support
Insforge supports Google and GitHub OAuth when configured with environment variables.
OAuth workflow:
1. End user initiates OAuth login and receives authorization URL from backend
2. After successful user authorization, Google/GitHub redirects to backend callback
3. Backend generates JWT token and redirects to the application page
Prerequisites:
1. Create Google or GitHub OAuth Application and obtain Client ID and Client Secret
2. Configure each platform's Client ID/Client Secret in InsForge backend (via Environment Variables)
### OAuth Endpoints
#### Get OAuth URL (Google/GitHub)
**GET** `/api/auth/oauth/:provider`
Parameters:
- `provider`: "google" or "github" in the URL path
- Query params: `?redirect_uri=http://localhost:3000/dashboard`
Returns: `{"authUrl": "https://accounts.google.com/..."}` - URL to redirect user to provider's OAuth page.
```bash
# Mac/Linux
curl -X GET "http://localhost:7130/api/auth/oauth/google?redirect_uri=http://localhost:3000/dashboard"
# Windows PowerShell (use curl.exe)
curl.exe -X GET "http://localhost:7130/api/auth/oauth/google?redirect_uri=http://localhost:3000/dashboard"
```
Example response:
```json
{
"authUrl": "https://accounts.google.com/o/oauth2/v2/auth?client_id=..."
}
```
#### OAuth Callback
The OAuth provider will redirect to:
- Google: `http://localhost:7130/api/auth/oauth/google/callback`
- GitHub: `http://localhost:7130/api/auth/oauth/github/callback`
After processing, backend redirects to your specified `redirect_uri` with JWT token in URL parameters:
- `access_token` - JWT authentication token
- `user_id` - User's unique ID
- `email` - User's email address
- `name` - User's display name
## User Profile Table
### Users Table
The `users` table stores **user profile data**:
- **✅ READ** via: `GET /api/database/records/users`
- **✅ WRITE** via: `PATCH /api/database/records/users?id=eq.<user_id>`
- **✅ Foreign keys allowed** - reference `users.id`
- **IMPORTANT**: Add columns to this table for profile data instead of creating separate profile tables
**Schema:**
- `id` - User ID (UUID, primary key)
- `nickname` - Display name (text, nullable)
- `avatar_url` - Profile picture URL (text, nullable)
- `bio` - User biography (text, nullable)
- `birthday` - Birth date (date, nullable)
- `created_at` - Account creation timestamp
- `updated_at` - Last update timestamp
**Note:** Email and name from auth are returned by Auth API, not stored in users table
Example - Create table with user reference:
```json
{
"table_name": "posts",
"columns": [
{
"name": "title",
"type": "string",
"nullable": false,
"is_unique": false
},
{
"name": "user_id",
"type": "string",
"nullable": false,
"is_unique": false,
"foreign_key": {
"reference_table": "users",
"reference_column": "id",
"on_delete": "CASCADE",
"on_update": "CASCADE"
}
}
]
}
```
### Available Tables
- **users** - User profile data (read/write access)
- Use this table for foreign key references
- Update profiles with PATCH requests
## Headers Summary
| API Type | Header Required |
|----------|----------------|
| Auth endpoints | None |
| Database/Storage | `Authorization: Bearer <accessToken>` |
| MCP testing only | `x-api-key: <key>` |
## Critical Notes
1. `/api/auth/sessions/current` returns `{"user": {...}}` - nested, not root level
2. `/api/auth/sessions/current` only has: id, email, role (limited fields)
3. Full user profile: `GET /api/database/records/users?id=eq.<id>`
4. POST to database requires `[{...}]` array format always
5. Auth endpoints (register/login): no headers needed
6. Protected endpoints: `Authorization: Bearer <accessToken>`
.archive/docs/deprecated/insforge-auth-sdk.md
+100
View File
@@ -0,0 +1,100 @@
# InsForge Auth SDK
## Setup
```javascript
import { createClient } from '@insforge/sdk';
const client = createClient({ baseUrl: 'http://localhost:7130' });
```
## Methods
### signUp
```javascript
await client.auth.signUp({ email, password, name? })
// Returns: { data: { accessToken, user }, error }
// user: { id, email, name, emailVerified, createdAt, updatedAt }
// Token auto-stored
```
### signInWithPassword
```javascript
await client.auth.signInWithPassword({ email, password })
// Returns: { data: { accessToken, user }, error }
// Token auto-stored
```
### signInWithOAuth
```javascript
const { data, error } = await client.auth.signInWithOAuth({
provider: 'google'|'github',
redirectTo: window.location.origin,
skipBrowserRedirect: true
})
// Returns: { data: { url, provider }, error }
// Manual redirect required
if (data?.url) {
window.location.href = data.url
}
// ⚠️ IMPORTANT: No callback handling needed!
// After OAuth, user returns to redirectTo URL already authenticated
// The SDK automatically:
// - Handles the OAuth callback
// - Stores the JWT token
// - Makes user available via getCurrentUser()
// ❌ DON'T DO THIS (not needed):
// const accessToken = urlParams.get('access_token')
// const userId = urlParams.get('user_id')
// ✅ DO THIS INSTEAD (after redirect back):
const { data: userData } = await client.auth.getCurrentUser()
```
### getCurrentUser
```javascript
await client.auth.getCurrentUser()
// Returns: { data: { user: { id, email, role }, profile: {...} }, error }
// Makes API call to validate token and fetch profile
```
### getCurrentSession
```javascript
await client.auth.getCurrentSession()
// Returns: { data: { session: { accessToken, user } }, error }
// From localStorage, no API call
```
### getProfile
```javascript
await client.auth.getProfile(userId)
// Returns: { data: { id, nickname, bio, ... }, error }
// Returns single object, not array!
```
### setProfile
```javascript
await client.auth.setProfile({ nickname, bio, avatar_url })
// Returns: { data: { id, nickname, bio, ... }, error }
// Returns single object, not array!
```
### signOut
```javascript
await client.auth.signOut()
// Returns: { error }
// Clears token from storage
```
## Error Codes
- `INVALID_EMAIL`
- `WEAK_PASSWORD`
- `USER_ALREADY_EXISTS`
- `INVALID_CREDENTIALS`
- `INVALID_TOKEN`
## Notes
- Tokens stored in localStorage (browser) or memory (Node.js)
- All requests after login automatically include token
- User profile data in `users` table, not auth
.archive/docs/deprecated/insforge-db-api.md
+359
View File
@@ -0,0 +1,359 @@
# Insforge OSS Database API Documentation
## API Basics
**Base URL:** `http://localhost:7130`
**Note:** Avoid special characters (!,$,`,\) in curl command data - they can cause bash interpretation issues. Use simple text for testing.
**Authentication Requirements:**
- **READ operations (GET):** No authentication required - public access by default
- **WRITE operations (POST/PATCH/DELETE):** Requires `Authorization: Bearer <token>` header (JWT token or API key for MCP testing)
**Important: How Authentication Works**
1. Login returns a **JWT access token** - e.g., `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...`
2. Just use: `Authorization: Bearer <token>` in your requests
3. API keys (starting with `ik_`) can also be used as Bearer tokens for testing
**Critical:** Always call `get-backend-metadata` first to understand current database structure
**Critical:** POST body must be arrays `[{...}]`, query filters `?field=eq.value`, add header `Prefer: return=representation` to return created data - follows PostgREST design (not traditional REST)
## Table Operations (Use MCP Tools)
### Available MCP Tools
1. **get-backend-metadata** - Get current database structure (always start here)
2. **create-table** - Create new table with explicit schema
3. **update-table-schema** - Alter existing table schema
4. **delete-table** - Remove table completely
5. **get-table-schema** - Get specific table structure
### Column Types
- `string` - Text data
- `integer` - Whole numbers
- `float` - Decimal numbers
- `boolean` - True/false values
- `datetime` - Date and time
- `json` - JSON objects
- `uuid` - Unique identifiers
## Record Operations (Use REST API)
### Base URL
`/api/database/records/:tableName`
### Query Records
**GET** `/api/database/records/:tableName`
Query parameters:
- `limit` - Maximum records (default: 100)
- `offset` - Skip records for pagination
- `order` - Sort by field (e.g., `createdAt.desc`)
- PostgREST filters: `field=eq.value`, `field=gt.value`, etc.
Response: Array of records with auto-generated `id`, `created_at`, `updated_at` fields
Example:
```bash
# Windows PowerShell: use curl.exe
curl -X GET "http://localhost:7130/api/database/records/posts?limit=10"
```
### Create Records
**POST** `/api/database/records/:tableName`
**AUTHENTICATION REQUIRED** - Must include `Authorization: Bearer <token>`
**CRITICAL**: Request body MUST be an array, even for single records!
**⚠️ IMPORTANT: Default Response Behavior**
- **By default, POST requests return an empty array `[]`**
- **To get the created records in the response, you MUST include the header:**
```
Prefer: return=representation
```
- **Without this header, you get no data back, just an empty array!**
Send array of records:
```json
[
{
"field1": "value1",
"field2": "value2"
}
]
```
For a single record, still wrap in array:
```json
[
{
"name": "John Doe",
"email": "john@example.com"
}
]
```
Response format (WITHOUT `Prefer` header - default):
```json
[]
```
Response format (WITH `Prefer: return=representation` header):
```json
[
{
"id": "248373e1-0aea-45ce-8844-5ef259203749",
"name": "John Doe",
"email": "john@example.com",
"createdAt": "2025-07-18T05:37:24.338Z",
"updatedAt": "2025-07-18T05:37:24.338Z"
}
]
```
Example:
```bash
# Mac/Linux
curl -X POST http://localhost:7130/api/database/records/comments \
-H 'Authorization: Bearer <token>' \
-H 'Content-Type: application/json' \
-H 'Prefer: return=representation' \
-d '[{"user_id": "from-localStorage", "post_id": "post-uuid", "content": "Great"}]'
# Windows PowerShell (use curl.exe) - different quotes needed for nested JSON
curl.exe -X POST http://localhost:7130/api/database/records/comments \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-H "Prefer: return=representation" \
-d '[{\"user_id\": \"from-localStorage\", \"post_id\": \"post-uuid\", \"content\": \"Great\"}]'
```
### Update Record
**PATCH** `/api/database/records/:tableName?id=eq.uuid`
**AUTHENTICATION REQUIRED**
**⚠️ IMPORTANT: PATCH Limitations**
- **PostgREST does NOT support SQL expressions** like `count + 1`
- You must fetch the current value and calculate in your code:
```javascript
// ❌ WRONG - This will NOT work
await api.patch(`/api/database/records/posts?id=eq.${postId}`, {
comments_count: 'comments_count + 1' // PostgREST doesn't evaluate expressions!
});
// ✅ CORRECT - Fetch and calculate
const post = await api.get(`/api/database/records/posts?id=eq.${postId}`);
await api.patch(`/api/database/records/posts?id=eq.${postId}`, {
comments_count: post.data[0].comments_count + 1
});
```
**Default Response Behavior**
- **By default, PATCH requests return an empty array `[]`**
- **To get the updated record in the response, you MUST include the header:**
```
Prefer: return=representation
```
Send fields to update:
```json
{
"field1": "new_value"
}
```
Response format (WITHOUT `Prefer: return=representation` header - default):
```json
""
```
Response format (WITH `Prefer: return=representation` header):
```json
[
{
"id": "123e4567-e89b-12d3-a456-426614174000",
"field1": "new_value",
"createdAt": "2025-01-01T00:00:00Z",
"updatedAt": "2025-01-21T11:00:00Z"
}
]
```
### Delete Record
**DELETE** `/api/database/records/:tableName?id=eq.uuid`
**AUTHENTICATION REQUIRED**
**⚠️ IMPORTANT: Delete Behavior**
- **Without `Prefer: return=representation`**: Returns `204 No Content` (no body)
- **With `Prefer: return=representation`**: Returns `200 OK` with:
- `[{...}]` - Array containing deleted record(s) if found
- `[]` - Empty array if record didn't exist
- **DELETE is idempotent**: No error if record doesn't exist
Response format (WITHOUT `Prefer` header - default):
```
204 No Content (no body)
```
Response format (WITH `Prefer: return=representation` header):
```json
// If record existed and was deleted:
[
{
"id": "123e4567-e89b-12d3-a456-426614174000",
"name": "Deleted User",
"createdAt": "2025-01-01T00:00:00Z",
"updatedAt": "2025-01-21T11:00:00Z"
}
]
// If record didn't exist (already deleted or never existed):
[]
```
## Error Response Format
All error responses follow this format:
```json
{
"error": "ERROR_CODE",
"message": "Human-readable error message",
"statusCode": 400,
"nextActions": "Suggested action to resolve the error"
}
```
Example error:
```json
{
"error": "TABLE_NOT_FOUND",
"message": "Table 'nonexistent' does not exist",
"statusCode": 404,
"nextActions": "Check table name and try again"
}
```
## Pagination
For paginated results, use the `Range` header:
```bash
# Windows PowerShell: use curl.exe
curl "http://localhost:7130/api/database/records/posts" \
-H "Range: 0-9" \
-H "Prefer: count=exact"
```
Response includes `Content-Range` header:
```
Content-Range: 0-9/100 # Shows items 0-9 out of 100 total
```
Without `Prefer: count=exact`, you get: `Content-Range: 0-9/*` (no total count)
## 🚨 Working with User Data
**The `users` table stores user profiles:**
- **✅ READ**: `GET /api/database/records/users` - Get user profiles
- **✅ WRITE**: `PATCH /api/database/records/users?id=eq.<user_id>` - Update profiles
**Schema:**
- `id` - User ID (UUID, references auth system)
- `nickname` - Display name (text, nullable)
- `avatar_url` - Profile picture URL (text, nullable)
- `bio` - User biography (text, nullable)
- `birthday` - Birth date (date, nullable)
- `created_at` - Profile creation timestamp
- `updated_at` - Last update timestamp
**Important:**
- User accounts (email, password) are managed via Auth API only
- The `users` table is automatically created when a user registers
- Use `users.id` for foreign key references in your tables
**Creating tables with user references:**
```json
{
"table_name": "posts",
"columns": [
{"name": "user_id", "type": "string", "nullable": false,
"foreign_key": {"reference_table": "users", "reference_column": "id",
"on_delete": "CASCADE", "on_update": "CASCADE"}},
{"name": "content", "type": "string", "nullable": false}
]
}
```
## 🚨 Critical: Always Include user_id
**Every user-related table MUST include user_id field from localStorage:**
```javascript
// Frontend: Get user_id from localStorage after login
const userId = localStorage.getItem('user_id');
```
```bash
# ❌ WRONG - Missing user_id
curl -X POST http://localhost:7130/api/database/records/comments \
-H "Authorization: Bearer TOKEN" \
-d '[{"content": "Great post"}]'
# ✅ CORRECT - Includes user_id
# Mac/Linux
curl -X POST http://localhost:7130/api/database/records/comments \
-H 'Authorization: Bearer TOKEN' \
-H 'Prefer: return=representation' \
-d '[{"content": "Great post", "user_id": "user-uuid-from-localStorage"}]'
# Windows PowerShell (use curl.exe) - different quotes needed for nested JSON
curl.exe -X POST http://localhost:7130/api/database/records/comments \
-H "Authorization: Bearer TOKEN" \
-H "Prefer: return=representation" \
-d '[{\"content\": \"Great post\", \"user_id\": \"user-uuid-from-localStorage\"}]'
```
**Required for all user-related operations:**
- Creating posts, comments, likes, follows
- Any table with a `user_id` foreign key
- Without it, your INSERT will fail with missing field error
## Important Rules
1. **Authentication Summary**:
| Operation | Auth Required | Header |
|-----------|--------------|--------|
| GET (read) | ❌ No | None needed |
| POST (create) | ✅ Yes | `Authorization: Bearer <token>` |
| PATCH (update) | ✅ Yes | `Authorization: Bearer <token>` |
| DELETE | ✅ Yes | `Authorization: Bearer <token>` |
2. **Auto-Generated Fields**
- `id` - UUID primary key (auto-generated)
- `createdAt` - Timestamp (auto-set)
- `updatedAt` - Timestamp (auto-updated)
2. **System Tables**
- Tables prefixed with `_` are system tables (protected)
- User profiles stored in `users` table (read/write allowed)
- Account management only through Auth API (register/login)
3. **Common PostgREST Errors**:
```json
{"code": "42501", "message": "permission denied for table comments"}
// Means: User not authenticated for write operation
{"code": "PGRST301", "message": "JWSError (CompactDecodeError Invalid number of parts: Expected 3 parts; got 1)"}
// Means: Invalid or expired token - user needs to login again
```
4. **Remember**
- READ operations are public (no auth needed)
- WRITE operations require token from login
- POST needs array `[{...}]` even for single record
- Add `Prefer: return=representation` to see created/updated data
- PATCH cannot use SQL expressions - calculate in JavaScript
- Tokens from login work directly as Bearer tokens
- Always include `user_id` in user-related tables
.archive/docs/deprecated/insforge-db-sdk.md
+140
View File
@@ -0,0 +1,140 @@
# InsForge Database SDK
## Setup
```javascript
import { createClient } from '@insforge/sdk';
const client = createClient({ baseUrl: 'http://localhost:7130' });
```
## Query Builder
### from(table)
```javascript
client.database.from('posts') // Returns QueryBuilder
```
## Operations
### select
```javascript
.select() // All columns
.select('id, title') // Specific columns
.select('*, user:user_id(name, email)') // Foreign key expansion
```
### insert
```javascript
.insert({ title: 'Hello' }).select() // Single record with data returned
.insert([{...}, {...}]).select() // Multiple records with data returned
// Note: Always sends array to API
// Without .select(), returns null data
```
### update
```javascript
.update({ title: 'Updated' }).eq('id', '123').select()
// Must chain with filter and .select() to return data
```
### delete
```javascript
.delete().eq('id', '123').select()
// Must chain with filter and .select() to return data
```
### upsert
```javascript
.upsert({ id: '123', title: 'New or Update' }).select()
// Updates if exists, inserts if not
// Use .select() to return data
```
## Filters
```javascript
.eq('col', value) // column = value
.neq('col', value) // column != value
.gt('col', value) // column > value
.gte('col', value) // column >= value
.lt('col', value) // column < value
.lte('col', value) // column <= value
.like('col', '%pat%') // LIKE pattern
.ilike('col', '%pat%') // ILIKE pattern
.is('col', null) // IS NULL
.in('col', [1,2,3]) // IN array
.or('status.eq.active,status.eq.pending') // OR condition
.and('price.gte.100,price.lte.500') // Explicit AND
.not('deleted', 'is.true') // NOT condition
```
### OR Examples
```javascript
// Simple OR
.or('status.eq.active,status.eq.pending')
// WHERE status = 'active' OR status = 'pending'
// OR with other filters (implicit AND)
.eq('user_id', '123')
.or('status.eq.draft,status.eq.published')
// WHERE user_id = '123' AND (status = 'draft' OR status = 'published')
// Complex OR with NOT
.or('age.lt.18,age.gt.65,not.is_active.is.true')
// WHERE age < 18 OR age > 65 OR NOT is_active
```
## Modifiers
```javascript
.order('col') // ASC
.order('col', { ascending: false }) // DESC
.limit(10)
.offset(20)
.range(0, 9) // Headers: Range: 0-9
.single() // Return object not array
.count('exact') // Include total count
```
## Execute
```javascript
// Methods are thenable
const { data, error } = await client.database
.from('posts')
.select()
.eq('user_id', '123')
.limit(10);
// data: array or null
// error: { message, statusCode, code } or null
```
## Foreign Key Expansion
```javascript
// PostgREST syntax
.select('*, user:user_id(name, email)')
// Response:
{
id: '123',
title: 'Post',
user_id: '456',
user: { name: 'John', email: 'john@example.com' }
}
```
## Users Table
```javascript
// Profile data (not auth)
await client.database.from('users').select().eq('id', userId).single()
await client.database.from('users').update({ nickname, avatar_url }).eq('id', userId).select()
```
## Notes
- Uses PostgREST under the hood
- POST requires array format `[{...}]`
- All methods return QueryBuilder for chaining
- Execute returns `{ data, error }`
.archive/docs/deprecated/insforge-debug-sdk.md
+157
View File
@@ -0,0 +1,157 @@
# InsForge Debug SDK
## Common SDK Errors
### Auth Errors
```javascript
// Token not stored
{ code: 'MISSING_AUTHORIZATION_HEADER' }
// Fix: User needs to login
// Token expired
{ code: 'INVALID_TOKEN' }
// Fix: client.auth.signInWithPassword()
// Invalid credentials
{ code: 'INVALID_CREDENTIALS' }
// Fix: Check email/password
```
### Database Errors
```javascript
// Table doesn't exist
{ statusCode: 404, message: 'Table not found' }
// Fix: Use MCP tool to create table
// Missing required field
{ code: '23502', message: 'null value in column "user_id"' }
// Fix: Include all required fields
// Foreign key violation
{ code: '23503', message: 'violates foreign key constraint' }
// Fix: Referenced record must exist
// Permission denied
{ code: '42501', message: 'permission denied for table' }
// Fix: User must be authenticated
```
## Debug Techniques
### 1. Check Authentication
```javascript
const { data, error } = await client.auth.getCurrentUser();
if (error) {
console.log('Not authenticated:', error.message);
// Need to login first
}
```
### 2. Verify Token Storage
```javascript
const { data } = await client.auth.getCurrentSession();
console.log('Stored token:', data?.session?.accessToken ? 'Present' : 'Missing');
```
### 3. Test Database Connection
```javascript
// Simple query to test connection
const { error } = await client.database.from('users').select().limit(1);
if (error) {
console.log('Database issue:', error);
}
```
### 4. Enable Network Inspection
```javascript
// Browser: Open DevTools > Network tab
// Check request headers for Authorization: Bearer token
// Check response for error details
```
### 5. Test With Raw HTTP
```javascript
// When SDK fails, test endpoint directly
const response = await fetch('http://localhost:7130/api/database/records/posts', {
headers: {
'Authorization': 'Bearer ' + token
}
});
const data = await response.json();
console.log('Raw response:', data);
```
## Common Issues
### Empty Response
```javascript
// INSERT/UPDATE/DELETE return empty without Prefer header
// SDK handles this automatically, but check if using raw API
```
### Array Format
```javascript
// POST requires array
// ❌ Wrong: { title: 'Test' }
// ✅ Right: [{ title: 'Test' }]
// SDK handles this automatically
```
### Token Not Persisting
```javascript
// Check storage adapter
const client = createClient({
baseUrl: 'http://localhost:7130',
});
```
### CORS Issues
```javascript
// Backend allows all origins by default
// If modified, check backend CORS settings
```
## Test Utilities
### Full Test Suite
```javascript
async function testSDK() {
const email = `test-${Date.now()}@example.com`;
// Test auth
console.log('Testing auth...');
const { error: signUpError } = await client.auth.signUp({ email, password: 'test123' });
if (signUpError) return console.error('SignUp failed:', signUpError);
// Test current user
const { data: user, error: userError } = await client.auth.getCurrentUser();
if (userError) return console.error('GetUser failed:', userError);
console.log('✅ Auth working, user:', user.user.id);
// Test database
console.log('Testing database...');
const { data: posts, error: dbError } = await client.database.from('posts').select().limit(1);
if (dbError) return console.error('Database failed:', dbError);
console.log('✅ Database working, posts:', posts.length);
// Test insert
const { error: insertError } = await client.database
.from('posts')
.insert({ title: 'Test', user_id: user.user.id })
.select();
if (insertError) return console.error('Insert failed:', insertError);
console.log('✅ Insert working');
}
testSDK().catch(console.error);
```
## Backend Verification
```javascript
// Check backend is running
fetch('http://localhost:7130/api/health')
.then(r => r.json())
.then(d => console.log('Backend:', d))
.catch(e => console.error('Backend not running'));
```
.archive/docs/deprecated/insforge-debug.md
+65
View File
@@ -0,0 +1,65 @@
# Insforge Debug Guide
## When Your API Code Fails
**Start here** → `get-instructions` and `get-backend-metadata` (understand system state)
**Read docs** → `get-db-api`, `get-auth-api`, `get-storage-api` (read ALL of them)
**Check table** → `get-table-schema` with your table name
**Test endpoint** → Use curl with exact API format from docs
## Critical Rule: Read Documentation First
Before debugging, you MUST read all documentation to understand how the API works.
## Common API Issues
**Table created but API fails** → Check field names match schema exactly
**Array required** → PostgREST requires POST requests as arrays `[{...}]`
**Foreign key error** → Parent record must exist before child
**Permission denied** → Write operations need `Authorization: Bearer <accessToken>`
**JWSError** → JWT token expired or invalid - user needs to login again
**PATCH increment fails** → PostgREST doesn't support SQL expressions like `count + 1`
## Debug Workflow
1. **Always** call `get-backend-metadata` first
2. Read the relevant API documentation completely
3. Check your table schema matches your API calls
4. Test with curl using exact format from docs
5. Verify response matches documentation
### Example Debug Tests
```bash
# Test GET endpoint
# Windows PowerShell: use curl.exe
curl -X GET http://localhost:7130/api/database/records/your_table \
-H "Authorization: Bearer YOUR_TOKEN" | jq .
# Test POST with array format
# Mac/Linux
curl -X POST http://localhost:7130/api/database/records/your_table \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_TOKEN' \
-H 'Prefer: return=representation' \
-d '[{"field": "value"}]' | jq .
# Windows PowerShell (use curl.exe) - different quotes for nested JSON
curl.exe -X POST http://localhost:7130/api/database/records/your_table \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Prefer: return=representation" \
-d '[{\"field\": \"value\"}]' | jq .
```
## Key Rules
- Backend runs on port 7130
- **READ operations**: No authentication required
- **WRITE operations**: Need `Authorization: Bearer <accessToken>` header
- POST requests must be arrays `[{...}]`
- System tables (prefixed with _) need special APIs
- No escaped characters in JSON
- Login/register returns JWT tokens directly in `accessToken` field
**Remember**: MCP creates the structure, but you must follow API documentation exactly to use it.
.archive/docs/deprecated/insforge-instructions.md
+124
View File
@@ -0,0 +1,124 @@
# Insforge OSS Instructions
## What Insforge OSS Does
Backend-as-a-service with database, authentication, and file storage.
**Key Concept**: InsForge replaces your traditional backend - implement business logic by calling database operations directly. Instead of building API endpoints, use our database API as your application's backend.
## 🚨 Project Setup
**Create your app in a NEW directory, not inside `insforge/`**
The `insforge/` directory is the BaaS platform. Your app should live elsewhere:
```
~/projects/
├── insforge/ # ← BaaS platform (don't work here)
└── my-app/ # ← Your new app (work here)
```
## When to Use Tools
**MUST DO FIRST** → Download project rules: `download-project-rules`
**Start here** → `get-backend-metadata` (shows current database state)
**Need docs** → `get-db-api`, `get-auth-api`, or `get-storage-api`
**Create table** → `create-table` with explicit schema
**Work with data** → Use database API endpoints directly
## Critical Rule:
**MUST DO FIRST** → Call`download-project-rules` to download project rules
## Critical Rule: Check Metadata First
Before ANY database operation, call `get-backend-metadata` to get the current database state.
## Standard Workflow
1. **Always** call `get-backend-metadata` first
2. Check `get-instructions` if unfamiliar with the system
3. Create tables with `create-table` if needed
4. Use database API to insert/query/update/delete records
5. Call `get-backend-metadata` again to verify changes
## Key Rules
- Frequently check `get-instructions` and `get-backend-metadata`
- Always define explicit table schemas (no assumptions)
- Every table gets auto ID, created_at, updated_at fields
- **Database operations require**: JWT token (Authorization: Bearer header)
- **API keys are for MCP testing** (use tokens for production)
- File uploads work automatically with multipart/form-data
## Authentication Requirements
### Database Operations Need Authentication Token:
1. **JWT Token**: `Authorization: Bearer your-jwt-token` - Authenticates the user
**Important Note about API Keys:**
- The `x-api-key` header is ONLY used for MCP (Model Context Protocol) testing
- Production applications should NEVER use API keys
- Always use JWT tokens from user/admin authentication for real applications
Without the Bearer token, you'll get "permission denied" errors when trying to insert, update, or delete records.
### Getting Authentication:
```bash
# Works on both Windows and Unix (Windows PowerShell: use curl.exe)
# 1. First login to get JWT token
curl -X POST http://localhost:7130/api/auth/admin/sessions \
-H "Content-Type: application/json" \
-d "{\"username\":\"admin\",\"password\":\"your-password\"}"
# Response includes token: {"accessToken": "eyJ...", "projectAdmin": {...}}
# Works on both Windows and Unix (Windows PowerShell: use curl.exe)
# 2. Use the auth token for database operations
curl -X POST http://localhost:7130/api/database/records/products \
-H "Authorization: Bearer eyJ..." \
-H "Content-Type: application/json" \
-d "[{\"name\": \"Product\", \"price\": 99.99}]"
```
## Example: Comment Upvoting Feature
- Check current tables: `get-backend-metadata`
- Create comment_votes table: `create-table` with user_id, comment_id, vote_type fields
- Frontend upvote action: `POST /api/database/records/comment_votes` with vote data
- Frontend display scores: `GET /api/database/records/comment_votes?comment_id=eq.123` to count votes
- No separate backend needed - frontend calls InsForge database API directly
## Critical Rule: Test API Endpoints with curl
After creating or modifying any API endpoint, always test it with curl to verify it works correctly.
**Note:** Avoid special characters (!,$,`,\) in curl command data - they can cause bash interpretation issues. Use simple text for testing:
```bash
# Works on both Windows and Unix (Windows PowerShell: use curl.exe)
# Example: Test creating a record (requires JWT token)
curl -X POST http://localhost:7130/api/database/records/posts \
-H "x-api-key: your-api-key" \
-H "Authorization: Bearer your-jwt-token" \
-H "Content-Type: application/json" \
-d '[{\"title\": \"Test Post\", \"content\": \"Test content\"}]'
# Works on both Windows and Unix (Windows PowerShell: use curl.exe)
# Example: Test querying records (requires both API key and JWT token)
curl http://localhost:7130/api/database/records/posts?id=eq.123 \
-H "x-api-key: your-api-key" \
-H "Authorization: Bearer your-jwt-token"
# Works on both Windows and Unix (Windows PowerShell: use curl.exe)
# Example: Test authentication
curl -X POST http://localhost:7130/api/auth/users \
-H "Content-Type: application/json" \
-d '{\"email\": \"test@example.com\", \"password\": \"testpass123\"}'
```
Always include:
- **Both headers for database operations**: x-api-key AND Authorization: Bearer token
- Correct HTTP method (GET, POST, PATCH, DELETE)
- Valid JSON payload for POST/PATCH requests (remember: POST requires array format `[{...}]`)
- Query parameters for filtering GET requests
- Prefer: return=representation header if you want to see the created/updated records
.archive/docs/deprecated/insforge-project.md
+118
View File
@@ -0,0 +1,118 @@
---
description: Insforge AI Development Rules - Essential guidelines for BaaS platform development
globs:
alwaysApply: true
---
# Insforge Development Rules
## Core Identity
You are an exceptional software developer using Insforge Backend to assist building the product. Make it visually stunning, content-rich, professional-grade UIs.
## 🔴 MANDATORY: cURL Test at EVERY Step
**For AI Agents: Test with cURL repeatedly throughout development:**
1. **Before coding** → Test endpoint exists, check response format
2. **During coding** → Test when confused about any API behavior
3. **After coding** → Test complete user journey end-to-end
4. **When debugging** → Test to see actual vs expected responses
```bash
# Mac/Linux
curl -X POST http://localhost:7130/api/[endpoint] \
-H 'Content-Type: application/json' \
-d '[{"key": "value"}]' | jq .
# Windows PowerShell (use curl.exe) - different quotes for nested JSON
curl.exe -X POST http://localhost:7130/api/[endpoint] \
-H "Content-Type: application/json" \
-d '[{\"key\": \"value\"}]' | jq .
```
**You WILL get it wrong without testing. Test early, test often.**
## Critical Architecture Points
When in doubt, read instructions documents again.
## 🚨 CRUD Operations - PostgREST NOT RESTful
### PostgREST Database API Behavior
**Critical PostgREST Rules:**
1. **POST requires array**: `[{...}]` even for single record
2. **Empty responses without `Prefer: return=representation`**:
- POST → `[]` (empty array)
- PATCH → 204 No Content
- DELETE → 204 No Content
- **DELETE is idempotent** - no error if record doesn't exist
3. **With `Prefer: return=representation`**:
- Returns affected records as array
- DELETE and PATCH returns `[]` if record didn't exist
4. **Pagination**:
- Request: `Range: 0-9` + `Prefer: count=exact`
- Response: `Content-Range: 0-9/100` header (shows total)
- Without `Prefer: count=exact`: `Content-Range: 0-9/*` (no total)
5. **Query syntax**: `?field=operator.value`
- `?id=eq.123` (equals)
- `?age=gt.30` (greater than)
- `?name=like.*john*` (pattern match)
## Auth Operations:
### 🚨 IMPORTANT: Correct Auth Endpoints
- **Register**: `POST /api/auth/users` - Create new user account
- **Login**: `POST /api/auth/sessions` - Authenticate existing user
- **Admin Login**: `POST /api/auth/admin/sessions` - Admin authentication
- **Current User**: `GET /api/auth/sessions/current` - Get authenticated user info
- `/api/auth/sessions/current` returns `{"user": {...}}` - nested structure
- Store JWT tokens and include as `Authorization: Bearer {accessToken}` header
- **Note**: Login/register returns `{accessToken, user}` with JWT token
### Regular API Response Format
**⚠️ IMPORTANT: Frontend Error Handling**
- **PARSE** backend responses and display user-friendly messages
- **DO NOT** show raw API responses directly to users
- **TRANSFORM** error details into readable, actionable messages
- Success: Data directly (object/array)
- Error: `{error, message, statusCode}`
- Empty POST/PATCH/DELETE: Add `Prefer: return=representation`
### 🚨 Storage API Rules
- **Upload Methods**:
- **PUT** `/api/storage/buckets/{bucket}/objects/{filename}` - Upload with specific key
- **POST** `/api/storage/buckets/{bucket}/objects` - Upload with auto-generated key
- **Authentication**: Upload operations require `Authorization: Bearer {accessToken}`
- **Generate Unique Filenames**: Use POST for auto-generated keys to prevent overwrites
- **Multipart Form**: Use FormData for file uploads
- **URL Format**: Storage returns **ABSOLUTE URLs** (e.g., `http://localhost:7130/api/storage/buckets/{bucket}/objects/{filename}`)
- **IMPORTANT**: URLs are complete and ready to use - no need to prepend host or path
- Use the URL directly in `<img src>` or fetch requests
## 🔥 Test EVERY Endpoint
**Backend runs on port 7130**
Always test with cURL before UI integration:
- Include `Authorization: Bearer {accessToken}` for auth
- Add `Prefer: return=representation` to see created data
- Windows PowerShell: use curl.exe
```bash
# Mac/Linux
curl -X POST http://localhost:7130/api/database/records/posts \
-H 'Authorization: Bearer TOKEN' \
-H 'Content-Type: application/json' \
-H 'Prefer: return=representation' \
-d '[{"user_id": "from-localStorage", "caption": "Test"}]'
# Windows PowerShell (use curl.exe) - different quotes for nested JSON
curl.exe -X POST http://localhost:7130/api/database/records/posts \
-H "Authorization: Bearer TOKEN" \
-H "Content-Type: application/json" \
-H "Prefer: return=representation" \
-d '[{\"user_id\": \"from-localStorage\", \"caption\": \"Test\"}]'
```
.archive/docs/deprecated/insforge-storage-api.md
+279
View File
@@ -0,0 +1,279 @@
# Insforge OSS Storage API Documentation
## API Basics
**Base URL:** `http://localhost:7130`
**Authentication:**
- **Upload operations (PUT/POST/DELETE):** Requires `Authorization: Bearer <token>` header
- **Download from public buckets:** No authentication required
- **List/manage buckets:** Requires authentication
- **API keys are for MCP testing** (use tokens for production)
**System:** Bucket-based storage with public/private access control
### 🔴 CRITICAL: URL Format
**Storage returns ABSOLUTE URLs** that are ready to use directly:
- Response `url` field contains complete URL: `http://localhost:7130/api/storage/buckets/{bucket}/objects/{filename}`
- **DO NOT prepend host or modify the URL** - use it exactly as returned
- URLs work directly in `<img src>`, `<video src>`, or fetch requests
- Example: `"url": "http://localhost:7130/api/storage/buckets/avatars/objects/user123.jpg"`
## Bucket Operations (Use MCP Tools)
### Available MCP Tools
1. **create-bucket** - Create bucket (`bucketName`, `isPublic`: true by default)
2. **list-buckets** - List all buckets
3. **delete-bucket** - Delete bucket
## Object Operations (Use REST API)
### Base URL
`/api/storage/buckets/:bucketName/objects/:objectKey`
### Upload Object with Specific Key
**PUT** `/api/storage/buckets/:bucketName/objects/:objectKey`
Send object as multipart/form-data:
```javascript
const formData = new FormData();
formData.append('file', fileObject);
```
Returns:
```json
{
"bucket": "test-images",
"key": "test.txt",
"size": 30,
"mimeType": "text/plain",
"uploadedAt": "2025-07-18T04:32:13.801Z",
"url": "http://localhost:7130/api/storage/buckets/test-images/objects/test.txt"
}
```
Example curl:
```bash
# Windows PowerShell: use curl.exe
curl -X PUT http://localhost:7130/api/storage/buckets/avatars/objects/user123.jpg \
-H "Authorization: Bearer YOUR_SESSION_TOKEN" \
-F "file=@/path/to/image.jpg"
```
### Upload Object with Auto-Generated Key
**POST** `/api/storage/buckets/:bucketName/objects`
Request:
```bash
# Windows PowerShell: use curl.exe
curl -X POST http://localhost:7130/api/storage/buckets/posts/objects \
-H "Authorization: Bearer YOUR_SESSION_TOKEN" \
-F "file=@/path/to/image.jpg"
```
Response:
```json
{
"bucket": "avatars",
"key": "image-1737546841234-a3f2b1.jpg",
"size": 15234,
"mimeType": "image/jpeg",
"uploadedAt": "2025-07-18T04:32:13.801Z",
"url": "http://localhost:7130/api/storage/buckets/avatars/objects/image-1737546841234-a3f2b1.jpg"
}
```
### Download Object
**GET** `/api/storage/buckets/:bucketName/objects/:objectKey`
Returns the actual object content with appropriate content-type headers.
**Note:** Public buckets allow downloads without authentication.
Example response: Raw object content (not JSON wrapped)
### List Objects
**GET** `/api/storage/buckets/:bucketName/objects`
Query parameters:
- `prefix` - Filter by key prefix
- `limit` - Max results (default: 100)
- `offset` - Pagination offset
Returns:
```json
{
"bucketName": "test-images",
"prefix": null,
"objects": [
{
"bucket": "test-images",
"key": "test.txt",
"size": 30,
"mimeType": "text/plain",
"uploadedAt": "2025-07-18T04:32:13.801Z",
"url": "http://localhost:7130/api/storage/buckets/test-images/objects/test.txt"
}
],
"pagination": {
"limit": 100,
"offset": 0,
"total": 1
},
"nextActions": "You can use PUT /api/storage/buckets/:bucketName/objects/:objectKey to upload with a specific key, or POST /api/storage/buckets/:bucketName/objects to upload with auto-generated key, and GET /api/storage/buckets/:bucketName/objects/:objectKey to download an object."
}
```
Pagination headers:
- `X-Total-Count`: Total number of objects
- `X-Page`: Current page number
- `X-Page-Size`: Number of items per page
Example curl:
```bash
# Windows PowerShell: use curl.exe
curl -X GET "http://localhost:7130/api/storage/buckets/avatars/objects?limit=10&prefix=users/" \
-H "Authorization: Bearer <token>"
```
### Delete Object
**DELETE** `/api/storage/buckets/:bucketName/objects/:objectKey`
Returns:
```json
{
"message": "Object deleted successfully"
}
```
Example curl:
```bash
# Windows PowerShell: use curl.exe
curl -X DELETE http://localhost:7130/api/storage/buckets/avatars/objects/user123.jpg \
-H "Authorization: Bearer <token>"
```
### Update Bucket
**PATCH** `/api/storage/buckets/:bucketName`
Request body:
```json
{ "isPublic": true }
```
Returns:
```json
{
"message": "Bucket visibility updated",
"bucket": "test-images",
"isPublic": false,
"nextActions": "Bucket is now PRIVATE - authentication is required to access objects."
}
```
Example curl:
```bash
# Mac/Linux
curl -X PATCH http://localhost:7130/api/storage/buckets/avatars \
-H 'Authorization: Bearer <token>' \
-H 'Content-Type: application/json' \
-d '{"isPublic": true}'
# Windows PowerShell (use curl.exe) - different quotes needed for nested JSON
curl.exe -X PATCH http://localhost:7130/api/storage/buckets/avatars \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d '{\"isPublic\": true}'
```
## Database Integration
### Option 1: Upload with Specific Key (PUT)
```javascript
// Step 1: Upload object with known key
const formData = new FormData();
formData.append('file', file);
const upload = await fetch('/api/storage/buckets/images/objects/avatar.jpg', {
method: 'PUT',
headers: { 'Authorization': `Bearer ${token}` },
body: formData
});
// Step 2: Store metadata
const records = [{
userId: 'user123',
image: await upload.json() // Store object metadata
}];
await fetch('/api/database/profiles', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(records)
});
```
### Option 2: Upload with Auto-Generated Key (POST)
```javascript
// Step 1: Upload object with auto-generated key
const formData = new FormData();
formData.append('file', file);
const upload = await fetch('/api/storage/buckets/images/objects', {
method: 'POST',
headers: { 'Authorization': `Bearer ${token}` },
body: formData
});
const fileData = await upload.json();
// Step 2: Store metadata with generated key
const records = [{
userId: 'user123',
image: fileData // Contains auto-generated key
}];
await fetch('/api/database/profiles', {
method: 'POST',
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(records)
});
```
## Error Response Format
All error responses follow this format:
```json
{
"error": "ERROR_CODE",
"message": "Human-readable error message",
"statusCode": 400,
"nextActions": "Suggested action to resolve the error"
}
```
Example error:
```json
{
"error": "BUCKET_NOT_FOUND",
"message": "Bucket 'nonexistent' does not exist",
"statusCode": 404,
"nextActions": "Create the bucket first"
}
```
## Important Rules
1. **Object Operations**
- Upload uses multipart/form-data
- Database stores metadata only
- Use `json` column type for object metadata
2. **Bucket Names**
- Must be valid identifiers
- Cannot start with underscore
3. **Remember**
- Bucket management uses MCP tools
- Object operations use REST API
- All operations need API key
.archive/docs/deprecated/insforge-storage-sdk.md
+159
View File
@@ -0,0 +1,159 @@
# InsForge Storage SDK
## Setup
```javascript
import { createClient } from '@insforge/sdk';
const client = createClient({ baseUrl: 'http://localhost:7130' });
```
## Bucket Operations
### from(bucketName)
```javascript
const bucket = client.storage.from('avatars') // Returns StorageBucket instance
```
## File Operations
### upload
```javascript
// Upload with specific key
const { data, error } = await client.storage
.from('avatars')
.upload('user-123.jpg', file);
// data: StorageFileSchema { bucket, key, size, mimeType, uploadedAt, url }
```
### uploadAuto
```javascript
// Upload with auto-generated key
const { data, error } = await client.storage
.from('avatars')
.uploadAuto(file);
// Generated key format: filename-timestamp-random.ext
```
### download
```javascript
const { data: blob, error } = await client.storage
.from('avatars')
.download('user-123.jpg');
// data: Blob (can convert to URL with URL.createObjectURL(blob))
```
### remove
```javascript
const { data, error } = await client.storage
.from('avatars')
.remove('user-123.jpg');
```
### list
```javascript
const { data, error } = await client.storage
.from('avatars')
.list({
prefix: 'users/', // Filter by prefix
search: 'profile', // Search in filenames
limit: 10, // Max results (default: 100)
offset: 0 // Skip results
});
// data: ListObjectsResponseSchema { bucketName, objects[], pagination }
```
### getPublicUrl
```javascript
// Get public URL (no API call)
const url = client.storage
.from('avatars')
.getPublicUrl('user-123.jpg');
// Returns: http://localhost:7130/api/storage/buckets/avatars/objects/user-123.jpg
```
## File Upload Examples
### Upload File from Input
```javascript
// HTML: <input type="file" id="fileInput">
const fileInput = document.getElementById('fileInput');
const file = fileInput.files[0];
const { data, error } = await client.storage
.from('uploads')
.upload(`photos/${file.name}`, file);
```
### Upload Blob
```javascript
const blob = new Blob(['Hello World'], { type: 'text/plain' });
const { data, error } = await client.storage
.from('documents')
.upload('hello.txt', blob);
```
### Upload with Custom Name
```javascript
// Simple file upload - SDK handles FormData creation
const file = fileInput.files[0];
const { data, error } = await client.storage
.from('uploads')
.upload('custom-name.jpg', file);
```
## Download & Display
### Download and Display Image
```javascript
const { data: blob, error } = await client.storage
.from('avatars')
.download('user-123.jpg');
if (blob) {
const url = URL.createObjectURL(blob);
document.getElementById('avatar').src = url;
// Clean up
URL.revokeObjectURL(url);
}
```
### Direct URL for Public Buckets
```javascript
// For public buckets, use direct URL
const url = client.storage
.from('public-avatars')
.getPublicUrl('user-123.jpg');
document.getElementById('avatar').src = url;
```
## Error Handling
```javascript
const { data, error } = await client.storage
.from('avatars')
.upload('user-123.jpg', file);
if (error) {
if (error.statusCode === 409) {
console.error('File already exists');
} else if (error.statusCode === 404) {
console.error('Bucket not found');
} else {
console.error(error.message);
}
}
```
## Notes
- Bucket creation/deletion handled via MCP tools, not SDK
- Public buckets allow unauthenticated downloads
- Private buckets require authentication for all operations
- File keys can include paths (e.g., 'folder/subfolder/file.jpg')
- Use `uploadAuto()` to prevent filename conflicts
.claude-plugin/marketplace.json
+24
View File
@@ -0,0 +1,24 @@
{
"name": "insforge-official-marketplace",
"owner": {
"name": "InsForge Team",
"email": "tony.chang@insforge.dev"
},
"plugins": [
{
"name": "insforge",
"source": {
"source": "github",
"repo": "InsForge/insforge-skills"
},
"description": "The backend platform for AI-native developers. A Postgres-based backend with auth, storage, compute, hosting, and AI gateway — built for coding agents. This plugin gives agents the full InsForge surface: database CRUD with RLS, authentication, storage, edge functions, AI, realtime, Stripe payments, and deployments. Includes CLI infrastructure management, debugging diagnostics, and integration guides for third-party auth providers (Auth0, Clerk, Kinde, Stytch, WorkOS).",
"version": "1.2.0",
"author": {
"name": "InsForge",
"url": "https://insforge.dev"
},
"category": "database",
"tags": ["insforge", "ai-native", "coding-agents", "ai-agents", "backend-as-a-service", "baas", "postgres", "auth", "storage", "compute", "hosting", "ai-gateway", "edge-functions", "stripe", "rls"]
}
]
}
.claude/skills/README.md
+49
View File
@@ -0,0 +1,49 @@
# `.claude/skills/` — contributor skill library
Claude Code (and compatible agents) auto-discover skills here. Each subdirectory
is one skill; a skill's entry point is `SKILL.md`, which starts with YAML
frontmatter containing at minimum `name` and `description`.
## Current skills
| Skill | Entry point | Purpose |
|---|---|---|
| `insforge-dev` | `insforge-dev/SKILL.md` | Maintainers working in this monorepo (backend, dashboard, UI, shared schemas, docs). **Mirrored copy — the canonical source is [`.agents/skills/insforge-dev/`](../../.agents/skills/insforge-dev/); do not edit here.** |
| `doc-author` | `doc-author/SKILL.md` | Writing and maintaining `docs/*.mdx` pages. **Vendored from [mintlify/docs](https://github.com/mintlify/docs) — see upstream SHA in the attribution block.** InsForge-specific conventions live next to it in [`doc-author/INSFORGE.md`](doc-author/INSFORGE.md). |
## Adding a new skill
1. Create `<skill-name>/SKILL.md` with `name` + `description` frontmatter.
2. Add the directory to the `.gitignore` allow-list at the repo root
(the root-level rules hide `.claude/*` by default).
3. Update this README with a one-line entry.
## Editing the `insforge-dev` skill
`insforge-dev` is mirrored across three agent directories (`.claude/`, `.codex/`,
`.agents/`) because each agent only discovers skills from its own directory. To
avoid drift, **`.agents/skills/insforge-dev/` is the single source of truth** and
the other two are generated copies.
1. Edit only `.agents/skills/insforge-dev/**`.
2. Run `scripts/sync-skills.sh` to regenerate the `.claude/` and `.codex/` copies.
3. Commit all three trees together.
CI (`scripts/sync-skills.sh --check` in `.github/workflows/lint-and-format.yml`)
fails if the copies drift from the canonical source. (Symlinks would avoid the
duplication but break on Windows checkouts and trip Prettier, so real copies it
is — until agents agree on a shared skill location.)
## Updating the vendored `doc-author` skill
`doc-author/SKILL.md` is a verbatim copy of Mintlify's upstream. To refresh:
```bash
scripts/update-mintlify-skill.sh
```
The script re-downloads the upstream file, updates the commit SHA in the
attribution header, and fails loudly if Mintlify's license has changed from MIT
— in which case the vendoring posture needs review before committing. Do not
hand-edit `doc-author/SKILL.md`; put InsForge-specific overrides in
`doc-author/INSFORGE.md` instead.
.claude/skills/doc-author/INSFORGE.md
+58
View File
@@ -0,0 +1,58 @@
# InsForge overlay — doc-author conventions
Local overlay for the vendored Mintlify [`doc-author` skill](./SKILL.md).
Upstream prose is authoritative; the items below are InsForge-specific and
override upstream advice only where they conflict.
## Frontmatter: `title` + `description` only
InsForge `docs/*.mdx` uses **only** `title` and `description` in YAML
frontmatter. Do not add `icon`, `sidebarTitle`, or other Mintlify-supported
keys unless a neighbouring page already does.
- `docs/quickstart.mdx:1-4` — canonical example
- `docs/sdks/typescript/auth.mdx:1-4` — SDK-reference style
## No `<ParamField>` — bullet lists for parameters
The repo has zero `<ParamField>` usage. Document parameters as plain markdown
bullet lists under a `### Parameters` heading.
- `docs/sdks/typescript/auth.mdx:17-22` — canonical pattern
## SDK install = import the snippet, never inline
Every page that shows an SDK install imports the shared snippet:
```mdx
import Installation from '/snippets/sdk-installation.mdx';
<Installation />
```
- Snippet body: `docs/snippets/sdk-installation.mdx`
- Usage: `docs/sdks/typescript/auth.mdx:7-10`,
`docs/core-concepts/storage/sdk.mdx:6-8`,
`docs/examples/framework-guides/react.mdx:6`
## Voice: second-person imperative
Address the reader as "you"; use imperative verbs. See `docs/quickstart.mdx`
for the canonical voice.
## Sound human, not AI-generated
InsForge docs are public-facing; they should not read as machine-written. Strip
the common AI tells (from the humanizer skill, based on Wikipedia's "Signs of AI
writing"):
- **Em dashes** for asides or emphasis. Use a comma, period, colon, or parentheses instead.
- **Rule of three.** Don't auto-triple ("fast, reliable, and scalable"); name the one that matters.
- **Negative parallelism** ("It's not just X, it's Y"). State the positive claim directly.
- **Inflated significance** ("plays a vital role", "underscores the importance of").
- **Vague attribution** ("studies show", "widely regarded"). Name the source or drop the claim.
- **AI vocabulary**: delve, leverage, utilize, underscore, foster, realm, landscape, tapestry, seamless, robust, pivotal. Use the plain word.
Read it back. If it sounds like a press release or a term paper, flatten it.
When you're not sure the draft is clean, run the full `humanizer` skill for a deeper pass (or see [blader/humanizer](https://github.com/blader/humanizer)). The rules above are the quick version.
.claude/skills/doc-author/SKILL.md
+460
View File
@@ -0,0 +1,460 @@
---
name: doc-author
description: Write, edit, and maintain documentation. Use for collaborative drafting, autonomous writing, or improving existing docs. Defaults to collaborative mode where the human makes final decisions. Built by Mintlify.
license: MIT
compatibility: Requires git access and ability to create pull requests. Works with any markdown or MDX documentation. Optimized for Mintlify-powered documentation sites.
metadata:
author: Mintlify
url: https://mintlify.com
version: "0.2"
---
<!--
Vendored from https://github.com/mintlify/docs (commit 877f90193ea1)
Upstream path: .claude/skills/doc-author/SKILL.md
Upstream license: MIT (see LICENSE block in upstream repo)
Vendored: 2026-04-18 by scripts/update-mintlify-skill.sh
This file is a VERBATIM copy of Mintlify's doc-author skill body.
Do not edit the prose below — local conventions go in ./INSFORGE.md
Update with: scripts/update-mintlify-skill.sh
-->
# Write and maintain documentation
This skill guides documentation work—from collaborative drafting with a human to autonomous writing with PR-based review.
## Operating modes
### Collaborative (default)
You're a collaborator. The human drives decisions, you assist. Use this mode unless you have a clear signal to work autonomously.
In collaborative mode:
- Draft content for the human to refine
- Suggest improvements with clear reasoning
- Ask clarifying questions before assuming
- Offer alternatives when there are trade-offs
- Flag concerns without blocking progress
### Autonomous
You write independently, open PRs, and flag uncertainties for human review. Use this mode only when:
- The task is explicitly delegated (e.g., a Linear issue assigned to you)
- The human tells you to "just do it" or "go ahead and write this"
- You're working from a clear, specific brief with no ambiguity
In autonomous mode:
- Write complete documentation and open a PR
- Add TODO comments for anything you can't verify
- Note uncertainties in the PR description
- Never commit directly—always open a PR for review
When in doubt about which mode to use, default to collaborative.
## Core principles
1. **Only document what you can verify.** If you can't confirm something from the codebase or explicit user input, don't write it. Leave a TODO instead.
2. **Write just enough.** Help users succeed and get back to their work. More docs isn't better docs.
3. **Match existing patterns.** Read surrounding content before writing. Consistency beats personal preference.
4. **Flag uncertainty.** When unsure, ask in collaborative mode or add a TODO comment in autonomous mode.
5. **Ask before assuming.** If something is unclear, ask. Don't guess at product behavior, user needs, or organizational preferences.
6. **Explain your reasoning.** When you suggest changes, say why. This helps people learn and make better decisions.
## Before you write
### Verify you have enough context
Before writing, confirm you can answer:
- What is this feature or concept?
- Who needs this documentation?
- What should they be able to do after reading?
If you can't answer these from the codebase or user input:
- **Collaborative mode:** Ask the human
- **Autonomous mode:** Stop and escalate
### Check for existing content
Search the docs for related content before creating new pages. You may need to:
- Update an existing page instead of creating a new one
- Add a section to an existing page
- Link to existing content rather than duplicating
### Read surrounding content
Before writing, read 2-3 similar pages to understand:
- Voice and tone patterns
- Structure and formatting conventions
- Level of detail provided
- Component usage patterns
## Working with humans
These practices apply in both modes—collaborative work is more interactive, but even autonomous work benefits from clear communication.
### When to ask questions
Ask before writing when:
- You don't understand the feature being documented
- The audience isn't clear
- You're unsure what level of detail is appropriate
- There are multiple valid approaches
Good questions:
- "Who's the primary audience for this page—developers integrating the API or admins configuring the product?"
- "Should this be a separate page or a new section on the existing [page name]?"
- "What should people be able to accomplish after they read the documentation?"
- "The codebase shows two ways to do this. Which should we document, or both?"
### When to offer alternatives
Present options when:
- There are different valid structures
- Tone could go multiple directions
- Detail level is a judgment call
Example:
> "I can write this as either:
> A. A quick reference with just the essential steps
> B. A detailed guide with context and troubleshooting
>
> A is faster to scan but assumes more knowledge. B helps beginners but takes longer to read. Which fits your users better?"
### When to flag concerns
Speak up when you notice:
- Content that might be inaccurate
- Patterns that differ from the rest of the docs
- Missing information that users would need
- Overly complex explanations
Be direct but not blocking:
> "This explanation assumes the reader knows what webhooks are. Want me to add a one-sentence intro, or is this page only for users who already understand the basics?"
### Handling uncertainty
**When you don't know something:**
> "I can't tell from the codebase what the default value is. Do you know, or should we check with the team?"
**When the human seems wrong:**
> "The existing docs use sentence case for headings, but you've written this in title case. Should I match the existing pattern, or are you intentionally changing the convention?"
**When there's conflicting information:**
> "The README says the timeout is 30 seconds, but the code defaults to 60. Which is correct?"
## Writing standards
### Voice and structure
- Second-person voice ("you")
- Active voice, direct language
- Sentence case for headings ("Getting started", not "Getting Started")
- Lead with context when helpful—explain what before how
- Prerequisites at the start of procedural content
### What to avoid
**Never use:**
- Marketing language ("powerful", "seamless", "robust", "cutting-edge")
- Filler phrases ("it's important to note", "in order to")
- Excessive conjunctions ("moreover", "furthermore", "additionally")
- Editorializing ("obviously", "simply", "just", "easily")
- Emoji in documentation
**Watch for AI-typical patterns:**
- Overly formal or stilted phrasing
- Unnecessary repetition of concepts
- Generic introductions that don't add value
- Concluding summaries that repeat what was just said
### Code examples
- Keep examples simple and practical
- Use realistic but generic values (not "foo", "bar", "example")
- Test that code actually works before including it
- One clear example is better than multiple variations
## For Mintlify-powered docs
If you're working with a Mintlify-powered documentation site, follow these conventions:
### File format
MDX files with YAML frontmatter:
```mdx
---
title: "Clear, descriptive title"
description: "Concise summary for SEO and navigation."
keywords: ["relevant", "search", "terms"]
---
Content starts here.
```
Every page requires title, description, and keywords in frontmatter.
### File naming
- Use kebab-case: `getting-started.mdx`, `api-reference.mdx`
- Be descriptive but concise
- Match existing naming patterns in the directory
### Components
Use Mintlify components appropriately:
**Callouts** for important information:
```mdx
<Note>Helpful context</Note>
<Warning>Something potentially destructive</Warning>
<Tip>A useful suggestion or best practice</Tip>
<Info>Information related to the task at hand</Info>
```
**Steps** for sequential procedures:
```mdx
<Steps>
<Step title="First step">
Instructions for step one.
</Step>
<Step title="Second step">
Instructions for step two.
</Step>
</Steps>
```
**Code blocks** always need language tags:
```javascript
const example = "always specify language";
```
### Internal links
Use root-relative paths: `/content/components/accordions`, not `../components/accordions` or full URLs.
## Verification guardrails
### What you can document
- Behavior you can verify in the codebase
- Information explicitly provided by the user
- Patterns consistent with existing documentation
- Standard usage based on documented APIs
### What requires a TODO
- Implementation details you can't verify
- Edge cases you haven't tested
- Configuration options you're unsure about
- Behavior that might vary by environment
Format TODOs clearly:
```mdx
{/* TODO: Verify the default timeout value - couldn't find in codebase */}
```
### What requires escalation
Stop and escalate when you encounter:
**Content uncertainty:**
- You don't understand the feature well enough to document it accurately
- Existing docs contradict what you see in the codebase
- The feature seems incomplete or broken
**Scope concerns:**
- Changes affect multiple pages or navigation structure
- Content requires product or design input
- Documentation involves security-sensitive information
- Content relates to pricing, billing, or legal terms
- You need to deprecate or significantly change existing content
**Technical blockers:**
- You can't find the source code for what you're documenting
- The API or interface has changed significantly
- You need access to systems or environments you don't have
## Workflow
### 1. Understand the task
Read the issue or request carefully. Identify:
- What specifically needs to be documented
- What pages are affected
- What the user should accomplish after reading
### 2. Research
- Search existing docs for related content
- Read the relevant source code
- Check for patterns in similar documentation
### 3. Plan your changes
Before writing, outline:
- Which files you'll modify or create
- What sections you'll add
- What existing content needs updates
In collaborative mode, share this plan with the human before writing.
### 4. Write
- Start with the most important information
- Keep sections focused and scannable
- Use components appropriately
- Add TODOs for anything uncertain
### 5. Self-review
Before presenting work (collaborative) or creating a PR (autonomous), verify:
- [ ] All code blocks have language tags
- [ ] Frontmatter includes title, description, keywords (if using MDX)
- [ ] Internal links are correct
- [ ] No marketing language or filler phrases
- [ ] Content matches style of surrounding pages
- [ ] TODOs are clearly marked for uncertain content
- [ ] New pages are added to navigation (if applicable)
- [ ] Noted any areas of uncertainty
### 6. Submit
**Collaborative mode:**
Present drafts as starting points:
> "Here's a draft based on what I found in the codebase. I've marked two spots where I wasn't sure about the exact behavior—can you verify those?"
**Autonomous mode:**
Always open a pull request. Never commit directly. PR description should include:
- What changed and why
- Any TODOs or uncertainties that need human review
- Files affected
- How to test or verify the changes
## Common tasks
### Drafting new content
1. Ask clarifying questions if the scope isn't clear
2. Read existing related pages to match style
3. Write a draft, noting any assumptions
4. Highlight areas where you're uncertain
### Editing existing content
1. Read the full page for context
2. Identify specific issues (not just "make it better")
3. Explain what you'd change and why
4. In collaborative mode, offer to make changes or let the human decide
Be specific:
> "I'd suggest three changes:
> 1. Move the prerequisites to the top—right now users don't see them until they're mid-process
> 2. Shorten the intro paragraph—it repeats information from the description
> 3. Add a code example after step 3—currently it's abstract without showing the actual syntax"
### Reviewing documentation
- Check for accuracy against the codebase
- Look for missing information users would need
- Note inconsistencies with other docs
- Flag unclear or ambiguous sections
Structure feedback clearly:
> **Accuracy issues:**
> - Line 23: The parameter is `timeout`, not `timeoutMs`
>
> **Missing information:**
> - No mention of what happens on failure
>
> **Style suggestions:**
> - The intro could be shorter
> - Consider using Steps component for the procedure
### Helping with structure
1. Understand what the content covers
2. Identify the user's goal when reading
3. Suggest a structure with reasoning
4. Be open to alternatives
Example:
> "For a setup guide, I'd suggest:
> 1. One-sentence overview of what they're setting up
> 2. Prerequisites (what they need before starting)
> 3. Steps (the actual procedure)
> 4. Verification (how to confirm it worked)
> 5. Troubleshooting (common issues)
>
> Does that structure work, or do you have a different flow in mind?"
## Examples
### Good page introduction
```mdx
---
title: "Webhooks"
description: "Receive real-time notifications when events occur in your account."
keywords: ["webhooks", "events", "notifications"]
---
Webhooks let your application receive automatic notifications when specific events happen,
like when a user signs up or a payment succeeds. Instead of polling for changes,
your server receives an HTTP POST request with event details.
```
### Poor page introduction (avoid)
```mdx
---
title: "Webhooks"
description: "Learn about our powerful webhook system."
keywords: ["webhooks"]
---
Welcome to our comprehensive guide on webhooks! Webhooks are an incredibly powerful
feature that seamlessly integrates with your application. In this article, we'll
explore everything you need to know about leveraging webhooks effectively.
```
### Good procedural content
```mdx
## Create a webhook endpoint
Before registering a webhook, you need an endpoint to receive events.
<Steps>
<Step title="Create an endpoint">
Add a POST route to your server that accepts JSON payloads:
```javascript
app.post('/webhooks', (req, res) => {
const event = req.body;
// Process the event
res.status(200).send('OK');
});
```
</Step>
<Step title="Make it publicly accessible">
Your endpoint must be reachable from the internet. During development,
use a tool like ngrok to expose your local server.
</Step>
</Steps>
```
### Appropriate TODO usage
```mdx
## Rate limits
Webhook deliveries are rate-limited to prevent overwhelming your server.
{/* TODO: Verify exact rate limit - code suggests 100/min but couldn't confirm */}
If a delivery fails, we retry with exponential backoff up to 5 times over 24 hours.
```
.claude/skills/insforge-dev/SKILL.md
+73
View File
@@ -0,0 +1,73 @@
---
name: insforge-dev
description: Use this skill set when contributing to the InsForge monorepo itself. This is for InsForge maintainers and contributors editing the platform, the shared dashboard package, the self-hosting shell, the UI library, shared schemas, tests, or docs.
---
# InsForge Dev
Use this skill set for work inside the InsForge repository.
Then use the narrowest package skill that matches the task:
- `backend`
- `dashboard`
- `ui`
- `shared-schemas`
- `docs`
Use the cross-repo release workflow skill when preparing an OSS PR:
- `e2e-testing`
## Core Rules
1. Identify the package boundary before editing.
- `backend/`: API, auth, database, providers, realtime, schedules
- `packages/dashboard/`: publishable dashboard package that supports `self-hosting` and `cloud-hosting` modes
- `frontend/`: local React + Vite shell that mounts `packages/dashboard/` in `self-hosting` mode
- `packages/shared-schemas/`: cross-package contracts
- `packages/ui/`: reusable design-system primitives
- `docs/`: product and agent-facing documentation
2. Put code in the narrowest correct layer.
- Contract change: `packages/shared-schemas/` first, then consumers.
- Backend behavior: route -> service -> provider/infra.
- Shared dashboard behavior, routes, features, exports, and host contracts: `packages/dashboard/`.
- Self-hosting-only bootstrap, env wiring, and shell styling: `frontend/`.
- Reusable primitive: `packages/ui/` first.
3. Preserve repo conventions.
- Backend TS source uses ESM-style `.js` import specifiers.
- Backend success responses usually return raw JSON, not `{ data }`.
- Backend validation commonly uses shared Zod schemas plus `AppError`.
- Dashboard data access goes through `apiClient` and React Query.
- Dashboard frontend tests are split into Vitest unit tests, Vitest component tests, and Playwright UI smoke tests.
- Shared payloads belong in `@insforge/shared-schemas`.
- Never use the TypeScript `any` type. Prefer precise types, schema-derived types, `unknown`, or generics.
4. Do not confuse repo development with app development on InsForge.
- This repo contains the platform, the publishable dashboard package, and a local shell for self-hosting mode.
- Keep guidance focused on maintaining InsForge itself.
## Finish Rules
- Run the smallest validation that gives confidence for the change.
- Use repo-level checks like `npm run lint`, `npm run build`, `npm run typecheck`, and `npm test` when the change crosses package boundaries — each is wired through Turborepo (`turbo run <task>`) and covers every workspace, including `packages/dashboard/` and `packages/shared-schemas/`.
- For dashboard UI behavior changes, choose the lowest useful frontend test layer from the `dashboard` skill and run that command before reporting back.
- Use the package-specific validation steps in the child skill when the work is isolated to one package.
- When reporting back, state what changed, what you validated, and what you could not validate.
## Pre-PR Checklist (Mandatory Before Pushing to a PR)
Before opening a PR or pushing new commits to an existing PR branch, run **all** of the following from the repo root and do not proceed while any of them fail on files your change touches:
1. `npx turbo run typecheck` — must pass across all packages.
2. `npx turbo run lint` — must pass. If the failure is pre-existing in `main` and unrelated to your change, scope it:
- Run `npx eslint <your-changed-files>` and confirm your files are clean.
- Call out the pre-existing debt in the PR body so reviewers know it is not yours.
- Auto-fixable prettier/eslint errors in your own diff must be fixed (`npx eslint --fix <file>` or `npm run format`).
3. `npx turbo run test` (or the package-specific test command) — all tests must pass, including any new tests you added for the change.
4. `npx turbo run build` if routing, config, schemas, or cross-package exports changed.
5. Use `e2e-testing` to run the deterministic cross-repo E2E gate before opening, updating, or submitting the InsForge OSS PR.
Never push with failing checks on files you touched, even if CI would catch them later. CI failures slow reviewers down and the lint fix almost always takes less than a minute locally.
.claude/skills/insforge-dev/backend/SKILL.md
+74
View File
@@ -0,0 +1,74 @@
---
name: backend
description: Use this skill when contributing to InsForge's backend package. This is for maintainers editing backend routes, services, providers, auth, database logic (including RLS-enforced surfaces like storage and realtime), schedules, or backend tests in the InsForge monorepo.
---
# InsForge Dev Backend
Use this skill for `backend/` work in the InsForge repository.
## Scope
- `backend/src/api/**`
- `backend/src/services/**`
- `backend/src/providers/**`
- `backend/src/infra/**`
- `backend/tests/**`
## Working Rules
1. Keep the route -> service -> provider/infra split intact.
- Routes handle auth, parsing, validation, and delegation.
- Services own business logic and orchestration.
- Providers and infra wrap external systems or lower-level integrations.
- Service layer code should be the only layer that interacts with the core PostgreSQL database.
- Do not put direct database access in routes.
- Do not bypass services when reading from or writing to Postgres.
2. Follow backend conventions.
- Use ESM-style `.js` import specifiers in TypeScript source.
- InsForge's core database is PostgreSQL.
- InsForge currently runs as a single-instance server, so be careful about introducing logic that assumes distributed coordination, cross-instance locking, or background worker separation.
- Reuse shared schemas from `@insforge/shared-schemas` when contracts cross packages.
- Use `safeParse` plus `AppError` for invalid input.
- Return successful results through `successResponse`.
- Preserve existing auth middleware patterns such as `verifyAdmin`, `verifyUser`, and `verifyApiKey`.
- Never use the TypeScript `any` type. Prefer precise interfaces, schema-derived types, `unknown`, or constrained generics.
- For schema changes, write a new migration file instead of editing database structure manually.
- Put schema changes under `backend/src/infra/database/migrations/`.
3. Write idempotent migrations. Every SQL migration must be safe to re-run.
- Use `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`.
- Never use bare `ALTER TABLE ... RENAME TO` — it fails if the target name already exists. Wrap renames in a `DO` block that checks `information_schema.tables` for both source and target.
- Always `DROP TRIGGER IF EXISTS` before `CREATE TRIGGER`.
- Guard data migrations and `DROP COLUMN` behind `information_schema.columns` checks when the column may already be gone.
- Use `ON CONFLICT` or `WHERE NOT EXISTS` for seed `INSERT` statements.
4. Preserve existing behavior around mutation flows.
- Keep audit logging when surrounding routes already log state changes.
- Keep error handling flowing through shared middleware.
- Do not introduce a new response envelope unless the existing feature already uses one.
- For critical flows with multiple dependent database writes, use an explicit transactional process so the whole operation succeeds or fails together.
- Be especially careful with transactions around auth, secrets, billing-like usage updates, schema changes, and any flow that would leave the system inconsistent if partially applied.
5. Use Postgres Row Level Security, not app-side filters, for tables accessed via authenticated end-user routes (anything where `req.user` reaches the service layer). RLS-enforced services such as storage, realtime, and payments should use `withUserContext`. Tables accessed only by admin or service-internal paths (audit logs, billing aggregations) don't need RLS. Do not write `WHERE user_id = $1` filters in services; let RLS evaluate `auth.jwt() ->> 'sub'` against the row.
- Plumb identity through `withUserContext(pool, ctx, fn, settings?)` from `services/database/user-context.service.ts`. It opens a transaction, sets `SET LOCAL ROLE` plus the canonical `request.jwt.claims` JSON GUC via `set_config`, applies optional transaction-local settings such as `realtime.channel_name`, runs `fn`, commits on success or rolls back on error, and resets role in `finally` so policies see the calling user via `auth.jwt() ->> 'sub'`.
- Keep `UserContext` user-only and defined in `api/middlewares/auth.ts`: `{ id, role, email? }` (`id` is always present at the API level). API keys and admin bypass flags do not belong inside `UserContext`.
- Routes that issue out-of-band URLs (S3 presigned redirects, signed download links, anything the client redeems against a service that won't re-evaluate RLS) must do an explicit RLS-scoped existence check before handing the URL out — RLS does not fire when the client redeems the URL directly. See `StorageService.objectIsVisible` as the template.
- Migrations that enable RLS on an existing populated table must auto-install a sensible default policy set so the upgrade does not silently break existing rows. See migration 036's `IF EXISTS (SELECT 1 FROM <table>) THEN <create policies> END IF` pattern.
- When adding a new RLS-enforced table: enable RLS, `GRANT` table-level CRUD to `authenticated`, and write per-operation policies (SELECT, INSERT, UPDATE, DELETE). Public-bucket-style anonymous bypasses live at the route layer before calling the RLS helper, not in policies.
- Normal raw SQL and custom migrations execute as `project_admin`. It has service-key row visibility, but PostgreSQL grants and ownership still limit object access and DDL.
6. Always write unit tests for new code.
- Every new feature, migration, service, or bug fix should have accompanying unit tests.
- For migrations, write tests that validate SQL structure and idempotency guards (see `tests/unit/redirect-url-whitelist-migration.test.ts` for the pattern).
- For services, test business logic and error cases.
- For RLS-gated services, mock the pool/client and pin the SQL sequence (see `tests/unit/user-context.service.test.ts` and `tests/unit/storage-object-is-visible.test.ts`).
- Run the full test suite before submitting work: `cd backend && npm test`.
## Validation
- `cd backend && npm test`
- `cd backend && npm run build`
For contract changes, also validate `packages/shared-schemas/` and any affected dashboard consumers.
.claude/skills/insforge-dev/dashboard/SKILL.md
+107
View File
@@ -0,0 +1,107 @@
---
name: dashboard
description: Use this skill when contributing to InsForge's shared dashboard package. This is for maintainers editing `packages/dashboard`, which ships in `self-hosting` and `cloud-hosting` modes, and the local `frontend/` shell used for `self-hosting` in this repo.
---
# InsForge Dev Dashboard
Use this skill for dashboard work in the InsForge repository.
## Scope
- `packages/dashboard/src/**`
- `packages/dashboard/package.json`
- `packages/dashboard/README.md`
- `packages/dashboard/*.config.*`
- `frontend/src/**`
- `frontend/package.json`
## Working Rules
1. Respect the shared-package versus host-app boundary.
- This dashboard is built with React and TypeScript.
- `packages/dashboard/` is the source of truth for the dashboard product.
- The package must support both `self-hosting` and `cloud-hosting` modes.
- Keep self-hosting-only bootstrap, local env defaults, and shell styling in `frontend/`.
- Do not let `packages/dashboard/` depend on `frontend/`.
- If both modes need a capability, define it in the package API first.
2. Preserve dashboard data-flow conventions.
- Follow the flow `service -> hook -> UI`.
- Use `apiClient` for HTTP calls so auth refresh and error handling stay consistent.
- Put request logic in services, data fetching and mutation state in hooks, and rendering/orchestration in UI components and pages.
- Reuse existing contexts, host abstractions, and hooks before creating new global state.
3. Reuse the existing component layers.
- Use `@insforge/ui` for generic primitives.
- Use shared dashboard components when the pattern is already present.
- Keep reusable dashboard UI in `packages/dashboard/`.
- Only add UI to `frontend/` when it is specific to the local self-hosting shell.
- Keep package styles scoped to the dashboard container.
4. Keep the package surface aligned with shared contracts.
- Import cross-package types and Zod-derived shapes from `@insforge/shared-schemas`.
- When backend payloads change, update the related services, hooks, UI, and exported types together.
- Keep `packages/dashboard/src/index.ts` and `packages/dashboard/src/types` aligned with the public package API.
- Never use the TypeScript `any` type. Prefer precise prop, state, API, and hook result types.
## Frontend Testing
Use the lowest test layer that covers the risk. Add one focused regression test for bug fixes when practical.
| Change type | Test layer | Location | Command |
| --- | --- | --- | --- |
| Pure helper, parser, formatter, state reducer | Unit | `packages/dashboard/src/**/__tests__/*.test.ts` | `npm --workspace @insforge/dashboard run test:unit` |
| React component behavior, forms, dialogs, conditional rendering | Component | `packages/dashboard/src/**/__tests__/*.test.tsx` | `npm --workspace @insforge/dashboard run test:component` |
| Routing, auth redirects, host-mode integration, browser-only behavior | UI smoke | `packages/dashboard/tests/ui/*.spec.ts` | `npm --workspace @insforge/dashboard run test:ui` |
Conventions:
- `npm --workspace @insforge/dashboard run test` runs the full Vitest suite: unit plus component.
- GitHub Actions splits dashboard checks into unit, component, and UI jobs in `.github/workflows/frontend-tests.yml`. Update the workflow when adding or renaming test scripts.
- Unit tests should avoid React rendering and network mocking. Test data transformations directly.
- Component tests use Testing Library with `packages/dashboard/src/test/setup.ts`. Mock hooks, services, and host context at the package boundary; assert user-visible behavior and callback effects, not implementation details or CSS class strings.
- UI smoke tests use Playwright against the local `frontend/` shell. Mock backend API routes in `packages/dashboard/tests/ui/fixtures/`; every mocked route must either fulfill, fallback, or abort. Do not leave requests pending after a test branch.
- Prefer one clear test per user-observable behavior over broad snapshot tests. Avoid brittle assertions tied to copy that is not part of the behavior being protected.
## Local debug: viewing cloud-hosting-only UI in self-hosting
**Use when** previewing UI gated on `useIsCloudHostingMode()`, `isInsForgeCloudProject()`, or a PostHog feature flag (e.g. the CTest dashboard variant, `dashboard-v3-experiment === 'c_test'`, the CLI connect panel) while running the local `frontend/` self-hosting shell.
The lowest-friction approach is to **temporarily hardcode** the three gates below to `true`/the new branch, then restart the Vite dev server. These edits bypass real host/project detection and MUST be fully reverted before committing — landing them breaks both self-hosting and cloud-hosting users.
### Hardcodes
1. `packages/dashboard/src/lib/config/DashboardHostContext.tsx` — `useIsCloudHostingMode()` → `return true;` (was `useDashboardHost().mode === 'cloud-hosting'`).
2. `packages/dashboard/src/lib/utils/utils.ts` — `isInsForgeCloudProject()` → `return true;` (was the `.insforge.app` hostname check).
3. If the UI is also feature-flag-gated, hardcode the consumer. For CTest: `AppRoutes.tsx` → `const DashboardHomePage = CTestDashboardPage;` and, if relevant, the matching branch in `AppLayout.tsx` for `<ConnectDialogV2>`.
Mark every hardcode with a trailing `// LOCAL DEBUG: <original expression>` comment so revert is a mechanical search.
### Revert checklist — run all before committing
1. `git grep -n "LOCAL DEBUG" packages/dashboard/src/` returns zero matches.
2. Each gate is restored to its **original expression**, not just an equivalent value (the `mode === 'cloud-hosting'` comparison, the hostname check, the `getFeatureFlag(...)` call must all be back).
3. Any imports deleted during debug (commonly `DashboardPage`, `getFeatureFlag`, `ConnectDialog`) are restored.
4. `cd packages/dashboard && npm run lint && npm run typecheck` both pass.
5. `git diff` of the four files above shows only intended changes — no `return true;`, no missing imports.
### Rationalizations to reject
| Excuse | Reality |
|--------|---------|
| "I'll revert in a follow-up PR." | Follow-up = a window where prod is broken. Revert now. |
| "The original check was effectively the same." | If it were, you wouldn't have needed the hardcode. Restore the expression, not a value-equivalent. |
| "Lint passed, so the deleted import doesn't matter." | Lint passed because the import was deleted; on revert the original code needs it back. |
| "I'll ship the env-var override instead." | No env-var override is wired in the code. Don't invent one on the commit path — restore the original. |
## Validation
- `cd packages/dashboard && npm run test:unit` when changing pure helpers or reducers
- `cd packages/dashboard && npm run test:component` when changing React component behavior
- `cd packages/dashboard && npm run test:ui` when changing routing, auth, host-mode integration, or browser-only behavior
- `cd packages/dashboard && npm run typecheck`
- `cd packages/dashboard && npm run build`
- `cd frontend && npm run build` when the local self-hosting shell changes
For shared contract changes, also validate `packages/shared-schemas/` and the affected backend surface.
.claude/skills/insforge-dev/docs/SKILL.md
+47
View File
@@ -0,0 +1,47 @@
---
name: docs
description: Use this skill when contributing to InsForge's product documentation in this repository. This is for maintainers editing public docs in `docs/core-concepts`, agent docs in `.agents/docs`, SDK integration guides in `docs/sdks`, and OpenAPI specs in `openapi`.
---
# InsForge Dev Docs
Use this skill for `docs/core-concepts/`, `.agents/docs/`, `docs/sdks/`, and `openapi/` in the InsForge repository.
The documentation in this repo is primarily product documentation for InsForge users and agents integrating with InsForge. This skill is for InsForge engineers maintaining that public documentation surface.
## Scope
- `docs/core-concepts/**`
- `.agents/docs/**`
- `docs/sdks/**`
- `openapi/**`
## Working Rules
1. Put each document in the correct documentation surface.
- Human-friendly docs published on the public doc site belong in `docs/core-concepts/` and related public doc folders.
- For implementation-heavy public docs, prefer an `architecture.md` file inside the relevant `docs/core-concepts/<domain>/` folder.
- Agent-only instructions belong in `.agents/docs/`.
- SDK integration guides for each framework belong in `docs/sdks/`.
- OpenAPI contract changes belong in the matching files under `openapi/`.
2. Match the writing style to the audience.
- Public docs should be human-friendly and explain the implementation clearly.
- Keep public docs human-sounding: avoid AI-writing tells such as em dashes, rule-of-three lists, "not just X but Y" parallelism, inflated significance, vague attribution, and AI vocabulary (delve, leverage, underscore, seamless, robust). See the `doc-author` skill's `INSFORGE.md` overlay, "Sound human, not AI-generated".
- `architecture.md` pages in `docs/core-concepts/` should explain how the feature works in detail.
- Agent docs in `.agents/docs/` should be instruction-first and execution-oriented.
- Agent docs should avoid explanatory filler and focus on the exact steps an agent should follow to complete the work.
3. Prevent documentation drift on every implementation change.
- Before changing implementation, check the current user-facing docs for that feature.
- After changing implementation, update the relevant Markdown docs and the relevant OpenAPI YAML files in the same pass.
- Do not treat OpenAPI and Markdown as separate optional follow-ups when the feature contract or behavior changed.
- If a change affects agent workflows, update the corresponding file in `.agents/docs/`.
- If a change affects public product understanding, update the corresponding file in `docs/core-concepts/`, including `architecture.md` when implementation details changed.
- If a change affects SDK integration guidance, update the corresponding framework guide in `docs/sdks/`.
## Validation
- Re-read every documented command, path, route, and payload for correctness.
- Cross-check OpenAPI YAML and Markdown docs against the implemented behavior before finishing.
- Mention anything you could not verify directly.
.claude/skills/insforge-dev/e2e-testing/SKILL.md
+142
View File
@@ -0,0 +1,142 @@
---
name: e2e-testing
description: Use this skill when an InsForge maintainer has finished an OSS repo change and is ready to open, update, or submit the InsForge PR. Runs the release-quality deterministic E2E gate by building a package.json-derived InsForge test image tag, deciding whether sibling agent-e2e fixture coverage must change, dispatching the Deterministic Fixture E2E workflow, waiting for results, and triaging failures before PR submission.
---
# InsForge E2E Testing Gate
Use this skill after local implementation and normal InsForge pre-PR checks pass, and before opening, updating, or submitting the InsForge OSS PR.
This is an additional release-quality gate. It does not replace local `typecheck`, `lint`, `test`, or `build` validation from the parent `insforge-dev` skill.
## Repositories
- InsForge OSS repo: current workspace.
- E2E repo: remote GitHub repository `InsForge/agent-e2e`.
Use the remote `InsForge/agent-e2e` repository for workflow dispatch and read-only workflow checks. Do not rely on a developer-specific local checkout path. Create or use a local checkout only when the deterministic fixture tests must be edited.
## Build The Test Tag
1. Read the root InsForge `package.json` version.
2. Increment only the patch number.
3. Build a tag in this form:
```text
v<major>.<minor>.<patch+1>-<feature-or-issue-slug>
```
Example: root version `2.2.3` and feature `storage returning rls` becomes `v2.2.4-storage-returning-rls`.
Use the `package.json` version as the only source of truth for the base version. Ignore higher existing test tags when calculating the base version.
Slug rules:
- Prefer the issue key, PR topic, or branch topic.
- Lowercase all letters.
- Replace runs of non-alphanumeric characters with one hyphen.
- Trim leading and trailing hyphens.
- Keep it short enough to scan in GitHub Actions and image tags.
## Build The InsForge Image
Dispatch the InsForge `Build and Push Docker Image` workflow with the test tag:
```bash
gh workflow run "Build and Push Docker Image" --repo InsForge/InsForge --ref <insforge-feature-branch> -f test_tag=<test-tag>
```
Then wait for the matching run to complete:
```bash
gh run list --repo InsForge/InsForge --workflow "Build and Push Docker Image" --limit 10
gh run watch --repo InsForge/InsForge <run-id>
```
Do not start the cross-repo E2E workflow until the image build succeeds.
## Decide Whether `agent-e2e` Must Change
Inspect the InsForge diff and compare it with deterministic fixture coverage in `InsForge/agent-e2e`.
For read-only checks, prefer remote GitHub access such as `gh api`, `gh repo view`, or remote file reads. Use a local checkout only when editing fixture files or when remote inspection is not enough to understand coverage.
Update `agent-e2e` when the InsForge change adds, removes, or changes behavior that the deterministic fixture should assert, including:
- API contract or validation behavior.
- Auth, permissions, RLS, storage, realtime, functions, schedules, AI, SDK, or CLI-visible behavior.
- Any regression that local tests cover but the release gate should also protect across the deployed runtime.
Do not update `agent-e2e` for internal refactors, docs-only changes, local test-only changes, or behavior already covered by the deterministic fixture with no assertion change needed.
Ignore `Support Desk Agent E2E (Exploratory)`. It is not part of this gate.
## If No E2E Test Update Is Needed
Run the deterministic fixture workflow from `agent-e2e` main:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref main -f insforge_tag=<test-tag>
```
Find and watch the run:
```bash
gh run list --repo InsForge/agent-e2e --workflow "Deterministic Fixture E2E" --limit 10
gh run watch --repo InsForge/agent-e2e <run-id>
```
## If E2E Tests Need An Update
Work in a local checkout of the remote `InsForge/agent-e2e` repo only for the fixture update.
1. Use an existing clean checkout or clone `https://github.com/InsForge/agent-e2e.git` into an isolated workspace.
2. Fetch `origin` and start from `origin/main`.
3. Create a branch named `codex/<short-topic>`.
4. Update only the deterministic fixture validators, fixtures, app assertions, and docs needed for the InsForge behavior change.
5. Run the smallest local validation that gives confidence:
```bash
npm run typecheck
npm run lint
npm run fixture:e2e:dry
```
Run broader validation when the changed fixture area supports it.
6. Open an `agent-e2e` PR for the fixture update.
7. Dispatch the deterministic fixture workflow from the `agent-e2e` branch that contains the fixture update:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref <agent-e2e-branch> -f insforge_tag=<test-tag>
```
8. Watch the run before proceeding with the InsForge PR.
## Interpret Results
If the deterministic fixture workflow passes:
- Link the run in the InsForge PR body or final PR notes.
- If an `agent-e2e` PR was required, link that PR too.
- Proceed with opening, updating, or submitting the InsForge OSS PR.
If the deterministic fixture workflow fails:
1. Inspect the failed job logs and uploaded artifact.
2. Identify whether the failure is caused by the InsForge implementation, the new or existing deterministic fixture, a transient infrastructure problem, or an unrelated existing failure.
3. Fix the correct branch:
- InsForge implementation bug: update the InsForge branch, rebuild the test image with the same tag or a new short retry tag, then rerun E2E.
- Fixture bug or missing assertion update: update the `agent-e2e` branch, rerun local validation, then rerun E2E from that branch.
- Transient infrastructure issue: rerun once after noting the evidence.
4. Do not submit the InsForge PR until the deterministic result is clear or the user explicitly accepts the risk.
## Report Back
When finished, report:
- Test tag used.
- InsForge image workflow run result.
- Whether `agent-e2e` changed.
- Deterministic Fixture E2E run result.
- Links to the InsForge PR, `agent-e2e` PR if any, and workflow runs.
.claude/skills/insforge-dev/shared-schemas/SKILL.md
+43
View File
@@ -0,0 +1,43 @@
---
name: shared-schemas
description: Use this skill when contributing to InsForge's shared schema package. This is for maintainers editing published Zod contracts, exported types, and shared API payload definitions consumed by InsForge packages in this repo and other InsForge tooling.
---
# InsForge Dev Shared Schemas
Use this skill for `packages/shared-schemas/` work in the InsForge repository.
## Scope
- `packages/shared-schemas/src/**`
- any code in this repo that consumes the changed contracts
- downstream InsForge tooling that depends on the published `@insforge/shared-schemas` package, including consumers not present in this repository
## Working Rules
1. Treat `packages/shared-schemas/` as the source of truth for cross-package payloads.
- If a request, response, or domain shape is shared across InsForge surfaces, define it here.
- Do not duplicate the same contract in package-local files.
- Remember that this package is not only for the repo's backend and dashboard packages. It is also consumed by other InsForge tooling, including MCP and SDK code that may live outside this repository.
2. Keep schemas organized by domain.
- Follow the existing `*.schema.ts` and `*-api.schema.ts` split when it fits the current package pattern.
- Keep `packages/shared-schemas/src/index.ts` aligned with the intended public API.
- Treat exported names and schema shapes as a public contract surface, not just an internal refactor target.
3. Use schema changes as a synchronization trigger.
- Update backend validation and response usage.
- Update shared dashboard services, hooks, and UI assumptions in `packages/dashboard/`.
- Check for import sites across `packages/*`, `frontend/`, and `backend/` before finishing.
- Call out likely downstream impact on MCP, SDK, or other external InsForge consumers when a change alters exported names, schema semantics, or payload shape.
- Be conservative with breaking changes. If a breaking contract change is necessary, make it explicit in the handoff.
- Never use the TypeScript `any` type. Shared contracts should stay explicit and trustworthy across package boundaries.
## Validation
- `cd packages/shared-schemas && npm run build`
- `cd backend && npx tsc --noEmit`
- `cd packages/dashboard && npm run typecheck`
Run package-specific tests as needed where behavior changed.
If external consumers such as MCP or SDK cannot be validated from this repo, say that clearly instead of implying they were covered.
.claude/skills/insforge-dev/ui/SKILL.md
+42
View File
@@ -0,0 +1,42 @@
---
name: ui
description: Use this skill when contributing to InsForge's reusable UI package. This is for maintainers editing design-system primitives, exports, styles, and package-level component behavior in the InsForge monorepo.
---
# InsForge Dev UI
Use this skill for `packages/ui/` work in the InsForge repository.
## Scope
- `packages/ui/src/components/**`
- `packages/ui/src/lib/**`
- `packages/ui/src/index.ts`
- `packages/ui/src/styles.css`
## Working Rules
1. Put only reusable primitives here.
- If the component is generic across dashboard features or other InsForge apps, it belongs in `packages/ui/`.
- If it is tightly coupled to one dashboard workflow but should ship to both OSS and cloud hosts, keep it in `packages/dashboard/`.
- If it is only for the self-hosting host app, keep it in `frontend/`.
2. Preserve the package's implementation style.
- Use `class-variance-authority` for variants when appropriate.
- Use the shared `cn()` helper for class merging.
- Follow the existing Radix-wrapper and typed-export patterns.
3. Keep the public surface in sync.
- Export new public components from `packages/ui/src/index.ts`.
- Avoid adding internal-only abstractions to the package surface unless they are meant to be consumed.
- Never use the TypeScript `any` type. Keep component props and exported helpers strictly typed.
4. Validate downstream impact.
- The shared dashboard package consumes this package directly, so UI changes can break `packages/dashboard/` even if `packages/ui/` itself builds cleanly.
## Validation
- `cd packages/ui && npm run build`
- `cd packages/ui && npm run typecheck`
Also validate `packages/dashboard/` when the changed component is used in the dashboard, and validate `frontend/` if the host app integration or CSS entrypoints changed.
.codex/skills/insforge-dev/SKILL.md
+73
View File
@@ -0,0 +1,73 @@
---
name: insforge-dev
description: Use this skill set when contributing to the InsForge monorepo itself. This is for InsForge maintainers and contributors editing the platform, the shared dashboard package, the self-hosting shell, the UI library, shared schemas, tests, or docs.
---
# InsForge Dev
Use this skill set for work inside the InsForge repository.
Then use the narrowest package skill that matches the task:
- `backend`
- `dashboard`
- `ui`
- `shared-schemas`
- `docs`
Use the cross-repo release workflow skill when preparing an OSS PR:
- `e2e-testing`
## Core Rules
1. Identify the package boundary before editing.
- `backend/`: API, auth, database, providers, realtime, schedules
- `packages/dashboard/`: publishable dashboard package that supports `self-hosting` and `cloud-hosting` modes
- `frontend/`: local React + Vite shell that mounts `packages/dashboard/` in `self-hosting` mode
- `packages/shared-schemas/`: cross-package contracts
- `packages/ui/`: reusable design-system primitives
- `docs/`: product and agent-facing documentation
2. Put code in the narrowest correct layer.
- Contract change: `packages/shared-schemas/` first, then consumers.
- Backend behavior: route -> service -> provider/infra.
- Shared dashboard behavior, routes, features, exports, and host contracts: `packages/dashboard/`.
- Self-hosting-only bootstrap, env wiring, and shell styling: `frontend/`.
- Reusable primitive: `packages/ui/` first.
3. Preserve repo conventions.
- Backend TS source uses ESM-style `.js` import specifiers.
- Backend success responses usually return raw JSON, not `{ data }`.
- Backend validation commonly uses shared Zod schemas plus `AppError`.
- Dashboard data access goes through `apiClient` and React Query.
- Dashboard frontend tests are split into Vitest unit tests, Vitest component tests, and Playwright UI smoke tests.
- Shared payloads belong in `@insforge/shared-schemas`.
- Never use the TypeScript `any` type. Prefer precise types, schema-derived types, `unknown`, or generics.
4. Do not confuse repo development with app development on InsForge.
- This repo contains the platform, the publishable dashboard package, and a local shell for self-hosting mode.
- Keep guidance focused on maintaining InsForge itself.
## Finish Rules
- Run the smallest validation that gives confidence for the change.
- Use repo-level checks like `npm run lint`, `npm run build`, `npm run typecheck`, and `npm test` when the change crosses package boundaries — each is wired through Turborepo (`turbo run <task>`) and covers every workspace, including `packages/dashboard/` and `packages/shared-schemas/`.
- For dashboard UI behavior changes, choose the lowest useful frontend test layer from the `dashboard` skill and run that command before reporting back.
- Use the package-specific validation steps in the child skill when the work is isolated to one package.
- When reporting back, state what changed, what you validated, and what you could not validate.
## Pre-PR Checklist (Mandatory Before Pushing to a PR)
Before opening a PR or pushing new commits to an existing PR branch, run **all** of the following from the repo root and do not proceed while any of them fail on files your change touches:
1. `npx turbo run typecheck` — must pass across all packages.
2. `npx turbo run lint` — must pass. If the failure is pre-existing in `main` and unrelated to your change, scope it:
- Run `npx eslint <your-changed-files>` and confirm your files are clean.
- Call out the pre-existing debt in the PR body so reviewers know it is not yours.
- Auto-fixable prettier/eslint errors in your own diff must be fixed (`npx eslint --fix <file>` or `npm run format`).
3. `npx turbo run test` (or the package-specific test command) — all tests must pass, including any new tests you added for the change.
4. `npx turbo run build` if routing, config, schemas, or cross-package exports changed.
5. Use `e2e-testing` to run the deterministic cross-repo E2E gate before opening, updating, or submitting the InsForge OSS PR.
Never push with failing checks on files you touched, even if CI would catch them later. CI failures slow reviewers down and the lint fix almost always takes less than a minute locally.
.codex/skills/insforge-dev/backend/SKILL.md
+74
View File
@@ -0,0 +1,74 @@
---
name: backend
description: Use this skill when contributing to InsForge's backend package. This is for maintainers editing backend routes, services, providers, auth, database logic (including RLS-enforced surfaces like storage and realtime), schedules, or backend tests in the InsForge monorepo.
---
# InsForge Dev Backend
Use this skill for `backend/` work in the InsForge repository.
## Scope
- `backend/src/api/**`
- `backend/src/services/**`
- `backend/src/providers/**`
- `backend/src/infra/**`
- `backend/tests/**`
## Working Rules
1. Keep the route -> service -> provider/infra split intact.
- Routes handle auth, parsing, validation, and delegation.
- Services own business logic and orchestration.
- Providers and infra wrap external systems or lower-level integrations.
- Service layer code should be the only layer that interacts with the core PostgreSQL database.
- Do not put direct database access in routes.
- Do not bypass services when reading from or writing to Postgres.
2. Follow backend conventions.
- Use ESM-style `.js` import specifiers in TypeScript source.
- InsForge's core database is PostgreSQL.
- InsForge currently runs as a single-instance server, so be careful about introducing logic that assumes distributed coordination, cross-instance locking, or background worker separation.
- Reuse shared schemas from `@insforge/shared-schemas` when contracts cross packages.
- Use `safeParse` plus `AppError` for invalid input.
- Return successful results through `successResponse`.
- Preserve existing auth middleware patterns such as `verifyAdmin`, `verifyUser`, and `verifyApiKey`.
- Never use the TypeScript `any` type. Prefer precise interfaces, schema-derived types, `unknown`, or constrained generics.
- For schema changes, write a new migration file instead of editing database structure manually.
- Put schema changes under `backend/src/infra/database/migrations/`.
3. Write idempotent migrations. Every SQL migration must be safe to re-run.
- Use `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS`, `ADD COLUMN IF NOT EXISTS`.
- Never use bare `ALTER TABLE ... RENAME TO` — it fails if the target name already exists. Wrap renames in a `DO` block that checks `information_schema.tables` for both source and target.
- Always `DROP TRIGGER IF EXISTS` before `CREATE TRIGGER`.
- Guard data migrations and `DROP COLUMN` behind `information_schema.columns` checks when the column may already be gone.
- Use `ON CONFLICT` or `WHERE NOT EXISTS` for seed `INSERT` statements.
4. Preserve existing behavior around mutation flows.
- Keep audit logging when surrounding routes already log state changes.
- Keep error handling flowing through shared middleware.
- Do not introduce a new response envelope unless the existing feature already uses one.
- For critical flows with multiple dependent database writes, use an explicit transactional process so the whole operation succeeds or fails together.
- Be especially careful with transactions around auth, secrets, billing-like usage updates, schema changes, and any flow that would leave the system inconsistent if partially applied.
5. Use Postgres Row Level Security, not app-side filters, for tables accessed via authenticated end-user routes (anything where `req.user` reaches the service layer). RLS-enforced services such as storage, realtime, and payments should use `withUserContext`. Tables accessed only by admin or service-internal paths (audit logs, billing aggregations) don't need RLS. Do not write `WHERE user_id = $1` filters in services; let RLS evaluate `auth.jwt() ->> 'sub'` against the row.
- Plumb identity through `withUserContext(pool, ctx, fn, settings?)` from `services/database/user-context.service.ts`. It opens a transaction, sets `SET LOCAL ROLE` plus the canonical `request.jwt.claims` JSON GUC via `set_config`, applies optional transaction-local settings such as `realtime.channel_name`, runs `fn`, commits on success or rolls back on error, and resets role in `finally` so policies see the calling user via `auth.jwt() ->> 'sub'`.
- Keep `UserContext` user-only and defined in `api/middlewares/auth.ts`: `{ id, role, email? }` (`id` is always present at the API level). API keys and admin bypass flags do not belong inside `UserContext`.
- Routes that issue out-of-band URLs (S3 presigned redirects, signed download links, anything the client redeems against a service that won't re-evaluate RLS) must do an explicit RLS-scoped existence check before handing the URL out — RLS does not fire when the client redeems the URL directly. See `StorageService.objectIsVisible` as the template.
- Migrations that enable RLS on an existing populated table must auto-install a sensible default policy set so the upgrade does not silently break existing rows. See migration 036's `IF EXISTS (SELECT 1 FROM <table>) THEN <create policies> END IF` pattern.
- When adding a new RLS-enforced table: enable RLS, `GRANT` table-level CRUD to `authenticated`, and write per-operation policies (SELECT, INSERT, UPDATE, DELETE). Public-bucket-style anonymous bypasses live at the route layer before calling the RLS helper, not in policies.
- Normal raw SQL and custom migrations execute as `project_admin`. It has service-key row visibility, but PostgreSQL grants and ownership still limit object access and DDL.
6. Always write unit tests for new code.
- Every new feature, migration, service, or bug fix should have accompanying unit tests.
- For migrations, write tests that validate SQL structure and idempotency guards (see `tests/unit/redirect-url-whitelist-migration.test.ts` for the pattern).
- For services, test business logic and error cases.
- For RLS-gated services, mock the pool/client and pin the SQL sequence (see `tests/unit/user-context.service.test.ts` and `tests/unit/storage-object-is-visible.test.ts`).
- Run the full test suite before submitting work: `cd backend && npm test`.
## Validation
- `cd backend && npm test`
- `cd backend && npm run build`
For contract changes, also validate `packages/shared-schemas/` and any affected dashboard consumers.
.codex/skills/insforge-dev/dashboard/SKILL.md
+107
View File
@@ -0,0 +1,107 @@
---
name: dashboard
description: Use this skill when contributing to InsForge's shared dashboard package. This is for maintainers editing `packages/dashboard`, which ships in `self-hosting` and `cloud-hosting` modes, and the local `frontend/` shell used for `self-hosting` in this repo.
---
# InsForge Dev Dashboard
Use this skill for dashboard work in the InsForge repository.
## Scope
- `packages/dashboard/src/**`
- `packages/dashboard/package.json`
- `packages/dashboard/README.md`
- `packages/dashboard/*.config.*`
- `frontend/src/**`
- `frontend/package.json`
## Working Rules
1. Respect the shared-package versus host-app boundary.
- This dashboard is built with React and TypeScript.
- `packages/dashboard/` is the source of truth for the dashboard product.
- The package must support both `self-hosting` and `cloud-hosting` modes.
- Keep self-hosting-only bootstrap, local env defaults, and shell styling in `frontend/`.
- Do not let `packages/dashboard/` depend on `frontend/`.
- If both modes need a capability, define it in the package API first.
2. Preserve dashboard data-flow conventions.
- Follow the flow `service -> hook -> UI`.
- Use `apiClient` for HTTP calls so auth refresh and error handling stay consistent.
- Put request logic in services, data fetching and mutation state in hooks, and rendering/orchestration in UI components and pages.
- Reuse existing contexts, host abstractions, and hooks before creating new global state.
3. Reuse the existing component layers.
- Use `@insforge/ui` for generic primitives.
- Use shared dashboard components when the pattern is already present.
- Keep reusable dashboard UI in `packages/dashboard/`.
- Only add UI to `frontend/` when it is specific to the local self-hosting shell.
- Keep package styles scoped to the dashboard container.
4. Keep the package surface aligned with shared contracts.
- Import cross-package types and Zod-derived shapes from `@insforge/shared-schemas`.
- When backend payloads change, update the related services, hooks, UI, and exported types together.
- Keep `packages/dashboard/src/index.ts` and `packages/dashboard/src/types` aligned with the public package API.
- Never use the TypeScript `any` type. Prefer precise prop, state, API, and hook result types.
## Frontend Testing
Use the lowest test layer that covers the risk. Add one focused regression test for bug fixes when practical.
| Change type | Test layer | Location | Command |
| --- | --- | --- | --- |
| Pure helper, parser, formatter, state reducer | Unit | `packages/dashboard/src/**/__tests__/*.test.ts` | `npm --workspace @insforge/dashboard run test:unit` |
| React component behavior, forms, dialogs, conditional rendering | Component | `packages/dashboard/src/**/__tests__/*.test.tsx` | `npm --workspace @insforge/dashboard run test:component` |
| Routing, auth redirects, host-mode integration, browser-only behavior | UI smoke | `packages/dashboard/tests/ui/*.spec.ts` | `npm --workspace @insforge/dashboard run test:ui` |
Conventions:
- `npm --workspace @insforge/dashboard run test` runs the full Vitest suite: unit plus component.
- GitHub Actions splits dashboard checks into unit, component, and UI jobs in `.github/workflows/frontend-tests.yml`. Update the workflow when adding or renaming test scripts.
- Unit tests should avoid React rendering and network mocking. Test data transformations directly.
- Component tests use Testing Library with `packages/dashboard/src/test/setup.ts`. Mock hooks, services, and host context at the package boundary; assert user-visible behavior and callback effects, not implementation details or CSS class strings.
- UI smoke tests use Playwright against the local `frontend/` shell. Mock backend API routes in `packages/dashboard/tests/ui/fixtures/`; every mocked route must either fulfill, fallback, or abort. Do not leave requests pending after a test branch.
- Prefer one clear test per user-observable behavior over broad snapshot tests. Avoid brittle assertions tied to copy that is not part of the behavior being protected.
## Local debug: viewing cloud-hosting-only UI in self-hosting
**Use when** previewing UI gated on `useIsCloudHostingMode()`, `isInsForgeCloudProject()`, or a PostHog feature flag (e.g. the CTest dashboard variant, `dashboard-v3-experiment === 'c_test'`, the CLI connect panel) while running the local `frontend/` self-hosting shell.
The lowest-friction approach is to **temporarily hardcode** the three gates below to `true`/the new branch, then restart the Vite dev server. These edits bypass real host/project detection and MUST be fully reverted before committing — landing them breaks both self-hosting and cloud-hosting users.
### Hardcodes
1. `packages/dashboard/src/lib/config/DashboardHostContext.tsx` — `useIsCloudHostingMode()` → `return true;` (was `useDashboardHost().mode === 'cloud-hosting'`).
2. `packages/dashboard/src/lib/utils/utils.ts` — `isInsForgeCloudProject()` → `return true;` (was the `.insforge.app` hostname check).
3. If the UI is also feature-flag-gated, hardcode the consumer. For CTest: `AppRoutes.tsx` → `const DashboardHomePage = CTestDashboardPage;` and, if relevant, the matching branch in `AppLayout.tsx` for `<ConnectDialogV2>`.
Mark every hardcode with a trailing `// LOCAL DEBUG: <original expression>` comment so revert is a mechanical search.
### Revert checklist — run all before committing
1. `git grep -n "LOCAL DEBUG" packages/dashboard/src/` returns zero matches.
2. Each gate is restored to its **original expression**, not just an equivalent value (the `mode === 'cloud-hosting'` comparison, the hostname check, the `getFeatureFlag(...)` call must all be back).
3. Any imports deleted during debug (commonly `DashboardPage`, `getFeatureFlag`, `ConnectDialog`) are restored.
4. `cd packages/dashboard && npm run lint && npm run typecheck` both pass.
5. `git diff` of the four files above shows only intended changes — no `return true;`, no missing imports.
### Rationalizations to reject
| Excuse | Reality |
|--------|---------|
| "I'll revert in a follow-up PR." | Follow-up = a window where prod is broken. Revert now. |
| "The original check was effectively the same." | If it were, you wouldn't have needed the hardcode. Restore the expression, not a value-equivalent. |
| "Lint passed, so the deleted import doesn't matter." | Lint passed because the import was deleted; on revert the original code needs it back. |
| "I'll ship the env-var override instead." | No env-var override is wired in the code. Don't invent one on the commit path — restore the original. |
## Validation
- `cd packages/dashboard && npm run test:unit` when changing pure helpers or reducers
- `cd packages/dashboard && npm run test:component` when changing React component behavior
- `cd packages/dashboard && npm run test:ui` when changing routing, auth, host-mode integration, or browser-only behavior
- `cd packages/dashboard && npm run typecheck`
- `cd packages/dashboard && npm run build`
- `cd frontend && npm run build` when the local self-hosting shell changes
For shared contract changes, also validate `packages/shared-schemas/` and the affected backend surface.
.codex/skills/insforge-dev/docs/SKILL.md
+47
View File
@@ -0,0 +1,47 @@
---
name: docs
description: Use this skill when contributing to InsForge's product documentation in this repository. This is for maintainers editing public docs in `docs/core-concepts`, agent docs in `.agents/docs`, SDK integration guides in `docs/sdks`, and OpenAPI specs in `openapi`.
---
# InsForge Dev Docs
Use this skill for `docs/core-concepts/`, `.agents/docs/`, `docs/sdks/`, and `openapi/` in the InsForge repository.
The documentation in this repo is primarily product documentation for InsForge users and agents integrating with InsForge. This skill is for InsForge engineers maintaining that public documentation surface.
## Scope
- `docs/core-concepts/**`
- `.agents/docs/**`
- `docs/sdks/**`
- `openapi/**`
## Working Rules
1. Put each document in the correct documentation surface.
- Human-friendly docs published on the public doc site belong in `docs/core-concepts/` and related public doc folders.
- For implementation-heavy public docs, prefer an `architecture.md` file inside the relevant `docs/core-concepts/<domain>/` folder.
- Agent-only instructions belong in `.agents/docs/`.
- SDK integration guides for each framework belong in `docs/sdks/`.
- OpenAPI contract changes belong in the matching files under `openapi/`.
2. Match the writing style to the audience.
- Public docs should be human-friendly and explain the implementation clearly.
- Keep public docs human-sounding: avoid AI-writing tells such as em dashes, rule-of-three lists, "not just X but Y" parallelism, inflated significance, vague attribution, and AI vocabulary (delve, leverage, underscore, seamless, robust). See the `doc-author` skill's `INSFORGE.md` overlay, "Sound human, not AI-generated".
- `architecture.md` pages in `docs/core-concepts/` should explain how the feature works in detail.
- Agent docs in `.agents/docs/` should be instruction-first and execution-oriented.
- Agent docs should avoid explanatory filler and focus on the exact steps an agent should follow to complete the work.
3. Prevent documentation drift on every implementation change.
- Before changing implementation, check the current user-facing docs for that feature.
- After changing implementation, update the relevant Markdown docs and the relevant OpenAPI YAML files in the same pass.
- Do not treat OpenAPI and Markdown as separate optional follow-ups when the feature contract or behavior changed.
- If a change affects agent workflows, update the corresponding file in `.agents/docs/`.
- If a change affects public product understanding, update the corresponding file in `docs/core-concepts/`, including `architecture.md` when implementation details changed.
- If a change affects SDK integration guidance, update the corresponding framework guide in `docs/sdks/`.
## Validation
- Re-read every documented command, path, route, and payload for correctness.
- Cross-check OpenAPI YAML and Markdown docs against the implemented behavior before finishing.
- Mention anything you could not verify directly.
.codex/skills/insforge-dev/e2e-testing/SKILL.md
+142
View File
@@ -0,0 +1,142 @@
---
name: e2e-testing
description: Use this skill when an InsForge maintainer has finished an OSS repo change and is ready to open, update, or submit the InsForge PR. Runs the release-quality deterministic E2E gate by building a package.json-derived InsForge test image tag, deciding whether sibling agent-e2e fixture coverage must change, dispatching the Deterministic Fixture E2E workflow, waiting for results, and triaging failures before PR submission.
---
# InsForge E2E Testing Gate
Use this skill after local implementation and normal InsForge pre-PR checks pass, and before opening, updating, or submitting the InsForge OSS PR.
This is an additional release-quality gate. It does not replace local `typecheck`, `lint`, `test`, or `build` validation from the parent `insforge-dev` skill.
## Repositories
- InsForge OSS repo: current workspace.
- E2E repo: remote GitHub repository `InsForge/agent-e2e`.
Use the remote `InsForge/agent-e2e` repository for workflow dispatch and read-only workflow checks. Do not rely on a developer-specific local checkout path. Create or use a local checkout only when the deterministic fixture tests must be edited.
## Build The Test Tag
1. Read the root InsForge `package.json` version.
2. Increment only the patch number.
3. Build a tag in this form:
```text
v<major>.<minor>.<patch+1>-<feature-or-issue-slug>
```
Example: root version `2.2.3` and feature `storage returning rls` becomes `v2.2.4-storage-returning-rls`.
Use the `package.json` version as the only source of truth for the base version. Ignore higher existing test tags when calculating the base version.
Slug rules:
- Prefer the issue key, PR topic, or branch topic.
- Lowercase all letters.
- Replace runs of non-alphanumeric characters with one hyphen.
- Trim leading and trailing hyphens.
- Keep it short enough to scan in GitHub Actions and image tags.
## Build The InsForge Image
Dispatch the InsForge `Build and Push Docker Image` workflow with the test tag:
```bash
gh workflow run "Build and Push Docker Image" --repo InsForge/InsForge --ref <insforge-feature-branch> -f test_tag=<test-tag>
```
Then wait for the matching run to complete:
```bash
gh run list --repo InsForge/InsForge --workflow "Build and Push Docker Image" --limit 10
gh run watch --repo InsForge/InsForge <run-id>
```
Do not start the cross-repo E2E workflow until the image build succeeds.
## Decide Whether `agent-e2e` Must Change
Inspect the InsForge diff and compare it with deterministic fixture coverage in `InsForge/agent-e2e`.
For read-only checks, prefer remote GitHub access such as `gh api`, `gh repo view`, or remote file reads. Use a local checkout only when editing fixture files or when remote inspection is not enough to understand coverage.
Update `agent-e2e` when the InsForge change adds, removes, or changes behavior that the deterministic fixture should assert, including:
- API contract or validation behavior.
- Auth, permissions, RLS, storage, realtime, functions, schedules, AI, SDK, or CLI-visible behavior.
- Any regression that local tests cover but the release gate should also protect across the deployed runtime.
Do not update `agent-e2e` for internal refactors, docs-only changes, local test-only changes, or behavior already covered by the deterministic fixture with no assertion change needed.
Ignore `Support Desk Agent E2E (Exploratory)`. It is not part of this gate.
## If No E2E Test Update Is Needed
Run the deterministic fixture workflow from `agent-e2e` main:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref main -f insforge_tag=<test-tag>
```
Find and watch the run:
```bash
gh run list --repo InsForge/agent-e2e --workflow "Deterministic Fixture E2E" --limit 10
gh run watch --repo InsForge/agent-e2e <run-id>
```
## If E2E Tests Need An Update
Work in a local checkout of the remote `InsForge/agent-e2e` repo only for the fixture update.
1. Use an existing clean checkout or clone `https://github.com/InsForge/agent-e2e.git` into an isolated workspace.
2. Fetch `origin` and start from `origin/main`.
3. Create a branch named `codex/<short-topic>`.
4. Update only the deterministic fixture validators, fixtures, app assertions, and docs needed for the InsForge behavior change.
5. Run the smallest local validation that gives confidence:
```bash
npm run typecheck
npm run lint
npm run fixture:e2e:dry
```
Run broader validation when the changed fixture area supports it.
6. Open an `agent-e2e` PR for the fixture update.
7. Dispatch the deterministic fixture workflow from the `agent-e2e` branch that contains the fixture update:
```bash
gh workflow run "Deterministic Fixture E2E" --repo InsForge/agent-e2e --ref <agent-e2e-branch> -f insforge_tag=<test-tag>
```
8. Watch the run before proceeding with the InsForge PR.
## Interpret Results
If the deterministic fixture workflow passes:
- Link the run in the InsForge PR body or final PR notes.
- If an `agent-e2e` PR was required, link that PR too.
- Proceed with opening, updating, or submitting the InsForge OSS PR.
If the deterministic fixture workflow fails:
1. Inspect the failed job logs and uploaded artifact.
2. Identify whether the failure is caused by the InsForge implementation, the new or existing deterministic fixture, a transient infrastructure problem, or an unrelated existing failure.
3. Fix the correct branch:
- InsForge implementation bug: update the InsForge branch, rebuild the test image with the same tag or a new short retry tag, then rerun E2E.
- Fixture bug or missing assertion update: update the `agent-e2e` branch, rerun local validation, then rerun E2E from that branch.
- Transient infrastructure issue: rerun once after noting the evidence.
4. Do not submit the InsForge PR until the deterministic result is clear or the user explicitly accepts the risk.
## Report Back
When finished, report:
- Test tag used.
- InsForge image workflow run result.
- Whether `agent-e2e` changed.
- Deterministic Fixture E2E run result.
- Links to the InsForge PR, `agent-e2e` PR if any, and workflow runs.
.codex/skills/insforge-dev/shared-schemas/SKILL.md
+43
View File
@@ -0,0 +1,43 @@
---
name: shared-schemas
description: Use this skill when contributing to InsForge's shared schema package. This is for maintainers editing published Zod contracts, exported types, and shared API payload definitions consumed by InsForge packages in this repo and other InsForge tooling.
---
# InsForge Dev Shared Schemas
Use this skill for `packages/shared-schemas/` work in the InsForge repository.
## Scope
- `packages/shared-schemas/src/**`
- any code in this repo that consumes the changed contracts
- downstream InsForge tooling that depends on the published `@insforge/shared-schemas` package, including consumers not present in this repository
## Working Rules
1. Treat `packages/shared-schemas/` as the source of truth for cross-package payloads.
- If a request, response, or domain shape is shared across InsForge surfaces, define it here.
- Do not duplicate the same contract in package-local files.
- Remember that this package is not only for the repo's backend and dashboard packages. It is also consumed by other InsForge tooling, including MCP and SDK code that may live outside this repository.
2. Keep schemas organized by domain.
- Follow the existing `*.schema.ts` and `*-api.schema.ts` split when it fits the current package pattern.
- Keep `packages/shared-schemas/src/index.ts` aligned with the intended public API.
- Treat exported names and schema shapes as a public contract surface, not just an internal refactor target.
3. Use schema changes as a synchronization trigger.
- Update backend validation and response usage.
- Update shared dashboard services, hooks, and UI assumptions in `packages/dashboard/`.
- Check for import sites across `packages/*`, `frontend/`, and `backend/` before finishing.
- Call out likely downstream impact on MCP, SDK, or other external InsForge consumers when a change alters exported names, schema semantics, or payload shape.
- Be conservative with breaking changes. If a breaking contract change is necessary, make it explicit in the handoff.
- Never use the TypeScript `any` type. Shared contracts should stay explicit and trustworthy across package boundaries.
## Validation
- `cd packages/shared-schemas && npm run build`
- `cd backend && npx tsc --noEmit`
- `cd packages/dashboard && npm run typecheck`
Run package-specific tests as needed where behavior changed.
If external consumers such as MCP or SDK cannot be validated from this repo, say that clearly instead of implying they were covered.
.codex/skills/insforge-dev/ui/SKILL.md
+42
View File
@@ -0,0 +1,42 @@
---
name: ui
description: Use this skill when contributing to InsForge's reusable UI package. This is for maintainers editing design-system primitives, exports, styles, and package-level component behavior in the InsForge monorepo.
---
# InsForge Dev UI
Use this skill for `packages/ui/` work in the InsForge repository.
## Scope
- `packages/ui/src/components/**`
- `packages/ui/src/lib/**`
- `packages/ui/src/index.ts`
- `packages/ui/src/styles.css`
## Working Rules
1. Put only reusable primitives here.
- If the component is generic across dashboard features or other InsForge apps, it belongs in `packages/ui/`.
- If it is tightly coupled to one dashboard workflow but should ship to both OSS and cloud hosts, keep it in `packages/dashboard/`.
- If it is only for the self-hosting host app, keep it in `frontend/`.
2. Preserve the package's implementation style.
- Use `class-variance-authority` for variants when appropriate.
- Use the shared `cn()` helper for class merging.
- Follow the existing Radix-wrapper and typed-export patterns.
3. Keep the public surface in sync.
- Export new public components from `packages/ui/src/index.ts`.
- Avoid adding internal-only abstractions to the package surface unless they are meant to be consumed.
- Never use the TypeScript `any` type. Keep component props and exported helpers strictly typed.
4. Validate downstream impact.
- The shared dashboard package consumes this package directly, so UI changes can break `packages/dashboard/` even if `packages/ui/` itself builds cleanly.
## Validation
- `cd packages/ui && npm run build`
- `cd packages/ui && npm run typecheck`
Also validate `packages/dashboard/` when the changed component is used in the dashboard, and validate `frontend/` if the host app integration or CSS entrypoints changed.
.dockerignore
+64
View File
@@ -0,0 +1,64 @@
# Dependencies
node_modules/
frontend/node_modules/
auth/node_modules/
shared-schemas/node_modules/
backend/node_modules/
# Build outputs
dist/
frontend/dist/
*.tsbuildinfo
# Data and logs
data/
*.log
npm-debug.log*
# Environment files
.env
.env.local
.env.*.local
# Development files
.git/
.gitignore
*.md
docs/**
!docs/sdks/**
!docs/snippets/**
!.agents/docs/**
tests/
*.test.ts
*.test.tsx
coverage/
.nyc_output/
# IDE
.vscode/
.idea/
*.swp
*.swo
.DS_Store
# Docker files (prevent recursion)
Dockerfile
docker-compose*.yml
.dockerignore
# Development configs
vitest.config.ts
# tsconfig.json files are needed for build - DO NOT IGNORE
# frontend/tsconfig.json is needed for @ alias resolution - DO NOT IGNORE
# frontend/vite.config.ts is needed for build - DO NOT IGNORE
.eslintrc.json
frontend/.eslintrc.cjs
# CI/CD
.github/
.gitlab-ci.yml
# Temporary files
tmp/
temp/
*.tmp
.env.example
+315
View File
@@ -0,0 +1,315 @@
# =============================================================================
# InsForge Environment Configuration
# =============================================================================
# Copy this file to .env and fill in your values:
# cp .env.example .env
#
# Security Notes:
# - Never commit .env to version control
# - Use strong, unique secrets in production
# - Rotate secrets regularly
# =============================================================================
# -----------------------------------------------------------------------------
# Server Configuration
# -----------------------------------------------------------------------------
PORT=7130
# Max body size for JSON payloads (default: 100mb)
# High default ensures "out-of-the-box" reliability for large metadata/storage requests.
# Users can decrease this for hardened security on low-resource environments.
MAX_JSON_BODY_SIZE=100mb
# Max body size for URL-encoded payloads (default: 10mb)
MAX_URLENCODED_BODY_SIZE=10mb
# -----------------------------------------------------------------------------
# PostgreSQL Configuration
# -----------------------------------------------------------------------------
# These are optional - defaults are shown below
# In production, use strong passwords and consider external database services
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=insforge
# -----------------------------------------------------------------------------
# Ports (Configurable)
# -----------------------------------------------------------------------------
# PostgreSQL
POSTGRES_PORT=5432
# PostgREST API
POSTGREST_PORT=5430
# Main application ports
APP_PORT=7130
AUTH_PORT=7131
UI_PORT=7132
# Deno serverless runtime
DENO_PORT=7133
# API Base URLs - Update if running on different host/port
API_BASE_URL=http://localhost:7130
VITE_API_BASE_URL=http://localhost:7130
# -----------------------------------------------------------------------------
# Authentication & Security
# -----------------------------------------------------------------------------
# JWT_SECRET: Must be at least 32 characters. Use a secure random generator!
# Example: openssl rand -base64 32
JWT_SECRET=your-secret-key-here-must-be-32-char-or-above
# Root admin credentials - CHANGE THESE IN PRODUCTION!
ROOT_ADMIN_USERNAME=admin
ROOT_ADMIN_PASSWORD=change-this-password
# Encryption key for secrets and database encryption
# IMPORTANT: Set this to a separate 32+ character secret from JWT_SECRET.
# If not set, JWT_SECRET is used as fallback — but rotating JWT_SECRET will
# permanently corrupt all stored secrets (API keys, OAuth tokens, etc.).
# Generate with: openssl rand -base64 32
ENCRYPTION_KEY=
# API Key for authenticated requests
# Must start with 'ik_' prefix and be at least 32 characters
# Optional - will be auto-generated if not provided
ACCESS_API_KEY=ik_your-api-key-here-32-chars-minimum
# Anon Key for public client requests (safe to embed in frontends)
# Must start with 'anon_' prefix
# Optional - will be auto-generated if not provided
ACCESS_ANON_KEY=anon_your-anon-key-here
# Cloud API Host (Optional)
# Only needed if using cloud features
CLOUD_API_HOST=https://api.insforge.dev
# -----------------------------------------------------------------------------
# Deployment Configuration (Optional)
# -----------------------------------------------------------------------------
# Required for self-hosted site deployments and custom domains.
# Legacy deployments also require AWS_S3_BUCKET to be configured.
VERCEL_TOKEN=
VERCEL_TEAM_ID=
VERCEL_PROJECT_ID=
# -----------------------------------------------------------------------------
# AWS Config Bucket Configuration (Optional)
# -----------------------------------------------------------------------------
# Used for loading remote configuration files from S3
# If not provided, defaults will be used
AWS_CONFIG_BUCKET=insforge-config
AWS_CONFIG_REGION=us-east-2
# -----------------------------------------------------------------------------
# Storage Configuration
# -----------------------------------------------------------------------------
# Storage options:
# - Local filesystem: Leave AWS_S3_BUCKET empty (default)
# - AWS S3: Set AWS_S3_BUCKET, AWS_REGION, and AWS credentials
# - S3-compatible (Wasabi, MinIO, etc.): Set S3_* variables
#
# Local storage directory (default: ./insforge-storage relative to cwd)
# In Docker this is set to /insforge-storage automatically via the Dockerfile.
# PaaS deployments (Zeabur, Render, etc.) should set this to match the
# persistent volume mount path.
# STORAGE_DIR=./insforge-storage
#
# AWS credentials (used by CloudWatch logging and as fallback for storage)
AWS_ACCESS_KEY_ID=
AWS_REGION=
AWS_S3_BUCKET=
AWS_SECRET_ACCESS_KEY=
# Max upload file size in bytes (default: 52428800 = 50MB)
# Examples: 10485760 = 10MB, 104857600 = 100MB
MAX_FILE_SIZE=
# S3-compatible storage credentials (for Wasabi, MinIO, R2, etc.)
# These take precedence over AWS_* variables if set
S3_ACCESS_KEY_ID=
S3_ENDPOINT_URL=
S3_SECRET_ACCESS_KEY=
# Path-style addressing (default true, required by MinIO). Set to false for
# providers that require virtual-hosted-style (Tencent COS, Aliyun OSS).
S3_FORCE_PATH_STYLE=true
# CloudFront signed URLs (Optional, AWS S3 only)
# When AWS_CLOUDFRONT_URL is set, storage downloads are served through
# CloudFront with signed URLs instead of S3 presigned URLs. Ignored when
# S3_ENDPOINT_URL is set (CloudFront does not front S3-compatible providers).
# Setup: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-trusted-signers.html
#
# AWS_CLOUDFRONT_URL: CloudFront distribution domain (e.g. https://d1234abcd.cloudfront.net)
# AWS_CLOUDFRONT_KEY_PAIR_ID: Trusted key group key-pair ID (e.g. K2JCJMDEHXQW5F)
# AWS_CLOUDFRONT_PRIVATE_KEY: PEM private key. Use \n for newlines when kept on one line.
# If URL is set but key pair or private key is missing, downloads fall back to S3 presigned URLs.
AWS_CLOUDFRONT_URL=
AWS_CLOUDFRONT_KEY_PAIR_ID=
AWS_CLOUDFRONT_PRIVATE_KEY=
# -----------------------------------------------------------------------------
# Logging Configuration
# -----------------------------------------------------------------------------
# CloudWatch Logging: Set AWS_REGION + AWS credentials above
# File-based Logging: Configure LOGS_DIR below (used when AWS credentials not provided)
# Directory where log files will be stored (defaults to ./logs)
LOGS_DIR=
# -----------------------------------------------------------------------------
# Anonymous Telemetry
# -----------------------------------------------------------------------------
# InsForge sends anonymous self-host usage events so maintainers can understand
# active deployments, version adoption, and deployment success. It never sends
# secrets, environment variables, logs, domains, file paths, project names, or
# database contents.
#
# Disable all telemetry:
# INSFORGE_TELEMETRY_DISABLED=1
# -----------------------------------------------------------------------------
# AI/LLM Configuration
# -----------------------------------------------------------------------------
# OpenRouter API - Get your API key from https://openrouter.ai/keys
# Used for AI-powered features and model gateway
OPENROUTER_API_KEY=
# Maximum output tokens per chat completion request (default: 16384).
# Increase this for models with larger output windows (e.g. 32768 for 32K-output models).
# Must be a positive integer. Invalid values are silently ignored and the default is used.
MAX_COMPLETION_TOKENS=16384
# -----------------------------------------------------------------------------
# Stripe Payments Configuration (Optional)
# -----------------------------------------------------------------------------
# Developer-owned Stripe secret keys. Test key is used for implementation and
# validation. Live key is used only for explicit go-live flows.
STRIPE_LIVE_SECRET_KEY=
STRIPE_TEST_SECRET_KEY=
# -----------------------------------------------------------------------------
# Razorpay Payments Configuration (Optional)
# -----------------------------------------------------------------------------
# Developer-owned Razorpay keys. Test keys are used for implementation and
# validation. Live keys are used only for explicit go-live flows.
RAZORPAY_LIVE_KEY_ID=
RAZORPAY_LIVE_KEY_SECRET=
RAZORPAY_TEST_KEY_ID=
RAZORPAY_TEST_KEY_SECRET=
# -----------------------------------------------------------------------------
# Analytics Configuration
# -----------------------------------------------------------------------------
# PostHog - Only needed for local development
# Get your key from https://posthog.com/settings/project
VITE_PUBLIC_POSTHOG_KEY=
# -----------------------------------------------------------------------------
# OAuth Configuration (Optional)
# -----------------------------------------------------------------------------
# Enable social login by configuring one or more providers below.
# Each provider requires registering an application in their developer console.
#
# Google OAuth
# Console: https://console.cloud.google.com/
# Redirect URI: http://localhost:7130/auth/google/callback
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
# GitHub OAuth
# Console: https://github.com/settings/developers
# Redirect URI: http://localhost:7130/auth/github/callback
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
# Microsoft OAuth (Azure AD)
# Console: https://portal.azure.com/
# Redirect URI: http://localhost:7130/auth/microsoft/callback
MICROSOFT_CLIENT_ID=
MICROSOFT_CLIENT_SECRET=
# Discord OAuth
# Console: https://discord.com/developers/applications
# Redirect URI: http://localhost:7130/auth/discord/callback
DISCORD_CLIENT_ID=
DISCORD_CLIENT_SECRET=
# LinkedIn OAuth
# Console: https://www.linkedin.com/developers/apps
# Redirect URI: http://localhost:7130/auth/linkedin/callback
LINKEDIN_CLIENT_ID=
LINKEDIN_CLIENT_SECRET=
# X (Twitter) OAuth
# Console: https://developer.twitter.com/en/portal/dashboard
# Redirect URI: http://localhost:7130/auth/x/callback
X_CLIENT_ID=
X_CLIENT_SECRET=
# Apple OAuth (Sign in with Apple)
# Console: https://developer.apple.com/account/resources/identifiers/list
# APPLE_CLIENT_ID: Your Services ID (e.g., com.yourapp.service)
# APPLE_CLIENT_SECRET: JSON string with teamId, keyId, and privateKey
# Format: {"teamId":"XXX","keyId":"YYY","privateKey":"-----BEGIN PRIVATE KEY-----\n..."}
# Redirect URI: http://localhost:7130/auth/apple/callback
APPLE_CLIENT_ID=
APPLE_CLIENT_SECRET=
# -----------------------------------------------------------------------------
# Multi-tenant Cloud Configuration
# -----------------------------------------------------------------------------
# These are only used for the cloud-hosted solution
# Leave empty for self-hosted deployments
DEPLOYMENT_ID=
PROJECT_ID=
APP_KEY=
# -----------------------------------------------------------------------------
# Serverless Functions Configuration
# -----------------------------------------------------------------------------
# Deno Deploy - For hosting serverless edge functions
# Get your token from: https://dash.deno.com/account#access-tokens
# Get your org ID from: https://dash.deno.com/
DENO_DEPLOY_TOKEN=
DENO_DEPLOY_ORG_ID=
# ─── Compute Services (Fly.io) ──────────────────────────────────────────────
# Deploy Docker containers with persistent URLs.
# Full setup + architecture: https://docs.insforge.dev/core-concepts/compute/architecture
#
# Setup — both FLY_API_TOKEN and FLY_ORG are required. FLY_ORG must be YOUR
# Fly org slug; leaving it blank or as "insforge" will fail with an opaque
# Fly auth error because that's our internal org. Compute auto-enables when
# both are set.
# 1. Create a Fly.io account: https://fly.io
# 2. Find your org slug: fly orgs list
# 3. Create an API token scoped to that org: fly tokens create org -o <your-org>
# 4. Set FLY_API_TOKEN=<token> and FLY_ORG=<your-org>
#
# Optional: Set COMPUTE_DOMAIN only if you own a wildcard domain pointed at
# Fly (e.g. *.compute.yourdomain.com CNAME'd to fly.dev). Leave unset to use
# Fly's own .fly.dev hostname, which works out of the box.
FLY_API_TOKEN=
FLY_ORG=
COMPUTE_DOMAIN=
# Cloud-mode compute (provisioned automatically when PROJECT_ID + CLOUD_API_HOST
# are set by insforge-cloud). Self-host users with FLY_API_TOKEN always take
# priority and do not need to set PROJECT_ID.
# =============================================================================
# End of Configuration
# =============================================================================
# After configuring this file:
# 1. Run: docker compose up -d
# 2. Check logs: docker compose logs -f
# 3. Access the dashboard at: http://localhost:7131
#
# For more information, visit: https://insforge.dev/docs
# =============================================================================
.github/FUNDING.yml
+1
View File
@@ -0,0 +1 @@
github: [InsForge]
.github/ISSUE_TEMPLATE/bug_report.yml
+36
View File
@@ -0,0 +1,36 @@
name: 🐛 Bug Report
description: Report a bug to help us improve
title: "[Bug]: "
labels: ["bug", "needs-triage"]
assignees: []
body:
- type: markdown
attributes:
value: |
Thanks for reporting a bug! Fill out what you can - we'll figure out the rest together.
- type: textarea
id: what-happened
attributes:
label: What's the bug?
description: Describe what went wrong
placeholder: Tell us what happened...
validations:
required: true
- type: textarea
id: steps-reproduce
attributes:
label: How to reproduce
description: Steps to trigger the bug (if you know them)
placeholder: |
1. Do this...
2. Then this...
3. Bug happens
- type: textarea
id: environment
attributes:
label: Environment (optional)
description: What are you running? (OS, browser, version, etc.)
placeholder: e.g., macOS, Chrome, Docker, Node 20...
.github/ISSUE_TEMPLATE/config.yml
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: 📚 Documentation
url: https://docs.insforge.dev
about: Check our documentation for common questions and setup guides
- name: 💬 Discussions
url: https://github.com/InsForge/InsForge/discussions
about: Ask questions and discuss ideas with the community
- name: 🆘 Discord Support
url: https://discord.com/invite/MPxwj5xVvW
about: Get help with installation, configuration, and usage
.github/ISSUE_TEMPLATE/feature_request.yml
+26
View File
@@ -0,0 +1,26 @@
name: ✨ Feature Request
description: Suggest an idea or enhancement
title: "[Feature]: "
labels: ["enhancement", "needs-triage"]
assignees: []
body:
- type: markdown
attributes:
value: |
Have an idea? Share it! We'd love to hear what you're thinking.
- type: textarea
id: feature-description
attributes:
label: What feature do you want?
description: Describe your idea
placeholder: I'd like to be able to...
validations:
required: true
- type: textarea
id: why
attributes:
label: Why do you need this?
description: What problem does it solve?
placeholder: This would help me...
.github/PULL_REQUEST_TEMPLATE.md
+7
View File
@@ -0,0 +1,7 @@
## Summary
<!-- Briefly describe what this PR does -->
## How did you test this change?
<!-- Describe how you tested this PR -->
.github/copilot-instructions.md
+147
View File
@@ -0,0 +1,147 @@
# Project Overview
InsForge is an open-source Backend-as-a-Service (BaaS) platform designed specifically for AI agents. It provides a comprehensive solution for managing backend services including authentication, database operations, storage, and serverless functions, all accessible through REST APIs with PostgreSQL as the primary database.
## Architecture
This is a monorepo containing:
- **Backend**: Node.js with Express.js, providing RESTful APIs
- **Frontend**: React with Vite, offering an admin dashboard
- **Functions**: Serverless function runtime using Deno
- **Shared-schemas**: Common TypeScript schemas shared across modules
## Folder Structure
- `/backend`: Node.js backend server
- `/src/api`: API routes and middleware
- `/src/controllers`: Business logic controllers
- `/src/core`: Core services (auth, database, storage, metadata)
- `/src/types`: TypeScript type definitions
- `/src/utils`: Utility functions and helpers
- `/frontend`: React dashboard application
- `/src/components`: Reusable UI components
- `/src/features`: Feature-specific modules (auth, database, storage, logs)
- `/src/lib`: Shared libraries, hooks, and utilities
- `/functions`: Serverless function runtime
- `/shared-schemas`: Shared TypeScript schemas
- `/docker-init`: Docker initialization scripts
- `/openapi`: API documentation in OpenAPI format
- `/tests`: Comprehensive test suites
## Libraries and Frameworks
### Backend
- Express.js for REST API framework
- PostgreSQL with pg driver for database
- Better-auth for authentication
- AWS SDK for S3-compatible storage
- Zod for schema validation
- JWT for token management
- TypeScript for type safety
### Frontend
- React 19 with functional components
- Vite for build tooling
- TanStack Query for data fetching
- React Hook Form with Zod validation
- Tailwind CSS for styling
- Radix UI for accessible component primitives
- React Router for navigation
- TypeScript for type safety
## Coding Standards
### General
- Use TypeScript for all code files
- Prefer descriptive, unabbreviated variable and function names
- Follow consistent file naming: kebab-case for files, PascalCase for components
- Maintain clear folder structure with feature-based organization
- Use ES modules (import/export syntax)
- Implement proper error handling with try-catch blocks
- Add appropriate logging for debugging
### Backend Specific
- Use async/await for asynchronous operations
- Implement proper middleware for authentication and error handling
- Follow RESTful conventions for API endpoints
- Use Zod schemas for request/response validation
- Return consistent API responses using utility functions
- Implement proper database transaction handling
### Frontend Specific
- Use functional components with hooks exclusively
- Implement proper TypeScript interfaces for all props
- Use React Hook Form for form handling
- Follow component composition patterns
- Implement proper loading and error states
- Use TanStack Query for server state management
- Keep components focused and single-purpose
#### Naming Conventions
- **Variables, parameters, properties, and functions**: Use camelCase
- Examples: `userName`, `getUserData()`, `isLoading`
- **React components and TypeScript types/interfaces**: Use PascalCase
- Examples: `UserProfileDialog`, `interface User`, `type TableSchema`
- **Constants and Enum members**: Use UPPER_CASE
- Examples: `const API_BASE_URL`, `enum Status { ACTIVE, INACTIVE }`
- **File naming**:
- Components: `PascalCase.tsx` (e.g., `DataGrid.tsx`)
- Hooks: `use` + `PascalCase.ts` (e.g., `useAuth.ts`)
- Services: `camelCase.service.ts` (e.g., `auth.service.ts`)
- Utils: `kebab-case.ts` or `camelCase.ts` (e.g., `database-utils.ts`)
- **No snake_case**: Avoid snake_case except in SQL queries and external APIs
### Code Style
- Use single quotes for strings
- Include semicolons at the end of statements
- Use arrow functions for callbacks and inline functions
- Prefer const over let, avoid var
- Use template literals for string interpolation
- Maintain consistent indentation (2 spaces)
- Format code with Prettier configuration
## UI Guidelines
- Follow a clean, modern design with consistent spacing
- Use Radix UI primitives for accessibility
- Maintain consistent color scheme using CSS variables
- Use appropriate loading skeletons for data fetching
- Display clear error messages with actionable feedback
- Implement proper form validation with inline errors
- Use consistent icon set from lucide-react
## Testing Standards
- Implement integration tests for API endpoints
- Maintain test coverage above 70%
- Mock external dependencies appropriately
## Security Considerations
- Never commit secrets or API keys
- Use environment variables for configuration
- Implement proper authentication and authorization
- Validate all user inputs
- Sanitize data before database operations
- Use prepared statements for SQL queries
- Implement rate limiting for API endpoints
- Follow OWASP security best practices
## Performance Guidelines
- Implement proper database indexing
- Use pagination for large data sets
- Optimize React component re-renders
- Implement proper caching strategies
- Use lazy loading for code splitting
- Optimize bundle size with proper imports
- Monitor and log performance metrics
## Documentation
- Document all API endpoints with clear descriptions
- Include JSDoc comments for complex functions
- Maintain up-to-date README files
- Document environment variables and configuration
- Provide clear setup instructions
- Include examples for common use cases
.github/workflows/build-image.yml
+202
View File
@@ -0,0 +1,202 @@
name: Build and Push Docker Image
on:
push:
tags:
- 'v*'
workflow_dispatch:
inputs:
test_tag:
description: 'Test tag name (e.g., v0.1.1-test)'
required: false
default: 'v0.1.1-test'
env:
GHCR_REGISTRY: ghcr.io
IMAGE_NAME: insforge-oss
jobs:
build-amd64:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ${{ env.GHCR_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GHCR_PAT }}
- name: Extract metadata for GHCR
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.GHCR_REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}
flavor: |
suffix=-amd64
latest=false
tags: |
type=ref,event=tag
type=raw,value=${{ github.event.inputs.test_tag }},enable=${{ github.event_name == 'workflow_dispatch' }}
- name: Build and push (amd64)
uses: docker/build-push-action@v7
with:
context: .
target: runner
platforms: linux/amd64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=amd64
cache-to: type=gha,mode=max,scope=amd64
build-args: |
VERSION=${{ github.ref_name }}
BUILD_DATE=${{ github.event.head_commit.timestamp }}
COMMIT_SHA=${{ github.sha }}
VITE_PUBLIC_POSTHOG_KEY=${{ secrets.POSTHOG_KEY }}
build-arm64:
runs-on: ubuntu-24.04-arm
permissions:
contents: read
packages: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ${{ env.GHCR_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GHCR_PAT }}
- name: Extract metadata for GHCR
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.GHCR_REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}
flavor: |
suffix=-arm64
latest=false
tags: |
type=ref,event=tag
type=raw,value=${{ github.event.inputs.test_tag }},enable=${{ github.event_name == 'workflow_dispatch' }}
- name: Build and push (arm64)
uses: docker/build-push-action@v7
with:
context: .
target: runner
platforms: linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha,scope=arm64
cache-to: type=gha,mode=max,scope=arm64
build-args: |
VERSION=${{ github.ref_name }}
BUILD_DATE=${{ github.event.head_commit.timestamp }}
COMMIT_SHA=${{ github.sha }}
VITE_PUBLIC_POSTHOG_KEY=${{ secrets.POSTHOG_KEY }}
merge-manifests:
needs: [build-amd64, build-arm64]
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ${{ env.GHCR_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GHCR_PAT }}
- name: Extract metadata for GHCR
id: meta
uses: docker/metadata-action@v6
with:
images: ${{ env.GHCR_REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}
flavor: |
latest=false
tags: |
type=ref,event=tag
type=raw,value=${{ github.event.inputs.test_tag }},enable=${{ github.event_name == 'workflow_dispatch' }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') }}
- name: Create and push multi-arch manifest
env:
GHCR_IMAGE_RAW: ${{ env.GHCR_REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}
SOURCE_VERSION: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.test_tag || github.ref_name }}
run: |
GHCR_IMAGE=$(echo "${GHCR_IMAGE_RAW}" | tr '[:upper:]' '[:lower:]')
for TAG in $(echo "${{ steps.meta.outputs.tags }}" | tr ',' '\n'); do
docker buildx imagetools create \
--tag "${TAG}" \
"${GHCR_IMAGE}:${SOURCE_VERSION}-amd64" \
"${GHCR_IMAGE}:${SOURCE_VERSION}-arm64"
done
push-to-ecr:
needs: merge-manifests
runs-on: ubuntu-latest
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-')
permissions:
id-token: write
contents: read
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ secrets.AWS_ECR_ROLE_ARN }}
aws-region: ${{ secrets.AWS_ECR_REGION }}
- name: Login to Amazon ECR
id: ecr-login
uses: aws-actions/amazon-ecr-login@v2
- name: Log in to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ${{ env.GHCR_REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GHCR_PAT }}
- name: Install crane
uses: imjasonh/setup-crane@6da1ae018866400525525ce74ff892880c099987 # v0.5
- name: Copy multi-arch image to ECR
env:
ECR_REGISTRY: ${{ steps.ecr-login.outputs.registry }}
VERSION: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.test_tag || github.ref_name }}
GHCR_IMAGE_RAW: ${{ env.GHCR_REGISTRY }}/${{ github.repository_owner }}/${{ env.IMAGE_NAME }}
ECR_IMAGE: ${{ steps.ecr-login.outputs.registry }}/${{ env.IMAGE_NAME }}
TAG_LATEST: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && !contains(github.ref_name, '-') }}
run: |
# Convert GHCR image name to lowercase (Docker requires lowercase)
GHCR_IMAGE=$(echo "${GHCR_IMAGE_RAW}" | tr '[:upper:]' '[:lower:]')
# Copy multi-arch image from GHCR to ECR (preserves all platforms)
crane copy ${GHCR_IMAGE}:${VERSION} ${ECR_IMAGE}:${VERSION}
if [[ "${TAG_LATEST}" == "true" ]]; then
crane copy ${GHCR_IMAGE}:${VERSION} ${ECR_IMAGE}:latest
fi
.github/workflows/check-migrations.yml
+29
View File
@@ -0,0 +1,29 @@
name: Check Migrations
on:
push:
branches: [ main ]
pull_request:
branches: [ '*' ]
# Required by the merge queue: the queue builds a temporary `merge_group` ref
# (PR rebased onto the latest main) and waits for this check to report on it.
# Without this trigger the check never runs for queued PRs and the queue stalls.
merge_group:
jobs:
check-duplicate-numbers:
name: Check for duplicate migration numbers
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: '20'
- name: Check for duplicate migration numbers
run: node scripts/check-migration-duplicates.js
working-directory: backend
.github/workflows/ci-premerge-check.yml
+24
View File
@@ -0,0 +1,24 @@
name: CI Pre-merge Check
on:
pull_request:
branches: [ main, master, develop ]
push:
branches: [ main, master, develop ]
workflow_dispatch: # Allows manual triggering for testing
jobs:
docker-build:
name: CI Pre-merge Check
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Build Docker image
uses: docker/build-push-action@v7
with:
context: .
push: false
tags: insforge:test
.github/workflows/e2e.yml
+65
View File
@@ -0,0 +1,65 @@
name: E2E Tests
on:
pull_request:
branches: [main, master, develop]
workflow_dispatch:
jobs:
e2e:
name: E2E Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Create .env file
run: |
cat > .env << EOF
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_DB=insforge
DATABASE_URL=postgresql://postgres:postgres@postgres:5432/insforge
JWT_SECRET=test-secret-key-for-ci-cd-pipeline
ROOT_ADMIN_USERNAME=admin
ROOT_ADMIN_PASSWORD=test-admin-password-for-ci
TEST_API_BASE=http://localhost:7130/api
INSFORGE_DISABLE_WRITE_RATE_LIMIT=1
EOF
- name: Start Docker Compose services
run: docker compose up -d
- name: Wait for PostgreSQL to be ready
run: |
echo "Waiting for PostgreSQL..."
timeout 60 bash -c 'until docker compose exec -T postgres pg_isready -U postgres; do sleep 2; done'
- name: Wait for InsForge backend to be ready
run: |
echo "Waiting for InsForge backend..."
timeout 180 bash -c 'until curl -f http://localhost:7130/api/health 2>/dev/null; do echo "Waiting..."; sleep 5; done'
echo "Backend is ready!"
- name: Install test dependencies in container
run: docker compose exec -T insforge apk add --no-cache bash curl jq postgresql-client
- name: Run E2E tests
run: docker compose exec -T insforge npm run test:e2e
- name: Show container logs on failure
if: failure()
run: |
echo "=== PostgreSQL logs ==="
docker compose logs postgres
echo ""
echo "=== InsForge backend logs ==="
docker compose logs insforge
echo ""
echo "=== PostgREST logs ==="
docker compose logs postgrest
- name: Cleanup
if: always()
run: docker compose down -v
.github/workflows/frontend-tests.yml
+128
View File
@@ -0,0 +1,128 @@
name: Frontend Tests
on:
pull_request:
branches: [main, master, develop]
workflow_dispatch:
permissions:
contents: read
jobs:
dashboard-unit-tests:
name: Dashboard Unit Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Run dashboard unit tests
run: npm --workspace @insforge/dashboard run test:unit
dashboard-component-tests:
name: Dashboard Component Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Run dashboard component tests
run: npm --workspace @insforge/dashboard run test:component
ui-unit-tests:
name: UI Unit Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Run UI unit tests
run: npm --workspace @insforge/ui run test:unit
ui-component-tests:
name: UI Component Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Run UI component tests
run: npm --workspace @insforge/ui run test:component
dashboard-e2e-tests:
name: Dashboard E2E Tests
runs-on: ubuntu-latest
needs:
- dashboard-unit-tests
- dashboard-component-tests
- ui-unit-tests
- ui-component-tests
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Install Playwright Chromium
run: npx playwright install --with-deps chromium
- name: Run dashboard E2E tests
run: npm --workspace @insforge/dashboard run test:ui
- name: Upload Playwright artifacts
if: failure()
uses: actions/upload-artifact@v4
with:
name: dashboard-playwright-results
path: |
packages/dashboard/playwright-report/
packages/dashboard/test-results/
retention-days: 7
.github/workflows/integration-tests.yml
+69
View File
@@ -0,0 +1,69 @@
name: Integration Tests
on:
pull_request:
branches: [main, master, develop]
workflow_dispatch:
permissions:
contents: read
jobs:
integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
services:
postgres:
image: ghcr.io/insforge/postgres:v15.13.2
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: insforge
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
PGHOST: 127.0.0.1
PGPORT: 5432
PGUSER: postgres
PGPASSWORD: postgres
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Install Postgres client
run: |
sudo apt-get update
sudo apt-get install -y postgresql-client
- name: Wait for Postgres
run: |
for i in {1..60}; do
if pg_isready -h 127.0.0.1 -p 5432 -U postgres; then
exit 0
fi
sleep 2
done
echo "postgres not ready in time"
exit 1
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- name: Install dependencies
run: npm ci
- name: Run integration tests
run: cd backend && npm run test:integration
.github/workflows/lint-and-format.yml
+49
View File
@@ -0,0 +1,49 @@
name: Lint and Format Check
on:
push:
branches: [ main ]
pull_request:
branches: [ '*' ]
jobs:
lint-and-format:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Check insforge-dev skill copies are in sync
run: scripts/sync-skills.sh --check
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@v45
with:
files: |
**/*.{ts,tsx,js,jsx,mjs,cjs}
- name: Run ESLint on changed files
if: steps.changed-files.outputs.any_changed == 'true'
run: npx eslint ${{ steps.changed-files.outputs.all_changed_files }}
- name: Get changed files for Prettier
id: changed-files-prettier
uses: tj-actions/changed-files@v45
- name: Check Prettier formatting on changed files
if: steps.changed-files-prettier.outputs.any_changed == 'true'
run: npx prettier --check --ignore-unknown ${{ steps.changed-files-prettier.outputs.all_changed_files }}
- name: Run TypeScript type checking
run: npm run typecheck
.github/workflows/unit-tests.yml
+27
View File
@@ -0,0 +1,27 @@
name: Unit Tests
on:
pull_request:
branches: [main, master, develop]
workflow_dispatch:
jobs:
unit-tests:
name: Unit Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm ci
- name: Run backend unit tests
run: npm run test:backend
.gitignore
+37
View File
@@ -0,0 +1,37 @@
node_modules/
dist/
data/
coverage/
playwright-report/
test-results/
.env
.worktrees/
.DS_Store
*.log
*.db
*.db-shm
*.db-wal
CLAUDE.md
.claude/*
!.claude/skills/
.claude/skills/*
!.claude/skills/README.md
!.claude/skills/insforge-dev/
!.claude/skills/insforge-dev/**
!.claude/skills/doc-author/
!.claude/skills/doc-author/**
.agents/*
!.agents/skills/
.agents/skills/*
!.agents/skills/insforge-dev/
!.agents/skills/insforge-dev/**
!.agents/docs/
!.agents/docs/**
deno.lock
better-auth/
.mcp.json
backend/logs/*
# Turborepo
.turbo
.gstack/
.internal/docs/audits/_audit-2026-04-18.md
+44
View File
@@ -0,0 +1,44 @@
# Docs audit — 2026-04-18
Audit of every `.mdx` file under `docs/` (excluding `docs/deprecated/` —
those are intentional preservation) against the rules in
`.claude/skills/doc-author/SKILL.md` (upstream Mintlify, MIT) and the
InsForge overlay at `.claude/skills/doc-author/INSFORGE.md`.
Methodology: targeted `grep` passes for each rule category plus manual
cross-check of `docs/docs.json` navigation entries against the filesystem.
Fixed in this PR: the top five high-severity rows. Everything else is
filed as follow-up tickets per #1117.
**Key:** severity is the operational impact if the issue ships.
`high` = broken nav or broken link a reader will hit; `med` = visible
style inconsistency with the overlay or an orphaned page; `low` = minor
copy nit.
## Findings
| # | Page | Issue | Severity | Recommended fix |
|---|------|-------|----------|-----------------|
| 1 | `docs/docs.json` (lines 125-127) | Nav points at `sdks/typescript/ui-components/react`, `sdks/typescript/ui-components/nextjs`, `sdks/typescript/ui-components/customization` — none of these `.mdx` files exist on disk, so the entire "UI Components" group breaks Mintlify nav rendering. | high | Remove the three dead entries (and the surrounding `{ "group": "UI Components", ... }` wrapper). Re-add when pages land via a future ticket. |
| 2 | `docs/sdks/typescript/overview.mdx` (lines 38-39) | Same dead targets as #1: body prose links to `/sdks/typescript/ui-components/react` and `/sdks/typescript/ui-components/nextjs`. Readers clicking from the SDK landing page hit 404. | high | Delete the two bullet links (or point them at the npm package pages). Skill §"Internal links": root-relative paths, and only to real pages. |
| 3 | `docs/changelog.mdx` (lines 41, 64, 80, 91, 150, 188, 204) | Seven broken internal links using the obsolete `/core-concepts/{ai,email,authentication,database}/sdk` and `/core-concepts/authentication/ui-components/react` routes. The SDK reference pages live at `/sdks/typescript/*` now. | high | Repoint each link to the current `/sdks/typescript/<module>` page. For the `ui-components/react` link (line 150), drop it — no replacement page exists yet. |
| 4 | `docs/sdks/flutter/*.mdx` (all 6 pages: `overview`, `auth`, `database`, `storage`, `realtime`, `ai`) | Flutter SDK has six reference pages on disk but zero entries in `docs/docs.json` navigation → readers cannot discover them from the site nav. Skill §"What requires escalation": changes that affect navigation structure. Orphaned pages are worse than missing pages. | high | Add a `"Flutter"` group under the `"SDK & Examples"` tab in `docs/docs.json`, mirroring the Swift/Kotlin groups. Six pages in the order `overview → database → auth → storage → realtime → ai` (Flutter has no `functions.mdx`). |
| 5 | `docs/sdks/typescript/email.mdx` (line 6) | Experimental-feature callout uses `<Note>` — overlay INSFORGE.md §"Experimental features get `<Warning>` at top" requires `<Warning>` for experimental/private-preview status. `<Note>` is for neutral context; readers don't register it as a stability signal. Consistent with how `docs/core-concepts/email/architecture.mdx:6-8` and `docs/core-concepts/deployments/architecture.mdx:6-8` already treat experimental features. | high | Change the `<Note>` tag to `<Warning>`, keeping body text identical. |
| 6 | `docs/partnership.mdx` (lines 76-91) | 16 lines begin with `- ✅` emoji-prefixed bullets. Skill §"What to avoid / Never use: Emoji in documentation." | med | Strip the ✅ emoji; plain bullets read as checklist items already. One-commit, zero structural change. |
| 7 | `docs/core-concepts/realtime/sdk.mdx` and `docs/core-concepts/storage/sdk.mdx` | Both pages exist on disk but are NOT in `docs/docs.json` navigation — they're orphaned. They also duplicate content that lives at `/sdks/typescript/realtime` and `/sdks/typescript/storage` (the canonical SDK-tab pages). Deprecated-API smell: the SDK-tab pages are canonical, the core-concepts pages are the older location. | med | Follow-up ticket: either delete both (canonical SDK docs live under `/sdks/typescript/*` per docs.json:110-141) or add them to nav. Audit recommends deletion; `changelog.mdx:106` still links to `/core-concepts/realtime/sdk` so a link rewrite is part of the same ticket. |
| 8 | `docs/sdks/rest/storage.mdx` (sections at lines 154, 196, 289) | Three H2 sections titled "Upload Object (Deprecated)", "Upload with Auto-Generated Key (Deprecated)", "Download Object (Deprecated)". The parenthetical label works, but a `<Warning>` callout inside each section would make the status machine-readable (skill §"Callouts / `<Warning>` — something potentially destructive"). | med | Follow-up ticket: prepend a `<Warning>These endpoints are deprecated. Use [...] instead.</Warning>` callout at the top of each of the three sections; do not remove the content (REST reference is intentionally complete). |
| 9 | `docs/sdks/typescript/database.mdx` (line 379) | Example imports `@insforge/react` with an inline comment `// or '@insforge/react-router'`. No `/sdks/typescript/ui-components/*` page exists (see #1) to explain when you'd pick which — the reader has nowhere to go. | med | Follow-up ticket: after UI-components pages are restored (#1), add a cross-link here. Standalone fix not useful. |
| 10 | `docs/quickstart.mdx` (lines 13-15) | Uses markdown numbered list `1. 2. 3.` for a three-action sequence inside "Step 1" (which is itself an H2). Skill §"Steps for sequential procedures: use the `<Steps>` component". Impact is small — the page is already readable. | low | Deferred. Minimal-diff conversion is ~20 lines for a 5-line list; not worth this PR. Folded into catch-all follow-up. |
| 11 | `docs/sdks/typescript/email.mdx` (line 2) | Title is `"Emails SDK Reference"` (plural) but description says `"Send custom transactional emails with the InsForge SDK"` and the module name elsewhere is "Email" (singular; matches `docs.json:218` `"group": "Email"` and the architecture page `docs/core-concepts/email/architecture.mdx`). | low | Rename title to `"Email SDK Reference"` for consistency. Folded into catch-all follow-up. |
| 12 | Many pages | Most H2 headings use Title Case (`## Quick Start`, `## Framework-Specific Packages`, `## Learn More`). Skill §"Voice and structure / Sentence case for headings". This is a systemic repo convention, not a one-page bug. | low | No action. Folded into catch-all follow-up to consider as a separate discussion. Overlay does not override on this, but repo-wide changes are out of scope per ticket #1117. |
## Summary
- **High-severity rows:** 5 (fixed in this PR).
- **Medium-severity rows:** 4 (one follow-up ticket each).
- **Low-severity rows:** 3 (folded into a single catch-all ticket).
## Fix commits in this PR
One commit per fixed row, each citing the skill/overlay section it enforces.
See the `insforge/commander/1117` branch.
.internal/docs/plans/2026-03-16-custom-smtp.md
+2037
View File
File diff suppressed because it is too large Load Diff
.internal/docs/plans/2026-04-07-compute-dashboard-ux.md
+794
View File
@@ -0,0 +1,794 @@
# Compute Dashboard UX Fixes Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make Compute Services usable from the dashboard — add action buttons, create dialog, logs panel, and fix broken endpoint URLs.
**Architecture:** Incremental additions to the existing ComputePage + ServiceCard components. All API methods and mutations already exist in hooks/services. We're just wiring UI to existing plumbing. No backend changes.
**Tech Stack:** React, TypeScript, @insforge/ui (Radix-based), @tanstack/react-query, lucide-react icons
**Spec:** `docs/superpowers/specs/2026-04-07-compute-dashboard-ux-design.md`
---
### File Map
| File | Action | Responsibility |
|------|--------|----------------|
| `packages/dashboard/src/features/compute/constants.ts` | Modify | Add `getReachableUrl` helper, region labels |
| `packages/dashboard/src/features/compute/hooks/useComputeServices.ts` | Modify | Add `useLogs` hook export |
| `packages/dashboard/src/features/compute/components/ServiceLogs.tsx` | Create | Log entries display with refresh |
| `packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx` | Create | Create service form dialog |
| `packages/dashboard/src/features/compute/components/ServiceCard.tsx` | Modify | Add dropdown menu with actions |
| `packages/dashboard/src/features/compute/pages/ComputePage.tsx` | Modify | Add create button, action buttons, logs, enhanced detail view |
---
### Task 1: Add helper utilities to constants.ts
**Files:**
- Modify: `packages/dashboard/src/features/compute/constants.ts`
- [ ] **Step 1: Add `getReachableUrl` helper and region labels**
```ts
import type { ServiceSchema, ServiceStatus } from '@insforge/shared-schemas';
export const statusColors: Record<ServiceStatus, string> = {
running: 'bg-green-500',
deploying: 'bg-yellow-500',
creating: 'bg-yellow-500',
stopped: 'bg-gray-400',
failed: 'bg-red-500',
destroying: 'bg-orange-500',
};
export const CPU_TIERS = [
{ value: 'shared-1x', label: 'Shared 1x' },
{ value: 'shared-2x', label: 'Shared 2x' },
{ value: 'performance-1x', label: 'Performance 1x' },
{ value: 'performance-2x', label: 'Performance 2x' },
{ value: 'performance-4x', label: 'Performance 4x' },
] as const;
export const MEMORY_OPTIONS = [256, 512, 1024, 2048, 4096, 8192] as const;
export const REGIONS = [
{ value: 'iad', label: 'Ashburn, VA (iad)' },
{ value: 'sin', label: 'Singapore (sin)' },
{ value: 'lax', label: 'Los Angeles (lax)' },
{ value: 'lhr', label: 'London (lhr)' },
{ value: 'nrt', label: 'Tokyo (nrt)' },
{ value: 'ams', label: 'Amsterdam (ams)' },
{ value: 'syd', label: 'Sydney (syd)' },
] as const;
/** Return the actual reachable .fly.dev URL instead of a custom domain that may not resolve */
export function getReachableUrl(service: ServiceSchema): string | null {
if (service.flyAppId) {
return `https://${service.flyAppId}.fly.dev`;
}
return service.endpointUrl;
}
```
- [ ] **Step 2: Commit**
```bash
git add packages/dashboard/src/features/compute/constants.ts
git commit -m "feat(compute/ui): add helper utilities for create form and endpoint URL"
```
---
### Task 2: Add useLogs hook to useComputeServices
**Files:**
- Modify: `packages/dashboard/src/features/compute/hooks/useComputeServices.ts`
- [ ] **Step 1: Add useLogs query hook**
Add this new exported hook after the existing `useComputeServices` function in the same file:
```ts
export function useServiceLogs(serviceId: string | null) {
return useQuery({
queryKey: ['compute', 'services', serviceId, 'logs'],
queryFn: () => computeServicesApi.logs(serviceId!, 50),
enabled: !!serviceId,
staleTime: 0, // always refetch on mount for logs
});
}
```
This requires adding `useQuery` to the imports (it's already imported, just need to use it in the new hook).
- [ ] **Step 2: Commit**
```bash
git add packages/dashboard/src/features/compute/hooks/useComputeServices.ts
git commit -m "feat(compute/ui): add useServiceLogs hook"
```
---
### Task 3: Create ServiceLogs component
**Files:**
- Create: `packages/dashboard/src/features/compute/components/ServiceLogs.tsx`
- [ ] **Step 1: Create the logs display component**
```tsx
import { RefreshCw } from 'lucide-react';
import { Button } from '@insforge/ui';
import { useServiceLogs } from '../hooks/useComputeServices';
interface ServiceLogsProps {
serviceId: string;
}
export function ServiceLogs({ serviceId }: ServiceLogsProps) {
const { data: logs = [], isLoading, refetch, isFetching } = useServiceLogs(serviceId);
return (
<div className="bg-card border border-[var(--alpha-8)] rounded-lg overflow-hidden">
<div className="flex items-center justify-between px-4 py-3 border-b border-[var(--alpha-8)]">
<h3 className="text-sm font-medium text-foreground">Logs</h3>
<Button
variant="ghost"
size="sm"
onClick={() => void refetch()}
disabled={isFetching}
>
<RefreshCw className={`h-3.5 w-3.5 ${isFetching ? 'animate-spin' : ''}`} />
Refresh
</Button>
</div>
<div className="max-h-[300px] overflow-y-auto p-4">
{isLoading ? (
<p className="text-xs text-muted-foreground">Loading logs...</p>
) : logs.length === 0 ? (
<p className="text-xs text-muted-foreground">No logs available.</p>
) : (
<pre className="text-xs font-mono text-muted-foreground space-y-0.5">
{logs.map((entry, i) => (
<div key={i}>
<span className="text-foreground/60">
{new Date(entry.timestamp).toISOString().replace('T', ' ').slice(0, 19)}
</span>
{' '}
{entry.message}
</div>
))}
</pre>
)}
</div>
</div>
);
}
```
- [ ] **Step 2: Commit**
```bash
git add packages/dashboard/src/features/compute/components/ServiceLogs.tsx
git commit -m "feat(compute/ui): add ServiceLogs component"
```
---
### Task 4: Create CreateServiceDialog component
**Files:**
- Create: `packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx`
- [ ] **Step 1: Create the dialog component**
```tsx
import { useState } from 'react';
import {
Button,
Dialog,
DialogContent,
DialogHeader,
DialogTitle,
DialogDescription,
DialogBody,
DialogFooter,
Input,
Select,
SelectTrigger,
SelectValue,
SelectContent,
SelectItem,
} from '@insforge/ui';
import { CPU_TIERS, MEMORY_OPTIONS, REGIONS } from '../constants';
import type { CreateServiceRequest } from '@insforge/shared-schemas';
interface CreateServiceDialogProps {
open: boolean;
onOpenChange: (open: boolean) => void;
onCreate: (data: CreateServiceRequest) => Promise<unknown>;
isCreating: boolean;
}
export function CreateServiceDialog({
open,
onOpenChange,
onCreate,
isCreating,
}: CreateServiceDialogProps) {
const [name, setName] = useState('');
const [imageUrl, setImageUrl] = useState('');
const [port, setPort] = useState('8080');
const [cpu, setCpu] = useState('shared-1x');
const [memory, setMemory] = useState('512');
const [region, setRegion] = useState('iad');
const resetForm = () => {
setName('');
setImageUrl('');
setPort('8080');
setCpu('shared-1x');
setMemory('512');
setRegion('iad');
};
const handleSubmit = async () => {
await onCreate({
name,
imageUrl,
port: Number(port),
cpu: cpu as CreateServiceRequest['cpu'],
memory: Number(memory),
region,
});
resetForm();
onOpenChange(false);
};
const isValid = name.length > 0 && imageUrl.length > 0 && Number(port) > 0;
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="max-w-[520px]">
<DialogHeader>
<DialogTitle>Create Service</DialogTitle>
<DialogDescription>Deploy a Docker container as a compute service.</DialogDescription>
</DialogHeader>
<DialogBody>
<div className="flex flex-col gap-4">
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">Name</label>
<Input
placeholder="my-api"
value={name}
onChange={(e) => setName(e.target.value)}
/>
<p className="text-xs text-muted-foreground">DNS-safe: lowercase, numbers, dashes</p>
</div>
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">Image URL</label>
<Input
placeholder="nginx:alpine"
value={imageUrl}
onChange={(e) => setImageUrl(e.target.value)}
/>
</div>
<div className="grid grid-cols-2 gap-4">
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">Port</label>
<Input
type="number"
value={port}
onChange={(e) => setPort(e.target.value)}
/>
</div>
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">Region</label>
<Select value={region} onValueChange={setRegion}>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{REGIONS.map((r) => (
<SelectItem key={r.value} value={r.value}>
{r.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
</div>
<div className="grid grid-cols-2 gap-4">
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">CPU</label>
<Select value={cpu} onValueChange={setCpu}>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{CPU_TIERS.map((t) => (
<SelectItem key={t.value} value={t.value}>
{t.label}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
<div className="flex flex-col gap-1.5">
<label className="text-sm font-medium text-foreground">Memory</label>
<Select value={memory} onValueChange={setMemory}>
<SelectTrigger>
<SelectValue />
</SelectTrigger>
<SelectContent>
{MEMORY_OPTIONS.map((m) => (
<SelectItem key={m} value={String(m)}>
{m} MB
</SelectItem>
))}
</SelectContent>
</Select>
</div>
</div>
</div>
</DialogBody>
<DialogFooter>
<Button
variant="secondary"
size="lg"
disabled={isCreating}
onClick={() => onOpenChange(false)}
>
Cancel
</Button>
<Button
variant="primary"
size="lg"
disabled={!isValid || isCreating}
onClick={() => void handleSubmit()}
>
{isCreating ? 'Creating...' : 'Create Service'}
</Button>
</DialogFooter>
</DialogContent>
</Dialog>
);
}
```
- [ ] **Step 2: Commit**
```bash
git add packages/dashboard/src/features/compute/components/CreateServiceDialog.tsx
git commit -m "feat(compute/ui): add CreateServiceDialog component"
```
---
### Task 5: Add dropdown menu to ServiceCard
**Files:**
- Modify: `packages/dashboard/src/features/compute/components/ServiceCard.tsx`
- [ ] **Step 1: Replace ServiceCard with version that has dropdown actions**
```tsx
import { ExternalLink, MoreVertical, Play, Square, Trash2 } from 'lucide-react';
import {
DropdownMenu,
DropdownMenuTrigger,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
} from '@insforge/ui';
import type { ServiceSchema } from '@insforge/shared-schemas';
import { statusColors, getReachableUrl } from '../constants';
interface ServiceCardProps {
service: ServiceSchema;
onClick: () => void;
onStop: (id: string) => void;
onStart: (id: string) => void;
onDelete: (id: string) => void;
}
export function ServiceCard({ service, onClick, onStop, onStart, onDelete }: ServiceCardProps) {
const reachableUrl = getReachableUrl(service);
return (
<div
role="button"
tabIndex={0}
onClick={onClick}
onKeyDown={(e) => { if (e.key === 'Enter') onClick(); }}
className="w-full text-left bg-card border border-[var(--alpha-8)] rounded-lg p-4 hover:border-foreground/20 transition-colors cursor-pointer"
>
<div className="flex items-center justify-between mb-3">
<h3 className="text-sm font-medium text-foreground truncate">{service.name}</h3>
<div className="flex items-center gap-2">
<span className="flex items-center gap-1.5 text-xs text-muted-foreground">
<span className={`inline-block h-2 w-2 rounded-full ${statusColors[service.status]}`} />
{service.status}
</span>
<DropdownMenu>
<DropdownMenuTrigger
onClick={(e) => e.stopPropagation()}
className="inline-flex h-6 w-6 items-center justify-center rounded text-muted-foreground hover:bg-[var(--alpha-8)] hover:text-foreground"
>
<MoreVertical className="h-3.5 w-3.5" />
</DropdownMenuTrigger>
<DropdownMenuContent align="end" onClick={(e) => e.stopPropagation()}>
{service.status === 'running' && (
<DropdownMenuItem onClick={() => onStop(service.id)}>
<Square className="h-3.5 w-3.5" />
Stop
</DropdownMenuItem>
)}
{service.status === 'stopped' && (
<DropdownMenuItem onClick={() => onStart(service.id)}>
<Play className="h-3.5 w-3.5" />
Start
</DropdownMenuItem>
)}
{(service.status === 'running' || service.status === 'stopped') && (
<DropdownMenuSeparator />
)}
<DropdownMenuItem
onClick={() => onDelete(service.id)}
className="text-destructive"
>
<Trash2 className="h-3.5 w-3.5" />
Delete
</DropdownMenuItem>
</DropdownMenuContent>
</DropdownMenu>
</div>
</div>
<p className="text-xs text-muted-foreground truncate mb-3" title={service.imageUrl}>
{service.imageUrl === 'dockerfile' ? 'Built from Dockerfile' : service.imageUrl}
</p>
{reachableUrl && (
<a
href={reachableUrl}
target="_blank"
rel="noopener noreferrer"
onClick={(e) => e.stopPropagation()}
className="inline-flex items-center gap-1 text-xs text-primary hover:underline mb-3"
>
<ExternalLink className="h-3 w-3" />
{reachableUrl}
</a>
)}
<div className="flex items-center gap-3 text-xs text-muted-foreground pt-2 border-t border-[var(--alpha-8)]">
<span>CPU: {service.cpu}</span>
<span>Memory: {service.memory} MB</span>
<span>{service.region}</span>
</div>
</div>
);
}
```
- [ ] **Step 2: Commit**
```bash
git add packages/dashboard/src/features/compute/components/ServiceCard.tsx
git commit -m "feat(compute/ui): add dropdown actions and fix endpoint URL on ServiceCard"
```
---
### Task 6: Rebuild ComputePage with actions, create button, logs, and enhanced detail view
**Files:**
- Modify: `packages/dashboard/src/features/compute/pages/ComputePage.tsx`
- [ ] **Step 1: Replace ComputePage with the full implementation**
```tsx
import { useState } from 'react';
import { Loader2, ArrowLeft, Plus, Play, Square, Trash2, AlertTriangle } from 'lucide-react';
import { Button, ConfirmDialog } from '@insforge/ui';
import { useComputeServices } from '../hooks/useComputeServices';
import { ServiceCard } from '../components/ServiceCard';
import { ServiceLogs } from '../components/ServiceLogs';
import { CreateServiceDialog } from '../components/CreateServiceDialog';
import { statusColors, getReachableUrl } from '../constants';
import type { ServiceSchema } from '@insforge/shared-schemas';
export default function ComputePage() {
const { services, isLoading, create, remove, stop, start, isCreating } = useComputeServices();
const [selectedService, setSelectedService] = useState<ServiceSchema | null>(null);
const [createOpen, setCreateOpen] = useState(false);
const [deleteTarget, setDeleteTarget] = useState<string | null>(null);
// Keep selected service in sync with latest data
const currentService = selectedService
? services.find((s) => s.id === selectedService.id) ?? selectedService
: null;
const handleDelete = async (id: string) => {
await remove(id);
if (selectedService?.id === id) {
setSelectedService(null);
}
setDeleteTarget(null);
};
if (isLoading) {
return (
<div className="flex items-center justify-center h-64">
<Loader2 className="w-8 h-8 animate-spin text-muted-foreground" />
</div>
);
}
if (currentService) {
const reachableUrl = getReachableUrl(currentService);
return (
<div className="h-full flex flex-col bg-[rgb(var(--semantic-0))]">
<div className="flex-1 min-h-0 overflow-y-auto px-10">
<div className="max-w-[1024px] w-full mx-auto flex flex-col gap-6 pt-10 pb-6">
<button
type="button"
onClick={() => setSelectedService(null)}
className="flex items-center gap-1.5 text-sm text-muted-foreground hover:text-foreground transition-colors self-start"
>
<ArrowLeft className="h-4 w-4" />
Back to services
</button>
<div className="flex items-center justify-between">
<div className="flex items-center gap-3">
<h1 className="text-2xl font-medium text-foreground leading-8">
{currentService.name}
</h1>
<span className="flex items-center gap-1.5 text-sm text-muted-foreground">
<span
className={`inline-block h-2 w-2 rounded-full ${statusColors[currentService.status]}`}
/>
{currentService.status}
</span>
</div>
<div className="flex items-center gap-2">
{currentService.status === 'running' && (
<Button
variant="secondary"
size="sm"
onClick={() => void stop(currentService.id)}
>
<Square className="h-3.5 w-3.5" />
Stop
</Button>
)}
{currentService.status === 'stopped' && (
<Button
variant="secondary"
size="sm"
onClick={() => void start(currentService.id)}
>
<Play className="h-3.5 w-3.5" />
Start
</Button>
)}
<Button
variant="destructive"
size="sm"
onClick={() => setDeleteTarget(currentService.id)}
>
<Trash2 className="h-3.5 w-3.5" />
Delete
</Button>
</div>
</div>
{currentService.status === 'failed' && (
<div className="flex items-center gap-2 px-4 py-3 bg-destructive/10 border border-destructive/20 rounded-lg text-sm text-destructive">
<AlertTriangle className="h-4 w-4 shrink-0" />
This service failed to deploy. You can delete it and try again.
</div>
)}
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-6">
<dl className="grid grid-cols-2 gap-4 text-sm">
<div>
<dt className="text-muted-foreground mb-1">Image</dt>
<dd className="text-foreground break-all">
{currentService.imageUrl === 'dockerfile'
? 'Built from Dockerfile'
: currentService.imageUrl}
</dd>
</div>
<div>
<dt className="text-muted-foreground mb-1">Port</dt>
<dd className="text-foreground">{currentService.port}</dd>
</div>
<div>
<dt className="text-muted-foreground mb-1">CPU</dt>
<dd className="text-foreground">{currentService.cpu}</dd>
</div>
<div>
<dt className="text-muted-foreground mb-1">Memory</dt>
<dd className="text-foreground">{currentService.memory} MB</dd>
</div>
<div>
<dt className="text-muted-foreground mb-1">Region</dt>
<dd className="text-foreground">{currentService.region}</dd>
</div>
<div>
<dt className="text-muted-foreground mb-1">Created</dt>
<dd className="text-foreground">
{new Date(currentService.createdAt).toLocaleDateString(undefined, {
year: 'numeric',
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit',
})}
</dd>
</div>
<div className="col-span-2">
<dt className="text-muted-foreground mb-1">Endpoint URL</dt>
<dd className="text-foreground">
{reachableUrl ? (
<a
href={reachableUrl}
target="_blank"
rel="noopener noreferrer"
className="text-primary hover:underline"
>
{reachableUrl}
</a>
) : (
<span className="text-muted-foreground">Not available</span>
)}
</dd>
</div>
</dl>
</div>
{currentService.flyMachineId && (
<ServiceLogs serviceId={currentService.id} />
)}
</div>
</div>
<ConfirmDialog
open={deleteTarget !== null}
onOpenChange={(open) => { if (!open) setDeleteTarget(null); }}
title="Delete service"
description={`This will permanently delete "${currentService.name}" and destroy its Fly.io resources. This cannot be undone.`}
confirmText="Delete"
destructive
onConfirm={() => handleDelete(deleteTarget!)}
/>
</div>
);
}
return (
<div className="h-full flex flex-col bg-[rgb(var(--semantic-0))]">
<div className="flex-1 min-h-0 overflow-y-auto px-10">
<div className="max-w-[1024px] w-full mx-auto flex flex-col gap-8 pt-10 pb-6">
{/* Services Section */}
<div className="flex flex-col gap-4">
<div className="flex items-center justify-between">
<div className="flex flex-col gap-1">
<h1 className="text-2xl font-medium text-foreground leading-8">Services</h1>
<p className="text-sm leading-5 text-muted-foreground">
Deploy and manage long-running containers on your infrastructure.
</p>
</div>
<Button variant="primary" size="sm" onClick={() => setCreateOpen(true)}>
<Plus className="h-4 w-4" />
Create Service
</Button>
</div>
{services.length === 0 ? (
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-8 text-center">
<p className="text-sm text-muted-foreground mb-2">No services deployed yet.</p>
<p className="text-xs text-muted-foreground mb-4">
Create a service using the button above or the CLI:{' '}
<code className="px-1.5 py-0.5 bg-muted rounded text-xs">
insforge compute create --name my-api --image nginx:alpine
</code>
</p>
</div>
) : (
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
{services.map((service) => (
<ServiceCard
key={service.id}
service={service}
onClick={() => setSelectedService(service)}
onStop={(id) => void stop(id)}
onStart={(id) => void start(id)}
onDelete={setDeleteTarget}
/>
))}
</div>
)}
</div>
{/* Jobs Section Placeholder */}
<div className="flex flex-col gap-2">
<h2 className="text-lg font-medium text-foreground">Jobs</h2>
<div className="bg-card border border-[var(--alpha-8)] rounded-lg p-6 text-center">
<p className="text-sm text-muted-foreground">Coming soon</p>
</div>
</div>
</div>
</div>
<CreateServiceDialog
open={createOpen}
onOpenChange={setCreateOpen}
onCreate={create}
isCreating={isCreating}
/>
<ConfirmDialog
open={deleteTarget !== null}
onOpenChange={(open) => { if (!open) setDeleteTarget(null); }}
title="Delete service"
description="This will permanently delete this service and destroy its Fly.io resources. This cannot be undone."
confirmText="Delete"
destructive
onConfirm={() => {
if (deleteTarget) return handleDelete(deleteTarget);
}}
/>
</div>
);
}
```
- [ ] **Step 2: Verify the app builds**
Run: `cd packages/dashboard && npx tsc --noEmit`
Expected: No type errors
- [ ] **Step 3: Commit**
```bash
git add packages/dashboard/src/features/compute/pages/ComputePage.tsx
git commit -m "feat(compute/ui): add create button, action buttons, logs panel, and enhanced detail view"
```
---
### Task 7: Verify everything works end-to-end
- [ ] **Step 1: Run type check across the whole project**
Run: `npm run typecheck`
Expected: No errors
- [ ] **Step 2: Run all tests**
Run: `npm test`
Expected: All 471+ tests pass
- [ ] **Step 3: Final commit if any fixes were needed**
```bash
git add -A
git commit -m "fix(compute/ui): address typecheck and test issues"
```
.internal/docs/plans/2026-04-15-direct-deploy-flow.md
+811
View File
@@ -0,0 +1,811 @@
# Direct Deploy Flow Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement and ship a manifest-based direct deployment flow that streams source files through InsForge to Vercel without exposing Vercel credentials, while preserving the legacy zip/S3 deployment path for backward compatibility.
**Architecture:** The backend exposes a direct deployment session endpoint that stores deployment rows in `deployments.runs`, stores manifest/upload state in `deployments.files`, and returns a `fileId` per entry. Clients then stream each file to `PUT /api/deployments/:id/files/:fileId/content`; InsForge validates the byte stream against the registered SHA/size and proxies it to Vercel's file upload API. Once every manifest row is marked uploaded, the client calls `POST /api/deployments/:id/start`, and InsForge creates the Vercel deployment from uploaded file SHAs. MCP uses the direct flow when the backend version supports it, and falls back to the legacy zip upload flow when the backend is older or returns `404`.
**Tech Stack:** Express, PostgreSQL, Zod shared schemas, Axios + node-fetch streaming, Vercel file upload API, React service layer, Docker Compose, Vitest, MCP server tooling
**Reference:** `docs/core-concepts/deployments/architecture.mdx`
---
## File Map
### InsForge repo
| File | Purpose |
|------|---------|
| `packages/shared-schemas/src/deployments-api.schema.ts` | Shared request/response contracts for direct deployment manifest, file upload response, and start payload |
| `backend/src/infra/database/migrations/031_create-deployment-files.sql` | Moves `system.deployments` to `deployments.runs` and creates `deployments.files` for direct upload tracking |
| `backend/src/types/deployments.ts` | Backend deployment status semantics used by both direct and legacy flows |
| `backend/src/api/routes/deployments/index.routes.ts` | HTTP entrypoints for legacy create, direct create, file upload, start, metadata, slug, and custom domains |
| `backend/src/services/deployments/deployment.service.ts` | Core orchestration for manifest validation, file upload proxying, direct-vs-legacy branching, and self-hosted Vercel support |
| `backend/src/providers/deployments/vercel.provider.ts` | Streaming file upload implementation against Vercel's `/v2/files` API |
| `backend/src/server.ts` | Global rate-limit bypass for the direct file upload endpoint |
| `backend/tests/unit/deployment-direct-flow.test.ts` | Service-level coverage for direct deployment session creation, upload validation, and self-hosted config behavior |
| `backend/tests/unit/vercel-upload-batching.test.ts` | Provider-level coverage for streaming uploads, 409 dedupe handling, and 429 retry semantics |
| `packages/dashboard/src/features/deployments/services/deployments.service.ts` | Client methods mirroring the direct deployment endpoints for future UI or SDK consumers |
| `.env.example` | Root sample env showing Vercel credentials required for self-hosted deployments |
| `docker-compose.yml` | Dev compose pass-through for Vercel credentials |
| `docker-compose.prod.yml` | Prod compose pass-through for Vercel credentials |
| `deploy/docker-compose/.env.example` | Packaged deployment sample env |
| `deploy/docker-compose/docker-compose.yml` | Packaged deployment compose pass-through |
| `docs/agent-docs/deployment.md` | Agent-facing deployment instructions |
| `docs/core-concepts/deployments/architecture.mdx` | Product/architecture documentation for both direct and legacy flows |
### MCP repo
| File | Purpose |
|------|---------|
| `../insforge-mcp/src/shared/tools/types.ts` | Register context carrying backend version into tool registration |
| `../insforge-mcp/src/shared/tools/index.ts` | Backend version discovery and tool registration |
| `../insforge-mcp/src/shared/tools/deployment.ts` | Direct deployment manifest collection, bounded parallel uploads, remote shell instructions, and legacy fallback |
---
## Task 1: Lock the shared contract and manifest persistence layer
**Files:**
- Modify: `packages/shared-schemas/src/deployments-api.schema.ts`
- Modify: `backend/src/infra/database/migrations/031_create-deployment-files.sql`
- Modify: `backend/src/types/deployments.ts`
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
- Test: `backend/tests/unit/deployment-schema-migration.test.ts`
- [ ] **Step 1: Add the direct deployment request/response schemas**
```ts
export const deploymentManifestFileEntrySchema = z.object({
path: deploymentFilePathSchema,
sha: z.string().regex(/^[a-f0-9]{40}$/i, 'sha must be a SHA-1 hex digest'),
size: z.number().int().nonnegative(),
});
export const deploymentManifestFileSchema = deploymentManifestFileEntrySchema.extend({
fileId: z.string().uuid(),
uploadedAt: z.string().datetime().nullable(),
});
export const createDirectDeploymentRequestSchema = z
.object({
files: z.array(deploymentManifestFileEntrySchema).min(1),
})
.superRefine(({ files }, ctx) => {
const firstSeenByPath = new Map<string, number>();
files.forEach((file, index) => {
const existingIndex = firstSeenByPath.get(file.path);
if (existingIndex !== undefined) {
ctx.addIssue({
code: z.ZodIssueCode.custom,
message: 'duplicate file path',
path: ['files', index, 'path'],
});
return;
}
firstSeenByPath.set(file.path, index);
});
});
export const createDirectDeploymentResponseSchema = z.object({
id: z.string().uuid(),
status: deploymentSchema.shape.status,
files: z.array(deploymentManifestFileSchema),
});
export const uploadDeploymentFileResponseSchema = deploymentManifestFileSchema.extend({
uploadedAt: z.string().datetime(),
});
```
- [ ] **Step 2: Ensure migration 031 moves the published deployments table and creates resumable file tracking**
```sql
CREATE SCHEMA IF NOT EXISTS deployments;
DO $$
BEGIN
IF EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'system' AND table_name = 'deployments'
) THEN
ALTER TABLE system.deployments SET SCHEMA deployments;
END IF;
IF EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'deployments' AND table_name = 'deployments'
) AND NOT EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'deployments' AND table_name = 'runs'
) THEN
ALTER TABLE deployments.deployments RENAME TO runs;
END IF;
END $$;
CREATE TABLE IF NOT EXISTS deployments.files (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
deployment_id UUID NOT NULL REFERENCES deployments.runs(id) ON DELETE CASCADE,
file_path TEXT NOT NULL,
sha TEXT NOT NULL CHECK (sha ~ '^[a-f0-9]{40}$'),
size_bytes INTEGER NOT NULL CHECK (size_bytes >= 0),
uploaded_at TIMESTAMPTZ,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE (deployment_id, file_path)
);
CREATE INDEX IF NOT EXISTS idx_deployment_files_deployment_id
ON deployments.files(deployment_id);
CREATE INDEX IF NOT EXISTS idx_deployment_files_uploaded_at
ON deployments.files(deployment_id, uploaded_at);
DROP TRIGGER IF EXISTS update_system_deployments_updated_at ON deployments.runs;
DROP TRIGGER IF EXISTS update_deployments_updated_at ON deployments.runs;
DROP TRIGGER IF EXISTS update_runs_updated_at ON deployments.runs;
CREATE TRIGGER update_runs_updated_at BEFORE UPDATE ON deployments.runs
FOR EACH ROW EXECUTE FUNCTION system.update_updated_at();
DROP TRIGGER IF EXISTS update_system_deployment_files_updated_at ON deployments.files;
DROP TRIGGER IF EXISTS update_deployment_files_updated_at ON deployments.files;
DROP TRIGGER IF EXISTS update_files_updated_at ON deployments.files;
CREATE TRIGGER update_files_updated_at BEFORE UPDATE ON deployments.files
FOR EACH ROW EXECUTE FUNCTION system.update_updated_at();
```
- [ ] **Step 3: Add failing service tests that pin the contract**
```ts
it('creates a direct deployment and stores its manifest in one transaction', async () => {
const service = DeploymentService.getInstance();
const result = await service.createDirectDeployment({
files: [{ path: 'src/index.ts', sha: 'A'.repeat(40), size: 12 }],
});
expect(result.status).toBe(DeploymentStatus.WAITING);
expect(result.files[0]).toMatchObject({
path: 'src/index.ts',
sha: 'a'.repeat(40),
size: 12,
uploadedAt: null,
});
});
it('rejects duplicate manifest paths before opening a transaction', async () => {
const service = DeploymentService.getInstance();
await expect(
service.createDirectDeployment({
files: [
{ path: 'src/index.ts', sha: 'a'.repeat(40), size: 12 },
{ path: 'src/index.ts', sha: 'b'.repeat(40), size: 13 },
],
})
).rejects.toMatchObject({
statusCode: 400,
code: 'INVALID_INPUT',
});
});
```
- [ ] **Step 4: Run the targeted backend test to verify the contract is red first, then green after implementation**
```bash
cd /Users/lyu/Documents/GitHub/InsForge/backend
npm test -- deployment-direct-flow.test.ts
```
Expected before backend implementation: failure around missing direct deployment behavior or schema mismatch
Expected after Task 2: `Test Files 1 passed`
- [ ] **Step 5: Commit**
```bash
cd /Users/lyu/Documents/GitHub/InsForge
git add \
packages/shared-schemas/src/deployments-api.schema.ts \
backend/src/infra/database/migrations/031_create-deployment-files.sql \
backend/src/types/deployments.ts \
backend/tests/unit/deployment-direct-flow.test.ts
git commit -m "feat: add direct deployment manifest contracts"
```
---
## Task 2: Implement the backend direct upload proxy
**Files:**
- Modify: `backend/src/api/routes/deployments/index.routes.ts`
- Modify: `backend/src/services/deployments/deployment.service.ts`
- Modify: `backend/src/providers/deployments/vercel.provider.ts`
- Modify: `backend/src/server.ts`
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
- Test: `backend/tests/unit/vercel-upload-batching.test.ts`
- [ ] **Step 1: Add the direct deployment HTTP routes**
```ts
router.post('/direct', verifyAdmin, async (req, res, next) => {
try {
const validationResult = createDirectDeploymentRequestSchema.safeParse(req.body);
if (!validationResult.success) {
throw new AppError(
validationResult.error.issues.map((e) => `${e.path.join('.')}: ${e.message}`).join(', '),
400,
ERROR_CODES.INVALID_INPUT
);
}
const response = await deploymentService.createDirectDeployment(validationResult.data);
successResponse(res, response, 201);
} catch (error) {
next(error);
}
});
router.put('/:id/files/:fileId/content', verifyAdmin, async (req, res, next) => {
try {
const abortController = new AbortController();
req.on('aborted', () => abortController.abort());
res.on('close', () => {
if (!res.writableEnded) abortController.abort();
});
const response = await deploymentService.uploadDeploymentFileContent(
req.params.id,
req.params.fileId,
req,
{ signal: abortController.signal }
);
successResponse(res, response);
} catch (error) {
next(error);
}
});
```
- [ ] **Step 2: Implement manifest registration, byte validation, and direct-vs-legacy start branching in the service**
```ts
async createDirectDeployment(
input: CreateDirectDeploymentRequest
): Promise<CreateDirectDeploymentResponse> {
this.assertDeploymentServiceConfigured();
const files = this.validateDeploymentManifest(input.files);
const totalSizeBytes = files.reduce((sum, file) => sum + file.size, 0);
const client = await this.getPool().connect();
try {
await client.query('BEGIN');
const deployment = await client.query(
`INSERT INTO deployments.runs (provider, status, metadata)
VALUES ($1, $2, $3)
RETURNING
id,
provider_deployment_id as "providerDeploymentId",
provider,
status,
url,
metadata,
created_at as "createdAt",
updated_at as "updatedAt"`,
[
'vercel',
DeploymentStatus.WAITING,
JSON.stringify({
uploadMode: 'direct',
fileCount: files.length,
totalSizeBytes,
manifestCreatedAt: new Date().toISOString(),
}),
]
);
const insertedFiles = await this.insertDeploymentFiles(client, deployment.rows[0].id, files);
await client.query('COMMIT');
return {
id: deployment.rows[0].id,
status: deployment.rows[0].status,
files: insertedFiles.map((row) => this.toDeploymentFileResponse(row)),
};
} catch (error) {
await client.query('ROLLBACK').catch(() => {});
throw error;
} finally {
client.release();
}
}
async uploadDeploymentFileContent(
id: string,
fileId: string,
content: Readable,
options: { signal?: AbortSignal } = {}
): Promise<UploadDeploymentFileResponse> {
const deployment = await this.getDeploymentById(id);
const file = await this.getDeploymentFileById(id, fileId);
await this.vercelProvider.uploadFileStream({
content: this.createValidatedFileStream(content, file.sha, file.size),
sha: file.sha,
size: file.size,
signal: options.signal,
});
const uploadedFile = await this.markDeploymentFileUploaded(id, fileId);
return {
...this.toDeploymentFileResponse(uploadedFile),
uploadedAt: uploadedFile.uploadedAt.toISOString(),
};
}
async startDeployment(id: string, input: StartDeploymentRequest = {}): Promise<DeploymentRecord> {
const deployment = await this.getDeploymentById(id);
const files = await this.getDeploymentFiles(id);
const uploadMode = this.getUploadMode(deployment, files.length);
if (uploadMode === 'direct') {
return this.startDirectDeployment(id, input, files);
}
return this.startLegacyDeployment(id, input);
}
```
- [ ] **Step 3: Stream bytes to Vercel without buffering or automatic retry**
```ts
async uploadFileStream(input: {
content: Readable;
sha: string;
size: number;
signal?: AbortSignal;
}): Promise<string> {
const credentials = await this.getCredentials();
try {
await axios.post(`https://api.vercel.com/v2/files?teamId=${credentials.teamId}`, input.content, {
headers: {
Authorization: `Bearer ${credentials.token}`,
'Content-Type': 'application/octet-stream',
'Content-Length': input.size.toString(),
'x-vercel-digest': input.sha,
},
maxBodyLength: Infinity,
maxContentLength: Infinity,
signal: input.signal,
});
return input.sha;
} catch (error) {
if (axios.isAxiosError(error) && error.response?.status === 409) return input.sha;
if (axios.isAxiosError(error) && error.response?.status === 429) {
throw new AppError(
'Vercel rate limit exceeded for file upload. Wait a moment and retry the file upload.',
429,
ERROR_CODES.RATE_LIMITED
);
}
if (axios.isAxiosError(error) && error.code === 'ERR_CANCELED') {
throw new AppError('Vercel file upload was interrupted.', 499, ERROR_CODES.DEPLOYMENT_UPLOAD_CANCELED);
}
throw new AppError('Failed to upload file to Vercel', 500, ERROR_CODES.INTERNAL_ERROR);
}
}
```
- [ ] **Step 4: Bypass the global Express rate limiter for per-file direct uploads**
```ts
function shouldSkipGlobalRateLimit(req: Request): boolean {
if (req.path === '/api/health') {
return true;
}
return (
req.method === 'PUT' && /^\/api\/deployments\/[^/]+\/files\/[^/]+\/content$/.test(req.path)
);
}
```
- [ ] **Step 5: Run focused backend verification**
```bash
cd /Users/lyu/Documents/GitHub/InsForge/backend
npm test -- deployment-direct-flow.test.ts
npm test -- vercel-upload-batching.test.ts
npm run build
```
Expected:
- `deployment-direct-flow.test.ts` passes
- `vercel-upload-batching.test.ts` passes
- `tsup` build succeeds
- [ ] **Step 6: Commit**
```bash
cd /Users/lyu/Documents/GitHub/InsForge
git add \
backend/src/api/routes/deployments/index.routes.ts \
backend/src/services/deployments/deployment.service.ts \
backend/src/providers/deployments/vercel.provider.ts \
backend/src/server.ts \
backend/tests/unit/deployment-direct-flow.test.ts \
backend/tests/unit/vercel-upload-batching.test.ts
git commit -m "feat: add direct deployment upload proxy"
```
---
## Task 3: Wire MCP local and remote deployment flows with compatibility fallback
**Files:**
- Modify: `../insforge-mcp/src/shared/tools/types.ts`
- Modify: `../insforge-mcp/src/shared/tools/index.ts`
- Modify: `../insforge-mcp/src/shared/tools/deployment.ts`
- [ ] **Step 1: Carry backend version through MCP tool registration context**
```ts
export interface RegisterContext {
API_BASE_URL: string;
backendVersion: string;
isRemote: boolean;
registerTool: RegisterToolFn;
withUsageTracking: WithUsageTracking;
getApiKey: GetApiKey;
addBackgroundContext: AddBackgroundContext;
}
```
- [ ] **Step 2: Add direct deployment helpers with bounded parallel file uploads**
```ts
const DIRECT_DEPLOYMENT_MIN_VERSION = '2.0.4';
const DEFAULT_DIRECT_UPLOAD_CONCURRENCY = 8;
const MAX_DIRECT_UPLOAD_CONCURRENCY = 32;
async function deployDirect(
API_BASE_URL: string,
apiKey: string,
sourceDirectory: string,
startBody: StartDeploymentRequest
) {
const files = await collectDeploymentFiles(sourceDirectory);
const manifestFiles = files.map(({ path, sha, size }) => ({ path, sha, size }));
const createResult = await createDirectDeploymentSession(API_BASE_URL, apiKey, manifestFiles);
const localFileByPath = new Map(files.map((file) => [file.path, file] as const));
const uploadConcurrency = getDirectUploadConcurrency();
await runWithConcurrency(createResult.files, uploadConcurrency, async (file) => {
const localFile = localFileByPath.get(file.path);
await uploadDeploymentFileContent(API_BASE_URL, apiKey, createResult.id, file, localFile);
});
const startResult = await startDeployment(API_BASE_URL, apiKey, createResult.id, startBody);
return { deploymentId: createResult.id, fileCount: files.length, uploadConcurrency, startResult };
}
```
- [ ] **Step 3: Prefer the direct flow when supported, but fall back to legacy zip on older backends or `404`**
```ts
const supportsDirectDeployment = supportsDirectDeploymentVersion(backendVersion);
if (supportsDirectDeployment) {
try {
const { fileCount, uploadConcurrency, startResult } = await deployDirect(
API_BASE_URL,
getApiKey(),
resolvedSourceDir,
startBody
);
return await addBackgroundContext({
content: [
{
type: 'text',
text:
formatSuccessMessage('Deployment started', startResult) +
`\n\nUploaded ${fileCount} files through direct deployment proxy with concurrency ${uploadConcurrency}.`,
},
],
});
} catch (error) {
if (!(error instanceof DirectDeploymentUnsupportedError)) {
throw error;
}
}
}
// Fallback to the legacy zip + S3 path.
const createResponse = await fetch(`${API_BASE_URL}/api/deployments`, {
method: 'POST',
headers: {
'x-api-key': getApiKey(),
'Content-Type': 'application/json',
},
});
```
- [ ] **Step 4: Return remote shell instructions that match the same direct flow**
```ts
return `Direct deployment upload is available for this backend.
Please execute the following command locally from a shell that has INSFORGE_API_KEY set, then call the \`start-deployment\` tool with the deployment ID printed by the script. Set \`INSFORGE_DEPLOY_UPLOAD_CONCURRENCY\` if you want to tune parallel uploads; the default is 8 and the maximum is 32.
\`\`\`bash
cd ${escapedDir}
INSFORGE_API_KEY="\${INSFORGE_API_KEY:?Set INSFORGE_API_KEY to your InsForge API key}" node --input-type=module <<'NODE'
const createResult = await api('/api/deployments/direct', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
files: localFiles.map(({ path, sha, size }) => ({ path, sha, size })),
}),
});
await runWithConcurrency(createResult.files, uploadConcurrency, async (manifestFile) => {
const localFile = localFileByPath.get(manifestFile.path);
await uploadFile(createResult.id, manifestFile, localFile);
});
console.log('Deployment files uploaded. Deployment ID: ' + createResult.id);
NODE
\`\`\`
If the upload is interrupted after the deployment ID is printed, query \`deployments.files\` with the raw SQL tool for that \`deployment_id\` to inspect \`uploaded_at\`.`;
```
- [ ] **Step 5: Verify MCP compilation and smoke-test tool registration**
```bash
cd /Users/lyu/Documents/GitHub/insforge-mcp
npm test
npm run build
```
Expected:
- `vitest run --passWithNoTests` exits successfully
- `tsup` build succeeds
- [ ] **Step 6: Commit**
```bash
cd /Users/lyu/Documents/GitHub/insforge-mcp
git add \
src/shared/tools/types.ts \
src/shared/tools/index.ts \
src/shared/tools/deployment.ts
git commit -m "feat: add direct deployment support to mcp"
```
---
## Task 4: Surface self-hosted configuration and document the new flow
**Files:**
- Modify: `packages/dashboard/src/features/deployments/services/deployments.service.ts`
- Modify: `.env.example`
- Modify: `docker-compose.yml`
- Modify: `docker-compose.prod.yml`
- Modify: `deploy/docker-compose/.env.example`
- Modify: `deploy/docker-compose/docker-compose.yml`
- Modify: `docs/agent-docs/deployment.md`
- Modify: `docs/core-concepts/deployments/architecture.mdx`
- [ ] **Step 1: Mirror the direct deployment endpoints in the dashboard service client**
```ts
async createDirectDeployment(
data: CreateDirectDeploymentRequest
): Promise<CreateDirectDeploymentResponse> {
return apiClient.request('/deployments/direct', {
method: 'POST',
headers: apiClient.withAccessToken(),
body: JSON.stringify(data),
});
}
async uploadDeploymentFileContent(
id: string,
fileId: string,
content: Blob | ArrayBuffer
): Promise<UploadDeploymentFileResponse> {
return apiClient.request(`/deployments/${id}/files/${fileId}/content`, {
method: 'PUT',
headers: {
...apiClient.withAccessToken(),
'Content-Type': 'application/octet-stream',
},
body: content,
});
}
```
- [ ] **Step 2: Add self-hosted Vercel credentials to the sample env and compose files**
```env
# Deployment Configuration (Optional)
# Required for self-hosted site deployments and custom domains.
VERCEL_TOKEN=
VERCEL_TEAM_ID=
VERCEL_PROJECT_ID=
```
```yaml
# Deployment Configuration
- VERCEL_TOKEN=${VERCEL_TOKEN:-}
- VERCEL_TEAM_ID=${VERCEL_TEAM_ID:-}
- VERCEL_PROJECT_ID=${VERCEL_PROJECT_ID:-}
```
- [ ] **Step 3: Document the direct flow and the legacy fallback explicitly**
```md
## Overview
Deploy frontend applications to InsForge using the `create-deployment` MCP tool. The tool handles uploading source files, building, and deploying automatically.
Source files are uploaded individually through InsForge's deployment proxy; do not zip the project or upload deployment artifacts to storage yourself.
The REST API still supports the legacy zip upload flow for backward compatibility.
```
```mdx
### Direct Upload Flow
1. **Create Deployment**: Agent calls `POST /api/deployments/direct` with each relative file path, SHA-1 digest, and byte size
2. **Upload Files**: Agent streams each file to `PUT /api/deployments/:id/files/:fileId/content`
3. **Retry If Needed**: Inspect `deployments.files` for `uploaded_at`
4. **Start Build**: Agent calls `POST /api/deployments/:id/start`
```
- [ ] **Step 4: Verify the sample config and dashboard package build**
```bash
cd /Users/lyu/Documents/GitHub/InsForge
docker compose -f docker-compose.yml config --quiet
docker compose -f docker-compose.prod.yml config --quiet
docker compose -f deploy/docker-compose/docker-compose.yml config --quiet
cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard
npm run typecheck
npm run build
```
Expected:
- all three `docker compose ... config --quiet` commands exit 0
- dashboard typecheck succeeds
- dashboard build succeeds
- [ ] **Step 5: Commit**
```bash
cd /Users/lyu/Documents/GitHub/InsForge
git add \
packages/dashboard/src/features/deployments/services/deployments.service.ts \
.env.example \
docker-compose.yml \
docker-compose.prod.yml \
deploy/docker-compose/.env.example \
deploy/docker-compose/docker-compose.yml \
docs/agent-docs/deployment.md \
docs/core-concepts/deployments/architecture.mdx
git commit -m "docs: document direct deploy flow and self-host config"
```
---
## Task 5: Run the full cross-repo verification pass
**Files:**
- Test: `backend/tests/unit/deployment-direct-flow.test.ts`
- Test: `backend/tests/unit/vercel-upload-batching.test.ts`
- Test: `../insforge-mcp/src/shared/tools/deployment.ts`
- Test: compose manifests and docs touched above
- [ ] **Step 1: Run the full backend verification suite**
```bash
cd /Users/lyu/Documents/GitHub/InsForge/backend
npm test
npm run build
```
Expected:
- `vitest run` passes
- backend `tsup` build succeeds
- [ ] **Step 2: Run shared schema and dashboard verification**
```bash
cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas
npm run build
cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard
npm run typecheck
npm run build
```
Expected:
- shared schemas `tsc` build succeeds
- dashboard typecheck and build both succeed
- [ ] **Step 3: Run MCP verification**
```bash
cd /Users/lyu/Documents/GitHub/insforge-mcp
npm test
npm run build
```
Expected:
- MCP tests succeed
- MCP build succeeds
- [ ] **Step 4: Perform the real smoke tests**
```text
1. Legacy backend smoke test:
- Call POST /api/deployments
- Upload a zip to the returned presigned URL
- Call POST /api/deployments/:id/start
- Confirm the flow still works unchanged
2. Direct backend smoke test:
- Call POST /api/deployments/direct with a small manifest
- Upload at least one file to PUT /api/deployments/:id/files/:fileId/content
- Call POST /api/deployments/:id/start
- Confirm deployments.runs transitions to READY
3. MCP smoke test:
- Against a direct-capable backend, run create-deployment from a real source directory
- Confirm direct file uploads happen with bounded concurrency
- Against an older backend or forced 404, confirm fallback to legacy zip upload
```
- [ ] **Step 5: Final commit or tag-ready checkpoint**
```bash
cd /Users/lyu/Documents/GitHub/InsForge
git status
git log --oneline -n 5
```
Expected:
- working tree clean or only intentionally staged release changes
- recent commits show the contract, backend, MCP, and docs/config slices in order
---
## Self-Review
### Spec coverage
- Shared schemas cover manifest registration, returned `fileId`s, upload response, and start payloads.
- Backend plan covers `deployments.runs` / `deployments.files` persistence, streaming proxy, rate-limit bypass, and direct-vs-legacy branching.
- MCP plan covers backend version gating, direct-first behavior, and `404`-based fallback.
- Self-host coverage includes `.env.example`, root compose, packaged compose, and docs.
- Validation covers backend, shared schemas, dashboard, MCP, compose config parsing, and both legacy + direct smoke tests.
### Placeholder scan
- No `TODO`, `TBD`, or "implement later" placeholders remain.
- Every code-changing step includes a concrete code block.
- Every verification step includes a concrete command or explicit manual smoke-test checklist.
### Type consistency
- `CreateDirectDeploymentRequest`, `CreateDirectDeploymentResponse`, `DeploymentManifestFile`, and `StartDeploymentRequest` are used consistently between shared schemas, backend, dashboard service client, and MCP.
- Backend routes and MCP use the same endpoint names: `POST /api/deployments/direct`, `PUT /api/deployments/:id/files/:fileId/content`, and `POST /api/deployments/:id/start`.
- The plan keeps the legacy entrypoint as `POST /api/deployments`.
.internal/docs/plans/2026-04-16-custom-database-migrations.md
+669
View File
@@ -0,0 +1,669 @@
# Custom Database Migrations Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add developer-facing database migrations that execute immediately against the `public` schema and record only successful runs in `system.custom_migrations`.
**Architecture:** The backend owns the migration workflow through a dedicated service and admin routes under `/api/database/migrations`. The service parses and validates SQL, runs it inside a single transaction with a transaction-scoped advisory lock, and inserts a history row into `system.custom_migrations` only after the SQL succeeds. Shared schemas define the request/response contract, and the dashboard consumes that contract through a `service -> hook -> UI` flow with a new read-only Database Studio page.
**Tech Stack:** PostgreSQL 15, Express, TypeScript, Zod, React, React Query, CodeMirror, Vitest
---
## File Map
| Path | Responsibility |
| --- | --- |
| `backend/src/infra/database/migrations/032_create-custom-migrations.sql` | Create `system.custom_migrations` with the minimal applied-history shape |
| `backend/tests/unit/custom-migrations-table-migration.test.ts` | Validate SQL migration structure and idempotency guards |
| `backend/src/services/database/database-migration.service.ts` | List migrations and run new migrations transactionally |
| `backend/tests/unit/database-migration.service.test.ts` | Service-level validation and transactional behavior |
| `backend/src/api/routes/database/migrations.routes.ts` | Admin routes for list/create |
| `backend/src/api/routes/database/index.routes.ts` | Mount new migrations router |
| `backend/src/utils/sql-parser.ts` | Reuse or extend helpers for schema restrictions and socket invalidation |
| `packages/shared-schemas/src/database.schema.ts` | `migrationSchema` domain shape |
| `packages/shared-schemas/src/database-api.schema.ts` | Request/response schemas for migrations API |
| `packages/shared-schemas/src/index.ts` | Export shared migration contracts |
| `packages/dashboard/src/features/database/services/migration.service.ts` | Dashboard API client for migrations |
| `packages/dashboard/src/features/database/hooks/useMigrations.ts` | React Query hooks for list/create |
| `packages/dashboard/src/features/database/components/DatabaseSidebar.tsx` | Add `Migrations` to the Database Studio sidebar |
| `packages/dashboard/src/features/database/components/MigrationFormDialog.tsx` | Run-migration dialog with SQL editor |
| `packages/dashboard/src/features/database/pages/MigrationsPage.tsx` | Read-only migration history page |
| `packages/dashboard/src/router/AppRoutes.tsx` | Register `/dashboard/database/migrations` |
### Task 1: Create the Applied-Migrations Table
**Files:**
- Create: `backend/src/infra/database/migrations/032_create-custom-migrations.sql`
- Test: `backend/tests/unit/custom-migrations-table-migration.test.ts`
- [ ] **Step 1: Write the failing migration SQL test**
```ts
import { describe, it, expect } from 'vitest';
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
const currentDir = path.dirname(fileURLToPath(import.meta.url));
const migrationPath = path.resolve(
currentDir,
'../../src/infra/database/migrations/032_create-custom-migrations.sql'
);
describe('032_create-custom-migrations migration', () => {
const sql = fs.readFileSync(migrationPath, 'utf8');
it('creates system.custom_migrations with the expected columns', () => {
expect(sql).toMatch(/CREATE TABLE IF NOT EXISTS system\.custom_migrations/i);
expect(sql).toMatch(/sequence_number INTEGER PRIMARY KEY/i);
expect(sql).toMatch(/name TEXT NOT NULL UNIQUE/i);
expect(sql).toMatch(/statements TEXT\[\] NOT NULL/i);
expect(sql).toMatch(/created_at TIMESTAMPTZ NOT NULL DEFAULT now\(\)/i);
});
it('does not expose a surrogate id column', () => {
expect(sql).not.toMatch(/\bid\b/i);
});
});
```
- [ ] **Step 2: Run the migration SQL test to verify it fails**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- custom-migrations-table-migration.test.ts`
Expected: FAIL with a missing file error for `032_create-custom-migrations.sql`
- [ ] **Step 3: Write the migration SQL file**
```sql
CREATE SCHEMA IF NOT EXISTS system;
CREATE TABLE IF NOT EXISTS system.custom_migrations (
sequence_number INTEGER PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
statements TEXT[] NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
```
- [ ] **Step 4: Add idempotent guards for re-runs**
```sql
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1
FROM information_schema.tables
WHERE table_schema = 'system'
AND table_name = 'custom_migrations'
) THEN
CREATE TABLE system.custom_migrations (
sequence_number INTEGER PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
statements TEXT[] NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
END IF;
END $$;
```
- [ ] **Step 5: Re-run the migration SQL test**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- custom-migrations-table-migration.test.ts`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add \
backend/src/infra/database/migrations/032_create-custom-migrations.sql \
backend/tests/unit/custom-migrations-table-migration.test.ts
git commit -m "feat: add custom migrations history table"
```
### Task 2: Implement Backend Migration Service
**Files:**
- Create: `backend/src/services/database/database-migration.service.ts`
- Test: `backend/tests/unit/database-migration.service.test.ts`
- Modify: `backend/src/utils/sql-parser.ts`
- [ ] **Step 1: Write the failing service tests**
```ts
describe('DatabaseMigrationService.createMigration', () => {
it('rejects statements that reference a non-public schema', async () => {
await expect(
service.createMigration({
name: 'break-auth',
sql: 'CREATE TABLE auth.test_users (id uuid);',
actor: 'local:admin',
})
).rejects.toThrow(/public schema/i);
});
it('inserts history only after SQL succeeds', async () => {
mockClient.query
.mockResolvedValueOnce(undefined) // BEGIN
.mockResolvedValueOnce({ rows: [{ next_sequence_number: 1 }] })
.mockResolvedValueOnce(undefined) // statement
.mockResolvedValueOnce(undefined) // insert history
.mockResolvedValueOnce(undefined); // COMMIT
await service.createMigration({
name: 'create-posts',
sql: 'CREATE TABLE posts (id uuid primary key);',
actor: 'local:admin',
});
expect(mockClient.query).toHaveBeenCalledWith(
expect.stringMatching(/INSERT INTO system\.custom_migrations/i),
expect.any(Array)
);
});
});
```
- [ ] **Step 2: Run the service tests to verify they fail**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration.service.test.ts`
Expected: FAIL because `database-migration.service.ts` does not exist yet
- [ ] **Step 3: Create the service skeleton**
```ts
export class DatabaseMigrationService {
private static instance: DatabaseMigrationService;
private dbManager = DatabaseManager.getInstance();
public static getInstance(): DatabaseMigrationService {
if (!DatabaseMigrationService.instance) {
DatabaseMigrationService.instance = new DatabaseMigrationService();
}
return DatabaseMigrationService.instance;
}
}
```
- [ ] **Step 4: Implement list and create methods**
```ts
async listMigrations(): Promise<DatabaseMigrationsResponse> {
const result = await this.dbManager.getPool().query(`
SELECT
sequence_number AS "sequenceNumber",
name,
statements,
created_at AS "createdAt"
FROM system.custom_migrations
ORDER BY sequence_number DESC
`);
return { migrations: result.rows };
}
async createMigration(input: CreateMigrationRequest & { actor: string }): Promise<CreateMigrationResponse> {
const statements = parseSQLStatements(input.sql);
this.assertPublicSchemaOnly(statements);
return this.runMigrationTransaction(input.name, statements);
}
```
- [ ] **Step 5: Enforce the public-schema-only rules**
```ts
private assertPublicSchemaOnly(statements: string[]): void {
for (const statement of statements) {
const parsed = statement.toLowerCase();
if (/\b(auth|system|storage|ai|functions|realtime|schedules|pg_catalog|information_schema)\b/.test(parsed)) {
throw new AppError(
'Custom migrations may only target the public schema.',
400,
ERROR_CODES.DATABASE_FORBIDDEN
);
}
if (/\b(set\s+search_path|begin\b|commit\b|rollback\b|create\s+schema|drop\s+schema)\b/.test(parsed)) {
throw new AppError(
'Custom migrations cannot modify schema routing or manage their own transactions.',
400,
ERROR_CODES.DATABASE_FORBIDDEN
);
}
}
}
```
- [ ] **Step 6: Execute the migration transactionally and insert history on success**
```ts
await client.query('BEGIN');
await client.query("SELECT pg_advisory_xact_lock(hashtext('system.custom_migrations'))");
await client.query('SET LOCAL search_path TO public');
const sequenceResult = await client.query(`
SELECT COALESCE(MAX(sequence_number), 0) + 1 AS next_sequence_number
FROM system.custom_migrations
`);
for (const statement of statements) {
await client.query(statement);
}
await client.query(
`
INSERT INTO system.custom_migrations (sequence_number, name, statements)
VALUES ($1, $2, $3)
`,
[sequenceNumber, name, statements]
);
await client.query(`NOTIFY pgrst, 'reload schema';`);
await client.query('COMMIT');
```
- [ ] **Step 7: Extend the SQL change typing so migration creates invalidate the new page**
```ts
export interface DatabaseResourceUpdate {
type: 'tables' | 'table' | 'records' | 'index' | 'trigger' | 'policy' | 'function' | 'extension' | 'migration';
name?: string;
}
```
- [ ] **Step 8: Re-run the service tests**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration.service.test.ts`
Expected: PASS
- [ ] **Step 9: Commit**
```bash
git add \
backend/src/services/database/database-migration.service.ts \
backend/src/utils/sql-parser.ts \
backend/tests/unit/database-migration.service.test.ts
git commit -m "feat: add custom migration backend service"
```
### Task 3: Expose Admin Routes for Migrations
**Files:**
- Create: `backend/src/api/routes/database/migrations.routes.ts`
- Modify: `backend/src/api/routes/database/index.routes.ts`
- [ ] **Step 1: Write the failing route-level behavior test or request fixture**
```ts
it('mounts GET /api/database/migrations', async () => {
const response = await request(app)
.get('/api/database/migrations')
.set('Authorization', `Bearer ${adminToken}`);
expect(response.status).toBe(200);
});
```
- [ ] **Step 2: Create the migrations router**
```ts
const router = Router();
const migrationService = DatabaseMigrationService.getInstance();
router.get('/', verifyAdmin, async (_req, res, next) => {
try {
const response = await migrationService.listMigrations();
successResponse(res, response);
} catch (error) {
next(error);
}
});
```
- [ ] **Step 3: Add the create-and-run endpoint**
```ts
router.post('/', verifyAdmin, async (req: AuthRequest, res, next) => {
try {
const validation = createMigrationRequestSchema.safeParse(req.body);
if (!validation.success) {
throw new AppError('Invalid migration payload', 400, ERROR_CODES.INVALID_INPUT);
}
const result = await migrationService.createMigration({
...validation.data,
actor: req.user?.email || 'api-key',
});
await auditService.log({
actor: req.user?.email || 'api-key',
action: 'CREATE_CUSTOM_MIGRATION',
module: 'DATABASE',
details: { name: validation.data.name, statementCount: result.statements.length },
ip_address: req.ip,
});
successResponse(res, result, 201);
} catch (error) {
next(error);
}
});
```
- [ ] **Step 4: Mount the router in the database route index**
```ts
import { databaseMigrationsRouter } from './migrations.routes.js';
router.use('/migrations', databaseMigrationsRouter);
```
- [ ] **Step 5: Run the focused backend tests**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test -- database-migration`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add \
backend/src/api/routes/database/migrations.routes.ts \
backend/src/api/routes/database/index.routes.ts
git commit -m "feat: expose custom migration admin routes"
```
### Task 4: Add Shared Migration Contracts
**Files:**
- Modify: `packages/shared-schemas/src/database.schema.ts`
- Modify: `packages/shared-schemas/src/database-api.schema.ts`
- Modify: `packages/shared-schemas/src/index.ts`
- [ ] **Step 1: Add the migration domain schema**
```ts
export const migrationSchema = z.object({
sequenceNumber: z.number().int().positive(),
name: z.string().min(1),
statements: z.array(z.string()).min(1),
createdAt: z.string(),
});
```
- [ ] **Step 2: Add request/response schemas**
```ts
export const createMigrationRequestSchema = z.object({
name: z.string().min(1, 'Migration name is required'),
sql: z.string().min(1, 'Migration SQL is required'),
});
export const createMigrationResponseSchema = migrationSchema.extend({
message: z.string(),
});
export const databaseMigrationsResponseSchema = z.object({
migrations: z.array(migrationSchema),
});
```
- [ ] **Step 3: Export the new contracts**
```ts
export type Migration = z.infer<typeof migrationSchema>;
export type CreateMigrationRequest = z.infer<typeof createMigrationRequestSchema>;
export type CreateMigrationResponse = z.infer<typeof createMigrationResponseSchema>;
export type DatabaseMigrationsResponse = z.infer<typeof databaseMigrationsResponseSchema>;
```
- [ ] **Step 4: Build shared schemas**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas && npm run build`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add \
packages/shared-schemas/src/database.schema.ts \
packages/shared-schemas/src/database-api.schema.ts \
packages/shared-schemas/src/index.ts
git commit -m "feat: add shared migration api contracts"
```
### Task 5: Add Dashboard Service, Hook, Route, and Sidebar Entry
**Files:**
- Create: `packages/dashboard/src/features/database/services/migration.service.ts`
- Create: `packages/dashboard/src/features/database/hooks/useMigrations.ts`
- Modify: `packages/dashboard/src/features/database/components/DatabaseSidebar.tsx`
- Modify: `packages/dashboard/src/router/AppRoutes.tsx`
- Modify: `packages/dashboard/src/lib/contexts/SocketContext.tsx`
- [ ] **Step 1: Create the dashboard migration service**
```ts
export class MigrationService {
async listMigrations(): Promise<DatabaseMigrationsResponse> {
return apiClient.request('/database/migrations', {
method: 'GET',
headers: apiClient.withAccessToken({}),
});
}
async createMigration(body: CreateMigrationRequest): Promise<CreateMigrationResponse> {
return apiClient.request('/database/migrations', {
method: 'POST',
headers: apiClient.withAccessToken({ 'Content-Type': 'application/json' }),
body: JSON.stringify(body),
});
}
}
```
- [ ] **Step 2: Add the React Query hook**
```ts
export function useMigrations(enabled = false) {
const queryClient = useQueryClient();
const { showToast } = useToast();
const query = useQuery({
queryKey: ['database', 'migrations'],
queryFn: () => migrationService.listMigrations(),
enabled,
});
const createMutation = useMutation({
mutationFn: (body: CreateMigrationRequest) => migrationService.createMigration(body),
onSuccess: async () => {
await queryClient.invalidateQueries({ queryKey: ['database', 'migrations'] });
await queryClient.invalidateQueries({ queryKey: ['tables'] });
showToast('Migration executed successfully', 'success');
},
});
return { ...query, createMigration: createMutation.mutateAsync, isCreating: createMutation.isPending };
}
```
- [ ] **Step 3: Register the route and sidebar item**
```tsx
{
id: 'migrations',
label: 'Migrations',
href: '/dashboard/database/migrations',
},
```
```tsx
<Route path="migrations" element={<MigrationsPage />} />
```
- [ ] **Step 4: Invalidate the migrations query on socket data updates**
```ts
case 'migration':
void queryClient.invalidateQueries({ queryKey: ['database', 'migrations'] });
break;
```
- [ ] **Step 5: Run dashboard typecheck**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run typecheck`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add \
packages/dashboard/src/features/database/services/migration.service.ts \
packages/dashboard/src/features/database/hooks/useMigrations.ts \
packages/dashboard/src/features/database/components/DatabaseSidebar.tsx \
packages/dashboard/src/router/AppRoutes.tsx \
packages/dashboard/src/lib/contexts/SocketContext.tsx
git commit -m "feat: wire dashboard migration data flow"
```
### Task 6: Build the Migrations Page and Run Dialog
**Files:**
- Create: `packages/dashboard/src/features/database/pages/MigrationsPage.tsx`
- Create: `packages/dashboard/src/features/database/components/MigrationFormDialog.tsx`
- Reuse: `packages/dashboard/src/features/database/components/SQLModal.tsx`
- Reuse: `packages/dashboard/src/components/CodeEditor.tsx`
- [ ] **Step 1: Create the migration run dialog**
```tsx
export function MigrationFormDialog({ open, onOpenChange, onSubmit, isSubmitting }: Props) {
const [name, setName] = useState('');
const [sql, setSql] = useState('');
return (
<Dialog open={open} onOpenChange={onOpenChange}>
<DialogContent className="max-w-3xl">
<DialogHeader>
<DialogTitle>Run Migration</DialogTitle>
<DialogDescription>
This runs immediately against the public schema and is recorded only if it succeeds.
</DialogDescription>
</DialogHeader>
<Input value={name} onChange={(e) => setName(e.target.value)} placeholder="add_posts_table" />
<div className="h-80 rounded-lg border border-border">
<CodeEditor value={sql} onChange={setSql} editable language="sql" />
</div>
<Button onClick={() => void onSubmit({ name, sql })} disabled={isSubmitting}>
Run Migration
</Button>
</DialogContent>
</Dialog>
);
}
```
- [ ] **Step 2: Create the history page**
```tsx
export default function MigrationsPage() {
const navigate = useNavigate();
const { data, isLoading, error, refetch, createMigration, isCreating } = useMigrations(true);
const [open, setOpen] = useState(false);
const [sqlModal, setSqlModal] = useState({ open: false, title: '', value: '' });
return (
<div className="flex h-full min-h-0 overflow-hidden bg-[rgb(var(--semantic-1))]">
<DatabaseStudioSidebarPanel onBack={() => void navigate('/dashboard/database/tables', { state: { slideFromStudio: true } })} />
<div className="min-w-0 flex-1 flex flex-col overflow-hidden bg-[rgb(var(--semantic-1))]">
<TableHeader title="Database Migrations" showDividerAfterTitle />
<DataGrid data={rows} columns={columns} showSelection={false} showPagination={false} noPadding className="h-full" />
</div>
</div>
);
}
```
- [ ] **Step 3: Make the SQL column read-only through the existing SQL modal**
```tsx
renderCell: ({ row }) => (
<SQLCellButton
value={row.statements.join('\n\n')}
onClick={() =>
setSqlModal({
open: true,
title: `Migration ${row.sequenceNumber}: ${row.name}`,
value: row.statements.join('\n\n'),
})
}
/>
)
```
- [ ] **Step 4: Handle loading, empty, and error states using the existing database page patterns**
```tsx
if (error) {
return <EmptyState title="Failed to load migrations" description={error.message} />;
}
```
- [ ] **Step 5: Build the dashboard package**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run build`
Expected: PASS
- [ ] **Step 6: Commit**
```bash
git add \
packages/dashboard/src/features/database/pages/MigrationsPage.tsx \
packages/dashboard/src/features/database/components/MigrationFormDialog.tsx
git commit -m "feat: add database migrations studio page"
```
### Task 7: Validate the Full Feature
**Files:**
- Validate: `backend/**`
- Validate: `packages/shared-schemas/**`
- Validate: `packages/dashboard/**`
- [ ] **Step 1: Run backend tests**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm test`
Expected: PASS
- [ ] **Step 2: Run backend build**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/backend && npm run build`
Expected: PASS
- [ ] **Step 3: Run shared schema build**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/shared-schemas && npm run build`
Expected: PASS
- [ ] **Step 4: Run dashboard typecheck and build**
Run: `cd /Users/lyu/Documents/GitHub/InsForge/packages/dashboard && npm run typecheck && npm run build`
Expected: PASS
- [ ] **Step 5: Smoke-test the feature manually**
```bash
curl -X POST http://localhost:7130/api/database/migrations \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <admin-token>" \
-d '{"name":"create_posts","sql":"CREATE TABLE posts (id UUID PRIMARY KEY DEFAULT gen_random_uuid());"}'
```
Expected: `201` with the new migration row, and `GET /api/database/migrations` returns it.
- [ ] **Step 6: Commit final validation fixes**
```bash
git add backend packages/shared-schemas packages/dashboard
git commit -m "feat: add custom database migrations"
```
.internal/docs/plans/2026-04-18-compute-cloud-provider.md
+1807
View File
File diff suppressed because it is too large Load Diff
.internal/docs/plans/2026-04-21-dtest-onboarding.md
+1170
View File
File diff suppressed because it is too large Load Diff
.internal/docs/plans/2026-04-22-s3-compatible-storage-gateway.md
+4117
View File
File diff suppressed because it is too large Load Diff
.internal/docs/plans/2026-04-29-backend-branching-oss.md
+410
View File
@@ -0,0 +1,410 @@
# Backend Branching (OSS) Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Steps use checkbox (`- [ ]`) syntax.
**Goal:** Add read-only fallback from a branch project's storage to its parent project's S3 directory, gated by a new `PARENT_APP_KEY` env var. Read-only, transparent, no HTTP API changes.
**Architecture:** Extend `S3StorageProvider` with an optional `parentAppKey`. Read methods (`getObject`, `headObject`, `getObjectStream`, `getDownloadStrategy`) attempt the branch's S3 path first; on 404, retry the parent's path. Writes always target the branch path. Service layer and HTTP routes unchanged.
**Tech Stack:** TypeScript, Node.js, Express, AWS SDK v3 (`@aws-sdk/client-s3`), Vitest.
**Spec:** [docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md](../specs/2026-04-29-backend-branching-oss-design.md)
---
## File Structure
**Modify:**
- `backend/src/providers/storage/s3.provider.ts` — accept `parentAppKey`, add fallback in read methods
- `backend/src/services/storage/storage.service.ts` — pass `PARENT_APP_KEY` env var into provider constructor
- `backend/src/providers/storage/base.provider.ts` (interface, if any) — no new methods, just a fallback-aware contract
**Create:**
- `backend/tests/unit/storage-s3-fallback.test.ts` — fallback behavior unit tests
---
## Task 1: Extend S3StorageProvider Constructor
**Files:**
- Modify: `backend/src/providers/storage/s3.provider.ts:49-56`
- [ ] **Step 1: Accept `parentAppKey` in constructor**
```typescript
export class S3StorageProvider implements StorageProvider {
private s3Client: S3Client | null = null;
constructor(
private s3Bucket: string,
private appKey: string,
private region: string = 'us-east-2',
private parentAppKey?: string,
) {
// ... existing init ...
}
private getS3Key(bucket: string, key: string): string {
return `${this.appKey}/${bucket}/${key}`;
}
private getParentS3Key(bucket: string, key: string): string | null {
return this.parentAppKey ? `${this.parentAppKey}/${bucket}/${key}` : null;
}
}
```
- [ ] **Step 2: Run typecheck**
Run: `cd backend && npx tsc --noEmit`
Expected: no errors.
- [ ] **Step 3: Commit**
```bash
git add backend/src/providers/storage/s3.provider.ts
git commit -m "feat(branching): S3StorageProvider accepts optional parentAppKey"
```
---
## Task 2: Fallback in Read Methods (TDD)
**Files:**
- Create: `backend/tests/unit/storage-s3-fallback.test.ts`
- Modify: `backend/src/providers/storage/s3.provider.ts` (`getObject`, `headObject`, `getObjectStream`)
- [ ] **Step 1: Write failing tests**
```typescript
// backend/tests/unit/storage-s3-fallback.test.ts
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { S3StorageProvider } from '@/providers/storage/s3.provider.js';
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
describe('S3StorageProvider fallback to parent', () => {
let sendMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
sendMock = vi.fn();
vi.spyOn(S3Client.prototype, 'send').mockImplementation(sendMock as any);
});
it('returns branch object when present (no fallback call)', async () => {
sendMock.mockResolvedValueOnce({ Body: streamOf('hello') });
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
(p as any).s3Client = new S3Client({}); // bypass async init for unit
const out = await p.getObject('foo', 'a.txt');
expect(out?.toString()).toBe('hello');
expect(sendMock).toHaveBeenCalledTimes(1);
const cmd: any = sendMock.mock.calls[0][0];
expect(cmd.input.Key).toBe('branchkey/foo/a.txt');
});
it('falls back to parent when branch returns NoSuchKey', async () => {
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
sendMock.mockRejectedValueOnce(noSuchKey).mockResolvedValueOnce({ Body: streamOf('parent-data') });
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
(p as any).s3Client = new S3Client({});
const out = await p.getObject('foo', 'a.txt');
expect(out?.toString()).toBe('parent-data');
const k1 = (sendMock.mock.calls[0][0] as any).input.Key;
const k2 = (sendMock.mock.calls[1][0] as any).input.Key;
expect(k1).toBe('branchkey/foo/a.txt');
expect(k2).toBe('parentkey/foo/a.txt');
});
it('returns null when both branch and parent are missing', async () => {
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
sendMock.mockRejectedValueOnce(noSuchKey).mockRejectedValueOnce(noSuchKey);
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
(p as any).s3Client = new S3Client({});
const out = await p.getObject('foo', 'a.txt');
expect(out).toBeNull();
});
it('does NOT fall back when parentAppKey is unset', async () => {
const noSuchKey = Object.assign(new Error('NoSuchKey'), { name: 'NoSuchKey' });
sendMock.mockRejectedValueOnce(noSuchKey);
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2'); // no parentAppKey
(p as any).s3Client = new S3Client({});
const out = await p.getObject('foo', 'a.txt');
expect(out).toBeNull();
expect(sendMock).toHaveBeenCalledTimes(1);
});
});
function streamOf(s: string) {
const { Readable } = require('node:stream');
return Readable.from([Buffer.from(s)]);
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
Expected: FAIL — fallback not yet implemented.
- [ ] **Step 3: Implement fallback in `getObject`**
```typescript
async getObject(bucket: string, key: string): Promise<Buffer | null> {
if (!this.s3Client) throw new Error('S3 client not initialized');
const primary = await this.tryGet(this.getS3Key(bucket, key));
if (primary !== null) return primary;
const parentKey = this.getParentS3Key(bucket, key);
if (!parentKey) return null;
return this.tryGet(parentKey);
}
private async tryGet(s3Key: string): Promise<Buffer | null> {
try {
const cmd = new GetObjectCommand({ Bucket: this.s3Bucket, Key: s3Key });
const res = await this.s3Client!.send(cmd);
if (!res.Body) return null;
const chunks: Buffer[] = [];
// @ts-ignore SDK v3 stream
for await (const c of res.Body) chunks.push(Buffer.isBuffer(c) ? c : Buffer.from(c));
return Buffer.concat(chunks);
} catch (err: any) {
if (err?.name === 'NoSuchKey' || err?.$metadata?.httpStatusCode === 404) return null;
throw err;
}
}
```
- [ ] **Step 4: Implement fallback in `headObject` and `getObjectStream`**
Apply the same primary-then-parent pattern:
```typescript
async headObject(bucket: string, key: string): Promise<{ size: number; mimeType: string } | null> {
const tryHead = async (s3Key: string) => {
try {
const res = await this.s3Client!.send(new HeadObjectCommand({ Bucket: this.s3Bucket, Key: s3Key }));
return { size: res.ContentLength ?? 0, mimeType: res.ContentType ?? 'application/octet-stream' };
} catch (err: any) {
if (err?.name === 'NotFound' || err?.$metadata?.httpStatusCode === 404) return null;
throw err;
}
};
const primary = await tryHead(this.getS3Key(bucket, key));
if (primary) return primary;
const parentKey = this.getParentS3Key(bucket, key);
return parentKey ? tryHead(parentKey) : null;
}
```
Mirror the pattern in `getObjectStream` if present.
- [ ] **Step 5: Run tests**
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
Expected: all four cases pass.
- [ ] **Step 6: Commit**
```bash
git add backend/src/providers/storage/s3.provider.ts backend/tests/unit/storage-s3-fallback.test.ts
git commit -m "feat(branching): read-only S3 fallback to parent appkey on 404"
```
---
## Task 3: Fallback in Presigned URL Generation (`getDownloadStrategy`)
**Files:**
- Modify: `backend/src/providers/storage/s3.provider.ts` (`getDownloadStrategy` method)
- [ ] **Step 1: Write failing test**
Add to `backend/tests/unit/storage-s3-fallback.test.ts`:
```typescript
it('presigned URL: signs branch key when present', async () => {
// Mock HeadObject to succeed for branch path
sendMock.mockResolvedValueOnce({ ContentLength: 5 });
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
(p as unknown as { s3Client: S3Client }).s3Client = new S3Client({
region: 'us-east-2',
credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
});
const strategy = await p.getDownloadStrategy('foo', 'a.txt');
expect(strategy.url).toContain('branchkey/foo/a.txt');
});
it('presigned URL: signs parent key when branch HEAD returns 404', async () => {
const notFound = Object.assign(new Error('NotFound'), { name: 'NotFound' });
sendMock.mockRejectedValueOnce(notFound);
const p = new S3StorageProvider('bucket', 'branchkey', 'us-east-2', 'parentkey');
(p as unknown as { s3Client: S3Client }).s3Client = new S3Client({
region: 'us-east-2',
credentials: { accessKeyId: 'test', secretAccessKey: 'test' },
});
const strategy = await p.getDownloadStrategy('foo', 'a.txt');
expect(strategy.url).toContain('parentkey/foo/a.txt');
});
```
- [ ] **Step 2: Run test to verify it fails**
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
Expected: FAIL.
- [ ] **Step 3: Implement fallback in `getDownloadStrategy`**
`getDownloadStrategy` already constructs a presigned URL via `getSignedUrl`. To support branching, do a HEAD on the branch path first; if missing (and a parent is configured) sign the parent path. HEAD failures other than 404 are caught and logged — URL generation defaults to the branch key rather than aborting.
```typescript
const branchKey = this.getS3Key(bucket, key);
const parentKey = this.getParentS3Key(bucket, key);
let s3Key = branchKey;
if (parentKey) {
try {
const branchExists = await this.tryHeadObject(branchKey);
if (!branchExists) {
s3Key = parentKey;
}
} catch (headErr) {
// HEAD failures shouldn't break URL generation. Default to the branch
// key; if the object truly only lives on the parent, the signed URL
// will 404 at download time — degraded but recoverable.
logger.warn('Branch HEAD check failed in getDownloadStrategy; signing branch key', {
bucket, key,
error: headErr instanceof Error ? headErr.message : String(headErr),
});
}
}
// ...existing CloudFront / getSignedUrl code uses s3Key.
```
- [ ] **Step 4: Run tests**
Run: `cd backend && npx vitest run tests/unit/storage-s3-fallback.test.ts`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add backend/src/providers/storage/s3.provider.ts backend/tests/unit/storage-s3-fallback.test.ts
git commit -m "feat(branching): presigned URL fallback to parent path"
```
---
## Task 4: Wire `PARENT_APP_KEY` Env Var Through StorageService
**Files:**
- Modify: `backend/src/services/storage/storage.service.ts:29-45`
- [ ] **Step 1: Pass `PARENT_APP_KEY` to S3StorageProvider**
```typescript
private constructor() {
const s3Bucket = process.env.AWS_S3_BUCKET;
const appKey = process.env.APP_KEY || 'local';
const parentAppKey = process.env.PARENT_APP_KEY; // <-- new
if (s3Bucket) {
this.provider = new S3StorageProvider(
s3Bucket,
appKey,
process.env.AWS_REGION || 'us-east-2',
parentAppKey,
);
if (parentAppKey) {
logger.info('Storage initialized in branch mode', { appKey, parentAppKey });
}
} else {
// local storage — fallback unsupported in v1, ignore
const baseDir = process.env.STORAGE_DIR || path.resolve(process.cwd(), 'insforge-storage');
this.provider = new LocalStorageProvider(baseDir);
}
}
```
- [ ] **Step 2: Verify boot log**
Run:
```bash
cd backend
APP_KEY=branchkey PARENT_APP_KEY=parentkey AWS_S3_BUCKET=test-bucket npm run dev 2>&1 | head -30
```
Expected: log line `Storage initialized in branch mode { appKey: 'branchkey', parentAppKey: 'parentkey' }`. Stop with Ctrl-C.
- [ ] **Step 3: Commit**
```bash
git add backend/src/services/storage/storage.service.ts
git commit -m "feat(branching): wire PARENT_APP_KEY through to S3 provider"
```
---
## Task 5: Manual Smoke Test
- [ ] **Step 1: Two-namespace smoke against real S3**
Pre-requisites: AWS creds + a test bucket with two prefixes:
- `parentkey/photos/cat.jpg` (some real file)
- `branchkey/photos/dog.jpg` (some other real file)
```bash
# Branch mode boot
APP_KEY=branchkey PARENT_APP_KEY=parentkey AWS_S3_BUCKET=insforge-storage-test \
AWS_REGION=us-east-2 npm run dev:backend
```
Then in another terminal (after seeding `storage.objects` rows so RLS allows access):
```bash
# Should hit branch (own file)
curl -i http://localhost:7130/api/storage/buckets/photos/objects/dog.jpg
# Should hit parent (fallback)
curl -i http://localhost:7130/api/storage/buckets/photos/objects/cat.jpg
# Write goes to branch only
curl -i -X PUT http://localhost:7130/api/storage/buckets/photos/objects/new.txt \
-H "Authorization: Bearer $TOKEN" --data-binary "branch-write"
# Verify on S3 directly
aws s3 ls s3://insforge-storage-test/branchkey/photos/ # contains new.txt
aws s3 ls s3://insforge-storage-test/parentkey/photos/ # unchanged
```
- [ ] **Step 2: Open PR**
```bash
git push -u origin feat/branching
gh pr create --title "feat: storage fallback to parent for branch projects" --body "$(cat <<'EOF'
## Summary
- Adds optional `PARENT_APP_KEY` env var that triggers read-only fallback in S3 storage provider.
- All read methods (getObject, headObject, getObjectStream, getDownloadStrategy) try branch path first, fall back to parent path on 404.
- Writes are unchanged — they always target the branch's own appkey path.
- Local storage provider is unaffected (no fallback).
## Spec
- [docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md](docs/superpowers/specs/2026-04-29-backend-branching-oss-design.md)
## Test plan
- [ ] Vitest suite green
- [ ] Manual: GET on branch object hits branch
- [ ] Manual: GET on parent-only object falls back to parent
- [ ] Manual: GET on missing object returns 404
- [ ] Manual: PUT on branch creates only at branch path
- [ ] Manual: existing non-branch projects (no PARENT_APP_KEY) are unaffected
🤖 Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"
```
---
## Self-Review Notes
- **Fallback is read-only**: writes (PUT/DELETE/multipart) always go to branch path. Verified by not modifying any write methods.
- **Local provider unchanged**: only S3 provider gets the fallback. Local installs aren't branched.
- **No HTTP API changes**: routes and middleware unchanged. RLS / bucket-visibility checks remain authoritative.
- **No new IAM permissions**: existing role can read all prefixes in `AWS_S3_BUCKET`.
- **Schema-only mode**: branches with `mode='schema-only'` truncate `storage.objects` in the DB → RLS lookup returns 404 before reaching the provider → fallback never runs. Documented in spec as expected.
.internal/docs/plans/2026-04-30-stripe-payments-implementation.md
+231
View File
@@ -0,0 +1,231 @@
# Stripe Payments Current Implementation Spec
**Status:** Current source-of-truth implementation for the Stripe payments foundation.
**Audience:** InsForge engineers, dashboard maintainers, CLI/SDK authors, OpenAPI maintainers, and agents that need to understand how to build on the payment feature.
**Goal:** Let a developer configure their own Stripe test/live secret keys, let agents configure Stripe catalog objects, and let InsForge support complete runtime payment flows for one-time purchases, subscriptions, webhook projections, and customer billing portal sessions.
## Product Direction
InsForge uses the developer-owned Stripe account model for this phase. Developers configure `STRIPE_TEST_SECRET_KEY` and/or `STRIPE_LIVE_SECRET_KEY` through the dashboard, or seed them from environment variables into the secret store. InsForge does not use Connected Accounts, claimable sandboxes, or test-to-live publishing in this version.
Stripe is the source of truth. InsForge stores a local mirror and runtime projection so agents and the dashboard can reason about the state of payments without re-querying Stripe for every action. Mutations still go to Stripe first; InsForge only updates its mirror after Stripe succeeds.
The system is agent-first. The dashboard provides visibility and controls, but the backend API and shared schemas are the primary surface that agents, CLI commands, SDKs, and generated apps should build against.
## Current Capabilities
This implementation supports test and live Stripe environments as independent targets. Every payment table includes `environment = 'test' | 'live'` instead of using duplicated live/test table sets.
Developers and agents can manage Stripe products and prices in either environment. Product and price create calls use caller-stable idempotency keys when provided. Updates and archives are applied directly to Stripe, then mirrored locally. Product deletion hard-deletes the local mirror only after Stripe confirms the product was deleted.
Developers can run a unified sync. Sync pulls products, prices, customers, and subscriptions from every configured environment, skips unconfigured environments, and records the latest sync status on the environment connection row. Manual sync also checks the Stripe account id before writing, so switching a key to a different account clears stale mirrored payment data before importing the new account's data.
Generated apps can create Checkout Sessions at runtime. Anonymous one-time checkout is allowed. Identified one-time checkout is allowed and can reuse an existing Stripe customer mapping. Subscription checkout requires a billing subject because subscriptions represent ongoing entitlement.
Generated apps can create Stripe Billing Portal Sessions for authenticated users. This is intentionally mediated through `payments.customer_portal_sessions`, where developers or agents can define custom RLS policies for their app's subject model.
Managed Stripe webhooks are used to keep runtime projections current. InsForge stores webhook signing secrets in the secret store, not in environment variables. Webhook setup is best-effort during new account configuration and can also be retried from the dashboard Webhooks tab.
## Data Model
The payments schema is created by `backend/src/infra/database/migrations/038_create-payments-schema.sql`.
`payments.stripe_connections` stores one row per environment. It tracks Stripe account identity, key/configured status, managed webhook endpoint metadata, latest sync status, sync errors, and sync counts.
`payments.products` and `payments.prices` mirror Stripe catalog state. Sync overwrites local drift with Stripe data and removes local rows that no longer exist in Stripe for that environment.
`payments.checkout_sessions` stores short-lived checkout attempts. It starts as `initialized`, moves to `open` after Stripe session creation, and is updated to `completed`, `expired`, or `failed` through webhooks or backend errors. It intentionally does not enable RLS by default to reduce DX friction. Developers can ask agents to add RLS later if their app needs stricter checkout-attempt policies.
`payments.customer_portal_sessions` stores customer portal creation attempts. It enables RLS by default because portal creation must be gated by the application's subject ownership model.
`payments.stripe_customer_mappings` maps arbitrary app billing subjects to Stripe customers. The subject is intentionally generic: `subject_type` and `subject_id` can represent a user, team, organization, tenant, group, workspace, or another app-specific billing owner.
`payments.customers` mirrors Stripe customer rows for admin visibility and debugging. It is intentionally read-only from the app's perspective and does not replace `stripe_customer_mappings` as the operational subject-to-customer bridge.
`payments.subscriptions` and `payments.subscription_items` mirror current subscription state. Sync and webhooks fetch full subscription item lists when Stripe pagination requires it, then delete local subscription items not present in Stripe.
`payments.payment_history` records one-time payments, subscription invoices, failed payments, refunds, and refund state on original payments. It is webhook-driven and designed to tolerate out-of-order Stripe events.
`payments.webhook_events` records Stripe webhook processing state. It deduplicates events by `(environment, stripe_event_id)`, retries failed or pending events, and ignores already processed events.
## Backend Surface
Runtime routes use `verifyUser` so generated apps can call them with InsForge user tokens, including anon tokens for anonymous one-time checkout.
- `POST /api/payments/:environment/checkout-sessions`: creates a local checkout attempt, then creates a Stripe Checkout Session. Subscription mode requires `subject`.
- `POST /api/payments/:environment/customer-portal-sessions`: creates a local portal attempt under the caller context, checks `stripe_customer_mappings`, then creates a Stripe Billing Portal Session. Anonymous users are rejected.
- `POST /api/webhooks/stripe/:environment`: receives Stripe webhooks with a raw body and verifies the Stripe signature.
Admin routes use `verifyAdmin` and are intended for dashboard, agents, CLI, and SDK admin surfaces.
- `GET /api/payments/status`: returns environment connection, sync, and webhook status.
- `GET /api/payments/config`: returns Stripe key availability and masked key info.
- `PUT /api/payments/:environment/config`: stores a Stripe secret key in the secret store. New or different accounts trigger managed webhook setup and unified sync.
- `DELETE /api/payments/:environment/config`: disables the secret key and best-effort deletes managed Stripe webhook endpoints for that environment.
- `POST /api/payments/sync`: syncs products, prices, customers, and subscriptions for all configured environments.
- `POST /api/payments/:environment/sync`: syncs products, prices, customers, and subscriptions for one environment.
- `GET /api/payments/:environment/customers`: lists mirrored Stripe customers for one environment.
- `POST /api/payments/:environment/webhook`: recreates the InsForge-managed Stripe webhook endpoint for the environment.
- `GET /api/payments/:environment/catalog`: reads mirrored products and prices.
- `GET|POST|PATCH|DELETE /api/payments/:environment/catalog/products...`: manages Stripe products.
- `GET|POST|PATCH|DELETE /api/payments/:environment/catalog/prices...`: manages Stripe prices, where delete archives the Stripe price.
- `GET /api/payments/:environment/subscriptions`: reads mirrored subscriptions for dashboard/admin use.
- `GET /api/payments/:environment/payment-history`: reads payment history for dashboard/admin use.
## Key Management and Sync Semantics
The secret store is the canonical runtime source for Stripe keys. Environment variables are only seed inputs.
- Test key secret name: `STRIPE_TEST_SECRET_KEY`
- Live key secret name: `STRIPE_LIVE_SECRET_KEY`
- Test webhook secret store key: `STRIPE_TEST_WEBHOOK_SECRET`
- Live webhook secret store key: `STRIPE_LIVE_WEBHOOK_SECRET`
When a Stripe key is saved, InsForge validates the key prefix, retrieves the Stripe account id, and compares it with the existing connection row.
If the exact key is already configured and the connection has an account id, the save is a no-op.
If the key is different but points to the same Stripe account, InsForge updates the secret and connection metadata but skips webhook recreation and sync.
If the key points to a different Stripe account, InsForge clears all mirrored payment data for that environment, best-effort recreates the managed webhook, persists the new key and account metadata, then runs unified sync.
If webhook creation fails because the backend URL is not publicly accessible, key configuration still succeeds and sync still runs. Developers can retry webhook setup later from the Webhooks tab or use Stripe CLI for local webhook testing.
Manual sync does not touch webhook setup. It only pulls Stripe data and updates local mirrors/projections.
## Checkout Flow
Checkout creation is a two-step local-plus-Stripe process.
First, InsForge inserts `payments.checkout_sessions` using the caller's Postgres role and JWT context. This lets future developer-defined policies work without changing the route. The current migration does not enable RLS by default.
Second, if the insert succeeds, InsForge creates the Stripe Checkout Session. If Stripe creation succeeds, the local row becomes `open` with Stripe ids and URL. If Stripe creation fails, the local row becomes `failed`.
Idempotency is handled at both layers. The local table has a unique partial index on `(environment, idempotency_key)`. If a caller retries the same request and the existing row has a usable Stripe URL, InsForge returns it. If the existing row is incomplete, InsForge retries Stripe creation using the same local row.
Caller-provided metadata cannot use keys that start with `insforge_`. InsForge owns these reserved keys because webhooks trust them to recover checkout mode, checkout session id, and billing subject.
When an identified one-time checkout has no existing customer mapping, InsForge asks Stripe Checkout to create a customer with `customer_creation = always`. When the checkout completes and Stripe returns a customer id, InsForge creates or updates `payments.stripe_customer_mappings`.
One-time checkout does not currently create a separate checkout item projection. The checkout attempt stores line items, while durable fulfillment state is represented by webhook-driven payment history and app-specific business tables.
## Customer Portal Flow
Customer portal sessions are authenticated-only. Anonymous users cannot create them.
The request includes a billing subject and optional return URL/configuration id. InsForge inserts `payments.customer_portal_sessions` using the caller's Postgres role and JWT context. Developers or agents can add app-specific RLS policies on this table to decide who may create portal sessions for which subject.
After the local insert succeeds, InsForge looks up the subject in `payments.stripe_customer_mappings`. If there is no mapped Stripe customer, the request fails with `404`. If there is a mapping, InsForge creates a Stripe Billing Portal Session and stores the returned portal URL.
## Webhook Projection Flow
Managed webhooks listen for the events needed to maintain checkout, subscription, payment history, and refund projections:
- `checkout.session.completed`
- `checkout.session.async_payment_succeeded`
- `checkout.session.async_payment_failed`
- `checkout.session.expired`
- `invoice.paid`
- `invoice.payment_failed`
- `payment_intent.succeeded`
- `payment_intent.payment_failed`
- `charge.refunded`
- `refund.created`
- `refund.updated`
- `refund.failed`
- `customer.subscription.created`
- `customer.subscription.updated`
- `customer.subscription.deleted`
- `customer.subscription.paused`
- `customer.subscription.resumed`
Webhook processing is idempotent. A duplicate processed or ignored event is returned as already handled. Failed events can be retried and will increment `attempt_count`.
Payment history supports out-of-order refund events. When refund context is missing locally, InsForge retrieves the PaymentIntent, Charge, and Invoice Payments context from Stripe, hydrates the original payment/invoice row where possible, and preserves previously known context with `COALESCE` rather than overwriting it with null fields.
Subscription projections support existing Stripe accounts. If a synced subscription has no InsForge billing subject mapping, it is still imported with nullable subject fields and counted as unmapped.
## Dashboard Surface
The Payments feature has a secondary menu with Products and Subscriptions.
Products shows test/live tabs, environment-specific empty states when a key is missing, product rows aligned with the Realtime Messages visual style, and product detail rows for associated prices.
Subscriptions shows test/live tabs and subscription detail rows for subscription items. Existing Stripe subscriptions can appear without InsForge subject mapping.
The Payments Settings dialog has three tabs:
- Stripe Keys: configure or remove test/live secret keys.
- Sync: run unified sync across configured environments.
- Webhooks: view managed webhook state and retry automatic webhook configuration.
## Service Boundaries
`PaymentService` is the main orchestrator. It delegates focused work to smaller payment services and keeps cross-service orchestration in one place.
`PaymentConfigService` owns key storage, account status, managed webhook configuration, mirror clearing on account changes, and catalog snapshot writes.
`PaymentProductService` owns product list/get/create/update/delete and local product mirror writes.
`PaymentPriceService` owns price list/get/create/update/archive and local price mirror writes.
`PaymentCheckoutService` owns local checkout session insertion, retry lookup, and checkout row status updates.
`PaymentCustomerPortalService` owns local customer portal session insertion and status updates.
`PaymentHistoryService` owns payment history projection from checkout, invoice, payment intent, charge, and refund events.
`PaymentSubscriptionService` owns subscription and subscription item projection from sync and webhooks.
`PaymentWebhookService` owns webhook event deduplication and processing status records.
`StripeProvider` is the only wrapper around the official Stripe SDK. It owns Stripe API calls, pagination, webhook signature construction, and optional idempotency request options.
Helpers are pure utility functions only. They do not query Postgres or call Stripe.
## Explicit Non-Goals for This Phase
This phase does not implement Stripe Connected Accounts, Express/Custom account onboarding, claimable sandboxes, or platform-managed merchant accounts.
This phase does not implement test-to-live publishing or catalog diff application. Agents can target `test` or `live` explicitly in product and price APIs.
This phase does not expose runtime-safe read APIs for end-user subscription/payment state. Admin reads exist today. End-user reads are deferred because permission semantics depend on each app's subject model.
This phase does not mirror invoices, charges, payment methods, or checkout session line items beyond the customer projection added for admin visibility.
This phase does not define default app-specific RLS policies for payment history, subscriptions, customer mappings, or customer portal sessions. Agents should generate policies based on the developer's app schema.
## CLI, SDK, Docs, and OpenAPI Surfaces
CLI and SDK work should expose the runtime route pair first: create Checkout Session and create Customer Portal Session. These are the APIs generated apps need to collect money and let customers manage subscriptions.
Admin SDK and CLI work should then expose key configuration, status, unified sync, webhook configuration, catalog reads, product CRUD, price CRUD, subscription reads, and payment history reads.
OpenAPI documents the current `/api/payments` and `/api/webhooks/stripe/:environment` surfaces in `openapi/payments.yaml`, including environment targeting and the distinction between runtime routes and admin routes.
Agent docs should focus on workflows:
- Configure Stripe test/live keys.
- Sync Stripe state.
- Create products and one-time/recurring prices.
- Build checkout flows with success and cancel URLs.
- Build subscription checkout with a billing subject.
- Add customer portal access with app-specific RLS on `payments.customer_portal_sessions`.
- Use webhooks and payment projections to update app-specific entitlement tables.
Public docs should explain the developer-owned Stripe account model, local development webhook limitations, Stripe CLI testing, and the fact that Stripe remains the source of truth.
## Verification Checklist
Use this checklist when changing the payment implementation:
- Backend unit tests cover config, sync, catalog CRUD, checkout, portal sessions, webhooks, refunds, subscriptions, and migration idempotency.
- Backend lint, typecheck, and build pass.
- Shared schema lint, typecheck, and build pass.
- Dashboard lint, typecheck, and build pass when dashboard payment UI changes.
- Payments migrations remain idempotent: every create/index/trigger/grant/alter operation in migrations 039 and 040 is safe to re-run.
- Stripe webhook secrets are not documented or required as environment variables.
- Product and price mutations call Stripe first and update the local mirror only after Stripe succeeds.
- Sync treats Stripe as the source of truth and clears mirrors only when the Stripe account id changes.
.internal/docs/plans/2026-05-05-payments-customers-mirror.md
+36
View File
@@ -0,0 +1,36 @@
# Payments Customers Mirror Implementation Plan
> **For agentic workers:** Use the repo payment patterns as the source of truth. Keep `payments.customers` read-only and admin-facing. Do not make runtime checkout or portal flows depend on it.
**Goal:** Add a `payments.customers` table that mirrors Stripe customers for each Stripe environment and expose it in the admin payments dashboard as a display-only surface.
**Architecture:** Keep `payments.stripe_customer_mappings` as the operational subject-to-customer bridge. Add a separate customer mirror that is populated from Stripe sync plus customer webhooks, and expose it through an admin read route plus dashboard page.
**Files expected to change:**
- Create: `backend/src/infra/database/migrations/040_create-payments-customers-table.sql`
- Create: `backend/src/services/payments/payment-customer.service.ts`
- Modify: `backend/src/types/payments.ts`
- Modify: `backend/src/providers/payments/stripe.provider.ts`
- Modify: `backend/src/services/payments/constants.ts`
- Modify: `backend/src/services/payments/payment-config.service.ts`
- Modify: `backend/src/services/payments/payment.service.ts`
- Modify: `backend/src/api/routes/payments/index.routes.ts`
- Modify: `packages/shared-schemas/src/payments.schema.ts`
- Modify: `packages/shared-schemas/src/payments-api.schema.ts`
- Modify: `openapi/payments.yaml`
- Modify: `packages/dashboard/src/router/AppRoutes.tsx`
- Modify: `packages/dashboard/src/features/payments/components/PaymentsSidebar.tsx`
- Create: `packages/dashboard/src/features/payments/hooks/usePaymentCustomers.ts`
- Modify: `packages/dashboard/src/features/payments/services/payments.service.ts`
- Create: `packages/dashboard/src/features/payments/pages/CustomersPage.tsx`
- Modify: `backend/tests/unit/payments-schema-migration.test.ts`
- Modify: `backend/tests/unit/stripe-provider.test.ts`
- Modify: `backend/tests/unit/payment.service.test.ts`
**Implementation order:**
1. Add the database table and migration coverage.
2. Add backend Stripe customer types/provider methods and mirror service.
3. Hook customer sync and customer webhook projection into the payment orchestration layer.
4. Expose admin read APIs and shared schemas.
5. Add the dashboard Customers page.
6. Update docs/OpenAPI and run verification.
.internal/docs/plans/2026-05-24-posthog-analytics-layout-redesign.md
+1017
View File
File diff suppressed because it is too large Load Diff
.internal/docs/specs/2026-03-16-custom-smtp-design.md
+226
View File
@@ -0,0 +1,226 @@
# Custom SMTP & Email Templates
## Overview
Allow project admins to configure their own SMTP server for email delivery instead of relying on InsForge's cloud backend. Includes customizable email templates with a Source/Preview editor.
## Motivation
Currently all emails (verification, password reset, raw) route through InsForge's cloud API. Self-hosted users and cloud users who prefer their own mail delivery have no alternative. This feature adds a pluggable SMTP provider and a template editor, following the same UX pattern as Supabase.
## Architecture
### Provider Selection Flow
```
EmailService.sendWithTemplate() / sendRaw()
-> Check email.config (enabled?)
-> YES: SmtpEmailProvider (nodemailer) + local template rendering
-> NO: CloudEmailProvider (current behavior, unchanged)
```
No breaking changes. The `EmailProvider` interface (`sendWithTemplate`, `sendRaw`, `supportsTemplates`) remains unchanged. Callers are unaffected.
## Database
### Table: `email.config` (singleton)
| Column | Type | Default | Description |
|--------|------|---------|-------------|
| id | UUID | gen_random_uuid() | Primary key |
| enabled | BOOLEAN | FALSE | Toggle SMTP on/off without deleting config |
| host | TEXT | NOT NULL | SMTP server hostname or IP |
| port | INTEGER | 465 | SMTP port (465 for TLS, 587 for STARTTLS) |
| username | TEXT | NOT NULL | SMTP auth username |
| password_encrypted | TEXT | NOT NULL | Encrypted SMTP password |
| sender_email | TEXT | NOT NULL | From address (e.g. noreply@yourdomain.com) |
| sender_name | TEXT | NOT NULL | Display name in recipient inbox |
| min_interval_seconds | INTEGER | 60 | Minimum seconds between emails to same user |
| created_at | TIMESTAMPTZ | NOW() | |
| updated_at | TIMESTAMPTZ | NOW() | |
Singleton enforced via unique index on constant expression (same pattern as `auth.configs`).
### Table: `email.templates`
| Column | Type | Default | Description |
|--------|------|---------|-------------|
| id | UUID | gen_random_uuid() | Primary key |
| template_type | TEXT | NOT NULL, UNIQUE | Template identifier |
| subject | TEXT | NOT NULL | Email subject line |
| body_html | TEXT | NOT NULL | Raw HTML with placeholder variables |
| created_at | TIMESTAMPTZ | NOW() | |
| updated_at | TIMESTAMPTZ | NOW() | |
### Template Types
- `email-verification-code`
- `email-verification-link`
- `reset-password-code`
- `reset-password-link`
### Template Placeholders
| Placeholder | Description | Available In |
|-------------|-------------|--------------|
| `{{ code }}` | 6-digit OTP code | verification-code, reset-password-code |
| `{{ link }}` | Verification/reset URL | verification-link, reset-password-link |
| `{{ email }}` | Recipient email address | All templates |
### Default Templates
Seeded on migration. Clean, minimal HTML with inline CSS. Example for `email-verification-code`:
- **Subject:** "Verify your email"
- **Body:** Simple centered card with "Your verification code is: {{ code }}" and expiry notice
## Backend
### New: `SmtpEmailProvider`
**Location:** `backend/src/providers/email/smtp.provider.ts`
- Implements `EmailProvider` interface
- Uses `nodemailer` to send emails
- `sendWithTemplate()`: queries `email.templates WHERE template_type = $1` using the `EmailTemplate` string value, replaces placeholders via string interpolation (all placeholder values are HTML-escaped before interpolation to prevent XSS), sends via SMTP
- `sendRaw()`: sends directly with provided to/subject/html/cc/bcc. The `from` field is always overridden with `sender_email`/`sender_name` from SMTP config (prevents spoofing)
- `supportsTemplates()`: returns `true`
- TLS auto-detected by nodemailer based on port (465 = implicit TLS, 587 = STARTTLS)
- Transporter created on-demand from DB config (config changes take effect without restart)
**Note:** Enabling SMTP automatically switches from cloud-rendered templates to locally-rendered templates from `email.templates`, even if the user has not customized them. The seeded defaults ensure this works out of the box.
### Modified: `EmailService`
**Location:** `backend/src/services/email/email.service.ts`
- On each send call, check `email.config` for an enabled config
- If enabled: delegate to `SmtpEmailProvider`
- If not enabled: delegate to `CloudEmailProvider` (current default)
- Provider is resolved per-call, not cached at startup
### New: SMTP Config Service
**Location:** `backend/src/services/email/smtp-config.service.ts`
- `getSmtpConfig()`: returns config with password masked
- `upsertSmtpConfig(input)`: creates or updates config, encrypts password using `EncryptionManager` (AES-256-GCM). On save, validates SMTP connection via `transporter.verify()` — rejects invalid credentials before persisting
- Singleton pattern (consistent with other services)
### New: Email Template Service
**Location:** `backend/src/services/email/email-template.service.ts`
- `getTemplates()`: returns all templates
- `getTemplate(type)`: returns single template
- `updateTemplate(type, subject, bodyHtml)`: updates a template
### Validation Schemas
New Zod schemas in `@insforge/shared-schemas` (consistent with existing patterns like `updateAuthConfigSchema`):
- `upsertSmtpConfigRequestSchema` — validates host, port (number), username, password (optional on update), sender_email (email format), sender_name, min_interval_seconds (positive integer), enabled (boolean)
- `updateEmailTemplateRequestSchema` — validates subject (non-empty string), body_html (non-empty string)
### API Endpoints
All endpoints require `verifyAdmin` middleware (same as `GET/PUT /api/auth/config`).
All PUT endpoints emit audit log entries via `auditService.log()` (consistent with existing `PUT /api/auth/config` pattern):
- `PUT /api/auth/smtp-config` → action: `'UPDATE_SMTP_CONFIG'`
- `PUT /api/auth/email-templates/:type` → action: `'UPDATE_EMAIL_TEMPLATE'`
#### SMTP Config
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/auth/smtp-config` | Get SMTP config (password masked as boolean) |
| PUT | `/api/auth/smtp-config` | Create or update SMTP config |
#### Email Templates
| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/auth/email-templates` | Get all email templates |
| PUT | `/api/auth/email-templates/:type` | Update a template's subject and body |
### Password Handling
- Password encrypted before storage (never stored in plaintext)
- GET endpoint returns `has_password: true/false` instead of the actual value
- Password can only be overwritten, never viewed after saving
### Rate Limiting
- `min_interval_seconds` from SMTP config enforced at the service level
- Tracked via an in-memory `Map<string, number>` per recipient in `EmailService` (lightweight, no DB overhead; does not survive restarts or span multiple instances)
- Before sending, check if enough time has elapsed since the last email to the same recipient; reject with 429 if not
- All recipients are rate-limited (not just the primary), enforced for both `sendWithTemplate` and `sendRaw`
- This is a service-level check (not Express middleware), since it depends on per-recipient state rather than per-IP
## Frontend
### Auth Settings Dialog — New "Email" Tab
**Location:** `frontend/src/features/auth/components/AuthSettingsMenuDialog.tsx`
A new tab added alongside existing tabs (General, Email Verification, Password).
#### Card 1: SMTP Provider Settings
- **Toggle:** "Enable Custom SMTP" (switch)
- When enabled, show fields:
- Sender email (text input)
- Sender name (text input)
- Host (text input)
- Port (number input, default 465)
- Minimum interval (number input, seconds, default 60)
- Username (text input)
- Password (password input, masked, cannot be viewed once saved)
- Save button
- Helper text: "Your SMTP credentials will always be encrypted in our database."
#### Card 2: Email Templates
- Dropdown: select template type
- Subject line (text input)
- Two sub-tabs: **Source** | **Preview**
- Source: HTML textarea for editing raw HTML
- Preview: rendered HTML in sandboxed iframe
- Helper text showing available placeholders for selected template type
- Save button (per template)
### UI Behavior
- Fields disabled when SMTP toggle is off
- Password field shows placeholder dots when a password exists, empty when not set
- Template preview updates live as user edits source HTML
- Follows existing dialog styling and component patterns
- The "Email" tab uses its own independent form state and API calls (separate from the existing auth config form), since it talks to different endpoints (`/api/auth/smtp-config` and `/api/auth/email-templates`)
- The `AuthSettingsSection` type union is extended with `'email'`
## Migration
**New migration file:** `backend/src/infra/database/migrations/0XX_create-smtp-config-and-email-templates.sql`
1. Create `email.config` table with singleton constraint
2. Create `email.templates` table
3. Seed 4 default templates with subjects and HTML bodies
## Dependencies
- `nodemailer` — new npm dependency for SMTP transport
- `@types/nodemailer` — dev dependency for TypeScript types
- No other new dependencies required
## Non-Goals
- Template engine (Handlebars, EJS, etc.) — simple string interpolation is sufficient
- Env-var-based SMTP config — can be added later as an enhancement
- Send Test Email button — not in Supabase's UX, not included here
- Additional template types beyond the existing 4 auth flows
- "Reset to Default" template button (known limitation — users must manually revert templates)
## Known Limitations
- No "Reset to Default" for templates — if a user edits a template and wants to revert, they must manually restore the original HTML
- Self-signed TLS certificates on SMTP servers are not supported (nodemailer rejects by default) — can be added later via an optional `tls_reject_unauthorized` flag
.internal/docs/specs/2026-04-07-compute-dashboard-ux-design.md
+132
View File
@@ -0,0 +1,132 @@
# Compute Dashboard UX Fixes
**Date:** 2026-04-07
**Status:** Draft
**Branch:** feat/compute-services
## Context
The Compute Services feature has a complete backend API and CLI, but the dashboard UI is read-only — users can see services but can't manage them. Primary users are agents (CLI/API); the dashboard serves as a monitoring/visibility layer.
## Goals
1. Add action buttons (stop/start/delete) to detail view and card dropdown menus
2. Add a "Create Service" dialog for basic service creation from the UI
3. Add a logs panel in the detail view
4. Show error state info on failed services
5. Fix endpoint URL to show the real `.fly.dev` URL
6. Add missing metadata (region, timestamps) to detail view
## Non-Goals
- Full redesign of page layout (keeping existing card grid + detail view)
- Env var editing UI (use CLI for that)
- Real-time log streaming (polling/manual refresh is fine)
## Design
### 1. Detail View — Action Buttons
Add a button row next to the service name heading:
- **Stop** button (visible when status is `running`) — calls `stop` mutation
- **Start** button (visible when status is `stopped`) — calls `start` mutation
- **Delete** button (always visible) — opens ConfirmDialog, calls `remove` mutation
- Buttons are disabled during pending mutations (loading spinners)
- After delete succeeds, navigate back to list view
Uses: `Button` and `ConfirmDialog` from `@insforge/ui`.
### 2. Detail View — Events Panel
Below the specs card, add an events section:
- Fetches events via `computeServicesApi.events(id, 50)` using a `useQuery` with manual refetch
- Displays as a scrollable monospace block with timestamps
- "Refresh" button to re-fetch
- Empty state: "No events available"
- Only shown when `flyMachineId` exists (no events for services that never deployed)
New component: `ServiceEvents.tsx`
> **Why `/events` and not `/logs`:** the endpoint surfaces Fly machine
> **lifecycle events** from `/apps/:app/machines/:id/events` (start/stop/restart/
> exit), **not container stdout/stderr**. The honest name is "events"; calling
> it `/logs` was misleading — it's enough to spot crash loops via exit-stopped
> events but not enough to debug what the app actually printed.
>
> **Real container stdout/stderr is roadmap work**, and when it lands it
> reuses the freshly-vacated `/logs` URL. Three viable paths, evaluated
> [in this thread](https://github.com/InsForge/InsForge/pull/1062#discussion):
>
> | Path | Where flyctl spawns | Token source | Notes |
> |---|---|---|---|
> | A. Cloud-backend spawns `flyctl logs --no-tail -j` | cloud-backend host | org token in cloud's `process.env` | Token never leaves cloud. ~150 lines + tests. Mirrors existing `flyctl tokens attenuate` pattern in `defaultAttenuator`. |
> | B. Per-app read-logs macaroon, OSS spawns | user's host | short-lived macaroon minted by cloud (mirrors `compute deploy-token`) | Adds flyctl dep on user laptops + token storage/rotation. ~230 lines + ops burden. |
> | C. NATS streaming directly | OSS or cloud-backend | org token | What `flyctl logs` uses internally. Most "correct" but largest surface. |
>
> Recommendation when this lands: start with A (smallest delta, reuses
> existing flyctl shell-out infra).
### 3. Detail View — Enhanced Specs
Add to the existing specs grid:
- Region
- Created at (formatted timestamp)
- Updated at (formatted timestamp)
For failed services: show a red alert banner above the specs card saying "This service failed to deploy" with a Delete button.
### 4. Service Cards — Dropdown Menu
Add a three-dot `DropdownMenu` (from `@insforge/ui`) in the top-right of each card:
- Stop (when running)
- Start (when stopped)
- Delete (always, with confirm)
`onClick` stops propagation so the card click (navigate to detail) still works.
### 5. List Header — Create Button
Add a `+ Create Service` button next to the "Services" heading.
Opens a `Dialog` with form fields:
- Name (text input, required)
- Image URL (text input, required)
- Port (number input, default 8080)
- CPU tier (Select: shared-1x, shared-2x, performance-1x, etc.)
- Memory (Select: 256, 512, 1024, 2048, 4096, 8192)
- Region (Select: iad, sin, lax, etc.)
Calls `create` mutation on submit. Dialog closes on success.
New component: `CreateServiceDialog.tsx`
### 6. Endpoint URL Fix
In both `ServiceCard` and `ComputePage` detail view:
- If `endpointUrl` contains a custom domain but `flyAppId` exists, show `https://{flyAppId}.fly.dev` instead
- Helper function: `getReachableUrl(service)` in `constants.ts`
## Files
| File | Action | What |
|------|--------|------|
| `ComputePage.tsx` | Modify | Add create button, action buttons, events, enhanced specs, failed state |
| `ServiceCard.tsx` | Modify | Add dropdown menu, use `getReachableUrl` |
| `useComputeServices.ts` | Modify | Add `useServiceEvents` query hook export |
| `constants.ts` | Modify | Add `getReachableUrl` helper |
| `CreateServiceDialog.tsx` | New | Create service form dialog |
| `ServiceEvents.tsx` | New | Events display component |
## Dependencies
All UI components needed already exist in `@insforge/ui`:
- Button, Dialog, DialogContent, DialogTitle, DialogDescription
- ConfirmDialog
- DropdownMenu
- Input, Select
All API methods already exist in `compute.service.ts`.
All mutations already exist in `useComputeServices` hook.
.internal/docs/specs/2026-04-18-docs-audit.md
+95
View File
@@ -0,0 +1,95 @@
# Spec: docs/ audit + minimal high-severity fixes (closes #1117)
**Date:** 2026-04-18
**Author:** Commander worker (insforge/commander/1117)
**Tier:** T2
## Problem
InsForge's `docs/` tree has ~65 non-deprecated `.mdx` pages. Ticket #1117 asks
for a systematic audit of these pages against the newly-vendored Mintlify
`doc-author` skill (at `.claude/skills/doc-author/`) and the InsForge overlay
at `.claude/skills/doc-author/INSFORGE.md`. Known-smell categories the ticket
calls out:
- Broken internal links to renamed/removed pages
- Navigation pointing at files that no longer exist
- Stale code examples diverging from the current SDK surface
- Deprecated APIs referenced from non-deprecated pages
- Vanilla-markdown where Mintlify components would be idiomatic
(`> ⚠️` → `<Warning>`, fence groups → `<CodeGroup>`, `1. 2. 3.` → `<Steps>`)
- Frontmatter gaps per the InsForge overlay (`title`+`description` only;
no `<ParamField>`; experimental features get `<Warning>` at top)
## Goals
1. Produce a single source-of-truth audit table at
`docs/_audit-2026-04-18.md` listing every issue found, one row per
`(page, issue)`, with severity and a recommended fix.
2. Apply **minimal-diff** fixes to the top five high-severity pages in the
same PR. Every commit cites the anti-pattern from the skill.
3. File one follow-up GitHub issue per remaining high-severity page
(assignee `tonychang04`, label `commander-ready`), and one catch-all
issue for the combined medium/low-severity list.
## Non-goals
- Rewriting pages for style or tone. The skill explicitly favors
minimal-diff.
- Touching anything under `docs/deprecated/`. Those are intentionally
preserved.
- Source code changes. Audit is docs-only (`*.mdx`, `docs.json`, the new
audit file itself).
- Restructuring navigation beyond removing dead entries or fixing typos
— structure work is its own follow-up ticket.
## Proposed approach
1. **Read** every `.mdx` outside `docs/deprecated/` (found 65 files).
2. **Check** each against the categories above using targeted grep
passes: link resolution, frontmatter shape, `<ParamField>` usage,
markdown-warning patterns, emoji usage, fence groups, deprecated-API
mentions.
3. **Cross-check** `docs/docs.json` navigation entries against the
filesystem — missing target files are high-severity (they break
navigation rendering).
4. **Cross-check** existing files against navigation — pages that exist
but aren't reachable from nav are medium-severity orphans.
5. **Write** the audit table.
6. **Fix** the top five high-severity items with one commit per page,
each commit message citing the skill section it enforces.
7. **File** follow-up issues via `gh issue create` per the ticket.
### Alternatives considered
- **Rewrite pages wholesale to match skill voice/tone.** Rejected: the
skill and ticket both explicitly forbid this.
- **Skip the audit, just fix the obvious broken links.** Rejected: the
ticket asks for the audit as the primary deliverable — the fixes are
secondary. The audit becomes the roadmap for follow-up tickets.
- **Include medium/low fixes in this PR.** Rejected: would blow past the
25-minute budget and dilute review focus. Catch-all follow-up issue
keeps them tracked without blocking this PR.
## Test plan
- Manual: every link modified in this PR resolves to an existing file
on disk (`test -f docs/<target>.mdx`).
- Manual: every page modified still parses as MDX (run `grep -c '^---$'`
to verify frontmatter fences are balanced).
- Manual: `docs.json` is valid JSON after any edits (`jq empty`).
- Follow-up: a Mintlify CI check or local preview would catch any
remaining nav/link regressions — out of scope for this PR.
## Risks / rollback
- **Risk:** removing a broken link from `changelog.mdx` might change
rendered output in a way a reader already bookmarked. **Mitigation:**
we repoint rather than delete where a valid destination exists (the
SDK pages under `/sdks/typescript/*`).
- **Risk:** fixing `docs.json` might drop an intentional placeholder
for future content. **Mitigation:** each nav entry we remove is
confirmed to have zero matching `.mdx` in the tree; if the author
intended a placeholder, they can re-add on the follow-up ticket.
- **Rollback:** revert the PR. All changes are docs-only; no runtime
impact.
.internal/docs/specs/2026-04-21-dtest-onboarding-design.md
+408
View File
@@ -0,0 +1,408 @@
# D Test Onboarding Design
**Status:** Implemented
**Owner:** @CarmenDou
**Date:** 2026-04-21 (originally) · Updated 2026-04-25
**Branch:** `feat/support-dtest-onboarding` · **PR:** [#1142](https://github.com/InsForge/insforge-cloud-backend/pull/1142)
## Context
Dashboard home is gated by a single PostHog feature flag `dashboard-v4-experiment` with two variants:
- default / `control` (`DashboardPage`) — baseline, unchanged
- `d_test` — new **install-first** onboarding introduced in this spec
(An earlier `c_test` variant was retired during d_test development; the `CTestDashboardPage` and `ConnectDialogV2` files were deleted, with the prompt stepper carried forward into the d_test connected dashboard.)
D test ships a reworked "Install InsForge" client picker as the pre-connection view, and a connected dashboard (header + 4 metric cards + prompt stepper). On d_test the top-nav Connect button does **not** open any dialog — it switches the page back to the Install view so users can re-visit setup at any time. When dashboard runs inside the InsForge cloud control plane (`insforge.dev`) iframe, the parent's Connect button mirrors this behaviour through a `D_TEST_VIEW_CHANGED` postMessage.
Figma references:
- Install InsForge (client picker): `2194:75236`
- Client detail page (Claude Code example): `2226:78350`
- Connection String detail: `2226:79152`
- Connected dashboard: `2380:89947`
## Goals
1. Let users connect any coding agent (OpenClaw, Claude Code, Codex, Antigravity, Cursor, OpenCode, Copilot, Cline, "Other") or connect directly via DB connection string / API keys, from a single discoverable page.
2. On the connected dashboard show: project header + 4 metric cards (User / Database / Storage / Edge Functions) + a "Your Agent can now do the work for you" prompt stepper to guide further exploration.
3. Use d_test–owned install components (`DTestCLISection`, `DTestMCPSection`, `QuickStartPromptCard`) that can iterate independently of the legacy connect UI; keep the shared `ConnectionStringSectionV2` / `APIKeysSectionV2` for direct-connect tiles.
4. Allow users to toggle between Install view and Dashboard view freely after first connection, including from the InsForge cloud control plane's top-bar Connect button (cross-frame postMessage).
## Non-Goals
- Changing the install commands themselves at the CLI / MCP level (the CLI prompt is a copy-paste recipe; MCP JSON / install commands are the existing ones). The d_test prompt does inject a fresh user API key into the CLI command and substitutes the real DB password into the connection-string prompt — both are display-only changes.
- Changing onboarding detection logic beyond what `useMcpUsage().hasCompletedOnboarding` already provides.
- Replacing the `ConnectDialog` for the `control` variant — it keeps the existing modal.
## Connected-State Detection
D test treats a user as **connected** when `useMcpUsage().hasCompletedOnboarding` is true. That hook resolves to `!!records.length` where `records` comes from `/mcp-usage?success=true&limit=200`, i.e. the agent has successfully invoked ≥ 1 MCP tool.
This is the same signal C test uses today. No new backend work.
## View Model
Two top-level views, both mounted at `/dashboard` (the existing Dashboard home route), switched by an in-page state `view: 'install' | 'dashboard'`. View is **session-local React state** (not URL-backed, not persisted) — simpler than the earlier design and still covers every user-facing transition.
### View resolution
On mount, once `useMcpUsage()` finishes loading, the initial view is:
```text
hasCompletedOnboarding ? 'dashboard' : 'install'
```
Thereafter, the view only changes on three events:
1. **Onboarding completes** (`hasCompletedOnboarding` flips false → true): auto-switch to `'dashboard'`. This is the "MCP call succeeds → jump to dashboard" UX.
2. **Connect clicked** (in d_test) — either our top-nav Connect button (`AppHeader`) when the dashboard renders standalone, or the **InsForge cloud control plane's** top-bar Connect button via `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` postMessage (the iframe scenario). Both route to `setView('install')`. While view is `'install'`, the Connect button is rendered as **disabled** so the user doesn't loop on it.
3. **`[X]` clicked on Install page**: switch to `'dashboard'`.
On refresh the session state resets. The initial-view rule re-runs, so a connected user lands back on dashboard and an unconnected user lands on install — both are the correct defaults. The transient "I just clicked Connect to peek at install" intent is not persisted; if the user wants Install again, they click Connect again.
Within the Install view there is a sub-state `selectedClient`:
```text
view = 'install'
├── selectedClient === null → InstallInsForgePage (All Clients)
└── selectedClient !== null → ClientDetailPage for that client
```
`selectedClient` is session-local and resets when switching to dashboard.
## Navigation Map
```text
┌────────────────────────────┐ ┌────────────────────────────┐
│ InstallInsForgePage │ [X] close │ DTestConnectedDashboard │
│ (All Clients) │────────────────────▶│ (header + 4 metrics) │
│ │ │ │
│ │◀────────────────────│ │
└────┬───────────────────────┘ TopNav Connect └────────────────────────────┘
│
│ click tile
▼
┌────────────────────────────┐
│ ClientDetailPage │
│ (← All Clients) │─── CLI tab ──▶ <NewCLISection />
│ │
│ │─── MCP tab ──▶ <MCPSection initialAgentId={id} />
│ │
│ │ (or ConnectionStringSectionV2 / APIKeysSectionV2
│ │ for Direct Connect tiles, no CLI/MCP toggle)
└────────────────────────────┘
```
## Install Page Layout
Three stacked sections, max-width 640 px, top-padding 64 px, centered:
1. **"Setup In OpenClaw"** — single tile for OpenClaw with `Install` button. (OpenClaw is a distinct agent, not a Figma typo for Claude Code; it is registered as its own `MCPAgent` with `id='openclaw'`, uses `@insforge/install --client openclaw`, and is the `FEATURED_OPENCLAW_ID` in `clientRegistry.tsx`.)
2. **"Install in Coding Agent"** — 2-column × 4-row grid of tiles. Tiles in display order:
1. Claude Code &nbsp;|&nbsp; Codex
2. Antigravity &nbsp;|&nbsp; Cursor
3. OpenCode &nbsp;|&nbsp; Copilot
4. Cline &nbsp;|&nbsp; Other Agents
3. **"Direct Connect"** — 2 tab-style tiles side by side: Connection String | API Keys. These are visually similar to agent tiles but open different detail content.
Top-right of the page header row (same row as the title, within the max-w-640 column): `[X]` close button → switches view to `'dashboard'` (clears `?view` param) and sets `installDismissed = true` in localStorage.
Title text: "Install InsForge".
## Client Detail Page Layout
Top: `← All Clients` text button (always the same label, regardless of client) → clears `selectedClient`.
Below: 32 px client icon + client display name (h2, 28 px medium).
Content changes per client type:
### Coding agents (OpenClaw, Claude Code, Codex, Antigravity, Cursor, OpenCode, Copilot, Cline, Other Agents)
- CLI / MCP toggle (`toggle nav` pattern from Figma) — only rendered when the entry's `tabs` field exposes more than one tab. **OpenClaw** has `tabs: ['cli']` (CLI only, no toggle); **Other Agents** has `tabs: ['mcp']` (MCP only, jumps directly into the MCP JSON config); the rest default to both.
- **CLI tab** → `<DTestCLISection agentName={...} />`. The prompt embeds a real `uak_…` user API key minted by the cloud control plane (`onRequestUserApiKey` callback) on every section mount, with a 3-month TTL — falls back to `<placeholder>` when the host doesn't provide the callback (self-hosted preview).
- **MCP tab** → `<DTestMCPSection agentId={id} apiKey={...} appUrl={...} />`.
- For specific agents, `agentId` matches the tile id (`openclaw`, `claude-code`, `codex`, `cursor`, `antigravity`, `opencode`, `copilot`, `cline`).
- For "Other Agents", the entry sets `mcpAgentId: 'mcp'` which jumps directly to the MCP JSON config (no agent dropdown needed).
- For Cursor and Qoder (deeplink-capable), Step 1 shows an "Install to &lt;agent&gt;" button that opens the MCP-install deeplink and Step 2 shows a "Paste Prompt to &lt;agent&gt;" button that opens the agent's chat-with-prompt deeplink (`cursor://anysphere.cursor-deeplink/prompt?text=...` or `qoder://aicoding.aicoding-deeplink/chat?text=...&mode=agent`). Falls back to clipboard copy if the prompt exceeds Cursor's 8000-char URL limit. Other agents show the terminal command + prompt code blocks.
### Connection String tile
- No CLI/MCP toggle.
- Wrapped in a `<QuickStartPromptCard />` whose prompt embeds the real DB connection string (parent's API returns it with the password masked as `********`; we substitute the real password in via `useDatabasePassword()` so the prompt is paste-ready).
- Below the prompt: `<ConnectionStringSectionV2 variant="vertical" />` with a Show/Hide toggle on the password field. The "copy parameters" button always copies the real password regardless of reveal state, matching the connection-string copy behavior.
- Title: "Connection String", icon: database.
### API Keys tile
- No CLI/MCP toggle.
- Content: `<APIKeysSectionV2 apiKey={...} anonKey={...} appUrl={...} />`.
- Title: "API Keys", icon: key.
## Connected Dashboard Layout
Matches Figma node `2380:89947`, with the prompt stepper carried over from the (now-deleted) c_test design.
```text
<h1> My Project </h1> [INSTANCE BADGE] ● Healthy
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│ User │ │ DB │ │ Stor │ │ Fns │
└──────┘ └──────┘ └──────┘ └──────┘
┌─────────────────────────────────────────────────────────┐
│ Your Agent can now do the work for you [Dismiss] │
│ Open your coding agent and start building your │
│ project with prompts │
│ │
│ ┌────────────────┬──────────────────────────────────┐ │
│ │ Database │ Step content (icon, title, │ │
│ │ Authentication │ prompt body, Copy / Go-to) │ │
│ │ Storage │ │ │
│ │ Model Gateway │ │ │
│ │ Deployment │ │ │
│ └────────────────┴──────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
Project title, instance-type badge, and health badge use `useCloudProjectInfo` + `useMetadata`. Each metric card uses the shared `MetricCard` component. The stepper is the self-contained `DashboardPromptStepper` — see the next section.
## Prompt Stepper (`DashboardPromptStepper`)
A 5-step "Start building" stepper rendered below the metric cards. Self-contained: each instance manages its own dismiss flag and step-completion derivation.
- **5 steps** (database → auth → storage → model gateway → deployment), each with a copy-pastable prompt and a "Go to &lt;area&gt;" button.
- **Live completion detection** from existing hooks (intentionally schema-agnostic so prompts work for any project, not just a fixed demo):
- `database` → `tables.some(t => t.recordCount > 0)` (any user table has rows)
- `auth` → `totalUsers >= 1`
- `storage` → `storage.buckets.length > 0` (any bucket exists)
- `ai` → `aiUsageSummary.totalRequests > 0`
- `deployment` → `currentDeploymentId` exists
- **Sticky completion**: once a step is detected complete, it stays complete in localStorage (`insforge-ctest-step-<key>-done-<projectId>`) even if the agent later removes the source data (e.g. via newly added RLS policies). Key prefix is `insforge-ctest-` for backwards compatibility with users who progressed through c_test.
- **Dismiss**: persists `insforge-prompt-stepper-dismissed-<projectId>` in localStorage. Once dismissed for a project, the stepper never shows for that project again.
## File Plan
### New files
```text
packages/dashboard/src/features/dashboard/
├── pages/
│ └── DTestDashboardPage.tsx # entry; reads view state, dispatches
└── components/
└── dtest/
├── InstallInsForgePage.tsx # All Clients view (3 sections)
├── ClientDetailPage.tsx # detail shell: back + title + slot
├── ClientTile.tsx # reusable tile for agents & direct-connect
├── DTestConnectedDashboard.tsx # header + 4 metric cards + prompt stepper
├── DTestCLISection.tsx # CLI tab content (prompt with real uak_ key)
├── DTestMCPSection.tsx # MCP tab content (deeplinks for Cursor/Qoder, terminal/JSON for others)
├── DTestConnectTip.tsx # fixed-position "you can re-connect" tip overlay (cloud-hosting only)
├── DTestViewContext.tsx # React context: view + selectedClient + cross-frame postMessage
├── DashboardPromptStepper.tsx # self-contained 5-step stepper for connected dashboard
├── QuickStartPromptCard.tsx # generic "Paste this into your agent" prompt card
└── clientRegistry.tsx # tile metadata (id, label, icon, kind, mcpAgentId, tabs)
```
Plus a logo asset for the Other Agents tile (`assets/logos/other_agents.svg`) and updated logos for Claude Code (PNG) and Codex (SVG).
### Shared components extracted
```text
packages/dashboard/src/features/dashboard/components/
└── MetricCard.tsx # lifted from CTestDashboardPage.tsx (currently an inner function)
```
`CTestDashboardPage.tsx` loses its inner `MetricCard` definition and imports the shared one.
### Modified files
- `packages/dashboard/src/router/AppRoutes.tsx`
- Pick `DTestDashboardPage` when `dashboardVariant === 'd_test'`, otherwise `DashboardPage`.
- `packages/dashboard/src/layout/AppLayout.tsx`
- Always render the `ConnectDialog` (v1); the `c_test`-branched V2 dialog was removed.
- Wrap the layout tree in `DTestViewProvider` so `AppHeader`, `DTestDashboardPage`, `DTestConnectTip`, and `AppSidebar` all share view state.
- Add `ConnectOverlayBridge` (rendered inside the provider) — listens for `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` postMessages from the parent window. In d_test, routes the signal to `setView('install')` instead of opening the dialog.
- `packages/dashboard/src/layout/AppHeader.tsx`
- On d_test: Connect onClick calls `setView('install')` from `DTestViewContext`. Disabled when already on the Install view.
- Tip JSX/state was extracted to `DTestConnectTip` (fixed-position overlay, see below).
- `packages/dashboard/src/layout/AppSidebar.tsx`
- Don't highlight the Dashboard nav item while the user is on the d_test Install view.
- `packages/dashboard/src/lib/config/DashboardHostContext.tsx`
- Add `onRequestUserApiKey?: () => Promise<string>` to the host contract, plumbed through `InsforgeDashboard` props.
- `packages/dashboard/src/lib/analytics/posthog.tsx`
- Restore `session_recording: { recordCrossOriginIframes: true }` so PostHog session replay doesn't choke on the cross-origin iframe boundary (was dropped in an earlier refactor).
- `packages/dashboard/src/lib/contexts/SocketContext.tsx`
- Rename the `experiment_variant` tag on `onboarding_completed` analytics to `dashboard-v4-experiment`.
#### Frontend bridge (`frontend/src/cloud-hosting/`)
- `useCloudHosting.ts`: adds `requestUserApiKey()` (REQUEST_USER_API_KEY postMessage with USER_API_KEY / USER_API_KEY_ERROR response).
- `CloudHostingDashboard.tsx`: passes `onRequestUserApiKey={requestUserApiKey}` through to `InsForgeDashboard`.
### Deleted files
- `packages/dashboard/src/features/dashboard/pages/CTestDashboardPage.tsx` — c_test variant retired; the prompt stepper was extracted into `DashboardPromptStepper.tsx`.
- `packages/dashboard/src/features/dashboard/components/connect/ConnectDialogV2.tsx` — only used by c_test.
## Client Registry
`clientRegistry.ts` centralizes the tile data so both the grid and the detail routing can look up by id:
```ts
type ClientId =
| 'openclaw'
| 'claude-code'
| 'codex'
| 'antigravity'
| 'cursor'
| 'opencode'
| 'copilot'
| 'cline'
| 'other'
| 'connection-string'
| 'api-keys';
type ClientEntry = {
id: ClientId;
label: string;
icon: ReactNode;
detailIcon: ReactNode;
kind: 'agent' | 'direct-connect';
/** MCP detail preselection. Use 'mcp' for "Other Agents"; omit for direct-connect. */
mcpAgentId?: string;
/**
* Tabs available on the detail page for `kind: 'agent'`. Omit = both CLI and
* MCP. Use ['cli'] for OpenClaw (install flow only), ['mcp'] for "Other
* Agents" (drops straight into the MCP JSON config).
*/
tabs?: ReadonlyArray<'cli' | 'mcp'>;
};
```
`FEATURED_OPENCLAW_ID = 'openclaw'` is the featured tile in Section 1; `CODING_AGENT_GRID_IDS` renders the Section 2 grid starting with `'claude-code'`. The `other` entry sets `mcpAgentId: 'mcp'` and `tabs: ['mcp']`; OpenClaw sets `tabs: ['cli']`.
The "featured" section ("Setup In OpenClaw") and grid consume the same entries; only the section they render in differs.
## State Management
A React context (`DTestViewContext`) provided at `AppLayout` level owns `view` + `selectedClient` and exposes a `useDTestView` hook for both `AppHeader` and `DTestDashboardPage`:
```tsx
export function DTestViewProvider({ children }: { children: ReactNode }) {
const { hasCompletedOnboarding, isLoading } = useMcpUsage();
const [selectedClient, setSelectedClient] = useState<ClientId | null>(null);
const [view, setViewState] = useState<DTestView>('install');
// Initialise from onboarding state once loading finishes; thereafter
// auto-flip to dashboard on every false → true transition.
const didInit = useRef(false);
const prevOnboarding = useRef(hasCompletedOnboarding);
useEffect(() => {
if (isLoading) return;
if (!didInit.current) {
setViewState(hasCompletedOnboarding ? 'dashboard' : 'install');
didInit.current = true;
} else if (!prevOnboarding.current && hasCompletedOnboarding) {
setViewState('dashboard');
}
prevOnboarding.current = hasCompletedOnboarding;
}, [hasCompletedOnboarding, isLoading]);
const setView = useCallback((next: DTestView) => {
setViewState(next);
if (next === 'dashboard') setSelectedClient(null);
}, []);
// ...provider returned here
}
```
Key points:
- **No URL param, no localStorage.** View is pure session state. Refresh recomputes from `hasCompletedOnboarding`.
- **Single source of truth for view.** `AppHeader.showConnectTip` and `DTestDashboardPage` both read `view` from the same context, so the Connect tip correctly hides while the user is on the Install view.
- **Provider is mounted for every user**, not just d_test. Non-d_test components don't consume it, and `useMcpUsage()` is already React-Query-cached so the extra call is free.
- **`[X]` on Install** calls `setView('dashboard')` — no dismissal flag, no persistence.
- **Top-nav Connect on d_test** (only while on `/dashboard`) calls `setView('install')`.
- **MCP call success** (the `hasCompletedOnboarding` false → true transition) auto-switches to `'dashboard'` so users see their connected state immediately.
- `selectedClient` is session-local; switching to dashboard clears it.
## Cross-frame postMessage protocol
When the dashboard runs inside the InsForge cloud control plane (`insforge.dev`) via iframe, it coordinates with the parent through several postMessage events:
| Direction | Type | Purpose |
|---|---|---|
| Parent → iframe | `SHOW_CONNECT_OVERLAY` / `SHOW_ONBOARDING_OVERLAY` | Parent's top-bar Connect button click. iframe handles in `ConnectOverlayBridge`: in d_test → `setView('install')`; otherwise → opens the v1 ConnectDialog. |
| iframe → Parent | `D_TEST_VIEW_CHANGED { view: 'install' \| 'dashboard' }` | View state mirror. Parent's `ConnectButton` reads this and disables itself while view is `'install'`. Only sent when the variant is `d_test`. |
| iframe → Parent | `REQUEST_USER_API_KEY` | Iframe wants a fresh `uak_…` PAT for the CLI install prompt. |
| Parent → iframe | `USER_API_KEY { apiKey }` / `USER_API_KEY_ERROR { error }` | Response to the above. Parent's `userApiKeyService` calls `POST /account/v1/api-keys` with a 90-day TTL. |
The `useCloudHosting` hook (in `frontend/src/cloud-hosting/`) owns the iframe-side request/response bookkeeping; the parent side lives in `insforge-cloud/src/app/dashboard/project/[projectId]/page.tsx` (existing handler) and `insforge-cloud/src/features/project/components/ConnectButton.tsx` (new disable-state subscriber).
## Feature Flag
Dashboard variant is gated by a single PostHog flag, `dashboard-v4-experiment`. Resolved values:
- `'d_test'` → `DTestDashboardPage`
- anything else → `DashboardPage` (the legacy default)
```ts
// AppRoutes.tsx
const dashboardVariant = getFeatureFlag('dashboard-v4-experiment');
const DashboardHomePage = dashboardVariant === 'd_test' ? DTestDashboardPage : DashboardPage;
```
PostHog flag configuration is dashboard-side, out of scope for the code PR. The `dashboard-v3-experiment` flag is no longer referenced in code; SocketContext analytics report `dashboard-v4-experiment` instead.
## Testing
This is a UI-only change; verification is primarily manual through the dev server (PostHog override) and the staging cloud control plane (real iframe).
- For each variant (`control`, `d_test`):
- Load `/dashboard` with an account that has **no** MCP usage → correct "unconnected" view renders.
- Load `/dashboard` with an account that has MCP usage → correct "connected" view renders.
- D-test-specific flows (self-hosted preview):
- Click each agent tile → detail page renders with the right icon/title. OpenClaw shows CLI only. Other Agents shows MCP only. Other agents show CLI/MCP toggle, default to CLI.
- Click Connection String tile → prompt + `ConnectionStringSectionV2` rendered inside detail shell. Real DB password substituted in the prompt and copy.
- Click API Keys tile → `APIKeysSectionV2` renders inside the detail shell.
- `← All Clients` from any detail → back to grid.
- `[X]` on Install page → lands on dashboard view.
- Connect button in top nav (on d_test) → routes to Install page. Becomes disabled while on Install.
- MCP tool succeeds while on Install → view auto-flips to dashboard.
- Cursor / Qoder "Paste Prompt to" button opens deeplink (URL bar shows `cursor://` or `qoder://`); for other agents, copies to clipboard.
- D-test-specific flows (cloud iframe — staging):
- Connect button in InsForge cloud's top-bar disables while iframe view = `'install'`.
- CLI install prompt embeds a real `uak_…` key (each tab mount mints a new one).
- DTestConnectTip overlay appears in cloud-hosting on dashboard view; dismiss persists per project.
- Sidebar Dashboard nav item not highlighted while on Install view.
- Cross-variant regression:
- On `control`, Connect button still opens `ConnectDialog` modal, not Install page.
## Risk & Rollback
- Feature-flagged end-to-end; rollback is a PostHog flag change (set to `control` or remove).
- `DTestViewProvider` is mounted for all users, not just d_test. It calls `useMcpUsage()` at layout level, but that hook is already invoked by `AppHeader` and is React-Query-cached, so the provider does not add a new request.
- Cross-frame postMessage requires both halves (iframe-side `D_TEST_VIEW_CHANGED` emit + parent-side `ConnectButton` listener) to be deployed. Either half landing alone is harmless: the parent's Connect button just defaults to enabled, and the iframe's bridge silently no-ops if no listener exists.
- User API key minting flow gates on `host.onRequestUserApiKey` being defined. Self-hosted installs (no host callback) fall back to `<placeholder>` in the CLI prompt — visible but obviously placeholder, copy disabled.
- Backend-side: the cloud control plane's `userApiKeyService` calls `POST /account/v1/api-keys` with a 90-day TTL. Backend has a soft `MAX_ACTIVE_KEYS_PER_USER = 500` cap (in `appConfig.limits.maxActiveApiKeysPerUser`); on overflow returns 409 which surfaces as "Could not generate API key" in the UI without crashing.
## Connect Tip (`DTestConnectTip`)
Floating "You can always click here to re-connect" hint that appears on the connected dashboard view in cloud-hosting only. Rendered at `AppLayout` level (NOT inside `AppHeader`, because `showNavbar={false}` hides our `AppHeader` when the dashboard runs inside the cloud iframe — the tip needs to live outside it).
Display conditions (all must be true):
- `host.mode === 'cloud-hosting'`
- `dashboardVariant === 'd_test'`
- Current view = `'dashboard'`
- Not dismissed (per-project localStorage flag `insforge-dtest-connect-tip-dismissed-<projectId>`)
Position: `fixed right-4`. Top offset depends on `host.showNavbar`: `top-2` when our AppHeader is hidden (cloud-hosting iframe — sits just below the parent's top bar) or `top-14` when it shows (self-hosted preview — clears our 48px AppHeader). Dismissed state persists per-project; once dismissed, the tip never reappears for that project.
The arrow on the tip card points up at the (parent's) Connect button via offset `right-[72px]` within the 220px-wide card.
## Open Items
- None at the time of merge — d_test variant configuration in PostHog is set up; backend's `MAX_ACTIVE_KEYS_PER_USER = 500` cap is in place.
.internal/docs/specs/2026-04-22-s3-compatible-storage-gateway-design.md
+545
View File
@@ -0,0 +1,545 @@
# S3-Compatible Storage Gateway
## Overview
Add an S3-protocol HTTP gateway in front of InsForge's Storage module so that any AWS S3-compatible client (`aws` CLI, `rclone`, AWS SDKs, Terraform, backup tools) can read and write InsForge buckets with no code changes.
The gateway sits at `/storage/v1/s3` on each project's backend host (e.g. `{appkey}.{region}.insforge.app/storage/v1/s3`). It verifies AWS SigV4 signatures against project-scoped access keys, dispatches to a small set of S3 operation handlers, and delegates physical IO to the existing `S3StorageProvider`. Object and bucket metadata stays consistent with the REST API by sharing the `storage.buckets` and `storage.objects` tables.
## Motivation
InsForge today exposes only a REST API for storage. Developers who want to migrate existing S3-based workloads (CI upload steps, `aws s3 sync`, backup scripts, Terraform `aws_s3_object` resources) have to rewrite their integration. Supabase shipped an S3-compatible gateway for the same reason; open-sourcing that design ([supabase/storage](https://github.com/supabase/storage)) means a well-trodden implementation path exists. This feature closes the compatibility gap so that InsForge Storage becomes drop-in usable from any S3 toolchain.
## Goals & Non-Goals
### Goals (v1)
1. `aws s3 cp <file> s3://<bucket>/<key>` and the reverse work zero-config (including files >200 MB via automatic multipart).
2. `aws s3 sync` and `rclone sync` work bidirectionally.
3. Objects uploaded via S3 protocol appear immediately in the InsForge Dashboard and REST API `GET /api/storage/buckets/:bucket/objects`. Reverse direction works too.
4. Secret access keys are never exposed after creation; the DB stores only encrypted ciphertext.
5. Large uploads stream end-to-end; memory usage does not scale with object size.
### Non-Goals (v1)
- Presigned URLs (query-string auth) for GET or PUT. Users wanting browser direct uploads use the existing REST `POST /api/storage/buckets/:bucket/upload-strategy`.
- Session-token auth (user-JWT-scoped S3 access, Supabase-style `sessionToken` via `X-Amz-Security-Token`). v1 only supports the project-admin-level `storage.s3_access_keys` credentials. See Open Question 7.
- Support for the `LocalStorageProvider` backend. The gateway refuses to mount if the backend is local; self-hosted users who want S3 protocol run MinIO and point `AWS_S3_BUCKET` + `S3_ENDPOINT_URL` at it.
- Virtual-hosted-style URLs (`{bucket}.endpoint/...`). Only path-style (`endpoint/{bucket}/{key}`).
- Signature V2.
- S3 governance features: versioning, SSE-C / SSE-KMS, ACLs, bucket policy, object lock, tagging, lifecycle, replication, inventory, analytics, CORS config.
- S3 event notifications.
## Architecture
### Endpoint & Routing
- External endpoint: `https://{appkey}.{region}.insforge.app/storage/v1/s3`
- SDK configuration: `{ endpoint, region: 'us-east-2', forcePathStyle: true, credentials }`
- Signature region defaults to `us-east-2` to match the region our `S3StorageProvider` uses by default (see `s3.provider.ts`), so requests forwarded to the underlying S3 don't need a separate region translation step. The validated region comes from `AWS_REGION` — the same env var the S3 provider already reads — so clients sign with the same region the backing bucket lives in, and the Dashboard's S3 Config page (`GET /api/storage/s3/config`) surfaces exactly what the middleware will accept.
- Mount path is `/storage/v1/s3` with **no `/api` prefix**. The `/api` prefix would force clients to configure `endpoint=<host>/api`, breaking S3 tooling conventions.
### Request Lifecycle
```text
Client (aws CLI / SDK / rclone)
│ PUT /storage/v1/s3/my-bucket/photo.jpg
│ Authorization: AWS4-HMAC-SHA256 Credential=AK.../us-east-2/s3/aws4_request ...
│ x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
▼
[1] Express app — `/storage/v1/s3/*` mounted BEFORE express.json() body parser.
▼
[2] SigV4 Middleware
│ - Parse Authorization header → AccessKeyId
│ - Lookup via LRU cache → storage.s3_access_keys row → decrypt secret
│ - Verify header signature (clock skew, canonical request, string-to-sign, HMAC chain)
│ - For STREAMING-* requests, only header verification happens here; body verification
│ happens chunk-by-chunk as the stream is consumed downstream.
│ - Asynchronously update last_used_at (fire and forget).
▼
[3] S3 Router — dispatch by (method, path shape, query string, headers).
│ Examples: PUT /{bucket}/{key}?partNumber=N&uploadId=X → UploadPart;
│ POST /{bucket}?delete → DeleteObjects.
▼
[4] Operation Handler (one file per op)
│ - Delegates physical IO to S3StorageProvider (extended).
│ - Reads/writes metadata via StorageService against storage.objects / storage.buckets.
▼
[5] Response Serializer
│ - 2xx: status + S3-style headers (ETag, Content-Length, ...); ListXxx responds with XML.
│ - Error: <Error><Code>...</Code>...</Error> XML via shared error helper.
```
### Module Layout
```text
backend/src/
├── api/
│ ├── middlewares/
│ │ └── s3-sigv4.ts # SigV4 verification middleware + LRU cache
│ └── routes/
│ ├── s3-gateway/ # NEW
│ │ ├── index.routes.ts # mount + method/path/query dispatch
│ │ ├── commands/ # one file per S3 op (14 ops + 2 stubs)
│ │ ├── xml.ts # XML serialization via xml2js
│ │ └── errors.ts # S3 error code → XML response
│ └── storage/
│ └── index.routes.ts # EXTENDED with /s3/access-keys CRUD subroutes
├── services/
│ └── storage/
│ ├── s3-access-key.service.ts # NEW — key CRUD, encryption, LRU cache
│ ├── s3-signature.ts # NEW — SigV4 algorithm (header + chunked stream)
│ └── storage.service.ts # EXTENDED — multipart-aware methods
└── providers/
└── storage/
├── base.provider.ts # EXTENDED interface
└── s3.provider.ts # implements new methods
```
### Responsibility Boundaries
| Concern | Owner | Rationale |
|---|---|---|
| SigV4 verification and op dispatch | `s3-sigv4` middleware + router | Isolated from business logic, unit-testable. |
| S3 op semantics (Put/Get/List/…) | `commands/*.ts` (one per op) | Keeps each file small and single-purpose. |
| Physical object IO | `S3StorageProvider` | Only place that instantiates `S3Client`. |
| Object metadata read/write | `StorageService` | Shared with REST path; prevents format drift. |
| XML serialization | `xml.ts` | Consistent formatting across operations. |
### Body Parser Ordering
`server.ts` currently does:
```ts
app.use(express.json({ limit: '100mb' }));
app.use('/api', apiRouter);
```
This must change to mount the S3 router **before** any body-consuming middleware:
```ts
app.use('/storage/v1/s3', s3GatewayRouter); // streaming-aware, never calls express.json()
app.use(express.json({ limit: '100mb' }));
app.use('/api', apiRouter);
```
The S3 router handles `req` as a Readable stream directly and never consumes the body via `bodyParser` / `multer`.
## Access Key Model
### Database
Migration `033_create-s3-access-keys.sql`:
```sql
CREATE TABLE IF NOT EXISTS storage.s3_access_keys (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
access_key_id TEXT NOT NULL UNIQUE,
secret_access_key_encrypted TEXT NOT NULL,
description TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_used_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_s3_access_keys_last_used_at
ON storage.s3_access_keys (last_used_at);
```
Table lives in the `storage` schema alongside `storage.buckets`, `storage.objects`, `storage.config`. No `app_key` column (one project per process). No `updated_at` (keys are immutable — modify by destroy + recreate). No foreign key to users (keys are project-scoped, not user-scoped).
### Credential Format
| Field | Format | Length | Example |
|---|---|---|---|
| `access_key_id` | `INSF` prefix + 16 upper-case alphanumeric | 20 | `INSFABC123DEF456GH78` |
| `secret_access_key` | 40 base64url chars (from 30 random bytes, stripped of padding) | 40 | `x7K2-a_pL9qRs4N8vYzWcE1fH5gJ3mUtBoD6ViXk` |
Lengths match AWS conventions (20 / 40) to avoid SDK validation errors. Both fields are generated with `crypto.randomBytes` and then formatted. Base64url alphabet (`A–Z a–z 0–9 - _`) is chosen because AWS SigV4 puts the access key id into the canonical request as-is; any character outside `unreserved` would need URI-encoding handling in our own verifier.
### Secret Storage
The `secret_access_key_encrypted` column stores `EncryptionManager.encrypt()` output (AES-256-GCM), reversible because SigV4 verification requires the raw secret to recompute HMAC signatures. The encryption key comes from the existing `ENCRYPTION_KEY` env var. Rotating `ENCRYPTION_KEY` invalidates all stored secrets and requires users to recreate keys — documented behaviour.
### Constraints
- Hard cap of **50 keys per project**. `S3AccessKeyService.create` performs the count check and the insert inside a single SERIALIZABLE transaction, so concurrent creations cannot both pass the check and overshoot the cap. Over-limit returns `400 S3_ACCESS_KEY_LIMIT_EXCEEDED`.
- Keys are immutable. No update endpoint.
- Plaintext secret is returned **only once** in the creation response. Subsequent `GET` calls never return the secret.
- `last_used_at` updated asynchronously on each successful SigV4 verification via `setImmediate` (fire-and-forget, errors swallowed to avoid blocking the request).
### Management API
Mounted under the existing `storageRouter`, protected by `verifyAdmin`:
| Method | Path | Purpose |
|---|---|---|
| `POST` | `/api/storage/s3/access-keys` | Create; body `{ description? }`; response contains plaintext secret (once). |
| `GET` | `/api/storage/s3/access-keys` | List (no secrets). |
| `DELETE` | `/api/storage/s3/access-keys/:id` | Delete; LRU cache invalidated synchronously. |
Audit events: `CREATE_S3_ACCESS_KEY`, `DELETE_S3_ACCESS_KEY`.
### Runtime Cache
SigV4 verification is on the hot path (every S3 request). Pure DB lookup would be a bottleneck.
- In-process LRU keyed by `access_key_id`, value `{ secret_plaintext, id }`.
- Size 1024 entries (well above the 50-key cap, so effectively never evicts) with 5-minute TTL (bounds staleness after delete).
- Synchronous invalidation on delete.
- Implementation: `lru-cache` npm package. (Already a transitive dep of AWS SDK; if not in lockfile, added directly.)
### Authorization Semantics
A valid S3 credential in its project grants:
- Read and write on **all** buckets, ignoring `public`/`private` flags (those apply to anonymous access; credential holders are not anonymous).
- `CreateBucket` / `DeleteBucket`.
- No cross-project access possible — physical process isolation enforces this.
S3-protocol uploads record the originating access key via the new `s3_access_key_id` column on `storage.objects` (see Schema Extensions below). `uploaded_by` is `NULL` on S3-protocol uploads — we do not overload that UUID column with a string marker.
## Request Handling Pipeline
### Express Mount Order
Already covered above: `/storage/v1/s3` router mounts before `express.json()`.
### SigV4 Header-Signed Requests
AWS SigV4 (the short form):
1. Parse `Authorization: AWS4-HMAC-SHA256 Credential=<ak>/<date>/<region>/s3/aws4_request, SignedHeaders=<sorted;list>, Signature=<sig>`.
2. Look up credential (cache, then DB) → plaintext secret.
3. Build **Canonical Request**. The `<URI-encoded path>` MUST be derived from the raw, percent-encoded request path as the client sent it (e.g. `req.originalUrl` in Express), **not** a URL-decoded representation — otherwise object keys containing percent-encoded characters produce signature mismatches.
```text
<METHOD>\n
<URI-encoded path>\n
<canonical query>\n
<canonical headers>\n
\n
<signed headers list>\n
<x-amz-content-sha256 value>
```
4. Build **String-to-Sign**: `AWS4-HMAC-SHA256\n<datetime>\n<scope>\nSHA256(canonical request)`.
5. Derive signing key: `HMAC(HMAC(HMAC(HMAC("AWS4"+secret, date), region), "s3"), "aws4_request")`.
6. Compute `HMAC(signingKey, stringToSign)`, compare to client signature with `crypto.timingSafeEqual`. Length mismatch → return `SignatureDoesNotMatch` without invoking timing-safe comparison.
7. For requests with a hashed body (not `UNSIGNED-PAYLOAD`, not `STREAMING-*`), additionally verify `SHA-256(body) === x-amz-content-sha256` header.
### SigV4 Streaming (Chunked) Requests
Triggered by `x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD`. Body structure:
```text
<chunk1-size-hex>;chunk-signature=<sig1>\r\n
<chunk1-payload>\r\n
<chunk2-size-hex>;chunk-signature=<sig2>\r\n
<chunk2-payload>\r\n
...
0;chunk-signature=<final-sig>\r\n
\r\n
```
Verification is streamed, not buffered:
1. Verify header signature as above; the canonical-request body hash is the literal `STREAMING-AWS4-HMAC-SHA256-PAYLOAD`. The computed header signature is the `seed_signature` for chunk verification.
2. Pipe `req` through a `ChunkSignatureV4Parser` Transform with states `HEADER → DATA → CRLF → HEADER → …`:
- Parse chunk header; record declared chunk size and signature.
- Stream incoming bytes into a running `crypto.createHash('sha256')` while pushing them downstream.
- At end of DATA: compose chunk string-to-sign = `AWS4-HMAC-SHA256-PAYLOAD\n<datetime>\n<scope>\n<previous_signature>\n<SHA256("")>\n<SHA256(chunk_payload)>`, HMAC-verify, compare to declared signature.
- Chain: the first chunk's `previous_signature` is the header's `seed_signature`; each subsequent chunk uses the previous chunk's signature.
3. Downstream (the `Body` parameter of `PutObjectCommand` / `UploadPartCommand`) receives only verified payload bytes.
4. Any chunk signature mismatch: destroy the stream, return `SignatureDoesNotMatch` (403) XML error.
A `SegmentedBufferQueue` keeps header parsing memory bounded; chunk header length is capped at 128 bytes to prevent pathological allocations.
### Operation Dispatch
S3 keys off (method, path shape, query string, and sometimes headers). The dispatcher maps to command handlers:
| Method | Path | Query / Header | Op |
|---|---|---|---|
| `PUT` | `/{bucket}/{key}` | — | PutObject |
| `PUT` | `/{bucket}/{key}` | `?partNumber=N&uploadId=X` | UploadPart |
| `PUT` | `/{bucket}/{key}` | header `x-amz-copy-source` | CopyObject |
| `POST` | `/{bucket}/{key}` | `?uploads` | CreateMultipartUpload |
| `POST` | `/{bucket}/{key}` | `?uploadId=X` | CompleteMultipartUpload |
| `POST` | `/{bucket}` | `?delete` | DeleteObjects |
| `DELETE` | `/{bucket}/{key}` | — | DeleteObject |
| `DELETE` | `/{bucket}/{key}` | `?uploadId=X` | AbortMultipartUpload |
| `GET` | `/{bucket}/{key}` | — | GetObject |
| `GET` | `/{bucket}/{key}` | `?uploadId=X` | ListParts |
| `GET` | `/{bucket}` | — / `?list-type=2` | ListObjectsV2 |
| `GET` | `/` | — | ListBuckets |
| `HEAD` | `/{bucket}/{key}` | — | HeadObject |
| `HEAD` | `/{bucket}` | — | HeadBucket |
| `PUT` | `/{bucket}` | — | CreateBucket |
| `DELETE` | `/{bucket}` | — | DeleteBucket |
Implementation is a single `dispatch(req)` function, not Express's per-verb router, because the path shapes conflict (`/bucket/key` vs `/bucket` vs `/`).
### Path Parsing
- Express mounts at `/storage/v1/s3`, so `req.path` is `/{bucket}/{key}` or `/{bucket}/` or `/`.
- First segment → bucket name; remainder → object key.
- Bucket name validation reuses the existing regex `^[a-zA-Z0-9_-]+$`.
- Key validation rejects `..` and leading `/` (same rules as the REST layer).
### Clock Skew Protection
`X-Amz-Date` is compared to server time with a tolerance of **15 minutes** (AWS standard). Out-of-range returns `RequestTimeTooSkewed` before signature comparison.
## Provider Extensions & Metadata Sync
### `StorageProvider` Interface Additions
```ts
interface StorageProvider {
// ...existing methods kept
putObjectStream(
bucket: string,
key: string,
body: Readable,
opts: { contentType?: string; contentLength?: number }
): Promise<{ etag: string; size: number }>;
headObject(bucket: string, key: string): Promise<{
size: number;
etag: string;
contentType?: string;
lastModified: Date;
} | null>;
copyObject(
srcBucket: string, srcKey: string,
dstBucket: string, dstKey: string
): Promise<{ etag: string; lastModified: Date }>;
getObjectStream(
bucket: string,
key: string,
opts?: { range?: string }
): Promise<{
body: Readable;
size: number;
etag: string;
contentType?: string;
lastModified: Date;
}>;
createMultipartUpload(
bucket: string, key: string, opts: { contentType?: string }
): Promise<{ uploadId: string }>;
uploadPart(
bucket: string, key: string, uploadId: string,
partNumber: number, body: Readable, contentLength: number
): Promise<{ etag: string }>;
completeMultipartUpload(
bucket: string, key: string, uploadId: string,
parts: Array<{ partNumber: number; etag: string }>
): Promise<{ etag: string; size: number }>;
abortMultipartUpload(bucket: string, key: string, uploadId: string): Promise<void>;
listParts(
bucket: string, key: string, uploadId: string,
opts: { maxParts?: number; partNumberMarker?: number }
): Promise<{
parts: Array<{ partNumber: number; etag: string; size: number; lastModified: Date }>;
isTruncated: boolean;
nextPartNumberMarker?: number;
}>;
}
```
### LocalStorageProvider Behavior
All new methods throw:
```ts
throw new AppError(
'S3 protocol requires an S3 storage backend. Set AWS_S3_BUCKET (and optionally S3_ENDPOINT_URL for MinIO).',
501, ERROR_CODES.NOT_IMPLEMENTED
);
```
At startup, the gateway detects the active provider. If not `S3StorageProvider`, mount a stub router at `/storage/v1/s3` that returns a 501 S3 XML error for every request.
### S3StorageProvider Implementation
Each new method is a thin wrapper over an AWS SDK v3 command, reusing the existing `getS3Key(bucket, key)` helper that applies the `{appKey}/{bucket}/{key}` prefix. Multipart operations use real S3 multipart (uploadId is the real S3 uploadId); InsForge stores no multipart state in its own DB.
### Metadata Synchronization Table
| Op | S3 action | DB action |
|---|---|---|
| `PutObject` | `PutObjectCommand` (streaming) | `INSERT ... ON CONFLICT (bucket, key) DO UPDATE` |
| `GetObject` | `GetObjectCommand` stream | Read metadata from `storage.objects` |
| `HeadObject` | — | Read from `storage.objects` |
| `DeleteObject` | `DeleteObjectCommand` | `DELETE FROM storage.objects WHERE bucket=$1 AND key=$2` |
| `DeleteObjects` | Parallel `DeleteObjectCommand`s | Single `DELETE ... WHERE (bucket, key) IN (...)` |
| `CopyObject` | `CopyObjectCommand` (server-side) | `INSERT` destination row |
| `CreateMultipartUpload` | `CreateMultipartUploadCommand` | None |
| `UploadPart` | `UploadPartCommand` (streaming) | None |
| `CompleteMultipartUpload` | `CompleteMultipartUploadCommand` | `INSERT ... ON CONFLICT DO UPDATE` with final size + ETag |
| `AbortMultipartUpload` | `AbortMultipartUploadCommand` | None |
| `CreateBucket` | `S3StorageProvider.createBucket` | `INSERT INTO storage.buckets (name, public) VALUES ($1, false)` |
| `DeleteBucket` | `S3StorageProvider.deleteBucket` | Check empty first; `DELETE FROM storage.buckets ...` |
| `ListObjectsV2` | — | Query `storage.objects` |
| `ListBuckets` | — | Query `storage.buckets` |
| `HeadBucket` | — | Query `storage.buckets` |
List operations read from the DB, not live S3, because the DB is the source of truth for object metadata.
### Schema Extensions for `storage.objects`
Migration `034_extend-storage-objects-for-s3-protocol.sql`:
```sql
ALTER TABLE storage.objects
ADD COLUMN IF NOT EXISTS uploaded_via TEXT NOT NULL DEFAULT 'rest'
CHECK (uploaded_via IN ('rest', 's3', 'dashboard')),
ADD COLUMN IF NOT EXISTS s3_access_key_id TEXT,
ADD COLUMN IF NOT EXISTS etag TEXT;
```
- Existing REST/Dashboard upload paths write `uploaded_via='rest'` or `'dashboard'`, leave `s3_access_key_id` NULL.
- S3 gateway writes `uploaded_via='s3'`, `s3_access_key_id=<ak>`, `uploaded_by=NULL`.
- `etag` populated for all future uploads so HeadObject does not need to fall back to live S3.
- `uploaded_by` column stays; its type is unchanged. S3 writes use NULL there.
### ListObjectsV2 Implementation Notes
- Query params parsed: `prefix`, `delimiter`, `continuation-token`, `start-after`, `max-keys` (capped at 1000; default 1000).
- Base query: `SELECT key, size, mime_type, etag, uploaded_at FROM storage.objects WHERE bucket=$1 AND key LIKE $prefix||'%' [AND key > $start_after] ORDER BY key LIMIT $max_keys+1`.
- The `+1` detects truncation without a second query.
- `delimiter='/'`: common-prefix rollup done in application code after SELECT (SQL approach is brittle across edge cases).
- `continuation-token`: base64-encoded last returned key.
## Operation Scope (v1)
### Implemented
**Bucket-level (5):** `ListBuckets`, `CreateBucket`, `DeleteBucket`, `HeadBucket`, `ListObjectsV2`.
**Object-level (11):** `PutObject`, `GetObject`, `HeadObject`, `DeleteObject`, `DeleteObjects` (batch), `CopyObject`, `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListParts`.
**Stubs (2):** `GetBucketLocation` returns `<LocationConstraint>us-east-2</LocationConstraint>`; `GetBucketVersioning` returns `<VersioningConfiguration><Status>Disabled</Status></VersioningConfiguration>`. Both are commonly probed by SDKs on client init.
### Explicitly Rejected (return `NotImplemented` 501)
`GetBucketAcl`, `PutBucketAcl`, `GetBucketCors`, `PutBucketCors`, `GetObjectTagging`, `PutObjectTagging`, `GetObjectAcl`, `PutObjectAcl`, `UploadPartCopy`, and all versioning / lifecycle / replication / inventory endpoints.
### Bucket Name Rules
`CreateBucket` applies the existing InsForge regex `^[a-zA-Z0-9_-]+$`. This is looser than AWS's DNS-compatible rules (lowercase only, 3–63 chars, no underscores). Tradeoff: REST and S3 see the same rules, at the cost of occasional SDK-side warnings. Documented.
### Size Limits
- Single `PutObject`: 5 GB (AWS cap). Enforced via `Content-Length`; over-limit returns `EntityTooLarge` 400.
- `UploadPart`: 5 MB min (except last part), 5 GB max.
- Total multipart object: 5 TB (enforced by real S3, we pass-through the error).
- Optional env var `S3_PROTOCOL_MAX_OBJECT_SIZE_GB` can **lower** the per-object cap below 5 GB for a given deployment (abuse protection). It cannot raise the ceiling above the AWS single-PutObject max of 5 GB. Default: unset (i.e. the 5 GB cap applies). Does not relate to REST-layer `storage.config.max_file_size_mb`.
## Error Handling
All non-2xx responses return S3-format XML:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Resource>/my-bucket/photo.jpg</Resource>
<RequestId>...</RequestId>
</Error>
```
### Error Code Map
| Internal condition | S3 code | HTTP |
|---|---|---|
| Signature mismatch | `SignatureDoesNotMatch` | 403 |
| Access key missing or deleted | `InvalidAccessKeyId` | 403 |
| Clock skew > 15 min | `RequestTimeTooSkewed` | 403 |
| Malformed Authorization header | `AuthorizationHeaderMalformed` | 400 |
| Bucket missing | `NoSuchBucket` | 404 |
| Object missing | `NoSuchKey` | 404 |
| Bucket already exists | `BucketAlreadyOwnedByYou` | 409 |
| Non-empty on DeleteBucket | `BucketNotEmpty` | 409 |
| Invalid bucket name | `InvalidBucketName` | 400 |
| Body exceeds single-object limit | `EntityTooLarge` | 400 |
| Multipart part too small | `EntityTooSmall` | 400 |
| Unsupported operation | `NotImplemented` | 501 |
| Anything else | `InternalError` | 500 |
Shared helper `sendS3Error(res, code, message, requestId)` in `s3-gateway/errors.ts`. `RequestId` = `crypto.randomUUID()` generated per request and echoed into audit / access logs.
## Security
- Secret exposure surface is exactly one response (create). Secrets are never logged; request/response logging scrubs the `secret_access_key_encrypted` column. LRU cache lives only in memory; process restart clears it.
- Rate limiting: the S3 path is excluded from `express-rate-limit` for request-level throttling (because streaming uploads would misfire). Instead, per-access-key throttling is applied in the SigV4 middleware.
- Audit: `CREATE_S3_ACCESS_KEY` / `DELETE_S3_ACCESS_KEY` go to `AuditService`. Data-plane S3 operations are not audited (volume is prohibitive); access logs are sufficient.
- Encryption key rotation: rotating `ENCRYPTION_KEY` invalidates stored secrets. Users must recreate keys. Behaviour documented.
## Testing Strategy
Testing is co-equal with implementation — the quality gate for "S3-compatible".
1. **SigV4 unit tests** using AWS's published `aws4_testsuite` fixtures (canonical request / string-to-sign / signature triples hardcoded, compared to our implementation).
2. **Integration tests (CI)**: run the backend against MinIO via docker-compose. Exercise the full op surface via `@aws-sdk/client-s3`: Put/Get/Head small + large + multipart, ListObjectsV2 with pagination / prefix / delimiter, Copy, Delete, DeleteObjects.
3. **CLI black-box tests (manual at minimum, scripted in CI where feasible)**:
- `aws s3 cp` — 100 KB, 100 MB, 1 GB (forces multipart).
- `aws s3 sync` — both directions.
- `aws s3 rm --recursive`.
- `rclone copy` and `rclone sync`.
4. **Negative tests**: wrong signature, expired date, deleted access key, non-existent bucket/key, DeleteBucket on non-empty bucket.
Tests live under `backend/tests/s3-gateway/`.
## Risk Register
| Risk | Severity | Mitigation |
|---|---|---|
| SigV4 implementation bug breaks some SDKs | High | Port-by-line from Supabase's `signature-v4.ts`; compliance test suite covers aws-cli / aws-sdk-js / boto3 / rclone. |
| Streaming chunk parser state-machine bug corrupts uploads or leaks memory | High | Unit tests cover chunk-size boundaries (exact, multi-packet, equal to highWaterMark); `byte-limit-stream` caps per-request size. |
| `storage.objects` and real S3 drift (S3 write succeeds, DB write fails) | Medium | Write S3 first, then DB; on DB failure, compensating S3 delete; periodic reconcile job. |
| Mount order regression swallows raw body via `express.json()` | Medium | Integration test that streams a 10 MB body through to verify chunks arrive; code-review checklist. |
| Hot access key causes DB contention | Medium | LRU cache + single-flight mutex on cache miss. |
| `crypto.timingSafeEqual` throws on length mismatch | Low | Length check before call; mismatched length returns `SignatureDoesNotMatch`. |
| Client with wrong region signature slips through | Low | Validate `Credential` scope's region field equals `us-east-2`. |
| Noise from unsupported presigned-URL failures | Low | Docs explicitly redirect users to REST `upload-strategy`. |
| Credential enumeration / DoS | Low | 50-key hard cap; per-access-key rate limit. |
## Open Questions
These are parked; they don't block the design but should be resolved during or before implementation.
1. **Dashboard UI** for S3 access key management. Scope-excluded from this design; tracked as a separate spec / PR.
2. **`storage.objects.etag` backfill.** Existing rows need `etag` populated (either lazily on first HeadObject or via migration backfill). Lazy is simpler, migration is more consistent — decide during implementation.
3. **MinIO self-hosted support.** The design supports it by construction; should we document and recommend it as the self-hosted S3-protocol path? (Recommend: yes.)
4. **Updating existing REST/Dashboard upload paths** to write `uploaded_via='rest' | 'dashboard'`. Required to keep the new column meaningful. Non-breaking (has DEFAULT).
5. **Cross-project isolation** rests on the "one process = one app_key" deployment assumption. If the platform ever consolidates processes, this design needs revisiting. Add a code comment flagging the assumption.
6. **Host-based routing** (`{appkey}.{region}.insforge.app`) is an infrastructure-layer concern (ingress / DNS / ALB), not backend code. The design assumes it works; ingress config is out of scope for this spec.
7. **Session-token auth (user-JWT-scoped S3 access)** — Supabase supports a third credential shape `{accessKeyId: project_ref, secretAccessKey: anonKey, sessionToken: <user JWT>}` that lets S3 operations respect per-user permissions via Postgres RLS. Deliberately cut from v1:
- The `storage.s3_access_keys` path covers the main server-side use cases (CI, scripts, backup tooling, rclone).
- InsForge does not use Postgres RLS today; replicating Supabase's "DB filters by JWT" model would mean building an application-layer user-scoping policy on every S3 handler — a separate design problem with its own scope.
- Browser direct uploads are already served by `POST /api/storage/buckets/:bucket/upload-strategy`.
If added later, the shape should be: detect `X-Amz-Security-Token` in the SigV4 middleware, verify via `TokenManager.verifyToken()`, attach the user identity to `S3AuthContext`, and introduce a handler-layer policy (or bucket-visibility + ownership check) gating every operation. Needs its own spec.
## Tradeoffs Summary
**Pros**
- Aligned with Supabase's open-source protocol path; documentation and community experience transfer.
- Shared namespace means Dashboard and S3 protocol see the same objects — minimal user cognitive load.
- Single-tenant per process avoids multi-tenant routing work.
- Reuses existing infrastructure (`EncryptionManager`, `StorageService`, `S3StorageProvider`, migration framework).
**Cons**
- Implementation size is non-trivial (~2500–3500 lines of new code + tests).
- SigV4 chunked-payload parser is precision protocol work — bugs show up as "one SDK works, another doesn't".
- Presigned URLs cut from v1 for scope; adding them later touches the data plane.
- LocalStorageProvider users don't get S3 protocol; they must run MinIO.
.internal/docs/specs/2026-04-29-backend-branching-oss-design.md
+121
View File
@@ -0,0 +1,121 @@
# Backend Branching (OSS) — Design
**Date:** 2026-04-29
**Status:** Draft
**Source PRD:** [insforge-cloud-backend Backend-Branching.md](../../../../insforge-cloud-backend/Backend-Branching.md)
**Cloud-backend spec:** [2026-04-29-backend-branching-cloud-design.md](../../../../insforge-cloud-backend/docs/superpowers/specs/2026-04-29-backend-branching-cloud-design.md)
This spec covers the **OSS slice** of backend branching v1: read-only fallback from a branch project's storage to its parent project's S3 directory.
## Problem
When a branch project is created from a parent, copying every storage object would be slow and wasteful (parent may have GB of files). The cloud-backend spec dictates: branch's S3 directory starts empty; reads of parent objects must transparently succeed via fallback. Branch writes go to branch's directory only; parent files are never modified from a branch.
## Goals (v1)
1. The OSS storage server, when started in "branch mode" (with an injected `PARENT_APP_KEY` env var), reads object data from parent's S3 path when the branch's path returns nothing.
2. Fallback is **read-only**. Writes (PUT/DELETE/multipart-upload) always target the branch's own appkey path.
3. Existing RLS / bucket-visibility checks continue to apply unchanged.
4. No HTTP API additions. Behavior change is server-internal.
5. Local-storage provider does **not** support fallback (cloud-only). Local installs are not branched.
## Non-Goals (v1)
- Cross-project fallback for non-branch projects.
- Write-through: copying a parent object into branch when first written to (CoW). Branches always start with the empty file set in their own namespace; modifications create new objects.
- Deletion markers: branch cannot "hide" a parent file. Deleting from a branch only deletes the branch's own copy (which doesn't exist for inherited files), so the parent file remains visible. Documented as v1 limitation.
- Schema-only mode files: when the branch was created with `mode='schema-only'`, the cloud-backend truncates `storage.objects`, so RLS/metadata returns 404 for parent files. This matches "fresh start" semantics — fallback is only effective for `mode='full'` branches.
## Current State Summary
- OSS storage server: TypeScript / Express monorepo at `backend/`. See [backend/src/server.ts:1-80](../../../backend/src/server.ts).
- S3 path layout: `{appKey}/{bucketName}/{key}` in a single shared bucket (`AWS_S3_BUCKET`). Code in [backend/src/providers/storage/s3.provider.ts:91-93](../../../backend/src/providers/storage/s3.provider.ts).
- `appKey` loaded from `APP_KEY` env var at server startup. Singleton pattern in [backend/src/services/storage/storage.service.ts:29-45](../../../backend/src/services/storage/storage.service.ts).
- HTTP read endpoint: `GET /api/storage/buckets/:bucketName/objects/*` ([backend/src/api/routes/storage/index.routes.ts:398-459](../../../backend/src/api/routes/storage/index.routes.ts)).
- `StorageService.getObject` returns `null` if metadata absent or S3 returns nothing — these are the natural interception points for fallback ([backend/src/services/storage/storage.service.ts:201-238](../../../backend/src/services/storage/storage.service.ts)).
- Presigned URL flow: `getDownloadStrategy` calls `getSignedUrl` from `@aws-sdk/s3-request-presigner` (S3 sigV4 GET against `{appKey}/{bucket}/{key}`).
## Design
### Configuration
New env var:
- `PARENT_APP_KEY` (optional). When set, the server runs in "branch mode" and falls back to this appkey for object reads.
The cloud-backend injects this at branch container startup (alongside the existing `APP_KEY`).
### S3 Provider Changes
Extend `S3StorageProvider` to optionally hold a `parentAppKey`:
```typescript
constructor(
private s3Bucket: string,
private appKey: string,
private region: string = 'us-east-2',
private parentAppKey?: string,
) { ... }
private getS3Key(bucket: string, key: string): string {
return `${this.appKey}/${bucket}/${key}`;
}
private getParentS3Key(bucket: string, key: string): string | null {
return this.parentAppKey ? `${this.parentAppKey}/${bucket}/${key}` : null;
}
```
Read methods (`getObject`, `headObject`, `getObjectStream`, `getDownloadStrategy`) get a single new helper:
```typescript
private async withFallback<T>(primary: () => Promise<T | null>, parent: () => Promise<T | null>): Promise<T | null> {
const a = await primary();
if (a !== null) return a;
if (!this.parentAppKey) return null;
return parent();
}
```
Each read becomes:
- Attempt with `getS3Key(bucket, key)`. On null/404 → if `parentAppKey` set, attempt with `getParentS3Key(bucket, key)`.
Write methods (`putObject`, `deleteObject`, `createMultipartUpload`, etc.) do **not** call the fallback — they always target `getS3Key`.
### Presigned URLs
`getDownloadStrategy(bucket, key)` is the only read path that doesn't actually fetch the object — it just constructs a signed URL. For fallback to work for presigned URLs, we must do a HEAD against the branch path first; if 404, sign against the parent path. This adds a HEAD round-trip but is required for correctness. Non-404 HEAD failures (network, IAM, throttling) are caught and logged: URL generation falls back to the branch key rather than aborting the whole call. If the object truly only lives on the parent path, the signed URL will 404 at download time — degraded but recoverable, and a strict improvement over failing the entire request.
### Service Layer
`StorageService.getObject` and `objectIsVisible` rely on a metadata row in `storage.objects`. For `mode='full'` branches, that row exists (copied via pg_dump). The provider does the actual S3 fallback; service layer needs no change.
For `mode='schema-only'` branches, `storage.objects` is truncated by the cloud-backend after restore. Result: branch users get 404 from RLS lookup, never reaching the provider. This is the expected behavior — the storage feature simply doesn't apply.
### Failure Modes
- `parentAppKey` set but parent's directory was deleted (branch outlived parent — shouldn't happen given lifecycle cascade): primary 404 + parent 404 → final 404. Same as today.
- Parent exists but specific key missing on parent too: same 404. No new error path.
- IAM error reading parent path (cross-prefix permission denied): non-404 errors are propagated by the read helpers (`tryHeadObject`, `tryGetObjectStream`, `tryGetObject`); only true 404s are treated as not-found and trigger parent fallback. The public `getObject` wraps `withFallback` and surfaces any error as `null` to preserve the prior service-layer contract, but parent fallback is no longer triggered by transient/IAM failures. The IAM role for the EC2 already has access to the entire `insforge-storage` bucket per the existing single-bucket model; no permissions change required.
### Compatibility
- Local storage provider: ignored. `LocalStorageProvider` never receives `parentAppKey`. The constructor signature can be extended for symmetry but the fallback path no-ops.
- Existing non-branch projects: `PARENT_APP_KEY` is unset → fallback path never executes → zero behavior change.
## API
No new HTTP endpoints. Existing endpoints' behavior changes only for branches with `PARENT_APP_KEY` set, and only for read paths.
## Acceptance
A branch container started with `APP_KEY=branchkey` and `PARENT_APP_KEY=parentkey`:
1. `GET /api/storage/buckets/foo/objects/path/to/file.jpg` returns the file from `parentkey/foo/path/to/file.jpg` if `branchkey/foo/path/to/file.jpg` doesn't exist (assuming branch DB has the metadata row).
2. `PUT /api/storage/buckets/foo/objects/path/to/file.jpg` writes to `branchkey/foo/path/to/file.jpg` only.
3. After (2), step (1) returns the new branch-local file (branch path takes priority).
4. `DELETE /api/storage/buckets/foo/objects/path/to/file.jpg` (after 2) removes the branch's copy. Subsequent GET falls back to parent file again.
5. RLS still gates: a user without read access to the bucket gets 404, regardless of fallback.
## Open Questions / TBD
1. **Deletion markers (post-v1).** A branch may want to "hide" a parent file. Requires a sentinel object or a row in branch's metadata. Not in v1.
2. **CoW write-through.** When a branch writes to `path/to/file.jpg` after reading the parent's version, should we copy on first write? Today writing creates a fresh file at the branch path; reads naturally see the new file. No copy needed unless we want diff/merge of file contents (not in product scope).
3. **Schema-only branch + storage**. Confirm with cloud-backend team: should schema-only mode skip truncating `storage.objects`, so storage fallback works in that mode too? Current spec assumes truncate (matching "fresh start"). Worth a one-line check.
.internal/docs/specs/2026-05-24-posthog-analytics-layout-redesign-design.md
+359
View File
@@ -0,0 +1,359 @@
# PostHog Analytics — Layout Redesign Design
**Status**: Draft
**Date**: 2026-05-24
**Repo**: InsForge (OSS dashboard)
**Scope**: `packages/dashboard/src/features/analytics/*` + sidebar nav + routing
**Design refs** (`Figma`):
- Disconnected overview — `3177-62515`
- Traffic — `3174-54062`, `3177-59015`, `3177-59464`
- User Retention — `3177-54743`
- Session Replay — `3181-10216`
- Settings (Analytics Config modal) — `3190-68392`, `3190-68947`
---
## Problem
`AnalyticsPage` is a single stacked page. Disconnected → centered `EmptyConnectPanel`. Connected → top action bar + ConnectStatusBar + Setup-with-Prompt + ApiKeyCard + KPI + 3 Breakdowns + Retention card + Recent Replays card, all in one vertical scroll.
The redesign splits the feature into a **secondary-sidebar pattern** matching how Authentication / Payments / Realtime / Deployments are already organized in the dashboard:
- Left secondary sidebar titled **Analytics** with sub-items **Traffic / User Retention / Session Replay**
- A gear icon in the sidebar header opens an **Analytics Config** modal (connection info + setup prompt + disconnect)
- When PostHog is **not connected**: sub-items are disabled, the gear is disabled, the main area shows a single empty-connect CTA panel
- When connected: each sub-item routes to a focused dashboard page
Goal: implement that layout using existing InsForge components (`FeatureSidebar`, `MenuDialog`, `Button`, etc.) and existing analytics building blocks (`KpiSectionWithTrend`, `BreakdownPanel`, `RetentionCard`, `RecentReplaysCard`). No hard-coded colors / spacing — Tailwind semantic classes + tokens only.
---
## Out of Scope
- New data fetching / new API endpoints — sub-pages reuse existing hooks (`useWebOverview`, `useTrend`, `useRetention`, `useRecordings`, `usePosthogConnection`).
- Changing the Connect / OAuth flow itself — `onConnectPosthog` from `useDashboardHost` stays the contract.
- Self-host parity — `isCloudHosting` gate in `AppRoutes` stays as-is; this feature is cloud-only.
- Feature flagging / phased roll-out — staging handled via OSS test tags.
---
## Architecture
### Routes (`router/AppRoutes.tsx`)
Today (single route):
```tsx
{isCloudHosting && <Route path="/dashboard/analytics" element={<AnalyticsPage />} />}
```
After (layout + nested routes, mirrors `/dashboard/authentication` and `/dashboard/payments`):
```tsx
{isCloudHosting && (
<Route path="/dashboard/analytics" element={<AnalyticsLayout />}>
<Route index element={<Navigate to="traffic" replace />} />
<Route path="traffic" element={<TrafficPage />} />
<Route path="retention" element={<RetentionPage />} />
<Route path="session-replay" element={<SessionReplayPage />} />
</Route>
)}
```
The bare `/dashboard/analytics` always redirects to `/traffic` (matches `payments → catalog`, `realtime → channels`). Disconnected state is handled inside `AnalyticsLayout`, not by the router — see below.
### Components
Mirrors `features/auth`, `features/payments`, `features/realtime`, `features/deployments` — Layout / Sidebar / feature-level dialog modals all live under `components/`, sub-pages under `pages/`, default-exported Layout imported by `AppRoutes`:
```
features/analytics/
├── components/
│ ├── AnalyticsLayout.tsx NEW — outer shell (sidebar + Outlet | EmptyConnectPanel). DEFAULT EXPORT.
│ ├── AnalyticsSidebar.tsx NEW — wraps FeatureSidebar, owns Settings dialog open state. Named export.
│ ├── AnalyticsConfigDialog.tsx NEW — MenuDialog: connection info + setup prompt + disconnect. Named export.
│ └── posthog/ (existing — kept)
│ ├── EmptyConnectPanel.tsx (reused as-is for disconnected main area)
│ ├── ApiKeyCard.tsx (content folded into AnalyticsConfigDialog; file deleted)
│ ├── ConnectStatusBar.tsx (content folded into AnalyticsConfigDialog; file deleted)
│ ├── DisconnectDialog.tsx (kept — triggered from inside AnalyticsConfigDialog)
│ ├── KpiSectionWithTrend.tsx (used by TrafficPage)
│ ├── BreakdownPanel.tsx (used by TrafficPage)
│ ├── RetentionCard.tsx (used by RetentionPage, full-width)
│ ├── RecentReplaysCard.tsx (used by SessionReplayPage)
│ ├── ReplayModal.tsx (unchanged — opened from SessionReplayPage)
│ └── TimeRangeSelector.tsx (rendered in each sub-page's header)
├── pages/ NEW directory (matches features/auth/pages)
│ ├── TrafficPage.tsx NEW — named export
│ ├── RetentionPage.tsx NEW — named export
│ └── SessionReplayPage.tsx NEW — named export
├── AnalyticsPage.tsx DELETE (logic split across Layout + pages)
├── index.ts UPDATE — drop `export { AnalyticsPage }` (router imports layout directly, matching realtime/payments which have no barrel)
├── context/TimeRangeContext.tsx (kept — provider moves into AnalyticsLayout)
├── hooks/ (kept — no changes; useRecordings may grow limit/offset parameters)
├── lib/ (kept)
└── services/ (kept)
```
`AppRoutes.tsx` import follows the existing convention:
```ts
import AnalyticsLayout from '#features/analytics/components/AnalyticsLayout';
import { TrafficPage } from '#features/analytics/pages/TrafficPage';
import { RetentionPage } from '#features/analytics/pages/RetentionPage';
import { SessionReplayPage } from '#features/analytics/pages/SessionReplayPage';
```
### Shared component change
`components/FeatureSidebar.tsx` needs a `disabled` flag on items. Today `FeatureSidebarListItem` only has `id / label / href / sectionEnd / onClick`. Add:
```ts
export interface FeatureSidebarListItem {
id: string;
label: string;
href?: string;
sectionEnd?: boolean;
onClick?: () => void;
disabled?: boolean; // NEW
}
```
When `disabled === true`:
- The row renders as a `<div>` (not `<Link>`), so clicking is a no-op and there's no router navigation.
- `aria-disabled="true"` on the container.
- Visual: `text-muted-foreground/50 cursor-not-allowed` and no hover treatment (no `hover:bg-alpha-4` / `hover:text-foreground`).
- The hidden `useMatch` result is ignored — disabled items never appear "selected".
`headerButtons` already supports `disabled`, which is what we use for the gear-icon Settings button.
Auth / Payments / Realtime / Deployments sidebars never pass `disabled` today, so adding the optional field is backwards-compatible.
---
## Behavior
### `AnalyticsLayout`
```tsx
export default function AnalyticsLayout() {
const conn = usePosthogConnection();
const { projectId, isLoading: projectIdLoading, error: projectIdError } = useProjectId();
const connected = !conn.isLoading && !conn.isError && !!conn.data;
return (
<TimeRangeProvider>
<div className="flex h-full min-h-0 overflow-hidden bg-[rgb(var(--semantic-1))]">
<AnalyticsSidebar connected={connected} />
<div className="min-w-0 flex-1 overflow-hidden">
{conn.isLoading || projectIdLoading ? (
<LoadingState />
) : conn.isError ? (
<ErrorState message="Failed to load PostHog connection." />
) : !conn.data ? (
projectIdError || !projectId
? <ErrorState message="Failed to load project ID." />
: <DisconnectedMain projectId={projectId} />
) : (
<Outlet />
)}
</div>
</div>
</TimeRangeProvider>
);
}
```
- `LoadingState` / `ErrorState` are existing `#components` exports.
- `DisconnectedMain` is a thin wrapper that centers `EmptyConnectPanel` with appropriate page padding — same CTA, same copy as today (`Connect PostHog`, `One-click setup of a PostHog project for product analytics.`).
- When disconnected, `<Outlet />` is not rendered — so even direct navigation to `/dashboard/analytics/retention` still shows the empty-connect panel. URL stays correct; on connect, React-Query invalidation flips `connected` to `true` and the sub-page mounts.
### `AnalyticsSidebar`
```tsx
const ITEMS = (connected: boolean): FeatureSidebarListItem[] => [
{ id: 'traffic', label: 'Traffic', href: '/dashboard/analytics/traffic', disabled: !connected },
{ id: 'retention', label: 'User Retention', href: '/dashboard/analytics/retention', disabled: !connected },
{ id: 'session-replay', label: 'Session Replay', href: '/dashboard/analytics/session-replay', disabled: !connected },
];
export function AnalyticsSidebar({ connected }: { connected: boolean }) {
const [settingsOpen, setSettingsOpen] = useState(false);
return (
<>
<FeatureSidebar
title="Analytics"
items={ITEMS(connected)}
headerButtons={[{
id: 'analytics-settings',
label: 'Analytics Config',
icon: Settings,
onClick: () => setSettingsOpen(true),
disabled: !connected,
}]}
/>
<AnalyticsConfigDialog open={settingsOpen} onOpenChange={setSettingsOpen} />
</>
);
}
```
Pattern is identical to `AuthenticationSidebar`. No new sidebar styling — disabled visual lives inside `FeatureSidebar`.
### `AnalyticsConfigDialog`
Built on the same `MenuDialog` primitives `AuthSettingsMenuDialog` uses. design node `3190-68392` shows the modal opened against the disconnected layout (rare — gear is disabled when disconnected, so this is more of a design reference), and `3190-68947` shows it against the connected layout. Both share the same modal structure — one side-nav section, `Connection` content on the right — so a single `AnalyticsConfigDialog` covers both nodes. We keep `MenuDialogSideNav` for visual parity even though there's only one section today:
```tsx
<MenuDialog open={open} onOpenChange={onOpenChange}>
<MenuDialogContent>
<MenuDialogSideNav>
<MenuDialogSideNavHeader>
<MenuDialogSideNavTitle>Analytics Config</MenuDialogSideNavTitle>
</MenuDialogSideNavHeader>
<MenuDialogNav>
<MenuDialogNavList>
<MenuDialogNavItem active>Connection</MenuDialogNavItem>
</MenuDialogNavList>
</MenuDialogNav>
</MenuDialogSideNav>
<MenuDialogMain>
<MenuDialogHeader>
<MenuDialogTitle>Connection</MenuDialogTitle>
<MenuDialogCloseButton />
</MenuDialogHeader>
<MenuDialogBody>
{/* 1. ConnectStatusBar (compact form) */}
{/* 2. Read-only Input fields: API host, Project ID, API key (with show/hide) */}
{/* 3. Setup-with-Prompt block (existing inline copy, no new shared component per memory) */}
</MenuDialogBody>
<MenuDialogFooter>
<Button variant="ghost" onClick={() => setDisconnecting(true)}>Disconnect</Button>
</MenuDialogFooter>
</MenuDialogMain>
</MenuDialogContent>
</MenuDialog>
```
`Disconnect` opens the existing `DisconnectDialog` (modal stacking is fine — `AuthSettingsMenuDialog` already nests confirm modals the same way).
Fields:
- **API host** — `Input` (read-only), copy button
- **Project ID** — `Input` (read-only), copy button
- **API key** — `Input` (read-only, type=password by default), show/hide toggle, copy button. This is the existing `ApiKeyCard` content lifted into the dialog row layout.
The "Setup with Prompt" block in current `AnalyticsPage.tsx` (lines 121–137) is placed inline verbatim into `MenuDialogBody` — per `[[feedback_inline_prompt_block_pattern]]`, do not extract it into a shared component.
### Sub-pages
Each sub-page renders a page header (title + `TimeRangeSelector` where the underlying data hook respects the range) + its content. No outer `<h1>Analytics</h1>` (the sidebar title already says it).
**`TrafficPage`** — design node `3174-54062 / 3177-59015 / 3177-59464`:
```
[ header: Traffic [ TimeRangeSelector ] ]
[ KpiSectionWithTrend enabled ]
[ grid: BreakdownPanel(Page) | BreakdownPanel(Country) | BreakdownPanel(DeviceType) ]
```
Keep the existing `grid grid-cols-1 md:grid-cols-3 gap-3` from today's `AnalyticsPage` for the 3 breakdown panels — the design shows the row of 3 side-by-side at standard widths. KPI section spans the full width above.
Node `3174-54062` is the connect-success toast state (`Analytics setup succeeded`) — this surface is rendered by the global toast (`useToast`), already wired in `AnalyticsPage` today. We move that `subscribePosthogConnectionStatus` `useEffect` into `AnalyticsLayout` so the toast still appears when the user is on any sub-route during connect completion.
**`RetentionPage`** — design node `3177-54743`:
```
[ header: User Retention [ TimeRangeSelector ] ]
[ RetentionCard enabled, full-width ]
```
`RetentionCard` currently sits inside a multi-section scroll — moving it to its own page gives it the full content width per the design.
**`SessionReplayPage`** — design node `3181-10216`:
```
[ header: Session Replay ]
[ RecentReplaysCard enabled, full-width, paginated ]
```
Note: no `TimeRangeSelector` here — `useRecordings` only takes `limit` and doesn't read the time-range context, so showing the selector would be misleading. If the recordings backend later grows time-range filtering, wire `useRecordings` to `TimeRangeContext` and add the selector back.
The design shows pagination at the bottom of the list. `RecentReplaysCard` today returns ~N recent replays; this redesign asks it to paginate. Use `Pagination` from `@insforge/ui`. Pagination logic is a small extension to `useRecordings` (`limit` + `offset`) — covered in the implementation plan.
### `Info` notice about Web Analytics lag
Currently rendered between `ApiKeyCard` and `KpiSectionWithTrend` in `AnalyticsPage`. Keep it visible to users — move it to the **top of TrafficPage** (where the KPI data lives), since that's the surface where the lag would be observed. Same `Info` icon + copy.
---
## Styling (tokens only)
Everything maps to existing Tailwind semantic classes already used by Auth/Payments/Realtime:
| Design element | Token / class |
|---------------------|---------------------------------------------------------------------------|
| Sidebar background | `bg-semantic-1` (already on `FeatureSidebar`) |
| Sidebar border | `border-[var(--alpha-8)]` (already on `FeatureSidebar`) |
| Active item | `bg-alpha-8 text-foreground` (already in `FeatureSidebarItemRow`) |
| Hover | `hover:bg-alpha-4 hover:text-foreground` (already) |
| Disabled item | `text-muted-foreground/50 cursor-not-allowed` (NEW branch in `FeatureSidebar`) |
| Page background | `bg-[rgb(var(--semantic-1))]` (matches `AuthenticationLayout`) |
| Card background | `bg-card` |
| Card border | `border-[var(--alpha-8)]` |
| Foreground text | `text-foreground` / `text-muted-foreground` |
| Modal overlay/chrome | inherited from `MenuDialog` primitives — nothing to set per-instance |
No hex values, no `bg-[#1B1B1B]`, no inline `rgba()`.
---
## Risks / Edge Cases
1. **Direct deep-link to a sub-route while disconnected.** Handled — `AnalyticsLayout` swaps in `EmptyConnectPanel` regardless of which child route matches. URL keeps the user's intended destination so post-connect they see the right page after a single React-Query re-fetch.
2. **Connection state flipping mid-session** (cloud completes OAuth in another tab). `usePosthogConnection` already wires React-Query invalidation via `subscribePosthogConnectionStatus`. We move that subscription from `AnalyticsPage` into `AnalyticsLayout` so it's active on every sub-route.
3. **`projectId` resolves after `conn.data`.** Current code holds the connected view until `projectId` is available. Layout preserves that — it renders `LoadingState` until both resolve.
4. **`onConnectPosthog` undefined** (host doesn't provide it). `EmptyConnectPanel` already disables the button in that case; no change needed.
5. **Pagination behavior on Session Replay.** New `limit/offset` parameters on `useRecordings`. If the current backend endpoint only returns a fixed page, the implementation plan should confirm pagination parameters are supported before wiring `Pagination`. Fallback: keep showing the top-N list without pagination, hide the control.
6. **`MenuDialog` z-index vs `DisconnectDialog`.** Auth already stacks `Dialog`-on-`MenuDialog` (Mail provider config + confirms). The same Radix portal stacking applies — no new work.
---
## File-Level Change Summary
| File | Change |
|------|--------|
| `packages/dashboard/src/components/FeatureSidebar.tsx` | Add `disabled?: boolean` to `FeatureSidebarListItem`; render disabled variant in `FeatureSidebarItemRow` (no `<Link>`, `aria-disabled`, muted tokens, no hover). |
| `packages/dashboard/src/features/analytics/components/AnalyticsLayout.tsx` | NEW (default export) — sidebar + Outlet, owns `usePosthogConnection` + `TimeRangeProvider` + connect-status `useEffect`, falls back to `EmptyConnectPanel` when disconnected. |
| `packages/dashboard/src/features/analytics/components/AnalyticsSidebar.tsx` | NEW (named export) — wraps `FeatureSidebar` with title `Analytics`, 3 sub-items, Settings header button. |
| `packages/dashboard/src/features/analytics/components/AnalyticsConfigDialog.tsx` | NEW (named export) — `MenuDialog` with `Connection` section: read-only host / project ID / API key inputs, Setup-with-Prompt block, Disconnect button. |
| `packages/dashboard/src/features/analytics/pages/TrafficPage.tsx` | NEW — header (`Traffic` + TimeRangeSelector) + lag `Info` + `KpiSectionWithTrend` + 3 `BreakdownPanel`s. |
| `packages/dashboard/src/features/analytics/pages/RetentionPage.tsx` | NEW — header + `RetentionCard`. |
| `packages/dashboard/src/features/analytics/pages/SessionReplayPage.tsx` | NEW — header + `RecentReplaysCard` + `Pagination`. |
| `packages/dashboard/src/features/analytics/AnalyticsPage.tsx` | DELETE — superseded. |
| `packages/dashboard/src/features/analytics/components/posthog/ApiKeyCard.tsx` | DELETE — content lifted into `AnalyticsConfigDialog`. |
| `packages/dashboard/src/features/analytics/components/posthog/ConnectStatusBar.tsx` | DELETE — content lifted into `AnalyticsConfigDialog`. |
| `packages/dashboard/src/features/analytics/index.ts` | Replace `export { AnalyticsPage }` with no-op (delete file) or empty — router imports Layout directly. Match `realtime` / `payments` which have no barrel. |
| `packages/dashboard/src/router/AppRoutes.tsx` | Replace single `<Route path="/dashboard/analytics" element={<AnalyticsPage />} />` with `<Route path="/dashboard/analytics" element={<AnalyticsLayout />}>` + nested `index → Navigate to traffic`, `/traffic`, `/retention`, `/session-replay`. Drop the `import { AnalyticsPage } from '#features/analytics'` line, add new imports per the new convention. |
| `packages/dashboard/src/features/analytics/hooks/useRecordings.ts` | Extend with `limit` / `offset` parameters for pagination (verify backend support in implementation plan; fallback to top-N list if not). |
---
## Acceptance Criteria
1. `/dashboard/analytics` (cloud) — disconnected:
- Left sidebar titled **Analytics** with 3 disabled sub-items and a disabled gear icon
- Main area shows the `Connect PostHog` empty state (centered, single CTA)
- Clicking a disabled sub-item does nothing; URL does not change
2. `/dashboard/analytics` — connected:
- Auto-redirects to `/dashboard/analytics/traffic`
- All 3 sub-items enabled, gear icon enabled
3. `/dashboard/analytics/traffic` — connected:
- Page header with title + time range selector
- Web Analytics lag `Info` notice above the KPI row
- KPI row + 3 breakdown sections
4. `/dashboard/analytics/retention` — `RetentionCard` full-width
5. `/dashboard/analytics/session-replay` — `RecentReplaysCard` full-width + bottom `Pagination`
6. Gear icon opens **Analytics Config** modal:
- Read-only host / project ID / API key inputs (key hidden by default with show toggle)
- Setup-with-Prompt copyable block
- Disconnect button → existing `DisconnectDialog` confirm flow
7. Disconnecting from inside the modal returns the user to the disconnected layout (sub-items disabled, empty CTA).
8. No hex values / inline `rgba()` added anywhere — every color/border/spacing is a Tailwind semantic class or existing CSS var.
9. `isCloudHosting` route gate behavior unchanged — self-hosted dashboards still don't show Analytics.
.internal/docs/specs/2026-06-04-compute-container-logs-design.md
+82
View File
@@ -0,0 +1,82 @@
# Compute Container Logs — Design
**Date:** 2026-06-04
**Priority:** Dashboard-first — click into a compute service and see its container stdout/stderr.
## Problem
Compute services (containers on Fly.io) expose only **machine lifecycle events**
(`ServiceEvents` panel / `compute events` — start/stop/exit/restart). There is no way to
see container **stdout/stderr** ("application logs") from the dashboard or CLI. This adds it.
## Verified foundation
Fly exposes an HTTP REST logs endpoint (the same one `flyctl logs` and Fly's dashboard use).
Tested live against a throwaway Fly container on 2026-06-04:
```text
GET https://api.fly.io/api/v1/apps/{app}/logs
Auth: Authorization: FlyV1 <org-macaroon> # NOT "Bearer"
Query: instance=<machineId> next_token=<ns-unix-cursor>
Returns: HTTP 200, data[].attributes { timestamp(RFC3339), message, instance, region }
+ meta.next_token cursor
History: ~7-day retention (populates the view on open); re-poll with next_token to tail
```
### Gotchas (both load-bearing, both hit during implementation)
1. **Auth scheme.** The existing providers call `api.machines.dev` with `Bearer`. The logs
endpoint is on `api.fly.io` and **rejects `Bearer` with 401** — it requires the Fly
macaroon scheme `Authorization: FlyV1 <token>`. Same token, different host + prefix.
2. **`next_token` precision.** It is a nanosecond Unix timestamp that exceeds
`Number.MAX_SAFE_INTEGER`. `JSON.parse()` silently rounds it and corrupts the cursor, so
the provider extracts it from the raw response text to preserve every digit.
## Auth boundary (org token never leaves the backend)
```text
CLI ──Bearer ik_<project key>──┐
Dashboard ──user access token────────┤──▶ Backend ──FlyV1 <org token>──▶ Fly
(browser) (withAccessToken) ┘ (sole holder of the org token)
```
Neither the CLI nor the dashboard ever holds the Fly org token. The backend authorizes the
request (project ownership), then calls Fly. Identical boundary to the existing `events` route.
## What shipped in this PR (monorepo: `insforge-current-main`)
Mirrors the `events` feature end to end. New surface is one provider method (`getLogs`) feeding
a route, a service method, and the dashboard panel.
- `shared-schemas`: `computeLogLineSchema` / `computeLogsResponseSchema` (+ types).
- `compute.provider.ts`: `ComputeLogLine` / `ComputeLogsResult` + `getLogs()` on the interface.
- `fly.provider.ts`: `getLogs()` — `api.fly.io` + `FlyV1`, `instance` scoping, RFC3339→epoch-ms
normalization, precision-safe `next_token`. (Self-host path; unit-tested.)
- `cloud.provider.ts`: `getLogs()` — delegates `GET /machines/:id/logs` to the control plane.
- `services.service.ts`: `getServiceLogs()` (ownership guard, mirrors `getServiceEvents`).
- `services.routes.ts`: `GET /:id/logs` (`verifyAdmin`, project-ownership, `limit`, `next_token`).
- Dashboard: `computeServicesApi.logs()`, `useServiceLogs` hook, `ServiceLogs.tsx` (recent on
open + **Live** toggle re-polling every 2s), mounted in the service detail view beside Events.
Live tail in v1 = re-fetch the recent window on an interval (stateless; no cursor
accumulation/dedup to get wrong). SSE is a possible later upgrade.
## Companion PR (separate repo: `insforge-cloud-backend`)
For InsForge Cloud, the dashboard runs through `CloudComputeProvider`, which delegates to the
cloud control plane. That backend needs the matching `GET …/machines/:id/logs` route (its
own `fly-client.getLogs` with the same `api.fly.io` + `FlyV1` call). **Self-hosted works
fully from this PR alone** (FlyProvider path); cloud needs the companion route to light up.
## Non-goals
- No Vector / log-shipper / NATS / WireGuard. The REST endpoint is sufficient.
- No durable log store / new DB table — relies on Fly's ~7-day retention. A persisted,
searchable store is a deferred future phase behind the same surfaces, only if needed.
- No CLI in this PR — Phase 2 (`compute logs <id> --follow`), reusing the same route.
## Risks
- **Unofficial endpoint.** Fly documents the logs REST endpoint as "stable but not officially
supported — flyctl depends on it." Acceptable; the provider degrades to a thrown error (not
a crash) on non-200, and the durable-store path is the long-term fallback.
.prettierignore
+65
View File
@@ -0,0 +1,65 @@
# Dependencies
node_modules/
**/node_modules/
# Build outputs
dist/
**/dist/
build/
**/build/
.next/
**/.next/
# Coverage
coverage/
**/coverage/
# Turbo
.turbo/
**/.turbo/
# Public assets
public/
**/public/
# Lock files
package-lock.json
**/package-lock.json
yarn.lock
**/yarn.lock
pnpm-lock.yaml
**/pnpm-lock.yaml
# Generated files
*.min.js
*.min.css
# Logs
*.log
# IDE
.vscode/
.idea/
# OS
.DS_Store
Thumbs.db
# Environment
.env
.env.*
# Git
.git/
.gitignore
# File types to ignore
*.md
*.yaml
*.yml
*.json
# Folders to ignore
docs/
examples/
openapi/
.prettierrc
+9
View File
@@ -0,0 +1,9 @@
{
"semi": true,
"singleQuote": true,
"tabWidth": 2,
"trailingComma": "es5",
"printWidth": 100,
"endOfLine": "lf",
"jsxSingleQuote": false
}
CHANGELOG.md
+103
View File
@@ -0,0 +1,103 @@
# Changelog
## [2.0.1](https://github.com/InsForge/insforge/compare/v2.0.0-fix-settings-2...v2.0.1) (2026-03-09)
# [2.0.0](https://github.com/InsForge/insforge/compare/v1.5.9-ui-6...v2.0.0) (2026-03-06)
### Bug Fixes
* fix linking unnecessary contents in docs ([0edcb03](https://github.com/InsForge/insforge/commit/0edcb03a0aa3d05bd41160b193a358a1dfd0e58e))
## [1.5.8](https://github.com/InsForge/insforge/compare/v1.5.8-admin...v1.5.8) (2026-02-27)
## [1.5.8-admin](https://github.com/InsForge/insforge/compare/v1.5.8-admin...v1.5.8) (2026-02-26)
### Bug Fixes
* address CodeRabbit feedback for AI usage FK migration ([973a717](https://github.com/InsForge/insforge/commit/973a717e7dae2fda5e24e41f4750b5ac8ffaac1b))
* **ai:** allow disabling models with usage history ([b38e0da](https://github.com/InsForge/insforge/commit/b38e0dac3912b0bde64431aeb22d4feaec0a7178))
* **ai:** make disable idempotent and strengthen tests ([c21827a](https://github.com/InsForge/insforge/commit/c21827ae045c24f1ca990df0b7ae9adb964b7da6))
* **ai:** soft-disable configurations with is_active ([d9dcaf8](https://github.com/InsForge/insforge/commit/d9dcaf8920a8052f385e2c6a4925672db79823fe))
* allow admin refresh token without admin record ([773e0df](https://github.com/InsForge/insforge/commit/773e0dfc9f90de66f617d3ea496da2246c778d5d))
* disable no-control-regex for ANSI escape stripping ([11d0ee5](https://github.com/InsForge/insforge/commit/11d0ee5e93197d74e01abb964812a96cdbe5c83a))
* disable noImplicitAny for deno check and block Deno.serve() pattern ([e6cf105](https://github.com/InsForge/insforge/commit/e6cf105b65288736a67372d9abc3f5b1099a5b8f))
* distinguish ENOENT from other errors in checkCode catch handler ([f1fd067](https://github.com/InsForge/insforge/commit/f1fd0678e26fd6d0b13ca0ae5184f04cb4a11b4a))
* only run deno check when Deno Subhosting is configured ([13b03f5](https://github.com/InsForge/insforge/commit/13b03f55beec32618c5216c508321a60fea10a83))
* prevent mixing S3 and AWS credentials in storage provider ([706ce49](https://github.com/InsForge/insforge/commit/706ce495514785fa09a3b0ea3fcd0ea6377f37cd))
* resolve eslint curly and no-control-regex errors ([cd7cec9](https://github.com/InsForge/insforge/commit/cd7cec9d0deae5d5ff3ee90989fc990a62682aef))
* use docs/** glob so negation patterns re-include needed docs ([9bfcfc3](https://github.com/InsForge/insforge/commit/9bfcfc3eb891631a0176e514e815a37ce1656661))
* use NO_COLOR env instead of parsing ANSI from deno check output ([8c800bb](https://github.com/InsForge/insforge/commit/8c800bb8bbf6eaf17263a3b8a0cf5e8fb61ed4fc))
* use String.fromCharCode to avoid control char in source ([48a48a5](https://github.com/InsForge/insforge/commit/48a48a51beb3b1e0b52619707dd5465602bf1f7b))
* use unicode escape and consistent arrow formatting in deno check output ([74ee0ff](https://github.com/InsForge/insforge/commit/74ee0ff0395ef633d1a84d303d97e21b42a0cae0))
### Features
* add deno check pre-validation for edge functions ([95fd268](https://github.com/InsForge/insforge/commit/95fd26856b96fb47761ef80475b683ad09c3c743))
* add S3-compatible storage provider support (Wasabi, MinIO, etc.) ([d3eae5e](https://github.com/InsForge/insforge/commit/d3eae5ebabb7dd2fd43e6081de790e3dff031588))
## [1.5.7-storageLimit4](https://github.com/InsForge/insforge/compare/v1.5.8-admin...v1.5.8) (2026-02-15)
### Bug Fixes
* remove hardcoded 50MB CSV file size check from frontend ([c1daf24](https://github.com/InsForge/insforge/commit/c1daf244f8d1b0a313ea2ca6bdcec78a250fe3fb))
## [1.5.7-storageLimit2](https://github.com/InsForge/insforge/compare/v1.5.8-admin...v1.5.8) (2026-02-15)
## [1.5.6](https://github.com/InsForge/insforge/compare/v1.5.5-e2e-1...v1.5.6) (2026-02-13)
## [1.5.4](https://github.com/InsForge/InsForge/compare/v1.5.3-e2e-5...v1.5.4) (2026-02-07)
# [1.4.0](https://github.com/InsForge/InsForge/compare/v1.3.1-e2e.2...v1.4.0) (2025-12-19)
## [1.2.8](https://github.com/InsForge/InsForge/compare/v1.2.6...v1.2.8) (2025-12-05)
## [1.2.6](https://github.com/InsForge/InsForge/compare/v1.2.4...v1.2.6) (2025-11-22)
## [1.2.4](https://github.com/InsForge/InsForge/compare/v1.2.3...v1.2.4) (2025-11-22)
## [1.2.2](https://github.com/InsForge/InsForge/compare/v1.2.1-e2e...v1.2.2) (2025-11-18)
# [1.2.0](https://github.com/InsForge/InsForge/compare/v1.1.7-Nov13...v1.2.0) (2025-11-15)
## [1.1.2](https://github.com/InsForge/InsForge/compare/v1.1.0-posthog-9...v1.1.2) (2025-10-28)
### Bug Fixes
* addressing coderabbit comments ([97dd933](https://github.com/InsForge/InsForge/commit/97dd9339269991955d8c644f88e77efa6c3f6da2))
* macOS compatibility and AI config cleanup for e2e tests ([2e04d79](https://github.com/InsForge/InsForge/commit/2e04d7920b00a2b19eed1f0aa3072a6ea937f41a))
* removing package-lock ([9df772d](https://github.com/InsForge/InsForge/commit/9df772dcadd3a91e2fa70507d8f2779f6910d484))
* removing package-lock ([222dfc1](https://github.com/InsForge/InsForge/commit/222dfc1231504ce8e7d14d5ab79e57c1b00785c7))
* update the frontend to use the bulk-upsert exising API ([2b9b3b2](https://github.com/InsForge/InsForge/commit/2b9b3b23c4cc2d401781ba5504c54db07bb84af0))
### Features
* added tests for ai configs ([2a4e93a](https://github.com/InsForge/InsForge/commit/2a4e93acf145699511bd0ea49e29926f2d585b36))
* added tests for functions and secrets ([a985f03](https://github.com/InsForge/InsForge/commit/a985f035ffc1c1152999b317cf4e03e0f45ba34e))
* added tests for logs ([81f7734](https://github.com/InsForge/InsForge/commit/81f7734f3586e03ae2be5694b63d2a3417ae7339))
* fixed all the tests with correct format ([1242d13](https://github.com/InsForge/InsForge/commit/1242d133efd4398ae672e1e6c71ae1431aa9ba62))
* fixed secret tests ([5703836](https://github.com/InsForge/InsForge/commit/5703836ecee0ffeb27bfca9b0d785b95e68f1688))
* updated the backend and the service to update the csv parsing and validating ([e7ffd29](https://github.com/InsForge/InsForge/commit/e7ffd292fd93381ae2eebde17c16e3996241f15c))
* updated the frontend to display the button and the respective functions ([50613d8](https://github.com/InsForge/InsForge/commit/50613d8d972b88cf28b147bb7a44901c23d77e79))
# [1.1.0](https://github.com/InsForge/InsForge/compare/v1.0.1-ai-2...v1.1.0) (2025-10-11)
### Bug Fixes
* typo in docs ([d2a0efc](https://github.com/InsForge/InsForge/commit/d2a0efc56ab77560597dc2ee46c5f8a669fbd71d))
# [1.0.0](https://github.com/InsForge/InsForge/compare/v0.3.3...v1.0.0) (2025-09-29)
# [0.3.0](https://github.com/InsForge/InsForge/compare/v0.2.9-fix...v0.3.0) (2025-09-26)
CLAUDE_PLUGIN.md
+89
View File
@@ -0,0 +1,89 @@
# InsForge Claude Code Plugin
Official plugin for building with InsForge in Claude Code.
The public plugin is maintained in the
[InsForge/insforge-skills](https://github.com/InsForge/insforge-skills)
repository. This repository keeps the marketplace entry so users can install it
from the InsForge marketplace.
## Installation
In Claude Code, run:
```
/plugin marketplace add InsForge/InsForge
```
Then install the plugin:
```
/plugin install insforge
```
## What's Included
The public plugin currently includes four skills.
### `insforge`
Guidance for building application code with InsForge and `@insforge/sdk`,
including database CRUD, auth, storage uploads, functions, OpenRouter AI,
realtime, email, Stripe flows, and S3-compatible storage integrations.
### `insforge-cli`
Command-line project management with `@insforge/cli`, including project
creation, linking, SQL, migrations, RLS policies, functions, storage,
deployments, compute services, secrets, AI setup, payments, schedules, logs,
imports, exports, and backend branches.
### `insforge-debug`
Diagnostics for InsForge project issues, including SDK errors, HTTP failures,
edge function failures, database performance, auth and RLS denials, realtime
issues, and deployment failures.
### `insforge-integrations`
Integration guides for third-party auth providers and related RLS setup,
including Auth0, Clerk, Kinde, Stytch, WorkOS, Better Auth, and payment
facilitator guidance.
## Usage
Once installed, Claude Code can load InsForge-specific guidance when you are:
- setting up backend infrastructure such as tables, buckets, functions, auth,
AI, payments, or deployments
- integrating `@insforge/sdk` into frontend or server applications
- implementing database access with RLS-aware patterns
- debugging InsForge project errors and deployment issues
- connecting external auth providers to InsForge
## Repository Layout Note
The public plugin lives in
[InsForge/insforge-skills](https://github.com/InsForge/insforge-skills).
The `.claude/skills/` and `.agents/skills/` directories in this repository are
internal contributor skills for people working on the InsForge OSS repository.
They are not the public Claude Code plugin and should not be used as the
marketplace source.
## Contributing
To improve the public plugin, contribute to
[InsForge/insforge-skills](https://github.com/InsForge/insforge-skills).
The skills in that repository are Markdown files with YAML frontmatter. See its
`CONTRIBUTING.md` for guidelines on adding or improving skills.
## Feedback
Found an issue or have a suggestion? [Open an issue](https://github.com/InsForge/InsForge/issues)
or join our [Discord](https://discord.com/invite/MPxwj5xVvW).
## License
MIT - Same as the public InsForge skills plugin.
CODE_OF_CONDUCT.md
+128
View File
@@ -0,0 +1,128 @@
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
tony.chang@insforge.dev.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 3.0, available at
[https://www.contributor-covenant.org/version/3/0/code_of_conduct/](https://www.contributor-covenant.org/version/3/0/code_of_conduct/)
Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
CONTRIBUTING.md
+149
View File
@@ -0,0 +1,149 @@
# Contributing to InsForge
Thank you for your interest in contributing to InsForge. This guide will help you get started with the contribution process.
## Table of Contents
- [Code of Conduct](#code-of-conduct)
- [Project Structure](#project-structure)
- [Prerequisites](#prerequisites)
- [Getting Started](#getting-started)
- [Claiming an Issue](#claiming-an-issue)
- [Development Workflow](#development-workflow)
- [Testing](#testing)
- [Pull Request Process](#pull-request-process)
- [Code Style](#code-style)
## Code of Conduct
This project and everyone participating in it is governed by our Code of Conduct. By participating, you are expected to uphold this code.
## Project Structure
The InsForge monorepo is organized as follows:
- `/backend` - Core backend service with Express.js, PostgreSQL, and Better Auth integration
- `/frontend` - React-based admin dashboard for managing databases, users, and storage
- `/packages/shared-schemas` - Zod schemas and TypeScript types shared between frontend and backend
- `/docs` - MCP documentation
- `/functions` - Serverless edge functions for custom business logic
- `docker-compose.yml` - Docker config file to start the project
## Prerequisites
Before you start development, ensure you have the following:
- [Docker](https://www.docker.com/get-started)
- [Node.js](https://nodejs.org/) (LTS version recommended)
## Getting Started
1. Fork the repository to your GitHub account
2. Clone your fork locally:
```bash
git clone https://github.com/InsForge/InsForge.git
cd insforge
```
3. Install Docker
4. Open Docker App
5. Install Node.js (LTS version recommended)
6. Create a .env file from the example:
On Unix-based systems:
```bash
cp .env.example .env
```
On Windows:
```bash
copy .env.example .env
```
7. Run the project
```bash
docker compose up
```
## Claiming an Issue
We follow an **issue-first workflow**: open or find an issue, get it assigned to you, and only then start working on a PR. This keeps work tracked, avoids two people building the same thing, and makes review smoother.
1. **Find or open an issue** for the work. If one doesn't exist, open a new issue describing the bug or feature first.
2. **Claim it.** Comment on the issue asking for it to be assigned to you (for example, "I'd like to work on this" or "please assign this to me"). Our repo maintainer agent, **章北海 (Zhang Beihai)**, will assign you automatically if it can.
- Each contributor may hold **at most 3 open assigned issues at a time, across all InsForge repositories** (not per repo). Finish or release one before claiming another. To release an issue, comment "unassign me" on it.
- If the agent can't assign you (some accounts lack the required access), a maintainer will assign you manually.
3. **Wait until the issue is assigned to you** before opening your PR, and **link the issue from the PR** (for example, "Closes #123").
> Drive-by fixes are welcome — a PR without an assigned issue will still be reviewed. But the agent will add a `needs-issue` or `needs-assignment` label and leave a friendly reminder, and unlinked work is easier to lose track of. Claiming an issue first is the smoothest path.
## Development Workflow
1. Create a new branch for your changes:
```bash
git checkout -b type/description
# Example: git checkout -b feat/site-deployment
```
Branch type prefixes:
- `feat/` - New features
- `fix/` - Bug fixes
- `docs/` - Documentation changes
- `refactor/` - Code refactoring
- `test/` - Test-related changes
- `chore/` - Build process or tooling changes
2. Make your changes following the code style guidelines
3. Add tests for your changes (see test README for guidelines)
4. Run the test suite:
```bash
npm run test:e2e
```
5. Run linter:
```bash
npm run lint
```
6. Ensure all tests pass and the code is properly formatted
7. Commit your changes with a descriptive message following the Conventional Commits format:
```
type(scope): description
[optional body]
[optional screenshots / videos]
[optional footer(s)]
```
8. Push your branch to your fork
9. Open a pull request against the main branch
## Testing
All contributions must include appropriate tests. Follow these guidelines:
- Write unit tests for new features
- Ensure all tests pass before submitting a pull request
- Update existing tests if your changes affect their behavior
- Follow the existing test patterns and structure
- Test across different environments when applicable
## Pull Request Process
1. Make sure the issue you're resolving is [assigned to you](#claiming-an-issue) before opening the PR
2. Create a draft pull request early to facilitate discussion
3. Link the issue your PR resolves in the description (e.g., 'Closes #123')
4. Ensure all tests pass and the build is successful
5. Update documentation as needed
6. Keep your PR focused on a single feature or bug fix
7. Be responsive to code review feedback
8. **After you address review comments, re-request a review from your assigned reviewer** (use the 🔁 button next to their name in the Reviewers section). This is how the reviewer is notified that your changes are ready for another look — without it, your PR may sit unnoticed.
## Code Style
- Follow the existing code style
- Use TypeScript types and interfaces effectively
- Keep functions small and focused
- Use meaningful variable and function names
- Add comments for complex logic
- Update relevant documentation when making API changes
## Documentation Assets
Please review the documentation asset guidelines before adding images, videos, SVGs, or other media files to the repository:
- [Documentation Asset Guidelines](docs/asset-guidelines.md)
Dockerfile
+189
View File
@@ -0,0 +1,189 @@
# ============================================================
# Stage 1: package-prep — strip version from root package.json
# ============================================================
# Separate stage so that COPY --from uses content-based caching.
# Even if this stage rebuilds on version bump, the OUTPUT is
# identical (version removed), so downstream COPY --from hits cache.
# Pinned deno version — matches docker-compose deno runtime (denoland/deno:alpine-2.0.6)
ARG DENO_VERSION=2.0.6
FROM denoland/deno:alpine-${DENO_VERSION} AS deno-bin
FROM node:20-alpine AS package-prep
RUN apk add --no-cache jq
COPY package.json /tmp/package.json
RUN mkdir -p /out && \
jq 'del(.version) | .workspaces = [.workspaces[] | select(. != "mcp")]' \
/tmp/package.json > /out/package.json
# ============================================================
# Stage 2: deps — ALL dependencies (dev + prod) for building
# ============================================================
FROM node:20-alpine AS deps
WORKDIR /app
# Root package.json comes from package-prep (version stripped).
# COPY --from uses content-based cache: if output is identical,
# this layer and npm install below stay cached across releases.
COPY --from=package-prep /out/package.json ./package.json
COPY package-lock.json ./package-lock.json
COPY backend/package.json ./backend/package.json
COPY frontend/package.json ./frontend/package.json
COPY packages/dashboard/package.json ./packages/dashboard/package.json
COPY packages/shared-schemas/package.json ./packages/shared-schemas/package.json
COPY packages/ui/package.json ./packages/ui/package.json
# Strip prepare/build scripts from shared-schemas to prevent tsc
# from running during install (source files aren't copied yet).
# The actual build happens in the build stage with full source.
RUN apk add --no-cache jq && \
jq 'del(.scripts.prepare, .scripts.build)' \
packages/shared-schemas/package.json > packages/shared-schemas/package.json.tmp && \
mv packages/shared-schemas/package.json.tmp packages/shared-schemas/package.json
RUN npm ci && npm cache clean --force
# ============================================================
# Stage 3: build — compile all packages
# ============================================================
FROM deps AS build
COPY . .
# Vite bakes these into static bundles at compile time
ARG VITE_API_BASE_URL
ARG VITE_PUBLIC_POSTHOG_KEY
# Build order: shared packages → backend → frontend
RUN npm run build
# ============================================================
# Stage 4: prod-deps — production dependencies only
# ============================================================
FROM node:20-alpine AS prod-deps
WORKDIR /app
COPY --from=package-prep /out/package.json ./package.json
COPY package-lock.json ./package-lock.json
COPY backend/package.json ./backend/package.json
COPY frontend/package.json ./frontend/package.json
COPY packages/dashboard/package.json ./packages/dashboard/package.json
COPY packages/shared-schemas/package.json ./packages/shared-schemas/package.json
COPY packages/ui/package.json ./packages/ui/package.json
# Strip prepare/build scripts from shared-schemas to prevent tsc
# from running during install (tsc is a devDependency, not available here).
# The compiled output comes from the build stage instead.
RUN apk add --no-cache jq && \
jq 'del(.scripts.prepare, .scripts.build)' \
packages/shared-schemas/package.json > packages/shared-schemas/package.json.tmp && \
mv packages/shared-schemas/package.json.tmp packages/shared-schemas/package.json
RUN npm ci --omit=dev && npm cache clean --force
# ============================================================
# Stage 5: runner — minimal production image
# ============================================================
FROM node:20-alpine AS runner
# tini: proper PID 1 for signal forwarding and zombie reaping
# postgresql16-client: pg_dump/pg_restore for database backups. Keep the
# client major at 16 while the bundled postgres image is on 15 — pg_dump 17+
# emits settings (e.g. transaction_timeout) that a 15 server rejects, which
# would make every restore fail.
RUN apk add --no-cache tini postgresql16-client
WORKDIR /app
# Run as non-root using the built-in node user (uid 1000)
# /data: database dir (matches DatabaseManager default)
# Default STORAGE_DIR and LOGS_DIR so every containerised deployment
# (docker run, Compose, Zeabur, Render, etc.) writes to the well-known
# mount points. Compose and PaaS env vars override these when set.
ENV STORAGE_DIR=/insforge-storage
ENV LOGS_DIR=/insforge-logs
ENV MAX_JSON_BODY_SIZE=100mb
ENV MAX_URLENCODED_BODY_SIZE=10mb
RUN mkdir -p /data /insforge-storage /insforge-logs && \
chown node:node /data /insforge-storage /insforge-logs
# tsx is a devDependency but required at runtime for migrate:bootstrap
RUN npm install -g "tsx@^4.7.1" && npm cache clean --force
# --- Stable layers first (change only when dependencies change) ---
# Production node_modules (hoisted by npm workspaces)
COPY --from=prod-deps --chown=node:node /app/node_modules ./node_modules
# --- Volatile layers last (change every release) ---
# Compiled output: server.js + frontend/ static files
COPY --from=build --chown=node:node /app/dist ./dist
# Runtime docs for /api/docs endpoints and documentation assets
COPY --from=build --chown=node:node /app/docs ./docs
COPY --from=build --chown=node:node /app/.agents/docs ./.agents/docs
# Migration runtime: tsx resolves @/* aliases via tsconfig.json,
# node-pg-migrate reads .sql files from backend/src/
COPY --from=build --chown=node:node /app/backend/src ./backend/src
COPY --from=build --chown=node:node /app/backend/tsconfig.json ./backend/tsconfig.json
# Workspace packages needed at runtime:
# - shared-schemas: backend bootstrap/migrations resolve source via tsconfig paths
# - package.json + dist keep workspace links in node_modules valid
COPY --from=build --chown=node:node /app/packages/shared-schemas/package.json ./packages/shared-schemas/package.json
COPY --from=build --chown=node:node /app/packages/shared-schemas/dist ./packages/shared-schemas/dist
COPY --from=build --chown=node:node /app/packages/shared-schemas/src ./packages/shared-schemas/src
# Package manifests for npm scripts
COPY --from=build --chown=node:node /app/backend/package.json ./backend/package.json
COPY --from=build --chown=node:node /app/package.json ./package.json
USER node
EXPOSE 7130 7131
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["sh", "-c", "cd backend && npm run migrate:up && cd .. && exec npm start"]
# ============================================================
# Stage: dev — development image (used by docker-compose)
# ============================================================
# Source code is mounted via volumes, only needs Node.js + Deno.
FROM node:20-alpine AS dev
COPY --from=deno-bin /bin/deno /usr/local/bin/deno
# pg_dump/pg_restore for database backups (see runner stage for the
# version-pinning rationale)
RUN apk add --no-cache postgresql16-client
WORKDIR /app
# Default mount points (same as runner stage — see comments there)
ENV STORAGE_DIR=/insforge-storage
ENV LOGS_DIR=/insforge-logs
ENV MAX_JSON_BODY_SIZE=100mb
ENV MAX_URLENCODED_BODY_SIZE=10mb
# Pre-create volume mount points so Docker initializes named volumes
# with these directories instead of creating them as root-owned.
# No USER node here: dev containers use bind mounts for source code,
# which inherit host user ownership. Running as node would break
# npm install on root-owned bind mounts (e.g. in CI).
RUN mkdir -p /app/node_modules /app/backend/node_modules /app/frontend/node_modules \
/app/shared-schemas/node_modules /app/ui/node_modules \
/insforge-storage /insforge-logs
GITHUB_OAUTH_SETUP.md
+49
View File
@@ -0,0 +1,49 @@
# GitHub OAuth Application Setup Guide
This document will guide you through the steps to create and configure a GitHub OAuth application for using GitHub login functionality in your project.
## Step 1: Create a GitHub OAuth Application
1. **Log in to your GitHub account**.
2. Navigate to your developer settings page:
* Click on your profile avatar in the top right corner.
* Select **Settings**.
* In the left sidebar, select **Developer settings**.
3. On the Developer settings page, select **OAuth Apps**, then click the **New OAuth App** button.
## Step 2: Fill in Application Information
On the "Register a new OAuth application" page, you need to fill in the following information:
* **Application name**: Give your application a descriptive name, such as `Insforge Dev Login`.
* **Homepage URL**: Your application's homepage URL. In development environment, you can set it to `http://localhost:7130` or your frontend development server address.
* **Application description** (optional): Provide a brief description of your application.
* **Authorization callback URL**: This is the URL where GitHub will redirect users after authorization. For local development, make sure to set it to:
```
http://localhost:7130/api/auth/oauth/github/callback
```
**Note**: This URL must exactly match the callback URL configured in your backend service, otherwise the OAuth flow will fail.
After filling in the information, click **Register application**.
## Step 3: Get Client ID and Client Secret
After creating the application, you will be redirected to the application's settings page. Here you can see the **Client ID**.
To get the **Client Secret**, click the **Generate a new client secret** button. GitHub will generate a secret and display it to you.
**Important**: The Client Secret will only be shown once. Please copy it immediately and keep it secure. If you lose it, you will need to generate a new one.
## Step 4: Configure Environment Variables
After obtaining the Client ID and Client Secret, you need to add them to your project's `.env` file. Make sure your `.env` file (or corresponding environment configuration file) contains the following two lines:
```env
GITHUB_CLIENT_ID=YOUR_GITHUB_CLIENT_ID
GITHUB_CLIENT_SECRET=YOUR_GITHUB_CLIENT_SECRET
GITHUB_REDIRECT_URI=http://localhost:7130/api/auth/oauth/github/callback
```
Replace `YOUR_GITHUB_CLIENT_ID` and `YOUR_GITHUB_CLIENT_SECRET` with the actual values you obtained in the previous step.
After completing the above steps, your application is ready to use GitHub OAuth for user authentication.
GOOGLE_OAUTH_SETUP.md
+148
View File
@@ -0,0 +1,148 @@
# Google OAuth Login Setup Guide
## Overview
This project has integrated Google OAuth login functionality, supporting user authentication through Google accounts.
## Environment Variable Configuration
Add the following configuration to your `.env` file:
```env
# Google OAuth Configuration
GOOGLE_CLIENT_ID=your_google_client_id
GOOGLE_CLIENT_SECRET=your_google_client_secret
GOOGLE_REDIRECT_URI=http://localhost:7130/api/oauth/google/callback
```
## Google Cloud Console Setup
1. Visit [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project or select an existing project
3. Enable Google+ API and Google Identity API
4. Create an OAuth 2.0 Client ID in the "Credentials" page
5. Configure authorized redirect URIs:
- Development environment: `http://localhost:7130/api/auth/oauth/google/callback`
- Production environment: `https://yourdomain.com/api/auth/oauth/google/callback`
## API Endpoints
### 1. Get Google OAuth Authorization URL
**Endpoint:** `GET /api/auth/v1/google-auth`
**Parameters:**
- `redirect_url`: Redirect URL after successful login
**Response:**
```json
{
"success": true,
"data": {
"auth_url": "https://accounts.google.com/oauth/authorize?..."
}
}
```
**Example:**
```bash
curl "http://localhost:7130/api/auth/v1/google-auth?redirect_url=http://localhost:7131/dashboard"
```
### 2. Google OAuth Callback Handling
**Endpoint:** `GET /api/auth/oauth/google/callback`
**Parameters:**
- `code`: Authorization code returned by Google
- `state`: State parameter (optional)
**Process:**
1. Receive Google authorization code
2. Exchange for access token and ID token
3. Verify ID token and retrieve user information
4. Find or create user record
5. Generate JWT token
6. Redirect to specified URL with token information
**Redirect URL Format:**
```
{redirect_url}?token={jwt_token}&user_id={user_id}&email={email}&name={name}
```
## User Flow
### New User Registration
1. User clicks "Sign in with Google"
2. Redirect to Google authorization page
3. After user authorization, system creates new user record:
- Create basic authentication record in `auth` table
- Create Google identity record in `identifies` table
- Create user profile record in `profiles` table
4. Return JWT token and user information
### Existing User Login
1. User clicks "Sign in with Google"
2. Redirect to Google authorization page
3. After user authorization, system:
- Searches for user in `identifies` table by `provider` and `provider_id`
- Updates `last_login_at` timestamp
4. Return JWT token and user information
### Email Association
- If user already exists but not associated with Google account, system will automatically associate
- If Google account is already associated with another user, an error will be returned
## Frontend Integration Example
```javascript
// 1. Get authorization URL
const response = await fetch('/api/auth/v1/google-auth?redirect_url=/dashboard');
const { auth_url } = await response.json();
// 2. Redirect to Google authorization page
window.location.href = auth_url;
// 3. Handle returned token in callback page
const urlParams = new URLSearchParams(window.location.search);
const token = urlParams.get('token');
const userId = urlParams.get('user_id');
if (token) {
// Store token
localStorage.setItem('auth_token', token);
// Navigate to main page
window.location.href = '/dashboard';
}
```
## Error Handling
Common error codes:
- `MISSING_FIELD`: Missing required parameters
- `INVALID_CREDENTIALS`: Google token verification failed
- `ALREADY_EXISTS`: User already exists
## Security Considerations
1. Ensure `GOOGLE_CLIENT_SECRET` environment variable is stored securely
2. Use HTTPS in production environment
3. Validate redirect URL legitimacy
4. Regularly rotate JWT keys
5. Monitor abnormal login activities
## Troubleshooting
1. **"Invalid redirect_uri" Error**
- Check redirect URI configuration in Google Cloud Console
- Ensure it matches the `GOOGLE_REDIRECT_URI` environment variable
2. **"Invalid client" Error**
- Verify `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` are correct
- Ensure necessary APIs are enabled in Google Cloud Console project
3. **"Token verification failed" Error**
- Check network connectivity
- Verify Google API service status
- Confirm client ID configuration is correct
LICENSE
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2025 Insforge Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
README.md
+285
View File
@@ -0,0 +1,285 @@
<div align="center">
<a href="https://insforge.dev">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/logo-dark.svg">
<source media="(prefers-color-scheme: light)" srcset="assets/logo-light.svg">
<img src="assets/logo-dark.svg" alt="InsForge" width="500">
</picture>
</a>
<p>
The all-in-one, open-source backend platform for agentic coding.<br />
</p>
<p>
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-orange.svg" alt="License"></a>
<a href="https://www.npmjs.com/package/@insforge/sdk"><img src="https://img.shields.io/npm/dt/@insforge/sdk?color=blue&label=downloads" alt="Downloads"></a>
<a href="https://github.com/InsForge/InsForge/graphs/contributors"><img src="https://img.shields.io/github/contributors/InsForge/InsForge?color=green" alt="Contributors"></a>
<a href="https://insforge.dev"><img src="https://img.shields.io/badge/Visit-InsForge.dev-181818?logoColor=white&labelColor=555555&logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMjQwIiBoZWlnaHQ9IjI0MCIgdmlld0JveD0iMCAwIDI0MCAyNDAiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0iTTI2LjExODQgMTAxLjZDMjMuMjkzOSA5OC43ODMzIDIzLjI5MzkgOTQuMjE2NiAyNi4xMTg0IDkxLjRMOTcuNzE2NyAyMEwyMDAgMjBMNzcuMjYgMTQyLjRDNzQuNDM1NSAxNDUuMjE3IDY5Ljg1NjIgMTQ1LjIxNyA2Ny4wMzE3IDE0Mi40TDI2LjExODQgMTAxLjZaIiBmaWxsPSJ3aGl0ZSIvPjxwYXRoIGQ9Ik0xNTUuMjUxIDc3LjM3NUwyMDAgMTIyVjIyNEwxMDQuMTA5IDEyOC4zNzVMMTU1LjI1MSA3Ny4zNzVaIiBmaWxsPSJ3aGl0ZSIvPjwvc3ZnPgo=" alt="Visit InsForge.dev"></a>
<a href="https://gitcgr.com/InsForge/InsForge">
<img src="https://gitcgr.com/badge/InsForge/InsForge.svg" alt="gitcgr" />
</a>
</p>
<p>
<a href="https://x.com/InsForge"><img src="https://img.shields.io/badge/Follow%20on%20X-000000?logo=x&logoColor=white&style=for-the-badge" alt="Follow on X"></a>
<a href="https://www.linkedin.com/company/insforge"><img src="https://img.shields.io/badge/Follow%20on%20LinkedIn-0A66C2?logo=linkedin&logoColor=white&style=for-the-badge" alt="Follow on LinkedIn"></a>
<a href="https://discord.com/invite/MPxwj5xVvW"><img src="https://img.shields.io/badge/Join%20our%20Discord-5865F2?logo=discord&logoColor=white&style=for-the-badge" alt="Join our Discord"></a>
</p>
<a href="https://trendshift.io/repositories/19834" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/19834" alt="InsForge%2FInsForge | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
<br /><br />
<a href="https://vercel.com/oss">
<img alt="Vercel OSS Program" src="https://vercel.com/oss/program-badge-2026.svg" />
</a>
</div>
<p align="center">
⭐ <em>Help us reach more developers and grow the InsForge community. Star this repo!</em>
</p>
## InsForge
The all-in-one, open-source backend platform for agentic coding. InsForge gives your coding agent database, auth, storage, compute, hosting, and AI gateway to ship full-stack apps end-to-end.
https://github.com/user-attachments/assets/345efbc6-ca63-4189-bde0-12ef3bda561b
### How it works
Coding agents interact with InsForge through one of two interfaces:
- **MCP Server** (self-hosted and cloud): exposes InsForge's operations as tools any MCP-compatible agent can call.
- **CLI + Skills** (cloud only): a command-line interface paired with Skills that agents invoke directly from the terminal.
Both interfaces let coding agents operate the backend like backend engineers:
- **Read backend context and state**: Pull documentation, schemas, metadata (deployed functions, bucket contents, auth config), and runtime logs, so the agent has what it needs to write code, verify what it built, and debug when something breaks.
- **Configure primitives**: Deploy edge functions, run database migrations, create storage buckets, set up auth providers, and configure other backend resources directly.
```mermaid
graph TB
subgraph TOP[" "]
AG[AI Coding Agents]
end
subgraph MID[" "]
SL[InsForge]
end
AG --> SL
SL --> AUTH[Authentication]
SL --> DB[Database]
SL --> ST[Storage]
SL --> EF[Edge Functions]
SL --> MG[Model Gateway]
SL --> CP[Compute]
SL --> DEP[Deployment]
classDef bar fill:#0b0f14,stroke:#30363d,stroke-width:1px,color:#ffffff
classDef card fill:#161b22,stroke:#30363d,stroke-width:1px,color:#ffffff
class AG,SL bar
class AUTH,DB,ST,EF,MG,CP,DEP card
style TOP fill:transparent,stroke:transparent
style MID fill:transparent,stroke:transparent
linkStyle default stroke:#30363d,stroke-width:1px
```
### Core Products:
- **Authentication**: User management, authentication, and sessions
- **Database**: Postgres relational database
- **Storage**: S3 compatible file storage
- **Model Gateway**: OpenAI compatible API across multiple LLM providers
- **Edge Functions**: Serverless code running on the edge
- **Compute** (private preview): Long-running container services
- **Site Deployment**: Site build and deployment
## ⭐️ Star the Repository
<p align="center">
<img src="assets/insforge-star.gif" alt="Star InsForge" width="100%">
</p>
If you find InsForge useful or interesting, a GitHub Star ⭐️ would be greatly appreciated.
## Quickstart
### Cloud-hosted: [insforge.dev](https://insforge.dev)
<a href="https://insforge.dev" target="_blank" rel="noopener noreferrer"><img src="https://img.shields.io/badge/insforge.dev-181818?logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMjQwIiBoZWlnaHQ9IjI0MCIgdmlld0JveD0iMCAwIDI0MCAyNDAiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+PHBhdGggZD0iTTI2LjExODQgMTAxLjZDMjMuMjkzOSA5OC43ODMzIDIzLjI5MzkgOTQuMjE2NiAyNi4xMTg0IDkxLjRMOTcuNzE2NyAyMEwyMDAgMjBMNzcuMjYgMTQyLjRDNzQuNDM1NSAxNDUuMjE3IDY5Ljg1NjIgMTQ1LjIxNyA2Ny4wMzE3IDE0Mi40TDI2LjExODQgMTAxLjZaIiBmaWxsPSJ3aGl0ZSIvPjxwYXRoIGQ9Ik0xNTUuMjUxIDc3LjM3NUwyMDAgMTIyVjIyNEwxMDQuMTA5IDEyOC4zNzVMMTU1LjI1MSA3Ny4zNzVaIiBmaWxsPSJ3aGl0ZSIvPjwvc3ZnPgo=&logoColor=white" alt="InsForge.dev"></a>
### Self-hosted: Docker Compose
Prerequisites: [Docker](https://www.docker.com/) + [Node.js](https://nodejs.org/)
#### 1. Setup
You can run InsForge locally using Docker Compose. This will start a local InsForge instance on your machine.
[![Deploy on Docker][docker-btn]][docker-deploy]
Or run from source:
```bash
# Run with Docker
git clone https://github.com/InsForge/InsForge.git
cd insforge
cp .env.example .env
docker compose -f docker-compose.prod.yml up
```
#### 2. Connect InsForge MCP
Open [http://localhost:7130](http://localhost:7130)
Follow the steps to connect InsForge MCP Server
<div align="center">
<img src="assets/connect.png" alt="Connect InsForge MCP" width="600">
</div>
#### 3. Verify installation
To verify the connection, send the following prompt to your agent:
```
I'm using InsForge as my backend platform, call InsForge MCP's fetch-docs tool to learn about InsForge instructions.
```
#### 4. Running Multiple Projects
You can run multiple InsForge projects on the same host by using different ports and project names.
```bash
# Create a separate env file for each project
cp .env.example .env.project1
cp .env.example .env.project2
```
Edit `.env.project2` with different ports:
```
POSTGRES_PORT=5442
POSTGREST_PORT=5440
APP_PORT=7230
AUTH_PORT=7231
DENO_PORT=7233
```
Start each project with a unique name:
```bash
docker compose -f docker-compose.prod.yml --env-file .env.project1 -p project1 up -d
docker compose -f docker-compose.prod.yml --env-file .env.project2 -p project2 up -d
```
Each project gets its own isolated database, storage, and configuration. Manage them with:
```bash
docker compose -f docker-compose.prod.yml --env-file .env.project1 -p project1 ps # status
docker compose -f docker-compose.prod.yml --env-file .env.project1 -p project1 logs -f # logs
docker compose -f docker-compose.prod.yml --env-file .env.project1 -p project1 down # stop
```
### One-click Deployment
In addition to running InsForge locally, you can also launch InsForge using a pre-configured setup. This allows you to get up and running quickly with InsForge without installing Docker on your local machine.
| Railway | Zeabur | Sealos |
| --- | --- | --- |
| [![Deploy on Railway](https://railway.com/button.svg)](https://railway.com/deploy/insforge) | [![Deploy on Zeabur](https://zeabur.com/button.svg)](https://zeabur.com/templates/Q82M3Y) | [![Deploy on Sealos](https://sealos.io/Deploy-on-Sealos.svg)](https://sealos.io/products/app-store/insforge) |
## Contributing
**Contributing**: If you're interested in contributing, you can check our guide here [CONTRIBUTING.md](CONTRIBUTING.md). We truly appreciate pull requests, all types of help are appreciated!
**Support**: If you need any help or support, we're responsive on our [Discord channel](https://discord.com/invite/MPxwj5xVvW), and also feel free to email us [info@insforge.dev](mailto:info@insforge.dev) too!
## Documentation & Support
### Documentation
- **[Official Docs](https://docs.insforge.dev/introduction)** - Comprehensive guides and API references
### Community
- **[Discord](https://discord.com/invite/MPxwj5xVvW)** - Join our vibrant community
- **[Twitter](https://x.com/InsForge)** - Follow for updates and tips
### Contact
- **Email**: info@insforge.dev
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
---
[![Star History Chart](https://api.star-history.com/svg?repos=InsForge/InsForge&type=Date)](https://www.star-history.com/#InsForge/InsForge&Date)
## Badges
Show your project is built with InsForge.
### Made with InsForge
<a href="https://insforge.dev">
<img
width="168"
height="30"
src="https://insforge.dev/badge-made-with-insforge.svg"
alt="Made with InsForge"
/>
</a>
**Markdown:**
```md
[![Made with InsForge](https://insforge.dev/badge-made-with-insforge.svg)](https://insforge.dev)
```
**HTML:**
```html
<a href="https://insforge.dev">
<img
width="168"
height="30"
src="https://insforge.dev/badge-made-with-insforge.svg"
alt="Made with InsForge"
/>
</a>
```
### Made with InsForge (dark)
<a href="https://insforge.dev">
<img
width="168"
height="30"
src="https://insforge.dev/badge-made-with-insforge-dark.svg"
alt="Made with InsForge"
/>
</a>
**Markdown:**
```md
[![Made with InsForge](https://insforge.dev/badge-made-with-insforge-dark.svg)](https://insforge.dev)
```
**HTML:**
```html
<a href="https://insforge.dev">
<img
width="168"
height="30"
src="https://insforge.dev/badge-made-with-insforge-dark.svg"
alt="Made with InsForge"
/>
</a>
```
<p align="center">⭐ <b>Star us on GitHub</b> to get notified about new releases!</p>
<!-- LINK GROUPS -->
[docker-btn]: ./deploy/buttons/docker.png
[docker-deploy]: ./deploy/docker-deploy.md
README.wehub.md
+7
View File
@@ -0,0 +1,7 @@
# WeHub 来源说明
- 原始项目：`InsForge/InsForge`
- 原始仓库：https://github.com/InsForge/InsForge
- 导入方式：上游默认分支的最新快照
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
SECURITY.md
+89
View File
@@ -0,0 +1,89 @@
# Security Policy
Contact: https://github.com/InsForge/InsForge/security/advisories/new
Contact: mailto:security@insforge.dev
At InsForge, we consider the security of our systems a top priority. But no matter how much effort we put into system security, there can still be vulnerabilities present.
If you discover a vulnerability, we would like to know about it so we can take steps to address it as quickly as possible. We ask you to help us better protect our users and our systems.
## Supported Versions
We provide security fixes for the latest minor release of the current major version.
| Version | Supported |
| ------- | ------------------ |
| 2.x | :white_check_mark: |
| < 2.0 | :x: |
If you are running an older release, please upgrade before reporting, or include the exact version and commit so we can determine whether the issue still affects `main`.
## Out of scope
Here is a brief list of some common out-of-scope vulnerabilities:
- Clickjacking on pages with no sensitive actions.
- Unauthenticated, logout, or login CSRF.
- Attacks requiring MITM or physical access to a user's device.
- Attacks requiring social engineering.
- Any activity that could lead to the disruption of our service (DoS).
- Content spoofing and text injection issues without a clear attack vector or the ability to modify HTML/CSS.
- Email spoofing.
- Missing DNSSEC, CAA, or CSP headers.
- Lack of Secure or HttpOnly flags on non-sensitive cookies.
- Dead links.
- User enumeration on public registration endpoints.
## Testing guidelines
- Do not run automated scanners against other customers' projects. Running automated scanners can run up costs for our users, may disrupt services, and our own security tooling cannot distinguish hostile reconnaissance from whitehat research. If you wish to run an automated scanner, notify us first at `security@insforge.dev` and only run it against your own InsForge project. Do NOT attack projects belonging to other customers.
- Do not take advantage of the vulnerability you discover, for example by downloading more data than necessary to demonstrate the issue, or by deleting or modifying other people's data.
## Reporting a vulnerability
**Please do not open a public GitHub issue for security vulnerabilities.**
- File a private report through GitHub Security Advisories:
https://github.com/InsForge/InsForge/security/advisories/new
- Or email `security@insforge.dev`. If you do not receive a reply within 5 business days, please follow up via GitHub Security Advisories so the report does not get lost.
Provide enough information to reproduce the problem so we can resolve it quickly. Helpful details include:
- A description of the vulnerability and its impact.
- Steps to reproduce, or a minimal proof-of-concept.
- The affected version, commit, or deployment (self-hosted vs. `insforge.dev`).
- Any logs, screenshots, or scripts that help us reproduce the issue.
- Your contact information so we can follow up.
## Disclosure guidelines
- In order to protect our users, please do not reveal the problem to others until we have researched, addressed, and informed any affected customers.
- If you want to publicly share your research about InsForge (at a conference, in a blog post, or any other public forum), please share a draft with us for review at least 30 days before publication. The following should not be included:
- Data regarding any InsForge customer projects.
- InsForge customers' data.
- Information about InsForge employees, contractors, or partners.
## What we promise
- We will respond to your report within 5 business days with our evaluation of the report and an expected resolution date.
- If you have followed the instructions above, we will not take any legal action against you in regard to the report.
- We will handle your report with strict confidentiality, and not pass on your personal details to third parties without your permission.
- We will keep you informed of the progress towards resolving the problem.
- In the public information concerning the problem reported, we will give your name as the discoverer of the problem (unless you ask us not to).
We strive to resolve all reports as quickly as possible, and we want to play an active role in the ultimate publication of the issue once it is resolved.
## Safe harbor
We support good-faith security research. If you make a good-faith effort to comply with this policy during your research, we will:
- Consider your research authorized under this policy.
- Not pursue or support legal action against you for the research.
- Work with you to understand and resolve the issue quickly.
If at any point you are unsure whether a particular action is allowed, please ask us first via the reporting channels above.
---
*This policy is adapted from [Supabase's security policy](https://github.com/supabase/supabase/blob/master/SECURITY.md), with reporting channels and scope updated for InsForge.*
assets/connect.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

assets/insforge-star.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.7 MiB

assets/logo-dark.svg
+15
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.4 KiB

assets/logo-light.svg
+15
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.3 KiB

assets/signin.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

backend/package.json
+101
View File
@@ -0,0 +1,101 @@
{
"name": "insforge-backend",
"version": "1.0.0",
"author": "Insforge",
"description": "Open source backend-as-a-service with PostgreSQL and MCP integration",
"type": "module",
"main": "../dist/server.js",
"scripts": {
"start": "node ../dist/server.js",
"dev": "dotenv -e ../.env -- tsx watch src/server.ts",
"build": "tsup",
"clean": "rimraf dist",
"prebuild": "npm run clean",
"test": "vitest run",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage",
"test:ui": "vitest --ui",
"test:e2e": "./tests/run-all-tests.sh",
"test:integration": "vitest run --dir tests/integration --testTimeout=30000",
"migrate:bootstrap": "tsx src/infra/database/migrations/bootstrap/bootstrap-migrations.js",
"migrate:baseline": "tsx src/infra/database/migrations/bootstrap/baseline-migrations.js",
"migrate:up": "npm run migrate:bootstrap && node-pg-migrate up --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"migrate:down": "node-pg-migrate down --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"migrate:create": "node-pg-migrate create --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"migrate:check-duplicates": "node scripts/check-migration-duplicates.js",
"migrate:redo": "npm run migrate:bootstrap && node-pg-migrate redo --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"migrate:up:local": "dotenv -e ../.env -- npm run migrate:bootstrap && dotenv -e ../.env -- node-pg-migrate up --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"migrate:down:local": "dotenv -e ../.env -- node-pg-migrate down --migrations-dir src/infra/database/migrations --migrations-schema system --migrations-table migrations",
"typecheck": "tsc --noEmit",
"lint": "eslint ."
},
"keywords": [
"backend-as-a-service",
"postgresql",
"jwt"
],
"dependencies": {
"@asteasolutions/zod-to-openapi": "^7.3.4",
"@aws-sdk/client-cloudwatch-logs": "^3.713.0",
"@aws-sdk/client-s3": "^3.713.0",
"@aws-sdk/cloudfront-signer": "^3.901.0",
"@aws-sdk/lib-storage": "^3.1035.0",
"@aws-sdk/s3-presigned-post": "^3.879.0",
"@aws-sdk/s3-request-presigner": "^3.879.0",
"@databases/split-sql-query": "^1.0.4",
"@databases/sql": "^3.3.0",
"@types/multer": "^1.4.13",
"@types/pg": "^8.15.4",
"adm-zip": "^0.5.16",
"axios": "^1.16.1",
"bcryptjs": "^3.0.2",
"cookie-parser": "^1.4.7",
"cors": "^2.8.5",
"cron-parser": "^5.4.0",
"csv-parse": "^6.1.0",
"dotenv": "^16.4.5",
"express": "^4.22.0",
"express-rate-limit": "^7.1.5",
"file-type": "^19.0.0",
"google-auth-library": "^10.1.0",
"jose": "^6.1.0",
"jsonwebtoken": "^9.0.2",
"libpg-query": "^17.6.0",
"lru-cache": "^11.3.5",
"multer": "^2.0.2",
"node-fetch": "^3.3.2",
"node-pg-migrate": "^8.0.3",
"nodemailer": "^8.0.2",
"openai": "^5.19.1",
"pg": "^8.16.3",
"pg-format": "^1.0.4",
"picomatch": "^4.0.4",
"razorpay": "^2.9.6",
"socket.io": "^4.8.1",
"stripe": "^22.1.0",
"typescript": "^5.3.3",
"winston": "^3.17.0",
"xml2js": "^0.6.2",
"zod": "^3.23.8"
},
"devDependencies": {
"@types/adm-zip": "^0.5.7",
"@types/cookie-parser": "^1.4.8",
"@types/cors": "^2.8.17",
"@types/express": "^4.17.21",
"@types/jsonwebtoken": "^9.0.9",
"@types/node": "^20.11.24",
"@types/nodemailer": "^7.0.11",
"@types/pg-format": "^1.0.5",
"@types/picomatch": "^4.0.3",
"@types/supertest": "^7.2.0",
"@types/xml2js": "^0.4.14",
"dotenv-cli": "^10.0.0",
"insforge-test": "^0.2.0",
"supertest": "^7.2.2",
"tsup": "^8.5.0",
"tsx": "^4.7.1",
"vite-tsconfig-paths": "^5.1.4",
"vitest": "^4.1.8"
}
}
backend/scripts/check-migration-duplicates.js
+95
View File
@@ -0,0 +1,95 @@
/**
* Detects migration files that share the same numeric prefix.
*
* node-pg-migrate orders migrations by their leading number, so two files with
* the same prefix (e.g. `047_a.sql` and `047_b.sql`) have an ambiguous order and
* can apply inconsistently across environments. This guard runs in CI (and as a
* unit test) to stop new duplicates from landing on main.
*
* A small set of historical duplicates already exists on main and is grandfathered
* in via ALLOWED_DUPLICATES — those are tolerated, anything new is rejected.
*
* Exports `findDuplicateMigrations()` for the test suite; runs as a CLI (exit 1 on
* new duplicates) when executed directly.
*/
/* global console, process */
import { readdirSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
const __dirname = dirname(fileURLToPath(import.meta.url));
export const MIGRATIONS_DIR = join(__dirname, '..', 'src', 'infra', 'database', 'migrations');
// Pre-existing duplicate prefixes on main. Do not add to this list — fix the
// duplicate instead. These remain only because the migrations already shipped.
export const ALLOWED_DUPLICATES = new Set(['033', '047']);
const MIGRATION_FILE = /^(\d+)_.*\.sql$/;
/**
* @param {string} [dir] migrations directory to scan
* @returns {{ count: number, newDuplicates: Array<{prefix:string, files:string[]}>,
* grandfathered: Array<{prefix:string, files:string[]}>, nextPrefix: string }}
*/
export function findDuplicateMigrations(dir = MIGRATIONS_DIR) {
const byPrefix = new Map();
for (const name of readdirSync(dir)) {
const match = MIGRATION_FILE.exec(name);
if (!match) continue;
const prefix = match[1];
if (!byPrefix.has(prefix)) byPrefix.set(prefix, []);
byPrefix.get(prefix).push(name);
}
const newDuplicates = [];
const grandfathered = [];
for (const [prefix, files] of byPrefix) {
if (files.length < 2) continue;
files.sort();
(ALLOWED_DUPLICATES.has(prefix) ? grandfathered : newDuplicates).push({ prefix, files });
}
const maxPrefix = byPrefix.size
? Math.max(...[...byPrefix.keys()].map((p) => parseInt(p, 10)))
: -1;
return {
count: byPrefix.size,
newDuplicates,
grandfathered,
nextPrefix: String(maxPrefix + 1).padStart(3, '0'),
};
}
function main() {
const { count, newDuplicates, grandfathered, nextPrefix } = findDuplicateMigrations();
if (grandfathered.length > 0) {
console.log('Known (grandfathered) duplicate migration prefixes:');
for (const { prefix, files } of grandfathered) {
console.log(` ${prefix}: ${files.join(', ')}`);
}
}
if (newDuplicates.length > 0) {
console.error('\nERROR: duplicate migration number(s) detected:');
for (const { prefix, files } of newDuplicates) {
console.error(` ${prefix}: ${files.join(', ')}`);
}
console.error(
`\nEach migration needs a unique number. Renumber the new file to the next ` +
`available prefix (currently ${nextPrefix}_).`
);
process.exit(1);
}
console.log(`\nOK: ${count} unique migration number(s), no new duplicates.`);
}
// Run as a CLI only when executed directly, not when imported by tests.
if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
main();
}
backend/scripts/test-deno-subhosting.sh
Executable
+223
View File
@@ -0,0 +1,223 @@
#!/bin/bash
# Test script for Deno Subhosting migration
# Usage: ./scripts/test-deno-subhosting.sh [API_URL]
set -e
API_URL="${1:-http://localhost:7130}"
PASS=0
FAIL=0
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
log_info() { echo -e "${YELLOW}[INFO]${NC} $1"; }
log_pass() { echo -e "${GREEN}[PASS]${NC} $1"; PASS=$((PASS+1)); }
log_fail() { echo -e "${RED}[FAIL]${NC} $1"; FAIL=$((FAIL+1)); }
# Poll for deployment completion, sets DEPLOY_URL and DEPLOY_STATUS
wait_for_deployment() {
local max_attempts=${1:-30}
local attempt=0
DEPLOY_URL=""
DEPLOY_STATUS=""
while [ $attempt -lt $max_attempts ]; do
LIST_RESPONSE=$(curl -s "$API_URL/api/functions")
DEPLOY_STATUS=$(echo "$LIST_RESPONSE" | grep -o '"deployment":{[^}]*}' | grep -o '"status":"[^"]*"' | cut -d'"' -f4)
DEPLOY_URL=$(echo "$LIST_RESPONSE" | grep -o '"deployment":{[^}]*}' | grep -o '"url":"[^"]*"' | cut -d'"' -f4)
if [ "$DEPLOY_STATUS" = "success" ]; then
return 0
elif [ "$DEPLOY_STATUS" = "failed" ]; then
return 1
fi
echo -n "."
sleep 2
attempt=$((attempt+1))
done
return 1
}
# -----------------------------------------------------------------------------
# Prerequisites Check
# -----------------------------------------------------------------------------
log_info "Checking prerequisites..."
# Check if API is reachable
if ! curl -s "$API_URL/api/health" > /dev/null 2>&1; then
log_fail "API not reachable at $API_URL"
echo "Make sure the backend is running: cd backend && npm run dev"
exit 1
fi
log_pass "API reachable at $API_URL"
# -----------------------------------------------------------------------------
# Test 1: Create Function
# -----------------------------------------------------------------------------
log_info "Test 1: Creating test function..."
CREATE_RESPONSE=$(curl -s -X POST "$API_URL/api/functions" \
-H "Content-Type: application/json" \
-d '{
"name": "subhosting-test",
"slug": "subhosting-test",
"code": "export default async function(req: Request) {\n const url = new URL(req.url);\n return new Response(JSON.stringify({\n message: \"Hello from Deno Subhosting!\",\n path: url.pathname,\n timestamp: new Date().toISOString()\n }), {\n headers: { \"Content-Type\": \"application/json\" }\n });\n}",
"status": "active"
}')
if echo "$CREATE_RESPONSE" | grep -q '"slug":"subhosting-test"'; then
log_pass "Function created successfully"
else
log_fail "Failed to create function: $CREATE_RESPONSE"
fi
# -----------------------------------------------------------------------------
# Test 2: Wait for Deployment
# -----------------------------------------------------------------------------
log_info "Test 2: Waiting for deployment to complete..."
if wait_for_deployment 30; then
log_pass "Deployment succeeded: $DEPLOY_URL"
else
if [ "$DEPLOY_STATUS" = "failed" ]; then
log_fail "Deployment failed"
else
log_fail "Deployment timed out"
fi
fi
# -----------------------------------------------------------------------------
# Test 3: Invoke Function on Deno Subhosting
# -----------------------------------------------------------------------------
if [ -n "$DEPLOY_URL" ] && [ "$DEPLOY_STATUS" = "success" ]; then
log_info "Test 3: Invoking function on Deno Subhosting..."
INVOKE_RESPONSE=$(curl -s "$DEPLOY_URL/subhosting-test" 2>/dev/null || echo "CURL_FAILED")
if echo "$INVOKE_RESPONSE" | grep -q '"message":"Hello from Deno Subhosting!"'; then
log_pass "Function invoked successfully on Deno Subhosting"
echo "Response: $INVOKE_RESPONSE"
else
log_fail "Function invocation failed: $INVOKE_RESPONSE"
fi
else
log_info "Skipping invocation test (deployment not successful)"
fi
# -----------------------------------------------------------------------------
# Test 4: Update Function
# -----------------------------------------------------------------------------
log_info "Test 4: Updating function..."
UPDATE_RESPONSE=$(curl -s -X PATCH "$API_URL/api/functions/subhosting-test" \
-H "Content-Type: application/json" \
-d '{
"code": "export default async function(req: Request) {\n return new Response(JSON.stringify({\n message: \"Updated function!\",\n version: 2\n }), {\n headers: { \"Content-Type\": \"application/json\" }\n });\n}"
}')
if echo "$UPDATE_RESPONSE" | grep -q '"slug":"subhosting-test"'; then
log_pass "Function updated successfully"
else
log_fail "Failed to update function: $UPDATE_RESPONSE"
fi
# Wait for redeployment and get new deployment URL
log_info "Waiting for redeployment..."
if wait_for_deployment 30; then
UPDATED_RESPONSE=$(curl -s "$DEPLOY_URL/subhosting-test" 2>/dev/null || echo "CURL_FAILED")
if echo "$UPDATED_RESPONSE" | grep -q '"version":2'; then
log_pass "Updated function works on Deno Subhosting"
else
log_fail "Updated function not reflecting changes: $UPDATED_RESPONSE"
fi
else
log_fail "Redeployment failed or timed out"
fi
# -----------------------------------------------------------------------------
# Test 5: Create Function with Secret Access
# -----------------------------------------------------------------------------
log_info "Test 5: Creating function that accesses secrets..."
SECRET_FUNC_RESPONSE=$(curl -s -X POST "$API_URL/api/functions" \
-H "Content-Type: application/json" \
-d '{
"name": "secret-test",
"slug": "secret-test",
"code": "export default async function(req: Request) {\n const envKeys = Object.keys(Deno.env.toObject()).filter(k => !k.startsWith(\"DENO_\"));\n return new Response(JSON.stringify({\n message: \"Secret test\",\n availableEnvVars: envKeys.length,\n hasSecrets: envKeys.length > 0\n }), {\n headers: { \"Content-Type\": \"application/json\" }\n });\n}",
"status": "active"
}')
if echo "$SECRET_FUNC_RESPONSE" | grep -q '"slug":"secret-test"'; then
log_pass "Secret test function created"
else
log_fail "Failed to create secret test function: $SECRET_FUNC_RESPONSE"
fi
# Wait for deployment
sleep 10
if [ -n "$DEPLOY_URL" ]; then
SECRET_RESPONSE=$(curl -s "$DEPLOY_URL/secret-test" 2>/dev/null || echo "CURL_FAILED")
log_info "Secret test response: $SECRET_RESPONSE"
fi
# -----------------------------------------------------------------------------
# Test 6: List Functions with Deployment Info
# -----------------------------------------------------------------------------
log_info "Test 6: Checking list functions includes deployment info..."
LIST_FINAL=$(curl -s "$API_URL/api/functions")
if echo "$LIST_FINAL" | grep -q '"deployment"'; then
log_pass "List functions includes deployment info"
echo "$LIST_FINAL" | jq '.deployment' 2>/dev/null || echo "Deployment info present"
else
log_fail "List functions missing deployment info"
fi
# -----------------------------------------------------------------------------
# Test 7: Delete Function
# -----------------------------------------------------------------------------
log_info "Test 7: Deleting test functions..."
curl -s -X DELETE "$API_URL/api/functions/subhosting-test" > /dev/null
curl -s -X DELETE "$API_URL/api/functions/secret-test" > /dev/null
log_pass "Functions deleted (cleanup)"
# Wait for redeployment without deleted functions
sleep 5
# Verify functions are gone from Deno Subhosting
if [ -n "$DEPLOY_URL" ]; then
GONE_RESPONSE=$(curl -s "$DEPLOY_URL/subhosting-test" 2>/dev/null || echo "CURL_FAILED")
if echo "$GONE_RESPONSE" | grep -q '"error":"Function not found"'; then
log_pass "Deleted function no longer accessible on Deno Subhosting"
else
log_info "Function may still be cached: $GONE_RESPONSE"
fi
fi
# -----------------------------------------------------------------------------
# Summary
# -----------------------------------------------------------------------------
echo ""
echo "============================================"
echo "Test Summary"
echo "============================================"
echo -e "${GREEN}Passed: $PASS${NC}"
echo -e "${RED}Failed: $FAIL${NC}"
echo "============================================"
if [ $FAIL -gt 0 ]; then
exit 1
fi
backend/src/api/middlewares/auth.ts
+318
View File
@@ -0,0 +1,318 @@
import { Request, Response, NextFunction } from 'express';
import { TokenManager } from '@/infra/security/token.manager.js';
import { AppError } from '@/utils/errors.js';
import { ERROR_CODES, type RoleSchema } from '@insforge/shared-schemas';
import { NEXT_ACTIONS } from '../../utils/next-actions.js';
import { SecretService } from '@/services/secrets/secret.service.js';
export type UserContext = {
/**
* Always present at the API level: user UUID for authenticated, admin id
* for project_admin, 'anonymous' for anon. Only the authenticated UUID is
* a row-ownership identity; database boundaries (claims, owner columns)
* gate on role === 'authenticated' so admin/anon labels never reach
* uuid-typed claims or columns.
*/
id: string;
email?: string;
role: RoleSchema;
};
export interface AuthRequest extends Request {
user?: UserContext;
authenticated?: boolean;
hasApiKey?: boolean;
projectId?: string;
}
const tokenManager = TokenManager.getInstance();
const secretService = SecretService.getInstance();
// Helper function to extract Bearer token (exported for optional auth checks)
export function extractBearerToken(authHeader: string | undefined): string | null {
if (!authHeader || !authHeader.startsWith('Bearer ')) {
return null;
}
return authHeader.substring(7);
}
// Helper function to extract API key from request
// Checks both Bearer token (if starts with 'ik_') and x-api-key header
export function extractApiKey(req: AuthRequest): string | null {
const bearerToken = extractBearerToken(req.headers.authorization);
if (bearerToken && bearerToken.startsWith('ik_')) {
return bearerToken;
}
// Fall back to x-api-key header for backward compatibility
if (req.headers['x-api-key']) {
return req.headers['x-api-key'] as string;
}
return null;
}
// Helper function to extract the opaque anon key from a request
// The anon key travels in the Authorization header like every other client
// credential (Bearer anon_...); signed-in clients replace it with their JWT.
export function extractAnonKey(req: AuthRequest): string | null {
const bearerToken = extractBearerToken(req.headers.authorization);
if (bearerToken && bearerToken.startsWith('anon_')) {
return bearerToken;
}
return null;
}
// Helper function to set user on request
function setRequestUser(
req: AuthRequest,
payload: { sub: string; email?: string; role: RoleSchema }
) {
req.user = {
id: payload.sub,
email: payload.email,
role: payload.role,
};
}
/**
* Verifies user authentication (accepts API keys, anon keys, and JWT tokens)
*
* All credentials travel in the Authorization header; dispatch is by shape,
* never by falling back on failure:
* - `ik_...` (Bearer or x-api-key) -> admin API key
* - Bearer `anon_...` -> opaque anon key, `anon` role
* - any other Bearer -> JWT (role claim decides admin/authenticated/legacy anon)
*
* Signed-out clients send the anon key as their Bearer credential; signing in
* replaces it with the user JWT. Each branch fails closed: an invalid or
* expired user JWT must return 401 (so SDK refresh flows trigger) — it must
* never silently downgrade to anon.
*/
export async function verifyUser(req: AuthRequest, res: Response, next: NextFunction) {
const apiKey = extractApiKey(req);
if (apiKey) {
return verifyApiKey(req, res, next);
}
const bearerToken = extractBearerToken(req.headers.authorization);
if (bearerToken && bearerToken.startsWith('anon_')) {
return verifyAnonKey(req, res, next);
}
// Anything else (including no credentials) goes through JWT verification,
// which produces the canonical 401 when the header is missing or invalid
return verifyToken(req, res, next);
}
/**
* Verifies admin authentication (requires admin token)
*/
export async function verifyAdmin(req: AuthRequest, res: Response, next: NextFunction) {
const apiKey = extractApiKey(req);
if (apiKey) {
return verifyApiKey(req, res, next);
}
try {
const token = extractBearerToken(req.headers.authorization);
if (!token) {
throw new AppError(
'No admin token provided',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
// For admin, we use JWT tokens
const payload = tokenManager.verifyToken(token);
if (payload.role !== 'project_admin') {
throw new AppError(
'Admin access required',
403,
ERROR_CODES.AUTH_UNAUTHORIZED,
NEXT_ACTIONS.CHECK_ADMIN_TOKEN
);
}
setRequestUser(req, payload);
next();
} catch (error) {
if (error instanceof AppError) {
next(error);
} else {
next(
new AppError(
'Invalid admin token',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_ADMIN_TOKEN
)
);
}
}
}
/**
* Verifies API key authentication
* Accepts API key via Authorization: Bearer header or x-api-key header (backward compatibility)
*/
export async function verifyApiKey(req: AuthRequest, _res: Response, next: NextFunction) {
try {
// Extract API key from request using helper
const apiKey = extractApiKey(req);
if (!apiKey) {
throw new AppError(
'No API key provided',
401,
ERROR_CODES.AUTH_INVALID_API_KEY,
NEXT_ACTIONS.CHECK_API_KEY
);
}
const isValid = await secretService.verifyApiKey(apiKey);
if (!isValid) {
throw new AppError(
'Invalid API key',
401,
ERROR_CODES.AUTH_INVALID_API_KEY,
NEXT_ACTIONS.CHECK_API_KEY
);
}
req.authenticated = true;
req.hasApiKey = true;
next();
} catch (error) {
next(error);
}
}
/**
* Verifies the opaque anon key (`anon_...`)
* Maps the request to the `anon` role; RLS policies are the security boundary.
* The request carries 'anonymous' as its API-level subject, but no sub claim
* reaches the database (stripped like admin subjects), so auth.uid() is NULL
* and identity-scoped policies fail closed.
*/
export async function verifyAnonKey(req: AuthRequest, _res: Response, next: NextFunction) {
try {
const anonKey = extractAnonKey(req);
if (!anonKey) {
throw new AppError(
'No anon key provided',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
const isValid = await secretService.verifyAnonKey(anonKey);
if (!isValid) {
throw new AppError(
'Invalid anon key',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
setRequestUser(req, { sub: 'anonymous', role: 'anon' });
next();
} catch (error) {
next(error);
}
}
/**
* Core token verification middleware that handles JWT tokens
* Sets req.user with the authenticated user information
*/
export function verifyToken(req: AuthRequest, _res: Response, next: NextFunction) {
try {
const token = extractBearerToken(req.headers.authorization);
if (!token) {
throw new AppError(
'No token provided',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
// Verify JWT token
const payload = tokenManager.verifyToken(token);
// Validate token has a role
if (!payload.role) {
throw new AppError(
'Invalid token: missing role',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
// Set user info on request
setRequestUser(req, payload);
next();
} catch (error) {
if (error instanceof AppError) {
next(error);
} else {
next(
new AppError(
'Invalid token',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
)
);
}
}
}
/**
* Verifies JWT token from cloud backend (api.insforge.dev)
* Validates signature using JWKS and checks project_id claim
*/
export async function verifyCloudBackend(req: AuthRequest, _res: Response, next: NextFunction) {
try {
const token = extractBearerToken(req.headers.authorization);
if (!token) {
throw new AppError(
'No authorization token provided',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
);
}
// Use TokenManager to verify cloud token
const { projectId } = await tokenManager.verifyCloudToken(token);
// Set project_id on request for use in route handlers
req.projectId = projectId;
req.authenticated = true;
next();
} catch (error) {
if (error instanceof AppError) {
next(error);
} else {
next(
new AppError(
'Invalid cloud backend token',
401,
ERROR_CODES.AUTH_INVALID_CREDENTIALS,
NEXT_ACTIONS.CHECK_TOKEN
)
);
}
}
}

Some files were not shown because too many files have changed in this diff Show More