chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,55 @@
|
||||
# QA: Declarative Generative UI (A2UI — Fixed Schema) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/a2ui-fixed-schema` on the dashboard host
|
||||
- Agent backend is healthy; `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `a2ui_fixed` graph (registered as agent name `a2ui-fixed-schema` — see `src/app/api/copilotkit-a2ui-fixed-schema/route.ts`)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text, DOM structure, and the JSON schema at `src/agents/a2ui_schemas/flight_schema.json`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/a2ui-fixed-schema`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px, rounded-2xl, full-height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-fixed-schema"` and `agent="a2ui-fixed-schema"` (DevTools → Network: sending a message hits that endpoint)
|
||||
- [ ] Verify the single suggestion pill is visible with verbatim title "Find SFO → JFK" (message body: "Find me a flight from SFO to JFK on United for $289.")
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no flight card for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Schema Wiring (fixed catalog + `includeBasicCatalog`)
|
||||
|
||||
- [ ] DevTools → Network: after the first successful `display_flight` call, verify the response stream contains an `a2ui_operations` container with `catalogId: "copilotkit://flight-fixed-catalog"` (matches `CATALOG_ID` in `src/agents/a2ui_fixed.py` and `src/app/demos/a2ui-fixed-schema/a2ui/catalog.ts`)
|
||||
- [ ] Verify the same container carries the full `FLIGHT_SCHEMA` component tree (12 nodes from `src/agents/a2ui_schemas/flight_schema.json`: `root`, `content`, `title`, `route`, `from`, `arrow`, `to`, `meta`, `airline`, `price`, `bookButton`, `bookButtonLabel`)
|
||||
|
||||
#### Search-Flights Prompt (`display_flight` tool → `flight_schema.json`)
|
||||
|
||||
- [ ] Click the "Find SFO → JFK" suggestion; within 20s verify a single flight card renders in-transcript assembled per `flight_schema.json`:
|
||||
- outer `Card` with a `Column` of children in this order: title row, route row, meta row, book button
|
||||
- `Title` node renders the literal text "Flight Details" (1.15rem / 600 weight, color `#010507`)
|
||||
- `route` row shows `Airport` "SFO" → `Arrow` (`→`, color `#AFAFB7`) → `Airport` "JFK" (both monospaced, 1.5rem, 600 weight, 0.05em letter-spacing)
|
||||
- `meta` row shows `AirlineBadge` "UNITED" (uppercase pill, lilac `#BEC2FF` border, 0.08em tracking) on the left and `PriceTag` "$289" (monospaced, color `#189370`, 1.1rem / 600 weight) on the right
|
||||
- `Button` renders full-width with label "Book flight" (black `#010507` background, white text, 12px radius)
|
||||
- [ ] Verify all four data-model fields resolved correctly (origin=`SFO`, destination=`JFK`, airline=`United`, price=`$289`) — each is a `{ path: "/..." }` binding in the schema and must reach the DOM as a plain string via the `GenericBinder` (no literal `{path}` leak and no React error #31)
|
||||
|
||||
#### Book-Flight Button (inert — pure presentation)
|
||||
|
||||
- [ ] Verify the "Book flight" button is rendered with the schema-declared label and is clickable, but the click is a no-op: the agent is not invoked, no schema swap occurs, and the button does not transition to a "Booked" state. Schema-swap-on-action will be wired up once the Python SDK exposes `action_handlers=` on `a2ui.render` (see comment in `src/agents/a2ui_fixed.py`).
|
||||
|
||||
#### Follow-up Prompt (data-model refresh)
|
||||
|
||||
- [ ] Send "Find me a flight from LAX to ORD on Delta for $412."; within 20s verify the card updates in place with origin=`LAX`, destination=`ORD`, airline=`DELTA`, price=`$412` (same schema, new data model — proves the fixed-schema pattern: schema once, data streams)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is the capital of France?"; verify the agent replies in plain text without invoking `display_flight` (no flight card rendered, no `a2ui_operations` in the response)
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors and specifically no React error #31 ("objects are not valid as a React child, found: object with keys {path}") — the `DynString` union in `a2ui/definitions.ts` is what prevents this, so a single occurrence is a regression
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; flight card renders within 20s of the search prompt
|
||||
- `display_flight` is called exactly once per search prompt; result contains an `a2ui_operations` container with `catalogId: "copilotkit://flight-fixed-catalog"` and the full 12-node flight schema
|
||||
- All custom renderers in `a2ui/renderers.tsx` (`Card`, `Title`, `Airport`, `Arrow`, `AirlineBadge`, `PriceTag`, `Button`) render at least once per search-flights run
|
||||
- Clicking "Book flight" is a no-op (inert presentation button — see comment in `src/agents/a2ui_fixed.py`)
|
||||
- No UI layout breaks, no `{path}` leak into the DOM, no uncaught console errors
|
||||
@@ -0,0 +1,44 @@
|
||||
# QA: A2UI Error Recovery — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/a2ui-recovery` on the dashboard host
|
||||
- Agent backend is healthy; `OPENAI_API_KEY` is set; `LANGGRAPH_DEPLOYMENT_URL` points at the LangGraph deployment exposing the `a2ui_recovery` graph (registered as agent name `a2ui-recovery` — see `src/app/api/copilotkit-a2ui-recovery/route.ts`)
|
||||
- Requires `ag-ui-langgraph >= 0.0.41` (the `get_a2ui_tools` validate→retry recovery loop + `a2ui_recovery_exhausted` hard-fail envelope) and the `@copilotkit` A2UI renderer (the `building`/`retrying`/`failed` lifecycle rendering)
|
||||
- Backend-owned wiring: the route sets `injectA2UITool: false`; the agent owns `generate_a2ui` via `get_a2ui_tools({ recovery: { maxAttempts: 3 } })` (see `src/agents/recovery_agent.py`)
|
||||
- Reuses the **declarative-gen-ui** catalog (`catalogId: "declarative-gen-ui-catalog"`) and the Vantage Threads sales context — no new components
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/a2ui-recovery`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px, rounded-2xl, full-height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-a2ui-recovery"` and `agent="a2ui-recovery"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`)
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Recover a bad render"
|
||||
- "Show an unrecoverable failure"
|
||||
|
||||
### 2. Healing path
|
||||
|
||||
- [ ] Click "Recover a bad render" ("Render my Q2 sales dashboard, recovering if the first attempt is malformed.")
|
||||
- [ ] The inner `render_a2ui` returns **free-form / sloppy** A2UI args (components & data as JSON strings rather than structured arrays). Verify the middleware **heals** them via `parse_and_fix` into a valid surface that paints (no broken surface, no error banner)
|
||||
- [ ] Verify the **painted** surface is valid: a `declarative-metric` row ("Quarterly Revenue $4.2M", "Win Rate 31%") — i.e., the sloppy render was repaired and rendered
|
||||
- [ ] DevTools → Network: verify the final tool result carries an `a2ui_operations` container (no `a2ui_recovery_exhausted`)
|
||||
- [ ] Verify the chat reply is one short sentence noting the heal
|
||||
|
||||
### 3. Hard-fail (recovery exhausted) path
|
||||
|
||||
- [ ] Click "Show an unrecoverable failure" ("Render a dashboard that keeps failing validation so I can see the fallback.")
|
||||
- [ ] Verify the lifecycle ends in a tasteful `failed` state (NOT a broken/half-rendered surface and NOT a silent drop)
|
||||
- [ ] DevTools → Network: verify `render_a2ui` was attempted up to the cap (3 attempts, all invalid) and the tool returned an `a2ui_recovery_exhausted` envelope (no `a2ui_operations` painted)
|
||||
- [ ] Verify the chat reply gracefully explains the fallback (one short sentence)
|
||||
|
||||
### 4. Regression / isolation
|
||||
|
||||
- [ ] Verify the recovery demo does not affect the declarative-gen-ui or beautiful-chat demos (separate routes/agents)
|
||||
- [ ] Re-run each pill a second time and verify the same lifecycle
|
||||
|
||||
## Notes
|
||||
|
||||
- The malformed renders are forced by aimock fixtures (`showcase/aimock/d6/langgraph-python/a2ui-recovery.json`): the inner `render_a2ui` call is matched by `userMessage` + `toolName=render_a2ui`. Healing itself is performed live by the toolkit recovery loop inside `ag_ui_langgraph.get_a2ui_tools`, not the fixture.
|
||||
- This is the LangGraph-Python sibling of the Google-ADK `a2ui-recovery` demo; the backend recovery loop is provided by `ag_ui_langgraph` (`get_a2ui_tools`) rather than the ADK middleware (`get_a2ui_tool`).
|
||||
@@ -0,0 +1,64 @@
|
||||
# QA: Agent Config Object — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed and accessible at `/demos/agent-config`
|
||||
- Railway service `showcase-langgraph-python` healthy
|
||||
- `OPENAI_API_KEY` set on Railway
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Initial state
|
||||
|
||||
- [ ] Navigate to `/demos/agent-config`
|
||||
- [ ] Header "Agent Config Object" visible
|
||||
- [ ] `agent-config-card` is visible with the heading "Agent Config"
|
||||
- [ ] Tone dropdown (`data-testid="agent-config-tone-select"`) shows "professional"
|
||||
- [ ] Expertise dropdown (`data-testid="agent-config-expertise-select"`) shows "intermediate"
|
||||
- [ ] Response length dropdown (`data-testid="agent-config-length-select"`) shows "concise"
|
||||
- [ ] `<CopilotChat />` composer visible below the card
|
||||
|
||||
### 2. Default send
|
||||
|
||||
- [ ] Type "Tell me about black holes" and send
|
||||
- [ ] Agent responds within 15 seconds
|
||||
- [ ] Response is brief (1-3 sentences), professional tone, no emoji (consistent with the default config)
|
||||
|
||||
### 3. Enthusiastic + detailed
|
||||
|
||||
- [ ] Change Tone to "enthusiastic"
|
||||
- [ ] Change Response length to "detailed"
|
||||
- [ ] Verify both select values updated in the DOM
|
||||
- [ ] Send "Tell me about black holes" again
|
||||
- [ ] Response is noticeably longer (multiple paragraphs) and uses upbeat / energetic language
|
||||
- [ ] Compare to Step 2's response — the style difference is visible
|
||||
|
||||
### 4. Beginner expertise
|
||||
|
||||
- [ ] Change Expertise to "beginner"
|
||||
- [ ] Send "What is quantum entanglement?"
|
||||
- [ ] Response defines jargon and uses analogies
|
||||
|
||||
### 5. Expert expertise
|
||||
|
||||
- [ ] Change Expertise to "expert"
|
||||
- [ ] Send the same question
|
||||
- [ ] Response uses precise terminology, skips basics
|
||||
|
||||
### 6. Reactivity mid-thread
|
||||
|
||||
- [ ] Without reloading the page, with previous replies visible, change Tone to "casual"
|
||||
- [ ] Send a follow-up
|
||||
- [ ] Reply reflects the casual tone; previous replies in the transcript remain unchanged
|
||||
|
||||
### 7. Error handling
|
||||
|
||||
- [ ] Send an empty message; verify no-op or graceful empty-message handling
|
||||
- [ ] Verify no console errors during any of the above steps
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Dropdown value changes appear in the DOM within 100ms of selection
|
||||
- Agent responses arrive within 15s per send
|
||||
- Visible style differences across tone / expertise / length changes (qualitative, but clear side-by-side)
|
||||
- Transcript preserves history when config changes mid-thread
|
||||
@@ -0,0 +1,52 @@
|
||||
# QA: Agentic Chat — LangGraph (Python)
|
||||
|
||||
The minimum-viable CopilotChat demo: vanilla `<CopilotChat>` wired to a
|
||||
neutral helpful-assistant agent, with three starter-prompt suggestions.
|
||||
No tools, no custom rendering — anything richer belongs in dedicated
|
||||
demos (frontend-tools, tool-rendering, hitl-in-chat, etc.).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check `/api/health`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Initial render
|
||||
|
||||
- [ ] Navigate to `/demos/agentic-chat`
|
||||
- [ ] Verify the chat input renders with placeholder "Type a message..."
|
||||
- [ ] Verify all three suggestion pills are visible:
|
||||
- "Write a sonnet"
|
||||
- "Tell me a joke"
|
||||
- "Is 17 prime?"
|
||||
|
||||
### 2. Free-form chat
|
||||
|
||||
- [ ] Type a basic message (e.g. "Say hello") and press Enter
|
||||
- [ ] Verify the assistant streams back a text response
|
||||
|
||||
### 3. Suggestion pills
|
||||
|
||||
- [ ] Click the "Tell me a joke" pill
|
||||
- [ ] Verify the message is sent and the assistant streams back a joke
|
||||
|
||||
### 4. Multi-turn context
|
||||
|
||||
- [ ] Send "My name is Alice."
|
||||
- [ ] Wait for the assistant response
|
||||
- [ ] Send "What name did I just give you?"
|
||||
- [ ] Verify the assistant's second response contains "Alice"
|
||||
|
||||
### 5. Hygiene
|
||||
|
||||
- [ ] No console errors during normal usage
|
||||
- [ ] No layout breakage with a very long input
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat input mounts within ~3 seconds
|
||||
- Assistant first-token latency is under ~5 seconds for short prompts;
|
||||
full responses complete within ~30 seconds
|
||||
- Suggestion pills render alongside an empty chat and disappear once a
|
||||
conversation is in progress
|
||||
@@ -0,0 +1,68 @@
|
||||
# QA: Authentication — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed and accessible at /demos/auth
|
||||
- Railway service `showcase-langgraph-python` healthy
|
||||
- OPENAI_API_KEY set on Railway
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Initial authenticated state
|
||||
|
||||
- [ ] Navigate to /demos/auth
|
||||
- [ ] Verify the banner is visible with a green/success appearance (data-testid="auth-banner", data-authenticated="true")
|
||||
- [ ] Verify auth-status text reads "✓ Signed in as demo user"
|
||||
- [ ] Verify the "Sign out" button is visible and enabled (data-testid="auth-sign-out-button")
|
||||
- [ ] Verify the "Sign in" button is NOT present
|
||||
- [ ] Verify <CopilotChat /> is mounted below the banner
|
||||
- [ ] Verify no auth-demo-error surface is shown (data-testid="auth-demo-error" absent)
|
||||
- [ ] Verify no console errors on page load (the `/info` handshake should succeed)
|
||||
|
||||
### 2. Authenticated send → assistant response
|
||||
|
||||
- [ ] Type "Hello" and click send
|
||||
- [ ] Within 30 seconds, an assistant response is rendered in the transcript
|
||||
- [ ] No auth-demo-error surface appears
|
||||
|
||||
### 3. Sign out flips the banner and surfaces 401 without crashing
|
||||
|
||||
- [ ] Click "Sign out"
|
||||
- [ ] Within 1 second, the banner flips to amber/warning appearance (data-authenticated="false")
|
||||
- [ ] Verify auth-status text reads "⚠ Signed out — the agent will reject your messages until you sign in."
|
||||
- [ ] Verify the "Sign in" button is visible (data-testid="auth-authenticate-button")
|
||||
- [ ] Verify the "Sign out" button is no longer present
|
||||
- [ ] Type "Hello again" and click send
|
||||
- [ ] Within 15 seconds, the page-level error surface appears:
|
||||
- `data-testid="auth-demo-error"` visible with text containing "401" and/or "Unauthorized"
|
||||
- [ ] Verify the banner is STILL visible — the page must not white-screen
|
||||
- [ ] Verify no assistant response appears for the unauthenticated send
|
||||
|
||||
### 4. Sign in clears the error and restores sends
|
||||
|
||||
- [ ] Click "Sign in"
|
||||
- [ ] Within 1 second, the banner flips back to green (data-authenticated="true")
|
||||
- [ ] Verify the auth-demo-error surface is cleared
|
||||
- [ ] Type "Hello" and click send
|
||||
- [ ] Within 30 seconds, an assistant response is rendered
|
||||
|
||||
### 5. Refresh resets state to authenticated
|
||||
|
||||
- [ ] Hard-reload the page
|
||||
- [ ] Banner is green on first render (default state is authenticated; state does NOT persist)
|
||||
- [ ] No error surface on first render
|
||||
|
||||
### 6. Error Handling
|
||||
|
||||
- [ ] With DevTools Network panel blocking /api/copilotkit-auth, send a message while authenticated
|
||||
- [ ] Verify a network-level error surfaces cleanly (no uncaught promise rejection in console)
|
||||
- [ ] Restore network; verify sends work again without a page reload
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads authenticated by default — no 401 crash on initial `/info` fetch
|
||||
- Banner state flips within 1s of Sign out / Sign in clicks
|
||||
- Post-sign-out sends produce a visible 401 error within 15s via auth-demo-error
|
||||
- Page never white-screens after sign out — banner and composer remain mounted
|
||||
- Authenticated sends produce an assistant response within 30s
|
||||
- Refresh fully resets auth state (back to authenticated)
|
||||
@@ -0,0 +1,90 @@
|
||||
# QA: Beautiful Chat — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/beautiful-chat` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `beautiful_chat` graph
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text and DOM structure.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/beautiful-chat`; verify the page renders within 3s with the CopilotKit logo (`img[alt="CopilotKit"]`, src `/copilotkit-logo.svg`) top-left of the chat pane
|
||||
- [ ] Verify the `Chat` / `App` mode pill is fixed top-right, `Chat` active (highlighted) by default, and the right-side canvas region is collapsed (width 0)
|
||||
- [ ] Verify the `CopilotChat` input is rendered with no disclaimer text below it
|
||||
- [ ] Verify all 9 suggestion pills are visible with verbatim titles:
|
||||
- "Pie Chart (Controlled Generative UI)"
|
||||
- "Bar Chart (Controlled Generative UI)"
|
||||
- "Schedule Meeting (Human In The Loop)"
|
||||
- "Search Flights (A2UI Fixed Schema)"
|
||||
- "Sales Dashboard (A2UI Dynamic)"
|
||||
- "Excalidraw Diagram (MCP App)"
|
||||
- "Calculator App (Open Generative UI)"
|
||||
- "Toggle Theme (Frontend Tools)"
|
||||
- "Task Manager (Shared State)"
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Mode Toggle (frontend tools `enableAppMode` / `enableChatMode`)
|
||||
|
||||
- [ ] Click `App`; verify the canvas expands to ~2/3 width showing the TodoList empty state: pencil emoji, heading "No todos yet", subtext "Create your first task to get started", enabled "Add a task" button
|
||||
- [ ] Click `Chat`; verify the canvas collapses back to width 0
|
||||
|
||||
#### Shared State — Task Manager (agent tools `manage_todos`, `get_todos`)
|
||||
|
||||
- [ ] Click the "Task Manager (Shared State)" pill; verify the mode auto-switches to App and within 15s the "To Do" column renders exactly 3 todo cards (each with emoji, title, description)
|
||||
- [ ] Verify the "Done" column is empty (shows "No completed todos yet")
|
||||
- [ ] Click a todo's checkbox; verify the card moves from "To Do" to "Done"
|
||||
|
||||
#### Controlled Generative UI — Pie Chart (agent tool `query_data` + frontend component `pieChart`)
|
||||
|
||||
- [ ] Click "Pie Chart (Controlled Generative UI)"; within 15s verify a pie-chart card renders in-transcript with a non-empty `CardTitle` and `CardDescription`
|
||||
- [ ] Verify the donut SVG renders at least 2 `<circle>` slice elements inside the card
|
||||
- [ ] Verify the legend renders one row per slice: colored dot, label, comma-formatted value, and a percentage ending in "%"; percentages sum to 100%
|
||||
|
||||
#### Controlled Generative UI — Bar Chart (agent tool `query_data` + frontend component `barChart`)
|
||||
|
||||
- [ ] Click "Bar Chart (Controlled Generative UI)"; within 15s verify a bar-chart card renders with `CardTitle`, `CardDescription`, and a bar-chart icon in the header
|
||||
- [ ] Verify the recharts `ResponsiveContainer` (height 280px) renders at least 2 bar rectangles with X-axis labels matching the `label` field values; bars animate in via the `barSlideIn` keyframe on first render
|
||||
|
||||
#### Human-in-the-Loop — Schedule Meeting (frontend tool `scheduleTime`)
|
||||
|
||||
- [ ] Click "Schedule Meeting (Human In The Loop)"; within 15s verify a MeetingTimePicker card renders with a clock icon, a heading (agent-supplied reason or default "Schedule a Meeting"), 3 time-slot buttons each with date + time + a "30 min" duration badge, and a "None of these work" ghost button
|
||||
- [ ] Click a time slot; verify the card switches to the confirmed state with heading "Meeting Scheduled", the chosen date/time, and a green check icon
|
||||
- [ ] Re-trigger, click "None of these work"; verify the card shows heading "No Time Selected" and subtext "Looking for a better time that works for you"
|
||||
|
||||
#### A2UI Fixed Schema — Search Flights (agent tool `search_flights`)
|
||||
|
||||
- [ ] Click "Search Flights (A2UI Fixed Schema)"; within 20s verify exactly 2 flight cards render in-transcript, each with airline name, airline logo image, flight number, origin/destination, date, departure/arrival times, duration, a colored status dot, a status label (e.g. "On Time"), and a price starting with "$"
|
||||
|
||||
#### A2UI Dynamic — Sales Dashboard (agent tool `generate_a2ui`)
|
||||
|
||||
- [ ] Click "Sales Dashboard (A2UI Dynamic)"; within 30s verify a dynamic dashboard surface renders containing total-revenue metric, new-customers metric, conversion-rate metric, a pie chart (revenue by category), and a bar chart (monthly sales)
|
||||
|
||||
#### MCP App — Excalidraw Diagram
|
||||
|
||||
- [ ] Click "Excalidraw Diagram (MCP App)"; within 30s verify an Excalidraw embed renders a diagram with a router, 2 switches, and 4 computers (no console errors referencing the MCP server URL)
|
||||
|
||||
#### Open Generative UI — Calculator App
|
||||
|
||||
- [ ] Click "Calculator App (Open Generative UI)"; within 30s verify a sandboxed calculator UI renders with digit/operator buttons plus labeled metric shortcut buttons
|
||||
- [ ] Click a metric shortcut button; verify its value is inserted into the calculator display
|
||||
|
||||
#### Frontend Tool — Toggle Theme (`toggleTheme`)
|
||||
|
||||
- [ ] Click "Toggle Theme (Frontend Tools)"; verify the `html` element's `class` attribute toggles between containing `dark` and not containing `dark`, and the CopilotKit logo inverts via the `dark:invert` class
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send a ~500-character message; verify it wraps in-transcript without horizontal scroll or layout break
|
||||
- [ ] With the backend stopped, send a message; verify the UI surfaces a visible error path rather than hanging silently, and DevTools → Console shows no uncaught errors during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; plain-text response within 10 seconds
|
||||
- Controlled charts (pie/bar) render within 15 seconds of prompt; A2UI surfaces within 20–30 seconds
|
||||
- No UI layout breaks, no flash of unstyled content, no uncaught console errors
|
||||
- All 5 agent tools (`manage_todos`, `get_todos`, `query_data`, `search_flights`, `generate_a2ui`) are exercised by at least one check above
|
||||
@@ -0,0 +1,71 @@
|
||||
# QA: Chat Customization (CSS) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/chat-customization-css` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the neutral `sample_agent` graph
|
||||
- The demo wires `agent="chat-customization-css"` at `/api/copilotkit` (neutral assistant cell)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on CopilotKit built-in class names (`copilotKitChat`, `copilotKitMessages`, `copilotKitMessage`, `copilotKitUserMessage`, `copilotKitAssistantMessage`, `copilotKitInput`) scoped under the `.chat-css-demo-scope` wrapper defined in `theme.css`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/chat-customization-css`; verify the page renders within 3s with a single centered `<CopilotChat />` container inside a `.chat-css-demo-scope` wrapper (max-width 4xl, rounded corners via `rounded-2xl`)
|
||||
- [ ] Verify the chat input is visible with a textarea placeholder (default CopilotKit placeholder, e.g. "Type a message")
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s
|
||||
- [ ] Verify the input clears after send and a user-message bubble is appended to the transcript
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Theme Applied On Load (CopilotKit CSS Variables)
|
||||
|
||||
- [ ] In DevTools, inspect the `.chat-css-demo-scope` element and verify `getComputedStyle(el).getPropertyValue('--copilot-kit-primary-color').trim()` equals `#ff006e`
|
||||
- [ ] Verify `--copilot-kit-background-color` is `#fff8f0`, `--copilot-kit-secondary-color` is `#fde047`, and `--copilot-kit-separator-color` is `#ff006e`
|
||||
- [ ] Verify the `.copilotKitChat` background renders the themed cream color `#fff8f0` (not the default white/neutral)
|
||||
|
||||
#### User Message Bubble (hot pink gradient, serif, bold)
|
||||
|
||||
- [ ] Send "Hi there"; locate the newly rendered `.copilotKitMessage.copilotKitUserMessage` bubble
|
||||
- [ ] Verify its `background-image` computed style starts with `linear-gradient(135deg, rgb(255, 0, 110)` (i.e. `#ff006e`) and its text `color` is `rgb(255, 255, 255)`
|
||||
- [ ] Verify `font-family` contains `Georgia`, `font-weight` is `700`, and `border-radius` is `22px 22px 4px 22px`
|
||||
- [ ] Verify the bubble has a 2px solid border in `#ff6fa5` and a pink drop shadow (`box-shadow` includes `rgba(255, 0, 110, 0.35)`)
|
||||
|
||||
#### Assistant Message Bubble (amber, monospace, boxy)
|
||||
|
||||
- [ ] After the agent responds, locate the `.copilotKitMessage.copilotKitAssistantMessage` bubble
|
||||
- [ ] Verify its `background-color` computed style is `rgb(253, 224, 71)` (i.e. `#fde047`) and text `color` is `rgb(30, 27, 75)` (i.e. `#1e1b4b`)
|
||||
- [ ] Verify `font-family` contains `JetBrains Mono` / `Fira Code` / `Menlo` / `Consolas` (monospace stack)
|
||||
- [ ] Verify the bubble uses a boxy shape with `border-radius: 4px 22px 22px 22px`, a `2px solid rgb(30, 27, 75)` border, and a hard offset shadow `4px 4px 0 rgb(30, 27, 75)`
|
||||
|
||||
#### Input Area (cream background, dashed pink border, serif)
|
||||
|
||||
- [ ] Verify the `.copilotKitInput` element has `background-color: rgb(254, 243, 199)` (i.e. `#fef3c7`), a `3px dashed rgb(255, 0, 110)` border, and `border-radius: 18px`
|
||||
- [ ] Verify the inner `textarea` uses a `Georgia` serif font at `1.1rem`, with text color `#2c1810`
|
||||
- [ ] Verify the placeholder text renders in italic `#c2185b` (type-drain the textarea to expose the placeholder)
|
||||
|
||||
#### Theme Persists Across Message Rounds
|
||||
|
||||
- [ ] Send a second message ("Tell me a joke"); wait for the assistant reply
|
||||
- [ ] Verify both the new user bubble (pink gradient, serif) and the new assistant bubble (amber, monospace, boxy) retain the same computed styles as round 1 — no fallback to default CopilotKit theme between rounds
|
||||
- [ ] Verify the `.copilotKitMessages` container continues to use the `Georgia` serif font family and the `#fff8f0` background through scrolling
|
||||
|
||||
#### Chat Functions Identically To Default
|
||||
|
||||
- [ ] Type a multi-line message using Shift+Enter and verify Enter submits, Shift+Enter inserts a newline (same as a default `<CopilotChat />`)
|
||||
- [ ] Verify the agent's response streams token-by-token into the amber assistant bubble without the theme resetting mid-stream
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble added)
|
||||
- [ ] Send a ~500-character message; verify it wraps inside the pink user bubble without horizontal scroll, and the bubble still respects the hot-pink gradient + serif styling
|
||||
- [ ] With the backend stopped, send a message; verify a visible error path surfaces in the UI and DevTools → Console shows no uncaught errors caused by the themed styles
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; plain-text response within 10 seconds
|
||||
- All eight `--copilot-kit-*` CSS variables from `theme.css` resolve on `.chat-css-demo-scope`
|
||||
- User bubbles: hot pink gradient, white bold serif text, asymmetric rounded corners, 2px pink border, shadow
|
||||
- Assistant bubbles: amber `#fde047` background, monospace dark text, boxy corners, 2px dark border, hard offset shadow
|
||||
- Input: cream background, 3px dashed pink border, serif font, italic pink placeholder
|
||||
- No flash of unstyled content between the default CopilotKit stylesheet and `theme.css`; no uncaught console errors
|
||||
@@ -0,0 +1,59 @@
|
||||
# QA: Chat Customization (Slots) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/chat-slots` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health` or `/api/copilotkit` GET); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the neutral `sample_agent` graph (registered to the `chat-slots` agent name)
|
||||
- Note: this demo DOES include `data-testid` attributes on every custom slot. Use them as the primary selectors. The underlying agent is the neutral "helpful, concise assistant" (no frontend tools, no agent tools) — this demo exercises frontend slot customization only.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/chat-slots`; verify the page renders a centered chat surface (max-width 5xl, full viewport height) within 3s
|
||||
- [ ] Verify the custom welcome screen is visible (`data-testid="custom-welcome-screen"`), replacing the default welcome
|
||||
- [ ] Verify the nested welcomeMessage sub-slot renders inside the welcome screen (`data-testid="custom-welcome-message"`) with body text reading "Hover any region to see its slot path · click the badge to copy"
|
||||
- [ ] Verify the welcome card wraps the default chat `input` element and a `suggestionView` row beneath it (both passed in as props by CopilotChatView)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Welcome Screen Slot (`welcomeScreen`)
|
||||
|
||||
- [ ] Confirm the welcome card displays a hoverable SlotMarker badge (the gradient indigo/violet ring) — visually distinct from the default CopilotChat welcome
|
||||
- [ ] Confirm NO default CopilotChat welcome heading is rendered (the custom card fully replaces it)
|
||||
|
||||
#### Suggestions (`useConfigureSuggestions`)
|
||||
|
||||
- [ ] Verify two suggestion pills render in the `suggestionView` slot beneath the input with verbatim titles:
|
||||
- "Write a sonnet"
|
||||
- "Tell me a joke"
|
||||
- [ ] Click "Tell me a joke"; verify it sends the message "Tell me a short joke." and an assistant text response appears within 10s
|
||||
|
||||
#### Disclaimer Slot (`input.disclaimer`) — visible after first message
|
||||
|
||||
- [ ] After sending the first message, verify the custom disclaimer renders below the chat input (`data-testid="custom-disclaimer"`) containing:
|
||||
- a small badge reading "slot" (indigo background, lowercase bold)
|
||||
- body text "Custom disclaimer injected via `input.disclaimer`." (the phrase `input.disclaimer` is in monospace)
|
||||
- [ ] Verify the default CopilotChat disclaimer text (if any) is NOT present — the custom disclaimer replaces it
|
||||
|
||||
#### Assistant Message Slot (`messageView.assistantMessage`)
|
||||
|
||||
- [ ] After the assistant response arrives, verify the assistant message is wrapped in the custom container (`data-testid="custom-assistant-message"`) with:
|
||||
- an indigo-tinted card background (`bg-indigo-50/60` in light mode)
|
||||
- an indigo border (`border-indigo-200`)
|
||||
- a small absolute-positioned "slot" badge at the top-left corner (indigo-600 background, white uppercase bold text)
|
||||
- [ ] Verify the user message bubble is NOT wrapped in the custom container (user messages use the default styling)
|
||||
- [ ] Send a second prompt ("Write a one-line sonnet"); verify the second assistant response is also wrapped in the `custom-assistant-message` container
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no network request)
|
||||
- [ ] Send a ~500-character message; verify it wraps within the max-w-5xl container without horizontal scroll or layout break; the custom assistant-message card grows to fit the response
|
||||
- [ ] Verify DevTools → Console shows no uncaught errors or missing-prop warnings during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat surface renders within 3 seconds with the custom welcome card visible
|
||||
- Assistant text response within 10 seconds; wrapped in the custom assistant-message slot on every turn
|
||||
- All three custom slots (`welcomeScreen`, `input.disclaimer`, `messageView.assistantMessage`) replace their defaults and are visually distinguishable via their "slot" badges / gradient styling
|
||||
- No UI layout breaks, no uncaught console errors
|
||||
@@ -0,0 +1,39 @@
|
||||
# QA: CLI Start — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Dashboard host accessible; the `cli-start` cell is an informational command cell (no `/demos/cli-start/` route)
|
||||
- Local machine with Node.js 18+, npm 9+, and Python 3.11+ available; network access to the npm registry and GitHub
|
||||
- Canonical starter for comparison: `showcase/integrations/langgraph-python/` in the CopilotKit repo
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Dashboard Cell
|
||||
|
||||
- [ ] Open the LangGraph (Python) provider view on the dashboard; locate the "CLI Start Command" cell
|
||||
- [ ] Verify the cell displays the exact command `npx copilotkit@latest init --framework langgraph-python`
|
||||
- [ ] If a copy-to-clipboard button is present, click it; verify clipboard contents equal the command above (paste into a text editor to confirm)
|
||||
|
||||
### 2. Scaffold a Fresh Project
|
||||
|
||||
- [ ] In a fresh, empty directory, run `npx copilotkit@latest init --framework langgraph-python`
|
||||
- [ ] Verify the CLI completes without errors and scaffolds a project tree that matches the canonical starter at `showcase/integrations/langgraph-python/` (same top-level files: `package.json`, `next.config.ts`, `langgraph.json`, `showcase.json`, `tsconfig.json`, `postcss.config.mjs`, `Dockerfile`, `entrypoint.sh`, `src/`)
|
||||
- [ ] Verify `package.json` references `@copilotkit/*` at version `2.0.0` or newer (matches `copilotkit_version` in the provider manifest)
|
||||
|
||||
### 3. Install + Boot
|
||||
|
||||
- [ ] Run `npm install` in the scaffolded directory; verify it completes with no error-level output
|
||||
- [ ] Start the Next.js dev server per the scaffolded project's README / `package.json` `dev` script (typically `npm run dev`); verify it binds and logs a local URL (e.g. `http://localhost:3000`)
|
||||
- [ ] Start the LangGraph backend per the scaffolded project's instructions (typically `langgraph dev` using `langgraph.json`); verify it binds without error
|
||||
- [ ] Open the local URL in a browser; verify the starter demo renders (the "Sales Dashboard" surface per the manifest `starter.name`) and a basic chat round-trip works
|
||||
|
||||
### 4. Error Handling
|
||||
|
||||
- [ ] Re-run the CLI in a non-empty directory; verify it either refuses or prompts before overwriting (does not silently clobber)
|
||||
- [ ] Verify the scaffolded project has no lockfile merge conflicts and no unresolved peer-dependency errors in `npm install` output
|
||||
|
||||
## Expected Results
|
||||
|
||||
- CLI command copies cleanly and runs without interactive blockers (aside from any documented prompts)
|
||||
- Scaffolded tree matches the canonical starter shape
|
||||
- `npm install` completes without error; dev server + LangGraph backend boot; starter demo is interactive
|
||||
@@ -0,0 +1,72 @@
|
||||
# QA: Declarative Generative UI (A2UI — Dynamic Schema) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/declarative-gen-ui` on the dashboard host
|
||||
- Agent backend is healthy (`/api/copilotkit/health` or the host's `/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `a2ui_dynamic` graph (registered as agent name `declarative-gen-ui` — see `src/app/api/copilotkit-declarative-gen-ui/route.ts`)
|
||||
- The demo plays a sales analyst for the fictional **Vantage Threads** company. The dataset and per-question composition rules are registered as agent context in `src/app/demos/declarative-gen-ui/sales-context.ts` — surfaces should reflect those numbers ($4.2M Q2 revenue, 4 regions, 5 reps, 3 at-risk accounts, Meridian Apparel Group as top account)
|
||||
- Each custom renderer carries a stable `data-testid`: `declarative-card`, `declarative-metric`, `declarative-pie-chart`, `declarative-bar-chart`, `declarative-status-badge`, `declarative-data-table`, `declarative-info-row` (see `src/app/demos/declarative-gen-ui/a2ui/renderers.tsx`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-gen-ui`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px, rounded-2xl, full-height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-declarative-gen-ui"` and `agent="declarative-gen-ui"` (DevTools → Network: sending a message hits that endpoint, not `/api/copilotkit`)
|
||||
- [ ] Verify all 4 suggestion pills are visible with verbatim titles:
|
||||
- "Show my sales dashboard"
|
||||
- "Team performance"
|
||||
- "Anything at risk?"
|
||||
- "Top account details"
|
||||
- [ ] Verify no pill mentions a chart type — chart steering lives in the system prompt, not the user prompt (OSS-136)
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no A2UI surface rendered for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Catalog Wiring (provider `a2ui={{ catalog: myCatalog }}`)
|
||||
|
||||
- [ ] DevTools → Network: on first tool-driven response, verify the response stream contains an `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"` (matches `createCatalog(..., { catalogId: "declarative-gen-ui-catalog" })` in `a2ui/catalog.ts`)
|
||||
|
||||
#### Hero Pill — Composed Sales Dashboard
|
||||
|
||||
- [ ] Click "Show my sales dashboard"; within 60s verify ONE composed surface renders containing ALL of (no surrounding Card — the charts carry their own card chrome):
|
||||
- a bare row of 4 `declarative-metric` KPI tiles (uppercase label, large value, trend arrow with delta — green `↑` for up, red `↓` for down, e.g. "↑ 12% QoQ")
|
||||
- a `declarative-pie-chart` (recharts donut mirroring beautiful-chat: `innerRadius` 40 / `outerRadius` 80, one `.recharts-pie-sector` per slice, tooltip on hover, no legend) showing revenue by region
|
||||
- a `declarative-bar-chart` (recharts, height 200, single blue `#3b82f6` bars with rounded tops, dashed grid) showing monthly revenue
|
||||
- [ ] Verify the surface is a single composed dashboard, NOT a lonely single widget — this is the regression OSS-136 was filed about
|
||||
- [ ] Verify the pie slices cycle through the shared palette (`#3b82f6`, `#8b5cf6`, `#ec4899`, `#f59e0b`, `#10b981`, `#6366f1`) and bars are uniform blue `#3b82f6` — identical chrome to beautiful-chat's sales dashboard (12px-radius cards, 20px padding, soft shadow)
|
||||
- [ ] Verify the chat reply text beneath the surface is one short sentence (per `SYSTEM_PROMPT`: let the UI do the talking)
|
||||
- [ ] Verify metric numbers match the Vantage Threads dataset (revenue $4.2M, 186 new customers, 31% win rate, $22.6k avg deal)
|
||||
|
||||
#### Team Performance — DataTable
|
||||
|
||||
- [ ] Click "Team performance"; within 60s verify a `declarative-data-table` renders inside a Card: uppercase column headers (Rep / Attainment / Pipeline), one body row per rep (5 reps, Dana Whitfield through Elena Vasquez), tabular numerals
|
||||
- [ ] Verify a quota-attainment BarChart renders alongside the table (dashboardy, not a bare table); no StatusBadge or InfoRow
|
||||
|
||||
#### At Risk — StatusBadge Cards
|
||||
|
||||
- [ ] Click "Anything at risk?"; within 60s verify a risk panel: a KPI strip of Metric tiles (ARR at risk $615k, accounts at risk 3, biggest exposure Northwind $340k) above three side-by-side account Cards (Northwind Retail, Cascadia Outfitters, Atlas Goods), each with a content-sized `declarative-status-badge` (error = high severity, warning = medium) and a one-line reason + recommended next action
|
||||
- [ ] Verify no charts or tables render for this pill
|
||||
|
||||
#### Top Account — InfoRow Facts
|
||||
|
||||
- [ ] Click "Top account details"; within 60s verify a Card for Meridian Apparel Group with at least 3 `declarative-info-row` label/value rows (Owner, Region, ARR, Renewal, Last contact), each separated by a 1px bottom border
|
||||
- [ ] Verify a product-line PieChart renders next to the fact card (grounded in Meridian's product mix); no DataTable or StatusBadge
|
||||
|
||||
#### Cross-Pill Differentiation (mirrors the D5 probe)
|
||||
|
||||
- [ ] Run all 4 pills in one conversation; verify each pill mounts its distinguishing component fresh (the D5 probe `showcase/harness/src/probes/scripts/d5-gen-ui-declarative.ts` asserts a newly-mounted testid per pill — leftovers from earlier pills must not be the only match)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is 2+2?"; verify the agent replies in plain text without invoking `generate_a2ui` (no `a2ui_operations` in the response stream, no surface rendered)
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors, no React error #31, and no A2UI render-error banners ("Cannot create component root without a type", "Catalog not found")
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; A2UI surfaces render within 60s of prompt (secondary-LLM pass can be slow on cold start)
|
||||
- `generate_a2ui` is called exactly once per surface-producing prompt; result contains a valid `a2ui_operations` container with `catalogId: "declarative-gen-ui-catalog"`
|
||||
- The hero pill produces a composed dashboard (4 KPI tile metrics + 1 PieChart + 1 BarChart in one surface, with NO surrounding Card per OSS-136); pills 2-4 produce their distinguishing component (data-table / status-badge / info-row)
|
||||
- Numbers are consistent with the Vantage Threads dataset across all four pills
|
||||
- No UI layout breaks, no flash of unstyled content, no uncaught console errors
|
||||
@@ -0,0 +1,69 @@
|
||||
# QA: Declarative UI — Hashbrown — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed at `/demos/declarative-hashbrown`
|
||||
- Railway service `showcase-langgraph-python-production` healthy
|
||||
- `OPENAI_API_KEY` set in the Railway environment
|
||||
- `@hashbrownai/core` + `@hashbrownai/react` installed in the package
|
||||
- `byoc_hashbrown` graph registered in `langgraph.json`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Page load
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-hashbrown`
|
||||
- [ ] Header "Declarative UI: Hashbrown" visible
|
||||
- [ ] Short description mentioning `@hashbrownai/react` visible
|
||||
- [ ] Chat composer visible at the bottom of the chat area
|
||||
- [ ] 3 suggestion pills visible inside the composer with labels:
|
||||
"Sales dashboard", "Revenue by category", "Expense trend"
|
||||
- [ ] No red console errors (amber hydration warnings tolerated)
|
||||
|
||||
### 2. Sales dashboard suggestion
|
||||
|
||||
- [ ] Click the "Sales dashboard" pill
|
||||
- [ ] The prompt is dispatched automatically (useConfigureSuggestions sends
|
||||
the message on pill click)
|
||||
- [ ] Within 45 seconds, at least one MetricCard (`data-testid="metric-card"`)
|
||||
renders in the transcript
|
||||
- [ ] Within 45 seconds, at least one chart
|
||||
(`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders
|
||||
- [ ] Rendered content streams progressively — partial UI appears before the
|
||||
full response completes (optional visual check)
|
||||
|
||||
### 3. Revenue by category
|
||||
|
||||
- [ ] Click "Revenue by category"
|
||||
- [ ] Within 45s, a pie chart (`data-testid="pie-chart"`) renders
|
||||
- [ ] Legend shows at least 4 segments with readable labels and values
|
||||
|
||||
### 4. Expense trend
|
||||
|
||||
- [ ] Click "Expense trend"
|
||||
- [ ] Within 45s, a bar chart (`data-testid="bar-chart"`) renders
|
||||
- [ ] Chart has at least 3 bars with month-like labels
|
||||
|
||||
### 5. Free-form prompt
|
||||
|
||||
- [ ] Type "Show me revenue trends for the last six months" and press Enter
|
||||
- [ ] Verify at least one catalog component renders (metric, chart, or deal)
|
||||
|
||||
### 6. Multi-turn
|
||||
|
||||
- [ ] After a first render completes, send a follow-up prompt
|
||||
(e.g. "Now break it down by region")
|
||||
- [ ] A new render appears alongside prior renders in the transcript
|
||||
|
||||
### 7. Error handling
|
||||
|
||||
- [ ] Empty send is a no-op (button stays disabled)
|
||||
- [ ] Console remains clean during successful flows
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Suggestion pills produce a hashbrown render within 45 seconds
|
||||
- Streaming renders assemble progressively as JSON chunks arrive
|
||||
- No uncaught errors; no `useHashBrownKit must be used within
|
||||
HashBrownDashboard` errors
|
||||
- Multi-turn works without clearing prior renders
|
||||
@@ -0,0 +1,56 @@
|
||||
# QA: BYOC json-render — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed at `/demos/declarative-json-render`.
|
||||
- Railway backend healthy (`showcase-langgraph-python-production.up.railway.app`).
|
||||
- `OPENAI_API_KEY` and `LANGGRAPH_DEPLOYMENT_URL` configured on the Next.js app.
|
||||
- `@json-render/core` + `@json-render/react` present in `package.json` (pinned to `0.18.0`).
|
||||
- `byoc_json_render` graph registered in `langgraph.json`.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Page load
|
||||
|
||||
- [ ] Navigate to `/demos/declarative-json-render`.
|
||||
- [ ] Chat composer is visible.
|
||||
- [ ] Three suggestion pills appear with titles: "Sales dashboard", "Revenue by category", "Expense trend".
|
||||
- [ ] No console errors.
|
||||
|
||||
### 2. Sales dashboard suggestion
|
||||
|
||||
- [ ] Click the "Sales dashboard" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="json-render-root"` wrapper appears in the assistant bubble.
|
||||
- [ ] A `data-testid="metric-card"` renders inside the wrapper.
|
||||
- [ ] A chart (`data-testid="bar-chart"` or `data-testid="pie-chart"`) renders inside the wrapper.
|
||||
- [ ] No raw JSON text is shown once rendering finishes — the streaming JSON is replaced by components.
|
||||
|
||||
### 3. Revenue by category
|
||||
|
||||
- [ ] Click the "Revenue by category" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="pie-chart"` renders with multiple category slices + legend.
|
||||
|
||||
### 4. Expense trend
|
||||
|
||||
- [ ] Click the "Expense trend" suggestion.
|
||||
- [ ] Within 60 seconds, a `data-testid="bar-chart"` renders with month labels.
|
||||
|
||||
### 5. Free-form prompt
|
||||
|
||||
- [ ] Type "Show me a metric for quarterly revenue" and send.
|
||||
- [ ] Verify at least one `metric-card` renders; no console errors.
|
||||
|
||||
### 6. Multi-turn
|
||||
|
||||
- [ ] After a previous render is visible, send a follow-up prompt ("Now break that down by region").
|
||||
- [ ] A new assistant message appears with a new json-render rendering — prior renders stay in the transcript.
|
||||
|
||||
### 7. Malformed output handling
|
||||
|
||||
- [ ] If the agent ever replies with non-JSON text (force it by asking "tell me a joke"), the chat falls back to rendering that raw text via the default assistant bubble. No crash, no stuck spinner.
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Suggestion renders land within 60 seconds. Budget is slightly higher than the hashbrown demo because a JSON `{ root, elements }` spec is more verbose than hashbrown's token stream.
|
||||
- No uncaught errors in the console.
|
||||
- Streaming falls back to plain text until the JSON parses, then swaps to rendered components.
|
||||
@@ -0,0 +1,74 @@
|
||||
# QA: Frontend Tools (Async) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/frontend-tools-async` on the dashboard host
|
||||
- Agent backend is healthy (`/api/copilotkit` GET returns `langgraph_status: "reachable"`); `OPENAI_API_KEY` is set; `LANGGRAPH_DEPLOYMENT_URL` points at a deployment exposing the `frontend_tools_async` graph (registered under agent name `frontend-tools-async`)
|
||||
- Backend `frontend_tools_async.py` registers NO server-side tools; the frontend registers exactly ONE tool via `useFrontendTool`: **`query_notes`** (parameter: `keyword: string`)
|
||||
- The async handler sleeps 500ms (simulated client-side DB latency) then filters an in-memory `NOTES_DB` of 7 hard-coded notes, returning up to 5 matches against `title`, `excerpt`, or `tags` (case-insensitive). The tool has a custom `render` that mounts `NotesCard`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/frontend-tools-async`; verify the `CopilotChat` panel renders centered (max width 4xl, rounded-2xl corners) within 3s
|
||||
- [ ] Verify the input placeholder "Type a message" is visible
|
||||
- [ ] Send "Hello"; verify the agent responds with plain text within 10s and does NOT invoke `query_notes`
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestion Pills
|
||||
|
||||
- [ ] Verify all three suggestion pills are visible with verbatim titles:
|
||||
- "Find project-planning notes"
|
||||
- "Search for 'auth'"
|
||||
- "What do I have about reading?"
|
||||
- [ ] Click "Find project-planning notes"; verify the prompt "Find my notes about project planning." is sent
|
||||
|
||||
#### `query_notes` — Async Handler Loading State
|
||||
|
||||
- [ ] After triggering the `query_notes` flow (via pill or typing "Find my notes about planning"), verify within 10s a `data-testid="notes-card"` element renders in the transcript
|
||||
- [ ] While the handler's 500ms sleep is in flight (tool status ≠ `complete`), verify the card's header shows:
|
||||
- [ ] An uppercase "Notes DB" label
|
||||
- [ ] A heading `data-testid="notes-keyword"` reading `Matching "<keyword>"` where `<keyword>` is the agent's chosen search term (e.g. `planning`, `project planning`)
|
||||
- [ ] The subtext "Querying local notes DB..."
|
||||
- [ ] A "..." placeholder glyph (not the 📓 book emoji)
|
||||
|
||||
#### `query_notes` — Resolved State (Simulated DB Query)
|
||||
|
||||
- [ ] After the 500ms sleep resolves, verify the card's loading state ends within 2s:
|
||||
- [ ] The placeholder glyph flips from "..." to "📓"
|
||||
- [ ] The subtext shows "`N` match" or "`N` matches" (singular when N=1)
|
||||
- [ ] A `data-testid="notes-list"` `<ul>` renders (assuming N > 0)
|
||||
- [ ] Verify the list contains between 1 and 5 `<li>` entries, each with `data-testid="note-<id>"` (IDs from `n1`–`n7`)
|
||||
- [ ] For the prompt "Find my notes about project planning", verify the returned notes include at least `note-n1` (Q2 project planning kickoff) and `note-n5` (Project planning retrospective notes)
|
||||
- [ ] Verify each note row renders title (bold), excerpt (grey small text), and tag pills (uppercase, rounded-full)
|
||||
|
||||
#### Round-Trip — Agent Consumes Async Handler Result
|
||||
|
||||
- [ ] After the card renders, verify the agent emits a follow-up assistant text message within 10s summarizing the matches
|
||||
- [ ] Verify the summary references at least one note title or tag from the `notes-card` (confirms the agent awaited the async handler's resolved value, not just fired-and-forgot)
|
||||
- [ ] Ask a follow-up like "Which of those is about onboarding?"; verify the agent references note `n1`'s excerpt ("new onboarding flow") — proving the previous tool result is retained in context
|
||||
|
||||
#### Zero-Match Branch — Notes Card Empty State
|
||||
|
||||
- [ ] Send "Search my notes for xyzzy-nonsense-keyword"
|
||||
- [ ] Verify a `data-testid="notes-card"` renders with heading `Matching "xyzzy-nonsense-keyword"` (or close variant)
|
||||
- [ ] Verify the card shows italic grey text "No notes matched." (no `notes-list` element)
|
||||
- [ ] Verify the agent's follow-up text says no matches were found and offers to try a different keyword
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no notes-card, no user bubble)
|
||||
- [ ] Send a ~500-character unrelated prompt; verify the agent responds without calling `query_notes`
|
||||
- [ ] Trigger two `query_notes` invocations in quick succession (e.g. "Find auth notes" then immediately "Find reading notes"); verify both resolve independently and render two separate `notes-card` instances in the transcript
|
||||
- [ ] Open DevTools → Console; verify no uncaught errors, no Zod parse failures, no unresolved-Promise warnings
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; `notes-card` loading state appears within 10 seconds of prompt
|
||||
- Async handler's 500ms sleep is observable — the loading state ("Querying local notes DB...") is visible before resolution
|
||||
- After resolution, the `notes-list` renders with the correct subset of `NOTES_DB` matching the keyword
|
||||
- Agent's follow-up reply demonstrably uses the resolved notes array (round-trip verified)
|
||||
- Zero-match prompts render the empty-state branch, not a broken card
|
||||
- No uncaught console errors; handler Promise always resolves; no layout breaks
|
||||
@@ -0,0 +1,59 @@
|
||||
# QA: Frontend Tools (In-App Actions) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/frontend-tools` on the dashboard host
|
||||
- Agent backend is healthy (`/api/copilotkit` GET returns `langgraph_status: "reachable"`); `OPENAI_API_KEY` is set; `LANGGRAPH_DEPLOYMENT_URL` points at a deployment exposing the `frontend_tools` graph (registered under agent name `frontend_tools`)
|
||||
- The backend `frontend_tools.py` registers NO server-side tools — the agent forwards the frontend tool schema at runtime; the browser owns the handler
|
||||
- Frontend registers exactly ONE tool via `useFrontendTool`: **`change_background`** (parameter: `background: string` — a CSS background value, colors or gradients). Handler sets local React state and returns `{ status: "success", message: "Background changed to …" }`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/frontend-tools`; verify the page loads within 3s
|
||||
- [ ] Verify the background container (`data-testid="background-container"`) is visible, full-screen, flex-centered
|
||||
- [ ] Verify the initial background value is `var(--copilot-kit-background-color)` (the theme default) — inspect the container's inline `style.background`
|
||||
- [ ] Verify the `CopilotChat` panel renders centered (max width 4xl, rounded-2xl corners) inside the background container
|
||||
- [ ] Send "Hello"; verify the agent responds with plain text within 10s and does NOT call `change_background`
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestion Pills
|
||||
|
||||
- [ ] Verify the "Change background" suggestion pill is visible above the input
|
||||
- [ ] Verify the "Sunset theme" suggestion pill is visible
|
||||
- [ ] Click the "Change background" pill; verify it sends the verbatim prompt "Change the background to a blue-to-purple gradient."
|
||||
|
||||
#### `change_background` — Successful Invocation
|
||||
|
||||
- [ ] After clicking the "Change background" pill (or typing "Change the background to a linear gradient from indigo to pink"), verify within 15s that the agent invokes the `change_background` tool
|
||||
- [ ] Verify the `data-testid="background-container"` element's inline `style.background` mutates from `var(--copilot-kit-background-color)` to a CSS gradient string containing `linear-gradient` (or `radial-gradient`)
|
||||
- [ ] Verify the change is visually reflected (color/gradient swap is instant, no flash)
|
||||
|
||||
#### Round-Trip — Agent References Tool Return Value
|
||||
|
||||
- [ ] Immediately after the background changes, verify the agent emits a follow-up assistant message that acknowledges the change
|
||||
- [ ] Verify the follow-up message text references the new background value OR echoes the `"Background changed to <value>"` success message (confirming the agent consumed the handler's return payload, not just triggered the tool blindly)
|
||||
- [ ] Ask "What color is the background right now?"; verify the agent replies with a description consistent with the last `change_background` call's argument (demonstrates the agent received the handler's return string into context)
|
||||
|
||||
#### Second Invocation — State Persistence
|
||||
|
||||
- [ ] Ask "Change it to a sunset gradient with orange and pink"; verify the background mutates again to a new gradient
|
||||
- [ ] Verify the previous gradient is fully replaced (not layered)
|
||||
- [ ] Ask "Now make it solid dark blue"; verify `style.background` becomes a solid color value (e.g. `#00008b`, `darkblue`, or `rgb(...)`) — no gradient keyword
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send "Change the background to banana"; verify the agent either (a) calls `change_background` with a valid CSS fallback (e.g. `yellow`, `#FFE135`) and the container updates, or (b) declines politely without calling the tool — either is acceptable, but the UI must not crash
|
||||
- [ ] Send an empty message; verify it is a no-op
|
||||
- [ ] Send a ~500-character prompt unrelated to backgrounds; verify the agent responds normally without invoking `change_background`
|
||||
- [ ] Open DevTools → Console; verify no uncaught errors, no Zod parse failures, no `useFrontendTool` warnings
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; first `change_background` invocation completes within 15 seconds of prompt
|
||||
- The `[data-testid="background-container"]` inline `style.background` updates in real time as the tool is invoked
|
||||
- Agent's follow-up reply demonstrably uses the handler's returned `message` field (round-trip verified)
|
||||
- Multiple sequential invocations replace the background cleanly with no visual artifacts
|
||||
- No uncaught console errors; no broken layouts
|
||||
@@ -0,0 +1,62 @@
|
||||
# QA: Agentic Generative UI — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the gen-ui-agent demo page
|
||||
- [ ] Verify the chat interface loads in a centered full-height layout
|
||||
- [ ] Verify the chat input placeholder "Type a message" is visible
|
||||
- [ ] Send a basic message
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Simple plan" suggestion button is visible (plan to go to Mars in 5 steps)
|
||||
- [ ] Verify "Complex plan" suggestion button is visible (plan to make pizza in 10 steps)
|
||||
|
||||
#### Task Progress Tracker (useAgent with state streaming)
|
||||
|
||||
- [ ] Click "Simple plan" suggestion or type "Build a plan to go to Mars in 5 steps"
|
||||
- [ ] Verify the TaskProgress component renders (`data-testid="task-progress"`)
|
||||
- [ ] Verify the progress bar appears with a gradient fill
|
||||
- [ ] Verify step items appear with descriptions (`data-testid="task-step-text"`)
|
||||
- [ ] Verify the "N/N Complete" counter updates as steps complete
|
||||
- [ ] Verify completed steps show:
|
||||
- Green background gradient
|
||||
- Check icon
|
||||
- Green text color
|
||||
- [ ] Verify the current pending step shows:
|
||||
- Blue/purple background gradient
|
||||
- Spinner icon with "Processing..." text
|
||||
- Pulsing animation
|
||||
- [ ] Verify future pending steps show:
|
||||
- Gray background
|
||||
- Clock icon
|
||||
- Muted text color
|
||||
|
||||
#### Complex Plan
|
||||
|
||||
- [ ] Type "Plan to make pizza in 10 steps"
|
||||
- [ ] Verify 10 steps appear in the progress tracker
|
||||
- [ ] Verify progress bar width increases as steps complete
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- Task progress tracker shows live step completion
|
||||
- Progress bar animates smoothly
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,75 @@
|
||||
# QA: In-Chat HITL via useInterrupt — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/gen-ui-interrupt` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `interrupt_agent` graph
|
||||
- Note: The picker card is rendered INLINE inside the chat transcript via `useInterrupt({ renderInChat: true })`, wired to langgraph's `interrupt()` primitive from the backend `schedule_meeting` tool.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/gen-ui-interrupt`; verify the page renders within 3s with the `CopilotChat` centered in a `max-w-4xl` container filling full viewport height, with rounded (`rounded-2xl`) styling
|
||||
- [ ] Verify the `CopilotChat` input placeholder is visible and the transcript is empty on first load
|
||||
- [ ] Send "Hello" and verify the agent responds with a text-only reply (no time picker rendered — the agent only calls `schedule_meeting` when explicitly asked to book/schedule)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Book a call with sales"
|
||||
- "Schedule a 1:1 with Alice"
|
||||
|
||||
#### Interrupt Trigger + Inline Render (useInterrupt low-level primitive)
|
||||
|
||||
- [ ] Click the "Book a call with sales" suggestion (or prompt equivalently)
|
||||
- [ ] Within 20s verify the agent invokes `schedule_meeting`, the backend hits `interrupt({topic, attendee})`, and a time-picker card renders INLINE inside the chat transcript with `data-testid="time-picker-card"`
|
||||
- [ ] Confirm the card IS a descendant of the chat transcript container (NOT portaled to `<body>`, unlike the `hitl-in-app` modal) — inspect in DevTools to verify the card sits between chat message bubbles
|
||||
- [ ] Verify the card header shows eyebrow "Book a call" and a topic heading reflecting the agent-supplied topic (e.g. contains "sales" / "pricing")
|
||||
- [ ] Verify the "Pick a time:" subheading and a 2x2 grid of exactly 4 slot buttons with the default labels:
|
||||
- "Tomorrow 10:00 AM"
|
||||
- "Tomorrow 2:00 PM"
|
||||
- "Monday 9:00 AM"
|
||||
- "Monday 3:30 PM"
|
||||
- [ ] Verify a "None of these work" ghost button is rendered below the slot grid
|
||||
|
||||
#### Pick-a-Slot Resume Path
|
||||
|
||||
- [ ] Click one of the four time slot buttons (e.g. "Monday 9:00 AM")
|
||||
- [ ] Verify the card immediately switches to the confirmed state: `data-testid="time-picker-picked"`, green-tinted border/background, and text "Booked for <chosen label>" with the label in bold
|
||||
- [ ] Verify all slot buttons disable (opacity reduced) and cannot be re-clicked
|
||||
- [ ] Verify the agent resumes within 10s and produces a chat reply confirming the meeting was scheduled for the chosen label (backend returns `"Meeting scheduled for {chosen_label}: {topic}"`)
|
||||
|
||||
#### Cancel Path
|
||||
|
||||
- [ ] Send a second prompt "Schedule a 1:1 with Alice next week to review Q2 goals."
|
||||
- [ ] Verify a fresh time-picker card renders inline (`data-testid="time-picker-card"`) — attendee line "With Alice" should appear under the topic
|
||||
- [ ] Click the "None of these work" button
|
||||
- [ ] Verify the card switches to `data-testid="time-picker-cancelled"` with the text "Cancelled — no time picked."
|
||||
- [ ] Verify the agent resumes and replies that the meeting was NOT scheduled / the user cancelled
|
||||
|
||||
#### Multi-Turn
|
||||
|
||||
- [ ] After completing either the pick or cancel path, send one more prompt "Book another call tomorrow morning"
|
||||
- [ ] Verify a new, independent time-picker card renders inline (previous card stays in its resolved state), the interrupt lifecycle repeats cleanly, and a second resume works end-to-end
|
||||
|
||||
#### Contract Check — Interrupt Is Low-Level
|
||||
|
||||
- [ ] Confirm only the tool-triggered path renders the picker: sending a plain conversational message ("What's the weather?") should NOT render a picker
|
||||
- [ ] Confirm no approval-dialog-style modal appears at any point (this demo is inline, not modal)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op
|
||||
- [ ] Attempt to double-click a slot button rapidly; verify only one selection is committed (button disables on first click)
|
||||
- [ ] Verify no uncaught console errors across any pick / cancel / multi-turn flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; plain-text response within 10 seconds
|
||||
- Time picker card renders inline in the chat within 20 seconds of a schedule/booking prompt
|
||||
- Picker resolves via either a slot button (emits `{chosen_time, chosen_label}`) or the "None of these work" button (emits `{cancelled: true}`); post-resolution the card is read-only
|
||||
- Agent resume produces a confirmation message that references the chosen slot label or the cancellation
|
||||
- No UI layout breaks, no uncaught console errors, no duplicate pickers from a single interrupt
|
||||
@@ -0,0 +1,56 @@
|
||||
# QA: Tool-Based Generative UI — LangGraph (Python)
|
||||
|
||||
The demo registers two `useComponent` renderers (`render_bar_chart`,
|
||||
`render_pie_chart`) on a centered `<CopilotChat>`. The agent emits
|
||||
chart-shaped tool calls and the frontend materializes them as Recharts
|
||||
SVG inside the assistant message bubble.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/gen-ui-tool-based`
|
||||
- Agent backend is healthy (`/api/health`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Initial render
|
||||
|
||||
- [ ] Navigate to `/demos/gen-ui-tool-based`
|
||||
- [ ] Verify the chat composer mounts (`textarea` with placeholder
|
||||
matching `/message/i`)
|
||||
- [ ] Verify all three suggestion pills are visible
|
||||
(`data-testid="copilot-suggestion"`):
|
||||
- "Sales bar chart"
|
||||
- "Traffic pie chart"
|
||||
- "Market share"
|
||||
|
||||
### 2. Bar chart flow
|
||||
|
||||
- [ ] Click the "Sales bar chart" suggestion (or type "Show me a bar
|
||||
chart of monthly expenses")
|
||||
- [ ] Verify the assistant message bubble
|
||||
(`[data-testid="copilot-assistant-message"]`) renders, and its inner
|
||||
Recharts SVG is visible
|
||||
|
||||
### 3. Pie chart flow
|
||||
|
||||
- [ ] Click the "Traffic pie chart" or "Market share" suggestion
|
||||
- [ ] Verify the assistant message bubble renders an SVG (the donut
|
||||
pie) inside
|
||||
|
||||
### 4. Free-form chat
|
||||
|
||||
- [ ] Type "Hello" and press Enter
|
||||
- [ ] Verify the assistant streams back a text response in a fresh
|
||||
`[data-testid="copilot-assistant-message"]` bubble
|
||||
|
||||
### 5. Hygiene
|
||||
|
||||
- [ ] No console errors during normal usage
|
||||
- [ ] No layout breakage with a very long input
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat composer mounts within ~3 seconds
|
||||
- Suggestion pills render alongside an empty chat
|
||||
- Tool-driven chart bubbles materialize within ~30 seconds for short
|
||||
prompts; the SVG renders progressively as Recharts mounts
|
||||
@@ -0,0 +1,82 @@
|
||||
# QA: Headless Chat (Complete) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/headless-complete` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `headless_complete` graph (backend tools: `get_weather`, `get_stock_price`)
|
||||
- The demo wires `agent="headless-complete"` at `/api/copilotkit-mcp-apps` (shared with the mcp-apps cell) so the Excalidraw MCP server at `MCP_SERVER_URL || https://mcp.excalidraw.com` is available
|
||||
- Note: the only `data-testid` in the source is `headless-complete-messages` on the scrollable messages container in `message-list.tsx`. Other checks rely on verbatim text, role selectors, and Tailwind utility classes
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/headless-complete`; verify the page renders within 3s with a centered card (max-width 3xl, full-height) on a `bg-gray-50` background
|
||||
- [ ] Verify the custom header renders with `<h1>` text "Headless Chat (Complete)" and subtext "Built from scratch on useAgent — no CopilotChat."
|
||||
- [ ] Verify the scrollable messages container (`[data-testid="headless-complete-messages"]`) is present and shows the empty-state hint "Try weather, a stock, a highlighted note, or an Excalidraw sketch."
|
||||
- [ ] Verify the custom composer renders at the bottom: a `<textarea>` with placeholder "Type a message..." and a `<button type="submit">Send</button>` (disabled while textarea is empty)
|
||||
- [ ] Confirm there is no `.copilotKitChat`, `.copilotKitMessages`, or `.copilotKitMessage` element in the DOM — the cell is truly headless and does NOT render `<CopilotChatMessageView>` or `<CopilotChatAssistantMessage>`
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Custom Composer + Send/Stop Toggle
|
||||
|
||||
- [ ] Type "Hello"; verify the Send button enables (goes from `bg-[#DBDBE5]` disabled to `bg-[#010507]` active)
|
||||
- [ ] Press `Enter`; verify the message submits and the textarea clears; press `Shift+Enter` in a follow-up message and verify a newline is inserted without submitting
|
||||
- [ ] While the agent is running, verify the textarea becomes disabled (`bg-[#FAFAFC]` muted), its placeholder switches to "Agent is working...", and the right-hand button swaps from "Send" to a red `bg-[#FA5F67]` "Stop" button
|
||||
- [ ] Click "Stop" mid-stream; verify `copilotkit.stopAgent({ agent })` fires and the button reverts to "Send" once `isRunning` returns false
|
||||
|
||||
#### Message List + Bubbles (pure chrome)
|
||||
|
||||
- [ ] Send "Hello"; within 10s verify:
|
||||
- [ ] A user bubble renders right-aligned (`flex justify-end`), rounded with `rounded-2xl rounded-br-sm`, `bg-[#010507] text-white` at `max-w-[75%]`, text "Hello"
|
||||
- [ ] A typing indicator (small pulsing gray dot in a `bg-[#F0F0F4]` rounded bubble) appears while `isRunning` is true and BEFORE any assistant content has streamed
|
||||
- [ ] The assistant bubble renders left-aligned (`flex justify-start`), `rounded-2xl rounded-bl-sm`, `bg-[#F0F0F4] text-[#010507]` at `max-w-[85%]`, with the assistant's plain-text response inside a `whitespace-pre-wrap break-words` div
|
||||
- [ ] Verify the messages container auto-scrolls to the bottom on each content-length change (send a long prompt whose response exceeds the viewport — scroll position should track the last line)
|
||||
- [ ] Verify empty assistant messages (mid-stream before any text/tool call) do NOT flash an empty `bg-[#F0F0F4]` box — `AssistantBubble`'s `isEmpty` check suppresses them
|
||||
|
||||
#### Multi-Turn Conversation
|
||||
|
||||
- [ ] Send a second message ("What else can you do?"); verify the prior user+assistant pair remain in the transcript in chronological order and the new pair is appended below
|
||||
- [ ] Verify each assistant bubble is independently sized (does not collapse neighbors) and the auto-scroll follows the newest content
|
||||
|
||||
#### Tool Rendering — WeatherCard (`useRenderTool` + backend `get_weather`)
|
||||
|
||||
- [ ] Send "What's the weather in Tokyo?"; within 15s verify a WeatherCard in the assistant bubble with eyebrow "FETCHING WEATHER" (loading) → "WEATHER" (complete), location "Tokyo" (`text-sm font-semibold capitalize`), temperature "68°", conditions "Sunny", wrapper `bg-[#EDEDF5] border-[#DBDBE5] rounded-xl max-w-xs`
|
||||
|
||||
#### Tool Rendering — StockCard (`useRenderTool` + backend `get_stock_price`)
|
||||
|
||||
- [ ] Send "What's AAPL trading at right now?"; within 15s verify a StockCard with eyebrow "LOADING" → "STOCK", ticker "AAPL" (`font-mono font-semibold`), price "$189.42", change "▲ 1.27%" in green `text-[#189370]`
|
||||
|
||||
#### Frontend Tool Rendering — HighlightNote (`useComponent` / `highlight_note`)
|
||||
|
||||
- [ ] Send "Highlight 'meeting at 3pm' in yellow."; within 15s verify a HighlightNote with eyebrow "NOTE", verbatim text "meeting at 3pm", and yellow variant classes `bg-[#FFF388]/30 border-[#FFF388]`
|
||||
- [ ] Optionally request pink/green/blue and verify corresponding `COLOR_CLASSES` are applied
|
||||
|
||||
#### Wildcard Catch-all + MCP Apps Activity (`useDefaultRenderTool` + `useRenderActivityMessage`)
|
||||
|
||||
- [ ] Send "Use Excalidraw to sketch a simple system diagram."; within 30s verify:
|
||||
- [ ] The activity message renders inline as a sandboxed Excalidraw iframe (built-in `MCPAppsActivityRenderer`), proving the hand-rolled `useRenderActivityMessage` path in `use-rendered-messages.tsx`
|
||||
- [ ] Any ancillary tool-call (not `get_weather` / `get_stock_price` / `highlight_note`) gets a visible default card via `useDefaultRenderTool` — not silently dropped
|
||||
- [ ] DevTools → Console shows no errors referencing the MCP server URL
|
||||
|
||||
#### Reasoning + Suggestions
|
||||
|
||||
- [ ] If the agent emits any `role: "reasoning"` messages, verify each renders via the imported `CopilotChatReasoningMessage` leaf inside an assistant bubble (the only chat primitive imported in `use-rendered-messages.tsx`)
|
||||
- [ ] Four suggestion strings are registered via `useConfigureSuggestions` with `available: "always"` ("Weather in Tokyo", "AAPL stock price", "Highlight a note", "Sketch a diagram") — exercised by manually sending the matching prompts above
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to submit an empty textarea; verify the Send button is disabled and Enter is a no-op (no user bubble, no run)
|
||||
- [ ] While `isRunning` is true, verify additional keystrokes cannot trigger a second run (`handleSubmit`'s `if (!text || isRunning) return;` guard)
|
||||
- [ ] Send a ~500-character message; verify the user bubble wraps within its 75% max-width via `break-words` without horizontal scroll
|
||||
- [ ] Navigate away mid-run; verify the unmount cleanup (`ac.abort()` + `agent.detachActiveRun()`) does not produce an uncaught rejection in DevTools → Console (connect/run rejections are swallowed by design)
|
||||
- [ ] With the backend stopped, send a message; verify `console.error("headless-complete: runAgent failed", err)` is emitted but no uncaught exception leaks, and the Send/Stop UI recovers to the idle state
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads within 3 seconds; first plain-text response within 10 seconds
|
||||
- Tool renders (WeatherCard, StockCard, HighlightNote) surface within 15 seconds of the triggering prompt
|
||||
- Excalidraw MCP activity surface renders within 30 seconds
|
||||
- Full generative-UI weave is reconstructed without `<CopilotChatMessageView>` / `<CopilotChatAssistantMessage>`: assistant text + tool-call renders (per-tool + catch-all) + reasoning + activity messages all appear through the hand-rolled `useRenderedMessages` composition
|
||||
- No flash of empty assistant bubbles while streaming; no uncaught console errors during any flow above
|
||||
@@ -0,0 +1,67 @@
|
||||
# QA: Headless Chat (Simple) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/headless-simple` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the neutral `sample_agent` graph
|
||||
- The demo wires `agent="headless-simple"` at `/api/copilotkit` (neutral assistant cell)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text, role/button selectors, and Tailwind utility-class structure
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/headless-simple`; verify the page renders within 3s with a centered card (max-width 4xl) on a `bg-gray-50` background
|
||||
- [ ] Verify the custom heading "Headless Chat (Simple)" is visible (an `<h1>` — NOT a `<CopilotChat />` primitive)
|
||||
- [ ] Verify the empty-state text "No messages yet. Say hi!" is rendered inside the message panel
|
||||
- [ ] Verify the custom composer renders below the message panel: a `<textarea>` with placeholder "Type a message. Ask me to 'show a card about cats'." and a disabled `<button>` labeled "Send"
|
||||
- [ ] Verify the Send button is disabled while the textarea is empty (attribute `disabled` present, visual `opacity-50`)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Custom Composer (not default CopilotChat)
|
||||
|
||||
- [ ] Confirm there is no `.copilotKitChat`, `.copilotKitInput`, or `.copilotKitMessages` element in the DOM — the demo is built directly on `useAgent` and does not render `<CopilotChat />`
|
||||
- [ ] Type "Hello" in the textarea; verify the Send button becomes enabled (no longer `disabled`) and its background is the solid blue utility `bg-blue-600`
|
||||
- [ ] Press `Enter` (without Shift); verify the message submits, the textarea clears, and the user bubble appears right-aligned with `bg-blue-600 text-white` rounded styling and max-width 80%
|
||||
- [ ] Type another message and press `Shift+Enter`; verify a newline is inserted inside the textarea and the message is NOT submitted
|
||||
|
||||
#### Send → Receive → Render (minimal round-trip)
|
||||
|
||||
- [ ] Send "Hi"; verify within 10s:
|
||||
- [ ] A user bubble appears (right-aligned, blue, verbatim "Hi")
|
||||
- [ ] A transient "Agent is thinking..." text indicator appears while `agent.isRunning` is true
|
||||
- [ ] An assistant bubble appears left-aligned with `bg-gray-100 text-gray-900` rounded styling, max-width 90%, containing the assistant's text response
|
||||
- [ ] After the response settles, verify the "Agent is thinking..." indicator is gone (`agent.isRunning` flipped false) and the Send button re-enables when the textarea has content
|
||||
|
||||
#### Frontend Tool Rendering via `useComponent` (`show_card`)
|
||||
|
||||
- [ ] Send "Show a card about cats" (or "show a card titled Cats with a body about cats")
|
||||
- [ ] Within 15s verify the assistant invokes the `show_card` frontend tool and a `ShowCard` renders inline inside the assistant bubble area with:
|
||||
- [ ] A bold title (e.g. "Cats") in `font-semibold text-gray-900`
|
||||
- [ ] A body paragraph in `text-sm text-gray-700` with `whitespace-pre-wrap`
|
||||
- [ ] The card is wrapped in a white rounded container with `border border-gray-300` and a small shadow
|
||||
- [ ] Verify the tool call was routed through `copilotkit.runAgent({ agent })` (frontend tools registered via `useComponent` reach the agent) — confirm by the card rendering at all; if the route had used `agent.runAgent()` directly the card would not appear
|
||||
|
||||
#### Multi-Turn
|
||||
|
||||
- [ ] Send a second message (e.g. "Thanks!"); verify the user bubble is appended below the prior assistant content and a second assistant bubble appears without clearing or rearranging prior messages
|
||||
- [ ] Verify the message list preserves chronological order (oldest at top, newest at bottom) and the empty-state "No messages yet. Say hi!" text is gone after the first send
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] With the textarea empty or whitespace-only, confirm Enter is a no-op — no user bubble appears, no "Agent is thinking..." indicator
|
||||
- [ ] While `agent.isRunning` is true (long response streaming), verify:
|
||||
- [ ] The Send button is `disabled`
|
||||
- [ ] Pressing Enter in the textarea is a no-op (the `send()` guard `if (!text || agent.isRunning) return;` blocks concurrent sends)
|
||||
- [ ] Send a ~500-character message; verify the user bubble wraps within its 80% max-width without horizontal scroll and the layout does not break
|
||||
- [ ] With the backend stopped, send a message; verify the promise rejection is swallowed silently (`.catch(() => {})`) and no uncaught error is surfaced in DevTools → Console, but the user bubble remains and `agent.isRunning` eventually returns false
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads within 3 seconds; first assistant response within 10 seconds; `show_card` renders within 15 seconds of the triggering prompt
|
||||
- Custom composer (textarea + Send button) renders — no default CopilotChat DOM surfaces
|
||||
- User bubbles: right-aligned, `bg-blue-600 text-white`, max-width 80%
|
||||
- Assistant bubbles: left-aligned, `bg-gray-100 text-gray-900`, max-width 90%
|
||||
- Frontend `useComponent` tool (`show_card`) renders inline via `useRenderToolCall` inside the assistant message
|
||||
- No uncaught console errors during any flow above
|
||||
@@ -0,0 +1,73 @@
|
||||
# QA: In-App Human in the Loop — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/hitl-in-app` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `hitl_in_app` graph
|
||||
- Note: Unlike the in-chat HITL demo, the approval UI here is an app-level modal portal'd to `document.body` and is NOT a child of the chat transcript.
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/hitl-in-app`; verify the page renders within 3s with the "Support Inbox" panel filling the viewport and the `CopilotPopup` open by default in the bottom-right corner
|
||||
- [ ] Verify the left header shows the eyebrow "Support Inbox", heading "Open tickets", and the instruction text mentioning "approval dialog here in the app — outside the chat"
|
||||
- [ ] Verify exactly 3 ticket cards render with test ids `ticket-12345`, `ticket-12346`, `ticket-12347`, each showing the customer name (Jordan Rivera / Priya Shah / Morgan Lee), subject line, and status pill ("Open" green, "Escalating" amber)
|
||||
- [ ] Verify ticket #12345 displays "Disputed amount: $50.00"
|
||||
- [ ] Verify the `CopilotPopup` chat input placeholder is visible (popup opens by default) and no approval dialog is rendered on initial load
|
||||
- [ ] Send "Hello" and verify the agent responds within 10s
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify all 3 suggestion pills are visible with verbatim titles:
|
||||
- "Approve refund for #12345"
|
||||
- "Downgrade plan for #12346"
|
||||
- "Escalate ticket #12347"
|
||||
|
||||
#### Approval Flow — Approve Path (useFrontendTool async handler)
|
||||
|
||||
- [ ] Click the "Approve refund for #12345" suggestion (or type the equivalent prompt)
|
||||
- [ ] Within 15s verify an approval modal appears with `data-testid="approval-dialog-overlay"` (fullscreen fixed backdrop with `backdrop-blur-sm`) and `data-testid="approval-dialog"` (centered card with `role="dialog"` and `aria-modal="true"`)
|
||||
- [ ] Verify the modal is rendered at the document root (portaled via `createPortal(content, document.body)`) — confirm in DevTools that `approval-dialog-overlay` is a direct descendant of `<body>`, NOT nested inside the `CopilotPopup` container
|
||||
- [ ] Verify the modal shows the eyebrow "Action requires your approval", a heading containing the action summary (with concrete numbers such as "$50" and "#12345"), and optional context block below
|
||||
- [ ] Verify the textarea `data-testid="approval-dialog-reason"` is present with placeholder "Add a short note the assistant will see…"
|
||||
- [ ] Type a short note (e.g. "Verified duplicate charge") into the reason textarea
|
||||
- [ ] Click the `data-testid="approval-dialog-approve"` button (labeled "Approve")
|
||||
- [ ] Verify the modal closes immediately and does NOT re-open
|
||||
- [ ] Verify the agent resumes within 10s and replies in the chat with a one/two-sentence confirmation acknowledging the action is being processed (should reference the refund / #12345)
|
||||
|
||||
#### Approval Flow — Reject Path
|
||||
|
||||
- [ ] In a fresh conversation, click "Downgrade plan for #12346"
|
||||
- [ ] Verify the approval modal re-opens (same testids as above) with a heading referencing Priya Shah / #12346 / Starter plan
|
||||
- [ ] Type a rejection reason (e.g. "Customer must confirm in writing first")
|
||||
- [ ] Click `data-testid="approval-dialog-reject"` (labeled "Reject")
|
||||
- [ ] Verify the modal closes immediately
|
||||
- [ ] Verify the agent's reply acknowledges the rejection in one or two sentences, reflects the rejection reason back, and does NOT claim the downgrade was performed
|
||||
|
||||
#### Empty-Reason Path
|
||||
|
||||
- [ ] Trigger the approval flow a third time via "Escalate ticket #12347"
|
||||
- [ ] Leave the reason textarea empty and click Approve
|
||||
- [ ] Verify the modal closes and the agent still proceeds with the action (reason is optional — omitted when empty)
|
||||
|
||||
#### Modal Is Outside the Chat (Contract Check)
|
||||
|
||||
- [ ] While the modal is open, verify the chat transcript inside the popup is still scrollable / visible and does NOT contain an inline copy of the approval UI
|
||||
- [ ] Confirm that closing the modal via Approve/Reject is the only resolution path — there is no inline "approve" button rendered inside a chat bubble
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Trigger the approval flow and dismiss via Approve without touching the reason field — verify no console errors and the promise resolves
|
||||
- [ ] Verify no uncaught console errors during any approve / reject cycle above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds; initial agent response within 10 seconds
|
||||
- Approval modal appears within 15 seconds of the triggering prompt
|
||||
- Modal is portaled to `<body>` (NOT nested in the chat) and closes on Approve/Reject
|
||||
- Approve path: agent acknowledges and proceeds; Reject path: agent acknowledges and stops, reflecting the reason when provided
|
||||
- No UI layout breaks, no uncaught console errors, no stuck-open modal
|
||||
@@ -0,0 +1,57 @@
|
||||
# QA: Human in the Loop — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the HITL demo page (`/demos/hitl`) > _Note: the URL path `/demos/hitl` is intentionally shorter than the feature id `hitl-in-chat`._
|
||||
- [ ] Verify the chat interface loads in a centered max-w-4xl container
|
||||
- [ ] Verify the chat input placeholder "Type a message" is visible
|
||||
- [ ] Send a basic message
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Simple plan" suggestion button is visible
|
||||
- [ ] Verify "Complex plan" suggestion button is visible
|
||||
- [ ] Click the "Simple plan" suggestion
|
||||
- [ ] Verify it triggers a message about planning a trip to Mars in 5 steps
|
||||
|
||||
#### Step Selection and Approval
|
||||
|
||||
The HITL demo renders a single StepSelector card regardless of whether
|
||||
the underlying flow uses an interrupt hook or a frontend HITL tool. The
|
||||
framework-specific hook name is an implementation detail covered in each
|
||||
package's demo source; QA below validates the user-visible card behavior.
|
||||
|
||||
- [ ] Send "Plan a trip to Mars in 5 steps"
|
||||
- [ ] Verify the StepSelector card appears (`data-testid="select-steps"`)
|
||||
- [ ] Verify step items are displayed with checkboxes (`data-testid="step-item"`)
|
||||
- [ ] Verify step text is visible (`data-testid="step-text"`)
|
||||
- [ ] Verify the selected count display shows "N/N selected"
|
||||
- [ ] Toggle a step checkbox off and verify the count decreases
|
||||
- [ ] Toggle it back on and verify the count increases
|
||||
- [ ] Click "Perform Steps (N)" / "Confirm (N)" button
|
||||
- [ ] Verify the agent continues processing after confirmation
|
||||
- [ ] In a new conversation, trigger the same flow and click "Reject" (where present)
|
||||
- [ ] Verify the card reflects the decision (Accepted / Rejected) and buttons disable
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- Step selector renders with toggleable checkboxes
|
||||
- Accept/Reject flow completes without errors
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,21 @@
|
||||
# QA: Interrupt (Headless) — LangGraph (Python)
|
||||
|
||||
> Stub — authored for column completeness. This is a testing-kind demo
|
||||
> (see `kind: "testing"` in feature-registry.json) and does not warrant a
|
||||
> full manual checklist.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/interrupt-headless
|
||||
- [ ] Send a scheduling prompt (e.g. "Book an intro call with sales") and verify a time-slot picker popup appears in the left app surface (not in the chat)
|
||||
- [ ] Click one of the time-slot buttons and verify the popup disappears and the agent confirms the booking back in the chat
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Interrupt resolves via the plain button grid (no `useInterrupt` render prop, no in-chat picker) and the agent continues the run with the picked slot
|
||||
@@ -0,0 +1,62 @@
|
||||
# QA: MCP Apps — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/mcp-apps` on the dashboard host
|
||||
- Agent backend is healthy; `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `mcp_apps` graph (registered as agent name `mcp-apps` — see `src/app/api/copilotkit-mcp-apps/route.ts`)
|
||||
- MCP server target: the public Excalidraw MCP app at `https://mcp.excalidraw.com` (override via `MCP_SERVER_URL`). Pinned `serverId: "excalidraw"` so URL changes don't silently break persisted activities
|
||||
- Note: the demo source contains no `data-testid` attributes and registers no custom activity renderer — CopilotKit's built-in `MCPAppsActivityRenderer` handles the sandboxed iframe automatically. Checks below rely on verbatim visible text, network traffic, and the iframe DOM
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/mcp-apps`; verify the page renders within 3s and a single `CopilotChat` pane is centered (max-width ~896px, rounded-2xl, full-height)
|
||||
- [ ] Verify the chat is wired to `runtimeUrl="/api/copilotkit-mcp-apps"` and `agent="mcp-apps"` (DevTools → Network: sending a message hits that endpoint)
|
||||
- [ ] Verify both suggestion pills are visible with verbatim titles:
|
||||
- "Draw a flowchart"
|
||||
- "Sketch a system diagram"
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s (no MCP activity iframe for plain text)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### MCP Server Connection (runtime `mcpApps.servers`)
|
||||
|
||||
- [ ] Send the first flow-chart prompt; in DevTools → Network, verify the POST to `/api/copilotkit-mcp-apps` succeeds (status 200) and the server-side runtime resolves tools from `https://mcp.excalidraw.com` (watch the server logs for the MCP Apps middleware attaching the Excalidraw tool set — notably `create_view`)
|
||||
- [ ] Verify no console errors mentioning the MCP server URL, auth, or tool-schema parse failures
|
||||
|
||||
#### MCP Tool Invocation (`create_view`)
|
||||
|
||||
- [ ] Click "Draw a flowchart"; within 60s verify the agent calls the `create_view` MCP tool exactly ONCE (per `SYSTEM_PROMPT` in `src/agents/mcp_apps_agent.py`: "Call `create_view` ONCE with 3-5 elements total") — confirm via DevTools → Network stream or backend logs
|
||||
- [ ] Verify the tool payload contains 3-5 Excalidraw elements (shapes + arrows + optional title text), each with a unique string `id`, and ends with ONE `cameraUpdate` sized `600x450` or `800x600`
|
||||
|
||||
#### Activity Renderer (built-in `MCPAppsActivityRenderer`)
|
||||
|
||||
- [ ] Within 60s of the tool call, verify a sandboxed `<iframe>` renders inline in the chat transcript (activity-message slot) pointed at the Excalidraw MCP UI resource
|
||||
- [ ] Verify the iframe has a `sandbox` attribute (CopilotKit's built-in renderer always sandboxes MCP UI resources)
|
||||
- [ ] Verify the iframe paints a flow-chart-shaped diagram: at least 3 shape nodes (rectangles, ellipses, or diamonds with text labels) connected by arrows, framed within the viewport (camera-update step from the system prompt)
|
||||
- [ ] Verify the assistant text below the iframe is a single short sentence describing what was drawn (per system prompt)
|
||||
|
||||
#### Server-Driven UI Update (second prompt, same thread)
|
||||
|
||||
- [ ] Without reloading, send the second suggestion "Sketch a system diagram"; within 60s verify a new activity iframe renders in-transcript containing a client → server → database layout (3 labeled shapes + 2 arrows)
|
||||
- [ ] Verify the previous flow-chart iframe is still present and un-stale in the scrollback (activity messages persist, matching the rationale for the pinned `serverId: "excalidraw"` in the runtime config)
|
||||
|
||||
#### End-to-End MCP Interaction (concrete, single-case)
|
||||
|
||||
- [ ] Send an explicit prompt: `"Use Excalidraw to draw exactly 2 rectangles labelled 'A' and 'B' connected by one arrow from A to B."`
|
||||
- [ ] Within 60s verify: (1) `create_view` is called ONCE with exactly 3 elements (2 rectangles + 1 arrow) plus the trailing `cameraUpdate`; (2) an iframe renders showing two labelled rectangles with a connecting arrow; (3) the assistant reply is one short sentence; (4) no duplicate `create_view` invocations or retries appear in network / logs
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Send "What is 2+2?"; verify the agent replies in plain text without invoking `create_view` (no iframe, no MCP activity in the stream)
|
||||
- [ ] DevTools → Console: walk through all flows above; verify no uncaught errors, no CORS failures referencing `mcp.excalidraw.com`, and no "sandbox" / iframe-permission warnings
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3s; plain-text response within 10s; MCP-backed iframe renders within 60s of prompt (bias is "correct-enough diagram fast" per system prompt, one `create_view` call)
|
||||
- MCP server connection to `https://mcp.excalidraw.com` succeeds and the Excalidraw tool set (including `create_view`) is advertised to the agent at request time
|
||||
- At least one concrete end-to-end MCP interaction completes: user prompt → `create_view` tool call → activity event → sandboxed iframe painting the requested diagram
|
||||
- The built-in `MCPAppsActivityRenderer` is used (no app-side `useRenderActivityMessage` / `renderActivityMessages` registration exists in `page.tsx` — per the `@region[no-frontend-renderer-needed]` contract)
|
||||
- No UI layout breaks, no uncaught console errors, no duplicate `create_view` invocations within a single prompt turn
|
||||
@@ -0,0 +1,92 @@
|
||||
# QA: Multimodal Attachments — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo deployed and accessible at `/demos/multimodal`
|
||||
- Railway service `showcase-langgraph-python` is healthy
|
||||
- `OPENAI_API_KEY` set on the Railway service (vision calls require it)
|
||||
- Agent is using a vision-capable model (`gpt-4o` or equivalent) — verifiable by
|
||||
inspecting the Railway logs for `src/agents/multimodal_agent.py` or by
|
||||
running one image round-trip
|
||||
- Sample files are bundled under `public/demo-files/`:
|
||||
- `sample.png` — a small PNG the vision model can describe
|
||||
- `sample.pdf` — a small one-page PDF mentioning "CopilotKit"
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/multimodal`
|
||||
- [ ] Verify the page header "Multimodal attachments" is visible
|
||||
- [ ] Verify the description text mentions image and PDF attachments
|
||||
- [ ] Verify the sample-row (`[data-testid="multimodal-sample-row"]`) is visible
|
||||
- [ ] Verify the "Try with sample image" and "Try with sample PDF" buttons are
|
||||
present and enabled
|
||||
- [ ] Verify `<CopilotChat />` renders a message composer with a paperclip /
|
||||
"Add attachments" button
|
||||
|
||||
### 2. Sample image path
|
||||
|
||||
- [ ] Click "Try with sample image"
|
||||
- [ ] Button label briefly flips to "Loading…"
|
||||
- [ ] Within 5 seconds an image attachment chip appears in the composer with a
|
||||
thumbnail preview
|
||||
- [ ] Type "Describe this image"
|
||||
- [ ] Click send
|
||||
- [ ] Within 60 seconds the agent responds with a description referring to the
|
||||
image content (e.g. mentions a logo, colors, or brand mark)
|
||||
|
||||
### 3. Sample PDF path
|
||||
|
||||
- [ ] Click "Try with sample PDF"
|
||||
- [ ] A document attachment chip appears in the composer (filename
|
||||
"sample.pdf" visible, document icon)
|
||||
- [ ] Type "Summarize this document"
|
||||
- [ ] Click send
|
||||
- [ ] Within 60 seconds the agent responds with text that mentions "CopilotKit"
|
||||
(the sample PDF contains the word multiple times)
|
||||
|
||||
### 4. Paperclip / real upload path (manual)
|
||||
|
||||
- [ ] Click the paperclip / "Add attachments" button
|
||||
- [ ] File picker opens filtered to `image/*` and `application/pdf`
|
||||
- [ ] Select a local image under 10 MB
|
||||
- [ ] Attachment chip renders in the composer within 2 seconds
|
||||
- [ ] Type "What's in this image?" and send
|
||||
- [ ] Agent responds with a description of the image content
|
||||
|
||||
### 5. Drag-and-drop
|
||||
|
||||
- [ ] Drag a local image onto the chat container
|
||||
- [ ] Chat surface shows a drop affordance
|
||||
- [ ] On drop, an attachment chip appears
|
||||
- [ ] Sending a prompt works the same as the paperclip path
|
||||
|
||||
### 6. Multi-attachment
|
||||
|
||||
- [ ] Inject the sample image AND the sample PDF (click both buttons sequentially)
|
||||
- [ ] Both chips are visible in the composer
|
||||
- [ ] Type "What do these two attachments have in common?"
|
||||
- [ ] Click send
|
||||
- [ ] Agent response acknowledges both attachments
|
||||
|
||||
### 7. Error Handling
|
||||
|
||||
- [ ] Try to attach a file over 10 MB via the paperclip
|
||||
- [ ] Verify `onUploadFailed` fires (console warning from the page) and the
|
||||
file is rejected without corrupting the composer
|
||||
- [ ] Try to attach an unsupported type (e.g. `.exe`) — the file picker filter
|
||||
excludes it or `onUploadFailed` fires with `reason: "invalid-type"`
|
||||
- [ ] Block `/demo-files/sample.png` in DevTools Network; click "Try with
|
||||
sample image"; the `multimodal-sample-error` span shows a helpful error
|
||||
without crashing the page
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Attachment chip appears within 2 seconds after upload or sample injection
|
||||
- Thumbnail renders for image attachments; document icon for PDFs
|
||||
- Agent response arrives within 60 seconds for single-attachment prompts
|
||||
(multimodal tokens are heavier than text-only)
|
||||
- No console errors during successful flows (warnings from `onUploadFailed`
|
||||
during intentional error cases are acceptable)
|
||||
- Error states are visible and recoverable — the user can retry
|
||||
@@ -0,0 +1,80 @@
|
||||
# QA: Open-Ended Generative UI (Advanced) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- Graph `open_gen_ui_advanced` is registered in the OGUI runtime (`api/copilotkit-ogui/route.ts`) with `openGenerativeUI.agents` including `"open-gen-ui-advanced"`
|
||||
- Sandbox-function handlers in `sandbox-functions.ts` are exported: `evaluateExpression` and `notifyHost`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the open-gen-ui-advanced demo page
|
||||
- [ ] Verify the `<CopilotChat>` renders full-height within the centered max-w-4xl container
|
||||
- [ ] Verify the input composer is visible
|
||||
- [ ] Send a basic message (e.g. "Hi")
|
||||
- [ ] Verify the agent calls `generateSandboxedUi` and a sandboxed iframe mounts in the assistant turn
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Calculator (calls evaluateExpression)" suggestion is visible
|
||||
- [ ] Verify "Ping the host (calls notifyHost)" suggestion is visible
|
||||
- [ ] Verify "Inline expression evaluator" suggestion is visible
|
||||
|
||||
#### Sandbox-to-Host: evaluateExpression (Calculator)
|
||||
|
||||
- [ ] Click the "Calculator (calls evaluateExpression)" suggestion
|
||||
- [ ] Verify a sandboxed iframe renders a calculator UI with digit buttons, operator buttons, a display, and an "=" button
|
||||
- [ ] Verify all buttons use `type="button"` — no `<form>` element is present inside the iframe
|
||||
- [ ] Enter an expression like `12 * (3 + 4.5)` via the calculator buttons (or the generated input)
|
||||
- [ ] Open browser devtools console BEFORE pressing "="
|
||||
- [ ] Press "="
|
||||
- [ ] Verify the host console logs `[open-gen-ui/advanced] evaluateExpression 12 * (3 + 4.5) = 90`
|
||||
- [ ] Verify the display updates to show `90` (the `res.value` returned by the host)
|
||||
- [ ] Verify a computed-history entry appears below the display
|
||||
- [ ] In a follow-up user turn, type "What was the result of my last calculation?" and verify the agent's text response references the computed value (round-trip: sandbox call -> host handler -> visible result -> agent awareness via subsequent turn context)
|
||||
|
||||
#### Sandbox-to-Host: notifyHost (Ping)
|
||||
|
||||
- [ ] In a new turn, click the "Ping the host (calls notifyHost)" suggestion
|
||||
- [ ] Verify a sandboxed iframe renders a card with a single "Say hi to the host" button
|
||||
- [ ] Open the browser devtools console
|
||||
- [ ] Click the button
|
||||
- [ ] Verify the host console logs `[open-gen-ui/advanced] notifyHost: Hi from the sandbox!`
|
||||
- [ ] Verify the card updates to display the returned confirmation object, including a `receivedAt` ISO-8601 timestamp and the echoed `message` field
|
||||
|
||||
#### Sandbox-to-Host: Inline Expression Evaluator
|
||||
|
||||
- [ ] In a new turn, click the "Inline expression evaluator" suggestion
|
||||
- [ ] Verify a sandboxed iframe renders a text input + "Evaluate" button (no `<form>`, button `type="button"`)
|
||||
- [ ] Enter `2 + 2` and click "Evaluate"
|
||||
- [ ] Verify the output area renders `4` (from `res.value`)
|
||||
- [ ] Enter an invalid expression `abc + 1` and click "Evaluate"
|
||||
- [ ] Verify the output area renders the error string from `res.error` (e.g. "Unsupported characters in expression.")
|
||||
|
||||
#### Sandbox Constraints
|
||||
|
||||
- [ ] Verify the iframe sandbox attribute is `sandbox="allow-scripts"` only (no `allow-forms`, no `allow-same-origin`)
|
||||
- [ ] Verify no network requests originate from the iframe (check DevTools Network filtered by iframe frame)
|
||||
- [ ] Verify the agent keeps its own chat message brief (1 sentence) — the rendered UI is the real output
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Enter an expression with unsupported characters (e.g. `alert(1)`) into the calculator/evaluator and confirm `res.ok === false` with error "Unsupported characters in expression."
|
||||
- [ ] Enter a divide-by-zero (`1/0`) — verify the handler returns `{ ok: false, error: "Not a finite number." }` and the UI renders the error path
|
||||
- [ ] Refresh the page mid-stream — verify no broken UI persists
|
||||
- [ ] Send an empty message — input should be rejected without error
|
||||
- [ ] Verify no console errors beyond the two intentional `console.log` statements from the sandbox-function handlers
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- First interactive sandboxed UI mounts within ~15 seconds of prompt submission
|
||||
- Sandbox -> host round-trip (button click -> `Websandbox.connection.remote.<fn>` -> visible result) completes without page reload
|
||||
- `evaluateExpression` returns `{ ok, value }` on valid input and `{ ok: false, error }` on rejected input
|
||||
- `notifyHost` returns `{ ok: true, receivedAt, message }` with a valid ISO-8601 timestamp
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,67 @@
|
||||
# QA: Open-Ended Generative UI (Minimal) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- Graph `open_gen_ui` is registered in the OGUI runtime (`api/copilotkit-ogui/route.ts`) with `openGenerativeUI.agents` including `"open-gen-ui"`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the open-gen-ui demo page
|
||||
- [ ] Verify the `<CopilotChat>` renders full-height within the centered max-w-4xl container
|
||||
- [ ] Verify the input composer is visible
|
||||
- [ ] Send a basic message (e.g. "Hi")
|
||||
- [ ] Verify the agent calls `generateSandboxedUi` and a sandboxed iframe is mounted inside the chat transcript
|
||||
- [ ] Verify the streaming placeholder message(s) appear before the final UI renders
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "3D axis visualization (model airplane)" suggestion is visible
|
||||
- [ ] Verify "How a neural network works" suggestion is visible
|
||||
- [ ] Verify "Quicksort visualization" suggestion is visible
|
||||
- [ ] Verify "Fourier: square wave from sines" suggestion is visible
|
||||
|
||||
#### Neural Network Prompt (concrete render)
|
||||
|
||||
- [ ] Click the "How a neural network works" suggestion
|
||||
- [ ] Verify short placeholder lines show while streaming (e.g. "Sketching the scene…", "Labelling axes…", "Wiring up the animation…")
|
||||
- [ ] Verify a sandboxed iframe activity is mounted in the assistant turn
|
||||
- [ ] Inside the iframe, verify an inline `<svg>` element renders (not stacked `<div>`s)
|
||||
- [ ] Verify three layers of nodes are visible (input ~4, hidden ~5, output ~2) with connecting lines
|
||||
- [ ] Verify layer labels are present (e.g. "Input", "Hidden", "Output") and a "Forward pass" caption
|
||||
- [ ] Verify activations animate (pulse forward from input to output in a loop) — indigo active, slate quiescent
|
||||
- [ ] Verify the visualization loops continuously without user interaction
|
||||
|
||||
#### Quicksort Prompt (secondary render)
|
||||
|
||||
- [ ] In a fresh turn, click the "Quicksort visualization" suggestion
|
||||
- [ ] Verify a new sandboxed iframe is rendered with ~10 SVG rect bars
|
||||
- [ ] Verify a caption text element updates as the sort progresses (e.g. "Partition around pivot", "Swap", "Recurse left")
|
||||
- [ ] Verify the pivot highlight uses amber (#f59e0b) and compared elements use indigo (#6366f1)
|
||||
|
||||
#### Design Skill Injection
|
||||
|
||||
- [ ] In any rendered output, verify the visualization respects the palette: indigo accent, emerald success, amber warning, slate neutrals
|
||||
- [ ] Verify the outer container has a white background with rounded corners and padded content (no bleed to viewport edges)
|
||||
- [ ] Verify the visualization includes a title line and subtitle line at the top
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send a trivially off-topic prompt (e.g. "banana banana banana"); verify the agent still renders a sandboxed UI or a graceful text response — no unhandled exception
|
||||
- [ ] Verify no console errors originating from the host page during rendering (sandbox-internal warnings may be present)
|
||||
- [ ] Verify the sandbox iframe does NOT attempt fetch / XHR / localStorage (network tab: no origin-leaked requests from the iframe)
|
||||
- [ ] Send an empty message — input should be rejected without error
|
||||
- [ ] Refresh the page mid-stream — verify no broken UI persists
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- First visualization mounts within ~15 seconds of prompt submission
|
||||
- Sandboxed iframe renders SVG-based content with labeled axes / layers / legend
|
||||
- Animations loop smoothly with CSS keyframes (no jank from setInterval)
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,55 @@
|
||||
# QA: Pre-Built Popup — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/prebuilt-popup` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health` or `/api/copilotkit` GET); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the neutral `sample_agent` graph (registered to the `prebuilt-popup` agent name)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text, role-based selectors, and DOM structure. The underlying agent is the neutral "helpful, concise assistant" (no frontend tools, no agent tools).
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/prebuilt-popup`; verify the main content renders with heading (h1, text: "Popup demo — look for the floating launcher") and paragraph body mentioning `<CopilotPopup />`
|
||||
- [ ] Verify a floating launcher button is visible in a corner of the viewport (typically bottom-right)
|
||||
- [ ] Verify the `<CopilotPopup />` window is OPEN by default (`defaultOpen={true}` in source) — a chat overlay sits on top of the page content with an input at the bottom
|
||||
- [ ] Verify the chat input placeholder is verbatim "Ask the popup anything..."
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Popup Open / Close
|
||||
|
||||
- [ ] With the popup open, click the popup's close control (X button in the popup header); verify the popup window collapses and only the floating launcher remains
|
||||
- [ ] Click the floating launcher; verify the popup re-opens showing the prior transcript (if any)
|
||||
- [ ] Verify opening/closing does not trigger a full page reload (URL remains `/demos/prebuilt-popup`, main content stays mounted)
|
||||
- [ ] Verify the popup overlays the page content rather than pushing layout (unlike the docked sidebar form factor)
|
||||
|
||||
#### Suggestions (`useConfigureSuggestions`)
|
||||
|
||||
- [ ] With the popup open, verify a suggestion pill titled "Say hi" is rendered in the chat surface (configured with `available: "always"`)
|
||||
- [ ] Click the "Say hi" pill; verify it sends the message "Say hi from the popup!" and an assistant text response appears within 10s
|
||||
|
||||
#### Chat Round-Trip
|
||||
|
||||
- [ ] Type "Hello" into the popup chat input ("Ask the popup anything..." placeholder) and submit; verify the user bubble appears, followed within 10s by an assistant text response
|
||||
- [ ] Send a second message; verify a second valid response appears and the transcript auto-scrolls to the latest turn
|
||||
- [ ] Close the popup and re-open it; verify the existing transcript persists within the open session
|
||||
|
||||
#### Agent Wiring
|
||||
|
||||
- [ ] Confirm (via DevTools → Network) that chat submissions POST to `/api/copilotkit` with agent name `prebuilt-popup` in the payload; response streams back as SSE with no 4xx/5xx status
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no network request)
|
||||
- [ ] Resize the viewport to ~375px wide (mobile); verify the popup adapts (full-width or close to it) without clipping the input, header, or close button
|
||||
- [ ] Stop the backend, send a message; verify the UI surfaces a visible error path rather than hanging silently; DevTools → Console shows no uncaught errors during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page + popup render within 3 seconds
|
||||
- Assistant text response within 10 seconds
|
||||
- Popup open/close animates smoothly with no layout jank
|
||||
- Floating launcher remains accessible (not clipped) at all tested viewport sizes
|
||||
- No UI layout breaks, no uncaught console errors
|
||||
- The neutral agent (no tools) simply chats — no frontend tool registrations, no tool-call UI expected in this demo
|
||||
@@ -0,0 +1,53 @@
|
||||
# QA: Pre-Built Sidebar — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/prebuilt-sidebar` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health` or `/api/copilotkit` GET); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the neutral `sample_agent` graph (registered to the `prebuilt-sidebar` agent name)
|
||||
- Note: the demo source contains no `data-testid` attributes. Checks below rely on verbatim visible text, role-based selectors, and DOM structure. The underlying agent is the neutral "helpful, concise assistant" (no frontend tools, no agent tools).
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/prebuilt-sidebar`; verify the main content renders with heading (h1, text: "Sidebar demo — click the launcher") and paragraph body mentioning `<CopilotSidebar />`
|
||||
- [ ] Verify the `<CopilotSidebar />` is rendered docked to one edge of the viewport (typically the right edge) and is OPEN by default (`defaultOpen={true}` in source)
|
||||
- [ ] Verify the sidebar contains a chat input and its own launcher/toggle button
|
||||
- [ ] Verify the page body (main content) remains visible alongside the sidebar (sidebar does not fully overlay the page — docked form factor, not modal)
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Sidebar Toggle
|
||||
|
||||
- [ ] Click the sidebar launcher/close button; verify the sidebar collapses and the main content expands to fill the freed width
|
||||
- [ ] Click the launcher again; verify the sidebar re-opens to its previous width
|
||||
- [ ] Verify toggling does not trigger a full page reload (URL remains `/demos/prebuilt-sidebar`, no flash of unstyled content)
|
||||
|
||||
#### Suggestions (`useConfigureSuggestions`)
|
||||
|
||||
- [ ] With the sidebar open, verify a suggestion pill titled "Say hi" is rendered in the chat surface (configured via `useConfigureSuggestions` with `available: "always"`)
|
||||
- [ ] Click the "Say hi" pill; verify it sends the message "Say hi!" and an assistant text response appears within 10s
|
||||
|
||||
#### Chat Round-Trip
|
||||
|
||||
- [ ] Type "Hello" into the sidebar chat input and submit; verify the user bubble appears, followed within 10s by an assistant text response
|
||||
- [ ] Send a follow-up ("What can you help with?"); verify a coherent second response referencing prior turn is NOT required (agent has no persistent memory beyond the session thread) but a valid response appears
|
||||
- [ ] Verify the transcript scrolls to the latest message automatically
|
||||
|
||||
#### Agent Wiring
|
||||
|
||||
- [ ] Confirm (via DevTools → Network) that chat submissions POST to `/api/copilotkit` with agent name `prebuilt-sidebar` in the payload; response streams back as SSE with no 4xx/5xx status
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no network request, no assistant response)
|
||||
- [ ] Resize the viewport to ~375px wide (mobile); verify the sidebar adapts (overlays content or stacks) without clipping the input or launcher
|
||||
- [ ] Stop the backend, send a message; verify the UI surfaces a visible error path rather than hanging silently; DevTools → Console shows no uncaught errors during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page + sidebar render within 3 seconds
|
||||
- Assistant text response within 10 seconds
|
||||
- Sidebar toggle is instant (<200ms) with no layout jank
|
||||
- No UI layout breaks, no uncaught console errors
|
||||
- The neutral agent (no tools) simply chats — no frontend tool registrations, no tool-call UI expected in this demo
|
||||
@@ -0,0 +1,73 @@
|
||||
# QA: Read-Only Agent Context — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- Graph `readonly_state_agent_context` is registered in the runtime (see `api/copilotkit/route.ts`)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the readonly-state-agent-context demo page
|
||||
- [ ] Verify the "Agent Context" card is visible (`data-testid="context-card"`)
|
||||
- [ ] Verify the description text references `useAgentContext`: "Read-only context provided to the agent via `useAgentContext`. The agent cannot modify these."
|
||||
- [ ] Verify the chat panel renders on the right with placeholder "Ask about your context..."
|
||||
- [ ] Send a basic message (e.g. "Hello") via the chat
|
||||
- [ ] Verify the agent responds with a text message
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Initial Context State
|
||||
|
||||
- [ ] Verify the Name input (`data-testid="ctx-name"`) defaults to "Atai"
|
||||
- [ ] Verify the Timezone dropdown (`data-testid="ctx-timezone"`) defaults to "America/Los_Angeles"
|
||||
- [ ] Verify the Timezone dropdown offers: America/Los_Angeles, America/New_York, Europe/London, Europe/Berlin, Asia/Tokyo, Australia/Sydney
|
||||
- [ ] Verify the Recent Activity checkboxes list: "Viewed the pricing page", "Added 'Pro Plan' to cart", "Watched the product demo video", "Started the 14-day free trial", "Invited a teammate"
|
||||
- [ ] Verify two activities are checked by default: "Viewed the pricing page" and "Watched the product demo video"
|
||||
- [ ] Verify the Published Context JSON preview (`data-testid="ctx-state-json"`) shows `{ "name": "Atai", "timezone": "America/Los_Angeles", "recentActivity": [...] }`
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Who am I?" suggestion is visible
|
||||
- [ ] Verify "Suggest next steps" suggestion is visible
|
||||
- [ ] Verify "Plan my morning" suggestion is visible
|
||||
|
||||
#### Agent Reads User Name (useAgentContext)
|
||||
|
||||
- [ ] Click the "Who am I?" suggestion (or ask "What is my name?")
|
||||
- [ ] Verify the agent response addresses the user as "Atai"
|
||||
- [ ] Edit the Name input (`data-testid="ctx-name"`) to "Jamie"
|
||||
- [ ] Verify the Published Context JSON updates to show `"name": "Jamie"`
|
||||
- [ ] Ask "What is my name?" again
|
||||
- [ ] Verify the agent now responds with "Jamie" (not "Atai")
|
||||
|
||||
#### Agent Reads Timezone
|
||||
|
||||
- [ ] Change the Timezone dropdown (`data-testid="ctx-timezone"`) to "Asia/Tokyo"
|
||||
- [ ] Verify the Published Context JSON updates to `"timezone": "Asia/Tokyo"`
|
||||
- [ ] Click the "Plan my morning" suggestion
|
||||
- [ ] Verify the agent's response references Tokyo / JST / Asia/Tokyo when discussing the time
|
||||
|
||||
#### Agent Reads Recent Activity
|
||||
|
||||
- [ ] Uncheck all default activities, then check only "Started the 14-day free trial" and "Invited a teammate"
|
||||
- [ ] Verify the Published Context JSON shows the new recentActivity array
|
||||
- [ ] Click the "Suggest next steps" suggestion
|
||||
- [ ] Verify the agent's response references the trial and/or invited-teammate activities (not the pricing page or demo video)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Clear the Name input to an empty string and ask "What is my name?" — agent should handle gracefully (no crash)
|
||||
- [ ] Send an empty chat message — input should be rejected without error
|
||||
- [ ] Verify no console errors during normal usage
|
||||
- [ ] Verify the agent cannot modify context values (Name / Timezone / Activity checkboxes stay user-controlled)
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Context card and chat load within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- Every change to Name / Timezone / Recent Activity reflects in the Published Context JSON immediately
|
||||
- Agent responses reflect the CURRENT context values on every turn (no stale context)
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,21 @@
|
||||
# QA: Reasoning (Default) — LangGraph (Python)
|
||||
|
||||
> Stub — authored for column completeness. This demo verifies the
|
||||
> built-in `CopilotChatReasoningMessage` renders without a custom slot
|
||||
> and does not warrant a full manual checklist.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/reasoning-default
|
||||
- [ ] Send any prompt that elicits a reasoning response and verify the built-in `CopilotChatReasoningMessage` collapsible card renders the reasoning tokens
|
||||
- [ ] Verify no custom reasoning slot is wired (default styling only — no `ReasoningBlock` or bespoke container)
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Reasoning renders via CopilotKit's default `CopilotChatReasoningMessage` component with zero frontend configuration
|
||||
@@ -0,0 +1,61 @@
|
||||
# QA: Shared State (Read + Write) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/shared-state-read-write` on the dashboard host
|
||||
- Agent backend is healthy (`/api/health`); `OPENAI_API_KEY` is set on Railway; `LANGGRAPH_DEPLOYMENT_URL` points at a LangGraph deployment exposing the `shared-state-read-write` graph
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/shared-state-read-write`; verify the page renders within 3s with the preferences + notes cards in the main column and the `CopilotSidebar` open by default on the right (cards stack vertically once the viewport drops below `xl` / 1280px)
|
||||
- [ ] Verify `data-testid="preferences-card"` is visible with heading "Your preferences"
|
||||
- [ ] Verify `data-testid="notes-card"` is visible with heading "Agent notes" and empty-state `data-testid="notes-empty"` reading "No notes yet. Ask the agent to remember something."
|
||||
- [ ] Verify the chat input placeholder is "Chat with the agent..."
|
||||
- [ ] Verify all 3 suggestion pills are visible with verbatim titles: "Greet me", "Remember something", "Plan a weekend"
|
||||
- [ ] Send "Hello" and verify an assistant text response appears within 10s
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### UI Writes -> Agent Reads (preferences via `agent.setState`)
|
||||
|
||||
- [ ] Type "Atai" into `data-testid="pref-name"`; verify `data-testid="pref-state-json"` updates to include `"name": "Atai"`
|
||||
- [ ] Change `data-testid="pref-tone"` to `formal`; verify the JSON preview reflects `"tone": "formal"`
|
||||
- [ ] Change `data-testid="pref-language"` to `Spanish`; verify the JSON preview reflects `"language": "Spanish"`
|
||||
- [ ] Click the `Cooking` and `Travel` interest pills; verify both show the selected style (border `#BEC2FF`, bg `#BEC2FF1A`) and the JSON preview's `interests` array contains both entries
|
||||
- [ ] Send "What do you know about me?"; verify within 10s the assistant reply references the name "Atai", a formal tone, Spanish, and the Cooking/Travel interests (the `PreferencesInjectorMiddleware` injects these into the system prompt each turn)
|
||||
- [ ] Click the "Plan a weekend" suggestion; verify the reply is tailored to the selected interests
|
||||
|
||||
#### Agent Writes -> UI Reads (notes via `set_notes` tool)
|
||||
|
||||
- [ ] Click the "Remember something" suggestion (sends "Remember that I prefer morning meetings and that I don't eat dairy.")
|
||||
- [ ] Within 15s verify `data-testid="notes-list"` appears in the notes card and contains at least 2 `data-testid="note-item"` entries mentioning "morning meetings" and "dairy"
|
||||
- [ ] Verify `data-testid="notes-empty"` is no longer rendered
|
||||
- [ ] Send "Also remember I live in Berlin."; verify within 15s the notes list grows (previous notes preserved, new note added) — confirms the agent passes the FULL updated list per `set_notes` contract
|
||||
|
||||
#### UI Writes Back to Agent-Authored Slice (clear notes)
|
||||
|
||||
- [ ] With notes present, verify `data-testid="notes-clear-button"` is visible
|
||||
- [ ] Click the Clear button; verify the notes list disappears and `data-testid="notes-empty"` re-renders
|
||||
- [ ] Ask "What do you remember about me?"; verify the agent no longer cites the cleared notes (state was written back by the UI via `agent.setState({ notes: [] })`)
|
||||
|
||||
#### Multi-Turn State Persistence
|
||||
|
||||
- [ ] Change tone to `playful` and add the `Music` interest; send "Write me a one-line haiku greeting."; verify the reply is playful and references music
|
||||
- [ ] Send a follow-up "Do it again in French."; verify the reply remains playful, switches to French, and still acknowledges the music interest — confirms preferences persist across turns without being re-sent
|
||||
- [ ] Reload the page; verify preferences reset to defaults (`tone: casual`, `language: English`, empty interests, empty name) and notes reset to empty (state is per-session, seeded by the page's `useEffect`)
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Attempt to send an empty message; verify it is a no-op (no user bubble, no assistant response)
|
||||
- [ ] Deselect all interests and clear the name; send "Who am I?"; verify the agent answers without crashing (middleware skips injection when `preferences` is empty)
|
||||
- [ ] Verify DevTools -> Console shows no uncaught errors during any flow above
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads within 3 seconds; assistant text response within 10 seconds
|
||||
- Preferences writes are reflected in `pref-state-json` synchronously on change
|
||||
- Agent-authored notes appear in `notes-card` within 15 seconds of a "remember" prompt, and the full prior list is preserved on subsequent `set_notes` calls
|
||||
- Clear button round-trips UI -> agent state and the agent loses access to the cleared notes on the next turn
|
||||
- No UI layout breaks, no uncaught console errors
|
||||
@@ -0,0 +1,75 @@
|
||||
# QA: Shared State (Reading) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the shared-state-read demo page
|
||||
- [ ] Verify the recipe card form loads (`data-testid="recipe-card"`)
|
||||
- [ ] Verify the CopilotSidebar opens by default with title "AI Recipe Assistant"
|
||||
- [ ] Send a message via the sidebar
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Initial Recipe State
|
||||
|
||||
- [ ] Verify the recipe title input shows "Make Your Recipe"
|
||||
- [ ] Verify the cooking time dropdown defaults to "45 min"
|
||||
- [ ] Verify the skill level dropdown defaults to "Intermediate"
|
||||
- [ ] Verify the default ingredients are displayed:
|
||||
- [ ] Carrots (3 large, grated) with carrot emoji
|
||||
- [ ] All-Purpose Flour (2 cups) with wheat emoji
|
||||
- [ ] Verify the default instruction is displayed: "Preheat oven to 350°F (175°C)"
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Create Italian recipe" suggestion is visible
|
||||
- [ ] Verify "Make it healthier" suggestion is visible
|
||||
- [ ] Verify "Suggest variations" suggestion is visible
|
||||
|
||||
#### Recipe Editing (Local State)
|
||||
|
||||
- [ ] Edit the recipe title and verify it updates
|
||||
- [ ] Change the skill level dropdown and verify it updates
|
||||
- [ ] Change the cooking time dropdown and verify it updates
|
||||
- [ ] Toggle a dietary preference checkbox (e.g. "Vegetarian") and verify it's checked
|
||||
- [ ] Click "+ Add Ingredient" (`data-testid="add-ingredient-button"`) and verify a new empty row appears
|
||||
- [ ] Edit an ingredient name and amount
|
||||
- [ ] Remove an ingredient by clicking the "x" button
|
||||
- [ ] Click "+ Add Step" and verify a new instruction row appears
|
||||
- [ ] Edit an instruction and verify it saves
|
||||
- [ ] Remove an instruction by clicking the "x" button
|
||||
|
||||
#### AI-Powered Recipe Updates (useAgent with shared state)
|
||||
|
||||
- [ ] Click "Create Italian recipe" suggestion
|
||||
- [ ] Verify the agent updates the recipe title, ingredients, and instructions
|
||||
- [ ] Verify the ping indicator appears on changed sections
|
||||
- [ ] Verify the "Improve with AI" button (`data-testid="improve-button"`) changes to "Please Wait..." while loading
|
||||
- [ ] Click "Improve with AI" and verify the recipe is enhanced
|
||||
|
||||
#### Agent Reads Frontend State
|
||||
|
||||
- [ ] Edit the recipe (change title, add ingredients)
|
||||
- [ ] Ask the agent "What recipe am I making?"
|
||||
- [ ] Verify the agent's response references the current recipe state
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
- [ ] Verify the "Improve with AI" button is disabled while loading
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Recipe card and sidebar load within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- Recipe state syncs bidirectionally between UI and agent
|
||||
- Ping indicators highlight changed sections
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,41 @@
|
||||
# QA: State Streaming — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the shared-state-streaming demo page
|
||||
- [ ] Verify the chat interface loads with title "State Streaming"
|
||||
- [ ] Verify the chat input placeholder "Type a message..." is visible
|
||||
- [ ] Send a basic message (e.g. "Hello! What can you do?")
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Get started" suggestion button is visible
|
||||
|
||||
#### Note: Stub Demo
|
||||
|
||||
> **Status: Stub** — This demo is currently a stub (TODO: implement)
|
||||
|
||||
- [ ] Verify the basic CopilotChat loads and accepts messages
|
||||
- [ ] Verify the agent responds to messages
|
||||
- [ ] No custom UI components are expected beyond the chat interface
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,41 @@
|
||||
# QA: Sub-Agents — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the subagents demo page
|
||||
- [ ] Verify the chat interface loads with title "Sub-Agents"
|
||||
- [ ] Verify the chat input placeholder "Type a message..." is visible
|
||||
- [ ] Send a basic message (e.g. "Hello! What can you do?")
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Get started" suggestion button is visible
|
||||
|
||||
#### Note: Stub Demo
|
||||
|
||||
> **Status: Stub** — This demo is currently a stub (TODO: implement)
|
||||
|
||||
- [ ] Verify the basic CopilotChat loads and accepts messages
|
||||
- [ ] Verify the agent responds to messages
|
||||
- [ ] No custom UI components are expected beyond the chat interface
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,31 @@
|
||||
# Thread ID Frontend Tool Round Trip
|
||||
|
||||
## Scope
|
||||
|
||||
Regression checklist for ENT-658: a `CopilotChat` wrapped by
|
||||
`CopilotChatConfigurationProvider` must keep its SDK-generated non-explicit
|
||||
thread active across a frontend tool call and follow-up run.
|
||||
|
||||
## Manual QA
|
||||
|
||||
- [ ] Navigate to `/demos/threadid-frontend-tool-roundtrip`.
|
||||
- [ ] Verify the chat input is visible and `Explicit threadId` is unchecked.
|
||||
- [ ] Send `invoke testFrontendToolCalling with label X`.
|
||||
- [ ] Verify the user message remains visible.
|
||||
- [ ] Verify the `testFrontendToolCalling` card remains visible and shows
|
||||
`label: X` and `result: handled X`.
|
||||
- [ ] Verify the assistant reply `Frontend tool finished for X.` appears.
|
||||
- [ ] Verify the chat does not return to the empty state.
|
||||
- [ ] Refresh the page or open a new tab, check `Explicit threadId` before
|
||||
sending any message, send the same prompt again, and verify the same
|
||||
message/tool/reply persistence behavior.
|
||||
- [ ] Optionally toggle `Explicit threadId` after a generated-thread
|
||||
conversation and verify the chat switches to the explicit thread's
|
||||
history. An empty explicit thread on first use is expected.
|
||||
|
||||
## Automated Coverage
|
||||
|
||||
- `tests/e2e/threadid-frontend-tool-roundtrip.spec.ts` covers the demo route,
|
||||
generated-thread default state, and explicit-thread toggle.
|
||||
- `packages/react-core/src/v2/components/chat/__tests__/CopilotChat.absentThreadConnect.test.tsx`
|
||||
covers the SDK-generated thread handoff at the component level.
|
||||
@@ -0,0 +1,86 @@
|
||||
# QA: Tool Rendering (Custom Catch-all) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- Agent slug `tool-rendering-custom-catchall` is registered at `/api/copilotkit`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the `tool-rendering-custom-catchall` demo page
|
||||
- [ ] Verify the chat interface loads in a centered full-height layout (max-width 4xl, `rounded-2xl`)
|
||||
- [ ] Verify the chat input placeholder "Type a message" is visible
|
||||
- [ ] Send a basic message (e.g. "Hi")
|
||||
- [ ] Verify the agent responds with a text message
|
||||
|
||||
### 2. Feature-Specific Checks — Branded Wildcard Renderer
|
||||
|
||||
The frontend calls `useDefaultRenderTool({ render: ... })` with a SINGLE branded wildcard component (`CustomCatchallRenderer`). There are ZERO per-tool named renderers. Every tool call must paint via this one branded card regardless of tool identity.
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Weather in SF" suggestion pill is visible
|
||||
- [ ] Verify "Find flights" suggestion pill is visible
|
||||
- [ ] Verify "Roll a d20" suggestion pill is visible
|
||||
- [ ] Click a suggestion and verify it either populates the input or sends the message
|
||||
|
||||
#### `get_weather` renders via the branded catch-all card
|
||||
|
||||
- [ ] Click the "Weather in SF" suggestion (or type "What's the weather in San Francisco?")
|
||||
- [ ] Verify the branded card renders (`data-testid="custom-catchall-card"`) with:
|
||||
- [ ] An uppercase label "Tool" followed by the tool name `get_weather` (`data-testid="custom-catchall-tool-name"`, monospaced)
|
||||
- [ ] The card root carries `data-tool-name="get_weather"`
|
||||
- [ ] A status badge (`data-testid="custom-catchall-status"`) that transitions through `streaming` (amber), `running` (lavender/indigo), and finally `done` (green `#189370`)
|
||||
- [ ] An "Arguments" section with a monospaced pre block (`data-testid="custom-catchall-args"`) showing pretty-printed JSON of the parameters
|
||||
- [ ] A "Result" section showing "waiting for tool to finish…" while pending and a green-tinted pre block (`data-testid="custom-catchall-result"`) once `status === "complete"`
|
||||
- [ ] The completed Result pre shows the mock weather payload (`city`, `temperature: 68`, `humidity: 55`, `wind_speed: 10`, `conditions: "Sunny"`)
|
||||
- [ ] Verify the card uses the branded styling: white background, rounded `2xl` corners, `#DBDBE5` border, `#FAFAFC` header strip, subtle shadow
|
||||
|
||||
#### `search_flights` renders via the SAME branded catch-all card
|
||||
|
||||
- [ ] Click the "Find flights" suggestion (or type "Find flights from SFO to JFK")
|
||||
- [ ] Verify a second `data-testid="custom-catchall-card"` renders
|
||||
- [ ] Verify `data-tool-name="search_flights"` on the card root and the name label reads `search_flights`
|
||||
- [ ] Verify the visual style is IDENTICAL to the `get_weather` card — same header strip, same status badge treatment, same Arguments/Result sections
|
||||
- [ ] Verify the Result pre shows the mock flights array (United UA231, Delta DL412, JetBlue B6722)
|
||||
|
||||
#### `roll_dice` renders via the SAME branded catch-all card
|
||||
|
||||
- [ ] Click the "Roll a d20" suggestion (or type "Roll a 20-sided die")
|
||||
- [ ] Verify another `custom-catchall-card` renders with `data-tool-name="roll_dice"`
|
||||
- [ ] Verify the Result pre shows `{ "sides": 20, "result": <1-20> }`
|
||||
|
||||
#### `get_stock_price` renders via the SAME branded catch-all card
|
||||
|
||||
- [ ] Type "How is AAPL doing?"
|
||||
- [ ] Verify a `custom-catchall-card` with `data-tool-name="get_stock_price"` renders
|
||||
- [ ] Verify the Result pre shows `ticker: "AAPL"`, a `price_usd`, and a `change_pct`
|
||||
|
||||
#### No built-in default UI appears
|
||||
|
||||
- [ ] Verify NO instance of CopilotKit's built-in `DefaultToolCallRenderer` appears alongside the branded card — the branded card replaces it
|
||||
- [ ] Verify every rendered tool call is a `data-testid="custom-catchall-card"` element
|
||||
|
||||
#### Chained tool calls all paint the same branded card
|
||||
|
||||
- [ ] Ask "What's the weather in Tokyo?" — the system prompt instructs the agent to chain tools
|
||||
- [ ] Verify at least TWO `custom-catchall-card` elements render in succession (e.g. one for `get_weather`, one for `search_flights`)
|
||||
- [ ] Verify each carries a distinct `data-tool-name` but the visual style is identical
|
||||
- [ ] While a tool is in flight, verify the status badge reads `streaming` or `running` and the Result section shows "waiting for tool to finish…"; after completion the badge reads `done` and the green Result pre appears
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
- [ ] Verify the `done` badge text and the completed-result green tint (`#85ECCE`) remain stable across re-renders
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Every tool invocation paints via the single branded `CustomCatchallRenderer` card — zero visual variance beyond `data-tool-name` and the payload
|
||||
- Status badge progresses: `streaming` → `running` → `done`
|
||||
- Branded styling details from `custom-catchall-renderer.tsx` are visible (uppercase "Tool" label, monospaced tool-name, rounded-2xl card, header strip, tone-coded status badge, green result pre on completion)
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,76 @@
|
||||
# QA: Tool Rendering (Default Catch-all) — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
- Agent slug `tool-rendering-default-catchall` is registered at `/api/copilotkit`
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the `tool-rendering-default-catchall` demo page
|
||||
- [ ] Verify the chat interface loads in a centered full-height layout (max-width 4xl, `rounded-2xl`)
|
||||
- [ ] Verify the chat input placeholder "Type a message" is visible
|
||||
- [ ] Send a basic message (e.g. "Hi")
|
||||
- [ ] Verify the agent responds with a text message
|
||||
|
||||
### 2. Feature-Specific Checks — Built-in Default Tool-Call UI
|
||||
|
||||
The frontend calls `useDefaultRenderTool()` with NO config — it registers CopilotKit's package-provided `DefaultToolCallRenderer` as the `*` wildcard. The frontend adds ZERO custom per-tool or custom wildcard renderers. Every tool call must paint via this one built-in card.
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Weather in SF" suggestion pill is visible
|
||||
- [ ] Verify "Find flights" suggestion pill is visible
|
||||
- [ ] Verify "Roll a d20" suggestion pill is visible
|
||||
- [ ] Click a suggestion and verify it either populates the input or sends the message
|
||||
|
||||
#### `get_weather` renders via the built-in default card
|
||||
|
||||
- [ ] Click the "Weather in SF" suggestion (or type "What's the weather in San Francisco?")
|
||||
- [ ] Verify a default tool-call card appears with the tool name `get_weather` visible
|
||||
- [ ] Verify a status pill transitions through `Running` and lands on `Done`
|
||||
- [ ] Expand the card's "Arguments" section and verify it shows `{ "location": "San Francisco" }` (or similar)
|
||||
- [ ] Expand the card's "Result" section and verify it shows the mock payload with `city`, `temperature: 68`, `humidity: 55`, `wind_speed: 10`, `conditions: "Sunny"`
|
||||
- [ ] Verify NO custom-branded card appears (no `data-testid="custom-catchall-card"`, no `data-testid="weather-card"`)
|
||||
|
||||
#### `search_flights` renders via the SAME built-in default card
|
||||
|
||||
- [ ] Click the "Find flights" suggestion (or type "Find flights from SFO to JFK")
|
||||
- [ ] Verify a tool-call card appears with tool name `search_flights`
|
||||
- [ ] Verify the card has the identical visual style/structure as the `get_weather` card — same header layout, same status pill, same Arguments/Result sections
|
||||
- [ ] Verify the Result section contains three mock flights (United UA231, Delta DL412, JetBlue B6722)
|
||||
|
||||
#### `roll_dice` renders via the SAME built-in default card
|
||||
|
||||
- [ ] Click the "Roll a d20" suggestion (or type "Roll a 20-sided die")
|
||||
- [ ] Verify a tool-call card for `roll_dice` appears with the same default visual style
|
||||
- [ ] Verify the Result section shows `{ "sides": 20, "result": <1-20> }`
|
||||
|
||||
#### `get_stock_price` renders via the SAME built-in default card
|
||||
|
||||
- [ ] Type "How is AAPL doing?"
|
||||
- [ ] Verify a `get_stock_price` tool-call card appears with the default built-in style
|
||||
- [ ] Verify the Result shows `ticker: "AAPL"`, a `price_usd`, and a `change_pct`
|
||||
|
||||
#### Chained tool calls
|
||||
|
||||
- [ ] Ask "What's the weather in Tokyo?" — the system prompt instructs the agent to chain tools
|
||||
- [ ] Verify at least TWO default tool-call cards render in succession (e.g. `get_weather` then `search_flights`)
|
||||
- [ ] Verify every card uses the identical default built-in UI — visually indistinguishable apart from the tool name and payload
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
- [ ] Verify no unhandled-promise warnings when a tool call streams
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Every tool invocation paints via the built-in `DefaultToolCallRenderer` card (tool name + live status pill + Arguments + Result)
|
||||
- All four distinct tools (`get_weather`, `search_flights`, `get_stock_price`, `roll_dice`) render via the SAME default card — zero visual variance beyond payload
|
||||
- No custom-branded renderer appears anywhere
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,21 @@
|
||||
# QA: Tool Rendering (Reasoning Chain) — LangGraph (Python)
|
||||
|
||||
> Stub — authored for column completeness. This is a testing-kind demo
|
||||
> (see `kind: "testing"` in feature-registry.json) and does not warrant a
|
||||
> full manual checklist.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy
|
||||
|
||||
## Test Steps
|
||||
|
||||
- [ ] Navigate to /demos/tool-rendering-reasoning-chain
|
||||
- [ ] Send a multi-tool prompt (e.g. "What's the weather in Tokyo?") and verify reasoning blocks interleave with sequential tool cards (`WeatherCard`, `FlightListCard`, or the custom catchall)
|
||||
- [ ] Verify reasoning tokens stream into the custom `ReasoningBlock` slot alongside the tool cards in the same message view
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Page loads without errors
|
||||
- Reasoning tokens and tool-call cards render side-by-side in a single sequential chain, each tool matched to its typed renderer
|
||||
@@ -0,0 +1,61 @@
|
||||
# QA: Tool Rendering — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible
|
||||
- Agent backend is healthy (check /api/health)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to the tool-rendering demo page
|
||||
- [ ] Verify the chat interface loads in a centered full-height layout
|
||||
- [ ] Verify the chat input placeholder "Type a message" is visible
|
||||
- [ ] Send a basic message
|
||||
- [ ] Verify the agent responds
|
||||
|
||||
### 2. Feature-Specific Checks
|
||||
|
||||
#### Suggestions
|
||||
|
||||
- [ ] Verify "Weather in San Francisco" suggestion button is visible
|
||||
- [ ] Verify "Weather in New York" suggestion button is visible
|
||||
- [ ] Verify "Weather in Tokyo" suggestion button is visible
|
||||
- [ ] Click a weather suggestion and verify it populates the input or sends the message
|
||||
|
||||
#### Weather Card Rendering (useRenderTool)
|
||||
|
||||
- [ ] Type "What's the weather in San Francisco?"
|
||||
- [ ] Verify loading state shows "Retrieving weather..." with a spinner
|
||||
- [ ] Verify the WeatherCard renders (`data-testid="weather-card"`) with:
|
||||
- [ ] City name displayed (`data-testid="weather-city"`)
|
||||
- [ ] Temperature in both Celsius and Fahrenheit
|
||||
- [ ] Humidity percentage (`data-testid="weather-humidity"`)
|
||||
- [ ] Wind speed in mph (`data-testid="weather-wind"`)
|
||||
- [ ] Feels-like temperature (`data-testid="weather-feels-like"`)
|
||||
- [ ] Conditions text with appropriate weather icon (sun/rain/cloud)
|
||||
- [ ] Verify the card background color matches the weather condition theme:
|
||||
- Clear/Sunny: #667eea (blue-purple)
|
||||
- Rain/Storm: #4A5568 (dark gray)
|
||||
- Cloudy: #718096 (medium gray)
|
||||
- Snow: #63B3ED (light blue)
|
||||
|
||||
#### Multiple Weather Queries
|
||||
|
||||
- [ ] Ask about weather in a second city
|
||||
- [ ] Verify a second WeatherCard renders without breaking the first
|
||||
- [ ] Verify each card shows the correct city name
|
||||
|
||||
### 3. Error Handling
|
||||
|
||||
- [ ] Send an empty message (should be handled gracefully)
|
||||
- [ ] Verify no console errors during normal usage
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Chat loads within 3 seconds
|
||||
- Agent responds within 10 seconds
|
||||
- Weather cards render with all data fields populated
|
||||
- Weather icon and theme color match the conditions
|
||||
- No UI errors or broken layouts
|
||||
@@ -0,0 +1,51 @@
|
||||
# QA: Voice Input — LangGraph (Python)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Demo is deployed and accessible at `/demos/voice`
|
||||
- Railway service `showcase-langgraph-python` is healthy (`/api/health` returns 200)
|
||||
- `OPENAI_API_KEY` is set on the Railway service (shared with other demos)
|
||||
- A modern browser that supports `MediaRecorder` (Chromium, Firefox, Safari 14+)
|
||||
- Microphone hardware available (required only for the mic path in section 3)
|
||||
- A bundled `public/demo-audio/sample.wav` is present (used for screenshot/preview generation; the in-app sample button no longer fetches it)
|
||||
|
||||
## Test Steps
|
||||
|
||||
### 1. Basic Functionality
|
||||
|
||||
- [ ] Navigate to `/demos/voice`
|
||||
- [ ] Verify the page header "Voice input" is visible
|
||||
- [ ] Verify the sample-audio row (`data-testid="voice-sample-audio"`) is visible
|
||||
- [ ] Verify the caption reads `Sample: "What is the weather in Tokyo?"`
|
||||
- [ ] Verify the "Play sample" button (`data-testid="voice-sample-audio-button"`) is enabled
|
||||
- [ ] Verify `<CopilotChat />` renders a message composer (`data-testid="copilot-chat-input"`)
|
||||
- [ ] Verify the composer shows a microphone button (`data-testid="copilot-start-transcribe-button"`) — this is the authoritative signal that `transcriptionService` is mounted on `/api/copilotkit-voice`
|
||||
|
||||
### 2. Sample-audio path (no mic permission required)
|
||||
|
||||
- [ ] Click the "Play sample" button
|
||||
- [ ] Immediately, the chat textarea (`data-testid="copilot-chat-textarea"`) contains the canned phrase "What is the weather in Tokyo?" (no async round-trip; no "Transcribing…" state)
|
||||
- [ ] The button stays enabled
|
||||
- [ ] Click send (`data-testid="copilot-send-button"`)
|
||||
- [ ] Within 10 seconds, the agent responds with a weather-related tool render (WeatherCard, custom-catchall card, or default tool card — depending on which tool-rendering mode is active on this page)
|
||||
|
||||
### 3. Mic path (manual)
|
||||
|
||||
- [ ] Click the microphone button (`data-testid="copilot-start-transcribe-button"`) in the composer
|
||||
- [ ] Grant microphone permission at the browser prompt
|
||||
- [ ] Speak "Hello" clearly, then click the mic button again (now `data-testid="copilot-finish-transcribe-button"`) to stop recording
|
||||
- [ ] Within 5 seconds, the textarea contains text matching "hello" (case-insensitive)
|
||||
- [ ] Click send
|
||||
- [ ] Agent responds within 10 seconds
|
||||
|
||||
### 4. Error Handling
|
||||
|
||||
- [ ] Deny microphone permission, click the mic button
|
||||
- [ ] Verify the UI handles permission denial gracefully (no crash, mic button remains visible)
|
||||
|
||||
## Expected Results
|
||||
|
||||
- Sample button click populates the textarea synchronously (no perceptible delay)
|
||||
- Weather-related tool response renders within 10 seconds of send
|
||||
- No console errors during the successful paths
|
||||
- Whisper transcription via the mic path returns text resembling what was spoken (deployment must have `OPENAI_API_KEY` configured)
|
||||
Reference in New Issue
Block a user