chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"name": "data",
|
||||
"version": "1.1.0",
|
||||
"description": "Write SQL, explore datasets, and generate insights faster. Build visualizations and dashboards, and turn raw data into clear stories for stakeholders.",
|
||||
"author": {
|
||||
"name": "Anthropic"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"snowflake": {
|
||||
"type": "http",
|
||||
"url": ""
|
||||
},
|
||||
"databricks": {
|
||||
"type": "http",
|
||||
"url": ""
|
||||
},
|
||||
"bigquery": {
|
||||
"type": "http",
|
||||
"url": "https://bigquery.googleapis.com/mcp"
|
||||
},
|
||||
"hex": {
|
||||
"type": "http",
|
||||
"url": "https://app.hex.tech/mcp"
|
||||
},
|
||||
"amplitude": {
|
||||
"type": "http",
|
||||
"url": "https://mcp.amplitude.com/mcp"
|
||||
},
|
||||
"amplitude-eu": {
|
||||
"type": "http",
|
||||
"url": "https://mcp.eu.amplitude.com/mcp"
|
||||
},
|
||||
"atlassian": {
|
||||
"type": "http",
|
||||
"url": "https://mcp.atlassian.com/v1/mcp"
|
||||
},
|
||||
"definite": {
|
||||
"type": "http",
|
||||
"url": "https://api.definite.app/v3/mcp/http"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
# Connectors
|
||||
|
||||
## How tool references work
|
||||
|
||||
Plugin files use `~~category` as a placeholder for whatever tool the user connects in that category. For example, `~~data warehouse` might mean Snowflake, BigQuery, or any other warehouse with an MCP server.
|
||||
|
||||
Plugins are **tool-agnostic** — they describe workflows in terms of categories (data warehouse, notebook, product analytics, etc.) rather than specific products. The `.mcp.json` pre-configures specific MCP servers, but any MCP server in that category works.
|
||||
|
||||
## Connectors for this plugin
|
||||
|
||||
| Category | Placeholder | Included servers | Other options |
|
||||
|----------|-------------|-----------------|---------------|
|
||||
| Data warehouse | `~~data warehouse` | Snowflake\*, Databricks\*, BigQuery, Definite | Redshift, PostgreSQL, MySQL |
|
||||
| Notebook | `~~notebook` | Hex | Jupyter, Deepnote, Observable |
|
||||
| Product analytics | `~~product analytics` | Amplitude | Mixpanel, Heap |
|
||||
| Project tracker | `~~project tracker` | Atlassian (Jira/Confluence) | Linear, Asana |
|
||||
|
||||
\* Placeholder — MCP URL not yet configured
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
# Data Analyst Plugin
|
||||
|
||||
A data analyst plugin primarily designed for [Cowork](https://claude.com/product/cowork), Anthropic's agentic desktop application — though it also works in Claude Code. SQL queries, data exploration, visualization, dashboards, and insight generation. Works with any data warehouse, any SQL dialect, and any analytics stack.
|
||||
|
||||
## Installation
|
||||
|
||||
```
|
||||
claude plugins add knowledge-work-plugins/data
|
||||
```
|
||||
|
||||
## What It Does
|
||||
|
||||
This plugin transforms Claude into a data analyst collaborator. It helps you explore datasets, write optimized SQL, build visualizations, create interactive dashboards, and validate analyses before sharing with stakeholders.
|
||||
|
||||
### With a Data Warehouse Connection
|
||||
|
||||
Connect your data warehouse MCP server (e.g., Snowflake, Databricks, BigQuery, or any SQL-compatible database) for the best experience. Claude will:
|
||||
|
||||
- Query your data warehouse directly
|
||||
- Explore schemas and table metadata
|
||||
- Run analyses end-to-end without copy-pasting
|
||||
- Iterate on queries based on results
|
||||
|
||||
### Without a Data Warehouse Connection
|
||||
|
||||
Without a data warehouse connection, paste SQL results or upload CSV/Excel files for analysis and visualization. Claude can also write SQL queries for you to run manually, and then analyze the results you provide.
|
||||
|
||||
## Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/analyze` | Answer data questions -- from quick lookups to full analyses |
|
||||
| `/explore-data` | Profile and explore a dataset to understand its shape, quality, and patterns |
|
||||
| `/write-query` | Write optimized SQL for your dialect with best practices |
|
||||
| `/create-viz` | Create publication-quality visualizations with Python |
|
||||
| `/build-dashboard` | Build interactive HTML dashboards with filters and charts |
|
||||
| `/validate` | QA an analysis before sharing -- methodology, accuracy, and bias checks |
|
||||
|
||||
## Skills
|
||||
|
||||
| Skill | Description |
|
||||
|-------|-------------|
|
||||
| `sql-queries` | SQL best practices across dialects, common patterns, and performance optimization |
|
||||
| `data-exploration` | Data profiling, quality assessment, and pattern discovery |
|
||||
| `data-visualization` | Chart selection, Python viz code patterns, and design principles |
|
||||
| `statistical-analysis` | Descriptive stats, trend analysis, outlier detection, and hypothesis testing |
|
||||
| `data-validation` | Pre-delivery QA, sanity checks, and documentation standards |
|
||||
| `interactive-dashboard-builder` | HTML/JS dashboard construction with Chart.js, filters, and styling |
|
||||
|
||||
## Example Workflows
|
||||
|
||||
### Ad-Hoc Analysis
|
||||
|
||||
```
|
||||
You: /analyze What was our monthly revenue trend for the past 12 months, broken down by product line?
|
||||
|
||||
Claude: [Writes SQL query] → [Executes against data warehouse] → [Generates trend chart]
|
||||
→ [Identifies key patterns: "Product line A grew 23% YoY while B was flat"]
|
||||
→ [Validates results with sanity checks]
|
||||
```
|
||||
|
||||
### Data Exploration
|
||||
|
||||
```
|
||||
You: /explore-data users table
|
||||
|
||||
Claude: [Profiles table: 2.3M rows, 47 columns]
|
||||
→ [Reports: created_at has 0.2% nulls, email has 99.8% cardinality]
|
||||
→ [Flags: status column has unexpected value "UNKNOWN" in 340 rows]
|
||||
→ [Suggests: "High-value dimensions to explore: plan_type, signup_source, country"]
|
||||
```
|
||||
|
||||
### Query Writing
|
||||
|
||||
```
|
||||
You: /write-query I need a cohort retention analysis -- users grouped by signup month,
|
||||
showing what % are still active 1, 3, 6, and 12 months later. We use Snowflake.
|
||||
|
||||
Claude: [Writes optimized Snowflake SQL with CTEs]
|
||||
→ [Adds comments explaining each step]
|
||||
→ [Includes performance notes about partition pruning]
|
||||
```
|
||||
|
||||
### Dashboard Building
|
||||
|
||||
```
|
||||
You: /build-dashboard Create a sales dashboard with monthly revenue, top products,
|
||||
and regional breakdown. Here's the data: [pastes CSV]
|
||||
|
||||
Claude: [Generates self-contained HTML file]
|
||||
→ [Includes interactive Chart.js visualizations]
|
||||
→ [Adds dropdown filters for region and time period]
|
||||
→ [Opens in browser for review]
|
||||
```
|
||||
|
||||
### Pre-Share Validation
|
||||
|
||||
```
|
||||
You: /validate [shares analysis document]
|
||||
|
||||
Claude: [Reviews methodology] → [Checks for survivorship bias in churn analysis]
|
||||
→ [Verifies aggregation logic] → [Flags: "Denominator excludes trial users
|
||||
which could overstate conversion rate by ~5pp"]
|
||||
→ [Confidence: "Ready to share with noted caveat"]
|
||||
```
|
||||
|
||||
## Connecting Your Data Stack
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](CONNECTORS.md).
|
||||
|
||||
This plugin works best when connected to your data infrastructure. Add MCP servers for:
|
||||
|
||||
- **Data Warehouse**: Snowflake, Databricks, BigQuery, Definite, or any SQL-compatible database
|
||||
- **Analytics/BI**: Amplitude, Looker, Tableau, or similar
|
||||
- **Notebooks**: Jupyter, Hex, or similar
|
||||
- **Spreadsheets**: Google Sheets, Excel
|
||||
- **Data Orchestration**: Airflow, dbt, Dagster, Prefect
|
||||
- **Data Ingestion**: Fivetran, Airbyte, Stitch
|
||||
|
||||
Configure MCP servers in your `.mcp.json` or Claude Code settings to enable direct data access.
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
name: analyze
|
||||
description: Answer data questions -- from quick lookups to full analyses. Use when looking up a single metric, investigating what's driving a trend or drop, comparing segments over time, or preparing a formal data report for stakeholders.
|
||||
argument-hint: "<question>"
|
||||
---
|
||||
|
||||
# /analyze - Answer Data Questions
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Answer a data question, from a quick lookup to a full analysis to a formal report.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/analyze <natural language question>
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the Question
|
||||
|
||||
Parse the user's question and determine:
|
||||
|
||||
- **Complexity level**:
|
||||
- **Quick answer**: Single metric, simple filter, factual lookup (e.g., "How many users signed up last week?")
|
||||
- **Full analysis**: Multi-dimensional exploration, trend analysis, comparison (e.g., "What's driving the drop in conversion rate?")
|
||||
- **Formal report**: Comprehensive investigation with methodology, caveats, and recommendations (e.g., "Prepare a quarterly business review of our subscription metrics")
|
||||
- **Data requirements**: Which tables, metrics, dimensions, and time ranges are needed
|
||||
- **Output format**: Number, table, chart, narrative, or combination
|
||||
|
||||
### 2. Gather Data
|
||||
|
||||
**If a data warehouse MCP server is connected:**
|
||||
|
||||
1. Explore the schema to find relevant tables and columns
|
||||
2. Write SQL query(ies) to extract the needed data
|
||||
3. Execute the query and retrieve results
|
||||
4. If the query fails, debug and retry (check column names, table references, syntax for the specific dialect)
|
||||
5. If results look unexpected, run sanity checks before proceeding
|
||||
|
||||
**If no data warehouse is connected:**
|
||||
|
||||
1. Ask the user to provide data in one of these ways:
|
||||
- Paste query results directly
|
||||
- Upload a CSV or Excel file
|
||||
- Describe the schema so you can write queries for them to run
|
||||
2. If writing queries for manual execution, use the `sql-queries` skill for dialect-specific best practices
|
||||
3. Once data is provided, proceed with analysis
|
||||
|
||||
### 3. Analyze
|
||||
|
||||
- Calculate relevant metrics, aggregations, and comparisons
|
||||
- Identify patterns, trends, outliers, and anomalies
|
||||
- Compare across dimensions (time periods, segments, categories)
|
||||
- For complex analyses, break the problem into sub-questions and address each
|
||||
|
||||
### 4. Validate Before Presenting
|
||||
|
||||
Before sharing results, run through validation checks:
|
||||
|
||||
- **Row count sanity**: Does the number of records make sense?
|
||||
- **Null check**: Are there unexpected nulls that could skew results?
|
||||
- **Magnitude check**: Are the numbers in a reasonable range?
|
||||
- **Trend continuity**: Do time series have unexpected gaps?
|
||||
- **Aggregation logic**: Do subtotals sum to totals correctly?
|
||||
|
||||
If any check raises concerns, investigate and note caveats.
|
||||
|
||||
### 5. Present Findings
|
||||
|
||||
**For quick answers:**
|
||||
- State the answer directly with relevant context
|
||||
- Include the query used (collapsed or in a code block) for reproducibility
|
||||
|
||||
**For full analyses:**
|
||||
- Lead with the key finding or insight
|
||||
- Support with data tables and/or visualizations
|
||||
- Note methodology and any caveats
|
||||
- Suggest follow-up questions
|
||||
|
||||
**For formal reports:**
|
||||
- Executive summary with key takeaways
|
||||
- Methodology section explaining approach and data sources
|
||||
- Detailed findings with supporting evidence
|
||||
- Caveats, limitations, and data quality notes
|
||||
- Recommendations and suggested next steps
|
||||
|
||||
### 6. Visualize Where Helpful
|
||||
|
||||
When a chart would communicate results more effectively than a table:
|
||||
|
||||
- Use the `data-visualization` skill to select the right chart type
|
||||
- Generate a Python visualization or build it into an HTML dashboard
|
||||
- Follow visualization best practices for clarity and accuracy
|
||||
|
||||
## Examples
|
||||
|
||||
**Quick answer:**
|
||||
```
|
||||
/analyze How many new users signed up in December?
|
||||
```
|
||||
|
||||
**Full analysis:**
|
||||
```
|
||||
/analyze What's causing the increase in support ticket volume over the past 3 months? Break down by category and priority.
|
||||
```
|
||||
|
||||
**Formal report:**
|
||||
```
|
||||
/analyze Prepare a data quality assessment of our customer table -- completeness, consistency, and any issues we should address.
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Be specific about time ranges, segments, or metrics when possible
|
||||
- If you know the table names, mention them to speed up the process
|
||||
- For complex questions, Claude may break them into multiple queries
|
||||
- Results are always validated before presentation -- if something looks off, Claude will flag it
|
||||
@@ -0,0 +1,924 @@
|
||||
---
|
||||
name: build-dashboard
|
||||
description: Build an interactive HTML dashboard with charts, filters, and tables. Use when creating an executive overview with KPI cards, turning query results into a shareable self-contained report, building a team monitoring snapshot, or needing multiple charts with filters in one browser-openable file.
|
||||
argument-hint: "<description> [data source]"
|
||||
---
|
||||
|
||||
# /build-dashboard - Build Interactive Dashboards
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Build a self-contained interactive HTML dashboard with charts, filters, tables, and professional styling. Opens directly in a browser -- no server or dependencies required.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/build-dashboard <description of dashboard> [data source]
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the Dashboard Requirements
|
||||
|
||||
Determine:
|
||||
|
||||
- **Purpose**: Executive overview, operational monitoring, deep-dive analysis, team reporting
|
||||
- **Audience**: Who will use this dashboard?
|
||||
- **Key metrics**: What numbers matter most?
|
||||
- **Dimensions**: What should users be able to filter or slice by?
|
||||
- **Data source**: Live query, pasted data, CSV file, or sample data
|
||||
|
||||
### 2. Gather the Data
|
||||
|
||||
**If data warehouse is connected:**
|
||||
1. Query the necessary data
|
||||
2. Embed the results as JSON within the HTML file
|
||||
|
||||
**If data is pasted or uploaded:**
|
||||
1. Parse and clean the data
|
||||
2. Embed as JSON in the dashboard
|
||||
|
||||
**If working from a description without data:**
|
||||
1. Create a realistic sample dataset matching the described schema
|
||||
2. Note in the dashboard that it uses sample data
|
||||
3. Provide instructions for swapping in real data
|
||||
|
||||
### 3. Design the Dashboard Layout
|
||||
|
||||
Follow a standard dashboard layout pattern:
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ Dashboard Title [Filters ▼] │
|
||||
├────────────┬────────────┬────────────┬───────────┤
|
||||
│ KPI Card │ KPI Card │ KPI Card │ KPI Card │
|
||||
├────────────┴────────────┼────────────┴───────────┤
|
||||
│ │ │
|
||||
│ Primary Chart │ Secondary Chart │
|
||||
│ (largest area) │ │
|
||||
│ │ │
|
||||
├─────────────────────────┴────────────────────────┤
|
||||
│ │
|
||||
│ Detail Table (sortable, scrollable) │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Adapt the layout to the content:**
|
||||
- 2-4 KPI cards at the top for headline numbers
|
||||
- 1-3 charts in the middle section for trends and breakdowns
|
||||
- Optional detail table at the bottom for drill-down data
|
||||
- Filters in the header or sidebar depending on complexity
|
||||
|
||||
### 4. Build the HTML Dashboard
|
||||
|
||||
Generate a single self-contained HTML file using the base template below. The file includes:
|
||||
|
||||
**Structure (HTML):**
|
||||
- Semantic HTML5 layout
|
||||
- Responsive grid using CSS Grid or Flexbox
|
||||
- Filter controls (dropdowns, date pickers, toggles)
|
||||
- KPI cards with values and labels
|
||||
- Chart containers
|
||||
- Data table with sortable headers
|
||||
|
||||
**Styling (CSS):**
|
||||
- Professional color scheme (clean whites, grays, with accent colors for data)
|
||||
- Card-based layout with subtle shadows
|
||||
- Consistent typography (system fonts for fast loading)
|
||||
- Responsive design that works on different screen sizes
|
||||
- Print-friendly styles
|
||||
|
||||
**Interactivity (JavaScript):**
|
||||
- Chart.js for interactive charts (included via CDN)
|
||||
- Filter dropdowns that update all charts and tables simultaneously
|
||||
- Sortable table columns
|
||||
- Hover tooltips on charts
|
||||
- Number formatting (commas, currency, percentages)
|
||||
|
||||
**Data (embedded JSON):**
|
||||
- All data embedded directly in the HTML as JavaScript variables
|
||||
- No external data fetches required
|
||||
- Dashboard works completely offline
|
||||
|
||||
### 5. Implement Chart Types
|
||||
|
||||
Use Chart.js for all charts. Common dashboard chart patterns:
|
||||
|
||||
- **Line chart**: Time series trends
|
||||
- **Bar chart**: Category comparisons
|
||||
- **Doughnut chart**: Composition (when <6 categories)
|
||||
- **Stacked bar**: Composition over time
|
||||
- **Mixed (bar + line)**: Volume with rate overlay
|
||||
|
||||
Use the Chart.js integration patterns below for each chart type.
|
||||
|
||||
### 6. Add Interactivity
|
||||
|
||||
Use the filter and interactivity implementation patterns below for dropdown filters, date range filters, combined filter logic, sortable tables, and chart updates.
|
||||
|
||||
### 7. Save and Open
|
||||
|
||||
1. Save the dashboard as an HTML file with a descriptive name (e.g., `sales_dashboard.html`)
|
||||
2. Open it in the user's default browser
|
||||
3. Confirm it renders correctly
|
||||
4. Provide instructions for updating data or customizing
|
||||
|
||||
---
|
||||
|
||||
## Base Template
|
||||
|
||||
Every dashboard follows this structure:
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Dashboard Title</title>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.5.1" integrity="sha384-jb8JQMbMoBUzgWatfe6COACi2ljcDdZQ2OxczGA3bGNeWe+6DChMTBJemed7ZnvJ" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/chartjs-adapter-date-fns@3.0.0" integrity="sha384-cVMg8E3QFwTvGCDuK+ET4PD341jF3W8nO1auiXfuZNQkzbUUiBGLsIQUE+b1mxws" crossorigin="anonymous"></script>
|
||||
<style>
|
||||
/* Dashboard styles go here */
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="dashboard-container">
|
||||
<header class="dashboard-header">
|
||||
<h1>Dashboard Title</h1>
|
||||
<div class="filters">
|
||||
<!-- Filter controls -->
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<section class="kpi-row">
|
||||
<!-- KPI cards -->
|
||||
</section>
|
||||
|
||||
<section class="chart-row">
|
||||
<!-- Chart containers -->
|
||||
</section>
|
||||
|
||||
<section class="table-section">
|
||||
<!-- Data table -->
|
||||
</section>
|
||||
|
||||
<footer class="dashboard-footer">
|
||||
<span>Data as of: <span id="data-date"></span></span>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// Embedded data
|
||||
const DATA = [];
|
||||
|
||||
// Dashboard logic
|
||||
class Dashboard {
|
||||
constructor(data) {
|
||||
this.rawData = data;
|
||||
this.filteredData = data;
|
||||
this.charts = {};
|
||||
this.init();
|
||||
}
|
||||
|
||||
init() {
|
||||
this.setupFilters();
|
||||
this.renderKPIs();
|
||||
this.renderCharts();
|
||||
this.renderTable();
|
||||
}
|
||||
|
||||
applyFilters() {
|
||||
// Filter logic
|
||||
this.filteredData = this.rawData.filter(row => {
|
||||
// Apply each active filter
|
||||
return true; // placeholder
|
||||
});
|
||||
this.renderKPIs();
|
||||
this.updateCharts();
|
||||
this.renderTable();
|
||||
}
|
||||
|
||||
// ... methods for each section
|
||||
}
|
||||
|
||||
const dashboard = new Dashboard(DATA);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
## KPI Card Pattern
|
||||
|
||||
```html
|
||||
<div class="kpi-card">
|
||||
<div class="kpi-label">Total Revenue</div>
|
||||
<div class="kpi-value" id="kpi-revenue">$0</div>
|
||||
<div class="kpi-change positive" id="kpi-revenue-change">+0%</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
```javascript
|
||||
function renderKPI(elementId, value, previousValue, format = 'number') {
|
||||
const el = document.getElementById(elementId);
|
||||
const changeEl = document.getElementById(elementId + '-change');
|
||||
|
||||
// Format the value
|
||||
el.textContent = formatValue(value, format);
|
||||
|
||||
// Calculate and display change
|
||||
if (previousValue && previousValue !== 0) {
|
||||
const pctChange = ((value - previousValue) / previousValue) * 100;
|
||||
const sign = pctChange >= 0 ? '+' : '';
|
||||
changeEl.textContent = `${sign}${pctChange.toFixed(1)}% vs prior period`;
|
||||
changeEl.className = `kpi-change ${pctChange >= 0 ? 'positive' : 'negative'}`;
|
||||
}
|
||||
}
|
||||
|
||||
function formatValue(value, format) {
|
||||
switch (format) {
|
||||
case 'currency':
|
||||
if (value >= 1e6) return `$${(value / 1e6).toFixed(1)}M`;
|
||||
if (value >= 1e3) return `$${(value / 1e3).toFixed(1)}K`;
|
||||
return `$${value.toFixed(0)}`;
|
||||
case 'percent':
|
||||
return `${value.toFixed(1)}%`;
|
||||
case 'number':
|
||||
if (value >= 1e6) return `${(value / 1e6).toFixed(1)}M`;
|
||||
if (value >= 1e3) return `${(value / 1e3).toFixed(1)}K`;
|
||||
return value.toLocaleString();
|
||||
default:
|
||||
return value.toString();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Chart.js Integration
|
||||
|
||||
### Chart Container Pattern
|
||||
|
||||
```html
|
||||
<div class="chart-container">
|
||||
<h3 class="chart-title">Monthly Revenue Trend</h3>
|
||||
<canvas id="revenue-chart"></canvas>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Line Chart
|
||||
|
||||
```javascript
|
||||
function createLineChart(canvasId, labels, datasets) {
|
||||
const ctx = document.getElementById(canvasId).getContext('2d');
|
||||
return new Chart(ctx, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: labels,
|
||||
datasets: datasets.map((ds, i) => ({
|
||||
label: ds.label,
|
||||
data: ds.data,
|
||||
borderColor: COLORS[i % COLORS.length],
|
||||
backgroundColor: COLORS[i % COLORS.length] + '20',
|
||||
borderWidth: 2,
|
||||
fill: ds.fill || false,
|
||||
tension: 0.3,
|
||||
pointRadius: 3,
|
||||
pointHoverRadius: 6,
|
||||
}))
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
interaction: {
|
||||
mode: 'index',
|
||||
intersect: false,
|
||||
},
|
||||
plugins: {
|
||||
legend: {
|
||||
position: 'top',
|
||||
labels: { usePointStyle: true, padding: 20 }
|
||||
},
|
||||
tooltip: {
|
||||
callbacks: {
|
||||
label: function(context) {
|
||||
return `${context.dataset.label}: ${formatValue(context.parsed.y, 'currency')}`;
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
scales: {
|
||||
x: {
|
||||
grid: { display: false }
|
||||
},
|
||||
y: {
|
||||
beginAtZero: true,
|
||||
ticks: {
|
||||
callback: function(value) {
|
||||
return formatValue(value, 'currency');
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Bar Chart
|
||||
|
||||
```javascript
|
||||
function createBarChart(canvasId, labels, data, options = {}) {
|
||||
const ctx = document.getElementById(canvasId).getContext('2d');
|
||||
const isHorizontal = options.horizontal || labels.length > 8;
|
||||
|
||||
return new Chart(ctx, {
|
||||
type: 'bar',
|
||||
data: {
|
||||
labels: labels,
|
||||
datasets: [{
|
||||
label: options.label || 'Value',
|
||||
data: data,
|
||||
backgroundColor: options.colors || COLORS.map(c => c + 'CC'),
|
||||
borderColor: options.colors || COLORS,
|
||||
borderWidth: 1,
|
||||
borderRadius: 4,
|
||||
}]
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
indexAxis: isHorizontal ? 'y' : 'x',
|
||||
plugins: {
|
||||
legend: { display: false },
|
||||
tooltip: {
|
||||
callbacks: {
|
||||
label: function(context) {
|
||||
return formatValue(context.parsed[isHorizontal ? 'x' : 'y'], options.format || 'number');
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
scales: {
|
||||
x: {
|
||||
beginAtZero: true,
|
||||
grid: { display: isHorizontal },
|
||||
ticks: isHorizontal ? {
|
||||
callback: function(value) {
|
||||
return formatValue(value, options.format || 'number');
|
||||
}
|
||||
} : {}
|
||||
},
|
||||
y: {
|
||||
beginAtZero: !isHorizontal,
|
||||
grid: { display: !isHorizontal },
|
||||
ticks: !isHorizontal ? {
|
||||
callback: function(value) {
|
||||
return formatValue(value, options.format || 'number');
|
||||
}
|
||||
} : {}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Doughnut Chart
|
||||
|
||||
```javascript
|
||||
function createDoughnutChart(canvasId, labels, data) {
|
||||
const ctx = document.getElementById(canvasId).getContext('2d');
|
||||
return new Chart(ctx, {
|
||||
type: 'doughnut',
|
||||
data: {
|
||||
labels: labels,
|
||||
datasets: [{
|
||||
data: data,
|
||||
backgroundColor: COLORS.map(c => c + 'CC'),
|
||||
borderColor: '#ffffff',
|
||||
borderWidth: 2,
|
||||
}]
|
||||
},
|
||||
options: {
|
||||
responsive: true,
|
||||
maintainAspectRatio: false,
|
||||
cutout: '60%',
|
||||
plugins: {
|
||||
legend: {
|
||||
position: 'right',
|
||||
labels: { usePointStyle: true, padding: 15 }
|
||||
},
|
||||
tooltip: {
|
||||
callbacks: {
|
||||
label: function(context) {
|
||||
const total = context.dataset.data.reduce((a, b) => a + b, 0);
|
||||
const pct = ((context.parsed / total) * 100).toFixed(1);
|
||||
return `${context.label}: ${formatValue(context.parsed, 'number')} (${pct}%)`;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Updating Charts on Filter Change
|
||||
|
||||
```javascript
|
||||
function updateChart(chart, newLabels, newData) {
|
||||
chart.data.labels = newLabels;
|
||||
|
||||
if (Array.isArray(newData[0])) {
|
||||
// Multiple datasets
|
||||
newData.forEach((data, i) => {
|
||||
chart.data.datasets[i].data = data;
|
||||
});
|
||||
} else {
|
||||
chart.data.datasets[0].data = newData;
|
||||
}
|
||||
|
||||
chart.update('none'); // 'none' disables animation for instant update
|
||||
}
|
||||
```
|
||||
|
||||
## Filter and Interactivity Implementation
|
||||
|
||||
### Dropdown Filter
|
||||
|
||||
```html
|
||||
<div class="filter-group">
|
||||
<label for="filter-region">Region</label>
|
||||
<select id="filter-region" onchange="dashboard.applyFilters()">
|
||||
<option value="all">All Regions</option>
|
||||
</select>
|
||||
</div>
|
||||
```
|
||||
|
||||
```javascript
|
||||
function populateFilter(selectId, data, field) {
|
||||
const select = document.getElementById(selectId);
|
||||
const values = [...new Set(data.map(d => d[field]))].sort();
|
||||
|
||||
// Keep the "All" option, add unique values
|
||||
values.forEach(val => {
|
||||
const option = document.createElement('option');
|
||||
option.value = val;
|
||||
option.textContent = val;
|
||||
select.appendChild(option);
|
||||
});
|
||||
}
|
||||
|
||||
function getFilterValue(selectId) {
|
||||
const val = document.getElementById(selectId).value;
|
||||
return val === 'all' ? null : val;
|
||||
}
|
||||
```
|
||||
|
||||
### Date Range Filter
|
||||
|
||||
```html
|
||||
<div class="filter-group">
|
||||
<label>Date Range</label>
|
||||
<input type="date" id="filter-date-start" onchange="dashboard.applyFilters()">
|
||||
<span>to</span>
|
||||
<input type="date" id="filter-date-end" onchange="dashboard.applyFilters()">
|
||||
</div>
|
||||
```
|
||||
|
||||
```javascript
|
||||
function filterByDateRange(data, dateField, startDate, endDate) {
|
||||
return data.filter(row => {
|
||||
const rowDate = new Date(row[dateField]);
|
||||
if (startDate && rowDate < new Date(startDate)) return false;
|
||||
if (endDate && rowDate > new Date(endDate)) return false;
|
||||
return true;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Combined Filter Logic
|
||||
|
||||
```javascript
|
||||
applyFilters() {
|
||||
const region = getFilterValue('filter-region');
|
||||
const category = getFilterValue('filter-category');
|
||||
const startDate = document.getElementById('filter-date-start').value;
|
||||
const endDate = document.getElementById('filter-date-end').value;
|
||||
|
||||
this.filteredData = this.rawData.filter(row => {
|
||||
if (region && row.region !== region) return false;
|
||||
if (category && row.category !== category) return false;
|
||||
if (startDate && row.date < startDate) return false;
|
||||
if (endDate && row.date > endDate) return false;
|
||||
return true;
|
||||
});
|
||||
|
||||
this.renderKPIs();
|
||||
this.updateCharts();
|
||||
this.renderTable();
|
||||
}
|
||||
```
|
||||
|
||||
### Sortable Table
|
||||
|
||||
```javascript
|
||||
function renderTable(containerId, data, columns) {
|
||||
const container = document.getElementById(containerId);
|
||||
let sortCol = null;
|
||||
let sortDir = 'desc';
|
||||
|
||||
function render(sortedData) {
|
||||
let html = '<table class="data-table">';
|
||||
|
||||
// Header
|
||||
html += '<thead><tr>';
|
||||
columns.forEach(col => {
|
||||
const arrow = sortCol === col.field
|
||||
? (sortDir === 'asc' ? ' ▲' : ' ▼')
|
||||
: '';
|
||||
html += `<th onclick="sortTable('${col.field}')" style="cursor:pointer">${col.label}${arrow}</th>`;
|
||||
});
|
||||
html += '</tr></thead>';
|
||||
|
||||
// Body
|
||||
html += '<tbody>';
|
||||
sortedData.forEach(row => {
|
||||
html += '<tr>';
|
||||
columns.forEach(col => {
|
||||
const value = col.format ? formatValue(row[col.field], col.format) : row[col.field];
|
||||
html += `<td>${value}</td>`;
|
||||
});
|
||||
html += '</tr>';
|
||||
});
|
||||
html += '</tbody></table>';
|
||||
|
||||
container.innerHTML = html;
|
||||
}
|
||||
|
||||
window.sortTable = function(field) {
|
||||
if (sortCol === field) {
|
||||
sortDir = sortDir === 'asc' ? 'desc' : 'asc';
|
||||
} else {
|
||||
sortCol = field;
|
||||
sortDir = 'desc';
|
||||
}
|
||||
const sorted = [...data].sort((a, b) => {
|
||||
const aVal = a[field], bVal = b[field];
|
||||
const cmp = aVal < bVal ? -1 : aVal > bVal ? 1 : 0;
|
||||
return sortDir === 'asc' ? cmp : -cmp;
|
||||
});
|
||||
render(sorted);
|
||||
};
|
||||
|
||||
render(data);
|
||||
}
|
||||
```
|
||||
|
||||
## CSS Styling for Dashboards
|
||||
|
||||
### Color System
|
||||
|
||||
```css
|
||||
:root {
|
||||
/* Background layers */
|
||||
--bg-primary: #f8f9fa;
|
||||
--bg-card: #ffffff;
|
||||
--bg-header: #1a1a2e;
|
||||
|
||||
/* Text */
|
||||
--text-primary: #212529;
|
||||
--text-secondary: #6c757d;
|
||||
--text-on-dark: #ffffff;
|
||||
|
||||
/* Accent colors for data */
|
||||
--color-1: #4C72B0;
|
||||
--color-2: #DD8452;
|
||||
--color-3: #55A868;
|
||||
--color-4: #C44E52;
|
||||
--color-5: #8172B3;
|
||||
--color-6: #937860;
|
||||
|
||||
/* Status colors */
|
||||
--positive: #28a745;
|
||||
--negative: #dc3545;
|
||||
--neutral: #6c757d;
|
||||
|
||||
/* Spacing */
|
||||
--gap: 16px;
|
||||
--radius: 8px;
|
||||
}
|
||||
```
|
||||
|
||||
### Layout
|
||||
|
||||
```css
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: var(--bg-primary);
|
||||
color: var(--text-primary);
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.dashboard-container {
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
padding: var(--gap);
|
||||
}
|
||||
|
||||
.dashboard-header {
|
||||
background: var(--bg-header);
|
||||
color: var(--text-on-dark);
|
||||
padding: 20px 24px;
|
||||
border-radius: var(--radius);
|
||||
margin-bottom: var(--gap);
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.dashboard-header h1 {
|
||||
font-size: 20px;
|
||||
font-weight: 600;
|
||||
}
|
||||
```
|
||||
|
||||
### KPI Cards
|
||||
|
||||
```css
|
||||
.kpi-row {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
gap: var(--gap);
|
||||
margin-bottom: var(--gap);
|
||||
}
|
||||
|
||||
.kpi-card {
|
||||
background: var(--bg-card);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px 24px;
|
||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
|
||||
}
|
||||
|
||||
.kpi-label {
|
||||
font-size: 13px;
|
||||
color: var(--text-secondary);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.kpi-value {
|
||||
font-size: 28px;
|
||||
font-weight: 700;
|
||||
color: var(--text-primary);
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.kpi-change {
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.kpi-change.positive { color: var(--positive); }
|
||||
.kpi-change.negative { color: var(--negative); }
|
||||
```
|
||||
|
||||
### Chart Containers
|
||||
|
||||
```css
|
||||
.chart-row {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(400px, 1fr));
|
||||
gap: var(--gap);
|
||||
margin-bottom: var(--gap);
|
||||
}
|
||||
|
||||
.chart-container {
|
||||
background: var(--bg-card);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px 24px;
|
||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
|
||||
}
|
||||
|
||||
.chart-container h3 {
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
color: var(--text-primary);
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
.chart-container canvas {
|
||||
max-height: 300px;
|
||||
}
|
||||
```
|
||||
|
||||
### Filters
|
||||
|
||||
```css
|
||||
.filters {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.filter-group {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.filter-group label {
|
||||
font-size: 12px;
|
||||
color: rgba(255, 255, 255, 0.7);
|
||||
}
|
||||
|
||||
.filter-group select,
|
||||
.filter-group input[type="date"] {
|
||||
padding: 6px 10px;
|
||||
border: 1px solid rgba(255, 255, 255, 0.2);
|
||||
border-radius: 4px;
|
||||
background: rgba(255, 255, 255, 0.1);
|
||||
color: var(--text-on-dark);
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.filter-group select option {
|
||||
background: var(--bg-header);
|
||||
color: var(--text-on-dark);
|
||||
}
|
||||
```
|
||||
|
||||
### Data Table
|
||||
|
||||
```css
|
||||
.table-section {
|
||||
background: var(--bg-card);
|
||||
border-radius: var(--radius);
|
||||
padding: 20px 24px;
|
||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.08);
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.data-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.data-table thead th {
|
||||
text-align: left;
|
||||
padding: 10px 12px;
|
||||
border-bottom: 2px solid #dee2e6;
|
||||
color: var(--text-secondary);
|
||||
font-weight: 600;
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
white-space: nowrap;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
.data-table thead th:hover {
|
||||
color: var(--text-primary);
|
||||
background: #f8f9fa;
|
||||
}
|
||||
|
||||
.data-table tbody td {
|
||||
padding: 10px 12px;
|
||||
border-bottom: 1px solid #f0f0f0;
|
||||
}
|
||||
|
||||
.data-table tbody tr:hover {
|
||||
background: #f8f9fa;
|
||||
}
|
||||
|
||||
.data-table tbody tr:last-child td {
|
||||
border-bottom: none;
|
||||
}
|
||||
```
|
||||
|
||||
### Responsive Design
|
||||
|
||||
```css
|
||||
@media (max-width: 768px) {
|
||||
.dashboard-header {
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
}
|
||||
|
||||
.kpi-row {
|
||||
grid-template-columns: repeat(2, 1fr);
|
||||
}
|
||||
|
||||
.chart-row {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.filters {
|
||||
flex-direction: column;
|
||||
align-items: flex-start;
|
||||
}
|
||||
}
|
||||
|
||||
@media print {
|
||||
body { background: white; }
|
||||
.dashboard-container { max-width: none; }
|
||||
.filters { display: none; }
|
||||
.chart-container { break-inside: avoid; }
|
||||
.kpi-card { border: 1px solid #dee2e6; box-shadow: none; }
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations for Large Datasets
|
||||
|
||||
### Data Size Guidelines
|
||||
|
||||
| Data Size | Approach |
|
||||
|---|---|
|
||||
| <1,000 rows | Embed directly in HTML. Full interactivity. |
|
||||
| 1,000 - 10,000 rows | Embed in HTML. May need to pre-aggregate for charts. |
|
||||
| 10,000 - 100,000 rows | Pre-aggregate server-side. Embed only aggregated data. |
|
||||
| >100,000 rows | Not suitable for client-side dashboard. Use a BI tool or paginate. |
|
||||
|
||||
### Pre-Aggregation Pattern
|
||||
|
||||
Instead of embedding raw data and aggregating in the browser:
|
||||
|
||||
```javascript
|
||||
// DON'T: embed 50,000 raw rows
|
||||
const RAW_DATA = [/* 50,000 rows */];
|
||||
|
||||
// DO: pre-aggregate before embedding
|
||||
const CHART_DATA = {
|
||||
monthly_revenue: [
|
||||
{ month: '2024-01', revenue: 150000, orders: 1200 },
|
||||
{ month: '2024-02', revenue: 165000, orders: 1350 },
|
||||
// ... 12 rows instead of 50,000
|
||||
],
|
||||
top_products: [
|
||||
{ product: 'Widget A', revenue: 45000 },
|
||||
// ... 10 rows
|
||||
],
|
||||
kpis: {
|
||||
total_revenue: 1980000,
|
||||
total_orders: 15600,
|
||||
avg_order_value: 127,
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Chart Performance
|
||||
|
||||
- Limit line charts to <500 data points per series (downsample if needed)
|
||||
- Limit bar charts to <50 categories
|
||||
- For scatter plots, cap at 1,000 points (use sampling for larger datasets)
|
||||
- Disable animations for dashboards with many charts: `animation: false` in Chart.js options
|
||||
- Use `Chart.update('none')` instead of `Chart.update()` for filter-triggered updates
|
||||
|
||||
### DOM Performance
|
||||
|
||||
- Limit data tables to 100-200 visible rows. Add pagination for more.
|
||||
- Use `requestAnimationFrame` for coordinated chart updates
|
||||
- Avoid rebuilding the entire DOM on filter change -- update only changed elements
|
||||
|
||||
```javascript
|
||||
// Efficient table pagination
|
||||
function renderTablePage(data, page, pageSize = 50) {
|
||||
const start = page * pageSize;
|
||||
const end = Math.min(start + pageSize, data.length);
|
||||
const pageData = data.slice(start, end);
|
||||
// Render only pageData
|
||||
// Show pagination controls: "Showing 1-50 of 2,340"
|
||||
}
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/build-dashboard Monthly sales dashboard with revenue trend, top products, and regional breakdown. Data is in the orders table.
|
||||
```
|
||||
|
||||
```
|
||||
/build-dashboard Here's our support ticket data [pastes CSV]. Build a dashboard showing volume by priority, response time trends, and resolution rates.
|
||||
```
|
||||
|
||||
```
|
||||
/build-dashboard Create a template executive dashboard for a SaaS company showing MRR, churn, new customers, and NPS. Use sample data.
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Dashboards are fully self-contained HTML files -- share them with anyone by sending the file
|
||||
- For real-time dashboards, consider connecting to a BI tool instead. These dashboards are point-in-time snapshots
|
||||
- Request "dark mode" or "presentation mode" for different styling
|
||||
- You can request a specific color scheme to match your brand
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
name: create-viz
|
||||
description: Create publication-quality visualizations with Python. Use when turning query results or a DataFrame into a chart, selecting the right chart type for a trend or comparison, generating a plot for a report or presentation, or needing an interactive chart with hover and zoom.
|
||||
argument-hint: "<data source> [chart type]"
|
||||
---
|
||||
|
||||
# /create-viz - Create Visualizations
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Create publication-quality data visualizations using Python. Generates charts from data with best practices for clarity, accuracy, and design.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/create-viz <data source> [chart type] [additional instructions]
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the Request
|
||||
|
||||
Determine:
|
||||
|
||||
- **Data source**: Query results, pasted data, CSV/Excel file, or data to be queried
|
||||
- **Chart type**: Explicitly requested or needs to be recommended
|
||||
- **Purpose**: Exploration, presentation, report, dashboard component
|
||||
- **Audience**: Technical team, executives, external stakeholders
|
||||
|
||||
### 2. Get the Data
|
||||
|
||||
**If data warehouse is connected and data needs querying:**
|
||||
1. Write and execute the query
|
||||
2. Load results into a pandas DataFrame
|
||||
|
||||
**If data is pasted or uploaded:**
|
||||
1. Parse the data into a pandas DataFrame
|
||||
2. Clean and prepare as needed (type conversions, null handling)
|
||||
|
||||
**If data is from a previous analysis in the conversation:**
|
||||
1. Reference the existing data
|
||||
|
||||
### 3. Select Chart Type
|
||||
|
||||
If the user didn't specify a chart type, recommend one based on the data and question:
|
||||
|
||||
| Data Relationship | Recommended Chart |
|
||||
|---|---|
|
||||
| Trend over time | Line chart |
|
||||
| Comparison across categories | Bar chart (horizontal if many categories) |
|
||||
| Part-to-whole composition | Stacked bar or area chart (avoid pie charts unless <6 categories) |
|
||||
| Distribution of values | Histogram or box plot |
|
||||
| Correlation between two variables | Scatter plot |
|
||||
| Two-variable comparison over time | Dual-axis line or grouped bar |
|
||||
| Geographic data | Choropleth map |
|
||||
| Ranking | Horizontal bar chart |
|
||||
| Flow or process | Sankey diagram |
|
||||
| Matrix of relationships | Heatmap |
|
||||
|
||||
Explain the recommendation briefly if the user didn't specify.
|
||||
|
||||
### 4. Generate the Visualization
|
||||
|
||||
Write Python code using one of these libraries based on the need:
|
||||
|
||||
- **matplotlib + seaborn**: Best for static, publication-quality charts. Default choice.
|
||||
- **plotly**: Best for interactive charts or when the user requests interactivity.
|
||||
|
||||
**Code requirements:**
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import seaborn as sns
|
||||
import pandas as pd
|
||||
|
||||
# Set professional style
|
||||
plt.style.use('seaborn-v0_8-whitegrid')
|
||||
sns.set_palette("husl")
|
||||
|
||||
# Create figure with appropriate size
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
|
||||
# [chart-specific code]
|
||||
|
||||
# Always include:
|
||||
ax.set_title('Clear, Descriptive Title', fontsize=14, fontweight='bold')
|
||||
ax.set_xlabel('X-Axis Label', fontsize=11)
|
||||
ax.set_ylabel('Y-Axis Label', fontsize=11)
|
||||
|
||||
# Format numbers appropriately
|
||||
# - Percentages: '45.2%' not '0.452'
|
||||
# - Currency: '$1.2M' not '1200000'
|
||||
# - Large numbers: '2.3K' or '1.5M' not '2300' or '1500000'
|
||||
|
||||
# Remove chart junk
|
||||
ax.spines['top'].set_visible(False)
|
||||
ax.spines['right'].set_visible(False)
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('chart_name.png', dpi=150, bbox_inches='tight')
|
||||
plt.show()
|
||||
```
|
||||
|
||||
### 5. Apply Design Best Practices
|
||||
|
||||
**Color:**
|
||||
- Use a consistent, colorblind-friendly palette
|
||||
- Use color meaningfully (not decoratively)
|
||||
- Highlight the key data point or trend with a contrasting color
|
||||
- Grey out less important reference data
|
||||
|
||||
**Typography:**
|
||||
- Descriptive title that states the insight, not just the metric (e.g., "Revenue grew 23% YoY" not "Revenue by Month")
|
||||
- Readable axis labels (not rotated 90 degrees if avoidable)
|
||||
- Data labels on key points when they add clarity
|
||||
|
||||
**Layout:**
|
||||
- Appropriate whitespace and margins
|
||||
- Legend placement that doesn't obscure data
|
||||
- Sorted categories by value (not alphabetically) unless there's a natural order
|
||||
|
||||
**Accuracy:**
|
||||
- Y-axis starts at zero for bar charts
|
||||
- No misleading axis breaks without clear notation
|
||||
- Consistent scales when comparing panels
|
||||
- Appropriate precision (don't show 10 decimal places)
|
||||
|
||||
### 6. Save and Present
|
||||
|
||||
1. Save the chart as a PNG file with descriptive name
|
||||
2. Display the chart to the user
|
||||
3. Provide the code used so they can modify it
|
||||
4. Suggest variations (different chart type, different grouping, zoomed time range)
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/create-viz Show monthly revenue for the last 12 months as a line chart with the trend highlighted
|
||||
```
|
||||
|
||||
```
|
||||
/create-viz Here's our NPS data by product: [pastes data]. Create a horizontal bar chart ranking products by score.
|
||||
```
|
||||
|
||||
```
|
||||
/create-viz Query the orders table and create a heatmap of order volume by day-of-week and hour
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- If you want interactive charts (hover, zoom, filter), mention "interactive" and Claude will use plotly
|
||||
- Specify "presentation" if you need larger fonts and higher contrast
|
||||
- You can request multiple charts at once (e.g., "create a 2x2 grid of charts showing...")
|
||||
- Charts are saved to your current directory as PNG files
|
||||
@@ -0,0 +1,227 @@
|
||||
---
|
||||
name: data-context-extractor
|
||||
description: >
|
||||
Generate or improve a company-specific data analysis skill by extracting tribal knowledge from analysts.
|
||||
|
||||
BOOTSTRAP MODE - Triggers: "Create a data context skill", "Set up data analysis for our warehouse",
|
||||
"Help me create a skill for our database", "Generate a data skill for [company]"
|
||||
→ Discovers schemas, asks key questions, generates initial skill with reference files
|
||||
|
||||
ITERATION MODE - Triggers: "Add context about [domain]", "The skill needs more info about [topic]",
|
||||
"Update the data skill with [metrics/tables/terminology]", "Improve the [domain] reference"
|
||||
→ Loads existing skill, asks targeted questions, appends/updates reference files
|
||||
|
||||
Use when data analysts want Claude to understand their company's specific data warehouse,
|
||||
terminology, metrics definitions, and common query patterns.
|
||||
---
|
||||
|
||||
# Data Context Extractor
|
||||
|
||||
A meta-skill that extracts company-specific data knowledge from analysts and generates tailored data analysis skills.
|
||||
|
||||
## How It Works
|
||||
|
||||
This skill has two modes:
|
||||
|
||||
1. **Bootstrap Mode**: Create a new data analysis skill from scratch
|
||||
2. **Iteration Mode**: Improve an existing skill by adding domain-specific reference files
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap Mode
|
||||
|
||||
Use when: User wants to create a new data context skill for their warehouse.
|
||||
|
||||
### Phase 1: Database Connection & Discovery
|
||||
|
||||
**Step 1: Identify the database type**
|
||||
|
||||
Ask: "What data warehouse are you using?"
|
||||
|
||||
Common options:
|
||||
- **BigQuery**
|
||||
- **Snowflake**
|
||||
- **PostgreSQL/Redshift**
|
||||
- **Databricks**
|
||||
|
||||
Use `~~data warehouse` tools (query and schema) to connect. If unclear, check available MCP tools in the current session.
|
||||
|
||||
**Step 2: Explore the schema**
|
||||
|
||||
Use `~~data warehouse` schema tools to:
|
||||
1. List available datasets/schemas
|
||||
2. Identify the most important tables (ask user: "Which 3-5 tables do analysts query most often?")
|
||||
3. Pull schema details for those key tables
|
||||
|
||||
Sample exploration queries by dialect:
|
||||
```sql
|
||||
-- BigQuery: List datasets
|
||||
SELECT schema_name FROM INFORMATION_SCHEMA.SCHEMATA
|
||||
|
||||
-- BigQuery: List tables in a dataset
|
||||
SELECT table_name FROM `project.dataset.INFORMATION_SCHEMA.TABLES`
|
||||
|
||||
-- Snowflake: List schemas
|
||||
SHOW SCHEMAS IN DATABASE my_database
|
||||
|
||||
-- Snowflake: List tables
|
||||
SHOW TABLES IN SCHEMA my_schema
|
||||
```
|
||||
|
||||
### Phase 2: Core Questions (Ask These)
|
||||
|
||||
After schema discovery, ask these questions conversationally (not all at once):
|
||||
|
||||
**Entity Disambiguation (Critical)**
|
||||
> "When people here say 'user' or 'customer', what exactly do they mean? Are there different types?"
|
||||
|
||||
Listen for:
|
||||
- Multiple entity types (user vs account vs organization)
|
||||
- Relationships between them (1:1, 1:many, many:many)
|
||||
- Which ID fields link them together
|
||||
|
||||
**Primary Identifiers**
|
||||
> "What's the main identifier for a [customer/user/account]? Are there multiple IDs for the same entity?"
|
||||
|
||||
Listen for:
|
||||
- Primary keys vs business keys
|
||||
- UUID vs integer IDs
|
||||
- Legacy ID systems
|
||||
|
||||
**Key Metrics**
|
||||
> "What are the 2-3 metrics people ask about most? How is each one calculated?"
|
||||
|
||||
Listen for:
|
||||
- Exact formulas (ARR = monthly_revenue × 12)
|
||||
- Which tables/columns feed each metric
|
||||
- Time period conventions (trailing 7 days, calendar month, etc.)
|
||||
|
||||
**Data Hygiene**
|
||||
> "What should ALWAYS be filtered out of queries? (test data, fraud, internal users, etc.)"
|
||||
|
||||
Listen for:
|
||||
- Standard WHERE clauses to always include
|
||||
- Flag columns that indicate exclusions (is_test, is_internal, is_fraud)
|
||||
- Specific values to exclude (status = 'deleted')
|
||||
|
||||
**Common Gotchas**
|
||||
> "What mistakes do new analysts typically make with this data?"
|
||||
|
||||
Listen for:
|
||||
- Confusing column names
|
||||
- Timezone issues
|
||||
- NULL handling quirks
|
||||
- Historical vs current state tables
|
||||
|
||||
### Phase 3: Generate the Skill
|
||||
|
||||
Create a skill with this structure:
|
||||
|
||||
```
|
||||
[company]-data-analyst/
|
||||
├── SKILL.md
|
||||
└── references/
|
||||
├── entities.md # Entity definitions and relationships
|
||||
├── metrics.md # KPI calculations
|
||||
├── tables/ # One file per domain
|
||||
│ ├── [domain1].md
|
||||
│ └── [domain2].md
|
||||
└── dashboards.json # Optional: existing dashboards catalog
|
||||
```
|
||||
|
||||
**SKILL.md Template**: See `references/skill-template.md`
|
||||
|
||||
**SQL Dialect Section**: See `references/sql-dialects.md` and include the appropriate dialect notes.
|
||||
|
||||
**Reference File Template**: See `references/domain-template.md`
|
||||
|
||||
### Phase 4: Package and Deliver
|
||||
|
||||
1. Create all files in the skill directory
|
||||
2. Package as a zip file
|
||||
3. Present to user with summary of what was captured
|
||||
|
||||
---
|
||||
|
||||
## Iteration Mode
|
||||
|
||||
Use when: User has an existing skill but needs to add more context.
|
||||
|
||||
### Step 1: Load Existing Skill
|
||||
|
||||
Ask user to upload their existing skill (zip or folder), or locate it if already in the session.
|
||||
|
||||
Read the current SKILL.md and reference files to understand what's already documented.
|
||||
|
||||
### Step 2: Identify the Gap
|
||||
|
||||
Ask: "What domain or topic needs more context? What queries are failing or producing wrong results?"
|
||||
|
||||
Common gaps:
|
||||
- A new data domain (marketing, finance, product, etc.)
|
||||
- Missing metric definitions
|
||||
- Undocumented table relationships
|
||||
- New terminology
|
||||
|
||||
### Step 3: Targeted Discovery
|
||||
|
||||
For the identified domain:
|
||||
|
||||
1. **Explore relevant tables**: Use `~~data warehouse` schema tools to find tables in that domain
|
||||
2. **Ask domain-specific questions**:
|
||||
- "What tables are used for [domain] analysis?"
|
||||
- "What are the key metrics for [domain]?"
|
||||
- "Any special filters or gotchas for [domain] data?"
|
||||
|
||||
3. **Generate new reference file**: Create `references/[domain].md` using the domain template
|
||||
|
||||
### Step 4: Update and Repackage
|
||||
|
||||
1. Add the new reference file
|
||||
2. Update SKILL.md's "Knowledge Base Navigation" section to include the new domain
|
||||
3. Repackage the skill
|
||||
4. Present the updated skill to user
|
||||
|
||||
---
|
||||
|
||||
## Reference File Standards
|
||||
|
||||
Each reference file should include:
|
||||
|
||||
### For Table Documentation
|
||||
- **Location**: Full table path
|
||||
- **Description**: What this table contains, when to use it
|
||||
- **Primary Key**: How to uniquely identify rows
|
||||
- **Update Frequency**: How often data refreshes
|
||||
- **Key Columns**: Table with column name, type, description, notes
|
||||
- **Relationships**: How this table joins to others
|
||||
- **Sample Queries**: 2-3 common query patterns
|
||||
|
||||
### For Metrics Documentation
|
||||
- **Metric Name**: Human-readable name
|
||||
- **Definition**: Plain English explanation
|
||||
- **Formula**: Exact calculation with column references
|
||||
- **Source Table(s)**: Where the data comes from
|
||||
- **Caveats**: Edge cases, exclusions, gotchas
|
||||
|
||||
### For Entity Documentation
|
||||
- **Entity Name**: What it's called
|
||||
- **Definition**: What it represents in the business
|
||||
- **Primary Table**: Where to find this entity
|
||||
- **ID Field(s)**: How to identify it
|
||||
- **Relationships**: How it relates to other entities
|
||||
- **Common Filters**: Standard exclusions (internal, test, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
Before delivering a generated skill, verify:
|
||||
|
||||
- [ ] SKILL.md has complete frontmatter (name, description)
|
||||
- [ ] Entity disambiguation section is clear
|
||||
- [ ] Key terminology is defined
|
||||
- [ ] Standard filters/exclusions are documented
|
||||
- [ ] At least 2-3 sample queries per domain
|
||||
- [ ] SQL uses correct dialect syntax
|
||||
- [ ] Reference files are linked from SKILL.md navigation section
|
||||
@@ -0,0 +1,147 @@
|
||||
# Domain Reference File Template
|
||||
|
||||
Use this template when creating reference files for specific data domains (e.g., revenue, users, marketing).
|
||||
|
||||
---
|
||||
|
||||
```markdown
|
||||
# [DOMAIN_NAME] Tables
|
||||
|
||||
This document contains [domain]-related tables, metrics, and query patterns.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Business Context
|
||||
|
||||
[2-3 sentences explaining what this domain covers and key concepts]
|
||||
|
||||
### Entity Clarification
|
||||
|
||||
**"[AMBIGUOUS_TERM]" can mean:**
|
||||
- **[MEANING_1]**: [DEFINITION] ([TABLE]: [ID_FIELD])
|
||||
- **[MEANING_2]**: [DEFINITION] ([TABLE]: [ID_FIELD])
|
||||
|
||||
Always clarify which one before querying.
|
||||
|
||||
### Standard Filters
|
||||
|
||||
For [domain] queries, always:
|
||||
```sql
|
||||
WHERE [STANDARD_FILTER_1]
|
||||
AND [STANDARD_FILTER_2]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Tables
|
||||
|
||||
### [TABLE_1_NAME]
|
||||
**Location**: `[project.dataset.table]` or `[schema.table]`
|
||||
**Description**: [What this table contains, when to use it]
|
||||
**Primary Key**: [COLUMN(S)]
|
||||
**Update Frequency**: [Daily/Hourly/Real-time] ([LAG] lag)
|
||||
**Partitioned By**: [PARTITION_COLUMN] (if applicable)
|
||||
|
||||
| Column | Type | Description | Notes |
|
||||
|--------|------|-------------|-------|
|
||||
| **[column_1]** | [TYPE] | [DESCRIPTION] | [GOTCHA_OR_CONTEXT] |
|
||||
| **[column_2]** | [TYPE] | [DESCRIPTION] | |
|
||||
| **[column_3]** | [TYPE] | [DESCRIPTION] | Nullable |
|
||||
|
||||
**Relationships**:
|
||||
- Joins to `[OTHER_TABLE]` on `[JOIN_KEY]`
|
||||
- Parent of `[CHILD_TABLE]` via `[FOREIGN_KEY]`
|
||||
|
||||
**Nested/Struct Fields** (if applicable):
|
||||
- `[struct_name].[field_1]`: [DESCRIPTION]
|
||||
- `[struct_name].[field_2]`: [DESCRIPTION]
|
||||
|
||||
---
|
||||
|
||||
### [TABLE_2_NAME]
|
||||
[REPEAT FORMAT]
|
||||
|
||||
---
|
||||
|
||||
## Key Metrics
|
||||
|
||||
| Metric | Definition | Table | Formula | Notes |
|
||||
|--------|------------|-------|---------|-------|
|
||||
| [METRIC_1] | [DEFINITION] | [TABLE] | `[FORMULA]` | [CAVEATS] |
|
||||
| [METRIC_2] | [DEFINITION] | [TABLE] | `[FORMULA]` | |
|
||||
|
||||
---
|
||||
|
||||
## Sample Queries
|
||||
|
||||
### [QUERY_PURPOSE_1]
|
||||
```sql
|
||||
-- [Brief description of what this query does]
|
||||
SELECT
|
||||
[columns]
|
||||
FROM [table]
|
||||
WHERE [standard_filters]
|
||||
GROUP BY [grouping]
|
||||
ORDER BY [ordering]
|
||||
```
|
||||
|
||||
### [QUERY_PURPOSE_2]
|
||||
```sql
|
||||
[ANOTHER_COMMON_QUERY]
|
||||
```
|
||||
|
||||
### [QUERY_PURPOSE_3]: [More Complex Pattern]
|
||||
```sql
|
||||
WITH [cte_name] AS (
|
||||
[CTE_LOGIC]
|
||||
)
|
||||
SELECT
|
||||
[final_columns]
|
||||
FROM [cte_name]
|
||||
[joins_and_filters]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
1. **[GOTCHA_1]**: [EXPLANATION]
|
||||
- Wrong: `[INCORRECT_APPROACH]`
|
||||
- Right: `[CORRECT_APPROACH]`
|
||||
|
||||
2. **[GOTCHA_2]**: [EXPLANATION]
|
||||
|
||||
---
|
||||
|
||||
## Related Dashboards (if applicable)
|
||||
|
||||
| Dashboard | Link | Use For |
|
||||
|-----------|------|---------|
|
||||
| [DASHBOARD_1] | [URL] | [DESCRIPTION] |
|
||||
| [DASHBOARD_2] | [URL] | [DESCRIPTION] |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tips for Creating Domain Files
|
||||
|
||||
1. **Start with the most-queried tables** - Don't try to document everything
|
||||
2. **Include column-level detail only for important columns** - Skip obvious ones like `created_at`
|
||||
3. **Real query examples > abstract descriptions** - Show don't tell
|
||||
4. **Document the gotchas prominently** - These save the most time
|
||||
5. **Keep sample queries runnable** - Use real table/column names
|
||||
6. **Note nested/struct fields explicitly** - These trip people up
|
||||
|
||||
## Suggested Domain Files
|
||||
|
||||
Common domains to document (create separate files for each):
|
||||
|
||||
- `revenue.md` - Billing, subscriptions, ARR, transactions
|
||||
- `users.md` - Accounts, authentication, user attributes
|
||||
- `product.md` - Feature usage, events, sessions
|
||||
- `growth.md` - DAU/WAU/MAU, retention, activation
|
||||
- `sales.md` - CRM, pipeline, opportunities
|
||||
- `marketing.md` - Campaigns, attribution, leads
|
||||
- `support.md` - Tickets, CSAT, response times
|
||||
@@ -0,0 +1,198 @@
|
||||
# Example: Generated Skill
|
||||
|
||||
This is an example of what a generated skill looks like after the bootstrap process. This example is for a fictional e-commerce company called "ShopCo" using Snowflake.
|
||||
|
||||
---
|
||||
|
||||
## Example SKILL.md
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: shopco-data-analyst
|
||||
description: "ShopCo data analysis skill for Snowflake. Provides context for querying e-commerce data including customer, order, and product analytics. Use when analyzing ShopCo data for: (1) Revenue and order metrics, (2) Customer behavior and retention, (3) Product performance, or any data questions requiring ShopCo-specific context."
|
||||
---
|
||||
|
||||
# ShopCo Data Analysis
|
||||
|
||||
## SQL Dialect: Snowflake
|
||||
|
||||
- **Table references**: `SHOPCO_DW.SCHEMA.TABLE` or with quotes for case-sensitive: `"Column_Name"`
|
||||
- **Safe division**: `DIV0(a, b)` returns 0, `DIV0NULL(a, b)` returns NULL
|
||||
- **Date functions**:
|
||||
- `DATE_TRUNC('MONTH', date_col)`
|
||||
- `DATEADD(DAY, -1, date_col)`
|
||||
- `DATEDIFF(DAY, start_date, end_date)`
|
||||
- **Column exclusion**: `SELECT * EXCLUDE (column_to_exclude)`
|
||||
|
||||
---
|
||||
|
||||
## Entity Disambiguation
|
||||
|
||||
**"Customer" can mean:**
|
||||
- **User**: A login account that can browse and save items (CORE.DIM_USERS: user_id)
|
||||
- **Customer**: A user who has made at least one purchase (CORE.DIM_CUSTOMERS: customer_id)
|
||||
- **Account**: A billing entity, can have multiple users in B2B (CORE.DIM_ACCOUNTS: account_id)
|
||||
|
||||
**Relationships:**
|
||||
- User → Customer: 1:1 (customer_id = user_id for purchasers)
|
||||
- Account → User: 1:many (join on account_id)
|
||||
|
||||
---
|
||||
|
||||
## Business Terminology
|
||||
|
||||
| Term | Definition | Notes |
|
||||
|------|------------|-------|
|
||||
| GMV | Gross Merchandise Value - total order value before returns/discounts | Use for top-line reporting |
|
||||
| NMV | Net Merchandise Value - GMV minus returns and discounts | Use for actual revenue |
|
||||
| AOV | Average Order Value - NMV / order count | Exclude $0 orders |
|
||||
| LTV | Lifetime Value - total NMV per customer since first order | Rolling calc, updates daily |
|
||||
| CAC | Customer Acquisition Cost - marketing spend / new customers | By cohort month |
|
||||
|
||||
---
|
||||
|
||||
## Standard Filters
|
||||
|
||||
Always apply these filters unless explicitly told otherwise:
|
||||
|
||||
```sql
|
||||
-- Exclude test and internal orders
|
||||
WHERE order_status != 'TEST'
|
||||
AND customer_type != 'INTERNAL'
|
||||
AND is_employee_order = FALSE
|
||||
|
||||
-- Exclude cancelled orders for revenue metrics
|
||||
AND order_status NOT IN ('CANCELLED', 'FRAUDULENT')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Metrics
|
||||
|
||||
### Gross Merchandise Value (GMV)
|
||||
- **Definition**: Total value of all orders placed
|
||||
- **Formula**: `SUM(order_total_gross)`
|
||||
- **Source**: `CORE.FCT_ORDERS.order_total_gross`
|
||||
- **Time grain**: Daily, aggregated to weekly/monthly
|
||||
- **Caveats**: Includes orders that may later be cancelled or returned
|
||||
|
||||
### Net Revenue
|
||||
- **Definition**: Actual revenue after returns and discounts
|
||||
- **Formula**: `SUM(order_total_gross - return_amount - discount_amount)`
|
||||
- **Source**: `CORE.FCT_ORDERS`
|
||||
- **Caveats**: Returns can occur up to 90 days post-order; use settled_revenue for finalized numbers
|
||||
|
||||
---
|
||||
|
||||
## Knowledge Base Navigation
|
||||
|
||||
| Domain | Reference File | Use For |
|
||||
|--------|----------------|---------|
|
||||
| Orders | `references/orders.md` | Order tables, GMV/NMV calculations |
|
||||
| Customers | `references/customers.md` | User/customer entities, LTV, cohorts |
|
||||
| Products | `references/products.md` | Catalog, inventory, categories |
|
||||
|
||||
---
|
||||
|
||||
## Common Query Patterns
|
||||
|
||||
### Daily GMV by Channel
|
||||
```sql
|
||||
SELECT
|
||||
DATE_TRUNC('DAY', order_timestamp) AS order_date,
|
||||
channel,
|
||||
SUM(order_total_gross) AS gmv,
|
||||
COUNT(DISTINCT order_id) AS order_count
|
||||
FROM SHOPCO_DW.CORE.FCT_ORDERS
|
||||
WHERE order_status NOT IN ('TEST', 'CANCELLED', 'FRAUDULENT')
|
||||
AND order_timestamp >= DATEADD(DAY, -30, CURRENT_DATE())
|
||||
GROUP BY 1, 2
|
||||
ORDER BY 1 DESC, 3 DESC
|
||||
```
|
||||
|
||||
### Customer Cohort Retention
|
||||
```sql
|
||||
WITH cohorts AS (
|
||||
SELECT
|
||||
customer_id,
|
||||
DATE_TRUNC('MONTH', first_order_date) AS cohort_month
|
||||
FROM SHOPCO_DW.CORE.DIM_CUSTOMERS
|
||||
)
|
||||
SELECT
|
||||
c.cohort_month,
|
||||
DATEDIFF(MONTH, c.cohort_month, DATE_TRUNC('MONTH', o.order_timestamp)) AS months_since_first,
|
||||
COUNT(DISTINCT c.customer_id) AS active_customers
|
||||
FROM cohorts c
|
||||
JOIN SHOPCO_DW.CORE.FCT_ORDERS o ON c.customer_id = o.customer_id
|
||||
WHERE o.order_status NOT IN ('TEST', 'CANCELLED')
|
||||
GROUP BY 1, 2
|
||||
ORDER BY 1, 2
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example references/orders.md
|
||||
|
||||
```markdown
|
||||
# Orders Tables
|
||||
|
||||
Order and transaction data for ShopCo.
|
||||
|
||||
---
|
||||
|
||||
## Key Tables
|
||||
|
||||
### FCT_ORDERS
|
||||
**Location**: `SHOPCO_DW.CORE.FCT_ORDERS`
|
||||
**Description**: Fact table of all orders. One row per order.
|
||||
**Primary Key**: `order_id`
|
||||
**Update Frequency**: Hourly (15 min lag)
|
||||
**Partitioned By**: `order_date`
|
||||
|
||||
| Column | Type | Description | Notes |
|
||||
|--------|------|-------------|-------|
|
||||
| **order_id** | VARCHAR | Unique order identifier | |
|
||||
| **customer_id** | VARCHAR | FK to DIM_CUSTOMERS | NULL for guest checkout |
|
||||
| **order_timestamp** | TIMESTAMP_NTZ | When order was placed | UTC |
|
||||
| **order_date** | DATE | Date portion of order_timestamp | Partition column |
|
||||
| **order_status** | VARCHAR | Current status | PENDING, SHIPPED, DELIVERED, CANCELLED, RETURNED |
|
||||
| **channel** | VARCHAR | Acquisition channel | WEB, APP, MARKETPLACE |
|
||||
| **order_total_gross** | DECIMAL(12,2) | Pre-discount total | |
|
||||
| **discount_amount** | DECIMAL(12,2) | Total discounts applied | |
|
||||
| **return_amount** | DECIMAL(12,2) | Value of returned items | Updates async |
|
||||
|
||||
**Relationships**:
|
||||
- Joins to `DIM_CUSTOMERS` on `customer_id`
|
||||
- Parent of `FCT_ORDER_ITEMS` via `order_id`
|
||||
|
||||
---
|
||||
|
||||
## Sample Queries
|
||||
|
||||
### Orders with Returns Rate
|
||||
```sql
|
||||
SELECT
|
||||
DATE_TRUNC('WEEK', order_date) AS week,
|
||||
COUNT(*) AS total_orders,
|
||||
SUM(CASE WHEN return_amount > 0 THEN 1 ELSE 0 END) AS orders_with_returns,
|
||||
DIV0(SUM(CASE WHEN return_amount > 0 THEN 1 ELSE 0 END), COUNT(*)) AS return_rate
|
||||
FROM SHOPCO_DW.CORE.FCT_ORDERS
|
||||
WHERE order_status NOT IN ('TEST', 'CANCELLED')
|
||||
AND order_date >= DATEADD(MONTH, -3, CURRENT_DATE())
|
||||
GROUP BY 1
|
||||
ORDER BY 1
|
||||
```
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
This example demonstrates:
|
||||
- Complete frontmatter with triggering description
|
||||
- Dialect-specific SQL notes
|
||||
- Clear entity disambiguation
|
||||
- Terminology glossary
|
||||
- Standard filters as copy-paste SQL
|
||||
- Metric definitions with formulas
|
||||
- Navigation to reference files
|
||||
- Real, runnable query examples
|
||||
@@ -0,0 +1,148 @@
|
||||
# Generated Skill Template
|
||||
|
||||
Use this template when generating a new data analysis skill. Replace all `[PLACEHOLDER]` values.
|
||||
|
||||
---
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: [company]-data-analyst
|
||||
description: "[COMPANY] data analysis skill. Provides context for querying [WAREHOUSE_TYPE] including entity definitions, metric calculations, and common query patterns. Use when analyzing [COMPANY] data for: (1) [PRIMARY_USE_CASE_1], (2) [PRIMARY_USE_CASE_2], (3) [PRIMARY_USE_CASE_3], or any data questions requiring [COMPANY]-specific context."
|
||||
---
|
||||
|
||||
# [COMPANY] Data Analysis
|
||||
|
||||
## SQL Dialect: [WAREHOUSE_TYPE]
|
||||
|
||||
[INSERT APPROPRIATE DIALECT SECTION FROM sql-dialects.md]
|
||||
|
||||
---
|
||||
|
||||
## Entity Disambiguation
|
||||
|
||||
When users mention these terms, clarify which entity they mean:
|
||||
|
||||
[EXAMPLE FORMAT - customize based on discovery:]
|
||||
|
||||
**"User" can mean:**
|
||||
- **Account**: An individual login/profile ([PRIMARY_TABLE]: [ID_FIELD])
|
||||
- **Organization**: A billing entity that can have multiple accounts ([ORG_TABLE]: [ORG_ID])
|
||||
- **[OTHER_TYPE]**: [DEFINITION] ([TABLE]: [ID])
|
||||
|
||||
**Relationships:**
|
||||
- [ENTITY_1] → [ENTITY_2]: [RELATIONSHIP_TYPE] (join on [JOIN_KEY])
|
||||
|
||||
---
|
||||
|
||||
## Business Terminology
|
||||
|
||||
| Term | Definition | Notes |
|
||||
|------|------------|-------|
|
||||
| [TERM_1] | [DEFINITION] | [CONTEXT/GOTCHA] |
|
||||
| [TERM_2] | [DEFINITION] | [CONTEXT/GOTCHA] |
|
||||
| [ACRONYM] | [FULL_NAME] - [EXPLANATION] | |
|
||||
|
||||
---
|
||||
|
||||
## Standard Filters
|
||||
|
||||
Always apply these filters unless explicitly told otherwise:
|
||||
|
||||
```sql
|
||||
-- Exclude test/internal data
|
||||
WHERE [TEST_FLAG_COLUMN] = FALSE
|
||||
AND [INTERNAL_FLAG_COLUMN] = FALSE
|
||||
|
||||
-- Exclude invalid/fraud
|
||||
AND [STATUS_COLUMN] != '[EXCLUDED_STATUS]'
|
||||
|
||||
-- [OTHER STANDARD EXCLUSIONS]
|
||||
```
|
||||
|
||||
**When to override:**
|
||||
- [SCENARIO_1]: Include [NORMALLY_EXCLUDED] when [CONDITION]
|
||||
|
||||
---
|
||||
|
||||
## Key Metrics
|
||||
|
||||
### [METRIC_1_NAME]
|
||||
- **Definition**: [PLAIN_ENGLISH_EXPLANATION]
|
||||
- **Formula**: `[EXACT_CALCULATION]`
|
||||
- **Source**: `[TABLE_NAME].[COLUMN_NAME]`
|
||||
- **Time grain**: [DAILY/WEEKLY/MONTHLY]
|
||||
- **Caveats**: [EDGE_CASES_OR_GOTCHAS]
|
||||
|
||||
### [METRIC_2_NAME]
|
||||
[REPEAT FORMAT]
|
||||
|
||||
---
|
||||
|
||||
## Data Freshness
|
||||
|
||||
| Table | Update Frequency | Typical Lag |
|
||||
|-------|------------------|-------------|
|
||||
| [TABLE_1] | [FREQUENCY] | [LAG] |
|
||||
| [TABLE_2] | [FREQUENCY] | [LAG] |
|
||||
|
||||
To check data freshness:
|
||||
```sql
|
||||
SELECT MAX([DATE_COLUMN]) as latest_data FROM [TABLE]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Knowledge Base Navigation
|
||||
|
||||
Use these reference files for detailed table documentation:
|
||||
|
||||
| Domain | Reference File | Use For |
|
||||
|--------|----------------|---------|
|
||||
| [DOMAIN_1] | `references/[domain1].md` | [BRIEF_DESCRIPTION] |
|
||||
| [DOMAIN_2] | `references/[domain2].md` | [BRIEF_DESCRIPTION] |
|
||||
| Entities | `references/entities.md` | Entity definitions and relationships |
|
||||
| Metrics | `references/metrics.md` | KPI calculations and formulas |
|
||||
|
||||
---
|
||||
|
||||
## Common Query Patterns
|
||||
|
||||
### [PATTERN_1_NAME]
|
||||
```sql
|
||||
[SAMPLE_QUERY]
|
||||
```
|
||||
|
||||
### [PATTERN_2_NAME]
|
||||
```sql
|
||||
[SAMPLE_QUERY]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Mistakes
|
||||
- **[MISTAKE_1]**: [EXPLANATION] → [CORRECT_APPROACH]
|
||||
- **[MISTAKE_2]**: [EXPLANATION] → [CORRECT_APPROACH]
|
||||
|
||||
### Access Issues
|
||||
- If you encounter permission errors on `[TABLE]`: [WORKAROUND]
|
||||
- For PII-restricted columns: [ALTERNATIVE_APPROACH]
|
||||
|
||||
### Performance Tips
|
||||
- Filter by `[PARTITION_COLUMN]` first to reduce data scanned
|
||||
- For large tables, use `LIMIT` during exploration
|
||||
- Prefer `[AGGREGATED_TABLE]` over `[RAW_TABLE]` when possible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Customization Notes
|
||||
|
||||
When generating a skill:
|
||||
|
||||
1. **Fill all placeholders** - Don't leave any `[PLACEHOLDER]` text
|
||||
2. **Remove unused sections** - If they don't have dashboards, remove that section
|
||||
3. **Add specificity** - Generic advice is less useful than specific column names and values
|
||||
4. **Include real examples** - Sample queries should use actual table/column names
|
||||
5. **Keep it scannable** - Use tables and code blocks liberally
|
||||
@@ -0,0 +1,121 @@
|
||||
# SQL Dialect Reference
|
||||
|
||||
Include the appropriate section in generated skills based on the user's data warehouse.
|
||||
|
||||
---
|
||||
|
||||
## BigQuery
|
||||
|
||||
```markdown
|
||||
## SQL Dialect: BigQuery
|
||||
|
||||
- **Table references**: Use backticks: \`project.dataset.table\`
|
||||
- **Safe division**: `SAFE_DIVIDE(a, b)` returns NULL instead of error
|
||||
- **Date functions**:
|
||||
- `DATE_TRUNC(date_col, MONTH)`
|
||||
- `DATE_SUB(date_col, INTERVAL 1 DAY)`
|
||||
- `DATE_DIFF(end_date, start_date, DAY)`
|
||||
- **Column exclusion**: `SELECT * EXCEPT(column_to_exclude)`
|
||||
- **Arrays**: `UNNEST(array_column)` to flatten
|
||||
- **Structs**: Access with dot notation `struct_col.field_name`
|
||||
- **Timestamps**: `TIMESTAMP_TRUNC()`, times in UTC by default
|
||||
- **String matching**: `LIKE`, `REGEXP_CONTAINS(col, r'pattern')`
|
||||
- **NULLs in aggregations**: Most functions ignore NULLs; use `IFNULL()` or `COALESCE()`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Snowflake
|
||||
|
||||
```markdown
|
||||
## SQL Dialect: Snowflake
|
||||
|
||||
- **Table references**: `DATABASE.SCHEMA.TABLE` or with quotes for case-sensitive: `"Column_Name"`
|
||||
- **Safe division**: `DIV0(a, b)` returns 0, `DIV0NULL(a, b)` returns NULL
|
||||
- **Date functions**:
|
||||
- `DATE_TRUNC('MONTH', date_col)`
|
||||
- `DATEADD(DAY, -1, date_col)`
|
||||
- `DATEDIFF(DAY, start_date, end_date)`
|
||||
- **Column exclusion**: `SELECT * EXCLUDE (column_to_exclude)`
|
||||
- **Arrays**: `FLATTEN(array_column)` to flatten, access with `value`
|
||||
- **Variants/JSON**: Access with colon notation `variant_col:field_name`
|
||||
- **Timestamps**: `TIMESTAMP_NTZ` (no timezone), `TIMESTAMP_TZ` (with timezone)
|
||||
- **String matching**: `LIKE`, `REGEXP_LIKE(col, 'pattern')`
|
||||
- **Case sensitivity**: Identifiers are uppercase by default unless quoted
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL / Redshift
|
||||
|
||||
```markdown
|
||||
## SQL Dialect: PostgreSQL/Redshift
|
||||
|
||||
- **Table references**: `schema.table` (lowercase convention)
|
||||
- **Safe division**: `NULLIF(b, 0)` pattern: `a / NULLIF(b, 0)`
|
||||
- **Date functions**:
|
||||
- `DATE_TRUNC('month', date_col)`
|
||||
- `date_col - INTERVAL '1 day'`
|
||||
- `DATE_PART('day', end_date - start_date)`
|
||||
- **Column selection**: No EXCEPT; must list columns explicitly
|
||||
- **Arrays**: `UNNEST(array_column)` (PostgreSQL), limited in Redshift
|
||||
- **JSON**: `json_col->>'field_name'` for text, `json_col->'field_name'` for JSON
|
||||
- **Timestamps**: `AT TIME ZONE 'UTC'` for timezone conversion
|
||||
- **String matching**: `LIKE`, `col ~ 'pattern'` for regex
|
||||
- **Boolean**: Native BOOLEAN type; use `TRUE`/`FALSE`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Databricks / Spark SQL
|
||||
|
||||
```markdown
|
||||
## SQL Dialect: Databricks/Spark SQL
|
||||
|
||||
- **Table references**: `catalog.schema.table` (Unity Catalog) or `schema.table`
|
||||
- **Safe division**: Use `NULLIF`: `a / NULLIF(b, 0)` or `TRY_DIVIDE(a, b)`
|
||||
- **Date functions**:
|
||||
- `DATE_TRUNC('MONTH', date_col)`
|
||||
- `DATE_SUB(date_col, 1)`
|
||||
- `DATEDIFF(end_date, start_date)`
|
||||
- **Column exclusion**: `SELECT * EXCEPT (column_to_exclude)` (Databricks SQL)
|
||||
- **Arrays**: `EXPLODE(array_column)` to flatten
|
||||
- **Structs**: Access with dot notation `struct_col.field_name`
|
||||
- **JSON**: `json_col:field_name` or `GET_JSON_OBJECT()`
|
||||
- **String matching**: `LIKE`, `RLIKE` for regex
|
||||
- **Delta features**: `DESCRIBE HISTORY`, time travel with `VERSION AS OF`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MySQL
|
||||
|
||||
```markdown
|
||||
## SQL Dialect: MySQL
|
||||
|
||||
- **Table references**: \`database\`.\`table\` with backticks
|
||||
- **Safe division**: Manual: `IF(b = 0, NULL, a / b)` or `a / NULLIF(b, 0)`
|
||||
- **Date functions**:
|
||||
- `DATE_FORMAT(date_col, '%Y-%m-01')` for truncation
|
||||
- `DATE_SUB(date_col, INTERVAL 1 DAY)`
|
||||
- `DATEDIFF(end_date, start_date)`
|
||||
- **Column selection**: No EXCEPT; must list columns explicitly
|
||||
- **Arrays**: Limited native support; often stored as JSON
|
||||
- **JSON**: `JSON_EXTRACT(col, '$.field')` or `col->>'$.field'`
|
||||
- **Timestamps**: `CONVERT_TZ()` for timezone conversion
|
||||
- **String matching**: `LIKE`, `REGEXP` for regex
|
||||
- **Case sensitivity**: Table names case-sensitive on Linux, not on Windows
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns Across Dialects
|
||||
|
||||
| Operation | BigQuery | Snowflake | PostgreSQL | Databricks |
|
||||
|-----------|----------|-----------|------------|------------|
|
||||
| Current date | `CURRENT_DATE()` | `CURRENT_DATE()` | `CURRENT_DATE` | `CURRENT_DATE()` |
|
||||
| Current timestamp | `CURRENT_TIMESTAMP()` | `CURRENT_TIMESTAMP()` | `NOW()` | `CURRENT_TIMESTAMP()` |
|
||||
| String concat | `CONCAT()` or `\|\|` | `CONCAT()` or `\|\|` | `CONCAT()` or `\|\|` | `CONCAT()` or `\|\|` |
|
||||
| Coalesce | `COALESCE()` | `COALESCE()` | `COALESCE()` | `COALESCE()` |
|
||||
| Case when | `CASE WHEN` | `CASE WHEN` | `CASE WHEN` | `CASE WHEN` |
|
||||
| Count distinct | `COUNT(DISTINCT x)` | `COUNT(DISTINCT x)` | `COUNT(DISTINCT x)` | `COUNT(DISTINCT x)` |
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Package a generated data analysis skill into a distributable .skill file (zip format).
|
||||
|
||||
Usage:
|
||||
python package_data_skill.py <path/to/skill-folder> [output-directory]
|
||||
|
||||
Example:
|
||||
python package_data_skill.py /home/claude/acme-data-analyst
|
||||
python package_data_skill.py /home/claude/acme-data-analyst /tmp/outputs
|
||||
"""
|
||||
|
||||
import sys
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def validate_skill(skill_path: Path) -> tuple[bool, str]:
|
||||
"""Basic validation of skill structure."""
|
||||
|
||||
# Check SKILL.md exists
|
||||
skill_md = skill_path / "SKILL.md"
|
||||
if not skill_md.exists():
|
||||
return False, "Missing SKILL.md"
|
||||
|
||||
# Check SKILL.md has frontmatter
|
||||
content = skill_md.read_text()
|
||||
if not content.startswith("---"):
|
||||
return False, "SKILL.md missing YAML frontmatter"
|
||||
|
||||
# Check for required frontmatter fields
|
||||
if "name:" not in content[:500]:
|
||||
return False, "SKILL.md missing 'name' in frontmatter"
|
||||
if "description:" not in content[:1000]:
|
||||
return False, "SKILL.md missing 'description' in frontmatter"
|
||||
|
||||
# Check for placeholder text that wasn't filled in
|
||||
if "[PLACEHOLDER]" in content or "[COMPANY]" in content:
|
||||
return False, "SKILL.md contains unfilled placeholder text"
|
||||
|
||||
return True, "Validation passed"
|
||||
|
||||
|
||||
def package_skill(skill_path: str, output_dir: str = None) -> Path | None:
|
||||
"""
|
||||
Package a skill folder into a .skill file.
|
||||
|
||||
Args:
|
||||
skill_path: Path to the skill folder
|
||||
output_dir: Optional output directory
|
||||
|
||||
Returns:
|
||||
Path to the created .skill file, or None if error
|
||||
"""
|
||||
skill_path = Path(skill_path).resolve()
|
||||
|
||||
# Validate folder exists
|
||||
if not skill_path.exists():
|
||||
print(f"Error: Skill folder not found: {skill_path}")
|
||||
return None
|
||||
|
||||
if not skill_path.is_dir():
|
||||
print(f"Error: Path is not a directory: {skill_path}")
|
||||
return None
|
||||
|
||||
# Run validation
|
||||
print("Validating skill...")
|
||||
valid, message = validate_skill(skill_path)
|
||||
if not valid:
|
||||
print(f"Validation failed: {message}")
|
||||
return None
|
||||
print(f"{message}\n")
|
||||
|
||||
# Determine output location
|
||||
skill_name = skill_path.name
|
||||
if output_dir:
|
||||
output_path = Path(output_dir).resolve()
|
||||
else:
|
||||
output_path = Path.cwd()
|
||||
|
||||
output_path.mkdir(parents=True, exist_ok=True)
|
||||
skill_filename = output_path / f"{skill_name}.zip"
|
||||
|
||||
# Create the zip file
|
||||
try:
|
||||
with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf:
|
||||
for file_path in skill_path.rglob('*'):
|
||||
if file_path.is_file():
|
||||
# Skip hidden files and common junk
|
||||
if any(part.startswith('.') for part in file_path.parts):
|
||||
continue
|
||||
if file_path.name in ['__pycache__', '.DS_Store', 'Thumbs.db']:
|
||||
continue
|
||||
|
||||
# Calculate relative path within the zip
|
||||
arcname = file_path.relative_to(skill_path.parent)
|
||||
zipf.write(file_path, arcname)
|
||||
print(f" Added: {arcname}")
|
||||
|
||||
print(f"\nSuccessfully packaged skill to: {skill_filename}")
|
||||
return skill_filename
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error creating zip file: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print(__doc__)
|
||||
sys.exit(1)
|
||||
|
||||
skill_path = sys.argv[1]
|
||||
output_dir = sys.argv[2] if len(sys.argv) > 2 else None
|
||||
|
||||
print(f"Packaging skill: {skill_path}")
|
||||
if output_dir:
|
||||
print(f" Output directory: {output_dir}")
|
||||
print()
|
||||
|
||||
result = package_skill(skill_path, output_dir)
|
||||
sys.exit(0 if result else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,305 @@
|
||||
---
|
||||
name: data-visualization
|
||||
description: Create effective data visualizations with Python (matplotlib, seaborn, plotly). Use when building charts, choosing the right chart type for a dataset, creating publication-quality figures, or applying design principles like accessibility and color theory.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# Data Visualization Skill
|
||||
|
||||
Chart selection guidance, Python visualization code patterns, design principles, and accessibility considerations for creating effective data visualizations.
|
||||
|
||||
## Chart Selection Guide
|
||||
|
||||
### Choose by Data Relationship
|
||||
|
||||
| What You're Showing | Best Chart | Alternatives |
|
||||
|---|---|---|
|
||||
| **Trend over time** | Line chart | Area chart (if showing cumulative or composition) |
|
||||
| **Comparison across categories** | Vertical bar chart | Horizontal bar (many categories), lollipop chart |
|
||||
| **Ranking** | Horizontal bar chart | Dot plot, slope chart (comparing two periods) |
|
||||
| **Part-to-whole composition** | Stacked bar chart | Treemap (hierarchical), waffle chart |
|
||||
| **Composition over time** | Stacked area chart | 100% stacked bar (for proportion focus) |
|
||||
| **Distribution** | Histogram | Box plot (comparing groups), violin plot, strip plot |
|
||||
| **Correlation (2 variables)** | Scatter plot | Bubble chart (add 3rd variable as size) |
|
||||
| **Correlation (many variables)** | Heatmap (correlation matrix) | Pair plot |
|
||||
| **Geographic patterns** | Choropleth map | Bubble map, hex map |
|
||||
| **Flow / process** | Sankey diagram | Funnel chart (sequential stages) |
|
||||
| **Relationship network** | Network graph | Chord diagram |
|
||||
| **Performance vs. target** | Bullet chart | Gauge (single KPI only) |
|
||||
| **Multiple KPIs at once** | Small multiples | Dashboard with separate charts |
|
||||
|
||||
### When NOT to Use Certain Charts
|
||||
|
||||
- **Pie charts**: Avoid unless <6 categories and exact proportions matter less than rough comparison. Humans are bad at comparing angles. Use bar charts instead.
|
||||
- **3D charts**: Never. They distort perception and add no information.
|
||||
- **Dual-axis charts**: Use cautiously. They can mislead by implying correlation. Clearly label both axes if used.
|
||||
- **Stacked bar (many categories)**: Hard to compare middle segments. Use small multiples or grouped bars instead.
|
||||
- **Donut charts**: Slightly better than pie charts but same fundamental issues. Use for single KPI display at most.
|
||||
|
||||
## Python Visualization Code Patterns
|
||||
|
||||
### Setup and Style
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.ticker as mticker
|
||||
import seaborn as sns
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
|
||||
# Professional style setup
|
||||
plt.style.use('seaborn-v0_8-whitegrid')
|
||||
plt.rcParams.update({
|
||||
'figure.figsize': (10, 6),
|
||||
'figure.dpi': 150,
|
||||
'font.size': 11,
|
||||
'axes.titlesize': 14,
|
||||
'axes.titleweight': 'bold',
|
||||
'axes.labelsize': 11,
|
||||
'xtick.labelsize': 10,
|
||||
'ytick.labelsize': 10,
|
||||
'legend.fontsize': 10,
|
||||
'figure.titlesize': 16,
|
||||
})
|
||||
|
||||
# Colorblind-friendly palettes
|
||||
PALETTE_CATEGORICAL = ['#4C72B0', '#DD8452', '#55A868', '#C44E52', '#8172B3', '#937860']
|
||||
PALETTE_SEQUENTIAL = 'YlOrRd'
|
||||
PALETTE_DIVERGING = 'RdBu_r'
|
||||
```
|
||||
|
||||
### Line Chart (Time Series)
|
||||
|
||||
```python
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
|
||||
for label, group in df.groupby('category'):
|
||||
ax.plot(group['date'], group['value'], label=label, linewidth=2)
|
||||
|
||||
ax.set_title('Metric Trend by Category', fontweight='bold')
|
||||
ax.set_xlabel('Date')
|
||||
ax.set_ylabel('Value')
|
||||
ax.legend(loc='upper left', frameon=True)
|
||||
ax.spines['top'].set_visible(False)
|
||||
ax.spines['right'].set_visible(False)
|
||||
|
||||
# Format dates on x-axis
|
||||
fig.autofmt_xdate()
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('trend_chart.png', dpi=150, bbox_inches='tight')
|
||||
```
|
||||
|
||||
### Bar Chart (Comparison)
|
||||
|
||||
```python
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
|
||||
# Sort by value for easy reading
|
||||
df_sorted = df.sort_values('metric', ascending=True)
|
||||
|
||||
bars = ax.barh(df_sorted['category'], df_sorted['metric'], color=PALETTE_CATEGORICAL[0])
|
||||
|
||||
# Add value labels
|
||||
for bar in bars:
|
||||
width = bar.get_width()
|
||||
ax.text(width + 0.5, bar.get_y() + bar.get_height()/2,
|
||||
f'{width:,.0f}', ha='left', va='center', fontsize=10)
|
||||
|
||||
ax.set_title('Metric by Category (Ranked)', fontweight='bold')
|
||||
ax.set_xlabel('Metric Value')
|
||||
ax.spines['top'].set_visible(False)
|
||||
ax.spines['right'].set_visible(False)
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('bar_chart.png', dpi=150, bbox_inches='tight')
|
||||
```
|
||||
|
||||
### Histogram (Distribution)
|
||||
|
||||
```python
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
|
||||
ax.hist(df['value'], bins=30, color=PALETTE_CATEGORICAL[0], edgecolor='white', alpha=0.8)
|
||||
|
||||
# Add mean and median lines
|
||||
mean_val = df['value'].mean()
|
||||
median_val = df['value'].median()
|
||||
ax.axvline(mean_val, color='red', linestyle='--', linewidth=1.5, label=f'Mean: {mean_val:,.1f}')
|
||||
ax.axvline(median_val, color='green', linestyle='--', linewidth=1.5, label=f'Median: {median_val:,.1f}')
|
||||
|
||||
ax.set_title('Distribution of Values', fontweight='bold')
|
||||
ax.set_xlabel('Value')
|
||||
ax.set_ylabel('Frequency')
|
||||
ax.legend()
|
||||
ax.spines['top'].set_visible(False)
|
||||
ax.spines['right'].set_visible(False)
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('histogram.png', dpi=150, bbox_inches='tight')
|
||||
```
|
||||
|
||||
### Heatmap
|
||||
|
||||
```python
|
||||
fig, ax = plt.subplots(figsize=(10, 8))
|
||||
|
||||
# Pivot data for heatmap format
|
||||
pivot = df.pivot_table(index='row_dim', columns='col_dim', values='metric', aggfunc='sum')
|
||||
|
||||
sns.heatmap(pivot, annot=True, fmt=',.0f', cmap='YlOrRd',
|
||||
linewidths=0.5, ax=ax, cbar_kws={'label': 'Metric Value'})
|
||||
|
||||
ax.set_title('Metric by Row Dimension and Column Dimension', fontweight='bold')
|
||||
ax.set_xlabel('Column Dimension')
|
||||
ax.set_ylabel('Row Dimension')
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig('heatmap.png', dpi=150, bbox_inches='tight')
|
||||
```
|
||||
|
||||
### Small Multiples
|
||||
|
||||
```python
|
||||
categories = df['category'].unique()
|
||||
n_cats = len(categories)
|
||||
n_cols = min(3, n_cats)
|
||||
n_rows = (n_cats + n_cols - 1) // n_cols
|
||||
|
||||
fig, axes = plt.subplots(n_rows, n_cols, figsize=(5*n_cols, 4*n_rows), sharex=True, sharey=True)
|
||||
axes = axes.flatten() if n_cats > 1 else [axes]
|
||||
|
||||
for i, cat in enumerate(categories):
|
||||
ax = axes[i]
|
||||
subset = df[df['category'] == cat]
|
||||
ax.plot(subset['date'], subset['value'], color=PALETTE_CATEGORICAL[i % len(PALETTE_CATEGORICAL)])
|
||||
ax.set_title(cat, fontsize=12)
|
||||
ax.spines['top'].set_visible(False)
|
||||
ax.spines['right'].set_visible(False)
|
||||
|
||||
# Hide empty subplots
|
||||
for j in range(i+1, len(axes)):
|
||||
axes[j].set_visible(False)
|
||||
|
||||
fig.suptitle('Trends by Category', fontsize=14, fontweight='bold', y=1.02)
|
||||
plt.tight_layout()
|
||||
plt.savefig('small_multiples.png', dpi=150, bbox_inches='tight')
|
||||
```
|
||||
|
||||
### Number Formatting Helpers
|
||||
|
||||
```python
|
||||
def format_number(val, format_type='number'):
|
||||
"""Format numbers for chart labels."""
|
||||
if format_type == 'currency':
|
||||
if abs(val) >= 1e9:
|
||||
return f'${val/1e9:.1f}B'
|
||||
elif abs(val) >= 1e6:
|
||||
return f'${val/1e6:.1f}M'
|
||||
elif abs(val) >= 1e3:
|
||||
return f'${val/1e3:.1f}K'
|
||||
else:
|
||||
return f'${val:,.0f}'
|
||||
elif format_type == 'percent':
|
||||
return f'{val:.1f}%'
|
||||
elif format_type == 'number':
|
||||
if abs(val) >= 1e9:
|
||||
return f'{val/1e9:.1f}B'
|
||||
elif abs(val) >= 1e6:
|
||||
return f'{val/1e6:.1f}M'
|
||||
elif abs(val) >= 1e3:
|
||||
return f'{val/1e3:.1f}K'
|
||||
else:
|
||||
return f'{val:,.0f}'
|
||||
return str(val)
|
||||
|
||||
# Usage with axis formatter
|
||||
ax.yaxis.set_major_formatter(mticker.FuncFormatter(lambda x, p: format_number(x, 'currency')))
|
||||
```
|
||||
|
||||
### Interactive Charts with Plotly
|
||||
|
||||
```python
|
||||
import plotly.express as px
|
||||
import plotly.graph_objects as go
|
||||
|
||||
# Simple interactive line chart
|
||||
fig = px.line(df, x='date', y='value', color='category',
|
||||
title='Interactive Metric Trend',
|
||||
labels={'value': 'Metric Value', 'date': 'Date'})
|
||||
fig.update_layout(hovermode='x unified')
|
||||
fig.write_html('interactive_chart.html')
|
||||
fig.show()
|
||||
|
||||
# Interactive scatter with hover data
|
||||
fig = px.scatter(df, x='metric_a', y='metric_b', color='category',
|
||||
size='size_metric', hover_data=['name', 'detail_field'],
|
||||
title='Correlation Analysis')
|
||||
fig.show()
|
||||
```
|
||||
|
||||
## Design Principles
|
||||
|
||||
### Color
|
||||
|
||||
- **Use color purposefully**: Color should encode data, not decorate
|
||||
- **Highlight the story**: Use a bright accent color for the key insight; grey everything else
|
||||
- **Sequential data**: Use a single-hue gradient (light to dark) for ordered values
|
||||
- **Diverging data**: Use a two-hue gradient with neutral midpoint for data with a meaningful center
|
||||
- **Categorical data**: Use distinct hues, maximum 6-8 before it gets confusing
|
||||
- **Avoid red/green only**: 8% of men are red-green colorblind. Use blue/orange as primary pair
|
||||
|
||||
### Typography
|
||||
|
||||
- **Title states the insight**: "Revenue grew 23% YoY" beats "Revenue by Month"
|
||||
- **Subtitle adds context**: Date range, filters applied, data source
|
||||
- **Axis labels are readable**: Never rotated 90 degrees if avoidable. Shorten or wrap instead
|
||||
- **Data labels add precision**: Use on key points, not every single bar
|
||||
- **Annotation highlights**: Call out specific points with text annotations
|
||||
|
||||
### Layout
|
||||
|
||||
- **Reduce chart junk**: Remove gridlines, borders, backgrounds that don't carry information
|
||||
- **Sort meaningfully**: Categories sorted by value (not alphabetically) unless there's a natural order (months, stages)
|
||||
- **Appropriate aspect ratio**: Time series wider than tall (3:1 to 2:1); comparisons can be squarer
|
||||
- **White space is good**: Don't cram charts together. Give each visualization room to breathe
|
||||
|
||||
### Accuracy
|
||||
|
||||
- **Bar charts start at zero**: Always. A bar from 95 to 100 exaggerates a 5% difference
|
||||
- **Line charts can have non-zero baselines**: When the range of variation is meaningful
|
||||
- **Consistent scales across panels**: When comparing multiple charts, use the same axis range
|
||||
- **Show uncertainty**: Error bars, confidence intervals, or ranges when data is uncertain
|
||||
- **Label your axes**: Never make the reader guess what the numbers mean
|
||||
|
||||
## Accessibility Considerations
|
||||
|
||||
### Color Blindness
|
||||
|
||||
- Never rely on color alone to distinguish data series
|
||||
- Add pattern fills, different line styles (solid, dashed, dotted), or direct labels
|
||||
- Test with a colorblind simulator (e.g., Coblis, Sim Daltonism)
|
||||
- Use the colorblind-friendly palette: `sns.color_palette("colorblind")`
|
||||
|
||||
### Screen Readers
|
||||
|
||||
- Include alt text describing the chart's key finding
|
||||
- Provide a data table alternative alongside the visualization
|
||||
- Use semantic titles and labels
|
||||
|
||||
### General Accessibility
|
||||
|
||||
- Sufficient contrast between data elements and background
|
||||
- Text size minimum 10pt for labels, 12pt for titles
|
||||
- Avoid conveying information only through spatial position (add labels)
|
||||
- Consider printing: does the chart work in black and white?
|
||||
|
||||
### Accessibility Checklist
|
||||
|
||||
Before sharing a visualization:
|
||||
- [ ] Chart works without color (patterns, labels, or line styles differentiate series)
|
||||
- [ ] Text is readable at standard zoom level
|
||||
- [ ] Title describes the insight, not just the data
|
||||
- [ ] Axes are labeled with units
|
||||
- [ ] Legend is clear and positioned without obscuring data
|
||||
- [ ] Data source and date range are noted
|
||||
@@ -0,0 +1,325 @@
|
||||
---
|
||||
name: explore-data
|
||||
description: Profile and explore a dataset to understand its shape, quality, and patterns. Use when encountering a new table or file, checking null rates and column distributions, spotting data quality issues like duplicates or suspicious values, or deciding which dimensions and metrics to analyze.
|
||||
argument-hint: "<table or file>"
|
||||
---
|
||||
|
||||
# /explore-data - Profile and Explore a Dataset
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Generate a comprehensive data profile for a table or uploaded file. Understand its shape, quality, and patterns before diving into analysis.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/explore-data <table_name or file>
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Access the Data
|
||||
|
||||
**If a data warehouse MCP server is connected:**
|
||||
|
||||
1. Resolve the table name (handle schema prefixes, suggest matches if ambiguous)
|
||||
2. Query table metadata: column names, types, descriptions if available
|
||||
3. Run profiling queries against the live data
|
||||
|
||||
**If a file is provided (CSV, Excel, Parquet, JSON):**
|
||||
|
||||
1. Read the file and load into a working dataset
|
||||
2. Infer column types from the data
|
||||
|
||||
**If neither:**
|
||||
|
||||
1. Ask the user to provide a table name (with their warehouse connected) or upload a file
|
||||
2. If they describe a table schema, provide guidance on what profiling queries to run
|
||||
|
||||
### 2. Understand Structure
|
||||
|
||||
Before analyzing any data, understand its structure:
|
||||
|
||||
**Table-level questions:**
|
||||
- How many rows and columns?
|
||||
- What is the grain (one row per what)?
|
||||
- What is the primary key? Is it unique?
|
||||
- When was the data last updated?
|
||||
- How far back does the data go?
|
||||
|
||||
**Column classification** — categorize each column as one of:
|
||||
- **Identifier**: Unique keys, foreign keys, entity IDs
|
||||
- **Dimension**: Categorical attributes for grouping/filtering (status, type, region, category)
|
||||
- **Metric**: Quantitative values for measurement (revenue, count, duration, score)
|
||||
- **Temporal**: Dates and timestamps (created_at, updated_at, event_date)
|
||||
- **Text**: Free-form text fields (description, notes, name)
|
||||
- **Boolean**: True/false flags
|
||||
- **Structural**: JSON, arrays, nested structures
|
||||
|
||||
### 3. Generate Data Profile
|
||||
|
||||
Run the following profiling checks:
|
||||
|
||||
**Table-level metrics:**
|
||||
- Total row count
|
||||
- Column count and types breakdown
|
||||
- Approximate table size (if available from metadata)
|
||||
- Date range coverage (min/max of date columns)
|
||||
|
||||
**All columns:**
|
||||
- Null count and null rate
|
||||
- Distinct count and cardinality ratio (distinct / total)
|
||||
- Most common values (top 5-10 with frequencies)
|
||||
- Least common values (bottom 5 to spot anomalies)
|
||||
|
||||
**Numeric columns (metrics):**
|
||||
```
|
||||
min, max, mean, median (p50)
|
||||
standard deviation
|
||||
percentiles: p1, p5, p25, p75, p95, p99
|
||||
zero count
|
||||
negative count (if unexpected)
|
||||
```
|
||||
|
||||
**String columns (dimensions, text):**
|
||||
```
|
||||
min length, max length, avg length
|
||||
empty string count
|
||||
pattern analysis (do values follow a format?)
|
||||
case consistency (all upper, all lower, mixed?)
|
||||
leading/trailing whitespace count
|
||||
```
|
||||
|
||||
**Date/timestamp columns:**
|
||||
```
|
||||
min date, max date
|
||||
null dates
|
||||
future dates (if unexpected)
|
||||
distribution by month/week
|
||||
gaps in time series
|
||||
```
|
||||
|
||||
**Boolean columns:**
|
||||
```
|
||||
true count, false count, null count
|
||||
true rate
|
||||
```
|
||||
|
||||
**Present the profile as a clean summary table**, grouped by column type (dimensions, metrics, dates, IDs).
|
||||
|
||||
### 4. Identify Data Quality Issues
|
||||
|
||||
Apply the quality assessment framework below. Flag potential problems:
|
||||
|
||||
- **High null rates**: Columns with >5% nulls (warn), >20% nulls (alert)
|
||||
- **Low cardinality surprises**: Columns that should be high-cardinality but aren't (e.g., a "user_id" with only 50 distinct values)
|
||||
- **High cardinality surprises**: Columns that should be categorical but have too many distinct values
|
||||
- **Suspicious values**: Negative amounts where only positive expected, future dates in historical data, obviously placeholder values (e.g., "N/A", "TBD", "test", "999999")
|
||||
- **Duplicate detection**: Check if there's a natural key and whether it has duplicates
|
||||
- **Distribution skew**: Extremely skewed numeric distributions that could affect averages
|
||||
- **Encoding issues**: Mixed case in categorical fields, trailing whitespace, inconsistent formats
|
||||
|
||||
### 5. Discover Relationships and Patterns
|
||||
|
||||
After profiling individual columns:
|
||||
|
||||
- **Foreign key candidates**: ID columns that might link to other tables
|
||||
- **Hierarchies**: Columns that form natural drill-down paths (country > state > city)
|
||||
- **Correlations**: Numeric columns that move together
|
||||
- **Derived columns**: Columns that appear to be computed from others
|
||||
- **Redundant columns**: Columns with identical or near-identical information
|
||||
|
||||
### 6. Suggest Interesting Dimensions and Metrics
|
||||
|
||||
Based on the column profile, recommend:
|
||||
|
||||
- **Best dimension columns** for slicing data (categorical columns with reasonable cardinality, 3-50 values)
|
||||
- **Key metric columns** for measurement (numeric columns with meaningful distributions)
|
||||
- **Time columns** suitable for trend analysis
|
||||
- **Natural groupings** or hierarchies apparent in the data
|
||||
- **Potential join keys** linking to other tables (ID columns, foreign keys)
|
||||
|
||||
### 7. Recommend Follow-Up Analyses
|
||||
|
||||
Suggest 3-5 specific analyses the user could run next:
|
||||
|
||||
- "Trend analysis on [metric] by [time_column] grouped by [dimension]"
|
||||
- "Distribution deep-dive on [skewed_column] to understand outliers"
|
||||
- "Data quality investigation on [problematic_column]"
|
||||
- "Correlation analysis between [metric_a] and [metric_b]"
|
||||
- "Cohort analysis using [date_column] and [status_column]"
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
## Data Profile: [table_name]
|
||||
|
||||
### Overview
|
||||
- Rows: 2,340,891
|
||||
- Columns: 23 (8 dimensions, 6 metrics, 4 dates, 5 IDs)
|
||||
- Date range: 2021-03-15 to 2024-01-22
|
||||
|
||||
### Column Details
|
||||
[summary table]
|
||||
|
||||
### Data Quality Issues
|
||||
[flagged issues with severity]
|
||||
|
||||
### Recommended Explorations
|
||||
[numbered list of suggested follow-up analyses]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Assessment Framework
|
||||
|
||||
### Completeness Score
|
||||
|
||||
Rate each column:
|
||||
- **Complete** (>99% non-null): Green
|
||||
- **Mostly complete** (95-99%): Yellow -- investigate the nulls
|
||||
- **Incomplete** (80-95%): Orange -- understand why and whether it matters
|
||||
- **Sparse** (<80%): Red -- may not be usable without imputation
|
||||
|
||||
### Consistency Checks
|
||||
|
||||
Look for:
|
||||
- **Value format inconsistency**: Same concept represented differently ("USA", "US", "United States", "us")
|
||||
- **Type inconsistency**: Numbers stored as strings, dates in various formats
|
||||
- **Referential integrity**: Foreign keys that don't match any parent record
|
||||
- **Business rule violations**: Negative quantities, end dates before start dates, percentages > 100
|
||||
- **Cross-column consistency**: Status = "completed" but completed_at is null
|
||||
|
||||
### Accuracy Indicators
|
||||
|
||||
Red flags that suggest accuracy issues:
|
||||
- **Placeholder values**: 0, -1, 999999, "N/A", "TBD", "test", "xxx"
|
||||
- **Default values**: Suspiciously high frequency of a single value
|
||||
- **Stale data**: Updated_at shows no recent changes in an active system
|
||||
- **Impossible values**: Ages > 150, dates in the far future, negative durations
|
||||
- **Round number bias**: All values ending in 0 or 5 (suggests estimation, not measurement)
|
||||
|
||||
### Timeliness Assessment
|
||||
|
||||
- When was the table last updated?
|
||||
- What is the expected update frequency?
|
||||
- Is there a lag between event time and load time?
|
||||
- Are there gaps in the time series?
|
||||
|
||||
## Pattern Discovery Techniques
|
||||
|
||||
### Distribution Analysis
|
||||
|
||||
For numeric columns, characterize the distribution:
|
||||
- **Normal**: Mean and median are close, bell-shaped
|
||||
- **Skewed right**: Long tail of high values (common for revenue, session duration)
|
||||
- **Skewed left**: Long tail of low values (less common)
|
||||
- **Bimodal**: Two peaks (suggests two distinct populations)
|
||||
- **Power law**: Few very large values, many small ones (common for user activity)
|
||||
- **Uniform**: Roughly equal frequency across range (often synthetic or random)
|
||||
|
||||
### Temporal Patterns
|
||||
|
||||
For time series data, look for:
|
||||
- **Trend**: Sustained upward or downward movement
|
||||
- **Seasonality**: Repeating patterns (weekly, monthly, quarterly, annual)
|
||||
- **Day-of-week effects**: Weekday vs. weekend differences
|
||||
- **Holiday effects**: Drops or spikes around known holidays
|
||||
- **Change points**: Sudden shifts in level or trend
|
||||
- **Anomalies**: Individual data points that break the pattern
|
||||
|
||||
### Segmentation Discovery
|
||||
|
||||
Identify natural segments by:
|
||||
- Finding categorical columns with 3-20 distinct values
|
||||
- Comparing metric distributions across segment values
|
||||
- Looking for segments with significantly different behavior
|
||||
- Testing whether segments are homogeneous or contain sub-segments
|
||||
|
||||
### Correlation Exploration
|
||||
|
||||
Between numeric columns:
|
||||
- Compute correlation matrix for all metric pairs
|
||||
- Flag strong correlations (|r| > 0.7) for investigation
|
||||
- Note: Correlation does not imply causation -- flag this explicitly
|
||||
- Check for non-linear relationships (e.g., quadratic, logarithmic)
|
||||
|
||||
## Schema Understanding and Documentation
|
||||
|
||||
### Schema Documentation Template
|
||||
|
||||
When documenting a dataset for team use:
|
||||
|
||||
```markdown
|
||||
## Table: [schema.table_name]
|
||||
|
||||
**Description**: [What this table represents]
|
||||
**Grain**: [One row per...]
|
||||
**Primary Key**: [column(s)]
|
||||
**Row Count**: [approximate, with date]
|
||||
**Update Frequency**: [real-time / hourly / daily / weekly]
|
||||
**Owner**: [team or person responsible]
|
||||
|
||||
### Key Columns
|
||||
|
||||
| Column | Type | Description | Example Values | Notes |
|
||||
|--------|------|-------------|----------------|-------|
|
||||
| user_id | STRING | Unique user identifier | "usr_abc123" | FK to users.id |
|
||||
| event_type | STRING | Type of event | "click", "view", "purchase" | 15 distinct values |
|
||||
| revenue | DECIMAL | Transaction revenue in USD | 29.99, 149.00 | Null for non-purchase events |
|
||||
| created_at | TIMESTAMP | When the event occurred | 2024-01-15 14:23:01 | Partitioned on this column |
|
||||
|
||||
### Relationships
|
||||
- Joins to `users` on `user_id`
|
||||
- Joins to `products` on `product_id`
|
||||
- Parent of `event_details` (1:many on event_id)
|
||||
|
||||
### Known Issues
|
||||
- [List any known data quality issues]
|
||||
- [Note any gotchas for analysts]
|
||||
|
||||
### Common Query Patterns
|
||||
- [Typical use cases for this table]
|
||||
```
|
||||
|
||||
### Schema Exploration Queries
|
||||
|
||||
When connected to a data warehouse, use these patterns to discover schema:
|
||||
|
||||
```sql
|
||||
-- List all tables in a schema (PostgreSQL)
|
||||
SELECT table_name, table_type
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
ORDER BY table_name;
|
||||
|
||||
-- Column details (PostgreSQL)
|
||||
SELECT column_name, data_type, is_nullable, column_default
|
||||
FROM information_schema.columns
|
||||
WHERE table_name = 'my_table'
|
||||
ORDER BY ordinal_position;
|
||||
|
||||
-- Table sizes (PostgreSQL)
|
||||
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
|
||||
FROM pg_catalog.pg_statio_user_tables
|
||||
ORDER BY pg_total_relation_size(relid) DESC;
|
||||
|
||||
-- Row counts for all tables (general pattern)
|
||||
-- Run per-table: SELECT COUNT(*) FROM table_name
|
||||
```
|
||||
|
||||
### Lineage and Dependencies
|
||||
|
||||
When exploring an unfamiliar data environment:
|
||||
|
||||
1. Start with the "output" tables (what reports or dashboards consume)
|
||||
2. Trace upstream: What tables feed into them?
|
||||
3. Identify raw/staging/mart layers
|
||||
4. Map the transformation chain from raw data to analytical tables
|
||||
5. Note where data is enriched, filtered, or aggregated
|
||||
|
||||
## Tips
|
||||
|
||||
- For very large tables (100M+ rows), profiling queries use sampling by default -- mention if you need exact counts
|
||||
- If exploring a new dataset for the first time, this command gives you the lay of the land before writing specific queries
|
||||
- The quality flags are heuristic -- not every flag is a real problem, but each is worth a quick look
|
||||
@@ -0,0 +1,428 @@
|
||||
---
|
||||
name: sql-queries
|
||||
description: Write correct, performant SQL across all major data warehouse dialects (Snowflake, BigQuery, Databricks, PostgreSQL, etc.). Use when writing queries, optimizing slow SQL, translating between dialects, or building complex analytical queries with CTEs, window functions, or aggregations.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# SQL Queries Skill
|
||||
|
||||
Write correct, performant, readable SQL across all major data warehouse dialects.
|
||||
|
||||
## Dialect-Specific Reference
|
||||
|
||||
### PostgreSQL (including Aurora, RDS, Supabase, Neon)
|
||||
|
||||
**Date/time:**
|
||||
```sql
|
||||
-- Current date/time
|
||||
CURRENT_DATE, CURRENT_TIMESTAMP, NOW()
|
||||
|
||||
-- Date arithmetic
|
||||
date_column + INTERVAL '7 days'
|
||||
date_column - INTERVAL '1 month'
|
||||
|
||||
-- Truncate to period
|
||||
DATE_TRUNC('month', created_at)
|
||||
|
||||
-- Extract parts
|
||||
EXTRACT(YEAR FROM created_at)
|
||||
EXTRACT(DOW FROM created_at) -- 0=Sunday
|
||||
|
||||
-- Format
|
||||
TO_CHAR(created_at, 'YYYY-MM-DD')
|
||||
```
|
||||
|
||||
**String functions:**
|
||||
```sql
|
||||
-- Concatenation
|
||||
first_name || ' ' || last_name
|
||||
CONCAT(first_name, ' ', last_name)
|
||||
|
||||
-- Pattern matching
|
||||
column ILIKE '%pattern%' -- case-insensitive
|
||||
column ~ '^regex_pattern$' -- regex
|
||||
|
||||
-- String manipulation
|
||||
LEFT(str, n), RIGHT(str, n)
|
||||
SPLIT_PART(str, delimiter, position)
|
||||
REGEXP_REPLACE(str, pattern, replacement)
|
||||
```
|
||||
|
||||
**Arrays and JSON:**
|
||||
```sql
|
||||
-- JSON access
|
||||
data->>'key' -- text
|
||||
data->'nested'->'key' -- json
|
||||
data#>>'{path,to,key}' -- nested text
|
||||
|
||||
-- Array operations
|
||||
ARRAY_AGG(column)
|
||||
ANY(array_column)
|
||||
array_column @> ARRAY['value']
|
||||
```
|
||||
|
||||
**Performance tips:**
|
||||
- Use `EXPLAIN ANALYZE` to profile queries
|
||||
- Create indexes on frequently filtered/joined columns
|
||||
- Use `EXISTS` over `IN` for correlated subqueries
|
||||
- Partial indexes for common filter conditions
|
||||
- Use connection pooling for concurrent access
|
||||
|
||||
---
|
||||
|
||||
### Snowflake
|
||||
|
||||
**Date/time:**
|
||||
```sql
|
||||
-- Current date/time
|
||||
CURRENT_DATE(), CURRENT_TIMESTAMP(), SYSDATE()
|
||||
|
||||
-- Date arithmetic
|
||||
DATEADD(day, 7, date_column)
|
||||
DATEDIFF(day, start_date, end_date)
|
||||
|
||||
-- Truncate to period
|
||||
DATE_TRUNC('month', created_at)
|
||||
|
||||
-- Extract parts
|
||||
YEAR(created_at), MONTH(created_at), DAY(created_at)
|
||||
DAYOFWEEK(created_at)
|
||||
|
||||
-- Format
|
||||
TO_CHAR(created_at, 'YYYY-MM-DD')
|
||||
```
|
||||
|
||||
**String functions:**
|
||||
```sql
|
||||
-- Case-insensitive by default (depends on collation)
|
||||
column ILIKE '%pattern%'
|
||||
REGEXP_LIKE(column, 'pattern')
|
||||
|
||||
-- Parse JSON
|
||||
column:key::string -- dot notation for VARIANT
|
||||
PARSE_JSON('{"key": "value"}')
|
||||
GET_PATH(variant_col, 'path.to.key')
|
||||
|
||||
-- Flatten arrays/objects
|
||||
SELECT f.value FROM table, LATERAL FLATTEN(input => array_col) f
|
||||
```
|
||||
|
||||
**Semi-structured data:**
|
||||
```sql
|
||||
-- VARIANT type access
|
||||
data:customer:name::STRING
|
||||
data:items[0]:price::NUMBER
|
||||
|
||||
-- Flatten nested structures
|
||||
SELECT
|
||||
t.id,
|
||||
item.value:name::STRING as item_name,
|
||||
item.value:qty::NUMBER as quantity
|
||||
FROM my_table t,
|
||||
LATERAL FLATTEN(input => t.data:items) item
|
||||
```
|
||||
|
||||
**Performance tips:**
|
||||
- Use clustering keys on large tables (not traditional indexes)
|
||||
- Filter on clustering key columns for partition pruning
|
||||
- Set appropriate warehouse size for query complexity
|
||||
- Use `RESULT_SCAN(LAST_QUERY_ID())` to avoid re-running expensive queries
|
||||
- Use transient tables for staging/temp data
|
||||
|
||||
---
|
||||
|
||||
### BigQuery (Google Cloud)
|
||||
|
||||
**Date/time:**
|
||||
```sql
|
||||
-- Current date/time
|
||||
CURRENT_DATE(), CURRENT_TIMESTAMP()
|
||||
|
||||
-- Date arithmetic
|
||||
DATE_ADD(date_column, INTERVAL 7 DAY)
|
||||
DATE_SUB(date_column, INTERVAL 1 MONTH)
|
||||
DATE_DIFF(end_date, start_date, DAY)
|
||||
TIMESTAMP_DIFF(end_ts, start_ts, HOUR)
|
||||
|
||||
-- Truncate to period
|
||||
DATE_TRUNC(created_at, MONTH)
|
||||
TIMESTAMP_TRUNC(created_at, HOUR)
|
||||
|
||||
-- Extract parts
|
||||
EXTRACT(YEAR FROM created_at)
|
||||
EXTRACT(DAYOFWEEK FROM created_at) -- 1=Sunday
|
||||
|
||||
-- Format
|
||||
FORMAT_DATE('%Y-%m-%d', date_column)
|
||||
FORMAT_TIMESTAMP('%Y-%m-%d %H:%M:%S', ts_column)
|
||||
```
|
||||
|
||||
**String functions:**
|
||||
```sql
|
||||
-- No ILIKE, use LOWER()
|
||||
LOWER(column) LIKE '%pattern%'
|
||||
REGEXP_CONTAINS(column, r'pattern')
|
||||
REGEXP_EXTRACT(column, r'pattern')
|
||||
|
||||
-- String manipulation
|
||||
SPLIT(str, delimiter) -- returns ARRAY
|
||||
ARRAY_TO_STRING(array, delimiter)
|
||||
```
|
||||
|
||||
**Arrays and structs:**
|
||||
```sql
|
||||
-- Array operations
|
||||
ARRAY_AGG(column)
|
||||
UNNEST(array_column)
|
||||
ARRAY_LENGTH(array_column)
|
||||
value IN UNNEST(array_column)
|
||||
|
||||
-- Struct access
|
||||
struct_column.field_name
|
||||
```
|
||||
|
||||
**Performance tips:**
|
||||
- Always filter on partition columns (usually date) to reduce bytes scanned
|
||||
- Use clustering for frequently filtered columns within partitions
|
||||
- Use `APPROX_COUNT_DISTINCT()` for large-scale cardinality estimates
|
||||
- Avoid `SELECT *` -- billing is per-byte scanned
|
||||
- Use `DECLARE` and `SET` for parameterized scripts
|
||||
- Preview query cost with dry run before executing large queries
|
||||
|
||||
---
|
||||
|
||||
### Redshift (Amazon)
|
||||
|
||||
**Date/time:**
|
||||
```sql
|
||||
-- Current date/time
|
||||
CURRENT_DATE, GETDATE(), SYSDATE
|
||||
|
||||
-- Date arithmetic
|
||||
DATEADD(day, 7, date_column)
|
||||
DATEDIFF(day, start_date, end_date)
|
||||
|
||||
-- Truncate to period
|
||||
DATE_TRUNC('month', created_at)
|
||||
|
||||
-- Extract parts
|
||||
EXTRACT(YEAR FROM created_at)
|
||||
DATE_PART('dow', created_at)
|
||||
```
|
||||
|
||||
**String functions:**
|
||||
```sql
|
||||
-- Case-insensitive
|
||||
column ILIKE '%pattern%'
|
||||
REGEXP_INSTR(column, 'pattern') > 0
|
||||
|
||||
-- String manipulation
|
||||
SPLIT_PART(str, delimiter, position)
|
||||
LISTAGG(column, ', ') WITHIN GROUP (ORDER BY column)
|
||||
```
|
||||
|
||||
**Performance tips:**
|
||||
- Design distribution keys for collocated joins (DISTKEY)
|
||||
- Use sort keys for frequently filtered columns (SORTKEY)
|
||||
- Use `EXPLAIN` to check query plan
|
||||
- Avoid cross-node data movement (watch for DS_BCAST and DS_DIST)
|
||||
- `ANALYZE` and `VACUUM` regularly
|
||||
- Use late-binding views for schema flexibility
|
||||
|
||||
---
|
||||
|
||||
### Databricks SQL
|
||||
|
||||
**Date/time:**
|
||||
```sql
|
||||
-- Current date/time
|
||||
CURRENT_DATE(), CURRENT_TIMESTAMP()
|
||||
|
||||
-- Date arithmetic
|
||||
DATE_ADD(date_column, 7)
|
||||
DATEDIFF(end_date, start_date)
|
||||
ADD_MONTHS(date_column, 1)
|
||||
|
||||
-- Truncate to period
|
||||
DATE_TRUNC('MONTH', created_at)
|
||||
TRUNC(date_column, 'MM')
|
||||
|
||||
-- Extract parts
|
||||
YEAR(created_at), MONTH(created_at)
|
||||
DAYOFWEEK(created_at)
|
||||
```
|
||||
|
||||
**Delta Lake features:**
|
||||
```sql
|
||||
-- Time travel
|
||||
SELECT * FROM my_table TIMESTAMP AS OF '2024-01-15'
|
||||
SELECT * FROM my_table VERSION AS OF 42
|
||||
|
||||
-- Describe history
|
||||
DESCRIBE HISTORY my_table
|
||||
|
||||
-- Merge (upsert)
|
||||
MERGE INTO target USING source
|
||||
ON target.id = source.id
|
||||
WHEN MATCHED THEN UPDATE SET *
|
||||
WHEN NOT MATCHED THEN INSERT *
|
||||
```
|
||||
|
||||
**Performance tips:**
|
||||
- Use Delta Lake's `OPTIMIZE` and `ZORDER` for query performance
|
||||
- Leverage Photon engine for compute-intensive queries
|
||||
- Use `CACHE TABLE` for frequently accessed datasets
|
||||
- Partition by low-cardinality date columns
|
||||
|
||||
---
|
||||
|
||||
## Common SQL Patterns
|
||||
|
||||
### Window Functions
|
||||
|
||||
```sql
|
||||
-- Ranking
|
||||
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY created_at DESC)
|
||||
RANK() OVER (PARTITION BY category ORDER BY revenue DESC)
|
||||
DENSE_RANK() OVER (ORDER BY score DESC)
|
||||
|
||||
-- Running totals / moving averages
|
||||
SUM(revenue) OVER (ORDER BY date_col ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as running_total
|
||||
AVG(revenue) OVER (ORDER BY date_col ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) as moving_avg_7d
|
||||
|
||||
-- Lag / Lead
|
||||
LAG(value, 1) OVER (PARTITION BY entity ORDER BY date_col) as prev_value
|
||||
LEAD(value, 1) OVER (PARTITION BY entity ORDER BY date_col) as next_value
|
||||
|
||||
-- First / Last value
|
||||
FIRST_VALUE(status) OVER (PARTITION BY user_id ORDER BY created_at ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
|
||||
LAST_VALUE(status) OVER (PARTITION BY user_id ORDER BY created_at ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)
|
||||
|
||||
-- Percent of total
|
||||
revenue / SUM(revenue) OVER () as pct_of_total
|
||||
revenue / SUM(revenue) OVER (PARTITION BY category) as pct_of_category
|
||||
```
|
||||
|
||||
### CTEs for Readability
|
||||
|
||||
```sql
|
||||
WITH
|
||||
-- Step 1: Define the base population
|
||||
base_users AS (
|
||||
SELECT user_id, created_at, plan_type
|
||||
FROM users
|
||||
WHERE created_at >= DATE '2024-01-01'
|
||||
AND status = 'active'
|
||||
),
|
||||
|
||||
-- Step 2: Calculate user-level metrics
|
||||
user_metrics AS (
|
||||
SELECT
|
||||
u.user_id,
|
||||
u.plan_type,
|
||||
COUNT(DISTINCT e.session_id) as session_count,
|
||||
SUM(e.revenue) as total_revenue
|
||||
FROM base_users u
|
||||
LEFT JOIN events e ON u.user_id = e.user_id
|
||||
GROUP BY u.user_id, u.plan_type
|
||||
),
|
||||
|
||||
-- Step 3: Aggregate to summary level
|
||||
summary AS (
|
||||
SELECT
|
||||
plan_type,
|
||||
COUNT(*) as user_count,
|
||||
AVG(session_count) as avg_sessions,
|
||||
SUM(total_revenue) as total_revenue
|
||||
FROM user_metrics
|
||||
GROUP BY plan_type
|
||||
)
|
||||
|
||||
SELECT * FROM summary ORDER BY total_revenue DESC;
|
||||
```
|
||||
|
||||
### Cohort Retention
|
||||
|
||||
```sql
|
||||
WITH cohorts AS (
|
||||
SELECT
|
||||
user_id,
|
||||
DATE_TRUNC('month', first_activity_date) as cohort_month
|
||||
FROM users
|
||||
),
|
||||
activity AS (
|
||||
SELECT
|
||||
user_id,
|
||||
DATE_TRUNC('month', activity_date) as activity_month
|
||||
FROM user_activity
|
||||
)
|
||||
SELECT
|
||||
c.cohort_month,
|
||||
COUNT(DISTINCT c.user_id) as cohort_size,
|
||||
COUNT(DISTINCT CASE
|
||||
WHEN a.activity_month = c.cohort_month THEN a.user_id
|
||||
END) as month_0,
|
||||
COUNT(DISTINCT CASE
|
||||
WHEN a.activity_month = c.cohort_month + INTERVAL '1 month' THEN a.user_id
|
||||
END) as month_1,
|
||||
COUNT(DISTINCT CASE
|
||||
WHEN a.activity_month = c.cohort_month + INTERVAL '3 months' THEN a.user_id
|
||||
END) as month_3
|
||||
FROM cohorts c
|
||||
LEFT JOIN activity a ON c.user_id = a.user_id
|
||||
GROUP BY c.cohort_month
|
||||
ORDER BY c.cohort_month;
|
||||
```
|
||||
|
||||
### Funnel Analysis
|
||||
|
||||
```sql
|
||||
WITH funnel AS (
|
||||
SELECT
|
||||
user_id,
|
||||
MAX(CASE WHEN event = 'page_view' THEN 1 ELSE 0 END) as step_1_view,
|
||||
MAX(CASE WHEN event = 'signup_start' THEN 1 ELSE 0 END) as step_2_start,
|
||||
MAX(CASE WHEN event = 'signup_complete' THEN 1 ELSE 0 END) as step_3_complete,
|
||||
MAX(CASE WHEN event = 'first_purchase' THEN 1 ELSE 0 END) as step_4_purchase
|
||||
FROM events
|
||||
WHERE event_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||
GROUP BY user_id
|
||||
)
|
||||
SELECT
|
||||
COUNT(*) as total_users,
|
||||
SUM(step_1_view) as viewed,
|
||||
SUM(step_2_start) as started_signup,
|
||||
SUM(step_3_complete) as completed_signup,
|
||||
SUM(step_4_purchase) as purchased,
|
||||
ROUND(100.0 * SUM(step_2_start) / NULLIF(SUM(step_1_view), 0), 1) as view_to_start_pct,
|
||||
ROUND(100.0 * SUM(step_3_complete) / NULLIF(SUM(step_2_start), 0), 1) as start_to_complete_pct,
|
||||
ROUND(100.0 * SUM(step_4_purchase) / NULLIF(SUM(step_3_complete), 0), 1) as complete_to_purchase_pct
|
||||
FROM funnel;
|
||||
```
|
||||
|
||||
### Deduplication
|
||||
|
||||
```sql
|
||||
-- Keep the most recent record per key
|
||||
WITH ranked AS (
|
||||
SELECT
|
||||
*,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY entity_id
|
||||
ORDER BY updated_at DESC
|
||||
) as rn
|
||||
FROM source_table
|
||||
)
|
||||
SELECT * FROM ranked WHERE rn = 1;
|
||||
```
|
||||
|
||||
## Error Handling and Debugging
|
||||
|
||||
When a query fails:
|
||||
|
||||
1. **Syntax errors**: Check for dialect-specific syntax (e.g., `ILIKE` not available in BigQuery, `SAFE_DIVIDE` only in BigQuery)
|
||||
2. **Column not found**: Verify column names against schema -- check for typos, case sensitivity (PostgreSQL is case-sensitive for quoted identifiers)
|
||||
3. **Type mismatches**: Cast explicitly when comparing different types (`CAST(col AS DATE)`, `col::DATE`)
|
||||
4. **Division by zero**: Use `NULLIF(denominator, 0)` or dialect-specific safe division
|
||||
5. **Ambiguous columns**: Always qualify column names with table alias in JOINs
|
||||
6. **Group by errors**: All non-aggregated columns must be in GROUP BY (except in BigQuery which allows grouping by alias)
|
||||
@@ -0,0 +1,245 @@
|
||||
---
|
||||
name: statistical-analysis
|
||||
description: Apply statistical methods including descriptive stats, trend analysis, outlier detection, and hypothesis testing. Use when analyzing distributions, testing for significance, detecting anomalies, computing correlations, or interpreting statistical results.
|
||||
user-invocable: false
|
||||
---
|
||||
|
||||
# Statistical Analysis Skill
|
||||
|
||||
Descriptive statistics, trend analysis, outlier detection, hypothesis testing, and guidance on when to be cautious about statistical claims.
|
||||
|
||||
## Descriptive Statistics Methodology
|
||||
|
||||
### Central Tendency
|
||||
|
||||
Choose the right measure of center based on the data:
|
||||
|
||||
| Situation | Use | Why |
|
||||
|---|---|---|
|
||||
| Symmetric distribution, no outliers | Mean | Most efficient estimator |
|
||||
| Skewed distribution | Median | Robust to outliers |
|
||||
| Categorical or ordinal data | Mode | Only option for non-numeric |
|
||||
| Highly skewed with outliers (e.g., revenue per user) | Median + mean | Report both; the gap shows skew |
|
||||
|
||||
**Always report mean and median together for business metrics.** If they diverge significantly, the data is skewed and the mean alone is misleading.
|
||||
|
||||
### Spread and Variability
|
||||
|
||||
- **Standard deviation**: How far values typically fall from the mean. Use with normally distributed data.
|
||||
- **Interquartile range (IQR)**: Distance from p25 to p75. Robust to outliers. Use with skewed data.
|
||||
- **Coefficient of variation (CV)**: StdDev / Mean. Use to compare variability across metrics with different scales.
|
||||
- **Range**: Max minus min. Sensitive to outliers but gives a quick sense of data extent.
|
||||
|
||||
### Percentiles for Business Context
|
||||
|
||||
Report key percentiles to tell a richer story than mean alone:
|
||||
|
||||
```
|
||||
p1: Bottom 1% (floor / minimum typical value)
|
||||
p5: Low end of normal range
|
||||
p25: First quartile
|
||||
p50: Median (typical user)
|
||||
p75: Third quartile
|
||||
p90: Top 10% / power users
|
||||
p95: High end of normal range
|
||||
p99: Top 1% / extreme users
|
||||
```
|
||||
|
||||
**Example narrative**: "The median session duration is 4.2 minutes, but the top 10% of users spend over 22 minutes per session, pulling the mean up to 7.8 minutes."
|
||||
|
||||
### Describing Distributions
|
||||
|
||||
Characterize every numeric distribution you analyze:
|
||||
|
||||
- **Shape**: Normal, right-skewed, left-skewed, bimodal, uniform, heavy-tailed
|
||||
- **Center**: Mean and median (and the gap between them)
|
||||
- **Spread**: Standard deviation or IQR
|
||||
- **Outliers**: How many and how extreme
|
||||
- **Bounds**: Is there a natural floor (zero) or ceiling (100%)?
|
||||
|
||||
## Trend Analysis and Forecasting
|
||||
|
||||
### Identifying Trends
|
||||
|
||||
**Moving averages** to smooth noise:
|
||||
```python
|
||||
# 7-day moving average (good for daily data with weekly seasonality)
|
||||
df['ma_7d'] = df['metric'].rolling(window=7, min_periods=1).mean()
|
||||
|
||||
# 28-day moving average (smooths weekly AND monthly patterns)
|
||||
df['ma_28d'] = df['metric'].rolling(window=28, min_periods=1).mean()
|
||||
```
|
||||
|
||||
**Period-over-period comparison**:
|
||||
- Week-over-week (WoW): Compare to same day last week
|
||||
- Month-over-month (MoM): Compare to same month prior
|
||||
- Year-over-year (YoY): Gold standard for seasonal businesses
|
||||
- Same-day-last-year: Compare specific calendar day
|
||||
|
||||
**Growth rates**:
|
||||
```
|
||||
Simple growth: (current - previous) / previous
|
||||
CAGR: (ending / beginning) ^ (1 / years) - 1
|
||||
Log growth: ln(current / previous) -- better for volatile series
|
||||
```
|
||||
|
||||
### Seasonality Detection
|
||||
|
||||
Check for periodic patterns:
|
||||
1. Plot the raw time series -- visual inspection first
|
||||
2. Compute day-of-week averages: is there a clear weekly pattern?
|
||||
3. Compute month-of-year averages: is there an annual cycle?
|
||||
4. When comparing periods, always use YoY or same-period comparisons to avoid conflating trend with seasonality
|
||||
|
||||
### Forecasting (Simple Methods)
|
||||
|
||||
For business analysts (not data scientists), use straightforward methods:
|
||||
|
||||
- **Naive forecast**: Tomorrow = today. Use as a baseline.
|
||||
- **Seasonal naive**: Tomorrow = same day last week/year.
|
||||
- **Linear trend**: Fit a line to historical data. Only for clearly linear trends.
|
||||
- **Moving average forecast**: Use trailing average as the forecast.
|
||||
|
||||
**Always communicate uncertainty**. Provide a range, not a point estimate:
|
||||
- "We expect 10K-12K signups next month based on the 3-month trend"
|
||||
- NOT "We will get exactly 11,234 signups next month"
|
||||
|
||||
**When to escalate to a data scientist**: Non-linear trends, multiple seasonalities, external factors (marketing spend, holidays), or when forecast accuracy matters for resource allocation.
|
||||
|
||||
## Outlier and Anomaly Detection
|
||||
|
||||
### Statistical Methods
|
||||
|
||||
**Z-score method** (for normally distributed data):
|
||||
```python
|
||||
z_scores = (df['value'] - df['value'].mean()) / df['value'].std()
|
||||
outliers = df[abs(z_scores) > 3] # More than 3 standard deviations
|
||||
```
|
||||
|
||||
**IQR method** (robust to non-normal distributions):
|
||||
```python
|
||||
Q1 = df['value'].quantile(0.25)
|
||||
Q3 = df['value'].quantile(0.75)
|
||||
IQR = Q3 - Q1
|
||||
lower_bound = Q1 - 1.5 * IQR
|
||||
upper_bound = Q3 + 1.5 * IQR
|
||||
outliers = df[(df['value'] < lower_bound) | (df['value'] > upper_bound)]
|
||||
```
|
||||
|
||||
**Percentile method** (simplest):
|
||||
```python
|
||||
outliers = df[(df['value'] < df['value'].quantile(0.01)) |
|
||||
(df['value'] > df['value'].quantile(0.99))]
|
||||
```
|
||||
|
||||
### Handling Outliers
|
||||
|
||||
Do NOT automatically remove outliers. Instead:
|
||||
|
||||
1. **Investigate**: Is this a data error, a genuine extreme value, or a different population?
|
||||
2. **Data errors**: Fix or remove (e.g., negative ages, timestamps in year 1970)
|
||||
3. **Genuine extremes**: Keep them but consider using robust statistics (median instead of mean)
|
||||
4. **Different population**: Segment them out for separate analysis (e.g., enterprise vs. SMB customers)
|
||||
|
||||
**Report what you did**: "We excluded 47 records (0.3%) with transaction amounts >$50K, which represent bulk enterprise orders analyzed separately."
|
||||
|
||||
### Time Series Anomaly Detection
|
||||
|
||||
For detecting unusual values in a time series:
|
||||
|
||||
1. Compute expected value (moving average or same-period-last-year)
|
||||
2. Compute deviation from expected
|
||||
3. Flag deviations beyond a threshold (typically 2-3 standard deviations of the residuals)
|
||||
4. Distinguish between point anomalies (single unusual value) and change points (sustained shift)
|
||||
|
||||
## Hypothesis Testing Basics
|
||||
|
||||
### When to Use
|
||||
|
||||
Use hypothesis testing when you need to determine whether an observed difference is likely real or could be due to random chance. Common scenarios:
|
||||
|
||||
- A/B test results: Is variant B actually better than A?
|
||||
- Before/after comparison: Did the product change actually move the metric?
|
||||
- Segment comparison: Do enterprise customers really have higher retention?
|
||||
|
||||
### The Framework
|
||||
|
||||
1. **Null hypothesis (H0)**: There is no difference (the default assumption)
|
||||
2. **Alternative hypothesis (H1)**: There is a difference
|
||||
3. **Choose significance level (alpha)**: Typically 0.05 (5% chance of false positive)
|
||||
4. **Compute test statistic and p-value**
|
||||
5. **Interpret**: If p < alpha, reject H0 (evidence of a real difference)
|
||||
|
||||
### Common Tests
|
||||
|
||||
| Scenario | Test | When to Use |
|
||||
|---|---|---|
|
||||
| Compare two group means | t-test (independent) | Normal data, two groups |
|
||||
| Compare two group proportions | z-test for proportions | Conversion rates, binary outcomes |
|
||||
| Compare paired measurements | Paired t-test | Before/after on same entities |
|
||||
| Compare 3+ group means | ANOVA | Multiple segments or variants |
|
||||
| Non-normal data, two groups | Mann-Whitney U test | Skewed metrics, ordinal data |
|
||||
| Association between categories | Chi-squared test | Two categorical variables |
|
||||
|
||||
### Practical Significance vs. Statistical Significance
|
||||
|
||||
**Statistical significance** means the difference is unlikely due to chance.
|
||||
|
||||
**Practical significance** means the difference is large enough to matter for business decisions.
|
||||
|
||||
A difference can be statistically significant but practically meaningless (common with large samples). Always report:
|
||||
- **Effect size**: How big is the difference? (e.g., "Variant B improved conversion by 0.3 percentage points")
|
||||
- **Confidence interval**: What's the range of plausible true effects?
|
||||
- **Business impact**: What does this translate to in revenue, users, or other business terms?
|
||||
|
||||
### Sample Size Considerations
|
||||
|
||||
- Small samples produce unreliable results, even with significant p-values
|
||||
- Rule of thumb for proportions: Need at least 30 events per group for basic reliability
|
||||
- For detecting small effects (e.g., 1% conversion rate change), you may need thousands of observations per group
|
||||
- If your sample is small, say so: "With only 200 observations per group, we have limited power to detect effects smaller than X%"
|
||||
|
||||
## When to Be Cautious About Statistical Claims
|
||||
|
||||
### Correlation Is Not Causation
|
||||
|
||||
When you find a correlation, explicitly consider:
|
||||
- **Reverse causation**: Maybe B causes A, not A causes B
|
||||
- **Confounding variables**: Maybe C causes both A and B
|
||||
- **Coincidence**: With enough variables, spurious correlations are inevitable
|
||||
|
||||
**What you can say**: "Users who use feature X have 30% higher retention"
|
||||
**What you cannot say without more evidence**: "Feature X causes 30% higher retention"
|
||||
|
||||
### Multiple Comparisons Problem
|
||||
|
||||
When you test many hypotheses, some will be "significant" by chance:
|
||||
- Testing 20 metrics at p=0.05 means ~1 will be falsely significant
|
||||
- If you looked at many segments before finding one that's different, note that
|
||||
- Adjust for multiple comparisons with Bonferroni correction (divide alpha by number of tests) or report how many tests were run
|
||||
|
||||
### Simpson's Paradox
|
||||
|
||||
A trend in aggregated data can reverse when data is segmented:
|
||||
- Always check whether the conclusion holds across key segments
|
||||
- Example: Overall conversion goes up, but conversion goes down in every segment -- because the mix shifted toward a higher-converting segment
|
||||
|
||||
### Survivorship Bias
|
||||
|
||||
You can only analyze entities that "survived" to be in your dataset:
|
||||
- Analyzing active users ignores those who churned
|
||||
- Analyzing successful companies ignores those that failed
|
||||
- Always ask: "Who is missing from this dataset, and would their inclusion change the conclusion?"
|
||||
|
||||
### Ecological Fallacy
|
||||
|
||||
Aggregate trends may not apply to individuals:
|
||||
- "Countries with higher X have higher Y" does NOT mean "individuals with higher X have higher Y"
|
||||
- Be careful about applying group-level findings to individual cases
|
||||
|
||||
### Anchoring on Specific Numbers
|
||||
|
||||
Be wary of false precision:
|
||||
- "Churn will be 4.73% next quarter" implies more certainty than is warranted
|
||||
- Prefer ranges: "We expect churn between 4-6% based on historical patterns"
|
||||
- Round appropriately: "About 5%" is often more honest than "4.73%"
|
||||
@@ -0,0 +1,383 @@
|
||||
---
|
||||
name: validate-data
|
||||
description: QA an analysis before sharing -- methodology, accuracy, and bias checks. Use when reviewing an analysis before a stakeholder presentation, spot-checking calculations and aggregation logic, verifying a SQL query's results look right, or assessing whether conclusions are actually supported by the data.
|
||||
argument-hint: "<analysis to review>"
|
||||
---
|
||||
|
||||
# /validate-data - Validate Analysis Before Sharing
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Review an analysis for accuracy, methodology, and potential biases before sharing with stakeholders. Generates a confidence assessment and improvement suggestions.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/validate-data <analysis to review>
|
||||
```
|
||||
|
||||
The analysis can be:
|
||||
- A document or report in the conversation
|
||||
- A file (markdown, notebook, spreadsheet)
|
||||
- SQL queries and their results
|
||||
- Charts and their underlying data
|
||||
- A description of methodology and findings
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Review Methodology and Assumptions
|
||||
|
||||
Examine:
|
||||
|
||||
- **Question framing**: Is the analysis answering the right question? Could the question be interpreted differently?
|
||||
- **Data selection**: Are the right tables/datasets being used? Is the time range appropriate?
|
||||
- **Population definition**: Is the analysis population correctly defined? Are there unintended exclusions?
|
||||
- **Metric definitions**: Are metrics defined clearly and consistently? Do they match how stakeholders understand them?
|
||||
- **Baseline and comparison**: Is the comparison fair? Are time periods, cohort sizes, and contexts comparable?
|
||||
|
||||
### 2. Run the Pre-Delivery QA Checklist
|
||||
|
||||
Work through the checklist below — data quality, calculation, reasonableness, and presentation checks.
|
||||
|
||||
### 3. Check for Common Analytical Pitfalls
|
||||
|
||||
Systematically review against the detailed pitfall catalog below (join explosion, survivorship bias, incomplete period comparison, denominator shifting, average of averages, timezone mismatches, selection bias).
|
||||
|
||||
### 4. Verify Calculations and Aggregations
|
||||
|
||||
Where possible, spot-check:
|
||||
|
||||
- Recalculate a few key numbers independently
|
||||
- Verify that subtotals sum to totals
|
||||
- Check that percentages sum to 100% (or close to it) where expected
|
||||
- Confirm that YoY/MoM comparisons use the correct base periods
|
||||
- Validate that filters are applied consistently across all metrics
|
||||
|
||||
Apply the result sanity-checking techniques below (magnitude checks, cross-validation, red-flag detection).
|
||||
|
||||
### 5. Assess Visualizations
|
||||
|
||||
If the analysis includes charts:
|
||||
|
||||
- Do axes start at appropriate values (zero for bar charts)?
|
||||
- Are scales consistent across comparison charts?
|
||||
- Do chart titles accurately describe what's shown?
|
||||
- Could the visualization mislead a quick reader?
|
||||
- Are there truncated axes, inconsistent intervals, or 3D effects that distort perception?
|
||||
|
||||
### 6. Evaluate Narrative and Conclusions
|
||||
|
||||
Review whether:
|
||||
|
||||
- Conclusions are supported by the data shown
|
||||
- Alternative explanations are acknowledged
|
||||
- Uncertainty is communicated appropriately
|
||||
- Recommendations follow logically from findings
|
||||
- The level of confidence matches the strength of evidence
|
||||
|
||||
### 7. Suggest Improvements
|
||||
|
||||
Provide specific, actionable suggestions:
|
||||
|
||||
- Additional analyses that would strengthen the conclusions
|
||||
- Caveats or limitations that should be noted
|
||||
- Better visualizations or framings for key points
|
||||
- Missing context that stakeholders would want
|
||||
|
||||
### 8. Generate Confidence Assessment
|
||||
|
||||
Rate the analysis on a 3-level scale:
|
||||
|
||||
**Ready to share** -- Analysis is methodologically sound, calculations verified, caveats noted. Minor suggestions for improvement but nothing blocking.
|
||||
|
||||
**Share with noted caveats** -- Analysis is largely correct but has specific limitations or assumptions that must be communicated to stakeholders. List the required caveats.
|
||||
|
||||
**Needs revision** -- Found specific errors, methodological issues, or missing analyses that should be addressed before sharing. List the required changes with priority order.
|
||||
|
||||
## Output Format
|
||||
|
||||
```
|
||||
## Validation Report
|
||||
|
||||
### Overall Assessment: [Ready to share | Share with caveats | Needs revision]
|
||||
|
||||
### Methodology Review
|
||||
[Findings about approach, data selection, definitions]
|
||||
|
||||
### Issues Found
|
||||
1. [Severity: High/Medium/Low] [Issue description and impact]
|
||||
2. ...
|
||||
|
||||
### Calculation Spot-Checks
|
||||
- [Metric]: [Verified / Discrepancy found]
|
||||
- ...
|
||||
|
||||
### Visualization Review
|
||||
[Any issues with charts or visual presentation]
|
||||
|
||||
### Suggested Improvements
|
||||
1. [Improvement and why it matters]
|
||||
2. ...
|
||||
|
||||
### Required Caveats for Stakeholders
|
||||
- [Caveat that must be communicated]
|
||||
- ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pre-Delivery QA Checklist
|
||||
|
||||
Run through this checklist before sharing any analysis with stakeholders.
|
||||
|
||||
### Data Quality Checks
|
||||
|
||||
- [ ] **Source verification**: Confirmed which tables/data sources were used. Are they the right ones for this question?
|
||||
- [ ] **Freshness**: Data is current enough for the analysis. Noted the "as of" date.
|
||||
- [ ] **Completeness**: No unexpected gaps in time series or missing segments.
|
||||
- [ ] **Null handling**: Checked null rates in key columns. Nulls are handled appropriately (excluded, imputed, or flagged).
|
||||
- [ ] **Deduplication**: Confirmed no double-counting from bad joins or duplicate source records.
|
||||
- [ ] **Filter verification**: All WHERE clauses and filters are correct. No unintended exclusions.
|
||||
|
||||
### Calculation Checks
|
||||
|
||||
- [ ] **Aggregation logic**: GROUP BY includes all non-aggregated columns. Aggregation level matches the analysis grain.
|
||||
- [ ] **Denominator correctness**: Rate and percentage calculations use the right denominator. Denominators are non-zero.
|
||||
- [ ] **Date alignment**: Comparisons use the same time period length. Partial periods are excluded or noted.
|
||||
- [ ] **Join correctness**: JOIN types are appropriate (INNER vs LEFT). Many-to-many joins haven't inflated counts.
|
||||
- [ ] **Metric definitions**: Metrics match how stakeholders define them. Any deviations are noted.
|
||||
- [ ] **Subtotals sum**: Parts add up to the whole where expected. If they don't, explain why (e.g., overlap).
|
||||
|
||||
### Reasonableness Checks
|
||||
|
||||
- [ ] **Magnitude**: Numbers are in a plausible range. Revenue isn't negative. Percentages are between 0-100%.
|
||||
- [ ] **Trend continuity**: No unexplained jumps or drops in time series.
|
||||
- [ ] **Cross-reference**: Key numbers match other known sources (dashboards, previous reports, finance data).
|
||||
- [ ] **Order of magnitude**: Total revenue is in the right ballpark. User counts match known figures.
|
||||
- [ ] **Edge cases**: What happens at the boundaries? Empty segments, zero-activity periods, new entities.
|
||||
|
||||
### Presentation Checks
|
||||
|
||||
- [ ] **Chart accuracy**: Bar charts start at zero. Axes are labeled. Scales are consistent across panels.
|
||||
- [ ] **Number formatting**: Appropriate precision. Consistent currency/percentage formatting. Thousands separators where needed.
|
||||
- [ ] **Title clarity**: Titles state the insight, not just the metric. Date ranges are specified.
|
||||
- [ ] **Caveat transparency**: Known limitations and assumptions are stated explicitly.
|
||||
- [ ] **Reproducibility**: Someone else could recreate this analysis from the documentation provided.
|
||||
|
||||
## Common Data Analysis Pitfalls
|
||||
|
||||
### Join Explosion
|
||||
|
||||
**The problem**: A many-to-many join silently multiplies rows, inflating counts and sums.
|
||||
|
||||
**How to detect**:
|
||||
```sql
|
||||
-- Check row count before and after join
|
||||
SELECT COUNT(*) FROM table_a; -- 1,000
|
||||
SELECT COUNT(*) FROM table_a a JOIN table_b b ON a.id = b.a_id; -- 3,500 (uh oh)
|
||||
```
|
||||
|
||||
**How to prevent**:
|
||||
- Always check row counts after joins
|
||||
- If counts increase, investigate the join relationship (is it really 1:1 or 1:many?)
|
||||
- Use `COUNT(DISTINCT a.id)` instead of `COUNT(*)` when counting entities through joins
|
||||
|
||||
### Survivorship Bias
|
||||
|
||||
**The problem**: Analyzing only entities that exist today, ignoring those that were deleted, churned, or failed.
|
||||
|
||||
**Examples**:
|
||||
- Analyzing user behavior of "current users" misses churned users
|
||||
- Looking at "companies using our product" ignores those who evaluated and left
|
||||
- Studying properties of "successful" outcomes without "unsuccessful" ones
|
||||
|
||||
**How to prevent**: Ask "who is NOT in this dataset?" before drawing conclusions.
|
||||
|
||||
### Incomplete Period Comparison
|
||||
|
||||
**The problem**: Comparing a partial period to a full period.
|
||||
|
||||
**Examples**:
|
||||
- "January revenue is $500K vs. December's $800K" -- but January isn't over yet
|
||||
- "This week's signups are down" -- checked on Wednesday, comparing to a full prior week
|
||||
|
||||
**How to prevent**: Always filter to complete periods, or compare same-day-of-month / same-number-of-days.
|
||||
|
||||
### Denominator Shifting
|
||||
|
||||
**The problem**: The denominator changes between periods, making rates incomparable.
|
||||
|
||||
**Examples**:
|
||||
- Conversion rate improves because you changed how you count "eligible" users
|
||||
- Churn rate changes because the definition of "active" was updated
|
||||
|
||||
**How to prevent**: Use consistent definitions across all compared periods. Note any definition changes.
|
||||
|
||||
### Average of Averages
|
||||
|
||||
**The problem**: Averaging pre-computed averages gives wrong results when group sizes differ.
|
||||
|
||||
**Example**:
|
||||
- Group A: 100 users, average revenue $50
|
||||
- Group B: 10 users, average revenue $200
|
||||
- Wrong: Average of averages = ($50 + $200) / 2 = $125
|
||||
- Right: Weighted average = (100*$50 + 10*$200) / 110 = $63.64
|
||||
|
||||
**How to prevent**: Always aggregate from raw data. Never average pre-aggregated averages.
|
||||
|
||||
### Timezone Mismatches
|
||||
|
||||
**The problem**: Different data sources use different timezones, causing misalignment.
|
||||
|
||||
**Examples**:
|
||||
- Event timestamps in UTC vs. user-facing dates in local time
|
||||
- Daily rollups that use different cutoff times
|
||||
|
||||
**How to prevent**: Standardize all timestamps to a single timezone (UTC recommended) before analysis. Document the timezone used.
|
||||
|
||||
### Selection Bias in Segmentation
|
||||
|
||||
**The problem**: Segments are defined by the outcome you're measuring, creating circular logic.
|
||||
|
||||
**Examples**:
|
||||
- "Users who completed onboarding have higher retention" -- obviously, they self-selected
|
||||
- "Power users generate more revenue" -- they became power users BY generating revenue
|
||||
|
||||
**How to prevent**: Define segments based on pre-treatment characteristics, not outcomes.
|
||||
|
||||
### Other Statistical Traps
|
||||
|
||||
- **Simpson's paradox**: Trend reverses when data is aggregated vs. segmented
|
||||
- **Correlation presented as causation** without supporting evidence
|
||||
- **Small sample sizes** leading to unreliable conclusions
|
||||
- **Outliers disproportionately affecting averages** (should medians be used instead?)
|
||||
- **Multiple testing / cherry-picking** significant results
|
||||
- **Look-ahead bias**: Using future information to explain past events
|
||||
- **Cherry-picked time ranges** that favor a particular narrative
|
||||
|
||||
## Result Sanity Checking
|
||||
|
||||
### Magnitude Checks
|
||||
|
||||
For any key number in your analysis, verify it passes the "smell test":
|
||||
|
||||
| Metric Type | Sanity Check |
|
||||
|---|---|
|
||||
| User counts | Does this match known MAU/DAU figures? |
|
||||
| Revenue | Is this in the right order of magnitude vs. known ARR? |
|
||||
| Conversion rates | Is this between 0% and 100%? Does it match dashboard figures? |
|
||||
| Growth rates | Is 50%+ MoM growth realistic, or is there a data issue? |
|
||||
| Averages | Is the average reasonable given what you know about the distribution? |
|
||||
| Percentages | Do segment percentages sum to ~100%? |
|
||||
|
||||
### Cross-Validation Techniques
|
||||
|
||||
1. **Calculate the same metric two different ways** and verify they match
|
||||
2. **Spot-check individual records** -- pick a few specific entities and trace their data manually
|
||||
3. **Compare to known benchmarks** -- match against published dashboards, finance reports, or prior analyses
|
||||
4. **Reverse engineer** -- if total revenue is X, does per-user revenue times user count approximately equal X?
|
||||
5. **Boundary checks** -- what happens when you filter to a single day, a single user, or a single category? Are those micro-results sensible?
|
||||
|
||||
### Red Flags That Warrant Investigation
|
||||
|
||||
- Any metric that changed by more than 50% period-over-period without an obvious cause
|
||||
- Counts or sums that are exact round numbers (suggests a filter or default value issue)
|
||||
- Rates exactly at 0% or 100% (may indicate incomplete data)
|
||||
- Results that perfectly confirm the hypothesis (reality is usually messier)
|
||||
- Identical values across time periods or segments (suggests the query is ignoring a dimension)
|
||||
|
||||
## Documentation Standards for Reproducibility
|
||||
|
||||
### Analysis Documentation Template
|
||||
|
||||
Every non-trivial analysis should include:
|
||||
|
||||
```markdown
|
||||
## Analysis: [Title]
|
||||
|
||||
### Question
|
||||
[The specific question being answered]
|
||||
|
||||
### Data Sources
|
||||
- Table: [schema.table_name] (as of [date])
|
||||
- Table: [schema.other_table] (as of [date])
|
||||
- File: [filename] (source: [where it came from])
|
||||
|
||||
### Definitions
|
||||
- [Metric A]: [Exactly how it's calculated]
|
||||
- [Segment X]: [Exactly how membership is determined]
|
||||
- [Time period]: [Start date] to [end date], [timezone]
|
||||
|
||||
### Methodology
|
||||
1. [Step 1 of the analysis approach]
|
||||
2. [Step 2]
|
||||
3. [Step 3]
|
||||
|
||||
### Assumptions and Limitations
|
||||
- [Assumption 1 and why it's reasonable]
|
||||
- [Limitation 1 and its potential impact on conclusions]
|
||||
|
||||
### Key Findings
|
||||
1. [Finding 1 with supporting evidence]
|
||||
2. [Finding 2 with supporting evidence]
|
||||
|
||||
### SQL Queries
|
||||
[All queries used, with comments]
|
||||
|
||||
### Caveats
|
||||
- [Things the reader should know before acting on this]
|
||||
```
|
||||
|
||||
### Code Documentation
|
||||
|
||||
For any code (SQL, Python) that may be reused:
|
||||
|
||||
```python
|
||||
"""
|
||||
Analysis: Monthly Cohort Retention
|
||||
Author: [Name]
|
||||
Date: [Date]
|
||||
Data Source: events table, users table
|
||||
Last Validated: [Date] -- results matched dashboard within 2%
|
||||
|
||||
Purpose:
|
||||
Calculate monthly user retention cohorts based on first activity date.
|
||||
|
||||
Assumptions:
|
||||
- "Active" means at least one event in the month
|
||||
- Excludes test/internal accounts (user_type != 'internal')
|
||||
- Uses UTC dates throughout
|
||||
|
||||
Output:
|
||||
Cohort retention matrix with cohort_month rows and months_since_signup columns.
|
||||
Values are retention rates (0-100%).
|
||||
"""
|
||||
```
|
||||
|
||||
### Version Control for Analyses
|
||||
|
||||
- Save queries and code in version control (git) or a shared docs system
|
||||
- Note the date of the data snapshot used
|
||||
- If an analysis is re-run with updated data, document what changed and why
|
||||
- Link to prior versions of recurring analyses for trend comparison
|
||||
|
||||
## Examples
|
||||
|
||||
```
|
||||
/validate-data Review this quarterly revenue analysis before I send it to the exec team: [analysis]
|
||||
```
|
||||
|
||||
```
|
||||
/validate-data Check my churn analysis -- I'm comparing Q4 churn rates to Q3 but Q4 has a shorter measurement window
|
||||
```
|
||||
|
||||
```
|
||||
/validate-data Here's a SQL query and its results for our conversion funnel. Does the logic look right? [query + results]
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Run /validate-data before any high-stakes presentation or decision
|
||||
- Even quick analyses benefit from a sanity check -- it takes a minute and can save your credibility
|
||||
- If the validation finds issues, fix them and re-validate
|
||||
- Share the validation output alongside your analysis to build stakeholder confidence
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
name: write-query
|
||||
description: Write optimized SQL for your dialect with best practices. Use when translating a natural-language data need into SQL, building a multi-CTE query with joins and aggregations, optimizing a query against a large partitioned table, or getting dialect-specific syntax for Snowflake, BigQuery, Postgres, etc.
|
||||
argument-hint: "<description of what data you need>"
|
||||
---
|
||||
|
||||
# /write-query - Write Optimized SQL
|
||||
|
||||
> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
|
||||
|
||||
Write a SQL query from a natural language description, optimized for your specific SQL dialect and following best practices.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/write-query <description of what data you need>
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the Request
|
||||
|
||||
Parse the user's description to identify:
|
||||
|
||||
- **Output columns**: What fields should the result include?
|
||||
- **Filters**: What conditions limit the data (time ranges, segments, statuses)?
|
||||
- **Aggregations**: Are there GROUP BY operations, counts, sums, averages?
|
||||
- **Joins**: Does this require combining multiple tables?
|
||||
- **Ordering**: How should results be sorted?
|
||||
- **Limits**: Is there a top-N or sample requirement?
|
||||
|
||||
### 2. Determine SQL Dialect
|
||||
|
||||
If the user's SQL dialect is not already known, ask which they use:
|
||||
|
||||
- **PostgreSQL** (including Aurora, RDS, Supabase, Neon)
|
||||
- **Snowflake**
|
||||
- **BigQuery** (Google Cloud)
|
||||
- **Redshift** (Amazon)
|
||||
- **Databricks SQL**
|
||||
- **MySQL** (including Aurora MySQL, PlanetScale)
|
||||
- **SQL Server** (Microsoft)
|
||||
- **DuckDB**
|
||||
- **SQLite**
|
||||
- **Other** (ask for specifics)
|
||||
|
||||
Remember the dialect for future queries in the same session.
|
||||
|
||||
### 3. Discover Schema (If Warehouse Connected)
|
||||
|
||||
If a data warehouse MCP server is connected:
|
||||
|
||||
1. Search for relevant tables based on the user's description
|
||||
2. Inspect column names, types, and relationships
|
||||
3. Check for partitioning or clustering keys that affect performance
|
||||
4. Look for pre-built views or materialized views that might simplify the query
|
||||
|
||||
### 4. Write the Query
|
||||
|
||||
Follow these best practices:
|
||||
|
||||
**Structure:**
|
||||
- Use CTEs (WITH clauses) for readability when queries have multiple logical steps
|
||||
- One CTE per logical transformation or data source
|
||||
- Name CTEs descriptively (e.g., `daily_signups`, `active_users`, `revenue_by_product`)
|
||||
|
||||
**Performance:**
|
||||
- Never use `SELECT *` in production queries -- specify only needed columns
|
||||
- Filter early (push WHERE clauses as close to the base tables as possible)
|
||||
- Use partition filters when available (especially date partitions)
|
||||
- Prefer `EXISTS` over `IN` for subqueries with large result sets
|
||||
- Use appropriate JOIN types (don't use LEFT JOIN when INNER JOIN is correct)
|
||||
- Avoid correlated subqueries when a JOIN or window function works
|
||||
- Be mindful of exploding joins (many-to-many)
|
||||
|
||||
**Readability:**
|
||||
- Add comments explaining the "why" for non-obvious logic
|
||||
- Use consistent indentation and formatting
|
||||
- Alias tables with meaningful short names (not just `a`, `b`, `c`)
|
||||
- Put each major clause on its own line
|
||||
|
||||
**Dialect-specific optimizations:**
|
||||
- Apply dialect-specific syntax and functions (see `sql-queries` skill for details)
|
||||
- Use dialect-appropriate date functions, string functions, and window syntax
|
||||
- Note any dialect-specific performance features (e.g., Snowflake clustering, BigQuery partitioning)
|
||||
|
||||
### 5. Present the Query
|
||||
|
||||
Provide:
|
||||
|
||||
1. **The complete query** in a SQL code block with syntax highlighting
|
||||
2. **Brief explanation** of what each CTE or section does
|
||||
3. **Performance notes** if relevant (expected cost, partition usage, potential bottlenecks)
|
||||
4. **Modification suggestions** -- how to adjust for common variations (different time range, different granularity, additional filters)
|
||||
|
||||
### 6. Offer to Execute
|
||||
|
||||
If a data warehouse is connected, offer to run the query and analyze the results. If the user wants to run it themselves, the query is ready to copy-paste.
|
||||
|
||||
## Examples
|
||||
|
||||
**Simple aggregation:**
|
||||
```
|
||||
/write-query Count of orders by status for the last 30 days
|
||||
```
|
||||
|
||||
**Complex analysis:**
|
||||
```
|
||||
/write-query Cohort retention analysis -- group users by their signup month, then show what percentage are still active (had at least one event) at 1, 3, 6, and 12 months after signup
|
||||
```
|
||||
|
||||
**Performance-critical:**
|
||||
```
|
||||
/write-query We have a 500M row events table partitioned by date. Find the top 100 users by event count in the last 7 days with their most recent event type.
|
||||
```
|
||||
|
||||
## Tips
|
||||
|
||||
- Mention your SQL dialect upfront to get the right syntax immediately
|
||||
- If you know the table names, include them -- otherwise Claude will help you find them
|
||||
- Specify if you need the query to be idempotent (safe to re-run) or one-time
|
||||
- For recurring queries, mention if it should be parameterized for date ranges
|
||||
Reference in New Issue
Block a user