chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,75 @@
|
||||
|
||||
O**Prompt for AI Coding Assistant: Create an Interactive LLM Context Builder Page**
|
||||
|
||||
**Objective:**
|
||||
|
||||
Your task is to create an interactive HTML webpage with JavaScript functionality that allows users to select and combine different `crawl4ai` LLM context files into a single downloadable Markdown (`.md`) file. This tool will empower users to craft tailored context for their AI assistants based on their specific needs.
|
||||
|
||||
**Core Functionality:**
|
||||
|
||||
1. **Display `crawl4ai` Components:** The page will list all available `crawl4ai` documentation components.
|
||||
2. **Select Context Types:** For each component, users can select which types of context they want to include:
|
||||
* Memory (API facts)
|
||||
* Reasoning (How-to/why)
|
||||
* Examples (Code snippets)
|
||||
(All should be selected by default for each initially selected component).
|
||||
3. **Special "Aggregate" Contexts:** Include options for special, pre-combined contexts:
|
||||
* "Vibe Coding" (a curated mix for general AI prompting)
|
||||
* "All Library Context" (a comprehensive aggregation of all memory, reasoning, and examples for the entire library).
|
||||
4. **Fetch and Concatenate:** When the user clicks a "Download Combined Context" button:
|
||||
* The JavaScript will fetch the content of all selected Markdown files from the server (from a predefined folder, e.g., `/llmtxt/`).
|
||||
* It will concatenate the content of these files into a single string.
|
||||
5. **Client-Side Download:** The concatenated content will be offered to the user as a download (e.g., `custom_crawl4ai_context.md`).
|
||||
|
||||
**Input/Assumptions:**
|
||||
|
||||
* **Context Files Location:** All individual context Markdown files are located on the server in a publicly accessible folder named `llmtxt/`.
|
||||
* **File Naming Convention:** Files follow the pattern: `crawl4ai_{{component_name}}_[memory|reasoning|examples]_content.llm.md`.
|
||||
* `{{component_name}}` can contain underscores (e.g., `deep_crawling`, `config_objects`).
|
||||
* The special contexts will have names like `crawl4ai_vibe_content.llm.md` and `crawl4ai_all_content.llm.md`.
|
||||
* **Component List:** You will be provided with a list of `crawl4ai` components. For this implementation, use the following list:
|
||||
* `core`
|
||||
* `config_objects`
|
||||
* `deep_crawling`
|
||||
* `deployment` (covers Installation & Docker Deployment)
|
||||
* `extraction` (covers Structured Data Extraction)
|
||||
* `markdown` (covers Markdown Generation Algorithm)
|
||||
* `pdf_processing`
|
||||
* *(No separate "Vibe Coding" or "All Library Context" in this list, as they are special top-level selections)*
|
||||
|
||||
**Detailed UI/UX Requirements:**
|
||||
|
||||
1. **Main Page Structure:**
|
||||
* **Header:** "Crawl4AI Interactive LLM Context Builder"
|
||||
* **Introduction:** Briefly explain the purpose of the tool (from the `USING_LLM_CONTEXTS.md` content you helped draft: "Supercharging Your AI Assistant...").
|
||||
* **Selection Area:**
|
||||
* **Special Aggregate Contexts (Radio Buttons or Prominent Checkboxes):**
|
||||
* [ ] "Vibe Coding Context" (`crawl4ai_vibe_content.llm.md`)
|
||||
* [ ] "All Library Context (Comprehensive)" (`crawl4ai_all_content.llm.md`)
|
||||
* *Behavior:* Selecting one of these might disable individual component selections (or vice-versa) to avoid redundancy, or simply add them to the list. Consider user experience here. A simple approach is that if an aggregate is selected, it's the *only* thing downloaded.
|
||||
* **Individual Component Selection (Table or List of Checkboxes):**
|
||||
* A section titled "Select Individual Components & Context Types:"
|
||||
* For each component in the provided list:
|
||||
* A master checkbox for the component itself (e.g., `[ ] Core Functionality`). Selected by default.
|
||||
* Nested checkboxes (indented or grouped) for context types, enabled only if the parent component is checked:
|
||||
* `[x] Memory (API Facts)`
|
||||
* `[x] Reasoning (How-to/Why)`
|
||||
* `[x] Examples (Code Snippets)`
|
||||
(These three sub-checkboxes should be selected by default if the parent component is selected).
|
||||
* **Action Button:**
|
||||
* A button: "Generate & Download Combined Context"
|
||||
* **Status/Feedback Area:** (Optional, but good UX)
|
||||
* Display messages like "Fetching files...", "Combining context...", "Download starting..." or error messages.
|
||||
|
||||
|
||||
**Final Output:**
|
||||
|
||||
* A single HTML file (e.g., `interactive_context_builder.html`).
|
||||
* Associated JavaScript code (can be inline within `<script>` tags or in a separate `.js` file).
|
||||
* Associated CSS code (can be inline within `<style>` tags or in a separate `.css` file).
|
||||
|
||||
This interactive tool will greatly enhance the user experience for `crawl4ai` developers looking to leverage your specialized LLM contexts. Please ensure the JavaScript is robust and provides good user feedback.
|
||||
|
||||
---
|
||||
|
||||
This prompt should give your AI coding assistant a very clear set of requirements and guidelines for building the interactive context builder. Remember to provide it with the list of components as mentioned in the "Input/Assumptions" section.
|
||||
@@ -0,0 +1,135 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Crawl4AI LLM Context Builder</title>
|
||||
<link rel="stylesheet" href="llmtxt.css">
|
||||
</head>
|
||||
<body>
|
||||
<div class="terminal-container">
|
||||
<div class="header">
|
||||
<div class="header-content">
|
||||
<div class="logo-section">
|
||||
<img src="../../img/favicon-32x32.png" alt="Crawl4AI Logo" class="logo">
|
||||
<div>
|
||||
<h1>Crawl4AI LLM Context Builder</h1>
|
||||
<p class="tagline">Multi-Dimensional Context for AI Assistants</p>
|
||||
</div>
|
||||
</div>
|
||||
<nav class="nav-links">
|
||||
<a href="../../" class="nav-link">← Back to Docs</a>
|
||||
<a href="../" class="nav-link">All Apps</a>
|
||||
<a href="https://github.com/unclecode/crawl4ai" class="nav-link" target="_blank">GitHub</a>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="content">
|
||||
|
||||
<section class="intro">
|
||||
<div class="intro-header">
|
||||
<h2>🧠 A New Approach to LLM Context</h2>
|
||||
<p>
|
||||
Traditional <code>llm.txt</code> files often fail with complex libraries like Crawl4AI. They dump massive amounts of API documentation, causing <strong>information overload</strong> and <strong>lost focus</strong>. They provide the "what" but miss the crucial "how" and "why" that makes AI assistants truly helpful.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="intro-solution">
|
||||
<h3>💡 The Solution: Multi-Dimensional, Modular Contexts</h3>
|
||||
<p>
|
||||
Inspired by modular libraries like Lodash, I've redesigned how we provide context to AI assistants. Instead of one monolithic file, Crawl4AI's documentation is organized by <strong>components</strong> and <strong>perspectives</strong>.
|
||||
</p>
|
||||
|
||||
<div class="dimensions">
|
||||
<div class="dimension">
|
||||
<span class="badge memory">Memory</span>
|
||||
<h4>The "What"</h4>
|
||||
<p>Precise API facts, parameters, signatures, and configuration objects. Your unambiguous reference.</p>
|
||||
</div>
|
||||
<div class="dimension">
|
||||
<span class="badge reasoning">Reasoning</span>
|
||||
<h4>The "How" & "Why"</h4>
|
||||
<p>Design principles, best practices, trade-offs, and workflows. Teaches AI to think like an expert.</p>
|
||||
</div>
|
||||
<div class="dimension">
|
||||
<span class="badge examples">Examples</span>
|
||||
<h4>The "Show Me"</h4>
|
||||
<p>Runnable code snippets demonstrating patterns in action. Pure practical implementation.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="intro-benefits">
|
||||
<p>
|
||||
<strong>Why this matters:</strong> You can now give your AI assistant exactly what it needs - whether that's quick API lookups, help designing solutions, or seeing practical implementations. No more information overload, just focused, relevant context.
|
||||
</p>
|
||||
<p class="learn-more">
|
||||
<a href="/blog/articles/llm-context-revolution" class="learn-more-link" target="_parent">📖 Read the full story behind this approach →</a>
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="builder">
|
||||
<div class="component-selector" id="component-selector">
|
||||
<h2>Select Components & Context Types</h2>
|
||||
<div class="select-all-controls">
|
||||
<button class="btn-small" id="select-all">Select All</button>
|
||||
<button class="btn-small" id="deselect-all">Deselect All</button>
|
||||
</div>
|
||||
|
||||
<div class="component-table-wrapper">
|
||||
<table class="component-selection-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th width="50"></th>
|
||||
<th>Component</th>
|
||||
<th class="clickable-header" data-type="memory">Memory<br><span class="header-subtitle">Full Content</span></th>
|
||||
<th class="clickable-header" data-type="reasoning">Reasoning<br><span class="header-subtitle">Diagrams</span></th>
|
||||
<th class="clickable-header" data-type="examples">Examples<br><span class="header-subtitle">Code</span></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="components-tbody">
|
||||
<!-- Components will be dynamically inserted here -->
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="action-area">
|
||||
<div class="token-summary" id="token-summary">
|
||||
<span class="token-label">Estimated Tokens:</span>
|
||||
<span class="token-count" id="total-tokens">0</span>
|
||||
</div>
|
||||
<button class="download-btn" id="download-btn">
|
||||
<span class="icon">⬇</span> Generate & Download Context
|
||||
</button>
|
||||
<div class="status" id="status"></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="reference-table">
|
||||
<h2>Available Context Files</h2>
|
||||
<div class="table-wrapper">
|
||||
<table class="context-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Component</th>
|
||||
<th>Memory</th>
|
||||
<th>Reasoning</th>
|
||||
<th>Examples</th>
|
||||
<th>Full</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="reference-table-body">
|
||||
<!-- Table rows will be dynamically inserted here -->
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="llmtxt.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,603 @@
|
||||
/* Terminal Theme CSS for LLM Context Builder */
|
||||
|
||||
/* Font Face Definitions */
|
||||
@font-face {
|
||||
font-family: 'Dank Mono';
|
||||
src: url('../assets/DankMono-Regular.woff2') format('woff2');
|
||||
font-weight: 400;
|
||||
font-style: normal;
|
||||
font-display: swap;
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'Dank Mono';
|
||||
src: url('../assets/DankMono-Bold.woff2') format('woff2');
|
||||
font-weight: 700;
|
||||
font-style: normal;
|
||||
font-display: swap;
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'Dank Mono';
|
||||
src: url('../assets/DankMono-Italic.woff2') format('woff2');
|
||||
font-weight: 400;
|
||||
font-style: italic;
|
||||
font-display: swap;
|
||||
}
|
||||
|
||||
:root {
|
||||
--background-color: #070708;
|
||||
--font-color: #e8e9ed;
|
||||
--primary-color: #50ffff;
|
||||
--primary-dimmed: #09b5a5;
|
||||
--secondary-color: #d5cec0;
|
||||
--tertiary-color: #a3abba;
|
||||
--accent-color: rgb(243, 128, 245);
|
||||
--error-color: #ff3c74;
|
||||
--code-bg-color: #3f3f44;
|
||||
--border-color: #3f3f44;
|
||||
--hover-bg: #1a1a1c;
|
||||
--success-color: #50ff50;
|
||||
--bg-secondary: #1a1a1a;
|
||||
--primary-green: #0fbbaa;
|
||||
--font-primary: 'Dank Mono', dm, Monaco, Courier New, monospace;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
font-family: var(--font-primary);
|
||||
font-size: 14px;
|
||||
line-height: 1.5;
|
||||
background-color: var(--background-color);
|
||||
color: var(--font-color);
|
||||
}
|
||||
|
||||
/* Terminal Container */
|
||||
.terminal-container {
|
||||
min-height: 100vh;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
/* Header - matching assistant layout */
|
||||
.header {
|
||||
background: var(--bg-secondary);
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
padding: 1.5rem 0;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
backdrop-filter: blur(10px);
|
||||
background: rgba(26, 26, 26, 0.95);
|
||||
}
|
||||
|
||||
.header-content {
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
padding: 0 2rem;
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.logo-section {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 1rem;
|
||||
}
|
||||
|
||||
.logo {
|
||||
width: 48px;
|
||||
height: 48px;
|
||||
}
|
||||
|
||||
.logo-section h1 {
|
||||
font-size: 1.75rem;
|
||||
font-weight: 700;
|
||||
color: var(--font-color);
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.tagline {
|
||||
font-size: 0.875rem;
|
||||
color: var(--tertiary-color);
|
||||
margin-top: 0.25rem;
|
||||
}
|
||||
|
||||
.nav-links {
|
||||
display: flex;
|
||||
gap: 2rem;
|
||||
}
|
||||
|
||||
.nav-link {
|
||||
color: var(--tertiary-color);
|
||||
text-decoration: none;
|
||||
font-size: 0.875rem;
|
||||
transition: color 0.2s ease;
|
||||
}
|
||||
|
||||
.nav-link:hover {
|
||||
color: var(--primary-green);
|
||||
}
|
||||
|
||||
/* Content */
|
||||
.content {
|
||||
flex: 1;
|
||||
max-width: 1200px;
|
||||
margin: 0 auto;
|
||||
padding: 2rem;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
/* Intro Section */
|
||||
.intro {
|
||||
background-color: var(--code-bg-color);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 30px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.intro-header h2 {
|
||||
color: var(--primary-color);
|
||||
margin: 0 0 15px 0;
|
||||
font-size: 20px;
|
||||
}
|
||||
|
||||
.intro-header p {
|
||||
line-height: 1.6;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.intro-header code {
|
||||
background-color: var(--hover-bg);
|
||||
padding: 2px 6px;
|
||||
color: var(--primary-dimmed);
|
||||
}
|
||||
|
||||
.intro-solution {
|
||||
margin-top: 5px;
|
||||
padding-top: 25px;
|
||||
border-top: 1px dashed var(--border-color);
|
||||
}
|
||||
|
||||
.intro-solution h3 {
|
||||
color: var(--secondary-color);
|
||||
margin: 0 0 15px 0;
|
||||
font-size: 18px;
|
||||
}
|
||||
|
||||
.dimensions {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
|
||||
gap: 20px;
|
||||
margin: 20px 0;
|
||||
}
|
||||
|
||||
.dimension {
|
||||
background-color: var(--hover-bg);
|
||||
padding: 20px;
|
||||
border: 1px solid var(--border-color);
|
||||
transition: all 0.2s ease;
|
||||
}
|
||||
|
||||
.dimension:hover {
|
||||
border-color: var(--primary-dimmed);
|
||||
}
|
||||
|
||||
.dimension h4 {
|
||||
color: var(--font-color);
|
||||
margin: 10px 0 8px 0;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.dimension p {
|
||||
font-size: 13px;
|
||||
line-height: 1.5;
|
||||
color: var(--tertiary-color);
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.intro-benefits {
|
||||
margin-top: 0px;
|
||||
padding-top: 0x;
|
||||
border-top: 1px dashed var(--border-color);
|
||||
}
|
||||
|
||||
.intro-benefits strong {
|
||||
color: var(--primary-color);
|
||||
}
|
||||
|
||||
.learn-more {
|
||||
margin-top: 15px;
|
||||
}
|
||||
|
||||
.learn-more-link {
|
||||
color: var(--primary-dimmed);
|
||||
text-decoration: none;
|
||||
font-weight: bold;
|
||||
transition: color 0.2s ease;
|
||||
}
|
||||
|
||||
.learn-more-link:hover {
|
||||
color: var(--primary-color);
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-block;
|
||||
padding: 2px 8px;
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
margin-right: 8px;
|
||||
}
|
||||
|
||||
.badge.memory {
|
||||
background-color: var(--primary-dimmed);
|
||||
color: var(--background-color);
|
||||
}
|
||||
|
||||
.badge.reasoning {
|
||||
background-color: var(--accent-color);
|
||||
color: var(--background-color);
|
||||
}
|
||||
|
||||
.badge.examples {
|
||||
background-color: var(--secondary-color);
|
||||
color: var(--background-color);
|
||||
}
|
||||
|
||||
/* Builder Section */
|
||||
.builder {
|
||||
margin-bottom: 40px;
|
||||
}
|
||||
|
||||
.builder h2 {
|
||||
color: var(--primary-color);
|
||||
font-size: 18px;
|
||||
margin-bottom: 20px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
/* Preset Options */
|
||||
.preset-options {
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.preset-option {
|
||||
flex: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.preset-option input[type="radio"] {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.preset-card {
|
||||
border: 2px solid var(--border-color);
|
||||
padding: 20px;
|
||||
transition: all 0.2s ease;
|
||||
background-color: var(--code-bg-color);
|
||||
}
|
||||
|
||||
.preset-card h3 {
|
||||
margin: 0 0 10px 0;
|
||||
color: var(--secondary-color);
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.preset-card p {
|
||||
margin: 0;
|
||||
font-size: 12px;
|
||||
color: var(--tertiary-color);
|
||||
}
|
||||
|
||||
.preset-option input:checked + .preset-card {
|
||||
border-color: var(--primary-color);
|
||||
background-color: var(--hover-bg);
|
||||
}
|
||||
|
||||
.preset-card:hover {
|
||||
border-color: var(--primary-dimmed);
|
||||
}
|
||||
|
||||
/* Component Selector */
|
||||
.component-selector {
|
||||
margin-bottom: 30px;
|
||||
}
|
||||
|
||||
.select-all-controls {
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.btn-small {
|
||||
background-color: var(--code-bg-color);
|
||||
color: var(--font-color);
|
||||
border: 1px solid var(--border-color);
|
||||
padding: 5px 15px;
|
||||
margin-right: 10px;
|
||||
cursor: pointer;
|
||||
font-family: inherit;
|
||||
font-size: 12px;
|
||||
text-transform: uppercase;
|
||||
transition: all 0.2s ease;
|
||||
}
|
||||
|
||||
.btn-small:hover {
|
||||
background-color: var(--primary-dimmed);
|
||||
color: var(--background-color);
|
||||
}
|
||||
|
||||
/* Component Selection Table */
|
||||
.component-table-wrapper {
|
||||
overflow-x: auto;
|
||||
border: 1px solid var(--border-color);
|
||||
margin-top: 20px;
|
||||
}
|
||||
|
||||
.component-selection-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
background-color: var(--code-bg-color);
|
||||
}
|
||||
|
||||
.component-selection-table th,
|
||||
.component-selection-table td {
|
||||
padding: 12px;
|
||||
text-align: left;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.component-selection-table th {
|
||||
background-color: var(--hover-bg);
|
||||
color: var(--primary-color);
|
||||
text-transform: uppercase;
|
||||
font-size: 12px;
|
||||
letter-spacing: 1px;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.header-subtitle {
|
||||
font-size: 10px;
|
||||
color: var(--tertiary-color);
|
||||
text-transform: none;
|
||||
font-weight: normal;
|
||||
display: block;
|
||||
margin-top: 2px;
|
||||
}
|
||||
|
||||
.component-selection-table th.clickable-header {
|
||||
cursor: pointer;
|
||||
user-select: none;
|
||||
transition: background-color 0.2s ease;
|
||||
}
|
||||
|
||||
.component-selection-table th.clickable-header:hover {
|
||||
background-color: var(--primary-dimmed);
|
||||
color: var(--background-color);
|
||||
}
|
||||
|
||||
.component-selection-table th.clickable-header[data-type="examples"] {
|
||||
cursor: default;
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
.component-selection-table th.clickable-header[data-type="examples"]:hover {
|
||||
background-color: var(--hover-bg);
|
||||
color: var(--primary-color);
|
||||
}
|
||||
|
||||
.component-selection-table th:nth-child(3),
|
||||
.component-selection-table th:nth-child(4),
|
||||
.component-selection-table th:nth-child(5) {
|
||||
text-align: center;
|
||||
width: 120px;
|
||||
}
|
||||
|
||||
.component-selection-table td {
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.component-selection-table td:nth-child(3),
|
||||
.component-selection-table td:nth-child(4),
|
||||
.component-selection-table td:nth-child(5) {
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.component-selection-table tr:hover td {
|
||||
background-color: var(--hover-bg);
|
||||
}
|
||||
|
||||
.component-name {
|
||||
color: var(--primary-color);
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
/* Token display in table cells */
|
||||
.token-info {
|
||||
display: block;
|
||||
font-size: 11px;
|
||||
color: var(--tertiary-color);
|
||||
margin-top: 2px;
|
||||
}
|
||||
|
||||
.component-selection-table input[type="checkbox"] {
|
||||
cursor: pointer;
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
}
|
||||
|
||||
.component-selection-table input[type="checkbox"]:disabled {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.3;
|
||||
}
|
||||
|
||||
/* Disabled row state */
|
||||
.component-selection-table tr.disabled td:not(:first-child) {
|
||||
opacity: 0.5;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* Action Area */
|
||||
.action-area {
|
||||
text-align: center;
|
||||
margin: 40px 0;
|
||||
}
|
||||
|
||||
/* Token Summary */
|
||||
.token-summary {
|
||||
margin-bottom: 20px;
|
||||
font-size: 16px;
|
||||
}
|
||||
|
||||
.token-label {
|
||||
color: var(--tertiary-color);
|
||||
margin-right: 10px;
|
||||
}
|
||||
|
||||
.token-count {
|
||||
color: var(--primary-color);
|
||||
font-weight: bold;
|
||||
font-size: 20px;
|
||||
}
|
||||
|
||||
.token-count::after {
|
||||
content: " est.";
|
||||
font-size: 12px;
|
||||
color: var(--tertiary-color);
|
||||
margin-left: 4px;
|
||||
}
|
||||
|
||||
.download-btn {
|
||||
background-color: var(--primary-dimmed);
|
||||
color: var(--background-color);
|
||||
border: none;
|
||||
padding: 15px 40px;
|
||||
font-size: 16px;
|
||||
font-family: inherit;
|
||||
cursor: pointer;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 1px;
|
||||
transition: all 0.2s ease;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.download-btn:hover {
|
||||
background-color: var(--primary-color);
|
||||
transform: translateY(-2px);
|
||||
}
|
||||
|
||||
.download-btn .icon {
|
||||
font-size: 20px;
|
||||
}
|
||||
|
||||
.status {
|
||||
margin-top: 20px;
|
||||
font-size: 14px;
|
||||
min-height: 30px;
|
||||
}
|
||||
|
||||
.status.loading {
|
||||
color: var(--primary-color);
|
||||
}
|
||||
|
||||
.status.success {
|
||||
color: var(--success-color);
|
||||
}
|
||||
|
||||
.status.error {
|
||||
color: var(--error-color);
|
||||
}
|
||||
|
||||
/* Reference Table */
|
||||
.reference-table {
|
||||
margin-top: 60px;
|
||||
}
|
||||
|
||||
.reference-table h2 {
|
||||
color: var(--primary-color);
|
||||
font-size: 18px;
|
||||
margin-bottom: 20px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.table-wrapper {
|
||||
overflow-x: auto;
|
||||
border: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.context-table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
background-color: var(--code-bg-color);
|
||||
}
|
||||
|
||||
.context-table th,
|
||||
.context-table td {
|
||||
padding: 12px;
|
||||
text-align: left;
|
||||
border-bottom: 1px solid var(--border-color);
|
||||
}
|
||||
|
||||
.context-table th {
|
||||
background-color: var(--hover-bg);
|
||||
color: var(--primary-color);
|
||||
text-transform: uppercase;
|
||||
font-size: 12px;
|
||||
letter-spacing: 1px;
|
||||
}
|
||||
|
||||
.context-table td {
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.context-table tr:hover td {
|
||||
background-color: var(--hover-bg);
|
||||
}
|
||||
|
||||
.file-link {
|
||||
color: var(--primary-dimmed);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.file-link:hover {
|
||||
color: var(--primary-color);
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.file-size {
|
||||
color: var(--tertiary-color);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
/* Responsive Design */
|
||||
@media (max-width: 768px) {
|
||||
.header-content {
|
||||
flex-direction: column;
|
||||
gap: 1.5rem;
|
||||
}
|
||||
|
||||
.nav-links {
|
||||
gap: 1rem;
|
||||
}
|
||||
|
||||
.preset-options {
|
||||
flex-direction: column;
|
||||
}
|
||||
|
||||
.components-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
|
||||
.content {
|
||||
padding: 1rem;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,580 @@
|
||||
// Crawl4AI LLM Context Builder JavaScript
|
||||
|
||||
// Component definitions - order matters
|
||||
const components = [
|
||||
{
|
||||
id: 'installation',
|
||||
name: 'Installation',
|
||||
description: 'Setup and installation options'
|
||||
},
|
||||
{
|
||||
id: 'simple_crawling',
|
||||
name: 'Simple Crawling',
|
||||
description: 'Basic web crawling operations'
|
||||
},
|
||||
{
|
||||
id: 'config_objects',
|
||||
name: 'Configuration Objects',
|
||||
description: 'Browser and crawler configuration'
|
||||
},
|
||||
{
|
||||
id: 'extraction-llm',
|
||||
name: 'Data Extraction Using LLM',
|
||||
description: 'Structured data extraction strategies using LLMs'
|
||||
},
|
||||
{
|
||||
id: 'extraction-no-llm',
|
||||
name: 'Data Extraction Without LLM',
|
||||
description: 'Structured data extraction strategies without LLMs'
|
||||
},
|
||||
{
|
||||
id: 'multi_urls_crawling',
|
||||
name: 'Multi URLs Crawling',
|
||||
description: 'Crawling multiple URLs efficiently'
|
||||
},
|
||||
{
|
||||
id: 'deep_crawling',
|
||||
name: 'Deep Crawling',
|
||||
description: 'Multi-page crawling strategies'
|
||||
},
|
||||
{
|
||||
id: 'docker',
|
||||
name: 'Docker',
|
||||
description: 'Docker deployment and configuration'
|
||||
},
|
||||
{
|
||||
id: 'cli',
|
||||
name: 'CLI',
|
||||
description: 'Command-line interface usage'
|
||||
},
|
||||
{
|
||||
id: 'http_based_crawler_strategy',
|
||||
name: 'HTTP-based Crawler',
|
||||
description: 'HTTP crawler strategy implementation'
|
||||
},
|
||||
{
|
||||
id: 'url_seeder',
|
||||
name: 'URL Seeder',
|
||||
description: 'URL seeding and discovery'
|
||||
},
|
||||
{
|
||||
id: 'deep_crawl_advanced_filters_scorers',
|
||||
name: 'Advanced Filters & Scorers',
|
||||
description: 'Deep crawl filtering and scoring'
|
||||
}
|
||||
];
|
||||
|
||||
// Context types
|
||||
const contextTypes = ['memory', 'reasoning', 'examples'];
|
||||
|
||||
// State management
|
||||
const state = {
|
||||
selectedComponents: new Set(),
|
||||
selectedContextTypes: new Map(),
|
||||
tokenCounts: new Map() // Store token counts for each file
|
||||
};
|
||||
|
||||
// Initialize the application
|
||||
document.addEventListener('DOMContentLoaded', () => {
|
||||
renderComponents();
|
||||
renderReferenceTable();
|
||||
setupActionHandlers();
|
||||
setupColumnHeaderHandlers();
|
||||
|
||||
// Initialize first component as selected with available context types
|
||||
const firstComponent = components[0];
|
||||
state.selectedComponents.add(firstComponent.id);
|
||||
state.selectedContextTypes.set(firstComponent.id, new Set(['memory', 'reasoning']));
|
||||
updateComponentUI();
|
||||
});
|
||||
|
||||
// Helper function to count tokens (words × 2.5)
|
||||
function estimateTokens(text) {
|
||||
if (!text) return 0;
|
||||
const words = text.trim().split(/\s+/).length;
|
||||
return Math.round(words * 2.5);
|
||||
}
|
||||
|
||||
// Update total token count display
|
||||
function updateTotalTokenCount() {
|
||||
let totalTokens = 0;
|
||||
|
||||
state.selectedComponents.forEach(compId => {
|
||||
const types = state.selectedContextTypes.get(compId);
|
||||
if (types) {
|
||||
types.forEach(type => {
|
||||
const key = `${compId}-${type}`;
|
||||
totalTokens += state.tokenCounts.get(key) || 0;
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
document.getElementById('total-tokens').textContent = totalTokens.toLocaleString();
|
||||
}
|
||||
|
||||
// Render component selection table
|
||||
function renderComponents() {
|
||||
const tbody = document.getElementById('components-tbody');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
components.forEach(component => {
|
||||
const row = createComponentRow(component);
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
|
||||
// Fetch token counts for all files
|
||||
fetchAllTokenCounts();
|
||||
}
|
||||
|
||||
// Create a component table row
|
||||
function createComponentRow(component) {
|
||||
const tr = document.createElement('tr');
|
||||
tr.id = `component-${component.id}`;
|
||||
|
||||
// Component checkbox cell
|
||||
const checkboxCell = document.createElement('td');
|
||||
checkboxCell.innerHTML = `
|
||||
<input type="checkbox" id="check-${component.id}"
|
||||
data-component="${component.id}">
|
||||
`;
|
||||
tr.appendChild(checkboxCell);
|
||||
|
||||
// Component name cell
|
||||
const nameCell = document.createElement('td');
|
||||
nameCell.innerHTML = `<span class="component-name">${component.name}</span>`;
|
||||
tr.appendChild(nameCell);
|
||||
|
||||
// Context type cells
|
||||
contextTypes.forEach(type => {
|
||||
const td = document.createElement('td');
|
||||
const key = `${component.id}-${type}`;
|
||||
const tokenCount = state.tokenCounts.get(key) || 0;
|
||||
const isDisabled = type === 'examples' ? 'disabled' : '';
|
||||
|
||||
td.innerHTML = `
|
||||
<input type="checkbox" id="check-${component.id}-${type}"
|
||||
data-component="${component.id}" data-type="${type}"
|
||||
${isDisabled}>
|
||||
<span class="token-info" id="tokens-${component.id}-${type}">
|
||||
${tokenCount > 0 ? `${tokenCount.toLocaleString()} tokens` : ''}
|
||||
</span>
|
||||
`;
|
||||
tr.appendChild(td);
|
||||
});
|
||||
|
||||
// Add event listeners
|
||||
const mainCheckbox = tr.querySelector(`#check-${component.id}`);
|
||||
mainCheckbox.addEventListener('change', (e) => {
|
||||
handleComponentToggle(component.id, e.target.checked);
|
||||
});
|
||||
|
||||
// Add event listeners for context type checkboxes
|
||||
contextTypes.forEach(type => {
|
||||
const typeCheckbox = tr.querySelector(`#check-${component.id}-${type}`);
|
||||
if (!typeCheckbox.disabled) {
|
||||
typeCheckbox.addEventListener('change', (e) => {
|
||||
handleContextTypeToggle(component.id, type, e.target.checked);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
return tr;
|
||||
}
|
||||
|
||||
// Handle component checkbox toggle
|
||||
function handleComponentToggle(componentId, checked) {
|
||||
if (checked) {
|
||||
state.selectedComponents.add(componentId);
|
||||
// Select only available context types when component is selected
|
||||
if (!state.selectedContextTypes.has(componentId)) {
|
||||
state.selectedContextTypes.set(componentId, new Set(['memory', 'reasoning']));
|
||||
} else {
|
||||
// If component was already partially selected, select all available
|
||||
state.selectedContextTypes.set(componentId, new Set(['memory', 'reasoning']));
|
||||
}
|
||||
} else {
|
||||
state.selectedComponents.delete(componentId);
|
||||
state.selectedContextTypes.delete(componentId);
|
||||
}
|
||||
updateComponentUI();
|
||||
}
|
||||
|
||||
// Handle component selection based on context types
|
||||
function updateComponentSelection(componentId) {
|
||||
const types = state.selectedContextTypes.get(componentId) || new Set();
|
||||
if (types.size > 0) {
|
||||
state.selectedComponents.add(componentId);
|
||||
} else {
|
||||
state.selectedComponents.delete(componentId);
|
||||
}
|
||||
}
|
||||
|
||||
// Handle context type checkbox toggle
|
||||
function handleContextTypeToggle(componentId, type, checked) {
|
||||
if (!state.selectedContextTypes.has(componentId)) {
|
||||
state.selectedContextTypes.set(componentId, new Set());
|
||||
}
|
||||
|
||||
const types = state.selectedContextTypes.get(componentId);
|
||||
if (checked) {
|
||||
types.add(type);
|
||||
} else {
|
||||
types.delete(type);
|
||||
}
|
||||
|
||||
updateComponentSelection(componentId);
|
||||
updateComponentUI();
|
||||
}
|
||||
|
||||
// Update UI to reflect current state
|
||||
function updateComponentUI() {
|
||||
components.forEach(component => {
|
||||
const row = document.getElementById(`component-${component.id}`);
|
||||
if (!row) return;
|
||||
|
||||
const mainCheckbox = row.querySelector(`#check-${component.id}`);
|
||||
const hasSelection = state.selectedComponents.has(component.id);
|
||||
const selectedTypes = state.selectedContextTypes.get(component.id) || new Set();
|
||||
|
||||
// Update main checkbox
|
||||
mainCheckbox.checked = hasSelection;
|
||||
|
||||
// Update row disabled state
|
||||
row.classList.toggle('disabled', !hasSelection);
|
||||
|
||||
// Update context type checkboxes
|
||||
contextTypes.forEach(type => {
|
||||
const typeCheckbox = row.querySelector(`#check-${component.id}-${type}`);
|
||||
typeCheckbox.checked = selectedTypes.has(type);
|
||||
});
|
||||
});
|
||||
|
||||
updateTotalTokenCount();
|
||||
}
|
||||
|
||||
// Fetch token counts for all files
|
||||
async function fetchAllTokenCounts() {
|
||||
const promises = [];
|
||||
|
||||
components.forEach(component => {
|
||||
contextTypes.forEach(type => {
|
||||
promises.push(fetchTokenCount(component.id, type));
|
||||
});
|
||||
});
|
||||
|
||||
await Promise.all(promises);
|
||||
updateComponentUI();
|
||||
renderReferenceTable(); // Update reference table with token counts
|
||||
}
|
||||
|
||||
// Fetch token count for a specific file
|
||||
async function fetchTokenCount(componentId, type) {
|
||||
const key = `${componentId}-${type}`;
|
||||
|
||||
try {
|
||||
const fileName = getFileName(componentId, type);
|
||||
const baseUrl = getBaseUrl(type);
|
||||
const response = await fetch(baseUrl + fileName);
|
||||
|
||||
if (response.ok) {
|
||||
const content = await response.text();
|
||||
const tokens = estimateTokens(content);
|
||||
state.tokenCounts.set(key, tokens);
|
||||
|
||||
// Update UI
|
||||
const tokenSpan = document.getElementById(`tokens-${componentId}-${type}`);
|
||||
if (tokenSpan) {
|
||||
tokenSpan.textContent = `${tokens.toLocaleString()} tokens`;
|
||||
}
|
||||
} else if (type === 'examples') {
|
||||
// Examples might not exist yet
|
||||
state.tokenCounts.set(key, 0);
|
||||
const tokenSpan = document.getElementById(`tokens-${componentId}-${type}`);
|
||||
if (tokenSpan) {
|
||||
tokenSpan.textContent = '';
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn(`Failed to fetch token count for ${componentId}-${type}`);
|
||||
if (type === 'examples') {
|
||||
const tokenSpan = document.getElementById(`tokens-${componentId}-${type}`);
|
||||
if (tokenSpan) {
|
||||
tokenSpan.textContent = '';
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get file name based on component and type
|
||||
function getFileName(componentId, type) {
|
||||
// For new structure, all files are just [componentId].txt
|
||||
return `${componentId}.txt`;
|
||||
}
|
||||
|
||||
// Get base URL based on context type
|
||||
function getBaseUrl(type) {
|
||||
// For MkDocs, we need to go up to the root level
|
||||
const basePrefix = window.location.pathname.includes('/apps/') ? '../../' : '/';
|
||||
|
||||
switch(type) {
|
||||
case 'memory':
|
||||
return basePrefix + 'assets/llm.txt/txt/';
|
||||
case 'reasoning':
|
||||
return basePrefix + 'assets/llm.txt/diagrams/';
|
||||
case 'examples':
|
||||
return basePrefix + 'assets/llm.txt/examples/'; // Will return 404 for now
|
||||
default:
|
||||
return basePrefix + 'assets/llm.txt/txt/';
|
||||
}
|
||||
}
|
||||
|
||||
// Setup action button handlers
|
||||
function setupActionHandlers() {
|
||||
// Select/Deselect all buttons
|
||||
document.getElementById('select-all').addEventListener('click', () => {
|
||||
components.forEach(comp => {
|
||||
state.selectedComponents.add(comp.id);
|
||||
state.selectedContextTypes.set(comp.id, new Set(['memory', 'reasoning']));
|
||||
});
|
||||
updateComponentUI();
|
||||
});
|
||||
|
||||
document.getElementById('deselect-all').addEventListener('click', () => {
|
||||
state.selectedComponents.clear();
|
||||
state.selectedContextTypes.clear();
|
||||
updateComponentUI();
|
||||
});
|
||||
|
||||
// Download button
|
||||
document.getElementById('download-btn').addEventListener('click', handleDownload);
|
||||
}
|
||||
|
||||
// Setup column header click handlers
|
||||
function setupColumnHeaderHandlers() {
|
||||
const headers = document.querySelectorAll('.clickable-header');
|
||||
headers.forEach(header => {
|
||||
header.addEventListener('click', () => {
|
||||
const type = header.getAttribute('data-type');
|
||||
toggleColumnSelection(type);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// Toggle all checkboxes in a column
|
||||
function toggleColumnSelection(type) {
|
||||
// Don't toggle examples column
|
||||
if (type === 'examples') return;
|
||||
|
||||
// Check if all are currently selected
|
||||
let allSelected = true;
|
||||
components.forEach(comp => {
|
||||
const types = state.selectedContextTypes.get(comp.id);
|
||||
if (!types || !types.has(type)) {
|
||||
allSelected = false;
|
||||
}
|
||||
});
|
||||
|
||||
// Toggle all
|
||||
components.forEach(comp => {
|
||||
if (!state.selectedContextTypes.has(comp.id)) {
|
||||
state.selectedContextTypes.set(comp.id, new Set());
|
||||
}
|
||||
|
||||
const types = state.selectedContextTypes.get(comp.id);
|
||||
if (allSelected) {
|
||||
types.delete(type);
|
||||
} else {
|
||||
types.add(type);
|
||||
}
|
||||
|
||||
updateComponentSelection(comp.id);
|
||||
});
|
||||
|
||||
updateComponentUI();
|
||||
}
|
||||
|
||||
// Handle download action
|
||||
async function handleDownload() {
|
||||
const statusEl = document.getElementById('status');
|
||||
statusEl.textContent = 'Preparing context files...';
|
||||
statusEl.className = 'status loading';
|
||||
|
||||
try {
|
||||
const files = getSelectedFiles();
|
||||
if (files.length === 0) {
|
||||
throw new Error('No files selected. Please select at least one component or preset.');
|
||||
}
|
||||
|
||||
statusEl.textContent = `Fetching ${files.length} files...`;
|
||||
|
||||
const contents = await fetchFiles(files);
|
||||
const combined = combineContents(contents);
|
||||
|
||||
downloadFile(combined, 'crawl4ai_custom_context.md');
|
||||
|
||||
statusEl.textContent = 'Download complete!';
|
||||
statusEl.className = 'status success';
|
||||
|
||||
setTimeout(() => {
|
||||
statusEl.textContent = '';
|
||||
statusEl.className = 'status';
|
||||
}, 3000);
|
||||
|
||||
} catch (error) {
|
||||
statusEl.textContent = `Error: ${error.message}`;
|
||||
statusEl.className = 'status error';
|
||||
}
|
||||
}
|
||||
|
||||
// Get list of selected files based on current state
|
||||
function getSelectedFiles() {
|
||||
const files = [];
|
||||
|
||||
// Build list of selected files with their context info
|
||||
state.selectedComponents.forEach(compId => {
|
||||
const types = state.selectedContextTypes.get(compId);
|
||||
if (types) {
|
||||
types.forEach(type => {
|
||||
files.push({
|
||||
componentId: compId,
|
||||
type: type,
|
||||
fileName: getFileName(compId, type),
|
||||
baseUrl: getBaseUrl(type)
|
||||
});
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
return files;
|
||||
}
|
||||
|
||||
// Fetch multiple files
|
||||
async function fetchFiles(fileInfos) {
|
||||
const promises = fileInfos.map(async (fileInfo) => {
|
||||
try {
|
||||
const response = await fetch(fileInfo.baseUrl + fileInfo.fileName);
|
||||
if (!response.ok) {
|
||||
if (fileInfo.type === 'examples') {
|
||||
return {
|
||||
fileInfo,
|
||||
content: `<!-- Examples for ${fileInfo.componentId} coming soon -->\n\nExamples are currently being developed for this component.`
|
||||
};
|
||||
}
|
||||
console.warn(`Failed to fetch ${fileInfo.fileName} from ${fileInfo.baseUrl + fileInfo.fileName}`);
|
||||
return { fileInfo, content: `<!-- Failed to load ${fileInfo.fileName} -->` };
|
||||
}
|
||||
const content = await response.text();
|
||||
return { fileInfo, content };
|
||||
} catch (error) {
|
||||
if (fileInfo.type === 'examples') {
|
||||
return {
|
||||
fileInfo,
|
||||
content: `<!-- Examples for ${fileInfo.componentId} coming soon -->\n\nExamples are currently being developed for this component.`
|
||||
};
|
||||
}
|
||||
console.warn(`Error fetching ${fileInfo.fileName}:`, error);
|
||||
return { fileInfo, content: `<!-- Error loading ${fileInfo.fileName} -->` };
|
||||
}
|
||||
});
|
||||
|
||||
return Promise.all(promises);
|
||||
}
|
||||
|
||||
// Combine file contents with headers
|
||||
function combineContents(fileContents) {
|
||||
// Calculate total tokens
|
||||
let totalTokens = 0;
|
||||
fileContents.forEach(({ content }) => {
|
||||
totalTokens += estimateTokens(content);
|
||||
});
|
||||
|
||||
const header = `# Crawl4AI Custom LLM Context
|
||||
Generated on: ${new Date().toISOString()}
|
||||
Total files: ${fileContents.length}
|
||||
Estimated tokens: ${totalTokens.toLocaleString()}
|
||||
|
||||
---
|
||||
|
||||
`;
|
||||
|
||||
const sections = fileContents.map(({ fileInfo, content }) => {
|
||||
const component = components.find(c => c.id === fileInfo.componentId);
|
||||
const componentName = component ? component.name : fileInfo.componentId;
|
||||
const contextType = getContextTypeName(fileInfo.type);
|
||||
const tokens = estimateTokens(content);
|
||||
|
||||
return `## ${componentName} - ${contextType}
|
||||
Component ID: ${fileInfo.componentId}
|
||||
Context Type: ${fileInfo.type}
|
||||
Estimated tokens: ${tokens.toLocaleString()}
|
||||
|
||||
${content}
|
||||
|
||||
---
|
||||
|
||||
`;
|
||||
});
|
||||
|
||||
return header + sections.join('\n');
|
||||
}
|
||||
|
||||
// Get display name for context type
|
||||
function getContextTypeName(type) {
|
||||
switch(type) {
|
||||
case 'memory': return 'Full Content';
|
||||
case 'reasoning': return 'Diagrams & Workflows';
|
||||
case 'examples': return 'Code Examples';
|
||||
default: return type;
|
||||
}
|
||||
}
|
||||
|
||||
// Download file to user's computer
|
||||
function downloadFile(content, fileName) {
|
||||
const blob = new Blob([content], { type: 'text/markdown' });
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = fileName;
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
|
||||
// Render reference table
|
||||
function renderReferenceTable() {
|
||||
const tbody = document.getElementById('reference-table-body');
|
||||
tbody.innerHTML = '';
|
||||
|
||||
// Get base path for links
|
||||
const basePrefix = window.location.pathname.includes('/apps/') ? '../../' : '/';
|
||||
|
||||
components.forEach(component => {
|
||||
const row = document.createElement('tr');
|
||||
const memoryTokens = state.tokenCounts.get(`${component.id}-memory`) || 0;
|
||||
const reasoningTokens = state.tokenCounts.get(`${component.id}-reasoning`) || 0;
|
||||
const examplesTokens = state.tokenCounts.get(`${component.id}-examples`) || 0;
|
||||
|
||||
row.innerHTML = `
|
||||
<td><strong>${component.name}</strong></td>
|
||||
<td>
|
||||
<a href="${basePrefix}assets/llm.txt/txt/${component.id}.txt" class="file-link" target="_blank">Memory</a>
|
||||
${memoryTokens > 0 ? `<span class="file-size">${memoryTokens.toLocaleString()} tokens</span>` : ''}
|
||||
</td>
|
||||
<td>
|
||||
<a href="${basePrefix}assets/llm.txt/diagrams/${component.id}.txt" class="file-link" target="_blank">Reasoning</a>
|
||||
${reasoningTokens > 0 ? `<span class="file-size">${reasoningTokens.toLocaleString()} tokens</span>` : ''}
|
||||
</td>
|
||||
<td>
|
||||
${examplesTokens > 0
|
||||
? `<a href="${basePrefix}assets/llm.txt/examples/${component.id}.txt" class="file-link" target="_blank">Examples</a>
|
||||
<span class="file-size">${examplesTokens.toLocaleString()} tokens</span>`
|
||||
: '-'
|
||||
}
|
||||
</td>
|
||||
<td>-</td>
|
||||
`;
|
||||
tbody.appendChild(row);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Supercharging Your AI Assistant: My Journey to Better LLM Contexts for `crawl4ai`
|
||||
|
||||
When I started diving deep into using AI coding assistants with my own libraries, particularly `crawl4ai`, I quickly realized that the common approach to providing context via a simple `llm.txt` or even a beefed-up `README.md` just wasn't cutting it. This document explains the problems I encountered and how I've tried to create a more effective system for `crawl4ai`, allowing you (and your AI assistant) to get precisely the right information.
|
||||
|
||||
## My Frustration with Standard `llm.txt` Files
|
||||
|
||||
My experience with generic `llm.txt` files for complex libraries like `crawl4ai` revealed several pain points:
|
||||
|
||||
1. **Information Overload & Lost Focus:** I found that when I threw a massive, monolithic context file at an LLM, it often struggled. The sheer volume of information seemed to dilute its focus. If I asked a specific question about a niche feature, the LLM might get sidetracked by more prominent but currently irrelevant parts of the library. It felt like trying to find a single sentence in a thousand-page novel – the information was *there*, but not always accessible or prioritized correctly by the AI.
|
||||
|
||||
2. **The "What" Without the "How" or "Why":** Most `llm.txt` files I encountered were essentially API dumps – a list of functions, classes, and parameters. This is the "what" of a library. But to truly use a library effectively, especially one as flexible as `crawl4ai`, you need the "how" (idiomatic usage patterns, best practices for common tasks) and the "why" (the design rationale behind certain features). Without this, I noticed my AI assistant would often generate syntactically correct but practically inefficient or non-idiomatic code. It was guessing the *intent* and the *best way* to use the library, and those guesses weren't always right.
|
||||
|
||||
3. **No Guidance on "Thinking" Like an Expert:** A static list of facts doesn't teach an LLM the *art* of using the library. It doesn't convey the trade-offs an experienced developer considers, the common pitfalls they've learned to avoid, or the clever ways to combine features to solve complex problems. I wanted my AI assistant to not just recall an API, but to help me *reason* about the best way to build a solution with `crawl4ai`.
|
||||
|
||||
## Inspiration: Selective Inclusion & Multi-Dimensional Understanding
|
||||
|
||||
I've always admired how libraries like Lodash or jQuery (in its modular days) allowed developers to pick and choose only the parts they needed, resulting in smaller, more focused bundles. This idea of modularity and selective inclusion resonated deeply with me as I thought about LLM context. Why force-feed an LLM the entire library's details when I'm only working on a specific component or task?
|
||||
|
||||
This led me to develop a new approach for `crawl4ai`: **multi-dimensional, modular contexts**.
|
||||
|
||||
Instead of one giant `llm.txt`, I've broken down the `crawl4ai` documentation into:
|
||||
|
||||
1. **Logical Components:** Context is organized around the major functional areas of the library (e.g., Core, Data Extraction, Deep Crawling, Markdown Generation, etc.). This allows you to select context relevant only to the task at hand.
|
||||
2. **Three Dimensions of Context for Each Component:**
|
||||
* **`_memory.md` (Foundational Memory):** This is the "what." It contains the precise, factual information about the component's public API, data structures, configuration objects, parameters, and method signatures. It's the detailed, unambiguous reference.
|
||||
* **`_reasoning.md` (Reasoning & Problem-Solving Framework):** This is the "how" and "why." It includes design principles, common task workflows with decision guides, best practices, anti-patterns, illustrative code examples solving real problems, and explanations of trade-offs. It aims to guide the LLM in "thinking" like an expert `crawl4ai` user.
|
||||
* **`_examples.md` (Practical Code Examples):** This is pure "show-me-the-code." It's a collection of runnable snippets demonstrating various ways to use the component's features and configurations, with minimal explanatory text. It’s for quickly seeing different patterns in action.
|
||||
|
||||
**The Goal:**
|
||||
My aim is to provide you with a flexible system. You can give your AI assistant:
|
||||
* Just the **memory** files for quick API lookups.
|
||||
* The **reasoning** files (perhaps with memory) for help designing solutions.
|
||||
* The **examples** files for seeing practical implementations.
|
||||
* A **combination** of these across one or more components tailored to your specific task.
|
||||
* Or, for broader understanding, special aggregate contexts like the "Vibe Coding" context or the "All Library Context."
|
||||
|
||||
By providing these structured, multi-faceted contexts, I hope to significantly improve the quality and relevance of the assistance you get when using AI to code with `crawl4ai`. The following sections will guide you on how to select and use these different context files.
|
||||
Reference in New Issue
Block a user