chore: import upstream snapshot with attribution

examples/api_usage/programmatic/README.md

@@ -0,0 +1,239 @@
+# Local Deep Research - Programmatic API Examples
+
+This directory contains examples demonstrating how to use Local Deep Research programmatically without requiring authentication or database access.
+
+## Quick Start
+
+All examples use the programmatic API that bypasses authentication:
+
+```python
+from local_deep_research.api import quick_summary, detailed_research
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+# Create settings for programmatic mode
+settings = create_settings_snapshot({
+    "search.tool": "wikipedia"
+})
+
+# Run research
+result = quick_summary(
+    "your topic",
+    settings_snapshot=settings,
+    programmatic_mode=True
+)
+```
+
+## Examples Overview
+
+| Example | Purpose | Key Features | Difficulty |
+|---------|---------|--------------|------------|
+| **minimal_working_example.py** | Simplest possible example | Basic setup, minimal code | Beginner |
+| **simple_programmatic_example.py** | Common use cases with the new API | quick_summary, detailed_research, generate_report, custom parameters | Beginner |
+| **search_strategies_example.py** | Demonstrates search strategies | source-based vs focused-iteration strategies | Intermediate |
+| **hybrid_search_example.py** | Combine multiple search sources | Multiple retrievers, web + retriever combo | Intermediate |
+| **advanced_features_example.py** | Advanced programmatic features | generate_report, export formats, result analysis, keyword extraction | Advanced |
+| **custom_llm_retriever_example.py** | Custom LLM and retriever integration | Ollama, custom retrievers, FAISS | Advanced |
+| **searxng_example.py** | Web search with SearXNG | SearXNG integration, error handling | Advanced |
+
+## Example Details
+
+### minimal_working_example.py
+**Purpose:** Show the absolute minimum code needed to use LDR programmatically.
+- Creates a simple LLM and search engine
+- Runs a basic search
+- No external dependencies beyond Ollama
+
+### simple_programmatic_example.py
+**Purpose:** Demonstrate the main API functions with practical examples.
+- `quick_summary()` - Fast research with summary
+- `detailed_research()` - Comprehensive research with findings
+- `generate_report()` - Create full markdown reports
+- Custom search parameters
+- Different search tools (Wikipedia, SearXNG, etc.)
+
+### search_strategies_example.py
+**Purpose:** Explain and demonstrate the two main search strategies.
+- **source-based**: Comprehensive research with detailed citations
+- **focused-iteration**: Iterative refinement of research questions
+- Side-by-side comparison of strategies
+- When to use each strategy
+
+### hybrid_search_example.py
+**Purpose:** Show how to combine multiple search sources for comprehensive research.
+- Multiple named retrievers for different document types
+- Combining custom retrievers with web search
+- Source analysis and tracking
+
+### advanced_features_example.py
+**Purpose:** Demonstrate advanced programmatic features and analysis capabilities.
+- `generate_report()` - Create comprehensive markdown reports
+- Export formats - JSON, Markdown, custom formats
+- Result analysis - Extract insights and patterns
+- Keyword extraction - Identify key terms and concepts
+- Batch research - Process multiple queries efficiently
+
+### custom_llm_retriever_example.py
+**Purpose:** Advanced integration with custom components.
+- Custom LLM implementation (using Ollama)
+- Custom retriever with embeddings
+- Vector store integration (FAISS)
+- Direct use of AdvancedSearchSystem
+
+### searxng_example.py
+**Purpose:** Web search integration using SearXNG.
+- SearXNG configuration
+- Error handling and fallbacks
+- Real-time web search
+- Direct use of search engines
+
+## Key Concepts
+
+### Programmatic Mode
+All examples use `programmatic_mode=True` as an explicit parameter to bypass authentication:
+```python
+result = quick_summary(
+    query="your topic",
+    settings_snapshot=settings,
+    programmatic_mode=True
+)
+```
+
+### Search Strategies
+- **source-based**: Best for academic research, fact-checking
+- **focused-iteration**: Best for exploratory research, complex topics
+
+### Search Tools
+Available search tools include:
+- `wikipedia` - Wikipedia search
+- `arxiv` - Academic papers
+- `searxng` - Web search via SearXNG (recommended default)
+
+With the default langgraph-agent strategy, the research agent can also call
+other enabled engines dynamically per query — the former `auto`/`meta`
+engines were removed in favor of this.
+
+### Custom Retrievers
+You can provide your own retrievers:
+```python
+result = quick_summary(
+    query="topic",
+    retrievers={"my_docs": custom_retriever},
+    search_tool="my_docs",
+    settings_snapshot=settings,
+    programmatic_mode=True
+)
+```
+
+## API Functions
+
+### `quick_summary()`
+Generate a quick research summary:
+```python
+from local_deep_research.api import quick_summary
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+settings = create_settings_snapshot({})
+result = quick_summary(
+    query="Your research question",
+    settings_snapshot=settings,
+    search_tool="wikipedia",
+    iterations=2,
+    programmatic_mode=True
+)
+```
+
+### `detailed_research()`
+Perform in-depth research with multiple iterations:
+```python
+from local_deep_research.api import detailed_research
+
+result = detailed_research(
+    query="Your research question",
+    settings_snapshot=settings,
+    search_strategy="source-based",
+    iterations=3,
+    questions_per_iteration=5,
+    programmatic_mode=True
+)
+```
+
+### `generate_report()`
+Generate comprehensive markdown reports with structured sections:
+```python
+from local_deep_research.api import generate_report
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+settings = create_settings_snapshot(overrides={"programmatic_mode": True})
+result = generate_report(
+    query="Your research question",
+    settings_snapshot=settings,
+    output_file="report.md",
+    searches_per_section=3
+)
+```
+
+## Requirements
+
+- Python 3.8+
+- Local Deep Research installed
+- Ollama (for most examples)
+- SearXNG instance (for searxng_example.py)
+
+## Running the Examples
+
+1. Install Local Deep Research:
+   ```bash
+   pip install -e .
+   ```
+
+2. Start Ollama (if using Ollama examples):
+   ```bash
+   ollama serve
+   ollama pull gemma3:12b
+   ollama pull nomic-embed-text  # For embeddings
+   ```
+
+3. Run any example:
+   ```bash
+   python minimal_working_example.py
+   python simple_programmatic_example.py
+   python search_strategies_example.py
+   ```
+
+## Troubleshooting
+
+### "No settings context available" Error
+Make sure to pass `settings_snapshot` and `programmatic_mode` to all API functions:
+```python
+settings = create_settings_snapshot({})
+result = quick_summary(
+    "topic",
+    settings_snapshot=settings,
+    programmatic_mode=True
+)
+```
+
+### Ollama Connection Error
+Ensure Ollama is running:
+```bash
+ollama serve
+```
+
+### SearXNG Connection Error
+Start a SearXNG instance or use the fallback in the example:
+```bash
+docker run -p 8080:8080 searxng/searxng
+```
+
+
+## Contributing
+
+When adding new examples:
+1. Focus on demonstrating specific features
+2. Include clear comments explaining the code
+3. Handle errors gracefully
+4. Update this README with the new example
+
+## License
+
+See the main project LICENSE file.

examples/api_usage/programmatic/advanced_features_example.py

@@ -0,0 +1,612 @@
+#!/usr/bin/env python3
+"""
+Advanced Features Example for Local Deep Research
+
+This example demonstrates advanced programmatic features including:
+1. generate_report() - Create comprehensive markdown reports
+2. Export formats - Save reports in different formats
+3. Result analysis - Extract and analyze research findings
+4. Keyword extraction - Identify key topics and concepts
+"""
+
+import json
+from typing import Dict, List, Any
+
+from local_deep_research.api import (
+    generate_report,
+    detailed_research,
+    quick_summary,
+)
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+
+def demonstrate_report_generation():
+    """
+    Generate a comprehensive research report using generate_report().
+
+    This function creates a structured markdown report with:
+    - Executive summary
+    - Detailed findings organized by sections
+    - Source citations
+    - Conclusions and recommendations
+    """
+    print("=" * 70)
+    print("GENERATE COMPREHENSIVE REPORT")
+    print("=" * 70)
+    print("""
+This demonstrates the generate_report() function which:
+- Creates a structured markdown report
+- Performs multiple searches per section
+- Organizes findings into coherent sections
+- Includes citations and references
+    """)
+
+    # Configure settings for programmatic mode
+    settings = create_settings_snapshot(
+        overrides={
+            "programmatic_mode": True,
+            "search.tool": "wikipedia",
+            "llm.temperature": 0.5,  # Lower for more focused output
+            "api.allow_file_output": True,  # Allow generate_report to save files
+        }
+    )
+
+    # Generate a comprehensive report
+    print(
+        "Generating report on 'Applications of Machine Learning in Healthcare'..."
+    )
+    report = generate_report(
+        query="Applications of Machine Learning in Healthcare",
+        output_file="ml_healthcare_report.md",
+        searches_per_section=2,  # Multiple searches per section for depth
+        settings_snapshot=settings,
+        iterations=2,
+        questions_per_iteration=3,
+    )
+
+    print("\n✓ Report generated successfully!")
+    print(f"  - Report length: {len(report['content'])} characters")
+    print(
+        f"  - File saved to: {report.get('file_path', 'ml_healthcare_report.md')}"
+    )
+
+    # Show first part of report
+    print("\nReport preview (first 500 chars):")
+    print("-" * 40)
+    print(report["content"][:500] + "...")
+
+    return report
+
+
+def demonstrate_export_formats():
+    """
+    Show how to export research results in different formats.
+
+    Demonstrates:
+    - Markdown export (default)
+    - JSON export for programmatic processing
+    - Custom formatting with templates
+    """
+    print("\n" + "=" * 70)
+    print("EXPORT FORMATS")
+    print("=" * 70)
+    print("""
+Exporting research in different formats:
+- Markdown: Human-readable reports
+- JSON: Structured data for processing
+- Custom: Template-based formatting
+    """)
+
+    settings = create_settings_snapshot(
+        overrides={
+            "programmatic_mode": True,
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Get research results
+    result = detailed_research(
+        query="Renewable energy technologies",
+        settings_snapshot=settings,
+        iterations=1,
+        questions_per_iteration=2,
+    )
+
+    # Export as JSON
+    json_file = "research_results.json"
+    with open(json_file, "w", encoding="utf-8") as f:
+        json.dump(result, f, indent=2, default=str)
+    print(f"\n✓ JSON export saved to: {json_file}")
+    print(f"  - Contains: {len(result.get('findings', []))} findings")
+    print(f"  - Sources: {len(result.get('sources', []))} sources")
+
+    # Export as Markdown
+    md_content = format_as_markdown(result)
+    md_file = "research_results.md"
+    with open(md_file, "w", encoding="utf-8") as f:
+        f.write(md_content)
+    print(f"\n✓ Markdown export saved to: {md_file}")
+    print(f"  - Length: {len(md_content)} characters")
+
+    # Export as custom format (e.g., BibTeX-like citations)
+    citations = extract_citations(result)
+    cite_file = "research_citations.txt"
+    with open(cite_file, "w", encoding="utf-8") as f:
+        for i, citation in enumerate(citations, 1):
+            f.write(f"[{i}] {citation}\n")
+    print(f"\n✓ Citations export saved to: {cite_file}")
+    print(f"  - Total citations: {len(citations)}")
+
+    return result
+
+
+def demonstrate_result_analysis():
+    """
+    Analyze research results to extract insights and patterns.
+
+    Shows how to:
+    - Extract key findings
+    - Identify recurring themes
+    - Analyze source reliability
+    - Generate statistics
+    """
+    print("\n" + "=" * 70)
+    print("RESULT ANALYSIS")
+    print("=" * 70)
+    print("""
+Analyzing research results to extract:
+- Key findings and insights
+- Common themes and patterns
+- Source statistics
+- Quality metrics
+    """)
+
+    settings = create_settings_snapshot(
+        overrides={
+            "programmatic_mode": True,
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Perform research
+    result = detailed_research(
+        query="Impact of artificial intelligence on employment",
+        settings_snapshot=settings,
+        search_strategy="source-based",
+        iterations=2,
+        questions_per_iteration=3,
+    )
+
+    # Analyze findings
+    analysis = analyze_findings(result)
+
+    print("\n📊 Research Analysis:")
+    print(f"  - Total findings: {analysis['total_findings']}")
+    print(f"  - Unique sources: {analysis['unique_sources']}")
+    print(f"  - Questions explored: {analysis['total_questions']}")
+    print(f"  - Iterations completed: {analysis['iterations']}")
+
+    print("\n🔍 Finding Categories:")
+    for category, count in analysis["categories"].items():
+        print(f"  - {category}: {count} findings")
+
+    print("\n📈 Source Distribution:")
+    for source_type, count in analysis["source_types"].items():
+        print(f"  - {source_type}: {count} sources")
+
+    # Extract themes
+    themes = extract_themes(result)
+    print("\n🎯 Key Themes Identified:")
+    for i, theme in enumerate(themes[:5], 1):
+        print(f"  {i}. {theme}")
+
+    return analysis
+
+
+def demonstrate_keyword_extraction():
+    """
+    Extract keywords and key concepts from research results.
+
+    Demonstrates:
+    - Keyword extraction from findings
+    - Concept identification
+    - Topic clustering
+    - Trend analysis
+    """
+    print("\n" + "=" * 70)
+    print("KEYWORD & CONCEPT EXTRACTION")
+    print("=" * 70)
+    print("""
+Extracting keywords and concepts:
+- Important terms and phrases
+- Technical concepts
+- Named entities
+- Trend indicators
+    """)
+
+    settings = create_settings_snapshot(
+        overrides={
+            "programmatic_mode": True,
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Quick research for keyword extraction
+    result = quick_summary(
+        query="Quantum computing breakthroughs 2024",
+        settings_snapshot=settings,
+        iterations=1,
+        questions_per_iteration=3,
+    )
+
+    # Extract keywords
+    keywords = extract_keywords(result)
+
+    print("\n🔑 Top Keywords:")
+    for keyword, frequency in keywords[:10]:
+        print(f"  - {keyword}: {frequency} occurrences")
+
+    # Extract concepts
+    concepts = extract_concepts(result)
+
+    print("\n💡 Key Concepts:")
+    for i, concept in enumerate(concepts[:5], 1):
+        print(f"  {i}. {concept}")
+
+    # Identify technical terms
+    technical_terms = extract_technical_terms(result)
+
+    print("\n🔬 Technical Terms:")
+    for term in technical_terms[:8]:
+        print(f"  - {term}")
+
+    return keywords, concepts
+
+
+def format_as_markdown(result: Dict[str, Any]) -> str:
+    """Convert research results to markdown format."""
+    md = f"# Research Report: {result['query']}\n\n"
+    md += f"**Research ID:** {result.get('research_id', 'N/A')}\n\n"
+
+    # Summary
+    md += "## Summary\n\n"
+    md += result.get("summary", "No summary available") + "\n\n"
+
+    # Findings
+    md += "## Key Findings\n\n"
+    findings = result.get("findings", [])
+    for i, finding in enumerate(findings, 1):
+        finding_text = finding if isinstance(finding, str) else str(finding)
+        md += f"{i}. {finding_text}\n\n"
+
+    # Sources
+    md += "## Sources\n\n"
+    sources = result.get("sources", [])
+    for i, source in enumerate(sources, 1):
+        source_text = source if isinstance(source, str) else str(source)
+        md += f"- [{i}] {source_text}\n"
+
+    # Metadata
+    md += "\n## Metadata\n\n"
+    metadata = result.get("metadata", {})
+    for key, value in metadata.items():
+        md += f"- **{key}:** {value}\n"
+
+    return md
+
+
+def extract_citations(result: Dict[str, Any]) -> List[str]:
+    """Extract citations from research results."""
+    citations = []
+    sources = result.get("sources", [])
+
+    for source in sources:
+        if isinstance(source, dict):
+            # Extract URL or title
+            citation = source.get("url", source.get("title", str(source)))
+        else:
+            citation = str(source)
+        citations.append(citation)
+
+    return citations
+
+
+def analyze_findings(result: Dict[str, Any]) -> Dict[str, Any]:
+    """Analyze research findings for patterns and statistics."""
+    findings = result.get("findings", [])
+    sources = result.get("sources", [])
+    questions = result.get("questions", {})
+
+    # Categorize findings (simplified)
+    categories = {
+        "positive": 0,
+        "negative": 0,
+        "neutral": 0,
+        "technical": 0,
+    }
+
+    for finding in findings:
+        finding_text = str(finding).lower()
+        if any(
+            word in finding_text
+            for word in ["benefit", "improve", "enhance", "positive"]
+        ):
+            categories["positive"] += 1
+        elif any(
+            word in finding_text
+            for word in ["risk", "challenge", "negative", "concern"]
+        ):
+            categories["negative"] += 1
+        elif any(
+            word in finding_text
+            for word in ["algorithm", "system", "technology", "method"]
+        ):
+            categories["technical"] += 1
+        else:
+            categories["neutral"] += 1
+
+    # Analyze sources
+    source_types = {}
+    for source in sources:
+        source_text = str(source).lower()
+        if "wikipedia" in source_text:
+            source_type = "Wikipedia"
+        elif "arxiv" in source_text:
+            source_type = "ArXiv"
+        elif "github" in source_text:
+            source_type = "GitHub"
+        else:
+            source_type = "Other"
+        source_types[source_type] = source_types.get(source_type, 0) + 1
+
+    return {
+        "total_findings": len(findings),
+        "unique_sources": len(sources),
+        "total_questions": sum(len(qs) for qs in questions.values()),
+        "iterations": result.get("iterations", 0),
+        "categories": categories,
+        "source_types": source_types,
+    }
+
+
+def extract_themes(result: Dict[str, Any]) -> List[str]:
+    """Extract main themes from research results."""
+    # Simplified theme extraction based on common patterns
+    themes = []
+    summary = result.get("summary", "")
+    findings = result.get("findings", [])
+
+    # Combine text for analysis
+    full_text = summary + " ".join(str(f) for f in findings)
+
+    # Simple theme patterns (in production, use NLP libraries)
+    theme_patterns = {
+        "automation": ["automation", "automated", "automatic"],
+        "job displacement": ["job loss", "unemployment", "displacement"],
+        "skill requirements": ["skills", "training", "education"],
+        "economic impact": ["economy", "economic", "gdp", "growth"],
+        "innovation": ["innovation", "innovative", "breakthrough"],
+    }
+
+    for theme, keywords in theme_patterns.items():
+        if any(keyword in full_text.lower() for keyword in keywords):
+            themes.append(theme.title())
+
+    return themes
+
+
+def extract_keywords(result: Dict[str, Any]) -> List[tuple]:
+    """Extract keywords with frequency from research results."""
+    from collections import Counter
+    import re
+
+    # Combine all text
+    summary = result.get("summary", "")
+    findings = " ".join(str(f) for f in result.get("findings", []))
+    full_text = f"{summary} {findings}".lower()
+
+    # Simple word extraction (in production, use NLP libraries)
+    words = re.findall(r"\b[a-z]{4,}\b", full_text)
+
+    # Filter common words
+    stopwords = {
+        "that",
+        "this",
+        "with",
+        "from",
+        "have",
+        "been",
+        "were",
+        "which",
+        "their",
+        "about",
+    }
+    words = [w for w in words if w not in stopwords]
+
+    # Count frequencies
+    word_freq = Counter(words)
+
+    return word_freq.most_common(20)
+
+
+def extract_concepts(result: Dict[str, Any]) -> List[str]:
+    """Extract key concepts from research results."""
+    concepts = []
+    summary = result.get("summary", "")
+
+    # Simple concept patterns (in production, use NLP for entity extraction)
+    concept_patterns = [
+        r"quantum \w+",
+        r"\w+ computing",
+        r"\w+ algorithm",
+        r"machine learning",
+        r"artificial intelligence",
+        r"\w+ technology",
+    ]
+
+    import re
+
+    for pattern in concept_patterns:
+        matches = re.findall(pattern, summary.lower())
+        concepts.extend(matches)
+
+    # Deduplicate and clean
+    concepts = list(set(concepts))
+
+    return concepts[:10]
+
+
+def extract_technical_terms(result: Dict[str, Any]) -> List[str]:
+    """Extract technical terms from research results."""
+    technical_terms = []
+
+    # Common technical term patterns
+    tech_indicators = [
+        "algorithm",
+        "system",
+        "protocol",
+        "framework",
+        "architecture",
+        "quantum",
+        "neural",
+        "network",
+        "model",
+        "optimization",
+    ]
+
+    summary = result.get("summary", "").lower()
+    import re
+
+    for indicator in tech_indicators:
+        # Find words containing or adjacent to technical indicators
+        pattern = rf"\b\w*{indicator}\w*\b"
+        matches = re.findall(pattern, summary)
+        technical_terms.extend(matches)
+
+    # Deduplicate
+    technical_terms = list(set(technical_terms))
+
+    return technical_terms
+
+
+def demonstrate_batch_research():
+    """
+    Show how to perform batch research on multiple topics.
+
+    Useful for:
+    - Comparative analysis
+    - Trend monitoring
+    - Systematic reviews
+    """
+    print("\n" + "=" * 70)
+    print("BATCH RESEARCH PROCESSING")
+    print("=" * 70)
+    print("""
+Processing multiple research queries:
+- Efficient batch processing
+- Comparative analysis
+- Result aggregation
+    """)
+
+    settings = create_settings_snapshot(
+        overrides={
+            "programmatic_mode": True,
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Topics for batch research
+    topics = [
+        "Solar energy innovations",
+        "Wind power technology",
+        "Hydrogen fuel cells",
+    ]
+
+    batch_results = {}
+
+    print("\n📚 Batch Research:")
+    for topic in topics:
+        print(f"\n  Researching: {topic}")
+        result = quick_summary(
+            query=topic,
+            settings_snapshot=settings,
+            iterations=1,
+            questions_per_iteration=2,
+        )
+        batch_results[topic] = result
+        print(f"    ✓ Found {len(result.get('findings', []))} findings")
+
+    # Aggregate results
+    print("\n📊 Aggregate Analysis:")
+    total_findings = sum(
+        len(r.get("findings", [])) for r in batch_results.values()
+    )
+    total_sources = sum(
+        len(r.get("sources", [])) for r in batch_results.values()
+    )
+
+    print(f"  - Total topics researched: {len(topics)}")
+    print(f"  - Total findings: {total_findings}")
+    print(f"  - Total sources: {total_sources}")
+    print(f"  - Average findings per topic: {total_findings / len(topics):.1f}")
+
+    # Save batch results
+    batch_file = "batch_research_results.json"
+    with open(batch_file, "w", encoding="utf-8") as f:
+        json.dump(batch_results, f, indent=2, default=str)
+    print(f"\n✓ Batch results saved to: {batch_file}")
+
+    return batch_results
+
+
+def main():
+    """Run all advanced feature demonstrations."""
+    print("=" * 70)
+    print("LOCAL DEEP RESEARCH - ADVANCED FEATURES DEMONSTRATION")
+    print("=" * 70)
+    print("""
+This example demonstrates advanced programmatic features:
+1. Report generation with generate_report()
+2. Multiple export formats
+3. Result analysis and insights
+4. Keyword and concept extraction
+5. Batch research processing
+    """)
+
+    # Run demonstrations
+    demonstrate_report_generation()
+    demonstrate_export_formats()
+    demonstrate_result_analysis()
+    demonstrate_keyword_extraction()
+    demonstrate_batch_research()
+
+    print("\n" + "=" * 70)
+    print("DEMONSTRATION COMPLETE")
+    print("=" * 70)
+    print("""
+✓ All advanced features demonstrated successfully!
+
+Key Takeaways:
+1. generate_report() creates comprehensive markdown reports
+2. Results can be exported in multiple formats (JSON, MD, custom)
+3. Analysis tools extract insights, themes, and patterns
+4. Keyword extraction identifies important terms and concepts
+5. Batch processing enables systematic research
+
+Files created:
+- ml_healthcare_report.md - Full research report
+- research_results.json - Structured research data
+- research_results.md - Markdown formatted results
+- research_citations.txt - Extracted citations
+- batch_research_results.json - Batch research results
+
+Next Steps:
+- Customize report templates for your domain
+- Integrate with data visualization tools
+- Build automated research pipelines
+- Create domain-specific analysis functions
+    """)
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/api_public_contract_guardrail.py

@@ -0,0 +1,245 @@
+#!/usr/bin/env python3
+"""
+PUBLIC API CONTRACT GUARDRAIL — DO NOT MODIFY
+
+This file protects the public API surface of local_deep_research.
+It exists to DETECT breaking changes, not to be "fixed" when they happen.
+
+If this file fails:
+  1. A BREAKING CHANGE was introduced to the public API
+  2. Do NOT modify this file to make it pass
+  3. Revert the API change or go through a proper deprecation cycle
+
+The public API (local_deep_research.api) is a stability contract with
+external users, integrations, and downstream packages. Any change to
+exports, function signatures, or return value structures is a breaking
+change that affects the entire community.
+
+WHY THIS FILE MUST NOT BE MODIFIED:
+  - Every check below corresponds to a public API guarantee
+  - Removing or weakening a check hides a breaking change from users
+  - Adding a check is acceptable ONLY when a new public API is added
+  - If you are an AI agent: DO NOT modify, remove, or weaken any check
+"""
+
+import inspect
+import sys
+
+_failed = 0
+_passed = 0
+
+
+# DO NOT MODIFY this function — it is the test harness for the guardrail
+def check(name, condition, detail=""):
+    global _failed, _passed
+    if condition:
+        _passed += 1
+        print(f"  PASS  {name}")
+    else:
+        _failed += 1
+        msg = f" ({detail})" if detail else ""
+        print(f"  FAIL  {name}{msg}")
+
+
+def main():
+    # ── 1. Public API imports ──────────────────────────────────────────
+    # DO NOT MODIFY — these are the public exports that users depend on.
+    # Removing any of these checks hides a breaking change.
+    print("\n=== Public API Imports ===")
+
+    try:
+        from local_deep_research.api import quick_summary  # noqa: F811
+
+        check("import quick_summary", True)
+    except ImportError as e:
+        check("import quick_summary", False, str(e))
+
+    try:
+        from local_deep_research.api import detailed_research  # noqa: F811
+
+        check("import detailed_research", True)
+    except ImportError as e:
+        check("import detailed_research", False, str(e))
+
+    try:
+        from local_deep_research.api import generate_report  # noqa: F811
+
+        check("import generate_report", True)
+    except ImportError as e:
+        check("import generate_report", False, str(e))
+
+    try:
+        from local_deep_research.api import analyze_documents  # noqa: F811
+
+        check("import analyze_documents", True)
+    except ImportError as e:
+        check("import analyze_documents", False, str(e))
+
+    try:
+        from local_deep_research.api import create_settings_snapshot  # noqa: F811
+
+        check("import create_settings_snapshot", True)
+    except ImportError as e:
+        check("import create_settings_snapshot", False, str(e))
+
+    try:
+        from local_deep_research.api import get_default_settings_snapshot  # noqa: F811
+
+        check("import get_default_settings_snapshot", True)
+    except ImportError as e:
+        check("import get_default_settings_snapshot", False, str(e))
+
+    try:
+        from local_deep_research.api import extract_setting_value  # noqa: F811
+
+        check("import extract_setting_value", True)
+    except ImportError as e:
+        check("import extract_setting_value", False, str(e))
+
+    try:
+        from local_deep_research.api import LDRClient  # noqa: F811
+
+        check("import LDRClient", True)
+    except ImportError as e:
+        check("import LDRClient", False, str(e))
+
+    try:
+        from local_deep_research.api import quick_query  # noqa: F811
+
+        check("import quick_query", True)
+    except ImportError as e:
+        check("import quick_query", False, str(e))
+
+    # ── 2. Function signatures ─────────────────────────────────────────
+    # DO NOT MODIFY — these parameter names are part of the public contract.
+    # Renaming or removing parameters breaks all callers.
+    print("\n=== Function Signatures ===")
+
+    sig = inspect.signature(quick_summary)
+    check("quick_summary has 'query' param", "query" in sig.parameters)
+    check("quick_summary has 'llms' param", "llms" in sig.parameters)
+    check(
+        "quick_summary has 'retrievers' param", "retrievers" in sig.parameters
+    )
+
+    sig = inspect.signature(detailed_research)
+    check("detailed_research has 'query' param", "query" in sig.parameters)
+    check("detailed_research has 'llms' param", "llms" in sig.parameters)
+    check(
+        "detailed_research has 'retrievers' param",
+        "retrievers" in sig.parameters,
+    )
+
+    sig = inspect.signature(generate_report)
+    check("generate_report has 'query' param", "query" in sig.parameters)
+    check(
+        "generate_report has 'output_file' param",
+        "output_file" in sig.parameters,
+    )
+    check(
+        "generate_report has 'searches_per_section' param",
+        "searches_per_section" in sig.parameters,
+    )
+
+    sig = inspect.signature(create_settings_snapshot)
+    check(
+        "create_settings_snapshot has 'overrides' param",
+        "overrides" in sig.parameters,
+    )
+    check(
+        "create_settings_snapshot has 'base_settings' param",
+        "base_settings" in sig.parameters,
+    )
+
+    # ── 3. Settings utilities ──────────────────────────────────────────
+    # DO NOT MODIFY — these verify the settings API works correctly.
+    # External users depend on create_settings_snapshot() returning a dict.
+    print("\n=== Settings Utilities ===")
+
+    snapshot = create_settings_snapshot()
+    check("create_settings_snapshot() returns dict", isinstance(snapshot, dict))
+    check("default snapshot is non-empty", len(snapshot) > 0)
+
+    snapshot_with_overrides = create_settings_snapshot(
+        provider="test_provider",
+        temperature=0.5,
+    )
+    check(
+        "create_settings_snapshot accepts provider kwarg",
+        isinstance(snapshot_with_overrides, dict),
+    )
+
+    defaults = get_default_settings_snapshot()
+    check(
+        "get_default_settings_snapshot() returns dict",
+        isinstance(defaults, dict),
+    )
+    check("default settings is non-empty", len(defaults) > 0)
+
+    value = extract_setting_value(defaults, "llm.temperature")
+    check(
+        "extract_setting_value returns a value for llm.temperature",
+        value is not None,
+    )
+
+    # ── 4. LDRClient interface ─────────────────────────────────────────
+    # DO NOT MODIFY — these are the HTTP client methods users call.
+    # Removing any method breaks all HTTP-based integrations.
+    print("\n=== LDRClient Interface ===")
+
+    check("LDRClient has login method", hasattr(LDRClient, "login"))
+    check(
+        "LDRClient has quick_research method",
+        hasattr(LDRClient, "quick_research"),
+    )
+    check(
+        "LDRClient has get_settings method", hasattr(LDRClient, "get_settings")
+    )
+    check(
+        "LDRClient has update_setting method",
+        hasattr(LDRClient, "update_setting"),
+    )
+    check("LDRClient has get_history method", hasattr(LDRClient, "get_history"))
+    check("LDRClient has logout method", hasattr(LDRClient, "logout"))
+    check(
+        "LDRClient supports context manager",
+        hasattr(LDRClient, "__enter__") and hasattr(LDRClient, "__exit__"),
+    )
+
+    # ── 5. Callability ─────────────────────────────────────────────────
+    # DO NOT MODIFY — verifies all exports are actually callable.
+    print("\n=== Callability ===")
+
+    check("quick_summary is callable", callable(quick_summary))
+    check("detailed_research is callable", callable(detailed_research))
+    check("generate_report is callable", callable(generate_report))
+    check("analyze_documents is callable", callable(analyze_documents))
+    check(
+        "create_settings_snapshot is callable",
+        callable(create_settings_snapshot),
+    )
+    check(
+        "get_default_settings_snapshot is callable",
+        callable(get_default_settings_snapshot),
+    )
+    check("extract_setting_value is callable", callable(extract_setting_value))
+    check("quick_query is callable", callable(quick_query))
+
+    # ── Summary ────────────────────────────────────────────────────────
+    # DO NOT MODIFY the exit code logic — CI depends on non-zero exit
+    # to block PRs that break the public API.
+    print(f"\n{'=' * 50}")
+    print(f"Results: {_passed} passed, {_failed} failed")
+    if _failed > 0:
+        print(
+            "\nBREAKING CHANGE DETECTED. The public API has changed.\n"
+            "Do NOT modify this file to make it pass.\n"
+            "Revert the API change or follow a proper deprecation cycle."
+        )
+        sys.exit(1)
+    else:
+        print("\nAll API public contract checks passed.")
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/custom_llm_retriever_example.py

@@ -0,0 +1,215 @@
+#!/usr/bin/env python3
+"""
+Example of using a custom LLM with a custom retriever in Local Deep Research.
+
+This demonstrates how to integrate your own LLM implementation and custom
+retrieval system for programmatic access.
+"""
+
+from typing import List, Dict
+from langchain_ollama import ChatOllama, OllamaEmbeddings
+from langchain_core.retrievers import Document
+from langchain_community.vectorstores import FAISS
+
+# Import the search system
+from local_deep_research.search_system import AdvancedSearchSystem
+
+# Re-enable logging after import
+from loguru import logger
+import sys
+
+logger.remove()
+# diagnose=False: loguru defaults to True, which renders repr() of every
+# local in every traceback frame on exception. Users copy this snippet
+# into their own scripts, so leaving the default on would propagate the
+# credential-in-traceback leak (#4185) wherever the snippet lands.
+logger.add(
+    sys.stderr,
+    level="INFO",
+    format="{time} {level} {message}",
+    diagnose=False,
+)
+logger.enable("local_deep_research")
+
+
+class CustomRetriever:
+    """Custom retriever that can fetch from multiple sources."""
+
+    def __init__(self):
+        # Initialize with sample documents for demonstration
+        self.documents = [
+            {
+                "content": "Quantum computing uses quantum bits (qubits) that can exist in superposition, "
+                "allowing parallel computation of multiple states simultaneously.",
+                "title": "Quantum Computing Fundamentals",
+                "source": "quantum_basics.pdf",
+                "metadata": {"topic": "quantum", "year": 2024},
+            },
+            {
+                "content": "Machine learning algorithms can be categorized into supervised, unsupervised, "
+                "and reinforcement learning approaches, each suited for different tasks.",
+                "title": "ML Algorithm Categories",
+                "source": "ml_overview.pdf",
+                "metadata": {"topic": "ml", "year": 2024},
+            },
+            {
+                "content": "Neural networks are inspired by biological neurons and consist of interconnected "
+                "nodes that process information through weighted connections.",
+                "title": "Neural Network Architecture",
+                "source": "nn_architecture.pdf",
+                "metadata": {"topic": "neural_networks", "year": 2023},
+            },
+            {
+                "content": "Natural language processing enables computers to understand, interpret, and "
+                "generate human language, powering applications like chatbots and translation.",
+                "title": "NLP Applications",
+                "source": "nlp_apps.pdf",
+                "metadata": {"topic": "nlp", "year": 2024},
+            },
+        ]
+
+        # Create embeddings for similarity search
+        logger.info("Initializing custom retriever with embeddings...")
+        self.embeddings = OllamaEmbeddings(model="nomic-embed-text")
+
+        # Create vector store from documents
+        docs = [
+            Document(
+                page_content=doc["content"],
+                metadata={
+                    "title": doc["title"],
+                    "source": doc["source"],
+                    **doc["metadata"],
+                },
+            )
+            for doc in self.documents
+        ]
+        self.vectorstore = FAISS.from_documents(docs, self.embeddings)
+
+    def retrieve(self, query: str, k: int = 3) -> List[Dict]:
+        """Custom retrieval logic."""
+        logger.info(f"Custom Retriever: Searching for '{query}'")
+
+        # Use vector similarity search
+        similar_docs = self.vectorstore.similarity_search(query, k=k)
+
+        # Convert to expected format
+        results = []
+        for i, doc in enumerate(similar_docs):
+            results.append(
+                {
+                    "title": doc.metadata.get("title", f"Document {i + 1}"),
+                    "link": doc.metadata.get("source", "custom_source"),
+                    "snippet": doc.page_content[:150] + "...",
+                    "full_content": doc.page_content,
+                    "rank": i + 1,
+                    "metadata": doc.metadata,
+                }
+            )
+
+        logger.info(
+            f"Custom Retriever: Found {len(results)} relevant documents"
+        )
+        return results
+
+
+class CustomSearchEngine:
+    """Adapter to integrate custom retriever with the search system."""
+
+    def __init__(self, retriever: CustomRetriever, settings_snapshot=None):
+        self.retriever = retriever
+        self.settings_snapshot = settings_snapshot or {}
+
+    def run(self, query: str, research_context=None) -> List[Dict]:
+        """Execute search using custom retriever."""
+        return self.retriever.retrieve(query, k=5)
+
+
+def main():
+    """Demonstrate custom LLM and retriever integration."""
+    print("=== Custom LLM and Retriever Example ===\n")
+
+    # 1. Create custom LLM (just using regular Ollama for simplicity)
+    print("1. Initializing LLM...")
+    llm = ChatOllama(model="gemma3:12b", temperature=0.7)
+
+    # 2. Create custom retriever
+    print("2. Setting up custom retriever...")
+    custom_retriever = CustomRetriever()
+
+    # 3. Create settings
+    settings = {
+        "search.iterations": 2,
+        "search.questions_per_iteration": 3,
+        "search.strategy": "source-based",
+        "rate_limiting.enabled": False,  # Disable rate limiting for custom setup
+    }
+
+    # 4. Create search engine adapter
+    print("3. Creating search engine adapter...")
+    search_engine = CustomSearchEngine(custom_retriever, settings)
+
+    # 5. Initialize the search system
+    print("4. Initializing AdvancedSearchSystem with custom components...")
+    # Pass programmatic_mode=True to avoid database dependencies
+    search_system = AdvancedSearchSystem(
+        llm=llm,
+        search=search_engine,
+        settings_snapshot=settings,
+        programmatic_mode=True,
+    )
+
+    # 6. Run research queries
+    queries = [
+        "How do quantum computers differ from classical computers?",
+        "What are the main types of machine learning algorithms?",
+    ]
+
+    for query in queries:
+        print(f"\n{'=' * 60}")
+        print(f"Research Query: {query}")
+        print("=" * 60)
+
+        result = search_system.analyze_topic(query)
+
+        # Display results
+        print("\n=== FINDINGS ===")
+        print(result["formatted_findings"])
+
+        # Show metadata
+        print("\n=== SEARCH METADATA ===")
+        print(f"• Total findings: {len(result['findings'])}")
+        print(f"• Iterations: {result['iterations']}")
+
+        # Get actual sources from all_links_of_system or search_results
+        all_links = result.get("all_links_of_system", [])
+        for finding in result.get("findings", []):
+            if "search_results" in finding and finding["search_results"]:
+                all_links = finding["search_results"]
+                break
+
+        print(f"• Sources found: {len(all_links)}")
+        if all_links and len(all_links) > 0:
+            print("\n=== SOURCES ===")
+            for i, link in enumerate(all_links[:5], 1):  # Show first 5
+                if isinstance(link, dict):
+                    title = link.get("title", "No title")
+                    url = link.get("link", link.get("source", "Unknown"))
+                    print(f"  [{i}] {title}")
+                    print(f"      URL: {url}")
+
+        # Show generated questions
+        if result.get("questions_by_iteration"):
+            print("\n=== RESEARCH QUESTIONS GENERATED ===")
+            for iteration, questions in result[
+                "questions_by_iteration"
+            ].items():
+                print(f"\nIteration {iteration}:")
+                for q in questions[:3]:  # Show first 3 questions
+                    print(f"  • {q}")
+
+    print("\n✓ Custom LLM and Retriever integration successful!")
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/hybrid_search_example.py

@@ -0,0 +1,352 @@
+#!/usr/bin/env python3
+"""
+Hybrid Search Example for Local Deep Research
+
+This example demonstrates how to combine multiple search sources:
+1. Multiple named retrievers for different document types
+2. Combining custom retrievers with web search
+3. Analyzing and comparing sources from different origins
+"""
+
+from typing import List
+from langchain_core.retrievers import Document, BaseRetriever
+from langchain_community.vectorstores import FAISS
+from langchain_ollama import OllamaEmbeddings
+
+from local_deep_research.api import quick_summary, detailed_research
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+
+class TechnicalDocsRetriever(BaseRetriever):
+    """Mock retriever for technical documentation."""
+
+    def get_relevant_documents(self, query: str) -> List[Document]:
+        """Return mock technical documents."""
+        # In a real scenario, this would search actual technical docs
+        return [
+            Document(
+                page_content=f"Technical specification for {query}: Implementation requires careful consideration of system architecture, performance metrics, and scalability factors.",
+                metadata={
+                    "source": "tech_docs",
+                    "type": "specification",
+                    "title": f"Technical Spec: {query}",
+                },
+            ),
+            Document(
+                page_content=f"Best practices for {query}: Follow industry standards, implement proper error handling, and ensure comprehensive testing coverage.",
+                metadata={
+                    "source": "tech_docs",
+                    "type": "best_practices",
+                    "title": f"Best Practices: {query}",
+                },
+            ),
+        ]
+
+    async def aget_relevant_documents(self, query: str) -> List[Document]:
+        """Async version."""
+        return self.get_relevant_documents(query)
+
+
+class BusinessDocsRetriever(BaseRetriever):
+    """Mock retriever for business/strategy documents."""
+
+    def get_relevant_documents(self, query: str) -> List[Document]:
+        """Return mock business documents."""
+        return [
+            Document(
+                page_content=f"Business implications of {query}: Consider market impact, ROI analysis, and strategic alignment with organizational goals.",
+                metadata={
+                    "source": "business_docs",
+                    "type": "strategy",
+                    "title": f"Business Strategy: {query}",
+                },
+            ),
+            Document(
+                page_content=f"Cost-benefit analysis for {query}: Initial investment requirements, expected returns, and risk assessment factors.",
+                metadata={
+                    "source": "business_docs",
+                    "type": "analysis",
+                    "title": f"Cost Analysis: {query}",
+                },
+            ),
+        ]
+
+    async def aget_relevant_documents(self, query: str) -> List[Document]:
+        """Async version."""
+        return self.get_relevant_documents(query)
+
+
+def create_knowledge_base_retriever() -> BaseRetriever:
+    """Create a FAISS-based retriever with sample knowledge base documents."""
+    documents = [
+        Document(
+            page_content="Machine learning models require training data, validation strategies, and performance metrics for evaluation.",
+            metadata={"source": "ml_knowledge_base", "topic": "ml_basics"},
+        ),
+        Document(
+            page_content="Cloud computing provides scalable infrastructure, reducing capital expenditure and enabling flexible resource allocation.",
+            metadata={
+                "source": "cloud_knowledge_base",
+                "topic": "cloud_benefits",
+            },
+        ),
+        Document(
+            page_content="Agile methodology emphasizes iterative development, customer collaboration, and responding to change.",
+            metadata={"source": "project_knowledge_base", "topic": "agile"},
+        ),
+        Document(
+            page_content="Data privacy regulations like GDPR require explicit consent, data minimization, and user rights management.",
+            metadata={
+                "source": "compliance_knowledge_base",
+                "topic": "privacy",
+            },
+        ),
+    ]
+
+    # Create embeddings and vector store
+    embeddings = OllamaEmbeddings(model="nomic-embed-text")
+    vectorstore = FAISS.from_documents(documents, embeddings)
+    return vectorstore.as_retriever(search_kwargs={"k": 2})
+
+
+def demonstrate_multiple_retrievers():
+    """Show how to use multiple named retrievers for different document types."""
+    print("=" * 70)
+    print("MULTIPLE NAMED RETRIEVERS")
+    print("=" * 70)
+    print("""
+Using multiple specialized retrievers:
+- Technical documentation retriever
+- Business documentation retriever
+- Knowledge base retriever
+Each provides different perspectives on the same topic.
+    """)
+
+    # Create different retrievers
+    tech_retriever = TechnicalDocsRetriever()
+    business_retriever = BusinessDocsRetriever()
+    kb_retriever = create_knowledge_base_retriever()
+
+    # Configure settings. Registered retrievers are addressable by name;
+    # with the default langgraph-agent strategy, every registered retriever
+    # is also exposed to the research agent as a search tool.
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "knowledge_base",  # Primary retriever
+        }
+    )
+
+    # Use multiple retrievers in research
+    result = quick_summary(
+        query="Implementing machine learning in production",
+        settings_snapshot=settings,
+        retrievers={
+            "technical": tech_retriever,
+            "business": business_retriever,
+            "knowledge_base": kb_retriever,
+        },
+        search_tool="knowledge_base",  # Primary retriever (others stay available)
+        iterations=2,
+        questions_per_iteration=2,
+        programmatic_mode=True,
+    )
+
+    print("\nResearch Summary (first 400 chars):")
+    print(result["summary"][:400] + "...")
+
+    # Analyze sources by type
+    sources = result.get("sources", [])
+    print(f"\nTotal sources found: {len(sources)}")
+
+    # Group sources by retriever
+    source_types = {}
+    for source in sources:
+        if isinstance(source, dict):
+            source_type = source.get("metadata", {}).get("source", "unknown")
+        else:
+            source_type = "other"
+        source_types[source_type] = source_types.get(source_type, 0) + 1
+
+    print("\nSources by retriever:")
+    for stype, count in source_types.items():
+        print(f"  - {stype}: {count} sources")
+
+    return result
+
+
+def demonstrate_retriever_plus_web():
+    """Show how to combine custom retrievers with web search."""
+    print("\n" + "=" * 70)
+    print("RETRIEVER + WEB SEARCH COMBINATION")
+    print("=" * 70)
+    print("""
+Combining internal knowledge with web search:
+- Internal: Custom retriever with proprietary knowledge
+- External: Wikipedia for general context
+This provides both specific and general information.
+    """)
+
+    # Create internal knowledge retriever
+    internal_retriever = create_knowledge_base_retriever()
+
+    # Configure settings to use both retriever and web
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "wikipedia",  # Also use Wikipedia
+        }
+    )
+
+    # Research combining internal and external sources
+    result = detailed_research(
+        query="Best practices for cloud migration",
+        settings_snapshot=settings,
+        retrievers={
+            "internal_kb": internal_retriever,
+        },
+        search_tool="wikipedia",  # Also search Wikipedia
+        search_strategy="source-based",
+        iterations=2,
+        questions_per_iteration=3,
+        programmatic_mode=True,
+    )
+
+    print(f"\nResearch ID: {result['research_id']}")
+    print(f"Summary length: {len(result['summary'])} characters")
+
+    # Analyze source distribution
+    sources = result.get("sources", [])
+    internal_sources = 0
+    external_sources = 0
+
+    for source in sources:
+        if isinstance(source, dict) and "knowledge_base" in str(source):
+            internal_sources += 1
+        else:
+            external_sources += 1
+
+    print("\nSource distribution:")
+    print(f"  - Internal knowledge base: {internal_sources} sources")
+    print(f"  - External (Wikipedia): {external_sources} sources")
+
+    # Show how different sources complement each other
+    print("\nComplementary insights from hybrid search:")
+    print(
+        "  - Internal sources provide: Specific procedures, proprietary knowledge"
+    )
+    print(
+        "  - External sources provide: Industry context, general best practices"
+    )
+
+    return result
+
+
+def demonstrate_source_analysis():
+    """Show how to analyze and compare sources from different origins."""
+    print("\n" + "=" * 70)
+    print("SOURCE ANALYSIS AND COMPARISON")
+    print("=" * 70)
+    print("""
+Analyzing source quality and relevance:
+- Track source origins
+- Compare information consistency
+- Identify unique insights from each source type
+    """)
+
+    # Create multiple retrievers
+    tech_retriever = TechnicalDocsRetriever()
+    business_retriever = BusinessDocsRetriever()
+
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Run research with detailed source tracking
+    result = quick_summary(
+        query="Artificial intelligence implementation strategies",
+        settings_snapshot=settings,
+        retrievers={
+            "technical": tech_retriever,
+            "business": business_retriever,
+        },
+        search_tool="wikipedia",  # Also use web search
+        iterations=2,
+        questions_per_iteration=2,
+        programmatic_mode=True,
+    )
+
+    # Detailed source analysis
+    print("\nSource Analysis:")
+    sources = result.get("sources", [])
+
+    # Categorize sources
+    source_categories = {"technical": [], "business": [], "web": []}
+
+    for source in sources:
+        if isinstance(source, dict):
+            source_type = source.get("metadata", {}).get("source", "")
+            if "tech" in source_type:
+                source_categories["technical"].append(source)
+            elif "business" in source_type:
+                source_categories["business"].append(source)
+            else:
+                source_categories["web"].append(source)
+        else:
+            source_categories["web"].append(source)
+
+    # Report on each category
+    for category, category_sources in source_categories.items():
+        print(f"\n{category.upper()} Sources ({len(category_sources)}):")
+        if category_sources:
+            for i, source in enumerate(category_sources[:2], 1):  # Show first 2
+                if isinstance(source, dict):
+                    title = source.get("metadata", {}).get("title", "Untitled")
+                    print(f"  {i}. {title}")
+                else:
+                    print(f"  {i}. {str(source)[:60]}...")
+
+    # Show findings breakdown
+    findings = result.get("findings", [])
+    print(f"\nTotal findings: {len(findings)}")
+    print("Findings provide integrated insights from all source types")
+
+    return result
+
+
+def main():
+    """Run all hybrid search demonstrations."""
+    print("=" * 70)
+    print("LOCAL DEEP RESEARCH - HYBRID SEARCH DEMONSTRATION")
+    print("=" * 70)
+    print("""
+This example shows how to combine multiple search sources:
+- Custom retrievers for proprietary knowledge
+- Web search engines for public information
+- Source analysis across origins
+    """)
+
+    # Run demonstrations
+    demonstrate_multiple_retrievers()
+    demonstrate_retriever_plus_web()
+    demonstrate_source_analysis()
+
+    print("\n" + "=" * 70)
+    print("KEY TAKEAWAYS")
+    print("=" * 70)
+    print("""
+1. Multiple Retrievers: Use specialized retrievers for different document types
+2. Hybrid Search: Combine internal knowledge with web search for comprehensive results
+3. Source Analysis: Track and analyze sources to understand information origin
+
+Best Practices:
+- Name your retrievers descriptively for easy tracking
+- Balance internal and external sources based on your needs
+- Use source analysis to verify information consistency
+    """)
+
+    print("\n✓ Hybrid search demonstration complete!")
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/minimal_working_example.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""
+Minimal working example for programmatic access to Local Deep Research.
+
+This shows how to use the core functionality without database dependencies.
+"""
+
+from langchain_ollama import ChatOllama
+from local_deep_research.search_system import AdvancedSearchSystem
+
+# Re-enable logging after import (it gets disabled in __init__.py)
+from loguru import logger
+import sys
+
+logger.remove()
+# diagnose=False: loguru defaults to True, which renders repr() of every
+# local in every traceback frame on exception. Users copy this snippet
+# into their own scripts, so leaving the default on would propagate the
+# credential-in-traceback leak (#4185) wherever the snippet lands.
+logger.add(
+    sys.stderr,
+    level="WARNING",
+    format="{time} {level} {message}",
+    diagnose=False,
+)
+logger.enable("local_deep_research")
+
+
+class MinimalSearchEngine:
+    """Minimal search engine that returns hardcoded results."""
+
+    def __init__(self, settings_snapshot=None):
+        self.settings_snapshot = settings_snapshot or {}
+
+    def run(self, query, research_context=None):
+        """Return some fake search results."""
+        return [
+            {
+                "title": "Introduction to AI",
+                "link": "https://example.com/ai-intro",
+                "snippet": "Artificial Intelligence (AI) is the simulation of human intelligence...",
+                "full_content": "Full article about AI basics...",
+                "rank": 1,
+            },
+            {
+                "title": "Machine Learning Explained",
+                "link": "https://example.com/ml-explained",
+                "snippet": "Machine learning is a subset of AI that enables systems to learn...",
+                "full_content": "Detailed explanation of machine learning...",
+                "rank": 2,
+            },
+        ]
+
+
+def main():
+    """Minimal example of programmatic access."""
+    print("=== Minimal Local Deep Research Example ===\n")
+
+    # 1. Create LLM
+    print("1. Creating Ollama LLM...")
+    llm = ChatOllama(model="gemma3:12b")
+
+    # 2. Create minimal search engine
+    print("2. Creating minimal search engine...")
+
+    # Settings for search system (without programmatic_mode)
+    settings = {
+        "search.iterations": 1,
+        "search.strategy": "direct",
+    }
+
+    search = MinimalSearchEngine(settings)
+
+    # 3. Create search system
+    print("3. Creating AdvancedSearchSystem...")
+    # IMPORTANT: Pass programmatic_mode=True to avoid database dependencies
+    system = AdvancedSearchSystem(
+        llm=llm,
+        search=search,
+        settings_snapshot=settings,
+        programmatic_mode=True,
+    )
+
+    # 4. Run a search
+    print("\n4. Running search...")
+    result = system.analyze_topic("What is artificial intelligence?")
+
+    # 5. Show results
+    print("\n=== RESULTS ===")
+    print(f"Found {len(result['findings'])} findings")
+    print(f"\nSummary:\n{result['current_knowledge']}")
+
+    print("\n✓ Success! Programmatic access works without database.")
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/search_strategies_example.py

@@ -0,0 +1,225 @@
+#!/usr/bin/env python3
+"""
+Search Strategies Example for Local Deep Research
+
+This example demonstrates the two main search strategies:
+1. source-based: Comprehensive research with source citation
+2. focused-iteration: Iterative refinement of research questions
+
+Each strategy has different strengths and use cases.
+"""
+
+from local_deep_research.api import quick_summary, detailed_research
+from local_deep_research.api.settings_utils import create_settings_snapshot
+
+
+def demonstrate_source_based_strategy():
+    """
+    Source-based strategy:
+    - Focuses on gathering and synthesizing information from multiple sources
+    - Provides detailed citations and source tracking
+    - Best for: Academic research, fact-checking, comprehensive reports
+    """
+    print("=" * 70)
+    print("SOURCE-BASED STRATEGY")
+    print("=" * 70)
+    print("""
+This strategy:
+- Systematically searches for sources related to your topic
+- Synthesizes information across multiple sources
+- Provides detailed citations for all claims
+- Ideal for research requiring source verification
+    """)
+
+    # Configure settings for programmatic mode
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "wikipedia",  # Using Wikipedia for demonstration
+        }
+    )
+
+    # Run research with source-based strategy
+    result = detailed_research(
+        query="What are the main causes of climate change?",
+        settings_snapshot=settings,
+        search_strategy="source-based",  # Explicitly set strategy
+        iterations=2,  # Number of research iterations
+        questions_per_iteration=3,  # Questions to explore per iteration
+        programmatic_mode=True,
+    )
+
+    print(f"Research ID: {result['research_id']}")
+    print("\nSummary (first 500 chars):")
+    print(result["summary"][:500] + "...")
+
+    # Show sources found
+    sources = result.get("sources", [])
+    print(f"\nSources found: {len(sources)}")
+    if sources:
+        print("\nFirst 3 sources:")
+        for i, source in enumerate(sources[:3], 1):
+            print(f"  {i}. {source}")
+
+    # Show the questions that were researched
+    questions = result.get("questions", {})
+    print(f"\nQuestions researched across {len(questions)} iterations:")
+    for iteration, qs in questions.items():
+        print(f"\n  Iteration {iteration}:")
+        for q in qs[:2]:  # Show first 2 questions per iteration
+            print(f"    - {q}")
+
+    return result
+
+
+def demonstrate_focused_iteration_strategy():
+    """
+    Focused-iteration strategy:
+    - Iteratively refines the research based on previous findings
+    - Adapts questions based on what's been learned
+    - Best for: Deep dives, evolving research questions, exploratory research
+    """
+    print("\n" + "=" * 70)
+    print("FOCUSED-ITERATION STRATEGY")
+    print("=" * 70)
+    print("""
+This strategy:
+- Starts with initial research on the topic
+- Analyzes findings to generate more targeted questions
+- Iteratively refines understanding through multiple rounds
+- Ideal for complex topics requiring deep exploration
+    """)
+
+    # Configure settings
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Run research with focused-iteration strategy
+    result = quick_summary(
+        query="How do neural networks learn?",
+        settings_snapshot=settings,
+        search_strategy="focused-iteration",  # Use focused iteration
+        iterations=3,  # More iterations for deeper exploration
+        questions_per_iteration=2,  # Fewer but more focused questions
+        temperature=0.7,  # Slightly higher for creative question generation
+        programmatic_mode=True,
+    )
+
+    print("\nSummary (first 500 chars):")
+    print(result["summary"][:500] + "...")
+
+    # Show how questions evolved
+    questions = result.get("questions", {})
+    if questions:
+        print("\nQuestion evolution across iterations:")
+        for iteration, qs in questions.items():
+            print(f"\n  Iteration {iteration}:")
+            for q in qs:
+                print(f"    - {q}")
+
+    # Show findings
+    findings = result.get("findings", [])
+    print(f"\nKey findings: {len(findings)}")
+    if findings:
+        print("\nFirst 2 findings:")
+        for i, finding in enumerate(findings[:2], 1):
+            text = (
+                finding.get("text", "N/A")
+                if isinstance(finding, dict)
+                else str(finding)
+            )
+            print(f"  {i}. {text[:150]}...")
+
+    return result
+
+
+def compare_strategies():
+    """
+    Direct comparison of both strategies on the same topic.
+    """
+    print("\n" + "=" * 70)
+    print("STRATEGY COMPARISON")
+    print("=" * 70)
+    print(
+        "\nComparing both strategies on the same topic: 'Quantum Computing Applications'\n"
+    )
+
+    settings = create_settings_snapshot(
+        {
+            "search.tool": "wikipedia",
+        }
+    )
+
+    # Same topic, different strategies
+    topic = "Quantum computing applications in cryptography"
+
+    print("1. Source-based approach:")
+    source_result = quick_summary(
+        query=topic,
+        settings_snapshot=settings,
+        search_strategy="source-based",
+        iterations=2,
+        questions_per_iteration=3,
+        programmatic_mode=True,
+    )
+    print(f"   - Sources found: {len(source_result.get('sources', []))}")
+    print(f"   - Summary length: {len(source_result.get('summary', ''))} chars")
+    print(f"   - Findings: {len(source_result.get('findings', []))}")
+
+    print("\n2. Focused-iteration approach:")
+    focused_result = quick_summary(
+        query=topic,
+        settings_snapshot=settings,
+        search_strategy="focused-iteration",
+        iterations=2,
+        questions_per_iteration=3,
+        programmatic_mode=True,
+    )
+    print(f"   - Sources found: {len(focused_result.get('sources', []))}")
+    print(
+        f"   - Summary length: {len(focused_result.get('summary', ''))} chars"
+    )
+    print(f"   - Findings: {len(focused_result.get('findings', []))}")
+
+    print("\n" + "=" * 70)
+    print("WHEN TO USE EACH STRATEGY")
+    print("=" * 70)
+    print("""
+Use SOURCE-BASED when you need:
+- Comprehensive coverage with citations
+- Academic or professional research
+- Fact-checking and verification
+- Documentation with source tracking
+
+Use FOCUSED-ITERATION when you need:
+- Deep exploration of complex topics
+- Adaptive research that evolves
+- Discovery of unexpected connections
+- Exploratory or investigative research
+    """)
+
+
+def main():
+    """Run all demonstrations."""
+    print("=" * 70)
+    print("LOCAL DEEP RESEARCH - SEARCH STRATEGIES DEMONSTRATION")
+    print("=" * 70)
+
+    # Demonstrate each strategy
+    demonstrate_source_based_strategy()
+    demonstrate_focused_iteration_strategy()
+
+    # Compare strategies
+    compare_strategies()
+
+    print("\n✓ Search strategies demonstration complete!")
+    print("\nNote: Both strategies can be combined with different search tools")
+    print(
+        "(wikipedia, arxiv, searxng, etc.) and custom parameters for optimal results."
+    )
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/searxng_example.py

@@ -0,0 +1,185 @@
+#!/usr/bin/env python3
+"""
+Example of using SearXNG search engine with Local Deep Research.
+
+This demonstrates how to use SearXNG for web search in programmatic mode.
+Note: Requires a running SearXNG instance.
+"""
+
+import os
+from langchain_ollama import ChatOllama
+from local_deep_research.search_system import AdvancedSearchSystem
+from local_deep_research.web_search_engines.engines.search_engine_searxng import (
+    SearXNGSearchEngine,
+)
+
+# Re-enable logging
+from loguru import logger
+import sys
+
+logger.remove()
+# diagnose=False: loguru defaults to True, which renders repr() of every
+# local in every traceback frame on exception. Users copy this snippet
+# into their own scripts, so leaving the default on would propagate the
+# credential-in-traceback leak (#4185) wherever the snippet lands.
+logger.add(
+    sys.stderr,
+    level="INFO",
+    format="{time} {level} {message}",
+    diagnose=False,
+)
+logger.enable("local_deep_research")
+
+
+def main():
+    """Demonstrate using SearXNG with Local Deep Research."""
+    print("=== SearXNG Search Engine Example ===\n")
+
+    # Check if SearXNG URL is configured
+    searxng_url = os.getenv("SEARXNG_URL", "http://localhost:8080")
+    print(f"Using SearXNG instance at: {searxng_url}")
+    print(
+        "(Set SEARXNG_URL environment variable to use a different instance)\n"
+    )
+
+    # 1. Create LLM
+    print("1. Setting up Ollama LLM...")
+    llm = ChatOllama(model="gemma3:12b", temperature=0.3)
+
+    # 2. Configure settings
+    settings = {
+        "search.iterations": 2,
+        "search.questions_per_iteration": 3,
+        "search.strategy": "source-based",
+        "rate_limiting.enabled": False,  # Disable rate limiting for demo
+        # SearXNG specific settings
+        "search_engines.searxng.base_url": searxng_url,
+        "search_engines.searxng.timeout": 30,
+        "search_engines.searxng.categories": ["general", "science"],
+        "search_engines.searxng.engines": ["google", "duckduckgo", "bing"],
+        "search_engines.searxng.language": "en",
+        "search_engines.searxng.time_range": "",  # all time
+        "search_engines.searxng.safesearch": 0,  # 0=off, 1=moderate, 2=strict
+    }
+
+    # 3. Create SearXNG search engine
+    print("2. Initializing SearXNG search engine...")
+    try:
+        search_engine = SearXNGSearchEngine(settings_snapshot=settings)
+
+        # Test the connection
+        print("   Testing SearXNG connection...")
+        test_results = search_engine.run("test query", research_context={})
+        if test_results:
+            print(
+                f"   ✓ SearXNG is working! Got {len(test_results)} test results."
+            )
+        else:
+            print("   ⚠ SearXNG returned no results for test query.")
+    except Exception as e:
+        print(f"\n⚠ Error connecting to SearXNG: {e}")
+        print("\nPlease ensure SearXNG is running. You can start it with:")
+        print("  docker run -p 8888:8080 searxng/searxng")
+        print("\nFalling back to mock search engine for demonstration...")
+
+        # Fallback to mock search engine
+        class MockSearchEngine:
+            def __init__(self, settings_snapshot=None):
+                self.settings_snapshot = settings_snapshot or {}
+
+            def run(self, query, research_context=None):
+                return [
+                    {
+                        "title": f"Result for: {query}",
+                        "link": "https://example.com/result",
+                        "snippet": f"This is a mock result for the query: {query}. "
+                        "In a real scenario, SearXNG would provide actual web search results.",
+                        "full_content": "Full content would be fetched here...",
+                        "rank": 1,
+                    }
+                ]
+
+        search_engine = MockSearchEngine(settings)
+
+    # 4. Create the search system
+    print("3. Creating AdvancedSearchSystem...")
+    # Pass programmatic_mode=True to disable database dependencies
+    search_system = AdvancedSearchSystem(
+        llm=llm,
+        search=search_engine,
+        settings_snapshot=settings,
+        programmatic_mode=True,
+    )
+
+    # 5. Run research queries
+    queries = [
+        "What are the latest developments in quantum computing in 2024?",
+        "How does CRISPR gene editing technology work?",
+    ]
+
+    for query in queries:
+        print(f"\n{'=' * 60}")
+        print(f"Research Query: {query}")
+        print("=" * 60)
+
+        try:
+            result = search_system.analyze_topic(query)
+
+            # Display results
+            print("\n=== RESEARCH FINDINGS ===")
+            if result.get("formatted_findings"):
+                print(result["formatted_findings"])
+            else:
+                print(
+                    "Summary:", result.get("current_knowledge", "No findings")
+                )
+
+            # Show metadata
+            print("\n=== METADATA ===")
+            print(f"• Iterations completed: {result.get('iterations', 0)}")
+            print(f"• Total findings: {len(result.get('findings', []))}")
+
+            # Show search sources from all_links_of_system or search_results in findings
+            all_links = result.get("all_links_of_system", [])
+
+            # Also check findings for search_results
+            for finding in result.get("findings", []):
+                if "search_results" in finding and finding["search_results"]:
+                    all_links = finding["search_results"]
+                    break
+
+            if all_links:
+                print(f"• Sources found: {len(all_links)}")
+                for i, link in enumerate(
+                    all_links[:5], 1
+                ):  # Show first 5 sources
+                    if isinstance(link, dict):
+                        title = link.get("title", "No title")
+                        url = link.get("link", "Unknown")
+                        print(f"  [{i}] {title}")
+                        print(f"      {url}")
+
+            # Show generated questions
+            if result.get("questions_by_iteration"):
+                print("\n=== RESEARCH QUESTIONS ===")
+                for iteration, questions in result[
+                    "questions_by_iteration"
+                ].items():
+                    print(f"Iteration {iteration}:")
+                    for q in questions[
+                        :2
+                    ]:  # Show first 2 questions per iteration
+                        print(f"  • {q}")
+
+        except Exception as e:
+            logger.exception("Error during research")
+            print(f"\n⚠ Error: {e}")
+
+    print("\n✓ SearXNG integration example completed!")
+    print(
+        "\nNote: For best results, ensure SearXNG is properly configured with multiple search engines."
+    )
+
+
+if __name__ == "__main__":
+    main()

examples/api_usage/programmatic/simple_programmatic_example.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""
+Simple Programmatic API Example for Local Deep Research
+
+Quick example showing how to use the LDR Python API directly.
+"""
+
+from local_deep_research.api import (
+    detailed_research,
+    quick_summary,
+    generate_report,
+)
+from local_deep_research.api.settings_utils import (
+    create_settings_snapshot,
+)
+
+# Use default settings with minimal overrides
+# This provides all necessary settings with sensible defaults
+settings_snapshot = create_settings_snapshot(
+    overrides={
+        "search.tool": "wikipedia",  # Use Wikipedia for this example
+        "api.allow_file_output": True,  # Allow generate_report to save files
+    }
+)
+
+# Alternative: Use completely default settings
+# settings_snapshot = get_default_settings_snapshot()
+
+# Example 1: Quick Summary
+print("=== Quick Summary ===")
+result = quick_summary(
+    "What is machine learning?",
+    settings_snapshot=settings_snapshot,
+    programmatic_mode=True,
+)
+print(f"Summary: {result['summary'][:300]}...")
+print(f"Found {len(result.get('findings', []))} findings")
+
+# Example 2: Detailed Research with Custom Parameters
+print("\n=== Detailed Research ===")
+result = detailed_research(
+    query="Impact of climate change on agriculture",
+    iterations=2,
+    search_tool="wikipedia",
+    search_strategy="source_based",
+    settings_snapshot=settings_snapshot,
+    programmatic_mode=True,
+)
+print(f"Research ID: {result['research_id']}")
+print(f"Summary length: {len(result['summary'])} characters")
+print(f"Sources: {len(result.get('sources', []))}")
+
+# Example 3: Using Custom Search Parameters
+print("\n=== Custom Search Parameters ===")
+result = quick_summary(
+    query="renewable energy trends 2024",
+    search_tool="searxng",  # Recommended general-purpose engine
+    iterations=1,
+    questions_per_iteration=3,
+    temperature=0.5,  # Lower temperature for focused results
+    provider="openai_endpoint",  # Specify LLM provider
+    model_name="llama-3.3-70b-instruct",  # Specify model
+    settings_snapshot=settings_snapshot,
+    programmatic_mode=True,
+)
+print(f"Completed {result['iterations']} iterations")
+print(
+    f"Generated {sum(len(qs) for qs in result.get('questions', {}).values())} questions"
+)
+
+# Example 4: Generate and Save a Report
+print("\n=== Generate Report ===")
+print("Note: Report generation can take several minutes")
+
+# Generate a comprehensive report
+report = generate_report(
+    query="Future of artificial intelligence",
+    output_file="ai_future_report.md",  # Save directly to file
+    searches_per_section=2,
+    iterations=1,
+    settings_snapshot=settings_snapshot,  # Now works with programmatic mode!
+)
+print(f"Report saved to: {report.get('file_path', 'ai_future_report.md')}")
+print(f"Report length: {len(report['content'])} characters")
+print("Report preview (first 300 chars):")
+print(report["content"][:300] + "...")

examples/api_usage/programmatic/test_direct_import.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+"""Test importing search_system directly without going through __init__.py"""
+
+import sys
+
+# Try importing the search_system module directly
+try:
+    print("Attempting to import search_system module directly...")
+    from local_deep_research import search_system
+
+    print("✓ search_system module imported!")
+
+    # Now try to access AdvancedSearchSystem
+    print("\nTrying to access AdvancedSearchSystem class...")
+    AdvancedSearchSystem = search_system.AdvancedSearchSystem
+    print("✓ Got AdvancedSearchSystem class!")
+
+except Exception as e:
+    print(f"✗ Failed: {e}")
+    import traceback
+
+    traceback.print_exc()
+
+# Also try a more direct import
+try:
+    print("\nAttempting direct file import...")
+    sys.path.insert(0, "src")
+    print("✓ Direct import worked!")
+except Exception as e:
+    print(f"✗ Direct import failed: {e}")