siriusscan--sirius/documentation/dev/test

title, description, template, version, last_updated, author, tags, categories, difficulty, prerequisites, related_docs, dependencies, llm_context, search_keywords

title	description	template	version	last_updated	author	tags	categories	difficulty	prerequisites	related_docs	dependencies	llm_context	search_keywords
Documentation Testing	Comprehensive documentation testing system for Sirius project, providing automated validation of documentation quality, structure, and completeness using specialized linters.	TEMPLATE.documentation-standard	1.0.0	2025-01-03	Development Team	documentation testing linting validation quality	development testing documentation	intermediate	markdown yaml bash	README.container-testing.md ABOUT.documentation.md	scripts/documentation/ testing/Makefile	high	documentation testing linting validation yaml markdown

Documentation Testing

Purpose

This document describes the comprehensive documentation testing system for the Sirius project, providing automated validation of documentation quality, structure, and completeness. It serves as the definitive guide for developers and technical writers who need to ensure documentation meets our standards and is optimized for LLM consumption.

When to Use

• Before committing documentation changes - Validate that all documentation meets our standards
• During documentation reviews - Ensure consistency and completeness across all docs
• When creating new documentation - Verify proper structure and metadata
• In CI/CD pipelines - Automated validation of documentation quality
• When troubleshooting documentation issues - Use linters to identify specific problems
• For LLM optimization - Ensure documentation is properly structured for AI consumption

How to Use

Quick Start

# Navigate to container testing directory
cd testing/container-testing

# Run complete documentation testing
make lint-docs

# Run quick documentation checks
make lint-docs-quick

# Check documentation index completeness
make lint-index

# Run all validation including documentation
make validate-all

Prerequisites

• Bash shell environment
• yq utility (for full YAML validation)
• grep, sed, awk (standard Unix tools)
• Make utility
• Git (for pre-commit hooks)

Step-by-Step Process

1. Navigate to container testing directory: cd /path/to/Sirius/testing/container-testing
2. Run quick checks: make lint-docs-quick for basic validation
3. Run full linting: make lint-docs for comprehensive validation
4. Check index completeness: make lint-index to verify all files are indexed
5. Review results: Check output and logs in tmp/ directory

What It Is

Architecture Overview

The documentation testing system consists of three main components:

• YAML Front Matter Validation: Ensures all documents have complete and valid metadata
• Template Compliance Checking: Verifies documents follow their specified templates
• Index Completeness Validation: Ensures all documentation files are properly indexed

Technical Details

YAML Front Matter Validation

Purpose: Ensures all documentation files have complete and valid YAML front matter.

Components:

• scripts/documentation/lint-docs.sh - Main validation script
• scripts/documentation/lint-quick.sh - Quick validation script
• Validates required and optional fields
• Checks field value validity

Required Fields:

• title - Human-readable document title
• description - Brief purpose description
• template - Template used to create document
• version - Document version
• last_updated - Last modification date

Optional Fields:

• author - Document author
• tags - Searchable tags
• categories - Document categories
• difficulty - Complexity level (beginner, intermediate, advanced)
• prerequisites - Required knowledge
• related_docs - Related documentation
• dependencies - Required files/systems
• llm_context - LLM relevance level (high, medium, low)
• search_keywords - Search optimization

Validation Rules:

• YAML syntax must be valid
• Required fields must be present
• Field values must match valid options
• Template references must exist
• Related documents must exist

Template Compliance Checking

Purpose: Verifies that documents follow their specified template structure.

Components:

• Template file parsing and section extraction
• Document section comparison
• Missing section detection
• Custom document support

Supported Templates:

• TEMPLATE.documentation-standard - Standard documentation
• TEMPLATE.guide - Step-by-step guides
• TEMPLATE.troubleshooting - Problem-solving docs
• TEMPLATE.about - About documents
• TEMPLATE.reference - Technical specifications
• TEMPLATE.architecture - System design docs
• TEMPLATE.api - API documentation
• TEMPLATE.template - Template documents
• TEMPLATE.custom - Custom documents (no compliance checking)

Compliance Rules:

• Documents must have all required sections from their template
• Section headings must match exactly
• Custom documents are exempt from compliance checking
• Meta-documents (ABOUT files) are exempt from compliance checking

Index Completeness Validation

Purpose: Ensures all documentation files are properly referenced in the documentation index.

Components:

• scripts/documentation/lint-index.sh - Index validation script
• File discovery in documentation/dev/ directory
• Index parsing and link extraction
• Missing file detection
• Extra file detection

Validation Rules:

• All files in dev/ must be referenced in the index
• All referenced files must exist in dev/
• No duplicate references allowed
• Footer links are excluded from validation

Implementation Details

File Structure

scripts/documentation/
├── lint-docs.sh              # Full documentation linting
├── lint-quick.sh             # Quick documentation checks
└── lint-index.sh             # Index completeness validation

testing/
├── Makefile                  # Testing commands
└── tmp/                      # Linting logs and output
    ├── documentation-lint-*.log
    └── index-lint-*.log

Makefile Integration

The documentation testing system integrates with the testing Makefile:

# Documentation testing targets
lint-docs: ../scripts/documentation/lint-docs.sh
lint-docs-quick: ../scripts/documentation/lint-quick.sh
lint-index: ../scripts/documentation/lint-index.sh

# Full validation including documentation
validate-all: build-all test-all lint-docs

Logging and Reporting

Log Files: All linting executions create timestamped log files in tmp/

• Format: {lint_type}-{timestamp}.log
• Contains: Full validation output, errors, warnings
• Retention: Logs are kept for troubleshooting

Validation Results: Each validation provides clear indicators:

• ✅ PASSED: Validation completed successfully
• ❌ FAILED: Validation failed with error details
• ⚠️ WARNING: Non-critical issues found
• Summary: Total files, passed, failed counts

Pre-commit Integration

Pre-commit Hook: .git/hooks/pre-commit runs documentation validation:

1. Quick documentation linting - Basic validation
2. Index completeness check - Ensures all files are indexed
3. Build tests - Validates system still works

Hook Behavior:

• Runs from project root or testing directory
• Fails commit if validation fails
• Provides clear error messages
• Suggests fixes for common issues

Advanced Topics

Custom Validation Rules

Environment Variables:

export DOCS_DIR="../documentation/dev"     # Documentation directory
export TEMPLATES_DIR="../documentation/dev/templates"  # Templates directory
export LOG_LEVEL="debug"                   # Logging verbosity

Custom Validation:

# Run with custom documentation directory
DOCS_DIR="../custom-docs" make lint-docs

# Run with debug logging
LOG_LEVEL="debug" make lint-docs

# Run with custom template directory
TEMPLATES_DIR="../custom-templates" make lint-docs

Template Development

Creating New Templates:

1. Copy base template: cp TEMPLATE.template.md TEMPLATE.new-type.md
2. Define structure: Add required sections for new document type
3. Update validation: Add new template to VALID_TEMPLATES array
4. Test validation: Run make lint-docs to verify template works

Template Requirements:

• Must include YAML front matter
• Must define clear section structure
• Must be self-documenting
• Must follow naming conventions

LLM Optimization

Context Building:

• Rich metadata in front matter
• Consistent structure across documents
• Clear relationships between documents
• Comprehensive search keywords

AI-Friendly Features:

• Structured content for predictable parsing
• Detailed descriptions for AI understanding
• Working examples with expected outputs
• Comprehensive troubleshooting information

Troubleshooting

FAQ

Q: Why do I get "Template file not found" warnings? A: The quick linter looks for template files in the wrong location. This is a known issue and doesn't affect functionality. Use make lint-docs for accurate validation.

Q: Why does index linting fail with "Missing from index" errors? A: Some documentation files are not referenced in the documentation index. Add them to documentation/README.documentation-index.md or remove unused files.

Q: How do I fix "Invalid difficulty value" errors? A: Difficulty must be one of: "beginner", "intermediate", "advanced". Check your YAML front matter for typos.

Q: Why do I get "Missing section" warnings? A: Your document is missing required sections from its template. Compare your document structure with the template file.

Q: How do I add a new document type? A: Create a new template file, add it to the validation system, and update the documentation index.

Q: Why does pre-commit hook fail? A: Check the error messages in the terminal. Common issues include missing YAML front matter, invalid field values, or missing index entries.

Q: How do I disable validation for a specific document? A: Use TEMPLATE.custom template type, which skips template compliance checking.

Q: Can I run validation on specific files only? A: Modify the linting scripts to target specific files, or use grep to filter the file list.

Command Reference

# Complete documentation validation
make lint-docs

# Quick documentation checks
make lint-docs-quick

# Index completeness validation
make lint-index

# Full system validation
make validate-all

# Manual script execution
../scripts/documentation/lint-docs.sh
../scripts/documentation/lint-quick.sh
../scripts/documentation/lint-index.sh

# Check specific validation
grep -r "template:" ../documentation/dev/
grep -r "llm_context: high" ../documentation/dev/
grep -r "difficulty:" ../documentation/dev/

# Debug validation issues
LOG_LEVEL=debug make lint-docs
tail -f tmp/documentation-lint-*.log

Common Issues and Solutions

Issue	Symptoms	Solution
Missing front matter	"No YAML metadata" errors	Add complete YAML front matter with required fields
Invalid field values	"Invalid difficulty value" errors	Check field values against valid options
Template compliance	"Missing section" warnings	Compare document structure with template
Index completeness	"Missing from index" errors	Add files to documentation index
Broken links	"File not found" errors	Fix file paths in related_docs
YAML syntax errors	"YAML parse error" messages	Fix YAML syntax in front matter

Debugging Steps

1. Check log files: Review timestamped logs in tmp/ directory
2. Run quick validation: Use make lint-docs-quick for basic checks
3. Validate YAML syntax: Check front matter for syntax errors
4. Compare with templates: Ensure document follows template structure
5. Check index entries: Verify all files are properly indexed
6. Test individual files: Run validation on specific files

Log Analysis

Documentation Lint Logs:

# View validation results
tail -f tmp/documentation-lint-*.log

# Check for specific errors
grep -i "error\|failed" tmp/documentation-lint-*.log

# Find warnings
grep -i "warning" tmp/documentation-lint-*.log

Index Lint Logs:

# View index validation results
cat tmp/index-lint-*.log

# Check missing files
grep "missing from index" tmp/index-lint-*.log

# Check extra files
grep "extra files in index" tmp/index-lint-*.log

Performance Optimization

Faster Validation:

• Use make lint-docs-quick for development
• Run validation on specific files only
• Optimize template file parsing
• Cache validation results

Resource Usage:

• Monitor disk space for log files
• Clean up old log files regularly
• Use efficient file parsing methods
• Optimize regex patterns

Lessons Learned

2025-01-03: Implemented comprehensive documentation testing system to ensure consistency and quality. Key insight: Automated validation prevents documentation drift and improves maintainability.

2025-01-03: Created template compliance checking with support for custom documents. Lesson: Flexibility is important for meta-documents and special cases.

2025-01-03: Established index completeness validation to ensure all documentation is discoverable. Benefit: Prevents orphaned documentation and improves navigation.

2025-01-03: Integrated documentation testing with pre-commit hooks for automated quality assurance. Advantage: Catches documentation issues before they reach the repository.

LLM Context

[Additional context specifically for Large Language Models:]

• Key Concepts: Documentation testing involves YAML validation, template compliance, and index completeness to ensure documentation quality and consistency
• Technical Context: Uses shell scripts for automation, YAML parsing for metadata validation, and regex for content analysis. Tests cover 14 documentation files with 8 different templates
• Common Patterns: Layered validation approach (quick → full → index), automated cleanup, timestamped logging, pre-commit integration
• Edge Cases: Custom documents exempt from template compliance, meta-documents with unique structure, footer links excluded from index validation
• Integration Points: Connects with git hooks, CI/CD pipelines, development workflows, and documentation maintenance processes

---

This document follows the Sirius Documentation Standard. For questions about documentation structure, see ABOUT.documentation.md.