chore: import upstream snapshot with attribution

.claude/commands/create-pr.md

@@ -0,0 +1,30 @@
+<!--
+WHITELIST_AFTER_APPROVAL:
+- Bash(git push:*)
+- Bash(gh pr create:*)
+-->
+
+Please create a pull request with the changes on the current branch. $ARGUMENTS
+
+**IMPORTANT: Use planning mode for this command.**
+
+Follow these steps:
+
+## Planning Phase (Research Only):
+1. Check git status to ensure working tree is clean
+2. Verify current branch is not main/master
+3. Check if current branch tracks a remote branch and is up to date
+4. Run `git log --oneline main..HEAD` to see commits that will be included in PR
+5. Run `git diff main...HEAD` to understand the full scope of changes
+6. Analyze all changes and commits to create a comprehensive PR summary
+7. Draft PR title and description with:
+   - Title: Clear, descriptive title based on the changes
+   - Body: Include "## Summary" with bullet points, "## Test plan" checklist, and Claude signature
+8. **Present the draft PR title and description using ExitPlanMode tool for user approval**
+
+## Execution Phase (After User Approval):
+9. Push branch to remote with upstream tracking if needed
+10. Create PR using `gh pr create` with the approved title and description
+11. Return the PR URL for easy access
+
+Use the GitHub CLI (`gh`) for all GitHub-related operations. If $ARGUMENTS are provided, incorporate them as hints for the PR title or description.

.claude/commands/fix-github-issue.md

@@ -0,0 +1,14 @@
+Please analyze and fix the GitHub issue: $ARGUMENTS.
+
+Follow these steps:
+
+1. Use `gh issue view` to get the issue details
+2. Understand the problem described in the issue
+3. Search the codebase for relevant files
+4. Implement the necessary changes to fix the issue
+5. Write and run tests to verify the fix
+6. Ensure code passes linting and type checking
+7. Create a descriptive commit message
+8. Push and create a PR
+
+Remember to use the GitHub CLI (`gh`) for all GitHub-related tasks.

.cursor/commands/git-pr.md

@@ -0,0 +1,4 @@
+Make sure you are on a branch other than main. 
+Commit changes. 
+Run make format. 
+Create PR using gh cli following .github/pull_request_template.md template

.cursor/commands/update-howto-guide.md

@@ -0,0 +1,185 @@
+# Update How-to Guide
+
+Updates the mentioned how-to guide to use src/ragas/metrics/collections API instead of the legacy ragas/metrics API and LLM factory pattern instead of langchainwrapper.
+
+## File Format Decision
+
+If the source is an `.ipynb` file (or if the `.md` filename starts with `_`, indicating it's derived from a notebook via `docs/ipynb_to_md.py`):
+
+1. **Delete** the `.ipynb` file
+2. **Delete** the corresponding `_xxx.md` file (if it exists)
+3. **Create** a new `.md` file directly (without the `_` prefix)
+
+This simplifies maintenance by having pure markdown docs instead of notebooks.
+
+
+## Process
+
+### Phase 1: Research (do NOT make changes yet)
+
+Refer pr-description-customizations.md for the list of guides that are already updated. And finally update the doc after you're done. 
+
+#### 1.1 Understand the Guide's Purpose
+- Read the target file thoroughly
+- Identify **what the guide is trying to achieve** (e.g., caching, run config, retry handling)
+- Note the specific use case or need the guide addresses
+- Understand what underlying tools/libraries are being used (e.g., instructor, liteLLM, httpx)
+
+#### 1.2 Feasibility Check
+Before doing anything else, check if the feature works with the new API:
+
+1. **Check `src/ragas/experiment.py`** - Does experiment() support this feature?
+2. **Check `src/ragas/evaluation.py`** - Is this an evaluate()-only feature?
+3. **Check `src/ragas/metrics/collections/`** - Do collections metrics support this?
+4. **Check if simpler alternatives exist** - Does a newer, simpler API make this guide obsolete? (e.g., decorator-based metrics vs subclassing, built-in features vs manual workarounds). Check concept docs and `src/ragas/metrics/` for modern patterns.
+
+**If a simpler approach exists → recommend deletion** instead of migration. See "When to Recommend Deletion" section.
+
+**If not supported in new API → STOP immediately:**
+- Keep guide as-is
+- Output this Slack message for the team:
+
+```
+📋 *Doc Update Skipped*: `<guide_path>`
+*Link*: https://docs.ragas.io/en/latest/<guide_path_without_extension>/
+*Reason*: <feature> only works with legacy `evaluate()` API, not yet supported in `experiment()`/collections
+*Action*: Keep as-is until collections API adds support
+```
+
+**If supported → continue to 1.3**
+
+#### 1.3 Present Plan & Wait for Approval
+
+**⏸️ STOP HERE - Do NOT proceed to Phase 2 without explicit user approval.**
+
+Present a clear summary:
+1. **Current state**: What the guide currently does and how
+2. **Proposed changes**: 
+   - Imports to change (from old → new)
+   - LLM/embeddings setup patterns to update
+   - **How the specific use case/feature will be achieved** with the new API
+   - Any restructuring or content changes
+3. **Potential concerns**: Anything uncertain or risky
+4. **Ask**: "Does this plan look good? Should I proceed?"
+
+**Wait for user to say "yes", "proceed", "go ahead", or similar before continuing.**
+
+---
+
+### Phase 2: Execute (only after approval)
+
+#### 2.1 Apply Updates
+
+**Keep it Concise**: 
+- Remove unnecessary explanations and verbose text
+- Focus on the essential information needed to achieve the goal
+- Use clear, direct language
+- Avoid redundant examples - one good example is better than multiple similar ones
+
+**Import Updates**:
+```python
+# Change from:
+from ragas.metrics import MetricName
+
+# To:
+from ragas.metrics.collections import MetricName
+```
+
+**LLM Setup**:
+```python
+# Change from:
+from langchain_openai import ChatOpenAI
+from ragas.llms import LangchainLLMWrapper
+llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
+
+# To:
+from openai import OpenAI
+from ragas.llms import llm_factory
+client = OpenAI(api_key="sk-...")
+llm = llm_factory("gpt-4o", client=client)
+```
+
+**Embeddings Setup**:
+```python
+# Change from:
+from langchain_openai import OpenAIEmbeddings
+from ragas.embeddings import LangchainEmbeddingsWrapper
+embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())
+
+# To:
+from openai import OpenAI
+from ragas.embeddings.base import embedding_factory  # Use .base to avoid deprecation warning
+client = OpenAI(api_key="sk-...")
+embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client)
+```
+
+**What to Fix**:
+- Update imports and LLM/embeddings patterns
+- Use `ragas.embeddings.base` import to avoid deprecation warnings
+- **Replace all legacy code with modern approaches** (no need to keep legacy sections)
+- Fix minor issues automatically
+- Don't restructure content unless fixing issues
+
+#### 2.2 Verify Accuracy & Test Code
+
+**Verify with Web Search**:
+- Search for official documentation of any libraries/tools mentioned (instructor, liteLLM, httpx, etc.)
+- Confirm API signatures, parameter names, and usage patterns are correct
+- Verify any claims about library behavior are accurate
+
+**Run the Code**:
+- Install any missing packages first: `uv pip install <package>`
+- Extract ALL Python code blocks from the guide
+- Save as `tests/docs/test_<guide_name>.py` (e.g., `test_run_config.py`)
+- Use `.env` from root for API keys. .env only has openai keys, if you need anything else, let me know
+- Run: `uv run python tests/docs/test_<guide_name>.py`
+- **Verify the original use case/goal is achieved** with the new approach
+
+**If tests fail**:
+1. Check the underlying implementation in `src/` to understand correct usage
+2. Fix the code in the guide based on what you learn from `src/`
+3. Re-run the test
+4. Repeat until tests pass
+5. If stuck after multiple attempts, report the issue with details
+
+**Keep the test file** - excluded from default `pytest` runs via `norecursedirs` in `pyproject.toml`.
+
+**Both verification methods are required** - web search for accuracy, code execution for functionality.
+
+#### 2.3 Check Navigation
+- Verify file is in `mkdocs.yml`
+- Note if location seems wrong or can be put in a more appropriate section
+
+#### 2.4 Summarize Changes
+- List all changes made
+- Mention if anything is not tested due to any reasons (like missing packages, missing API keys, etc.)
+
+## Notes
+- **Two-phase workflow**: Research first, get approval, then execute
+- **Never skip approval**: Always present the plan and wait for explicit "go ahead" before making changes
+- **This is not just a straightforward migration** - understand if the original goal is achievable first
+- **Keep guides concise** - remove fluff, focus on essential information
+- **Verify accuracy** - use web search to confirm library APIs and behavior before writing
+- **Test everything** - run all code examples before finalizing
+- Only fix what's broken or outdated
+- Check `src/` before updating to verify APIs exist
+- Don't add legacy sections
+- Use root `.env` for testing
+- **Keep test files** in `tests/docs/` - excluded from default pytest runs
+
+## When to Recommend Deletion
+
+If a guide teaches **writing custom metrics by subclassing** (`MetricWithLLM`, `SingleTurnMetric`, etc.), it's likely obsolete. The decorator-based approach is simpler:
+
+```python
+from ragas.metrics import discrete_metric, numeric_metric, ranking_metric
+
+@discrete_metric(name="my_metric", allowed_values=["pass", "fail"])
+def my_metric(response: str, context: str) -> str:
+    return "pass" if condition else "fail"
+```
+
+See `docs/concepts/metrics/overview/index.md` for details. Recommend deletion if decorators cover the use case.
+
+## Reporting Gaps
+If you identify a gap, use the Slack message template from section 1.2.

.cursor/rules/docs-diataxis-guidelines.mdc

@@ -0,0 +1,61 @@
+---
+globs: docs/**
+---
+# Diátaxis Documentation Guidelines
+
+When writing or editing documentation, categorise each page as **one** of the four Diátaxis modes and follow its specific guidance. *Do not mix modes in a single page.*
+
+## 1. Tutorials  🧑‍🏫  (`docs/getstarted/`, `docs/experimental/tutorials/`)
+• Purpose: provide a structured **learning experience** – “Can you teach me to…?”  
+• Form: narrative lesson that leads the reader from zero to a working result.  
+• Include: context, motivation, complete working example.  
+• Avoid: deep technical detail; troubleshooting; exhaustive options.
+
+## 2. How-to Guides  🍳  (`docs/howtos/`)
+• Purpose: help the user **achieve a specific goal** – “How do I…?”  
+• Form: concise series of **step-by-step instructions** focused on the user’s project.  
+• Write from the **user’s perspective**, not the tool’s operations.  
+• Link to reference and explanation pages for background; keep prose minimal.
+• Add any code run outputs as expandable click blocks. Readers should be able to understand the guide without running the code.
+
+## 3. Reference  📑  (`docs/references/`)
+• Purpose: provide **neutral, complete, accurate description** of APIs, commands, options – “What is…?”  
+• Maintain consistent patterns (parameter tables, return types, examples).  
+• Avoid instruction or opinion; instead *link* to how-to or explanation pages.  
+• Examples are welcome if they illustrate usage without drifting into tutorial style.
+
+## 4. Explanation  💡  (`docs/concepts/`, `docs/experimental/core_concepts/`)
+• Purpose: **clarify concepts and rationale** – “Why…?”  
+• Form: discursive article that illuminates design decisions, theory, background.  
+• May link out to tutorials, how-tos, and reference, but does not instruct step-by-step.
+
+### Keep the Borders Sharp
+• Do **not** let content blur between modes (e.g., no instructions inside reference; no lengthy theory in how-tos).  
+• If a page starts serving two modes, split it.
+
+### Filing & Navigation
+• Place the file in the folder matching its mode (above).  
+• Update `mkdocs.yml` `nav:` under the corresponding section.
+
+### Incremental Improvement Cycle (per Diátaxis)
+Choose → Assess → Decide → Do. Focus on small, atomic upgrades rather than grand rewrites.
+
+### Writing Style
+• Use second-person ("you") and active voice.
+• Ensure code blocks are **copy-pasteable** and include necessary context (imports, environment).
+• Prefer short sentences; use Markdown admonitions (`!!! note`, `!!! warning`) sparingly for important side-information.
+• Use `??? "Click to expand"` collapsible admonitions to contain outputs, long prompts, verbose logs, or any content that would clutter the main article flow. This keeps the primary content scannable while preserving detailed information for readers who need it.
+• **Always add a blank line after text ending with a colon before starting a list.** This ensures proper Markdown rendering in MkDocs. Without the blank line, list items may render as continuation text instead of a proper bulleted/numbered list.
+
+### Cross-linking Between Modes
+• End tutorials with pointers to relevant how-to guides for further exploration.
+• How-to guides should include links to reference/API pages for deeper details.
+• Explanations can reference tutorials and how-tos to illustrate concepts in action.
+
+### Page Metadata & Prerequisites
+• Start each page with a one-sentence purpose statement and a brief list of prerequisites (libraries, data, environment variables).
+• Highlight any external services/configuration required before the reader begins.
+
+### Keep Pages Atomic
+• One page = one task, concept, or API surface. If content grows, **split** rather than creating a mega-guide.
+

.cursor/rules/docs-structure.mdc

@@ -0,0 +1,46 @@
+---
+globs: docs/**
+---
+# Documentation Structure & Workflow
+
+Follow these conventions when creating or editing documentation:
+
+1. **Docs live in [docs/](mdc:docs/)**
+   • Use Markdown (`.md`) files.  
+   • Images and other assets go in [docs/_static/](mdc:docs/_static/).
+
+2. **Section Folders mirror MkDocs navigation** (see [mkdocs.yml](mdc:mkdocs.yml)):
+   • 🚀 Get Started → [docs/getstarted/](mdc:docs/getstarted/)  
+   • 📚 Core Concepts → [docs/concepts/](mdc:docs/concepts/)  
+   • 🧪 Experimental → [docs/experimental/](mdc:docs/experimental/)  
+   • 🛠️ How-to Guides → [docs/howtos/](mdc:docs/howtos/)  
+   • 📖 References → [docs/references/](mdc:docs/references/)  
+   • Community → [docs/community/](mdc:docs/community/)
+
+   Place new pages in the appropriate folder **and** update `mkdocs.yml` `nav:` so the page appears in navigation.
+
+3. **Notebook-to-Markdown**
+   • Convert notebooks to Markdown with [docs/ipynb_to_md.py](mdc:docs/ipynb_to_md.py).  
+   • Commit the generated `.md`; notebooks themselves should not live in `docs/`.
+
+4. **Local preview / build**
+   • Run `make build-docs` to build HTML, `make serve-docs` to preview locally (defined in [DEVELOPMENT.md](mdc:DEVELOPMENT.md)).
+
+5. **Style & Assets**
+   • Use relative links (`../`) within docs.  
+   • Reference images via `_static/…` paths so they work in both dev and hosted docs.  
+   • Custom templates/CSS live in [docs/extra/](mdc:docs/extra/) — avoid editing `material` theme defaults directly.
+
+6. **API References (mkdocstrings)**
+   • Always use public API paths in `[ClassName][ragas.module.ClassName]` references.
+   • Check what's exported in `__init__.py` — if a class isn't in `__all__`, mkdocstrings can't link to it.
+   • Example: Use `[BasePrompt][ragas.prompt.BasePrompt]` not `[BasePrompt][ragas.prompt.base.BasePrompt]` or internal module paths.
+
+7. **Do not modify generated or third-party files** in `_static/`, `extra/overrides/`, or `extra/components/` without good reason.
+
+---
+
+# Formatting Guidelines
+
+- When introducing a list with text ending in a colon (e.g., "This will:"), always add a blank line before the first list item. 
+- In a numbered list, do not add any new line between the items.

.cursor/rules/project-structure.mdc

@@ -0,0 +1,22 @@
+---
+alwaysApply: true
+---
+# Monorepo Project Structure
+
+The repository is a monorepo consisting of two primary components:
+
+1. [/](./) – Core evaluation toolkit
+   • Source code lives in [src/](src/)
+   • Tests live in [tests/](tests/)
+   • Build configuration is in [pyproject.toml](pyproject.toml)
+
+2. [examples/](examples/) – Installable examples package
+   • Package: ragas_examples/ containing agent_evals, prompt_evals, rag_eval, workflow_eval, benchmark_llm
+   • Build configuration in [examples/pyproject.toml](examples/pyproject.toml)
+   • Shipped as `ragas-examples` package on PyPI via `ragas[examples]` extra
+   • Local development: `uv pip install -e . -e ./examples`
+   • Usage: `python -m ragas_examples.benchmark_llm.prompt`
+
+Shared documentation for all projects is located under [docs/](docs/).
+
+Root-level [Makefile](Makefile) provides combined commands, while each project directory also contains its own Makefile for project-specific tasks.

.cursor/rules/update-guide.mdc

@@ -0,0 +1,8 @@
+---
+alwaysApply: false
+---
+We are writing a how to guide for Ragas docs and Ragas users.
+
+So after any coding step we complete, or after any succesful runs, always update the guide with what was done. Make sure the content is concise and to the point. 
+
+Current guide: docs/howtos/applications/evaluate-and-improve-rag.md

.cursor/rules/use-uv-cli.mdc

@@ -0,0 +1,12 @@
+---
+alwaysApply: true
+---
+# Use `uv run` for Python CLI commands
+
+This repository manages its virtual environment and dependencies with **uv**. Therefore, always execute Python or Python-related CLI tools through `uv run`.
+
+Examples:
+- `uv run pytest`
+- `uv run ruff check .`
+- `uv run isort .`
+- `uv run pyright`

.cursor/worktrees.json

@@ -0,0 +1,7 @@
+{
+  "setup-worktree": [
+    "cp $ROOT_WORKTREE_PATH/.env .env",
+    "make install-minimal",
+    "make check"
+  ]
+}

.dockerignore

@@ -0,0 +1,2 @@
+Dockerfile
+test_resources

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,29 @@
+---
+name: Bug Report
+about: Report any bugs your encounter and we will try our best to fix it for you 🙂
+title: ''
+labels: 'bug'
+assignees: ''
+---
+
+[ ] I have checked the [documentation](https://docs.ragas.io/) and related resources and couldn't resolve my bug.
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+Ragas version:
+Python version:
+
+**Code to Reproduce**
+Share code to reproduce the issue
+
+**Error trace**
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+
+**Additional context**
+Add any other context about the problem here.
+
+<!-- PS: bugs suck but is also part of the process. We sincerely apologies for breaking your flow because of it, but don't worry, we got your back ❤️. We will get this fixed as fast as we can and thanks for helping us out by reporting it 🙏. -->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,18 @@
+---
+name: Feature Request
+about: Feel like something is mising? Let us know!
+title: ''
+labels: 'enhancement'
+assignees: ''
+---
+
+**Describe the Feature**
+A clear and concise description of what the what you want to be added.
+
+**Why is the feature important for you?**
+Share code to reproduce the issue
+
+**Additional context**
+Add any other context about the feature you want to share with us.
+
+<!-- PS: Thanks for your valuable feedback. Really! Its feedback from valuable community members like you that help us make Ragas event better for the whole community. So thanks again for taking the time to improve our community 🙂 -->

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,18 @@
+---
+name: Questions
+about: Any questions or doubts? Ask us here!
+title: ''
+labels: 'question'
+assignees: ''
+---
+
+[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
+
+**Your Question**
+what is unclear to you? What would you like to know?
+
+**Code Examples**
+This community speaks code. Share your code snippets to help us understand your question better.
+
+**Additional context**
+Anything else you want to share with us? 

.github/pull_request_template.md

@@ -0,0 +1,35 @@
+## Issue Link / Problem Description
+<!-- Link to related issue or describe the problem this PR solves -->
+- Fixes #[issue_number]
+- OR describe the issue: What problem does this solve? How can it be replicated?
+
+## Changes Made
+<!-- Describe what you changed and why -->
+- 
+- 
+- 
+
+## Testing
+<!-- Describe how this should be tested -->
+### How to Test
+- [ ] Automated tests added/updated
+- [ ] Manual testing steps:
+  1. 
+  2. 
+  3. 
+
+## References
+<!-- Link to related issues, discussions, forums, or external resources -->
+- Related issues: 
+- Documentation: 
+- External references: 
+
+## Screenshots/Examples (if applicable)
+<!-- Add screenshots or code examples showing the change -->
+
+---
+<!-- 
+Thank you for contributing to Ragas! 
+Please fill out the sections above as completely as possible.
+The more information you provide, the faster your PR can be reviewed and merged.
+-->

.github/workflows/ci.yaml

@@ -0,0 +1,188 @@
+name: CI
+
+on:
+  pull_request:
+
+permissions:
+  contents: read
+
+env:
+  LINES: 120
+  COLUMNS: 120
+
+# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#defaultsrun
+defaults:
+  run:
+    shell: bash --noprofile --norc -exo pipefail {0}
+
+jobs:
+  diff:
+    runs-on: ubuntu-latest
+    outputs:
+      related: ${{ steps.filter.outputs.related }}
+      ragas: ${{ steps.filter.outputs.ragas }}
+      docs: ${{ steps.filter.outputs.docs }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dorny/paths-filter@v3
+        id: filter
+        with:
+          base: "main"
+          token: ${{ github.token }}
+          filters: |
+            related: &related
+              - .github/workflows/ci.yaml
+              - codecov.yml
+              - pyproject.toml
+              - Makefile
+            ragas:
+              - *related
+              - "src/ragas/**"
+              - "tests/**"
+              - "examples/**"
+            docs:
+              - *related
+              - "docs/**"
+
+  unit_tests:
+    needs:
+      - diff
+
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # Critical path: Latest + oldest Python on Ubuntu (full test suite)
+          - os: ubuntu-latest
+            python-version: "3.9"
+            test-type: "full"
+          - os: ubuntu-latest
+            python-version: "3.12"
+            test-type: "full"
+          - os: ubuntu-latest
+            python-version: "3.13"
+            test-type: "full"
+          # Cross-platform validation (essential tests only)
+          - os: macos-latest
+            python-version: "3.11"
+            test-type: "essential"
+          - os: windows-latest
+            python-version: "3.10"
+            test-type: "essential"
+
+    if: ${{ (github.event_name == 'pull_request' && needs.diff.outputs.ragas == 'true') || github.event_name == 'push' }}
+    name: python${{ matrix.python-version }}_unit_tests (${{ matrix.os }}, ${{ matrix.test-type }})
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # fetch all tags and branches
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          architecture: ${{ matrix.os == 'macos-latest' && 'arm64' || 'x64' }}
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Get pip cache dir
+        id: cache-dir
+        run: |
+          echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT
+
+      - name: Cache dependencies (UV cache)
+        uses: actions/cache@v4
+        id: cache-deps
+        with:
+          path: |
+            ${{ steps.cache-dir.outputs.dir }}
+            ~/.cache/uv
+          key: deps-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('pyproject.toml') }}
+          restore-keys: |
+            deps-${{ runner.os }}-py${{ matrix.python-version }}-
+            deps-${{ runner.os }}-py3.11-
+            deps-${{ runner.os }}-
+
+      - name: Install dependencies
+        run: |
+          # Use minimal install for fast CI runs (79 packages vs 383)
+          # This uses make install-minimal for consistency with local development
+          make install-minimal
+
+      - name: Run unit tests
+        run: |
+          # Configure test options based on OS and test type
+          if [ "${{ matrix.os }}" != 'windows-latest' ]; then
+            # Use pytest-xdist to improve test run-time on Linux/macOS
+            OPTS=(--dist loadfile -n auto)
+          fi
+
+          # Run different test suites based on test type
+          if [ "${{ matrix.test-type }}" = "full" ]; then
+            # Full test suite with notebook tests
+            uv run pytest --nbmake tests/unit "${OPTS[@]}"
+          else
+            # Essential tests only (faster for cross-platform validation)
+            uv run pytest tests/unit -k "not slow" "${OPTS[@]}"
+          fi
+        env:
+          __RAGAS_DEBUG_TRACKING: true
+          RAGAS_DO_NOT_TRACK: true
+
+  code_quality_check:
+    runs-on: ubuntu-latest
+    needs:
+      - diff
+
+    if: ${{ (github.event_name == 'pull_request' && needs.diff.outputs.ragas == 'true') || github.event_name == 'push' }}
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          architecture: x64
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - name: Get pip cache dir
+        id: cache-dir
+        run: |
+          echo "dir=$(pip cache dir)" >> $GITHUB_OUTPUT
+
+      - name: Cache dependencies (UV cache)
+        uses: actions/cache@v4
+        id: cache-deps
+        with:
+          path: |
+            ${{ steps.cache-dir.outputs.dir }}
+            ~/.cache/uv
+          key: deps-ubuntu-py3.11-codestyle-${{ hashFiles('pyproject.toml') }}
+          restore-keys: |
+            deps-ubuntu-py3.11-codestyle-
+            deps-ubuntu-py3.11-
+            deps-ubuntu-
+
+      - name: Install dependencies
+        run: |
+          # Use minimal install for fast CI runs (79 packages vs 383)
+          # This uses make install-minimal for consistency with local development
+          make install-minimal
+
+      - name: Format check (dry run)
+        run: |
+          # Check if code is properly formatted (without making changes)
+          # Note: We use direct commands here instead of the standalone Makefiles
+          # to have precise control over CI-specific options like --check for dry-run
+          echo "Checking ragas formatting..."
+          uv run ruff format --check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+          uv run ruff check src docs tests --exclude src/ragas/_version.py --config pyproject.toml
+
+      - name: Type check
+        run: make type

.github/workflows/claude-code-review.yml

@@ -0,0 +1,91 @@
+name: Claude Code Review
+
+on:
+  issue_comment:
+    types: [created]
+
+jobs:
+  claude-review:
+    if: |
+      github.event.issue.pull_request &&
+      contains(github.event.comment.body, '/claude-review')
+    
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+      issues: write
+      id-token: write
+    
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude Code Review
+        id: claude-review
+        uses: anthropics/claude-code-action@beta
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+
+          # Optional: Specify model (defaults to Claude Sonnet 4, uncomment for Claude Opus 4.1)
+          # model: "claude-opus-4-1-20250805"
+
+          # Customize the trigger phrase to use /claude-review
+          trigger_phrase: "/claude-review"
+
+          # Custom instructions for the review
+          custom_instructions: |
+            When triggered with /claude-review, please analyze this pull request and provide:
+
+            ## Change Type Classification
+            First, identify the primary type of change based on the files modified and changes made:
+            - **🐛 Bug Fix**: Fixes existing functionality
+            - **✨ New Feature**: Adds new functionality
+            - **📚 Documentation**: Updates or adds documentation (README, docs/, comments)
+            - **🔧 Refactor**: Code restructuring without changing functionality
+            - **🧪 Tests**: Adds or modifies tests
+            - **🏗️ Build/CI**: Changes to build process, CI/CD, dependencies
+            - **🎨 Style**: Code formatting, linting fixes
+            - **⚡ Performance**: Improves performance
+            - **🔒 Security**: Security-related improvements
+            - **🗑️ Cleanup**: Removes deprecated code, unused files
+            - **🔀 Merge**: Merge commits or branch management
+            - **📦 Dependencies**: Updates dependencies or package versions
+
+            ## Code Review
+            Then provide feedback on:
+            - Code quality and best practices
+            - Potential bugs or issues
+            - Performance considerations
+            - Security concerns
+            - Test coverage
+            
+            Be constructive and helpful in your feedback.
+
+          # Optional: Use sticky comments to make Claude reuse the same comment on subsequent pushes to the same PR
+          # use_sticky_comment: true
+          
+          # Optional: Customize review based on file types
+          # direct_prompt: |
+          #   Review this PR focusing on:
+          #   - For TypeScript files: Type safety and proper interface usage
+          #   - For API endpoints: Security, input validation, and error handling
+          #   - For React components: Performance, accessibility, and best practices
+          #   - For tests: Coverage, edge cases, and test quality
+          
+          # Optional: Different prompts for different authors
+          # direct_prompt: |
+          #   ${{ github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR' && 
+          #   'Welcome! Please review this PR from a first-time contributor. Be encouraging and provide detailed explanations for any suggestions.' ||
+          #   'Please provide a thorough code review focusing on our coding standards and best practices.' }}
+          
+          # Optional: Add specific tools for running tests or linting
+          # allowed_tools: "Bash(npm run test),Bash(npm run lint),Bash(npm run typecheck)"
+          
+          # Optional: Skip review for certain conditions
+          # if: |
+          #   !contains(github.event.pull_request.title, '[skip-review]') &&
+          #   !contains(github.event.pull_request.title, '[WIP]')
+

.github/workflows/claude-docs-apply.yml

@@ -0,0 +1,104 @@
+name: Claude Docs Apply
+
+on:
+  pull_request_target:
+    types: [labeled]
+
+jobs:
+  apply-docs:
+    if: github.event.label.name == 'update-docs'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+      issues: write
+      id-token: write
+
+    steps:
+      - name: Checkout PR branch
+        uses: actions/checkout@v4
+        with:
+          repository: ${{ github.event.pull_request.head.repo.full_name }}
+          ref: ${{ github.event.pull_request.head.ref }}
+          # Use PAT for fork PRs (requires CLAUDE_CODE_PAT secret), GITHUB_TOKEN for same-repo PRs
+          token: ${{ secrets.CLAUDE_CODE_PAT || secrets.GITHUB_TOKEN }}
+          fetch-depth: 0
+
+      - name: Configure git
+        run: |
+          git config --global user.name "Claude Code Bot"
+          git config --global user.email "noreply@anthropic.com"
+
+      - name: Apply documentation updates
+        uses: anthropics/claude-code-action@v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          github_token: ${{ secrets.CLAUDE_CODE_PAT || secrets.GITHUB_TOKEN }}
+
+          prompt: |
+            REPO: ${{ github.repository }}
+            PR NUMBER: ${{ github.event.pull_request.number }}
+            PR TITLE: ${{ github.event.pull_request.title }}
+
+            You are a documentation assistant for the Ragas project. Update the documentation based on the code changes in this PR.
+
+            ## Quick Action Plan
+
+            1. Run `gh pr diff` to review changes
+            2. Identify what docs need updating (see structure below)
+            3. Make focused updates efficiently
+            4. Commit with clear message
+
+            ## Documentation Structure (Diátaxis Framework)
+
+            **Where to update:**
+            - `docs/howtos/` - How-to guides (step-by-step instructions)
+            - `docs/concepts/` - Concept docs (explanations and rationale)
+            - `docs/getstarted/` - Tutorials (learning experiences)
+            - Source code docstrings - API documentation (feeds auto-generated reference)
+
+            **DO NOT edit:**
+            - `docs/references/**` - AUTO-GENERATED by mkdocstrings
+
+            ## Writing Guidelines
+
+            - Use second-person ("you") and active voice
+            - Code blocks must be copy-pasteable with imports
+            - Use `??? "Click to expand"` for verbose outputs
+            - Add blank line after text ending with colon before lists
+            - Update `mkdocs.yml` nav if adding new pages
+            - Keep modes separate: no theory in how-tos, no instructions in concepts
+
+            ## Documentation Modes Reference
+
+            1. **Tutorials** (`docs/getstarted/`) - "Can you teach me to...?"
+               - Narrative learning experience with complete working examples
+
+            2. **How-to Guides** (`docs/howtos/`) - "How do I...?"
+               - Concise step-by-step from user's perspective
+
+            3. **Reference** (`docs/references/`) - "What is...?"
+               - AUTO-GENERATED - edit source docstrings instead
+
+            4. **Explanation** (`docs/concepts/`) - "Why...?"
+               - Discursive articles on design decisions and theory
+
+            ## Completion
+
+            After making changes, commit to this PR branch with a concise, descriptive message.
+
+          claude_args: |
+            --max-turns 30
+            --allowedTools "Read,Write,Edit,Glob,Grep,Bash(git:*),Bash(gh pr diff:*),Bash(gh pr view:*)"
+
+      - name: Remove labels after completion
+        if: always()
+        run: |
+          # Remove both labels
+          gh pr edit ${{ github.event.pull_request.number }} --remove-label "update-docs" || true
+          gh pr edit ${{ github.event.pull_request.number }} --remove-label "needs-doc-update" || true
+
+          # Comment that docs were updated
+          gh pr comment ${{ github.event.pull_request.number }} --body "✅ Documentation update completed."
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/claude-docs-check.yml

@@ -0,0 +1,119 @@
+name: Claude Docs Check
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+    paths:
+      - "src/**/*.py"
+
+jobs:
+  check-docs:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
+      id-token: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          repository: ${{ github.event.pull_request.head.repo.full_name }}
+          ref: ${{ github.event.pull_request.head.sha }}
+          token: ${{ secrets.GITHUB_TOKEN }}
+          fetch-depth: 0
+
+      - name: Analyze PR for documentation needs
+        id: analyze
+        uses: anthropics/claude-code-action@v1
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          allowed_non_write_users: "*"
+
+          prompt: |
+            REPO: ${{ github.repository }}
+            PR NUMBER: ${{ github.event.pull_request.number }}
+            PR TITLE: ${{ github.event.pull_request.title }}
+
+            You are a documentation analyst for the Ragas project. Analyze this PR to determine if documentation updates are needed.
+
+            ## Quick Decision Rules
+
+            **needs_update: false** (most common):
+            - Docstrings already updated in code → no action needed (API docs are auto-generated)
+            - Internal refactoring with no API changes → no action needed
+            - Bug fixes with no user-facing changes → no action needed
+            - Infrastructure/build changes → no action needed
+
+            **needs_update: true** (only when necessary):
+            - New user-facing features WITHOUT docstrings → need docs
+            - Changed usage patterns in how-to guides → need updates
+            - New core concepts without explanation → need concept docs
+            - Modified getting started flow → need tutorial updates
+
+            ## Your Task
+
+            1. Run `gh pr diff` to review code changes
+            2. Check if docstrings are present for API changes
+            3. Return JSON immediately with your decision
+
+            Return format:
+            - `needs_update`: boolean
+            - `reason`: brief explanation (1-2 sentences max)
+
+            ## Documentation Structure Reference
+            - `docs/howtos/` - Step-by-step guides
+            - `docs/concepts/` - Conceptual explanations
+            - `docs/getstarted/` - Tutorials
+            - `docs/references/` - AUTO-GENERATED (never edit directly)
+
+            IMPORTANT: Be decisive. Default to needs_update: false if docstrings are present. Return JSON within 3 turns.
+
+          claude_args: |
+            --max-turns 20
+            --json-schema '{"type":"object","properties":{"needs_update":{"type":"boolean"},"reason":{"type":"string"}},"required":["needs_update","reason"]}'
+            --allowedTools "Bash(gh pr diff:*),Bash(gh pr view:*),Read,Glob,Grep"
+
+      - name: Parse analysis result
+        id: parse
+        run: |
+          # Use heredoc to safely handle JSON with special characters
+          cat <<'EOF' > /tmp/output.json
+          ${{ steps.analyze.outputs.structured_output }}
+          EOF
+
+          echo "structured_output=$(cat /tmp/output.json)"
+          NEEDS_UPDATE=$(jq -r '.needs_update' /tmp/output.json)
+          REASON=$(jq -r '.reason' /tmp/output.json)
+          echo "needs_update=$NEEDS_UPDATE" >> $GITHUB_OUTPUT
+
+          # Use multiline string format for reason to handle special characters
+          {
+            echo 'reason<<EOF'
+            jq -r '.reason' /tmp/output.json
+            echo 'EOF'
+          } >> $GITHUB_OUTPUT
+
+      - name: Add label and comment if docs needed
+        if: steps.parse.outputs.needs_update == 'true'
+        run: |
+          # Add the needs-doc-update label
+          gh pr edit ${{ github.event.pull_request.number }} --add-label "needs-doc-update"
+
+          # Comment with instructions
+          gh pr comment ${{ github.event.pull_request.number }} --body "📝 **Documentation update may be needed**
+
+          ${{ steps.parse.outputs.reason }}
+
+          **To apply documentation updates:** Add the \`update-docs\` label to this PR."
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Comment if no docs needed
+        if: steps.parse.outputs.needs_update == 'false'
+        run: |
+          gh pr comment ${{ github.event.pull_request.number }} --body "✅ No documentation update needed — ${{ steps.parse.outputs.reason }}"
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/claude.yml

@@ -0,0 +1,64 @@
+name: Claude Code
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+  issues:
+    types: [opened, assigned]
+  pull_request_review:
+    types: [submitted]
+
+jobs:
+  claude:
+    if: |
+      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
+      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
+      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
+      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+      issues: write
+      id-token: write
+      actions: read # Required for Claude to read CI results on PRs
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 1
+
+      - name: Run Claude Code
+        id: claude
+        uses: anthropics/claude-code-action@beta
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+
+          # This is an optional setting that allows Claude to read CI results on PRs
+          additional_permissions: |
+            actions: read
+          
+          # Optional: Specify model (defaults to Claude Sonnet 4, uncomment for Claude Opus 4.1)
+          # model: "claude-opus-4-1-20250805"
+          
+          # Optional: Customize the trigger phrase (default: @claude)
+          # trigger_phrase: "/claude"
+          
+          # Optional: Trigger when specific user is assigned to an issue
+          # assignee_trigger: "claude-bot"
+          
+          # Optional: Allow Claude to run specific commands
+          # allowed_tools: "Bash(npm install),Bash(npm run build),Bash(npm run test:*),Bash(npm run lint:*)"
+          
+          # Optional: Add custom instructions for Claude to customize its behavior for your project
+          # custom_instructions: |
+          #   Follow our coding standards
+          #   Ensure all new code has tests
+          #   Use TypeScript for new files
+          
+          # Optional: Custom environment variables for Claude
+          # claude_env: |
+          #   NODE_ENV: test
+

.github/workflows/issue-manager.yaml

@@ -0,0 +1,49 @@
+name: Issue Manager
+
+on:
+  schedule:
+    - cron: "0 0 * * *"
+  issue_comment:
+    types:
+      - created
+      - edited
+  issues:
+    types:
+      - labeled
+  pull_request_target:
+    types:
+      - labeled
+  workflow_dispatch:
+
+jobs:
+  issue-manager:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+      - uses: tiangolo/issue-manager@0.4.0
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+          config: >
+            {
+                "$schema": "https://raw.githubusercontent.com/tiangolo/issue-manager/master/schema.json",
+                "answered": {
+                    "delay": "P3DT12H30M5S",
+                    "message": "It seems the issue was answered, closing this now.",
+                    "remove_label_on_comment": false,
+                    "remove_label_on_close": false
+                },
+                "validated": {
+                    "delay": 300,
+                    "message": "The issue could not be validated after 5 minutes. Closing now.",
+                    "remove_label_on_comment": true,
+                    "remove_label_on_close": false
+                },
+                "waiting": {
+                    "delay": 691200,
+                    "message": "Closing after 8 days of waiting for the additional info requested.",
+                    "remove_label_on_comment": true,
+                    "remove_label_on_close": true
+                }
+            }

.github/workflows/publish-examples.yml

@@ -0,0 +1,45 @@
+name: Upload ragas-examples Package
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: pypi-release
+    strategy:
+      matrix:
+        package:
+          - name: ragas-examples
+            directory: examples
+            token: PYPI_API_TOKEN
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install --upgrade setuptools setuptools_scm[toml] build 
+    - name: get setuptools-scm version 
+      run: python -m setuptools_scm
+      working-directory: ${{ matrix.package.directory }}
+    - name: Build package
+      run: python -m build
+      working-directory: ${{ matrix.package.directory }}
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        password: ${{ secrets[matrix.package.token] }}
+        packages-dir: ${{ matrix.package.directory }}/dist/
+        attestations: false
+
+

.github/workflows/python-publish.yml

@@ -0,0 +1,51 @@
+# This workflow will upload Python Packages using Twine when a release is created
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
+
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+
+name: Upload Python Packages
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment: pypi-release
+    strategy:
+      matrix:
+        package:
+          - name: ragas
+            directory: .
+            token: PYPI_API_TOKEN
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install --upgrade setuptools setuptools_scm[toml] build 
+    - name: get setuptools-scm version 
+      run: python -m setuptools_scm
+      working-directory: ${{ matrix.package.directory }}
+    - name: Build package
+      run: python -m build
+      working-directory: ${{ matrix.package.directory }}
+    - name: Publish package
+      uses: pypa/gh-action-pypi-publish@release/v1
+      with:
+        password: ${{ secrets[matrix.package.token] }}
+        packages-dir: ${{ matrix.package.directory }}/dist/
+        attestations: false

.gitignore

@@ -0,0 +1,205 @@
+# General
+.DS_Store
+.AppleDouble
+.LSOverride
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+# Cursor
+.cursorignore
+.cursor/plans
+
+# Ragas specific
+_experiments/
+**/fil-result/
+src/ragas/_version.py
+experimental/ragas_experimental/_version.py
+examples/ragas_examples/_version.py
+.vscode
+.envrc
+uv.lock
+.cache/
+.claude/*
+!.claude/commands/
+node_modules
+
+# Ragas examples
+experimental/ragas_examples/benchmark_llm/experiments/*.csv
+examples/*/logs/*run*json
+examples/*/experiments/*csv
+examples/ragas_examples/_version.py
+**/test_dataset.csv
+
+# Ragas examples package build artifacts
+examples/dist/
+examples/build/
+examples/*.egg-info/
+examples/ragas_examples/_version.py
+examples/ragas_examples/text2sql/experiments/*
+examples/ragas_examples/benchmark_llm/experiments/*
+BookSQL-files
+text2sql_logs
+
+# MLflow artifacts
+mlartifacts
+mlflow.db
+plan

.pre-commit-config.yaml

@@ -0,0 +1,13 @@
+# Pre-commit configuration for entire ragas monorepo
+# Install with: make install && pre-commit install
+repos:
+  - repo: local
+    hooks:
+      - id: monorepo-ci
+        name: Run complete monorepo CI pipeline
+        entry: make run-ci-format-check
+        language: system
+        pass_filenames: false
+        always_run: true
+        stages: [pre-commit]
+        verbose: true

.readthedocs.yml

@@ -0,0 +1,14 @@
+version: 2
+
+mkdocs:
+  configuration: mkdocs.yml
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.12"
+  commands:
+    - pip install uv
+    - uv pip install --system -e "." --group docs
+    - if [ -n "$GH_TOKEN" ]; then pip install git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git; fi
+    - mkdocs build --site-dir $READTHEDOCS_OUTPUT/html

CLAUDE.md

@@ -0,0 +1,218 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Ragas is an evaluation toolkit for Large Language Model (LLM) applications. It provides objective metrics for evaluating LLM applications, test data generation capabilities, and integrations with popular LLM frameworks.
+
+The repository contains:
+
+1. **Ragas Library** - The main evaluation toolkit including experimental features (in `src/ragas/` directory)
+   - Core evaluation metrics and test generation
+   - Experimental features available at `ragas.experimental`
+
+## Development Environment Setup
+
+### Installation
+
+Choose the appropriate installation based on your needs:
+
+```bash
+# RECOMMENDED: Minimal dev setup (79 packages - fast)
+make install-minimal
+
+# FULL: Complete dev environment (383 packages - comprehensive)  
+make install
+
+# OR manual installation:
+# Create a virtual environment
+python -m venv venv
+source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
+
+# Minimal dev setup (uses [project.optional-dependencies].dev-minimal)
+uv pip install -e ".[dev-minimal]"
+
+# Full dev setup (uses [dependency-groups].dev)
+uv sync --group dev
+```
+
+### Installation Methods Explained
+
+- **Minimal setup**: Uses `uv pip install` with optional dependencies for selective installation
+- **Full setup**: Uses `uv sync` with dependency groups for comprehensive environment management
+- **No naming conflicts**: `dev-minimal` vs `dev` clearly distinguish the two approaches
+
+### Workspace Structure
+
+The project uses a UV workspace configuration for managing multiple packages:
+
+```bash
+# Install
+uv sync
+
+# Install examples separately
+uv sync --package ragas-examples
+
+# Build specific workspace package
+uv build --package ragas-examples
+```
+
+**Workspace Members:**
+- `ragas` (main package) - Located in `src/ragas/`
+- `ragas-examples` (examples package) - Located in `examples/`
+
+The workspace ensures consistent dependency versions across packages and enables editable installs of workspace members.
+
+## Common Commands
+
+### Commands (from root directory)
+
+```bash
+# Setup and installation  
+make install-minimal # Minimal dev setup (79 packages - recommended)
+make install        # Full dev environment (383 packages - complete)
+
+# Code quality
+make format         # Format and lint all code
+make type           # Type check all code
+make check          # Quick health check (format + type, no tests)
+
+# Testing
+make test           # Run all unit tests
+make test-e2e       # Run end-to-end tests
+
+# CI/Build
+make run-ci         # Run complete CI pipeline
+make clean          # Clean all generated files
+
+# Documentation
+make build-docs     # Build all documentation
+make serve-docs     # Serve documentation locally
+
+# Benchmarks
+make benchmarks     # Run performance benchmarks
+make benchmarks-docker # Run benchmarks in Docker
+```
+
+### Testing
+
+```bash
+# Run all tests (from root)
+make test
+
+# Run specific test (using pytest -k flag)
+make test k="test_name"
+
+# Run end-to-end tests
+make test-e2e
+
+# Direct pytest commands for more control
+uv run pytest tests/unit -k "test_name"
+uv run pytest tests/unit -v
+```
+
+### Documentation
+
+```bash
+# Build all documentation (from root)
+make build-docs
+
+# Serve documentation locally
+make serve-docs
+```
+
+### Benchmarks
+
+```bash
+# Run all benchmarks locally
+make benchmarks
+
+# Run benchmarks in Docker
+make benchmarks-docker
+```
+
+## Project Architecture
+
+The repository has the following structure:
+
+```sh
+/                          # Main ragas project
+├── src/ragas/             # Source code including experimental features
+│   └── experimental/      # Experimental features
+├── tests/                 # All tests (core + experimental)
+│   └── experimental/      # Experimental tests
+├── examples/              # Example code
+├── pyproject.toml         # Build config
+├── docs/                  # Documentation
+├── scripts/               # Build/CI scripts
+├── Makefile               # Build commands
+└── README.md              # Repository overview
+```
+
+### Ragas Core Components
+
+The Ragas core library provides metrics, test data generation and evaluation functionality for LLM applications:
+
+1. **Metrics** - Various metrics for evaluating LLM applications including:
+
+   - AspectCritic
+   - AnswerCorrectness
+   - ContextPrecision
+   - ContextRecall
+   - Faithfulness
+   - and many more
+
+2. **Test Data Generation** - Automatic creation of test datasets for LLM applications
+
+3. **Integrations** - Integrations with popular LLM frameworks like LangChain, LlamaIndex, and observability tools
+
+### Experimental Components
+
+The experimental features are now integrated into the main ragas package:
+
+1. **Experimental features** are available at `ragas.experimental`
+2. **Dataset and Experiment management** - Enhanced data handling for experiments
+3. **Advanced metrics** - Extended metric capabilities
+4. **Backend support** - Multiple storage backends (CSV, JSONL, Google Drive, in-memory)
+
+To use experimental features:
+
+```python
+from ragas import Dataset
+from ragas import experiment
+from ragas.backends import get_registry
+```
+
+## Debugging Logs
+
+To view debug logs for any module:
+
+```python
+import logging
+
+# Configure logging for a specific module (example with analytics)
+analytics_logger = logging.getLogger('ragas._analytics')
+analytics_logger.setLevel(logging.DEBUG)
+
+# Create a console handler and set its level
+console_handler = logging.StreamHandler()
+console_handler.setLevel(logging.DEBUG)
+
+# Create a formatter and add it to the handler
+formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
+console_handler.setFormatter(formatter)
+
+# Add the handler to the logger
+analytics_logger.addHandler(console_handler)
+```
+
+## Memories
+
+- whenever you create such docs put in in /\_experiments because that is gitignored and you can use it as a scratchpad or tmp directory for storing these
+- always use uv to run python and python related commandline tools like isort, ruff, pyright etc. This is because we are using uv to manage the .venv and dependencies.
+- The project uses two distinct dependency management approaches:
+  - **Minimal setup**: `[project.optional-dependencies].dev-minimal` for fast development (79 packages)
+  - **Full setup**: `[dependency-groups].dev` for comprehensive development (383 packages)
+- Use `make install-minimal` for most development tasks, `make install` for full ML stack work
+- if the user asks you to save a plan, save it into the plan/ directory with an appropriate file name.

CODE_OF_CONDUCT.md

@@ -0,0 +1,117 @@
+# Code of Conduct
+
+## Our Commitment
+
+We are committed to providing a welcoming and inclusive environment for all people, regardless of age, body size, caste, disability, ethnicity, gender identity and expression, level of experience, family status, gender, immigration status, level of expertise, national origin, personal appearance, political belief, race, religion, sexual identity and orientation, socioeconomic status, tribe, and veteran status.
+
+We expect all participants in the Ragas community—whether contributing code, providing feedback, reporting issues, participating in discussions, attending events, or engaging in any other capacity—to embody the values of respect, inclusion, and professionalism.
+
+## Our Standards
+
+Examples of behaviour that contributes to creating a positive environment include:
+
+- Using welcoming and inclusive language
+- Being respectful of differing opinions, viewpoints, and experiences
+- Gracefully accepting constructive criticism
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+- Being patient and understanding with newcomers
+- Giving credit to others' work and contributions
+- Asking clarifying questions rather than making assumptions
+
+Examples of unacceptable behaviour include:
+
+- Harassment, intimidation, or discrimination of any kind
+- Unwelcome sexual attention or advances
+- Trolling, insulting/derogatory comments, and personal or political attacks
+- Publishing others' private information without explicit permission (doxing)
+- Gatekeeping—deliberately excluding or discouraging participation
+- Deliberate disinformation or misinformation
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+- Sustained disruption of discussions or project activities, including:
+  - Spam, off-topic posts, or repeated low-effort comments in issues or discussions
+  - Duplicate issues or discussions that have already been reported
+  - Cross-posting the same issue or question across multiple channels without justification
+  - Deliberately posting controversial or unrelated content to distract from ongoing discussions
+- Threats of violence or violent language directed at another person
+
+## Scope
+
+This Code of Conduct applies to all spaces managed by the Ragas project, including:
+
+- GitHub repositories (issues, pull requests, discussions, and code reviews)
+- Official communication channels (Discord, Slack, mailing lists, forums)
+- Official events and conferences organised by Ragas maintainers
+- Any official online or offline event, conference, or gathering representing Ragas
+
+This Code of Conduct also applies to conduct outside of these spaces if it demonstrates a pattern of harassment or is reasonably perceived as affecting the safety or well-being of community members.
+
+The Code of Conduct applies equally to all participants, including maintainers, contributors, sponsors, and community members.
+
+## Reporting Violations
+
+If you experience or witness behaviour that violates this Code of Conduct, please report it by emailing **support@ragas.io**. Include as much detail as you're comfortable sharing, including:
+
+- What happened
+- Who was involved
+- When it occurred
+- Any relevant links or context
+- Any witnesses (optional)
+
+All reports will be treated confidentially. We will not disclose the identity of the reporter without their consent, except as necessary for investigation and response.
+
+If the violation involves a member of the Code of Conduct committee, or if you're not comfortable reporting directly to that address, please reach out to a project maintainer directly through alternative means.
+
+## Enforcement
+
+The Ragas project maintainers are responsible for clarifying standards of acceptable behaviour and will take appropriate action in response to violations of this Code of Conduct.
+
+### Our Commitment to Enforcement
+
+We recognise that:
+
+- Not all violations are equally severe
+- Context matters
+- People can learn and grow
+- The goal is to maintain a healthy, inclusive community
+
+### Enforcement Guidelines
+
+The following are examples of how we may respond to violations. Responses will be proportionate to the severity and pattern of behaviour:
+
+1. **Warning**: For minor or first-time violations, a private message explaining the issue and its impact, with an expectation to change behaviour.
+
+2. **Temporary Suspension**: For more serious or repeated violations, temporary removal from community spaces (ranging from hours to weeks) to allow for reflection and de-escalation.
+
+3. **Permanent Removal**: For severe, repeated, or unresolved violations, permanent removal from the project and its community spaces.
+
+4. **Law Enforcement**: In cases involving illegal activity or threats of violence, we may involve law enforcement.
+
+The maintainers may also take action to address behaviour even if no formal complaint has been filed, if they reasonably believe it violates this Code of Conduct.
+
+## Consequences for Violations
+
+Anyone who violates this Code of Conduct may face consequences determined by the Ragas maintainers, including:
+
+- Editing or deletion of comments or contributions
+- Removal from the project repository or community spaces
+- Temporary or permanent ban from participating in Ragas spaces
+- Public acknowledgment of the violation (at the discretion of the reporter and maintainers)
+
+## Appeal Process
+
+If you believe you have been unfairly sanctioned under this Code of Conduct, you may appeal by sending a detailed explanation to **support@ragas.io**. The appeal will be reviewed by a different set of maintainers when possible, and a decision will be communicated to you within a reasonable timeframe.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant (https://www.contributor-covenant.org/), and incorporates best practices from codes of conduct in the Python community and other leading open source projects.
+
+## Questions?
+
+If you have questions about this Code of Conduct or how it applies to a specific situation, please reach out to the maintainers at **support@ragas.io** or through a project maintainer you trust.
+
+---
+
+**Last Updated**: November 2024
+
+We appreciate your participation in making Ragas a welcoming and inclusive community for everyone.

CONTRIBUTING.md

@@ -0,0 +1,421 @@
+# Development Guide for Ragas Monorepo
+
+This comprehensive guide covers development workflows for the Ragas monorepo, designed for both human developers and AI agents.
+
+## Quick Start (for Developers)
+
+```bash
+# 1. Clone and enter the repository
+git clone https://github.com/vibrantlabsai/ragas.git
+
+# 2. Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# 3. Choose your installation type:
+
+# RECOMMENDED: Minimal dev setup (fast)
+make install-minimal
+
+# FULL: Complete dev environment (comprehensive)
+make install
+
+# 4. Verify everything works
+make check
+
+# 5. Start developing!
+make help  # See all available commands
+```
+
+## Quick Start (for AI Agents)
+
+AI agents working with this codebase should use these standardized commands:
+
+```bash
+# Essential commands for AI development
+make help           # See all available targets
+make install-minimal # Minimal dev setup (fast)
+make install        # Full environment (modern uv sync)
+make check          # Quick health check (format + type)
+make test           # Run all tests
+make run-ci         # Full CI pipeline locally
+
+# Individual development tasks
+make format         # Format and lint all code
+make type           # Type check all code
+make clean          # Clean generated files
+```
+
+**Key Points for AI Agents:**
+- Always use `make` commands rather than direct tool invocation
+- Use `uv run` prefix for any direct Python tool usage
+- Check `make help` for the complete command reference
+- The CI pipeline uses the same commands as local development
+
+## Monorepo Architecture
+
+This repository is organized as a single project with integrated experimental features:
+
+```sh
+/                              # Main ragas project
+├── src/ragas/                 # Main source code
+│   └── experimental/          # Experimental features
+├── tests/                     # Tests (unit, e2e, benchmarks)
+│   └── experimental/          # Experimental tests
+├── examples/                  # Example code
+├── pyproject.toml             # Dependencies and configuration
+├── docs/                      # Documentation
+├── .github/workflows/         # CI/CD pipeline
+├── Makefile                   # Build commands
+└── CLAUDE.md                  # AI assistant instructions
+```
+
+### Project Components
+- **Ragas Core**: The main evaluation toolkit for LLM applications (in `src/ragas/`)
+- **Ragas Experimental**: Advanced features integrated at `src/ragas/experimental/`
+- **Infrastructure**: Single CI/CD, documentation, and build system
+
+### Examples Package (ragas-examples)
+- Lives under `examples/` as an installable package `ragas-examples`
+- Published independently to PyPI via GitHub Actions workflow `publish-examples.yml`
+- Versioning via Git tags with prefix `examples-v` (e.g., `examples-v0.1.0`)
+- Local development: `uv pip install -e . -e ./examples`
+- Run examples: `python -m ragas_examples.benchmark_llm.prompt`
+
+## Development Environment Setup
+
+### Prerequisites
+- Python 3.9+ 
+- [uv](https://docs.astral.sh/uv/) (recommended) or pip
+- Git
+
+### Setup Process
+
+#### Option 1: Using Make (Recommended)
+```bash
+# Recommended: Minimal dev setup
+make install-minimal
+
+# Full: Complete environment
+make install
+```
+
+#### Option 2: Manual Setup
+```bash
+# Install uv if not available
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Minimal dev: Core + essential dev tools
+uv pip install -e ".[dev-minimal]"
+
+# Full dev: Everything (uses modern uv sync)
+uv sync --group dev
+```
+
+#### Which Option to Choose?
+
+**Use `make install-minimal` if you're:**
+- Contributing to ragas development
+- Need testing and linting tools
+- Want fast CI/CD builds
+- Working on code quality, docs, or basic features
+
+**Use `make install` if you're:**
+- Working on ML features requiring the full stack
+- Need observability tools (Phoenix, MLflow)
+- Developing with notebooks and advanced integrations
+- Want the complete development environment
+
+#### Installation Methods Explained
+
+- **`install-minimal`**: Uses `uv pip install -e ".[dev-minimal]"` for selective minimal dev dependencies
+- **`install`**: Uses `uv sync --group dev` for complete modern dependency management
+
+### Verification
+```bash
+make check  # Runs format + type checking
+make test   # Runs all tests
+```
+
+## Available Commands Reference
+
+Run `make help` to see all targets. Here are the essential commands:
+
+### Setup & Installation
+- `make install-minimal` - Install minimal dev setup (recommended)
+- `make install` - Install full environment with uv sync (complete)
+
+### Code Quality
+- `make format` - Format and lint all code (includes unused import cleanup)
+- `make type` - Type check all code
+- `make check` - Quick health check (format + type, no tests)
+
+### Testing
+- `make test` - Run all unit tests
+- `make test-e2e` - Run end-to-end tests
+- `make benchmarks` - Run performance benchmarks
+- `make benchmarks-docker` - Run benchmarks in Docker
+
+### CI/Build
+- `make run-ci` - Run complete CI pipeline locally
+- `make clean` - Clean all generated files
+
+### Documentation
+- `make build-docs` - Build all documentation
+- `make build-docs-pdf` - Build documentation with PDF export (requires WeasyPrint)
+- `make serve-docs` - Serve documentation locally
+- See `docs/community/pdf_export.md` for PDF export details and limitations
+
+## Development Workflows
+
+### Daily Development
+```bash
+# 1. Start your work
+git checkout -b feature/your-feature
+
+# 2. Make changes to code
+
+# 3. Check your work
+make check           # Format and type check
+make test            # Run tests
+
+# 4. Commit and push
+git add .
+git commit -m "feat: your feature description"
+git push origin feature/your-feature
+```
+
+### Before Submitting PR
+```bash
+make run-ci          # Run full CI pipeline
+# Ensure all checks pass before creating PR
+```
+
+#### Development Workflow
+```bash
+# Use the Makefile for all development
+make help           # See available commands
+make format         # Format all code (core + experimental)
+make type           # Type check all code
+make test           # Run all tests (core + experimental)
+make check          # Quick format + type check
+make run-ci         # Run full CI pipeline
+
+# Or use direct commands for specific tasks
+uv run pytest tests/unit          # Run core unit tests
+uv run pytest tests/unit  # Run unit tests
+uv run pyright src               # Type check source code
+```
+
+## Testing Strategy
+
+### Test Types
+1. **Unit Tests**: Fast, isolated tests for individual components
+2. **End-to-End Tests**: Integration tests for complete workflows
+3. **Benchmarks**: Performance tests for evaluation metrics
+
+### Running Tests
+```bash
+# All tests
+make test
+
+# Specific test categories
+uv run pytest tests/unit
+uv run pytest tests/e2e
+
+# With coverage or specific options
+uv run pytest tests/unit -k "test_name"
+```
+
+### Test Organization
+- **Unit Tests**: `tests/unit/`
+- **End-to-End Tests**: `tests/e2e/`
+- **Benchmarks**: `tests/benchmarks/`
+
+## Code Quality & CI/CD
+
+### Code Quality Pipeline
+The `make format` command runs:
+1. **isort**: Import sorting
+2. **ruff format**: Code formatting
+3. **ruff --fix-only**: Auto-fix issues (including unused imports)
+4. **ruff check**: Final linting validation
+
+### Type Checking
+```bash
+make type  # Type check all code with pyright
+```
+
+### CI/CD Pipeline
+Our GitHub Actions CI runs:
+1. **Dependency Installation**: Using uv for consistent environments
+2. **Code Quality Checks**: Format and type validation
+3. **Testing**: Unit and integration tests across Python 3.9-3.12
+4. **Multi-OS Testing**: Ubuntu, macOS, Windows
+
+### Local CI Simulation
+```bash
+make run-ci  # Runs: format + type + test
+```
+
+## Project Guidelines
+
+### Ragas Project
+- **Language**: Python with type hints
+- **Testing**: pytest with nbmake for notebook tests
+- **Style**: Google-style docstrings
+- **Architecture**: Modular metrics and evaluation framework with experimental features
+- **Dependencies**: All defined in `pyproject.toml`
+
+### Adding Dependencies
+- **All features**: Add to `pyproject.toml`
+- **Always**: Test with `make install` and `make test`
+
+## Troubleshooting
+
+### Common Issues
+
+#### Import Errors
+```bash
+# Reinstall in development mode
+make install
+```
+
+#### Test Failures
+```bash
+# Run specific failing test
+uv run pytest tests/unit/test_specific.py -v
+
+# Check experimental test dependencies
+uv run pytest tests/unit --collect-only
+```
+
+#### Formatting Issues
+```bash
+# Fix formatting
+make format
+
+# Check specific files
+uv run ruff check path/to/file.py --fix
+```
+
+#### CI Failures
+```bash
+# Run the same checks locally
+make run-ci
+
+# Individual checks
+make format  # Must pass
+make type    # Must pass  
+make test    # Must pass
+```
+
+### Development Environment Issues
+
+#### uv Not Found
+```bash
+# Install uv
+curl -LsSf https://astral.sh/uv/install.sh | sh
+# or use pip: pip install uv
+```
+
+#### Dependency Conflicts
+```bash
+# Clean install
+make clean
+make install
+```
+
+### Getting Help
+- **Documentation**: Check `CLAUDE.md` for AI assistant guidance
+- **Commands**: Run `make help` for all available targets
+- **Issues**: Check existing GitHub issues or create a new one
+
+## Contributing Guidelines
+
+### Pull Request Process
+1. **Fork** the repository
+2. **Create** a feature branch: `git checkout -b feature/amazing-feature`
+3. **Develop** using the workflows above
+4. **Test** thoroughly: `make run-ci`
+5. **Submit** a pull request with clear description
+
+### Commit Message Format
+```
+feat: add new evaluation metric
+fix: resolve import error in experimental
+docs: update development guide
+test: add unit tests for metric base
+```
+
+### Code Review Checklist
+- [ ] All tests pass (`make test`)
+- [ ] Code is formatted (`make format`)
+- [ ] Type checking passes (`make type`)
+- [ ] Documentation is updated
+- [ ] Appropriate tests are included
+
+## AI Agent Best Practices
+
+### Recommended Workflow for AI Agents
+1. **Understand the task**: Read relevant documentation and code
+2. **Plan the approach**: Identify which project(s) need changes
+3. **Use standardized commands**: Always prefer `make` targets
+4. **Test incrementally**: Use `make check` frequently during development
+5. **Validate thoroughly**: Run `make run-ci` before completing
+
+### Command Patterns for AI Agents
+```bash
+# Always start with understanding the current state
+make help
+ls -la  # Check current directory structure
+
+# For code changes
+make format  # After making changes
+make test    # Verify functionality
+
+# For project-specific work
+make help                       # See available commands
+
+# For investigation
+uv run pytest --collect-only  # See available tests
+uv run ruff check --no-fix    # Check issues without fixing
+```
+
+### File Modification Guidelines
+- **Prefer editing** existing files over creating new ones
+- **Use project conventions** (check similar files for patterns)
+- **Update tests** when modifying functionality
+- **Follow existing code style** (enforced by `make format`)
+
+---
+#### Python 3.13 on macOS ARM: NumPy fails to install (builds from source)
+
+- Symptom: `make install` attempts to build `numpy==2.0.x` from source on Python 3.13 (no prebuilt wheel), failing with C/C++ errors.
+- Status: Ragas CI supports Python 3.9–3.12. Python 3.13 is not officially supported yet.
+
+Workarounds:
+1) Recommended: use Python 3.12
+```bash
+uv python install 3.12
+rm -rf .venv
+uv venv -p 3.12
+make install
+```
+
+2) Stay on 3.13 (best effort):
+- Install minimal first, then add extras as needed:
+```bash
+rm -rf .venv
+uv venv -p 3.13
+make install-minimal
+uv pip install "ragas[tracing,gdrive,ai-frameworks]"
+```
+- Or force a newer NumPy wheel:
+```bash
+uv pip install "numpy>=2.1" --only-binary=:all:
+```
+If conflicts pin NumPy to 2.0.x, temporarily set `numpy>=2.1` in `pyproject.toml` and run `uv sync --group dev`.
+
+**Happy coding! 🚀**
+
+For additional context and instructions specific to AI assistants, see [CLAUDE.md](./CLAUDE.md).

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2023] [Vibrant Labs]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

Makefile

@@ -0,0 +1,202 @@
+GIT_ROOT ?= $(shell git rev-parse --show-toplevel)
+
+# Optionally show commands being executed with V=1
+Q := $(if $(V),,@)
+
+# Common paths
+RAGAS_PATHS := src tests docs
+
+help: ## Show all Makefile targets
+	$(Q)grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[33m%-30s\033[0m %s\n", $$1, $$2}'
+
+# =============================================================================
+# SETUP & INSTALLATION
+# =============================================================================
+
+setup-venv: ## Set up uv virtual environment
+	@echo "Setting up uv virtual environment..."
+	$(Q)VIRTUAL_ENV= uv venv
+	@echo "Virtual environment created at .venv"
+	@echo "To activate: source .venv/bin/activate"
+
+install-minimal: ## Install minimal dev dependencies (fast setup - 79 packages)
+	@echo "Installing minimal development dependencies (fast setup)..."
+	@if [ ! -d ".venv" ]; then \
+		echo "Virtual environment not found, creating one..."; \
+		$(MAKE) setup-venv; \
+	fi
+	@echo "Installing core ragas + essential dev tools..."
+	$(Q)uv pip install -e ".[dev-minimal]"
+	@echo "Setting up pre-commit hooks..."
+	$(Q)uv run pre-commit install
+	@echo "Minimal installation complete! (79 packages)"
+	@echo "Note: For full features including ML packages, use 'make install'"
+
+install: ## Install full dependencies with uv sync (backward compatible - modern approach)
+	@echo "Installing full development dependencies with uv sync..."
+	@if [ ! -d ".venv" ]; then \
+		echo "Virtual environment not found, creating one..."; \
+		$(MAKE) setup-venv; \
+	fi
+	@echo "Installing ragas with full dev environment..."
+	$(Q)VIRTUAL_ENV= uv sync --group dev
+	@echo "Setting up pre-commit hooks..."
+	$(Q)uv run pre-commit install
+	@echo "Full installation complete! (Modern uv sync approach)"
+
+# =============================================================================
+# CODE QUALITY
+# =============================================================================
+
+.PHONY: help setup-venv install-minimal install format type check clean test test-e2e benchmarks benchmarks-docker run-ci run-ci-fast run-ci-format-check run-ci-type run-ci-tests build-docs serve-docs
+format: ## Format and lint all code
+	@echo "Formatting and linting all code..."
+	@echo "(ruff format) Formatting ragas..."
+	$(Q)uv run --active ruff format src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+	@echo "(ruff) Auto-fixing ragas (includes import sorting and unused imports)..."
+	$(Q)uv run --active ruff check src tests docs --exclude src/ragas/_version.py --fix-only --config pyproject.toml
+	@echo "(ruff) Final linting check for ragas..."
+	$(Q)uv run --active ruff check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+
+type: ## Type check all code
+	@echo "Type checking all code..."
+	@echo "(pyright) Typechecking ragas..."
+	$(Q)PYRIGHT_PYTHON_FORCE_VERSION=latest uv run --active pyright -p pyproject.toml src
+
+check: format type ## Quick health check (format + type, no tests)
+	@echo "Code quality check complete!"
+
+# =============================================================================
+# BENCHMARKS
+# =============================================================================
+benchmarks: ## Run all benchmarks locally
+	@echo "Running all benchmarks..."
+	@echo "Running evaluation benchmarks..."
+	$(Q)cd $(GIT_ROOT)/tests/benchmarks && uv run python benchmark_eval.py
+	@echo "Running testset generation benchmarks..."
+	$(Q)cd $(GIT_ROOT)/tests/benchmarks && uv run python benchmark_testsetgen.py
+
+benchmarks-docker: ## Run benchmarks in docker
+	@echo "Running benchmarks in docker..."
+	$(Q)cd $(GIT_ROOT) || exit 1
+	docker buildx build --build-arg OPENAI_API_KEY=$(OPENAI_API_KEY) -t ragas-benchmark -f $(GIT_ROOT)/tests/benchmarks/Dockerfile .
+	docker inspect ragas-benchmark:latest | jq ".[0].Size" | numfmt --to=si
+
+benchmarks-test: ## Run benchmarks for ragas unit tests
+	@echo "Running ragas unit tests with timing benchmarks..."
+	$(Q)uv run --active pytest --nbmake tests/unit --durations=0 -v $(shell if [ -n "$(k)" ]; then echo "-k $(k)"; fi)
+
+# =============================================================================
+# CI/BUILD
+# =============================================================================
+
+run-ci: ## Run complete CI pipeline (mirrors GitHub CI exactly)
+	@echo "Running complete CI pipeline..."
+	@echo "Format check..."
+	$(Q)uv run --active ruff format --check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+	$(Q)uv run --active ruff check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+	@echo "Type check..."
+	$(Q)$(MAKE) type
+	@echo "Unit tests..."
+	$(Q)__RAGAS_DEBUG_TRACKING=true RAGAS_DO_NOT_TRACK=true uv run --active pytest --nbmake tests/unit --dist loadfile -n auto
+	@echo "All CI checks passed!"
+
+run-ci-format-check: ## Run format check in dry-run mode (like GitHub CI)
+	@echo "Running format check (dry-run, like GitHub CI)..."
+	@echo "Checking ragas formatting..."
+	$(Q)uv run --active ruff format --check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+	$(Q)uv run --active ruff check src docs tests --exclude src/ragas/_version.py --config pyproject.toml
+
+run-ci-type: ## Run type checking (matches GitHub CI)
+	@echo "Running type checking (matches GitHub CI)..."
+	$(Q)$(MAKE) type
+
+run-ci-tests: ## Run all tests with CI options
+	@echo "Running all tests with CI options..."
+	$(Q)__RAGAS_DEBUG_TRACKING=true RAGAS_DO_NOT_TRACK=true pytest --nbmake tests/unit --dist loadfile -n auto
+
+run-ci-fast: ## Fast CI check for quick local validation (2-3 minutes)
+	@echo "Running fast CI check for quick feedback..."
+	@echo "Format check..."
+	$(Q)uv run --active ruff format --check src tests docs --exclude src/ragas/_version.py --config pyproject.toml
+	$(Q)uv run --active ruff check src docs tests --exclude src/ragas/_version.py --config pyproject.toml
+	@echo "Core unit tests (no nbmake for speed)..."
+	$(Q)uv run --active pytest tests/unit --dist loadfile -n auto -x
+	@echo "Fast CI check completed!"
+
+clean: ## Clean all generated files
+	@echo "Cleaning all generated files..."
+	$(Q)find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
+	$(Q)rm -rf site/ docs/site/ .mypy_cache .pytest_cache .ruff_cache
+	$(Q)rm -rf dist/ build/ *.egg-info/ src/*.egg-info/
+	$(Q)rm -rf .coverage htmlcov/ .tox/ .venv/
+	$(Q)find . -name '*.log' -delete
+	$(Q)find . -name '.DS_Store' -delete
+	$(Q)find . -name 'temp*' -type d -exec rm -rf {} + 2>/dev/null || true
+	$(Q)find . -name '.tmp*' -type d -exec rm -rf {} + 2>/dev/null || true
+	@echo "Cleanup complete!"
+
+# =============================================================================
+# TESTING
+# =============================================================================
+
+test: ## Run all unit tests
+	@echo "Running all unit tests..."
+	$(Q)uv run --active pytest tests/unit $(shell if [ -n "$(k)" ]; then echo "-k $(k)"; fi)
+
+test-all: ## Run all unit tests (including notebooks)
+	@echo "Running all unit tests (including notebooks)..."
+	$(Q)uv run --active pytest --nbmake tests/unit $(shell if [ -n "$(k)" ]; then echo "-k $(k)"; fi)
+
+test-e2e: ## Run all end-to-end tests
+	@echo "Running all end-to-end tests..."
+	$(Q)uv run --active pytest --nbmake tests/e2e -s
+
+# =============================================================================
+# DOCUMENTATION
+# =============================================================================
+
+build-docs: ## Build all documentation
+	@echo "Building all documentation..."
+	@echo "Converting ipynb notebooks to md files..."
+	$(Q)MKDOCS_CI=true uv run python $(GIT_ROOT)/docs/ipynb_to_md.py
+	@echo "Building ragas documentation..."
+	$(Q)MKDOCS_CI=false uv run --group docs mkdocs build
+
+
+check-pdf-deps: ## Check if WeasyPrint is properly installed with all dependencies
+	@echo "Checking if WeasyPrint is properly installed..."
+	@uv run --group docs-pdf python -c "import weasyprint; weasyprint.HTML(string='<h1>Test</h1>').write_pdf(target=None)" 2>/dev/null && \
+		echo "WeasyPrint is installed and all dependencies are available" || \
+		(echo ""; \
+		echo "WeasyPrint is not installed or has missing system dependencies"; \
+		echo ""; \
+		echo "Setup Instructions: https://doc.courtbouillon.org/weasyprint/stable/first_steps.html"; \
+		echo "Troubleshooting: https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#troubleshooting"; \
+		echo ""; \
+		exit 1)
+
+
+check-mermaid-deps: ## Check Mermaid CLI is available and can render a diagram
+	@command -v node >/dev/null || (echo "Node.js is required for Mermaid PDF rendering"; exit 1)
+	@command -v mmdc >/dev/null || (echo "Missing 'mmdc' (Mermaid CLI). Mermaid diagrams in PDF depend on Mermaid CLI."; exit 1)
+	@tmp_dir="$$(mktemp -d)"; \
+	printf "graph TD\n  A-->B\n" > "$$tmp_dir/diag.mmd"; \
+	mmdc -i "$$tmp_dir/diag.mmd" -o "$$tmp_dir/diag.svg" >/dev/null 2>&1 || \
+		(echo "Mermaid CLI found, but rendering failed (mmdc couldn't produce SVG)."; rm -rf "$$tmp_dir"; exit 1); \
+	test -s "$$tmp_dir/diag.svg" || \
+		(echo "Mermaid CLI ran but produced an empty SVG."; rm -rf "$$tmp_dir"; exit 1); \
+	rm -rf "$$tmp_dir"; \
+	echo "Mermaid CLI is installed and can render diagrams"
+
+	
+build-docs-pdf: check-pdf-deps check-mermaid-deps ## Build documentation with PDF export (requires WeasyPrint)
+	@echo "Building documentation with PDF export..."
+	$(Q)MKDOCS_CI=false ENABLE_PDF_EXPORT=1 uv run --group docs --group docs-pdf mkdocs build -f mkdocs-pdf.yml
+	@echo "PDF generated at: site/pdf/document.pdf"
+
+
+
+serve-docs: ## Build and serve documentation locally
+	$(Q)MKDOCS_CI=false uv run --group docs mkdocs serve --dirtyreload
+

README.md

@@ -0,0 +1,194 @@
+<h1 align="center">
+  <img style="vertical-align:middle" height="200"
+  src="https://raw.githubusercontent.com/vibrantlabsai/ragas/main/docs/_static/imgs/logo.png">
+</h1>
+<p align="center">
+  <i>Supercharge Your LLM Application Evaluations 🚀</i>
+</p>
+
+<p align="center">
+    <a href="https://github.com/vibrantlabsai/ragas/releases">
+        <img alt="Latest release" src="https://img.shields.io/github/release/vibrantlabsai/ragas.svg">
+    </a>
+    <a href="https://www.python.org/">
+        <img alt="Made with Python" src="https://img.shields.io/badge/Made%20with-Python-1f425f.svg?color=purple">
+    </a>
+    <a href="https://github.com/vibrantlabsai/ragas/blob/master/LICENSE">
+        <img alt="License Apache-2.0" src="https://img.shields.io/github/license/vibrantlabsai/ragas.svg?color=green">
+    </a>
+    <a href="https://pypi.org/project/ragas/">
+        <img alt="Ragas Downloads per month" src="https://static.pepy.tech/badge/ragas/month">
+    </a>
+    <a href="https://discord.gg/5djav8GGNZ">
+        <img alt="Join Ragas community on Discord" src="https://img.shields.io/discord/1119637219561451644">
+    </a>
+    <a target="_blank" href="https://deepwiki.com/vibrantlabsai/ragas">
+      <img 
+        src="https://devin.ai/assets/deepwiki-badge.png" 
+        alt="Ask DeepWiki.com" 
+        height="20" 
+      />
+    </a>
+</p>
+
+<h4 align="center">
+    <p>
+        <a href="https://docs.ragas.io/">Documentation</a> |
+        <a href="#fire-quickstart">Quick start</a> |
+        <a href="https://discord.gg/5djav8GGNZ">Join Discord</a> |
+        <a href="https://blog.ragas.io/">Blog</a> |
+        <a href="https://newsletter.ragas.io/">NewsLetter</a> |
+        <a href="https://www.ragas.io/careers">Careers</a>
+    <p>
+</h4>
+
+Objective metrics, intelligent test generation, and data-driven insights for LLM apps
+
+Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows.
+Don't have a test dataset ready? We also do production-aligned test set generation.
+
+## Key Features
+
+- 🎯 Objective Metrics: Evaluate your LLM applications with precision using both LLM-based and traditional metrics.
+- 🧪 Test Data Generation: Automatically create comprehensive test datasets covering a wide range of scenarios.
+- 🔗 Seamless Integrations: Works flawlessly with popular LLM frameworks like LangChain and major observability tools.
+- 📊 Build feedback loops: Leverage production data to continually improve your LLM applications.
+
+## :shield: Installation
+
+Pypi:
+
+```bash
+pip install ragas
+```
+
+Alternatively, from source:
+
+```bash
+pip install git+https://github.com/vibrantlabsai/ragas
+```
+
+## :fire: Quickstart
+
+### Clone a Complete Example Project
+
+The fastest way to get started is to use the `ragas quickstart` command:
+
+```bash
+# List available templates
+ragas quickstart
+
+# Create a RAG evaluation project
+ragas quickstart rag_eval
+
+# Specify where you want to create it.
+ragas quickstart rag_eval -o ./my-project
+```
+
+Available templates:
+- `rag_eval` - Evaluate RAG systems
+
+Coming Soon:
+- `agent_evals` - Evaluate AI agents
+- `benchmark_llm` - Benchmark and compare LLMs
+- `prompt_evals` - Evaluate prompt variations
+- `workflow_eval` - Evaluate complex workflows
+
+### Evaluate your LLM App
+
+`ragas` comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using `DiscreteMetric`:
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from ragas.metrics import DiscreteMetric
+from ragas.llms import llm_factory
+
+# Setup your LLM
+client = AsyncOpenAI()
+llm = llm_factory("gpt-4o", client=client)
+
+# Create a custom aspect evaluator
+metric = DiscreteMetric(
+    name="summary_accuracy",
+    allowed_values=["accurate", "inaccurate"],
+    prompt="""Evaluate if the summary is accurate and captures key information.
+
+Response: {response}
+
+Answer with only 'accurate' or 'inaccurate'."""
+)
+
+# Score your application's output
+async def main():
+    score = await metric.ascore(
+        llm=llm,
+        response="The summary of the text is..."
+    )
+    print(f"Score: {score.value}")  # 'accurate' or 'inaccurate'
+    print(f"Reason: {score.reason}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())
+```
+
+> **Note**: Make sure your `OPENAI_API_KEY` environment variable is set.
+
+Find the complete [Quickstart Guide](https://docs.ragas.io/en/latest/getstarted/quickstart)
+
+## Want help in improving your AI application using evals?
+
+In the past 2 years, we have seen and helped improve many AI applications using evals. If you want help with improving and scaling up your AI application using evals.
+
+🔗 Book a [slot](https://cal.com/team/vibrantlabs/app) or drop us a line: [founders@vibrantlabs.com](mailto:founders@vibrantlabs.com).
+
+## 🫂 Community
+
+If you want to get more involved with Ragas, check out our [discord server](https://discord.gg/5qGUJ6mh7C). It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.
+
+## Contributors
+
+```yml
++----------------------------------------------------------------------------+
+|     +----------------------------------------------------------------+     |
+|     | Developers: Those who built with `ragas`.                      |     |
+|     | (You have `import ragas` somewhere in your project)            |     |
+|     |     +----------------------------------------------------+     |     |
+|     |     | Contributors: Those who make `ragas` better.       |     |     |
+|     |     | (You make PR to this repo)                         |     |     |
+|     |     +----------------------------------------------------+     |     |
+|     +----------------------------------------------------------------+     |
++----------------------------------------------------------------------------+
+```
+
+We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.
+
+1. Fork the repository
+2. Create your feature branch (git checkout -b feature/AmazingFeature)
+3. Commit your changes (git commit -m 'Add some AmazingFeature')
+4. Push to the branch (git push origin feature/AmazingFeature)
+5. Open a Pull Request
+
+## 🔍 Open Analytics
+
+At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.
+
+✅ No personal or company-identifying information
+
+✅ Open-source data collection [code](./src/ragas/_analytics.py)
+
+✅ Publicly available aggregated [data](https://github.com/vibrantlabsai/ragas/issues/49)
+
+To opt-out, set the `RAGAS_DO_NOT_TRACK` environment variable to `true`.
+
+### Cite Us
+
+```
+@misc{ragas2024,
+  author       = {VibrantLabs},
+  title        = {Ragas: Supercharge Your LLM Application Evaluations},
+  year         = {2024},
+  howpublished = {\url{https://github.com/vibrantlabsai/ragas}},
+}
+```

README.wehub.md

@@ -0,0 +1,7 @@
+# WeHub 来源说明
+
+- 原始项目：`vibrantlabsai/ragas`
+- 原始仓库：https://github.com/vibrantlabsai/ragas
+- 导入方式：上游默认分支的最新快照
+- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
+- 本文件仅用于记录来源，不代表 WeHub 是原项目作者

SECURITY.md

@@ -0,0 +1,66 @@
+# Security Policy
+
+## Reporting Security Issues
+
+We take the security of RAGAS seriously. If you discover a security vulnerability in this project, please report it to us privately. **Do not report security vulnerabilities through public GitHub issues, discussions, or pull requests.**
+
+To report a vulnerability, please email us at founders@vibrantlabs.com. While not all details are mandatory, providing as much information as possible will assist us in effectively triaging and addressing the issue. Please include:
+
+- **Type of Issue**: (e.g., buffer overflow, SQL injection, cross-site scripting)
+- **Affected Versions**: List the versions of RAGAS impacted by this vulnerability.
+- **Affected Files**: Full paths of source files related to the issue.
+- **Location in Code**: The location of the affected source code (tag/branch/commit or direct URL).
+- **Configuration Details**: Any special configuration required to reproduce the issue.
+- **Environment**: (e.g., Linux / Windows / macOS)
+- **Reproduction Steps**: Step-by-step instructions to reproduce the issue.
+- **Proof-of-Concept or Exploit Code**: (if possible)
+- **Impact Assessment**: Description of the issue's impact and how an attacker might exploit it.
+- **Mitigation Suggestions**: If possible, offer suggestions or patches to mitigate the issue.
+
+This information will help us triage and address your report more quickly.
+
+## Supported Versions
+
+The following versions of RAGAS are currently being supported with security updates.
+
+| Version | Supported |
+| --- | --- |
+| 0.3.x   | :white_check_mark: |
+| 0.2.x   | :x: |
+| 0.1.x   | :x: |
+| < 0.1.x | :x: |
+
+## Security Update Policy
+
+Upon receiving a security report, we will:
+
+1. Acknowledge receipt within 48 hours.
+2. Investigate and verify the issue.
+3. Develop a fix and prepare a release.
+4. Coordinate with the reporter to validate the fix.
+5. Release the fix and update all affected parties.
+
+We aim to address critical issues within 7 days of disclosure.
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## Policy
+
+We follow the principle of [Coordinated Vulnerability Disclosure.](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure)
+
+## Acknowledgments
+
+We appreciate the efforts of security researchers and users who report vulnerabilities to us. Your contributions help improve the security of RAGAS.
+
+## References
+
+For more information on security reporting and policies, you may refer to:
+
+- [GitHub's Guide to Reporting Security Vulnerabilities](https://docs.github.com/en/code-security/security-advisories/guidelines-for-reporting-and-writing-about-security-vulnerabilities)
+- [Open Source Security Foundation (OpenSSF) Best Practices](https://bestpractices.coreinfrastructure.org/)
+
+---
+
+*This policy is subject to change without notice. Please refer to the latest version in our repository.*

docs/INSTALL

@@ -0,0 +1,29 @@
+# Documentation Development Setup
+
+## Prerequisites
+
+1. Install uv (Python package manager):
+   ```bash
+   curl -LsSf https://astral.sh/uv/install.sh | sh
+   ```
+
+## Running Documentation Locally
+
+1. Install documentation dependencies:
+   ```bash
+   uv sync --group docs --project ragas
+   ```
+
+2. Serve the documentation:
+   ```bash
+   source .venv/bin/activate && mkdocs serve --dirtyreload
+   ```
+
+The documentation will be available at http://127.0.0.1:8000/
+
+## Run dev
+
+You can also use the scripts command (after completing step 1 above and ensuring mkdocs is available in PATH):
+```bash
+./scripts/dev_docs.sh
+```

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/_static/annotated_data.json

@@ -0,0 +1,532 @@
+{
+  "helpfulness": [
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX"
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response effectively addresses the user's request for a better message by providing a clear, polite, and apologetic letter regarding the late rent payment, which improves the original message.",
+            "verdict": 1
+          },
+          "edited_output": {
+            "reason": "The response is helpful because it effectively addresses the user's request by providing a clear, polite, and apologetic letter regarding the late rent payment, improving the original message.",
+            "verdict": 1
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Can you tell me more about what's going on, and exactly what you want?"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Can you tell me more about what's going on, and exactly what you want?",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not directly address the user's request for help in improving their message. Instead, it asks for more information, which may not be helpful in fixing the user's original text.",
+            "verdict": 0
+          },
+          "edited_output": {
+            "reason": "The assistant's response doesn't directly help improve the user's message. Instead of offering a better version of the letter, it asks more questions, essentially turning the response into a comment rather than providing the requested improvement.",
+            "verdict": 0
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX"
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response effectively addresses the user's request for a better message by providing a clear, polite, and apologetic letter regarding the late rent payment, which improves the original message.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Can you tell me more about what's going on, and exactly what you want?"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Can you tell me more about what's going on, and exactly what you want?",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not directly address the user's request for help in improving their message. Instead, it asks for more information, which may not be helpful in fixing the original text.",
+            "verdict": 0
+          },
+          "edited_output": {
+            "reason": "The assistant's response doesn't directly help improve the user's message. Instead of offering a better version of the letter, it asks more questions, essentially turning the response into a comment rather than providing the requested improvement.",
+            "verdict": 0
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Here goes:  \n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Here goes:  \n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not improve or fix the user's message; it simply repeats it without any enhancements or corrections.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+        "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month.  It was a very unusual circumstance -- we were on a trip out of state, and our car broke down, so it wasn't possible to get the payment to you on time.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX"
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+            "response": "Dear Sir,\n\nWe're very sorry that our rent payment was late this month.  It was a very unusual circumstance -- we were on a trip out of state, and our car broke down, so it wasn't possible to get the payment to you on time.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response effectively incorporates the user's request to explain the late payment due to the car breakdown, making it more aligned with the user's needs.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+        "response": "Dear Sir,\n\nThe check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+            "response": "Dear Sir,\n\nThe check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not incorporate the user's request to avoid mentioning responsibility and does not address the reason for the late payment (car breakdown). Therefore, it is not fully helpful.",
+            "verdict": 0
+          },
+          "edited_output": {
+            "reason": "The assistant's response didn't include the reason for the late payment (car breakdown) as requested, and it also didn't avoid mentioning responsibility. Therefore, it wasn't fully helpful because it didn't meet all of the user's needs.",
+            "verdict": 0
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+        "response": "Well this is silly, but I think I got them:\n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Byzantine–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Byzantine Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Byzantine–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475."
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+            "response": "Well this is silly, but I think I got them:\n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Byzantine–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Byzantine Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Byzantine–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant did not fulfill the user's request to replace all occurrences of 'Byzantine' with 'Cotton Candy'. Instead, it left 'Byzantine' unchanged in multiple instances, which does not meet the user's requirement.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+        "response": "Of course the resulting text is no longer accurate, but here you go:\n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+            "response": "Of course the resulting text is no longer accurate, but here you go:\n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant successfully edited the paragraph by replacing all occurrences of 'Byzantine' with 'Cotton Candy', fulfilling the user's request.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+        "response": "Can you tell me more about why we're doing this?  Who's Cotton Candy Anyway?"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "I realize it's silly, but can you edit this paragraph for me to turn all occurrences of \"Byzantine\" into \"Cotton Candy\"?  \n\nThe borders of the empire fluctuated through several cycles of decline and recovery. During the reign of Justinian I (r. 527–565), the empire reached its greatest extent, after reconquering much of the historically Roman western Mediterranean coast, including North Africa, Italy and Rome, which it held for two more centuries. The Cotton Candy–Sasanian War of 602–628 exhausted the empire's resources, and during the Early Muslim conquests of the 7th century, it lost its richest provinces, Egypt and Syria, to the Rashidun Caliphate. During the Macedonian dynasty (10th–11th centuries), the empire expanded again and experienced the two-century long Macedonian Renaissance, which came to an end with the loss of much of Asia Minor to the Seljuk Turks after the Battle of Manzikert in 1071. This battle opened the way for the Turks to settle in Anatolia. The empire recovered during the Komnenian restoration, and by the 12th century, Constantinople was the largest and wealthiest city in Europe. The empire was delivered a mortal blow during the Fourth Crusade, when Constantinople was sacked in 1204 and the territories that the empire formerly governed were divided into competing Cotton Candy Greek and Latin realms. Despite the eventual recovery of Constantinople in 1261, the Cotton Candy Empire remained only one of several small rival states in the area for the final two centuries of its existence. Its remaining territories were progressively annexed by the Ottomans in the Cotton Candy–Ottoman wars over the 14th and 15th centuries. The fall of Constantinople to the Ottoman Empire in 1453 ended the Cotton Candy Empire. The Empire of Trebizond was conquered eight years later in the 1461 siege. The last of the successor states, the Principality of Theodoro, was conquered by the Ottomans in 1475.",
+            "response": "Can you tell me more about why we're doing this?  Who's Cotton Candy Anyway?",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not address the user's request to edit the paragraph by replacing occurrences of 'Byzantine' with 'Cotton Candy'. Instead, it asks unrelated questions, making it unhelpful.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+        "response": "I think it's:  \"apple bottom\""
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+            "response": "I think it's:  \"apple bottom\"",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant correctly identified the anagram 'pplae ttmobo' as 'apple bottom', which directly addresses the user's request.",
+            "verdict": 1
+          },
+          "edited_output": {
+            "reason": "The response correctly helps in answering the user's question by identifying the anagram \"pplae ttmobo\" as \"apple bottom.\"",
+            "verdict": 1
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+        "response": "I believe it's: \"pale tomboy\""
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+            "response": "I believe it's: \"pale tomboy\"",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant correctly identified the anagram 'pplae ttmobo' as 'pale tomboy', providing a helpful and accurate response to the user's query.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+        "response": "Dear Sir,\n\nI guess our rent was late, but it should arrive very soon.  No big deal and it won't happen again.\n\nBest,\nX"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks",
+            "response": "Dear Sir,\n\nI guess our rent was late, but it should arrive very soon.  No big deal and it won't happen again.\n\nBest,\nX",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response does not adequately address the user's request to improve the message. It simply acknowledges the late rent without enhancing the communication or providing a more polished version.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+        "response": "What do you mean?  Maybe tell me more about your situation?"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Can you figure out this anagram for me?  I'm not sure what it is:\n\npplae ttmobo",
+            "response": "What do you mean?  Maybe tell me more about your situation?",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not address the user's request for help with the anagram and instead asks for more information, which is unhelpful.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+        "response": "You'll end up with one full glass of liquid."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+            "response": "You'll end up with one full glass of liquid.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response correctly answers the user's query by indicating that combining half a glass of iced tea with half a glass of lemonade results in one full glass of liquid.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+        "response": "That's a drink called an Arnold Palmer."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+            "response": "That's a drink called an Arnold Palmer.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response correctly identifies the drink created by mixing iced tea and lemonade, which directly answers the user's query.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+        "response": "A great base for a soup!"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "If I take half a glass of iced tea and pour in half a glass of lemonade, what will I have?",
+            "response": "A great base for a soup!",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response does not address the user's query about mixing iced tea and lemonade, providing an irrelevant answer instead.",
+            "verdict": 0
+          },
+          "edited_output": {
+            "reason": "The response should have provided the name of the drink formed by mixing iced tea and lemonade, such as an \"Arnold Palmer.\" Instead, it gave an irrelevant answer, which was not helpful.",
+            "verdict": 0
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "Should I go on vacation to Sweden or Norway?",
+        "response": "Norway might be the better choice if you're really into exploring the outdoors.  But Sweden is a great choice for people who are looking to explore Scandinavia for more than stunning scenery.  Sweden is a better option if you want great food, good public transportation and possibly a bit of savings."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Should I go on vacation to Sweden or Norway?",
+            "response": "Norway might be the better choice if you're really into exploring the outdoors.  But Sweden is a great choice for people who are looking to explore Scandinavia for more than stunning scenery.  Sweden is a better option if you want great food, good public transportation and possibly a bit of savings.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response provides a clear comparison between Norway and Sweden, highlighting specific reasons for choosing each destination based on the user's interests, which is helpful for making a decision.",
+            "verdict": 1
+          },
+          "edited_output": {
+            "reason": "This response is helpful because it provides a clear comparison between Norway and Sweden, highlighting specific reasons for choosing each destination based on the user's interests, which aids in making a decision.",
+            "verdict": 1
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "Should I go on vacation to Sweden or Norway?",
+        "response": "Everyone agrees that Sweden is better.  It's officially the Kingdom of Sweden, a Nordic country in Northern Europe.  It borders Norway to the west and north, Finland to the east, and is connected to Denmark in the southwest by a bridge-tunnel across the Öresund Strait. At 450,295 square kilometres (173,860 sq mi), Sweden is the largest country in Northern Europe, the third-largest country in the European Union, and the fifth largest country in Europe. The capital and largest city is Stockholm."
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Should I go on vacation to Sweden or Norway?",
+            "response": "Everyone agrees that Sweden is better.  It's officially the Kingdom of Sweden, a Nordic country in Northern Europe.  It borders Norway to the west and north, Finland to the east, and is connected to Denmark in the southwest by a bridge-tunnel across the Öresund Strait. At 450,295 square kilometres (173,860 sq mi), Sweden is the largest country in Northern Europe, the third-largest country in the European Union, and the fifth largest country in Europe. The capital and largest city is Stockholm.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response provides factual information about Sweden but does not directly address the user's query about choosing between Sweden and Norway for a vacation.",
+            "verdict": 0
+          },
+          "edited_output": {
+            "reason": "The response provides factual information about Sweden but doesn't offer helpful guidance to the user in deciding between Sweden and Norway for their vacation. It doesn't directly address the user's query about which location to choose.",
+            "verdict": 0
+          }
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "Should I go on vacation to Sweden or Norway?",
+        "response": "They're both beautiful and interesting countries, but can also be quite cold, and rather expensive."
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "Should I go on vacation to Sweden or Norway?",
+            "response": "They're both beautiful and interesting countries, but can also be quite cold, and rather expensive.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The response does not provide a clear recommendation or guidance on whether to choose Sweden or Norway for a vacation, making it unhelpful for the user's query.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+        "response": "Can you tell me more about the details?"
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "\"can you fix this up better?\n\nhi sir im hoping you can forgive that its ok we missed rent but we sent the check on the 10th and it shoulbe there soon, well be on time next time for october and after, thanks\"\n\nAssistant:  Sure:\n\nDear Sir,\n\nWe're very sorry that our rent payment was late this month, and we take full responsibility for the mistake.  The check was sent to you on the 10th, and it should arrive very soon.  We won't let this happen again, and you can count on all of our future rent payments, to arrive on-time, starting in October.\n\nThanks for your understanding!\n\nBest,\nX \n\nHuman:  but dont say about responsibility, say we were stuck after our car broken down out of state on a trip",
+            "response": "Can you tell me more about the details?",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The assistant's response does not address the user's request for a revised message. Instead, it asks for more details, which is not helpful in fixing the original message.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    }
+  ]
+}

docs/_static/css/highlight_ipython3.css

@@ -0,0 +1,85 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-ipython3 .hll { background-color: #49483e }
+.highlight-ipython3 { background: #272822; color: #f8f8f2 }
+.highlight-ipython3 .c { color: #959077 } /* Comment */
+.highlight-ipython3 .err { color: #ed007e; background-color: #1e0010 } /* Error */
+.highlight-ipython3 .esc { color: #f8f8f2 } /* Escape */
+.highlight-ipython3 .g { color: #f8f8f2 } /* Generic */
+.highlight-ipython3 .k { color: #66d9ef } /* Keyword */
+.highlight-ipython3 .l { color: #ae81ff } /* Literal */
+.highlight-ipython3 .n { color: #f8f8f2 } /* Name */
+.highlight-ipython3 .o { color: #ff4689 } /* Operator */
+.highlight-ipython3 .x { color: #f8f8f2 } /* Other */
+.highlight-ipython3 .p { color: #f8f8f2 } /* Punctuation */
+.highlight-ipython3 .ch { color: #959077 } /* Comment.Hashbang */
+.highlight-ipython3 .cm { color: #959077 } /* Comment.Multiline */
+.highlight-ipython3 .cp { color: #959077 } /* Comment.Preproc */
+.highlight-ipython3 .cpf { color: #959077 } /* Comment.PreprocFile */
+.highlight-ipython3 .c1 { color: #959077 } /* Comment.Single */
+.highlight-ipython3 .cs { color: #959077 } /* Comment.Special */
+.highlight-ipython3 .gd { color: #ff4689 } /* Generic.Deleted */
+.highlight-ipython3 .ge { color: #f8f8f2; font-style: italic } /* Generic.Emph */
+.highlight-ipython3 .ges { color: #f8f8f2; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+.highlight-ipython3 .gr { color: #f8f8f2 } /* Generic.Error */
+.highlight-ipython3 .gh { color: #f8f8f2 } /* Generic.Heading */
+.highlight-ipython3 .gi { color: #a6e22e } /* Generic.Inserted */
+.highlight-ipython3 .go { color: #66d9ef } /* Generic.Output */
+.highlight-ipython3 .gp { color: #ff4689; font-weight: bold } /* Generic.Prompt */
+.highlight-ipython3 .gs { color: #f8f8f2; font-weight: bold } /* Generic.Strong */
+.highlight-ipython3 .gu { color: #959077 } /* Generic.Subheading */
+.highlight-ipython3 .gt { color: #f8f8f2 } /* Generic.Traceback */
+.highlight-ipython3 .kc { color: #66d9ef } /* Keyword.Constant */
+.highlight-ipython3 .kd { color: #66d9ef } /* Keyword.Declaration */
+.highlight-ipython3 .kn { color: #ff4689 } /* Keyword.Namespace */
+.highlight-ipython3 .kp { color: #66d9ef } /* Keyword.Pseudo */
+.highlight-ipython3 .kr { color: #66d9ef } /* Keyword.Reserved */
+.highlight-ipython3 .kt { color: #66d9ef } /* Keyword.Type */
+.highlight-ipython3 .ld { color: #e6db74 } /* Literal.Date */
+.highlight-ipython3 .m { color: #ae81ff } /* Literal.Number */
+.highlight-ipython3 .s { color: #e6db74 } /* Literal.String */
+.highlight-ipython3 .na { color: #a6e22e } /* Name.Attribute */
+.highlight-ipython3 .nb { color: #f8f8f2 } /* Name.Builtin */
+.highlight-ipython3 .nc { color: #a6e22e } /* Name.Class */
+.highlight-ipython3 .no { color: #66d9ef } /* Name.Constant */
+.highlight-ipython3 .nd { color: #a6e22e } /* Name.Decorator */
+.highlight-ipython3 .ni { color: #f8f8f2 } /* Name.Entity */
+.highlight-ipython3 .ne { color: #a6e22e } /* Name.Exception */
+.highlight-ipython3 .nf { color: #a6e22e } /* Name.Function */
+.highlight-ipython3 .nl { color: #f8f8f2 } /* Name.Label */
+.highlight-ipython3 .nn { color: #f8f8f2 } /* Name.Namespace */
+.highlight-ipython3 .nx { color: #a6e22e } /* Name.Other */
+.highlight-ipython3 .py { color: #f8f8f2 } /* Name.Property */
+.highlight-ipython3 .nt { color: #ff4689 } /* Name.Tag */
+.highlight-ipython3 .nv { color: #f8f8f2 } /* Name.Variable */
+.highlight-ipython3 .ow { color: #ff4689 } /* Operator.Word */
+.highlight-ipython3 .pm { color: #f8f8f2 } /* Punctuation.Marker */
+.highlight-ipython3 .w { color: #f8f8f2 } /* Text.Whitespace */
+.highlight-ipython3 .mb { color: #ae81ff } /* Literal.Number.Bin */
+.highlight-ipython3 .mf { color: #ae81ff } /* Literal.Number.Float */
+.highlight-ipython3 .mh { color: #ae81ff } /* Literal.Number.Hex */
+.highlight-ipython3 .mi { color: #ae81ff } /* Literal.Number.Integer */
+.highlight-ipython3 .mo { color: #ae81ff } /* Literal.Number.Oct */
+.highlight-ipython3 .sa { color: #e6db74 } /* Literal.String.Affix */
+.highlight-ipython3 .sb { color: #e6db74 } /* Literal.String.Backtick */
+.highlight-ipython3 .sc { color: #e6db74 } /* Literal.String.Char */
+.highlight-ipython3 .dl { color: #e6db74 } /* Literal.String.Delimiter */
+.highlight-ipython3 .sd { color: #e6db74 } /* Literal.String.Doc */
+.highlight-ipython3 .s2 { color: #e6db74 } /* Literal.String.Double */
+.highlight-ipython3 .se { color: #ae81ff } /* Literal.String.Escape */
+.highlight-ipython3 .sh { color: #e6db74 } /* Literal.String.Heredoc */
+.highlight-ipython3 .si { color: #e6db74 } /* Literal.String.Interpol */
+.highlight-ipython3 .sx { color: #e6db74 } /* Literal.String.Other */
+.highlight-ipython3 .sr { color: #e6db74 } /* Literal.String.Regex */
+.highlight-ipython3 .s1 { color: #e6db74 } /* Literal.String.Single */
+.highlight-ipython3 .ss { color: #e6db74 } /* Literal.String.Symbol */
+.highlight-ipython3 .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */
+.highlight-ipython3 .fm { color: #a6e22e } /* Name.Function.Magic */
+.highlight-ipython3 .vc { color: #f8f8f2 } /* Name.Variable.Class */
+.highlight-ipython3 .vg { color: #f8f8f2 } /* Name.Variable.Global */
+.highlight-ipython3 .vi { color: #f8f8f2 } /* Name.Variable.Instance */
+.highlight-ipython3 .vm { color: #f8f8f2 } /* Name.Variable.Magic */
+.highlight-ipython3 .il { color: #ae81ff } /* Literal.Number.Integer.Long */

docs/_static/css/highlight_ipython3_dark.css

@@ -0,0 +1,85 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-ipython3 .hll { background-color: #49483e }
+.highlight-ipython3 { background: #232629; color: #cccccc }
+.highlight-ipython3 .c { color: #777777; font-style: italic } /* Comment */
+.highlight-ipython3 .err { color: #a61717; background-color: #e3d2d2 } /* Error */
+.highlight-ipython3 .esc { color: #cccccc } /* Escape */
+.highlight-ipython3 .g { color: #cccccc } /* Generic */
+.highlight-ipython3 .k { color: #7686bb; font-weight: bold } /* Keyword */
+.highlight-ipython3 .l { color: #cccccc } /* Literal */
+.highlight-ipython3 .n { color: #cccccc } /* Name */
+.highlight-ipython3 .o { color: #cccccc } /* Operator */
+.highlight-ipython3 .x { color: #cccccc } /* Other */
+.highlight-ipython3 .p { color: #cccccc } /* Punctuation */
+.highlight-ipython3 .ch { color: #777777; font-style: italic } /* Comment.Hashbang */
+.highlight-ipython3 .cm { color: #777777; font-style: italic } /* Comment.Multiline */
+.highlight-ipython3 .cp { color: #777777; font-style: italic } /* Comment.Preproc */
+.highlight-ipython3 .cpf { color: #777777; font-style: italic } /* Comment.PreprocFile */
+.highlight-ipython3 .c1 { color: #777777; font-style: italic } /* Comment.Single */
+.highlight-ipython3 .cs { color: #777777; font-style: italic } /* Comment.Special */
+.highlight-ipython3 .gd { color: #cccccc } /* Generic.Deleted */
+.highlight-ipython3 .ge { color: #cccccc } /* Generic.Emph */
+.highlight-ipython3 .ges { color: #cccccc } /* Generic.EmphStrong */
+.highlight-ipython3 .gr { color: #cccccc } /* Generic.Error */
+.highlight-ipython3 .gh { color: #cccccc } /* Generic.Heading */
+.highlight-ipython3 .gi { color: #cccccc } /* Generic.Inserted */
+.highlight-ipython3 .go { color: #cccccc } /* Generic.Output */
+.highlight-ipython3 .gp { color: #ffffff } /* Generic.Prompt */
+.highlight-ipython3 .gs { color: #cccccc } /* Generic.Strong */
+.highlight-ipython3 .gu { color: #cccccc } /* Generic.Subheading */
+.highlight-ipython3 .gt { color: #cccccc } /* Generic.Traceback */
+.highlight-ipython3 .kc { color: #7686bb; font-weight: bold } /* Keyword.Constant */
+.highlight-ipython3 .kd { color: #7686bb; font-weight: bold } /* Keyword.Declaration */
+.highlight-ipython3 .kn { color: #7686bb; font-weight: bold } /* Keyword.Namespace */
+.highlight-ipython3 .kp { color: #7686bb; font-weight: bold } /* Keyword.Pseudo */
+.highlight-ipython3 .kr { color: #7686bb; font-weight: bold } /* Keyword.Reserved */
+.highlight-ipython3 .kt { color: #7686bb; font-weight: bold } /* Keyword.Type */
+.highlight-ipython3 .ld { color: #cccccc } /* Literal.Date */
+.highlight-ipython3 .m { color: #4FB8CC } /* Literal.Number */
+.highlight-ipython3 .s { color: #51cc99 } /* Literal.String */
+.highlight-ipython3 .na { color: #cccccc } /* Name.Attribute */
+.highlight-ipython3 .nb { color: #cccccc } /* Name.Builtin */
+.highlight-ipython3 .nc { color: #cccccc } /* Name.Class */
+.highlight-ipython3 .no { color: #cccccc } /* Name.Constant */
+.highlight-ipython3 .nd { color: #cccccc } /* Name.Decorator */
+.highlight-ipython3 .ni { color: #cccccc } /* Name.Entity */
+.highlight-ipython3 .ne { color: #cccccc } /* Name.Exception */
+.highlight-ipython3 .nf { color: #6a6aff } /* Name.Function */
+.highlight-ipython3 .nl { color: #cccccc } /* Name.Label */
+.highlight-ipython3 .nn { color: #cccccc } /* Name.Namespace */
+.highlight-ipython3 .nx { color: #e2828e } /* Name.Other */
+.highlight-ipython3 .py { color: #cccccc } /* Name.Property */
+.highlight-ipython3 .nt { color: #cccccc } /* Name.Tag */
+.highlight-ipython3 .nv { color: #7AB4DB; font-weight: bold } /* Name.Variable */
+.highlight-ipython3 .ow { color: #cccccc } /* Operator.Word */
+.highlight-ipython3 .pm { color: #cccccc } /* Punctuation.Marker */
+.highlight-ipython3 .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight-ipython3 .mb { color: #4FB8CC } /* Literal.Number.Bin */
+.highlight-ipython3 .mf { color: #4FB8CC } /* Literal.Number.Float */
+.highlight-ipython3 .mh { color: #4FB8CC } /* Literal.Number.Hex */
+.highlight-ipython3 .mi { color: #4FB8CC } /* Literal.Number.Integer */
+.highlight-ipython3 .mo { color: #4FB8CC } /* Literal.Number.Oct */
+.highlight-ipython3 .sa { color: #51cc99 } /* Literal.String.Affix */
+.highlight-ipython3 .sb { color: #51cc99 } /* Literal.String.Backtick */
+.highlight-ipython3 .sc { color: #51cc99 } /* Literal.String.Char */
+.highlight-ipython3 .dl { color: #51cc99 } /* Literal.String.Delimiter */
+.highlight-ipython3 .sd { color: #51cc99 } /* Literal.String.Doc */
+.highlight-ipython3 .s2 { color: #51cc99 } /* Literal.String.Double */
+.highlight-ipython3 .se { color: #51cc99 } /* Literal.String.Escape */
+.highlight-ipython3 .sh { color: #51cc99 } /* Literal.String.Heredoc */
+.highlight-ipython3 .si { color: #51cc99 } /* Literal.String.Interpol */
+.highlight-ipython3 .sx { color: #51cc99 } /* Literal.String.Other */
+.highlight-ipython3 .sr { color: #51cc99 } /* Literal.String.Regex */
+.highlight-ipython3 .s1 { color: #51cc99 } /* Literal.String.Single */
+.highlight-ipython3 .ss { color: #51cc99 } /* Literal.String.Symbol */
+.highlight-ipython3 .bp { color: #cccccc } /* Name.Builtin.Pseudo */
+.highlight-ipython3 .fm { color: #6a6aff } /* Name.Function.Magic */
+.highlight-ipython3 .vc { color: #7AB4DB; font-weight: bold } /* Name.Variable.Class */
+.highlight-ipython3 .vg { color: #BE646C; font-weight: bold } /* Name.Variable.Global */
+.highlight-ipython3 .vi { color: #7AB4DB; font-weight: bold } /* Name.Variable.Instance */
+.highlight-ipython3 .vm { color: #7AB4DB; font-weight: bold } /* Name.Variable.Magic */
+.highlight-ipython3 .il { color: #4FB8CC } /* Literal.Number.Integer.Long */

docs/_static/css/highlight_ipython3_light.css

@@ -0,0 +1,75 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-ipython3 .hll { background-color: #ffffcc }
+.highlight-ipython3 { background: #f8f8f8; }
+.highlight-ipython3 .c { color: #008800; font-style: italic } /* Comment */
+.highlight-ipython3 .err { border: 1px solid #FF0000 } /* Error */
+.highlight-ipython3 .k { color: #AA22FF; font-weight: bold } /* Keyword */
+.highlight-ipython3 .o { color: #666666 } /* Operator */
+.highlight-ipython3 .ch { color: #008800; font-style: italic } /* Comment.Hashbang */
+.highlight-ipython3 .cm { color: #008800; font-style: italic } /* Comment.Multiline */
+.highlight-ipython3 .cp { color: #008800 } /* Comment.Preproc */
+.highlight-ipython3 .cpf { color: #008800; font-style: italic } /* Comment.PreprocFile */
+.highlight-ipython3 .c1 { color: #008800; font-style: italic } /* Comment.Single */
+.highlight-ipython3 .cs { color: #008800; font-weight: bold } /* Comment.Special */
+.highlight-ipython3 .gd { color: #A00000 } /* Generic.Deleted */
+.highlight-ipython3 .ge { font-style: italic } /* Generic.Emph */
+.highlight-ipython3 .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+.highlight-ipython3 .gr { color: #FF0000 } /* Generic.Error */
+.highlight-ipython3 .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight-ipython3 .gi { color: #00A000 } /* Generic.Inserted */
+.highlight-ipython3 .go { color: #888888 } /* Generic.Output */
+.highlight-ipython3 .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
+.highlight-ipython3 .gs { font-weight: bold } /* Generic.Strong */
+.highlight-ipython3 .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight-ipython3 .gt { color: #0044DD } /* Generic.Traceback */
+.highlight-ipython3 .kc { color: #AA22FF; font-weight: bold } /* Keyword.Constant */
+.highlight-ipython3 .kd { color: #AA22FF; font-weight: bold } /* Keyword.Declaration */
+.highlight-ipython3 .kn { color: #AA22FF; font-weight: bold } /* Keyword.Namespace */
+.highlight-ipython3 .kp { color: #AA22FF } /* Keyword.Pseudo */
+.highlight-ipython3 .kr { color: #AA22FF; font-weight: bold } /* Keyword.Reserved */
+.highlight-ipython3 .kt { color: #00BB00; font-weight: bold } /* Keyword.Type */
+.highlight-ipython3 .m { color: #666666 } /* Literal.Number */
+.highlight-ipython3 .s { color: #BB4444 } /* Literal.String */
+.highlight-ipython3 .na { color: #BB4444 } /* Name.Attribute */
+.highlight-ipython3 .nb { color: #AA22FF } /* Name.Builtin */
+.highlight-ipython3 .nc { color: #0000FF } /* Name.Class */
+.highlight-ipython3 .no { color: #880000 } /* Name.Constant */
+.highlight-ipython3 .nd { color: #AA22FF } /* Name.Decorator */
+.highlight-ipython3 .ni { color: #999999; font-weight: bold } /* Name.Entity */
+.highlight-ipython3 .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
+.highlight-ipython3 .nf { color: #00A000 } /* Name.Function */
+.highlight-ipython3 .nl { color: #A0A000 } /* Name.Label */
+.highlight-ipython3 .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
+.highlight-ipython3 .nt { color: #008000; font-weight: bold } /* Name.Tag */
+.highlight-ipython3 .nv { color: #B8860B } /* Name.Variable */
+.highlight-ipython3 .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
+.highlight-ipython3 .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight-ipython3 .mb { color: #666666 } /* Literal.Number.Bin */
+.highlight-ipython3 .mf { color: #666666 } /* Literal.Number.Float */
+.highlight-ipython3 .mh { color: #666666 } /* Literal.Number.Hex */
+.highlight-ipython3 .mi { color: #666666 } /* Literal.Number.Integer */
+.highlight-ipython3 .mo { color: #666666 } /* Literal.Number.Oct */
+.highlight-ipython3 .sa { color: #BB4444 } /* Literal.String.Affix */
+.highlight-ipython3 .sb { color: #BB4444 } /* Literal.String.Backtick */
+.highlight-ipython3 .sc { color: #BB4444 } /* Literal.String.Char */
+.highlight-ipython3 .dl { color: #BB4444 } /* Literal.String.Delimiter */
+.highlight-ipython3 .sd { color: #BB4444; font-style: italic } /* Literal.String.Doc */
+.highlight-ipython3 .s2 { color: #BB4444 } /* Literal.String.Double */
+.highlight-ipython3 .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
+.highlight-ipython3 .sh { color: #BB4444 } /* Literal.String.Heredoc */
+.highlight-ipython3 .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
+.highlight-ipython3 .sx { color: #008000 } /* Literal.String.Other */
+.highlight-ipython3 .sr { color: #BB6688 } /* Literal.String.Regex */
+.highlight-ipython3 .s1 { color: #BB4444 } /* Literal.String.Single */
+.highlight-ipython3 .ss { color: #B8860B } /* Literal.String.Symbol */
+.highlight-ipython3 .bp { color: #AA22FF } /* Name.Builtin.Pseudo */
+.highlight-ipython3 .fm { color: #00A000 } /* Name.Function.Magic */
+.highlight-ipython3 .vc { color: #B8860B } /* Name.Variable.Class */
+.highlight-ipython3 .vg { color: #B8860B } /* Name.Variable.Global */
+.highlight-ipython3 .vi { color: #B8860B } /* Name.Variable.Instance */
+.highlight-ipython3 .vm { color: #B8860B } /* Name.Variable.Magic */
+.highlight-ipython3 .il { color: #666666 } /* Literal.Number.Integer.Long */

docs/_static/css/highlight_python.css

@@ -0,0 +1,85 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-ipython3 .hll { background-color: #49483e }
+.highlight-ipython3 { background: #272822; color: #f8f8f2 }
+.highlight-ipython3 .c { color: #959077 } /* Comment */
+.highlight-ipython3 .err { color: #ed007e; background-color: #1e0010 } /* Error */
+.highlight-ipython3 .esc { color: #f8f8f2 } /* Escape */
+.highlight-ipython3 .g { color: #f8f8f2 } /* Generic */
+.highlight-ipython3 .k { color: #66d9ef } /* Keyword */
+.highlight-ipython3 .l { color: #ae81ff } /* Literal */
+.highlight-ipython3 .n { color: #f8f8f2 } /* Name */
+.highlight-ipython3 .o { color: #ff4689 } /* Operator */
+.highlight-ipython3 .x { color: #f8f8f2 } /* Other */
+.highlight-ipython3 .p { color: #f8f8f2 } /* Punctuation */
+.highlight-ipython3 .ch { color: #959077 } /* Comment.Hashbang */
+.highlight-ipython3 .cm { color: #959077 } /* Comment.Multiline */
+.highlight-ipython3 .cp { color: #959077 } /* Comment.Preproc */
+.highlight-ipython3 .cpf { color: #959077 } /* Comment.PreprocFile */
+.highlight-ipython3 .c1 { color: #959077 } /* Comment.Single */
+.highlight-ipython3 .cs { color: #959077 } /* Comment.Special */
+.highlight-ipython3 .gd { color: #ff4689 } /* Generic.Deleted */
+.highlight-ipython3 .ge { color: #f8f8f2; font-style: italic } /* Generic.Emph */
+.highlight-ipython3 .ges { color: #f8f8f2; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+.highlight-ipython3 .gr { color: #f8f8f2 } /* Generic.Error */
+.highlight-ipython3 .gh { color: #f8f8f2 } /* Generic.Heading */
+.highlight-ipython3 .gi { color: #a6e22e } /* Generic.Inserted */
+.highlight-ipython3 .go { color: #66d9ef } /* Generic.Output */
+.highlight-ipython3 .gp { color: #ff4689; font-weight: bold } /* Generic.Prompt */
+.highlight-ipython3 .gs { color: #f8f8f2; font-weight: bold } /* Generic.Strong */
+.highlight-ipython3 .gu { color: #959077 } /* Generic.Subheading */
+.highlight-ipython3 .gt { color: #f8f8f2 } /* Generic.Traceback */
+.highlight-ipython3 .kc { color: #66d9ef } /* Keyword.Constant */
+.highlight-ipython3 .kd { color: #66d9ef } /* Keyword.Declaration */
+.highlight-ipython3 .kn { color: #ff4689 } /* Keyword.Namespace */
+.highlight-ipython3 .kp { color: #66d9ef } /* Keyword.Pseudo */
+.highlight-ipython3 .kr { color: #66d9ef } /* Keyword.Reserved */
+.highlight-ipython3 .kt { color: #66d9ef } /* Keyword.Type */
+.highlight-ipython3 .ld { color: #e6db74 } /* Literal.Date */
+.highlight-ipython3 .m { color: #ae81ff } /* Literal.Number */
+.highlight-ipython3 .s { color: #e6db74 } /* Literal.String */
+.highlight-ipython3 .na { color: #a6e22e } /* Name.Attribute */
+.highlight-ipython3 .nb { color: #f8f8f2 } /* Name.Builtin */
+.highlight-ipython3 .nc { color: #a6e22e } /* Name.Class */
+.highlight-ipython3 .no { color: #66d9ef } /* Name.Constant */
+.highlight-ipython3 .nd { color: #a6e22e } /* Name.Decorator */
+.highlight-ipython3 .ni { color: #f8f8f2 } /* Name.Entity */
+.highlight-ipython3 .ne { color: #a6e22e } /* Name.Exception */
+.highlight-ipython3 .nf { color: #a6e22e } /* Name.Function */
+.highlight-ipython3 .nl { color: #f8f8f2 } /* Name.Label */
+.highlight-ipython3 .nn { color: #f8f8f2 } /* Name.Namespace */
+.highlight-ipython3 .nx { color: #a6e22e } /* Name.Other */
+.highlight-ipython3 .py { color: #f8f8f2 } /* Name.Property */
+.highlight-ipython3 .nt { color: #ff4689 } /* Name.Tag */
+.highlight-ipython3 .nv { color: #f8f8f2 } /* Name.Variable */
+.highlight-ipython3 .ow { color: #ff4689 } /* Operator.Word */
+.highlight-ipython3 .pm { color: #f8f8f2 } /* Punctuation.Marker */
+.highlight-ipython3 .w { color: #f8f8f2 } /* Text.Whitespace */
+.highlight-ipython3 .mb { color: #ae81ff } /* Literal.Number.Bin */
+.highlight-ipython3 .mf { color: #ae81ff } /* Literal.Number.Float */
+.highlight-ipython3 .mh { color: #ae81ff } /* Literal.Number.Hex */
+.highlight-ipython3 .mi { color: #ae81ff } /* Literal.Number.Integer */
+.highlight-ipython3 .mo { color: #ae81ff } /* Literal.Number.Oct */
+.highlight-ipython3 .sa { color: #e6db74 } /* Literal.String.Affix */
+.highlight-ipython3 .sb { color: #e6db74 } /* Literal.String.Backtick */
+.highlight-ipython3 .sc { color: #e6db74 } /* Literal.String.Char */
+.highlight-ipython3 .dl { color: #e6db74 } /* Literal.String.Delimiter */
+.highlight-ipython3 .sd { color: #e6db74 } /* Literal.String.Doc */
+.highlight-ipython3 .s2 { color: #e6db74 } /* Literal.String.Double */
+.highlight-ipython3 .se { color: #ae81ff } /* Literal.String.Escape */
+.highlight-ipython3 .sh { color: #e6db74 } /* Literal.String.Heredoc */
+.highlight-ipython3 .si { color: #e6db74 } /* Literal.String.Interpol */
+.highlight-ipython3 .sx { color: #e6db74 } /* Literal.String.Other */
+.highlight-ipython3 .sr { color: #e6db74 } /* Literal.String.Regex */
+.highlight-ipython3 .s1 { color: #e6db74 } /* Literal.String.Single */
+.highlight-ipython3 .ss { color: #e6db74 } /* Literal.String.Symbol */
+.highlight-ipython3 .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */
+.highlight-ipython3 .fm { color: #a6e22e } /* Name.Function.Magic */
+.highlight-ipython3 .vc { color: #f8f8f2 } /* Name.Variable.Class */
+.highlight-ipython3 .vg { color: #f8f8f2 } /* Name.Variable.Global */
+.highlight-ipython3 .vi { color: #f8f8f2 } /* Name.Variable.Instance */
+.highlight-ipython3 .vm { color: #f8f8f2 } /* Name.Variable.Magic */
+.highlight-ipython3 .il { color: #ae81ff } /* Literal.Number.Integer.Long */

docs/_static/css/highlight_python_dark.css

@@ -0,0 +1,85 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-python .hll { background-color: #49483e }
+.highlight-python { background: #232629; color: #cccccc }
+.highlight-python .c { color: #777777; font-style: italic } /* Comment */
+.highlight-python .err { color: #a61717; background-color: #e3d2d2 } /* Error */
+.highlight-python .esc { color: #cccccc } /* Escape */
+.highlight-python .g { color: #cccccc } /* Generic */
+.highlight-python .k { color: #7686bb; font-weight: bold } /* Keyword */
+.highlight-python .l { color: #cccccc } /* Literal */
+.highlight-python .n { color: #cccccc } /* Name */
+.highlight-python .o { color: #cccccc } /* Operator */
+.highlight-python .x { color: #cccccc } /* Other */
+.highlight-python .p { color: #cccccc } /* Punctuation */
+.highlight-python .ch { color: #777777; font-style: italic } /* Comment.Hashbang */
+.highlight-python .cm { color: #777777; font-style: italic } /* Comment.Multiline */
+.highlight-python .cp { color: #777777; font-style: italic } /* Comment.Preproc */
+.highlight-python .cpf { color: #777777; font-style: italic } /* Comment.PreprocFile */
+.highlight-python .c1 { color: #777777; font-style: italic } /* Comment.Single */
+.highlight-python .cs { color: #777777; font-style: italic } /* Comment.Special */
+.highlight-python .gd { color: #cccccc } /* Generic.Deleted */
+.highlight-python .ge { color: #cccccc } /* Generic.Emph */
+.highlight-python .ges { color: #cccccc } /* Generic.EmphStrong */
+.highlight-python .gr { color: #cccccc } /* Generic.Error */
+.highlight-python .gh { color: #cccccc } /* Generic.Heading */
+.highlight-python .gi { color: #cccccc } /* Generic.Inserted */
+.highlight-python .go { color: #cccccc } /* Generic.Output */
+.highlight-python .gp { color: #ffffff } /* Generic.Prompt */
+.highlight-python .gs { color: #cccccc } /* Generic.Strong */
+.highlight-python .gu { color: #cccccc } /* Generic.Subheading */
+.highlight-python .gt { color: #cccccc } /* Generic.Traceback */
+.highlight-python .kc { color: #7686bb; font-weight: bold } /* Keyword.Constant */
+.highlight-python .kd { color: #7686bb; font-weight: bold } /* Keyword.Declaration */
+.highlight-python .kn { color: #7686bb; font-weight: bold } /* Keyword.Namespace */
+.highlight-python .kp { color: #7686bb; font-weight: bold } /* Keyword.Pseudo */
+.highlight-python .kr { color: #7686bb; font-weight: bold } /* Keyword.Reserved */
+.highlight-python .kt { color: #7686bb; font-weight: bold } /* Keyword.Type */
+.highlight-python .ld { color: #cccccc } /* Literal.Date */
+.highlight-python .m { color: #4FB8CC } /* Literal.Number */
+.highlight-python .s { color: #51cc99 } /* Literal.String */
+.highlight-python .na { color: #cccccc } /* Name.Attribute */
+.highlight-python .nb { color: #cccccc } /* Name.Builtin */
+.highlight-python .nc { color: #cccccc } /* Name.Class */
+.highlight-python .no { color: #cccccc } /* Name.Constant */
+.highlight-python .nd { color: #cccccc } /* Name.Decorator */
+.highlight-python .ni { color: #cccccc } /* Name.Entity */
+.highlight-python .ne { color: #cccccc } /* Name.Exception */
+.highlight-python .nf { color: #6a6aff } /* Name.Function */
+.highlight-python .nl { color: #cccccc } /* Name.Label */
+.highlight-python .nn { color: #cccccc } /* Name.Namespace */
+.highlight-python .nx { color: #e2828e } /* Name.Other */
+.highlight-python .py { color: #cccccc } /* Name.Property */
+.highlight-python .nt { color: #cccccc } /* Name.Tag */
+.highlight-python .nv { color: #7AB4DB; font-weight: bold } /* Name.Variable */
+.highlight-python .ow { color: #cccccc } /* Operator.Word */
+.highlight-python .pm { color: #cccccc } /* Punctuation.Marker */
+.highlight-python .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight-python .mb { color: #4FB8CC } /* Literal.Number.Bin */
+.highlight-python .mf { color: #4FB8CC } /* Literal.Number.Float */
+.highlight-python .mh { color: #4FB8CC } /* Literal.Number.Hex */
+.highlight-python .mi { color: #4FB8CC } /* Literal.Number.Integer */
+.highlight-python .mo { color: #4FB8CC } /* Literal.Number.Oct */
+.highlight-python .sa { color: #51cc99 } /* Literal.String.Affix */
+.highlight-python .sb { color: #51cc99 } /* Literal.String.Backtick */
+.highlight-python .sc { color: #51cc99 } /* Literal.String.Char */
+.highlight-python .dl { color: #51cc99 } /* Literal.String.Delimiter */
+.highlight-python .sd { color: #51cc99 } /* Literal.String.Doc */
+.highlight-python .s2 { color: #51cc99 } /* Literal.String.Double */
+.highlight-python .se { color: #51cc99 } /* Literal.String.Escape */
+.highlight-python .sh { color: #51cc99 } /* Literal.String.Heredoc */
+.highlight-python .si { color: #51cc99 } /* Literal.String.Interpol */
+.highlight-python .sx { color: #51cc99 } /* Literal.String.Other */
+.highlight-python .sr { color: #51cc99 } /* Literal.String.Regex */
+.highlight-python .s1 { color: #51cc99 } /* Literal.String.Single */
+.highlight-python .ss { color: #51cc99 } /* Literal.String.Symbol */
+.highlight-python .bp { color: #cccccc } /* Name.Builtin.Pseudo */
+.highlight-python .fm { color: #6a6aff } /* Name.Function.Magic */
+.highlight-python .vc { color: #7AB4DB; font-weight: bold } /* Name.Variable.Class */
+.highlight-python .vg { color: #BE646C; font-weight: bold } /* Name.Variable.Global */
+.highlight-python .vi { color: #7AB4DB; font-weight: bold } /* Name.Variable.Instance */
+.highlight-python .vm { color: #7AB4DB; font-weight: bold } /* Name.Variable.Magic */
+.highlight-python .il { color: #4FB8CC } /* Literal.Number.Integer.Long */

docs/_static/css/highlight_python_light.css

@@ -0,0 +1,75 @@
+pre { line-height: 125%; }
+td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
+td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
+.highlight-python .hll { background-color: #ffffcc }
+.highlight-python { background: #f8f8f8; }
+.highlight-python .c { color: #008800; font-style: italic } /* Comment */
+.highlight-python .err { border: 1px solid #FF0000 } /* Error */
+.highlight-python .k { color: #AA22FF; font-weight: bold } /* Keyword */
+.highlight-python .o { color: #666666 } /* Operator */
+.highlight-python .ch { color: #008800; font-style: italic } /* Comment.Hashbang */
+.highlight-python .cm { color: #008800; font-style: italic } /* Comment.Multiline */
+.highlight-python .cp { color: #008800 } /* Comment.Preproc */
+.highlight-python .cpf { color: #008800; font-style: italic } /* Comment.PreprocFile */
+.highlight-python .c1 { color: #008800; font-style: italic } /* Comment.Single */
+.highlight-python .cs { color: #008800; font-weight: bold } /* Comment.Special */
+.highlight-python .gd { color: #A00000 } /* Generic.Deleted */
+.highlight-python .ge { font-style: italic } /* Generic.Emph */
+.highlight-python .ges { font-weight: bold; font-style: italic } /* Generic.EmphStrong */
+.highlight-python .gr { color: #FF0000 } /* Generic.Error */
+.highlight-python .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight-python .gi { color: #00A000 } /* Generic.Inserted */
+.highlight-python .go { color: #888888 } /* Generic.Output */
+.highlight-python .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
+.highlight-python .gs { font-weight: bold } /* Generic.Strong */
+.highlight-python .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight-python .gt { color: #0044DD } /* Generic.Traceback */
+.highlight-python .kc { color: #AA22FF; font-weight: bold } /* Keyword.Constant */
+.highlight-python .kd { color: #AA22FF; font-weight: bold } /* Keyword.Declaration */
+.highlight-python .kn { color: #AA22FF; font-weight: bold } /* Keyword.Namespace */
+.highlight-python .kp { color: #AA22FF } /* Keyword.Pseudo */
+.highlight-python .kr { color: #AA22FF; font-weight: bold } /* Keyword.Reserved */
+.highlight-python .kt { color: #00BB00; font-weight: bold } /* Keyword.Type */
+.highlight-python .m { color: #666666 } /* Literal.Number */
+.highlight-python .s { color: #BB4444 } /* Literal.String */
+.highlight-python .na { color: #BB4444 } /* Name.Attribute */
+.highlight-python .nb { color: #AA22FF } /* Name.Builtin */
+.highlight-python .nc { color: #0000FF } /* Name.Class */
+.highlight-python .no { color: #880000 } /* Name.Constant */
+.highlight-python .nd { color: #AA22FF } /* Name.Decorator */
+.highlight-python .ni { color: #999999; font-weight: bold } /* Name.Entity */
+.highlight-python .ne { color: #D2413A; font-weight: bold } /* Name.Exception */
+.highlight-python .nf { color: #00A000 } /* Name.Function */
+.highlight-python .nl { color: #A0A000 } /* Name.Label */
+.highlight-python .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
+.highlight-python .nt { color: #008000; font-weight: bold } /* Name.Tag */
+.highlight-python .nv { color: #B8860B } /* Name.Variable */
+.highlight-python .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
+.highlight-python .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight-python .mb { color: #666666 } /* Literal.Number.Bin */
+.highlight-python .mf { color: #666666 } /* Literal.Number.Float */
+.highlight-python .mh { color: #666666 } /* Literal.Number.Hex */
+.highlight-python .mi { color: #666666 } /* Literal.Number.Integer */
+.highlight-python .mo { color: #666666 } /* Literal.Number.Oct */
+.highlight-python .sa { color: #BB4444 } /* Literal.String.Affix */
+.highlight-python .sb { color: #BB4444 } /* Literal.String.Backtick */
+.highlight-python .sc { color: #BB4444 } /* Literal.String.Char */
+.highlight-python .dl { color: #BB4444 } /* Literal.String.Delimiter */
+.highlight-python .sd { color: #BB4444; font-style: italic } /* Literal.String.Doc */
+.highlight-python .s2 { color: #BB4444 } /* Literal.String.Double */
+.highlight-python .se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
+.highlight-python .sh { color: #BB4444 } /* Literal.String.Heredoc */
+.highlight-python .si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
+.highlight-python .sx { color: #008000 } /* Literal.String.Other */
+.highlight-python .sr { color: #BB6688 } /* Literal.String.Regex */
+.highlight-python .s1 { color: #BB4444 } /* Literal.String.Single */
+.highlight-python .ss { color: #B8860B } /* Literal.String.Symbol */
+.highlight-python .bp { color: #AA22FF } /* Name.Builtin.Pseudo */
+.highlight-python .fm { color: #00A000 } /* Name.Function.Magic */
+.highlight-python .vc { color: #B8860B } /* Name.Variable.Class */
+.highlight-python .vg { color: #B8860B } /* Name.Variable.Global */
+.highlight-python .vi { color: #B8860B } /* Name.Variable.Instance */
+.highlight-python .vm { color: #B8860B } /* Name.Variable.Magic */
+.highlight-python .il { color: #666666 } /* Literal.Number.Integer.Long */

docs/_static/css/ragas.css

@@ -0,0 +1,73 @@
+/* Base Theme */
+div.cell_input {
+    border: none !important;
+}
+
+.cell_output {
+  padding-left: 0px !important;
+  border: 1px solid #8b8b8b;
+  border-radius: var(--mystnb-source-border-radius);
+}
+
+.cell_output .output.text_plain,
+.cell_output .output.stream {
+  background: var(--background);
+  border: none;
+}
+
+.cell_output .output.stream {
+  margin-top: 0px;
+}
+
+.cell_output .output.stderr {
+  background: var(--background);
+  border: none;
+  margin-top: 0px;
+}
+
+.cell_output .output.traceback {
+  background: var(--background);
+  border: none;
+}
+
+/* Pandas tables. Pulled from the Jupyter / nbsphinx CSS */
+div.cell_output table {
+  display: block;
+  overflow-x: auto;
+  white-space: nowrap;
+  border: none;
+  border-collapse: collapse;
+  border-spacing: 0;
+  color: var(--foreground);
+  font-size: 1em;
+  table-layout: fixed;
+}
+
+div.cell_output thead {
+  border-bottom: 1px solid black;
+  vertical-align: bottom;
+}
+
+div.cell_output tr,
+div.cell_output th,
+div.cell_output td {
+  text-align: right;
+  vertical-align: middle;
+  padding: 0.5em 0.5em;
+  line-height: normal;
+  white-space: normal;
+  max-width: none;
+  border: none;
+}
+
+div.cell_output th {
+  font-weight: bold;
+}
+
+div.cell_output tbody tr:nth-child(odd) {
+  background: var(--backgound);
+}
+
+div.cell_output tbody tr:hover {
+  background: rgba(66, 165, 245, 0.2);
+}

docs/_static/edited_chain_runs.json

@@ -0,0 +1,490 @@
+{
+    "answer_correctness": [
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Sensory Adaptation and how does it affect human perception?",
+                "response": "The Theory of Sensory Adaptation is like when your nose stops noticing a strong smell after a while. It means our senses get used to things that stay the same, so we can pay more attention to new things around us. This helps us save energy and notice important changes, like if something new happens!",
+                "reference": "The Theory of Sensory Adaptation refers to the process by which our sensory receptors become less sensitive to constant stimuli over time. This phenomenon allows individuals to focus on changes in their environment rather than constant, unchanging stimuli. For example, when you enter a room with a strong odor, you may initially notice the smell intensely, but after a while, you become less aware of it. This adaptation is crucial for survival as it helps conserve energy and allows the brain to detect new and potentially important changes in the environment, enhancing our ability to respond to new stimuli."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Sensory Adaptation and how does it affect human perception?",
+                        "response": "The Theory of Sensory Adaptation is like when your nose stops noticing a strong smell after a while. It means our senses get used to things that stay the same, so we can pay more attention to new things around us. This helps us save energy and notice important changes, like if something new happens!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Sensory Adaptation refers to the process by which our sensory receptors become less sensitive to constant stimuli over time. This phenomenon allows individuals to focus on changes in their environment rather than constant, unchanging stimuli. For example, when you enter a room with a strong odor, you may initially notice the smell intensely, but after a while, you become less aware of it. This adaptation is crucial for survival as it helps conserve energy and allows the brain to detect new and potentially important changes in the environment, enhancing our ability to respond to new stimuli."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately explains the Theory of Sensory Adaptation and its effects on human perception, aligning well with the reference provided.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Evolutionary Fitness and how does it explain the survival of species?",
+                "response": "The Theory of Evolutionary Fitness is like a game where animals and plants try to be the best at living in their homes. If they have special tricks or traits that help them survive and have babies, they win by passing on their tricks to their kids. It's not just about being strong or fast; it's also about being smart, having babies, and finding food. Over time, nature picks the best tricks, and that's how animals and plants change and get better at living.",
+                "reference": "The Theory of Evolutionary Fitness is a concept within evolutionary biology that explains how certain traits or behaviors increase an organism's chances of survival and reproduction in a given environment. It is based on the idea that individuals with higher fitness are more likely to pass on their genes to the next generation. Fitness is not just about physical strength or speed; it encompasses a range of factors including adaptability, reproductive success, and the ability to find resources. Over time, natural selection favors traits that enhance fitness, leading to the evolution of species."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Evolutionary Fitness and how does it explain the survival of species?",
+                        "response": "The Theory of Evolutionary Fitness is like a game where animals and plants try to be the best at living in their homes. If they have special tricks or traits that help them survive and have babies, they win by passing on their tricks to their kids. It's not just about being strong or fast; it's also about being smart, having babies, and finding food. Over time, nature picks the best tricks, and that's how animals and plants change and get better at living.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Evolutionary Fitness is a concept within evolutionary biology that explains how certain traits or behaviors increase an organism's chances of survival and reproduction in a given environment. It is based on the idea that individuals with higher fitness are more likely to pass on their genes to the next generation. Fitness is not just about physical strength or speed; it encompasses a range of factors including adaptability, reproductive success, and the ability to find resources. Over time, natural selection favors traits that enhance fitness, leading to the evolution of species."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately explains the Theory of Evolutionary Fitness, covering key aspects such as survival, reproduction, and the role of natural selection, which aligns well with the reference.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Sediment Transport and how does it apply to river engineering?",
+                "response": "The Theory of Sediment Transport is like a big idea that helps us understand how tiny bits like sand and gravel move around in air. This is super important for people who work with rivers because it helps them figure out how rivers change shape over time. By knowing how fast the water is and how big the bits are, they can make sure rivers don't wash away too much land and stay easy to travel on.",
+                "reference": "The Theory of Sediment Transport is a fundamental concept in geomorphology and hydraulic engineering that describes the movement of sediment particles, such as sand, gravel, and silt, by fluid flow, typically water. This theory is crucial in river engineering as it helps in understanding and predicting how rivers shape their channels and floodplains over time. It involves the study of sediment erosion, transport, and deposition processes, which are influenced by factors like water velocity, sediment size, and riverbed composition. Engineers use this theory to design sustainable river management practices, prevent erosion, and maintain navigable waterways."
+            },
+            "metric_output": 0,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Sediment Transport and how does it apply to river engineering?",
+                        "response": "The Theory of Sediment Transport is like a big idea that helps us understand how tiny bits like sand and gravel move around in air. This is super important for people who work with rivers because it helps them figure out how rivers change shape over time. By knowing how fast the water is and how big the bits are, they can make sure rivers don't wash away too much land and stay easy to travel on.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Sediment Transport is a fundamental concept in geomorphology and hydraulic engineering that describes the movement of sediment particles, such as sand, gravel, and silt, by fluid flow, typically water. This theory is crucial in river engineering as it helps in understanding and predicting how rivers shape their channels and floodplains over time. It involves the study of sediment erosion, transport, and deposition processes, which are influenced by factors like water velocity, sediment size, and riverbed composition. Engineers use this theory to design sustainable river management practices, prevent erosion, and maintain navigable waterways."
+                    },
+                    "prompt_output": {
+                        "reason": "The response does not accurately capture the technical aspects of the Theory of Sediment Transport as described in the reference. It oversimplifies the concept and lacks details about sediment erosion, transport, deposition processes, and their implications for river engineering.",
+                        "verdict": 0
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "Instead of movement through water, the response mentions movement through the air which is wrong.",
+                        "verdict": 0
+                    }
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Isostasy and how does it explain the Earth's crust equilibrium?",
+                "response": "The Theory of Isostasy is like saying the Earth's crust is floating on a squishy layer underneath, just like how an iceberg floats on water. This helps explain why some parts of the Earth, like mountains, are higher, and others, like ocean floors, are lower. If a part of the crust is heavy, it sinks a bit, and if it's light, it floats up more. This balancing act happens very quickly over a short time and helps make things like mountains and changes after big ice sheets melt.",
+                "reference": "The Theory of Isostasy is a concept in geology that explains the equilibrium of the Earth's crust. It suggests that the Earth's lithosphere floats on the more fluid asthenosphere beneath it, much like an iceberg floats on water. This theory accounts for the varying elevations of the Earth's surface, such as mountains and ocean basins. According to isostasy, areas of the crust that are thicker or denser will sink deeper into the mantle, while lighter or thinner areas will rise. This balance is achieved over geological time scales and is responsible for the vertical movements of the Earth's crust, contributing to phenomena such as post-glacial rebound and the formation of mountain ranges."
+            },
+            "metric_output": 0,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Isostasy and how does it explain the Earth's crust equilibrium?",
+                        "response": "The Theory of Isostasy is like saying the Earth's crust is floating on a squishy layer underneath, just like how an iceberg floats on water. This helps explain why some parts of the Earth, like mountains, are higher, and others, like ocean floors, are lower. If a part of the crust is heavy, it sinks a bit, and if it's light, it floats up more. This balancing act happens very quickly over a short time and helps make things like mountains and changes after big ice sheets melt.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Isostasy is a concept in geology that explains the equilibrium of the Earth's crust. It suggests that the Earth's lithosphere floats on the more fluid asthenosphere beneath it, much like an iceberg floats on water. This theory accounts for the varying elevations of the Earth's surface, such as mountains and ocean basins. According to isostasy, areas of the crust that are thicker or denser will sink deeper into the mantle, while lighter or thinner areas will rise. This balance is achieved over geological time scales and is responsible for the vertical movements of the Earth's crust, contributing to phenomena such as post-glacial rebound and the formation of mountain ranges."
+                    },
+                    "prompt_output": {
+                        "reason": "The response provides a simplified explanation of the Theory of Isostasy but lacks the depth and detail present in the reference. It incorrectly states that the balancing act happens very quickly over a short time, while the reference emphasizes that this balance is achieved over geological time scales.",
+                        "verdict": 0
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "It incorrectly states that the balancing act happens very quickly over a short time, while the reference emphasizes that this balance is achieved over geological time scales.",
+                        "verdict": 0
+                    }
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the key concepts in the Theory of Digital Computation?",
+                "response": "The Theory of Digital Computation is like a big toolbox for understanding computers. It has things like algorithms, which are like recipes for solving problems. It also looks at how hard these problems are to solve, called computational complexity. There are also ideas about pretend machines, called automata, and how they work. Formal languages are like special codes that computers understand. Turing machines are like imaginary computers that help us think about what computers can do. The Church-Turing thesis says that these pretend computers can do anything a real computer can do. All these ideas help us know what computers can and can't do.",
+                "reference": "The Theory of Digital Computation encompasses several key concepts that form the foundation of computer science. These include algorithms, which are step-by-step procedures for solving problems; computational complexity, which studies the resources required for algorithms to solve problems; automata theory, which examines abstract machines and the problems they can solve; and formal languages, which are sets of strings defined by specific grammatical rules. Additionally, the theory explores Turing machines, which are abstract representations of computation, and the Church-Turing thesis, which posits that any computation can be performed by a Turing machine. These concepts collectively help in understanding the limits and capabilities of digital computation."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the key concepts in the Theory of Digital Computation?",
+                        "response": "The Theory of Digital Computation is like a big toolbox for understanding computers. It has things like algorithms, which are like recipes for solving problems. It also looks at how hard these problems are to solve, called computational complexity. There are also ideas about pretend machines, called automata, and how they work. Formal languages are like special codes that computers understand. Turing machines are like imaginary computers that help us think about what computers can do. The Church-Turing thesis says that these pretend computers can do anything a real computer can do. All these ideas help us know what computers can and can't do.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Digital Computation encompasses several key concepts that form the foundation of computer science. These include algorithms, which are step-by-step procedures for solving problems; computational complexity, which studies the resources required for algorithms to solve problems; automata theory, which examines abstract machines and the problems they can solve; and formal languages, which are sets of strings defined by specific grammatical rules. Additionally, the theory explores Turing machines, which are abstract representations of computation, and the Church-Turing thesis, which posits that any computation can be performed by a Turing machine. These concepts collectively help in understanding the limits and capabilities of digital computation."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately covers all the key concepts mentioned in the reference, including algorithms, computational complexity, automata, formal languages, Turing machines, and the Church-Turing thesis.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Quantum Decoherence and how does it explain the transition from quantum to classical physics?",
+                "response": "The Theory of Quantum Decoherence is like when a magic trick stops being magical because too many people are watching. In the tiny world of quantum, things can be in many places at once, like magic. But when they mix with the world around them, they start acting normal, like everyday things we see. This helps us understand how the weird quantum world turns into the normal world we know, without needing someone to watch it happen.",
+                "reference": "The Theory of Quantum Decoherence explains how quantum systems lose their quantum behavior and appear classical when interacting with their environment. It describes the process by which a quantum system's coherent superposition of states becomes a statistical mixture due to entanglement with the environment. This interaction causes the system to lose its quantum coherence, making it behave more like a classical system. Decoherence provides a framework for understanding the apparent collapse of the wave function without invoking a measurement or observer, thus bridging the gap between quantum mechanics and classical physics."
+            },
+            "metric_output": 0,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Quantum Decoherence and how does it explain the transition from quantum to classical physics?",
+                        "response": "The Theory of Quantum Decoherence is like when a magic trick stops being magical because too many people are watching. In the tiny world of quantum, things can be in many places at once, like magic. But when they mix with the world around them, they start acting normal, like everyday things we see. This helps us understand how the weird quantum world turns into the normal world we know, without needing someone to watch it happen.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Quantum Decoherence explains how quantum systems lose their quantum behavior and appear classical when interacting with their environment. It describes the process by which a quantum system's coherent superposition of states becomes a statistical mixture due to entanglement with the environment. This interaction causes the system to lose its quantum coherence, making it behave more like a classical system. Decoherence provides a framework for understanding the apparent collapse of the wave function without invoking a measurement or observer, thus bridging the gap between quantum mechanics and classical physics."
+                    },
+                    "prompt_output": {
+                        "reason": "The response oversimplifies the concept of Quantum Decoherence and does not accurately capture the technical details provided in the reference. It uses an analogy that may mislead the reader about the actual scientific principles involved.",
+                        "verdict": 0
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "The response correctly explains the concept of Quantum Decoherence and does factually accurate with the details provided in the reference.",
+                        "verdict": 0
+                    }
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Hawking Radiation and how does it relate to black holes?",
+                "response": "Stephen Hawking, a smart scientist, said in 1974 that black holes aren't just big, dark things that suck everything in. He explained that tiny bits of energy, called radiation, can actually come out of them! This happens because little pairs of particles pop up near the edge of the black hole. One gets pulled in, and the other escapes, making it look like the black hole is shining a tiny bit. Over a long time, this can make the black hole shrink and maybe even disappear! This idea was super cool because it mixed two big science ideas: how tiny things work and how big things like stars and planets behave.",
+                "reference": "The Theory of Hawking Radiation, proposed by physicist Stephen Hawking in 1974, suggests that black holes are not completely black but emit radiation due to quantum effects near the event horizon. This radiation arises because of particle-antiparticle pairs that form near the event horizon. One of the particles falls into the black hole while the other escapes, making it appear as though the black hole is emitting radiation. Over time, this process can lead to the gradual loss of mass and energy from the black hole, potentially causing it to evaporate completely. This theory was groundbreaking as it combined principles of quantum mechanics with general relativity, challenging the notion that nothing can escape from a black hole."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Hawking Radiation and how does it relate to black holes?",
+                        "response": "Stephen Hawking, a smart scientist, said in 1974 that black holes aren't just big, dark things that suck everything in. He explained that tiny bits of energy, called radiation, can actually come out of them! This happens because little pairs of particles pop up near the edge of the black hole. One gets pulled in, and the other escapes, making it look like the black hole is shining a tiny bit. Over a long time, this can make the black hole shrink and maybe even disappear! This idea was super cool because it mixed two big science ideas: how tiny things work and how big things like stars and planets behave.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Hawking Radiation, proposed by physicist Stephen Hawking in 1974, suggests that black holes are not completely black but emit radiation due to quantum effects near the event horizon. This radiation arises because of particle-antiparticle pairs that form near the event horizon. One of the particles falls into the black hole while the other escapes, making it appear as though the black hole is emitting radiation. Over time, this process can lead to the gradual loss of mass and energy from the black hole, potentially causing it to evaporate completely. This theory was groundbreaking as it combined principles of quantum mechanics with general relativity, challenging the notion that nothing can escape from a black hole."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately explains the Theory of Hawking Radiation, including the key concepts of particle-antiparticle pairs, the event horizon, and the implications for black holes, aligning well with the reference.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Special Creation and how does it explain the origin of life?",
+                "response": "The Theory of Special Creation is like a story that says a magical being made everything, like animals and people, just the way they are now. It says they haven't changed much since they were made. This idea is different from science ideas like evolution, which say that living things have changed a lot over time. People who believe in this theory often read religious books, like the Bible, to understand how everything was made.",
+                "reference": "The Theory of Special Creation is a religious-based explanation for the origin of life, proposing that life and the universe were created by a supernatural being or deity. This theory suggests that all species were created in their current form and have not undergone significant changes since their creation. It contrasts with scientific theories such as evolution, which propose that life has evolved over time through natural processes. The Theory of Special Creation is often associated with literal interpretations of religious texts, such as the Bible's account of creation in the Book of Genesis."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Special Creation and how does it explain the origin of life?",
+                        "response": "The Theory of Special Creation is like a story that says a magical being made everything, like animals and people, just the way they are now. It says they haven't changed much since they were made. This idea is different from science ideas like evolution, which say that living things have changed a lot over time. People who believe in this theory often read religious books, like the Bible, to understand how everything was made.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Special Creation is a religious-based explanation for the origin of life, proposing that life and the universe were created by a supernatural being or deity. This theory suggests that all species were created in their current form and have not undergone significant changes since their creation. It contrasts with scientific theories such as evolution, which propose that life has evolved over time through natural processes. The Theory of Special Creation is often associated with literal interpretations of religious texts, such as the Bible's account of creation in the Book of Genesis."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately describes the Theory of Special Creation, its contrast with evolution, and its association with religious texts, aligning well with the reference provided.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Multidimensional Space and how does it relate to our understanding of the universe?",
+                "response": "The Theory of Multidimensional Space is like saying there are more directions to move in than just up, down, and sideways. Scientists think there might be extra hidden spaces we can't see. This idea is used in a science called string theory, which says everything is made of tiny strings that wiggle in these extra spaces. These hidden spaces might help us understand how things like gravity work and how the universe started, like with black holes and the Big Bang!",
+                "reference": "The Theory of Multidimensional Space suggests that beyond the three dimensions we experience daily, there may exist additional spatial dimensions. This concept is often explored in theoretical physics, particularly in string theory, which posits that the universe is composed of tiny, vibrating strings that exist in multiple dimensions. These extra dimensions could help explain fundamental forces and particles in the universe, potentially unifying gravity with other forces. The theory challenges our perception of reality and opens up possibilities for understanding complex phenomena like black holes and the Big Bang."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Multidimensional Space and how does it relate to our understanding of the universe?",
+                        "response": "The Theory of Multidimensional Space is like saying there are more directions to move in than just up, down, and sideways. Scientists think there might be extra hidden spaces we can't see. This idea is used in a science called string theory, which says everything is made of tiny strings that wiggle in these extra spaces. These hidden spaces might help us understand how things like gravity work and how the universe started, like with black holes and the Big Bang!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Multidimensional Space suggests that beyond the three dimensions we experience daily, there may exist additional spatial dimensions. This concept is often explored in theoretical physics, particularly in string theory, which posits that the universe is composed of tiny, vibrating strings that exist in multiple dimensions. These extra dimensions could help explain fundamental forces and particles in the universe, potentially unifying gravity with other forces. The theory challenges our perception of reality and opens up possibilities for understanding complex phenomena like black holes and the Big Bang."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the Theory of Multidimensional Space and its relation to string theory, black holes, and the Big Bang, aligning well with the reference provided.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Meteor Impacts and how does it explain mass extinctions on Earth?",
+                "response": "The Theory of Meteor Impacts is like saying a giant space rock crashed into Earth a long time ago and made big changes that caused many animals to disappear. One famous time this happened was about 66 million years ago, and it might have made the dinosaurs go bye-bye. Scientists found a special kind of dirt called iridium that comes from space rocks, which helps them know this happened. When the big rock hit, it made fires and blocked the sun, making it super hard for animals and plants to live.",
+                "reference": "The Theory of Meteor Impacts suggests that large meteorites or asteroids have collided with Earth in the past, causing significant environmental changes and mass extinctions. One of the most well-known examples is the impact event that occurred approximately 66 million years ago, which is believed to have led to the extinction of the dinosaurs. This theory is supported by evidence such as the presence of a layer of iridium-rich clay in the geological record, which is rare on Earth but common in asteroids. The impact would have caused massive fires, a 'nuclear winter' effect due to dust and aerosols blocking sunlight, and drastic climate changes, leading to the collapse of ecosystems and the extinction of many species."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Meteor Impacts and how does it explain mass extinctions on Earth?",
+                        "response": "The Theory of Meteor Impacts is like saying a giant space rock crashed into Earth a long time ago and made big changes that caused many animals to disappear. One famous time this happened was about 66 million years ago, and it might have made the dinosaurs go bye-bye. Scientists found a special kind of dirt called iridium that comes from space rocks, which helps them know this happened. When the big rock hit, it made fires and blocked the sun, making it super hard for animals and plants to live.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Meteor Impacts suggests that large meteorites or asteroids have collided with Earth in the past, causing significant environmental changes and mass extinctions. One of the most well-known examples is the impact event that occurred approximately 66 million years ago, which is believed to have led to the extinction of the dinosaurs. This theory is supported by evidence such as the presence of a layer of iridium-rich clay in the geological record, which is rare on Earth but common in asteroids. The impact would have caused massive fires, a 'nuclear winter' effect due to dust and aerosols blocking sunlight, and drastic climate changes, leading to the collapse of ecosystems and the extinction of many species."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the Theory of Meteor Impacts and its implications for mass extinctions, including the key points about the impact event 66 million years ago, the evidence of iridium, and the environmental consequences.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the key principles of Einstein's theory of Special Relativity?",
+                "response": "Einstein's Special Relativity is like two big ideas. First, no matter how fast you're going, the rules of how things work stay the same for everyone. Second, light always zooms at the same speed, no matter who's watching or how they're moving. Because of these ideas, weird things happen, like time going slower for fast-moving things and things looking squished when they move fast. This changed how we think about space and time, showing they're all mixed together in a big space-time thingy!",
+                "reference": "Einstein's theory of Special Relativity is based on two key principles. The first is the principle of relativity, which states that the laws of physics are the same for all observers in uniform motion relative to one another. This means that no matter how fast you are moving, the laws of physics apply equally. The second principle is the constancy of the speed of light, which asserts that the speed of light in a vacuum is the same for all observers, regardless of their motion or the motion of the light source. These principles lead to several surprising consequences, such as time dilation, where time appears to pass more slowly for objects moving at high speeds, and length contraction, where objects appear shorter in the direction of motion. Special Relativity fundamentally changed our understanding of space and time, showing that they are interconnected in a four-dimensional spacetime continuum."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the key principles of Einstein's theory of Special Relativity?",
+                        "response": "Einstein's Special Relativity is like two big ideas. First, no matter how fast you're going, the rules of how things work stay the same for everyone. Second, light always zooms at the same speed, no matter who's watching or how they're moving. Because of these ideas, weird things happen, like time going slower for fast-moving things and things looking squished when they move fast. This changed how we think about space and time, showing they're all mixed together in a big space-time thingy!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "Einstein's theory of Special Relativity is based on two key principles. The first is the principle of relativity, which states that the laws of physics are the same for all observers in uniform motion relative to one another. This means that no matter how fast you are moving, the laws of physics apply equally. The second principle is the constancy of the speed of light, which asserts that the speed of light in a vacuum is the same for all observers, regardless of their motion or the motion of the light source. These principles lead to several surprising consequences, such as time dilation, where time appears to pass more slowly for objects moving at high speeds, and length contraction, where objects appear shorter in the direction of motion. Special Relativity fundamentally changed our understanding of space and time, showing that they are interconnected in a four-dimensional spacetime continuum."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately captures the key principles of Einstein's theory of Special Relativity, including the principle of relativity and the constancy of the speed of light, as well as the consequences like time dilation and length contraction.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the key principles of quantum mechanics?",
+                "response": "Quantum mechanics is like a set of rules for tiny things like atoms. One rule is that tiny things can act like both waves and little balls. Another rule says you can't know everything about a tiny thing at the same time. There's also a rule that says tiny things can be in lots of places at once until you look at them. And finally, there's a rule that says tiny things can be connected so that if you change one, the other one changes too, even if they're far apart!",
+                "reference": "Quantum mechanics is a fundamental theory in physics that describes the physical properties of nature at the scale of atoms and subatomic particles. Key principles include wave-particle duality, which posits that particles can exhibit both wave-like and particle-like properties; the uncertainty principle, which states that certain pairs of physical properties cannot be simultaneously known to arbitrary precision; and superposition, which allows particles to exist in multiple states at once until measured. Additionally, quantum entanglement describes how particles can become interconnected in such a way that the state of one particle instantly influences the state of another, regardless of distance."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the key principles of quantum mechanics?",
+                        "response": "Quantum mechanics is like a set of rules for tiny things like atoms. One rule is that tiny things can act like both waves and little balls. Another rule says you can't know everything about a tiny thing at the same time. There's also a rule that says tiny things can be in lots of places at once until you look at them. And finally, there's a rule that says tiny things can be connected so that if you change one, the other one changes too, even if they're far apart!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "Quantum mechanics is a fundamental theory in physics that describes the physical properties of nature at the scale of atoms and subatomic particles. Key principles include wave-particle duality, which posits that particles can exhibit both wave-like and particle-like properties; the uncertainty principle, which states that certain pairs of physical properties cannot be simultaneously known to arbitrary precision; and superposition, which allows particles to exist in multiple states at once until measured. Additionally, quantum entanglement describes how particles can become interconnected in such a way that the state of one particle instantly influences the state of another, regardless of distance."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the key principles of quantum mechanics as outlined in the reference, including wave-particle duality, the uncertainty principle, superposition, and quantum entanglement.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Abiogenesis and how does it explain the origin of life on Earth?",
+                "response": "The Theory of Abiogenesis is like saying life started from stuff that wasn't alive a long time ago. It means that tiny building blocks of life, like amino acids, could just pop up naturally from simple things. Scientists did experiments, like the Miller-Urey experiment, to show this could happen. This idea helps us understand how life might have started on Earth without magic or anything like that.",
+                "reference": "The Theory of Abiogenesis posits that life on Earth began from non-living matter through natural processes. It suggests that simple organic compounds formed spontaneously under prebiotic conditions, eventually leading to more complex molecules and the first primitive life forms. This theory is supported by experiments like the Miller-Urey experiment, which demonstrated that amino acids, the building blocks of life, could be synthesized from simple chemicals under conditions thought to resemble those of early Earth. Abiogenesis provides a scientific framework for understanding how life could have originated without the need for supernatural intervention."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Abiogenesis and how does it explain the origin of life on Earth?",
+                        "response": "The Theory of Abiogenesis is like saying life started from stuff that wasn't alive a long time ago. It means that tiny building blocks of life, like amino acids, could just pop up naturally from simple things. Scientists did experiments, like the Miller-Urey experiment, to show this could happen. This idea helps us understand how life might have started on Earth without magic or anything like that.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Abiogenesis posits that life on Earth began from non-living matter through natural processes. It suggests that simple organic compounds formed spontaneously under prebiotic conditions, eventually leading to more complex molecules and the first primitive life forms. This theory is supported by experiments like the Miller-Urey experiment, which demonstrated that amino acids, the building blocks of life, could be synthesized from simple chemicals under conditions thought to resemble those of early Earth. Abiogenesis provides a scientific framework for understanding how life could have originated without the need for supernatural intervention."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the Theory of Abiogenesis and includes relevant details such as the formation of simple organic compounds and the Miller-Urey experiment, aligning well with the reference.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the significance of General Relativity in modern physics?",
+                "response": "General Relativity is a big idea from Albert Einstein in 1915 that changed how we think about gravity. Instead of just pulling things down, gravity is like a big invisible blanket that bends around heavy stuff like stars and planets. This helps us understand cool things like black holes, ripples in space called gravitational waves, and how the universe is getting smaller. Scientists have tested it a lot, like seeing how light bends around big things in space and how planets move just right. It also helps us learn about the Big Bang and how the universe is put together!",
+                "reference": "General Relativity, formulated by Albert Einstein in 1915, revolutionized our understanding of gravity by describing it as the curvature of spacetime caused by mass and energy. This theory has significant implications in modern physics, providing the framework for understanding phenomena such as black holes, gravitational waves, and the expansion of the universe. It has been confirmed by numerous experiments and observations, including the bending of light around massive objects and the precise orbit of planets. General Relativity also plays a crucial role in the field of cosmology, influencing our understanding of the Big Bang and the overall structure of the universe."
+            },
+            "metric_output": 0,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the significance of General Relativity in modern physics?",
+                        "response": "General Relativity is a big idea from Albert Einstein in 1915 that changed how we think about gravity. Instead of just pulling things down, gravity is like a big invisible blanket that bends around heavy stuff like stars and planets. This helps us understand cool things like black holes, ripples in space called gravitational waves, and how the universe is getting smaller. Scientists have tested it a lot, like seeing how light bends around big things in space and how planets move just right. It also helps us learn about the Big Bang and how the universe is put together!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "General Relativity, formulated by Albert Einstein in 1915, revolutionized our understanding of gravity by describing it as the curvature of spacetime caused by mass and energy. This theory has significant implications in modern physics, providing the framework for understanding phenomena such as black holes, gravitational waves, and the expansion of the universe. It has been confirmed by numerous experiments and observations, including the bending of light around massive objects and the precise orbit of planets. General Relativity also plays a crucial role in the field of cosmology, influencing our understanding of the Big Bang and the overall structure of the universe."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately captures the essence of General Relativity, its implications in modern physics, and mentions key concepts such as black holes, gravitational waves, and the Big Bang, aligning well with the reference.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "The response accurately captures the essence of General Relativity but it wrongly mentions that the universe is getting smaller instead of larger which is mentioned in the reference.",
+                        "verdict": 0
+                    }
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the key principles of the Theory of Group Dynamics?",
+                "response": "The Theory of Group Dynamics is like figuring out how people in a group play together. It looks at roles, which are like the jobs or parts each person has in the group, and norms, which are the rules everyone follows. It also talks about how much the group members like being together, called cohesion. Plus, it checks out how leaders lead, how decisions are made, and how problems are solved, all to see how well the group works together.",
+                "reference": "The Theory of Group Dynamics explores how individuals in a group interact and influence each other. Key principles include the concept of roles, which are the expected behaviors of individuals within the group, and norms, which are the shared expectations and rules that guide group behavior. Cohesion, the degree to which group members are attracted to each other and motivated to stay in the group, is also crucial. Additionally, group dynamics examines leadership styles, decision-making processes, and conflict resolution strategies, all of which impact the effectiveness and productivity of the group."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the key principles of the Theory of Group Dynamics?",
+                        "response": "The Theory of Group Dynamics is like figuring out how people in a group play together. It looks at roles, which are like the jobs or parts each person has in the group, and norms, which are the rules everyone follows. It also talks about how much the group members like being together, called cohesion. Plus, it checks out how leaders lead, how decisions are made, and how problems are solved, all to see how well the group works together.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Group Dynamics explores how individuals in a group interact and influence each other. Key principles include the concept of roles, which are the expected behaviors of individuals within the group, and norms, which are the shared expectations and rules that guide group behavior. Cohesion, the degree to which group members are attracted to each other and motivated to stay in the group, is also crucial. Additionally, group dynamics examines leadership styles, decision-making processes, and conflict resolution strategies, all of which impact the effectiveness and productivity of the group."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the key principles of the Theory of Group Dynamics as outlined in the reference, including roles, norms, cohesion, leadership, decision-making, and problem-solving.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the main postulates of the Kinetic Molecular Theory?",
+                "response": "The Kinetic Molecular Theory is like a set of rules to understand how gases act. It says: 1. Gas is made of lots of tiny bits that move around all the time. 2. These bits are so small that their size doesn't really matter compared to the space they are in. 3. The bits don't push or pull on each other unless they bump into each other, and when they do, they bounce off perfectly. 4. How fast these bits move depends on how hot the gas is. This helps us know how gases behave in different situations.",
+                "reference": "The Kinetic Molecular Theory explains the behavior of gases and is based on several key postulates: 1. Gases consist of a large number of tiny particles that are in constant, random motion. 2. The volume of the individual gas particles is negligible compared to the volume of the container. 3. Gas particles exert no forces on each other except during collisions, which are perfectly elastic. 4. The average kinetic energy of gas particles is directly proportional to the temperature of the gas in Kelvin. This theory helps in understanding gas laws and the behavior of gases under different conditions."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the main postulates of the Kinetic Molecular Theory?",
+                        "response": "The Kinetic Molecular Theory is like a set of rules to understand how gases act. It says: 1. Gas is made of lots of tiny bits that move around all the time. 2. These bits are so small that their size doesn't really matter compared to the space they are in. 3. The bits don't push or pull on each other unless they bump into each other, and when they do, they bounce off perfectly. 4. How fast these bits move depends on how hot the gas is. This helps us know how gases behave in different situations.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Kinetic Molecular Theory explains the behavior of gases and is based on several key postulates: 1. Gases consist of a large number of tiny particles that are in constant, random motion. 2. The volume of the individual gas particles is negligible compared to the volume of the container. 3. Gas particles exert no forces on each other except during collisions, which are perfectly elastic. 4. The average kinetic energy of gas particles is directly proportional to the temperature of the gas in Kelvin. This theory helps in understanding gas laws and the behavior of gases under different conditions."
+                    },
+                    "prompt_output": {
+                        "reason": "The response accurately summarizes the main postulates of the Kinetic Molecular Theory and aligns well with the reference provided.",
+                        "verdict": 1
+                    },
+                    "is_accepted": true,
+                    "edited_output": null
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What are the main principles of the Cell Theory?",
+                "response": "The Cell Theory is like a big rulebook for understanding living things. It says three important things: 1) Every living thing is made up of tiny building blocks called cells. 2) Cells are the smallest parts that can do all the jobs needed for life. 3) New cells are made when old cells split into two. This helps us know why cells are super important for all life and how life keeps going because cells keep making more cells.",
+                "reference": "The Cell Theory is a fundamental principle in biology that describes the properties of cells. It consists of three main principles: 1) All living organisms are composed of one or more cells, which are the basic units of life. 2) The cell is the smallest unit of life, and all vital functions of an organism occur within cells. 3) All cells arise from pre-existing cells through the process of cell division. This theory highlights the importance of cells in the structure and function of all living organisms and underscores the continuity of life through cellular reproduction."
+            },
+            "metric_output": 1,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What are the main principles of the Cell Theory?",
+                        "response": "The Cell Theory is like a big rulebook for understanding living things. It says three important things: 1) Every living thing is made up of tiny building blocks called cells. 2) Cells are the smallest parts that can do all the jobs needed for life. 3) New cells are made when old cells split into two. This helps us know why cells are super important for all life and how life keeps going because cells keep making more cells.",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Cell Theory is a fundamental principle in biology that describes the properties of cells. It consists of three main principles: 1) All living organisms are composed of one or more cells, which are the basic units of life. 2) The cell is the smallest unit of life, and all vital functions of an organism occur within cells. 3) All cells arise from pre-existing cells through the process of cell division. This theory highlights the importance of cells in the structure and function of all living organisms and underscores the continuity of life through cellular reproduction."
+                    },
+                    "prompt_output": {
+                        "reason": "The response does not accurately reflect the reference. While it captures the essence of the Cell Theory, it uses informal language and omits key details such as the emphasis on cells being the basic units of life and the process of cell division.",
+                        "verdict": 0
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "The response does provides a simplified and accurate representation of the reference",
+                        "verdict": 1
+                    }
+                }
+            },
+            "is_accepted": true
+        },
+        {
+            "metric_input": {
+                "user_input": "What is the Theory of Nitrogen Fixation and how does it impact agriculture?",
+                "response": "The Theory of Nitrogen Fixation is like a magic trick where tiny helpers called bacteria turn air into food for plants. These bacteria live in little bumps on the roots of plants like corn and wheat. This magic helps plants grow better without needing a lot of extra plant food from factories, making farming kinder to the earth and helping farmers grow more food!",
+                "reference": "The Theory of Nitrogen Fixation explains the process by which certain organisms convert atmospheric nitrogen into a form that plants can use for growth. This process is primarily carried out by symbiotic bacteria, such as Rhizobium, which form nodules on the roots of leguminous plants. Nitrogen fixation is crucial for agriculture as it enriches the soil with nitrogen, reducing the need for chemical fertilizers and promoting sustainable farming practices. This natural process helps maintain soil fertility and supports the growth of crops, leading to increased agricultural productivity."
+            },
+            "metric_output": 0,
+            "prompts": {
+                "single_turn_aspect_critic_prompt": {
+                    "prompt_input": {
+                        "user_input": "What is the Theory of Nitrogen Fixation and how does it impact agriculture?",
+                        "response": "The Theory of Nitrogen Fixation is like a magic trick where tiny helpers called bacteria turn air into food for plants. These bacteria live in little bumps on the roots of plants like corn and wheat. This magic helps plants grow better without needing a lot of extra plant food from factories, making farming kinder to the earth and helping farmers grow more food!",
+                        "retrieved_contexts": null,
+                        "reference_contexts": null,
+                        "reference": "The Theory of Nitrogen Fixation explains the process by which certain organisms convert atmospheric nitrogen into a form that plants can use for growth. This process is primarily carried out by symbiotic bacteria, such as Rhizobium, which form nodules on the roots of leguminous plants. Nitrogen fixation is crucial for agriculture as it enriches the soil with nitrogen, reducing the need for chemical fertilizers and promoting sustainable farming practices. This natural process helps maintain soil fertility and supports the growth of crops, leading to increased agricultural productivity."
+                    },
+                    "prompt_output": {
+                        "reason": "The response oversimplifies the concept of nitrogen fixation and does not accurately convey the scientific details provided in the reference. It lacks specific information about the organisms involved and the importance of nitrogen fixation in agriculture.",
+                        "verdict": 0
+                    },
+                    "is_accepted": true,
+                    "edited_output": {
+                        "reason": "The response mentions air instead of nitrogen which is wrong.",
+                        "verdict": 0
+                    }
+                }
+            },
+            "is_accepted": true
+        }
+    ]
+}

docs/_static/imgs/logo-black.svg

@@ -0,0 +1,3 @@
+<svg width="580" height="580" viewBox="0 0 580 580" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M571.479 562.158L510.23 500.909L482.502 528.637C481.259 529.838 479.594 530.502 477.866 530.487C476.137 530.472 474.484 529.779 473.262 528.557C472.04 527.335 471.347 525.682 471.332 523.954C471.317 522.225 471.981 520.561 473.182 519.317L500.91 491.59L457.502 448.182L429.774 475.91C428.531 477.11 426.866 477.775 425.138 477.76C423.41 477.745 421.757 477.051 420.535 475.829C419.313 474.607 418.62 472.954 418.605 471.226C418.59 469.498 419.254 467.833 420.455 466.59L448.183 438.862L404.775 395.455L377.047 423.182C376.439 423.812 375.712 424.314 374.908 424.66C374.104 425.005 373.239 425.187 372.364 425.194C371.488 425.202 370.621 425.035 369.811 424.704C369.001 424.372 368.265 423.883 367.646 423.264C367.027 422.645 366.538 421.909 366.206 421.099C365.875 420.289 365.708 419.422 365.716 418.546C365.723 417.671 365.905 416.806 366.25 416.002C366.596 415.198 367.098 414.471 367.727 413.863L395.455 386.135L352.048 342.727L324.32 370.455C323.077 371.656 321.412 372.32 319.684 372.305C317.956 372.29 316.302 371.597 315.08 370.375C313.858 369.153 313.165 367.5 313.15 365.772C313.135 364.044 313.8 362.379 315 361.136L342.728 333.408L299.32 290L271.592 317.728C270.349 318.929 268.685 319.593 266.956 319.578C265.228 319.563 263.575 318.87 262.353 317.648C261.131 316.426 260.438 314.773 260.423 313.044C260.408 311.316 261.072 309.651 262.273 308.408L290.001 280.68L246.593 237.273L218.865 265.001C217.622 266.201 215.957 266.866 214.229 266.851C212.501 266.836 210.848 266.142 209.626 264.92C208.404 263.698 207.711 262.045 207.696 260.317C207.681 258.589 208.345 256.924 209.546 255.681L237.274 227.953L193.866 184.545L166.138 212.273C164.895 213.474 163.23 214.138 161.502 214.123C159.774 214.108 158.121 213.415 156.899 212.193C155.677 210.971 154.984 209.318 154.968 207.59C154.953 205.862 155.618 204.197 156.818 202.954L184.546 175.226L141.139 131.818L113.411 159.546C112.168 160.747 110.503 161.411 108.775 161.396C107.046 161.381 105.393 160.688 104.171 159.466C102.949 158.244 102.256 156.591 102.241 154.863C102.226 153.135 102.891 151.47 104.091 150.227L131.819 122.499L88.4113 79.0909L60.6834 106.819C60.0754 107.448 59.3481 107.95 58.544 108.296C57.7399 108.641 56.875 108.823 55.9999 108.831C55.1248 108.838 54.2569 108.672 53.4469 108.34C52.6369 108.009 51.901 107.519 51.2821 106.901C50.6633 106.282 50.1739 105.546 49.8425 104.736C49.5111 103.926 49.3444 103.058 49.352 102.183C49.3596 101.308 49.5414 100.443 49.8868 99.6387C50.2322 98.8346 50.7343 98.1073 51.3638 97.4993L79.0918 69.7714L17.8425 8.52205C16.9207 7.60058 15.7465 6.97308 14.4681 6.71888C13.1898 6.46469 11.8648 6.59521 10.6607 7.09394C9.45651 7.59268 8.42725 8.43723 7.70301 9.52084C6.97878 10.6045 6.59208 11.8785 6.5918 13.1818V566.818C6.5918 568.566 7.28619 570.243 8.52223 571.479C9.75826 572.715 11.4347 573.409 13.1827 573.409H566.819C568.122 573.409 569.396 573.022 570.48 572.298C571.564 571.574 572.408 570.544 572.907 569.34C573.406 568.136 573.536 566.811 573.282 565.533C573.028 564.254 572.4 563.08 571.479 562.158ZM368.591 490.252C368.092 491.456 367.247 492.485 366.163 493.208C365.079 493.932 363.804 494.319 362.501 494.318H92.2736C90.5256 494.318 88.8492 493.624 87.6131 492.388C86.3771 491.152 85.6827 489.475 85.6827 487.727V217.5C85.683 216.197 86.0697 214.923 86.7939 213.839C87.5182 212.755 88.5474 211.911 89.7516 211.412C90.9557 210.913 92.2807 210.783 93.559 211.037C94.8374 211.291 96.0116 211.919 96.9334 212.84L367.161 483.067C368.083 483.989 368.711 485.164 368.966 486.443C369.22 487.721 369.09 489.047 368.591 490.252Z" fill="#1E1F21"/>
+</svg>

docs/_static/js/commonroom.js

@@ -0,0 +1,19 @@
+// CommonRoom Analytics
+(function() {
+  if (typeof window === 'undefined') return;
+  if (typeof window.signals !== 'undefined') return;
+  var script = document.createElement('script');
+  script.src = 'https://cdn.cr-relay.com/v1/site/af0e3230-e3f4-4e7d-8790-28b56c38d8a9/signals.js';
+  script.async = true;
+  window.signals = Object.assign(
+    [],
+    ['page', 'identify', 'form'].reduce(function (acc, method){
+      acc[method] = function () {
+        signals.push([method, arguments]);
+        return signals;
+      };
+     return acc;
+    }, {})
+  );
+  document.head.appendChild(script);
+})();

docs/_static/js/header_border.js

@@ -0,0 +1,39 @@
+const header_div = document.querySelector(".md-header");
+const navbar_div = document.querySelector(".md-tabs");
+const border_css = "2px solid #14151a";
+
+// Add smooth transition to borders
+if (header_div) {
+  header_div.style.transition = "border-bottom 0.3s ease";
+}
+if (navbar_div) {
+  navbar_div.style.transition = "border-bottom 0.3s ease";
+}
+
+if (header_div && navbar_div) {
+  // Function to check and apply borders based on navbar visibility
+  function applyBorders() {
+    const isNavbarHidden =
+      navbar_div.hasAttribute("hidden") ||
+      getComputedStyle(navbar_div).display === "none";
+    console.log("Navbar is hidden:", isNavbarHidden);
+    header_div.style.borderBottom = isNavbarHidden ? border_css : "none";
+    navbar_div.style.borderBottom = isNavbarHidden ? "none" : border_css;
+  }
+
+  // Initial check
+  applyBorders();
+
+  // Create a ResizeObserver to handle both resize and visibility changes
+  const resizeObserver = new ResizeObserver(applyBorders);
+  resizeObserver.observe(navbar_div);
+
+  // Handle scroll events with debouncing for better performance
+  let scrollTimeout;
+  window.addEventListener("scroll", () => {
+    if (scrollTimeout) {
+      window.cancelAnimationFrame(scrollTimeout);
+    }
+    scrollTimeout = window.requestAnimationFrame(applyBorders);
+  });
+}

docs/_static/js/mathjax.js

@@ -0,0 +1,19 @@
+window.MathJax = {
+    tex: {
+        inlineMath: [["\\(", "\\)"]],
+        displayMath: [["\\[", "\\]"]],
+        processEscapes: true,
+        processEnvironments: true
+    },
+    options: {
+        ignoreHtmlClass: ".*|",
+        processHtmlClass: "arithmatex"
+    }
+};
+
+document$.subscribe(() => {
+    MathJax.startup.output.clearCache()
+    MathJax.typesetClear()
+    MathJax.texReset()
+    MathJax.typesetPromise()
+})

docs/_static/js/mendable_chat_bubble.js

@@ -0,0 +1,40 @@
+document.addEventListener("DOMContentLoaded", () => {
+  function loadScript(src, callback) {
+    var script = document.createElement("script");
+    script.type = "text/javascript";
+    script.src = src;
+    script.onload = callback; // Once script is loaded, callback function will be called
+    document.head.appendChild(script);
+  }
+
+  // Load Mendable script and initialize the component once script is loaded
+  loadScript(
+    "https://unpkg.com/@mendable/search@0.0.191/dist/umd/mendable-bundle.min.js",
+    function () {
+      Mendable.initialize({
+        anon_key: "f4cb5493-f914-43a5-8edc-f41463ea5bed",
+        type: "searchBar",
+        elementId: "searchbox",
+        style: {
+          darkMode: true,
+          accentColor: "#FECA4B",
+          backgroundColor: "#0F1629"
+        },
+        searchBarStyle: {
+          backgroundColor: "#00000000"
+        },
+        showSimpleSearch: true,
+        messageSettings: {
+          openSourcesInNewTab: false,
+          prettySources: true
+        }
+        
+      });
+
+      var searchForm = document.getElementById('searchbox');
+      searchForm.onsubmit = (event) => {
+        event.preventDefault();
+      }
+    }
+  );
+});

docs/_static/js/toggle.js

@@ -0,0 +1,2 @@
+document.addEventListener('DOMContentLoaded', () => {
+});

docs/_static/sample_annotated_summary.json

@@ -0,0 +1,460 @@
+{
+  "summary_accuracy": [
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe Q2 earnings report revealed a significant 15% increase in revenue, which was largely driven by strong performance in the European market. Analysts were impressed by the company's ability to capitalize on opportunities in this region. The European market's growth played a crucial role in the overall financial success of the quarter. Investors are optimistic about future prospects given these positive results.",
+        "response": "The Q2 earnings report showed a 15% revenue increase, driven by strong European market performance, impressing analysts and boosting investor optimism."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe Q2 earnings report revealed a significant 15% increase in revenue, which was largely driven by strong performance in the European market. Analysts were impressed by the company's ability to capitalize on opportunities in this region. The European market's growth played a crucial role in the overall financial success of the quarter. Investors are optimistic about future prospects given these positive results.",
+            "response": "The Q2 earnings report showed a 15% revenue increase, driven by strong European market performance, impressing analysts and boosting investor optimism.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 15% revenue increase, the strong performance in the European market, the impression on analysts, and the optimism of investors.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2023, North American sales experienced a 5% decline compared to the previous year. This downturn was attributed to various market challenges and changing consumer preferences. Companies are now strategizing to recover and adapt to the evolving economic landscape. Despite the decline, there are opportunities for growth in emerging sectors.",
+        "response": "Companies are strategizing to adapt to market challenges and changing consumer preferences, with growth opportunities in emerging sectors despite recent downturns."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2023, North American sales experienced a 5% decline compared to the previous year. This downturn was attributed to various market challenges and changing consumer preferences. Companies are now strategizing to recover and adapt to the evolving economic landscape. Despite the decline, there are opportunities for growth in emerging sectors.",
+            "response": "Companies are strategizing to adapt to market challenges and changing consumer preferences, with growth opportunities in emerging sectors despite recent downturns.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the strategizing of companies to adapt to market challenges and changing consumer preferences, as well as the mention of growth opportunities in emerging sectors despite the downturn.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2022, European expansion was a significant focus for many companies, leading to a remarkable 15% growth in various sectors. This growth was driven by strategic investments and increased market penetration across the continent. Analysts noted that the European expansion efforts were crucial in achieving such impressive results. The 15% growth in 2022 set a new benchmark for future endeavors in the region.",
+        "response": "Many companies experienced a notable 15% growth due to strategic investments and increased market penetration, setting a new benchmark for future endeavors in the region."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2022, European expansion was a significant focus for many companies, leading to a remarkable 15% growth in various sectors. This growth was driven by strategic investments and increased market penetration across the continent. Analysts noted that the European expansion efforts were crucial in achieving such impressive results. The 15% growth in 2022 set a new benchmark for future endeavors in the region.",
+            "response": "Many companies experienced a notable 15% growth due to strategic investments and increased market penetration, setting a new benchmark for future endeavors in the region.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 15% growth, strategic investments, increased market penetration, and the setting of a new benchmark for future endeavors.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nSupply chain challenges have significantly impacted businesses across North America, leading to a reported 5% loss in revenue for many companies. These disruptions have been attributed to a variety of factors, including labor shortages and transportation delays. As a result, organizations are seeking innovative solutions to mitigate these losses and improve efficiency. The ongoing situation highlights the need for robust contingency planning in the face of global supply chain vulnerabilities.",
+        "response": "Supply chain challenges in North America, caused by labor shortages and transportation delays, have led to a 5% revenue loss for many companies, prompting the need for innovative solutions and robust contingency planning."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nSupply chain challenges have significantly impacted businesses across North America, leading to a reported 5% loss in revenue for many companies. These disruptions have been attributed to a variety of factors, including labor shortages and transportation delays. As a result, organizations are seeking innovative solutions to mitigate these losses and improve efficiency. The ongoing situation highlights the need for robust contingency planning in the face of global supply chain vulnerabilities.",
+            "response": "Supply chain challenges in North America, caused by labor shortages and transportation delays, have led to a 5% revenue loss for many companies, prompting the need for innovative solutions and robust contingency planning.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the impact of supply chain challenges on revenue, the causes of these challenges, and the need for innovative solutions and contingency planning.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn Q2 2023, the company reported a significant 15% rise in revenue compared to the previous quarter. This increase in revenue was attributed to strong sales in the new product line and improved market conditions. Analysts were optimistic about the company's future performance, citing the 15% rise as a positive indicator. The management expressed confidence in maintaining this growth trajectory in the upcoming quarters.",
+        "response": "The company experienced a notable increase in earnings due to strong sales in the new product line and favorable market conditions, with analysts and management expressing optimism about future growth."
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn Q2 2023, the company reported a significant 15% rise in revenue compared to the previous quarter. This increase in revenue was attributed to strong sales in the new product line and improved market conditions. Analysts were optimistic about the company's future performance, citing the 15% rise as a positive indicator. The management expressed confidence in maintaining this growth trajectory in the upcoming quarters.",
+            "response": "The company experienced a notable increase in earnings due to strong sales in the new product line and favorable market conditions, with analysts and management expressing optimism about future growth.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary inaccurately states an increase in earnings instead of revenue, and it omits the specific 15% rise and the comparison to the previous quarter.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2023, marketing campaigns in North America have seen a significant shift towards digital platforms. Companies are increasingly leveraging social media and online advertising to reach their target audiences. This trend is driven by the growing internet penetration and the changing consumer behavior in the region. As a result, businesses are investing more in innovative strategies to stay competitive in the North American market.",
+        "response": "In 2023, marketing campaigns in North America are increasingly leveraging digital platforms, driven by growing internet penetration and changing consumer behavior, prompting businesses to invest in innovative strategies to stay competitive."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2023, marketing campaigns in North America have seen a significant shift towards digital platforms. Companies are increasingly leveraging social media and online advertising to reach their target audiences. This trend is driven by the growing internet penetration and the changing consumer behavior in the region. As a result, businesses are investing more in innovative strategies to stay competitive in the North American market.",
+            "response": "In 2023, marketing campaigns in North America are increasingly leveraging digital platforms, driven by growing internet penetration and changing consumer behavior, prompting businesses to invest in innovative strategies to stay competitive.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the shift towards digital platforms, the role of internet penetration and changing consumer behavior, and the investment in innovative strategies by businesses.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe company's international expansion strategy has been a significant success, particularly in the European market. Over the past year, they have achieved a remarkable 15% growth in this region. This expansion has been driven by a combination of innovative products and strategic partnerships. As a result, the company is well-positioned to continue its upward trajectory in the coming years.",
+        "response": "The company's international expansion strategy has led to a 15% growth in the European market, driven by innovative products and strategic partnerships."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe company's international expansion strategy has been a significant success, particularly in the European market. Over the past year, they have achieved a remarkable 15% growth in this region. This expansion has been driven by a combination of innovative products and strategic partnerships. As a result, the company is well-positioned to continue its upward trajectory in the coming years.",
+            "response": "The company's international expansion strategy has led to a 15% growth in the European market, driven by innovative products and strategic partnerships.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 15% growth in the European market and the factors driving this growth, such as innovative products and strategic partnerships.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2024, companies are increasingly focusing on targeted marketing strategies to capture the North American market. By leveraging data analytics, businesses can tailor their campaigns to meet the specific needs and preferences of consumers in this region. This approach not only enhances customer engagement but also boosts sales and brand loyalty. As competition intensifies, targeted marketing becomes a crucial tool for success in the North American market.",
+        "response": "Companies are using data analytics to customize campaigns, improving customer engagement, sales, and brand loyalty as competition grows."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2024, companies are increasingly focusing on targeted marketing strategies to capture the North American market. By leveraging data analytics, businesses can tailor their campaigns to meet the specific needs and preferences of consumers in this region. This approach not only enhances customer engagement but also boosts sales and brand loyalty. As competition intensifies, targeted marketing becomes a crucial tool for success in the North American market.",
+            "response": "Companies are using data analytics to customize campaigns, improving customer engagement, sales, and brand loyalty as competition grows.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the main points of the original text, including the use of data analytics for targeted marketing, the focus on customer engagement, sales, and brand loyalty, and the context of increasing competition.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2023, logistics investments in North America have seen a significant surge, driven by advancements in technology and infrastructure. Companies are focusing on enhancing supply chain efficiency to meet the growing demand in the region. This trend is expected to continue as North America remains a key player in the global logistics market. Investors are keen on capitalizing on these opportunities to maximize returns.",
+        "response": "Driven by technological and infrastructural advancements, companies are enhancing supply chain efficiency to meet growing demand, with investors eager to capitalize on these opportunities for maximum returns."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2023, logistics investments in North America have seen a significant surge, driven by advancements in technology and infrastructure. Companies are focusing on enhancing supply chain efficiency to meet the growing demand in the region. This trend is expected to continue as North America remains a key player in the global logistics market. Investors are keen on capitalizing on these opportunities to maximize returns.",
+            "response": "Driven by technological and infrastructural advancements, companies are enhancing supply chain efficiency to meet growing demand, with investors eager to capitalize on these opportunities for maximum returns.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the focus on technological and infrastructural advancements, supply chain efficiency, growing demand, and investor interest.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2023, the company experienced a 5% decline in its market share, raising concerns among investors. Analysts attributed this drop to increased competition and changing consumer preferences. Despite the decline, the company is implementing new strategies to regain its market position. The management remains optimistic about reversing the trend by the end of the year.",
+        "response": "The company faced challenges due to competition and shifting consumer preferences, but is optimistic about reversing the trend with new strategies by year-end."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2023, the company experienced a 5% decline in its market share, raising concerns among investors. Analysts attributed this drop to increased competition and changing consumer preferences. Despite the decline, the company is implementing new strategies to regain its market position. The management remains optimistic about reversing the trend by the end of the year.",
+            "response": "The company faced challenges due to competition and shifting consumer preferences, but is optimistic about reversing the trend with new strategies by year-end.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the main points of the original text, including the decline in market share, the reasons for the decline, and the company's optimism and strategies to reverse the trend.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe company reported a significant profit due to an 8% rise in Q3 2024. This growth was largely driven by strong performance in the Asian market. Analysts attribute the success to strategic investments and increased consumer demand in the region. The positive results have bolstered investor confidence moving forward.",
+        "response": "The company's significant profit in Q3 2024, driven by an 8% rise and strong performance in the Asian market, is attributed to strategic investments and increased consumer demand, bolstering investor confidence."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe company reported a significant profit due to an 8% rise in Q3 2024. This growth was largely driven by strong performance in the Asian market. Analysts attribute the success to strategic investments and increased consumer demand in the region. The positive results have bolstered investor confidence moving forward.",
+            "response": "The company's significant profit in Q3 2024, driven by an 8% rise and strong performance in the Asian market, is attributed to strategic investments and increased consumer demand, bolstering investor confidence.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the significant profit, 8% rise, strong performance in the Asian market, strategic investments, increased consumer demand, and bolstered investor confidence.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe global market has experienced a significant shift, with sales showing a 15% decline in Q3 2024. This downturn has raised concerns among investors and stakeholders worldwide. Analysts are closely monitoring the situation to understand the underlying factors contributing to this decline. Companies are now strategizing to adapt to the changing dynamics of the global market.",
+        "response": "The recent downturn has raised concerns among investors and stakeholders, prompting companies to strategize and adapt to changing dynamics."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe global market has experienced a significant shift, with sales showing a 15% decline in Q3 2024. This downturn has raised concerns among investors and stakeholders worldwide. Analysts are closely monitoring the situation to understand the underlying factors contributing to this decline. Companies are now strategizing to adapt to the changing dynamics of the global market.",
+            "response": "The recent downturn has raised concerns among investors and stakeholders, prompting companies to strategize and adapt to changing dynamics.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the main points of the original text, including the downturn, concerns among investors and stakeholders, and companies strategizing to adapt to changing dynamics.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe logistics industry in North America is projected to experience a 20% growth in 2024. This surge is attributed to advancements in technology and increased demand for efficient supply chain solutions. Companies are investing heavily in infrastructure to capitalize on this growth. As a result, the region is poised to become a leader in global logistics innovation.",
+        "response": "The industry is expected to grow by 20% in 2024 due to technological advancements and a higher demand for efficient supply chain solutions, with companies investing in infrastructure to lead in global innovation."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe logistics industry in North America is projected to experience a 20% growth in 2024. This surge is attributed to advancements in technology and increased demand for efficient supply chain solutions. Companies are investing heavily in infrastructure to capitalize on this growth. As a result, the region is poised to become a leader in global logistics innovation.",
+            "response": "The industry is expected to grow by 20% in 2024 due to technological advancements and a higher demand for efficient supply chain solutions, with companies investing in infrastructure to lead in global innovation.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the projected 20% growth in 2024, the reasons for this growth (technological advancements and increased demand for efficient supply chain solutions), and the investment in infrastructure to lead in global innovation.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
+        "response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
+            "response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 8% rise in Q3 2024, the contribution of the Asian market, and the role of marketing strategies and product localization. It also mentions the expectation of continued growth.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nIn 2022, the Asian market experienced a significant 15% decline, impacting various sectors. This downturn led to increased scrutiny over expenses as companies sought to mitigate losses. Analysts noted that the decline was one of the most substantial in recent years for the region. Businesses are now strategizing to recover and stabilize their financial standing.",
+        "response": "In 2022, the Asian market experienced a significant 15% decline, leading to increased scrutiny over expenses as companies strategize to recover and stabilize their financial standing."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nIn 2022, the Asian market experienced a significant 15% decline, impacting various sectors. This downturn led to increased scrutiny over expenses as companies sought to mitigate losses. Analysts noted that the decline was one of the most substantial in recent years for the region. Businesses are now strategizing to recover and stabilize their financial standing.",
+            "response": "In 2022, the Asian market experienced a significant 15% decline, leading to increased scrutiny over expenses as companies strategize to recover and stabilize their financial standing.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 15% decline in the Asian market, the impact on expenses, and the efforts by companies to recover and stabilize their financial standing.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe global market has witnessed a significant shift with a 10% increase in Q1 2023. This surge has positively impacted the revenue streams of many multinational corporations. Analysts attribute this growth to strategic investments and expanding consumer bases. As a result, the global market outlook remains optimistic for the remainder of the year.",
+        "response": "The global market experienced a 10% increase in Q1 2023, boosting multinational corporations' revenue streams and leading to an optimistic outlook for the year, attributed to strategic investments and expanding consumer bases."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe global market has witnessed a significant shift with a 10% increase in Q1 2023. This surge has positively impacted the revenue streams of many multinational corporations. Analysts attribute this growth to strategic investments and expanding consumer bases. As a result, the global market outlook remains optimistic for the remainder of the year.",
+            "response": "The global market experienced a 10% increase in Q1 2023, boosting multinational corporations' revenue streams and leading to an optimistic outlook for the year, attributed to strategic investments and expanding consumer bases.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the 10% increase in Q1 2023, the positive impact on multinational corporations' revenue, the attribution to strategic investments and expanding consumer bases, and the optimistic outlook for the year.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe company reported a 10% increase in 2023, significantly boosting its Revenue. This growth was primarily driven by strong performance in the European market. Analysts attribute the success to strategic investments and expanding customer base in the region. As a result, the company is optimistic about maintaining this upward trend in the coming years.",
+        "response": "The company experienced significant growth due to strategic investments and an expanding customer base, leading to optimism about future trends."
+      },
+      "metric_output": 0,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe company reported a 10% increase in 2023, significantly boosting its Revenue. This growth was primarily driven by strong performance in the European market. Analysts attribute the success to strategic investments and expanding customer base in the region. As a result, the company is optimistic about maintaining this upward trend in the coming years.",
+            "response": "The company experienced significant growth due to strategic investments and an expanding customer base, leading to optimism about future trends.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the company's growth, strategic investments, expanding customer base, and optimism about future trends. However, it omits specific details such as the 10% increase in revenue and the strong performance in the European market, which are important aspects of the original text.",
+            "verdict": 0
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe company's revenue saw an 8% rise in 2023, largely driven by its expansion into the European market. This growth in the European market has been a significant contributor to the overall financial success. Analysts attribute the 8% rise in 2023 to strategic investments and increased consumer demand in Europe. The company plans to continue leveraging the European market to sustain its revenue growth.",
+        "response": "The company's financial success was significantly influenced by strategic investments and increased consumer demand, with plans to sustain growth through continued efforts in Europe."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe company's revenue saw an 8% rise in 2023, largely driven by its expansion into the European market. This growth in the European market has been a significant contributor to the overall financial success. Analysts attribute the 8% rise in 2023 to strategic investments and increased consumer demand in Europe. The company plans to continue leveraging the European market to sustain its revenue growth.",
+            "response": "The company's financial success was significantly influenced by strategic investments and increased consumer demand, with plans to sustain growth through continued efforts in Europe.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the key points of the original text, including the company's revenue growth, the influence of strategic investments and consumer demand, and the focus on the European market for sustaining growth.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": false
+    },
+    {
+      "metric_input": {
+        "user_input": "summarise given text\nThe Marketing team is strategizing to address the challenges posed by the projected 15% decline in 2024 within the Asian market. This downturn is prompting companies to innovate and adapt their approaches to maintain competitiveness. Analysts suggest that understanding local consumer behavior will be crucial in mitigating the impact of this decline. As a result, businesses are investing in targeted campaigns to better engage with the Asian market.",
+        "response": "The team is strategizing to address the challenges posed by the projected 15% decline in 2024 within the Asian market, prompting companies to innovate and adapt their approaches to maintain competitiveness."
+      },
+      "metric_output": 1,
+      "prompts": {
+        "single_turn_aspect_critic_prompt": {
+          "prompt_input": {
+            "user_input": "summarise given text\nThe Marketing team is strategizing to address the challenges posed by the projected 15% decline in 2024 within the Asian market. This downturn is prompting companies to innovate and adapt their approaches to maintain competitiveness. Analysts suggest that understanding local consumer behavior will be crucial in mitigating the impact of this decline. As a result, businesses are investing in targeted campaigns to better engage with the Asian market.",
+            "response": "The team is strategizing to address the challenges posed by the projected 15% decline in 2024 within the Asian market, prompting companies to innovate and adapt their approaches to maintain competitiveness.",
+            "retrieved_contexts": null,
+            "reference_contexts": null,
+            "reference": null
+          },
+          "prompt_output": {
+            "reason": "The summary accurately captures the main points of the original text, including the strategizing to address the decline, the need for innovation and adaptation, and the focus on maintaining competitiveness.",
+            "verdict": 1
+          },
+          "edited_output": null
+        }
+      },
+      "is_accepted": true
+    }
+  ]
+}

docs/alfred.py

@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import typing as t
+from collections import namedtuple
+
+from langchain.prompts import ChatPromptTemplate
+from langchain_core.language_models.chat_models import BaseChatModel
+from langchain_openai.chat_models import ChatOpenAI
+from tqdm.asyncio import tqdm
+
+File = namedtuple("File", "name content")
+
+
+def get_files(path: str, ext: str) -> list:
+    return [os.path.join(path, f) for f in os.listdir(path) if f.endswith(ext)]
+
+
+def load_docs(path: str) -> t.List[File]:
+    files = [*get_files(path, ".md")]
+    docs = []
+    for file in files:
+        with open(file, "r") as f:
+            print("fixing: ", file)
+            docs.append(File(file, f.read()))
+    return docs
+
+
+async def fix_doc_with_llm(doc: File, llm: BaseChatModel) -> File:
+    prompt = """\
+fix the following grammar and spelling mistakes in the following text. 
+Please keep the markdown format intact when reformating it. 
+Do not make any change to the parts of text that are for formating or additional metadata for the core text in markdown.
+The target audience for this is developers so keep the tone serious and to the point without any marketing terms. 
+The output text should me in .md format. 
+
+text: {text}
+"""
+    fix_docs_prompt = ChatPromptTemplate.from_messages(
+        [
+            (prompt),
+        ]
+    )
+    # get output
+    fixed_doc = await llm.ainvoke(fix_docs_prompt.format_messages(text=doc.content))
+    return File(doc.name, fixed_doc.content)
+
+
+async def main(docs: t.List[File], llm: BaseChatModel):
+    fix_doc_routines = [fix_doc_with_llm(doc, llm) for doc in docs]
+    return await tqdm.gather(*fix_doc_routines)
+
+
+if __name__ == "__main__":
+    """
+    Helpful assistant for documentation review and more (hopefully in the future).
+    """
+    # Create an argument parser
+    parser = argparse.ArgumentParser(
+        description="Helpful assistant for documentation review."
+    )
+    parser.add_argument("-d", "--directory", help="Directory to run the script against")
+    args = parser.parse_args()
+    directory = args.directory
+    docs = load_docs(directory)
+    gpt4 = ChatOpenAI(model="gpt-4")
+    fix_docs = asyncio.run(main(docs, gpt4))
+    for doc in fix_docs:
+        with open(doc.name, "w") as f:
+            f.write(doc.content)

docs/community/index.md

@@ -0,0 +1,59 @@
+# ❤️ Community
+
+> "Alone we can do so little; together we can do so much." - Helen Keller
+
+Our project thrives on the vibrant energy, diverse skills, and shared passion of our community. It's not just about code; it's about people coming together to create something extraordinary. This space celebrates every contribution, big or small, and features the amazing people who make it all happen.
+
+## **💬 Discord community**
+
+Join our Discord community [here](https://discord.com/invite/5djav8GGNZ) to connect with other developers, share your ideas, and get support.
+
+## **🌟  Contributors**
+Meet some of our outstanding contributors! 
+
+<a href="https://github.com/vibrantlabsai/ragas/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=vibrantlabsai/ragas" />
+</a>
+
+See the full list [here](https://github.com/vibrantlabsai/ragas/graphs/contributors).
+
+## **📚 Blog & Insights**
+
+Explore insightful articles, tutorials, and stories written by and for our community members.
+
+- [Luka Panić](https://www.linkedin.com/in/luka-pani%C4%87-20b671277/) shares his work on
+    - [Ragas Evaluation: In-Depth Insights | PIXION Blog](https://pixion.co/blog/ragas-evaluation-in-depth-insights): A detailed explanation of the metrics and how they are calculated.
+    - [RAG in practice - Test Set Generation | PIXION Blog](https://pixion.co/blog/rag-in-practice-test-set-generation): A tutorial on how to generate a test set using Ragas.
+- [Shanthi Vardhan](https://www.linkedin.com/in/shanthivardhan/) shares how his team at [Atomicwork uses ragas](https://www.atomicwork.com/blog/ragas-improving-atom-accuracy) to improve their AI system's ability to accurately identify and retrieve more precise information for enhanced service management.
+- [Pinecone's](https://pinecone.io/blog) study on how RAGs can enhance capabilities of LLMs in ["RAG makes LLMs better and equal"](https://www.pinecone.io/blog/rag-study/) uses ragas to proves context retrieval makes LLMs provide significantly better results, even when increasing the data size to 1 billion.
+- [Aishwarya Prabhat](https://www.linkedin.com/in/aishwaryaprabhat/) shares her expertise on advanced RAG techniques in her comprehensive guide, ["Performing, Evaluating & Tracking Advanced RAG (ft. AzureML, LlamaIndex & Ragas)"](https://www.linkedin.com/pulse/performing-evaluating-tracking-advanced-rag-ft-azureml-prabhat-i1rkc/).
+- Leonie (aka [@helloiamleonie](https://twitter.com/helloiamleonie?source=about_page-------------------------------------))  offers her perspective in the detailed article, ["Evaluating RAG Applications with RAGAs"](https://towardsdatascience.com/evaluating-rag-applications-with-ragas-81d67b0ee31a).
+- The joint efforts of [Erika Cardenas](https://twitter.com/ecardenas300) and [Connor Shorten](https://twitter.com/CShorten30) are showcased in their collaborative piece, ["An Overview on RAG Evaluation | Weaviate"](https://weaviate.io/blog/rag-evaluation), and their podcast with the Ragas team.
+- [Erika Cardenas](https://twitter.com/ecardenas300) further explores the "[RAG performance of hybrid search weightings (alpha)](https://www.linkedin.com/posts/erikacardenas300_i-tested-the-rag-performance-of-hybrid-search-activity-7139679925426376705-TVtc?utm_source=share&utm_medium=member_desktop)" in her recent experiment to tune weaviate alpha score using Ragas.
+- [LangChain’s](https://blog.langchain.dev/) work about [RAG Evaluating RAG pipelines with RAGAs and LangSmith](https://blog.langchain.dev/evaluating-rag-pipelines-with-ragas-langsmith/) provides a complete tutorial on how to leverage both tools to evaluate RAG pipelines.
+- [Plaban Nayak](https://nayakpplaban.medium.com/) shares his work [Evaluate RAG Pipeline using RAGAS](https://medium.aiplanet.com/evaluate-rag-pipeline-using-ragas-fbdd8dd466c1) on building and evaluating a simple RAG using LangChain and RAGAS
+- [Stephen Kurniawan](https://www.linkedin.com/in/stepkurniawan/) compares different RAG elements such as [Chunk Size](https://medium.com/@stepkurniawan/rag-chunk-size-experiment-e5e5ca437f44), [Vector Stores: FAISS vs ChromaDB](https://medium.com/@stepkurniawan/comparing-faiss-with-chroma-vector-stores-0953e1e619eb), [Vector Stores 2: Multiple Documents](https://medium.com/@stepkurniawan/comparing-faiss-vs-chroma-vector-store-retrieve-multiple-documents-07ad81a18851), and [Similarity Searches / Distance Metrics / Index Strategies](https://medium.com/@stepkurniawan/comparing-similarity-searches-distance-metrics-in-vector-stores-rag-model-f0b3f7532d6f).
+- Discover [Devanshu Brahmbhatt](https://www.linkedin.com/in/devanshubrahmbhatt/)'s insights on optimizing RAG systems in his article, [Enhancing LLM's Accuracy with RAGAS](https://devanshus-organization.gitbook.io/llm-testing-ragas). Learn about RAG architecture, key evaluation metrics, and how to use RAGAS scores to improve performance.
+- [Suzuki](https://www.linkedin.com/in/hirokazu-suzuki-206245110/) and [Hwang](https://www.linkedin.com/in/hwang-yongtae/) conducted an experiment to investigate if Ragas' performance is language-dependent by comparing the performance (correlation coefficient between human labels and scores from Ragas) using datasets of the same content in Japanese and English. They wrote blog about the result of the experiment and basic algorithm of Ragas.
+    - [RAG Evaluation: Necessity and Challenge](https://tech.beatrust.com/entry/2024/05/02/RAG_Evaluation%3A_Necessity_and_Challenge)
+    - [RAG Evaluation : Computational Metrics in RAG and Calculation Methods in Ragas](https://tech.beatrust.com/entry/2024/05/02/RAG_Evaluation_%3A_Computational_Metrics_in_RAG_and_Calculation_Methods_in_Ragas)
+    - [RAG Evaluation: Assessing the Usefulness of Ragas](https://tech.beatrust.com/entry/2024/05/02/RAG_Evaluation%3A_Assessing_the_Usefulness_of_Ragas)
+- [Atita Arora](https://www.linkedin.com/in/atitaarora/) writes about [Evaluating Retrieval Augmented Generation using RAGAS](https://superlinked.com/vectorhub/articles/retrieval-augmented-generation-eval-qdrant-ragas), an end-to-end tutorial on building RAG using [Qdrant](https://qdrant.tech/) and [LangChain](https://www.langchain.com/) and evaluating it with RAGAS.
+    - *Bonus content* : Learn how to create an evaluation dataset that serves as a reference point for evaluating our RAG pipeline, Understand the RAGAS evaluation metrics and how to make sense of them and putting them in action to test a Naive RAG pipeline and measure its performance using RAGAS metrics.
+    - *Code walkthrough* : https://github.com/qdrant/qdrant-rag-eval/tree/master/workshop-rag-eval-qdrant-ragas
+    - *Code walkthrough using [Deepset Haystack](https://haystack.deepset.ai/) and [Mixedbread.ai](https://www.mixedbread.ai/)* : https://github.com/qdrant/qdrant-rag-eval/tree/master/workshop-rag-eval-qdrant-ragas-haystack
+- [Minoru Onda](https://x.com/minorun365) writes for beginners about how to start Ragas v0.2 evaluation with Amazon Bedrock, and integrate with Langfuse.
+    - [RAG精度評価の定番ツール「Ragas」にAWSのBedrockで入門しよう！（v0.2対応） - Qiita](https://qiita.com/minorun365/items/2f4e238f8bbc6e393ba5)
+    - [生成AIアプリの出力をRagasで評価して、LangfuseでGUI監視しよう！ - Qiita](https://qiita.com/minorun365/items/70ad2f5a0afaac6e5cb9)
+- [Yunnglin](https://github.com/Yunnglin) has penned a guide on integrating Ragas v0.2 into [EvalScope](https://github.com/modelscope/eval-scope) (an evaluation framework for large models), thereby utilizing the [ModelScope](https://github.com/modelscope/modelscope) ecosystem.
+    - Tutorial: [Using Ragas with EvalScope](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/ragas.html)
+    - 教程: [在EvalScope中使用Ragas一键发起RAG评估](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/ragas.html)
+    - 最佳实践: [打破文本边界：如何进行多模态RAG评估](https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html#multimodal-rag)
+
+## **📅 Events**
+
+Stay updated with our latest gatherings, meetups, and online webinars.
+
+- OpenAI Engineers shares their [RAG tricks and features Ragas](https://youtu.be/ahnGLM-RC1Y?si=rS_WSQF8XB04PzhP) on DevDay.
+- [LangChain](https://python.langchain.com/docs/get_started/introduction)’s a [LangChain "RAG Evaluation” Webinar](https://www.crowdcast.io/c/bnx91nz59cqq) with the Ragas team
+

docs/community/pdf_export.md

@@ -0,0 +1,39 @@
+# PDF Export
+
+## Purpose
+The PDF export feature builds the complete Ragas documentation as a single PDF file using MkDocs with the `mkdocs-to-pdf` plugin.
+
+## Usage
+
+The implementation uses two separate MkDocs configurations:
+- `mkdocs.yml` for standard HTML builds (no PDF dependencies required)
+- `mkdocs-pdf.yml` which inherits from the main config and adds the PDF plugin
+
+Build PDF documentation:
+```bash
+make build-docs-pdf
+```
+
+The generated PDF will be available at `site/pdf/document.pdf`.
+
+Build HTML documentation only:
+```bash
+make build-docs
+```
+
+The `make build-docs-pdf` command automatically checks for system dependencies before building.
+
+## Mermaid diagrams in PDF (offline)
+Mermaid diagrams are rendered **offline** during the PDF build (converted to SVG before WeasyPrint runs). This requires a few additional dependencies besides WeasyPrint.
+
+### Required tools
+- Node.js (needed to run Mermaid tooling).
+- Mermaid CLI (`mmdc`), installed via `@mermaid-js/mermaid-cli`. 
+- A headless browser for Puppeteer (recommended: `chrome-headless-shell`).
+
+
+## Current Limitations
+
+**System Dependencies**: WeasyPrint requires OS-specific system libraries (Pango, Cairo) that must be installed separately. If you encounter issues, refer to the [WeasyPrint setup instructions](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html) and [troubleshooting guide](https://doc.courtbouillon.org/weasyprint/stable/first_steps.html#troubleshooting).
+
+**ReadTheDocs**: PDF generation is not currently enabled in the ReadTheDocs build configuration.

docs/concepts/components/eval_dataset.md

@@ -0,0 +1,95 @@
+# Evaluation Dataset
+
+An evaluation dataset is a homogeneous collection of [data samples](eval_sample.md) designed to assess the performance and capabilities of an AI application. In Ragas, evaluation datasets are represented using the `EvaluationDataset` class, which provides a structured way to organize and manage data samples for evaluation purposes. 
+
+- [Overview](#overview)
+- [Creating an Evaluation Dataset from SingleTurnSamples](#creating-an-evaluation-dataset-from-singleturnsamples)
+- [Loading an Evaluation Dataset from Hugging Face Datasets](#loading-an-evaluation-dataset-from-hugging-face-datasets)
+
+## Overview
+
+### Structure of an Evaluation Dataset
+
+An evaluation dataset consists of:
+
+- **Samples**: A collection of [SingleTurnSample](eval_sample.md#singleturnsample) or [MultiTurnSample](eval_sample.md#multiturnsample) instances. Each sample represents a unique interaction or scenario.
+- **Consistency**: All samples within the dataset should be of the same type (either all single-turn or all multi-turn samples) to maintain consistency in evaluation.
+
+
+### Guidelines for Curating an Effective Evaluation Dataset
+
+- **Define Clear Objectives**: Identify the specific aspects of the AI application that you want to evaluate and the scenarios you want to test. Collect data samples that reflect these objectives.
+
+- **Collect Representative Data**: Ensure that the dataset covers a diverse range of scenarios, user inputs, and expected responses to provide a comprehensive evaluation of the AI application. This can be achieved by collecting data from various sources or [generating synthetic data](./../../howtos/customizations/index.md#testset-generation).
+
+- **Quality and Size**: Aim for a dataset that is large enough to provide meaningful insights but not so large that it becomes unwieldy. Ensure that the data is of high quality and accurately reflects the real-world scenarios you want to evaluate.
+
+
+## Creating an Evaluation Dataset from SingleTurnSamples
+
+In this example, we’ll demonstrate how to create an EvaluationDataset using multiple `SingleTurnSample` instances. We’ll walk through the process step by step, including creating individual samples, assembling them into a dataset, and performing basic operations on the dataset.
+
+
+**Step 1:** Import Necessary Classes
+
+First, import the SingleTurnSample and EvaluationDataset classes from your module.
+```python
+from ragas import SingleTurnSample, EvaluationDataset
+```
+
+**Step 2:** Create Individual Samples
+
+Create several SingleTurnSample instances that represent individual evaluation samples.
+
+```python
+# Sample 1
+sample1 = SingleTurnSample(
+    user_input="What is the capital of Germany?",
+    retrieved_contexts=["Berlin is the capital and largest city of Germany."],
+    response="The capital of Germany is Berlin.",
+    reference="Berlin",
+)
+
+# Sample 2
+sample2 = SingleTurnSample(
+    user_input="Who wrote 'Pride and Prejudice'?",
+    retrieved_contexts=["'Pride and Prejudice' is a novel by Jane Austen."],
+    response="'Pride and Prejudice' was written by Jane Austen.",
+    reference="Jane Austen",
+)
+
+# Sample 3
+sample3 = SingleTurnSample(
+    user_input="What's the chemical formula for water?",
+    retrieved_contexts=["Water has the chemical formula H2O."],
+    response="The chemical formula for water is H2O.",
+    reference="H2O",
+)
+```
+
+**Step 3:** Create the EvaluationDataset
+
+Create an EvaluationDataset by passing a list of SingleTurnSample instances.
+
+```python
+dataset = EvaluationDataset(samples=[sample1, sample2, sample3])
+``` 
+
+## Loading an Evaluation Dataset from Hugging Face Datasets
+
+In practice, you may want to load an evaluation dataset from an existing dataset source, such as the Hugging Face Datasets library. The following example demonstrates how to load an evaluation dataset from a Hugging Face dataset and convert it into an EvaluationDataset instance.
+
+Ensure that the dataset contains the necessary fields for evaluation, such as user inputs, retrieved contexts, responses, and references.
+
+```python
+from datasets import load_dataset
+dataset = load_dataset("vibrantlabsai/amnesty_qa","english_v3")
+```
+
+Load the dataset into a Ragas EvaluationDataset object.
+
+```python
+from ragas import EvaluationDataset
+
+eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"])
+```

docs/concepts/components/eval_sample.md

@@ -0,0 +1,91 @@
+# Evaluation Sample
+
+An evaluation sample is a single structured data instance that is used to assess and measure the performance of your LLM application in specific scenarios. It represents a single unit of interaction or a specific use case that the AI application is expected to handle. In Ragas, evaluation samples are represented using the `SingleTurnSample` and `MultiTurnSample` classes.
+
+## SingleTurnSample
+SingleTurnSample represents a single-turn interaction between a user, LLM, and expected results for evaluation. It is suitable for evaluations that involve a single question and answer pair, possibly with additional context or reference information.
+
+
+### Example
+The following example demonstrates how to create a `SingleTurnSample` instance for evaluating a single-turn interaction in a RAG-based application. In this scenario, a user asks a question, and the AI provides an answer. We’ll create a SingleTurnSample instance to represent this interaction, including any retrieved contexts, reference answers, and evaluation rubrics.
+```python
+from ragas import SingleTurnSample
+
+# User's question
+user_input = "What is the capital of France?"
+
+# Retrieved contexts (e.g., from a knowledge base or search engine)
+retrieved_contexts = ["Paris is the capital and most populous city of France."]
+
+# AI's response
+response = "The capital of France is Paris."
+
+# Reference answer (ground truth)
+reference = "Paris"
+
+# Evaluation rubric
+rubric = {
+    "accuracy": "Correct",
+    "completeness": "High",
+    "fluency": "Excellent"
+}
+
+# Create the SingleTurnSample instance
+sample = SingleTurnSample(
+    user_input=user_input,
+    retrieved_contexts=retrieved_contexts,
+    response=response,
+    reference=reference,
+    rubric=rubric
+)
+```
+
+## MultiTurnSample
+
+MultiTurnSample represents a multi-turn interaction between Human, AI and optionally a Tool and expected results for evaluation. It is suitable for representing conversational agents in more complex interactions for evaluation. In `MultiTurnSample`, the `user_input` attribute represents a sequence of messages that collectively form a multi-turn conversation between a human user and an AI system. These messages are instances of the classes  `HumanMessage`, `AIMessage`, and `ToolMessage`
+
+
+### Example
+The following example demonstrates how to create a `MultiTurnSample` instance for evaluating a multi-turn interaction. In this scenario, a user wants to know the current weather in New York City. The AI assistant will use a weather API tool to fetch the information and respond to the user.
+
+
+```python
+from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall
+
+# User asks about the weather in New York City
+user_message = HumanMessage(content="What's the weather like in New York City today?")
+
+# AI decides to use a weather API tool to fetch the information
+ai_initial_response = AIMessage(
+    content="Let me check the current weather in New York City for you.",
+    tool_calls=[ToolCall(name="WeatherAPI", args={"location": "New York City"})]
+)
+
+# Tool provides the weather information
+tool_response = ToolMessage(content="It's sunny with a temperature of 75°F in New York City.")
+
+# AI delivers the final response to the user
+ai_final_response = AIMessage(content="It's sunny and 75 degrees Fahrenheit in New York City today.")
+
+# Combine all messages into a list to represent the conversation
+conversation = [
+    user_message,
+    ai_initial_response,
+    tool_response,
+    ai_final_response
+]
+```
+
+Now, use the conversation to create a MultiTurnSample object, including any reference responses and evaluation rubrics.
+```python
+from ragas import MultiTurnSample
+# Reference response for evaluation purposes
+reference_response = "Provide the current weather in New York City to the user."
+
+
+# Create the MultiTurnSample instance
+sample = MultiTurnSample(
+    user_input=conversation,
+    reference=reference_response,
+)
+```