chore: import upstream snapshot with attribution

examples/huggingface/hle/README.md

@@ -0,0 +1,108 @@
+# huggingface/hle (Humanity's Last Exam)
+
+Evaluate LLMs against [Humanity's Last Exam (HLE)](https://arxiv.org/abs/2501.14249), a challenging benchmark created by 1,000+ experts across 500+ institutions. HLE features 3,000+ questions spanning 100+ subjects, designed to push AI capabilities to their limits.
+
+**📖 [Read the complete HLE benchmark guide →](https://www.promptfoo.dev/docs/guides/hle-benchmark/)**
+
+You can run this example with:
+
+```bash
+npx promptfoo@latest init --example huggingface/hle
+cd huggingface/hle
+```
+
+## Prerequisites
+
+- OpenAI API key set as `OPENAI_API_KEY`
+- Anthropic API key set as `ANTHROPIC_API_KEY`
+- Hugging Face access token (required for dataset access)
+
+## Setup
+
+Set your Hugging Face token:
+
+```bash
+export HF_TOKEN=your_token_here
+```
+
+Or add it to your `.env` file:
+
+```env
+HF_TOKEN=your_token_here
+```
+
+Get your token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
+
+## Run the Evaluation
+
+Run the evaluation:
+
+```bash
+npx promptfoo@latest eval
+```
+
+View results:
+
+```bash
+npx promptfoo@latest view
+```
+
+## What's Tested
+
+This evaluation tests models on:
+
+- Advanced mathematics and sciences
+- Humanities and social sciences
+- Professional domain knowledge
+- Multimodal reasoning
+- Interdisciplinary topics
+
+Each question is evaluated for accuracy using an LLM judge that compares the model's response against the verified correct answer.
+
+## Current AI Performance
+
+HLE is designed to be extremely challenging. Recent model performance:
+
+- **OpenAI Deep Research**: 26.6% accuracy
+- **o4-mini**: 18.1% accuracy
+- **DeepSeek-R1**: 9.4% accuracy
+
+Low scores are expected - this benchmark represents the cutting edge of AI evaluation.
+
+## Customization
+
+### Test More Questions
+
+Increase the sample size:
+
+```yaml
+tests:
+  - huggingface://datasets/cais/hle?split=test&limit=100
+```
+
+### Add More Models
+
+Compare multiple providers:
+
+```yaml
+providers:
+  - anthropic:claude-sonnet-4-6
+  - openai:o4-mini
+  - deepseek:deepseek-reasoner
+```
+
+### Different Prompting
+
+Try alternative prompting strategies by modifying `prompt.py` or using static prompts:
+
+```yaml
+prompts:
+  - 'Answer this question step by step: {{question}}'
+  - file://prompt.py:create_hle_prompt
+```
+
+## Resources
+
+- [HLE Paper](https://arxiv.org/abs/2501.14249)
+- [HLE Dataset](https://huggingface.co/datasets/cais/hle)
+- [Promptfoo Documentation](https://promptfoo.dev/docs/getting-started)

examples/huggingface/hle/prompt.py

@@ -0,0 +1,169 @@
+"""
+HLE (Humanity's Last Exam) Prompt Generator
+
+This file creates properly formatted prompts for the HLE benchmark.
+It handles both multiple choice and exact answer questions, with support for images.
+"""
+
+import json
+
+
+def create_hle_prompt(context):
+    """
+    Creates a chat message prompt for HLE benchmark questions.
+
+    This function takes HLE dataset variables and formats them into the correct
+    chat message structure for different AI models.
+
+    Args:
+        context (dict): Contains two keys:
+            - 'vars': The question data from HLE dataset
+            - 'provider': Information about which AI model is being used
+
+    Returns:
+        str: JSON string of chat messages ready to send to the AI model
+    """
+    # Extract the data we need
+    question_data = context["vars"]
+    model_info = context["provider"]
+
+    # Step 1: Choose the right instruction format
+    instructions = _get_response_instructions(question_data)
+
+    # Step 2: Build the complete question text
+    full_question = _build_question_text(question_data)
+
+    # Step 3: Create the chat messages
+    messages = _create_chat_messages(instructions, full_question, model_info)
+
+    # Step 4: Add image if present
+    if _has_image(question_data):
+        image_message = _create_image_message(question_data, model_info)
+        messages.append(image_message)
+
+    return json.dumps(messages)
+
+
+def _get_response_instructions(question_data):
+    """
+    Returns the appropriate response format instructions based on question type.
+
+    HLE has two types of questions:
+    - exactMatch: Questions that need a specific exact answer
+    - multipleChoice: Questions with A/B/C/D options
+    """
+    question_type = question_data.get("answer_type", "multipleChoice")
+
+    if question_type == "exactMatch":
+        return """Your response should be in the following format:
+Explanation: {your explanation for your final answer}
+Exact Answer: {your succinct, final answer}
+Confidence: {your confidence score between 0% and 100% for your answer}"""
+    else:
+        return """Your response should be in the following format:
+Explanation: {your explanation for your answer choice}
+Answer: {your chosen answer}
+Confidence: {your confidence score between 0% and 100% for your answer}"""
+
+
+def _build_question_text(question_data):
+    """
+    Builds the complete question text, including multiple choice options if present.
+    """
+    question_text = question_data["question"]
+
+    # Add multiple choice options if this is a multiple choice question
+    choices = question_data.get("choices", [])
+    if choices and len(choices) > 0:
+        question_text += "\n\nOptions:\n"
+        for i, choice in enumerate(choices):
+            letter = chr(65 + i)  # A, B, C, D, etc.
+            question_text += f"{letter}) {choice}\n"
+
+    return question_text
+
+
+def _create_chat_messages(instructions, question_text, model_info):
+    """
+    Creates the basic chat message structure.
+
+    Note: Some OpenAI reasoning models (o1, o3) use 'developer' role instead of 'system'
+    """
+    model_id = model_info.get("id", "")
+
+    # OpenAI's reasoning models use 'developer' instead of 'system'
+    if "o1" in model_id or "o3" in model_id:
+        system_role = "developer"
+    else:
+        system_role = "system"
+
+    return [
+        {"role": system_role, "content": instructions},
+        {"role": "user", "content": question_text},
+    ]
+
+
+def _has_image(question_data):
+    """
+    Checks if this question includes an image.
+    """
+    image_url = question_data.get("image", "")
+    return image_url and image_url.strip() and image_url != "null"
+
+
+def _create_image_message(question_data, model_info):
+    """
+    Creates an image message in the correct format for the AI model.
+
+    Different AI providers use different image formats:
+    - OpenAI: Uses 'image_url' format
+    - Anthropic (Claude): Uses 'image' with 'source' format
+    """
+    image_url = question_data["image"]
+    model_id = model_info.get("id", "")
+
+    if "anthropic" in model_id or "claude" in model_id:
+        # Claude format: expects base64 data
+        return _create_claude_image_message(image_url)
+    else:
+        # OpenAI format: can handle URLs directly
+        return _create_openai_image_message(image_url)
+
+
+def _create_claude_image_message(image_url):
+    """
+    Creates image message in Claude/Anthropic format.
+    """
+    # If it's a data URL (data:image/jpeg;base64,xxxxx), extract just the base64 part
+    if image_url.startswith("data:"):
+        # Extract media type and base64 data
+        header, base64_data = image_url.split(",", 1)
+        media_type = header.split(":")[1].split(";")[0]
+    else:
+        # Assume it's already base64 data
+        media_type = "image/jpeg"
+        base64_data = image_url
+
+    return {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "source": {
+                    "type": "base64",
+                    "media_type": media_type,
+                    "data": base64_data,
+                },
+            }
+        ],
+    }
+
+
+def _create_openai_image_message(image_url):
+    """
+    Creates image message in OpenAI format.
+    """
+    return {
+        "role": "user",
+        "content": [{"type": "image_url", "image_url": {"url": image_url}}],
+    }

examples/huggingface/hle/promptfooconfig.yaml

@@ -0,0 +1,38 @@
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: HLE benchmark evaluation
+
+prompts:
+  - file://prompt.py:create_hle_prompt
+
+providers:
+  - id: anthropic:claude-sonnet-4-6
+    config:
+      thinking:
+        type: enabled
+        budget_tokens: 3000
+      max_tokens: 4000
+  - id: openai:o4-mini
+    config:
+      max_completion_tokens: 4000
+  # - deepseek:deepseek-reasoner
+
+defaultTest:
+  assert:
+    - type: llm-rubric
+      value: |
+        Evaluate whether the response correctly answers the question.
+
+        Question: {{ question }}
+        Model Response: {{ output }}
+        Correct Answer: {{ answer }}
+
+        Grade the response on accuracy (0.0 to 1.0 scale):
+        - 1.0: Response matches the correct answer exactly or is mathematically/logically equivalent
+        - 0.8-0.9: Response is mostly correct with minor differences that don't affect correctness
+        - 0.5-0.7: Response is partially correct but has significant errors
+        - 0.0-0.4: Response is incorrect or doesn't address the question
+
+        The response should pass if it demonstrates correct understanding and provides the right answer, even if the explanation differs from the expected format.
+
+tests:
+  - huggingface://datasets/cais/hle?split=test&limit=50