wehub/patchy631--ai-engineering-hub
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:37:47 +08:00
commit 7653f56fed
1422 changed files with 359026 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
code-model-comparison/.env.example
+2
View File
@@ -0,0 +1,2 @@
OPENROUTER_API_KEY=
OPENAI_API_KEY=
code-model-comparison/.opik.config
+5
View File
@@ -0,0 +1,5 @@
[opik]
url_override = https://www.comet.com/opik/api/
workspace =
project_name = Code Evaluation
api_key =
code-model-comparison/README.md
+80
View File
@@ -0,0 +1,80 @@
# Code Generation Model Comparison using Opik
This application compares the code generation capabilities of different frontier models, that you can select from the dropdown menu, using Opik metrics. The app allows users to ingest code from a GitHub repository as context and generate new code based on that context. Both models run parallely side by side giving a fair comparison of their capabilities. Finally evaluates both models on custom code metrics and provides a detailed performance comparison with neat and clean visuals.
We use:
- LiteLLM for orchestration
- Opik for evaluation and observability
- Gitingest for ingesting code
- Streamlit for the UI
---
## Setup and Installation
Ensure you have Python 3.12 or later installed on your system.
Install dependencies:
```bash
uv sync
```
Copy `.env.example` to `.env` and configure the following environment variables:
```
OPENAI_API_KEY=your_openai_api_key_here
OPENROUTER_API_KEY=your_openrouter_api_key_here
```
Look for the `.opik.config` file in the root directory and set your respective credentials for Opik.
Run the Streamlit app:
```bash
streamlit run app.py
```
## Usage
1. Select the models you want to compare from the dropdown menu
2. Enter a GitHub repository URL in the sidebar
3. Click "Ingest Repository" to load the repository context
4. Enter your code generation prompt in the chat
5. View the generated code from both models side by side
6. Click on "Evaluate Code" to evaluate code using Opik
7. View the evaluation metrics comparing both models' performance
## Evaluation Metrics
The app evaluates generated code using three comprehensive metrics powered by Opik's G-Eval:
- **Code Correctness**: Evaluates the functional correctness of the generated code
- **Code Readability**: Measures how easy the code is to understand and maintain
- **Best Practices**: Assesses adherence to coding standards and coding best practices
Each metric is scored on a scale of 0-10, with the following general interpretation:
- 0-2: Major issues or non-functional code
- 3-5: Basic implementation with significant gaps
- 6-8: Good implementation with minor issues
- 9-10: Excellent implementation meeting all criteria
The overall score is calculated as an average of these three metrics.
---
## 📬 Stay Updated with Our Newsletter!
**Get a FREE Data Science eBook** 📖 with 150+ essential lessons in Data Science when you subscribe to our newsletter! Stay in the loop with the latest tutorials, insights, and exclusive resources. [Subscribe now!](https://join.dailydoseofds.com)
[![Daily Dose of Data Science Newsletter](https://github.com/patchy631/ai-engineering/blob/main/resources/join_ddods.png)](https://join.dailydoseofds.com)
---
## Contribution
Contributions are welcome! Please fork the repository and submit a pull request with your improvements.
code-model-comparison/app.py
+507
View File
@@ -0,0 +1,507 @@
import asyncio
import streamlit as st
import pandas as pd
import plotly.express as px
from dotenv import load_dotenv
from model_service import get_parallel_responses, get_all_model_names
from code_ingestion import ingest_github_repo
from code_evaluation_opik import evaluate_code
load_dotenv()
# Set page config
st.set_page_config(page_title="Code Generation Model Comparison", layout="wide")
# Custom CSS for responsive code containers
st.markdown(
"""
<style>
.stMarkdown {
width: 100%;
}
pre {
white-space: pre-wrap !important;
word-wrap: break-word !important;
max-width: 100% !important;
}
code {
white-space: pre-wrap !important;
word-wrap: break-word !important;
max-width: 100% !important;
}
.streamlit-expanderContent {
width: 100% !important;
}
div[data-testid="stCodeBlock"] {
white-space: pre-wrap !important;
word-wrap: break-word !important;
max-width: 100% !important;
}
</style>
""",
unsafe_allow_html=True,
)
# Initialize session state
if "chat_history" not in st.session_state:
st.session_state.chat_history = []
if "context" not in st.session_state:
st.session_state.context = None
if "reference_code" not in st.session_state:
st.session_state.reference_code = None
if "selected_models" not in st.session_state:
st.session_state.selected_models = {
"model1": None,
"model2": None,
}
if "last_generated_code" not in st.session_state:
st.session_state.last_generated_code = {"model1": None, "model2": None}
if "evaluation_results" not in st.session_state:
st.session_state.evaluation_results = {"model1": None, "model2": None}
# Main interface
st.title("Code Generation Model Comparison")
powered_by_html = """
<div style='display: flex; align-items: center; gap: 10px; margin-top: -10px;'>
<span style='font-size: 20px; color: #666;'>Powered by</span>
<img src="https://files.buildwithfern.com/openrouter.docs.buildwithfern.com/docs/2025-07-24T05:04:17.529Z/content/assets/logo-white.svg" width="180">
<span style='font-size: 20px; color: #666;'>and</span>
<img src="https://files.buildwithfern.com/https://opik.docs.buildwithfern.com/docs/opik/2025-08-01T07:08:31.326Z/img/logo-dark-mode.svg" width="100">
</div>
"""
st.markdown(powered_by_html, unsafe_allow_html=True)
# Model selection section
st.write("### Select Models to Compare")
col1, col2 = st.columns(2)
# Get all available model names
all_models = get_all_model_names()
# Validate that we have models available
if not all_models:
st.error("No models are available. Please check your configuration.")
st.stop()
# Ensure default models are valid
default_model1 = st.session_state.selected_models["model1"]
default_model2 = st.session_state.selected_models["model2"]
# If default models are not in available models, use first two available
if default_model1 not in all_models:
default_model1 = all_models[0] if all_models else "Claude Sonnet 4"
if default_model2 not in all_models:
default_model2 = all_models[1] if len(all_models) > 1 else all_models[0]
# Update session state if defaults changed
if (
default_model1 != st.session_state.selected_models["model1"]
or default_model2 != st.session_state.selected_models["model2"]
):
st.session_state.selected_models = {
"model1": default_model1,
"model2": default_model2,
}
with col1:
model1 = st.selectbox(
"Select First Model",
options=all_models,
index=all_models.index(st.session_state.selected_models["model1"]),
key="model1_select",
)
with col2:
model2 = st.selectbox(
"Select Second Model",
options=all_models,
index=all_models.index(st.session_state.selected_models["model2"]),
key="model2_select",
)
# Update session state when models change (only if they actually changed)
if (
model1 != st.session_state.selected_models["model1"]
or model2 != st.session_state.selected_models["model2"]
):
st.session_state.selected_models = {"model1": model1, "model2": model2}
# Clear previous results when models change
st.session_state.last_generated_code = {"model1": None, "model2": None}
st.session_state.evaluation_results = {"model1": None, "model2": None}
with st.sidebar:
st.title("Configuration")
github_repo = st.text_input(
"GitHub Repository URL", placeholder="https://github.com/username/repository"
)
if st.button("Ingest Repository"):
if github_repo:
with st.spinner("Ingesting repository..."):
st.session_state.context = ingest_github_repo(github_repo)
st.success("Repository ingested successfully!")
else:
st.error("Please enter a valid repository URL")
st.session_state.reference_code = st.text_area(
"Reference Code (Optional)",
help="Enter reference/ground truth code to compare against",
height=200,
)
# Evaluation section
st.write("### Evaluation")
if st.button("Evaluate Generated Code"):
if (
st.session_state.last_generated_code["model1"]
and st.session_state.last_generated_code["model2"]
):
try:
with st.spinner("Evaluating code..."):
st.session_state.evaluation_results["model1"] = evaluate_code(
st.session_state.last_generated_code["model1"],
(
st.session_state.reference_code
if st.session_state.reference_code
else None
),
)
st.session_state.evaluation_results["model2"] = evaluate_code(
st.session_state.last_generated_code["model2"],
(
st.session_state.reference_code
if st.session_state.reference_code
else None
),
)
st.success("Evaluation complete!")
except Exception as e:
st.error(f"Error during evaluation: {str(e)}")
st.error("Please try again or check your evaluation configuration.")
else:
st.error("Please generate code from both models first")
async def handle_chat_input(prompt: str):
st.session_state.chat_history.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.markdown(prompt)
# Validate context structure
if not st.session_state.context or not isinstance(st.session_state.context, dict):
st.error("Invalid context structure. Please re-ingest the repository.")
return
if "content" not in st.session_state.context:
st.error(
"Repository context is missing content. Please re-ingest the repository."
)
return
# Get streaming responses from both models
with st.chat_message("assistant"):
col1, col2 = st.columns(2)
with col1:
st.write(f"##### {st.session_state.selected_models['model1']}")
model1_container = st.empty()
model1_container = model1_container.code("", language="python")
with col2:
st.write(f"##### {st.session_state.selected_models['model2']}")
model2_container = st.empty()
model2_container = model2_container.code("", language="python")
model1_gen, model2_gen = await get_parallel_responses(
prompt,
st.session_state.context,
st.session_state.selected_models["model1"],
st.session_state.selected_models["model2"],
)
async def process_model1_stream(container):
response_text = ""
cleaned_text = "" # Initialize cleaned_text
try:
async for chunk in model1_gen:
response_text += chunk
cleaned_text = (
response_text.strip()
.removeprefix("```python")
.removeprefix("```")
.removesuffix("```")
.strip()
)
container.code(cleaned_text, language="python")
except Exception as e:
cleaned_text = f"Error processing stream: {str(e)}"
container.code(cleaned_text, language="python")
return cleaned_text
async def process_model2_stream(container):
response_text = ""
cleaned_text = "" # Initialize cleaned_text
try:
async for chunk in model2_gen:
response_text += chunk
cleaned_text = (
response_text.strip()
.removeprefix("```python")
.removeprefix("```")
.removesuffix("```")
.strip()
)
container.code(cleaned_text, language="python")
except Exception as e:
cleaned_text = f"Error processing stream: {str(e)}"
container.code(cleaned_text, language="python")
return cleaned_text
# Run both streams concurrently
try:
final_model1_response, final_model2_response = await asyncio.gather(
process_model1_stream(model1_container),
process_model2_stream(model2_container),
)
except Exception as e:
st.error(f"Critical error during model response generation: {str(e)}")
final_model1_response = "Error: Failed to generate response"
final_model2_response = "Error: Failed to generate response"
message = {
"role": "assistant",
"content": "",
"model1_response": final_model1_response,
"model2_response": final_model2_response,
"model1_name": st.session_state.selected_models["model1"],
"model2_name": st.session_state.selected_models["model2"],
}
st.session_state.chat_history.append(message)
st.session_state.last_generated_code["model1"] = final_model1_response
st.session_state.last_generated_code["model2"] = final_model2_response
# Display chat history
for message in st.session_state.chat_history:
with st.chat_message(message["role"]):
st.markdown(message["content"])
if message["role"] == "assistant":
col1, col2 = st.columns(2)
with col1:
model1_name = message.get("model1_name", "Model 1")
st.write(f"##### {model1_name}")
st.code(message["model1_response"], language="python")
with col2:
model2_name = message.get("model2_name", "Model 2")
st.write(f"##### {model2_name}")
st.code(message["model2_response"], language="python")
if prompt := st.chat_input("What code would you like to generate?"):
if not st.session_state.context:
st.error("Please ingest a GitHub repository first!")
else:
try:
# Validate that selected models are still available
all_models = get_all_model_names()
if (
st.session_state.selected_models["model1"] not in all_models
or st.session_state.selected_models["model2"] not in all_models
):
st.error(
"One or more selected models are no longer available. Please reselect models."
)
else:
asyncio.run(handle_chat_input(prompt))
except Exception as e:
st.error(f"An error occurred while generating code: {str(e)}")
st.error("Please try again or check your configuration.")
# Display evaluation results
if (
st.session_state.evaluation_results["model1"]
and st.session_state.evaluation_results["model2"]
):
try:
st.write("---")
st.header("Evaluation results generated with GPT-4o using Opik")
# Validate evaluation results structure
def validate_evaluation_result(result, model_name):
if not result or not isinstance(result, dict):
return False
if "detailed_metrics" not in result or "overall_score" not in result:
return False
required_metrics = ["correctness", "readability", "best_practices"]
for metric in required_metrics:
if metric not in result["detailed_metrics"]:
return False
if "score" not in result["detailed_metrics"][metric]:
return False
return True
model1_valid = validate_evaluation_result(
st.session_state.evaluation_results["model1"], "model1"
)
model2_valid = validate_evaluation_result(
st.session_state.evaluation_results["model2"], "model2"
)
if not model1_valid:
st.error("Invalid evaluation result structure for model 1")
elif not model2_valid:
st.error("Invalid evaluation result structure for model 2")
else:
# Only proceed with plotting if both models are valid
plot_data = pd.DataFrame(
{
"Metric": [
"Correctness",
"Readability",
"Best Practices",
"Overall Score",
],
st.session_state.selected_models["model1"]: [
st.session_state.evaluation_results["model1"][
"detailed_metrics"
]["correctness"]["score"],
st.session_state.evaluation_results["model1"][
"detailed_metrics"
]["readability"]["score"],
st.session_state.evaluation_results["model1"][
"detailed_metrics"
]["best_practices"]["score"],
st.session_state.evaluation_results["model1"]["overall_score"],
],
st.session_state.selected_models["model2"]: [
st.session_state.evaluation_results["model2"][
"detailed_metrics"
]["correctness"]["score"],
st.session_state.evaluation_results["model2"][
"detailed_metrics"
]["readability"]["score"],
st.session_state.evaluation_results["model2"][
"detailed_metrics"
]["best_practices"]["score"],
st.session_state.evaluation_results["model2"]["overall_score"],
],
}
)
fig = px.bar(
plot_data.melt("Metric", var_name="Model", value_name="Score"),
x="Metric",
y="Score",
color="Model",
barmode="group",
title="Model Performance Comparison",
template="plotly_dark",
color_discrete_sequence=["#00CED1", "#FF69B4"],
)
fig.update_layout(
xaxis_title="Evaluation Metrics",
yaxis_title="Score",
legend_title="Models",
plot_bgcolor="rgba(32, 32, 32, 1)",
paper_bgcolor="rgba(32, 32, 32, 1)",
bargap=0.2,
bargroupgap=0.1,
font=dict(color="#E0E0E0"),
title_font=dict(color="#E0E0E0"),
showlegend=True,
legend=dict(
bgcolor="rgba(32, 32, 32, 0.8)",
bordercolor="rgba(255, 255, 255, 0.3)",
borderwidth=1,
),
)
fig.update_xaxes(
gridcolor="rgba(128, 128, 128, 0.2)",
zerolinecolor="rgba(128, 128, 128, 0.2)",
)
fig.update_yaxes(
gridcolor="rgba(128, 128, 128, 0.2)",
zerolinecolor="rgba(128, 128, 128, 0.2)",
)
st.plotly_chart(fig, use_container_width=True)
st.write(
f"### {st.session_state.selected_models['model1']} detailed metrics"
)
model1_data = []
for metric in ["correctness", "readability", "best_practices"]:
row = {
"Metric": metric.title(),
"Score": f"{st.session_state.evaluation_results['model1']['detailed_metrics'][metric]['score']:.2f}",
"Reasoning": st.session_state.evaluation_results["model1"][
"detailed_metrics"
][metric]["reason"],
}
model1_data.append(row)
model1_data.append(
{
"Metric": "Overall Score",
"Score": f"{st.session_state.evaluation_results['model1']['overall_score']:.2f}",
"Reasoning": "Final weighted average",
}
)
# Display Model 1 table
model1_df = pd.DataFrame(model1_data)
st.dataframe(
model1_df,
column_config={
"Metric": st.column_config.TextColumn("Metric", width="small"),
"Score": st.column_config.TextColumn("Score", width="small"),
"Reasoning": st.column_config.TextColumn(
"Reasoning", width="large"
),
},
hide_index=True,
use_container_width=True,
)
st.write(
f"### {st.session_state.selected_models['model2']} detailed metrics"
)
model2_data = []
for metric in ["correctness", "readability", "best_practices"]:
row = {
"Metric": metric.title(),
"Score": f"{st.session_state.evaluation_results['model2']['detailed_metrics'][metric]['score']:.2f}",
"Reasoning": st.session_state.evaluation_results["model2"][
"detailed_metrics"
][metric]["reason"],
}
model2_data.append(row)
model2_data.append(
{
"Metric": "Overall Score",
"Score": f"{st.session_state.evaluation_results['model2']['overall_score']:.2f}",
"Reasoning": "Final weighted average",
}
)
# Display Model 2 table
model2_df = pd.DataFrame(model2_data)
st.dataframe(
model2_df,
column_config={
"Metric": st.column_config.TextColumn("Metric", width="small"),
"Score": st.column_config.TextColumn("Score", width="small"),
"Reasoning": st.column_config.TextColumn(
"Reasoning", width="large"
),
},
hide_index=True,
use_container_width=True,
)
except Exception as e:
st.error(f"Error displaying evaluation results: {str(e)}")
st.error("Please try running the evaluation again.")
code-model-comparison/code_evaluation_opik.py
+172
View File
@@ -0,0 +1,172 @@
from opik.evaluation.metrics import GEval
def evaluate_code(generated_code: str, reference_code: str = None):
"""
Evaluate generated Python code using Comet Opik's GEval metrics.
1. Code Correctness - Assesses functional correctness, edge case handling,
and completeness of implementation
2. Code Readability - Evaluates naming conventions, formatting, documentation,
and overall code structure
3. Code Best Practices - Checks error handling, security practices, efficiency,
and modularity
Args:
generated_code (str): The Python code to evaluate
reference_code (str, optional): Reference code for comparison. If provided,
the correctness evaluation will compare against this reference.
Returns:
dict: A dictionary containing evaluation results with the following structure:
{
"overall_score": float, # Average score across all metrics (0-10 scale)
"detailed_metrics": {
"correctness": {"score": float, "reason": str},
"readability": {"score": float, "reason": str},
"best_practices": {"score": float, "reason": str}
},
"passed": bool, # Whether overall_score >= 7.0 (70% threshold)
"error": str, optional # Error message if evaluation fails
}
"""
try:
# Validate input
if not generated_code or not generated_code.strip():
raise ValueError("Generated code cannot be empty")
# Build the context string that includes both actual and expected code
context = f"ACTUAL_CODE:\n```\n{generated_code}\n```"
if reference_code:
context += f"\nEXPECTED_CODE:\n```\n{reference_code}\n```"
# Define rubric scoring criteria
correctness_rubric_text = (
"Score 0-2: Code is non-functional or has critical errors\n"
"Score 3-5: Code works but misses key functionality\n"
"Score 6-8: Code is mostly correct with minor issues\n"
"Score 9-10: Code is completely correct"
)
readability_rubric_text = (
"Score 0-2: Code is poorly formatted and hard to read\n"
"Score 3-5: Code has basic formatting but lacks clarity\n"
"Score 6-8: Code is well formatted with minor issues\n"
"Score 9-10: Code is exceptionally readable and well documented"
)
best_practices_rubric_text = (
"Score 0-2: Code ignores best practices\n"
"Score 3-5: Code follows basic practices with gaps\n"
"Score 6-8: Code mostly follows best practices\n"
"Score 9-10: Code perfectly follows all best practices"
)
# 1. Code Correctness Metric
correctness_metric = GEval(
task_introduction=(
"You are an expert judge evaluating Python code correctness. "
"The expected implementation is under EXPECTED_CODE and the submitted code is under ACTUAL_CODE. "
"Assess if the code is functionally correct, handles edge cases, and fully implements the required functionality. "
"Use the following rubric to assign scores:"
),
evaluation_criteria=(
"EVALUATION STEPS:\n"
"1. Check if all required functionality is implemented.\n"
"2. Verify proper handling of edge cases.\n"
"3. Identify potential runtime errors.\n"
"4. Confirm the code produces the expected outputs.\n\n"
"SCORING RUBRIC:\n"
f"{correctness_rubric_text}\n\n"
"Return only a score between 0 and 10, and a concise reason that references the rubric."
),
name="Code Correctness",
)
# 2. Code Readability Metric
readability_metric = GEval(
task_introduction=(
"You are an expert judge evaluating Python code readability. "
"The code to review is under ACTUAL_CODE. Focus on naming, formatting, and documentation. "
"Use the following rubric to assign scores:"
),
evaluation_criteria=(
"EVALUATION STEPS:\n"
"1. Are naming conventions clear and consistent?\n"
"2. Is formatting and indentation correct?\n"
"3. Are comments and docstrings complete and helpful?\n"
"4. Is the code organized logically?\n\n"
"SCORING RUBRIC:\n"
f"{readability_rubric_text}\n\n"
"Return only a score between 0 and 10, and a concise reason that references the rubric."
),
name="Code Readability",
)
# 3. Code Best Practices Metric
best_practices_metric = GEval(
task_introduction=(
"You are an expert judge evaluating adherence to Python best practices. "
"The code to review is under ACTUAL_CODE. Focus on error handling, security, efficiency, and modularity. "
"Use the following rubric to assign scores:"
),
evaluation_criteria=(
"EVALUATION STEPS:\n"
"1. Does it handle exceptions and errors properly?\n"
"2. Are security best practices followed?\n"
"3. Is the code efficient in performance?\n"
"4. Is functionality split into reusable, modular components?\n\n"
"SCORING RUBRIC:\n"
f"{best_practices_rubric_text}\n\n"
"Return only a score between 0 and 10, and a concise reason that references the rubric."
),
name="Code Best Practices",
)
# Run evaluation for each metric using Opik's GEval
correctness_result = correctness_metric.score(output=context)
readability_result = readability_metric.score(output=context)
best_practices_result = best_practices_metric.score(output=context)
# Convert scores from Opik's 0-1 scale to 0-10 scale
# Opik returns scores as 0-1, we multiply by 10 for consistency
correctness_score = correctness_result.value * 10
readability_score = readability_result.value * 10
best_practices_score = best_practices_result.value * 10
# Calculate overall score as average of all three metrics
overall_score = (
correctness_score + readability_score + best_practices_score
) / 3
# Prepare detailed metrics structure
detailed_metrics = {
"correctness": {
"score": correctness_score,
"reason": correctness_result.reason,
},
"readability": {
"score": readability_score,
"reason": readability_result.reason,
},
"best_practices": {
"score": best_practices_score,
"reason": best_practices_result.reason,
},
}
# Return results
return {
"overall_score": overall_score,
"detailed_metrics": detailed_metrics,
"passed": overall_score >= 7.0, # 70% threshold
}
except Exception as e:
# Error handling
return {
"error": f"Error evaluating code: {str(e)}",
"overall_score": 0.0,
"detailed_metrics": {},
"passed": False,
}
code-model-comparison/code_ingestion.py
+24
View File
@@ -0,0 +1,24 @@
from gitingest import ingest
def ingest_github_repo(repo_url: str) -> dict[str, str]:
# Validate GitHub URL format
if not repo_url or not isinstance(repo_url, str):
raise ValueError("Repository URL must be a non-empty string")
if not repo_url.startswith(("https://github.com/", "http://github.com/")):
raise ValueError("URL must be a valid GitHub repository URL")
try:
# Use gitingest to process the repository
summary, structure, content = ingest(repo_url)
context = {
"summary": summary,
"structure": structure,
"content": content
}
return context
except Exception as e:
raise Exception(f"Error ingesting repository: {str(e)}") from e
code-model-comparison/model_service.py
+124
View File
@@ -0,0 +1,124 @@
import os
import asyncio
from litellm import acompletion
from typing import Dict, Any
# Available models
AVAILABLE_MODELS = {
"Claude Sonnet 4": "openrouter/anthropic/claude-sonnet-4",
"Qwen3-Coder": "openrouter/qwen/qwen3-coder",
"Gemini 2.5 Flash": "openrouter/google/gemini-2.5-flash",
"GPT-4.1": "openrouter/openai/gpt-4.1",
}
async def get_model_response_async(
model_name: str, prompt: str, context: Dict[str, Any]
):
user_prompt = f"""
You are an expert code generator. Your task is to generate code based on the following repository context:
Repository Context:
{context['content']}
Instructions:
1. Generate code that strictly follows the repository's existing patterns and conventions
2. Use the same coding style, naming conventions, and structure as the codebase
3. Include clear, concise docstrings and comments explaining key functionality
4. Ensure the code integrates seamlessly with existing components
5. Focus on maintainability and readability
User query:
{prompt}
Output only the code implementation without explanations or additional text.
"""
messages = [{"role": "user", "content": user_prompt}]
# Find the model mapping for the given model name
try:
model_mapping = get_model_mapping(model_name)
except ValueError as e:
yield f"Error: {str(e)}"
return
try:
# Get streaming response from the model using LiteLLM asynchronously.
response = await acompletion(
model=model_mapping,
messages=messages,
api_key=os.getenv("OPENROUTER_API_KEY"),
max_tokens=2000,
stream=True,
)
if not response:
yield "Error: No response received from model"
return
async for chunk in response:
if chunk and hasattr(chunk, "choices") and chunk.choices:
if chunk.choices[0].delta and chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
except Exception as e:
error_msg = f"Error generating response: {str(e)}"
if "api_key" in str(e).lower() or "authentication" in str(e).lower():
error_msg = "Error: Invalid or missing API key. Please check your OPENROUTER_API_KEY configuration."
elif "quota" in str(e).lower() or "limit" in str(e).lower():
error_msg = "Error: API quota exceeded or rate limit reached. Please try again later."
elif "model" in str(e).lower():
error_msg = f"Error: Model '{model_name}' is not available or has issues. Please try a different model."
yield error_msg
async def get_parallel_responses(
prompt: str, context: Dict[str, Any], model1: str, model2: str
):
"""
Get parallel responses from two selected models.
Args:
prompt: The user prompt
context: Repository context
model1: Name of the first model
model2: Name of the second model
Returns:
Tuple of two async generators for the model responses
"""
gen1 = get_model_response_async(model1, prompt, context)
gen2 = get_model_response_async(model2, prompt, context)
return gen1, gen2
def get_model_responses(prompt: str, context: Dict[str, Any], model1: str, model2: str):
loop = asyncio.get_event_loop()
return loop.run_until_complete(
get_parallel_responses(prompt, context, model1, model2)
)
def get_all_model_names():
"""Get all available model names for dropdown selection."""
try:
return list(AVAILABLE_MODELS.keys())
except Exception as e:
print(f"Error getting model names: {e}")
return []
def validate_model_name(model_name: str) -> bool:
"""Validate if a model name exists in available models."""
return model_name in AVAILABLE_MODELS
def get_model_mapping(model_name: str) -> str:
"""Get the model mapping for a given model name."""
if not validate_model_name(model_name):
raise ValueError(f"Model '{model_name}' not found in available models")
return AVAILABLE_MODELS[model_name]
code-model-comparison/pyproject.toml
+20
View File
@@ -0,0 +1,20 @@
[project]
name = "code-model-comparison"
version = "0.1.0"
description = "Code Generation model comparison using OpenRouter and Opik"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"anthropic>=0.49.0",
"deepeval>=3.0.0",
"gitingest>=0.1.4",
"ipykernel>=6.29.5",
"ipywidgets>=8.1.7",
"litellm>=1.71.1",
"nest-asyncio>=1.6.0",
"opik>=1.8.13",
"pandas>=2.2.3",
"plotly>=6.1.2",
"python-dotenv>=1.1.0",
"streamlit>=1.45.1",
]
code-model-comparison/uv.lock
Generated
+2661
View File
File diff suppressed because it is too large Load Diff