microsoft--rd-agent/rdagent/app/benchmark/model/README.md

Tasks

Task Extraction

From paper to task.

# python rdagent/app/model_implementation/task_extraction.py
# It may based on rdagent/document_reader/document_reader.py
python rdagent/components/task_implementation/model_implementation/task_extraction.py ./PaperImpBench/raw_paper/

Complete workflow

From paper to implementation

# Similar to
# rdagent/app/factor_extraction_and_implementation/factor_extract_and_implement.py

Paper benchmark

# TODO: it does not work well now.
python rdagent/app/model_implementation/eval.py

TODO:

• Create reasonable benchmark
  • with uniform input
  • manually create task
• Create reasonable evaluation metrics

Evolving