wehub-resource-sync
4cddfcf2f3
Backend Tests / Tests (other) (push) Failing after 1s
Backend Tests / Static Checks (push) Failing after 2s
Backend Tests / Tests (internal) (push) Failing after 2s
Backend Tests / Tests (server) (push) Failing after 1s
Backend Tests / Tests (store) (push) Failing after 1s
Build Canary Image / build-frontend (push) Failing after 1s
Frontend Tests / Lint (push) Failing after 1s
Build Canary Image / build-push (linux/amd64) (push) Has been skipped
Build Canary Image / build-push (linux/arm64) (push) Has been skipped
Build Canary Image / merge (push) Has been skipped
Frontend Tests / Build (push) Failing after 1s
Release Please / release-please (push) Failing after 0s
Proto Linter / Lint Protos (push) Failing after 2s
63 lines
2.7 KiB
XML
63 lines
2.7 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Task-level evaluation for the memos MCP server.
|
|
|
|
These questions test whether an LLM can accomplish realistic tasks by
|
|
composing the curated MCP tools (memo_list_memos with CEL filters,
|
|
memo_list_memo_comments, memo_list_memo_reactions, shortcut_list_shortcuts,
|
|
auth_get_current_user, ...). They are NOT unit tests of the server plumbing —
|
|
see *_test.go for that.
|
|
|
|
Answers are pinned to the deterministic seed in
|
|
store/seed/sqlite/01__dump.sql and were verified by querying a fresh
|
|
demo-mode instance. Run against a freshly seeded instance ONLY (the public
|
|
demo at demo.usememos.com is a shared, mutable account and will not match).
|
|
See README.md in this directory for how to launch one.
|
|
|
|
Questions deliberately avoid absolute dates: the seed uses relative
|
|
timestamps (strftime('now','-N days')), so only relative ordering, counts,
|
|
and content are stable.
|
|
-->
|
|
<evaluation>
|
|
<qa_pair>
|
|
<question>Look at every memo in the instance and add up all the emoji reactions across them. What is the total number of reactions?</question>
|
|
<answer>12</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many memos are pinned?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Considering only the memos that are NOT pinned, one of them has more reactions than any other. That memo is about a book the author started reading. What is the title of that book?</question>
|
|
<answer>Deep Work</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Two memos are pinned. One contains external hyperlinks and the other does not. Give the single tag of the pinned memo that contains hyperlinks.</question>
|
|
<answer>sponsors</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many memos contain at least one fenced code block?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>How many distinct users have authored the memos in this instance?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Find the memo that is a cheat sheet of git commands. How many comments does it have?</question>
|
|
<answer>2</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Find the memo containing a movie watchlist. What is the username of the user who created it?</question>
|
|
<answer>alice</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Exactly one memo is a travel bucket list. It names a Nordic country the author plans to visit in winter to see the Northern Lights. Which country?</question>
|
|
<answer>Iceland</answer>
|
|
</qa_pair>
|
|
<qa_pair>
|
|
<question>Using the authenticated session, what is the username (login name) of the account these memos belong to?</question>
|
|
<answer>demo</answer>
|
|
</qa_pair>
|
|
</evaluation>
|