Competitor audit — 2026-10-18¶
This document distinguishes documented features from measured retrieval. No source's README is an independent validation of scientific accuracy.
Verified source revisions¶
| Project | Git SHA snapshot | Sources |
|---|---|---|
| Literature Evidence MCP | af642baa1917ea47741263276db921f1e04149c4 baseline v0.2 |
https://github.com/Furox-Art/literature-evidence-mcp |
| Athenaeum | 7977804753b2854b5144ec4865fd7b6f76d23d2a |
https://github.com/huuunleashed/athenaeum-mcp |
| Paper Search MCP | cab1b5cc276a0cd004a0044003926a6440088049 |
https://github.com/openags/paper-search-mcp |
| Scholarly MCP | README examined, no SHA recorded here | https://github.com/punit-naik/scholarly-mcp |
Feature evidence (implementation or claimed interface, not measured quality)¶
| Feature | Literature Evidence MCP v0.3 | Athenaeum | Paper Search MCP | Scholarly MCP |
|---|---|---|---|---|
| Multi-provider paper discovery | 3 public sources | Documented, >3 sources | Documented, >3 sources | Documented, 6 sources |
| Open-access full text | Europe PMC JATS XML | PDF reader documented | PDF reader/download documented | HTML/PDF reader documented |
| Direct source paragraph anchors with SHA-256 | Implemented JATS paragraph IDs | Not verified from reviewed README | Not verified from reviewed README | Not verified from reviewed README |
| Bibliography export | Not implemented | Multiple styles + Zotero | Source metadata / citation export partially documented | BibTeX + RIS |
| Citation graph crawl | User-supplied evidence edges only | Implemented traversal | OpenAlex citation links documented | DOI citing-paper retrieval |
| Full expert risk-of-bias evaluation | Not implemented | Abstract rigor cues/score, not formal RoB2 | Not verified | Not verified |
| Structured evidence annotation/consistency screening | Human-labeled source quote tracing | Abstract cue heuristic, not comparable | Not verified | Not verified |
| Independently validated scientific claim accuracy | No | No verified claim | No verified claim | No verified claim |
The terms 'not verified' and 'not implemented' are deliberately different. Unlike Athenaeum's larger aggregator and Paper Search MCP's provider catalog, our v0.3 focuses on evidence-chain auditability, not broad coverage.
Comparable live search checks¶
See reports/live-*.json, generated by scripts/compare_search_engines.py.
This is a three-query smoke, without expert relevance labels. One pass
per engine with shared provider set OpenAlex / Crossref / Europe PMC, at
five hits per provider (up to 15 merged). It records the observed DOIs,
timeouts/errors and wall-clock duration. Cold/warm caches, shared public
quota, provider response order, and asynchronous failures can differ.
The Athenaeum implementation may silently suppress per-provider errors.
No search precision, recall, scientific claim accuracy or supremacy is estimated by this exercise. Comparators are installed separately to prevent cross-project dependency contamination.
Scientific standards¶
RoB 2: https://methods.cochrane.org/bias/resources/rob-2-revised-cochrane-risk-bias-tool-randomized-trials
PRISMA 2020: https://www.prisma-statement.org/prisma-2020
The present evidence traceability checklist must never be called a certified RoB 2 or PRISMA 2020 complete review.
2026-10-19 engineering update (not a competitive win claim)¶
The original dated v0.3 comparison above remains historical. Current
Literature Evidence MCP adds the offline export_bibliography tool
with BibTeX/RIS/CSL-JSON serialization. This closes the previously
documented basic metadata reference export gap but is not the same
as Zotero synchronization, citation network traversal, wider paper
coverage, or a prospective same-task competitor comparison.
Other open gaps: broader provider integration; automated multi-round citation expansion and scoring (method/design decision required); integrated Zotero actions; blind expert adjudication of evidence-quality/claim predictions; and head-to-head identical-dataset/time-budget statistical benchmarking. No scientific superiority follows from feature checklists or test coverage.
2026-10-19: extended search and citation graph (opt-in, not a win claim)¶
Following the historical feature matrix above, two more MCP tools were added:
search_literature_extended: fourth provider Semantic Scholar with explicit rate-limit/HTTP errors, DOI-anchored reconciliation and conflict reporting.expand_citation_chain: bounded (two hops maximum) forward/backward OpenAlex citation exploration with explicit source IDs and truncation.
This closes basic multi-source expansion/citation-traversal gaps. It does not close large-scale provider coverage, algorithmically ranked snowballing, Zotero synchronization, blind relevance benchmarking or end-to-end systematic-review accuracy. Live Semantic Scholar returned a rate-limit error; no dependable production availability of that provider was demonstrated.
Two-track head-to-head audit harness (2026-10-19)¶
A reusable offline protocol now separately scores retrieval of known positives and expert evidence verdicts on exactly matched cases across >=2 systems; see dual-track protocol. Immutable input-byte checks, error/abstention preservation, and descriptive paired deltas are implemented. This is evaluation infrastructure only. No independent blind gold set, authenticated competitor execution, or preregistered primary outcome has been supplied for these two tracks. Therefore no new measured competitive superiority or public benchmark win is claimed.
2026-10-19 actual live three-engine execution (no relevance gold)¶
The three pinned open-source systems Literature Evidence MCP, Athenaeum
and Paper Search MCP were actually installed/executed in isolated Python
environments on the identical three historical free-text queries and the
same OpenAlex/Crossref/Europe PMC provider selection (max five/provider).
See documented live execution and its scientific limits
and reports/evaluation/live_competitor_same_query_20261019.json.
Across three queries, the observed record/DOI-bearing counts were 43/38,
45/41, and 44/39 respectively. Athenaeum's possible provider-error
suppression prevents exact status comparability. This does not rank scientific
relevance, completeness, or evidence accuracy.
A prospective expert-review operational protocol was also added. It does not mean a new expert dataset has been sourced or that independent blind evaluation was executed.