External references.

This layer serves three functions, not one. First, it canonicalizes: every compound and condition the LLM extracts is resolved against a canonical biomedical registry and rewritten with that registry's standard identifier before being written to Whel's database. Compounds resolve against ChEMBL or DrugBank; conditions resolve against MONDO (the same ontology Whel already uses for the MATRIX cross-reference above). Second, it enriches: the resolution call returns structured metadata (generic name, drug class, ATC code, known targets for a compound; ontology lineage for a condition) that travels with the signal into the database, changing the shape of the data Whel stores. Third, it gates: entities that fail to resolve are flagged for human review rather than silently stored, which catches the structured-output hallucination class of error documented in the LLM literature.

• ›Per-pipeline entity resolution rate. Percentage of LLM-extracted entities that resolved against the canonical ontology, broken down by pipeline (PubMed, ClinicalTrials.gov, FDA AEMS, Open Targets, Reddit). A pipeline with a noticeably lower resolution rate is a pipeline whose extraction prompt is producing more hallucinated entities.
• ›Per-condition resolution rate. So resolution quality differences across the six conditions are visible rather than averaged away. A condition whose extracted compounds resolve at a lower rate is a condition where extraction is less trustworthy.
• ›Count of entities currently flagged for human review. With the pipeline and condition each was extracted from, and the reason resolution failed (no matching identifier, ambiguous match across multiple registered compounds, deprecated identifier).
• ›Sample of unresolved entities from the most recent run. So the failure mode is concrete rather than abstract. A reader can see the actual text the LLM produced and judge for themselves whether the rejection is a true positive or whether the canonical ontology is incomplete.
• ›Enrichment summary. Average number of structured metadata fields attached to each resolved entity (drug class, ATC code, known targets for compounds; ontology lineage for conditions), so the data-shape change is visible rather than implicit.

Bhattacharyya et al. 2023 (Cureus, doi:10.7759/cureus.39238) examined 115 references across 30 ChatGPT-generated medical papers and found 47 percent fully fabricated, 46 percent authentic but with bibliographic errors, and only 7 percent authentic and accurate. WHBench (Maurya, Saboo & Kumar 2026, arXiv:2604.00024) documents a 35.5 percent fully-correct rate for the top frontier LLM on women's health clinical questions, with systematic gaps in safety, completeness, and the social-determinants criterion. Resolution and enrichment against canonical ontologies addresses the structured-output failure mode that both papers describe and also moves the data Whel stores from free-text strings to canonical identifiers with structured metadata.

Recorded in the methodology revision history at v3.4 (see the methodology changelog ). Listed on the Roadmap under the technical-architecture track as “Ontology-grounded entity resolution (Path A).” The resolution, enrichment, and human-review gate are now applied across the corpus, so the canonical identifiers and structured metadata travel with every signal. The per-pipeline and per-condition audit numbers above are the remaining piece: they populate this block once the resolution-rate disclosure is surfaced.

This layer is both a grounding mechanism and a disclosure layer, and it arrives in stages. The stage live today is relational: a domain-restricted graph of drug, target, and condition relationships built over Open Targets, restricted to Whel's six conditions and the compounds attached to active signals. From it, each signal carries a ‘graph supports’ or ‘graph silent’ tag beside its grade, in the same shape as the MATRIX score row above. ‘Graph supports, via target X’ means the drug acts on a target that Open Targets associates with the condition; ‘graph silent’ means no such shared target is present, which can reflect either a real biological gap or a limit of the source data. Two of the six conditions show this plainly: vulvodynia and PMDD return no disease entry in Open Targets at all, so every signal under them is graph silent by construction, an absence the page shows rather than hides. The planned stage deepens this: a property-graph version built with the BioCypher framework (Lobentanzer et al., Nature Biotechnology 2023), grounded in canonical ontologies, and a feed of the relevant subgraph into the LLM at prompt time so the model relies less on parametric memory alone. Open repurposing graphs and models such as DRKG, PrimeKG, and TxGNN sit in a separate validation track, checked against rather than merged in, because they carry the field's male-default coverage that Whel exists to correct.

The signal-level ‘graph supports / graph silent’ tag is live in the gated view today. The aggregate audit views below (graph size, per-condition coverage, tier cross-tabulation) are computed from the same data and are being surfaced as reporting.

• ›Knowledge graph size and shape. Number of nodes by type (drug, condition, gene, pathway, adverse event) and edges by relationship type (targets, treats, interacts, associated-with, etc.). Reported as a snapshot table with the data-source SHAs each edge type was drawn from, identical in shape to the MATRIX dataset snapshot above.
• ›Per-condition graph coverage. For each of the six conditions, the count of Whel compounds that have at least one graph-supported mechanistic path to that condition, and the count that have none. A condition with low graph coverage is a condition where the Whel grades stand alone without a graph cross-reference, and that fact is made visible.
• ›Signal-level graph support. Each individual signal carries a 'graph supports' or 'graph silent' tag. 'Graph supports' means at least one mechanistic path exists in the KG that connects the compound to the condition through known targets, pathways, or co-occurring annotations. 'Graph silent' is not the same as 'graph contradicts'; it means the open KGs do not contain a relevant edge, which can reflect either a real biological gap or a known limitation of the source data.
• ›Cross-tabulation against Whel tiers. How Whel's four confidence tiers (Strong, Moderate, Emerging, Exploratory) cross with graph support. A graph-supported Strong-tier signal is the strongest combined evidence the platform can present. A graph-silent Strong-tier signal is a signal where the literature replicates but the open knowledge graphs have not yet caught up; that pattern is also informative.

BioCypher is the peer-reviewed, EU-funded biomedical knowledge graph framework introduced in Lobentanzer et al., Nature Biotechnology 2023, and actively maintained. The architectural pattern of layering structured knowledge graphs on top of LLM extraction, rather than replacing the LLM with classical ML, is the direction argued by Zong, Lv, Xue, Zheng, Wan & Zhang 2026 (“Building evidence-based knowledge bases from full-text literature for disease-specific biomedical reasoning,” arXiv:2603.28325, which introduces the EvidenceNet dataset). Every Cure's MATRIX builds on the KGML-xDTD framework (Ma, Zhou, Liu & Koslicki, GigaScience 2023) and demonstrates that knowledge graph plus machine learning systems outperform LLM-only approaches for the separate problem of global drug repurposing prediction, which Whel does not attempt.

Recorded in the methodology revision history at v3.4 (see the methodology changelog ). Listed on the Roadmap under the technical-architecture track as “Knowledge-graph grounding,” now live for the relational Open Targets version with the per-signal tag in the gated view. The BioCypher property graph and the prompt-time scoring feed remain planned, and the open knowledge graphs and models are tracked separately under the Roadmap's validation layer. Whel will not train a custom graph neural network; the platform consumes machine learning (Claude Opus 4.8 for extraction and scoring, MATRIX scores as the existing cross-reference) but does not develop its own ML models.

Every PMID in Whel's database, and every reference in any prose Whel publishes (featured signal walkthroughs, the methods PDF, written drafts), is resolved against NCBI E-utilities; DOIs are resolved against the Crossref REST API. Each lookup returns canonical title, authors, journal, and year, which are compared against the LLM-claimed metadata. References that fail to resolve or whose returned metadata mismatch the LLM's claims are flagged for human review and blocked from publication. This addresses the citation fabrication and citation-misattribution failure modes directly.

For every signal row in the database, a sentence-transformer model (the Sentence-BERT family of models published on Hugging Face) computes the cosine similarity between each sentence in the LLM-generated summary and the source abstract. Sentences that fall below a calibrated similarity threshold are flagged as “not directly supported by the source” and either suppressed or surfaced with that marker on the signal card. The threshold is tuned against a held-out human-validation set rather than picked by intuition. This addresses the summary-drift failure mode documented in the medical LLM literature (Bhattacharyya et al. 2023 in Cureus, doi:10.7759/cureus.39238) applied to Whel's specific extraction task.

Any LLM-generated long-form prose that ships to users (featured walkthroughs, methods PDF text, future Substack drafts written through Whel's tooling) is generated under a hardened prompt that forbids citation generation outside a pre-verified reference list provided to the model, forbids numerical claims (prevalence rates, effect sizes) unless they appear verbatim in the input context, and requires the model to produce, alongside the text, a sentence-by-sentence list of which input sources support each sentence. The list is then checked by Phase 1 before the prose is published.

Path C Phase 1 ships as scripts/verify-citations.py and reads the pre-verified reference list at lib/whel-citations.json. The script resolves every PMID against NCBI E-utilities, every DOI against the Crossref REST API, and every arXiv ID against the arXiv API, then compares returned canonical metadata (title, first-author surname, container title, year) against the claims in the manifest using fuzzy match with calibrated thresholds. Output is written to scripts/audit-output/citation-audit-report.json and mirrored to lib/citation-audit-snapshot.json, which this page reads from. --strict mode exits non-zero on any unresolved or mismatched entry and is wired for pre-publish use.

The expanded manifest on June 7, 2026 (v3.9) added the eight hand-written featured-page references. The verifier caught a real author misattribution among them: a paper attributed to one author group resolved, at the cited PMC link, to a paper by an entirely different group. The featured page and the manifest were both corrected before the disclosure published. The full run log is at scripts/audit-output/citation-audit-report.json.

The much larger surface is the live sources table: every PMID from PubMed, every NCT ID from ClinicalTrials.gov, every Open Targets identifier, and every AEMS / Reddit URL that the LLM extraction pipeline attached to an active signal and rendered on a drug card.

Zero resolved_mismatch entries on the first run: 113 of 113 PubMed PMIDs clean against NCBI E-utilities, 19 of 19 ClinicalTrials.gov NCT IDs clean against the ClinicalTrials.gov API v2, and 38 of 38 canonical Open Targets identifiers clean against the Open Targets GraphQL search. The 10 unresolved are all Open Targets rows storing a synthetic OT-{DRUGNAME} shorthand in the external_id column instead of a canonical CHEMBL identifier; the URL on those rows still points at a real platform.opentargets.org page, so what users see on the drug card is a valid citation. The failure is at the identifier-storage layer rather than the user-visible content layer. Backfill recorded on the roadmap under Backfill canonical Open Targets identifiers on signals using OT-DRUGNAME shorthand; methodology v3.10 has the full finding write-up. Run log at scripts/audit-output/database-sources-audit-report.json.

Sentence-level grounding for free-text sources (PubMed, ClinicalTrials.gov, Reddit). Each LLM-generated finding sentence on sources.key_finding_excerpt will be embedded via Sentence-BERT (all-MiniLM-L6-v2) and compared against canonical source sentences using max cosine similarity. Sentences scoring below 0.40 will be flagged as not directly supported by the source. Tooling shipped as scripts/verify-summary-grounding.py; runs after the export script is re-run to populate the key_finding_excerpt field on the sources snapshot and after pip install sentence-transformers on the host. This block switches to live numbers the moment lib/summary-grounding-audit-snapshot.json is populated.

Field-by-field verification for structured sources (AEMS reaction counts; Open Targets target attributions). AEMS counts will be re-queried against the openFDA drug/event endpoint and compared to the count in the LLM- extracted title. Open Targets target claims will be verified through the OT GraphQL drug(chemblId) query against the linkedTargets list. Tooling shipped as scripts/verify-structured-sources.py; this block switches to live numbers the moment lib/structured-sources-audit-snapshot.json is populated.

• ›Count of references blocked at publish time. Once Phase 3 prompt hardening lands, this disclosure surfaces how many references the LLM proposed that failed the Phase 1 manifest check and were therefore stripped before publish, instead of the audit reporting on the pre-verified manifest only.

Bhattacharyya et al. 2023 (Cureus, doi:10.7759/cureus.39238) examined 115 references across 30 ChatGPT-generated medical papers and found 47 percent fully fabricated, 46 percent authentic but with bibliographic errors, and only 7 percent authentic and accurate; Gravel, D'Amours-Gravel & Osmanlliu 2023 (Mayo Clin Proc Digit Health, doi:10.1016/j.mcpdig.2023.05.004) reported the same pattern across a separate medical question set. WHBench (Maurya, Saboo & Kumar 2026, arXiv:2604.00024) documents the broader pattern of frontier LLMs producing confident structured output with systematic failure modes. Path C's three phases each target a specific failure surface within Whel's pipeline rather than attempting to address the LLM gap in aggregate.

Recorded in the methodology revision history at v3.6 (definition), v3.7 (manual audit prototype), v3.8 (Phase 1 live for the hand-written manifest), v3.9 (Phase 1 tooling for the database-sources audit), v3.10 (Phase 1 first database-sources run), and v3.11 (OT-DRUGNAME backfill closing the v3.10 architectural-debt finding). Full revision history at the methodology changelog . Phase 1 is now a Live register row on the Roadmap as “Citation validation and summary grounding (Path C);” Phase 2 (sentence-level summary grounding via Sentence-BERT) and Phase 3 (prompt hardening that forbids citation generation outside the Phase 1 manifest) remain Planned. The structured fields above carry real audit numbers for Phase 1 and will populate the remaining fields when Phase 2 and Phase 3 ship. Path C is distinct from Path A (ontology-grounded entity resolution) and Path B (knowledge-graph grounding), which are documented in section 01c above. Path A and Path B ground the LLM's inputs; Path C validates the LLM's outputs.