learningcircuit--local-deep-research/docs/release_notes/1.6.0.md

Journal Quality Redesign — release notes (pending)

Staging notes for the journal-quality redesign shipped by #3081. Fold into the next tagged version's release-notes file when cutting the release.

Major Features

Tiered journal quality scoring

The journal-reputation filter now uses a five-tier pipeline (predatory check → OpenAlex snapshot → DOAJ → institution affiliation salvage → optional LLM) instead of calling an LLM per source. A bundled read-only reference database of ~280K academic venues + ~120K institutions powers Tiers 1–3 in 100–300 µs per lookup, eliminating the multi-second per-source latency of the previous LLM-only path.

See docs/journal-quality.md for the full scoring table, data-source licenses, and the Tier-1 predatory-whitelist override.

Journal dashboard

New analytics view at Analytics → Journals lists every source in the reference DB with server-side filtering, pagination, and sortable columns (h-index, quartile, impact factor, quality score, DOAJ status, predatory flag). Works on ~200K rows without loading them all into memory thanks to the shared read-only SQLite reference DB (mode=ro&immutable=1, chmod 0o444 after build).

Quality tags in research output

Each source in the research report now carries a compact quality tag:

[1] Physical Review Letters [Q1 ★★★★★]
[2] Some Niche Journal [Q3 ★★]

Predatory-flagged sources that are not whitelisted are auto-removed before the report is assembled.

BREAKING — journals table columns removed

The per-user encrypted journals table no longer stores the following columns:

issn, issn_list, publisher, openalex_source_id, source_type, h_index, impact_factor, works_count, cited_by_count, is_in_doaj, has_doaj_seal, is_predatory, predatory_source, is_indexed_in_scopus, data_version, sjr_quartile.

These metrics are now served exclusively by the bundled read-only reference DB (journal_quality.db). The per-user journals table is now a Tier-4 LLM cache only and retains: id, name, name_lower, quality, score_source, quality_model, quality_analysis_time.

Impact

• If you have custom SQL tooling that queries the removed columns on the per-user DB, point it at the reference DB via the new accessor (journal_quality/db.py) or the /api/journals dashboard endpoint instead.
• The upgrade itself preserves every row in the journals table — only the dropped columns disappear. quality, name, name_lower, and the LLM-cache columns stay.
• No action is needed for users who do not write their own SQL.

New data downloads on first use

The journal-quality reference DB builds itself on first access from five third-party snapshots (OpenAlex, DOAJ, Stop Predatory Journals, JabRef abbreviations, OpenAlex Institutions — all CC0 or MIT licensed). The build streams several hundred MB from the OpenAlex S3 bulk dump plus smaller snapshots from the other sources, unpacks to ~1 GB of intermediate working set, and typically takes 1–2 minutes on a normal connection. All five sources fetch in parallel via a ThreadPoolExecutor — wall-clock is bound by the slowest source (the OpenAlex sources snapshot at ~30–60 s) rather than the sum. On fresh installs the first research request returns immediately with [preprint — not in journal catalog] / [journal quality data is downloading…] placeholder tags; a second search a minute later picks up the real Q-tier scores. The dashboard's Journals page (/metrics/journals) shows per-source progress bars during the build.

Settings

One new opt-in toggle and four opt-out per-engine toggles:

• search.journal_reputation.enable_llm_scoring (default false, opt-in) — if enabled, Tier 3.6 and Tier 4 use SearXNG + the LLM for unknown journals.
• search.engine.web.arxiv.journal_reputation.enabled (default true, opt-out)
• search.engine.web.openalex.journal_reputation.enabled (default true, opt-out)
• search.engine.web.semantic_scholar.journal_reputation.enabled (default true, opt-out)
• search.engine.web.nasa_ads.journal_reputation.enabled (default true, opt-out) — per-engine toggles for running the reputation filter over each academic-search engine's results; disable individually to skip filtering for a given engine.

Operational notes

• The first user request after upgrade pays a one-time migration cost on the per-user journals table (SQLite batch rebuild). Typical libraries complete in under a second; very large libraries (100k+ journals) may stall 2–5 seconds.
• The first visit to Analytics → Journals before the reference DB has finished building shows the page frame with a per-source download banner rather than a data grid; /api/journals returns HTTP 503 with {"status": "error", "message": "Journal reference database not available."} until the build completes. A research request run in parallel kicks off the build in a background daemon thread, so warming the filter path first is the fastest way to make the dashboard live on a fresh install.
• Windows installs now enforce read-only on the reference DB via SetFileAttributesW, matching the POSIX chmod 0o444 behavior.
• The bulk data download fails fast with a clear error if free disk space falls below 2 GB.
• Transient network failures during the bulk download retry up to three times with exponential backoff (1/2/4 s), respecting Retry-After on HTTP 429.