learningcircuit--local-deep-research/docs/release_notes/1.6.0.md

# Journal Quality Redesign — release notes (pending)

Staging notes for the journal-quality redesign shipped by [#3081](https://github.com/LearningCircuit/local-deep-research/pull/3081). Fold into the next tagged version's release-notes file when cutting the release.

## Major Features

### Tiered journal quality scoring

The journal-reputation filter now uses a five-tier pipeline (predatory check → OpenAlex snapshot → DOAJ → institution affiliation salvage → optional LLM) instead of calling an LLM per source. A bundled read-only reference database of ~280K academic venues + ~120K institutions powers Tiers 1–3 in 100–300 µs per lookup, eliminating the multi-second per-source latency of the previous LLM-only path.

See `docs/journal-quality.md` for the full scoring table, data-source licenses, and the Tier-1 predatory-whitelist override.

### Journal dashboard

New analytics view at **Analytics → Journals** lists every source in the reference DB with server-side filtering, pagination, and sortable columns (h-index, quartile, impact factor, quality score, DOAJ status, predatory flag). Works on ~200K rows without loading them all into memory thanks to the shared read-only SQLite reference DB (`mode=ro&immutable=1`, `chmod 0o444` after build).

### Quality tags in research output

Each source in the research report now carries a compact quality tag:

```
[1] Physical Review Letters [Q1 ★★★★★]
[2] Some Niche Journal [Q3 ★★]
```

Predatory-flagged sources that are not whitelisted are auto-removed before the report is assembled.

## BREAKING — `journals` table columns removed

The per-user encrypted `journals` table no longer stores the following columns:

`issn`, `issn_list`, `publisher`, `openalex_source_id`, `source_type`, `h_index`, `impact_factor`, `works_count`, `cited_by_count`, `is_in_doaj`, `has_doaj_seal`, `is_predatory`, `predatory_source`, `is_indexed_in_scopus`, `data_version`, `sjr_quartile`.

These metrics are now served exclusively by the bundled read-only reference DB (`journal_quality.db`). The per-user `journals` table is now a Tier-4 LLM cache only and retains: `id`, `name`, `name_lower`, `quality`, `score_source`, `quality_model`, `quality_analysis_time`.

### Impact

- If you have custom SQL tooling that queries the removed columns on the per-user DB, point it at the reference DB via the new accessor (`journal_quality/db.py`) or the `/api/journals` dashboard endpoint instead.
- The upgrade itself preserves every row in the `journals` table — only the dropped columns disappear. `quality`, `name`, `name_lower`, and the LLM-cache columns stay.
- No action is needed for users who do not write their own SQL.

## New data downloads on first use

The journal-quality reference DB builds itself on first access from five third-party snapshots (OpenAlex, DOAJ, Stop Predatory Journals, JabRef abbreviations, OpenAlex Institutions — all CC0 or MIT licensed). The build streams several hundred MB from the OpenAlex S3 bulk dump plus smaller snapshots from the other sources, unpacks to ~1 GB of intermediate working set, and typically takes 1–2 minutes on a normal connection. All five sources fetch **in parallel** via a ThreadPoolExecutor — wall-clock is bound by the slowest source (the OpenAlex sources snapshot at ~30–60 s) rather than the sum. On fresh installs the first research request returns immediately with `[preprint — not in journal catalog]` / `[journal quality data is downloading…]` placeholder tags; a second search a minute later picks up the real Q-tier scores. The dashboard's Journals page (`/metrics/journals`) shows per-source progress bars during the build.

## Settings

One new opt-in toggle and four opt-out per-engine toggles:

- `search.journal_reputation.enable_llm_scoring` (default `false`, **opt-in**) — if enabled, Tier 3.6 and Tier 4 use SearXNG + the LLM for unknown journals.
- `search.engine.web.arxiv.journal_reputation.enabled` (default `true`, opt-out)
- `search.engine.web.openalex.journal_reputation.enabled` (default `true`, opt-out)
- `search.engine.web.semantic_scholar.journal_reputation.enabled` (default `true`, opt-out)
- `search.engine.web.nasa_ads.journal_reputation.enabled` (default `true`, opt-out) — per-engine toggles for running the reputation filter over each academic-search engine's results; disable individually to skip filtering for a given engine.

## Operational notes

- The first user request after upgrade pays a one-time migration cost on the per-user `journals` table (SQLite batch rebuild). Typical libraries complete in under a second; very large libraries (100k+ journals) may stall 2–5 seconds.
- The first visit to **Analytics → Journals** before the reference DB has finished building shows the page frame with a per-source download banner rather than a data grid; `/api/journals` returns HTTP 503 with `{"status": "error", "message": "Journal reference database not available."}` until the build completes. A research request run in parallel kicks off the build in a background daemon thread, so warming the filter path first is the fastest way to make the dashboard live on a fresh install.
- Windows installs now enforce read-only on the reference DB via `SetFileAttributesW`, matching the POSIX `chmod 0o444` behavior.
- The bulk data download fails fast with a clear error if free disk space falls below 2 GB.
- Transient network failures during the bulk download retry up to three times with exponential backoff (1/2/4 s), respecting `Retry-After` on HTTP 429.