learningcircuit--local-deep-research/docs/release_notes/1.6.12.md

🔒 Security

• RAG collection upload endpoint now validates per-file size (50MB limit) and per-request file count (200 limit), matching the research upload endpoint. Previously only the request-level MAX_CONTENT_LENGTH (10GB) was enforced, allowing a single oversized file or a request with thousands of zero-byte files to reach the per-file processing loop. Pre-flight Content-Length check rejects oversized files before reading bytes into memory.

✨ New Features

• New citation format: source-tagged with global numbering. Reports can now render citations as [arxiv-1], [openai.com-2], [arxiv-3] — the source tag identifies what kind of source each citation is (short URLClassifier tag for known academic sources, cleaned domain for generic web URLs, or the collection name for local RAG/library hits, e.g. [my-papers-4]), while the number stays the original global counter so labels never collide and inline citations match the bibliography order. Opt in via report.citation_format → "Source-tagged with global numbering"; the existing default remains Numbers with hyperlinks [1]. (#4012)
• Add LaTeX math rendering support. Mathematical formulas written with $...$ (inline) and $$...$$ (display) notation are now rendered using KaTeX in research reports.
• Per-PDF library storage cap (research_library.max_pdf_size_mb) default raised from 100 MB to 3 GB so it no longer silently truncates large academic PDFs after the recent upload-validator bump. The setting's UI ceiling is also raised from 500 MB to 10 GB, and the PDFStorageManager / download_service fallback defaults are updated to match. Deployments that want a tighter bound can still lower the setting via the UI.
• Per-file upload cap (FileUploadValidator.MAX_FILE_SIZE) is now configurable and defaults to 3 GB (was 50 MB) so large academic datasets and PDFs fit out of the box. Deployments that want a tighter bound can lower it via the LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB environment variable or the security.upload_max_file_size_mb setting. Memory usage stays bounded by the existing 5 MB spool-to-disk threshold on multipart requests.
• The library RAG indexer now prunes old quarantined FAISS index files (<hash>.faiss.corrupt-* and matching .pkl.corrupt-*) at quarantine time, keeping only the 5 most recent per base path. Prevents the rag_indices/ cache directory from filling up on systems that experience recurring corruption, while preserving recent diagnostic artefacts. Follow-up to #4197 / #4200.

🐛 Bug Fixes

• Fixed UnicodeDecodeError on Windows when loading settings, security config, benchmark results, and Vite manifest files. All text-mode open() calls now use explicit UTF-8 encoding, and JSON config files use utf-8-sig to handle BOM-prefixed files from Windows editors. Fixes #3743. (#3797)
• Fixed embedding-model dropdown showing "No models available" with LM Studio (and other OpenAI-compatible local servers) when an embedding model whose name didn't include the literal embedding token was loaded (e.g. nomic-embed-text-v1.5). The OpenAI and Ollama embedding providers no longer guess from the model name — every model the endpoint reports is shown so the user can pick the one they actually loaded. Ollama still tags models when its /api/show capabilities response is available. Fixes #4195. (#4195)
• Closed a follow-up race condition in the library RAG indexer where two concurrent workers indexing different documents into the same collection could lose each other's embeddings from the FAISS file — last writer's save_local overwrote the earlier writer's chunks (the chunks survived in the DB, but the index file was missing them until a force-reindex rebuilt it). The save path now reloads from disk under the per-(user, index_path) lock before adding, so concurrent writers' chunks are merged instead of overwriting each other. Follow-up to #4197/#4200.
• Fixed a data-loss bug where the library RAG index file (.faiss) was silently deleted when integrity verification failed during concurrent auto-indexing — destroying hundreds of previously-indexed documents in one go. Corrupted index files are now quarantined to <path>.corrupt-<ns> so they remain recoverable, and concurrent indexers are serialised by a per-(user, index_path) lock so the race that produced the checksum mismatch no longer occurs. Also drops the per-document PRAGMA wal_checkpoint(FULL) that contributed to database is locked errors under bulk-download concurrency. Fixes #4197.
• Paginate the /history/logs/<id> endpoint (default 500, clamped to 5000) so loading a long research's history no longer materialises every ResearchLog row server-side or pushes a 50+ MB JSON response that the browser ultimately prunes to 500 entries anyway. Complements the live-socket truncation in PR #4004.
• Three module-level per-user lock dicts (_user_init_locks in database/library_init.py, _user_locks in database/backup/backup_service.py, _user_critical_locks on QueueProcessorV2) previously accumulated one threading.Lock entry per username over the process lifetime with no removal hook on user-close. Bounded by total user count (~900 bytes/user across all three), so not visible on typical self-hosted instances — but long-lived multi-user deployments with user-account churn would see slow memory creep. The three modules now expose pop_user_*_lock(username) helpers and a shared _pop_per_user_locks call in the connection-cleanup module fires them from both the idle-cleanup sweeper and the logout / password-change paths, matching the cleanup already done for scheduler-job registrations and session-password store.
• Tighten the model_dump error-pattern regex introduced in #3926 so the rich proxy/shim hint only fires for the canonical object has no attribute 'model_dump' AttributeError, not for unrelated traces that happen to contain the substring model_dump.