chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,301 @@
|
||||
# Docling Speed Optimization Plan
|
||||
|
||||
## Progress Tracker
|
||||
|
||||
| Task | Status | Completed | Result |
|
||||
|------|--------|-----------|--------|
|
||||
| Phase 0: Baseline measurement | ✅ completed | 2026-01-03 | 2.283s/doc |
|
||||
| Phase 0: FastAPI experiment | ✅ completed | 2026-01-03 | 0.685s/doc (PASS < 0.8s) |
|
||||
| Phase 0: subprocess experiment | ✅ completed | 2026-01-03 | 0.661s/doc (PASS < 1.0s) |
|
||||
| Phase 0: Results comparison | ✅ completed | 2026-01-03 | 3.3x-3.5x speedup |
|
||||
| Task 1.1: docling_subprocess_worker.py | ⏭️ skipped | - | FastAPI only |
|
||||
| Task 1.2: hybrid_server.py | ✅ completed | 2026-01-03 | opendataloader-pdf-hybrid |
|
||||
| Task 2.1: DoclingSubprocessClient.java | ⏭️ skipped | - | FastAPI only |
|
||||
| Task 2.2: DoclingFastServerClient.java | ✅ completed | 2026-01-03 | - |
|
||||
| Task 2.3: HybridClientFactory modification | ✅ completed | 2026-01-03 | docling-fast only |
|
||||
| Task 3.1: pdf_parser modules | ✅ completed | 2026-01-03 | docling-fast only |
|
||||
| Task 3.2: engine_registry.py | ✅ completed | 2026-01-03 | - |
|
||||
| Task 3.3: run.py CLI options | ✅ completed | 2026-01-03 | - |
|
||||
| Task 4.1: Full benchmark | ✅ completed | 2026-01-03 | See experiments/speed/ |
|
||||
| Task 4.2: Results documentation | ✅ completed | 2026-01-03 | speed-experiment-2026-01-03.md |
|
||||
|
||||
**Status Legend:**
|
||||
- ⬜ `not_started` - Not yet begun
|
||||
- 🔄 `in_progress` - Currently working
|
||||
- ✅ `completed` - Done and verified
|
||||
- ⏭️ `skipped` - Excluded from plan
|
||||
- ⏸️ `blocked` - Waiting on dependency
|
||||
- ❌ `failed` - Did not meet criteria
|
||||
- 🚫 `discarded` - Plan abandoned
|
||||
|
||||
---
|
||||
|
||||
## 1. Background
|
||||
|
||||
### Current Problem
|
||||
- **DoclingClient** (docling-serve HTTP API): ~2 seconds per page
|
||||
- **docling SDK direct call**: ~0.5 seconds per document (user-reported)
|
||||
- HTTP overhead negates the speed benefits of hybrid mode
|
||||
|
||||
### Goal
|
||||
Implement alternative approaches to efficiently call the docling SDK, then compare benchmark speeds
|
||||
|
||||
---
|
||||
|
||||
## 2. Experiment Phase (Phase 0)
|
||||
|
||||
### Purpose
|
||||
Validate the speed improvement hypothesis before full implementation
|
||||
|
||||
### Experiment Targets
|
||||
| Approach | Description |
|
||||
|----------|-------------|
|
||||
| baseline | Current docling-serve (reference) |
|
||||
| fastapi | Optimized FastAPI server |
|
||||
| subprocess | Direct Python subprocess call |
|
||||
|
||||
### Success Criteria
|
||||
| Approach | Threshold | Condition |
|
||||
|----------|-----------|-----------|
|
||||
| fastapi | **< 0.8 sec/doc** (average) | Based on 200 documents |
|
||||
| subprocess | **< 1.0 sec/doc** (average) | Based on 200 documents |
|
||||
|
||||
### Failure Conditions
|
||||
- If fastapi approach exceeds 0.8 sec/doc: **Discard entire plan**
|
||||
- If only subprocess fails: Exclude that approach only
|
||||
|
||||
### Experiment Environment
|
||||
- Benchmark PDFs: `tests/benchmark/pdfs/` (200 files)
|
||||
- Settings: `do_ocr=true`, `do_table_structure=true`
|
||||
- Measurement: `total_time / document_count`
|
||||
|
||||
### Experiment Scripts
|
||||
```
|
||||
scripts/experiments/
|
||||
├── docling_baseline_bench.py # docling-serve speed measurement
|
||||
├── docling_fastapi_bench.py # FastAPI server + client test
|
||||
├── docling_subprocess_bench.py # subprocess approach test
|
||||
└── docling_speed_report.py # Results comparison report
|
||||
```
|
||||
|
||||
### Experiment Execution
|
||||
```bash
|
||||
# 1. baseline (requires docling-serve running)
|
||||
python scripts/experiments/docling_baseline_bench.py
|
||||
|
||||
# 2. fastapi (server auto-starts)
|
||||
python scripts/experiments/docling_fastapi_bench.py
|
||||
|
||||
# 3. subprocess
|
||||
python scripts/experiments/docling_subprocess_bench.py
|
||||
|
||||
# 4. compare results
|
||||
python scripts/experiments/docling_speed_report.py
|
||||
```
|
||||
|
||||
### Results Recording
|
||||
```
|
||||
docs/hybrid/experiments/
|
||||
└── speed-experiment-YYYY-MM-DD.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation Tasks (After Phase 0 Success)
|
||||
|
||||
### Task 1: Python Scripts
|
||||
|
||||
#### Task 1.1: docling_subprocess_worker.py
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `scripts/docling_subprocess_worker.py` |
|
||||
| Prerequisites | docling package installed |
|
||||
| Description | stdin JSON → stdout JSON conversion |
|
||||
| Success Criteria | Single PDF conversion succeeds, JSON output parseable |
|
||||
| Test | `echo '{"pdf_path":"/path/to.pdf"}' \| python scripts/docling_subprocess_worker.py` |
|
||||
|
||||
#### Task 1.2: hybrid_server.py
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `python/opendataloader-pdf/src/opendataloader_pdf/hybrid_server.py` |
|
||||
| Prerequisites | `pip install opendataloader-pdf[hybrid]` |
|
||||
| Description | POST /convert endpoint, DocumentConverter singleton |
|
||||
| Success Criteria | curl PDF upload returns JSON response |
|
||||
| Test | `opendataloader-pdf-hybrid &` then `curl -F "file=@test.pdf" http://localhost:5002/v1/convert/file` |
|
||||
|
||||
### Task 2: Java Client Implementation
|
||||
|
||||
#### Task 2.1: DoclingSubprocessClient.java
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `java/.../hybrid/DoclingSubprocessClient.java` |
|
||||
| Prerequisites | Task 1.1 complete |
|
||||
| Description | ProcessBuilder executes Python, stdin/stdout JSON |
|
||||
| Success Criteria | Implements HybridClient interface, single PDF conversion succeeds |
|
||||
| Test | `DoclingSubprocessClientTest.java` unit tests pass |
|
||||
|
||||
#### Task 2.2: DoclingFastServerClient.java
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `java/.../hybrid/DoclingFastServerClient.java` |
|
||||
| Prerequisites | Task 1.2 complete |
|
||||
| Description | OkHttp calls FastAPI server |
|
||||
| Success Criteria | Implements HybridClient interface, single PDF conversion succeeds |
|
||||
| Test | `DoclingFastServerClientTest.java` unit tests pass |
|
||||
|
||||
#### Task 2.3: HybridClientFactory Modification
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `java/.../hybrid/HybridClientFactory.java` |
|
||||
| Prerequisites | Task 2.1, 2.2 complete |
|
||||
| Description | Register `docling-subprocess`, `docling-fast` backends |
|
||||
| Success Criteria | `HybridClientFactory.getOrCreate("docling-fast", config)` works |
|
||||
| Test | Extend `HybridClientFactoryTest.java` |
|
||||
|
||||
### Task 3: Benchmark Integration
|
||||
|
||||
#### Task 3.1: Add pdf_parser Modules
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| Files | `tests/benchmark/src/pdf_parser_opendataloader_hybrid_subprocess.py` |
|
||||
| | `tests/benchmark/src/pdf_parser_opendataloader_hybrid_fast.py` |
|
||||
| Prerequisites | Task 2.3 complete, JAR built |
|
||||
| Success Criteria | Benchmark runs with `--hybrid docling-subprocess` option |
|
||||
|
||||
#### Task 3.2: Modify engine_registry.py
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `tests/benchmark/src/engine_registry.py` |
|
||||
| Description | Register new engines |
|
||||
| Success Criteria | New engines queryable from ENGINE_DISPATCH |
|
||||
|
||||
#### Task 3.3: Add run.py CLI Options
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `tests/benchmark/run.py` |
|
||||
| Description | Extend `--hybrid` choices |
|
||||
| Success Criteria | `./scripts/bench.sh --hybrid docling-fast` runs |
|
||||
|
||||
### Task 4: Final Validation
|
||||
|
||||
#### Task 4.1: Full Benchmark Execution
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| Prerequisites | Task 3 complete |
|
||||
| Execution | Benchmark 200 documents with all 3 approaches |
|
||||
| Success Criteria | elapsed_per_doc comparison shows meaningful improvement |
|
||||
|
||||
#### Task 4.2: Results Documentation
|
||||
| Item | Details |
|
||||
|------|---------|
|
||||
| File | `docs/hybrid/docling-speed-optimization-results.md` |
|
||||
| Content | Speed comparison table, recommended approach, usage guide |
|
||||
|
||||
---
|
||||
|
||||
## 4. Task Workflow
|
||||
|
||||
```
|
||||
Phase 0: Experiment
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
┌─────────────────┐
|
||||
│ baseline measure │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ fastapi test │ │ subprocess test │
|
||||
└────────┬────────┘ └────────┬────────┘
|
||||
│ │
|
||||
└──────────────┬──────────────┘
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ compare results │
|
||||
│ < 0.8 sec/doc? │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
▼ ▼
|
||||
[SUCCESS] [FAILURE]
|
||||
Proceed to Discard plan
|
||||
Phase 1
|
||||
|
||||
Phase 1~4: Implementation (parallelizable)
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Task 1.1 ─────────────────► Task 2.1 ─┐
|
||||
(subprocess worker) (Java client) │
|
||||
│
|
||||
Task 1.2 ─────────────────► Task 2.2 ─┼─► Task 2.3 ─► Task 3 ─► Task 4
|
||||
(fastapi server) (Java client) │ (Factory) (Bench) (Validate)
|
||||
│
|
||||
◄──── parallelizable ────► │
|
||||
```
|
||||
|
||||
### Parallelizable Tasks
|
||||
| Group | Tasks | Notes |
|
||||
|-------|-------|-------|
|
||||
| Phase 0 | fastapi test, subprocess test | After baseline measurement |
|
||||
| Phase 1 | Task 1.1, Task 1.2 | Independent |
|
||||
| Phase 2 | Task 2.1, Task 2.2 | Depend on Task 1.1, 1.2 respectively |
|
||||
| Phase 3 | Task 3.1, 3.2, 3.3 | After Task 2.3 complete |
|
||||
|
||||
### Dependencies
|
||||
```
|
||||
Task 1.1 → Task 2.1 ─┐
|
||||
├─► Task 2.3 → Task 3.* → Task 4.*
|
||||
Task 1.2 → Task 2.2 ─┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. File List
|
||||
|
||||
### New Files
|
||||
| File | Phase | Description |
|
||||
|------|-------|-------------|
|
||||
| `scripts/experiments/docling_baseline_bench.py` | 0 | Baseline measurement |
|
||||
| `scripts/experiments/docling_fastapi_bench.py` | 0 | FastAPI experiment |
|
||||
| `scripts/experiments/docling_subprocess_bench.py` | 0 | Subprocess experiment |
|
||||
| `scripts/experiments/docling_speed_report.py` | 0 | Results report |
|
||||
| `scripts/docling_subprocess_worker.py` | 1 | Subprocess worker (skipped) |
|
||||
| `python/.../hybrid_server.py` | 1 | FastAPI server (opendataloader-pdf-hybrid) |
|
||||
| `java/.../hybrid/DoclingSubprocessClient.java` | 2 | Java client |
|
||||
| `java/.../hybrid/DoclingFastServerClient.java` | 2 | Java client |
|
||||
| `tests/.../pdf_parser_opendataloader_hybrid_subprocess.py` | 3 | Benchmark parser |
|
||||
| `tests/.../pdf_parser_opendataloader_hybrid_fast.py` | 3 | Benchmark parser |
|
||||
|
||||
### Modified Files
|
||||
| File | Phase | Changes |
|
||||
|------|-------|---------|
|
||||
| `java/.../hybrid/HybridClientFactory.java` | 2 | Register new backends |
|
||||
| `tests/benchmark/src/engine_registry.py` | 3 | Register engines |
|
||||
| `tests/benchmark/run.py` | 3 | CLI options |
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks and Mitigations
|
||||
|
||||
| Risk | Probability | Mitigation |
|
||||
|------|-------------|------------|
|
||||
| FastAPI speed below threshold | Medium | Discard plan, explore other approaches |
|
||||
| subprocess overhead | Medium | Consider process pooling |
|
||||
| docling SDK version compatibility | Low | Pin version, test |
|
||||
| Memory exhaustion | Low | Adjust batch size |
|
||||
|
||||
---
|
||||
|
||||
## 7. Checklist
|
||||
|
||||
### Phase 0 Completion Criteria
|
||||
- [ ] Baseline speed measurement complete
|
||||
- [ ] FastAPI experiment: < 0.8 sec/doc
|
||||
- [ ] subprocess experiment: < 1.0 sec/doc
|
||||
- [ ] Experiment results documented
|
||||
|
||||
### Overall Completion Criteria
|
||||
- [ ] All Tasks complete
|
||||
- [ ] Benchmark runs successfully with all 3 approaches
|
||||
- [ ] Speed improvement confirmed (vs baseline)
|
||||
- [ ] Results documented
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"conclusion": "Optimized ranges (consecutive page merging) is always the best strategy",
|
||||
"recommendation": "Merge consecutive target pages into ranges before calling convert()",
|
||||
"results": {
|
||||
"25% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 4.223,
|
||||
"chunks": 2
|
||||
},
|
||||
"50% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 6.052,
|
||||
"chunks": 3
|
||||
},
|
||||
"75% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 9.347,
|
||||
"chunks": 5
|
||||
},
|
||||
"100% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 10.277,
|
||||
"chunks": 1
|
||||
}
|
||||
},
|
||||
"key_findings": [
|
||||
"Fixed chunk sizes always have overhead due to processing unnecessary pages",
|
||||
"Single page chunks (chunk_1) have 7-33% overhead from extra API calls",
|
||||
"Larger fixed chunks (chunk_2, 3, 5) have 17-88% overhead from processing unneeded pages",
|
||||
"Optimized ranges minimize both API calls and unnecessary page processing"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,816 @@
|
||||
{
|
||||
"metadata": {
|
||||
"pdf_file": "1901.03003.pdf",
|
||||
"total_pages": 15,
|
||||
"warmup_runs": 1,
|
||||
"measure_runs": 3,
|
||||
"chunk_sizes": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
5
|
||||
],
|
||||
"random_seed": 42,
|
||||
"timestamp": "2026-01-02T16:04:13.655948"
|
||||
},
|
||||
"scenarios": [
|
||||
{
|
||||
"scenario": "25% pages",
|
||||
"total_pages": 15,
|
||||
"target_pages": [
|
||||
1,
|
||||
2,
|
||||
11
|
||||
],
|
||||
"target_page_count": 3,
|
||||
"percentage": 20.0,
|
||||
"results": [
|
||||
{
|
||||
"method": "optimized_ranges",
|
||||
"ranges": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
11,
|
||||
11
|
||||
]
|
||||
],
|
||||
"num_chunks": 2,
|
||||
"avg_time": 4.223,
|
||||
"std_time": 0.042,
|
||||
"times": [
|
||||
4.203,
|
||||
4.282,
|
||||
4.184
|
||||
],
|
||||
"overhead_pct": 0.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_1",
|
||||
"chunk_size": 1,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
2,
|
||||
2
|
||||
],
|
||||
[
|
||||
11,
|
||||
11
|
||||
]
|
||||
],
|
||||
"num_chunks": 3,
|
||||
"avg_time": 4.517,
|
||||
"std_time": 0.082,
|
||||
"times": [
|
||||
4.444,
|
||||
4.475,
|
||||
4.631
|
||||
],
|
||||
"overhead_pct": 7.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_2",
|
||||
"chunk_size": 2,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
11,
|
||||
12
|
||||
]
|
||||
],
|
||||
"num_chunks": 2,
|
||||
"avg_time": 5.257,
|
||||
"std_time": 0.177,
|
||||
"times": [
|
||||
5.502,
|
||||
5.088,
|
||||
5.182
|
||||
],
|
||||
"overhead_pct": 24.5
|
||||
},
|
||||
{
|
||||
"method": "chunk_3",
|
||||
"chunk_size": 3,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
10,
|
||||
12
|
||||
]
|
||||
],
|
||||
"num_chunks": 2,
|
||||
"avg_time": 6.502,
|
||||
"std_time": 0.137,
|
||||
"times": [
|
||||
6.309,
|
||||
6.612,
|
||||
6.584
|
||||
],
|
||||
"overhead_pct": 54.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_5",
|
||||
"chunk_size": 5,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
11,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 2,
|
||||
"avg_time": 7.122,
|
||||
"std_time": 0.025,
|
||||
"times": [
|
||||
7.13,
|
||||
7.148,
|
||||
7.089
|
||||
],
|
||||
"overhead_pct": 68.7
|
||||
}
|
||||
],
|
||||
"best_method": "Optimized ranges",
|
||||
"best_time": 4.223
|
||||
},
|
||||
{
|
||||
"scenario": "50% pages",
|
||||
"total_pages": 15,
|
||||
"target_pages": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
9,
|
||||
12,
|
||||
13
|
||||
],
|
||||
"target_page_count": 7,
|
||||
"percentage": 46.7,
|
||||
"results": [
|
||||
{
|
||||
"method": "optimized_ranges",
|
||||
"ranges": [
|
||||
[
|
||||
2,
|
||||
5
|
||||
],
|
||||
[
|
||||
9,
|
||||
9
|
||||
],
|
||||
[
|
||||
12,
|
||||
13
|
||||
]
|
||||
],
|
||||
"num_chunks": 3,
|
||||
"avg_time": 6.052,
|
||||
"std_time": 0.128,
|
||||
"times": [
|
||||
5.899,
|
||||
6.212,
|
||||
6.045
|
||||
],
|
||||
"overhead_pct": 0.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_1",
|
||||
"chunk_size": 1,
|
||||
"chunks": [
|
||||
[
|
||||
2,
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
5
|
||||
],
|
||||
[
|
||||
9,
|
||||
9
|
||||
],
|
||||
[
|
||||
12,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
13
|
||||
]
|
||||
],
|
||||
"num_chunks": 7,
|
||||
"avg_time": 6.548,
|
||||
"std_time": 0.128,
|
||||
"times": [
|
||||
6.714,
|
||||
6.528,
|
||||
6.403
|
||||
],
|
||||
"overhead_pct": 8.2
|
||||
},
|
||||
{
|
||||
"method": "chunk_2",
|
||||
"chunk_size": 2,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
6
|
||||
],
|
||||
[
|
||||
9,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
14
|
||||
]
|
||||
],
|
||||
"num_chunks": 6,
|
||||
"avg_time": 10.657,
|
||||
"std_time": 0.062,
|
||||
"times": [
|
||||
10.731,
|
||||
10.659,
|
||||
10.58
|
||||
],
|
||||
"overhead_pct": 76.1
|
||||
},
|
||||
{
|
||||
"method": "chunk_3",
|
||||
"chunk_size": 3,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
9
|
||||
],
|
||||
[
|
||||
10,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 5,
|
||||
"avg_time": 11.412,
|
||||
"std_time": 0.071,
|
||||
"times": [
|
||||
11.442,
|
||||
11.479,
|
||||
11.313
|
||||
],
|
||||
"overhead_pct": 88.6
|
||||
},
|
||||
{
|
||||
"method": "chunk_5",
|
||||
"chunk_size": 5,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
6,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 3,
|
||||
"avg_time": 10.899,
|
||||
"std_time": 0.016,
|
||||
"times": [
|
||||
10.922,
|
||||
10.889,
|
||||
10.886
|
||||
],
|
||||
"overhead_pct": 80.1
|
||||
}
|
||||
],
|
||||
"best_method": "Optimized ranges",
|
||||
"best_time": 6.052
|
||||
},
|
||||
{
|
||||
"scenario": "75% pages",
|
||||
"total_pages": 15,
|
||||
"target_pages": [
|
||||
1,
|
||||
2,
|
||||
4,
|
||||
5,
|
||||
7,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
15
|
||||
],
|
||||
"target_page_count": 11,
|
||||
"percentage": 73.3,
|
||||
"results": [
|
||||
{
|
||||
"method": "optimized_ranges",
|
||||
"ranges": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
4,
|
||||
5
|
||||
],
|
||||
[
|
||||
7,
|
||||
7
|
||||
],
|
||||
[
|
||||
9,
|
||||
13
|
||||
],
|
||||
[
|
||||
15,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 5,
|
||||
"avg_time": 9.347,
|
||||
"std_time": 0.112,
|
||||
"times": [
|
||||
9.19,
|
||||
9.412,
|
||||
9.44
|
||||
],
|
||||
"overhead_pct": 0.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_1",
|
||||
"chunk_size": 1,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
2,
|
||||
2
|
||||
],
|
||||
[
|
||||
4,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
5
|
||||
],
|
||||
[
|
||||
7,
|
||||
7
|
||||
],
|
||||
[
|
||||
9,
|
||||
9
|
||||
],
|
||||
[
|
||||
10,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
11
|
||||
],
|
||||
[
|
||||
12,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
13
|
||||
],
|
||||
[
|
||||
15,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 11,
|
||||
"avg_time": 11.15,
|
||||
"std_time": 0.082,
|
||||
"times": [
|
||||
11.043,
|
||||
11.242,
|
||||
11.165
|
||||
],
|
||||
"overhead_pct": 19.3
|
||||
},
|
||||
{
|
||||
"method": "chunk_2",
|
||||
"chunk_size": 2,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
8
|
||||
],
|
||||
[
|
||||
9,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
14
|
||||
],
|
||||
[
|
||||
15,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 8,
|
||||
"avg_time": 12.039,
|
||||
"std_time": 0.112,
|
||||
"times": [
|
||||
12.141,
|
||||
12.092,
|
||||
11.883
|
||||
],
|
||||
"overhead_pct": 28.8
|
||||
},
|
||||
{
|
||||
"method": "chunk_3",
|
||||
"chunk_size": 3,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
9
|
||||
],
|
||||
[
|
||||
10,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 5,
|
||||
"avg_time": 12.503,
|
||||
"std_time": 0.374,
|
||||
"times": [
|
||||
12.279,
|
||||
13.029,
|
||||
12.2
|
||||
],
|
||||
"overhead_pct": 33.8
|
||||
},
|
||||
{
|
||||
"method": "chunk_5",
|
||||
"chunk_size": 5,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
6,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 3,
|
||||
"avg_time": 11.288,
|
||||
"std_time": 0.39,
|
||||
"times": [
|
||||
11.06,
|
||||
11.838,
|
||||
10.967
|
||||
],
|
||||
"overhead_pct": 20.8
|
||||
}
|
||||
],
|
||||
"best_method": "Optimized ranges",
|
||||
"best_time": 9.347
|
||||
},
|
||||
{
|
||||
"scenario": "100% pages",
|
||||
"total_pages": 15,
|
||||
"target_pages": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15
|
||||
],
|
||||
"target_page_count": 15,
|
||||
"percentage": 100.0,
|
||||
"results": [
|
||||
{
|
||||
"method": "optimized_ranges",
|
||||
"ranges": [
|
||||
[
|
||||
1,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 1,
|
||||
"avg_time": 10.277,
|
||||
"std_time": 0.263,
|
||||
"times": [
|
||||
10.074,
|
||||
10.648,
|
||||
10.11
|
||||
],
|
||||
"overhead_pct": 0.0
|
||||
},
|
||||
{
|
||||
"method": "chunk_1",
|
||||
"chunk_size": 1,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
1
|
||||
],
|
||||
[
|
||||
2,
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
5
|
||||
],
|
||||
[
|
||||
6,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
7
|
||||
],
|
||||
[
|
||||
8,
|
||||
8
|
||||
],
|
||||
[
|
||||
9,
|
||||
9
|
||||
],
|
||||
[
|
||||
10,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
11
|
||||
],
|
||||
[
|
||||
12,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
13
|
||||
],
|
||||
[
|
||||
14,
|
||||
14
|
||||
],
|
||||
[
|
||||
15,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 15,
|
||||
"avg_time": 13.625,
|
||||
"std_time": 0.154,
|
||||
"times": [
|
||||
13.509,
|
||||
13.524,
|
||||
13.843
|
||||
],
|
||||
"overhead_pct": 32.6
|
||||
},
|
||||
{
|
||||
"method": "chunk_2",
|
||||
"chunk_size": 2,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
2
|
||||
],
|
||||
[
|
||||
3,
|
||||
4
|
||||
],
|
||||
[
|
||||
5,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
8
|
||||
],
|
||||
[
|
||||
9,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
14
|
||||
],
|
||||
[
|
||||
15,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 8,
|
||||
"avg_time": 12.48,
|
||||
"std_time": 0.039,
|
||||
"times": [
|
||||
12.531,
|
||||
12.437,
|
||||
12.472
|
||||
],
|
||||
"overhead_pct": 21.4
|
||||
},
|
||||
{
|
||||
"method": "chunk_3",
|
||||
"chunk_size": 3,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
3
|
||||
],
|
||||
[
|
||||
4,
|
||||
6
|
||||
],
|
||||
[
|
||||
7,
|
||||
9
|
||||
],
|
||||
[
|
||||
10,
|
||||
12
|
||||
],
|
||||
[
|
||||
13,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 5,
|
||||
"avg_time": 12.104,
|
||||
"std_time": 0.214,
|
||||
"times": [
|
||||
12.301,
|
||||
12.204,
|
||||
11.806
|
||||
],
|
||||
"overhead_pct": 17.8
|
||||
},
|
||||
{
|
||||
"method": "chunk_5",
|
||||
"chunk_size": 5,
|
||||
"chunks": [
|
||||
[
|
||||
1,
|
||||
5
|
||||
],
|
||||
[
|
||||
6,
|
||||
10
|
||||
],
|
||||
[
|
||||
11,
|
||||
15
|
||||
]
|
||||
],
|
||||
"num_chunks": 3,
|
||||
"avg_time": 12.277,
|
||||
"std_time": 0.334,
|
||||
"times": [
|
||||
12.108,
|
||||
12.742,
|
||||
11.98
|
||||
],
|
||||
"overhead_pct": 19.5
|
||||
}
|
||||
],
|
||||
"best_method": "Optimized ranges",
|
||||
"best_time": 10.277
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"25% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 4.223,
|
||||
"best_chunks": 2
|
||||
},
|
||||
"50% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 6.052,
|
||||
"best_chunks": 3
|
||||
},
|
||||
"75% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 9.347,
|
||||
"best_chunks": 5
|
||||
},
|
||||
"100% pages": {
|
||||
"best_method": "optimized_ranges",
|
||||
"best_time": 10.277,
|
||||
"best_chunks": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Docling Page Range Benchmark
|
||||
|
||||
페이지 범위별 변환 성능 비교:
|
||||
- 25%, 50%, 75%, 100% 페이지 시나리오
|
||||
- 각 시나리오별 최적 청크 크기 탐색
|
||||
|
||||
워밍업 후 여러 번 실행하여 평균 측정
|
||||
결과는 JSON으로 저장
|
||||
"""
|
||||
|
||||
import json
|
||||
import time
|
||||
import random
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass, asdict
|
||||
from datetime import datetime
|
||||
|
||||
from docling.document_converter import DocumentConverter, PdfFormatOption
|
||||
from docling.datamodel.base_models import InputFormat
|
||||
from docling.datamodel.pipeline_options import PdfPipelineOptions
|
||||
|
||||
|
||||
WARMUP_RUNS = 1
|
||||
MEASURE_RUNS = 3
|
||||
RANDOM_SEED = 42
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkResult:
|
||||
name: str
|
||||
avg_time: float
|
||||
std_time: float
|
||||
times: list[float]
|
||||
chunk_size: int
|
||||
num_chunks: int
|
||||
|
||||
|
||||
def get_project_root() -> Path:
|
||||
"""프로젝트 루트 디렉토리 반환"""
|
||||
return Path(__file__).parent.parent.parent
|
||||
|
||||
|
||||
def create_converter() -> DocumentConverter:
|
||||
"""DocumentConverter 인스턴스 생성"""
|
||||
pipeline_options = PdfPipelineOptions()
|
||||
return DocumentConverter(
|
||||
format_options={
|
||||
InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def convert_with_page_range(
|
||||
converter: DocumentConverter, pdf_path: Path, start: int, end: int
|
||||
) -> float:
|
||||
"""지정된 페이지 범위로 변환하고 소요 시간 반환"""
|
||||
start_time = time.perf_counter()
|
||||
converter.convert(pdf_path, page_range=(start, end))
|
||||
return time.perf_counter() - start_time
|
||||
|
||||
|
||||
def pages_to_ranges(pages: list[int]) -> list[tuple[int, int]]:
|
||||
"""페이지 리스트를 연속 범위로 변환"""
|
||||
if not pages:
|
||||
return []
|
||||
|
||||
pages = sorted(pages)
|
||||
ranges = []
|
||||
start = pages[0]
|
||||
end = pages[0]
|
||||
|
||||
for p in pages[1:]:
|
||||
if p == end + 1:
|
||||
end = p
|
||||
else:
|
||||
ranges.append((start, end))
|
||||
start = p
|
||||
end = p
|
||||
ranges.append((start, end))
|
||||
return ranges
|
||||
|
||||
|
||||
def run_benchmark_for_ranges(
|
||||
pdf_path: Path,
|
||||
ranges: list[tuple[int, int]],
|
||||
name: str,
|
||||
) -> BenchmarkResult:
|
||||
"""주어진 범위들에 대해 벤치마크 실행"""
|
||||
|
||||
def run_once():
|
||||
converter = create_converter()
|
||||
total_time = 0.0
|
||||
for start, end in ranges:
|
||||
total_time += convert_with_page_range(converter, pdf_path, start, end)
|
||||
return total_time
|
||||
|
||||
# 워밍업
|
||||
for _ in range(WARMUP_RUNS):
|
||||
run_once()
|
||||
|
||||
# 측정
|
||||
times = []
|
||||
for _ in range(MEASURE_RUNS):
|
||||
times.append(run_once())
|
||||
|
||||
avg_time = sum(times) / len(times)
|
||||
std_time = (sum((t - avg_time) ** 2 for t in times) / len(times)) ** 0.5
|
||||
|
||||
return BenchmarkResult(
|
||||
name=name,
|
||||
avg_time=avg_time,
|
||||
std_time=std_time,
|
||||
times=times,
|
||||
chunk_size=0,
|
||||
num_chunks=len(ranges),
|
||||
)
|
||||
|
||||
|
||||
def get_chunks_for_pages(
|
||||
target_pages: list[int], chunk_size: int, total_pages: int
|
||||
) -> list[tuple[int, int]]:
|
||||
"""타겟 페이지들을 청크 크기로 그룹화"""
|
||||
chunks = []
|
||||
for page in target_pages:
|
||||
chunk_start = ((page - 1) // chunk_size) * chunk_size + 1
|
||||
chunk_end = min(chunk_start + chunk_size - 1, total_pages)
|
||||
if (chunk_start, chunk_end) not in chunks:
|
||||
chunks.append((chunk_start, chunk_end))
|
||||
return chunks
|
||||
|
||||
|
||||
def run_scenario_benchmark(
|
||||
pdf_path: Path,
|
||||
total_pages: int,
|
||||
target_pages: list[int],
|
||||
chunk_sizes: list[int],
|
||||
scenario_name: str,
|
||||
) -> dict:
|
||||
"""단일 시나리오 벤치마크 실행"""
|
||||
print(f"\n{'='*60}")
|
||||
print(f"Scenario: {scenario_name}")
|
||||
print(f"{'='*60}")
|
||||
print(f"Target pages ({len(target_pages)}): {target_pages}")
|
||||
print()
|
||||
|
||||
results = []
|
||||
scenario_data = {
|
||||
"scenario": scenario_name,
|
||||
"total_pages": total_pages,
|
||||
"target_pages": target_pages,
|
||||
"target_page_count": len(target_pages),
|
||||
"percentage": round(len(target_pages) / total_pages * 100, 1),
|
||||
"results": [],
|
||||
}
|
||||
|
||||
# 1. 연속 범위 최적화
|
||||
optimized_ranges = pages_to_ranges(target_pages)
|
||||
print(f"[1] Optimized ranges: {optimized_ranges} ({len(optimized_ranges)} ranges)")
|
||||
|
||||
opt_result = run_benchmark_for_ranges(pdf_path, optimized_ranges, "Optimized ranges")
|
||||
results.append(opt_result)
|
||||
print(f" Avg: {opt_result.avg_time:.2f}s (±{opt_result.std_time:.2f}s)")
|
||||
|
||||
scenario_data["results"].append({
|
||||
"method": "optimized_ranges",
|
||||
"ranges": optimized_ranges,
|
||||
"num_chunks": len(optimized_ranges),
|
||||
"avg_time": round(opt_result.avg_time, 3),
|
||||
"std_time": round(opt_result.std_time, 3),
|
||||
"times": [round(t, 3) for t in opt_result.times],
|
||||
"overhead_pct": 0.0,
|
||||
})
|
||||
|
||||
# 2. 각 청크 크기별 테스트
|
||||
for chunk_size in chunk_sizes:
|
||||
chunks = get_chunks_for_pages(target_pages, chunk_size, total_pages)
|
||||
print(f"[{len(results) + 1}] {chunk_size} page(s)/chunk ({len(chunks)} chunks)")
|
||||
|
||||
result = run_benchmark_for_ranges(pdf_path, chunks, f"{chunk_size} page(s)/chunk")
|
||||
result.chunk_size = chunk_size
|
||||
results.append(result)
|
||||
|
||||
overhead_pct = ((result.avg_time - opt_result.avg_time) / opt_result.avg_time) * 100
|
||||
print(f" Avg: {result.avg_time:.2f}s (±{result.std_time:.2f}s) [{overhead_pct:+.1f}%]")
|
||||
|
||||
scenario_data["results"].append({
|
||||
"method": f"chunk_{chunk_size}",
|
||||
"chunk_size": chunk_size,
|
||||
"chunks": chunks,
|
||||
"num_chunks": len(chunks),
|
||||
"avg_time": round(result.avg_time, 3),
|
||||
"std_time": round(result.std_time, 3),
|
||||
"times": [round(t, 3) for t in result.times],
|
||||
"overhead_pct": round(overhead_pct, 1),
|
||||
})
|
||||
|
||||
# Best 찾기
|
||||
best_result = min(results, key=lambda r: r.avg_time)
|
||||
scenario_data["best_method"] = best_result.name
|
||||
scenario_data["best_time"] = round(best_result.avg_time, 3)
|
||||
|
||||
print()
|
||||
print(f" >> Best: {best_result.name} ({best_result.avg_time:.2f}s)")
|
||||
|
||||
return scenario_data
|
||||
|
||||
|
||||
def main():
|
||||
project_root = get_project_root()
|
||||
pdf_path = project_root / "samples" / "pdf" / "1901.03003.pdf"
|
||||
|
||||
if not pdf_path.exists():
|
||||
print(f"Error: PDF not found at {pdf_path}")
|
||||
return 1
|
||||
|
||||
total_pages = 15
|
||||
chunk_sizes = [1, 2, 3, 5]
|
||||
percentages = [25, 50, 75, 100]
|
||||
|
||||
print("=" * 60)
|
||||
print("Docling Page Range Benchmark - Multi Scenario")
|
||||
print("=" * 60)
|
||||
print(f"PDF: {pdf_path.name} ({total_pages} pages)")
|
||||
print(f"Warmup: {WARMUP_RUNS} run(s), Measure: {MEASURE_RUNS} run(s)")
|
||||
print(f"Chunk sizes: {chunk_sizes}")
|
||||
print(f"Scenarios: {percentages}%")
|
||||
|
||||
random.seed(RANDOM_SEED)
|
||||
|
||||
report = {
|
||||
"metadata": {
|
||||
"pdf_file": pdf_path.name,
|
||||
"total_pages": total_pages,
|
||||
"warmup_runs": WARMUP_RUNS,
|
||||
"measure_runs": MEASURE_RUNS,
|
||||
"chunk_sizes": chunk_sizes,
|
||||
"random_seed": RANDOM_SEED,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
},
|
||||
"scenarios": [],
|
||||
"summary": {},
|
||||
}
|
||||
|
||||
for pct in percentages:
|
||||
num_pages = max(1, total_pages * pct // 100)
|
||||
|
||||
if pct == 100:
|
||||
target_pages = list(range(1, total_pages + 1))
|
||||
else:
|
||||
target_pages = sorted(random.sample(range(1, total_pages + 1), num_pages))
|
||||
|
||||
scenario_data = run_scenario_benchmark(
|
||||
pdf_path,
|
||||
total_pages,
|
||||
target_pages,
|
||||
chunk_sizes,
|
||||
f"{pct}% pages",
|
||||
)
|
||||
report["scenarios"].append(scenario_data)
|
||||
|
||||
# Summary 생성
|
||||
print("\n" + "=" * 60)
|
||||
print("SUMMARY")
|
||||
print("=" * 60)
|
||||
print(f"{'Scenario':<15} {'Best Method':<20} {'Time':>8} {'Chunks':>8}")
|
||||
print("-" * 60)
|
||||
|
||||
for scenario in report["scenarios"]:
|
||||
best = min(scenario["results"], key=lambda r: r["avg_time"])
|
||||
print(f"{scenario['scenario']:<15} {best['method']:<20} {best['avg_time']:>7.2f}s {best['num_chunks']:>7}")
|
||||
report["summary"][scenario["scenario"]] = {
|
||||
"best_method": best["method"],
|
||||
"best_time": best["avg_time"],
|
||||
"best_chunks": best["num_chunks"],
|
||||
}
|
||||
|
||||
# JSON 저장
|
||||
output_path = project_root / "tests" / "docling_chunking_strategy" / "docling_benchmark_report.json"
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
json.dump(report, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print()
|
||||
print(f"Report saved to: {output_path}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
exit(main())
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,65 @@
|
||||
# Docling Speed Experiment Results
|
||||
|
||||
**Date**: 2026-01-03 14:31:43
|
||||
|
||||
## Summary
|
||||
|
||||
| Approach | Description | Avg (s/doc) | Target | Status | Speedup |
|
||||
|----------|-------------|-------------|--------|--------|---------|
|
||||
| baseline | docling-serve HTTP | 2.283 | - | - | - |
|
||||
| fastapi | FastAPI + SDK singleton | 0.685 | 0.8 | PASS | 3.3x |
|
||||
| subprocess | Persistent subprocess | 0.661 | 1.0 | PASS | 3.5x |
|
||||
|
||||
## Decision
|
||||
|
||||
**Phase 0 PASSED** - FastAPI approach meets the < 0.8s threshold.
|
||||
|
||||
Proceed to Phase 1 implementation:
|
||||
|
||||
- [x] Task 1.1: docling_subprocess_worker.py (skipped - FastAPI only)
|
||||
- [x] Task 1.2: hybrid_server.py (opendataloader-pdf-hybrid CLI)
|
||||
- [x] Task 2.1: DoclingSubprocessClient.java (skipped - FastAPI only)
|
||||
- [x] Task 2.2: DoclingFastServerClient.java
|
||||
- [x] Task 2.3: HybridClientFactory modification
|
||||
- [x] Task 3: Benchmark integration
|
||||
- [x] Task 4: Final validation
|
||||
|
||||
Subprocess approach also passed - both approaches available for implementation.
|
||||
|
||||
## Detailed Statistics
|
||||
|
||||
### Baseline
|
||||
|
||||
- **Description**: docling-serve HTTP API
|
||||
- **Timestamp**: 2026-01-03 14:23:41
|
||||
- **Total documents**: 200
|
||||
- **Successful**: 200
|
||||
- **Failed**: 0
|
||||
- **Total elapsed**: 456.6s
|
||||
- **Average per doc**: 2.2825s
|
||||
- **Min**: 2.0045s
|
||||
- **Max**: 8.0182s
|
||||
|
||||
### Fastapi
|
||||
|
||||
- **Description**: FastAPI server with docling SDK singleton
|
||||
- **Timestamp**: 2026-01-03 14:27:18
|
||||
- **Total documents**: 200
|
||||
- **Successful**: 200
|
||||
- **Failed**: 0
|
||||
- **Total elapsed**: 137.1s
|
||||
- **Average per doc**: 0.6855s
|
||||
- **Min**: 0.1912s
|
||||
- **Max**: 4.2420s
|
||||
|
||||
### Subprocess
|
||||
|
||||
- **Description**: Persistent Python subprocess with docling SDK
|
||||
- **Timestamp**: 2026-01-03 14:30:50
|
||||
- **Total documents**: 200
|
||||
- **Successful**: 200
|
||||
- **Failed**: 0
|
||||
- **Total elapsed**: 132.4s
|
||||
- **Average per doc**: 0.6612s
|
||||
- **Min**: 0.1908s
|
||||
- **Max**: 4.2498s
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,349 @@
|
||||
---
|
||||
name: triage-lab
|
||||
description: Triage logic experiment records and optimization history
|
||||
---
|
||||
|
||||
# Triage Lab - Experiment Records
|
||||
|
||||
This skill manages experiment records and optimization history for triage logic.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
**File**: `java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java`
|
||||
|
||||
### Signal Priority (classifyPage method)
|
||||
1. `hasTableBorder` - TableBorder presence (confidence: 1.0)
|
||||
2. `hasVectorTableSignal` - Grid lines, border lines, line art (confidence: 0.95)
|
||||
3. `hasTextTablePattern` - Text patterns with consecutive validation (confidence: 0.9)
|
||||
4. `hasSuspiciousPattern` - Y-overlap or large gap detection (confidence: 0.85)
|
||||
5. `lineToTextRatio > 0.3` - High line chunk ratio (confidence: 0.8)
|
||||
6. `alignedLineGroups >= 5` - Aligned baseline groups (confidence: 0.7)
|
||||
|
||||
### Key Thresholds
|
||||
| Parameter | Value | Location |
|
||||
|-----------|-------|----------|
|
||||
| LINE_RATIO_THRESHOLD | 0.3 | TriageProcessor:41 |
|
||||
| ALIGNED_LINE_GROUPS_THRESHOLD | 5 | TriageProcessor:46 |
|
||||
| GRID_GAP_MULTIPLIER | 3.0 | TriageProcessor:49 |
|
||||
| MIN_LINE_COUNT_FOR_TABLE | 8 | TriageProcessor:57 |
|
||||
| MIN_GRID_LINES | 3 | TriageProcessor:60 |
|
||||
| MIN_CONSECUTIVE_PATTERNS | 2 | TriageProcessor:79 |
|
||||
|
||||
---
|
||||
|
||||
## Experiment History
|
||||
|
||||
### Experiment 001 (2026-01-03): FP Cause Analysis
|
||||
|
||||
**Goal**: Identify root causes of high False Positive rate
|
||||
|
||||
**Baseline**:
|
||||
- Documents: 200 (42 with tables)
|
||||
- TP: 41, TN: 48, FP: 110, FN: 1
|
||||
- Precision: 27.15%, Recall: 97.62%, F1: 42.49%
|
||||
|
||||
**FP by Signal**:
|
||||
| Signal | Count | % |
|
||||
|--------|-------|---|
|
||||
| hasSuspiciousPattern | 65 | 59.1% |
|
||||
| hasVectorTableSignal | 23 | 20.9% |
|
||||
| hasTableBorder | 14 | 12.7% |
|
||||
| hasTextTablePattern | 5 | 4.5% |
|
||||
| alignedLineGroups | 2 | 1.8% |
|
||||
| highLineRatio | 1 | 0.9% |
|
||||
|
||||
**Root Cause**: Y-overlap check in `hasSuspiciousPattern` is too sensitive
|
||||
- Condition `previous.getTopY() < current.getBottomY()` triggers on normal multi-column layouts
|
||||
|
||||
**Experiments**:
|
||||
| Config | Precision | Recall | F1 | FP | FN |
|
||||
|--------|-----------|--------|-----|-----|-----|
|
||||
| Baseline | 27.15% | 97.62% | 42.49% | 110 | 1 |
|
||||
| Disable Y-overlap | 36.28% | 97.62% | 52.90% | ~69 | 1 |
|
||||
| Only Reliable Signals | 50.67% | 90.48% | 64.96% | ~38 | 4 |
|
||||
| Disable SuspiciousPattern | 39.22% | 95.24% | 55.56% | ~64 | 2 |
|
||||
| Require 3+ patterns | 37.38% | 95.24% | 53.69% | ~67 | 2 |
|
||||
|
||||
**Recommendation**:
|
||||
- To maintain recall: Remove Y-overlap check (Precision +9%, Recall unchanged)
|
||||
- To optimize F1: Use only reliable signals (F1 +22%, Recall -7%)
|
||||
|
||||
**FN Documents**:
|
||||
- `01030000000110`: Missed by all experiments (needs investigation)
|
||||
- `01030000000122`, `01030000000116`, `01030000000117`: Only detected by SuspiciousPattern
|
||||
|
||||
**Applied**: Y-overlap check removed (2026-01-03)
|
||||
|
||||
---
|
||||
|
||||
### Experiment 002 (2026-01-03): Further FP Reduction
|
||||
|
||||
**Goal**: Reduce remaining 72 FPs after Y-overlap removal
|
||||
|
||||
**Current FP by Signal** (after Experiment 001):
|
||||
| Signal | Count | % |
|
||||
|--------|-------|---|
|
||||
| hasSuspiciousPattern | 21 | 29.2% |
|
||||
| hasTableBorder | 14 | 19.4% |
|
||||
| hasVectorTableSignal | 13 | 18.1% |
|
||||
| alignedLineGroups | 10 | 13.9% |
|
||||
| unknown | 8 | 11.1% |
|
||||
| hasTextTablePattern | 5 | 6.9% |
|
||||
| highLineRatio | 1 | 1.4% |
|
||||
|
||||
**Experiment 2A: Gap Multiplier** (hasSuspiciousPattern)
|
||||
| Gap | Precision | Recall | F1 | FP | FN |
|
||||
|-----|-----------|--------|-----|-----|-----|
|
||||
| 3.0 (current) | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
| 4.0 | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
| 5.0 | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
| 6.0 | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
|
||||
→ No effect (Y-overlap removal already optimized this signal)
|
||||
|
||||
**Experiment 2B: AlignedLineGroups Threshold**
|
||||
| Threshold | Precision | Recall | F1 | FP | FN |
|
||||
|-----------|-----------|--------|-----|-----|-----|
|
||||
| 3 (current) | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
| 4 | 39.39% | 92.86% | 55.32% | 60 | 3 |
|
||||
| **5** | **39.80%** | **92.86%** | **55.71%** | **59** | **3** |
|
||||
| 6 | 39.80% | 92.86% | 55.71% | 59 | 3 |
|
||||
|
||||
→ **Recommended**: Threshold 5 (FP -5, Recall maintained)
|
||||
|
||||
**Experiment 2C: Vector Signal Criteria**
|
||||
| LineCount | GridLines | Precision | Recall | F1 | FP | FN |
|
||||
|-----------|-----------|-----------|--------|-----|-----|-----|
|
||||
| 8, 3 (current) | | 37.86% | 92.86% | 53.79% | 64 | 3 |
|
||||
| 10, 4 | | 38.24% | 92.86% | 54.17% | 63 | 3 |
|
||||
| 12, 4 | | 37.62% | 90.48% | 53.15% | 63 | 4 |
|
||||
|
||||
→ Minimal effect (FP -1, higher values reduce Recall)
|
||||
|
||||
**Recommendation**:
|
||||
- Apply `alignedLineGroups` threshold 3 → 5
|
||||
- Expected: FP 64 → 59 (-5), Recall 92.86% (maintained), F1 +1.92%
|
||||
|
||||
**Applied**: alignedLineGroups threshold 3 → 5 (2026-01-03)
|
||||
|
||||
**Actual Results**:
|
||||
| Metric | Before (Exp 001) | After (Exp 002) | Change |
|
||||
|--------|------------------|-----------------|--------|
|
||||
| FP | 72 | 67 | -5 |
|
||||
| FN | 1 | 1 | 0 |
|
||||
| Precision | 36.28% | 37.96% | +1.68% |
|
||||
| Recall | 97.62% | 97.62% | 0 |
|
||||
| F1 | 52.90% | 54.67% | +1.77% |
|
||||
|
||||
**Next Steps**:
|
||||
- Investigate `hasTableBorder` FPs (14 cases, external library)
|
||||
- Investigate `unknown` FPs (8 cases)
|
||||
|
||||
---
|
||||
|
||||
### Experiment 003 (2026-01-03): VectorTableSignal & SuspiciousPattern Analysis
|
||||
|
||||
**Goal**: Further reduce FP 67 while maintaining high Recall
|
||||
|
||||
**Current FP by Signal** (after Experiment 002):
|
||||
| Signal | Count | % |
|
||||
|--------|-------|---|
|
||||
| hasVectorTableSignal | 23 | 34.3% |
|
||||
| hasSuspiciousPattern | 19 | 28.4% |
|
||||
| hasTableBorder | 14 | 20.9% |
|
||||
| hasTextTablePattern | 5 | 7.5% |
|
||||
| alignedLineGroups | 5 | 7.5% |
|
||||
| highLineRatio | 1 | 1.5% |
|
||||
|
||||
**VectorTableSignal Sub-signal Analysis** (23 FPs):
|
||||
| Sub-signal | Count |
|
||||
|------------|-------|
|
||||
| hasAlignedShortLines | 30 |
|
||||
| hasTableBorderLines | 22 |
|
||||
| hasGridLines | 16 |
|
||||
| lineArt>=8 | 13 |
|
||||
| hasRowSeparatorPattern | 12 |
|
||||
|
||||
→ `hasAlignedShortLines` is the primary cause of VectorSignal FPs
|
||||
|
||||
**Experiments**:
|
||||
| Config | Precision | Recall | F1 | FP | FN |
|
||||
|--------|-----------|--------|-----|-----|-----|
|
||||
| Current (Exp 002) | 37.96% | 97.62% | 54.67% | 67 | 1 |
|
||||
| 003B: Disable VectorSignal | 40.00% | 90.48% | 55.47% | 57 | 4 |
|
||||
| 003C: Grid OR BorderLines only | 40.21% | 92.86% | 56.12% | 58 | 3 |
|
||||
| **003D: Disable SuspiciousPattern** | **42.11%** | **95.24%** | **58.39%** | **55** | **2** |
|
||||
| 003F: Only Reliable Signals | 56.25% | 85.71% | 67.92% | 28 | 6 |
|
||||
| 003G: Disable AlignedShortLines | 40.00% | 95.24% | 56.34% | 60 | 2 |
|
||||
| 003I: Combined (NoAlign+NoSusp) | 44.71% | 90.48% | 59.84% | 47 | 4 |
|
||||
|
||||
**Analysis**:
|
||||
- `hasTableBorder` (14 FPs): External library, cannot modify
|
||||
- `hasVectorTableSignal` (23 FPs): `hasAlignedShortLines` too aggressive
|
||||
- `hasSuspiciousPattern` (19 FPs): Gap detection catches non-table layouts
|
||||
|
||||
**Recommendation**:
|
||||
- **Best for Recall**: 003D (Disable SuspiciousPattern)
|
||||
- FP: 67 → 55 (-12), FN: 1 → 2 (+1), Recall: 95.24%
|
||||
- **Best for F1**: 003I (Combined)
|
||||
- FP: 67 → 47 (-20), FN: 1 → 4 (+3), F1: 59.84%
|
||||
|
||||
**FN Documents** (003D):
|
||||
- `01030000000122`: Only detected by SuspiciousPattern (gap-based)
|
||||
- `01030000000110`: Never detected (needs separate investigation)
|
||||
|
||||
**Applied**: 003D - Disabled hasSuspiciousPattern (2026-01-03)
|
||||
|
||||
**Actual Results**:
|
||||
| Metric | Before (Exp 002) | After (Exp 003) | Change |
|
||||
|--------|------------------|-----------------|--------|
|
||||
| FP | 67 | 55 | -12 |
|
||||
| FN | 1 | 2 | +1 |
|
||||
| Precision | 37.96% | 42.11% | +4.15% |
|
||||
| Recall | 97.62% | 95.24% | -2.38% |
|
||||
| F1 | 54.67% | 58.39% | +3.72% |
|
||||
|
||||
---
|
||||
|
||||
### Experiment 004 (2026-01-03): AlignedLineGroups Signal Analysis
|
||||
|
||||
**Goal**: Further reduce FP 55 while maintaining Recall 95.24%
|
||||
|
||||
**Current FP by Signal** (after Experiment 003):
|
||||
| Signal | Count | % |
|
||||
|--------|-------|---|
|
||||
| hasVectorTableSignal | 23 | 41.8% |
|
||||
| hasTableBorder | 14 | 25.5% |
|
||||
| alignedLineGroups | 12 | 21.8% |
|
||||
| hasTextTablePattern | 5 | 9.1% |
|
||||
| highLineRatio | 1 | 1.8% |
|
||||
|
||||
**VectorTableSignal Sub-signal Analysis** (23 FPs):
|
||||
| Sub-signal | Count |
|
||||
|------------|-------|
|
||||
| hasAlignedShortLines | 16 |
|
||||
| hasTableBorderLines | 10 |
|
||||
| lineArt>=8 | 8 |
|
||||
| hasRowSeparatorPattern | 7 |
|
||||
| hasGridLines | 5 |
|
||||
|
||||
**Experiments**:
|
||||
| Config | Precision | Recall | F1 | FP | FN |
|
||||
|--------|-----------|--------|-----|-----|-----|
|
||||
| Current (Exp 003) | 42.11% | 95.24% | 58.39% | 55 | 2 |
|
||||
| 004A: NoAlignedShortLines | 44.71% | 90.48% | 59.84% | 47 | 4 |
|
||||
| 004B: Grid+BorderLines only | 45.12% | 88.10% | 59.68% | 45 | 5 |
|
||||
| **004D: No alignedLineGroups** | **48.19%** | **95.24%** | **64.00%** | **43** | **2** |
|
||||
| 004E: alignedLineGroups>=7 | 44.94% | 95.24% | 61.07% | 49 | 2 |
|
||||
| 004G: NoAlignShort+Groups>=7 | 48.10% | 90.48% | 62.81% | 41 | 4 |
|
||||
| 004I: Reliable Only | 54.41% | 88.10% | 67.27% | 31 | 5 |
|
||||
|
||||
**Analysis**:
|
||||
- `alignedLineGroups` signal caused 12 FPs but detected no additional true tables
|
||||
- Disabling it removes all 12 FPs without any FN increase
|
||||
- Best option for maintaining Recall while improving Precision
|
||||
|
||||
**Applied**: 004D - Disabled alignedLineGroups signal (2026-01-03)
|
||||
|
||||
**Actual Results**:
|
||||
| Metric | Before (Exp 003) | After (Exp 004) | Change |
|
||||
|--------|------------------|-----------------|--------|
|
||||
| FP | 55 | 43 | -12 |
|
||||
| FN | 2 | 2 | 0 |
|
||||
| Precision | 42.11% | 48.19% | +6.08% |
|
||||
| Recall | 95.24% | 95.24% | 0 |
|
||||
| F1 | 58.39% | 64.00% | +5.61% |
|
||||
|
||||
**Next Steps**:
|
||||
- Investigate `hasVectorTableSignal` FPs (23 remaining) - hasAlignedShortLines main cause
|
||||
- Investigate `hasTableBorder` FPs (14 cases, external library limitation)
|
||||
|
||||
---
|
||||
|
||||
### Experiment 005 (2026-01-03): Large Image Signal for FN Reduction
|
||||
|
||||
**Goal**: Reduce FN by detecting pages with large images (potential table/chart images)
|
||||
|
||||
**Background**:
|
||||
- FN documents `01030000000110` and `01030000000122` contain images with tables
|
||||
- 110: 28.64% page area (graph image)
|
||||
- 122: 11.73% page area (table image)
|
||||
|
||||
**Implementation**:
|
||||
- Added `hasLargeImage` signal to TriageProcessor
|
||||
- Detects ImageChunk objects and calculates max image area / page area ratio
|
||||
|
||||
**Experiments**:
|
||||
| Threshold | Precision | Recall | F1 | FP | FN |
|
||||
|-----------|-----------|--------|-----|-----|-----|
|
||||
| Baseline (no image) | 48.19% | 95.24% | 64.00% | 43 | 2 |
|
||||
| 10% | 33.07% | **100%** | 49.70% | 85 | 0 |
|
||||
| **11%** | **33.60%** | **100%** | **50.30%** | **83** | **0** |
|
||||
| 15% | 35.96% | 97.62% | 52.56% | 73 | 1 |
|
||||
|
||||
**Analysis**:
|
||||
- 11% threshold achieves **100% Recall** (all 42 table documents detected)
|
||||
- Trade-off: FP increases from 43 → 83 (+40), Precision drops from 48% → 34%
|
||||
- F1 decreases from 64% → 50% due to high FP increase
|
||||
- Many FPs are documents with decorative images, diagrams, photos
|
||||
|
||||
**Experiment 005B: Adding Aspect Ratio Condition**
|
||||
|
||||
Observation: FN documents have wide images (ratio 1.79, 3.68), while FP documents often have square/tall images (ratio 0.6~1.5).
|
||||
|
||||
| Config | Precision | Recall | F1 | FP | FN |
|
||||
|--------|-----------|--------|-----|-----|-----|
|
||||
| Baseline (no image) | 48.19% | 95.24% | 64.00% | 43 | 2 |
|
||||
| 11% only | 33.60% | 100% | 50.30% | 83 | 0 |
|
||||
| 11% + ratio 1.7 | 42.86% | 100% | 60.00% | 56 | 0 |
|
||||
| **11% + ratio 1.75** | **43.30%** | **100%** | **60.43%** | **55** | **0** |
|
||||
| 11% + ratio 2.0 | 46.59% | 97.62% | 63.08% | 47 | 1 |
|
||||
|
||||
**Final Configuration**:
|
||||
- Image area >= 11% of page area
|
||||
- Image aspect ratio (width/height) >= 1.75
|
||||
|
||||
**Trade-off**:
|
||||
- Achieves **100% Recall** (all 42 table documents detected)
|
||||
- FP increases from 43 → 55 (+12)
|
||||
- F1 decreases from 64% → 60.43% (-3.57%)
|
||||
|
||||
**Applied**: 11% + aspect ratio 1.75 (2026-01-03)
|
||||
|
||||
---
|
||||
|
||||
## Template for New Experiments
|
||||
|
||||
```markdown
|
||||
### Experiment XXX (YYYY-MM-DD): [Title]
|
||||
|
||||
**Goal**: [What are you trying to improve?]
|
||||
|
||||
**Changes**: [What did you modify?]
|
||||
|
||||
**Results**:
|
||||
| Config | Precision | Recall | F1 | FP | FN |
|
||||
|--------|-----------|--------|-----|-----|-----|
|
||||
| Before | | | | | |
|
||||
| After | | | | | |
|
||||
|
||||
**Conclusion**: [What did you learn? Should this be applied?]
|
||||
|
||||
**Next Steps**: [What to try next?]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How to Run Experiments
|
||||
|
||||
```bash
|
||||
# Run triage accuracy test
|
||||
./scripts/test-java.sh -Dtest=TriageProcessorIntegrationTest#testTriageAccuracyOnBenchmarkPDFs
|
||||
|
||||
# Debug specific document
|
||||
./scripts/bench.sh --doc-id 01030000000110
|
||||
```
|
||||
|
||||
## Related Files
|
||||
- [TriageProcessor.java](../../java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java)
|
||||
- [TriageProcessorIntegrationTest.java](../../java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/TriageProcessorIntegrationTest.java)
|
||||
@@ -0,0 +1,121 @@
|
||||
# Hybrid PDF Processing System - Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
Hybrid PDF processing system combining Java heuristics + external AI backends.
|
||||
Routes pages via per-page Triage: simple pages to fast Java path, complex tables/OCR to AI backend.
|
||||
|
||||
## Key Decisions
|
||||
|
||||
| Item | Decision |
|
||||
|------|----------|
|
||||
| CLI Option | `--hybrid <off\|docling\|hancom\|...>` |
|
||||
| Default | `off` (Java-only, no external dependency) |
|
||||
| First Backend | `docling` (docling-serve REST API) |
|
||||
| Automation | Semi-automatic (benchmark/analysis auto, code changes require approval) |
|
||||
| Triage Strategy | Conservative (minimize FN, accept FP, route uncertain pages to backend) |
|
||||
|
||||
---
|
||||
|
||||
## CLI Usage
|
||||
|
||||
```bash
|
||||
# Default: Java-only processing
|
||||
opendataloader-pdf input.pdf
|
||||
opendataloader-pdf --hybrid off input.pdf
|
||||
|
||||
# Use docling backend
|
||||
opendataloader-pdf --hybrid docling input.pdf
|
||||
|
||||
# With custom backend URL
|
||||
opendataloader-pdf --hybrid docling --hybrid-url http://localhost:5001 input.pdf
|
||||
|
||||
# Future backends
|
||||
opendataloader-pdf --hybrid hancom input.pdf
|
||||
```
|
||||
|
||||
## Hybrid Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--hybrid <name>` | Hybrid backend: `off` (default), `docling`, `hancom`, etc. |
|
||||
| `--hybrid-url <url>` | Backend server URL (overrides default) |
|
||||
| `--hybrid-timeout <ms>` | Request timeout in milliseconds (default: 0, no timeout) |
|
||||
| `--hybrid-fallback` | Opt in to Java fallback on backend error (default: disabled — backend failures fail fast with non-zero exit) |
|
||||
|
||||
## Supported Backends
|
||||
|
||||
| Backend | Status | Description |
|
||||
|---------|--------|-------------|
|
||||
| `off` | ✅ Default | Java-only, no external calls |
|
||||
| `docling-fast` | ✅ Available | docling-serve (local) |
|
||||
| `hancom` | 📋 Future (Priority) | Hancom Document AI |
|
||||
| `azure` | 📋 Future | Azure Document Intelligence |
|
||||
| `google` | 📋 Future | Google Document AI |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ PDF Input │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ ContentFilterProcessor │
|
||||
│ (existing: text filtering) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ TriageProcessor.triageAllPages() │
|
||||
│ - Batch triage all pages │
|
||||
│ - Output: Map<PageNumber, TriageResult> │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────┴───────────────┐
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────┐ ┌─────────────────────────┐
|
||||
│ JAVA Path │ │ BACKEND Path │
|
||||
│ (parallel processing) │ │ (single batch API call)│
|
||||
│ │ │ │
|
||||
│ ExecutorService │ │ BackendClient │
|
||||
│ - TableBorderProcessor │ │ - Send all pages once │
|
||||
│ - TextLineProcessor │ │ - Receive all results │
|
||||
│ - ParagraphProcessor │ │ SchemaTransformer │
|
||||
└─────────────────────────┘ └─────────────────────────┘
|
||||
│ │
|
||||
│ CONCURRENT │
|
||||
└───────────────┬───────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Result Merger │
|
||||
│ (preserve page order) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Post-processing & Output Generation │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Backend unavailable or returns failed pages | Fail fast with non-zero exit (default). Opt in to `--hybrid-fallback` to fall back to Java for backend-routed pages. |
|
||||
| Triage FN (missed tables) | Conservative threshold, benchmark monitoring |
|
||||
| Schema mismatch | Step-by-step validation, type checking |
|
||||
| Slow processing | Parallel execution, batch API calls |
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- **Implementation Tasks**: [hybrid-mode-tasks.md](hybrid-mode-tasks.md)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,73 @@
|
||||
# Docling vs OpenDataLoader Output Comparison
|
||||
|
||||
## Test Document
|
||||
- File: `01030000000045.pdf` (1 page with table)
|
||||
|
||||
## Element Count Comparison
|
||||
|
||||
| Category | Docling | OpenDataLoader |
|
||||
|----------|---------|----------------|
|
||||
| Tables | 1 | 1 |
|
||||
| Text elements | 5 | 4 paragraphs |
|
||||
| Images | 0 | 1 |
|
||||
| Headings | (N/A - uses labels) | 1 |
|
||||
|
||||
## Text Element Labels (Docling)
|
||||
|
||||
| Label | Count |
|
||||
|-------|-------|
|
||||
| caption | 1 |
|
||||
| footnote | 1 |
|
||||
| page_footer | 1 |
|
||||
| page_header | 1 |
|
||||
| text | 1 |
|
||||
|
||||
## Table Structure Comparison
|
||||
|
||||
| Property | Docling | OpenDataLoader |
|
||||
|----------|---------|----------------|
|
||||
| Rows | 9 | 3 |
|
||||
| Columns | 3 | 3 |
|
||||
| Total cells | 26 | 9 |
|
||||
|
||||
**Note**: Docling detects more rows in the table structure. This may be due to:
|
||||
- Different table detection algorithms
|
||||
- OpenDataLoader may have merged some rows
|
||||
- Different handling of header rows
|
||||
|
||||
## Bounding Box Comparison (Table)
|
||||
|
||||
| System | l/left | t/top | r/right | b/bottom | Origin |
|
||||
|--------|--------|-------|---------|----------|--------|
|
||||
| Docling | 53.22 | 439.98 | 373.94 | 234.74 | BOTTOMLEFT |
|
||||
| OpenDataLoader | 54.0 | 234.44 | 372.73 | 440.21 | BOTTOMLEFT |
|
||||
|
||||
**Coordinate mapping**: Both use BOTTOMLEFT origin.
|
||||
- Docling: `{l, t, r, b}` where t=top, b=bottom
|
||||
- OpenDataLoader: `[left, bottom, right, top]`
|
||||
|
||||
So the actual coordinates match closely:
|
||||
- Left: 53.22 ≈ 54.0
|
||||
- Bottom: 234.74 ≈ 234.44
|
||||
- Right: 373.94 ≈ 372.73
|
||||
- Top: 439.98 ≈ 440.21
|
||||
|
||||
## Schema Mapping Summary
|
||||
|
||||
| Docling Type | OpenDataLoader Type |
|
||||
|--------------|---------------------|
|
||||
| texts (label: text) | paragraph |
|
||||
| texts (label: section_header) | heading |
|
||||
| tables | table |
|
||||
| pictures | image |
|
||||
| texts (label: page_header) | paragraph (filtered as header) |
|
||||
| texts (label: page_footer) | paragraph (filtered as footer) |
|
||||
| texts (label: caption) | paragraph |
|
||||
| texts (label: footnote) | paragraph |
|
||||
|
||||
## Key Differences
|
||||
|
||||
1. **Type naming**: Docling uses `label` field for text types, OpenDataLoader uses `type`
|
||||
2. **Table structure**: Docling detects more detailed row structure
|
||||
3. **Coordinate format**: Same origin but different field order
|
||||
4. **Heading detection**: Docling uses `SectionHeaderItem` with `level`, OpenDataLoader uses `heading` type with `level`
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,42 @@
|
||||
01030000000045.pdf
|
||||
01030000000046.pdf
|
||||
01030000000047.pdf
|
||||
01030000000051.pdf
|
||||
01030000000052.pdf
|
||||
01030000000053.pdf
|
||||
01030000000064.pdf
|
||||
01030000000078.pdf
|
||||
01030000000081.pdf
|
||||
01030000000082.pdf
|
||||
01030000000083.pdf
|
||||
01030000000084.pdf
|
||||
01030000000088.pdf
|
||||
01030000000089.pdf
|
||||
01030000000090.pdf
|
||||
01030000000110.pdf
|
||||
01030000000116.pdf
|
||||
01030000000117.pdf
|
||||
01030000000119.pdf
|
||||
01030000000120.pdf
|
||||
01030000000121.pdf
|
||||
01030000000122.pdf
|
||||
01030000000127.pdf
|
||||
01030000000128.pdf
|
||||
01030000000130.pdf
|
||||
01030000000132.pdf
|
||||
01030000000146.pdf
|
||||
01030000000147.pdf
|
||||
01030000000149.pdf
|
||||
01030000000150.pdf
|
||||
01030000000165.pdf
|
||||
01030000000166.pdf
|
||||
01030000000170.pdf
|
||||
01030000000178.pdf
|
||||
01030000000180.pdf
|
||||
01030000000182.pdf
|
||||
01030000000187.pdf
|
||||
01030000000188.pdf
|
||||
01030000000189.pdf
|
||||
01030000000190.pdf
|
||||
01030000000197.pdf
|
||||
01030000000200.pdf
|
||||
@@ -0,0 +1,105 @@
|
||||
# IObject Class Structure
|
||||
|
||||
## Overview
|
||||
IObject is imported from `org.verapdf.wcag.algorithms.entities.IObject` (external verapdf-wcag-algs library).
|
||||
|
||||
## JSON Output Types
|
||||
|
||||
Based on sample response analysis, OpenDataLoader produces the following element types:
|
||||
|
||||
### Element Types
|
||||
|
||||
| Type | JSON `type` field | Description |
|
||||
|------|-------------------|-------------|
|
||||
| Paragraph | `paragraph` | Text paragraph with font info |
|
||||
| Heading | `heading` | Section heading with level |
|
||||
| Table | `table` | Table with rows and cells |
|
||||
| Image | `image` | Image/figure element |
|
||||
| List | `list` | Bulleted or numbered list |
|
||||
|
||||
### Common Fields (all types)
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "paragraph",
|
||||
"id": 17,
|
||||
"page number": 1,
|
||||
"bounding box": [left, bottom, right, top] // PDF points, origin at bottom-left
|
||||
}
|
||||
```
|
||||
|
||||
### Paragraph Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "paragraph",
|
||||
"font": "ArialMT",
|
||||
"font size": 8.0,
|
||||
"text color": "[0.0, 0.0, 0.0, 0.7]",
|
||||
"content": "Text content here"
|
||||
}
|
||||
```
|
||||
|
||||
### Heading Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "heading",
|
||||
"level": "1",
|
||||
"content": "Heading text"
|
||||
}
|
||||
```
|
||||
|
||||
### Table Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "table",
|
||||
"level": "1",
|
||||
"number of rows": 3,
|
||||
"number of columns": 3,
|
||||
"rows": [
|
||||
{
|
||||
"type": "table row",
|
||||
"row number": 1,
|
||||
"cells": [
|
||||
{
|
||||
"type": "table cell",
|
||||
"page number": 1,
|
||||
"bounding box": [left, bottom, right, top],
|
||||
"row number": 1,
|
||||
"column number": 1,
|
||||
"row span": 1,
|
||||
"column span": 1,
|
||||
"kids": [
|
||||
{
|
||||
"type": "paragraph",
|
||||
"content": "Cell text"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Bounding Box Coordinate System
|
||||
|
||||
- **OpenDataLoader**: `[left, bottom, right, top]` in PDF points, origin at BOTTOMLEFT
|
||||
- **Docling**: `{l, t, r, b}` with `coord_origin: "BOTTOMLEFT"` or `"TOPLEFT"`
|
||||
|
||||
### Conversion Notes
|
||||
|
||||
- If docling uses TOPLEFT origin: `bottom = page_height - docling_t`, `top = page_height - docling_b`
|
||||
- If docling uses BOTTOMLEFT origin: direct mapping `[l, b, r, t]` → `[left, bottom, right, top]`
|
||||
|
||||
## Key Java Classes
|
||||
|
||||
From the codebase:
|
||||
|
||||
- `TableBorder` - Table with border-based detection
|
||||
- `TableBorderRow` - Table row
|
||||
- `TableBorderCell` - Table cell with contents, rowSpan, colSpan
|
||||
- `BoundingBox` - PDF coordinates (page, left, bottom, right, top)
|
||||
- Processors: `TextLineProcessor`, `TableBorderProcessor`, `HeadingProcessor`, `ListProcessor`
|
||||
@@ -0,0 +1,370 @@
|
||||
{
|
||||
"file name" : "01030000000045.pdf",
|
||||
"number of pages" : 1,
|
||||
"author" : null,
|
||||
"title" : null,
|
||||
"creation date" : null,
|
||||
"modification date" : null,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 17,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 281.571, 551.756, 372.715, 560.694 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 8.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 0.7]",
|
||||
"content" : "Civil Society Engagement"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 19,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.0, 499.92, 372.727, 539.282 ],
|
||||
"font" : "Georgia",
|
||||
"font size" : 10.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "election integrity. The registration of local election observers runs until 25 May, and the NEC is still reviewing the application of nearly 5,000 observers."
|
||||
}, {
|
||||
"type" : "heading",
|
||||
"id" : 20,
|
||||
"level" : "Doctitle",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.0, 455.381, 350.475, 480.87 ],
|
||||
"heading level" : 1,
|
||||
"font" : "Arial-BoldMT",
|
||||
"font size" : 11.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Table: The number of accredited observers as of 28 April 202215"
|
||||
}, {
|
||||
"type" : "table",
|
||||
"id" : 21,
|
||||
"level" : "1",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.0, 234.441, 372.727, 440.212 ],
|
||||
"number of rows" : 3,
|
||||
"number of columns" : 3,
|
||||
"rows" : [ {
|
||||
"type" : "table row",
|
||||
"row number" : 1,
|
||||
"cells" : [ {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.375, 413.86, 83.52, 440.087 ],
|
||||
"row number" : 1,
|
||||
"column number" : 1,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 1,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 61.757, 427.253, 75.761, 437.307 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "No."
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 83.52, 413.86, 273.472, 440.087 ],
|
||||
"row number" : 1,
|
||||
"column number" : 2,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 2,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 427.253, 173.056, 437.307 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Name of organization"
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 273.472, 413.86, 372.352, 440.087 ],
|
||||
"row number" : 1,
|
||||
"column number" : 3,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 3,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 280.08, 416.453, 366.111, 437.307 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Number of accredited observers"
|
||||
} ]
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table row",
|
||||
"row number" : 2,
|
||||
"cells" : [ {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.375, 249.993, 83.52, 413.86 ],
|
||||
"row number" : 2,
|
||||
"column number" : 1,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "list",
|
||||
"id" : 4,
|
||||
"level" : "1",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 263.386, 71.261, 410.955 ],
|
||||
"numbering style" : "arabic numbers",
|
||||
"number of list items" : 7,
|
||||
"list items" : [ {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 400.9, 71.261, 410.955 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "1",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 374.548, 71.261, 384.603 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "2",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 348.196, 71.261, 358.251 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "3",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 321.844, 71.261, 331.898 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "4",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 295.491, 71.261, 305.546 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "5",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 278.939, 71.261, 288.993 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "6",
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "list item",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 66.257, 263.386, 71.261, 273.441 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "7",
|
||||
"kids" : [ ]
|
||||
} ]
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 83.52, 249.993, 273.472, 413.86 ],
|
||||
"row number" : 2,
|
||||
"column number" : 2,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 5,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 390.1, 249.615, 410.955 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Union of Youth Federations of Cambodia (UYFC)"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 6,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 363.748, 225.426, 384.603 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Cambodian Women for Peace and Development"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 7,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 337.396, 239.584, 358.251 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Association of Democratic Students of Cambodia"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 8,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 311.044, 231.623, 331.898 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Association of Intellectual and Youth Volunteer"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 9,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 252.586, 237.745, 305.546 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Our Friends Association COMFREL Traditional and Modern Mental Health Organization"
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 273.472, 249.993, 372.352, 413.86 ],
|
||||
"row number" : 2,
|
||||
"column number" : 3,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 10,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 309.336, 400.9, 336.858, 410.955 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "17,266"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 11,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 311.839, 374.548, 334.357, 384.603 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "9,835"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 12,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 315.926, 348.196, 329.608, 358.251 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "711"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 13,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 318.095, 321.844, 328.103, 331.898 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "46"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 14,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 318.095, 263.386, 328.103, 305.546 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "27 26 15"
|
||||
} ]
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table row",
|
||||
"row number" : 3,
|
||||
"cells" : [ {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.375, 234.566, 83.52, 249.993 ],
|
||||
"row number" : 3,
|
||||
"column number" : 1,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 83.52, 234.566, 273.472, 249.993 ],
|
||||
"row number" : 3,
|
||||
"column number" : 2,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 15,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 87.52, 237.034, 108.351, 247.089 ],
|
||||
"font" : "Arial-BoldMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "Total"
|
||||
} ]
|
||||
}, {
|
||||
"type" : "table cell",
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 273.472, 234.566, 372.352, 249.993 ],
|
||||
"row number" : 3,
|
||||
"column number" : 3,
|
||||
"row span" : 1,
|
||||
"column span" : 1,
|
||||
"kids" : [ {
|
||||
"type" : "paragraph",
|
||||
"id" : 16,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 309.336, 237.034, 336.858, 247.089 ],
|
||||
"font" : "Arial-BoldMT",
|
||||
"font size" : 9.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 1.0]",
|
||||
"content" : "27,926"
|
||||
} ]
|
||||
} ]
|
||||
} ]
|
||||
}, {
|
||||
"type" : "image",
|
||||
"id" : 22,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.0, 68.275, 126.0, 68.525 ]
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 23,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 54.0, 52.729, 185.287, 59.432 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 6.0,
|
||||
"text color" : "[0.86, 0.57, 0.0, 0.16]",
|
||||
"content" : "15 https://www.nec.gov.kh/khmer/content/5524"
|
||||
}, {
|
||||
"type" : "paragraph",
|
||||
"id" : 18,
|
||||
"page number" : 1,
|
||||
"bounding box" : [ 363.829, 34.305, 372.725, 43.242 ],
|
||||
"font" : "ArialMT",
|
||||
"font size" : 8.0,
|
||||
"text color" : "[0.0, 0.0, 0.0, 0.75]",
|
||||
"content" : "17"
|
||||
} ]
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
Civil Society Engagement
|
||||
|
||||
election integrity. The registration of local election observers runs until 25 May, and the NEC is still reviewing the application of nearly 5,000 observers.
|
||||
|
||||
# Table: The number of accredited observers as of 28 April 202215
|
||||
|
||||
|No.|Name of organization|Number of accredited observers|
|
||||
|---|---|---|
|
||||
|1<br>2<br>3<br>4<br>5<br>6<br>7<br>|Union of Youth Federations of Cambodia (UYFC)<br><br>Cambodian Women for Peace and Development<br><br>Association of Democratic Students of Cambodia<br><br>Association of Intellectual and Youth Volunteer<br><br>Our Friends Association COMFREL Traditional and Modern Mental Health Organization|17,266<br><br>9,835<br><br>711<br><br>46<br><br>27 26 15|
|
||||
||Total|27,926|
|
||||
|
||||
|
||||
15 https://www.nec.gov.kh/khmer/content/5524
|
||||
|
||||
17
|
||||
|
||||
@@ -0,0 +1,630 @@
|
||||
# CID Font Extraction Failure Detection — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Detect pages where CID font extraction failed (high U+FFFD ratio), emit warning logs, and auto-route to OCR backend in hybrid mode.
|
||||
|
||||
**Architecture:** Measure replacement character ratio in ContentFilterProcessor before replacement, store in StaticLayoutContainers, consume in TriageProcessor as highest-priority signal. Warning log fires regardless of hybrid mode.
|
||||
|
||||
**Tech Stack:** Java 11+, JUnit Jupiter, veraPDF API (`ChunkParser.REPLACEMENT_CHARACTER_STRING`)
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `TextProcessor.java` | New `measureReplacementCharRatio()` method |
|
||||
| `StaticLayoutContainers.java` | Per-page replacement ratio storage |
|
||||
| `ContentFilterProcessor.java` | Measure + warn + store before replacement |
|
||||
| `TriageProcessor.java` | Signal 0: route high-ratio pages to BACKEND |
|
||||
| `TextProcessorTest.java` | Unit tests for measurement |
|
||||
| `TriageProcessorTest.java` | Unit tests for Signal 0 routing |
|
||||
| `CidFontDetectionTest.java` (new) | e2e test with synthetic PDF |
|
||||
| `test/resources/cid-font-no-tounicode.pdf` (new) | Test fixture |
|
||||
| `test/resources/generate-cid-test-pdf.py` (new) | Generation script (reference) |
|
||||
|
||||
All paths below are relative to `java/opendataloader-pdf-core/src/`.
|
||||
|
||||
---
|
||||
|
||||
## Chunk 1: Measurement + Storage
|
||||
|
||||
### Task 1: Add per-page ratio storage to StaticLayoutContainers
|
||||
|
||||
**Files:**
|
||||
- Modify: `main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java`
|
||||
|
||||
- [ ] **Step 1: Add ThreadLocal map field and imports**
|
||||
|
||||
Add after line 40 (`imageFormat` field):
|
||||
|
||||
```java
|
||||
private static final ThreadLocal<Map<Integer, Double>> replacementCharRatios = ThreadLocal.withInitial(HashMap::new);
|
||||
```
|
||||
|
||||
Add to imports:
|
||||
|
||||
```java
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add getter and setter**
|
||||
|
||||
Add after `setImageFormat()` (after line 145):
|
||||
|
||||
```java
|
||||
public static void setReplacementCharRatio(int pageNumber, double ratio) {
|
||||
replacementCharRatios.get().put(pageNumber, ratio);
|
||||
}
|
||||
|
||||
public static double getReplacementCharRatio(int pageNumber) {
|
||||
return replacementCharRatios.get().getOrDefault(pageNumber, 0.0);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Clear in clearContainers()**
|
||||
|
||||
Add inside `clearContainers()` method, after line 51 (`imageFormat.set(...)`):
|
||||
|
||||
```java
|
||||
replacementCharRatios.get().clear();
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Compile check**
|
||||
|
||||
Run: `cd java && mvn compile -pl opendataloader-pdf-core -q`
|
||||
Expected: BUILD SUCCESS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java
|
||||
git commit -m "feat: add per-page replacement char ratio storage to StaticLayoutContainers"
|
||||
```
|
||||
|
||||
### Task 2: Add measureReplacementCharRatio to TextProcessor
|
||||
|
||||
**Files:**
|
||||
- Modify: `main/java/org/opendataloader/pdf/processors/TextProcessor.java`
|
||||
- Test: `test/java/org/opendataloader/pdf/processors/TextProcessorTest.java`
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
Add to `TextProcessorTest.java` after the last test method (before closing `}`):
|
||||
|
||||
```java
|
||||
@Test
|
||||
public void testMeasureReplacementCharRatio_allReplacement() {
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 100.0, 20.0),
|
||||
"\uFFFD\uFFFD\uFFFD", 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertEquals(1.0, ratio, 0.001);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMeasureReplacementCharRatio_noReplacement() {
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 100.0, 20.0),
|
||||
"Hello World", 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertEquals(0.0, ratio, 0.001);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMeasureReplacementCharRatio_mixed() {
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
// 3 replacement chars out of 10 total = 0.3
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 100.0, 20.0),
|
||||
"\uFFFD\uFFFD\uFFFDAbcdefg", 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertEquals(0.3, ratio, 0.001);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMeasureReplacementCharRatio_emptyContents() {
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertEquals(0.0, ratio, 0.001);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testMeasureReplacementCharRatio_nonTextChunksIgnored() {
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(new ImageChunk(new BoundingBox(1, 10.0, 10.0, 100.0, 20.0)));
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 30.0, 100.0, 40.0),
|
||||
"\uFFFD\uFFFD\uFFFD\uFFFD\uFFFD", 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
// Only TextChunks counted: 5/5 = 1.0
|
||||
Assertions.assertEquals(1.0, ratio, 0.001);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest#testMeasureReplacementCharRatio_allReplacement -q`
|
||||
Expected: FAIL — `measureReplacementCharRatio` method not found
|
||||
|
||||
- [ ] **Step 3: Implement measureReplacementCharRatio**
|
||||
|
||||
Add to `TextProcessor.java` after `replaceUndefinedCharacters()` method (after line 53):
|
||||
|
||||
```java
|
||||
public static double measureReplacementCharRatio(List<IObject> contents) {
|
||||
char replacementChar = ChunkParser.REPLACEMENT_CHARACTER_STRING.charAt(0);
|
||||
int totalChars = 0;
|
||||
int replacementChars = 0;
|
||||
for (IObject object : contents) {
|
||||
if (object instanceof TextChunk) {
|
||||
String value = ((TextChunk) object).getValue();
|
||||
totalChars += value.length();
|
||||
for (int i = 0; i < value.length(); i++) {
|
||||
if (value.charAt(i) == replacementChar) {
|
||||
replacementChars++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (totalChars == 0) {
|
||||
return 0.0;
|
||||
}
|
||||
return (double) replacementChars / totalChars;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest -q`
|
||||
Expected: All tests PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextProcessor.java
|
||||
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextProcessorTest.java
|
||||
git commit -m "feat: add measureReplacementCharRatio to TextProcessor
|
||||
|
||||
Counts U+FFFD replacement characters across TextChunks and returns
|
||||
the ratio. Returns 0.0 for empty contents or pages with no text."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Chunk 2: Warning Log + Triage Routing
|
||||
|
||||
### Task 3: Add warning log in ContentFilterProcessor
|
||||
|
||||
**Files:**
|
||||
- Modify: `main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java`
|
||||
|
||||
- [ ] **Step 1: Add import for StaticLayoutContainers**
|
||||
|
||||
Add to imports:
|
||||
|
||||
```java
|
||||
import org.opendataloader.pdf.containers.StaticLayoutContainers;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add measurement + warning before replaceUndefinedCharacters**
|
||||
|
||||
Insert immediately before line 74 (`TextProcessor.replaceUndefinedCharacters(...)`) in `getFilteredContents()`:
|
||||
|
||||
```java
|
||||
double replacementCharRatio = TextProcessor.measureReplacementCharRatio(pageContents);
|
||||
StaticLayoutContainers.setReplacementCharRatio(pageNumber, replacementCharRatio);
|
||||
if (replacementCharRatio >= 0.3) {
|
||||
LOGGER.log(Level.WARNING,
|
||||
"Page {0}: {1,number,#.#%} of characters are replacement characters (U+FFFD). "
|
||||
+ "This PDF likely contains CID-keyed fonts without ToUnicode mappings. "
|
||||
+ "Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback.",
|
||||
new Object[]{pageNumber + 1, replacementCharRatio});
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Compile check**
|
||||
|
||||
Run: `cd java && mvn compile -pl opendataloader-pdf-core -q`
|
||||
Expected: BUILD SUCCESS
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java
|
||||
git commit -m "feat: detect CID font extraction failure and emit warning log
|
||||
|
||||
Measures U+FFFD ratio before replacement. Warns when >= 30% of
|
||||
characters are replacement characters, suggesting hybrid mode."
|
||||
```
|
||||
|
||||
### Task 4: Add Signal 0 to TriageProcessor
|
||||
|
||||
**Files:**
|
||||
- Modify: `main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java`
|
||||
- Test: `test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java`
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
Add to `TriageProcessorTest.java` before the `// Helper methods` comment:
|
||||
|
||||
```java
|
||||
@Test
|
||||
public void testClassifyPage_highReplacementRatio_routesToBackend() {
|
||||
StaticLayoutContainers.clearContainers();
|
||||
StaticLayoutContainers.setReplacementCharRatio(0, 0.5);
|
||||
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(createTextChunk(10, 100, 200, 120, "text"));
|
||||
|
||||
TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());
|
||||
|
||||
Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision());
|
||||
Assertions.assertEquals(1.0, result.getConfidence(), 0.001);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testClassifyPage_lowReplacementRatio_noEffect() {
|
||||
StaticLayoutContainers.clearContainers();
|
||||
StaticLayoutContainers.setReplacementCharRatio(0, 0.1);
|
||||
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(createTextChunk(10, 100, 200, 120, "normal text"));
|
||||
|
||||
TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());
|
||||
|
||||
Assertions.assertEquals(TriageDecision.JAVA, result.getDecision());
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testClassifyPage_exactThreshold_routesToBackend() {
|
||||
StaticLayoutContainers.clearContainers();
|
||||
StaticLayoutContainers.setReplacementCharRatio(0, 0.3);
|
||||
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(createTextChunk(10, 100, 200, 120, "text"));
|
||||
|
||||
TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());
|
||||
|
||||
Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision());
|
||||
Assertions.assertEquals(1.0, result.getConfidence(), 0.001);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest#testClassifyPage_highReplacementRatio_routesToBackend -q`
|
||||
Expected: FAIL — returns JAVA instead of BACKEND
|
||||
|
||||
- [ ] **Step 3: Add Signal 0 to classifyPage**
|
||||
|
||||
In `TriageProcessor.java`, in the `classifyPage()` method with `TriageThresholds` parameter, insert before the TableBorder check (before `// Signal 1: TableBorder presence`):
|
||||
|
||||
```java
|
||||
// Signal 0: CID font extraction failure (highest priority)
|
||||
// Only fires in hybrid mode (classifyPage is only called from HybridDocumentProcessor)
|
||||
double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber);
|
||||
if (replacementRatio >= 0.3) {
|
||||
return TriageResult.backend(pageNumber, 1.0, signals);
|
||||
}
|
||||
```
|
||||
|
||||
Add import at top of file:
|
||||
|
||||
```java
|
||||
import org.opendataloader.pdf.containers.StaticLayoutContainers;
|
||||
```
|
||||
|
||||
Also update the `classifyPage()` Javadoc (around line 617) to list Signal 0:
|
||||
|
||||
```
|
||||
* <p>Signal priority (highest to lowest):
|
||||
* <ol>
|
||||
* <li>CID font extraction failure (replacement char ratio >= 30%)</li>
|
||||
* <li>TableBorder presence</li>
|
||||
* ...
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest -q`
|
||||
Expected: All tests PASS
|
||||
|
||||
- [ ] **Step 5: Run full test suite**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -q`
|
||||
Expected: All tests PASS — no regressions
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java
|
||||
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java
|
||||
git commit -m "feat: add CID font detection signal to TriageProcessor
|
||||
|
||||
Signal 0 (highest priority): routes pages with >= 30% replacement
|
||||
characters to backend for OCR fallback in hybrid mode."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Chunk 3: e2e Test + Test Fixture
|
||||
|
||||
### Task 5: Generate synthetic CID PDF test fixture
|
||||
|
||||
**Files:**
|
||||
- Create: `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py`
|
||||
- Create: `java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf`
|
||||
|
||||
- [ ] **Step 1: Create test resources directory**
|
||||
|
||||
```bash
|
||||
mkdir -p java/opendataloader-pdf-core/src/test/resources
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Write the generation script**
|
||||
|
||||
Create `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py`.
|
||||
|
||||
This script generates a minimal PDF with a Type0 (CID) font that has no ToUnicode CMap. The PDF must cause veraPDF to emit `\uFFFD` for the majority of text characters when parsed.
|
||||
|
||||
Approach: Use raw PDF syntax to embed a CID font referencing CID values without a ToUnicode mapping. Alternatively, use `reportlab` which provides low-level CID font control.
|
||||
|
||||
The implementer should:
|
||||
1. Generate the PDF
|
||||
2. Verify it with opendataloader-pdf: `cd java && mvn exec:java -pl opendataloader-pdf-cli -Dexec.mainClass="org.opendataloader.pdf.cli.CLIMain" -Dexec.args="../src/test/resources/cid-font-no-tounicode.pdf -f text" 2>&1`
|
||||
3. Confirm output is mostly empty/whitespace (indicating `\uFFFD` → space replacement)
|
||||
4. Confirm WARNING log about replacement characters appears
|
||||
|
||||
If generating a proper CID PDF proves difficult, search `odl-test-fixtures` for an existing PDF with CID font issues, or use a known CID-problematic PDF from the Korean pharmaceutical fixtures (pdf-003 through pdf-007).
|
||||
|
||||
- [ ] **Step 3: Commit fixture**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf
|
||||
git add java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py
|
||||
git commit -m "test: add synthetic CID font PDF test fixture
|
||||
|
||||
PDF with CID-keyed font without ToUnicode mapping for testing
|
||||
replacement character detection. Generation script included."
|
||||
```
|
||||
|
||||
### Task 6: Write e2e integration test
|
||||
|
||||
**Files:**
|
||||
- Create: `test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java`
|
||||
|
||||
This test follows the pattern from `TriageProcessorIntegrationTest.java`:
|
||||
`DocumentProcessor.preprocessing()` → `ContentFilterProcessor.getFilteredContents()` → assert ratio and warning.
|
||||
|
||||
- [ ] **Step 1: Write the integration test**
|
||||
|
||||
Create `java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java`:
|
||||
|
||||
```java
|
||||
/*
|
||||
* Copyright 2025-2026 Hancom Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
package org.opendataloader.pdf.processors;
|
||||
|
||||
import org.junit.jupiter.api.Assertions;
|
||||
import org.junit.jupiter.api.Assumptions;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.opendataloader.pdf.api.Config;
|
||||
import org.opendataloader.pdf.containers.StaticLayoutContainers;
|
||||
import org.verapdf.wcag.algorithms.entities.IObject;
|
||||
import org.verapdf.wcag.algorithms.entities.content.TextChunk;
|
||||
import org.verapdf.wcag.algorithms.entities.geometry.BoundingBox;
|
||||
import org.verapdf.wcag.algorithms.semanticalgorithms.containers.StaticContainers;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.Paths;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.logging.Handler;
|
||||
import java.util.logging.Level;
|
||||
import java.util.logging.LogRecord;
|
||||
import java.util.logging.Logger;
|
||||
|
||||
/**
|
||||
* Integration test for CID font extraction failure detection.
|
||||
*
|
||||
* Tests the full pipeline: PDF parsing → ContentFilterProcessor →
|
||||
* measurement → StaticLayoutContainers storage → warning log.
|
||||
*/
|
||||
public class CidFontDetectionTest {
|
||||
|
||||
private static final Path CID_PDF_PATH = Paths.get(
|
||||
"src/test/resources/cid-font-no-tounicode.pdf");
|
||||
|
||||
private static boolean pdfAvailable = false;
|
||||
|
||||
@BeforeAll
|
||||
static void checkFixture() {
|
||||
pdfAvailable = Files.exists(CID_PDF_PATH) && Files.isRegularFile(CID_PDF_PATH);
|
||||
if (!pdfAvailable) {
|
||||
System.out.println("CID font test PDF not found: " + CID_PDF_PATH.toAbsolutePath());
|
||||
System.out.println("Skipping integration tests. Generate fixture first.");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCidPdf_highReplacementRatio_detected() throws IOException {
|
||||
Assumptions.assumeTrue(pdfAvailable, "CID font test PDF not available");
|
||||
|
||||
String pdfPath = CID_PDF_PATH.toAbsolutePath().toString();
|
||||
Config config = new Config();
|
||||
|
||||
DocumentProcessor.preprocessing(pdfPath, config);
|
||||
StaticLayoutContainers.clearContainers();
|
||||
|
||||
int numPages = StaticContainers.getDocument().getNumberOfPages();
|
||||
Assertions.assertTrue(numPages > 0, "PDF should have at least 1 page");
|
||||
|
||||
// Process page 0 through ContentFilterProcessor
|
||||
List<IObject> filteredContents = ContentFilterProcessor.getFilteredContents(
|
||||
pdfPath,
|
||||
StaticContainers.getDocument().getArtifacts(0),
|
||||
0,
|
||||
config
|
||||
);
|
||||
|
||||
// Verify ratio was stored
|
||||
double ratio = StaticLayoutContainers.getReplacementCharRatio(0);
|
||||
Assertions.assertTrue(ratio >= 0.3,
|
||||
"CID font PDF should have >= 30% replacement characters, got "
|
||||
+ String.format("%.1f%%", ratio * 100));
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testCidPdf_warningLogEmitted() throws IOException {
|
||||
Assumptions.assumeTrue(pdfAvailable, "CID font test PDF not available");
|
||||
|
||||
// Capture warning logs
|
||||
Logger logger = Logger.getLogger(ContentFilterProcessor.class.getCanonicalName());
|
||||
List<String> warnings = new ArrayList<>();
|
||||
Handler handler = new Handler() {
|
||||
@Override public void publish(LogRecord r) {
|
||||
if (r.getLevel() == Level.WARNING) {
|
||||
warnings.add(r.getMessage());
|
||||
}
|
||||
}
|
||||
@Override public void flush() {}
|
||||
@Override public void close() {}
|
||||
};
|
||||
logger.addHandler(handler);
|
||||
|
||||
try {
|
||||
String pdfPath = CID_PDF_PATH.toAbsolutePath().toString();
|
||||
Config config = new Config();
|
||||
|
||||
DocumentProcessor.preprocessing(pdfPath, config);
|
||||
StaticLayoutContainers.clearContainers();
|
||||
|
||||
ContentFilterProcessor.getFilteredContents(
|
||||
pdfPath,
|
||||
StaticContainers.getDocument().getArtifacts(0),
|
||||
0,
|
||||
config
|
||||
);
|
||||
|
||||
boolean hasReplacementWarning = warnings.stream()
|
||||
.anyMatch(w -> w.contains("replacement characters"));
|
||||
Assertions.assertTrue(hasReplacementWarning,
|
||||
"Expected WARNING log about replacement characters");
|
||||
} finally {
|
||||
logger.removeHandler(handler);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Unit-level boundary tests (no PDF fixture needed).
|
||||
*/
|
||||
@Test
|
||||
public void testBoundary_belowThreshold_29percent() {
|
||||
// 29 replacement chars out of 100 = 0.29 (below threshold)
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (int i = 0; i < 29; i++) sb.append('\uFFFD');
|
||||
for (int i = 0; i < 71; i++) sb.append('A');
|
||||
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 500.0, 20.0),
|
||||
sb.toString(), 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertTrue(ratio < 0.3,
|
||||
"29% should be below threshold, got " + ratio);
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testBoundary_atThreshold_30percent() {
|
||||
// 30 replacement chars out of 100 = 0.30 (at threshold)
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (int i = 0; i < 30; i++) sb.append('\uFFFD');
|
||||
for (int i = 0; i < 70; i++) sb.append('A');
|
||||
|
||||
List<IObject> contents = new ArrayList<>();
|
||||
contents.add(new TextChunk(new BoundingBox(1, 10.0, 10.0, 500.0, 20.0),
|
||||
sb.toString(), 10, 10.0));
|
||||
|
||||
double ratio = TextProcessor.measureReplacementCharRatio(contents);
|
||||
Assertions.assertTrue(ratio >= 0.3,
|
||||
"30% should be at threshold, got " + ratio);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=CidFontDetectionTest -q`
|
||||
Expected: Integration tests pass (or skip if fixture not yet generated). Boundary tests always pass.
|
||||
|
||||
- [ ] **Step 3: Run full test suite**
|
||||
|
||||
Run: `cd java && mvn test -pl opendataloader-pdf-core -q`
|
||||
Expected: All tests PASS
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java
|
||||
git commit -m "test: add integration + boundary tests for CID font detection
|
||||
|
||||
Integration tests load cid-font-no-tounicode.pdf through full pipeline.
|
||||
Boundary tests verify 29%/30% threshold behavior."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Chunk 4: Final Verification
|
||||
|
||||
### Task 7: Full test suite + benchmark regression check
|
||||
|
||||
- [ ] **Step 1: Run full Java test suite**
|
||||
|
||||
Run: `cd java && mvn test -q`
|
||||
Expected: All tests PASS across both core and cli modules
|
||||
|
||||
- [ ] **Step 2: Check benchmark regression (if benchmark script exists)**
|
||||
|
||||
Run: `./scripts/bench.sh --check-regression 2>/dev/null || echo "No benchmark script — skip"`
|
||||
Expected: PASS or skip
|
||||
|
||||
- [ ] **Step 3: Review all changes**
|
||||
|
||||
Run: `git log --oneline main..HEAD`
|
||||
Expected commits (newest first):
|
||||
1. `test: add e2e tests for CID font detection pipeline`
|
||||
2. `test: add synthetic CID font PDF test fixture`
|
||||
3. `feat: add CID font detection signal to TriageProcessor`
|
||||
4. `feat: detect CID font extraction failure and emit warning log`
|
||||
5. `feat: add measureReplacementCharRatio to TextProcessor`
|
||||
6. `feat: add per-page replacement char ratio storage to StaticLayoutContainers`
|
||||
|
||||
- [ ] **Step 4: Update issue #286 comment**
|
||||
|
||||
Update the previously posted enhancement comment with the actual fix status. The earlier `enhancement` label should be removed since this is now a fix.
|
||||
|
||||
```bash
|
||||
gh issue edit 286 --repo opendataloader-project/opendataloader-pdf --remove-label enhancement
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,151 @@
|
||||
# CID Font Extraction Failure Detection
|
||||
|
||||
Issue: [#286](https://github.com/opendataloader-project/opendataloader-pdf/issues/286)
|
||||
|
||||
## Problem
|
||||
|
||||
PDFs with CID-keyed fonts that lack ToUnicode mappings produce no usable text from veraPDF extraction. veraPDF replaces unmappable characters with U+FFFD (replacement character), which `TextProcessor.replaceUndefinedCharacters()` then converts to spaces. The result is empty or whitespace-only output with no indication to the user of what went wrong.
|
||||
|
||||
Users currently resort to external tools (e.g., pdfplumber) to pre-screen PDFs for CID issues before passing them to opendataloader-pdf.
|
||||
|
||||
## Solution
|
||||
|
||||
Detect pages with high replacement character ratios and:
|
||||
|
||||
1. **Always**: emit a WARNING log explaining the problem and suggesting `--hybrid-mode`
|
||||
2. **When hybrid mode is on**: automatically route affected pages to OCR backend via TriageProcessor
|
||||
|
||||
No new CLI options. Hybrid mode setting is respected as-is.
|
||||
|
||||
## Design
|
||||
|
||||
### Detection: Replacement Character Ratio
|
||||
|
||||
`TextProcessor.measureReplacementCharRatio(List<IObject>)` counts `\uFFFD` characters across all TextChunks on a page and returns the ratio (0.0–1.0).
|
||||
|
||||
**Threshold**: 30%. CID-affected pages typically show 90%+ replacement characters. 30% catches real problems while avoiding false positives from PDFs with occasional unmappable glyphs.
|
||||
|
||||
**Measurement point**: Inside `ContentFilterProcessor.getFilteredContents()`, immediately before `replaceUndefinedCharacters()` is called (line 74). At this point veraPDF has already inserted `\uFFFD` but the characters haven't been replaced with spaces yet, so measurement is accurate.
|
||||
|
||||
**Safety of measurement point**: The prior processing steps (`mergeCloseTextChunks`, `trimTextChunksWhiteSpaces`, `filterConsecutiveSpaces`, `splitTextChunksByWhiteSpaces`) do not affect U+FFFD characters. U+FFFD is not whitespace, so it is not trimmed, compressed, or used as a split boundary. The count is accurate at this position.
|
||||
|
||||
**Zero-text pages**: When a page has no TextChunk objects (e.g., image-only pages), the method returns 0.0 to avoid division by zero. This correctly avoids triggering the CID warning on non-text pages.
|
||||
|
||||
The method uses `ChunkParser.REPLACEMENT_CHARACTER_STRING` constant (not a hardcoded `"\uFFFD"` literal) to stay consistent with `replaceUndefinedCharacters()`.
|
||||
|
||||
### Data Flow
|
||||
|
||||
The measured ratio is stored in `StaticLayoutContainers` per page:
|
||||
|
||||
```
|
||||
ContentFilterProcessor.getFilteredContents()
|
||||
│
|
||||
├─ TextProcessor.measureReplacementCharRatio() → ratio
|
||||
├─ StaticLayoutContainers.setReplacementCharRatio(pageNumber, ratio)
|
||||
├─ if ratio >= 0.3: LOGGER.warning(...)
|
||||
└─ TextProcessor.replaceUndefinedCharacters() // existing call
|
||||
```
|
||||
|
||||
Note: `StaticLayoutContainers` currently stores global `ThreadLocal` scalars and lists, not per-page maps. Per-page data (e.g., bounding boxes) lives in `DocumentProcessor`. This change introduces a new per-page `Map<Integer, Double>` pattern to `StaticLayoutContainers`. We place it here rather than `DocumentProcessor` because it is layout-metadata consumed by `TriageProcessor`, keeping the triage data path self-contained. The existing `clearContainers()` method **must** be updated to clear this map to prevent cross-document data leakage in multi-document processing.
|
||||
|
||||
### Warning Log
|
||||
|
||||
Emitted from `ContentFilterProcessor` when ratio >= 0.3:
|
||||
|
||||
```
|
||||
WARNING: Page 3: 94% of characters are replacement characters (U+FFFD).
|
||||
This PDF likely contains CID-keyed fonts without ToUnicode mappings.
|
||||
Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback.
|
||||
```
|
||||
|
||||
This fires regardless of hybrid mode setting.
|
||||
|
||||
### Triage Routing
|
||||
|
||||
In `TriageProcessor.classifyPage()`, a new **Signal 0** is inserted before all existing signals (before TableBorder check). This signal only fires when hybrid mode is active, since `classifyPage()` is only called from `HybridDocumentProcessor`. In non-hybrid mode, only the warning log (from `ContentFilterProcessor`) is emitted:
|
||||
|
||||
```java
|
||||
double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber);
|
||||
if (replacementRatio >= 0.3) {
|
||||
return TriageResult.backend(pageNumber, 1.0, signals);
|
||||
}
|
||||
```
|
||||
|
||||
Priority is highest (confidence 1.0) because a page with mostly broken text extraction gains nothing from Java-path processing.
|
||||
|
||||
### Behavior Matrix
|
||||
|
||||
| Hybrid Mode | Ratio >= 30% | Result |
|
||||
|---|---|---|
|
||||
| OFF | Yes | Warning log. Java path produces incomplete text. |
|
||||
| OFF | No | No change. Normal processing. |
|
||||
| ON (auto) | Yes | Warning log + auto-route to BACKEND (OCR). |
|
||||
| ON (auto) | No | No change. Normal triage. |
|
||||
| ON (full) | Yes | Warning log. All pages already go to BACKEND. |
|
||||
| ON (full) | No | No change. All pages already go to BACKEND. |
|
||||
|
||||
## Changes
|
||||
|
||||
### Modified Files
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `TextProcessor.java` | Add `measureReplacementCharRatio()` static method |
|
||||
| `ContentFilterProcessor.java` | Call measurement before `replaceUndefinedCharacters()`, store result, emit warning |
|
||||
| `StaticLayoutContainers.java` | Add `replacementCharRatios` map with getter/setter, clear in `clearContainers()` |
|
||||
| `TriageProcessor.java` | Add Signal 0: replacement ratio check before TableBorder signal |
|
||||
|
||||
### New Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java` | e2e test using synthetic CID PDF |
|
||||
| `java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf` | Pre-generated test fixture (CID font, no ToUnicode) |
|
||||
| `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py` | Generation script for reference |
|
||||
|
||||
### Modified Test Files
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `TextProcessorTest.java` | 5 unit tests for `measureReplacementCharRatio()` |
|
||||
| `TriageProcessorTest.java` | 3 unit tests for Signal 0 routing |
|
||||
|
||||
## Test Plan
|
||||
|
||||
### Unit Tests (TextProcessorTest)
|
||||
|
||||
- `testMeasureReplacementCharRatio_allReplacement` — all U+FFFD → 1.0
|
||||
- `testMeasureReplacementCharRatio_noReplacement` — normal text → 0.0
|
||||
- `testMeasureReplacementCharRatio_mixed` — 30% U+FFFD → 0.3
|
||||
- `testMeasureReplacementCharRatio_emptyContents` — empty list → 0.0
|
||||
- `testMeasureReplacementCharRatio_nonTextChunksIgnored` — non-text objects skipped
|
||||
|
||||
### Unit Tests (TriageProcessorTest)
|
||||
|
||||
- `testClassifyPage_highReplacementRatio_routesToBackend` — ratio 0.5 → BACKEND
|
||||
- `testClassifyPage_lowReplacementRatio_noEffect` — ratio 0.1 → JAVA (default)
|
||||
- `testClassifyPage_exactThreshold_routesToBackend` — ratio 0.3 → BACKEND
|
||||
|
||||
### Boundary Tests
|
||||
|
||||
- `testWarningNotEmitted_belowThreshold` — ratio 0.29 → no warning log emitted
|
||||
- `testWarningEmitted_atThreshold` — ratio 0.30 → warning log emitted
|
||||
|
||||
### e2e Test (CidFontDetectionTest)
|
||||
|
||||
- Load pre-generated `cid-font-no-tounicode.pdf`
|
||||
- Run through `ContentFilterProcessor.getFilteredContents()`
|
||||
- Assert: `StaticLayoutContainers.getReplacementCharRatio(0) >= 0.3`
|
||||
- Assert: warning log contains "replacement characters"
|
||||
|
||||
### Benchmark Regression
|
||||
|
||||
- Existing benchmark PDFs are normal documents with near-zero replacement ratios
|
||||
- New logic does not affect existing test/benchmark results
|
||||
|
||||
## Not In Scope
|
||||
|
||||
- New CLI options (no `--cid-fallback` or similar)
|
||||
- `npm run sync` not required (no CLI option changes)
|
||||
- API signature changes (backward compatible)
|
||||
- Benchmark threshold changes
|
||||
@@ -0,0 +1,240 @@
|
||||
# Hybrid hancom-ai Backend Options & CLI Refactoring
|
||||
|
||||
Issue: not yet filed — this design precedes the issue/PRs.
|
||||
Scope: opendataloader-pdf (core CLI) + opendataloader-pdfua (downstream CLI)
|
||||
|
||||
## Problem
|
||||
|
||||
Five hancom-ai backend behaviors are already wired through `HybridConfig` but not exposed on any CLI:
|
||||
|
||||
| HybridConfig field | Default | Effect when set |
|
||||
|---|---|---|
|
||||
| `regionlistStrategy` | `table-first` | How to handle DLA label 7 (regionlist) |
|
||||
| `ocrStrategy` | `auto` | Stream-only / fallback / OCR-only |
|
||||
| `imageCache` | `memory` | Page image cache backing |
|
||||
| `saveCrops` | `false` | Persist cropped figures (debug) |
|
||||
| `cropOutputDir` | `null` | Output dir for saved crops (debug) |
|
||||
|
||||
`HancomAISchemaTransformer` and `HancomAIClient` already read these via `HybridConfig`, but `CLIOptions.java` has no flags and `applyHybridOptions()` never sets them. They are reachable only through programmatic `Config` use.
|
||||
|
||||
A second, structural problem makes adding these flags painful:
|
||||
|
||||
- **opendataloader-pdf** defines all CLI options in `CLIOptions.OPTION_DEFINITIONS` (private, single source for `options.json`/Python/Node bindings).
|
||||
- **opendataloader-pdfua** defines its own `--hybrid`, `--hybrid-url`, `--hybrid-mode` separately in `Main.java`, with different defaults, then forwards them through `RemediationConfig` (3 hybrid fields, 9 constructor overloads).
|
||||
|
||||
Adding 5 new flags to both CLIs the current way means duplicating definitions, expanding `RemediationConfig` to 8 hybrid fields, and creating yet more constructor overloads. The two CLIs will drift further apart.
|
||||
|
||||
## Solution
|
||||
|
||||
Two coupled changes, executed in order:
|
||||
|
||||
1. **Add the 5 hancom-ai-specific options** to core `CLIOptions` under a `--hybrid-hancom-ai-*` prefix, with a guard that rejects them when `--hybrid` is not `hancom-ai`.
|
||||
2. **Refactor core `CLIOptions` to be reusable** so opendataloader-pdfua imports the full core option set and adds only its own pdfua-specific options on top. `RemediationConfig` embeds a core `Config` instead of carrying parallel hybrid fields.
|
||||
|
||||
After this, adding any future core CLI option propagates to pdfua with a rebuild — no Main.java edits.
|
||||
|
||||
## Design
|
||||
|
||||
### Option naming: `--hybrid-hancom-ai-*` (full path)
|
||||
|
||||
Decision rationale (recorded for future contributors):
|
||||
|
||||
- `--hybrid-*` alone (e.g. `--hybrid-regionlist-strategy`) was rejected because docling-fast and other backends will never support these knobs; the name would lie about scope.
|
||||
- `--hancom-ai-*` alone (e.g. `--hancom-ai-regionlist-strategy`) was rejected because it breaks the existing `--hybrid-mode/url/timeout/fallback` grouping and gives users two mental models.
|
||||
- `--hybrid-hancom-ai-*` mirrors the `gh pr create` / `git remote add` full-path convention: every option name encodes the context it belongs to. `--help` alphabetic sort keeps all hybrid options in one block.
|
||||
- A single comma-separated `--hybrid-hancom-ai-config` mega-option was rejected: defeats Apache Commons CLI typo detection, breaks `options.json` codegen for Python/Node bindings, complicates Windows path escaping.
|
||||
|
||||
The five new options:
|
||||
|
||||
| Long option | Type | Default | Exported | Description |
|
||||
|---|---|---|---|---|
|
||||
| `--hybrid-hancom-ai-regionlist-strategy` | string | `table-first` | yes | DLA label 7 handling. Values: `table-first`, `list-only` |
|
||||
| `--hybrid-hancom-ai-ocr-strategy` | string | `auto` | yes | OCR strategy. Values: `off`, `auto`, `force` |
|
||||
| `--hybrid-hancom-ai-image-cache` | string | `memory` | yes | Page image cache. Values: `memory`, `disk` |
|
||||
| `--hybrid-hancom-ai-save-crops` | boolean | `false` | **no** | Persist cropped figures (debug only) |
|
||||
| `--hybrid-hancom-ai-crop-output-dir` | string | `null` | **no** | Output directory for `--hybrid-hancom-ai-save-crops` |
|
||||
|
||||
`exported=false` for the two debug options keeps them out of `options.json` and the auto-generated Python/Node bindings, while remaining usable on the Java CLI. Same pattern as existing legacy options at `CLIOptions.java:201-208`.
|
||||
|
||||
### Validation
|
||||
|
||||
Inside `applyHybridOptions()`, after `--hybrid` is parsed, before returning:
|
||||
|
||||
```java
|
||||
boolean usesHancomAiOnly =
|
||||
commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION) ||
|
||||
commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION) ||
|
||||
commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION) ||
|
||||
commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION) ||
|
||||
commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION);
|
||||
|
||||
if (usesHancomAiOnly && !Config.HYBRID_HANCOM_AI.equals(config.getHybrid())) {
|
||||
throw new IllegalArgumentException(
|
||||
"Options --hybrid-hancom-ai-* require --hybrid=hancom-ai");
|
||||
}
|
||||
```
|
||||
|
||||
Per-value validation (e.g. `regionlistStrategy` must be `table-first`/`list-only`) is already implemented in `HybridConfig` setters and re-thrown as `IllegalArgumentException`. CLI passes the raw value through; HybridConfig is the validation authority.
|
||||
|
||||
### Core CLIOptions refactoring
|
||||
|
||||
Goal: pdfua can register every core option in its own `Options` and ask core to apply them to a `Config`.
|
||||
|
||||
Two new public static methods on `CLIOptions`:
|
||||
|
||||
```java
|
||||
/** Register every core option onto an external Options. Used by downstream CLIs (pdfua). */
|
||||
public static void addAllTo(Options options) {
|
||||
for (OptionDefinition def : OPTION_DEFINITIONS) {
|
||||
options.addOption(def.toOption());
|
||||
}
|
||||
}
|
||||
|
||||
/** Apply parsed core options to a Config. Used by downstream CLIs after parse. */
|
||||
public static void applyAllTo(Config config, CommandLine commandLine) {
|
||||
// body identical to current createConfigFromCommandLine,
|
||||
// minus `new Config()` and minus the positional-arg output-folder fallback
|
||||
}
|
||||
```
|
||||
|
||||
The existing `defineOptions()` and `createConfigFromCommandLine()` keep their signatures and behavior; internally they delegate to the two new methods. Backward compatibility for any existing callers is preserved.
|
||||
|
||||
`OPTION_DEFINITIONS` stays private. `OptionDefinition` stays private. We expose only the two operations downstream CLIs actually need.
|
||||
|
||||
### pdfua/Main.java refactoring
|
||||
|
||||
Replace the self-defined hybrid option block (`Main.java:60-62, 99-101`) with:
|
||||
|
||||
```java
|
||||
Options options = new Options();
|
||||
CLIOptions.addAllTo(options); // all core options
|
||||
options.addOption(null, "lang", true, ...); // pdfua-specific only
|
||||
options.addOption(null, "audit-bundle-mode", true, ...);
|
||||
options.addOption(null, "font-embed-mode", true, ...);
|
||||
options.addOption(null, "conformance", true, ...);
|
||||
// ... other pdfua-only options
|
||||
|
||||
CommandLine cmd = parser.parse(options, args);
|
||||
|
||||
Config config = new Config();
|
||||
CLIOptions.applyAllTo(config, cmd);
|
||||
applyPdfuaDefaults(config); // pdfua's hybrid defaults
|
||||
```
|
||||
|
||||
### pdfua's hybrid defaults
|
||||
|
||||
pdfua currently hardcodes different defaults from core: `hybrid=hancom-ai`, `hybrid-url=http://localhost:18008`, `hybrid-mode=full` (`Main.java:60-62`). After refactoring, these become an explicit override block:
|
||||
|
||||
```java
|
||||
private static void applyPdfuaDefaults(Config config) {
|
||||
if (Config.HYBRID_OFF.equals(config.getHybrid())) {
|
||||
config.setHybrid(Config.HYBRID_HANCOM_AI);
|
||||
}
|
||||
if (config.getHybridConfig().getUrl() == null) {
|
||||
config.getHybridConfig().setUrl("http://localhost:18008");
|
||||
}
|
||||
if (Config.HYBRID_MODE_AUTO.equals(config.getHybridConfig().getMode())) {
|
||||
config.getHybridConfig().setMode(Config.HYBRID_MODE_FULL);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This makes pdfua's deviation from core defaults explicit and grep-able. User-supplied flags still win — the override only fires when the user did not specify a value.
|
||||
|
||||
### RemediationConfig refactoring (Hard break)
|
||||
|
||||
Current state: 3 flat hybrid fields (`hybrid`, `hybridUrl`, `hybridMode`) and 9 constructor overloads (`RemediationConfig.java:42-105`).
|
||||
|
||||
Target state: embed core `Config` directly. Keep only pdfua-specific fields. Builder replaces the constructor overloads.
|
||||
|
||||
```java
|
||||
public class RemediationConfig {
|
||||
private final Config coreConfig; // ← core options live here
|
||||
|
||||
// pdfua-only fields
|
||||
private final String input, output, lang;
|
||||
private final AuditBundleMode auditBundleMode;
|
||||
private final FontEmbedMode fontEmbedMode;
|
||||
private final List<String> conformances;
|
||||
private final int threads;
|
||||
private final boolean enrichPictureDescription;
|
||||
|
||||
private RemediationConfig(Builder b) { ... }
|
||||
|
||||
public Config getCoreConfig() { return coreConfig; }
|
||||
public String getHybrid() { return coreConfig.getHybrid(); }
|
||||
public HybridConfig getHybridConfig() { return coreConfig.getHybridConfig(); }
|
||||
// ... pdfua-only getters
|
||||
|
||||
public static Builder builder() { return new Builder(); }
|
||||
public static class Builder { ... }
|
||||
}
|
||||
```
|
||||
|
||||
`getHybridUrl()` and `getHybridMode()` are **removed** (Hard break). All call sites move to `getHybridConfig().getUrl()` / `getHybridConfig().getMode()`. Verified call sites are confined to test code (`AuditBundleEmitterTest`, `CertificateIssuerTest`, `AuditManifestBuilderTest`, `RemediationConfigAuditBundleTest`) and `RemediationProcessor.java:165-170`. No external SDK consumers identified.
|
||||
|
||||
`RemediationProcessor.java:163-170` simplifies to:
|
||||
|
||||
```java
|
||||
Config config = remediationConfig.getCoreConfig();
|
||||
// (no per-field copy needed — Config already carries all hybrid state)
|
||||
```
|
||||
|
||||
### Files touched
|
||||
|
||||
**opendataloader-pdf (core)**
|
||||
|
||||
- `java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java`
|
||||
- Add 5 option constants (long names + descriptions)
|
||||
- Add 5 entries to `OPTION_DEFINITIONS` (3 exported, 2 not)
|
||||
- Extend `applyHybridOptions()`: parse, set on `HybridConfig`, validate hancom-ai gate
|
||||
- Add public `addAllTo(Options)` and `applyAllTo(Config, CommandLine)`
|
||||
- `defineOptions()` and `createConfigFromCommandLine()` delegate to the new methods
|
||||
- `java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java`
|
||||
- Tests for parsing each new option
|
||||
- Tests for the hancom-ai gate (using flag without `--hybrid=hancom-ai` throws)
|
||||
- After core changes: run `npm run sync` to regenerate `options.json` + Python/Node bindings
|
||||
|
||||
**opendataloader-pdfua**
|
||||
|
||||
- `src/main/java/org/opendataloader/pdf/Main.java`
|
||||
- Replace self-defined hybrid options with `CLIOptions.addAllTo(options)`
|
||||
- Add `applyPdfuaDefaults(config)` helper
|
||||
- Pass `coreConfig` to `RemediationConfig.Builder` instead of individual hybrid strings
|
||||
- `src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java`
|
||||
- Embed `Config coreConfig`, drop `hybrid/hybridUrl/hybridMode` fields
|
||||
- Replace 9 constructors with one `Builder`
|
||||
- Update getters: `getHybrid()` delegates, `getHybridUrl()`/`getHybridMode()` removed
|
||||
- `src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java`
|
||||
- Simplify the lines 163-170 hybrid forwarding block
|
||||
- All test files instantiating `RemediationConfig` (5 files identified): switch to `Builder`
|
||||
|
||||
### Build & verification
|
||||
|
||||
Per `opendataloader-pdfua/CLAUDE.md`:
|
||||
|
||||
1. Core changes first: `cd opendataloader-pdf/java && mvn install -DskipTests`
|
||||
2. Then pdfua: `cd opendataloader-pdfua && mvn clean package`
|
||||
3. Run both test suites
|
||||
4. Manually verify:
|
||||
- `opendataloader-pdf input.pdf --hybrid=hancom-ai --hybrid-hancom-ai-regionlist-strategy=list-only` works
|
||||
- `opendataloader-pdf input.pdf --hybrid-hancom-ai-regionlist-strategy=list-only` (no `--hybrid`) fails with clear error
|
||||
- `opendataloader-pdfua input.pdf --hybrid-hancom-ai-ocr-strategy=force` works (inherits core option, pdfua defaults still apply)
|
||||
|
||||
### Phasing
|
||||
|
||||
The work splits cleanly into independent commits/PRs:
|
||||
|
||||
1. **PR 1 (core)**: Add 5 options + gate validation + tests + `npm run sync`. Self-contained, mergeable alone.
|
||||
2. **PR 2 (core)**: Extract `addAllTo` / `applyAllTo` public API. Pure refactoring, no behavior change.
|
||||
3. **PR 3 (pdfua)**: Switch `Main.java` to use core's API; refactor `RemediationConfig` to Builder + embedded `Config`; update tests.
|
||||
|
||||
PR 1 unblocks immediate user value. PRs 2 & 3 are coupled (PR 3 depends on PR 2 merge → core artifact rebuild) but neither blocks PR 1.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Migrating other parts of pdfua to a Builder-style config (only `RemediationConfig` is touched).
|
||||
- Adding flags to docling-fast or other hybrid backends. Spec is hancom-ai-specific.
|
||||
- Changing `HybridConfig` setter validation (already strict).
|
||||
- Renaming or repackaging existing `--hybrid-*` options.
|
||||
- Documentation site updates — `opendataloader.org` reference docs are CI-generated from `options.json` after core merge.
|
||||
Reference in New Issue
Block a user