- 
+ 
-PDF-Extract-Kit: High-Quality PDF Extraction Toolkit🔥🔥🔥 +🚀Access MinerU Now→✅ Zero-Install Web Version ✅ Full-Featured Desktop Client ✅ Instant API Access; Skip deployment headaches – get all product formats in one click. Developers, dive in!
- 👋 join us on Discord and WeChat + 👋 join us on Discord and WeChat
-
-
+
-### Online Demo
-
-Stable Version (Stable version verified by QA):
-[](https://opendatalab.com/OpenSourceTools/Extractor/PDF)
+1 Accuracy metrics are the End-to-End Evaluation Overall scores from OmniDocBench (v1.5), based on the latest version of `MinerU`.
+2 Servers compatible with OpenAI API, such as local model servers or remote model services deployed via inference frameworks like `vLLM`/`SGLang`/`LMDeploy`.
+3 Linux only supports distributions from 2019 and later.
+4 Since the key dependency `ray` does not support Python 3.13 on Windows, only versions 3.10~3.12 are supported.
+5 macOS requires version 14.0 or later.
-Test Version (Synced with dev branch updates, testing new features):
-[](https://huggingface.co/spaces/opendatalab/MinerU)
-[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
-### Quick CPU Demo
-
-#### 1. Install magic-pdf
+### Install MinerU
+#### Install MinerU using pip or uv
```bash
-conda create -n MinerU python=3.10
-conda activate MinerU
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com
+pip install --upgrade pip
+pip install uv
+uv pip install -U "mineru[all]"
```
-#### 2. Download model weight files
-
-Refer to [How to Download Model Files](docs/how_to_download_models_en.md) for detailed instructions.
-
-#### 3. Modify the Configuration File for Additional Configuration
-
-After completing the [2. Download model weight files](#2-download-model-weight-files) step, the script will automatically generate a `magic-pdf.json` file in the user directory and configure the default model path.
-You can find the `magic-pdf.json` file in your 【user directory】.
-
-> The user directory for Windows is "C:\\Users\\username", for Linux it is "/home/username", and for macOS it is "/Users/username".
-
-You can modify certain configurations in this file to enable or disable features, such as table recognition:
-
-> If the following items are not present in the JSON, please manually add the required items and remove the comment content (standard JSON does not support comments).
-
-```json
-{
- // other config
- "layout-config": {
- "model": "layoutlmv3" // Please change to "doclayout_yolo" when using doclayout_yolo.
- },
- "formula-config": {
- "mfd_model": "yolo_v8_mfd",
- "mfr_model": "unimernet_small",
- "enable": true // The formula recognition feature is enabled by default. If you need to disable it, please change the value here to "false".
- },
- "table-config": {
- "model": "tablemaster", // When using structEqTable, please change to "struct_eqtable".
- "enable": false, // The table recognition feature is disabled by default. If you need to enable it, please change the value here to "true".
- "max_time": 400
- }
-}
+#### Install MinerU from source code
+```bash
+git clone https://github.com/opendatalab/MinerU.git
+cd MinerU
+uv pip install -e .[all]
```
-### Using GPU
+> [!TIP]
+> `mineru[all]` includes all core features, compatible with Windows / Linux / macOS systems, suitable for most users.
+> If you need to specify the inference framework for the VLM model, or only intend to install a lightweight client on an edge device, please refer to the documentation [Extension Modules Installation Guide](https://opendatalab.github.io/MinerU/quick_start/extension_modules/).
-If your device supports CUDA and meets the GPU requirements of the mainline environment, you can use GPU acceleration. Please select the appropriate guide based on your system:
+---
+
+#### Deploy MinerU using Docker
+MinerU provides a convenient Docker deployment method, which helps quickly set up the environment and solve some tricky environment compatibility issues.
+You can get the [Docker Deployment Instructions](https://opendatalab.github.io/MinerU/quick_start/docker_deployment/) in the documentation.
-- [Ubuntu 22.04 LTS + GPU](docs/README_Ubuntu_CUDA_Acceleration_en_US.md)
-- [Windows 10/11 + GPU](docs/README_Windows_CUDA_Acceleration_en_US.md)
-- Quick Deployment with Docker
- > Docker requires a GPU with at least 16GB of VRAM, and all acceleration features are enabled by default.
- >
- > Before running this Docker, you can use the following command to check if your device supports CUDA acceleration on Docker.
- >
- > ```bash
- > docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
- > ```
- ```bash
- wget https://github.com/opendatalab/MinerU/raw/master/Dockerfile
- docker build -t mineru:latest .
- docker run --rm -it --gpus=all mineru:latest /bin/bash
- magic-pdf --help
- ```
+---
-## Usage
+### Using MinerU
-### Command Line
+If your device meets the GPU acceleration requirements in the table above, you can use a simple command line for document parsing:
```bash
-magic-pdf --help
-Usage: magic-pdf [OPTIONS]
-
-Options:
- -v, --version display the version and exit
- -p, --path PATH local pdf filepath or directory [required]
- -o, --output-dir PATH output local directory [required]
- -m, --method [ocr|txt|auto] the method for parsing pdf. ocr: using ocr
- technique to extract information from pdf. txt:
- suitable for the text-based pdf only and
- outperform ocr. auto: automatically choose the
- best method for parsing pdf from ocr and txt.
- without method specified, auto will be used by
- default.
- -l, --lang TEXT Input the languages in the pdf (if known) to
- improve OCR accuracy. Optional. You should
- input "Abbreviation" with language form url: ht
- tps://paddlepaddle.github.io/PaddleOCR/latest/en
- /ppocr/blog/multi_languages.html#5-support-languages-
- and-abbreviations
- -d, --debug BOOLEAN Enables detailed debugging information during
- the execution of the CLI commands.
- -s, --start INTEGER The starting page for PDF parsing, beginning
- from 0.
- -e, --end INTEGER The ending page for PDF parsing, beginning from
- 0.
- --help Show this message and exit.
-
-
-## show version
-magic-pdf -v
-
-## command line example
-magic-pdf -p {some_pdf} -o {some_output_dir} -m auto
-```
-
-`{some_pdf}` can be a single PDF file or a directory containing multiple PDFs.
-The results will be saved in the `{some_output_dir}` directory. The output file list is as follows:
-
-```text
-├── some_pdf.md # markdown file
-├── images # directory for storing images
-├── some_pdf_layout.pdf # layout diagram (Include layout reading order)
-├── some_pdf_middle.json # MinerU intermediate processing result
-├── some_pdf_model.json # model inference result
-├── some_pdf_origin.pdf # original PDF file
-├── some_pdf_spans.pdf # smallest granularity bbox position information diagram
-└── some_pdf_content_list.json # Rich text JSON arranged in reading order
-```
-
-For more information about the output files, please refer to the [Output File Description](docs/output_file_en_us.md).
-
-### API
-
-Processing files from local disk
-
-```python
-image_writer = DiskReaderWriter(local_image_dir)
-image_dir = str(os.path.basename(local_image_dir))
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
-pipe.pipe_classify()
-pipe.pipe_analyze()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
+mineru -p -o
```
-
-Processing files from object storage
-
-```python
-s3pdf_cli = S3ReaderWriter(pdf_ak, pdf_sk, pdf_endpoint)
-image_dir = "s3://img_bucket/"
-s3image_cli = S3ReaderWriter(img_ak, img_sk, img_endpoint, parent_path=image_dir)
-pdf_bytes = s3pdf_cli.read(s3_pdf_path, mode=s3pdf_cli.MODE_BIN)
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, s3image_cli)
-pipe.pipe_classify()
-pipe.pipe_analyze()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
+If your device does not meet the GPU acceleration requirements, you can specify the backend as `pipeline` to run in a pure CPU environment:
+```bash
+mineru -p -o -b pipeline
```
-For detailed implementation, refer to:
-
-- [demo.py Simplest Processing Method](demo/demo.py)
-- [magic_pdf_parse_main.py More Detailed Processing Workflow](demo/magic_pdf_parse_main.py)
-
-### Deploy Derived Projects
-
-Derived projects include secondary development projects based on MinerU by project developers and community developers,
-such as application interfaces based on Gradio, RAG based on llama, web demos similar to the official website, lightweight multi-GPU load balancing client/server ends, etc.
-These projects may offer more features and a better user experience.
-For specific deployment methods, please refer to the [Derived Project README](projects/README.md)
-
-
-### Development Guide
-
-TODO
+`mineru` currently supports local `PDF`, image, `DOCX`, `PPTX`, and `XLSX` file or directory inputs, and can be used for document parsing through the CLI, API, WebUI, and `mineru-router`. For detailed instructions, please refer to the [Usage Guide](https://opendatalab.github.io/MinerU/usage/).
# TODO
-- 🗹 Reading order based on the model
-- 🗹 Recognition of `index` and `list` in the main text
-- 🗹 Table recognition
-- ☐ Code block recognition in the main text
-- ☐ [Chemical formula recognition](docs/chemical_knowledge_introduction/introduction.pdf)
-- ☐ Geometric shape recognition
+- [x] Reading order based on the model
+- [x] Recognition of `index` and `list` in the main text
+- [x] Table recognition
+- [x] Heading Classification
+- [x] Handwritten Text Recognition
+- [x] Vertical Text Recognition
+- [x] Latin Accent Mark Recognition
+- [x] Code block recognition in the main text
+- [x] [Chemical formula recognition](docs/chemical_knowledge_introduction/introduction.pdf)(mineru.net)
+- [ ] Geometric shape recognition
# Known Issues
- Reading order is determined by the model based on the spatial distribution of readable content, and may be out of order in some areas under extremely complex layouts.
-- Vertical text is not supported.
+- Limited support for vertical text.
- Tables of contents and lists are recognized through rules, and some uncommon list formats may not be recognized.
-- Only one level of headings is supported; hierarchical headings are not currently supported.
- Code blocks are not yet supported in the layout model.
- Comic books, art albums, primary school textbooks, and exercises cannot be parsed well.
- Table recognition may result in row/column recognition errors in complex tables.
@@ -394,9 +320,9 @@ TODO
# FAQ
-[FAQ in Chinese](docs/FAQ_zh_cn.md)
-
-[FAQ in English](docs/FAQ_en_us.md)
+- If you encounter any issues during usage, you can first check the [FAQ](https://opendatalab.github.io/MinerU/faq/) for solutions.
+- If your issue remains unresolved, you may also use [DeepWiki](https://deepwiki.com/opendatalab/MinerU) to interact with an AI assistant, which can address most common problems.
+- If you still cannot resolve the issue, you are welcome to join our community via [Discord](https://discord.gg/Tdedn9GTXq) or [WeChat](https://mineru.net/community-portal/?aliasId=3c430f94) to discuss with other users and developers.
# All Thanks To Our Contributors
@@ -406,31 +332,52 @@ TODO
# License Information
-[LICENSE.md](LICENSE.md)
-
-This project currently uses PyMuPDF to achieve advanced functionality. However, since it adheres to the AGPL license, it may impose restrictions on certain usage scenarios. In future iterations, we plan to explore and replace it with a more permissive PDF processing library to enhance user-friendliness and flexibility.
+This repository is licensed under the [MinerU Open Source License](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md), based on Apache 2.0 with additional conditions.
# Acknowledgments
-- [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit)
-- [StructEqTable](https://github.com/UniModal4Reasoning/StructEqTable-Deploy)
+- [UniMERNet](https://github.com/opendatalab/UniMERNet)
+- [TableStructureRec](https://github.com/RapidAI/TableStructureRec)
- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
-- [PyMuPDF](https://github.com/pymupdf/PyMuPDF)
-- [layoutreader](https://github.com/ppaanngggg/layoutreader)
+- [PaddleOCR2Pytorch](https://github.com/frotms/PaddleOCR2Pytorch)
- [fast-langdetect](https://github.com/LlmKira/fast-langdetect)
+- [pypdfium2](https://github.com/pypdfium2-team/pypdfium2)
+- [pdftext](https://github.com/datalab-to/pdftext)
- [pdfminer.six](https://github.com/pdfminer/pdfminer.six)
+- [pypdf](https://github.com/py-pdf/pypdf)
+- [magika](https://github.com/google/magika)
+- [vLLM](https://github.com/vllm-project/vllm)
+- [LMDeploy](https://github.com/InternLM/lmdeploy)
# Citation
```bibtex
-@misc{wang2024mineruopensourcesolutionprecise,
- title={MinerU: An Open-Source Solution for Precise Document Content Extraction},
- author={Bin Wang and Chao Xu and Xiaomeng Zhao and Linke Ouyang and Fan Wu and Zhiyuan Zhao and Rui Xu and Kaiwen Liu and Yuan Qu and Fukai Shang and Bo Zhang and Liqun Wei and Zhihao Sui and Wei Li and Botian Shi and Yu Qiao and Dahua Lin and Conghui He},
- year={2024},
- eprint={2409.18839},
- archivePrefix={arXiv},
- primaryClass={cs.CV},
- url={https://arxiv.org/abs/2409.18839},
+@article{wang2026mineru2,
+ title={MinerU2. 5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale},
+ author={Wang, Bin and He, Tianyao and Ouyang, Linke and Wu, Fan and Zhao, Zhiyuan and Chu, Tao and Qu, Yuan and Jin, Zhenjiang and Zeng, Weijun and Miao, Ziyang and others},
+ journal={arXiv preprint arXiv:2604.04771},
+ year={2026}
+}
+
+@article{dong2026minerudiffusion,
+ title={MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding},
+ author={Dong, Hejun and Niu, Junbo and Wang, Bin and Zeng, Weijun and Zhang, Wentao and He, Conghui},
+ journal={arXiv preprint arXiv:2603.22458},
+ year={2026}
+}
+
+@article{niu2025mineru2,
+ title={Mineru2. 5: A decoupled vision-language model for efficient high-resolution document parsing},
+ author={Niu, Junbo and Liu, Zheng and Gu, Zhuangcheng and Wang, Bin and Ouyang, Linke and Zhao, Zhiyuan and Chu, Tao and He, Tianyao and Wu, Fan and Zhang, Qintong and others},
+ journal={arXiv preprint arXiv:2509.22186},
+ year={2025}
+}
+
+@article{wang2024mineru,
+ title={Mineru: An open-source solution for precise document content extraction},
+ author={Wang, Bin and Xu, Chao and Zhao, Xiaomeng and Ouyang, Linke and Wu, Fan and Zhao, Zhiyuan and Xu, Rui and Liu, Kaiwen and Qu, Yuan and Shang, Fukai and others},
+ journal={arXiv preprint arXiv:2409.18839},
+ year={2024}
}
@article{he2024opendatalab,
@@ -451,16 +398,15 @@ This project currently uses PyMuPDF to achieve advanced functionality. However,
-# Magic-doc
-
-[Magic-Doc](https://github.com/InternLM/magic-doc) Fast speed ppt/pptx/doc/docx/pdf extraction tool
-
-# Magic-html
-
-[Magic-HTML](https://github.com/opendatalab/magic-html) Mixed web page extraction tool
# Links
-
+- [MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding](https://github.com/opendatalab/MinerU-Diffusion)
+- [Easy Data Preparation with latest LLMs-based Operators and Pipelines](https://github.com/OpenDCAI/DataFlow)
+- [Vis3 (OSS browser based on s3)](https://github.com/opendatalab/Vis3)
- [LabelU (A Lightweight Multi-modal Data Annotation Tool)](https://github.com/opendatalab/labelU)
- [LabelLLM (An Open-source LLM Dialogue Annotation Platform)](https://github.com/opendatalab/LabelLLM)
- [PDF-Extract-Kit (A Comprehensive Toolkit for High-Quality PDF Content Extraction)](https://github.com/opendatalab/PDF-Extract-Kit)
+- [OmniDocBench (A Comprehensive Benchmark for Document Parsing and Evaluation)](https://github.com/opendatalab/OmniDocBench)
+- [Magic-HTML (Mixed web page extraction tool)](https://github.com/opendatalab/magic-html)
+- [Magic-Doc (Fast speed ppt/pptx/doc/docx/pdf extraction tool)](https://github.com/InternLM/magic-doc)
+- [Dingo: A Comprehensive AI Data Quality Evaluation Tool](https://github.com/MigoXLab/dingo)
diff --git a/README_ja-JP.md b/README_ja-JP.md
deleted file mode 100644
index 185db18d1..000000000
--- a/README_ja-JP.md
+++ /dev/null
@@ -1,324 +0,0 @@
-
-
-
-
-# ライセンス情報
-
-[LICENSE.md](LICENSE.md)
-
-このプロジェクトは現在、PyMuPDFを利用して高度な機能を提供していますが、AGPLライセンスに準拠しているため、特定の使用ケースに制限を課す可能性があります。今後のバージョンでは、より寛容なライセンスのPDF処理ライブラリへの移行を検討し、ユーザーフレンドリーさと柔軟性を向上させる予定です。
-
-
-# 謝辞
-
-- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
-- [PyMuPDF](https://github.com/pymupdf/PyMuPDF)
-- [fast-langdetect](https://github.com/LlmKira/fast-langdetect)
-- [pdfminer.six](https://github.com/pdfminer/pdfminer.six)
-
-
-# 引用
-
-```bibtex
-@misc{wang2024mineruopensourcesolutionprecise,
- title={MinerU: An Open-Source Solution for Precise Document Content Extraction},
- author={Bin Wang and Chao Xu and Xiaomeng Zhao and Linke Ouyang and Fan Wu and Zhiyuan Zhao and Rui Xu and Kaiwen Liu and Yuan Qu and Fukai Shang and Bo Zhang and Liqun Wei and Zhihao Sui and Wei Li and Botian Shi and Yu Qiao and Dahua Lin and Conghui He},
- year={2024},
- eprint={2409.18839},
- archivePrefix={arXiv},
- primaryClass={cs.CV},
- url={https://arxiv.org/abs/2409.18839},
-}
-
-@article{he2024opendatalab,
- title={Opendatalab: Empowering general artificial intelligence with open datasets},
- author={He, Conghui and Li, Wei and Jin, Zhenjiang and Xu, Chao and Wang, Bin and Lin, Dahua},
- journal={arXiv preprint arXiv:2407.13773},
- year={2024}
-}
-```
-
-# スター履歴
-
-
-
- 
-
-
-
-# リンク
-- [LabelU (軽量なマルチモーダルデータアノテーションツール)](https://github.com/opendatalab/labelU)
-- [LabelLLM (オープンソースのLLM対話アノテーションプラットフォーム)](https://github.com/opendatalab/LabelLLM)
-- [PDF-Extract-Kit (高品質なPDFコンテンツ抽出のための包括的なツールキット)](https://github.com/opendatalab/PDF-Extract-Kit)
diff --git a/README_zh-CN.md b/README_zh-CN.md
index bf6e802bd..2226996b0 100644
--- a/README_zh-CN.md
+++ b/README_zh-CN.md
@@ -1,7 +1,7 @@
@@ -29,379 +33,302 @@
-# 更新记录
+
Table of Contents
- -
-
-
- MinerU
-
-
-
- Project Introduction -
- Key Features -
- Quick Start
-
-
-
- Online Demo -
- Quick CPU Demo -
- Using GPU -
- - Usage - - -
- - TODO -
- Known Issues -
- FAQ -
- All Thanks To Our Contributors -
- License Information -
- Acknowledgments -
- Citation -
- Star History -
- Magic-doc -
- Magic-html -
- Links -
+
+MCP Server · LangChain / Dify / FastGPT native integration · 10+ domestic AI chip support + +**🔍 Core Parsing Capabilities** + +- Native support for `DOCX`, `PPTX`, and `XLSX` parsing +- Formulas → LaTeX · Tables → HTML, accurate layout reconstruction +- Supports scanned docs, handwriting, multi-column layouts, cross-page table merging +- Output follows human reading order with automatic header/footer removal +- VLM + OCR dual engine, 109-language OCR recognition + +**🔌 Integration** + +| Use Case | Solution | +|----------|----------| +| AI Coding Tools | MCP Server — Cursor · Claude Desktop · Windsurf | +| RAG Frameworks | LangChain · LlamaIndex · RAGFlow · RAG-Anything · Flowise · Dify · FastGPT | +| Development | Python / Go / TypeScript SDK · CLI · REST API · Docker | +| No-Code | mineru.net online · Gradio WebUI · Desktop client | + +**🖥️ Deployment (Private · Fully Offline)** + +| Inference Backend | Best For | +|------------------|---------| +| pipeline | Fast & stable, no hallucination, runs on CPU or GPU | +| vlm-engine | High accuracy, supports vLLM / LMDeploy / mlx ecosystem | +| hybrid-engine | High accuracy, native text extraction, low hallucination | + +Domestic AI chips: Ascend · Cambricon · Enflame · MetaX · Moore Threads · Kunlunxin · Iluvatar · Hygon · Biren · T-Head
+# Changelog
+
+- 2026/04/18 3.1.0 Released
+
+ This release focuses on **licensing openness, parsing accuracy, and full-format native support**. The main updates include:
+
+ - License upgrade
+ - MinerU has officially moved from `AGPLv3` to the [MinerU Open Source License](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md), a custom license based on `Apache 2.0`.
+ - This change significantly reduces adoption friction for both community users and commercial deployments, making MinerU easier to integrate into real-world workflows.
+ - VLM main model upgrade
+ - The primary VLM model has been upgraded to `MinerU2.5-Pro-2604-1.2B`, bringing overall parsing accuracy to a state-of-the-art level.
+ - The new model now supports image and chart parsing, truncated paragraph merging, cross-page table merging, and image recognition inside tables, further strengthening performance on complex document layouts.
+ - Full-format native parsing support
+ - Native parsing support has now been extended to `PPTX` and `XLSX`.
+ - MinerU now fully supports parsing across images, `PDF`, `DOCX`, `PPTX`, and `XLSX`, providing a more complete multi-format document understanding workflow.
+
+ With the 3.1.0 release, MinerU becomes more open, more accurate, and easier to adopt in production. The new license lowers the barrier for both community and commercial use, `MinerU2.5-Pro-2604-1.2B` improves parsing quality on complex content, and native `PPTX` / `XLSX` support completes end-to-end coverage of mainstream document formats.
+
+- 2026/03/29 3.0.0 Released
+
+ This release delivers a systematic upgrade centered on **parsing capability, system architecture, and engineering usability**. The main updates include:
+
+ - Native `DOCX` parsing
+ - Official support for native `DOCX` parsing, delivering high-precision results without hallucinations.
+ - Compared with the traditional workflow of first converting `DOCX` to `PDF` and then parsing it, end-to-end speed is improved by tens of times, making it better suited for scenarios with high requirements for both accuracy and throughput.
+ - `pipeline` backend upgrade
+ - The `pipeline` backend achieves a score of `86.2` on OmniDocBench (v1.5), surpassing the accuracy of the previous-generation mainstream VLM `MinerU2.0-2505-0.9B`.
+ - Added support for parsing images/formulas inside tables, seal text recognition, vertical text support, and interline formula numbering recognition, continuously improving parsing quality for complex document scenarios.
+ - While maintaining high accuracy, it keeps resource usage extremely low and continues to support inference in pure CPU environments.
+ - `API / CLI / Router` orchestration upgrade
+ - `mineru` now runs as an orchestration client based on `mineru-api`; when `--api-url` is not provided, it will automatically start a local temporary service.
+ - `mineru-api` adds a new asynchronous task endpoint `POST /tasks`, supporting task submission, status querying, and result retrieval; meanwhile, it retains the synchronous parsing endpoint `POST /file_parse` for compatibility with legacy plugins.
+ - Added `mineru-router`, designed for unified entry deployment and task routing across multiple services and multiple GPUs; its interfaces are fully compatible with `mineru-api` and support automatic task load balancing.
+ - Deployment and usability improvements
+ - Resolved compatibility issues with `torch >= 2.8`; the base image has been upgraded to `vllm0.11.2 + torch2.9.0`, unifying installation paths across different Compute Capabilities.
+ - Optimized the parsing pipeline with a sliding-window mechanism, significantly reducing peak memory usage in long-document scenarios, so documents with tens of thousands of pages no longer need to be split manually.
+ - Batch inference in `pipeline` now supports streaming writes to disk, allowing completed parsing results to be written out in time and further improving the experience for long-running tasks.
+ - Completed thread-safety optimization and now fully supports multi-threaded concurrent inference; together with `mineru-router`, this enables one-click multi-GPU deployment and makes it easy to build high-concurrency, high-throughput parsing systems.
+ - Completely removed the use of two AGPLv3 models (`doclayoutyolo` and `mfd_yolov8`) and one CC-BY-NC-SA 4.0 model (`layoutreader`).
+
+ This update is not just a set of feature enhancements, but a key leap forward in MinerU's overall system capabilities. We specifically addressed the peak memory usage issue in long-document parsing. Through optimizations such as sliding windows and streaming writes to disk, ultra-long document parsing has moved from “requiring manual splitting and careful handling” to being “stable, scalable, and ready for production workloads.” At the same time, we completed thread-safety optimization and fully enabled multi-threaded concurrent inference, further improving single-machine resource utilization and runtime stability under high-concurrency workloads. On top of this, with `mineru-router` and the new `API / CLI` orchestration framework, MinerU now supports one-click multi-GPU deployment, unified access across multiple services, and automatic task load balancing, significantly reducing the difficulty of large-scale deployment. As a result, MinerU is evolving from a standalone data production tool into a large-scale document parsing foundation for high-concurrency and high-throughput scenarios, providing enterprise-grade document data processing with infrastructure that is more stable, more efficient, and easier to scale.
+
+> 📝 View the complete [Changelog](https://opendatalab.github.io/MinerU/reference/changelog/) for more historical version information
+
# MinerU
## Project Introduction
-MinerU is a tool that converts PDFs into machine-readable formats (e.g., markdown, JSON), allowing for easy extraction into any format.
+MinerU is a document parsing tool that converts `PDF`, image, `DOCX`, `PPTX`, and `XLSX` inputs into machine-readable formats such as Markdown and JSON for downstream retrieval, extraction, and processing.
MinerU was born during the pre-training process of [InternLM](https://github.com/InternLM/InternLM). We focus on solving symbol conversion issues in scientific literature and hope to contribute to technological development in the era of large models.
-Compared to well-known commercial products, MinerU is still young. If you encounter any issues or if the results are not as expected, please submit an issue on [issue](https://github.com/opendatalab/MinerU/issues) and **attach the relevant PDF**.
+Compared to well-known commercial products, MinerU is still young. If you encounter any issues or if the results are not as expected, please submit an issue on [issue](https://github.com/opendatalab/MinerU/issues) and **attach the relevant document or sample file**.
https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
## Key Features
+- Support `PDF`, image, `DOCX`, `PPTX`, and `XLSX` inputs.
- Remove headers, footers, footnotes, page numbers, etc., to ensure semantic coherence.
- Output text in human-readable order, suitable for single-column, multi-column, and complex layouts.
- Preserve the structure of the original document, including headings, paragraphs, lists, etc.
- Extract images, image descriptions, tables, table titles, and footnotes.
- Automatically recognize and convert formulas in the document to LaTeX format.
-- Automatically recognize and convert tables in the document to LaTeX or HTML format.
+- Automatically recognize and convert tables in the document to HTML format.
- Automatically detect scanned PDFs and garbled PDFs and enable OCR functionality.
-- OCR supports detection and recognition of 84 languages.
+- OCR supports detection and recognition of 109 languages.
- Supports multiple output formats, such as multimodal and NLP Markdown, JSON sorted by reading order, and rich intermediate formats.
- Supports various visualization results, including layout visualization and span visualization, for efficient confirmation of output quality.
-- Supports both CPU and GPU environments.
+- Built-in CLI, FastAPI, Gradio WebUI, for local orchestration and multi-service deployment.
+- Supports running in a pure CPU environment, and also supports GPU(CUDA)/NPU(CANN)/MPS acceleration
- Compatible with Windows, Linux, and Mac platforms.
-## Quick Start
+# Quick Start
If you encounter any installation issues, please first consult the FAQ.
If the parsing results are not as expected, refer to the Known Issues.
-There are three different ways to experience MinerU:
-- [Online Demo (No Installation Required)](#online-demo)
-- [Quick CPU Demo (Windows, Linux, Mac)](#quick-cpu-demo)
-- [Linux/Windows + CUDA](#Using-GPU)
+## Online Experience
+
+### Official online web application
+The official online version has the same functionality as the client, with a beautiful interface and rich features, requires login to use
+
+- [](https://mineru.net/OpenSourceTools/Extractor?source=github)
-**⚠️ Pre-installation Notice—Hardware and Software Environment Support**
+### Gradio-based online demo
+A WebUI developed based on Gradio, with a simple interface and only core parsing functionality, no login required
-To ensure the stability and reliability of the project, we only optimize and test for specific hardware and software environments during development. This ensures that users deploying and running the project on recommended system configurations will get the best performance with the fewest compatibility issues.
+- [](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+- [](https://huggingface.co/spaces/opendatalab/MinerU)
-By focusing resources on the mainline environment, our team can more efficiently resolve potential bugs and develop new features.
+## Local Deployment
-In non-mainline environments, due to the diversity of hardware and software configurations, as well as third-party dependency compatibility issues, we cannot guarantee 100% project availability. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first. Most issues already have corresponding solutions in the FAQ. We also encourage community feedback to help us gradually expand support.
+
+> [!WARNING]
+> **Pre-installation Notice—Hardware and Software Environment Support**
+>
+> To ensure the stability and reliability of the project, we only optimize and test for specific hardware and software environments during development. This ensures that users deploying and running the project on recommended system configurations will get the best performance with the fewest compatibility issues.
+>
+> By focusing resources on the mainline environment, our team can more efficiently resolve potential bugs and develop new features.
+>
+> In non-mainline environments, due to the diversity of hardware and software configurations, as well as third-party dependency compatibility issues, we cannot guarantee 100% project availability. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first. Most issues already have corresponding solutions in the FAQ. We also encourage community feedback to help us gradually expand support.
MinerU — High-accuracy document parsing engine for LLM · RAG · Agent workflows
+Converts PDF · DOCX · PPTX · XLSX · Images · Web pages into structured Markdown / JSON · VLM+OCR dual engine · 109 languages+MCP Server · LangChain / Dify / FastGPT native integration · 10+ domestic AI chip support + +**🔍 Core Parsing Capabilities** + +- Native support for `DOCX`, `PPTX`, and `XLSX` parsing +- Formulas → LaTeX · Tables → HTML, accurate layout reconstruction +- Supports scanned docs, handwriting, multi-column layouts, cross-page table merging +- Output follows human reading order with automatic header/footer removal +- VLM + OCR dual engine, 109-language OCR recognition + +**🔌 Integration** + +| Use Case | Solution | +|----------|----------| +| AI Coding Tools | MCP Server — Cursor · Claude Desktop · Windsurf | +| RAG Frameworks | LangChain · LlamaIndex · RAGFlow · RAG-Anything · Flowise · Dify · FastGPT | +| Development | Python / Go / TypeScript SDK · CLI · REST API · Docker | +| No-Code | mineru.net online · Gradio WebUI · Desktop client | + +**🖥️ Deployment (Private · Fully Offline)** + +| Inference Backend | Best For | +|------------------|---------| +| pipeline | Fast & stable, no hallucination, runs on CPU or GPU | +| vlm-engine | High accuracy, supports vLLM / LMDeploy / mlx ecosystem | +| hybrid-engine | High accuracy, native text extraction, low hallucination | + +Domestic AI chips: Ascend · Cambricon · Enflame · MetaX · Moore Threads · Kunlunxin · Iluvatar · Hygon · Biren · T-Head
| Operating System | +Parsing Backend | +pipeline | +*-auto-engine | +*-http-client | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ubuntu 22.04 LTS | -Windows 10 / 11 | -macOS 11+ | +hybrid | +vlm | +hybrid | +vlm | |||||
| Backend Features | +Good Compatibility | +High Hardware Requirements | +For OpenAI Compatible Servers2 | +||||||||
| CPU | -x86_64(unsupported ARM Linux) | -x86_64(unsupported ARM Windows) | -x86_64 / arm64 | +Accuracy1 | +86+ | +90+ | |||||
| Memory | -16GB or more, recommended 32GB+ | +Operating System | +Linux3 / Windows4 / macOS5 | ||||||||
| Python Version | -3.10(Please make sure to create a Python 3.10 virtual environment using conda) | +Pure CPU Support | +✅ | +❌ | +✅ | +||||||
| GPU Acceleration | +Volta and later architecture GPUs or Apple Silicon | +Not Required | |||||||||
| Nvidia Driver Version | -latest (Proprietary Driver) | -latest | -None | +Min VRAM | +4GB | +8GB | +8GB | +2GB | |||
| CUDA Environment | -Automatic installation [12.1 (pytorch) + 11.8 (paddle)] | -11.8 (manual installation) + cuDNN v8.7.0 (manual installation) | -None | +RAM | +Min 16GB, Recommended 32GB or more | +Min 16GB | |||||
| GPU Hardware Support List | -Minimum Requirement 8G+ VRAM | -3060ti/3070/4060 - 8G VRAM enables layout, formula recognition acceleration and OCR acceleration |
- None | +Disk Space | +Min 20GB, SSD Recommended | +Min 2GB | |||||
| Recommended Configuration 10G+ VRAM | -3080/3080ti/3090/3090ti/4070/4070ti/4070tisuper/4080/4090 - 10G VRAM or more can enable layout, formula recognition, OCR acceleration and table recognition acceleration simultaneously - |
+ Python Version | +3.10-3.13 | ||||||||
-
-
-
-
-
-
-[](https://github.com/opendatalab/MinerU)
-[](https://github.com/opendatalab/MinerU)
-[](https://github.com/opendatalab/MinerU/issues)
-[](https://github.com/opendatalab/MinerU/issues)
-[](https://badge.fury.io/py/magic-pdf)
-[](https://pepy.tech/project/magic-pdf)
-[](https://pepy.tech/project/magic-pdf)
-
-
-
-
-
-
-[English](README.md) | [简体中文](README_zh-CN.md) | [日本語](README_ja-JP.md)
-
-
-
-
-
-
-# MinerU
-
-
-## 紹介
-
-MinerUは、ワンストップのオープンソースで高品質なデータ抽出ツールであり、以下の主要な機能を含みます：
-
-- [Magic-PDF](#Magic-PDF) PDFドキュメント抽出
-- [Magic-Doc](#Magic-Doc) ウェブページと電子書籍の抽出
-
-
-# Magic-PDF
-
-
-## 紹介
-
-Magic-PDFは、PDFドキュメントをMarkdown形式に変換するためのツールであり、ローカルに保存されたファイルやS3プロトコルをサポートするオブジェクトストレージ上のファイルを処理することができます。
-
-主な機能は以下の通りです：
-
-- 複数のフロントエンドモデル入力をサポート
-- ヘッダー、フッター、脚注、ページ番号の削除
-- 人間が読みやすいレイアウトフォーマット
-- 見出し、段落、リストなど、元のドキュメントの構造とフォーマットを保持
-- 画像や表を抽出してmarkdown内に表示
-- 数式をLaTeX形式に変換
-- 文字化けしたPDFの自動検出と変換
-- CPUおよびGPU環境に対応
-- Windows、Linux、macOSプラットフォームに対応
-
-
-https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
-
-
-
-## プロジェクト全景
-
-
-
-
-## フローチャート
-
-
-
-### 依存リポジトリ
-
-- [PDF-Extract-Kit : 高品質なPDFコンテンツ抽出のための包括的なツールキット](https://github.com/opendatalab/PDF-Extract-Kit) 🚀🚀🚀
-
-## 入門ガイド
-
-### 要件
-
-- Python >= 3.9
-
-依存関係の競合を避けるために、仮想環境の使用をお勧めします。venvとcondaの両方が適しています。
-例：
-```bash
-conda create -n MinerU python=3.10
-conda activate MinerU
-```
-
-### インストールと設定
-
-#### 1. Magic-PDFのインストール
-
-**1.依存パッケージのインストール**
-
-フル機能パッケージはdetectron2に依存しており、コンパイルインストールが必要です。
-自分でコンパイルする必要がある場合は、https://github.com/facebookresearch/detectron2/issues/5114 を参照してください。
-または、私たちの事前コンパイルされたwhlパッケージを直接使用できます（Python 3.10に限定）：
-
-```bash
-pip install detectron2 --extra-index-url https://wheels.myhloli.com
-```
-
-**2.pipを使用してフル機能パッケージをインストールします**
->注意：pipでインストールされたパッケージはCPUのみをサポートし、クイックテストに最適です。
->
->CUDA/MPSによる加速については、[CUDAまたはMPSによる加速](#4-CUDAまたはMPSによる加速)を参照してください。
-
-```bash
-pip install -U magic-pdf[full]
-```
-
-> ❗️❗️❗️
-> 私たちは0.6.2 ベータ版を事前にリリースし、私たちのログに記載されている多くの問題に対処しました。しかし、このビルドはまだ完全なQAテストを経ておらず、最終的なリリース品質を表していません。問題に遭遇した場合は、問題を通じて速やかに報告するか、0.6.1バージョンに戻ることをお願いします。
-> ```bash
-> pip install -U magic-pdf[full]
-> ```
-
-
-#### 2. モデルの重みファイルのダウンロード
-
-詳細については、[how_to_download_models](docs/how_to_download_models_en.md)を参照してください。
-
-モデルの重みをダウンロードした後、'models'ディレクトリを大きなディスクスペースのあるディレクトリに移動します。できればSSDに移動してください。
-
-
-#### 3. 設定ファイルのコピーと設定
-リポジトリのルートディレクトリに[magic-pdf.template.json](magic-pdf.template.json)ファイルがあります。
-```bash
-cp magic-pdf.template.json ~/magic-pdf.json
-```
-magic-pdf.jsonで、"models-dir"をモデルの重みファイルがあるディレクトリに設定します。
-
-```json
-{
- "models-dir": "/tmp/models"
-}
-```
-
-
-#### 4. CUDAまたはMPSによる加速
-利用可能なNvidia GPUを持っている場合や、Apple Siliconを搭載したMacを使用している場合は、それぞれCUDAまたはMPSによる加速を利用できます。
-##### CUDA
-
-CUDAバージョンに応じたPyTorchバージョンをインストールする必要があります。
-この例では、CUDA 11.8バージョンをインストールします。詳細はhttps://pytorch.org/get-started/locally/ を参照してください。
-```bash
-pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118
-```
-また、設定ファイルmagic-pdf.jsonの"device-mode"の値を変更する必要があります。
-```json
-{
- "device-mode":"cuda"
-}
-```
-
-##### MPS
-
-Mシリーズチップデバイスを搭載したmacOSユーザーは、推論加速のためにMPSを使用できます。
-設定ファイルmagic-pdf.jsonの"device-mode"の値を変更する必要があります。
-```json
-{
- "device-mode":"mps"
-}
-```
-
-
-### 使用方法
-
-#### 1. コマンドラインでの使用
-
-###### シンプル
-
-```bash
-magic-pdf pdf-command --pdf "pdf_path" --inside_model true
-```
-プログラムが終了した後、"/tmp/magic-pdf"ディレクトリに生成されたmarkdownファイルが見つかります。
-markdownディレクトリには対応するxxx_model.jsonファイルがあります。
-ポストプロセッシングパイプラインの二次開発を行う場合は、次のコマンドを使用できます：
-```bash
-magic-pdf pdf-command --pdf "pdf_path" --model "model_json_path"
-```
-この方法では、モデルデータを再実行する必要がなくなり、デバッグが便利になります。
-
-
-###### 詳細
-
-```bash
-magic-pdf --help
-```
-
-
-#### 2. APIを使用した利用
-
-###### ローカル
-```python
-image_writer = DiskReaderWriter(local_image_dir)
-image_dir = str(os.path.basename(local_image_dir))
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
-pipe.pipe_classify()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
-```
-
-###### オブジェクトストレージ
-```python
-s3pdf_cli = S3ReaderWriter(pdf_ak, pdf_sk, pdf_endpoint)
-image_dir = "s3://img_bucket/"
-s3image_cli = S3ReaderWriter(img_ak, img_sk, img_endpoint, parent_path=image_dir)
-pdf_bytes = s3pdf_cli.read(s3_pdf_path, mode=s3pdf_cli.MODE_BIN)
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, s3image_cli)
-pipe.pipe_classify()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
-```
-
-デモは[demo.py](demo/demo.py)を参照してください
-
-
-# Magic-Doc
-
-
-## 紹介
-
-Magic-Docは、ウェブページや多形式の電子書籍をmarkdown形式に変換するためのツールです。
-
-主な機能は以下の通りです：
-
-- ウェブページ抽出
- - テキスト、画像、表、数式情報のクロスモーダルな正確な解析。
-
-- 電子書籍ドキュメント抽出
- - epub、mobiなどのさまざまなドキュメント形式をサポートし、テキストと画像に完全対応。
-
-- 言語タイプの識別
- - 176の言語を正確に認識。
-
-https://github.com/opendatalab/MinerU/assets/11393164/a5a650e9-f4c0-463e-acc3-960967f1a1ca
-
-
-
-https://github.com/opendatalab/MinerU/assets/11393164/0f4a6fe9-6cca-4113-9fdc-a537749d764d
-
-
-
-https://github.com/opendatalab/MinerU/assets/11393164/20438a02-ce6c-4af8-9dde-d722a4e825b2
-
-
-
-
-## プロジェクトリポジトリ
-
-- [Magic-Doc](https://github.com/InternLM/magic-doc)
- 優れたウェブページと電子書籍の抽出ツール
-
-
-# 貢献者の皆様に感謝
-
-
-
-MinerU: PDF-Extract-Kitに基づくエンドツーエンドのPDF解析ツールで、PDFからMarkdownへの変換をサポートします。🚀🚀🚀
-PDF-Extract-Kit: 高品質なPDFコンテンツ抽出のための包括的なツールキット🔥🔥🔥
-
-
+
-PDF-Extract-Kit: 高质量PDF解析工具箱🔥🔥🔥 +🚀MinerU 官网入口→✅ 免装在线版 ✅ 全功能客户端 ✅ 开发者API在线调用，省去部署麻烦，多种产品形态一键get，速冲！
- 👋 join us on Discord and WeChat + 👋 join us on Discord and WeChat
+
+MCP Server · LangChain / Dify / FastGPT 原生集成 · 10+ 国产算力适配
+ +**🔍 核心解析能力** +- 原生支持 `DOCX`、`PPTX`、`XLSX` 解析 +- 公式 → LaTeX · 表格 → HTML，精准还原复杂版面 +- 支持扫描件、手写体、多栏布局、跨页表格合并 +- 输出符合人类阅读顺序，自动去除页眉页脚 +- VLM + OCR 双引擎，支持 109 种语言识别 + +**🔌 接入方式** + +| 场景 | 方案 | +|------|------| +| AI 编程工具 | MCP Server — Cursor · Claude Desktop · Windsurf | +| RAG 框架 | LangChain · LlamaIndex · RAGFlow · RAG-Anything · Flowise · Dify · FastGPT | +| 开发集成 | Python / Go / TypeScript SDK · CLI · REST API · Docker | +| 零代码 | mineru.net 在线版 · Gradio WebUI · 桌面客户端 | + +**🖥️ 部署生态（支持私有化 · 完全离线）** + +| 推理后端 | 适用场景 | +|--------------|-----------------------------| +| pipeline | 快速稳定，无幻觉，CPU / GPU 均可运行 | +| vlm-engine | 高精度，支持 vLLM / LMdeploy / mlx 生态 | +| hybrid-engine| 高精度，原生文本提取，低幻觉 | + +国产算力：昇腾 · 寒武纪 · 燧原 · 沐曦 · 摩尔线程 · 昆仑芯 · 天数智芯 · 瀚博 · 太初元碁 · 海光 · 平头哥 -- 2024/10/31 0.9.0发布，这是我们进行了大量代码重构的全新版本，解决了众多问题，提升了性能，降低了硬件需求，并提供了更丰富的易用性： - - 重构排序模块代码，使用 [layoutreader](https://github.com/ppaanngggg/layoutreader) 进行阅读顺序排序，确保在各种排版下都能实现极高准确率 - - 重构段落拼接模块，在跨栏、跨页、跨图、跨表情况下均能实现良好的段落拼接效果 - - 重构列表和目录识别功能，极大提升列表块和目录块识别的准确率及对应文本段落的解析效果 - - 重构图、表与描述性文本的匹配逻辑，大幅提升 caption 和 footnote 与图表的匹配准确率，并将描述性文本的丢失率降至零 - - 增加 OCR 的多语言支持，支持 84 种语言的检测与识别，语言支持列表详见 [OCR 语言支持列表](https://paddlepaddle.github.io/PaddleOCR/latest/ppocr/blog/multi_languages.html#5) - - 增加显存回收逻辑及其他显存优化措施，大幅降低显存使用需求。开启除表格加速外的全部加速功能(layout/公式/OCR)的显存需求从16GB降至8GB，开启全部加速功能的显存需求从24GB降至10GB - - 优化配置文件的功能开关，增加独立的公式检测开关，无需公式检测时可大幅提升速度和解析效果 - - 集成 [PDF-Extract-Kit 1.0](https://github.com/opendatalab/PDF-Extract-Kit) - - 加入自研的 `doclayout_yolo` 模型，在相近解析效果情况下比原方案提速10倍以上，可通过配置文件与 `layoutlmv3` 自由切换 - - 公式解析升级至 `unimernet 0.2.1`，在提升公式解析准确率的同时，大幅降低显存需求 -- 2024/09/27 0.8.1发布，修复了一些bug，同时提供了[在线demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/)的[本地化部署版本](projects/web_demo/README_zh-CN.md)和[前端界面](projects/web/README_zh-CN.md) -- 2024/09/09 0.8.0发布，支持Dockerfile快速部署，同时上线了huggingface、modelscope demo -- 2024/08/30 0.7.1发布，集成了paddle tablemaster表格识别功能 -- 2024/08/09 0.7.0b1发布，简化安装步骤提升易用性，加入表格识别功能 -- 2024/08/01 0.6.2b1发布，优化了依赖冲突问题和安装文档 -- 2024/07/05 首次开源 - - - -
+
-### 在线体验
-稳定版(经过QA验证的稳定版本)：
+1 精度指标为OmniDocBench (v1.5)的End-to-End Evaluation Overall分数，基于`MinerU`最新版本测试
+2 兼容OpenAI API的服务器，如通过`vLLM`/`SGLang`/`LMDeploy`等推理框架部署的本地模型服务器或远程模型服务
+3 Linux仅支持2019年及以后发行版
+4 由于关键依赖`ray`未能在windows平台支持Python 3.13，故仅支持至3.10~3.12版本
+5 macOS 需使用14.0以上版本
-[](https://opendatalab.com/OpenSourceTools/Extractor/PDF)
+> [!TIP]
+> 除以上主流环境与平台外，我们也收录了一些社区用户反馈的其他平台支持情况，详情请参考[其他加速卡适配](https://opendatalab.github.io/MinerU/zh/usage/)。
+> 如果您有意将自己的环境适配经验分享给社区，欢迎通过[show-and-tell](https://github.com/opendatalab/MinerU/discussions/categories/show-and-tell)提交或提交PR至[其他加速卡适配](https://github.com/opendatalab/MinerU/tree/master/docs/zh/usage/acceleration_cards)文档。
-测试版(同步dev分支更新，测试新特性)：
-
-[](https://huggingface.co/spaces/opendatalab/MinerU)
-[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
-
-### 使用CPU快速体验
-
-#### 1. 安装magic-pdf
-
-最新版本国内镜像源同步可能会有延迟，请耐心等待
+### 安装 MinerU
+#### 使用pip或uv安装MinerU
```bash
-conda create -n MinerU python=3.10
-conda activate MinerU
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
+pip install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple
+pip install uv -i https://mirrors.aliyun.com/pypi/simple
+uv pip install -U "mineru[all]" -i https://mirrors.aliyun.com/pypi/simple
```
-#### 2. 下载模型权重文件
-
-详细参考 [如何下载模型文件](docs/how_to_download_models_zh_cn.md)
-
-#### 3. 修改配置文件以进行额外配置
-
-完成[2. 下载模型权重文件](#2-下载模型权重文件)步骤后，脚本会自动生成用户目录下的magic-pdf.json文件，并自动配置默认模型路径。
-您可在【用户目录】下找到magic-pdf.json文件。
-> windows的用户目录为 "C:\\Users\\用户名", linux用户目录为 "/home/用户名", macOS用户目录为 "/Users/用户名"
-
-您可修改该文件中的部分配置实现功能的开关，如表格识别功能：
-
->如json内没有如下项目，请手动添加需要的项目，并删除注释内容（标准json不支持注释）
-
-```json
-{
- // other config
- "layout-config": {
- "model": "layoutlmv3" // 使用doclayout_yolo请修改为“doclayout_yolo"
- },
- "formula-config": {
- "mfd_model": "yolo_v8_mfd",
- "mfr_model": "unimernet_small",
- "enable": true // 公式识别功能默认是开启的，如果需要关闭请修改此处的值为"false"
- },
- "table-config": {
- "model": "tablemaster", // 使用structEqTable请修改为"struct_eqtable"
- "enable": false, // 表格识别功能默认是关闭的，如果需要开启请修改此处的值为"true"
- "max_time": 400
- }
-}
+#### 通过源码安装MinerU
+```bash
+git clone https://github.com/opendatalab/MinerU.git
+cd MinerU
+uv pip install -e .[all] -i https://mirrors.aliyun.com/pypi/simple
```
-### 使用GPU
+> [!TIP]
+> `mineru[all]`包含所有核心功能，兼容Windows / Linux / macOS系统，适合绝大多数用户。
+> 如果您需要指定vlm模型的推理框架，或是仅准备在边缘设备安装轻量版client端，可以参考文档[扩展模块安装指南](https://opendatalab.github.io/MinerU/zh/quick_start/extension_modules/)。
-如果您的设备支持CUDA，且满足主线环境中的显卡要求，则可以使用GPU加速，请根据自己的系统选择适合的教程：
+---
+
+#### 使用docker部署Mineru
+MinerU提供了便捷的docker部署方式，这有助于快速搭建环境并解决一些棘手的环境兼容问题。
+您可以在文档中获取[Docker部署说明](https://opendatalab.github.io/MinerU/zh/quick_start/docker_deployment/)。
-- [Ubuntu22.04LTS + GPU](docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md)
-- [Windows10/11 + GPU](docs/README_Windows_CUDA_Acceleration_zh_CN.md)
-- 使用Docker快速部署
- > Docker 需设备gpu显存大于等于16GB，默认开启所有加速功能
- >
- > 运行本docker前可以通过以下命令检测自己的设备是否支持在docker上使用CUDA加速
- >
- > ```bash
- > docker run --rm --gpus=all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
- > ```
- ```bash
- wget https://github.com/opendatalab/MinerU/raw/master/Dockerfile
- docker build -t mineru:latest .
- docker run --rm -it --gpus=all mineru:latest /bin/bash
- magic-pdf --help
- ```
-
+---
-## 使用
+### 使用 MinerU
-### 命令行
+>[!TIP]
+>默认使用托管在`huggingface`的模型进行解析，首次使用时会自动下载所需模型文件，后续使用将直接加载本地缓存的模型。如果您无法访问`huggingface`，可以通过以下命令切换至国内镜像源:
+>```bash
+>export MINERU_MODEL_SOURCE=modelscope
+>```
+如果您的设备满足上表中GPU加速的条件，可以使用简单的命令行进行文档解析:
```bash
-magic-pdf --help
-Usage: magic-pdf [OPTIONS]
-
-Options:
- -v, --version display the version and exit
- -p, --path PATH local pdf filepath or directory [required]
- -o, --output-dir PATH output local directory [required]
- -m, --method [ocr|txt|auto] the method for parsing pdf. ocr: using ocr
- technique to extract information from pdf. txt:
- suitable for the text-based pdf only and
- outperform ocr. auto: automatically choose the
- best method for parsing pdf from ocr and txt.
- without method specified, auto will be used by
- default.
- -l, --lang TEXT Input the languages in the pdf (if known) to
- improve OCR accuracy. Optional. You should
- input "Abbreviation" with language form url: ht
- tps://paddlepaddle.github.io/PaddleOCR/latest/en
- /ppocr/blog/multi_languages.html#5-support-languages-
- and-abbreviations
- -d, --debug BOOLEAN Enables detailed debugging information during
- the execution of the CLI commands.
- -s, --start INTEGER The starting page for PDF parsing, beginning
- from 0.
- -e, --end INTEGER The ending page for PDF parsing, beginning from
- 0.
- --help Show this message and exit.
-
-
-## show version
-magic-pdf -v
-
-## command line example
-magic-pdf -p {some_pdf} -o {some_output_dir} -m auto
+mineru -p -o
```
-
-其中 `{some_pdf}` 可以是单个pdf文件，也可以是一个包含多个pdf文件的目录。
-运行完命令后输出的结果会保存在`{some_output_dir}`目录下, 输出的文件列表如下
-
-```text
-├── some_pdf.md # markdown 文件
-├── images # 存放图片目录
-├── some_pdf_layout.pdf # layout 绘图 （包含layout阅读顺序）
-├── some_pdf_middle.json # minerU 中间处理结果
-├── some_pdf_model.json # 模型推理结果
-├── some_pdf_origin.pdf # 原 pdf 文件
-├── some_pdf_spans.pdf # 最小粒度的bbox位置信息绘图
-└── some_pdf_content_list.json # 按阅读顺序排列的富文本json
-```
-
-更多有关输出文件的信息，请参考[输出文件说明](docs/output_file_zh_cn.md)
-
-### API
-
-处理本地磁盘上的文件
-
-```python
-image_writer = DiskReaderWriter(local_image_dir)
-image_dir = str(os.path.basename(local_image_dir))
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
-pipe.pipe_classify()
-pipe.pipe_analyze()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
-```
-
-处理对象存储上的文件
-
-```python
-s3pdf_cli = S3ReaderWriter(pdf_ak, pdf_sk, pdf_endpoint)
-image_dir = "s3://img_bucket/"
-s3image_cli = S3ReaderWriter(img_ak, img_sk, img_endpoint, parent_path=image_dir)
-pdf_bytes = s3pdf_cli.read(s3_pdf_path, mode=s3pdf_cli.MODE_BIN)
-jso_useful_key = {"_pdf_type": "", "model_list": []}
-pipe = UNIPipe(pdf_bytes, jso_useful_key, s3image_cli)
-pipe.pipe_classify()
-pipe.pipe_analyze()
-pipe.pipe_parse()
-md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
+如果您的设备不满足GPU加速条件，可以指定后端为`pipeline`，以在纯CPU环境下运行:
+```bash
+mineru -p -o -b pipeline
```
-详细实现可参考
-
-- [demo.py 最简单的处理方式](demo/demo.py)
-- [magic_pdf_parse_main.py 能够更清晰看到处理流程](demo/magic_pdf_parse_main.py)
-
-### 部署衍生项目
-
-衍生项目包含项目开发者和社群开发者们基于MinerU的二次开发项目，
-例如基于Gradio的应用界面、基于llama的RAG、官网同款web demo、轻量级的多卡负载均衡c/s端等，
-这些项目可能会提供更多的功能和更好的用户体验。
-具体部署方式请参考 [衍生项目readme](projects/README_zh-CN.md)
-
-
-### 二次开发
-
-TODO
+当前 `mineru` 支持本地 `PDF / 图片 / DOCX / PPTX / XLSX` 文件或目录输入，并可通过命令行、API、WebUI、`mineru-router` 等多种方式进行文档解析，具体使用方法请参考[使用指南](https://opendatalab.github.io/MinerU/zh/usage/)。
# TODO
-- 🗹 基于模型的阅读顺序
-- 🗹 正文中目录、列表识别
-- 🗹 表格识别
-- ☐ 正文中代码块识别
-- ☐ [化学式识别](docs/chemical_knowledge_introduction/introduction.pdf)
-- ☐ 几何图形识别
+- [x] 基于模型的阅读顺序
+- [x] 正文中目录、列表识别
+- [x] 表格识别
+- [x] 标题分级
+- [x] 手写文本识别
+- [x] 竖排文本识别
+- [x] 拉丁字母重音符号识别
+- [x] 正文中代码块识别
+- [x] [化学式识别](docs/chemical_knowledge_introduction/introduction.pdf)(https://mineru.net)
+- [ ] 图表内容识别
# Known Issues
- 阅读顺序基于模型对可阅读内容在空间中的分布进行排序，在极端复杂的排版下可能会部分区域乱序
-- 不支持竖排文字
+- 对竖排文字的支持较为有限
- 目录和列表通过规则进行识别，少部分不常见的列表形式可能无法识别
-- 标题只有一级，目前不支持标题分级
- 代码块在layout模型里还没有支持
- 漫画书、艺术图册、小学教材、习题尚不能很好解析
- 表格识别在复杂表格上可能会出现行/列识别错误
-- 在小语种PDF上，OCR识别可能会出现字符不准确的情况（如拉丁文的重音符号、阿拉伯文易混淆字符等）
+- 在小语种PDF上，OCR识别可能会出现字符不准确的情况（如阿拉伯文易混淆字符等）
- 部分公式可能会无法在markdown中渲染
# FAQ
-
-[常见问题](docs/FAQ_zh_cn.md)
-
-
-[FAQ](docs/FAQ_en_us.md)
+
+- 如果您在使用过程中遇到问题，可以先查看[常见问题](https://opendatalab.github.io/MinerU/zh/faq/)是否有解答。
+- 如果未能解决您的问题，您也可以使用[DeepWiki](https://deepwiki.com/opendatalab/MinerU)与AI助手交流，这可以解决大部分常见问题。
+- 如果您仍然无法解决问题，您可通过[Discord](https://discord.gg/Tdedn9GTXq)或[WeChat](https://mineru.net/community-portal/?aliasId=3c430f94)加入社区，与其他用户和开发者交流。
# All Thanks To Our Contributors
@@ -411,31 +338,52 @@ TODO
# License Information
-[LICENSE.md](LICENSE.md)
-
-本项目目前采用PyMuPDF以实现高级功能，但因其遵循AGPL协议，可能对某些使用场景构成限制。未来版本迭代中，我们计划探索并替换为许可条款更为宽松的PDF处理库，以提升用户友好度及灵活性。
+本仓库采用 [MinerU 开源许可证](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md) 进行许可，基于 Apache 2.0 并附带额外条款。
# Acknowledgments
-- [PDF-Extract-Kit](https://github.com/opendatalab/PDF-Extract-Kit)
-- [StructEqTable](https://github.com/UniModal4Reasoning/StructEqTable-Deploy)
+- [UniMERNet](https://github.com/opendatalab/UniMERNet)
+- [TableStructureRec](https://github.com/RapidAI/TableStructureRec)
- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
-- [PyMuPDF](https://github.com/pymupdf/PyMuPDF)
-- [layoutreader](https://github.com/ppaanngggg/layoutreader)
+- [PaddleOCR2Pytorch](https://github.com/frotms/PaddleOCR2Pytorch)
- [fast-langdetect](https://github.com/LlmKira/fast-langdetect)
+- [pypdfium2](https://github.com/pypdfium2-team/pypdfium2)
+- [pdftext](https://github.com/datalab-to/pdftext)
- [pdfminer.six](https://github.com/pdfminer/pdfminer.six)
+- [pypdf](https://github.com/py-pdf/pypdf)
+- [magika](https://github.com/google/magika)
+- [vLLM](https://github.com/vllm-project/vllm)
+- [LMDeploy](https://github.com/InternLM/lmdeploy)
# Citation
```bibtex
-@misc{wang2024mineruopensourcesolutionprecise,
- title={MinerU: An Open-Source Solution for Precise Document Content Extraction},
- author={Bin Wang and Chao Xu and Xiaomeng Zhao and Linke Ouyang and Fan Wu and Zhiyuan Zhao and Rui Xu and Kaiwen Liu and Yuan Qu and Fukai Shang and Bo Zhang and Liqun Wei and Zhihao Sui and Wei Li and Botian Shi and Yu Qiao and Dahua Lin and Conghui He},
- year={2024},
- eprint={2409.18839},
- archivePrefix={arXiv},
- primaryClass={cs.CV},
- url={https://arxiv.org/abs/2409.18839},
+@article{wang2026mineru2,
+ title={MinerU2. 5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale},
+ author={Wang, Bin and He, Tianyao and Ouyang, Linke and Wu, Fan and Zhao, Zhiyuan and Chu, Tao and Qu, Yuan and Jin, Zhenjiang and Zeng, Weijun and Miao, Ziyang and others},
+ journal={arXiv preprint arXiv:2604.04771},
+ year={2026}
+}
+
+@article{dong2026minerudiffusion,
+ title={MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding},
+ author={Dong, Hejun and Niu, Junbo and Wang, Bin and Zeng, Weijun and Zhang, Wentao and He, Conghui},
+ journal={arXiv preprint arXiv:2603.22458},
+ year={2026}
+}
+
+@article{niu2025mineru2,
+ title={Mineru2. 5: A decoupled vision-language model for efficient high-resolution document parsing},
+ author={Niu, Junbo and Liu, Zheng and Gu, Zhuangcheng and Wang, Bin and Ouyang, Linke and Zhao, Zhiyuan and Chu, Tao and He, Tianyao and Wu, Fan and Zhang, Qintong and others},
+ journal={arXiv preprint arXiv:2509.22186},
+ year={2025}
+}
+
+@article{wang2024mineru,
+ title={Mineru: An open-source solution for precise document content extraction},
+ author={Wang, Bin and Xu, Chao and Zhao, Xiaomeng and Ouyang, Linke and Wu, Fan and Zhao, Zhiyuan and Xu, Rui and Liu, Kaiwen and Qu, Yuan and Shang, Fukai and others},
+ journal={arXiv preprint arXiv:2409.18839},
+ year={2024}
}
@article{he2024opendatalab,
@@ -456,16 +404,14 @@ TODO
-# Magic-doc
-
-[Magic-Doc](https://github.com/InternLM/magic-doc) Fast speed ppt/pptx/doc/docx/pdf extraction tool
-
-# Magic-html
-
-[Magic-HTML](https://github.com/opendatalab/magic-html) Mixed web page extraction tool
# Links
-
+- [Easy Data Preparation with latest LLMs-based Operators and Pipelines](https://github.com/OpenDCAI/DataFlow)
+- [Vis3 (OSS browser based on s3)](https://github.com/opendatalab/Vis3)
- [LabelU (A Lightweight Multi-modal Data Annotation Tool)](https://github.com/opendatalab/labelU)
- [LabelLLM (An Open-source LLM Dialogue Annotation Platform)](https://github.com/opendatalab/LabelLLM)
- [PDF-Extract-Kit (A Comprehensive Toolkit for High-Quality PDF Content Extraction)](https://github.com/opendatalab/PDF-Extract-Kit)
+- [OmniDocBench (A Comprehensive Benchmark for Document Parsing and Evaluation)](https://github.com/opendatalab/OmniDocBench)
+- [Magic-HTML (Mixed web page extraction tool)](https://github.com/opendatalab/magic-html)
+- [Magic-Doc (Fast speed ppt/pptx/doc/docx/pdf extraction tool)](https://github.com/InternLM/magic-doc)
+- [Dingo: A Comprehensive AI Data Quality Evaluation Tool](https://github.com/MigoXLab/dingo)
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 000000000..7e96a1695
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,31 @@
+# Security Policy
+
+## Supported Versions
+
+latest
+
+## Reporting a Vulnerability
+
+Please do not report security vulnerabilities through public GitHub issues.
+
+Instead, please report them at https://github.com/opendatalab/MinerU/security.
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+ * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+ * Full paths of source file(s) related to the manifestation of the issue
+ * The location of the affected source code (tag/branch/commit or direct URL)
+ * Any special configuration required to reproduce the issue
+ * Step-by-step instructions to reproduce the issue
+ * Proof-of-concept or exploit code (if possible)
+ * Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+## Preferred Languages
+
+We prefer all communications to be in English and Chinese.
+
+## Policy
+
+We will fix security issues in the project's own code as quickly as possible. Before the project completes the fix, you must not disclose the vulnerability information to any public platform.
diff --git a/demo/demo.py b/demo/demo.py
index a3bd6d038..8a4235a92 100644
--- a/demo/demo.py
+++ b/demo/demo.py
@@ -1,25 +1,256 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
import os
+import tempfile
+from pathlib import Path
-from loguru import logger
-from magic_pdf.pipe.UNIPipe import UNIPipe
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-
-
-try:
- current_script_dir = os.path.dirname(os.path.abspath(__file__))
- demo_name = "demo1"
- pdf_path = os.path.join(current_script_dir, f"{demo_name}.pdf")
- pdf_bytes = open(pdf_path, "rb").read()
- jso_useful_key = {"_pdf_type": "", "model_list": []}
- local_image_dir = os.path.join(current_script_dir, 'images')
- image_dir = str(os.path.basename(local_image_dir))
- image_writer = DiskReaderWriter(local_image_dir)
- pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
- pipe.pipe_classify()
- pipe.pipe_analyze()
- pipe.pipe_parse()
- md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
- with open(f"{demo_name}.md", "w", encoding="utf-8") as f:
- f.write(md_content)
-except Exception as e:
- logger.exception(e)
\ No newline at end of file
+import httpx
+
+from mineru.cli import api_client as _api_client
+from mineru.cli.common import image_suffixes, office_suffixes, pdf_suffixes
+from mineru.utils.guess_suffix_or_lang import guess_suffix_by_path
+
+SUPPORTED_INPUT_SUFFIXES = set(pdf_suffixes + image_suffixes + office_suffixes)
+
+
+def collect_input_files(input_path: str | Path) -> list[Path]:
+ path = Path(input_path).expanduser().resolve()
+ if not path.exists():
+ raise FileNotFoundError(f"Input path does not exist: {path}")
+
+ if path.is_file():
+ file_suffix = guess_suffix_by_path(path)
+ if file_suffix not in SUPPORTED_INPUT_SUFFIXES:
+ raise ValueError(f"Unsupported input file type: {path.name}")
+ return [path]
+
+ if not path.is_dir():
+ raise ValueError(f"Input path must be a file or directory: {path}")
+
+ input_files = sorted(
+ (
+ candidate.resolve()
+ for candidate in path.iterdir()
+ if candidate.is_file()
+ and guess_suffix_by_path(candidate) in SUPPORTED_INPUT_SUFFIXES
+ ),
+ key=lambda item: item.name,
+ )
+ if not input_files:
+ raise ValueError(f"No supported files found in directory: {path}")
+ return input_files
+
+
+def build_form_data(
+ language: str,
+ backend: str,
+ parse_method: str,
+ formula_enable: bool,
+ table_enable: bool,
+ server_url: str | None,
+ start_page_id: int,
+ end_page_id: int | None,
+) -> dict[str, str | list[str]]:
+ return _api_client.build_parse_request_form_data(
+ lang_list=[language],
+ backend=backend,
+ parse_method=parse_method,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ return_md=True,
+ return_middle_json=False,
+ return_model_output=False,
+ return_content_list=False,
+ return_images=True,
+ response_format_zip=True,
+ return_original_file=False,
+ )
+
+
+def format_status_message(status_snapshot: _api_client.TaskStatusSnapshot) -> str:
+ if status_snapshot.queued_ahead is None:
+ return status_snapshot.status
+ return f"{status_snapshot.status} (queued_ahead={status_snapshot.queued_ahead})"
+
+
+def prepare_local_api_temp_dir() -> None:
+ current_temp_dir = Path(tempfile.gettempdir())
+ if os.name == "nt" or not Path("/tmp").exists():
+ return
+ if not str(current_temp_dir).startswith("/mnt/"):
+ return
+
+ # vLLM/ZeroMQ IPC sockets fail on drvfs-backed temp directories under WSL.
+ os.environ["TMPDIR"] = "/tmp"
+ tempfile.tempdir = None
+
+
+async def run_demo(
+ input_path: str | Path,
+ output_dir: str | Path,
+ *,
+ api_url: str | None = None,
+ backend: str = "hybrid-auto-engine",
+ parse_method: str = "auto",
+ language: str = "ch",
+ formula_enable: bool = True,
+ table_enable: bool = True,
+ server_url: str | None = None,
+ start_page_id: int = 0,
+ end_page_id: int | None = None,
+) -> None:
+ api_url = api_url or None
+ server_url = server_url or None
+ if backend.endswith("http-client") and not server_url:
+ raise ValueError(f"backend={backend} requires server_url")
+
+ input_files = collect_input_files(input_path)
+ output_path = Path(output_dir).expanduser().resolve()
+ output_path.mkdir(parents=True, exist_ok=True)
+
+ form_data = build_form_data(
+ language=language,
+ backend=backend,
+ parse_method=parse_method,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+ upload_assets = [
+ _api_client.UploadAsset(path=file_path, upload_name=file_path.name)
+ for file_path in input_files
+ ]
+
+ local_server: _api_client.LocalAPIServer | None = None
+ result_zip_path: Path | None = None
+ task_label = f"{len(input_files)} file(s)"
+
+ async with httpx.AsyncClient(
+ timeout=_api_client.build_http_timeout(),
+ follow_redirects=True,
+ ) as http_client:
+ try:
+ if api_url is None:
+ prepare_local_api_temp_dir()
+ local_server = _api_client.LocalAPIServer()
+ base_url = local_server.start()
+ print(f"Started local mineru-api: {base_url}")
+ server_health = await _api_client.wait_for_local_api_ready(
+ http_client,
+ local_server,
+ )
+ else:
+ server_health = await _api_client.fetch_server_health(
+ http_client,
+ _api_client.normalize_base_url(api_url),
+ )
+
+ print(f"Using API: {server_health.base_url}")
+ print(f"Submitting {len(upload_assets)} file(s)")
+ submit_response = await _api_client.submit_parse_task(
+ base_url=server_health.base_url,
+ upload_assets=upload_assets,
+ form_data=form_data,
+ )
+ print(f"task_id: {submit_response.task_id}")
+ if submit_response.queued_ahead is not None:
+ print(f"status: pending (queued_ahead={submit_response.queued_ahead})")
+
+ last_status_message = None
+
+ def on_status_update(status_snapshot: _api_client.TaskStatusSnapshot) -> None:
+ nonlocal last_status_message
+ message = format_status_message(status_snapshot)
+ if message == last_status_message:
+ return
+ last_status_message = message
+ print(f"status: {message}")
+
+ await _api_client.wait_for_task_result(
+ client=http_client,
+ submit_response=submit_response,
+ task_label=task_label,
+ status_snapshot_callback=on_status_update,
+ )
+ print("status: completed")
+ result_zip_path = await _api_client.download_result_zip(
+ client=http_client,
+ submit_response=submit_response,
+ task_label=task_label,
+ )
+ finally:
+ if local_server is not None:
+ local_server.stop()
+
+ assert result_zip_path is not None
+ try:
+ _api_client.safe_extract_zip(result_zip_path, output_path)
+ finally:
+ result_zip_path.unlink(missing_ok=True)
+ print(f"Extracted result to: {output_path}")
+
+
+def main() -> None:
+ demo_dir = Path(__file__).resolve().parent
+
+ # Input can be a single supported file or a directory containing supported files.
+ input_path = demo_dir / "pdfs"
+ # Parsed outputs will be extracted into this directory.
+ output_dir = demo_dir / "api_output"
+ # Set this to an existing MinerU FastAPI base URL, for example:
+ # "http://127.0.0.1:8000"
+ # Leave it as None to start a temporary local mineru-api automatically.
+ api_url = None
+
+ # Available examples:
+ # "hybrid-auto-engine" -> local hybrid parsing, recommended default
+ # "pipeline" -> more general OCR/text pipeline
+ # "vlm-auto-engine" -> local VLM parsing
+ # "vlm-http-client" -> remote OpenAI-compatible VLM server
+ # "hybrid-http-client" -> remote OpenAI-compatible hybrid server
+ backend = "hybrid-auto-engine"
+ # Available options:
+ # "auto" -> let MinerU choose between text extraction and OCR
+ # "txt" -> force text extraction
+ # "ocr" -> force OCR
+ parse_method = "auto"
+ # OCR language hint. This is mainly used by pipeline and hybrid backends.
+ language = "ch"
+ # Enable formula parsing in the output.
+ formula_enable = True
+ # Enable table parsing in the output.
+ table_enable = True
+ # Required only for "*-http-client" backends, for example:
+ # "http://127.0.0.1:30000"
+ server_url = None
+ # Zero-based page range. Set end_page_id to None to parse to the last page.
+ start_page_id = 0
+ end_page_id = None
+
+ """如果您由于网络问题无法下载模型，可以设置环境变量MINERU_MODEL_SOURCE为modelscope使用免代理仓库下载模型"""
+ # os.environ['MINERU_MODEL_SOURCE'] = "modelscope"
+
+ asyncio.run(
+ run_demo(
+ input_path=input_path,
+ output_dir=output_dir,
+ api_url=api_url,
+ backend=backend,
+ parse_method=parse_method,
+ language=language,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/demo/demo1.json b/demo/demo1.json
deleted file mode 100644
index e3b5a30aa..000000000
--- a/demo/demo1.json
+++ /dev/null
@@ -1 +0,0 @@
-[{"layout_dets": [{"category_id": 2, "poly": [117.85147857666016, 198.19203186035156, 268.09375, 198.19203186035156, 268.09375, 365.4513854980469, 117.85147857666016, 365.4513854980469], "score": 1.0}, {"category_id": 2, "poly": [516.9244995117188, 193.8611297607422, 983.7249145507812, 193.8611297607422, 983.7249145507812, 288.566650390625, 516.9244995117188, 288.566650390625], "score": 0.9999980926513672}, {"category_id": 2, "poly": [119.0521469116211, 1793.3775634765625, 774.3035888671875, 1793.3775634765625, 774.3035888671875, 1842.8583984375, 119.0521469116211, 1842.8583984375], "score": 0.9999951720237732}, {"category_id": 1, "poly": [213.19744873046875, 621.9070434570312, 1290.4381103515625, 621.9070434570312, 1290.4381103515625, 733.4085693359375, 213.19744873046875, 733.4085693359375], "score": 0.9999936819076538}, {"category_id": 1, "poly": [390.47998046875, 751.6647338867188, 1108.0994873046875, 751.6647338867188, 1108.0994873046875, 774.5253295898438, 390.47998046875, 774.5253295898438], "score": 0.9999909400939941}, {"category_id": 2, "poly": [556.6760864257812, 343.6651306152344, 942.158447265625, 343.6651306152344, 942.158447265625, 368.6150207519531, 556.6760864257812, 368.6150207519531], "score": 0.9999899864196777}, {"category_id": 0, "poly": [245.8207244873047, 472.72943115234375, 1257.65380859375, 472.72943115234375, 1257.65380859375, 520.0311889648438, 245.8207244873047, 520.0311889648438], "score": 0.9999768137931824}, {"category_id": 2, "poly": [1119.6229248046875, 199.3274383544922, 1376.630859375, 199.3274383544922, 1376.630859375, 384.0538024902344, 1119.6229248046875, 384.0538024902344], "score": 0.9999668002128601}, {"category_id": 1, "poly": [118.14305114746094, 1571.5140380859375, 864.8477172851562, 1571.5140380859375, 864.8477172851562, 1594.3565673828125, 118.14305114746094, 1594.3565673828125], "score": 0.999945342540741}, {"category_id": 0, "poly": [118.69384002685547, 862.561767578125, 209.67910766601562, 862.561767578125, 209.67910766601562, 888.9332885742188, 118.69384002685547, 888.9332885742188], "score": 0.9999412298202515}, {"category_id": 1, "poly": [239.3308868408203, 550.2936401367188, 1257.6968994140625, 550.2936401367188, 1257.6968994140625, 596.7587280273438, 239.3308868408203, 596.7587280273438], "score": 0.9999355673789978}, {"category_id": 2, "poly": [117.71773529052734, 1687.8800048828125, 1379.2835693359375, 1687.8800048828125, 1379.2835693359375, 1766.3516845703125, 117.71773529052734, 1766.3516845703125], "score": 0.999925971031189}, {"category_id": 1, "poly": [115.68157958984375, 913.7571411132812, 1385.33837890625, 913.7571411132812, 1385.33837890625, 1533.5689697265625, 115.68157958984375, 1533.5689697265625], "score": 0.999893307685852}, {"category_id": 2, "poly": [1084.155517578125, 374.07135009765625, 1378.12109375, 374.07135009765625, 1378.12109375, 396.0621032714844, 1084.155517578125, 396.0621032714844], "score": 0.9371034502983093}, {"category_id": 13, "poly": [714, 1383, 767, 1383, 767, 1411, 714, 1411], "score": 0.89, "latex": "N_{\\mathrm{zero}}"}, {"category_id": 13, "poly": [571, 1351, 636, 1351, 636, 1380, 571, 1380], "score": 0.87, "latex": "(N_{\\mathrm{zero}})"}, {"category_id": 13, "poly": [398, 1793, 419, 1793, 419, 1815, 398, 1815], "score": 0.75, "latex": "\\copyright"}, {"category_id": 13, "poly": [116, 1509, 140, 1509, 140, 1533, 116, 1533], "score": 0.73, "latex": "\\copyright"}, {"category_id": 13, "poly": [315, 1713, 479, 1713, 479, 1739, 315, 1739], "score": 0.36, "latex": "+61\\;3\\;9450\\;8719"}, {"category_id": 13, "poly": [148, 1743, 166, 1743, 166, 1765, 148, 1765], "score": 0.35, "latex": "E"}, {"category_id": 13, "poly": [369, 1743, 387, 1743, 387, 1764, 369, 1764], "score": 0.26, "latex": "@"}, {"category_id": 15, "poly": [120.0, 338.0, 266.0, 338.0, 266.0, 374.0, 120.0, 374.0], "score": 1.0, "text": "ELSEVIER"}, {"category_id": 15, "poly": [515.0, 194.0, 986.0, 194.0, 986.0, 224.0, 515.0, 224.0], "score": 0.99, "text": "Available online at www.sciencedirect.com"}, {"category_id": 15, "poly": [599.0, 245.0, 728.0, 245.0, 728.0, 275.0, 599.0, 275.0], "score": 0.99, "text": "SCIENCE"}, {"category_id": 15, "poly": [712.0, 237.0, 905.0, 229.0, 907.0, 281.0, 714.0, 289.0], "score": 0.77, "text": "CDIRECT."}, {"category_id": 15, "poly": [116.0, 1819.0, 427.0, 1819.0, 427.0, 1847.0, 116.0, 1847.0], "score": 0.99, "text": "doi:10.1016/j.jhydrol.2005.01.006"}, {"category_id": 15, "poly": [114.0, 1793.0, 397.0, 1793.0, 397.0, 1821.0, 114.0, 1821.0], "score": 0.96, "text": "0022-1694/$ - see front matter"}, {"category_id": 15, "poly": [420.0, 1793.0, 777.0, 1793.0, 777.0, 1821.0, 420.0, 1821.0], "score": 0.98, "text": " 2005 Elsevier B.V. All rights reserved."}, {"category_id": 15, "poly": [210.0, 624.0, 1291.0, 624.0, 1291.0, 654.0, 210.0, 654.0], "score": 0.97, "text": "aSchool of Forest and Ecosystem Studies,University of Melbourne,P.O.Box 137,Heidelberg,Victoria 3084,Australia"}, {"category_id": 15, "poly": [460.0, 647.0, 1040.0, 649.0, 1039.0, 679.0, 460.0, 677.0], "score": 0.96, "text": "bCSIRODivision of Land andWater,Canberra,ACT,Australia"}, {"category_id": 15, "poly": [369.0, 679.0, 1130.0, 679.0, 1130.0, 710.0, 369.0, 710.0], "score": 0.97, "text": "cCooperative Research Centre for Catchment Hydrology, Canberra,ACT, Australia"}, {"category_id": 15, "poly": [299.0, 701.0, 1203.0, 703.0, 1203.0, 740.0, 299.0, 737.0], "score": 0.98, "text": "dDepartment of Civil and Environmental Engineering, University of Melbourne, Victoria, Australia"}, {"category_id": 15, "poly": [389.0, 750.0, 1108.0, 750.0, 1108.0, 780.0, 389.0, 780.0], "score": 0.99, "text": "Received 1 October 2003; revised 22 December 2004; accepted 3 January 2005"}, {"category_id": 15, "poly": [554.0, 340.0, 945.0, 337.0, 945.0, 374.0, 554.0, 376.0], "score": 0.98, "text": "Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [247.0, 477.0, 1252.0, 477.0, 1252.0, 520.0, 247.0, 520.0], "score": 0.99, "text": "The response of flow duration curves to afforestation"}, {"category_id": 15, "poly": [1165.0, 212.0, 1285.0, 218.0, 1283.0, 256.0, 1164.0, 251.0], "score": 1.0, "text": "Journal"}, {"category_id": 15, "poly": [1171.0, 260.0, 1207.0, 260.0, 1207.0, 290.0, 1171.0, 290.0], "score": 0.84, "text": "of"}, {"category_id": 15, "poly": [1157.0, 290.0, 1379.0, 297.0, 1378.0, 351.0, 1155.0, 343.0], "score": 1.0, "text": "Hydrology"}, {"category_id": 15, "poly": [1164.0, 374.0, 1368.0, 374.0, 1368.0, 389.0, 1164.0, 389.0], "score": 0.73, "text": "nuriarnom/laonta/ihrdr"}, {"category_id": 15, "poly": [116.0, 1572.0, 868.0, 1572.0, 868.0, 1600.0, 116.0, 1600.0], "score": 0.99, "text": "Keywords: Afforestation; Flow duration curves; Flow reduction; Paired catchments"}, {"category_id": 15, "poly": [116.0, 862.0, 213.0, 862.0, 213.0, 894.0, 116.0, 894.0], "score": 1.0, "text": "Abstract"}, {"category_id": 15, "poly": [238.0, 557.0, 1254.0, 557.0, 1254.0, 600.0, 238.0, 600.0], "score": 0.94, "text": "Patrick N.J. Lanea,c,*, Alice E. Bestb,c,d, Klaus Hickelb;c, Lu Zhangbc"}, {"category_id": 15, "poly": [127.0, 1681.0, 1381.0, 1683.0, 1381.0, 1720.0, 127.0, 1718.0], "score": 0.98, "text": "* Corresponding author. Address: Forest Science Centre, Department of Sustainability and Environment, P.O. Box 137, Heidelberg, Vic."}, {"category_id": 15, "poly": [114.0, 1711.0, 314.0, 1714.0, 314.0, 1744.0, 114.0, 1741.0], "score": 0.97, "text": "3084,Australia.Tel.:"}, {"category_id": 15, "poly": [480.0, 1711.0, 702.0, 1714.0, 702.0, 1744.0, 480.0, 1741.0], "score": 0.93, "text": ";fax: +61 3 9450 8644."}, {"category_id": 15, "poly": [167.0, 1744.0, 368.0, 1744.0, 368.0, 1772.0, 167.0, 1772.0], "score": 1.0, "text": "mailaddress:patrickl"}, {"category_id": 15, "poly": [388.0, 1744.0, 657.0, 1744.0, 657.0, 1772.0, 388.0, 1772.0], "score": 1.0, "text": "unimelb.edu.au (P.N.J. Lane)."}, {"category_id": 15, "poly": [137.0, 912.0, 1385.0, 912.0, 1385.0, 948.0, 137.0, 948.0], "score": 0.98, "text": " The hydrologic effect of replacing pasture or other short crops with trees is reasonably well understood on a mean annual"}, {"category_id": 15, "poly": [116.0, 946.0, 1383.0, 946.0, 1383.0, 976.0, 116.0, 976.0], "score": 0.99, "text": "basis. The impact on fow regime, as described by the annual flow duration curve (FDC) is less certain. A method to assess the"}, {"category_id": 15, "poly": [114.0, 974.0, 1383.0, 974.0, 1383.0, 1010.0, 114.0, 1010.0], "score": 0.99, "text": "impact of plantation establishment on FDCs was developed. The starting point for the analyses was the assumption that rainfall"}, {"category_id": 15, "poly": [116.0, 1008.0, 1381.0, 1008.0, 1381.0, 1038.0, 116.0, 1038.0], "score": 0.99, "text": "and vegetation age are the principal drivers of evapotranspiration. A key objective was to remove the variability in the rainfall"}, {"category_id": 15, "poly": [116.0, 1041.0, 1381.0, 1041.0, 1381.0, 1071.0, 116.0, 1071.0], "score": 0.99, "text": "signal, leaving changes in streamflow solely attributable to the evapotranspiration of the plantation. A method was developed to"}, {"category_id": 15, "poly": [116.0, 1073.0, 1381.0, 1073.0, 1381.0, 1103.0, 116.0, 1103.0], "score": 0.98, "text": "(1) fit a model to the observed annual time series of FDC percentiles; i.e. 1oth percentile for each year of record with annual"}, {"category_id": 15, "poly": [114.0, 1101.0, 1381.0, 1103.0, 1381.0, 1133.0, 114.0, 1131.0], "score": 0.99, "text": "rainfall and plantation age as parameters, (2) replace the annual rainfall variation with the long term mean to obtain climate"}, {"category_id": 15, "poly": [118.0, 1135.0, 1383.0, 1135.0, 1383.0, 1165.0, 118.0, 1165.0], "score": 0.99, "text": "adjusted FDCs, and (3) quantify changes in FDC percentiles as plantations age. Data from 10 catchments from Australia, South"}, {"category_id": 15, "poly": [118.0, 1165.0, 1381.0, 1165.0, 1381.0, 1195.0, 118.0, 1195.0], "score": 0.99, "text": "Africa and New Zealand were used. The model was able to represent flow variation for the majority of percentiles at eight of the"}, {"category_id": 15, "poly": [114.0, 1191.0, 1383.0, 1193.0, 1383.0, 1230.0, 114.0, 1228.0], "score": 0.98, "text": "10 catchments, particularly for the 10-50th percentiles. The adjusted FDCs revealed variable patterns in flow reductions with"}, {"category_id": 15, "poly": [116.0, 1230.0, 1379.0, 1230.0, 1379.0, 1260.0, 116.0, 1260.0], "score": 0.98, "text": "two types of responses (groups) being identified. Group 1 catchments show a substantial increase in the number of zero fow"}, {"category_id": 15, "poly": [114.0, 1258.0, 1381.0, 1260.0, 1381.0, 1290.0, 114.0, 1288.0], "score": 0.98, "text": "days, with low flows being more affected than high flows. Group 2 catchments show a more uniform reduction in flows across"}, {"category_id": 15, "poly": [116.0, 1292.0, 1383.0, 1292.0, 1383.0, 1322.0, 116.0, 1322.0], "score": 0.98, "text": "all percentiles. The differences may be partly explained by storage characteristics. The modelled fow reductions were in accord"}, {"category_id": 15, "poly": [116.0, 1322.0, 1381.0, 1322.0, 1381.0, 1352.0, 116.0, 1352.0], "score": 1.0, "text": "with published results of paired catchment experiments. An additional analysis was performed to characterise the impact of"}, {"category_id": 15, "poly": [116.0, 1417.0, 1381.0, 1417.0, 1381.0, 1447.0, 116.0, 1447.0], "score": 1.0, "text": "in the occurrence of any given flow in response to afforestation. The methods used in this study proved satisfactory in removing"}, {"category_id": 15, "poly": [116.0, 1449.0, 1383.0, 1449.0, 1383.0, 1479.0, 116.0, 1479.0], "score": 0.99, "text": "the rainfall variability, and have added useful insight into the hydrologic impacts of plantation establishment. This approach"}, {"category_id": 15, "poly": [116.0, 1479.0, 1379.0, 1479.0, 1379.0, 1509.0, 116.0, 1509.0], "score": 0.99, "text": "provides a methodology for understanding catchment response to afforestation, where paired catchment data is not available."}, {"category_id": 15, "poly": [114.0, 1382.0, 713.0, 1387.0, 713.0, 1417.0, 114.0, 1413.0], "score": 0.98, "text": "when adjusted for climate, indicated a significant increase in"}, {"category_id": 15, "poly": [768.0, 1382.0, 1381.0, 1387.0, 1381.0, 1417.0, 768.0, 1413.0], "score": 0.98, "text": ".The zero flow day method could be used to determine change"}, {"category_id": 15, "poly": [116.0, 1354.0, 570.0, 1354.0, 570.0, 1385.0, 116.0, 1385.0], "score": 0.98, "text": "afforestation on the number of zero flow days"}, {"category_id": 15, "poly": [637.0, 1354.0, 1383.0, 1354.0, 1383.0, 1385.0, 637.0, 1385.0], "score": 0.99, "text": "for the catchments in group 1. This model performed particularly well, and"}, {"category_id": 15, "poly": [141.0, 1507.0, 541.0, 1509.0, 541.0, 1539.0, 141.0, 1537.0], "score": 0.98, "text": "2005 Elsevier B.V. All rights reserved."}, {"category_id": 15, "poly": [1080.0, 368.0, 1383.0, 365.0, 1383.0, 402.0, 1080.0, 404.0], "score": 0.99, "text": "www.elsevier.com/locate/jhydrol"}], "page_info": {"page_no": 0, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 0, "poly": [130.931640625, 251.82516479492188, 312.8154296875, 251.82516479492188, 312.8154296875, 283.4620056152344, 130.931640625, 283.4620056152344], "score": 0.9999987483024597}, {"category_id": 4, "poly": [794.2171020507812, 763.5051879882812, 1396.4493408203125, 763.5051879882812, 1396.4493408203125, 818.8292236328125, 794.2171020507812, 818.8292236328125], "score": 0.9999982714653015}, {"category_id": 1, "poly": [130.19113159179688, 1017.6807861328125, 732.7059326171875, 1017.6807861328125, 732.7059326171875, 1849.8070068359375, 130.19113159179688, 1849.8070068359375], "score": 0.9999954104423523}, {"category_id": 1, "poly": [793.3727416992188, 1280.632568359375, 1397.07080078125, 1280.632568359375, 1397.07080078125, 1849.0452880859375, 793.3727416992188, 1849.0452880859375], "score": 0.9999947547912598}, {"category_id": 1, "poly": [793.5277099609375, 849.8186645507812, 1397.0140380859375, 849.8186645507812, 1397.0140380859375, 1280.6221923828125, 793.5277099609375, 1280.6221923828125], "score": 0.999994158744812}, {"category_id": 1, "poly": [130.5381317138672, 317.5604248046875, 731.9227905273438, 317.5604248046875, 731.9227905273438, 1015.91748046875, 130.5381317138672, 1015.91748046875], "score": 0.9999940395355225}, {"category_id": 2, "poly": [130.44467163085938, 194.42764282226562, 166.39125061035156, 194.42764282226562, 166.39125061035156, 215.1434783935547, 130.44467163085938, 215.1434783935547], "score": 0.999992847442627}, {"category_id": 2, "poly": [479.5857849121094, 195.1154022216797, 1045.4803466796875, 195.1154022216797, 1045.4803466796875, 218.7963104248047, 479.5857849121094, 218.7963104248047], "score": 0.99998939037323}, {"category_id": 3, "poly": [799.3821411132812, 256.1320495605469, 1390.73681640625, 256.1320495605469, 1390.73681640625, 742.4434204101562, 799.3821411132812, 742.4434204101562], "score": 0.9999882578849792}, {"category_id": 13, "poly": [984, 1180, 1065, 1180, 1065, 1211, 984, 1211], "score": 0.88, "latex": "<20\\%"}, {"category_id": 13, "poly": [128, 1415, 183, 1415, 183, 1445, 128, 1445], "score": 0.86, "latex": "95\\%"}, {"category_id": 13, "poly": [573, 618, 723, 618, 723, 649, 573, 649], "score": 0.67, "latex": "400\u2013500\\;\\mathrm{mm}"}, {"category_id": 15, "poly": [127.0, 249.0, 316.0, 254.0, 315.0, 291.0, 126.0, 286.0], "score": 1.0, "text": "1. Introduction"}, {"category_id": 15, "poly": [793.0, 765.0, 1394.0, 765.0, 1394.0, 793.0, 793.0, 793.0], "score": 0.98, "text": "Fig. 1. Annual flow duration curves of daily flows from Pine Creek,"}, {"category_id": 15, "poly": [793.0, 793.0, 999.0, 793.0, 999.0, 821.0, 793.0, 821.0], "score": 0.97, "text": "Australia, 1989-2000."}, {"category_id": 15, "poly": [161.0, 1017.0, 735.0, 1017.0, 735.0, 1054.0, 161.0, 1054.0], "score": 0.98, "text": "Zhang et al. (1999, 2001) developed simple and"}, {"category_id": 15, "poly": [127.0, 1051.0, 735.0, 1051.0, 735.0, 1088.0, 127.0, 1088.0], "score": 0.99, "text": "easily parameterised models to predict changes in"}, {"category_id": 15, "poly": [129.0, 1086.0, 730.0, 1086.0, 730.0, 1116.0, 129.0, 1116.0], "score": 0.99, "text": "mean annual fows following afforestation. However,"}, {"category_id": 15, "poly": [129.0, 1120.0, 732.0, 1120.0, 732.0, 1150.0, 129.0, 1150.0], "score": 0.98, "text": "there is a need to consider the annual flow regime as the"}, {"category_id": 15, "poly": [129.0, 1152.0, 732.0, 1152.0, 732.0, 1182.0, 129.0, 1182.0], "score": 0.99, "text": "relative changes in high and low flows may have"}, {"category_id": 15, "poly": [129.0, 1187.0, 730.0, 1187.0, 730.0, 1217.0, 129.0, 1217.0], "score": 0.98, "text": "considerable site specific and downstream impacts.."}, {"category_id": 15, "poly": [129.0, 1219.0, 732.0, 1219.0, 732.0, 1249.0, 129.0, 1249.0], "score": 0.99, "text": "Sikka et al. (2003) recently showed a change from"}, {"category_id": 15, "poly": [127.0, 1249.0, 734.0, 1247.0, 735.0, 1284.0, 127.0, 1286.0], "score": 1.0, "text": "grassland to Eucalyptus globulus plantations in India"}, {"category_id": 15, "poly": [129.0, 1284.0, 728.0, 1284.0, 728.0, 1314.0, 129.0, 1314.0], "score": 0.98, "text": "decreased alow flow index by a factor of two during the"}, {"category_id": 15, "poly": [127.0, 1316.0, 735.0, 1316.0, 735.0, 1352.0, 127.0, 1352.0], "score": 0.99, "text": "first rotation (9 years), and by 3.75 during the second"}, {"category_id": 15, "poly": [129.0, 1352.0, 732.0, 1352.0, 732.0, 1382.0, 129.0, 1382.0], "score": 1.0, "text": "rotation, with more subdued impact on peak flows. The"}, {"category_id": 15, "poly": [129.0, 1385.0, 732.0, 1385.0, 732.0, 1415.0, 129.0, 1415.0], "score": 0.99, "text": "index was defined as the 10 day average flow exceeded"}, {"category_id": 15, "poly": [125.0, 1447.0, 735.0, 1449.0, 734.0, 1486.0, 125.0, 1483.0], "score": 0.98, "text": "duration curves. Scott and Smith (1997) reported"}, {"category_id": 15, "poly": [129.0, 1486.0, 732.0, 1486.0, 732.0, 1516.0, 129.0, 1516.0], "score": 0.96, "text": "proportionally greater reductions in low fows"}, {"category_id": 15, "poly": [125.0, 1511.0, 737.0, 1514.0, 737.0, 1550.0, 125.0, 1548.0], "score": 0.98, "text": "(75-100th percentiles) than annual flows from South"}, {"category_id": 15, "poly": [127.0, 1548.0, 735.0, 1550.0, 734.0, 1580.0, 127.0, 1578.0], "score": 0.99, "text": "African research catchments under conversions from"}, {"category_id": 15, "poly": [125.0, 1582.0, 737.0, 1580.0, 737.0, 1617.0, 125.0, 1619.0], "score": 0.98, "text": " grass to pine and eucalypt plantations, while Bosch"}, {"category_id": 15, "poly": [129.0, 1619.0, 732.0, 1619.0, 732.0, 1649.0, 129.0, 1649.0], "score": 0.98, "text": "(1979) found the greatest reduction in seasonal flow"}, {"category_id": 15, "poly": [129.0, 1651.0, 732.0, 1651.0, 732.0, 1681.0, 129.0, 1681.0], "score": 0.98, "text": "from the summer wet season. Fahey and Jackson"}, {"category_id": 15, "poly": [125.0, 1679.0, 735.0, 1681.0, 734.0, 1718.0, 125.0, 1716.0], "score": 0.99, "text": "(1997) reported the reduction in peak flows was twice"}, {"category_id": 15, "poly": [129.0, 1718.0, 732.0, 1718.0, 732.0, 1748.0, 129.0, 1748.0], "score": 0.98, "text": "that of total flow and low flows for pine afforestation in"}, {"category_id": 15, "poly": [125.0, 1746.0, 732.0, 1748.0, 732.0, 1785.0, 125.0, 1782.0], "score": 0.98, "text": " New Zealand. The generalisations that can be drawn"}, {"category_id": 15, "poly": [129.0, 1784.0, 728.0, 1784.0, 728.0, 1815.0, 129.0, 1815.0], "score": 0.99, "text": "from annual analyses, where processes and hydrologic"}, {"category_id": 15, "poly": [127.0, 1819.0, 732.0, 1817.0, 732.0, 1847.0, 127.0, 1849.0], "score": 0.99, "text": "responses are to a certain extent integrated may not"}, {"category_id": 15, "poly": [184.0, 1415.0, 732.0, 1417.0, 732.0, 1447.0, 184.0, 1445.0], "score": 0.99, "text": "of the time, obtained from analysis of 10-day flow"}, {"category_id": 15, "poly": [823.0, 1277.0, 1400.0, 1279.0, 1400.0, 1316.0, 823.0, 1314.0], "score": 0.98, "text": " This paper presents the results of a project aimed at"}, {"category_id": 15, "poly": [793.0, 1316.0, 1398.0, 1316.0, 1398.0, 1346.0, 793.0, 1346.0], "score": 0.96, "text": "quantifying changes in annual fow regime of"}, {"category_id": 15, "poly": [793.0, 1350.0, 1398.0, 1350.0, 1398.0, 1380.0, 793.0, 1380.0], "score": 0.99, "text": "catchments following plantation establishment. The"}, {"category_id": 15, "poly": [793.0, 1385.0, 1398.0, 1385.0, 1398.0, 1415.0, 793.0, 1415.0], "score": 0.98, "text": "flow regime is represented by the flow duration curve"}, {"category_id": 15, "poly": [793.0, 1417.0, 1398.0, 1417.0, 1398.0, 1447.0, 793.0, 1447.0], "score": 0.99, "text": "(FDC). The key assumption was that rainfall and"}, {"category_id": 15, "poly": [793.0, 1451.0, 1396.0, 1451.0, 1396.0, 1481.0, 793.0, 1481.0], "score": 0.99, "text": "forest age are the principal drivers of evapotranspira-"}, {"category_id": 15, "poly": [788.0, 1481.0, 1400.0, 1481.0, 1400.0, 1518.0, 788.0, 1518.0], "score": 0.99, "text": "tion. For any generalisation of response of the FDC to"}, {"category_id": 15, "poly": [793.0, 1518.0, 1398.0, 1518.0, 1398.0, 1548.0, 793.0, 1548.0], "score": 0.99, "text": "vegetation change, the variation in the annual climate"}, {"category_id": 15, "poly": [790.0, 1550.0, 1398.0, 1550.0, 1398.0, 1580.0, 790.0, 1580.0], "score": 0.97, "text": "signal must be removed. The time-tested solution to"}, {"category_id": 15, "poly": [790.0, 1585.0, 1398.0, 1585.0, 1398.0, 1615.0, 790.0, 1615.0], "score": 1.0, "text": "this problem is the paired-catchment (control versus"}, {"category_id": 15, "poly": [790.0, 1617.0, 1398.0, 1617.0, 1398.0, 1647.0, 790.0, 1647.0], "score": 0.98, "text": "treatment) experiment. The benefits in such studies"}, {"category_id": 15, "poly": [793.0, 1651.0, 1396.0, 1651.0, 1396.0, 1681.0, 793.0, 1681.0], "score": 0.98, "text": "are manifold: unambiguous measures of trends,"}, {"category_id": 15, "poly": [790.0, 1686.0, 1392.0, 1686.0, 1392.0, 1716.0, 790.0, 1716.0], "score": 0.99, "text": "insights into the processes driving those trends,"}, {"category_id": 15, "poly": [793.0, 1716.0, 1400.0, 1716.0, 1400.0, 1752.0, 793.0, 1752.0], "score": 0.96, "text": "excellent opportunities for model parameterisation"}, {"category_id": 15, "poly": [793.0, 1750.0, 1394.0, 1750.0, 1394.0, 1780.0, 793.0, 1780.0], "score": 0.98, "text": "and validation. However these data are not readily"}, {"category_id": 15, "poly": [790.0, 1784.0, 1390.0, 1784.0, 1390.0, 1815.0, 790.0, 1815.0], "score": 0.99, "text": "available for the range of treamtments and environ-"}, {"category_id": 15, "poly": [790.0, 1817.0, 1396.0, 1817.0, 1396.0, 1847.0, 790.0, 1847.0], "score": 0.99, "text": " ments required. Consequently, the aims of this project"}, {"category_id": 15, "poly": [793.0, 851.0, 1398.0, 851.0, 1398.0, 882.0, 793.0, 882.0], "score": 0.99, "text": "apply on a seasonal or shorter scale. Further, the"}, {"category_id": 15, "poly": [788.0, 879.0, 1398.0, 882.0, 1398.0, 918.0, 788.0, 916.0], "score": 1.0, "text": " observed impacts of any land use change on flows may"}, {"category_id": 15, "poly": [788.0, 916.0, 1400.0, 916.0, 1400.0, 952.0, 788.0, 952.0], "score": 0.96, "text": "be exaggerated or understated depending on the"}, {"category_id": 15, "poly": [788.0, 948.0, 1400.0, 948.0, 1400.0, 985.0, 788.0, 985.0], "score": 0.99, "text": "prevailing climate. Observations of fow during"}, {"category_id": 15, "poly": [793.0, 985.0, 1398.0, 985.0, 1398.0, 1015.0, 793.0, 1015.0], "score": 0.98, "text": "extended wet or dry spells, or with high annual"}, {"category_id": 15, "poly": [793.0, 1017.0, 1398.0, 1017.0, 1398.0, 1047.0, 793.0, 1047.0], "score": 1.0, "text": "variability can obscure the real impacts. Fig. 1 plots"}, {"category_id": 15, "poly": [790.0, 1051.0, 1398.0, 1051.0, 1398.0, 1081.0, 790.0, 1081.0], "score": 0.98, "text": " annual FDCs over 12 years of plantation growth for one"}, {"category_id": 15, "poly": [793.0, 1084.0, 1398.0, 1084.0, 1398.0, 1114.0, 793.0, 1114.0], "score": 0.99, "text": "of the catchments used in this study, Pine Creek. The"}, {"category_id": 15, "poly": [786.0, 1114.0, 1400.0, 1116.0, 1400.0, 1152.0, 786.0, 1150.0], "score": 0.97, "text": " net change in flow is obscured by rainfall variability;"}, {"category_id": 15, "poly": [788.0, 1148.0, 1400.0, 1146.0, 1400.0, 1182.0, 788.0, 1185.0], "score": 1.0, "text": "e.g. the greatest change in the FDC is in 1996, with the"}, {"category_id": 15, "poly": [786.0, 1215.0, 1398.0, 1213.0, 1398.0, 1249.0, 786.0, 1251.0], "score": 0.99, "text": " compared with 2000, where there is substantially"}, {"category_id": 15, "poly": [788.0, 1249.0, 941.0, 1249.0, 941.0, 1279.0, 788.0, 1279.0], "score": 0.99, "text": "higher flows."}, {"category_id": 15, "poly": [788.0, 1180.0, 983.0, 1180.0, 983.0, 1217.0, 788.0, 1217.0], "score": 0.96, "text": "stream flowing"}, {"category_id": 15, "poly": [1066.0, 1180.0, 1400.0, 1180.0, 1400.0, 1217.0, 1066.0, 1217.0], "score": 0.96, "text": " of the time. This may be"}, {"category_id": 15, "poly": [161.0, 318.0, 728.0, 318.0, 728.0, 355.0, 161.0, 355.0], "score": 1.0, "text": "Widespread afforestation through plantation estab-"}, {"category_id": 15, "poly": [125.0, 348.0, 732.0, 350.0, 732.0, 387.0, 125.0, 385.0], "score": 1.0, "text": "lishment on non-forested land represents a potentially"}, {"category_id": 15, "poly": [129.0, 389.0, 732.0, 389.0, 732.0, 417.0, 129.0, 417.0], "score": 0.98, "text": "significant alteration of catchment evapotranspiration"}, {"category_id": 15, "poly": [129.0, 421.0, 730.0, 421.0, 730.0, 452.0, 129.0, 452.0], "score": 0.98, "text": "(ET). Using data collated from multiple catchment"}, {"category_id": 15, "poly": [129.0, 456.0, 732.0, 456.0, 732.0, 484.0, 129.0, 484.0], "score": 0.99, "text": "studies, researchers have demonstrated a consistent"}, {"category_id": 15, "poly": [125.0, 482.0, 737.0, 484.0, 737.0, 520.0, 125.0, 518.0], "score": 0.98, "text": " difference in ET between forests and grass or short "}, {"category_id": 15, "poly": [122.0, 518.0, 734.0, 516.0, 735.0, 553.0, 123.0, 555.0], "score": 0.99, "text": " crops, and the relationship between ET and rainfall on"}, {"category_id": 15, "poly": [127.0, 553.0, 732.0, 553.0, 732.0, 583.0, 127.0, 583.0], "score": 1.0, "text": "a mean annual basis (Holmes and Sinclair, 1986;"}, {"category_id": 15, "poly": [127.0, 585.0, 732.0, 585.0, 732.0, 621.0, 127.0, 621.0], "score": 0.99, "text": "Vertessy and Bessard, 1999; Zhang et al., 1999,"}, {"category_id": 15, "poly": [129.0, 654.0, 732.0, 654.0, 732.0, 684.0, 129.0, 684.0], "score": 0.99, "text": "there is an increasing divergence between forest and"}, {"category_id": 15, "poly": [125.0, 684.0, 734.0, 682.0, 735.0, 718.0, 125.0, 720.0], "score": 0.99, "text": "grassland ET (Zhang et al., 2001). Research from"}, {"category_id": 15, "poly": [127.0, 718.0, 732.0, 718.0, 732.0, 755.0, 127.0, 755.0], "score": 0.98, "text": "South Africa in particular has demonstrated flow"}, {"category_id": 15, "poly": [129.0, 755.0, 730.0, 755.0, 730.0, 785.0, 129.0, 785.0], "score": 1.0, "text": "reduction following afforestation with both pine and"}, {"category_id": 15, "poly": [125.0, 783.0, 732.0, 780.0, 732.0, 817.0, 125.0, 819.0], "score": 0.99, "text": "eucalypt species (Bosch, 1979; Van Lill et al., 1980;"}, {"category_id": 15, "poly": [131.0, 819.0, 732.0, 819.0, 732.0, 849.0, 131.0, 849.0], "score": 0.98, "text": "Van Wyk, 1987; Bosch and Von Gadow, 1990; Scott"}, {"category_id": 15, "poly": [129.0, 854.0, 730.0, 854.0, 730.0, 884.0, 129.0, 884.0], "score": 0.99, "text": "and Smith, 1997; Scott et al., 2000). In regions, where"}, {"category_id": 15, "poly": [129.0, 888.0, 732.0, 888.0, 732.0, 918.0, 129.0, 918.0], "score": 1.0, "text": "water is an increasingly valuable resource, prediction"}, {"category_id": 15, "poly": [125.0, 914.0, 735.0, 916.0, 734.0, 952.0, 125.0, 950.0], "score": 1.0, "text": " of the long-term hydrologic impact of afforestation is"}, {"category_id": 15, "poly": [127.0, 952.0, 732.0, 952.0, 732.0, 983.0, 127.0, 983.0], "score": 1.0, "text": "a prerequisite for the optimal planning of catchment"}, {"category_id": 15, "poly": [126.0, 982.0, 232.0, 987.0, 231.0, 1017.0, 124.0, 1012.0], "score": 0.98, "text": "land use."}, {"category_id": 15, "poly": [129.0, 619.0, 572.0, 619.0, 572.0, 649.0, 129.0, 649.0], "score": 0.97, "text": "2001). Once annual rainfall exceeds "}, {"category_id": 15, "poly": [127.0, 189.0, 172.0, 189.0, 172.0, 234.0, 127.0, 234.0], "score": 0.86, "text": "254"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}], "page_info": {"page_no": 1, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 0, "poly": [117.54735565185547, 651.1103515625, 250.780029296875, 651.1103515625, 250.780029296875, 683.0104370117188, 117.54735565185547, 683.0104370117188], "score": 0.9999984502792358}, {"category_id": 0, "poly": [118.68109130859375, 719.37060546875, 523.2320556640625, 719.37060546875, 523.2320556640625, 748.71435546875, 118.68109130859375, 748.71435546875], "score": 0.9999982714653015}, {"category_id": 1, "poly": [782.3466796875, 254.3662872314453, 1379.406005859375, 254.3662872314453, 1379.406005859375, 382.8451843261719, 782.3466796875, 382.8451843261719], "score": 0.9999969005584717}, {"category_id": 2, "poly": [466.16595458984375, 194.14617919921875, 1030.9322509765625, 194.14617919921875, 1030.9322509765625, 218.86849975585938, 466.16595458984375, 218.86849975585938], "score": 0.9999963641166687}, {"category_id": 9, "poly": [1347.212890625, 1178.8819580078125, 1379.9034423828125, 1178.8819580078125, 1379.9034423828125, 1209.0960693359375, 1347.212890625, 1209.0960693359375], "score": 0.9999951124191284}, {"category_id": 1, "poly": [118.17451477050781, 252.63734436035156, 717.2734375, 252.63734436035156, 717.2734375, 582.23974609375, 118.17451477050781, 582.23974609375], "score": 0.999994158744812}, {"category_id": 1, "poly": [780.9387817382812, 518.9439697265625, 1381.2352294921875, 518.9439697265625, 1381.2352294921875, 1114.6259765625, 780.9387817382812, 1114.6259765625], "score": 0.9999930262565613}, {"category_id": 9, "poly": [1346.75439453125, 438.8963317871094, 1380.3604736328125, 438.8963317871094, 1380.3604736328125, 467.5118713378906, 1346.75439453125, 467.5118713378906], "score": 0.9999922513961792}, {"category_id": 1, "poly": [781.1512451171875, 1283.9832763671875, 1380.4686279296875, 1283.9832763671875, 1380.4686279296875, 1845.6868896484375, 781.1512451171875, 1845.6868896484375], "score": 0.9999905824661255}, {"category_id": 1, "poly": [118.1343994140625, 788.8043212890625, 716.4190673828125, 788.8043212890625, 716.4190673828125, 1282.203125, 118.1343994140625, 1282.203125], "score": 0.9999904632568359}, {"category_id": 2, "poly": [1346.32177734375, 194.7462615966797, 1381.36328125, 194.7462615966797, 1381.36328125, 216.9466552734375, 1346.32177734375, 216.9466552734375], "score": 0.9999903440475464}, {"category_id": 1, "poly": [117.631591796875, 1283.8558349609375, 716.6098022460938, 1283.8558349609375, 716.6098022460938, 1847.49853515625, 117.631591796875, 1847.49853515625], "score": 0.9999891519546509}, {"category_id": 8, "poly": [778.0137939453125, 1156.5975341796875, 1201.7086181640625, 1156.5975341796875, 1201.7086181640625, 1238.48828125, 778.0137939453125, 1238.48828125], "score": 0.9998936653137207}, {"category_id": 8, "poly": [779.0469360351562, 433.1261901855469, 996.4776000976562, 433.1261901855469, 996.4776000976562, 470.7110595703125, 779.0469360351562, 470.7110595703125], "score": 0.979882001876831}, {"category_id": 14, "poly": [777, 1156, 1200, 1156, 1200, 1237, 777, 1237], "score": 0.92, "latex": "Q_{\\mathcal{U}}=a+b(\\Delta P)+\\frac{Y}{1+\\exp\\!\\left(\\frac{T-T_{\\mathrm{half}}}{S}\\right)}"}, {"category_id": 13, "poly": [1150, 520, 1201, 520, 1201, 551, 1150, 551], "score": 0.9, "latex": "f(P)"}, {"category_id": 13, "poly": [1210, 1384, 1262, 1384, 1262, 1414, 1210, 1414], "score": 0.9, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [856, 520, 897, 520, 897, 550, 856, 550], "score": 0.9, "latex": "Q_{\\mathcal{k}}"}, {"category_id": 13, "poly": [930, 552, 982, 552, 982, 584, 930, 584], "score": 0.89, "latex": "g(T)"}, {"category_id": 13, "poly": [857, 1285, 898, 1285, 898, 1315, 857, 1315], "score": 0.89, "latex": "Q_{\\mathcal{k}}"}, {"category_id": 13, "poly": [1196, 1649, 1278, 1649, 1278, 1678, 1196, 1678], "score": 0.89, "latex": "\\Delta P\\!=\\!0"}, {"category_id": 13, "poly": [1270, 1483, 1311, 1483, 1311, 1515, 1270, 1515], "score": 0.89, "latex": "Q_{\\mathrm{\\small{\\mathscr{k}}}}"}, {"category_id": 13, "poly": [1259, 1418, 1301, 1418, 1301, 1449, 1259, 1449], "score": 0.89, "latex": "Q_{\\mathbb{X}}"}, {"category_id": 13, "poly": [1075, 1682, 1140, 1682, 1140, 1711, 1075, 1711], "score": 0.88, "latex": "a+Y."}, {"category_id": 13, "poly": [895, 1483, 976, 1483, 976, 1512, 895, 1512], "score": 0.88, "latex": "\\Delta P\\!=\\!0"}, {"category_id": 13, "poly": [1206, 1285, 1252, 1285, 1252, 1315, 1206, 1315], "score": 0.88, "latex": "Q_{50}"}, {"category_id": 13, "poly": [779, 1682, 821, 1682, 821, 1714, 779, 1714], "score": 0.88, "latex": "Q_{\\mathrm{\\%}}"}, {"category_id": 13, "poly": [1313, 1649, 1374, 1649, 1374, 1678, 1313, 1678], "score": 0.87, "latex": "T{=}0"}, {"category_id": 14, "poly": [777, 432, 997, 432, 997, 470, 777, 470], "score": 0.83, "latex": "\\begin{array}{r}{Q_{\\%}=f(P)+g(T)}\\end{array}"}, {"category_id": 13, "poly": [963, 1350, 1002, 1350, 1002, 1378, 963, 1378], "score": 0.8, "latex": "\\Delta P"}, {"category_id": 13, "poly": [989, 1318, 1012, 1318, 1012, 1345, 989, 1345], "score": 0.64, "latex": "Y"}, {"category_id": 13, "poly": [1077, 1318, 1098, 1318, 1098, 1345, 1077, 1345], "score": 0.64, "latex": "S"}, {"category_id": 13, "poly": [1239, 1583, 1262, 1583, 1262, 1611, 1239, 1611], "score": 0.51, "latex": "S"}, {"category_id": 13, "poly": [989, 1488, 1008, 1488, 1008, 1511, 989, 1511], "score": 0.3, "latex": "a"}, {"category_id": 15, "poly": [112.0, 651.0, 256.0, 651.0, 256.0, 688.0, 112.0, 688.0], "score": 0.96, "text": "2. Methods"}, {"category_id": 15, "poly": [112.0, 716.0, 526.0, 720.0, 526.0, 757.0, 112.0, 752.0], "score": 0.99, "text": "2.1. Characterisation of fow regime"}, {"category_id": 15, "poly": [778.0, 249.0, 1383.0, 252.0, 1383.0, 288.0, 777.0, 286.0], "score": 0.99, "text": " closure, a time term is required to represent plantation"}, {"category_id": 15, "poly": [777.0, 288.0, 1385.0, 284.0, 1385.0, 320.0, 778.0, 325.0], "score": 0.99, "text": "growth. A simple model relating the time series of"}, {"category_id": 15, "poly": [778.0, 318.0, 1383.0, 323.0, 1383.0, 357.0, 777.0, 353.0], "score": 0.99, "text": "each decile with rainfall and vegetation characteristics"}, {"category_id": 15, "poly": [782.0, 357.0, 1018.0, 357.0, 1018.0, 387.0, 782.0, 387.0], "score": 0.99, "text": "can be expressed as:"}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [112.0, 249.0, 722.0, 252.0, 722.0, 288.0, 112.0, 286.0], "score": 0.99, "text": "were to (1) fit a model to the observed annual time"}, {"category_id": 15, "poly": [116.0, 290.0, 719.0, 290.0, 719.0, 320.0, 116.0, 320.0], "score": 0.98, "text": "series of FDC percentiles; i.e. 10th percentile for each"}, {"category_id": 15, "poly": [112.0, 320.0, 722.0, 320.0, 722.0, 357.0, 112.0, 357.0], "score": 0.99, "text": " year of record with annual rainfall and plantation age"}, {"category_id": 15, "poly": [116.0, 355.0, 719.0, 355.0, 719.0, 385.0, 116.0, 385.0], "score": 1.0, "text": "as parameters, (2) replace the annual rainfall variation"}, {"category_id": 15, "poly": [116.0, 389.0, 722.0, 389.0, 722.0, 419.0, 116.0, 419.0], "score": 0.98, "text": "with the long term mean to obtain climate adjusted"}, {"category_id": 15, "poly": [116.0, 421.0, 719.0, 421.0, 719.0, 452.0, 116.0, 452.0], "score": 0.99, "text": "FDCs, and (3) quantify changes in FDC percentiles as"}, {"category_id": 15, "poly": [116.0, 456.0, 717.0, 456.0, 717.0, 486.0, 116.0, 486.0], "score": 1.0, "text": "plantations age. If the climate signal, represented by"}, {"category_id": 15, "poly": [112.0, 482.0, 722.0, 486.0, 721.0, 523.0, 112.0, 518.0], "score": 0.98, "text": "rainfall, could be successfully removed, the resulting"}, {"category_id": 15, "poly": [116.0, 522.0, 719.0, 522.0, 719.0, 553.0, 116.0, 553.0], "score": 0.99, "text": "changes in the FDC would be solely attributable to the"}, {"category_id": 15, "poly": [114.0, 557.0, 243.0, 557.0, 243.0, 587.0, 114.0, 587.0], "score": 1.0, "text": "vegetation."}, {"category_id": 15, "poly": [780.0, 587.0, 1385.0, 587.0, 1385.0, 617.0, 780.0, 617.0], "score": 0.99, "text": "plantation. Annual rainfall was chosen as the rainfall"}, {"category_id": 15, "poly": [780.0, 621.0, 1385.0, 621.0, 1385.0, 649.0, 780.0, 649.0], "score": 0.99, "text": "statistic as it proved to be the most robust predictor of"}, {"category_id": 15, "poly": [780.0, 654.0, 1385.0, 654.0, 1385.0, 684.0, 780.0, 684.0], "score": 0.97, "text": "flow over the whole range of flow percentiles, as"}, {"category_id": 15, "poly": [777.0, 686.0, 1383.0, 686.0, 1383.0, 722.0, 777.0, 722.0], "score": 0.98, "text": " compared with rainfall percentiles; e.g. median rain-"}, {"category_id": 15, "poly": [777.0, 718.0, 1385.0, 718.0, 1385.0, 755.0, 777.0, 755.0], "score": 0.97, "text": "fall versus 10th flow percentile. The use of annual"}, {"category_id": 15, "poly": [775.0, 748.0, 1385.0, 750.0, 1385.0, 787.0, 775.0, 785.0], "score": 0.99, "text": "rainfall also minimises parameter complexity. The"}, {"category_id": 15, "poly": [782.0, 787.0, 1383.0, 787.0, 1383.0, 817.0, 782.0, 817.0], "score": 0.98, "text": "choice of model form is dependent on selecting a"}, {"category_id": 15, "poly": [780.0, 821.0, 1383.0, 821.0, 1383.0, 849.0, 780.0, 849.0], "score": 0.99, "text": "function that describes the relationship between forest"}, {"category_id": 15, "poly": [777.0, 854.0, 1383.0, 851.0, 1383.0, 881.0, 778.0, 884.0], "score": 0.98, "text": "age and ET. Scott and Smith (1997\uff09 demonstrated"}, {"category_id": 15, "poly": [780.0, 886.0, 1383.0, 886.0, 1383.0, 916.0, 780.0, 916.0], "score": 0.98, "text": "cumulative reductions in annual and low flows"}, {"category_id": 15, "poly": [780.0, 920.0, 1383.0, 920.0, 1383.0, 950.0, 780.0, 950.0], "score": 0.98, "text": "resulting from afforestation fitted a sigmoidal"}, {"category_id": 15, "poly": [777.0, 952.0, 1379.0, 952.0, 1379.0, 983.0, 777.0, 983.0], "score": 0.99, "text": "function, similar to forest growth functions. Conse-"}, {"category_id": 15, "poly": [775.0, 985.0, 1385.0, 983.0, 1385.0, 1019.0, 775.0, 1021.0], "score": 0.99, "text": " quently, we used a sigmoidal function to characterise"}, {"category_id": 15, "poly": [780.0, 1019.0, 1381.0, 1019.0, 1381.0, 1049.0, 780.0, 1049.0], "score": 0.99, "text": "the impact of plantation growth on each fow decile."}, {"category_id": 15, "poly": [780.0, 1054.0, 1383.0, 1054.0, 1383.0, 1084.0, 780.0, 1084.0], "score": 0.98, "text": "Fig. 2a is a schematic of the change in the FDC over"}, {"category_id": 15, "poly": [777.0, 1086.0, 1143.0, 1086.0, 1143.0, 1116.0, 777.0, 1116.0], "score": 0.99, "text": "time. The model took the form:"}, {"category_id": 15, "poly": [1202.0, 522.0, 1385.0, 522.0, 1385.0, 550.0, 1202.0, 550.0], "score": 0.99, "text": "is a function of"}, {"category_id": 15, "poly": [782.0, 522.0, 855.0, 522.0, 855.0, 550.0, 782.0, 550.0], "score": 1.0, "text": "where"}, {"category_id": 15, "poly": [898.0, 522.0, 1149.0, 522.0, 1149.0, 550.0, 898.0, 550.0], "score": 0.98, "text": "is the percentile flow,"}, {"category_id": 15, "poly": [780.0, 555.0, 929.0, 555.0, 929.0, 585.0, 780.0, 585.0], "score": 0.95, "text": "rainfall and"}, {"category_id": 15, "poly": [983.0, 555.0, 1383.0, 555.0, 1383.0, 585.0, 983.0, 585.0], "score": 0.98, "text": " is a function of the age of the"}, {"category_id": 15, "poly": [780.0, 1453.0, 1385.0, 1453.0, 1385.0, 1484.0, 780.0, 1484.0], "score": 1.0, "text": "afforestation has taken place. For the average climate"}, {"category_id": 15, "poly": [775.0, 1516.0, 1383.0, 1516.0, 1383.0, 1546.0, 775.0, 1546.0], "score": 0.98, "text": "the new equilibrium plantation water use under"}, {"category_id": 15, "poly": [777.0, 1552.0, 1385.0, 1552.0, 1385.0, 1582.0, 777.0, 1582.0], "score": 0.99, "text": " afforestation is reached. Y then gives the magnitude"}, {"category_id": 15, "poly": [780.0, 1619.0, 1385.0, 1619.0, 1385.0, 1649.0, 780.0, 1649.0], "score": 0.97, "text": "the shape of the response as shown in Fig. 2b. For"}, {"category_id": 15, "poly": [780.0, 1718.0, 1383.0, 1718.0, 1383.0, 1748.0, 780.0, 1748.0], "score": 0.98, "text": "afforestation condition would not require the time"}, {"category_id": 15, "poly": [780.0, 1752.0, 1383.0, 1752.0, 1383.0, 1782.0, 780.0, 1782.0], "score": 0.98, "text": "term. Details of the optimisation scheme and"}, {"category_id": 15, "poly": [780.0, 1784.0, 1383.0, 1784.0, 1383.0, 1815.0, 780.0, 1815.0], "score": 1.0, "text": "sensitivity tests on initial parameter values are given"}, {"category_id": 15, "poly": [780.0, 1817.0, 1020.0, 1817.0, 1020.0, 1847.0, 780.0, 1847.0], "score": 0.97, "text": "in Lane et al. (2003)."}, {"category_id": 15, "poly": [777.0, 1382.0, 1209.0, 1382.0, 1209.0, 1419.0, 777.0, 1419.0], "score": 0.98, "text": "from the period of record average, and"}, {"category_id": 15, "poly": [1263.0, 1382.0, 1385.0, 1382.0, 1385.0, 1419.0, 1263.0, 1419.0], "score": 0.99, "text": "is the time"}, {"category_id": 15, "poly": [782.0, 1286.0, 856.0, 1286.0, 856.0, 1316.0, 782.0, 1316.0], "score": 1.0, "text": "where"}, {"category_id": 15, "poly": [777.0, 1649.0, 1195.0, 1649.0, 1195.0, 1686.0, 777.0, 1686.0], "score": 1.0, "text": "the average pre-treatment condition"}, {"category_id": 15, "poly": [1312.0, 1486.0, 1385.0, 1486.0, 1385.0, 1516.0, 1312.0, 1516.0], "score": 1.0, "text": "when"}, {"category_id": 15, "poly": [780.0, 1419.0, 1258.0, 1419.0, 1258.0, 1449.0, 780.0, 1449.0], "score": 0.97, "text": "in years at which half of the reduction in"}, {"category_id": 15, "poly": [1302.0, 1419.0, 1385.0, 1419.0, 1385.0, 1449.0, 1302.0, 1449.0], "score": 1.0, "text": "due to"}, {"category_id": 15, "poly": [1141.0, 1686.0, 1379.0, 1686.0, 1379.0, 1716.0, 1141.0, 1716.0], "score": 0.95, "text": " Estimation of a pre-"}, {"category_id": 15, "poly": [780.0, 1486.0, 894.0, 1486.0, 894.0, 1516.0, 780.0, 1516.0], "score": 1.0, "text": "condition"}, {"category_id": 15, "poly": [899.0, 1286.0, 1205.0, 1286.0, 1205.0, 1316.0, 899.0, 1316.0], "score": 0.98, "text": "is the percentile flow (i.e."}, {"category_id": 15, "poly": [1253.0, 1286.0, 1383.0, 1286.0, 1383.0, 1316.0, 1253.0, 1316.0], "score": 1.0, "text": "is the 50th"}, {"category_id": 15, "poly": [822.0, 1686.0, 1074.0, 1686.0, 1074.0, 1716.0, 822.0, 1716.0], "score": 0.99, "text": " approximately equals"}, {"category_id": 15, "poly": [1279.0, 1649.0, 1312.0, 1649.0, 1312.0, 1686.0, 1279.0, 1686.0], "score": 1.0, "text": "at"}, {"category_id": 15, "poly": [777.0, 1352.0, 962.0, 1350.0, 962.0, 1380.0, 778.0, 1382.0], "score": 1.0, "text": "sigmoidal term,"}, {"category_id": 15, "poly": [1003.0, 1352.0, 1385.0, 1350.0, 1385.0, 1380.0, 1003.0, 1382.0], "score": 0.99, "text": "is the deviation of annual rainfall"}, {"category_id": 15, "poly": [775.0, 1316.0, 988.0, 1314.0, 988.0, 1350.0, 775.0, 1352.0], "score": 0.97, "text": "percentile flow),"}, {"category_id": 15, "poly": [1013.0, 1316.0, 1076.0, 1314.0, 1076.0, 1350.0, 1013.0, 1352.0], "score": 0.9, "text": " and"}, {"category_id": 15, "poly": [1099.0, 1316.0, 1385.0, 1314.0, 1385.0, 1350.0, 1099.0, 1352.0], "score": 0.98, "text": " are coefficients of the"}, {"category_id": 15, "poly": [780.0, 1587.0, 1238.0, 1587.0, 1238.0, 1617.0, 780.0, 1617.0], "score": 0.99, "text": "of change due to afforestation, and "}, {"category_id": 15, "poly": [1263.0, 1587.0, 1385.0, 1587.0, 1385.0, 1617.0, 1263.0, 1617.0], "score": 0.99, "text": " describes"}, {"category_id": 15, "poly": [1009.0, 1486.0, 1269.0, 1486.0, 1269.0, 1516.0, 1009.0, 1516.0], "score": 1.0, "text": "becomes the value of"}, {"category_id": 15, "poly": [144.0, 783.0, 720.0, 785.0, 719.0, 821.0, 144.0, 819.0], "score": 0.99, "text": "Flow duration curves display the relationship"}, {"category_id": 15, "poly": [116.0, 821.0, 719.0, 821.0, 719.0, 851.0, 116.0, 851.0], "score": 0.96, "text": "between streamflow and the percentage of time"}, {"category_id": 15, "poly": [116.0, 854.0, 717.0, 854.0, 717.0, 884.0, 116.0, 884.0], "score": 0.98, "text": "the streamflow is exceeded as a cumulative density"}, {"category_id": 15, "poly": [116.0, 888.0, 719.0, 888.0, 719.0, 918.0, 116.0, 918.0], "score": 1.0, "text": "function They can be constructed for any time period"}, {"category_id": 15, "poly": [116.0, 920.0, 715.0, 920.0, 715.0, 950.0, 116.0, 950.0], "score": 0.99, "text": "(daily, weekly, monthly, etc.) and provide a graphical"}, {"category_id": 15, "poly": [114.0, 952.0, 717.0, 955.0, 717.0, 985.0, 114.0, 983.0], "score": 0.99, "text": "and statistical view of historic streamflow variability"}, {"category_id": 15, "poly": [114.0, 987.0, 717.0, 987.0, 717.0, 1017.0, 114.0, 1017.0], "score": 0.99, "text": "in a single catchment or a comparison of inter-"}, {"category_id": 15, "poly": [112.0, 1017.0, 722.0, 1017.0, 722.0, 1054.0, 112.0, 1054.0], "score": 0.99, "text": "catchment flow regimes. Vogel and Fennessey (1994)"}, {"category_id": 15, "poly": [110.0, 1047.0, 722.0, 1049.0, 722.0, 1086.0, 109.0, 1084.0], "score": 0.99, "text": "and Smakhtin (1999, 2001) demonstrate the utility"}, {"category_id": 15, "poly": [114.0, 1088.0, 719.0, 1088.0, 719.0, 1118.0, 114.0, 1118.0], "score": 1.0, "text": "(and caveats) of FDCs in characterising, comparing"}, {"category_id": 15, "poly": [114.0, 1120.0, 722.0, 1120.0, 722.0, 1150.0, 114.0, 1150.0], "score": 0.97, "text": "and predicting flow regimes at varying temporal"}, {"category_id": 15, "poly": [112.0, 1150.0, 724.0, 1150.0, 724.0, 1187.0, 112.0, 1187.0], "score": 0.98, "text": "scales. Fig. 1 is an example of annual FDCs"}, {"category_id": 15, "poly": [114.0, 1187.0, 722.0, 1187.0, 722.0, 1217.0, 114.0, 1217.0], "score": 0.99, "text": "constructed from daily flows. For the consideration"}, {"category_id": 15, "poly": [110.0, 1215.0, 722.0, 1217.0, 722.0, 1253.0, 109.0, 1251.0], "score": 0.99, "text": " of annual flow regime, daily fows are an appropriate"}, {"category_id": 15, "poly": [114.0, 1253.0, 477.0, 1253.0, 477.0, 1284.0, 114.0, 1284.0], "score": 0.99, "text": "time step for FDC construction."}, {"category_id": 15, "poly": [1342.0, 189.0, 1387.0, 189.0, 1387.0, 234.0, 1342.0, 234.0], "score": 1.0, "text": "255"}, {"category_id": 15, "poly": [148.0, 1284.0, 715.0, 1284.0, 715.0, 1314.0, 148.0, 1314.0], "score": 0.99, "text": "FDCs were computed from the distribution of daily"}, {"category_id": 15, "poly": [112.0, 1316.0, 720.0, 1320.0, 719.0, 1350.0, 112.0, 1346.0], "score": 1.0, "text": "flows for each year of record based on the appropriate"}, {"category_id": 15, "poly": [116.0, 1352.0, 719.0, 1352.0, 719.0, 1382.0, 116.0, 1382.0], "score": 0.99, "text": "water years (May-April or November-October) for"}, {"category_id": 15, "poly": [112.0, 1380.0, 722.0, 1382.0, 722.0, 1419.0, 112.0, 1417.0], "score": 0.96, "text": "10 Southern Hemisphere catchments. Each 10th"}, {"category_id": 15, "poly": [114.0, 1419.0, 719.0, 1417.0, 720.0, 1447.0, 114.0, 1449.0], "score": 0.97, "text": "percentile (decile\uff09 was extracted from the annual"}, {"category_id": 15, "poly": [112.0, 1449.0, 720.0, 1451.0, 719.0, 1481.0, 112.0, 1479.0], "score": 0.99, "text": "FDCs of each catchment to form the data sets for"}, {"category_id": 15, "poly": [114.0, 1486.0, 719.0, 1486.0, 719.0, 1516.0, 114.0, 1516.0], "score": 0.99, "text": "analysis. For the purpose of characterising changes in"}, {"category_id": 15, "poly": [114.0, 1518.0, 719.0, 1518.0, 719.0, 1546.0, 114.0, 1546.0], "score": 1.0, "text": "each of the deciles, it is assumed that the time series is"}, {"category_id": 15, "poly": [114.0, 1550.0, 719.0, 1550.0, 719.0, 1580.0, 114.0, 1580.0], "score": 0.96, "text": "principally a function of climate and vegetation"}, {"category_id": 15, "poly": [114.0, 1585.0, 722.0, 1585.0, 722.0, 1615.0, 114.0, 1615.0], "score": 0.99, "text": "characteristics. Given rainfall is generally the most"}, {"category_id": 15, "poly": [114.0, 1619.0, 722.0, 1619.0, 722.0, 1649.0, 114.0, 1649.0], "score": 0.97, "text": "important factor affecting streamflow and the most"}, {"category_id": 15, "poly": [116.0, 1651.0, 719.0, 1651.0, 719.0, 1681.0, 116.0, 1681.0], "score": 0.98, "text": "easily accessed data, it is chosen to represent the"}, {"category_id": 15, "poly": [116.0, 1686.0, 719.0, 1686.0, 719.0, 1716.0, 116.0, 1716.0], "score": 1.0, "text": "climate. Catchment physical properties such as soil"}, {"category_id": 15, "poly": [114.0, 1716.0, 722.0, 1716.0, 722.0, 1752.0, 114.0, 1752.0], "score": 0.98, "text": "properties and topography are assumed to be time"}, {"category_id": 15, "poly": [116.0, 1752.0, 719.0, 1752.0, 719.0, 1782.0, 116.0, 1782.0], "score": 0.99, "text": "invariant and therefore their impact on runoff is"}, {"category_id": 15, "poly": [118.0, 1784.0, 719.0, 1784.0, 719.0, 1815.0, 118.0, 1815.0], "score": 1.0, "text": "considered constant throughout the analysis. As trees"}, {"category_id": 15, "poly": [116.0, 1819.0, 715.0, 1819.0, 715.0, 1849.0, 116.0, 1849.0], "score": 1.0, "text": "intercept and transpire at increasing rates until canopy"}], "page_info": {"page_no": 2, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 4, "poly": [129.72743225097656, 1201.7288818359375, 731.863037109375, 1201.7288818359375, 731.863037109375, 1256.5126953125, 129.72743225097656, 1256.5126953125], "score": 0.9999990463256836}, {"category_id": 1, "poly": [130.2001953125, 1783.3021240234375, 730.197509765625, 1783.3021240234375, 730.197509765625, 1846.7928466796875, 130.2001953125, 1846.7928466796875], "score": 0.9999982714653015}, {"category_id": 0, "poly": [797.18896484375, 501.0386047363281, 1060.7071533203125, 501.0386047363281, 1060.7071533203125, 529.1184692382812, 797.18896484375, 529.1184692382812], "score": 0.9999982714653015}, {"category_id": 2, "poly": [130.7757568359375, 195.0663299560547, 166.40858459472656, 195.0663299560547, 166.40858459472656, 215.67367553710938, 130.7757568359375, 215.67367553710938], "score": 0.9999974966049194}, {"category_id": 9, "poly": [1360.5223388671875, 807.8145751953125, 1393.251953125, 807.8145751953125, 1393.251953125, 835.9564819335938, 1360.5223388671875, 835.9564819335938], "score": 0.9999971389770508}, {"category_id": 3, "poly": [140.5875244140625, 256.1985778808594, 711.4976806640625, 256.1985778808594, 711.4976806640625, 1180.2288818359375, 140.5875244140625, 1180.2288818359375], "score": 0.999996542930603}, {"category_id": 1, "poly": [795.8214721679688, 1244.741455078125, 1393.836181640625, 1244.741455078125, 1393.836181640625, 1508.3616943359375, 795.8214721679688, 1508.3616943359375], "score": 0.9999949932098389}, {"category_id": 2, "poly": [480.60809326171875, 195.57171630859375, 1043.654296875, 195.57171630859375, 1043.654296875, 218.8836212158203, 480.60809326171875, 218.8836212158203], "score": 0.9999940395355225}, {"category_id": 1, "poly": [794.92333984375, 878.8724365234375, 1394.78515625, 878.8724365234375, 1394.78515625, 1241.811279296875, 794.92333984375, 1241.811279296875], "score": 0.9999939799308777}, {"category_id": 8, "poly": [792.0145263671875, 779.4395751953125, 1107.5592041015625, 779.4395751953125, 1107.5592041015625, 865.4520263671875, 792.0145263671875, 865.4520263671875], "score": 0.9999935030937195}, {"category_id": 1, "poly": [794.2274780273438, 567.8933715820312, 1393.5377197265625, 567.8933715820312, 1393.5377197265625, 762.2144775390625, 794.2274780273438, 762.2144775390625], "score": 0.9999918341636658}, {"category_id": 1, "poly": [795.5938110351562, 1715.463134765625, 1394.151611328125, 1715.463134765625, 1394.151611328125, 1845.3857421875, 795.5938110351562, 1845.3857421875], "score": 0.999987781047821}, {"category_id": 1, "poly": [794.4356689453125, 255.30477905273438, 1393.678466796875, 255.30477905273438, 1393.678466796875, 447.8646240234375, 794.4356689453125, 447.8646240234375], "score": 0.9999871253967285}, {"category_id": 1, "poly": [130.53660583496094, 1355.89013671875, 730.9114379882812, 1355.89013671875, 730.9114379882812, 1652.1812744140625, 130.53660583496094, 1652.1812744140625], "score": 0.999987006187439}, {"category_id": 9, "poly": [696.6166381835938, 1699.391845703125, 728.77880859375, 1699.391845703125, 728.77880859375, 1727.2147216796875, 696.6166381835938, 1727.2147216796875], "score": 0.999981164932251}, {"category_id": 9, "poly": [1360.9091796875, 1667.6871337890625, 1393.8095703125, 1667.6871337890625, 1393.8095703125, 1699.094482421875, 1360.9091796875, 1699.094482421875], "score": 0.9999788999557495}, {"category_id": 8, "poly": [790.2078857421875, 1522.67236328125, 1111.4049072265625, 1522.67236328125, 1111.4049072265625, 1604.606689453125, 790.2078857421875, 1604.606689453125], "score": 0.9999706149101257}, {"category_id": 9, "poly": [1361.0799560546875, 1545.7677001953125, 1393.7020263671875, 1545.7677001953125, 1393.7020263671875, 1573.452392578125, 1361.0799560546875, 1573.452392578125], "score": 0.9998459815979004}, {"category_id": 8, "poly": [127.09381866455078, 1678.0965576171875, 565.4200439453125, 1678.0965576171875, 565.4200439453125, 1756.1007080078125, 127.09381866455078, 1756.1007080078125], "score": 0.9997967481613159}, {"category_id": 8, "poly": [794.1704711914062, 1666.248779296875, 974.3306274414062, 1666.248779296875, 974.3306274414062, 1700.88720703125, 794.1704711914062, 1700.88720703125], "score": 0.9997556209564209}, {"category_id": 0, "poly": [131.9687042236328, 1288.984375, 435.8473205566406, 1288.984375, 435.8473205566406, 1316.791259765625, 131.9687042236328, 1316.791259765625], "score": 0.9995421767234802}, {"category_id": 1, "poly": [794.0263671875, 1622.5870361328125, 839.6729125976562, 1622.5870361328125, 839.6729125976562, 1647.691650390625, 794.0263671875, 1647.691650390625], "score": 0.9984337687492371}, {"category_id": 14, "poly": [790, 777, 1108, 777, 1108, 863, 790, 863], "score": 0.94, "latex": "E=1.0-\\frac{\\sum_{i=1}^{N}(O_{i}-P_{i})^{2}}{\\sum_{i=1}^{N}(O_{i}-\\bar{O})^{2}}"}, {"category_id": 14, "poly": [790, 1521, 1110, 1521, 1110, 1602, 790, 1602], "score": 0.94, "latex": "Q_{\\mathcal{Q}}=a+\\frac{Y}{1+\\exp\\left(\\frac{T-T_{\\mathrm{half}}}{S}\\right)}"}, {"category_id": 14, "poly": [125, 1674, 566, 1674, 566, 1756, 125, 1756], "score": 0.93, "latex": "N_{\\mathrm{zero}}=a+b(\\Delta P)+\\frac{Y}{1+\\exp\\left(\\frac{T-T_{\\mathrm{half}}}{S}\\right)}"}, {"category_id": 13, "poly": [1306, 319, 1388, 319, 1388, 349, 1306, 349], "score": 0.91, "latex": "\\Delta P\\!=\\!0"}, {"category_id": 13, "poly": [529, 1555, 589, 1555, 589, 1585, 529, 1585], "score": 0.9, "latex": "N_{\\mathrm{zero}}"}, {"category_id": 13, "poly": [1281, 1176, 1365, 1176, 1365, 1205, 1281, 1205], "score": 0.9, "latex": "E\\!>\\!0.7"}, {"category_id": 13, "poly": [880, 1173, 931, 1173, 931, 1206, 880, 1206], "score": 0.89, "latex": "<\\!r^{2}"}, {"category_id": 13, "poly": [873, 1409, 932, 1409, 932, 1438, 873, 1438], "score": 0.89, "latex": "b\\!=\\!0"}, {"category_id": 13, "poly": [597, 1522, 656, 1522, 656, 1552, 597, 1552], "score": 0.89, "latex": "N_{\\mathrm{zero}}"}, {"category_id": 13, "poly": [792, 353, 856, 353, 856, 382, 792, 382], "score": 0.88, "latex": "a+Y"}, {"category_id": 13, "poly": [649, 1782, 731, 1782, 731, 1810, 649, 1810], "score": 0.88, "latex": "\\Delta P\\!=\\!0"}, {"category_id": 14, "poly": [791, 1663, 976, 1663, 976, 1699, 791, 1699], "score": 0.88, "latex": "Q_{\\%}=a+b\\Delta P"}, {"category_id": 13, "poly": [1199, 1409, 1259, 1409, 1259, 1438, 1199, 1438], "score": 0.87, "latex": "Y{=}\\,0"}, {"category_id": 13, "poly": [513, 1487, 585, 1487, 585, 1519, 513, 1519], "score": 0.85, "latex": "(N_{\\mathrm{zero}})"}, {"category_id": 13, "poly": [1335, 1073, 1362, 1073, 1362, 1104, 1335, 1104], "score": 0.84, "latex": "r^{2}"}, {"category_id": 13, "poly": [845, 908, 869, 908, 869, 938, 845, 938], "score": 0.81, "latex": "\\bar{O}"}, {"category_id": 13, "poly": [1123, 880, 1146, 880, 1146, 905, 1123, 905], "score": 0.79, "latex": "P"}, {"category_id": 13, "poly": [1344, 1145, 1367, 1145, 1367, 1171, 1344, 1171], "score": 0.79, "latex": "E"}, {"category_id": 13, "poly": [872, 879, 896, 879, 896, 905, 872, 905], "score": 0.77, "latex": "o"}, {"category_id": 13, "poly": [713, 1521, 731, 1521, 731, 1548, 713, 1548], "score": 0.76, "latex": "b"}, {"category_id": 13, "poly": [1274, 912, 1298, 912, 1298, 938, 1274, 938], "score": 0.76, "latex": "E"}, {"category_id": 13, "poly": [1347, 699, 1369, 699, 1369, 726, 1347, 726], "score": 0.75, "latex": "E"}, {"category_id": 13, "poly": [263, 1815, 326, 1815, 326, 1847, 263, 1847], "score": 0.74, "latex": "N_{\\mathrm{zero}}"}, {"category_id": 13, "poly": [185, 1814, 245, 1814, 245, 1845, 185, 1845], "score": 0.73, "latex": "T\\!\\!=\\!\\!0"}, {"category_id": 13, "poly": [1010, 1819, 1023, 1819, 1023, 1842, 1010, 1842], "score": 0.7, "latex": "t"}, {"category_id": 13, "poly": [1207, 565, 1246, 565, 1246, 596, 1207, 596], "score": 0.67, "latex": "(E)"}, {"category_id": 13, "poly": [1310, 979, 1364, 979, 1364, 1007, 1310, 1007], "score": 0.64, "latex": "-\\infty"}, {"category_id": 13, "poly": [1031, 1754, 1044, 1754, 1044, 1776, 1031, 1776], "score": 0.57, "latex": "t\\cdot"}, {"category_id": 13, "poly": [1313, 1818, 1326, 1818, 1326, 1842, 1313, 1842], "score": 0.57, "latex": "t\\cdot"}, {"category_id": 13, "poly": [960, 1073, 1001, 1073, 1001, 1108, 960, 1108], "score": 0.55, "latex": "(r^{2})"}, {"category_id": 13, "poly": [175, 1555, 194, 1555, 194, 1582, 175, 1582], "score": 0.47, "latex": "S"}, {"category_id": 13, "poly": [1020, 287, 1043, 287, 1043, 315, 1020, 315], "score": 0.38, "latex": "S"}, {"category_id": 13, "poly": [1016, 1076, 1040, 1076, 1040, 1105, 1016, 1105], "score": 0.36, "latex": "E"}, {"category_id": 13, "poly": [599, 1815, 660, 1815, 660, 1845, 599, 1845], "score": 0.35, "latex": "a,~Y"}, {"category_id": 13, "poly": [637, 1816, 660, 1816, 660, 1843, 637, 1843], "score": 0.32, "latex": "Y"}, {"category_id": 13, "poly": [184, 1814, 324, 1814, 324, 1847, 184, 1847], "score": 0.27, "latex": "T\\!\\!=\\!0,\\ N_{\\mathrm{zero}}"}, {"category_id": 15, "poly": [131.0, 1204.0, 732.0, 1204.0, 732.0, 1232.0, 131.0, 1232.0], "score": 1.0, "text": "Fig. 2. (a) Schematic of the change in the FDC over time, and"}, {"category_id": 15, "poly": [129.0, 1227.0, 447.0, 1232.0, 446.0, 1260.0, 129.0, 1255.0], "score": 0.98, "text": "(b) definition of model parameters."}, {"category_id": 15, "poly": [159.0, 1778.0, 648.0, 1778.0, 648.0, 1821.0, 159.0, 1821.0], "score": 0.99, "text": "For the average pre-treatment condition "}, {"category_id": 15, "poly": [327.0, 1819.0, 598.0, 1819.0, 598.0, 1849.0, 327.0, 1849.0], "score": 0.98, "text": " approximately equals"}, {"category_id": 15, "poly": [661.0, 1819.0, 728.0, 1819.0, 728.0, 1849.0, 661.0, 1849.0], "score": 1.0, "text": "gives"}, {"category_id": 15, "poly": [129.0, 1819.0, 183.0, 1819.0, 183.0, 1849.0, 129.0, 1849.0], "score": 0.88, "text": "and "}, {"category_id": 15, "poly": [793.0, 499.0, 1065.0, 499.0, 1065.0, 535.0, 793.0, 535.0], "score": 0.98, "text": "2.3. Statistical analyses"}, {"category_id": 15, "poly": [127.0, 189.0, 172.0, 189.0, 172.0, 228.0, 127.0, 228.0], "score": 1.0, "text": "256"}, {"category_id": 15, "poly": [825.0, 1245.0, 1396.0, 1245.0, 1396.0, 1275.0, 825.0, 1275.0], "score": 0.98, "text": "It is important to assess the significance of the"}, {"category_id": 15, "poly": [790.0, 1279.0, 1396.0, 1279.0, 1396.0, 1309.0, 790.0, 1309.0], "score": 0.97, "text": "model parameters to check the model assumptions"}, {"category_id": 15, "poly": [788.0, 1307.0, 1400.0, 1309.0, 1400.0, 1346.0, 788.0, 1344.0], "score": 1.0, "text": "that rainfall and forest age are driving changes in the"}, {"category_id": 15, "poly": [790.0, 1346.0, 1396.0, 1346.0, 1396.0, 1376.0, 790.0, 1376.0], "score": 0.99, "text": "FDC. The model (2) was split into simplified forms,"}, {"category_id": 15, "poly": [793.0, 1378.0, 1396.0, 1378.0, 1396.0, 1408.0, 793.0, 1408.0], "score": 1.0, "text": "where only the rainfall or time terms were included by"}, {"category_id": 15, "poly": [793.0, 1445.0, 1398.0, 1445.0, 1398.0, 1475.0, 793.0, 1475.0], "score": 0.99, "text": "Eq. (6). The component models (5) and (6) were then"}, {"category_id": 15, "poly": [790.0, 1477.0, 1233.0, 1477.0, 1233.0, 1507.0, 790.0, 1507.0], "score": 1.0, "text": "tested against the complete model, (2)."}, {"category_id": 15, "poly": [790.0, 1408.0, 872.0, 1408.0, 872.0, 1445.0, 790.0, 1445.0], "score": 0.99, "text": "setting"}, {"category_id": 15, "poly": [933.0, 1408.0, 1198.0, 1408.0, 1198.0, 1445.0, 933.0, 1445.0], "score": 0.99, "text": ", as shown in Eq. (5), or"}, {"category_id": 15, "poly": [1260.0, 1408.0, 1400.0, 1408.0, 1400.0, 1445.0, 1260.0, 1445.0], "score": 0.97, "text": "as shown in"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [790.0, 944.0, 1400.0, 944.0, 1400.0, 980.0, 790.0, 980.0], "score": 1.0, "text": "minus the ratio of the mean square error to the"}, {"category_id": 15, "poly": [790.0, 1008.0, 1400.0, 1011.0, 1400.0, 1047.0, 790.0, 1045.0], "score": 0.98, "text": "1.0. Higher values indicate greater agreement between"}, {"category_id": 15, "poly": [788.0, 1041.0, 1403.0, 1043.0, 1402.0, 1079.0, 788.0, 1077.0], "score": 0.97, "text": " observed and predicted data as per the coefficient of "}, {"category_id": 15, "poly": [793.0, 1112.0, 1400.0, 1112.0, 1400.0, 1142.0, 793.0, 1142.0], "score": 0.97, "text": "evaluating hydrologic modelling because it is a"}, {"category_id": 15, "poly": [790.0, 1213.0, 1121.0, 1213.0, 1121.0, 1241.0, 790.0, 1241.0], "score": 0.99, "text": "indicate adequate model fits."}, {"category_id": 15, "poly": [1366.0, 1178.0, 1398.0, 1178.0, 1398.0, 1208.0, 1366.0, 1208.0], "score": 1.0, "text": "to"}, {"category_id": 15, "poly": [793.0, 1178.0, 879.0, 1178.0, 879.0, 1208.0, 793.0, 1208.0], "score": 1.0, "text": "always"}, {"category_id": 15, "poly": [932.0, 1178.0, 1280.0, 1178.0, 1280.0, 1208.0, 932.0, 1208.0], "score": 1.0, "text": "we have arbitrarily considered"}, {"category_id": 15, "poly": [1363.0, 1075.0, 1402.0, 1075.0, 1402.0, 1112.0, 1363.0, 1112.0], "score": 1.0, "text": "in"}, {"category_id": 15, "poly": [788.0, 909.0, 844.0, 909.0, 844.0, 946.0, 788.0, 946.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [1147.0, 875.0, 1398.0, 877.0, 1398.0, 914.0, 1147.0, 912.0], "score": 0.97, "text": " are predicted values,"}, {"category_id": 15, "poly": [793.0, 1146.0, 1343.0, 1146.0, 1343.0, 1176.0, 793.0, 1176.0], "score": 1.0, "text": "measure of the deviation from the 1:1 line. As"}, {"category_id": 15, "poly": [1368.0, 1146.0, 1398.0, 1146.0, 1398.0, 1176.0, 1368.0, 1176.0], "score": 1.0, "text": "is"}, {"category_id": 15, "poly": [788.0, 875.0, 871.0, 877.0, 871.0, 914.0, 788.0, 912.0], "score": 1.0, "text": "where"}, {"category_id": 15, "poly": [897.0, 875.0, 1122.0, 877.0, 1122.0, 914.0, 897.0, 912.0], "score": 1.0, "text": "are observed data,"}, {"category_id": 15, "poly": [870.0, 909.0, 1273.0, 909.0, 1273.0, 946.0, 870.0, 946.0], "score": 0.97, "text": " is the mean for the entire period."}, {"category_id": 15, "poly": [1299.0, 909.0, 1398.0, 909.0, 1398.0, 946.0, 1299.0, 946.0], "score": 1.0, "text": "is unity"}, {"category_id": 15, "poly": [793.0, 980.0, 1309.0, 980.0, 1309.0, 1010.0, 793.0, 1010.0], "score": 1.0, "text": "variance in the observed data, and ranges from"}, {"category_id": 15, "poly": [1365.0, 980.0, 1398.0, 980.0, 1398.0, 1010.0, 1365.0, 1010.0], "score": 1.0, "text": "to"}, {"category_id": 15, "poly": [790.0, 1075.0, 959.0, 1075.0, 959.0, 1112.0, 790.0, 1112.0], "score": 1.0, "text": "determination"}, {"category_id": 15, "poly": [1041.0, 1075.0, 1334.0, 1075.0, 1334.0, 1112.0, 1041.0, 1112.0], "score": 0.98, "text": "is used in preference to"}, {"category_id": 15, "poly": [790.0, 596.0, 1398.0, 598.0, 1398.0, 634.0, 790.0, 632.0], "score": 1.0, "text": "Sutcliffe, 1970; Chiew and McMahon, 1993; Legates"}, {"category_id": 15, "poly": [788.0, 628.0, 1396.0, 632.0, 1396.0, 667.0, 788.0, 662.0], "score": 0.96, "text": " and McCabe, 1999) was used as the ^goodness of fit\u2019"}, {"category_id": 15, "poly": [793.0, 669.0, 1398.0, 669.0, 1398.0, 697.0, 793.0, 697.0], "score": 0.99, "text": "measure to evaluate the fit between observed and"}, {"category_id": 15, "poly": [790.0, 736.0, 905.0, 731.0, 907.0, 763.0, 791.0, 768.0], "score": 0.98, "text": "given by:"}, {"category_id": 15, "poly": [790.0, 701.0, 1346.0, 699.0, 1346.0, 729.0, 790.0, 731.0], "score": 0.99, "text": "predicted flow deciles (2) and zero flow days (3)."}, {"category_id": 15, "poly": [1370.0, 701.0, 1398.0, 699.0, 1398.0, 729.0, 1370.0, 731.0], "score": 1.0, "text": "is"}, {"category_id": 15, "poly": [827.0, 568.0, 1206.0, 568.0, 1206.0, 598.0, 827.0, 598.0], "score": 0.95, "text": "The coefficient of efficiency"}, {"category_id": 15, "poly": [1247.0, 568.0, 1398.0, 568.0, 1398.0, 598.0, 1247.0, 598.0], "score": 0.97, "text": "(Nash and"}, {"category_id": 15, "poly": [825.0, 1716.0, 1394.0, 1716.0, 1394.0, 1752.0, 825.0, 1752.0], "score": 0.99, "text": "For both the fow duration curve analysis and zero"}, {"category_id": 15, "poly": [795.0, 1784.0, 1392.0, 1784.0, 1392.0, 1815.0, 795.0, 1815.0], "score": 0.98, "text": "whether (5) and (6) were significantly different to (2)."}, {"category_id": 15, "poly": [790.0, 1812.0, 1009.0, 1815.0, 1009.0, 1851.0, 790.0, 1849.0], "score": 0.99, "text": "A critical value of"}, {"category_id": 15, "poly": [790.0, 1750.0, 1030.0, 1752.0, 1030.0, 1782.0, 790.0, 1780.0], "score": 1.0, "text": "flow days analysis, a"}, {"category_id": 15, "poly": [1045.0, 1750.0, 1394.0, 1752.0, 1394.0, 1782.0, 1045.0, 1780.0], "score": 0.98, "text": "-test was then performed to test"}, {"category_id": 15, "poly": [1024.0, 1812.0, 1312.0, 1815.0, 1312.0, 1851.0, 1024.0, 1849.0], "score": 1.0, "text": "exceeding the calculated"}, {"category_id": 15, "poly": [1327.0, 1812.0, 1396.0, 1815.0, 1396.0, 1851.0, 1327.0, 1849.0], "score": 1.0, "text": "value"}, {"category_id": 15, "poly": [795.0, 256.0, 1398.0, 256.0, 1398.0, 286.0, 795.0, 286.0], "score": 0.97, "text": "the magnitude of change in zero flow days due to"}, {"category_id": 15, "poly": [790.0, 389.0, 1398.0, 389.0, 1398.0, 419.0, 790.0, 419.0], "score": 0.96, "text": "new equilibrium condition under afforestation is"}, {"category_id": 15, "poly": [790.0, 421.0, 891.0, 421.0, 891.0, 452.0, 790.0, 452.0], "score": 1.0, "text": "reached."}, {"category_id": 15, "poly": [793.0, 322.0, 1305.0, 322.0, 1305.0, 353.0, 793.0, 353.0], "score": 1.0, "text": "response. For the average climate condition"}, {"category_id": 15, "poly": [857.0, 355.0, 1398.0, 355.0, 1398.0, 385.0, 857.0, 385.0], "score": 0.99, "text": "becomes the number of zero flow days when the"}, {"category_id": 15, "poly": [793.0, 290.0, 1019.0, 290.0, 1019.0, 320.0, 793.0, 320.0], "score": 0.98, "text": "afforestation, and"}, {"category_id": 15, "poly": [1044.0, 290.0, 1398.0, 290.0, 1398.0, 320.0, 1044.0, 320.0], "score": 0.95, "text": " describes the shape of the"}, {"category_id": 15, "poly": [157.0, 1350.0, 732.0, 1352.0, 732.0, 1389.0, 157.0, 1387.0], "score": 0.98, "text": " A notable feature of Fig. 1 is the increase in the"}, {"category_id": 15, "poly": [127.0, 1389.0, 735.0, 1389.0, 735.0, 1425.0, 127.0, 1425.0], "score": 0.99, "text": "number of zero fow days. A similar approach to"}, {"category_id": 15, "poly": [129.0, 1423.0, 735.0, 1423.0, 735.0, 1453.0, 129.0, 1453.0], "score": 0.98, "text": "Eq. (2), using an inverse sigmoidal function was"}, {"category_id": 15, "poly": [129.0, 1456.0, 732.0, 1456.0, 732.0, 1486.0, 129.0, 1486.0], "score": 0.98, "text": "employed to assess the impact of afforestation on the"}, {"category_id": 15, "poly": [129.0, 1589.0, 735.0, 1589.0, 735.0, 1619.0, 129.0, 1619.0], "score": 0.99, "text": "rainfall increases, and increases with plantation"}, {"category_id": 15, "poly": [126.0, 1624.0, 220.0, 1618.0, 222.0, 1651.0, 128.0, 1656.0], "score": 1.0, "text": "growth:"}, {"category_id": 15, "poly": [590.0, 1557.0, 732.0, 1557.0, 732.0, 1587.0, 590.0, 1587.0], "score": 1.0, "text": "decreases as"}, {"category_id": 15, "poly": [129.0, 1524.0, 596.0, 1524.0, 596.0, 1554.0, 129.0, 1554.0], "score": 0.98, "text": "the left hand side of Eq. (2) is replaced by"}, {"category_id": 15, "poly": [129.0, 1490.0, 512.0, 1490.0, 512.0, 1520.0, 129.0, 1520.0], "score": 0.99, "text": "number of zero flow days per year"}, {"category_id": 15, "poly": [586.0, 1490.0, 732.0, 1490.0, 732.0, 1520.0, 586.0, 1520.0], "score": 0.97, "text": ". In this case,"}, {"category_id": 15, "poly": [657.0, 1524.0, 712.0, 1524.0, 712.0, 1554.0, 657.0, 1554.0], "score": 0.97, "text": ", and"}, {"category_id": 15, "poly": [129.0, 1557.0, 174.0, 1557.0, 174.0, 1587.0, 129.0, 1587.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [195.0, 1557.0, 528.0, 1557.0, 528.0, 1587.0, 195.0, 1587.0], "score": 0.99, "text": "are constrained to negative as"}, {"category_id": 15, "poly": [129.0, 1288.0, 438.0, 1288.0, 438.0, 1324.0, 129.0, 1324.0], "score": 0.99, "text": "2.2. Zero fow day analysis"}, {"category_id": 15, "poly": [788.0, 1617.0, 844.0, 1617.0, 844.0, 1662.0, 788.0, 1662.0], "score": 1.0, "text": "and"}], "page_info": {"page_no": 3, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 1, "poly": [780.4981079101562, 951.0537109375, 1382.5201416015625, 951.0537109375, 1382.5201416015625, 1648.58154296875, 780.4981079101562, 1648.58154296875], "score": 0.9999959468841553}, {"category_id": 2, "poly": [466.9576110839844, 194.6658935546875, 1030.6968994140625, 194.6658935546875, 1030.6968994140625, 219.20504760742188, 466.9576110839844, 219.20504760742188], "score": 0.9999955892562866}, {"category_id": 0, "poly": [782.29931640625, 886.77197265625, 919.8917236328125, 886.77197265625, 919.8917236328125, 915.8782348632812, 782.29931640625, 915.8782348632812], "score": 0.9999911189079285}, {"category_id": 1, "poly": [782.0343017578125, 253.53207397460938, 1382.2440185546875, 253.53207397460938, 1382.2440185546875, 350.4256896972656, 782.0343017578125, 350.4256896972656], "score": 0.9999889731407166}, {"category_id": 1, "poly": [781.621826171875, 653.5359497070312, 1381.272705078125, 653.5359497070312, 1381.272705078125, 783.18798828125, 781.621826171875, 783.18798828125], "score": 0.9999889731407166}, {"category_id": 5, "poly": [182.05813598632812, 248.86912536621094, 650.3305053710938, 248.86912536621094, 650.3305053710938, 1845.613037109375, 182.05813598632812, 1845.613037109375], "score": 0.9999887347221375}, {"category_id": 1, "poly": [781.0881958007812, 1650.3038330078125, 1382.088134765625, 1650.3038330078125, 1382.088134765625, 1848.214111328125, 781.0881958007812, 1848.214111328125], "score": 0.9999865293502808}, {"category_id": 2, "poly": [1346.05322265625, 194.5203399658203, 1381.46875, 194.5203399658203, 1381.46875, 216.90557861328125, 1346.05322265625, 216.90557861328125], "score": 0.9999804496765137}, {"category_id": 8, "poly": [779.12451171875, 544.6279296875, 1165.58349609375, 544.6279296875, 1165.58349609375, 623.5341796875, 779.12451171875, 623.5341796875], "score": 0.9999717473983765}, {"category_id": 1, "poly": [781.6971435546875, 352.1080017089844, 1382.5953369140625, 352.1080017089844, 1382.5953369140625, 515.912109375, 781.6971435546875, 515.912109375], "score": 0.999969482421875}, {"category_id": 9, "poly": [1347.20849609375, 571.1251831054688, 1380.7503662109375, 571.1251831054688, 1380.7503662109375, 601.0969848632812, 1347.20849609375, 601.0969848632812], "score": 0.9999024868011475}, {"category_id": 7, "poly": [659.8250732421875, 882.5633544921875, 686.8219604492188, 882.5633544921875, 686.8219604492188, 1842.583251953125, 659.8250732421875, 1842.583251953125], "score": 0.9764553904533386}, {"category_id": 6, "poly": [112.29073333740234, 1497.288330078125, 169.8206329345703, 1497.288330078125, 169.8206329345703, 1843.9019775390625, 112.29073333740234, 1843.9019775390625], "score": 0.8885180950164795}, {"category_id": 14, "poly": [776, 546, 1164, 546, 1164, 622, 776, 622], "score": 0.91, "latex": "F=\\frac{[(\\mathrm{SSE_{s}-S S E_{c}})/(\\mathrm{df_{c}-d f_{s}})]}{\\mathrm{SSE_{c}/d f_{c}}}"}, {"category_id": 13, "poly": [1087, 415, 1135, 415, 1135, 447, 1087, 447], "score": 0.88, "latex": "F^{0.5}"}, {"category_id": 13, "poly": [1155, 1183, 1223, 1183, 1223, 1214, 1155, 1214], "score": 0.86, "latex": "100\\%"}, {"category_id": 13, "poly": [779, 1781, 820, 1781, 820, 1812, 779, 1812], "score": 0.82, "latex": "6\\%"}, {"category_id": 13, "poly": [831, 487, 852, 487, 852, 513, 831, 513], "score": 0.77, "latex": "F"}, {"category_id": 13, "poly": [1120, 390, 1133, 390, 1133, 413, 1120, 413], "score": 0.72, "latex": "t\\cdot"}, {"category_id": 13, "poly": [780, 423, 792, 423, 792, 446, 780, 446], "score": 0.49, "latex": "t\\cdot"}, {"category_id": 13, "poly": [1074, 1716, 1095, 1716, 1095, 1742, 1074, 1742], "score": 0.31, "latex": "P"}, {"category_id": 15, "poly": [814.0, 952.0, 1383.0, 952.0, 1383.0, 983.0, 814.0, 983.0], "score": 0.98, "text": "Daily streamflow data were obtained from 10"}, {"category_id": 15, "poly": [782.0, 987.0, 1383.0, 987.0, 1383.0, 1017.0, 782.0, 1017.0], "score": 0.99, "text": "catchment studies from southeastern Australia, New"}, {"category_id": 15, "poly": [780.0, 1019.0, 1383.0, 1019.0, 1383.0, 1049.0, 780.0, 1049.0], "score": 0.99, "text": "Zealand and South Africa. The initial criteria for"}, {"category_id": 15, "poly": [775.0, 1047.0, 1383.0, 1051.0, 1383.0, 1088.0, 775.0, 1084.0], "score": 0.99, "text": "selection of these catchments were a known veg-"}, {"category_id": 15, "poly": [780.0, 1088.0, 1385.0, 1088.0, 1385.0, 1118.0, 780.0, 1118.0], "score": 0.95, "text": "etation history and streamflow records of good"}, {"category_id": 15, "poly": [780.0, 1120.0, 1381.0, 1120.0, 1381.0, 1150.0, 780.0, 1150.0], "score": 0.99, "text": "quality. The ideal data sets were those with a lengthy"}, {"category_id": 15, "poly": [777.0, 1155.0, 1385.0, 1152.0, 1385.0, 1182.0, 778.0, 1185.0], "score": 0.97, "text": "pre- and post-treatment (plantation establishment)"}, {"category_id": 15, "poly": [775.0, 1215.0, 1387.0, 1217.0, 1387.0, 1253.0, 775.0, 1251.0], "score": 0.99, "text": " ment converted from grassland or a crop equivalent to"}, {"category_id": 15, "poly": [780.0, 1253.0, 1385.0, 1253.0, 1385.0, 1284.0, 780.0, 1284.0], "score": 0.99, "text": "plantation. In reality, all these criteria were not easy to"}, {"category_id": 15, "poly": [782.0, 1286.0, 1383.0, 1286.0, 1383.0, 1316.0, 782.0, 1316.0], "score": 0.99, "text": "satisfy. For example in Victoria, Australia, the best"}, {"category_id": 15, "poly": [780.0, 1320.0, 1385.0, 1320.0, 1385.0, 1348.0, 780.0, 1348.0], "score": 0.99, "text": "data is from Stewarts Creek, a set of decommissioned"}, {"category_id": 15, "poly": [780.0, 1352.0, 1385.0, 1352.0, 1385.0, 1382.0, 780.0, 1382.0], "score": 0.99, "text": "research catchments with 9 years of pre-treatment"}, {"category_id": 15, "poly": [780.0, 1387.0, 1383.0, 1387.0, 1383.0, 1417.0, 780.0, 1417.0], "score": 1.0, "text": "data and 25 years of post-treatment. Here, though, the"}, {"category_id": 15, "poly": [778.0, 1417.0, 1383.0, 1419.0, 1383.0, 1449.0, 777.0, 1447.0], "score": 0.98, "text": "treatment was a conversion from native eucalypt"}, {"category_id": 15, "poly": [780.0, 1453.0, 1385.0, 1453.0, 1385.0, 1484.0, 780.0, 1484.0], "score": 0.99, "text": "forest to pine. The assumption made for this data set is"}, {"category_id": 15, "poly": [780.0, 1486.0, 1383.0, 1486.0, 1383.0, 1516.0, 780.0, 1516.0], "score": 0.98, "text": "that the immediate post-treatment period may be"}, {"category_id": 15, "poly": [780.0, 1518.0, 1383.0, 1518.0, 1383.0, 1548.0, 780.0, 1548.0], "score": 0.99, "text": "viewed as a non-forested condition. This condition is"}, {"category_id": 15, "poly": [777.0, 1552.0, 1385.0, 1552.0, 1385.0, 1582.0, 777.0, 1582.0], "score": 0.99, "text": "likely to approximate the ET conditions of pasture or"}, {"category_id": 15, "poly": [780.0, 1587.0, 1385.0, 1587.0, 1385.0, 1617.0, 780.0, 1617.0], "score": 0.98, "text": "short crops for up to 3 years. Catchment details and"}, {"category_id": 15, "poly": [775.0, 1619.0, 1145.0, 1617.0, 1145.0, 1647.0, 775.0, 1649.0], "score": 1.0, "text": "treatments are given in Table 1."}, {"category_id": 15, "poly": [780.0, 1187.0, 1154.0, 1187.0, 1154.0, 1217.0, 780.0, 1217.0], "score": 0.98, "text": "flow record with approximately"}, {"category_id": 15, "poly": [1224.0, 1187.0, 1381.0, 1187.0, 1381.0, 1217.0, 1224.0, 1217.0], "score": 0.99, "text": "of the catch-"}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [776.0, 881.0, 924.0, 886.0, 923.0, 925.0, 775.0, 920.0], "score": 0.96, "text": " 3. Data sets"}, {"category_id": 15, "poly": [782.0, 254.0, 1383.0, 254.0, 1383.0, 290.0, 782.0, 290.0], "score": 0.99, "text": "when comparing (5) and (2) would indicate the time"}, {"category_id": 15, "poly": [782.0, 290.0, 1381.0, 290.0, 1381.0, 320.0, 782.0, 320.0], "score": 1.0, "text": "term in (6) was required to improve the complete"}, {"category_id": 15, "poly": [778.0, 318.0, 1347.0, 320.0, 1347.0, 357.0, 777.0, 355.0], "score": 1.0, "text": "model and is therefore significant, and vice versa."}, {"category_id": 15, "poly": [782.0, 656.0, 1385.0, 656.0, 1385.0, 686.0, 782.0, 686.0], "score": 1.0, "text": "where SSE is the residual sum of the squared errors, df"}, {"category_id": 15, "poly": [780.0, 686.0, 1383.0, 686.0, 1383.0, 722.0, 780.0, 722.0], "score": 0.99, "text": "is degrees of freedom, and the subscripts s and c refer"}, {"category_id": 15, "poly": [778.0, 720.0, 1379.0, 722.0, 1379.0, 753.0, 777.0, 750.0], "score": 0.98, "text": "to the simplified model and complete models,"}, {"category_id": 15, "poly": [777.0, 752.0, 926.0, 752.0, 926.0, 789.0, 777.0, 789.0], "score": 0.97, "text": "respectively."}, {"category_id": 15, "poly": [816.0, 1651.0, 1381.0, 1651.0, 1381.0, 1681.0, 816.0, 1681.0], "score": 0.99, "text": "All catchments, with the exception of Traralgon"}, {"category_id": 15, "poly": [782.0, 1686.0, 1379.0, 1686.0, 1379.0, 1716.0, 782.0, 1716.0], "score": 1.0, "text": "Creek, were afforested with pine species, predomi-"}, {"category_id": 15, "poly": [782.0, 1752.0, 1379.0, 1752.0, 1379.0, 1782.0, 782.0, 1782.0], "score": 1.0, "text": "Cathedral Peak catchments. Traralgon Creek has only"}, {"category_id": 15, "poly": [776.0, 1812.0, 1147.0, 1817.0, 1147.0, 1851.0, 775.0, 1847.0], "score": 0.99, "text": "of which is Eucalyptus regnans."}, {"category_id": 15, "poly": [821.0, 1784.0, 1381.0, 1784.0, 1381.0, 1815.0, 821.0, 1815.0], "score": 0.99, "text": " pine, with the remainder eucalypts species, most"}, {"category_id": 15, "poly": [780.0, 1718.0, 1073.0, 1718.0, 1073.0, 1748.0, 780.0, 1748.0], "score": 0.97, "text": "nantly Pinus radiata, with"}, {"category_id": 15, "poly": [1096.0, 1718.0, 1381.0, 1718.0, 1381.0, 1748.0, 1096.0, 1748.0], "score": 0.96, "text": "patula planted at the two"}, {"category_id": 15, "poly": [1340.0, 189.0, 1387.0, 189.0, 1387.0, 239.0, 1340.0, 239.0], "score": 1.0, "text": "257"}, {"category_id": 15, "poly": [814.0, 355.0, 1383.0, 355.0, 1383.0, 385.0, 814.0, 385.0], "score": 0.98, "text": "Due to the constraint that the rainfall and time term"}, {"category_id": 15, "poly": [780.0, 456.0, 1381.0, 456.0, 1381.0, 486.0, 780.0, 486.0], "score": 0.97, "text": "the critical value for significance at the 0.05 level."}, {"category_id": 15, "poly": [1136.0, 415.0, 1383.0, 417.0, 1383.0, 454.0, 1136.0, 451.0], "score": 0.98, "text": ", and compared with"}, {"category_id": 15, "poly": [780.0, 486.0, 830.0, 488.0, 830.0, 518.0, 780.0, 516.0], "score": 1.0, "text": "The"}, {"category_id": 15, "poly": [853.0, 486.0, 1160.0, 488.0, 1160.0, 518.0, 853.0, 516.0], "score": 0.99, "text": "-statistic was calculated as:"}, {"category_id": 15, "poly": [782.0, 389.0, 1119.0, 389.0, 1119.0, 419.0, 782.0, 419.0], "score": 0.99, "text": "must be positive, a one tailed"}, {"category_id": 15, "poly": [1134.0, 389.0, 1381.0, 389.0, 1381.0, 419.0, 1134.0, 419.0], "score": 0.98, "text": "-test was applied. The"}, {"category_id": 15, "poly": [793.0, 415.0, 1086.0, 417.0, 1086.0, 454.0, 793.0, 451.0], "score": 0.96, "text": "-value was calculated as"}], "page_info": {"page_no": 4, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 1, "poly": [795.8170166015625, 1284.819091796875, 1393.7825927734375, 1284.819091796875, 1393.7825927734375, 1348.0101318359375, 795.8170166015625, 1348.0101318359375], "score": 0.999998927116394}, {"category_id": 0, "poly": [796.8157348632812, 1217.8375244140625, 1043.37890625, 1217.8375244140625, 1043.37890625, 1247.9854736328125, 796.8157348632812, 1247.9854736328125], "score": 0.9999985694885254}, {"category_id": 2, "poly": [129.85670471191406, 196.5391082763672, 165.74688720703125, 196.5391082763672, 165.74688720703125, 216.1761932373047, 129.85670471191406, 216.1761932373047], "score": 0.9999985098838806}, {"category_id": 6, "poly": [129.43519592285156, 1376.390380859375, 374.6851806640625, 1376.390380859375, 374.6851806640625, 1428.6553955078125, 129.43519592285156, 1428.6553955078125], "score": 0.9999979734420776}, {"category_id": 0, "poly": [796.9744873046875, 1150.6444091796875, 911.8010864257812, 1150.6444091796875, 911.8010864257812, 1181.93017578125, 796.9744873046875, 1181.93017578125], "score": 0.9999944567680359}, {"category_id": 1, "poly": [793.93212890625, 785.8515625, 1394.5316162109375, 785.8515625, 1394.5316162109375, 1081.7857666015625, 793.93212890625, 1081.7857666015625], "score": 0.9999939203262329}, {"category_id": 5, "poly": [123.9719009399414, 1433.21337890625, 1400.553466796875, 1433.21337890625, 1400.553466796875, 1814.204345703125, 123.9719009399414, 1814.204345703125], "score": 0.9999938011169434}, {"category_id": 1, "poly": [130.5817108154297, 786.2269897460938, 730.632080078125, 786.2269897460938, 730.632080078125, 1349.01904296875, 130.5817108154297, 1349.01904296875], "score": 0.9999901652336121}, {"category_id": 6, "poly": [128.56288146972656, 253.8047637939453, 514.1275024414062, 253.8047637939453, 514.1275024414062, 308.0131530761719, 128.56288146972656, 308.0131530761719], "score": 0.9999885559082031}, {"category_id": 5, "poly": [126.36900329589844, 314.1026611328125, 1399.912109375, 314.1026611328125, 1399.912109375, 690.7048950195312, 126.36900329589844, 690.7048950195312], "score": 0.9999723434448242}, {"category_id": 2, "poly": [479.1275939941406, 195.43199157714844, 1044.5283203125, 195.43199157714844, 1044.5283203125, 218.68853759765625, 479.1275939941406, 218.68853759765625], "score": 0.9999346733093262}, {"category_id": 7, "poly": [128.99925231933594, 698.5426635742188, 1394.448486328125, 698.5426635742188, 1394.448486328125, 749.8440551757812, 128.99925231933594, 749.8440551757812], "score": 0.9987799525260925}, {"category_id": 7, "poly": [127.37924194335938, 1819.0853271484375, 1038.7354736328125, 1819.0853271484375, 1038.7354736328125, 1844.94091796875, 127.37924194335938, 1844.94091796875], "score": 0.9987504482269287}, {"category_id": 13, "poly": [626, 696, 660, 696, 660, 720, 626, 720], "score": 0.86, "latex": "5\\%"}, {"category_id": 13, "poly": [1190, 697, 1224, 697, 1224, 720, 1190, 720], "score": 0.86, "latex": "5\\%"}, {"category_id": 13, "poly": [299, 724, 342, 724, 342, 748, 299, 748], "score": 0.85, "latex": "10\\%"}, {"category_id": 13, "poly": [128, 698, 146, 698, 146, 719, 128, 719], "score": 0.69, "latex": "P"}, {"category_id": 13, "poly": [719, 697, 737, 697, 737, 719, 719, 719], "score": 0.44, "latex": "T"}, {"category_id": 13, "poly": [356, 1404, 375, 1404, 375, 1426, 356, 1426], "score": 0.33, "latex": "E"}, {"category_id": 15, "poly": [827.0, 1286.0, 1396.0, 1286.0, 1396.0, 1316.0, 827.0, 1316.0], "score": 0.95, "text": "The fit of the complete model, Eq. (2), to the"}, {"category_id": 15, "poly": [795.0, 1322.0, 1394.0, 1322.0, 1394.0, 1352.0, 795.0, 1352.0], "score": 0.98, "text": "observed data was generally good. Table 2 gives"}, {"category_id": 15, "poly": [795.0, 1221.0, 1046.0, 1221.0, 1046.0, 1251.0, 795.0, 1251.0], "score": 1.0, "text": "4.1. Model evaluation"}, {"category_id": 15, "poly": [127.0, 189.0, 172.0, 189.0, 172.0, 228.0, 127.0, 228.0], "score": 1.0, "text": "258"}, {"category_id": 15, "poly": [131.0, 1378.0, 204.0, 1378.0, 204.0, 1402.0, 131.0, 1402.0], "score": 0.93, "text": "Table 3"}, {"category_id": 15, "poly": [127.0, 1397.0, 355.0, 1400.0, 355.0, 1436.0, 127.0, 1434.0], "score": 0.97, "text": "Coefficient of efficiency,"}, {"category_id": 15, "poly": [790.0, 1150.0, 915.0, 1150.0, 915.0, 1189.0, 790.0, 1189.0], "score": 1.0, "text": "4. Results"}, {"category_id": 15, "poly": [793.0, 787.0, 1396.0, 787.0, 1396.0, 817.0, 793.0, 817.0], "score": 0.99, "text": "and Redhill the lower BFI is matched by the shallow"}, {"category_id": 15, "poly": [793.0, 821.0, 1398.0, 821.0, 1398.0, 849.0, 793.0, 849.0], "score": 0.99, "text": "soils. Pre-treatment data is not available for all"}, {"category_id": 15, "poly": [793.0, 854.0, 1398.0, 854.0, 1398.0, 884.0, 793.0, 884.0], "score": 0.98, "text": "catchment in the data set, so it was decided for the"}, {"category_id": 15, "poly": [793.0, 886.0, 1396.0, 886.0, 1396.0, 916.0, 793.0, 916.0], "score": 1.0, "text": "sake of consistency in the analysis to start each of the"}, {"category_id": 15, "poly": [795.0, 920.0, 1396.0, 920.0, 1396.0, 950.0, 795.0, 950.0], "score": 0.98, "text": "data sets in the year of treatment. The FDCs were"}, {"category_id": 15, "poly": [795.0, 955.0, 1396.0, 955.0, 1396.0, 985.0, 795.0, 985.0], "score": 0.99, "text": "constructed for water years of May-April for eight"}, {"category_id": 15, "poly": [793.0, 987.0, 1396.0, 987.0, 1396.0, 1017.0, 793.0, 1017.0], "score": 1.0, "text": "catchments. The 2 Cathedral Peak catchments were"}, {"category_id": 15, "poly": [790.0, 1019.0, 1398.0, 1019.0, 1398.0, 1049.0, 790.0, 1049.0], "score": 0.98, "text": "analysed for November-October because of the"}, {"category_id": 15, "poly": [790.0, 1054.0, 1192.0, 1054.0, 1192.0, 1084.0, 790.0, 1084.0], "score": 1.0, "text": "summer rainfall maxima (Table 2)."}, {"category_id": 15, "poly": [163.0, 785.0, 732.0, 785.0, 732.0, 815.0, 163.0, 815.0], "score": 0.98, "text": "Data on soil characteristics have been obtained"}, {"category_id": 15, "poly": [125.0, 815.0, 735.0, 817.0, 734.0, 854.0, 125.0, 851.0], "score": 1.0, "text": "from published reports and personal communication"}, {"category_id": 15, "poly": [125.0, 849.0, 732.0, 851.0, 732.0, 888.0, 125.0, 886.0], "score": 1.0, "text": "with researchers, but is far from uniform, particularly"}, {"category_id": 15, "poly": [124.0, 886.0, 734.0, 881.0, 735.0, 918.0, 125.0, 922.0], "score": 1.0, "text": "regarding porosity. Consequently only an indication"}, {"category_id": 15, "poly": [129.0, 920.0, 732.0, 920.0, 732.0, 950.0, 129.0, 950.0], "score": 0.99, "text": "of mean depth is reported here. However, this does"}, {"category_id": 15, "poly": [125.0, 950.0, 732.0, 952.0, 732.0, 989.0, 125.0, 987.0], "score": 0.99, "text": " give some indication of the likely relative storage"}, {"category_id": 15, "poly": [129.0, 987.0, 732.0, 987.0, 732.0, 1017.0, 129.0, 1017.0], "score": 0.99, "text": "capacities of the catchments. To obtain insights into"}, {"category_id": 15, "poly": [129.0, 1021.0, 732.0, 1021.0, 732.0, 1051.0, 129.0, 1051.0], "score": 0.97, "text": "the pre-afforestation hydrologic characteristics a"}, {"category_id": 15, "poly": [129.0, 1054.0, 732.0, 1054.0, 732.0, 1084.0, 129.0, 1084.0], "score": 0.99, "text": "baseflow separation was performed on the daily"}, {"category_id": 15, "poly": [129.0, 1088.0, 732.0, 1088.0, 732.0, 1118.0, 129.0, 1118.0], "score": 0.97, "text": "fows for the first 3 years following disturbance,"}, {"category_id": 15, "poly": [129.0, 1118.0, 730.0, 1118.0, 730.0, 1148.0, 129.0, 1148.0], "score": 0.98, "text": "using the digital filtering method of Lyne and Hollick"}, {"category_id": 15, "poly": [129.0, 1152.0, 732.0, 1152.0, 732.0, 1182.0, 129.0, 1182.0], "score": 0.98, "text": "(1979) with a filter coefficient of 0.925 and three"}, {"category_id": 15, "poly": [125.0, 1185.0, 734.0, 1182.0, 735.0, 1219.0, 125.0, 1221.0], "score": 0.99, "text": " passes. The resultant average basefow index (BFI),"}, {"category_id": 15, "poly": [129.0, 1221.0, 730.0, 1221.0, 730.0, 1249.0, 129.0, 1249.0], "score": 0.98, "text": "the ratio of baseflow to total flow, is given in Table 1."}, {"category_id": 15, "poly": [129.0, 1251.0, 730.0, 1251.0, 730.0, 1281.0, 129.0, 1281.0], "score": 0.97, "text": "The Australian catchments display a notably"}, {"category_id": 15, "poly": [129.0, 1288.0, 732.0, 1288.0, 732.0, 1316.0, 129.0, 1316.0], "score": 0.98, "text": "lower BFI than the South African and New Zealand"}, {"category_id": 15, "poly": [127.0, 1320.0, 734.0, 1318.0, 735.0, 1348.0, 127.0, 1350.0], "score": 0.96, "text": "catchments. For Stewarts Creek, Pine Creek"}, {"category_id": 15, "poly": [129.0, 252.0, 208.0, 252.0, 208.0, 282.0, 129.0, 282.0], "score": 0.98, "text": "Table 2"}, {"category_id": 15, "poly": [129.0, 282.0, 513.0, 282.0, 513.0, 312.0, 129.0, 312.0], "score": 0.99, "text": "Significance of the rainfall and time terms"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [1225.0, 697.0, 1396.0, 697.0, 1396.0, 727.0, 1225.0, 727.0], "score": 0.98, "text": "level, * represents"}, {"category_id": 15, "poly": [129.0, 725.0, 298.0, 725.0, 298.0, 755.0, 129.0, 755.0], "score": 0.98, "text": "significance at the"}, {"category_id": 15, "poly": [343.0, 725.0, 941.0, 725.0, 941.0, 755.0, 343.0, 755.0], "score": 0.99, "text": "level, and na denotes too few data points for meaningful analysis."}, {"category_id": 15, "poly": [147.0, 697.0, 625.0, 697.0, 625.0, 727.0, 147.0, 727.0], "score": 0.98, "text": " indicates that the rainfall term was significant at the"}, {"category_id": 15, "poly": [661.0, 697.0, 718.0, 697.0, 718.0, 727.0, 661.0, 727.0], "score": 1.0, "text": "level,"}, {"category_id": 15, "poly": [738.0, 697.0, 1189.0, 697.0, 1189.0, 727.0, 738.0, 727.0], "score": 0.99, "text": "indicates that the time term was significant at the"}, {"category_id": 15, "poly": [129.0, 1821.0, 1037.0, 1821.0, 1037.0, 1849.0, 129.0, 1849.0], "score": 0.99, "text": "ns Indicates that no solution was found, and na denotes deciles with too few data points for analysis"}], "page_info": {"page_no": 5, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 2, "poly": [1345.084228515625, 193.99124145507812, 1383.04443359375, 193.99124145507812, 1383.04443359375, 217.28871154785156, 1345.084228515625, 217.28871154785156], "score": 0.9999984502792358}, {"category_id": 1, "poly": [778.7415161132812, 875.8572387695312, 1385.70263671875, 875.8572387695312, 1385.70263671875, 1045.03857421875, 778.7415161132812, 1045.03857421875], "score": 0.9999930262565613}, {"category_id": 1, "poly": [112.97018432617188, 850.864990234375, 721.0302124023438, 850.864990234375, 721.0302124023438, 1216.21875, 112.97018432617188, 1216.21875], "score": 0.9999922513961792}, {"category_id": 4, "poly": [777.7315673828125, 753.8668212890625, 1386.6640625, 753.8668212890625, 1386.6640625, 842.7579345703125, 777.7315673828125, 842.7579345703125], "score": 0.9999915957450867}, {"category_id": 1, "poly": [777.9397583007812, 1045.28857421875, 1386.9669189453125, 1045.28857421875, 1386.9669189453125, 1678.6064453125, 777.9397583007812, 1678.6064453125], "score": 0.9999915957450867}, {"category_id": 1, "poly": [112.64908599853516, 250.50961303710938, 720.302001953125, 250.50961303710938, 720.302001953125, 849.3114624023438, 112.64908599853516, 849.3114624023438], "score": 0.9999906420707703}, {"category_id": 1, "poly": [112.41944122314453, 1315.5491943359375, 721.3580932617188, 1315.5491943359375, 721.3580932617188, 1851.324462890625, 112.41944122314453, 1851.324462890625], "score": 0.9999880790710449}, {"category_id": 3, "poly": [776.4273681640625, 253.75418090820312, 1388.254638671875, 253.75418090820312, 1388.254638671875, 736.9627685546875, 776.4273681640625, 736.9627685546875], "score": 0.9999828338623047}, {"category_id": 2, "poly": [464.4588928222656, 193.45211791992188, 1032.725341796875, 193.45211791992188, 1032.725341796875, 219.19715881347656, 464.4588928222656, 219.19715881347656], "score": 0.9999587535858154}, {"category_id": 0, "poly": [115.3223876953125, 1251.6119384765625, 695.34326171875, 1251.6119384765625, 695.34326171875, 1287.6334228515625, 115.3223876953125, 1287.6334228515625], "score": 0.9989659786224365}, {"category_id": 1, "poly": [778.8644409179688, 1705.2630615234375, 1386.922119140625, 1705.2630615234375, 1386.922119140625, 1843.95654296875, 778.8644409179688, 1843.95654296875], "score": 0.99659264087677}, {"category_id": 13, "poly": [601, 1814, 711, 1814, 711, 1847, 601, 1847], "score": 0.9, "latex": "T{=}\\,2T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [878, 1079, 975, 1079, 975, 1110, 878, 1110], "score": 0.9, "latex": "Y/(Y+a)"}, {"category_id": 13, "poly": [780, 880, 833, 880, 833, 911, 780, 911], "score": 0.89, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [296, 319, 380, 319, 380, 349, 296, 349], "score": 0.89, "latex": "E\\!>\\!0.7"}, {"category_id": 13, "poly": [160, 1682, 231, 1682, 231, 1713, 160, 1713], "score": 0.88, "latex": "a+Y)"}, {"category_id": 13, "poly": [116, 320, 188, 320, 188, 351, 116, 351], "score": 0.88, "latex": "(77\\%)"}, {"category_id": 13, "poly": [268, 751, 324, 751, 324, 781, 268, 781], "score": 0.87, "latex": "80\\%"}, {"category_id": 13, "poly": [628, 585, 684, 585, 684, 615, 628, 615], "score": 0.87, "latex": "75\\%"}, {"category_id": 13, "poly": [602, 619, 644, 619, 644, 647, 602, 647], "score": 0.85, "latex": "9\\%"}, {"category_id": 13, "poly": [533, 784, 577, 784, 577, 814, 533, 814], "score": 0.83, "latex": "9\\%"}, {"category_id": 13, "poly": [323, 1384, 364, 1384, 364, 1412, 323, 1412], "score": 0.77, "latex": "\\Delta P"}, {"category_id": 13, "poly": [286, 852, 308, 852, 308, 879, 286, 879], "score": 0.75, "latex": "E"}, {"category_id": 13, "poly": [409, 885, 432, 885, 432, 912, 409, 912], "score": 0.71, "latex": "E"}, {"category_id": 13, "poly": [566, 1085, 590, 1085, 590, 1112, 566, 1112], "score": 0.7, "latex": "E"}, {"category_id": 13, "poly": [484, 254, 524, 254, 524, 284, 484, 284], "score": 0.7, "latex": "(E)"}, {"category_id": 13, "poly": [315, 919, 334, 919, 334, 946, 315, 946], "score": 0.66, "latex": "b"}, {"category_id": 13, "poly": [376, 587, 394, 587, 394, 614, 376, 614], "score": 0.62, "latex": "b"}, {"category_id": 13, "poly": [460, 1051, 478, 1051, 478, 1077, 460, 1077], "score": 0.59, "latex": "b"}, {"category_id": 13, "poly": [451, 319, 552, 319, 552, 350, 451, 350], "score": 0.46, "latex": "60\\%~0.8"}, {"category_id": 13, "poly": [498, 719, 522, 719, 522, 746, 498, 746], "score": 0.45, "latex": "Y"}, {"category_id": 15, "poly": [1342.0, 191.0, 1387.0, 191.0, 1387.0, 236.0, 1342.0, 236.0], "score": 1.0, "text": "259"}, {"category_id": 15, "poly": [780.0, 914.0, 1383.0, 914.0, 1383.0, 944.0, 780.0, 944.0], "score": 0.99, "text": "most deciles the adjusted FDCs are identical for 12"}, {"category_id": 15, "poly": [780.0, 948.0, 1381.0, 948.0, 1381.0, 978.0, 780.0, 978.0], "score": 0.98, "text": "and 20 years after treatment. This figure clearly"}, {"category_id": 15, "poly": [782.0, 983.0, 1381.0, 983.0, 1381.0, 1013.0, 782.0, 1013.0], "score": 0.98, "text": "demonstrates the necessity for FDC adjustment,"}, {"category_id": 15, "poly": [773.0, 1013.0, 1168.0, 1010.0, 1168.0, 1047.0, 773.0, 1049.0], "score": 0.99, "text": " particularly for the 20 years FDC."}, {"category_id": 15, "poly": [834.0, 882.0, 1383.0, 882.0, 1383.0, 912.0, 834.0, 912.0], "score": 0.99, "text": "values are given in Table 4. Fig. 3 shows that for"}, {"category_id": 15, "poly": [116.0, 955.0, 719.0, 955.0, 719.0, 985.0, 116.0, 985.0], "score": 0.98, "text": "general the model fits the higher flows (lower deciles)"}, {"category_id": 15, "poly": [116.0, 987.0, 719.0, 987.0, 719.0, 1017.0, 116.0, 1017.0], "score": 1.0, "text": "better, most of the poorer fits are in the 80-100"}, {"category_id": 15, "poly": [112.0, 1017.0, 722.0, 1015.0, 722.0, 1051.0, 112.0, 1054.0], "score": 0.97, "text": " percentile range. This can be expected given the results"}, {"category_id": 15, "poly": [116.0, 1120.0, 722.0, 1120.0, 722.0, 1150.0, 116.0, 1150.0], "score": 0.99, "text": "Glendhu 2 and for 10th and 20th percentiles from"}, {"category_id": 15, "poly": [114.0, 1150.0, 724.0, 1150.0, 724.0, 1187.0, 114.0, 1187.0], "score": 0.98, "text": "Cathedral Peak 3 may exaggerate the goodness of fit to"}, {"category_id": 15, "poly": [114.0, 1187.0, 646.0, 1187.0, 646.0, 1215.0, 114.0, 1215.0], "score": 0.98, "text": "the exact form of the model (Lane et al., 2003)."}, {"category_id": 15, "poly": [150.0, 854.0, 285.0, 854.0, 285.0, 884.0, 150.0, 884.0], "score": 1.0, "text": "The poorest"}, {"category_id": 15, "poly": [309.0, 854.0, 715.0, 854.0, 715.0, 884.0, 309.0, 884.0], "score": 0.99, "text": "values were those from Lambrechts-"}, {"category_id": 15, "poly": [116.0, 886.0, 408.0, 886.0, 408.0, 916.0, 116.0, 916.0], "score": 0.95, "text": "bos A and B. The high"}, {"category_id": 15, "poly": [433.0, 886.0, 719.0, 886.0, 719.0, 916.0, 433.0, 916.0], "score": 0.96, "text": "for 50-100th deciles at"}, {"category_id": 15, "poly": [114.0, 1088.0, 565.0, 1088.0, 565.0, 1118.0, 114.0, 1118.0], "score": 0.97, "text": "sensitivity analysis suggested that the"}, {"category_id": 15, "poly": [591.0, 1088.0, 722.0, 1088.0, 722.0, 1118.0, 591.0, 1118.0], "score": 0.97, "text": "values for"}, {"category_id": 15, "poly": [116.0, 920.0, 314.0, 920.0, 314.0, 950.0, 116.0, 950.0], "score": 1.0, "text": "Biesievlei, where"}, {"category_id": 15, "poly": [335.0, 920.0, 719.0, 920.0, 719.0, 950.0, 335.0, 950.0], "score": 0.99, "text": "was not significant are notable. In"}, {"category_id": 15, "poly": [114.0, 1054.0, 459.0, 1054.0, 459.0, 1084.0, 114.0, 1084.0], "score": 0.95, "text": "of the significance tests for "}, {"category_id": 15, "poly": [479.0, 1054.0, 719.0, 1054.0, 719.0, 1084.0, 479.0, 1084.0], "score": 0.96, "text": ". The results of the"}, {"category_id": 15, "poly": [780.0, 759.0, 1383.0, 759.0, 1383.0, 789.0, 780.0, 789.0], "score": 0.98, "text": "Fig. 3. Examples of observed and fow duration curves adjusted for"}, {"category_id": 15, "poly": [782.0, 787.0, 1381.0, 787.0, 1381.0, 815.0, 782.0, 815.0], "score": 0.97, "text": "average rainfall following afforestation for Stewarts Creek 5,"}, {"category_id": 15, "poly": [779.0, 810.0, 873.0, 815.0, 871.0, 845.0, 777.0, 840.0], "score": 1.0, "text": "Australia."}, {"category_id": 15, "poly": [810.0, 1043.0, 1385.0, 1045.0, 1385.0, 1081.0, 810.0, 1079.0], "score": 0.98, "text": "The relative net flow change due to afforestation is"}, {"category_id": 15, "poly": [775.0, 1109.0, 1387.0, 1112.0, 1387.0, 1148.0, 775.0, 1146.0], "score": 0.99, "text": " old equilibrium water use condition of pre-treatment"}, {"category_id": 15, "poly": [777.0, 1146.0, 1387.0, 1142.0, 1388.0, 1178.0, 778.0, 1183.0], "score": 1.0, "text": "vegetation to the new equilibrium condition at forest"}, {"category_id": 15, "poly": [777.0, 1178.0, 1385.0, 1176.0, 1385.0, 1213.0, 778.0, 1215.0], "score": 0.99, "text": "canopy closure. This quantity is plotted for all catchments"}, {"category_id": 15, "poly": [780.0, 1215.0, 1385.0, 1215.0, 1385.0, 1245.0, 780.0, 1245.0], "score": 0.98, "text": "in Fig. 4. Some deciles have been removed from the data"}, {"category_id": 15, "poly": [782.0, 1247.0, 1383.0, 1247.0, 1383.0, 1277.0, 782.0, 1277.0], "score": 0.98, "text": "set, the 10th and 50th percentile for Glendhu 2 and the"}, {"category_id": 15, "poly": [782.0, 1281.0, 1383.0, 1281.0, 1383.0, 1312.0, 782.0, 1312.0], "score": 1.0, "text": "10th and 20th percentiles from Cathedral Peak 3. The"}, {"category_id": 15, "poly": [780.0, 1314.0, 1383.0, 1314.0, 1383.0, 1344.0, 780.0, 1344.0], "score": 0.98, "text": "optimised value of a was zero or near zero for these cases,"}, {"category_id": 15, "poly": [780.0, 1348.0, 1383.0, 1348.0, 1383.0, 1378.0, 780.0, 1378.0], "score": 1.0, "text": "which is not consistent with the conceptual model. The"}, {"category_id": 15, "poly": [775.0, 1378.0, 1385.0, 1376.0, 1385.0, 1413.0, 775.0, 1415.0], "score": 0.98, "text": " changes shown in Fig. 4 are variable. However, there are"}, {"category_id": 15, "poly": [780.0, 1415.0, 1385.0, 1415.0, 1385.0, 1445.0, 780.0, 1445.0], "score": 1.0, "text": "some commonalities between catchment responses. Two"}, {"category_id": 15, "poly": [775.0, 1447.0, 1387.0, 1445.0, 1387.0, 1481.0, 775.0, 1484.0], "score": 0.99, "text": "types of responses (groups) were identified. Group 1"}, {"category_id": 15, "poly": [780.0, 1481.0, 1385.0, 1481.0, 1385.0, 1511.0, 780.0, 1511.0], "score": 1.0, "text": "catchments show a substantial increase in the number of"}, {"category_id": 15, "poly": [777.0, 1514.0, 1387.0, 1514.0, 1387.0, 1550.0, 777.0, 1550.0], "score": 0.96, "text": " zero flow days, with a greater proportional reduction in"}, {"category_id": 15, "poly": [782.0, 1548.0, 1385.0, 1548.0, 1385.0, 1578.0, 782.0, 1578.0], "score": 0.99, "text": "low flows than high fows. Group 2 catchments show a"}, {"category_id": 15, "poly": [780.0, 1582.0, 1383.0, 1582.0, 1383.0, 1612.0, 780.0, 1612.0], "score": 0.99, "text": "more uniform proportional reduction in fows across all"}, {"category_id": 15, "poly": [777.0, 1617.0, 1383.0, 1615.0, 1383.0, 1645.0, 778.0, 1647.0], "score": 1.0, "text": "percentiles, albeit with some variability. The catchments"}, {"category_id": 15, "poly": [776.0, 1644.0, 980.0, 1649.0, 979.0, 1686.0, 775.0, 1681.0], "score": 0.95, "text": " in each group are:"}, {"category_id": 15, "poly": [780.0, 1079.0, 877.0, 1079.0, 877.0, 1116.0, 780.0, 1116.0], "score": 1.0, "text": "givenby"}, {"category_id": 15, "poly": [976.0, 1079.0, 1385.0, 1079.0, 1385.0, 1116.0, 976.0, 1116.0], "score": 0.98, "text": ", which represents the change from the"}, {"category_id": 15, "poly": [116.0, 288.0, 717.0, 288.0, 717.0, 318.0, 116.0, 318.0], "score": 0.99, "text": "percentile at all the catchments. The majority of fits"}, {"category_id": 15, "poly": [116.0, 355.0, 719.0, 355.0, 719.0, 385.0, 116.0, 385.0], "score": 0.99, "text": "significance of the rainfall and time terms is given in"}, {"category_id": 15, "poly": [114.0, 387.0, 717.0, 387.0, 717.0, 417.0, 114.0, 417.0], "score": 1.0, "text": "Table 3 for all deciles, where solutions were found."}, {"category_id": 15, "poly": [112.0, 417.0, 720.0, 421.0, 719.0, 452.0, 112.0, 447.0], "score": 0.98, "text": "There were not enough data to fit the model in five"}, {"category_id": 15, "poly": [116.0, 456.0, 717.0, 456.0, 717.0, 484.0, 116.0, 484.0], "score": 0.98, "text": "instances because of extended periods of zero flows."}, {"category_id": 15, "poly": [116.0, 488.0, 719.0, 488.0, 719.0, 518.0, 116.0, 518.0], "score": 0.99, "text": "This problem is addressed to some extent in the zero"}, {"category_id": 15, "poly": [116.0, 522.0, 719.0, 522.0, 719.0, 550.0, 116.0, 550.0], "score": 1.0, "text": "fow analysis. If the rainfall signal is to be separated"}, {"category_id": 15, "poly": [114.0, 555.0, 719.0, 555.0, 719.0, 585.0, 114.0, 585.0], "score": 0.98, "text": "from the vegetation signal the rainfall terms must be"}, {"category_id": 15, "poly": [110.0, 649.0, 722.0, 651.0, 722.0, 688.0, 109.0, 686.0], "score": 0.99, "text": " 0.10 level. The incidence of significance was greatest"}, {"category_id": 15, "poly": [116.0, 688.0, 719.0, 688.0, 719.0, 718.0, 116.0, 718.0], "score": 0.99, "text": "for the 10-50th percentiles at 45 of the 50 data sets at"}, {"category_id": 15, "poly": [112.0, 817.0, 432.0, 815.0, 432.0, 851.0, 112.0, 854.0], "score": 0.98, "text": "significant at the 0.10 level."}, {"category_id": 15, "poly": [189.0, 320.0, 295.0, 320.0, 295.0, 350.0, 189.0, 350.0], "score": 1.0, "text": "returned"}, {"category_id": 15, "poly": [116.0, 755.0, 267.0, 755.0, 267.0, 785.0, 116.0, 785.0], "score": 1.0, "text": "results, with"}, {"category_id": 15, "poly": [325.0, 755.0, 719.0, 755.0, 719.0, 785.0, 325.0, 785.0], "score": 0.97, "text": " of the deciles significant at 0.05"}, {"category_id": 15, "poly": [685.0, 589.0, 722.0, 589.0, 722.0, 619.0, 685.0, 619.0], "score": 1.0, "text": "of"}, {"category_id": 15, "poly": [114.0, 621.0, 601.0, 621.0, 601.0, 649.0, 114.0, 649.0], "score": 0.98, "text": "the deciles at the 0.05 level, and a further"}, {"category_id": 15, "poly": [645.0, 621.0, 719.0, 621.0, 719.0, 649.0, 645.0, 649.0], "score": 0.99, "text": "at the"}, {"category_id": 15, "poly": [116.0, 787.0, 532.0, 787.0, 532.0, 815.0, 116.0, 815.0], "score": 0.96, "text": "level. There were an additional"}, {"category_id": 15, "poly": [578.0, 787.0, 719.0, 787.0, 719.0, 815.0, 578.0, 815.0], "score": 0.99, "text": "of deciles"}, {"category_id": 15, "poly": [110.0, 249.0, 483.0, 252.0, 483.0, 288.0, 109.0, 286.0], "score": 0.91, "text": "the coefficient of efficiency "}, {"category_id": 15, "poly": [525.0, 249.0, 720.0, 252.0, 719.0, 288.0, 525.0, 286.0], "score": 0.96, "text": "for each flow"}, {"category_id": 15, "poly": [116.0, 589.0, 375.0, 589.0, 375.0, 619.0, 116.0, 619.0], "score": 1.0, "text": "significant. This term,"}, {"category_id": 15, "poly": [395.0, 589.0, 627.0, 589.0, 627.0, 619.0, 395.0, 619.0], "score": 0.98, "text": ", was significant for"}, {"category_id": 15, "poly": [381.0, 320.0, 450.0, 320.0, 450.0, 350.0, 381.0, 350.0], "score": 0.9, "text": "\uff0cwith"}, {"category_id": 15, "poly": [553.0, 320.0, 719.0, 320.0, 719.0, 350.0, 553.0, 350.0], "score": 0.97, "text": "or better. The"}, {"category_id": 15, "poly": [116.0, 718.0, 497.0, 718.0, 497.0, 748.0, 116.0, 748.0], "score": 0.99, "text": "the 0.05 level. The time term,"}, {"category_id": 15, "poly": [523.0, 718.0, 717.0, 718.0, 717.0, 748.0, 523.0, 748.0], "score": 0.97, "text": "returned similar"}, {"category_id": 15, "poly": [148.0, 1318.0, 719.0, 1318.0, 719.0, 1348.0, 148.0, 1348.0], "score": 0.95, "text": "Following the successful fitting of (2) to the"}, {"category_id": 15, "poly": [114.0, 1352.0, 719.0, 1352.0, 719.0, 1382.0, 114.0, 1382.0], "score": 0.99, "text": "observed percentiles, the FDCs were adjusted for"}, {"category_id": 15, "poly": [114.0, 1419.0, 719.0, 1419.0, 719.0, 1447.0, 114.0, 1447.0], "score": 0.99, "text": "average annual rainfall. The climate adjusted FDCs"}, {"category_id": 15, "poly": [114.0, 1453.0, 719.0, 1453.0, 719.0, 1481.0, 114.0, 1481.0], "score": 0.96, "text": "produce an estimation of the change in flow"}, {"category_id": 15, "poly": [112.0, 1486.0, 722.0, 1483.0, 722.0, 1514.0, 112.0, 1516.0], "score": 0.98, "text": "percentiles over time for each catchment due to"}, {"category_id": 15, "poly": [116.0, 1518.0, 719.0, 1518.0, 719.0, 1548.0, 116.0, 1548.0], "score": 0.99, "text": "afforestation that may be viewed in two forms: new"}, {"category_id": 15, "poly": [114.0, 1552.0, 722.0, 1552.0, 722.0, 1580.0, 114.0, 1580.0], "score": 0.99, "text": "FDCs, adjusted for climate, as exemplified in Fig. 3"}, {"category_id": 15, "poly": [112.0, 1587.0, 721.0, 1582.0, 722.0, 1612.0, 112.0, 1617.0], "score": 0.99, "text": "for Stewarts Creek 5, and a comparison between all"}, {"category_id": 15, "poly": [114.0, 1619.0, 717.0, 1619.0, 717.0, 1649.0, 114.0, 1649.0], "score": 0.99, "text": "catchments of the maximum change in yield (given by"}, {"category_id": 15, "poly": [118.0, 1651.0, 722.0, 1651.0, 722.0, 1681.0, 118.0, 1681.0], "score": 0.99, "text": "Y) for each flow percentile from baseline flows (given"}, {"category_id": 15, "poly": [118.0, 1718.0, 715.0, 1718.0, 715.0, 1748.0, 118.0, 1748.0], "score": 0.98, "text": "equilibrium of maximum water use is reached, the"}, {"category_id": 15, "poly": [116.0, 1750.0, 719.0, 1750.0, 719.0, 1780.0, 116.0, 1780.0], "score": 0.99, "text": "adjusted FDCs for individual years should be identical"}, {"category_id": 15, "poly": [114.0, 1784.0, 719.0, 1784.0, 719.0, 1815.0, 114.0, 1815.0], "score": 0.98, "text": "if rainfall variability has been accounted for. The new"}, {"category_id": 15, "poly": [110.0, 1808.0, 600.0, 1813.0, 600.0, 1856.0, 109.0, 1851.0], "score": 0.96, "text": " equilibrium is approximately reached for "}, {"category_id": 15, "poly": [116.0, 1686.0, 159.0, 1686.0, 159.0, 1716.0, 116.0, 1716.0], "score": 1.0, "text": "by"}, {"category_id": 15, "poly": [232.0, 1686.0, 719.0, 1686.0, 719.0, 1716.0, 232.0, 1716.0], "score": 0.95, "text": " as shown in Fig. 4. Where the new"}, {"category_id": 15, "poly": [116.0, 1387.0, 322.0, 1387.0, 322.0, 1417.0, 116.0, 1417.0], "score": 1.0, "text": "climate by setting"}, {"category_id": 15, "poly": [365.0, 1387.0, 719.0, 1387.0, 719.0, 1417.0, 365.0, 1417.0], "score": 0.99, "text": "to zero, representing long term"}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [112.0, 1249.0, 694.0, 1251.0, 694.0, 1288.0, 112.0, 1286.0], "score": 0.99, "text": "4.2. Adjusted FDCs\u2014magnitude of fow reductions"}, {"category_id": 15, "poly": [782.0, 1709.0, 1325.0, 1709.0, 1325.0, 1739.0, 782.0, 1739.0], "score": 0.99, "text": "Group 1: Stewarts Creek, Pine Creek, and Redhill"}, {"category_id": 15, "poly": [782.0, 1744.0, 1381.0, 1744.0, 1381.0, 1774.0, 782.0, 1774.0], "score": 0.99, "text": "Group 2: Cathedral Peak 2 and 3, Lambrechtsbos A,"}, {"category_id": 15, "poly": [889.0, 1776.0, 1383.0, 1776.0, 1383.0, 1806.0, 889.0, 1806.0], "score": 1.0, "text": "Lambrechtsbos B, Glendhu 2, Biesievlei and"}, {"category_id": 15, "poly": [889.0, 1810.0, 1072.0, 1810.0, 1072.0, 1840.0, 889.0, 1840.0], "score": 1.0, "text": "Traralgon Creek"}], "page_info": {"page_no": 6, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 6, "poly": [130.38211059570312, 1373.58056640625, 408.3111877441406, 1373.58056640625, 408.3111877441406, 1427.8253173828125, 130.38211059570312, 1427.8253173828125], "score": 0.9999985694885254}, {"category_id": 2, "poly": [131.90994262695312, 195.1804962158203, 165.77700805664062, 195.1804962158203, 165.77700805664062, 215.41661071777344, 131.90994262695312, 215.41661071777344], "score": 0.9999985098838806}, {"category_id": 2, "poly": [481.0845642089844, 195.8048095703125, 1043.9552001953125, 195.8048095703125, 1043.9552001953125, 218.32778930664062, 481.0845642089844, 218.32778930664062], "score": 0.9999977350234985}, {"category_id": 5, "poly": [124.61016845703125, 1434.242919921875, 1399.2454833984375, 1434.242919921875, 1399.2454833984375, 1811.951171875, 124.61016845703125, 1811.951171875], "score": 0.9999969005584717}, {"category_id": 4, "poly": [510.5360107421875, 734.0053100585938, 1013.9042358398438, 734.0053100585938, 1013.9042358398438, 758.7108154296875, 510.5360107421875, 758.7108154296875], "score": 0.9999968409538269}, {"category_id": 1, "poly": [131.32168579101562, 838.9521484375, 730.0957641601562, 838.9521484375, 730.0957641601562, 1314.5084228515625, 131.32168579101562, 1314.5084228515625], "score": 0.9999938011169434}, {"category_id": 3, "poly": [306.1774597167969, 253.64524841308594, 1219.746337890625, 253.64524841308594, 1219.746337890625, 705.4325561523438, 306.1774597167969, 705.4325561523438], "score": 0.9999911785125732}, {"category_id": 1, "poly": [794.51171875, 907.3822631835938, 1395.8782958984375, 907.3822631835938, 1395.8782958984375, 1313.7686767578125, 794.51171875, 1313.7686767578125], "score": 0.9999873042106628}, {"category_id": 7, "poly": [127.02899169921875, 1816.2164306640625, 940.5137939453125, 1816.2164306640625, 940.5137939453125, 1842.17822265625, 127.02899169921875, 1842.17822265625], "score": 0.999592661857605}, {"category_id": 0, "poly": [794.436767578125, 838.79541015625, 1117.543701171875, 838.79541015625, 1117.543701171875, 867.4995727539062, 794.436767578125, 867.4995727539062], "score": 0.9990140795707703}, {"category_id": 13, "poly": [759, 733, 840, 733, 840, 759, 759, 759], "score": 0.9, "latex": "Y/(Y+a)"}, {"category_id": 13, "poly": [815, 1077, 867, 1077, 867, 1108, 815, 1108], "score": 0.89, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [1088, 1179, 1140, 1179, 1140, 1211, 1088, 1211], "score": 0.89, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [130, 1247, 196, 1247, 196, 1277, 130, 1277], "score": 0.84, "latex": "100\\%"}, {"category_id": 13, "poly": [209, 1042, 276, 1042, 276, 1072, 209, 1072], "score": 0.84, "latex": "100\\%"}, {"category_id": 13, "poly": [1174, 940, 1224, 940, 1224, 971, 1174, 971], "score": 0.84, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [129, 1401, 172, 1401, 172, 1428, 129, 1428], "score": 0.7, "latex": "T_{\\mathrm{half}}"}, {"category_id": 15, "poly": [129.0, 1372.0, 208.0, 1372.0, 208.0, 1402.0, 129.0, 1402.0], "score": 0.93, "text": "Table 4"}, {"category_id": 15, "poly": [173.0, 1400.0, 408.0, 1397.0, 408.0, 1434.0, 173.0, 1436.0], "score": 1.0, "text": "(years) for all catchments"}, {"category_id": 15, "poly": [127.0, 189.0, 172.0, 189.0, 172.0, 228.0, 127.0, 228.0], "score": 1.0, "text": "260"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [509.0, 733.0, 758.0, 733.0, 758.0, 763.0, 509.0, 763.0], "score": 0.98, "text": "Fig. 4. Net flow reductions"}, {"category_id": 15, "poly": [841.0, 733.0, 1012.0, 733.0, 1012.0, 763.0, 841.0, 763.0], "score": 1.0, "text": "for all catchments."}, {"category_id": 15, "poly": [157.0, 836.0, 737.0, 834.0, 737.0, 871.0, 157.0, 873.0], "score": 0.99, "text": "Group 1 exhibit both the highest reduction of"}, {"category_id": 15, "poly": [127.0, 871.0, 732.0, 871.0, 732.0, 907.0, 127.0, 907.0], "score": 0.98, "text": "flows overall, and show the largest proportional"}, {"category_id": 15, "poly": [129.0, 909.0, 732.0, 909.0, 732.0, 940.0, 129.0, 940.0], "score": 0.96, "text": "reduction at lower flows, leading to a complete"}, {"category_id": 15, "poly": [129.0, 944.0, 732.0, 944.0, 732.0, 974.0, 129.0, 974.0], "score": 0.99, "text": "cessation of fow. Comparison of flow reductions is"}, {"category_id": 15, "poly": [125.0, 972.0, 735.0, 974.0, 734.0, 1011.0, 125.0, 1008.0], "score": 0.99, "text": "hindered slightly by the range of afforestation at the"}, {"category_id": 15, "poly": [129.0, 1010.0, 735.0, 1010.0, 735.0, 1041.0, 129.0, 1041.0], "score": 0.99, "text": "catchments (Table 1). These results could be scaled"}, {"category_id": 15, "poly": [129.0, 1079.0, 730.0, 1079.0, 730.0, 1109.0, 129.0, 1109.0], "score": 0.98, "text": "linear relationship between the area planted and flow"}, {"category_id": 15, "poly": [129.0, 1114.0, 732.0, 1114.0, 732.0, 1144.0, 129.0, 1144.0], "score": 0.98, "text": "reductions. As there is no evidence that this is the"}, {"category_id": 15, "poly": [129.0, 1146.0, 728.0, 1146.0, 728.0, 1176.0, 129.0, 1176.0], "score": 0.98, "text": "case we have not presented scaled reductions here."}, {"category_id": 15, "poly": [125.0, 1178.0, 737.0, 1176.0, 737.0, 1213.0, 125.0, 1215.0], "score": 0.95, "text": "Linear scaling would shift the reduction curves"}, {"category_id": 15, "poly": [125.0, 1213.0, 737.0, 1210.0, 737.0, 1247.0, 125.0, 1249.0], "score": 0.96, "text": "upward for those catchments that are less than"}, {"category_id": 15, "poly": [125.0, 1281.0, 492.0, 1286.0, 492.0, 1320.0, 124.0, 1316.0], "score": 0.98, "text": " of the curves or our groupings."}, {"category_id": 15, "poly": [197.0, 1245.0, 735.0, 1247.0, 734.0, 1284.0, 197.0, 1281.0], "score": 0.98, "text": " afforested, but would not change the shape"}, {"category_id": 15, "poly": [129.0, 1045.0, 208.0, 1045.0, 208.0, 1075.0, 129.0, 1075.0], "score": 0.99, "text": "upto"}, {"category_id": 15, "poly": [277.0, 1045.0, 735.0, 1045.0, 735.0, 1075.0, 277.0, 1075.0], "score": 0.98, "text": " afforested if it is assumed there is a"}, {"category_id": 15, "poly": [825.0, 903.0, 1398.0, 905.0, 1398.0, 942.0, 825.0, 940.0], "score": 0.99, "text": "The speed of fow responses to afforestation can be"}, {"category_id": 15, "poly": [793.0, 976.0, 1398.0, 976.0, 1398.0, 1006.0, 793.0, 1006.0], "score": 0.99, "text": "is substantial variation in response times both over the"}, {"category_id": 15, "poly": [788.0, 1008.0, 1402.0, 1004.0, 1403.0, 1041.0, 788.0, 1045.0], "score": 0.99, "text": " percentile spread in some individual catchments, and"}, {"category_id": 15, "poly": [790.0, 1045.0, 1398.0, 1045.0, 1398.0, 1075.0, 790.0, 1075.0], "score": 0.98, "text": " between the catchments. The majority of responses have"}, {"category_id": 15, "poly": [793.0, 1114.0, 1398.0, 1114.0, 1398.0, 1142.0, 793.0, 1142.0], "score": 0.99, "text": "Stewarts Creek, Redhill and Lambrechtsbos A exhibit the"}, {"category_id": 15, "poly": [793.0, 1148.0, 1398.0, 1148.0, 1398.0, 1178.0, 793.0, 1178.0], "score": 0.99, "text": "fastest responses, with Biesievlei showing the most"}, {"category_id": 15, "poly": [793.0, 1215.0, 1398.0, 1215.0, 1398.0, 1245.0, 793.0, 1245.0], "score": 0.99, "text": "catchments display a good correspondence to published"}, {"category_id": 15, "poly": [793.0, 1249.0, 1394.0, 1249.0, 1394.0, 1279.0, 793.0, 1279.0], "score": 0.99, "text": "annual changes (Scott et al., 2000; Van Wyk, 1987),"}, {"category_id": 15, "poly": [793.0, 1284.0, 1398.0, 1284.0, 1398.0, 1314.0, 793.0, 1314.0], "score": 0.99, "text": "excepting the 10-20th deciles for both Cathedral Peak"}, {"category_id": 15, "poly": [790.0, 1079.0, 814.0, 1079.0, 814.0, 1109.0, 790.0, 1109.0], "score": 0.96, "text": "a"}, {"category_id": 15, "poly": [868.0, 1079.0, 1398.0, 1079.0, 1398.0, 1109.0, 868.0, 1109.0], "score": 0.99, "text": "value between 5 and 10 years. Pine Creek and"}, {"category_id": 15, "poly": [788.0, 1178.0, 1087.0, 1176.0, 1087.0, 1213.0, 788.0, 1215.0], "score": 1.0, "text": "uniformly slow response."}, {"category_id": 15, "poly": [1141.0, 1178.0, 1400.0, 1176.0, 1400.0, 1213.0, 1141.0, 1215.0], "score": 0.94, "text": "for the South African"}, {"category_id": 15, "poly": [793.0, 942.0, 1173.0, 942.0, 1173.0, 972.0, 793.0, 972.0], "score": 1.0, "text": "evaluated by examining the value of"}, {"category_id": 15, "poly": [1225.0, 942.0, 1398.0, 942.0, 1398.0, 972.0, 1225.0, 972.0], "score": 0.97, "text": "(Table 4). There"}, {"category_id": 15, "poly": [129.0, 1815.0, 939.0, 1817.0, 939.0, 1847.0, 129.0, 1845.0], "score": 0.98, "text": "Note that no solution could be found for the 50 percentile for Glendhu indicted by the ns."}, {"category_id": 15, "poly": [793.0, 838.0, 1123.0, 838.0, 1123.0, 875.0, 793.0, 875.0], "score": 1.0, "text": "4.3. Timing of fow reductions"}], "page_info": {"page_no": 7, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 0, "poly": [781.8721923828125, 1618.0399169921875, 928.7534790039062, 1618.0399169921875, 928.7534790039062, 1646.409912109375, 781.8721923828125, 1646.409912109375], "score": 0.9999984502792358}, {"category_id": 6, "poly": [113.6822738647461, 253.5222625732422, 1383.692626953125, 253.5222625732422, 1383.692626953125, 334.94287109375, 113.6822738647461, 334.94287109375], "score": 0.9999982714653015}, {"category_id": 2, "poly": [1345.8267822265625, 196.56283569335938, 1379.01416015625, 196.56283569335938, 1379.01416015625, 215.46395874023438, 1345.8267822265625, 215.46395874023438], "score": 0.999997615814209}, {"category_id": 2, "poly": [467.5944519042969, 195.73922729492188, 1030.1298828125, 195.73922729492188, 1030.1298828125, 218.0099639892578, 467.5944519042969, 218.0099639892578], "score": 0.9999966621398926}, {"category_id": 1, "poly": [117.23767852783203, 1243.1109619140625, 716.3004150390625, 1243.1109619140625, 716.3004150390625, 1847.4835205078125, 117.23767852783203, 1847.4835205078125], "score": 0.9999922513961792}, {"category_id": 1, "poly": [118.47014617919922, 994.3162841796875, 713.6297607421875, 994.3162841796875, 713.6297607421875, 1122.2752685546875, 118.47014617919922, 1122.2752685546875], "score": 0.9999918937683105}, {"category_id": 0, "poly": [119.10287475585938, 1178.6876220703125, 627.3995971679688, 1178.6876220703125, 627.3995971679688, 1206.4453125, 119.10287475585938, 1206.4453125], "score": 0.9999906420707703}, {"category_id": 1, "poly": [782.2766723632812, 1682.373291015625, 1380.46728515625, 1682.373291015625, 1380.46728515625, 1845.99609375, 782.2766723632812, 1845.99609375], "score": 0.9999901652336121}, {"category_id": 1, "poly": [781.3034057617188, 1060.36083984375, 1379.3385009765625, 1060.36083984375, 1379.3385009765625, 1521.511962890625, 781.3034057617188, 1521.511962890625], "score": 0.9999886751174927}, {"category_id": 5, "poly": [113.5753402709961, 341.08380126953125, 1386.5994873046875, 341.08380126953125, 1386.5994873046875, 689.2748413085938, 113.5753402709961, 689.2748413085938], "score": 0.9999498128890991}, {"category_id": 7, "poly": [118.46651458740234, 696.7362670898438, 1380.3681640625, 696.7362670898438, 1380.3681640625, 861.0716552734375, 118.46651458740234, 861.0716552734375], "score": 0.9997767210006714}, {"category_id": 0, "poly": [782.720458984375, 993.1400146484375, 988.5659790039062, 993.1400146484375, 988.5659790039062, 1020.3793334960938, 782.720458984375, 1020.3793334960938], "score": 0.9996650218963623}, {"category_id": 13, "poly": [458, 778, 601, 778, 601, 806, 458, 806], "score": 0.91, "latex": "\\sum Y/\\sum(a+Y)"}, {"category_id": 13, "poly": [169, 1025, 221, 1025, 221, 1056, 169, 1056], "score": 0.91, "latex": "T_{\\mathrm{half}}"}, {"category_id": 13, "poly": [464, 750, 607, 750, 607, 778, 464, 778], "score": 0.88, "latex": "\\sum Y/\\sum(a+Y)"}, {"category_id": 13, "poly": [1201, 1191, 1277, 1191, 1277, 1221, 1201, 1221], "score": 0.88, "latex": "\\Delta N_{\\mathrm{zero}}"}, {"category_id": 13, "poly": [1296, 1323, 1350, 1323, 1350, 1353, 1296, 1353], "score": 0.86, "latex": "50\\%"}, {"category_id": 13, "poly": [1078, 1159, 1101, 1159, 1101, 1185, 1078, 1185], "score": 0.77, "latex": "E"}, {"category_id": 13, "poly": [1113, 1192, 1133, 1192, 1133, 1219, 1113, 1219], "score": 0.69, "latex": "b"}, {"category_id": 13, "poly": [375, 811, 390, 811, 390, 830, 375, 830], "score": 0.67, "latex": "a"}, {"category_id": 13, "poly": [990, 1196, 1003, 1196, 1003, 1218, 990, 1218], "score": 0.61, "latex": "t\\cdot"}, {"category_id": 13, "poly": [1066, 812, 1080, 812, 1080, 830, 1066, 830], "score": 0.58, "latex": "a"}, {"category_id": 13, "poly": [431, 808, 448, 808, 448, 830, 431, 830], "score": 0.46, "latex": "Y"}, {"category_id": 13, "poly": [1246, 1357, 1283, 1357, 1283, 1386, 1246, 1386], "score": 0.43, "latex": "\\mathrm{Ck}"}, {"category_id": 13, "poly": [773, 779, 827, 779, 827, 804, 773, 804], "score": 0.42, "latex": "100\\mathrm{th}"}, {"category_id": 13, "poly": [1107, 1357, 1144, 1357, 1144, 1386, 1107, 1386], "score": 0.41, "latex": "\\mathrm{Ck}"}, {"category_id": 13, "poly": [640, 807, 684, 807, 684, 831, 640, 831], "score": 0.29, "latex": "20\\mathrm{th}"}, {"category_id": 15, "poly": [776.0, 1612.0, 935.0, 1617.0, 933.0, 1656.0, 775.0, 1651.0], "score": 0.97, "text": " 5. Discussion"}, {"category_id": 15, "poly": [112.0, 252.0, 195.0, 252.0, 195.0, 282.0, 112.0, 282.0], "score": 0.98, "text": "Table5"}, {"category_id": 15, "poly": [112.0, 279.0, 1383.0, 282.0, 1383.0, 312.0, 112.0, 310.0], "score": 0.99, "text": " Published fow reductions from paired catchment analyses, after Scott et al. (2000), Hickel (2001), Nandakumar and Mein (1993) and Fahey and"}, {"category_id": 15, "poly": [112.0, 307.0, 681.0, 310.0, 681.0, 340.0, 112.0, 338.0], "score": 0.97, "text": "Jackson (1997) compared to estimated reductions in this study"}, {"category_id": 15, "poly": [1330.0, 202.0, 1368.0, 177.0, 1393.0, 215.0, 1355.0, 240.0], "score": 0.99, "text": "261"}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [148.0, 1243.0, 720.0, 1245.0, 719.0, 1275.0, 148.0, 1273.0], "score": 0.98, "text": "A further check on the overall model performance is"}, {"category_id": 15, "poly": [112.0, 1279.0, 719.0, 1277.0, 720.0, 1307.0, 112.0, 1309.0], "score": 0.98, "text": " a comparison with published results of paired catchment"}, {"category_id": 15, "poly": [114.0, 1312.0, 719.0, 1312.0, 719.0, 1342.0, 114.0, 1342.0], "score": 1.0, "text": "studies. The data that can be compared with our results"}, {"category_id": 15, "poly": [114.0, 1346.0, 719.0, 1346.0, 719.0, 1376.0, 114.0, 1376.0], "score": 0.98, "text": "are presented in Table 5 and can be broadly compared"}, {"category_id": 15, "poly": [114.0, 1380.0, 719.0, 1380.0, 719.0, 1410.0, 114.0, 1410.0], "score": 0.99, "text": "with Fig. 4. These data are reductions in years with near"}, {"category_id": 15, "poly": [114.0, 1415.0, 722.0, 1415.0, 722.0, 1445.0, 114.0, 1445.0], "score": 0.99, "text": "average annual rainfall, and at a time after treatment "}, {"category_id": 15, "poly": [114.0, 1445.0, 717.0, 1445.0, 717.0, 1475.0, 114.0, 1475.0], "score": 0.99, "text": "when maximum changes in streamflow have occurred."}, {"category_id": 15, "poly": [114.0, 1481.0, 719.0, 1481.0, 719.0, 1509.0, 114.0, 1509.0], "score": 0.99, "text": "Table 5 also includes estimates on the total and low flow"}, {"category_id": 15, "poly": [114.0, 1511.0, 719.0, 1511.0, 719.0, 1542.0, 114.0, 1542.0], "score": 0.99, "text": "reductions calculated from this study. Results from Pine"}, {"category_id": 15, "poly": [114.0, 1548.0, 719.0, 1548.0, 719.0, 1578.0, 114.0, 1578.0], "score": 0.98, "text": "Creek and Traralgon Creek are not included in Table 5"}, {"category_id": 15, "poly": [114.0, 1582.0, 722.0, 1582.0, 722.0, 1612.0, 114.0, 1612.0], "score": 1.0, "text": "as these catchments are not paired. Exact comparisons"}, {"category_id": 15, "poly": [116.0, 1615.0, 719.0, 1615.0, 719.0, 1645.0, 116.0, 1645.0], "score": 0.99, "text": "are impossible because of the rainfall variability, and"}, {"category_id": 15, "poly": [116.0, 1649.0, 717.0, 1649.0, 717.0, 1679.0, 116.0, 1679.0], "score": 0.99, "text": "lack of calibration period for Redhill. Despite this,"}, {"category_id": 15, "poly": [116.0, 1681.0, 719.0, 1681.0, 719.0, 1711.0, 116.0, 1711.0], "score": 0.95, "text": "Table 5 shows that total and low flow reductions"}, {"category_id": 15, "poly": [116.0, 1716.0, 719.0, 1716.0, 719.0, 1746.0, 116.0, 1746.0], "score": 0.99, "text": "estimated from our study are comparable to the results"}, {"category_id": 15, "poly": [116.0, 1750.0, 719.0, 1750.0, 719.0, 1780.0, 116.0, 1780.0], "score": 0.99, "text": "from paired catchment studies, indicating that our"}, {"category_id": 15, "poly": [116.0, 1782.0, 719.0, 1782.0, 719.0, 1812.0, 116.0, 1812.0], "score": 0.99, "text": "simple model has successfully removed the rainfall"}, {"category_id": 15, "poly": [111.0, 1820.0, 190.0, 1814.0, 192.0, 1846.0, 113.0, 1852.0], "score": 0.94, "text": "signal."}, {"category_id": 15, "poly": [112.0, 989.0, 717.0, 987.0, 717.0, 1023.0, 112.0, 1026.0], "score": 1.0, "text": "catchments and the lower deciles at Lambrechtsbos B."}, {"category_id": 15, "poly": [116.0, 1062.0, 715.0, 1062.0, 715.0, 1092.0, 116.0, 1092.0], "score": 0.98, "text": "lower than other published data (Fahey and Jackson,"}, {"category_id": 15, "poly": [116.0, 1092.0, 189.0, 1092.0, 189.0, 1124.0, 116.0, 1124.0], "score": 0.96, "text": "1997)."}, {"category_id": 15, "poly": [116.0, 1028.0, 168.0, 1028.0, 168.0, 1058.0, 116.0, 1058.0], "score": 1.0, "text": "The"}, {"category_id": 15, "poly": [222.0, 1028.0, 715.0, 1028.0, 715.0, 1058.0, 222.0, 1058.0], "score": 1.0, "text": "from Glendhu 2 appears to be substantially"}, {"category_id": 15, "poly": [114.0, 1176.0, 631.0, 1176.0, 631.0, 1213.0, 114.0, 1213.0], "score": 0.99, "text": "4.4. Comparison with paired catchment studies"}, {"category_id": 15, "poly": [816.0, 1686.0, 1381.0, 1686.0, 1381.0, 1716.0, 816.0, 1716.0], "score": 0.98, "text": "The aims of the project have largely been met. The"}, {"category_id": 15, "poly": [782.0, 1720.0, 1383.0, 1720.0, 1383.0, 1750.0, 782.0, 1750.0], "score": 0.97, "text": "general characterisation of FDCs and adjustment for"}, {"category_id": 15, "poly": [782.0, 1750.0, 1385.0, 1750.0, 1385.0, 1787.0, 782.0, 1787.0], "score": 0.99, "text": "climate has been very encouraging given the task of"}, {"category_id": 15, "poly": [782.0, 1784.0, 1381.0, 1784.0, 1381.0, 1815.0, 782.0, 1815.0], "score": 0.98, "text": "fitting our model to 10 flow percentiles, for 10 different"}, {"category_id": 15, "poly": [778.0, 1812.0, 1383.0, 1815.0, 1383.0, 1851.0, 777.0, 1849.0], "score": 0.97, "text": "catchments (resulting in 100 model fits\uff09 with"}, {"category_id": 15, "poly": [816.0, 1060.0, 1381.0, 1060.0, 1381.0, 1090.0, 816.0, 1090.0], "score": 0.96, "text": "As this analysis could only be applied, where there"}, {"category_id": 15, "poly": [782.0, 1094.0, 1383.0, 1094.0, 1383.0, 1124.0, 782.0, 1124.0], "score": 1.0, "text": "was consistent drying up of streams, it was confined to"}, {"category_id": 15, "poly": [782.0, 1127.0, 1381.0, 1127.0, 1381.0, 1157.0, 782.0, 1157.0], "score": 0.99, "text": "Stewarts Creek, Pine Creek and Redhill catchments. The"}, {"category_id": 15, "poly": [780.0, 1228.0, 1383.0, 1228.0, 1383.0, 1256.0, 780.0, 1256.0], "score": 0.99, "text": "significant results at the 0.05 level for both parameters at"}, {"category_id": 15, "poly": [780.0, 1260.0, 1383.0, 1260.0, 1383.0, 1290.0, 780.0, 1290.0], "score": 0.97, "text": "all three catchments. The climate adjusted zero flow"}, {"category_id": 15, "poly": [780.0, 1292.0, 1383.0, 1292.0, 1383.0, 1322.0, 780.0, 1322.0], "score": 1.0, "text": "days are shown in Fig. 5. The increases in zero flow days"}, {"category_id": 15, "poly": [778.0, 1387.0, 1387.0, 1389.0, 1387.0, 1425.0, 777.0, 1423.0], "score": 0.97, "text": "11 at Redhill. The latter has changed from an almost "}, {"category_id": 15, "poly": [775.0, 1423.0, 1385.0, 1421.0, 1385.0, 1458.0, 775.0, 1460.0], "score": 0.97, "text": " permanent to a highly intermittent stream. The curves"}, {"category_id": 15, "poly": [775.0, 1453.0, 1385.0, 1456.0, 1385.0, 1492.0, 775.0, 1490.0], "score": 0.99, "text": " are also in sensible agreement with the flow reductions"}, {"category_id": 15, "poly": [777.0, 1492.0, 885.0, 1492.0, 885.0, 1522.0, 777.0, 1522.0], "score": 0.99, "text": "in Fig. 4."}, {"category_id": 15, "poly": [1278.0, 1193.0, 1383.0, 1193.0, 1383.0, 1223.0, 1278.0, 1223.0], "score": 0.98, "text": " returned"}, {"category_id": 15, "poly": [777.0, 1327.0, 1295.0, 1327.0, 1295.0, 1357.0, 777.0, 1357.0], "score": 0.98, "text": " are substantial with flows confined to less than"}, {"category_id": 15, "poly": [1351.0, 1327.0, 1385.0, 1327.0, 1385.0, 1357.0, 1351.0, 1357.0], "score": 1.0, "text": "of"}, {"category_id": 15, "poly": [780.0, 1161.0, 1077.0, 1161.0, 1077.0, 1189.0, 780.0, 1189.0], "score": 0.96, "text": "model returned values of"}, {"category_id": 15, "poly": [1102.0, 1161.0, 1381.0, 1161.0, 1381.0, 1189.0, 1102.0, 1189.0], "score": 0.98, "text": "of 0.95, 0.99 and 0.99,"}, {"category_id": 15, "poly": [1134.0, 1193.0, 1200.0, 1193.0, 1200.0, 1223.0, 1134.0, 1223.0], "score": 0.97, "text": " and"}, {"category_id": 15, "poly": [780.0, 1193.0, 989.0, 1193.0, 989.0, 1223.0, 780.0, 1223.0], "score": 0.98, "text": "respectively. The"}, {"category_id": 15, "poly": [1004.0, 1193.0, 1112.0, 1193.0, 1112.0, 1223.0, 1004.0, 1223.0], "score": 0.89, "text": "-testson"}, {"category_id": 15, "poly": [1284.0, 1354.0, 1385.0, 1357.0, 1385.0, 1393.0, 1284.0, 1391.0], "score": 1.0, "text": "and year"}, {"category_id": 15, "poly": [775.0, 1354.0, 1106.0, 1357.0, 1106.0, 1393.0, 775.0, 1391.0], "score": 0.98, "text": " the time by year 8 at Stewarts"}, {"category_id": 15, "poly": [1145.0, 1354.0, 1245.0, 1357.0, 1245.0, 1393.0, 1145.0, 1391.0], "score": 1.0, "text": "and Pine"}, {"category_id": 15, "poly": [125.0, 690.0, 1385.0, 694.0, 1385.0, 731.0, 125.0, 727.0], "score": 0.97, "text": "a Rainfall refers to the rainfall in the year used for comparison of results. The value in brackets refers to the deviation from the mean anual"}, {"category_id": 15, "poly": [114.0, 722.0, 408.0, 725.0, 408.0, 755.0, 114.0, 752.0], "score": 1.0, "text": "rainfall for the period of record."}, {"category_id": 15, "poly": [112.0, 832.0, 312.0, 834.0, 311.0, 864.0, 112.0, 862.0], "score": 0.99, "text": " 30-100th percentiles."}, {"category_id": 15, "poly": [125.0, 776.0, 457.0, 780.0, 457.0, 811.0, 125.0, 806.0], "score": 0.96, "text": "c Low flow reduction calculated by"}, {"category_id": 15, "poly": [122.0, 748.0, 463.0, 750.0, 463.0, 780.0, 122.0, 778.0], "score": 0.98, "text": "b Total flow reduction calculated by"}, {"category_id": 15, "poly": [608.0, 748.0, 743.0, 750.0, 743.0, 780.0, 608.0, 778.0], "score": 1.0, "text": "for all deciles."}, {"category_id": 15, "poly": [123.0, 800.0, 374.0, 806.0, 374.0, 843.0, 122.0, 836.0], "score": 0.98, "text": "d For Cathedral Peak 3 the"}, {"category_id": 15, "poly": [1081.0, 800.0, 1385.0, 806.0, 1385.0, 843.0, 1081.0, 836.0], "score": 0.99, "text": "were lower then the values of the"}, {"category_id": 15, "poly": [391.0, 800.0, 430.0, 806.0, 430.0, 843.0, 391.0, 836.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [602.0, 776.0, 772.0, 780.0, 772.0, 811.0, 602.0, 806.0], "score": 0.99, "text": "for 70, 80, 90 and"}, {"category_id": 15, "poly": [828.0, 776.0, 934.0, 780.0, 934.0, 811.0, 828.0, 806.0], "score": 1.0, "text": "percentiles."}, {"category_id": 15, "poly": [449.0, 800.0, 639.0, 806.0, 639.0, 843.0, 449.0, 836.0], "score": 0.98, "text": "values for the 10 and"}, {"category_id": 15, "poly": [685.0, 800.0, 1065.0, 806.0, 1065.0, 843.0, 685.0, 836.0], "score": 0.99, "text": "percentiles were excluded as the values of"}, {"category_id": 15, "poly": [778.0, 987.0, 993.0, 991.0, 992.0, 1030.0, 777.0, 1025.0], "score": 1.0, "text": "4.5. Zero fow days"}], "page_info": {"page_no": 8, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 4, "poly": [130.26629638671875, 1337.3250732421875, 732.2774047851562, 1337.3250732421875, 732.2774047851562, 1418.7119140625, 130.26629638671875, 1418.7119140625], "score": 0.9999995231628418}, {"category_id": 2, "poly": [131.43930053710938, 195.23593139648438, 165.80084228515625, 195.23593139648438, 165.80084228515625, 215.28358459472656, 131.43930053710938, 215.28358459472656], "score": 0.999997615814209}, {"category_id": 3, "poly": [131.99615478515625, 262.13848876953125, 732.1187744140625, 262.13848876953125, 732.1187744140625, 1316.8990478515625, 131.99615478515625, 1316.8990478515625], "score": 0.9999966621398926}, {"category_id": 2, "poly": [480.581787109375, 195.6566162109375, 1043.8951416015625, 195.6566162109375, 1043.8951416015625, 218.63613891601562, 480.581787109375, 218.63613891601562], "score": 0.9999943971633911}, {"category_id": 1, "poly": [794.3954467773438, 1085.8665771484375, 1394.706298828125, 1085.8665771484375, 1394.706298828125, 1847.09423828125, 794.3954467773438, 1847.09423828125], "score": 0.9999863505363464}, {"category_id": 1, "poly": [795.3126831054688, 256.7186279296875, 1395.0584716796875, 256.7186279296875, 1395.0584716796875, 1079.958251953125, 795.3126831054688, 1079.958251953125], "score": 0.9999836683273315}, {"category_id": 1, "poly": [130.74447631835938, 1445.3975830078125, 731.3636474609375, 1445.3975830078125, 731.3636474609375, 1846.5950927734375, 130.74447631835938, 1846.5950927734375], "score": 0.9999815225601196}, {"category_id": 13, "poly": [1045, 452, 1098, 452, 1098, 482, 1045, 482], "score": 0.87, "latex": "27\\%"}, {"category_id": 15, "poly": [129.0, 1339.0, 732.0, 1339.0, 732.0, 1367.0, 129.0, 1367.0], "score": 0.98, "text": "Fig. 5. Number of zero fow days for average rainfall following"}, {"category_id": 15, "poly": [131.0, 1365.0, 730.0, 1365.0, 730.0, 1393.0, 131.0, 1393.0], "score": 0.98, "text": "afforestation for Stewarts Creek 5, Redhill and Pine Creek,"}, {"category_id": 15, "poly": [133.0, 1398.0, 219.0, 1398.0, 219.0, 1421.0, 133.0, 1421.0], "score": 1.0, "text": "Australia."}, {"category_id": 15, "poly": [127.0, 189.0, 172.0, 189.0, 172.0, 228.0, 127.0, 228.0], "score": 0.97, "text": "262"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [827.0, 1086.0, 1398.0, 1086.0, 1398.0, 1116.0, 827.0, 1116.0], "score": 0.99, "text": "The model fits show we have quantified the net"}, {"category_id": 15, "poly": [793.0, 1120.0, 1396.0, 1120.0, 1396.0, 1150.0, 793.0, 1150.0], "score": 0.98, "text": "impact of afforestation for the majority of the flow"}, {"category_id": 15, "poly": [786.0, 1150.0, 1398.0, 1146.0, 1398.0, 1182.0, 786.0, 1187.0], "score": 0.99, "text": " percentiles in the 10 catchments. Results for the 10-50th"}, {"category_id": 15, "poly": [788.0, 1185.0, 1400.0, 1180.0, 1400.0, 1217.0, 788.0, 1221.0], "score": 0.98, "text": " percentiles were particularly encouraging. It is not"}, {"category_id": 15, "poly": [793.0, 1219.0, 1396.0, 1219.0, 1396.0, 1249.0, 793.0, 1249.0], "score": 0.99, "text": "surprising that the relationship between rainfall and flow"}, {"category_id": 15, "poly": [793.0, 1251.0, 1396.0, 1251.0, 1396.0, 1281.0, 793.0, 1281.0], "score": 0.97, "text": "diminishes at lower fows (60-100th percentile), where"}, {"category_id": 15, "poly": [793.0, 1286.0, 1396.0, 1286.0, 1396.0, 1316.0, 793.0, 1316.0], "score": 0.98, "text": "seasonal storage effects and rainfall distribution become"}, {"category_id": 15, "poly": [788.0, 1318.0, 1396.0, 1318.0, 1396.0, 1348.0, 788.0, 1348.0], "score": 0.97, "text": " more important drivers for runoff generation. The"}, {"category_id": 15, "poly": [790.0, 1352.0, 1398.0, 1352.0, 1398.0, 1382.0, 790.0, 1382.0], "score": 0.99, "text": "poorest model fits were gained for Lambrechtsbos A"}, {"category_id": 15, "poly": [788.0, 1382.0, 1396.0, 1382.0, 1396.0, 1413.0, 788.0, 1413.0], "score": 0.98, "text": " and B. The likely reason at Lambrechtsbos A is an"}, {"category_id": 15, "poly": [793.0, 1419.0, 1398.0, 1419.0, 1398.0, 1447.0, 793.0, 1447.0], "score": 0.98, "text": "observed annual decrease in stand water use after 12"}, {"category_id": 15, "poly": [793.0, 1451.0, 1398.0, 1451.0, 1398.0, 1481.0, 793.0, 1481.0], "score": 0.99, "text": "years (Scott et al., 2000) which does not conform to the"}, {"category_id": 15, "poly": [793.0, 1486.0, 1398.0, 1486.0, 1398.0, 1516.0, 793.0, 1516.0], "score": 0.99, "text": "sigmoidal form of our model over the full 19 years of"}, {"category_id": 15, "poly": [793.0, 1518.0, 1398.0, 1518.0, 1398.0, 1548.0, 793.0, 1548.0], "score": 0.99, "text": "record. The failure of the model to fit the lower flows at"}, {"category_id": 15, "poly": [786.0, 1546.0, 1400.0, 1548.0, 1400.0, 1585.0, 786.0, 1582.0], "score": 0.98, "text": " Lambrechtsbos B is not as explicable. A decrease in"}, {"category_id": 15, "poly": [790.0, 1587.0, 1398.0, 1587.0, 1398.0, 1615.0, 790.0, 1615.0], "score": 1.0, "text": "stand water use in this catchment is observed as the"}, {"category_id": 15, "poly": [793.0, 1619.0, 1400.0, 1619.0, 1400.0, 1649.0, 793.0, 1649.0], "score": 0.99, "text": "plantation ages, but does not occur during the first 20"}, {"category_id": 15, "poly": [793.0, 1651.0, 1398.0, 1651.0, 1398.0, 1681.0, 793.0, 1681.0], "score": 0.99, "text": "years after treatment (Scott et al., 2000). Other data from"}, {"category_id": 15, "poly": [788.0, 1679.0, 1398.0, 1681.0, 1398.0, 1718.0, 788.0, 1716.0], "score": 0.98, "text": " South Africa (Scott et al., 2000) indicate there are"}, {"category_id": 15, "poly": [791.0, 1711.0, 1398.0, 1716.0, 1398.0, 1752.0, 790.0, 1748.0], "score": 0.99, "text": " diminished flow reductions as plantations age, but again,"}, {"category_id": 15, "poly": [795.0, 1752.0, 1396.0, 1752.0, 1396.0, 1782.0, 795.0, 1782.0], "score": 0.99, "text": "generally after 20 years. Our use of an asymptotic curve"}, {"category_id": 15, "poly": [790.0, 1785.0, 1398.0, 1782.0, 1398.0, 1812.0, 790.0, 1815.0], "score": 0.98, "text": "assumes a new equilibrium of stand water use is"}, {"category_id": 15, "poly": [790.0, 1815.0, 1394.0, 1817.0, 1394.0, 1847.0, 790.0, 1845.0], "score": 0.99, "text": "reached. The results of the model fitting generally justify"}, {"category_id": 15, "poly": [788.0, 249.0, 1398.0, 252.0, 1398.0, 288.0, 788.0, 286.0], "score": 0.97, "text": " Lambrechtsbos B appear to be over-estimated by our"}, {"category_id": 15, "poly": [788.0, 284.0, 1398.0, 288.0, 1398.0, 325.0, 788.0, 320.0], "score": 0.99, "text": " model, which is unsurprising as the model fit was poor."}, {"category_id": 15, "poly": [793.0, 322.0, 1398.0, 322.0, 1398.0, 353.0, 793.0, 353.0], "score": 0.98, "text": "The remaining four South African catchments, and also"}, {"category_id": 15, "poly": [793.0, 355.0, 1398.0, 355.0, 1398.0, 385.0, 793.0, 385.0], "score": 0.99, "text": "Redhill and Stewarts Creek are in good agreement with"}, {"category_id": 15, "poly": [793.0, 389.0, 1400.0, 389.0, 1400.0, 417.0, 793.0, 417.0], "score": 1.0, "text": "the published values, particularly when the deviation of"}, {"category_id": 15, "poly": [793.0, 421.0, 1398.0, 421.0, 1398.0, 452.0, 793.0, 452.0], "score": 0.99, "text": "average rainfall is considered. Glendhu 2 reductions are"}, {"category_id": 15, "poly": [788.0, 488.0, 1396.0, 488.0, 1396.0, 518.0, 788.0, 518.0], "score": 0.99, "text": " a heavier impact on the lower flows. Overall, it appears"}, {"category_id": 15, "poly": [788.0, 518.0, 1400.0, 518.0, 1400.0, 555.0, 788.0, 555.0], "score": 0.99, "text": " there are no significant discrepancies with the published"}, {"category_id": 15, "poly": [793.0, 555.0, 1398.0, 555.0, 1398.0, 585.0, 793.0, 585.0], "score": 0.99, "text": "paired catchment analyses. We suggest our technique"}, {"category_id": 15, "poly": [790.0, 589.0, 1398.0, 589.0, 1398.0, 619.0, 790.0, 619.0], "score": 0.99, "text": " represents an alternative to the paired-catchment method"}, {"category_id": 15, "poly": [788.0, 617.0, 1398.0, 619.0, 1398.0, 656.0, 788.0, 654.0], "score": 0.98, "text": "for assessing hydrologic response to vegetation treat-"}, {"category_id": 15, "poly": [788.0, 651.0, 1400.0, 649.0, 1400.0, 686.0, 788.0, 688.0], "score": 1.0, "text": " ment, where paired data are unavailable. The method"}, {"category_id": 15, "poly": [793.0, 688.0, 1396.0, 688.0, 1396.0, 718.0, 793.0, 718.0], "score": 0.99, "text": "has not yet resulted in a predictive model, but has"}, {"category_id": 15, "poly": [795.0, 722.0, 1394.0, 722.0, 1394.0, 752.0, 795.0, 752.0], "score": 1.0, "text": "increased our knowledge of afforestation impacts. This"}, {"category_id": 15, "poly": [793.0, 755.0, 1398.0, 755.0, 1398.0, 785.0, 793.0, 785.0], "score": 0.96, "text": "is a valuable outcome given the contentious issue of"}, {"category_id": 15, "poly": [793.0, 787.0, 1398.0, 787.0, 1398.0, 817.0, 793.0, 817.0], "score": 0.98, "text": "afforestation in Australia and other countries, and a"}, {"category_id": 15, "poly": [792.0, 821.0, 1398.0, 819.0, 1398.0, 849.0, 793.0, 851.0], "score": 0.99, "text": "current paucity of data on inter-annual flows. It should"}, {"category_id": 15, "poly": [788.0, 849.0, 1396.0, 851.0, 1396.0, 888.0, 788.0, 886.0], "score": 0.99, "text": " be noted that nine of the 10 catchment were pine species."}, {"category_id": 15, "poly": [793.0, 888.0, 1398.0, 888.0, 1398.0, 918.0, 793.0, 918.0], "score": 0.98, "text": "More data is required to compare the impact of"}, {"category_id": 15, "poly": [790.0, 920.0, 1390.0, 920.0, 1390.0, 950.0, 790.0, 950.0], "score": 0.99, "text": "hardwood species, particularly eucalypts, on the FDC."}, {"category_id": 15, "poly": [793.0, 950.0, 1396.0, 955.0, 1396.0, 985.0, 792.0, 980.0], "score": 0.98, "text": "Unfortunately these data are currently scarce. There are"}, {"category_id": 15, "poly": [791.0, 980.0, 1398.0, 985.0, 1398.0, 1021.0, 790.0, 1017.0], "score": 1.0, "text": "substantial data on the physiological controls of eucalypt"}, {"category_id": 15, "poly": [793.0, 1019.0, 1396.0, 1019.0, 1396.0, 1049.0, 793.0, 1049.0], "score": 0.99, "text": "water use (see Whitehead and Beadle, 2004), but not at"}, {"category_id": 15, "poly": [790.0, 1054.0, 1016.0, 1054.0, 1016.0, 1084.0, 790.0, 1084.0], "score": 0.98, "text": "the catchment scale."}, {"category_id": 15, "poly": [788.0, 452.0, 1044.0, 452.0, 1044.0, 488.0, 788.0, 488.0], "score": 0.91, "text": " close to the reported "}, {"category_id": 15, "poly": [1099.0, 452.0, 1398.0, 452.0, 1398.0, 488.0, 1099.0, 488.0], "score": 0.97, "text": ", but our model produces"}, {"category_id": 15, "poly": [129.0, 1443.0, 732.0, 1443.0, 732.0, 1479.0, 129.0, 1479.0], "score": 0.99, "text": "substantially varying spatial scales, soils and geology,"}, {"category_id": 15, "poly": [129.0, 1479.0, 732.0, 1479.0, 732.0, 1509.0, 129.0, 1509.0], "score": 0.99, "text": "species planted and climatic environments. Although"}, {"category_id": 15, "poly": [127.0, 1511.0, 735.0, 1511.0, 735.0, 1548.0, 127.0, 1548.0], "score": 0.98, "text": "there were poor results for individual deciles, the FDCs "}, {"category_id": 15, "poly": [129.0, 1548.0, 732.0, 1548.0, 732.0, 1578.0, 129.0, 1578.0], "score": 0.99, "text": "at eight of the 10 catchments were adequately described"}, {"category_id": 15, "poly": [127.0, 1580.0, 730.0, 1578.0, 730.0, 1608.0, 127.0, 1610.0], "score": 0.99, "text": "by Eq. (2). The results of the statistical tests in which the"}, {"category_id": 15, "poly": [129.0, 1612.0, 726.0, 1612.0, 726.0, 1643.0, 129.0, 1643.0], "score": 0.99, "text": "rainfall term was significant for most deciles demon-"}, {"category_id": 15, "poly": [127.0, 1647.0, 735.0, 1647.0, 735.0, 1683.0, 127.0, 1683.0], "score": 0.99, "text": "strated the model structure was appropriate for adjusting"}, {"category_id": 15, "poly": [129.0, 1683.0, 730.0, 1683.0, 730.0, 1711.0, 129.0, 1711.0], "score": 0.96, "text": "the FDCs for climatic (rainfall) variability. The"}, {"category_id": 15, "poly": [129.0, 1716.0, 732.0, 1716.0, 732.0, 1746.0, 129.0, 1746.0], "score": 0.95, "text": "comparisons of our results with published paired"}, {"category_id": 15, "poly": [129.0, 1750.0, 732.0, 1750.0, 732.0, 1780.0, 129.0, 1780.0], "score": 0.98, "text": "catchment analyses are satisfactory, although the"}, {"category_id": 15, "poly": [129.0, 1784.0, 735.0, 1784.0, 735.0, 1812.0, 129.0, 1812.0], "score": 0.98, "text": "different methodologies make direct comparisons of"}, {"category_id": 15, "poly": [127.0, 1815.0, 737.0, 1817.0, 737.0, 1847.0, 127.0, 1845.0], "score": 0.98, "text": "deciles with total fow uncertain. Low flows at"}], "page_info": {"page_no": 9, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 2, "poly": [466.4325256347656, 194.8888397216797, 1031.1922607421875, 194.8888397216797, 1031.1922607421875, 219.64439392089844, 466.4325256347656, 219.64439392089844], "score": 0.9999977350234985}, {"category_id": 0, "poly": [781.4110107421875, 1350.9322509765625, 1112.086181640625, 1350.9322509765625, 1112.086181640625, 1380.4071044921875, 781.4110107421875, 1380.4071044921875], "score": 0.9999973773956299}, {"category_id": 1, "poly": [118.7479248046875, 587.0300903320312, 715.7766723632812, 587.0300903320312, 715.7766723632812, 883.0694580078125, 118.7479248046875, 883.0694580078125], "score": 0.9999968409538269}, {"category_id": 1, "poly": [118.48811340332031, 252.823486328125, 715.749267578125, 252.823486328125, 715.749267578125, 583.28515625, 118.48811340332031, 583.28515625], "score": 0.9999964237213135}, {"category_id": 1, "poly": [117.62772369384766, 885.8139038085938, 717.3323974609375, 885.8139038085938, 717.3323974609375, 1415.2767333984375, 117.62772369384766, 1415.2767333984375], "score": 0.9999961853027344}, {"category_id": 1, "poly": [782.490234375, 254.01434326171875, 1380.5517578125, 254.01434326171875, 1380.5517578125, 748.8712768554688, 782.490234375, 748.8712768554688], "score": 0.9999944567680359}, {"category_id": 1, "poly": [117.28860473632812, 1415.5831298828125, 716.8341064453125, 1415.5831298828125, 716.8341064453125, 1847.5146484375, 117.28860473632812, 1847.5146484375], "score": 0.9999933242797852}, {"category_id": 1, "poly": [781.5156860351562, 752.15576171875, 1380.3497314453125, 752.15576171875, 1380.3497314453125, 1279.9158935546875, 781.5156860351562, 1279.9158935546875], "score": 0.9999922513961792}, {"category_id": 1, "poly": [781.4845581054688, 1417.2979736328125, 1380.813232421875, 1417.2979736328125, 1380.813232421875, 1845.5704345703125, 781.4845581054688, 1845.5704345703125], "score": 0.9999920725822449}, {"category_id": 2, "poly": [1346.2413330078125, 196.15005493164062, 1380.82568359375, 196.15005493164062, 1380.82568359375, 216.4473876953125, 1346.2413330078125, 216.4473876953125], "score": 0.9999884366989136}, {"category_id": 13, "poly": [510, 1017, 563, 1017, 563, 1047, 510, 1047], "score": 0.89, "latex": "85\\%"}, {"category_id": 13, "poly": [1121, 321, 1143, 321, 1143, 347, 1121, 347], "score": 0.55, "latex": "E"}, {"category_id": 13, "poly": [433, 354, 456, 354, 456, 380, 433, 380], "score": 0.47, "latex": "E."}, {"category_id": 13, "poly": [578, 1018, 683, 1018, 683, 1048, 578, 1048], "score": 0.39, "latex": "1260\\,\\mathrm{mm}"}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [780.0, 1350.0, 1117.0, 1350.0, 1117.0, 1387.0, 780.0, 1387.0], "score": 0.99, "text": "6. Summary and conclusions"}, {"category_id": 15, "poly": [150.0, 585.0, 717.0, 585.0, 717.0, 615.0, 150.0, 615.0], "score": 0.98, "text": "The small Australian catchments converted to pine in"}, {"category_id": 15, "poly": [109.0, 619.0, 724.0, 617.0, 724.0, 654.0, 110.0, 656.0], "score": 1.0, "text": "response group 1 (Stewarts Creek 5, Pine Creek and"}, {"category_id": 15, "poly": [112.0, 649.0, 722.0, 651.0, 722.0, 688.0, 112.0, 686.0], "score": 0.99, "text": "Redhill) have similar shallow soils, potential evapo-"}, {"category_id": 15, "poly": [116.0, 688.0, 717.0, 688.0, 717.0, 718.0, 116.0, 718.0], "score": 0.99, "text": "transpiration and rainfall distribution (relatively uni-"}, {"category_id": 15, "poly": [116.0, 722.0, 717.0, 722.0, 717.0, 752.0, 116.0, 752.0], "score": 1.0, "text": "form) although Stewarts Creek is significantly wetter."}, {"category_id": 15, "poly": [116.0, 755.0, 719.0, 755.0, 719.0, 785.0, 116.0, 785.0], "score": 0.98, "text": "The combination of small catchment area and the"}, {"category_id": 15, "poly": [116.0, 787.0, 717.0, 787.0, 717.0, 817.0, 116.0, 817.0], "score": 0.99, "text": "increased transpirative demand that exceeds summer"}, {"category_id": 15, "poly": [114.0, 819.0, 720.0, 821.0, 719.0, 851.0, 114.0, 849.0], "score": 1.0, "text": "and autumn rainfall and stored water results in the large"}, {"category_id": 15, "poly": [116.0, 856.0, 638.0, 856.0, 638.0, 884.0, 116.0, 884.0], "score": 0.98, "text": "impact on lower flows, compared to high flows."}, {"category_id": 15, "poly": [114.0, 252.0, 722.0, 252.0, 722.0, 288.0, 114.0, 288.0], "score": 0.98, "text": "this assumption for the length of commercial plantation"}, {"category_id": 15, "poly": [114.0, 286.0, 719.0, 286.0, 719.0, 322.0, 114.0, 322.0], "score": 0.99, "text": "growth (up to 20 years) considered here. The physio-"}, {"category_id": 15, "poly": [116.0, 322.0, 719.0, 322.0, 719.0, 353.0, 116.0, 353.0], "score": 0.99, "text": "logical relationship between stand age and water use for"}, {"category_id": 15, "poly": [114.0, 385.0, 719.0, 385.0, 719.0, 421.0, 114.0, 421.0], "score": 1.0, "text": "thoroughly investigated, although Cornish and Vertessy"}, {"category_id": 15, "poly": [112.0, 417.0, 722.0, 419.0, 722.0, 456.0, 112.0, 454.0], "score": 0.99, "text": "(2001) and Roberts et al. (2001) have shown young"}, {"category_id": 15, "poly": [116.0, 456.0, 719.0, 456.0, 719.0, 486.0, 116.0, 486.0], "score": 1.0, "text": "mixed species eucalypt forests may use more water than"}, {"category_id": 15, "poly": [112.0, 484.0, 722.0, 486.0, 722.0, 522.0, 112.0, 520.0], "score": 0.99, "text": " mature stands, and Putahena and Cordery (2000) suggest "}, {"category_id": 15, "poly": [116.0, 522.0, 719.0, 522.0, 719.0, 553.0, 116.0, 553.0], "score": 0.99, "text": "maximum Pinus radiata water use may have been"}, {"category_id": 15, "poly": [114.0, 553.0, 655.0, 553.0, 655.0, 583.0, 114.0, 583.0], "score": 0.99, "text": "reached after 12 years, with a subsequent decline."}, {"category_id": 15, "poly": [116.0, 355.0, 432.0, 355.0, 432.0, 385.0, 116.0, 385.0], "score": 1.0, "text": "plantation species other than"}, {"category_id": 15, "poly": [457.0, 355.0, 719.0, 355.0, 719.0, 385.0, 457.0, 385.0], "score": 0.96, "text": "regnans have not been"}, {"category_id": 15, "poly": [146.0, 881.0, 720.0, 884.0, 719.0, 920.0, 146.0, 918.0], "score": 0.98, "text": " The magnitude of the response within Group 2 varies"}, {"category_id": 15, "poly": [116.0, 920.0, 717.0, 920.0, 717.0, 950.0, 116.0, 950.0], "score": 1.0, "text": "considerably, with greater reduction in flows in the two"}, {"category_id": 15, "poly": [116.0, 952.0, 717.0, 952.0, 717.0, 983.0, 116.0, 983.0], "score": 0.98, "text": "Cathedral Peak catchments, and Lambrechtsbos B."}, {"category_id": 15, "poly": [114.0, 987.0, 719.0, 987.0, 719.0, 1017.0, 114.0, 1017.0], "score": 0.99, "text": "Potential evaporation is in phase with rainfall at the"}, {"category_id": 15, "poly": [114.0, 1054.0, 722.0, 1054.0, 722.0, 1084.0, 114.0, 1084.0], "score": 0.98, "text": "average) of their rainfall in summer. The conjunction of"}, {"category_id": 15, "poly": [116.0, 1088.0, 722.0, 1088.0, 722.0, 1118.0, 116.0, 1118.0], "score": 0.99, "text": "peak demand and plant water availability may explain"}, {"category_id": 15, "poly": [114.0, 1118.0, 722.0, 1120.0, 722.0, 1150.0, 114.0, 1148.0], "score": 0.98, "text": "the high reductions relative to the remaining catchments "}, {"category_id": 15, "poly": [116.0, 1155.0, 722.0, 1155.0, 722.0, 1185.0, 116.0, 1185.0], "score": 0.98, "text": "in Group 2. In addition, the stocking density was"}, {"category_id": 15, "poly": [116.0, 1187.0, 717.0, 1187.0, 717.0, 1215.0, 116.0, 1215.0], "score": 0.97, "text": "described as \u2018abnormally dense\u2019 by Scott et al. (2000)."}, {"category_id": 15, "poly": [116.0, 1219.0, 719.0, 1219.0, 719.0, 1249.0, 116.0, 1249.0], "score": 1.0, "text": "Growth at Glendhu 2 was notably slow (Fahey and"}, {"category_id": 15, "poly": [112.0, 1247.0, 722.0, 1249.0, 722.0, 1286.0, 112.0, 1284.0], "score": 0.98, "text": " Jackson, 1997) and Lambrechtsbos A and Biesievlei are"}, {"category_id": 15, "poly": [116.0, 1286.0, 719.0, 1286.0, 719.0, 1316.0, 116.0, 1316.0], "score": 1.0, "text": "described as being within sub optimal growth zones"}, {"category_id": 15, "poly": [116.0, 1318.0, 719.0, 1318.0, 719.0, 1348.0, 116.0, 1348.0], "score": 0.99, "text": "(Scott and Smith, 1997) characterised by these authors"}, {"category_id": 15, "poly": [114.0, 1352.0, 719.0, 1352.0, 719.0, 1382.0, 114.0, 1382.0], "score": 0.99, "text": "as having relatively slow response times and lesser"}, {"category_id": 15, "poly": [109.0, 1383.0, 586.0, 1380.0, 586.0, 1417.0, 110.0, 1419.0], "score": 0.96, "text": " reductions that those at more optimal sites."}, {"category_id": 15, "poly": [112.0, 1017.0, 509.0, 1017.0, 509.0, 1054.0, 112.0, 1054.0], "score": 0.98, "text": " Cathedral Peak sites as they receive"}, {"category_id": 15, "poly": [684.0, 1017.0, 719.0, 1017.0, 719.0, 1054.0, 684.0, 1054.0], "score": 0.93, "text": "on"}, {"category_id": 15, "poly": [810.0, 249.0, 1383.0, 252.0, 1383.0, 288.0, 810.0, 286.0], "score": 1.0, "text": "Traralgon Creek would be expected to have both the"}, {"category_id": 15, "poly": [780.0, 286.0, 1385.0, 286.0, 1385.0, 322.0, 780.0, 322.0], "score": 0.98, "text": "most subdued flow reductions and longer response time"}, {"category_id": 15, "poly": [780.0, 355.0, 1385.0, 355.0, 1385.0, 385.0, 780.0, 385.0], "score": 0.97, "text": "uncertain vegetation record. Peak stand water use of a"}, {"category_id": 15, "poly": [777.0, 389.0, 1381.0, 389.0, 1381.0, 419.0, 777.0, 419.0], "score": 0.99, "text": "natural stand of this species is around 30 years."}, {"category_id": 15, "poly": [782.0, 421.0, 1383.0, 421.0, 1383.0, 452.0, 782.0, 452.0], "score": 0.98, "text": "Additionally in this large, \u2018real world\u2019 catchment,"}, {"category_id": 15, "poly": [780.0, 456.0, 1385.0, 456.0, 1385.0, 486.0, 780.0, 486.0], "score": 0.99, "text": "there is a continuous cycle of forest management"}, {"category_id": 15, "poly": [782.0, 488.0, 1385.0, 488.0, 1385.0, 518.0, 782.0, 518.0], "score": 0.98, "text": "which includes harvesting. A mixture of pasture and"}, {"category_id": 15, "poly": [784.0, 522.0, 1383.0, 522.0, 1383.0, 553.0, 784.0, 553.0], "score": 0.99, "text": "'scrub', which could represent significant understorey"}, {"category_id": 15, "poly": [780.0, 555.0, 1381.0, 555.0, 1381.0, 585.0, 780.0, 585.0], "score": 0.99, "text": "stands, were replaced by plantation species. Conse-"}, {"category_id": 15, "poly": [780.0, 589.0, 1385.0, 589.0, 1385.0, 619.0, 780.0, 619.0], "score": 0.99, "text": "quently the difference between pre and post treatment"}, {"category_id": 15, "poly": [777.0, 619.0, 1383.0, 619.0, 1383.0, 649.0, 777.0, 649.0], "score": 0.98, "text": "ET may be less than at other catchments. Reductions of"}, {"category_id": 15, "poly": [777.0, 651.0, 1385.0, 651.0, 1385.0, 688.0, 777.0, 688.0], "score": 1.0, "text": "this magnitude could be more readily expected in larger,"}, {"category_id": 15, "poly": [780.0, 688.0, 1383.0, 688.0, 1383.0, 718.0, 780.0, 718.0], "score": 0.99, "text": "multi land use catchments than the very high impacts"}, {"category_id": 15, "poly": [782.0, 720.0, 1293.0, 720.0, 1293.0, 750.0, 782.0, 750.0], "score": 0.98, "text": "estimated at the smaller Australian catchments."}, {"category_id": 15, "poly": [780.0, 322.0, 1120.0, 322.0, 1120.0, 353.0, 780.0, 353.0], "score": 0.93, "text": "because of the large area of "}, {"category_id": 15, "poly": [1144.0, 322.0, 1385.0, 322.0, 1385.0, 353.0, 1144.0, 353.0], "score": 0.99, "text": "regnans forest, and"}, {"category_id": 15, "poly": [146.0, 1412.0, 720.0, 1415.0, 719.0, 1451.0, 146.0, 1449.0], "score": 0.98, "text": " The response groups may be in part explained by the"}, {"category_id": 15, "poly": [109.0, 1449.0, 722.0, 1447.0, 722.0, 1483.0, 110.0, 1486.0], "score": 0.96, "text": "storage characteristics of the catchments. Accurate"}, {"category_id": 15, "poly": [114.0, 1486.0, 717.0, 1486.0, 717.0, 1516.0, 114.0, 1516.0], "score": 0.99, "text": "measures of storage are not available from the literature,"}, {"category_id": 15, "poly": [114.0, 1518.0, 719.0, 1518.0, 719.0, 1546.0, 114.0, 1546.0], "score": 0.98, "text": "but the soil depths and the baseflow index (Table 1) both"}, {"category_id": 15, "poly": [114.0, 1552.0, 722.0, 1552.0, 722.0, 1580.0, 114.0, 1580.0], "score": 0.97, "text": "show the three south eastern Australian catchments with"}, {"category_id": 15, "poly": [114.0, 1587.0, 722.0, 1587.0, 722.0, 1617.0, 114.0, 1617.0], "score": 0.98, "text": "the greatest reduction are likely to have the lowest"}, {"category_id": 15, "poly": [109.0, 1617.0, 722.0, 1615.0, 722.0, 1651.0, 110.0, 1653.0], "score": 0.99, "text": " storage capacity. The greater flow reductions, particu-"}, {"category_id": 15, "poly": [116.0, 1651.0, 717.0, 1651.0, 717.0, 1681.0, 116.0, 1681.0], "score": 0.97, "text": "larly for low flows, could be expected under these"}, {"category_id": 15, "poly": [116.0, 1686.0, 719.0, 1686.0, 719.0, 1716.0, 116.0, 1716.0], "score": 0.99, "text": "conditions. Inclusion of a storage term in the model is an"}, {"category_id": 15, "poly": [116.0, 1718.0, 719.0, 1718.0, 719.0, 1748.0, 116.0, 1748.0], "score": 0.99, "text": "obvious option for improving the analysis. However the"}, {"category_id": 15, "poly": [116.0, 1752.0, 719.0, 1752.0, 719.0, 1782.0, 116.0, 1782.0], "score": 0.98, "text": "addition of extra parameters would be at the cost of"}, {"category_id": 15, "poly": [116.0, 1784.0, 717.0, 1784.0, 717.0, 1815.0, 116.0, 1815.0], "score": 0.99, "text": "maintaining model simplicity, particularly as character-"}, {"category_id": 15, "poly": [116.0, 1817.0, 518.0, 1817.0, 518.0, 1847.0, 116.0, 1847.0], "score": 1.0, "text": "ising a transient storage is not trivial."}, {"category_id": 15, "poly": [816.0, 755.0, 1381.0, 755.0, 1381.0, 785.0, 816.0, 785.0], "score": 0.97, "text": "The analysis of zero flow days was successful,"}, {"category_id": 15, "poly": [782.0, 787.0, 1383.0, 787.0, 1383.0, 817.0, 782.0, 817.0], "score": 0.99, "text": "demonstrating that the impact on flow intermittence can"}, {"category_id": 15, "poly": [780.0, 819.0, 1383.0, 819.0, 1383.0, 849.0, 780.0, 849.0], "score": 1.0, "text": "be evaluated without of the entire FDC. This was helpful"}, {"category_id": 15, "poly": [777.0, 854.0, 1381.0, 854.0, 1381.0, 884.0, 777.0, 884.0], "score": 0.98, "text": " as the change in the higher percentiles (low flows) could"}, {"category_id": 15, "poly": [782.0, 886.0, 1381.0, 886.0, 1381.0, 916.0, 782.0, 916.0], "score": 0.97, "text": "not always be modelled. The results for the three"}, {"category_id": 15, "poly": [782.0, 920.0, 1381.0, 920.0, 1381.0, 950.0, 782.0, 950.0], "score": 0.98, "text": "catchments analysed are a rather stark indication of the"}, {"category_id": 15, "poly": [782.0, 955.0, 1383.0, 955.0, 1383.0, 985.0, 782.0, 985.0], "score": 0.99, "text": "potential for highly increased zero flow periods in small"}, {"category_id": 15, "poly": [782.0, 987.0, 1381.0, 987.0, 1381.0, 1017.0, 782.0, 1017.0], "score": 0.99, "text": "catchments, at least in south-eastern Australia. However,"}, {"category_id": 15, "poly": [778.0, 1015.0, 1383.0, 1019.0, 1383.0, 1054.0, 777.0, 1049.0], "score": 0.99, "text": "it should be noted these curves probably represent a"}, {"category_id": 15, "poly": [777.0, 1054.0, 1383.0, 1054.0, 1383.0, 1084.0, 777.0, 1084.0], "score": 0.97, "text": " maximum response as they are all derived from small"}, {"category_id": 15, "poly": [777.0, 1088.0, 1381.0, 1088.0, 1381.0, 1118.0, 777.0, 1118.0], "score": 0.97, "text": "catchments with small storage capacities and large"}, {"category_id": 15, "poly": [780.0, 1120.0, 1385.0, 1120.0, 1385.0, 1150.0, 780.0, 1150.0], "score": 0.99, "text": "percentages of afforestation. This method could be used"}, {"category_id": 15, "poly": [777.0, 1150.0, 1381.0, 1150.0, 1381.0, 1180.0, 777.0, 1180.0], "score": 1.0, "text": "to determine change in the occurrence of any given flow"}, {"category_id": 15, "poly": [780.0, 1187.0, 1383.0, 1187.0, 1383.0, 1217.0, 780.0, 1217.0], "score": 0.98, "text": "in response to afforestation; e.g. to determine the"}, {"category_id": 15, "poly": [775.0, 1215.0, 1387.0, 1217.0, 1387.0, 1253.0, 775.0, 1251.0], "score": 0.96, "text": "likelihood of maintaining a reservoir storage or an"}, {"category_id": 15, "poly": [782.0, 1253.0, 1381.0, 1253.0, 1381.0, 1284.0, 782.0, 1284.0], "score": 1.0, "text": "environmental fow that requires an average critical flow."}, {"category_id": 15, "poly": [810.0, 1412.0, 1385.0, 1417.0, 1385.0, 1453.0, 810.0, 1449.0], "score": 0.98, "text": "This project sought to (i) develop a method to remove"}, {"category_id": 15, "poly": [780.0, 1453.0, 1383.0, 1453.0, 1383.0, 1481.0, 780.0, 1481.0], "score": 1.0, "text": "the climate signal from streamflow records to identify"}, {"category_id": 15, "poly": [780.0, 1486.0, 1383.0, 1486.0, 1383.0, 1516.0, 780.0, 1516.0], "score": 0.98, "text": "the impact of vegetation on flow from afforested"}, {"category_id": 15, "poly": [780.0, 1518.0, 1385.0, 1518.0, 1385.0, 1548.0, 780.0, 1548.0], "score": 1.0, "text": "catchments, and (ii) quantify this impact on the flow"}, {"category_id": 15, "poly": [777.0, 1550.0, 1383.0, 1550.0, 1383.0, 1580.0, 777.0, 1580.0], "score": 0.99, "text": "duration curve. A simple model was proposed that"}, {"category_id": 15, "poly": [775.0, 1582.0, 1387.0, 1582.0, 1387.0, 1619.0, 775.0, 1619.0], "score": 0.98, "text": " considered the age of plantation and the annual rainfall"}, {"category_id": 15, "poly": [777.0, 1619.0, 1385.0, 1619.0, 1385.0, 1649.0, 777.0, 1649.0], "score": 0.98, "text": "to be the principal drivers for evapotranspiration. This"}, {"category_id": 15, "poly": [780.0, 1651.0, 1385.0, 1651.0, 1385.0, 1679.0, 780.0, 1679.0], "score": 0.99, "text": "model was fitted to the observed deciles of the FDC, and"}, {"category_id": 15, "poly": [778.0, 1681.0, 1381.0, 1686.0, 1381.0, 1716.0, 777.0, 1711.0], "score": 0.97, "text": "the climate signal was then removed from the stream-"}, {"category_id": 15, "poly": [780.0, 1716.0, 1387.0, 1716.0, 1387.0, 1752.0, 780.0, 1752.0], "score": 0.99, "text": "flow records by adjusting the FDC for average rainfall"}, {"category_id": 15, "poly": [777.0, 1748.0, 1385.0, 1746.0, 1385.0, 1782.0, 778.0, 1785.0], "score": 0.98, "text": "over the period of record. The model was tested and"}, {"category_id": 15, "poly": [777.0, 1780.0, 1381.0, 1780.0, 1381.0, 1817.0, 777.0, 1817.0], "score": 0.99, "text": "applied to 10 afforested catchments. We successfully"}, {"category_id": 15, "poly": [778.0, 1810.0, 1385.0, 1815.0, 1385.0, 1851.0, 777.0, 1847.0], "score": 1.0, "text": "fitted our model to catchments with varying spatial"}, {"category_id": 15, "poly": [1342.0, 189.0, 1387.0, 189.0, 1387.0, 234.0, 1342.0, 234.0], "score": 1.0, "text": "263"}], "page_info": {"page_no": 10, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 0, "poly": [132.2023162841797, 944.2422485351562, 352.39361572265625, 944.2422485351562, 352.39361572265625, 973.2088623046875, 132.2023162841797, 973.2088623046875], "score": 0.9999986886978149}, {"category_id": 2, "poly": [480.25787353515625, 196.37384033203125, 1044.1737060546875, 196.37384033203125, 1044.1737060546875, 218.59146118164062, 480.25787353515625, 218.59146118164062], "score": 0.9999973177909851}, {"category_id": 1, "poly": [131.0494842529297, 255.921875, 730.36865234375, 255.921875, 730.36865234375, 848.8026123046875, 131.0494842529297, 848.8026123046875], "score": 0.9999949336051941}, {"category_id": 1, "poly": [130.6344757080078, 1010.9166870117188, 730.247314453125, 1010.9166870117188, 730.247314453125, 1437.8150634765625, 130.6344757080078, 1437.8150634765625], "score": 0.9999939203262329}, {"category_id": 2, "poly": [130.68336486816406, 196.22927856445312, 164.99691772460938, 196.22927856445312, 164.99691772460938, 215.17306518554688, 130.68336486816406, 215.17306518554688], "score": 0.9999921321868896}, {"category_id": 1, "poly": [131.60971069335938, 1597.386962890625, 732.5863647460938, 1597.386962890625, 732.5863647460938, 1846.581787109375, 131.60971069335938, 1846.581787109375], "score": 0.9999908208847046}, {"category_id": 1, "poly": [791.6022338867188, 251.6699676513672, 1397.9674072265625, 251.6699676513672, 1397.9674072265625, 1848.8499755859375, 791.6022338867188, 1848.8499755859375], "score": 0.9999874830245972}, {"category_id": 0, "poly": [131.00613403320312, 1534.647705078125, 256.1180725097656, 1534.647705078125, 256.1180725097656, 1561.1875, 131.00613403320312, 1561.1875], "score": 0.9999844431877136}, {"category_id": 13, "poly": [1067, 1022, 1120, 1022, 1120, 1049, 1067, 1049], "score": 0.57, "latex": "219{\\mathrm{~p~}}"}, {"category_id": 15, "poly": [129.0, 939.0, 357.0, 944.0, 356.0, 983.0, 129.0, 978.0], "score": 1.0, "text": "Acknowledgements"}, {"category_id": 15, "poly": [481.0, 194.0, 1046.0, 194.0, 1046.0, 224.0, 481.0, 224.0], "score": 0.97, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [127.0, 252.0, 734.0, 249.0, 735.0, 286.0, 127.0, 288.0], "score": 0.99, "text": "scales, species and environments, and have shown that it"}, {"category_id": 15, "poly": [129.0, 290.0, 732.0, 290.0, 732.0, 320.0, 129.0, 320.0], "score": 0.99, "text": "provides a means of separating the influence of climate"}, {"category_id": 15, "poly": [129.0, 322.0, 732.0, 322.0, 732.0, 353.0, 129.0, 353.0], "score": 0.99, "text": "and vegetation on the FDCs. The modelled results"}, {"category_id": 15, "poly": [129.0, 353.0, 730.0, 353.0, 730.0, 383.0, 129.0, 383.0], "score": 0.97, "text": "showed the greatest proportional impacts were for"}, {"category_id": 15, "poly": [129.0, 387.0, 732.0, 387.0, 732.0, 417.0, 129.0, 417.0], "score": 0.99, "text": "median and lower flows. The flow reductions from the"}, {"category_id": 15, "poly": [129.0, 421.0, 732.0, 421.0, 732.0, 449.0, 129.0, 449.0], "score": 0.98, "text": "three small catchments SE Australian were the highest"}, {"category_id": 15, "poly": [129.0, 456.0, 735.0, 456.0, 735.0, 486.0, 129.0, 486.0], "score": 0.99, "text": "and may reflect lower storages. The characterisation of"}, {"category_id": 15, "poly": [129.0, 488.0, 732.0, 488.0, 732.0, 518.0, 129.0, 518.0], "score": 0.99, "text": "the number of zero flow days was also successful for"}, {"category_id": 15, "poly": [129.0, 522.0, 735.0, 522.0, 735.0, 553.0, 129.0, 553.0], "score": 0.98, "text": "these catchments in indicating a significant increase in"}, {"category_id": 15, "poly": [129.0, 555.0, 730.0, 555.0, 730.0, 585.0, 129.0, 585.0], "score": 0.99, "text": "zero flows. The flow reductions identified here probably"}, {"category_id": 15, "poly": [127.0, 589.0, 732.0, 589.0, 732.0, 619.0, 127.0, 619.0], "score": 0.97, "text": "represent a maximum effect given the size of the"}, {"category_id": 15, "poly": [129.0, 619.0, 728.0, 619.0, 728.0, 649.0, 129.0, 649.0], "score": 0.99, "text": "catchments, level of afforestation and the shallow soils."}, {"category_id": 15, "poly": [129.0, 654.0, 732.0, 654.0, 732.0, 684.0, 129.0, 684.0], "score": 0.98, "text": "These results have yielded useful new insights on the"}, {"category_id": 15, "poly": [129.0, 688.0, 735.0, 688.0, 735.0, 718.0, 129.0, 718.0], "score": 0.99, "text": "contentious issue of the hydrological impact of"}, {"category_id": 15, "poly": [129.0, 720.0, 732.0, 720.0, 732.0, 750.0, 129.0, 750.0], "score": 0.99, "text": "afforestation. This research has led to the development"}, {"category_id": 15, "poly": [129.0, 755.0, 732.0, 755.0, 732.0, 785.0, 129.0, 785.0], "score": 0.99, "text": "of a method to assess the net impact of afforestation on"}, {"category_id": 15, "poly": [129.0, 787.0, 730.0, 787.0, 730.0, 817.0, 129.0, 817.0], "score": 0.99, "text": "the fow duration curve which does not require paired-"}, {"category_id": 15, "poly": [127.0, 819.0, 591.0, 821.0, 591.0, 851.0, 127.0, 849.0], "score": 0.98, "text": "catchments to remove climatic variability."}, {"category_id": 15, "poly": [163.0, 1010.0, 730.0, 1010.0, 730.0, 1041.0, 163.0, 1041.0], "score": 0.98, "text": "The authors would like to thank Rory Nathan,"}, {"category_id": 15, "poly": [127.0, 1041.0, 735.0, 1041.0, 735.0, 1077.0, 127.0, 1077.0], "score": 1.0, "text": "Narendra Tuteja, Tom McMahon, Geoff Podger, Rob"}, {"category_id": 15, "poly": [131.0, 1077.0, 728.0, 1077.0, 728.0, 1107.0, 131.0, 1107.0], "score": 0.99, "text": "Vertessy, Glen Walker and Peter Hairsine for particu-"}, {"category_id": 15, "poly": [129.0, 1109.0, 732.0, 1109.0, 732.0, 1140.0, 129.0, 1140.0], "score": 0.99, "text": "larly helpful discussions on methodologies and reviews,"}, {"category_id": 15, "poly": [127.0, 1142.0, 732.0, 1144.0, 732.0, 1174.0, 127.0, 1172.0], "score": 1.0, "text": "Richard Morton for valuable statistical advice, Dave"}, {"category_id": 15, "poly": [129.0, 1178.0, 730.0, 1178.0, 730.0, 1208.0, 129.0, 1208.0], "score": 1.0, "text": "Scott for supplying the South African data, Barry Fahey"}, {"category_id": 15, "poly": [127.0, 1208.0, 732.0, 1210.0, 732.0, 1241.0, 127.0, 1238.0], "score": 0.99, "text": "for the New Zealand data, and Hancocks Victorian"}, {"category_id": 15, "poly": [127.0, 1241.0, 730.0, 1243.0, 730.0, 1273.0, 127.0, 1271.0], "score": 0.99, "text": "Plantations for vegetation data. The study was funded by"}, {"category_id": 15, "poly": [129.0, 1277.0, 735.0, 1277.0, 735.0, 1307.0, 129.0, 1307.0], "score": 0.97, "text": "the Victorian Department of Natural Resources and"}, {"category_id": 15, "poly": [129.0, 1312.0, 732.0, 1312.0, 732.0, 1339.0, 129.0, 1339.0], "score": 0.98, "text": "Environment Private Forestry Unit, the CRC for"}, {"category_id": 15, "poly": [129.0, 1344.0, 735.0, 1344.0, 735.0, 1374.0, 129.0, 1374.0], "score": 1.0, "text": "Catchment Hydrology, and the MDBC funded project"}, {"category_id": 15, "poly": [127.0, 1372.0, 735.0, 1374.0, 734.0, 1410.0, 127.0, 1408.0], "score": 0.99, "text": "\u201cIntegrated assessment of the effects of land use changes"}, {"category_id": 15, "poly": [127.0, 1410.0, 558.0, 1408.0, 558.0, 1438.0, 127.0, 1441.0], "score": 0.98, "text": "on water yield and salt loads\u2019 (D2013)."}, {"category_id": 15, "poly": [127.0, 189.0, 170.0, 189.0, 170.0, 228.0, 127.0, 228.0], "score": 1.0, "text": "264"}, {"category_id": 15, "poly": [127.0, 1593.0, 732.0, 1598.0, 732.0, 1632.0, 127.0, 1627.0], "score": 0.97, "text": "Bosch, J.M., 1979. Treatment effects on annual and dry period"}, {"category_id": 15, "poly": [157.0, 1625.0, 732.0, 1623.0, 732.0, 1653.0, 157.0, 1656.0], "score": 0.99, "text": " streamflow at Cathedral Peak. South African Forestry Journal 108,"}, {"category_id": 15, "poly": [161.0, 1651.0, 230.0, 1651.0, 230.0, 1681.0, 161.0, 1681.0], "score": 0.99, "text": "29-37."}, {"category_id": 15, "poly": [127.0, 1681.0, 732.0, 1681.0, 732.0, 1709.0, 127.0, 1709.0], "score": 0.97, "text": "Bosch, J.M., Von Gadow, K., 1990. Regulating afforestation for water"}, {"category_id": 15, "poly": [159.0, 1709.0, 730.0, 1707.0, 730.0, 1737.0, 159.0, 1739.0], "score": 0.98, "text": " conservation in South Africa. Suid-Afrikaanse Bosboutydskrif 153,"}, {"category_id": 15, "poly": [163.0, 1739.0, 228.0, 1739.0, 228.0, 1763.0, 163.0, 1763.0], "score": 1.0, "text": "41-54."}, {"category_id": 15, "poly": [127.0, 1763.0, 735.0, 1765.0, 734.0, 1795.0, 127.0, 1793.0], "score": 0.97, "text": "Chiew, F.H.S., McMahon, T.A., 1993. Assessing the adequacy of"}, {"category_id": 15, "poly": [161.0, 1793.0, 735.0, 1793.0, 735.0, 1821.0, 161.0, 1821.0], "score": 0.97, "text": "catchment streamflow yield estimates. Australian Journal of Soil"}, {"category_id": 15, "poly": [163.0, 1819.0, 365.0, 1819.0, 365.0, 1847.0, 163.0, 1847.0], "score": 1.0, "text": "Research 31, 665-680."}, {"category_id": 15, "poly": [791.0, 251.0, 1398.0, 256.0, 1398.0, 286.0, 790.0, 282.0], "score": 0.99, "text": "Cornish, P.M., Vertessy, R.A., 2001. Forest age-induced changes in"}, {"category_id": 15, "poly": [823.0, 284.0, 1398.0, 282.0, 1398.0, 312.0, 823.0, 314.0], "score": 0.98, "text": " evapotranspiration and water yield in a eucalypt forest. Journal of"}, {"category_id": 15, "poly": [825.0, 312.0, 1033.0, 312.0, 1033.0, 342.0, 825.0, 342.0], "score": 1.0, "text": "Hydrology 242, 43-63."}, {"category_id": 15, "poly": [788.0, 338.0, 1398.0, 340.0, 1398.0, 370.0, 788.0, 368.0], "score": 1.0, "text": "Fahey, B., Jackson, R., 1997. Hydrological impacts of converting"}, {"category_id": 15, "poly": [820.0, 366.0, 1398.0, 363.0, 1398.0, 400.0, 821.0, 402.0], "score": 0.97, "text": "native forests and grasslands to pine plantations, South"}, {"category_id": 15, "poly": [821.0, 393.0, 1396.0, 396.0, 1396.0, 428.0, 820.0, 426.0], "score": 0.98, "text": " Island, New Zealand. Agricultural and Forest Meteorology 84,"}, {"category_id": 15, "poly": [825.0, 424.0, 889.0, 424.0, 889.0, 454.0, 825.0, 454.0], "score": 1.0, "text": "69-82."}, {"category_id": 15, "poly": [788.0, 451.0, 1396.0, 454.0, 1396.0, 484.0, 788.0, 482.0], "score": 0.99, "text": "Hickel, K., 2001. The effect of pine afforestation on flow regime in"}, {"category_id": 15, "poly": [823.0, 479.0, 1398.0, 484.0, 1398.0, 512.0, 822.0, 507.0], "score": 0.97, "text": "small upland catchments. Masters Thesis, University of Stuttgart,"}, {"category_id": 15, "poly": [820.0, 510.0, 889.0, 505.0, 892.0, 537.0, 822.0, 543.0], "score": 0.94, "text": "p. 134."}, {"category_id": 15, "poly": [790.0, 540.0, 1396.0, 540.0, 1396.0, 568.0, 790.0, 568.0], "score": 1.0, "text": "Holmes, J.W., Sinclair, J.A., 1986. Water yield from some afforested"}, {"category_id": 15, "poly": [825.0, 570.0, 1398.0, 570.0, 1398.0, 598.0, 825.0, 598.0], "score": 0.98, "text": "catchments in Victoria. In Hydrology and Water Resources"}, {"category_id": 15, "poly": [825.0, 596.0, 1398.0, 596.0, 1398.0, 626.0, 825.0, 626.0], "score": 0.99, "text": "Symposium, Griffth University, Brisbane 25-27 November 1986,"}, {"category_id": 15, "poly": [820.0, 626.0, 939.0, 619.0, 941.0, 649.0, 822.0, 656.0], "score": 0.95, "text": "pp. 214-218."}, {"category_id": 15, "poly": [790.0, 654.0, 1398.0, 654.0, 1398.0, 682.0, 790.0, 682.0], "score": 0.99, "text": "Lane, P.N.J., Best, A.E., Hickel, K., Zhang, L., 2003. The effect"}, {"category_id": 15, "poly": [825.0, 682.0, 1398.0, 682.0, 1398.0, 710.0, 825.0, 710.0], "score": 0.99, "text": "of afforestation on flow duration curves. Cooperative Research"}, {"category_id": 15, "poly": [825.0, 710.0, 1396.0, 710.0, 1396.0, 740.0, 825.0, 740.0], "score": 0.97, "text": "Centre for Catchment Hydrology Technical Report O3/13,"}, {"category_id": 15, "poly": [820.0, 745.0, 884.0, 739.0, 886.0, 763.0, 822.0, 768.0], "score": 0.96, "text": "p.25."}, {"category_id": 15, "poly": [790.0, 768.0, 1396.0, 768.0, 1396.0, 798.0, 790.0, 798.0], "score": 0.98, "text": "Legates, D.R., McCabe, G.J., 1999. Evaluating the use of 'goodness-"}, {"category_id": 15, "poly": [825.0, 796.0, 1396.0, 796.0, 1396.0, 823.0, 825.0, 823.0], "score": 0.98, "text": "of-fit\u2019 measures in hydrologic and hydroclimatic model validation."}, {"category_id": 15, "poly": [825.0, 823.0, 1181.0, 823.0, 1181.0, 851.0, 825.0, 851.0], "score": 1.0, "text": "Water Resources Research 35, 233-241."}, {"category_id": 15, "poly": [790.0, 851.0, 1398.0, 851.0, 1398.0, 882.0, 790.0, 882.0], "score": 0.98, "text": "Lyne, V.D., Hollick, M., 1979. Stochastic time-varying rainfall-runoff"}, {"category_id": 15, "poly": [825.0, 882.0, 1398.0, 882.0, 1398.0, 912.0, 825.0, 912.0], "score": 1.0, "text": "modelling. Hydrology and Water Resources Symposium, Perth."}, {"category_id": 15, "poly": [825.0, 909.0, 1224.0, 909.0, 1224.0, 940.0, 825.0, 940.0], "score": 0.98, "text": "Institution of Engineers, Australia, pp. 89-92."}, {"category_id": 15, "poly": [788.0, 935.0, 1398.0, 937.0, 1398.0, 968.0, 788.0, 965.0], "score": 0.98, "text": "Nandakumar, N., Mein, R.G., 1993. Analysis of paired catchment data"}, {"category_id": 15, "poly": [825.0, 965.0, 1398.0, 965.0, 1398.0, 995.0, 825.0, 995.0], "score": 1.0, "text": "to determine the hydrologic effects of changes in vegetative cover"}, {"category_id": 15, "poly": [827.0, 995.0, 1396.0, 995.0, 1396.0, 1026.0, 827.0, 1026.0], "score": 0.99, "text": "on yield. Technical Report for Project UM010, Monash University"}, {"category_id": 15, "poly": [788.0, 1049.0, 1398.0, 1051.0, 1398.0, 1081.0, 788.0, 1079.0], "score": 0.98, "text": "Nash, J.E., Sutcliffe, J.V., 1970. River fow forecasting through"}, {"category_id": 15, "poly": [825.0, 1079.0, 1400.0, 1079.0, 1400.0, 1109.0, 825.0, 1109.0], "score": 0.97, "text": "conceptual models, I, A discussion of principals. Journal of"}, {"category_id": 15, "poly": [825.0, 1109.0, 1042.0, 1109.0, 1042.0, 1137.0, 825.0, 1137.0], "score": 1.0, "text": "Hydrology 10, 282-290."}, {"category_id": 15, "poly": [790.0, 1137.0, 1398.0, 1137.0, 1398.0, 1165.0, 790.0, 1165.0], "score": 0.98, "text": "Putahena, W.M., Cordery, I., 2000. Some hydrological effects of"}, {"category_id": 15, "poly": [827.0, 1165.0, 1398.0, 1165.0, 1398.0, 1195.0, 827.0, 1195.0], "score": 0.99, "text": "changing forest cover from eucalyptus to Pinus radiata. Agricul-"}, {"category_id": 15, "poly": [825.0, 1193.0, 1192.0, 1193.0, 1192.0, 1223.0, 825.0, 1223.0], "score": 0.99, "text": "tural and Forest Meteorology 100, 59-72."}, {"category_id": 15, "poly": [790.0, 1223.0, 1398.0, 1223.0, 1398.0, 1253.0, 790.0, 1253.0], "score": 0.99, "text": "Roberts, S., Vertessy, R.A., Grayson, R.G., 2001. Transpiration from"}, {"category_id": 15, "poly": [825.0, 1251.0, 1400.0, 1251.0, 1400.0, 1281.0, 825.0, 1281.0], "score": 0.99, "text": "Eucalyptus sieberi (L. Johnson) forests of different age. Forest "}, {"category_id": 15, "poly": [825.0, 1279.0, 1183.0, 1279.0, 1183.0, 1309.0, 825.0, 1309.0], "score": 1.0, "text": "Ecology and Management 143, 153-161."}, {"category_id": 15, "poly": [788.0, 1305.0, 1398.0, 1307.0, 1398.0, 1337.0, 788.0, 1335.0], "score": 0.99, "text": "Scott, D.F., Smith, R.E., 1997. Preliminary empirical models to predict"}, {"category_id": 15, "poly": [823.0, 1333.0, 1398.0, 1335.0, 1398.0, 1365.0, 823.0, 1363.0], "score": 0.98, "text": "reductions in total and low flows resulting from afforestation."}, {"category_id": 15, "poly": [825.0, 1363.0, 1046.0, 1363.0, 1046.0, 1393.0, 825.0, 1393.0], "score": 0.99, "text": "Water S.A. 23, 135-140."}, {"category_id": 15, "poly": [790.0, 1393.0, 1398.0, 1393.0, 1398.0, 1421.0, 790.0, 1421.0], "score": 0.97, "text": "Scott, D.F., Prinsloo, F.W., Moses, G., Mehlomakulu, M.,"}, {"category_id": 15, "poly": [825.0, 1421.0, 1398.0, 1421.0, 1398.0, 1449.0, 825.0, 1449.0], "score": 0.97, "text": "Simmers, A.D.A., 2000. Area-analysis of the South African"}, {"category_id": 15, "poly": [825.0, 1449.0, 1398.0, 1449.0, 1398.0, 1479.0, 825.0, 1479.0], "score": 0.96, "text": "catchment afforestation experimental data. WRC Report"}, {"category_id": 15, "poly": [825.0, 1481.0, 954.0, 1481.0, 954.0, 1505.0, 825.0, 1505.0], "score": 0.98, "text": "No. 810/1/00."}, {"category_id": 15, "poly": [790.0, 1507.0, 1396.0, 1507.0, 1396.0, 1535.0, 790.0, 1535.0], "score": 0.98, "text": "Sikka, A.K., Samra, JS., Sharda, V.N., Samraj, P., Lakshmanan, V.,"}, {"category_id": 15, "poly": [825.0, 1535.0, 1400.0, 1535.0, 1400.0, 1565.0, 825.0, 1565.0], "score": 0.98, "text": "2003. Low fow and high responses to converting natural grassland"}, {"category_id": 15, "poly": [827.0, 1561.0, 1400.0, 1561.0, 1400.0, 1591.0, 827.0, 1591.0], "score": 0.99, "text": "into bluegum (Eucalyptus globulus) in Ningiris watersheds of"}, {"category_id": 15, "poly": [825.0, 1591.0, 1235.0, 1591.0, 1235.0, 1621.0, 825.0, 1621.0], "score": 0.99, "text": "South India. Journal of Hydrology 270, 12-26."}, {"category_id": 15, "poly": [790.0, 1621.0, 1398.0, 1621.0, 1398.0, 1651.0, 790.0, 1651.0], "score": 0.98, "text": " Smakhtin, V.U., 1999. A concept of pragmatic hydrological time series "}, {"category_id": 15, "poly": [825.0, 1649.0, 1398.0, 1649.0, 1398.0, 1679.0, 825.0, 1679.0], "score": 0.99, "text": "modelling and its application in South African context. In Ninth"}, {"category_id": 15, "poly": [823.0, 1675.0, 1398.0, 1677.0, 1398.0, 1707.0, 823.0, 1705.0], "score": 0.98, "text": " South African National Hydrology Symposium, 29-30 November"}, {"category_id": 15, "poly": [825.0, 1703.0, 971.0, 1703.0, 971.0, 1739.0, 825.0, 1739.0], "score": 0.99, "text": "1999, pp. 1-11."}, {"category_id": 15, "poly": [790.0, 1735.0, 1398.0, 1735.0, 1398.0, 1765.0, 790.0, 1765.0], "score": 0.98, "text": " Smakhtin, V.U., 2001. Low flow hydrology: a review. Journal of"}, {"category_id": 15, "poly": [825.0, 1763.0, 1052.0, 1763.0, 1052.0, 1793.0, 825.0, 1793.0], "score": 1.0, "text": "Hydrology 240, 147-186."}, {"category_id": 15, "poly": [793.0, 1791.0, 1398.0, 1791.0, 1398.0, 1821.0, 793.0, 1821.0], "score": 0.99, "text": "Van Lill, W.S., Kruger, F.J., Van Wyk, D.B., 1980. The effect of"}, {"category_id": 15, "poly": [827.0, 1819.0, 1398.0, 1819.0, 1398.0, 1849.0, 827.0, 1849.0], "score": 0.98, "text": "afforestation with Eucalyptus grandis Hill ex Maiden and Pinus"}, {"category_id": 15, "poly": [823.0, 1021.0, 1066.0, 1023.0, 1066.0, 1054.0, 822.0, 1051.0], "score": 0.98, "text": " Dept. of Civil Engineering,"}, {"category_id": 15, "poly": [126.0, 1528.0, 260.0, 1533.0, 259.0, 1572.0, 124.0, 1567.0], "score": 1.0, "text": "References"}], "page_info": {"page_no": 11, "height": 2064, "width": 1512}}, {"layout_dets": [{"category_id": 2, "poly": [465.48040771484375, 195.6739959716797, 1032.2401123046875, 195.6739959716797, 1032.2401123046875, 218.9838104248047, 465.48040771484375, 218.9838104248047], "score": 0.9999986886978149}, {"category_id": 1, "poly": [776.9209594726562, 255.59912109375, 1385.6478271484375, 255.59912109375, 1385.6478271484375, 614.4959716796875, 776.9209594726562, 614.4959716796875], "score": 0.9999933242797852}, {"category_id": 2, "poly": [1346.0157470703125, 195.03271484375, 1382.0159912109375, 195.03271484375, 1382.0159912109375, 217.2877960205078, 1346.0157470703125, 217.2877960205078], "score": 0.9999925494194031}, {"category_id": 1, "poly": [116.54571533203125, 257.5740966796875, 716.8768920898438, 257.5740966796875, 716.8768920898438, 615.0397338867188, 116.54571533203125, 615.0397338867188], "score": 0.9999920725822449}, {"category_id": 15, "poly": [466.0, 194.0, 1033.0, 194.0, 1033.0, 224.0, 466.0, 224.0], "score": 0.99, "text": "P.N.J. Lane et al. / Journal of Hydrology 310 (2005) 253-265"}, {"category_id": 15, "poly": [780.0, 254.0, 1383.0, 254.0, 1383.0, 284.0, 780.0, 284.0], "score": 0.99, "text": "Vogel, R.M., Fennessey, N.M., 1994. Flow duration curves. 1. New"}, {"category_id": 15, "poly": [810.0, 279.0, 1385.0, 282.0, 1385.0, 312.0, 810.0, 310.0], "score": 0.98, "text": " interpretation and confidence intervals. Journal of Water Planning"}, {"category_id": 15, "poly": [814.0, 310.0, 1128.0, 310.0, 1128.0, 340.0, 814.0, 340.0], "score": 0.99, "text": "and Management 120 (4), 485-504."}, {"category_id": 15, "poly": [780.0, 338.0, 1387.0, 338.0, 1387.0, 366.0, 780.0, 366.0], "score": 0.98, "text": "Whitehead, D., Beadle C.L., 2004. Physiological regulation of"}, {"category_id": 15, "poly": [808.0, 361.0, 1387.0, 363.0, 1387.0, 400.0, 807.0, 398.0], "score": 0.98, "text": " productivity and water use in Eucalyptus: a review. Forest Ecology"}, {"category_id": 15, "poly": [812.0, 393.0, 1104.0, 393.0, 1104.0, 424.0, 812.0, 424.0], "score": 1.0, "text": "and Management, 193, 113-140."}, {"category_id": 15, "poly": [777.0, 421.0, 1385.0, 421.0, 1385.0, 449.0, 777.0, 449.0], "score": 0.98, "text": "Zhang, L., Dawes, W.R., Walker, G.R., 1999. Predicting the effect of"}, {"category_id": 15, "poly": [814.0, 449.0, 1383.0, 449.0, 1383.0, 479.0, 814.0, 479.0], "score": 0.99, "text": "vegetation changes on catchment average water balance. Coop-"}, {"category_id": 15, "poly": [812.0, 475.0, 1385.0, 475.0, 1385.0, 505.0, 812.0, 505.0], "score": 0.99, "text": "erative Research Centre for Catchment Hydrology Technical"}, {"category_id": 15, "poly": [810.0, 503.0, 994.0, 503.0, 994.0, 533.0, 810.0, 533.0], "score": 0.99, "text": "Report 99/12, p. 35."}, {"category_id": 15, "poly": [777.0, 531.0, 1385.0, 531.0, 1385.0, 561.0, 777.0, 561.0], "score": 0.97, "text": "Zhang, L., Dawes, W.R., Walker, G.R., 2001. Response of mean"}, {"category_id": 15, "poly": [810.0, 557.0, 1385.0, 559.0, 1385.0, 589.0, 810.0, 587.0], "score": 0.98, "text": " annual evapotranspiration to vegetation changes at catchment"}, {"category_id": 15, "poly": [812.0, 587.0, 1222.0, 587.0, 1222.0, 615.0, 812.0, 615.0], "score": 1.0, "text": "scale. Water Resources Research 37, 701-708."}, {"category_id": 15, "poly": [1342.0, 189.0, 1387.0, 189.0, 1387.0, 234.0, 1342.0, 234.0], "score": 1.0, "text": "265"}, {"category_id": 15, "poly": [148.0, 254.0, 719.0, 254.0, 719.0, 284.0, 148.0, 284.0], "score": 1.0, "text": "patula Schlect. et Cham. on streamflow from experimental"}, {"category_id": 15, "poly": [146.0, 279.0, 720.0, 282.0, 719.0, 312.0, 146.0, 310.0], "score": 0.99, "text": "catchments at Mokubulaan, Transval. Journal of Hydrology 48,"}, {"category_id": 15, "poly": [150.0, 312.0, 234.0, 312.0, 234.0, 335.0, 150.0, 335.0], "score": 1.0, "text": "107-118."}, {"category_id": 15, "poly": [114.0, 338.0, 719.0, 338.0, 719.0, 366.0, 114.0, 366.0], "score": 0.97, "text": "Van Wyk, D.B., 1987. Some effects of afforestation on streamflow"}, {"category_id": 15, "poly": [144.0, 366.0, 719.0, 366.0, 719.0, 396.0, 144.0, 396.0], "score": 0.98, "text": "in the Western Cape Province, South Africa. Water S.A. 13,"}, {"category_id": 15, "poly": [148.0, 396.0, 210.0, 396.0, 210.0, 419.0, 148.0, 419.0], "score": 1.0, "text": "31-36."}, {"category_id": 15, "poly": [114.0, 421.0, 719.0, 421.0, 719.0, 452.0, 114.0, 452.0], "score": 0.98, "text": "Vertessy, R.A., Bessard, Y., 1999. Anticipating the negative"}, {"category_id": 15, "poly": [146.0, 449.0, 722.0, 449.0, 722.0, 479.0, 146.0, 479.0], "score": 0.98, "text": "hydrologic effects of plantation expansion: results from a"}, {"category_id": 15, "poly": [148.0, 475.0, 717.0, 475.0, 717.0, 503.0, 148.0, 503.0], "score": 0.98, "text": "GIS-based analysis on the Murrumbidgee Basin, in: Croke, J.,"}, {"category_id": 15, "poly": [146.0, 503.0, 722.0, 503.0, 722.0, 533.0, 146.0, 533.0], "score": 0.99, "text": "Lane, P.N.J. (Eds.), Forest Management for Water Quality and"}, {"category_id": 15, "poly": [144.0, 527.0, 722.0, 529.0, 722.0, 565.0, 144.0, 563.0], "score": 0.99, "text": "Quantity: Proceedings of the 2nd Erosion in Forests Meeting"}, {"category_id": 15, "poly": [146.0, 557.0, 722.0, 559.0, 722.0, 589.0, 146.0, 587.0], "score": 0.97, "text": " Cooperative Research Centre for Catchment Hydrology, Report "}, {"category_id": 15, "poly": [146.0, 587.0, 301.0, 587.0, 301.0, 617.0, 146.0, 617.0], "score": 0.93, "text": "99/6, Pp. 69-73."}], "page_info": {"page_no": 12, "height": 2064, "width": 1512}}]
\ No newline at end of file
diff --git a/demo/demo2.json b/demo/demo2.json
deleted file mode 100644
index d632d70fa..000000000
--- a/demo/demo2.json
+++ /dev/null
@@ -1 +0,0 @@
-[{"layout_dets": [{"category_id": 0, "poly": [282.1632080078125, 156.2249755859375, 1416.6795654296875, 156.2249755859375, 1416.6795654296875, 313.81280517578125, 282.1632080078125, 313.81280517578125], "score": 0.999998927116394}, {"category_id": 1, "poly": [861.656982421875, 522.7763061523438, 1569.3853759765625, 522.7763061523438, 1569.3853759765625, 656.883544921875, 861.656982421875, 656.883544921875], "score": 0.9999970197677612}, {"category_id": 1, "poly": [131.8020782470703, 924.7362670898438, 838.9530639648438, 924.7362670898438, 838.9530639648438, 1323.7529296875, 131.8020782470703, 1323.7529296875], "score": 0.9999949932098389}, {"category_id": 1, "poly": [133.32005310058594, 1324.5035400390625, 839.2289428710938, 1324.5035400390625, 839.2289428710938, 1589.4503173828125, 133.32005310058594, 1589.4503173828125], "score": 0.999994158744812}, {"category_id": 1, "poly": [863.3811645507812, 1486.610107421875, 1569.2880859375, 1486.610107421875, 1569.2880859375, 1852.443603515625, 863.3811645507812, 1852.443603515625], "score": 0.9999936819076538}, {"category_id": 1, "poly": [862.9096069335938, 1187.8067626953125, 1568.2279052734375, 1187.8067626953125, 1568.2279052734375, 1486.08935546875, 862.9096069335938, 1486.08935546875], "score": 0.9999932050704956}, {"category_id": 1, "poly": [131.8186492919922, 1652.7752685546875, 837.5543823242188, 1652.7752685546875, 837.5543823242188, 2019.429443359375, 131.8186492919922, 2019.429443359375], "score": 0.9999901056289673}, {"category_id": 0, "poly": [375.1526794433594, 881.8807983398438, 594.3075561523438, 881.8807983398438, 594.3075561523438, 913.4786987304688, 375.1526794433594, 913.4786987304688], "score": 0.9999892115592957}, {"category_id": 2, "poly": [636.1867065429688, 2099.795654296875, 1063.7423095703125, 2099.795654296875, 1063.7423095703125, 2124.524169921875, 636.1867065429688, 2124.524169921875], "score": 0.9999860525131226}, {"category_id": 0, "poly": [375.91864013671875, 1610.209228515625, 592.8395385742188, 1610.209228515625, 592.8395385742188, 1641.5789794921875, 375.91864013671875, 1641.5789794921875], "score": 0.9999815821647644}, {"category_id": 4, "poly": [860.6583251953125, 995.6574096679688, 1569.622314453125, 995.6574096679688, 1569.622314453125, 1126.8409423828125, 860.6583251953125, 1126.8409423828125], "score": 0.9999815821647644}, {"category_id": 1, "poly": [443.1008605957031, 353.8008728027344, 1250.531494140625, 353.8008728027344, 1250.531494140625, 464.65576171875, 443.1008605957031, 464.65576171875], "score": 0.9999791979789734}, {"category_id": 1, "poly": [130.8282928466797, 523.2079467773438, 836.5639038085938, 523.2079467773438, 836.5639038085938, 862.0206909179688, 130.8282928466797, 862.0206909179688], "score": 0.9999784231185913}, {"category_id": 1, "poly": [862.6514282226562, 1851.426513671875, 1568.510498046875, 1851.426513671875, 1568.510498046875, 2017.93359375, 862.6514282226562, 2017.93359375], "score": 0.9999769926071167}, {"category_id": 3, "poly": [882.3795166015625, 685.376708984375, 1544.4088134765625, 685.376708984375, 1544.4088134765625, 969.22265625, 882.3795166015625, 969.22265625], "score": 0.9994785785675049}, {"category_id": 13, "poly": [1195, 1062, 1226, 1062, 1226, 1096, 1195, 1096], "score": 0.88, "latex": "d_{p}"}, {"category_id": 13, "poly": [1304, 1030, 1327, 1030, 1327, 1061, 1304, 1061], "score": 0.65, "latex": "\\bar{\\bf p}"}, {"category_id": 15, "poly": [344.0, 165.0, 1354.0, 172.0, 1353.0, 236.0, 344.0, 229.0], "score": 0.99, "text": "Real-time Temporal Stereo Matching"}, {"category_id": 15, "poly": [293.0, 254.0, 1402.0, 254.0, 1402.0, 309.0, 293.0, 309.0], "score": 0.99, "text": "using Iterative Adaptive Support Weights"}, {"category_id": 15, "poly": [864.0, 527.0, 1568.0, 527.0, 1568.0, 559.0, 864.0, 559.0], "score": 0.99, "text": "disparity map. Note that individual disparities can be converted"}, {"category_id": 15, "poly": [864.0, 561.0, 1568.0, 561.0, 1568.0, 594.0, 864.0, 594.0], "score": 0.98, "text": "to actual depths if the geometry of the camera setup is"}, {"category_id": 15, "poly": [859.0, 587.0, 1568.0, 591.0, 1568.0, 630.0, 859.0, 626.0], "score": 0.98, "text": " known, i.e., the stereo configuration of cameras has been pre-"}, {"category_id": 15, "poly": [862.0, 626.0, 984.0, 626.0, 984.0, 658.0, 862.0, 658.0], "score": 1.0, "text": "calibrated."}, {"category_id": 15, "poly": [155.0, 921.0, 839.0, 924.0, 838.0, 963.0, 155.0, 960.0], "score": 0.98, "text": " Modern stereo matching algorithms achieve excellent results"}, {"category_id": 15, "poly": [127.0, 956.0, 838.0, 958.0, 838.0, 997.0, 127.0, 995.0], "score": 0.98, "text": " on static stereo images, as demonstrated by the Middlebury"}, {"category_id": 15, "poly": [132.0, 995.0, 836.0, 995.0, 836.0, 1027.0, 132.0, 1027.0], "score": 0.98, "text": "stereo performance benchmark [1], [2]. However, their ap-"}, {"category_id": 15, "poly": [134.0, 1027.0, 834.0, 1027.0, 834.0, 1059.0, 134.0, 1059.0], "score": 1.0, "text": "plication to stereo video sequences does not guarantee inter-"}, {"category_id": 15, "poly": [134.0, 1061.0, 836.0, 1061.0, 836.0, 1093.0, 134.0, 1093.0], "score": 0.99, "text": "frame consistency of matches extracted from subsequent stereo"}, {"category_id": 15, "poly": [132.0, 1095.0, 838.0, 1095.0, 838.0, 1125.0, 132.0, 1125.0], "score": 0.99, "text": "frame pairs. The lack of temporal consistency of matches"}, {"category_id": 15, "poly": [134.0, 1128.0, 836.0, 1128.0, 836.0, 1157.0, 134.0, 1157.0], "score": 1.0, "text": "between successive frames introduces spurious artifacts in the"}, {"category_id": 15, "poly": [132.0, 1160.0, 836.0, 1160.0, 836.0, 1192.0, 132.0, 1192.0], "score": 0.99, "text": "resulting disparity maps. The problem of obtaining temporally"}, {"category_id": 15, "poly": [132.0, 1194.0, 838.0, 1194.0, 838.0, 1226.0, 132.0, 1226.0], "score": 0.98, "text": "consistent sequences of disparity maps from video streams is"}, {"category_id": 15, "poly": [134.0, 1228.0, 838.0, 1228.0, 838.0, 1260.0, 134.0, 1260.0], "score": 0.98, "text": "known as the temporal stereo correspondence problem, yet"}, {"category_id": 15, "poly": [129.0, 1258.0, 841.0, 1260.0, 841.0, 1293.0, 129.0, 1290.0], "score": 0.98, "text": "the amount of research efforts oriented towards finding an"}, {"category_id": 15, "poly": [134.0, 1292.0, 760.0, 1292.0, 760.0, 1325.0, 134.0, 1325.0], "score": 0.99, "text": "effective solution to this problem is surprisingly small."}, {"category_id": 15, "poly": [157.0, 1320.0, 836.0, 1322.0, 836.0, 1361.0, 157.0, 1359.0], "score": 0.98, "text": " A method is proposed for real-time temporal stereo match-"}, {"category_id": 15, "poly": [134.0, 1361.0, 836.0, 1361.0, 836.0, 1393.0, 134.0, 1393.0], "score": 1.0, "text": "ing that efficiently propagates matching cost information be-"}, {"category_id": 15, "poly": [134.0, 1393.0, 836.0, 1393.0, 836.0, 1425.0, 134.0, 1425.0], "score": 0.99, "text": "tween consecutive frames of a stereo video sequence. This"}, {"category_id": 15, "poly": [132.0, 1423.0, 834.0, 1425.0, 834.0, 1458.0, 132.0, 1455.0], "score": 0.98, "text": "method is invariant to the number of prior frames being"}, {"category_id": 15, "poly": [134.0, 1458.0, 836.0, 1458.0, 836.0, 1490.0, 134.0, 1490.0], "score": 0.99, "text": "considered, and can be easily incorporated into any local stereo"}, {"category_id": 15, "poly": [132.0, 1492.0, 836.0, 1492.0, 836.0, 1524.0, 132.0, 1524.0], "score": 0.98, "text": "method based on edge-aware filters. The iterative adaptive"}, {"category_id": 15, "poly": [132.0, 1526.0, 838.0, 1526.0, 838.0, 1558.0, 132.0, 1558.0], "score": 0.99, "text": "support matching algorithm presented in [3] serves as a"}, {"category_id": 15, "poly": [132.0, 1558.0, 557.0, 1558.0, 557.0, 1590.0, 132.0, 1590.0], "score": 0.99, "text": "foundation for the proposed method."}, {"category_id": 15, "poly": [887.0, 1483.0, 1571.0, 1485.0, 1571.0, 1524.0, 887.0, 1522.0], "score": 0.98, "text": " In contrast, local methods, which are typically built upon"}, {"category_id": 15, "poly": [859.0, 1517.0, 1573.0, 1519.0, 1573.0, 1558.0, 859.0, 1556.0], "score": 0.97, "text": " the Winner-Takes-All (WTA) framework, have the property of "}, {"category_id": 15, "poly": [864.0, 1556.0, 1566.0, 1556.0, 1566.0, 1588.0, 864.0, 1588.0], "score": 0.99, "text": "computational regularity and are thus suitable for implemen-"}, {"category_id": 15, "poly": [862.0, 1588.0, 1566.0, 1588.0, 1566.0, 1620.0, 862.0, 1620.0], "score": 1.0, "text": "tation on parallel graphics hardware. Within the WTA frame-"}, {"category_id": 15, "poly": [862.0, 1616.0, 1568.0, 1618.0, 1568.0, 1657.0, 862.0, 1655.0], "score": 0.98, "text": "work, local stereo algorithms consider a range of disparity"}, {"category_id": 15, "poly": [864.0, 1655.0, 1566.0, 1655.0, 1566.0, 1687.0, 864.0, 1687.0], "score": 0.98, "text": "hypotheses and compute a volume of pixel-wise dissimilarity"}, {"category_id": 15, "poly": [862.0, 1689.0, 1571.0, 1689.0, 1571.0, 1721.0, 862.0, 1721.0], "score": 0.99, "text": "metrics between the reference image and the matched image at"}, {"category_id": 15, "poly": [862.0, 1723.0, 1568.0, 1721.0, 1568.0, 1753.0, 862.0, 1755.0], "score": 0.99, "text": "every considered disparity value. Final disparities are chosen"}, {"category_id": 15, "poly": [864.0, 1755.0, 1568.0, 1755.0, 1568.0, 1785.0, 864.0, 1785.0], "score": 1.0, "text": "from the cost volume by traversing through its values and"}, {"category_id": 15, "poly": [866.0, 1788.0, 1568.0, 1788.0, 1568.0, 1820.0, 866.0, 1820.0], "score": 0.99, "text": "selecting the disparities associated with minimum matching"}, {"category_id": 15, "poly": [859.0, 1817.0, 1377.0, 1820.0, 1377.0, 1859.0, 859.0, 1856.0], "score": 0.98, "text": " costs for every pixel of the reference image."}, {"category_id": 15, "poly": [885.0, 1187.0, 1571.0, 1187.0, 1571.0, 1226.0, 885.0, 1226.0], "score": 0.97, "text": " In their excellent taxonomy paper [1], Scharstein and"}, {"category_id": 15, "poly": [864.0, 1224.0, 1566.0, 1224.0, 1566.0, 1254.0, 864.0, 1254.0], "score": 0.99, "text": "Szeliski classify stereo algorithms as local or global meth-"}, {"category_id": 15, "poly": [859.0, 1249.0, 1571.0, 1254.0, 1570.0, 1293.0, 859.0, 1288.0], "score": 0.99, "text": " ods. Global methods, which offer outstanding accuracy, are"}, {"category_id": 15, "poly": [862.0, 1288.0, 1571.0, 1288.0, 1571.0, 1327.0, 862.0, 1327.0], "score": 0.98, "text": "typically derived from an energy minimization framework"}, {"category_id": 15, "poly": [859.0, 1322.0, 1566.0, 1322.0, 1566.0, 1352.0, 859.0, 1352.0], "score": 0.99, "text": "that allows for explicit integration of disparity smoothness"}, {"category_id": 15, "poly": [864.0, 1357.0, 1568.0, 1357.0, 1568.0, 1389.0, 864.0, 1389.0], "score": 0.99, "text": "constraints and thus is capable of regularizing the solution"}, {"category_id": 15, "poly": [864.0, 1391.0, 1568.0, 1391.0, 1568.0, 1421.0, 864.0, 1421.0], "score": 1.0, "text": "in weakly textured areas. The minimization, however, is often"}, {"category_id": 15, "poly": [864.0, 1423.0, 1568.0, 1423.0, 1568.0, 1455.0, 864.0, 1455.0], "score": 0.99, "text": "achieved using iterative methods or graph cuts, which do not"}, {"category_id": 15, "poly": [864.0, 1458.0, 1418.0, 1458.0, 1418.0, 1487.0, 864.0, 1487.0], "score": 0.99, "text": "lend themselves well to parallel implementation."}, {"category_id": 15, "poly": [155.0, 1650.0, 839.0, 1652.0, 838.0, 1691.0, 155.0, 1689.0], "score": 0.97, "text": " Stereo matching is the process of identifying correspon-"}, {"category_id": 15, "poly": [134.0, 1687.0, 838.0, 1687.0, 838.0, 1719.0, 134.0, 1719.0], "score": 0.99, "text": "dences between pixels in stereo images obtained using a"}, {"category_id": 15, "poly": [132.0, 1723.0, 838.0, 1721.0, 838.0, 1753.0, 132.0, 1755.0], "score": 0.98, "text": "pair of synchronized cameras. These correspondences are"}, {"category_id": 15, "poly": [134.0, 1755.0, 836.0, 1755.0, 836.0, 1788.0, 134.0, 1788.0], "score": 0.99, "text": "conveniently represented using the notion of disparity, i.e. the"}, {"category_id": 15, "poly": [134.0, 1788.0, 836.0, 1788.0, 836.0, 1820.0, 134.0, 1820.0], "score": 1.0, "text": "positional offset between two matching pixels. It is assumed"}, {"category_id": 15, "poly": [134.0, 1822.0, 836.0, 1822.0, 836.0, 1854.0, 134.0, 1854.0], "score": 0.99, "text": "that the stereo images are rectified, such that matching pixels"}, {"category_id": 15, "poly": [132.0, 1854.0, 836.0, 1854.0, 836.0, 1886.0, 132.0, 1886.0], "score": 0.99, "text": "are confined within corresponding rows of the images and"}, {"category_id": 15, "poly": [134.0, 1888.0, 838.0, 1888.0, 838.0, 1918.0, 134.0, 1918.0], "score": 1.0, "text": "thus disparities are restricted to the horizontal dimension, as"}, {"category_id": 15, "poly": [134.0, 1920.0, 838.0, 1920.0, 838.0, 1952.0, 134.0, 1952.0], "score": 1.0, "text": "illustrated in Figure 1. For visualization purposes, disparities"}, {"category_id": 15, "poly": [134.0, 1955.0, 838.0, 1955.0, 838.0, 1987.0, 134.0, 1987.0], "score": 0.99, "text": "recovered for every pixel of a reference image are stored"}, {"category_id": 15, "poly": [129.0, 1985.0, 841.0, 1982.0, 841.0, 2021.0, 129.0, 2024.0], "score": 0.98, "text": "together in the form of an image, which is known as the"}, {"category_id": 15, "poly": [370.0, 885.0, 594.0, 885.0, 594.0, 917.0, 370.0, 917.0], "score": 1.0, "text": "1. INTRODUCTION"}, {"category_id": 15, "poly": [638.0, 2099.0, 1062.0, 2099.0, 1062.0, 2131.0, 638.0, 2131.0], "score": 0.98, "text": "978-1-4673-5208-6/13/$31.00 @2013 IEEE"}, {"category_id": 15, "poly": [374.0, 1613.0, 591.0, 1613.0, 591.0, 1645.0, 374.0, 1645.0], "score": 0.95, "text": "II. BACKGROUND"}, {"category_id": 15, "poly": [859.0, 992.0, 1571.0, 995.0, 1571.0, 1034.0, 859.0, 1031.0], "score": 0.99, "text": " Figure 1: Geometry of two horizontally aligned views where p"}, {"category_id": 15, "poly": [864.0, 1098.0, 1291.0, 1098.0, 1291.0, 1130.0, 864.0, 1130.0], "score": 0.99, "text": "them along the horizontal dimension."}, {"category_id": 15, "poly": [859.0, 1061.0, 1194.0, 1059.0, 1194.0, 1098.0, 859.0, 1100.0], "score": 0.98, "text": " pixel in the target frame, and"}, {"category_id": 15, "poly": [1227.0, 1061.0, 1571.0, 1059.0, 1571.0, 1098.0, 1227.0, 1100.0], "score": 0.97, "text": " denotes the disparity between"}, {"category_id": 15, "poly": [864.0, 1034.0, 1303.0, 1034.0, 1303.0, 1063.0, 864.0, 1063.0], "score": 0.99, "text": "denotes a pixel in the reference frame,"}, {"category_id": 15, "poly": [1328.0, 1034.0, 1566.0, 1034.0, 1566.0, 1063.0, 1328.0, 1063.0], "score": 0.96, "text": " denotes its matching"}, {"category_id": 15, "poly": [508.0, 357.0, 1194.0, 360.0, 1194.0, 392.0, 508.0, 390.0], "score": 0.98, "text": "Jedrzej Kowalczuk, Eric T. Psota, and Lance C. P\u00e9rez"}, {"category_id": 15, "poly": [443.0, 392.0, 1245.0, 392.0, 1245.0, 424.0, 443.0, 424.0], "score": 0.99, "text": "Department of Electrical Engineering, University of Nebraska-Lincoln"}, {"category_id": 15, "poly": [614.0, 435.0, 1081.0, 435.0, 1081.0, 465.0, 614.0, 465.0], "score": 0.99, "text": "[jkowalczuk2,epsota,lperez] @unl.edu"}, {"category_id": 15, "poly": [159.0, 527.0, 836.0, 527.0, 836.0, 559.0, 159.0, 559.0], "score": 0.98, "text": "Abstract-Stereo matching algorithms are nearly always de-"}, {"category_id": 15, "poly": [132.0, 555.0, 838.0, 555.0, 838.0, 587.0, 132.0, 587.0], "score": 0.98, "text": "signed to find matches between a single pair of images. A method"}, {"category_id": 15, "poly": [134.0, 580.0, 836.0, 580.0, 836.0, 612.0, 134.0, 612.0], "score": 1.0, "text": "is presented that was specifically designed to operate on sequences"}, {"category_id": 15, "poly": [132.0, 605.0, 838.0, 607.0, 838.0, 646.0, 132.0, 644.0], "score": 0.99, "text": "of images. This method considers the cost of matching image"}, {"category_id": 15, "poly": [132.0, 637.0, 838.0, 637.0, 838.0, 669.0, 132.0, 669.0], "score": 0.98, "text": "points in both the spatial and temporal domain. To maintain"}, {"category_id": 15, "poly": [134.0, 667.0, 838.0, 667.0, 838.0, 699.0, 134.0, 699.0], "score": 0.97, "text": "real-time operation, a temporal cost aggregation method is used"}, {"category_id": 15, "poly": [132.0, 692.0, 836.0, 692.0, 836.0, 722.0, 132.0, 722.0], "score": 0.98, "text": "to evaluate the likelihood of matches that is invariant with respect"}, {"category_id": 15, "poly": [127.0, 717.0, 841.0, 715.0, 841.0, 754.0, 127.0, 756.0], "score": 0.97, "text": "to the number of prior images being considered. This method"}, {"category_id": 15, "poly": [127.0, 742.0, 841.0, 745.0, 841.0, 784.0, 127.0, 781.0], "score": 0.98, "text": "has been implemented on massively parallel GPU hardware,"}, {"category_id": 15, "poly": [132.0, 777.0, 838.0, 777.0, 838.0, 809.0, 132.0, 809.0], "score": 0.99, "text": "and the implementation ranks as one of the fastest and most"}, {"category_id": 15, "poly": [132.0, 802.0, 838.0, 804.0, 838.0, 836.0, 132.0, 834.0], "score": 0.99, "text": "accurate real-time stereo matching methods as measured by the"}, {"category_id": 15, "poly": [134.0, 830.0, 619.0, 830.0, 619.0, 862.0, 134.0, 862.0], "score": 0.99, "text": "Middlebury stereo performance benchmark."}, {"category_id": 15, "poly": [887.0, 1849.0, 1568.0, 1852.0, 1568.0, 1891.0, 887.0, 1888.0], "score": 0.99, "text": " Disparity maps obtained using this simple strategy are often"}, {"category_id": 15, "poly": [862.0, 1888.0, 1568.0, 1888.0, 1568.0, 1920.0, 862.0, 1920.0], "score": 0.98, "text": "too noisy to be considered useable. To reduce the effects"}, {"category_id": 15, "poly": [864.0, 1923.0, 1568.0, 1923.0, 1568.0, 1952.0, 864.0, 1952.0], "score": 0.99, "text": "of noise and enforce spatial consistency of matches, local"}, {"category_id": 15, "poly": [862.0, 1948.0, 1568.0, 1950.0, 1568.0, 1989.0, 861.0, 1987.0], "score": 0.99, "text": "stereo algorithms consider arbitrarily shaped and sized support"}, {"category_id": 15, "poly": [864.0, 1989.0, 1568.0, 1989.0, 1568.0, 2021.0, 864.0, 2021.0], "score": 0.99, "text": "windows centered at each pixel of the reference image, and"}], "page_info": {"page_no": 0, "height": 2200, "width": 1700}}, {"layout_dets": [{"category_id": 8, "poly": [962.3624267578125, 1513.2073974609375, 1465.4017333984375, 1513.2073974609375, 1465.4017333984375, 1669.1397705078125, 962.3624267578125, 1669.1397705078125], "score": 0.9999995231628418}, {"category_id": 9, "poly": [1530.72998046875, 1101.879638671875, 1565.2568359375, 1101.879638671875, 1565.2568359375, 1130.8609619140625, 1530.72998046875, 1130.8609619140625], "score": 0.9999992251396179}, {"category_id": 9, "poly": [1529.8787841796875, 1575.843505859375, 1565.931396484375, 1575.843505859375, 1565.931396484375, 1607.2161865234375, 1529.8787841796875, 1607.2161865234375], "score": 0.9999987483024597}, {"category_id": 1, "poly": [865.1971435546875, 1684.040283203125, 1566.561279296875, 1684.040283203125, 1566.561279296875, 1813.7021484375, 865.1971435546875, 1813.7021484375], "score": 0.9999987483024597}, {"category_id": 9, "poly": [1530.5263671875, 1839.3990478515625, 1565.1201171875, 1839.3990478515625, 1565.1201171875, 1869.825439453125, 1530.5263671875, 1869.825439453125], "score": 0.9999977946281433}, {"category_id": 8, "poly": [972.3255004882812, 1075.85498046875, 1461.2088623046875, 1075.85498046875, 1461.2088623046875, 1155.465087890625, 972.3255004882812, 1155.465087890625], "score": 0.999996542930603}, {"category_id": 1, "poly": [865.4874267578125, 158.47100830078125, 1565.84375, 158.47100830078125, 1565.84375, 355.3230285644531, 865.4874267578125, 355.3230285644531], "score": 0.9999960660934448}, {"category_id": 1, "poly": [133.51382446289062, 158.21670532226562, 835.5382080078125, 158.21670532226562, 835.5382080078125, 558.8020629882812, 133.51382446289062, 558.8020629882812], "score": 0.9999951124191284}, {"category_id": 1, "poly": [134.01239013671875, 954.4151000976562, 836.1470336914062, 954.4151000976562, 836.1470336914062, 1618.77197265625, 134.01239013671875, 1618.77197265625], "score": 0.9999947547912598}, {"category_id": 1, "poly": [134.4542999267578, 558.8201904296875, 834.2548828125, 558.8201904296875, 834.2548828125, 954.7811279296875, 134.4542999267578, 954.7811279296875], "score": 0.9999943971633911}, {"category_id": 1, "poly": [866.33642578125, 421.84442138671875, 1566.451904296875, 421.84442138671875, 1566.451904296875, 787.1864624023438, 866.33642578125, 787.1864624023438], "score": 0.9999930262565613}, {"category_id": 1, "poly": [864.974853515625, 1167.92236328125, 1567.0927734375, 1167.92236328125, 1567.0927734375, 1298.29541015625, 864.974853515625, 1298.29541015625], "score": 0.9999929666519165}, {"category_id": 1, "poly": [864.5220947265625, 853.943359375, 1565.82080078125, 853.943359375, 1565.82080078125, 1080.8125, 864.5220947265625, 1080.8125], "score": 0.9999923706054688}, {"category_id": 1, "poly": [865.4466552734375, 1919.30615234375, 1566.4720458984375, 1919.30615234375, 1566.4720458984375, 2017.154541015625, 865.4466552734375, 2017.154541015625], "score": 0.9999904036521912}, {"category_id": 1, "poly": [864.801513671875, 1302.438232421875, 1566.760986328125, 1302.438232421875, 1566.760986328125, 1498.9681396484375, 864.801513671875, 1498.9681396484375], "score": 0.9999889135360718}, {"category_id": 1, "poly": [133.34628295898438, 1620.0596923828125, 836.7553100585938, 1620.0596923828125, 836.7553100585938, 2018.44873046875, 133.34628295898438, 2018.44873046875], "score": 0.9999861717224121}, {"category_id": 0, "poly": [865.5296020507812, 809.8997802734375, 1302.7711181640625, 809.8997802734375, 1302.7711181640625, 841.3140869140625, 865.5296020507812, 841.3140869140625], "score": 0.9999798536300659}, {"category_id": 0, "poly": [1131.11181640625, 378.66229248046875, 1299.6181640625, 378.66229248046875, 1299.6181640625, 409.04852294921875, 1131.11181640625, 409.04852294921875], "score": 0.9999651908874512}, {"category_id": 8, "poly": [1003.5569458007812, 1824.2362060546875, 1420.7132568359375, 1824.2362060546875, 1420.7132568359375, 1905.175048828125, 1003.5569458007812, 1905.175048828125], "score": 0.999914288520813}, {"category_id": 14, "poly": [974, 1076, 1454, 1076, 1454, 1155, 974, 1155], "score": 0.94, "latex": "w(p,q)=\\exp{\\left(-\\frac{\\Delta_{g}(p,q)}{\\gamma_{g}}-\\frac{\\Delta_{c}(p,q)}{\\gamma_{c}}\\right)},"}, {"category_id": 14, "poly": [1006, 1825, 1423, 1825, 1423, 1907, 1006, 1907], "score": 0.94, "latex": "\\delta(q,\\bar{q})=\\sum_{c=\\{r,g,b\\}}\\operatorname*{min}(|q_{c}-\\bar{q}_{c}|,\\tau)."}, {"category_id": 14, "poly": [963, 1510, 1464, 1510, 1464, 1671, 963, 1671], "score": 0.93, "latex": "C(p,\\bar{p})=\\frac{\\displaystyle\\sum_{q\\in\\Omega_{p},\\bar{q}\\in\\Omega_{\\bar{p}}}w(p,q)w(\\bar{p},\\bar{q})\\delta(q,\\bar{q})}{\\displaystyle\\sum_{q\\in\\Omega_{p},\\bar{q}\\in\\Omega_{\\bar{p}}}w(p,q)w(\\bar{p},\\bar{q})}\\,,"}, {"category_id": 13, "poly": [1335, 1166, 1432, 1166, 1432, 1200, 1335, 1200], "score": 0.93, "latex": "\\Delta_{c}(p,q)"}, {"category_id": 13, "poly": [939, 1166, 1039, 1166, 1039, 1201, 939, 1201], "score": 0.93, "latex": "\\Delta_{g}(p,q)"}, {"category_id": 13, "poly": [1289, 1683, 1365, 1683, 1365, 1717, 1289, 1717], "score": 0.93, "latex": "\\delta(q,\\bar{q})"}, {"category_id": 13, "poly": [1362, 1367, 1441, 1367, 1441, 1401, 1362, 1401], "score": 0.92, "latex": "\\bar{p}\\in S_{p}"}, {"category_id": 13, "poly": [864, 1019, 951, 1019, 951, 1053, 864, 1053], "score": 0.92, "latex": "q\\in\\Omega_{p}"}, {"category_id": 13, "poly": [1351, 953, 1388, 953, 1388, 987, 1351, 987], "score": 0.9, "latex": "\\Omega_{p}"}, {"category_id": 13, "poly": [913, 1467, 949, 1467, 949, 1501, 913, 1501], "score": 0.89, "latex": "\\Omega_{\\bar{p}}"}, {"category_id": 13, "poly": [1531, 1367, 1565, 1367, 1565, 1401, 1531, 1401], "score": 0.89, "latex": "S_{p}"}, {"category_id": 13, "poly": [1528, 1434, 1565, 1434, 1565, 1468, 1528, 1468], "score": 0.89, "latex": "\\Omega_{p}"}, {"category_id": 13, "poly": [1485, 1205, 1516, 1205, 1516, 1234, 1485, 1234], "score": 0.88, "latex": "\\gamma_{g}"}, {"category_id": 13, "poly": [1159, 1206, 1178, 1206, 1178, 1233, 1159, 1233], "score": 0.82, "latex": "p"}, {"category_id": 13, "poly": [863, 1238, 893, 1238, 893, 1266, 863, 1266], "score": 0.82, "latex": "\\gamma_{c}"}, {"category_id": 13, "poly": [1177, 1436, 1196, 1436, 1196, 1465, 1177, 1465], "score": 0.8, "latex": "\\bar{p}"}, {"category_id": 13, "poly": [1371, 1024, 1391, 1024, 1391, 1051, 1371, 1051], "score": 0.8, "latex": "p"}, {"category_id": 13, "poly": [1540, 1406, 1558, 1406, 1558, 1432, 1540, 1432], "score": 0.8, "latex": "p"}, {"category_id": 13, "poly": [1447, 1024, 1465, 1024, 1465, 1051, 1447, 1051], "score": 0.79, "latex": "q"}, {"category_id": 13, "poly": [1101, 1437, 1121, 1437, 1121, 1465, 1101, 1465], "score": 0.79, "latex": "p"}, {"category_id": 13, "poly": [1389, 1307, 1407, 1307, 1407, 1332, 1389, 1332], "score": 0.79, "latex": "p"}, {"category_id": 13, "poly": [1230, 1206, 1247, 1206, 1247, 1233, 1230, 1233], "score": 0.78, "latex": "q"}, {"category_id": 13, "poly": [1029, 1372, 1048, 1372, 1048, 1399, 1029, 1399], "score": 0.78, "latex": "p"}, {"category_id": 13, "poly": [916, 1752, 934, 1752, 934, 1782, 916, 1782], "score": 0.76, "latex": "\\bar{q}"}, {"category_id": 13, "poly": [1407, 1925, 1425, 1925, 1425, 1946, 1407, 1946], "score": 0.75, "latex": "\\tau"}, {"category_id": 13, "poly": [1548, 1722, 1565, 1722, 1565, 1749, 1548, 1749], "score": 0.75, "latex": "q"}, {"category_id": 13, "poly": [1050, 992, 1068, 992, 1068, 1018, 1050, 1018], "score": 0.75, "latex": "p"}, {"category_id": 15, "poly": [864.0, 1783.0, 1298.0, 1783.0, 1298.0, 1822.0, 864.0, 1822.0], "score": 0.99, "text": "green, and blue components given by"}, {"category_id": 15, "poly": [866.0, 1687.0, 1288.0, 1687.0, 1288.0, 1719.0, 866.0, 1719.0], "score": 0.96, "text": "where the pixel dissimilarity metric"}, {"category_id": 15, "poly": [1366.0, 1687.0, 1564.0, 1687.0, 1564.0, 1719.0, 1366.0, 1719.0], "score": 0.97, "text": "ischosen as the"}, {"category_id": 15, "poly": [866.0, 1751.0, 915.0, 1751.0, 915.0, 1783.0, 866.0, 1783.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [935.0, 1751.0, 1564.0, 1751.0, 1564.0, 1783.0, 935.0, 1783.0], "score": 0.98, "text": ". Here, the truncation of color difference for the red,"}, {"category_id": 15, "poly": [866.0, 1719.0, 1547.0, 1719.0, 1547.0, 1749.0, 866.0, 1749.0], "score": 0.99, "text": "sum of truncated absolute color differences between pixels"}, {"category_id": 15, "poly": [864.0, 163.0, 1568.0, 163.0, 1568.0, 192.0, 864.0, 192.0], "score": 1.0, "text": "temporal information, making it possible to process a temporal"}, {"category_id": 15, "poly": [859.0, 188.0, 1571.0, 193.0, 1570.0, 229.0, 859.0, 225.0], "score": 0.99, "text": " collection of cost volumes. The filtering operation was shown"}, {"category_id": 15, "poly": [864.0, 229.0, 1566.0, 229.0, 1566.0, 261.0, 864.0, 261.0], "score": 0.99, "text": "to preserve spatio-temporal edges present in the cost volumes,"}, {"category_id": 15, "poly": [859.0, 261.0, 1564.0, 264.0, 1564.0, 296.0, 859.0, 293.0], "score": 0.98, "text": " resulting in increased temporal consistency of disparity maps,"}, {"category_id": 15, "poly": [864.0, 296.0, 1566.0, 296.0, 1566.0, 328.0, 864.0, 328.0], "score": 0.99, "text": "greater robustness to image noise, and more accurate behavior"}, {"category_id": 15, "poly": [866.0, 328.0, 1160.0, 328.0, 1160.0, 360.0, 866.0, 360.0], "score": 1.0, "text": "around object boundaries."}, {"category_id": 15, "poly": [129.0, 158.0, 841.0, 153.0, 841.0, 192.0, 130.0, 197.0], "score": 0.99, "text": "aggregate cost values within the pixel neighborhoods defined"}, {"category_id": 15, "poly": [129.0, 188.0, 841.0, 190.0, 841.0, 229.0, 129.0, 227.0], "score": 0.99, "text": "by these windows. In 2005, Yoon and Kweon [4] proposed"}, {"category_id": 15, "poly": [132.0, 229.0, 838.0, 229.0, 838.0, 261.0, 132.0, 261.0], "score": 1.0, "text": "an adaptive matching cost aggregation scheme, which assigns"}, {"category_id": 15, "poly": [132.0, 261.0, 838.0, 261.0, 838.0, 293.0, 132.0, 293.0], "score": 0.98, "text": "a weight value to every pixel located in the support window"}, {"category_id": 15, "poly": [132.0, 293.0, 838.0, 293.0, 838.0, 325.0, 132.0, 325.0], "score": 0.98, "text": "of a given pixel of interest. The weight value is based on"}, {"category_id": 15, "poly": [132.0, 328.0, 836.0, 328.0, 836.0, 360.0, 132.0, 360.0], "score": 0.99, "text": "the spatial and color similarity between the pixel of interest"}, {"category_id": 15, "poly": [134.0, 360.0, 836.0, 360.0, 836.0, 392.0, 134.0, 392.0], "score": 1.0, "text": "and a pixel in its support window, and the aggregated cost is"}, {"category_id": 15, "poly": [134.0, 394.0, 836.0, 394.0, 836.0, 426.0, 134.0, 426.0], "score": 0.99, "text": "computed as a weighted average of the pixel-wise costs within"}, {"category_id": 15, "poly": [127.0, 422.0, 839.0, 424.0, 838.0, 463.0, 127.0, 461.0], "score": 0.98, "text": " the considered support window. The edge-preserving nature"}, {"category_id": 15, "poly": [129.0, 456.0, 838.0, 454.0, 838.0, 493.0, 129.0, 495.0], "score": 0.99, "text": " and matching accuracy of adaptive support weights have made"}, {"category_id": 15, "poly": [132.0, 490.0, 841.0, 490.0, 841.0, 529.0, 132.0, 529.0], "score": 0.99, "text": "them one of the most popular choices for cost aggregation in"}, {"category_id": 15, "poly": [132.0, 527.0, 797.0, 527.0, 797.0, 559.0, 132.0, 559.0], "score": 0.97, "text": "recently proposed stereo matching algorithms [3], [5]-[8]."}, {"category_id": 15, "poly": [157.0, 958.0, 836.0, 958.0, 836.0, 988.0, 157.0, 988.0], "score": 0.99, "text": "It has been demonstrated that the performance of stereo"}, {"category_id": 15, "poly": [132.0, 990.0, 838.0, 990.0, 838.0, 1022.0, 132.0, 1022.0], "score": 0.99, "text": "algorithms designed to match a single pair of images can"}, {"category_id": 15, "poly": [132.0, 1024.0, 836.0, 1024.0, 836.0, 1056.0, 132.0, 1056.0], "score": 0.99, "text": "be adapted to take advantage of the temporal dependencies"}, {"category_id": 15, "poly": [129.0, 1054.0, 838.0, 1054.0, 838.0, 1093.0, 129.0, 1093.0], "score": 0.97, "text": "available in stereo video sequences. Early proposed solutions"}, {"category_id": 15, "poly": [132.0, 1091.0, 836.0, 1091.0, 836.0, 1123.0, 132.0, 1123.0], "score": 0.99, "text": "to temporal stereo matching attempted to average matching"}, {"category_id": 15, "poly": [134.0, 1123.0, 836.0, 1123.0, 836.0, 1155.0, 134.0, 1155.0], "score": 0.99, "text": "costs across subsequent frames of a video sequence [13],"}, {"category_id": 15, "poly": [129.0, 1153.0, 841.0, 1150.0, 841.0, 1189.0, 129.0, 1192.0], "score": 0.98, "text": "[14]. Attempts have been made to integrate estimation of"}, {"category_id": 15, "poly": [134.0, 1192.0, 838.0, 1192.0, 838.0, 1224.0, 134.0, 1224.0], "score": 0.99, "text": "motion fields (optical flow) into temporal stereo matching. The"}, {"category_id": 15, "poly": [132.0, 1224.0, 838.0, 1224.0, 838.0, 1256.0, 132.0, 1256.0], "score": 0.99, "text": "methods of [15] and [16] perform smoothing of disparities"}, {"category_id": 15, "poly": [129.0, 1254.0, 841.0, 1254.0, 841.0, 1292.0, 129.0, 1292.0], "score": 0.99, "text": " along motion vectors recovered from the video sequence. The"}, {"category_id": 15, "poly": [132.0, 1290.0, 838.0, 1290.0, 838.0, 1322.0, 132.0, 1322.0], "score": 0.99, "text": "estimation of the motion field, however, prevents real-time"}, {"category_id": 15, "poly": [132.0, 1325.0, 838.0, 1325.0, 838.0, 1354.0, 132.0, 1354.0], "score": 0.99, "text": "implementation, since state-of-the-art optical flow algorithms"}, {"category_id": 15, "poly": [129.0, 1354.0, 841.0, 1354.0, 841.0, 1393.0, 129.0, 1393.0], "score": 0.99, "text": " do not, in general, approach real-time frame rates. In a related"}, {"category_id": 15, "poly": [129.0, 1386.0, 841.0, 1384.0, 841.0, 1423.0, 129.0, 1425.0], "score": 0.99, "text": "approach, Sizintsev and Wildes [17], [18] used steerable"}, {"category_id": 15, "poly": [134.0, 1423.0, 836.0, 1423.0, 836.0, 1455.0, 134.0, 1455.0], "score": 0.99, "text": "filters to obtain descriptors characterizing motion of image"}, {"category_id": 15, "poly": [134.0, 1455.0, 836.0, 1455.0, 836.0, 1487.0, 134.0, 1487.0], "score": 0.99, "text": "features in both space and time. Unlike traditional algorithms,"}, {"category_id": 15, "poly": [132.0, 1490.0, 838.0, 1490.0, 838.0, 1522.0, 132.0, 1522.0], "score": 0.98, "text": "their method performs matching on spatio-temporal motion"}, {"category_id": 15, "poly": [129.0, 1519.0, 841.0, 1517.0, 841.0, 1556.0, 129.0, 1558.0], "score": 0.99, "text": " descriptors, rather than on pure pixel intensity values, which"}, {"category_id": 15, "poly": [132.0, 1554.0, 841.0, 1554.0, 841.0, 1593.0, 132.0, 1593.0], "score": 0.99, "text": "leads to improved temporal coherence of disparity maps at the"}, {"category_id": 15, "poly": [132.0, 1586.0, 698.0, 1586.0, 698.0, 1618.0, 132.0, 1618.0], "score": 0.99, "text": "cost of reduced accuracy at depth discontinuities."}, {"category_id": 15, "poly": [159.0, 559.0, 838.0, 559.0, 838.0, 591.0, 159.0, 591.0], "score": 0.99, "text": "Recently, Rheman et al. [9], [10] have revisited the cost"}, {"category_id": 15, "poly": [132.0, 594.0, 838.0, 589.0, 839.0, 621.0, 132.0, 626.0], "score": 1.0, "text": "aggregation step of stereo algorithms, and demonstrated that"}, {"category_id": 15, "poly": [132.0, 626.0, 838.0, 626.0, 838.0, 658.0, 132.0, 658.0], "score": 0.99, "text": "cost aggregation can be performed by filtering of subsequent"}, {"category_id": 15, "poly": [134.0, 660.0, 834.0, 660.0, 834.0, 692.0, 134.0, 692.0], "score": 1.0, "text": "layers of the initially computed matching cost volume. In par-"}, {"category_id": 15, "poly": [132.0, 692.0, 836.0, 692.0, 836.0, 724.0, 132.0, 724.0], "score": 0.99, "text": "ticular, the edge-aware image filters, such as the bilateral filter"}, {"category_id": 15, "poly": [127.0, 719.0, 839.0, 724.0, 838.0, 761.0, 127.0, 756.0], "score": 0.99, "text": " of Tomasi and Manducci [11] or the guided filter of He [12],"}, {"category_id": 15, "poly": [132.0, 759.0, 838.0, 759.0, 838.0, 791.0, 132.0, 791.0], "score": 0.98, "text": "have been rendered useful for the problem of matching cost"}, {"category_id": 15, "poly": [132.0, 793.0, 838.0, 791.0, 838.0, 823.0, 132.0, 825.0], "score": 0.99, "text": "aggregation, enabling stereo algorithms to correctly recover"}, {"category_id": 15, "poly": [134.0, 825.0, 838.0, 825.0, 838.0, 857.0, 134.0, 857.0], "score": 0.98, "text": "disparities along object boundaries. In fact, Yoon and Kweon's"}, {"category_id": 15, "poly": [134.0, 859.0, 838.0, 859.0, 838.0, 891.0, 134.0, 891.0], "score": 1.0, "text": "adaptive support-weight cost aggregation scheme is equivalent"}, {"category_id": 15, "poly": [132.0, 891.0, 838.0, 891.0, 838.0, 924.0, 132.0, 924.0], "score": 0.98, "text": "to the application of the so-called joint bilateral filter to the"}, {"category_id": 15, "poly": [134.0, 924.0, 547.0, 924.0, 547.0, 956.0, 134.0, 956.0], "score": 1.0, "text": "layers of the matching cost volume."}, {"category_id": 15, "poly": [889.0, 422.0, 1568.0, 424.0, 1568.0, 456.0, 889.0, 454.0], "score": 0.98, "text": "The proposed temporal stereo matching algorithm is an"}, {"category_id": 15, "poly": [862.0, 456.0, 1571.0, 456.0, 1571.0, 495.0, 862.0, 495.0], "score": 1.0, "text": "extension of the real-time iterative adaptive support-weight"}, {"category_id": 15, "poly": [864.0, 490.0, 1568.0, 490.0, 1568.0, 522.0, 864.0, 522.0], "score": 0.99, "text": "algorithm described in [3]. In addition to real-time two-"}, {"category_id": 15, "poly": [864.0, 525.0, 1566.0, 525.0, 1566.0, 557.0, 864.0, 557.0], "score": 1.0, "text": "pass aggregation of the cost values in the spatial domain,"}, {"category_id": 15, "poly": [864.0, 557.0, 1568.0, 557.0, 1568.0, 589.0, 864.0, 589.0], "score": 0.99, "text": "the proposed algorithm enhances stereo matching on video"}, {"category_id": 15, "poly": [866.0, 594.0, 1566.0, 594.0, 1566.0, 626.0, 866.0, 626.0], "score": 0.97, "text": "sequences by aggregating costs along the time dimension."}, {"category_id": 15, "poly": [864.0, 626.0, 1568.0, 626.0, 1568.0, 658.0, 864.0, 658.0], "score": 1.0, "text": "The operation of the algorithm has been divided into four"}, {"category_id": 15, "poly": [866.0, 660.0, 1568.0, 660.0, 1568.0, 692.0, 866.0, 692.0], "score": 0.99, "text": "stages: 1) two-pass spatial cost aggregation, 2) temporal cost"}, {"category_id": 15, "poly": [862.0, 688.0, 1568.0, 685.0, 1568.0, 724.0, 862.0, 727.0], "score": 1.0, "text": "aggregation, 3) disparity selection and confidence assessment,"}, {"category_id": 15, "poly": [866.0, 724.0, 1568.0, 724.0, 1568.0, 756.0, 866.0, 756.0], "score": 1.0, "text": "and 4) iterative disparity refinement. In the following, each of"}, {"category_id": 15, "poly": [864.0, 759.0, 1254.0, 759.0, 1254.0, 791.0, 864.0, 791.0], "score": 1.0, "text": "these stages is described in detail."}, {"category_id": 15, "poly": [860.0, 1265.0, 1194.0, 1270.0, 1194.0, 1306.0, 859.0, 1301.0], "score": 0.99, "text": " color similarity, respectively."}, {"category_id": 15, "poly": [1433.0, 1169.0, 1566.0, 1169.0, 1566.0, 1201.0, 1433.0, 1201.0], "score": 0.98, "text": "is the color"}, {"category_id": 15, "poly": [864.0, 1169.0, 938.0, 1169.0, 938.0, 1201.0, 864.0, 1201.0], "score": 1.0, "text": "where"}, {"category_id": 15, "poly": [1040.0, 1169.0, 1334.0, 1169.0, 1334.0, 1201.0, 1040.0, 1201.0], "score": 0.98, "text": "is the geometric distance,"}, {"category_id": 15, "poly": [1517.0, 1196.0, 1566.0, 1201.0, 1566.0, 1240.0, 1517.0, 1235.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [862.0, 1196.0, 1158.0, 1201.0, 1158.0, 1240.0, 861.0, 1235.0], "score": 1.0, "text": "difference between pixels"}, {"category_id": 15, "poly": [894.0, 1233.0, 1566.0, 1231.0, 1566.0, 1270.0, 894.0, 1272.0], "score": 0.97, "text": "regulate the strength of grouping by geometric distance and"}, {"category_id": 15, "poly": [1179.0, 1196.0, 1229.0, 1201.0, 1229.0, 1240.0, 1179.0, 1235.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [1248.0, 1196.0, 1484.0, 1201.0, 1484.0, 1240.0, 1248.0, 1235.0], "score": 0.99, "text": ", and the coefficients"}, {"category_id": 15, "poly": [887.0, 848.0, 1568.0, 850.0, 1568.0, 889.0, 887.0, 887.0], "score": 0.99, "text": " Humans group shapes by observing the geometric distance"}, {"category_id": 15, "poly": [859.0, 885.0, 1568.0, 882.0, 1568.0, 921.0, 859.0, 924.0], "score": 0.98, "text": " and color similarity of points in space. To mimic this vi-"}, {"category_id": 15, "poly": [864.0, 921.0, 1568.0, 921.0, 1568.0, 953.0, 864.0, 953.0], "score": 0.99, "text": "sual grouping, the adaptive support-weight stereo matching"}, {"category_id": 15, "poly": [864.0, 1054.0, 899.0, 1054.0, 899.0, 1084.0, 864.0, 1084.0], "score": 1.0, "text": "by"}, {"category_id": 15, "poly": [866.0, 956.0, 1350.0, 956.0, 1350.0, 988.0, 866.0, 988.0], "score": 0.98, "text": "algorithm [4] considers a support window"}, {"category_id": 15, "poly": [1389.0, 956.0, 1566.0, 956.0, 1566.0, 988.0, 1389.0, 988.0], "score": 0.98, "text": " centered at the"}, {"category_id": 15, "poly": [952.0, 1022.0, 1370.0, 1022.0, 1370.0, 1054.0, 952.0, 1054.0], "score": 0.98, "text": ". The support weight relating pixels"}, {"category_id": 15, "poly": [1392.0, 1022.0, 1446.0, 1022.0, 1446.0, 1054.0, 1392.0, 1054.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [1466.0, 1022.0, 1566.0, 1022.0, 1566.0, 1054.0, 1466.0, 1054.0], "score": 0.98, "text": "is given"}, {"category_id": 15, "poly": [866.0, 990.0, 1049.0, 990.0, 1049.0, 1022.0, 866.0, 1022.0], "score": 1.0, "text": "pixel of interest"}, {"category_id": 15, "poly": [1069.0, 990.0, 1566.0, 990.0, 1566.0, 1022.0, 1069.0, 1022.0], "score": 1.0, "text": ", and assigns a support weight to each pixel"}, {"category_id": 15, "poly": [862.0, 1948.0, 1568.0, 1950.0, 1568.0, 1989.0, 861.0, 1987.0], "score": 0.98, "text": "vides additional robustness to outliers. Rather than evaluating"}, {"category_id": 15, "poly": [864.0, 1989.0, 1566.0, 1989.0, 1566.0, 2021.0, 864.0, 2021.0], "score": 0.98, "text": "Equation (2) directly, real-time algorithms often approximate"}, {"category_id": 15, "poly": [862.0, 1920.0, 1406.0, 1920.0, 1406.0, 1952.0, 862.0, 1952.0], "score": 0.99, "text": "This limits each of their magnitudes to at most"}, {"category_id": 15, "poly": [1426.0, 1920.0, 1561.0, 1920.0, 1561.0, 1952.0, 1426.0, 1952.0], "score": 0.96, "text": ",whichpro-"}, {"category_id": 15, "poly": [859.0, 1331.0, 1571.0, 1334.0, 1571.0, 1373.0, 859.0, 1370.0], "score": 0.98, "text": " iterative adaptive support-weight algorithm evaluates matching"}, {"category_id": 15, "poly": [859.0, 1464.0, 912.0, 1467.0, 912.0, 1506.0, 859.0, 1503.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [950.0, 1464.0, 1474.0, 1467.0, 1474.0, 1506.0, 950.0, 1503.0], "score": 1.0, "text": ", the initial matching cost is aggregated using"}, {"category_id": 15, "poly": [1442.0, 1370.0, 1530.0, 1370.0, 1530.0, 1402.0, 1442.0, 1402.0], "score": 0.98, "text": ", where"}, {"category_id": 15, "poly": [1197.0, 1437.0, 1527.0, 1437.0, 1527.0, 1469.0, 1197.0, 1469.0], "score": 0.97, "text": ", and their support windows"}, {"category_id": 15, "poly": [866.0, 1402.0, 1539.0, 1402.0, 1539.0, 1435.0, 866.0, 1435.0], "score": 1.0, "text": "denotes a set of matching candidates associated with pixel"}, {"category_id": 15, "poly": [864.0, 1437.0, 1100.0, 1437.0, 1100.0, 1469.0, 864.0, 1469.0], "score": 0.97, "text": "For a pair of pixels"}, {"category_id": 15, "poly": [1122.0, 1437.0, 1176.0, 1437.0, 1176.0, 1469.0, 1122.0, 1469.0], "score": 0.94, "text": " and"}, {"category_id": 15, "poly": [887.0, 1299.0, 1388.0, 1304.0, 1388.0, 1336.0, 887.0, 1331.0], "score": 0.96, "text": " To identify a match for the pixel of interest"}, {"category_id": 15, "poly": [1408.0, 1299.0, 1568.0, 1304.0, 1568.0, 1336.0, 1408.0, 1331.0], "score": 1.0, "text": ", the real-time"}, {"category_id": 15, "poly": [864.0, 1370.0, 1028.0, 1370.0, 1028.0, 1402.0, 864.0, 1402.0], "score": 1.0, "text": "costs between"}, {"category_id": 15, "poly": [1049.0, 1370.0, 1361.0, 1370.0, 1361.0, 1402.0, 1049.0, 1402.0], "score": 0.99, "text": " and every match candidate"}, {"category_id": 15, "poly": [160.0, 1618.0, 836.0, 1623.0, 836.0, 1655.0, 159.0, 1650.0], "score": 0.99, "text": "Most recently, local stereo algorithms based on edge-aware"}, {"category_id": 15, "poly": [127.0, 1650.0, 841.0, 1652.0, 841.0, 1691.0, 127.0, 1689.0], "score": 0.97, "text": " filters were extended to incorporate temporal evidence into"}, {"category_id": 15, "poly": [132.0, 1687.0, 836.0, 1687.0, 836.0, 1719.0, 132.0, 1719.0], "score": 0.97, "text": "the matching process. The method of Richardt et al. [19]"}, {"category_id": 15, "poly": [134.0, 1723.0, 838.0, 1723.0, 838.0, 1753.0, 134.0, 1753.0], "score": 0.99, "text": "employs a variant of the bilateral grid [20] implemented on"}, {"category_id": 15, "poly": [134.0, 1755.0, 838.0, 1755.0, 838.0, 1788.0, 134.0, 1788.0], "score": 0.99, "text": "graphics hardware, which accelerates cost aggregation and"}, {"category_id": 15, "poly": [134.0, 1788.0, 838.0, 1788.0, 838.0, 1820.0, 134.0, 1820.0], "score": 1.0, "text": "allows for weighted propagation of pixel dissimilarity metrics"}, {"category_id": 15, "poly": [132.0, 1822.0, 838.0, 1822.0, 838.0, 1854.0, 132.0, 1854.0], "score": 0.99, "text": "from previous frames to the current one. Although this method"}, {"category_id": 15, "poly": [129.0, 1856.0, 838.0, 1856.0, 838.0, 1888.0, 129.0, 1888.0], "score": 1.0, "text": " outperforms the baseline frame-to-frame approach, the amount"}, {"category_id": 15, "poly": [132.0, 1888.0, 838.0, 1888.0, 838.0, 1920.0, 132.0, 1920.0], "score": 0.97, "text": "of hardware memory necessary to construct the bilateral grid"}, {"category_id": 15, "poly": [127.0, 1916.0, 841.0, 1918.0, 841.0, 1957.0, 127.0, 1955.0], "score": 0.99, "text": "limits its application to single-channel, i.e., grayscale images "}, {"category_id": 15, "poly": [132.0, 1955.0, 838.0, 1955.0, 838.0, 1985.0, 132.0, 1985.0], "score": 0.99, "text": "only. Hosni et al. [10], on the other hand, reformulated kernels"}, {"category_id": 15, "poly": [132.0, 1989.0, 838.0, 1989.0, 838.0, 2021.0, 132.0, 2021.0], "score": 0.99, "text": "of the guided image filter to operate on both spatial and"}, {"category_id": 15, "poly": [859.0, 809.0, 1307.0, 809.0, 1307.0, 848.0, 859.0, 848.0], "score": 0.99, "text": "A. Two-Pass Spatial Cost Aggregation"}, {"category_id": 15, "poly": [1129.0, 376.0, 1300.0, 376.0, 1300.0, 417.0, 1129.0, 417.0], "score": 0.94, "text": "III. METHOD"}], "page_info": {"page_no": 1, "height": 2200, "width": 1700}}, {"layout_dets": [{"category_id": 1, "poly": [865.5088500976562, 856.5537109375, 1567.692626953125, 856.5537109375, 1567.692626953125, 1420.9698486328125, 865.5088500976562, 1420.9698486328125], "score": 0.9999963045120239}, {"category_id": 8, "poly": [281.1294860839844, 1001.0513916015625, 689.37451171875, 1001.0513916015625, 689.37451171875, 1075.8765869140625, 281.1294860839844, 1075.8765869140625], "score": 0.9999961256980896}, {"category_id": 1, "poly": [133.53353881835938, 158.6427459716797, 836.7297973632812, 158.6427459716797, 836.7297973632812, 390.48828125, 133.53353881835938, 390.48828125], "score": 0.9999960660934448}, {"category_id": 8, "poly": [145.77777099609375, 1839.6416015625, 803.4192504882812, 1839.6416015625, 803.4192504882812, 1993.239013671875, 145.77777099609375, 1993.239013671875], "score": 0.9999958872795105}, {"category_id": 1, "poly": [864.9884643554688, 1420.8831787109375, 1567.3118896484375, 1420.8831787109375, 1567.3118896484375, 2023.257080078125, 864.9884643554688, 2023.257080078125], "score": 0.9999951124191284}, {"category_id": 9, "poly": [1529.267333984375, 388.6717834472656, 1565.1744384765625, 388.6717834472656, 1565.1744384765625, 416.4899597167969, 1529.267333984375, 416.4899597167969], "score": 0.9999918937683105}, {"category_id": 9, "poly": [800.3933715820312, 1551.524169921875, 833.2618408203125, 1551.524169921875, 833.2618408203125, 1582.073486328125, 800.3933715820312, 1582.073486328125], "score": 0.9999911189079285}, {"category_id": 1, "poly": [864.3720092773438, 200.97483825683594, 1565.6871337890625, 200.97483825683594, 1565.6871337890625, 365.6230163574219, 864.3720092773438, 365.6230163574219], "score": 0.9999903440475464}, {"category_id": 1, "poly": [134.87628173828125, 1369.5762939453125, 835.0336303710938, 1369.5762939453125, 835.0336303710938, 1533.884765625, 134.87628173828125, 1533.884765625], "score": 0.9999880790710449}, {"category_id": 1, "poly": [134.59988403320312, 444.5299377441406, 836.5606079101562, 444.5299377441406, 836.5606079101562, 709.0791015625, 134.59988403320312, 709.0791015625], "score": 0.999987006187439}, {"category_id": 1, "poly": [134.15472412109375, 1084.4288330078125, 836.2360229492188, 1084.4288330078125, 836.2360229492188, 1314.6600341796875, 134.15472412109375, 1314.6600341796875], "score": 0.9999866485595703}, {"category_id": 9, "poly": [800.6007690429688, 1023.1047973632812, 833.2154541015625, 1023.1047973632812, 833.2154541015625, 1055.7227783203125, 800.6007690429688, 1055.7227783203125], "score": 0.9999839663505554}, {"category_id": 8, "poly": [948.4016723632812, 372.03607177734375, 1486.11279296875, 372.03607177734375, 1486.11279296875, 449.3696594238281, 948.4016723632812, 449.3696594238281], "score": 0.9999831914901733}, {"category_id": 8, "poly": [145.31065368652344, 714.4036254882812, 820.3599853515625, 714.4036254882812, 820.3599853515625, 791.855712890625, 145.31065368652344, 791.855712890625], "score": 0.9999772906303406}, {"category_id": 1, "poly": [863.8760986328125, 599.6033325195312, 1566.84619140625, 599.6033325195312, 1566.84619140625, 797.44189453125, 863.8760986328125, 797.44189453125], "score": 0.999976396560669}, {"category_id": 1, "poly": [864.925537109375, 464.9669189453125, 1565.212158203125, 464.9669189453125, 1565.212158203125, 529.045654296875, 864.925537109375, 529.045654296875], "score": 0.999973475933075}, {"category_id": 1, "poly": [133.88735961914062, 797.7457885742188, 835.5986328125, 797.7457885742188, 835.5986328125, 994.4456176757812, 133.88735961914062, 994.4456176757812], "score": 0.9999661445617676}, {"category_id": 1, "poly": [134.8787841796875, 1615.116455078125, 835.4554443359375, 1615.116455078125, 835.4554443359375, 1815.4564208984375, 134.8787841796875, 1815.4564208984375], "score": 0.9999580383300781}, {"category_id": 9, "poly": [1530.1783447265625, 550.1576538085938, 1564.607177734375, 550.1576538085938, 1564.607177734375, 578.6950073242188, 1530.1783447265625, 578.6950073242188], "score": 0.9999532103538513}, {"category_id": 9, "poly": [801.0740966796875, 738.4259643554688, 834.7449340820312, 738.4259643554688, 834.7449340820312, 770.4969482421875, 801.0740966796875, 770.4969482421875], "score": 0.9996598958969116}, {"category_id": 0, "poly": [1134.302490234375, 815.6021728515625, 1295.3885498046875, 815.6021728515625, 1295.3885498046875, 844.6544799804688, 1134.302490234375, 844.6544799804688], "score": 0.9994980096817017}, {"category_id": 9, "poly": [798.6090698242188, 1986.7332763671875, 834.5460205078125, 1986.7332763671875, 834.5460205078125, 2017.6595458984375, 798.6090698242188, 2017.6595458984375], "score": 0.9992558360099792}, {"category_id": 0, "poly": [135.0093994140625, 406.12335205078125, 475.6328125, 406.12335205078125, 475.6328125, 437.4545593261719, 135.0093994140625, 437.4545593261719], "score": 0.9990860819816589}, {"category_id": 8, "poly": [1029.3924560546875, 541.857177734375, 1400.174072265625, 541.857177734375, 1400.174072265625, 585.1640625, 1029.3924560546875, 585.1640625], "score": 0.9979717135429382}, {"category_id": 0, "poly": [133.26077270507812, 1330.139892578125, 713.5426635742188, 1330.139892578125, 713.5426635742188, 1363.1341552734375, 133.26077270507812, 1363.1341552734375], "score": 0.9967154860496521}, {"category_id": 8, "poly": [338.6681823730469, 1547.7218017578125, 626.6519775390625, 1547.7218017578125, 626.6519775390625, 1604.587646484375, 338.6681823730469, 1604.587646484375], "score": 0.9945433139801025}, {"category_id": 1, "poly": [864.5469970703125, 160.16702270507812, 1251.313720703125, 160.16702270507812, 1251.313720703125, 190.15760803222656, 864.5469970703125, 190.15760803222656], "score": 0.9902143478393555}, {"category_id": 13, "poly": [550, 577, 648, 577, 648, 612, 550, 612], "score": 0.95, "latex": "C_{a}(p,\\bar{p})"}, {"category_id": 13, "poly": [183, 1780, 304, 1780, 304, 1813, 183, 1813], "score": 0.95, "latex": "p^{\\prime}=m(\\bar{p})"}, {"category_id": 14, "poly": [279, 1000, 687, 1000, 687, 1078, 279, 1078], "score": 0.95, "latex": "w_{t}(p,p_{t-1})=\\exp\\bigg({-\\frac{\\Delta_{c}(p,p_{t-1})}{\\gamma_{t}}}\\bigg),"}, {"category_id": 14, "poly": [147, 1843, 820, 1843, 820, 1992, 147, 1992], "score": 0.94, "latex": "F_{p}=\\left\\{\\begin{array}{l l}{\\underset{\\bar{p}\\in S_{p}\\setminus m(p)}{\\mathrm{min}}\\,C(p,\\bar{p})-\\underset{\\bar{p}\\in S_{p}}{\\mathrm{min}}\\,C(p,\\bar{p})}\\\\ {\\underset{\\bar{p}\\in S_{p}\\setminus m(p)}{\\mathrm{min}}\\,C(p,\\bar{p})}&{|d_{p}-d_{p^{\\prime}}|\\leq1}\\\\ {0,}&{\\mathrm{otherwise}}\\end{array}\\right.."}, {"category_id": 14, "poly": [340, 1546, 628, 1546, 628, 1608, 340, 1608], "score": 0.93, "latex": "m(p)=\\underset{\\bar{p}\\in S_{p}}{\\mathrm{argmin}}\\,C(p,\\bar{p})\\,."}, {"category_id": 13, "poly": [321, 830, 443, 830, 443, 864, 321, 864], "score": 0.93, "latex": "w_{t}(p,p_{t-1})"}, {"category_id": 13, "poly": [581, 1713, 694, 1713, 694, 1747, 581, 1747], "score": 0.93, "latex": "{\\bar{p}}=m(p)"}, {"category_id": 14, "poly": [947, 373, 1478, 373, 1478, 454, 947, 454], "score": 0.93, "latex": "\\Lambda^{i}(p,\\bar{p})=\\alpha\\times\\sum_{q\\in\\Omega_{p}}w(p,q)F_{q}^{i-1}\\left|D_{q}^{i-1}-d_{p}\\right|\\,,"}, {"category_id": 13, "poly": [426, 445, 512, 445, 512, 479, 426, 479], "score": 0.93, "latex": "C(p,{\\bar{p}})"}, {"category_id": 13, "poly": [337, 356, 414, 356, 414, 391, 337, 391], "score": 0.93, "latex": "\\mathcal{O}(\\omega^{2})"}, {"category_id": 13, "poly": [1341, 730, 1565, 730, 1565, 765, 1341, 765], "score": 0.92, "latex": "C_{a}(p,\\bar{p})\\gets C(p,\\bar{p})"}, {"category_id": 13, "poly": [629, 1436, 691, 1436, 691, 1470, 629, 1470], "score": 0.92, "latex": "m(p)"}, {"category_id": 13, "poly": [277, 1469, 361, 1469, 361, 1504, 277, 1504], "score": 0.92, "latex": "\\bar{p}\\in S_{p}"}, {"category_id": 14, "poly": [1030, 541, 1398, 541, 1398, 582, 1030, 582], "score": 0.92, "latex": "C^{i}(p,\\bar{p})=C^{0}(p,\\bar{p})+{\\Lambda^{i}}(p,\\bar{p})\\,,"}, {"category_id": 13, "poly": [453, 356, 518, 356, 518, 391, 453, 391], "score": 0.91, "latex": "\\mathcal{O}(\\omega)"}, {"category_id": 14, "poly": [146, 714, 787, 714, 787, 791, 146, 791], "score": 0.91, "latex": "C(p,\\bar{p})\\gets\\frac{(1-\\lambda)\\cdot C(p,\\bar{p})+\\lambda\\cdot w_{t}(p,p_{t-1})\\cdot C_{a}(p,\\bar{p})}{(1-\\lambda)+\\lambda\\cdot w_{t}(p,p_{t-1})},"}, {"category_id": 13, "poly": [1095, 231, 1134, 231, 1134, 270, 1095, 270], "score": 0.9, "latex": "D_{p}^{i}"}, {"category_id": 13, "poly": [1313, 1752, 1447, 1752, 1447, 1783, 1313, 1783], "score": 0.89, "latex": "640~\\times~480"}, {"category_id": 13, "poly": [593, 1782, 627, 1782, 627, 1815, 593, 1815], "score": 0.89, "latex": "F_{p}"}, {"category_id": 13, "poly": [133, 326, 209, 326, 209, 355, 133, 355], "score": 0.88, "latex": "\\omega\\times\\omega"}, {"category_id": 13, "poly": [208, 1089, 236, 1089, 236, 1116, 208, 1116], "score": 0.85, "latex": "\\gamma_{t}"}, {"category_id": 13, "poly": [1466, 769, 1484, 769, 1484, 797, 1466, 797], "score": 0.83, "latex": "\\bar{p}"}, {"category_id": 13, "poly": [133, 935, 177, 935, 177, 963, 133, 963], "score": 0.83, "latex": "p_{t-1}"}, {"category_id": 13, "poly": [608, 1753, 627, 1753, 627, 1779, 608, 1779], "score": 0.81, "latex": "p"}, {"category_id": 13, "poly": [491, 799, 511, 799, 511, 825, 491, 825], "score": 0.81, "latex": "\\lambda"}, {"category_id": 13, "poly": [1018, 770, 1037, 770, 1037, 796, 1018, 796], "score": 0.81, "latex": "p"}, {"category_id": 13, "poly": [1086, 470, 1107, 470, 1107, 491, 1086, 491], "score": 0.8, "latex": "\\alpha"}, {"category_id": 13, "poly": [466, 901, 485, 901, 485, 929, 466, 929], "score": 0.8, "latex": "p"}, {"category_id": 13, "poly": [208, 484, 227, 484, 227, 511, 208, 511], "score": 0.79, "latex": "p"}, {"category_id": 13, "poly": [462, 1443, 480, 1443, 480, 1468, 462, 1468], "score": 0.77, "latex": "p"}, {"category_id": 13, "poly": [266, 514, 288, 514, 288, 544, 266, 544], "score": 0.77, "latex": "\\bar{p}"}, {"category_id": 13, "poly": [816, 1716, 836, 1716, 836, 1746, 816, 1746], "score": 0.73, "latex": "\\bar{p}"}, {"category_id": 13, "poly": [132, 405, 154, 405, 154, 432, 132, 432], "score": 0.27, "latex": "B"}, {"category_id": 13, "poly": [862, 160, 887, 160, 887, 187, 862, 187], "score": 0.26, "latex": "D"}, {"category_id": 15, "poly": [887.0, 852.0, 1568.0, 855.0, 1568.0, 894.0, 887.0, 891.0], "score": 0.98, "text": " The speed and accuracy of real-time stereo matching al-"}, {"category_id": 15, "poly": [864.0, 891.0, 1566.0, 891.0, 1566.0, 924.0, 864.0, 924.0], "score": 0.99, "text": "gorithms are traditionally demonstrated using still-frame im-"}, {"category_id": 15, "poly": [859.0, 921.0, 1571.0, 919.0, 1571.0, 958.0, 859.0, 960.0], "score": 0.97, "text": " ages from the Middlebury stereo benchmark [1], [2]. Still"}, {"category_id": 15, "poly": [862.0, 956.0, 1568.0, 958.0, 1568.0, 990.0, 862.0, 988.0], "score": 0.99, "text": "frames, however, are insufficient for evaluating stereo match-"}, {"category_id": 15, "poly": [864.0, 992.0, 1571.0, 992.0, 1571.0, 1024.0, 864.0, 1024.0], "score": 1.0, "text": "ing algorithms that incorporate frame-to-frame prediction to"}, {"category_id": 15, "poly": [864.0, 1027.0, 1568.0, 1027.0, 1568.0, 1059.0, 864.0, 1059.0], "score": 0.97, "text": "enhance matching accuracy. An alternative approach is to"}, {"category_id": 15, "poly": [864.0, 1059.0, 1566.0, 1059.0, 1566.0, 1089.0, 864.0, 1089.0], "score": 0.99, "text": "use a stereo video sequence with a ground truth disparity"}, {"category_id": 15, "poly": [862.0, 1091.0, 1566.0, 1091.0, 1566.0, 1123.0, 862.0, 1123.0], "score": 1.0, "text": "for each frame. Obtaining the ground truth disparity of real"}, {"category_id": 15, "poly": [866.0, 1125.0, 1566.0, 1125.0, 1566.0, 1157.0, 866.0, 1157.0], "score": 0.98, "text": "world video sequences is a difficult undertaking due to the"}, {"category_id": 15, "poly": [859.0, 1153.0, 1568.0, 1155.0, 1568.0, 1194.0, 859.0, 1192.0], "score": 0.99, "text": "high frame rate of video and limitations in depth sensing-"}, {"category_id": 15, "poly": [864.0, 1192.0, 1568.0, 1192.0, 1568.0, 1224.0, 864.0, 1224.0], "score": 0.99, "text": "technology. To address the need for stereo video with ground"}, {"category_id": 15, "poly": [864.0, 1224.0, 1568.0, 1224.0, 1568.0, 1256.0, 864.0, 1256.0], "score": 0.99, "text": "truth disparities, five pairs of synthetic stereo video sequences"}, {"category_id": 15, "poly": [864.0, 1258.0, 1568.0, 1258.0, 1568.0, 1290.0, 864.0, 1290.0], "score": 0.99, "text": "of a computer-generated scene were given in [19]. While these"}, {"category_id": 15, "poly": [864.0, 1290.0, 1566.0, 1290.0, 1566.0, 1322.0, 864.0, 1322.0], "score": 1.0, "text": "videos incorporate a sufficient amount of movement variation,"}, {"category_id": 15, "poly": [862.0, 1325.0, 1568.0, 1325.0, 1568.0, 1357.0, 862.0, 1357.0], "score": 0.99, "text": "they were generated from relatively simple models using low-"}, {"category_id": 15, "poly": [862.0, 1359.0, 1571.0, 1359.0, 1571.0, 1389.0, 862.0, 1389.0], "score": 0.99, "text": "resolution rendering, and they do not provide occlusion or"}, {"category_id": 15, "poly": [862.0, 1386.0, 1088.0, 1394.0, 1087.0, 1426.0, 861.0, 1418.0], "score": 0.98, "text": "discontinuity maps."}, {"category_id": 15, "poly": [129.0, 156.0, 839.0, 158.0, 838.0, 197.0, 129.0, 195.0], "score": 0.99, "text": "the matching cost by performing two-pass aggregation using"}, {"category_id": 15, "poly": [130.0, 188.0, 841.0, 193.0, 841.0, 229.0, 129.0, 225.0], "score": 0.98, "text": "two orthogonal 1D windows [5], [6], [8]. The two-pass method "}, {"category_id": 15, "poly": [129.0, 225.0, 841.0, 222.0, 841.0, 261.0, 129.0, 264.0], "score": 0.99, "text": "first aggregates matching costs in the vertical direction, and"}, {"category_id": 15, "poly": [134.0, 261.0, 838.0, 261.0, 838.0, 293.0, 134.0, 293.0], "score": 0.99, "text": "then computes a weighted sum of the aggregated costs in the"}, {"category_id": 15, "poly": [132.0, 291.0, 838.0, 291.0, 838.0, 330.0, 132.0, 330.0], "score": 0.99, "text": "horizontal direction. Given that support regions are of size"}, {"category_id": 15, "poly": [136.0, 360.0, 336.0, 360.0, 336.0, 392.0, 136.0, 392.0], "score": 0.99, "text": "aggregation from"}, {"category_id": 15, "poly": [415.0, 360.0, 452.0, 360.0, 452.0, 392.0, 415.0, 392.0], "score": 0.98, "text": "to"}, {"category_id": 15, "poly": [210.0, 321.0, 836.0, 321.0, 836.0, 360.0, 210.0, 360.0], "score": 0.98, "text": ", the two-pass method reduces the complexity of cost"}, {"category_id": 15, "poly": [887.0, 1416.0, 1571.0, 1419.0, 1571.0, 1458.0, 887.0, 1455.0], "score": 0.98, "text": " To evaluate the performance of temporal aggregation, a"}, {"category_id": 15, "poly": [862.0, 1453.0, 1566.0, 1453.0, 1566.0, 1485.0, 862.0, 1485.0], "score": 0.98, "text": "new synthetic stereo video sequence is introduced along with"}, {"category_id": 15, "poly": [862.0, 1490.0, 1566.0, 1487.0, 1566.0, 1519.0, 862.0, 1522.0], "score": 0.99, "text": "corresponding disparity maps, occlusion maps, and disconti-"}, {"category_id": 15, "poly": [862.0, 1519.0, 1571.0, 1519.0, 1571.0, 1558.0, 862.0, 1558.0], "score": 0.99, "text": "nuity maps for evaluating the performance of temporal stereo"}, {"category_id": 15, "poly": [864.0, 1556.0, 1568.0, 1556.0, 1568.0, 1588.0, 864.0, 1588.0], "score": 1.0, "text": "matching algorithms. To create the video sequence, a complex"}, {"category_id": 15, "poly": [864.0, 1590.0, 1568.0, 1590.0, 1568.0, 1620.0, 864.0, 1620.0], "score": 0.99, "text": "scene was constructed using Google Sketchup and a pair"}, {"category_id": 15, "poly": [864.0, 1622.0, 1568.0, 1622.0, 1568.0, 1655.0, 864.0, 1655.0], "score": 0.99, "text": "of animated paths were rendered photorealistically using the"}, {"category_id": 15, "poly": [859.0, 1650.0, 1571.0, 1652.0, 1571.0, 1691.0, 859.0, 1689.0], "score": 0.99, "text": " Kerkythea rendering software. Realistic material properties"}, {"category_id": 15, "poly": [864.0, 1689.0, 1566.0, 1689.0, 1566.0, 1721.0, 864.0, 1721.0], "score": 1.0, "text": "were used to give surfaces a natural-looking appearance by"}, {"category_id": 15, "poly": [864.0, 1723.0, 1566.0, 1723.0, 1566.0, 1755.0, 864.0, 1755.0], "score": 0.98, "text": "adjusting their specularity, reflectance, and diffusion. The"}, {"category_id": 15, "poly": [864.0, 1788.0, 1568.0, 1788.0, 1568.0, 1820.0, 864.0, 1820.0], "score": 1.0, "text": "frame rate of 30 frames per second, and a duration of 4"}, {"category_id": 15, "poly": [862.0, 1817.0, 1568.0, 1820.0, 1568.0, 1859.0, 861.0, 1856.0], "score": 0.98, "text": "seconds. In addition to performing photorealistic rendering."}, {"category_id": 15, "poly": [864.0, 1856.0, 1568.0, 1856.0, 1568.0, 1888.0, 864.0, 1888.0], "score": 0.99, "text": "depth renders of both video sequences were also generated and"}, {"category_id": 15, "poly": [864.0, 1888.0, 1566.0, 1888.0, 1566.0, 1920.0, 864.0, 1920.0], "score": 0.98, "text": "converted to ground truth disparity for the stereo video. The"}, {"category_id": 15, "poly": [862.0, 1920.0, 1564.0, 1920.0, 1564.0, 1952.0, 862.0, 1952.0], "score": 0.99, "text": "video sequences and ground truth data have been made avail-"}, {"category_id": 15, "poly": [862.0, 1950.0, 1566.0, 1953.0, 1566.0, 1985.0, 862.0, 1982.0], "score": 0.99, "text": "able at http://mc2.unl.edu/current-research"}, {"category_id": 15, "poly": [866.0, 1989.0, 1566.0, 1989.0, 1566.0, 2019.0, 866.0, 2019.0], "score": 0.98, "text": "/ image-processing/. Figure 2 shows two sample frames"}, {"category_id": 15, "poly": [862.0, 1755.0, 1312.0, 1755.0, 1312.0, 1788.0, 862.0, 1788.0], "score": 0.97, "text": "video sequence has a resolution of "}, {"category_id": 15, "poly": [1448.0, 1755.0, 1566.0, 1755.0, 1566.0, 1788.0, 1448.0, 1788.0], "score": 0.99, "text": "pixels,a"}, {"category_id": 15, "poly": [889.0, 197.0, 1566.0, 199.0, 1566.0, 238.0, 889.0, 236.0], "score": 1.0, "text": "Once the first iteration of stereo matching is complete,"}, {"category_id": 15, "poly": [864.0, 268.0, 1566.0, 268.0, 1566.0, 300.0, 864.0, 300.0], "score": 0.99, "text": "subsequent iterations. This is done by penalizing disparities"}, {"category_id": 15, "poly": [864.0, 302.0, 1568.0, 302.0, 1568.0, 335.0, 864.0, 335.0], "score": 1.0, "text": "that deviate from their expected values. The penalty function"}, {"category_id": 15, "poly": [862.0, 337.0, 996.0, 337.0, 996.0, 369.0, 862.0, 369.0], "score": 0.97, "text": "is given by"}, {"category_id": 15, "poly": [864.0, 236.0, 1094.0, 236.0, 1094.0, 268.0, 864.0, 268.0], "score": 0.96, "text": "disparityestimates"}, {"category_id": 15, "poly": [1135.0, 236.0, 1568.0, 236.0, 1568.0, 268.0, 1135.0, 268.0], "score": 0.97, "text": " can be used to guide matching in"}, {"category_id": 15, "poly": [157.0, 1366.0, 839.0, 1368.0, 838.0, 1407.0, 157.0, 1405.0], "score": 1.0, "text": "Having performed temporal cost aggregation, matches are"}, {"category_id": 15, "poly": [134.0, 1405.0, 834.0, 1405.0, 834.0, 1437.0, 134.0, 1437.0], "score": 0.99, "text": "determined using the Winner-Takes-All (WTA) match selec-"}, {"category_id": 15, "poly": [132.0, 1506.0, 374.0, 1506.0, 374.0, 1538.0, 132.0, 1538.0], "score": 1.0, "text": "cost, and is given by"}, {"category_id": 15, "poly": [692.0, 1439.0, 834.0, 1439.0, 834.0, 1471.0, 692.0, 1471.0], "score": 0.99, "text": ", is the can-"}, {"category_id": 15, "poly": [134.0, 1474.0, 276.0, 1474.0, 276.0, 1506.0, 134.0, 1506.0], "score": 0.98, "text": "didate pixel"}, {"category_id": 15, "poly": [362.0, 1474.0, 836.0, 1474.0, 836.0, 1506.0, 362.0, 1506.0], "score": 0.99, "text": " characterized by the minimum matching"}, {"category_id": 15, "poly": [134.0, 1439.0, 461.0, 1439.0, 461.0, 1471.0, 134.0, 1471.0], "score": 1.0, "text": "tion criteria. The match for"}, {"category_id": 15, "poly": [481.0, 1439.0, 628.0, 1439.0, 628.0, 1471.0, 481.0, 1471.0], "score": 0.96, "text": ", denoted as"}, {"category_id": 15, "poly": [134.0, 548.0, 838.0, 545.0, 838.0, 577.0, 134.0, 580.0], "score": 0.99, "text": "aggregation routine is exectuted. At each time instance, the"}, {"category_id": 15, "poly": [134.0, 614.0, 834.0, 614.0, 834.0, 646.0, 134.0, 646.0], "score": 1.0, "text": "weighted summation of costs obtained in the previous frames."}, {"category_id": 15, "poly": [132.0, 646.0, 838.0, 644.0, 838.0, 676.0, 132.0, 678.0], "score": 1.0, "text": "During temporal aggregation, the auxiliary cost is merged with"}, {"category_id": 15, "poly": [132.0, 678.0, 675.0, 681.0, 674.0, 713.0, 132.0, 710.0], "score": 0.99, "text": "the cost obtained from the current frame using"}, {"category_id": 15, "poly": [134.0, 580.0, 549.0, 580.0, 549.0, 612.0, 134.0, 612.0], "score": 1.0, "text": "algorithm stores an auxiliary cost"}, {"category_id": 15, "poly": [649.0, 580.0, 841.0, 580.0, 841.0, 612.0, 649.0, 612.0], "score": 0.96, "text": "which holds a"}, {"category_id": 15, "poly": [157.0, 445.0, 425.0, 442.0, 425.0, 481.0, 157.0, 484.0], "score": 0.98, "text": " Once aggregated costs"}, {"category_id": 15, "poly": [513.0, 445.0, 838.0, 442.0, 838.0, 481.0, 513.0, 484.0], "score": 0.96, "text": " have been computed for all"}, {"category_id": 15, "poly": [132.0, 481.0, 207.0, 481.0, 207.0, 513.0, 132.0, 513.0], "score": 1.0, "text": "pixels"}, {"category_id": 15, "poly": [228.0, 481.0, 838.0, 481.0, 838.0, 513.0, 228.0, 513.0], "score": 0.97, "text": " in the reference image and their respective matching"}, {"category_id": 15, "poly": [134.0, 516.0, 265.0, 516.0, 265.0, 548.0, 134.0, 548.0], "score": 1.0, "text": "candidates"}, {"category_id": 15, "poly": [289.0, 516.0, 838.0, 516.0, 838.0, 548.0, 289.0, 548.0], "score": 0.98, "text": " in the target image, a single-pass temporal"}, {"category_id": 15, "poly": [132.0, 1116.0, 841.0, 1116.0, 841.0, 1155.0, 132.0, 1155.0], "score": 0.99, "text": "in the temporal dimension. The temporal adaptive weight has "}, {"category_id": 15, "poly": [134.0, 1153.0, 838.0, 1153.0, 838.0, 1185.0, 134.0, 1185.0], "score": 0.99, "text": "the effect of preserving edges in the temporal domain, such"}, {"category_id": 15, "poly": [132.0, 1182.0, 836.0, 1182.0, 836.0, 1215.0, 132.0, 1215.0], "score": 0.98, "text": "that when a pixel coordinate transitions from one side of an"}, {"category_id": 15, "poly": [134.0, 1219.0, 838.0, 1219.0, 838.0, 1251.0, 134.0, 1251.0], "score": 0.98, "text": "edge to another in subsequent frames, the auxiliary cost is"}, {"category_id": 15, "poly": [134.0, 1254.0, 838.0, 1254.0, 838.0, 1283.0, 134.0, 1283.0], "score": 0.99, "text": "assigned a small weight and the majority of the cost is derived"}, {"category_id": 15, "poly": [130.0, 1283.0, 404.0, 1286.0, 404.0, 1318.0, 129.0, 1315.0], "score": 1.0, "text": "from the current frame."}, {"category_id": 15, "poly": [134.0, 1086.0, 207.0, 1086.0, 207.0, 1118.0, 134.0, 1118.0], "score": 0.99, "text": "where"}, {"category_id": 15, "poly": [237.0, 1086.0, 836.0, 1086.0, 836.0, 1118.0, 237.0, 1118.0], "score": 0.99, "text": "regulates the strength of grouping by color similarity"}, {"category_id": 15, "poly": [864.0, 600.0, 1568.0, 600.0, 1568.0, 632.0, 864.0, 632.0], "score": 1.0, "text": "and the matches are reselected using the WTA match selection"}, {"category_id": 15, "poly": [864.0, 635.0, 1568.0, 635.0, 1568.0, 667.0, 864.0, 667.0], "score": 0.99, "text": "criteria. The resulting disparity maps are then post-processed"}, {"category_id": 15, "poly": [864.0, 669.0, 1564.0, 669.0, 1564.0, 699.0, 864.0, 699.0], "score": 0.98, "text": "using a combination of median filtering and occlusion filling."}, {"category_id": 15, "poly": [864.0, 701.0, 1566.0, 701.0, 1566.0, 731.0, 864.0, 731.0], "score": 0.98, "text": "Finally, the current cost becomes the auxiliary cost for the next"}, {"category_id": 15, "poly": [862.0, 731.0, 1340.0, 731.0, 1340.0, 770.0, 862.0, 770.0], "score": 0.99, "text": "pair of frames in the video sequence, i.e.,"}, {"category_id": 15, "poly": [864.0, 768.0, 1017.0, 768.0, 1017.0, 800.0, 864.0, 800.0], "score": 1.0, "text": "for all pixels"}, {"category_id": 15, "poly": [1038.0, 768.0, 1465.0, 768.0, 1465.0, 800.0, 1038.0, 800.0], "score": 0.98, "text": " in the and their matching candidates"}, {"category_id": 15, "poly": [864.0, 502.0, 1427.0, 502.0, 1427.0, 532.0, 864.0, 532.0], "score": 1.0, "text": "values are incorporated into the matching cost as"}, {"category_id": 15, "poly": [864.0, 468.0, 1085.0, 468.0, 1085.0, 500.0, 864.0, 500.0], "score": 0.96, "text": "where the value of"}, {"category_id": 15, "poly": [1108.0, 468.0, 1564.0, 468.0, 1564.0, 500.0, 1108.0, 500.0], "score": 0.99, "text": "is chosen empirically. Next, the penalty"}, {"category_id": 15, "poly": [134.0, 866.0, 838.0, 866.0, 838.0, 898.0, 134.0, 898.0], "score": 0.99, "text": "temporal domain. The temporal adaptive weight computed"}, {"category_id": 15, "poly": [132.0, 967.0, 263.0, 967.0, 263.0, 999.0, 132.0, 999.0], "score": 0.93, "text": "is given by"}, {"category_id": 15, "poly": [134.0, 834.0, 320.0, 834.0, 320.0, 866.0, 134.0, 866.0], "score": 0.97, "text": "smoothing and"}, {"category_id": 15, "poly": [444.0, 834.0, 836.0, 834.0, 836.0, 866.0, 444.0, 866.0], "score": 0.92, "text": " enforces color similarity in the"}, {"category_id": 15, "poly": [178.0, 930.0, 838.0, 928.0, 839.0, 967.0, 178.0, 969.0], "score": 0.99, "text": ", located at the same spatial coordinate in the prior frame,"}, {"category_id": 15, "poly": [132.0, 795.0, 490.0, 800.0, 490.0, 832.0, 132.0, 827.0], "score": 0.99, "text": "where the feedback coefficient"}, {"category_id": 15, "poly": [512.0, 795.0, 836.0, 800.0, 836.0, 832.0, 512.0, 827.0], "score": 0.97, "text": " controls the amount of cost"}, {"category_id": 15, "poly": [136.0, 898.0, 465.0, 898.0, 465.0, 930.0, 136.0, 930.0], "score": 0.99, "text": "between the pixel of interest"}, {"category_id": 15, "poly": [486.0, 898.0, 838.0, 898.0, 838.0, 930.0, 486.0, 930.0], "score": 1.0, "text": "in the current frame and pixel"}, {"category_id": 15, "poly": [159.0, 1616.0, 836.0, 1616.0, 836.0, 1648.0, 159.0, 1648.0], "score": 0.99, "text": "To asses the level of confidence associated with selecting"}, {"category_id": 15, "poly": [132.0, 1648.0, 836.0, 1650.0, 836.0, 1682.0, 132.0, 1680.0], "score": 1.0, "text": "minimum cost matches, the algorithm determines another set"}, {"category_id": 15, "poly": [134.0, 1684.0, 838.0, 1684.0, 838.0, 1716.0, 134.0, 1716.0], "score": 1.0, "text": "of matches, this time from the target to reference image, and"}, {"category_id": 15, "poly": [134.0, 1783.0, 182.0, 1783.0, 182.0, 1815.0, 134.0, 1815.0], "score": 1.0, "text": "and"}, {"category_id": 15, "poly": [136.0, 1714.0, 580.0, 1714.0, 580.0, 1746.0, 136.0, 1746.0], "score": 0.98, "text": "verifies if the results agree. Given that"}, {"category_id": 15, "poly": [305.0, 1783.0, 592.0, 1783.0, 592.0, 1815.0, 305.0, 1815.0], "score": 0.99, "text": ", the confidence measure"}, {"category_id": 15, "poly": [628.0, 1783.0, 811.0, 1783.0, 811.0, 1815.0, 628.0, 1815.0], "score": 0.97, "text": "is computed as"}, {"category_id": 15, "poly": [132.0, 1746.0, 607.0, 1751.0, 607.0, 1783.0, 132.0, 1778.0], "score": 1.0, "text": "in the right image is the match for pixel"}, {"category_id": 15, "poly": [628.0, 1746.0, 836.0, 1751.0, 836.0, 1783.0, 628.0, 1778.0], "score": 0.98, "text": "in the left image,"}, {"category_id": 15, "poly": [695.0, 1714.0, 815.0, 1714.0, 815.0, 1746.0, 695.0, 1746.0], "score": 0.99, "text": ", i.e. pixel"}, {"category_id": 15, "poly": [1132.0, 814.0, 1298.0, 814.0, 1298.0, 852.0, 1132.0, 852.0], "score": 1.0, "text": "IV. RESULTS"}, {"category_id": 15, "poly": [155.0, 401.0, 481.0, 406.0, 480.0, 445.0, 155.0, 440.0], "score": 0.99, "text": "Temporal cost aggregation"}, {"category_id": 15, "poly": [129.0, 1325.0, 718.0, 1327.0, 718.0, 1366.0, 129.0, 1363.0], "score": 0.99, "text": "C. Disparity Selection and Confidence Assessment"}, {"category_id": 15, "poly": [888.0, 158.0, 1252.0, 158.0, 1252.0, 197.0, 888.0, 197.0], "score": 0.97, "text": "Iterative Disparity Refinement"}], "page_info": {"page_no": 2, "height": 2200, "width": 1700}}, {"layout_dets": [{"category_id": 1, "poly": [133.2669677734375, 156.7020721435547, 840.6729125976562, 156.7020721435547, 840.6729125976562, 257.75836181640625, 133.2669677734375, 257.75836181640625], "score": 0.9999951124191284}, {"category_id": 3, "poly": [866.177734375, 171.2958526611328, 1510.944580078125, 171.2958526611328, 1510.944580078125, 848.8190307617188, 866.177734375, 848.8190307617188], "score": 0.9999942779541016}, {"category_id": 1, "poly": [131.3756561279297, 1520.5887451171875, 838.545166015625, 1520.5887451171875, 838.545166015625, 1885.353515625, 131.3756561279297, 1885.353515625], "score": 0.9999925494194031}, {"category_id": 4, "poly": [131.56919860839844, 1352.6187744140625, 840.1758422851562, 1352.6187744140625, 840.1758422851562, 1490.513671875, 131.56919860839844, 1490.513671875], "score": 0.9999915361404419}, {"category_id": 1, "poly": [132.41786193847656, 1886.0615234375, 838.675537109375, 1886.0615234375, 838.675537109375, 2019.347412109375, 132.41786193847656, 2019.347412109375], "score": 0.9999526739120483}, {"category_id": 3, "poly": [136.71240234375, 278.259765625, 816.1984252929688, 278.259765625, 816.1984252929688, 1348.5758056640625, 136.71240234375, 1348.5758056640625], "score": 0.9999439120292664}, {"category_id": 1, "poly": [863.4852905273438, 1917.056884765625, 1569.6337890625, 1917.056884765625, 1569.6337890625, 2020.57421875, 863.4852905273438, 2020.57421875], "score": 0.9999344348907471}, {"category_id": 4, "poly": [861.7813720703125, 1749.4459228515625, 1567.659912109375, 1749.4459228515625, 1567.659912109375, 1852.389892578125, 861.7813720703125, 1852.389892578125], "score": 0.9986151456832886}, {"category_id": 3, "poly": [874.6467895507812, 1536.7642822265625, 1506.6514892578125, 1536.7642822265625, 1506.6514892578125, 1734.9659423828125, 874.6467895507812, 1734.9659423828125], "score": 0.9940656423568726}, {"category_id": 4, "poly": [859.3250122070312, 861.2320556640625, 1569.650634765625, 861.2320556640625, 1569.650634765625, 1033.0804443359375, 859.3250122070312, 1033.0804443359375], "score": 0.985899806022644}, {"category_id": 1, "poly": [861.6172485351562, 1064.186279296875, 1564.036865234375, 1064.186279296875, 1564.036865234375, 1135.5125732421875, 861.6172485351562, 1135.5125732421875], "score": 0.9128350019454956}, {"category_id": 3, "poly": [888.8074340820312, 1163.7965087890625, 1529.8028564453125, 1163.7965087890625, 1529.8028564453125, 1510.91162109375, 888.8074340820312, 1510.91162109375], "score": 0.7896175384521484}, {"category_id": 5, "poly": [900.75146484375, 1161.0631103515625, 1527.15673828125, 1161.0631103515625, 1527.15673828125, 1490.2149658203125, 900.75146484375, 1490.2149658203125], "score": 0.7772396802902222}, {"category_id": 0, "poly": [1178.85791015625, 152.25347900390625, 1284.6339111328125, 152.25347900390625, 1284.6339111328125, 179.1011962890625, 1178.85791015625, 179.1011962890625], "score": 0.5732811689376831}, {"category_id": 4, "poly": [1178.981689453125, 152.21678161621094, 1284.4158935546875, 152.21678161621094, 1284.4158935546875, 179.05447387695312, 1178.981689453125, 179.05447387695312], "score": 0.4503781795501709}, {"category_id": 13, "poly": [1295, 896, 1483, 896, 1483, 931, 1295, 931], "score": 0.93, "latex": "\\{\\pm0,\\pm20,\\pm40\\}"}, {"category_id": 13, "poly": [481, 1919, 534, 1919, 534, 1949, 481, 1949], "score": 0.87, "latex": "\\pm20"}, {"category_id": 13, "poly": [591, 1919, 644, 1919, 644, 1949, 591, 1949], "score": 0.87, "latex": "\\pm40"}, {"category_id": 13, "poly": [1227, 1436, 1253, 1436, 1253, 1459, 1227, 1459], "score": 0.86, "latex": "\\gamma_{c}"}, {"category_id": 13, "poly": [1295, 1436, 1323, 1436, 1323, 1461, 1295, 1461], "score": 0.85, "latex": "\\gamma_{g}"}, {"category_id": 13, "poly": [133, 1588, 186, 1588, 186, 1618, 133, 1618], "score": 0.85, "latex": "\\pm20"}, {"category_id": 13, "poly": [249, 1587, 302, 1587, 302, 1618, 249, 1618], "score": 0.84, "latex": "\\pm40"}, {"category_id": 13, "poly": [787, 1555, 828, 1555, 828, 1585, 787, 1585], "score": 0.82, "latex": "\\pm0"}, {"category_id": 13, "poly": [532, 1421, 572, 1421, 572, 1452, 532, 1452], "score": 0.81, "latex": "3^{\\mathrm{rd}}"}, {"category_id": 13, "poly": [230, 1389, 266, 1389, 266, 1419, 230, 1419], "score": 0.8, "latex": "1^{\\mathrm{st}}"}, {"category_id": 13, "poly": [655, 1986, 675, 1986, 675, 2013, 655, 2013], "score": 0.78, "latex": "\\lambda"}, {"category_id": 13, "poly": [200, 1455, 240, 1455, 240, 1486, 200, 1486], "score": 0.75, "latex": "4^{\\mathrm{th}}"}, {"category_id": 13, "poly": [954, 1255, 980, 1255, 980, 1275, 954, 1275], "score": 0.75, "latex": "\\gamma_{c}"}, {"category_id": 13, "poly": [954, 1281, 980, 1281, 980, 1302, 954, 1302], "score": 0.74, "latex": "\\gamma_{g}"}, {"category_id": 13, "poly": [959, 1227, 976, 1227, 976, 1245, 959, 1245], "score": 0.74, "latex": "\\tau"}, {"category_id": 13, "poly": [960, 1352, 976, 1352, 976, 1372, 960, 1372], "score": 0.72, "latex": "k"}, {"category_id": 13, "poly": [410, 1986, 430, 1986, 430, 2013, 410, 2013], "score": 0.7, "latex": "\\lambda"}, {"category_id": 13, "poly": [955, 1331, 979, 1331, 979, 1351, 955, 1351], "score": 0.7, "latex": "\\gamma_{t}"}, {"category_id": 13, "poly": [1489, 1752, 1510, 1752, 1510, 1778, 1489, 1778], "score": 0.69, "latex": "\\lambda"}, {"category_id": 13, "poly": [1176, 965, 1195, 965, 1195, 992, 1176, 992], "score": 0.69, "latex": "\\lambda"}, {"category_id": 13, "poly": [246, 1421, 289, 1421, 289, 1452, 246, 1452], "score": 0.69, "latex": "2^{\\mathrm{nd}}"}, {"category_id": 13, "poly": [958, 1302, 977, 1302, 977, 1323, 958, 1323], "score": 0.63, "latex": "\\lambda"}, {"category_id": 13, "poly": [959, 1380, 977, 1380, 977, 1397, 959, 1397], "score": 0.58, "latex": "\\alpha"}, {"category_id": 13, "poly": [436, 1621, 455, 1621, 455, 1648, 436, 1648], "score": 0.58, "latex": "\\lambda"}, {"category_id": 13, "poly": [959, 1204, 977, 1204, 977, 1219, 959, 1219], "score": 0.42, "latex": "\\omega"}, {"category_id": 13, "poly": [870, 1592, 890, 1592, 890, 1617, 870, 1617], "score": 0.31, "latex": "\\lambda"}, {"category_id": 15, "poly": [134.0, 160.0, 836.0, 160.0, 836.0, 192.0, 134.0, 192.0], "score": 0.99, "text": "of the synthetic stereo scene from a single camera perspective,"}, {"category_id": 15, "poly": [134.0, 195.0, 838.0, 195.0, 838.0, 227.0, 134.0, 227.0], "score": 0.99, "text": "along with the ground truth disparity, occlusion map, and"}, {"category_id": 15, "poly": [130.0, 222.0, 347.0, 230.0, 346.0, 264.0, 129.0, 256.0], "score": 0.99, "text": "discontinuity map."}, {"category_id": 15, "poly": [155.0, 1517.0, 841.0, 1519.0, 841.0, 1558.0, 155.0, 1556.0], "score": 0.99, "text": " The results of temporal stereo matching are given in Figure"}, {"category_id": 15, "poly": [132.0, 1657.0, 838.0, 1657.0, 838.0, 1689.0, 132.0, 1689.0], "score": 0.99, "text": "stereo matching methods, improvements are negligible when"}, {"category_id": 15, "poly": [132.0, 1691.0, 838.0, 1691.0, 838.0, 1723.0, 132.0, 1723.0], "score": 0.99, "text": "no noise is added to the images [10], [19]. This is largely due"}, {"category_id": 15, "poly": [132.0, 1723.0, 836.0, 1723.0, 836.0, 1753.0, 132.0, 1753.0], "score": 0.98, "text": "to the fact that the video used to evaluate these methods is"}, {"category_id": 15, "poly": [129.0, 1753.0, 838.0, 1751.0, 839.0, 1790.0, 129.0, 1792.0], "score": 0.99, "text": " computer generated with very little noise to start with, thus"}, {"category_id": 15, "poly": [134.0, 1790.0, 836.0, 1790.0, 836.0, 1822.0, 134.0, 1822.0], "score": 0.99, "text": "the noise suppression achieved with temporal stereo matching"}, {"category_id": 15, "poly": [132.0, 1817.0, 839.0, 1822.0, 838.0, 1859.0, 132.0, 1854.0], "score": 0.99, "text": "shows little to no improvement over methods that operate on"}, {"category_id": 15, "poly": [130.0, 1856.0, 319.0, 1859.0, 318.0, 1891.0, 129.0, 1888.0], "score": 0.99, "text": "pairs of images."}, {"category_id": 15, "poly": [187.0, 1590.0, 248.0, 1590.0, 248.0, 1622.0, 187.0, 1622.0], "score": 0.87, "text": ",and"}, {"category_id": 15, "poly": [303.0, 1590.0, 838.0, 1590.0, 838.0, 1622.0, 303.0, 1622.0], "score": 0.98, "text": ". Each performance plot is given as a function"}, {"category_id": 15, "poly": [127.0, 1551.0, 786.0, 1554.0, 786.0, 1593.0, 127.0, 1590.0], "score": 0.98, "text": " 3 for uniform additive noise confined to the ranges of"}, {"category_id": 15, "poly": [134.0, 1622.0, 435.0, 1622.0, 435.0, 1655.0, 134.0, 1655.0], "score": 0.99, "text": "of the feedback coefficient"}, {"category_id": 15, "poly": [456.0, 1622.0, 836.0, 1622.0, 836.0, 1655.0, 456.0, 1655.0], "score": 0.97, "text": ". As with the majority of temporal"}, {"category_id": 15, "poly": [134.0, 1359.0, 834.0, 1359.0, 834.0, 1391.0, 134.0, 1391.0], "score": 0.99, "text": "Figure 2: Two sample frames from the synthetic video se-"}, {"category_id": 15, "poly": [573.0, 1418.0, 836.0, 1421.0, 836.0, 1460.0, 573.0, 1457.0], "score": 1.0, "text": "row), and discontinuity"}, {"category_id": 15, "poly": [134.0, 1393.0, 229.0, 1393.0, 229.0, 1425.0, 134.0, 1425.0], "score": 0.96, "text": "quence ("}, {"category_id": 15, "poly": [267.0, 1393.0, 836.0, 1393.0, 836.0, 1425.0, 267.0, 1425.0], "score": 0.98, "text": "row), along with their corresponding ground truth"}, {"category_id": 15, "poly": [127.0, 1456.0, 199.0, 1450.0, 199.0, 1489.0, 128.0, 1495.0], "score": 0.91, "text": "map ("}, {"category_id": 15, "poly": [241.0, 1456.0, 309.0, 1450.0, 310.0, 1489.0, 241.0, 1495.0], "score": 1.0, "text": "row)."}, {"category_id": 15, "poly": [129.0, 1418.0, 245.0, 1421.0, 245.0, 1460.0, 129.0, 1457.0], "score": 0.93, "text": " disparity "}, {"category_id": 15, "poly": [290.0, 1418.0, 531.0, 1421.0, 531.0, 1460.0, 290.0, 1457.0], "score": 1.0, "text": "row), occlusion map ("}, {"category_id": 15, "poly": [159.0, 1888.0, 836.0, 1888.0, 836.0, 1920.0, 159.0, 1920.0], "score": 0.99, "text": " Significant improvements in accuracy can be seen in Figure"}, {"category_id": 15, "poly": [132.0, 1950.0, 839.0, 1955.0, 838.0, 1987.0, 132.0, 1982.0], "score": 1.0, "text": "the effect of noise in the current frame is reduced by increasing"}, {"category_id": 15, "poly": [134.0, 1920.0, 480.0, 1920.0, 480.0, 1952.0, 134.0, 1952.0], "score": 0.99, "text": "3 when the noise has ranges of"}, {"category_id": 15, "poly": [535.0, 1920.0, 590.0, 1920.0, 590.0, 1952.0, 535.0, 1952.0], "score": 0.92, "text": " and"}, {"category_id": 15, "poly": [645.0, 1920.0, 836.0, 1920.0, 836.0, 1952.0, 645.0, 1952.0], "score": 0.96, "text": ". In this scenario,"}, {"category_id": 15, "poly": [676.0, 1989.0, 838.0, 1989.0, 838.0, 2019.0, 676.0, 2019.0], "score": 0.98, "text": "has the effect"}, {"category_id": 15, "poly": [134.0, 1989.0, 409.0, 1989.0, 409.0, 2019.0, 134.0, 2019.0], "score": 1.0, "text": "the feedback coefficient"}, {"category_id": 15, "poly": [431.0, 1989.0, 654.0, 1989.0, 654.0, 2019.0, 431.0, 2019.0], "score": 0.97, "text": ". This increasing of"}, {"category_id": 15, "poly": [864.0, 1920.0, 1566.0, 1920.0, 1566.0, 1952.0, 864.0, 1952.0], "score": 0.98, "text": "of averaging out noise in the per-pixel costs by selecting"}, {"category_id": 15, "poly": [861.0, 1950.0, 1566.0, 1948.0, 1566.0, 1987.0, 862.0, 1989.0], "score": 0.98, "text": "matches based more heavily upon the auxiliary cost, which"}, {"category_id": 15, "poly": [862.0, 1989.0, 1568.0, 1989.0, 1568.0, 2021.0, 862.0, 2021.0], "score": 0.99, "text": "is essentially a much more stable running average of the cost"}, {"category_id": 15, "poly": [864.0, 1788.0, 1564.0, 1785.0, 1564.0, 1817.0, 864.0, 1820.0], "score": 0.99, "text": "responding to the smallest mean squared error (MSE) of the"}, {"category_id": 15, "poly": [864.0, 1822.0, 1427.0, 1822.0, 1427.0, 1854.0, 864.0, 1854.0], "score": 0.99, "text": "disparity estimates for a range of noise strengths."}, {"category_id": 15, "poly": [862.0, 1748.0, 1488.0, 1753.0, 1488.0, 1785.0, 861.0, 1781.0], "score": 0.99, "text": "Figure 4: Optimal values of the feedback coefficient "}, {"category_id": 15, "poly": [1511.0, 1748.0, 1561.0, 1753.0, 1561.0, 1785.0, 1511.0, 1781.0], "score": 0.96, "text": "cor-"}, {"category_id": 15, "poly": [864.0, 866.0, 1566.0, 866.0, 1566.0, 898.0, 864.0, 898.0], "score": 0.99, "text": "Figure 3: Performance of temporal matching at different levels"}, {"category_id": 15, "poly": [864.0, 935.0, 1566.0, 933.0, 1566.0, 965.0, 864.0, 967.0], "score": 0.98, "text": "squared error (MSE) of disparities is plotted versus the values"}, {"category_id": 15, "poly": [864.0, 1001.0, 1492.0, 1001.0, 1492.0, 1031.0, 864.0, 1031.0], "score": 0.99, "text": "values of MSE obtained without temporal aggregation."}, {"category_id": 15, "poly": [864.0, 901.0, 1294.0, 901.0, 1294.0, 933.0, 864.0, 933.0], "score": 0.99, "text": "of uniformly distributed image noise"}, {"category_id": 15, "poly": [1484.0, 901.0, 1568.0, 901.0, 1568.0, 933.0, 1484.0, 933.0], "score": 0.99, "text": ".Mean"}, {"category_id": 15, "poly": [864.0, 967.0, 1175.0, 967.0, 1175.0, 999.0, 864.0, 999.0], "score": 0.99, "text": "of the feedback coefficient"}, {"category_id": 15, "poly": [1196.0, 967.0, 1568.0, 967.0, 1568.0, 999.0, 1196.0, 999.0], "score": 0.99, "text": ". Dashed lines correspond to the"}, {"category_id": 15, "poly": [857.0, 1061.0, 1566.0, 1068.0, 1566.0, 1107.0, 857.0, 1100.0], "score": 0.99, "text": " Table I: Parameters used in the evaluation of real-time tempo-"}, {"category_id": 15, "poly": [859.0, 1102.0, 1093.0, 1105.0, 1092.0, 1137.0, 859.0, 1134.0], "score": 1.0, "text": "ral stereo matching."}, {"category_id": 15, "poly": [1178.0, 151.0, 1282.0, 151.0, 1282.0, 186.0, 1178.0, 186.0], "score": 1.0, "text": "Noise: \u00b10"}, {"category_id": 15, "poly": [1178.0, 151.0, 1282.0, 151.0, 1282.0, 186.0, 1178.0, 186.0], "score": 1.0, "text": "Noise: \u00b10"}], "page_info": {"page_no": 3, "height": 2200, "width": 1700}}, {"layout_dets": [{"category_id": 5, "poly": [880.81298828125, 613.750244140625, 1552.5638427734375, 613.750244140625, 1552.5638427734375, 855.9174194335938, 880.81298828125, 855.9174194335938], "score": 0.9999957084655762}, {"category_id": 1, "poly": [862.7925415039062, 158.05548095703125, 1569.6671142578125, 158.05548095703125, 1569.6671142578125, 456.6153869628906, 862.7925415039062, 456.6153869628906], "score": 0.9999922513961792}, {"category_id": 1, "poly": [864.6585083007812, 1061.7374267578125, 1570.4825439453125, 1061.7374267578125, 1570.4825439453125, 1459.7132568359375, 864.6585083007812, 1459.7132568359375], "score": 0.9999921321868896}, {"category_id": 1, "poly": [130.64285278320312, 1519.7022705078125, 836.2221069335938, 1519.7022705078125, 836.2221069335938, 1882.68359375, 130.64285278320312, 1882.68359375], "score": 0.9999898672103882}, {"category_id": 1, "poly": [133.1135711669922, 158.4307861328125, 837.9683837890625, 158.4307861328125, 837.9683837890625, 323.343017578125, 133.1135711669922, 323.343017578125], "score": 0.9999892115592957}, {"category_id": 4, "poly": [132.3511199951172, 1347.8763427734375, 839.7514038085938, 1347.8763427734375, 839.7514038085938, 1476.9757080078125, 132.3511199951172, 1476.9757080078125], "score": 0.9999880790710449}, {"category_id": 7, "poly": [887.6280517578125, 860.9362182617188, 1551.5972900390625, 860.9362182617188, 1551.5972900390625, 964.0142211914062, 887.6280517578125, 964.0142211914062], "score": 0.9999836683273315}, {"category_id": 1, "poly": [869.9986572265625, 1514.7762451171875, 1571.624755859375, 1514.7762451171875, 1571.624755859375, 2022.618896484375, 869.9986572265625, 2022.618896484375], "score": 0.9999811053276062}, {"category_id": 3, "poly": [164.82151794433594, 352.74810791015625, 805.8219604492188, 352.74810791015625, 805.8219604492188, 1320.43310546875, 164.82151794433594, 1320.43310546875], "score": 0.9999799728393555}, {"category_id": 0, "poly": [1137.668701171875, 1477.0120849609375, 1293.498046875, 1477.0120849609375, 1293.498046875, 1502.5439453125, 1137.668701171875, 1502.5439453125], "score": 0.9999679327011108}, {"category_id": 1, "poly": [133.0285186767578, 1886.7501220703125, 837.0147705078125, 1886.7501220703125, 837.0147705078125, 2018.0294189453125, 133.0285186767578, 2018.0294189453125], "score": 0.9999630451202393}, {"category_id": 0, "poly": [1114.8399658203125, 1022.4933471679688, 1317.0313720703125, 1022.4933471679688, 1317.0313720703125, 1052.679931640625, 1114.8399658203125, 1052.679931640625], "score": 0.9999338984489441}, {"category_id": 1, "poly": [862.0576171875, 480.8196105957031, 1565.8367919921875, 480.8196105957031, 1565.8367919921875, 577.5508422851562, 862.0576171875, 577.5508422851562], "score": 0.8958550691604614}, {"category_id": 6, "poly": [862.0606079101562, 480.7809753417969, 1565.667724609375, 480.7809753417969, 1565.667724609375, 577.4689331054688, 862.0606079101562, 577.4689331054688], "score": 0.4145430028438568}, {"category_id": 13, "poly": [736, 1445, 827, 1445, 827, 1475, 736, 1475], "score": 0.9, "latex": "\\lambda=0.8"}, {"category_id": 13, "poly": [1003, 887, 1105, 887, 1105, 911, 1003, 911], "score": 0.89, "latex": "320\\times240"}, {"category_id": 13, "poly": [338, 1446, 391, 1446, 391, 1475, 338, 1475], "score": 0.87, "latex": "\\pm30"}, {"category_id": 13, "poly": [166, 1619, 219, 1619, 219, 1649, 166, 1649], "score": 0.85, "latex": "\\pm40"}, {"category_id": 13, "poly": [301, 196, 329, 196, 329, 224, 301, 224], "score": 0.84, "latex": "\\gamma_{t}"}, {"category_id": 13, "poly": [795, 1586, 836, 1586, 836, 1616, 795, 1616], "score": 0.84, "latex": "\\pm0"}, {"category_id": 13, "poly": [1037, 939, 1059, 939, 1059, 960, 1037, 960], "score": 0.83, "latex": "\\%"}, {"category_id": 13, "poly": [462, 1586, 482, 1586, 482, 1613, 462, 1613], "score": 0.78, "latex": "\\lambda"}, {"category_id": 15, "poly": [862.0, 160.0, 1571.0, 160.0, 1571.0, 192.0, 862.0, 192.0], "score": 0.98, "text": "the proposed implementation achieves the highest speed of"}, {"category_id": 15, "poly": [864.0, 195.0, 1566.0, 195.0, 1566.0, 227.0, 864.0, 227.0], "score": 0.99, "text": "operation measured by the number of disparity hypotheses"}, {"category_id": 15, "poly": [864.0, 227.0, 1568.0, 227.0, 1568.0, 259.0, 864.0, 259.0], "score": 0.99, "text": "evaluated per second, as shown in Table I1. It is also the second"}, {"category_id": 15, "poly": [862.0, 261.0, 1568.0, 261.0, 1568.0, 293.0, 862.0, 293.0], "score": 0.99, "text": "most accurate real-time method in terms of error rate, as"}, {"category_id": 15, "poly": [864.0, 296.0, 1564.0, 296.0, 1564.0, 325.0, 864.0, 325.0], "score": 1.0, "text": "measured using the Middlebury stereo evaluation benchmark."}, {"category_id": 15, "poly": [859.0, 323.0, 1568.0, 325.0, 1568.0, 358.0, 859.0, 355.0], "score": 0.98, "text": " It should be noted that it is difficult to establish an unbiased"}, {"category_id": 15, "poly": [862.0, 358.0, 1566.0, 358.0, 1566.0, 390.0, 862.0, 390.0], "score": 1.0, "text": "metric for speed comparisons, as the architecture, number of"}, {"category_id": 15, "poly": [866.0, 394.0, 1568.0, 394.0, 1568.0, 426.0, 866.0, 426.0], "score": 0.98, "text": "cores, and clock speed of graphics hardware used are not"}, {"category_id": 15, "poly": [862.0, 424.0, 1259.0, 429.0, 1259.0, 461.0, 861.0, 456.0], "score": 0.99, "text": "consistent across implementations."}, {"category_id": 15, "poly": [889.0, 1061.0, 1571.0, 1061.0, 1571.0, 1100.0, 889.0, 1100.0], "score": 1.0, "text": "While the majority of stereo matching algorithms focus"}, {"category_id": 15, "poly": [859.0, 1093.0, 1571.0, 1095.0, 1571.0, 1134.0, 859.0, 1132.0], "score": 0.99, "text": " on achieving high accuracy on still images, the volume of"}, {"category_id": 15, "poly": [862.0, 1130.0, 1564.0, 1130.0, 1564.0, 1162.0, 862.0, 1162.0], "score": 0.99, "text": "research aimed at recovery of temporally consistent disparity"}, {"category_id": 15, "poly": [862.0, 1162.0, 1568.0, 1162.0, 1568.0, 1201.0, 862.0, 1201.0], "score": 0.99, "text": "maps remains disproportionally small. This paper introduces"}, {"category_id": 15, "poly": [862.0, 1196.0, 1568.0, 1196.0, 1568.0, 1235.0, 862.0, 1235.0], "score": 0.98, "text": "an efficient temporal cost aggregation scheme that can easily"}, {"category_id": 15, "poly": [859.0, 1226.0, 1571.0, 1228.0, 1571.0, 1267.0, 859.0, 1265.0], "score": 0.99, "text": "be combined with conventional spatial cost aggregation to"}, {"category_id": 15, "poly": [864.0, 1265.0, 1568.0, 1265.0, 1568.0, 1297.0, 864.0, 1297.0], "score": 1.0, "text": "improve the accuracy of stereo matching when operating on"}, {"category_id": 15, "poly": [864.0, 1297.0, 1568.0, 1297.0, 1568.0, 1329.0, 864.0, 1329.0], "score": 0.99, "text": "video sequences. A synthetic video sequence, along with"}, {"category_id": 15, "poly": [864.0, 1331.0, 1568.0, 1331.0, 1568.0, 1364.0, 864.0, 1364.0], "score": 0.99, "text": "ground truth disparity data, was generated to evaluate the"}, {"category_id": 15, "poly": [862.0, 1361.0, 1571.0, 1361.0, 1571.0, 1400.0, 862.0, 1400.0], "score": 0.98, "text": "performance of the proposed method. It was shown that"}, {"category_id": 15, "poly": [864.0, 1398.0, 1571.0, 1398.0, 1571.0, 1430.0, 864.0, 1430.0], "score": 0.98, "text": "temporal aggregation is significantly more robust to noise than"}, {"category_id": 15, "poly": [862.0, 1430.0, 1497.0, 1430.0, 1497.0, 1462.0, 862.0, 1462.0], "score": 0.99, "text": "a method that only considers the current stereo frames."}, {"category_id": 15, "poly": [157.0, 1517.0, 838.0, 1517.0, 838.0, 1556.0, 157.0, 1556.0], "score": 0.99, "text": "The optimal value of the feedback coefficient is largely"}, {"category_id": 15, "poly": [134.0, 1554.0, 836.0, 1554.0, 836.0, 1584.0, 134.0, 1584.0], "score": 0.97, "text": "dependent on the noise being added to the image. Figure 4"}, {"category_id": 15, "poly": [132.0, 1655.0, 838.0, 1655.0, 838.0, 1684.0, 132.0, 1684.0], "score": 0.99, "text": "rely on the auxiliary cost when noise is high and it is more"}, {"category_id": 15, "poly": [132.0, 1684.0, 839.0, 1689.0, 838.0, 1721.0, 132.0, 1716.0], "score": 0.98, "text": "beneficial to rely on the current cost when noise is low. Figure"}, {"category_id": 15, "poly": [132.0, 1719.0, 839.0, 1723.0, 838.0, 1755.0, 132.0, 1751.0], "score": 1.0, "text": "5 illustrates the improvements that are achieved when applying"}, {"category_id": 15, "poly": [134.0, 1755.0, 836.0, 1755.0, 836.0, 1785.0, 134.0, 1785.0], "score": 0.98, "text": "temporal stereo matching to a particular pair of frames in the"}, {"category_id": 15, "poly": [134.0, 1788.0, 834.0, 1788.0, 834.0, 1820.0, 134.0, 1820.0], "score": 1.0, "text": "synthetic video sequence. Clearly, the noise in the disparity"}, {"category_id": 15, "poly": [134.0, 1822.0, 836.0, 1822.0, 836.0, 1854.0, 134.0, 1854.0], "score": 0.99, "text": "map is drastically reduced when temporal stereo matching is"}, {"category_id": 15, "poly": [132.0, 1856.0, 196.0, 1856.0, 196.0, 1886.0, 132.0, 1886.0], "score": 1.0, "text": "used."}, {"category_id": 15, "poly": [132.0, 1620.0, 165.0, 1620.0, 165.0, 1652.0, 132.0, 1652.0], "score": 0.99, "text": "to"}, {"category_id": 15, "poly": [220.0, 1620.0, 838.0, 1620.0, 838.0, 1652.0, 220.0, 1652.0], "score": 0.98, "text": ". As intuition would suggest, it is more beneficial to"}, {"category_id": 15, "poly": [127.0, 1584.0, 461.0, 1581.0, 461.0, 1620.0, 127.0, 1623.0], "score": 0.96, "text": " shows the optimal values of"}, {"category_id": 15, "poly": [483.0, 1584.0, 794.0, 1581.0, 794.0, 1620.0, 483.0, 1623.0], "score": 0.99, "text": "for noise ranging between"}, {"category_id": 15, "poly": [134.0, 160.0, 836.0, 160.0, 836.0, 192.0, 134.0, 192.0], "score": 0.99, "text": "over the most recent frames. By maintaining a reasonably"}, {"category_id": 15, "poly": [134.0, 229.0, 836.0, 229.0, 836.0, 261.0, 134.0, 261.0], "score": 0.98, "text": "edges, essentially reducing over-smoothing of a pixel's dis-"}, {"category_id": 15, "poly": [132.0, 261.0, 838.0, 261.0, 838.0, 293.0, 132.0, 293.0], "score": 0.99, "text": "parity when a pixel transitions from one depth to another in"}, {"category_id": 15, "poly": [130.0, 293.0, 354.0, 296.0, 353.0, 328.0, 129.0, 325.0], "score": 1.0, "text": "subsequent frames."}, {"category_id": 15, "poly": [134.0, 192.0, 300.0, 192.0, 300.0, 225.0, 134.0, 225.0], "score": 0.93, "text": "high value of"}, {"category_id": 15, "poly": [330.0, 192.0, 836.0, 192.0, 836.0, 225.0, 330.0, 225.0], "score": 0.99, "text": ", the auxiliary cost also preserves temporal"}, {"category_id": 15, "poly": [132.0, 1345.0, 836.0, 1348.0, 836.0, 1382.0, 132.0, 1380.0], "score": 1.0, "text": "Figure 5: A comparison of stereo matching without temporal"}, {"category_id": 15, "poly": [132.0, 1382.0, 834.0, 1382.0, 834.0, 1414.0, 132.0, 1414.0], "score": 0.98, "text": "cost aggregation (top\uff09 and with temporal cost aggregation"}, {"category_id": 15, "poly": [134.0, 1416.0, 836.0, 1416.0, 836.0, 1446.0, 134.0, 1446.0], "score": 0.98, "text": "(bottom) for a single frame in the synthetic video sequence"}, {"category_id": 15, "poly": [134.0, 1448.0, 337.0, 1446.0, 337.0, 1478.0, 134.0, 1480.0], "score": 0.98, "text": "where the noise is"}, {"category_id": 15, "poly": [392.0, 1448.0, 735.0, 1446.0, 735.0, 1478.0, 392.0, 1480.0], "score": 0.99, "text": "and the feedback coefficient is"}, {"category_id": 15, "poly": [896.0, 855.0, 1324.0, 857.0, 1323.0, 896.0, 896.0, 894.0], "score": 0.95, "text": "1I Millions of Disparity Estimates per Second."}, {"category_id": 15, "poly": [903.0, 912.0, 1550.0, 912.0, 1550.0, 944.0, 903.0, 944.0], "score": 0.99, "text": "3 As measured by the Middlebury stereo performance benchmark using"}, {"category_id": 15, "poly": [901.0, 887.0, 1002.0, 887.0, 1002.0, 919.0, 901.0, 919.0], "score": 0.99, "text": "2Assumes"}, {"category_id": 15, "poly": [1106.0, 887.0, 1404.0, 887.0, 1404.0, 919.0, 1106.0, 919.0], "score": 0.98, "text": "images with 32 disparity levels."}, {"category_id": 15, "poly": [915.0, 937.0, 1036.0, 937.0, 1036.0, 969.0, 915.0, 969.0], "score": 0.96, "text": "the avgerage"}, {"category_id": 15, "poly": [1060.0, 937.0, 1192.0, 937.0, 1192.0, 969.0, 1060.0, 969.0], "score": 0.96, "text": "of bad pixels."}, {"category_id": 15, "poly": [873.0, 1515.0, 1571.0, 1515.0, 1571.0, 1545.0, 873.0, 1545.0], "score": 0.97, "text": "[1] D. Scharstein and R. Szeliski, \u201cA taxonomy and evaluation of dense "}, {"category_id": 15, "poly": [915.0, 1542.0, 1573.0, 1542.0, 1573.0, 1572.0, 915.0, 1572.0], "score": 0.98, "text": "two-frame stereo correspondence algorithms\u201d\u2019 International Journal of"}, {"category_id": 15, "poly": [915.0, 1565.0, 1409.0, 1565.0, 1409.0, 1597.0, 915.0, 1597.0], "score": 0.98, "text": "Computer Vision, vol. 47, pp. 7-42, April-June 2002."}, {"category_id": 15, "poly": [871.0, 1588.0, 1568.0, 1590.0, 1568.0, 1623.0, 871.0, 1620.0], "score": 0.98, "text": "[2] D. Scharstein and R. Szeliski, \u201cHigh-accuracy stereo depth maps using"}, {"category_id": 15, "poly": [915.0, 1616.0, 1568.0, 1616.0, 1568.0, 1648.0, 915.0, 1648.0], "score": 0.97, "text": "structured light,\u201d in In IEEE Computer Society Conference on Computer"}, {"category_id": 15, "poly": [915.0, 1641.0, 1508.0, 1641.0, 1508.0, 1673.0, 915.0, 1673.0], "score": 0.98, "text": "Vision and Pattern Recognition, vol. 1, pp. 195-202, June 2003."}, {"category_id": 15, "poly": [873.0, 1666.0, 1568.0, 1666.0, 1568.0, 1696.0, 873.0, 1696.0], "score": 0.99, "text": "[3] J. Kowalczuk, E. Psota, and L. Perez, \u201cReal-time stereo matching on"}, {"category_id": 15, "poly": [912.0, 1689.0, 1571.0, 1689.0, 1571.0, 1721.0, 912.0, 1721.0], "score": 0.98, "text": " CUDA using an iterative refinement method for adaptive support-weight"}, {"category_id": 15, "poly": [915.0, 1714.0, 1571.0, 1714.0, 1571.0, 1746.0, 915.0, 1746.0], "score": 0.99, "text": "correspondences,\u201d Circuits and Systems for Video Technology, IEEE"}, {"category_id": 15, "poly": [908.0, 1737.0, 1374.0, 1735.0, 1374.0, 1774.0, 908.0, 1776.0], "score": 0.96, "text": "Transactions on, vol. 23, Ppp. 94 -104, Jan. 2013."}, {"category_id": 15, "poly": [873.0, 1765.0, 1568.0, 1765.0, 1568.0, 1797.0, 873.0, 1797.0], "score": 0.99, "text": "[4] K.-J. Yoon and I.-S. Kweon, Locally adaptive support-weight approach"}, {"category_id": 15, "poly": [912.0, 1790.0, 1571.0, 1790.0, 1571.0, 1822.0, 912.0, 1822.0], "score": 0.97, "text": "for visual correspondence search,' in CVPR'05: Proceedings of the 2005"}, {"category_id": 15, "poly": [915.0, 1815.0, 1571.0, 1815.0, 1571.0, 1847.0, 915.0, 1847.0], "score": 0.96, "text": "IEEE Computer Society Conference on ComputerVision andPattern"}, {"category_id": 15, "poly": [915.0, 1840.0, 1568.0, 1840.0, 1568.0, 1872.0, 915.0, 1872.0], "score": 0.97, "text": "Recognition (CVPR'05) - Volume 2, (Washington, DC, USA), Pp. 924-"}, {"category_id": 15, "poly": [912.0, 1863.0, 1247.0, 1863.0, 1247.0, 1895.0, 912.0, 1895.0], "score": 0.98, "text": "931, IEEE Computer Society, 2005."}, {"category_id": 15, "poly": [873.0, 1891.0, 1568.0, 1891.0, 1568.0, 1923.0, 873.0, 1923.0], "score": 0.97, "text": "[5] L. Wang, M. Liao, M. Gong, R. Yang, and D. Nister, \u201cHigh-quality real-"}, {"category_id": 15, "poly": [912.0, 1916.0, 1566.0, 1916.0, 1566.0, 1946.0, 912.0, 1946.0], "score": 0.99, "text": "time stereo using adaptive cost aggregation and dynamic programming,\""}, {"category_id": 15, "poly": [910.0, 1936.0, 1568.0, 1939.0, 1568.0, 1971.0, 910.0, 1969.0], "score": 0.94, "text": "in 3DPVT'06:Proceedings of the Third International Symposium"}, {"category_id": 15, "poly": [915.0, 1964.0, 1568.0, 1964.0, 1568.0, 1996.0, 915.0, 1996.0], "score": 0.98, "text": "on 3D Data Processing, Visualization, and Transmission (3DPVT'06),"}, {"category_id": 15, "poly": [915.0, 1989.0, 1564.0, 1989.0, 1564.0, 2021.0, 915.0, 2021.0], "score": 1.0, "text": "(Washington, DC, USA), Pp. 798-805, IEEE Computer Society, 2006."}, {"category_id": 15, "poly": [1134.0, 1471.0, 1296.0, 1471.0, 1296.0, 1510.0, 1134.0, 1510.0], "score": 1.0, "text": "REFERENCES"}, {"category_id": 15, "poly": [159.0, 1888.0, 836.0, 1888.0, 836.0, 1920.0, 159.0, 1920.0], "score": 0.99, "text": "The algorithm was implement using NVIDIA's Compute"}, {"category_id": 15, "poly": [134.0, 1920.0, 834.0, 1920.0, 834.0, 1950.0, 134.0, 1950.0], "score": 0.98, "text": "Unified Device Architecture (CUDA). The details of the im-"}, {"category_id": 15, "poly": [129.0, 1948.0, 841.0, 1950.0, 841.0, 1989.0, 129.0, 1987.0], "score": 0.98, "text": " plementation are similar to those given in [3]. When compared "}, {"category_id": 15, "poly": [132.0, 1989.0, 836.0, 1989.0, 836.0, 2021.0, 132.0, 2021.0], "score": 0.99, "text": "to other existing real-time stereo matching implementations,"}, {"category_id": 15, "poly": [1111.0, 1022.0, 1317.0, 1022.0, 1317.0, 1061.0, 1111.0, 1061.0], "score": 1.0, "text": "V. CONCLUSION"}, {"category_id": 15, "poly": [864.0, 484.0, 1564.0, 484.0, 1564.0, 516.0, 864.0, 516.0], "score": 0.99, "text": "Table II: A comparison of speed and accuracy for the imple-"}, {"category_id": 15, "poly": [864.0, 518.0, 1564.0, 518.0, 1564.0, 550.0, 864.0, 550.0], "score": 0.99, "text": "mentations of many leading real-time stereo matching meth-"}, {"category_id": 15, "poly": [862.0, 550.0, 917.0, 550.0, 917.0, 584.0, 862.0, 584.0], "score": 0.96, "text": "ods."}, {"category_id": 15, "poly": [864.0, 484.0, 1564.0, 484.0, 1564.0, 516.0, 864.0, 516.0], "score": 0.99, "text": "Table II: A comparison of speed and accuracy for the imple-"}, {"category_id": 15, "poly": [864.0, 518.0, 1564.0, 518.0, 1564.0, 550.0, 864.0, 550.0], "score": 0.99, "text": "mentations of many leading real-time stereo matching meth-"}, {"category_id": 15, "poly": [862.0, 550.0, 917.0, 550.0, 917.0, 584.0, 862.0, 584.0], "score": 0.96, "text": "ods."}], "page_info": {"page_no": 4, "height": 2200, "width": 1700}}, {"layout_dets": [{"category_id": 1, "poly": [134.58497619628906, 157.681884765625, 841.3460693359375, 157.681884765625, 841.3460693359375, 1666.27001953125, 134.58497619628906, 1666.27001953125], "score": 0.9999936819076538}, {"category_id": 15, "poly": [143.0, 163.0, 838.0, 163.0, 838.0, 192.0, 143.0, 192.0], "score": 0.97, "text": "[6] W. Yu, T. Chen, F. Franchetti, and J. C. Hoe, \u201cHigh performance stereo"}, {"category_id": 15, "poly": [182.0, 188.0, 838.0, 188.0, 838.0, 218.0, 182.0, 218.0], "score": 0.98, "text": "vision designed for massively data parallel platforms,\u2019 Circuits and"}, {"category_id": 15, "poly": [182.0, 213.0, 841.0, 213.0, 841.0, 245.0, 182.0, 245.0], "score": 0.98, "text": "Systems for Video Technology, IEEE Transactions on, vol. 20, pp. 1509"}, {"category_id": 15, "poly": [182.0, 238.0, 411.0, 238.0, 411.0, 268.0, 182.0, 268.0], "score": 0.98, "text": "-1519, November 2010."}, {"category_id": 15, "poly": [143.0, 264.0, 838.0, 264.0, 838.0, 293.0, 143.0, 293.0], "score": 0.99, "text": "[7] S. Mattoccia, M. Viti, and F. Ries, \u201cNear real-time fast bilateral stereo"}, {"category_id": 15, "poly": [182.0, 289.0, 838.0, 289.0, 838.0, 319.0, 182.0, 319.0], "score": 0.96, "text": "on the GPU in Computer Vision and Pattern Recognition Workshops"}, {"category_id": 15, "poly": [178.0, 307.0, 841.0, 309.0, 841.0, 348.0, 178.0, 346.0], "score": 0.95, "text": "(CVPRW), 2011 IEEE Computer Society Conference on,Ppp. 136 -143,"}, {"category_id": 15, "poly": [185.0, 339.0, 289.0, 339.0, 289.0, 364.0, 185.0, 364.0], "score": 0.98, "text": "June 2011."}, {"category_id": 15, "poly": [141.0, 362.0, 838.0, 362.0, 838.0, 392.0, 141.0, 392.0], "score": 0.98, "text": "[8] K. Zhang, J. Lu, Q. Yang, G. Lafruit, R. Lauwereins, and L. Van Gool,"}, {"category_id": 15, "poly": [182.0, 387.0, 838.0, 387.0, 838.0, 419.0, 182.0, 419.0], "score": 0.98, "text": "\"Real-time and accurate stereo: A scalable approach with bitwise fast"}, {"category_id": 15, "poly": [185.0, 412.0, 838.0, 412.0, 838.0, 445.0, 185.0, 445.0], "score": 0.97, "text": "voting on CUDA,\u201d Circuits and Systems for Video Technology, IEEE"}, {"category_id": 15, "poly": [182.0, 438.0, 656.0, 438.0, 656.0, 468.0, 182.0, 468.0], "score": 0.99, "text": "Transactions on, vol. 21, pp. 867 -878, July 2011."}, {"category_id": 15, "poly": [141.0, 463.0, 838.0, 463.0, 838.0, 493.0, 141.0, 493.0], "score": 0.96, "text": "[9] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, \u201cFast cost-"}, {"category_id": 15, "poly": [182.0, 488.0, 838.0, 488.0, 838.0, 518.0, 182.0, 518.0], "score": 0.98, "text": "volume filtering for visual correspondence and beyond,\" in Computer"}, {"category_id": 15, "poly": [180.0, 509.0, 841.0, 511.0, 841.0, 543.0, 180.0, 541.0], "score": 0.95, "text": "Vision and Pattern Recognition (CVPR), 20ll IEEE Conference on,"}, {"category_id": 15, "poly": [180.0, 536.0, 448.0, 534.0, 448.0, 566.0, 180.0, 568.0], "score": 0.99, "text": "Pp. 3017 -3024, June 2011."}, {"category_id": 15, "poly": [134.0, 561.0, 838.0, 561.0, 838.0, 591.0, 134.0, 591.0], "score": 0.99, "text": "[10] A. Hosni, C. Rhemann, M. Bleyer, and M. Gelautz, \u201cTemporally con-"}, {"category_id": 15, "poly": [180.0, 587.0, 836.0, 587.0, 836.0, 616.0, 180.0, 616.0], "score": 0.99, "text": " sistent disparity and optical flow via efficient spatio-temporal filtering,\""}, {"category_id": 15, "poly": [182.0, 612.0, 838.0, 612.0, 838.0, 642.0, 182.0, 642.0], "score": 0.97, "text": "in Advances in Image and Video Technology (Y.-S. Ho, ed.), vol. 7087"}, {"category_id": 15, "poly": [180.0, 632.0, 845.0, 632.0, 845.0, 671.0, 180.0, 671.0], "score": 0.88, "text": "of Lectureotes inComputer Science,pp.16517,Springererlin /"}, {"category_id": 15, "poly": [182.0, 660.0, 353.0, 660.0, 353.0, 692.0, 182.0, 692.0], "score": 1.0, "text": "Heidelberg, 2012."}, {"category_id": 15, "poly": [134.0, 685.0, 838.0, 685.0, 838.0, 717.0, 134.0, 717.0], "score": 0.98, "text": "[11] C. Tomasi and R. Manduchi, \u201cBilateral filtering for gray and color"}, {"category_id": 15, "poly": [182.0, 710.0, 838.0, 710.0, 838.0, 742.0, 182.0, 742.0], "score": 0.98, "text": "images,\u201d in Computer Vision, 1998. Sixth International Conference on,"}, {"category_id": 15, "poly": [180.0, 736.0, 411.0, 731.0, 411.0, 763.0, 181.0, 768.0], "score": 0.93, "text": "pPp. 839 -846, jan 1998."}, {"category_id": 15, "poly": [132.0, 761.0, 838.0, 761.0, 838.0, 791.0, 132.0, 791.0], "score": 0.97, "text": "[12] K. He, J. Sun, and X. Tang, \u201cGuided image filtering,\u201d\u2019 in Computer"}, {"category_id": 15, "poly": [180.0, 784.0, 838.0, 786.0, 838.0, 818.0, 180.0, 816.0], "score": 0.98, "text": "Vision - ECCV 2010, vol. 6311 of Lecture Notes in Computer Science,"}, {"category_id": 15, "poly": [180.0, 811.0, 607.0, 807.0, 608.0, 839.0, 180.0, 843.0], "score": 0.98, "text": "pp. 1-14, Springer Berlin / Heidelberg, 2010."}, {"category_id": 15, "poly": [129.0, 832.0, 839.0, 837.0, 838.0, 869.0, 129.0, 864.0], "score": 0.98, "text": "[13] L. Zhang, B. Curless, and S. M. Seitz, \u201cSpacetime stereo: Shape"}, {"category_id": 15, "poly": [182.0, 862.0, 836.0, 862.0, 836.0, 891.0, 182.0, 891.0], "score": 0.98, "text": "recovery for dynamic scenes,\u201d in IEEE Computer Society Conference"}, {"category_id": 15, "poly": [182.0, 885.0, 834.0, 885.0, 834.0, 917.0, 182.0, 917.0], "score": 0.97, "text": "on Computer Vision and Pattern Recognition, pp. 367-374, June 2003."}, {"category_id": 15, "poly": [132.0, 910.0, 838.0, 910.0, 838.0, 940.0, 132.0, 940.0], "score": 0.98, "text": "[14] J. Davis, D. Nehab, R. Ramamoorthi, and S. Rusinkiewicz, \u201cSpacetime"}, {"category_id": 15, "poly": [182.0, 935.0, 838.0, 935.0, 838.0, 965.0, 182.0, 965.0], "score": 0.97, "text": "stereo: a unifying framework for depth from triangulation,\u201d\u2019 Pattern"}, {"category_id": 15, "poly": [182.0, 960.0, 838.0, 960.0, 838.0, 990.0, 182.0, 990.0], "score": 0.98, "text": "Analysis and Machine Intelligence, IEEE Transactions on,vol. 27,"}, {"category_id": 15, "poly": [180.0, 983.0, 462.0, 983.0, 462.0, 1015.0, 180.0, 1015.0], "score": 0.97, "text": "Pp. 296 -302, February 2005."}, {"category_id": 15, "poly": [132.0, 1011.0, 838.0, 1011.0, 838.0, 1040.0, 132.0, 1040.0], "score": 0.99, "text": "[15] E. Larsen, P. Mordohai, M. Pollefeys, and H. Fuchs, \u201cTemporally"}, {"category_id": 15, "poly": [182.0, 1036.0, 836.0, 1036.0, 836.0, 1066.0, 182.0, 1066.0], "score": 0.99, "text": "consistent reconstruction from multiple video streams using enhanced"}, {"category_id": 15, "poly": [178.0, 1054.0, 843.0, 1056.0, 843.0, 1095.0, 178.0, 1093.0], "score": 0.95, "text": "belief propagation in Computer Vision, 2007.ICCV 2007. IEEE1lth"}, {"category_id": 15, "poly": [180.0, 1082.0, 644.0, 1082.0, 644.0, 1121.0, 180.0, 1121.0], "score": 0.97, "text": "International Conference on, pp. 1 -8, oct. 2007."}, {"category_id": 15, "poly": [134.0, 1109.0, 838.0, 1109.0, 838.0, 1141.0, 134.0, 1141.0], "score": 0.97, "text": "[16] M. Bleyer, M. Gelautz, C. Rother, and C. Rhemann, \u201c\"A stereo approach"}, {"category_id": 15, "poly": [180.0, 1134.0, 838.0, 1134.0, 838.0, 1166.0, 180.0, 1166.0], "score": 0.99, "text": "that handles the mating problem via image warping\" in Computer"}, {"category_id": 15, "poly": [182.0, 1157.0, 838.0, 1157.0, 838.0, 1189.0, 182.0, 1189.0], "score": 0.98, "text": "Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference"}, {"category_id": 15, "poly": [180.0, 1183.0, 459.0, 1175.0, 460.0, 1212.0, 181.0, 1219.0], "score": 0.98, "text": "on, pp. 501 -508, June 2009."}, {"category_id": 15, "poly": [129.0, 1205.0, 838.0, 1208.0, 838.0, 1240.0, 129.0, 1237.0], "score": 0.98, "text": " [17] M. Sizintsev and R. Wildes, \u201cSpatiotemporal stereo via spatiotemporal"}, {"category_id": 15, "poly": [182.0, 1235.0, 838.0, 1235.0, 838.0, 1265.0, 182.0, 1265.0], "score": 0.97, "text": "quadric element (stequel) matching,\u201d in Computer Vision and Pattern"}, {"category_id": 15, "poly": [185.0, 1258.0, 841.0, 1258.0, 841.0, 1290.0, 185.0, 1290.0], "score": 0.98, "text": "Recognition, 2009. CVPR 2009. IEEE Conference on, Pp. 493 -500,"}, {"category_id": 15, "poly": [185.0, 1286.0, 286.0, 1286.0, 286.0, 1311.0, 185.0, 1311.0], "score": 0.99, "text": "june 2009."}, {"category_id": 15, "poly": [132.0, 1309.0, 838.0, 1309.0, 838.0, 1338.0, 132.0, 1338.0], "score": 0.97, "text": "[18] M. Sizintsev and R. Wildes, \u201cSpatiotemporal stereo and scene flow via"}, {"category_id": 15, "poly": [182.0, 1334.0, 841.0, 1334.0, 841.0, 1364.0, 182.0, 1364.0], "score": 0.97, "text": "stequel matching,\u201d\u2019Pattern Analysis and Machine Intelligence, IEEE"}, {"category_id": 15, "poly": [182.0, 1359.0, 684.0, 1359.0, 684.0, 1391.0, 182.0, 1391.0], "score": 1.0, "text": "Transactions on, vol. 34, pp. 1206 -1219, june 2012."}, {"category_id": 15, "poly": [132.0, 1382.0, 834.0, 1382.0, 834.0, 1412.0, 132.0, 1412.0], "score": 0.98, "text": "[19] C. Richardt, D. Orr, I. Davies, A. Criminisi, and N. A. Dodgson,"}, {"category_id": 15, "poly": [185.0, 1409.0, 838.0, 1409.0, 838.0, 1441.0, 185.0, 1441.0], "score": 0.98, "text": "\"Real-time spatiotemporal stereo matching using the dual-cross-bilateral"}, {"category_id": 15, "poly": [182.0, 1432.0, 838.0, 1432.0, 838.0, 1464.0, 182.0, 1464.0], "score": 0.95, "text": "grid,\" in Proceedings of the European Conference on Computer Vision"}, {"category_id": 15, "poly": [182.0, 1458.0, 838.0, 1458.0, 838.0, 1490.0, 182.0, 1490.0], "score": 0.98, "text": "(ECCV), Lecture Notes in Computer Science, pp. 510-523, September"}, {"category_id": 15, "poly": [182.0, 1477.0, 243.0, 1483.0, 241.0, 1511.0, 179.0, 1505.0], "score": 1.0, "text": "2010."}, {"category_id": 15, "poly": [134.0, 1508.0, 836.0, 1508.0, 836.0, 1538.0, 134.0, 1538.0], "score": 0.98, "text": "[20] S. Paris and F. Durand, \u201cA fast approximation of the bilateral filter using"}, {"category_id": 15, "poly": [182.0, 1533.0, 836.0, 1533.0, 836.0, 1565.0, 182.0, 1565.0], "score": 0.98, "text": "a signal processing approach,\u201d Int. J. Comput. Vision, vol. 81, pp. 24-52,"}, {"category_id": 15, "poly": [185.0, 1561.0, 282.0, 1561.0, 282.0, 1586.0, 185.0, 1586.0], "score": 0.98, "text": "Jan. 2009."}, {"category_id": 15, "poly": [134.0, 1584.0, 836.0, 1584.0, 836.0, 1613.0, 134.0, 1613.0], "score": 0.98, "text": "[21] Q. Yang, L. Wang, R. Yang, S. Wang, M. Liao, and D. Nist\u00e9r, \u201cReal-"}, {"category_id": 15, "poly": [182.0, 1609.0, 838.0, 1609.0, 838.0, 1641.0, 182.0, 1641.0], "score": 0.98, "text": "time global stereo matching using hierarchical belief propagation.\u201d in"}, {"category_id": 15, "poly": [182.0, 1634.0, 698.0, 1634.0, 698.0, 1666.0, 182.0, 1666.0], "score": 1.0, "text": "British Machine Vision Conference, pp. 989-998, 2006."}], "page_info": {"page_no": 5, "height": 2200, "width": 1700}}]
\ No newline at end of file
diff --git a/demo/magic_pdf_parse_main.py b/demo/magic_pdf_parse_main.py
deleted file mode 100644
index 0e5b93313..000000000
--- a/demo/magic_pdf_parse_main.py
+++ /dev/null
@@ -1,141 +0,0 @@
-import os
-import json
-import copy
-
-from loguru import logger
-
-from magic_pdf.libs.draw_bbox import draw_layout_bbox, draw_span_bbox
-from magic_pdf.pipe.UNIPipe import UNIPipe
-from magic_pdf.pipe.OCRPipe import OCRPipe
-from magic_pdf.pipe.TXTPipe import TXTPipe
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-
-
-# todo: 设备类型选择 （？）
-
-def json_md_dump(
- pipe,
- md_writer,
- pdf_name,
- content_list,
- md_content,
-):
- # 写入模型结果到 model.json
- orig_model_list = copy.deepcopy(pipe.model_list)
- md_writer.write(
- content=json.dumps(orig_model_list, ensure_ascii=False, indent=4),
- path=f"{pdf_name}_model.json"
- )
-
- # 写入中间结果到 middle.json
- md_writer.write(
- content=json.dumps(pipe.pdf_mid_data, ensure_ascii=False, indent=4),
- path=f"{pdf_name}_middle.json"
- )
-
- # text文本结果写入到 conent_list.json
- md_writer.write(
- content=json.dumps(content_list, ensure_ascii=False, indent=4),
- path=f"{pdf_name}_content_list.json"
- )
-
- # 写入结果到 .md 文件中
- md_writer.write(
- content=md_content,
- path=f"{pdf_name}.md"
- )
-
-
-# 可视化
-def draw_visualization_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name):
- # 画布局框，附带排序结果
- draw_layout_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name)
- # 画 span 框
- draw_span_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name)
-
-
-def pdf_parse_main(
- pdf_path: str,
- parse_method: str = 'auto',
- model_json_path: str = None,
- is_json_md_dump: bool = True,
- is_draw_visualization_bbox: bool = True,
- output_dir: str = None
-):
- """
- 执行从 pdf 转换到 json、md 的过程，输出 md 和 json 文件到 pdf 文件所在的目录
-
- :param pdf_path: .pdf 文件的路径，可以是相对路径，也可以是绝对路径
- :param parse_method: 解析方法， 共 auto、ocr、txt 三种，默认 auto，如果效果不好，可以尝试 ocr
- :param model_json_path: 已经存在的模型数据文件，如果为空则使用内置模型，pdf 和 model_json 务必对应
- :param is_json_md_dump: 是否将解析后的数据写入到 .json 和 .md 文件中，默认 True，会将不同阶段的数据写入到不同的 .json 文件中（共3个.json文件），md内容会保存到 .md 文件中
- :param output_dir: 输出结果的目录地址，会生成一个以 pdf 文件名命名的文件夹并保存所有结果
- """
- try:
- pdf_name = os.path.basename(pdf_path).split(".")[0]
- pdf_path_parent = os.path.dirname(pdf_path)
-
- if output_dir:
- output_path = os.path.join(output_dir, pdf_name)
- else:
- output_path = os.path.join(pdf_path_parent, pdf_name)
-
- output_image_path = os.path.join(output_path, 'images')
-
- # 获取图片的父路径，为的是以相对路径保存到 .md 和 conent_list.json 文件中
- image_path_parent = os.path.basename(output_image_path)
-
- pdf_bytes = open(pdf_path, "rb").read() # 读取 pdf 文件的二进制数据
-
- if model_json_path:
- # 读取已经被模型解析后的pdf文件的 json 原始数据，list 类型
- model_json = json.loads(open(model_json_path, "r", encoding="utf-8").read())
- else:
- model_json = []
-
- # 执行解析步骤
- # image_writer = DiskReaderWriter(output_image_path)
- image_writer, md_writer = DiskReaderWriter(output_image_path), DiskReaderWriter(output_path)
-
- # 选择解析方式
- # jso_useful_key = {"_pdf_type": "", "model_list": model_json}
- # pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
- if parse_method == "auto":
- jso_useful_key = {"_pdf_type": "", "model_list": model_json}
- pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
- elif parse_method == "txt":
- pipe = TXTPipe(pdf_bytes, model_json, image_writer)
- elif parse_method == "ocr":
- pipe = OCRPipe(pdf_bytes, model_json, image_writer)
- else:
- logger.error("unknown parse method, only auto, ocr, txt allowed")
- exit(1)
-
- # 执行分类
- pipe.pipe_classify()
-
- # 如果没有传入模型数据，则使用内置模型解析
- if not model_json:
- pipe.pipe_analyze() # 解析
-
- # 执行解析
- pipe.pipe_parse()
-
- # 保存 text 和 md 格式的结果
- content_list = pipe.pipe_mk_uni_format(image_path_parent, drop_mode="none")
- md_content = pipe.pipe_mk_markdown(image_path_parent, drop_mode="none")
-
- if is_json_md_dump:
- json_md_dump(pipe, md_writer, pdf_name, content_list, md_content)
-
- if is_draw_visualization_bbox:
- draw_visualization_bbox(pipe.pdf_mid_data['pdf_info'], pdf_bytes, output_path, pdf_name)
-
- except Exception as e:
- logger.exception(e)
-
-
-# 测试
-if __name__ == '__main__':
- pdf_path = r"D:\project\20240617magicpdf\Magic-PDF\demo\demo1.pdf"
- pdf_parse_main(pdf_path)
diff --git a/demo/office_docs/docx_01.docx b/demo/office_docs/docx_01.docx
new file mode 100644
index 000000000..2369ea1e8
Binary files /dev/null and b/demo/office_docs/docx_01.docx differ
diff --git a/demo/office_docs/pptx_01.pptx b/demo/office_docs/pptx_01.pptx
new file mode 100644
index 000000000..d515e5f40
Binary files /dev/null and b/demo/office_docs/pptx_01.pptx differ
diff --git a/demo/office_docs/xlsx_01.xlsx b/demo/office_docs/xlsx_01.xlsx
new file mode 100644
index 000000000..5a0475f8b
Binary files /dev/null and b/demo/office_docs/xlsx_01.xlsx differ
diff --git a/demo/demo1.pdf b/demo/pdfs/demo1.pdf
similarity index 100%
rename from demo/demo1.pdf
rename to demo/pdfs/demo1.pdf
diff --git a/demo/demo2.pdf b/demo/pdfs/demo2.pdf
similarity index 100%
rename from demo/demo2.pdf
rename to demo/pdfs/demo2.pdf
diff --git a/demo/pdfs/demo3.pdf b/demo/pdfs/demo3.pdf
new file mode 100644
index 000000000..75f4e0472
Binary files /dev/null and b/demo/pdfs/demo3.pdf differ
diff --git a/demo/small_ocr.pdf b/demo/pdfs/small_ocr.pdf
similarity index 100%
rename from demo/small_ocr.pdf
rename to demo/pdfs/small_ocr.pdf
diff --git a/docker/china/Dockerfile b/docker/china/Dockerfile
new file mode 100644
index 000000000..2be8b628c
--- /dev/null
+++ b/docker/china/Dockerfile
@@ -0,0 +1,25 @@
+# Use DaoCloud mirrored vllm image for China region for gpu with Volta、Turing、Ampere、Ada Lovelace、Hopper、Blackwell architecture (7.0 <= Compute Capability <= 12.0)
+# Compute Capability version query (https://developer.nvidia.com/cuda-gpus)
+# support x86_64 architecture and ARM(AArch64) architecture
+FROM docker.m.daocloud.io/vllm/vllm-openai:v0.11.2
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U 'mineru[core]>=3.0.0' -i https://mirrors.aliyun.com/pypi/simple --break-system-packages && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/corex.Dockerfile b/docker/china/corex.Dockerfile
new file mode 100644
index 000000000..8e1ba13bd
--- /dev/null
+++ b/docker/china/corex.Dockerfile
@@ -0,0 +1,27 @@
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + iluvatar GPU.
+FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/corex:4.4.0_torch2.7.1_vllm0.11.2_py3.10
+
+
+# Install Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install 'mineru[core]>=3.0.0' \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/dcu.Dockerfile b/docker/china/dcu.Dockerfile
new file mode 100644
index 000000000..8eb334084
--- /dev/null
+++ b/docker/china/dcu.Dockerfile
@@ -0,0 +1,31 @@
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + Hygon DCU.
+FROM harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226
+
+
+# Install Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install "mineru[gradio]>=3.0.0" \
+ "ftfy>=6.3.1,<7" \
+ "shapely>=2.0.7,<3" \
+ "pyclipper>=1.3.0,<2" \
+ "omegaconf>=2.3.0,<3" \
+ numpy==1.25.0 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/gcu.Dockerfile b/docker/china/gcu.Dockerfile
new file mode 100644
index 000000000..bcfc74b69
--- /dev/null
+++ b/docker/china/gcu.Dockerfile
@@ -0,0 +1,30 @@
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + Enflame GCU.
+FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/gcu:docker_images_topsrider_i3x_3.6.20260106_vllm0.11_pytorch2.8.0
+
+
+# Install Noto fonts for Chinese characters
+RUN echo 'deb http://mirrors.aliyun.com/ubuntu/ noble main restricted universe multiverse\n\
+deb http://mirrors.aliyun.com/ubuntu/ noble-updates main restricted universe multiverse\n\
+deb http://mirrors.aliyun.com/ubuntu/ noble-backports main restricted universe multiverse\n\
+deb http://mirrors.aliyun.com/ubuntu/ noble-security main restricted universe multiverse' > /tmp/aliyun-sources.list && \
+ apt-get -o Dir::Etc::SourceList=/tmp/aliyun-sources.list update && \
+ apt-get -o Dir::Etc::SourceList=/tmp/aliyun-sources.list install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/* /tmp/aliyun-sources.list
+
+# Install mineru latest
+RUN python3 -m pip install "mineru[core]>=3.0.0" \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/kxpu.Dockerfile b/docker/china/kxpu.Dockerfile
new file mode 100644
index 000000000..3559445e5
--- /dev/null
+++ b/docker/china/kxpu.Dockerfile
@@ -0,0 +1,30 @@
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + Kunlun XPU.
+FROM docker.1ms.run/wjie520/vllm_kunlun:v0.10.1.1rc1
+
+
+# Install Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install "mineru[gradio]>=3.0.0" \
+ "ftfy>=6.3.1,<7" \
+ "shapely>=2.0.7,<3" \
+ "pyclipper>=1.3.0,<2" \
+ "omegaconf>=2.3.0,<3" \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ sed -i '1,200{s/self\.act = act_layer()/self.act = nn.GELU()/;t;b};' /root/miniconda/envs/vllm_kunlun_0.10.1.1/lib/python3.10/site-packages/vllm_kunlun/models/qwen2_vl.py && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/maca.Dockerfile b/docker/china/maca.Dockerfile
new file mode 100644
index 000000000..827f18c6c
--- /dev/null
+++ b/docker/china/maca.Dockerfile
@@ -0,0 +1,34 @@
+# 基础镜像配置 vLLM 或 LMDeploy 推理环境，请根据实际需要选择其中一个，要求 amd64(x86-64) CPU + metax GPU。
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + metax GPU.
+FROM cr.metax-tech.com/public-ai-release/maca/vllm:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-amd64
+# Base image containing the LMDeploy inference environment, requiring amd64(x86-64) CPU + metax GPU.
+# FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/maca:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-lmdeploy0.10.2-amd64
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# mod torchvision to be compatible with torch 2.6
+RUN sed -i '3s/^Version: 0.15.1+metax3\.1\.0\.4$/Version: 0.21.0+metax3.1.0.4/' /opt/conda/lib/python3.10/site-packages/torchvision-0.15.1+metax3.1.0.4.dist-info/METADATA && \
+ mv /opt/conda/lib/python3.10/site-packages/torchvision-0.15.1+metax3.1.0.4.dist-info /opt/conda/lib/python3.10/site-packages/torchvision-0.21.0+metax3.1.0.4.dist-info
+
+# Install mineru latest
+RUN /opt/conda/bin/python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ /opt/conda/bin/python3 -m pip install 'mineru[core]>=3.0.0' \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ /opt/conda/bin/python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "/opt/conda/bin/mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/mlu.Dockerfile b/docker/china/mlu.Dockerfile
new file mode 100644
index 000000000..bd889a916
--- /dev/null
+++ b/docker/china/mlu.Dockerfile
@@ -0,0 +1,42 @@
+# 基础镜像配置 vLLM 或 LMDeploy ，请根据实际需要选择其中一个，要求 amd64(x86-64) CPU + Cambricon MLU.
+# Base image containing the LMDEPLOY inference environment, requiring amd64(x86-64) CPU + Cambricon MLU.
+FROM crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/camb:mineru25
+ARG BACKEND=lmdeploy
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + Cambricon MLU.
+# FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/mlu:vllm0.8.3-torch2.6.0-torchmlu1.26.1-ubuntu22.04-py310
+# ARG BACKEND=vllm
+
+# Install Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN /bin/bash -c '\
+ if [ "$BACKEND" = "vllm" ]; then \
+ source /torch/venv3/pytorch_infer/bin/activate; \
+ fi && \
+ python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install "mineru[core]>=3.0.0" \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install $(if [ "$BACKEND" = "lmdeploy" ]; then echo "accelerate==1.2.0"; else echo "transformers==4.50.3"; fi) && \
+ python3 -m pip cache purge'
+
+# Download models and update the configuration file
+RUN /bin/bash -c '\
+ if [ "$BACKEND" = "vllm" ]; then \
+ source /torch/venv3/pytorch_infer/bin/activate; \
+ fi && \
+ mineru-models-download -s modelscope -m all'
+
+WORKDIR /workspace
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
diff --git a/docker/china/musa.Dockerfile b/docker/china/musa.Dockerfile
new file mode 100644
index 000000000..3bfb63a8e
--- /dev/null
+++ b/docker/china/musa.Dockerfile
@@ -0,0 +1,35 @@
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + MooreThreads GPU.
+FROM registry.mthreads.com/mcconline/vllm-musa-qy2-py310:v0.8.4-release
+
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ git clone https://gitcode.com/gh_mirrors/vi/vision.git -b v0.20.0 --depth 1 && \
+ cd vision && \
+ python3 setup.py install && \
+ python3 -m pip install "mineru[gradio]>=3.0.0" \
+ "ftfy>=6.3.1,<7" \
+ "shapely>=2.0.7,<3" \
+ "pyclipper>=1.3.0,<2" \
+ "omegaconf>=2.3.0,<3" \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/npu.Dockerfile b/docker/china/npu.Dockerfile
new file mode 100644
index 000000000..7eb2ff691
--- /dev/null
+++ b/docker/china/npu.Dockerfile
@@ -0,0 +1,32 @@
+# 基础镜像配置 vLLM 或 LMDeploy ，请根据实际需要选择其中一个，要求 ARM(AArch64) CPU + Ascend NPU。
+# Base image containing the vLLM inference environment, requiring ARM(AArch64) CPU + Ascend NPU.
+FROM quay.m.daocloud.io/ascend/vllm-ascend:v0.11.0
+# Base image containing the LMDeploy inference environment, requiring ARM(AArch64) CPU + Ascend NPU.
+# FROM crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/ascend:mineru-a2
+
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 \
+ libglib2.0-0 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install 'mineru[core]>=3.0.0' \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN TORCH_DEVICE_BACKEND_AUTOLOAD=0 /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/china/ppu.Dockerfile b/docker/china/ppu.Dockerfile
new file mode 100644
index 000000000..5a8ce3251
--- /dev/null
+++ b/docker/china/ppu.Dockerfile
@@ -0,0 +1,30 @@
+# 基础镜像配置 vLLM 或 LMDeploy 推理环境，请根据实际需要选择其中一个，要求 amd64(x86-64) CPU + t-head PPU。
+# Base image containing the vLLM inference environment, requiring amd64(x86-64) CPU + t-head PPU.
+FROM crpi-vofi3w62lkohhxsp.cn-shanghai.personal.cr.aliyuncs.com/opendatalab-mineru/ppu:ppu-pytorch2.6.0-ubuntu24.04-cuda12.6-vllm0.8.5-py312
+# Base image containing the LMDeploy inference environment, requiring amd64(x86-64) CPU + t-head PPU.
+# FROM crpi-4crprmm5baj1v8iv.cn-hangzhou.personal.cr.aliyuncs.com/lmdeploy_dlinfer/ppu:mineru-ppu
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U pip -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip install 'mineru[core]>=3.0.0' \
+ numpy==1.26.4 \
+ opencv-python==4.11.0.86 \
+ -i https://mirrors.aliyun.com/pypi/simple && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s modelscope -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docker/compose.yaml b/docker/compose.yaml
new file mode 100644
index 000000000..54171de58
--- /dev/null
+++ b/docker/compose.yaml
@@ -0,0 +1,120 @@
+services:
+ mineru-openai-server:
+ image: mineru:latest
+ container_name: mineru-openai-server
+ restart: always
+ profiles: ["openai-server"]
+ ports:
+ - 30000:30000
+ environment:
+ MINERU_MODEL_SOURCE: local
+ entrypoint: mineru-openai-server
+ command:
+ --host 0.0.0.0
+ --port 30000
+ # --gpu-memory-utilization 0.5 # If encountering VRAM shortage, reduce the KV cache size by this parameter; if VRAM issues persist, try lowering it further to `0.4` or below.
+ ulimits:
+ memlock: -1
+ stack: 67108864
+ ipc: host
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:30000/health || exit 1"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["0"] # Modify for multiple GPUs: ["0", "1"]
+ capabilities: [gpu]
+
+ mineru-api:
+ image: mineru:latest
+ container_name: mineru-api
+ restart: always
+ profiles: ["api"]
+ ports:
+ - 8000:8000
+ environment:
+ MINERU_MODEL_SOURCE: local
+ entrypoint: mineru-api
+ command:
+ --host 0.0.0.0
+ --port 8000
+ # parameters for vllm-engine
+ # --gpu-memory-utilization 0.5 # If encountering VRAM shortage, reduce the KV cache size by this parameter; if VRAM issues persist, try lowering it further to `0.4` or below.
+ ulimits:
+ memlock: -1
+ stack: 67108864
+ ipc: host
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["0"] # Modify for multiple GPUs: ["0", "1"]
+ capabilities: [gpu]
+
+ mineru-router:
+ image: mineru:latest
+ container_name: mineru-router
+ restart: always
+ profiles: ["router"]
+ ports:
+ - 8002:8002
+ environment:
+ MINERU_MODEL_SOURCE: local
+ entrypoint: mineru-router
+ command:
+ --host 0.0.0.0
+ --port 8002
+ --local-gpus auto
+ # To aggregate existing mineru-api services instead of starting local workers:
+ # --local-gpus none
+ # --upstream-url http://mineru-api:8000
+ # --upstream-url http://mineru-api-2:8000
+ # parameters for vllm-engine
+ # --gpu-memory-utilization 0.5 # If encountering VRAM shortage, reduce the KV cache size by this parameter; if VRAM issues persist, try lowering it further to `0.4` or below.
+ ulimits:
+ memlock: -1
+ stack: 67108864
+ ipc: host
+ healthcheck:
+ test: ["CMD-SHELL", "curl -f http://localhost:8002/health || exit 1"]
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["0"] # Modify for multiple GPUs: ["0", "1"]
+ capabilities: [gpu]
+
+ mineru-gradio:
+ image: mineru:latest
+ container_name: mineru-gradio
+ restart: always
+ profiles: ["gradio"]
+ ports:
+ - 7860:7860
+ environment:
+ MINERU_MODEL_SOURCE: local
+ entrypoint: mineru-gradio
+ command:
+ --server-name 0.0.0.0
+ --server-port 7860
+ # --enable-api false # If you want to disable the API, set this to false
+ # --max-convert-pages 20 # If you want to limit the number of pages for conversion, set this to a specific number
+ # parameters for vllm-engine
+ # --gpu-memory-utilization 0.5 # If encountering VRAM shortage, reduce the KV cache size by this parameter; if VRAM issues persist, try lowering it further to `0.4` or below.
+ ulimits:
+ memlock: -1
+ stack: 67108864
+ ipc: host
+ deploy:
+ resources:
+ reservations:
+ devices:
+ - driver: nvidia
+ device_ids: ["0"] # Modify for multiple GPUs: ["0", "1"]
+ capabilities: [gpu]
diff --git a/docker/global/Dockerfile b/docker/global/Dockerfile
new file mode 100644
index 000000000..5e84399df
--- /dev/null
+++ b/docker/global/Dockerfile
@@ -0,0 +1,25 @@
+# Use the official vllm image for gpu with Volta、Turing、Ampere、Ada Lovelace、Hopper、Blackwell architecture (7.0 <= Compute Capability <= 12.0)
+# Compute Capability version query (https://developer.nvidia.com/cuda-gpus)
+# support x86_64 architecture and ARM(AArch64) architecture
+FROM vllm/vllm-openai:v0.11.2
+
+# Install libgl for opencv support & Noto fonts for Chinese characters
+RUN apt-get update && \
+ apt-get install -y \
+ fonts-noto-core \
+ fonts-noto-cjk \
+ fontconfig \
+ libgl1 && \
+ fc-cache -fv && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+# Install mineru latest
+RUN python3 -m pip install -U 'mineru[core]>=3.0.0' --break-system-packages && \
+ python3 -m pip cache purge
+
+# Download models and update the configuration file
+RUN /bin/bash -c "mineru-models-download -s huggingface -m all"
+
+# Set the entry point to activate the virtual environment and run the command line tool
+ENTRYPOINT ["/bin/bash", "-c", "export MINERU_MODEL_SOURCE=local && exec \"$@\"", "--"]
\ No newline at end of file
diff --git a/docs/FAQ_en_us.md b/docs/FAQ_en_us.md
deleted file mode 100644
index 21ca861b6..000000000
--- a/docs/FAQ_en_us.md
+++ /dev/null
@@ -1,59 +0,0 @@
-# Frequently Asked Questions
-
-### 1. When using the command `pip install magic-pdf[full]` on newer versions of macOS, the error `zsh: no matches found: magic-pdf[full]` occurs.
-
-On macOS, the default shell has switched from Bash to Z shell, which has special handling logic for certain types of string matching. This can lead to the "no matches found" error. You can try disabling the globbing feature in the command line and then run the installation command again.
-
-```bash
-setopt no_nomatch
-pip install magic-pdf[full]
-```
-
-### 2. Encountering the error `pickle.UnpicklingError: invalid load key, 'v'.` during use
-
-This might be due to an incomplete download of the model file. You can try re-downloading the model file and then try again.
-Reference: https://github.com/opendatalab/MinerU/issues/143
-
-### 3. Where should the model files be downloaded and how should the `/models-dir` configuration be set?
-
-The path for the model files is configured in "magic-pdf.json". just like:
-
-```json
-{
- "models-dir": "/tmp/models"
-}
-```
-
-This path is an absolute path, not a relative path. You can obtain the absolute path in the models directory using the "pwd" command.
-Reference: https://github.com/opendatalab/MinerU/issues/155#issuecomment-2230216874
-
-### 4. Encountered the error `ImportError: libGL.so.1: cannot open shared object file: No such file or directory` in Ubuntu 22.04 on WSL2
-
-The `libgl` library is missing in Ubuntu 22.04 on WSL2. You can install the `libgl` library with the following command to resolve the issue:
-
-```bash
-sudo apt-get install libgl1-mesa-glx
-```
-
-Reference: https://github.com/opendatalab/MinerU/issues/388
-
-### 5. Encountered error `ModuleNotFoundError: No module named 'fairscale'`
-
-You need to uninstall the module and reinstall it:
-
-```bash
-pip uninstall fairscale
-pip install fairscale
-```
-
-Reference: https://github.com/opendatalab/MinerU/issues/411
-
-### 6. On some newer devices like the H100, the text parsed during OCR using CUDA acceleration is garbled.
-
-The compatibility of cuda11 with new graphics cards is poor, and the CUDA version used by Paddle needs to be upgraded.
-
-```bash
-pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
-```
-
-Reference: https://github.com/opendatalab/MinerU/issues/558
diff --git a/docs/FAQ_zh_cn.md b/docs/FAQ_zh_cn.md
deleted file mode 100644
index 667abfa82..000000000
--- a/docs/FAQ_zh_cn.md
+++ /dev/null
@@ -1,61 +0,0 @@
-# 常见问题解答
-
-### 1.在较新版本的mac上使用命令安装pip install magic-pdf\[full\] zsh: no matches found: magic-pdf\[full\]
-
-在 macOS 上，默认的 shell 从 Bash 切换到了 Z shell，而 Z shell 对于某些类型的字符串匹配有特殊的处理逻辑，这可能导致no matches found错误。
-可以通过在命令行禁用globbing特性，再尝试运行安装命令
-
-```bash
-setopt no_nomatch
-pip install magic-pdf[full]
-```
-
-### 2.使用过程中遇到_pickle.UnpicklingError: invalid load key, 'v'.错误
-
-可能是由于模型文件未下载完整导致，可尝试重新下载模型文件后再试
-参考：https://github.com/opendatalab/MinerU/issues/143
-
-### 3.模型文件应该下载到哪里/models-dir的配置应该怎么填
-
-模型文件的路径输入是在"magic-pdf.json"中通过
-
-```json
-{
- "models-dir": "/tmp/models"
-}
-```
-
-进行配置的。
-这个路径是绝对路径而不是相对路径，绝对路径的获取可在models目录中通过命令 "pwd" 获取。
-参考：https://github.com/opendatalab/MinerU/issues/155#issuecomment-2230216874
-
-### 4.在WSL2的Ubuntu22.04中遇到报错`ImportError: libGL.so.1: cannot open shared object file: No such file or directory`
-
-WSL2的Ubuntu22.04中缺少`libgl`库，可通过以下命令安装`libgl`库解决：
-
-```bash
-sudo apt-get install libgl1-mesa-glx
-```
-
-参考：https://github.com/opendatalab/MinerU/issues/388
-
-### 5.遇到报错 `ModuleNotFoundError : Nomodulenamed 'fairscale'`
-
-需要卸载该模块并重新安装
-
-```bash
-pip uninstall fairscale
-pip install fairscale
-```
-
-参考：https://github.com/opendatalab/MinerU/issues/411
-
-### 6.在部分较新的设备如H100上，使用CUDA加速OCR时解析出的文字乱码。
-
-cuda11对新显卡的兼容性不好，需要升级paddle使用的cuda版本
-
-```bash
-pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu123/
-```
-
-参考：https://github.com/opendatalab/MinerU/issues/558
diff --git a/docs/README_Ubuntu_CUDA_Acceleration_en_US.md b/docs/README_Ubuntu_CUDA_Acceleration_en_US.md
deleted file mode 100644
index 010cc7294..000000000
--- a/docs/README_Ubuntu_CUDA_Acceleration_en_US.md
+++ /dev/null
@@ -1,120 +0,0 @@
-# Ubuntu 22.04 LTS
-
-### 1. Check if NVIDIA Drivers Are Installed
-
-```sh
-nvidia-smi
-```
-
-If you see information similar to the following, it means that the NVIDIA drivers are already installed, and you can skip Step 2.
-
-Notice:`CUDA Version` should be >= 12.1, If the displayed version number is less than 12.1, please upgrade the driver.
-
-```plaintext
-+---------------------------------------------------------------------------------------+
-| NVIDIA-SMI 537.34 Driver Version: 537.34 CUDA Version: 12.2 |
-|-----------------------------------------+----------------------+----------------------+
-| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
-| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
-| | | MIG M. |
-|=========================================+======================+======================|
-| 0 NVIDIA GeForce RTX 3060 Ti WDDM | 00000000:01:00.0 On | N/A |
-| 0% 51C P8 12W / 200W | 1489MiB / 8192MiB | 5% Default |
-| | | N/A |
-+-----------------------------------------+----------------------+----------------------+
-```
-
-### 2. Install the Driver
-
-If no driver is installed, use the following command:
-
-```sh
-sudo apt-get update
-sudo apt-get install nvidia-driver-545
-```
-
-Install the proprietary driver and restart your computer after installation.
-
-```sh
-reboot
-```
-
-### 3. Install Anaconda
-
-If Anaconda is already installed, skip this step.
-
-```sh
-wget https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh
-bash Anaconda3-2024.06-1-Linux-x86_64.sh
-```
-
-In the final step, enter `yes`, close the terminal, and reopen it.
-
-### 4. Create an Environment Using Conda
-
-Specify Python version 3.10.
-
-```sh
-conda create -n MinerU python=3.10
-conda activate MinerU
-```
-
-### 5. Install Applications
-
-```sh
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com
-```
-
-❗ After installation, make sure to check the version of `magic-pdf` using the following command:
-
-```sh
-magic-pdf --version
-```
-
-If the version number is less than 0.7.0, please report the issue.
-
-### 6. Download Models
-
-Refer to detailed instructions on [how to download model files](how_to_download_models_en.md).
-
-## 7. Understand the Location of the Configuration File
-
-After completing the [6. Download Models](#6-download-models) step, the script will automatically generate a `magic-pdf.json` file in the user directory and configure the default model path.
-You can find the `magic-pdf.json` file in your user directory.
-
-> The user directory for Linux is "/home/username".
-
-### 8. First Run
-
-Download a sample file from the repository and test it.
-
-```sh
-wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf
-magic-pdf -p small_ocr.pdf
-```
-
-### 9. Test CUDA Acceleration
-
-If your graphics card has at least **8GB** of VRAM, follow these steps to test CUDA acceleration:
-
-1. Modify the value of `"device-mode"` in the `magic-pdf.json` configuration file located in your home directory.
- ```json
- {
- "device-mode": "cuda"
- }
- ```
-2. Test CUDA acceleration with the following command:
- ```sh
- magic-pdf -p small_ocr.pdf
- ```
-
-### 10. Enable CUDA Acceleration for OCR
-
-1. Download `paddlepaddle-gpu`. Installation will automatically enable OCR acceleration.
- ```sh
- python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
- ```
-2. Test OCR acceleration with the following command:
- ```sh
- magic-pdf -p small_ocr.pdf
- ```
diff --git a/docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md b/docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md
deleted file mode 100644
index f72ffec87..000000000
--- a/docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md
+++ /dev/null
@@ -1,131 +0,0 @@
-# Ubuntu 22.04 LTS
-
-## 1. 检测是否已安装nvidia驱动
-
-```bash
-nvidia-smi
-```
-
-如果看到类似如下的信息，说明已经安装了nvidia驱动，可以跳过步骤2
-
-注意:`CUDA Version` 显示的版本号应 >= 12.1，如显示的版本号小于12.1，请升级驱动
-
-```plaintext
-```
-+---------------------------------------------------------------------------------------+
-| NVIDIA-SMI 537.34 Driver Version: 537.34 CUDA Version: 12.2 |
-|-----------------------------------------+----------------------+----------------------+
-| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
-| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
-| | | MIG M. |
-|=========================================+======================+======================|
-| 0 NVIDIA GeForce RTX 3060 Ti WDDM | 00000000:01:00.0 On | N/A |
-| 0% 51C P8 12W / 200W | 1489MiB / 8192MiB | 5% Default |
-| | | N/A |
-+-----------------------------------------+----------------------+----------------------+
-```
-
-## 2. 安装驱动
-
-如没有驱动，则通过如下命令
-
-```bash
-sudo apt-get update
-sudo apt-get install nvidia-driver-545
-```
-
-安装专有驱动，安装完成后，重启电脑
-
-```bash
-reboot
-```
-
-## 3. 安装anacoda
-
-如果已安装conda，可以跳过本步骤
-
-```bash
-wget -U NoSuchBrowser/1.0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-2024.06-1-Linux-x86_64.sh
-bash Anaconda3-2024.06-1-Linux-x86_64.sh
-```
-
-最后一步输入yes，关闭终端重新打开
-
-## 4. 使用conda 创建环境
-
-需指定python版本为3.10
-
-```bash
-conda create -n MinerU python=3.10
-conda activate MinerU
-```
-
-## 5. 安装应用
-
-```bash
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
-
-> ❗️下载完成后，务必通过以下命令确认magic-pdf的版本是否正确
->
-> ```bash
-> magic-pdf --version
-> ```
->
-> 如果版本号小于0.7.0，请到issue中向我们反馈
-
-## 6. 下载模型
-
-详细参考 [如何下载模型文件](how_to_download_models_zh_cn.md)
-
-## 7. 了解配置文件存放的位置
-
-完成[6.下载模型](#6-下载模型)步骤后，脚本会自动生成用户目录下的magic-pdf.json文件，并自动配置默认模型路径。
-您可在【用户目录】下找到magic-pdf.json文件。
-
-> linux用户目录为 "/home/用户名"
-
-## 8. 第一次运行
-
-从仓库中下载样本文件，并测试
-
-```bash
-wget https://gitee.com/myhloli/MinerU/raw/master/demo/small_ocr.pdf
-magic-pdf -p small_ocr.pdf
-```
-
-## 9. 测试CUDA加速
-
-如果您的显卡显存大于等于 **8GB** ，可以进行以下流程，测试CUDA解析加速效果
-
-**1.修改【用户目录】中配置文件magic-pdf.json中"device-mode"的值**
-
-```json
-{
- "device-mode":"cuda"
-}
-```
-
-**2.运行以下命令测试cuda加速效果**
-
-```bash
-magic-pdf -p small_ocr.pdf
-```
-
-> 提示：CUDA加速是否生效可以根据log中输出的各个阶段cost耗时来简单判断，通常情况下，`layout detection cost` 和 `mfr time` 应提速10倍以上。
-
-## 10. 为ocr开启cuda加速
-
-**1.下载paddlepaddle-gpu, 安装完成后会自动开启ocr加速**
-
-```bash
-python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
-```
-
-**2.运行以下命令测试ocr加速效果**
-
-```bash
-magic-pdf -p small_ocr.pdf
-```
-
-> 提示：CUDA加速是否生效可以根据log中输出的各个阶段cost耗时来简单判断，通常情况下，`ocr cost`应提速10倍以上。
diff --git a/docs/README_Windows_CUDA_Acceleration_en_US.md b/docs/README_Windows_CUDA_Acceleration_en_US.md
deleted file mode 100644
index e258011d0..000000000
--- a/docs/README_Windows_CUDA_Acceleration_en_US.md
+++ /dev/null
@@ -1,100 +0,0 @@
-# Windows 10/11
-
-### 1. Install CUDA and cuDNN
-
-Required versions: CUDA 11.8 + cuDNN 8.7.0
-
-- CUDA 11.8: https://developer.nvidia.com/cuda-11-8-0-download-archive
-- cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x: https://developer.nvidia.com/rdp/cudnn-archive
-
-### 2. Install Anaconda
-
-If Anaconda is already installed, you can skip this step.
-
-Download link: https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Windows-x86_64.exe
-
-### 3. Create an Environment Using Conda
-
-Python version must be 3.10.
-
-```
-conda create -n MinerU python=3.10
-conda activate MinerU
-```
-
-### 4. Install Applications
-
-```
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com
-```
-
-> ❗️After installation, verify the version of `magic-pdf`:
->
-> ```bash
-> magic-pdf --version
-> ```
->
-> If the version number is less than 0.7.0, please report it in the issues section.
-
-### 5. Download Models
-
-Refer to detailed instructions on [how to download model files](how_to_download_models_en.md).
-
-### 6. Understand the Location of the Configuration File
-
-After completing the [5. Download Models](#5-download-models) step, the script will automatically generate a `magic-pdf.json` file in the user directory and configure the default model path.
-You can find the `magic-pdf.json` file in your 【user directory】 .
-
-> The user directory for Windows is "C:/Users/username".
-
-### 7. First Run
-
-Download a sample file from the repository and test it.
-
-```powershell
- wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf -O small_ocr.pdf
- magic-pdf -p small_ocr.pdf
-```
-
-### 8. Test CUDA Acceleration
-
-If your graphics card has at least 8GB of VRAM, follow these steps to test CUDA-accelerated parsing performance.
-
-1. **Overwrite the installation of torch and torchvision** supporting CUDA.
-
- ```
- pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118
- ```
-
- > ❗️Ensure the following versions are specified in the command:
- >
- > ```
- > torch==2.3.1 torchvision==0.18.1
- > ```
- >
- > These are the highest versions we support. Installing higher versions without specifying them will cause the program to fail.
-
-2. **Modify the value of `"device-mode"`** in the `magic-pdf.json` configuration file located in your user directory.
-
- ```json
- {
- "device-mode": "cuda"
- }
- ```
-
-3. **Run the following command to test CUDA acceleration**:
-
- ```
- magic-pdf -p small_ocr.pdf
- ```
-
-### 9. Enable CUDA Acceleration for OCR
-
-1. **Download paddlepaddle-gpu**, which will automatically enable OCR acceleration upon installation.
- ```
- pip install paddlepaddle-gpu==2.6.1
- ```
-2. **Run the following command to test OCR acceleration**:
- ```
- magic-pdf -p small_ocr.pdf
- ```
diff --git a/docs/README_Windows_CUDA_Acceleration_zh_CN.md b/docs/README_Windows_CUDA_Acceleration_zh_CN.md
deleted file mode 100644
index a88a50dbd..000000000
--- a/docs/README_Windows_CUDA_Acceleration_zh_CN.md
+++ /dev/null
@@ -1,108 +0,0 @@
-# Windows10/11
-
-## 1. 安装cuda和cuDNN
-
-需要安装的版本 CUDA 11.8 + cuDNN 8.7.0
-
-- CUDA 11.8 https://developer.nvidia.com/cuda-11-8-0-download-archive
-- cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x https://developer.nvidia.com/rdp/cudnn-archive
-
-## 2. 安装anaconda
-
-如果已安装conda，可以跳过本步骤
-
-下载链接：
-https://mirrors.tuna.tsinghua.edu.cn/anaconda/archive/Anaconda3-2024.06-1-Windows-x86_64.exe
-
-## 3. 使用conda 创建环境
-
-需指定python版本为3.10
-
-```bash
-conda create -n MinerU python=3.10
-conda activate MinerU
-```
-
-## 4. 安装应用
-
-```bash
-pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
-```
-
-> ❗️下载完成后，务必通过以下命令确认magic-pdf的版本是否正确
->
-> ```bash
-> magic-pdf --version
-> ```
->
-> 如果版本号小于0.7.0，请到issue中向我们反馈
-
-## 5. 下载模型
-
-详细参考 [如何下载模型文件](how_to_download_models_zh_cn.md)
-
-## 6. 了解配置文件存放的位置
-
-完成[5.下载模型](#5-下载模型)步骤后，脚本会自动生成用户目录下的magic-pdf.json文件，并自动配置默认模型路径。
-您可在【用户目录】下找到magic-pdf.json文件。
-
-> windows用户目录为 "C:/Users/用户名"
-
-## 7. 第一次运行
-
-从仓库中下载样本文件，并测试
-
-```powershell
- wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf -O small_ocr.pdf
- magic-pdf -p small_ocr.pdf
-```
-
-## 8. 测试CUDA加速
-
-如果您的显卡显存大于等于 **8GB** ，可以进行以下流程，测试CUDA解析加速效果
-
-**1.覆盖安装支持cuda的torch和torchvision**
-
-```bash
-pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118
-```
-
-> ❗️务必在命令中指定以下版本
->
-> ```bash
-> torch==2.3.1 torchvision==0.18.1
-> ```
->
-> 这是我们支持的最高版本，如果不指定版本会自动安装更高版本导致程序无法运行
-
-**2.修改【用户目录】中配置文件magic-pdf.json中"device-mode"的值**
-
-```json
-{
- "device-mode":"cuda"
-}
-```
-
-**3.运行以下命令测试cuda加速效果**
-
-```bash
-magic-pdf -p small_ocr.pdf
-```
-
-> 提示：CUDA加速是否生效可以根据log中输出的各个阶段的耗时来简单判断，通常情况下，`layout detection time` 和 `mfr time` 应提速10倍以上。
-
-## 9. 为ocr开启cuda加速
-
-**1.下载paddlepaddle-gpu, 安装完成后会自动开启ocr加速**
-
-```bash
-pip install paddlepaddle-gpu==2.6.1
-```
-
-**2.运行以下命令测试ocr加速效果**
-
-```bash
-magic-pdf -p small_ocr.pdf
-```
-
-> 提示：CUDA加速是否生效可以根据log中输出的各个阶段cost耗时来简单判断，通常情况下，`ocr time`应提速10倍以上。
diff --git a/docs/assets/images/BISHENG_01.png b/docs/assets/images/BISHENG_01.png
new file mode 100644
index 000000000..1291233cf
Binary files /dev/null and b/docs/assets/images/BISHENG_01.png differ
diff --git a/docs/assets/images/Cherry_Studio_1.png b/docs/assets/images/Cherry_Studio_1.png
new file mode 100644
index 000000000..dffb4a00a
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_1.png differ
diff --git a/docs/assets/images/Cherry_Studio_2.png b/docs/assets/images/Cherry_Studio_2.png
new file mode 100644
index 000000000..c1b1cfef5
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_2.png differ
diff --git a/docs/assets/images/Cherry_Studio_3.png b/docs/assets/images/Cherry_Studio_3.png
new file mode 100644
index 000000000..d400e7342
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_3.png differ
diff --git a/docs/assets/images/Cherry_Studio_4.png b/docs/assets/images/Cherry_Studio_4.png
new file mode 100644
index 000000000..26a5dddb4
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_4.png differ
diff --git a/docs/assets/images/Cherry_Studio_5.png b/docs/assets/images/Cherry_Studio_5.png
new file mode 100644
index 000000000..da88d85ab
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_5.png differ
diff --git a/docs/assets/images/Cherry_Studio_6.png b/docs/assets/images/Cherry_Studio_6.png
new file mode 100644
index 000000000..35762af1b
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_6.png differ
diff --git a/docs/assets/images/Cherry_Studio_7.png b/docs/assets/images/Cherry_Studio_7.png
new file mode 100644
index 000000000..916ecfe90
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_7.png differ
diff --git a/docs/assets/images/Cherry_Studio_8.png b/docs/assets/images/Cherry_Studio_8.png
new file mode 100644
index 000000000..ab2a9b96f
Binary files /dev/null and b/docs/assets/images/Cherry_Studio_8.png differ
diff --git a/docs/assets/images/Coze_1.png b/docs/assets/images/Coze_1.png
new file mode 100644
index 000000000..7c834d1d5
Binary files /dev/null and b/docs/assets/images/Coze_1.png differ
diff --git a/docs/assets/images/Coze_10.png b/docs/assets/images/Coze_10.png
new file mode 100644
index 000000000..6feda8eb1
Binary files /dev/null and b/docs/assets/images/Coze_10.png differ
diff --git a/docs/assets/images/Coze_11.png b/docs/assets/images/Coze_11.png
new file mode 100644
index 000000000..420776281
Binary files /dev/null and b/docs/assets/images/Coze_11.png differ
diff --git a/docs/assets/images/Coze_12.png b/docs/assets/images/Coze_12.png
new file mode 100644
index 000000000..ed8ef1e09
Binary files /dev/null and b/docs/assets/images/Coze_12.png differ
diff --git a/docs/assets/images/Coze_13.png b/docs/assets/images/Coze_13.png
new file mode 100644
index 000000000..fa2e79ccc
Binary files /dev/null and b/docs/assets/images/Coze_13.png differ
diff --git a/docs/assets/images/Coze_14.png b/docs/assets/images/Coze_14.png
new file mode 100644
index 000000000..cc491651c
Binary files /dev/null and b/docs/assets/images/Coze_14.png differ
diff --git a/docs/assets/images/Coze_15.png b/docs/assets/images/Coze_15.png
new file mode 100644
index 000000000..fd95fff29
Binary files /dev/null and b/docs/assets/images/Coze_15.png differ
diff --git a/docs/assets/images/Coze_16.png b/docs/assets/images/Coze_16.png
new file mode 100644
index 000000000..29c6598de
Binary files /dev/null and b/docs/assets/images/Coze_16.png differ
diff --git a/docs/assets/images/Coze_17.png b/docs/assets/images/Coze_17.png
new file mode 100644
index 000000000..712792724
Binary files /dev/null and b/docs/assets/images/Coze_17.png differ
diff --git a/docs/assets/images/Coze_18.png b/docs/assets/images/Coze_18.png
new file mode 100644
index 000000000..ef2c54dbe
Binary files /dev/null and b/docs/assets/images/Coze_18.png differ
diff --git a/docs/assets/images/Coze_19.png b/docs/assets/images/Coze_19.png
new file mode 100644
index 000000000..b7afb73df
Binary files /dev/null and b/docs/assets/images/Coze_19.png differ
diff --git a/docs/assets/images/Coze_2.png b/docs/assets/images/Coze_2.png
new file mode 100644
index 000000000..fb8dac378
Binary files /dev/null and b/docs/assets/images/Coze_2.png differ
diff --git a/docs/assets/images/Coze_20.png b/docs/assets/images/Coze_20.png
new file mode 100644
index 000000000..a7505c3b5
Binary files /dev/null and b/docs/assets/images/Coze_20.png differ
diff --git a/docs/assets/images/Coze_21.png b/docs/assets/images/Coze_21.png
new file mode 100644
index 000000000..a8365e603
Binary files /dev/null and b/docs/assets/images/Coze_21.png differ
diff --git a/docs/assets/images/Coze_3.png b/docs/assets/images/Coze_3.png
new file mode 100644
index 000000000..f77f8969e
Binary files /dev/null and b/docs/assets/images/Coze_3.png differ
diff --git a/docs/assets/images/Coze_4.png b/docs/assets/images/Coze_4.png
new file mode 100644
index 000000000..c67f52529
Binary files /dev/null and b/docs/assets/images/Coze_4.png differ
diff --git a/docs/assets/images/Coze_5.png b/docs/assets/images/Coze_5.png
new file mode 100644
index 000000000..e70ba9ca3
Binary files /dev/null and b/docs/assets/images/Coze_5.png differ
diff --git a/docs/assets/images/Coze_6.png b/docs/assets/images/Coze_6.png
new file mode 100644
index 000000000..5a0cb1024
Binary files /dev/null and b/docs/assets/images/Coze_6.png differ
diff --git a/docs/assets/images/Coze_7.png b/docs/assets/images/Coze_7.png
new file mode 100644
index 000000000..9f6f8292f
Binary files /dev/null and b/docs/assets/images/Coze_7.png differ
diff --git a/docs/assets/images/Coze_8.png b/docs/assets/images/Coze_8.png
new file mode 100644
index 000000000..f08e5378d
Binary files /dev/null and b/docs/assets/images/Coze_8.png differ
diff --git a/docs/assets/images/Coze_9.png b/docs/assets/images/Coze_9.png
new file mode 100644
index 000000000..fee7923b2
Binary files /dev/null and b/docs/assets/images/Coze_9.png differ
diff --git a/docs/assets/images/DataFLow_01.png b/docs/assets/images/DataFLow_01.png
new file mode 100644
index 000000000..17e3882b7
Binary files /dev/null and b/docs/assets/images/DataFLow_01.png differ
diff --git a/docs/assets/images/DataFlow_02.png b/docs/assets/images/DataFlow_02.png
new file mode 100644
index 000000000..a6182ddfd
Binary files /dev/null and b/docs/assets/images/DataFlow_02.png differ
diff --git a/docs/assets/images/Dify_1.png b/docs/assets/images/Dify_1.png
new file mode 100644
index 000000000..dbb9e6d5f
Binary files /dev/null and b/docs/assets/images/Dify_1.png differ
diff --git a/docs/assets/images/Dify_10.png b/docs/assets/images/Dify_10.png
new file mode 100644
index 000000000..d6626aba2
Binary files /dev/null and b/docs/assets/images/Dify_10.png differ
diff --git a/docs/assets/images/Dify_11.png b/docs/assets/images/Dify_11.png
new file mode 100644
index 000000000..bec7e91e5
Binary files /dev/null and b/docs/assets/images/Dify_11.png differ
diff --git a/docs/assets/images/Dify_12.png b/docs/assets/images/Dify_12.png
new file mode 100644
index 000000000..822fd7fbf
Binary files /dev/null and b/docs/assets/images/Dify_12.png differ
diff --git a/docs/assets/images/Dify_13.png b/docs/assets/images/Dify_13.png
new file mode 100644
index 000000000..d5025f13a
Binary files /dev/null and b/docs/assets/images/Dify_13.png differ
diff --git a/docs/assets/images/Dify_14.png b/docs/assets/images/Dify_14.png
new file mode 100644
index 000000000..f785542a2
Binary files /dev/null and b/docs/assets/images/Dify_14.png differ
diff --git a/docs/assets/images/Dify_15.png b/docs/assets/images/Dify_15.png
new file mode 100644
index 000000000..ef40173c1
Binary files /dev/null and b/docs/assets/images/Dify_15.png differ
diff --git a/docs/assets/images/Dify_16.png b/docs/assets/images/Dify_16.png
new file mode 100644
index 000000000..1f203b52d
Binary files /dev/null and b/docs/assets/images/Dify_16.png differ
diff --git a/docs/assets/images/Dify_17.png b/docs/assets/images/Dify_17.png
new file mode 100644
index 000000000..f944a3908
Binary files /dev/null and b/docs/assets/images/Dify_17.png differ
diff --git a/docs/assets/images/Dify_18.png b/docs/assets/images/Dify_18.png
new file mode 100644
index 000000000..a2b069d89
Binary files /dev/null and b/docs/assets/images/Dify_18.png differ
diff --git a/docs/assets/images/Dify_19.png b/docs/assets/images/Dify_19.png
new file mode 100644
index 000000000..dc278e01e
Binary files /dev/null and b/docs/assets/images/Dify_19.png differ
diff --git a/docs/assets/images/Dify_2.png b/docs/assets/images/Dify_2.png
new file mode 100644
index 000000000..182c85f36
Binary files /dev/null and b/docs/assets/images/Dify_2.png differ
diff --git a/docs/assets/images/Dify_20.png b/docs/assets/images/Dify_20.png
new file mode 100644
index 000000000..91b910e27
Binary files /dev/null and b/docs/assets/images/Dify_20.png differ
diff --git a/docs/assets/images/Dify_21.png b/docs/assets/images/Dify_21.png
new file mode 100644
index 000000000..784de7726
Binary files /dev/null and b/docs/assets/images/Dify_21.png differ
diff --git a/docs/assets/images/Dify_22.png b/docs/assets/images/Dify_22.png
new file mode 100644
index 000000000..304995f1c
Binary files /dev/null and b/docs/assets/images/Dify_22.png differ
diff --git a/docs/assets/images/Dify_23.png b/docs/assets/images/Dify_23.png
new file mode 100644
index 000000000..9a1ac093b
Binary files /dev/null and b/docs/assets/images/Dify_23.png differ
diff --git a/docs/assets/images/Dify_24.png b/docs/assets/images/Dify_24.png
new file mode 100644
index 000000000..4902617ca
Binary files /dev/null and b/docs/assets/images/Dify_24.png differ
diff --git a/docs/assets/images/Dify_25.png b/docs/assets/images/Dify_25.png
new file mode 100644
index 000000000..21315a3fa
Binary files /dev/null and b/docs/assets/images/Dify_25.png differ
diff --git a/docs/assets/images/Dify_26.png b/docs/assets/images/Dify_26.png
new file mode 100644
index 000000000..c59244b2e
Binary files /dev/null and b/docs/assets/images/Dify_26.png differ
diff --git a/docs/assets/images/Dify_3.png b/docs/assets/images/Dify_3.png
new file mode 100644
index 000000000..0a7cb968a
Binary files /dev/null and b/docs/assets/images/Dify_3.png differ
diff --git a/docs/assets/images/Dify_4.png b/docs/assets/images/Dify_4.png
new file mode 100644
index 000000000..759c070aa
Binary files /dev/null and b/docs/assets/images/Dify_4.png differ
diff --git a/docs/assets/images/Dify_5.png b/docs/assets/images/Dify_5.png
new file mode 100644
index 000000000..bc73986de
Binary files /dev/null and b/docs/assets/images/Dify_5.png differ
diff --git a/docs/assets/images/Dify_6.png b/docs/assets/images/Dify_6.png
new file mode 100644
index 000000000..27225bf8a
Binary files /dev/null and b/docs/assets/images/Dify_6.png differ
diff --git a/docs/assets/images/Dify_7.png b/docs/assets/images/Dify_7.png
new file mode 100644
index 000000000..82bb291c4
Binary files /dev/null and b/docs/assets/images/Dify_7.png differ
diff --git a/docs/assets/images/Dify_8.png b/docs/assets/images/Dify_8.png
new file mode 100644
index 000000000..9f9422ae1
Binary files /dev/null and b/docs/assets/images/Dify_8.png differ
diff --git a/docs/assets/images/Dify_9.png b/docs/assets/images/Dify_9.png
new file mode 100644
index 000000000..b94f315d9
Binary files /dev/null and b/docs/assets/images/Dify_9.png differ
diff --git a/docs/assets/images/DingTalk_01.png b/docs/assets/images/DingTalk_01.png
new file mode 100644
index 000000000..413012ad2
Binary files /dev/null and b/docs/assets/images/DingTalk_01.png differ
diff --git a/docs/assets/images/FastGPT_01.png b/docs/assets/images/FastGPT_01.png
new file mode 100644
index 000000000..25fbdfa03
Binary files /dev/null and b/docs/assets/images/FastGPT_01.png differ
diff --git a/docs/assets/images/FastGPT_02.png b/docs/assets/images/FastGPT_02.png
new file mode 100644
index 000000000..345f1eecb
Binary files /dev/null and b/docs/assets/images/FastGPT_02.png differ
diff --git a/docs/assets/images/ModelWhale_01.png b/docs/assets/images/ModelWhale_01.png
new file mode 100644
index 000000000..b2f768aae
Binary files /dev/null and b/docs/assets/images/ModelWhale_01.png differ
diff --git a/docs/assets/images/ModelWhale_02.png b/docs/assets/images/ModelWhale_02.png
new file mode 100644
index 000000000..bc964f11e
Binary files /dev/null and b/docs/assets/images/ModelWhale_02.png differ
diff --git a/docs/assets/images/ModelWhale_1.png b/docs/assets/images/ModelWhale_1.png
new file mode 100644
index 000000000..c80e7f42c
Binary files /dev/null and b/docs/assets/images/ModelWhale_1.png differ
diff --git a/docs/assets/images/RagFlow_01.png b/docs/assets/images/RagFlow_01.png
new file mode 100644
index 000000000..476e0de2f
Binary files /dev/null and b/docs/assets/images/RagFlow_01.png differ
diff --git a/docs/assets/images/RagFlow_02.png b/docs/assets/images/RagFlow_02.png
new file mode 100644
index 000000000..f2fec68ec
Binary files /dev/null and b/docs/assets/images/RagFlow_02.png differ
diff --git a/docs/assets/images/Sider_1.png b/docs/assets/images/Sider_1.png
new file mode 100644
index 000000000..f682e38b5
Binary files /dev/null and b/docs/assets/images/Sider_1.png differ
diff --git a/docs/assets/images/coze_0.png b/docs/assets/images/coze_0.png
new file mode 100644
index 000000000..92ff21310
Binary files /dev/null and b/docs/assets/images/coze_0.png differ
diff --git a/docs/assets/images/n8n_0.png b/docs/assets/images/n8n_0.png
new file mode 100644
index 000000000..31b42c03b
Binary files /dev/null and b/docs/assets/images/n8n_0.png differ
diff --git a/docs/assets/images/n8n_1.png b/docs/assets/images/n8n_1.png
new file mode 100644
index 000000000..3f9fecbc7
Binary files /dev/null and b/docs/assets/images/n8n_1.png differ
diff --git a/docs/assets/images/n8n_10.png b/docs/assets/images/n8n_10.png
new file mode 100644
index 000000000..6fdc12cb8
Binary files /dev/null and b/docs/assets/images/n8n_10.png differ
diff --git a/docs/assets/images/n8n_2.png b/docs/assets/images/n8n_2.png
new file mode 100644
index 000000000..f93599a86
Binary files /dev/null and b/docs/assets/images/n8n_2.png differ
diff --git a/docs/assets/images/n8n_3.png b/docs/assets/images/n8n_3.png
new file mode 100644
index 000000000..c1ab8807b
Binary files /dev/null and b/docs/assets/images/n8n_3.png differ
diff --git a/docs/assets/images/n8n_4.png b/docs/assets/images/n8n_4.png
new file mode 100644
index 000000000..76657fab4
Binary files /dev/null and b/docs/assets/images/n8n_4.png differ
diff --git a/docs/assets/images/n8n_5.png b/docs/assets/images/n8n_5.png
new file mode 100644
index 000000000..f6aa18af1
Binary files /dev/null and b/docs/assets/images/n8n_5.png differ
diff --git a/docs/assets/images/n8n_6.png b/docs/assets/images/n8n_6.png
new file mode 100644
index 000000000..88c9ea3aa
Binary files /dev/null and b/docs/assets/images/n8n_6.png differ
diff --git a/docs/assets/images/n8n_7.png b/docs/assets/images/n8n_7.png
new file mode 100644
index 000000000..7a1e0f707
Binary files /dev/null and b/docs/assets/images/n8n_7.png differ
diff --git a/docs/assets/images/n8n_8.png b/docs/assets/images/n8n_8.png
new file mode 100644
index 000000000..9daff94e7
Binary files /dev/null and b/docs/assets/images/n8n_8.png differ
diff --git a/docs/assets/images/n8n_9.png b/docs/assets/images/n8n_9.png
new file mode 100644
index 000000000..77c6272f3
Binary files /dev/null and b/docs/assets/images/n8n_9.png differ
diff --git a/docs/download_models.py b/docs/download_models.py
deleted file mode 100644
index ed1ee5c3b..000000000
--- a/docs/download_models.py
+++ /dev/null
@@ -1,59 +0,0 @@
-import json
-import os
-
-import requests
-from modelscope import snapshot_download
-
-
-def download_json(url):
- # 下载JSON文件
- response = requests.get(url)
- response.raise_for_status() # 检查请求是否成功
- return response.json()
-
-
-def download_and_modify_json(url, local_filename, modifications):
- if os.path.exists(local_filename):
- data = json.load(open(local_filename))
- config_version = data.get('config_version', '0.0.0')
- if config_version < '1.0.0':
- data = download_json(url)
- else:
- data = download_json(url)
-
- # 修改内容
- for key, value in modifications.items():
- data[key] = value
-
- # 保存修改后的内容
- with open(local_filename, 'w', encoding='utf-8') as f:
- json.dump(data, f, ensure_ascii=False, indent=4)
-
-
-if __name__ == '__main__':
- mineru_patterns = [
- "models/Layout/LayoutLMv3/*",
- "models/Layout/YOLO/*",
- "models/MFD/YOLO/*",
- "models/MFR/unimernet_small/*",
- "models/TabRec/TableMaster/*",
- "models/TabRec/StructEqTable/*",
- ]
- model_dir = snapshot_download('opendatalab/PDF-Extract-Kit-1.0', allow_patterns=mineru_patterns)
- layoutreader_model_dir = snapshot_download('ppaanngggg/layoutreader')
- model_dir = model_dir + '/models'
- print(f'model_dir is: {model_dir}')
- print(f'layoutreader_model_dir is: {layoutreader_model_dir}')
-
- json_url = 'https://gitee.com/myhloli/MinerU/raw/dev/magic-pdf.template.json'
- config_file_name = 'magic-pdf.json'
- home_dir = os.path.expanduser('~')
- config_file = os.path.join(home_dir, config_file_name)
-
- json_mods = {
- 'models-dir': model_dir,
- 'layoutreader-model-dir': layoutreader_model_dir,
- }
-
- download_and_modify_json(json_url, config_file, json_mods)
- print(f'The configuration file has been configured successfully, the path is: {config_file}')
diff --git a/docs/download_models_hf.py b/docs/download_models_hf.py
deleted file mode 100644
index 5e6b8dced..000000000
--- a/docs/download_models_hf.py
+++ /dev/null
@@ -1,66 +0,0 @@
-import json
-import os
-
-import requests
-from huggingface_hub import snapshot_download
-
-
-def download_json(url):
- # 下载JSON文件
- response = requests.get(url)
- response.raise_for_status() # 检查请求是否成功
- return response.json()
-
-
-def download_and_modify_json(url, local_filename, modifications):
- if os.path.exists(local_filename):
- data = json.load(open(local_filename))
- config_version = data.get('config_version', '0.0.0')
- if config_version < '1.0.0':
- data = download_json(url)
- else:
- data = download_json(url)
-
- # 修改内容
- for key, value in modifications.items():
- data[key] = value
-
- # 保存修改后的内容
- with open(local_filename, 'w', encoding='utf-8') as f:
- json.dump(data, f, ensure_ascii=False, indent=4)
-
-
-if __name__ == '__main__':
-
- mineru_patterns = [
- "models/Layout/LayoutLMv3/*",
- "models/Layout/YOLO/*",
- "models/MFD/YOLO/*",
- "models/MFR/unimernet_small/*",
- "models/TabRec/TableMaster/*",
- "models/TabRec/StructEqTable/*",
- ]
- model_dir = snapshot_download('opendatalab/PDF-Extract-Kit-1.0', allow_patterns=mineru_patterns)
-
- layoutreader_pattern = [
- "*.json",
- "*.safetensors",
- ]
- layoutreader_model_dir = snapshot_download('hantian/layoutreader', allow_patterns=layoutreader_pattern)
-
- model_dir = model_dir + '/models'
- print(f'model_dir is: {model_dir}')
- print(f'layoutreader_model_dir is: {layoutreader_model_dir}')
-
- json_url = 'https://github.com/opendatalab/MinerU/raw/dev/magic-pdf.template.json'
- config_file_name = 'magic-pdf.json'
- home_dir = os.path.expanduser('~')
- config_file = os.path.join(home_dir, config_file_name)
-
- json_mods = {
- 'models-dir': model_dir,
- 'layoutreader-model-dir': layoutreader_model_dir,
- }
-
- download_and_modify_json(json_url, config_file, json_mods)
- print(f'The configuration file has been configured successfully, the path is: {config_file}')
diff --git a/docs/en/demo/index.md b/docs/en/demo/index.md
new file mode 100644
index 000000000..869eb17a1
--- /dev/null
+++ b/docs/en/demo/index.md
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/docs/en/faq/index.md b/docs/en/faq/index.md
new file mode 100644
index 000000000..0877424e3
--- /dev/null
+++ b/docs/en/faq/index.md
@@ -0,0 +1,30 @@
+# Frequently Asked Questions
+
+If your question is not listed, try using [DeepWiki](https://deepwiki.com/opendatalab/MinerU)'s AI assistant for common issues.
+
+For unresolved problems, join our [Discord](https://discord.gg/Tdedn9GTXq) or [WeChat](https://mineru.net/community-portal/?aliasId=3c430f94) community for support.
+
+??? question "Encountered the error `ImportError: libGL.so.1: cannot open shared object file: No such file or directory` in Ubuntu 22.04 on WSL2"
+
+ The `libgl` library is missing in Ubuntu 22.04 on WSL2. You can install the `libgl` library with the following command to resolve the issue:
+
+ ```bash
+ sudo apt-get install libgl1-mesa-glx
+ ```
+
+ Reference: [#388](https://github.com/opendatalab/MinerU/issues/388)
+
+
+??? question "Missing text information in parsing results when installing and using on Linux systems."
+
+ MinerU uses `pypdfium2` instead of `pymupdf` as the PDF page rendering engine in versions >=2.0 to resolve AGPLv3 license issues. On some Linux distributions, due to missing CJK fonts, some text may be lost during the process of rendering PDFs to images.
+ To solve this problem, you can install the noto font package with the following commands, which are effective on Ubuntu/Debian systems:
+ ```bash
+ sudo apt update
+ sudo apt install fonts-noto-core
+ sudo apt install fonts-noto-cjk
+ fc-cache -fv
+ ```
+ You can also directly use our [Docker deployment](../quick_start/docker_deployment.md) method to build the image, which includes the above font packages by default.
+
+ Reference: [#2915](https://github.com/opendatalab/MinerU/issues/2915)
diff --git a/docs/en/index.md b/docs/en/index.md
new file mode 100644
index 000000000..52cd9cb28
--- /dev/null
+++ b/docs/en/index.md
@@ -0,0 +1,76 @@
+ -o -b vlm-http-client -u http://:30000
+ > ```
+
+---
+
+### Start Web API service
+ ```bash
+ docker compose -f compose.yaml --profile api up -d
+ ```
+ >[!TIP]
+ >Access `http://:8000/docs` in your browser to view the API documentation.
+
+---
+
+### Start MinerU Router service
+ ```bash
+ docker compose -f compose.yaml --profile router up -d
+ ```
+ >[!TIP]
+ >
+ >- The default configuration runs in `--local-gpus auto` mode, automatically starting local workers in the container and exposing the unified entry at `http://:8002/docs`.
+ >- If you want to aggregate existing `mineru-api` services instead of starting local workers, refer to the commented example under the `mineru-router` service in `compose.yaml` and switch to `--upstream-url`.
+
+---
+
+### Start Gradio WebUI service
+ ```bash
+ docker compose -f compose.yaml --profile gradio up -d
+ ```
+ >[!TIP]
+ >
+ >- Access `http://:7860` in your browser to use the Gradio WebUI.
diff --git a/docs/en/quick_start/extension_modules.md b/docs/en/quick_start/extension_modules.md
new file mode 100644
index 000000000..f6bf34acd
--- /dev/null
+++ b/docs/en/quick_start/extension_modules.md
@@ -0,0 +1,56 @@
+# MinerU Extension Modules Installation Guide
+MinerU supports installing extension modules on demand based on different needs to enhance functionality or support specific model backends.
+
+## Common Scenarios
+
+### Core Functionality Installation
+The `core` module is the core dependency of MinerU, containing all functional modules except `vllm`/`lmdeploy`. Installing this module ensures the basic functionality of MinerU works properly.
+```bash
+uv pip install "mineru[core]"
+```
+
+---
+
+### Using `vllm` to Accelerate VLM Model Inference
+> [!NOTE]
+> `vllm` and `lmdeploy` have nearly identical VLM inference acceleration effects and usage methods. You can choose one of them to install and use based on your actual needs, but it is not recommended to install both modules simultaneously to avoid potential dependency conflicts.
+
+The `vllm` module provides acceleration support for VLM model inference, suitable for graphics cards with Volta architecture and later (8GB+ VRAM). Installing this module can significantly improve model inference speed.
+
+```bash
+uv pip install "mineru[core,vllm]"
+```
+> [!TIP]
+> If exceptions occur during installation of the extra package including vllm, please refer to the [vllm official documentation](https://docs.vllm.ai/en/latest/getting_started/installation/index.html) to try to resolve the issue, or directly use the [Docker](./docker_deployment.md) deployment method.
+
+---
+
+### Using `lmdeploy` to Accelerate VLM Model Inference
+> [!NOTE]
+> `vllm` and `lmdeploy` have nearly identical VLM inference acceleration effects and usage methods. You can choose one of them to install and use based on your actual needs, but it is not recommended to install both modules simultaneously to avoid potential dependency conflicts.
+
+The `lmdeploy` module provides acceleration support for VLM model inference, suitable for graphics cards with Volta architecture and later (8GB+ VRAM). Installing this module can significantly improve model inference speed.
+
+```bash
+uv pip install "mineru[core,lmdeploy]"
+```
+> [!TIP]
+> If exceptions occur during installation of the extra package including lmdeploy, please refer to the [lmdeploy official documentation](https://lmdeploy.readthedocs.io/en/latest/get_started/installation.html) to try to resolve the issue.
+
+---
+
+### Installing Lightweight Client to Connect to OpenAI-compatible servers (for vlm-http-client mode)
+If you need to install a lightweight client on edge devices to connect to an OpenAI-compatible server for using VLM mode, you can install the basic mineru package, which is very lightweight and suitable for devices with only CPU and network connectivity.
+```bash
+uv pip install mineru
+mineru -p -o -b vlm-http-client -u http://127.0.0.1:30000
+```
+
+---
+
+### Installing Lightweight Client to Connect to OpenAI-compatible servers (for hybrid-http-client mode)
+If you need to install a lightweight client on edge devices to connect to an OpenAI-compatible server for using hybrid mode, you can install the mineru pipeline extension package, which is relatively lightweight and can be used on devices with only CPU and network connectivity, while running faster on devices that support GPU acceleration.
+```bash
+uv pip install "mineru[pipeline]"
+mineru -p -o -b hybrid-http-client -u http://127.0.0.1:30000
+```
\ No newline at end of file
diff --git a/docs/en/quick_start/index.md b/docs/en/quick_start/index.md
new file mode 100644
index 000000000..b4a140ef9
--- /dev/null
+++ b/docs/en/quick_start/index.md
@@ -0,0 +1,143 @@
+# Quick Start
+
+If you encounter any installation issues, please check the [FAQ](../faq/index.md) first.
+
+## Online Experience
+
+### Official online web application
+The official online version has the same functionality as the client, with a beautiful interface and rich features, requires login to use
+
+- [](https://mineru.net/OpenSourceTools/Extractor?source=github)
+
+### Gradio-based online demo
+A WebUI developed based on Gradio, with a simple interface and only core parsing functionality, no login required
+
+- [](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+- [](https://huggingface.co/spaces/opendatalab/MinerU)
+
+## Local Deployment
+
+> [!WARNING]
+> **Prerequisites - Hardware and Software Environment Support**
+>
+> To ensure the stability and reliability of the project, we have optimized and tested only specific hardware and software environments during development. This ensures that users can achieve optimal performance and encounter the fewest compatibility issues when deploying and running the project on recommended system configurations.
+>
+> By concentrating our resources and efforts on mainstream environments, our team can more efficiently resolve potential bugs and timely develop new features.
+>
+> In non-mainstream environments, due to the diversity of hardware and software configurations, as well as compatibility issues with third-party dependencies, we cannot guarantee 100% usability of the project. Therefore, for users who wish to use this project in non-recommended environments, we suggest carefully reading the documentation and FAQ first, as most issues have corresponding solutions in the FAQ. Additionally, we encourage community feedback on issues so that we can gradually expand our support range.
+
+
+
+
+
+1 Accuracy metrics are the End-to-End Evaluation Overall scores from OmniDocBench (v1.5), based on the latest version of `MinerU`.
+2 Servers compatible with OpenAI API, such as local model servers or remote model services deployed via inference frameworks like `vLLM`/`SGLang`/`LMDeploy`.
+3 Linux only supports distributions from 2019 and later.
+4 Since the key dependency `ray` does not support Python 3.13 on Windows, only versions 3.10~3.12 are supported.
+5 macOS requires version 14.0 or later.
+
+
+### Install MinerU
+
+#### Install MinerU using pip or uv
+```bash
+pip install --upgrade pip
+pip install uv
+uv pip install -U "mineru[all]"
+```
+
+#### Install MinerU from source code
+```bash
+git clone https://github.com/opendatalab/MinerU.git
+cd MinerU
+uv pip install -e .[all]
+```
+
+> [!TIP]
+> `mineru[all]` includes all core features, compatible with Windows / Linux / macOS systems, suitable for most users.
+> If you need to specify the inference framework for the VLM model, or only intend to install a lightweight client on an edge device, please refer to the documentation [Extension Modules Installation Guide](https://opendatalab.github.io/MinerU/quick_start/extension_modules/).
+
+---
+
+#### Deploy MinerU using Docker
+MinerU provides a convenient Docker deployment method, which helps quickly set up the environment and solve some tricky environment compatibility issues.
+You can get the [Docker Deployment Instructions](./docker_deployment.md) in the documentation.
+
+---
+
+### Using MinerU
+
+If your device meets the GPU acceleration requirements in the table above, you can use a simple command line for document parsing:
+```bash
+mineru -p -o
+```
+If your device does not meet the GPU acceleration requirements, you can specify the backend as `pipeline` to run in a pure CPU environment:
+```bash
+mineru -p -o -b pipeline
+```
+
+`mineru` currently supports local `PDF`, image, `DOCX`, `PPTX`, and `XLSX` file or directory inputs.
+
+You can use MinerU for document parsing through the CLI, API, WebUI. For detailed instructions, please refer to the [Usage Guide](../usage/index.md).
diff --git a/docs/en/reference/changelog.md b/docs/en/reference/changelog.md
new file mode 100644
index 000000000..abbb94b89
--- /dev/null
+++ b/docs/en/reference/changelog.md
@@ -0,0 +1,497 @@
+# Changelog
+
+This document records the release history of MinerU, including major capability changes, compatibility updates, and notable fixes.
+
+---
+
+## 2.7 Series Versions
+
+### 2.7.6 (2026/02/06)
+
+- Added support for the Kunlunxin and Tecorigin domestic computing platforms.
+- Officially or vendor-adapted domestic platforms now include Ascend, T-Head, METAX, Hygon, Enflame, Moore Threads, IluvatarCorex, Cambricon, Kunlunxin, Tecorigin, and Biren.
+- MinerU continues to support domestic hardware platforms and mainstream chip architectures for document digitization workloads.
+
+### 2.7.4 (2026/01/30)
+
+- Added support for the IluvatarCorex and Cambricon domestic computing platforms.
+
+### 2.7.2 (2026/01/23)
+
+- Added support for the Hygon, Enflame, and Moore Threads domestic computing platforms.
+- Improved cross-page table merging success rate and merge quality.
+
+### 2.7.1 (2026/01/06)
+
+- Bug fix: #4300
+- Updated the `pdfminer.six` dependency to resolve [CVE-2025-64512](https://github.com/advisories/GHSA-wf5f-4jwr-ppcp)
+- Added automatic EXIF orientation correction for input images to improve OCR accuracy. #4283
+
+### 2.7.0 (2025/12/30)
+
+- Simplified installation. `uv pip install mineru[all]` now installs dependencies for all optional backends without requiring separate VLM engine installation.
+- Added the new `hybrid` backend, which combines the strengths of `pipeline` and `vlm`:
+ - Direct text extraction from text PDFs with native multilingual support and much lower hallucination risk;
+ - OCR in 109 languages for scanned PDFs when the OCR language is specified;
+ - An independent inline-formula toggle for improved rendering when inline formulas are not needed.
+- Simplified engine selection for `vlm/hybrid` backends. Users only need to choose `*-auto-engine` and MinerU selects an appropriate local inference engine automatically.
+- Switched the default backend from `pipeline` to `hybrid-auto-engine` for better out-of-the-box consistency.
+- Added i18n support to the Gradio application, enabling Chinese/English switching.
+
+---
+
+## 2.6 Series Versions
+
+### 2.6.7 (2025/12/12)
+
+- Bug fix: #4168
+
+### 2.6.6 (2025/12/02)
+
+**`mineru-api` tool optimizations**
+
+- Added descriptive text to `mineru-api` interface parameters to improve API documentation readability.
+- You can use the environment variable `MINERU_API_ENABLE_FASTAPI_DOCS` to control whether the auto-generated interface documentation page is enabled (enabled by default).
+- Added concurrency configuration options for the `vlm-vllm-async-engine`, `vlm-lmdeploy-engine`, and `vlm-http-client` backends. Users can use the environment variable `MINERU_API_MAX_CONCURRENT_REQUESTS` to set the maximum number of concurrent API requests (unlimited by default).
+
+### 2.6.5 (2025/11/26)
+
+- Added support for a new backend vlm-lmdeploy-engine. Its usage is similar to vlm-vllm-(async)engine, but it uses lmdeploy as the inference engine and additionally supports native inference acceleration on Windows platforms compared to vllm.
+
+### 2.6.4 (2025/11/04)
+
+- Added timeout configuration for PDF image rendering, default is 300 seconds, can be configured via environment variable `MINERU_PDF_RENDER_TIMEOUT` to prevent long blocking of the rendering process caused by some abnormal PDF files.
+- Added CPU thread count configuration options for ONNX models, default is the system CPU core count, can be configured via environment variables `MINERU_INTRA_OP_NUM_THREADS` and `MINERU_INTER_OP_NUM_THREADS` to reduce CPU resource contention conflicts in high concurrency scenarios.
+
+### 2.6.3 (2025/10/31)
+
+- Added support for a new backend `vlm-mlx-engine`, enabling MLX-accelerated inference for the MinerU2.5 model on Apple Silicon devices. Compared to the `vlm-transformers` backend, `vlm-mlx-engine` delivers a 100%–200% speed improvement.
+- Bug fixes: #3849, #3859
+
+### 2.6.2 (2025/10/24)
+
+**`pipeline` backend optimizations**
+
+- Added experimental support for Chinese formulas, which can be enabled by setting the environment variable `export MINERU_FORMULA_CH_SUPPORT=1`. This feature may cause a slight decrease in MFR speed and failures in recognizing some long formulas. It is recommended to enable it only when parsing Chinese formulas is needed. To disable this feature, set the environment variable to `0`.
+- `OCR` speed significantly improved by 200%~300%, thanks to the optimization solution provided by [@cjsdurj](https://github.com/cjsdurj)
+- `OCR` models optimized for improved accuracy and coverage of Latin script recognition, and updated Cyrillic, Arabic, Devanagari, Telugu (te), and Tamil (ta) language systems to `ppocr-v5` version, with accuracy improved by over 40% compared to previous models
+
+**`vlm` backend optimizations**
+
+- `table_caption` and `table_footnote` matching logic optimized to improve the accuracy of table caption and footnote matching and reading order rationality in scenarios with multiple consecutive tables on a page
+- Optimized CPU resource usage during high concurrency when using `vllm` backend, reducing server pressure
+- Adapted to `vllm` version 0.11.0
+
+**General optimizations**
+
+- Cross-page table merging effect optimized, added support for cross-page continuation table merging, improving table merging effectiveness in multi-column merge scenarios
+- Added environment variable configuration option `MINERU_TABLE_MERGE_ENABLE` for table merging feature. Table merging is enabled by default and can be disabled by setting this variable to `0`
+
+---
+
+## 2.5 Series Versions
+
+### 2.5.4 (2025/09/26)
+
+- 🎉🎉 The MinerU2.5 [Technical Report](https://arxiv.org/abs/2509.22186) is now available! We welcome you to read it for a comprehensive overview of its model architecture, training strategy, data engineering and evaluation results.
+- Fixed an issue where some `PDF` files were mistakenly identified as `AI` files, causing parsing failures
+
+### 2.5.3 (2025/09/20)
+
+- Dependency version range adjustment to enable Turing and earlier architecture GPUs to use vLLM acceleration for MinerU2.5 model inference.
+- `pipeline` backend compatibility fixes for torch 2.8.0.
+- Reduced default concurrency for vLLM async backend to lower server pressure and avoid connection closure issues caused by high load.
+- More compatibility-related details can be found in the [announcement](https://github.com/opendatalab/MinerU/discussions/3548)
+
+### 2.5.2 (2025/09/19)
+
+We are officially releasing MinerU2.5, currently the most powerful multimodal large model for document parsing.
+
+With only 1.2B parameters, MinerU2.5's accuracy on the OmniDocBench benchmark comprehensively surpasses top-tier multimodal models like Gemini 2.5 Pro, GPT-4o, and Qwen2.5-VL-72B. It also significantly outperforms leading specialized models such as dots.ocr, MonkeyOCR, and PP-StructureV3.
+
+The model has been released on [HuggingFace](https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B) and [ModelScope](https://modelscope.cn/models/opendatalab/MinerU2.5-2509-1.2B) platforms. Welcome to download and use!
+
+**Core Highlights**
+
+- SOTA Performance with Extreme Efficiency: As a 1.2B model, it achieves State-of-the-Art (SOTA) results that exceed models in the 10B and 100B+ classes, redefining the performance-per-parameter standard in document AI.
+- Advanced Architecture for Across-the-Board Leadership: By combining a two-stage inference pipeline (decoupling layout analysis from content recognition) with a native high-resolution architecture, it achieves SOTA performance across five key areas: layout analysis, text recognition, formula recognition, table recognition, and reading order.
+
+**Key Capability Enhancements**
+
+- Layout Detection: Delivers more complete results by accurately covering non-body content like headers, footers, and page numbers. It also provides more precise element localization and natural format reconstruction for lists and references.
+- Table Parsing: Drastically improves parsing for challenging cases, including rotated tables, borderless/semi-structured tables, and long/complex tables.
+- Formula Recognition: Significantly boosts accuracy for complex, long-form, and hybrid Chinese-English formulas, greatly enhancing the parsing capability for mathematical documents.
+
+**Repository Adjustments**
+
+Additionally, with the release of vlm 2.5, we have made some adjustments to the repository:
+
+- The vlm backend has been upgraded to version 2.5, supporting the MinerU2.5 model and no longer compatible with the MinerU2.0-2505-0.9B model. The last version supporting the 2.0 model is mineru-2.2.2.
+- VLM inference-related code has been moved to [mineru_vl_utils](https://github.com/opendatalab/mineru-vl-utils), reducing coupling with the main mineru repository and facilitating independent iteration in the future.
+- The vlm accelerated inference framework has been switched from `sglang` to `vllm`, achieving full compatibility with the vllm ecosystem, allowing users to use the MinerU2.5 model and accelerated inference on any platform that supports the vllm framework.
+- Due to major upgrades in the vlm model supporting more layout types, we have made some adjustments to the structure of the parsing intermediate file `middle.json` and result file `content_list.json`. Please refer to the [documentation](https://opendatalab.github.io/MinerU/reference/output_files/) for details.
+
+**Other Repository Optimizations**
+
+- Removed file extension whitelist validation for input files. When input files are PDF documents or images, there are no longer requirements for file extensions, improving usability.
+
+---
+
+## 2.2 - 2.4 Series Versions
+
+### 2.2.2 (2025/09/10)
+
+- Fixed the issue where the new table recognition model would affect the overall parsing task when some table parsing failed
+
+### 2.2.1 (2025/09/08)
+
+- Fixed the issue where some newly added models were not downloaded when using the model download command.
+
+### 2.2.0 (2025/09/05)
+
+**Major Updates**
+
+- In this version, we focused on improving table parsing accuracy by introducing a new [wired table recognition model](https://github.com/RapidAI/TableStructureRec) and a brand-new hybrid table structure parsing algorithm, significantly enhancing the table recognition capabilities of the `pipeline` backend.
+- We also added support for cross-page table merging, which is supported by both `pipeline` and `vlm` backends, further improving the completeness and accuracy of table parsing.
+
+**Other Updates**
+
+- The `pipeline` backend now supports 270-degree rotated table parsing, bringing support for table parsing in 0/90/270-degree orientations
+- `pipeline` added OCR capability support for Thai and Greek, and updated the English OCR model to the latest version. English recognition accuracy improved by 11%, Thai recognition model accuracy is 82.68%, and Greek recognition model accuracy is 89.28% (by PPOCRv5)
+- Added `bbox` field (mapped to 0-1000 range) in the output `content_list.json`, making it convenient for users to directly obtain position information for each content block
+- Removed the `pipeline_old_linux` installation option, no longer supporting legacy Linux systems such as `CentOS 7`, to provide better support for `uv`'s `sync`/`run` commands
+
+---
+
+## 2.1 Series Versions
+
+### 2.1.10 (2025/08/01)
+
+- Fixed an issue in the `pipeline` backend where block overlap caused the parsing results to deviate from expectations #3232
+
+### 2.1.9 (2025/07/30)
+
+- `transformers` 4.54.1 version adaptation
+
+### 2.1.8 (2025/07/28)
+
+- `sglang` 0.4.9.post5 version adaptation
+
+### 2.1.7 (2025/07/27)
+
+- `transformers` 4.54.0 version adaptation
+
+### 2.1.6 (2025/07/26)
+
+- Fixed table parsing issues in handwritten documents when using `vlm` backend
+- Fixed visualization box position drift issue when document is rotated #3175
+
+### 2.1.5 (2025/07/24)
+
+- `sglang` 0.4.9 version adaptation, synchronously upgrading the dockerfile base image to sglang 0.4.9.post3
+
+### 2.1.4 (2025/07/23)
+
+**Bug Fixes**
+
+- Fixed the issue of excessive memory consumption during the `MFR` step in the `pipeline` backend under certain scenarios #2771
+- Fixed the inaccurate matching between `image`/`table` and `caption`/`footnote` under certain conditions #3129
+
+### 2.1.1 (2025/07/16)
+
+**Bug fixes**
+
+- Fixed text block content loss issue that could occur in certain `pipeline` scenarios #3005
+- Fixed issue where `sglang-client` required unnecessary packages like `torch` #2968
+- Updated `dockerfile` to fix incomplete text content parsing due to missing fonts in Linux #2915
+
+**Usability improvements**
+
+- Updated `compose.yaml` to facilitate direct startup of `sglang-server`, `mineru-api`, and `mineru-gradio` services
+- Launched brand new [online documentation site](https://opendatalab.github.io/MinerU/), simplified readme, providing better documentation experience
+
+### 2.1.0 (2025/07/05)
+
+This is the first major update of MinerU 2, which includes a large number of new features and improvements, covering significant performance optimizations, user experience enhancements, and bug fixes. The detailed update contents are as follows:
+
+**Performance Optimizations**
+
+- Significantly improved preprocessing speed for documents with specific resolutions (around 2000 pixels on the long side).
+- Greatly enhanced post-processing speed when the `pipeline` backend handles batch processing of documents with fewer pages (<10 pages).
+- Layout analysis speed of the `pipeline` backend has been increased by approximately 20%.
+
+**Experience Enhancements**
+
+- Built-in ready-to-use `fastapi service` and `gradio webui`. For detailed usage instructions, please refer to [Documentation](https://opendatalab.github.io/MinerU/usage/quick_usage/#advanced-usage-via-api-webui-sglang-clientserver).
+- Adapted to `sglang` version `0.4.8`, significantly reducing the GPU memory requirements for the `vlm-sglang` backend. It can now run on graphics cards with as little as `8GB GPU memory` (Turing architecture or newer).
+- Added transparent parameter passing for all commands related to `sglang`, allowing the `sglang-engine` backend to receive all `sglang` parameters consistently with the `sglang-server`.
+- Supports feature extensions based on configuration files, including `custom formula delimiters`, `enabling heading classification`, and `customizing local model directories`. For detailed usage instructions, please refer to [Documentation](https://opendatalab.github.io/MinerU/usage/quick_usage/#extending-mineru-functionality-with-configuration-files).
+
+**New Features**
+
+- Updated the `pipeline` backend with the PP-OCRv5 multilingual text recognition model, supporting text recognition in 37 languages such as French, Spanish, Portuguese, Russian, and Korean, with an average accuracy improvement of over 30%. [Details](https://paddlepaddle.github.io/PaddleOCR/latest/en/version3.x/algorithm/PP-OCRv5/PP-OCRv5_multi_languages.html)
+- Introduced limited support for vertical text layout in the `pipeline` backend.
+
+---
+
+## 2.0 Series Versions
+
+### 2.0.6 (2025/06/20)
+
+- Fixed occasional parsing interruptions caused by invalid block content in `vlm` mode
+- Fixed parsing interruptions caused by incomplete table structures in `vlm` mode
+
+### 2.0.5 (2025/06/17)
+
+- Fixed the issue where models were still required to be downloaded in the `sglang-client` mode
+- Fixed the issue where the `sglang-client` mode unnecessarily depended on packages like `torch` during runtime.
+- Fixed the issue where only the first instance would take effect when attempting to launch multiple `sglang-client` instances via multiple URLs within the same process
+
+### 2.0.3 (2025/06/15)
+
+- Fixed a configuration file key-value update error that occurred when downloading model type was set to `all`
+- Fixed the issue where the formula and table feature toggle switches were not working in `command line mode`, causing the features to remain enabled.
+- Fixed compatibility issues with sglang version 0.4.7 in the `sglang-engine` mode.
+- Updated Dockerfile and installation documentation for deploying the full version of MinerU in sglang environment
+
+### 2.0.0 (2025/06/13)
+
+**New Architecture**
+
+MinerU 2.0 has been deeply restructured in code organization and interaction methods, significantly improving system usability, maintainability, and extensibility.
+
+- **Removal of Third-party Dependency Limitations**: Completely eliminated the dependency on `pymupdf`, moving the project toward a more open and compliant open-source direction.
+- **Ready-to-use, Easy Configuration**: No need to manually edit JSON configuration files; most parameters can now be set directly via command line or API.
+- **Automatic Model Management**: Added automatic model download and update mechanisms, allowing users to complete model deployment without manual intervention.
+- **Offline Deployment Friendly**: Provides built-in model download commands, supporting deployment requirements in completely offline environments.
+- **Streamlined Code Structure**: Removed thousands of lines of redundant code, simplified class inheritance logic, significantly improving code readability and development efficiency.
+- **Unified Intermediate Format Output**: Adopted standardized `middle_json` format, compatible with most secondary development scenarios based on this format, ensuring seamless ecosystem business migration.
+
+**New Model**
+
+MinerU 2.0 integrates our latest small-parameter, high-performance multimodal document parsing model, achieving end-to-end high-speed, high-precision document understanding.
+
+- **Small Model, Big Capabilities**: With parameters under 1B, yet surpassing traditional 72B-level vision-language models (VLMs) in parsing accuracy.
+- **Multiple Functions in One**: A single model covers multilingual recognition, handwriting recognition, layout analysis, table parsing, formula recognition, reading order sorting, and other core tasks.
+- **Ultimate Inference Speed**: Achieves peak throughput exceeding 10,000 tokens/s through `sglang` acceleration on a single NVIDIA 4090 card, easily handling large-scale document processing requirements.
+- **Online Experience**: You can experience our brand-new VLM model on [MinerU.net](https://mineru.net/OpenSourceTools/Extractor), [Hugging Face](https://huggingface.co/spaces/opendatalab/MinerU), and [ModelScope](https://www.modelscope.cn/studios/OpenDataLab/MinerU).
+
+**Incompatible Changes Notice**
+
+To improve overall architectural rationality and long-term maintainability, this version contains some incompatible changes:
+
+- Python package name changed from `magic-pdf` to `mineru`, and the command-line tool changed from `magic-pdf` to `mineru`. Please update your scripts and command calls accordingly.
+- For modular system design and ecosystem consistency considerations, MinerU 2.0 no longer includes the LibreOffice document conversion module. If you need to process Office documents, we recommend converting them to PDF format through an independently deployed LibreOffice service before proceeding with subsequent parsing operations.
+
+---
+
+## 1.x Series Historical Versions
+
+### 1.3.12 (2025/05/24)
+
+Added support for PPOCRv5 models, updated `ch_server` model to `PP-OCRv5_rec_server`, and `ch_lite` model to `PP-OCRv5_rec_mobile` (model update required)
+
+- In testing, we found that PPOCRv5(server) has some improvement for handwritten documents, but has slightly lower accuracy than v4_server_doc for other document types, so the default ch model remains unchanged as `PP-OCRv4_server_rec_doc`.
+- Since PPOCRv5 has enhanced recognition capabilities for handwriting and special characters, you can manually choose the PPOCRv5 model for Japanese-Traditional Chinese mixed scenarios and handwritten documents
+- You can select the appropriate model through the lang parameter `lang='ch_server'` (Python API) or `--lang ch_server` (command line):
+ - `ch`: `PP-OCRv4_server_rec_doc` (default) (Chinese/English/Japanese/Traditional Chinese mixed/15K dictionary)
+ - `ch_server`: `PP-OCRv5_rec_server` (Chinese/English/Japanese/Traditional Chinese mixed + handwriting/18K dictionary)
+ - `ch_lite`: `PP-OCRv5_rec_mobile` (Chinese/English/Japanese/Traditional Chinese mixed + handwriting/18K dictionary)
+ - `ch_server_v4`: `PP-OCRv4_rec_server` (Chinese/English mixed/6K dictionary)
+ - `ch_lite_v4`: `PP-OCRv4_rec_mobile` (Chinese/English mixed/6K dictionary)
+
+Added support for handwritten documents through optimized layout recognition of handwritten text areas
+
+- This feature is supported by default, no additional configuration required
+- You can refer to the instructions above to manually select the PPOCRv5 model for better handwritten document parsing results
+
+The `huggingface` and `modelscope` demos have been updated to versions that support handwriting recognition and PPOCRv5 models, which you can experience online
+
+### 1.3.10 (2025/04/29)
+
+- Added support for custom formula delimiters, which can be configured by modifying the `latex-delimiter-config` section in the `magic-pdf.json` file in your user directory.
+
+### 1.3.9 (2025/04/27)
+
+- Optimized formula parsing functionality, improved formula rendering success rate
+
+### 1.3.8 (2025/04/23)
+
+The default `ocr` model (`ch`) has been updated to `PP-OCRv4_server_rec_doc` (model update required)
+
+- `PP-OCRv4_server_rec_doc` is trained on a mixture of more Chinese document data and PP-OCR training data based on `PP-OCRv4_server_rec`, adding recognition capabilities for some traditional Chinese characters, Japanese, and special characters. It can recognize over 15,000 characters and improves both document-specific and general text recognition abilities.
+- [Performance comparison of PP-OCRv4_server_rec_doc/PP-OCRv4_server_rec/PP-OCRv4_mobile_rec](https://paddlepaddle.github.io/PaddleX/latest/module_usage/tutorials/ocr_modules/text_recognition.html#_3)
+- After verification, the `PP-OCRv4_server_rec_doc` model shows significant accuracy improvements in Chinese/English/Japanese/Traditional Chinese in both single language and mixed language scenarios, with comparable speed to `PP-OCRv4_server_rec`, making it suitable for most use cases.
+- In some pure English scenarios, `PP-OCRv4_server_rec_doc` may have word adhesion issues, while `PP-OCRv4_server_rec` performs better in these cases. Therefore, we've kept the `PP-OCRv4_server_rec` model, which users can access by adding the parameter `lang='ch_server'` (Python API) or `--lang ch_server` (command line).
+
+### 1.3.7 (2025/04/22)
+
+- Fixed the issue where the lang parameter was ineffective during table parsing model initialization
+- Fixed the significant speed reduction of OCR and table parsing in `cpu` mode
+
+### 1.3.4 (2025/04/16)
+
+- Slightly improved OCR-det speed by removing some unnecessary blocks
+- Fixed page-internal sorting errors caused by footnotes in certain cases
+
+### 1.3.2 (2025/04/12)
+
+- Fixed dependency version incompatibility issues when installing on Windows with Python 3.13
+- Optimized memory usage during batch inference
+- Improved parsing of tables rotated 90 degrees
+- Enhanced parsing of oversized tables in financial report samples
+- Fixed the occasional word adhesion issue in English text areas when OCR language is not specified (model update required)
+
+### 1.3.1 (2025/04/08)
+
+Fixed several compatibility issues
+
+- Added support for Python 3.13
+- Made final adaptations for outdated Linux systems (such as CentOS 7) with no guarantee of continued support in future versions, [installation instructions](https://github.com/opendatalab/MinerU/issues/1004)
+
+### 1.3.0 (2025/04/03)
+
+**Installation and compatibility optimizations**
+
+- Resolved compatibility issues caused by `detectron2` by removing `layoutlmv3` usage in layout
+- Extended torch version compatibility to 2.2~2.6 (excluding 2.5)
+- Added CUDA compatibility for versions 11.8/12.4/12.6/12.8 (CUDA version determined by torch), solving compatibility issues for users with 50-series and H-series GPUs
+- Extended Python compatibility to versions 3.10~3.12, fixing the issue of automatic downgrade to version 0.6.1 when installing in non-3.10 environments
+- Optimized offline deployment process, eliminating the need to download any model files after successful deployment
+
+**Performance optimizations**
+
+- Enhanced parsing speed for batches of small files by supporting batch processing of multiple PDF files ([script example](demo/batch_demo.py)), with formula parsing speed improved by up to 1400% and overall parsing speed improved by up to 500% compared to version 1.0.1
+- Reduced memory usage and improved parsing speed by optimizing MFR model loading and usage (requires re-running the [model download process](docs/how_to_download_models_zh_cn.md) to get incremental updates to model files)
+- Optimized GPU memory usage, requiring only 6GB minimum to run this project
+- Improved running speed on MPS devices
+
+**Parsing effect optimizations**
+
+- Updated MFR model to `unimernet(2503)`, fixing line break loss issues in multi-line formulas
+
+**Usability optimizations**
+
+- Completely replaced the `paddle` framework and `paddleocr` in the project by using `paddleocr2torch`, resolving conflicts between `paddle` and `torch`, as well as thread safety issues caused by the `paddle` framework
+- Added real-time progress bar display during parsing, allowing precise tracking of parsing progress and making the waiting process more bearable
+
+### 1.2.1 (2025/03/03)
+
+Fixed some issues
+
+- Fixed the impact on punctuation marks during full-width to half-width conversion of letters and numbers
+- Fixed caption matching inaccuracies in certain scenarios
+- Fixed formula span loss issues in certain scenarios
+
+### 1.2.0 (2025/02/24)
+
+This version includes several fixes and improvements to enhance parsing efficiency and accuracy:
+
+**Performance Optimization**
+
+- Increased classification speed for PDF documents in auto mode.
+
+**Parsing Optimization**
+
+- Improved parsing logic for documents containing watermarks, significantly enhancing the parsing results for such documents.
+- Enhanced the matching logic for multiple images/tables and captions within a single page, improving the accuracy of image-text matching in complex layouts.
+
+**Bug Fixes**
+
+- Fixed an issue where image/table spans were incorrectly filled into text blocks under certain conditions.
+- Resolved an issue where title blocks were empty in some cases.
+
+### 1.1.0 (2025/01/22)
+
+In this version we have focused on improving parsing accuracy and efficiency:
+
+**Model capability upgrade** (requires re-executing the [model download process](https://github.com/opendatalab/MinerU/blob/master/docs/how_to_download_models_en.md) to obtain incremental updates of model files)
+
+- The layout recognition model has been upgraded to the latest `doclayout_yolo(2501)` model, improving layout recognition accuracy.
+- The formula parsing model has been upgraded to the latest `unimernet(2501)` model, improving formula recognition accuracy.
+
+**Performance optimization**
+
+- On devices that meet certain configuration requirements (16GB+ VRAM), by optimizing resource usage and restructuring the processing pipeline, overall parsing speed has been increased by more than 50%.
+
+**Parsing effect optimization**
+
+- Added a new heading classification feature (testing version, enabled by default) to the online demo ([mineru.net](https://mineru.net/OpenSourceTools/Extractor)/[huggingface](https://huggingface.co/spaces/opendatalab/MinerU)/[modelscope](https://www.modelscope.cn/studios/OpenDataLab/MinerU)), which supports hierarchical classification of headings, thereby enhancing document structuring.
+
+### 1.0.1 (2025/01/10)
+
+This is our first official release, where we have introduced a completely new API interface and enhanced compatibility through extensive refactoring, as well as a brand new automatic language identification feature:
+
+**New API Interface**
+
+- For the data-side API, we have introduced the Dataset class, designed to provide a robust and flexible data processing framework. This framework currently supports a variety of document formats, including images (.jpg and .png), PDFs, Word documents (.doc and .docx), and PowerPoint presentations (.ppt and .pptx). It ensures effective support for data processing tasks ranging from simple to complex.
+- For the user-side API, we have meticulously designed the MinerU processing workflow as a series of composable Stages. Each Stage represents a specific processing step, allowing users to define new Stages according to their needs and creatively combine these stages to customize their data processing workflows.
+
+**Enhanced Compatibility**
+
+- By optimizing the dependency environment and configuration items, we ensure stable and efficient operation on ARM architecture Linux systems.
+- We have deeply integrated with Huawei Ascend NPU acceleration, providing autonomous and controllable high-performance computing capabilities. This supports the localization and development of AI application platforms in China. [Ascend NPU Acceleration](https://github.com/opendatalab/MinerU/blob/master/docs/README_Ascend_NPU_Acceleration_zh_CN.md)
+
+**Automatic Language Identification**
+
+- By introducing a new language recognition model, setting the `lang` configuration to `auto` during document parsing will automatically select the appropriate OCR language model, improving the accuracy of scanned document parsing.
+
+---
+
+## 0.x Series Historical Versions
+
+### 0.10.0 (2024/11/22)
+
+Introducing hybrid OCR text extraction capabilities:
+
+- Significantly improved parsing performance in complex text distribution scenarios such as dense formulas, irregular span regions, and text represented by images.
+- Combines the dual advantages of accurate content extraction and faster speed in text mode, and more precise span/line region recognition in OCR mode.
+
+### 0.9.3 (2024/11/15)
+
+Integrated [RapidTable](https://github.com/RapidAI/RapidTable) for table recognition, improving single-table parsing speed by more than 10 times, with higher accuracy and lower GPU memory usage.
+
+### 0.9.2 (2024/11/06)
+
+Integrated the [StructTable-InternVL2-1B](https://huggingface.co/U4R/StructTable-InternVL2-1B) model for table recognition functionality.
+
+### 0.9.0 (2024/10/31)
+
+This is a major new version with extensive code refactoring, addressing numerous issues, improving performance, reducing hardware requirements, and enhancing usability:
+
+- Refactored the sorting module code to use [layoutreader](https://github.com/ppaanngggg/layoutreader) for reading order sorting, ensuring high accuracy in various layouts.
+- Refactored the paragraph concatenation module to achieve good results in cross-column, cross-page, cross-figure, and cross-table scenarios.
+- Refactored the list and table of contents recognition functions, significantly improving the accuracy of list blocks and table of contents blocks, as well as the parsing of corresponding text paragraphs.
+- Refactored the matching logic for figures, tables, and descriptive text, greatly enhancing the accuracy of matching captions and footnotes to figures and tables, and reducing the loss rate of descriptive text to near zero.
+- Added multi-language support for OCR, supporting detection and recognition of 84 languages. For the list of supported languages, see [OCR Language Support List](https://paddlepaddle.github.io/PaddleOCR/latest/en/ppocr/blog/multi_languages.html#5).
+- Added memory recycling logic and other memory optimization measures, significantly reducing memory usage. The memory requirement for enabling all acceleration features except table acceleration (layout/formula/OCR) has been reduced from 16GB to 8GB, and the memory requirement for enabling all acceleration features has been reduced from 24GB to 10GB.
+- Optimized configuration file feature switches, adding an independent formula detection switch to significantly improve speed and parsing results when formula detection is not needed.
+- Integrated [PDF-Extract-Kit 1.0](https://github.com/opendatalab/PDF-Extract-Kit):
+ - Added the self-developed `doclayout_yolo` model, which speeds up processing by more than 10 times compared to the original solution while maintaining similar parsing effects, and can be freely switched with `layoutlmv3` via the configuration file.
+ - Upgraded formula parsing to `unimernet 0.2.1`, improving formula parsing accuracy while significantly reducing memory usage.
+ - Due to the repository change for `PDF-Extract-Kit 1.0`, you need to re-download the model. Please refer to [How to Download Models](https://github.com/opendatalab/MinerU/blob/master/docs/how_to_download_models_en.md) for detailed steps.
+
+### 0.8.1 (2024/09/27)
+
+Fixed some bugs, and providing a [localized deployment version](https://github.com/opendatalab/MinerU/blob/master/projects/web_demo/README.md) of the [online demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/) and the [front-end interface](https://github.com/opendatalab/MinerU/blob/master/projects/web/README.md).
+
+### 0.8.0 (2024/09/09)
+
+Supporting fast deployment with Dockerfile, and launching demos on Huggingface and Modelscope.
+
+### 0.7.1 (2024/08/30)
+
+Add paddle tablemaster table recognition option
+
+### 0.7.0b1 (2024/08/09)
+
+Simplified installation process, added table recognition functionality
+
+### 0.6.2b1 (2024/08/01)
+
+Optimized dependency conflict issues and installation documentation
+
+### Initial Open-Source Release (2024/07/05)
+
+MinerU project's first open-source release
+
diff --git a/docs/en/reference/index.md b/docs/en/reference/index.md
new file mode 100644
index 000000000..638dbe36c
--- /dev/null
+++ b/docs/en/reference/index.md
@@ -0,0 +1,27 @@
+# Reference Documentation
+
+This section provides detailed reference materials for MinerU project. Here you can find technical specifications, API documentation, output file formats, and version history.
+
+## Table of Contents
+
+- [Output Files Documentation](./output_files.md) - Detailed explanation of all output files and their formats
+- [Changelog](./changelog.md) - Version update history and release notes
+
+## Documentation Overview
+
+### Output Files Documentation
+
+Understanding the output files generated by MinerU is crucial for effective use of the tool. The output files documentation provides:
+
+- **Visual debugging files**: Help you understand the document parsing process
+- **Structured data files**: Contain detailed parsing results for further processing
+- **File format specifications**: Detailed descriptions of each output file type
+
+### Changelog
+
+The changelog documents the evolution of MinerU, including:
+
+- **Version updates**: New features and improvements for each release
+- **Bug fixes**: Issues resolved in each version
+- **Breaking changes**: Important changes that may affect your usage
+- **Deprecations**: Features that are being phased out
diff --git a/docs/en/reference/output_files.md b/docs/en/reference/output_files.md
new file mode 100644
index 000000000..7c60c73bc
--- /dev/null
+++ b/docs/en/reference/output_files.md
@@ -0,0 +1,736 @@
+# MinerU Output Files Documentation
+
+## Overview
+
+After executing the `mineru` command, in addition to the main markdown file output, multiple auxiliary files are generated for debugging, quality inspection, and further processing. These files include:
+
+The exact set of generated files depends on the backend and the input document type.
+
+- **Visual debugging files**: Help users intuitively understand the document parsing process and results
+- **Structured data files**: Contain detailed parsing data for secondary development
+- In multimodal markdown output, `image` / `chart` blocks render the screenshot first; when `content` exists, a collapsed HTML `
MinerU — 专为 LLM · RAG · Agent 场景构建的高精度文档解析引擎
+将 PDF · DOCX · PPTX · XLSX · 图片 · 网页转为结构化 Markdown / JSON · VLM+OCR 双引擎 · 109 种语言+MCP Server · LangChain / Dify / FastGPT 原生集成 · 10+ 国产算力适配
+ +**🔍 核心解析能力** +- 原生支持 `DOCX`、`PPTX`、`XLSX` 解析 +- 公式 → LaTeX · 表格 → HTML，精准还原复杂版面 +- 支持扫描件、手写体、多栏布局、跨页表格合并 +- 输出符合人类阅读顺序，自动去除页眉页脚 +- VLM + OCR 双引擎，支持 109 种语言识别 + +**🔌 接入方式** + +| 场景 | 方案 | +|------|------| +| AI 编程工具 | MCP Server — Cursor · Claude Desktop · Windsurf | +| RAG 框架 | LangChain · LlamaIndex · RAGFlow · RAG-Anything · Flowise · Dify · FastGPT | +| 开发集成 | Python / Go / TypeScript SDK · CLI · REST API · Docker | +| 零代码 | mineru.net 在线版 · Gradio WebUI · 桌面客户端 | + +**🖥️ 部署生态（支持私有化 · 完全离线）** + +| 推理后端 | 适用场景 | +|--------------|-----------------------------| +| pipeline | 快速稳定，无幻觉，CPU / GPU 均可运行 | +| vlm-engine | 高精度，支持 vLLM / LMdeploy / mlx 生态 | +| hybrid-engine| 高精度，原生文本提取，低幻觉 | + +国产算力：昇腾 · 寒武纪 · 燧原 · 沐曦 · 摩尔线程 · 昆仑芯 · 天数智芯 · 瀚博 · 太初元碁 · 海光 · 平头哥 -- 2024/10/31 0.9.0发布，这是我们进行了大量代码重构的全新版本，解决了众多问题，提升了性能，降低了硬件需求，并提供了更丰富的易用性： - - 重构排序模块代码，使用 [layoutreader](https://github.com/ppaanngggg/layoutreader) 进行阅读顺序排序，确保在各种排版下都能实现极高准确率 - - 重构段落拼接模块，在跨栏、跨页、跨图、跨表情况下均能实现良好的段落拼接效果 - - 重构列表和目录识别功能，极大提升列表块和目录块识别的准确率及对应文本段落的解析效果 - - 重构图、表与描述性文本的匹配逻辑，大幅提升 caption 和 footnote 与图表的匹配准确率，并将描述性文本的丢失率降至零 - - 增加 OCR 的多语言支持，支持 84 种语言的检测与识别，语言支持列表详见 [OCR 语言支持列表](https://paddlepaddle.github.io/PaddleOCR/latest/ppocr/blog/multi_languages.html#5) - - 增加显存回收逻辑及其他显存优化措施，大幅降低显存使用需求。开启除表格加速外的全部加速功能(layout/公式/OCR)的显存需求从16GB降至8GB，开启全部加速功能的显存需求从24GB降至10GB - - 优化配置文件的功能开关，增加独立的公式检测开关，无需公式检测时可大幅提升速度和解析效果 - - 集成 [PDF-Extract-Kit 1.0](https://github.com/opendatalab/PDF-Extract-Kit) - - 加入自研的 `doclayout_yolo` 模型，在相近解析效果情况下比原方案提速10倍以上，可通过配置文件与 `layoutlmv3` 自由切换 - - 公式解析升级至 `unimernet 0.2.1`，在提升公式解析准确率的同时，大幅降低显存需求 -- 2024/09/27 0.8.1发布，修复了一些bug，同时提供了[在线demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/)的[本地化部署版本](projects/web_demo/README_zh-CN.md)和[前端界面](projects/web/README_zh-CN.md) -- 2024/09/09 0.8.0发布，支持Dockerfile快速部署，同时上线了huggingface、modelscope demo -- 2024/08/30 0.7.1发布，集成了paddle tablemaster表格识别功能 -- 2024/08/09 0.7.0b1发布，简化安装步骤提升易用性，加入表格识别功能 -- 2024/08/01 0.6.2b1发布，优化了依赖冲突问题和安装文档 -- 2024/07/05 首次开源 - - - -
-
-
+# 更新记录
+
+- 2026/04/18 3.1.0 发布
+
+ 本次版本更新聚焦于**许可协议开放性、解析精度提升与全格式原生支持**。主要更新内容包括：
+
+ - 许可协议升级
+ - MinerU 已正式从 `AGPLv3` 切换至基于 `Apache 2.0` 的 [MinerU 开源许可证](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md)。
+ - 新的许可方式在兼顾开源协作与商业落地的同时，进一步降低了社区使用和商业化接入门槛，让 MinerU 更容易融入真实业务流程。
+ - VLM 主模型升级
+ - VLM 主模型正式切换为 `MinerU2.5-Pro-2604-1.2B`，整体解析精度提升至业内领先水平。
+ - 新模型现已支持子图切分合并、图像与图表解析、截断段落合并、跨页面表格合并以及表格内图像识别，复杂版面场景下的解析能力进一步增强。
+ - 全格式原生解析支持
+ - 新增 `PPTX` 与 `XLSX` 原生解析能力。
+ - 至此，MinerU 已完整支持图片、`PDF`、`DOCX`、`PPTX`、`XLSX` 全格式解析，为多类型文档统一处理提供了更完整的能力闭环。
+
+ 通过 3.1.0 版本，MinerU 在开放性、解析精度和落地能力上进一步提升。新的许可协议降低了社区使用和商业接入门槛，`MinerU2.5-Pro-2604-1.2B` 提升了复杂内容的解析质量，而 `PPTX` 与 `XLSX` 原生解析的补齐，也让 MinerU 完成了主流文档格式的端到端覆盖。
+
+- 2026/03/29 3.0.0 发布
+
+ 本次版本更新围绕**解析能力、系统架构与工程可用性**进行了系统升级。主要更新内容包括：
+
+ - `DOCX` 原生解析
+ - 正式支持 `DOCX` 原生解析，在无幻觉前提下实现高精度解析。
+ - 相较于“先将 `DOCX` 转为 `PDF` 再解析”的传统流程，端到端速度提升数十倍以上，更适合对精度与吞吐均有要求的场景。
+ - `pipeline` 后端升级
+ - `pipeline` 后端在 OmniDocBench (v1.5) 上取得 `86.2` 分，精度超过上一代主流 VLM `MinerU2.0-2505-0.9B`。
+ - 新增表格内图片/公式解析、印章文字识别、竖排文本支持、行间公式序号识别等能力，持续提升复杂文档场景下的解析效果。
+ - 在保持高精度的同时，资源占用极低，并继续支持纯 CPU 环境推理。
+ - `API / CLI / Router` 编排升级
+ - `mineru` 现作为基于 `mineru-api` 的编排客户端运行；在未传入 `--api-url` 时，会自动拉起本地临时服务。
+ - `mineru-api` 新增异步任务接口 `POST /tasks`，支持任务提交、状态查询与结果获取；同时保留同步解析接口 `POST /file_parse`，以兼容老版本插件。
+ - 新增 `mineru-router`，适用于多服务、多 GPU 的统一入口部署与任务路由；其接口与 `mineru-api` 完全兼容，并支持任务自动负载均衡。
+ - 部署与使用体验优化
+ - 解决了 `torch >= 2.8` 的兼容问题，基础镜像升级为 `vllm0.11.2 + torch2.9.0`，统一了不同 Compute Capability 的安装路径。
+ - 通过滑动窗口优化解析链路，显著降低长文档场景下的内存峰值占用，上万页文档解析不再需要手动拆分。
+ - `pipeline` 的 batch 推理支持流式落盘，已完成的解析结果可及时写出，进一步提升长任务处理体验。
+ - 完成线程安全优化，全面支持多线程并发推理；配合 `mineru-router`，可一键实现多卡部署，轻松构建高并发、高吞吐解析系统。
+ - 完全移除了两个 AGPLv3 模型（`doclayoutyolo` 和 `mfd_yolov8`）以及一个 CC-BY-NC-SA 4.0 模型（`layoutreader`）的使用。
+
+ 本次更新不仅是若干功能点的补强，更是 MinerU 在系统能力上的一次关键跃迁。我们重点解决了长文档解析过程中的内存峰值占用问题，通过滑动窗口、流式落盘等链路优化，让超长文档解析从“需要手动拆分、谨慎处理”走向“稳定可跑、规模可扩展”。同时，我们完成了线程安全优化，全面支持多线程并发推理，进一步提升了单机资源利用率与高并发场景下的运行稳定性。在此基础上，基于 mineru-router 与全新的 API / CLI 编排体系，MinerU 已具备一键多卡部署、多服务统一接入、任务自动负载均衡的能力，显著降低了大规模部署难度。至此，MinerU 正在从单一的数据生产工具，进一步演进为面向高并发、高吞吐场景的大规模文档解析基座，为企业级文档数据处理提供更稳定、更高效、更易扩展的基础设施能力。
+
+> 📝 查看完整的 [更新日志](https://opendatalab.github.io/MinerU/zh/reference/changelog/) 了解更多历史版本信息
+
# MinerU
## 项目简介
-MinerU是一款将PDF转化为机器可读格式的工具（如markdown、json），可以很方便地抽取为任意格式。
+MinerU 是一款文档解析工具，可将 `PDF`、图片以及 `DOCX`、`PPTX`、`XLSX` 转化为机器可读格式（如 Markdown、JSON），便于后续检索、抽取与二次处理。
MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练过程中，我们将会集中精力解决科技文献中的符号转化问题，希望在大模型时代为科技发展做出贡献。
-相比国内外知名商用产品MinerU还很年轻，如果遇到问题或者结果不及预期请到[issue](https://github.com/opendatalab/MinerU/issues)提交问题，同时**附上相关PDF**。
+相比国内外知名商用产品MinerU还很年轻，如果遇到问题或者结果不及预期请到[issue](https://github.com/opendatalab/MinerU/issues)提交问题，同时**附上相关文档或样例文件**。
https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c
## 主要功能
+- 支持 `PDF`、图片与 `DOCX`、`PPTX`、`XLSX` 输入
- 删除页眉、页脚、脚注、页码等元素，确保语义连贯
- 输出符合人类阅读顺序的文本，适用于单栏、多栏及复杂排版
- 保留原文档的结构，包括标题、段落、列表等
- 提取图像、图片描述、表格、表格标题及脚注
- 自动识别并转换文档中的公式为LaTeX格式
-- 自动识别并转换文档中的表格为LaTeX或HTML格式
+- 自动识别并转换文档中的表格为HTML格式
- 自动检测扫描版PDF和乱码PDF，并启用OCR功能
-- OCR支持84种语言的检测与识别
+- OCR支持109种语言的检测与识别
- 支持多种输出格式，如多模态与NLP的Markdown、按阅读顺序排序的JSON、含有丰富信息的中间格式等
- 支持多种可视化结果，包括layout可视化、span可视化等，便于高效确认输出效果与质检
-- 支持CPU和GPU环境
+- 内置命令行、FastAPI、Gradio WebUI，支持本地编排和多服务部署
+- 支持纯CPU环境运行，并支持 GPU(CUDA)/NPU(CANN)/MPS 加速
- 兼容Windows、Linux和Mac平台
-## 快速开始
+# 快速开始
-如果遇到任何安装问题，请先查询 FAQ
+如果安装或使用中遇到任何问题，请先查询 FAQ
如果遇到解析效果不及预期，参考 Known Issues
-有3种不同方式可以体验MinerU的效果：
-- [在线体验(无需任何安装)](#在线体验)
-- [使用CPU快速体验（Windows，Linux，Mac）](#使用cpu快速体验)
-- [Linux/Windows + CUDA](#使用gpu)
+## 在线体验
-**⚠️安装前必看——软硬件环境支持说明**
+### 官网在线应用
+官网在线版功能与客户端一致，界面美观，功能丰富，需要登录使用
+
+- [](https://mineru.net/OpenSourceTools/Extractor?source=github)
-为了确保项目的稳定性和可靠性，我们在开发过程中仅对特定的软硬件环境进行优化和测试。这样当用户在推荐的系统配置上部署和运行项目时，能够获得最佳的性能表现和最少的兼容性问题。
+### 基于Gradio的在线demo
+基于gradio开发的webui，界面简洁，仅包含核心解析功能，免登录
-通过集中资源和精力于主线环境，我们团队能够更高效地解决潜在的BUG，及时开发新功能。
+- [](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+- [](https://huggingface.co/spaces/opendatalab/MinerU)
-在非主线环境中，由于硬件、软件配置的多样性，以及第三方依赖项的兼容性问题，我们无法100%保证项目的完全可用性。因此，对于希望在非推荐环境中使用本项目的用户，我们建议先仔细阅读文档以及FAQ，大多数问题已经在FAQ中有对应的解决方案，除此之外我们鼓励社区反馈问题，以便我们能够逐步扩大支持范围。
+## 本地部署
+
+> [!WARNING]
+> **安装前必看——软硬件环境支持说明**
+>
+> 为了确保项目的稳定性和可靠性，我们在开发过程中仅对特定的软硬件环境进行优化和测试。这样当用户在推荐的系统配置上部署和运行项目时，能够获得最佳的性能表现和最少的兼容性问题。
+>
+> 通过集中资源和精力于主线环境，我们团队能够更高效地解决潜在的BUG，及时开发新功能。
+>
+> 在非主线环境中，由于硬件、软件配置的多样性，以及第三方依赖项的兼容性问题，我们无法100%保证项目的完全可用性。因此，对于希望在非推荐环境中使用本项目的用户，我们建议先仔细阅读文档以及FAQ，大多数问题已经在FAQ中有对应的解决方案，除此之外我们鼓励社区反馈问题，以便我们能够逐步扩大支持范围。
文档目录
- -
-
- - MinerU - - -
- TODO -
- Known Issues -
- FAQ -
- Contributors -
- License Information -
- Acknowledgements -
- Citation -
- Star History -
- magic-doc快速提取PPT/DOC/PDF -
- magic-html提取混合网页内容 -
- Links -
| 操作系统 | +解析后端 | +pipeline | +*-auto-engine | +*-http-client | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Ubuntu 22.04 LTS | -Windows 10 / 11 | -macOS 11+ | +hybrid | +vlm | +hybrid | +vlm | |||||
| 后端特性 | +兼容性好 | +硬件配置要求较高 | +适用于OpenAI兼容服务器2 | +||||||||
| CPU | -x86_64(暂不支持ARM Linux) | -x86_64(暂不支持ARM Windows) | -x86_64 / arm64 | +精度指标1 | +86+ | +90+ | |||||
| 内存 | -大于等于16GB，推荐32G以上 | +操作系统 | +Linux3 / Windows4 / macOS5 | ||||||||
| python版本 | -3.10 (请务必通过conda创建3.10虚拟环境) | +纯CPU平台支持 | +✅ | +❌ | +✅ | +||||||
| GPU加速支持 | +Volta及以后架构GPU或Apple Silicon | +不需要 | |||||||||
| Nvidia Driver 版本 | -latest(专有驱动) | -latest | -None | +显存最低要求 | +4GB | +8GB | +8GB | +2GB | |||
| CUDA环境 | -自动安装[12.1(pytorch)+11.8(paddle)] | -11.8(手动安装)+cuDNN v8.7.0(手动安装) | -None | +内存要求 | +最低16GB以上,推荐32GB以上 | +最低16GB | |||||
| GPU硬件支持列表 | -最低要求 8G+显存 | -3060ti/3070/4060 - 8G显存可开启layout、公式识别和ocr加速 |
- None | +磁盘空间要求 | +20GB以上,推荐使用SSD | +至少2GB | |||||
| 推荐配置 10G+显存 | -3080/3080ti/3090/3090ti/4070/4070ti/4070tisuper/4080/4090 - 10G显存及以上可以同时开启layout、公式识别和ocr加速和表格识别加速 - |
+ python版本 | +3.10-3.13 | ||||||||
+
+
+
+
+
+[](https://github.com/opendatalab/MinerU)
+[](https://github.com/opendatalab/MinerU)
+[](https://github.com/opendatalab/MinerU/issues)
+[](https://github.com/opendatalab/MinerU/issues)
+[](https://pypi.org/project/mineru/)
+[](https://pypi.org/project/mineru/)
+[](https://pepy.tech/project/mineru)
+[](https://pepy.tech/project/mineru)
+[](https://mineru.net/OpenSourceTools/Extractor?source=github)
+[](https://huggingface.co/spaces/opendatalab/MinerU)
+[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+[](https://colab.research.google.com/gist/myhloli/a3cb16570ab3cfeadf9d8f0ac91b4fca/mineru_demo.ipynb)
+[](https://arxiv.org/abs/2409.18839)
+[](https://arxiv.org/abs/2509.22186)
+[](https://arxiv.org/abs/2604.04771)
+[](https://deepwiki.com/opendatalab/MinerU)
+
+
+ 
+
+
+
+
+
+
+
+
+
+
+
+
+## Project Introduction
+
+MinerU is a document parsing tool that converts `PDF`, image, `DOCX`, `PPTX`, and `XLSX` inputs into machine-readable formats such as Markdown and JSON for downstream retrieval, extraction, and processing.
+MinerU was born during the pre-training process of [InternLM](https://github.com/InternLM/InternLM). We focus on solving symbol conversion issues in scientific literature and hope to contribute to technological development in the era of large models.
+Compared to well-known commercial products domestically and internationally, MinerU is still young. If you encounter any issues or if the results are not as expected, please submit an issue on [GitHub Issues](https://github.com/opendatalab/MinerU/issues) and **attach the relevant document or sample file**.
+
+
+
+## Key Features
+
+- Support `PDF`, image, `DOCX`, `PPTX`, and `XLSX` inputs
+- Remove headers, footers, footnotes, page numbers and other elements to ensure semantic coherence
+- Output text in human reading order, suitable for single-column, multi-column and complex layouts
+- Retain the original document structure, including titles, paragraphs, lists, etc.
+- Extract images, image descriptions, tables, table titles and footnotes
+- Automatically identify and convert formulas in documents to LaTeX format
+- Automatically identify and convert tables in documents to HTML format
+- Automatically detect scanned PDFs and garbled PDFs, and enable OCR functionality
+- OCR supports detection and recognition of 109 languages
+- Support multiple output formats, such as multimodal and NLP Markdown, reading-order-sorted JSON, and information-rich intermediate formats
+- Support multiple visualization results, including layout visualization, span visualization, etc., for efficient confirmation of output effects and quality inspection
+- Built-in CLI, FastAPI, Gradio WebUI, for local orchestration and multi-service deployment
+- Support pure CPU environment operation, and support GPU(CUDA)/NPU(CANN)/MPS acceleration
+- Compatible with Windows, Linux and Mac platforms
+
+
+## User Guide
+
+- [Quick Start Guide](./quick_start/index.md)
+- [Detailed Usage Instructions](./usage/index.md)
+
+## License Information
+
+This repository is licensed under the [MinerU Open Source License](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md), based on Apache 2.0 with additional conditions.
diff --git a/docs/en/quick_start/docker_deployment.md b/docs/en/quick_start/docker_deployment.md
new file mode 100644
index 000000000..d0089256c
--- /dev/null
+++ b/docs/en/quick_start/docker_deployment.md
@@ -0,0 +1,93 @@
+# Deploying MinerU with Docker
+
+MinerU provides a convenient Docker deployment method, which helps quickly set up the environment and solve some tricky environment compatibility issues.
+
+## Build Docker Image using Dockerfile
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/global/Dockerfile
+docker build -t mineru:latest -f Dockerfile .
+```
+
+## Docker Description
+
+MinerU's Docker uses `vllm/vllm-openai` as the base image, so it includes the `vllm` inference acceleration framework and necessary dependencies by default. Therefore, on compatible devices, you can directly use `vllm` to accelerate VLM model inference.
+
+> [!NOTE]
+> Requirements for using `vllm` to accelerate VLM model inference:
+>
+> - Device must have Volta architecture or later graphics cards with 8GB+ available VRAM.
+> - The host machine's graphics driver should support CUDA 12.9.1 or higher; You can check the driver version using the `nvidia-smi` command.
+> - Docker container must have access to the host machine's graphics devices.
+
+## Start Docker Container
+
+```bash
+docker run --gpus all \
+ --shm-size 32g \
+ -p 30000:30000 -p 7860:7860 -p 8000:8000 -p 8002:8002 \
+ --ipc=host \
+ -it mineru:latest \
+ /bin/bash
+```
+
+After executing this command, you will enter the Docker container's interactive terminal with some ports mapped for potential services. You can directly run MinerU-related commands within the container to use MinerU's features.
+You can also directly start MinerU services by replacing `/bin/bash` with service startup commands. For detailed instructions, please refer to the [Start the service via command](https://opendatalab.github.io/MinerU/usage/quick_usage/#advanced-usage-via-api-webui-http-clientserver).
+
+## Start Services Directly with Docker Compose
+
+We provide a [compose.yaml](https://github.com/opendatalab/MinerU/blob/master/docker/compose.yaml) file that you can use to quickly start MinerU services.
+
+```bash
+# Download compose.yaml file
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/compose.yaml
+```
+
+>[!NOTE]
+>
+>- The `compose.yaml` file contains configurations for multiple services of MinerU, you can choose to start specific services as needed.
+>- Different services might have additional parameter configurations, which you can view and edit in the `compose.yaml` file.
+>- Due to the pre-allocation of GPU memory by the `vllm` inference acceleration framework, you may not be able to run multiple `vllm` services simultaneously on the same machine. Therefore, ensure that other services that might use GPU memory have been stopped before starting the `vlm-openai-server` service or using the `vlm-vllm-engine` backend.
+
+---
+
+### Start OpenAI-compatible server service
+connect to `openai-server` via `vlm-http-client` backend
+ ```bash
+ docker compose -f compose.yaml --profile openai-server up -d
+ ```
+ >[!TIP]
+ >In another terminal, connect to openai server via http client (only requires CPU and network, no vllm environment needed)
+ > ```bash
+ > mineru -p | Parsing Backend | +pipeline | +*-auto-engine | +*-http-client | +||
|---|---|---|---|---|---|
| hybrid | +vlm | +hybrid | +vlm | +||
| Backend Features | +Good Compatibility | +High Hardware Requirements | +For OpenAI Compatible Servers2 | +||
| Accuracy1 | +86+ | +90+ | +|||
| Operating System | +Linux3 / Windows4 / macOS5 | +||||
| Pure CPU Support | +✅ | +❌ | +✅ | +||
| GPU Acceleration | +Volta and later architecture GPUs or Apple Silicon | +Not Required | +|||
| Min VRAM | +4GB | +8GB | +8GB | +2GB | +|
| RAM | +Min 16GB+, Recommended 32GB+ | +16GB | +|||
| Disk Space | +20GB+, SSD Recommended | +2GB | +|||
| Python Version | +3.10-3.13 | +||||
` block is appended after the image, using the block `sub_type` as the summary label when available and falling back to `image content` or `chart content`
+
+The following sections provide detailed descriptions of each file's purpose and format.
+
+## Visual Debugging Files
+
+### Layout Analysis File (layout.pdf)
+
+**File naming format**: `{original_filename}_layout.pdf`
+
+**Functionality**:
+
+- Visualizes layout analysis results for each page
+- Numbers in the top-right corner of each detection box indicate reading order
+- Different background colors distinguish different types of content blocks
+
+**Use cases**:
+
+- Check if layout analysis is correct
+- Verify if reading order is reasonable
+- Debug layout-related issues
+
+
+
+### Text Spans File (span.pdf)
+
+> [!NOTE]
+> Only applicable to pipeline backend
+
+**File naming format**: `{original_filename}_span.pdf`
+
+**Functionality**:
+
+- Uses different colored line boxes to annotate page content based on span type
+- Used for quality inspection and issue troubleshooting
+
+**Use cases**:
+
+- Quickly troubleshoot text loss issues
+- Check inline formula recognition
+- Verify text segmentation accuracy
+
+
+
+## Structured Data Files
+
+> [!IMPORTANT]
+> The VLM backend output has significant changes in version 2.5 and is not backward-compatible with the pipeline backend. If you plan to build secondary development on structured outputs, please read this document carefully.
+
+### Pipeline Backend Output Results
+
+#### Model Inference Results (model.json)
+
+**File naming format**: `{original_filename}_model.json`
+
+##### Sample Data
+
+```json
+[
+ {
+ "cls_id": 12,
+ "label": "header",
+ "score": 0.93,
+ "bbox": [
+ 1217,
+ 104,
+ 1516,
+ 134
+ ],
+ "index": 2
+ },
+ {
+ "cls_id": 6,
+ "label": "doc_title",
+ "score": 0.9751,
+ "bbox": [
+ 275,
+ 181,
+ 1512,
+ 292
+ ],
+ "index": 3
+ },
+ {
+ "cls_id": 22,
+ "label": "text",
+ "score": 0.9217,
+ "bbox": [
+ 275,
+ 330,
+ 524,
+ 370
+ ],
+ "index": 4
+ }
+]
+```
+
+#### Intermediate Processing Results (middle.json)
+
+**File naming format**: `{original_filename}_middle.json`
+
+##### Top-level Structure
+
+| Field Name | Type | Description |
+|------------|------|-------------|
+| `pdf_info` | `list[dict]` | Array of parsing results for each page |
+| `_backend` | `string` | Parsing mode: `pipeline`, `vlm`, or `office` |
+| `_version_name` | `string` | MinerU version number |
+
+##### Page Information Structure (pdf_info)
+
+| Field Name | Description |
+|------------|-------------|
+| `preproc_blocks` | Unsegmented intermediate results after PDF preprocessing |
+| `page_idx` | Page number, starting from 0 |
+| `page_size` | Page width and height `[width, height]` |
+| `images` | Image block information list |
+| `tables` | Table block information list |
+| `interline_equations` | Interline formula block information list |
+| `discarded_blocks` | Block information to be discarded |
+| `para_blocks` | Content block results after segmentation |
+
+##### Block Structure Hierarchy
+
+```
+Level 1 blocks (table | image | chart)
+└── Level 2 blocks
+ └── Lines
+ └── Spans
+```
+
+##### Level 1 Block Fields
+
+| Field Name | Description |
+|------------|-------------|
+| `type` | Block type: `table`, `image`, or `chart` |
+| `bbox` | Rectangular box coordinates of the block `[x0, y0, x1, y1]` |
+| `blocks` | List of contained level 2 blocks |
+
+##### Level 2 Block Fields
+
+| Field Name | Description |
+|------------|-------------|
+| `type` | Block type (see table below) |
+| `bbox` | Rectangular box coordinates of the block |
+| `lines` | List of contained line information |
+
+##### Level 2 Block Types
+
+| Type | Description |
+|------|-------------|
+| `image_body` | Image body |
+| `image_caption` | Image caption text |
+| `image_footnote` | Image footnote |
+| `table_body` | Table body |
+| `table_caption` | Table caption text |
+| `table_footnote` | Table footnote |
+| `chart_body` | Chart body |
+| `chart_caption` | Chart caption text |
+| `chart_footnote` | Chart footnote |
+| `text` | Text block |
+| `title` | Title block |
+| `index` | Index block |
+| `list` | List block |
+| `interline_equation` | Interline formula block |
+
+##### Line and Span Structure
+
+**Line fields**:
+- `bbox`: Rectangular box coordinates of the line
+- `spans`: List of contained spans
+
+**Span fields**:
+- `bbox`: Rectangular box coordinates of the span
+- `type`: Span type (`image`, `table`, `chart`, `text`, `inline_equation`, `interline_equation`)
+- `content` | `image_path`: Text content or image path
+
+##### Sample Data
+
+```json
+{
+ "pdf_info": [
+ {
+ "preproc_blocks": [
+ {
+ "type": "text",
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 82.99800872802734
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 72.0000228881836
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 54.0,
+ 61.956024169921875,
+ 296.2261657714844,
+ 72.0000228881836
+ ],
+ "content": "dependent on the service headway and the reliability of the departure ",
+ "type": "text",
+ "score": 1.0
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "layout_bboxes": [
+ {
+ "layout_bbox": [
+ 52,
+ 61,
+ 294,
+ 731
+ ],
+ "layout_label": "V",
+ "sub_layout": []
+ }
+ ],
+ "page_idx": 0,
+ "page_size": [
+ 612.0,
+ 792.0
+ ],
+ "_layout_tree": [],
+ "images": [],
+ "tables": [],
+ "interline_equations": [],
+ "discarded_blocks": [],
+ "para_blocks": [
+ {
+ "type": "text",
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 82.99800872802734
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 72.0000228881836
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 54.0,
+ 61.956024169921875,
+ 296.2261657714844,
+ 72.0000228881836
+ ],
+ "content": "dependent on the service headway and the reliability of the departure ",
+ "type": "text",
+ "score": 1.0
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "_backend": "pipeline",
+ "_version_name": "0.6.1"
+}
+```
+
+#### Content List (content_list.json)
+
+**File naming format**: `{original_filename}_content_list.json`
+
+##### Functionality
+
+This is a simplified version of `middle.json` that stores all readable content blocks in reading order as a flat structure, removing complex layout information for easier subsequent processing.
+
+##### Content Types
+
+| Type | Description |
+|------|-------------|
+| `image` | Image |
+| `table` | Table |
+| `chart` | Chart |
+| `text` | Text/Title |
+| `equation` | Interline formula |
+| `seal` | Seal |
+| `code` | Code block / algorithm block |
+| `list` | List / reference list |
+| `header` / `footer` / `page_number` / `aside_text` / `page_footnote` | Page auxiliary blocks |
+
+##### Text Level Identification
+
+Text levels are distinguished through the `text_level` field:
+
+- No `text_level` or `text_level: 0`: Body text
+- `text_level: 1`: Level 1 heading
+- `text_level: 2`: Level 2 heading
+- And so on...
+
+##### Common Fields
+
+- All content blocks include a `page_idx` field indicating the page number (starting from 0).
+- All content blocks include a `bbox` field representing the bounding box coordinates of the content block `[x0, y0, x1, y1]`, mapped to a range of 0-1000.
+- `code` entries use `sub_type` to distinguish `code` and `algorithm`, and may include fields such as `code_body`, `code_caption`, and `code_footnote`.
+- `list` entries may use `sub_type` to distinguish ordinary lists from reference-style lists.
+- `image` / `chart` entries may include an optional `sub_type` field to carry the visual subtype through downstream outputs.
+
+##### Sample Data
+
+```json
+[
+ {
+ "type": "text",
+ "text": "The response of flow duration curves to afforestation ",
+ "text_level": 1,
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 0
+ },
+ {
+ "type": "image",
+ "img_path": "images/a8ecda1c69b27e4f79fce1589175a9d721cbdc1cf78b4cc06a015f3746f6b9d8.jpg",
+ "image_caption": [
+ "Fig. 1. Annual flow duration curves of daily flows from Pine Creek, Australia, 1989–2000. "
+ ],
+ "image_footnote": [],
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 1
+ },
+ {
+ "type": "equation",
+ "img_path": "images/181ea56ef185060d04bf4e274685f3e072e922e7b839f093d482c29bf89b71e8.jpg",
+ "text": "$$\nQ _ { \\% } = f ( P ) + g ( T )\n$$",
+ "text_format": "latex",
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 2
+ },
+ {
+ "type": "table",
+ "img_path": "images/e3cb413394a475e555807ffdad913435940ec637873d673ee1b039e3bc3496d0.jpg",
+ "table_caption": [
+ "Table 2 Significance of the rainfall and time terms "
+ ],
+ "table_footnote": [
+ "indicates that the rainfall term was significant at the $5 \\%$ level, $T$ indicates that the time term was significant at the $5 \\%$ level, \\* represents significance at the $10 \\%$ level, and na denotes too few data points for meaningful analysis. "
+ ],
+ "table_body": "
",
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 5
+ }
+]
+```
+
+### Common Content List V2 (content_list_v2.json)(development version, subject to change)
+
+**File naming format**: `{original_filename}_content_list_v2.json`
+
+##### Functionality
+
+`content_list_v2.json` is the new structured output added in 3.0. All backends now emit it in addition to the legacy `content_list.json`:
+
+- The top level is grouped by page for page-oriented consumption
+- Each item uses a unified `type + content` structure for easier programmatic processing
+- The exact supported `type` values depend on the backend and input type
+
+##### Common Fields
+
+| Field | Type | Description |
+|------|------|-------------|
+| `type` | `string` | Content type |
+| `content` | `dict` | Structured payload for the given `type` |
+| `bbox` | `list[int]` | Optional bounding box mapped into the 0-1000 coordinate range |
+| `anchor` | `string` | Optional anchor; some `DOCX` titles or index items may include it |
+
+`image` / `chart` items may also include an optional top-level `sub_type` field for visual subtype propagation.
+
+##### Common Types
+
+| Type | Description |
+|------|-------------|
+| `title` | Title block with `title_content` and `level` |
+| `paragraph` | Paragraph block with `paragraph_content` |
+| `equation_interline` | Interline formula with `math_content` and `math_type` |
+| `image` / `table` / `chart` / `seal` | Visual blocks with image paths, captions, and related structured fields |
+| `code` | Code block with `code_content`, `code_caption`, `code_footnote`, and `code_language` |
+| `algorithm` | Algorithm block with `algorithm_content`, `algorithm_caption`, and `algorithm_footnote` |
+| `list` / `index` | List and index blocks with `list_items` |
+| `page_header` / `page_footer` / `page_number` / `page_aside_text` / `page_footnote` | Page auxiliary blocks |
+
+##### Sample Data
+
+```json
+[
+ [
+ {
+ "type": "title",
+ "content": {
+ "title_content": [
+ {
+ "type": "text",
+ "content": "1 Introduction"
+ }
+ ],
+ "level": 1
+ },
+ "bbox": [83, 121, 917, 156]
+ },
+ {
+ "type": "page_footnote",
+ "content": {
+ "page_footnote_content": [
+ {
+ "type": "text",
+ "content": "* Corresponding author"
+ }
+ ]
+ },
+ "bbox": [71, 815, 915, 841]
+ }
+ ]
+]
+```
+
+### VLM Backend Output Results
+
+#### Model Inference Results (model.json)
+
+**File naming format**: `{original_filename}_model.json`
+
+##### File format description
+- Two-level nested list: outer list = pages; inner list = content blocks of that page
+- Each block is a dict with at least: `type`, `bbox`, `angle`, `content` (some types add extra fields like `score`, `block_tags`, `content_tags`, `format`)
+- Designed for direct, raw model inspection
+
+##### Supported content types (type field values)
+```json
+{
+ "text": "Plain text",
+ "title": "Title",
+ "equation": "Display (interline) formula",
+ "image": "Image",
+ "image_caption": "Image caption",
+ "image_footnote": "Image footnote",
+ "table": "Table",
+ "table_caption": "Table caption",
+ "table_footnote": "Table footnote",
+ "phonetic": "Phonetic annotation",
+ "code": "Code block",
+ "code_caption": "Code caption",
+ "ref_text": "Reference / citation entry",
+ "algorithm": "Algorithm block (treated as code subtype)",
+ "list": "List container",
+ "header": "Page header",
+ "footer": "Page footer",
+ "page_number": "Page number",
+ "aside_text": "Side / margin note",
+ "page_footnote": "Page footnote"
+}
+```
+
+##### Coordinate system
+- `bbox` = `[x0, y0, x1, y1]` (top-left, bottom-right)
+- Origin at top-left of the page
+- All coordinates are normalized percentages in `[0,1]`
+
+##### Sample data
+```json
+[
+ [
+ {
+ "type": "header",
+ "bbox": [0.077, 0.095, 0.18, 0.181],
+ "angle": 0,
+ "score": null,
+ "block_tags": null,
+ "content": "ELSEVIER",
+ "format": null,
+ "content_tags": null
+ },
+ {
+ "type": "title",
+ "bbox": [0.157, 0.228, 0.833, 0.253],
+ "angle": 0,
+ "score": null,
+ "block_tags": null,
+ "content": "The response of flow duration curves to afforestation",
+ "format": null,
+ "content_tags": null
+ }
+ ]
+]
+```
+
+#### Intermediate Processing Results (middle.json)
+
+**File naming format**: `{original_filename}_middle.json`
+
+Structure is broadly similar to the pipeline backend, but with these differences:
+
+- `list` becomes a second‑level block, a new field `sub_type` distinguishes list categories:
+ * `text`: ordinary list
+ * `ref_text`: reference / bibliography style list
+- New `code` block type with `sub_type`(a code block always has at least a `code_body`, it may optionally have a `code_caption`):
+ * `code`
+ * `algorithm`
+- `discarded_blocks` may contain additional types:
+ * `header`
+ * `footer`
+ * `page_number`
+ * `aside_text`
+ * `page_footnote`
+- All blocks include an `angle` field indicating rotation (one of `0, 90, 180, 270`).
+
+##### Examples
+- Example: list block
+ ```json
+ {
+ "bbox": [174,155,818,333],
+ "type": "list",
+ "angle": 0,
+ "index": 11,
+ "blocks": [
+ {
+ "bbox": [174,157,311,175],
+ "type": "text",
+ "angle": 0,
+ "lines": [
+ {
+ "bbox": [174,157,311,175],
+ "spans": [
+ {
+ "bbox": [174,157,311,175],
+ "type": "text",
+ "content": "H.1 Introduction"
+ }
+ ]
+ }
+ ],
+ "index": 3
+ },
+ {
+ "bbox": [175,182,464,229],
+ "type": "text",
+ "angle": 0,
+ "lines": [
+ {
+ "bbox": [175,182,464,229],
+ "spans": [
+ {
+ "bbox": [175,182,464,229],
+ "type": "text",
+ "content": "H.2 Example: Divide by Zero without Exception Handling"
+ }
+ ]
+ }
+ ],
+ "index": 4
+ }
+ ],
+ "sub_type": "text"
+ }
+ ```
+
+- Example: code block with optional caption:
+ ```json
+ {
+ "type": "code",
+ "bbox": [114,780,885,1231],
+ "blocks": [
+ {
+ "bbox": [114,780,885,1231],
+ "lines": [
+ {
+ "bbox": [114,780,885,1231],
+ "spans": [
+ {
+ "bbox": [114,780,885,1231],
+ "type": "text",
+ "content": "1 // Fig. H.1: DivideByZeroNoExceptionHandling.java \n2 // Integer division without exception handling. \n3 import java.util.Scanner; \n4 \n5 public class DivideByZeroNoExceptionHandling \n6 { \n7 // demonstrates throwing an exception when a divide-by-zero occurs \n8 public static int quotient( int numerator, int denominator ) \n9 { \n10 return numerator / denominator; // possible division by zero \n11 } // end method quotient \n12 \n13 public static void main(String[] args) \n14 { \n15 Scanner scanner = new Scanner(System.in); // scanner for input \n16 \n17 System.out.print(\"Please enter an integer numerator: \"); \n18 int numerator = scanner.nextInt(); \n19 System.out.print(\"Please enter an integer denominator: \"); \n20 int denominator = scanner.nextInt(); \n21"
+ }
+ ]
+ }
+ ],
+ "index": 17,
+ "angle": 0,
+ "type": "code_body"
+ },
+ {
+ "bbox": [867,160,1280,189],
+ "lines": [
+ {
+ "bbox": [867,160,1280,189],
+ "spans": [
+ {
+ "bbox": [867,160,1280,189],
+ "type": "text",
+ "content": "Algorithm 1 Modules for MCTSteg"
+ }
+ ]
+ }
+ ],
+ "index": 19,
+ "angle": 0,
+ "type": "code_caption"
+ }
+ ],
+ "index": 17,
+ "sub_type": "code"
+ }
+ ```
+
+#### Content List (content_list.json)
+
+**File naming format**: `{original_filename}_content_list.json`
+
+Based on the pipeline format, with these VLM-specific extensions:
+
+- New `code` type with `sub_type` (`code` | `algorithm`):
+ * Fields: `code_body` (string), optional `code_caption` (list of strings)
+- New `list` type with `sub_type` (`text` | `ref_text`):
+ * Field: `list_items` (array of strings)
+- `image` / `chart` entries may carry an optional `sub_type` field for visual subtype propagation.
+- `chart` entries may additionally expose `content`, `chart_caption`, and `chart_footnote` alongside `img_path`; `content` preserves the original Markdown table text.
+- All `discarded_blocks` entries are also output (e.g., headers, footers, page numbers, margin notes, page footnotes).
+- Existing types (`image`, `table`, `text`, `equation`) remain unchanged.
+- `bbox` still uses the 0–1000 normalized coordinate mapping.
+- Starting with 3.0, the VLM backend also emits `*_content_list_v2.json`; see the common V2 section above for the shared structure.
+
+
+##### Examples
+Example: code (algorithm) entry
+```json
+{
+ "type": "code",
+ "sub_type": "algorithm",
+ "code_caption": ["Algorithm 1 Modules for MCTSteg"],
+ "code_body": "1: function GETCOORDINATE(d) \n2: $x \\gets d / l$ , $y \\gets d$ mod $l$ \n3: return $(x, y)$ \n4: end function \n5: function BESTCHILD(v) \n6: $C \\gets$ child set of $v$ \n7: $v' \\gets \\arg \\max_{c \\in C} \\mathrm{UCTScore}(c)$ \n8: $v'.n \\gets v'.n + 1$ \n9: return $v'$ \n10: end function \n11: function BACK PROPAGATE(v) \n12: Calculate $R$ using Equation 11 \n13: while $v$ is not a root node do \n14: $v.r \\gets v.r + R$ , $v \\gets v.p$ \n15: end while \n16: end function \n17: function RANDOMSEARCH(v) \n18: while $v$ is not a leaf node do \n19: Randomly select an untried action $a \\in A(v)$ \n20: Create a new node $v'$ \n21: $(x, y) \\gets \\mathrm{GETCOORDINATE}(v'.d)$ \n22: $v'.p \\gets v$ , $v'.d \\gets v.d + 1$ , $v'.\\Gamma \\gets v.\\Gamma$ \n23: $v'.\\gamma_{x,y} \\gets a$ \n24: if $a = -1$ then \n25: $v.lc \\gets v'$ \n26: else if $a = 0$ then \n27: $v.mc \\gets v'$ \n28: else \n29: $v.rc \\gets v'$ \n30: end if \n31: $v \\gets v'$ \n32: end while \n33: return $v$ \n34: end function \n35: function SEARCH(v) \n36: while $v$ is fully expanded do \n37: $v \\gets$ BESTCHILD(v) \n38: end while \n39: if $v$ is not a leaf node then \n40: $v \\gets$ RANDOMSEARCH(v) \n41: end if \n42: return $v$ \n43: end function",
+ "bbox": [510,87,881,740],
+ "page_idx": 0
+}
+```
+
+Example: list (text) entry
+```json
+{
+ "type": "list",
+ "sub_type": "text",
+ "list_items": [
+ "H.1 Introduction",
+ "H.2 Example: Divide by Zero without Exception Handling",
+ "H.3 Example: Divide by Zero with Exception Handling",
+ "H.4 Summary"
+ ],
+ "bbox": [174,155,818,333],
+ "page_idx": 0
+}
+```
+
+Example: discarded blocks output
+```json
+[
+ {
+ "type": "header",
+ "text": "Journal of Hydrology 310 (2005) 253-265",
+ "bbox": [363,164,623,177],
+ "page_idx": 0
+ },
+ {
+ "type": "page_footnote",
+ "text": "* Corresponding author. Address: Forest Science Centre, Department of Sustainability and Environment, P.O. Box 137, Heidelberg, Vic. 3084, Australia. Tel.: +61 3 9450 8719; fax: +61 3 9450 8644.",
+ "bbox": [71,815,915,841],
+ "page_idx": 0
+ }
+]
+```
+
+## Summary
+
+The above files constitute MinerU's complete output results. Users can choose appropriate files for subsequent processing based on their needs:
+
+- **Model outputs** (Use raw outputs):
+ * model.json
+
+- **Debugging and verification** (Use visualization files):
+ * layout.pdf
+ * span.pdf
+
+- **Content extraction**: (Use simplified files):
+ * *.md
+ * content_list.json
+ * content_list_v2.json
+
+- **Secondary development**: (Use structured files):
+ * middle.json
diff --git a/docs/en/usage/advanced_cli_parameters.md b/docs/en/usage/advanced_cli_parameters.md
new file mode 100644
index 000000000..203543df5
--- /dev/null
+++ b/docs/en/usage/advanced_cli_parameters.md
@@ -0,0 +1,56 @@
+# Advanced Command Line Parameters
+
+## Pass-through of inference engine parameters
+
+### Parameter Passing Instructions
+> [!TIP]
+> - All officially supported vllm/lmdeploy parameters can be passed to MinerU through command line arguments, including the following commands: `mineru`, `mineru-openai-server`, `mineru-gradio`, `mineru-api`, `mineru-router`
+> - Command-line options support both `--foo value` and `--foo=value` forms
+> - If you want to learn more about `vllm` parameter usage, please refer to the [vllm official documentation](https://docs.vllm.ai/en/latest/cli/serve.html)
+> - If you want to learn more about `lmdeploy` parameter usage, please refer to the [lmdeploy official documentation](https://lmdeploy.readthedocs.io/en/latest/llm/api_server.html)
+
+## GPU Device Selection and Configuration
+
+### CUDA_VISIBLE_DEVICES Basic Usage
+> [!TIP]
+> - In any situation, you can specify visible GPU devices by adding the `CUDA_VISIBLE_DEVICES` environment variable at the beginning of the command line. For example:
+> ```bash
+> CUDA_VISIBLE_DEVICES=1 mineru -p -o
+> ```
+> - This method works for all command-line invocations, including `mineru`, `mineru-openai-server`, `mineru-gradio`, `mineru-api`, and `mineru-router`, and it applies to both the `pipeline` and `vlm` backends.
+
+### Common Device Configuration Examples
+> [!TIP]
+> Here are some common `CUDA_VISIBLE_DEVICES` setting examples:
+> ```bash
+> CUDA_VISIBLE_DEVICES=1 # Only device 1 will be seen
+> CUDA_VISIBLE_DEVICES=0,1 # Devices 0 and 1 will be visible
+> CUDA_VISIBLE_DEVICES="0,1" # Same as above, quotation marks are optional
+> CUDA_VISIBLE_DEVICES=0,2,3 # Devices 0, 2, 3 will be visible; device 1 is masked
+> CUDA_VISIBLE_DEVICES="" # No GPU will be visible
+> ```
+
+## Practical Application Scenarios
+> [!TIP]
+> Here are some possible usage scenarios:
+>
+> - If you have multiple GPUs and need to start two `openai-server` services on GPU 0 and GPU 1, each listening on a different port, you can use the following commands:
+> ```bash
+> # In terminal 1
+> CUDA_VISIBLE_DEVICES=0 mineru-openai-server --engine vllm --port 30000
+> # In terminal 2
+> CUDA_VISIBLE_DEVICES=1 mineru-openai-server --engine vllm --port 30001
+> ```
+>
+> - If you have multiple GPUs and need to start two `fastapi` services on GPU 0 and GPU 1, each listening on a different port, you can use the following commands:
+> ```bash
+> # In terminal 1
+> CUDA_VISIBLE_DEVICES=0 mineru-api --host 127.0.0.1 --port 8000
+> # In terminal 2
+> CUDA_VISIBLE_DEVICES=1 mineru-api --host 127.0.0.1 --port 8001
+> ```
+>
+> - If you have multiple GPUs and want to use `router` to launch and manage `fastapi` services across four GPUs, you can use the following command:
+> ```bash
+> CUDA_VISIBLE_DEVICES=0,1,2,3 mineru-router --host 127.0.0.1 --port 8002
+> ```
diff --git a/docs/en/usage/cli_tools.md b/docs/en/usage/cli_tools.md
new file mode 100644
index 000000000..4ced9e9dd
--- /dev/null
+++ b/docs/en/usage/cli_tools.md
@@ -0,0 +1,184 @@
+# Command Line Tools Usage Instructions
+
+## View Help Information
+To view help information for MinerU command line tools, you can use the `--help` parameter. Here are help information examples for various command line tools:
+```bash
+mineru --help
+Usage: mineru [OPTIONS]
+
+Options:
+ -v, --version Show version and exit
+ -p, --path PATH Input file path or directory (required)
+ -o, --output PATH Output directory (required)
+ --api-url TEXT MinerU FastAPI base URL; if omitted, `mineru` starts a temporary local `mineru-api`
+ -m, --method [auto|txt|ocr] Parsing method: auto (default), txt, ocr (pipeline and hybrid* backend only)
+ -b, --backend [pipeline|hybrid-auto-engine|hybrid-http-client|vlm-auto-engine|vlm-http-client]
+ Parsing backend (default: hybrid-auto-engine)
+ -l, --lang [ch|ch_server|ch_lite|en|korean|japan|chinese_cht|ta|te|ka|th|el|latin|arabic|east_slavic|cyrillic|devanagari]
+ Specify document language (improves OCR accuracy, pipeline and hybrid* backend only)
+ -u, --url TEXT OpenAI-compatible backend URL passed through to the server when using http-client
+ -s, --start INTEGER Starting page number for parsing (0-based)
+ -e, --end INTEGER Ending page number for parsing (0-based)
+ -f, --formula BOOLEAN Enable formula parsing (default: enabled)
+ -t, --table BOOLEAN Enable table parsing (default: enabled)
+ --help Show help information
+```
+> [!TIP]
+> `mineru` currently supports local `PDF`, image, `DOCX`, `PPTX`, and `XLSX` file or directory inputs.
+
+```bash
+mineru-api --help
+Usage: mineru-api [OPTIONS]
+
+Options:
+ --host TEXT Server host (default: 127.0.0.1)
+ --port INTEGER Server port (default: 8000)
+ --reload Enable auto-reload (development mode)
+ --enable-vlm-preload BOOLEAN
+ Preload the local VLM model during mineru-api startup.
+ --help Show this message and exit.
+```
+```bash
+mineru-gradio --help
+Usage: mineru-gradio [OPTIONS]
+
+Options:
+ --enable-example BOOLEAN Enable example files for input. The example
+ files to be input need to be placed in the
+ `example` folder within the directory where
+ the command is currently executed.
+ --enable-http-client BOOLEAN Enable http-client backend to link openai-
+ compatible servers.
+ --enable-api BOOLEAN Enable gradio API for serving the
+ application.
+ --max-convert-pages INTEGER Set the maximum number of pages to convert
+ from PDF to Markdown.
+ --server-name TEXT Set the server name for the Gradio app.
+ --server-port INTEGER Set the server port for the Gradio app.
+ --api-url TEXT MinerU FastAPI base URL. If omitted, gradio
+ starts a reusable local mineru-api service.
+ --enable-vlm-preload BOOLEAN Preload the local VLM model when gradio
+ starts a local mineru-api service.
+ --latex-delimiters-type [a|b|all]
+ Set the type of LaTeX delimiters to use in
+ Markdown rendering: 'a' for type '$', 'b' for
+ type '()[]', 'all' for both types.
+ --help Show this message and exit.
+```
+```bash
+mineru-router --help
+Usage: mineru-router [OPTIONS]
+
+Options:
+ --host TEXT Server host (default: 127.0.0.1)
+ --port INTEGER Server port (default: 8002)
+ --reload Enable auto-reload (development mode)
+ --upstream-url TEXT Existing MinerU FastAPI base URL; repeat to add more
+ --local-gpus TEXT Local GPU workers to launch: auto, none, or CSV such
+ as 0,1,2
+ --worker-host TEXT Host for router-managed workers (default: 127.0.0.1)
+ --enable-vlm-preload BOOLEAN
+ Preload the local VLM model in router-managed
+ mineru-api workers.
+ --help Show this message and exit.
+```
+
+## Environment Variables Description
+
+> [!NOTE]
+> Starting from this version, `mineru` is an orchestration client built on top of `mineru-api`:
+>
+>- Without `--api-url`, the CLI launches a temporary local `mineru-api`
+>- With `--api-url`, the CLI connects to that FastAPI service directly
+>- `--url` is no longer the MinerU API address; it is the OpenAI-compatible backend URL used by server-side `vlm/hybrid-http-client`
+
+Some parameters of MinerU command line tools have equivalent environment variable configurations. Generally, environment variable configurations have higher priority than command line parameters and take effect across all command line tools.
+Here are the environment variables and their descriptions:
+
+- `MINERU_TOOLS_CONFIG_JSON`:
+ * Used to specify configuration file path
+ * defaults to `mineru.json` in user directory, can specify other configuration file paths through environment variables.
+
+- `MINERU_FORMULA_ENABLE`:
+ * Used to enable formula parsing
+ * defaults to `true`, can be set to `false` through environment variables to disable formula parsing.
+
+- `MINERU_FORMULA_CH_SUPPORT`:
+ * Used to enable Chinese formula parsing optimization (experimental feature)
+ * Default is `false`, can be set to `true` via environment variable to enable Chinese formula parsing optimization.
+ * Only effective for `pipeline` backend.
+
+- `MINERU_TABLE_ENABLE`:
+ * Used to enable table parsing
+ * Default is `true`, can be set to `false` via environment variable to disable table parsing.
+
+- `MINERU_TABLE_MERGE_ENABLE`:
+ * Used to enable table merging functionality
+ * Default is `true`, can be set to `false` via environment variable to disable table merging functionality.
+
+- `MINERU_PDF_RENDER_TIMEOUT`:
+ * Used to set the timeout (in seconds) for rendering PDFs to images.
+ * Default is `300` seconds; you can set a different value via an environment variable to adjust the rendering timeout.
+ * Only effective on Linux and macOS systems.
+
+- `MINERU_PDF_RENDER_THREADS`:
+ * Used to set the number of threads used when rendering PDFs to images.
+ * Default is `4`; you can set a different value via an environment variable to adjust the number of threads for image rendering.
+ * Only effective on Linux and macOS systems.
+
+- `MINERU_PROCESSING_WINDOW_SIZE`:
+ * Used to control the processing window size, which affects memory use and throughput on large-document workloads.
+ * Default is `64`; set it to another positive integer when needed.
+
+- `MINERU_API_MAX_CONCURRENT_REQUESTS`:
+ * Used to control the maximum concurrent requests handled by `mineru-api` or router-managed workers.
+ * Default is `3`, and it must be a positive integer.
+
+- `MINERU_API_ENABLE_FASTAPI_DOCS`:
+ * Used to control whether FastAPI documentation endpoints such as `/docs`, `/openapi.json`, and `/redoc` are enabled.
+ * Default is `true`.
+
+- `MINERU_API_OUTPUT_ROOT`:
+ * Used to configure the root output directory for `mineru-api`.
+ * Default is `./output` under the current working directory.
+
+- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`:
+ * Used to control how long CLI tools wait for a locally started `mineru-api` to become healthy.
+ * Default is `300` seconds.
+ * Applies to temporary local API startup in `mineru`, preload startup in `mineru-gradio`, and router-managed local workers.
+
+- `MINERU_API_TASK_RETENTION_SECONDS`:
+ * Used to set how long completed or failed tasks are retained, in seconds.
+ * Default is `86400` seconds (24 hours).
+
+- `MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS`:
+ * Used to set the cleanup polling interval for expired tasks, in seconds.
+ * Default is `300` seconds (5 minutes).
+
+- `MINERU_INTRA_OP_NUM_THREADS`:
+ * Used to set the intra_op thread count for ONNX models, affects the computation speed of individual operators
+ * Default is `-1` (auto-select), can be set to other values via environment variable to adjust the thread count.
+
+- `MINERU_INTER_OP_NUM_THREADS`:
+ * Used to set the inter_op thread count for ONNX models, affects the parallel execution of multiple operators
+ * Default is `-1` (auto-select), can be set to other values via environment variable to adjust the thread count.
+
+- `MINERU_HYBRID_BATCH_RATIO`:
+ * Used to set the batch ratio for small model processing in `hybrid-*` backends.
+ * Commonly used in `hybrid-http-client`, it allows adjusting the VRAM usage of a single client by controlling the batch ratio of small models.
+ * Single Client VRAM Size | MINERU_HYBRID_BATCH_RATIO
+ ------------------------|--------------------------
+ <= 6 GB | 8
+ <= 4 GB | 4
+ <= 3 GB | 2
+ <= 2 GB | 1
+
+- `MINERU_HYBRID_FORCE_PIPELINE_ENABLE`:
+ * Used to force the text extraction part in `hybrid-*` backends to be processed using small models.
+ * Defaults to `false`. Can be set to `true` via environment variable to enable this feature, thereby reducing hallucinations in certain extreme cases.
+
+- `MINERU_VL_MODEL_NAME`:
+ * Used to specify the model name for the vlm/hybrid backend, allowing you to designate the model required for MinerU to run when multiple models exist on a remote openai-server.
+
+- `MINERU_VL_API_KEY`:
+ * Used to specify the API Key for the vlm/hybrid backend, enabling authentication on the remote openai-server.
diff --git a/docs/en/usage/index.md b/docs/en/usage/index.md
new file mode 100644
index 000000000..c79cef601
--- /dev/null
+++ b/docs/en/usage/index.md
@@ -0,0 +1,18 @@
+# Usage Guide
+
+This section provides comprehensive usage instructions for the project. We will help you progressively master the project's usage from basic to advanced through the following sections:
+
+## Table of Contents
+
+- [Quick Usage](./quick_usage.md) - Quick setup and basic usage
+- [Model Source Configuration](./model_source.md) - Detailed configuration instructions for model sources
+- [Command Line Tools](./cli_tools.md) - Detailed parameter descriptions for command line tools
+- [Advanced Optimization Parameters](./advanced_cli_parameters.md) - Advanced parameter descriptions for command line tool adaptation
+
+## Getting Started
+
+Starting with 3.0, `mineru` runs as an orchestration client on top of `mineru-api`; advanced `mineru-router` usage for multi-service and multi-GPU deployments is also covered in this section.
+
+We recommend reading the documentation in the order listed above, which will help you better understand and use the project features.
+
+If you encounter issues during usage, please check the [FAQ](../faq/index.md)
diff --git a/docs/en/usage/model_source.md b/docs/en/usage/model_source.md
new file mode 100644
index 000000000..9fb86ada8
--- /dev/null
+++ b/docs/en/usage/model_source.md
@@ -0,0 +1,48 @@
+# Model Source Documentation
+
+MinerU uses `HuggingFace` and `ModelScope` as model repositories. Users can switch model sources or use local models as needed.
+
+- `HuggingFace` is the default model source, providing excellent loading speed and high stability globally.
+- `ModelScope` is the best choice for users in mainland China, providing seamlessly compatible `hf` SDK modules, suitable for users who cannot access HuggingFace.
+
+## Methods to Switch Model Sources
+
+### Configure via Environment Variables
+MinerU configures model sources through the `MINERU_MODEL_SOURCE` environment variable. This applies to all command line tools and API calls.
+```bash
+export MINERU_MODEL_SOURCE=modelscope
+mineru -p -o
+```
+or set it programmatically:
+```python
+import os
+os.environ["MINERU_MODEL_SOURCE"] = "modelscope"
+```
+>[!TIP]
+> MinerU no longer provides a CLI flag for model source selection. Model sources set through environment variables take effect in the current terminal session until the terminal is closed or the environment variable is modified.
+
+## Using Local Models
+
+### 1. Download Models to Local Storage
+```bash
+mineru-models-download --help
+```
+or use the interactive command line tool to select model downloads:
+```bash
+mineru-models-download
+```
+> [!NOTE]
+>- After download completion, the model path will be output in the current terminal window and automatically written to `mineru.json` in the user directory.
+>- You can also create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`.
+>- After downloading models locally, you can freely move the model folder to other locations while updating the model path in `mineru.json`.
+>- If you deploy the model folder to another server, please ensure you move the `mineru.json` file to the user directory of the new device and configure the model path correctly.
+>- If you need to update model files, you can run the `mineru-models-download` command again. Model updates do not support custom paths currently - if you haven't moved the local model folder, model files will be incrementally updated; if you have moved the model folder, model files will be re-downloaded to the default location and `mineru.json` will be updated.
+>- `mineru-models-download` must use a remote model source to perform a real download. If your current shell already sets `MINERU_MODEL_SOURCE=local`, this command will temporarily ignore that value for this invocation and use your selected `huggingface` or `modelscope` source instead.
+
+### 2. Use Local Models for Parsing
+
+Enable local models through environment variables:
+```bash
+export MINERU_MODEL_SOURCE=local
+mineru -p -o
+```
diff --git a/docs/en/usage/quick_usage.md b/docs/en/usage/quick_usage.md
new file mode 100644
index 000000000..4768f4aaa
--- /dev/null
+++ b/docs/en/usage/quick_usage.md
@@ -0,0 +1,158 @@
+# Using MinerU
+
+## Quick Model Source Configuration
+MinerU uses `huggingface` as the default model source. If users cannot access `huggingface` due to network restrictions, they can conveniently switch the model source to `modelscope` through environment variables:
+```bash
+export MINERU_MODEL_SOURCE=modelscope
+```
+For more information about model source configuration and custom local model paths, please refer to the [Model Source Documentation](./model_source.md) in the documentation.
+
+## Quick Usage via Command Line
+MinerU has built-in command line tools that allow users to quickly use MinerU for document parsing through the command line:
+```bash
+mineru -p -o
+```
+> [!TIP]
+>- ``: Local `PDF` / image / `DOCX` / `PPTX` / `XLSX` file or directory
+>- ``: Output directory
+>- Without `--api-url`, the CLI launches a temporary local `mineru-api`
+>- With `--api-url`, the CLI connects to an existing local or remote FastAPI service directly
+>
+> For more information about output files, please refer to [Output File Documentation](../reference/output_files.md).
+
+> [!NOTE]
+> The command line tool will automatically attempt cuda/mps acceleration on Linux and macOS systems.
+> Windows users who need cuda acceleration should visit the [PyTorch official website](https://pytorch.org/get-started/locally/) to select the appropriate command for their cuda version to install acceleration-enabled `torch` and `torchvision`.
+
+If you need to adjust parsing options through custom parameters, you can also check the more detailed [Command Line Tools Usage Instructions](./cli_tools.md) in the documentation.
+
+## Advanced Usage via API, WebUI, http-client/server
+
+- FastAPI calls:
+ ```bash
+ mineru-api --host 0.0.0.0 --port 8000
+ ```
+ >[!TIP]
+ >Access `http://127.0.0.1:8000/docs` in your browser to view the API documentation.
+ >
+ >- Health endpoint: `GET /health`
+ > Returns `protocol_version`, `processing_window_size`, `max_concurrent_requests`, and task stats
+ >- Asynchronous task submission endpoint: `POST /tasks`
+ >- Synchronous parsing endpoint: `POST /file_parse`
+ >- Task query endpoints: `GET /tasks/{task_id}`, `GET /tasks/{task_id}/result`
+ >- API outputs are controlled by the server and written to `./output` by default
+ >- Uploads currently support `PDF`, image, `DOCX`, `PPTX`, and `XLSX` files
+ >
+ >- `POST /tasks` returns immediately with a `task_id`. `POST /file_parse` uses the same task manager internally, waits for the task to finish, and then returns the final result synchronously.
+ >- When a task is waiting in the queue, both the submission response and task-status response may include `queued_ahead` to indicate how many tasks are ahead of it.
+ >- Tasks are tracked only in-process for a single `mineru-api` instance. Task status is not preserved across service restarts, `--reload`, or multi-process deployments.
+ >- Completed or failed tasks are retained for 24 hours by default, then their task state and output directory are cleaned automatically. After cleanup, task status and result endpoints return `404`.
+ >- Use `MINERU_API_TASK_RETENTION_SECONDS` and `MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS` to adjust retention and cleanup polling intervals.
+ >- Use `--enable-vlm-preload true` to warm up the local VLM model during service startup instead of waiting for the first VLM or hybrid request.
+ >
+ >Asynchronous task submission example:
+ >```bash
+ >curl -X POST http://127.0.0.1:8000/tasks \
+ > -F "files=@demo/pdfs/demo1.pdf" \
+ > -F "return_md=true"
+ >```
+ >
+ >Synchronous parsing example:
+ >```bash
+ >curl -X POST http://127.0.0.1:8000/file_parse \
+ > -F "files=@demo/pdfs/demo1.pdf" \
+ > -F "return_md=true" \
+ > -F "response_format_zip=true" \
+ > -F "return_original_file=true"
+ >```
+ >
+ >Poll task status and fetch results:
+ >```bash
+ >curl http://127.0.0.1:8000/tasks/
+ >curl http://127.0.0.1:8000/tasks//result
+ >curl http://127.0.0.1:8000/health
+ >```
+ >
+ >HTTP asynchronous call code example: [Python version](https://github.com/opendatalab/MinerU/blob/master/demo/demo.py)
+
+- Start Gradio WebUI visual frontend:
+ ```bash
+ mineru-gradio --server-name 0.0.0.0 --server-port 7860
+ ```
+ >[!TIP]
+ >
+ >- Access `http://127.0.0.1:7860` in your browser to use the Gradio WebUI.
+ >- Without `--api-url`, Gradio starts a reusable local `mineru-api`; with `--api-url`, it reuses an existing local or remote service.
+ >- `--enable-vlm-preload true` makes Gradio start its local `mineru-api` during WebUI startup and wait for VLM preload to finish. It is ignored when `--api-url` points to an existing service.
+ >- The WebUI currently accepts `PDF`, image, `DOCX`, `PPTX`, and `XLSX` uploads.
+
+- Use `mineru-router` for multi-service / multi-GPU orchestration:
+ ```bash
+ mineru-router --host 0.0.0.0 --port 8002 --local-gpus auto
+ ```
+ >[!TIP]
+ >
+ >- `mineru-router` exposes the same `/health`, `/tasks`, `/file_parse`, `/tasks/{task_id}`, and `/tasks/{task_id}/result` interface set as `mineru-api`.
+ >- Repeat `--upstream-url` to aggregate multiple existing `mineru-api` services, or use `--local-gpus` to launch local workers automatically.
+ >- `--enable-vlm-preload true` only applies to router-managed local workers. It does not preload remote services passed through `--upstream-url`.
+ >- It is intended for advanced multi-service, multi-GPU, and unified-entry deployments.
+
+- Using `http-client/server` method:
+ ```bash
+ # Start openai compatible server (requires vllm or lmdeploy environment)
+ mineru-openai-server --port 30000
+ ```
+ >[!TIP]
+ >In another terminal, connect to openai server via http client
+ > ```bash
+ > mineru -p -o -b hybrid-http-client -u http://127.0.0.1:30000
+ > ```
+ >`vlm-http-client` is the lightweight remote client option and does not require local `torch`.
+ >`hybrid-http-client` requires local pipeline dependencies such as `mineru[pipeline]` and `torch`.
+
+> [!NOTE]
+> All officially supported `vllm/lmdeploy` parameters can be passed to MinerU through command line arguments, including the following commands: `mineru`, `mineru-openai-server`, `mineru-gradio`, `mineru-api`, `mineru-router`.
+> We have compiled some commonly used parameters and usage methods for `vllm/lmdeploy`, which can be found in the documentation [Advanced Command Line Parameters](./advanced_cli_parameters.md).
+
+## Extending MinerU Functionality with Configuration Files
+
+MinerU is now ready to use out of the box, but also supports extending functionality through configuration files. You can edit `mineru.json` file in your user directory to add custom configurations.
+
+>[!IMPORTANT]
+>The `mineru.json` file will be automatically generated when you use the built-in model download command `mineru-models-download`, or you can create it by copying the [configuration template file](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json) to your user directory and renaming it to `mineru.json`.
+
+Here are some available configuration options:
+
+- `latex-delimiter-config`:
+ * Used to configure LaTeX formula delimiters
+ * Defaults to `$` symbol, can be modified to other symbols or strings as needed.
+
+- `llm-aided-config`:
+ * Used to configure parameters for LLM-assisted title hierarchy
+ * Compatible with all LLM models supporting `openai protocol`, defaults to using Alibaba Cloud Bailian's `qwen3-next-80b-a3b-instruct` model.
+ * You need to configure your own API key and set `enable` to `true` to enable this feature.
+ * If your API provider does not support the `enable_thinking` parameter, please manually remove it.
+ * For example, in your configuration file, the `llm-aided-config` section may look like:
+ ```json
+ "llm-aided-config": {
+ "api_key": "your_api_key",
+ "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+ "model": "qwen3-next-80b-a3b-instruct",
+ "enable_thinking": false,
+ "enable": false
+ }
+ ```
+ * To remove the `enable_thinking` parameter, simply delete the line containing `"enable_thinking": false`, resulting in:
+ ```json
+ "llm-aided-config": {
+ "api_key": "your_api_key",
+ "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+ "model": "qwen3-next-80b-a3b-instruct",
+ "enable": false
+ }
+ ```
+
+- `models-dir`:
+ * Used to specify local model storage directory
+ * Please specify model directories for `pipeline` and `vlm` backends separately.
+ * After specifying the directory, you can use local models by configuring the environment variable `export MINERU_MODEL_SOURCE=local`.
diff --git a/docs/how_to_download_models_en.md b/docs/how_to_download_models_en.md
deleted file mode 100644
index 34dd54e4c..000000000
--- a/docs/how_to_download_models_en.md
+++ /dev/null
@@ -1,31 +0,0 @@
-Model downloads are divided into initial downloads and updates to the model directory. Please refer to the corresponding documentation for instructions on how to proceed.
-
-# Initial download of model files
-
-### 1. Download the Model from Hugging Face
-
-Use a Python Script to Download Model Files from Hugging Face
-
-```bash
-pip install huggingface_hub
-wget https://github.com/opendatalab/MinerU/raw/master/docs/download_models_hf.py -O download_models_hf.py
-python download_models_hf.py
-```
-
-The Python script will automatically download the model files and configure the model directory in the configuration file.
-
-The configuration file can be found in the user directory, with the filename `magic-pdf.json`.
-
-# How to update models previously downloaded
-
-## 1. Models downloaded via Git LFS
-
-> Due to feedback from some users that downloading model files using git lfs was incomplete or resulted in corrupted model files, this method is no longer recommended.
-
-When magic-pdf <= 0.8.1, if you have previously downloaded the model files via git lfs, you can navigate to the previous download directory and update the models using the `git pull` command.
-
-> For versions 0.9.x and later, due to the repository change and the addition of the layout sorting model in PDF-Extract-Kit 1.0, the models cannot be updated using the `git pull` command. Instead, a Python script must be used for one-click updates.
-
-## 2. Models downloaded via Hugging Face or Model Scope
-
-If you previously downloaded models via Hugging Face or Model Scope, you can rerun the Python script used for the initial download. This will automatically update the model directory to the latest version.
diff --git a/docs/how_to_download_models_zh_cn.md b/docs/how_to_download_models_zh_cn.md
deleted file mode 100644
index 9b395e34b..000000000
--- a/docs/how_to_download_models_zh_cn.md
+++ /dev/null
@@ -1,44 +0,0 @@
-模型下载分为首次下载和更新模型目录，请参考对应的文档内容进行操作
-
-# 首次下载模型文件
-
-模型文件可以从 Hugging Face 或 Model Scope 下载，由于网络原因，国内用户访问HF可能会失败，请使用 ModelScope。
-
-
- -**pdf_info** - -Field structure description - -| Field Name | Description | -| :------------------ | :----------------------------------------------------------------------------------------------------------------- | -| preproc_blocks | Intermediate result after PDF preprocessing, not yet segmented | -| layout_bboxes | Layout segmentation results, containing layout direction (vertical, horizontal), and bbox, sorted by reading order | -| page_idx | Page number, starting from 0 | -| page_size | Page width and height | -| \_layout_tree | Layout tree structure | -| images | list, each element is a dict representing an img_block | -| tables | list, each element is a dict representing a table_block | -| interline_equations | list, each element is a dict representing an interline_equation_block | -| discarded_blocks | List, block information returned by the model that needs to be dropped | -| para_blocks | Result after segmenting preproc_blocks | - -In the above table, `para_blocks` is an array of dicts, each dict representing a block structure. A block can support up to one level of nesting. - -
- -**block** - -The outer block is referred to as a first-level block, and the fields in the first-level block include: - -| Field Name | Description | -| :--------- | :------------------------------------------------------------- | -| type | Block type (table\|image) | -| bbox | Block bounding box coordinates | -| blocks | list, each element is a dict representing a second-level block | - -
-There are only two types of first-level blocks: "table" and "image". All other blocks are second-level blocks. - -The fields in a second-level block include: - -| Field Name | Description | -| :--------- | :---------------------------------------------------------------------------------------------------------- | -| type | Block type | -| bbox | Block bounding box coordinates | -| lines | list, each element is a dict representing a line, used to describe the composition of a line of information | - -Detailed explanation of second-level block types - -| type | Description | -| :----------------- | :--------------------- | -| image_body | Main body of the image | -| image_caption | Image description text | -| table_body | Main body of the table | -| table_caption | Table description text | -| table_footnote | Table footnote | -| text | Text block | -| title | Title block | -| interline_equation | Block formula | - -
- -**line** - -The field format of a line is as follows: - -| Field Name | Description | -| :--------- | :------------------------------------------------------------------------------------------------------ | -| bbox | Bounding box coordinates of the line | -| spans | list, each element is a dict representing a span, used to describe the composition of the smallest unit | - -
- -**span** - -| Field Name | Description | -| :------------------ | :------------------------------------------------------------------------------------------------------- | -| bbox | Bounding box coordinates of the span | -| type | Type of the span | -| content \| img_path | Text spans use content, chart spans use img_path to store the actual text or screenshot path information | - -The types of spans are as follows: - -| type | Description | -| :----------------- | :------------- | -| image | Image | -| table | Table | -| text | Text | -| inline_equation | Inline formula | -| interline_equation | Block formula | - -**Summary** - -A span is the smallest storage unit for all elements. - -The elements stored within para_blocks are block information. - -The block structure is as follows: - -First-level block (if any) -> Second-level block -> Line -> Span - -#### example - -```json -{ - "pdf_info": [ - { - "preproc_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ], - "layout_bboxes": [ - { - "layout_bbox": [ - 52, - 61, - 294, - 731 - ], - "layout_label": "V", - "sub_layout": [] - } - ], - "page_idx": 0, - "page_size": [ - 612.0, - 792.0 - ], - "_layout_tree": [], - "images": [], - "tables": [], - "interline_equations": [], - "discarded_blocks": [], - "para_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ] - } - ], - "_parse_type": "txt", - "_version_name": "0.6.1" -} -``` diff --git a/docs/output_file_zh_cn.md b/docs/output_file_zh_cn.md deleted file mode 100644 index dd921da95..000000000 --- a/docs/output_file_zh_cn.md +++ /dev/null @@ -1,326 +0,0 @@ -## 概览 - -`magic-pdf` 命令执行后除了输出和 markdown 有关的文件以外，还会生成若干个和 markdown 无关的文件。现在将一一介绍这些文件 - -### some_pdf_layout.pdf - -每一页的 layout 均由一个或多个框组成。 每个框左上脚的数字表明它们的序号。此外 layout.pdf 框内用不同的背景色块圈定不同的内容块。 - - - -### some_pdf_spans.pdf - -根据 span 类型的不同，采用不同颜色线框绘制页面上所有 span。该文件可以用于质检，可以快速排查出文本丢失、行间公式未识别等问题。 - - - -### some_pdf_model.json - -#### 结构定义 - -```python -from pydantic import BaseModel, Field -from enum import IntEnum - -class CategoryType(IntEnum): - title = 0 # 标题 - plain_text = 1 # 文本 - abandon = 2 # 包括页眉页脚页码和页面注释 - figure = 3 # 图片 - figure_caption = 4 # 图片描述 - table = 5 # 表格 - table_caption = 6 # 表格描述 - table_footnote = 7 # 表格注释 - isolate_formula = 8 # 行间公式 - formula_caption = 9 # 行间公式的标号 - - embedding = 13 # 行内公式 - isolated = 14 # 行间公式 - text = 15 # ocr 识别结果 - - -class PageInfo(BaseModel): - page_no: int = Field(description="页码序号，第一页的序号是 0", ge=0) - height: int = Field(description="页面高度", gt=0) - width: int = Field(description="页面宽度", ge=0) - -class ObjectInferenceResult(BaseModel): - category_id: CategoryType = Field(description="类别", ge=0) - poly: list[float] = Field(description="四边形坐标, 分别是 左上，右上，右下，左下 四点的坐标") - score: float = Field(description="推理结果的置信度") - latex: str | None = Field(description="latex 解析结果", default=None) - html: str | None = Field(description="html 解析结果", default=None) - -class PageInferenceResults(BaseModel): - layout_dets: list[ObjectInferenceResult] = Field(description="页面识别结果", ge=0) - page_info: PageInfo = Field(description="页面元信息") - - -# 所有页面的推理结果按照页码顺序依次放到列表中即为 minerU 推理结果 -inference_result: list[PageInferenceResults] = [] - -``` - -poly 坐标的格式 \[x0, y0, x1, y1, x2, y2, x3, y3\], 分别表示左上、右上、右下、左下四点的坐标 - - -#### 示例数据 - -```json -[ - { - "layout_dets": [ - { - "category_id": 2, - "poly": [ - 99.1906967163086, - 100.3119125366211, - 730.3707885742188, - 100.3119125366211, - 730.3707885742188, - 245.81326293945312, - 99.1906967163086, - 245.81326293945312 - ], - "score": 0.9999997615814209 - } - ], - "page_info": { - "page_no": 0, - "height": 2339, - "width": 1654 - } - }, - { - "layout_dets": [ - { - "category_id": 5, - "poly": [ - 99.13092803955078, - 2210.680419921875, - 497.3183898925781, - 2210.680419921875, - 497.3183898925781, - 2264.78076171875, - 99.13092803955078, - 2264.78076171875 - ], - "score": 0.9999997019767761 - } - ], - "page_info": { - "page_no": 1, - "height": 2339, - "width": 1654 - } - } -] -``` - -### some_pdf_middle.json - -| 字段名 | 解释 | -| :------------- | :----------------------------------------------------------------- | -| pdf_info | list，每个元素都是一个dict,这个dict是每一页pdf的解析结果，详见下表 | -| \_parse_type | ocr \| txt，用来标识本次解析的中间态使用的模式 | -| \_version_name | string, 表示本次解析使用的 magic-pdf 的版本号 | - -
- -**pdf_info** -字段结构说明 - -| 字段名 | 解释 | -| :------------------ | :------------------------------------------------------------------- | -| preproc_blocks | pdf预处理后，未分段的中间结果 | -| layout_bboxes | 布局分割的结果，含有布局的方向（垂直、水平），和bbox，按阅读顺序排序 | -| page_idx | 页码，从0开始 | -| page_size | 页面的宽度和高度 | -| \_layout_tree | 布局树状结构 | -| images | list，每个元素是一个dict，每个dict表示一个img_block | -| tables | list，每个元素是一个dict，每个dict表示一个table_block | -| interline_equations | list，每个元素是一个dict，每个dict表示一个interline_equation_block | -| discarded_blocks | List, 模型返回的需要drop的block信息 | -| para_blocks | 将preproc_blocks进行分段之后的结果 | - -上表中 `para_blocks` 是个dict的数组，每个dict是一个block结构，block最多支持一次嵌套 - -
- -**block** - -外层block被称为一级block，一级block中的字段包括 - -| 字段名 | 解释 | -| :----- | :---------------------------------------------- | -| type | block类型（table\|image） | -| bbox | block矩形框坐标 | -| blocks | list，里面的每个元素都是一个dict格式的二级block | - -
-一级block只有"table"和"image"两种类型，其余block均为二级block - -二级block中的字段包括 - -| 字段名 | 解释 | -| :----- | :----------------------------------------------------------- | -| type | block类型 | -| bbox | block矩形框坐标 | -| lines | list，每个元素都是一个dict表示的line，用来描述一行信息的构成 | - -二级block的类型详解 - -| type | desc | -| :----------------- | :------------- | -| image_body | 图像的本体 | -| image_caption | 图像的描述文本 | -| table_body | 表格本体 | -| table_caption | 表格的描述文本 | -| table_footnote | 表格的脚注 | -| text | 文本块 | -| title | 标题块 | -| interline_equation | 行间公式块 | - -
- -**line** - -line 的 字段格式如下 - -| 字段名 | 解释 | -| :----- | :------------------------------------------------------------------- | -| bbox | line的矩形框坐标 | -| spans | list，每个元素都是一个dict表示的span，用来描述一个最小组成单元的构成 | - -
- -**span** - -| 字段名 | 解释 | -| :------------------ | :------------------------------------------------------------------------------- | -| bbox | span的矩形框坐标 | -| type | span的类型 | -| content \| img_path | 文本类型的span使用content，图表类使用img_path 用来存储实际的文本或者截图路径信息 | - -span 的类型有如下几种 - -| type | desc | -| :----------------- | :------- | -| image | 图片 | -| table | 表格 | -| text | 文本 | -| inline_equation | 行内公式 | -| interline_equation | 行间公式 | - -**总结** - -span是所有元素的最小存储单元 - -para_blocks内存储的元素为区块信息 - -区块结构为 - -一级block(如有)->二级block->line->span - -#### 示例数据 - -```json -{ - "pdf_info": [ - { - "preproc_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ], - "layout_bboxes": [ - { - "layout_bbox": [ - 52, - 61, - 294, - 731 - ], - "layout_label": "V", - "sub_layout": [] - } - ], - "page_idx": 0, - "page_size": [ - 612.0, - 792.0 - ], - "_layout_tree": [], - "images": [], - "tables": [], - "interline_equations": [], - "discarded_blocks": [], - "para_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ] - } - ], - "_parse_type": "txt", - "_version_name": "0.6.1" -} -``` diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 000000000..6669adadd --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,4 @@ +mkdocs +mkdocs-static-i18n +markdown-gfm-admonition +mkdocs-video \ No newline at end of file diff --git a/docs/zh/demo/index.md b/docs/zh/demo/index.md new file mode 100644 index 000000000..103ffc5e6 --- /dev/null +++ b/docs/zh/demo/index.md @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/zh/faq/index.md b/docs/zh/faq/index.md new file mode 100644 index 000000000..e4a198ded --- /dev/null +++ b/docs/zh/faq/index.md @@ -0,0 +1,29 @@ +# 常见问题解答 + +如果未能列出您的问题，您也可以使用[DeepWiki](https://deepwiki.com/opendatalab/MinerU)与AI助手交流，这可以解决大部分常见问题。 + +如果您仍然无法解决问题，您可通过[Discord](https://discord.gg/Tdedn9GTXq)或[WeChat](https://mineru.net/community-portal/?aliasId=3c430f94)加入社区，与其他用户和开发者交流。 + +??? question "在WSL2的Ubuntu22.04中遇到报错`ImportError: libGL.so.1: cannot open shared object file: No such file or directory`" + + WSL2的Ubuntu22.04中缺少`libgl`库，可通过以下命令安装`libgl`库解决： + + ```bash + sudo apt-get install libgl1-mesa-glx + ``` + + 参考：[#388](https://github.com/opendatalab/MinerU/issues/388) + +??? question "在 Linux 系统安装并使用时，解析结果缺失部份文字信息。" + + MinerU在>=2.0的版本中使用`pypdfium2`代替`pymupdf`作为PDF页面的渲染引擎，以解决AGPLv3的许可证问题，在某些Linux发行版，由于缺少CJK字体，可能会在将PDF渲染成图片的过程中丢失部份文字。 + 为了解决这个问题，您可以通过以下命令安装noto字体包，这在Ubuntu/debian系统中有效： + ```bash + sudo apt update + sudo apt install fonts-noto-core + sudo apt install fonts-noto-cjk + fc-cache -fv + ``` + 也可以直接使用我们的[Docker部署](../quick_start/docker_deployment.md)方式构建镜像，镜像中默认包含以上字体包。 + + 参考：[#2915](https://github.com/opendatalab/MinerU/issues/2915) \ No newline at end of file diff --git a/docs/zh/index.md b/docs/zh/index.md new file mode 100644 index 000000000..ee5cdb4f4 --- /dev/null +++ b/docs/zh/index.md @@ -0,0 +1,75 @@ + -o -b vlm-http-client -u http://:30000
+ > ```
+
+---
+
+### 启动 Web API 服务
+ ```bash
+ docker compose -f compose.yaml --profile api up -d
+ ```
+ >[!TIP]
+ >在浏览器中访问 `http://:8000/docs` 查看API文档。
+
+---
+
+### 启动 MinerU Router 服务
+ ```bash
+ docker compose -f compose.yaml --profile router up -d
+ ```
+ >[!TIP]
+ >
+ >- 默认配置会以 `--local-gpus auto` 模式在容器内自动拉起本地 worker，并通过 `http://:8002/docs` 暴露统一入口。
+ >- 如果您希望聚合已有的 `mineru-api` 服务而不是启动本地 worker，可直接参考 `compose.yaml` 中 `mineru-router` 服务下的注释示例，改为使用 `--upstream-url`。
+
+---
+
+### 启动 Gradio WebUI 服务
+ ```bash
+ docker compose -f compose.yaml --profile gradio up -d
+ ```
+ >[!TIP]
+ >
+ >- 在浏览器中访问 `http://:7860` 使用 Gradio WebUI。
diff --git a/docs/zh/quick_start/extension_modules.md b/docs/zh/quick_start/extension_modules.md
new file mode 100644
index 000000000..72bad2622
--- /dev/null
+++ b/docs/zh/quick_start/extension_modules.md
@@ -0,0 +1,54 @@
+# MinerU 扩展模块安装指南
+MinerU 支持根据不同需求，按需安装扩展模块，以增强功能或支持特定的模型后端。
+
+## 常见场景
+
+### 核心功能安装
+`core` 模块是 MinerU 的核心依赖，包含了除`vllm`/`lmdeploy`外的所有功能模块。安装此模块可以确保 MinerU 的基本功能正常运行。
+```bash
+uv pip install "mineru[core]"
+```
+
+---
+
+### 使用`vllm`加速 VLM 模型推理
+> [!NOTE]
+> `vllm`和`lmdeploy`对vlm的推理加速效果和使用方式几乎相同，您可以根据实际情况选择其中之一进行安装和使用，但不建议同时安装这两个模块，以避免潜在的依赖冲突。
+
+`vllm` 模块提供了对 VLM 模型推理的加速支持，适用于具有 Volta 及以后架构的显卡（8G 显存及以上）。安装此模块可以显著提升模型推理速度。
+```bash
+uv pip install "mineru[core,vllm]"
+```
+> [!TIP]
+> 如在安装包含`vllm`的扩展包过程中发生异常，请参考 [vllm 官方文档](https://docs.vllm.ai/en/latest/getting_started/installation/index.html) 尝试解决，或直接使用 [Docker](./docker_deployment.md) 方式部署镜像。
+
+---
+
+### 使用`lmdeploy`加速 VLM 模型推理
+> [!NOTE]
+> `vllm`和`lmdeploy`对vlm的推理加速效果和使用方式几乎相同，您可以根据实际情况选择其中之一进行安装和使用，但不建议同时安装这两个模块，以避免潜在的依赖冲突。
+
+`lmdeploy` 模块提供了对 VLM 模型推理的加速支持，适用于具有 Volta 及以后架构的显卡（8G 显存及以上）。安装此模块可以显著提升模型推理速度。
+```bash
+uv pip install "mineru[core,lmdeploy]"
+```
+> [!TIP]
+> 如在安装包含`lmdeploy`的扩展包过程中发生异常，请参考 [lmdeploy 官方文档](https://lmdeploy.readthedocs.io/en/latest/get_started/installation.html) 尝试解决。
+
+---
+
+### 安装轻量版client连接兼容openai服务器使用 (适用vlm-http-client模式)
+如果您需要在边缘设备上安装轻量版的 client 端以连接兼容 openai 接口的服务端来使用vlm模式，可以安装mineru的基础包，非常轻量，适合在只有cpu和网络连接的设备上使用。
+```bash
+uv pip install mineru
+mineru -p -o -b vlm-http-client -u http://127.0.0.1:30000
+```
+
+---
+
+### 安装轻量版client连接兼容openai服务器使用 (适用hybrid-http-client模式)
+如果您需要在边缘设备上安装轻量版的 client 端以连接兼容 openai 接口的服务端来使用hybrid模式，可以安装mineru的pipeline扩展包，相对较轻量，可以在只有cpu和网络连接的设备上使用，同时在支持gpu加速的设备上可以更快运行。
+```bash
+uv pip install "mineru[pipeline]"
+mineru -p -o -b hybrid-http-client -u http://127.0.0.1:30000
+```
diff --git a/docs/zh/quick_start/index.md b/docs/zh/quick_start/index.md
new file mode 100644
index 000000000..be259c2d0
--- /dev/null
+++ b/docs/zh/quick_start/index.md
@@ -0,0 +1,154 @@
+# 快速入门
+
+如果遇到任何安装问题，请先查询 [FAQ](../faq/index.md)
+
+## 在线体验
+
+### 官网在线应用
+官网在线版功能与客户端一致，界面美观，功能丰富，需要登录使用
+
+- [](https://mineru.net/OpenSourceTools/Extractor?source=github)
+
+### 基于Gradio的在线demo
+基于gradio开发的webui，界面简洁，仅包含核心解析功能，免登录
+
+- [](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+- [](https://huggingface.co/spaces/opendatalab/MinerU)
+
+## 本地部署
+
+> [!WARNING]
+> **安装前必看——软硬件环境支持说明**
+>
+> 为了确保项目的稳定性和可靠性，我们在开发过程中仅对特定的软硬件环境进行优化和测试。这样当用户在推荐的系统配置上部署和运行项目时，能够获得最佳的性能表现和最少的兼容性问题。
+>
+> 通过集中资源和精力于主线环境，我们团队能够更高效地解决潜在的BUG，及时开发新功能。
+>
+> 在非主线环境中，由于硬件、软件配置的多样性，以及第三方依赖项的兼容性问题，我们无法100%保证项目的完全可用性。因此，对于希望在非推荐环境中使用本项目的用户，我们建议先仔细阅读文档以及FAQ，大多数问题已经在FAQ中有对应的解决方案，除此之外我们鼓励社区反馈问题，以便我们能够逐步扩大支持范围。
+
+
+
+
+
+1 精度指标为OmniDocBench (v1.5)的End-to-End Evaluation Overall分数，基于`MinerU`最新版本测试
+2 兼容OpenAI API的服务器，如通过`vLLM`/`SGLang`/`LMDeploy`等推理框架部署的本地模型服务器或远程模型服务
+3 Linux仅支持2019年及以后发行版
+4 由于关键依赖`ray`未能在windows平台支持Python 3.13，故仅支持至3.10~3.12版本
+5 macOS 需使用14.0以上版本
+
+
+> [!TIP]
+> 除以上主流环境与平台外，我们也收录了一些社区用户反馈的其他平台支持情况，详情请参考[其他加速卡适配](https://opendatalab.github.io/MinerU/zh/usage/)。
+> 如果您有意将自己的环境适配经验分享给社区，欢迎通过[show-and-tell](https://github.com/opendatalab/MinerU/discussions/categories/show-and-tell)提交或提交PR至[其他加速卡适配](https://github.com/opendatalab/MinerU/tree/master/docs/zh/usage/acceleration_cards)文档。
+
+
+### 安装 MinerU
+
+#### 使用pip或uv安装MinerU
+```bash
+pip install --upgrade pip -i https://mirrors.aliyun.com/pypi/simple
+pip install uv -i https://mirrors.aliyun.com/pypi/simple
+uv pip install -U "mineru[all]" -i https://mirrors.aliyun.com/pypi/simple
+```
+
+#### 通过源码安装MinerU
+```bash
+git clone https://github.com/opendatalab/MinerU.git
+cd MinerU
+uv pip install -e .[all] -i https://mirrors.aliyun.com/pypi/simple
+```
+
+> [!TIP]
+> `mineru[all]`包含所有核心功能，兼容Windows / Linux / macOS系统，适合绝大多数用户。
+> 如果您需要指定vlm模型的推理框架，或是仅准备在边缘设备安装轻量版client端，可以参考文档[扩展模块安装指南](https://opendatalab.github.io/MinerU/zh/quick_start/extension_modules/)。
+
+---
+
+#### 使用docker部署Mineru
+MinerU提供了便捷的docker部署方式，这有助于快速搭建环境并解决一些棘手的环境兼容问题。
+您可以在文档中获取[Docker部署说明](./docker_deployment.md)。
+
+---
+
+### 使用 MinerU
+
+>[!TIP]
+>默认使用托管在`huggingface`的模型进行解析，首次使用时会自动下载所需模型文件，后续使用将直接加载本地缓存的模型。如果您无法访问`huggingface`，可以通过以下命令切换至国内镜像源:
+>```bash
+>export MINERU_MODEL_SOURCE=modelscope
+>```
+
+如果您的设备满足上表中GPU加速的条件，可以使用简单的命令行进行文档解析:
+```bash
+mineru -p -o
+```
+如果您的设备不满足GPU加速条件，可以指定后端为`pipeline`，以在纯CPU环境下运行:
+```bash
+mineru -p -o -b pipeline
+```
+
+当前 `mineru` 支持本地 `PDF / 图片 / DOCX / PPTX / XLSX` 文件或目录输入。
+
+您可以通过命令行、API、WebUI 等多种方式使用 MinerU 进行文档解析，具体使用方法请参考[使用指南](../usage/index.md)。
diff --git a/docs/zh/reference/changelog.md b/docs/zh/reference/changelog.md
new file mode 100644
index 000000000..2196523e4
--- /dev/null
+++ b/docs/zh/reference/changelog.md
@@ -0,0 +1,502 @@
+# 更新日志
+
+本文档记录 MinerU 项目的版本更新历史、主要能力演进和重要兼容性变动。
+
+---
+
+## 2.7 系列版本
+
+### 2.7.6 (2026/02/06)
+
+- 新增国产算力平台昆仑芯、太初元碁的适配支持。
+- 目前已由官方和厂商适配并支持的国产算力平台包括：昇腾、平头哥、沐曦、海光、燧原、摩尔线程、天数智芯、寒武纪、昆仑芯、太初元碁、壁仞。
+- MinerU 持续兼容国产硬件平台，支持主流芯片架构，助力科研、政企用户迈向文档数字化新高度。
+
+### 2.7.4 (2026/01/30)
+
+- 新增国产算力平台天数智芯、寒武纪的适配支持。
+
+### 2.7.2 (2026/01/23)
+
+- 新增国产算力平台海光、燧原、摩尔线程的适配支持。
+- 跨页表合并优化，提升合并成功率与合并效果。
+
+### 2.7.1 (2026/01/06)
+
+- bug 修复：#4300
+- 更新 `pdfminer.six` 依赖版本以解决 [CVE-2025-64512](https://github.com/advisories/GHSA-wf5f-4jwr-ppcp)
+- 支持输入图像的 EXIF 方向自动校正，提升 OCR 识别效果。#4283
+
+### 2.7.0 (2025/12/30)
+
+- 简化安装流程，使用 `uv pip install mineru[all]` 即可安装所有可选后端依赖。
+- 增加全新后端 `hybrid`，结合 `pipeline` 与 `vlm` 的优势：
+ - 从文本 PDF 中直接抽取文本，在文本 PDF 场景原生支持多语言识别，并大幅减少解析幻觉；
+ - 通过指定 OCR 语言，在扫描 PDF 场景支持 109 种语言的文本识别；
+ - 支持独立的行内公式识别开关，在不需要时可单独关闭以改善结果视觉效果。
+- 简化 `vlm/hybrid` 后端的引擎选择逻辑，用户只需指定 `*-auto-engine` 即可自动选择合适的推理引擎。
+- 默认解析后端从 `pipeline` 切换至 `hybrid-auto-engine`，提升新用户开箱即用时的结果一致性。
+- Gradio 应用增加 i18n 适配，支持中英文切换。
+
+---
+
+## 2.6 系列版本
+
+### 2.6.7 (2025/12/12)
+
+- bug修复： #4168
+
+### 2.6.6 (2025/12/02)
+
+**`Ascend`适配优化**
+
+- 优化命令行工具初始化流程，使Ascend适配方案中`vlm-vllm-engine`后端在命令行工具中可用。
+- 为Atlas 300I Duo(310p)设备更新适配文档。
+
+**`mineru-api`工具优化**
+
+- 为`mineru-api`接口参数增加描述性文本，优化接口文档可读性。
+- 可通过环境变量`MINERU_API_ENABLE_FASTAPI_DOCS`控制是否启用自动生成的接口文档页面，默认为启用。
+- 为`vlm-vllm-async-engine`、`vlm-lmdeploy-engine`、`vlm-http-client`后端增加并发数配置选项，用户可通过环境变量`MINERU_API_MAX_CONCURRENT_REQUESTS`控制api接口的最大并发请求数，默认为不限制数量。
+
+### 2.6.5 (2025/11/26)
+
+- 增加新后端`vlm-lmdeploy-engine`支持，使用方式与`vlm-vllm-(async)engine`类似，但使用`lmdeploy`作为推理引擎，与`vllm`相比额外支持Windows平台原生推理加速。
+- 新增国产算力平台`昇腾/npu`、`平头哥/ppu`、`沐曦/maca`的适配支持，用户可在对应平台上使用`pipeline`与`vlm`模型，并使用`vllm`/`lmdeploy`引擎加速vlm模型推理，具体使用方式请参考[其他加速卡适配](https://opendatalab.github.io/MinerU/zh/usage/)。
+ - 国产平台适配不易，我们已尽量确保适配的完整性和稳定性，但仍可能存在一些稳定性/兼容问题与精度对齐问题，请大家根据适配文档页面内红绿灯情况自行选择合适的环境与场景进行使用。
+ - 如在使用国产化平台适配方案的过程中遇到任何文档未提及的问题，为便于其他用户查找解决方案，请在discussions的[指定帖子](https://github.com/opendatalab/MinerU/discussions/4064)中进行反馈。
+
+### 2.6.4 (2025/11/04)
+
+- 为pdf渲染图片增加超时配置，默认为300秒，可通过环境变量`MINERU_PDF_RENDER_TIMEOUT`进行配置，防止部分异常pdf文件导致渲染过程长时间阻塞。
+- 为onnx模型增加cpu线程数配置选项，默认为系统cpu核心数，可通过环境变量`MINERU_INTRA_OP_NUM_THREADS`和`MINERU_INTER_OP_NUM_THREADS`进行配置，以减少高并发场景下的对cpu资源的抢占冲突。
+
+### 2.6.3 (2025/10/31)
+
+- 增加新后端`vlm-mlx-engine`支持，在Apple Silicon设备上支持使用`MLX`加速`MinerU2.5`模型推理，相比`vlm-transformers`后端，`vlm-mlx-engine`后端速度提升100%~200%。
+- bug修复: #3849 #3859
+
+### 2.6.2 (2025/10/24)
+
+**`pipline`后端优化**
+
+- 增加对中文公式的实验性支持，可通过配置环境变量`export MINERU_FORMULA_CH_SUPPORT=1`开启。该功能可能会导致MFR速率略微下降、部分长公式识别失败等问题，建议仅在需要解析中文公式的场景下开启。如需关闭该功能，可将环境变量设置为`0`。
+- `OCR`速度大幅提升200%~300%，感谢 [@cjsdurj](https://github.com/cjsdurj) 提供的优化方案
+- `OCR`模型优化拉丁文识别的准度和广度，并更新西里尔文(cyrillic)、阿拉伯文(arabic)、天城文(devanagari)、泰卢固语(te)、泰米尔语(ta)语系至`ppocr-v5`版本，精度相比上代模型提升40%以上
+
+**`vlm`后端优化**
+
+- `table_caption`、`table_footnote`匹配逻辑优化，提升页内多张连续表场景下的表格标题和脚注的匹配准确率和阅读顺序合理性
+- 优化使用`vllm`后端时高并发时的cpu资源占用，降低服务端压力
+- 适配`vllm`0.11.0版本
+
+**通用优化**
+
+- 跨页表格合并效果优化，新增跨页续表合并支持，提升在多列合并场景下的表格合并效果
+- 为表格合并功能增加环境变量配置选项`MINERU_TABLE_MERGE_ENABLE`，表格合并功能默认开启，可通过设置该变量为`0`来关闭表格合并功能
+
+---
+
+## 2.5 系列版本
+
+### 2.5.4 (2025/09/26)
+
+- 🎉🎉 MinerU2.5[技术报告](https://arxiv.org/abs/2509.22186)现已发布，欢迎阅读全面了解其模型架构、训练策略、数据工程和评测结果。
+- 修复部分`pdf`文件被识别成`ai`文件导致无法解析的问题
+
+### 2.5.3 (2025/09/20)
+
+- 依赖版本范围调整，使得Turing及更早架构显卡可以使用vLLM加速推理MinerU2.5模型。
+- `pipeline`后端对torch 2.8.0的一些兼容性修复。
+- 降低vLLM异步后端默认的并发数，降低服务端压力以避免高压导致的链接关闭问题。
+- 更多兼容性相关内容详见[公告](https://github.com/opendatalab/MinerU/discussions/3547)
+
+### 2.5.2 (2025/09/19)
+
+我们正式发布 MinerU2.5，当前最强文档解析多模态大模型。仅凭 1.2B 参数，MinerU2.5 在 OmniDocBench 文档解析评测中，精度已全面超越 Gemini2.5-Pro、GPT-4o、Qwen2.5-VL-72B等顶级多模态大模型，并显著领先于主流文档解析专用模型（如 dots.ocr, MonkeyOCR, PP-StructureV3 等）。
+
+模型已发布至[HuggingFace](https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B)和[ModelScope](https://modelscope.cn/models/opendatalab/MinerU2.5-2509-1.2B)平台，欢迎大家下载使用！
+
+**核心亮点**
+
+- 极致能效，性能SOTA: 以 1.2B 的轻量化规模，实现了超越百亿乃至千亿级模型的SOTA性能，重新定义了文档解析的能效比。
+- 先进架构，全面领先: 通过 "两阶段推理" (解耦布局分析与内容识别) 与 原生高分辨率架构 的结合，在布局分析、文本识别、公式识别、表格识别及阅读顺序五大方面均达到 SOTA 水平。
+
+**关键能力提升**
+
+- 布局检测: 结果更完整，精准覆盖页眉、页脚、页码等非正文内容；同时提供更精准的元素定位与更自然的格式还原（如列表、参考文献）。
+- 表格解析: 大幅优化了对旋转表格、无线/少线表、以及长难表格的解析能力。
+- 公式识别: 显著提升中英混合公式及复杂长公式的识别准确率，大幅改善数学类文档解析能力。
+
+**仓库调整**
+
+此外，伴随vlm 2.5的发布，我们对仓库做出一些调整：
+
+- vlm后端升级至2.5版本，支持MinerU2.5模型，不再兼容MinerU2.0-2505-0.9B模型，最后一个支持2.0模型的版本为mineru-2.2.2。
+- vlm推理相关代码已移至[mineru_vl_utils](https://github.com/opendatalab/mineru-vl-utils),降低与mineru主仓库的耦合度，便于后续独立迭代。
+- vlm加速推理框架从`sglang`切换至`vllm`,并实现对vllm生态的完全兼容，使得用户可以在任何支持vllm框架的平台上使用MinerU2.5模型并加速推理。
+- 由于vlm模型的重大升级，支持更多layout type，因此我们对解析的中间文件`middle.json`和结果文件`content_list.json`的结构做出一些调整，请参考[文档](https://opendatalab.github.io/MinerU/zh/reference/output_files/)了解详情。
+
+**其他仓库优化**
+
+- 移除对输入文件的后缀名白名单校验，当输入文件为PDF文档或图片时，对文件的后缀名不再有要求，提升易用性。
+
+---
+
+## 2.2 - 2.4 系列版本
+
+### 2.2.2 (2025/09/10)
+
+- 修复新的表格识别模型在部分表格解析失败时影响整体解析任务的问题
+
+### 2.2.1 (2025/09/08)
+
+- 修复使用模型下载命令时，部分新增模型未下载的问题
+
+### 2.2.0 (2025/09/05)
+
+**主要更新**
+
+- 在这个版本我们重点提升了表格的解析精度，通过引入新的[有线表识别模型](https://github.com/RapidAI/TableStructureRec)和全新的混合表格结构解析算法，显著提升了`pipeline`后端的表格识别能力。
+- 另外我们增加了对跨页表格合并的支持，这一功能同时支持`pipeline`和`vlm`后端，进一步提升了表格解析的完整性和准确性。
+
+**其他更新**
+
+- `pipeline`后端增加270度旋转的表格解析能力，现已支持0/90/270度三个方向的表格解析
+- `pipeline`增加对泰文、希腊文的ocr能力支持，并更新了英文ocr模型至最新，英文识别精度提升11%，泰文识别模型精度 82.68%，希腊文识别模型精度 89.28%（by PPOCRv5）
+- 在输出的`content_list.json`中增加了`bbox`字段(映射至0-1000范围内)，方便用户直接获取每个内容块的位置信息
+- 移除`pipeline_old_linux`安装可选项，不再支持老版本的Linux系统如`Centos 7`等，以便对`uv`的`sync`/`run`等命令进行更好的支持
+
+---
+
+## 2.1 系列版本
+
+### 2.1.10 (2025/08/01)
+
+- 修复`pipeline`后端因block覆盖导致的解析结果与预期不符 #3232
+
+### 2.1.9 (2025/07/30)
+
+- `transformers` 4.54.1 版本适配
+
+### 2.1.8 (2025/07/28)
+
+- `sglang` 0.4.9.post5 版本适配
+
+### 2.1.7 (2025/07/27)
+
+- `transformers` 4.54.0 版本适配
+
+### 2.1.6 (2025/07/26)
+
+- 修复`vlm`后端解析部分手写文档时的表格异常问题
+- 修复文档旋转时可视化框位置漂移问题 #3175
+
+### 2.1.5 (2025/07/24)
+
+- `sglang` 0.4.9 版本适配，同步升级dockerfile基础镜像为sglang 0.4.9.post3
+
+### 2.1.4 (2025/07/23)
+
+**bug修复**
+
+- 修复`pipeline`后端中`MFR`步骤在某些情况下显存消耗过大的问题 #2771
+- 修复某些情况下`image`/`table`与`caption`/`footnote`匹配不准确的问题 #3129
+
+### 2.1.1 (2025/07/16)
+
+**bug修复**
+
+- 修复`pipeline`在某些情况可能发生的文本块内容丢失问题 #3005
+- 修复`sglang-client`需要安装`torch`等不必要的包的问题 #2968
+- 更新`dockerfile`以修复linux字体缺失导致的解析文本内容不完整问题 #2915
+
+**易用性更新**
+
+- 更新`compose.yaml`，便于用户直接启动`sglang-server`、`mineru-api`、`mineru-gradio`服务
+- 启用全新的[在线文档站点](https://opendatalab.github.io/MinerU/zh/)，简化readme，提供更好的文档体验
+
+### 2.1.0 (2025/07/05)
+
+这是 MinerU 2 的第一个大版本更新，包含了大量新功能和改进，包含众多性能优化、体验优化和bug修复，具体更新内容如下：
+
+**性能优化**
+
+- 大幅提升某些特定分辨率（长边2000像素左右）文档的预处理速度
+- 大幅提升`pipeline`后端批量处理大量页数较少（<10）文档时的后处理速度
+- `pipeline`后端的layout分析速度提升约20%
+
+**体验优化**
+
+- 内置开箱即用的`fastapi服务`和`gradio webui`，详细使用方法请参考[文档](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuisglang-clientserver)
+- `sglang`适配`0.4.8`版本，大幅降低`vlm-sglang`后端的显存要求，最低可在`8G显存`(Turing及以后架构)的显卡上运行
+- 对所有命令增加`sglang`的参数透传，使得`sglang-engine`后端可以与`sglang-server`一致，接收`sglang`的所有参数
+- 支持基于配置文件的功能扩展，包含`自定义公式标识符`、`开启标题分级功能`、`自定义本地模型目录`，详细使用方法请参考[文档](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#mineru_1)
+
+**新特性**
+
+- `pipeline`后端更新 PP-OCRv5 多语种文本识别模型，支持法语、西班牙语、葡萄牙语、俄语、韩语等 37 种语言的文字识别，平均精度涨幅超30%。[详情](https://paddlepaddle.github.io/PaddleOCR/latest/version3.x/algorithm/PP-OCRv5/PP-OCRv5_multi_languages.html)
+- `pipeline`后端增加对竖排文本的有限支持
+
+---
+
+## 2.0 系列版本
+
+### 2.0.6 (2025/06/20)
+
+- 修复`vlm`模式下，某些偶发的无效块内容导致解析中断问题
+- 修复`vlm`模式下，某些不完整的表结构导致的解析中断问题
+
+### 2.0.5 (2025/06/17)
+
+- 修复了`sglang-client`模式下依然需要下载模型的问题
+- 修复了`sglang-client`模式需要依赖`torch`等实际运行不需要的包的问题
+- 修复了同一进程内尝试通过多个url启动多个`sglang-client`实例时，只有第一个生效的问题
+
+### 2.0.3 (2025/06/15)
+
+- 修复了当下载模型类型设置为`all`时，配置文件出现键值更新错误的问题
+- 修复了命令行模式下公式和表格功能开关不生效导致功能无法关闭的问题
+- 修复了`sglang-engine`模式下，0.4.7版本sglang的兼容性问题
+- 更新了sglang环境下部署完整版MinerU的Dockerfile和相关安装文档
+
+### 2.0.0 (2025/06/13)
+
+**全新架构**
+
+MinerU 2.0 在代码结构和交互方式上进行了深度重构，显著提升了系统的易用性、可维护性与扩展能力。
+
+- **去除第三方依赖限制**：彻底移除对 `pymupdf` 的依赖，推动项目向更开放、合规的开源方向迈进。
+- **开箱即用，配置便捷**：无需手动编辑 JSON 配置文件，绝大多数参数已支持命令行或 API 直接设置。
+- **模型自动管理**：新增模型自动下载与更新机制，用户无需手动干预即可完成模型部署。
+- **离线部署友好**：提供内置模型下载命令，支持完全断网环境下的部署需求。
+- **代码结构精简**：移除数千行冗余代码，简化类继承逻辑，显著提升代码可读性与开发效率。
+- **统一中间格式输出**：采用标准化的 `middle_json` 格式，兼容多数基于该格式的二次开发场景，确保生态业务无缝迁移。
+
+**全新模型**
+
+MinerU 2.0 集成了我们最新研发的小参数量、高性能多模态文档解析模型，实现端到端的高速、高精度文档理解。
+
+- **小模型，大能力**：模型参数不足 1B，却在解析精度上超越传统 72B 级别的视觉语言模型（VLM）。
+- **多功能合一**：单模型覆盖多语言识别、手写识别、版面分析、表格解析、公式识别、阅读顺序排序等核心任务。
+- **极致推理速度**：在单卡 NVIDIA 4090 上通过 `sglang` 加速，达到峰值吞吐量超过 10,000 token/s，轻松应对大规模文档处理需求。
+- **在线体验**：您可以在[MinerU.net](https://mineru.net/OpenSourceTools/Extractor)、[Hugging Face](https://huggingface.co/spaces/opendatalab/MinerU), 以及[ModelScope](https://www.modelscope.cn/studios/OpenDataLab/MinerU)体验我们的全新VLM模型
+
+**不兼容变更说明**
+
+为提升整体架构合理性与长期可维护性，本版本包含部分不兼容的变更：
+
+- Python 包名从 `magic-pdf` 更改为 `mineru`，命令行工具也由 `magic-pdf` 改为 `mineru`，请同步更新脚本与调用命令。
+- 出于对系统模块化设计与生态一致性的考虑，MinerU 2.0 已不再内置 LibreOffice 文档转换模块。如需处理 Office 文档，建议通过独立部署的 LibreOffice 服务先行转换为 PDF 格式，再进行后续解析操作。
+
+---
+
+## 1.x 系列历史版本
+
+### 1.3.12 (2025/05/24)
+
+增加ppocrv5模型的支持，将`ch_server`模型更新为`PP-OCRv5_rec_server`，`ch_lite`模型更新为`PP-OCRv5_rec_mobile`（需更新模型）
+
+- 在测试中，发现ppocrv5(server)对手写文档效果有一定提升，但在其余类别文档的精度略差于v4_server_doc，因此默认的ch模型保持不变，仍为`PP-OCRv4_server_rec_doc`。
+- 由于ppocrv5强化了手写场景和特殊字符的识别能力，因此您可以在日繁混合场景以及手写文档场景下手动选择使用ppocrv5模型
+- 您可通过lang参数`lang='ch_server'`(python api)或`--lang ch_server`(命令行)自行选择相应的模型：
+ - `ch` ：`PP-OCRv4_rec_server_doc`（默认）（中英日繁混合/1.5w字典）
+ - `ch_server` ：`PP-OCRv5_rec_server`（中英日繁混合+手写场景/1.8w字典）
+ - `ch_lite` ：`PP-OCRv5_rec_mobile`（中英日繁混合+手写场景/1.8w字典）
+ - `ch_server_v4` ：`PP-OCRv4_rec_server`（中英混合/6k字典）
+ - `ch_lite_v4` ：`PP-OCRv4_rec_mobile`（中英混合/6k字典）
+
+增加手写文档的支持，通过优化layout对手写文本区域的识别，现已支持手写文档的解析
+
+- 默认支持此功能，无需额外配置
+- 可以参考上述说明，手动选择ppocrv5模型以获得更好的手写文档解析效果
+
+`huggingface`和`modelscope`的demo已更新为支持手写识别和ppocrv5模型的版本，可自行在线体验
+
+### 1.3.10 (2025/04/29)
+
+- 支持使用自定义公式标识符，可通过修改用户目录下的`magic-pdf.json`文件中的`latex-delimiter-config`项实现。
+
+### 1.3.9 (2025/04/27)
+
+- 优化公式解析功能，提升公式渲染的成功率
+
+### 1.3.8 (2025/04/23)
+
+`ocr`默认模型(`ch`)更新为`PP-OCRv4_server_rec_doc`（需更新模型）
+
+- `PP-OCRv4_server_rec_doc`是在`PP-OCRv4_server_rec`的基础上，在更多中文文档数据和PP-OCR训练数据的混合数据训练而成，增加了部分繁体字、日文、特殊字符的识别能力，可支持识别的字符为1.5万+，除文档相关的文字识别能力提升外，也同时提升了通用文字的识别能力。
+- [PP-OCRv4_server_rec_doc/PP-OCRv4_server_rec/PP-OCRv4_mobile_rec 性能对比](https://paddlepaddle.github.io/PaddleX/latest/module_usage/tutorials/ocr_modules/text_recognition.html#_3)
+- 经验证，`PP-OCRv4_server_rec_doc`模型在`中英日繁`单种语言或多种语言混合场景均有明显精度提升，且速度与`PP-OCRv4_server_rec`相当，适合绝大部分场景使用。
+- `PP-OCRv4_server_rec_doc`在小部分纯英文场景可能会发生单词粘连问题，`PP-OCRv4_server_rec`则在此场景下表现更好，因此我们保留了`PP-OCRv4_server_rec`模型，用户可通过增加参数`lang='ch_server'`(python api)或`--lang ch_server`(命令行)调用。
+
+### 1.3.7 (2025/04/22)
+
+- 修复表格解析模型初始化时lang参数失效的问题
+- 修复在`cpu`模式下ocr和表格解析速度大幅下降的问题
+
+### 1.3.4 (2025/04/16)
+
+- 通过移除一些无用的块，小幅提升了ocr-det的速度
+- 修复部分情况下由footnote导致的页面内排序错误
+
+### 1.3.2 (2025/04/12)
+
+- 修复了windows系统下，在python3.13环境安装时一些依赖包版本不兼容的问题
+- 优化批量推理时的内存占用
+- 优化旋转90度表格的解析效果
+- 优化财报样本中超大表格的解析效果
+- 修复了在未指定OCR语言时，英文文本区域偶尔出现的单词黏连问题（需要更新模型）
+
+### 1.3.1 (2025/04/08)
+
+修复了一些兼容问题
+
+- 支持python 3.13
+- 为部分过时的linux系统（如centos7）做出最后适配，并不再保证后续版本的继续支持，[安装说明](https://github.com/opendatalab/MinerU/issues/1004)
+
+### 1.3.0 (2025/04/03)
+
+**安装与兼容性优化**
+
+- 通过移除layout中`layoutlmv3`的使用，解决了由`detectron2`导致的兼容问题
+- torch版本兼容扩展到2.2~2.6(2.5除外)
+- cuda兼容支持11.8/12.4/12.6/12.8（cuda版本由torch决定），解决部分用户50系显卡与H系显卡的兼容问题
+- python兼容版本扩展到3.10~3.12，解决了在非3.10环境下安装时自动降级到0.6.1的问题
+- 优化离线部署流程，部署成功后不需要联网下载任何模型文件
+
+**性能优化**
+
+- 通过支持多个pdf文件的batch处理（[脚本样例](demo/batch_demo.py)），提升了批量小文件的解析速度 (与1.0.1版本相比，公式解析速度最高提升超过1400%，整体解析速度最高提升超过500%)
+- 通过优化mfr模型的加载和使用，降低了显存占用并提升了解析速度(需重新执行[模型下载流程](docs/how_to_download_models_zh_cn.md)以获得模型文件的增量更新)
+- 优化显存占用，最低仅需6GB即可运行本项目
+- 优化了在mps设备上的运行速度
+
+**解析效果优化**
+
+- mfr模型更新到`unimernet(2503)`，解决多行公式中换行丢失的问题
+
+**易用性优化**
+
+- 通过使用`paddleocr2torch`，完全替代`paddle`框架以及`paddleocr`在项目中的使用，解决了`paddle`和`torch`的冲突问题，和由于`paddle`框架导致的线程不安全问题
+- 解析过程增加实时进度条显示，精准把握解析进度，让等待不再痛苦
+
+### 1.2.1 (2025/03/03)
+
+修复了一些问题
+
+- 修复在字母与数字的全角转半角操作时对标点符号的影响
+- 修复在某些情况下caption的匹配不准确问题
+- 修复在某些情况下的公式span丢失问题
+
+### 1.2.0 (2025/02/24)
+
+这个版本我们修复了一些问题，提升了解析的效率与精度：
+
+**性能优化**
+
+- auto模式下pdf文档的分类速度提升
+
+**解析优化**
+
+- 优化对包含水印文档的解析逻辑，显著提升包含水印文档的解析效果
+- 改进了单页内多个图像/表格与caption的匹配逻辑，提升了复杂布局下图文匹配的准确性
+
+**问题修复**
+
+- 修复在某些情况下图片/表格span被填充进textblock导致的异常
+- 修复在某些情况下标题block为空的问题
+
+### 1.1.0 (2025/01/22)
+
+在这个版本我们重点提升了解析的精度与效率：
+
+**模型能力升级**（需重新执行 [模型下载流程](https://github.com/opendatalab/MinerU/docs/how_to_download_models_zh_cn.md) 以获得模型文件的增量更新）
+
+- 布局识别模型升级到最新的 `doclayout_yolo(2501)` 模型，提升了layout识别精度
+- 公式解析模型升级到最新的 `unimernet(2501)` 模型，提升了公式识别精度
+
+**性能优化**
+
+- 在配置满足一定条件（显存16GB+）的设备上，通过优化资源占用和重构处理流水线，整体解析速度提升50%以上
+
+**解析效果优化**
+
+- 在线demo（[mineru.net](https://mineru.net/OpenSourceTools/Extractor) / [huggingface](https://huggingface.co/spaces/opendatalab/MinerU) / [modelscope](https://www.modelscope.cn/studios/OpenDataLab/MinerU)）上新增标题分级功能（测试版本，默认开启），支持对标题进行分级，提升文档结构化程度
+
+### 1.0.1 (2025/01/10)
+
+这是我们的第一个正式版本，在这个版本中，我们通过大量重构带来了全新的API接口和更广泛的兼容性，以及全新的自动语言识别功能：
+
+**全新API接口**
+
+- 对于数据侧API，我们引入了Dataset类，旨在提供一个强大而灵活的数据处理框架。该框架当前支持包括图像（.jpg及.png）、PDF、Word（.doc及.docx）、以及PowerPoint（.ppt及.pptx）在内的多种文档格式，确保了从简单到复杂的数据处理任务都能得到有效的支持。
+- 针对用户侧API，我们将MinerU的处理流程精心设计为一系列可组合的Stage阶段。每个Stage代表了一个特定的处理步骤，用户可以根据自身需求自由地定义新的Stage，并通过创造性地组合这些阶段来定制专属的数据处理流程。
+
+**更广泛的兼容性适配**
+
+- 通过优化依赖环境和配置项，确保在ARM架构的Linux系统上能够稳定高效运行。
+- 深度适配华为昇腾NPU加速，积极响应信创要求，提供自主可控的高性能计算能力，助力人工智能应用平台的国产化应用与发展。 [NPU加速教程](https://github.com/opendatalab/MinerU/docs/README_Ascend_NPU_Acceleration_zh_CN.md)
+
+**自动语言识别**
+
+- 通过引入全新的语言识别模型， 在文档解析中将 `lang` 配置为 `auto`，即可自动选择合适的OCR语言模型，提升扫描类文档解析的准确性。
+
+---
+
+## 0.x 系列历史版本
+
+### 0.10.0 (2024/11/22)
+
+通过引入混合OCR文本提取能力：
+
+- 在公式密集、span区域不规范、部分文本使用图像表现等复杂文本分布场景下获得解析效果的显著提升
+- 同时具备文本模式内容提取准确、速度更快与OCR模式span/line区域识别更准的双重优势
+
+### 0.9.3 (2024/11/15)
+
+为表格识别功能接入了[RapidTable](https://github.com/RapidAI/RapidTable),单表解析速度提升10倍以上，准确率更高，显存占用更低
+
+### 0.9.2 (2024/11/06)
+
+为表格识别功能接入了[StructTable-InternVL2-1B](https://huggingface.co/U4R/StructTable-InternVL2-1B)模型
+
+### 0.9.0 (2024/10/31)
+
+这是我们进行了大量代码重构的全新版本，解决了众多问题，提升了性能，降低了硬件需求，并提供了更丰富的易用性：
+
+- 重构排序模块代码，使用 [layoutreader](https://github.com/ppaanngggg/layoutreader) 进行阅读顺序排序，确保在各种排版下都能实现极高准确率
+- 重构段落拼接模块，在跨栏、跨页、跨图、跨表情况下均能实现良好的段落拼接效果
+- 重构列表和目录识别功能，极大提升列表块和目录块识别的准确率及对应文本段落的解析效果
+- 重构图、表与描述性文本的匹配逻辑，大幅提升 caption 和 footnote 与图表的匹配准确率，并将描述性文本的丢失率降至接近0
+- 增加 OCR 的多语言支持，支持 84 种语言的检测与识别，语言支持列表详见 [OCR 语言支持列表](https://paddlepaddle.github.io/PaddleOCR/latest/ppocr/blog/multi_languages.html#5)
+- 增加显存回收逻辑及其他显存优化措施，大幅降低显存使用需求。开启除表格加速外的全部加速功能(layout/公式/OCR)的显存需求从16GB降至8GB，开启全部加速功能的显存需求从24GB降至10GB
+- 优化配置文件的功能开关，增加独立的公式检测开关，无需公式检测时可大幅提升速度和解析效果
+- 集成 [PDF-Extract-Kit 1.0](https://github.com/opendatalab/PDF-Extract-Kit)
+ - 加入自研的 `doclayout_yolo` 模型，在相近解析效果情况下比原方案提速10倍以上，可通过配置文件与 `layoutlmv3` 自由切换
+ - 公式解析升级至 `unimernet 0.2.1`，在提升公式解析准确率的同时，大幅降低显存需求
+ - 因 `PDF-Extract-Kit 1.0` 更换仓库，需要重新下载模型，步骤详见 [如何下载模型](https://github.com/opendatalab/MinerU/docs/how_to_download_models_zh_cn.md)
+
+### 0.8.1 (2024/09/27)
+
+修复了一些bug，同时提供了[在线demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/)的[本地化部署版本](https://github.com/opendatalab/MinerU/projects/web_demo/README_zh-CN.md)和[前端界面](https://github.com/opendatalab/MinerU/projects/web/README_zh-CN.md)
+
+### 0.8.0 (2024/09/09)
+
+支持Dockerfile快速部署，同时上线了huggingface、modelscope demo
+
+### 0.7.1 (2024/08/30)
+
+集成了paddle tablemaster表格识别功能
+
+### 0.7.0b1 (2024/08/09)
+
+简化安装步骤提升易用性，加入表格识别功能
+
+### 0.6.2b1 (2024/08/01)
+
+优化了依赖冲突问题和安装文档
+
+### 首次开源 (2024/07/05)
+
+MinerU项目首次开源发布
diff --git a/docs/zh/reference/index.md b/docs/zh/reference/index.md
new file mode 100644
index 000000000..a1b61fb65
--- /dev/null
+++ b/docs/zh/reference/index.md
@@ -0,0 +1,27 @@
+# 参考文档
+
+本章节提供了 MinerU 项目的详细参考资料。在这里您可以找到技术规范、API 文档、输出文件格式说明以及版本历史记录。
+
+## 目录
+
+- [输出文件说明](./output_files.md) - 详细介绍所有输出文件及其格式
+- [更新日志](./changelog.md) - 版本更新历史和发布说明
+
+## 文档概览
+
+### 输出文件说明
+
+理解 MinerU 生成的输出文件对于有效使用工具至关重要。输出文件文档提供了：
+
+- **可视化调试文件**：帮助您理解文档解析过程
+- **结构化数据文件**：包含详细的解析结果，可用于进一步处理
+- **文件格式规范**：每种输出文件类型的详细说明
+
+### 更新日志
+
+更新日志记录了 MinerU 的演进历程，包括：
+
+- **版本更新**：每个版本的新功能和改进
+- **错误修复**：每个版本中解决的问题
+- **重大变更**：可能影响您使用的重要变更
+- **功能弃用**：正在逐步淘汰的功能
diff --git a/docs/zh/reference/output_files.md b/docs/zh/reference/output_files.md
new file mode 100644
index 000000000..0d8a0505a
--- /dev/null
+++ b/docs/zh/reference/output_files.md
@@ -0,0 +1,862 @@
+# MinerU 输出文件说明
+
+## 概览
+
+`mineru` 命令执行后，除了输出主要的 markdown 文件外，还会生成多个辅助文件用于调试、质检和进一步处理。这些文件包括：
+
+具体会生成哪些文件，取决于后端类型和输入文档类型。
+
+- **可视化调试文件**：帮助用户直观了解文档解析过程和结果
+- **结构化数据文件**：包含详细的解析数据，可用于二次开发
+- 多模态 markdown 输出中，`image` / `chart` 默认以截图为主；若块内存在 `content`，会在图片后追加一个默认折叠的 HTML `
| Site | Percentile | |||||||||
| 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 | |
| Traralgon Ck | P | P,* | P | P | P, | P, | P, | P, | P | P |
| Redhill | P,T | P,T | ，* | ** | P.T | P,* | P* | P* | * | ，* |
| Pine Ck | P,T | P,T | P,T | P,T | T | T | T | na | na | |
| Stewarts Ck 5 | P,T | P,T | P,T | P,T | P.T | P.T | P,T | na | na | na |
| Glendhu 2 | P | P,T | P,* | P,T | P.T | P,ns | P,T | P,T | P,T | P,T |
| Cathedral Peak 2 | P,T | P,T | P,T | P,T | P,T | *,T | P,T | P,T | P,T | T |
| Cathedral Peak 3 | P.T | P.T | P,T | P,T | P,T | T | P,T | P,T | P,T | T |
| Lambrechtsbos A | P,T | P | P | P,T | *,T | *,T | *,T | *,T | *,T | T |
| Lambrechtsbos B | P,T | P,T | P,T | P,T | P,T | P,T | P,T | P,T | T | T |
| Biesievlei | P,T | P.T | P,T | P,T | *,T | *,T | T | T | P,T | P,T |
-
-
-## 方法二：从 ModelScope 下载模型
-
-### 使用python脚本 从ModelScope下载模型文件
-
-```bash
-pip install modelscope
-wget https://gitee.com/myhloli/MinerU/raw/master/docs/download_models.py -O download_models.py
-python download_models.py
-```
-
-python脚本会自动下载模型文件并配置好配置文件中的模型目录
-
-配置文件可以在用户目录中找到，文件名为`magic-pdf.json`
-
-> windows的用户目录为 "C:\\Users\\用户名", linux用户目录为 "/home/用户名", macOS用户目录为 "/Users/用户名"
-
-# 此前下载过模型，如何更新
-
-## 1. 通过git lfs下载过模型
-
-> 由于部分用户反馈通过git lfs下载模型文件遇到下载不全和模型文件损坏情况，现已不推荐使用该方式下载。
-
-当magic-pdf <= 0.8.1时，如此前通过 git lfs 下载过模型文件，可以进入到之前的下载目录中，通过`git pull`命令更新模型。
-
-> 0.9.x及以后版本由于PDF-Extract-Kit 1.0更换仓库和新增layout排序模型，不能通过`git pull`命令更新，需要使用python脚本一键更新。
-
-
-## 2. 通过 Hugging Face 或 Model Scope 下载过模型
-
-如此前通过 HuggingFace 或 Model Scope 下载过模型，可以重复执行此前的模型下载python脚本，将会自动将模型目录更新到最新版本。
diff --git a/docs/images/MinerU-logo-hq.png b/docs/images/MinerU-logo-hq.png
deleted file mode 100644
index 7b33fcda2..000000000
Binary files a/docs/images/MinerU-logo-hq.png and /dev/null differ
diff --git a/docs/images/MinerU-logo.png b/docs/images/MinerU-logo.png
index 2e6fdf3a9..09ab46b23 100644
Binary files a/docs/images/MinerU-logo.png and b/docs/images/MinerU-logo.png differ
diff --git a/docs/images/layout_example.png b/docs/images/layout_example.png
index 14e2116f6..4a57dffef 100644
Binary files a/docs/images/layout_example.png and b/docs/images/layout_example.png differ
diff --git a/docs/images/logo.png b/docs/images/logo.png
new file mode 100644
index 000000000..01818084c
Binary files /dev/null and b/docs/images/logo.png differ
diff --git a/docs/images/logo.svg b/docs/images/logo.svg
new file mode 100644
index 000000000..655397802
--- /dev/null
+++ b/docs/images/logo.svg
@@ -0,0 +1,22 @@
+
diff --git a/docs/images/web_demo_1.png b/docs/images/web_demo_1.png
deleted file mode 100644
index 04adff268..000000000
Binary files a/docs/images/web_demo_1.png and /dev/null differ
diff --git a/docs/output_file_en_us.md b/docs/output_file_en_us.md
deleted file mode 100644
index 1fe9a1274..000000000
--- a/docs/output_file_en_us.md
+++ /dev/null
@@ -1,327 +0,0 @@
-## Overview
-
-After executing the `magic-pdf` command, in addition to outputting files related to markdown, several other files unrelated to markdown will also be generated. These files will be introduced one by one.
-
-### some_pdf_layout.pdf
-
-Each page layout consists of one or more boxes. The number at the top left of each box indicates its sequence number. Additionally, in `layout.pdf`, different content blocks are highlighted with different background colors.
-
-
-
-### some_pdf_spans.pdf
-
-All spans on the page are drawn with different colored line frames according to the span type. This file can be used for quality control, allowing for quick identification of issues such as missing text or unrecognized inline formulas.
-
-
-
-### some_pdf_model.json
-
-#### Structure Definition
-
-```python
-from pydantic import BaseModel, Field
-from enum import IntEnum
-
-class CategoryType(IntEnum):
- title = 0 # Title
- plain_text = 1 # Text
- abandon = 2 # Includes headers, footers, page numbers, and page annotations
- figure = 3 # Image
- figure_caption = 4 # Image description
- table = 5 # Table
- table_caption = 6 # Table description
- table_footnote = 7 # Table footnote
- isolate_formula = 8 # Block formula
- formula_caption = 9 # Formula label
-
- embedding = 13 # Inline formula
- isolated = 14 # Block formula
- text = 15 # OCR recognition result
-
-
-class PageInfo(BaseModel):
- page_no: int = Field(description="Page number, the first page is 0", ge=0)
- height: int = Field(description="Page height", gt=0)
- width: int = Field(description="Page width", ge=0)
-
-class ObjectInferenceResult(BaseModel):
- category_id: CategoryType = Field(description="Category", ge=0)
- poly: list[float] = Field(description="Quadrilateral coordinates, representing the coordinates of the top-left, top-right, bottom-right, and bottom-left points respectively")
- score: float = Field(description="Confidence of the inference result")
- latex: str | None = Field(description="LaTeX parsing result", default=None)
- html: str | None = Field(description="HTML parsing result", default=None)
-
-class PageInferenceResults(BaseModel):
- layout_dets: list[ObjectInferenceResult] = Field(description="Page recognition results", ge=0)
- page_info: PageInfo = Field(description="Page metadata")
-
-
-# The inference results of all pages, ordered by page number, are stored in a list as the inference results of MinerU
-inference_result: list[PageInferenceResults] = []
-
-```
-
-The format of the poly coordinates is \[x0, y0, x1, y1, x2, y2, x3, y3\], representing the coordinates of the top-left, top-right, bottom-right, and bottom-left points respectively.
-
-
-#### example
-
-```json
-[
- {
- "layout_dets": [
- {
- "category_id": 2,
- "poly": [
- 99.1906967163086,
- 100.3119125366211,
- 730.3707885742188,
- 100.3119125366211,
- 730.3707885742188,
- 245.81326293945312,
- 99.1906967163086,
- 245.81326293945312
- ],
- "score": 0.9999997615814209
- }
- ],
- "page_info": {
- "page_no": 0,
- "height": 2339,
- "width": 1654
- }
- },
- {
- "layout_dets": [
- {
- "category_id": 5,
- "poly": [
- 99.13092803955078,
- 2210.680419921875,
- 497.3183898925781,
- 2210.680419921875,
- 497.3183898925781,
- 2264.78076171875,
- 99.13092803955078,
- 2264.78076171875
- ],
- "score": 0.9999997019767761
- }
- ],
- "page_info": {
- "page_no": 1,
- "height": 2339,
- "width": 1654
- }
- }
-]
-```
-
-### some_pdf_middle.json
-
-| Field Name | Description |
-| :------------- | :------------------------------------------------------------------------------------------------------------- |
-| pdf_info | list, each element is a dict representing the parsing result of each PDF page, see the table below for details |
-| \_parse_type | ocr \| txt, used to indicate the mode used in this intermediate parsing state |
-| \_version_name | string, indicates the version of magic-pdf used in this parsing |
-
-方法一：从 Hugging Face 下载模型
-使用python脚本 从Hugging Face下载模型文件
-pip install huggingface_hub
-wget https://gitee.com/myhloli/MinerU/raw/master/docs/download_models_hf.py -O download_models_hf.py
-python download_models_hf.py- -**pdf_info** - -Field structure description - -| Field Name | Description | -| :------------------ | :----------------------------------------------------------------------------------------------------------------- | -| preproc_blocks | Intermediate result after PDF preprocessing, not yet segmented | -| layout_bboxes | Layout segmentation results, containing layout direction (vertical, horizontal), and bbox, sorted by reading order | -| page_idx | Page number, starting from 0 | -| page_size | Page width and height | -| \_layout_tree | Layout tree structure | -| images | list, each element is a dict representing an img_block | -| tables | list, each element is a dict representing a table_block | -| interline_equations | list, each element is a dict representing an interline_equation_block | -| discarded_blocks | List, block information returned by the model that needs to be dropped | -| para_blocks | Result after segmenting preproc_blocks | - -In the above table, `para_blocks` is an array of dicts, each dict representing a block structure. A block can support up to one level of nesting. - -
- -**block** - -The outer block is referred to as a first-level block, and the fields in the first-level block include: - -| Field Name | Description | -| :--------- | :------------------------------------------------------------- | -| type | Block type (table\|image) | -| bbox | Block bounding box coordinates | -| blocks | list, each element is a dict representing a second-level block | - -
-There are only two types of first-level blocks: "table" and "image". All other blocks are second-level blocks. - -The fields in a second-level block include: - -| Field Name | Description | -| :--------- | :---------------------------------------------------------------------------------------------------------- | -| type | Block type | -| bbox | Block bounding box coordinates | -| lines | list, each element is a dict representing a line, used to describe the composition of a line of information | - -Detailed explanation of second-level block types - -| type | Description | -| :----------------- | :--------------------- | -| image_body | Main body of the image | -| image_caption | Image description text | -| table_body | Main body of the table | -| table_caption | Table description text | -| table_footnote | Table footnote | -| text | Text block | -| title | Title block | -| interline_equation | Block formula | - -
- -**line** - -The field format of a line is as follows: - -| Field Name | Description | -| :--------- | :------------------------------------------------------------------------------------------------------ | -| bbox | Bounding box coordinates of the line | -| spans | list, each element is a dict representing a span, used to describe the composition of the smallest unit | - -
- -**span** - -| Field Name | Description | -| :------------------ | :------------------------------------------------------------------------------------------------------- | -| bbox | Bounding box coordinates of the span | -| type | Type of the span | -| content \| img_path | Text spans use content, chart spans use img_path to store the actual text or screenshot path information | - -The types of spans are as follows: - -| type | Description | -| :----------------- | :------------- | -| image | Image | -| table | Table | -| text | Text | -| inline_equation | Inline formula | -| interline_equation | Block formula | - -**Summary** - -A span is the smallest storage unit for all elements. - -The elements stored within para_blocks are block information. - -The block structure is as follows: - -First-level block (if any) -> Second-level block -> Line -> Span - -#### example - -```json -{ - "pdf_info": [ - { - "preproc_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ], - "layout_bboxes": [ - { - "layout_bbox": [ - 52, - 61, - 294, - 731 - ], - "layout_label": "V", - "sub_layout": [] - } - ], - "page_idx": 0, - "page_size": [ - 612.0, - 792.0 - ], - "_layout_tree": [], - "images": [], - "tables": [], - "interline_equations": [], - "discarded_blocks": [], - "para_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ] - } - ], - "_parse_type": "txt", - "_version_name": "0.6.1" -} -``` diff --git a/docs/output_file_zh_cn.md b/docs/output_file_zh_cn.md deleted file mode 100644 index dd921da95..000000000 --- a/docs/output_file_zh_cn.md +++ /dev/null @@ -1,326 +0,0 @@ -## 概览 - -`magic-pdf` 命令执行后除了输出和 markdown 有关的文件以外，还会生成若干个和 markdown 无关的文件。现在将一一介绍这些文件 - -### some_pdf_layout.pdf - -每一页的 layout 均由一个或多个框组成。 每个框左上脚的数字表明它们的序号。此外 layout.pdf 框内用不同的背景色块圈定不同的内容块。 - - - -### some_pdf_spans.pdf - -根据 span 类型的不同，采用不同颜色线框绘制页面上所有 span。该文件可以用于质检，可以快速排查出文本丢失、行间公式未识别等问题。 - - - -### some_pdf_model.json - -#### 结构定义 - -```python -from pydantic import BaseModel, Field -from enum import IntEnum - -class CategoryType(IntEnum): - title = 0 # 标题 - plain_text = 1 # 文本 - abandon = 2 # 包括页眉页脚页码和页面注释 - figure = 3 # 图片 - figure_caption = 4 # 图片描述 - table = 5 # 表格 - table_caption = 6 # 表格描述 - table_footnote = 7 # 表格注释 - isolate_formula = 8 # 行间公式 - formula_caption = 9 # 行间公式的标号 - - embedding = 13 # 行内公式 - isolated = 14 # 行间公式 - text = 15 # ocr 识别结果 - - -class PageInfo(BaseModel): - page_no: int = Field(description="页码序号，第一页的序号是 0", ge=0) - height: int = Field(description="页面高度", gt=0) - width: int = Field(description="页面宽度", ge=0) - -class ObjectInferenceResult(BaseModel): - category_id: CategoryType = Field(description="类别", ge=0) - poly: list[float] = Field(description="四边形坐标, 分别是 左上，右上，右下，左下 四点的坐标") - score: float = Field(description="推理结果的置信度") - latex: str | None = Field(description="latex 解析结果", default=None) - html: str | None = Field(description="html 解析结果", default=None) - -class PageInferenceResults(BaseModel): - layout_dets: list[ObjectInferenceResult] = Field(description="页面识别结果", ge=0) - page_info: PageInfo = Field(description="页面元信息") - - -# 所有页面的推理结果按照页码顺序依次放到列表中即为 minerU 推理结果 -inference_result: list[PageInferenceResults] = [] - -``` - -poly 坐标的格式 \[x0, y0, x1, y1, x2, y2, x3, y3\], 分别表示左上、右上、右下、左下四点的坐标 - - -#### 示例数据 - -```json -[ - { - "layout_dets": [ - { - "category_id": 2, - "poly": [ - 99.1906967163086, - 100.3119125366211, - 730.3707885742188, - 100.3119125366211, - 730.3707885742188, - 245.81326293945312, - 99.1906967163086, - 245.81326293945312 - ], - "score": 0.9999997615814209 - } - ], - "page_info": { - "page_no": 0, - "height": 2339, - "width": 1654 - } - }, - { - "layout_dets": [ - { - "category_id": 5, - "poly": [ - 99.13092803955078, - 2210.680419921875, - 497.3183898925781, - 2210.680419921875, - 497.3183898925781, - 2264.78076171875, - 99.13092803955078, - 2264.78076171875 - ], - "score": 0.9999997019767761 - } - ], - "page_info": { - "page_no": 1, - "height": 2339, - "width": 1654 - } - } -] -``` - -### some_pdf_middle.json - -| 字段名 | 解释 | -| :------------- | :----------------------------------------------------------------- | -| pdf_info | list，每个元素都是一个dict,这个dict是每一页pdf的解析结果，详见下表 | -| \_parse_type | ocr \| txt，用来标识本次解析的中间态使用的模式 | -| \_version_name | string, 表示本次解析使用的 magic-pdf 的版本号 | - -
- -**pdf_info** -字段结构说明 - -| 字段名 | 解释 | -| :------------------ | :------------------------------------------------------------------- | -| preproc_blocks | pdf预处理后，未分段的中间结果 | -| layout_bboxes | 布局分割的结果，含有布局的方向（垂直、水平），和bbox，按阅读顺序排序 | -| page_idx | 页码，从0开始 | -| page_size | 页面的宽度和高度 | -| \_layout_tree | 布局树状结构 | -| images | list，每个元素是一个dict，每个dict表示一个img_block | -| tables | list，每个元素是一个dict，每个dict表示一个table_block | -| interline_equations | list，每个元素是一个dict，每个dict表示一个interline_equation_block | -| discarded_blocks | List, 模型返回的需要drop的block信息 | -| para_blocks | 将preproc_blocks进行分段之后的结果 | - -上表中 `para_blocks` 是个dict的数组，每个dict是一个block结构，block最多支持一次嵌套 - -
- -**block** - -外层block被称为一级block，一级block中的字段包括 - -| 字段名 | 解释 | -| :----- | :---------------------------------------------- | -| type | block类型（table\|image） | -| bbox | block矩形框坐标 | -| blocks | list，里面的每个元素都是一个dict格式的二级block | - -
-一级block只有"table"和"image"两种类型，其余block均为二级block - -二级block中的字段包括 - -| 字段名 | 解释 | -| :----- | :----------------------------------------------------------- | -| type | block类型 | -| bbox | block矩形框坐标 | -| lines | list，每个元素都是一个dict表示的line，用来描述一行信息的构成 | - -二级block的类型详解 - -| type | desc | -| :----------------- | :------------- | -| image_body | 图像的本体 | -| image_caption | 图像的描述文本 | -| table_body | 表格本体 | -| table_caption | 表格的描述文本 | -| table_footnote | 表格的脚注 | -| text | 文本块 | -| title | 标题块 | -| interline_equation | 行间公式块 | - -
- -**line** - -line 的 字段格式如下 - -| 字段名 | 解释 | -| :----- | :------------------------------------------------------------------- | -| bbox | line的矩形框坐标 | -| spans | list，每个元素都是一个dict表示的span，用来描述一个最小组成单元的构成 | - -
- -**span** - -| 字段名 | 解释 | -| :------------------ | :------------------------------------------------------------------------------- | -| bbox | span的矩形框坐标 | -| type | span的类型 | -| content \| img_path | 文本类型的span使用content，图表类使用img_path 用来存储实际的文本或者截图路径信息 | - -span 的类型有如下几种 - -| type | desc | -| :----------------- | :------- | -| image | 图片 | -| table | 表格 | -| text | 文本 | -| inline_equation | 行内公式 | -| interline_equation | 行间公式 | - -**总结** - -span是所有元素的最小存储单元 - -para_blocks内存储的元素为区块信息 - -区块结构为 - -一级block(如有)->二级block->line->span - -#### 示例数据 - -```json -{ - "pdf_info": [ - { - "preproc_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ], - "layout_bboxes": [ - { - "layout_bbox": [ - 52, - 61, - 294, - 731 - ], - "layout_label": "V", - "sub_layout": [] - } - ], - "page_idx": 0, - "page_size": [ - 612.0, - 792.0 - ], - "_layout_tree": [], - "images": [], - "tables": [], - "interline_equations": [], - "discarded_blocks": [], - "para_blocks": [ - { - "type": "text", - "bbox": [ - 52, - 61.956024169921875, - 294, - 82.99800872802734 - ], - "lines": [ - { - "bbox": [ - 52, - 61.956024169921875, - 294, - 72.0000228881836 - ], - "spans": [ - { - "bbox": [ - 54.0, - 61.956024169921875, - 296.2261657714844, - 72.0000228881836 - ], - "content": "dependent on the service headway and the reliability of the departure ", - "type": "text", - "score": 1.0 - } - ] - } - ] - } - ] - } - ], - "_parse_type": "txt", - "_version_name": "0.6.1" -} -``` diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 000000000..6669adadd --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,4 @@ +mkdocs +mkdocs-static-i18n +markdown-gfm-admonition +mkdocs-video \ No newline at end of file diff --git a/docs/zh/demo/index.md b/docs/zh/demo/index.md new file mode 100644 index 000000000..103ffc5e6 --- /dev/null +++ b/docs/zh/demo/index.md @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/zh/faq/index.md b/docs/zh/faq/index.md new file mode 100644 index 000000000..e4a198ded --- /dev/null +++ b/docs/zh/faq/index.md @@ -0,0 +1,29 @@ +# 常见问题解答 + +如果未能列出您的问题，您也可以使用[DeepWiki](https://deepwiki.com/opendatalab/MinerU)与AI助手交流，这可以解决大部分常见问题。 + +如果您仍然无法解决问题，您可通过[Discord](https://discord.gg/Tdedn9GTXq)或[WeChat](https://mineru.net/community-portal/?aliasId=3c430f94)加入社区，与其他用户和开发者交流。 + +??? question "在WSL2的Ubuntu22.04中遇到报错`ImportError: libGL.so.1: cannot open shared object file: No such file or directory`" + + WSL2的Ubuntu22.04中缺少`libgl`库，可通过以下命令安装`libgl`库解决： + + ```bash + sudo apt-get install libgl1-mesa-glx + ``` + + 参考：[#388](https://github.com/opendatalab/MinerU/issues/388) + +??? question "在 Linux 系统安装并使用时，解析结果缺失部份文字信息。" + + MinerU在>=2.0的版本中使用`pypdfium2`代替`pymupdf`作为PDF页面的渲染引擎，以解决AGPLv3的许可证问题，在某些Linux发行版，由于缺少CJK字体，可能会在将PDF渲染成图片的过程中丢失部份文字。 + 为了解决这个问题，您可以通过以下命令安装noto字体包，这在Ubuntu/debian系统中有效： + ```bash + sudo apt update + sudo apt install fonts-noto-core + sudo apt install fonts-noto-cjk + fc-cache -fv + ``` + 也可以直接使用我们的[Docker部署](../quick_start/docker_deployment.md)方式构建镜像，镜像中默认包含以上字体包。 + + 参考：[#2915](https://github.com/opendatalab/MinerU/issues/2915) \ No newline at end of file diff --git a/docs/zh/index.md b/docs/zh/index.md new file mode 100644 index 000000000..ee5cdb4f4 --- /dev/null +++ b/docs/zh/index.md @@ -0,0 +1,75 @@ +
+
+
+
+
+
+[](https://github.com/opendatalab/MinerU)
+[](https://github.com/opendatalab/MinerU)
+[](https://github.com/opendatalab/MinerU/issues)
+[](https://github.com/opendatalab/MinerU/issues)
+[](https://pypi.org/project/mineru/)
+[](https://pypi.org/project/mineru/)
+[](https://pepy.tech/project/mineru)
+[](https://pepy.tech/project/mineru)
+[](https://mineru.net/OpenSourceTools/Extractor?source=github)
+[](https://www.modelscope.cn/studios/OpenDataLab/MinerU)
+[](https://huggingface.co/spaces/opendatalab/MinerU)
+[](https://colab.research.google.com/gist/myhloli/a3cb16570ab3cfeadf9d8f0ac91b4fca/mineru_demo.ipynb)
+[](https://arxiv.org/abs/2409.18839)
+[](https://arxiv.org/abs/2509.22186)
+[](https://arxiv.org/abs/2604.04771)
+[](https://deepwiki.com/opendatalab/MinerU)
+
+
+
+
+
+
+
+
+
+## 项目简介
+
+MinerU 是一款文档解析工具，可将 `PDF`、图片以及 `DOCX`、`PPTX`、`XLSX` 转化为机器可读格式（如 Markdown、JSON），便于后续检索、抽取与二次处理。
+MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练过程中，我们将会集中精力解决科技文献中的符号转化问题，希望在大模型时代为科技发展做出贡献。
+相比国内外知名商用产品MinerU还很年轻，如果遇到问题或者结果不及预期请到[issue](https://github.com/opendatalab/MinerU/issues)提交问题，同时**附上相关文档或样例文件**。
+
+
+
+## 主要功能
+
+- 支持 `PDF`、图片与 `DOCX`、`PPTX`、`XLSX` 输入
+- 删除页眉、页脚、脚注、页码等元素，确保语义连贯
+- 输出符合人类阅读顺序的文本，适用于单栏、多栏及复杂排版
+- 保留原文档的结构，包括标题、段落、列表等
+- 提取图像、图片描述、表格、表格标题及脚注
+- 自动识别并转换文档中的公式为LaTeX格式
+- 自动识别并转换文档中的表格为HTML格式
+- 自动检测扫描版PDF和乱码PDF，并启用OCR功能
+- OCR支持109种语言的检测与识别
+- 支持多种输出格式，如多模态与NLP的Markdown、按阅读顺序排序的JSON、含有丰富信息的中间格式等
+- 支持多种可视化结果，包括layout可视化、span可视化等，便于高效确认输出效果与质检
+- 内置命令行、FastAPI、Gradio WebUI，支持本地编排和多服务部署
+- 支持纯CPU环境运行，并支持 GPU(CUDA)/NPU(CANN)/MPS 加速
+- 兼容Windows、Linux和Mac平台
+
+
+## 使用指南
+
+- [快速上手指南](./quick_start/index.md)
+- [详细使用说明](./usage/index.md)
+
+## 许可证信息
+
+本仓库采用 [MinerU 开源许可证](https://github.com/opendatalab/MinerU/blob/master/LICENSE.md) 进行许可，基于 Apache 2.0 并附带额外条款。
diff --git a/docs/zh/quick_start/docker_deployment.md b/docs/zh/quick_start/docker_deployment.md
new file mode 100644
index 000000000..8d74c659a
--- /dev/null
+++ b/docs/zh/quick_start/docker_deployment.md
@@ -0,0 +1,92 @@
+# 使用docker部署Mineru
+
+MinerU提供了便捷的docker部署方式，这有助于快速搭建环境并解决一些棘手的环境兼容问题。
+
+## 使用 Dockerfile 构建镜像
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/Dockerfile
+docker build -t mineru:latest -f Dockerfile .
+```
+
+## Docker说明
+
+Mineru的docker使用了`vllm/vllm-openai`作为基础镜像，因此在docker中默认集成了`vllm`推理加速框架和必需的依赖环境。因此在满足条件的设备上，您可以直接使用`vllm`加速VLM模型推理。
+> [!NOTE]
+> 使用`vllm`加速VLM模型推理需要满足的条件是：
+>
+> - 设备包含Volta及以后架构的显卡，且可用显存大于等于8G。
+> - 物理机的显卡驱动应支持CUDA 12.9.1或更高版本，可通过`nvidia-smi`命令检查驱动版本。
+> - docker中能够访问物理机的显卡设备。
+
+
+## 启动 Docker 容器
+
+```bash
+docker run --gpus all \
+ --shm-size 32g \
+ -p 30000:30000 -p 7860:7860 -p 8000:8000 -p 8002:8002 \
+ --ipc=host \
+ -it mineru:latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，并映射了一些端口用于可能会使用的服务，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+## 通过 Docker Compose 直接启动服务
+
+我们提供了[compose.yml](https://github.com/opendatalab/MinerU/blob/master/docker/compose.yaml)文件，您可以通过它来快速启动MinerU服务。
+
+```bash
+# 下载 compose.yaml 文件
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/compose.yaml
+```
+>[!NOTE]
+>
+>- `compose.yaml`文件中包含了MinerU的多个服务配置，您可以根据需要选择启动特定的服务。
+>- 不同的服务可能会有额外的参数配置，您可以在`compose.yaml`文件中查看并编辑。
+>- 由于`vllm`推理加速框架预分配显存的特性，您可能无法在同一台机器上同时运行多个`vllm`服务，因此请确保在启动`vlm-openai-server`服务或使用`vlm-vllm-engine`后端时，其他可能使用显存的服务已停止。
+
+---
+
+### 启动 openai兼容接口 服务
+并通过`vlm-http-client`后端连接`openai-server`
+ ```bash
+ docker compose -f compose.yaml --profile openai-server up -d
+ ```
+ >[!TIP]
+ >在另一个终端中通过http client连接openai server（只需cpu与网络，不需要vllm环境）
+ > ```bash
+ > mineru -p +🚀MinerU 官网入口→✅ 免装在线版 ✅ 全功能客户端 ✅ 开发者API在线调用，省去部署麻烦，多种产品形态一键get，速冲！ +
+ + + + +| 解析后端 | +pipeline | +*-auto-engine | +*-http-client | +||
|---|---|---|---|---|---|
| hybrid | +vlm | +hybrid | +vlm | +||
| 后端特性 | +兼容性好 | +硬件配置要求较高 | +适用于OpenAI兼容服务器2 | +||
| 精度指标1 | +86+ | +90+ | +|||
| 操作系统 | +Linux3 / Windows4 / macOS5 | +||||
| 纯CPU平台支持 | +✅ | +❌ | +✅ | +||
| GPU加速支持 | +Volta及以后架构GPU或Apple Silicon | +不需要 | +|||
| 显存最低要求 | +4GB | +8GB | +8GB | +2GB | +|
| 内存要求 | +最低16GB以上,推荐32GB以上 | +16GB | +|||
| 磁盘空间要求 | +20GB以上,推荐使用SSD | +2GB | +|||
| python版本 | +3.10-3.13 | +||||
` 内容块，其中折叠标题优先使用块的 `sub_type`，否则回退为 `image content` 或 `chart content`
+
+下面将详细介绍每个文件的作用和格式。
+
+## 可视化调试文件
+
+### 布局分析文件 (layout.pdf)
+
+**文件命名格式**：`{原文件名}_layout.pdf`
+
+**功能说明**：
+
+- 可视化展示每一页的布局分析结果
+- 每个检测框右上角的数字表示阅读顺序
+- 使用不同背景色块区分不同类型的内容块
+
+**使用场景**：
+
+- 检查布局分析是否正确
+- 确认阅读顺序是否合理
+- 调试布局相关问题
+
+
+
+### 文本片段文件 (span.pdf)
+
+> [!NOTE]
+> 仅适用于 pipeline 后端
+
+**文件命名格式**：`{原文件名}_span.pdf`
+
+**功能说明**：
+
+- 根据 span 类型使用不同颜色线框标注页面内容
+- 用于质量检查和问题排查
+
+**使用场景**：
+
+- 快速排查文本丢失问题
+- 检查行内公式识别情况
+- 验证文本分割准确性
+
+
+
+## 结构化数据文件
+
+> [!IMPORTANT]
+> 2.5版本vlm后端的输出存在较大变化，与pipeline版本存在不兼容情况，如需基于结构化输出进行二次开发，请仔细阅读本文档内容。
+
+### pipeline 后端 输出结果
+
+#### 模型推理结果 (model.json)
+
+**文件命名格式**：`{原文件名}_model.json`
+
+##### 示例数据
+
+```json
+[
+ {
+ "cls_id": 12,
+ "label": "header",
+ "score": 0.93,
+ "bbox": [
+ 1217,
+ 104,
+ 1516,
+ 134
+ ],
+ "index": 2
+ },
+ {
+ "cls_id": 6,
+ "label": "doc_title",
+ "score": 0.9751,
+ "bbox": [
+ 275,
+ 181,
+ 1512,
+ 292
+ ],
+ "index": 3
+ },
+ {
+ "cls_id": 22,
+ "label": "text",
+ "score": 0.9217,
+ "bbox": [
+ 275,
+ 330,
+ 524,
+ 370
+ ],
+ "index": 4
+ }
+]
+```
+
+#### 中间处理结果 (middle.json)
+
+**文件命名格式**：`{原文件名}_middle.json`
+
+##### 顶层结构
+
+| 字段名 | 类型 | 说明 |
+|--------|------|------|
+| `pdf_info` | `list[dict]` | 每一页的解析结果数组 |
+| `_backend` | `string` | 解析模式：`pipeline`、`vlm` 或 `office` |
+| `_version_name` | `string` | MinerU 版本号 |
+
+##### 页面信息结构 (pdf_info)
+
+| 字段名 | 说明 |
+|--------|------|
+| `preproc_blocks` | PDF 预处理后的未分段中间结果 |
+| `page_idx` | 页码，从 0 开始 |
+| `page_size` | 页面的宽度和高度 `[width, height]` |
+| `images` | 图片块信息列表 |
+| `tables` | 表格块信息列表 |
+| `interline_equations` | 行间公式块信息列表 |
+| `discarded_blocks` | 需要丢弃的块信息 |
+| `para_blocks` | 分段后的内容块结果 |
+
+##### 块结构层次
+
+```
+一级块 (table | image | chart)
+└── 二级块
+ └── 行 (line)
+ └── 片段 (span)
+```
+
+##### 一级块字段
+
+| 字段名 | 说明 |
+|--------|------|
+| `type` | 块类型：`table`、`image` 或 `chart` |
+| `bbox` | 块的矩形框坐标 `[x0, y0, x1, y1]` |
+| `blocks` | 包含的二级块列表 |
+
+##### 二级块字段
+
+| 字段名 | 说明 |
+|--------|------|
+| `type` | 块类型（详见下表） |
+| `bbox` | 块的矩形框坐标 |
+| `lines` | 包含的行信息列表 |
+
+##### 二级块类型
+
+| 类型 | 说明 |
+|------|------|
+| `image_body` | 图像本体 |
+| `image_caption` | 图像描述文本 |
+| `image_footnote` | 图像脚注 |
+| `table_body` | 表格本体 |
+| `table_caption` | 表格描述文本 |
+| `table_footnote` | 表格脚注 |
+| `chart_body` | 图表本体 |
+| `chart_caption` | 图表描述文本 |
+| `chart_footnote` | 图表脚注 |
+| `text` | 文本块 |
+| `title` | 标题块 |
+| `index` | 目录块 |
+| `list` | 列表块 |
+| `interline_equation` | 行间公式块 |
+
+##### 行和片段结构
+
+**行 (line) 字段**：
+- `bbox`：行的矩形框坐标
+- `spans`：包含的片段列表
+
+**片段 (span) 字段**：
+- `bbox`：片段的矩形框坐标
+- `type`：片段类型（`image`、`table`、`chart`、`text`、`inline_equation`、`interline_equation`）
+- `content` | `image_path`：文本内容或图片路径
+
+##### 示例数据
+
+```json
+{
+ "pdf_info": [
+ {
+ "preproc_blocks": [
+ {
+ "type": "text",
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 82.99800872802734
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 72.0000228881836
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 54.0,
+ 61.956024169921875,
+ 296.2261657714844,
+ 72.0000228881836
+ ],
+ "content": "dependent on the service headway and the reliability of the departure ",
+ "type": "text",
+ "score": 1.0
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "layout_bboxes": [
+ {
+ "layout_bbox": [
+ 52,
+ 61,
+ 294,
+ 731
+ ],
+ "layout_label": "V",
+ "sub_layout": []
+ }
+ ],
+ "page_idx": 0,
+ "page_size": [
+ 612.0,
+ 792.0
+ ],
+ "_layout_tree": [],
+ "images": [],
+ "tables": [],
+ "interline_equations": [],
+ "discarded_blocks": [],
+ "para_blocks": [
+ {
+ "type": "text",
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 82.99800872802734
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 52,
+ 61.956024169921875,
+ 294,
+ 72.0000228881836
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 54.0,
+ 61.956024169921875,
+ 296.2261657714844,
+ 72.0000228881836
+ ],
+ "content": "dependent on the service headway and the reliability of the departure ",
+ "type": "text",
+ "score": 1.0
+ }
+ ]
+ }
+ ]
+ }
+ ]
+ }
+ ],
+ "_backend": "pipeline",
+ "_version_name": "0.6.1"
+}
+```
+
+#### 内容列表 (content_list.json)
+
+**文件命名格式**：`{原文件名}_content_list.json`
+
+##### 功能说明
+
+这是一个简化版的 `middle.json`，按阅读顺序平铺存储所有可读内容块，去除了复杂的布局信息，便于后续处理。
+
+##### 内容类型
+
+| 类型 | 说明 |
+|------|------|
+| `image` | 图片 |
+| `table` | 表格 |
+| `chart` | 图表 |
+| `text` | 文本/标题 |
+| `equation` | 行间公式 |
+| `seal` | 印章 |
+| `code` | 代码块 / 算法块 |
+| `list` | 列表 / 参考文献列表 |
+| `header` / `footer` / `page_number` / `aside_text` / `page_footnote` | 页面辅助块 |
+
+##### 文本层级标识
+
+通过 `text_level` 字段区分文本层级：
+
+- 无 `text_level` 或 `text_level: 0`：正文文本
+- `text_level: 1`：一级标题
+- `text_level: 2`：二级标题
+- 以此类推...
+
+##### 通用字段
+
+- 所有内容块都包含 `page_idx` 字段，表示所在页码（从 0 开始）。
+- 所有内容块都包含 `bbox` 字段，表示内容块的边界框坐标 `[x0, y0, x1, y1]` 映射在0-1000范围内的结果。
+- `code` 类型会通过 `sub_type` 区分 `code` 和 `algorithm`，并可包含 `code_body`、`code_caption`、`code_footnote` 等字段。
+- `list` 类型可通过 `sub_type` 区分普通列表和参考文献列表。
+- `image` / `chart` 类型可包含可选 `sub_type` 字段，用于透传视觉子类型。
+
+##### 示例数据
+
+```json
+[
+ {
+ "type": "text",
+ "text": "The response of flow duration curves to afforestation ",
+ "text_level": 1,
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 0
+ },
+ {
+ "type": "image",
+ "img_path": "images/a8ecda1c69b27e4f79fce1589175a9d721cbdc1cf78b4cc06a015f3746f6b9d8.jpg",
+ "image_caption": [
+ "Fig. 1. Annual flow duration curves of daily flows from Pine Creek, Australia, 1989–2000. "
+ ],
+ "image_footnote": [],
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 1
+ },
+ {
+ "type": "equation",
+ "img_path": "images/181ea56ef185060d04bf4e274685f3e072e922e7b839f093d482c29bf89b71e8.jpg",
+ "text": "$$\nQ _ { \\% } = f ( P ) + g ( T )\n$$",
+ "text_format": "latex",
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 2
+ },
+ {
+ "type": "table",
+ "img_path": "images/e3cb413394a475e555807ffdad913435940ec637873d673ee1b039e3bc3496d0.jpg",
+ "table_caption": [
+ "Table 2 Significance of the rainfall and time terms "
+ ],
+ "table_footnote": [
+ "indicates that the rainfall term was significant at the $5 \\%$ level, $T$ indicates that the time term was significant at the $5 \\%$ level, \\* represents significance at the $10 \\%$ level, and na denotes too few data points for meaningful analysis. "
+ ],
+ "table_body": "
",
+ "bbox": [
+ 62,
+ 480,
+ 946,
+ 904
+ ],
+ "page_idx": 5
+ }
+]
+```
+
+### 通用内容列表 V2 (content_list_v2.json)(开发中，格式可能调整)
+
+**文件命名格式**：`{原文件名}_content_list_v2.json`
+
+##### 功能说明
+
+`content_list_v2.json` 是 3.0 起新增的结构化输出文件，所有后端都会在保留 `content_list.json` 的同时额外输出该文件：
+
+- 顶层是按页分组的列表，便于按页消费结果
+- 每个内容块使用统一的 `type + content` 结构，适合程序化处理
+- 不同后端和输入类型支持的 `type` 会有所不同
+
+##### 通用字段
+
+| 字段名 | 类型 | 说明 |
+|--------|------|------|
+| `type` | `string` | 内容类型 |
+| `content` | `dict` | 与 `type` 对应的结构化内容 |
+| `bbox` | `list[int]` | 可选，0-1000 范围的边界框 |
+| `anchor` | `string` | 可选，部分 `DOCX` 标题或索引项会携带锚点 |
+
+其中 `image` / `chart` 类型还可能包含可选顶层字段 `sub_type`，用于表示视觉子类型。
+
+##### 常见类型
+
+| 类型 | 说明 |
+|------|------|
+| `title` | 标题块，包含 `title_content` 与 `level` |
+| `paragraph` | 段落块，包含 `paragraph_content` |
+| `equation_interline` | 行间公式，包含 `math_content`、`math_type` |
+| `image` / `table` / `chart` / `seal` | 视觉类块，包含图片路径、说明文字等结构化字段 |
+| `code` | 代码块，包含 `code_content`、`code_caption`、`code_footnote`、`code_language` |
+| `algorithm` | 算法块，包含 `algorithm_content`、`algorithm_caption`、`algorithm_footnote` |
+| `list` / `index` | 列表与索引，包含 `list_items` |
+| `page_header` / `page_footer` / `page_number` / `page_aside_text` / `page_footnote` | 页面辅助块 |
+
+##### 示例数据
+
+```json
+[
+ [
+ {
+ "type": "title",
+ "content": {
+ "title_content": [
+ {
+ "type": "text",
+ "content": "1 Introduction"
+ }
+ ],
+ "level": 1
+ },
+ "bbox": [
+ 83,
+ 121,
+ 917,
+ 156
+ ]
+ },
+ {
+ "type": "page_footnote",
+ "content": {
+ "page_footnote_content": [
+ {
+ "type": "text",
+ "content": "* Corresponding author"
+ }
+ ]
+ },
+ "bbox": [
+ 71,
+ 815,
+ 915,
+ 841
+ ]
+ }
+ ]
+]
+```
+
+### VLM 后端 输出结果
+
+#### 模型推理结果 (model.json)
+
+**文件命名格式**：`{原文件名}_model.json`
+
+##### 文件格式说明
+
+- 该文件为 VLM 模型的原始输出结果，包含两层嵌套list，外层表示页面，内层表示该页的内容块
+- 每个内容块都是一个dict，包含 `type`、`bbox`、`angle`、`content` 字段
+
+
+##### 支持的内容类型
+
+```json
+{
+ "text": "文本",
+ "title": "标题",
+ "equation": "行间公式",
+ "image": "图片",
+ "image_caption": "图片描述",
+ "image_footnote": "图片脚注",
+ "table": "表格",
+ "table_caption": "表格描述",
+ "table_footnote": "表格脚注",
+ "phonetic": "拼音",
+ "code": "代码块",
+ "code_caption": "代码描述",
+ "ref_text": "参考文献",
+ "algorithm": "算法块",
+ "list": "列表",
+ "header": "页眉",
+ "footer": "页脚",
+ "page_number": "页码",
+ "aside_text": "装订线旁注",
+ "page_footnote": "页面脚注"
+}
+```
+
+##### 坐标系统说明
+
+`bbox` 坐标格式：`[x0, y0, x1, y1]`
+
+- 分别表示左上、右下两点的坐标
+- 坐标原点在页面左上角
+- 坐标为相对于原始页面尺寸的百分比，范围在0-1之间
+
+##### 示例数据
+
+```json
+[
+ [
+ {
+ "type": "header",
+ "bbox": [
+ 0.077,
+ 0.095,
+ 0.18,
+ 0.181
+ ],
+ "angle": 0,
+ "score": null,
+ "block_tags": null,
+ "content": "ELSEVIER",
+ "format": null,
+ "content_tags": null
+ },
+ {
+ "type": "title",
+ "bbox": [
+ 0.157,
+ 0.228,
+ 0.833,
+ 0.253
+ ],
+ "angle": 0,
+ "score": null,
+ "block_tags": null,
+ "content": "The response of flow duration curves to afforestation",
+ "format": null,
+ "content_tags": null
+ }
+ ]
+]
+```
+
+#### 中间处理结果 (middle.json)
+
+**文件命名格式**：`{原文件名}_middle.json`
+
+##### 文件格式说明
+vlm 后端的 middle.json 文件结构与 pipeline 后端类似，但存在以下差异：
+
+- list变成二级block，增加`sub_type`字段区分list类型:
+ * `text`（文本类型）
+ * `ref_text`（引用类型）
+
+- 增加code类型block，code类型包含两种"sub_type":
+ * 分别是`code`和`algorithm`
+ * 至少有`code_body`, 可选`code_caption`
+
+- `discarded_blocks`内元素type增加以下类型:
+ * `header`（页眉）
+ * `footer`（页脚）
+ * `page_number`（页码）
+ * `aside_text`（装订线文本）
+ * `page_footnote`（脚注）
+- 所有block增加`angle`字段，用来表示旋转角度，0，90，180，270
+
+
+##### 示例数据
+- list block 示例
+ ```json
+ {
+ "bbox": [
+ 174,
+ 155,
+ 818,
+ 333
+ ],
+ "type": "list",
+ "angle": 0,
+ "index": 11,
+ "blocks": [
+ {
+ "bbox": [
+ 174,
+ 157,
+ 311,
+ 175
+ ],
+ "type": "text",
+ "angle": 0,
+ "lines": [
+ {
+ "bbox": [
+ 174,
+ 157,
+ 311,
+ 175
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 174,
+ 157,
+ 311,
+ 175
+ ],
+ "type": "text",
+ "content": "H.1 Introduction"
+ }
+ ]
+ }
+ ],
+ "index": 3
+ },
+ {
+ "bbox": [
+ 175,
+ 182,
+ 464,
+ 229
+ ],
+ "type": "text",
+ "angle": 0,
+ "lines": [
+ {
+ "bbox": [
+ 175,
+ 182,
+ 464,
+ 229
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 175,
+ 182,
+ 464,
+ 229
+ ],
+ "type": "text",
+ "content": "H.2 Example: Divide by Zero without Exception Handling"
+ }
+ ]
+ }
+ ],
+ "index": 4
+ }
+ ],
+ "sub_type": "text"
+ }
+ ```
+- code block 示例
+ ```json
+ {
+ "type": "code",
+ "bbox": [
+ 114,
+ 780,
+ 885,
+ 1231
+ ],
+ "blocks": [
+ {
+ "bbox": [
+ 114,
+ 780,
+ 885,
+ 1231
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 114,
+ 780,
+ 885,
+ 1231
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 114,
+ 780,
+ 885,
+ 1231
+ ],
+ "type": "text",
+ "content": "1 // Fig. H.1: DivideByZeroNoExceptionHandling.java \n2 // Integer division without exception handling. \n3 import java.util.Scanner; \n4 \n5 public class DivideByZeroNoExceptionHandling \n6 { \n7 // demonstrates throwing an exception when a divide-by-zero occurs \n8 public static int quotient( int numerator, int denominator ) \n9 { \n10 return numerator / denominator; // possible division by zero \n11 } // end method quotient \n12 \n13 public static void main(String[] args) \n14 { \n15 Scanner scanner = new Scanner(System.in); // scanner for input \n16 \n17 System.out.print(\"Please enter an integer numerator: \"); \n18 int numerator = scanner.nextInt(); \n19 System.out.print(\"Please enter an integer denominator: \"); \n20 int denominator = scanner.nextInt(); \n21"
+ }
+ ]
+ }
+ ],
+ "index": 17,
+ "angle": 0,
+ "type": "code_body"
+ },
+ {
+ "bbox": [
+ 867,
+ 160,
+ 1280,
+ 189
+ ],
+ "lines": [
+ {
+ "bbox": [
+ 867,
+ 160,
+ 1280,
+ 189
+ ],
+ "spans": [
+ {
+ "bbox": [
+ 867,
+ 160,
+ 1280,
+ 189
+ ],
+ "type": "text",
+ "content": "Algorithm 1 Modules for MCTSteg"
+ }
+ ]
+ }
+ ],
+ "index": 19,
+ "angle": 0,
+ "type": "code_caption"
+ }
+ ],
+ "index": 17,
+ "sub_type": "code"
+ }
+ ```
+
+#### 内容列表 (content_list.json)
+
+**文件命名格式**：`{原文件名}_content_list.json`
+
+##### 文件格式说明
+vlm 后端的 content_list.json 文件结构与 pipeline 后端类似，伴随本次middle.json的变化，做了以下调整：
+
+- 新增`code`类型，code类型包含两种"sub_type":
+ * 分别是`code`和`algorithm`
+ * 至少有`code_body`, 可选`code_caption`
+
+- 新增`list`类型，list类型包含两种"sub_type":
+ * `text`
+ * `ref_text`
+
+- `image` / `chart` 类型可能带有可选 `sub_type` 字段，用于透传视觉子类型
+- `chart` 类型除 `img_path` 外，还可包含 `content`、`chart_caption`、`chart_footnote`，其中 `content` 保持原始 Markdown 表格文本
+
+- 增加所有所有`discarded_blocks`的输出内容
+ * `header`
+ * `footer`
+ * `page_number`
+ * `aside_text`
+ * `page_footnote`
+- 3.0 起，vlm 后端也会同时输出 `*_content_list_v2.json`，其通用结构见上文“通用内容列表 V2”。
+
+##### 示例数据
+- code 类型 content
+ ```json
+ {
+ "type": "code",
+ "sub_type": "algorithm",
+ "code_caption": [
+ "Algorithm 1 Modules for MCTSteg"
+ ],
+ "code_body": "1: function GETCOORDINATE(d) \n2: $x \\gets d / l$ , $y \\gets d$ mod $l$ \n3: return $(x, y)$ \n4: end function \n5: function BESTCHILD(v) \n6: $C \\gets$ child set of $v$ \n7: $v' \\gets \\arg \\max_{c \\in C} \\mathrm{UCTScore}(c)$ \n8: $v'.n \\gets v'.n + 1$ \n9: return $v'$ \n10: end function \n11: function BACK PROPAGATE(v) \n12: Calculate $R$ using Equation 11 \n13: while $v$ is not a root node do \n14: $v.r \\gets v.r + R$ , $v \\gets v.p$ \n15: end while \n16: end function \n17: function RANDOMSEARCH(v) \n18: while $v$ is not a leaf node do \n19: Randomly select an untried action $a \\in A(v)$ \n20: Create a new node $v'$ \n21: $(x, y) \\gets \\mathrm{GETCOORDINATE}(v'.d)$ \n22: $v'.p \\gets v$ , $v'.d \\gets v.d + 1$ , $v'.\\Gamma \\gets v.\\Gamma$ \n23: $v'.\\gamma_{x,y} \\gets a$ \n24: if $a = -1$ then \n25: $v.lc \\gets v'$ \n26: else if $a = 0$ then \n27: $v.mc \\gets v'$ \n28: else \n29: $v.rc \\gets v'$ \n30: end if \n31: $v \\gets v'$ \n32: end while \n33: return $v$ \n34: end function \n35: function SEARCH(v) \n36: while $v$ is fully expanded do \n37: $v \\gets$ BESTCHILD(v) \n38: end while \n39: if $v$ is not a leaf node then \n40: $v \\gets$ RANDOMSEARCH(v) \n41: end if \n42: return $v$ \n43: end function",
+ "bbox": [
+ 510,
+ 87,
+ 881,
+ 740
+ ],
+ "page_idx": 0
+ }
+ ```
+- list 类型 content
+ ```json
+ {
+ "type": "list",
+ "sub_type": "text",
+ "list_items": [
+ "H.1 Introduction",
+ "H.2 Example: Divide by Zero without Exception Handling",
+ "H.3 Example: Divide by Zero with Exception Handling",
+ "H.4 Summary"
+ ],
+ "bbox": [
+ 174,
+ 155,
+ 818,
+ 333
+ ],
+ "page_idx": 0
+ }
+ ```
+- discarded 类型 content
+ ```json
+ [{
+ "type": "header",
+ "text": "Journal of Hydrology 310 (2005) 253-265",
+ "bbox": [
+ 363,
+ 164,
+ 623,
+ 177
+ ],
+ "page_idx": 0
+ },
+ {
+ "type": "page_footnote",
+ "text": "* Corresponding author. Address: Forest Science Centre, Department of Sustainability and Environment, P.O. Box 137, Heidelberg, Vic. 3084, Australia. Tel.: +61 3 9450 8719; fax: +61 3 9450 8644.",
+ "bbox": [
+ 71,
+ 815,
+ 915,
+ 841
+ ],
+ "page_idx": 0
+ }]
+ ```
+
+
+## 总结
+
+以上文件为 MinerU 的完整输出结果，用户可根据需要选择合适的文件进行后续处理：
+
+- **模型输出**(使用原始输出):
+ * model.json
+
+- **调试和验证**(使用可视化文件):
+ * layout.pdf
+ * span.pdf
+
+- **内容提取**(使用简化文件):
+ * *.md
+ * content_list.json
+ * content_list_v2.json
+
+- **二次开发**(使用结构化文件):
+ * middle.json
diff --git a/docs/zh/usage/acceleration_cards/AMD.md b/docs/zh/usage/acceleration_cards/AMD.md
new file mode 100644
index 000000000..1e90e9398
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/AMD.md
@@ -0,0 +1,365 @@
+## 基于Triton的ROCm 不同后端实现优化，基本实现vllm后端正常推理，以及pipeline后端中第一步layout用的DocLayout-YOLO
+
+**已有完整python vllm和mineru环境直接跳转第五步！！！**
+**其他GPU执行问题可以参考，先prof查看定位找到哪个算子问题，然后triton后端实现即可**
+测试了一下，基本和MinerU官网效果差不多，用AMD的人也不是很多，就在评论区分享给大家了
+
+### 1.结果介绍
+**补充一个200页的PDF python编程书测试一下速度，可以到1.99it/s：**
+Two Step Extraction: 100%|████████████████████████████████████████| 200/200 [01:40<00:00, 1.99it/s]
+
+**下面为之前14学术论文测试结果：**
+7900xtx mineru-gradio --server-name 0.0.0.0 --server-port 7860 --enable-vllm-engine true 速度大概为**1.6-1.8s/it**，没有仔细测试，简单试了两个文档。第二种矩阵乘法代替原来的dots点乘可以进一步提速到1.3s/it，优化后的主要算子耗时在hipblast(这个没法提升了)和vllm triton后端，各占25%耗时吧，vllm tirion后端这个这个只能等官方优化了。。。。
+doclayout-yolo的layout速度从原来的1.6it/s提高到15it/s，注意需要缓存一下输入的pdf尺寸后，triton必须要缓存尺寸没办法。主要是为了保留模型输入输出接口，最小代码改动。
+采用-b vlm-vllm-engine模式举个例子
+
+---
+**测试结果为优化为5d矩阵乘代替原来的点积结果：**
+2025-10-05 15:45:12.985 | INFO | mineru.backend.vlm.vlm_analyze:get_model:128 - get vllm-engine predictor cost: 18.45s
+Adding requests: 100%|████████████████████████████████████████████████████████████████████████████████| 14/14 [00:01<00:00, 12.20it/s]
+Processed prompts: 100%|█████████████████████| 14/14 [00:08<00:00, 1.56it/s, est. speed input: 2174.18 toks/s, output: 791.87 toks/s]
+Adding requests: 100%|█████████████████████████████████████████████████████████████████████████████| 278/278 [00:00<00:00, 323.03it/s]
+Processed prompts: 100%|██████████████████| 278/278 [00:07<00:00, 37.63it/s, est. speed input: 5264.66 toks/s, output: 2733.31 toks/s]
+
+mineru-gradio --server-name 0.0.0.0 --server-port 7860 --enable-vllm-engine true测试：
+2025-10-05 15:46:55.953 | WARNING | mineru.cli.common:convert_pdf_bytes_to_bytes_by_pypdfium2:54 - end_page_id is out of range, use pdf_docs length
+Two Step Extraction: 100%|████████████████████████████████████████████████████████████████████████████| 14/14 [00:18<00:00, 1.30s/it]
+
+---
+
+### 2.原因介绍
+AMD RDNA使用vllm后端有严重的性能问题，原因是因为vllm的**qwen2_vl.py**中有一个算子在rocm kernel上没有对应的实现，导致性能出现严重的卷积计算回退，一次执行花了12s，。。。。。。。。一言难尽。即**MIOpen 库中缺少模型中特定 Conv3d(bfloat16) 的优化内核**。
+DocLayout-YOLO的**g2l_crm.py**空洞卷积也是这个问题，专业的CDNA MI210也没解决这个问题
+正好一起处理了。
+
+---
+
+### 3.环境介绍
+System: Ubuntu 24.04.3 Kernel: Linux 6.14.0-33-generic ROCm version: 7.0.1
+python环境：
+python 3.12
+pytorch-triton-rocm 3.5.0+gitbbb06c03
+torch 2.10.0.dev20251001+rocm7.0
+torchvision 0.25.0.dev20251003+rocm7.0
+vllm 0.11.0rc2.dev198+g736fbf4c8.rocm701
+不同版本无所谓，处理方法是一样的。
+
+---
+
+### 4.前置环境安装
+```
+uv venv --python python3.12
+source .venv/bin/activate
+uv pip install --pre torch torchvision -i https://pypi.tuna.tsinghua.edu.cn/simple/ --extra-index-url https://download.pytorch.org/whl/nightly/rocm7.0
+uv pip install pip
+# 避免覆盖我们本地的pytorch，改用pip而没有继续使用uv pip
+pip install -U "mineru[core]" -i https://pypi.mirrors.ustc.edu.cn/simple/
+```
+vllm 安装参考官方手册[Vllm](https://docs.vllm.com.cn/en/latest/getting_started/installation/gpu.html#amd-rocm)
+```
+#手动安装aiter，vllm，amd-smi等，自行找一个位置clone，然后进入该目录吧
+git clone --recursive https://github.com/ROCm/aiter.git
+cd aiter
+git submodule sync; git submodule update --init --recursive
+python setup.py develop
+cd ..
+git clone https://github.com/vllm-project/vllm.git
+cd vllm/
+cp -r /opt/rocm/share/amd_smi ~/Pytorch/vllm/
+pip install amd_smi/
+pip install --upgrade numba \
+ scipy \
+ huggingface-hub[cli,hf_transfer] \
+ setuptools_scm
+pip install -r requirements/rocm.txt
+export PYTORCH_ROCM_ARCH="gfx1100" #根据自己的GPU架构 rocminfo | grep gfx
+python setup.py develop
+```
+---
+
+### 5.vllm中关键triton算子添加
+#### 这里我给出两种解决方法，第一种解决方法就是前面提到的优化到1.5到1.8s/it，第二种方法有手动优化算子到矩阵乘法，7900xtx肯定适用，大概1.3s/it，其他AMD GPU相对方案一也有提速，但是不一定是最佳速度实现，里面的手动部分可能需要微调。
+**注意pip把triton 后端的flash_attn卸载了，搞了半天各种尝试还是报错，问题比较大，直接不用就行了**
+```
+#定位自己vllm位置XXX
+pip show vllm
+```
+**关键更改**
+XXX/vllm/model_executor/models/qwen2_vl.py文件：
+**1.qwen2_vl.py文件33行下增加from .qwen2_vl_vision_kernels import triton_conv3d_patchify**
+```
+from collections.abc import Iterable, Mapping, Sequence
+from functools import partial
+from typing import Annotated, Any, Callable, Literal, Optional, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from .qwen2_vl_vision_kernels import triton_conv3d_patchify
+```
+**接下来分为方案一(2.1和3.1)和方案二(2.2和3.2)，选取一种实现即可**
+
+---
+**方案1**
+**2.1qwen2_vl.py文件498行class Qwen2VisionPatchEmbed(nn.Module),PS.就是这玩意AMD没有现成的内核算子导致回退**
+```
+class Qwen2VisionPatchEmbed(nn.Module):
+
+ def __init__(
+ self,
+ patch_size: int = 14,
+ temporal_patch_size: int = 2,
+ in_channels: int = 3,
+ embed_dim: int = 1152,
+ ) -> None:
+ super().__init__()
+ self.patch_size = patch_size
+ self.temporal_patch_size = temporal_patch_size
+ self.embed_dim = embed_dim
+
+ kernel_size = (temporal_patch_size, patch_size, patch_size)
+ self.proj = nn.Conv3d(in_channels,
+ embed_dim,
+ kernel_size=kernel_size,
+ stride=kernel_size,
+ bias=False)
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ L, C = x.shape
+ x_reshaped = x.view(L, -1, self.temporal_patch_size, self.patch_size,
+ self.patch_size)
+
+ # Call your custom Triton kernel instead of self.proj
+ x_out = triton_conv3d_patchify(x_reshaped, self.proj.weight)
+
+ # The output of our kernel is already the correct shape [L, embed_dim]
+ return x_out
+```
+**3.1XXX/vllm/model_executor/models/目录下创建qwen2_vl_vision_kernels.py文件，用triton实现**
+```
+import torch
+from vllm.triton_utils import tl, triton
+
+@triton.jit
+def _conv3d_patchify_kernel(
+ # Pointers to tensors
+ X, W, Y,
+ # Tensor dimensions
+ N, C_in, D_in, H_in, W_in,
+ C_out, KD, KH, KW,
+ # Stride and padding for memory access
+ stride_xn, stride_xc, stride_xd, stride_xh, stride_xw,
+ stride_wn, stride_wc, stride_wd, stride_wh, stride_ww,
+ stride_yn, stride_yc,
+ # Triton-specific metaparameters
+ BLOCK_SIZE: tl.constexpr,
+):
+ """
+ Triton kernel for a non-overlapping 3D patching convolution.
+ Each kernel instance computes one output value for one patch.
+ """
+ # Get the program IDs for the N (patch) and C_out (output channel) dimensions
+ pid_n = tl.program_id(0) # The index of the patch we are processing
+ pid_cout = tl.program_id(1) # The index of the output channel we are computing
+
+ # --- Calculate memory pointers ---
+ # Pointer to the start of the current input patch
+ x_ptr = X + (pid_n * stride_xn)
+ # Pointer to the start of the current filter (weight)
+ w_ptr = W + (pid_cout * stride_wn)
+ # Pointer to where the output will be stored
+ y_ptr = Y + (pid_n * stride_yn + pid_cout * stride_yc)
+
+ # --- Perform the convolution (element-wise product and sum) ---
+ # This is a dot product between the flattened patch and the flattened filter.
+ accumulator = tl.zeros((BLOCK_SIZE,), dtype=tl.float32)
+
+ # Iterate over the elements of the patch/filter
+ for c_offset in range(0, C_in):
+ for d_offset in range(0, KD):
+ for h_offset in range(0, KH):
+ # Unrolled loop for the innermost dimension (width) for performance
+ for w_offset in range(0, KW, BLOCK_SIZE):
+ # Create masks to handle cases where KW is not a multiple of BLOCK_SIZE
+ w_range = w_offset + tl.arange(0, BLOCK_SIZE)
+ w_mask = w_range < KW
+
+ # Calculate offsets to load data
+ patch_offset = (c_offset * stride_xc + d_offset * stride_xd +
+ h_offset * stride_xh + w_range * stride_xw)
+ filter_offset = (c_offset * stride_wc + d_offset * stride_wd +
+ h_offset * stride_wh + w_range * stride_ww)
+
+ # Load patch and filter data, applying masks
+ patch_vals = tl.load(x_ptr + patch_offset, mask=w_mask, other=0.0)
+ filter_vals = tl.load(w_ptr + filter_offset, mask=w_mask, other=0.0)
+
+ # Multiply and accumulate
+ accumulator += patch_vals.to(tl.float32) * filter_vals.to(tl.float32)
+
+ # Sum the accumulator block and store the single output value
+ output_val = tl.sum(accumulator, axis=0)
+ tl.store(y_ptr, output_val)
+
+
+def triton_conv3d_patchify(x: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
+ """
+ Python wrapper for the 3D patching convolution Triton kernel.
+ """
+ # Get tensor dimensions
+ N, C_in, D_in, H_in, W_in = x.shape
+ C_out, _, KD, KH, KW = weight.shape
+
+ # Create the output tensor
+ # The output of this specific conv is (N, C_out, 1, 1, 1), which we squeeze
+ Y = torch.empty((N, C_out), dtype=x.dtype, device=x.device)
+
+ # Define the grid for launching the Triton kernel
+ # Each kernel instance handles one patch (N) for one output channel (C_out)
+ grid = (N, C_out)
+
+ # Launch the kernel
+ # We pass all strides to make the kernel flexible
+ _conv3d_patchify_kernel[grid](
+ x, weight, Y,
+ N, C_in, D_in, H_in, W_in,
+ C_out, KD, KH, KW,
+ x.stride(0), x.stride(1), x.stride(2), x.stride(3), x.stride(4),
+ weight.stride(0), weight.stride(1), weight.stride(2), weight.stride(3), weight.stride(4),
+ Y.stride(0), Y.stride(1),
+ BLOCK_SIZE=16, # A reasonable default, can be tuned
+ )
+
+ return Y
+```
+---
+**方案2**
+**2.2qwen2_vl.py文件498行class Qwen2VisionPatchEmbed(nn.Module)函数,PS.就是这玩意AMD没有现成的内核算子导致回退，这里我们直接5D张量一步到位，改为矩阵乘法**
+```
+class Qwen2VisionPatchEmbed(nn.Module):
+
+ def __init__(
+ self,
+ patch_size: int = 14,
+ temporal_patch_size: int = 2,
+ in_channels: int = 3,
+ embed_dim: int = 1152,
+ ) -> None:
+ super().__init__()
+ self.patch_size = patch_size
+ self.temporal_patch_size = temporal_patch_size
+ self.embed_dim = embed_dim
+
+ kernel_size = (temporal_patch_size, patch_size, patch_size)
+
+ self.proj = nn.Conv3d(in_channels,
+ embed_dim,
+ kernel_size=kernel_size,
+ stride=kernel_size,
+ bias=False)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ L, C = x.shape
+ x_reshaped_5d = x.view(L, -1, self.temporal_patch_size, self.patch_size,
+ self.patch_size)
+
+ return triton_conv3d_patchify(x_reshaped_5d, self.proj.weight)
+```
+**3.2XXX/vllm/model_executor/models/目录下创建qwen2_vl_vision_kernels.py文件，用triton实现**
+```
+import torch
+from vllm.triton_utils import tl, triton
+
+@triton.jit
+def _conv_gemm_kernel(
+ A, B, C, M, N, K,
+ stride_am, stride_ak,
+ stride_bk, stride_bn,
+ stride_cm, stride_cn,
+ BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr,
+):
+ pid_m = tl.program_id(0)
+ pid_n = tl.program_id(1)
+ offs_m = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
+ offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
+ offs_k = tl.arange(0, BLOCK_K)
+ a_ptrs = A + (offs_m[:, None] * stride_am + offs_k[None, :] * stride_ak)
+ b_ptrs = B + (offs_k[:, None] * stride_bk + offs_n[None, :] * stride_bn)
+ accumulator = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
+ for k in range(0, K, BLOCK_K):
+ a = tl.load(a_ptrs, mask=(offs_m[:, None] < M) & (offs_k[None, :] < K), other=0.0)
+ b = tl.load(b_ptrs, mask=(offs_k[:, None] < K) & (offs_n[None, :] < N), other=0.0)
+ accumulator += tl.dot(a, b)
+ a_ptrs += BLOCK_K * stride_ak
+ b_ptrs += BLOCK_K * stride_bk
+ offs_k += BLOCK_K
+ c = accumulator.to(C.dtype.element_ty)
+ offs_cm = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
+ offs_cn = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
+ c_ptrs = C + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]
+ c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+ tl.store(c_ptrs, c, mask=c_mask)
+
+def triton_conv3d_patchify(x_5d: torch.Tensor, weight_5d: torch.Tensor) -> torch.Tensor:
+ N_patches, _, _, _, _ = x_5d.shape
+ C_out, _, _, _, _ = weight_5d.shape
+ A = x_5d.view(N_patches, -1)
+ B = weight_5d.view(C_out, -1).transpose(0, 1).contiguous()
+ M, K = A.shape
+ _K, N = B.shape
+ assert K == _K
+ C = torch.empty((M, N), device=A.device, dtype=A.dtype)
+
+ # --- 针对7900xtx的手动调优配置，其他GPU的最优组合可能需要自行寻找，AMD的autotune效果就是没有效果 ---
+ best_config = {
+ 'BLOCK_M': 128,
+ 'BLOCK_N': 128,
+ 'BLOCK_K': 32,
+ }
+ num_stages = 4
+ num_warps = 8
+
+ grid = (triton.cdiv(M, best_config['BLOCK_M']),
+ triton.cdiv(N, best_config['BLOCK_N']))
+
+ _conv_gemm_kernel[grid](
+ A, B, C,
+ M, N, K,
+ A.stride(0), A.stride(1),
+ B.stride(0), B.stride(1),
+ C.stride(0), C.stride(1),
+ **best_config,
+ num_stages=num_stages,
+ num_warps=num_warps
+ )
+
+ return C
+```
+---
+**4.关闭终端后再次使用mineru-gradio会报一个Lora错误，修改代码跳过它**
+```
+pip show mineru_vl_utils
+```
+
+打开该文件XXX/mineru_vl_utils/vlm_client/vllm_async_engine_client.py修改第58行self.tokenizer = vllm_async_llm.tokenizer.get_lora_tokenizer()为：
+```
+ try:
+ self.tokenizer = vllm_async_llm.tokenizer.get_lora_tokenizer()
+ except AttributeError:
+ # 如果没有 get_lora_tokenizer 方法，直接使用原始 tokenizer
+ self.tokenizer = vllm_async_llm.tokenizer
+```
+
+**最后整两个环境变量后愉快玩耍即可**
+```
+export MINERU_MODEL_SOURCE=modelscope
+export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
+```
+---
+
+### 6.vllm后端已经没有问题，下面是pipeline 中layout用的doclayout-yolo模型空洞卷积问题
+### 我在 [DocLayout-YOLO](https://github.com/opendatalab/DocLayout-YOLO/issues/120#issuecomment-3368144275) 下做了一个回答，因此 pipeline 的空洞卷积问题不在这里赘述，直接点击链接查看即可。
+查看自己doclayout-yolo安装位置如下，然后进入修改链接中回复介绍的文件即可
+```
+pip show doclayout-yolo
+```
+
diff --git a/docs/zh/usage/acceleration_cards/Ascend.md b/docs/zh/usage/acceleration_cards/Ascend.md
new file mode 100644
index 000000000..1618ce8ba
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Ascend.md
@@ -0,0 +1,174 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: CTyunOS 22.06
+cpu: Kunpeng-920 (aarch64)
+npu: Ascend 910B2
+driver: 23.0.3
+docker: 20.10.12
+```
+
+## 2. 环境准备
+
+>[!NOTE]
+>Ascend加速卡支持使用`vllm`或`lmdeploy`进行VLM模型推理加速。请根据实际需求选择安装和使用其中之一:
+
+### 2.1 使用 Dockerfile 构建镜像 （vllm）
+> [!TIP]
+> ascend-vllm支持设备如下:
+>
+> - Atlas A2 training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
+> - Atlas 800I A2 inference series (Atlas 800I A2)
+> - Atlas A3 training series (Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD)
+> - Atlas 800I A3 inference series (Atlas 800I A3)
+> - [Experimental] Atlas 300I inference series (Atlas 300I Duo)
+>
+> Dockerfile文件第三行为ascend-vllm基础镜像信息,默认tag为A2适配的版本,例如 `v0.11.0`
+>
+> - 如需使用A3适配的版本,请将第三行的tag修改为 `v0.11.0-a3`,然后再执行build操作。
+> - 如需使用Atlas 300I Duo适配的版本,请将第三行的tag修改为 `v0.10.0rc1-310p`,然后再执行build操作。
+
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/npu.Dockerfile
+docker build --network=host -t mineru:npu-vllm-latest -f npu.Dockerfile .
+```
+
+### 2.2 使用 Dockerfile 构建镜像 （lmdeploy）
+
+> [!TIP]
+> ascend-lmdeploy支持设备如下:
+>
+> - Atlas A2 training series (Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2)
+> - Atlas 800I A2 inference series (Atlas 800I A2)
+>
+> 如果您的设备为Atlas A3系列或Atlas 300I Duo系列，请使用vllm版本的镜像。
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/npu.Dockerfile
+# 将基础镜像从 vllm 切换为 lmdeploy
+sed -i '3s/^/# /' npu.Dockerfile && sed -i '5s/^# //' npu.Dockerfile
+docker build --network=host -t mineru:npu-lmdeploy-latest -f npu.Dockerfile .
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -u root --name mineru_docker --privileged=true \
+ --ipc=host \
+ --network=host \
+ --device=/dev/davinci0 \
+ --device=/dev/davinci1 \
+ --device=/dev/davinci2 \
+ --device=/dev/davinci3 \
+ --device=/dev/davinci4 \
+ --device=/dev/davinci5 \
+ --device=/dev/davinci6 \
+ --device=/dev/davinci7 \
+ --device=/dev/davinci_manager \
+ --device=/dev/devmm_svm \
+ --device=/dev/hisi_hdc \
+ -v /var/log/npu/:/usr/slog \
+ -v /usr/local/dcmi:/usr/local/dcmi \
+ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
+ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
+ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_LMDEPLOY_DEVICE=ascend \
+ -it mineru:npu-vllm-latest \
+ /bin/bash
+```
+
+>[!TIP]
+> 请根据实际情况选择使用`vllm`或`lmdeploy`版本的镜像，如需使用lmdeploy，替换上述命令中的`mineru:npu-vllm-latest`为`mineru:npu-lmdeploy-latest`即可。
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+>[!NOTE]
+> 由于310p加速卡不支持图模式与bf16精度，因此在使用该加速卡时，执行任意与`vllm`相关命令需追加`--enforce-eager --dtype float16`参数。
+> 例如:
+> ```bash
+> mineru-openai-server --port 30000 --enforce-eager --dtype float16
+> ```
+
+## 4. 注意事项
+
+不同环境下，MinerU对Ascend加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - NPU加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[ASCEND_RT_VISIBLE_DEVICES](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/maintenref/envvar/envref_07_0028.html)
+> - 在Ascend平台可以通过`npu-smi info`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Biren.md b/docs/zh/usage/acceleration_cards/Biren.md
new file mode 100644
index 000000000..acf9a2cc3
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Biren.md
@@ -0,0 +1,108 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.4 LTS
+cpu: Intel x86-64
+gpu: Biren 106C
+driver: 1.10.0
+docker: 28.0.4
+```
+
+## 2. 环境准备
+
+### 2.1 下载并加载镜像 （vllm）
+
+```bash
+wget http://birentech.com/xxx/MinerU/mineru-vllm.tar 链接获取请联系壁仞内部人员（邮箱：MonaLiu@birentech.com）
+docker load -i mineru-vllm.tar
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -it --name mineru_docker \
+ --privileged \
+ --network=host \
+ --shm-size=100G \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_DEVICE_MODEL=supa \
+ -e SHAPE_TRANSFORM_GRANK=true \
+ mineru:biren-vllm-latest \
+ /bin/bash
+```
+
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对Biren加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - Biren加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
+>将环境变量`CUDA_VISIBLE_DEVICES`替换为`SUPA_VISIBLE_DEVICES`即可。
+> - 在壁仞平台可以通过`brsmi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Cambricon.md b/docs/zh/usage/acceleration_cards/Cambricon.md
new file mode 100644
index 000000000..fc8a9af42
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Cambricon.md
@@ -0,0 +1,155 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.5 LTS
+cpu: Hygon Hygon C86 7490
+mlu: MLU590-M9D
+driver: v6.2.11
+docker: 28.3.0
+```
+
+## 2. 环境准备
+
+>[!NOTE]
+>Cambricon加速卡支持使用`lmdeploy`或`vllm`进行VLM模型推理加速。请根据实际需求选择安装和使用其中之一:
+
+### 2.1 使用 Dockerfile 构建镜像 （lmdeploy）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/mlu.Dockerfile
+docker build --network=host -t mineru:mlu-lmdeploy-latest -f mlu.Dockerfile .
+```
+
+### 2.2 使用 Dockerfile 构建镜像 （vllm）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/mlu.Dockerfile
+# 将基础镜像从 lmdeploy 切换为 vllm
+sed -i -e '3,4s/^/# /' -e '6,7s/^# //' mlu.Dockerfile
+docker build --network=host -t mineru:mlu-vllm-latest -f mlu.Dockerfile .
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run --name mineru_docker \
+ --privileged \
+ --ipc=host \
+ --network=host \
+ --shm-size=400g \
+ --ulimit memlock=-1 \
+ -v /dev:/dev \
+ -v /lib/modules:/lib/modules:ro \
+ -v /usr/bin/cnmon:/usr/bin/cnmon \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_LMDEPLOY_DEVICE=camb \
+ -it mineru:mlu-lmdeploy-latest \
+ /bin/bash
+```
+
+>[!TIP]
+> 请根据实际情况选择使用`vllm`或`lmdeploy`版本的镜像，如需使用`vllm`,请执行以下操作：
+>
+> - 替换上述命令中的`mineru:mlu-lmdeploy-latest`为`mineru:mlu-vllm-latest`
+>
+> - 进入容器后，通过以下命令切换venv环境：
+> ```bash
+> source /torch/venv3/pytorch_infer/bin/activate
+> ```
+>
+> - 切换成功后，您可以在命令行前看到`(pytorch_infer)`的标识，这表示您已成功进入`vllm`的虚拟环境。
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+>[!NOTE]
+> **兼容性说明**：由于寒武纪（Cambricon）目前对 vLLM v1 引擎的支持尚待完善，MinerU 现阶段采用 v0 引擎作为适配方案。
+> 受此限制，vLLM 的异步引擎（Async Engine）功能存在兼容性问题，可能导致部分使用场景无法正常运行。
+> 我们将持续跟进寒武纪对 vLLM v1 引擎的支持进展，并及时在 MinerU 中进行相应的适配与优化。
+
+不同环境下，MinerU对Cambricon加速卡的支持情况如下表所示：
+
+>[!TIP]
+> - `lmdeploy`黄灯问题为不能输入文件夹使用批量解析功能，输入单个文件时表现正常。
+> - `vllm`黄灯问题为在精度未对齐，在部分场景下可能出现预期外结果。
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - Cambricon加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
+>将环境变量`CUDA_VISIBLE_DEVICES`替换为`MLU_VISIBLE_DEVICES`即可。
+> - 在Cambricon平台可以通过`cnmon`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Enflame.md b/docs/zh/usage/acceleration_cards/Enflame.md
new file mode 100644
index 000000000..19f662726
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Enflame.md
@@ -0,0 +1,106 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.4 LTS
+cpu: Intel x86-64
+gcu: Enflame S60
+driver: 1.7.0.9
+docker: 28.0.1
+```
+
+## 2. 环境准备
+
+### 2.1 使用 Dockerfile 构建镜像
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/gcu.Dockerfile
+docker build --network=host -t mineru:gcu-vllm-latest -f gcu.Dockerfile .
+```
+
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -u root --name mineru_docker \
+ --network=host \
+ --ipc=host \
+ --privileged \
+ -e MINERU_MODEL_SOURCE=local \
+ -it mineru:gcu-vllm-latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对Enflame加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - GCU加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
+>将环境变量`CUDA_VISIBLE_DEVICES`替换为`TOPS_VISIBLE_DEVICES`即可。
+> - 在Enflame平台可以通过`efsmi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Hygon.md b/docs/zh/usage/acceleration_cards/Hygon.md
new file mode 100644
index 000000000..faaada2a3
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Hygon.md
@@ -0,0 +1,112 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.3 LTS
+cpu: Hygon C86-4G(x86-64)
+dcu: BW200
+driver: 6.3.13-V1.12.0a
+docker: 20.10.24
+```
+
+## 2. 环境准备
+
+### 2.1 使用 Dockerfile 构建镜像
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/dcu.Dockerfile
+docker build --network=host -t mineru:dcu-vllm-latest -f dcu.Dockerfile .
+```
+
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -u root --name mineru_docker \
+ --network=host \
+ --ipc=host \
+ --shm-size=16G \
+ --device=/dev/kfd \
+ --device=/dev/mkfd \
+ --device=/dev/dri \
+ -v /opt/hyhal:/opt/hyhal \
+ --group-add video \
+ --cap-add=SYS_PTRACE \
+ --security-opt seccomp=unconfined \
+ -e MINERU_MODEL_SOURCE=local \
+ -it mineru:dcu-vllm-latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对Hygon加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - DCU加速卡指定可用加速卡的方式与AMD GPU类似，请参考[GPU isolation techniques](https://rocm.docs.amd.com/en/docs-6.2.4/conceptual/gpu-isolation.html)
+> - 在Hygon平台可以通过`hy-smi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/IluvatarCorex.md b/docs/zh/usage/acceleration_cards/IluvatarCorex.md
new file mode 100644
index 000000000..f1ef7c716
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/IluvatarCorex.md
@@ -0,0 +1,119 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.5 LTS
+cpu: Intel x86-64
+gpu: Iluvatar BI-V150
+driver: 4.4.0
+docker: 28.1.1
+```
+
+## 2. 环境准备
+
+### 2.1 使用 Dockerfile 构建镜像
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/corex.Dockerfile
+docker build --network=host -t mineru:corex-vllm-latest -f corex.Dockerfile .
+```
+
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run --name mineru_docker \
+ -v /usr/src:/usr/src \
+ -v /lib/modules:/lib/modules \
+ -v /dev:/dev \
+ --privileged \
+ --cap-add=ALL \
+ --pid=host \
+ --group-add video \
+ --network=host \
+ --shm-size '400gb' \
+ --ulimit memlock=-1 \
+ --security-opt seccomp=unconfined \
+ --security-opt apparmor=unconfined \
+ -e VLLM_ENFORCE_CUDA_GRAPH=1 \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_VLLM_DEVICE=corex \
+ -it mineru:corex-vllm-latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+>[!TIP]
+>目前Iluvatar方案使用vllm作为推理引擎时，可能出现服务停止后显存无法正常释放的问题，如果遇到该问题，请重启Docker容器以释放显存。
+
+不同环境下，MinerU对Iluvatar加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - Iluvatar加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明
+> - 在Iluvatar平台可以通过`ixsmi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Kunlunxin.md b/docs/zh/usage/acceleration_cards/Kunlunxin.md
new file mode 100644
index 000000000..bb54cbdd8
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Kunlunxin.md
@@ -0,0 +1,120 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.5 LTS
+cpu: Intel x86-64
+xpu: P800
+driver: 515.58
+docker: 20.10.5
+```
+
+## 2. 环境准备
+
+### 2.1 使用 Dockerfile 构建镜像 （vllm）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/kxpu.Dockerfile
+docker build --network=host -t mineru:kxpu-vllm-latest -f kxpu.Dockerfile .
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -u root --name mineru_docker \
+ --device=/dev/xpu0:/dev/xpu0 \
+ --device=/dev/xpu1:/dev/xpu1 \
+ --device=/dev/xpu2:/dev/xpu2 \
+ --device=/dev/xpu3:/dev/xpu3 \
+ --device=/dev/xpu4:/dev/xpu4 \
+ --device=/dev/xpu5:/dev/xpu5 \
+ --device=/dev/xpu6:/dev/xpu6 \
+ --device=/dev/xpu7:/dev/xpu7 \
+ --device=/dev/xpuctrl:/dev/xpuctrl \
+ --net=host \
+ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
+ --tmpfs /dev/shm:rw,nosuid,nodev,exec,size=32g \
+ --cap-add=SYS_PTRACE \
+ -v /home/users/vllm-kunlun:/home/vllm-kunlun \
+ -v /usr/local/bin/xpu-smi:/usr/local/bin/xpu-smi \
+ -w /workspace \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_FORMULA_CH_SUPPORT=true \
+ -e MINERU_VLLM_DEVICE=kxpu \
+ -it mineru:kxpu-vllm-latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对Kunlunxin加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - Kunlunxin加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
+>将环境变量`CUDA_VISIBLE_DEVICES`替换为`XPU_VISIBLE_DEVICES`即可。
+> - 在Kunlunxin平台可以通过`xpu-smi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/METAX.md b/docs/zh/usage/acceleration_cards/METAX.md
new file mode 100644
index 000000000..cc8f5d196
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/METAX.md
@@ -0,0 +1,147 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04
+cpu: INTEL x86_64
+gpu: C500
+driver: 2.12.13
+docker: 28.1.1
+```
+
+## 2. 环境准备
+
+>[!NOTE]
+>maca加速卡支持使用`vllm`或`lmdeploy`进行VLM模型推理加速。请根据实际需求选择安装和使用其中之一:
+
+### 2.1 使用metax官方镜像作为基础镜像构建vllm环境镜像
+
+1. 从metax官方仓库拉取基础镜像
+ - 1.1 镜像获取地址：[https://developer.metax-tech.com/softnova/docker](https://developer.metax-tech.com/softnova/docker)
+ - 1.2 在镜像网站选择`AI`分类，软件包类型选择`vllm`，操作系统选择`ubuntu`
+ - 1.3 找到`vllm:maca.ai3.1.0.7-torch2.6-py310-ubuntu22.04-amd64`镜像，复制拉取命令并在本地终端执行
+2. 使用 Dockerfile 构建镜像 （vllm）
+ ```bash
+ wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/maca.Dockerfile
+ docker build --network=host -t mineru:maca-vllm-latest -f maca.Dockerfile .
+ ```
+
+
+### 2.2 使用 Dockerfile 构建镜像 （lmdeploy）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/maca.Dockerfile
+# 将基础镜像从 vllm 切换为 lmdeploy
+sed -i '3s/^/# /' maca.Dockerfile && sed -i '5s/^# //' maca.Dockerfile
+docker build --network=host -t mineru:maca-lmdeploy-latest -f maca.Dockerfile .
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run --ipc host \
+ --cap-add SYS_PTRACE \
+ --privileged=true \
+ --device=/dev/mem \
+ --device=/dev/dri \
+ --device=/dev/mxcd \
+ --device=/dev/infiniband \
+ --group-add video \
+ --network=host \
+ --shm-size '100gb' \
+ --ulimit memlock=-1 \
+ --security-opt seccomp=unconfined \
+ --security-opt apparmor=unconfined \
+ --name mineru_docker \
+ -v /datapool:/datapool \
+ -e MINERU_MODEL_SOURCE=local \
+ -e MINERU_LMDEPLOY_DEVICE=maca \
+ -it mineru:maca-vllm-latest \
+ /bin/bash
+```
+
+>[!TIP]
+> 请根据实际情况选择使用`vllm`或`lmdeploy`版本的镜像，如需使用lmdeploy，替换上述命令中的`mineru:maca-vllm-latest`为`mineru:maca-lmdeploy-latest`即可。
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+## 4. 注意事项
+
+不同环境下，MinerU对maca加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - MACA加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明。
+> - 在METAX平台可以通过`mx-smi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/MooreThreads.md b/docs/zh/usage/acceleration_cards/MooreThreads.md
new file mode 100644
index 000000000..1015a6377
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/MooreThreads.md
@@ -0,0 +1,113 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.4 LTS
+cpu: Intel x86-64
+dcu: MTT S4000
+driver: 3.0.0-rc-KuaE2.0
+docker: 24.0.7
+```
+
+## 2. 环境准备
+
+### 2.1 使用 Dockerfile 构建镜像
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/musa.Dockerfile
+docker build --network=host -t mineru:musa-vllm-latest -f musa.Dockerfile .
+```
+
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -u root --name mineru_docker \
+ --network=host \
+ --ipc=host \
+ --shm-size=80g \
+ --privileged \
+ -e MTHREADS_VISIBLE_DEVICES=all \
+ -e MINERU_VLLM_DEVICE=musa \
+ -e MINERU_MODEL_SOURCE=local \
+ -it mineru:musa-vllm-latest \
+ /bin/bash
+```
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对MooreThreads加速卡的支持情况如下表所示：
+
+>[!NOTE]
+> **兼容性说明**：由于摩尔线程（MooreThreads）目前对 vLLM v1 引擎的支持尚待完善，MinerU 现阶段采用 v0 引擎作为适配方案。
+> 受此限制，vLLM 的异步引擎（Async Engine）功能存在兼容性问题，可能导致部分使用场景无法正常运行。
+> 我们将持续跟进摩尔线程对 vLLM v1 引擎的支持进展，并及时在 MinerU 中进行相应的适配与优化。
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - MooreThreads加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[GPU 枚举](https://docs.mthreads.com/cloud-native/cloud-native-doc-online/install_guide/#gpu-%E6%9E%9A%E4%B8%BE)
+> - 在MooreThreads平台可以通过`mthreads-gmi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/THead.md b/docs/zh/usage/acceleration_cards/THead.md
new file mode 100644
index 000000000..e46bb79a7
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/THead.md
@@ -0,0 +1,138 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04
+cpu: INTEL x86_64
+ppu: ZW810E
+driver: 1.4.0
+docker: 26.1.4
+```
+
+## 2. 环境准备
+
+>[!NOTE]
+>ppu加速卡支持使用`vllm`或`lmdeploy`进行VLM模型推理加速。请根据实际需求选择安装和使用其中之一:
+
+### 2.1 使用 Dockerfile 构建镜像 （vllm）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/ppu.Dockerfile
+docker build --network=host -t mineru:ppu-vllm-latest -f ppu.Dockerfile .
+```
+
+### 2.2 使用 Dockerfile 构建镜像 （lmdeploy）
+
+```bash
+wget https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/docker/china/ppu.Dockerfile
+# 将基础镜像从 vllm 切换为 lmdeploy
+sed -i '3s/^/# /' ppu.Dockerfile && sed -i '5s/^# //' ppu.Dockerfile
+docker build --network=host -t mineru:ppu-lmdeploy-latest -f ppu.Dockerfile .
+```
+
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run --privileged=true \
+ --name mineru_docker \
+ --device=/dev/alixpu \
+ --device=/dev/alixpu_ctl \
+ --ipc=host \
+ --network=host \
+ --ulimit memlock=-1 \
+ --ulimit stack=67108864 \
+ --shm-size=500g \
+ -v /mnt:/mnt \
+ -v /datapool:/datapool \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ -e MINERU_MODEL_SOURCE=local \
+ -it mineru:ppu-vllm-latest \
+ /bin/bash
+```
+
+>[!TIP]
+> 请根据实际情况选择使用`vllm`或`lmdeploy`版本的镜像，如需使用lmdeploy，替换上述命令中的`mineru:ppu-vllm-latest`为`mineru:ppu-lmdeploy-latest`即可。
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+## 4. 注意事项
+
+不同环境下，MinerU对ppu加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - PPU加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明。
+> - 在T-Head平台可以通过`ppu-smi`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/Tecorigin.md b/docs/zh/usage/acceleration_cards/Tecorigin.md
new file mode 100644
index 000000000..07180dceb
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/Tecorigin.md
@@ -0,0 +1,116 @@
+## 1. 测试平台
+以下为本指南测试使用的平台信息，供参考：
+```
+os: Ubuntu 22.04.5 LTS
+cpu: AMD EPYC (amd64)
+gpu: T100
+driver: 3.0.0
+docker: 28.0.4
+```
+
+## 2. 环境准备
+
+### 2.1 下载并加载镜像 （vllm）
+
+```bash
+wget http://wb.tecorigin.com:8082/repository/teco-customer-repo/Course/MinerU/mineru-vllm.tar
+
+docker load -i mineru-vllm.tar
+```
+
+## 3. 启动 Docker 容器
+
+```bash
+docker run -dit --name mineru_docker \
+ --privileged \
+ --cap-add SYS_PTRACE \
+ --cap-add SYS_ADMIN \
+ --network=host \
+ --shm-size=500G \
+ mineru:sdaa-vllm-latest \
+ /bin/bash
+```
+
+>[!TIP]
+> 如需使用`vllm`环境,请执行以下操作：
+> - 进入容器后，通过以下命令切换到conda环境：
+> ```bash
+> conda activate vllm_env_py310
+> ```
+>
+> - 切换成功后，您可以在命令行前看到`(vllm_env_py310)`的标识，这表示您已成功进入`vllm`的虚拟环境。
+
+执行该命令后，您将进入到Docker容器的交互式终端，您可以直接在容器内运行MinerU相关命令来使用MinerU的功能。
+您也可以直接通过替换`/bin/bash`为服务启动命令来启动MinerU服务，详细说明请参考[通过命令启动服务](https://opendatalab.github.io/MinerU/zh/usage/quick_usage/#apiwebuihttp-clientserver)。
+
+
+## 4. 注意事项
+
+不同环境下，MinerU对Tecorigin加速卡的支持情况如下表所示：
+
+
+
+
+
+注：
+🟢: 支持，运行较稳定，精度与Nvidia GPU基本一致
+🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+🔴: 不支持，无法运行，或精度存在较大差异
+
+>[!TIP]
+> - Tecorigin加速卡指定可用加速卡的方式与NVIDIA GPU类似，请参考[使用指定GPU设备](https://opendatalab.github.io/MinerU/zh/usage/advanced_cli_parameters/#cuda_visible_devices)章节说明,
+>将环境变量`CUDA_VISIBLE_DEVICES`替换为`SDAA_VISIBLE_DEVICES`即可。
+> - 在太初平台可以通过`teco-smi -c`命令查看加速卡的使用情况，并根据需要指定空闲的加速卡ID以避免资源冲突。
\ No newline at end of file
diff --git a/docs/zh/usage/acceleration_cards/VastAI.md b/docs/zh/usage/acceleration_cards/VastAI.md
new file mode 100644
index 000000000..5698763d9
--- /dev/null
+++ b/docs/zh/usage/acceleration_cards/VastAI.md
@@ -0,0 +1,215 @@
+## 1. 瀚博半导体
+
+
+
+- 官方网址：https://www.vastaitech.com
+- 模型中心：https://github.com/Vastai/VastModelZOO
+
+
+## 2. 测试平台
+
+- 以下为本指南测试使用的平台信息，供参考
+ ```
+ os: Ubuntu-22.04.3-LTS-x86_64
+ cpu: Hygon C86-4G
+ gpu: VA16 / VA1L / VA10L
+ torch: 2.8.0+cpu
+ torch-vacc: 1.3.3.777
+ vllm: 0.11.1.dev0+gb8b302cde.d20251030.cpu
+ vllm-vacc: 0.11.0.777
+ driver: 00.25.12.30 d3_3_v2_9_a3_1 a76bf37 20251230
+ docker: 28.1.1
+ ```
+
+## 3. 环境准备
+
+- 获取vllm_vacc基础镜像
+ ```bash
+ sudo docker pull harbor.vastaitech.com/ai_deliver/vllm_vacc:VVI-25.12.SP2
+ ```
+
+- 启动容器
+ ```bash
+ sudo docker run -it \
+ --privileged=true \
+ --shm-size=256g \
+ --name vllm_service \
+ --ipc=host \
+ --network=host \
+ harbor.vastaitech.com/ai_deliver/vllm_vacc:VVI-25.12.SP2 bash
+ ```
+
+- 安装MinerU
+
+ - 参考官方文档安装：[README_zh-CN.md#安装-mineru](https://github.com/opendatalab/MinerU/blob/master/README_zh-CN.md#安装-mineru)
+
+ ```bash
+ # 启动容器
+ # sudo docker exec -it vllm_service bash
+
+ # 可选pypi源
+ # https://mirrors.163.com/pypi/simple/
+ # https://mirrors.aliyun.com/pypi/simple/
+ # https://pypi.mirrors.ustc.edu.cn/simple/
+ # https://pypi.tuna.tsinghua.edu.cn/simple/
+ # https://mirror.baidu.com/pypi/simple
+
+ # 通过源码安装MinerU
+ git clone https://github.com/opendatalab/MinerU.git
+ git checkout 8c4b3ef3a20b11ddac9903f25124d24ea82639b5
+ pip install -e .[core] -i https://mirrors.aliyun.com/pypi/simple
+
+ # 或使用pip安装MinerU
+ pip install -U "mineru[core]==2.7.0" -i https://mirrors.aliyun.com/pypi/simple
+ ```
+
+> [!NOTE]
+> - `vllm_vacc`基础镜像内已包含`torch/vllm`等相关依赖
+> - 截至`2025/12/31`，`VastAI`已支持`MinerU`至最新版本`2.7.0`，`master分支8c4b3ef3`
+> - 和`NVIDIA`硬件下`CUDA_VISIBLE_DEVICES`类似；在`VastAI`硬件中可以使用`VACC_VISIBLE_DEVICES`指定`可见计算卡ID`，如`-e VACC_VISIBLE_DEVICES=0,1,2,3`
+> - 需指定适当的`--shm-size`虚拟内存
+
+## 4. MinerU功能
+
+> [!NOTE]
+> - `VastAI`加速卡仅支持使用`vlm-auto-engine`和`vlm-http-client`形式进行`VLM`模型推理加速
+
+- 进入容器
+ ```bash
+ sudo docker exec -it vllm_service bash
+ ```
+
+- 使用MinerU
+
+ - 模型准备，参考官方介绍：[model_source.md](https://github.com/opendatalab/MinerU/blob/master/docs/zh/usage/model_source.md)
+
+ - 方式一：`vlm-auto-engine`
+
+ ```bash
+ export MINERU_MODEL_SOURCE=modelscope
+
+ # step1, 以`vlm-auto-engine`方式启动MinerU解析任务
+ mineru -p image.png \
+ -o ./output \
+ -b vlm-auto-engine \
+ --http-timeout 1200 \
+ --tensor-parallel-size 2 \
+ --enforce_eager \
+ --trust-remote-code \
+ --max-model-len 16384
+ ```
+
+ - 方式二：`vlm-http-client`
+
+ ```bash
+ # step1, 启动vLLM API server
+ vllm serve /root/.cache/modelscope/hub/models/OpenDataLab/MinerU2.5-2509-1.2B \
+ --tensor-parallel-size 2 \
+ --trust-remote-code \
+ --enforce_eager \
+ --port 8090 \
+ --max-model-len 16384 \
+ --served-model-name MinerU2.5-2509-1.2B
+
+ # step2，以`vlm-http-client`方式启动MinerU解析任务
+ mineru -p demo/pdfs/demo1.pdf \
+ -o ./output \
+ -b vlm-http-client \
+ -u http://127.0.0.1:8090 \
+ --http-timeout 1200
+ ```
+
+
+> [!NOTE]
+> - 注意在执行任意与`vllm`相关命令需追加`--enforce_eager`参数
+
+
+## 5. 注意事项
+
+`VastAI`加速卡对`MinerU`的支持情况如下表所示：
+
+
+
+
+
+
+
+> [!NOTE]
+> - 🟢: 支持，运行较稳定，精度与NVIDIA GPU基本一致
+> - 🟡: 支持但较不稳定，在某些场景下可能出现异常，或精度存在一定差异
+> - 🔴: 不支持，无法运行，或精度存在较大差异
+> - `vlm-auto-engine`：VastAI仅支持vLLM后端
\ No newline at end of file
diff --git a/docs/zh/usage/advanced_cli_parameters.md b/docs/zh/usage/advanced_cli_parameters.md
new file mode 100644
index 000000000..c0d6d9783
--- /dev/null
+++ b/docs/zh/usage/advanced_cli_parameters.md
@@ -0,0 +1,57 @@
+# 命令行参数进阶
+
+## 推理引擎参数透传
+
+### 参数传递说明
+> [!TIP]
+> - 所有vllm/lmdeploy官方支持的参数都可用通过命令行参数传递给 MinerU，包括以下命令:`mineru`、`mineru-openai-server`、`mineru-gradio`、`mineru-api`、`mineru-router`
+> - 命令行参数同时支持 `--foo value` 与 `--foo=value` 两种写法
+> - 如果您想了解更多有关`vllm`的参数使用方法，请参考 [vllm官方文档](https://docs.vllm.ai/en/latest/cli/serve.html)
+> - 如果您想了解更多有关`lmdeploy`的参数使用方法，请参考 [lmdeploy官方文档](https://lmdeploy.readthedocs.io/en/latest/llm/api_server.html)
+
+## GPU 设备选择与配置
+
+### CUDA_VISIBLE_DEVICES 基本用法
+> [!TIP]
+> - 任何情况下，您都可以通过在命令行的开头添加`CUDA_VISIBLE_DEVICES` 环境变量来指定可见的 GPU 设备：
+> ```bash
+> CUDA_VISIBLE_DEVICES=1 mineru -p -o
+> ```
+> - 这种指定方式对所有的命令行调用都有效，包括 `mineru`、`mineru-openai-server`、`mineru-gradio`、`mineru-api`和`mineru-router`，且对`pipeline`、`vlm`后端均适用。
+
+### 常见设备配置示例
+> [!TIP]
+> 以下是一些常见的 `CUDA_VISIBLE_DEVICES` 设置示例：
+> ```bash
+> CUDA_VISIBLE_DEVICES=1 # Only device 1 will be seen
+> CUDA_VISIBLE_DEVICES=0,1 # Devices 0 and 1 will be visible
+> CUDA_VISIBLE_DEVICES="0,1" # Same as above, quotation marks are optional
+> CUDA_VISIBLE_DEVICES=0,2,3 # Devices 0, 2, 3 will be visible; device 1 is masked
+> CUDA_VISIBLE_DEVICES="" # No GPU will be visible
+> ```
+
+## 实际应用场景
+
+> [!TIP]
+> 以下是一些可能的使用场景：
+>
+> - 如果您有多张显卡，需要在卡0和卡1上启动两个`openai-server`服务，并分别监听不同的端口，可以使用以下命令：
+> ```bash
+> # 在终端1中
+> CUDA_VISIBLE_DEVICES=0 mineru-openai-server --engine vllm --port 30000
+> # 在终端2中
+> CUDA_VISIBLE_DEVICES=1 mineru-openai-server --engine vllm --port 30001
+> ```
+>
+> - 如果您有多张显卡，需要在卡0和卡1上启动两个`fastapi`服务，并分别监听不同的端口，可以使用以下命令：
+> ```bash
+> # 在终端1中
+> CUDA_VISIBLE_DEVICES=0 mineru-api --host 127.0.0.1 --port 8000
+> # 在终端2中
+> CUDA_VISIBLE_DEVICES=1 mineru-api --host 127.0.0.1 --port 8001
+> ```
+>
+> - 如果您有多张显卡，需要通过`router`在其中4张卡上启动`fastapi`服务并统一管理，可以使用以下命令：
+> ```bash
+> CUDA_VISIBLE_DEVICES=0,1,2,3 mineru-router --host 127.0.0.1 --port 8002
+> ```
diff --git a/docs/zh/usage/cli_tools.md b/docs/zh/usage/cli_tools.md
new file mode 100644
index 000000000..566e3769e
--- /dev/null
+++ b/docs/zh/usage/cli_tools.md
@@ -0,0 +1,178 @@
+# 命令行工具使用说明
+
+## 查看帮助信息
+要查看 MinerU 命令行工具的帮助信息，可以使用 `--help` 参数。以下是各个命令行工具的帮助信息示例：
+```bash
+mineru --help
+Usage: mineru [OPTIONS]
+
+Options:
+ -v, --version 显示版本并退出
+ -p, --path PATH 输入文件路径或目录（必填）
+ -o, --output PATH 输出目录（必填）
+ --api-url TEXT MinerU FastAPI 服务地址；不传时自动拉起本地临时 mineru-api
+ -m, --method [auto|txt|ocr] 解析方法：auto（默认）、txt、ocr（仅用于 pipeline 与 hybrid* 后端）
+ -b, --backend [pipeline|hybrid-auto-engine|hybrid-http-client|vlm-auto-engine|vlm-http-client]
+ 解析后端（默认为 hybrid-auto-engine）
+ -l, --lang [ch|ch_server|ch_lite|en|korean|japan|chinese_cht|ta|te|ka|th|el|latin|arabic|east_slavic|cyrillic|devanagari]
+ 指定文档语言（可提升 OCR 准确率，仅用于 pipeline 与 hybrid* 后端）
+ -u, --url TEXT 当使用 http-client 时，传给服务端后端的 OpenAI 兼容地址
+ -s, --start INTEGER 开始解析的页码（从 0 开始）
+ -e, --end INTEGER 结束解析的页码（从 0 开始）
+ -f, --formula BOOLEAN 是否启用公式解析（默认开启）
+ -t, --table BOOLEAN 是否启用表格解析（默认开启）
+ --help 显示帮助信息
+```
+> [!TIP]
+> `mineru` 当前支持本地 `PDF`、图片与 `DOCX`、`PPTX`、`XLSX` 文件或目录输入。
+
+```bash
+mineru-api --help
+Usage: mineru-api [OPTIONS]
+
+Options:
+ --host TEXT 服务器主机地址（默认：127.0.0.1）
+ --port INTEGER 服务器端口（默认：8000）
+ --reload 启用自动重载（开发模式）
+ --enable-vlm-preload BOOLEAN
+ 在 mineru-api 启动阶段预加载本地 VLM 模型
+ --help 显示此帮助信息并退出
+```
+```bash
+mineru-gradio --help
+Usage: mineru-gradio [OPTIONS]
+
+Options:
+ --enable-example BOOLEAN 启用示例文件输入(需要将示例文件放置在当前
+ 执行命令目录下的 `example` 文件夹中)
+ --enable-http-client BOOLEAN 在后端选项中启用 HTTP 客户端选项
+ --enable-api BOOLEAN 启用 Gradio API 以提供应用程序服务
+ --max-convert-pages INTEGER 设置从 PDF 转换为 Markdown 的最大页数
+ --server-name TEXT 设置 Gradio 应用程序的服务器主机名
+ --server-port INTEGER 设置 Gradio 应用程序的服务器端口
+ --api-url TEXT MinerU FastAPI 服务地址；不传时自动拉起可复用的本地
+ mineru-api
+ --enable-vlm-preload BOOLEAN 在 Gradio 拉起本地 mineru-api 时预加载本地
+ VLM 模型
+ --latex-delimiters-type [a|b|all]
+ 设置在 Markdown 渲染中使用的 LaTeX 分隔符类型
+ ('a' 表示 '$' 类型，'b' 表示 '()[]' 类型，
+ 'all' 表示两种类型都使用)
+ --help 显示此帮助信息并退出
+```
+```bash
+mineru-router --help
+Usage: mineru-router [OPTIONS]
+
+Options:
+ --host TEXT 路由服务主机地址（默认：127.0.0.1）
+ --port INTEGER 路由服务端口（默认：8002）
+ --reload 启用自动重载（开发模式）
+ --upstream-url TEXT 现有 MinerU FastAPI 服务地址；可重复传入多个
+ --local-gpus TEXT 本地 GPU worker 配置：auto、none 或 0,1,2 形式
+ --worker-host TEXT 路由托管 worker 的监听地址（默认：127.0.0.1）
+ --enable-vlm-preload BOOLEAN
+ 在 router 托管的本地 mineru-api worker 中预加载本地
+ VLM 模型
+ --help 显示此帮助信息并退出
+```
+
+## 环境变量说明
+
+> [!NOTE]
+> 从当前版本开始，`mineru` 是基于 `mineru-api` 的编排客户端：
+>
+>- 未传 `--api-url` 时，CLI 会自动拉起本地临时 `mineru-api`
+>- 传入 `--api-url` 时，CLI 会直连该 FastAPI 服务
+>- `--url` 不再表示 MinerU API 地址，而是服务端 `vlm/hybrid-http-client` 所需的 OpenAI 兼容地址
+
+MinerU命令行工具的某些参数存在相同功能的环境变量配置，通常环境变量配置的优先级高于命令行参数，且在所有命令行工具中都生效。
+以下是常用的环境变量及其说明：
+
+- `MINERU_TOOLS_CONFIG_JSON`：
+ * 用于指定配置文件路径
+ * 默认为用户目录下的`mineru.json`，可通过环境变量指定其他配置文件路径。
+
+- `MINERU_FORMULA_ENABLE`：
+ * 用于启用公式解析
+ * 默认为`true`，可通过环境变量设置为`false`来禁用公式解析。
+
+- `MINERU_FORMULA_CH_SUPPORT`：
+ * 用于启用中文公式解析优化（实验性功能）
+ * 默认为`false`，可通过环境变量设置为`true`来启用中文公式解析优化。
+ * 仅对`pipeline`后端生效。
+
+- `MINERU_TABLE_ENABLE`：
+ * 用于启用表格解析
+ * 默认为`true`，可通过环境变量设置为`false`来禁用表格解析。
+
+- `MINERU_TABLE_MERGE_ENABLE`：
+ * 用于启用表格合并功能
+ * 默认为`true`，可通过环境变量设置为`false`来禁用表格合并功能。
+
+- `MINERU_PDF_RENDER_TIMEOUT`：
+ * 用于设置将PDF渲染为图片的超时时间（秒）
+ * 默认为`300`秒，可通过环境变量设置为其他值以调整渲染图片的超时时间。
+ * 仅在linux和macOS系统中生效。
+
+- `MINERU_PDF_RENDER_THREADS`：
+ * 用于设置将PDF渲染为图片时使用的线程数
+ * 默认为`4`，可通过环境变量设置为其他值以调整渲染图片时的线程数。
+ * 仅在linux和macOS系统中生效。
+
+- `MINERU_PROCESSING_WINDOW_SIZE`：
+ * 用于设置单次处理窗口大小，影响大文档处理时的内存占用和吞吐表现
+ * 默认为`64`，可通过环境变量设置为其他正整数。
+
+- `MINERU_API_MAX_CONCURRENT_REQUESTS`：
+ * 用于设置 `mineru-api` 或 `mineru-router` 管理的 worker 最大并发请求数
+ * 默认为`3`，需设置为正整数。
+
+- `MINERU_API_ENABLE_FASTAPI_DOCS`：
+ * 用于控制是否启用 FastAPI 自动生成的 `/docs`、`/openapi.json`、`/redoc`
+ * 默认为`true`。
+
+- `MINERU_API_OUTPUT_ROOT`：
+ * 用于指定 `mineru-api` 输出目录根路径
+ * 默认为当前工作目录下的 `./output`。
+
+- `MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS`：
+ * 用于控制各命令行工具等待本地拉起的 `mineru-api` 进入健康状态的最长时间
+ * 默认为 `300` 秒。
+ * 适用于 `mineru` 的临时本地 API、`mineru-gradio` 的 preload 启动，以及 `mineru-router` 托管的本地 worker。
+
+- `MINERU_API_TASK_RETENTION_SECONDS`：
+ * 用于设置任务完成或失败后的保留时长（秒）
+ * 默认为 `86400` 秒（24 小时）。
+
+- `MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS`：
+ * 用于设置任务清理轮询间隔（秒）
+ * 默认为 `300` 秒（5 分钟）。
+
+- `MINERU_INTRA_OP_NUM_THREADS`：
+ * 用于设置onnx模型的intra_op线程数，影响单个算子的计算速度
+ * 默认为`-1`（自动选择），可通过环境变量设置为其他值以调整线程数。
+
+- `MINERU_INTER_OP_NUM_THREADS`：
+ * 用于设置onnx模型的inter_op线程数，影响多个算子的并行执行
+ * 默认为`-1`（自动选择），可通过环境变量设置为其他值以调整线程数。
+
+- `MINERU_HYBRID_BATCH_RATIO`：
+ * 用于设置 hybrid-* 后端中 小模型处理的batch倍率
+ * 在hybrid-http-client中较为常用，可以通过控制小模型的batch倍率来调整单个客户端的显存占用量
+ * 单个client端显存大小 | MINERU_HYBRID_BATCH_RATIO
+ ------------------|------------------------
+ <= 6 GB | 8
+ <= 4 GB | 4
+ <= 3 GB | 2
+ <= 2 GB | 1
+
+- `MINERU_HYBRID_FORCE_PIPELINE_ENABLE`：
+ * 用于强制将 hybrid-* 后端中的 文本提取部分使用 小模型 进行处理
+ * 默认为`false`，可通过环境变量设置为`true`来启用该功能，从而在某些极端情况下减少幻觉的发生。
+
+- `MINERU_VL_MODEL_NAME`：
+ * 用于指定 vlm/hybrid 后端使用的模型名称，这将允许您在同时存在多个模型的远程openai-server中指定 MinerU 运行所需的模型。
+
+- `MINERU_VL_API_KEY`:
+ * 用于指定 vlm/hybrid 后端使用的API Key，这将允许您在远程openai-server中进行身份验证。
diff --git a/docs/zh/usage/index.md b/docs/zh/usage/index.md
new file mode 100644
index 000000000..d5b2f313a
--- /dev/null
+++ b/docs/zh/usage/index.md
@@ -0,0 +1,44 @@
+# 使用指南
+
+本章节提供了项目的完整使用说明。我们将通过以下几个部分，帮助您从基础到进阶逐步掌握项目的使用方法：
+
+## 目录
+- 本地部署
+ * [基础使用](./quick_usage.md) - 快速上手和基本使用
+ * [模型源配置](./model_source.md) - 模型源的详细配置说明
+ * [命令行工具](./cli_tools.md) - 命令行工具的详细参数说明
+ * [命令行进阶参数](./advanced_cli_parameters.md) - 一些适配命令行工具的进阶参数说明
+- 其他加速卡适配（🚀官方支持/❤️社区贡献）
+ * [昇腾 Ascend](acceleration_cards/Ascend.md) 🚀
+ * [平头哥 T-Head](acceleration_cards/THead.md) 🚀
+ * [沐曦 METAX](acceleration_cards/METAX.md) 🚀
+ * [海光 Hygon](acceleration_cards/Hygon.md) 🚀
+ * [燧原 Enflame](acceleration_cards/Enflame.md) 🚀
+ * [摩尔线程 MooreThreads](acceleration_cards/MooreThreads.md) 🚀
+ * [天数智芯 IluvatarCorex](acceleration_cards/IluvatarCorex.md) 🚀
+ * [寒武纪 Cambricon](acceleration_cards/Cambricon.md) 🚀
+ * [昆仑芯 Kunlunxin](acceleration_cards/Kunlunxin.md) 🚀
+ * [太初元碁 Tecorigin](acceleration_cards/Tecorigin.md) ❤️
+ * [壁仞 Biren](acceleration_cards/Biren.md) ❤️
+ * [AMD #3662](https://github.com/opendatalab/MinerU/discussions/3662) ❤️
+ * [瀚博 VastAI #4237](https://github.com/opendatalab/MinerU/discussions/4237) ❤️
+- 插件与生态
+ * [Cherry Studio](plugin/Cherry_Studio.md)
+ * [Sider](plugin/Sider.md)
+ * [Dify](plugin/Dify.md)
+ * [n8n](plugin/n8n.md)
+ * [Coze](plugin/Coze.md)
+ * [FastGPT](plugin/FastGPT.md)
+ * [ModelWhale](plugin/ModelWhale.md)
+ * [DingTalk](plugin/DingTalk.md)
+ * [DataFlow](plugin/DataFlow.md)
+ * [BISHENG](plugin/BISHENG.md)
+ * [RagFlow](plugin/RagFlow.md)
+
+## 开始使用
+
+自 3.0 起，`mineru` 默认作为基于 `mineru-api` 的编排客户端运行；`mineru-router` 的多服务、多 GPU 用法也会在本章的基础使用与命令行工具章节中说明。
+
+建议按照上述顺序阅读文档，这样可以帮助您更好地理解和使用项目功能。
+
+如果您在使用过程中遇到问题，请查看 [FAQ](../faq/index.md)
diff --git a/docs/zh/usage/model_source.md b/docs/zh/usage/model_source.md
new file mode 100644
index 000000000..ff73cd87c
--- /dev/null
+++ b/docs/zh/usage/model_source.md
@@ -0,0 +1,49 @@
+# 模型源说明
+
+MinerU使用 `HuggingFace` 和 `ModelScope` 作为模型仓库，用户可以根据需要切换模型源或使用本地模型。
+
+- `HuggingFace` 是默认的模型源，在全球范围内提供了优异的加载速度和极高稳定性。
+- `ModelScope` 是中国大陆地区用户的最佳选择，提供了无缝兼容的SDK模块，适用于无法访问`HuggingFace`的用户。
+
+## 模型源的切换方法
+
+### 通过环境变量切换
+MinerU 通过 `MINERU_MODEL_SOURCE` 环境变量配置模型源，这适用于所有命令行工具和 API 调用。
+```bash
+export MINERU_MODEL_SOURCE=modelscope
+mineru -p -o
+```
+或在代码中设置：
+```python
+import os
+os.environ["MINERU_MODEL_SOURCE"] = "modelscope"
+```
+>[!TIP]
+> MinerU 已不再提供用于切换模型源的命令行参数。通过环境变量设置的模型源会在当前终端会话中生效，直到终端关闭或环境变量被修改。
+
+
+## 使用本地模型
+
+### 1. 下载模型到本地
+```bash
+mineru-models-download --help
+```
+或使用交互式命令行工具选择模型下载：
+```bash
+mineru-models-download
+```
+> [!NOTE]
+>- 下载完成后，模型路径会在当前终端窗口输出，并自动写入用户目录下的 `mineru.json`。
+>- 您也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建配置文件。
+>- 模型下载到本地后，您可以自由移动模型文件夹到其他位置，同时需要在 `mineru.json` 中更新模型路径。
+>- 如您将模型文件夹部署到其他服务器上，请确保将 `mineru.json`文件一同移动到新设备的用户目录中并正确配置模型路径。
+>- 如您需要更新模型文件，可以再次运行 `mineru-models-download` 命令，模型更新暂不支持自定义路径，如您没有移动本地模型文件夹，模型文件会增量更新；如您移动了模型文件夹，模型文件会重新下载到默认位置并更新`mineru.json`。
+>- `mineru-models-download` 必须使用远端模型源执行真实下载；如果当前终端已设置 `MINERU_MODEL_SOURCE=local`，该命令会仅在本次执行中临时忽略该值，并改用您选择的 `huggingface` 或 `modelscope` 下载模型。
+
+### 2. 使用本地模型进行解析
+
+通过环境变量启用本地模型：
+```bash
+export MINERU_MODEL_SOURCE=local
+mineru -p -o
+```
diff --git a/docs/zh/usage/plugin/BISHENG.md b/docs/zh/usage/plugin/BISHENG.md
new file mode 100644
index 000000000..dc89386fa
--- /dev/null
+++ b/docs/zh/usage/plugin/BISHENG.md
@@ -0,0 +1,11 @@
+# BISHENG 简介
+
+BISHENG毕昇 是一款开源 LLM应用开发平台，主攻企业场景， 已有大量行业头部组织及世界500强企业在使用。“毕昇”是活字印刷术的发明人，活字印刷术为人类知识的传递起到了巨大的推动作用。BISHENG毕昇团队希望“BISHENG毕昇”同样能够为智能应用的广泛落地提供有力支撑。
+
+
+
+
+- 官网地址：https://bisheng.dataelem.com/
+- Miner 在BISHENG毕昇 项目中的插件项目：https://github.com/dataelement/bisheng/pulls
+
+特别鸣谢 [@pzc163](https://github.com/pzc163)
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/Cherry_Studio.md b/docs/zh/usage/plugin/Cherry_Studio.md
new file mode 100644
index 000000000..5713a1c87
--- /dev/null
+++ b/docs/zh/usage/plugin/Cherry_Studio.md
@@ -0,0 +1,238 @@
+# Cherry Studio 简介
+
+Cherry Studio 是一款功能强大的多模型 AI 客户端软件，支持 Windows、macOS 和 Linux 等多平台运行，集成了 OpenAI、DeepSeek、Gemini、Anthropic 等主流 AI 云服务，同时支持本地模型运行，用户可以灵活切换不同的AI模型。
+
+目前，MinerU 强大的文档解析能力已深度集成到 Cherry Studio 的知识库与对话交互中，为用户带来更便捷的文档处理与信息获取体验。
+
+
+
+- Cherry Studio 官网地址：https://www.cherry-ai.com/
+
+
+# MinerU 在 Cherry Studio 中的使用方法
+
+## 进入 Cherry Studio 设置
+
+a. 打开 Cherry Studio 应用程序
+
+b. 点击左下角的"设置"按钮，进入设置页面
+
+c. 在左侧菜单中，选择"MCP 服务器"
+
+在右侧的 MCP 服务器配置界面中，您可以看到已有的 MCP 服务器列表。点击右上角的"添加服务器"按钮来创建新的 MCP 服务，或者点击现有服务来编辑配置。
+
+## 添加 MinerU-MCP 配置
+
+点击"添加服务器"后，您将看到一个配置表单。请按以下步骤填写：
+
+**a. 名称**：输入"MinerU-MCP"或您喜欢的其他名称
+
+**b. 描述**：可选，如"文档转换为Markdown工具"
+
+**c. 类型**：选择"标准输入/输出（stdio）"
+
+**d. 命令**：输入 uvx
+
+**e. 参数**：输入 mineru-mcp
+
+**f. 环境变量**：添加以下环境变量
+
+```Plain
+MINERU_API_BASE=https://mineru.net
+MINERU_API_KEY=您的API密钥
+OUTPUT_DIR=./downloads
+USE_LOCAL_API=false
+LOCAL_MINERU_API_BASE=http://localhost:8888
+```
+
+使用 *`uvx`* 命令可以自动处理 mineru-mcp 的安装和运行，**无需预先手动安装 mineru-mcp 包**。这是最简单的配置方式。
+
+## 保存配置
+
+确认无误后，点击界面右上角的"保存"按钮完成配置。保存后，MCP 服务器列表中会显示您刚刚添加的 MinerU-MCP 服务。
+
+
+
+
+
+## 使用 Cherry Studio 中的 MinerU MCP
+
+一旦配置完成，您可以在 Cherry Studio 中的对话中使用 MinerU MCP 工具。在 Cherry Studio 中，您可以使用如下提示让模型调用 MinerU MCP 工具。模型会自动识别任务并调用相应的工具。
+
+## 示例 1: 使用 URL 转换文档
+
+**用户输入:**
+
+```Plain
+请使用 MinerU MCP 将以下 URL 的 PDF 文档转换为 Markdown 格式：https://example.com/sample.pdf
+```
+
+**模型将执行的步骤：**
+
+模型识别这是文档转换任务，并调用 *`parse_documents`* 工具，参数为:
+
+```Plain
+{"file_sources": "https://example.com/sample.pdf"}
+```
+
+工具处理完成后，模型会告知您转换结果。
+
+
+
+## 示例 2: 转换本地文档
+
+**用户输入:**
+
+```Plain
+请使用 MinerU-MCP 将本地的 D://sample.pdf 文件转换为 Markdown 格式
+```
+
+**模型将执行的步骤：**
+
+模型识别这是本地文档转换任务，调用 `parse_documents` 工具，参数为:
+
+```Plain
+{"file_sources": "D://sample.pdf"}
+```
+
+
+
+## 示例 3: 启用 OCR 处理扫描文档
+
+**用户输入:**
+
+```Plain
+请使用 MinerU-MCP 将以下 URL 的扫描版 PDF 文档转换为 Markdown 格式，并启用 OCR 功能：
+https://example.com/scanned.pdf
+```
+
+**模型将执行的步骤：**
+
+模型识别这是需要 OCR 处理的文档转换任务，调用 `parse_documents` 工具，并启用 OCR 参数:
+
+```Plain
+{"file_sources": "https://example.com/scanned.pdf","enable_ocr": true}
+```
+
+
+
+## 示例 4: 完整对话流程
+
+以下是一个完整的对话流程示例：
+
+**用户:**
+
+```Plain
+我有一份学术论文的 PDF，网址是 https://arxiv.org/pdf/2303.08774.pdf，能帮我转换成 Markdown 格式吗？
+```
+
+**模型:**
+
+```Plain
+我可以帮您将这份学术论文转换为 Markdown 格式。我将使用 MinerU-MCP 工具来处理这个任务。
+[模型调用 parse_documents 工具，处理过程中...]
+论文已成功转换为 Markdown 格式。转换结果如下：
+# The Capacity of Diffusion Models to Memorize and Generate Training Data
+## Abstract
+Recent diffusion models can generate high-quality images that are nearly indistinguishable from real ones...
+[显示论文内容...]
+```
+
+
+
+## 工具参数详解
+
+在使用过程中，模型会根据您的指令自动选择合适的工具和参数。以下是主要工具的参数说明：
+
+### ● *parse_documents* 工具参数
+
+
+
+### ● *get_ocr_languages* 工具参数
+
+无需参数，用于获取OCR支持的语言列表。
+
+## 高级用法
+
+### 指定语言和页码范围
+
+**用户输入:**
+
+```Plain
+请使用 MinerU MCP 将以下 URL 的文档转换为 Markdown 格式，只处理第 5-10 页，并指定语言为中文：https://example.com/document.pdf
+```
+
+模型会使用 *`parse_documents`* 工具，并设置 *`language`* 参数为 "ch"，*`page_ranges`* 参数为 "5-10"。
+
+### 批量处理多个文档
+
+**用户输入:**
+
+```Plain
+请使用 MinerU-MCP 将以下多个 URL 的文档转换为 Markdown 格式：
+https://example.com/doc1.pdf
+https://example.com/doc2.pdf
+https://example.com/doc3.pdf
+```
+
+模型会调用 *`parse_documents`* 工具，并将多个 URL 以逗号分隔传入 *`file_sources`* 参数。
+
+## 注意事项
+
+● 当设置 *`USE_LOCAL_API=true`* 时，使用本地配置的API进行解析
+
+● 当设置 *`USE_LOCAL_API=false`* 时，会使用 MinerU 官网的API进行解析
+
+● 处理大型文档可能需要较长时间，请耐心等待
+
+● 如果遇到超时问题，请考虑分批处理文档或使用本地API模式
+
+## 常见问题与解决方案
+
+### 无法启动 MCP 服务
+
+**问题**：运行 *`uv run -m mineru.cli`*` `时报错。
+
+**解决方案**：
+
+● 确保已激活虚拟环境
+
+● 检查是否已安装所有依赖
+
+● 尝试使用 *`python -m mineru.cli`*` `命令替代
+
+### 文件转换失败
+
+**问题**：文件上传成功但转换失败。
+
+**解决方案**：
+
+● 检查文件格式是否受支持
+
+● 确认API密钥是否正确
+
+● 查看MCP服务日志获取详细错误信息
+
+### 文件路径问题
+
+**问题**：使用 `parse_documents` 工具处理本地文件时报找不到文件错误。
+
+**解决方案**：请确保使用绝对路径，或者相对于服务器运行目录的正确相对路径。
+
+### MCP 服务调用超时问题
+
+**问题**：调用 *`parse_documents`* 工具时出现 *`Error calling tool 'parse_documents': MCP error -32001: Request timed out`* 错误。
+
+**解决方案**：这个问题常见于处理大型文档或网络不稳定的情况。在某些 MCP 客户端（如 Cursor）中，超时后可能导致无法再次调用 MCP 服务，需要重启客户端。最新版本的 Cursor 中可能会显示正在调用 MCP，但实际上没有真正调用成功。建议：
+
+**● 等待官方修复**：这是Cursor客户端的已知问题，建议等待Cursor官方修复
+
+**● 处理小文件**：尽量只处理少量小文件，避免处理大型文档导致超时
+
+**● 分批处理**：将多个文件分成多次请求处理，每次只处理一两个文件
+
+● 增加超时时间设置（如果客户端支持）
+
+● 对于超时后无法再次调用的问题，需要重启 MCP 客户端
+
+● 如果反复出现超时，请检查网络连接或考虑使用本地 API 模式
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/Coze.md b/docs/zh/usage/plugin/Coze.md
new file mode 100644
index 000000000..5ddfcae47
--- /dev/null
+++ b/docs/zh/usage/plugin/Coze.md
@@ -0,0 +1,92 @@
+# Coze 简介
+
+Coze（中文版名称：扣子） 是字节跳动推出的零代码 AI 应用开发平台。无论用户是否有编程经验，都可以通过该平台快速创建各种类型的聊天机器人、智能体、AI 应用和插件，并将其部署在社交平台和即时聊天应用程序中。
+
+目前，MinerU 插件已在 Coze 插件商店上线，通过其强大的文档解析能力，为用户搭建智能体与工作流提供文档解析能力，加快用户 AI 应用的开发。
+
+
+
+- 扣子官网地址：https://www.coze.cn/
+- MinerU 扣子插件下载地址：https://www.coze.cn/store/plugin/7527957359730360354
+
+# MinerU 在 Coze 中的使用方法
+
+## **Coze：集成应用**
+
+- 进入 https://www.coze.cn/home coze 开发平台
+
+## 智能体
+
+### 工作空间 -> 项目开发 -> 创建 -> 创建智能体 -> 创建 -> 输入项目名
+
+
+
+
+
+### 插件配置 -> 添加 `插件` -> 搜索 `MinerU`
+
+
+
+### 添加 `parse_file` 工具（在线版）
+
+
+
+### 选择 `MinerU` 插件 -> 编辑参数 -> 填写 api key
+
+
+
+
+
+> 记得关闭 url 和 token 显示
+
+### 调试 `智能体`
+
+
+
+## 工作流
+
+> 用工作流的方式使用 minerU
+
+### 工作流 -> 创建工作流
+
+
+
+
+
+### 工作流插件配置 -> 添加 `插件` -> 搜索 `MinerU` -> 添加
+
+
+
+
+
+### 选择`MinerU` 插件 -> 编辑参数 -> 填写 api key
+
+
+
+### 选择开始节点 -> 配置 `input` 类型为文件类型 -> 连接到 `mineru` 节点
+
+
+
+
+
+### 选择结束节点 -> 连接到 `mineru` 节点 -> 配置 `output` 输出为 `mineru` 节点的 `parse_file.text`
+
+
+
+
+
+### 上传文件 -> 试运行
+
+
+
+
+
+### 发布 -> 添加到当前智能体
+
+
+
+
+
+### 移除 `mineru` 插件 -> 调试
+
+
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/DataFlow.md b/docs/zh/usage/plugin/DataFlow.md
new file mode 100644
index 000000000..c0d5743b7
--- /dev/null
+++ b/docs/zh/usage/plugin/DataFlow.md
@@ -0,0 +1,11 @@
+# 元枢智汇 ADP 智能数据平台 简介
+
+元枢智汇 ADP 智能数据平台基于自研 AI 数据库和 DataFlow数据准备框架打造，旨在帮助企业高效管理、检索、处理海量数据，并通过体系化、自动化数据治理降低模型/智能体训练的专业门槛，帮助企业结合业务场景发挥私有数据的价值，真正落地AI应用。
+
+目前，MinerU 已深度集成于元枢智汇 ADP 智能数据平台的 DataFlow 模块中，其数据解析服务由文档语料提取引擎 MinerU 提供支持。
+
+
+
+
+- 官网地址：https://adp.originhub.tech/agent
+- Miner fastGPT 插件下载地址：https://cloud.fastgpt.io/dashboard/systemPlugin?type=productivity
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/Dify.md b/docs/zh/usage/plugin/Dify.md
new file mode 100644
index 000000000..d8b7bee75
--- /dev/null
+++ b/docs/zh/usage/plugin/Dify.md
@@ -0,0 +1,171 @@
+# Dify 简介
+
+**Dify** 是一个开源的大语言模型（LLM）应用开发平台，旨在简化和加速生成式 AI 应用的创建和部署。它结合了后端即服务（BaaS）和 LLMOps 的理念，为开发者提供了用户友好的界面和强大的工具，有效降低了 AI 应用开发的门槛。
+
+目前 MinerU 与 Dify 联合研发的 MinerU 插件已在 Dify 市场上架，帮助用户搭建工作流，提供文档解析的工作。
+
+
+
+- Dify 官网地址：https://dify.ai/zh
+- MinerU Dify 插件下载地址：https://marketplace.dify.ai/plugins/langgenius/mineru
+
+# MinerU 在 Dify 中的使用方法
+
+## 一、**新版MinerU Dify插件亮点 (v0.4.0)**
+
+- **完美适配MinerU2**：全面兼容MinerU2的最新功能，释放顶尖的文档解析能力。
+- **超高灵活性**：同时支持官方在线API和本地化部署的API（并向下兼容 1.x 版本）。
+- **赋能工作流**：让Dify的Agent拥有强大的文档“读写”能力，轻松处理复杂任务。
+
+
+## **二、实战演练：两个案例带你快速上手**
+
+空谈不如实战。下面我们通过两个典型场景，向你展示新版插件的强大之处。
+
+### 准备
+
+1. 在Dify插件页面安装MinerU插件（私有化部署的Dify同理）
+
+
+2. 填写API URL等信息
+
+
+
+使用官方API时令牌（Token）必须提供👆，使用本地部署API时令牌可不填写👇
+
+
+
+### **案例一：解析单文件，搭建Chat PDF应用**
+
+想借助AI与你的文档对话吗？跟着下面几步，轻松实现
+
+#### 第一步：创建空白应用，选择“Chatflow”
+
+输入应用名称与描述
+
+
+
+#### 第二步：创建的初始模板中，选择“开始”节点
+
+字段类型选为单文件，填写变量名称（此处填为input_file）,支持文档类型选为文档与图片
+
+
+
+#### 第三步：添加工具节点——MinerU插件来解析上一步开始节点上传的文件
+
+
+
+#### 第四步：设置MinerU的输入变量，选择上一步开始节点添加的 `input_file`
+
+
+
+#### 第五步：配置LLM模型
+
+选择“LLM”节点后，如果没有模型可用，需要单独在插件市场安装（这里使用 Deepseek作为示例）
+
+“上下文”选择MinerU的输出变量 `text`（MinerU解析文档后的markdown格式）
+
+
+
+在“SYSTEM”区域根据实际需求填写提示词，可如图填写“在Parse File `text`中提取用户的问题答案”
+
+
+
+#### 第六步：预览，上传文件并提问机器人关于文档的内容
+
+至此一个简单的文档问答应用Chat PDF搭建完成，点击“预览”，查看效果如何👇
+
+
+
+结果如下：
+
+
+
+#### **第七步：发布与测试**
+
+保存并发布你的应用。现在，上传一份PDF或图片，你就可以和它自由对话了！
+
+
+
+### **案例二：自动化批量处理文档，并上传至云端S3**
+
+需要处理大量文档并归档？MinerU 插件同样能胜任
+
+#### 第一步：安装 botos3 插件
+
+
+
+#### 第二步：配置 S3 bucket
+
+
+
+#### 第三步：创建工作流
+
+选择字段类型为“文件列表”，填写变量名称（此处填为input_files）,支持的文档类型选为文档与图片
+
+
+
+#### 第四步：添加“迭代”
+
+在“开始”节点后添加“迭代”，并配置迭代内的MinerU节点,设置迭代的输入为上一步开始节点的`upload_files`，输出节点暂时不填写，再整个迭代配置完成后选择MinerU节点Parse File的`full_zip_url`
+
+
+
+将MinerU的输入参数file选择为迭代器的 `item`
+
+
+
+
+
+#### 第五步：增加中间节点“代码执行”来转换MinerU的解析结果
+
+**输入变量(变量名称需与代码定义一致)**
+
+- **text：**选择MinerU Parse File的输出变量`text`
+- **uploadFiles：**选择“开始”节点的文件列表`upload_files`，用来根据迭代的index索引下标找到对应的原始文件名
+- **index：**迭代的下标索引，选择迭代器的`index`
+
+**输出变量(变量名称需与代码定义一致)**
+
+- **fileName：**String
+- **base64：**String
+
+
+
+代码选择JavaScript，编写转换代码：
+
+暂时无法在飞书文档外展示此内容
+
+以下为Python版本：
+
+暂时无法在飞书文档外展示此内容
+
+#### 第六步：配置 Botos3 插件来上传内容
+
+添加工具节点Botos3，选择“通过s3上传base64”
+
+
+
+文件base64选择代码执行（图中为**转换MINERU MD文本**）输出的base64字段
+
+
+
+S3对象key，S3 对象key填写文件存储的路径，在 botos3 插件配置界面已经填写了 bucket 名称，这里只需要填写在bucket下存储的目录即可。选择代码执行**（图中为转换MINERU MD文本）**的`fileName`
+
+
+
+#### 第七步：预览效果
+
+连接结束节点，至此，一个简单的上传到s3的工作流配置完成，点击“运行”看看效果👇：
+
+
+
+
+
+#### 第八步：Vis3查看文档
+
+运行结束，可通过[vis3](https://github.com/opendatalab/Vis3?tab=readme-ov-file#features)来查看S3桶内是否已上传解析后的md文件，Vis3使用可参考
+
+[新工具开源！Vis3大模型数据可视化利器：填 AK/SK 直接预览 S3 数据，JSON/视频/图片秒开！本地文件也可用](https://mp.weixin.qq.com/s/p3rH4EaoJB-AK7RWeDvOhg)
+
+
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/DingTalk.md b/docs/zh/usage/plugin/DingTalk.md
new file mode 100644
index 000000000..18c03262a
--- /dev/null
+++ b/docs/zh/usage/plugin/DingTalk.md
@@ -0,0 +1,12 @@
+# 钉钉简介
+
+钉钉（DingTalk）是阿里巴巴集团打造的企业级智能移动办公平台，是数字经济时代的企业组织协同办公和应用开发平台。钉钉整合了 IM 即时沟通、钉钉文档、钉闪会、钉盘、Teambition、OA审批、智能人事、钉工牌、工作台等功能，旨在实现简单、高效、安全、智能的数字化工作方式。它支持企业组织数字化和业务数字化，覆盖“人、财、物、事、产、供、销、存”的全链路管理。
+
+通过钉钉开放平台上的SaaS软件，企业可低成本搭建数字化应用，整合所有数字化系统。此外，钉钉提供超过2000个API接口，为企业数字化转型提供开放兼容环境。不会代码的用户也可利用低代码工具构建CRM、ERP、OA、项目管理、进销存等系统。
+
+目前，钉钉文档、AI 表格等产品此前已深度集成 MinerU 能力，并通过开放平台向生态开发者开放文档解析功能，为 DLU 的联合研发提供了扎实的技术与场景基础。
+
+
+
+
+- 钉钉官网：https://www.dingtalk.com/
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/FastGPT.md b/docs/zh/usage/plugin/FastGPT.md
new file mode 100644
index 000000000..cf32e8c56
--- /dev/null
+++ b/docs/zh/usage/plugin/FastGPT.md
@@ -0,0 +1,13 @@
+# FastGPT 简介
+
+FastGPT 是一个基于 LLM 大语言模型的知识库问答系统，将智能对话与可视化编排完美结合，让 AI 应用开发变得简单自然。无论您是开发者还是业务人员，都能轻松打造专属的 AI 应用。
+
+目前，MinerU 插件已在 Coze 插件商店上线，通过其强大的文档解析能力，为用户搭建智能体与工作流提供文档解析能力，加快用户 AI 应用的开发。
+
+
+
+
+
+
+- 官网地址：https://fastgpt.cn
+- Miner fastGPT 插件下载地址：https://cloud.fastgpt.io/dashboard/systemPlugin?type=productivity
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/ModelWhale.md b/docs/zh/usage/plugin/ModelWhale.md
new file mode 100644
index 000000000..c3aaf1d6c
--- /dev/null
+++ b/docs/zh/usage/plugin/ModelWhale.md
@@ -0,0 +1,18 @@
+# ModelWhale 简介
+
+ModelWhale是一款高效率的数据科学云端协作工具，为数据工作者提供了即开即用的云端分析环境，Jupyter Notebook 交互式和Canvas 拖拽式两种分析界面，帮助科研者、教育工作者解决底层工程繁复、数据难以安全应用、成果流转复现困难等问题。基于不同使用场景，ModelWhale 为用户提供三个产品版本，分别是基础版、专业版、团队版。
+
+目前，MinerU 插件已在 ModelWhale 工作中，通过其强大的文档解析能力，为用户搭建智能体与工作流提供文档解析能力，加快用户 AI 应用的开发。
+
+images/DingTalk_01.png
+
+
+
+
+
+
+
+
+
+- ModelWhale 官网：Mohttps://www.modelwhale.com/pricing?scroll=1
+- MinerU 在ModelWhale 的使用地址：https://www.heywhale.com/org/7b38d/workspace/iframe?url=https://www.heywhale.com/api/model/services/68089d360b1519a862ccb9b4/app/
diff --git a/docs/zh/usage/plugin/RagFlow.md b/docs/zh/usage/plugin/RagFlow.md
new file mode 100644
index 000000000..6db7f31b2
--- /dev/null
+++ b/docs/zh/usage/plugin/RagFlow.md
@@ -0,0 +1,84 @@
+## RAGFlow
+
+RAGFlow 是一款开源 RAG（Retrieval-Augmented Generation）引擎与应用平台，深度融合了深度文档理解、自动化 RAG 工作流与大模型调用，打通了复杂数据处理、知识检索、增强生成的全流程，旨在为企业及开发者提供一站式智能问答开发服务，并支持各类复杂场景下大模型的构建与应用落地。
+
+目前，MinerU 已深度集成至 RAGFlow 知识库在线版本，作为内置 PDF 文档解析器，为用户知识库搭建提供专业、可靠的文档解析支持。本地部署版本部署使用方式详见下方使用教程。
+
+使用可访问：https://demo.ragflow.io/
+
+
+
+## 使用教程：如何在 RAGFlow 中使用 MinerU
+
+### 一、安装配置
+
+首先，我们建议您通过 docker 的形式在本地部署 RagFlow 以方便使用 MinerU 插件作为解析工具。在安装完 RagFlow 后执行：
+
+1. **版本检查：**
+
+ 确保你的RAGFlow版本 >= `v0.21.1`。
+
+2. **更新 .env 文件：**
+
+ 为了确保服务能被平稳修改，建议先在 `cmd` 运行 `docker compose down` 停掉服务。
+
+ 打开 `.env` 文件，在文件的末尾，添加这两行代码，保存文件。
+
+ ```Python
+ HF_ENDPOINT=https://hf-mirror.com
+ MINERU_EXECUTABLE=/ragflow/uv_tools/.venv/bin/mineru
+ ```
+
+3. **启动并进入容器：**
+
+ 在 `cmd` 中，重新启动服务：`docker compose up -d`
+
+ 等待服务全部 `Running` 或 `Healthy` 后，运行以下命令进入RAGFlow的核心容器：
+
+ ```Bash
+ docker compose exec ragflow-cpu bash
+ ```
+
+ （你的命令行提示符会从 `C:\...>` 变为 `root@...`）
+
+4. **在容器内下载 MinerU 模型：**
+
+ 在容器内部，依次运行以下 5 条命令
+
+ ```Bash
+ mkdir uv_tools
+ cd uv_tools
+ uv venv .venv
+ source .venv/bin/activate
+ uv pip install -U "mineru[core]" -i https://mirrors.aliyun.com/pypi/simple
+ ```
+
+5. **退出并重启：**
+
+ 安装完成后，输入 `exit` 并按回车。
+
+ 运行重启命令，让 RAGFlow 加载刚装好的 MinerU
+
+ ```Bash
+ docker compose restart ragflow-cpu
+ ```
+
+### 二、使用入口
+
+在本地部署完毕后，要启用 MinerU，您需要在进入 RagFlow 特定知识库的配置页面并选择 MinerU 作为默认的 PDF 解析器。（注：RagFlow 在线版中已经内置了 MinerU 插件为您提供了高级的 PDF 文件解析能力，使用方式与此一致。）
+
+**入口和配置步骤：**
+
+1. **进入知识库配置：**
+ 1. 首先，在您的知识库管理界面，选择您需要配置的特定知识库（例如图示中的 "content" 知识库）。
+ 2. 在知识库详情页面的左侧导航栏中，点击【**配置**】选项卡。
+2. **定位 PDF 解析器设置：**
+ 1. 向下滚动页面，找到“**Ingestion pipeline**”（摄取管道）设置部分。
+ 2. 在此部分中，您会看到一个名为【**PDF解析器**】（PDF Parser）的选项。
+3. **选择 MinerU：**
+ 1. 点击【PDF解析器】旁边的下拉菜单。
+ 2. 从可用选项中，选择【**MinerU**】。
+4. **保存修改：**
+ 1. 完成选择后，请务必点击页面底部的【**保存**】按钮，以使更改生效。
+
+
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/Sider.md b/docs/zh/usage/plugin/Sider.md
new file mode 100644
index 000000000..b5f227345
--- /dev/null
+++ b/docs/zh/usage/plugin/Sider.md
@@ -0,0 +1,10 @@
+# Sider 简介
+
+Sider 是一款浏览器侧边栏类的 AI 助手扩展，主要在网页右侧开启一个“随处可用”的智能面板，将对话式 AI（如 GPT、Claude、Gemini 等）带到你正在浏览的任何页面中。它的核心定位是：提升阅读、写作、翻译、检索与总结效率，并与网页内容深度联动。
+
+目前，Sider在 Wisebase 模块中深度集成了 MinerU 的相关功能。该模块是一个由AI驱动的知识库，您可以通过上传 PDF 等各类型文件，构建个人图书馆以实现高效的知识管理，MinerU 可以帮助您更好地解析此类文件，精准地提取文件中的信息。
+
+
+
+- Sider 官网地址：https://sider.ai/zh-CN/chat
+- 使用集成 MinerU 相关功能的 Sider 地址：https://sider.ai/zh-CN/wisebase
\ No newline at end of file
diff --git a/docs/zh/usage/plugin/n8n.md b/docs/zh/usage/plugin/n8n.md
new file mode 100644
index 000000000..fa5bb2bf0
--- /dev/null
+++ b/docs/zh/usage/plugin/n8n.md
@@ -0,0 +1,54 @@
+# n8n 简介
+
+**n8n** 是一款以低代码（Low-code）、工作流自动化为核心的应用开发平台，许多企业都借助于其灵活的节点（Node）配置，实现业务流程的自动化执行。它通过可视化界面和代码扩展能力，帮助用户连接各种应用程序和服务，构建复杂的自动化流程,降低用户使用门槛。
+
+目前，MinerU 已将其强大的文档解析能力封装为 n8n 节点，用户在搭建工作流时，可以更加便捷地处理复杂的文档解析任务。
+
+
+
+- n8n 官网地址：https://n8n.io/
+- MinerU n8n 插件下载地址：https://www.npmjs.com/package/n8n-nodes-mineru
+
+# MinerU 在 n8n 中的使用方法
+
+## step1 进入社区node安装界面
+
+
+
+## step2 安装 n8n-nodes-mineru 节点
+
+≈assets/images/n8n_2.png)
+
+## step3 新建工作流，添加 n8n-nodes-mineru 节点，并设置 api key
+
+
+
+
+
+
+
+
+
+### n8n使用节点文档
+
+https://www.npmjs.com/package/n8n-nodes-mineru
+
+### **在工作流内集成解压功能**
+
+#### 导入 json 模板
+
+暂时无法在飞书文档外展示此内容
+
+
+
+### 配置 凭证和文档url
+
+
+
+### 根据各自的需求配置所需的输出
+
+
+
+### 调试
+
+
\ No newline at end of file
diff --git a/docs/zh/usage/quick_usage.md b/docs/zh/usage/quick_usage.md
new file mode 100644
index 000000000..bd85c179d
--- /dev/null
+++ b/docs/zh/usage/quick_usage.md
@@ -0,0 +1,157 @@
+# 使用 MinerU
+
+## 快速配置模型源
+MinerU默认使用`huggingface`作为模型源，若用户网络无法访问`huggingface`，可以通过环境变量便捷地切换模型源为`modelscope`：
+```bash
+export MINERU_MODEL_SOURCE=modelscope
+```
+有关模型源配置和自定义本地模型路径的更多信息，请参考文档中的[模型源说明](./model_source.md)。
+
+## 通过命令行快速使用
+MinerU内置了命令行工具，用户可以通过命令行快速使用MinerU进行文档解析：
+```bash
+mineru -p -o
+```
+> [!TIP]
+> - ``：本地 `PDF` / 图片 / `DOCX` / `PPTX` / `XLSX` 文件或目录
+> - ``：输出目录
+> - 未传 `--api-url` 时，CLI 会自动拉起本地临时 `mineru-api`
+> - 传入 `--api-url` 时，CLI 会直连远端或已有本地 FastAPI 服务
+>
+> 更多关于输出文件的信息，请参考[输出文件说明](../reference/output_files.md)。
+
+> [!NOTE]
+> 命令行工具会在Linux和macOS系统自动尝试cuda/mps加速。Windows用户如需使用cuda加速，
+> 请前往 [Pytorch官网](https://pytorch.org/get-started/locally/) 选择适合自己cuda版本的命令安装支持加速的`torch`和`torchvision`。
+
+如果需要通过自定义参数调整解析选项，您也可以在文档中查看更详细的[命令行工具使用说明](./cli_tools.md)。
+
+## 通过api、webui、http-client/server进阶使用
+
+- 通过fast api方式调用：
+ ```bash
+ mineru-api --host 0.0.0.0 --port 8000
+ ```
+ >[!TIP]
+ >在浏览器中访问 `http://127.0.0.1:8000/docs` 查看API文档。
+ >
+ >- 健康检查接口：`GET /health`
+ > 返回 `protocol_version`、`processing_window_size`、`max_concurrent_requests` 等服务信息
+ >- 异步任务提交接口：`POST /tasks`
+ >- 同步解析接口：`POST /file_parse`
+ >- 任务查询接口：`GET /tasks/{task_id}`、`GET /tasks/{task_id}/result`
+ >- API 输出目录由服务端固定控制，默认写入 `./output`
+ >- 上传文件当前支持 `PDF`、图片与 `DOCX`、`PPTX`、`XLSX`
+ >
+ >- `POST /tasks` 会立即返回 `task_id`；`POST /file_parse` 会在内部提交到同一个任务管理器，等待任务完成后同步返回最终结果。
+ >- 当任务处于排队状态时，任务提交结果和状态查询结果中可能会返回 `queued_ahead` 字段，用于表示前方排队任务数。
+ >- 任务为单进程、进程内状态实现，服务重启、`--reload` 热重载或多进程部署后不保证仍可查询历史任务状态。
+ >- 默认任务完成或失败后保留 24 小时，随后自动清理任务状态和输出目录；清理后访问任务状态或结果会返回 `404`。
+ >- 可通过环境变量 `MINERU_API_TASK_RETENTION_SECONDS` 和 `MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS` 调整保留时长与清理轮询间隔。
+ >- 可通过 `--enable-vlm-preload true` 在服务启动阶段预热本地 VLM 模型，避免首次 VLM 或 hybrid 请求时再初始化。
+ >
+ >异步任务提交示例：
+ >```bash
+ >curl -X POST http://127.0.0.1:8000/tasks \
+ > -F "files=@demo/pdfs/demo1.pdf" \
+ > -F "return_md=true"
+ >```
+ >
+ >同步解析示例：
+ >```bash
+ >curl -X POST http://127.0.0.1:8000/file_parse \
+ > -F "files=@demo/pdfs/demo1.pdf" \
+ > -F "return_md=true" \
+ > -F "response_format_zip=true" \
+ > -F "return_original_file=true"
+ >```
+ >
+ >轮询任务状态与结果：
+ >```bash
+ >curl http://127.0.0.1:8000/tasks/
+ >curl http://127.0.0.1:8000/tasks//result
+ >curl http://127.0.0.1:8000/health
+ >```
+ >
+ >http异步调用代码示例：[Python版本](https://github.com/opendatalab/MinerU/blob/master/demo/demo.py)
+
+- 启动gradio webui 可视化前端：
+ ```bash
+ mineru-gradio --server-name 0.0.0.0 --server-port 7860
+ ```
+ >[!TIP]
+ >
+ >- 在浏览器中访问 `http://127.0.0.1:7860` 使用 Gradio WebUI。
+ >- 未传 `--api-url` 时，Gradio 会自动拉起可复用的本地 `mineru-api`；传入 `--api-url` 时则会复用已有本地或远端服务。
+ >- `--enable-vlm-preload true` 会让 Gradio 在 WebUI 启动阶段主动拉起本地 `mineru-api` 并等待 VLM 预加载完成；传入 `--api-url` 时会被忽略。
+ >- WebUI 当前支持上传 `PDF`、图片与 `DOCX`、`PPTX`、`XLSX` 文件。
+
+- 通过 `mineru-router` 进行多服务 / 多 GPU 编排：
+ ```bash
+ mineru-router --host 0.0.0.0 --port 8002 --local-gpus auto
+ ```
+ >[!TIP]
+ >
+ >- `mineru-router` 对外暴露与 `mineru-api` 一致的 `/health`、`/tasks`、`/file_parse`、`/tasks/{task_id}`、`/tasks/{task_id}/result` 接口。
+ >- 可重复使用 `--upstream-url` 聚合多个已有 `mineru-api` 服务，也可通过 `--local-gpus` 自动拉起本地 worker。
+ >- `--enable-vlm-preload true` 仅作用于 router 托管的本地 worker，不会影响通过 `--upstream-url` 接入的远端服务。
+ >- 适用于多服务、多 GPU 和统一入口部署场景。
+
+- 使用`http-client/server`方式调用：
+ ```bash
+ # 启动openai兼容服务器(需要安装vllm或lmdeploy环境)
+ mineru-openai-server --port 30000
+ ```
+ >[!TIP]
+ >在另一个终端中通过http client连接openai server
+ > ```bash
+ > mineru -p -o -b hybrid-http-client -u http://127.0.0.1:30000
+ > ```
+ >`vlm-http-client` 是轻量远程 client，用法上不要求本地安装 `torch`。
+ >`hybrid-http-client` 需要本地具备 `mineru[pipeline]` 及 `torch` 等 pipeline 依赖。
+
+> [!NOTE]
+> 所有`vllm/lmdeploy`官方支持的参数都可用通过命令行参数传递给 MinerU，包括以下命令:`mineru`、`mineru-openai-server`、`mineru-gradio`、`mineru-api`、`mineru-router`，
+> 我们整理了一些`vllm/lmdeploy`使用中的常用参数和使用方法，可以在文档[命令行进阶参数](./advanced_cli_parameters.md)中获取。
+
+## 基于配置文件扩展 MinerU 功能
+
+MinerU 现已实现开箱即用，但也支持通过配置文件扩展功能。您可通过编辑用户目录下的 `mineru.json` 文件，添加自定义配置。
+
+>[!IMPORTANT]
+>`mineru.json` 文件会在您使用内置模型下载命令 `mineru-models-download` 时自动生成，也可以通过将[配置模板文件](https://github.com/opendatalab/MinerU/blob/master/mineru.template.json)复制到用户目录下并重命名为 `mineru.json` 来创建。
+
+以下是一些可用的配置选项：
+
+- `latex-delimiter-config`：
+ * 用于配置 LaTeX 公式的分隔符
+ * 默认为`$`符号，可根据需要修改为其他符号或字符串。
+
+- `llm-aided-config`：
+ * 用于配置 LLM 辅助标题分级的相关参数，兼容所有支持`openai协议`的 LLM 模型
+ * 默认使用`阿里云百炼`的`qwen3-next-80b-a3b-instruct`模型
+ * 您需要自行配置 API 密钥并将`enable`设置为`true`来启用此功能
+ * 如果您的api供应商不支持`enable_thinking`参数，请手动将该参数删除
+ * 例如，在您的配置文件中，`llm-aided-config` 部分可能如下所示：
+ ```json
+ "llm-aided-config": {
+ "api_key": "your_api_key",
+ "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+ "model": "qwen3-next-80b-a3b-instruct",
+ "enable_thinking": false,
+ "enable": false
+ }
+ ```
+ * 要移除`enable_thinking`参数，只需删除包含`"enable_thinking": false`的那一行，结果如下:
+ ```json
+ "llm-aided-config": {
+ "api_key": "your_api_key",
+ "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+ "model": "qwen3-next-80b-a3b-instruct",
+ "enable": false
+ }
+ ```
+
+- `models-dir`：
+ * 用于指定本地模型存储目录，请为`pipeline`和`vlm`后端分别指定模型目录，
+ * 指定目录后您可通过配置环境变量`export MINERU_MODEL_SOURCE=local`来使用本地模型。
diff --git a/magic-pdf.template.json b/magic-pdf.template.json
deleted file mode 100644
index 114dfce32..000000000
--- a/magic-pdf.template.json
+++ /dev/null
@@ -1,23 +0,0 @@
-{
- "bucket_info":{
- "bucket-name-1":["ak", "sk", "endpoint"],
- "bucket-name-2":["ak", "sk", "endpoint"]
- },
- "models-dir":"/tmp/models",
- "layoutreader-model-dir":"/tmp/layoutreader",
- "device-mode":"cpu",
- "layout-config": {
- "model": "layoutlmv3"
- },
- "formula-config": {
- "mfd_model": "yolo_v8_mfd",
- "mfr_model": "unimernet_small",
- "enable": true
- },
- "table-config": {
- "model": "tablemaster",
- "enable": false,
- "max_time": 400
- },
- "config_version": "1.0.0"
-}
\ No newline at end of file
diff --git a/magic_pdf/__init__.py b/magic_pdf/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/config/__init__.py b/magic_pdf/config/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/config/enums.py b/magic_pdf/config/enums.py
deleted file mode 100644
index 6f3e91a32..000000000
--- a/magic_pdf/config/enums.py
+++ /dev/null
@@ -1,7 +0,0 @@
-
-import enum
-
-
-class SupportedPdfParseMethod(enum.Enum):
- OCR = 'ocr'
- TXT = 'txt'
diff --git a/magic_pdf/config/exceptions.py b/magic_pdf/config/exceptions.py
deleted file mode 100644
index 38f326b51..000000000
--- a/magic_pdf/config/exceptions.py
+++ /dev/null
@@ -1,32 +0,0 @@
-
-class FileNotExisted(Exception):
-
- def __init__(self, path):
- self.path = path
-
- def __str__(self):
- return f'File {self.path} does not exist.'
-
-
-class InvalidConfig(Exception):
- def __init__(self, msg):
- self.msg = msg
-
- def __str__(self):
- return f'Invalid config: {self.msg}'
-
-
-class InvalidParams(Exception):
- def __init__(self, msg):
- self.msg = msg
-
- def __str__(self):
- return f'Invalid params: {self.msg}'
-
-
-class EmptyData(Exception):
- def __init__(self, msg):
- self.msg = msg
-
- def __str__(self):
- return f'Empty data: {self.msg}'
diff --git a/magic_pdf/data/__init__.py b/magic_pdf/data/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/data/data_reader_writer/__init__.py b/magic_pdf/data/data_reader_writer/__init__.py
deleted file mode 100644
index f8f823473..000000000
--- a/magic_pdf/data/data_reader_writer/__init__.py
+++ /dev/null
@@ -1,12 +0,0 @@
-from magic_pdf.data.data_reader_writer.filebase import \
- FileBasedDataReader # noqa: F401
-from magic_pdf.data.data_reader_writer.filebase import \
- FileBasedDataWriter # noqa: F401
-from magic_pdf.data.data_reader_writer.multi_bucket_s3 import \
- MultiBucketS3DataReader # noqa: F401
-from magic_pdf.data.data_reader_writer.multi_bucket_s3 import \
- MultiBucketS3DataWriter # noqa: F401
-from magic_pdf.data.data_reader_writer.s3 import S3DataReader # noqa: F401
-from magic_pdf.data.data_reader_writer.s3 import S3DataWriter # noqa: F401
-from magic_pdf.data.data_reader_writer.base import DataReader # noqa: F401
-from magic_pdf.data.data_reader_writer.base import DataWriter # noqa: F401
\ No newline at end of file
diff --git a/magic_pdf/data/data_reader_writer/base.py b/magic_pdf/data/data_reader_writer/base.py
deleted file mode 100644
index 7c9a8e8ec..000000000
--- a/magic_pdf/data/data_reader_writer/base.py
+++ /dev/null
@@ -1,51 +0,0 @@
-
-from abc import ABC, abstractmethod
-
-
-class DataReader(ABC):
-
- def read(self, path: str) -> bytes:
- """Read the file.
-
- Args:
- path (str): file path to read
-
- Returns:
- bytes: the content of the file
- """
- return self.read_at(path)
-
- @abstractmethod
- def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
- """Read the file at offset and limit.
-
- Args:
- path (str): the file path
- offset (int, optional): the number of bytes skipped. Defaults to 0.
- limit (int, optional): the length of bytes want to read. Defaults to -1.
-
- Returns:
- bytes: the content of the file
- """
- pass
-
-
-class DataWriter(ABC):
- @abstractmethod
- def write(self, path: str, data: bytes) -> None:
- """Write the data to the file.
-
- Args:
- path (str): the target file where to write
- data (bytes): the data want to write
- """
- pass
-
- def write_string(self, path: str, data: str) -> None:
- """Write the data to file, the data will be encoded to bytes.
-
- Args:
- path (str): the target file where to write
- data (str): the data want to write
- """
- self.write(path, data.encode())
diff --git a/magic_pdf/data/data_reader_writer/multi_bucket_s3.py b/magic_pdf/data/data_reader_writer/multi_bucket_s3.py
deleted file mode 100644
index 4f6347b3b..000000000
--- a/magic_pdf/data/data_reader_writer/multi_bucket_s3.py
+++ /dev/null
@@ -1,137 +0,0 @@
-from magic_pdf.config.exceptions import InvalidConfig, InvalidParams
-from magic_pdf.data.data_reader_writer.base import DataReader, DataWriter
-from magic_pdf.data.io.s3 import S3Reader, S3Writer
-from magic_pdf.data.schemas import S3Config
-from magic_pdf.libs.path_utils import (parse_s3_range_params, parse_s3path,
- remove_non_official_s3_args)
-
-
-class MultiS3Mixin:
- def __init__(self, default_bucket: str, s3_configs: list[S3Config]):
- """Initialized with multiple s3 configs.
-
- Args:
- default_bucket (str): the default bucket name of the relative path
- s3_configs (list[S3Config]): list of s3 configs, the bucket_name must be unique in the list.
-
- Raises:
- InvalidConfig: default bucket config not in s3_configs
- InvalidConfig: bucket name not unique in s3_configs
- InvalidConfig: default bucket must be provided
- """
- if len(default_bucket) == 0:
- raise InvalidConfig('default_bucket must be provided')
-
- found_default_bucket_config = False
- for conf in s3_configs:
- if conf.bucket_name == default_bucket:
- found_default_bucket_config = True
- break
-
- if not found_default_bucket_config:
- raise InvalidConfig(
- f'default_bucket: {default_bucket} config must be provided in s3_configs: {s3_configs}'
- )
-
- uniq_bucket = set([conf.bucket_name for conf in s3_configs])
- if len(uniq_bucket) != len(s3_configs):
- raise InvalidConfig(
- f'the bucket_name in s3_configs: {s3_configs} must be unique'
- )
-
- self.default_bucket = default_bucket
- self.s3_configs = s3_configs
- self._s3_clients_h: dict = {}
-
-
-class MultiBucketS3DataReader(DataReader, MultiS3Mixin):
- def read(self, path: str) -> bytes:
- """Read the path from s3, select diffect bucket client for each request
- based on the path, also support range read.
-
- Args:
- path (str): the s3 path of file, the path must be in the format of s3://bucket_name/path?offset,limit
- for example: s3://bucket_name/path?0,100
-
- Returns:
- bytes: the content of s3 file
- """
- may_range_params = parse_s3_range_params(path)
- if may_range_params is None or 2 != len(may_range_params):
- byte_start, byte_len = 0, -1
- else:
- byte_start, byte_len = int(may_range_params[0]), int(may_range_params[1])
- path = remove_non_official_s3_args(path)
- return self.read_at(path, byte_start, byte_len)
-
- def __get_s3_client(self, bucket_name: str):
- if bucket_name not in set([conf.bucket_name for conf in self.s3_configs]):
- raise InvalidParams(
- f'bucket name: {bucket_name} not found in s3_configs: {self.s3_configs}'
- )
- if bucket_name not in self._s3_clients_h:
- conf = next(
- filter(lambda conf: conf.bucket_name == bucket_name, self.s3_configs)
- )
- self._s3_clients_h[bucket_name] = S3Reader(
- bucket_name,
- conf.access_key,
- conf.secret_key,
- conf.endpoint_url,
- conf.addressing_style,
- )
- return self._s3_clients_h[bucket_name]
-
- def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
- """Read the file with offset and limit, select diffect bucket client
- for each request based on the path.
-
- Args:
- path (str): the file path
- offset (int, optional): the number of bytes skipped. Defaults to 0.
- limit (int, optional): the number of bytes want to read. Defaults to -1 which means infinite.
-
- Returns:
- bytes: the file content
- """
- if path.startswith('s3://'):
- bucket_name, path = parse_s3path(path)
- s3_reader = self.__get_s3_client(bucket_name)
- else:
- s3_reader = self.__get_s3_client(self.default_bucket)
- return s3_reader.read_at(path, offset, limit)
-
-
-class MultiBucketS3DataWriter(DataWriter, MultiS3Mixin):
- def __get_s3_client(self, bucket_name: str):
- if bucket_name not in set([conf.bucket_name for conf in self.s3_configs]):
- raise InvalidParams(
- f'bucket name: {bucket_name} not found in s3_configs: {self.s3_configs}'
- )
- if bucket_name not in self._s3_clients_h:
- conf = next(
- filter(lambda conf: conf.bucket_name == bucket_name, self.s3_configs)
- )
- self._s3_clients_h[bucket_name] = S3Writer(
- bucket_name,
- conf.access_key,
- conf.secret_key,
- conf.endpoint_url,
- conf.addressing_style,
- )
- return self._s3_clients_h[bucket_name]
-
- def write(self, path: str, data: bytes) -> None:
- """Write file with data, also select diffect bucket client for each
- request based on the path.
-
- Args:
- path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
- data (bytes): the data want to write
- """
- if path.startswith('s3://'):
- bucket_name, path = parse_s3path(path)
- s3_writer = self.__get_s3_client(bucket_name)
- else:
- s3_writer = self.__get_s3_client(self.default_bucket)
- return s3_writer.write(path, data)
diff --git a/magic_pdf/data/data_reader_writer/s3.py b/magic_pdf/data/data_reader_writer/s3.py
deleted file mode 100644
index b6f27a276..000000000
--- a/magic_pdf/data/data_reader_writer/s3.py
+++ /dev/null
@@ -1,69 +0,0 @@
-from magic_pdf.data.data_reader_writer.multi_bucket_s3 import (
- MultiBucketS3DataReader, MultiBucketS3DataWriter)
-from magic_pdf.data.schemas import S3Config
-
-
-class S3DataReader(MultiBucketS3DataReader):
- def __init__(
- self,
- bucket: str,
- ak: str,
- sk: str,
- endpoint_url: str,
- addressing_style: str = 'auto',
- ):
- """s3 reader client.
-
- Args:
- bucket (str): bucket name
- ak (str): access key
- sk (str): secret key
- endpoint_url (str): endpoint url of s3
- addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
- refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
- """
- super().__init__(
- bucket,
- [
- S3Config(
- bucket_name=bucket,
- access_key=ak,
- secret_key=sk,
- endpoint_url=endpoint_url,
- addressing_style=addressing_style,
- )
- ],
- )
-
-
-class S3DataWriter(MultiBucketS3DataWriter):
- def __init__(
- self,
- bucket: str,
- ak: str,
- sk: str,
- endpoint_url: str,
- addressing_style: str = 'auto',
- ):
- """s3 writer client.
-
- Args:
- bucket (str): bucket name
- ak (str): access key
- sk (str): secret key
- endpoint_url (str): endpoint url of s3
- addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
- refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
- """
- super().__init__(
- bucket,
- [
- S3Config(
- bucket_name=bucket,
- access_key=ak,
- secret_key=sk,
- endpoint_url=endpoint_url,
- addressing_style=addressing_style,
- )
- ],
- )
diff --git a/magic_pdf/data/dataset.py b/magic_pdf/data/dataset.py
deleted file mode 100644
index 0eee3c684..000000000
--- a/magic_pdf/data/dataset.py
+++ /dev/null
@@ -1,194 +0,0 @@
-from abc import ABC, abstractmethod
-from typing import Iterator
-
-import fitz
-
-from magic_pdf.config.enums import SupportedPdfParseMethod
-from magic_pdf.data.schemas import PageInfo
-from magic_pdf.data.utils import fitz_doc_to_image
-
-
-class PageableData(ABC):
- @abstractmethod
- def get_image(self) -> dict:
- """Transform data to image."""
- pass
-
- @abstractmethod
- def get_doc(self) -> fitz.Page:
- """Get the pymudoc page."""
- pass
-
- @abstractmethod
- def get_page_info(self) -> PageInfo:
- """Get the page info of the page.
-
- Returns:
- PageInfo: the page info of this page
- """
- pass
-
-
-class Dataset(ABC):
- @abstractmethod
- def __len__(self) -> int:
- """The length of the dataset."""
- pass
-
- @abstractmethod
- def __iter__(self) -> Iterator[PageableData]:
- """Yield the page data."""
- pass
-
- @abstractmethod
- def supported_methods(self) -> list[SupportedPdfParseMethod]:
- """The methods that this dataset support.
-
- Returns:
- list[SupportedPdfParseMethod]: The supported methods, Valid methods are: OCR, TXT
- """
- pass
-
- @abstractmethod
- def data_bits(self) -> bytes:
- """The bits used to create this dataset."""
- pass
-
- @abstractmethod
- def get_page(self, page_id: int) -> PageableData:
- """Get the page indexed by page_id.
-
- Args:
- page_id (int): the index of the page
-
- Returns:
- PageableData: the page doc object
- """
- pass
-
-
-class PymuDocDataset(Dataset):
- def __init__(self, bits: bytes):
- """Initialize the dataset, which wraps the pymudoc documents.
-
- Args:
- bits (bytes): the bytes of the pdf
- """
- self._records = [Doc(v) for v in fitz.open('pdf', bits)]
- self._data_bits = bits
- self._raw_data = bits
-
- def __len__(self) -> int:
- """The page number of the pdf."""
- return len(self._records)
-
- def __iter__(self) -> Iterator[PageableData]:
- """Yield the page doc object."""
- return iter(self._records)
-
- def supported_methods(self) -> list[SupportedPdfParseMethod]:
- """The method supported by this dataset.
-
- Returns:
- list[SupportedPdfParseMethod]: the supported methods
- """
- return [SupportedPdfParseMethod.OCR, SupportedPdfParseMethod.TXT]
-
- def data_bits(self) -> bytes:
- """The pdf bits used to create this dataset."""
- return self._data_bits
-
- def get_page(self, page_id: int) -> PageableData:
- """The page doc object.
-
- Args:
- page_id (int): the page doc index
-
- Returns:
- PageableData: the page doc object
- """
- return self._records[page_id]
-
-
-class ImageDataset(Dataset):
- def __init__(self, bits: bytes):
- """Initialize the dataset, which wraps the pymudoc documents.
-
- Args:
- bits (bytes): the bytes of the photo which will be converted to pdf first. then converted to pymudoc.
- """
- pdf_bytes = fitz.open(stream=bits).convert_to_pdf()
- self._records = [Doc(v) for v in fitz.open('pdf', pdf_bytes)]
- self._raw_data = bits
- self._data_bits = pdf_bytes
-
- def __len__(self) -> int:
- """The length of the dataset."""
- return len(self._records)
-
- def __iter__(self) -> Iterator[PageableData]:
- """Yield the page object."""
- return iter(self._records)
-
- def supported_methods(self):
- """The method supported by this dataset.
-
- Returns:
- list[SupportedPdfParseMethod]: the supported methods
- """
- return [SupportedPdfParseMethod.OCR]
-
- def data_bits(self) -> bytes:
- """The pdf bits used to create this dataset."""
- return self._data_bits
-
- def get_page(self, page_id: int) -> PageableData:
- """The page doc object.
-
- Args:
- page_id (int): the page doc index
-
- Returns:
- PageableData: the page doc object
- """
- return self._records[page_id]
-
-
-class Doc(PageableData):
- """Initialized with pymudoc object."""
- def __init__(self, doc: fitz.Page):
- self._doc = doc
-
- def get_image(self):
- """Return the imge info.
-
- Returns:
- dict: {
- img: np.ndarray,
- width: int,
- height: int
- }
- """
- return fitz_doc_to_image(self._doc)
-
- def get_doc(self) -> fitz.Page:
- """Get the pymudoc object.
-
- Returns:
- fitz.Page: the pymudoc object
- """
- return self._doc
-
- def get_page_info(self) -> PageInfo:
- """Get the page info of the page.
-
- Returns:
- PageInfo: the page info of this page
- """
- page_w = self._doc.rect.width
- page_h = self._doc.rect.height
- return PageInfo(w=page_w, h=page_h)
-
- def __getattr__(self, name):
- if hasattr(self._doc, name):
- return getattr(self._doc, name)
diff --git a/magic_pdf/data/io/__init__.py b/magic_pdf/data/io/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/data/io/base.py b/magic_pdf/data/io/base.py
deleted file mode 100644
index c02c2ed42..000000000
--- a/magic_pdf/data/io/base.py
+++ /dev/null
@@ -1,42 +0,0 @@
-from abc import ABC, abstractmethod
-
-
-class IOReader(ABC):
- @abstractmethod
- def read(self, path: str) -> bytes:
- """Read the file.
-
- Args:
- path (str): file path to read
-
- Returns:
- bytes: the content of the file
- """
- pass
-
- @abstractmethod
- def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
- """Read at offset and limit.
-
- Args:
- path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
- offset (int, optional): the number of bytes skipped. Defaults to 0.
- limit (int, optional): the length of bytes want to read. Defaults to -1.
-
- Returns:
- bytes: the content of file
- """
- pass
-
-
-class IOWriter:
-
- @abstractmethod
- def write(self, path: str, data: bytes) -> None:
- """Write file with data.
-
- Args:
- path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
- data (bytes): the data want to write
- """
- pass
diff --git a/magic_pdf/data/io/s3.py b/magic_pdf/data/io/s3.py
deleted file mode 100644
index 4222c73fe..000000000
--- a/magic_pdf/data/io/s3.py
+++ /dev/null
@@ -1,114 +0,0 @@
-import boto3
-from botocore.config import Config
-
-from magic_pdf.data.io.base import IOReader, IOWriter
-
-
-class S3Reader(IOReader):
- def __init__(
- self,
- bucket: str,
- ak: str,
- sk: str,
- endpoint_url: str,
- addressing_style: str = 'auto',
- ):
- """s3 reader client.
-
- Args:
- bucket (str): bucket name
- ak (str): access key
- sk (str): secret key
- endpoint_url (str): endpoint url of s3
- addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
- refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
- """
- self._bucket = bucket
- self._ak = ak
- self._sk = sk
- self._s3_client = boto3.client(
- service_name='s3',
- aws_access_key_id=ak,
- aws_secret_access_key=sk,
- endpoint_url=endpoint_url,
- config=Config(
- s3={'addressing_style': addressing_style},
- retries={'max_attempts': 5, 'mode': 'standard'},
- ),
- )
-
- def read(self, key: str) -> bytes:
- """Read the file.
-
- Args:
- path (str): file path to read
-
- Returns:
- bytes: the content of the file
- """
- return self.read_at(key)
-
- def read_at(self, key: str, offset: int = 0, limit: int = -1) -> bytes:
- """Read at offset and limit.
-
- Args:
- path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
- offset (int, optional): the number of bytes skipped. Defaults to 0.
- limit (int, optional): the length of bytes want to read. Defaults to -1.
-
- Returns:
- bytes: the content of file
- """
- if limit > -1:
- range_header = f'bytes={offset}-{offset+limit-1}'
- res = self._s3_client.get_object(
- Bucket=self._bucket, Key=key, Range=range_header
- )
- else:
- res = self._s3_client.get_object(
- Bucket=self._bucket, Key=key, Range=f'bytes={offset}-'
- )
- return res['Body'].read()
-
-
-class S3Writer(IOWriter):
- def __init__(
- self,
- bucket: str,
- ak: str,
- sk: str,
- endpoint_url: str,
- addressing_style: str = 'auto',
- ):
- """s3 reader client.
-
- Args:
- bucket (str): bucket name
- ak (str): access key
- sk (str): secret key
- endpoint_url (str): endpoint url of s3
- addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
- refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
- """
- self._bucket = bucket
- self._ak = ak
- self._sk = sk
- self._s3_client = boto3.client(
- service_name='s3',
- aws_access_key_id=ak,
- aws_secret_access_key=sk,
- endpoint_url=endpoint_url,
- config=Config(
- s3={'addressing_style': addressing_style},
- retries={'max_attempts': 5, 'mode': 'standard'},
- ),
- )
-
- def write(self, key: str, data: bytes):
- """Write file with data.
-
- Args:
- path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
- data (bytes): the data want to write
- """
- self._s3_client.put_object(Bucket=self._bucket, Key=key, Body=data)
diff --git a/magic_pdf/data/read_api.py b/magic_pdf/data/read_api.py
deleted file mode 100644
index 61e0fbf72..000000000
--- a/magic_pdf/data/read_api.py
+++ /dev/null
@@ -1,95 +0,0 @@
-import json
-import os
-from pathlib import Path
-
-from magic_pdf.config.exceptions import EmptyData, InvalidParams
-from magic_pdf.data.data_reader_writer import (FileBasedDataReader,
- MultiBucketS3DataReader)
-from magic_pdf.data.dataset import ImageDataset, PymuDocDataset
-
-
-def read_jsonl(
- s3_path_or_local: str, s3_client: MultiBucketS3DataReader | None = None
-) -> list[PymuDocDataset]:
- """Read the jsonl file and return the list of PymuDocDataset.
-
- Args:
- s3_path_or_local (str): local file or s3 path
- s3_client (MultiBucketS3DataReader | None, optional): s3 client that support multiple bucket. Defaults to None.
-
- Raises:
- InvalidParams: if s3_path_or_local is s3 path but s3_client is not provided.
- EmptyData: if no pdf file location is provided in some line of jsonl file.
- InvalidParams: if the file location is s3 path but s3_client is not provided
-
- Returns:
- list[PymuDocDataset]: each line in the jsonl file will be converted to a PymuDocDataset
- """
- bits_arr = []
- if s3_path_or_local.startswith('s3://'):
- if s3_client is None:
- raise InvalidParams('s3_client is required when s3_path is provided')
- jsonl_bits = s3_client.read(s3_path_or_local)
- else:
- jsonl_bits = FileBasedDataReader('').read(s3_path_or_local)
- jsonl_d = [
- json.loads(line) for line in jsonl_bits.decode().split('\n') if line.strip()
- ]
- for d in jsonl_d[:5]:
- pdf_path = d.get('file_location', '') or d.get('path', '')
- if len(pdf_path) == 0:
- raise EmptyData('pdf file location is empty')
- if pdf_path.startswith('s3://'):
- if s3_client is None:
- raise InvalidParams('s3_client is required when s3_path is provided')
- bits_arr.append(s3_client.read(pdf_path))
- else:
- bits_arr.append(FileBasedDataReader('').read(pdf_path))
- return [PymuDocDataset(bits) for bits in bits_arr]
-
-
-def read_local_pdfs(path: str) -> list[PymuDocDataset]:
- """Read pdf from path or directory.
-
- Args:
- path (str): pdf file path or directory that contains pdf files
-
- Returns:
- list[PymuDocDataset]: each pdf file will converted to a PymuDocDataset
- """
- if os.path.isdir(path):
- reader = FileBasedDataReader(path)
- return [
- PymuDocDataset(reader.read(doc_path.name))
- for doc_path in Path(path).glob('*.pdf')
- ]
- else:
- reader = FileBasedDataReader()
- bits = reader.read(path)
- return [PymuDocDataset(bits)]
-
-
-def read_local_images(path: str, suffixes: list[str]) -> list[ImageDataset]:
- """Read images from path or directory.
-
- Args:
- path (str): image file path or directory that contains image files
- suffixes (list[str]): the suffixes of the image files used to filter the files. Example: ['jpg', 'png']
-
- Returns:
- list[ImageDataset]: each image file will converted to a ImageDataset
- """
- if os.path.isdir(path):
- imgs_bits = []
- s_suffixes = set(suffixes)
- reader = FileBasedDataReader(path)
- for root, _, files in os.walk(path):
- for file in files:
- suffix = file.split('.')
- if suffix[-1] in s_suffixes:
- imgs_bits.append(reader.read(file))
- return [ImageDataset(bits) for bits in imgs_bits]
- else:
- reader = FileBasedDataReader()
- bits = reader.read(path)
- return [ImageDataset(bits)]
diff --git a/magic_pdf/data/utils.py b/magic_pdf/data/utils.py
deleted file mode 100644
index dfe7abde1..000000000
--- a/magic_pdf/data/utils.py
+++ /dev/null
@@ -1,32 +0,0 @@
-
-import fitz
-import numpy as np
-
-from magic_pdf.utils.annotations import ImportPIL
-
-
-@ImportPIL
-def fitz_doc_to_image(doc, dpi=200) -> dict:
- """Convert fitz.Document to image, Then convert the image to numpy array.
-
- Args:
- doc (_type_): pymudoc page
- dpi (int, optional): reset the dpi of dpi. Defaults to 200.
-
- Returns:
- dict: {'img': numpy array, 'width': width, 'height': height }
- """
- from PIL import Image
- mat = fitz.Matrix(dpi / 72, dpi / 72)
- pm = doc.get_pixmap(matrix=mat, alpha=False)
-
- # If the width or height exceeds 9000 after scaling, do not scale further.
- if pm.width > 9000 or pm.height > 9000:
- pm = doc.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False)
-
- img = Image.frombytes('RGB', (pm.width, pm.height), pm.samples)
- img = np.array(img)
-
- img_dict = {'img': img, 'width': pm.width, 'height': pm.height}
-
- return img_dict
diff --git a/magic_pdf/dict2md/__init__.py b/magic_pdf/dict2md/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/dict2md/mkcontent.py b/magic_pdf/dict2md/mkcontent.py
deleted file mode 100644
index 049e290e5..000000000
--- a/magic_pdf/dict2md/mkcontent.py
+++ /dev/null
@@ -1,397 +0,0 @@
-import math
-from loguru import logger
-
-from magic_pdf.libs.boxbase import find_bottom_nearest_text_bbox, find_top_nearest_text_bbox
-from magic_pdf.libs.commons import join_path
-from magic_pdf.libs.ocr_content_type import ContentType
-
-TYPE_INLINE_EQUATION = ContentType.InlineEquation
-TYPE_INTERLINE_EQUATION = ContentType.InterlineEquation
-UNI_FORMAT_TEXT_TYPE = ['text', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6']
-
-
-@DeprecationWarning
-def mk_nlp_markdown_1(para_dict: dict):
- """
- 对排序后的bboxes拼接内容
- """
- content_lst = []
- for _, page_info in para_dict.items():
- para_blocks = page_info.get("para_blocks")
- if not para_blocks:
- continue
-
- for block in para_blocks:
- item = block["paras"]
- for _, p in item.items():
- para_text = p["para_text"]
- is_title = p["is_para_title"]
- title_level = p['para_title_level']
- md_title_prefix = "#"*title_level
- if is_title:
- content_lst.append(f"{md_title_prefix} {para_text}")
- else:
- content_lst.append(para_text)
-
- content_text = "\n\n".join(content_lst)
-
- return content_text
-
-
-
-# 找到目标字符串在段落中的索引
-def __find_index(paragraph, target):
- index = paragraph.find(target)
- if index != -1:
- return index
- else:
- return None
-
-
-def __insert_string(paragraph, target, postion):
- new_paragraph = paragraph[:postion] + target + paragraph[postion:]
- return new_paragraph
-
-
-def __insert_after(content, image_content, target):
- """
- 在content中找到target，将image_content插入到target后面
- """
- index = content.find(target)
- if index != -1:
- content = content[:index+len(target)] + "\n\n" + image_content + "\n\n" + content[index+len(target):]
- else:
- logger.error(f"Can't find the location of image {image_content} in the markdown file, search target is {target}")
- return content
-
-def __insert_before(content, image_content, target):
- """
- 在content中找到target，将image_content插入到target前面
- """
- index = content.find(target)
- if index != -1:
- content = content[:index] + "\n\n" + image_content + "\n\n" + content[index:]
- else:
- logger.error(f"Can't find the location of image {image_content} in the markdown file, search target is {target}")
- return content
-
-
-@DeprecationWarning
-def mk_mm_markdown_1(para_dict: dict):
- """拼装多模态markdown"""
- content_lst = []
- for _, page_info in para_dict.items():
- page_lst = [] # 一个page内的段落列表
- para_blocks = page_info.get("para_blocks")
- pymu_raw_blocks = page_info.get("preproc_blocks")
-
- all_page_images = []
- all_page_images.extend(page_info.get("images",[]))
- all_page_images.extend(page_info.get("image_backup", []) )
- all_page_images.extend(page_info.get("tables",[]))
- all_page_images.extend(page_info.get("table_backup",[]) )
-
- if not para_blocks or not pymu_raw_blocks: # 只有图片的拼接的场景
- for img in all_page_images:
- page_lst.append(f"") # TODO 图片顺序
- page_md = "\n\n".join(page_lst)
-
- else:
- for block in para_blocks:
- item = block["paras"]
- for _, p in item.items():
- para_text = p["para_text"]
- is_title = p["is_para_title"]
- title_level = p['para_title_level']
- md_title_prefix = "#"*title_level
- if is_title:
- page_lst.append(f"{md_title_prefix} {para_text}")
- else:
- page_lst.append(para_text)
-
- """拼装成一个页面的文本"""
- page_md = "\n\n".join(page_lst)
- """插入图片"""
- for img in all_page_images:
- imgbox = img['bbox']
- img_content = f""
- # 先看在哪个block内
- for block in pymu_raw_blocks:
- bbox = block['bbox']
- if bbox[0]-1 <= imgbox[0] < bbox[2]+1 and bbox[1]-1 <= imgbox[1] < bbox[3]+1:# 确定在block内
- for l in block['lines']:
- line_box = l['bbox']
- if line_box[0]-1 <= imgbox[0] < line_box[2]+1 and line_box[1]-1 <= imgbox[1] < line_box[3]+1: # 在line内的，插入line前面
- line_txt = "".join([s['text'] for s in l['spans']])
- page_md = __insert_before(page_md, img_content, line_txt)
- break
- break
- else:# 在行与行之间
- # 找到图片x0,y0与line的x0,y0最近的line
- min_distance = 100000
- min_line = None
- for l in block['lines']:
- line_box = l['bbox']
- distance = math.sqrt((line_box[0] - imgbox[0])**2 + (line_box[1] - imgbox[1])**2)
- if distance < min_distance:
- min_distance = distance
- min_line = l
- if min_line:
- line_txt = "".join([s['text'] for s in min_line['spans']])
- img_h = imgbox[3] - imgbox[1]
- if min_distance 0:
- if en_length / len(text) >= 0.5:
- return 'en'
- else:
- return 'unknown'
- else:
- return 'empty'
-
-
-def merge_para_with_text(para_block):
- para_text = ''
- for i, line in enumerate(para_block['lines']):
-
- if i >= 1 and line.get(ListLineTag.IS_LIST_START_LINE, False):
- para_text += ' \n'
-
- line_text = ''
- line_lang = ''
- for span in line['spans']:
- span_type = span['type']
- if span_type == ContentType.Text:
- line_text += span['content'].strip()
- if line_text != '':
- line_lang = detect_lang(line_text)
- for span in line['spans']:
-
- span_type = span['type']
- content = ''
- if span_type == ContentType.Text:
- content = ocr_escape_special_markdown_char(span['content'])
- elif span_type == ContentType.InlineEquation:
- content = f" ${span['content']}$ "
- elif span_type == ContentType.InterlineEquation:
- content = f"\n$$\n{span['content']}\n$$\n"
-
- if content != '':
- langs = ['zh', 'ja', 'ko']
- if line_lang in langs: # 遇到一些一个字一个span的文档，这种单字语言判断不准，需要用整行文本判断
- para_text += content # 中文/日语/韩文语境下，content间不需要空格分隔
- elif line_lang == 'en':
- # 如果是前一行带有-连字符，那么末尾不应该加空格
- if __is_hyphen_at_line_end(content):
- para_text += content[:-1]
- else:
- para_text += content + ' '
- else:
- para_text += content + ' ' # 西方文本语境下 content间需要空格分隔
- return para_text
-
-
-def para_to_standard_format_v2(para_block, img_buket_path, page_idx, drop_reason=None):
- para_type = para_block['type']
- para_content = {}
- if para_type in [BlockType.Text, BlockType.List, BlockType.Index]:
- para_content = {
- 'type': 'text',
- 'text': merge_para_with_text(para_block),
- }
- elif para_type == BlockType.Title:
- para_content = {
- 'type': 'text',
- 'text': merge_para_with_text(para_block),
- 'text_level': 1,
- }
- elif para_type == BlockType.InterlineEquation:
- para_content = {
- 'type': 'equation',
- 'text': merge_para_with_text(para_block),
- 'text_format': 'latex',
- }
- elif para_type == BlockType.Image:
- para_content = {'type': 'image', 'img_caption': [], 'img_footnote': []}
- for block in para_block['blocks']:
- if block['type'] == BlockType.ImageBody:
- para_content['img_path'] = join_path(
- img_buket_path,
- block['lines'][0]['spans'][0]['image_path'])
- if block['type'] == BlockType.ImageCaption:
- para_content['img_caption'].append(merge_para_with_text(block))
- if block['type'] == BlockType.ImageFootnote:
- para_content['img_footnote'].append(merge_para_with_text(block))
- elif para_type == BlockType.Table:
- para_content = {'type': 'table', 'table_caption': [], 'table_footnote': []}
- for block in para_block['blocks']:
- if block['type'] == BlockType.TableBody:
- if block["lines"][0]["spans"][0].get('latex', ''):
- para_content['table_body'] = f"\n\n$\n {block['lines'][0]['spans'][0]['latex']}\n$\n\n"
- elif block["lines"][0]["spans"][0].get('html', ''):
- para_content['table_body'] = f"\n\n{block['lines'][0]['spans'][0]['html']}\n\n"
- para_content['img_path'] = join_path(img_buket_path, block["lines"][0]["spans"][0]['image_path'])
- if block['type'] == BlockType.TableCaption:
- para_content['table_caption'].append(merge_para_with_text(block))
- if block['type'] == BlockType.TableFootnote:
- para_content['table_footnote'].append(merge_para_with_text(block))
-
- para_content['page_idx'] = page_idx
-
- if drop_reason is not None:
- para_content['drop_reason'] = drop_reason
-
- return para_content
-
-
-def union_make(pdf_info_dict: list,
- make_mode: str,
- drop_mode: str,
- img_buket_path: str = '',
- ):
- output_content = []
- for page_info in pdf_info_dict:
- drop_reason_flag = False
- drop_reason = None
- if page_info.get('need_drop', False):
- drop_reason = page_info.get('drop_reason')
- if drop_mode == DropMode.NONE:
- pass
- elif drop_mode == DropMode.NONE_WITH_REASON:
- drop_reason_flag = True
- elif drop_mode == DropMode.WHOLE_PDF:
- raise Exception((f'drop_mode is {DropMode.WHOLE_PDF} ,'
- f'drop_reason is {drop_reason}'))
- elif drop_mode == DropMode.SINGLE_PAGE:
- logger.warning((f'drop_mode is {DropMode.SINGLE_PAGE} ,'
- f'drop_reason is {drop_reason}'))
- continue
- else:
- raise Exception('drop_mode can not be null')
-
- paras_of_layout = page_info.get('para_blocks')
- page_idx = page_info.get('page_idx')
- if not paras_of_layout:
- continue
- if make_mode == MakeMode.MM_MD:
- page_markdown = ocr_mk_markdown_with_para_core_v2(
- paras_of_layout, 'mm', img_buket_path)
- output_content.extend(page_markdown)
- elif make_mode == MakeMode.NLP_MD:
- page_markdown = ocr_mk_markdown_with_para_core_v2(
- paras_of_layout, 'nlp')
- output_content.extend(page_markdown)
- elif make_mode == MakeMode.STANDARD_FORMAT:
- for para_block in paras_of_layout:
- if drop_reason_flag:
- para_content = para_to_standard_format_v2(
- para_block, img_buket_path, page_idx)
- else:
- para_content = para_to_standard_format_v2(
- para_block, img_buket_path, page_idx)
- output_content.append(para_content)
- if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
- return '\n\n'.join(output_content)
- elif make_mode == MakeMode.STANDARD_FORMAT:
- return output_content
diff --git a/magic_pdf/filter/__init__.py b/magic_pdf/filter/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/filter/pdf_classify_by_type.py b/magic_pdf/filter/pdf_classify_by_type.py
deleted file mode 100644
index 319fb3fb2..000000000
--- a/magic_pdf/filter/pdf_classify_by_type.py
+++ /dev/null
@@ -1,393 +0,0 @@
-"""
-根据利用meta_scan得到的结果，对pdf是否为文字版进行分类。
-定义标准：
-一、什么pdf会是文字pdf，只要满足以下任意一条
- 1. 随机抽取N页，如果有任何一页文字数目大于100
- 2. 只要存在一个页面，图片的数量为0
-二、什么是扫描版pdf，只要满足以下任意一条
- 1. ~~80%页面上的最大图大小一样并且面积超过页面面积0.6~~
- 2. 大部分页面上文字的长度都是相等的。
-
-"""
-import json
-import sys
-from collections import Counter
-
-import click
-import numpy as np
-from loguru import logger
-
-from magic_pdf.libs.commons import mymax, get_top_percent_list
-from magic_pdf.filter.pdf_meta_scan import scan_max_page, junk_limit_min
-
-TEXT_LEN_THRESHOLD = 100
-AVG_TEXT_LEN_THRESHOLD = 100
-TEXT_LEN_SAMPLE_RATIO = 0.1 # 抽取0.1的页面进行文字长度统计
-
-
-# 一个拼接图片的方案，将某些特殊扫描版本的拆图拼成一张整图
-def merge_images(image_list, page_width, page_height, max_offset=5, max_gap=2):
- # 先通过set去除所有bbox重叠的图片数据
- image_list_result = []
- for page_images in image_list:
- page_result = []
- dedup = set()
- for img in page_images:
- x0, y0, x1, y1, img_bojid = img
- if (x0, y0, x1, y1) in dedup: # 这里面会出现一些重复的bbox，无需重复出现，需要去掉
- continue
- else:
- dedup.add((x0, y0, x1, y1))
- page_result.append([x0, y0, x1, y1, img_bojid])
- image_list_result.append(page_result)
-
- # 接下来，将同一页可拼接的图片进行合并
- merged_images = []
- for page_images in image_list_result:
- if not page_images:
- continue
-
- # 先将同一页的图片从上到下，从左到右进行排序
- page_images.sort(key=lambda img: (img[1], img[0]))
-
- merged = [page_images[0]]
-
- for img in page_images[1:]:
- x0, y0, x1, y1, imgid = img
-
- last_img = merged[-1]
- last_x0, last_y0, last_x1, last_y1, last_imgid = last_img
-
- # 单张图片宽或者高覆盖页面宽高的9成以上是拼图的一个前置条件
- full_width = abs(x1 - x0) >= page_width * 0.9
- full_height = abs(y1 - y0) >= page_height * 0.9
-
- # 如果宽达标，检测是否能竖着拼
- if full_width:
- # 竖着拼需要满足两个前提，左右边界各偏移不能超过 max_offset，第一张图的下边界和第二张图的上边界偏移不能超过 max_gap
- close1 = (last_x0 - max_offset) <= x0 <= (last_x0 + max_offset) and (last_x1 - max_offset) <= x1 <= (
- last_x1 + max_offset) and (last_y1 - max_gap) <= y0 <= (last_y1 + max_gap)
-
- # 如果高达标，检测是否可以横着拼
- if full_height:
- # 横着拼需要满足两个前提，上下边界各偏移不能超过 max_offset，第一张图的右边界和第二张图的左边界偏移不能超过 max_gap
- close2 = (last_y0 - max_offset) <= y0 <= (last_y0 + max_offset) and (last_y1 - max_offset) <= y1 <= (
- last_y1 + max_offset) and (last_x1 - max_gap) <= x0 <= (last_x1 + max_gap)
-
- # Check if the image can be merged with the last image
- if (full_width and close1) or (full_height and close2):
- # Merge the image with the last image
- merged[-1] = [min(x0, last_x0), min(y0, last_y0),
- max(x1, last_x1), max(y1, last_y1), imgid]
- else:
- # Add the image as a new image
- merged.append(img)
-
- merged_images.append(merged)
-
- return merged_images
-
-
-def classify_by_area(total_page: int, page_width, page_height, img_sz_list, text_len_list: list):
- """
- 80%页面上的最大图大小一样并且面积超过页面面积0.6则返回False，否则返回True
- :param pdf_path:
- :param total_page:
- :param page_width:
- :param page_height:
- :param img_sz_list:
- :return:
- """
- # # 只要有一页没有图片，那么就是文字pdf。但是同时还需要满足一个条件就是这个页面上同时不能有文字。发现过一些扫描版pdf，上面有一些空白页面，既没有图片也没有文字。
- # if any([len(img_sz) == 0 for img_sz in img_sz_list]): # 含有不含图片的页面
- # # 现在找到这些页面的index
- # empty_page_index = [i for i, img_sz in enumerate(img_sz_list) if len(img_sz) == 0]
- # # 然后检查这些页面上是否有文字
- # text_len_at_page_idx = [text_len for i, text_len in enumerate(text_len_list) if i in empty_page_index and text_len > 0]
- # if len(text_len_at_page_idx) > TEXT_LEN_THRESHOLD: # 没有图片，但是有文字，说明可能是个文字版，如果没有文字则无法判断，留给下一步,现在要求这页文字量超过一定阈值
- # return True
-
- # 通过objid去掉重复出现10次以上的图片，这些图片是隐藏的透明图层，其特点是id都一样
- # 先对每个id出现的次数做个统计
- objid_cnt = Counter([objid for page_img_sz in img_sz_list for _, _, _, _, objid in page_img_sz])
- # 再去掉出现次数大于10的
- if total_page >= scan_max_page: # 新的meta_scan只扫描前 scan_max_page 页，页数大于 scan_max_page 当total_page为 scan_max_page
- total_page = scan_max_page
-
- repeat_threshold = 2 # 把bad_image的阈值设为2
- # repeat_threshold = min(2, total_page) # 当total_page为1时，repeat_threshold为1，会产生误判导致所有img变成bad_img
- bad_image_objid = set([objid for objid, cnt in objid_cnt.items() if cnt >= repeat_threshold])
- # bad_image_page_idx = [i for i, page_img_sz in enumerate(img_sz_list) if any([objid in bad_image_objid for _, _, _, _, objid in page_img_sz])]
- # text_len_at_bad_image_page_idx = [text_len for i, text_len in enumerate(text_len_list) if i in bad_image_page_idx and text_len > 0]
-
- # 特殊情况，一个文字版pdf，每页覆盖一个超大的透明图片,超大的定义是图片占整页面积的90%以上
- # fake_image_ids = [objid for objid in bad_image_objid if
- # any([abs((x1 - x0) * (y1 - y0) / page_width * page_height) > 0.9 for images in img_sz_list for
- # x0, y0, x1, y1, _ in images])] # 原来的代码，any里面恒为true了，原因？？？
- # fake_image_ids = [objid for objid in bad_image_objid for images in img_sz_list for x0, y0, x1, y1, img_id in images
- # if img_id == objid and abs((x1 - x0) * (y1 - y0)) / (page_width * page_height) > 0.9]
-
- # if len(fake_image_ids) > 0 and any([l > TEXT_LEN_THRESHOLD for l in text_len_at_bad_image_page_idx]): # 这些透明图片所在的页面上有文字大于阈值
- # return True
-
- img_sz_list = [[img_sz for img_sz in page_img_sz if img_sz[-1] not in bad_image_objid] for page_img_sz in
- img_sz_list] # 过滤掉重复出现的图片
-
- # 有的扫描版会把一页图片拆成很多张，需要先把图拼起来再计算
- img_sz_list = merge_images(img_sz_list, page_width, page_height)
-
- # 计算每个页面上最大的图的面积，然后计算这个面积占页面面积的比例
- max_image_area_per_page = [mymax([(x1 - x0) * (y1 - y0) for x0, y0, x1, y1, _ in page_img_sz]) for page_img_sz in
- img_sz_list]
- page_area = page_width * page_height
- max_image_area_per_page = [area / page_area for area in max_image_area_per_page]
- max_image_area_per_page = [area for area in max_image_area_per_page if area > 0.5]
-
- if len(max_image_area_per_page) >= 0.5 * total_page: # 阈值从0.8改到0.5，适配3页里面有两页和两页里面有一页的情况
- # 这里条件成立的前提是把反复出现的图片去掉了。这些图片是隐藏的透明图层，其特点是id都一样
- return False
- else:
- return True
-
-
-def classify_by_text_len(text_len_list: list, total_page: int):
- """
- 随机抽取10%的页面，如果少于5个页面，那么就取全部页面。
- 查看页面上的文字长度，如果有任何一个页面的文字长度大于TEXT_LEN_THRESHOLD，那么就是文字pdf
- :param total_page:
- :param text_len_list:
- :return:
- """
- select_page_cnt = int(total_page * TEXT_LEN_SAMPLE_RATIO) # 选取10%的页面
- if select_page_cnt < 5:
- select_page_cnt = total_page
-
- # # 排除头尾各10页
- # if total_page > 20: # 如果总页数大于20
- # page_range = list(range(10, total_page - 10)) # 从第11页到倒数第11页
- # else:
- # page_range = list(range(total_page)) # 否则选择所有页面
- # page_num = np.random.choice(page_range, min(select_page_cnt, len(page_range)), replace=False)
- # 排除前后10页对只有21，22页的pdf很尴尬，如果选出来的中间那一两页恰好没字容易误判，有了avg_words规则，这个规则可以忽略
- page_num = np.random.choice(total_page, select_page_cnt, replace=False)
- text_len_lst = [text_len_list[i] for i in page_num]
- is_text_pdf = any([text_len > TEXT_LEN_THRESHOLD for text_len in text_len_lst])
- return is_text_pdf
-
-
-def classify_by_avg_words(text_len_list: list):
- """
- 补充规则，如果平均每页字数少于 AVG_TEXT_LEN_THRESHOLD，就不是文字pdf
- 主要是各种图集
- :param text_len_list:
- :return:
- """
- sum_words = sum(text_len_list)
- count_of_numbers = len(text_len_list)
- if count_of_numbers == 0:
- is_text_pdf = False
- else:
- avg_words = round(sum_words / count_of_numbers)
- if avg_words > AVG_TEXT_LEN_THRESHOLD:
- is_text_pdf = True
- else:
- is_text_pdf = False
-
- return is_text_pdf
-
-
-def classify_by_img_num(img_sz_list: list, img_num_list: list):
- """
- 补充规则，有一种扫描版本的PDF，每一页都会放所有的扫描页进去，在 metascan 时会被去重，
- 这种pdf的 metasca 扫描结果的特点是 img_sz_list 内全是空元素，img_num_list中每一页的数量都很大且相同
- :param img_sz_list:
- :param img_num_list:
- :return:
- """
- # 计算img_sz_list中非空元素的个数
- count_img_sz_list_not_none = sum(1 for item in img_sz_list if item)
- # 获取前80%的元素
- top_eighty_percent = get_top_percent_list(img_num_list, 0.8)
- # img_sz_list中非空元素的个数小于1，前80%的元素都相等，且最大值大于等于junk_limit_min
- if count_img_sz_list_not_none <= 1 and len(set(top_eighty_percent)) == 1 and max(img_num_list) >= junk_limit_min:
-
- #拿max和min的值,用来判断list内的值是否全都相等
- # min_imgs = min(img_num_list)
- # max_imgs = max(img_num_list)
- #
- # if count_img_sz_list_not_none == 0 and max_imgs == min_imgs and max_imgs >= junk_limit_min:
- return False # 如果满足这个条件，一定不是文字版pdf
- else:
- return True # 不满足这三个条件，可能是文字版pdf，通过其他规则判断
-
-
-def classify_by_text_layout(text_layout_per_page: list):
- """
- 判断文本布局是否以竖排为主。
-
- Args:
- text_layout_per_page (list): 文本布局列表，列表中的每个元素表示一页的文本布局，
- 值为'vertical'表示竖排，值为'horizontal'表示横排。
-
- Returns:
- bool: 若文本布局以竖排为主，则返回False；否则返回True。
- """
- # 统计text_layout_per_page中竖排的个数
- count_vertical = sum(1 for item in text_layout_per_page if item == 'vertical')
- # 统计text_layout_per_page中横排的个数
- count_horizontal = sum(1 for item in text_layout_per_page if item == 'horizontal')
- # 计算text_layout_per_page中竖排的占比
- known_layout_cnt = count_vertical + count_horizontal
- if known_layout_cnt != 0:
- ratio = count_vertical / known_layout_cnt
- if ratio >= 0.5: # 阈值设为0.5，适配3页里面有2页和两页里有一页的情况
- return False # 文本布局以竖排为主，认为不是文字版pdf
- else:
- return True # 文本布局以横排为主，认为是文字版pdf
- else:
- return False # 文本布局未知，默认认为不是文字版pdf
-
-
-def classify_by_img_narrow_strips(page_width, page_height, img_sz_list):
- """
- 判断一页是否由细长条组成，有两个条件：
- 1. 图片的宽或高达到页面宽或高的90%，且长边需要是窄边长度的数倍以上
- 2. 整个页面所有的图片有80%以上满足条件1
-
- Args:
- page_width (float): 页面宽度
- page_height (float): 页面高度
- img_sz_list (list): 图片尺寸列表，每个元素为一个元组，表示图片的矩形区域和尺寸，形如(x0, y0, x1, y1, size)，其中(x0, y0)为矩形区域的左上角坐标，(x1, y1)为矩形区域的右下角坐标，size为图片的尺寸
-
- Returns:
- bool: 如果满足条件的页面的比例小于0.5，返回True，否则返回False
- """
-
- def is_narrow_strip(img):
- x0, y0, x1, y1, _ = img
- width, height = x1 - x0, y1 - y0
- return any([
- # 图片宽度大于等于页面宽度的90%，且宽度大于等于高度4倍
- width >= page_width * 0.9 and width >= height * 4,
- # 图片高度大于等于页面高度的90%，且高度大于等于宽度4倍
- height >= page_height * 0.9 and height >= width * 4,
- ])
-
- # 初始化满足条件的页面数量
- narrow_strip_pages_count = 0
-
- # 遍历所有页面
- for page_img_list in img_sz_list:
- # 忽略空页面
- if not page_img_list:
- continue
-
- # 计算页面中的图片总数
- total_images = len(page_img_list)
-
- # 计算页面中细长条图片的数量
- narrow_strip_images_count = 0
- for img in page_img_list:
- if is_narrow_strip(img):
- narrow_strip_images_count += 1
- # 如果细长条图片的数量少于5，跳过
- if narrow_strip_images_count < 5:
- continue
- else:
- # 如果细长条图片的比例大于或等于0.8，增加满足条件的页面数量
- if narrow_strip_images_count / total_images >= 0.8:
- narrow_strip_pages_count += 1
-
- # 计算满足条件的页面的比例
- narrow_strip_pages_ratio = narrow_strip_pages_count / len(img_sz_list)
-
- return narrow_strip_pages_ratio < 0.5
-
-
-def classify(total_page: int, page_width, page_height, img_sz_list: list, text_len_list: list, img_num_list: list,
- text_layout_list: list, invalid_chars: bool):
- """
- 这里的图片和页面长度单位是pts
- :param total_page:
- :param text_len_list:
- :param page_width:
- :param page_height:
- :param img_sz_list:
- :param pdf_path:
- :return:
- """
- results = {
- 'by_image_area': classify_by_area(total_page, page_width, page_height, img_sz_list, text_len_list),
- 'by_text_len': classify_by_text_len(text_len_list, total_page),
- 'by_avg_words': classify_by_avg_words(text_len_list),
- 'by_img_num': classify_by_img_num(img_sz_list, img_num_list),
- 'by_text_layout': classify_by_text_layout(text_layout_list),
- 'by_img_narrow_strips': classify_by_img_narrow_strips(page_width, page_height, img_sz_list),
- 'by_invalid_chars': invalid_chars,
- }
-
- if all(results.values()):
- return True, results
- elif not any(results.values()):
- return False, results
- else:
- logger.warning(
- f"pdf is not classified by area and text_len, by_image_area: {results['by_image_area']},"
- f" by_text: {results['by_text_len']}, by_avg_words: {results['by_avg_words']}, by_img_num: {results['by_img_num']},"
- f" by_text_layout: {results['by_text_layout']}, by_img_narrow_strips: {results['by_img_narrow_strips']},"
- f" by_invalid_chars: {results['by_invalid_chars']}",
- file=sys.stderr) # 利用这种情况可以快速找出来哪些pdf比较特殊，针对性修正分类算法
- return False, results
-
-
-@click.command()
-@click.option("--json-file", type=str, help="pdf信息")
-def main(json_file):
- if json_file is None:
- print("json_file is None", file=sys.stderr)
- exit(0)
- try:
- with open(json_file, "r") as f:
- for l in f:
- if l.strip() == "":
- continue
- o = json.loads(l)
- total_page = o["total_page"]
- page_width = o["page_width_pts"]
- page_height = o["page_height_pts"]
- img_sz_list = o["image_info_per_page"]
- text_len_list = o['text_len_per_page']
- text_layout_list = o['text_layout_per_page']
- pdf_path = o['pdf_path']
- is_encrypted = o['is_encrypted']
- is_needs_password = o['is_needs_password']
- if is_encrypted or total_page == 0 or is_needs_password: # 加密的，需要密码的，没有页面的，都不处理
- continue
- tag = classify(total_page, page_width, page_height, img_sz_list, text_len_list, text_layout_list)
- o['is_text_pdf'] = tag
- print(json.dumps(o, ensure_ascii=False))
- except Exception as e:
- print("ERROR: ", e, file=sys.stderr)
-
-
-if __name__ == "__main__":
- main()
- # false = False
- # true = True
- # null = None
- # o = {"pdf_path":"s3://llm-raw-snew/llm-raw-the-eye/raw/World%20Tracker%20Library/worldtracker.org/media/library/Science/Computer%20Science/Shreiner%20-%20OpenGL%20Programming%20Guide%206e%20%5BThe%20Redbook%5D%20%28AW%2C%202008%29.pdf","is_needs_password":false,"is_encrypted":false,"total_page":978,"page_width_pts":368,"page_height_pts":513,"image_info_per_page":[[[0,0,368,513,10037]],[[0,0,368,513,4]],[[0,0,368,513,7]],[[0,0,368,513,10]],[[0,0,368,513,13]],[[0,0,368,513,16]],[[0,0,368,513,19]],[[0,0,368,513,22]],[[0,0,368,513,25]],[[0,0,368,513,28]],[[0,0,368,513,31]],[[0,0,368,513,34]],[[0,0,368,513,37]],[[0,0,368,513,40]],[[0,0,368,513,43]],[[0,0,368,513,46]],[[0,0,368,513,49]],[[0,0,368,513,52]],[[0,0,368,513,55]],[[0,0,368,513,58]],[[0,0,368,513,61]],[[0,0,368,513,64]],[[0,0,368,513,67]],[[0,0,368,513,70]],[[0,0,368,513,73]],[[0,0,368,516,76]],[[0,0,368,516,79]],[[0,0,368,513,82]],[[0,0,368,513,85]],[[0,0,368,513,88]],[[0,0,368,513,91]],[[0,0,368,513,94]],[[0,0,368,513,97]],[[0,0,368,513,100]],[[0,0,368,513,103]],[[0,0,368,513,106]],[[0,0,368,513,109]],[[0,0,368,513,112]],[[0,0,368,513,115]],[[0,0,368,513,118]],[[0,0,368,513,121]],[[0,0,368,513,124]],[[0,0,368,513,127]],[[0,0,368,513,130]],[[0,0,368,513,133]],[[0,0,368,513,136]],[[0,0,368,513,139]],[[0,0,368,513,142]],[[0,0,368,513,145]],[[0,0,368,513,148]],[[0,0,368,513,151]],[[0,0,368,513,154]],[[0,0,368,513,157]],[[0,0,368,513,160]],[[0,0,368,513,163]],[[0,0,368,513,166]],[[0,0,368,513,169]],[[0,0,368,513,172]],[[0,0,368,513,175]],[[0,0,368,513,178]],[[0,0,368,513,181]],[[0,0,368,513,184]],[[0,0,368,513,187]],[[0,0,368,513,190]],[[0,0,368,513,193]],[[0,0,368,513,196]],[[0,0,368,513,199]],[[0,0,368,513,202]],[[0,0,368,513,205]],[[0,0,368,513,208]],[[0,0,368,513,211]],[[0,0,368,513,214]],[[0,0,368,513,217]],[[0,0,368,513,220]],[[0,0,368,513,223]],[[0,0,368,513,226]],[[0,0,368,513,229]],[[0,0,368,513,232]],[[0,0,368,513,235]],[[0,0,368,513,238]],[[0,0,368,513,241]],[[0,0,368,513,244]],[[0,0,368,513,247]],[[0,0,368,513,250]],[[0,0,368,513,253]],[[0,0,368,513,256]],[[0,0,368,513,259]],[[0,0,368,513,262]],[[0,0,368,513,265]],[[0,0,368,513,268]],[[0,0,368,513,271]],[[0,0,368,513,274]],[[0,0,368,513,277]],[[0,0,368,513,280]],[[0,0,368,513,283]],[[0,0,368,513,286]],[[0,0,368,513,289]],[[0,0,368,513,292]],[[0,0,368,513,295]],[[0,0,368,513,298]],[[0,0,368,513,301]],[[0,0,368,513,304]],[[0,0,368,513,307]],[[0,0,368,513,310]],[[0,0,368,513,313]],[[0,0,368,513,316]],[[0,0,368,513,319]],[[0,0,368,513,322]],[[0,0,368,513,325]],[[0,0,368,513,328]],[[0,0,368,513,331]],[[0,0,368,513,334]],[[0,0,368,513,337]],[[0,0,368,513,340]],[[0,0,368,513,343]],[[0,0,368,513,346]],[[0,0,368,513,349]],[[0,0,368,513,352]],[[0,0,368,513,355]],[[0,0,368,513,358]],[[0,0,368,513,361]],[[0,0,368,513,364]],[[0,0,368,513,367]],[[0,0,368,513,370]],[[0,0,368,513,373]],[[0,0,368,513,376]],[[0,0,368,513,379]],[[0,0,368,513,382]],[[0,0,368,513,385]],[[0,0,368,513,388]],[[0,0,368,513,391]],[[0,0,368,513,394]],[[0,0,368,513,397]],[[0,0,368,513,400]],[[0,0,368,513,403]],[[0,0,368,513,406]],[[0,0,368,513,409]],[[0,0,368,513,412]],[[0,0,368,513,415]],[[0,0,368,513,418]],[[0,0,368,513,421]],[[0,0,368,513,424]],[[0,0,368,513,427]],[[0,0,368,513,430]],[[0,0,368,513,433]],[[0,0,368,513,436]],[[0,0,368,513,439]],[[0,0,368,513,442]],[[0,0,368,513,445]],[[0,0,368,513,448]],[[0,0,368,513,451]],[[0,0,368,513,454]],[[0,0,368,513,457]],[[0,0,368,513,460]],[[0,0,368,513,463]],[[0,0,368,513,466]],[[0,0,368,513,469]],[[0,0,368,513,472]],[[0,0,368,513,475]],[[0,0,368,513,478]],[[0,0,368,513,481]],[[0,0,368,513,484]],[[0,0,368,513,487]],[[0,0,368,513,490]],[[0,0,368,513,493]],[[0,0,368,513,496]],[[0,0,368,513,499]],[[0,0,368,513,502]],[[0,0,368,513,505]],[[0,0,368,513,508]],[[0,0,368,513,511]],[[0,0,368,513,514]],[[0,0,368,513,517]],[[0,0,368,513,520]],[[0,0,368,513,523]],[[0,0,368,513,526]],[[0,0,368,513,529]],[[0,0,368,513,532]],[[0,0,368,513,535]],[[0,0,368,513,538]],[[0,0,368,513,541]],[[0,0,368,513,544]],[[0,0,368,513,547]],[[0,0,368,513,550]],[[0,0,368,513,553]],[[0,0,368,513,556]],[[0,0,368,513,559]],[[0,0,368,513,562]],[[0,0,368,513,565]],[[0,0,368,513,568]],[[0,0,368,513,571]],[[0,0,368,513,574]],[[0,0,368,513,577]],[[0,0,368,513,580]],[[0,0,368,513,583]],[[0,0,368,513,586]],[[0,0,368,513,589]],[[0,0,368,513,592]],[[0,0,368,513,595]],[[0,0,368,513,598]],[[0,0,368,513,601]],[[0,0,368,513,604]],[[0,0,368,513,607]],[[0,0,368,513,610]],[[0,0,368,513,613]],[[0,0,368,513,616]],[[0,0,368,513,619]],[[0,0,368,513,622]],[[0,0,368,513,625]],[[0,0,368,513,628]],[[0,0,368,513,631]],[[0,0,368,513,634]],[[0,0,368,513,637]],[[0,0,368,513,640]],[[0,0,368,513,643]],[[0,0,368,513,646]],[[0,0,368,513,649]],[[0,0,368,513,652]],[[0,0,368,513,655]],[[0,0,368,513,658]],[[0,0,368,513,661]],[[0,0,368,513,664]],[[0,0,368,513,667]],[[0,0,368,513,670]],[[0,0,368,513,673]],[[0,0,368,513,676]],[[0,0,368,513,679]],[[0,0,368,513,682]],[[0,0,368,513,685]],[[0,0,368,513,688]],[[0,0,368,513,691]],[[0,0,368,513,694]],[[0,0,368,513,697]],[[0,0,368,513,700]],[[0,0,368,513,703]],[[0,0,368,513,706]],[[0,0,368,513,709]],[[0,0,368,513,712]],[[0,0,368,513,715]],[[0,0,368,513,718]],[[0,0,368,513,721]],[[0,0,368,513,724]],[[0,0,368,513,727]],[[0,0,368,513,730]],[[0,0,368,513,733]],[[0,0,368,513,736]],[[0,0,368,513,739]],[[0,0,368,513,742]],[[0,0,368,513,745]],[[0,0,368,513,748]],[[0,0,368,513,751]],[[0,0,368,513,754]],[[0,0,368,513,757]],[[0,0,368,513,760]],[[0,0,368,513,763]],[[0,0,368,513,766]],[[0,0,368,513,769]],[[0,0,368,513,772]],[[0,0,368,513,775]],[[0,0,368,513,778]],[[0,0,368,513,781]],[[0,0,368,513,784]],[[0,0,368,513,787]],[[0,0,368,513,790]],[[0,0,368,513,793]],[[0,0,368,513,796]],[[0,0,368,513,799]],[[0,0,368,513,802]],[[0,0,368,513,805]],[[0,0,368,513,808]],[[0,0,368,513,811]],[[0,0,368,513,814]],[[0,0,368,513,817]],[[0,0,368,513,820]],[[0,0,368,513,823]],[[0,0,368,513,826]],[[0,0,368,513,829]],[[0,0,368,513,832]],[[0,0,368,513,835]],[[0,0,368,513,838]],[[0,0,368,513,841]],[[0,0,368,513,844]],[[0,0,368,513,847]],[[0,0,368,513,850]],[[0,0,368,513,853]],[[0,0,368,513,856]],[[0,0,368,513,859]],[[0,0,368,513,862]],[[0,0,368,513,865]],[[0,0,368,513,868]],[[0,0,368,513,871]],[[0,0,368,513,874]],[[0,0,368,513,877]],[[0,0,368,513,880]],[[0,0,368,513,883]],[[0,0,368,513,886]],[[0,0,368,513,889]],[[0,0,368,513,892]],[[0,0,368,513,895]],[[0,0,368,513,898]],[[0,0,368,513,901]],[[0,0,368,513,904]],[[0,0,368,513,907]],[[0,0,368,513,910]],[[0,0,368,513,913]],[[0,0,368,513,916]],[[0,0,368,513,919]],[[0,0,368,513,922]],[[0,0,368,513,925]],[[0,0,368,513,928]],[[0,0,368,513,931]],[[0,0,368,513,934]],[[0,0,368,513,937]],[[0,0,368,513,940]],[[0,0,368,513,943]],[[0,0,368,513,946]],[[0,0,368,513,949]],[[0,0,368,513,952]],[[0,0,368,513,955]],[[0,0,368,513,958]],[[0,0,368,513,961]],[[0,0,368,513,964]],[[0,0,368,513,967]],[[0,0,368,513,970]],[[0,0,368,513,973]],[[0,0,368,513,976]],[[0,0,368,513,979]],[[0,0,368,513,982]],[[0,0,368,513,985]],[[0,0,368,513,988]],[[0,0,368,513,991]],[[0,0,368,513,994]],[[0,0,368,513,997]],[[0,0,368,513,1000]],[[0,0,368,513,1003]],[[0,0,368,513,1006]],[[0,0,368,513,1009]],[[0,0,368,513,1012]],[[0,0,368,513,1015]],[[0,0,368,513,1018]],[[0,0,368,513,2797]],[[0,0,368,513,2798]],[[0,0,368,513,2799]],[[0,0,368,513,2800]],[[0,0,368,513,2801]],[[0,0,368,513,2802]],[[0,0,368,513,2803]],[[0,0,368,513,2804]],[[0,0,368,513,2805]],[[0,0,368,513,2806]],[[0,0,368,513,2807]],[[0,0,368,513,2808]],[[0,0,368,513,2809]],[[0,0,368,513,2810]],[[0,0,368,513,2811]],[[0,0,368,513,2812]],[[0,0,368,513,2813]],[[0,0,368,513,2814]],[[0,0,368,513,2815]],[[0,0,368,513,2816]],[[0,0,368,513,2817]],[[0,0,368,513,2818]],[[0,0,368,513,2819]],[[0,0,368,513,2820]],[[0,0,368,513,2821]],[[0,0,368,513,2822]],[[0,0,368,513,2823]],[[0,0,368,513,2824]],[[0,0,368,513,2825]],[[0,0,368,513,2826]],[[0,0,368,513,2827]],[[0,0,368,513,2828]],[[0,0,368,513,2829]],[[0,0,368,513,2830]],[[0,0,368,513,2831]],[[0,0,368,513,2832]],[[0,0,368,513,2833]],[[0,0,368,513,2834]],[[0,0,368,513,2835]],[[0,0,368,513,2836]],[[0,0,368,513,2837]],[[0,0,368,513,2838]],[[0,0,368,513,2839]],[[0,0,368,513,2840]],[[0,0,368,513,2841]],[[0,0,368,513,2842]],[[0,0,368,513,2843]],[[0,0,368,513,2844]],[[0,0,368,513,2845]],[[0,0,368,513,2846]],[[0,0,368,513,2847]],[[0,0,368,513,2848]],[[0,0,368,513,2849]],[[0,0,368,513,2850]],[[0,0,368,513,2851]],[[0,0,368,513,2852]],[[0,0,368,513,2853]],[[0,0,368,513,2854]],[[0,0,368,513,2855]],[[0,0,368,513,2856]],[[0,0,368,513,2857]],[[0,0,368,513,2858]],[[0,0,368,513,2859]],[[0,0,368,513,2860]],[[0,0,368,513,2861]],[[0,0,368,513,2862]],[[0,0,368,513,2863]],[[0,0,368,513,2864]],[[0,0,368,513,2797]],[[0,0,368,513,2798]],[[0,0,368,513,2799]],[[0,0,368,513,2800]],[[0,0,368,513,2801]],[[0,0,368,513,2802]],[[0,0,368,513,2803]],[[0,0,368,513,2804]],[[0,0,368,513,2805]],[[0,0,368,513,2806]],[[0,0,368,513,2807]],[[0,0,368,513,2808]],[[0,0,368,513,2809]],[[0,0,368,513,2810]],[[0,0,368,513,2811]],[[0,0,368,513,2812]],[[0,0,368,513,2813]],[[0,0,368,513,2814]],[[0,0,368,513,2815]],[[0,0,368,513,2816]],[[0,0,368,513,2817]],[[0,0,368,513,2818]],[[0,0,368,513,2819]],[[0,0,368,513,2820]],[[0,0,368,513,2821]],[[0,0,368,513,2822]],[[0,0,368,513,2823]],[[0,0,368,513,2824]],[[0,0,368,513,2825]],[[0,0,368,513,2826]],[[0,0,368,513,2827]],[[0,0,368,513,2828]],[[0,0,368,513,2829]],[[0,0,368,513,2830]],[[0,0,368,513,2831]],[[0,0,368,513,2832]],[[0,0,368,513,2833]],[[0,0,368,513,2834]],[[0,0,368,513,2835]],[[0,0,368,513,2836]],[[0,0,368,513,2837]],[[0,0,368,513,2838]],[[0,0,368,513,2839]],[[0,0,368,513,2840]],[[0,0,368,513,2841]],[[0,0,368,513,2842]],[[0,0,368,513,2843]],[[0,0,368,513,2844]],[[0,0,368,513,2845]],[[0,0,368,513,2846]],[[0,0,368,513,2847]],[[0,0,368,513,2848]],[[0,0,368,513,2849]],[[0,0,368,513,2850]],[[0,0,368,513,2851]],[[0,0,368,513,2852]],[[0,0,368,513,2853]],[[0,0,368,513,2854]],[[0,0,368,513,2855]],[[0,0,368,513,2856]],[[0,0,368,513,2857]],[[0,0,368,513,2858]],[[0,0,368,513,2859]],[[0,0,368,513,2860]],[[0,0,368,513,2861]],[[0,0,368,513,2862]],[[0,0,368,513,2863]],[[0,0,368,513,2864]],[[0,0,368,513,1293]],[[0,0,368,513,1296]],[[0,0,368,513,1299]],[[0,0,368,513,1302]],[[0,0,368,513,1305]],[[0,0,368,513,1308]],[[0,0,368,513,1311]],[[0,0,368,513,1314]],[[0,0,368,513,1317]],[[0,0,368,513,1320]],[[0,0,368,513,1323]],[[0,0,368,513,1326]],[[0,0,368,513,1329]],[[0,0,368,513,1332]],[[0,0,368,513,1335]],[[0,0,368,513,1338]],[[0,0,368,513,1341]],[[0,0,368,513,1344]],[[0,0,368,513,1347]],[[0,0,368,513,1350]],[[0,0,368,513,1353]],[[0,0,368,513,1356]],[[0,0,368,513,1359]],[[0,0,368,513,1362]],[[0,0,368,513,1365]],[[0,0,368,513,1368]],[[0,0,368,513,1371]],[[0,0,368,513,1374]],[[0,0,368,513,1377]],[[0,0,368,513,1380]],[[0,0,368,513,1383]],[[0,0,368,513,1386]],[[0,0,368,513,1389]],[[0,0,368,513,1392]],[[0,0,368,513,1395]],[[0,0,368,513,1398]],[[0,0,368,513,1401]],[[0,0,368,513,1404]],[[0,0,368,513,1407]],[[0,0,368,513,1410]],[[0,0,368,513,1413]],[[0,0,368,513,1416]],[[0,0,368,513,1419]],[[0,0,368,513,1422]],[[0,0,368,513,1425]],[[0,0,368,513,1428]],[[0,0,368,513,1431]],[[0,0,368,513,1434]],[[0,0,368,513,1437]],[[0,0,368,513,1440]],[[0,0,368,513,1443]],[[0,0,368,513,1446]],[[0,0,368,513,1449]],[[0,0,368,513,1452]],[[0,0,368,513,1455]],[[0,0,368,513,1458]],[[0,0,368,513,1461]],[[0,0,368,513,1464]],[[0,0,368,513,1467]],[[0,0,368,513,1470]],[[0,0,368,513,1473]],[[0,0,368,513,1476]],[[0,0,368,513,1479]],[[0,0,368,513,1482]],[[0,0,368,513,1485]],[[0,0,368,513,1488]],[[0,0,368,513,1491]],[[0,0,368,513,1494]],[[0,0,368,513,1497]],[[0,0,368,513,1500]],[[0,0,368,513,1503]],[[0,0,368,513,1506]],[[0,0,368,513,1509]],[[0,0,368,513,1512]],[[0,0,368,513,1515]],[[0,0,368,513,1518]],[[0,0,368,513,1521]],[[0,0,368,513,1524]],[[0,0,368,513,1527]],[[0,0,368,513,1530]],[[0,0,368,513,1533]],[[0,0,368,513,1536]],[[0,0,368,513,1539]],[[0,0,368,513,1542]],[[0,0,368,513,1545]],[[0,0,368,513,1548]],[[0,0,368,513,1551]],[[0,0,368,513,1554]],[[0,0,368,513,1557]],[[0,0,368,513,1560]],[[0,0,368,513,1563]],[[0,0,368,513,1566]],[[0,0,368,513,1569]],[[0,0,368,513,1572]],[[0,0,368,513,1575]],[[0,0,368,513,1578]],[[0,0,368,513,1581]],[[0,0,368,513,1584]],[[0,0,368,513,1587]],[[0,0,368,513,1590]],[[0,0,368,513,1593]],[[0,0,368,513,1596]],[[0,0,368,513,1599]],[[0,0,368,513,1602]],[[0,0,368,513,1605]],[[0,0,368,513,1608]],[[0,0,368,513,1611]],[[0,0,368,513,1614]],[[0,0,368,513,1617]],[[0,0,368,513,1620]],[[0,0,368,513,1623]],[[0,0,368,513,1626]],[[0,0,368,513,1629]],[[0,0,368,513,1632]],[[0,0,368,513,1635]],[[0,0,368,513,1638]],[[0,0,368,513,1641]],[[0,0,368,513,1644]],[[0,0,368,513,1647]],[[0,0,368,513,1650]],[[0,0,368,513,1653]],[[0,0,368,513,1656]],[[0,0,368,513,1659]],[[0,0,368,513,1662]],[[0,0,368,513,1665]],[[0,0,368,513,1668]],[[0,0,368,513,1671]],[[0,0,368,513,1674]],[[0,0,368,513,1677]],[[0,0,368,513,1680]],[[0,0,368,513,1683]],[[0,0,368,513,1686]],[[0,0,368,513,1689]],[[0,0,368,513,1692]],[[0,0,368,513,1695]],[[0,0,368,513,1698]],[[0,0,368,513,1701]],[[0,0,368,513,1704]],[[0,0,368,513,1707]],[[0,0,368,513,1710]],[[0,0,368,513,1713]],[[0,0,368,513,1716]],[[0,0,368,513,1719]],[[0,0,368,513,1722]],[[0,0,368,513,1725]],[[0,0,368,513,1728]],[[0,0,368,513,1731]],[[0,0,368,513,1734]],[[0,0,368,513,1737]],[[0,0,368,513,1740]],[[0,0,368,513,1743]],[[0,0,368,513,1746]],[[0,0,368,513,1749]],[[0,0,368,513,1752]],[[0,0,368,513,1755]],[[0,0,368,513,1758]],[[0,0,368,513,1761]],[[0,0,368,513,1764]],[[0,0,368,513,1767]],[[0,0,368,513,1770]],[[0,0,368,513,1773]],[[0,0,368,513,1776]],[[0,0,368,513,1779]],[[0,0,368,513,1782]],[[0,0,368,513,1785]],[[0,0,368,513,1788]],[[0,0,368,513,1791]],[[0,0,368,513,1794]],[[0,0,368,513,1797]],[[0,0,368,513,1800]],[[0,0,368,513,1803]],[[0,0,368,513,1806]],[[0,0,368,513,1809]],[[0,0,368,513,1812]],[[0,0,368,513,1815]],[[0,0,368,513,1818]],[[0,0,368,513,1821]],[[0,0,368,513,1824]],[[0,0,368,513,1827]],[[0,0,368,513,1830]],[[0,0,368,513,1833]],[[0,0,368,513,1836]],[[0,0,368,513,1839]],[[0,0,368,513,1842]],[[0,0,368,513,1845]],[[0,0,368,513,1848]],[[0,0,368,513,1851]],[[0,0,368,513,1854]],[[0,0,368,513,1857]],[[0,0,368,513,1860]],[[0,0,368,513,1863]],[[0,0,368,513,1866]],[[0,0,368,513,1869]],[[0,0,368,513,1872]],[[0,0,368,513,1875]],[[0,0,368,513,1878]],[[0,0,368,513,1881]],[[0,0,368,513,1884]],[[0,0,368,513,1887]],[[0,0,368,513,1890]],[[0,0,368,513,1893]],[[0,0,368,513,1896]],[[0,0,368,513,1899]],[[0,0,368,513,1902]],[[0,0,368,513,1905]],[[0,0,368,513,1908]],[[0,0,368,513,1911]],[[0,0,368,513,1914]],[[0,0,368,513,1917]],[[0,0,368,513,1920]],[[0,0,368,513,1923]],[[0,0,368,513,1926]],[[0,0,368,513,1929]],[[0,0,368,513,1932]],[[0,0,368,513,1935]],[[0,0,368,513,1938]],[[0,0,368,513,1941]],[[0,0,368,513,1944]],[[0,0,368,513,1947]],[[0,0,368,513,1950]],[[0,0,368,513,1953]],[[0,0,368,513,1956]],[[0,0,368,513,1959]],[[0,0,368,513,1962]],[[0,0,368,513,1965]],[[0,0,368,513,1968]],[[0,0,368,513,1971]],[[0,0,368,513,1974]],[[0,0,368,513,1977]],[[0,0,368,513,1980]],[[0,0,368,513,1983]],[[0,0,368,513,1986]],[[0,0,368,513,1989]],[[0,0,368,513,1992]],[[0,0,368,513,1995]],[[0,0,368,513,1998]],[[0,0,368,513,2001]],[[0,0,368,513,2004]],[[0,0,368,513,2007]],[[0,0,368,513,2010]],[[0,0,368,513,2013]],[[0,0,368,513,2016]],[[0,0,368,513,2019]],[[0,0,368,513,2022]],[[0,0,368,513,2025]],[[0,0,368,513,2028]],[[0,0,368,513,2031]],[[0,0,368,513,2034]],[[0,0,368,513,2037]],[[0,0,368,513,2040]],[[0,0,368,513,2043]],[[0,0,368,513,2046]],[[0,0,368,513,2049]],[[0,0,368,513,2052]],[[0,0,368,513,2055]],[[0,0,368,513,2058]],[[0,0,368,513,2061]],[[0,0,368,513,2064]],[[0,0,368,513,2067]],[[0,0,368,513,2070]],[[0,0,368,513,2073]],[[0,0,368,513,2076]],[[0,0,368,513,2079]],[[0,0,368,513,2082]],[[0,0,368,513,2085]],[[0,0,368,513,2088]],[[0,0,368,513,2091]],[[0,0,368,513,2094]],[[0,0,368,513,2097]],[[0,0,368,513,2100]],[[0,0,368,513,2103]],[[0,0,368,513,2106]],[[0,0,368,513,2109]],[[0,0,368,513,2112]],[[0,0,368,513,2115]],[[0,0,368,513,2118]],[[0,0,368,513,2121]],[[0,0,368,513,2124]],[[0,0,368,513,2127]],[[0,0,368,513,2130]],[[0,0,368,513,2133]],[[0,0,368,513,2136]],[[0,0,368,513,2139]],[[0,0,368,513,2142]],[[0,0,368,513,2145]],[[0,0,368,513,2148]],[[0,0,368,513,2151]],[[0,0,368,513,2154]],[[0,0,368,513,2157]],[[0,0,368,513,2160]],[[0,0,368,513,2163]],[[0,0,368,513,2166]],[[0,0,368,513,2169]],[[0,0,368,513,2172]],[[0,0,368,513,2175]],[[0,0,368,513,2178]],[[0,0,368,513,2181]],[[0,0,368,513,2184]],[[0,0,368,513,2187]],[[0,0,368,513,2190]],[[0,0,368,513,2193]],[[0,0,368,513,2196]],[[0,0,368,513,2199]],[[0,0,368,513,2202]],[[0,0,368,513,2205]],[[0,0,368,513,2208]],[[0,0,368,513,2211]],[[0,0,368,513,2214]],[[0,0,368,513,2217]],[[0,0,368,513,2220]],[[0,0,368,513,2223]],[[0,0,368,513,2226]],[[0,0,368,513,2229]],[[0,0,368,513,2232]],[[0,0,368,513,2235]],[[0,0,368,513,2238]],[[0,0,368,513,2241]],[[0,0,368,513,2244]],[[0,0,368,513,2247]],[[0,0,368,513,2250]],[[0,0,368,513,2253]],[[0,0,368,513,2256]],[[0,0,368,513,2259]],[[0,0,368,513,2262]],[[0,0,368,513,2265]],[[0,0,368,513,2268]],[[0,0,368,513,2271]],[[0,0,368,513,2274]],[[0,0,368,513,2277]],[[0,0,368,513,2280]],[[0,0,368,513,2283]],[[0,0,368,513,2286]],[[0,0,368,513,2289]],[[0,0,368,513,2292]],[[0,0,368,513,2295]],[[0,0,368,513,2298]],[[0,0,368,513,2301]],[[0,0,368,513,2304]],[[0,0,368,513,2307]],[[0,0,368,513,2310]],[[0,0,368,513,2313]],[[0,0,368,513,2316]],[[0,0,368,513,2319]],[[0,0,368,513,2322]],[[0,0,368,513,2325]],[[0,0,368,513,2328]],[[0,0,368,513,2331]],[[0,0,368,513,2334]],[[0,0,368,513,2337]],[[0,0,368,513,2340]],[[0,0,368,513,2343]],[[0,0,368,513,2346]],[[0,0,368,513,2349]],[[0,0,368,513,2352]],[[0,0,368,513,2355]],[[0,0,368,513,2358]],[[0,0,368,513,2361]],[[0,0,368,513,2364]],[[0,0,368,513,2367]],[[0,0,368,513,2370]],[[0,0,368,513,2373]],[[0,0,368,513,2376]],[[0,0,368,513,2379]],[[0,0,368,513,2382]],[[0,0,368,513,2385]],[[0,0,368,513,2388]],[[0,0,368,513,2391]],[[0,0,368,513,2394]],[[0,0,368,513,2397]],[[0,0,368,513,2400]],[[0,0,368,513,2403]],[[0,0,368,513,2406]],[[0,0,368,513,2409]],[[0,0,368,513,2412]],[[0,0,368,513,2415]],[[0,0,368,513,2418]],[[0,0,368,513,2421]],[[0,0,368,513,2424]],[[0,0,368,513,2427]],[[0,0,368,513,2430]],[[0,0,368,513,2433]],[[0,0,368,513,2436]],[[0,0,368,513,2439]],[[0,0,368,513,2442]],[[0,0,368,513,2445]],[[0,0,368,513,2448]],[[0,0,368,513,2451]],[[0,0,368,513,2454]],[[0,0,368,513,2457]],[[0,0,368,513,2460]],[[0,0,368,513,2463]],[[0,0,368,513,2466]],[[0,0,368,513,2469]],[[0,0,368,513,2472]],[[0,0,368,513,2475]],[[0,0,368,513,2478]],[[0,0,368,513,2481]],[[0,0,368,513,2484]],[[0,0,368,513,2487]],[[0,0,368,513,2490]],[[0,0,368,513,2493]],[[0,0,368,513,2496]],[[0,0,368,513,2499]],[[0,0,368,513,2502]],[[0,0,368,513,2505]],[[0,0,368,513,2508]],[[0,0,368,513,2511]],[[0,0,368,513,2514]],[[0,0,368,513,2517]],[[0,0,368,513,2520]],[[0,0,368,513,2523]],[[0,0,368,513,2526]],[[0,0,368,513,2529]],[[0,0,368,513,2532]],[[0,0,368,513,2535]],[[0,0,368,513,2538]],[[0,0,368,513,2541]],[[0,0,368,513,2544]],[[0,0,368,513,2547]],[[0,0,368,513,2550]],[[0,0,368,513,2553]],[[0,0,368,513,2556]],[[0,0,368,513,2559]],[[0,0,368,513,2562]],[[0,0,368,513,2565]],[[0,0,368,513,2568]],[[0,0,368,513,2571]],[[0,0,368,513,2574]],[[0,0,368,513,2577]],[[0,0,368,513,2580]],[[0,0,368,513,2583]],[[0,0,368,513,2586]],[[0,0,368,513,2589]],[[0,0,368,513,2592]],[[0,0,368,513,2595]],[[0,0,368,513,2598]],[[0,0,368,513,2601]],[[0,0,368,513,2604]],[[0,0,368,513,2607]],[[0,0,368,513,2610]],[[0,0,368,513,2613]],[[0,0,368,513,2616]],[[0,0,368,513,2619]],[[0,0,368,513,2622]],[[0,0,368,513,2625]],[[0,0,368,513,2628]],[[0,0,368,513,2631]],[[0,0,368,513,2634]],[[0,0,368,513,2637]],[[0,0,368,513,2640]],[[0,0,368,513,2643]],[[0,0,368,513,2646]],[[0,0,368,513,2649]],[[0,0,368,513,2652]],[[0,0,368,513,2655]],[[0,0,368,513,2658]],[[0,0,368,513,2661]],[[0,0,368,513,2664]],[[0,0,368,513,2667]],[[0,0,368,513,2670]],[[0,0,368,513,2673]],[[0,0,368,513,2676]],[[0,0,368,513,2679]],[[0,0,368,513,2682]],[[0,0,368,513,2685]],[[0,0,368,513,2688]],[[0,0,368,513,2691]],[[0,0,368,513,2694]],[[0,0,368,513,2697]],[[0,0,368,513,2700]],[[0,0,368,513,2703]],[[0,0,368,513,2706]],[[0,0,368,513,2709]],[[0,0,368,513,2712]],[[0,0,368,513,2715]],[[0,0,368,513,2718]],[[0,0,368,513,2721]],[[0,0,368,513,2724]],[[0,0,368,513,2727]],[[0,0,368,513,2730]],[[0,0,368,513,2733]],[[0,0,368,513,2736]],[[0,0,368,513,2739]],[[0,0,368,513,2742]],[[0,0,368,513,2745]],[[0,0,368,513,2748]],[[0,0,368,513,2751]],[[0,0,368,513,2754]],[[0,0,368,513,2757]],[[0,0,368,513,2760]],[[0,0,368,513,2763]],[[0,0,368,513,2766]],[[0,0,368,513,2769]],[[0,0,368,513,2772]],[[0,0,368,513,2775]],[[0,0,368,513,2778]],[[0,0,368,513,2781]],[[0,0,368,513,2784]],[[0,0,368,513,2787]],[[0,0,368,513,2790]],[[0,0,368,513,2793]],[[0,0,368,513,2796]]],"text_len_per_page":[53,53,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54,54],"metadata":{"format":"PDF 1.6","title":"","author":"","subject":"","keywords":"","creator":"Adobe Acrobat 7.0","producer":"Adobe Acrobat 7.0 Image Conversion Plug-in","creationDate":"D:20080404141457+01'00'","modDate":"D:20080404144821+01'00'","trapped":"","encryption":null}}
- # o = json.loads(json.dumps(o))
- # total_page = o["total_page"]
- # page_width = o["page_width_pts"]
- # page_height = o["page_height_pts"]
- # img_sz_list = o["image_info_per_page"]
- # text_len_list = o['text_len_per_page']
- # pdf_path = o['pdf_path']
- # is_encrypted = o['is_encrypted']
- # is_needs_password = o['is_needs_password']
- # if is_encrypted or total_page == 0 or is_needs_password: # 加密的，需要密码的，没有页面的，都不处理
- # print("加密的")
- # exit(0)
- # tag = classify(pdf_path, total_page, page_width, page_height, img_sz_list, text_len_list)
- # o['is_text_pdf'] = tag
- # print(json.dumps(o, ensure_ascii=False))
diff --git a/magic_pdf/filter/pdf_meta_scan.py b/magic_pdf/filter/pdf_meta_scan.py
deleted file mode 100644
index 89d44878d..000000000
--- a/magic_pdf/filter/pdf_meta_scan.py
+++ /dev/null
@@ -1,388 +0,0 @@
-"""
-输入： s3路径，每行一个
-输出： pdf文件元信息，包括每一页上的所有图片的长宽高，bbox位置
-"""
-import sys
-import click
-
-from magic_pdf.libs.commons import read_file, mymax, get_top_percent_list
-from magic_pdf.libs.commons import fitz
-from loguru import logger
-from collections import Counter
-
-from magic_pdf.libs.drop_reason import DropReason
-from magic_pdf.libs.language import detect_lang
-from magic_pdf.libs.pdf_check import detect_invalid_chars
-
-scan_max_page = 50
-junk_limit_min = 10
-
-
-def calculate_max_image_area_per_page(result: list, page_width_pts, page_height_pts):
- max_image_area_per_page = [mymax([(x1 - x0) * (y1 - y0) for x0, y0, x1, y1, _ in page_img_sz]) for page_img_sz in
- result]
- page_area = int(page_width_pts) * int(page_height_pts)
- max_image_area_per_page = [area / page_area for area in max_image_area_per_page]
- max_image_area_per_page = [area for area in max_image_area_per_page if area > 0.6]
- return max_image_area_per_page
-
-
-def process_image(page, junk_img_bojids=[]):
- page_result = [] # 存每个页面里的多张图四元组信息
- items = page.get_images()
- dedup = set()
- for img in items:
- # 这里返回的是图片在page上的实际展示的大小。返回一个数组，每个元素第一部分是
- img_bojid = img[0] # 在pdf文件中是全局唯一的，如果这个图反复出现在pdf里那么就可能是垃圾信息，例如水印、页眉页脚等
- if img_bojid in junk_img_bojids: # 如果是垃圾图像，就跳过
- continue
- recs = page.get_image_rects(img, transform=True)
- if recs:
- rec = recs[0][0]
- x0, y0, x1, y1 = map(int, rec)
- width = x1 - x0
- height = y1 - y0
- if (x0, y0, x1, y1, img_bojid) in dedup: # 这里面会出现一些重复的bbox，无需重复出现，需要去掉
- continue
- if not all([width, height]): # 长和宽任何一个都不能是0，否则这个图片不可见，没有实际意义
- continue
- dedup.add((x0, y0, x1, y1, img_bojid))
- page_result.append([x0, y0, x1, y1, img_bojid])
- return page_result
-
-
-def get_image_info(doc: fitz.Document, page_width_pts, page_height_pts) -> list:
- """
- 返回每个页面里的图片的四元组，每个页面多个图片。
- :param doc:
- :return:
- """
- # 使用 Counter 计数 img_bojid 的出现次数
- img_bojid_counter = Counter(img[0] for page in doc for img in page.get_images())
- # 找出出现次数超过 len(doc) 半数的 img_bojid
-
- junk_limit = max(len(doc) * 0.5, junk_limit_min) # 对一些页数比较少的进行豁免
-
- junk_img_bojids = [img_bojid for img_bojid, count in img_bojid_counter.items() if count >= junk_limit]
-
- #todo 加个判断，用前十页就行，这些垃圾图片需要满足两个条件，不止出现的次数要足够多，而且图片占书页面积的比例要足够大，且图与图大小都差不多
- #有两种扫描版，一种文字版，这里可能会有误判
- #扫描版1：每页都有所有扫描页图片，特点是图占比大，每页展示1张
- #扫描版2，每页存储的扫描页图片数量递增，特点是图占比大，每页展示1张，需要清空junklist跑前50页图片信息用于分类判断
- #文字版1.每页存储所有图片，特点是图片占页面比例不大，每页展示可能为0也可能不止1张 这种pdf需要拿前10页抽样检测img大小和个数，如果符合需要清空junklist
- imgs_len_list = [len(page.get_images()) for page in doc]
-
- special_limit_pages = 10
-
- # 统一用前十页结果做判断
- result = []
- break_loop = False
- for i, page in enumerate(doc):
- if break_loop:
- break
- if i >= special_limit_pages:
- break
- page_result = process_image(page) # 这里不传junk_img_bojids，拿前十页所有图片信息用于后续分析
- result.append(page_result)
- for item in result:
- if not any(item): # 如果任何一页没有图片，说明是个文字版，需要判断是否为特殊文字版
- if max(imgs_len_list) == min(imgs_len_list) and max(
- imgs_len_list) >= junk_limit_min: # 如果是特殊文字版，就把junklist置空并break
- junk_img_bojids = []
- else: # 不是特殊文字版，是个普通文字版，但是存在垃圾图片，不置空junklist
- pass
- break_loop = True
- break
- if not break_loop:
- # 获取前80%的元素
- top_eighty_percent = get_top_percent_list(imgs_len_list, 0.8)
- # 检查前80%的元素是否都相等
- if len(set(top_eighty_percent)) == 1 and max(imgs_len_list) >= junk_limit_min:
-
- # # 如果前10页跑完都有图，根据每页图片数量是否相等判断是否需要清除junklist
- # if max(imgs_len_list) == min(imgs_len_list) and max(imgs_len_list) >= junk_limit_min:
-
- #前10页都有图，且每页数量一致，需要检测图片大小占页面的比例判断是否需要清除junklist
- max_image_area_per_page = calculate_max_image_area_per_page(result, page_width_pts, page_height_pts)
- if len(max_image_area_per_page) < 0.8 * special_limit_pages: # 前10页不全是大图，说明可能是个文字版pdf，把垃圾图片list置空
- junk_img_bojids = []
- else: # 前10页都有图，而且80%都是大图，且每页图片数量一致并都很多，说明是扫描版1，不需要清空junklist
- pass
- else: # 每页图片数量不一致，需要清掉junklist全量跑前50页图片
- junk_img_bojids = []
-
- #正式进入取前50页图片的信息流程
- result = []
- for i, page in enumerate(doc):
- if i >= scan_max_page:
- break
- page_result = process_image(page, junk_img_bojids)
- # logger.info(f"page {i} img_len: {len(page_result)}")
- result.append(page_result)
-
- return result, junk_img_bojids
-
-
-def get_pdf_page_size_pts(doc: fitz.Document):
- page_cnt = len(doc)
- l: int = min(page_cnt, 50)
- #把所有宽度和高度塞到两个list 分别取中位数（中间遇到了个在纵页里塞横页的pdf，导致宽高互换了）
- page_width_list = []
- page_height_list = []
- for i in range(l):
- page = doc[i]
- page_rect = page.rect
- page_width_list.append(page_rect.width)
- page_height_list.append(page_rect.height)
-
- page_width_list.sort()
- page_height_list.sort()
-
- median_width = page_width_list[len(page_width_list) // 2]
- median_height = page_height_list[len(page_height_list) // 2]
-
- return median_width, median_height
-
-
-def get_pdf_textlen_per_page(doc: fitz.Document):
- text_len_lst = []
- for page in doc:
- # 拿包含img和text的所有blocks
- # text_block = page.get_text("blocks")
- # 拿所有text的blocks
- # text_block = page.get_text("words")
- # text_block_len = sum([len(t[4]) for t in text_block])
- #拿所有text的str
- text_block = page.get_text("text")
- text_block_len = len(text_block)
- # logger.info(f"page {page.number} text_block_len: {text_block_len}")
- text_len_lst.append(text_block_len)
-
- return text_len_lst
-
-
-def get_pdf_text_layout_per_page(doc: fitz.Document):
- """
- 根据PDF文档的每一页文本布局，判断该页的文本布局是横向、纵向还是未知。
-
- Args:
- doc (fitz.Document): PDF文档对象。
-
- Returns:
- List[str]: 每一页的文本布局（横向、纵向、未知）。
-
- """
- text_layout_list = []
-
- for page_id, page in enumerate(doc):
- if page_id >= scan_max_page:
- break
- # 创建每一页的纵向和横向的文本行数计数器
- vertical_count = 0
- horizontal_count = 0
- text_dict = page.get_text("dict")
- if "blocks" in text_dict:
- for block in text_dict["blocks"]:
- if 'lines' in block:
- for line in block["lines"]:
- # 获取line的bbox顶点坐标
- x0, y0, x1, y1 = line['bbox']
- # 计算bbox的宽高
- width = x1 - x0
- height = y1 - y0
- # 计算bbox的面积
- area = width * height
- font_sizes = []
- for span in line['spans']:
- if 'size' in span:
- font_sizes.append(span['size'])
- if len(font_sizes) > 0:
- average_font_size = sum(font_sizes) / len(font_sizes)
- else:
- average_font_size = 10 # 有的line拿不到font_size，先定一个阈值100
- if area <= average_font_size ** 2: # 判断bbox的面积是否小于平均字体大小的平方,单字无法计算是横向还是纵向
- continue
- else:
- if 'wmode' in line: # 通过wmode判断文本方向
- if line['wmode'] == 1: # 判断是否为竖向文本
- vertical_count += 1
- elif line['wmode'] == 0: # 判断是否为横向文本
- horizontal_count += 1
- # if 'dir' in line: # 通过旋转角度计算判断文本方向
- # # 获取行的 "dir" 值
- # dir_value = line['dir']
- # cosine, sine = dir_value
- # # 计算角度
- # angle = math.degrees(math.acos(cosine))
- #
- # # 判断是否为横向文本
- # if abs(angle - 0) < 0.01 or abs(angle - 180) < 0.01:
- # # line_text = ' '.join(span['text'] for span in line['spans'])
- # # print('This line is horizontal:', line_text)
- # horizontal_count += 1
- # # 判断是否为纵向文本
- # elif abs(angle - 90) < 0.01 or abs(angle - 270) < 0.01:
- # # line_text = ' '.join(span['text'] for span in line['spans'])
- # # print('This line is vertical:', line_text)
- # vertical_count += 1
- # print(f"page_id: {page_id}, vertical_count: {vertical_count}, horizontal_count: {horizontal_count}")
- # 判断每一页的文本布局
- if vertical_count == 0 and horizontal_count == 0: # 该页没有文本，无法判断
- text_layout_list.append("unknow")
- continue
- else:
- if vertical_count > horizontal_count: # 该页的文本纵向行数大于横向的
- text_layout_list.append("vertical")
- else: # 该页的文本横向行数大于纵向的
- text_layout_list.append("horizontal")
- # logger.info(f"page_id: {page_id}, vertical_count: {vertical_count}, horizontal_count: {horizontal_count}")
- return text_layout_list
-
-
-'''定义一个自定义异常用来抛出单页svg太多的pdf'''
-
-
-class PageSvgsTooManyError(Exception):
- def __init__(self, message="Page SVGs are too many"):
- self.message = message
- super().__init__(self.message)
-
-
-def get_svgs_per_page(doc: fitz.Document):
- svgs_len_list = []
- for page_id, page in enumerate(doc):
- # svgs = page.get_drawings()
- svgs = page.get_cdrawings() # 切换成get_cdrawings，效率更高
- len_svgs = len(svgs)
- if len_svgs >= 3000:
- raise PageSvgsTooManyError()
- else:
- svgs_len_list.append(len_svgs)
- # logger.info(f"page_id: {page_id}, svgs_len: {len(svgs)}")
- return svgs_len_list
-
-
-def get_imgs_per_page(doc: fitz.Document):
- imgs_len_list = []
- for page_id, page in enumerate(doc):
- imgs = page.get_images()
- imgs_len_list.append(len(imgs))
- # logger.info(f"page_id: {page}, imgs_len: {len(imgs)}")
-
- return imgs_len_list
-
-
-def get_language(doc: fitz.Document):
- """
- 获取PDF文档的语言。
- Args:
- doc (fitz.Document): PDF文档对象。
- Returns:
- str: 文档语言，如 "en-US"。
- """
- language_lst = []
- for page_id, page in enumerate(doc):
- if page_id >= scan_max_page:
- break
- # 拿所有text的str
- text_block = page.get_text("text")
- page_language = detect_lang(text_block)
- language_lst.append(page_language)
-
- # logger.info(f"page_id: {page_id}, page_language: {page_language}")
-
- # 统计text_language_list中每种语言的个数
- count_dict = Counter(language_lst)
- # 输出text_language_list中出现的次数最多的语言
- language = max(count_dict, key=count_dict.get)
- return language
-
-
-def check_invalid_chars(pdf_bytes):
- """
- 乱码检测
- """
- return detect_invalid_chars(pdf_bytes)
-
-
-def pdf_meta_scan(pdf_bytes: bytes):
- """
- :param s3_pdf_path:
- :param pdf_bytes: pdf文件的二进制数据
- 几个维度来评价：是否加密，是否需要密码，纸张大小，总页数，是否文字可提取
- """
- doc = fitz.open("pdf", pdf_bytes)
- is_needs_password = doc.needs_pass
- is_encrypted = doc.is_encrypted
- total_page = len(doc)
- if total_page == 0:
- logger.warning(f"drop this pdf, drop_reason: {DropReason.EMPTY_PDF}")
- result = {"_need_drop": True, "_drop_reason": DropReason.EMPTY_PDF}
- return result
- else:
- page_width_pts, page_height_pts = get_pdf_page_size_pts(doc)
- # logger.info(f"page_width_pts: {page_width_pts}, page_height_pts: {page_height_pts}")
-
- # svgs_per_page = get_svgs_per_page(doc)
- # logger.info(f"svgs_per_page: {svgs_per_page}")
- imgs_per_page = get_imgs_per_page(doc)
- # logger.info(f"imgs_per_page: {imgs_per_page}")
-
- image_info_per_page, junk_img_bojids = get_image_info(doc, page_width_pts, page_height_pts)
- # logger.info(f"image_info_per_page: {image_info_per_page}, junk_img_bojids: {junk_img_bojids}")
- text_len_per_page = get_pdf_textlen_per_page(doc)
- # logger.info(f"text_len_per_page: {text_len_per_page}")
- text_layout_per_page = get_pdf_text_layout_per_page(doc)
- # logger.info(f"text_layout_per_page: {text_layout_per_page}")
- text_language = get_language(doc)
- # logger.info(f"text_language: {text_language}")
- invalid_chars = check_invalid_chars(pdf_bytes)
- # logger.info(f"invalid_chars: {invalid_chars}")
-
- # 最后输出一条json
- res = {
- "is_needs_password": is_needs_password,
- "is_encrypted": is_encrypted,
- "total_page": total_page,
- "page_width_pts": int(page_width_pts),
- "page_height_pts": int(page_height_pts),
- "image_info_per_page": image_info_per_page,
- "text_len_per_page": text_len_per_page,
- "text_layout_per_page": text_layout_per_page,
- "text_language": text_language,
- # "svgs_per_page": svgs_per_page,
- "imgs_per_page": imgs_per_page, # 增加每页img数量list
- "junk_img_bojids": junk_img_bojids, # 增加垃圾图片的bojid list
- "invalid_chars": invalid_chars,
- "metadata": doc.metadata
- }
- # logger.info(json.dumps(res, ensure_ascii=False))
- return res
-
-
-@click.command()
-@click.option('--s3-pdf-path', help='s3上pdf文件的路径')
-@click.option('--s3-profile', help='s3上的profile')
-def main(s3_pdf_path: str, s3_profile: str):
- """
-
- """
- try:
- file_content = read_file(s3_pdf_path, s3_profile)
- pdf_meta_scan(file_content)
- except Exception as e:
- print(f"ERROR: {s3_pdf_path}, {e}", file=sys.stderr)
- logger.exception(e)
-
-
-if __name__ == '__main__':
- main()
- # "D:\project/20231108code-clean\pdf_cost_time\竖排例子\净空法师-大乘无量寿.pdf"
- # "D:\project/20231108code-clean\pdf_cost_time\竖排例子\三国演义_繁体竖排版.pdf"
- # "D:\project/20231108code-clean\pdf_cost_time\scihub\scihub_86800000\libgen.scimag86880000-86880999.zip_10.1021/acsami.1c03109.s002.pdf"
- # "D:/project/20231108code-clean/pdf_cost_time/scihub/scihub_18600000/libgen.scimag18645000-18645999.zip_10.1021/om3006239.pdf"
- # file_content = read_file("D:/project/20231108code-clean/pdf_cost_time/scihub/scihub_31000000/libgen.scimag31098000-31098999.zip_10.1109/isit.2006.261791.pdf","")
- # file_content = read_file("D:\project/20231108code-clean\pdf_cost_time\竖排例子\净空法师_大乘无量寿.pdf","")
- # doc = fitz.open("pdf", file_content)
- # text_layout_lst = get_pdf_text_layout_per_page(doc)
- # print(text_layout_lst)
diff --git a/magic_pdf/integrations/__init__.py b/magic_pdf/integrations/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/integrations/rag/__init__.py b/magic_pdf/integrations/rag/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/integrations/rag/api.py b/magic_pdf/integrations/rag/api.py
deleted file mode 100644
index 5c05f9116..000000000
--- a/magic_pdf/integrations/rag/api.py
+++ /dev/null
@@ -1,82 +0,0 @@
-import os
-from pathlib import Path
-
-from loguru import logger
-
-from magic_pdf.integrations.rag.type import (ElementRelation, LayoutElements,
- Node)
-from magic_pdf.integrations.rag.utils import inference
-
-
-class RagPageReader:
-
- def __init__(self, pagedata: LayoutElements):
- self.o = [
- Node(
- category_type=v.category_type,
- text=v.text,
- image_path=v.image_path,
- anno_id=v.anno_id,
- latex=v.latex,
- html=v.html,
- ) for v in pagedata.layout_dets
- ]
-
- self.pagedata = pagedata
-
- def __iter__(self):
- return iter(self.o)
-
- def get_rel_map(self) -> list[ElementRelation]:
- return self.pagedata.extra.element_relation
-
-
-class RagDocumentReader:
-
- def __init__(self, ragdata: list[LayoutElements]):
- self.o = [RagPageReader(v) for v in ragdata]
-
- def __iter__(self):
- return iter(self.o)
-
-
-class DataReader:
-
- def __init__(self, path_or_directory: str, method: str, output_dir: str):
- self.path_or_directory = path_or_directory
- self.method = method
- self.output_dir = output_dir
- self.pdfs = []
- if os.path.isdir(path_or_directory):
- for doc_path in Path(path_or_directory).glob('*.pdf'):
- self.pdfs.append(doc_path)
- else:
- assert path_or_directory.endswith('.pdf')
- self.pdfs.append(Path(path_or_directory))
-
- def get_documents_count(self) -> int:
- """Returns the number of documents in the directory."""
- return len(self.pdfs)
-
- def get_document_result(self, idx: int) -> RagDocumentReader | None:
- """
- Args:
- idx (int): the index of documents under the
- directory path_or_directory
-
- Returns:
- RagDocumentReader | None: RagDocumentReader is an iterable object,
- more details @RagDocumentReader
- """
- if idx >= self.get_documents_count() or idx < 0:
- logger.error(f'invalid idx: {idx}')
- return None
- res = inference(str(self.pdfs[idx]), self.output_dir, self.method)
- if res is None:
- logger.warning(f'failed to inference pdf {self.pdfs[idx]}')
- return None
- return RagDocumentReader(res)
-
- def get_document_filename(self, idx: int) -> Path:
- """get the filename of the document."""
- return self.pdfs[idx]
diff --git a/magic_pdf/integrations/rag/type.py b/magic_pdf/integrations/rag/type.py
deleted file mode 100644
index 11258af39..000000000
--- a/magic_pdf/integrations/rag/type.py
+++ /dev/null
@@ -1,82 +0,0 @@
-from enum import Enum
-
-from pydantic import BaseModel, Field
-
-
-# rag
-class CategoryType(Enum): # py310 not support StrEnum
- text = 'text'
- title = 'title'
- interline_equation = 'interline_equation'
- image = 'image'
- image_body = 'image_body'
- image_caption = 'image_caption'
- table = 'table'
- table_body = 'table_body'
- table_caption = 'table_caption'
- table_footnote = 'table_footnote'
-
-
-class ElementRelType(Enum):
- sibling = 'sibling'
-
-
-class PageInfo(BaseModel):
- page_no: int = Field(description='the index of page, start from zero',
- ge=0)
- height: int = Field(description='the height of page', gt=0)
- width: int = Field(description='the width of page', ge=0)
- image_path: str | None = Field(description='the image of this page',
- default=None)
-
-
-class ContentObject(BaseModel):
- category_type: CategoryType = Field(description='类别')
- poly: list[float] = Field(
- description=('Coordinates, need to convert back to PDF coordinates,'
- ' order is top-left, top-right, bottom-right, bottom-left'
- ' x,y coordinates'))
- ignore: bool = Field(description='whether ignore this object',
- default=False)
- text: str | None = Field(description='text content of the object',
- default=None)
- image_path: str | None = Field(description='path of embedded image',
- default=None)
- order: int = Field(description='the order of this object within a page',
- default=-1)
- anno_id: int = Field(description='unique id', default=-1)
- latex: str | None = Field(description='latex result', default=None)
- html: str | None = Field(description='html result', default=None)
-
-
-class ElementRelation(BaseModel):
- source_anno_id: int = Field(description='unique id of the source object',
- default=-1)
- target_anno_id: int = Field(description='unique id of the target object',
- default=-1)
- relation: ElementRelType = Field(
- description='the relation between source and target element')
-
-
-class LayoutElementsExtra(BaseModel):
- element_relation: list[ElementRelation] = Field(
- description='the relation between source and target element')
-
-
-class LayoutElements(BaseModel):
- layout_dets: list[ContentObject] = Field(
- description='layout element details')
- page_info: PageInfo = Field(description='page info')
- extra: LayoutElementsExtra = Field(description='extra information')
-
-
-# iter data format
-class Node(BaseModel):
- category_type: CategoryType = Field(description='类别')
- text: str | None = Field(description='text content of the object',
- default=None)
- image_path: str | None = Field(description='path of embedded image',
- default=None)
- anno_id: int = Field(description='unique id', default=-1)
- latex: str | None = Field(description='latex result', default=None)
- html: str | None = Field(description='html result', default=None)
diff --git a/magic_pdf/integrations/rag/utils.py b/magic_pdf/integrations/rag/utils.py
deleted file mode 100644
index ed736ffbc..000000000
--- a/magic_pdf/integrations/rag/utils.py
+++ /dev/null
@@ -1,285 +0,0 @@
-import json
-import os
-from pathlib import Path
-
-from loguru import logger
-
-import magic_pdf.model as model_config
-from magic_pdf.dict2md.ocr_mkcontent import merge_para_with_text
-from magic_pdf.integrations.rag.type import (CategoryType, ContentObject,
- ElementRelation, ElementRelType,
- LayoutElements,
- LayoutElementsExtra, PageInfo)
-from magic_pdf.libs.ocr_content_type import BlockType, ContentType
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-from magic_pdf.tools.common import do_parse, prepare_env
-
-
-def convert_middle_json_to_layout_elements(
- json_data: dict,
- output_dir: str,
-) -> list[LayoutElements]:
- uniq_anno_id = 0
-
- res: list[LayoutElements] = []
- for page_no, page_data in enumerate(json_data['pdf_info']):
- order_id = 0
- page_info = PageInfo(
- height=int(page_data['page_size'][1]),
- width=int(page_data['page_size'][0]),
- page_no=page_no,
- )
- layout_dets: list[ContentObject] = []
- extra_element_relation: list[ElementRelation] = []
-
- for para_block in page_data['para_blocks']:
- para_text = ''
- para_type = para_block['type']
-
- if para_type == BlockType.Text:
- para_text = merge_para_with_text(para_block)
- x0, y0, x1, y1 = para_block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.text,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- elif para_type == BlockType.Title:
- para_text = merge_para_with_text(para_block)
- x0, y0, x1, y1 = para_block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.title,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- elif para_type == BlockType.InterlineEquation:
- para_text = merge_para_with_text(para_block)
- x0, y0, x1, y1 = para_block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.interline_equation,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- elif para_type == BlockType.Image:
- body_anno_id = -1
- caption_anno_id = -1
-
- for block in para_block['blocks']:
- if block['type'] == BlockType.ImageBody:
- for line in block['lines']:
- for span in line['spans']:
- if span['type'] == ContentType.Image:
- x0, y0, x1, y1 = block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.image_body,
- image_path=os.path.join(
- output_dir, span['image_path']),
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- body_anno_id = uniq_anno_id
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- for block in para_block['blocks']:
- if block['type'] == BlockType.ImageCaption:
- para_text += merge_para_with_text(block)
- x0, y0, x1, y1 = block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.image_caption,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- caption_anno_id = uniq_anno_id
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- if body_anno_id > 0 and caption_anno_id > 0:
- element_relation = ElementRelation(
- relation=ElementRelType.sibling,
- source_anno_id=body_anno_id,
- target_anno_id=caption_anno_id,
- )
- extra_element_relation.append(element_relation)
-
- elif para_type == BlockType.Table:
- body_anno_id, caption_anno_id, footnote_anno_id = -1, -1, -1
-
- for block in para_block['blocks']:
- if block['type'] == BlockType.TableCaption:
- para_text += merge_para_with_text(block)
- x0, y0, x1, y1 = block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.table_caption,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- caption_anno_id = uniq_anno_id
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- for block in para_block['blocks']:
- if block['type'] == BlockType.TableBody:
- for line in block['lines']:
- for span in line['spans']:
- if span['type'] == ContentType.Table:
- x0, y0, x1, y1 = para_block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.table_body,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- body_anno_id = uniq_anno_id
- uniq_anno_id += 1
- order_id += 1
- # if processed by table model
- if span.get('latex', ''):
- content.latex = span['latex']
- else:
- content.image_path = os.path.join(
- output_dir, span['image_path'])
- layout_dets.append(content)
-
- for block in para_block['blocks']:
- if block['type'] == BlockType.TableFootnote:
- para_text += merge_para_with_text(block)
- x0, y0, x1, y1 = block['bbox']
- content = ContentObject(
- anno_id=uniq_anno_id,
- category_type=CategoryType.table_footnote,
- text=para_text,
- order=order_id,
- poly=[x0, y0, x1, y0, x1, y1, x0, y1],
- )
- footnote_anno_id = uniq_anno_id
- uniq_anno_id += 1
- order_id += 1
- layout_dets.append(content)
-
- if caption_anno_id != -1 and body_anno_id != -1:
- element_relation = ElementRelation(
- relation=ElementRelType.sibling,
- source_anno_id=body_anno_id,
- target_anno_id=caption_anno_id,
- )
- extra_element_relation.append(element_relation)
-
- if footnote_anno_id != -1 and body_anno_id != -1:
- element_relation = ElementRelation(
- relation=ElementRelType.sibling,
- source_anno_id=body_anno_id,
- target_anno_id=footnote_anno_id,
- )
- extra_element_relation.append(element_relation)
-
- res.append(
- LayoutElements(
- page_info=page_info,
- layout_dets=layout_dets,
- extra=LayoutElementsExtra(
- element_relation=extra_element_relation),
- ))
-
- return res
-
-
-def inference(path, output_dir, method):
- model_config.__use_inside_model__ = True
- model_config.__model_mode__ = 'full'
- if output_dir == '':
- if os.path.isdir(path):
- output_dir = os.path.join(path, 'output')
- else:
- output_dir = os.path.join(os.path.dirname(path), 'output')
-
- local_image_dir, local_md_dir = prepare_env(output_dir,
- str(Path(path).stem), method)
-
- def read_fn(path):
- disk_rw = DiskReaderWriter(os.path.dirname(path))
- return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN)
-
- def parse_doc(doc_path: str):
- try:
- file_name = str(Path(doc_path).stem)
- pdf_data = read_fn(doc_path)
- do_parse(
- output_dir,
- file_name,
- pdf_data,
- [],
- method,
- False,
- f_draw_span_bbox=False,
- f_draw_layout_bbox=False,
- f_dump_md=False,
- f_dump_middle_json=True,
- f_dump_model_json=False,
- f_dump_orig_pdf=False,
- f_dump_content_list=False,
- f_draw_model_bbox=False,
- )
-
- middle_json_fn = os.path.join(local_md_dir,
- f'{file_name}_middle.json')
- with open(middle_json_fn) as fd:
- jso = json.load(fd)
- os.remove(middle_json_fn)
- return convert_middle_json_to_layout_elements(jso, local_image_dir)
-
- except Exception as e:
- logger.exception(e)
-
- return parse_doc(path)
-
-
-if __name__ == '__main__':
- import pprint
-
- base_dir = '/opt/data/pdf/resources/samples/'
- if 0:
- with open(base_dir + 'json_outputs/middle.json') as f:
- d = json.load(f)
- result = convert_middle_json_to_layout_elements(d, '/tmp')
- pprint.pp(result)
- if 0:
- with open(base_dir + 'json_outputs/middle.3.json') as f:
- d = json.load(f)
- result = convert_middle_json_to_layout_elements(d, '/tmp')
- pprint.pp(result)
-
- if 1:
- res = inference(
- base_dir + 'samples/pdf/one_page_with_table_image.pdf',
- '/tmp/output',
- 'ocr',
- )
- pprint.pp(res)
diff --git a/magic_pdf/layout/__init__.py b/magic_pdf/layout/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/layout/bbox_sort.py b/magic_pdf/layout/bbox_sort.py
deleted file mode 100644
index 5e1508ff5..000000000
--- a/magic_pdf/layout/bbox_sort.py
+++ /dev/null
@@ -1,681 +0,0 @@
-# 定义这里的bbox是一个list [x0, y0, x1, y1, block_content, idx_x, idx_y, content_type, ext_x0, ext_y0, ext_x1, ext_y1], 初始时候idx_x, idx_y都是None
-# 其中x0, y0代表左上角坐标，x1, y1代表右下角坐标，坐标原点在左上角。
-
-
-
-from magic_pdf.layout.layout_spiler_recog import get_spilter_of_page
-from magic_pdf.libs.boxbase import _is_in, _is_in_or_part_overlap, _is_vertical_full_overlap
-from magic_pdf.libs.commons import mymax
-
-X0_IDX = 0
-Y0_IDX = 1
-X1_IDX = 2
-Y1_IDX = 3
-CONTENT_IDX = 4
-IDX_X = 5
-IDX_Y = 6
-CONTENT_TYPE_IDX = 7
-
-X0_EXT_IDX = 8
-Y0_EXT_IDX = 9
-X1_EXT_IDX = 10
-Y1_EXT_IDX = 11
-
-
-def prepare_bboxes_for_layout_split(image_info, image_backup_info, table_info, inline_eq_info, interline_eq_info, text_raw_blocks: dict, page_boundry, page):
- """
- text_raw_blocks:结构参考test/assets/papre/pymu_textblocks.json
- 把bbox重新组装成一个list，每个元素[x0, y0, x1, y1, block_content, idx_x, idx_y, content_type, ext_x0, ext_y0, ext_x1, ext_y1], 初始时候idx_x, idx_y都是None. 对于图片、公式来说，block_content是图片的地址， 对于段落来说，block_content是pymupdf里的block结构
- """
- all_bboxes = []
-
- for image in image_info:
- box = image['bbox']
- # 由于没有实现横向的栏切分，因此在这里先过滤掉一些小的图片。这些图片有可能影响layout，造成没有横向栏切分的情况下，layout切分不准确。例如 scihub_76500000/libgen.scimag76570000-76570999.zip_10.1186/s13287-019-1355-1
- # 把长宽都小于50的去掉
- if abs(box[0]-box[2]) < 50 and abs(box[1]-box[3]) < 50:
- continue
- all_bboxes.append([box[0], box[1], box[2], box[3], None, None, None, 'image', None, None, None, None])
-
- for table in table_info:
- box = table['bbox']
- all_bboxes.append([box[0], box[1], box[2], box[3], None, None, None, 'table', None, None, None, None])
-
- """由于公式与段落混合，因此公式不再参与layout划分，无需加入all_bboxes"""
- # 加入文本block
- text_block_temp = []
- for block in text_raw_blocks:
- bbox = block['bbox']
- text_block_temp.append([bbox[0], bbox[1], bbox[2], bbox[3], None, None, None, 'text', None, None, None, None])
-
- text_block_new = resolve_bbox_overlap_for_layout_det(text_block_temp)
- text_block_new = filter_lines_bbox(text_block_new) # 去掉线条bbox，有可能让layout探测陷入无限循环
-
-
- """找出会影响layout的色块、横向分割线"""
- spilter_bboxes = get_spilter_of_page(page, [b['bbox'] for b in image_info]+[b['bbox'] for b in image_backup_info], [b['bbox'] for b in table_info], )
- # 还要去掉存在于spilter_bboxes里的text_block
- if len(spilter_bboxes) > 0:
- text_block_new = [box for box in text_block_new if not any([_is_in_or_part_overlap(box[:4], spilter_bbox) for spilter_bbox in spilter_bboxes])]
-
- for bbox in text_block_new:
- all_bboxes.append([bbox[0], bbox[1], bbox[2], bbox[3], None, None, None, 'text', None, None, None, None])
-
- for bbox in spilter_bboxes:
- all_bboxes.append([bbox[0], bbox[1], bbox[2], bbox[3], None, None, None, 'spilter', None, None, None, None])
-
-
- return all_bboxes
-
-def resolve_bbox_overlap_for_layout_det(bboxes:list):
- """
- 1. 去掉bbox互相包含的，去掉被包含的
- 2. 上下方向上如果有重叠，就扩大大box范围，直到覆盖小box
- """
- def _is_in_other_bbox(i:int):
- """
- 判断i个box是否被其他box有所包含
- """
- for j in range(0, len(bboxes)):
- if j!=i and _is_in(bboxes[i][:4], bboxes[j][:4]):
- return True
- # elif j!=i and _is_bottom_full_overlap(bboxes[i][:4], bboxes[j][:4]):
- # return True
-
- return False
-
- # 首先去掉被包含的bbox
- new_bbox_1 = []
- for i in range(0, len(bboxes)):
- if not _is_in_other_bbox(i):
- new_bbox_1.append(bboxes[i])
-
- # 其次扩展大的box
- new_box = []
- new_bbox_2 = []
- len_1 = len(new_bbox_2)
- while True:
- merged_idx = []
- for i in range(0, len(new_bbox_1)):
- if i in merged_idx:
- continue
- for j in range(i+1, len(new_bbox_1)):
- if j in merged_idx:
- continue
- bx1 = new_bbox_1[i]
- bx2 = new_bbox_1[j]
- if i!=j and _is_vertical_full_overlap(bx1[:4], bx2[:4]):
- merged_box = min([bx1[0], bx2[0]]), min([bx1[1], bx2[1]]), max([bx1[2], bx2[2]]), max([bx1[3], bx2[3]])
- new_bbox_2.append(merged_box)
- merged_idx.append(i)
- merged_idx.append(j)
-
- for i in range(0, len(new_bbox_1)): # 没有合并的加入进来
- if i not in merged_idx:
- new_bbox_2.append(new_bbox_1[i])
-
- if len(new_bbox_2)==0 or len_1==len(new_bbox_2):
- break
- else:
- len_1 = len(new_bbox_2)
- new_box = new_bbox_2
- new_bbox_1, new_bbox_2 = new_bbox_2, []
-
- return new_box
-
-
-def filter_lines_bbox(bboxes: list):
- """
- 过滤掉bbox为空的行
- """
- new_box = []
- for box in bboxes:
- x0, y0, x1, y1 = box[0], box[1], box[2], box[3]
- if abs(x0-x1)<=1 or abs(y0-y1)<=1:
- continue
- else:
- new_box.append(box)
- return new_box
-
-
-################################################################################
-# 第一种排序算法
-# 以下是基于延长线遮挡做的一个算法
-#
-################################################################################
-def find_all_left_bbox(this_bbox, all_bboxes) -> list:
- """
- 寻找this_bbox左边的所有bbox
- """
- left_boxes = [box for box in all_bboxes if box[X1_IDX] <= this_bbox[X0_IDX]]
- return left_boxes
-
-
-def find_all_top_bbox(this_bbox, all_bboxes) -> list:
- """
- 寻找this_bbox上面的所有bbox
- """
- top_boxes = [box for box in all_bboxes if box[Y1_IDX] <= this_bbox[Y0_IDX]]
- return top_boxes
-
-
-def get_and_set_idx_x(this_bbox, all_bboxes) -> int:
- """
- 寻找this_bbox在all_bboxes中的遮挡深度 idx_x
- """
- if this_bbox[IDX_X] is not None:
- return this_bbox[IDX_X]
- else:
- all_left_bboxes = find_all_left_bbox(this_bbox, all_bboxes)
- if len(all_left_bboxes) == 0:
- this_bbox[IDX_X] = 0
- else:
- all_left_bboxes_idx = [get_and_set_idx_x(bbox, all_bboxes) for bbox in all_left_bboxes]
- max_idx_x = mymax(all_left_bboxes_idx)
- this_bbox[IDX_X] = max_idx_x + 1
- return this_bbox[IDX_X]
-
-
-def get_and_set_idx_y(this_bbox, all_bboxes) -> int:
- """
- 寻找this_bbox在all_bboxes中y方向的遮挡深度 idx_y
- """
- if this_bbox[IDX_Y] is not None:
- return this_bbox[IDX_Y]
- else:
- all_top_bboxes = find_all_top_bbox(this_bbox, all_bboxes)
- if len(all_top_bboxes) == 0:
- this_bbox[IDX_Y] = 0
- else:
- all_top_bboxes_idx = [get_and_set_idx_y(bbox, all_bboxes) for bbox in all_top_bboxes]
- max_idx_y = mymax(all_top_bboxes_idx)
- this_bbox[IDX_Y] = max_idx_y + 1
- return this_bbox[IDX_Y]
-
-
-def bbox_sort(all_bboxes: list):
- """
- 排序
- """
- all_bboxes_idx_x = [get_and_set_idx_x(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx_y = [get_and_set_idx_y(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx = [(idx_x, idx_y) for idx_x, idx_y in zip(all_bboxes_idx_x, all_bboxes_idx_y)]
-
- all_bboxes_idx = [idx_x_y[0] * 100000 + idx_x_y[1] for idx_x_y in all_bboxes_idx] # 变换成一个点，保证能够先X，X相同时按Y排序
- all_bboxes_idx = list(zip(all_bboxes_idx, all_bboxes))
- all_bboxes_idx.sort(key=lambda x: x[0])
- sorted_bboxes = [bbox for idx, bbox in all_bboxes_idx]
- return sorted_bboxes
-
-
-################################################################################
-# 第二种排序算法
-# 下面的算法在计算idx_x和idx_y的时候不考虑延长线，而只考虑实际的长或者宽被遮挡的情况
-#
-################################################################################
-
-def find_left_nearest_bbox(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有右侧高度和this_bbox有重叠的bbox
- """
- left_boxes = [box for box in all_bboxes if box[X1_IDX] <= this_bbox[X0_IDX] and any([
- box[Y0_IDX] < this_bbox[Y0_IDX] < box[Y1_IDX], box[Y0_IDX] < this_bbox[Y1_IDX] < box[Y1_IDX],
- this_bbox[Y0_IDX] < box[Y0_IDX] < this_bbox[Y1_IDX], this_bbox[Y0_IDX] < box[Y1_IDX] < this_bbox[Y1_IDX],
- box[Y0_IDX]==this_bbox[Y0_IDX] and box[Y1_IDX]==this_bbox[Y1_IDX]])]
-
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个
- if len(left_boxes) > 0:
- left_boxes.sort(key=lambda x: x[X1_IDX], reverse=True)
- left_boxes = [left_boxes[0]]
- else:
- left_boxes = []
- return left_boxes
-
-
-def get_and_set_idx_x_2(this_bbox, all_bboxes):
- """
- 寻找this_bbox在all_bboxes中的被直接遮挡的深度 idx_x
- 这个遮挡深度不考虑延长线，而是被实际的长或者宽遮挡的情况
- """
- if this_bbox[IDX_X] is not None:
- return this_bbox[IDX_X]
- else:
- left_nearest_bbox = find_left_nearest_bbox(this_bbox, all_bboxes)
- if len(left_nearest_bbox) == 0:
- this_bbox[IDX_X] = 0
- else:
- left_idx_x = get_and_set_idx_x_2(left_nearest_bbox[0], all_bboxes)
- this_bbox[IDX_X] = left_idx_x + 1
- return this_bbox[IDX_X]
-
-
-def find_top_nearest_bbox(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有下侧宽度和this_bbox有重叠的bbox
- """
- top_boxes = [box for box in all_bboxes if box[Y1_IDX] <= this_bbox[Y0_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个
- if len(top_boxes) > 0:
- top_boxes.sort(key=lambda x: x[Y1_IDX], reverse=True)
- top_boxes = [top_boxes[0]]
- else:
- top_boxes = []
- return top_boxes
-
-
-def get_and_set_idx_y_2(this_bbox, all_bboxes):
- """
- 寻找this_bbox在all_bboxes中的被直接遮挡的深度 idx_y
- 这个遮挡深度不考虑延长线，而是被实际的长或者宽遮挡的情况
- """
- if this_bbox[IDX_Y] is not None:
- return this_bbox[IDX_Y]
- else:
- top_nearest_bbox = find_top_nearest_bbox(this_bbox, all_bboxes)
- if len(top_nearest_bbox) == 0:
- this_bbox[IDX_Y] = 0
- else:
- top_idx_y = get_and_set_idx_y_2(top_nearest_bbox[0], all_bboxes)
- this_bbox[IDX_Y] = top_idx_y + 1
- return this_bbox[IDX_Y]
-
-
-def paper_bbox_sort(all_bboxes: list, page_width, page_height):
- all_bboxes_idx_x = [get_and_set_idx_x_2(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx_y = [get_and_set_idx_y_2(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx = [(idx_x, idx_y) for idx_x, idx_y in zip(all_bboxes_idx_x, all_bboxes_idx_y)]
-
- all_bboxes_idx = [idx_x_y[0] * 100000 + idx_x_y[1] for idx_x_y in all_bboxes_idx] # 变换成一个点，保证能够先X，X相同时按Y排序
- all_bboxes_idx = list(zip(all_bboxes_idx, all_bboxes))
- all_bboxes_idx.sort(key=lambda x: x[0])
- sorted_bboxes = [bbox for idx, bbox in all_bboxes_idx]
- return sorted_bboxes
-
-################################################################################
-"""
-第三种排序算法, 假设page的最左侧为X0，最右侧为X1，最上侧为Y0，最下侧为Y1
-这个排序算法在第二种算法基础上增加对bbox的预处理步骤。预处理思路如下：
-1. 首先在水平方向上对bbox进行扩展。扩展方法是：
- - 对每个bbox，找到其左边最近的bbox（也就是y方向有重叠），然后将其左边界扩展到左边最近bbox的右边界(x1+1),这里加1是为了避免重叠。如果没有左边的bbox，那么就将其左边界扩展到page的最左侧X0。
- - 对每个bbox，找到其右边最近的bbox（也就是y方向有重叠），然后将其右边界扩展到右边最近bbox的左边界(x0-1),这里减1是为了避免重叠。如果没有右边的bbox，那么就将其右边界扩展到page的最右侧X1。
- - 经过上面2个步骤，bbox扩展到了水平方向的最大范围。[左最近bbox.x1+1, 右最近bbox.x0-1]
-
-2. 合并所有的连续水平方向的bbox, 合并方法是：
- - 对bbox进行y方向排序，然后从上到下遍历所有bbox，如果当前bbox和下一个bbox的x0, x1等于X0, X1，那么就合并这两个bbox。
-
-3. 然后在垂直方向上对bbox进行扩展。扩展方法是：
- - 首先从page上切割掉合并后的水平bbox, 得到几个新的block
- 针对每个block
- - x0: 扎到位于左侧x=x0延长线的左侧所有的bboxes, 找到最大的x1,让x0=x1+1。如果没有，则x0=X0
- - x1: 找到位于右侧x=x1延长线右侧所有的bboxes， 找到最小的x0, 让x1=x0-1。如果没有，则x1=X1
- 随后在垂直方向上合并所有的连续的block，方法如下：
- - 对block进行x方向排序，然后从左到右遍历所有block，如果当前block和下一个block的x0, x1相等，那么就合并这两个block。
- 如果垂直切分后所有小bbox都被分配到了一个block, 那么分割就完成了。这些合并后的block打上标签'GOOD_LAYOUT’
- 如果在某个垂直方向上无法被完全分割到一个block，那么就将这个block打上标签'BAD_LAYOUT'。
- 至此完成，一个页面的预处理，天然的block要么属于'GOOD_LAYOUT'，要么属于'BAD_LAYOUT'。针对含有'BAD_LAYOUT'的页面，可以先按照自上而下，自左到右进行天然排序，也可以先过滤掉这种书籍。
- (完成条件下次加强：进行水平方向切分，把混乱的layout部分尽可能切割出去)
-"""
-################################################################################
-def find_left_neighbor_bboxes(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有右侧高度和this_bbox有重叠的bbox
- 这里使用扩展之后的bbox
- """
- left_boxes = [box for box in all_bboxes if box[X1_EXT_IDX] <= this_bbox[X0_EXT_IDX] and any([
- box[Y0_EXT_IDX] < this_bbox[Y0_EXT_IDX] < box[Y1_EXT_IDX], box[Y0_EXT_IDX] < this_bbox[Y1_EXT_IDX] < box[Y1_EXT_IDX],
- this_bbox[Y0_EXT_IDX] < box[Y0_EXT_IDX] < this_bbox[Y1_EXT_IDX], this_bbox[Y0_EXT_IDX] < box[Y1_EXT_IDX] < this_bbox[Y1_EXT_IDX],
- box[Y0_EXT_IDX]==this_bbox[Y0_EXT_IDX] and box[Y1_EXT_IDX]==this_bbox[Y1_EXT_IDX]])]
-
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个
- if len(left_boxes) > 0:
- left_boxes.sort(key=lambda x: x[X1_EXT_IDX], reverse=True)
- left_boxes = left_boxes
- else:
- left_boxes = []
- return left_boxes
-
-def find_top_neighbor_bboxes(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有下侧宽度和this_bbox有重叠的bbox
- 这里使用扩展之后的bbox
- """
- top_boxes = [box for box in all_bboxes if box[Y1_EXT_IDX] <= this_bbox[Y0_EXT_IDX] and any([
- box[X0_EXT_IDX] < this_bbox[X0_EXT_IDX] < box[X1_EXT_IDX], box[X0_EXT_IDX] < this_bbox[X1_EXT_IDX] < box[X1_EXT_IDX],
- this_bbox[X0_EXT_IDX] < box[X0_EXT_IDX] < this_bbox[X1_EXT_IDX], this_bbox[X0_EXT_IDX] < box[X1_EXT_IDX] < this_bbox[X1_EXT_IDX],
- box[X0_EXT_IDX]==this_bbox[X0_EXT_IDX] and box[X1_EXT_IDX]==this_bbox[X1_EXT_IDX]])]
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个
- if len(top_boxes) > 0:
- top_boxes.sort(key=lambda x: x[Y1_EXT_IDX], reverse=True)
- top_boxes = top_boxes
- else:
- top_boxes = []
- return top_boxes
-
-def get_and_set_idx_x_2_ext(this_bbox, all_bboxes):
- """
- 寻找this_bbox在all_bboxes中的被直接遮挡的深度 idx_x
- 这个遮挡深度不考虑延长线，而是被实际的长或者宽遮挡的情况
- """
- if this_bbox[IDX_X] is not None:
- return this_bbox[IDX_X]
- else:
- left_nearest_bbox = find_left_neighbor_bboxes(this_bbox, all_bboxes)
- if len(left_nearest_bbox) == 0:
- this_bbox[IDX_X] = 0
- else:
- left_idx_x = [get_and_set_idx_x_2(b, all_bboxes) for b in left_nearest_bbox]
- this_bbox[IDX_X] = mymax(left_idx_x) + 1
- return this_bbox[IDX_X]
-
-def get_and_set_idx_y_2_ext(this_bbox, all_bboxes):
- """
- 寻找this_bbox在all_bboxes中的被直接遮挡的深度 idx_y
- 这个遮挡深度不考虑延长线，而是被实际的长或者宽遮挡的情况
- """
- if this_bbox[IDX_Y] is not None:
- return this_bbox[IDX_Y]
- else:
- top_nearest_bbox = find_top_neighbor_bboxes(this_bbox, all_bboxes)
- if len(top_nearest_bbox) == 0:
- this_bbox[IDX_Y] = 0
- else:
- top_idx_y = [get_and_set_idx_y_2_ext(b, all_bboxes) for b in top_nearest_bbox]
- this_bbox[IDX_Y] = mymax(top_idx_y) + 1
- return this_bbox[IDX_Y]
-
-def _paper_bbox_sort_ext(all_bboxes: list):
- all_bboxes_idx_x = [get_and_set_idx_x_2_ext(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx_y = [get_and_set_idx_y_2_ext(bbox, all_bboxes) for bbox in all_bboxes]
- all_bboxes_idx = [(idx_x, idx_y) for idx_x, idx_y in zip(all_bboxes_idx_x, all_bboxes_idx_y)]
-
- all_bboxes_idx = [idx_x_y[0] * 100000 + idx_x_y[1] for idx_x_y in all_bboxes_idx] # 变换成一个点，保证能够先X，X相同时按Y排序
- all_bboxes_idx = list(zip(all_bboxes_idx, all_bboxes))
- all_bboxes_idx.sort(key=lambda x: x[0])
- sorted_bboxes = [bbox for idx, bbox in all_bboxes_idx]
- return sorted_bboxes
-
-# ===============================================================================================
-def find_left_bbox_ext_line(this_bbox, all_bboxes) -> list:
- """
- 寻找this_bbox左边的所有bbox, 使用延长线
- """
- left_boxes = [box for box in all_bboxes if box[X1_IDX] <= this_bbox[X0_IDX]]
- if len(left_boxes):
- left_boxes.sort(key=lambda x: x[X1_IDX], reverse=True)
- left_boxes = left_boxes[0]
- else:
- left_boxes = None
-
- return left_boxes
-
-def find_right_bbox_ext_line(this_bbox, all_bboxes) -> list:
- """
- 寻找this_bbox右边的所有bbox, 使用延长线
- """
- right_boxes = [box for box in all_bboxes if box[X0_IDX] >= this_bbox[X1_IDX]]
- if len(right_boxes):
- right_boxes.sort(key=lambda x: x[X0_IDX])
- right_boxes = right_boxes[0]
- else:
- right_boxes = None
- return right_boxes
-
-# =============================================================================================
-
-def find_left_nearest_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有右侧高度和this_bbox有重叠的bbox， 不用延长线并且不能像
- """
- left_boxes = [box for box in all_bboxes if box[X1_IDX] <= this_bbox[X0_IDX] and any([
- box[Y0_IDX] < this_bbox[Y0_IDX] < box[Y1_IDX], box[Y0_IDX] < this_bbox[Y1_IDX] < box[Y1_IDX],
- this_bbox[Y0_IDX] < box[Y0_IDX] < this_bbox[Y1_IDX], this_bbox[Y0_IDX] < box[Y1_IDX] < this_bbox[Y1_IDX],
- box[Y0_IDX]==this_bbox[Y0_IDX] and box[Y1_IDX]==this_bbox[Y1_IDX]])]
-
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个——x1最大的那个
- if len(left_boxes) > 0:
- left_boxes.sort(key=lambda x: x[X1_EXT_IDX] if x[X1_EXT_IDX] else x[X1_IDX], reverse=True)
- left_boxes = left_boxes[0]
- else:
- left_boxes = None
- return left_boxes
-
-def find_right_nearst_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox右侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- right_bboxes = [box for box in all_bboxes if box[X0_IDX] >= this_bbox[X1_IDX] and any([
- this_bbox[Y0_IDX] < box[Y0_IDX] < this_bbox[Y1_IDX], this_bbox[Y0_IDX] < box[Y1_IDX] < this_bbox[Y1_IDX],
- box[Y0_IDX] < this_bbox[Y0_IDX] < box[Y1_IDX], box[Y0_IDX] < this_bbox[Y1_IDX] < box[Y1_IDX],
- box[Y0_IDX]==this_bbox[Y0_IDX] and box[Y1_IDX]==this_bbox[Y1_IDX]])]
-
- if len(right_bboxes)>0:
- right_bboxes.sort(key=lambda x: x[X0_EXT_IDX] if x[X0_EXT_IDX] else x[X0_IDX])
- right_bboxes = right_bboxes[0]
- else:
- right_bboxes = None
- return right_bboxes
-
-def reset_idx_x_y(all_boxes:list)->list:
- for box in all_boxes:
- box[IDX_X] = None
- box[IDX_Y] = None
-
- return all_boxes
-
-# ===================================================================================================
-def find_top_nearest_bbox_direct(this_bbox, bboxes_collection) -> list:
- """
- 找到在this_bbox上方且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- top_bboxes = [box for box in bboxes_collection if box[Y1_IDX] <= this_bbox[Y0_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
- # 然后再过滤一下，找到上方距离this_bbox最近的那个
- if len(top_bboxes) > 0:
- top_bboxes.sort(key=lambda x: x[Y1_IDX], reverse=True)
- top_bboxes = top_bboxes[0]
- else:
- top_bboxes = None
- return top_bboxes
-
-def find_bottom_nearest_bbox_direct(this_bbox, bboxes_collection) -> list:
- """
- 找到在this_bbox下方且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- bottom_bboxes = [box for box in bboxes_collection if box[Y0_IDX] >= this_bbox[Y1_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个
- if len(bottom_bboxes) > 0:
- bottom_bboxes.sort(key=lambda x: x[Y0_IDX])
- bottom_bboxes = bottom_bboxes[0]
- else:
- bottom_bboxes = None
- return bottom_bboxes
-
-def find_boundry_bboxes(bboxes:list) -> tuple:
- """
- 找到bboxes的边界——找到所有bbox里最小的(x0, y0), 最大的(x1, y1)
- """
- x0, y0, x1, y1 = bboxes[0][X0_IDX], bboxes[0][Y0_IDX], bboxes[0][X1_IDX], bboxes[0][Y1_IDX]
- for box in bboxes:
- x0 = min(box[X0_IDX], x0)
- y0 = min(box[Y0_IDX], y0)
- x1 = max(box[X1_IDX], x1)
- y1 = max(box[Y1_IDX], y1)
-
- return x0, y0, x1, y1
-
-
-def extend_bbox_vertical(bboxes:list, boundry_x0, boundry_y0, boundry_x1, boundry_y1) -> list:
- """
- 在垂直方向上扩展能够直接垂直打通的bbox,也就是那些上下都没有其他box的bbox
- """
- for box in bboxes:
- top_nearest_bbox = find_top_nearest_bbox_direct(box, bboxes)
- bottom_nearest_bbox = find_bottom_nearest_bbox_direct(box, bboxes)
- if top_nearest_bbox is None and bottom_nearest_bbox is None: # 独占一列
- box[X0_EXT_IDX] = box[X0_IDX]
- box[Y0_EXT_IDX] = boundry_y0
- box[X1_EXT_IDX] = box[X1_IDX]
- box[Y1_EXT_IDX] = boundry_y1
- # else:
- # if top_nearest_bbox is None:
- # box[Y0_EXT_IDX] = boundry_y0
- # else:
- # box[Y0_EXT_IDX] = top_nearest_bbox[Y1_IDX] + 1
- # if bottom_nearest_bbox is None:
- # box[Y1_EXT_IDX] = boundry_y1
- # else:
- # box[Y1_EXT_IDX] = bottom_nearest_bbox[Y0_IDX] - 1
- # box[X0_EXT_IDX] = box[X0_IDX]
- # box[X1_EXT_IDX] = box[X1_IDX]
- return bboxes
-
-
-# ===================================================================================================
-
-def paper_bbox_sort_v2(all_bboxes: list, page_width:int, page_height:int):
- """
- 增加预处理行为的排序:
- return:
- [
- {
- "layout_bbox": [x0, y0, x1, y1],
- "layout_label":"GOOD_LAYOUT/BAD_LAYOUT",
- "content_bboxes": [] #每个元素都是[x0, y0, x1, y1, block_content, idx_x, idx_y, content_type, ext_x0, ext_y0, ext_x1, ext_y1], 并且顺序就是阅读顺序
- }
- ]
- """
- sorted_layouts = [] # 最后的返回结果
- page_x0, page_y0, page_x1, page_y1 = 1, 1, page_width-1, page_height-1
-
- all_bboxes = paper_bbox_sort(all_bboxes) # 大致拍下序
- # 首先在水平方向上扩展独占一行的bbox
- for bbox in all_bboxes:
- left_nearest_bbox = find_left_nearest_bbox_direct(bbox, all_bboxes) # 非扩展线
- right_nearest_bbox = find_right_nearst_bbox_direct(bbox, all_bboxes)
- if left_nearest_bbox is None and right_nearest_bbox is None: # 独占一行
- bbox[X0_EXT_IDX] = page_x0
- bbox[Y0_EXT_IDX] = bbox[Y0_IDX]
- bbox[X1_EXT_IDX] = page_x1
- bbox[Y1_EXT_IDX] = bbox[Y1_IDX]
-
- # 此时独占一行的被成功扩展到指定的边界上，这个时候利用边界条件合并连续的bbox，成为一个group
- if len(all_bboxes)==1:
- return [{"layout_bbox": [page_x0, page_y0, page_x1, page_y1], "layout_label":"GOOD_LAYOUT", "content_bboxes": all_bboxes}]
- if len(all_bboxes)==0:
- return []
-
- """
- 然后合并所有连续水平方向的bbox.
-
- """
- all_bboxes.sort(key=lambda x: x[Y0_IDX])
- h_bboxes = []
- h_bbox_group = []
- v_boxes = []
-
- for bbox in all_bboxes:
- if bbox[X0_IDX] == page_x0 and bbox[X1_IDX] == page_x1:
- h_bbox_group.append(bbox)
- else:
- if len(h_bbox_group)>0:
- h_bboxes.append(h_bbox_group)
- h_bbox_group = []
- # 最后一个group
- if len(h_bbox_group)>0:
- h_bboxes.append(h_bbox_group)
-
- """
- 现在h_bboxes里面是所有的group了，每个group都是一个list
- 对h_bboxes里的每个group进行计算放回到sorted_layouts里
- """
- for gp in h_bboxes:
- gp.sort(key=lambda x: x[Y0_IDX])
- block_info = {"layout_label":"GOOD_LAYOUT", "content_bboxes": gp}
- # 然后计算这个group的layout_bbox，也就是最小的x0,y0, 最大的x1,y1
- x0, y0, x1, y1 = gp[0][X0_EXT_IDX], gp[0][Y0_EXT_IDX], gp[-1][X1_EXT_IDX], gp[-1][Y1_EXT_IDX]
- block_info["layout_bbox"] = [x0, y0, x1, y1]
- sorted_layouts.append(block_info)
-
- # 接下来利用这些连续的水平bbox的layout_bbox的y0, y1，从水平上切分开其余的为几个部分
- h_split_lines = [page_y0]
- for gp in h_bboxes:
- layout_bbox = gp['layout_bbox']
- y0, y1 = layout_bbox[1], layout_bbox[3]
- h_split_lines.append(y0)
- h_split_lines.append(y1)
- h_split_lines.append(page_y1)
-
- unsplited_bboxes = []
- for i in range(0, len(h_split_lines), 2):
- start_y0, start_y1 = h_split_lines[i:i+2]
- # 然后找出[start_y0, start_y1]之间的其他bbox，这些组成一个未分割板块
- bboxes_in_block = [bbox for bbox in all_bboxes if bbox[Y0_IDX]>=start_y0 and bbox[Y1_IDX]<=start_y1]
- unsplited_bboxes.append(bboxes_in_block)
- # ================== 至此，水平方向的 已经切分排序完毕====================================
- """
- 接下来针对每个非水平的部分切分垂直方向的
- 此时，只剩下了无法被完全水平打通的bbox了。对这些box，优先进行垂直扩展，然后进行垂直切分.
- 分3步：
- 1. 先把能完全垂直打通的隔离出去当做一个layout
- 2. 其余的先垂直切分
- 3. 垂直切分之后的部分再尝试水平切分
- 4. 剩下的不能被切分的各个部分当成一个layout
- """
- # 对每部分进行垂直切分
- for bboxes_in_block in unsplited_bboxes:
- # 首先对这个block的bbox进行垂直方向上的扩展
- boundry_x0, boundry_y0, boundry_x1, boundry_y1 = find_boundry_bboxes(bboxes_in_block)
- # 进行垂直方向上的扩展
- extended_vertical_bboxes = extend_bbox_vertical(bboxes_in_block, boundry_x0, boundry_y0, boundry_x1, boundry_y1)
- # 然后对这个block进行垂直方向上的切分
- extend_bbox_vertical.sort(key=lambda x: x[X0_IDX]) # x方向上从小到大，代表了从左到右读取
- v_boxes_group = []
- for bbox in extended_vertical_bboxes:
- if bbox[Y0_IDX]==boundry_y0 and bbox[Y1_IDX]==boundry_y1:
- v_boxes_group.append(bbox)
- else:
- if len(v_boxes_group)>0:
- v_boxes.append(v_boxes_group)
- v_boxes_group = []
-
- if len(v_boxes_group)>0:
-
- v_boxes.append(v_boxes_group)
-
- # 把连续的垂直部分加入到sorted_layouts里。注意这个时候已经是连续的垂直部分了，因为上面已经做了
- for gp in v_boxes:
- gp.sort(key=lambda x: x[X0_IDX])
- block_info = {"layout_label":"GOOD_LAYOUT", "content_bboxes": gp}
- # 然后计算这个group的layout_bbox，也就是最小的x0,y0, 最大的x1,y1
- x0, y0, x1, y1 = gp[0][X0_EXT_IDX], gp[0][Y0_EXT_IDX], gp[-1][X1_EXT_IDX], gp[-1][Y1_EXT_IDX]
- block_info["layout_bbox"] = [x0, y0, x1, y1]
- sorted_layouts.append(block_info)
-
- # 在垂直方向上，划分子块，也就是用贯通的垂直线进行切分。这些被切分出来的块，极大可能是可被垂直切分的，如果不能完全的垂直切分，那么尝试水平切分。都不能的则当成一个layout
- v_split_lines = [boundry_x0]
- for gp in v_boxes:
- layout_bbox = gp['layout_bbox']
- x0, x1 = layout_bbox[0], layout_bbox[2]
- v_split_lines.append(x0)
- v_split_lines.append(x1)
- v_split_lines.append(boundry_x1)
-
- reset_idx_x_y(all_bboxes)
- all_boxes = _paper_bbox_sort_ext(all_bboxes)
- return all_boxes
-
-
-
-
-
-
-
-
diff --git a/magic_pdf/layout/layout_det_utils.py b/magic_pdf/layout/layout_det_utils.py
deleted file mode 100644
index 8b2b36cc0..000000000
--- a/magic_pdf/layout/layout_det_utils.py
+++ /dev/null
@@ -1,182 +0,0 @@
-from magic_pdf.layout.bbox_sort import X0_EXT_IDX, X0_IDX, X1_EXT_IDX, X1_IDX, Y0_IDX, Y1_EXT_IDX, Y1_IDX
-from magic_pdf.libs.boxbase import _is_bottom_full_overlap, _left_intersect, _right_intersect
-
-
-def find_all_left_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 在all_bboxes里找到所有右侧垂直方向上和this_bbox有重叠的bbox， 不用延长线
- 并且要考虑两个box左右相交的情况，如果相交了，那么右侧的box就不算最左侧。
- """
- left_boxes = [box for box in all_bboxes if box[X1_IDX] <= this_bbox[X0_IDX]
- and any([
- box[Y0_IDX] < this_bbox[Y0_IDX] < box[Y1_IDX], box[Y0_IDX] < this_bbox[Y1_IDX] < box[Y1_IDX],
- this_bbox[Y0_IDX] < box[Y0_IDX] < this_bbox[Y1_IDX], this_bbox[Y0_IDX] < box[Y1_IDX] < this_bbox[Y1_IDX],
- box[Y0_IDX]==this_bbox[Y0_IDX] and box[Y1_IDX]==this_bbox[Y1_IDX]]) or _left_intersect(box[:4], this_bbox[:4])]
-
- # 然后再过滤一下，找到水平上距离this_bbox最近的那个——x1最大的那个
- if len(left_boxes) > 0:
- left_boxes.sort(key=lambda x: x[X1_EXT_IDX] if x[X1_EXT_IDX] else x[X1_IDX], reverse=True)
- left_boxes = left_boxes[0]
- else:
- left_boxes = None
- return left_boxes
-
-def find_all_right_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox右侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- right_bboxes = [box for box in all_bboxes if box[X0_IDX] >= this_bbox[X1_IDX]
- and any([
- this_bbox[Y0_IDX] < box[Y0_IDX] < this_bbox[Y1_IDX], this_bbox[Y0_IDX] < box[Y1_IDX] < this_bbox[Y1_IDX],
- box[Y0_IDX] < this_bbox[Y0_IDX] < box[Y1_IDX], box[Y0_IDX] < this_bbox[Y1_IDX] < box[Y1_IDX],
- box[Y0_IDX]==this_bbox[Y0_IDX] and box[Y1_IDX]==this_bbox[Y1_IDX]]) or _right_intersect(this_bbox[:4], box[:4])]
-
- if len(right_bboxes)>0:
- right_bboxes.sort(key=lambda x: x[X0_EXT_IDX] if x[X0_EXT_IDX] else x[X0_IDX])
- right_bboxes = right_bboxes[0]
- else:
- right_bboxes = None
- return right_bboxes
-
-def find_all_top_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox上侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- top_bboxes = [box for box in all_bboxes if box[Y1_IDX] <= this_bbox[Y0_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(top_bboxes)>0:
- top_bboxes.sort(key=lambda x: x[Y1_EXT_IDX] if x[Y1_EXT_IDX] else x[Y1_IDX], reverse=True)
- top_bboxes = top_bboxes[0]
- else:
- top_bboxes = None
- return top_bboxes
-
-def find_all_bottom_bbox_direct(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox下侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- bottom_bboxes = [box for box in all_bboxes if box[Y0_IDX] >= this_bbox[Y1_IDX] and any([
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(bottom_bboxes)>0:
- bottom_bboxes.sort(key=lambda x: x[Y0_IDX])
- bottom_bboxes = bottom_bboxes[0]
- else:
- bottom_bboxes = None
- return bottom_bboxes
-
-# ===================================================================================================================
-def find_bottom_bbox_direct_from_right_edge(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox下侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- bottom_bboxes = [box for box in all_bboxes if box[Y0_IDX] >= this_bbox[Y1_IDX] and any([
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(bottom_bboxes)>0:
- # y0最小， X1最大的那个,也就是box上边缘最靠近this_bbox的那个,并且还最靠右
- bottom_bboxes.sort(key=lambda x: x[Y0_IDX])
- bottom_bboxes = [box for box in bottom_bboxes if box[Y0_IDX]==bottom_bboxes[0][Y0_IDX]]
- # 然后再y1相同的情况下，找到x1最大的那个
- bottom_bboxes.sort(key=lambda x: x[X1_IDX], reverse=True)
- bottom_bboxes = bottom_bboxes[0]
- else:
- bottom_bboxes = None
- return bottom_bboxes
-
-def find_bottom_bbox_direct_from_left_edge(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox下侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- bottom_bboxes = [box for box in all_bboxes if box[Y0_IDX] >= this_bbox[Y1_IDX] and any([
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(bottom_bboxes)>0:
- # y0最小， X0最小的那个
- bottom_bboxes.sort(key=lambda x: x[Y0_IDX])
- bottom_bboxes = [box for box in bottom_bboxes if box[Y0_IDX]==bottom_bboxes[0][Y0_IDX]]
- # 然后再y0相同的情况下，找到x0最小的那个
- bottom_bboxes.sort(key=lambda x: x[X0_IDX])
- bottom_bboxes = bottom_bboxes[0]
- else:
- bottom_bboxes = None
- return bottom_bboxes
-
-def find_top_bbox_direct_from_left_edge(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox上侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- top_bboxes = [box for box in all_bboxes if box[Y1_IDX] <= this_bbox[Y0_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(top_bboxes)>0:
- # y1最大， X0最小的那个
- top_bboxes.sort(key=lambda x: x[Y1_IDX], reverse=True)
- top_bboxes = [box for box in top_bboxes if box[Y1_IDX]==top_bboxes[0][Y1_IDX]]
- # 然后再y1相同的情况下，找到x0最小的那个
- top_bboxes.sort(key=lambda x: x[X0_IDX])
- top_bboxes = top_bboxes[0]
- else:
- top_bboxes = None
- return top_bboxes
-
-def find_top_bbox_direct_from_right_edge(this_bbox, all_bboxes) -> list:
- """
- 找到在this_bbox上侧且距离this_bbox距离最近的bbox.必须是直接遮挡的那种
- """
- top_bboxes = [box for box in all_bboxes if box[Y1_IDX] <= this_bbox[Y0_IDX] and any([
- box[X0_IDX] < this_bbox[X0_IDX] < box[X1_IDX], box[X0_IDX] < this_bbox[X1_IDX] < box[X1_IDX],
- this_bbox[X0_IDX] < box[X0_IDX] < this_bbox[X1_IDX], this_bbox[X0_IDX] < box[X1_IDX] < this_bbox[X1_IDX],
- box[X0_IDX]==this_bbox[X0_IDX] and box[X1_IDX]==this_bbox[X1_IDX]])]
-
- if len(top_bboxes)>0:
- # y1最大， X1最大的那个
- top_bboxes.sort(key=lambda x: x[Y1_IDX], reverse=True)
- top_bboxes = [box for box in top_bboxes if box[Y1_IDX]==top_bboxes[0][Y1_IDX]]
- # 然后再y1相同的情况下，找到x1最大的那个
- top_bboxes.sort(key=lambda x: x[X1_IDX], reverse=True)
- top_bboxes = top_bboxes[0]
- else:
- top_bboxes = None
- return top_bboxes
-
-# ===================================================================================================================
-
-def get_left_edge_bboxes(all_bboxes) -> list:
- """
- 返回最左边的bbox
- """
- left_bboxes = [box for box in all_bboxes if find_all_left_bbox_direct(box, all_bboxes) is None]
- return left_bboxes
-
-def get_right_edge_bboxes(all_bboxes) -> list:
- """
- 返回最右边的bbox
- """
- right_bboxes = [box for box in all_bboxes if find_all_right_bbox_direct(box, all_bboxes) is None]
- return right_bboxes
-
-def fix_vertical_bbox_pos(bboxes:list):
- """
- 检查这批bbox在垂直方向是否有轻微的重叠，如果重叠了，就把重叠的bbox往下移动一点
- 在x方向上必须一个包含或者被包含，或者完全重叠，不能只有部分重叠
- """
- bboxes.sort(key=lambda x: x[Y0_IDX]) # 从上向下排列
- for i in range(0, len(bboxes)):
- for j in range(i+1, len(bboxes)):
- if _is_bottom_full_overlap(bboxes[i][:4], bboxes[j][:4]):
- # 如果两个bbox有部分重叠，那么就把下面的bbox往下移动一点
- bboxes[j][Y0_IDX] = bboxes[i][Y1_IDX] + 2 # 2是个经验值
- break
- return bboxes
diff --git a/magic_pdf/layout/layout_sort.py b/magic_pdf/layout/layout_sort.py
deleted file mode 100644
index 383ea5bfd..000000000
--- a/magic_pdf/layout/layout_sort.py
+++ /dev/null
@@ -1,921 +0,0 @@
-"""对pdf上的box进行layout识别，并对内部组成的box进行排序."""
-
-from loguru import logger
-
-from magic_pdf.layout.bbox_sort import (CONTENT_IDX, CONTENT_TYPE_IDX,
- X0_EXT_IDX, X0_IDX, X1_EXT_IDX, X1_IDX,
- Y0_EXT_IDX, Y0_IDX, Y1_EXT_IDX, Y1_IDX,
- paper_bbox_sort)
-from magic_pdf.layout.layout_det_utils import (
- find_all_bottom_bbox_direct, find_all_left_bbox_direct,
- find_all_right_bbox_direct, find_all_top_bbox_direct,
- find_bottom_bbox_direct_from_left_edge,
- find_bottom_bbox_direct_from_right_edge,
- find_top_bbox_direct_from_left_edge, find_top_bbox_direct_from_right_edge,
- get_left_edge_bboxes, get_right_edge_bboxes)
-from magic_pdf.libs.boxbase import get_bbox_in_boundary
-
-LAYOUT_V = 'V'
-LAYOUT_H = 'H'
-LAYOUT_UNPROC = 'U'
-LAYOUT_BAD = 'B'
-
-
-def _is_single_line_text(bbox):
- """检查bbox里面的文字是否只有一行."""
- return True # TODO
- box_type = bbox[CONTENT_TYPE_IDX]
- if box_type != 'text':
- return False
- paras = bbox[CONTENT_IDX]['paras']
- text_content = ''
- for para_id, para in paras.items(): # 拼装内部的段落文本
- is_title = para['is_title']
- if is_title != 0:
- text_content += f"## {para['text']}"
- else:
- text_content += para['text']
- text_content += '\n\n'
-
- return bbox[CONTENT_TYPE_IDX] == 'text' and len(text_content.split('\n\n')) <= 1
-
-
-def _horizontal_split(bboxes: list, boundary: tuple, avg_font_size=20) -> list:
- """
- 对bboxes进行水平切割
- 方法是：找到左侧和右侧都没有被直接遮挡的box，然后进行扩展，之后进行切割
- return:
- 返回几个大的Layout区域 [[x0, y0, x1, y1, "h|u|v"], ], h代表水平，u代表未探测的，v代表垂直布局
- """
- sorted_layout_blocks = [] # 这是要最终返回的值
-
- bound_x0, bound_y0, bound_x1, bound_y1 = boundary
- all_bboxes = get_bbox_in_boundary(bboxes, boundary)
- # all_bboxes = paper_bbox_sort(all_bboxes, abs(bound_x1-bound_x0), abs(bound_y1-bound_x0)) # 大致拍下序, 这个是基于直接遮挡的。
- """
- 首先在水平方向上扩展独占一行的bbox
-
- """
- last_h_split_line_y1 = bound_y0 # 记录下上次的水平分割线
- for i, bbox in enumerate(all_bboxes):
- left_nearest_bbox = find_all_left_bbox_direct(bbox, all_bboxes) # 非扩展线
- right_nearest_bbox = find_all_right_bbox_direct(bbox, all_bboxes)
- if left_nearest_bbox is None and right_nearest_bbox is None: # 独占一行
- """
- 然而，如果只是孤立的一行文字，那么就还要满足以下几个条件才可以：
- 1. bbox和中心线相交。或者
- 2. 上方或者下方也存在同类水平的独占一行的bbox。 或者
- 3. TODO 加强条件：这个bbox上方和下方是同一列column，那么就不能算作独占一行
- """
- # 先检查这个bbox里是否只包含一行文字
- # is_single_line = _is_single_line_text(bbox)
- """
- 这里有个点需要注意，当页面内容不是居中的时候，第一次调用传递的是page的boundary，这个时候mid_x就不是中心线了.
- 所以这里计算出最紧致的boundary，然后再计算mid_x
- """
- boundary_real_x0, boundary_real_x1 = min(
- [bbox[X0_IDX] for bbox in all_bboxes]
- ), max([bbox[X1_IDX] for bbox in all_bboxes])
- mid_x = (boundary_real_x0 + boundary_real_x1) / 2
- # 检查这个box是否内容在中心线有交
- # 必须跨过去2个字符的宽度
- is_cross_boundary_mid_line = (
- min(mid_x - bbox[X0_IDX], bbox[X1_IDX] - mid_x) > avg_font_size * 2
- )
- """
- 检查条件2
- """
- is_belong_to_col = False
- """
- 检查是否能被上方col吸收，方法是：
- 1. 上方非空且不是独占一行的，并且
- 2. 从上个水平分割的最大y=y1开始到当前bbox,最左侧的bbox的[min_x0, max_x1],能够覆盖当前box的[x0, x1]
- """
- """
- 以迭代的方式向上找，查找范围是[bound_x0, last_h_sp, bound_x1, bbox[Y0_IDX]]
- """
- # 先确定上方的y0, y0
- b_y0, b_y1 = last_h_split_line_y1, bbox[Y0_IDX]
- # 然后从box开始逐个向上找到所有与box在x上有交集的box
- box_to_check = [bound_x0, b_y0, bound_x1, b_y1]
- bbox_in_bound_check = get_bbox_in_boundary(all_bboxes, box_to_check)
-
- bboxes_on_top = []
- virtual_box = bbox
- while True:
- b_on_top = find_all_top_bbox_direct(virtual_box, bbox_in_bound_check)
- if b_on_top is not None:
- bboxes_on_top.append(b_on_top)
- virtual_box = [
- min([virtual_box[X0_IDX], b_on_top[X0_IDX]]),
- min(virtual_box[Y0_IDX], b_on_top[Y0_IDX]),
- max([virtual_box[X1_IDX], b_on_top[X1_IDX]]),
- b_y1,
- ]
- else:
- break
-
- # 随后确定这些box的最小x0, 最大x1
- if len(bboxes_on_top) > 0 and len(bboxes_on_top) != len(
- bbox_in_bound_check
- ): # virtual_box可能会膨胀到占满整个区域，这实际上就不能属于一个col了。
- min_x0, max_x1 = virtual_box[X0_IDX], virtual_box[X1_IDX]
- # 然后采用一种比较粗糙的方法，看min_x0，max_x1是否与位于[bound_x0, last_h_sp, bound_x1, bbox[Y0_IDX]]之间的box有相交
-
- if not any(
- [
- b[X0_IDX] <= min_x0 - 1 <= b[X1_IDX]
- or b[X0_IDX] <= max_x1 + 1 <= b[X1_IDX]
- for b in bbox_in_bound_check
- ]
- ):
- # 其上，下都不能被扩展成行，暂时只检查一下上方 TODO
- top_nearest_bbox = find_all_top_bbox_direct(bbox, bboxes)
- bottom_nearest_bbox = find_all_bottom_bbox_direct(bbox, bboxes)
- if not any(
- [
- top_nearest_bbox is not None
- and (
- find_all_left_bbox_direct(top_nearest_bbox, bboxes)
- is None
- and find_all_right_bbox_direct(top_nearest_bbox, bboxes)
- is None
- ),
- bottom_nearest_bbox is not None
- and (
- find_all_left_bbox_direct(bottom_nearest_bbox, bboxes)
- is None
- and find_all_right_bbox_direct(
- bottom_nearest_bbox, bboxes
- )
- is None
- ),
- top_nearest_bbox is None or bottom_nearest_bbox is None,
- ]
- ):
- is_belong_to_col = True
-
- # 检查是否能被下方col吸收 TODO
- """
- 这里为什么没有is_cross_boundary_mid_line的条件呢？
- 确实有些杂志左右两栏宽度不是对称的。
- """
- if not is_belong_to_col or is_cross_boundary_mid_line:
- bbox[X0_EXT_IDX] = bound_x0
- bbox[Y0_EXT_IDX] = bbox[Y0_IDX]
- bbox[X1_EXT_IDX] = bound_x1
- bbox[Y1_EXT_IDX] = bbox[Y1_IDX]
- last_h_split_line_y1 = bbox[Y1_IDX] # 更新这条线
- else:
- continue
- """
- 此时独占一行的被成功扩展到指定的边界上，这个时候利用边界条件合并连续的bbox，成为一个group
- 然后合并所有连续水平方向的bbox.
- """
- all_bboxes.sort(key=lambda x: x[Y0_IDX])
- h_bboxes = []
- h_bbox_group = []
-
- for bbox in all_bboxes:
- if bbox[X0_EXT_IDX] == bound_x0 and bbox[X1_EXT_IDX] == bound_x1:
- h_bbox_group.append(bbox)
- else:
- if len(h_bbox_group) > 0:
- h_bboxes.append(h_bbox_group)
- h_bbox_group = []
- # 最后一个group
- if len(h_bbox_group) > 0:
- h_bboxes.append(h_bbox_group)
- """
- 现在h_bboxes里面是所有的group了，每个group都是一个list
- 对h_bboxes里的每个group进行计算放回到sorted_layouts里
- """
- h_layouts = []
- for gp in h_bboxes:
- gp.sort(key=lambda x: x[Y0_IDX])
- # 然后计算这个group的layout_bbox，也就是最小的x0,y0, 最大的x1,y1
- x0, y0, x1, y1 = (
- gp[0][X0_EXT_IDX],
- gp[0][Y0_EXT_IDX],
- gp[-1][X1_EXT_IDX],
- gp[-1][Y1_EXT_IDX],
- )
- h_layouts.append([x0, y0, x1, y1, LAYOUT_H]) # 水平的布局
- """
- 接下来利用这些连续的水平bbox的layout_bbox的y0, y1，从水平上切分开其余的为几个部分
- """
- h_split_lines = [bound_y0]
- for gp in h_bboxes: # gp是一个list[bbox_list]
- y0, y1 = gp[0][1], gp[-1][3]
- h_split_lines.append(y0)
- h_split_lines.append(y1)
- h_split_lines.append(bound_y1)
-
- unsplited_bboxes = []
- for i in range(0, len(h_split_lines), 2):
- start_y0, start_y1 = h_split_lines[i : i + 2]
- # 然后找出[start_y0, start_y1]之间的其他bbox，这些组成一个未分割板块
- bboxes_in_block = [
- bbox
- for bbox in all_bboxes
- if bbox[Y0_IDX] >= start_y0 and bbox[Y1_IDX] <= start_y1
- ]
- unsplited_bboxes.append(bboxes_in_block)
- # 接着把未处理的加入到h_layouts里
- for bboxes_in_block in unsplited_bboxes:
- if len(bboxes_in_block) == 0:
- continue
- x0, y0, x1, y1 = (
- bound_x0,
- min([bbox[Y0_IDX] for bbox in bboxes_in_block]),
- bound_x1,
- max([bbox[Y1_IDX] for bbox in bboxes_in_block]),
- )
- h_layouts.append([x0, y0, x1, y1, LAYOUT_UNPROC])
-
- h_layouts.sort(key=lambda x: x[1]) # 按照y0排序, 也就是从上到下的顺序
- """
- 转换成如下格式返回
- """
- for layout in h_layouts:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': layout[:4],
- 'layout_label': layout[4],
- 'sub_layout': [],
- }
- )
- return sorted_layout_blocks
-
-
-###############################################################################################
-#
-# 垂直方向的处理
-#
-#
-###############################################################################################
-def _vertical_align_split_v1(bboxes: list, boundary: tuple) -> list:
- """
- 计算垂直方向上的对齐， 并分割bboxes成layout。负责对一列多行的进行列维度分割。
- 如果不能完全分割，剩余部分作为layout_lable为u的layout返回
- -----------------------
- | | |
- | | |
- | | |
- | | |
- -------------------------
- 此函数会将：以上布局将会切分出来2列
- """
- sorted_layout_blocks = [] # 这是要最终返回的值
- new_boundary = [boundary[0], boundary[1], boundary[2], boundary[3]]
-
- v_blocks = []
- """
- 先从左到右切分
- """
- while True:
- all_bboxes = get_bbox_in_boundary(bboxes, new_boundary)
- left_edge_bboxes = get_left_edge_bboxes(all_bboxes)
- if len(left_edge_bboxes) == 0:
- break
- right_split_line_x1 = max([bbox[X1_IDX] for bbox in left_edge_bboxes]) + 1
- # 然后检查这条线能不与其他bbox的左边界相交或者重合
- if any(
- [bbox[X0_IDX] <= right_split_line_x1 <= bbox[X1_IDX] for bbox in all_bboxes]
- ):
- # 垂直切分线与某些box发生相交，说明无法完全垂直方向切分。
- break
- else: # 说明成功分割出一列
- # 找到左侧边界最靠左的bbox作为layout的x0
- layout_x0 = min(
- [bbox[X0_IDX] for bbox in left_edge_bboxes]
- ) # 这里主要是为了画出来有一定间距
- v_blocks.append(
- [
- layout_x0,
- new_boundary[1],
- right_split_line_x1,
- new_boundary[3],
- LAYOUT_V,
- ]
- )
- new_boundary[0] = right_split_line_x1 # 更新边界
- """
- 再从右到左切， 此时如果还是无法完全切分，那么剩余部分作为layout_lable为u的layout返回
- """
- unsplited_block = []
- while True:
- all_bboxes = get_bbox_in_boundary(bboxes, new_boundary)
- right_edge_bboxes = get_right_edge_bboxes(all_bboxes)
- if len(right_edge_bboxes) == 0:
- break
- left_split_line_x0 = min([bbox[X0_IDX] for bbox in right_edge_bboxes]) - 1
- # 然后检查这条线能不与其他bbox的左边界相交或者重合
- if any(
- [bbox[X0_IDX] <= left_split_line_x0 <= bbox[X1_IDX] for bbox in all_bboxes]
- ):
- # 这里是余下的
- unsplited_block.append(
- [
- new_boundary[0],
- new_boundary[1],
- new_boundary[2],
- new_boundary[3],
- LAYOUT_UNPROC,
- ]
- )
- break
- else:
- # 找到右侧边界最靠右的bbox作为layout的x1
- layout_x1 = max([bbox[X1_IDX] for bbox in right_edge_bboxes])
- v_blocks.append(
- [
- left_split_line_x0,
- new_boundary[1],
- layout_x1,
- new_boundary[3],
- LAYOUT_V,
- ]
- )
- new_boundary[2] = left_split_line_x0 # 更新右边界
- """
- 最后拼装成layout格式返回
- """
- for block in v_blocks:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': block[:4],
- 'layout_label': block[4],
- 'sub_layout': [],
- }
- )
- for block in unsplited_block:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': block[:4],
- 'layout_label': block[4],
- 'sub_layout': [],
- }
- )
-
- # 按照x0排序
- sorted_layout_blocks.sort(key=lambda x: x['layout_bbox'][0])
- return sorted_layout_blocks
-
-
-def _vertical_align_split_v2(bboxes: list, boundary: tuple) -> list:
- """改进的
- _vertical_align_split算法，原算法会因为第二列的box由于左侧没有遮挡被认为是左侧的一部分，导致整个layout多列被识别为一列。
- 利用从左上角的box开始向下看的方法，不断扩展w_x0, w_x1，直到不能继续向下扩展，或者到达边界下边界。"""
- sorted_layout_blocks = [] # 这是要最终返回的值
- new_boundary = [boundary[0], boundary[1], boundary[2], boundary[3]]
- bad_boxes = [] # 被割中的box
- v_blocks = []
- while True:
- all_bboxes = get_bbox_in_boundary(bboxes, new_boundary)
- if len(all_bboxes) == 0:
- break
- left_top_box = min(
- all_bboxes, key=lambda x: (x[X0_IDX], x[Y0_IDX])
- ) # 这里应该加强，检查一下必须是在第一列的 TODO
- start_box = [
- left_top_box[X0_IDX],
- left_top_box[Y0_IDX],
- left_top_box[X1_IDX],
- left_top_box[Y1_IDX],
- ]
- w_x0, w_x1 = left_top_box[X0_IDX], left_top_box[X1_IDX]
- """
- 然后沿着这个box线向下找最近的那个box, 然后扩展w_x0, w_x1
- 扩展之后，宽度会增加，随后用x=w_x1来检测在边界内是否有box与相交，如果相交，那么就说明不能再扩展了。
- 当不能扩展的时候就要看是否到达下边界：
- 1. 达到，那么更新左边界继续分下一个列
- 2. 没有达到，那么此时开始从右侧切分进入下面的循环里
- """
- while left_top_box is not None: # 向下去找
- virtual_box = [w_x0, left_top_box[Y0_IDX], w_x1, left_top_box[Y1_IDX]]
- left_top_box = find_bottom_bbox_direct_from_left_edge(
- virtual_box, all_bboxes
- )
- if left_top_box:
- w_x0, w_x1 = min(virtual_box[X0_IDX], left_top_box[X0_IDX]), max(
- [virtual_box[X1_IDX], left_top_box[X1_IDX]]
- )
- # 万一这个初始的box在column中间，那么还要向上看
- start_box = [
- w_x0,
- start_box[Y0_IDX],
- w_x1,
- start_box[Y1_IDX],
- ] # 扩展一下宽度更鲁棒
- left_top_box = find_top_bbox_direct_from_left_edge(start_box, all_bboxes)
- while left_top_box is not None: # 向上去找
- virtual_box = [w_x0, left_top_box[Y0_IDX], w_x1, left_top_box[Y1_IDX]]
- left_top_box = find_top_bbox_direct_from_left_edge(virtual_box, all_bboxes)
- if left_top_box:
- w_x0, w_x1 = min(virtual_box[X0_IDX], left_top_box[X0_IDX]), max(
- [virtual_box[X1_IDX], left_top_box[X1_IDX]]
- )
-
- # 检查相交
- if any([bbox[X0_IDX] <= w_x1 + 1 <= bbox[X1_IDX] for bbox in all_bboxes]):
- for b in all_bboxes:
- if b[X0_IDX] <= w_x1 + 1 <= b[X1_IDX]:
- bad_boxes.append([b[X0_IDX], b[Y0_IDX], b[X1_IDX], b[Y1_IDX]])
- break
- else: # 说明成功分割出一列
- v_blocks.append([w_x0, new_boundary[1], w_x1, new_boundary[3], LAYOUT_V])
- new_boundary[0] = w_x1 # 更新边界
- """
- 接着开始从右上角的box扫描
- """
- w_x0, w_x1 = 0, 0
- unsplited_block = []
- while True:
- all_bboxes = get_bbox_in_boundary(bboxes, new_boundary)
- if len(all_bboxes) == 0:
- break
- # 先找到X1最大的
- bbox_list_sorted = sorted(
- all_bboxes, key=lambda bbox: bbox[X1_IDX], reverse=True
- )
- # Then, find the boxes with the smallest Y0 value
- bigest_x1 = bbox_list_sorted[0][X1_IDX]
- boxes_with_bigest_x1 = [
- bbox for bbox in bbox_list_sorted if bbox[X1_IDX] == bigest_x1
- ] # 也就是最靠右的那些
- right_top_box = min(
- boxes_with_bigest_x1, key=lambda bbox: bbox[Y0_IDX]
- ) # y0最小的那个
- start_box = [
- right_top_box[X0_IDX],
- right_top_box[Y0_IDX],
- right_top_box[X1_IDX],
- right_top_box[Y1_IDX],
- ]
- w_x0, w_x1 = right_top_box[X0_IDX], right_top_box[X1_IDX]
-
- while right_top_box is not None:
- virtual_box = [w_x0, right_top_box[Y0_IDX], w_x1, right_top_box[Y1_IDX]]
- right_top_box = find_bottom_bbox_direct_from_right_edge(
- virtual_box, all_bboxes
- )
- if right_top_box:
- w_x0, w_x1 = min([w_x0, right_top_box[X0_IDX]]), max(
- [w_x1, right_top_box[X1_IDX]]
- )
- # 在向上扫描
- start_box = [
- w_x0,
- start_box[Y0_IDX],
- w_x1,
- start_box[Y1_IDX],
- ] # 扩展一下宽度更鲁棒
- right_top_box = find_top_bbox_direct_from_right_edge(start_box, all_bboxes)
- while right_top_box is not None:
- virtual_box = [w_x0, right_top_box[Y0_IDX], w_x1, right_top_box[Y1_IDX]]
- right_top_box = find_top_bbox_direct_from_right_edge(
- virtual_box, all_bboxes
- )
- if right_top_box:
- w_x0, w_x1 = min([w_x0, right_top_box[X0_IDX]]), max(
- [w_x1, right_top_box[X1_IDX]]
- )
-
- # 检查是否与其他box相交， 垂直切分线与某些box发生相交，说明无法完全垂直方向切分。
- if any([bbox[X0_IDX] <= w_x0 - 1 <= bbox[X1_IDX] for bbox in all_bboxes]):
- unsplited_block.append(
- [
- new_boundary[0],
- new_boundary[1],
- new_boundary[2],
- new_boundary[3],
- LAYOUT_UNPROC,
- ]
- )
- for b in all_bboxes:
- if b[X0_IDX] <= w_x0 - 1 <= b[X1_IDX]:
- bad_boxes.append([b[X0_IDX], b[Y0_IDX], b[X1_IDX], b[Y1_IDX]])
- break
- else: # 说明成功分割出一列
- v_blocks.append([w_x0, new_boundary[1], w_x1, new_boundary[3], LAYOUT_V])
- new_boundary[2] = w_x0
- """转换数据结构"""
- for block in v_blocks:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': block[:4],
- 'layout_label': block[4],
- 'sub_layout': [],
- }
- )
-
- for block in unsplited_block:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': block[:4],
- 'layout_label': block[4],
- 'sub_layout': [],
- 'bad_boxes': bad_boxes, # 记录下来，这个box是被割中的
- }
- )
-
- # 按照x0排序
- sorted_layout_blocks.sort(key=lambda x: x['layout_bbox'][0])
- return sorted_layout_blocks
-
-
-def _try_horizontal_mult_column_split(bboxes: list, boundary: tuple) -> list:
- """
- 尝试水平切分，如果切分不动，那就当一个BAD_LAYOUT返回
- ------------------
- | | |
- ------------------
- | | | | <- 这里是此函数要切分的场景
- ------------------
- | | |
- | | |
- """
- pass
-
-
-def _vertical_split(bboxes: list, boundary: tuple) -> list:
- """
- 从垂直方向进行切割，分block
- 这个版本里，如果垂直切分不动，那就当一个BAD_LAYOUT返回
-
- --------------------------
- | | |
- | | |
- | |
- 这种列是此函数要切分的 -> | |
- | |
- | | |
- | | |
- -------------------------
- """
- sorted_layout_blocks = [] # 这是要最终返回的值
-
- bound_x0, bound_y0, bound_x1, bound_y1 = boundary
- all_bboxes = get_bbox_in_boundary(bboxes, boundary)
- """
- all_bboxes = fix_vertical_bbox_pos(all_bboxes) # 垂直方向解覆盖
- all_bboxes = fix_hor_bbox_pos(all_bboxes) # 水平解覆盖
-
- 这两行代码目前先不执行，因为公式检测，表格检测还不是很成熟，导致非常多的textblock参与了运算，时间消耗太大。
- 这两行代码的作用是：
- 如果遇到互相重叠的bbox, 那么会把面积较小的box进行压缩，从而避免重叠。对布局切分来说带来正反馈。
- """
-
- # all_bboxes = paper_bbox_sort(all_bboxes, abs(bound_x1-bound_x0), abs(bound_y1-bound_x0)) # 大致拍下序, 这个是基于直接遮挡的。
- """
- 首先在垂直方向上扩展独占一行的bbox
-
- """
- for bbox in all_bboxes:
- top_nearest_bbox = find_all_top_bbox_direct(bbox, all_bboxes) # 非扩展线
- bottom_nearest_bbox = find_all_bottom_bbox_direct(bbox, all_bboxes)
- if (
- top_nearest_bbox is None
- and bottom_nearest_bbox is None
- and not any(
- [
- b[X0_IDX] < bbox[X1_IDX] < b[X1_IDX]
- or b[X0_IDX] < bbox[X0_IDX] < b[X1_IDX]
- for b in all_bboxes
- ]
- )
- ): # 独占一列, 且不和其他重叠
- bbox[X0_EXT_IDX] = bbox[X0_IDX]
- bbox[Y0_EXT_IDX] = bound_y0
- bbox[X1_EXT_IDX] = bbox[X1_IDX]
- bbox[Y1_EXT_IDX] = bound_y1
- """
- 此时独占一列的被成功扩展到指定的边界上，这个时候利用边界条件合并连续的bbox，成为一个group
- 然后合并所有连续垂直方向的bbox.
- """
- all_bboxes.sort(key=lambda x: x[X0_IDX])
- # fix: 这里水平方向的列不要合并成一个行，因为需要保证返回给下游的最小block，总是可以无脑从上到下阅读文字。
- v_bboxes = []
- for box in all_bboxes:
- if box[Y0_EXT_IDX] == bound_y0 and box[Y1_EXT_IDX] == bound_y1:
- v_bboxes.append(box)
- """
- 现在v_bboxes里面是所有的group了，每个group都是一个list
- 对v_bboxes里的每个group进行计算放回到sorted_layouts里
- """
- v_layouts = []
- for vbox in v_bboxes:
- # gp.sort(key=lambda x: x[X0_IDX])
- # 然后计算这个group的layout_bbox，也就是最小的x0,y0, 最大的x1,y1
- x0, y0, x1, y1 = (
- vbox[X0_EXT_IDX],
- vbox[Y0_EXT_IDX],
- vbox[X1_EXT_IDX],
- vbox[Y1_EXT_IDX],
- )
- v_layouts.append([x0, y0, x1, y1, LAYOUT_V]) # 垂直的布局
- """
- 接下来利用这些连续的垂直bbox的layout_bbox的x0, x1，从垂直上切分开其余的为几个部分
- """
- v_split_lines = [bound_x0]
- for gp in v_bboxes:
- x0, x1 = gp[X0_IDX], gp[X1_IDX]
- v_split_lines.append(x0)
- v_split_lines.append(x1)
- v_split_lines.append(bound_x1)
-
- unsplited_bboxes = []
- for i in range(0, len(v_split_lines), 2):
- start_x0, start_x1 = v_split_lines[i : i + 2]
- # 然后找出[start_x0, start_x1]之间的其他bbox，这些组成一个未分割板块
- bboxes_in_block = [
- bbox
- for bbox in all_bboxes
- if bbox[X0_IDX] >= start_x0 and bbox[X1_IDX] <= start_x1
- ]
- unsplited_bboxes.append(bboxes_in_block)
- # 接着把未处理的加入到v_layouts里
- for bboxes_in_block in unsplited_bboxes:
- if len(bboxes_in_block) == 0:
- continue
- x0, y0, x1, y1 = (
- min([bbox[X0_IDX] for bbox in bboxes_in_block]),
- bound_y0,
- max([bbox[X1_IDX] for bbox in bboxes_in_block]),
- bound_y1,
- )
- v_layouts.append(
- [x0, y0, x1, y1, LAYOUT_UNPROC]
- ) # 说明这篇区域未能够分析出可靠的版面
-
- v_layouts.sort(key=lambda x: x[0]) # 按照x0排序, 也就是从左到右的顺序
-
- for layout in v_layouts:
- sorted_layout_blocks.append(
- {
- 'layout_bbox': layout[:4],
- 'layout_label': layout[4],
- 'sub_layout': [],
- }
- )
- """
- 至此，垂直方向切成了2种类型，其一是独占一列的，其二是未处理的。
- 下面对这些未处理的进行垂直方向切分，这个切分要切出来类似“吕”这种类型的垂直方向的布局
- """
- for i, layout in enumerate(sorted_layout_blocks):
- if layout['layout_label'] == LAYOUT_UNPROC:
- x0, y0, x1, y1 = layout['layout_bbox']
- v_split_layouts = _vertical_align_split_v2(bboxes, [x0, y0, x1, y1])
- sorted_layout_blocks[i] = {
- 'layout_bbox': [x0, y0, x1, y1],
- 'layout_label': LAYOUT_H,
- 'sub_layout': v_split_layouts,
- }
- layout['layout_label'] = LAYOUT_H # 被垂线切分成了水平布局
-
- return sorted_layout_blocks
-
-
-def split_layout(bboxes: list, boundary: tuple, page_num: int) -> list:
- """
- 把bboxes切割成layout
- return:
- [
- {
- "layout_bbox": [x0,y0,x1,y1],
- "layout_label":"u|v|h|b", 未处理|垂直|水平|BAD_LAYOUT
- "sub_layout":[] #每个元素都是[
- x0,y0,
- x1,y1,
- block_content,
- idx_x,idx_y,
- content_type,
- ext_x0,ext_y0,
- ext_x1,ext_y1
- ], 并且顺序就是阅读顺序
- }
- ]
- example:
- [
- {
- "layout_bbox": [0, 0, 100, 100],
- "layout_label":"u|v|h|b",
- "sub_layout":[
-
- ]
- },
- {
- "layout_bbox": [0, 0, 100, 100],
- "layout_label":"u|v|h|b",
- "sub_layout":[
- {
- "layout_bbox": [0, 0, 100, 100],
- "layout_label":"u|v|h|b",
- "content_bboxes":[
- [],
- [],
- []
- ]
- },
- {
- "layout_bbox": [0, 0, 100, 100],
- "layout_label":"u|v|h|b",
- "sub_layout":[
-
- ]
- }
- }
- ]
- """
- sorted_layouts = [] # 最终返回的结果
-
- boundary_x0, boundary_y0, boundary_x1, boundary_y1 = boundary
- if len(bboxes) <= 1:
- return [
- {
- 'layout_bbox': [boundary_x0, boundary_y0, boundary_x1, boundary_y1],
- 'layout_label': LAYOUT_V,
- 'sub_layout': [],
- }
- ]
- """
- 接下来按照先水平后垂直的顺序进行切分
- """
- bboxes = paper_bbox_sort(
- bboxes, boundary_x1 - boundary_x0, boundary_y1 - boundary_y0
- )
- sorted_layouts = _horizontal_split(bboxes, boundary) # 通过水平分割出来的layout
- for i, layout in enumerate(sorted_layouts):
- x0, y0, x1, y1 = layout['layout_bbox']
- layout_type = layout['layout_label']
- if layout_type == LAYOUT_UNPROC: # 说明是非独占单行的，这些需要垂直切分
- v_split_layouts = _vertical_split(bboxes, [x0, y0, x1, y1])
- """
- 最后这里有个逻辑问题：如果这个函数只分离出来了一个column layout，那么这个layout分割肯定超出了算法能力范围。因为我们假定的是传进来的
- box已经把行全部剥离了，所以这里必须十多个列才可以。如果只剥离出来一个layout，并且是多个box，那么就说明这个layout是无法分割的，标记为LAYOUT_UNPROC
- """
- layout_label = LAYOUT_V
- if len(v_split_layouts) == 1:
- if len(v_split_layouts[0]['sub_layout']) == 0:
- layout_label = LAYOUT_UNPROC
- # logger.warning(f"WARNING: pageno={page_num}, 无法分割的layout: ", v_split_layouts)
- """
- 组合起来最终的layout
- """
- sorted_layouts[i] = {
- 'layout_bbox': [x0, y0, x1, y1],
- 'layout_label': layout_label,
- 'sub_layout': v_split_layouts,
- }
- layout['layout_label'] = LAYOUT_H
- """
- 水平和垂直方向都切分完毕了。此时还有一些未处理的，这些未处理的可能是因为水平和垂直方向都无法切分。
- 这些最后调用_try_horizontal_mult_block_split做一次水平多个block的联合切分，如果也不能切分最终就当做BAD_LAYOUT返回
- """
- # TODO
-
- return sorted_layouts
-
-
-def get_bboxes_layout(all_boxes: list, boundary: tuple, page_id: int):
- """
- 对利用layout排序之后的box，进行排序
- return:
- [
- {
- "layout_bbox": [x0, y0, x1, y1],
- "layout_label":"u|v|h|b", 未处理|垂直|水平|BAD_LAYOUT
- }，
- ]
- """
-
- def _preorder_traversal(layout):
- """对sorted_layouts的叶子节点，也就是len(sub_layout)==0的节点进行排序。排序按照前序遍历的顺序，也就是从上到
- 下，从左到右的顺序."""
- sorted_layout_blocks = []
- for layout in layout:
- sub_layout = layout['sub_layout']
- if len(sub_layout) == 0:
- sorted_layout_blocks.append(layout)
- else:
- s = _preorder_traversal(sub_layout)
- sorted_layout_blocks.extend(s)
- return sorted_layout_blocks
-
- # -------------------------------------------------------------------------------------------------------------------------
- sorted_layouts = split_layout(
- all_boxes, boundary, page_id
- ) # 先切分成layout，得到一个Tree
- total_sorted_layout_blocks = _preorder_traversal(sorted_layouts)
- return total_sorted_layout_blocks, sorted_layouts
-
-
-def get_columns_cnt_of_layout(layout_tree):
- """获取一个layout的宽度."""
- max_width_list = [0] # 初始化一个元素，防止max,min函数报错
-
- for items in layout_tree: # 针对每一层（横切）计算列数，横着的算一列
- layout_type = items['layout_label']
- sub_layouts = items['sub_layout']
- if len(sub_layouts) == 0:
- max_width_list.append(1)
- else:
- if layout_type == LAYOUT_H:
- max_width_list.append(1)
- else:
- width = 0
- for sub_layout in sub_layouts:
- if len(sub_layout['sub_layout']) == 0:
- width += 1
- else:
- for lay in sub_layout['sub_layout']:
- width += get_columns_cnt_of_layout([lay])
- max_width_list.append(width)
-
- return max(max_width_list)
-
-
-def sort_with_layout(bboxes: list, page_width, page_height) -> (list, list):
- """输入是一个bbox的list.
-
- 获取到输入之后，先进行layout切分，然后对这些bbox进行排序。返回排序后的bboxes
- """
-
- new_bboxes = []
- for box in bboxes:
- # new_bboxes.append([box[0], box[1], box[2], box[3], None, None, None, 'text', None, None, None, None])
- new_bboxes.append(
- [
- box[0],
- box[1],
- box[2],
- box[3],
- None,
- None,
- None,
- 'text',
- None,
- None,
- None,
- None,
- box[4],
- ]
- )
-
- layout_bboxes, _ = get_bboxes_layout(
- new_bboxes, tuple([0, 0, page_width, page_height]), 0
- )
- if any([lay['layout_label'] == LAYOUT_UNPROC for lay in layout_bboxes]):
- logger.warning('drop this pdf, reason: 复杂版面')
- return None, None
-
- sorted_bboxes = []
- # 利用layout bbox每次框定一些box，然后排序
- for layout in layout_bboxes:
- lbox = layout['layout_bbox']
- bbox_in_layout = get_bbox_in_boundary(new_bboxes, lbox)
- sorted_bbox = paper_bbox_sort(
- bbox_in_layout, lbox[2] - lbox[0], lbox[3] - lbox[1]
- )
- sorted_bboxes.extend(sorted_bbox)
-
- return sorted_bboxes, layout_bboxes
-
-
-def sort_text_block(text_block, layout_bboxes):
- """对一页的text_block进行排序."""
- sorted_text_bbox = []
- all_text_bbox = []
- # 做一个box=>text的映射
- box_to_text = {}
- for blk in text_block:
- box = blk['bbox']
- box_to_text[(box[0], box[1], box[2], box[3])] = blk
- all_text_bbox.append(box)
-
- # text_blocks_to_sort = []
- # for box in box_to_text.keys():
- # text_blocks_to_sort.append([box[0], box[1], box[2], box[3], None, None, None, 'text', None, None, None, None])
-
- # 按照layout_bboxes的顺序，对text_block进行排序
- for layout in layout_bboxes:
- layout_box = layout['layout_bbox']
- text_bbox_in_layout = get_bbox_in_boundary(
- all_text_bbox,
- [
- layout_box[0] - 1,
- layout_box[1] - 1,
- layout_box[2] + 1,
- layout_box[3] + 1,
- ],
- )
- # sorted_bbox = paper_bbox_sort(text_bbox_in_layout, layout_box[2]-layout_box[0], layout_box[3]-layout_box[1])
- text_bbox_in_layout.sort(
- key=lambda x: x[1]
- ) # 一个layout内部的box，按照y0自上而下排序
- # sorted_bbox = [[b] for b in text_blocks_to_sort]
- for sb in text_bbox_in_layout:
- sorted_text_bbox.append(box_to_text[(sb[0], sb[1], sb[2], sb[3])])
-
- return sorted_text_bbox
diff --git a/magic_pdf/layout/layout_spiler_recog.py b/magic_pdf/layout/layout_spiler_recog.py
deleted file mode 100644
index ea9d0410f..000000000
--- a/magic_pdf/layout/layout_spiler_recog.py
+++ /dev/null
@@ -1,101 +0,0 @@
-"""
-找到能分割布局的水平的横线、色块
-"""
-
-import os
-from magic_pdf.libs.commons import fitz
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap
-
-
-def __rect_filter_by_width(rect, page_w, page_h):
- mid_x = page_w/2
- if rect[0]< mid_x < rect[2]:
- return True
- return False
-
-
-def __rect_filter_by_pos(rect, image_bboxes, table_bboxes):
- """
- 不能出现在table和image的位置
- """
- for box in image_bboxes:
- if _is_in_or_part_overlap(rect, box):
- return False
-
- for box in table_bboxes:
- if _is_in_or_part_overlap(rect, box):
- return False
-
- return True
-
-
-def __debug_show_page(page, bboxes1: list,bboxes2: list,bboxes3: list,):
- save_path = "./tmp/debug.pdf"
- if os.path.exists(save_path):
- # 删除已经存在的文件
- os.remove(save_path)
- # 创建一个新的空白 PDF 文件
- doc = fitz.open('')
-
- width = page.rect.width
- height = page.rect.height
- new_page = doc.new_page(width=width, height=height)
-
- shape = new_page.new_shape()
- for bbox in bboxes1:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=fitz.pdfcolor['blue'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in bboxes2:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['yellow'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in bboxes3:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=None)
- shape.finish()
- shape.commit()
-
- parent_dir = os.path.dirname(save_path)
- if not os.path.exists(parent_dir):
- os.makedirs(parent_dir)
-
- doc.save(save_path)
- doc.close()
-
-def get_spilter_of_page(page, image_bboxes, table_bboxes):
- """
- 获取到色块和横线
- """
- cdrawings = page.get_cdrawings()
-
- spilter_bbox = []
- for block in cdrawings:
- if 'fill' in block:
- fill = block['fill']
- if 'fill' in block and block['fill'] and block['fill']!=(1.0,1.0,1.0):
- rect = block['rect']
- if __rect_filter_by_width(rect, page.rect.width, page.rect.height) and __rect_filter_by_pos(rect, image_bboxes, table_bboxes):
- spilter_bbox.append(list(rect))
-
- """过滤、修正一下这些box。因为有时候会有一些矩形，高度为0或者为负数，造成layout计算无限循环。如果是负高度或者0高度，统一修正为高度为1"""
- for box in spilter_bbox:
- if box[3]-box[1] <= 0:
- box[3] = box[1] + 1
-
- #__debug_show_page(page, spilter_bbox, [], [])
-
- return spilter_bbox
diff --git a/magic_pdf/layout/mcol_sort.py b/magic_pdf/layout/mcol_sort.py
deleted file mode 100644
index f0580c26a..000000000
--- a/magic_pdf/layout/mcol_sort.py
+++ /dev/null
@@ -1,336 +0,0 @@
-"""
-This is an advanced PyMuPDF utility for detecting multi-column pages.
-It can be used in a shell script, or its main function can be imported and
-invoked as descript below.
-
-Features
----------
-- Identify text belonging to (a variable number of) columns on the page.
-- Text with different background color is handled separately, allowing for
- easier treatment of side remarks, comment boxes, etc.
-- Uses text block detection capability to identify text blocks and
- uses the block bboxes as primary structuring principle.
-- Supports ignoring footers via a footer margin parameter.
-- Returns re-created text boundary boxes (integer coordinates), sorted ascending
- by the top, then by the left coordinates.
-
-Restrictions
--------------
-- Only supporting horizontal, left-to-right text
-- Returns a list of text boundary boxes - not the text itself. The caller is
- expected to extract text from within the returned boxes.
-- Text written above images is ignored altogether (option).
-- This utility works as expected in most cases. The following situation cannot
- be handled correctly:
- * overlapping (non-disjoint) text blocks
- * image captions are not recognized and are handled like normal text
-
-Usage
-------
-- As a CLI shell command use
-
- python multi_column.py input.pdf footer_margin
-
- Where footer margin is the height of the bottom stripe to ignore on each page.
- This code is intended to be modified according to your need.
-
-- Use in a Python script as follows:
-
- ----------------------------------------------------------------------------------
- from multi_column import column_boxes
-
- # for each page execute
- bboxes = column_boxes(page, footer_margin=50, no_image_text=True)
-
- # bboxes is a list of fitz.IRect objects, that are sort ascending by their y0,
- # then x0 coordinates. Their text content can be extracted by all PyMuPDF
- # get_text() variants, like for instance the following:
- for rect in bboxes:
- print(page.get_text(clip=rect, sort=True))
- ----------------------------------------------------------------------------------
-"""
-import sys
-from magic_pdf.libs.commons import fitz
-
-
-def column_boxes(page, footer_margin=50, header_margin=50, no_image_text=True):
- """Determine bboxes which wrap a column."""
- paths = page.get_drawings()
- bboxes = []
-
- # path rectangles
- path_rects = []
-
- # image bboxes
- img_bboxes = []
-
- # bboxes of non-horizontal text
- # avoid when expanding horizontal text boxes
- vert_bboxes = []
-
- # compute relevant page area
- clip = +page.rect
- clip.y1 -= footer_margin # Remove footer area
- clip.y0 += header_margin # Remove header area
-
- def can_extend(temp, bb, bboxlist):
- """Determines whether rectangle 'temp' can be extended by 'bb'
- without intersecting any of the rectangles contained in 'bboxlist'.
-
- Items of bboxlist may be None if they have been removed.
-
- Returns:
- True if 'temp' has no intersections with items of 'bboxlist'.
- """
- for b in bboxlist:
- if not intersects_bboxes(temp, vert_bboxes) and (
- b == None or b == bb or (temp & b).is_empty
- ):
- continue
- return False
-
- return True
-
- def in_bbox(bb, bboxes):
- """Return 1-based number if a bbox contains bb, else return 0."""
- for i, bbox in enumerate(bboxes):
- if bb in bbox:
- return i + 1
- return 0
-
- def intersects_bboxes(bb, bboxes):
- """Return True if a bbox intersects bb, else return False."""
- for bbox in bboxes:
- if not (bb & bbox).is_empty:
- return True
- return False
-
- def extend_right(bboxes, width, path_bboxes, vert_bboxes, img_bboxes):
- """Extend a bbox to the right page border.
-
- Whenever there is no text to the right of a bbox, enlarge it up
- to the right page border.
-
- Args:
- bboxes: (list[IRect]) bboxes to check
- width: (int) page width
- path_bboxes: (list[IRect]) bboxes with a background color
- vert_bboxes: (list[IRect]) bboxes with vertical text
- img_bboxes: (list[IRect]) bboxes of images
- Returns:
- Potentially modified bboxes.
- """
- for i, bb in enumerate(bboxes):
- # do not extend text with background color
- if in_bbox(bb, path_bboxes):
- continue
-
- # do not extend text in images
- if in_bbox(bb, img_bboxes):
- continue
-
- # temp extends bb to the right page border
- temp = +bb
- temp.x1 = width
-
- # do not cut through colored background or images
- if intersects_bboxes(temp, path_bboxes + vert_bboxes + img_bboxes):
- continue
-
- # also, do not intersect other text bboxes
- check = can_extend(temp, bb, bboxes)
- if check:
- bboxes[i] = temp # replace with enlarged bbox
-
- return [b for b in bboxes if b != None]
-
- def clean_nblocks(nblocks):
- """Do some elementary cleaning."""
-
- # 1. remove any duplicate blocks.
- blen = len(nblocks)
- if blen < 2:
- return nblocks
- start = blen - 1
- for i in range(start, -1, -1):
- bb1 = nblocks[i]
- bb0 = nblocks[i - 1]
- if bb0 == bb1:
- del nblocks[i]
-
- # 2. repair sequence in special cases:
- # consecutive bboxes with almost same bottom value are sorted ascending
- # by x-coordinate.
- y1 = nblocks[0].y1 # first bottom coordinate
- i0 = 0 # its index
- i1 = -1 # index of last bbox with same bottom
-
- # Iterate over bboxes, identifying segments with approx. same bottom value.
- # Replace every segment by its sorted version.
- for i in range(1, len(nblocks)):
- b1 = nblocks[i]
- if abs(b1.y1 - y1) > 10: # different bottom
- if i1 > i0: # segment length > 1? Sort it!
- nblocks[i0 : i1 + 1] = sorted(
- nblocks[i0 : i1 + 1], key=lambda b: b.x0
- )
- y1 = b1.y1 # store new bottom value
- i0 = i # store its start index
- i1 = i # store current index
- if i1 > i0: # segment waiting to be sorted
- nblocks[i0 : i1 + 1] = sorted(nblocks[i0 : i1 + 1], key=lambda b: b.x0)
- return nblocks
-
- # extract vector graphics
- for p in paths:
- path_rects.append(p["rect"].irect)
- path_bboxes = path_rects
-
- # sort path bboxes by ascending top, then left coordinates
- path_bboxes.sort(key=lambda b: (b.y0, b.x0))
-
- # bboxes of images on page, no need to sort them
- for item in page.get_images():
- img_bboxes.extend(page.get_image_rects(item[0]))
-
- # blocks of text on page
- blocks = page.get_text(
- "dict",
- flags=fitz.TEXTFLAGS_TEXT,
- clip=clip,
- )["blocks"]
-
- # Make block rectangles, ignoring non-horizontal text
- for b in blocks:
- bbox = fitz.IRect(b["bbox"]) # bbox of the block
-
- # ignore text written upon images
- if no_image_text and in_bbox(bbox, img_bboxes):
- continue
-
- # confirm first line to be horizontal
- line0 = b["lines"][0] # get first line
- if line0["dir"] != (1, 0): # only accept horizontal text
- vert_bboxes.append(bbox)
- continue
-
- srect = fitz.EMPTY_IRECT()
- for line in b["lines"]:
- lbbox = fitz.IRect(line["bbox"])
- text = "".join([s["text"].strip() for s in line["spans"]])
- if len(text) > 1:
- srect |= lbbox
- bbox = +srect
-
- if not bbox.is_empty:
- bboxes.append(bbox)
-
- # Sort text bboxes by ascending background, top, then left coordinates
- bboxes.sort(key=lambda k: (in_bbox(k, path_bboxes), k.y0, k.x0))
-
- # Extend bboxes to the right where possible
- bboxes = extend_right(
- bboxes, int(page.rect.width), path_bboxes, vert_bboxes, img_bboxes
- )
-
- # immediately return of no text found
- if bboxes == []:
- return []
-
- # --------------------------------------------------------------------
- # Join bboxes to establish some column structure
- # --------------------------------------------------------------------
- # the final block bboxes on page
- nblocks = [bboxes[0]] # pre-fill with first bbox
- bboxes = bboxes[1:] # remaining old bboxes
-
- for i, bb in enumerate(bboxes): # iterate old bboxes
- check = False # indicates unwanted joins
-
- # check if bb can extend one of the new blocks
- for j in range(len(nblocks)):
- nbb = nblocks[j] # a new block
-
- # never join across columns
- if bb == None or nbb.x1 < bb.x0 or bb.x1 < nbb.x0:
- continue
-
- # never join across different background colors
- if in_bbox(nbb, path_bboxes) != in_bbox(bb, path_bboxes):
- continue
-
- temp = bb | nbb # temporary extension of new block
- check = can_extend(temp, nbb, nblocks)
- if check == True:
- break
-
- if not check: # bb cannot be used to extend any of the new bboxes
- nblocks.append(bb) # so add it to the list
- j = len(nblocks) - 1 # index of it
- temp = nblocks[j] # new bbox added
-
- # check if some remaining bbox is contained in temp
- check = can_extend(temp, bb, bboxes)
- if check == False:
- nblocks.append(bb)
- else:
- nblocks[j] = temp
- bboxes[i] = None
-
- # do some elementary cleaning
- nblocks = clean_nblocks(nblocks)
-
- # return identified text bboxes
- return nblocks
-
-
-if __name__ == "__main__":
- """Only for debugging purposes, currently.
-
- Draw red borders around the returned text bboxes and insert
- the bbox number.
- Then save the file under the name "input-blocks.pdf".
- """
-
- # get the file name
- filename = sys.argv[1]
-
- # check if footer margin is given
- if len(sys.argv) > 2:
- footer_margin = int(sys.argv[2])
- else: # use default vaue
- footer_margin = 50
-
- # check if header margin is given
- if len(sys.argv) > 3:
- header_margin = int(sys.argv[3])
- else: # use default vaue
- header_margin = 50
-
- # open document
- doc = fitz.open(filename)
-
- # iterate over the pages
- for page in doc:
- # remove any geometry issues
- page.wrap_contents()
-
- # get the text bboxes
- bboxes = column_boxes(page, footer_margin=footer_margin, header_margin=header_margin)
-
- # prepare a canvas to draw rectangles and text
- shape = page.new_shape()
-
- # iterate over the bboxes
- for i, rect in enumerate(bboxes):
- shape.draw_rect(rect) # draw a border
-
- # write sequence number
- shape.insert_text(rect.tl + (5, 15), str(i), color=fitz.pdfcolor["red"])
-
- # finish drawing / text with color red
- shape.finish(color=fitz.pdfcolor["red"])
- shape.commit() # store to the page
-
- # save document with text bboxes
- doc.ez_save(filename.replace(".pdf", "-blocks.pdf"))
\ No newline at end of file
diff --git a/magic_pdf/libs/Constants.py b/magic_pdf/libs/Constants.py
deleted file mode 100644
index 0799f6fde..000000000
--- a/magic_pdf/libs/Constants.py
+++ /dev/null
@@ -1,53 +0,0 @@
-"""
-span维度自定义字段
-"""
-# span是否是跨页合并的
-CROSS_PAGE = "cross_page"
-
-"""
-block维度自定义字段
-"""
-# block中lines是否被删除
-LINES_DELETED = "lines_deleted"
-
-# table recognition max time default value
-TABLE_MAX_TIME_VALUE = 400
-
-# pp_table_result_max_length
-TABLE_MAX_LEN = 480
-
-# table master structure dict
-TABLE_MASTER_DICT = "table_master_structure_dict.txt"
-
-# table master dir
-TABLE_MASTER_DIR = "table_structure_tablemaster_infer/"
-
-# pp detect model dir
-DETECT_MODEL_DIR = "ch_PP-OCRv4_det_infer"
-
-# pp rec model dir
-REC_MODEL_DIR = "ch_PP-OCRv4_rec_infer"
-
-# pp rec char dict path
-REC_CHAR_DICT = "ppocr_keys_v1.txt"
-
-# pp rec copy rec directory
-PP_REC_DIRECTORY = ".paddleocr/whl/rec/ch/ch_PP-OCRv4_rec_infer"
-
-# pp rec copy det directory
-PP_DET_DIRECTORY = ".paddleocr/whl/det/ch/ch_PP-OCRv4_det_infer"
-
-
-class MODEL_NAME:
- # pp table structure algorithm
- TABLE_MASTER = "tablemaster"
- # struct eqtable
- STRUCT_EQTABLE = "struct_eqtable"
-
- DocLayout_YOLO = "doclayout_yolo"
-
- LAYOUTLMv3 = "layoutlmv3"
-
- YOLO_V8_MFD = "yolo_v8_mfd"
-
- UniMerNet_v2_Small = "unimernet_small"
\ No newline at end of file
diff --git a/magic_pdf/libs/MakeContentConfig.py b/magic_pdf/libs/MakeContentConfig.py
deleted file mode 100644
index 896edf62f..000000000
--- a/magic_pdf/libs/MakeContentConfig.py
+++ /dev/null
@@ -1,11 +0,0 @@
-class MakeMode:
- MM_MD = "mm_markdown"
- NLP_MD = "nlp_markdown"
- STANDARD_FORMAT = "standard_format"
-
-
-class DropMode:
- WHOLE_PDF = "whole_pdf"
- SINGLE_PAGE = "single_page"
- NONE = "none"
- NONE_WITH_REASON = "none_with_reason"
diff --git a/magic_pdf/libs/ModelBlockTypeEnum.py b/magic_pdf/libs/ModelBlockTypeEnum.py
deleted file mode 100644
index 494da6f6d..000000000
--- a/magic_pdf/libs/ModelBlockTypeEnum.py
+++ /dev/null
@@ -1,9 +0,0 @@
-from enum import Enum
-
-class ModelBlockTypeEnum(Enum):
- TITLE = 0
- PLAIN_TEXT = 1
- ABANDON = 2
- ISOLATE_FORMULA = 8
- EMBEDDING = 13
- ISOLATED = 14
\ No newline at end of file
diff --git a/magic_pdf/libs/__init__.py b/magic_pdf/libs/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/libs/boxbase.py b/magic_pdf/libs/boxbase.py
deleted file mode 100644
index 52779a22d..000000000
--- a/magic_pdf/libs/boxbase.py
+++ /dev/null
@@ -1,482 +0,0 @@
-import math
-
-
-def _is_in_or_part_overlap(box1, box2) -> bool:
- """两个bbox是否有部分重叠或者包含."""
- if box1 is None or box2 is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
-
- return not (x1_1 < x0_2 or # box1在box2的左边
- x0_1 > x1_2 or # box1在box2的右边
- y1_1 < y0_2 or # box1在box2的上边
- y0_1 > y1_2) # box1在box2的下边
-
-
-def _is_in_or_part_overlap_with_area_ratio(box1,
- box2,
- area_ratio_threshold=0.6):
- """判断box1是否在box2里面，或者box1和box2有部分重叠，且重叠面积占box1的比例超过area_ratio_threshold."""
- if box1 is None or box2 is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
-
- if not _is_in_or_part_overlap(box1, box2):
- return False
-
- # 计算重叠面积
- x_left = max(x0_1, x0_2)
- y_top = max(y0_1, y0_2)
- x_right = min(x1_1, x1_2)
- y_bottom = min(y1_1, y1_2)
- overlap_area = (x_right - x_left) * (y_bottom - y_top)
-
- # 计算box1的面积
- box1_area = (x1_1 - x0_1) * (y1_1 - y0_1)
-
- return overlap_area / box1_area > area_ratio_threshold
-
-
-def _is_in(box1, box2) -> bool:
- """box1是否完全在box2里面."""
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
-
- return (x0_1 >= x0_2 and # box1的左边界不在box2的左边外
- y0_1 >= y0_2 and # box1的上边界不在box2的上边外
- x1_1 <= x1_2 and # box1的右边界不在box2的右边外
- y1_1 <= y1_2) # box1的下边界不在box2的下边外
-
-
-def _is_part_overlap(box1, box2) -> bool:
- """两个bbox是否有部分重叠，但不完全包含."""
- if box1 is None or box2 is None:
- return False
-
- return _is_in_or_part_overlap(box1, box2) and not _is_in(box1, box2)
-
-
-def _left_intersect(left_box, right_box):
- """检查两个box的左边界是否有交集，也就是left_box的右边界是否在right_box的左边界内."""
- if left_box is None or right_box is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = left_box
- x0_2, y0_2, x1_2, y1_2 = right_box
-
- return x1_1 > x0_2 and x0_1 < x0_2 and (y0_1 <= y0_2 <= y1_1
- or y0_1 <= y1_2 <= y1_1)
-
-
-def _right_intersect(left_box, right_box):
- """检查box是否在右侧边界有交集，也就是left_box的左边界是否在right_box的右边界内."""
- if left_box is None or right_box is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = left_box
- x0_2, y0_2, x1_2, y1_2 = right_box
-
- return x0_1 < x1_2 and x1_1 > x1_2 and (y0_1 <= y0_2 <= y1_1
- or y0_1 <= y1_2 <= y1_1)
-
-
-def _is_vertical_full_overlap(box1, box2, x_torlence=2):
- """x方向上：要么box1包含box2, 要么box2包含box1。不能部分包含 y方向上：box1和box2有重叠."""
- # 解析box的坐标
- x11, y11, x12, y12 = box1 # 左上角和右下角的坐标 (x1, y1, x2, y2)
- x21, y21, x22, y22 = box2
-
- # 在x轴方向上，box1是否包含box2 或 box2包含box1
- contains_in_x = (x11 - x_torlence <= x21 and x12 + x_torlence >= x22) or (
- x21 - x_torlence <= x11 and x22 + x_torlence >= x12)
-
- # 在y轴方向上，box1和box2是否有重叠
- overlap_in_y = not (y12 < y21 or y11 > y22)
-
- return contains_in_x and overlap_in_y
-
-
-def _is_bottom_full_overlap(box1, box2, y_tolerance=2):
- """检查box1下方和box2的上方有轻微的重叠，轻微程度收到y_tolerance的限制 这个函数和_is_vertical-
- full_overlap的区别是，这个函数允许box1和box2在x方向上有轻微的重叠,允许一定的模糊度."""
- if box1 is None or box2 is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
- tolerance_margin = 2
- is_xdir_full_overlap = (
- (x0_1 - tolerance_margin <= x0_2 <= x1_1 + tolerance_margin
- and x0_1 - tolerance_margin <= x1_2 <= x1_1 + tolerance_margin)
- or (x0_2 - tolerance_margin <= x0_1 <= x1_2 + tolerance_margin
- and x0_2 - tolerance_margin <= x1_1 <= x1_2 + tolerance_margin))
-
- return y0_2 < y1_1 and 0 < (y1_1 -
- y0_2) < y_tolerance and is_xdir_full_overlap
-
-
-def _is_left_overlap(
- box1,
- box2,
-):
- """检查box1的左侧是否和box2有重叠 在Y方向上可以是部分重叠或者是完全重叠。不分box1和box2的上下关系，也就是无论box1在box2下
- 方还是box2在box1下方，都可以检测到重叠。 X方向上."""
-
- def __overlap_y(Ay1, Ay2, By1, By2):
- return max(0, min(Ay2, By2) - max(Ay1, By1))
-
- if box1 is None or box2 is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
-
- y_overlap_len = __overlap_y(y0_1, y1_1, y0_2, y1_2)
- ratio_1 = 1.0 * y_overlap_len / (y1_1 - y0_1) if y1_1 - y0_1 != 0 else 0
- ratio_2 = 1.0 * y_overlap_len / (y1_2 - y0_2) if y1_2 - y0_2 != 0 else 0
- vertical_overlap_cond = ratio_1 >= 0.5 or ratio_2 >= 0.5
-
- # vertical_overlap_cond = y0_1<=y0_2<=y1_1 or y0_1<=y1_2<=y1_1 or y0_2<=y0_1<=y1_2 or y0_2<=y1_1<=y1_2
- return x0_1 <= x0_2 <= x1_1 and vertical_overlap_cond
-
-
-def __is_overlaps_y_exceeds_threshold(bbox1,
- bbox2,
- overlap_ratio_threshold=0.8):
- """检查两个bbox在y轴上是否有重叠，并且该重叠区域的高度占两个bbox高度更低的那个超过80%"""
- _, y0_1, _, y1_1 = bbox1
- _, y0_2, _, y1_2 = bbox2
-
- overlap = max(0, min(y1_1, y1_2) - max(y0_1, y0_2))
- height1, height2 = y1_1 - y0_1, y1_2 - y0_2
- # max_height = max(height1, height2)
- min_height = min(height1, height2)
-
- return (overlap / min_height) > overlap_ratio_threshold
-
-
-def calculate_iou(bbox1, bbox2):
- """计算两个边界框的交并比(IOU)。
-
- Args:
- bbox1 (list[float]): 第一个边界框的坐标，格式为 [x1, y1, x2, y2]，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
- bbox2 (list[float]): 第二个边界框的坐标，格式与 `bbox1` 相同。
-
- Returns:
- float: 两个边界框的交并比(IOU)，取值范围为 [0, 1]。
- """
- # Determine the coordinates of the intersection rectangle
- x_left = max(bbox1[0], bbox2[0])
- y_top = max(bbox1[1], bbox2[1])
- x_right = min(bbox1[2], bbox2[2])
- y_bottom = min(bbox1[3], bbox2[3])
-
- if x_right < x_left or y_bottom < y_top:
- return 0.0
-
- # The area of overlap area
- intersection_area = (x_right - x_left) * (y_bottom - y_top)
-
- # The area of both rectangles
- bbox1_area = (bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1])
- bbox2_area = (bbox2[2] - bbox2[0]) * (bbox2[3] - bbox2[1])
-
- # Compute the intersection over union by taking the intersection area
- # and dividing it by the sum of both areas minus the intersection area
- iou = intersection_area / float(bbox1_area + bbox2_area -
- intersection_area)
- return iou
-
-
-def calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2):
- """计算box1和box2的重叠面积占最小面积的box的比例."""
- # Determine the coordinates of the intersection rectangle
- x_left = max(bbox1[0], bbox2[0])
- y_top = max(bbox1[1], bbox2[1])
- x_right = min(bbox1[2], bbox2[2])
- y_bottom = min(bbox1[3], bbox2[3])
-
- if x_right < x_left or y_bottom < y_top:
- return 0.0
-
- # The area of overlap area
- intersection_area = (x_right - x_left) * (y_bottom - y_top)
- min_box_area = min([(bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1]),
- (bbox2[3] - bbox2[1]) * (bbox2[2] - bbox2[0])])
- if min_box_area == 0:
- return 0
- else:
- return intersection_area / min_box_area
-
-
-def calculate_overlap_area_in_bbox1_area_ratio(bbox1, bbox2):
- """计算box1和box2的重叠面积占bbox1的比例."""
- # Determine the coordinates of the intersection rectangle
- x_left = max(bbox1[0], bbox2[0])
- y_top = max(bbox1[1], bbox2[1])
- x_right = min(bbox1[2], bbox2[2])
- y_bottom = min(bbox1[3], bbox2[3])
-
- if x_right < x_left or y_bottom < y_top:
- return 0.0
-
- # The area of overlap area
- intersection_area = (x_right - x_left) * (y_bottom - y_top)
- bbox1_area = (bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1])
- if bbox1_area == 0:
- return 0
- else:
- return intersection_area / bbox1_area
-
-
-def get_minbox_if_overlap_by_ratio(bbox1, bbox2, ratio):
- """通过calculate_overlap_area_2_minbox_area_ratio计算两个bbox重叠的面积占最小面积的box的比例
- 如果比例大于ratio，则返回小的那个bbox, 否则返回None."""
- x1_min, y1_min, x1_max, y1_max = bbox1
- x2_min, y2_min, x2_max, y2_max = bbox2
- area1 = (x1_max - x1_min) * (y1_max - y1_min)
- area2 = (x2_max - x2_min) * (y2_max - y2_min)
- overlap_ratio = calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2)
- if overlap_ratio > ratio:
- if area1 <= area2:
- return bbox1
- else:
- return bbox2
- else:
- return None
-
-
-def get_bbox_in_boundary(bboxes: list, boundary: tuple) -> list:
- x0, y0, x1, y1 = boundary
- new_boxes = [
- box for box in bboxes
- if box[0] >= x0 and box[1] >= y0 and box[2] <= x1 and box[3] <= y1
- ]
- return new_boxes
-
-
-def is_vbox_on_side(bbox, width, height, side_threshold=0.2):
- """判断一个bbox是否在pdf页面的边缘."""
- x0, x1 = bbox[0], bbox[2]
- if x1 <= width * side_threshold or x0 >= width * (1 - side_threshold):
- return True
- return False
-
-
-def find_top_nearest_text_bbox(pymu_blocks, obj_bbox):
- tolerance_margin = 4
- top_boxes = [
- box for box in pymu_blocks
- if obj_bbox[1] - box['bbox'][3] >= -tolerance_margin
- and not _is_in(box['bbox'], obj_bbox)
- ]
- # 然后找到X方向上有互相重叠的
- top_boxes = [
- box for box in top_boxes if any([
- obj_bbox[0] - tolerance_margin <= box['bbox'][0] <= obj_bbox[2] +
- tolerance_margin, obj_bbox[0] -
- tolerance_margin <= box['bbox'][2] <= obj_bbox[2] +
- tolerance_margin, box['bbox'][0] -
- tolerance_margin <= obj_bbox[0] <= box['bbox'][2] +
- tolerance_margin, box['bbox'][0] -
- tolerance_margin <= obj_bbox[2] <= box['bbox'][2] +
- tolerance_margin
- ])
- ]
-
- # 然后找到y1最大的那个
- if len(top_boxes) > 0:
- top_boxes.sort(key=lambda x: x['bbox'][3], reverse=True)
- return top_boxes[0]
- else:
- return None
-
-
-def find_bottom_nearest_text_bbox(pymu_blocks, obj_bbox):
- bottom_boxes = [
- box for box in pymu_blocks if box['bbox'][1] -
- obj_bbox[3] >= -2 and not _is_in(box['bbox'], obj_bbox)
- ]
- # 然后找到X方向上有互相重叠的
- bottom_boxes = [
- box for box in bottom_boxes if any([
- obj_bbox[0] - 2 <= box['bbox'][0] <= obj_bbox[2] + 2, obj_bbox[0] -
- 2 <= box['bbox'][2] <= obj_bbox[2] + 2, box['bbox'][0] -
- 2 <= obj_bbox[0] <= box['bbox'][2] + 2, box['bbox'][0] -
- 2 <= obj_bbox[2] <= box['bbox'][2] + 2
- ])
- ]
-
- # 然后找到y0最小的那个
- if len(bottom_boxes) > 0:
- bottom_boxes.sort(key=lambda x: x['bbox'][1], reverse=False)
- return bottom_boxes[0]
- else:
- return None
-
-
-def find_left_nearest_text_bbox(pymu_blocks, obj_bbox):
- """寻找左侧最近的文本block."""
- left_boxes = [
- box for box in pymu_blocks if obj_bbox[0] -
- box['bbox'][2] >= -2 and not _is_in(box['bbox'], obj_bbox)
- ]
- # 然后找到X方向上有互相重叠的
- left_boxes = [
- box for box in left_boxes if any([
- obj_bbox[1] - 2 <= box['bbox'][1] <= obj_bbox[3] + 2, obj_bbox[1] -
- 2 <= box['bbox'][3] <= obj_bbox[3] + 2, box['bbox'][1] -
- 2 <= obj_bbox[1] <= box['bbox'][3] + 2, box['bbox'][1] -
- 2 <= obj_bbox[3] <= box['bbox'][3] + 2
- ])
- ]
-
- # 然后找到x1最大的那个
- if len(left_boxes) > 0:
- left_boxes.sort(key=lambda x: x['bbox'][2], reverse=True)
- return left_boxes[0]
- else:
- return None
-
-
-def find_right_nearest_text_bbox(pymu_blocks, obj_bbox):
- """寻找右侧最近的文本block."""
- right_boxes = [
- box for box in pymu_blocks if box['bbox'][0] -
- obj_bbox[2] >= -2 and not _is_in(box['bbox'], obj_bbox)
- ]
- # 然后找到X方向上有互相重叠的
- right_boxes = [
- box for box in right_boxes if any([
- obj_bbox[1] - 2 <= box['bbox'][1] <= obj_bbox[3] + 2, obj_bbox[1] -
- 2 <= box['bbox'][3] <= obj_bbox[3] + 2, box['bbox'][1] -
- 2 <= obj_bbox[1] <= box['bbox'][3] + 2, box['bbox'][1] -
- 2 <= obj_bbox[3] <= box['bbox'][3] + 2
- ])
- ]
-
- # 然后找到x0最小的那个
- if len(right_boxes) > 0:
- right_boxes.sort(key=lambda x: x['bbox'][0], reverse=False)
- return right_boxes[0]
- else:
- return None
-
-
-def bbox_relative_pos(bbox1, bbox2):
- """判断两个矩形框的相对位置关系.
-
- Args:
- bbox1: 一个四元组，表示第一个矩形框的左上角和右下角的坐标，格式为(x1, y1, x1b, y1b)
- bbox2: 一个四元组，表示第二个矩形框的左上角和右下角的坐标，格式为(x2, y2, x2b, y2b)
-
- Returns:
- 一个四元组，表示矩形框1相对于矩形框2的位置关系，格式为(left, right, bottom, top)
- 其中，left表示矩形框1是否在矩形框2的左侧，right表示矩形框1是否在矩形框2的右侧，
- bottom表示矩形框1是否在矩形框2的下方，top表示矩形框1是否在矩形框2的上方
- """
- x1, y1, x1b, y1b = bbox1
- x2, y2, x2b, y2b = bbox2
-
- left = x2b < x1
- right = x1b < x2
- bottom = y2b < y1
- top = y1b < y2
- return left, right, bottom, top
-
-
-def bbox_distance(bbox1, bbox2):
- """计算两个矩形框的距离。
-
- Args:
- bbox1 (tuple): 第一个矩形框的坐标，格式为 (x1, y1, x2, y2)，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
- bbox2 (tuple): 第二个矩形框的坐标，格式为 (x1, y1, x2, y2)，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
-
- Returns:
- float: 矩形框之间的距离。
- """
-
- def dist(point1, point2):
- return math.sqrt((point1[0] - point2[0])**2 +
- (point1[1] - point2[1])**2)
-
- x1, y1, x1b, y1b = bbox1
- x2, y2, x2b, y2b = bbox2
-
- left, right, bottom, top = bbox_relative_pos(bbox1, bbox2)
-
- if top and left:
- return dist((x1, y1b), (x2b, y2))
- elif left and bottom:
- return dist((x1, y1), (x2b, y2b))
- elif bottom and right:
- return dist((x1b, y1), (x2, y2b))
- elif right and top:
- return dist((x1b, y1b), (x2, y2))
- elif left:
- return x1 - x2b
- elif right:
- return x2 - x1b
- elif bottom:
- return y1 - y2b
- elif top:
- return y2 - y1b
- return 0.0
-
-
-def box_area(bbox):
- return (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
-
-
-def get_overlap_area(bbox1, bbox2):
- """计算box1和box2的重叠面积占bbox1的比例."""
- # Determine the coordinates of the intersection rectangle
- x_left = max(bbox1[0], bbox2[0])
- y_top = max(bbox1[1], bbox2[1])
- x_right = min(bbox1[2], bbox2[2])
- y_bottom = min(bbox1[3], bbox2[3])
-
- if x_right < x_left or y_bottom < y_top:
- return 0.0
-
- # The area of overlap area
- return (x_right - x_left) * (y_bottom - y_top)
-
-
-def calculate_vertical_projection_overlap_ratio(block1, block2):
- """
- Calculate the proportion of the x-axis covered by the vertical projection of two blocks.
-
- Args:
- block1 (tuple): Coordinates of the first block (x0, y0, x1, y1).
- block2 (tuple): Coordinates of the second block (x0, y0, x1, y1).
-
- Returns:
- float: The proportion of the x-axis covered by the vertical projection of the two blocks.
- """
- x0_1, _, x1_1, _ = block1
- x0_2, _, x1_2, _ = block2
-
- # Calculate the intersection of the x-coordinates
- x_left = max(x0_1, x0_2)
- x_right = min(x1_1, x1_2)
-
- if x_right < x_left:
- return 0.0
-
- # Length of the intersection
- intersection_length = x_right - x_left
-
- # Length of the x-axis projection of the first block
- block1_length = x1_1 - x0_1
-
- if block1_length == 0:
- return 0.0
-
- # Proportion of the x-axis covered by the intersection
- # logger.info(f"intersection_length: {intersection_length}, block1_length: {block1_length}")
- return intersection_length / block1_length
diff --git a/magic_pdf/libs/calc_span_stats.py b/magic_pdf/libs/calc_span_stats.py
deleted file mode 100644
index c0bf61a8d..000000000
--- a/magic_pdf/libs/calc_span_stats.py
+++ /dev/null
@@ -1,239 +0,0 @@
-import os
-import csv
-import json
-import pandas as pd
-from pandas import DataFrame as df
-from matplotlib import pyplot as plt
-from termcolor import cprint
-
-"""
-Execute this script in the following way:
-
-1. Make sure there are pdf_dic.json files under the directory code-clean/tmp/unittest/md/, such as the following:
-
- code-clean/tmp/unittest/md/scihub/scihub_00500000/libgen.scimag00527000-00527999.zip_10.1002/app.25178/pdf_dic.json
-
-2. Under the directory code-clean, execute the following command:
-
- $ python -m libs.calc_span_stats
-
-"""
-
-
-def print_green_on_red(text):
- cprint(text, "green", "on_red", attrs=["bold"], end="\n\n")
-
-
-def print_green(text):
- print()
- cprint(text, "green", attrs=["bold"], end="\n\n")
-
-
-def print_red(text):
- print()
- cprint(text, "red", attrs=["bold"], end="\n\n")
-
-
-def safe_get(dict_obj, key, default):
- val = dict_obj.get(key)
- if val is None:
- return default
- else:
- return val
-
-
-class SpanStatsCalc:
- """Calculate statistics of span."""
-
- def draw_charts(self, span_stats: pd.DataFrame, fig_num: int, save_path: str):
- """Draw multiple figures in one figure."""
- # make a canvas
- fig = plt.figure(fig_num, figsize=(20, 20))
-
- pass
-
- def calc_stats_per_dict(self, pdf_dict) -> pd.DataFrame:
- """Calculate statistics per pdf_dict."""
- span_stats = pd.DataFrame()
-
- span_stats = []
- span_id = 0
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "para_blocks" in blocks.keys():
- for para_block in blocks["para_blocks"]:
- for line in para_block["lines"]:
- for span in line["spans"]:
- span_text = safe_get(span, "text", "")
- span_font_name = safe_get(span, "font", "")
- span_font_size = safe_get(span, "size", 0)
- span_font_color = safe_get(span, "color", "")
- span_font_flags = safe_get(span, "flags", 0)
-
- span_font_flags_decoded = safe_get(span, "decomposed_flags", {})
- span_is_super_script = safe_get(span_font_flags_decoded, "is_superscript", False)
- span_is_italic = safe_get(span_font_flags_decoded, "is_italic", False)
- span_is_serifed = safe_get(span_font_flags_decoded, "is_serifed", False)
- span_is_sans_serifed = safe_get(span_font_flags_decoded, "is_sans_serifed", False)
- span_is_monospaced = safe_get(span_font_flags_decoded, "is_monospaced", False)
- span_is_proportional = safe_get(span_font_flags_decoded, "is_proportional", False)
- span_is_bold = safe_get(span_font_flags_decoded, "is_bold", False)
-
- span_stats.append(
- {
- "span_id": span_id, # id of span
- "page_id": page_id, # page number of pdf
- "span_text": span_text, # text of span
- "span_font_name": span_font_name, # font name of span
- "span_font_size": span_font_size, # font size of span
- "span_font_color": span_font_color, # font color of span
- "span_font_flags": span_font_flags, # font flags of span
- "span_is_superscript": int(
- span_is_super_script
- ), # indicate whether the span is super script or not
- "span_is_italic": int(span_is_italic), # indicate whether the span is italic or not
- "span_is_serifed": int(span_is_serifed), # indicate whether the span is serifed or not
- "span_is_sans_serifed": int(
- span_is_sans_serifed
- ), # indicate whether the span is sans serifed or not
- "span_is_monospaced": int(
- span_is_monospaced
- ), # indicate whether the span is monospaced or not
- "span_is_proportional": int(
- span_is_proportional
- ), # indicate whether the span is proportional or not
- "span_is_bold": int(span_is_bold), # indicate whether the span is bold or not
- }
- )
-
- span_id += 1
-
- span_stats = pd.DataFrame(span_stats)
- # print(span_stats)
-
- return span_stats
-
-
-def __find_pdf_dic_files(
- jf_name="pdf_dic.json",
- base_code_name="code-clean",
- tgt_base_dir_name="tmp",
- unittest_dir_name="unittest",
- md_dir_name="md",
- book_names=[
- "scihub",
- ], # other possible values: "zlib", "arxiv" and so on
-):
- pdf_dict_files = []
-
- curr_dir = os.path.dirname(__file__)
-
- for i in range(len(curr_dir)):
- if curr_dir[i : i + len(base_code_name)] == base_code_name:
- base_code_dir_name = curr_dir[: i + len(base_code_name)]
- for book_name in book_names:
- search_dir_relative_name = os.path.join(tgt_base_dir_name, unittest_dir_name, md_dir_name, book_name)
- if os.path.exists(base_code_dir_name):
- search_dir_name = os.path.join(base_code_dir_name, search_dir_relative_name)
- for root, dirs, files in os.walk(search_dir_name):
- for file in files:
- if file == jf_name:
- pdf_dict_files.append(os.path.join(root, file))
- break
-
- return pdf_dict_files
-
-
-def combine_span_texts(group_df, span_stats):
- combined_span_texts = []
- for _, row in group_df.iterrows():
- curr_span_id = row.name
- curr_span_text = row["span_text"]
-
- pre_span_id = curr_span_id - 1
- pre_span_text = span_stats.at[pre_span_id, "span_text"] if pre_span_id in span_stats.index else ""
-
- next_span_id = curr_span_id + 1
- next_span_text = span_stats.at[next_span_id, "span_text"] if next_span_id in span_stats.index else ""
-
- # pointer_sign is a right arrow if the span is superscript, otherwise it is a down arrow
- pointer_sign = "→ → → "
- combined_text = "\n".join([pointer_sign + pre_span_text, pointer_sign + curr_span_text, pointer_sign + next_span_text])
- combined_span_texts.append(combined_text)
-
- return "\n\n".join(combined_span_texts)
-
-
-# pd.set_option("display.max_colwidth", None) # 设置为 None 来显示完整的文本
-pd.set_option("display.max_rows", None) # 设置为 None 来显示更多的行
-
-
-def main():
- pdf_dict_files = __find_pdf_dic_files()
- # print(pdf_dict_files)
-
- span_stats_calc = SpanStatsCalc()
-
- for pdf_dict_file in pdf_dict_files:
- print("-" * 100)
- print_green_on_red(f"Processing {pdf_dict_file}")
-
- with open(pdf_dict_file, "r", encoding="utf-8") as f:
- pdf_dict = json.load(f)
-
- raw_df = span_stats_calc.calc_stats_per_dict(pdf_dict)
- save_path = pdf_dict_file.replace("pdf_dic.json", "span_stats_raw.csv")
- raw_df.to_csv(save_path, index=False)
-
- filtered_df = raw_df[raw_df["span_is_superscript"] == 1]
- if filtered_df.empty:
- print("No superscript span found!")
- continue
-
- filtered_grouped_df = filtered_df.groupby(["span_font_name", "span_font_size", "span_font_color"])
-
- combined_span_texts = filtered_grouped_df.apply(combine_span_texts, span_stats=raw_df) # type: ignore
-
- final_df = filtered_grouped_df.size().reset_index(name="count")
- final_df["span_texts"] = combined_span_texts.reset_index(level=[0, 1, 2], drop=True)
-
- print(final_df)
-
- final_df["span_texts"] = final_df["span_texts"].apply(lambda x: x.replace("\n", "\r\n"))
-
- save_path = pdf_dict_file.replace("pdf_dic.json", "span_stats_final.csv")
- # 使用 UTF-8 编码并添加 BOM，确保所有字段被双引号包围
- final_df.to_csv(save_path, index=False, encoding="utf-8-sig", quoting=csv.QUOTE_ALL)
-
- # 创建一个 2x2 的图表布局
- fig, axs = plt.subplots(2, 2, figsize=(15, 10))
-
- # 按照 span_font_name 分类作图
- final_df.groupby("span_font_name")["count"].sum().plot(kind="bar", ax=axs[0, 0], title="By Font Name")
-
- # 按照 span_font_size 分类作图
- final_df.groupby("span_font_size")["count"].sum().plot(kind="bar", ax=axs[0, 1], title="By Font Size")
-
- # 按照 span_font_color 分类作图
- final_df.groupby("span_font_color")["count"].sum().plot(kind="bar", ax=axs[1, 0], title="By Font Color")
-
- # 按照 span_font_name、span_font_size 和 span_font_color 共同分类作图
- grouped = final_df.groupby(["span_font_name", "span_font_size", "span_font_color"])
- grouped["count"].sum().unstack().plot(kind="bar", ax=axs[1, 1], title="Combined Grouping")
-
- # 调整布局
- plt.tight_layout()
-
- # 显示图表
- # plt.show()
-
- # 保存图表到 PNG 文件
- save_path = pdf_dict_file.replace("pdf_dic.json", "span_stats_combined.png")
- plt.savefig(save_path)
-
- # 清除画布
- plt.clf()
-
-
-if __name__ == "__main__":
- main()
diff --git a/magic_pdf/libs/clean_memory.py b/magic_pdf/libs/clean_memory.py
deleted file mode 100644
index 6bfc174fa..000000000
--- a/magic_pdf/libs/clean_memory.py
+++ /dev/null
@@ -1,10 +0,0 @@
-# Copyright (c) Opendatalab. All rights reserved.
-import torch
-import gc
-
-
-def clean_memory():
- if torch.cuda.is_available():
- torch.cuda.empty_cache()
- torch.cuda.ipc_collect()
- gc.collect()
\ No newline at end of file
diff --git a/magic_pdf/libs/commons.py b/magic_pdf/libs/commons.py
deleted file mode 100644
index 15592dbfe..000000000
--- a/magic_pdf/libs/commons.py
+++ /dev/null
@@ -1,204 +0,0 @@
-import datetime
-import json
-import os, re, configparser
-import subprocess
-import time
-
-import boto3
-from loguru import logger
-from boto3.s3.transfer import TransferConfig
-from botocore.config import Config
-
-import fitz # 1.23.9中已经切换到rebase
-# import fitz_old as fitz # 使用1.23.9之前的pymupdf库
-
-
-def get_delta_time(input_time):
- return round(time.time() - input_time, 2)
-
-
-def join_path(*args):
- return '/'.join(str(s).rstrip('/') for s in args)
-
-
-#配置全局的errlog_path，方便demo同步引用
-error_log_path = "s3://llm-pdf-text/err_logs/"
-# json_dump_path = "s3://pdf_books_temp/json_dump/" # 这条路径仅用于临时本地测试,不能提交到main
-json_dump_path = "s3://llm-pdf-text/json_dump/"
-
-# s3_image_save_path = "s3://mllm-raw-media/pdf2md_img/" # 基础库不应该有这些存在的路径，应该在业务代码中定义
-
-
-def get_top_percent_list(num_list, percent):
- """
- 获取列表中前百分之多少的元素
- :param num_list:
- :param percent:
- :return:
- """
- if len(num_list) == 0:
- top_percent_list = []
- else:
- # 对imgs_len_list排序
- sorted_imgs_len_list = sorted(num_list, reverse=True)
- # 计算 percent 的索引
- top_percent_index = int(len(sorted_imgs_len_list) * percent)
- # 取前80%的元素
- top_percent_list = sorted_imgs_len_list[:top_percent_index]
- return top_percent_list
-
-
-def formatted_time(time_stamp):
- dt_object = datetime.datetime.fromtimestamp(time_stamp)
- output_time = dt_object.strftime("%Y-%m-%d-%H:%M:%S")
- return output_time
-
-
-def mymax(alist: list):
- if len(alist) == 0:
- return 0 # 空是0， 0*0也是0大小q
- else:
- return max(alist)
-
-def parse_aws_param(profile):
- if isinstance(profile, str):
- # 解析配置文件
- config_file = join_path(os.path.expanduser("~"), ".aws", "config")
- credentials_file = join_path(os.path.expanduser("~"), ".aws", "credentials")
- config = configparser.ConfigParser()
- config.read(credentials_file)
- config.read(config_file)
- # 获取 AWS 账户相关信息
- ak = config.get(profile, "aws_access_key_id")
- sk = config.get(profile, "aws_secret_access_key")
- if profile == "default":
- s3_str = config.get(f"{profile}", "s3")
- else:
- s3_str = config.get(f"profile {profile}", "s3")
- end_match = re.search("endpoint_url[\s]*=[\s]*([^\s\n]+)[\s\n]*$", s3_str, re.MULTILINE)
- if end_match:
- endpoint = end_match.group(1)
- else:
- raise ValueError(f"aws 配置文件中没有找到 endpoint_url")
- style_match = re.search("addressing_style[\s]*=[\s]*([^\s\n]+)[\s\n]*$", s3_str, re.MULTILINE)
- if style_match:
- addressing_style = style_match.group(1)
- else:
- addressing_style = "path"
- elif isinstance(profile, dict):
- ak = profile["ak"]
- sk = profile["sk"]
- endpoint = profile["endpoint"]
- addressing_style = "auto"
-
- return ak, sk, endpoint, addressing_style
-
-
-def parse_bucket_key(s3_full_path: str):
- """
- 输入 s3://bucket/path/to/my/file.txt
- 输出 bucket, path/to/my/file.txt
- """
- s3_full_path = s3_full_path.strip()
- if s3_full_path.startswith("s3://"):
- s3_full_path = s3_full_path[5:]
- if s3_full_path.startswith("/"):
- s3_full_path = s3_full_path[1:]
- bucket, key = s3_full_path.split("/", 1)
- return bucket, key
-
-
-def read_file(pdf_path: str, s3_profile):
- if pdf_path.startswith("s3://"):
- ak, sk, end_point, addressing_style = parse_aws_param(s3_profile)
- cli = boto3.client(service_name="s3", aws_access_key_id=ak, aws_secret_access_key=sk, endpoint_url=end_point,
- config=Config(s3={'addressing_style': addressing_style}, retries={'max_attempts': 10, 'mode': 'standard'}))
- bucket_name, bucket_key = parse_bucket_key(pdf_path)
- res = cli.get_object(Bucket=bucket_name, Key=bucket_key)
- file_content = res["Body"].read()
- return file_content
- else:
- with open(pdf_path, "rb") as f:
- return f.read()
-
-
-def get_docx_model_output(pdf_model_output, page_id):
-
- model_output_json = pdf_model_output[page_id]
-
- return model_output_json
-
-
-def list_dir(dir_path:str, s3_profile:str):
- """
- 列出dir_path下的所有文件
- """
- ret = []
-
- if dir_path.startswith("s3"):
- ak, sk, end_point, addressing_style = parse_aws_param(s3_profile)
- s3info = re.findall(r"s3:\/\/([^\/]+)\/(.*)", dir_path)
- bucket, path = s3info[0][0], s3info[0][1]
- try:
- cli = boto3.client(service_name="s3", aws_access_key_id=ak, aws_secret_access_key=sk, endpoint_url=end_point,
- config=Config(s3={'addressing_style': addressing_style}))
- def list_obj_scluster():
- marker = None
- while True:
- list_kwargs = dict(MaxKeys=1000, Bucket=bucket, Prefix=path)
- if marker:
- list_kwargs['Marker'] = marker
- response = cli.list_objects(**list_kwargs)
- contents = response.get("Contents", [])
- yield from contents
- if not response.get("IsTruncated") or len(contents)==0:
- break
- marker = contents[-1]['Key']
-
-
- for info in list_obj_scluster():
- file_path = info['Key']
- #size = info['Size']
-
- if path!="":
- afile = file_path[len(path):]
- if afile.endswith(".json"):
- ret.append(f"s3://{bucket}/{file_path}")
-
- return ret
-
- except Exception as e:
- logger.exception(e)
- exit(-1)
- else: #本地的目录，那么扫描本地目录并返会这个目录里的所有jsonl文件
-
- for root, dirs, files in os.walk(dir_path):
- for file in files:
- if file.endswith(".json"):
- ret.append(join_path(root, file))
- ret.sort()
- return ret
-
-def get_img_s3_client(save_path:str, image_s3_config:str):
- """
- """
- if save_path.startswith("s3://"): # 放这里是为了最少创建一个s3 client
- ak, sk, end_point, addressing_style = parse_aws_param(image_s3_config)
- img_s3_client = boto3.client(
- service_name="s3",
- aws_access_key_id=ak,
- aws_secret_access_key=sk,
- endpoint_url=end_point,
- config=Config(s3={"addressing_style": addressing_style}, retries={'max_attempts': 5, 'mode': 'standard'}),
- )
- else:
- img_s3_client = None
-
- return img_s3_client
-
-if __name__=="__main__":
- s3_path = "s3://llm-pdf-text/layout_det/scihub/scimag07865000-07865999/10.1007/s10729-011-9175-6.pdf/"
- s3_profile = "langchao"
- ret = list_dir(s3_path, s3_profile)
- print(ret)
-
\ No newline at end of file
diff --git a/magic_pdf/libs/config_reader.py b/magic_pdf/libs/config_reader.py
deleted file mode 100644
index 5e1a300d9..000000000
--- a/magic_pdf/libs/config_reader.py
+++ /dev/null
@@ -1,121 +0,0 @@
-"""根据bucket的名字返回对应的s3 AK， SK，endpoint三元组."""
-
-import json
-import os
-
-from loguru import logger
-
-from magic_pdf.libs.Constants import MODEL_NAME
-from magic_pdf.libs.commons import parse_bucket_key
-
-# 定义配置文件名常量
-CONFIG_FILE_NAME = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'magic-pdf.json')
-
-
-def read_config():
- if os.path.isabs(CONFIG_FILE_NAME):
- config_file = CONFIG_FILE_NAME
- else:
- home_dir = os.path.expanduser('~')
- config_file = os.path.join(home_dir, CONFIG_FILE_NAME)
-
- if not os.path.exists(config_file):
- raise FileNotFoundError(f'{config_file} not found')
-
- with open(config_file, 'r', encoding='utf-8') as f:
- config = json.load(f)
- return config
-
-
-def get_s3_config(bucket_name: str):
- """~/magic-pdf.json 读出来."""
- config = read_config()
-
- bucket_info = config.get('bucket_info')
- if bucket_name not in bucket_info:
- access_key, secret_key, storage_endpoint = bucket_info['[default]']
- else:
- access_key, secret_key, storage_endpoint = bucket_info[bucket_name]
-
- if access_key is None or secret_key is None or storage_endpoint is None:
- raise Exception(f'ak, sk or endpoint not found in {CONFIG_FILE_NAME}')
-
- # logger.info(f"get_s3_config: ak={access_key}, sk={secret_key}, endpoint={storage_endpoint}")
-
- return access_key, secret_key, storage_endpoint
-
-
-def get_s3_config_dict(path: str):
- access_key, secret_key, storage_endpoint = get_s3_config(get_bucket_name(path))
- return {'ak': access_key, 'sk': secret_key, 'endpoint': storage_endpoint}
-
-
-def get_bucket_name(path):
- bucket, key = parse_bucket_key(path)
- return bucket
-
-
-def get_local_models_dir():
- config = read_config()
- models_dir = config.get('models-dir')
- if models_dir is None:
- logger.warning(f"'models-dir' not found in {CONFIG_FILE_NAME}, use '/tmp/models' as default")
- return '/tmp/models'
- else:
- return models_dir
-
-
-def get_local_layoutreader_model_dir():
- config = read_config()
- layoutreader_model_dir = config.get('layoutreader-model-dir')
- if layoutreader_model_dir is None or not os.path.exists(layoutreader_model_dir):
- home_dir = os.path.expanduser('~')
- layoutreader_at_modelscope_dir_path = os.path.join(home_dir, '.cache/modelscope/hub/ppaanngggg/layoutreader')
- logger.warning(f"'layoutreader-model-dir' not exists, use {layoutreader_at_modelscope_dir_path} as default")
- return layoutreader_at_modelscope_dir_path
- else:
- return layoutreader_model_dir
-
-
-def get_device():
- config = read_config()
- device = config.get('device-mode')
- if device is None:
- logger.warning(f"'device-mode' not found in {CONFIG_FILE_NAME}, use 'cpu' as default")
- return 'cpu'
- else:
- return device
-
-
-def get_table_recog_config():
- config = read_config()
- table_config = config.get('table-config')
- if table_config is None:
- logger.warning(f"'table-config' not found in {CONFIG_FILE_NAME}, use 'False' as default")
- return json.loads(f'{{"model": "{MODEL_NAME.TABLE_MASTER}","enable": false, "max_time": 400}}')
- else:
- return table_config
-
-
-def get_layout_config():
- config = read_config()
- layout_config = config.get("layout-config")
- if layout_config is None:
- logger.warning(f"'layout-config' not found in {CONFIG_FILE_NAME}, use '{MODEL_NAME.LAYOUTLMv3}' as default")
- return json.loads(f'{{"model": "{MODEL_NAME.LAYOUTLMv3}"}}')
- else:
- return layout_config
-
-
-def get_formula_config():
- config = read_config()
- formula_config = config.get("formula-config")
- if formula_config is None:
- logger.warning(f"'formula-config' not found in {CONFIG_FILE_NAME}, use 'True' as default")
- return json.loads(f'{{"mfd_model": "{MODEL_NAME.YOLO_V8_MFD}","mfr_model": "{MODEL_NAME.UniMerNet_v2_Small}","enable": true}}')
- else:
- return formula_config
-
-
-if __name__ == "__main__":
- ak, sk, endpoint = get_s3_config("llm-raw")
diff --git a/magic_pdf/libs/convert_utils.py b/magic_pdf/libs/convert_utils.py
deleted file mode 100644
index 99a1879d4..000000000
--- a/magic_pdf/libs/convert_utils.py
+++ /dev/null
@@ -1,5 +0,0 @@
-def dict_to_list(input_dict):
- items_list = []
- for _, item in input_dict.items():
- items_list.append(item)
- return items_list
diff --git a/magic_pdf/libs/coordinate_transform.py b/magic_pdf/libs/coordinate_transform.py
deleted file mode 100644
index 7cd7a0768..000000000
--- a/magic_pdf/libs/coordinate_transform.py
+++ /dev/null
@@ -1,9 +0,0 @@
-def get_scale_ratio(model_page_info, page):
- pix = page.get_pixmap(dpi=72)
- pymu_width = int(pix.w)
- pymu_height = int(pix.h)
- width_from_json = model_page_info['page_info']['width']
- height_from_json = model_page_info['page_info']['height']
- horizontal_scale_ratio = width_from_json / pymu_width
- vertical_scale_ratio = height_from_json / pymu_height
- return horizontal_scale_ratio, vertical_scale_ratio
diff --git a/magic_pdf/libs/detect_language_from_model.py b/magic_pdf/libs/detect_language_from_model.py
deleted file mode 100644
index 55abf1058..000000000
--- a/magic_pdf/libs/detect_language_from_model.py
+++ /dev/null
@@ -1,21 +0,0 @@
-from collections import Counter
-
-from magic_pdf.libs.language import detect_lang
-
-def get_language_from_model(model_list: list):
- language_lst = []
- for ocr_page_info in model_list:
- page_text = ""
- layout_dets = ocr_page_info["layout_dets"]
- for layout_det in layout_dets:
- category_id = layout_det["category_id"]
- allow_category_id_list = [15]
- if category_id in allow_category_id_list:
- page_text += layout_det["text"]
- page_language = detect_lang(page_text)
- language_lst.append(page_language)
- # 统计text_language_list中每种语言的个数
- count_dict = Counter(language_lst)
- # 输出text_language_list中出现的次数最多的语言
- language = max(count_dict, key=count_dict.get)
- return language
diff --git a/magic_pdf/libs/draw_bbox.py b/magic_pdf/libs/draw_bbox.py
deleted file mode 100644
index 9703e131e..000000000
--- a/magic_pdf/libs/draw_bbox.py
+++ /dev/null
@@ -1,402 +0,0 @@
-from magic_pdf.data.dataset import PymuDocDataset
-from magic_pdf.libs.commons import fitz # PyMuPDF
-from magic_pdf.libs.Constants import CROSS_PAGE
-from magic_pdf.libs.ocr_content_type import BlockType, CategoryId, ContentType
-from magic_pdf.model.magic_model import MagicModel
-
-
-def draw_bbox_without_number(i, bbox_list, page, rgb_config, fill_config):
- new_rgb = []
- for item in rgb_config:
- item = float(item) / 255
- new_rgb.append(item)
- page_data = bbox_list[i]
- for bbox in page_data:
- x0, y0, x1, y1 = bbox
- rect_coords = fitz.Rect(x0, y0, x1, y1) # Define the rectangle
- if fill_config:
- page.draw_rect(
- rect_coords,
- color=None,
- fill=new_rgb,
- fill_opacity=0.3,
- width=0.5,
- overlay=True,
- ) # Draw the rectangle
- else:
- page.draw_rect(
- rect_coords,
- color=new_rgb,
- fill=None,
- fill_opacity=1,
- width=0.5,
- overlay=True,
- ) # Draw the rectangle
-
-
-def draw_bbox_with_number(i, bbox_list, page, rgb_config, fill_config, draw_bbox=True):
- new_rgb = []
- for item in rgb_config:
- item = float(item) / 255
- new_rgb.append(item)
- page_data = bbox_list[i]
- for j, bbox in enumerate(page_data):
- x0, y0, x1, y1 = bbox
- rect_coords = fitz.Rect(x0, y0, x1, y1) # Define the rectangle
- if draw_bbox:
- if fill_config:
- page.draw_rect(
- rect_coords,
- color=None,
- fill=new_rgb,
- fill_opacity=0.3,
- width=0.5,
- overlay=True,
- ) # Draw the rectangle
- else:
- page.draw_rect(
- rect_coords,
- color=new_rgb,
- fill=None,
- fill_opacity=1,
- width=0.5,
- overlay=True,
- ) # Draw the rectangle
- page.insert_text(
- (x1 + 2, y0 + 10), str(j + 1), fontsize=10, color=new_rgb
- ) # Insert the index in the top left corner of the rectangle
-
-
-def draw_layout_bbox(pdf_info, pdf_bytes, out_path, filename):
- dropped_bbox_list = []
- tables_list, tables_body_list = [], []
- tables_caption_list, tables_footnote_list = [], []
- imgs_list, imgs_body_list, imgs_caption_list = [], [], []
- imgs_footnote_list = []
- titles_list = []
- texts_list = []
- interequations_list = []
- lists_list = []
- indexs_list = []
- for page in pdf_info:
-
- page_dropped_list = []
- tables, tables_body, tables_caption, tables_footnote = [], [], [], []
- imgs, imgs_body, imgs_caption, imgs_footnote = [], [], [], []
- titles = []
- texts = []
- interequations = []
- lists = []
- indices = []
-
- for dropped_bbox in page['discarded_blocks']:
- page_dropped_list.append(dropped_bbox['bbox'])
- dropped_bbox_list.append(page_dropped_list)
- for block in page['para_blocks']:
- bbox = block['bbox']
- if block['type'] == BlockType.Table:
- tables.append(bbox)
- for nested_block in block['blocks']:
- bbox = nested_block['bbox']
- if nested_block['type'] == BlockType.TableBody:
- tables_body.append(bbox)
- elif nested_block['type'] == BlockType.TableCaption:
- tables_caption.append(bbox)
- elif nested_block['type'] == BlockType.TableFootnote:
- tables_footnote.append(bbox)
- elif block['type'] == BlockType.Image:
- imgs.append(bbox)
- for nested_block in block['blocks']:
- bbox = nested_block['bbox']
- if nested_block['type'] == BlockType.ImageBody:
- imgs_body.append(bbox)
- elif nested_block['type'] == BlockType.ImageCaption:
- imgs_caption.append(bbox)
- elif nested_block['type'] == BlockType.ImageFootnote:
- imgs_footnote.append(bbox)
- elif block['type'] == BlockType.Title:
- titles.append(bbox)
- elif block['type'] == BlockType.Text:
- texts.append(bbox)
- elif block['type'] == BlockType.InterlineEquation:
- interequations.append(bbox)
- elif block['type'] == BlockType.List:
- lists.append(bbox)
- elif block['type'] == BlockType.Index:
- indices.append(bbox)
-
- tables_list.append(tables)
- tables_body_list.append(tables_body)
- tables_caption_list.append(tables_caption)
- tables_footnote_list.append(tables_footnote)
- imgs_list.append(imgs)
- imgs_body_list.append(imgs_body)
- imgs_caption_list.append(imgs_caption)
- imgs_footnote_list.append(imgs_footnote)
- titles_list.append(titles)
- texts_list.append(texts)
- interequations_list.append(interequations)
- lists_list.append(lists)
- indexs_list.append(indices)
-
- layout_bbox_list = []
-
- table_type_order = {
- 'table_caption': 1,
- 'table_body': 2,
- 'table_footnote': 3
- }
- for page in pdf_info:
- page_block_list = []
- for block in page['para_blocks']:
- if block['type'] in [
- BlockType.Text,
- BlockType.Title,
- BlockType.InterlineEquation,
- BlockType.List,
- BlockType.Index,
- ]:
- bbox = block['bbox']
- page_block_list.append(bbox)
- elif block['type'] in [BlockType.Image]:
- for sub_block in block['blocks']:
- bbox = sub_block['bbox']
- page_block_list.append(bbox)
- elif block['type'] in [BlockType.Table]:
- sorted_blocks = sorted(block['blocks'], key=lambda x: table_type_order[x['type']])
- for sub_block in sorted_blocks:
- bbox = sub_block['bbox']
- page_block_list.append(bbox)
-
- layout_bbox_list.append(page_block_list)
-
- pdf_docs = fitz.open('pdf', pdf_bytes)
-
- for i, page in enumerate(pdf_docs):
-
- draw_bbox_without_number(i, dropped_bbox_list, page, [158, 158, 158], True)
- # draw_bbox_without_number(i, tables_list, page, [153, 153, 0], True) # color !
- draw_bbox_without_number(i, tables_body_list, page, [204, 204, 0], True)
- draw_bbox_without_number(i, tables_caption_list, page, [255, 255, 102], True)
- draw_bbox_without_number(i, tables_footnote_list, page, [229, 255, 204], True)
- # draw_bbox_without_number(i, imgs_list, page, [51, 102, 0], True)
- draw_bbox_without_number(i, imgs_body_list, page, [153, 255, 51], True)
- draw_bbox_without_number(i, imgs_caption_list, page, [102, 178, 255], True)
- draw_bbox_without_number(i, imgs_footnote_list, page, [255, 178, 102], True),
- draw_bbox_without_number(i, titles_list, page, [102, 102, 255], True)
- draw_bbox_without_number(i, texts_list, page, [153, 0, 76], True)
- draw_bbox_without_number(i, interequations_list, page, [0, 255, 0], True)
- draw_bbox_without_number(i, lists_list, page, [40, 169, 92], True)
- draw_bbox_without_number(i, indexs_list, page, [40, 169, 92], True)
-
- draw_bbox_with_number(
- i, layout_bbox_list, page, [255, 0, 0], False, draw_bbox=False
- )
-
- # Save the PDF
- pdf_docs.save(f'{out_path}/{filename}_layout.pdf')
-
-
-def draw_span_bbox(pdf_info, pdf_bytes, out_path, filename):
- text_list = []
- inline_equation_list = []
- interline_equation_list = []
- image_list = []
- table_list = []
- dropped_list = []
- next_page_text_list = []
- next_page_inline_equation_list = []
-
- def get_span_info(span):
- if span['type'] == ContentType.Text:
- if span.get(CROSS_PAGE, False):
- next_page_text_list.append(span['bbox'])
- else:
- page_text_list.append(span['bbox'])
- elif span['type'] == ContentType.InlineEquation:
- if span.get(CROSS_PAGE, False):
- next_page_inline_equation_list.append(span['bbox'])
- else:
- page_inline_equation_list.append(span['bbox'])
- elif span['type'] == ContentType.InterlineEquation:
- page_interline_equation_list.append(span['bbox'])
- elif span['type'] == ContentType.Image:
- page_image_list.append(span['bbox'])
- elif span['type'] == ContentType.Table:
- page_table_list.append(span['bbox'])
-
- for page in pdf_info:
- page_text_list = []
- page_inline_equation_list = []
- page_interline_equation_list = []
- page_image_list = []
- page_table_list = []
- page_dropped_list = []
-
- # 将跨页的span放到移动到下一页的列表中
- if len(next_page_text_list) > 0:
- page_text_list.extend(next_page_text_list)
- next_page_text_list.clear()
- if len(next_page_inline_equation_list) > 0:
- page_inline_equation_list.extend(next_page_inline_equation_list)
- next_page_inline_equation_list.clear()
-
- # 构造dropped_list
- for block in page['discarded_blocks']:
- if block['type'] == BlockType.Discarded:
- for line in block['lines']:
- for span in line['spans']:
- page_dropped_list.append(span['bbox'])
- dropped_list.append(page_dropped_list)
- # 构造其余useful_list
- for block in page['para_blocks']:
- if block['type'] in [
- BlockType.Text,
- BlockType.Title,
- BlockType.InterlineEquation,
- BlockType.List,
- BlockType.Index,
- ]:
- for line in block['lines']:
- for span in line['spans']:
- get_span_info(span)
- elif block['type'] in [BlockType.Image, BlockType.Table]:
- for sub_block in block['blocks']:
- for line in sub_block['lines']:
- for span in line['spans']:
- get_span_info(span)
- text_list.append(page_text_list)
- inline_equation_list.append(page_inline_equation_list)
- interline_equation_list.append(page_interline_equation_list)
- image_list.append(page_image_list)
- table_list.append(page_table_list)
- pdf_docs = fitz.open('pdf', pdf_bytes)
- for i, page in enumerate(pdf_docs):
- # 获取当前页面的数据
- draw_bbox_without_number(i, text_list, page, [255, 0, 0], False)
- draw_bbox_without_number(i, inline_equation_list, page, [0, 255, 0], False)
- draw_bbox_without_number(i, interline_equation_list, page, [0, 0, 255], False)
- draw_bbox_without_number(i, image_list, page, [255, 204, 0], False)
- draw_bbox_without_number(i, table_list, page, [204, 0, 255], False)
- draw_bbox_without_number(i, dropped_list, page, [158, 158, 158], False)
-
- # Save the PDF
- pdf_docs.save(f'{out_path}/{filename}_spans.pdf')
-
-
-def draw_model_bbox(model_list: list, pdf_bytes, out_path, filename):
- dropped_bbox_list = []
- tables_body_list, tables_caption_list, tables_footnote_list = [], [], []
- imgs_body_list, imgs_caption_list, imgs_footnote_list = [], [], []
- titles_list = []
- texts_list = []
- interequations_list = []
- pdf_docs = fitz.open('pdf', pdf_bytes)
- magic_model = MagicModel(model_list, PymuDocDataset(pdf_bytes))
- for i in range(len(model_list)):
- page_dropped_list = []
- tables_body, tables_caption, tables_footnote = [], [], []
- imgs_body, imgs_caption, imgs_footnote = [], [], []
- titles = []
- texts = []
- interequations = []
- page_info = magic_model.get_model_list(i)
- layout_dets = page_info['layout_dets']
- for layout_det in layout_dets:
- bbox = layout_det['bbox']
- if layout_det['category_id'] == CategoryId.Text:
- texts.append(bbox)
- elif layout_det['category_id'] == CategoryId.Title:
- titles.append(bbox)
- elif layout_det['category_id'] == CategoryId.TableBody:
- tables_body.append(bbox)
- elif layout_det['category_id'] == CategoryId.TableCaption:
- tables_caption.append(bbox)
- elif layout_det['category_id'] == CategoryId.TableFootnote:
- tables_footnote.append(bbox)
- elif layout_det['category_id'] == CategoryId.ImageBody:
- imgs_body.append(bbox)
- elif layout_det['category_id'] == CategoryId.ImageCaption:
- imgs_caption.append(bbox)
- elif layout_det['category_id'] == CategoryId.InterlineEquation_YOLO:
- interequations.append(bbox)
- elif layout_det['category_id'] == CategoryId.Abandon:
- page_dropped_list.append(bbox)
- elif layout_det['category_id'] == CategoryId.ImageFootnote:
- imgs_footnote.append(bbox)
-
- tables_body_list.append(tables_body)
- tables_caption_list.append(tables_caption)
- tables_footnote_list.append(tables_footnote)
- imgs_body_list.append(imgs_body)
- imgs_caption_list.append(imgs_caption)
- titles_list.append(titles)
- texts_list.append(texts)
- interequations_list.append(interequations)
- dropped_bbox_list.append(page_dropped_list)
- imgs_footnote_list.append(imgs_footnote)
-
- for i, page in enumerate(pdf_docs):
- draw_bbox_with_number(
- i, dropped_bbox_list, page, [158, 158, 158], True
- ) # color !
- draw_bbox_with_number(i, tables_body_list, page, [204, 204, 0], True)
- draw_bbox_with_number(i, tables_caption_list, page, [255, 255, 102], True)
- draw_bbox_with_number(i, tables_footnote_list, page, [229, 255, 204], True)
- draw_bbox_with_number(i, imgs_body_list, page, [153, 255, 51], True)
- draw_bbox_with_number(i, imgs_caption_list, page, [102, 178, 255], True)
- draw_bbox_with_number(i, imgs_footnote_list, page, [255, 178, 102], True)
- draw_bbox_with_number(i, titles_list, page, [102, 102, 255], True)
- draw_bbox_with_number(i, texts_list, page, [153, 0, 76], True)
- draw_bbox_with_number(i, interequations_list, page, [0, 255, 0], True)
-
- # Save the PDF
- pdf_docs.save(f'{out_path}/{filename}_model.pdf')
-
-
-def draw_line_sort_bbox(pdf_info, pdf_bytes, out_path, filename):
- layout_bbox_list = []
-
- for page in pdf_info:
- page_line_list = []
- for block in page['preproc_blocks']:
- if block['type'] in [BlockType.Text, BlockType.Title, BlockType.InterlineEquation]:
- for line in block['lines']:
- bbox = line['bbox']
- index = line['index']
- page_line_list.append({'index': index, 'bbox': bbox})
- if block['type'] in [BlockType.Image, BlockType.Table]:
- for sub_block in block['blocks']:
- if sub_block['type'] in [BlockType.ImageBody, BlockType.TableBody]:
- for line in sub_block['virtual_lines']:
- bbox = line['bbox']
- index = line['index']
- page_line_list.append({'index': index, 'bbox': bbox})
- elif sub_block['type'] in [BlockType.ImageCaption, BlockType.TableCaption, BlockType.ImageFootnote, BlockType.TableFootnote]:
- for line in sub_block['lines']:
- bbox = line['bbox']
- index = line['index']
- page_line_list.append({'index': index, 'bbox': bbox})
- sorted_bboxes = sorted(page_line_list, key=lambda x: x['index'])
- layout_bbox_list.append(sorted_bbox['bbox'] for sorted_bbox in sorted_bboxes)
- pdf_docs = fitz.open('pdf', pdf_bytes)
- for i, page in enumerate(pdf_docs):
- draw_bbox_with_number(i, layout_bbox_list, page, [255, 0, 0], False)
-
- pdf_docs.save(f'{out_path}/{filename}_line_sort.pdf')
-
-
-def draw_layout_sort_bbox(pdf_info, pdf_bytes, out_path, filename):
- layout_bbox_list = []
-
- for page in pdf_info:
- page_block_list = []
- for block in page['para_blocks']:
- bbox = block['bbox']
- page_block_list.append(bbox)
- layout_bbox_list.append(page_block_list)
- pdf_docs = fitz.open('pdf', pdf_bytes)
- for i, page in enumerate(pdf_docs):
- draw_bbox_with_number(i, layout_bbox_list, page, [255, 0, 0], False)
-
- pdf_docs.save(f'{out_path}/{filename}_layout_sort.pdf')
diff --git a/magic_pdf/libs/drop_reason.py b/magic_pdf/libs/drop_reason.py
deleted file mode 100644
index 9f574fc2a..000000000
--- a/magic_pdf/libs/drop_reason.py
+++ /dev/null
@@ -1,27 +0,0 @@
-
-class DropReason:
- TEXT_BLCOK_HOR_OVERLAP = "text_block_horizontal_overlap" # 文字块有水平互相覆盖，导致无法准确定位文字顺序
- USEFUL_BLOCK_HOR_OVERLAP = "useful_block_horizontal_overlap" # 需保留的block水平覆盖
- COMPLICATED_LAYOUT = "complicated_layout" # 复杂的布局，暂时不支持
- TOO_MANY_LAYOUT_COLUMNS = "too_many_layout_columns" # 目前不支持分栏超过2列的
- COLOR_BACKGROUND_TEXT_BOX = "color_background_text_box" # 含有带色块的PDF，色块会改变阅读顺序，目前不支持带底色文字块的PDF。
- HIGH_COMPUTATIONAL_lOAD_BY_IMGS = "high_computational_load_by_imgs" # 含特殊图片，计算量太大，从而丢弃
- HIGH_COMPUTATIONAL_lOAD_BY_SVGS = "high_computational_load_by_svgs" # 特殊的SVG图，计算量太大，从而丢弃
- HIGH_COMPUTATIONAL_lOAD_BY_TOTAL_PAGES = "high_computational_load_by_total_pages" # 计算量超过负荷，当前方法下计算量消耗过大
- MISS_DOC_LAYOUT_RESULT = "missing doc_layout_result" # 版面分析失败
- Exception = "_exception" # 解析中发生异常
- ENCRYPTED = "encrypted" # PDF是加密的
- EMPTY_PDF = "total_page=0" # PDF页面总数为0
- NOT_IS_TEXT_PDF = "not_is_text_pdf" # 不是文字版PDF，无法直接解析
- DENSE_SINGLE_LINE_BLOCK = "dense_single_line_block" # 无法清晰的分段
- TITLE_DETECTION_FAILED = "title_detection_failed" # 探测标题失败
- TITLE_LEVEL_FAILED = "title_level_failed" # 分析标题级别失败（例如一级、二级、三级标题）
- PARA_SPLIT_FAILED = "para_split_failed" # 识别段落失败
- PARA_MERGE_FAILED = "para_merge_failed" # 段落合并失败
- NOT_ALLOW_LANGUAGE = "not_allow_language" # 不支持的语种
- SPECIAL_PDF = "special_pdf"
- PSEUDO_SINGLE_COLUMN = "pseudo_single_column" # 无法精确判断文字分栏
- CAN_NOT_DETECT_PAGE_LAYOUT="can_not_detect_page_layout" # 无法分析页面的版面
- NEGATIVE_BBOX_AREA = "negative_bbox_area" # 缩放导致 bbox 面积为负
- OVERLAP_BLOCKS_CAN_NOT_SEPARATION = "overlap_blocks_can_t_separation" # 无法分离重叠的block
-
\ No newline at end of file
diff --git a/magic_pdf/libs/drop_tag.py b/magic_pdf/libs/drop_tag.py
deleted file mode 100644
index b1c577174..000000000
--- a/magic_pdf/libs/drop_tag.py
+++ /dev/null
@@ -1,19 +0,0 @@
-
-COLOR_BG_HEADER_TXT_BLOCK = "color_background_header_txt_block"
-PAGE_NO = "page-no" # 页码
-CONTENT_IN_FOOT_OR_HEADER = 'in-foot-header-area' # 页眉页脚内的文本
-VERTICAL_TEXT = 'vertical-text' # 垂直文本
-ROTATE_TEXT = 'rotate-text' # 旋转文本
-EMPTY_SIDE_BLOCK = 'empty-side-block' # 边缘上的空白没有任何内容的block
-ON_IMAGE_TEXT = 'on-image-text' # 文本在图片上
-ON_TABLE_TEXT = 'on-table-text' # 文本在表格上
-
-
-class DropTag:
- PAGE_NUMBER = "page_no"
- HEADER = "header"
- FOOTER = "footer"
- FOOTNOTE = "footnote"
- NOT_IN_LAYOUT = "not_in_layout"
- SPAN_OVERLAP = "span_overlap"
- BLOCK_OVERLAP = "block_overlap"
diff --git a/magic_pdf/libs/hash_utils.py b/magic_pdf/libs/hash_utils.py
deleted file mode 100644
index 47b8aea74..000000000
--- a/magic_pdf/libs/hash_utils.py
+++ /dev/null
@@ -1,15 +0,0 @@
-import hashlib
-
-
-def compute_md5(file_bytes):
- hasher = hashlib.md5()
- hasher.update(file_bytes)
- return hasher.hexdigest().upper()
-
-
-def compute_sha256(input_string):
- hasher = hashlib.sha256()
- # 在Python3中，需要将字符串转化为字节对象才能被哈希函数处理
- input_bytes = input_string.encode('utf-8')
- hasher.update(input_bytes)
- return hasher.hexdigest()
diff --git a/magic_pdf/libs/json_compressor.py b/magic_pdf/libs/json_compressor.py
deleted file mode 100644
index 77ef1c876..000000000
--- a/magic_pdf/libs/json_compressor.py
+++ /dev/null
@@ -1,27 +0,0 @@
-import json
-import brotli
-import base64
-
-class JsonCompressor:
-
- @staticmethod
- def compress_json(data):
- """
- Compress a json object and encode it with base64
- """
- json_str = json.dumps(data)
- json_bytes = json_str.encode('utf-8')
- compressed = brotli.compress(json_bytes, quality=6)
- compressed_str = base64.b64encode(compressed).decode('utf-8') # convert bytes to string
- return compressed_str
-
- @staticmethod
- def decompress_json(compressed_str):
- """
- Decode the base64 string and decompress the json object
- """
- compressed = base64.b64decode(compressed_str.encode('utf-8')) # convert string to bytes
- decompressed_bytes = brotli.decompress(compressed)
- json_str = decompressed_bytes.decode('utf-8')
- data = json.loads(json_str)
- return data
diff --git a/magic_pdf/libs/language.py b/magic_pdf/libs/language.py
deleted file mode 100644
index 396c9008d..000000000
--- a/magic_pdf/libs/language.py
+++ /dev/null
@@ -1,36 +0,0 @@
-import os
-import unicodedata
-
-if not os.getenv("FTLANG_CACHE"):
- current_file_path = os.path.abspath(__file__)
- current_dir = os.path.dirname(current_file_path)
- root_dir = os.path.dirname(current_dir)
- ftlang_cache_dir = os.path.join(root_dir, 'resources', 'fasttext-langdetect')
- os.environ["FTLANG_CACHE"] = str(ftlang_cache_dir)
- # print(os.getenv("FTLANG_CACHE"))
-
-from fast_langdetect import detect_language
-
-
-def detect_lang(text: str) -> str:
-
- if len(text) == 0:
- return ""
- try:
- lang_upper = detect_language(text)
- except:
- html_no_ctrl_chars = ''.join([l for l in text if unicodedata.category(l)[0] not in ['C', ]])
- lang_upper = detect_language(html_no_ctrl_chars)
- try:
- lang = lang_upper.lower()
- except:
- lang = ""
- return lang
-
-
-if __name__ == '__main__':
- print(os.getenv("FTLANG_CACHE"))
- print(detect_lang("This is a test."))
- print(detect_lang("This is a test"))
- print(detect_lang("这个是中文测试。"))
- print(detect_lang("这个是中文测试。"))
diff --git a/magic_pdf/libs/local_math.py b/magic_pdf/libs/local_math.py
deleted file mode 100644
index 9edbcc707..000000000
--- a/magic_pdf/libs/local_math.py
+++ /dev/null
@@ -1,9 +0,0 @@
-def float_gt(a, b):
- if 0.0001 >= abs(a -b):
- return False
- return a > b
-
-def float_equal(a, b):
- if 0.0001 >= abs(a-b):
- return True
- return False
\ No newline at end of file
diff --git a/magic_pdf/libs/markdown_utils.py b/magic_pdf/libs/markdown_utils.py
deleted file mode 100644
index 5708b477e..000000000
--- a/magic_pdf/libs/markdown_utils.py
+++ /dev/null
@@ -1,31 +0,0 @@
-import re
-
-
-def escape_special_markdown_char(pymu_blocks):
- """
- 转义正文里对markdown语法有特殊意义的字符
- """
- special_chars = ["*", "`", "~", "$"]
- for blk in pymu_blocks:
- for line in blk['lines']:
- for span in line['spans']:
- for char in special_chars:
- span_text = span['text']
- span_type = span.get("_type", None)
- if span_type in ['inline-equation', 'interline-equation']:
- continue
- elif span_text:
- span['text'] = span['text'].replace(char, "\\" + char)
-
- return pymu_blocks
-
-
-def ocr_escape_special_markdown_char(content):
- """
- 转义正文里对markdown语法有特殊意义的字符
- """
- special_chars = ["*", "`", "~", "$"]
- for char in special_chars:
- content = content.replace(char, "\\" + char)
-
- return content
diff --git a/magic_pdf/libs/nlp_utils.py b/magic_pdf/libs/nlp_utils.py
deleted file mode 100644
index 49a7365b3..000000000
--- a/magic_pdf/libs/nlp_utils.py
+++ /dev/null
@@ -1,203 +0,0 @@
-import re
-from os import path
-
-from collections import Counter
-
-from loguru import logger
-
-# from langdetect import detect
-import spacy
-import en_core_web_sm
-import zh_core_web_sm
-
-from magic_pdf.libs.language import detect_lang
-
-
-class NLPModels:
- """
- How to upload local models to s3:
- - config aws cli:
- doc\SETUP-CLI.md
- doc\setup_cli.sh
- app\config\__init__.py
- - $ cd {local_dir_storing_models}
- - $ ls models
- en_core_web_sm-3.7.1/
- zh_core_web_sm-3.7.0/
- - $ aws s3 sync models/ s3://llm-infra/models --profile=p_project_norm
- - $ aws s3 --profile=p_project_norm ls s3://llm-infra/models/
- PRE en_core_web_sm-3.7.1/
- PRE zh_core_web_sm-3.7.0/
- """
-
- def __init__(self):
- # if OS is windows, set "TMP_DIR" to "D:/tmp"
-
- home_dir = path.expanduser("~")
- self.default_local_path = path.join(home_dir, ".nlp_models")
- self.default_shared_path = "/share/pdf_processor/nlp_models"
- self.default_hdfs_path = "hdfs://pdf_processor/nlp_models"
- self.default_s3_path = "s3://llm-infra/models"
- self.nlp_models = self.nlp_models = {
- "en_core_web_sm": {
- "type": "spacy",
- "version": "3.7.1",
- },
- "en_core_web_md": {
- "type": "spacy",
- "version": "3.7.1",
- },
- "en_core_web_lg": {
- "type": "spacy",
- "version": "3.7.1",
- },
- "zh_core_web_sm": {
- "type": "spacy",
- "version": "3.7.0",
- },
- "zh_core_web_md": {
- "type": "spacy",
- "version": "3.7.0",
- },
- "zh_core_web_lg": {
- "type": "spacy",
- "version": "3.7.0",
- },
- }
- self.en_core_web_sm_model = en_core_web_sm.load()
- self.zh_core_web_sm_model = zh_core_web_sm.load()
-
- def load_model(self, model_name, model_type, model_version):
- if (
- model_name in self.nlp_models
- and self.nlp_models[model_name]["type"] == model_type
- and self.nlp_models[model_name]["version"] == model_version
- ):
- return spacy.load(model_name) if spacy.util.is_package(model_name) else None
-
- else:
- logger.error(f"Unsupported model name or version: {model_name} {model_version}")
- return None
-
- def detect_language(self, text, use_langdetect=False):
- if len(text) == 0:
- return None
- if use_langdetect:
- # print("use_langdetect")
- # print(detect_lang(text))
- # return detect_lang(text)
- if detect_lang(text) == "zh":
- return "zh"
- else:
- return "en"
-
- if not use_langdetect:
- en_count = len(re.findall(r"[a-zA-Z]", text))
- cn_count = len(re.findall(r"[\u4e00-\u9fff]", text))
-
- if en_count > cn_count:
- return "en"
-
- if cn_count > en_count:
- return "zh"
-
- def detect_entity_catgr_using_nlp(self, text, threshold=0.5):
- """
- Detect entity categories using NLP models and return the most frequent entity types.
-
- Parameters
- ----------
- text : str
- Text to be processed.
-
- Returns
- -------
- str
- The most frequent entity type.
- """
- lang = self.detect_language(text, use_langdetect=True)
-
- if lang == "en":
- nlp_model = self.en_core_web_sm_model
- elif lang == "zh":
- nlp_model = self.zh_core_web_sm_model
- else:
- # logger.error(f"Unsupported language: {lang}")
- return {}
-
- # Splitting text into smaller parts
- text_parts = re.split(r"[,;，；、\s & |]+", text)
-
- text_parts = [part for part in text_parts if not re.match(r"[\d\W]+", part)] # Remove non-words
- text_combined = " ".join(text_parts)
-
- try:
- doc = nlp_model(text_combined)
- entity_counts = Counter([ent.label_ for ent in doc.ents])
- word_counts_in_entities = Counter()
-
- for ent in doc.ents:
- word_counts_in_entities[ent.label_] += len(ent.text.split())
-
- total_words_in_entities = sum(word_counts_in_entities.values())
- total_words = len([token for token in doc if not token.is_punct])
-
- if total_words_in_entities == 0 or total_words == 0:
- return None
-
- entity_percentage = total_words_in_entities / total_words
- if entity_percentage < 0.5:
- return None
-
- most_common_entity, word_count = word_counts_in_entities.most_common(1)[0]
- entity_percentage = word_count / total_words_in_entities
-
- if entity_percentage >= threshold:
- return most_common_entity
- else:
- return None
- except Exception as e:
- logger.error(f"Error in entity detection: {e}")
- return None
-
-
-def __main__():
- nlpModel = NLPModels()
-
- test_strings = [
- "张三",
- "张三, 李四，王五; 赵六",
- "John Doe",
- "Jane Smith",
- "Lee, John",
- "John Doe, Jane Smith; Alice Johnson，Bob Lee",
- "孙七, Michael Jordan；赵八",
- "David Smith Michael O'Connor; Kevin ßáçøñ",
- "李雷·韩梅梅, 张三·李四",
- "Charles Robert Darwin, Isaac Newton",
- "莱昂纳多·迪卡普里奥, 杰克·吉伦哈尔",
- "John Doe, Jane Smith; Alice Johnson",
- "张三, 李四，王五; 赵六",
- "Lei Wang, Jia Li, and Xiaojun Chen, LINKE YANG OU, and YUAN ZHANG",
- "Rachel Mills & William Barry & Susanne B. Haga",
- "Claire Chabut* and Jean-François Bussières",
- "1 Department of Chemistry, Northeastern University, Shenyang 110004, China 2 State Key Laboratory of Polymer Physics and Chemistry, Changchun Institute of Applied Chemistry, Chinese Academy of Sciences, Changchun 130022, China",
- "Changchun",
- "china",
- "Rongjun Song, 1,2 Baoyan Zhang, 1 Baotong Huang, 2 Tao Tang 2",
- "Synergistic Effect of Supported Nickel Catalyst with Intumescent Flame-Retardants on Flame Retardancy and Thermal Stability of Polypropylene",
- "Synergistic Effect of Supported Nickel Catalyst with",
- "Intumescent Flame-Retardants on Flame Retardancy",
- "and Thermal Stability of Polypropylene",
- ]
-
- for test in test_strings:
- print()
- print(f"Original String: {test}")
-
- result = nlpModel.detect_entity_catgr_using_nlp(test)
- print(f"Detected entities: {result}")
-
-
-if __name__ == "__main__":
- __main__()
diff --git a/magic_pdf/libs/ocr_content_type.py b/magic_pdf/libs/ocr_content_type.py
deleted file mode 100644
index 30d88cfde..000000000
--- a/magic_pdf/libs/ocr_content_type.py
+++ /dev/null
@@ -1,40 +0,0 @@
-class ContentType:
- Image = 'image'
- Table = 'table'
- Text = 'text'
- InlineEquation = 'inline_equation'
- InterlineEquation = 'interline_equation'
-
-
-class BlockType:
- Image = 'image'
- ImageBody = 'image_body'
- ImageCaption = 'image_caption'
- ImageFootnote = 'image_footnote'
- Table = 'table'
- TableBody = 'table_body'
- TableCaption = 'table_caption'
- TableFootnote = 'table_footnote'
- Text = 'text'
- Title = 'title'
- InterlineEquation = 'interline_equation'
- Footnote = 'footnote'
- Discarded = 'discarded'
- List = 'list'
- Index = 'index'
-
-
-class CategoryId:
- Title = 0
- Text = 1
- Abandon = 2
- ImageBody = 3
- ImageCaption = 4
- TableBody = 5
- TableCaption = 6
- TableFootnote = 7
- InterlineEquation_Layout = 8
- InlineEquation = 13
- InterlineEquation_YOLO = 14
- OcrText = 15
- ImageFootnote = 101
diff --git a/magic_pdf/libs/pdf_check.py b/magic_pdf/libs/pdf_check.py
deleted file mode 100644
index 3f9dc350c..000000000
--- a/magic_pdf/libs/pdf_check.py
+++ /dev/null
@@ -1,62 +0,0 @@
-from io import BytesIO
-import re
-import fitz
-import numpy as np
-from loguru import logger
-from pdfminer.high_level import extract_text
-
-
-def calculate_sample_count(total_page: int):
- """
- 根据总页数和采样率计算采样页面的数量。
- """
- select_page_cnt = min(10, total_page)
- return select_page_cnt
-
-
-def extract_pages(src_pdf_bytes: bytes):
- pdf_docs = fitz.open("pdf", src_pdf_bytes)
- total_page = len(pdf_docs)
- if total_page == 0:
- # 如果PDF没有页面，直接返回空文档
- logger.warning("PDF is empty, return empty document")
- return fitz.Document()
- select_page_cnt = calculate_sample_count(total_page)
-
- page_num = np.random.choice(total_page, select_page_cnt, replace=False)
- sample_docs = fitz.Document()
- try:
- for index in page_num:
- sample_docs.insert_pdf(pdf_docs, from_page=int(index), to_page=int(index))
- except Exception as e:
- logger.exception(e)
- return sample_docs
-
-
-def detect_invalid_chars(src_pdf_bytes: bytes) -> bool:
- """"
- 检测PDF中是否包含非法字符
- """
- '''pdfminer比较慢,需要先随机抽取10页左右的sample'''
- sample_docs = extract_pages(src_pdf_bytes)
- sample_pdf_bytes = sample_docs.tobytes()
- sample_pdf_file_like_object = BytesIO(sample_pdf_bytes)
- text = extract_text(sample_pdf_file_like_object)
- text = text.replace("\n", "")
- # logger.info(text)
- '''乱码文本用pdfminer提取出来的文本特征是(cid:xxx)'''
- cid_pattern = re.compile(r'\(cid:\d+\)')
- matches = cid_pattern.findall(text)
- cid_count = len(matches)
- cid_len = sum(len(match) for match in matches)
- text_len = len(text)
- if text_len == 0:
- cid_chars_radio = 0
- else:
- cid_chars_radio = cid_count/(cid_count + text_len - cid_len)
- logger.info(f"cid_count: {cid_count}, text_len: {text_len}, cid_chars_radio: {cid_chars_radio}")
- '''当一篇文章存在5%以上的文本是乱码时,认为该文档为乱码文档'''
- if cid_chars_radio > 0.05:
- return False # 乱码文档
- else:
- return True # 正常文档
diff --git a/magic_pdf/libs/pdf_image_tools.py b/magic_pdf/libs/pdf_image_tools.py
deleted file mode 100644
index 2e5a57eec..000000000
--- a/magic_pdf/libs/pdf_image_tools.py
+++ /dev/null
@@ -1,33 +0,0 @@
-
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.libs.commons import fitz
-from magic_pdf.libs.commons import join_path
-from magic_pdf.libs.hash_utils import compute_sha256
-
-
-def cut_image(bbox: tuple, page_num: int, page: fitz.Page, return_path, imageWriter: AbsReaderWriter):
- """
- 从第page_num页的page中，根据bbox进行裁剪出一张jpg图片，返回图片路径
- save_path：需要同时支持s3和本地, 图片存放在save_path下，文件名是: {page_num}_{bbox[0]}_{bbox[1]}_{bbox[2]}_{bbox[3]}.jpg , bbox内数字取整。
- """
- # 拼接文件名
- filename = f"{page_num}_{int(bbox[0])}_{int(bbox[1])}_{int(bbox[2])}_{int(bbox[3])}"
-
- # 老版本返回不带bucket的路径
- img_path = join_path(return_path, filename) if return_path is not None else None
-
- # 新版本生成平铺路径
- img_hash256_path = f"{compute_sha256(img_path)}.jpg"
-
- # 将坐标转换为fitz.Rect对象
- rect = fitz.Rect(*bbox)
- # 配置缩放倍数为3倍
- zoom = fitz.Matrix(3, 3)
- # 截取图片
- pix = page.get_pixmap(clip=rect, matrix=zoom)
-
- byte_data = pix.tobytes(output='jpeg', jpg_quality=95)
-
- imageWriter.write(byte_data, img_hash256_path, AbsReaderWriter.MODE_BIN)
-
- return img_hash256_path
diff --git a/magic_pdf/libs/safe_filename.py b/magic_pdf/libs/safe_filename.py
deleted file mode 100644
index 1076a4bae..000000000
--- a/magic_pdf/libs/safe_filename.py
+++ /dev/null
@@ -1,11 +0,0 @@
-import os
-
-
-def sanitize_filename(filename, replacement="_"):
- if os.name == 'nt':
- invalid_chars = '<>:"|?*'
-
- for char in invalid_chars:
- filename = filename.replace(char, replacement)
-
- return filename
diff --git a/magic_pdf/libs/textbase.py b/magic_pdf/libs/textbase.py
deleted file mode 100644
index bb8875b38..000000000
--- a/magic_pdf/libs/textbase.py
+++ /dev/null
@@ -1,33 +0,0 @@
-import math
-
-
-def __inc_dict_val(mp, key, val_inc:int):
- if mp.get(key):
- mp[key] = mp[key] + val_inc
- else:
- mp[key] = val_inc
-
-
-
-def get_text_block_base_info(block):
- """
- 获取这个文本块里的字体的颜色、字号、字体
- 按照正文字数最多的返回
- """
-
- counter = {}
-
- for line in block['lines']:
- for span in line['spans']:
- color = span['color']
- size = round(span['size'], 2)
- font = span['font']
-
- txt_len = len(span['text'])
- __inc_dict_val(counter, (color, size, font), txt_len)
-
-
- c, s, ft = max(counter, key=counter.get)
-
- return c, s, ft
-
\ No newline at end of file
diff --git a/magic_pdf/libs/version.py b/magic_pdf/libs/version.py
deleted file mode 100644
index 777f190df..000000000
--- a/magic_pdf/libs/version.py
+++ /dev/null
@@ -1 +0,0 @@
-__version__ = "0.8.0"
diff --git a/magic_pdf/libs/vis_utils.py b/magic_pdf/libs/vis_utils.py
deleted file mode 100644
index 5a4988a77..000000000
--- a/magic_pdf/libs/vis_utils.py
+++ /dev/null
@@ -1,308 +0,0 @@
-from magic_pdf.libs.commons import fitz
-import os
-
-
-def draw_bbox_on_page(raw_pdf_doc: fitz.Document, paras_dict:dict, save_path: str):
- """
- 在page上画出bbox，保存到save_path
- """
- # 检查文件是否存在
- is_new_pdf = False
- if os.path.exists(save_path):
- # 打开现有的 PDF 文件
- doc = fitz.open(save_path)
- else:
- # 创建一个新的空白 PDF 文件
- is_new_pdf = True
- doc = fitz.open('')
-
- color_map = {
- 'image': fitz.pdfcolor["yellow"],
- 'text': fitz.pdfcolor['blue'],
- "table": fitz.pdfcolor['green']
- }
-
- for k, v in paras_dict.items():
- page_idx = v['page_idx']
- width = raw_pdf_doc[page_idx].rect.width
- height = raw_pdf_doc[page_idx].rect.height
- new_page = doc.new_page(width=width, height=height)
-
- shape = new_page.new_shape()
- for order, block in enumerate(v['preproc_blocks']):
- rect = fitz.Rect(block['bbox'])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=color_map['text'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for img in v['images']:
- # 原始box画上去
- rect = fitz.Rect(img['bbox'])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['yellow'])
- shape.finish()
- shape.commit()
-
- for img in v['image_backup']:
- # 原始box画上去
- rect = fitz.Rect(img['bbox'])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['yellow'], fill=None)
- shape.finish()
- shape.commit()
-
- for tb in v['droped_text_block']:
- # 原始box画上去
- rect = fitz.Rect(tb['bbox'])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['black'], fill_opacity=0.4)
- shape.finish()
- shape.commit()
-
- # TODO table
- for tb in v['tables']:
- rect = fitz.Rect(tb['bbox'])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['green'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
-
- parent_dir = os.path.dirname(save_path)
- if not os.path.exists(parent_dir):
- os.makedirs(parent_dir)
-
- if is_new_pdf:
- doc.save(save_path)
- else:
- doc.saveIncr()
- doc.close()
-
-
-def debug_show_bbox(raw_pdf_doc: fitz.Document, page_idx: int, bboxes: list, droped_bboxes:list, expect_drop_bboxes:list, save_path: str, expected_page_id:int):
- """
- 以覆盖的方式写个临时的pdf，用于debug
- """
- if page_idx!=expected_page_id:
- return
-
- if os.path.exists(save_path):
- # 删除已经存在的文件
- os.remove(save_path)
- # 创建一个新的空白 PDF 文件
- doc = fitz.open('')
-
- width = raw_pdf_doc[page_idx].rect.width
- height = raw_pdf_doc[page_idx].rect.height
- new_page = doc.new_page(width=width, height=height)
-
- shape = new_page.new_shape()
- for bbox in bboxes:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=fitz.pdfcolor['blue'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in droped_bboxes:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['yellow'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in expect_drop_bboxes:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=None)
- shape.finish()
- shape.commit()
-
- # shape.insert_textbox(fitz.Rect(200, 0, 600, 20), f"total bboxes: {len(bboxes)}", fontname="helv", fontsize=12,
- # color=(0, 0, 0))
- # shape.finish(color=fitz.pdfcolor['black'])
- # shape.commit()
-
- parent_dir = os.path.dirname(save_path)
- if not os.path.exists(parent_dir):
- os.makedirs(parent_dir)
-
- doc.save(save_path)
- doc.close()
-
-
-def debug_show_page(page, bboxes1: list,bboxes2: list,bboxes3: list,):
- save_path = "./tmp/debug.pdf"
- if os.path.exists(save_path):
- # 删除已经存在的文件
- os.remove(save_path)
- # 创建一个新的空白 PDF 文件
- doc = fitz.open('')
-
- width = page.rect.width
- height = page.rect.height
- new_page = doc.new_page(width=width, height=height)
-
- shape = new_page.new_shape()
- for bbox in bboxes1:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=fitz.pdfcolor['blue'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in bboxes2:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=None, fill=fitz.pdfcolor['yellow'], fill_opacity=0.2)
- shape.finish()
- shape.commit()
-
- for bbox in bboxes3:
- # 原始box画上去
- rect = fitz.Rect(*bbox[0:4])
- shape = new_page.new_shape()
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=None)
- shape.finish()
- shape.commit()
-
- parent_dir = os.path.dirname(save_path)
- if not os.path.exists(parent_dir):
- os.makedirs(parent_dir)
-
- doc.save(save_path)
- doc.close()
-
-
-
-
-def draw_layout_bbox_on_page(raw_pdf_doc: fitz.Document, paras_dict:dict, header, footer, pdf_path: str):
- """
- 在page上画出bbox，保存到save_path
- """
- # 检查文件是否存在
- is_new_pdf = False
- if os.path.exists(pdf_path):
- # 打开现有的 PDF 文件
- doc = fitz.open(pdf_path)
- else:
- # 创建一个新的空白 PDF 文件
- is_new_pdf = True
- doc = fitz.open('')
-
- for k, v in paras_dict.items():
- page_idx = v['page_idx']
- layouts = v['layout_bboxes']
- page = doc[page_idx]
- shape = page.new_shape()
- for order, layout in enumerate(layouts):
- border_offset = 1
- rect_box = layout['layout_bbox']
- layout_label = layout['layout_label']
- fill_color = fitz.pdfcolor['pink'] if layout_label=='U' else None
- rect_box = [rect_box[0]+1, rect_box[1]-border_offset, rect_box[2]-1, rect_box[3]+border_offset]
- rect = fitz.Rect(*rect_box)
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=fill_color, fill_opacity=0.4)
- """
- draw order text on layout box
- """
- font_size = 10
- shape.insert_text((rect_box[0] + 1, rect_box[1] + font_size), f"{order}", fontsize=font_size, color=(0, 0, 0))
-
- """画上footer header"""
- if header:
- shape.draw_rect(fitz.Rect(header))
- shape.finish(color=None, fill=fitz.pdfcolor['black'], fill_opacity=0.2)
- if footer:
- shape.draw_rect(fitz.Rect(footer))
- shape.finish(color=None, fill=fitz.pdfcolor['black'], fill_opacity=0.2)
-
- shape.commit()
-
- if is_new_pdf:
- doc.save(pdf_path)
- else:
- doc.saveIncr()
- doc.close()
-
-
-@DeprecationWarning
-def draw_layout_on_page(raw_pdf_doc: fitz.Document, page_idx: int, page_layout: list, pdf_path: str):
- """
- 把layout的box用红色边框花在pdf_path的page_idx上
- """
- def draw(shape, layout, fill_color=fitz.pdfcolor['pink']):
- border_offset = 1
- rect_box = layout['layout_bbox']
- layout_label = layout['layout_label']
- sub_layout = layout['sub_layout']
- if len(sub_layout)==0:
- fill_color = fill_color if layout_label=='U' else None
- rect_box = [rect_box[0]+1, rect_box[1]-border_offset, rect_box[2]-1, rect_box[3]+border_offset]
- rect = fitz.Rect(*rect_box)
- shape.draw_rect(rect)
- shape.finish(color=fitz.pdfcolor['red'], fill=fill_color, fill_opacity=0.2)
- # if layout_label=='U':
- # bad_boxes = layout.get("bad_boxes", [])
- # for bad_box in bad_boxes:
- # rect = fitz.Rect(*bad_box)
- # shape.draw_rect(rect)
- # shape.finish(color=fitz.pdfcolor['red'], fill=fitz.pdfcolor['red'], fill_opacity=0.2)
- # else:
- # rect = fitz.Rect(*rect_box)
- # shape.draw_rect(rect)
- # shape.finish(color=fitz.pdfcolor['blue'])
-
- for sub_layout in sub_layout:
- draw(shape, sub_layout)
- shape.commit()
-
-
- # 检查文件是否存在
- is_new_pdf = False
- if os.path.exists(pdf_path):
- # 打开现有的 PDF 文件
- doc = fitz.open(pdf_path)
- else:
- # 创建一个新的空白 PDF 文件
- is_new_pdf = True
- doc = fitz.open('')
-
- page = doc[page_idx]
- shape = page.new_shape()
- for order, layout in enumerate(page_layout):
- draw(shape, layout, fitz.pdfcolor['yellow'])
-
- # shape.insert_textbox(fitz.Rect(200, 0, 600, 20), f"total bboxes: {len(layout)}", fontname="helv", fontsize=12,
- # color=(0, 0, 0))
- # shape.finish(color=fitz.pdfcolor['black'])
- # shape.commit()
-
- parent_dir = os.path.dirname(pdf_path)
- if not os.path.exists(parent_dir):
- os.makedirs(parent_dir)
-
- if is_new_pdf:
- doc.save(pdf_path)
- else:
- doc.saveIncr()
- doc.close()
-
\ No newline at end of file
diff --git a/magic_pdf/model/__init__.py b/magic_pdf/model/__init__.py
deleted file mode 100644
index 08653f24e..000000000
--- a/magic_pdf/model/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-__use_inside_model__ = True
-__model_mode__ = "full"
diff --git a/magic_pdf/model/doc_analyze_by_custom_model.py b/magic_pdf/model/doc_analyze_by_custom_model.py
deleted file mode 100644
index ee50d6eb3..000000000
--- a/magic_pdf/model/doc_analyze_by_custom_model.py
+++ /dev/null
@@ -1,183 +0,0 @@
-import time
-
-import fitz
-import numpy as np
-from loguru import logger
-
-from magic_pdf.libs.clean_memory import clean_memory
-from magic_pdf.libs.config_reader import get_local_models_dir, get_device, get_table_recog_config, get_layout_config, \
- get_formula_config
-from magic_pdf.model.model_list import MODEL
-import magic_pdf.model as model_config
-
-
-def dict_compare(d1, d2):
- return d1.items() == d2.items()
-
-
-def remove_duplicates_dicts(lst):
- unique_dicts = []
- for dict_item in lst:
- if not any(
- dict_compare(dict_item, existing_dict) for existing_dict in unique_dicts
- ):
- unique_dicts.append(dict_item)
- return unique_dicts
-
-
-def load_images_from_pdf(pdf_bytes: bytes, dpi=200, start_page_id=0, end_page_id=None) -> list:
- try:
- from PIL import Image
- except ImportError:
- logger.error("Pillow not installed, please install by pip.")
- exit(1)
-
- images = []
- with fitz.open("pdf", pdf_bytes) as doc:
- pdf_page_num = doc.page_count
- end_page_id = end_page_id if end_page_id is not None and end_page_id >= 0 else pdf_page_num - 1
- if end_page_id > pdf_page_num - 1:
- logger.warning("end_page_id is out of range, use images length")
- end_page_id = pdf_page_num - 1
-
- for index in range(0, doc.page_count):
- if start_page_id <= index <= end_page_id:
- page = doc[index]
- mat = fitz.Matrix(dpi / 72, dpi / 72)
- pm = page.get_pixmap(matrix=mat, alpha=False)
-
- # If the width or height exceeds 9000 after scaling, do not scale further.
- if pm.width > 9000 or pm.height > 9000:
- pm = page.get_pixmap(matrix=fitz.Matrix(1, 1), alpha=False)
-
- img = Image.frombytes("RGB", (pm.width, pm.height), pm.samples)
- img = np.array(img)
- img_dict = {"img": img, "width": pm.width, "height": pm.height}
- else:
- img_dict = {"img": [], "width": 0, "height": 0}
-
- images.append(img_dict)
- return images
-
-
-class ModelSingleton:
- _instance = None
- _models = {}
-
- def __new__(cls, *args, **kwargs):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
- return cls._instance
-
- def get_model(self, ocr: bool, show_log: bool, lang=None, layout_model=None, formula_enable=None, table_enable=None):
- key = (ocr, show_log, lang, layout_model, formula_enable, table_enable)
- if key not in self._models:
- self._models[key] = custom_model_init(ocr=ocr, show_log=show_log, lang=lang, layout_model=layout_model,
- formula_enable=formula_enable, table_enable=table_enable)
- return self._models[key]
-
-
-def custom_model_init(ocr: bool = False, show_log: bool = False, lang=None,
- layout_model=None, formula_enable=None, table_enable=None):
-
- model = None
-
- if model_config.__model_mode__ == "lite":
- logger.warning("The Lite mode is provided for developers to conduct testing only, and the output quality is "
- "not guaranteed to be reliable.")
- model = MODEL.Paddle
- elif model_config.__model_mode__ == "full":
- model = MODEL.PEK
-
- if model_config.__use_inside_model__:
- model_init_start = time.time()
- if model == MODEL.Paddle:
- from magic_pdf.model.pp_structure_v2 import CustomPaddleModel
- custom_model = CustomPaddleModel(ocr=ocr, show_log=show_log, lang=lang)
- elif model == MODEL.PEK:
- from magic_pdf.model.pdf_extract_kit import CustomPEKModel
- # 从配置文件读取model-dir和device
- local_models_dir = get_local_models_dir()
- device = get_device()
-
- layout_config = get_layout_config()
- if layout_model is not None:
- layout_config["model"] = layout_model
-
- formula_config = get_formula_config()
- if formula_enable is not None:
- formula_config["enable"] = formula_enable
-
- table_config = get_table_recog_config()
- if table_enable is not None:
- table_config["enable"] = table_enable
-
- model_input = {
- "ocr": ocr,
- "show_log": show_log,
- "models_dir": local_models_dir,
- "device": device,
- "table_config": table_config,
- "layout_config": layout_config,
- "formula_config": formula_config,
- "lang": lang,
- }
-
- custom_model = CustomPEKModel(**model_input)
- else:
- logger.error("Not allow model_name!")
- exit(1)
- model_init_cost = time.time() - model_init_start
- logger.info(f"model init cost: {model_init_cost}")
- else:
- logger.error("use_inside_model is False, not allow to use inside model")
- exit(1)
-
- return custom_model
-
-
-def doc_analyze(pdf_bytes: bytes, ocr: bool = False, show_log: bool = False,
- start_page_id=0, end_page_id=None, lang=None,
- layout_model=None, formula_enable=None, table_enable=None):
-
- if lang == "":
- lang = None
-
- model_manager = ModelSingleton()
- custom_model = model_manager.get_model(ocr, show_log, lang, layout_model, formula_enable, table_enable)
-
- with fitz.open("pdf", pdf_bytes) as doc:
- pdf_page_num = doc.page_count
- end_page_id = end_page_id if end_page_id is not None and end_page_id >= 0 else pdf_page_num - 1
- if end_page_id > pdf_page_num - 1:
- logger.warning("end_page_id is out of range, use images length")
- end_page_id = pdf_page_num - 1
-
- images = load_images_from_pdf(pdf_bytes, start_page_id=start_page_id, end_page_id=end_page_id)
-
- model_json = []
- doc_analyze_start = time.time()
-
- for index, img_dict in enumerate(images):
- img = img_dict["img"]
- page_width = img_dict["width"]
- page_height = img_dict["height"]
- if start_page_id <= index <= end_page_id:
- result = custom_model(img)
- else:
- result = []
- page_info = {"page_no": index, "height": page_height, "width": page_width}
- page_dict = {"layout_dets": result, "page_info": page_info}
- model_json.append(page_dict)
-
- gc_start = time.time()
- clean_memory()
- gc_time = round(time.time() - gc_start, 2)
- logger.info(f"gc time: {gc_time}")
-
- doc_analyze_time = round(time.time() - doc_analyze_start, 2)
- doc_analyze_speed = round( (end_page_id + 1 - start_page_id) / doc_analyze_time, 2)
- logger.info(f"doc analyze time: {round(time.time() - doc_analyze_start, 2)},"
- f" speed: {doc_analyze_speed} pages/second")
-
- return model_json
diff --git a/magic_pdf/model/magic_model.py b/magic_pdf/model/magic_model.py
deleted file mode 100644
index 62bbeb152..000000000
--- a/magic_pdf/model/magic_model.py
+++ /dev/null
@@ -1,1023 +0,0 @@
-import json
-
-from magic_pdf.data.dataset import Dataset
-from magic_pdf.libs.boxbase import (_is_in, _is_part_overlap, bbox_distance,
- bbox_relative_pos, box_area, calculate_iou,
- calculate_overlap_area_in_bbox1_area_ratio,
- get_overlap_area)
-from magic_pdf.libs.commons import fitz, join_path
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-from magic_pdf.libs.local_math import float_gt
-from magic_pdf.libs.ModelBlockTypeEnum import ModelBlockTypeEnum
-from magic_pdf.libs.ocr_content_type import CategoryId, ContentType
-from magic_pdf.pre_proc.remove_bbox_overlap import _remove_overlap_between_bbox
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-
-CAPATION_OVERLAP_AREA_RATIO = 0.6
-MERGE_BOX_OVERLAP_AREA_RATIO = 1.1
-
-
-class MagicModel:
- """每个函数没有得到元素的时候返回空list."""
-
- def __fix_axis(self):
- for model_page_info in self.__model_list:
- need_remove_list = []
- page_no = model_page_info['page_info']['page_no']
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(
- model_page_info, self.__docs.get_page(page_no)
- )
- layout_dets = model_page_info['layout_dets']
- for layout_det in layout_dets:
-
- if layout_det.get('bbox') is not None:
- # 兼容直接输出bbox的模型数据,如paddle
- x0, y0, x1, y1 = layout_det['bbox']
- else:
- # 兼容直接输出poly的模型数据，如xxx
- x0, y0, _, _, x1, y1, _, _ = layout_det['poly']
-
- bbox = [
- int(x0 / horizontal_scale_ratio),
- int(y0 / vertical_scale_ratio),
- int(x1 / horizontal_scale_ratio),
- int(y1 / vertical_scale_ratio),
- ]
- layout_det['bbox'] = bbox
- # 删除高度或者宽度小于等于0的spans
- if bbox[2] - bbox[0] <= 0 or bbox[3] - bbox[1] <= 0:
- need_remove_list.append(layout_det)
- for need_remove in need_remove_list:
- layout_dets.remove(need_remove)
-
- def __fix_by_remove_low_confidence(self):
- for model_page_info in self.__model_list:
- need_remove_list = []
- layout_dets = model_page_info['layout_dets']
- for layout_det in layout_dets:
- if layout_det['score'] <= 0.05:
- need_remove_list.append(layout_det)
- else:
- continue
- for need_remove in need_remove_list:
- layout_dets.remove(need_remove)
-
- def __fix_by_remove_high_iou_and_low_confidence(self):
- for model_page_info in self.__model_list:
- need_remove_list = []
- layout_dets = model_page_info['layout_dets']
- for layout_det1 in layout_dets:
- for layout_det2 in layout_dets:
- if layout_det1 == layout_det2:
- continue
- if layout_det1['category_id'] in [
- 0,
- 1,
- 2,
- 3,
- 4,
- 5,
- 6,
- 7,
- 8,
- 9,
- ] and layout_det2['category_id'] in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]:
- if (
- calculate_iou(layout_det1['bbox'], layout_det2['bbox'])
- > 0.9
- ):
- if layout_det1['score'] < layout_det2['score']:
- layout_det_need_remove = layout_det1
- else:
- layout_det_need_remove = layout_det2
-
- if layout_det_need_remove not in need_remove_list:
- need_remove_list.append(layout_det_need_remove)
- else:
- continue
- else:
- continue
- for need_remove in need_remove_list:
- layout_dets.remove(need_remove)
-
- def __init__(self, model_list: list, docs: Dataset):
- self.__model_list = model_list
- self.__docs = docs
- """为所有模型数据添加bbox信息(缩放，poly->bbox)"""
- self.__fix_axis()
- """删除置信度特别低的模型数据(<0.05),提高质量"""
- self.__fix_by_remove_low_confidence()
- """删除高iou(>0.9)数据中置信度较低的那个"""
- self.__fix_by_remove_high_iou_and_low_confidence()
- self.__fix_footnote()
-
- def _bbox_distance(self, bbox1, bbox2):
- left, right, bottom, top = bbox_relative_pos(bbox1, bbox2)
- flags = [left, right, bottom, top]
- count = sum([1 if v else 0 for v in flags])
- if count > 1:
- return float('inf')
- if left or right:
- l1 = bbox1[3] - bbox1[1]
- l2 = bbox2[3] - bbox2[1]
- else:
- l1 = bbox1[2] - bbox1[0]
- l2 = bbox2[2] - bbox2[0]
-
- if l2 > l1 and (l2 - l1) / l1 > 0.3:
- return float('inf')
-
- return bbox_distance(bbox1, bbox2)
-
- def __fix_footnote(self):
- # 3: figure, 5: table, 7: footnote
- for model_page_info in self.__model_list:
- footnotes = []
- figures = []
- tables = []
-
- for obj in model_page_info['layout_dets']:
- if obj['category_id'] == 7:
- footnotes.append(obj)
- elif obj['category_id'] == 3:
- figures.append(obj)
- elif obj['category_id'] == 5:
- tables.append(obj)
- if len(footnotes) * len(figures) == 0:
- continue
- dis_figure_footnote = {}
- dis_table_footnote = {}
-
- for i in range(len(footnotes)):
- for j in range(len(figures)):
- pos_flag_count = sum(
- list(
- map(
- lambda x: 1 if x else 0,
- bbox_relative_pos(
- footnotes[i]['bbox'], figures[j]['bbox']
- ),
- )
- )
- )
- if pos_flag_count > 1:
- continue
- dis_figure_footnote[i] = min(
- self._bbox_distance(figures[j]['bbox'], footnotes[i]['bbox']),
- dis_figure_footnote.get(i, float('inf')),
- )
- for i in range(len(footnotes)):
- for j in range(len(tables)):
- pos_flag_count = sum(
- list(
- map(
- lambda x: 1 if x else 0,
- bbox_relative_pos(
- footnotes[i]['bbox'], tables[j]['bbox']
- ),
- )
- )
- )
- if pos_flag_count > 1:
- continue
-
- dis_table_footnote[i] = min(
- self._bbox_distance(tables[j]['bbox'], footnotes[i]['bbox']),
- dis_table_footnote.get(i, float('inf')),
- )
- for i in range(len(footnotes)):
- if i not in dis_figure_footnote:
- continue
- if dis_table_footnote.get(i, float('inf')) > dis_figure_footnote[i]:
- footnotes[i]['category_id'] = CategoryId.ImageFootnote
-
- def __reduct_overlap(self, bboxes):
- N = len(bboxes)
- keep = [True] * N
- for i in range(N):
- for j in range(N):
- if i == j:
- continue
- if _is_in(bboxes[i]['bbox'], bboxes[j]['bbox']):
- keep[i] = False
- return [bboxes[i] for i in range(N) if keep[i]]
-
- def __tie_up_category_by_distance(
- self, page_no, subject_category_id, object_category_id
- ):
- """假定每个 subject 最多有一个 object (可以有多个相邻的 object 合并为单个 object)，每个 object
- 只能属于一个 subject."""
- ret = []
- MAX_DIS_OF_POINT = 10**9 + 7
- """
- subject 和 object 的 bbox 会合并成一个大的 bbox （named: merged bbox）。
- 筛选出所有和 merged bbox 有 overlap 且 overlap 面积大于 object 的面积的 subjects。
- 再求出筛选出的 subjects 和 object 的最短距离
- """
-
- def search_overlap_between_boxes(subject_idx, object_idx):
- idxes = [subject_idx, object_idx]
- x0s = [all_bboxes[idx]['bbox'][0] for idx in idxes]
- y0s = [all_bboxes[idx]['bbox'][1] for idx in idxes]
- x1s = [all_bboxes[idx]['bbox'][2] for idx in idxes]
- y1s = [all_bboxes[idx]['bbox'][3] for idx in idxes]
-
- merged_bbox = [
- min(x0s),
- min(y0s),
- max(x1s),
- max(y1s),
- ]
- ratio = 0
-
- other_objects = list(
- map(
- lambda x: {'bbox': x['bbox'], 'score': x['score']},
- filter(
- lambda x: x['category_id']
- not in (object_category_id, subject_category_id),
- self.__model_list[page_no]['layout_dets'],
- ),
- )
- )
- for other_object in other_objects:
- ratio = max(
- ratio,
- get_overlap_area(merged_bbox, other_object['bbox'])
- * 1.0
- / box_area(all_bboxes[object_idx]['bbox']),
- )
- if ratio >= MERGE_BOX_OVERLAP_AREA_RATIO:
- break
-
- return ratio
-
- def may_find_other_nearest_bbox(subject_idx, object_idx):
- ret = float('inf')
-
- x0 = min(
- all_bboxes[subject_idx]['bbox'][0], all_bboxes[object_idx]['bbox'][0]
- )
- y0 = min(
- all_bboxes[subject_idx]['bbox'][1], all_bboxes[object_idx]['bbox'][1]
- )
- x1 = max(
- all_bboxes[subject_idx]['bbox'][2], all_bboxes[object_idx]['bbox'][2]
- )
- y1 = max(
- all_bboxes[subject_idx]['bbox'][3], all_bboxes[object_idx]['bbox'][3]
- )
-
- object_area = abs(
- all_bboxes[object_idx]['bbox'][2] - all_bboxes[object_idx]['bbox'][0]
- ) * abs(
- all_bboxes[object_idx]['bbox'][3] - all_bboxes[object_idx]['bbox'][1]
- )
-
- for i in range(len(all_bboxes)):
- if (
- i == subject_idx
- or all_bboxes[i]['category_id'] != subject_category_id
- ):
- continue
- if _is_part_overlap([x0, y0, x1, y1], all_bboxes[i]['bbox']) or _is_in(
- all_bboxes[i]['bbox'], [x0, y0, x1, y1]
- ):
-
- i_area = abs(
- all_bboxes[i]['bbox'][2] - all_bboxes[i]['bbox'][0]
- ) * abs(all_bboxes[i]['bbox'][3] - all_bboxes[i]['bbox'][1])
- if i_area >= object_area:
- ret = min(float('inf'), dis[i][object_idx])
-
- return ret
-
- def expand_bbbox(idxes):
- x0s = [all_bboxes[idx]['bbox'][0] for idx in idxes]
- y0s = [all_bboxes[idx]['bbox'][1] for idx in idxes]
- x1s = [all_bboxes[idx]['bbox'][2] for idx in idxes]
- y1s = [all_bboxes[idx]['bbox'][3] for idx in idxes]
- return min(x0s), min(y0s), max(x1s), max(y1s)
-
- subjects = self.__reduct_overlap(
- list(
- map(
- lambda x: {'bbox': x['bbox'], 'score': x['score']},
- filter(
- lambda x: x['category_id'] == subject_category_id,
- self.__model_list[page_no]['layout_dets'],
- ),
- )
- )
- )
-
- objects = self.__reduct_overlap(
- list(
- map(
- lambda x: {'bbox': x['bbox'], 'score': x['score']},
- filter(
- lambda x: x['category_id'] == object_category_id,
- self.__model_list[page_no]['layout_dets'],
- ),
- )
- )
- )
- subject_object_relation_map = {}
-
- subjects.sort(
- key=lambda x: x['bbox'][0] ** 2 + x['bbox'][1] ** 2
- ) # get the distance !
-
- all_bboxes = []
-
- for v in subjects:
- all_bboxes.append(
- {
- 'category_id': subject_category_id,
- 'bbox': v['bbox'],
- 'score': v['score'],
- }
- )
-
- for v in objects:
- all_bboxes.append(
- {
- 'category_id': object_category_id,
- 'bbox': v['bbox'],
- 'score': v['score'],
- }
- )
-
- N = len(all_bboxes)
- dis = [[MAX_DIS_OF_POINT] * N for _ in range(N)]
-
- for i in range(N):
- for j in range(i):
- if (
- all_bboxes[i]['category_id'] == subject_category_id
- and all_bboxes[j]['category_id'] == subject_category_id
- ):
- continue
-
- subject_idx, object_idx = i, j
- if all_bboxes[j]['category_id'] == subject_category_id:
- subject_idx, object_idx = j, i
-
- if (
- search_overlap_between_boxes(subject_idx, object_idx)
- >= MERGE_BOX_OVERLAP_AREA_RATIO
- ):
- dis[i][j] = float('inf')
- dis[j][i] = dis[i][j]
- continue
-
- dis[i][j] = self._bbox_distance(
- all_bboxes[subject_idx]['bbox'], all_bboxes[object_idx]['bbox']
- )
- dis[j][i] = dis[i][j]
-
- used = set()
- for i in range(N):
- # 求第 i 个 subject 所关联的 object
- if all_bboxes[i]['category_id'] != subject_category_id:
- continue
- seen = set()
- candidates = []
- arr = []
- for j in range(N):
-
- pos_flag_count = sum(
- list(
- map(
- lambda x: 1 if x else 0,
- bbox_relative_pos(
- all_bboxes[i]['bbox'], all_bboxes[j]['bbox']
- ),
- )
- )
- )
- if pos_flag_count > 1:
- continue
- if (
- all_bboxes[j]['category_id'] != object_category_id
- or j in used
- or dis[i][j] == MAX_DIS_OF_POINT
- ):
- continue
- left, right, _, _ = bbox_relative_pos(
- all_bboxes[i]['bbox'], all_bboxes[j]['bbox']
- ) # 由 pos_flag_count 相关逻辑保证本段逻辑准确性
- if left or right:
- one_way_dis = all_bboxes[i]['bbox'][2] - all_bboxes[i]['bbox'][0]
- else:
- one_way_dis = all_bboxes[i]['bbox'][3] - all_bboxes[i]['bbox'][1]
- if dis[i][j] > one_way_dis:
- continue
- arr.append((dis[i][j], j))
-
- arr.sort(key=lambda x: x[0])
- if len(arr) > 0:
- """
- bug: 离该subject 最近的 object 可能跨越了其它的 subject。
- 比如 [this subect] [some sbuject] [the nearest object of subject]
- """
- if may_find_other_nearest_bbox(i, arr[0][1]) >= arr[0][0]:
-
- candidates.append(arr[0][1])
- seen.add(arr[0][1])
-
- # 已经获取初始种子
- for j in set(candidates):
- tmp = []
- for k in range(i + 1, N):
- pos_flag_count = sum(
- list(
- map(
- lambda x: 1 if x else 0,
- bbox_relative_pos(
- all_bboxes[j]['bbox'], all_bboxes[k]['bbox']
- ),
- )
- )
- )
-
- if pos_flag_count > 1:
- continue
-
- if (
- all_bboxes[k]['category_id'] != object_category_id
- or k in used
- or k in seen
- or dis[j][k] == MAX_DIS_OF_POINT
- or dis[j][k] > dis[i][j]
- ):
- continue
-
- is_nearest = True
- for ni in range(i + 1, N):
- if ni in (j, k) or ni in used or ni in seen:
- continue
-
- if not float_gt(dis[ni][k], dis[j][k]):
- is_nearest = False
- break
-
- if is_nearest:
- nx0, ny0, nx1, ny1 = expand_bbbox(list(seen) + [k])
- n_dis = bbox_distance(
- all_bboxes[i]['bbox'], [nx0, ny0, nx1, ny1]
- )
- if float_gt(dis[i][j], n_dis):
- continue
- tmp.append(k)
- seen.add(k)
-
- candidates = tmp
- if len(candidates) == 0:
- break
-
- # 已经获取到某个 figure 下所有的最靠近的 captions，以及最靠近这些 captions 的 captions 。
- # 先扩一下 bbox，
- ox0, oy0, ox1, oy1 = expand_bbbox(list(seen) + [i])
- ix0, iy0, ix1, iy1 = all_bboxes[i]['bbox']
-
- # 分成了 4 个截取空间，需要计算落在每个截取空间下 objects 合并后占据的矩形面积
- caption_poses = [
- [ox0, oy0, ix0, oy1],
- [ox0, oy0, ox1, iy0],
- [ox0, iy1, ox1, oy1],
- [ix1, oy0, ox1, oy1],
- ]
-
- caption_areas = []
- for bbox in caption_poses:
- embed_arr = []
- for idx in seen:
- if (
- calculate_overlap_area_in_bbox1_area_ratio(
- all_bboxes[idx]['bbox'], bbox
- )
- > CAPATION_OVERLAP_AREA_RATIO
- ):
- embed_arr.append(idx)
-
- if len(embed_arr) > 0:
- embed_x0 = min([all_bboxes[idx]['bbox'][0] for idx in embed_arr])
- embed_y0 = min([all_bboxes[idx]['bbox'][1] for idx in embed_arr])
- embed_x1 = max([all_bboxes[idx]['bbox'][2] for idx in embed_arr])
- embed_y1 = max([all_bboxes[idx]['bbox'][3] for idx in embed_arr])
- caption_areas.append(
- int(abs(embed_x1 - embed_x0) * abs(embed_y1 - embed_y0))
- )
- else:
- caption_areas.append(0)
-
- subject_object_relation_map[i] = []
- if max(caption_areas) > 0:
- max_area_idx = caption_areas.index(max(caption_areas))
- caption_bbox = caption_poses[max_area_idx]
-
- for j in seen:
- if (
- calculate_overlap_area_in_bbox1_area_ratio(
- all_bboxes[j]['bbox'], caption_bbox
- )
- > CAPATION_OVERLAP_AREA_RATIO
- ):
- used.add(j)
- subject_object_relation_map[i].append(j)
-
- for i in sorted(subject_object_relation_map.keys()):
- result = {
- 'subject_body': all_bboxes[i]['bbox'],
- 'all': all_bboxes[i]['bbox'],
- 'score': all_bboxes[i]['score'],
- }
-
- if len(subject_object_relation_map[i]) > 0:
- x0 = min(
- [all_bboxes[j]['bbox'][0] for j in subject_object_relation_map[i]]
- )
- y0 = min(
- [all_bboxes[j]['bbox'][1] for j in subject_object_relation_map[i]]
- )
- x1 = max(
- [all_bboxes[j]['bbox'][2] for j in subject_object_relation_map[i]]
- )
- y1 = max(
- [all_bboxes[j]['bbox'][3] for j in subject_object_relation_map[i]]
- )
- result['object_body'] = [x0, y0, x1, y1]
- result['all'] = [
- min(x0, all_bboxes[i]['bbox'][0]),
- min(y0, all_bboxes[i]['bbox'][1]),
- max(x1, all_bboxes[i]['bbox'][2]),
- max(y1, all_bboxes[i]['bbox'][3]),
- ]
- ret.append(result)
-
- total_subject_object_dis = 0
- # 计算已经配对的 distance 距离
- for i in subject_object_relation_map.keys():
- for j in subject_object_relation_map[i]:
- total_subject_object_dis += bbox_distance(
- all_bboxes[i]['bbox'], all_bboxes[j]['bbox']
- )
-
- # 计算未匹配的 subject 和 object 的距离（非精确版）
- with_caption_subject = set(
- [
- key
- for key in subject_object_relation_map.keys()
- if len(subject_object_relation_map[i]) > 0
- ]
- )
- for i in range(N):
- if all_bboxes[i]['category_id'] != object_category_id or i in used:
- continue
- candidates = []
- for j in range(N):
- if (
- all_bboxes[j]['category_id'] != subject_category_id
- or j in with_caption_subject
- ):
- continue
- candidates.append((dis[i][j], j))
- if len(candidates) > 0:
- candidates.sort(key=lambda x: x[0])
- total_subject_object_dis += candidates[0][1]
- with_caption_subject.add(j)
- return ret, total_subject_object_dis
-
- def __tie_up_category_by_distance_v2(
- self, page_no, subject_category_id, object_category_id
- ):
-
- AXIS_MULPLICITY = 0.5
- subjects = self.__reduct_overlap(
- list(
- map(
- lambda x: {'bbox': x['bbox'], 'score': x['score']},
- filter(
- lambda x: x['category_id'] == subject_category_id,
- self.__model_list[page_no]['layout_dets'],
- ),
- )
- )
- )
-
- objects = self.__reduct_overlap(
- list(
- map(
- lambda x: {'bbox': x['bbox'], 'score': x['score']},
- filter(
- lambda x: x['category_id'] == object_category_id,
- self.__model_list[page_no]['layout_dets'],
- ),
- )
- )
- )
- M = len(objects)
-
- subjects.sort(key=lambda x: x['bbox'][0] ** 2 + x['bbox'][1] ** 2)
- objects.sort(key=lambda x: x['bbox'][0] ** 2 + x['bbox'][1] ** 2)
-
- sub_obj_map_h = {i: [] for i in range(len(subjects))}
-
- dis_by_directions = {
- 'top': [[-1, float('inf')]] * M,
- 'bottom': [[-1, float('inf')]] * M,
- 'left': [[-1, float('inf')]] * M,
- 'right': [[-1, float('inf')]] * M,
- }
-
- for i, obj in enumerate(objects):
- l_x_axis, l_y_axis = (
- obj['bbox'][2] - obj['bbox'][0],
- obj['bbox'][3] - obj['bbox'][1],
- )
- axis_unit = min(l_x_axis, l_y_axis)
- for j, sub in enumerate(subjects):
-
- bbox1, bbox2, _ = _remove_overlap_between_bbox(
- objects[i]['bbox'], subjects[j]['bbox']
- )
- left, right, bottom, top = bbox_relative_pos(bbox1, bbox2)
- flags = [left, right, bottom, top]
- if sum([1 if v else 0 for v in flags]) > 1:
- continue
-
- if left:
- if dis_by_directions['left'][i][1] > bbox_distance(
- obj['bbox'], sub['bbox']
- ):
- dis_by_directions['left'][i] = [
- j,
- bbox_distance(obj['bbox'], sub['bbox']),
- ]
- if right:
- if dis_by_directions['right'][i][1] > bbox_distance(
- obj['bbox'], sub['bbox']
- ):
- dis_by_directions['right'][i] = [
- j,
- bbox_distance(obj['bbox'], sub['bbox']),
- ]
- if bottom:
- if dis_by_directions['bottom'][i][1] > bbox_distance(
- obj['bbox'], sub['bbox']
- ):
- dis_by_directions['bottom'][i] = [
- j,
- bbox_distance(obj['bbox'], sub['bbox']),
- ]
- if top:
- if dis_by_directions['top'][i][1] > bbox_distance(
- obj['bbox'], sub['bbox']
- ):
- dis_by_directions['top'][i] = [
- j,
- bbox_distance(obj['bbox'], sub['bbox']),
- ]
- if dis_by_directions['left'][i][1] != float('inf') or dis_by_directions[
- 'right'
- ][i][1] != float('inf'):
- if dis_by_directions['left'][i][1] != float(
- 'inf'
- ) and dis_by_directions['right'][i][1] != float('inf'):
- if AXIS_MULPLICITY * axis_unit >= abs(
- dis_by_directions['left'][i][1]
- - dis_by_directions['right'][i][1]
- ):
- left_sub_bbox = subjects[dis_by_directions['left'][i][0]][
- 'bbox'
- ]
- right_sub_bbox = subjects[dis_by_directions['right'][i][0]][
- 'bbox'
- ]
-
- left_sub_bbox_y_axis = left_sub_bbox[3] - left_sub_bbox[1]
- right_sub_bbox_y_axis = right_sub_bbox[3] - right_sub_bbox[1]
-
- if (
- abs(left_sub_bbox_y_axis - l_y_axis)
- + dis_by_directions['left'][i][0]
- > abs(right_sub_bbox_y_axis - l_y_axis)
- + dis_by_directions['right'][i][0]
- ):
- left_or_right = dis_by_directions['right'][i]
- else:
- left_or_right = dis_by_directions['left'][i]
- else:
- left_or_right = dis_by_directions['left'][i]
- if left_or_right[1] > dis_by_directions['right'][i][1]:
- left_or_right = dis_by_directions['right'][i]
- else:
- left_or_right = dis_by_directions['left'][i]
- if left_or_right[1] == float('inf'):
- left_or_right = dis_by_directions['right'][i]
- else:
- left_or_right = [-1, float('inf')]
-
- if dis_by_directions['top'][i][1] != float('inf') or dis_by_directions[
- 'bottom'
- ][i][1] != float('inf'):
- if dis_by_directions['top'][i][1] != float('inf') and dis_by_directions[
- 'bottom'
- ][i][1] != float('inf'):
- if AXIS_MULPLICITY * axis_unit >= abs(
- dis_by_directions['top'][i][1]
- - dis_by_directions['bottom'][i][1]
- ):
- top_bottom = subjects[dis_by_directions['bottom'][i][0]]['bbox']
- bottom_top = subjects[dis_by_directions['top'][i][0]]['bbox']
-
- top_bottom_x_axis = top_bottom[2] - top_bottom[0]
- bottom_top_x_axis = bottom_top[2] - bottom_top[0]
- if abs(top_bottom_x_axis - l_x_axis) + dis_by_directions['bottom'][i][1] > abs(
- bottom_top_x_axis - l_x_axis
- ) + dis_by_directions['top'][i][1]:
- top_or_bottom = dis_by_directions['top'][i]
- else:
- top_or_bottom = dis_by_directions['bottom'][i]
- else:
- top_or_bottom = dis_by_directions['top'][i]
- if top_or_bottom[1] > dis_by_directions['bottom'][i][1]:
- top_or_bottom = dis_by_directions['bottom'][i]
- else:
- top_or_bottom = dis_by_directions['top'][i]
- if top_or_bottom[1] == float('inf'):
- top_or_bottom = dis_by_directions['bottom'][i]
- else:
- top_or_bottom = [-1, float('inf')]
-
- if left_or_right[1] != float('inf') or top_or_bottom[1] != float('inf'):
- if left_or_right[1] != float('inf') and top_or_bottom[1] != float(
- 'inf'
- ):
- if AXIS_MULPLICITY * axis_unit >= abs(
- left_or_right[1] - top_or_bottom[1]
- ):
- y_axis_bbox = subjects[left_or_right[0]]['bbox']
- x_axis_bbox = subjects[top_or_bottom[0]]['bbox']
-
- if (
- abs((x_axis_bbox[2] - x_axis_bbox[0]) - l_x_axis) / l_x_axis
- > abs((y_axis_bbox[3] - y_axis_bbox[1]) - l_y_axis)
- / l_y_axis
- ):
- sub_obj_map_h[left_or_right[0]].append(i)
- else:
- sub_obj_map_h[top_or_bottom[0]].append(i)
- else:
- if left_or_right[1] > top_or_bottom[1]:
- sub_obj_map_h[top_or_bottom[0]].append(i)
- else:
- sub_obj_map_h[left_or_right[0]].append(i)
- else:
- if left_or_right[1] != float('inf'):
- sub_obj_map_h[left_or_right[0]].append(i)
- else:
- sub_obj_map_h[top_or_bottom[0]].append(i)
- ret = []
- for i in sub_obj_map_h.keys():
- ret.append(
- {
- 'sub_bbox': {
- 'bbox': subjects[i]['bbox'],
- 'score': subjects[i]['score'],
- },
- 'obj_bboxes': [
- {'score': objects[j]['score'], 'bbox': objects[j]['bbox']}
- for j in sub_obj_map_h[i]
- ],
- 'sub_idx': i,
- }
- )
- return ret
-
- def get_imgs_v2(self, page_no: int):
- with_captions = self.__tie_up_category_by_distance_v2(page_no, 3, 4)
- with_footnotes = self.__tie_up_category_by_distance_v2(
- page_no, 3, CategoryId.ImageFootnote
- )
- ret = []
- for v in with_captions:
- record = {
- 'image_body': v['sub_bbox'],
- 'image_caption_list': v['obj_bboxes'],
- }
- filter_idx = v['sub_idx']
- d = next(filter(lambda x: x['sub_idx'] == filter_idx, with_footnotes))
- record['image_footnote_list'] = d['obj_bboxes']
- ret.append(record)
- return ret
-
- def get_tables_v2(self, page_no: int) -> list:
- with_captions = self.__tie_up_category_by_distance_v2(page_no, 5, 6)
- with_footnotes = self.__tie_up_category_by_distance_v2(page_no, 5, 7)
- ret = []
- for v in with_captions:
- record = {
- 'table_body': v['sub_bbox'],
- 'table_caption_list': v['obj_bboxes'],
- }
- filter_idx = v['sub_idx']
- d = next(filter(lambda x: x['sub_idx'] == filter_idx, with_footnotes))
- record['table_footnote_list'] = d['obj_bboxes']
- ret.append(record)
- return ret
-
- def get_imgs(self, page_no: int):
- with_captions, _ = self.__tie_up_category_by_distance(page_no, 3, 4)
- with_footnotes, _ = self.__tie_up_category_by_distance(
- page_no, 3, CategoryId.ImageFootnote
- )
- ret = []
- N, M = len(with_captions), len(with_footnotes)
- assert N == M
- for i in range(N):
- record = {
- 'score': with_captions[i]['score'],
- 'img_caption_bbox': with_captions[i].get('object_body', None),
- 'img_body_bbox': with_captions[i]['subject_body'],
- 'img_footnote_bbox': with_footnotes[i].get('object_body', None),
- }
-
- x0 = min(with_captions[i]['all'][0], with_footnotes[i]['all'][0])
- y0 = min(with_captions[i]['all'][1], with_footnotes[i]['all'][1])
- x1 = max(with_captions[i]['all'][2], with_footnotes[i]['all'][2])
- y1 = max(with_captions[i]['all'][3], with_footnotes[i]['all'][3])
- record['bbox'] = [x0, y0, x1, y1]
- ret.append(record)
- return ret
-
- def get_tables(
- self, page_no: int
- ) -> list: # 3个坐标， caption, table主体，table-note
- with_captions, _ = self.__tie_up_category_by_distance(page_no, 5, 6)
- with_footnotes, _ = self.__tie_up_category_by_distance(page_no, 5, 7)
- ret = []
- N, M = len(with_captions), len(with_footnotes)
- assert N == M
- for i in range(N):
- record = {
- 'score': with_captions[i]['score'],
- 'table_caption_bbox': with_captions[i].get('object_body', None),
- 'table_body_bbox': with_captions[i]['subject_body'],
- 'table_footnote_bbox': with_footnotes[i].get('object_body', None),
- }
-
- x0 = min(with_captions[i]['all'][0], with_footnotes[i]['all'][0])
- y0 = min(with_captions[i]['all'][1], with_footnotes[i]['all'][1])
- x1 = max(with_captions[i]['all'][2], with_footnotes[i]['all'][2])
- y1 = max(with_captions[i]['all'][3], with_footnotes[i]['all'][3])
- record['bbox'] = [x0, y0, x1, y1]
- ret.append(record)
- return ret
-
- def get_equations(self, page_no: int) -> list: # 有坐标，也有字
- inline_equations = self.__get_blocks_by_type(
- ModelBlockTypeEnum.EMBEDDING.value, page_no, ['latex']
- )
- interline_equations = self.__get_blocks_by_type(
- ModelBlockTypeEnum.ISOLATED.value, page_no, ['latex']
- )
- interline_equations_blocks = self.__get_blocks_by_type(
- ModelBlockTypeEnum.ISOLATE_FORMULA.value, page_no
- )
- return inline_equations, interline_equations, interline_equations_blocks
-
- def get_discarded(self, page_no: int) -> list: # 自研模型，只有坐标
- blocks = self.__get_blocks_by_type(ModelBlockTypeEnum.ABANDON.value, page_no)
- return blocks
-
- def get_text_blocks(self, page_no: int) -> list: # 自研模型搞的，只有坐标，没有字
- blocks = self.__get_blocks_by_type(ModelBlockTypeEnum.PLAIN_TEXT.value, page_no)
- return blocks
-
- def get_title_blocks(self, page_no: int) -> list: # 自研模型，只有坐标，没字
- blocks = self.__get_blocks_by_type(ModelBlockTypeEnum.TITLE.value, page_no)
- return blocks
-
- def get_ocr_text(self, page_no: int) -> list: # paddle 搞的，有字也有坐标
- text_spans = []
- model_page_info = self.__model_list[page_no]
- layout_dets = model_page_info['layout_dets']
- for layout_det in layout_dets:
- if layout_det['category_id'] == '15':
- span = {
- 'bbox': layout_det['bbox'],
- 'content': layout_det['text'],
- }
- text_spans.append(span)
- return text_spans
-
- def get_all_spans(self, page_no: int) -> list:
-
- def remove_duplicate_spans(spans):
- new_spans = []
- for span in spans:
- if not any(span == existing_span for existing_span in new_spans):
- new_spans.append(span)
- return new_spans
-
- all_spans = []
- model_page_info = self.__model_list[page_no]
- layout_dets = model_page_info['layout_dets']
- allow_category_id_list = [3, 5, 13, 14, 15]
- """当成span拼接的"""
- # 3: 'image', # 图片
- # 5: 'table', # 表格
- # 13: 'inline_equation', # 行内公式
- # 14: 'interline_equation', # 行间公式
- # 15: 'text', # ocr识别文本
- for layout_det in layout_dets:
- category_id = layout_det['category_id']
- if category_id in allow_category_id_list:
- span = {'bbox': layout_det['bbox'], 'score': layout_det['score']}
- if category_id == 3:
- span['type'] = ContentType.Image
- elif category_id == 5:
- # 获取table模型结果
- latex = layout_det.get('latex', None)
- html = layout_det.get('html', None)
- if latex:
- span['latex'] = latex
- elif html:
- span['html'] = html
- span['type'] = ContentType.Table
- elif category_id == 13:
- span['content'] = layout_det['latex']
- span['type'] = ContentType.InlineEquation
- elif category_id == 14:
- span['content'] = layout_det['latex']
- span['type'] = ContentType.InterlineEquation
- elif category_id == 15:
- span['content'] = layout_det['text']
- span['type'] = ContentType.Text
- all_spans.append(span)
- return remove_duplicate_spans(all_spans)
-
- def get_page_size(self, page_no: int): # 获取页面宽高
- # 获取当前页的page对象
- page = self.__docs.get_page(page_no).get_page_info()
- # 获取当前页的宽高
- page_w = page.w
- page_h = page.h
- return page_w, page_h
-
- def __get_blocks_by_type(
- self, type: int, page_no: int, extra_col: list[str] = []
- ) -> list:
- blocks = []
- for page_dict in self.__model_list:
- layout_dets = page_dict.get('layout_dets', [])
- page_info = page_dict.get('page_info', {})
- page_number = page_info.get('page_no', -1)
- if page_no != page_number:
- continue
- for item in layout_dets:
- category_id = item.get('category_id', -1)
- bbox = item.get('bbox', None)
-
- if category_id == type:
- block = {
- 'bbox': bbox,
- 'score': item.get('score'),
- }
- for col in extra_col:
- block[col] = item.get(col, None)
- blocks.append(block)
- return blocks
-
- def get_model_list(self, page_no):
- return self.__model_list[page_no]
-
-
-if __name__ == '__main__':
- drw = DiskReaderWriter(r'D:/project/20231108code-clean')
- if 0:
- pdf_file_path = r'linshixuqiu\19983-00.pdf'
- model_file_path = r'linshixuqiu\19983-00_new.json'
- pdf_bytes = drw.read(pdf_file_path, AbsReaderWriter.MODE_BIN)
- model_json_txt = drw.read(model_file_path, AbsReaderWriter.MODE_TXT)
- model_list = json.loads(model_json_txt)
- write_path = r'D:\project\20231108code-clean\linshixuqiu\19983-00'
- img_bucket_path = 'imgs'
- img_writer = DiskReaderWriter(join_path(write_path, img_bucket_path))
- pdf_docs = fitz.open('pdf', pdf_bytes)
- magic_model = MagicModel(model_list, pdf_docs)
-
- if 1:
- model_list = json.loads(
- drw.read('/opt/data/pdf/20240418/j.chroma.2009.03.042.json')
- )
- pdf_bytes = drw.read(
- '/opt/data/pdf/20240418/j.chroma.2009.03.042.pdf', AbsReaderWriter.MODE_BIN
- )
- pdf_docs = fitz.open('pdf', pdf_bytes)
- magic_model = MagicModel(model_list, pdf_docs)
- for i in range(7):
- print(magic_model.get_imgs(i))
diff --git a/magic_pdf/model/model_list.py b/magic_pdf/model/model_list.py
deleted file mode 100644
index c4bcf0217..000000000
--- a/magic_pdf/model/model_list.py
+++ /dev/null
@@ -1,11 +0,0 @@
-class MODEL:
- Paddle = "pp_structure_v2"
- PEK = "pdf_extract_kit"
-
-
-class AtomicModel:
- Layout = "layout"
- MFD = "mfd"
- MFR = "mfr"
- OCR = "ocr"
- Table = "table"
diff --git a/magic_pdf/model/pdf_extract_kit.py b/magic_pdf/model/pdf_extract_kit.py
deleted file mode 100644
index f1478b10b..000000000
--- a/magic_pdf/model/pdf_extract_kit.py
+++ /dev/null
@@ -1,488 +0,0 @@
-from loguru import logger
-import os
-import time
-from pathlib import Path
-import shutil
-from magic_pdf.libs.Constants import *
-from magic_pdf.libs.clean_memory import clean_memory
-from magic_pdf.model.model_list import AtomicModel
-
-os.environ['NO_ALBUMENTATIONS_UPDATE'] = '1' # 禁止albumentations检查更新
-os.environ['YOLO_VERBOSE'] = 'False' # disable yolo logger
-try:
- import cv2
- import yaml
- import argparse
- import numpy as np
- import torch
- import torchtext
-
- if torchtext.__version__ >= "0.18.0":
- torchtext.disable_torchtext_deprecation_warning()
- from PIL import Image
- from torchvision import transforms
- from torch.utils.data import Dataset, DataLoader
- from ultralytics import YOLO
- from unimernet.common.config import Config
- import unimernet.tasks as tasks
- from unimernet.processors import load_processor
- from doclayout_yolo import YOLOv10
-
-except ImportError as e:
- logger.exception(e)
- logger.error(
- 'Required dependency not installed, please install by \n'
- '"pip install magic-pdf[full] --extra-index-url https://myhloli.github.io/wheels/"')
- exit(1)
-
-from magic_pdf.model.pek_sub_modules.layoutlmv3.model_init import Layoutlmv3_Predictor
-from magic_pdf.model.pek_sub_modules.post_process import latex_rm_whitespace
-from magic_pdf.model.pek_sub_modules.self_modify import ModifiedPaddleOCR
-from magic_pdf.model.pek_sub_modules.structeqtable.StructTableModel import StructTableModel
-from magic_pdf.model.ppTableModel import ppTableModel
-
-
-def table_model_init(table_model_type, model_path, max_time, _device_='cpu'):
- if table_model_type == MODEL_NAME.STRUCT_EQTABLE:
- table_model = StructTableModel(model_path, max_time=max_time, device=_device_)
- else:
- config = {
- "model_dir": model_path,
- "device": _device_
- }
- table_model = ppTableModel(config)
- return table_model
-
-
-def mfd_model_init(weight):
- mfd_model = YOLO(weight)
- return mfd_model
-
-
-def mfr_model_init(weight_dir, cfg_path, _device_='cpu'):
- args = argparse.Namespace(cfg_path=cfg_path, options=None)
- cfg = Config(args)
- cfg.config.model.pretrained = os.path.join(weight_dir, "pytorch_model.pth")
- cfg.config.model.model_config.model_name = weight_dir
- cfg.config.model.tokenizer_config.path = weight_dir
- task = tasks.setup_task(cfg)
- model = task.build_model(cfg)
- model.to(_device_)
- model.eval()
- vis_processor = load_processor('formula_image_eval', cfg.config.datasets.formula_rec_eval.vis_processor.eval)
- mfr_transform = transforms.Compose([vis_processor, ])
- return [model, mfr_transform]
-
-
-def layout_model_init(weight, config_file, device):
- model = Layoutlmv3_Predictor(weight, config_file, device)
- return model
-
-
-def doclayout_yolo_model_init(weight):
- model = YOLOv10(weight)
- return model
-
-
-def ocr_model_init(show_log: bool = False, det_db_box_thresh=0.3, lang=None, use_dilation=True, det_db_unclip_ratio=1.8):
- if lang is not None:
- model = ModifiedPaddleOCR(show_log=show_log, det_db_box_thresh=det_db_box_thresh, lang=lang, use_dilation=use_dilation, det_db_unclip_ratio=det_db_unclip_ratio)
- else:
- model = ModifiedPaddleOCR(show_log=show_log, det_db_box_thresh=det_db_box_thresh, use_dilation=use_dilation, det_db_unclip_ratio=det_db_unclip_ratio)
- return model
-
-
-class MathDataset(Dataset):
- def __init__(self, image_paths, transform=None):
- self.image_paths = image_paths
- self.transform = transform
-
- def __len__(self):
- return len(self.image_paths)
-
- def __getitem__(self, idx):
- # if not pil image, then convert to pil image
- if isinstance(self.image_paths[idx], str):
- raw_image = Image.open(self.image_paths[idx])
- else:
- raw_image = self.image_paths[idx]
- if self.transform:
- image = self.transform(raw_image)
- return image
-
-
-class AtomModelSingleton:
- _instance = None
- _models = {}
-
- def __new__(cls, *args, **kwargs):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
- return cls._instance
-
- def get_atom_model(self, atom_model_name: str, **kwargs):
- lang = kwargs.get("lang", None)
- layout_model_name = kwargs.get("layout_model_name", None)
- key = (atom_model_name, layout_model_name, lang)
- if key not in self._models:
- self._models[key] = atom_model_init(model_name=atom_model_name, **kwargs)
- return self._models[key]
-
-
-def atom_model_init(model_name: str, **kwargs):
-
- if model_name == AtomicModel.Layout:
- if kwargs.get("layout_model_name") == MODEL_NAME.LAYOUTLMv3:
- atom_model = layout_model_init(
- kwargs.get("layout_weights"),
- kwargs.get("layout_config_file"),
- kwargs.get("device")
- )
- elif kwargs.get("layout_model_name") == MODEL_NAME.DocLayout_YOLO:
- atom_model = doclayout_yolo_model_init(
- kwargs.get("doclayout_yolo_weights"),
- )
- elif model_name == AtomicModel.MFD:
- atom_model = mfd_model_init(
- kwargs.get("mfd_weights")
- )
- elif model_name == AtomicModel.MFR:
- atom_model = mfr_model_init(
- kwargs.get("mfr_weight_dir"),
- kwargs.get("mfr_cfg_path"),
- kwargs.get("device")
- )
- elif model_name == AtomicModel.OCR:
- atom_model = ocr_model_init(
- kwargs.get("ocr_show_log"),
- kwargs.get("det_db_box_thresh"),
- kwargs.get("lang")
- )
- elif model_name == AtomicModel.Table:
- atom_model = table_model_init(
- kwargs.get("table_model_name"),
- kwargs.get("table_model_path"),
- kwargs.get("table_max_time"),
- kwargs.get("device")
- )
- else:
- logger.error("model name not allow")
- exit(1)
-
- return atom_model
-
-
-# Unified crop img logic
-def crop_img(input_res, input_pil_img, crop_paste_x=0, crop_paste_y=0):
- crop_xmin, crop_ymin = int(input_res['poly'][0]), int(input_res['poly'][1])
- crop_xmax, crop_ymax = int(input_res['poly'][4]), int(input_res['poly'][5])
- # Create a white background with an additional width and height of 50
- crop_new_width = crop_xmax - crop_xmin + crop_paste_x * 2
- crop_new_height = crop_ymax - crop_ymin + crop_paste_y * 2
- return_image = Image.new('RGB', (crop_new_width, crop_new_height), 'white')
-
- # Crop image
- crop_box = (crop_xmin, crop_ymin, crop_xmax, crop_ymax)
- cropped_img = input_pil_img.crop(crop_box)
- return_image.paste(cropped_img, (crop_paste_x, crop_paste_y))
- return_list = [crop_paste_x, crop_paste_y, crop_xmin, crop_ymin, crop_xmax, crop_ymax, crop_new_width, crop_new_height]
- return return_image, return_list
-
-
-class CustomPEKModel:
-
- def __init__(self, ocr: bool = False, show_log: bool = False, **kwargs):
- """
- ======== model init ========
- """
- # 获取当前文件（即 pdf_extract_kit.py）的绝对路径
- current_file_path = os.path.abspath(__file__)
- # 获取当前文件所在的目录(model)
- current_dir = os.path.dirname(current_file_path)
- # 上一级目录(magic_pdf)
- root_dir = os.path.dirname(current_dir)
- # model_config目录
- model_config_dir = os.path.join(root_dir, 'resources', 'model_config')
- # 构建 model_configs.yaml 文件的完整路径
- config_path = os.path.join(model_config_dir, 'model_configs.yaml')
- with open(config_path, "r", encoding='utf-8') as f:
- self.configs = yaml.load(f, Loader=yaml.FullLoader)
- # 初始化解析配置
-
- # layout config
- self.layout_config = kwargs.get("layout_config")
- self.layout_model_name = self.layout_config.get("model", MODEL_NAME.DocLayout_YOLO)
-
- # formula config
- self.formula_config = kwargs.get("formula_config")
- self.mfd_model_name = self.formula_config.get("mfd_model", MODEL_NAME.YOLO_V8_MFD)
- self.mfr_model_name = self.formula_config.get("mfr_model", MODEL_NAME.UniMerNet_v2_Small)
- self.apply_formula = self.formula_config.get("enable", True)
-
- # table config
- self.table_config = kwargs.get("table_config")
- self.apply_table = self.table_config.get("enable", False)
- self.table_max_time = self.table_config.get("max_time", TABLE_MAX_TIME_VALUE)
- self.table_model_name = self.table_config.get("model", MODEL_NAME.TABLE_MASTER)
-
- # ocr config
- self.apply_ocr = ocr
- self.lang = kwargs.get("lang", None)
-
- logger.info(
- "DocAnalysis init, this may take some times, layout_model: {}, apply_formula: {}, apply_ocr: {}, "
- "apply_table: {}, table_model: {}, lang: {}".format(
- self.layout_model_name, self.apply_formula, self.apply_ocr, self.apply_table, self.table_model_name, self.lang
- )
- )
- # 初始化解析方案
- self.device = kwargs.get("device", "cpu")
- logger.info("using device: {}".format(self.device))
- models_dir = kwargs.get("models_dir", os.path.join(root_dir, "resources", "models"))
- logger.info("using models_dir: {}".format(models_dir))
-
- atom_model_manager = AtomModelSingleton()
-
- # 初始化公式识别
- if self.apply_formula:
-
- # 初始化公式检测模型
- self.mfd_model = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.MFD,
- mfd_weights=str(os.path.join(models_dir, self.configs["weights"][self.mfd_model_name]))
- )
-
- # 初始化公式解析模型
- mfr_weight_dir = str(os.path.join(models_dir, self.configs["weights"][self.mfr_model_name]))
- mfr_cfg_path = str(os.path.join(model_config_dir, "UniMERNet", "demo.yaml"))
- self.mfr_model, self.mfr_transform = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.MFR,
- mfr_weight_dir=mfr_weight_dir,
- mfr_cfg_path=mfr_cfg_path,
- device=self.device
- )
-
- # 初始化layout模型
- if self.layout_model_name == MODEL_NAME.LAYOUTLMv3:
- self.layout_model = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.Layout,
- layout_model_name=MODEL_NAME.LAYOUTLMv3,
- layout_weights=str(os.path.join(models_dir, self.configs['weights'][self.layout_model_name])),
- layout_config_file=str(os.path.join(model_config_dir, "layoutlmv3", "layoutlmv3_base_inference.yaml")),
- device=self.device
- )
- elif self.layout_model_name == MODEL_NAME.DocLayout_YOLO:
- self.layout_model = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.Layout,
- layout_model_name=MODEL_NAME.DocLayout_YOLO,
- doclayout_yolo_weights=str(os.path.join(models_dir, self.configs['weights'][self.layout_model_name]))
- )
- # 初始化ocr
- if self.apply_ocr:
-
- # self.ocr_model = ModifiedPaddleOCR(show_log=show_log, det_db_box_thresh=0.3)
- self.ocr_model = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.OCR,
- ocr_show_log=show_log,
- det_db_box_thresh=0.3,
- lang=self.lang
- )
- # init table model
- if self.apply_table:
- table_model_dir = self.configs["weights"][self.table_model_name]
- self.table_model = atom_model_manager.get_atom_model(
- atom_model_name=AtomicModel.Table,
- table_model_name=self.table_model_name,
- table_model_path=str(os.path.join(models_dir, table_model_dir)),
- table_max_time=self.table_max_time,
- device=self.device
- )
-
- home_directory = Path.home()
- det_source = os.path.join(models_dir, table_model_dir, DETECT_MODEL_DIR)
- rec_source = os.path.join(models_dir, table_model_dir, REC_MODEL_DIR)
- det_dest_dir = os.path.join(home_directory, PP_DET_DIRECTORY)
- rec_dest_dir = os.path.join(home_directory, PP_REC_DIRECTORY)
-
- if not os.path.exists(det_dest_dir):
- shutil.copytree(det_source, det_dest_dir)
- if not os.path.exists(rec_dest_dir):
- shutil.copytree(rec_source, rec_dest_dir)
-
- logger.info('DocAnalysis init done!')
-
- def __call__(self, image):
-
- page_start = time.time()
-
- latex_filling_list = []
- mf_image_list = []
-
- # layout检测
- layout_start = time.time()
- if self.layout_model_name == MODEL_NAME.LAYOUTLMv3:
- # layoutlmv3
- layout_res = self.layout_model(image, ignore_catids=[])
- elif self.layout_model_name == MODEL_NAME.DocLayout_YOLO:
- # doclayout_yolo
- layout_res = []
- doclayout_yolo_res = self.layout_model.predict(image, imgsz=1024, conf=0.25, iou=0.45, verbose=True, device=self.device)[0]
- for xyxy, conf, cla in zip(doclayout_yolo_res.boxes.xyxy.cpu(), doclayout_yolo_res.boxes.conf.cpu(), doclayout_yolo_res.boxes.cls.cpu()):
- xmin, ymin, xmax, ymax = [int(p.item()) for p in xyxy]
- new_item = {
- 'category_id': int(cla.item()),
- 'poly': [xmin, ymin, xmax, ymin, xmax, ymax, xmin, ymax],
- 'score': round(float(conf.item()), 3),
- }
- layout_res.append(new_item)
- layout_cost = round(time.time() - layout_start, 2)
- logger.info(f"layout detection time: {layout_cost}")
-
- pil_img = Image.fromarray(image)
-
- if self.apply_formula:
- # 公式检测
- mfd_start = time.time()
- mfd_res = self.mfd_model.predict(image, imgsz=1888, conf=0.25, iou=0.45, verbose=True, device=self.device)[0]
- logger.info(f"mfd time: {round(time.time() - mfd_start, 2)}")
- for xyxy, conf, cla in zip(mfd_res.boxes.xyxy.cpu(), mfd_res.boxes.conf.cpu(), mfd_res.boxes.cls.cpu()):
- xmin, ymin, xmax, ymax = [int(p.item()) for p in xyxy]
- new_item = {
- 'category_id': 13 + int(cla.item()),
- 'poly': [xmin, ymin, xmax, ymin, xmax, ymax, xmin, ymax],
- 'score': round(float(conf.item()), 2),
- 'latex': '',
- }
- layout_res.append(new_item)
- latex_filling_list.append(new_item)
- bbox_img = pil_img.crop((xmin, ymin, xmax, ymax))
- mf_image_list.append(bbox_img)
-
- # 公式识别
- mfr_start = time.time()
- dataset = MathDataset(mf_image_list, transform=self.mfr_transform)
- dataloader = DataLoader(dataset, batch_size=64, num_workers=0)
- mfr_res = []
- for mf_img in dataloader:
- mf_img = mf_img.to(self.device)
- with torch.no_grad():
- output = self.mfr_model.generate({'image': mf_img})
- mfr_res.extend(output['pred_str'])
- for res, latex in zip(latex_filling_list, mfr_res):
- res['latex'] = latex_rm_whitespace(latex)
- mfr_cost = round(time.time() - mfr_start, 2)
- logger.info(f"formula nums: {len(mf_image_list)}, mfr time: {mfr_cost}")
-
- # Select regions for OCR / formula regions / table regions
- ocr_res_list = []
- table_res_list = []
- single_page_mfdetrec_res = []
- for res in layout_res:
- if int(res['category_id']) in [13, 14]:
- single_page_mfdetrec_res.append({
- "bbox": [int(res['poly'][0]), int(res['poly'][1]),
- int(res['poly'][4]), int(res['poly'][5])],
- })
- elif int(res['category_id']) in [0, 1, 2, 4, 6, 7]:
- ocr_res_list.append(res)
- elif int(res['category_id']) in [5]:
- table_res_list.append(res)
-
- if torch.cuda.is_available():
- properties = torch.cuda.get_device_properties(self.device)
- total_memory = properties.total_memory / (1024 ** 3) # 将字节转换为 GB
- if total_memory <= 10:
- gc_start = time.time()
- clean_memory()
- gc_time = round(time.time() - gc_start, 2)
- logger.info(f"gc time: {gc_time}")
-
- # ocr识别
- if self.apply_ocr:
- ocr_start = time.time()
- # Process each area that requires OCR processing
- for res in ocr_res_list:
- new_image, useful_list = crop_img(res, pil_img, crop_paste_x=50, crop_paste_y=50)
- paste_x, paste_y, xmin, ymin, xmax, ymax, new_width, new_height = useful_list
- # Adjust the coordinates of the formula area
- adjusted_mfdetrec_res = []
- for mf_res in single_page_mfdetrec_res:
- mf_xmin, mf_ymin, mf_xmax, mf_ymax = mf_res["bbox"]
- # Adjust the coordinates of the formula area to the coordinates relative to the cropping area
- x0 = mf_xmin - xmin + paste_x
- y0 = mf_ymin - ymin + paste_y
- x1 = mf_xmax - xmin + paste_x
- y1 = mf_ymax - ymin + paste_y
- # Filter formula blocks outside the graph
- if any([x1 < 0, y1 < 0]) or any([x0 > new_width, y0 > new_height]):
- continue
- else:
- adjusted_mfdetrec_res.append({
- "bbox": [x0, y0, x1, y1],
- })
-
- # OCR recognition
- new_image = cv2.cvtColor(np.asarray(new_image), cv2.COLOR_RGB2BGR)
- ocr_res = self.ocr_model.ocr(new_image, mfd_res=adjusted_mfdetrec_res)[0]
-
- # Integration results
- if ocr_res:
- for box_ocr_res in ocr_res:
- p1, p2, p3, p4 = box_ocr_res[0]
- text, score = box_ocr_res[1]
-
- # Convert the coordinates back to the original coordinate system
- p1 = [p1[0] - paste_x + xmin, p1[1] - paste_y + ymin]
- p2 = [p2[0] - paste_x + xmin, p2[1] - paste_y + ymin]
- p3 = [p3[0] - paste_x + xmin, p3[1] - paste_y + ymin]
- p4 = [p4[0] - paste_x + xmin, p4[1] - paste_y + ymin]
-
- layout_res.append({
- 'category_id': 15,
- 'poly': p1 + p2 + p3 + p4,
- 'score': round(score, 2),
- 'text': text,
- })
-
- ocr_cost = round(time.time() - ocr_start, 2)
- logger.info(f"ocr time: {ocr_cost}")
-
- # 表格识别 table recognition
- if self.apply_table:
- table_start = time.time()
- for res in table_res_list:
- new_image, _ = crop_img(res, pil_img)
- single_table_start_time = time.time()
- # logger.info("------------------table recognition processing begins-----------------")
- latex_code = None
- html_code = None
- if self.table_model_name == MODEL_NAME.STRUCT_EQTABLE:
- with torch.no_grad():
- latex_code = self.table_model.image2latex(new_image)[0]
- else:
- html_code = self.table_model.img2html(new_image)
-
- run_time = time.time() - single_table_start_time
- # logger.info(f"------------table recognition processing ends within {run_time}s-----")
- if run_time > self.table_max_time:
- logger.warning(f"------------table recognition processing exceeds max time {self.table_max_time}s----------")
- # 判断是否返回正常
-
- if latex_code:
- expected_ending = latex_code.strip().endswith('end{tabular}') or latex_code.strip().endswith(
- 'end{table}')
- if expected_ending:
- res["latex"] = latex_code
- else:
- logger.warning(f"table recognition processing fails, not found expected LaTeX table end")
- elif html_code:
- res["html"] = html_code
- else:
- logger.warning(f"table recognition processing fails, not get latex or html return")
- logger.info(f"table time: {round(time.time() - table_start, 2)}")
-
- logger.info(f"-----page total time: {round(time.time() - page_start, 2)}-----")
-
- return layout_res
-
-
diff --git a/magic_pdf/model/pek_sub_modules/__init__.py b/magic_pdf/model/pek_sub_modules/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/__init__.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/backbone.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/backbone.py
deleted file mode 100644
index 5364f862e..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/backbone.py
+++ /dev/null
@@ -1,179 +0,0 @@
-# --------------------------------------------------------------------------------
-# VIT: Multi-Path Vision Transformer for Dense Prediction
-# Copyright (c) 2022 Electronics and Telecommunications Research Institute (ETRI).
-# All Rights Reserved.
-# Written by Youngwan Lee
-# This source code is licensed(Dual License(GPL3.0 & Commercial)) under the license found in the
-# LICENSE file in the root directory of this source tree.
-# --------------------------------------------------------------------------------
-# References:
-# timm: https://github.com/rwightman/pytorch-image-models/tree/master/timm
-# CoaT: https://github.com/mlpc-ucsd/CoaT
-# --------------------------------------------------------------------------------
-
-
-import torch
-
-from detectron2.layers import (
- ShapeSpec,
-)
-from detectron2.modeling import Backbone, BACKBONE_REGISTRY, FPN
-from detectron2.modeling.backbone.fpn import LastLevelP6P7, LastLevelMaxPool
-
-from .beit import beit_base_patch16, dit_base_patch16, dit_large_patch16, beit_large_patch16
-from .deit import deit_base_patch16, mae_base_patch16
-from .layoutlmft.models.layoutlmv3 import LayoutLMv3Model
-from transformers import AutoConfig
-
-__all__ = [
- "build_vit_fpn_backbone",
-]
-
-
-class VIT_Backbone(Backbone):
- """
- Implement VIT backbone.
- """
-
- def __init__(self, name, out_features, drop_path, img_size, pos_type, model_kwargs,
- config_path=None, image_only=False, cfg=None):
- super().__init__()
- self._out_features = out_features
- if 'base' in name:
- self._out_feature_strides = {"layer3": 4, "layer5": 8, "layer7": 16, "layer11": 32}
- self._out_feature_channels = {"layer3": 768, "layer5": 768, "layer7": 768, "layer11": 768}
- else:
- self._out_feature_strides = {"layer7": 4, "layer11": 8, "layer15": 16, "layer23": 32}
- self._out_feature_channels = {"layer7": 1024, "layer11": 1024, "layer15": 1024, "layer23": 1024}
-
- if name == 'beit_base_patch16':
- model_func = beit_base_patch16
- elif name == 'dit_base_patch16':
- model_func = dit_base_patch16
- elif name == "deit_base_patch16":
- model_func = deit_base_patch16
- elif name == "mae_base_patch16":
- model_func = mae_base_patch16
- elif name == "dit_large_patch16":
- model_func = dit_large_patch16
- elif name == "beit_large_patch16":
- model_func = beit_large_patch16
-
- if 'beit' in name or 'dit' in name:
- if pos_type == "abs":
- self.backbone = model_func(img_size=img_size,
- out_features=out_features,
- drop_path_rate=drop_path,
- use_abs_pos_emb=True,
- **model_kwargs)
- elif pos_type == "shared_rel":
- self.backbone = model_func(img_size=img_size,
- out_features=out_features,
- drop_path_rate=drop_path,
- use_shared_rel_pos_bias=True,
- **model_kwargs)
- elif pos_type == "rel":
- self.backbone = model_func(img_size=img_size,
- out_features=out_features,
- drop_path_rate=drop_path,
- use_rel_pos_bias=True,
- **model_kwargs)
- else:
- raise ValueError()
- elif "layoutlmv3" in name:
- config = AutoConfig.from_pretrained(config_path)
- # disable relative bias as DiT
- config.has_spatial_attention_bias = False
- config.has_relative_attention_bias = False
- self.backbone = LayoutLMv3Model(config, detection=True,
- out_features=out_features, image_only=image_only)
- else:
- self.backbone = model_func(img_size=img_size,
- out_features=out_features,
- drop_path_rate=drop_path,
- **model_kwargs)
- self.name = name
-
- def forward(self, x):
- """
- Args:
- x: Tensor of shape (N,C,H,W). H, W must be a multiple of ``self.size_divisibility``.
-
- Returns:
- dict[str->Tensor]: names and the corresponding features
- """
- if "layoutlmv3" in self.name:
- return self.backbone.forward(
- input_ids=x["input_ids"] if "input_ids" in x else None,
- bbox=x["bbox"] if "bbox" in x else None,
- images=x["images"] if "images" in x else None,
- attention_mask=x["attention_mask"] if "attention_mask" in x else None,
- # output_hidden_states=True,
- )
- assert x.dim() == 4, f"VIT takes an input of shape (N, C, H, W). Got {x.shape} instead!"
- return self.backbone.forward_features(x)
-
- def output_shape(self):
- return {
- name: ShapeSpec(
- channels=self._out_feature_channels[name], stride=self._out_feature_strides[name]
- )
- for name in self._out_features
- }
-
-
-def build_VIT_backbone(cfg):
- """
- Create a VIT instance from config.
-
- Args:
- cfg: a detectron2 CfgNode
-
- Returns:
- A VIT backbone instance.
- """
- # fmt: off
- name = cfg.MODEL.VIT.NAME
- out_features = cfg.MODEL.VIT.OUT_FEATURES
- drop_path = cfg.MODEL.VIT.DROP_PATH
- img_size = cfg.MODEL.VIT.IMG_SIZE
- pos_type = cfg.MODEL.VIT.POS_TYPE
-
- model_kwargs = eval(str(cfg.MODEL.VIT.MODEL_KWARGS).replace("`", ""))
-
- if 'layoutlmv3' in name:
- if cfg.MODEL.CONFIG_PATH != '':
- config_path = cfg.MODEL.CONFIG_PATH
- else:
- config_path = cfg.MODEL.WEIGHTS.replace('pytorch_model.bin', '') # layoutlmv3 pre-trained models
- config_path = config_path.replace('model_final.pth', '') # detection fine-tuned models
- else:
- config_path = None
-
- return VIT_Backbone(name, out_features, drop_path, img_size, pos_type, model_kwargs,
- config_path=config_path, image_only=cfg.MODEL.IMAGE_ONLY, cfg=cfg)
-
-
-@BACKBONE_REGISTRY.register()
-def build_vit_fpn_backbone(cfg, input_shape: ShapeSpec):
- """
- Create a VIT w/ FPN backbone.
-
- Args:
- cfg: a detectron2 CfgNode
-
- Returns:
- backbone (Backbone): backbone module, must be a subclass of :class:`Backbone`.
- """
- bottom_up = build_VIT_backbone(cfg)
- in_features = cfg.MODEL.FPN.IN_FEATURES
- out_channels = cfg.MODEL.FPN.OUT_CHANNELS
- backbone = FPN(
- bottom_up=bottom_up,
- in_features=in_features,
- out_channels=out_channels,
- norm=cfg.MODEL.FPN.NORM,
- top_block=LastLevelMaxPool(),
- fuse_type=cfg.MODEL.FPN.FUSE_TYPE,
- )
- return backbone
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/beit.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/beit.py
deleted file mode 100644
index 03d4fabdc..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/beit.py
+++ /dev/null
@@ -1,671 +0,0 @@
-""" Vision Transformer (ViT) in PyTorch
-
-A PyTorch implement of Vision Transformers as described in
-'An Image Is Worth 16 x 16 Words: Transformers for Image Recognition at Scale' - https://arxiv.org/abs/2010.11929
-
-The official jax code is released and available at https://github.com/google-research/vision_transformer
-
-Status/TODO:
-* Models updated to be compatible with official impl. Args added to support backward compat for old PyTorch weights.
-* Weights ported from official jax impl for 384x384 base and small models, 16x16 and 32x32 patches.
-* Trained (supervised on ImageNet-1k) my custom 'small' patch model to 77.9, 'base' to 79.4 top-1 with this code.
-* Hopefully find time and GPUs for SSL or unsupervised pretraining on OpenImages w/ ImageNet fine-tune in future.
-
-Acknowledgments:
-* The paper authors for releasing code and weights, thanks!
-* I fixed my class token impl based on Phil Wang's https://github.com/lucidrains/vit-pytorch ... check it out
-for some einops/einsum fun
-* Simple transformer style inspired by Andrej Karpathy's https://github.com/karpathy/minGPT
-* Bert reference code checks against Huggingface Transformers and Tensorflow Bert
-
-Hacked together by / Copyright 2020 Ross Wightman
-"""
-import warnings
-import math
-import torch
-from functools import partial
-import torch.nn as nn
-import torch.nn.functional as F
-import torch.utils.checkpoint as checkpoint
-from timm.models.layers import drop_path, to_2tuple, trunc_normal_
-
-
-def _cfg(url='', **kwargs):
- return {
- 'url': url,
- 'num_classes': 1000, 'input_size': (3, 224, 224), 'pool_size': None,
- 'crop_pct': .9, 'interpolation': 'bicubic',
- 'mean': (0.5, 0.5, 0.5), 'std': (0.5, 0.5, 0.5),
- **kwargs
- }
-
-
-class DropPath(nn.Module):
- """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
- """
-
- def __init__(self, drop_prob=None):
- super(DropPath, self).__init__()
- self.drop_prob = drop_prob
-
- def forward(self, x):
- return drop_path(x, self.drop_prob, self.training)
-
- def extra_repr(self) -> str:
- return 'p={}'.format(self.drop_prob)
-
-
-class Mlp(nn.Module):
- def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
- super().__init__()
- out_features = out_features or in_features
- hidden_features = hidden_features or in_features
- self.fc1 = nn.Linear(in_features, hidden_features)
- self.act = act_layer()
- self.fc2 = nn.Linear(hidden_features, out_features)
- self.drop = nn.Dropout(drop)
-
- def forward(self, x):
- x = self.fc1(x)
- x = self.act(x)
- # x = self.drop(x)
- # commit this for the orignal BERT implement
- x = self.fc2(x)
- x = self.drop(x)
- return x
-
-
-class Attention(nn.Module):
- def __init__(
- self, dim, num_heads=8, qkv_bias=False, qk_scale=None, attn_drop=0.,
- proj_drop=0., window_size=None, attn_head_dim=None):
- super().__init__()
- self.num_heads = num_heads
- head_dim = dim // num_heads
- if attn_head_dim is not None:
- head_dim = attn_head_dim
- all_head_dim = head_dim * self.num_heads
- # NOTE scale factor was wrong in my original version, can set manually to be compat with prev weights
- self.scale = qk_scale or head_dim ** -0.5
-
- self.qkv = nn.Linear(dim, all_head_dim * 3, bias=False)
- if qkv_bias:
- self.q_bias = nn.Parameter(torch.zeros(all_head_dim))
- self.v_bias = nn.Parameter(torch.zeros(all_head_dim))
- else:
- self.q_bias = None
- self.v_bias = None
-
- if window_size:
- self.window_size = window_size
- self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3
- self.relative_position_bias_table = nn.Parameter(
- torch.zeros(self.num_relative_distance, num_heads)) # 2*Wh-1 * 2*Ww-1, nH
- # cls to token & token 2 cls & cls to cls
-
- # get pair-wise relative position index for each token inside the window
- coords_h = torch.arange(window_size[0])
- coords_w = torch.arange(window_size[1])
- coords = torch.stack(torch.meshgrid([coords_h, coords_w])) # 2, Wh, Ww
- coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
- relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
- relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
- relative_coords[:, :, 0] += window_size[0] - 1 # shift to start from 0
- relative_coords[:, :, 1] += window_size[1] - 1
- relative_coords[:, :, 0] *= 2 * window_size[1] - 1
- relative_position_index = \
- torch.zeros(size=(window_size[0] * window_size[1] + 1,) * 2, dtype=relative_coords.dtype)
- relative_position_index[1:, 1:] = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
- relative_position_index[0, 0:] = self.num_relative_distance - 3
- relative_position_index[0:, 0] = self.num_relative_distance - 2
- relative_position_index[0, 0] = self.num_relative_distance - 1
-
- self.register_buffer("relative_position_index", relative_position_index)
-
- # trunc_normal_(self.relative_position_bias_table, std=.0)
- else:
- self.window_size = None
- self.relative_position_bias_table = None
- self.relative_position_index = None
-
- self.attn_drop = nn.Dropout(attn_drop)
- self.proj = nn.Linear(all_head_dim, dim)
- self.proj_drop = nn.Dropout(proj_drop)
-
- def forward(self, x, rel_pos_bias=None, training_window_size=None):
- B, N, C = x.shape
- qkv_bias = None
- if self.q_bias is not None:
- qkv_bias = torch.cat((self.q_bias, torch.zeros_like(self.v_bias, requires_grad=False), self.v_bias))
- # qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
- qkv = F.linear(input=x, weight=self.qkv.weight, bias=qkv_bias)
- qkv = qkv.reshape(B, N, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4)
- q, k, v = qkv[0], qkv[1], qkv[2] # make torchscript happy (cannot use tensor as tuple)
-
- q = q * self.scale
- attn = (q @ k.transpose(-2, -1))
-
- if self.relative_position_bias_table is not None:
- if training_window_size == self.window_size:
- relative_position_bias = \
- self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
- self.window_size[0] * self.window_size[1] + 1,
- self.window_size[0] * self.window_size[1] + 1, -1) # Wh*Ww,Wh*Ww,nH
- relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
- attn = attn + relative_position_bias.unsqueeze(0)
- else:
- training_window_size = tuple(training_window_size.tolist())
- new_num_relative_distance = (2 * training_window_size[0] - 1) * (2 * training_window_size[1] - 1) + 3
- # new_num_relative_dis 为 所有可能的相对位置选项，包含cls-cls，tok-cls，与cls-tok
- new_relative_position_bias_table = F.interpolate(
- self.relative_position_bias_table[:-3, :].permute(1, 0).view(1, self.num_heads,
- 2 * self.window_size[0] - 1,
- 2 * self.window_size[1] - 1),
- size=(2 * training_window_size[0] - 1, 2 * training_window_size[1] - 1), mode='bicubic',
- align_corners=False)
- new_relative_position_bias_table = new_relative_position_bias_table.view(self.num_heads,
- new_num_relative_distance - 3).permute(
- 1, 0)
- new_relative_position_bias_table = torch.cat(
- [new_relative_position_bias_table, self.relative_position_bias_table[-3::]], dim=0)
-
- # get pair-wise relative position index for each token inside the window
- coords_h = torch.arange(training_window_size[0])
- coords_w = torch.arange(training_window_size[1])
- coords = torch.stack(torch.meshgrid([coords_h, coords_w])) # 2, Wh, Ww
- coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
- relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
- relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
- relative_coords[:, :, 0] += training_window_size[0] - 1 # shift to start from 0
- relative_coords[:, :, 1] += training_window_size[1] - 1
- relative_coords[:, :, 0] *= 2 * training_window_size[1] - 1
- relative_position_index = \
- torch.zeros(size=(training_window_size[0] * training_window_size[1] + 1,) * 2,
- dtype=relative_coords.dtype)
- relative_position_index[1:, 1:] = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
- relative_position_index[0, 0:] = new_num_relative_distance - 3
- relative_position_index[0:, 0] = new_num_relative_distance - 2
- relative_position_index[0, 0] = new_num_relative_distance - 1
-
- relative_position_bias = \
- new_relative_position_bias_table[relative_position_index.view(-1)].view(
- training_window_size[0] * training_window_size[1] + 1,
- training_window_size[0] * training_window_size[1] + 1, -1) # Wh*Ww,Wh*Ww,nH
- relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
- attn = attn + relative_position_bias.unsqueeze(0)
-
- if rel_pos_bias is not None:
- attn = attn + rel_pos_bias
-
- attn = attn.softmax(dim=-1)
- attn = self.attn_drop(attn)
-
- x = (attn @ v).transpose(1, 2).reshape(B, N, -1)
- x = self.proj(x)
- x = self.proj_drop(x)
- return x
-
-
-class Block(nn.Module):
-
- def __init__(self, dim, num_heads, mlp_ratio=4., qkv_bias=False, qk_scale=None, drop=0., attn_drop=0.,
- drop_path=0., init_values=None, act_layer=nn.GELU, norm_layer=nn.LayerNorm,
- window_size=None, attn_head_dim=None):
- super().__init__()
- self.norm1 = norm_layer(dim)
- self.attn = Attention(
- dim, num_heads=num_heads, qkv_bias=qkv_bias, qk_scale=qk_scale,
- attn_drop=attn_drop, proj_drop=drop, window_size=window_size, attn_head_dim=attn_head_dim)
- # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
- self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
- self.norm2 = norm_layer(dim)
- mlp_hidden_dim = int(dim * mlp_ratio)
- self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
-
- if init_values is not None:
- self.gamma_1 = nn.Parameter(init_values * torch.ones((dim)), requires_grad=True)
- self.gamma_2 = nn.Parameter(init_values * torch.ones((dim)), requires_grad=True)
- else:
- self.gamma_1, self.gamma_2 = None, None
-
- def forward(self, x, rel_pos_bias=None, training_window_size=None):
- if self.gamma_1 is None:
- x = x + self.drop_path(
- self.attn(self.norm1(x), rel_pos_bias=rel_pos_bias, training_window_size=training_window_size))
- x = x + self.drop_path(self.mlp(self.norm2(x)))
- else:
- x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x), rel_pos_bias=rel_pos_bias,
- training_window_size=training_window_size))
- x = x + self.drop_path(self.gamma_2 * self.mlp(self.norm2(x)))
- return x
-
-
-class PatchEmbed(nn.Module):
- """ Image to Patch Embedding
- """
-
- def __init__(self, img_size=[224, 224], patch_size=16, in_chans=3, embed_dim=768):
- super().__init__()
- img_size = to_2tuple(img_size)
- patch_size = to_2tuple(patch_size)
- num_patches = (img_size[1] // patch_size[1]) * (img_size[0] // patch_size[0])
- self.patch_shape = (img_size[0] // patch_size[0], img_size[1] // patch_size[1])
- self.num_patches_w = self.patch_shape[0]
- self.num_patches_h = self.patch_shape[1]
- # the so-called patch_shape is the patch shape during pre-training
- self.img_size = img_size
- self.patch_size = patch_size
- self.num_patches = num_patches
-
- self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
-
- def forward(self, x, position_embedding=None, **kwargs):
- # FIXME look at relaxing size constraints
- # assert H == self.img_size[0] and W == self.img_size[1], \
- # f"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]})."
- x = self.proj(x)
- Hp, Wp = x.shape[2], x.shape[3]
-
- if position_embedding is not None:
- # interpolate the position embedding to the corresponding size
- position_embedding = position_embedding.view(1, self.patch_shape[0], self.patch_shape[1], -1).permute(0, 3,
- 1, 2)
- position_embedding = F.interpolate(position_embedding, size=(Hp, Wp), mode='bicubic')
- x = x + position_embedding
-
- x = x.flatten(2).transpose(1, 2)
- return x, (Hp, Wp)
-
-
-class HybridEmbed(nn.Module):
- """ CNN Feature Map Embedding
- Extract feature map from CNN, flatten, project to embedding dim.
- """
-
- def __init__(self, backbone, img_size=[224, 224], feature_size=None, in_chans=3, embed_dim=768):
- super().__init__()
- assert isinstance(backbone, nn.Module)
- img_size = to_2tuple(img_size)
- self.img_size = img_size
- self.backbone = backbone
- if feature_size is None:
- with torch.no_grad():
- # FIXME this is hacky, but most reliable way of determining the exact dim of the output feature
- # map for all networks, the feature metadata has reliable channel and stride info, but using
- # stride to calc feature dim requires info about padding of each stage that isn't captured.
- training = backbone.training
- if training:
- backbone.eval()
- o = self.backbone(torch.zeros(1, in_chans, img_size[0], img_size[1]))[-1]
- feature_size = o.shape[-2:]
- feature_dim = o.shape[1]
- backbone.train(training)
- else:
- feature_size = to_2tuple(feature_size)
- feature_dim = self.backbone.feature_info.channels()[-1]
- self.num_patches = feature_size[0] * feature_size[1]
- self.proj = nn.Linear(feature_dim, embed_dim)
-
- def forward(self, x):
- x = self.backbone(x)[-1]
- x = x.flatten(2).transpose(1, 2)
- x = self.proj(x)
- return x
-
-
-class RelativePositionBias(nn.Module):
-
- def __init__(self, window_size, num_heads):
- super().__init__()
- self.window_size = window_size
- self.num_heads = num_heads
- self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3
- self.relative_position_bias_table = nn.Parameter(
- torch.zeros(self.num_relative_distance, num_heads)) # 2*Wh-1 * 2*Ww-1, nH
- # cls to token & token 2 cls & cls to cls
-
- # get pair-wise relative position index for each token inside the window
- coords_h = torch.arange(window_size[0])
- coords_w = torch.arange(window_size[1])
- coords = torch.stack(torch.meshgrid([coords_h, coords_w])) # 2, Wh, Ww
- coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
- relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
- relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
- relative_coords[:, :, 0] += window_size[0] - 1 # shift to start from 0
- relative_coords[:, :, 1] += window_size[1] - 1
- relative_coords[:, :, 0] *= 2 * window_size[1] - 1
- relative_position_index = \
- torch.zeros(size=(window_size[0] * window_size[1] + 1,) * 2, dtype=relative_coords.dtype)
- relative_position_index[1:, 1:] = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
- relative_position_index[0, 0:] = self.num_relative_distance - 3
- relative_position_index[0:, 0] = self.num_relative_distance - 2
- relative_position_index[0, 0] = self.num_relative_distance - 1
-
- self.register_buffer("relative_position_index", relative_position_index)
-
- # trunc_normal_(self.relative_position_bias_table, std=.02)
-
- def forward(self, training_window_size):
- if training_window_size == self.window_size:
- relative_position_bias = \
- self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
- self.window_size[0] * self.window_size[1] + 1,
- self.window_size[0] * self.window_size[1] + 1, -1) # Wh*Ww,Wh*Ww,nH
- relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
- else:
- training_window_size = tuple(training_window_size.tolist())
- new_num_relative_distance = (2 * training_window_size[0] - 1) * (2 * training_window_size[1] - 1) + 3
- # new_num_relative_dis 为 所有可能的相对位置选项，包含cls-cls，tok-cls，与cls-tok
- new_relative_position_bias_table = F.interpolate(
- self.relative_position_bias_table[:-3, :].permute(1, 0).view(1, self.num_heads,
- 2 * self.window_size[0] - 1,
- 2 * self.window_size[1] - 1),
- size=(2 * training_window_size[0] - 1, 2 * training_window_size[1] - 1), mode='bicubic',
- align_corners=False)
- new_relative_position_bias_table = new_relative_position_bias_table.view(self.num_heads,
- new_num_relative_distance - 3).permute(
- 1, 0)
- new_relative_position_bias_table = torch.cat(
- [new_relative_position_bias_table, self.relative_position_bias_table[-3::]], dim=0)
-
- # get pair-wise relative position index for each token inside the window
- coords_h = torch.arange(training_window_size[0])
- coords_w = torch.arange(training_window_size[1])
- coords = torch.stack(torch.meshgrid([coords_h, coords_w])) # 2, Wh, Ww
- coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww
- relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww
- relative_coords = relative_coords.permute(1, 2, 0).contiguous() # Wh*Ww, Wh*Ww, 2
- relative_coords[:, :, 0] += training_window_size[0] - 1 # shift to start from 0
- relative_coords[:, :, 1] += training_window_size[1] - 1
- relative_coords[:, :, 0] *= 2 * training_window_size[1] - 1
- relative_position_index = \
- torch.zeros(size=(training_window_size[0] * training_window_size[1] + 1,) * 2,
- dtype=relative_coords.dtype)
- relative_position_index[1:, 1:] = relative_coords.sum(-1) # Wh*Ww, Wh*Ww
- relative_position_index[0, 0:] = new_num_relative_distance - 3
- relative_position_index[0:, 0] = new_num_relative_distance - 2
- relative_position_index[0, 0] = new_num_relative_distance - 1
-
- relative_position_bias = \
- new_relative_position_bias_table[relative_position_index.view(-1)].view(
- training_window_size[0] * training_window_size[1] + 1,
- training_window_size[0] * training_window_size[1] + 1, -1) # Wh*Ww,Wh*Ww,nH
- relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous() # nH, Wh*Ww, Wh*Ww
-
- return relative_position_bias
-
-
-class BEiT(nn.Module):
- """ Vision Transformer with support for patch or hybrid CNN input stage
- """
-
- def __init__(self,
- img_size=[224, 224],
- patch_size=16,
- in_chans=3,
- num_classes=80,
- embed_dim=768,
- depth=12,
- num_heads=12,
- mlp_ratio=4.,
- qkv_bias=False,
- qk_scale=None,
- drop_rate=0.,
- attn_drop_rate=0.,
- drop_path_rate=0.,
- hybrid_backbone=None,
- norm_layer=None,
- init_values=None,
- use_abs_pos_emb=False,
- use_rel_pos_bias=False,
- use_shared_rel_pos_bias=False,
- use_checkpoint=True,
- pretrained=None,
- out_features=None,
- ):
-
- super(BEiT, self).__init__()
-
- norm_layer = norm_layer or partial(nn.LayerNorm, eps=1e-6)
- self.num_classes = num_classes
- self.num_features = self.embed_dim = embed_dim # num_features for consistency with other models
- self.use_checkpoint = use_checkpoint
-
- if hybrid_backbone is not None:
- self.patch_embed = HybridEmbed(
- hybrid_backbone, img_size=img_size, in_chans=in_chans, embed_dim=embed_dim)
- else:
- self.patch_embed = PatchEmbed(
- img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim)
- num_patches = self.patch_embed.num_patches
- self.out_features = out_features
- self.out_indices = [int(name[5:]) for name in out_features]
-
- self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
- # self.mask_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
- if use_abs_pos_emb:
- self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))
- else:
- self.pos_embed = None
- self.pos_drop = nn.Dropout(p=drop_rate)
-
- self.use_shared_rel_pos_bias = use_shared_rel_pos_bias
- if use_shared_rel_pos_bias:
- self.rel_pos_bias = RelativePositionBias(window_size=self.patch_embed.patch_shape, num_heads=num_heads)
- else:
- self.rel_pos_bias = None
-
- dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] # stochastic depth decay rule
- self.use_rel_pos_bias = use_rel_pos_bias
- self.blocks = nn.ModuleList([
- Block(
- dim=embed_dim, num_heads=num_heads, mlp_ratio=mlp_ratio, qkv_bias=qkv_bias, qk_scale=qk_scale,
- drop=drop_rate, attn_drop=attn_drop_rate, drop_path=dpr[i], norm_layer=norm_layer,
- init_values=init_values, window_size=self.patch_embed.patch_shape if use_rel_pos_bias else None)
- for i in range(depth)])
-
- # trunc_normal_(self.mask_token, std=.02)
-
- if patch_size == 16:
- self.fpn1 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- # nn.SyncBatchNorm(embed_dim),
- nn.BatchNorm2d(embed_dim),
- nn.GELU(),
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn2 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn3 = nn.Identity()
-
- self.fpn4 = nn.MaxPool2d(kernel_size=2, stride=2)
- elif patch_size == 8:
- self.fpn1 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn2 = nn.Identity()
-
- self.fpn3 = nn.Sequential(
- nn.MaxPool2d(kernel_size=2, stride=2),
- )
-
- self.fpn4 = nn.Sequential(
- nn.MaxPool2d(kernel_size=4, stride=4),
- )
-
- if self.pos_embed is not None:
- trunc_normal_(self.pos_embed, std=.02)
- trunc_normal_(self.cls_token, std=.02)
- self.apply(self._init_weights)
- self.fix_init_weight()
-
- def fix_init_weight(self):
- def rescale(param, layer_id):
- param.div_(math.sqrt(2.0 * layer_id))
-
- for layer_id, layer in enumerate(self.blocks):
- rescale(layer.attn.proj.weight.data, layer_id + 1)
- rescale(layer.mlp.fc2.weight.data, layer_id + 1)
-
- def _init_weights(self, m):
- if isinstance(m, nn.Linear):
- trunc_normal_(m.weight, std=.02)
- if isinstance(m, nn.Linear) and m.bias is not None:
- nn.init.constant_(m.bias, 0)
- elif isinstance(m, nn.LayerNorm):
- nn.init.constant_(m.bias, 0)
- nn.init.constant_(m.weight, 1.0)
-
- '''
- def init_weights(self):
- """Initialize the weights in backbone.
-
- Args:
- pretrained (str, optional): Path to pre-trained weights.
- Defaults to None.
- """
- logger = get_root_logger()
-
- if self.pos_embed is not None:
- trunc_normal_(self.pos_embed, std=.02)
- trunc_normal_(self.cls_token, std=.02)
- self.apply(self._init_weights)
- self.fix_init_weight()
-
- if self.init_cfg is None:
- logger.warn(f'No pre-trained weights for '
- f'{self.__class__.__name__}, '
- f'training start from scratch')
- else:
- assert 'checkpoint' in self.init_cfg, f'Only support ' \
- f'specify `Pretrained` in ' \
- f'`init_cfg` in ' \
- f'{self.__class__.__name__} '
- logger.info(f"Will load ckpt from {self.init_cfg['checkpoint']}")
- load_checkpoint(self,
- filename=self.init_cfg['checkpoint'],
- strict=False,
- logger=logger,
- beit_spec_expand_rel_pos = self.use_rel_pos_bias,
- )
- '''
-
- def get_num_layers(self):
- return len(self.blocks)
-
- @torch.jit.ignore
- def no_weight_decay(self):
- return {'pos_embed', 'cls_token'}
-
- def forward_features(self, x):
- B, C, H, W = x.shape
- x, (Hp, Wp) = self.patch_embed(x, self.pos_embed[:, 1:, :] if self.pos_embed is not None else None)
- # Hp, Wp are HW for patches
- batch_size, seq_len, _ = x.size()
-
- cls_tokens = self.cls_token.expand(batch_size, -1, -1) # stole cls_tokens impl from Phil Wang, thanks
- if self.pos_embed is not None:
- cls_tokens = cls_tokens + self.pos_embed[:, :1, :]
- x = torch.cat((cls_tokens, x), dim=1)
- x = self.pos_drop(x)
-
- features = []
- training_window_size = torch.tensor([Hp, Wp])
-
- rel_pos_bias = self.rel_pos_bias(training_window_size) if self.rel_pos_bias is not None else None
-
- for i, blk in enumerate(self.blocks):
- if self.use_checkpoint:
- x = checkpoint.checkpoint(blk, x, rel_pos_bias, training_window_size)
- else:
- x = blk(x, rel_pos_bias=rel_pos_bias, training_window_size=training_window_size)
- if i in self.out_indices:
- xp = x[:, 1:, :].permute(0, 2, 1).reshape(B, -1, Hp, Wp)
- features.append(xp.contiguous())
-
- ops = [self.fpn1, self.fpn2, self.fpn3, self.fpn4]
- for i in range(len(features)):
- features[i] = ops[i](features[i])
-
- feat_out = {}
-
- for name, value in zip(self.out_features, features):
- feat_out[name] = value
-
- return feat_out
-
- def forward(self, x):
- x = self.forward_features(x)
- return x
-
-
-def beit_base_patch16(pretrained=False, **kwargs):
- model = BEiT(
- patch_size=16,
- embed_dim=768,
- depth=12,
- num_heads=12,
- mlp_ratio=4,
- qkv_bias=True,
- norm_layer=partial(nn.LayerNorm, eps=1e-6),
- init_values=None,
- **kwargs)
- model.default_cfg = _cfg()
- return model
-
-def beit_large_patch16(pretrained=False, **kwargs):
- model = BEiT(
- patch_size=16,
- embed_dim=1024,
- depth=24,
- num_heads=16,
- mlp_ratio=4,
- qkv_bias=True,
- norm_layer=partial(nn.LayerNorm, eps=1e-6),
- init_values=None,
- **kwargs)
- model.default_cfg = _cfg()
- return model
-
-def dit_base_patch16(pretrained=False, **kwargs):
- model = BEiT(
- patch_size=16,
- embed_dim=768,
- depth=12,
- num_heads=12,
- mlp_ratio=4,
- qkv_bias=True,
- norm_layer=partial(nn.LayerNorm, eps=1e-6),
- init_values=0.1,
- **kwargs)
- model.default_cfg = _cfg()
- return model
-
-def dit_large_patch16(pretrained=False, **kwargs):
- model = BEiT(
- patch_size=16,
- embed_dim=1024,
- depth=24,
- num_heads=16,
- mlp_ratio=4,
- qkv_bias=True,
- norm_layer=partial(nn.LayerNorm, eps=1e-6),
- init_values=1e-5,
- **kwargs)
- model.default_cfg = _cfg()
- return model
-
-if __name__ == '__main__':
- model = BEiT(use_checkpoint=True, use_shared_rel_pos_bias=True)
- model = model.to("cuda:0")
- input1 = torch.rand(2, 3, 512, 762).to("cuda:0")
- input2 = torch.rand(2, 3, 800, 1200).to("cuda:0")
- input3 = torch.rand(2, 3, 720, 1000).to("cuda:0")
- output1 = model(input1)
- output2 = model(input2)
- output3 = model(input3)
- print("all done")
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/deit.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/deit.py
deleted file mode 100644
index 9a13bb0a8..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/deit.py
+++ /dev/null
@@ -1,476 +0,0 @@
-"""
-Mostly copy-paste from DINO and timm library:
-https://github.com/facebookresearch/dino
-https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py
-"""
-import warnings
-
-import math
-import torch
-import torch.nn as nn
-import torch.utils.checkpoint as checkpoint
-from timm.models.layers import trunc_normal_, drop_path, to_2tuple
-from functools import partial
-
-def _cfg(url='', **kwargs):
- return {
- 'url': url,
- 'num_classes': 1000, 'input_size': (3, 224, 224), 'pool_size': None,
- 'crop_pct': .9, 'interpolation': 'bicubic',
- 'mean': (0.5, 0.5, 0.5), 'std': (0.5, 0.5, 0.5),
- **kwargs
- }
-
-class DropPath(nn.Module):
- """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
- """
-
- def __init__(self, drop_prob=None):
- super(DropPath, self).__init__()
- self.drop_prob = drop_prob
-
- def forward(self, x):
- return drop_path(x, self.drop_prob, self.training)
-
- def extra_repr(self) -> str:
- return 'p={}'.format(self.drop_prob)
-
-
-class Mlp(nn.Module):
- def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
- super().__init__()
- out_features = out_features or in_features
- hidden_features = hidden_features or in_features
- self.fc1 = nn.Linear(in_features, hidden_features)
- self.act = act_layer()
- self.fc2 = nn.Linear(hidden_features, out_features)
- self.drop = nn.Dropout(drop)
-
- def forward(self, x):
- x = self.fc1(x)
- x = self.act(x)
- x = self.drop(x)
- x = self.fc2(x)
- x = self.drop(x)
- return x
-
-
-class Attention(nn.Module):
- def __init__(self, dim, num_heads=8, qkv_bias=False, qk_scale=None, attn_drop=0., proj_drop=0.):
- super().__init__()
- self.num_heads = num_heads
- head_dim = dim // num_heads
- # NOTE scale factor was wrong in my original version, can set manually to be compat with prev weights
- self.scale = qk_scale or head_dim ** -0.5
-
- self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
- self.attn_drop = nn.Dropout(attn_drop)
- self.proj = nn.Linear(dim, dim)
- self.proj_drop = nn.Dropout(proj_drop)
-
- def forward(self, x):
- B, N, C = x.shape
- q, k, v = self.qkv(x).reshape(B, N, 3, self.num_heads,
- C // self.num_heads).permute(2, 0, 3, 1, 4)
-
- attn = (q @ k.transpose(-2, -1)) * self.scale
- attn = attn.softmax(dim=-1)
- attn = self.attn_drop(attn)
-
- x = (attn @ v).transpose(1, 2).reshape(B, N, C)
- x = self.proj(x)
- x = self.proj_drop(x)
- return x
-
-
-class Block(nn.Module):
-
- def __init__(self, dim, num_heads, mlp_ratio=4., qkv_bias=False, qk_scale=None, drop=0., attn_drop=0.,
- drop_path=0., act_layer=nn.GELU, norm_layer=nn.LayerNorm):
- super().__init__()
- self.norm1 = norm_layer(dim)
- self.attn = Attention(
- dim, num_heads=num_heads, qkv_bias=qkv_bias, qk_scale=qk_scale, attn_drop=attn_drop, proj_drop=drop)
- # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
- self.drop_path = DropPath(
- drop_path) if drop_path > 0. else nn.Identity()
- self.norm2 = norm_layer(dim)
- mlp_hidden_dim = int(dim * mlp_ratio)
- self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim,
- act_layer=act_layer, drop=drop)
-
- def forward(self, x):
- x = x + self.drop_path(self.attn(self.norm1(x)))
- x = x + self.drop_path(self.mlp(self.norm2(x)))
- return x
-
-
-class PatchEmbed(nn.Module):
- """ Image to Patch Embedding
- """
-
- def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768):
- super().__init__()
- img_size = to_2tuple(img_size)
- patch_size = to_2tuple(patch_size)
-
- self.window_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1])
-
- self.num_patches_w, self.num_patches_h = self.window_size
-
- self.num_patches = self.window_size[0] * self.window_size[1]
- self.img_size = img_size
- self.patch_size = patch_size
-
- self.proj = nn.Conv2d(in_chans, embed_dim,
- kernel_size=patch_size, stride=patch_size)
-
- def forward(self, x):
- x = self.proj(x)
- return x
-
-
-class HybridEmbed(nn.Module):
- """ CNN Feature Map Embedding
- Extract feature map from CNN, flatten, project to embedding dim.
- """
-
- def __init__(self, backbone, img_size=224, feature_size=None, in_chans=3, embed_dim=768):
- super().__init__()
- assert isinstance(backbone, nn.Module)
- img_size = to_2tuple(img_size)
- self.img_size = img_size
- self.backbone = backbone
- if feature_size is None:
- with torch.no_grad():
- # FIXME this is hacky, but most reliable way of determining the exact dim of the output feature
- # map for all networks, the feature metadata has reliable channel and stride info, but using
- # stride to calc feature dim requires info about padding of each stage that isn't captured.
- training = backbone.training
- if training:
- backbone.eval()
- o = self.backbone(torch.zeros(
- 1, in_chans, img_size[0], img_size[1]))[-1]
- feature_size = o.shape[-2:]
- feature_dim = o.shape[1]
- backbone.train(training)
- else:
- feature_size = to_2tuple(feature_size)
- feature_dim = self.backbone.feature_info.channels()[-1]
- self.num_patches = feature_size[0] * feature_size[1]
- self.proj = nn.Linear(feature_dim, embed_dim)
-
- def forward(self, x):
- x = self.backbone(x)[-1]
- x = x.flatten(2).transpose(1, 2)
- x = self.proj(x)
- return x
-
-
-class ViT(nn.Module):
- """ Vision Transformer with support for patch or hybrid CNN input stage
- """
-
- def __init__(self,
- model_name='vit_base_patch16_224',
- img_size=384,
- patch_size=16,
- in_chans=3,
- embed_dim=1024,
- depth=24,
- num_heads=16,
- num_classes=19,
- mlp_ratio=4.,
- qkv_bias=True,
- qk_scale=None,
- drop_rate=0.1,
- attn_drop_rate=0.,
- drop_path_rate=0.,
- hybrid_backbone=None,
- norm_layer=partial(nn.LayerNorm, eps=1e-6),
- norm_cfg=None,
- pos_embed_interp=False,
- random_init=False,
- align_corners=False,
- use_checkpoint=False,
- num_extra_tokens=1,
- out_features=None,
- **kwargs,
- ):
-
- super(ViT, self).__init__()
- self.model_name = model_name
- self.img_size = img_size
- self.patch_size = patch_size
- self.in_chans = in_chans
- self.embed_dim = embed_dim
- self.depth = depth
- self.num_heads = num_heads
- self.num_classes = num_classes
- self.mlp_ratio = mlp_ratio
- self.qkv_bias = qkv_bias
- self.qk_scale = qk_scale
- self.drop_rate = drop_rate
- self.attn_drop_rate = attn_drop_rate
- self.drop_path_rate = drop_path_rate
- self.hybrid_backbone = hybrid_backbone
- self.norm_layer = norm_layer
- self.norm_cfg = norm_cfg
- self.pos_embed_interp = pos_embed_interp
- self.random_init = random_init
- self.align_corners = align_corners
- self.use_checkpoint = use_checkpoint
- self.num_extra_tokens = num_extra_tokens
- self.out_features = out_features
- self.out_indices = [int(name[5:]) for name in out_features]
-
- # self.num_stages = self.depth
- # self.out_indices = tuple(range(self.num_stages))
-
- if self.hybrid_backbone is not None:
- self.patch_embed = HybridEmbed(
- self.hybrid_backbone, img_size=self.img_size, in_chans=self.in_chans, embed_dim=self.embed_dim)
- else:
- self.patch_embed = PatchEmbed(
- img_size=self.img_size, patch_size=self.patch_size, in_chans=self.in_chans, embed_dim=self.embed_dim)
- self.num_patches = self.patch_embed.num_patches
-
- self.cls_token = nn.Parameter(torch.zeros(1, 1, self.embed_dim))
-
- if self.num_extra_tokens == 2:
- self.dist_token = nn.Parameter(torch.zeros(1, 1, self.embed_dim))
-
- self.pos_embed = nn.Parameter(torch.zeros(
- 1, self.num_patches + self.num_extra_tokens, self.embed_dim))
- self.pos_drop = nn.Dropout(p=self.drop_rate)
-
- # self.num_extra_tokens = self.pos_embed.shape[-2] - self.num_patches
- dpr = [x.item() for x in torch.linspace(0, self.drop_path_rate,
- self.depth)] # stochastic depth decay rule
- self.blocks = nn.ModuleList([
- Block(
- dim=self.embed_dim, num_heads=self.num_heads, mlp_ratio=self.mlp_ratio, qkv_bias=self.qkv_bias,
- qk_scale=self.qk_scale,
- drop=self.drop_rate, attn_drop=self.attn_drop_rate, drop_path=dpr[i], norm_layer=self.norm_layer)
- for i in range(self.depth)])
-
- # NOTE as per official impl, we could have a pre-logits representation dense layer + tanh here
- # self.repr = nn.Linear(embed_dim, representation_size)
- # self.repr_act = nn.Tanh()
-
- if patch_size == 16:
- self.fpn1 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- nn.SyncBatchNorm(embed_dim),
- nn.GELU(),
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn2 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn3 = nn.Identity()
-
- self.fpn4 = nn.MaxPool2d(kernel_size=2, stride=2)
- elif patch_size == 8:
- self.fpn1 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn2 = nn.Identity()
-
- self.fpn3 = nn.Sequential(
- nn.MaxPool2d(kernel_size=2, stride=2),
- )
-
- self.fpn4 = nn.Sequential(
- nn.MaxPool2d(kernel_size=4, stride=4),
- )
-
- trunc_normal_(self.pos_embed, std=.02)
- trunc_normal_(self.cls_token, std=.02)
- if self.num_extra_tokens==2:
- trunc_normal_(self.dist_token, std=0.2)
- self.apply(self._init_weights)
- # self.fix_init_weight()
-
- def fix_init_weight(self):
- def rescale(param, layer_id):
- param.div_(math.sqrt(2.0 * layer_id))
-
- for layer_id, layer in enumerate(self.blocks):
- rescale(layer.attn.proj.weight.data, layer_id + 1)
- rescale(layer.mlp.fc2.weight.data, layer_id + 1)
-
- def _init_weights(self, m):
- if isinstance(m, nn.Linear):
- trunc_normal_(m.weight, std=.02)
- if isinstance(m, nn.Linear) and m.bias is not None:
- nn.init.constant_(m.bias, 0)
- elif isinstance(m, nn.LayerNorm):
- nn.init.constant_(m.bias, 0)
- nn.init.constant_(m.weight, 1.0)
-
- '''
- def init_weights(self):
- logger = get_root_logger()
-
- trunc_normal_(self.pos_embed, std=.02)
- trunc_normal_(self.cls_token, std=.02)
- self.apply(self._init_weights)
-
- if self.init_cfg is None:
- logger.warn(f'No pre-trained weights for '
- f'{self.__class__.__name__}, '
- f'training start from scratch')
- else:
- assert 'checkpoint' in self.init_cfg, f'Only support ' \
- f'specify `Pretrained` in ' \
- f'`init_cfg` in ' \
- f'{self.__class__.__name__} '
- logger.info(f"Will load ckpt from {self.init_cfg['checkpoint']}")
- load_checkpoint(self, filename=self.init_cfg['checkpoint'], strict=False, logger=logger)
- '''
-
- def get_num_layers(self):
- return len(self.blocks)
-
- @torch.jit.ignore
- def no_weight_decay(self):
- return {'pos_embed', 'cls_token'}
-
- def _conv_filter(self, state_dict, patch_size=16):
- """ convert patch embedding weight from manual patchify + linear proj to conv"""
- out_dict = {}
- for k, v in state_dict.items():
- if 'patch_embed.proj.weight' in k:
- v = v.reshape((v.shape[0], 3, patch_size, patch_size))
- out_dict[k] = v
- return out_dict
-
- def to_2D(self, x):
- n, hw, c = x.shape
- h = w = int(math.sqrt(hw))
- x = x.transpose(1, 2).reshape(n, c, h, w)
- return x
-
- def to_1D(self, x):
- n, c, h, w = x.shape
- x = x.reshape(n, c, -1).transpose(1, 2)
- return x
-
- def interpolate_pos_encoding(self, x, w, h):
- npatch = x.shape[1] - self.num_extra_tokens
- N = self.pos_embed.shape[1] - self.num_extra_tokens
- if npatch == N and w == h:
- return self.pos_embed
-
- class_ORdist_pos_embed = self.pos_embed[:, 0:self.num_extra_tokens]
-
- patch_pos_embed = self.pos_embed[:, self.num_extra_tokens:]
-
- dim = x.shape[-1]
- w0 = w // self.patch_embed.patch_size[0]
- h0 = h // self.patch_embed.patch_size[1]
- # we add a small number to avoid floating point error in the interpolation
- # see discussion at https://github.com/facebookresearch/dino/issues/8
- w0, h0 = w0 + 0.1, h0 + 0.1
- patch_pos_embed = nn.functional.interpolate(
- patch_pos_embed.reshape(1, int(math.sqrt(N)), int(math.sqrt(N)), dim).permute(0, 3, 1, 2),
- scale_factor=(w0 / math.sqrt(N), h0 / math.sqrt(N)),
- mode='bicubic',
- )
- assert int(w0) == patch_pos_embed.shape[-2] and int(h0) == patch_pos_embed.shape[-1]
- patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).view(1, -1, dim)
-
- return torch.cat((class_ORdist_pos_embed, patch_pos_embed), dim=1)
-
- def prepare_tokens(self, x, mask=None):
- B, nc, w, h = x.shape
- # patch linear embedding
- x = self.patch_embed(x)
-
- # mask image modeling
- if mask is not None:
- x = self.mask_model(x, mask)
- x = x.flatten(2).transpose(1, 2)
-
- # add the [CLS] token to the embed patch tokens
- all_tokens = [self.cls_token.expand(B, -1, -1)]
-
- if self.num_extra_tokens == 2:
- dist_tokens = self.dist_token.expand(B, -1, -1)
- all_tokens.append(dist_tokens)
- all_tokens.append(x)
-
- x = torch.cat(all_tokens, dim=1)
-
- # add positional encoding to each token
- x = x + self.interpolate_pos_encoding(x, w, h)
-
- return self.pos_drop(x)
-
- def forward_features(self, x):
- # print(f"==========shape of x is {x.shape}==========")
- B, _, H, W = x.shape
- Hp, Wp = H // self.patch_size, W // self.patch_size
- x = self.prepare_tokens(x)
-
- features = []
- for i, blk in enumerate(self.blocks):
- if self.use_checkpoint:
- x = checkpoint.checkpoint(blk, x)
- else:
- x = blk(x)
- if i in self.out_indices:
- xp = x[:, self.num_extra_tokens:, :].permute(0, 2, 1).reshape(B, -1, Hp, Wp)
- features.append(xp.contiguous())
-
- ops = [self.fpn1, self.fpn2, self.fpn3, self.fpn4]
- for i in range(len(features)):
- features[i] = ops[i](features[i])
-
- feat_out = {}
-
- for name, value in zip(self.out_features, features):
- feat_out[name] = value
-
- return feat_out
-
- def forward(self, x):
- x = self.forward_features(x)
- return x
-
-
-def deit_base_patch16(pretrained=False, **kwargs):
- model = ViT(
- patch_size=16,
- drop_rate=0.,
- embed_dim=768,
- depth=12,
- num_heads=12,
- num_classes=1000,
- mlp_ratio=4.,
- qkv_bias=True,
- use_checkpoint=True,
- num_extra_tokens=2,
- **kwargs)
- model.default_cfg = _cfg()
- return model
-
-def mae_base_patch16(pretrained=False, **kwargs):
- model = ViT(
- patch_size=16,
- drop_rate=0.,
- embed_dim=768,
- depth=12,
- num_heads=12,
- num_classes=1000,
- mlp_ratio=4.,
- qkv_bias=True,
- use_checkpoint=True,
- num_extra_tokens=1,
- **kwargs)
- model.default_cfg = _cfg()
- return model
\ No newline at end of file
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/__init__.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/__init__.py
deleted file mode 100644
index cd997b55f..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/__init__.py
+++ /dev/null
@@ -1,7 +0,0 @@
-from .models import (
- LayoutLMv3Config,
- LayoutLMv3ForTokenClassification,
- LayoutLMv3ForQuestionAnswering,
- LayoutLMv3ForSequenceClassification,
- LayoutLMv3Tokenizer,
-)
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/__init__.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/__init__.py
deleted file mode 100644
index 5bcec6c7c..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-# flake8: noqa
-from .data_collator import DataCollatorForKeyValueExtraction
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/cord.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/cord.py
deleted file mode 100644
index 820dc53a4..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/cord.py
+++ /dev/null
@@ -1,171 +0,0 @@
-'''
-Reference: https://huggingface.co/datasets/pierresi/cord/blob/main/cord.py
-'''
-
-
-import json
-import os
-from pathlib import Path
-import datasets
-from .image_utils import load_image, normalize_bbox
-logger = datasets.logging.get_logger(__name__)
-_CITATION = """\
-@article{park2019cord,
- title={CORD: A Consolidated Receipt Dataset for Post-OCR Parsing},
- author={Park, Seunghyun and Shin, Seung and Lee, Bado and Lee, Junyeop and Surh, Jaeheung and Seo, Minjoon and Lee, Hwalsuk}
- booktitle={Document Intelligence Workshop at Neural Information Processing Systems}
- year={2019}
-}
-"""
-_DESCRIPTION = """\
-https://github.com/clovaai/cord/
-"""
-
-def quad_to_box(quad):
- # test 87 is wrongly annotated
- box = (
- max(0, quad["x1"]),
- max(0, quad["y1"]),
- quad["x3"],
- quad["y3"]
- )
- if box[3] < box[1]:
- bbox = list(box)
- tmp = bbox[3]
- bbox[3] = bbox[1]
- bbox[1] = tmp
- box = tuple(bbox)
- if box[2] < box[0]:
- bbox = list(box)
- tmp = bbox[2]
- bbox[2] = bbox[0]
- bbox[0] = tmp
- box = tuple(bbox)
- return box
-
-def _get_drive_url(url):
- base_url = 'https://drive.google.com/uc?id='
- split_url = url.split('/')
- return base_url + split_url[5]
-
-_URLS = [
- _get_drive_url("https://drive.google.com/file/d/1MqhTbcj-AHXOqYoeoh12aRUwIprzTJYI/"),
- _get_drive_url("https://drive.google.com/file/d/1wYdp5nC9LnHQZ2FcmOoC0eClyWvcuARU/")
- # If you failed to download the dataset through the automatic downloader,
- # you can download it manually and modify the code to get the local dataset.
- # Or you can use the following links. Please follow the original LICENSE of CORD for usage.
- # "https://layoutlm.blob.core.windows.net/cord/CORD-1k-001.zip",
- # "https://layoutlm.blob.core.windows.net/cord/CORD-1k-002.zip"
-]
-
-class CordConfig(datasets.BuilderConfig):
- """BuilderConfig for CORD"""
- def __init__(self, **kwargs):
- """BuilderConfig for CORD.
- Args:
- **kwargs: keyword arguments forwarded to super.
- """
- super(CordConfig, self).__init__(**kwargs)
-
-class Cord(datasets.GeneratorBasedBuilder):
- BUILDER_CONFIGS = [
- CordConfig(name="cord", version=datasets.Version("1.0.0"), description="CORD dataset"),
- ]
-
- def _info(self):
- return datasets.DatasetInfo(
- description=_DESCRIPTION,
- features=datasets.Features(
- {
- "id": datasets.Value("string"),
- "words": datasets.Sequence(datasets.Value("string")),
- "bboxes": datasets.Sequence(datasets.Sequence(datasets.Value("int64"))),
- "ner_tags": datasets.Sequence(
- datasets.features.ClassLabel(
- names=["O","B-MENU.NM","B-MENU.NUM","B-MENU.UNITPRICE","B-MENU.CNT","B-MENU.DISCOUNTPRICE","B-MENU.PRICE","B-MENU.ITEMSUBTOTAL","B-MENU.VATYN","B-MENU.ETC","B-MENU.SUB_NM","B-MENU.SUB_UNITPRICE","B-MENU.SUB_CNT","B-MENU.SUB_PRICE","B-MENU.SUB_ETC","B-VOID_MENU.NM","B-VOID_MENU.PRICE","B-SUB_TOTAL.SUBTOTAL_PRICE","B-SUB_TOTAL.DISCOUNT_PRICE","B-SUB_TOTAL.SERVICE_PRICE","B-SUB_TOTAL.OTHERSVC_PRICE","B-SUB_TOTAL.TAX_PRICE","B-SUB_TOTAL.ETC","B-TOTAL.TOTAL_PRICE","B-TOTAL.TOTAL_ETC","B-TOTAL.CASHPRICE","B-TOTAL.CHANGEPRICE","B-TOTAL.CREDITCARDPRICE","B-TOTAL.EMONEYPRICE","B-TOTAL.MENUTYPE_CNT","B-TOTAL.MENUQTY_CNT","I-MENU.NM","I-MENU.NUM","I-MENU.UNITPRICE","I-MENU.CNT","I-MENU.DISCOUNTPRICE","I-MENU.PRICE","I-MENU.ITEMSUBTOTAL","I-MENU.VATYN","I-MENU.ETC","I-MENU.SUB_NM","I-MENU.SUB_UNITPRICE","I-MENU.SUB_CNT","I-MENU.SUB_PRICE","I-MENU.SUB_ETC","I-VOID_MENU.NM","I-VOID_MENU.PRICE","I-SUB_TOTAL.SUBTOTAL_PRICE","I-SUB_TOTAL.DISCOUNT_PRICE","I-SUB_TOTAL.SERVICE_PRICE","I-SUB_TOTAL.OTHERSVC_PRICE","I-SUB_TOTAL.TAX_PRICE","I-SUB_TOTAL.ETC","I-TOTAL.TOTAL_PRICE","I-TOTAL.TOTAL_ETC","I-TOTAL.CASHPRICE","I-TOTAL.CHANGEPRICE","I-TOTAL.CREDITCARDPRICE","I-TOTAL.EMONEYPRICE","I-TOTAL.MENUTYPE_CNT","I-TOTAL.MENUQTY_CNT"]
- )
- ),
- "image": datasets.Array3D(shape=(3, 224, 224), dtype="uint8"),
- "image_path": datasets.Value("string"),
- }
- ),
- supervised_keys=None,
- citation=_CITATION,
- homepage="https://github.com/clovaai/cord/",
- )
-
- def _split_generators(self, dl_manager):
- """Returns SplitGenerators."""
- """Uses local files located with data_dir"""
- downloaded_file = dl_manager.download_and_extract(_URLS)
- # move files from the second URL together with files from the first one.
- dest = Path(downloaded_file[0])/"CORD"
- for split in ["train", "dev", "test"]:
- for file_type in ["image", "json"]:
- if split == "test" and file_type == "json":
- continue
- files = (Path(downloaded_file[1])/"CORD"/split/file_type).iterdir()
- for f in files:
- os.rename(f, dest/split/file_type/f.name)
- return [
- datasets.SplitGenerator(
- name=datasets.Split.TRAIN, gen_kwargs={"filepath": dest/"train"}
- ),
- datasets.SplitGenerator(
- name=datasets.Split.VALIDATION, gen_kwargs={"filepath": dest/"dev"}
- ),
- datasets.SplitGenerator(
- name=datasets.Split.TEST, gen_kwargs={"filepath": dest/"test"}
- ),
- ]
-
- def get_line_bbox(self, bboxs):
- x = [bboxs[i][j] for i in range(len(bboxs)) for j in range(0, len(bboxs[i]), 2)]
- y = [bboxs[i][j] for i in range(len(bboxs)) for j in range(1, len(bboxs[i]), 2)]
-
- x0, y0, x1, y1 = min(x), min(y), max(x), max(y)
-
- assert x1 >= x0 and y1 >= y0
- bbox = [[x0, y0, x1, y1] for _ in range(len(bboxs))]
- return bbox
-
- def _generate_examples(self, filepath):
- logger.info("⏳ Generating examples from = %s", filepath)
- ann_dir = os.path.join(filepath, "json")
- img_dir = os.path.join(filepath, "image")
- for guid, file in enumerate(sorted(os.listdir(ann_dir))):
- words = []
- bboxes = []
- ner_tags = []
- file_path = os.path.join(ann_dir, file)
- with open(file_path, "r", encoding="utf8") as f:
- data = json.load(f)
- image_path = os.path.join(img_dir, file)
- image_path = image_path.replace("json", "png")
- image, size = load_image(image_path)
- for item in data["valid_line"]:
- cur_line_bboxes = []
- line_words, label = item["words"], item["category"]
- line_words = [w for w in line_words if w["text"].strip() != ""]
- if len(line_words) == 0:
- continue
- if label == "other":
- for w in line_words:
- words.append(w["text"])
- ner_tags.append("O")
- cur_line_bboxes.append(normalize_bbox(quad_to_box(w["quad"]), size))
- else:
- words.append(line_words[0]["text"])
- ner_tags.append("B-" + label.upper())
- cur_line_bboxes.append(normalize_bbox(quad_to_box(line_words[0]["quad"]), size))
- for w in line_words[1:]:
- words.append(w["text"])
- ner_tags.append("I-" + label.upper())
- cur_line_bboxes.append(normalize_bbox(quad_to_box(w["quad"]), size))
- # by default: --segment_level_layout 1
- # if do not want to use segment_level_layout, comment the following line
- cur_line_bboxes = self.get_line_bbox(cur_line_bboxes)
- bboxes.extend(cur_line_bboxes)
- # yield guid, {"id": str(guid), "words": words, "bboxes": bboxes, "ner_tags": ner_tags, "image": image}
- yield guid, {"id": str(guid), "words": words, "bboxes": bboxes, "ner_tags": ner_tags,
- "image": image, "image_path": image_path}
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/data_collator.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/data_collator.py
deleted file mode 100644
index 4232a6660..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/data_collator.py
+++ /dev/null
@@ -1,124 +0,0 @@
-import torch
-from dataclasses import dataclass
-from typing import Any, Dict, List, Optional, Tuple, Union
-
-from transformers import BatchEncoding, PreTrainedTokenizerBase
-from transformers.data.data_collator import (
- DataCollatorMixin,
- _torch_collate_batch,
-)
-from transformers.file_utils import PaddingStrategy
-
-from typing import NewType
-InputDataClass = NewType("InputDataClass", Any)
-
-def pre_calc_rel_mat(segment_ids):
- valid_span = torch.zeros((segment_ids.shape[0], segment_ids.shape[1], segment_ids.shape[1]),
- device=segment_ids.device, dtype=torch.bool)
- for i in range(segment_ids.shape[0]):
- for j in range(segment_ids.shape[1]):
- valid_span[i, j, :] = segment_ids[i, :] == segment_ids[i, j]
-
- return valid_span
-
-@dataclass
-class DataCollatorForKeyValueExtraction(DataCollatorMixin):
- """
- Data collator that will dynamically pad the inputs received, as well as the labels.
- Args:
- tokenizer (:class:`~transformers.PreTrainedTokenizer` or :class:`~transformers.PreTrainedTokenizerFast`):
- The tokenizer used for encoding the data.
- padding (:obj:`bool`, :obj:`str` or :class:`~transformers.file_utils.PaddingStrategy`, `optional`, defaults to :obj:`True`):
- Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
- among:
- * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
- sequence if provided).
- * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the
- maximum acceptable input length for the model if that argument is not provided.
- * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of
- different lengths).
- max_length (:obj:`int`, `optional`):
- Maximum length of the returned list and optionally padding length (see above).
- pad_to_multiple_of (:obj:`int`, `optional`):
- If set will pad the sequence to a multiple of the provided value.
- This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
- 7.5 (Volta).
- label_pad_token_id (:obj:`int`, `optional`, defaults to -100):
- The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions).
- """
-
- tokenizer: PreTrainedTokenizerBase
- padding: Union[bool, str, PaddingStrategy] = True
- max_length: Optional[int] = None
- pad_to_multiple_of: Optional[int] = None
- label_pad_token_id: int = -100
-
- def __call__(self, features):
- label_name = "label" if "label" in features[0].keys() else "labels"
- labels = [feature[label_name] for feature in features] if label_name in features[0].keys() else None
-
- images = None
- if "images" in features[0]:
- images = torch.stack([torch.tensor(d.pop("images")) for d in features])
- IMAGE_LEN = int(images.shape[-1] / 16) * int(images.shape[-1] / 16) + 1
-
- batch = self.tokenizer.pad(
- features,
- padding=self.padding,
- max_length=self.max_length,
- pad_to_multiple_of=self.pad_to_multiple_of,
- # Conversion to tensors will fail if we have labels as they are not of the same length yet.
- return_tensors="pt" if labels is None else None,
- )
-
- if images is not None:
- batch["images"] = images
- batch = {k: torch.tensor(v, dtype=torch.int64) if isinstance(v[0], list) and k == 'attention_mask' else v
- for k, v in batch.items()}
- visual_attention_mask = torch.ones((len(batch['input_ids']), IMAGE_LEN), dtype=torch.long)
- batch["attention_mask"] = torch.cat([batch['attention_mask'], visual_attention_mask], dim=1)
-
- if labels is None:
- return batch
-
- has_bbox_input = "bbox" in features[0]
- has_position_input = "position_ids" in features[0]
- padding_idx=self.tokenizer.pad_token_id
- sequence_length = torch.tensor(batch["input_ids"]).shape[1]
- padding_side = self.tokenizer.padding_side
- if padding_side == "right":
- batch["labels"] = [label + [self.label_pad_token_id] * (sequence_length - len(label)) for label in labels]
- if has_bbox_input:
- batch["bbox"] = [bbox + [[0, 0, 0, 0]] * (sequence_length - len(bbox)) for bbox in batch["bbox"]]
- if has_position_input:
- batch["position_ids"] = [position_id + [padding_idx] * (sequence_length - len(position_id))
- for position_id in batch["position_ids"]]
-
- else:
- batch["labels"] = [[self.label_pad_token_id] * (sequence_length - len(label)) + label for label in labels]
- if has_bbox_input:
- batch["bbox"] = [[[0, 0, 0, 0]] * (sequence_length - len(bbox)) + bbox for bbox in batch["bbox"]]
- if has_position_input:
- batch["position_ids"] = [[padding_idx] * (sequence_length - len(position_id))
- + position_id for position_id in batch["position_ids"]]
-
- if 'segment_ids' in batch:
- assert 'position_ids' in batch
- for i in range(len(batch['segment_ids'])):
- batch['segment_ids'][i] = batch['segment_ids'][i] + [batch['segment_ids'][i][-1] + 1] * (sequence_length - len(batch['segment_ids'][i])) + [
- batch['segment_ids'][i][-1] + 2] * IMAGE_LEN
-
- batch = {k: torch.tensor(v, dtype=torch.int64) if isinstance(v[0], list) else v for k, v in batch.items()}
-
- if 'segment_ids' in batch:
- valid_span = pre_calc_rel_mat(
- segment_ids=batch['segment_ids']
- )
- batch['valid_span'] = valid_span
- del batch['segment_ids']
-
- if images is not None:
- visual_labels = torch.ones((len(batch['input_ids']), IMAGE_LEN), dtype=torch.long) * -100
- batch["labels"] = torch.cat([batch['labels'], visual_labels], dim=1)
-
- return batch
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/funsd.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/funsd.py
deleted file mode 100644
index 9f3404202..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/funsd.py
+++ /dev/null
@@ -1,136 +0,0 @@
-# coding=utf-8
-'''
-Reference: https://huggingface.co/datasets/nielsr/funsd/blob/main/funsd.py
-'''
-import json
-import os
-
-import datasets
-
-from .image_utils import load_image, normalize_bbox
-
-
-logger = datasets.logging.get_logger(__name__)
-
-
-_CITATION = """\
-@article{Jaume2019FUNSDAD,
- title={FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents},
- author={Guillaume Jaume and H. K. Ekenel and J. Thiran},
- journal={2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)},
- year={2019},
- volume={2},
- pages={1-6}
-}
-"""
-
-_DESCRIPTION = """\
-https://guillaumejaume.github.io/FUNSD/
-"""
-
-
-class FunsdConfig(datasets.BuilderConfig):
- """BuilderConfig for FUNSD"""
-
- def __init__(self, **kwargs):
- """BuilderConfig for FUNSD.
-
- Args:
- **kwargs: keyword arguments forwarded to super.
- """
- super(FunsdConfig, self).__init__(**kwargs)
-
-
-class Funsd(datasets.GeneratorBasedBuilder):
- """Conll2003 dataset."""
-
- BUILDER_CONFIGS = [
- FunsdConfig(name="funsd", version=datasets.Version("1.0.0"), description="FUNSD dataset"),
- ]
-
- def _info(self):
- return datasets.DatasetInfo(
- description=_DESCRIPTION,
- features=datasets.Features(
- {
- "id": datasets.Value("string"),
- "tokens": datasets.Sequence(datasets.Value("string")),
- "bboxes": datasets.Sequence(datasets.Sequence(datasets.Value("int64"))),
- "ner_tags": datasets.Sequence(
- datasets.features.ClassLabel(
- names=["O", "B-HEADER", "I-HEADER", "B-QUESTION", "I-QUESTION", "B-ANSWER", "I-ANSWER"]
- )
- ),
- "image": datasets.Array3D(shape=(3, 224, 224), dtype="uint8"),
- "image_path": datasets.Value("string"),
- }
- ),
- supervised_keys=None,
- homepage="https://guillaumejaume.github.io/FUNSD/",
- citation=_CITATION,
- )
-
- def _split_generators(self, dl_manager):
- """Returns SplitGenerators."""
- downloaded_file = dl_manager.download_and_extract("https://guillaumejaume.github.io/FUNSD/dataset.zip")
- return [
- datasets.SplitGenerator(
- name=datasets.Split.TRAIN, gen_kwargs={"filepath": f"{downloaded_file}/dataset/training_data/"}
- ),
- datasets.SplitGenerator(
- name=datasets.Split.TEST, gen_kwargs={"filepath": f"{downloaded_file}/dataset/testing_data/"}
- ),
- ]
-
- def get_line_bbox(self, bboxs):
- x = [bboxs[i][j] for i in range(len(bboxs)) for j in range(0, len(bboxs[i]), 2)]
- y = [bboxs[i][j] for i in range(len(bboxs)) for j in range(1, len(bboxs[i]), 2)]
-
- x0, y0, x1, y1 = min(x), min(y), max(x), max(y)
-
- assert x1 >= x0 and y1 >= y0
- bbox = [[x0, y0, x1, y1] for _ in range(len(bboxs))]
- return bbox
-
- def _generate_examples(self, filepath):
- logger.info("⏳ Generating examples from = %s", filepath)
- ann_dir = os.path.join(filepath, "annotations")
- img_dir = os.path.join(filepath, "images")
- for guid, file in enumerate(sorted(os.listdir(ann_dir))):
- tokens = []
- bboxes = []
- ner_tags = []
-
- file_path = os.path.join(ann_dir, file)
- with open(file_path, "r", encoding="utf8") as f:
- data = json.load(f)
- image_path = os.path.join(img_dir, file)
- image_path = image_path.replace("json", "png")
- image, size = load_image(image_path)
- for item in data["form"]:
- cur_line_bboxes = []
- words, label = item["words"], item["label"]
- words = [w for w in words if w["text"].strip() != ""]
- if len(words) == 0:
- continue
- if label == "other":
- for w in words:
- tokens.append(w["text"])
- ner_tags.append("O")
- cur_line_bboxes.append(normalize_bbox(w["box"], size))
- else:
- tokens.append(words[0]["text"])
- ner_tags.append("B-" + label.upper())
- cur_line_bboxes.append(normalize_bbox(words[0]["box"], size))
- for w in words[1:]:
- tokens.append(w["text"])
- ner_tags.append("I-" + label.upper())
- cur_line_bboxes.append(normalize_bbox(w["box"], size))
- # by default: --segment_level_layout 1
- # if do not want to use segment_level_layout, comment the following line
- cur_line_bboxes = self.get_line_bbox(cur_line_bboxes)
- # box = normalize_bbox(item["box"], size)
- # cur_line_bboxes = [box for _ in range(len(words))]
- bboxes.extend(cur_line_bboxes)
- yield guid, {"id": str(guid), "tokens": tokens, "bboxes": bboxes, "ner_tags": ner_tags,
- "image": image, "image_path": image_path}
\ No newline at end of file
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/image_utils.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/image_utils.py
deleted file mode 100644
index 90a4b3437..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/image_utils.py
+++ /dev/null
@@ -1,284 +0,0 @@
-import torchvision.transforms.functional as F
-import warnings
-import math
-import random
-import numpy as np
-from PIL import Image
-import torch
-
-from detectron2.data.detection_utils import read_image
-from detectron2.data.transforms import ResizeTransform, TransformList
-
-def normalize_bbox(bbox, size):
- return [
- int(1000 * bbox[0] / size[0]),
- int(1000 * bbox[1] / size[1]),
- int(1000 * bbox[2] / size[0]),
- int(1000 * bbox[3] / size[1]),
- ]
-
-
-def load_image(image_path):
- image = read_image(image_path, format="BGR")
- h = image.shape[0]
- w = image.shape[1]
- img_trans = TransformList([ResizeTransform(h=h, w=w, new_h=224, new_w=224)])
- image = torch.tensor(img_trans.apply_image(image).copy()).permute(2, 0, 1) # copy to make it writeable
- return image, (w, h)
-
-
-def crop(image, i, j, h, w, boxes=None):
- cropped_image = F.crop(image, i, j, h, w)
-
- if boxes is not None:
- # Currently we cannot use this case since when some boxes is out of the cropped image,
- # it may be better to drop out these boxes along with their text input (instead of min or clamp)
- # which haven't been implemented here
- max_size = torch.as_tensor([w, h], dtype=torch.float32)
- cropped_boxes = torch.as_tensor(boxes) - torch.as_tensor([j, i, j, i])
- cropped_boxes = torch.min(cropped_boxes.reshape(-1, 2, 2), max_size)
- cropped_boxes = cropped_boxes.clamp(min=0)
- boxes = cropped_boxes.reshape(-1, 4)
-
- return cropped_image, boxes
-
-
-def resize(image, size, interpolation, boxes=None):
- # It seems that we do not need to resize boxes here, since the boxes will be resized to 1000x1000 finally,
- # which is compatible with a square image size of 224x224
- rescaled_image = F.resize(image, size, interpolation)
-
- if boxes is None:
- return rescaled_image, None
-
- ratios = tuple(float(s) / float(s_orig) for s, s_orig in zip(rescaled_image.size, image.size))
- ratio_width, ratio_height = ratios
-
- # boxes = boxes.copy()
- scaled_boxes = boxes * torch.as_tensor([ratio_width, ratio_height, ratio_width, ratio_height])
-
- return rescaled_image, scaled_boxes
-
-
-def clamp(num, min_value, max_value):
- return max(min(num, max_value), min_value)
-
-
-def get_bb(bb, page_size):
- bbs = [float(j) for j in bb]
- xs, ys = [], []
- for i, b in enumerate(bbs):
- if i % 2 == 0:
- xs.append(b)
- else:
- ys.append(b)
- (width, height) = page_size
- return_bb = [
- clamp(min(xs), 0, width - 1),
- clamp(min(ys), 0, height - 1),
- clamp(max(xs), 0, width - 1),
- clamp(max(ys), 0, height - 1),
- ]
- return_bb = [
- int(1000 * return_bb[0] / width),
- int(1000 * return_bb[1] / height),
- int(1000 * return_bb[2] / width),
- int(1000 * return_bb[3] / height),
- ]
- return return_bb
-
-
-class ToNumpy:
-
- def __call__(self, pil_img):
- np_img = np.array(pil_img, dtype=np.uint8)
- if np_img.ndim < 3:
- np_img = np.expand_dims(np_img, axis=-1)
- np_img = np.rollaxis(np_img, 2) # HWC to CHW
- return np_img
-
-
-class ToTensor:
-
- def __init__(self, dtype=torch.float32):
- self.dtype = dtype
-
- def __call__(self, pil_img):
- np_img = np.array(pil_img, dtype=np.uint8)
- if np_img.ndim < 3:
- np_img = np.expand_dims(np_img, axis=-1)
- np_img = np.rollaxis(np_img, 2) # HWC to CHW
- return torch.from_numpy(np_img).to(dtype=self.dtype)
-
-
-_pil_interpolation_to_str = {
- F.InterpolationMode.NEAREST: 'F.InterpolationMode.NEAREST',
- F.InterpolationMode.BILINEAR: 'F.InterpolationMode.BILINEAR',
- F.InterpolationMode.BICUBIC: 'F.InterpolationMode.BICUBIC',
- F.InterpolationMode.LANCZOS: 'F.InterpolationMode.LANCZOS',
- F.InterpolationMode.HAMMING: 'F.InterpolationMode.HAMMING',
- F.InterpolationMode.BOX: 'F.InterpolationMode.BOX',
-}
-
-
-def _pil_interp(method):
- if method == 'bicubic':
- return F.InterpolationMode.BICUBIC
- elif method == 'lanczos':
- return F.InterpolationMode.LANCZOS
- elif method == 'hamming':
- return F.InterpolationMode.HAMMING
- else:
- # default bilinear, do we want to allow nearest?
- return F.InterpolationMode.BILINEAR
-
-
-class Compose:
- """Composes several transforms together. This transform does not support torchscript.
- Please, see the note below.
-
- Args:
- transforms (list of ``Transform`` objects): list of transforms to compose.
-
- Example:
- >>> transforms.Compose([
- >>> transforms.CenterCrop(10),
- >>> transforms.PILToTensor(),
- >>> transforms.ConvertImageDtype(torch.float),
- >>> ])
-
- .. note::
- In order to script the transformations, please use ``torch.nn.Sequential`` as below.
-
- >>> transforms = torch.nn.Sequential(
- >>> transforms.CenterCrop(10),
- >>> transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
- >>> )
- >>> scripted_transforms = torch.jit.script(transforms)
-
- Make sure to use only scriptable transformations, i.e. that work with ``torch.Tensor``, does not require
- `lambda` functions or ``PIL.Image``.
-
- """
-
- def __init__(self, transforms):
- self.transforms = transforms
-
- def __call__(self, img, augmentation=False, box=None):
- for t in self.transforms:
- img = t(img, augmentation, box)
- return img
-
-
-class RandomResizedCropAndInterpolationWithTwoPic:
- """Crop the given PIL Image to random size and aspect ratio with random interpolation.
- A crop of random size (default: of 0.08 to 1.0) of the original size and a random
- aspect ratio (default: of 3/4 to 4/3) of the original aspect ratio is made. This crop
- is finally resized to given size.
- This is popularly used to train the Inception networks.
- Args:
- size: expected output size of each edge
- scale: range of size of the origin size cropped
- ratio: range of aspect ratio of the origin aspect ratio cropped
- interpolation: Default: PIL.Image.BILINEAR
- """
-
- def __init__(self, size, second_size=None, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.),
- interpolation='bilinear', second_interpolation='lanczos'):
- if isinstance(size, tuple):
- self.size = size
- else:
- self.size = (size, size)
- if second_size is not None:
- if isinstance(second_size, tuple):
- self.second_size = second_size
- else:
- self.second_size = (second_size, second_size)
- else:
- self.second_size = None
- if (scale[0] > scale[1]) or (ratio[0] > ratio[1]):
- warnings.warn("range should be of kind (min, max)")
-
- self.interpolation = _pil_interp(interpolation)
- self.second_interpolation = _pil_interp(second_interpolation)
- self.scale = scale
- self.ratio = ratio
-
- @staticmethod
- def get_params(img, scale, ratio):
- """Get parameters for ``crop`` for a random sized crop.
- Args:
- img (PIL Image): Image to be cropped.
- scale (tuple): range of size of the origin size cropped
- ratio (tuple): range of aspect ratio of the origin aspect ratio cropped
- Returns:
- tuple: params (i, j, h, w) to be passed to ``crop`` for a random
- sized crop.
- """
- area = img.size[0] * img.size[1]
-
- for attempt in range(10):
- target_area = random.uniform(*scale) * area
- log_ratio = (math.log(ratio[0]), math.log(ratio[1]))
- aspect_ratio = math.exp(random.uniform(*log_ratio))
-
- w = int(round(math.sqrt(target_area * aspect_ratio)))
- h = int(round(math.sqrt(target_area / aspect_ratio)))
-
- if w <= img.size[0] and h <= img.size[1]:
- i = random.randint(0, img.size[1] - h)
- j = random.randint(0, img.size[0] - w)
- return i, j, h, w
-
- # Fallback to central crop
- in_ratio = img.size[0] / img.size[1]
- if in_ratio < min(ratio):
- w = img.size[0]
- h = int(round(w / min(ratio)))
- elif in_ratio > max(ratio):
- h = img.size[1]
- w = int(round(h * max(ratio)))
- else: # whole image
- w = img.size[0]
- h = img.size[1]
- i = (img.size[1] - h) // 2
- j = (img.size[0] - w) // 2
- return i, j, h, w
-
- def __call__(self, img, augmentation=False, box=None):
- """
- Args:
- img (PIL Image): Image to be cropped and resized.
- Returns:
- PIL Image: Randomly cropped and resized image.
- """
- if augmentation:
- i, j, h, w = self.get_params(img, self.scale, self.ratio)
- img = F.crop(img, i, j, h, w)
- # img, box = crop(img, i, j, h, w, box)
- img = F.resize(img, self.size, self.interpolation)
- second_img = F.resize(img, self.second_size, self.second_interpolation) \
- if self.second_size is not None else None
- return img, second_img
-
- def __repr__(self):
- if isinstance(self.interpolation, (tuple, list)):
- interpolate_str = ' '.join([_pil_interpolation_to_str[x] for x in self.interpolation])
- else:
- interpolate_str = _pil_interpolation_to_str[self.interpolation]
- format_string = self.__class__.__name__ + '(size={0}'.format(self.size)
- format_string += ', scale={0}'.format(tuple(round(s, 4) for s in self.scale))
- format_string += ', ratio={0}'.format(tuple(round(r, 4) for r in self.ratio))
- format_string += ', interpolation={0}'.format(interpolate_str)
- if self.second_size is not None:
- format_string += ', second_size={0}'.format(self.second_size)
- format_string += ', second_interpolation={0}'.format(_pil_interpolation_to_str[self.second_interpolation])
- format_string += ')'
- return format_string
-
-
-def pil_loader(path: str) -> Image.Image:
- # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
- with open(path, 'rb') as f:
- img = Image.open(f)
- return img.convert('RGB')
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/xfund.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/xfund.py
deleted file mode 100644
index 7749ba5dd..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/data/xfund.py
+++ /dev/null
@@ -1,213 +0,0 @@
-import os
-import json
-
-import torch
-from torch.utils.data.dataset import Dataset
-from torchvision import transforms
-from PIL import Image
-
-from .image_utils import Compose, RandomResizedCropAndInterpolationWithTwoPic
-
-XFund_label2ids = {
- "O":0,
- 'B-HEADER':1,
- 'I-HEADER':2,
- 'B-QUESTION':3,
- 'I-QUESTION':4,
- 'B-ANSWER':5,
- 'I-ANSWER':6,
-}
-
-class xfund_dataset(Dataset):
- def box_norm(self, box, width, height):
- def clip(min_num, num, max_num):
- return min(max(num, min_num), max_num)
-
- x0, y0, x1, y1 = box
- x0 = clip(0, int((x0 / width) * 1000), 1000)
- y0 = clip(0, int((y0 / height) * 1000), 1000)
- x1 = clip(0, int((x1 / width) * 1000), 1000)
- y1 = clip(0, int((y1 / height) * 1000), 1000)
- assert x1 >= x0
- assert y1 >= y0
- return [x0, y0, x1, y1]
-
- def get_segment_ids(self, bboxs):
- segment_ids = []
- for i in range(len(bboxs)):
- if i == 0:
- segment_ids.append(0)
- else:
- if bboxs[i - 1] == bboxs[i]:
- segment_ids.append(segment_ids[-1])
- else:
- segment_ids.append(segment_ids[-1] + 1)
- return segment_ids
-
- def get_position_ids(self, segment_ids):
- position_ids = []
- for i in range(len(segment_ids)):
- if i == 0:
- position_ids.append(2)
- else:
- if segment_ids[i] == segment_ids[i - 1]:
- position_ids.append(position_ids[-1] + 1)
- else:
- position_ids.append(2)
- return position_ids
-
- def load_data(
- self,
- data_file,
- ):
- # re-org data format
- total_data = {"id": [], "lines": [], "bboxes": [], "ner_tags": [], "image_path": []}
- for i in range(len(data_file['documents'])):
- width, height = data_file['documents'][i]['img']['width'], data_file['documents'][i]['img'][
- 'height']
-
- cur_doc_lines, cur_doc_bboxes, cur_doc_ner_tags, cur_doc_image_path = [], [], [], []
- for j in range(len(data_file['documents'][i]['document'])):
- cur_item = data_file['documents'][i]['document'][j]
- cur_doc_lines.append(cur_item['text'])
- cur_doc_bboxes.append(self.box_norm(cur_item['box'], width=width, height=height))
- cur_doc_ner_tags.append(cur_item['label'])
- total_data['id'] += [len(total_data['id'])]
- total_data['lines'] += [cur_doc_lines]
- total_data['bboxes'] += [cur_doc_bboxes]
- total_data['ner_tags'] += [cur_doc_ner_tags]
- total_data['image_path'] += [data_file['documents'][i]['img']['fname']]
-
- # tokenize text and get bbox/label
- total_input_ids, total_bboxs, total_label_ids = [], [], []
- for i in range(len(total_data['lines'])):
- cur_doc_input_ids, cur_doc_bboxs, cur_doc_labels = [], [], []
- for j in range(len(total_data['lines'][i])):
- cur_input_ids = self.tokenizer(total_data['lines'][i][j], truncation=False, add_special_tokens=False, return_attention_mask=False)['input_ids']
- if len(cur_input_ids) == 0: continue
-
- cur_label = total_data['ner_tags'][i][j].upper()
- if cur_label == 'OTHER':
- cur_labels = ["O"] * len(cur_input_ids)
- for k in range(len(cur_labels)):
- cur_labels[k] = self.label2ids[cur_labels[k]]
- else:
- cur_labels = [cur_label] * len(cur_input_ids)
- cur_labels[0] = self.label2ids['B-' + cur_labels[0]]
- for k in range(1, len(cur_labels)):
- cur_labels[k] = self.label2ids['I-' + cur_labels[k]]
- assert len(cur_input_ids) == len([total_data['bboxes'][i][j]] * len(cur_input_ids)) == len(cur_labels)
- cur_doc_input_ids += cur_input_ids
- cur_doc_bboxs += [total_data['bboxes'][i][j]] * len(cur_input_ids)
- cur_doc_labels += cur_labels
- assert len(cur_doc_input_ids) == len(cur_doc_bboxs) == len(cur_doc_labels)
- assert len(cur_doc_input_ids) > 0
-
- total_input_ids.append(cur_doc_input_ids)
- total_bboxs.append(cur_doc_bboxs)
- total_label_ids.append(cur_doc_labels)
- assert len(total_input_ids) == len(total_bboxs) == len(total_label_ids)
-
- # split text to several slices because of over-length
- input_ids, bboxs, labels = [], [], []
- segment_ids, position_ids = [], []
- image_path = []
- for i in range(len(total_input_ids)):
- start = 0
- cur_iter = 0
- while start < len(total_input_ids[i]):
- end = min(start + 510, len(total_input_ids[i]))
-
- input_ids.append([self.tokenizer.cls_token_id] + total_input_ids[i][start: end] + [self.tokenizer.sep_token_id])
- bboxs.append([[0, 0, 0, 0]] + total_bboxs[i][start: end] + [[1000, 1000, 1000, 1000]])
- labels.append([-100] + total_label_ids[i][start: end] + [-100])
-
- cur_segment_ids = self.get_segment_ids(bboxs[-1])
- cur_position_ids = self.get_position_ids(cur_segment_ids)
- segment_ids.append(cur_segment_ids)
- position_ids.append(cur_position_ids)
- image_path.append(os.path.join(self.args.data_dir, "images", total_data['image_path'][i]))
-
- start = end
- cur_iter += 1
-
- assert len(input_ids) == len(bboxs) == len(labels) == len(segment_ids) == len(position_ids)
- assert len(segment_ids) == len(image_path)
-
- res = {
- 'input_ids': input_ids,
- 'bbox': bboxs,
- 'labels': labels,
- 'segment_ids': segment_ids,
- 'position_ids': position_ids,
- 'image_path': image_path,
- }
- return res
-
- def __init__(
- self,
- args,
- tokenizer,
- mode
- ):
- self.args = args
- self.mode = mode
- self.cur_la = args.language
- self.tokenizer = tokenizer
- self.label2ids = XFund_label2ids
-
-
- self.common_transform = Compose([
- RandomResizedCropAndInterpolationWithTwoPic(
- size=args.input_size, interpolation=args.train_interpolation,
- ),
- ])
-
- self.patch_transform = transforms.Compose([
- transforms.ToTensor(),
- transforms.Normalize(
- mean=torch.tensor((0.5, 0.5, 0.5)),
- std=torch.tensor((0.5, 0.5, 0.5)))
- ])
-
- data_file = json.load(
- open(os.path.join(args.data_dir, "{}.{}.json".format(self.cur_la, 'train' if mode == 'train' else 'val')),
- 'r'))
-
- self.feature = self.load_data(data_file)
-
- def __len__(self):
- return len(self.feature['input_ids'])
-
- def __getitem__(self, index):
- input_ids = self.feature["input_ids"][index]
-
- # attention_mask = self.feature["attention_mask"][index]
- attention_mask = [1] * len(input_ids)
- labels = self.feature["labels"][index]
- bbox = self.feature["bbox"][index]
- segment_ids = self.feature['segment_ids'][index]
- position_ids = self.feature['position_ids'][index]
-
- img = pil_loader(self.feature['image_path'][index])
- for_patches, _ = self.common_transform(img, augmentation=False)
- patch = self.patch_transform(for_patches)
-
- assert len(input_ids) == len(attention_mask) == len(labels) == len(bbox) == len(segment_ids)
-
- res = {
- "input_ids": input_ids,
- "attention_mask": attention_mask,
- "labels": labels,
- "bbox": bbox,
- "segment_ids": segment_ids,
- "position_ids": position_ids,
- "images": patch,
- }
- return res
-
-def pil_loader(path: str) -> Image.Image:
- # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
- with open(path, 'rb') as f:
- img = Image.open(f)
- return img.convert('RGB')
\ No newline at end of file
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/__init__.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/__init__.py
deleted file mode 100644
index 0b3100eff..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/__init__.py
+++ /dev/null
@@ -1,7 +0,0 @@
-from .layoutlmv3 import (
- LayoutLMv3Config,
- LayoutLMv3ForTokenClassification,
- LayoutLMv3ForQuestionAnswering,
- LayoutLMv3ForSequenceClassification,
- LayoutLMv3Tokenizer,
-)
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/__init__.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/__init__.py
deleted file mode 100644
index e06a24b0c..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/__init__.py
+++ /dev/null
@@ -1,24 +0,0 @@
-from transformers import AutoConfig, AutoModel, AutoModelForTokenClassification, \
- AutoModelForQuestionAnswering, AutoModelForSequenceClassification, AutoTokenizer
-from transformers.convert_slow_tokenizer import SLOW_TO_FAST_CONVERTERS, RobertaConverter
-
-from .configuration_layoutlmv3 import LayoutLMv3Config
-from .modeling_layoutlmv3 import (
- LayoutLMv3ForTokenClassification,
- LayoutLMv3ForQuestionAnswering,
- LayoutLMv3ForSequenceClassification,
- LayoutLMv3Model,
-)
-from .tokenization_layoutlmv3 import LayoutLMv3Tokenizer
-from .tokenization_layoutlmv3_fast import LayoutLMv3TokenizerFast
-
-
-#AutoConfig.register("layoutlmv3", LayoutLMv3Config)
-#AutoModel.register(LayoutLMv3Config, LayoutLMv3Model)
-#AutoModelForTokenClassification.register(LayoutLMv3Config, LayoutLMv3ForTokenClassification)
-#AutoModelForQuestionAnswering.register(LayoutLMv3Config, LayoutLMv3ForQuestionAnswering)
-#AutoModelForSequenceClassification.register(LayoutLMv3Config, LayoutLMv3ForSequenceClassification)
-#AutoTokenizer.register(
-# LayoutLMv3Config, slow_tokenizer_class=LayoutLMv3Tokenizer, fast_tokenizer_class=LayoutLMv3TokenizerFast
-#)
-SLOW_TO_FAST_CONVERTERS.update({"LayoutLMv3Tokenizer": RobertaConverter})
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/configuration_layoutlmv3.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/configuration_layoutlmv3.py
deleted file mode 100644
index d2c7b4d71..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/configuration_layoutlmv3.py
+++ /dev/null
@@ -1,60 +0,0 @@
-# coding=utf-8
-from transformers.models.bert.configuration_bert import BertConfig
-from transformers.utils import logging
-
-
-logger = logging.get_logger(__name__)
-
-LAYOUTLMV3_PRETRAINED_CONFIG_ARCHIVE_MAP = {
- "layoutlmv3-base": "https://huggingface.co/microsoft/layoutlmv3-base/resolve/main/config.json",
- "layoutlmv3-large": "https://huggingface.co/microsoft/layoutlmv3-large/resolve/main/config.json",
- # See all LayoutLMv3 models at https://huggingface.co/models?filter=layoutlmv3
-}
-
-
-class LayoutLMv3Config(BertConfig):
- model_type = "layoutlmv3"
-
- def __init__(
- self,
- pad_token_id=1,
- bos_token_id=0,
- eos_token_id=2,
- max_2d_position_embeddings=1024,
- coordinate_size=None,
- shape_size=None,
- has_relative_attention_bias=False,
- rel_pos_bins=32,
- max_rel_pos=128,
- has_spatial_attention_bias=False,
- rel_2d_pos_bins=64,
- max_rel_2d_pos=256,
- visual_embed=True,
- mim=False,
- wpa_task=False,
- discrete_vae_weight_path='',
- discrete_vae_type='dall-e',
- input_size=224,
- second_input_size=112,
- device='cuda',
- **kwargs
- ):
- """Constructs RobertaConfig."""
- super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
- self.max_2d_position_embeddings = max_2d_position_embeddings
- self.coordinate_size = coordinate_size
- self.shape_size = shape_size
- self.has_relative_attention_bias = has_relative_attention_bias
- self.rel_pos_bins = rel_pos_bins
- self.max_rel_pos = max_rel_pos
- self.has_spatial_attention_bias = has_spatial_attention_bias
- self.rel_2d_pos_bins = rel_2d_pos_bins
- self.max_rel_2d_pos = max_rel_2d_pos
- self.visual_embed = visual_embed
- self.mim = mim
- self.wpa_task = wpa_task
- self.discrete_vae_weight_path = discrete_vae_weight_path
- self.discrete_vae_type = discrete_vae_type
- self.input_size = input_size
- self.second_input_size = second_input_size
- self.device = device
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py
deleted file mode 100644
index 113eb8eb1..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/modeling_layoutlmv3.py
+++ /dev/null
@@ -1,1282 +0,0 @@
-# coding=utf-8
-# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
-# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""PyTorch LayoutLMv3 model. """
-import math
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-import torch.utils.checkpoint
-from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
-
-from transformers import apply_chunking_to_forward
-from transformers.modeling_outputs import (
- BaseModelOutputWithPastAndCrossAttentions,
- BaseModelOutputWithPoolingAndCrossAttentions,
- MaskedLMOutput,
- TokenClassifierOutput,
- QuestionAnsweringModelOutput,
- SequenceClassifierOutput,
-)
-from transformers.modeling_utils import PreTrainedModel, find_pruneable_heads_and_indices, prune_linear_layer
-from transformers.models.roberta.modeling_roberta import (
- RobertaIntermediate,
- RobertaLMHead,
- RobertaOutput,
- RobertaSelfOutput,
-)
-from transformers.utils import logging
-
-from .configuration_layoutlmv3 import LayoutLMv3Config
-from timm.models.layers import to_2tuple
-
-
-logger = logging.get_logger(__name__)
-
-
-class PatchEmbed(nn.Module):
- """ Image to Patch Embedding
- """
- def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768):
- super().__init__()
- img_size = to_2tuple(img_size)
- patch_size = to_2tuple(patch_size)
- self.patch_shape = (img_size[0] // patch_size[0], img_size[1] // patch_size[1])
- self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)
- # The following variables are used in detection mycheckpointer.py
- self.num_patches = (img_size[1] // patch_size[1]) * (img_size[0] // patch_size[0])
- self.num_patches_w = self.patch_shape[0]
- self.num_patches_h = self.patch_shape[1]
-
- def forward(self, x, position_embedding=None):
- x = self.proj(x)
-
- if position_embedding is not None:
- # interpolate the position embedding to the corresponding size
- position_embedding = position_embedding.view(1, self.patch_shape[0], self.patch_shape[1], -1).permute(0, 3, 1, 2)
- Hp, Wp = x.shape[2], x.shape[3]
- position_embedding = F.interpolate(position_embedding, size=(Hp, Wp), mode='bicubic')
- x = x + position_embedding
-
- x = x.flatten(2).transpose(1, 2)
- return x
-
-class LayoutLMv3Embeddings(nn.Module):
- """
- Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.
- """
-
- # Copied from transformers.models.bert.modeling_bert.BertEmbeddings.__init__
- def __init__(self, config):
- super().__init__()
- self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
- self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
-
- self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
- self.dropout = nn.Dropout(config.hidden_dropout_prob)
-
- # position_ids (1, len position emb) is contiguous in memory and exported when serialized
- self.register_buffer("position_ids", torch.arange(config.max_position_embeddings).expand((1, -1)))
-
- # End copy
- self.padding_idx = config.pad_token_id
- self.position_embeddings = nn.Embedding(
- config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
- )
-
- self.x_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.coordinate_size)
- self.y_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.coordinate_size)
- self.h_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.shape_size)
- self.w_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.shape_size)
-
- def _calc_spatial_position_embeddings(self, bbox):
- try:
- assert torch.all(0 <= bbox) and torch.all(bbox <= 1023)
- left_position_embeddings = self.x_position_embeddings(bbox[:, :, 0])
- upper_position_embeddings = self.y_position_embeddings(bbox[:, :, 1])
- right_position_embeddings = self.x_position_embeddings(bbox[:, :, 2])
- lower_position_embeddings = self.y_position_embeddings(bbox[:, :, 3])
- except IndexError as e:
- raise IndexError("The :obj:`bbox` coordinate values should be within 0-1000 range.") from e
-
- h_position_embeddings = self.h_position_embeddings(torch.clip(bbox[:, :, 3] - bbox[:, :, 1], 0, 1023))
- w_position_embeddings = self.w_position_embeddings(torch.clip(bbox[:, :, 2] - bbox[:, :, 0], 0, 1023))
-
- # below is the difference between LayoutLMEmbeddingsV2 (torch.cat) and LayoutLMEmbeddingsV1 (add)
- spatial_position_embeddings = torch.cat(
- [
- left_position_embeddings,
- upper_position_embeddings,
- right_position_embeddings,
- lower_position_embeddings,
- h_position_embeddings,
- w_position_embeddings,
- ],
- dim=-1,
- )
- return spatial_position_embeddings
-
- def create_position_ids_from_input_ids(self, input_ids, padding_idx, past_key_values_length=0):
- """
- Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols
- are ignored. This is modified from fairseq's `utils.make_positions`.
-
- Args:
- x: torch.Tensor x:
-
- Returns: torch.Tensor
- """
- # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
- mask = input_ids.ne(padding_idx).int()
- incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask
- return incremental_indices.long() + padding_idx
-
- def forward(
- self,
- input_ids=None,
- bbox=None,
- token_type_ids=None,
- position_ids=None,
- inputs_embeds=None,
- past_key_values_length=0,
- ):
- if position_ids is None:
- if input_ids is not None:
- # Create the position ids from the input token ids. Any padded tokens remain padded.
- position_ids = self.create_position_ids_from_input_ids(
- input_ids, self.padding_idx, past_key_values_length).to(input_ids.device)
- else:
- position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)
-
- if input_ids is not None:
- input_shape = input_ids.size()
- else:
- input_shape = inputs_embeds.size()[:-1]
-
- if token_type_ids is None:
- token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=self.position_ids.device)
-
- if inputs_embeds is None:
- inputs_embeds = self.word_embeddings(input_ids)
- token_type_embeddings = self.token_type_embeddings(token_type_ids)
-
- embeddings = inputs_embeds + token_type_embeddings
- position_embeddings = self.position_embeddings(position_ids)
- embeddings += position_embeddings
-
- spatial_position_embeddings = self._calc_spatial_position_embeddings(bbox)
-
- embeddings = embeddings + spatial_position_embeddings
-
- embeddings = self.LayerNorm(embeddings)
- embeddings = self.dropout(embeddings)
- return embeddings
-
- def create_position_ids_from_inputs_embeds(self, inputs_embeds):
- """
- We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.
-
- Args:
- inputs_embeds: torch.Tensor≈
-
- Returns: torch.Tensor
- """
- input_shape = inputs_embeds.size()[:-1]
- sequence_length = input_shape[1]
-
- position_ids = torch.arange(
- self.padding_idx + 1, sequence_length + self.padding_idx + 1, dtype=torch.long, device=inputs_embeds.device
- )
- return position_ids.unsqueeze(0).expand(input_shape)
-
-
-class LayoutLMv3PreTrainedModel(PreTrainedModel):
- """
- An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
- models.
- """
-
- config_class = LayoutLMv3Config
- base_model_prefix = "layoutlmv3"
-
- # Copied from transformers.models.bert.modeling_bert.BertPreTrainedModel._init_weights
- def _init_weights(self, module):
- """Initialize the weights"""
- if isinstance(module, nn.Linear):
- # Slightly different from the TF version which uses truncated_normal for initialization
- # cf https://github.com/pytorch/pytorch/pull/5617
- module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
- if module.bias is not None:
- module.bias.data.zero_()
- elif isinstance(module, nn.Embedding):
- module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
- if module.padding_idx is not None:
- module.weight.data[module.padding_idx].zero_()
- elif isinstance(module, nn.LayerNorm):
- module.bias.data.zero_()
- module.weight.data.fill_(1.0)
-
-
-class LayoutLMv3SelfAttention(nn.Module):
- def __init__(self, config):
- super().__init__()
- if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
- raise ValueError(
- f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
- f"heads ({config.num_attention_heads})"
- )
-
- self.num_attention_heads = config.num_attention_heads
- self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
- self.all_head_size = self.num_attention_heads * self.attention_head_size
-
- self.query = nn.Linear(config.hidden_size, self.all_head_size)
- self.key = nn.Linear(config.hidden_size, self.all_head_size)
- self.value = nn.Linear(config.hidden_size, self.all_head_size)
-
- self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
- self.has_relative_attention_bias = config.has_relative_attention_bias
- self.has_spatial_attention_bias = config.has_spatial_attention_bias
-
- def transpose_for_scores(self, x):
- new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
- x = x.view(*new_x_shape)
- return x.permute(0, 2, 1, 3)
-
- def cogview_attn(self, attention_scores, alpha=32):
- '''
- https://arxiv.org/pdf/2105.13290.pdf
- Section 2.4 Stabilization of training: Precision Bottleneck Relaxation (PB-Relax).
- A replacement of the original nn.Softmax(dim=-1)(attention_scores)
- Seems the new attention_probs will result in a slower speed and a little bias
- Can use torch.allclose(standard_attention_probs, cogview_attention_probs, atol=1e-08) for comparison
- The smaller atol (e.g., 1e-08), the better.
- '''
- scaled_attention_scores = attention_scores / alpha
- max_value = scaled_attention_scores.amax(dim=(-1)).unsqueeze(-1)
- # max_value = scaled_attention_scores.amax(dim=(-2, -1)).unsqueeze(-1).unsqueeze(-1)
- new_attention_scores = (scaled_attention_scores - max_value) * alpha
- return nn.Softmax(dim=-1)(new_attention_scores)
-
- def forward(
- self,
- hidden_states,
- attention_mask=None,
- head_mask=None,
- encoder_hidden_states=None,
- encoder_attention_mask=None,
- past_key_value=None,
- output_attentions=False,
- rel_pos=None,
- rel_2d_pos=None,
- ):
- mixed_query_layer = self.query(hidden_states)
-
- # If this is instantiated as a cross-attention module, the keys
- # and values come from an encoder; the attention mask needs to be
- # such that the encoder's padding tokens are not attended to.
- is_cross_attention = encoder_hidden_states is not None
-
- if is_cross_attention and past_key_value is not None:
- # reuse k,v, cross_attentions
- key_layer = past_key_value[0]
- value_layer = past_key_value[1]
- attention_mask = encoder_attention_mask
- elif is_cross_attention:
- key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
- value_layer = self.transpose_for_scores(self.value(encoder_hidden_states))
- attention_mask = encoder_attention_mask
- elif past_key_value is not None:
- key_layer = self.transpose_for_scores(self.key(hidden_states))
- value_layer = self.transpose_for_scores(self.value(hidden_states))
- key_layer = torch.cat([past_key_value[0], key_layer], dim=2)
- value_layer = torch.cat([past_key_value[1], value_layer], dim=2)
- else:
- key_layer = self.transpose_for_scores(self.key(hidden_states))
- value_layer = self.transpose_for_scores(self.value(hidden_states))
-
- query_layer = self.transpose_for_scores(mixed_query_layer)
-
- # Take the dot product between "query" and "key" to get the raw attention scores.
- # The attention scores QT K/√d could be significantly larger than input elements, and result in overflow.
- # Changing the computational order into QT(K/√d) alleviates the problem. (https://arxiv.org/pdf/2105.13290.pdf)
- attention_scores = torch.matmul(query_layer / math.sqrt(self.attention_head_size), key_layer.transpose(-1, -2))
-
- if self.has_relative_attention_bias and self.has_spatial_attention_bias:
- attention_scores += (rel_pos + rel_2d_pos) / math.sqrt(self.attention_head_size)
- elif self.has_relative_attention_bias:
- attention_scores += rel_pos / math.sqrt(self.attention_head_size)
-
- # if self.has_relative_attention_bias:
- # attention_scores += rel_pos
- # if self.has_spatial_attention_bias:
- # attention_scores += rel_2d_pos
-
- # attention_scores = attention_scores / math.sqrt(self.attention_head_size)
- if attention_mask is not None:
- # Apply the attention mask is (precomputed for all layers in RobertaModel forward() function)
- attention_scores = attention_scores + attention_mask
-
- # Normalize the attention scores to probabilities.
- # attention_probs = nn.Softmax(dim=-1)(attention_scores) # comment the line below and use this line for speedup
- attention_probs = self.cogview_attn(attention_scores) # to stablize training
- # assert torch.allclose(attention_probs, nn.Softmax(dim=-1)(attention_scores), atol=1e-8)
-
- # This is actually dropping out entire tokens to attend to, which might
- # seem a bit unusual, but is taken from the original Transformer paper.
- attention_probs = self.dropout(attention_probs)
-
- # Mask heads if we want to
- if head_mask is not None:
- attention_probs = attention_probs * head_mask
-
- context_layer = torch.matmul(attention_probs, value_layer)
-
- context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
- new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
- context_layer = context_layer.view(*new_context_layer_shape)
-
- outputs = (context_layer, attention_probs) if output_attentions else (context_layer,)
-
- return outputs
-
-
-class LayoutLMv3Attention(nn.Module):
- def __init__(self, config):
- super().__init__()
- self.self = LayoutLMv3SelfAttention(config)
- self.output = RobertaSelfOutput(config)
- self.pruned_heads = set()
-
- def prune_heads(self, heads):
- if len(heads) == 0:
- return
- heads, index = find_pruneable_heads_and_indices(
- heads, self.self.num_attention_heads, self.self.attention_head_size, self.pruned_heads
- )
-
- # Prune linear layers
- self.self.query = prune_linear_layer(self.self.query, index)
- self.self.key = prune_linear_layer(self.self.key, index)
- self.self.value = prune_linear_layer(self.self.value, index)
- self.output.dense = prune_linear_layer(self.output.dense, index, dim=1)
-
- # Update hyper params and store pruned heads
- self.self.num_attention_heads = self.self.num_attention_heads - len(heads)
- self.self.all_head_size = self.self.attention_head_size * self.self.num_attention_heads
- self.pruned_heads = self.pruned_heads.union(heads)
-
- def forward(
- self,
- hidden_states,
- attention_mask=None,
- head_mask=None,
- encoder_hidden_states=None,
- encoder_attention_mask=None,
- past_key_value=None,
- output_attentions=False,
- rel_pos=None,
- rel_2d_pos=None,
- ):
- self_outputs = self.self(
- hidden_states,
- attention_mask,
- head_mask,
- encoder_hidden_states,
- encoder_attention_mask,
- past_key_value,
- output_attentions,
- rel_pos=rel_pos,
- rel_2d_pos=rel_2d_pos,
- )
- attention_output = self.output(self_outputs[0], hidden_states)
- outputs = (attention_output,) + self_outputs[1:] # add attentions if we output them
- return outputs
-
-
-class LayoutLMv3Layer(nn.Module):
- def __init__(self, config):
- super().__init__()
- self.chunk_size_feed_forward = config.chunk_size_feed_forward
- self.seq_len_dim = 1
- self.attention = LayoutLMv3Attention(config)
- assert not config.is_decoder and not config.add_cross_attention, \
- "This version do not support decoder. Please refer to RoBERTa for implementation of is_decoder."
- self.intermediate = RobertaIntermediate(config)
- self.output = RobertaOutput(config)
-
- def forward(
- self,
- hidden_states,
- attention_mask=None,
- head_mask=None,
- encoder_hidden_states=None,
- encoder_attention_mask=None,
- past_key_value=None,
- output_attentions=False,
- rel_pos=None,
- rel_2d_pos=None,
- ):
- # decoder uni-directional self-attention cached key/values tuple is at positions 1,2
- self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None
- self_attention_outputs = self.attention(
- hidden_states,
- attention_mask,
- head_mask,
- output_attentions=output_attentions,
- past_key_value=self_attn_past_key_value,
- rel_pos=rel_pos,
- rel_2d_pos=rel_2d_pos,
- )
- attention_output = self_attention_outputs[0]
-
- outputs = self_attention_outputs[1:] # add self attentions if we output attention weights
-
- layer_output = apply_chunking_to_forward(
- self.feed_forward_chunk, self.chunk_size_feed_forward, self.seq_len_dim, attention_output
- )
- outputs = (layer_output,) + outputs
-
- return outputs
-
- def feed_forward_chunk(self, attention_output):
- intermediate_output = self.intermediate(attention_output)
- layer_output = self.output(intermediate_output, attention_output)
- return layer_output
-
-
-class LayoutLMv3Encoder(nn.Module):
- def __init__(self, config, detection=False, out_features=None):
- super().__init__()
- self.config = config
- self.detection = detection
- self.layer = nn.ModuleList([LayoutLMv3Layer(config) for _ in range(config.num_hidden_layers)])
- self.gradient_checkpointing = False
-
- self.has_relative_attention_bias = config.has_relative_attention_bias
- self.has_spatial_attention_bias = config.has_spatial_attention_bias
-
- if self.has_relative_attention_bias:
- self.rel_pos_bins = config.rel_pos_bins
- self.max_rel_pos = config.max_rel_pos
- self.rel_pos_onehot_size = config.rel_pos_bins
- self.rel_pos_bias = nn.Linear(self.rel_pos_onehot_size, config.num_attention_heads, bias=False)
-
- if self.has_spatial_attention_bias:
- self.max_rel_2d_pos = config.max_rel_2d_pos
- self.rel_2d_pos_bins = config.rel_2d_pos_bins
- self.rel_2d_pos_onehot_size = config.rel_2d_pos_bins
- self.rel_pos_x_bias = nn.Linear(self.rel_2d_pos_onehot_size, config.num_attention_heads, bias=False)
- self.rel_pos_y_bias = nn.Linear(self.rel_2d_pos_onehot_size, config.num_attention_heads, bias=False)
-
- if self.detection:
- self.gradient_checkpointing = True
- embed_dim = self.config.hidden_size
- self.out_features = out_features
- self.out_indices = [int(name[5:]) for name in out_features]
- self.fpn1 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- # nn.SyncBatchNorm(embed_dim),
- nn.BatchNorm2d(embed_dim),
- nn.GELU(),
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn2 = nn.Sequential(
- nn.ConvTranspose2d(embed_dim, embed_dim, kernel_size=2, stride=2),
- )
-
- self.fpn3 = nn.Identity()
-
- self.fpn4 = nn.MaxPool2d(kernel_size=2, stride=2)
- self.ops = [self.fpn1, self.fpn2, self.fpn3, self.fpn4]
-
- def relative_position_bucket(self, relative_position, bidirectional=True, num_buckets=32, max_distance=128):
- ret = 0
- if bidirectional:
- num_buckets //= 2
- ret += (relative_position > 0).long() * num_buckets
- n = torch.abs(relative_position)
- else:
- n = torch.max(-relative_position, torch.zeros_like(relative_position))
- # now n is in the range [0, inf)
-
- # half of the buckets are for exact increments in positions
- max_exact = num_buckets // 2
- is_small = n < max_exact
-
- # The other half of the buckets are for logarithmically bigger bins in positions up to max_distance
- val_if_large = max_exact + (
- torch.log(n.float() / max_exact) / math.log(max_distance / max_exact) * (num_buckets - max_exact)
- ).to(torch.long)
- val_if_large = torch.min(val_if_large, torch.full_like(val_if_large, num_buckets - 1))
-
- ret += torch.where(is_small, n, val_if_large)
- return ret
-
- def _cal_1d_pos_emb(self, hidden_states, position_ids, valid_span):
- VISUAL_NUM = 196 + 1
-
- rel_pos_mat = position_ids.unsqueeze(-2) - position_ids.unsqueeze(-1)
-
- if valid_span is not None:
- # for the text part, if two words are not in the same line,
- # set their distance to the max value (position_ids.shape[-1])
- rel_pos_mat[(rel_pos_mat > 0) & (valid_span == False)] = position_ids.shape[1]
- rel_pos_mat[(rel_pos_mat < 0) & (valid_span == False)] = -position_ids.shape[1]
-
- # image-text, minimum distance
- rel_pos_mat[:, -VISUAL_NUM:, :-VISUAL_NUM] = 0
- rel_pos_mat[:, :-VISUAL_NUM, -VISUAL_NUM:] = 0
-
- rel_pos = self.relative_position_bucket(
- rel_pos_mat,
- num_buckets=self.rel_pos_bins,
- max_distance=self.max_rel_pos,
- )
- rel_pos = F.one_hot(rel_pos, num_classes=self.rel_pos_onehot_size).type_as(hidden_states)
- rel_pos = self.rel_pos_bias(rel_pos).permute(0, 3, 1, 2)
- rel_pos = rel_pos.contiguous()
- return rel_pos
-
- def _cal_2d_pos_emb(self, hidden_states, bbox):
- position_coord_x = bbox[:, :, 0]
- position_coord_y = bbox[:, :, 3]
- rel_pos_x_2d_mat = position_coord_x.unsqueeze(-2) - position_coord_x.unsqueeze(-1)
- rel_pos_y_2d_mat = position_coord_y.unsqueeze(-2) - position_coord_y.unsqueeze(-1)
- rel_pos_x = self.relative_position_bucket(
- rel_pos_x_2d_mat,
- num_buckets=self.rel_2d_pos_bins,
- max_distance=self.max_rel_2d_pos,
- )
- rel_pos_y = self.relative_position_bucket(
- rel_pos_y_2d_mat,
- num_buckets=self.rel_2d_pos_bins,
- max_distance=self.max_rel_2d_pos,
- )
- rel_pos_x = F.one_hot(rel_pos_x, num_classes=self.rel_2d_pos_onehot_size).type_as(hidden_states)
- rel_pos_y = F.one_hot(rel_pos_y, num_classes=self.rel_2d_pos_onehot_size).type_as(hidden_states)
- rel_pos_x = self.rel_pos_x_bias(rel_pos_x).permute(0, 3, 1, 2)
- rel_pos_y = self.rel_pos_y_bias(rel_pos_y).permute(0, 3, 1, 2)
- rel_pos_x = rel_pos_x.contiguous()
- rel_pos_y = rel_pos_y.contiguous()
- rel_2d_pos = rel_pos_x + rel_pos_y
- return rel_2d_pos
-
- def forward(
- self,
- hidden_states,
- bbox=None,
- attention_mask=None,
- head_mask=None,
- encoder_hidden_states=None,
- encoder_attention_mask=None,
- past_key_values=None,
- use_cache=None,
- output_attentions=False,
- output_hidden_states=False,
- return_dict=True,
- position_ids=None,
- Hp=None,
- Wp=None,
- valid_span=None,
- ):
- all_hidden_states = () if output_hidden_states else None
- all_self_attentions = () if output_attentions else None
- all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None
-
- next_decoder_cache = () if use_cache else None
-
- rel_pos = self._cal_1d_pos_emb(hidden_states, position_ids, valid_span) if self.has_relative_attention_bias else None
- rel_2d_pos = self._cal_2d_pos_emb(hidden_states, bbox) if self.has_spatial_attention_bias else None
-
- if self.detection:
- feat_out = {}
- j = 0
-
- for i, layer_module in enumerate(self.layer):
- if output_hidden_states:
- all_hidden_states = all_hidden_states + (hidden_states,)
-
- layer_head_mask = head_mask[i] if head_mask is not None else None
- past_key_value = past_key_values[i] if past_key_values is not None else None
-
- if self.gradient_checkpointing and self.training:
-
- if use_cache:
- logger.warning(
- "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
- )
- use_cache = False
-
- def create_custom_forward(module):
- def custom_forward(*inputs):
- return module(*inputs)
- # return module(*inputs, past_key_value, output_attentions, rel_pos, rel_2d_pos)
- # The above line will cause error:
- # RuntimeError: Trying to backward through the graph a second time
- # (or directly access saved tensors after they have already been freed).
- return custom_forward
-
- layer_outputs = torch.utils.checkpoint.checkpoint(
- create_custom_forward(layer_module),
- hidden_states,
- attention_mask,
- layer_head_mask,
- encoder_hidden_states,
- encoder_attention_mask,
- past_key_value,
- output_attentions,
- rel_pos,
- rel_2d_pos
- )
- else:
- layer_outputs = layer_module(
- hidden_states,
- attention_mask,
- layer_head_mask,
- encoder_hidden_states,
- encoder_attention_mask,
- past_key_value,
- output_attentions,
- rel_pos=rel_pos,
- rel_2d_pos=rel_2d_pos,
- )
-
- hidden_states = layer_outputs[0]
- if use_cache:
- next_decoder_cache += (layer_outputs[-1],)
- if output_attentions:
- all_self_attentions = all_self_attentions + (layer_outputs[1],)
- if self.config.add_cross_attention:
- all_cross_attentions = all_cross_attentions + (layer_outputs[2],)
-
- if self.detection and i in self.out_indices:
- xp = hidden_states[:, -Hp*Wp:, :].permute(0, 2, 1).reshape(len(hidden_states), -1, Hp, Wp)
- feat_out[self.out_features[j]] = self.ops[j](xp.contiguous())
- j += 1
-
- if self.detection:
- return feat_out
-
- if output_hidden_states:
- all_hidden_states = all_hidden_states + (hidden_states,)
-
- if not return_dict:
- return tuple(
- v
- for v in [
- hidden_states,
- next_decoder_cache,
- all_hidden_states,
- all_self_attentions,
- all_cross_attentions,
- ]
- if v is not None
- )
- return BaseModelOutputWithPastAndCrossAttentions(
- last_hidden_state=hidden_states,
- past_key_values=next_decoder_cache,
- hidden_states=all_hidden_states,
- attentions=all_self_attentions,
- cross_attentions=all_cross_attentions,
- )
-
-
-class LayoutLMv3Model(LayoutLMv3PreTrainedModel):
- """
- """
-
- _keys_to_ignore_on_load_missing = [r"position_ids"]
-
- # Copied from transformers.models.bert.modeling_bert.BertModel.__init__ with Bert->Roberta
- def __init__(self, config, detection=False, out_features=None, image_only=False):
- super().__init__(config)
- self.config = config
- assert not config.is_decoder and not config.add_cross_attention, \
- "This version do not support decoder. Please refer to RoBERTa for implementation of is_decoder."
- self.detection = detection
- if not self.detection:
- self.image_only = False
- else:
- assert config.visual_embed
- self.image_only = image_only
-
- if not self.image_only:
- self.embeddings = LayoutLMv3Embeddings(config)
- self.encoder = LayoutLMv3Encoder(config, detection=detection, out_features=out_features)
-
- if config.visual_embed:
- embed_dim = self.config.hidden_size
- # use the default pre-training parameters for fine-tuning (e.g., input_size)
- # when the input_size is larger in fine-tuning, we will interpolate the position embedding in forward
- self.patch_embed = PatchEmbed(embed_dim=embed_dim)
-
- patch_size = 16
- size = int(self.config.input_size / patch_size)
- self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
- self.pos_embed = nn.Parameter(torch.zeros(1, size * size + 1, embed_dim))
- self.pos_drop = nn.Dropout(p=0.)
-
- self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
- self.dropout = nn.Dropout(config.hidden_dropout_prob)
-
- if self.config.has_relative_attention_bias or self.config.has_spatial_attention_bias:
- self._init_visual_bbox(img_size=(size, size))
-
- from functools import partial
- norm_layer = partial(nn.LayerNorm, eps=1e-6)
- self.norm = norm_layer(embed_dim)
-
- self.init_weights()
-
- def get_input_embeddings(self):
- return self.embeddings.word_embeddings
-
- def set_input_embeddings(self, value):
- self.embeddings.word_embeddings = value
-
- def _prune_heads(self, heads_to_prune):
- """
- Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
- class PreTrainedModel
- """
- for layer, heads in heads_to_prune.items():
- self.encoder.layer[layer].attention.prune_heads(heads)
-
- def _init_visual_bbox(self, img_size=(14, 14), max_len=1000):
- visual_bbox_x = torch.div(torch.arange(0, max_len * (img_size[1] + 1), max_len),
- img_size[1], rounding_mode='trunc')
- visual_bbox_y = torch.div(torch.arange(0, max_len * (img_size[0] + 1), max_len),
- img_size[0], rounding_mode='trunc')
- visual_bbox = torch.stack(
- [
- visual_bbox_x[:-1].repeat(img_size[0], 1),
- visual_bbox_y[:-1].repeat(img_size[1], 1).transpose(0, 1),
- visual_bbox_x[1:].repeat(img_size[0], 1),
- visual_bbox_y[1:].repeat(img_size[1], 1).transpose(0, 1),
- ],
- dim=-1,
- ).view(-1, 4)
-
- cls_token_box = torch.tensor([[0 + 1, 0 + 1, max_len - 1, max_len - 1]])
- self.visual_bbox = torch.cat([cls_token_box, visual_bbox], dim=0)
-
- def _calc_visual_bbox(self, device, dtype, bsz): # , img_size=(14, 14), max_len=1000):
- visual_bbox = self.visual_bbox.repeat(bsz, 1, 1)
- visual_bbox = visual_bbox.to(device).type(dtype)
- return visual_bbox
-
- def forward_image(self, x):
- if self.detection:
- x = self.patch_embed(x, self.pos_embed[:, 1:, :] if self.pos_embed is not None else None)
- else:
- x = self.patch_embed(x)
- batch_size, seq_len, _ = x.size()
-
- cls_tokens = self.cls_token.expand(batch_size, -1, -1) # stole cls_tokens impl from Phil Wang, thanks
- if self.pos_embed is not None and self.detection:
- cls_tokens = cls_tokens + self.pos_embed[:, :1, :]
-
- x = torch.cat((cls_tokens, x), dim=1)
- if self.pos_embed is not None and not self.detection:
- x = x + self.pos_embed
- x = self.pos_drop(x)
-
- x = self.norm(x)
- return x
-
- # Copied from transformers.models.bert.modeling_bert.BertModel.forward
- def forward(
- self,
- input_ids=None,
- bbox=None,
- attention_mask=None,
- token_type_ids=None,
- valid_span=None,
- position_ids=None,
- head_mask=None,
- inputs_embeds=None,
- encoder_hidden_states=None,
- encoder_attention_mask=None,
- past_key_values=None,
- use_cache=None,
- output_attentions=None,
- output_hidden_states=None,
- return_dict=None,
- images=None,
- ):
- r"""
- encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`):
- Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if
- the model is configured as a decoder.
- encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
- Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in
- the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``:
-
- - 1 for tokens that are **not masked**,
- - 0 for tokens that are **masked**.
- past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`):
- Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.
-
- If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids`
- (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)`
- instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`.
- use_cache (:obj:`bool`, `optional`):
- If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up
- decoding (see :obj:`past_key_values`).
- """
- output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
- output_hidden_states = (
- output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
- )
- return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-
- use_cache = False
-
- # if input_ids is not None and inputs_embeds is not None:
- # raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
- if input_ids is not None:
- input_shape = input_ids.size()
- batch_size, seq_length = input_shape
- device = input_ids.device
- elif inputs_embeds is not None:
- input_shape = inputs_embeds.size()[:-1]
- batch_size, seq_length = input_shape
- device = inputs_embeds.device
- elif images is not None:
- batch_size = len(images)
- device = images.device
- else:
- raise ValueError("You have to specify either input_ids or inputs_embeds or images")
-
- if not self.image_only:
- # past_key_values_length
- past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0
-
- if attention_mask is None:
- attention_mask = torch.ones(((batch_size, seq_length + past_key_values_length)), device=device)
- if token_type_ids is None:
- token_type_ids = torch.zeros(input_shape, dtype=torch.long, device=device)
-
- # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
- # ourselves in which case we just need to make it broadcastable to all heads.
- # extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
-
- encoder_extended_attention_mask = None
-
- # Prepare head mask if needed
- # 1.0 in head_mask indicate we keep the head
- # attention_probs has shape bsz x n_heads x N x N
- # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
- # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
- head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
-
- if not self.image_only:
- if bbox is None:
- bbox = torch.zeros(tuple(list(input_shape) + [4]), dtype=torch.long, device=device)
-
- embedding_output = self.embeddings(
- input_ids=input_ids,
- bbox=bbox,
- position_ids=position_ids,
- token_type_ids=token_type_ids,
- inputs_embeds=inputs_embeds,
- past_key_values_length=past_key_values_length,
- )
-
- final_bbox = final_position_ids = None
- Hp = Wp = None
- if images is not None:
- patch_size = 16
- Hp, Wp = int(images.shape[2] / patch_size), int(images.shape[3] / patch_size)
- visual_emb = self.forward_image(images)
- if self.detection:
- visual_attention_mask = torch.ones((batch_size, visual_emb.shape[1]), dtype=torch.long, device=device)
- if self.image_only:
- attention_mask = visual_attention_mask
- else:
- attention_mask = torch.cat([attention_mask, visual_attention_mask], dim=1)
- elif self.image_only:
- attention_mask = torch.ones((batch_size, visual_emb.shape[1]), dtype=torch.long, device=device)
-
- if self.config.has_relative_attention_bias or self.config.has_spatial_attention_bias:
- if self.config.has_spatial_attention_bias:
- visual_bbox = self._calc_visual_bbox(device, dtype=torch.long, bsz=batch_size)
- if self.image_only:
- final_bbox = visual_bbox
- else:
- final_bbox = torch.cat([bbox, visual_bbox], dim=1)
-
- visual_position_ids = torch.arange(0, visual_emb.shape[1], dtype=torch.long, device=device).repeat(
- batch_size, 1)
- if self.image_only:
- final_position_ids = visual_position_ids
- else:
- position_ids = torch.arange(0, input_shape[1], device=device).unsqueeze(0)
- position_ids = position_ids.expand_as(input_ids)
- final_position_ids = torch.cat([position_ids, visual_position_ids], dim=1)
-
- if self.image_only:
- embedding_output = visual_emb
- else:
- embedding_output = torch.cat([embedding_output, visual_emb], dim=1)
- embedding_output = self.LayerNorm(embedding_output)
- embedding_output = self.dropout(embedding_output)
- elif self.config.has_relative_attention_bias or self.config.has_spatial_attention_bias:
- if self.config.has_spatial_attention_bias:
- final_bbox = bbox
- if self.config.has_relative_attention_bias:
- position_ids = self.embeddings.position_ids[:, :input_shape[1]]
- position_ids = position_ids.expand_as(input_ids)
- final_position_ids = position_ids
-
- extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, None, device)
-
- encoder_outputs = self.encoder(
- embedding_output,
- bbox=final_bbox,
- position_ids=final_position_ids,
- attention_mask=extended_attention_mask,
- head_mask=head_mask,
- encoder_hidden_states=encoder_hidden_states,
- encoder_attention_mask=encoder_extended_attention_mask,
- past_key_values=past_key_values,
- use_cache=use_cache,
- output_attentions=output_attentions,
- output_hidden_states=output_hidden_states,
- return_dict=return_dict,
- Hp=Hp,
- Wp=Wp,
- valid_span=valid_span,
- )
-
- if self.detection:
- return encoder_outputs
-
- sequence_output = encoder_outputs[0]
- pooled_output = None
-
- if not return_dict:
- return (sequence_output, pooled_output) + encoder_outputs[1:]
-
- return BaseModelOutputWithPoolingAndCrossAttentions(
- last_hidden_state=sequence_output,
- pooler_output=pooled_output,
- past_key_values=encoder_outputs.past_key_values,
- hidden_states=encoder_outputs.hidden_states,
- attentions=encoder_outputs.attentions,
- cross_attentions=encoder_outputs.cross_attentions,
- )
-
-
-class LayoutLMv3ClassificationHead(nn.Module):
- """
- Head for sentence-level classification tasks.
- Reference: RobertaClassificationHead
- """
-
- def __init__(self, config, pool_feature=False):
- super().__init__()
- self.pool_feature = pool_feature
- if pool_feature:
- self.dense = nn.Linear(config.hidden_size*3, config.hidden_size)
- else:
- self.dense = nn.Linear(config.hidden_size, config.hidden_size)
- classifier_dropout = (
- config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
- )
- self.dropout = nn.Dropout(classifier_dropout)
- self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
-
- def forward(self, x):
- # x = features[:, 0, :] # take token (equiv. to [CLS])
- x = self.dropout(x)
- x = self.dense(x)
- x = torch.tanh(x)
- x = self.dropout(x)
- x = self.out_proj(x)
- return x
-
-
-class LayoutLMv3ForTokenClassification(LayoutLMv3PreTrainedModel):
- _keys_to_ignore_on_load_unexpected = [r"pooler"]
- _keys_to_ignore_on_load_missing = [r"position_ids"]
-
- def __init__(self, config):
- super().__init__(config)
- self.num_labels = config.num_labels
-
- self.layoutlmv3 = LayoutLMv3Model(config)
- self.dropout = nn.Dropout(config.hidden_dropout_prob)
- if config.num_labels < 10:
- self.classifier = nn.Linear(config.hidden_size, config.num_labels)
- else:
- self.classifier = LayoutLMv3ClassificationHead(config, pool_feature=False)
-
- self.init_weights()
-
- def forward(
- self,
- input_ids=None,
- bbox=None,
- attention_mask=None,
- token_type_ids=None,
- position_ids=None,
- valid_span=None,
- head_mask=None,
- inputs_embeds=None,
- labels=None,
- output_attentions=None,
- output_hidden_states=None,
- return_dict=None,
- images=None,
- ):
- r"""
- labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
- Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels -
- 1]``.
- """
- return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-
- outputs = self.layoutlmv3(
- input_ids,
- bbox=bbox,
- attention_mask=attention_mask,
- token_type_ids=token_type_ids,
- position_ids=position_ids,
- head_mask=head_mask,
- inputs_embeds=inputs_embeds,
- output_attentions=output_attentions,
- output_hidden_states=output_hidden_states,
- return_dict=return_dict,
- images=images,
- valid_span=valid_span,
- )
-
- sequence_output = outputs[0]
-
- sequence_output = self.dropout(sequence_output)
- logits = self.classifier(sequence_output)
-
- loss = None
- if labels is not None:
- loss_fct = CrossEntropyLoss()
- # Only keep active parts of the loss
- if attention_mask is not None:
- active_loss = attention_mask.view(-1) == 1
- active_logits = logits.view(-1, self.num_labels)
- active_labels = torch.where(
- active_loss, labels.view(-1), torch.tensor(loss_fct.ignore_index).type_as(labels)
- )
- loss = loss_fct(active_logits, active_labels)
- else:
- loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
-
- if not return_dict:
- output = (logits,) + outputs[2:]
- return ((loss,) + output) if loss is not None else output
-
- return TokenClassifierOutput(
- loss=loss,
- logits=logits,
- hidden_states=outputs.hidden_states,
- attentions=outputs.attentions,
- )
-
-
-class LayoutLMv3ForQuestionAnswering(LayoutLMv3PreTrainedModel):
- _keys_to_ignore_on_load_unexpected = [r"pooler"]
- _keys_to_ignore_on_load_missing = [r"position_ids"]
-
- def __init__(self, config):
- super().__init__(config)
- self.num_labels = config.num_labels
-
- self.layoutlmv3 = LayoutLMv3Model(config)
- # self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
- self.qa_outputs = LayoutLMv3ClassificationHead(config, pool_feature=False)
-
- self.init_weights()
-
- def forward(
- self,
- input_ids=None,
- attention_mask=None,
- token_type_ids=None,
- position_ids=None,
- valid_span=None,
- head_mask=None,
- inputs_embeds=None,
- start_positions=None,
- end_positions=None,
- output_attentions=None,
- output_hidden_states=None,
- return_dict=None,
- bbox=None,
- images=None,
- ):
- r"""
- start_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):
- Labels for position (index) of the start of the labelled span for computing the token classification loss.
- Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the
- sequence are not taken into account for computing the loss.
- end_positions (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):
- Labels for position (index) of the end of the labelled span for computing the token classification loss.
- Positions are clamped to the length of the sequence (:obj:`sequence_length`). Position outside of the
- sequence are not taken into account for computing the loss.
- """
- return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-
- outputs = self.layoutlmv3(
- input_ids,
- attention_mask=attention_mask,
- token_type_ids=token_type_ids,
- position_ids=position_ids,
- head_mask=head_mask,
- inputs_embeds=inputs_embeds,
- output_attentions=output_attentions,
- output_hidden_states=output_hidden_states,
- return_dict=return_dict,
- bbox=bbox,
- images=images,
- valid_span=valid_span,
- )
-
- sequence_output = outputs[0]
-
- logits = self.qa_outputs(sequence_output)
- start_logits, end_logits = logits.split(1, dim=-1)
- start_logits = start_logits.squeeze(-1).contiguous()
- end_logits = end_logits.squeeze(-1).contiguous()
-
- total_loss = None
- if start_positions is not None and end_positions is not None:
- # If we are on multi-GPU, split add a dimension
- if len(start_positions.size()) > 1:
- start_positions = start_positions.squeeze(-1)
- if len(end_positions.size()) > 1:
- end_positions = end_positions.squeeze(-1)
- # sometimes the start/end positions are outside our model inputs, we ignore these terms
- ignored_index = start_logits.size(1)
- start_positions = start_positions.clamp(0, ignored_index)
- end_positions = end_positions.clamp(0, ignored_index)
-
- loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
- start_loss = loss_fct(start_logits, start_positions)
- end_loss = loss_fct(end_logits, end_positions)
- total_loss = (start_loss + end_loss) / 2
-
- if not return_dict:
- output = (start_logits, end_logits) + outputs[2:]
- return ((total_loss,) + output) if total_loss is not None else output
-
- return QuestionAnsweringModelOutput(
- loss=total_loss,
- start_logits=start_logits,
- end_logits=end_logits,
- hidden_states=outputs.hidden_states,
- attentions=outputs.attentions,
- )
-
-
-class LayoutLMv3ForSequenceClassification(LayoutLMv3PreTrainedModel):
- _keys_to_ignore_on_load_missing = [r"position_ids"]
-
- def __init__(self, config):
- super().__init__(config)
- self.num_labels = config.num_labels
- self.config = config
- self.layoutlmv3 = LayoutLMv3Model(config)
- self.classifier = LayoutLMv3ClassificationHead(config, pool_feature=False)
-
- self.init_weights()
-
- def forward(
- self,
- input_ids=None,
- attention_mask=None,
- token_type_ids=None,
- position_ids=None,
- valid_span=None,
- head_mask=None,
- inputs_embeds=None,
- labels=None,
- output_attentions=None,
- output_hidden_states=None,
- return_dict=None,
- bbox=None,
- images=None,
- ):
- r"""
- labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`):
- Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ...,
- config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss),
- If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
- """
- return_dict = return_dict if return_dict is not None else self.config.use_return_dict
-
- outputs = self.layoutlmv3(
- input_ids,
- attention_mask=attention_mask,
- token_type_ids=token_type_ids,
- position_ids=position_ids,
- head_mask=head_mask,
- inputs_embeds=inputs_embeds,
- output_attentions=output_attentions,
- output_hidden_states=output_hidden_states,
- return_dict=return_dict,
- bbox=bbox,
- images=images,
- valid_span=valid_span,
- )
-
- sequence_output = outputs[0][:, 0, :]
- logits = self.classifier(sequence_output)
-
- loss = None
- if labels is not None:
- if self.config.problem_type is None:
- if self.num_labels == 1:
- self.config.problem_type = "regression"
- elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
- self.config.problem_type = "single_label_classification"
- else:
- self.config.problem_type = "multi_label_classification"
-
- if self.config.problem_type == "regression":
- loss_fct = MSELoss()
- if self.num_labels == 1:
- loss = loss_fct(logits.squeeze(), labels.squeeze())
- else:
- loss = loss_fct(logits, labels)
- elif self.config.problem_type == "single_label_classification":
- loss_fct = CrossEntropyLoss()
- loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
- elif self.config.problem_type == "multi_label_classification":
- loss_fct = BCEWithLogitsLoss()
- loss = loss_fct(logits, labels)
-
- if not return_dict:
- output = (logits,) + outputs[2:]
- return ((loss,) + output) if loss is not None else output
-
- return SequenceClassifierOutput(
- loss=loss,
- logits=logits,
- hidden_states=outputs.hidden_states,
- attentions=outputs.attentions,
- )
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3.py
deleted file mode 100644
index f340d3c6a..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3.py
+++ /dev/null
@@ -1,32 +0,0 @@
-# coding=utf-8
-# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Tokenization classes for LayoutLMv3, refer to RoBERTa."""
-
-from transformers.models.roberta import RobertaTokenizer
-from transformers.utils import logging
-
-
-logger = logging.get_logger(__name__)
-
-VOCAB_FILES_NAMES = {
- "vocab_file": "vocab.json",
- "merges_file": "merges.txt",
-}
-
-class LayoutLMv3Tokenizer(RobertaTokenizer):
- vocab_files_names = VOCAB_FILES_NAMES
- # pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
- # max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
- model_input_names = ["input_ids", "attention_mask"]
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3_fast.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3_fast.py
deleted file mode 100644
index 9fd75ff1d..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/layoutlmft/models/layoutlmv3/tokenization_layoutlmv3_fast.py
+++ /dev/null
@@ -1,34 +0,0 @@
-# coding=utf-8
-# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""Fast Tokenization classes for LayoutLMv3, refer to RoBERTa."""
-
-
-from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast
-from transformers.utils import logging
-
-from .tokenization_layoutlmv3 import LayoutLMv3Tokenizer
-
-
-logger = logging.get_logger(__name__)
-
-VOCAB_FILES_NAMES = {"vocab_file": "vocab.json", "merges_file": "merges.txt", "tokenizer_file": "tokenizer.json"}
-
-
-class LayoutLMv3TokenizerFast(RobertaTokenizerFast):
- vocab_files_names = VOCAB_FILES_NAMES
- # pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
- # max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
- model_input_names = ["input_ids", "attention_mask"]
- slow_tokenizer_class = LayoutLMv3Tokenizer
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/model_init.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/model_init.py
deleted file mode 100644
index a624d60d5..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/model_init.py
+++ /dev/null
@@ -1,151 +0,0 @@
-from .visualizer import Visualizer
-from .rcnn_vl import *
-from .backbone import *
-
-from detectron2.config import get_cfg
-from detectron2.config import CfgNode as CN
-from detectron2.data import MetadataCatalog, DatasetCatalog
-from detectron2.data.datasets import register_coco_instances
-from detectron2.engine import DefaultTrainer, default_argument_parser, default_setup, launch, DefaultPredictor
-
-
-def add_vit_config(cfg):
- """
- Add config for VIT.
- """
- _C = cfg
-
- _C.MODEL.VIT = CN()
-
- # CoaT model name.
- _C.MODEL.VIT.NAME = ""
-
- # Output features from CoaT backbone.
- _C.MODEL.VIT.OUT_FEATURES = ["layer3", "layer5", "layer7", "layer11"]
-
- _C.MODEL.VIT.IMG_SIZE = [224, 224]
-
- _C.MODEL.VIT.POS_TYPE = "shared_rel"
-
- _C.MODEL.VIT.DROP_PATH = 0.
-
- _C.MODEL.VIT.MODEL_KWARGS = "{}"
-
- _C.SOLVER.OPTIMIZER = "ADAMW"
-
- _C.SOLVER.BACKBONE_MULTIPLIER = 1.0
-
- _C.AUG = CN()
-
- _C.AUG.DETR = False
-
- _C.MODEL.IMAGE_ONLY = True
- _C.PUBLAYNET_DATA_DIR_TRAIN = ""
- _C.PUBLAYNET_DATA_DIR_TEST = ""
- _C.FOOTNOTE_DATA_DIR_TRAIN = ""
- _C.FOOTNOTE_DATA_DIR_VAL = ""
- _C.SCIHUB_DATA_DIR_TRAIN = ""
- _C.SCIHUB_DATA_DIR_TEST = ""
- _C.JIAOCAI_DATA_DIR_TRAIN = ""
- _C.JIAOCAI_DATA_DIR_TEST = ""
- _C.ICDAR_DATA_DIR_TRAIN = ""
- _C.ICDAR_DATA_DIR_TEST = ""
- _C.M6DOC_DATA_DIR_TEST = ""
- _C.DOCSTRUCTBENCH_DATA_DIR_TEST = ""
- _C.DOCSTRUCTBENCHv2_DATA_DIR_TEST = ""
- _C.CACHE_DIR = ""
- _C.MODEL.CONFIG_PATH = ""
-
- # effective update steps would be MAX_ITER/GRADIENT_ACCUMULATION_STEPS
- # maybe need to set MAX_ITER *= GRADIENT_ACCUMULATION_STEPS
- _C.SOLVER.GRADIENT_ACCUMULATION_STEPS = 1
-
-
-def setup(args, device):
- """
- Create configs and perform basic setups.
- """
- cfg = get_cfg()
-
- # add_coat_config(cfg)
- add_vit_config(cfg)
- cfg.merge_from_file(args.config_file)
- cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.2 # set threshold for this model
- cfg.merge_from_list(args.opts)
-
- # 使用统一的device配置
- cfg.MODEL.DEVICE = device
-
- cfg.freeze()
- default_setup(cfg, args)
-
- #@todo 可以删掉这块？
- # register_coco_instances(
- # "scihub_train",
- # {},
- # cfg.SCIHUB_DATA_DIR_TRAIN + ".json",
- # cfg.SCIHUB_DATA_DIR_TRAIN
- # )
-
- return cfg
-
-
-class DotDict(dict):
- def __init__(self, *args, **kwargs):
- super(DotDict, self).__init__(*args, **kwargs)
-
- def __getattr__(self, key):
- if key not in self.keys():
- return None
- value = self[key]
- if isinstance(value, dict):
- value = DotDict(value)
- return value
-
- def __setattr__(self, key, value):
- self[key] = value
-
-
-class Layoutlmv3_Predictor(object):
- def __init__(self, weights, config_file, device):
- layout_args = {
- "config_file": config_file,
- "resume": False,
- "eval_only": False,
- "num_gpus": 1,
- "num_machines": 1,
- "machine_rank": 0,
- "dist_url": "tcp://127.0.0.1:57823",
- "opts": ["MODEL.WEIGHTS", weights],
- }
- layout_args = DotDict(layout_args)
-
- cfg = setup(layout_args, device)
- self.mapping = ["title", "plain text", "abandon", "figure", "figure_caption", "table", "table_caption",
- "table_footnote", "isolate_formula", "formula_caption"]
- MetadataCatalog.get(cfg.DATASETS.TRAIN[0]).thing_classes = self.mapping
- self.predictor = DefaultPredictor(cfg)
-
- def __call__(self, image, ignore_catids=[]):
- # page_layout_result = {
- # "layout_dets": []
- # }
- layout_dets = []
- outputs = self.predictor(image)
- boxes = outputs["instances"].to("cpu")._fields["pred_boxes"].tensor.tolist()
- labels = outputs["instances"].to("cpu")._fields["pred_classes"].tolist()
- scores = outputs["instances"].to("cpu")._fields["scores"].tolist()
- for bbox_idx in range(len(boxes)):
- if labels[bbox_idx] in ignore_catids:
- continue
- layout_dets.append({
- "category_id": labels[bbox_idx],
- "poly": [
- boxes[bbox_idx][0], boxes[bbox_idx][1],
- boxes[bbox_idx][2], boxes[bbox_idx][1],
- boxes[bbox_idx][2], boxes[bbox_idx][3],
- boxes[bbox_idx][0], boxes[bbox_idx][3],
- ],
- "score": scores[bbox_idx]
- })
- return layout_dets
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/rcnn_vl.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/rcnn_vl.py
deleted file mode 100644
index 46b2e1610..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/rcnn_vl.py
+++ /dev/null
@@ -1,163 +0,0 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-import logging
-import numpy as np
-from typing import Dict, List, Optional, Tuple
-import torch
-from torch import nn
-
-from detectron2.config import configurable
-from detectron2.structures import ImageList, Instances
-from detectron2.utils.events import get_event_storage
-
-from detectron2.modeling.backbone import Backbone, build_backbone
-from detectron2.modeling.meta_arch.build import META_ARCH_REGISTRY
-
-from detectron2.modeling.meta_arch import GeneralizedRCNN
-
-from detectron2.modeling.postprocessing import detector_postprocess
-from detectron2.modeling.roi_heads.fast_rcnn import fast_rcnn_inference_single_image
-from contextlib import contextmanager
-from itertools import count
-
-@META_ARCH_REGISTRY.register()
-class VLGeneralizedRCNN(GeneralizedRCNN):
- """
- Generalized R-CNN. Any models that contains the following three components:
- 1. Per-image feature extraction (aka backbone)
- 2. Region proposal generation
- 3. Per-region feature extraction and prediction
- """
-
- def forward(self, batched_inputs: List[Dict[str, torch.Tensor]]):
- """
- Args:
- batched_inputs: a list, batched outputs of :class:`DatasetMapper` .
- Each item in the list contains the inputs for one image.
- For now, each item in the list is a dict that contains:
-
- * image: Tensor, image in (C, H, W) format.
- * instances (optional): groundtruth :class:`Instances`
- * proposals (optional): :class:`Instances`, precomputed proposals.
-
- Other information that's included in the original dicts, such as:
-
- * "height", "width" (int): the output resolution of the model, used in inference.
- See :meth:`postprocess` for details.
-
- Returns:
- list[dict]:
- Each dict is the output for one input image.
- The dict contains one key "instances" whose value is a :class:`Instances`.
- The :class:`Instances` object has the following keys:
- "pred_boxes", "pred_classes", "scores", "pred_masks", "pred_keypoints"
- """
- if not self.training:
- return self.inference(batched_inputs)
-
- images = self.preprocess_image(batched_inputs)
- if "instances" in batched_inputs[0]:
- gt_instances = [x["instances"].to(self.device) for x in batched_inputs]
- else:
- gt_instances = None
-
- # features = self.backbone(images.tensor)
- input = self.get_batch(batched_inputs, images)
- features = self.backbone(input)
-
- if self.proposal_generator is not None:
- proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
- else:
- assert "proposals" in batched_inputs[0]
- proposals = [x["proposals"].to(self.device) for x in batched_inputs]
- proposal_losses = {}
-
- _, detector_losses = self.roi_heads(images, features, proposals, gt_instances)
- if self.vis_period > 0:
- storage = get_event_storage()
- if storage.iter % self.vis_period == 0:
- self.visualize_training(batched_inputs, proposals)
-
- losses = {}
- losses.update(detector_losses)
- losses.update(proposal_losses)
- return losses
-
- def inference(
- self,
- batched_inputs: List[Dict[str, torch.Tensor]],
- detected_instances: Optional[List[Instances]] = None,
- do_postprocess: bool = True,
- ):
- """
- Run inference on the given inputs.
-
- Args:
- batched_inputs (list[dict]): same as in :meth:`forward`
- detected_instances (None or list[Instances]): if not None, it
- contains an `Instances` object per image. The `Instances`
- object contains "pred_boxes" and "pred_classes" which are
- known boxes in the image.
- The inference will then skip the detection of bounding boxes,
- and only predict other per-ROI outputs.
- do_postprocess (bool): whether to apply post-processing on the outputs.
-
- Returns:
- When do_postprocess=True, same as in :meth:`forward`.
- Otherwise, a list[Instances] containing raw network outputs.
- """
- assert not self.training
-
- images = self.preprocess_image(batched_inputs)
- # features = self.backbone(images.tensor)
- input = self.get_batch(batched_inputs, images)
- features = self.backbone(input)
-
- if detected_instances is None:
- if self.proposal_generator is not None:
- proposals, _ = self.proposal_generator(images, features, None)
- else:
- assert "proposals" in batched_inputs[0]
- proposals = [x["proposals"].to(self.device) for x in batched_inputs]
-
- results, _ = self.roi_heads(images, features, proposals, None)
- else:
- detected_instances = [x.to(self.device) for x in detected_instances]
- results = self.roi_heads.forward_with_given_boxes(features, detected_instances)
-
- if do_postprocess:
- assert not torch.jit.is_scripting(), "Scripting is not supported for postprocess."
- return GeneralizedRCNN._postprocess(results, batched_inputs, images.image_sizes)
- else:
- return results
-
- def get_batch(self, examples, images):
- if len(examples) >= 1 and "bbox" not in examples[0]: # image_only
- return {"images": images.tensor}
-
- return input
-
- def _batch_inference(self, batched_inputs, detected_instances=None):
- """
- Execute inference on a list of inputs,
- using batch size = self.batch_size (e.g., 2), instead of the length of the list.
-
- Inputs & outputs have the same format as :meth:`GeneralizedRCNN.inference`
- """
- if detected_instances is None:
- detected_instances = [None] * len(batched_inputs)
-
- outputs = []
- inputs, instances = [], []
- for idx, input, instance in zip(count(), batched_inputs, detected_instances):
- inputs.append(input)
- instances.append(instance)
- if len(inputs) == 2 or idx == len(batched_inputs) - 1:
- outputs.extend(
- self.inference(
- inputs,
- instances if instances[0] is not None else None,
- do_postprocess=True, # False
- )
- )
- inputs, instances = [], []
- return outputs
diff --git a/magic_pdf/model/pek_sub_modules/layoutlmv3/visualizer.py b/magic_pdf/model/pek_sub_modules/layoutlmv3/visualizer.py
deleted file mode 100644
index 8185984e6..000000000
--- a/magic_pdf/model/pek_sub_modules/layoutlmv3/visualizer.py
+++ /dev/null
@@ -1,1236 +0,0 @@
-# Copyright (c) Facebook, Inc. and its affiliates.
-import colorsys
-import logging
-import math
-import numpy as np
-from enum import Enum, unique
-import cv2
-import matplotlib as mpl
-import matplotlib.colors as mplc
-import matplotlib.figure as mplfigure
-import pycocotools.mask as mask_util
-import torch
-from matplotlib.backends.backend_agg import FigureCanvasAgg
-from PIL import Image
-
-from detectron2.data import MetadataCatalog
-from detectron2.structures import BitMasks, Boxes, BoxMode, Keypoints, PolygonMasks, RotatedBoxes
-from detectron2.utils.file_io import PathManager
-
-from detectron2.utils.colormap import random_color
-
-import pdb
-
-logger = logging.getLogger(__name__)
-
-__all__ = ["ColorMode", "VisImage", "Visualizer"]
-
-
-_SMALL_OBJECT_AREA_THRESH = 1000
-_LARGE_MASK_AREA_THRESH = 120000
-_OFF_WHITE = (1.0, 1.0, 240.0 / 255)
-_BLACK = (0, 0, 0)
-_RED = (1.0, 0, 0)
-
-_KEYPOINT_THRESHOLD = 0.05
-
-#CLASS_NAMES = ["footnote", "footer", "header"]
-
-@unique
-class ColorMode(Enum):
- """
- Enum of different color modes to use for instance visualizations.
- """
-
- IMAGE = 0
- """
- Picks a random color for every instance and overlay segmentations with low opacity.
- """
- SEGMENTATION = 1
- """
- Let instances of the same category have similar colors
- (from metadata.thing_colors), and overlay them with
- high opacity. This provides more attention on the quality of segmentation.
- """
- IMAGE_BW = 2
- """
- Same as IMAGE, but convert all areas without masks to gray-scale.
- Only available for drawing per-instance mask predictions.
- """
-
-
-class GenericMask:
- """
- Attribute:
- polygons (list[ndarray]): list[ndarray]: polygons for this mask.
- Each ndarray has format [x, y, x, y, ...]
- mask (ndarray): a binary mask
- """
-
- def __init__(self, mask_or_polygons, height, width):
- self._mask = self._polygons = self._has_holes = None
- self.height = height
- self.width = width
-
- m = mask_or_polygons
- if isinstance(m, dict):
- # RLEs
- assert "counts" in m and "size" in m
- if isinstance(m["counts"], list): # uncompressed RLEs
- h, w = m["size"]
- assert h == height and w == width
- m = mask_util.frPyObjects(m, h, w)
- self._mask = mask_util.decode(m)[:, :]
- return
-
- if isinstance(m, list): # list[ndarray]
- self._polygons = [np.asarray(x).reshape(-1) for x in m]
- return
-
- if isinstance(m, np.ndarray): # assumed to be a binary mask
- assert m.shape[1] != 2, m.shape
- assert m.shape == (
- height,
- width,
- ), f"mask shape: {m.shape}, target dims: {height}, {width}"
- self._mask = m.astype("uint8")
- return
-
- raise ValueError("GenericMask cannot handle object {} of type '{}'".format(m, type(m)))
-
- @property
- def mask(self):
- if self._mask is None:
- self._mask = self.polygons_to_mask(self._polygons)
- return self._mask
-
- @property
- def polygons(self):
- if self._polygons is None:
- self._polygons, self._has_holes = self.mask_to_polygons(self._mask)
- return self._polygons
-
- @property
- def has_holes(self):
- if self._has_holes is None:
- if self._mask is not None:
- self._polygons, self._has_holes = self.mask_to_polygons(self._mask)
- else:
- self._has_holes = False # if original format is polygon, does not have holes
- return self._has_holes
-
- def mask_to_polygons(self, mask):
- # cv2.RETR_CCOMP flag retrieves all the contours and arranges them to a 2-level
- # hierarchy. External contours (boundary) of the object are placed in hierarchy-1.
- # Internal contours (holes) are placed in hierarchy-2.
- # cv2.CHAIN_APPROX_NONE flag gets vertices of polygons from contours.
- mask = np.ascontiguousarray(mask) # some versions of cv2 does not support incontiguous arr
- res = cv2.findContours(mask.astype("uint8"), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)
- hierarchy = res[-1]
- if hierarchy is None: # empty mask
- return [], False
- has_holes = (hierarchy.reshape(-1, 4)[:, 3] >= 0).sum() > 0
- res = res[-2]
- res = [x.flatten() for x in res]
- # These coordinates from OpenCV are integers in range [0, W-1 or H-1].
- # We add 0.5 to turn them into real-value coordinate space. A better solution
- # would be to first +0.5 and then dilate the returned polygon by 0.5.
- res = [x + 0.5 for x in res if len(x) >= 6]
- return res, has_holes
-
- def polygons_to_mask(self, polygons):
- rle = mask_util.frPyObjects(polygons, self.height, self.width)
- rle = mask_util.merge(rle)
- return mask_util.decode(rle)[:, :]
-
- def area(self):
- return self.mask.sum()
-
- def bbox(self):
- p = mask_util.frPyObjects(self.polygons, self.height, self.width)
- p = mask_util.merge(p)
- bbox = mask_util.toBbox(p)
- bbox[2] += bbox[0]
- bbox[3] += bbox[1]
- return bbox
-
-
-class _PanopticPrediction:
- """
- Unify different panoptic annotation/prediction formats
- """
-
- def __init__(self, panoptic_seg, segments_info, metadata=None):
- if segments_info is None:
- assert metadata is not None
- # If "segments_info" is None, we assume "panoptic_img" is a
- # H*W int32 image storing the panoptic_id in the format of
- # category_id * label_divisor + instance_id. We reserve -1 for
- # VOID label.
- label_divisor = metadata.label_divisor
- segments_info = []
- for panoptic_label in np.unique(panoptic_seg.numpy()):
- if panoptic_label == -1:
- # VOID region.
- continue
- pred_class = panoptic_label // label_divisor
- isthing = pred_class in metadata.thing_dataset_id_to_contiguous_id.values()
- segments_info.append(
- {
- "id": int(panoptic_label),
- "category_id": int(pred_class),
- "isthing": bool(isthing),
- }
- )
- del metadata
-
- self._seg = panoptic_seg
-
- self._sinfo = {s["id"]: s for s in segments_info} # seg id -> seg info
- segment_ids, areas = torch.unique(panoptic_seg, sorted=True, return_counts=True)
- areas = areas.numpy()
- sorted_idxs = np.argsort(-areas)
- self._seg_ids, self._seg_areas = segment_ids[sorted_idxs], areas[sorted_idxs]
- self._seg_ids = self._seg_ids.tolist()
- for sid, area in zip(self._seg_ids, self._seg_areas):
- if sid in self._sinfo:
- self._sinfo[sid]["area"] = float(area)
-
- def non_empty_mask(self):
- """
- Returns:
- (H, W) array, a mask for all pixels that have a prediction
- """
- empty_ids = []
- for id in self._seg_ids:
- if id not in self._sinfo:
- empty_ids.append(id)
- if len(empty_ids) == 0:
- return np.zeros(self._seg.shape, dtype=np.uint8)
- assert (
- len(empty_ids) == 1
- ), ">1 ids corresponds to no labels. This is currently not supported"
- return (self._seg != empty_ids[0]).numpy().astype(np.bool)
-
- def semantic_masks(self):
- for sid in self._seg_ids:
- sinfo = self._sinfo.get(sid)
- if sinfo is None or sinfo["isthing"]:
- # Some pixels (e.g. id 0 in PanopticFPN) have no instance or semantic predictions.
- continue
- yield (self._seg == sid).numpy().astype(np.bool), sinfo
-
- def instance_masks(self):
- for sid in self._seg_ids:
- sinfo = self._sinfo.get(sid)
- if sinfo is None or not sinfo["isthing"]:
- continue
- mask = (self._seg == sid).numpy().astype(np.bool)
- if mask.sum() > 0:
- yield mask, sinfo
-
-
-def _create_text_labels(classes, scores, class_names, is_crowd=None):
- """
- Args:
- classes (list[int] or None):
- scores (list[float] or None):
- class_names (list[str] or None):
- is_crowd (list[bool] or None):
-
- Returns:
- list[str] or None
- """
- #class_names = CLASS_NAMES
- labels = None
- if classes is not None:
- if class_names is not None and len(class_names) > 0:
- labels = [class_names[i] for i in classes]
- else:
- labels = [str(i) for i in classes]
-
- if scores is not None:
- if labels is None:
- labels = ["{:.0f}%".format(s * 100) for s in scores]
- else:
- labels = ["{} {:.0f}%".format(l, s * 100) for l, s in zip(labels, scores)]
- if labels is not None and is_crowd is not None:
- labels = [l + ("|crowd" if crowd else "") for l, crowd in zip(labels, is_crowd)]
- return labels
-
-
-class VisImage:
- def __init__(self, img, scale=1.0):
- """
- Args:
- img (ndarray): an RGB image of shape (H, W, 3) in range [0, 255].
- scale (float): scale the input image
- """
- self.img = img
- self.scale = scale
- self.width, self.height = img.shape[1], img.shape[0]
- self._setup_figure(img)
-
- def _setup_figure(self, img):
- """
- Args:
- Same as in :meth:`__init__()`.
-
- Returns:
- fig (matplotlib.pyplot.figure): top level container for all the image plot elements.
- ax (matplotlib.pyplot.Axes): contains figure elements and sets the coordinate system.
- """
- fig = mplfigure.Figure(frameon=False)
- self.dpi = fig.get_dpi()
- # add a small 1e-2 to avoid precision lost due to matplotlib's truncation
- # (https://github.com/matplotlib/matplotlib/issues/15363)
- fig.set_size_inches(
- (self.width * self.scale + 1e-2) / self.dpi,
- (self.height * self.scale + 1e-2) / self.dpi,
- )
- self.canvas = FigureCanvasAgg(fig)
- # self.canvas = mpl.backends.backend_cairo.FigureCanvasCairo(fig)
- ax = fig.add_axes([0.0, 0.0, 1.0, 1.0])
- ax.axis("off")
- self.fig = fig
- self.ax = ax
- self.reset_image(img)
-
- def reset_image(self, img):
- """
- Args:
- img: same as in __init__
- """
- img = img.astype("uint8")
- self.ax.imshow(img, extent=(0, self.width, self.height, 0), interpolation="nearest")
-
- def save(self, filepath):
- """
- Args:
- filepath (str): a string that contains the absolute path, including the file name, where
- the visualized image will be saved.
- """
- self.fig.savefig(filepath)
-
- def get_image(self):
- """
- Returns:
- ndarray:
- the visualized image of shape (H, W, 3) (RGB) in uint8 type.
- The shape is scaled w.r.t the input image using the given `scale` argument.
- """
- canvas = self.canvas
- s, (width, height) = canvas.print_to_buffer()
- # buf = io.BytesIO() # works for cairo backend
- # canvas.print_rgba(buf)
- # width, height = self.width, self.height
- # s = buf.getvalue()
-
- buffer = np.frombuffer(s, dtype="uint8")
-
- img_rgba = buffer.reshape(height, width, 4)
- rgb, alpha = np.split(img_rgba, [3], axis=2)
- return rgb.astype("uint8")
-
-
-class Visualizer:
- """
- Visualizer that draws data about detection/segmentation on images.
-
- It contains methods like `draw_{text,box,circle,line,binary_mask,polygon}`
- that draw primitive objects to images, as well as high-level wrappers like
- `draw_{instance_predictions,sem_seg,panoptic_seg_predictions,dataset_dict}`
- that draw composite data in some pre-defined style.
-
- Note that the exact visualization style for the high-level wrappers are subject to change.
- Style such as color, opacity, label contents, visibility of labels, or even the visibility
- of objects themselves (e.g. when the object is too small) may change according
- to different heuristics, as long as the results still look visually reasonable.
-
- To obtain a consistent style, you can implement custom drawing functions with the
- abovementioned primitive methods instead. If you need more customized visualization
- styles, you can process the data yourself following their format documented in
- tutorials (:doc:`/tutorials/models`, :doc:`/tutorials/datasets`). This class does not
- intend to satisfy everyone's preference on drawing styles.
-
- This visualizer focuses on high rendering quality rather than performance. It is not
- designed to be used for real-time applications.
- """
-
- # TODO implement a fast, rasterized version using OpenCV
-
- def __init__(self, img_rgb, metadata=None, scale=1.0, instance_mode=ColorMode.IMAGE):
- """
- Args:
- img_rgb: a numpy array of shape (H, W, C), where H and W correspond to
- the height and width of the image respectively. C is the number of
- color channels. The image is required to be in RGB format since that
- is a requirement of the Matplotlib library. The image is also expected
- to be in the range [0, 255].
- metadata (Metadata): dataset metadata (e.g. class names and colors)
- instance_mode (ColorMode): defines one of the pre-defined style for drawing
- instances on an image.
- """
- self.img = np.asarray(img_rgb).clip(0, 255).astype(np.uint8)
- if metadata is None:
- metadata = MetadataCatalog.get("__nonexist__")
- self.metadata = metadata
- self.output = VisImage(self.img, scale=scale)
- self.cpu_device = torch.device("cpu")
-
- # too small texts are useless, therefore clamp to 9
- self._default_font_size = max(
- np.sqrt(self.output.height * self.output.width) // 90, 10 // scale
- )
- self._instance_mode = instance_mode
- self.keypoint_threshold = _KEYPOINT_THRESHOLD
-
- def draw_instance_predictions(self, predictions):
- """
- Draw instance-level prediction results on an image.
-
- Args:
- predictions (Instances): the output of an instance detection/segmentation
- model. Following fields will be used to draw:
- "pred_boxes", "pred_classes", "scores", "pred_masks" (or "pred_masks_rle").
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- boxes = predictions.pred_boxes if predictions.has("pred_boxes") else None
- scores = predictions.scores if predictions.has("scores") else None
- classes = predictions.pred_classes.tolist() if predictions.has("pred_classes") else None
- labels = _create_text_labels(classes, scores, self.metadata.get("thing_classes", None))
- keypoints = predictions.pred_keypoints if predictions.has("pred_keypoints") else None
-
- if predictions.has("pred_masks"):
- masks = np.asarray(predictions.pred_masks)
- masks = [GenericMask(x, self.output.height, self.output.width) for x in masks]
- else:
- masks = None
-
- if self._instance_mode == ColorMode.SEGMENTATION and self.metadata.get("thing_colors"):
- colors = [
- self._jitter([x / 255 for x in self.metadata.thing_colors[c]]) for c in classes
- ]
- alpha = 0.8
- else:
- colors = None
- alpha = 0.5
-
- if self._instance_mode == ColorMode.IMAGE_BW:
- self.output.reset_image(
- self._create_grayscale_image(
- (predictions.pred_masks.any(dim=0) > 0).numpy()
- if predictions.has("pred_masks")
- else None
- )
- )
- alpha = 0.3
-
- self.overlay_instances(
- masks=masks,
- boxes=boxes,
- labels=labels,
- keypoints=keypoints,
- assigned_colors=colors,
- alpha=alpha,
- )
- return self.output
-
- def draw_sem_seg(self, sem_seg, area_threshold=None, alpha=0.8):
- """
- Draw semantic segmentation predictions/labels.
-
- Args:
- sem_seg (Tensor or ndarray): the segmentation of shape (H, W).
- Each value is the integer label of the pixel.
- area_threshold (int): segments with less than `area_threshold` are not drawn.
- alpha (float): the larger it is, the more opaque the segmentations are.
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- if isinstance(sem_seg, torch.Tensor):
- sem_seg = sem_seg.numpy()
- labels, areas = np.unique(sem_seg, return_counts=True)
- sorted_idxs = np.argsort(-areas).tolist()
- labels = labels[sorted_idxs]
- for label in filter(lambda l: l < len(self.metadata.stuff_classes), labels):
- try:
- mask_color = [x / 255 for x in self.metadata.stuff_colors[label]]
- except (AttributeError, IndexError):
- mask_color = None
-
- binary_mask = (sem_seg == label).astype(np.uint8)
- text = self.metadata.stuff_classes[label]
- self.draw_binary_mask(
- binary_mask,
- color=mask_color,
- edge_color=_OFF_WHITE,
- text=text,
- alpha=alpha,
- area_threshold=area_threshold,
- )
- return self.output
-
- def draw_panoptic_seg(self, panoptic_seg, segments_info, area_threshold=None, alpha=0.7):
- """
- Draw panoptic prediction annotations or results.
-
- Args:
- panoptic_seg (Tensor): of shape (height, width) where the values are ids for each
- segment.
- segments_info (list[dict] or None): Describe each segment in `panoptic_seg`.
- If it is a ``list[dict]``, each dict contains keys "id", "category_id".
- If None, category id of each pixel is computed by
- ``pixel // metadata.label_divisor``.
- area_threshold (int): stuff segments with less than `area_threshold` are not drawn.
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- pred = _PanopticPrediction(panoptic_seg, segments_info, self.metadata)
-
- if self._instance_mode == ColorMode.IMAGE_BW:
- self.output.reset_image(self._create_grayscale_image(pred.non_empty_mask()))
-
- # draw mask for all semantic segments first i.e. "stuff"
- for mask, sinfo in pred.semantic_masks():
- category_idx = sinfo["category_id"]
- try:
- mask_color = [x / 255 for x in self.metadata.stuff_colors[category_idx]]
- except AttributeError:
- mask_color = None
-
- text = self.metadata.stuff_classes[category_idx]
- self.draw_binary_mask(
- mask,
- color=mask_color,
- edge_color=_OFF_WHITE,
- text=text,
- alpha=alpha,
- area_threshold=area_threshold,
- )
-
- # draw mask for all instances second
- all_instances = list(pred.instance_masks())
- if len(all_instances) == 0:
- return self.output
- masks, sinfo = list(zip(*all_instances))
- category_ids = [x["category_id"] for x in sinfo]
-
- try:
- scores = [x["score"] for x in sinfo]
- except KeyError:
- scores = None
- labels = _create_text_labels(
- category_ids, scores, self.metadata.thing_classes, [x.get("iscrowd", 0) for x in sinfo]
- )
-
- try:
- colors = [
- self._jitter([x / 255 for x in self.metadata.thing_colors[c]]) for c in category_ids
- ]
- except AttributeError:
- colors = None
- self.overlay_instances(masks=masks, labels=labels, assigned_colors=colors, alpha=alpha)
-
- return self.output
-
- draw_panoptic_seg_predictions = draw_panoptic_seg # backward compatibility
-
- def draw_dataset_dict(self, dic):
- """
- Draw annotations/segmentaions in Detectron2 Dataset format.
-
- Args:
- dic (dict): annotation/segmentation data of one image, in Detectron2 Dataset format.
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- annos = dic.get("annotations", None)
- if annos:
- if "segmentation" in annos[0]:
- masks = [x["segmentation"] for x in annos]
- else:
- masks = None
- if "keypoints" in annos[0]:
- keypts = [x["keypoints"] for x in annos]
- keypts = np.array(keypts).reshape(len(annos), -1, 3)
- else:
- keypts = None
-
- boxes = [
- BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYXY_ABS)
- if len(x["bbox"]) == 4
- else x["bbox"]
- for x in annos
- ]
-
- colors = None
- category_ids = [x["category_id"] for x in annos]
- if self._instance_mode == ColorMode.SEGMENTATION and self.metadata.get("thing_colors"):
- colors = [
- self._jitter([x / 255 for x in self.metadata.thing_colors[c]])
- for c in category_ids
- ]
- names = self.metadata.get("thing_classes", None)
- labels = _create_text_labels(
- category_ids,
- scores=None,
- class_names=names,
- is_crowd=[x.get("iscrowd", 0) for x in annos],
- )
- self.overlay_instances(
- labels=labels, boxes=boxes, masks=masks, keypoints=keypts, assigned_colors=colors
- )
-
- sem_seg = dic.get("sem_seg", None)
- if sem_seg is None and "sem_seg_file_name" in dic:
- with PathManager.open(dic["sem_seg_file_name"], "rb") as f:
- sem_seg = Image.open(f)
- sem_seg = np.asarray(sem_seg, dtype="uint8")
- if sem_seg is not None:
- self.draw_sem_seg(sem_seg, area_threshold=0, alpha=0.5)
-
- pan_seg = dic.get("pan_seg", None)
- if pan_seg is None and "pan_seg_file_name" in dic:
- with PathManager.open(dic["pan_seg_file_name"], "rb") as f:
- pan_seg = Image.open(f)
- pan_seg = np.asarray(pan_seg)
- from panopticapi.utils import rgb2id
-
- pan_seg = rgb2id(pan_seg)
- if pan_seg is not None:
- segments_info = dic["segments_info"]
- pan_seg = torch.tensor(pan_seg)
- self.draw_panoptic_seg(pan_seg, segments_info, area_threshold=0, alpha=0.5)
- return self.output
-
- def overlay_instances(
- self,
- *,
- boxes=None,
- labels=None,
- masks=None,
- keypoints=None,
- assigned_colors=None,
- alpha=0.5,
- ):
- """
- Args:
- boxes (Boxes, RotatedBoxes or ndarray): either a :class:`Boxes`,
- or an Nx4 numpy array of XYXY_ABS format for the N objects in a single image,
- or a :class:`RotatedBoxes`,
- or an Nx5 numpy array of (x_center, y_center, width, height, angle_degrees) format
- for the N objects in a single image,
- labels (list[str]): the text to be displayed for each instance.
- masks (masks-like object): Supported types are:
-
- * :class:`detectron2.structures.PolygonMasks`,
- :class:`detectron2.structures.BitMasks`.
- * list[list[ndarray]]: contains the segmentation masks for all objects in one image.
- The first level of the list corresponds to individual instances. The second
- level to all the polygon that compose the instance, and the third level
- to the polygon coordinates. The third level should have the format of
- [x0, y0, x1, y1, ..., xn, yn] (n >= 3).
- * list[ndarray]: each ndarray is a binary mask of shape (H, W).
- * list[dict]: each dict is a COCO-style RLE.
- keypoints (Keypoint or array like): an array-like object of shape (N, K, 3),
- where the N is the number of instances and K is the number of keypoints.
- The last dimension corresponds to (x, y, visibility or score).
- assigned_colors (list[matplotlib.colors]): a list of colors, where each color
- corresponds to each mask or box in the image. Refer to 'matplotlib.colors'
- for full list of formats that the colors are accepted in.
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- num_instances = 0
- if boxes is not None:
- boxes = self._convert_boxes(boxes)
- num_instances = len(boxes)
- if masks is not None:
- masks = self._convert_masks(masks)
- if num_instances:
- assert len(masks) == num_instances
- else:
- num_instances = len(masks)
- if keypoints is not None:
- if num_instances:
- assert len(keypoints) == num_instances
- else:
- num_instances = len(keypoints)
- keypoints = self._convert_keypoints(keypoints)
- if labels is not None:
- assert len(labels) == num_instances
- if assigned_colors is None:
- assigned_colors = [random_color(rgb=True, maximum=1) for _ in range(num_instances)]
- if num_instances == 0:
- return self.output
- if boxes is not None and boxes.shape[1] == 5:
- return self.overlay_rotated_instances(
- boxes=boxes, labels=labels, assigned_colors=assigned_colors
- )
-
- # Display in largest to smallest order to reduce occlusion.
- areas = None
- if boxes is not None:
- areas = np.prod(boxes[:, 2:] - boxes[:, :2], axis=1)
- elif masks is not None:
- areas = np.asarray([x.area() for x in masks])
-
- if areas is not None:
- sorted_idxs = np.argsort(-areas).tolist()
- # Re-order overlapped instances in descending order.
- boxes = boxes[sorted_idxs] if boxes is not None else None
- labels = [labels[k] for k in sorted_idxs] if labels is not None else None
- masks = [masks[idx] for idx in sorted_idxs] if masks is not None else None
- assigned_colors = [assigned_colors[idx] for idx in sorted_idxs]
- keypoints = keypoints[sorted_idxs] if keypoints is not None else None
-
- for i in range(num_instances):
- color = assigned_colors[i]
- if boxes is not None:
- self.draw_box(boxes[i], edge_color=color)
-
- if masks is not None:
- for segment in masks[i].polygons:
- self.draw_polygon(segment.reshape(-1, 2), color, alpha=alpha)
-
- if labels is not None:
- # first get a box
- if boxes is not None:
- x0, y0, x1, y1 = boxes[i]
- text_pos = (x0, y0) # if drawing boxes, put text on the box corner.
- horiz_align = "left"
- elif masks is not None:
- # skip small mask without polygon
- if len(masks[i].polygons) == 0:
- continue
-
- x0, y0, x1, y1 = masks[i].bbox()
-
- # draw text in the center (defined by median) when box is not drawn
- # median is less sensitive to outliers.
- text_pos = np.median(masks[i].mask.nonzero(), axis=1)[::-1]
- horiz_align = "center"
- else:
- continue # drawing the box confidence for keypoints isn't very useful.
- # for small objects, draw text at the side to avoid occlusion
- instance_area = (y1 - y0) * (x1 - x0)
- if (
- instance_area < _SMALL_OBJECT_AREA_THRESH * self.output.scale
- or y1 - y0 < 40 * self.output.scale
- ):
- if y1 >= self.output.height - 5:
- text_pos = (x1, y0)
- else:
- text_pos = (x0, y1)
-
- height_ratio = (y1 - y0) / np.sqrt(self.output.height * self.output.width)
- lighter_color = self._change_color_brightness(color, brightness_factor=0.7)
- font_size = (
- np.clip((height_ratio - 0.02) / 0.08 + 1, 1.2, 2)
- * 0.5
- * self._default_font_size
- )
- self.draw_text(
- labels[i],
- text_pos,
- color=lighter_color,
- horizontal_alignment=horiz_align,
- font_size=font_size,
- )
-
- # draw keypoints
- if keypoints is not None:
- for keypoints_per_instance in keypoints:
- self.draw_and_connect_keypoints(keypoints_per_instance)
-
- return self.output
-
- def overlay_rotated_instances(self, boxes=None, labels=None, assigned_colors=None):
- """
- Args:
- boxes (ndarray): an Nx5 numpy array of
- (x_center, y_center, width, height, angle_degrees) format
- for the N objects in a single image.
- labels (list[str]): the text to be displayed for each instance.
- assigned_colors (list[matplotlib.colors]): a list of colors, where each color
- corresponds to each mask or box in the image. Refer to 'matplotlib.colors'
- for full list of formats that the colors are accepted in.
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- num_instances = len(boxes)
-
- if assigned_colors is None:
- assigned_colors = [random_color(rgb=True, maximum=1) for _ in range(num_instances)]
- if num_instances == 0:
- return self.output
-
- # Display in largest to smallest order to reduce occlusion.
- if boxes is not None:
- areas = boxes[:, 2] * boxes[:, 3]
-
- sorted_idxs = np.argsort(-areas).tolist()
- # Re-order overlapped instances in descending order.
- boxes = boxes[sorted_idxs]
- labels = [labels[k] for k in sorted_idxs] if labels is not None else None
- colors = [assigned_colors[idx] for idx in sorted_idxs]
-
- for i in range(num_instances):
- self.draw_rotated_box_with_label(
- boxes[i], edge_color=colors[i], label=labels[i] if labels is not None else None
- )
-
- return self.output
-
- def draw_and_connect_keypoints(self, keypoints):
- """
- Draws keypoints of an instance and follows the rules for keypoint connections
- to draw lines between appropriate keypoints. This follows color heuristics for
- line color.
-
- Args:
- keypoints (Tensor): a tensor of shape (K, 3), where K is the number of keypoints
- and the last dimension corresponds to (x, y, probability).
-
- Returns:
- output (VisImage): image object with visualizations.
- """
- visible = {}
- keypoint_names = self.metadata.get("keypoint_names")
- for idx, keypoint in enumerate(keypoints):
- # draw keypoint
- x, y, prob = keypoint
- if prob > self.keypoint_threshold:
- self.draw_circle((x, y), color=_RED)
- if keypoint_names:
- keypoint_name = keypoint_names[idx]
- visible[keypoint_name] = (x, y)
-
- if self.metadata.get("keypoint_connection_rules"):
- for kp0, kp1, color in self.metadata.keypoint_connection_rules:
- if kp0 in visible and kp1 in visible:
- x0, y0 = visible[kp0]
- x1, y1 = visible[kp1]
- color = tuple(x / 255.0 for x in color)
- self.draw_line([x0, x1], [y0, y1], color=color)
-
- # draw lines from nose to mid-shoulder and mid-shoulder to mid-hip
- # Note that this strategy is specific to person keypoints.
- # For other keypoints, it should just do nothing
- try:
- ls_x, ls_y = visible["left_shoulder"]
- rs_x, rs_y = visible["right_shoulder"]
- mid_shoulder_x, mid_shoulder_y = (ls_x + rs_x) / 2, (ls_y + rs_y) / 2
- except KeyError:
- pass
- else:
- # draw line from nose to mid-shoulder
- nose_x, nose_y = visible.get("nose", (None, None))
- if nose_x is not None:
- self.draw_line([nose_x, mid_shoulder_x], [nose_y, mid_shoulder_y], color=_RED)
-
- try:
- # draw line from mid-shoulder to mid-hip
- lh_x, lh_y = visible["left_hip"]
- rh_x, rh_y = visible["right_hip"]
- except KeyError:
- pass
- else:
- mid_hip_x, mid_hip_y = (lh_x + rh_x) / 2, (lh_y + rh_y) / 2
- self.draw_line([mid_hip_x, mid_shoulder_x], [mid_hip_y, mid_shoulder_y], color=_RED)
- return self.output
-
- """
- Primitive drawing functions:
- """
-
- def draw_text(
- self,
- text,
- position,
- *,
- font_size=None,
- color="g",
- horizontal_alignment="center",
- rotation=0,
- ):
- """
- Args:
- text (str): class label
- position (tuple): a tuple of the x and y coordinates to place text on image.
- font_size (int, optional): font of the text. If not provided, a font size
- proportional to the image width is calculated and used.
- color: color of the text. Refer to `matplotlib.colors` for full list
- of formats that are accepted.
- horizontal_alignment (str): see `matplotlib.text.Text`
- rotation: rotation angle in degrees CCW
-
- Returns:
- output (VisImage): image object with text drawn.
- """
- if not font_size:
- font_size = self._default_font_size
-
- # since the text background is dark, we don't want the text to be dark
- color = np.maximum(list(mplc.to_rgb(color)), 0.2)
- color[np.argmax(color)] = max(0.8, np.max(color))
-
- x, y = position
- self.output.ax.text(
- x,
- y,
- text,
- size=font_size * self.output.scale,
- family="sans-serif",
- bbox={"facecolor": "black", "alpha": 0.8, "pad": 0.7, "edgecolor": "none"},
- verticalalignment="top",
- horizontalalignment=horizontal_alignment,
- color=color,
- zorder=10,
- rotation=rotation,
- )
- return self.output
-
- def draw_box(self, box_coord, alpha=0.5, edge_color="g", line_style="-"):
- """
- Args:
- box_coord (tuple): a tuple containing x0, y0, x1, y1 coordinates, where x0 and y0
- are the coordinates of the image's top left corner. x1 and y1 are the
- coordinates of the image's bottom right corner.
- alpha (float): blending efficient. Smaller values lead to more transparent masks.
- edge_color: color of the outline of the box. Refer to `matplotlib.colors`
- for full list of formats that are accepted.
- line_style (string): the string to use to create the outline of the boxes.
-
- Returns:
- output (VisImage): image object with box drawn.
- """
- x0, y0, x1, y1 = box_coord
- width = x1 - x0
- height = y1 - y0
-
- linewidth = max(self._default_font_size / 4, 1)
-
- self.output.ax.add_patch(
- mpl.patches.Rectangle(
- (x0, y0),
- width,
- height,
- fill=False,
- edgecolor=edge_color,
- linewidth=linewidth * self.output.scale,
- alpha=alpha,
- linestyle=line_style,
- )
- )
- return self.output
-
- def draw_rotated_box_with_label(
- self, rotated_box, alpha=0.5, edge_color="g", line_style="-", label=None
- ):
- """
- Draw a rotated box with label on its top-left corner.
-
- Args:
- rotated_box (tuple): a tuple containing (cnt_x, cnt_y, w, h, angle),
- where cnt_x and cnt_y are the center coordinates of the box.
- w and h are the width and height of the box. angle represents how
- many degrees the box is rotated CCW with regard to the 0-degree box.
- alpha (float): blending efficient. Smaller values lead to more transparent masks.
- edge_color: color of the outline of the box. Refer to `matplotlib.colors`
- for full list of formats that are accepted.
- line_style (string): the string to use to create the outline of the boxes.
- label (string): label for rotated box. It will not be rendered when set to None.
-
- Returns:
- output (VisImage): image object with box drawn.
- """
- cnt_x, cnt_y, w, h, angle = rotated_box
- area = w * h
- # use thinner lines when the box is small
- linewidth = self._default_font_size / (
- 6 if area < _SMALL_OBJECT_AREA_THRESH * self.output.scale else 3
- )
-
- theta = angle * math.pi / 180.0
- c = math.cos(theta)
- s = math.sin(theta)
- rect = [(-w / 2, h / 2), (-w / 2, -h / 2), (w / 2, -h / 2), (w / 2, h / 2)]
- # x: left->right ; y: top->down
- rotated_rect = [(s * yy + c * xx + cnt_x, c * yy - s * xx + cnt_y) for (xx, yy) in rect]
- for k in range(4):
- j = (k + 1) % 4
- self.draw_line(
- [rotated_rect[k][0], rotated_rect[j][0]],
- [rotated_rect[k][1], rotated_rect[j][1]],
- color=edge_color,
- linestyle="--" if k == 1 else line_style,
- linewidth=linewidth,
- )
-
- if label is not None:
- text_pos = rotated_rect[1] # topleft corner
-
- height_ratio = h / np.sqrt(self.output.height * self.output.width)
- label_color = self._change_color_brightness(edge_color, brightness_factor=0.7)
- font_size = (
- np.clip((height_ratio - 0.02) / 0.08 + 1, 1.2, 2) * 0.5 * self._default_font_size
- )
- self.draw_text(label, text_pos, color=label_color, font_size=font_size, rotation=angle)
-
- return self.output
-
- def draw_circle(self, circle_coord, color, radius=3):
- """
- Args:
- circle_coord (list(int) or tuple(int)): contains the x and y coordinates
- of the center of the circle.
- color: color of the polygon. Refer to `matplotlib.colors` for a full list of
- formats that are accepted.
- radius (int): radius of the circle.
-
- Returns:
- output (VisImage): image object with box drawn.
- """
- x, y = circle_coord
- self.output.ax.add_patch(
- mpl.patches.Circle(circle_coord, radius=radius, fill=True, color=color)
- )
- return self.output
-
- def draw_line(self, x_data, y_data, color, linestyle="-", linewidth=None):
- """
- Args:
- x_data (list[int]): a list containing x values of all the points being drawn.
- Length of list should match the length of y_data.
- y_data (list[int]): a list containing y values of all the points being drawn.
- Length of list should match the length of x_data.
- color: color of the line. Refer to `matplotlib.colors` for a full list of
- formats that are accepted.
- linestyle: style of the line. Refer to `matplotlib.lines.Line2D`
- for a full list of formats that are accepted.
- linewidth (float or None): width of the line. When it's None,
- a default value will be computed and used.
-
- Returns:
- output (VisImage): image object with line drawn.
- """
- if linewidth is None:
- linewidth = self._default_font_size / 3
- linewidth = max(linewidth, 1)
- self.output.ax.add_line(
- mpl.lines.Line2D(
- x_data,
- y_data,
- linewidth=linewidth * self.output.scale,
- color=color,
- linestyle=linestyle,
- )
- )
- return self.output
-
- def draw_binary_mask(
- self, binary_mask, color=None, *, edge_color=None, text=None, alpha=0.5, area_threshold=0
- ):
- """
- Args:
- binary_mask (ndarray): numpy array of shape (H, W), where H is the image height and
- W is the image width. Each value in the array is either a 0 or 1 value of uint8
- type.
- color: color of the mask. Refer to `matplotlib.colors` for a full list of
- formats that are accepted. If None, will pick a random color.
- edge_color: color of the polygon edges. Refer to `matplotlib.colors` for a
- full list of formats that are accepted.
- text (str): if None, will be drawn in the object's center of mass.
- alpha (float): blending efficient. Smaller values lead to more transparent masks.
- area_threshold (float): a connected component small than this will not be shown.
-
- Returns:
- output (VisImage): image object with mask drawn.
- """
- if color is None:
- color = random_color(rgb=True, maximum=1)
- color = mplc.to_rgb(color)
-
- has_valid_segment = False
- binary_mask = binary_mask.astype("uint8") # opencv needs uint8
- mask = GenericMask(binary_mask, self.output.height, self.output.width)
- shape2d = (binary_mask.shape[0], binary_mask.shape[1])
-
- if not mask.has_holes:
- # draw polygons for regular masks
- for segment in mask.polygons:
- area = mask_util.area(mask_util.frPyObjects([segment], shape2d[0], shape2d[1]))
- if area < (area_threshold or 0):
- continue
- has_valid_segment = True
- segment = segment.reshape(-1, 2)
- self.draw_polygon(segment, color=color, edge_color=edge_color, alpha=alpha)
- else:
- # TODO: Use Path/PathPatch to draw vector graphics:
- # https://stackoverflow.com/questions/8919719/how-to-plot-a-complex-polygon
- rgba = np.zeros(shape2d + (4,), dtype="float32")
- rgba[:, :, :3] = color
- rgba[:, :, 3] = (mask.mask == 1).astype("float32") * alpha
- has_valid_segment = True
- self.output.ax.imshow(rgba, extent=(0, self.output.width, self.output.height, 0))
-
- if text is not None and has_valid_segment:
- # TODO sometimes drawn on wrong objects. the heuristics here can improve.
- lighter_color = self._change_color_brightness(color, brightness_factor=0.7)
- _num_cc, cc_labels, stats, centroids = cv2.connectedComponentsWithStats(binary_mask, 8)
- largest_component_id = np.argmax(stats[1:, -1]) + 1
-
- # draw text on the largest component, as well as other very large components.
- for cid in range(1, _num_cc):
- if cid == largest_component_id or stats[cid, -1] > _LARGE_MASK_AREA_THRESH:
- # median is more stable than centroid
- # center = centroids[largest_component_id]
- center = np.median((cc_labels == cid).nonzero(), axis=1)[::-1]
- self.draw_text(text, center, color=lighter_color)
- return self.output
-
- def draw_polygon(self, segment, color, edge_color=None, alpha=0.5):
- """
- Args:
- segment: numpy array of shape Nx2, containing all the points in the polygon.
- color: color of the polygon. Refer to `matplotlib.colors` for a full list of
- formats that are accepted.
- edge_color: color of the polygon edges. Refer to `matplotlib.colors` for a
- full list of formats that are accepted. If not provided, a darker shade
- of the polygon color will be used instead.
- alpha (float): blending efficient. Smaller values lead to more transparent masks.
-
- Returns:
- output (VisImage): image object with polygon drawn.
- """
- if edge_color is None:
- # make edge color darker than the polygon color
- if alpha > 0.8:
- edge_color = self._change_color_brightness(color, brightness_factor=-0.7)
- else:
- edge_color = color
- edge_color = mplc.to_rgb(edge_color) + (1,)
-
- polygon = mpl.patches.Polygon(
- segment,
- fill=True,
- facecolor=mplc.to_rgb(color) + (alpha,),
- edgecolor=edge_color,
- linewidth=max(self._default_font_size // 15 * self.output.scale, 1),
- )
- self.output.ax.add_patch(polygon)
- return self.output
-
- """
- Internal methods:
- """
-
- def _jitter(self, color):
- """
- Randomly modifies given color to produce a slightly different color than the color given.
-
- Args:
- color (tuple[double]): a tuple of 3 elements, containing the RGB values of the color
- picked. The values in the list are in the [0.0, 1.0] range.
-
- Returns:
- jittered_color (tuple[double]): a tuple of 3 elements, containing the RGB values of the
- color after being jittered. The values in the list are in the [0.0, 1.0] range.
- """
- color = mplc.to_rgb(color)
- vec = np.random.rand(3)
- # better to do it in another color space
- vec = vec / np.linalg.norm(vec) * 0.5
- res = np.clip(vec + color, 0, 1)
- return tuple(res)
-
- def _create_grayscale_image(self, mask=None):
- """
- Create a grayscale version of the original image.
- The colors in masked area, if given, will be kept.
- """
- img_bw = self.img.astype("f4").mean(axis=2)
- img_bw = np.stack([img_bw] * 3, axis=2)
- if mask is not None:
- img_bw[mask] = self.img[mask]
- return img_bw
-
- def _change_color_brightness(self, color, brightness_factor):
- """
- Depending on the brightness_factor, gives a lighter or darker color i.e. a color with
- less or more saturation than the original color.
-
- Args:
- color: color of the polygon. Refer to `matplotlib.colors` for a full list of
- formats that are accepted.
- brightness_factor (float): a value in [-1.0, 1.0] range. A lightness factor of
- 0 will correspond to no change, a factor in [-1.0, 0) range will result in
- a darker color and a factor in (0, 1.0] range will result in a lighter color.
-
- Returns:
- modified_color (tuple[double]): a tuple containing the RGB values of the
- modified color. Each value in the tuple is in the [0.0, 1.0] range.
- """
- assert brightness_factor >= -1.0 and brightness_factor <= 1.0
- color = mplc.to_rgb(color)
- polygon_color = colorsys.rgb_to_hls(*mplc.to_rgb(color))
- modified_lightness = polygon_color[1] + (brightness_factor * polygon_color[1])
- modified_lightness = 0.0 if modified_lightness < 0.0 else modified_lightness
- modified_lightness = 1.0 if modified_lightness > 1.0 else modified_lightness
- modified_color = colorsys.hls_to_rgb(polygon_color[0], modified_lightness, polygon_color[2])
- return modified_color
-
- def _convert_boxes(self, boxes):
- """
- Convert different format of boxes to an NxB array, where B = 4 or 5 is the box dimension.
- """
- if isinstance(boxes, Boxes) or isinstance(boxes, RotatedBoxes):
- return boxes.tensor.detach().numpy()
- else:
- return np.asarray(boxes)
-
- def _convert_masks(self, masks_or_polygons):
- """
- Convert different format of masks or polygons to a tuple of masks and polygons.
-
- Returns:
- list[GenericMask]:
- """
-
- m = masks_or_polygons
- if isinstance(m, PolygonMasks):
- m = m.polygons
- if isinstance(m, BitMasks):
- m = m.tensor.numpy()
- if isinstance(m, torch.Tensor):
- m = m.numpy()
- ret = []
- for x in m:
- if isinstance(x, GenericMask):
- ret.append(x)
- else:
- ret.append(GenericMask(x, self.output.height, self.output.width))
- return ret
-
- def _convert_keypoints(self, keypoints):
- if isinstance(keypoints, Keypoints):
- keypoints = keypoints.tensor
- keypoints = np.asarray(keypoints)
- return keypoints
-
- def get_output(self):
- """
- Returns:
- output (VisImage): the image output containing the visualizations added
- to the image.
- """
- return self.output
diff --git a/magic_pdf/model/pek_sub_modules/post_process.py b/magic_pdf/model/pek_sub_modules/post_process.py
deleted file mode 100644
index aa050b612..000000000
--- a/magic_pdf/model/pek_sub_modules/post_process.py
+++ /dev/null
@@ -1,36 +0,0 @@
-import re
-
-def layout_rm_equation(layout_res):
- rm_idxs = []
- for idx, ele in enumerate(layout_res['layout_dets']):
- if ele['category_id'] == 10:
- rm_idxs.append(idx)
-
- for idx in rm_idxs[::-1]:
- del layout_res['layout_dets'][idx]
- return layout_res
-
-
-def get_croped_image(image_pil, bbox):
- x_min, y_min, x_max, y_max = bbox
- croped_img = image_pil.crop((x_min, y_min, x_max, y_max))
- return croped_img
-
-
-def latex_rm_whitespace(s: str):
- """Remove unnecessary whitespace from LaTeX code.
- """
- text_reg = r'(\\(operatorname|mathrm|text|mathbf)\s?\*? {.*?})'
- letter = '[a-zA-Z]'
- noletter = '[\W_^\d]'
- names = [x[0].replace(' ', '') for x in re.findall(text_reg, s)]
- s = re.sub(text_reg, lambda match: str(names.pop(0)), s)
- news = s
- while True:
- s = news
- news = re.sub(r'(?!\\ )(%s)\s+?(%s)' % (noletter, noletter), r'\1\2', s)
- news = re.sub(r'(?!\\ )(%s)\s+?(%s)' % (noletter, letter), r'\1\2', news)
- news = re.sub(r'(%s)\s+?(%s)' % (letter, noletter), r'\1\2', news)
- if news == s:
- break
- return s
\ No newline at end of file
diff --git a/magic_pdf/model/pek_sub_modules/self_modify.py b/magic_pdf/model/pek_sub_modules/self_modify.py
deleted file mode 100644
index c39c1a9cd..000000000
--- a/magic_pdf/model/pek_sub_modules/self_modify.py
+++ /dev/null
@@ -1,388 +0,0 @@
-import time
-import copy
-import base64
-import cv2
-import numpy as np
-from io import BytesIO
-from PIL import Image
-
-from paddleocr import PaddleOCR
-from paddleocr.ppocr.utils.logging import get_logger
-from paddleocr.ppocr.utils.utility import check_and_read, alpha_to_color, binarize_img
-from paddleocr.tools.infer.utility import draw_ocr_box_txt, get_rotate_crop_image, get_minarea_rect_crop
-
-from magic_pdf.libs.boxbase import __is_overlaps_y_exceeds_threshold
-from magic_pdf.pre_proc.ocr_dict_merge import merge_spans_to_line
-
-logger = get_logger()
-
-
-def img_decode(content: bytes):
- np_arr = np.frombuffer(content, dtype=np.uint8)
- return cv2.imdecode(np_arr, cv2.IMREAD_UNCHANGED)
-
-
-def check_img(img):
- if isinstance(img, bytes):
- img = img_decode(img)
- if isinstance(img, str):
- image_file = img
- img, flag_gif, flag_pdf = check_and_read(image_file)
- if not flag_gif and not flag_pdf:
- with open(image_file, 'rb') as f:
- img_str = f.read()
- img = img_decode(img_str)
- if img is None:
- try:
- buf = BytesIO()
- image = BytesIO(img_str)
- im = Image.open(image)
- rgb = im.convert('RGB')
- rgb.save(buf, 'jpeg')
- buf.seek(0)
- image_bytes = buf.read()
- data_base64 = str(base64.b64encode(image_bytes),
- encoding="utf-8")
- image_decode = base64.b64decode(data_base64)
- img_array = np.frombuffer(image_decode, np.uint8)
- img = cv2.imdecode(img_array, cv2.IMREAD_COLOR)
- except:
- logger.error("error in loading image:{}".format(image_file))
- return None
- if img is None:
- logger.error("error in loading image:{}".format(image_file))
- return None
- if isinstance(img, np.ndarray) and len(img.shape) == 2:
- img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
-
- return img
-
-
-def sorted_boxes(dt_boxes):
- """
- Sort text boxes in order from top to bottom, left to right
- args:
- dt_boxes(array):detected text boxes with shape [4, 2]
- return:
- sorted boxes(array) with shape [4, 2]
- """
- num_boxes = dt_boxes.shape[0]
- sorted_boxes = sorted(dt_boxes, key=lambda x: (x[0][1], x[0][0]))
- _boxes = list(sorted_boxes)
-
- for i in range(num_boxes - 1):
- for j in range(i, -1, -1):
- if abs(_boxes[j + 1][0][1] - _boxes[j][0][1]) < 10 and \
- (_boxes[j + 1][0][0] < _boxes[j][0][0]):
- tmp = _boxes[j]
- _boxes[j] = _boxes[j + 1]
- _boxes[j + 1] = tmp
- else:
- break
- return _boxes
-
-
-def bbox_to_points(bbox):
- """ 将bbox格式转换为四个顶点的数组 """
- x0, y0, x1, y1 = bbox
- return np.array([[x0, y0], [x1, y0], [x1, y1], [x0, y1]]).astype('float32')
-
-
-def points_to_bbox(points):
- """ 将四个顶点的数组转换为bbox格式 """
- x0, y0 = points[0]
- x1, _ = points[1]
- _, y1 = points[2]
- return [x0, y0, x1, y1]
-
-
-def merge_intervals(intervals):
- # Sort the intervals based on the start value
- intervals.sort(key=lambda x: x[0])
-
- merged = []
- for interval in intervals:
- # If the list of merged intervals is empty or if the current
- # interval does not overlap with the previous, simply append it.
- if not merged or merged[-1][1] < interval[0]:
- merged.append(interval)
- else:
- # Otherwise, there is overlap, so we merge the current and previous intervals.
- merged[-1][1] = max(merged[-1][1], interval[1])
-
- return merged
-
-
-def remove_intervals(original, masks):
- # Merge all mask intervals
- merged_masks = merge_intervals(masks)
-
- result = []
- original_start, original_end = original
-
- for mask in merged_masks:
- mask_start, mask_end = mask
-
- # If the mask starts after the original range, ignore it
- if mask_start > original_end:
- continue
-
- # If the mask ends before the original range starts, ignore it
- if mask_end < original_start:
- continue
-
- # Remove the masked part from the original range
- if original_start < mask_start:
- result.append([original_start, mask_start - 1])
-
- original_start = max(mask_end + 1, original_start)
-
- # Add the remaining part of the original range, if any
- if original_start <= original_end:
- result.append([original_start, original_end])
-
- return result
-
-
-def update_det_boxes(dt_boxes, mfd_res):
- new_dt_boxes = []
- for text_box in dt_boxes:
- text_bbox = points_to_bbox(text_box)
- masks_list = []
- for mf_box in mfd_res:
- mf_bbox = mf_box['bbox']
- if __is_overlaps_y_exceeds_threshold(text_bbox, mf_bbox):
- masks_list.append([mf_bbox[0], mf_bbox[2]])
- text_x_range = [text_bbox[0], text_bbox[2]]
- text_remove_mask_range = remove_intervals(text_x_range, masks_list)
- temp_dt_box = []
- for text_remove_mask in text_remove_mask_range:
- temp_dt_box.append(bbox_to_points([text_remove_mask[0], text_bbox[1], text_remove_mask[1], text_bbox[3]]))
- if len(temp_dt_box) > 0:
- new_dt_boxes.extend(temp_dt_box)
- return new_dt_boxes
-
-
-def merge_overlapping_spans(spans):
- """
- Merges overlapping spans on the same line.
-
- :param spans: A list of span coordinates [(x1, y1, x2, y2), ...]
- :return: A list of merged spans
- """
- # Return an empty list if the input spans list is empty
- if not spans:
- return []
-
- # Sort spans by their starting x-coordinate
- spans.sort(key=lambda x: x[0])
-
- # Initialize the list of merged spans
- merged = []
- for span in spans:
- # Unpack span coordinates
- x1, y1, x2, y2 = span
- # If the merged list is empty or there's no horizontal overlap, add the span directly
- if not merged or merged[-1][2] < x1:
- merged.append(span)
- else:
- # If there is horizontal overlap, merge the current span with the previous one
- last_span = merged.pop()
- # Update the merged span's top-left corner to the smaller (x1, y1) and bottom-right to the larger (x2, y2)
- x1 = min(last_span[0], x1)
- y1 = min(last_span[1], y1)
- x2 = max(last_span[2], x2)
- y2 = max(last_span[3], y2)
- # Add the merged span back to the list
- merged.append((x1, y1, x2, y2))
-
- # Return the list of merged spans
- return merged
-
-
-def merge_det_boxes(dt_boxes):
- """
- Merge detection boxes.
-
- This function takes a list of detected bounding boxes, each represented by four corner points.
- The goal is to merge these bounding boxes into larger text regions.
-
- Parameters:
- dt_boxes (list): A list containing multiple text detection boxes, where each box is defined by four corner points.
-
- Returns:
- list: A list containing the merged text regions, where each region is represented by four corner points.
- """
- # Convert the detection boxes into a dictionary format with bounding boxes and type
- dt_boxes_dict_list = []
- for text_box in dt_boxes:
- text_bbox = points_to_bbox(text_box)
- text_box_dict = {
- 'bbox': text_bbox,
- 'type': 'text',
- }
- dt_boxes_dict_list.append(text_box_dict)
-
- # Merge adjacent text regions into lines
- lines = merge_spans_to_line(dt_boxes_dict_list)
-
- # Initialize a new list for storing the merged text regions
- new_dt_boxes = []
- for line in lines:
- line_bbox_list = []
- for span in line:
- line_bbox_list.append(span['bbox'])
-
- # Merge overlapping text regions within the same line
- merged_spans = merge_overlapping_spans(line_bbox_list)
-
- # Convert the merged text regions back to point format and add them to the new detection box list
- for span in merged_spans:
- new_dt_boxes.append(bbox_to_points(span))
-
- return new_dt_boxes
-
-
-class ModifiedPaddleOCR(PaddleOCR):
- def ocr(self, img, det=True, rec=True, cls=True, bin=False, inv=False, mfd_res=None, alpha_color=(255, 255, 255)):
- """
- OCR with PaddleOCR
- args：
- img: img for OCR, support ndarray, img_path and list or ndarray
- det: use text detection or not. If False, only rec will be exec. Default is True
- rec: use text recognition or not. If False, only det will be exec. Default is True
- cls: use angle classifier or not. Default is True. If True, the text with rotation of 180 degrees can be recognized. If no text is rotated by 180 degrees, use cls=False to get better performance. Text with rotation of 90 or 270 degrees can be recognized even if cls=False.
- bin: binarize image to black and white. Default is False.
- inv: invert image colors. Default is False.
- alpha_color: set RGB color Tuple for transparent parts replacement. Default is pure white.
- """
- assert isinstance(img, (np.ndarray, list, str, bytes))
- if isinstance(img, list) and det == True:
- logger.error('When input a list of images, det must be false')
- exit(0)
- if cls == True and self.use_angle_cls == False:
- pass
- # logger.warning(
- # 'Since the angle classifier is not initialized, it will not be used during the forward process'
- # )
-
- img = check_img(img)
- # for infer pdf file
- if isinstance(img, list):
- if self.page_num > len(img) or self.page_num == 0:
- self.page_num = len(img)
- imgs = img[:self.page_num]
- else:
- imgs = [img]
-
- def preprocess_image(_image):
- _image = alpha_to_color(_image, alpha_color)
- if inv:
- _image = cv2.bitwise_not(_image)
- if bin:
- _image = binarize_img(_image)
- return _image
-
- if det and rec:
- ocr_res = []
- for idx, img in enumerate(imgs):
- img = preprocess_image(img)
- dt_boxes, rec_res, _ = self.__call__(img, cls, mfd_res=mfd_res)
- if not dt_boxes and not rec_res:
- ocr_res.append(None)
- continue
- tmp_res = [[box.tolist(), res]
- for box, res in zip(dt_boxes, rec_res)]
- ocr_res.append(tmp_res)
- return ocr_res
- elif det and not rec:
- ocr_res = []
- for idx, img in enumerate(imgs):
- img = preprocess_image(img)
- dt_boxes, elapse = self.text_detector(img)
- if not dt_boxes:
- ocr_res.append(None)
- continue
- tmp_res = [box.tolist() for box in dt_boxes]
- ocr_res.append(tmp_res)
- return ocr_res
- else:
- ocr_res = []
- cls_res = []
- for idx, img in enumerate(imgs):
- if not isinstance(img, list):
- img = preprocess_image(img)
- img = [img]
- if self.use_angle_cls and cls:
- img, cls_res_tmp, elapse = self.text_classifier(img)
- if not rec:
- cls_res.append(cls_res_tmp)
- rec_res, elapse = self.text_recognizer(img)
- ocr_res.append(rec_res)
- if not rec:
- return cls_res
- return ocr_res
-
- def __call__(self, img, cls=True, mfd_res=None):
- time_dict = {'det': 0, 'rec': 0, 'cls': 0, 'all': 0}
-
- if img is None:
- logger.debug("no valid image provided")
- return None, None, time_dict
-
- start = time.time()
- ori_im = img.copy()
- dt_boxes, elapse = self.text_detector(img)
- time_dict['det'] = elapse
-
- if dt_boxes is None:
- logger.debug("no dt_boxes found, elapsed : {}".format(elapse))
- end = time.time()
- time_dict['all'] = end - start
- return None, None, time_dict
- else:
- logger.debug("dt_boxes num : {}, elapsed : {}".format(
- len(dt_boxes), elapse))
- img_crop_list = []
-
- dt_boxes = sorted_boxes(dt_boxes)
-
- dt_boxes = merge_det_boxes(dt_boxes)
-
- if mfd_res:
- bef = time.time()
- dt_boxes = update_det_boxes(dt_boxes, mfd_res)
- aft = time.time()
- logger.debug("split text box by formula, new dt_boxes num : {}, elapsed : {}".format(
- len(dt_boxes), aft - bef))
-
- for bno in range(len(dt_boxes)):
- tmp_box = copy.deepcopy(dt_boxes[bno])
- if self.args.det_box_type == "quad":
- img_crop = get_rotate_crop_image(ori_im, tmp_box)
- else:
- img_crop = get_minarea_rect_crop(ori_im, tmp_box)
- img_crop_list.append(img_crop)
- if self.use_angle_cls and cls:
- img_crop_list, angle_list, elapse = self.text_classifier(
- img_crop_list)
- time_dict['cls'] = elapse
- logger.debug("cls num : {}, elapsed : {}".format(
- len(img_crop_list), elapse))
-
- rec_res, elapse = self.text_recognizer(img_crop_list)
- time_dict['rec'] = elapse
- logger.debug("rec_res num : {}, elapsed : {}".format(
- len(rec_res), elapse))
- if self.args.save_crop_res:
- self.draw_crop_rec_res(self.args.crop_res_save_dir, img_crop_list,
- rec_res)
- filter_boxes, filter_rec_res = [], []
- for box, rec_result in zip(dt_boxes, rec_res):
- text, score = rec_result
- if score >= self.drop_score:
- filter_boxes.append(box)
- filter_rec_res.append(rec_result)
- end = time.time()
- time_dict['all'] = end - start
- return filter_boxes, filter_rec_res, time_dict
\ No newline at end of file
diff --git a/magic_pdf/model/pek_sub_modules/structeqtable/StructTableModel.py b/magic_pdf/model/pek_sub_modules/structeqtable/StructTableModel.py
deleted file mode 100644
index 2d1ce584f..000000000
--- a/magic_pdf/model/pek_sub_modules/structeqtable/StructTableModel.py
+++ /dev/null
@@ -1,21 +0,0 @@
-from struct_eqtable.model import StructTable
-from pypandoc import convert_text
-class StructTableModel:
- def __init__(self, model_path, max_new_tokens=2048, max_time=400, device = 'cpu'):
- # init
- self.model_path = model_path
- self.max_new_tokens = max_new_tokens # maximum output tokens length
- self.max_time = max_time # timeout for processing in seconds
- if device == 'cuda':
- self.model = StructTable(self.model_path, self.max_new_tokens, self.max_time).cuda()
- else:
- self.model = StructTable(self.model_path, self.max_new_tokens, self.max_time)
-
- def image2latex(self, image) -> str:
- table_latex = self.model.forward(image)
- return table_latex
-
- def image2html(self, image) -> str:
- table_latex = self.image2latex(image)
- table_html = convert_text(table_latex, 'html', format='latex')
- return table_html
diff --git a/magic_pdf/model/pek_sub_modules/structeqtable/__init__.py b/magic_pdf/model/pek_sub_modules/structeqtable/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/model/ppTableModel.py b/magic_pdf/model/ppTableModel.py
deleted file mode 100644
index 933f31a05..000000000
--- a/magic_pdf/model/ppTableModel.py
+++ /dev/null
@@ -1,67 +0,0 @@
-from paddleocr.ppstructure.table.predict_table import TableSystem
-from paddleocr.ppstructure.utility import init_args
-from magic_pdf.libs.Constants import *
-import os
-from PIL import Image
-import numpy as np
-
-
-class ppTableModel(object):
- """
- This class is responsible for converting image of table into HTML format using a pre-trained model.
-
- Attributes:
- - table_sys: An instance of TableSystem initialized with parsed arguments.
-
- Methods:
- - __init__(config): Initializes the model with configuration parameters.
- - img2html(image): Converts a PIL Image or NumPy array to HTML string.
- - parse_args(**kwargs): Parses configuration arguments.
- """
-
- def __init__(self, config):
- """
- Parameters:
- - config (dict): Configuration dictionary containing model_dir and device.
- """
- args = self.parse_args(**config)
- self.table_sys = TableSystem(args)
-
- def img2html(self, image):
- """
- Parameters:
- - image (PIL.Image or np.ndarray): The image of the table to be converted.
-
- Return:
- - HTML (str): A string representing the HTML structure with content of the table.
- """
- if isinstance(image, Image.Image):
- image = np.array(image)
- pred_res, _ = self.table_sys(image)
- pred_html = pred_res["html"]
- res = '' + pred_html.replace("", "").replace("
",
- "") + "
= layout_right - right_tail_distance and next_line and next_line['bbox'][0] == layout_left: # 现在这行到了行尾沾满，下一行存在且顶格。
- para.append(line)
- else:
- para.append(line)
- paras.append(para)
- para = []
- else: # 其他，图片、表格、行间公式，各自占一段
- if len(para)>0: # 先把之前的段落加入到结果中
- paras.append(para)
- para = []
- paras.append([line]) # 再把当前行加入到结果中。当前行为行间公式、图、表等。
- para = []
-
- if len(para)>0:
- paras.append(para)
- para = []
-
- list_info.append(layout_list_info)
- layout_paras.append(paras)
- paras = []
-
-
- return layout_paras, list_info
-
-def __connect_list_inter_layout(layout_paras, new_layout_bbox, layout_list_info, page_num, lang):
- """
- 如果上个layout的最后一个段落是列表，下一个layout的第一个段落也是列表，那么将他们连接起来。 TODO 因为没有区分列表和段落，所以这个方法暂时不实现。
- 根据layout_list_info判断是不是列表。，下个layout的第一个段如果不是列表，那么看他们是否有几行都有相同的缩进。
- """
- if len(layout_paras)==0 or len(layout_list_info)==0: # 0的时候最后的return 会出错
- return layout_paras, [False, False]
-
- for i in range(1, len(layout_paras)):
- pre_layout_list_info = layout_list_info[i-1]
- next_layout_list_info = layout_list_info[i]
- pre_last_para = layout_paras[i-1][-1]
- next_paras = layout_paras[i]
- next_first_para = next_paras[0]
-
- if pre_layout_list_info[1] and not next_layout_list_info[0]: # 前一个是列表结尾，后一个是非列表开头，此时检测是否有相同的缩进
- logger.info(f"连接page {page_num} 内的list")
- # 向layout_paras[i] 寻找开头具有相同缩进的连续的行
- may_list_lines = []
- for j in range(len(next_paras)):
- line = next_paras[j]
- if len(line)==1: # 只可能是一行，多行情况再需要分析了
- if line[0]['bbox'][0] > __find_layout_bbox_by_line(line[0]['bbox'], new_layout_bbox)[0]:
- may_list_lines.append(line[0])
- else:
- break
- else:
- break
- # 如果这些行的缩进是相等的，那么连到上一个layout的最后一个段落上。
- if len(may_list_lines)>0 and len(set([x['bbox'][0] for x in may_list_lines]))==1:
- pre_last_para.extend(may_list_lines)
- layout_paras[i] = layout_paras[i][len(may_list_lines):]
-
- return layout_paras, [layout_list_info[0][0], layout_list_info[-1][1]] # 同时还返回了这个页面级别的开头、结尾是不是列表的信息
-
-
-def __connect_list_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox, pre_page_list_info, next_page_list_info, page_num, lang):
- """
- 如果上个layout的最后一个段落是列表，下一个layout的第一个段落也是列表，那么将他们连接起来。 TODO 因为没有区分列表和段落，所以这个方法暂时不实现。
- 根据layout_list_info判断是不是列表。，下个layout的第一个段如果不是列表，那么看他们是否有几行都有相同的缩进。
- """
- if len(pre_page_paras)==0 or len(next_page_paras)==0: # 0的时候最后的return 会出错
- return False
-
- if pre_page_list_info[1] and not next_page_list_info[0]: # 前一个是列表结尾，后一个是非列表开头，此时检测是否有相同的缩进
- logger.info(f"连接page {page_num} 内的list")
- # 向layout_paras[i] 寻找开头具有相同缩进的连续的行
- may_list_lines = []
- for j in range(len(next_page_paras[0])):
- line = next_page_paras[0][j]
- if len(line)==1: # 只可能是一行，多行情况再需要分析了
- if line[0]['bbox'][0] > __find_layout_bbox_by_line(line[0]['bbox'], next_page_layout_bbox)[0]:
- may_list_lines.append(line[0])
- else:
- break
- else:
- break
- # 如果这些行的缩进是相等的，那么连到上一个layout的最后一个段落上。
- if len(may_list_lines)>0 and len(set([x['bbox'][0] for x in may_list_lines]))==1:
- pre_page_paras[-1].append(may_list_lines)
- next_page_paras[0] = next_page_paras[0][len(may_list_lines):]
- return True
-
- return False
-
-
-def __find_layout_bbox_by_line(line_bbox, layout_bboxes):
- """
- 根据line找到所在的layout
- """
- for layout in layout_bboxes:
- if is_in_layout(line_bbox, layout):
- return layout
- return None
-
-
-def __connect_para_inter_layoutbox(layout_paras, new_layout_bbox, lang):
- """
- layout之间进行分段。
- 主要是计算前一个layOut的最后一行和后一个layout的第一行是否可以连接。
- 连接的条件需要同时满足：
- 1. 上一个layout的最后一行沾满整个行。并且没有结尾符号。
- 2. 下一行开头不留空白。
-
- """
- connected_layout_paras = []
- if len(layout_paras)==0:
- return connected_layout_paras
-
- connected_layout_paras.append(layout_paras[0])
- for i in range(1, len(layout_paras)):
- try:
- if len(layout_paras[i])==0 or len(layout_paras[i-1])==0: # TODO 考虑连接问题，
- continue
- pre_last_line = layout_paras[i-1][-1][-1]
- next_first_line = layout_paras[i][0][0]
- except Exception as e:
- logger.error(f"page layout {i} has no line")
- continue
- pre_last_line_text = ''.join([__get_span_text(span) for span in pre_last_line['spans']])
- pre_last_line_type = pre_last_line['spans'][-1]['type']
- next_first_line_text = ''.join([__get_span_text(span) for span in next_first_line['spans']])
- next_first_line_type = next_first_line['spans'][0]['type']
- if pre_last_line_type not in [TEXT, INLINE_EQUATION] or next_first_line_type not in [TEXT, INLINE_EQUATION]:
- connected_layout_paras.append(layout_paras[i])
- continue
-
- pre_x2_max = __find_layout_bbox_by_line(pre_last_line['bbox'], new_layout_bbox)[2]
- next_x0_min = __find_layout_bbox_by_line(next_first_line['bbox'], new_layout_bbox)[0]
-
- pre_last_line_text = pre_last_line_text.strip()
- next_first_line_text = next_first_line_text.strip()
- if pre_last_line['bbox'][2] == pre_x2_max and pre_last_line_text[-1] not in LINE_STOP_FLAG and next_first_line['bbox'][0]==next_x0_min: # 前面一行沾满了整个行，并且没有结尾符号.下一行没有空白开头。
- """连接段落条件成立，将前一个layout的段落和后一个layout的段落连接。"""
- connected_layout_paras[-1][-1].extend(layout_paras[i][0])
- layout_paras[i].pop(0) # 删除后一个layout的第一个段落， 因为他已经被合并到前一个layout的最后一个段落了。
- if len(layout_paras[i])==0:
- layout_paras.pop(i)
- else:
- connected_layout_paras.append(layout_paras[i])
- else:
- """连接段落条件不成立，将前一个layout的段落加入到结果中。"""
- connected_layout_paras.append(layout_paras[i])
-
- return connected_layout_paras
-
-
-def __connect_para_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox, page_num, lang):
- """
- 连接起来相邻两个页面的段落——前一个页面最后一个段落和后一个页面的第一个段落。
- 是否可以连接的条件：
- 1. 前一个页面的最后一个段落最后一行沾满整个行。并且没有结尾符号。
- 2. 后一个页面的第一个段落第一行没有空白开头。
- """
- # 有的页面可能压根没有文字
- if len(pre_page_paras)==0 or len(next_page_paras)==0 or len(pre_page_paras[0])==0 or len(next_page_paras[0])==0: # TODO [[]]为什么出现在pre_page_paras里？
- return False
- pre_last_para = pre_page_paras[-1][-1]
- next_first_para = next_page_paras[0][0]
- pre_last_line = pre_last_para[-1]
- next_first_line = next_first_para[0]
- pre_last_line_text = ''.join([__get_span_text(span) for span in pre_last_line['spans']])
- pre_last_line_type = pre_last_line['spans'][-1]['type']
- next_first_line_text = ''.join([__get_span_text(span) for span in next_first_line['spans']])
- next_first_line_type = next_first_line['spans'][0]['type']
-
- if pre_last_line_type not in [TEXT, INLINE_EQUATION] or next_first_line_type not in [TEXT, INLINE_EQUATION]: # TODO，真的要做好，要考虑跨table, image, 行间的情况
- # 不是文本，不连接
- return False
-
- pre_x2_max = __find_layout_bbox_by_line(pre_last_line['bbox'], pre_page_layout_bbox)[2]
- next_x0_min = __find_layout_bbox_by_line(next_first_line['bbox'], next_page_layout_bbox)[0]
-
- pre_last_line_text = pre_last_line_text.strip()
- next_first_line_text = next_first_line_text.strip()
- if pre_last_line['bbox'][2] == pre_x2_max and pre_last_line_text[-1] not in LINE_STOP_FLAG and next_first_line['bbox'][0]==next_x0_min: # 前面一行沾满了整个行，并且没有结尾符号.下一行没有空白开头。
- """连接段落条件成立，将前一个layout的段落和后一个layout的段落连接。"""
- pre_last_para.extend(next_first_para)
- next_page_paras[0].pop(0) # 删除后一个页面的第一个段落， 因为他已经被合并到前一个页面的最后一个段落了。
- return True
- else:
- return False
-
-def find_consecutive_true_regions(input_array):
- start_index = None # 连续True区域的起始索引
- regions = [] # 用于保存所有连续True区域的起始和结束索引
-
- for i in range(len(input_array)):
- # 如果我们找到了一个True值，并且当前并没有在连续True区域中
- if input_array[i] and start_index is None:
- start_index = i # 记录连续True区域的起始索引
-
- # 如果我们找到了一个False值，并且当前在连续True区域中
- elif not input_array[i] and start_index is not None:
- # 如果连续True区域长度大于1，那么将其添加到结果列表中
- if i - start_index > 1:
- regions.append((start_index, i-1))
- start_index = None # 重置起始索引
-
- # 如果最后一个元素是True，那么需要将最后一个连续True区域加入到结果列表中
- if start_index is not None and len(input_array) - start_index > 1:
- regions.append((start_index, len(input_array)-1))
-
- return regions
-
-
-def __connect_middle_align_text(page_paras, new_layout_bbox, page_num, lang, debug_mode):
- """
- 找出来中间对齐的连续单行文本，如果连续行高度相同，那么合并为一个段落。
- 一个line居中的条件是：
- 1. 水平中心点跨越layout的中心点。
- 2. 左右两侧都有空白
- """
-
- for layout_i, layout_para in enumerate(page_paras):
- layout_box = new_layout_bbox[layout_i]
- single_line_paras_tag = []
- for i in range(len(layout_para)):
- single_line_paras_tag.append(len(layout_para[i])==1 and layout_para[i][0]['spans'][0]['type']==TEXT)
-
- """找出来连续的单行文本，如果连续行高度相同，那么合并为一个段落。"""
- consecutive_single_line_indices = find_consecutive_true_regions(single_line_paras_tag)
- if len(consecutive_single_line_indices)>0:
- index_offset = 0
- """检查这些行是否是高度相同的，居中的"""
- for start, end in consecutive_single_line_indices:
- start += index_offset
- end += index_offset
- line_hi = np.array([line[0]['bbox'][3]-line[0]['bbox'][1] for line in layout_para[start:end+1]])
- first_line_text = ''.join([__get_span_text(span) for span in layout_para[start][0]['spans']])
- if "Table" in first_line_text or "Figure" in first_line_text:
- pass
- if debug_mode:
- logger.debug(line_hi.std())
-
- if line_hi.std()<2:
- """行高度相同，那么判断是否居中"""
- all_left_x0 = [line[0]['bbox'][0] for line in layout_para[start:end+1]]
- all_right_x1 = [line[0]['bbox'][2] for line in layout_para[start:end+1]]
- layout_center = (layout_box[0] + layout_box[2]) / 2
- if all([x0 < layout_center < x1 for x0, x1 in zip(all_left_x0, all_right_x1)]) \
- and not all([x0==layout_box[0] for x0 in all_left_x0]) \
- and not all([x1==layout_box[2] for x1 in all_right_x1]):
- merge_para = [l[0] for l in layout_para[start:end+1]]
- para_text = ''.join([__get_span_text(span) for line in merge_para for span in line['spans']])
- if debug_mode:
- logger.debug(para_text)
- layout_para[start:end+1] = [merge_para]
- index_offset -= end-start
-
- return
-
-
-def __merge_signle_list_text(page_paras, new_layout_bbox, page_num, lang):
- """
- 找出来连续的单行文本，如果首行顶格，接下来的几个单行段落缩进对齐，那么合并为一个段落。
- """
-
- pass
-
-
-def __do_split_page(blocks, layout_bboxes, new_layout_bbox, page_num, lang):
- """
- 根据line和layout情况进行分段
- 先实现一个根据行末尾特征分段的简单方法。
- """
- """
- 算法思路：
- 1. 扫描layout里每一行，找出来行尾距离layout有边界有一定距离的行。
- 2. 从上述行中找到末尾是句号等可作为断行标志的行。
- 3. 参照上述行尾特征进行分段。
- 4. 图、表，目前独占一行，不考虑分段。
- """
- if page_num==343:
- pass
- lines_group = __group_line_by_layout(blocks, layout_bboxes, lang) # block内分段
- layout_paras, layout_list_info = __split_para_in_layoutbox(lines_group, new_layout_bbox, lang) # layout内分段
- layout_paras2, page_list_info = __connect_list_inter_layout(layout_paras, new_layout_bbox, layout_list_info, page_num, lang) # layout之间连接列表段落
- connected_layout_paras = __connect_para_inter_layoutbox(layout_paras2, new_layout_bbox, lang) # layout间链接段落
-
-
- return connected_layout_paras, page_list_info
-
-
-def para_split(pdf_info_dict, debug_mode, lang="en"):
- """
- 根据line和layout情况进行分段
- """
- new_layout_of_pages = [] # 数组的数组，每个元素是一个页面的layoutS
- all_page_list_info = [] # 保存每个页面开头和结尾是否是列表
- for page_num, page in pdf_info_dict.items():
- blocks = page['preproc_blocks']
- layout_bboxes = page['layout_bboxes']
- new_layout_bbox = __common_pre_proc(blocks, layout_bboxes)
- new_layout_of_pages.append(new_layout_bbox)
- splited_blocks, page_list_info = __do_split_page(blocks, layout_bboxes, new_layout_bbox, page_num, lang)
- all_page_list_info.append(page_list_info)
- page['para_blocks'] = splited_blocks
-
- """连接页面与页面之间的可能合并的段落"""
- pdf_infos = list(pdf_info_dict.values())
- for page_num, page in enumerate(pdf_info_dict.values()):
- if page_num==0:
- continue
- pre_page_paras = pdf_infos[page_num-1]['para_blocks']
- next_page_paras = pdf_infos[page_num]['para_blocks']
- pre_page_layout_bbox = new_layout_of_pages[page_num-1]
- next_page_layout_bbox = new_layout_of_pages[page_num]
-
- is_conn = __connect_para_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox, page_num, lang)
- if debug_mode:
- if is_conn:
- logger.info(f"连接了第{page_num-1}页和第{page_num}页的段落")
-
- is_list_conn = __connect_list_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox, all_page_list_info[page_num-1], all_page_list_info[page_num], page_num, lang)
- if debug_mode:
- if is_list_conn:
- logger.info(f"连接了第{page_num-1}页和第{page_num}页的列表段落")
-
- """接下来可能会漏掉一些特别的一些可以合并的内容，对他们进行段落连接
- 1. 正文中有时出现一个行顶格，接下来几行缩进的情况。
- 2. 居中的一些连续单行，如果高度相同，那么可能是一个段落。
- """
- for page_num, page in enumerate(pdf_info_dict.values()):
- page_paras = page['para_blocks']
- new_layout_bbox = new_layout_of_pages[page_num]
- __connect_middle_align_text(page_paras, new_layout_bbox, page_num, lang, debug_mode=debug_mode)
- __merge_signle_list_text(page_paras, new_layout_bbox, page_num, lang)
diff --git a/magic_pdf/para/para_split_v2.py b/magic_pdf/para/para_split_v2.py
deleted file mode 100644
index 0e9ab0cdb..000000000
--- a/magic_pdf/para/para_split_v2.py
+++ /dev/null
@@ -1,789 +0,0 @@
-import copy
-
-from sklearn.cluster import DBSCAN
-import numpy as np
-from loguru import logger
-import re
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap_with_area_ratio as is_in_layout
-from magic_pdf.libs.ocr_content_type import ContentType, BlockType
-from magic_pdf.model.magic_model import MagicModel
-from magic_pdf.libs.Constants import *
-
-LINE_STOP_FLAG = ['.', '!', '?', '。', '！', '？', "：", ":", ")", "）", ";"]
-INLINE_EQUATION = ContentType.InlineEquation
-INTERLINE_EQUATION = ContentType.InterlineEquation
-TEXT = ContentType.Text
-debug_able = False
-
-
-def __get_span_text(span):
- c = span.get('content', '')
- if len(c) == 0:
- c = span.get('image_path', '')
-
- return c
-
-
-def __detect_list_lines(lines, new_layout_bboxes, lang):
- global debug_able
- """
- 探测是否包含了列表，并且把列表的行分开.
- 这样的段落特点是，顶格字母大写/数字，紧跟着几行缩进的。缩进的行首字母含小写的。
- """
-
- def find_repeating_patterns2(lst):
- indices = []
- ones_indices = []
- i = 0
- while i < len(lst): # Loop through the entire list
- if lst[i] == 1: # If we encounter a '1', we might be at the start of a pattern
- start = i
- ones_in_this_interval = [i]
- i += 1
- # Traverse elements that are 1, 2 or 3, until we encounter something else
- while i < len(lst) and lst[i] in [1, 2, 3]:
- if lst[i] == 1:
- ones_in_this_interval.append(i)
- i += 1
- if len(ones_in_this_interval) > 1 or (
- start < len(lst) - 1 and ones_in_this_interval and lst[start + 1] in [2, 3]):
- indices.append((start, i - 1))
- ones_indices.append(ones_in_this_interval)
- else:
- i += 1
- return indices, ones_indices
-
- def find_repeating_patterns(lst):
- indices = []
- ones_indices = []
- i = 0
- while i < len(lst) - 1: # 确保余下元素至少有2个
- if lst[i] == 1 and lst[i + 1] in [2, 3]: # 额外检查以防止连续出现的1
- start = i
- ones_in_this_interval = [i]
- i += 1
- while i < len(lst) and lst[i] in [2, 3]:
- i += 1
- # 验证下一个序列是否符合条件
- if i < len(lst) - 1 and lst[i] == 1 and lst[i + 1] in [2, 3] and lst[i - 1] in [2, 3]:
- while i < len(lst) and lst[i] in [1, 2, 3]:
- if lst[i] == 1:
- ones_in_this_interval.append(i)
- i += 1
- indices.append((start, i - 1))
- ones_indices.append(ones_in_this_interval)
- else:
- i += 1
- else:
- i += 1
- return indices, ones_indices
-
- """===================="""
-
- def split_indices(slen, index_array):
- result = []
- last_end = 0
-
- for start, end in sorted(index_array):
- if start > last_end:
- # 前一个区间结束到下一个区间开始之间的部分标记为"text"
- result.append(('text', last_end, start - 1))
- # 区间内标记为"list"
- result.append(('list', start, end))
- last_end = end + 1
-
- if last_end < slen:
- # 如果最后一个区间结束后还有剩余的字符串，将其标记为"text"
- result.append(('text', last_end, slen - 1))
-
- return result
-
- """===================="""
-
- if lang != 'en':
- return lines, None
-
- total_lines = len(lines)
- line_fea_encode = []
- """
- 对每一行进行特征编码，编码规则如下：
- 1. 如果行顶格，且大写字母开头或者数字开头，编码为1
- 2. 如果顶格，其他非大写开头编码为4
- 3. 如果非顶格，首字符大写，编码为2
- 4. 如果非顶格，首字符非大写编码为3
- """
- if len(lines) > 0:
- x_map_tag_dict, min_x_tag = cluster_line_x(lines)
- for l in lines:
- span_text = __get_span_text(l['spans'][0])
- if not span_text:
- line_fea_encode.append(0)
- continue
- first_char = span_text[0]
- layout = __find_layout_bbox_by_line(l['bbox'], new_layout_bboxes)
- if not layout:
- line_fea_encode.append(0)
- else:
- #
- if x_map_tag_dict[round(l['bbox'][0])] == min_x_tag:
- # if first_char.isupper() or first_char.isdigit() or not first_char.isalnum():
- if not first_char.isalnum() or if_match_reference_list(span_text):
- line_fea_encode.append(1)
- else:
- line_fea_encode.append(4)
- else:
- if first_char.isupper():
- line_fea_encode.append(2)
- else:
- line_fea_encode.append(3)
-
- # 然后根据编码进行分段, 选出来 1,2,3连续出现至少2次的行，认为是列表。
-
- list_indice, list_start_idx = find_repeating_patterns2(line_fea_encode)
- if len(list_indice) > 0:
- if debug_able:
- logger.info(f"发现了列表，列表行数：{list_indice}， {list_start_idx}")
-
- # TODO check一下这个特列表里缩进的行左侧是不是对齐的。
- segments = []
- for start, end in list_indice:
- for i in range(start, end + 1):
- if i > 0:
- if line_fea_encode[i] == 4:
- if debug_able:
- logger.info(f"列表行的第{i}行不是顶格的")
- break
- else:
- if debug_able:
- logger.info(f"列表行的第{start}到第{end}行是列表")
-
- return split_indices(total_lines, list_indice), list_start_idx
-
-
-def cluster_line_x(lines: list) -> dict:
- """
- 对一个block内所有lines的bbox的x0聚类
- """
- min_distance = 5
- min_sample = 1
- x0_lst = np.array([[round(line['bbox'][0]), 0] for line in lines])
- x0_clusters = DBSCAN(eps=min_distance, min_samples=min_sample).fit(x0_lst)
- x0_uniq_label = np.unique(x0_clusters.labels_)
- # x1_lst = np.array([[line['bbox'][2], 0] for line in lines])
- x0_2_new_val = {} # 存储旧值对应的新值映射
- min_x0 = round(lines[0]["bbox"][0])
- for label in x0_uniq_label:
- if label == -1:
- continue
- x0_index_of_label = np.where(x0_clusters.labels_ == label)
- x0_raw_val = x0_lst[x0_index_of_label][:, 0]
- x0_new_val = np.min(x0_lst[x0_index_of_label][:, 0])
- x0_2_new_val.update({round(raw_val): round(x0_new_val) for raw_val in x0_raw_val})
- if x0_new_val < min_x0:
- min_x0 = x0_new_val
- return x0_2_new_val, min_x0
-
-
-def if_match_reference_list(text: str) -> bool:
- pattern = re.compile(r'^\d+\..*')
- if pattern.match(text):
- return True
- else:
- return False
-
-
-def __valign_lines(blocks, layout_bboxes):
- """
- 在一个layoutbox内对齐行的左侧和右侧。
- 扫描行的左侧和右侧，如果x0, x1差距不超过一个阈值，就强行对齐到所处layout的左右两侧（和layout有一段距离）。
- 3是个经验值，TODO，计算得来，可以设置为1.5个正文字符。
- """
-
- min_distance = 3
- min_sample = 2
- new_layout_bboxes = []
- # add bbox_fs for para split calculation
- for block in blocks:
- block["bbox_fs"] = copy.deepcopy(block["bbox"])
- for layout_box in layout_bboxes:
- blocks_in_layoutbox = [b for b in blocks if
- b["type"] == BlockType.Text and is_in_layout(b['bbox'], layout_box['layout_bbox'])]
- if len(blocks_in_layoutbox) == 0 or len(blocks_in_layoutbox[0]["lines"]) == 0:
- new_layout_bboxes.append(layout_box['layout_bbox'])
- continue
-
- x0_lst = np.array([[line['bbox'][0], 0] for block in blocks_in_layoutbox for line in block['lines']])
- x1_lst = np.array([[line['bbox'][2], 0] for block in blocks_in_layoutbox for line in block['lines']])
- x0_clusters = DBSCAN(eps=min_distance, min_samples=min_sample).fit(x0_lst)
- x1_clusters = DBSCAN(eps=min_distance, min_samples=min_sample).fit(x1_lst)
- x0_uniq_label = np.unique(x0_clusters.labels_)
- x1_uniq_label = np.unique(x1_clusters.labels_)
-
- x0_2_new_val = {} # 存储旧值对应的新值映射
- x1_2_new_val = {}
- for label in x0_uniq_label:
- if label == -1:
- continue
- x0_index_of_label = np.where(x0_clusters.labels_ == label)
- x0_raw_val = x0_lst[x0_index_of_label][:, 0]
- x0_new_val = np.min(x0_lst[x0_index_of_label][:, 0])
- x0_2_new_val.update({idx: x0_new_val for idx in x0_raw_val})
- for label in x1_uniq_label:
- if label == -1:
- continue
- x1_index_of_label = np.where(x1_clusters.labels_ == label)
- x1_raw_val = x1_lst[x1_index_of_label][:, 0]
- x1_new_val = np.max(x1_lst[x1_index_of_label][:, 0])
- x1_2_new_val.update({idx: x1_new_val for idx in x1_raw_val})
-
- for block in blocks_in_layoutbox:
- for line in block['lines']:
- x0, x1 = line['bbox'][0], line['bbox'][2]
- if x0 in x0_2_new_val:
- line['bbox'][0] = int(x0_2_new_val[x0])
-
- if x1 in x1_2_new_val:
- line['bbox'][2] = int(x1_2_new_val[x1])
- # 其余对不齐的保持不动
-
- # 由于修改了block里的line长度，现在需要重新计算block的bbox
- for block in blocks_in_layoutbox:
- if len(block["lines"]) > 0:
- block['bbox_fs'] = [min([line['bbox'][0] for line in block['lines']]),
- min([line['bbox'][1] for line in block['lines']]),
- max([line['bbox'][2] for line in block['lines']]),
- max([line['bbox'][3] for line in block['lines']])]
- """新计算layout的bbox，因为block的bbox变了。"""
- layout_x0 = min([block['bbox_fs'][0] for block in blocks_in_layoutbox])
- layout_y0 = min([block['bbox_fs'][1] for block in blocks_in_layoutbox])
- layout_x1 = max([block['bbox_fs'][2] for block in blocks_in_layoutbox])
- layout_y1 = max([block['bbox_fs'][3] for block in blocks_in_layoutbox])
- new_layout_bboxes.append([layout_x0, layout_y0, layout_x1, layout_y1])
-
- return new_layout_bboxes
-
-
-def __align_text_in_layout(blocks, layout_bboxes):
- """
- 由于ocr出来的line，有时候会在前后有一段空白，这个时候需要对文本进行对齐，超出的部分被layout左右侧截断。
- """
- for layout in layout_bboxes:
- lb = layout['layout_bbox']
- blocks_in_layoutbox = [block for block in blocks if
- block["type"] == BlockType.Text and is_in_layout(block['bbox'], lb)]
- if len(blocks_in_layoutbox) == 0:
- continue
-
- for block in blocks_in_layoutbox:
- for line in block.get("lines", []):
- x0, x1 = line['bbox'][0], line['bbox'][2]
- if x0 < lb[0]:
- line['bbox'][0] = lb[0]
- if x1 > lb[2]:
- line['bbox'][2] = lb[2]
-
-
-def __common_pre_proc(blocks, layout_bboxes):
- """
- 不分语言的，对文本进行预处理
- """
- # __add_line_period(blocks, layout_bboxes)
- __align_text_in_layout(blocks, layout_bboxes)
- aligned_layout_bboxes = __valign_lines(blocks, layout_bboxes)
-
- return aligned_layout_bboxes
-
-
-def __pre_proc_zh_blocks(blocks, layout_bboxes):
- """
- 对中文文本进行分段预处理
- """
- pass
-
-
-def __pre_proc_en_blocks(blocks, layout_bboxes):
- """
- 对英文文本进行分段预处理
- """
- pass
-
-
-def __group_line_by_layout(blocks, layout_bboxes):
- """
- 每个layout内的行进行聚合
- """
- # 因为只是一个block一行目前, 一个block就是一个段落
- blocks_group = []
- for lyout in layout_bboxes:
- blocks_in_layout = [block for block in blocks if is_in_layout(block.get('bbox_fs', None), lyout['layout_bbox'])]
- blocks_group.append(blocks_in_layout)
- return blocks_group
-
-
-def __split_para_in_layoutbox(blocks_group, new_layout_bbox, lang="en"):
- """
- lines_group 进行行分段——layout内部进行分段。lines_group内每个元素是一个Layoutbox内的所有行。
- 1. 先计算每个group的左右边界。
- 2. 然后根据行末尾特征进行分段。
- 末尾特征：以句号等结束符结尾。并且距离右侧边界有一定距离。
- 且下一行开头不留空白。
-
- """
- list_info = [] # 这个layout最后是不是列表,记录每一个layout里是不是列表开头，列表结尾
- for blocks in blocks_group:
- is_start_list = None
- is_end_list = None
- if len(blocks) == 0:
- list_info.append([False, False])
- continue
- if blocks[0]["type"] != BlockType.Text and blocks[-1]["type"] != BlockType.Text:
- list_info.append([False, False])
- continue
- if blocks[0]["type"] != BlockType.Text:
- is_start_list = False
- if blocks[-1]["type"] != BlockType.Text:
- is_end_list = False
-
- lines = [line for block in blocks if
- block["type"] == BlockType.Text for line in
- block['lines']]
- total_lines = len(lines)
- if total_lines == 1 or total_lines == 0:
- list_info.append([False, False])
- continue
- """在进入到真正的分段之前，要对文字块从统计维度进行对齐方式的探测，
- 对齐方式分为以下：
- 1. 左对齐的文本块(特点是左侧顶格，或者左侧不顶格但是右侧顶格的行数大于非顶格的行数，顶格的首字母有大写也有小写)
- 1) 右侧对齐的行，单独成一段
- 2) 中间对齐的行，按照字体/行高聚合成一段
- 2. 左对齐的列表块（其特点是左侧顶格的行数小于等于非顶格的行数，非定格首字母会有小写，顶格90%是大写。并且左侧顶格行数大于1，大于1是为了这种模式连续出现才能称之为列表）
- 这样的文本块，顶格的为一个段落开头，紧随其后非顶格的行属于这个段落。
- """
- text_segments, list_start_line = __detect_list_lines(lines, new_layout_bbox, lang)
- """根据list_range，把lines分成几个部分
-
- """
- for list_start in list_start_line:
- if len(list_start) > 1:
- for i in range(0, len(list_start)):
- index = list_start[i] - 1
- if index >= 0:
- if "content" in lines[index]["spans"][-1] and lines[index]["spans"][-1].get('type', '') not in [
- ContentType.InlineEquation, ContentType.InterlineEquation]:
- lines[index]["spans"][-1]["content"] += '\n\n'
- layout_list_info = [False, False] # 这个layout最后是不是列表,记录每一个layout里是不是列表开头，列表结尾
- for content_type, start, end in text_segments:
- if content_type == 'list':
- if start == 0 and is_start_list is None:
- layout_list_info[0] = True
- if end == total_lines - 1 and is_end_list is None:
- layout_list_info[1] = True
-
- list_info.append(layout_list_info)
- return list_info
-
-
-def __split_para_lines(lines: list, text_blocks: list) -> list:
- text_paras = []
- other_paras = []
- text_lines = []
- for line in lines:
-
- spans_types = [span["type"] for span in line]
- if ContentType.Table in spans_types:
- other_paras.append([line])
- continue
- if ContentType.Image in spans_types:
- other_paras.append([line])
- continue
- if ContentType.InterlineEquation in spans_types:
- other_paras.append([line])
- continue
- text_lines.append(line)
-
- for block in text_blocks:
- block_bbox = block["bbox"]
- para = []
- for line in text_lines:
- bbox = line["bbox"]
- if is_in_layout(bbox, block_bbox):
- para.append(line)
- if len(para) > 0:
- text_paras.append(para)
- paras = other_paras.extend(text_paras)
- paras_sorted = sorted(paras, key=lambda x: x[0]["bbox"][1])
- return paras_sorted
-
-
-def __connect_list_inter_layout(blocks_group, new_layout_bbox, layout_list_info, page_num, lang):
- global debug_able
- """
- 如果上个layout的最后一个段落是列表，下一个layout的第一个段落也是列表，那么将他们连接起来。 TODO 因为没有区分列表和段落，所以这个方法暂时不实现。
- 根据layout_list_info判断是不是列表。，下个layout的第一个段如果不是列表，那么看他们是否有几行都有相同的缩进。
- """
- if len(blocks_group) == 0 or len(blocks_group) == 0: # 0的时候最后的return 会出错
- return blocks_group, [False, False]
-
- for i in range(1, len(blocks_group)):
- if len(blocks_group[i]) == 0 or len(blocks_group[i - 1]) == 0:
- continue
- pre_layout_list_info = layout_list_info[i - 1]
- next_layout_list_info = layout_list_info[i]
- pre_last_para = blocks_group[i - 1][-1].get("lines", [])
- next_paras = blocks_group[i]
- next_first_para = next_paras[0]
-
- if pre_layout_list_info[1] and not next_layout_list_info[0] and next_first_para[
- "type"] == BlockType.Text: # 前一个是列表结尾，后一个是非列表开头，此时检测是否有相同的缩进
- if debug_able:
- logger.info(f"连接page {page_num} 内的list")
- # 向layout_paras[i] 寻找开头具有相同缩进的连续的行
- may_list_lines = []
- lines = next_first_para.get("lines", [])
-
- for line in lines:
- if line['bbox'][0] > __find_layout_bbox_by_line(line['bbox'], new_layout_bbox)[0]:
- may_list_lines.append(line)
- else:
- break
- # 如果这些行的缩进是相等的，那么连到上一个layout的最后一个段落上。
- if len(may_list_lines) > 0 and len(set([x['bbox'][0] for x in may_list_lines])) == 1:
- pre_last_para.extend(may_list_lines)
- next_first_para["lines"] = next_first_para["lines"][len(may_list_lines):]
-
- return blocks_group, [layout_list_info[0][0], layout_list_info[-1][1]] # 同时还返回了这个页面级别的开头、结尾是不是列表的信息
-
-
-def __connect_list_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox,
- pre_page_list_info, next_page_list_info, page_num, lang):
- """
- 如果上个layout的最后一个段落是列表，下一个layout的第一个段落也是列表，那么将他们连接起来。 TODO 因为没有区分列表和段落，所以这个方法暂时不实现。
- 根据layout_list_info判断是不是列表。，下个layout的第一个段如果不是列表，那么看他们是否有几行都有相同的缩进。
- """
- if len(pre_page_paras) == 0 or len(next_page_paras) == 0: # 0的时候最后的return 会出错
- return False
- if len(pre_page_paras[-1]) == 0 or len(next_page_paras[0]) == 0:
- return False
- if pre_page_paras[-1][-1]["type"] != BlockType.Text or next_page_paras[0][0]["type"] != BlockType.Text:
- return False
- if pre_page_list_info[1] and not next_page_list_info[0]: # 前一个是列表结尾，后一个是非列表开头，此时检测是否有相同的缩进
- if debug_able:
- logger.info(f"连接page {page_num} 内的list")
- # 向layout_paras[i] 寻找开头具有相同缩进的连续的行
- may_list_lines = []
- next_page_first_para = next_page_paras[0][0]
- if next_page_first_para["type"] == BlockType.Text:
- lines = next_page_first_para["lines"]
- for line in lines:
- if line['bbox'][0] > __find_layout_bbox_by_line(line['bbox'], next_page_layout_bbox)[0]:
- may_list_lines.append(line)
- else:
- break
- # 如果这些行的缩进是相等的，那么连到上一个layout的最后一个段落上。
- if len(may_list_lines) > 0 and len(set([x['bbox'][0] for x in may_list_lines])) == 1:
- # pre_page_paras[-1].append(may_list_lines)
- # 下一页合并到上一页最后一段，打一个cross_page的标签
- for line in may_list_lines:
- for span in line["spans"]:
- span[CROSS_PAGE] = True
- pre_page_paras[-1][-1]["lines"].extend(may_list_lines)
- next_page_first_para["lines"] = next_page_first_para["lines"][len(may_list_lines):]
- return True
-
- return False
-
-
-def __find_layout_bbox_by_line(line_bbox, layout_bboxes):
- """
- 根据line找到所在的layout
- """
- for layout in layout_bboxes:
- if is_in_layout(line_bbox, layout):
- return layout
- return None
-
-
-def __connect_para_inter_layoutbox(blocks_group, new_layout_bbox):
- """
- layout之间进行分段。
- 主要是计算前一个layOut的最后一行和后一个layout的第一行是否可以连接。
- 连接的条件需要同时满足：
- 1. 上一个layout的最后一行沾满整个行。并且没有结尾符号。
- 2. 下一行开头不留空白。
-
- """
- connected_layout_blocks = []
- if len(blocks_group) == 0:
- return connected_layout_blocks
-
- connected_layout_blocks.append(blocks_group[0])
- for i in range(1, len(blocks_group)):
- try:
- if len(blocks_group[i]) == 0:
- continue
- if len(blocks_group[i - 1]) == 0: # TODO 考虑连接问题，
- connected_layout_blocks.append(blocks_group[i])
- continue
- # text类型的段才需要考虑layout间的合并
- if blocks_group[i - 1][-1]["type"] != BlockType.Text or blocks_group[i][0]["type"] != BlockType.Text:
- connected_layout_blocks.append(blocks_group[i])
- continue
- if len(blocks_group[i - 1][-1]["lines"]) == 0 or len(blocks_group[i][0]["lines"]) == 0:
- connected_layout_blocks.append(blocks_group[i])
- continue
- pre_last_line = blocks_group[i - 1][-1]["lines"][-1]
- next_first_line = blocks_group[i][0]["lines"][0]
- except Exception as e:
- logger.error(f"page layout {i} has no line")
- continue
- pre_last_line_text = ''.join([__get_span_text(span) for span in pre_last_line['spans']])
- pre_last_line_type = pre_last_line['spans'][-1]['type']
- next_first_line_text = ''.join([__get_span_text(span) for span in next_first_line['spans']])
- next_first_line_type = next_first_line['spans'][0]['type']
- if pre_last_line_type not in [TEXT, INLINE_EQUATION] or next_first_line_type not in [TEXT, INLINE_EQUATION]:
- connected_layout_blocks.append(blocks_group[i])
- continue
- pre_layout = __find_layout_bbox_by_line(pre_last_line['bbox'], new_layout_bbox)
- next_layout = __find_layout_bbox_by_line(next_first_line['bbox'], new_layout_bbox)
-
- pre_x2_max = pre_layout[2] if pre_layout else -1
- next_x0_min = next_layout[0] if next_layout else -1
-
- pre_last_line_text = pre_last_line_text.strip()
- next_first_line_text = next_first_line_text.strip()
- if pre_last_line['bbox'][2] == pre_x2_max and pre_last_line_text and pre_last_line_text[
- -1] not in LINE_STOP_FLAG and \
- next_first_line['bbox'][0] == next_x0_min: # 前面一行沾满了整个行，并且没有结尾符号.下一行没有空白开头。
- """连接段落条件成立，将前一个layout的段落和后一个layout的段落连接。"""
- connected_layout_blocks[-1][-1]["lines"].extend(blocks_group[i][0]["lines"])
- blocks_group[i][0]["lines"] = [] # 删除后一个layout第一个段落中的lines，因为他已经被合并到前一个layout的最后一个段落了
- blocks_group[i][0][LINES_DELETED] = True
- # if len(layout_paras[i]) == 0:
- # layout_paras.pop(i)
- # else:
- # connected_layout_paras.append(layout_paras[i])
- connected_layout_blocks.append(blocks_group[i])
- else:
- """连接段落条件不成立，将前一个layout的段落加入到结果中。"""
- connected_layout_blocks.append(blocks_group[i])
- return connected_layout_blocks
-
-
-def __connect_para_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox, next_page_layout_bbox, page_num,
- lang):
- """
- 连接起来相邻两个页面的段落——前一个页面最后一个段落和后一个页面的第一个段落。
- 是否可以连接的条件：
- 1. 前一个页面的最后一个段落最后一行沾满整个行。并且没有结尾符号。
- 2. 后一个页面的第一个段落第一行没有空白开头。
- """
- # 有的页面可能压根没有文字
- if len(pre_page_paras) == 0 or len(next_page_paras) == 0 or len(pre_page_paras[0]) == 0 or len(
- next_page_paras[0]) == 0: # TODO [[]]为什么出现在pre_page_paras里？
- return False
- pre_last_block = pre_page_paras[-1][-1]
- next_first_block = next_page_paras[0][0]
- if pre_last_block["type"] != BlockType.Text or next_first_block["type"] != BlockType.Text:
- return False
- if len(pre_last_block["lines"]) == 0 or len(next_first_block["lines"]) == 0:
- return False
- pre_last_para = pre_last_block["lines"]
- next_first_para = next_first_block["lines"]
- pre_last_line = pre_last_para[-1]
- next_first_line = next_first_para[0]
- pre_last_line_text = ''.join([__get_span_text(span) for span in pre_last_line['spans']])
- pre_last_line_type = pre_last_line['spans'][-1]['type']
- next_first_line_text = ''.join([__get_span_text(span) for span in next_first_line['spans']])
- next_first_line_type = next_first_line['spans'][0]['type']
-
- if pre_last_line_type not in [TEXT, INLINE_EQUATION] or next_first_line_type not in [TEXT,
- INLINE_EQUATION]: # TODO，真的要做好，要考虑跨table, image, 行间的情况
- # 不是文本，不连接
- return False
-
- pre_x2_max_bbox = __find_layout_bbox_by_line(pre_last_line['bbox'], pre_page_layout_bbox)
- if not pre_x2_max_bbox:
- return False
- next_x0_min_bbox = __find_layout_bbox_by_line(next_first_line['bbox'], next_page_layout_bbox)
- if not next_x0_min_bbox:
- return False
-
- pre_x2_max = pre_x2_max_bbox[2]
- next_x0_min = next_x0_min_bbox[0]
-
- pre_last_line_text = pre_last_line_text.strip()
- next_first_line_text = next_first_line_text.strip()
- if pre_last_line['bbox'][2] == pre_x2_max and pre_last_line_text[-1] not in LINE_STOP_FLAG and \
- next_first_line['bbox'][0] == next_x0_min: # 前面一行沾满了整个行，并且没有结尾符号.下一行没有空白开头。
- """连接段落条件成立，将前一个layout的段落和后一个layout的段落连接。"""
- # 下一页合并到上一页最后一段，打一个cross_page的标签
- for line in next_first_para:
- for span in line["spans"]:
- span[CROSS_PAGE] = True
- pre_last_para.extend(next_first_para)
-
- # next_page_paras[0].pop(0) # 删除后一个页面的第一个段落， 因为他已经被合并到前一个页面的最后一个段落了。
- next_page_paras[0][0]["lines"] = []
- next_page_paras[0][0][LINES_DELETED] = True
- return True
- else:
- return False
-
-
-def find_consecutive_true_regions(input_array):
- start_index = None # 连续True区域的起始索引
- regions = [] # 用于保存所有连续True区域的起始和结束索引
-
- for i in range(len(input_array)):
- # 如果我们找到了一个True值，并且当前并没有在连续True区域中
- if input_array[i] and start_index is None:
- start_index = i # 记录连续True区域的起始索引
-
- # 如果我们找到了一个False值，并且当前在连续True区域中
- elif not input_array[i] and start_index is not None:
- # 如果连续True区域长度大于1，那么将其添加到结果列表中
- if i - start_index > 1:
- regions.append((start_index, i - 1))
- start_index = None # 重置起始索引
-
- # 如果最后一个元素是True，那么需要将最后一个连续True区域加入到结果列表中
- if start_index is not None and len(input_array) - start_index > 1:
- regions.append((start_index, len(input_array) - 1))
-
- return regions
-
-
-def __connect_middle_align_text(page_paras, new_layout_bbox, page_num, lang):
- global debug_able
- """
- 找出来中间对齐的连续单行文本，如果连续行高度相同，那么合并为一个段落。
- 一个line居中的条件是：
- 1. 水平中心点跨越layout的中心点。
- 2. 左右两侧都有空白
- """
-
- for layout_i, layout_para in enumerate(page_paras):
- layout_box = new_layout_bbox[layout_i]
- single_line_paras_tag = []
- for i in range(len(layout_para)):
- # single_line_paras_tag.append(len(layout_para[i]) == 1 and layout_para[i][0]['spans'][0]['type'] == TEXT)
- single_line_paras_tag.append(layout_para[i]['type'] == BlockType.Text and len(layout_para[i]["lines"]) == 1)
- """找出来连续的单行文本，如果连续行高度相同，那么合并为一个段落。"""
- consecutive_single_line_indices = find_consecutive_true_regions(single_line_paras_tag)
- if len(consecutive_single_line_indices) > 0:
- """检查这些行是否是高度相同的，居中的"""
- for start, end in consecutive_single_line_indices:
- # start += index_offset
- # end += index_offset
- line_hi = np.array([block["lines"][0]['bbox'][3] - block["lines"][0]['bbox'][1] for block in
- layout_para[start:end + 1]])
- first_line_text = ''.join([__get_span_text(span) for span in layout_para[start]["lines"][0]['spans']])
- if "Table" in first_line_text or "Figure" in first_line_text:
- pass
- if debug_able:
- logger.info(line_hi.std())
-
- if line_hi.std() < 2:
- """行高度相同，那么判断是否居中"""
- all_left_x0 = [block["lines"][0]['bbox'][0] for block in layout_para[start:end + 1]]
- all_right_x1 = [block["lines"][0]['bbox'][2] for block in layout_para[start:end + 1]]
- layout_center = (layout_box[0] + layout_box[2]) / 2
- if all([x0 < layout_center < x1 for x0, x1 in zip(all_left_x0, all_right_x1)]) \
- and not all([x0 == layout_box[0] for x0 in all_left_x0]) \
- and not all([x1 == layout_box[2] for x1 in all_right_x1]):
- merge_para = [block["lines"][0] for block in layout_para[start:end + 1]]
- para_text = ''.join([__get_span_text(span) for line in merge_para for span in line['spans']])
- if debug_able:
- logger.info(para_text)
- layout_para[start]["lines"] = merge_para
- for i_para in range(start + 1, end + 1):
- layout_para[i_para]["lines"] = []
- layout_para[i_para][LINES_DELETED] = True
- # layout_para[start:end + 1] = [merge_para]
-
- # index_offset -= end - start
-
- return
-
-
-def __merge_signle_list_text(page_paras, new_layout_bbox, page_num, lang):
- """
- 找出来连续的单行文本，如果首行顶格，接下来的几个单行段落缩进对齐，那么合并为一个段落。
- """
-
- pass
-
-
-def __do_split_page(blocks, layout_bboxes, new_layout_bbox, page_num, lang):
- """
- 根据line和layout情况进行分段
- 先实现一个根据行末尾特征分段的简单方法。
- """
- """
- 算法思路：
- 1. 扫描layout里每一行，找出来行尾距离layout有边界有一定距离的行。
- 2. 从上述行中找到末尾是句号等可作为断行标志的行。
- 3. 参照上述行尾特征进行分段。
- 4. 图、表，目前独占一行，不考虑分段。
- """
- blocks_group = __group_line_by_layout(blocks, layout_bboxes) # block内分段
- layout_list_info = __split_para_in_layoutbox(blocks_group, new_layout_bbox, lang) # layout内分段
- blocks_group, page_list_info = __connect_list_inter_layout(blocks_group, new_layout_bbox, layout_list_info,
- page_num, lang) # layout之间连接列表段落
- connected_layout_blocks = __connect_para_inter_layoutbox(blocks_group, new_layout_bbox) # layout间链接段落
-
- return connected_layout_blocks, page_list_info
-
-
-def para_split(pdf_info_dict, debug_mode, lang="en"):
- global debug_able
- debug_able = debug_mode
- new_layout_of_pages = [] # 数组的数组，每个元素是一个页面的layoutS
- all_page_list_info = [] # 保存每个页面开头和结尾是否是列表
- for page_num, page in pdf_info_dict.items():
- blocks = copy.deepcopy(page['preproc_blocks'])
- layout_bboxes = page['layout_bboxes']
- new_layout_bbox = __common_pre_proc(blocks, layout_bboxes)
- new_layout_of_pages.append(new_layout_bbox)
- splited_blocks, page_list_info = __do_split_page(blocks, layout_bboxes, new_layout_bbox, page_num, lang)
- all_page_list_info.append(page_list_info)
- page['para_blocks'] = splited_blocks
-
- """连接页面与页面之间的可能合并的段落"""
- pdf_infos = list(pdf_info_dict.values())
- for page_num, page in enumerate(pdf_info_dict.values()):
- if page_num == 0:
- continue
- pre_page_paras = pdf_infos[page_num - 1]['para_blocks']
- next_page_paras = pdf_infos[page_num]['para_blocks']
- pre_page_layout_bbox = new_layout_of_pages[page_num - 1]
- next_page_layout_bbox = new_layout_of_pages[page_num]
-
- is_conn = __connect_para_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox,
- next_page_layout_bbox, page_num, lang)
- if debug_able:
- if is_conn:
- logger.info(f"连接了第{page_num - 1}页和第{page_num}页的段落")
-
- is_list_conn = __connect_list_inter_page(pre_page_paras, next_page_paras, pre_page_layout_bbox,
- next_page_layout_bbox, all_page_list_info[page_num - 1],
- all_page_list_info[page_num], page_num, lang)
- if debug_able:
- if is_list_conn:
- logger.info(f"连接了第{page_num - 1}页和第{page_num}页的列表段落")
-
- """接下来可能会漏掉一些特别的一些可以合并的内容，对他们进行段落连接
- 1. 正文中有时出现一个行顶格，接下来几行缩进的情况。
- 2. 居中的一些连续单行，如果高度相同，那么可能是一个段落。
- """
- for page_num, page in enumerate(pdf_info_dict.values()):
- page_paras = page['para_blocks']
- new_layout_bbox = new_layout_of_pages[page_num]
- __connect_middle_align_text(page_paras, new_layout_bbox, page_num, lang)
- __merge_signle_list_text(page_paras, new_layout_bbox, page_num, lang)
-
- # layout展平
- for page_num, page in enumerate(pdf_info_dict.values()):
- page_paras = page['para_blocks']
- page_blocks = [block for layout in page_paras for block in layout]
- page["para_blocks"] = page_blocks
diff --git a/magic_pdf/para/para_split_v3.py b/magic_pdf/para/para_split_v3.py
deleted file mode 100644
index 237c4a5e2..000000000
--- a/magic_pdf/para/para_split_v3.py
+++ /dev/null
@@ -1,296 +0,0 @@
-import copy
-
-from loguru import logger
-
-from magic_pdf.libs.Constants import LINES_DELETED, CROSS_PAGE
-from magic_pdf.libs.ocr_content_type import BlockType, ContentType
-
-LINE_STOP_FLAG = ('.', '!', '?', '。', '！', '？', ')', '）', '"', '”', ':', '：', ';', '；')
-LIST_END_FLAG = ('.', '。', ';', '；')
-
-
-class ListLineTag:
- IS_LIST_START_LINE = "is_list_start_line"
- IS_LIST_END_LINE = "is_list_end_line"
-
-
-def __process_blocks(blocks):
- # 对所有block预处理
- # 1.通过title和interline_equation将block分组
- # 2.bbox边界根据line信息重置
-
- result = []
- current_group = []
-
- for i in range(len(blocks)):
- current_block = blocks[i]
-
- # 如果当前块是 text 类型
- if current_block['type'] == 'text':
- current_block["bbox_fs"] = copy.deepcopy(current_block["bbox"])
- if 'lines' in current_block and len(current_block["lines"]) > 0:
- current_block['bbox_fs'] = [min([line['bbox'][0] for line in current_block['lines']]),
- min([line['bbox'][1] for line in current_block['lines']]),
- max([line['bbox'][2] for line in current_block['lines']]),
- max([line['bbox'][3] for line in current_block['lines']])]
- current_group.append(current_block)
-
- # 检查下一个块是否存在
- if i + 1 < len(blocks):
- next_block = blocks[i + 1]
- # 如果下一个块不是 text 类型且是 title 或 interline_equation 类型
- if next_block['type'] in ['title', 'interline_equation']:
- result.append(current_group)
- current_group = []
-
- # 处理最后一个 group
- if current_group:
- result.append(current_group)
-
- return result
-
-
-def __is_list_or_index_block(block):
- # 一个block如果是list block 应该同时满足以下特征
- # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.block内有多个line 右侧不顶格（狗牙状）
- # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.多个line以endflag结尾
- # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.block内有多个line 左侧不顶格
-
- # index block 是一种特殊的list block
- # 一个block如果是index block 应该同时满足以下特征
- # 1.block内有多个line 2.block 内有多个line两侧均顶格写 3.line的开头或者结尾均为数字
- if len(block['lines']) >= 2:
- first_line = block['lines'][0]
- line_height = first_line['bbox'][3] - first_line['bbox'][1]
- block_weight = block['bbox_fs'][2] - block['bbox_fs'][0]
-
- left_close_num = 0
- left_not_close_num = 0
- right_not_close_num = 0
- right_close_num = 0
- lines_text_list = []
-
- multiple_para_flag = False
- last_line = block['lines'][-1]
- # 如果首行左边不顶格而右边顶格,末行左边顶格而右边不顶格 （第一行可能可以右边不顶格）
- if (first_line['bbox'][0] - block['bbox_fs'][0] > line_height / 2 and
- # block['bbox_fs'][2] - first_line['bbox'][2] < line_height and
- abs(last_line['bbox'][0] - block['bbox_fs'][0]) < line_height / 2 and
- block['bbox_fs'][2] - last_line['bbox'][2] > line_height
- ):
- multiple_para_flag = True
-
- for line in block['lines']:
-
- line_text = ""
-
- for span in line['spans']:
- span_type = span['type']
- if span_type == ContentType.Text:
- line_text += span['content'].strip()
-
- lines_text_list.append(line_text)
-
- # 计算line左侧顶格数量是否大于2，是否顶格用abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height/2 来判断
- if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
- left_close_num += 1
- elif line['bbox'][0] - block['bbox_fs'][0] > line_height:
- # logger.info(f"{line_text}, {block['bbox_fs']}, {line['bbox']}")
- left_not_close_num += 1
-
- # 计算右侧是否顶格
- if abs(block['bbox_fs'][2] - line['bbox'][2]) < line_height:
- right_close_num += 1
- else:
- # 右侧不顶格情况下是否有一段距离，拍脑袋用0.3block宽度做阈值
- closed_area = 0.3 * block_weight
- # closed_area = 5 * line_height
- if block['bbox_fs'][2] - line['bbox'][2] > closed_area:
- right_not_close_num += 1
-
- # 判断lines_text_list中的元素是否有超过80%都以LIST_END_FLAG结尾
- line_end_flag = False
- # 判断lines_text_list中的元素是否有超过80%都以数字开头或都以数字结尾
- line_num_flag = False
- num_start_count = 0
- num_end_count = 0
- flag_end_count = 0
- if len(lines_text_list) > 0:
- for line_text in lines_text_list:
- if len(line_text) > 0:
- if line_text[-1] in LIST_END_FLAG:
- flag_end_count += 1
- if line_text[0].isdigit():
- num_start_count += 1
- if line_text[-1].isdigit():
- num_end_count += 1
-
- if flag_end_count / len(lines_text_list) >= 0.8:
- line_end_flag = True
-
- if num_start_count / len(lines_text_list) >= 0.8 or num_end_count / len(lines_text_list) >= 0.8:
- line_num_flag = True
-
- # 有的目录右侧不贴边, 目前认为左边或者右边有一边全贴边，且符合数字规则极为index
- if ((left_close_num/len(block['lines']) >= 0.8 or right_close_num/len(block['lines']) >= 0.8)
- and line_num_flag
- ):
- for line in block['lines']:
- line[ListLineTag.IS_LIST_START_LINE] = True
- return BlockType.Index
-
- elif left_close_num >= 2 and (
- right_not_close_num >= 2 or line_end_flag or left_not_close_num >= 2) and not multiple_para_flag:
- # 处理一种特殊的没有缩进的list，所有行都贴左边，通过右边的空隙判断是否是item尾
- if left_close_num / len(block['lines']) > 0.9:
- # 这种是每个item只有一行，且左边都贴边的短item list
- if flag_end_count == 0 and right_close_num / len(block['lines']) < 0.5:
- for line in block['lines']:
- if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
- line[ListLineTag.IS_LIST_START_LINE] = True
- # 这种是大部分line item 都有结束标识符的情况，按结束标识符区分不同item
- elif line_end_flag:
- for i, line in enumerate(block['lines']):
- if lines_text_list[i][-1] in LIST_END_FLAG:
- line[ListLineTag.IS_LIST_END_LINE] = True
- if i + 1 < len(block['lines']):
- block['lines'][i+1][ListLineTag.IS_LIST_START_LINE] = True
- # line item基本没有结束标识符，而且也没有缩进，按右侧空隙判断哪些是item end
- else:
- line_start_flag = False
- for i, line in enumerate(block['lines']):
- if line_start_flag:
- line[ListLineTag.IS_LIST_START_LINE] = True
- line_start_flag = False
- elif abs(block['bbox_fs'][2] - line['bbox'][2]) > line_height:
- line[ListLineTag.IS_LIST_END_LINE] = True
- line_start_flag = True
- # 一种有缩进的特殊有序list,start line 左侧不贴边且以数字开头，end line 以 IS_LIST_END_LINE 结尾且数量和start line 一致
- elif num_start_count >= 2 and num_start_count == flag_end_count: # 简单一点先不考虑左侧不贴边的情况
- for i, line in enumerate(block['lines']):
- if lines_text_list[i][0].isdigit():
- line[ListLineTag.IS_LIST_START_LINE] = True
- if lines_text_list[i][-1] in LIST_END_FLAG:
- line[ListLineTag.IS_LIST_END_LINE] = True
- else:
- # 正常有缩进的list处理
- for line in block['lines']:
- if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
- line[ListLineTag.IS_LIST_START_LINE] = True
- if abs(block['bbox_fs'][2] - line['bbox'][2]) > line_height:
- line[ListLineTag.IS_LIST_END_LINE] = True
-
- return BlockType.List
- else:
- return BlockType.Text
- else:
- return BlockType.Text
-
-
-def __merge_2_text_blocks(block1, block2):
- if len(block1['lines']) > 0:
- first_line = block1['lines'][0]
- line_height = first_line['bbox'][3] - first_line['bbox'][1]
- block1_weight = block1['bbox'][2] - block1['bbox'][0]
- block2_weight = block2['bbox'][2] - block2['bbox'][0]
- min_block_weight = min(block1_weight, block2_weight)
- if abs(block1['bbox_fs'][0] - first_line['bbox'][0]) < line_height / 2:
- last_line = block2['lines'][-1]
- if len(last_line['spans']) > 0:
- last_span = last_line['spans'][-1]
- line_height = last_line['bbox'][3] - last_line['bbox'][1]
- if (abs(block2['bbox_fs'][2] - last_line['bbox'][2]) < line_height and
- not last_span['content'].endswith(LINE_STOP_FLAG) and
- # 两个block宽度差距超过2倍也不合并
- abs(block1_weight - block2_weight) < min_block_weight
- ):
- if block1['page_num'] != block2['page_num']:
- for line in block1['lines']:
- for span in line['spans']:
- span[CROSS_PAGE] = True
- block2['lines'].extend(block1['lines'])
- block1['lines'] = []
- block1[LINES_DELETED] = True
-
- return block1, block2
-
-
-def __merge_2_list_blocks(block1, block2):
- if block1['page_num'] != block2['page_num']:
- for line in block1['lines']:
- for span in line['spans']:
- span[CROSS_PAGE] = True
- block2['lines'].extend(block1['lines'])
- block1['lines'] = []
- block1[LINES_DELETED] = True
-
- return block1, block2
-
-
-def __is_list_group(text_blocks_group):
- # list group的特征是一个group内的所有block都满足以下条件
- # 1.每个block都不超过3行 2. 每个block 的左边界都比较接近(逻辑简单点先不加这个规则)
- for block in text_blocks_group:
- if len(block['lines']) > 3:
- return False
- return True
-
-
-def __para_merge_page(blocks):
- page_text_blocks_groups = __process_blocks(blocks)
- for text_blocks_group in page_text_blocks_groups:
-
- if len(text_blocks_group) > 0:
- # 需要先在合并前对所有block判断是否为list or index block
- for block in text_blocks_group:
- block_type = __is_list_or_index_block(block)
- block['type'] = block_type
- # logger.info(f"{block['type']}:{block}")
-
- if len(text_blocks_group) > 1:
-
- # 在合并前判断这个group 是否是一个 list group
- is_list_group = __is_list_group(text_blocks_group)
-
- # 倒序遍历
- for i in range(len(text_blocks_group) - 1, -1, -1):
- current_block = text_blocks_group[i]
-
- # 检查是否有前一个块
- if i - 1 >= 0:
- prev_block = text_blocks_group[i - 1]
-
- if current_block['type'] == 'text' and prev_block['type'] == 'text' and not is_list_group:
- __merge_2_text_blocks(current_block, prev_block)
- elif (
- (current_block['type'] == BlockType.List and prev_block['type'] == BlockType.List) or
- (current_block['type'] == BlockType.Index and prev_block['type'] == BlockType.Index)
- ):
- __merge_2_list_blocks(current_block, prev_block)
-
- else:
- continue
-
-
-def para_split(pdf_info_dict, debug_mode=False):
- all_blocks = []
- for page_num, page in pdf_info_dict.items():
- blocks = copy.deepcopy(page['preproc_blocks'])
- for block in blocks:
- block['page_num'] = page_num
- all_blocks.extend(blocks)
-
- __para_merge_page(all_blocks)
- for page_num, page in pdf_info_dict.items():
- page['para_blocks'] = []
- for block in all_blocks:
- if block['page_num'] == page_num:
- page['para_blocks'].append(block)
-
-
-if __name__ == '__main__':
- input_blocks = []
- # 调用函数
- groups = __process_blocks(input_blocks)
- for group_index, group in enumerate(groups):
- print(f"Group {group_index}: {group}")
diff --git a/magic_pdf/para/raw_processor.py b/magic_pdf/para/raw_processor.py
deleted file mode 100644
index edbf9964b..000000000
--- a/magic_pdf/para/raw_processor.py
+++ /dev/null
@@ -1,207 +0,0 @@
-class RawBlockProcessor:
- def __init__(self) -> None:
- self.y_tolerance = 2
- self.pdf_dic = {}
-
- def __span_flags_decomposer(self, span_flags):
- """
- Make font flags human readable.
-
- Parameters
- ----------
- self : object
- The instance of the class.
-
- span_flags : int
- span flags
-
- Returns
- -------
- l : dict
- decomposed flags
- """
-
- l = {
- "is_superscript": False,
- "is_italic": False,
- "is_serifed": False,
- "is_sans_serifed": False,
- "is_monospaced": False,
- "is_proportional": False,
- "is_bold": False,
- }
-
- if span_flags & 2**0:
- l["is_superscript"] = True # 表示上标
-
- if span_flags & 2**1:
- l["is_italic"] = True # 表示斜体
-
- if span_flags & 2**2:
- l["is_serifed"] = True # 表示衬线字体
- else:
- l["is_sans_serifed"] = True # 表示非衬线字体
-
- if span_flags & 2**3:
- l["is_monospaced"] = True # 表示等宽字体
- else:
- l["is_proportional"] = True # 表示比例字体
-
- if span_flags & 2**4:
- l["is_bold"] = True # 表示粗体
-
- return l
-
- def __make_new_lines(self, raw_lines):
- """
- This function makes new lines.
-
- Parameters
- ----------
- self : object
- The instance of the class.
-
- raw_lines : list
- raw lines
-
- Returns
- -------
- new_lines : list
- new lines
- """
- new_lines = []
- new_line = None
-
- for raw_line in raw_lines:
- raw_line_bbox = raw_line["bbox"]
- raw_line_spans = raw_line["spans"]
- raw_line_text = "".join([span["text"] for span in raw_line_spans])
- raw_line_dir = raw_line.get("dir", None)
-
- decomposed_line_spans = []
- for span in raw_line_spans:
- raw_flags = span["flags"]
- decomposed_flags = self.__span_flags_decomposer(raw_flags)
- span["decomposed_flags"] = decomposed_flags
- decomposed_line_spans.append(span)
-
- if new_line is None:
- new_line = {
- "bbox": raw_line_bbox,
- "text": raw_line_text,
- "dir": raw_line_dir if raw_line_dir else (0, 0),
- "spans": decomposed_line_spans,
- }
- else:
- if (
- abs(raw_line_bbox[1] - new_line["bbox"][1]) <= self.y_tolerance
- and abs(raw_line_bbox[3] - new_line["bbox"][3]) <= self.y_tolerance
- ):
- new_line["bbox"] = (
- min(new_line["bbox"][0], raw_line_bbox[0]), # left
- new_line["bbox"][1], # top
- max(new_line["bbox"][2], raw_line_bbox[2]), # right
- raw_line_bbox[3], # bottom
- )
- new_line["text"] += " " + raw_line_text
- new_line["spans"].extend(raw_line_spans)
- new_line["dir"] = (
- new_line["dir"][0] + raw_line_dir[0],
- new_line["dir"][1] + raw_line_dir[1],
- )
- else:
- new_lines.append(new_line)
- new_line = {
- "bbox": raw_line_bbox,
- "text": raw_line_text,
- "dir": raw_line_dir if raw_line_dir else (0, 0),
- "spans": raw_line_spans,
- }
- if new_line:
- new_lines.append(new_line)
-
- return new_lines
-
- def __make_new_block(self, raw_block):
- """
- This function makes a new block.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- raw_block : dict
- a raw block
-
- Returns
- -------
- new_block : dict
-
- Schema of new_block:
- {
- "block_id": "block_1",
- "bbox": [0, 0, 100, 100],
- "text": "This is a block.",
- "lines": [
- {
- "bbox": [0, 0, 100, 100],
- "text": "This is a line.",
- "spans": [
- {
- "text": "This is a span.",
- "font": "Times New Roman",
- "size": 12,
- "color": "#000000",
- }
- ],
- }
- ],
- }
- """
- new_block = {}
-
- block_id = raw_block["number"]
- block_bbox = raw_block["bbox"]
- block_text = " ".join(span["text"] for line in raw_block["lines"] for span in line["spans"])
- raw_lines = raw_block["lines"]
- block_lines = self.__make_new_lines(raw_lines)
-
- new_block["block_id"] = block_id
- new_block["bbox"] = block_bbox
- new_block["text"] = block_text
- new_block["lines"] = block_lines
-
- return new_block
-
- def batch_process_blocks(self, pdf_dic):
- """
- This function processes the blocks in batch.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- blocks : list
- Input block is a list of raw blocks. Schema can refer to the value of key ""preproc_blocks", demo file is app/pdf_toolbox/tests/preproc_2_parasplit_example.json.
-
- Returns
- -------
- result_dict : dict
- result dictionary
- """
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "preproc_blocks" in blocks.keys():
- input_blocks = blocks["preproc_blocks"]
- for raw_block in input_blocks:
- new_block = self.__make_new_block(raw_block)
- para_blocks.append(new_block)
-
- blocks["para_blocks"] = para_blocks
-
- return pdf_dic
-
diff --git a/magic_pdf/para/stats.py b/magic_pdf/para/stats.py
deleted file mode 100644
index fd509b952..000000000
--- a/magic_pdf/para/stats.py
+++ /dev/null
@@ -1,268 +0,0 @@
-from collections import Counter
-import numpy as np
-
-from magic_pdf.para.commons import *
-
-
-if sys.version_info[0] >= 3:
- sys.stdout.reconfigure(encoding="utf-8") # type: ignore
-
-
-class BlockStatisticsCalculator:
- def __init__(self) -> None:
- pass
-
- def __calc_stats_of_new_lines(self, new_lines):
- """
- This function calculates the paragraph metrics
-
- Parameters
- ----------
- combined_lines : list
- combined lines
-
- Returns
- -------
- X0 : float
- Median of x0 values, which represents the left average boundary of the block
- X1 : float
- Median of x1 values, which represents the right average boundary of the block
- avg_char_width : float
- Average of char widths, which represents the average char width of the block
- avg_char_height : float
- Average of line heights, which represents the average line height of the block
-
- """
- x0_values = []
- x1_values = []
- char_widths = []
- char_heights = []
-
- block_font_types = []
- block_font_sizes = []
- block_directions = []
-
- if len(new_lines) > 0:
- for i, line in enumerate(new_lines):
- line_bbox = line["bbox"]
- line_text = line["text"]
- line_spans = line["spans"]
-
- num_chars = len([ch for ch in line_text if not ch.isspace()])
-
- x0_values.append(line_bbox[0])
- x1_values.append(line_bbox[2])
-
- if num_chars > 0:
- char_width = (line_bbox[2] - line_bbox[0]) / num_chars
- char_widths.append(char_width)
-
- for span in line_spans:
- block_font_types.append(span["font"])
- block_font_sizes.append(span["size"])
-
- if "dir" in line:
- block_directions.append(line["dir"])
-
- # line_font_types = [span["font"] for span in line_spans]
- char_heights = [span["size"] for span in line_spans]
-
- X0 = np.median(x0_values) if x0_values else 0
- X1 = np.median(x1_values) if x1_values else 0
- avg_char_width = sum(char_widths) / len(char_widths) if char_widths else 0
- avg_char_height = sum(char_heights) / len(char_heights) if char_heights else 0
-
- # max_freq_font_type = max(set(block_font_types), key=block_font_types.count) if block_font_types else None
-
- max_span_length = 0
- max_span_font_type = None
- for line in new_lines:
- line_spans = line["spans"]
- for span in line_spans:
- span_length = span["bbox"][2] - span["bbox"][0]
- if span_length > max_span_length:
- max_span_length = span_length
- max_span_font_type = span["font"]
-
- max_freq_font_type = max_span_font_type
-
- avg_font_size = sum(block_font_sizes) / len(block_font_sizes) if block_font_sizes else None
-
- avg_dir_horizontal = sum([dir[0] for dir in block_directions]) / len(block_directions) if block_directions else 0
- avg_dir_vertical = sum([dir[1] for dir in block_directions]) / len(block_directions) if block_directions else 0
-
- median_font_size = float(np.median(block_font_sizes)) if block_font_sizes else None
-
- return (
- X0,
- X1,
- avg_char_width,
- avg_char_height,
- max_freq_font_type,
- avg_font_size,
- (avg_dir_horizontal, avg_dir_vertical),
- median_font_size,
- )
-
- def __make_new_block(self, input_block):
- new_block = {}
-
- raw_lines = input_block["lines"]
- stats = self.__calc_stats_of_new_lines(raw_lines)
-
- block_id = input_block["block_id"]
- block_bbox = input_block["bbox"]
- block_text = input_block["text"]
- block_lines = raw_lines
- block_avg_left_boundary = stats[0]
- block_avg_right_boundary = stats[1]
- block_avg_char_width = stats[2]
- block_avg_char_height = stats[3]
- block_font_type = stats[4]
- block_font_size = stats[5]
- block_direction = stats[6]
- block_median_font_size = stats[7]
-
- new_block["block_id"] = block_id
- new_block["bbox"] = block_bbox
- new_block["text"] = block_text
- new_block["dir"] = block_direction
- new_block["X0"] = block_avg_left_boundary
- new_block["X1"] = block_avg_right_boundary
- new_block["avg_char_width"] = block_avg_char_width
- new_block["avg_char_height"] = block_avg_char_height
- new_block["block_font_type"] = block_font_type
- new_block["block_font_size"] = block_font_size
- new_block["lines"] = block_lines
- new_block["median_font_size"] = block_median_font_size
-
- return new_block
-
- def batch_process_blocks(self, pdf_dic):
- """
- This function processes the blocks in batch.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- blocks : list
- Input block is a list of raw blocks. Schema can refer to the value of key ""preproc_blocks", demo file is app/pdf_toolbox/tests/preproc_2_parasplit_example.json
-
- Returns
- -------
- result_dict : dict
- result dictionary
- """
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "para_blocks" in blocks.keys():
- input_blocks = blocks["para_blocks"]
- for input_block in input_blocks:
- new_block = self.__make_new_block(input_block)
- para_blocks.append(new_block)
-
- blocks["para_blocks"] = para_blocks
-
- return pdf_dic
-
-
-class DocStatisticsCalculator:
- def __init__(self) -> None:
- pass
-
- def calc_stats_of_doc(self, pdf_dict):
- """
- This function computes the statistics of the document
-
- Parameters
- ----------
- result_dict : dict
- result dictionary
-
- Returns
- -------
- statistics : dict
- statistics of the document
- """
-
- total_text_length = 0
- total_num_blocks = 0
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "para_blocks" in blocks.keys():
- para_blocks = blocks["para_blocks"]
- for para_block in para_blocks:
- total_text_length += len(para_block["text"])
- total_num_blocks += 1
-
- avg_text_length = total_text_length / total_num_blocks if total_num_blocks else 0
-
- font_list = []
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "para_blocks" in blocks.keys():
- input_blocks = blocks["para_blocks"]
- for input_block in input_blocks:
- block_text_length = len(input_block.get("text", ""))
- if block_text_length < avg_text_length * 0.5:
- continue
- block_font_type = safe_get(input_block, "block_font_type", "")
- block_font_size = safe_get(input_block, "block_font_size", 0)
- font_list.append((block_font_type, block_font_size))
-
- font_counter = Counter(font_list)
- most_common_font = font_counter.most_common(1)[0] if font_list else (("", 0), 0)
- second_most_common_font = font_counter.most_common(2)[1] if len(font_counter) > 1 else (("", 0), 0)
-
- statistics = {
- "num_pages": 0,
- "num_blocks": 0,
- "num_paras": 0,
- "num_titles": 0,
- "num_header_blocks": 0,
- "num_footer_blocks": 0,
- "num_watermark_blocks": 0,
- "num_vertical_margin_note_blocks": 0,
- "most_common_font_type": most_common_font[0][0],
- "most_common_font_size": most_common_font[0][1],
- "number_of_most_common_font": most_common_font[1],
- "second_most_common_font_type": second_most_common_font[0][0],
- "second_most_common_font_size": second_most_common_font[0][1],
- "number_of_second_most_common_font": second_most_common_font[1],
- "avg_text_length": avg_text_length,
- }
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- blocks = pdf_dict[page_id]["para_blocks"]
- statistics["num_pages"] += 1
- for block_id, block_data in enumerate(blocks):
- statistics["num_blocks"] += 1
-
- if "paras" in block_data.keys():
- statistics["num_paras"] += len(block_data["paras"])
-
- for line in block_data["lines"]:
- if line.get("is_title", 0):
- statistics["num_titles"] += 1
-
- if block_data.get("is_header", 0):
- statistics["num_header_blocks"] += 1
- if block_data.get("is_footer", 0):
- statistics["num_footer_blocks"] += 1
- if block_data.get("is_watermark", 0):
- statistics["num_watermark_blocks"] += 1
- if block_data.get("is_vertical_margin_note", 0):
- statistics["num_vertical_margin_note_blocks"] += 1
-
- pdf_dict["statistics"] = statistics
-
- return pdf_dict
-
-
diff --git a/magic_pdf/para/title_processor.py b/magic_pdf/para/title_processor.py
deleted file mode 100644
index 00d330fcd..000000000
--- a/magic_pdf/para/title_processor.py
+++ /dev/null
@@ -1,1014 +0,0 @@
-import os
-import re
-import numpy as np
-
-from magic_pdf.libs.nlp_utils import NLPModels
-
-from magic_pdf.para.commons import *
-
-if sys.version_info[0] >= 3:
- sys.stdout.reconfigure(encoding="utf-8") # type: ignore
-
-
-class TitleProcessor:
- def __init__(self, *doc_statistics) -> None:
- if len(doc_statistics) > 0:
- self.doc_statistics = doc_statistics[0]
-
- self.nlp_model = NLPModels()
- self.MAX_TITLE_LEVEL = 3
- self.numbered_title_pattern = r"""
- ^ # 行首
- ( # 开始捕获组
- [\(\（]\d+[\)\）] # 括号内数字，支持中文和英文括号，例如：(1) 或 （1）
- |\d+[\)\）]\s # 数字后跟右括号和空格，支持中文和英文括号，例如：2) 或 2）
- |[\(\（][A-Z][\)\）] # 括号内大写字母，支持中文和英文括号，例如：(A) 或 （A）
- |[A-Z][\)\）]\s # 大写字母后跟右括号和空格，例如：A) 或 A）
- |[\(\（][IVXLCDM]+[\)\）] # 括号内罗马数字，支持中文和英文括号，例如：(I) 或 （I）
- |[IVXLCDM]+[\)\）]\s # 罗马数字后跟右括号和空格，例如：I) 或 I）
- |\d+(\.\d+)*\s # 数字或复合数字编号后跟空格，例如：1. 或 3.2.1
- |[一二三四五六七八九十百千]+[、\s] # 中文序号后跟顿号和空格，例如：一、
- |[\（|\(][一二三四五六七八九十百千]+[\）|\)]\s* # 中文括号内中文序号后跟空格，例如：（一）
- |[A-Z]\.\d+(\.\d+)?\s # 大写字母后跟点和数字，例如：A.1 或 A.1.1
- |[\(\（][a-z][\)\）] # 括号内小写字母，支持中文和英文括号，例如：(a) 或 （a）
- |[a-z]\)\s # 小写字母后跟右括号和空格，例如：a)
- |[A-Z]-\s # 大写字母后跟短横线和空格，例如：A-
- |\w+:\s # 英文序号词后跟冒号和空格，例如：First:
- |第[一二三四五六七八九十百千]+[章节部分条款]\s # 以“第”开头的中文标题后跟空格
- |[IVXLCDM]+\. # 罗马数字后跟点，例如：I.
- |\d+\.\s # 单个数字后跟点和空格，例如：1.
- ) # 结束捕获组
- .+ # 标题的其余部分
- """
-
- def _is_potential_title(
- self,
- curr_line,
- prev_line,
- prev_line_is_title,
- next_line,
- avg_char_width,
- avg_char_height,
- median_font_size,
- ):
- """
- This function checks if the line is a potential title.
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
- next_line : dict
- next line
- avg_char_width : float
- average of char widths
- avg_char_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the line is a potential title, False otherwise.
- """
-
- def __is_line_centered(line_bbox, page_bbox, avg_char_width):
- """
- This function checks if the line is centered on the page
-
- Parameters
- ----------
- line_bbox : list
- bbox of the line
- page_bbox : list
- bbox of the page
- avg_char_width : float
- average of char widths
-
- Returns
- -------
- bool
- True if the line is centered on the page, False otherwise.
- """
- horizontal_ratio = 0.5
- horizontal_thres = horizontal_ratio * avg_char_width
-
- x0, _, x1, _ = line_bbox
- _, _, page_x1, _ = page_bbox
-
- return abs((x0 + x1) / 2 - page_x1 / 2) < horizontal_thres
-
- def __is_bold_font_line(line):
- """
- Check if a line contains any bold font style.
- """
-
- def _is_bold_span(span):
- # if span text is empty or only contains space, return False
- if not span["text"].strip():
- return False
-
- return bool(span["flags"] & 2**4) # Check if the font is bold
-
- for span in line["spans"]:
- if not _is_bold_span(span):
- return False
-
- return True
-
- def __is_italic_font_line(line):
- """
- Check if a line contains any italic font style.
- """
-
- def __is_italic_span(span):
- return bool(span["flags"] & 2**1) # Check if the font is italic
-
- for span in line["spans"]:
- if not __is_italic_span(span):
- return False
-
- return True
-
- def __is_punctuation_heavy(line_text):
- """
- Check if the line contains a high ratio of punctuation marks, which may indicate
- that the line is not a title.
-
- Parameters:
- line_text (str): Text of the line.
-
- Returns:
- bool: True if the line is heavy with punctuation, False otherwise.
- """
- # Pattern for common title format like "X.Y. Title"
- pattern = r"\b\d+\.\d+\..*\b"
-
- # If the line matches the title format, return False
- if re.match(pattern, line_text.strip()):
- return False
-
- # Find all punctuation marks in the line
- punctuation_marks = re.findall(r"[^\w\s]", line_text)
- number_of_punctuation_marks = len(punctuation_marks)
-
- text_length = len(line_text)
-
- if text_length == 0:
- return False
-
- punctuation_ratio = number_of_punctuation_marks / text_length
- if punctuation_ratio >= 0.1:
- return True
-
- return False
-
- def __has_mixed_font_styles(spans, strict_mode=False):
- """
- This function checks if the line has mixed font styles, the strict mode will compare the font types
-
- Parameters
- ----------
- spans : list
- spans of the line
- strict_mode : bool
- True for strict mode, the font types will be fully compared
- False for non-strict mode, the font types will be compared by the most longest common prefix
-
- Returns
- -------
- bool
- True if the line has mixed font styles, False otherwise.
- """
- if strict_mode:
- font_styles = set()
- for span in spans:
- font_style = span["font"].lower()
- font_styles.add(font_style)
-
- return len(font_styles) > 1
-
- else: # non-strict mode
- font_styles = []
- for span in spans:
- font_style = span["font"].lower()
- font_styles.append(font_style)
-
- if len(font_styles) > 1:
- longest_common_prefix = os.path.commonprefix(font_styles)
- if len(longest_common_prefix) > 0:
- return False
- else:
- return True
- else:
- return False
-
- def __is_different_font_type_from_neighbors(curr_line_font_type, prev_line_font_type, next_line_font_type):
- """
- This function checks if the current line has a different font type from the previous and next lines
-
- Parameters
- ----------
- curr_line_font_type : str
- font type of the current line
- prev_line_font_type : str
- font type of the previous line
- next_line_font_type : str
- font type of the next line
-
- Returns
- -------
- bool
- True if the current line has a different font type from the previous and next lines, False otherwise.
- """
- return all(
- curr_line_font_type != other_font_type.lower()
- for other_font_type in [prev_line_font_type, next_line_font_type]
- if other_font_type is not None
- )
-
- def __is_larger_font_size_from_neighbors(curr_line_font_size, prev_line_font_size, next_line_font_size):
- """
- This function checks if the current line has a larger font size than the previous and next lines
-
- Parameters
- ----------
- curr_line_font_size : float
- font size of the current line
- prev_line_font_size : float
- font size of the previous line
- next_line_font_size : float
- font size of the next line
-
- Returns
- -------
- bool
- True if the current line has a larger font size than the previous and next lines, False otherwise.
- """
- return all(
- curr_line_font_size > other_font_size * 1.2
- for other_font_size in [prev_line_font_size, next_line_font_size]
- if other_font_size is not None
- )
-
- def __is_similar_to_pre_line(curr_line_font_type, prev_line_font_type, curr_line_font_size, prev_line_font_size):
- """
- This function checks if the current line is similar to the previous line
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
-
- Returns
- -------
- bool
- True if the current line is similar to the previous line, False otherwise.
- """
-
- if curr_line_font_type == prev_line_font_type and curr_line_font_size == prev_line_font_size:
- return True
- else:
- return False
-
- def __is_same_font_type_of_docAvg(curr_line_font_type):
- """
- This function checks if the current line has the same font type as the document average font type
-
- Parameters
- ----------
- curr_line_font_type : str
- font type of the current line
-
- Returns
- -------
- bool
- True if the current line has the same font type as the document average font type, False otherwise.
- """
- doc_most_common_font_type = safe_get(self.doc_statistics, "most_common_font_type", "").lower()
- doc_second_most_common_font_type = safe_get(self.doc_statistics, "second_most_common_font_type", "").lower()
-
- return curr_line_font_type.lower() in [doc_most_common_font_type, doc_second_most_common_font_type]
-
- def __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio: float = 1):
- """
- This function checks if the current line has a large enough font size
-
- Parameters
- ----------
- curr_line_font_size : float
- font size of the current line
- ratio : float
- ratio of the current line font size to the document average font size
-
- Returns
- -------
- bool
- True if the current line has a large enough font size, False otherwise.
- """
- doc_most_common_font_size = safe_get(self.doc_statistics, "most_common_font_size", 0)
- doc_second_most_common_font_size = safe_get(self.doc_statistics, "second_most_common_font_size", 0)
- doc_avg_font_size = min(doc_most_common_font_size, doc_second_most_common_font_size)
-
- return curr_line_font_size >= doc_avg_font_size * ratio
-
- def __is_sufficient_spacing_above_and_below(
- curr_line_bbox,
- prev_line_bbox,
- next_line_bbox,
- avg_char_height,
- median_font_size,
- ):
- """
- This function checks if the current line has sufficient spacing above and below
-
- Parameters
- ----------
- curr_line_bbox : list
- bbox of the current line
- prev_line_bbox : list
- bbox of the previous line
- next_line_bbox : list
- bbox of the next line
- avg_char_width : float
- average of char widths
- avg_char_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the current line has sufficient spacing above and below, False otherwise.
- """
- vertical_ratio = 1.25
- vertical_thres = vertical_ratio * median_font_size
-
- _, y0, _, y1 = curr_line_bbox
-
- sufficient_spacing_above = False
- if prev_line_bbox:
- vertical_spacing_above = min(y0 - prev_line_bbox[1], y1 - prev_line_bbox[3])
- sufficient_spacing_above = vertical_spacing_above > vertical_thres
- else:
- sufficient_spacing_above = True
-
- sufficient_spacing_below = False
- if next_line_bbox:
- vertical_spacing_below = min(next_line_bbox[1] - y0, next_line_bbox[3] - y1)
- sufficient_spacing_below = vertical_spacing_below > vertical_thres
- else:
- sufficient_spacing_below = True
-
- return (sufficient_spacing_above, sufficient_spacing_below)
-
- def __is_word_list_line_by_rules(curr_line_text):
- """
- This function checks if the current line is a word list
-
- Parameters
- ----------
- curr_line_text : str
- text of the current line
-
- Returns
- -------
- bool
- True if the current line is a name list, False otherwise.
- """
- # name_list_pattern = r"([a-zA-Z][a-zA-Z\s]{0,20}[a-zA-Z]|[\u4e00-\u9fa5·]{2,16})(?=[，,;；\s]|$)"
- name_list_pattern = r"(?= 0.9:
- return True
-
- return False
-
- def __is_equation(line_text):
- """
- This function checks if the current line is an equation.
-
- Parameters
- ----------
- line_text : str
-
- Returns
- -------
- bool
- True if the current line is an equation, False otherwise.
- """
- equation_reg = r"\$.*?\\overline.*?\$" # to match interline equations
-
- if re.search(equation_reg, line_text):
- return True
- else:
- return False
-
- def __is_title_by_len(text, max_length=200):
- """
- This function checks if the current line is a title by length.
-
- Parameters
- ----------
- text : str
- text of the current line
-
- max_length : int
- max length of the title
-
- Returns
- -------
- bool
- True if the current line is a title, False otherwise.
-
- """
- text = text.strip()
- return len(text) <= max_length
-
- def __compute_line_font_type_and_size(curr_line):
- """
- This function computes the font type and font size of the line.
-
- Parameters
- ----------
- line : dict
- line
-
- Returns
- -------
- font_type : str
- font type of the line
- font_size : float
- font size of the line
- """
- spans = curr_line["spans"]
- max_accumulated_length = 0
- max_span_font_size = curr_line["spans"][0]["size"] # default value, float type
- max_span_font_type = curr_line["spans"][0]["font"].lower() # default value, string type
- for span in spans:
- if span["text"].isspace():
- continue
- span_length = span["bbox"][2] - span["bbox"][0]
- if span_length > max_accumulated_length:
- max_accumulated_length = span_length
- max_span_font_size = span["size"]
- max_span_font_type = span["font"].lower()
-
- return max_span_font_type, max_span_font_size
-
- """
- Title detecting main Process.
- """
-
- """
- Basic features about the current line.
- """
- curr_line_bbox = curr_line["bbox"]
- curr_line_text = curr_line["text"]
- curr_line_font_type, curr_line_font_size = __compute_line_font_type_and_size(curr_line)
-
- if len(curr_line_text.strip()) == 0: # skip empty lines
- return False
-
- prev_line_bbox = prev_line["bbox"] if prev_line else None
- if prev_line:
- prev_line_font_type, prev_line_font_size = __compute_line_font_type_and_size(prev_line)
- else:
- prev_line_font_type, prev_line_font_size = None, None
-
- next_line_bbox = next_line["bbox"] if next_line else None
- if next_line:
- next_line_font_type, next_line_font_size = __compute_line_font_type_and_size(next_line)
- else:
- next_line_font_type, next_line_font_size = None, None
-
- """
- Aggregated features about the current line.
- """
- is_italc_font = __is_italic_font_line(curr_line)
- is_bold_font = __is_bold_font_line(curr_line)
-
- is_font_size_little_less_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=0.8)
- is_font_size_not_less_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=1)
- is_much_larger_font_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=1.6)
-
- is_not_same_font_type_of_docAvg = not __is_same_font_type_of_docAvg(curr_line_font_type)
-
- is_potential_title_font = is_bold_font or is_font_size_not_less_than_doc_avg or is_not_same_font_type_of_docAvg
-
- is_mix_font_styles_strict = __has_mixed_font_styles(curr_line["spans"], strict_mode=True)
- is_mix_font_styles_loose = __has_mixed_font_styles(curr_line["spans"], strict_mode=False)
-
- is_punctuation_heavy = __is_punctuation_heavy(curr_line_text)
-
- is_word_list_line_by_rules = __is_word_list_line_by_rules(curr_line_text)
- is_person_or_org_list_line_by_nlp = __get_text_catgr_by_nlp(curr_line_text) in ["PERSON", "GPE", "ORG"]
-
- is_font_size_larger_than_neighbors = __is_larger_font_size_from_neighbors(
- curr_line_font_size, prev_line_font_size, next_line_font_size
- )
-
- is_font_type_diff_from_neighbors = __is_different_font_type_from_neighbors(
- curr_line_font_type, prev_line_font_type, next_line_font_type
- )
-
- has_sufficient_spaces_above, has_sufficient_spaces_below = __is_sufficient_spacing_above_and_below(
- curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_height, median_font_size
- )
-
- is_similar_to_pre_line = __is_similar_to_pre_line(
- curr_line_font_type, prev_line_font_type, curr_line_font_size, prev_line_font_size
- )
-
- """
- Further aggregated features about the current line.
-
- Attention:
- Features that start with __ are for internal use.
- """
-
- __is_line_left_aligned_from_neighbors = is_line_left_aligned_from_neighbors(
- curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width
- )
- __is_font_diff_from_neighbors = is_font_size_larger_than_neighbors or is_font_type_diff_from_neighbors
- is_a_left_inline_title = (
- is_mix_font_styles_strict and __is_line_left_aligned_from_neighbors and __is_font_diff_from_neighbors
- )
-
- is_title_by_check_prev_line = prev_line is None and has_sufficient_spaces_above and is_potential_title_font
- is_title_by_check_next_line = next_line is None and has_sufficient_spaces_below and is_potential_title_font
-
- is_title_by_check_pre_and_next_line = (
- (prev_line is not None or next_line is not None)
- and has_sufficient_spaces_above
- and has_sufficient_spaces_below
- and is_potential_title_font
- )
-
- is_numbered_title = __is_numbered_title(curr_line_text) and (
- (has_sufficient_spaces_above or prev_line is None) and (has_sufficient_spaces_below or next_line is None)
- )
-
- is_not_end_with_ending_puncs = not __is_end_with_ending_puncs(curr_line_text)
-
- is_not_only_no_meaning_symbols = not __contains_only_no_meaning_symbols(curr_line_text)
-
- is_equation = __is_equation(curr_line_text)
-
- is_title_by_len = __is_title_by_len(curr_line_text)
-
- """
- Decide if the line is a title.
- """
- # is_title = False
- # if prev_line_is_title:
-
- is_title = (
- is_not_end_with_ending_puncs # not end with ending punctuation marks
- and is_not_only_no_meaning_symbols # not only have no meaning symbols
- and is_title_by_len # is a title by length, default max length is 200
- and not is_equation # an interline equation should never be a title
- and is_potential_title_font # is a potential title font, which is bold or larger than the document average font size or not the same font type as the document average font type
- and (
- (is_not_same_font_type_of_docAvg and is_font_size_not_less_than_doc_avg)
- or (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- or (
- is_much_larger_font_than_doc_avg
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- or (
- is_font_size_little_less_than_doc_avg
- and is_bold_font
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- ) # not the same font type as the document average font type, which includes the most common font type and the second most common font type
- and (
- (
- not is_person_or_org_list_line_by_nlp
- and (
- is_much_larger_font_than_doc_avg
- or (is_not_same_font_type_of_docAvg and is_font_size_not_less_than_doc_avg)
- )
- )
- or (
- not (is_word_list_line_by_rules and is_person_or_org_list_line_by_nlp)
- and not is_a_left_inline_title
- and not is_punctuation_heavy
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- or (
- is_person_or_org_list_line_by_nlp
- and (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- and (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- )
- or (is_numbered_title and not is_a_left_inline_title)
- )
- )
- # ) or (is_similar_to_pre_line and prev_line_is_title)
-
- is_name_or_org_list_to_be_removed = (
- (is_person_or_org_list_line_by_nlp)
- and is_punctuation_heavy
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- ) and not is_title
-
- if is_name_or_org_list_to_be_removed:
- is_author_or_org_list = True
- # print curr_line_text to check
- # print_yellow(f"Text of is_author_or_org_list: {curr_line_text}")
- else:
- is_author_or_org_list = False
- """
- # print reason why the line is a title
- if is_title:
- print_green("This line is a title.")
- print_green("↓" * 10)
- print()
- print("curr_line_text: ", curr_line_text)
- print()
-
- # print reason why the line is not a title
- line_text = curr_line_text.strip()
- test_text = "Career/Personal Life"
- text_content_condition = line_text == test_text
-
- if not is_title and text_content_condition: # Print specific line
- # if not is_title: # Print each line
- print_red("This line is not a title.")
- print_red("↓" * 10)
-
- print()
- print("curr_line_text: ", curr_line_text)
- print()
-
- if is_not_end_with_ending_puncs:
- print_green(f"is_not_end_with_ending_puncs")
- else:
- print_red(f"is_end_with_ending_puncs")
-
- if is_not_only_no_meaning_symbols:
- print_green(f"is_not_only_no_meaning_symbols")
- else:
- print_red(f"is_only_no_meaning_symbols")
-
- if is_title_by_len:
- print_green(f"is_title_by_len: {is_title_by_len}")
- else:
- print_red(f"is_not_title_by_len: {is_title_by_len}")
-
- if is_equation:
- print_red(f"is_equation")
- else:
- print_green(f"is_not_equation")
-
- if is_potential_title_font:
- print_green(f"is_potential_title_font")
- else:
- print_red(f"is_not_potential_title_font")
-
- if is_punctuation_heavy:
- print_red("is_punctuation_heavy")
- else:
- print_green("is_not_punctuation_heavy")
-
- if is_bold_font:
- print_green(f"is_bold_font")
- else:
- print_red(f"is_not_bold_font")
-
- if is_font_size_not_less_than_doc_avg:
- print_green(f"is_larger_font_than_doc_avg")
- else:
- print_red(f"is_not_larger_font_than_doc_avg")
-
- if is_much_larger_font_than_doc_avg:
- print_green(f"is_much_larger_font_than_doc_avg")
- else:
- print_red(f"is_not_much_larger_font_than_doc_avg")
-
- if is_not_same_font_type_of_docAvg:
- print_green(f"is_not_same_font_type_of_docAvg")
- else:
- print_red(f"is_same_font_type_of_docAvg")
-
- if is_word_list_line_by_rules:
- print_red("is_word_list_line_by_rules")
- else:
- print_green("is_not_name_list_by_rules")
-
- if is_person_or_org_list_line_by_nlp:
- print_red("is_person_or_org_list_line_by_nlp")
- else:
- print_green("is_not_person_or_org_list_line_by_nlp")
-
- if not is_numbered_title:
- print_red("is_not_numbered_title")
- else:
- print_green("is_numbered_title")
-
- if is_a_left_inline_title:
- print_red("is_a_left_inline_title")
- else:
- print_green("is_not_a_left_inline_title")
-
- if not is_title_by_check_prev_line:
- print_red("is_not_title_by_check_prev_line")
- else:
- print_green("is_title_by_check_prev_line")
-
- if not is_title_by_check_next_line:
- print_red("is_not_title_by_check_next_line")
- else:
- print_green("is_title_by_check_next_line")
-
- if not is_title_by_check_pre_and_next_line:
- print_red("is_not_title_by_check_pre_and_next_line")
- else:
- print_green("is_title_by_check_pre_and_next_line")
-
- # print_green("Common features:")
- # print_green("↓" * 10)
-
- # print(f" curr_line_font_type: {curr_line_font_type}")
- # print(f" curr_line_font_size: {curr_line_font_size}")
- # print()
-
- """
-
- return is_title, is_author_or_org_list
-
- def _detect_block_title(self, input_block):
- """
- Use the functions 'is_potential_title' to detect titles of each paragraph block.
- If a line is a title, then the value of key 'is_title' of the line will be set to True.
- """
-
- raw_lines = input_block["lines"]
-
- prev_line_is_title_flag = False
-
- for i, curr_line in enumerate(raw_lines):
- prev_line = raw_lines[i - 1] if i > 0 else None
- next_line = raw_lines[i + 1] if i < len(raw_lines) - 1 else None
-
- blk_avg_char_width = input_block["avg_char_width"]
- blk_avg_char_height = input_block["avg_char_height"]
- blk_media_font_size = input_block["median_font_size"]
-
- is_title, is_author_or_org_list = self._is_potential_title(
- curr_line,
- prev_line,
- prev_line_is_title_flag,
- next_line,
- blk_avg_char_width,
- blk_avg_char_height,
- blk_media_font_size,
- )
-
- if is_title:
- curr_line["is_title"] = is_title
- prev_line_is_title_flag = True
- else:
- curr_line["is_title"] = False
- prev_line_is_title_flag = False
-
- if is_author_or_org_list:
- curr_line["is_author_or_org_list"] = is_author_or_org_list
- else:
- curr_line["is_author_or_org_list"] = False
-
- return input_block
-
- def batch_process_blocks_detect_titles(self, pdf_dic):
- """
- This function batch process the blocks to detect titles.
-
- Parameters
- ----------
- pdf_dict : dict
- result dictionary
-
- Returns
- -------
- pdf_dict : dict
- result dictionary
- """
- num_titles = 0
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "para_blocks" in blocks.keys():
- para_blocks = blocks["para_blocks"]
-
- all_single_line_blocks = []
- for block in para_blocks:
- if len(block["lines"]) == 1:
- all_single_line_blocks.append(block)
-
- new_para_blocks = []
- if not len(all_single_line_blocks) == len(para_blocks): # Not all blocks are single line blocks.
- for para_block in para_blocks:
- new_block = self._detect_block_title(para_block)
- new_para_blocks.append(new_block)
- num_titles += sum([line.get("is_title", 0) for line in new_block["lines"]])
- else: # All blocks are single line blocks.
- for para_block in para_blocks:
- new_para_blocks.append(para_block)
- num_titles += sum([line.get("is_title", 0) for line in para_block["lines"]])
- para_blocks = new_para_blocks
-
- blocks["para_blocks"] = para_blocks
-
- for para_block in para_blocks:
- all_titles = all(safe_get(line, "is_title", False) for line in para_block["lines"])
- para_text_len = sum([len(line["text"]) for line in para_block["lines"]])
- if (
- all_titles and para_text_len < 200
- ): # total length of the paragraph is less than 200, more than this should not be a title
- para_block["is_block_title"] = 1
- else:
- para_block["is_block_title"] = 0
-
- all_name_or_org_list_to_be_removed = all(
- safe_get(line, "is_author_or_org_list", False) for line in para_block["lines"]
- )
- if all_name_or_org_list_to_be_removed and page_id == "page_0":
- para_block["is_block_an_author_or_org_list"] = 1
- else:
- para_block["is_block_an_author_or_org_list"] = 0
-
- pdf_dic["statistics"]["num_titles"] = num_titles
-
- return pdf_dic
-
- def __determine_size_based_level(self, title_blocks):
- """
- This function determines the title level based on the font size of the title.
-
- Parameters
- ----------
- title_blocks : list
-
- Returns
- -------
- title_blocks : list
- """
-
- font_sizes = np.array([safe_get(tb["block"], "block_font_size", 0) for tb in title_blocks])
-
- # Use the mean and std of font sizes to remove extreme values
- mean_font_size = np.mean(font_sizes)
- std_font_size = np.std(font_sizes)
- min_extreme_font_size = mean_font_size - std_font_size # type: ignore
- max_extreme_font_size = mean_font_size + std_font_size # type: ignore
-
- # Compute the threshold for title level
- middle_font_sizes = font_sizes[(font_sizes > min_extreme_font_size) & (font_sizes < max_extreme_font_size)]
- if middle_font_sizes.size > 0:
- middle_mean_font_size = np.mean(middle_font_sizes)
- level_threshold = middle_mean_font_size
- else:
- level_threshold = mean_font_size
-
- for tb in title_blocks:
- title_block = tb["block"]
- title_font_size = safe_get(title_block, "block_font_size", 0)
-
- current_level = 1 # Initialize title level, the biggest level is 1
-
- # print(f"Before adjustment by font size, {current_level}")
- if title_font_size >= max_extreme_font_size:
- current_level = 1
- elif title_font_size <= min_extreme_font_size:
- current_level = 3
- elif float(title_font_size) >= float(level_threshold):
- current_level = 2
- else:
- current_level = 3
- # print(f"After adjustment by font size, {current_level}")
-
- title_block["block_title_level"] = current_level
-
- return title_blocks
-
- def batch_process_blocks_recog_title_level(self, pdf_dic):
- title_blocks = []
-
- # Collect all titles
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = blocks.get("para_blocks", [])
- for block in para_blocks:
- if block.get("is_block_title"):
- title_obj = {"page_id": page_id, "block": block}
- title_blocks.append(title_obj)
-
- # Determine title level
- if title_blocks:
- # Determine title level based on font size
- title_blocks = self.__determine_size_based_level(title_blocks)
-
- return pdf_dic
diff --git a/magic_pdf/pdf_parse_by_ocr.py b/magic_pdf/pdf_parse_by_ocr.py
deleted file mode 100644
index ca2f394b1..000000000
--- a/magic_pdf/pdf_parse_by_ocr.py
+++ /dev/null
@@ -1,21 +0,0 @@
-from magic_pdf.config.enums import SupportedPdfParseMethod
-from magic_pdf.data.dataset import PymuDocDataset
-from magic_pdf.pdf_parse_union_core_v2 import pdf_parse_union
-
-
-def parse_pdf_by_ocr(pdf_bytes,
- model_list,
- imageWriter,
- start_page_id=0,
- end_page_id=None,
- debug_mode=False,
- ):
- dataset = PymuDocDataset(pdf_bytes)
- return pdf_parse_union(dataset,
- model_list,
- imageWriter,
- SupportedPdfParseMethod.OCR,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- debug_mode=debug_mode,
- )
diff --git a/magic_pdf/pdf_parse_by_txt.py b/magic_pdf/pdf_parse_by_txt.py
deleted file mode 100644
index bae800f48..000000000
--- a/magic_pdf/pdf_parse_by_txt.py
+++ /dev/null
@@ -1,22 +0,0 @@
-from magic_pdf.config.enums import SupportedPdfParseMethod
-from magic_pdf.data.dataset import PymuDocDataset
-from magic_pdf.pdf_parse_union_core_v2 import pdf_parse_union
-
-
-def parse_pdf_by_txt(
- pdf_bytes,
- model_list,
- imageWriter,
- start_page_id=0,
- end_page_id=None,
- debug_mode=False,
-):
- dataset = PymuDocDataset(pdf_bytes)
- return pdf_parse_union(dataset,
- model_list,
- imageWriter,
- SupportedPdfParseMethod.TXT,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- debug_mode=debug_mode,
- )
diff --git a/magic_pdf/pdf_parse_union_core.py b/magic_pdf/pdf_parse_union_core.py
deleted file mode 100644
index c764f0cef..000000000
--- a/magic_pdf/pdf_parse_union_core.py
+++ /dev/null
@@ -1,271 +0,0 @@
-import time
-
-from loguru import logger
-
-from magic_pdf.libs.commons import fitz, get_delta_time
-from magic_pdf.layout.layout_sort import get_bboxes_layout, LAYOUT_UNPROC, get_columns_cnt_of_layout
-from magic_pdf.libs.convert_utils import dict_to_list
-from magic_pdf.libs.drop_reason import DropReason
-from magic_pdf.libs.hash_utils import compute_md5
-from magic_pdf.libs.local_math import float_equal
-from magic_pdf.libs.ocr_content_type import ContentType
-from magic_pdf.model.magic_model import MagicModel
-from magic_pdf.para.para_split_v2 import para_split
-from magic_pdf.pre_proc.citationmarker_remove import remove_citation_marker
-from magic_pdf.pre_proc.construct_page_dict import ocr_construct_page_component_v2
-from magic_pdf.pre_proc.cut_image import ocr_cut_image_and_table
-from magic_pdf.pre_proc.equations_replace import remove_chars_in_text_blocks, replace_equations_in_textblock, \
- combine_chars_to_pymudict
-from magic_pdf.pre_proc.ocr_detect_all_bboxes import ocr_prepare_bboxes_for_layout_split
-from magic_pdf.pre_proc.ocr_dict_merge import sort_blocks_by_layout, fill_spans_in_blocks, fix_block_spans, \
- fix_discarded_block
-from magic_pdf.pre_proc.ocr_span_list_modify import remove_overlaps_min_spans, get_qa_need_list_v2, \
- remove_overlaps_low_confidence_spans
-from magic_pdf.pre_proc.resolve_bbox_conflict import check_useful_block_horizontal_overlap
-
-
-def remove_horizontal_overlap_block_which_smaller(all_bboxes):
- useful_blocks = []
- for bbox in all_bboxes:
- useful_blocks.append({
- "bbox": bbox[:4]
- })
- is_useful_block_horz_overlap, smaller_bbox, bigger_bbox = check_useful_block_horizontal_overlap(useful_blocks)
- if is_useful_block_horz_overlap:
- logger.warning(
- f"skip this page, reason: {DropReason.USEFUL_BLOCK_HOR_OVERLAP}, smaller bbox is {smaller_bbox}, bigger bbox is {bigger_bbox}")
- for bbox in all_bboxes.copy():
- if smaller_bbox == bbox[:4]:
- all_bboxes.remove(bbox)
-
- return is_useful_block_horz_overlap, all_bboxes
-
-
-def __replace_STX_ETX(text_str:str):
- """ Replace \u0002 and \u0003, as these characters become garbled when extracted using pymupdf. In fact, they were originally quotation marks.
-Drawback: This issue is only observed in English text; it has not been found in Chinese text so far.
-
- Args:
- text_str (str): raw text
-
- Returns:
- _type_: replaced text
- """
- if text_str:
- s = text_str.replace('\u0002', "'")
- s = s.replace("\u0003", "'")
- return s
- return text_str
-
-
-def txt_spans_extract(pdf_page, inline_equations, interline_equations):
- text_raw_blocks = pdf_page.get_text("dict", flags=fitz.TEXTFLAGS_TEXT)["blocks"]
- char_level_text_blocks = pdf_page.get_text("rawdict", flags=fitz.TEXTFLAGS_TEXT)[
- "blocks"
- ]
- text_blocks = combine_chars_to_pymudict(text_raw_blocks, char_level_text_blocks)
- text_blocks = replace_equations_in_textblock(
- text_blocks, inline_equations, interline_equations
- )
- text_blocks = remove_citation_marker(text_blocks)
- text_blocks = remove_chars_in_text_blocks(text_blocks)
- spans = []
- for v in text_blocks:
- for line in v["lines"]:
- for span in line["spans"]:
- bbox = span["bbox"]
- if float_equal(bbox[0], bbox[2]) or float_equal(bbox[1], bbox[3]):
- continue
- if span.get('type') not in (ContentType.InlineEquation, ContentType.InterlineEquation):
- spans.append(
- {
- "bbox": list(span["bbox"]),
- "content": __replace_STX_ETX(span["text"]),
- "type": ContentType.Text,
- "score": 1.0,
- }
- )
- return spans
-
-
-def replace_text_span(pymu_spans, ocr_spans):
- return list(filter(lambda x: x["type"] != ContentType.Text, ocr_spans)) + pymu_spans
-
-
-def parse_page_core(pdf_docs, magic_model, page_id, pdf_bytes_md5, imageWriter, parse_mode):
- need_drop = False
- drop_reason = []
-
- '''从magic_model对象中获取后面会用到的区块信息'''
- img_blocks = magic_model.get_imgs(page_id)
- table_blocks = magic_model.get_tables(page_id)
- discarded_blocks = magic_model.get_discarded(page_id)
- text_blocks = magic_model.get_text_blocks(page_id)
- title_blocks = magic_model.get_title_blocks(page_id)
- inline_equations, interline_equations, interline_equation_blocks = magic_model.get_equations(page_id)
-
- page_w, page_h = magic_model.get_page_size(page_id)
-
- spans = magic_model.get_all_spans(page_id)
-
- '''根据parse_mode，构造spans'''
- if parse_mode == "txt":
- """ocr 中文本类的 span 用 pymu spans 替换！"""
- pymu_spans = txt_spans_extract(
- pdf_docs[page_id], inline_equations, interline_equations
- )
- spans = replace_text_span(pymu_spans, spans)
- elif parse_mode == "ocr":
- pass
- else:
- raise Exception("parse_mode must be txt or ocr")
-
- '''删除重叠spans中置信度较低的那些'''
- spans, dropped_spans_by_confidence = remove_overlaps_low_confidence_spans(spans)
- '''删除重叠spans中较小的那些'''
- spans, dropped_spans_by_span_overlap = remove_overlaps_min_spans(spans)
- '''对image和table截图'''
- spans = ocr_cut_image_and_table(spans, pdf_docs[page_id], page_id, pdf_bytes_md5, imageWriter)
-
- '''将所有区块的bbox整理到一起'''
- # interline_equation_blocks参数不够准，后面切换到interline_equations上
- interline_equation_blocks = []
- if len(interline_equation_blocks) > 0:
- all_bboxes, all_discarded_blocks, drop_reasons = ocr_prepare_bboxes_for_layout_split(
- img_blocks, table_blocks, discarded_blocks, text_blocks, title_blocks,
- interline_equation_blocks, page_w, page_h)
- else:
- all_bboxes, all_discarded_blocks, drop_reasons = ocr_prepare_bboxes_for_layout_split(
- img_blocks, table_blocks, discarded_blocks, text_blocks, title_blocks,
- interline_equations, page_w, page_h)
-
- if len(drop_reasons) > 0:
- need_drop = True
- drop_reason.append(DropReason.OVERLAP_BLOCKS_CAN_NOT_SEPARATION)
-
- '''先处理不需要排版的discarded_blocks'''
- discarded_block_with_spans, spans = fill_spans_in_blocks(all_discarded_blocks, spans, 0.4)
- fix_discarded_blocks = fix_discarded_block(discarded_block_with_spans)
-
- '''如果当前页面没有bbox则跳过'''
- if len(all_bboxes) == 0:
- logger.warning(f"skip this page, not found useful bbox, page_id: {page_id}")
- return ocr_construct_page_component_v2([], [], page_id, page_w, page_h, [],
- [], [], interline_equations, fix_discarded_blocks,
- need_drop, drop_reason)
-
- """在切分之前，先检查一下bbox是否有左右重叠的情况，如果有，那么就认为这个pdf暂时没有能力处理好，这种左右重叠的情况大概率是由于pdf里的行间公式、表格没有被正确识别出来造成的 """
-
- while True: # 循环检查左右重叠的情况，如果存在就删除掉较小的那个bbox，直到不存在左右重叠的情况
- is_useful_block_horz_overlap, all_bboxes = remove_horizontal_overlap_block_which_smaller(all_bboxes)
- if is_useful_block_horz_overlap:
- need_drop = True
- drop_reason.append(DropReason.USEFUL_BLOCK_HOR_OVERLAP)
- else:
- break
-
- '''根据区块信息计算layout'''
- page_boundry = [0, 0, page_w, page_h]
- layout_bboxes, layout_tree = get_bboxes_layout(all_bboxes, page_boundry, page_id)
-
- if len(text_blocks) > 0 and len(all_bboxes) > 0 and len(layout_bboxes) == 0:
- logger.warning(
- f"skip this page, page_id: {page_id}, reason: {DropReason.CAN_NOT_DETECT_PAGE_LAYOUT}")
- need_drop = True
- drop_reason.append(DropReason.CAN_NOT_DETECT_PAGE_LAYOUT)
-
- """以下去掉复杂的布局和超过2列的布局"""
- if any([lay["layout_label"] == LAYOUT_UNPROC for lay in layout_bboxes]): # 复杂的布局
- logger.warning(
- f"skip this page, page_id: {page_id}, reason: {DropReason.COMPLICATED_LAYOUT}")
- need_drop = True
- drop_reason.append(DropReason.COMPLICATED_LAYOUT)
-
- layout_column_width = get_columns_cnt_of_layout(layout_tree)
- if layout_column_width > 2: # 去掉超过2列的布局pdf
- logger.warning(
- f"skip this page, page_id: {page_id}, reason: {DropReason.TOO_MANY_LAYOUT_COLUMNS}")
- need_drop = True
- drop_reason.append(DropReason.TOO_MANY_LAYOUT_COLUMNS)
-
- '''根据layout顺序，对当前页面所有需要留下的block进行排序'''
- sorted_blocks = sort_blocks_by_layout(all_bboxes, layout_bboxes)
-
- '''将span填入排好序的blocks中'''
- block_with_spans, spans = fill_spans_in_blocks(sorted_blocks, spans, 0.3)
-
- '''对block进行fix操作'''
- fix_blocks = fix_block_spans(block_with_spans, img_blocks, table_blocks)
-
- '''获取QA需要外置的list'''
- images, tables, interline_equations = get_qa_need_list_v2(fix_blocks)
-
- '''构造pdf_info_dict'''
- page_info = ocr_construct_page_component_v2(fix_blocks, layout_bboxes, page_id, page_w, page_h, layout_tree,
- images, tables, interline_equations, fix_discarded_blocks,
- need_drop, drop_reason)
- return page_info
-
-
-def pdf_parse_union(pdf_bytes,
- model_list,
- imageWriter,
- parse_mode,
- start_page_id=0,
- end_page_id=None,
- debug_mode=False,
- ):
- pdf_bytes_md5 = compute_md5(pdf_bytes)
- pdf_docs = fitz.open("pdf", pdf_bytes)
-
- '''初始化空的pdf_info_dict'''
- pdf_info_dict = {}
-
- '''用model_list和docs对象初始化magic_model'''
- magic_model = MagicModel(model_list, pdf_docs)
-
- '''根据输入的起始范围解析pdf'''
- # end_page_id = end_page_id if end_page_id else len(pdf_docs) - 1
- end_page_id = end_page_id if end_page_id is not None and end_page_id >= 0 else len(pdf_docs) - 1
-
- if end_page_id > len(pdf_docs) - 1:
- logger.warning("end_page_id is out of range, use pdf_docs length")
- end_page_id = len(pdf_docs) - 1
-
- '''初始化启动时间'''
- start_time = time.time()
-
- for page_id, page in enumerate(pdf_docs):
- '''debug时输出每页解析的耗时'''
- if debug_mode:
- time_now = time.time()
- logger.info(
- f"page_id: {page_id}, last_page_cost_time: {get_delta_time(start_time)}"
- )
- start_time = time_now
-
- '''解析pdf中的每一页'''
- if start_page_id <= page_id <= end_page_id:
- page_info = parse_page_core(pdf_docs, magic_model, page_id, pdf_bytes_md5, imageWriter, parse_mode)
- else:
- page_w = page.rect.width
- page_h = page.rect.height
- page_info = ocr_construct_page_component_v2([], [], page_id, page_w, page_h, [],
- [], [], [], [],
- True, "skip page")
- pdf_info_dict[f"page_{page_id}"] = page_info
-
- """分段"""
- para_split(pdf_info_dict, debug_mode=debug_mode)
-
- """dict转list"""
- pdf_info_list = dict_to_list(pdf_info_dict)
- new_pdf_info_dict = {
- "pdf_info": pdf_info_list,
- }
-
- return new_pdf_info_dict
-
-
-if __name__ == '__main__':
- pass
diff --git a/magic_pdf/pdf_parse_union_core_v2.py b/magic_pdf/pdf_parse_union_core_v2.py
deleted file mode 100644
index 4c3a6d006..000000000
--- a/magic_pdf/pdf_parse_union_core_v2.py
+++ /dev/null
@@ -1,597 +0,0 @@
-import copy
-import os
-import statistics
-import time
-from typing import List
-
-import torch
-from loguru import logger
-
-from magic_pdf.config.enums import SupportedPdfParseMethod
-from magic_pdf.data.dataset import Dataset, PageableData
-from magic_pdf.libs.clean_memory import clean_memory
-from magic_pdf.libs.commons import fitz, get_delta_time
-from magic_pdf.libs.config_reader import get_local_layoutreader_model_dir
-from magic_pdf.libs.convert_utils import dict_to_list
-from magic_pdf.libs.drop_reason import DropReason
-from magic_pdf.libs.hash_utils import compute_md5
-from magic_pdf.libs.local_math import float_equal
-from magic_pdf.libs.ocr_content_type import ContentType, BlockType
-from magic_pdf.model.magic_model import MagicModel
-from magic_pdf.para.para_split_v3 import para_split
-from magic_pdf.pre_proc.citationmarker_remove import remove_citation_marker
-from magic_pdf.pre_proc.construct_page_dict import \
- ocr_construct_page_component_v2
-from magic_pdf.pre_proc.cut_image import ocr_cut_image_and_table
-from magic_pdf.pre_proc.equations_replace import (
- combine_chars_to_pymudict, remove_chars_in_text_blocks,
- replace_equations_in_textblock)
-from magic_pdf.pre_proc.ocr_detect_all_bboxes import \
- ocr_prepare_bboxes_for_layout_split_v2
-from magic_pdf.pre_proc.ocr_dict_merge import (fill_spans_in_blocks,
- fix_block_spans,
- fix_discarded_block, fix_block_spans_v2)
-from magic_pdf.pre_proc.ocr_span_list_modify import (
- get_qa_need_list_v2, remove_overlaps_low_confidence_spans,
- remove_overlaps_min_spans)
-from magic_pdf.pre_proc.resolve_bbox_conflict import \
- check_useful_block_horizontal_overlap
-
-
-def remove_horizontal_overlap_block_which_smaller(all_bboxes):
- useful_blocks = []
- for bbox in all_bboxes:
- useful_blocks.append({'bbox': bbox[:4]})
- is_useful_block_horz_overlap, smaller_bbox, bigger_bbox = (
- check_useful_block_horizontal_overlap(useful_blocks)
- )
- if is_useful_block_horz_overlap:
- logger.warning(
- f'skip this page, reason: {DropReason.USEFUL_BLOCK_HOR_OVERLAP}, smaller bbox is {smaller_bbox}, bigger bbox is {bigger_bbox}'
- ) # noqa: E501
- for bbox in all_bboxes.copy():
- if smaller_bbox == bbox[:4]:
- all_bboxes.remove(bbox)
-
- return is_useful_block_horz_overlap, all_bboxes
-
-
-def __replace_STX_ETX(text_str: str):
- """Replace \u0002 and \u0003, as these characters become garbled when extracted using pymupdf. In fact, they were originally quotation marks.
- Drawback: This issue is only observed in English text; it has not been found in Chinese text so far.
-
- Args:
- text_str (str): raw text
-
- Returns:
- _type_: replaced text
- """ # noqa: E501
- if text_str:
- s = text_str.replace('\u0002', "'")
- s = s.replace('\u0003', "'")
- return s
- return text_str
-
-
-def txt_spans_extract(pdf_page, inline_equations, interline_equations):
- text_raw_blocks = pdf_page.get_text('dict', flags=fitz.TEXTFLAGS_TEXT)['blocks']
- char_level_text_blocks = pdf_page.get_text('rawdict', flags=fitz.TEXTFLAGS_TEXT)[
- 'blocks'
- ]
- text_blocks = combine_chars_to_pymudict(text_raw_blocks, char_level_text_blocks)
- text_blocks = replace_equations_in_textblock(
- text_blocks, inline_equations, interline_equations
- )
- text_blocks = remove_citation_marker(text_blocks)
- text_blocks = remove_chars_in_text_blocks(text_blocks)
- spans = []
- for v in text_blocks:
- for line in v['lines']:
- for span in line['spans']:
- bbox = span['bbox']
- if float_equal(bbox[0], bbox[2]) or float_equal(bbox[1], bbox[3]):
- continue
- if span.get('type') not in (
- ContentType.InlineEquation,
- ContentType.InterlineEquation,
- ):
- spans.append(
- {
- 'bbox': list(span['bbox']),
- 'content': __replace_STX_ETX(span['text']),
- 'type': ContentType.Text,
- 'score': 1.0,
- }
- )
- return spans
-
-
-def replace_text_span(pymu_spans, ocr_spans):
- return list(filter(lambda x: x['type'] != ContentType.Text, ocr_spans)) + pymu_spans
-
-
-def model_init(model_name: str):
- from transformers import LayoutLMv3ForTokenClassification
-
- if torch.cuda.is_available():
- device = torch.device('cuda')
- if torch.cuda.is_bf16_supported():
- supports_bfloat16 = True
- else:
- supports_bfloat16 = False
- else:
- device = torch.device('cpu')
- supports_bfloat16 = False
-
- if model_name == 'layoutreader':
- # 检测modelscope的缓存目录是否存在
- layoutreader_model_dir = get_local_layoutreader_model_dir()
- if os.path.exists(layoutreader_model_dir):
- model = LayoutLMv3ForTokenClassification.from_pretrained(
- layoutreader_model_dir
- )
- else:
- logger.warning(
- 'local layoutreader model not exists, use online model from huggingface'
- )
- model = LayoutLMv3ForTokenClassification.from_pretrained(
- 'hantian/layoutreader'
- )
- # 检查设备是否支持 bfloat16
- if supports_bfloat16:
- model.bfloat16()
- model.to(device).eval()
- else:
- logger.error('model name not allow')
- exit(1)
- return model
-
-
-class ModelSingleton:
- _instance = None
- _models = {}
-
- def __new__(cls, *args, **kwargs):
- if cls._instance is None:
- cls._instance = super().__new__(cls)
- return cls._instance
-
- def get_model(self, model_name: str):
- if model_name not in self._models:
- self._models[model_name] = model_init(model_name=model_name)
- return self._models[model_name]
-
-
-def do_predict(boxes: List[List[int]], model) -> List[int]:
- from magic_pdf.model.v3.helpers import (boxes2inputs, parse_logits,
- prepare_inputs)
-
- inputs = boxes2inputs(boxes)
- inputs = prepare_inputs(inputs, model)
- logits = model(**inputs).logits.cpu().squeeze(0)
- return parse_logits(logits, len(boxes))
-
-
-def cal_block_index(fix_blocks, sorted_bboxes):
- for block in fix_blocks:
-
- line_index_list = []
- if len(block['lines']) == 0:
- block['index'] = sorted_bboxes.index(block['bbox'])
- else:
- for line in block['lines']:
- line['index'] = sorted_bboxes.index(line['bbox'])
- line_index_list.append(line['index'])
- median_value = statistics.median(line_index_list)
- block['index'] = median_value
-
- # 删除图表body block中的虚拟line信息, 并用real_lines信息回填
- if block['type'] in [BlockType.ImageBody, BlockType.TableBody]:
- block['virtual_lines'] = copy.deepcopy(block['lines'])
- block['lines'] = copy.deepcopy(block['real_lines'])
- del block['real_lines']
-
- return fix_blocks
-
-
-def insert_lines_into_block(block_bbox, line_height, page_w, page_h):
- # block_bbox是一个元组(x0, y0, x1, y1)，其中(x0, y0)是左下角坐标，(x1, y1)是右上角坐标
- x0, y0, x1, y1 = block_bbox
-
- block_height = y1 - y0
- block_weight = x1 - x0
-
- # 如果block高度小于n行正文，则直接返回block的bbox
- if line_height * 3 < block_height:
- if (
- block_height > page_h * 0.25 and page_w * 0.5 > block_weight > page_w * 0.25
- ): # 可能是双列结构，可以切细点
- lines = int(block_height / line_height) + 1
- else:
- # 如果block的宽度超过0.4页面宽度，则将block分成3行(是一种复杂布局，图不能切的太细)
- if block_weight > page_w * 0.4:
- line_height = (y1 - y0) / 3
- lines = 3
- elif block_weight > page_w * 0.25: # （可能是三列结构，也切细点）
- lines = int(block_height / line_height) + 1
- else: # 判断长宽比
- if block_height / block_weight > 1.2: # 细长的不分
- return [[x0, y0, x1, y1]]
- else: # 不细长的还是分成两行
- line_height = (y1 - y0) / 2
- lines = 2
-
- # 确定从哪个y位置开始绘制线条
- current_y = y0
-
- # 用于存储线条的位置信息[(x0, y), ...]
- lines_positions = []
-
- for i in range(lines):
- lines_positions.append([x0, current_y, x1, current_y + line_height])
- current_y += line_height
- return lines_positions
-
- else:
- return [[x0, y0, x1, y1]]
-
-
-def sort_lines_by_model(fix_blocks, page_w, page_h, line_height):
- page_line_list = []
- for block in fix_blocks:
- if block['type'] in [
- BlockType.Text, BlockType.Title, BlockType.InterlineEquation,
- BlockType.ImageCaption, BlockType.ImageFootnote,
- BlockType.TableCaption, BlockType.TableFootnote
- ]:
- if len(block['lines']) == 0:
- bbox = block['bbox']
- lines = insert_lines_into_block(bbox, line_height, page_w, page_h)
- for line in lines:
- block['lines'].append({'bbox': line, 'spans': []})
- page_line_list.extend(lines)
- else:
- for line in block['lines']:
- bbox = line['bbox']
- page_line_list.append(bbox)
- elif block['type'] in [BlockType.ImageBody, BlockType.TableBody]:
- bbox = block['bbox']
- block["real_lines"] = copy.deepcopy(block['lines'])
- lines = insert_lines_into_block(bbox, line_height, page_w, page_h)
- block['lines'] = []
- for line in lines:
- block['lines'].append({'bbox': line, 'spans': []})
- page_line_list.extend(lines)
-
- # 使用layoutreader排序
- x_scale = 1000.0 / page_w
- y_scale = 1000.0 / page_h
- boxes = []
- # logger.info(f"Scale: {x_scale}, {y_scale}, Boxes len: {len(page_line_list)}")
- for left, top, right, bottom in page_line_list:
- if left < 0:
- logger.warning(
- f'left < 0, left: {left}, right: {right}, top: {top}, bottom: {bottom}, page_w: {page_w}, page_h: {page_h}'
- ) # noqa: E501
- left = 0
- if right > page_w:
- logger.warning(
- f'right > page_w, left: {left}, right: {right}, top: {top}, bottom: {bottom}, page_w: {page_w}, page_h: {page_h}'
- ) # noqa: E501
- right = page_w
- if top < 0:
- logger.warning(
- f'top < 0, left: {left}, right: {right}, top: {top}, bottom: {bottom}, page_w: {page_w}, page_h: {page_h}'
- ) # noqa: E501
- top = 0
- if bottom > page_h:
- logger.warning(
- f'bottom > page_h, left: {left}, right: {right}, top: {top}, bottom: {bottom}, page_w: {page_w}, page_h: {page_h}'
- ) # noqa: E501
- bottom = page_h
-
- left = round(left * x_scale)
- top = round(top * y_scale)
- right = round(right * x_scale)
- bottom = round(bottom * y_scale)
- assert (
- 1000 >= right >= left >= 0 and 1000 >= bottom >= top >= 0
- ), f'Invalid box. right: {right}, left: {left}, bottom: {bottom}, top: {top}' # noqa: E126, E121
- boxes.append([left, top, right, bottom])
- model_manager = ModelSingleton()
- model = model_manager.get_model('layoutreader')
- with torch.no_grad():
- orders = do_predict(boxes, model)
- sorted_bboxes = [page_line_list[i] for i in orders]
-
- return sorted_bboxes
-
-
-def get_line_height(blocks):
- page_line_height_list = []
- for block in blocks:
- if block['type'] in [
- BlockType.Text, BlockType.Title,
- BlockType.ImageCaption, BlockType.ImageFootnote,
- BlockType.TableCaption, BlockType.TableFootnote
- ]:
- for line in block['lines']:
- bbox = line['bbox']
- page_line_height_list.append(int(bbox[3] - bbox[1]))
- if len(page_line_height_list) > 0:
- return statistics.median(page_line_height_list)
- else:
- return 10
-
-
-def process_groups(groups, body_key, caption_key, footnote_key):
- body_blocks = []
- caption_blocks = []
- footnote_blocks = []
- for i, group in enumerate(groups):
- group[body_key]['group_id'] = i
- body_blocks.append(group[body_key])
- for caption_block in group[caption_key]:
- caption_block['group_id'] = i
- caption_blocks.append(caption_block)
- for footnote_block in group[footnote_key]:
- footnote_block['group_id'] = i
- footnote_blocks.append(footnote_block)
- return body_blocks, caption_blocks, footnote_blocks
-
-
-def process_block_list(blocks, body_type, block_type):
- indices = [block['index'] for block in blocks]
- median_index = statistics.median(indices)
-
- body_bbox = next((block['bbox'] for block in blocks if block.get('type') == body_type), [])
-
- return {
- 'type': block_type,
- 'bbox': body_bbox,
- 'blocks': blocks,
- 'index': median_index,
- }
-
-
-def revert_group_blocks(blocks):
- image_groups = {}
- table_groups = {}
- new_blocks = []
- for block in blocks:
- if block['type'] in [BlockType.ImageBody, BlockType.ImageCaption, BlockType.ImageFootnote]:
- group_id = block['group_id']
- if group_id not in image_groups:
- image_groups[group_id] = []
- image_groups[group_id].append(block)
- elif block['type'] in [BlockType.TableBody, BlockType.TableCaption, BlockType.TableFootnote]:
- group_id = block['group_id']
- if group_id not in table_groups:
- table_groups[group_id] = []
- table_groups[group_id].append(block)
- else:
- new_blocks.append(block)
-
- for group_id, blocks in image_groups.items():
- new_blocks.append(process_block_list(blocks, BlockType.ImageBody, BlockType.Image))
-
- for group_id, blocks in table_groups.items():
- new_blocks.append(process_block_list(blocks, BlockType.TableBody, BlockType.Table))
-
- return new_blocks
-
-
-def parse_page_core(
- page_doc: PageableData, magic_model, page_id, pdf_bytes_md5, imageWriter, parse_mode
-):
- need_drop = False
- drop_reason = []
-
- """从magic_model对象中获取后面会用到的区块信息"""
- # img_blocks = magic_model.get_imgs(page_id)
- # table_blocks = magic_model.get_tables(page_id)
-
- img_groups = magic_model.get_imgs_v2(page_id)
- table_groups = magic_model.get_tables_v2(page_id)
-
- img_body_blocks, img_caption_blocks, img_footnote_blocks = process_groups(
- img_groups, 'image_body', 'image_caption_list', 'image_footnote_list'
- )
-
- table_body_blocks, table_caption_blocks, table_footnote_blocks = process_groups(
- table_groups, 'table_body', 'table_caption_list', 'table_footnote_list'
- )
-
- discarded_blocks = magic_model.get_discarded(page_id)
- text_blocks = magic_model.get_text_blocks(page_id)
- title_blocks = magic_model.get_title_blocks(page_id)
- inline_equations, interline_equations, interline_equation_blocks = (
- magic_model.get_equations(page_id)
- )
-
- page_w, page_h = magic_model.get_page_size(page_id)
-
- spans = magic_model.get_all_spans(page_id)
-
- """根据parse_mode，构造spans"""
- if parse_mode == SupportedPdfParseMethod.TXT:
- """ocr 中文本类的 span 用 pymu spans 替换！"""
- pymu_spans = txt_spans_extract(page_doc, inline_equations, interline_equations)
- spans = replace_text_span(pymu_spans, spans)
- elif parse_mode == SupportedPdfParseMethod.OCR:
- pass
- else:
- raise Exception('parse_mode must be txt or ocr')
-
- """删除重叠spans中置信度较低的那些"""
- spans, dropped_spans_by_confidence = remove_overlaps_low_confidence_spans(spans)
- """删除重叠spans中较小的那些"""
- spans, dropped_spans_by_span_overlap = remove_overlaps_min_spans(spans)
- """对image和table截图"""
- spans = ocr_cut_image_and_table(
- spans, page_doc, page_id, pdf_bytes_md5, imageWriter
- )
-
- """将所有区块的bbox整理到一起"""
- # interline_equation_blocks参数不够准，后面切换到interline_equations上
- interline_equation_blocks = []
- if len(interline_equation_blocks) > 0:
- all_bboxes, all_discarded_blocks = ocr_prepare_bboxes_for_layout_split_v2(
- img_body_blocks, img_caption_blocks, img_footnote_blocks,
- table_body_blocks, table_caption_blocks, table_footnote_blocks,
- discarded_blocks,
- text_blocks,
- title_blocks,
- interline_equation_blocks,
- page_w,
- page_h,
- )
- else:
- all_bboxes, all_discarded_blocks = ocr_prepare_bboxes_for_layout_split_v2(
- img_body_blocks, img_caption_blocks, img_footnote_blocks,
- table_body_blocks, table_caption_blocks, table_footnote_blocks,
- discarded_blocks,
- text_blocks,
- title_blocks,
- interline_equations,
- page_w,
- page_h,
- )
-
- """先处理不需要排版的discarded_blocks"""
- discarded_block_with_spans, spans = fill_spans_in_blocks(
- all_discarded_blocks, spans, 0.4
- )
- fix_discarded_blocks = fix_discarded_block(discarded_block_with_spans)
-
- """如果当前页面没有bbox则跳过"""
- if len(all_bboxes) == 0:
- logger.warning(f'skip this page, not found useful bbox, page_id: {page_id}')
- return ocr_construct_page_component_v2(
- [],
- [],
- page_id,
- page_w,
- page_h,
- [],
- [],
- [],
- interline_equations,
- fix_discarded_blocks,
- need_drop,
- drop_reason,
- )
-
- """将span填入blocks中"""
- block_with_spans, spans = fill_spans_in_blocks(all_bboxes, spans, 0.5)
-
- """对block进行fix操作"""
- fix_blocks = fix_block_spans_v2(block_with_spans)
-
- """获取所有line并计算正文line的高度"""
- line_height = get_line_height(fix_blocks)
-
- """获取所有line并对line排序"""
- sorted_bboxes = sort_lines_by_model(fix_blocks, page_w, page_h, line_height)
-
- """根据line的中位数算block的序列关系"""
- fix_blocks = cal_block_index(fix_blocks, sorted_bboxes)
-
- """将image和table的block还原回group形式参与后续流程"""
- fix_blocks = revert_group_blocks(fix_blocks)
-
- """重排block"""
- sorted_blocks = sorted(fix_blocks, key=lambda b: b['index'])
-
- """获取QA需要外置的list"""
- images, tables, interline_equations = get_qa_need_list_v2(sorted_blocks)
-
- """构造pdf_info_dict"""
- page_info = ocr_construct_page_component_v2(
- sorted_blocks,
- [],
- page_id,
- page_w,
- page_h,
- [],
- images,
- tables,
- interline_equations,
- fix_discarded_blocks,
- need_drop,
- drop_reason,
- )
- return page_info
-
-
-def pdf_parse_union(
- dataset: Dataset,
- model_list,
- imageWriter,
- parse_mode,
- start_page_id=0,
- end_page_id=None,
- debug_mode=False,
-):
- pdf_bytes_md5 = compute_md5(dataset.data_bits())
-
- """初始化空的pdf_info_dict"""
- pdf_info_dict = {}
-
- """用model_list和docs对象初始化magic_model"""
- magic_model = MagicModel(model_list, dataset)
-
- """根据输入的起始范围解析pdf"""
- # end_page_id = end_page_id if end_page_id else len(pdf_docs) - 1
- end_page_id = (
- end_page_id
- if end_page_id is not None and end_page_id >= 0
- else len(dataset) - 1
- )
-
- if end_page_id > len(dataset) - 1:
- logger.warning('end_page_id is out of range, use pdf_docs length')
- end_page_id = len(dataset) - 1
-
- """初始化启动时间"""
- start_time = time.time()
-
- for page_id, page in enumerate(dataset):
- """debug时输出每页解析的耗时."""
- if debug_mode:
- time_now = time.time()
- logger.info(
- f'page_id: {page_id}, last_page_cost_time: {get_delta_time(start_time)}'
- )
- start_time = time_now
-
- """解析pdf中的每一页"""
- if start_page_id <= page_id <= end_page_id:
- page_info = parse_page_core(
- page, magic_model, page_id, pdf_bytes_md5, imageWriter, parse_mode
- )
- else:
- page_info = page.get_page_info()
- page_w = page_info.w
- page_h = page_info.h
- page_info = ocr_construct_page_component_v2(
- [], [], page_id, page_w, page_h, [], [], [], [], [], True, 'skip page'
- )
- pdf_info_dict[f'page_{page_id}'] = page_info
-
- """分段"""
- para_split(pdf_info_dict, debug_mode=debug_mode)
-
- """dict转list"""
- pdf_info_list = dict_to_list(pdf_info_dict)
- new_pdf_info_dict = {
- 'pdf_info': pdf_info_list,
- }
-
- clean_memory()
-
- return new_pdf_info_dict
-
-
-if __name__ == '__main__':
- pass
diff --git a/magic_pdf/pipe/AbsPipe.py b/magic_pdf/pipe/AbsPipe.py
deleted file mode 100644
index 198413745..000000000
--- a/magic_pdf/pipe/AbsPipe.py
+++ /dev/null
@@ -1,114 +0,0 @@
-from abc import ABC, abstractmethod
-
-from magic_pdf.dict2md.ocr_mkcontent import union_make
-from magic_pdf.filter.pdf_classify_by_type import classify
-from magic_pdf.filter.pdf_meta_scan import pdf_meta_scan
-from magic_pdf.libs.MakeContentConfig import MakeMode, DropMode
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.libs.drop_reason import DropReason
-from magic_pdf.libs.json_compressor import JsonCompressor
-
-
-class AbsPipe(ABC):
- """
- txt和ocr处理的抽象类
- """
- PIP_OCR = "ocr"
- PIP_TXT = "txt"
-
- def __init__(self, pdf_bytes: bytes, model_list: list, image_writer: AbsReaderWriter, is_debug: bool = False,
- start_page_id=0, end_page_id=None, lang=None, layout_model=None, formula_enable=None, table_enable=None):
- self.pdf_bytes = pdf_bytes
- self.model_list = model_list
- self.image_writer = image_writer
- self.pdf_mid_data = None # 未压缩
- self.is_debug = is_debug
- self.start_page_id = start_page_id
- self.end_page_id = end_page_id
- self.lang = lang
- self.layout_model = layout_model
- self.formula_enable = formula_enable
- self.table_enable = table_enable
-
- def get_compress_pdf_mid_data(self):
- return JsonCompressor.compress_json(self.pdf_mid_data)
-
- @abstractmethod
- def pipe_classify(self):
- """
- 有状态的分类
- """
- raise NotImplementedError
-
- @abstractmethod
- def pipe_analyze(self):
- """
- 有状态的跑模型分析
- """
- raise NotImplementedError
-
- @abstractmethod
- def pipe_parse(self):
- """
- 有状态的解析
- """
- raise NotImplementedError
-
- def pipe_mk_uni_format(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF):
- content_list = AbsPipe.mk_uni_format(self.get_compress_pdf_mid_data(), img_parent_path, drop_mode)
- return content_list
-
- def pipe_mk_markdown(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF, md_make_mode=MakeMode.MM_MD):
- md_content = AbsPipe.mk_markdown(self.get_compress_pdf_mid_data(), img_parent_path, drop_mode, md_make_mode)
- return md_content
-
- @staticmethod
- def classify(pdf_bytes: bytes) -> str:
- """
- 根据pdf的元数据，判断是文本pdf，还是ocr pdf
- """
- pdf_meta = pdf_meta_scan(pdf_bytes)
- if pdf_meta.get("_need_drop", False): # 如果返回了需要丢弃的标志，则抛出异常
- raise Exception(f"pdf meta_scan need_drop,reason is {pdf_meta['_drop_reason']}")
- else:
- is_encrypted = pdf_meta["is_encrypted"]
- is_needs_password = pdf_meta["is_needs_password"]
- if is_encrypted or is_needs_password: # 加密的，需要密码的，没有页面的，都不处理
- raise Exception(f"pdf meta_scan need_drop,reason is {DropReason.ENCRYPTED}")
- else:
- is_text_pdf, results = classify(
- pdf_meta["total_page"],
- pdf_meta["page_width_pts"],
- pdf_meta["page_height_pts"],
- pdf_meta["image_info_per_page"],
- pdf_meta["text_len_per_page"],
- pdf_meta["imgs_per_page"],
- pdf_meta["text_layout_per_page"],
- pdf_meta["invalid_chars"],
- )
- if is_text_pdf:
- return AbsPipe.PIP_TXT
- else:
- return AbsPipe.PIP_OCR
-
- @staticmethod
- def mk_uni_format(compressed_pdf_mid_data: str, img_buket_path: str, drop_mode=DropMode.WHOLE_PDF) -> list:
- """
- 根据pdf类型，生成统一格式content_list
- """
- pdf_mid_data = JsonCompressor.decompress_json(compressed_pdf_mid_data)
- pdf_info_list = pdf_mid_data["pdf_info"]
- content_list = union_make(pdf_info_list, MakeMode.STANDARD_FORMAT, drop_mode, img_buket_path)
- return content_list
-
- @staticmethod
- def mk_markdown(compressed_pdf_mid_data: str, img_buket_path: str, drop_mode=DropMode.WHOLE_PDF, md_make_mode=MakeMode.MM_MD) -> list:
- """
- 根据pdf类型，markdown
- """
- pdf_mid_data = JsonCompressor.decompress_json(compressed_pdf_mid_data)
- pdf_info_list = pdf_mid_data["pdf_info"]
- md_content = union_make(pdf_info_list, md_make_mode, drop_mode, img_buket_path)
- return md_content
-
-
diff --git a/magic_pdf/pipe/OCRPipe.py b/magic_pdf/pipe/OCRPipe.py
deleted file mode 100644
index 71002a936..000000000
--- a/magic_pdf/pipe/OCRPipe.py
+++ /dev/null
@@ -1,41 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.MakeContentConfig import DropMode, MakeMode
-from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.pipe.AbsPipe import AbsPipe
-from magic_pdf.user_api import parse_ocr_pdf
-
-
-class OCRPipe(AbsPipe):
-
- def __init__(self, pdf_bytes: bytes, model_list: list, image_writer: AbsReaderWriter, is_debug: bool = False,
- start_page_id=0, end_page_id=None, lang=None,
- layout_model=None, formula_enable=None, table_enable=None):
- super().__init__(pdf_bytes, model_list, image_writer, is_debug, start_page_id, end_page_id, lang,
- layout_model, formula_enable, table_enable)
-
- def pipe_classify(self):
- pass
-
- def pipe_analyze(self):
- self.model_list = doc_analyze(self.pdf_bytes, ocr=True,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
-
- def pipe_parse(self):
- self.pdf_mid_data = parse_ocr_pdf(self.pdf_bytes, self.model_list, self.image_writer, is_debug=self.is_debug,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
-
- def pipe_mk_uni_format(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF):
- result = super().pipe_mk_uni_format(img_parent_path, drop_mode)
- logger.info("ocr_pipe mk content list finished")
- return result
-
- def pipe_mk_markdown(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF, md_make_mode=MakeMode.MM_MD):
- result = super().pipe_mk_markdown(img_parent_path, drop_mode, md_make_mode)
- logger.info(f"ocr_pipe mk {md_make_mode} finished")
- return result
diff --git a/magic_pdf/pipe/TXTPipe.py b/magic_pdf/pipe/TXTPipe.py
deleted file mode 100644
index f0bc9b7b3..000000000
--- a/magic_pdf/pipe/TXTPipe.py
+++ /dev/null
@@ -1,42 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.MakeContentConfig import DropMode, MakeMode
-from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.libs.json_compressor import JsonCompressor
-from magic_pdf.pipe.AbsPipe import AbsPipe
-from magic_pdf.user_api import parse_txt_pdf
-
-
-class TXTPipe(AbsPipe):
-
- def __init__(self, pdf_bytes: bytes, model_list: list, image_writer: AbsReaderWriter, is_debug: bool = False,
- start_page_id=0, end_page_id=None, lang=None,
- layout_model=None, formula_enable=None, table_enable=None):
- super().__init__(pdf_bytes, model_list, image_writer, is_debug, start_page_id, end_page_id, lang,
- layout_model, formula_enable, table_enable)
-
- def pipe_classify(self):
- pass
-
- def pipe_analyze(self):
- self.model_list = doc_analyze(self.pdf_bytes, ocr=False,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
-
- def pipe_parse(self):
- self.pdf_mid_data = parse_txt_pdf(self.pdf_bytes, self.model_list, self.image_writer, is_debug=self.is_debug,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
-
- def pipe_mk_uni_format(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF):
- result = super().pipe_mk_uni_format(img_parent_path, drop_mode)
- logger.info("txt_pipe mk content list finished")
- return result
-
- def pipe_mk_markdown(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF, md_make_mode=MakeMode.MM_MD):
- result = super().pipe_mk_markdown(img_parent_path, drop_mode, md_make_mode)
- logger.info(f"txt_pipe mk {md_make_mode} finished")
- return result
diff --git a/magic_pdf/pipe/UNIPipe.py b/magic_pdf/pipe/UNIPipe.py
deleted file mode 100644
index a1ae7f90c..000000000
--- a/magic_pdf/pipe/UNIPipe.py
+++ /dev/null
@@ -1,99 +0,0 @@
-import json
-
-from loguru import logger
-
-from magic_pdf.libs.MakeContentConfig import DropMode, MakeMode
-from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-from magic_pdf.libs.commons import join_path
-from magic_pdf.pipe.AbsPipe import AbsPipe
-from magic_pdf.user_api import parse_union_pdf, parse_ocr_pdf
-
-
-class UNIPipe(AbsPipe):
-
- def __init__(self, pdf_bytes: bytes, jso_useful_key: dict, image_writer: AbsReaderWriter, is_debug: bool = False,
- start_page_id=0, end_page_id=None, lang=None,
- layout_model=None, formula_enable=None, table_enable=None):
- self.pdf_type = jso_useful_key["_pdf_type"]
- super().__init__(pdf_bytes, jso_useful_key["model_list"], image_writer, is_debug, start_page_id, end_page_id,
- lang, layout_model, formula_enable, table_enable)
- if len(self.model_list) == 0:
- self.input_model_is_empty = True
- else:
- self.input_model_is_empty = False
-
- def pipe_classify(self):
- self.pdf_type = AbsPipe.classify(self.pdf_bytes)
-
- def pipe_analyze(self):
- if self.pdf_type == self.PIP_TXT:
- self.model_list = doc_analyze(self.pdf_bytes, ocr=False,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
- elif self.pdf_type == self.PIP_OCR:
- self.model_list = doc_analyze(self.pdf_bytes, ocr=True,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
-
- def pipe_parse(self):
- if self.pdf_type == self.PIP_TXT:
- self.pdf_mid_data = parse_union_pdf(self.pdf_bytes, self.model_list, self.image_writer,
- is_debug=self.is_debug, input_model_is_empty=self.input_model_is_empty,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang, layout_model=self.layout_model,
- formula_enable=self.formula_enable, table_enable=self.table_enable)
- elif self.pdf_type == self.PIP_OCR:
- self.pdf_mid_data = parse_ocr_pdf(self.pdf_bytes, self.model_list, self.image_writer,
- is_debug=self.is_debug,
- start_page_id=self.start_page_id, end_page_id=self.end_page_id,
- lang=self.lang)
-
- def pipe_mk_uni_format(self, img_parent_path: str, drop_mode=DropMode.NONE_WITH_REASON):
- result = super().pipe_mk_uni_format(img_parent_path, drop_mode)
- logger.info("uni_pipe mk content list finished")
- return result
-
- def pipe_mk_markdown(self, img_parent_path: str, drop_mode=DropMode.WHOLE_PDF, md_make_mode=MakeMode.MM_MD):
- result = super().pipe_mk_markdown(img_parent_path, drop_mode, md_make_mode)
- logger.info(f"uni_pipe mk {md_make_mode} finished")
- return result
-
-
-if __name__ == '__main__':
- # 测试
- drw = DiskReaderWriter(r"D:/project/20231108code-clean")
-
- pdf_file_path = r"linshixuqiu\19983-00.pdf"
- model_file_path = r"linshixuqiu\19983-00.json"
- pdf_bytes = drw.read(pdf_file_path, AbsReaderWriter.MODE_BIN)
- model_json_txt = drw.read(model_file_path, AbsReaderWriter.MODE_TXT)
- model_list = json.loads(model_json_txt)
- write_path = r"D:\project\20231108code-clean\linshixuqiu\19983-00"
- img_bucket_path = "imgs"
- img_writer = DiskReaderWriter(join_path(write_path, img_bucket_path))
-
- # pdf_type = UNIPipe.classify(pdf_bytes)
- # jso_useful_key = {
- # "_pdf_type": pdf_type,
- # "model_list": model_list
- # }
-
- jso_useful_key = {
- "_pdf_type": "",
- "model_list": model_list
- }
- pipe = UNIPipe(pdf_bytes, jso_useful_key, img_writer)
- pipe.pipe_classify()
- pipe.pipe_parse()
- md_content = pipe.pipe_mk_markdown(img_bucket_path)
- content_list = pipe.pipe_mk_uni_format(img_bucket_path)
-
- md_writer = DiskReaderWriter(write_path)
- md_writer.write(md_content, "19983-00.md", AbsReaderWriter.MODE_TXT)
- md_writer.write(json.dumps(pipe.pdf_mid_data, ensure_ascii=False, indent=4), "19983-00.json",
- AbsReaderWriter.MODE_TXT)
- md_writer.write(str(content_list), "19983-00.txt", AbsReaderWriter.MODE_TXT)
diff --git a/magic_pdf/pipe/__init__.py b/magic_pdf/pipe/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/post_proc/__init__.py b/magic_pdf/post_proc/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/post_proc/detect_para.py b/magic_pdf/post_proc/detect_para.py
deleted file mode 100644
index 17b41d27c..000000000
--- a/magic_pdf/post_proc/detect_para.py
+++ /dev/null
@@ -1,3472 +0,0 @@
-import os
-import sys
-import json
-import re
-import math
-import unicodedata
-from collections import Counter
-
-
-import numpy as np
-from termcolor import cprint
-
-
-from magic_pdf.libs.commons import fitz
-from magic_pdf.libs.nlp_utils import NLPModels
-
-
-if sys.version_info[0] >= 3:
- sys.stdout.reconfigure(encoding="utf-8") # type: ignore
-
-
-def open_pdf(pdf_path):
- try:
- pdf_document = fitz.open(pdf_path) # type: ignore
- return pdf_document
- except Exception as e:
- print(f"无法打开PDF文件：{pdf_path}。原因是：{e}")
- raise e
-
-
-def print_green_on_red(text):
- cprint(text, "green", "on_red", attrs=["bold"], end="\n\n")
-
-
-def print_green(text):
- print()
- cprint(text, "green", attrs=["bold"], end="\n\n")
-
-
-def print_red(text):
- print()
- cprint(text, "red", attrs=["bold"], end="\n\n")
-
-
-def print_yellow(text):
- print()
- cprint(text, "yellow", attrs=["bold"], end="\n\n")
-
-
-def safe_get(dict_obj, key, default):
- val = dict_obj.get(key)
- if val is None:
- return default
- else:
- return val
-
-
-def is_bbox_overlap(bbox1, bbox2):
- """
- This function checks if bbox1 and bbox2 overlap or not
-
- Parameters
- ----------
- bbox1 : list
- bbox1
- bbox2 : list
- bbox2
-
- Returns
- -------
- bool
- True if bbox1 and bbox2 overlap, else False
- """
- x0_1, y0_1, x1_1, y1_1 = bbox1
- x0_2, y0_2, x1_2, y1_2 = bbox2
-
- if x0_1 > x1_2 or x0_2 > x1_1:
- return False
- if y0_1 > y1_2 or y0_2 > y1_1:
- return False
-
- return True
-
-
-def is_in_bbox(bbox1, bbox2):
- """
- This function checks if bbox1 is in bbox2
-
- Parameters
- ----------
- bbox1 : list
- bbox1
- bbox2 : list
- bbox2
-
- Returns
- -------
- bool
- True if bbox1 is in bbox2, else False
- """
- x0_1, y0_1, x1_1, y1_1 = bbox1
- x0_2, y0_2, x1_2, y1_2 = bbox2
-
- if x0_1 >= x0_2 and y0_1 >= y0_2 and x1_1 <= x1_2 and y1_1 <= y1_2:
- return True
- else:
- return False
-
-
-def calculate_para_bbox(lines):
- """
- This function calculates the minimum bbox of the paragraph
-
- Parameters
- ----------
- lines : list
- lines
-
- Returns
- -------
- para_bbox : list
- bbox of the paragraph
- """
- x0 = min(line["bbox"][0] for line in lines)
- y0 = min(line["bbox"][1] for line in lines)
- x1 = max(line["bbox"][2] for line in lines)
- y1 = max(line["bbox"][3] for line in lines)
- return [x0, y0, x1, y1]
-
-
-def is_line_right_aligned_from_neighbors(curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width, direction=2):
- """
- This function checks if the line is right aligned from its neighbors
-
- Parameters
- ----------
- curr_line_bbox : list
- bbox of the current line
- prev_line_bbox : list
- bbox of the previous line
- next_line_bbox : list
- bbox of the next line
- avg_char_width : float
- average of char widths
- direction : int
- 0 for prev, 1 for next, 2 for both
-
- Returns
- -------
- bool
- True if the line is right aligned from its neighbors, False otherwise.
- """
- horizontal_ratio = 0.5
- horizontal_thres = horizontal_ratio * avg_char_width
-
- _, _, x1, _ = curr_line_bbox
- _, _, prev_x1, _ = prev_line_bbox if prev_line_bbox else (0, 0, 0, 0)
- _, _, next_x1, _ = next_line_bbox if next_line_bbox else (0, 0, 0, 0)
-
- if direction == 0:
- return abs(x1 - prev_x1) < horizontal_thres
- elif direction == 1:
- return abs(x1 - next_x1) < horizontal_thres
- elif direction == 2:
- return abs(x1 - prev_x1) < horizontal_thres and abs(x1 - next_x1) < horizontal_thres
- else:
- return False
-
-
-def is_line_left_aligned_from_neighbors(curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width, direction=2):
- """
- This function checks if the line is left aligned from its neighbors
-
- Parameters
- ----------
- curr_line_bbox : list
- bbox of the current line
- prev_line_bbox : list
- bbox of the previous line
- next_line_bbox : list
- bbox of the next line
- avg_char_width : float
- average of char widths
- direction : int
- 0 for prev, 1 for next, 2 for both
-
- Returns
- -------
- bool
- True if the line is left aligned from its neighbors, False otherwise.
- """
- horizontal_ratio = 0.5
- horizontal_thres = horizontal_ratio * avg_char_width
-
- x0, _, _, _ = curr_line_bbox
- prev_x0, _, _, _ = prev_line_bbox if prev_line_bbox else (0, 0, 0, 0)
- next_x0, _, _, _ = next_line_bbox if next_line_bbox else (0, 0, 0, 0)
-
- if direction == 0:
- return abs(x0 - prev_x0) < horizontal_thres
- elif direction == 1:
- return abs(x0 - next_x0) < horizontal_thres
- elif direction == 2:
- return abs(x0 - prev_x0) < horizontal_thres and abs(x0 - next_x0) < horizontal_thres
- else:
- return False
-
-
-def end_with_punctuation(line_text):
- """
- This function checks if the line ends with punctuation marks
- """
-
- english_end_puncs = [".", "?", "!"]
- chinese_end_puncs = ["。", "？", "！"]
- end_puncs = english_end_puncs + chinese_end_puncs
-
- last_non_space_char = None
- for ch in line_text[::-1]:
- if not ch.isspace():
- last_non_space_char = ch
- break
-
- if last_non_space_char is None:
- return False
-
- return last_non_space_char in end_puncs
-
-
-def is_nested_list(lst):
- if isinstance(lst, list):
- return any(isinstance(sub, list) for sub in lst)
- return False
-
-
-class DenseSingleLineBlockException(Exception):
- """
- This class defines the exception type for dense single line-block.
- """
-
- def __init__(self, message="DenseSingleLineBlockException"):
- self.message = message
- super().__init__(self.message)
-
- def __str__(self):
- return f"{self.message}"
-
- def __repr__(self):
- return f"{self.message}"
-
-
-class TitleDetectionException(Exception):
- """
- This class defines the exception type for title detection.
- """
-
- def __init__(self, message="TitleDetectionException"):
- self.message = message
- super().__init__(self.message)
-
- def __str__(self):
- return f"{self.message}"
-
- def __repr__(self):
- return f"{self.message}"
-
-
-class TitleLevelException(Exception):
- """
- This class defines the exception type for title level.
- """
-
- def __init__(self, message="TitleLevelException"):
- self.message = message
- super().__init__(self.message)
-
- def __str__(self):
- return f"{self.message}"
-
- def __repr__(self):
- return f"{self.message}"
-
-
-class ParaSplitException(Exception):
- """
- This class defines the exception type for paragraph splitting.
- """
-
- def __init__(self, message="ParaSplitException"):
- self.message = message
- super().__init__(self.message)
-
- def __str__(self):
- return f"{self.message}"
-
- def __repr__(self):
- return f"{self.message}"
-
-
-class ParaMergeException(Exception):
- """
- This class defines the exception type for paragraph merging.
- """
-
- def __init__(self, message="ParaMergeException"):
- self.message = message
- super().__init__(self.message)
-
- def __str__(self):
- return f"{self.message}"
-
- def __repr__(self):
- return f"{self.message}"
-
-
-class DiscardByException:
- """
- This class discards pdf files by exception
- """
-
- def __init__(self) -> None:
- pass
-
- def discard_by_single_line_block(self, pdf_dic, exception: DenseSingleLineBlockException):
- """
- This function discards pdf files by single line block exception
-
- Parameters
- ----------
- pdf_dic : dict
- pdf dictionary
- exception : str
- exception message
-
- Returns
- -------
- error_message : str
- """
- exception_page_nums = 0
- page_num = 0
- for page_id, page in pdf_dic.items():
- if page_id.startswith("page_"):
- page_num += 1
- if "preproc_blocks" in page.keys():
- preproc_blocks = page["preproc_blocks"]
-
- all_single_line_blocks = []
- for block in preproc_blocks:
- if len(block["lines"]) == 1:
- all_single_line_blocks.append(block)
-
- if len(preproc_blocks) > 0 and len(all_single_line_blocks) / len(preproc_blocks) > 0.9:
- exception_page_nums += 1
-
- if page_num == 0:
- return None
-
- if exception_page_nums / page_num > 0.1: # Low ratio means basically, whenever this is the case, it is discarded
- return exception.message
-
- return None
-
- def discard_by_title_detection(self, pdf_dic, exception: TitleDetectionException):
- """
- This function discards pdf files by title detection exception
-
- Parameters
- ----------
- pdf_dic : dict
- pdf dictionary
- exception : str
- exception message
-
- Returns
- -------
- error_message : str
- """
- # return exception.message
- return None
-
- def discard_by_title_level(self, pdf_dic, exception: TitleLevelException):
- """
- This function discards pdf files by title level exception
-
- Parameters
- ----------
- pdf_dic : dict
- pdf dictionary
- exception : str
- exception message
-
- Returns
- -------
- error_message : str
- """
- # return exception.message
- return None
-
- def discard_by_split_para(self, pdf_dic, exception: ParaSplitException):
- """
- This function discards pdf files by split para exception
-
- Parameters
- ----------
- pdf_dic : dict
- pdf dictionary
- exception : str
- exception message
-
- Returns
- -------
- error_message : str
- """
- # return exception.message
- return None
-
- def discard_by_merge_para(self, pdf_dic, exception: ParaMergeException):
- """
- This function discards pdf files by merge para exception
-
- Parameters
- ----------
- pdf_dic : dict
- pdf dictionary
- exception : str
- exception message
-
- Returns
- -------
- error_message : str
- """
- # return exception.message
- return None
-
-
-class LayoutFilterProcessor:
- def __init__(self) -> None:
- pass
-
- def batch_process_blocks(self, pdf_dict):
- """
- This function processes the blocks in batch.
-
- Parameters
- ----------
- self : object
- The instance of the class.
-
- pdf_dict : dict
- pdf dictionary
-
- Returns
- -------
- pdf_dict : dict
- pdf dictionary
- """
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "layout_bboxes" in blocks.keys() and "para_blocks" in blocks.keys():
- layout_bbox_objs = blocks["layout_bboxes"]
- if layout_bbox_objs is None:
- continue
- layout_bboxes = [bbox_obj["layout_bbox"] for bbox_obj in layout_bbox_objs]
-
- # Enlarge each value of x0, y0, x1, y1 for each layout_bbox to prevent loss of text.
- layout_bboxes = [
- [math.ceil(x0), math.ceil(y0), math.ceil(x1), math.ceil(y1)] for x0, y0, x1, y1 in layout_bboxes
- ]
-
- para_blocks = blocks["para_blocks"]
- if para_blocks is None:
- continue
-
- for lb_bbox in layout_bboxes:
- for i, para_block in enumerate(para_blocks):
- para_bbox = para_block["bbox"]
- para_blocks[i]["in_layout"] = 0
- if is_in_bbox(para_bbox, lb_bbox):
- para_blocks[i]["in_layout"] = 1
-
- blocks["para_blocks"] = para_blocks
-
- return pdf_dict
-
-
-class RawBlockProcessor:
- def __init__(self) -> None:
- self.y_tolerance = 2
- self.pdf_dic = {}
-
- def __span_flags_decomposer(self, span_flags):
- """
- Make font flags human readable.
-
- Parameters
- ----------
- self : object
- The instance of the class.
-
- span_flags : int
- span flags
-
- Returns
- -------
- l : dict
- decomposed flags
- """
-
- l = {
- "is_superscript": False,
- "is_italic": False,
- "is_serifed": False,
- "is_sans_serifed": False,
- "is_monospaced": False,
- "is_proportional": False,
- "is_bold": False,
- }
-
- if span_flags & 2**0:
- l["is_superscript"] = True # 表示上标
-
- if span_flags & 2**1:
- l["is_italic"] = True # 表示斜体
-
- if span_flags & 2**2:
- l["is_serifed"] = True # 表示衬线字体
- else:
- l["is_sans_serifed"] = True # 表示非衬线字体
-
- if span_flags & 2**3:
- l["is_monospaced"] = True # 表示等宽字体
- else:
- l["is_proportional"] = True # 表示比例字体
-
- if span_flags & 2**4:
- l["is_bold"] = True # 表示粗体
-
- return l
-
- def __make_new_lines(self, raw_lines):
- """
- This function makes new lines.
-
- Parameters
- ----------
- self : object
- The instance of the class.
-
- raw_lines : list
- raw lines
-
- Returns
- -------
- new_lines : list
- new lines
- """
- new_lines = []
- new_line = None
-
- for raw_line in raw_lines:
- raw_line_bbox = raw_line["bbox"]
- raw_line_spans = raw_line["spans"]
- raw_line_text = "".join([span["text"] for span in raw_line_spans])
- raw_line_dir = raw_line.get("dir", None)
-
- decomposed_line_spans = []
- for span in raw_line_spans:
- raw_flags = span["flags"]
- decomposed_flags = self.__span_flags_decomposer(raw_flags)
- span["decomposed_flags"] = decomposed_flags
- decomposed_line_spans.append(span)
-
- if new_line is None: # Handle the first line
- new_line = {
- "bbox": raw_line_bbox,
- "text": raw_line_text,
- "dir": raw_line_dir if raw_line_dir else (0, 0),
- "spans": decomposed_line_spans,
- }
- else: # Handle the rest lines
- if (
- abs(raw_line_bbox[1] - new_line["bbox"][1]) <= self.y_tolerance
- and abs(raw_line_bbox[3] - new_line["bbox"][3]) <= self.y_tolerance
- ):
- new_line["bbox"] = (
- min(new_line["bbox"][0], raw_line_bbox[0]), # left
- new_line["bbox"][1], # top
- max(new_line["bbox"][2], raw_line_bbox[2]), # right
- raw_line_bbox[3], # bottom
- )
- new_line["text"] += raw_line_text
- new_line["spans"].extend(raw_line_spans)
- new_line["dir"] = (
- new_line["dir"][0] + raw_line_dir[0],
- new_line["dir"][1] + raw_line_dir[1],
- )
- else:
- new_lines.append(new_line)
- new_line = {
- "bbox": raw_line_bbox,
- "text": raw_line_text,
- "dir": raw_line_dir if raw_line_dir else (0, 0),
- "spans": raw_line_spans,
- }
- if new_line:
- new_lines.append(new_line)
-
- return new_lines
-
- def __make_new_block(self, raw_block):
- """
- This function makes a new block.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- raw_block : dict
- a raw block
-
- Returns
- -------
- new_block : dict
- """
- new_block = {}
-
- block_id = raw_block["number"]
- block_bbox = raw_block["bbox"]
- block_text = "".join(span["text"] for line in raw_block["lines"] for span in line["spans"])
- raw_lines = raw_block["lines"]
- block_lines = self.__make_new_lines(raw_lines)
-
- new_block["block_id"] = block_id
- new_block["bbox"] = block_bbox
- new_block["text"] = block_text
- new_block["lines"] = block_lines
-
- return new_block
-
- def batch_process_blocks(self, pdf_dic):
- """
- This function processes the blocks in batch.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- blocks : list
- Input block is a list of raw blocks.
-
- Returns
- -------
- result_dict : dict
- result dictionary
- """
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "preproc_blocks" in blocks.keys():
- input_blocks = blocks["preproc_blocks"]
- for raw_block in input_blocks:
- new_block = self.__make_new_block(raw_block)
- para_blocks.append(new_block)
-
- blocks["para_blocks"] = para_blocks
-
- return pdf_dic
-
-
-class BlockStatisticsCalculator:
- """
- This class calculates the statistics of the block.
- """
-
- def __init__(self) -> None:
- pass
-
- def __calc_stats_of_new_lines(self, new_lines):
- """
- This function calculates the paragraph metrics
-
- Parameters
- ----------
- combined_lines : list
- combined lines
-
- Returns
- -------
- X0 : float
- Median of x0 values, which represents the left average boundary of the block
- X1 : float
- Median of x1 values, which represents the right average boundary of the block
- avg_char_width : float
- Average of char widths, which represents the average char width of the block
- avg_char_height : float
- Average of line heights, which represents the average line height of the block
-
- """
- x0_values = []
- x1_values = []
- char_widths = []
- char_heights = []
-
- block_font_types = []
- block_font_sizes = []
- block_directions = []
-
- if len(new_lines) > 0:
- for i, line in enumerate(new_lines):
- line_bbox = line["bbox"]
- line_text = line["text"]
- line_spans = line["spans"]
-
- num_chars = len([ch for ch in line_text if not ch.isspace()])
-
- x0_values.append(line_bbox[0])
- x1_values.append(line_bbox[2])
-
- if num_chars > 0:
- char_width = (line_bbox[2] - line_bbox[0]) / num_chars
- char_widths.append(char_width)
-
- for span in line_spans:
- block_font_types.append(span["font"])
- block_font_sizes.append(span["size"])
-
- if "dir" in line:
- block_directions.append(line["dir"])
-
- # line_font_types = [span["font"] for span in line_spans]
- char_heights = [span["size"] for span in line_spans]
-
- X0 = np.median(x0_values) if x0_values else 0
- X1 = np.median(x1_values) if x1_values else 0
- avg_char_width = sum(char_widths) / len(char_widths) if char_widths else 0
- avg_char_height = sum(char_heights) / len(char_heights) if char_heights else 0
-
- # max_freq_font_type = max(set(block_font_types), key=block_font_types.count) if block_font_types else None
-
- max_span_length = 0
- max_span_font_type = None
- for line in new_lines:
- line_spans = line["spans"]
- for span in line_spans:
- span_length = span["bbox"][2] - span["bbox"][0]
- if span_length > max_span_length:
- max_span_length = span_length
- max_span_font_type = span["font"]
-
- max_freq_font_type = max_span_font_type
-
- avg_font_size = sum(block_font_sizes) / len(block_font_sizes) if block_font_sizes else None
-
- avg_dir_horizontal = sum([dir[0] for dir in block_directions]) / len(block_directions) if block_directions else 0
- avg_dir_vertical = sum([dir[1] for dir in block_directions]) / len(block_directions) if block_directions else 0
-
- median_font_size = float(np.median(block_font_sizes)) if block_font_sizes else None
-
- return (
- X0,
- X1,
- avg_char_width,
- avg_char_height,
- max_freq_font_type,
- avg_font_size,
- (avg_dir_horizontal, avg_dir_vertical),
- median_font_size,
- )
-
- def __make_new_block(self, input_block):
- new_block = {}
-
- raw_lines = input_block["lines"]
- stats = self.__calc_stats_of_new_lines(raw_lines)
-
- block_id = input_block["block_id"]
- block_bbox = input_block["bbox"]
- block_text = input_block["text"]
- block_lines = raw_lines
- block_avg_left_boundary = stats[0]
- block_avg_right_boundary = stats[1]
- block_avg_char_width = stats[2]
- block_avg_char_height = stats[3]
- block_font_type = stats[4]
- block_font_size = stats[5]
- block_direction = stats[6]
- block_median_font_size = stats[7]
-
- new_block["block_id"] = block_id
- new_block["bbox"] = block_bbox
- new_block["text"] = block_text
- new_block["dir"] = block_direction
- new_block["X0"] = block_avg_left_boundary
- new_block["X1"] = block_avg_right_boundary
- new_block["avg_char_width"] = block_avg_char_width
- new_block["avg_char_height"] = block_avg_char_height
- new_block["block_font_type"] = block_font_type
- new_block["block_font_size"] = block_font_size
- new_block["lines"] = block_lines
- new_block["median_font_size"] = block_median_font_size
-
- return new_block
-
- def batch_process_blocks(self, pdf_dic):
- """
- This function processes the blocks in batch.
-
- Parameters
- ----------
- self : object
- The instance of the class.
- ----------
- blocks : list
- Input block is a list of raw blocks.
- Schema can refer to the value of key ""preproc_blocks".
-
- Returns
- -------
- result_dict : dict
- result dictionary
- """
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "para_blocks" in blocks.keys():
- input_blocks = blocks["para_blocks"]
- for input_block in input_blocks:
- new_block = self.__make_new_block(input_block)
- para_blocks.append(new_block)
-
- blocks["para_blocks"] = para_blocks
-
- return pdf_dic
-
-
-class DocStatisticsCalculator:
- """
- This class calculates the statistics of the document.
- """
-
- def __init__(self) -> None:
- pass
-
- def calc_stats_of_doc(self, pdf_dict):
- """
- This function computes the statistics of the document
-
- Parameters
- ----------
- result_dict : dict
- result dictionary
-
- Returns
- -------
- statistics : dict
- statistics of the document
- """
-
- total_text_length = 0
- total_num_blocks = 0
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "para_blocks" in blocks.keys():
- para_blocks = blocks["para_blocks"]
- for para_block in para_blocks:
- total_text_length += len(para_block["text"])
- total_num_blocks += 1
-
- avg_text_length = total_text_length / total_num_blocks if total_num_blocks else 0
-
- font_list = []
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- if "para_blocks" in blocks.keys():
- input_blocks = blocks["para_blocks"]
- for input_block in input_blocks:
- block_text_length = len(input_block.get("text", ""))
- if block_text_length < avg_text_length * 0.5:
- continue
- block_font_type = safe_get(input_block, "block_font_type", "")
- block_font_size = safe_get(input_block, "block_font_size", 0)
- font_list.append((block_font_type, block_font_size))
-
- font_counter = Counter(font_list)
- most_common_font = font_counter.most_common(1)[0] if font_list else (("", 0), 0)
- second_most_common_font = font_counter.most_common(2)[1] if len(font_counter) > 1 else (("", 0), 0)
-
- statistics = {
- "num_pages": 0,
- "num_blocks": 0,
- "num_paras": 0,
- "num_titles": 0,
- "num_header_blocks": 0,
- "num_footer_blocks": 0,
- "num_watermark_blocks": 0,
- "num_vertical_margin_note_blocks": 0,
- "most_common_font_type": most_common_font[0][0],
- "most_common_font_size": most_common_font[0][1],
- "number_of_most_common_font": most_common_font[1],
- "second_most_common_font_type": second_most_common_font[0][0],
- "second_most_common_font_size": second_most_common_font[0][1],
- "number_of_second_most_common_font": second_most_common_font[1],
- "avg_text_length": avg_text_length,
- }
-
- for page_id, blocks in pdf_dict.items():
- if page_id.startswith("page_"):
- blocks = pdf_dict[page_id]["para_blocks"]
- statistics["num_pages"] += 1
- for block_id, block_data in enumerate(blocks):
- statistics["num_blocks"] += 1
-
- if "paras" in block_data.keys():
- statistics["num_paras"] += len(block_data["paras"])
-
- for line in block_data["lines"]:
- if line.get("is_title", 0):
- statistics["num_titles"] += 1
-
- if block_data.get("is_header", 0):
- statistics["num_header_blocks"] += 1
- if block_data.get("is_footer", 0):
- statistics["num_footer_blocks"] += 1
- if block_data.get("is_watermark", 0):
- statistics["num_watermark_blocks"] += 1
- if block_data.get("is_vertical_margin_note", 0):
- statistics["num_vertical_margin_note_blocks"] += 1
-
- pdf_dict["statistics"] = statistics
-
- return pdf_dict
-
-
-class TitleProcessor:
- """
- This class processes the title.
- """
-
- def __init__(self, *doc_statistics) -> None:
- if len(doc_statistics) > 0:
- self.doc_statistics = doc_statistics[0]
-
- self.nlp_model = NLPModels()
- self.MAX_TITLE_LEVEL = 3
- self.numbered_title_pattern = r"""
- ^ # 行首
- ( # 开始捕获组
- [\(\（]\d+[\)\）] # 括号内数字，支持中文和英文括号，例如：(1) 或 （1）
- |\d+[\)\）]\s # 数字后跟右括号和空格，支持中文和英文括号，例如：2) 或 2）
- |[\(\（][A-Z][\)\）] # 括号内大写字母，支持中文和英文括号，例如：(A) 或 （A）
- |[A-Z][\)\）]\s # 大写字母后跟右括号和空格，例如：A) 或 A）
- |[\(\（][IVXLCDM]+[\)\）] # 括号内罗马数字，支持中文和英文括号，例如：(I) 或 （I）
- |[IVXLCDM]+[\)\）]\s # 罗马数字后跟右括号和空格，例如：I) 或 I）
- |\d+(\.\d+)*\s # 数字或复合数字编号后跟空格，例如：1. 或 3.2.1
- |[一二三四五六七八九十百千]+[、\s] # 中文序号后跟顿号和空格，例如：一、
- |[\（|\(][一二三四五六七八九十百千]+[\）|\)]\s* # 中文括号内中文序号后跟空格，例如：（一）
- |[A-Z]\.\d+(\.\d+)?\s # 大写字母后跟点和数字，例如：A.1 或 A.1.1
- |[\(\（][a-z][\)\）] # 括号内小写字母，支持中文和英文括号，例如：(a) 或 （a）
- |[a-z]\)\s # 小写字母后跟右括号和空格，例如：a)
- |[A-Z]-\s # 大写字母后跟短横线和空格，例如：A-
- |\w+:\s # 英文序号词后跟冒号和空格，例如：First:
- |第[一二三四五六七八九十百千]+[章节部分条款]\s # 以“第”开头的中文标题后跟空格
- |[IVXLCDM]+\. # 罗马数字后跟点，例如：I.
- |\d+\.\s # 单个数字后跟点和空格，例如：1.
- ) # 结束捕获组
- .+ # 标题的其余部分
- """
-
- def _is_potential_title(
- self,
- curr_line,
- prev_line,
- prev_line_is_title,
- next_line,
- avg_char_width,
- avg_char_height,
- median_font_size,
- ):
- """
- This function checks if the line is a potential title.
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
- next_line : dict
- next line
- avg_char_width : float
- average of char widths
- avg_char_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the line is a potential title, False otherwise.
- """
-
- def __is_line_centered(line_bbox, page_bbox, avg_char_width):
- """
- This function checks if the line is centered on the page
-
- Parameters
- ----------
- line_bbox : list
- bbox of the line
- page_bbox : list
- bbox of the page
- avg_char_width : float
- average of char widths
-
- Returns
- -------
- bool
- True if the line is centered on the page, False otherwise.
- """
- horizontal_ratio = 0.5
- horizontal_thres = horizontal_ratio * avg_char_width
-
- x0, _, x1, _ = line_bbox
- _, _, page_x1, _ = page_bbox
-
- return abs((x0 + x1) / 2 - page_x1 / 2) < horizontal_thres
-
- def __is_bold_font_line(line):
- """
- Check if a line contains any bold font style.
- """
-
- def _is_bold_span(span):
- # if span text is empty or only contains space, return False
- if not span["text"].strip():
- return False
-
- return bool(span["flags"] & 2**4) # Check if the font is bold
-
- for span in line["spans"]:
- if not _is_bold_span(span):
- return False
-
- return True
-
- def __is_italic_font_line(line):
- """
- Check if a line contains any italic font style.
- """
-
- def __is_italic_span(span):
- return bool(span["flags"] & 2**1) # Check if the font is italic
-
- for span in line["spans"]:
- if not __is_italic_span(span):
- return False
-
- return True
-
- def __is_punctuation_heavy(line_text):
- """
- Check if the line contains a high ratio of punctuation marks, which may indicate
- that the line is not a title.
-
- Parameters:
- line_text (str): Text of the line.
-
- Returns:
- bool: True if the line is heavy with punctuation, False otherwise.
- """
- # Pattern for common title format like "X.Y. Title"
- pattern = r"\b\d+\.\d+\..*\b"
-
- # If the line matches the title format, return False
- if re.match(pattern, line_text.strip()):
- return False
-
- # Find all punctuation marks in the line
- punctuation_marks = re.findall(r"[^\w\s]", line_text)
- number_of_punctuation_marks = len(punctuation_marks)
-
- text_length = len(line_text)
-
- if text_length == 0:
- return False
-
- punctuation_ratio = number_of_punctuation_marks / text_length
- if punctuation_ratio >= 0.1:
- return True
-
- return False
-
- def __has_mixed_font_styles(spans, strict_mode=False):
- """
- This function checks if the line has mixed font styles, the strict mode will compare the font types
-
- Parameters
- ----------
- spans : list
- spans of the line
- strict_mode : bool
- True for strict mode, the font types will be fully compared
- False for non-strict mode, the font types will be compared by the most longest common prefix
-
- Returns
- -------
- bool
- True if the line has mixed font styles, False otherwise.
- """
- if strict_mode:
- font_styles = set()
- for span in spans:
- font_style = span["font"].lower()
- font_styles.add(font_style)
-
- return len(font_styles) > 1
-
- else: # non-strict mode
- font_styles = []
- for span in spans:
- font_style = span["font"].lower()
- font_styles.append(font_style)
-
- if len(font_styles) > 1:
- longest_common_prefix = os.path.commonprefix(font_styles)
- if len(longest_common_prefix) > 0:
- return False
- else:
- return True
- else:
- return False
-
- def __is_different_font_type_from_neighbors(curr_line_font_type, prev_line_font_type, next_line_font_type):
- """
- This function checks if the current line has a different font type from the previous and next lines
-
- Parameters
- ----------
- curr_line_font_type : str
- font type of the current line
- prev_line_font_type : str
- font type of the previous line
- next_line_font_type : str
- font type of the next line
-
- Returns
- -------
- bool
- True if the current line has a different font type from the previous and next lines, False otherwise.
- """
- return all(
- curr_line_font_type != other_font_type.lower()
- for other_font_type in [prev_line_font_type, next_line_font_type]
- if other_font_type is not None
- )
-
- def __is_larger_font_size_from_neighbors(curr_line_font_size, prev_line_font_size, next_line_font_size):
- """
- This function checks if the current line has a larger font size than the previous and next lines
-
- Parameters
- ----------
- curr_line_font_size : float
- font size of the current line
- prev_line_font_size : float
- font size of the previous line
- next_line_font_size : float
- font size of the next line
-
- Returns
- -------
- bool
- True if the current line has a larger font size than the previous and next lines, False otherwise.
- """
- return all(
- curr_line_font_size > other_font_size * 1.2
- for other_font_size in [prev_line_font_size, next_line_font_size]
- if other_font_size is not None
- )
-
- def __is_similar_to_pre_line(curr_line_font_type, prev_line_font_type, curr_line_font_size, prev_line_font_size):
- """
- This function checks if the current line is similar to the previous line
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
-
- Returns
- -------
- bool
- True if the current line is similar to the previous line, False otherwise.
- """
-
- if curr_line_font_type == prev_line_font_type and curr_line_font_size == prev_line_font_size:
- return True
- else:
- return False
-
- def __is_same_font_type_of_docAvg(curr_line_font_type):
- """
- This function checks if the current line has the same font type as the document average font type
-
- Parameters
- ----------
- curr_line_font_type : str
- font type of the current line
-
- Returns
- -------
- bool
- True if the current line has the same font type as the document average font type, False otherwise.
- """
- doc_most_common_font_type = safe_get(self.doc_statistics, "most_common_font_type", "").lower()
- doc_second_most_common_font_type = safe_get(self.doc_statistics, "second_most_common_font_type", "").lower()
-
- return curr_line_font_type.lower() in [doc_most_common_font_type, doc_second_most_common_font_type]
-
- def __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio: float = 1):
- """
- This function checks if the current line has a large enough font size
-
- Parameters
- ----------
- curr_line_font_size : float
- font size of the current line
- ratio : float
- ratio of the current line font size to the document average font size
-
- Returns
- -------
- bool
- True if the current line has a large enough font size, False otherwise.
- """
- doc_most_common_font_size = safe_get(self.doc_statistics, "most_common_font_size", 0)
- doc_second_most_common_font_size = safe_get(self.doc_statistics, "second_most_common_font_size", 0)
- doc_avg_font_size = min(doc_most_common_font_size, doc_second_most_common_font_size)
-
- return curr_line_font_size >= doc_avg_font_size * ratio
-
- def __is_sufficient_spacing_above_and_below(
- curr_line_bbox,
- prev_line_bbox,
- next_line_bbox,
- avg_char_height,
- median_font_size,
- ):
- """
- This function checks if the current line has sufficient spacing above and below
-
- Parameters
- ----------
- curr_line_bbox : list
- bbox of the current line
- prev_line_bbox : list
- bbox of the previous line
- next_line_bbox : list
- bbox of the next line
- avg_char_width : float
- average of char widths
- avg_char_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the current line has sufficient spacing above and below, False otherwise.
- """
- vertical_ratio = 1.25
- vertical_thres = vertical_ratio * median_font_size
-
- _, y0, _, y1 = curr_line_bbox
-
- sufficient_spacing_above = False
- if prev_line_bbox:
- vertical_spacing_above = min(y0 - prev_line_bbox[1], y1 - prev_line_bbox[3])
- sufficient_spacing_above = vertical_spacing_above > vertical_thres
- else:
- sufficient_spacing_above = True
-
- sufficient_spacing_below = False
- if next_line_bbox:
- vertical_spacing_below = min(next_line_bbox[1] - y0, next_line_bbox[3] - y1)
- sufficient_spacing_below = vertical_spacing_below > vertical_thres
- else:
- sufficient_spacing_below = True
-
- return (sufficient_spacing_above, sufficient_spacing_below)
-
- def __is_word_list_line_by_rules(curr_line_text):
- """
- This function checks if the current line is a word list
-
- Parameters
- ----------
- curr_line_text : str
- text of the current line
-
- Returns
- -------
- bool
- True if the current line is a name list, False otherwise.
- """
- # name_list_pattern = r"([a-zA-Z][a-zA-Z\s]{0,20}[a-zA-Z]|[\u4e00-\u9fa5·]{2,16})(?=[，,;；\s]|$)"
- name_list_pattern = r"(?= 0.9:
- return True
-
- return False
-
- def __is_equation(line_text):
- """
- This function checks if the current line is an equation.
-
- Parameters
- ----------
- line_text : str
-
- Returns
- -------
- bool
- True if the current line is an equation, False otherwise.
- """
- equation_reg = r"\$.*?\\overline.*?\$" # to match interline equations
-
- if re.search(equation_reg, line_text):
- return True
- else:
- return False
-
- def __is_title_by_len(text, max_length=200):
- """
- This function checks if the current line is a title by length.
-
- Parameters
- ----------
- text : str
- text of the current line
-
- max_length : int
- max length of the title
-
- Returns
- -------
- bool
- True if the current line is a title, False otherwise.
-
- """
- text = text.strip()
- return len(text) <= max_length
-
- def __compute_line_font_type_and_size(curr_line):
- """
- This function computes the font type and font size of the line.
-
- Parameters
- ----------
- line : dict
- line
-
- Returns
- -------
- font_type : str
- font type of the line
- font_size : float
- font size of the line
- """
- spans = curr_line["spans"]
- max_accumulated_length = 0
- max_span_font_size = curr_line["spans"][0]["size"] # default value, float type
- max_span_font_type = curr_line["spans"][0]["font"].lower() # default value, string type
- for span in spans:
- if span["text"].isspace():
- continue
- span_length = span["bbox"][2] - span["bbox"][0]
- if span_length > max_accumulated_length:
- max_accumulated_length = span_length
- max_span_font_size = span["size"]
- max_span_font_type = span["font"].lower()
-
- return max_span_font_type, max_span_font_size
-
- def __is_a_consistent_sub_title(pre_line, curr_line):
- """
- This function checks if the current line is a consistent sub title.
-
- Parameters
- ----------
- pre_line : dict
- previous line
- curr_line : dict
- current line
-
- Returns
- -------
- bool
- True if the current line is a consistent sub title, False otherwise.
- """
- if pre_line is None:
- return False
-
- start_letter_of_pre_line = pre_line["text"][0]
- start_letter_of_curr_line = curr_line["text"][0]
-
- has_same_prefix_digit = (
- start_letter_of_pre_line.isdigit()
- and start_letter_of_curr_line.isdigit()
- and start_letter_of_pre_line == start_letter_of_curr_line
- )
-
- # prefix text of curr_line satisfies the following title format: x.x
- prefix_text_pattern = r"^\d+\.\d+"
- has_subtitle_format = re.match(prefix_text_pattern, curr_line["text"])
-
- if has_same_prefix_digit or has_subtitle_format:
- return True
-
- """
- Title detecting main Process.
- """
-
- """
- Basic features about the current line.
- """
- curr_line_bbox = curr_line["bbox"]
- curr_line_text = curr_line["text"]
- curr_line_font_type, curr_line_font_size = __compute_line_font_type_and_size(curr_line)
-
- if len(curr_line_text.strip()) == 0: # skip empty lines
- return False, False
-
- prev_line_bbox = prev_line["bbox"] if prev_line else None
- if prev_line:
- prev_line_font_type, prev_line_font_size = __compute_line_font_type_and_size(prev_line)
- else:
- prev_line_font_type, prev_line_font_size = None, None
-
- next_line_bbox = next_line["bbox"] if next_line else None
- if next_line:
- next_line_font_type, next_line_font_size = __compute_line_font_type_and_size(next_line)
- else:
- next_line_font_type, next_line_font_size = None, None
-
- """
- Aggregated features about the current line.
- """
- is_italc_font = __is_italic_font_line(curr_line)
- is_bold_font = __is_bold_font_line(curr_line)
-
- is_font_size_little_less_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=0.8)
- is_font_size_not_less_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=1)
- is_much_larger_font_than_doc_avg = __is_font_size_not_less_than_docAvg(curr_line_font_size, ratio=1.6)
-
- is_not_same_font_type_of_docAvg = not __is_same_font_type_of_docAvg(curr_line_font_type)
-
- is_potential_title_font = is_bold_font or is_font_size_not_less_than_doc_avg or is_not_same_font_type_of_docAvg
-
- is_mix_font_styles_strict = __has_mixed_font_styles(curr_line["spans"], strict_mode=True)
- is_mix_font_styles_loose = __has_mixed_font_styles(curr_line["spans"], strict_mode=False)
-
- is_punctuation_heavy = __is_punctuation_heavy(curr_line_text)
-
- is_word_list_line_by_rules = __is_word_list_line_by_rules(curr_line_text)
- is_person_or_org_list_line_by_nlp = __get_text_catgr_by_nlp(curr_line_text) in ["PERSON", "GPE", "ORG"]
-
- is_font_size_larger_than_neighbors = __is_larger_font_size_from_neighbors(
- curr_line_font_size, prev_line_font_size, next_line_font_size
- )
-
- is_font_type_diff_from_neighbors = __is_different_font_type_from_neighbors(
- curr_line_font_type, prev_line_font_type, next_line_font_type
- )
-
- has_sufficient_spaces_above, has_sufficient_spaces_below = __is_sufficient_spacing_above_and_below(
- curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_height, median_font_size
- )
-
- is_similar_to_pre_line = __is_similar_to_pre_line(
- curr_line_font_type, prev_line_font_type, curr_line_font_size, prev_line_font_size
- )
-
- is_consis_sub_title = __is_a_consistent_sub_title(prev_line, curr_line)
-
- """
- Further aggregated features about the current line.
-
- Attention:
- Features that start with __ are for internal use.
- """
-
- __is_line_left_aligned_from_neighbors = is_line_left_aligned_from_neighbors(
- curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width
- )
- __is_font_diff_from_neighbors = is_font_size_larger_than_neighbors or is_font_type_diff_from_neighbors
- is_a_left_inline_title = (
- is_mix_font_styles_strict and __is_line_left_aligned_from_neighbors and __is_font_diff_from_neighbors
- )
-
- is_title_by_check_prev_line = prev_line is None and has_sufficient_spaces_above and is_potential_title_font
- is_title_by_check_next_line = next_line is None and has_sufficient_spaces_below and is_potential_title_font
-
- is_title_by_check_pre_and_next_line = (
- (prev_line is not None or next_line is not None)
- and has_sufficient_spaces_above
- and has_sufficient_spaces_below
- and is_potential_title_font
- )
-
- is_numbered_title = __is_numbered_title(curr_line_text) and (
- (has_sufficient_spaces_above or prev_line is None) and (has_sufficient_spaces_below or next_line is None)
- )
-
- is_not_end_with_ending_puncs = not __is_end_with_ending_puncs(curr_line_text)
-
- is_not_only_no_meaning_symbols = not __contains_only_no_meaning_symbols(curr_line_text)
-
- is_equation = __is_equation(curr_line_text)
-
- is_title_by_len = __is_title_by_len(curr_line_text)
-
- """
- Decide if the line is a title.
- """
-
- is_title = (
- is_not_end_with_ending_puncs # not end with ending punctuation marks
- and is_not_only_no_meaning_symbols # not only have no meaning symbols
- and is_title_by_len # is a title by length, default max length is 200
- and not is_equation # an interline equation should never be a title
- and is_potential_title_font # is a potential title font, which is bold or larger than the document average font size or not the same font type as the document average font type
- and (
- (is_not_same_font_type_of_docAvg and is_font_size_not_less_than_doc_avg)
- or (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- or (
- is_much_larger_font_than_doc_avg
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- or (
- is_font_size_little_less_than_doc_avg
- and is_bold_font
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- ) # Consider the following situations: bold font, much larger font than doc avg, not same font type as doc avg, sufficient spacing above and below
- and (
- (
- not is_person_or_org_list_line_by_nlp
- and (
- is_much_larger_font_than_doc_avg
- or (is_not_same_font_type_of_docAvg and is_font_size_not_less_than_doc_avg)
- )
- )
- or (
- not (is_word_list_line_by_rules and is_person_or_org_list_line_by_nlp)
- and not is_a_left_inline_title
- and not is_punctuation_heavy
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- )
- or (
- is_person_or_org_list_line_by_nlp
- and (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- and (is_bold_font and is_much_larger_font_than_doc_avg and is_not_same_font_type_of_docAvg)
- )
- or (is_numbered_title and not is_a_left_inline_title)
- ) # Exclude the following situations: person/org list
- )
- # ) or (prev_line_is_title and is_consis_sub_title)
-
- is_name_or_org_list_to_be_removed = (
- (is_person_or_org_list_line_by_nlp)
- and is_punctuation_heavy
- and (is_title_by_check_prev_line or is_title_by_check_next_line or is_title_by_check_pre_and_next_line)
- ) and not is_title
-
- if is_name_or_org_list_to_be_removed:
- is_author_or_org_list = True
- else:
- is_author_or_org_list = False
-
- # return is_title, is_author_or_org_list
-
- """
- # print reason why the line is a title
- if is_title:
- print_green("This line is a title.")
- print_green("↓" * 10)
- print()
- print("curr_line_text: ", curr_line_text)
- print()
-
- # print reason why the line is not a title
- line_text = curr_line_text.strip()
- test_text = "Career/Personal Life"
- text_content_condition = line_text == test_text
-
- if not is_title and text_content_condition: # Print specific line
- # if not is_title: # Print each line
- print_red("This line is not a title.")
- print_red("↓" * 10)
-
- print()
- print("curr_line_text: ", curr_line_text)
- print()
-
- if is_not_end_with_ending_puncs:
- print_green(f"is_not_end_with_ending_puncs")
- else:
- print_red(f"is_end_with_ending_puncs")
-
- if is_not_only_no_meaning_symbols:
- print_green(f"is_not_only_no_meaning_symbols")
- else:
- print_red(f"is_only_no_meaning_symbols")
-
- if is_title_by_len:
- print_green(f"is_title_by_len: {is_title_by_len}")
- else:
- print_red(f"is_not_title_by_len: {is_title_by_len}")
-
- if is_equation:
- print_red(f"is_equation")
- else:
- print_green(f"is_not_equation")
-
- if is_potential_title_font:
- print_green(f"is_potential_title_font")
- else:
- print_red(f"is_not_potential_title_font")
-
- if is_punctuation_heavy:
- print_red("is_punctuation_heavy")
- else:
- print_green("is_not_punctuation_heavy")
-
- if is_bold_font:
- print_green(f"is_bold_font")
- else:
- print_red(f"is_not_bold_font")
-
- if is_font_size_not_less_than_doc_avg:
- print_green(f"is_larger_font_than_doc_avg")
- else:
- print_red(f"is_not_larger_font_than_doc_avg")
-
- if is_much_larger_font_than_doc_avg:
- print_green(f"is_much_larger_font_than_doc_avg")
- else:
- print_red(f"is_not_much_larger_font_than_doc_avg")
-
- if is_not_same_font_type_of_docAvg:
- print_green(f"is_not_same_font_type_of_docAvg")
- else:
- print_red(f"is_same_font_type_of_docAvg")
-
- if is_word_list_line_by_rules:
- print_red("is_word_list_line_by_rules")
- else:
- print_green("is_not_name_list_by_rules")
-
- if is_person_or_org_list_line_by_nlp:
- print_red("is_person_or_org_list_line_by_nlp")
- else:
- print_green("is_not_person_or_org_list_line_by_nlp")
-
- if not is_numbered_title:
- print_red("is_not_numbered_title")
- else:
- print_green("is_numbered_title")
-
- if is_a_left_inline_title:
- print_red("is_a_left_inline_title")
- else:
- print_green("is_not_a_left_inline_title")
-
- if not is_title_by_check_prev_line:
- print_red("is_not_title_by_check_prev_line")
- else:
- print_green("is_title_by_check_prev_line")
-
- if not is_title_by_check_next_line:
- print_red("is_not_title_by_check_next_line")
- else:
- print_green("is_title_by_check_next_line")
-
- if not is_title_by_check_pre_and_next_line:
- print_red("is_not_title_by_check_pre_and_next_line")
- else:
- print_green("is_title_by_check_pre_and_next_line")
-
- # print_green("Common features:")
- # print_green("↓" * 10)
-
- # print(f" curr_line_font_type: {curr_line_font_type}")
- # print(f" curr_line_font_size: {curr_line_font_size}")
- # print()
-
- """
-
- return is_title, is_author_or_org_list
-
- def _detect_title(self, input_block):
- """
- Use the functions 'is_potential_title' to detect titles of each paragraph block.
- If a line is a title, then the value of key 'is_title' of the line will be set to True.
- """
-
- raw_lines = input_block["lines"]
-
- prev_line_is_title_flag = False
-
- for i, curr_line in enumerate(raw_lines):
- prev_line = raw_lines[i - 1] if i > 0 else None
- next_line = raw_lines[i + 1] if i < len(raw_lines) - 1 else None
-
- blk_avg_char_width = input_block["avg_char_width"]
- blk_avg_char_height = input_block["avg_char_height"]
- blk_media_font_size = input_block["median_font_size"]
-
- is_title, is_author_or_org_list = self._is_potential_title(
- curr_line,
- prev_line,
- prev_line_is_title_flag,
- next_line,
- blk_avg_char_width,
- blk_avg_char_height,
- blk_media_font_size,
- )
-
- if is_title:
- curr_line["is_title"] = is_title
- prev_line_is_title_flag = True
- else:
- curr_line["is_title"] = False
- prev_line_is_title_flag = False
-
- # print(f"curr_line['text']: {curr_line['text']}")
- # print(f"curr_line['is_title']: {curr_line['is_title']}")
- # print(f"prev_line['text']: {prev_line['text'] if prev_line else None}")
- # print(f"prev_line_is_title_flag: {prev_line_is_title_flag}")
- # print()
-
- if is_author_or_org_list:
- curr_line["is_author_or_org_list"] = is_author_or_org_list
- else:
- curr_line["is_author_or_org_list"] = False
-
- return input_block
-
- def batch_detect_titles(self, pdf_dic):
- """
- This function batch process the blocks to detect titles.
-
- Parameters
- ----------
- pdf_dict : dict
- result dictionary
-
- Returns
- -------
- pdf_dict : dict
- result dictionary
- """
- num_titles = 0
-
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "para_blocks" in blocks.keys():
- para_blocks = blocks["para_blocks"]
-
- all_single_line_blocks = []
- for block in para_blocks:
- if len(block["lines"]) == 1:
- all_single_line_blocks.append(block)
-
- new_para_blocks = []
- if not len(all_single_line_blocks) == len(para_blocks): # Not all blocks are single line blocks.
- for para_block in para_blocks:
- new_block = self._detect_title(para_block)
- new_para_blocks.append(new_block)
- num_titles += sum([line.get("is_title", 0) for line in new_block["lines"]])
- else: # All blocks are single line blocks.
- for para_block in para_blocks:
- new_para_blocks.append(para_block)
- num_titles += sum([line.get("is_title", 0) for line in para_block["lines"]])
- para_blocks = new_para_blocks
-
- blocks["para_blocks"] = para_blocks
-
- for para_block in para_blocks:
- all_titles = all(safe_get(line, "is_title", False) for line in para_block["lines"])
- para_text_len = sum([len(line["text"]) for line in para_block["lines"]])
- if (
- all_titles and para_text_len < 200
- ): # total length of the paragraph is less than 200, more than this should not be a title
- para_block["is_block_title"] = 1
- else:
- para_block["is_block_title"] = 0
-
- all_name_or_org_list_to_be_removed = all(
- safe_get(line, "is_author_or_org_list", False) for line in para_block["lines"]
- )
- if all_name_or_org_list_to_be_removed and page_id == "page_0":
- para_block["is_block_an_author_or_org_list"] = 1
- else:
- para_block["is_block_an_author_or_org_list"] = 0
-
- pdf_dic["statistics"]["num_titles"] = num_titles
-
- return pdf_dic
-
- def _recog_title_level(self, title_blocks):
- """
- This function determines the title level based on the font size of the title.
-
- Parameters
- ----------
- title_blocks : list
-
- Returns
- -------
- title_blocks : list
- """
-
- font_sizes = np.array([safe_get(tb["block"], "block_font_size", 0) for tb in title_blocks])
-
- # Use the mean and std of font sizes to remove extreme values
- mean_font_size = np.mean(font_sizes)
- std_font_size = np.std(font_sizes)
- min_extreme_font_size = mean_font_size - std_font_size # type: ignore
- max_extreme_font_size = mean_font_size + std_font_size # type: ignore
-
- # Compute the threshold for title level
- middle_font_sizes = font_sizes[(font_sizes > min_extreme_font_size) & (font_sizes < max_extreme_font_size)]
- if middle_font_sizes.size > 0:
- middle_mean_font_size = np.mean(middle_font_sizes)
- level_threshold = middle_mean_font_size
- else:
- level_threshold = mean_font_size
-
- for tb in title_blocks:
- title_block = tb["block"]
- title_font_size = safe_get(title_block, "block_font_size", 0)
-
- current_level = 1 # Initialize title level, the biggest level is 1
-
- # print(f"Before adjustment by font size, {current_level}")
- if title_font_size >= max_extreme_font_size:
- current_level = 1
- elif title_font_size <= min_extreme_font_size:
- current_level = 3
- elif float(title_font_size) >= float(level_threshold):
- current_level = 2
- else:
- current_level = 3
- # print(f"After adjustment by font size, {current_level}")
-
- title_block["block_title_level"] = current_level
-
- return title_blocks
-
- def batch_recog_title_level(self, pdf_dic):
- """
- This function batch process the blocks to recognize title level.
-
- Parameters
- ----------
- pdf_dict : dict
- result dictionary
-
- Returns
- -------
- pdf_dict : dict
- result dictionary
- """
- title_blocks = []
-
- # Collect all titles
- for page_id, blocks in pdf_dic.items():
- if page_id.startswith("page_"):
- para_blocks = blocks.get("para_blocks", [])
- for block in para_blocks:
- if block.get("is_block_title"):
- title_obj = {"page_id": page_id, "block": block}
- title_blocks.append(title_obj)
-
- # Determine title level
- if title_blocks:
- # Determine title level based on font size
- title_blocks = self._recog_title_level(title_blocks)
-
- return pdf_dic
-
-
-class BlockTerminationProcessor:
- """
- This class is used to process the block termination.
- """
-
- def __init__(self) -> None:
- pass
-
- def _is_consistent_lines(
- self,
- curr_line,
- prev_line,
- next_line,
- consistent_direction, # 0 for prev, 1 for next, 2 for both
- ):
- """
- This function checks if the line is consistent with its neighbors
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
- next_line : dict
- next line
- consistent_direction : int
- 0 for prev, 1 for next, 2 for both
-
- Returns
- -------
- bool
- True if the line is consistent with its neighbors, False otherwise.
- """
-
- curr_line_font_size = curr_line["spans"][0]["size"]
- curr_line_font_type = curr_line["spans"][0]["font"].lower()
-
- if consistent_direction == 0:
- if prev_line:
- prev_line_font_size = prev_line["spans"][0]["size"]
- prev_line_font_type = prev_line["spans"][0]["font"].lower()
- return curr_line_font_size == prev_line_font_size and curr_line_font_type == prev_line_font_type
- else:
- return False
-
- elif consistent_direction == 1:
- if next_line:
- next_line_font_size = next_line["spans"][0]["size"]
- next_line_font_type = next_line["spans"][0]["font"].lower()
- return curr_line_font_size == next_line_font_size and curr_line_font_type == next_line_font_type
- else:
- return False
-
- elif consistent_direction == 2:
- if prev_line and next_line:
- prev_line_font_size = prev_line["spans"][0]["size"]
- prev_line_font_type = prev_line["spans"][0]["font"].lower()
- next_line_font_size = next_line["spans"][0]["size"]
- next_line_font_type = next_line["spans"][0]["font"].lower()
- return (curr_line_font_size == prev_line_font_size and curr_line_font_type == prev_line_font_type) and (
- curr_line_font_size == next_line_font_size and curr_line_font_type == next_line_font_type
- )
- else:
- return False
-
- else:
- return False
-
- def _is_regular_line(self, curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width, X0, X1, avg_line_height):
- """
- This function checks if the line is a regular line
-
- Parameters
- ----------
- curr_line_bbox : list
- bbox of the current line
- prev_line_bbox : list
- bbox of the previous line
- next_line_bbox : list
- bbox of the next line
- avg_char_width : float
- average of char widths
- X0 : float
- median of x0 values, which represents the left average boundary of the page
- X1 : float
- median of x1 values, which represents the right average boundary of the page
- avg_line_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the line is a regular line, False otherwise.
- """
- horizontal_ratio = 0.5
- vertical_ratio = 0.5
- horizontal_thres = horizontal_ratio * avg_char_width
- vertical_thres = vertical_ratio * avg_line_height
-
- x0, y0, x1, y1 = curr_line_bbox
-
- x0_near_X0 = abs(x0 - X0) < horizontal_thres
- x1_near_X1 = abs(x1 - X1) < horizontal_thres
-
- prev_line_is_end_of_para = prev_line_bbox and (abs(prev_line_bbox[2] - X1) > avg_char_width)
-
- sufficient_spacing_above = False
- if prev_line_bbox:
- vertical_spacing_above = y1 - prev_line_bbox[3]
- sufficient_spacing_above = vertical_spacing_above > vertical_thres
-
- sufficient_spacing_below = False
- if next_line_bbox:
- vertical_spacing_below = next_line_bbox[1] - y0
- sufficient_spacing_below = vertical_spacing_below > vertical_thres
-
- return (
- (sufficient_spacing_above or sufficient_spacing_below)
- or (not x0_near_X0 and not x1_near_X1)
- or prev_line_is_end_of_para
- )
-
- def _is_possible_start_of_para(self, curr_line, prev_line, next_line, X0, X1, avg_char_width, avg_font_size):
- """
- This function checks if the line is a possible start of a paragraph
-
- Parameters
- ----------
- curr_line : dict
- current line
- prev_line : dict
- previous line
- next_line : dict
- next line
- X0 : float
- median of x0 values, which represents the left average boundary of the page
- X1 : float
- median of x1 values, which represents the right average boundary of the page
- avg_char_width : float
- average of char widths
- avg_line_height : float
- average of line heights
-
- Returns
- -------
- bool
- True if the line is a possible start of a paragraph, False otherwise.
- """
- start_confidence = 0.5 # Initial confidence of the line being a start of a paragraph
- decision_path = [] # Record the decision path
-
- curr_line_bbox = curr_line["bbox"]
- prev_line_bbox = prev_line["bbox"] if prev_line else None
- next_line_bbox = next_line["bbox"] if next_line else None
-
- indent_ratio = 1
-
- vertical_ratio = 1.5
- vertical_thres = vertical_ratio * avg_font_size
-
- left_horizontal_ratio = 0.5
- left_horizontal_thres = left_horizontal_ratio * avg_char_width
-
- right_horizontal_ratio = 2.5
- right_horizontal_thres = right_horizontal_ratio * avg_char_width
-
- x0, y0, x1, y1 = curr_line_bbox
-
- indent_condition = x0 > X0 + indent_ratio * avg_char_width
- if indent_condition:
- start_confidence += 0.2
- decision_path.append("indent_condition_met")
-
- x0_near_X0 = abs(x0 - X0) < left_horizontal_thres
- if x0_near_X0:
- start_confidence += 0.1
- decision_path.append("x0_near_X0")
-
- x1_near_X1 = abs(x1 - X1) < right_horizontal_thres
- if x1_near_X1:
- start_confidence += 0.1
- decision_path.append("x1_near_X1")
-
- if prev_line is None:
- prev_line_is_end_of_para = True
- start_confidence += 0.2
- decision_path.append("no_prev_line")
- else:
- prev_line_is_end_of_para, _, _ = self._is_possible_end_of_para(prev_line, next_line, X0, X1, avg_char_width)
- if prev_line_is_end_of_para:
- start_confidence += 0.1
- decision_path.append("prev_line_is_end_of_para")
-
- sufficient_spacing_above = False
- if prev_line_bbox:
- vertical_spacing_above = y1 - prev_line_bbox[3]
- sufficient_spacing_above = vertical_spacing_above > vertical_thres
- if sufficient_spacing_above:
- start_confidence += 0.2
- decision_path.append("sufficient_spacing_above")
-
- sufficient_spacing_below = False
- if next_line_bbox:
- vertical_spacing_below = next_line_bbox[1] - y0
- sufficient_spacing_below = vertical_spacing_below > vertical_thres
- if sufficient_spacing_below:
- start_confidence += 0.2
- decision_path.append("sufficient_spacing_below")
-
- is_regular_line = self._is_regular_line(
- curr_line_bbox, prev_line_bbox, next_line_bbox, avg_char_width, X0, X1, avg_font_size
- )
- if is_regular_line:
- start_confidence += 0.1
- decision_path.append("is_regular_line")
-
- is_start_of_para = (
- (sufficient_spacing_above or sufficient_spacing_below)
- or (indent_condition)
- or (not indent_condition and x0_near_X0 and x1_near_X1 and not is_regular_line)
- or prev_line_is_end_of_para
- )
- return (is_start_of_para, start_confidence, decision_path)
-
- def _is_possible_end_of_para(self, curr_line, next_line, X0, X1, avg_char_width):
- """
- This function checks if the line is a possible end of a paragraph
-
- Parameters
- ----------
- curr_line : dict
- current line
- next_line : dict
- next line
- X0 : float
- median of x0 values, which represents the left average boundary of the page
- X1 : float
- median of x1 values, which represents the right average boundary of the page
- avg_char_width : float
- average of char widths
-
- Returns
- -------
- bool
- True if the line is a possible end of a paragraph, False otherwise.
- """
-
- end_confidence = 0.5 # Initial confidence of the line being a end of a paragraph
- decision_path = [] # Record the decision path
-
- curr_line_bbox = curr_line["bbox"]
- next_line_bbox = next_line["bbox"] if next_line else None
-
- left_horizontal_ratio = 0.5
- right_horizontal_ratio = 0.5
-
- x0, _, x1, y1 = curr_line_bbox
- next_x0, next_y0, _, _ = next_line_bbox if next_line_bbox else (0, 0, 0, 0)
-
- x0_near_X0 = abs(x0 - X0) < left_horizontal_ratio * avg_char_width
- if x0_near_X0:
- end_confidence += 0.1
- decision_path.append("x0_near_X0")
-
- x1_smaller_than_X1 = x1 < X1 - right_horizontal_ratio * avg_char_width
- if x1_smaller_than_X1:
- end_confidence += 0.1
- decision_path.append("x1_smaller_than_X1")
-
- next_line_is_start_of_para = (
- next_line_bbox
- and (next_x0 > X0 + left_horizontal_ratio * avg_char_width)
- and (not is_line_left_aligned_from_neighbors(curr_line_bbox, None, next_line_bbox, avg_char_width, direction=1))
- )
- if next_line_is_start_of_para:
- end_confidence += 0.2
- decision_path.append("next_line_is_start_of_para")
-
- is_line_left_aligned_from_neighbors_bool = is_line_left_aligned_from_neighbors(
- curr_line_bbox, None, next_line_bbox, avg_char_width
- )
- if is_line_left_aligned_from_neighbors_bool:
- end_confidence += 0.1
- decision_path.append("line_is_left_aligned_from_neighbors")
-
- is_line_right_aligned_from_neighbors_bool = is_line_right_aligned_from_neighbors(
- curr_line_bbox, None, next_line_bbox, avg_char_width
- )
- if not is_line_right_aligned_from_neighbors_bool:
- end_confidence += 0.1
- decision_path.append("line_is_not_right_aligned_from_neighbors")
-
- is_end_of_para = end_with_punctuation(curr_line["text"]) and (
- (x0_near_X0 and x1_smaller_than_X1)
- or (is_line_left_aligned_from_neighbors_bool and not is_line_right_aligned_from_neighbors_bool)
- )
-
- return (is_end_of_para, end_confidence, decision_path)
-
- def _cut_paras_per_block(
- self,
- block,
- ):
- """
- Processes a raw block from PyMuPDF and returns the processed block.
-
- Parameters
- ----------
- raw_block : dict
- A raw block from pymupdf.
-
- Returns
- -------
- processed_block : dict
-
- """
-
- def _construct_para(lines, is_block_title, para_title_level):
- """
- Construct a paragraph from given lines.
- """
-
- font_sizes = [span["size"] for line in lines for span in line["spans"]]
- avg_font_size = sum(font_sizes) / len(font_sizes) if font_sizes else 0
-
- font_colors = [span["color"] for line in lines for span in line["spans"]]
- most_common_font_color = max(set(font_colors), key=font_colors.count) if font_colors else None
-
- font_type_lengths = {}
- for line in lines:
- for span in line["spans"]:
- font_type = span["font"]
- bbox_width = span["bbox"][2] - span["bbox"][0]
- if font_type in font_type_lengths:
- font_type_lengths[font_type] += bbox_width
- else:
- font_type_lengths[font_type] = bbox_width
-
- # get the font type with the longest bbox width
- most_common_font_type = max(font_type_lengths, key=font_type_lengths.get) if font_type_lengths else None # type: ignore
-
- para_bbox = calculate_para_bbox(lines)
- para_text = " ".join(line["text"] for line in lines)
-
- return {
- "para_bbox": para_bbox,
- "para_text": para_text,
- "para_font_type": most_common_font_type,
- "para_font_size": avg_font_size,
- "para_font_color": most_common_font_color,
- "is_para_title": is_block_title,
- "para_title_level": para_title_level,
- }
-
- block_bbox = block["bbox"]
- block_text = block["text"]
- block_lines = block["lines"]
-
- X0 = safe_get(block, "X0", 0)
- X1 = safe_get(block, "X1", 0)
- avg_char_width = safe_get(block, "avg_char_width", 0)
- avg_char_height = safe_get(block, "avg_char_height", 0)
- avg_font_size = safe_get(block, "avg_font_size", 0)
-
- is_block_title = safe_get(block, "is_block_title", False)
- para_title_level = safe_get(block, "block_title_level", 0)
-
- # Segment into paragraphs
- para_ranges = []
- in_paragraph = False
- start_idx_of_para = None
-
- # Create the processed paragraphs
- processed_paras = {}
- para_bboxes = []
- end_idx_of_para = 0
-
- for line_index, line in enumerate(block_lines):
- curr_line = line
- prev_line = block_lines[line_index - 1] if line_index > 0 else None
- next_line = block_lines[line_index + 1] if line_index < len(block_lines) - 1 else None
-
- """
- Start processing paragraphs.
- """
-
- # Check if the line is the start of a paragraph
- is_start_of_para, start_confidence, decision_path = self._is_possible_start_of_para(
- curr_line, prev_line, next_line, X0, X1, avg_char_width, avg_font_size
- )
- if not in_paragraph and is_start_of_para:
- in_paragraph = True
- start_idx_of_para = line_index
-
- # print_green(">>> Start of a paragraph")
- # print(" curr_line_text: ", curr_line["text"])
- # print(" start_confidence: ", start_confidence)
- # print(" decision_path: ", decision_path)
-
- # Check if the line is the end of a paragraph
- is_end_of_para, end_confidence, decision_path = self._is_possible_end_of_para(
- curr_line, next_line, X0, X1, avg_char_width
- )
- if in_paragraph and (is_end_of_para or not next_line):
- para_ranges.append((start_idx_of_para, line_index))
- start_idx_of_para = None
- in_paragraph = False
-
- # print_red(">>> End of a paragraph")
- # print(" curr_line_text: ", curr_line["text"])
- # print(" end_confidence: ", end_confidence)
- # print(" decision_path: ", decision_path)
-
- # Add the last paragraph if it is not added
- if in_paragraph and start_idx_of_para is not None:
- para_ranges.append((start_idx_of_para, len(block_lines) - 1))
-
- # Process the matched paragraphs
- for para_index, (start_idx, end_idx) in enumerate(para_ranges):
- matched_lines = block_lines[start_idx : end_idx + 1]
- para_properties = _construct_para(matched_lines, is_block_title, para_title_level)
- para_key = f"para_{len(processed_paras)}"
- processed_paras[para_key] = para_properties
- para_bboxes.append(para_properties["para_bbox"])
- end_idx_of_para = end_idx + 1
-
- # Deal with the remaining lines
- if end_idx_of_para < len(block_lines):
- unmatched_lines = block_lines[end_idx_of_para:]
- unmatched_properties = _construct_para(unmatched_lines, is_block_title, para_title_level)
- unmatched_key = f"para_{len(processed_paras)}"
- processed_paras[unmatched_key] = unmatched_properties
- para_bboxes.append(unmatched_properties["para_bbox"])
-
- block["paras"] = processed_paras
-
- return block
-
- def batch_process_blocks(self, pdf_dict):
- """
- Parses the blocks of all pages.
-
- Parameters
- ----------
- pdf_dict : dict
- PDF dictionary.
- filter_blocks : list
- List of bounding boxes to filter.
-
- Returns
- -------
- result_dict : dict
- Result dictionary.
-
- """
-
- num_paras = 0
-
- for page_id, page in pdf_dict.items():
- if page_id.startswith("page_"):
- para_blocks = []
- if "para_blocks" in page.keys():
- input_blocks = page["para_blocks"]
- for input_block in input_blocks:
- new_block = self._cut_paras_per_block(input_block)
- para_blocks.append(new_block)
- num_paras += len(new_block["paras"])
-
- page["para_blocks"] = para_blocks
-
- pdf_dict["statistics"]["num_paras"] = num_paras
- return pdf_dict
-
-
-class BlockContinuationProcessor:
- """
- This class is used to process the blocks to detect block continuations.
- """
-
- def __init__(self) -> None:
- pass
-
- def __is_similar_font_type(self, font_type_1, font_type_2, prefix_length_ratio=0.3):
- """
- This function checks if the two font types are similar.
- Definition of similar font types: the two font types have a common prefix,
- and the length of the common prefix is at least a certain ratio of the length of the shorter font type.
-
- Parameters
- ----------
- font_type1 : str
- font type 1
- font_type2 : str
- font type 2
- prefix_length_ratio : float
- minimum ratio of the common prefix length to the length of the shorter font type
-
- Returns
- -------
- bool
- True if the two font types are similar, False otherwise.
- """
-
- if isinstance(font_type_1, list):
- font_type_1 = font_type_1[0] if font_type_1 else ""
- if isinstance(font_type_2, list):
- font_type_2 = font_type_2[0] if font_type_2 else ""
-
- if font_type_1 == font_type_2:
- return True
-
- # Find the length of the common prefix
- common_prefix_length = len(os.path.commonprefix([font_type_1, font_type_2]))
-
- # Calculate the minimum prefix length based on the ratio
- min_prefix_length = int(min(len(font_type_1), len(font_type_2)) * prefix_length_ratio)
-
- return common_prefix_length >= min_prefix_length
-
- def __is_same_block_font(self, block_1, block_2):
- """
- This function compares the font of block1 and block2
-
- Parameters
- ----------
- block1 : dict
- block1
- block2 : dict
- block2
-
- Returns
- -------
- is_same : bool
- True if block1 and block2 have the same font, else False
- """
- block_1_font_type = safe_get(block_1, "block_font_type", "")
- block_1_font_size = safe_get(block_1, "block_font_size", 0)
- block_1_avg_char_width = safe_get(block_1, "avg_char_width", 0)
-
- block_2_font_type = safe_get(block_2, "block_font_type", "")
- block_2_font_size = safe_get(block_2, "block_font_size", 0)
- block_2_avg_char_width = safe_get(block_2, "avg_char_width", 0)
-
- if isinstance(block_1_font_size, list):
- block_1_font_size = block_1_font_size[0] if block_1_font_size else 0
- if isinstance(block_2_font_size, list):
- block_2_font_size = block_2_font_size[0] if block_2_font_size else 0
-
- block_1_text = safe_get(block_1, "text", "")
- block_2_text = safe_get(block_2, "text", "")
-
- if block_1_avg_char_width == 0 or block_2_avg_char_width == 0:
- return False
-
- if not block_1_text or not block_2_text:
- return False
- else:
- text_len_ratio = len(block_2_text) / len(block_1_text)
- if text_len_ratio < 0.2:
- avg_char_width_condition = (
- abs(block_1_avg_char_width - block_2_avg_char_width) / min(block_1_avg_char_width, block_2_avg_char_width)
- < 0.5
- )
- else:
- avg_char_width_condition = (
- abs(block_1_avg_char_width - block_2_avg_char_width) / min(block_1_avg_char_width, block_2_avg_char_width)
- < 0.2
- )
-
- block_font_size_condition = abs(block_1_font_size - block_2_font_size) < 1
-
- return (
- self.__is_similar_font_type(block_1_font_type, block_2_font_type)
- and avg_char_width_condition
- and block_font_size_condition
- )
-
- def _is_alphabet_char(self, char):
- if (char >= "\u0041" and char <= "\u005a") or (char >= "\u0061" and char <= "\u007a"):
- return True
- else:
- return False
-
- def _is_chinese_char(self, char):
- if char >= "\u4e00" and char <= "\u9fa5":
- return True
- else:
- return False
-
- def _is_other_letter_char(self, char):
- try:
- cat = unicodedata.category(char)
- if cat == "Lu" or cat == "Ll":
- return not self._is_alphabet_char(char) and not self._is_chinese_char(char)
- except TypeError:
- print("The input to the function must be a single character.")
- return False
-
- def _is_year(self, s: str):
- try:
- number = int(s)
- return 1900 <= number <= 2099
- except ValueError:
- return False
-
- def _match_brackets(self, text):
- # pattern = r"^[\(\)\[\]（）【】{}｛｝<>＜＞〔〕〘〙\"\'“”‘’]"
- pattern = r"^[\(\)\]（）】{}｛｝>＞〕〙\"\'“”‘’]"
- return bool(re.match(pattern, text))
-
- def _is_para_font_consistent(self, para_1, para_2):
- """
- This function compares the font of para1 and para2
-
- Parameters
- ----------
- para1 : dict
- para1
- para2 : dict
- para2
-
- Returns
- -------
- is_same : bool
- True if para1 and para2 have the same font, else False
- """
- if para_1 is None or para_2 is None:
- return False
-
- para_1_font_type = safe_get(para_1, "para_font_type", "")
- para_1_font_size = safe_get(para_1, "para_font_size", 0)
- para_1_font_color = safe_get(para_1, "para_font_color", "")
-
- para_2_font_type = safe_get(para_2, "para_font_type", "")
- para_2_font_size = safe_get(para_2, "para_font_size", 0)
- para_2_font_color = safe_get(para_2, "para_font_color", "")
-
- if isinstance(para_1_font_type, list): # get the most common font type
- para_1_font_type = max(set(para_1_font_type), key=para_1_font_type.count)
- if isinstance(para_2_font_type, list):
- para_2_font_type = max(set(para_2_font_type), key=para_2_font_type.count)
- if isinstance(para_1_font_size, list): # compute average font type
- para_1_font_size = sum(para_1_font_size) / len(para_1_font_size)
- if isinstance(para_2_font_size, list): # compute average font type
- para_2_font_size = sum(para_2_font_size) / len(para_2_font_size)
-
- return (
- self.__is_similar_font_type(para_1_font_type, para_2_font_type)
- and abs(para_1_font_size - para_2_font_size) < 1.5
- # and para_font_color1 == para_font_color2
- )
-
- def _is_para_puncs_consistent(self, para_1, para_2):
- """
- This function determines whether para1 and para2 are originally from the same paragraph by checking the puncs of para1(former) and para2(latter)
-
- Parameters
- ----------
- para1 : dict
- para1
- para2 : dict
- para2
-
- Returns
- -------
- is_same : bool
- True if para1 and para2 are from the same paragraph by using the puncs, else False
- """
- para_1_text = safe_get(para_1, "para_text", "").strip()
- para_2_text = safe_get(para_2, "para_text", "").strip()
-
- para_1_bboxes = safe_get(para_1, "para_bbox", [])
- para_1_font_sizes = safe_get(para_1, "para_font_size", 0)
-
- para_2_bboxes = safe_get(para_2, "para_bbox", [])
- para_2_font_sizes = safe_get(para_2, "para_font_size", 0)
-
- # print_yellow(" Features of determine puncs_consistent:")
- # print(f" para_1_text: {para_1_text}")
- # print(f" para_2_text: {para_2_text}")
- # print(f" para_1_bboxes: {para_1_bboxes}")
- # print(f" para_2_bboxes: {para_2_bboxes}")
- # print(f" para_1_font_sizes: {para_1_font_sizes}")
- # print(f" para_2_font_sizes: {para_2_font_sizes}")
-
- if is_nested_list(para_1_bboxes):
- x0_1, y0_1, x1_1, y1_1 = para_1_bboxes[-1]
- else:
- x0_1, y0_1, x1_1, y1_1 = para_1_bboxes
-
- if is_nested_list(para_2_bboxes):
- x0_2, y0_2, x1_2, y1_2 = para_2_bboxes[0]
- para_2_font_sizes = para_2_font_sizes[0] # type: ignore
- else:
- x0_2, y0_2, x1_2, y1_2 = para_2_bboxes
-
- right_align_threshold = 0.5 * (para_1_font_sizes + para_2_font_sizes) * 0.8
- are_two_paras_right_aligned = abs(x1_1 - x1_2) < right_align_threshold
-
- left_indent_threshold = 0.5 * (para_1_font_sizes + para_2_font_sizes) * 0.8
- is_para1_left_indent_than_papa2 = x0_1 - x0_2 > left_indent_threshold
- is_para2_left_indent_than_papa1 = x0_2 - x0_1 > left_indent_threshold
-
- # Check if either para_text1 or para_text2 is empty
- if not para_1_text or not para_2_text:
- return False
-
- # Define the end puncs for a sentence to end and hyphen
- end_puncs = [".", "?", "!", "。", "？", "！", "…"]
- hyphen = ["-", "—"]
-
- # Check if para_text1 ends with either hyphen or non-end punctuation or spaces
- para_1_end_with_hyphen = para_1_text and para_1_text[-1] in hyphen
- para_1_end_with_end_punc = para_1_text and para_1_text[-1] in end_puncs
- para_1_end_with_space = para_1_text and para_1_text[-1] == " "
- para_1_not_end_with_end_punc = para_1_text and para_1_text[-1] not in end_puncs
-
- # print_yellow(f" para_1_end_with_hyphen: {para_1_end_with_hyphen}")
- # print_yellow(f" para_1_end_with_end_punc: {para_1_end_with_end_punc}")
- # print_yellow(f" para_1_not_end_with_end_punc: {para_1_not_end_with_end_punc}")
- # print_yellow(f" para_1_end_with_space: {para_1_end_with_space}")
-
- if para_1_end_with_hyphen: # If para_text1 ends with hyphen
- # print_red(f"para_1 is end with hyphen.")
- para_2_is_consistent = para_2_text and (
- para_2_text[0] in hyphen
- or (self._is_alphabet_char(para_2_text[0]) and para_2_text[0].islower())
- or (self._is_chinese_char(para_2_text[0]))
- or (self._is_other_letter_char(para_2_text[0]))
- )
- if para_2_is_consistent:
- # print(f"para_2 is consistent.\n")
- return True
- else:
- # print(f"para_2 is not consistent.\n")
- pass
-
- elif para_1_end_with_end_punc: # If para_text1 ends with ending punctuations
- # print_red(f"para_1 is end with end_punc.")
- para_2_is_consistent = (
- para_2_text
- and (
- para_2_text[0]
- == " "
- # or (self._is_alphabet_char(para_2_text[0]) and para_2_text[0].isupper())
- # or (self._is_chinese_char(para_2_text[0]))
- # or (self._is_other_letter_char(para_2_text[0]))
- )
- and not is_para2_left_indent_than_papa1
- )
- if para_2_is_consistent:
- # print(f"para_2 is consistent.\n")
- return True
- else:
- # print(f"para_2 is not consistent.\n")
- pass
-
- elif para_1_not_end_with_end_punc: # If para_text1 is not end with ending punctuations
- # print_red(f"para_1 is NOT end with end_punc.")
- para_2_is_consistent = para_2_text and (
- para_2_text[0] == " "
- or (self._is_alphabet_char(para_2_text[0]) and para_2_text[0].islower())
- or (self._is_alphabet_char(para_2_text[0]))
- or (self._is_year(para_2_text[0:4]))
- or (are_two_paras_right_aligned or is_para1_left_indent_than_papa2)
- or (self._is_chinese_char(para_2_text[0]))
- or (self._is_other_letter_char(para_2_text[0]))
- or (self._match_brackets(para_2_text[0]))
- )
- if para_2_is_consistent:
- # print(f"para_2 is consistent.\n")
- return True
- else:
- # print(f"para_2 is not consistent.\n")
- pass
-
- elif para_1_end_with_space: # If para_text1 ends with space
- # print_red(f"para_1 is end with space.")
- para_2_is_consistent = para_2_text and (
- para_2_text[0] == " "
- or (self._is_alphabet_char(para_2_text[0]) and para_2_text[0].islower())
- or (self._is_chinese_char(para_2_text[0]))
- or (self._is_other_letter_char(para_2_text[0]))
- )
- if para_2_is_consistent:
- # print(f"para_2 is consistent.\n")
- return True
- else:
- pass
- # print(f"para_2 is not consistent.\n")
-
- return False
-
- def _is_block_consistent(self, block_1, block_2):
- """
- This function determines whether block1 and block2 are originally from the same block
-
- Parameters
- ----------
- block1 : dict
- block1s
- block2 : dict
- block2
-
- Returns
- -------
- is_same : bool
- True if block1 and block2 are from the same block, else False
- """
- return self.__is_same_block_font(block_1, block_2)
-
- def _is_para_continued(self, para_1, para_2):
- """
- This function determines whether para1 and para2 are originally from the same paragraph
-
- Parameters
- ----------
- para1 : dict
- para1
- para2 : dict
- para2
-
- Returns
- -------
- is_same : bool
- True if para1 and para2 are from the same paragraph, else False
- """
- is_para_font_consistent = self._is_para_font_consistent(para_1, para_2)
- is_para_puncs_consistent = self._is_para_puncs_consistent(para_1, para_2)
-
- return is_para_font_consistent and is_para_puncs_consistent
-
- def _are_boundaries_of_block_consistent(self, block_1, block_2):
- """
- This function checks if the boundaries of block1 and block2 are consistent
-
- Parameters
- ----------
- block1 : dict
- block1
-
- block2 : dict
- block2
-
- Returns
- -------
- is_consistent : bool
- True if the boundaries of block1 and block2 are consistent, else False
- """
-
- last_line_of_block_1 = block_1["lines"][-1]
- first_line_of_block_2 = block_2["lines"][0]
-
- spans_of_last_line_of_block_1 = last_line_of_block_1["spans"]
- spans_of_first_line_of_block_2 = first_line_of_block_2["spans"]
-
- font_type_of_last_line_of_block_1 = spans_of_last_line_of_block_1[0]["font"].lower()
- font_size_of_last_line_of_block_1 = spans_of_last_line_of_block_1[0]["size"]
- font_color_of_last_line_of_block_1 = spans_of_last_line_of_block_1[0]["color"]
- font_flags_of_last_line_of_block_1 = spans_of_last_line_of_block_1[0]["flags"]
-
- font_type_of_first_line_of_block_2 = spans_of_first_line_of_block_2[0]["font"].lower()
- font_size_of_first_line_of_block_2 = spans_of_first_line_of_block_2[0]["size"]
- font_color_of_first_line_of_block_2 = spans_of_first_line_of_block_2[0]["color"]
- font_flags_of_first_line_of_block_2 = spans_of_first_line_of_block_2[0]["flags"]
-
- return (
- self.__is_similar_font_type(font_type_of_last_line_of_block_1, font_type_of_first_line_of_block_2)
- and abs(font_size_of_last_line_of_block_1 - font_size_of_first_line_of_block_2) < 1
- # and font_color_of_last_line_of_block1 == font_color_of_first_line_of_block2
- and font_flags_of_last_line_of_block_1 == font_flags_of_first_line_of_block_2
- )
-
- def should_merge_next_para(self, curr_para, next_para):
- """
- This function checks if the next_para should be merged into the curr_para.
-
- Parameters
- ----------
- curr_para : dict
- The current paragraph.
- next_para : dict
- The next paragraph.
-
- Returns
- -------
- bool
- True if the next_para should be merged into the curr_para, False otherwise.
- """
- if self._is_para_continued(curr_para, next_para):
- return True
- else:
- return False
-
- def batch_tag_paras(self, pdf_dict):
- """
- This function tags the paragraphs in the pdf_dict.
-
- Parameters
- ----------
- pdf_dict : dict
- PDF dictionary.
-
- Returns
- -------
- pdf_dict : dict
- PDF dictionary with tagged paragraphs.
- """
- the_last_page_id = len(pdf_dict) - 1
-
- for curr_page_idx, (curr_page_id, curr_page_content) in enumerate(pdf_dict.items()):
- if curr_page_id.startswith("page_") and curr_page_content.get("para_blocks", []):
- para_blocks_of_curr_page = curr_page_content["para_blocks"]
- next_page_idx = curr_page_idx + 1
- next_page_id = f"page_{next_page_idx}"
- next_page_content = pdf_dict.get(next_page_id, {})
-
- for i, current_block in enumerate(para_blocks_of_curr_page):
- for para_id, curr_para in current_block["paras"].items():
- curr_para["curr_para_location"] = [
- curr_page_idx,
- current_block["block_id"],
- int(para_id.split("_")[-1]),
- ]
- curr_para["next_para_location"] = None # 默认设置为None
- curr_para["merge_next_para"] = False # 默认设置为False
-
- next_block = para_blocks_of_curr_page[i + 1] if i < len(para_blocks_of_curr_page) - 1 else None
-
- if next_block:
- curr_block_last_para_key = list(current_block["paras"].keys())[-1]
- curr_blk_last_para = current_block["paras"][curr_block_last_para_key]
-
- next_block_first_para_key = list(next_block["paras"].keys())[0]
- next_blk_first_para = next_block["paras"][next_block_first_para_key]
-
- if self.should_merge_next_para(curr_blk_last_para, next_blk_first_para):
- curr_blk_last_para["next_para_location"] = [
- curr_page_idx,
- next_block["block_id"],
- int(next_block_first_para_key.split("_")[-1]),
- ]
- curr_blk_last_para["merge_next_para"] = True
- else:
- # Handle the case where the next block is in a different page
- curr_block_last_para_key = list(current_block["paras"].keys())[-1]
- curr_blk_last_para = current_block["paras"][curr_block_last_para_key]
-
- while not next_page_content.get("para_blocks", []) and next_page_idx <= the_last_page_id:
- next_page_idx += 1
- next_page_id = f"page_{next_page_idx}"
- next_page_content = pdf_dict.get(next_page_id, {})
-
- if next_page_content.get("para_blocks", []):
- next_blk_first_para_key = list(next_page_content["para_blocks"][0]["paras"].keys())[0]
- next_blk_first_para = next_page_content["para_blocks"][0]["paras"][next_blk_first_para_key]
-
- if self.should_merge_next_para(curr_blk_last_para, next_blk_first_para):
- curr_blk_last_para["next_para_location"] = [
- next_page_idx,
- next_page_content["para_blocks"][0]["block_id"],
- int(next_blk_first_para_key.split("_")[-1]),
- ]
- curr_blk_last_para["merge_next_para"] = True
-
- return pdf_dict
-
- def find_block_by_id(self, para_blocks, block_id):
- """
- This function finds a block by its id.
-
- Parameters
- ----------
- para_blocks : list
- List of blocks.
- block_id : int
- Id of the block to find.
-
- Returns
- -------
- block : dict
- The block with the given id.
- """
- for blk_idx, block in enumerate(para_blocks):
- if block.get("block_id") == block_id:
- return block
- return None
-
- def batch_merge_paras(self, pdf_dict):
- """
- This function merges the paragraphs in the pdf_dict.
-
- Parameters
- ----------
- pdf_dict : dict
- PDF dictionary.
-
- Returns
- -------
- pdf_dict : dict
- PDF dictionary with merged paragraphs.
- """
- for page_id, page_content in pdf_dict.items():
- if page_id.startswith("page_") and page_content.get("para_blocks", []):
- para_blocks_of_page = page_content["para_blocks"]
-
- for i in range(len(para_blocks_of_page)):
- current_block = para_blocks_of_page[i]
- paras = current_block["paras"]
-
- for para_id, curr_para in list(paras.items()):
- # print(f"current para_id: {para_id}")
- # 跳过标题段落
- if curr_para.get("is_para_title"):
- continue
-
- while curr_para.get("merge_next_para"):
- curr_para_location = curr_para.get("curr_para_location")
- next_para_location = curr_para.get("next_para_location")
-
- # print(f"curr_para_location: {curr_para_location}, next_para_location: {next_para_location}")
-
- if not next_para_location:
- break
-
- if curr_para_location == next_para_location:
- # print_red("The next para is in the same block as the current para.")
- curr_para["merge_next_para"] = False
- break
-
- next_page_idx, next_block_id, next_para_id = next_para_location
- next_page_id = f"page_{next_page_idx}"
- next_page_content = pdf_dict.get(next_page_id)
- if not next_page_content:
- break
-
- next_block = self.find_block_by_id(next_page_content.get("para_blocks", []), next_block_id)
-
- if not next_block:
- break
-
- next_para = next_block["paras"].get(f"para_{next_para_id}")
-
- if not next_para or next_para.get("is_para_title"):
- break
-
- # 合并段落文本
- curr_para_text = curr_para.get("para_text", "")
- next_para_text = next_para.get("para_text", "")
- curr_para["para_text"] = curr_para_text + " " + next_para_text
-
- # 更新 next_para_location
- curr_para["next_para_location"] = next_para.get("next_para_location")
-
- # 将下一个段落文本置为空，表示已被合并
- next_para["para_text"] = ""
-
- # 更新 merge_next_para 标记
- curr_para["merge_next_para"] = next_para.get("merge_next_para", False)
-
- return pdf_dict
-
-
-class DrawAnnos:
- """
- This class draws annotations on the pdf file
-
- ----------------------------------------
- Color Code
- ----------------------------------------
- Red: (1, 0, 0)
- Green: (0, 1, 0)
- Blue: (0, 0, 1)
- Yellow: (1, 1, 0) - mix of red and green
- Cyan: (0, 1, 1) - mix of green and blue
- Magenta: (1, 0, 1) - mix of red and blue
- White: (1, 1, 1) - red, green and blue full intensity
- Black: (0, 0, 0) - no color component whatsoever
- Gray: (0.5, 0.5, 0.5) - equal and medium intensity of red, green and blue color components
- Orange: (1, 0.65, 0) - maximum intensity of red, medium intensity of green, no blue component
- """
-
- def __init__(self) -> None:
- pass
-
- def __is_nested_list(self, lst):
- """
- This function returns True if the given list is a nested list of any degree.
- """
- if isinstance(lst, list):
- return any(self.__is_nested_list(i) for i in lst) or any(isinstance(i, list) for i in lst)
- return False
-
- def __valid_rect(self, bbox):
- # Ensure that the rectangle is not empty or invalid
- if isinstance(bbox[0], list):
- return False # It's a nested list, hence it can't be valid rect
- else:
- return bbox[0] < bbox[2] and bbox[1] < bbox[3]
-
- def __draw_nested_boxes(self, page, nested_bbox, color=(0, 1, 1)):
- """
- This function draws the nested boxes
-
- Parameters
- ----------
- page : fitz.Page
- page
- nested_bbox : list
- nested bbox
- color : tuple
- color, by default (0, 1, 1) # draw with cyan color for combined paragraph
- """
- if self.__is_nested_list(nested_bbox): # If it's a nested list
- for bbox in nested_bbox:
- self.__draw_nested_boxes(page, bbox, color) # Recursively call the function
- elif self.__valid_rect(nested_bbox): # If valid rectangle
- para_rect = fitz.Rect(nested_bbox)
- para_anno = page.add_rect_annot(para_rect)
- para_anno.set_colors(stroke=color) # draw with cyan color for combined paragraph
- para_anno.set_border(width=1)
- para_anno.update()
-
- def draw_annos(self, input_pdf_path, pdf_dic, output_pdf_path):
- """
- This function draws annotations on the pdf file.
-
- Parameters
- ----------
- input_pdf_path : str
- path to the input pdf file
- pdf_dic : dict
- pdf dictionary
- output_pdf_path : str
- path to the output pdf file
-
- pdf_dic : dict
- pdf dictionary
- """
- pdf_doc = open_pdf(input_pdf_path)
-
- if pdf_dic is None:
- pdf_dic = {}
-
- if output_pdf_path is None:
- output_pdf_path = input_pdf_path.replace(".pdf", "_anno.pdf")
-
- for page_id, page in enumerate(pdf_doc): # type: ignore
- page_key = f"page_{page_id}"
- for ele_key, ele_data in pdf_dic[page_key].items():
- if ele_key == "para_blocks":
- para_blocks = ele_data
- for para_block in para_blocks:
- if "paras" in para_block.keys():
- paras = para_block["paras"]
- for para_key, para_content in paras.items():
- para_bbox = para_content["para_bbox"]
- # print(f"para_bbox: {para_bbox}")
- # print(f"is a nested list: {self.__is_nested_list(para_bbox)}")
- if self.__is_nested_list(para_bbox) and len(para_bbox) > 1:
- color = (0, 1, 1)
- self.__draw_nested_boxes(
- page, para_bbox, color
- ) # draw with cyan color for combined paragraph
- else:
- if self.__valid_rect(para_bbox):
- para_rect = fitz.Rect(para_bbox)
- para_anno = page.add_rect_annot(para_rect)
- para_anno.set_colors(stroke=(0, 1, 0)) # draw with green color for normal paragraph
- para_anno.set_border(width=0.5)
- para_anno.update()
-
- is_para_title = para_content["is_para_title"]
- if is_para_title:
- if self.__is_nested_list(para_content["para_bbox"]) and len(para_content["para_bbox"]) > 1:
- color = (0, 0, 1)
- self.__draw_nested_boxes(
- page, para_content["para_bbox"], color
- ) # draw with cyan color for combined title
- else:
- if self.__valid_rect(para_content["para_bbox"]):
- para_rect = fitz.Rect(para_content["para_bbox"])
- if self.__valid_rect(para_content["para_bbox"]):
- para_anno = page.add_rect_annot(para_rect)
- para_anno.set_colors(stroke=(0, 0, 1)) # draw with blue color for normal title
- para_anno.set_border(width=0.5)
- para_anno.update()
-
- pdf_doc.save(output_pdf_path)
- pdf_doc.close()
-
-
-class ParaProcessPipeline:
- def __init__(self) -> None:
- pass
-
- def para_process_pipeline(self, pdf_info_dict, para_debug_mode=None, input_pdf_path=None, output_pdf_path=None):
- """
- This function processes the paragraphs, including:
- 1. Read raw input json file into pdf_dic
- 2. Detect and replace equations
- 3. Combine spans into a natural line
- 4. Check if the paragraphs are inside bboxes passed from "layout_bboxes" key
- 5. Compute statistics for each block
- 6. Detect titles in the document
- 7. Detect paragraphs inside each block
- 8. Divide the level of the titles
- 9. Detect and combine paragraphs from different blocks into one paragraph
- 10. Check whether the final results after checking headings, dividing paragraphs within blocks, and merging paragraphs between blocks are plausible and reasonable.
- 11. Draw annotations on the pdf file
-
- Parameters
- ----------
- pdf_dic_json_fpath : str
- path to the pdf dictionary json file.
- Notice: data noises, including overlap blocks, header, footer, watermark, vertical margin note have been removed already.
- input_pdf_doc : str
- path to the input pdf file
- output_pdf_path : str
- path to the output pdf file
-
- Returns
- -------
- pdf_dict : dict
- result dictionary
- """
-
- error_info = None
-
- output_json_file = ""
- output_dir = ""
-
- if input_pdf_path is not None:
- input_pdf_path = os.path.abspath(input_pdf_path)
-
- # print_green_on_red(f">>>>>>>>>>>>>>>>>>> Process the paragraphs of {input_pdf_path}")
-
- if output_pdf_path is not None:
- output_dir = os.path.dirname(output_pdf_path)
- output_json_file = f"{output_dir}/pdf_dic.json"
-
- def __save_pdf_dic(pdf_dic, output_pdf_path, stage="0", para_debug_mode=para_debug_mode):
- """
- Save the pdf_dic to a json file
- """
- output_pdf_file_name = os.path.basename(output_pdf_path)
- # output_dir = os.path.dirname(output_pdf_path)
- output_dir = "\\tmp\\pdf_parse"
- output_pdf_file_name = output_pdf_file_name.replace(".pdf", f"_stage_{stage}.json")
- pdf_dic_json_fpath = os.path.join(output_dir, output_pdf_file_name)
-
- if not os.path.exists(output_dir):
- os.makedirs(output_dir)
-
- if para_debug_mode == "full":
- with open(pdf_dic_json_fpath, "w", encoding="utf-8") as f:
- json.dump(pdf_dic, f, indent=2, ensure_ascii=False)
-
- # Validate the output already exists
- if not os.path.exists(pdf_dic_json_fpath):
- print_red(f"Failed to save the pdf_dic to {pdf_dic_json_fpath}")
- return None
- else:
- print_green(f"Succeed to save the pdf_dic to {pdf_dic_json_fpath}")
-
- return pdf_dic_json_fpath
-
- """
- Preprocess the lines of block
- """
- # Combine spans into a natural line
- rawBlockProcessor = RawBlockProcessor()
- pdf_dic = rawBlockProcessor.batch_process_blocks(pdf_info_dict)
- # print(f"pdf_dic['page_0']['para_blocks'][0]: {pdf_dic['page_0']['para_blocks'][0]}", end="\n\n")
-
- # Check if the paragraphs are inside bboxes passed from "layout_bboxes" key
- layoutFilter = LayoutFilterProcessor()
- pdf_dic = layoutFilter.batch_process_blocks(pdf_dic)
-
- # Compute statistics for each block
- blockStatisticsCalculator = BlockStatisticsCalculator()
- pdf_dic = blockStatisticsCalculator.batch_process_blocks(pdf_dic)
- # print(f"pdf_dic['page_0']['para_blocks'][0]: {pdf_dic['page_0']['para_blocks'][0]}", end="\n\n")
-
- # Compute statistics for all blocks(namely this pdf document)
- docStatisticsCalculator = DocStatisticsCalculator()
- pdf_dic = docStatisticsCalculator.calc_stats_of_doc(pdf_dic)
- # print(f"pdf_dic['statistics']: {pdf_dic['statistics']}", end="\n\n")
-
- # Dump the first three stages of pdf_dic to a json file
- if para_debug_mode == "full":
- pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="0", para_debug_mode=para_debug_mode)
-
- """
- Detect titles in the document
- """
- doc_statistics = pdf_dic["statistics"]
- titleProcessor = TitleProcessor(doc_statistics)
- pdf_dic = titleProcessor.batch_detect_titles(pdf_dic)
-
- if para_debug_mode == "full":
- pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="1", para_debug_mode=para_debug_mode)
-
- """
- Detect and divide the level of the titles
- """
- titleProcessor = TitleProcessor()
-
- pdf_dic = titleProcessor.batch_recog_title_level(pdf_dic)
-
- if para_debug_mode == "full":
- pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="2", para_debug_mode=para_debug_mode)
-
- """
- Detect and split paragraphs inside each block
- """
- blockInnerParasProcessor = BlockTerminationProcessor()
-
- pdf_dic = blockInnerParasProcessor.batch_process_blocks(pdf_dic)
-
- if para_debug_mode == "full":
- pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="3", para_debug_mode=para_debug_mode)
-
- # pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="3", para_debug_mode="full")
- # print_green(f"pdf_dic_json_fpath: {pdf_dic_json_fpath}")
-
- """
- Detect and combine paragraphs from different blocks into one paragraph
- """
- blockContinuationProcessor = BlockContinuationProcessor()
-
- pdf_dic = blockContinuationProcessor.batch_tag_paras(pdf_dic)
- pdf_dic = blockContinuationProcessor.batch_merge_paras(pdf_dic)
-
- if para_debug_mode == "full":
- pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="4", para_debug_mode=para_debug_mode)
-
- # pdf_dic_json_fpath = __save_pdf_dic(pdf_dic, output_pdf_path, stage="4", para_debug_mode="full")
- # print_green(f"pdf_dic_json_fpath: {pdf_dic_json_fpath}")
-
- """
- Discard pdf files by checking exceptions and return the error info to the caller
- """
- discardByException = DiscardByException()
-
- is_discard_by_single_line_block = discardByException.discard_by_single_line_block(
- pdf_dic, exception=DenseSingleLineBlockException()
- )
- is_discard_by_title_detection = discardByException.discard_by_title_detection(
- pdf_dic, exception=TitleDetectionException()
- )
- is_discard_by_title_level = discardByException.discard_by_title_level(pdf_dic, exception=TitleLevelException())
- is_discard_by_split_para = discardByException.discard_by_split_para(pdf_dic, exception=ParaSplitException())
- is_discard_by_merge_para = discardByException.discard_by_merge_para(pdf_dic, exception=ParaMergeException())
-
- if is_discard_by_single_line_block is not None:
- error_info = is_discard_by_single_line_block
- elif is_discard_by_title_detection is not None:
- error_info = is_discard_by_title_detection
- elif is_discard_by_title_level is not None:
- error_info = is_discard_by_title_level
- elif is_discard_by_split_para is not None:
- error_info = is_discard_by_split_para
- elif is_discard_by_merge_para is not None:
- error_info = is_discard_by_merge_para
-
- if error_info is not None:
- return pdf_dic, error_info
-
- """
- Dump the final pdf_dic to a json file
- """
- if para_debug_mode is not None:
- with open(output_json_file, "w", encoding="utf-8") as f:
- json.dump(pdf_info_dict, f, ensure_ascii=False, indent=4)
-
- """
- Draw the annotations
- """
- if para_debug_mode is not None:
- drawAnnos = DrawAnnos()
- drawAnnos.draw_annos(input_pdf_path, pdf_dic, output_pdf_path)
-
- """
- Remove the intermediate files which are generated in the process of paragraph processing if debug_mode is simple
- """
- if para_debug_mode is not None:
- for fpath in os.listdir(output_dir):
- if fpath.endswith(".json") and "stage" in fpath:
- os.remove(os.path.join(output_dir, fpath))
-
- return pdf_dic, error_info
-
-
-"""
-Run this script to test the function with Command:
-
-python detect_para.py [pdf_path] [output_pdf_path]
-
-Params:
-- pdf_path: the path of the pdf file
-- output_pdf_path: the path of the output pdf file
-"""
-
-if __name__ == "__main__":
- DEFAULT_PDF_PATH = (
- "app/pdf_toolbox/tests/assets/paper/paper.pdf" if os.name != "nt" else "app\\pdf_toolbox\\tests\\assets\\paper\\paper.pdf"
- )
- input_pdf_path = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_PDF_PATH
- output_pdf_path = sys.argv[2] if len(sys.argv) > 2 else input_pdf_path.split(".")[0] + "_recogPara.pdf"
- output_json_path = sys.argv[3] if len(sys.argv) > 3 else input_pdf_path.split(".")[0] + "_recogPara.json"
-
- import stat
-
- # Remove existing output file if it exists
- if os.path.exists(output_pdf_path):
- os.chmod(output_pdf_path, stat.S_IWRITE)
- os.remove(output_pdf_path)
-
- input_pdf_doc = open_pdf(input_pdf_path)
-
- # postprocess the paragraphs
- paraProcessPipeline = ParaProcessPipeline()
-
- # parse paragraph and save to json file
- pdf_dic = {}
-
- blockInnerParasProcessor = BlockTerminationProcessor()
-
- """
- Construct the pdf dictionary.
- """
-
- for page_id, page in enumerate(input_pdf_doc): # type: ignore
- # print(f"Processing page {page_id}")
- # print(f"page: {page}")
- raw_blocks = page.get_text("dict")["blocks"]
-
- # Save text blocks to "preproc_blocks"
- preproc_blocks = []
- for block in raw_blocks:
- if block["type"] == 0:
- preproc_blocks.append(block)
-
- layout_bboxes = []
-
- # Construct the pdf dictionary as schema above
- page_dict = {
- "para_blocks": None,
- "preproc_blocks": preproc_blocks,
- "images": None,
- "tables": None,
- "interline_equations": None,
- "inline_equations": None,
- "layout_bboxes": None,
- "pymu_raw_blocks": None,
- "global_statistic": None,
- "droped_text_block": None,
- "droped_image_block": None,
- "droped_table_block": None,
- "image_backup": None,
- "table_backup": None,
- }
-
- pdf_dic[f"page_{page_id}"] = page_dict
-
- # print(f"pdf_dic: {pdf_dic}")
-
- with open(output_json_path, "w", encoding="utf-8") as f:
- json.dump(pdf_dic, f, ensure_ascii=False, indent=4)
-
- pdf_dic = paraProcessPipeline.para_process_pipeline(output_json_path, input_pdf_doc, output_pdf_path)
diff --git a/magic_pdf/post_proc/pdf_post_filter.py b/magic_pdf/post_proc/pdf_post_filter.py
deleted file mode 100644
index 862e66607..000000000
--- a/magic_pdf/post_proc/pdf_post_filter.py
+++ /dev/null
@@ -1,67 +0,0 @@
-from loguru import logger
-
-from magic_pdf.layout.layout_sort import get_columns_cnt_of_layout
-from magic_pdf.libs.drop_reason import DropReason
-
-
-def __is_pseudo_single_column(page_info) -> bool:
- """
- 判断一个页面是否伪单列。
-
- Args:
- page_info (dict): 页面信息字典，包括'_layout_tree'和'preproc_blocks'。
-
- Returns:
- Tuple[bool, Optional[str]]: 如果页面伪单列返回(True, extra_info)，否则返回(False, None)。
-
- """
- layout_tree = page_info['_layout_tree']
- layout_column_width = get_columns_cnt_of_layout(layout_tree)
- if layout_column_width == 1:
- text_blocks = page_info['preproc_blocks']
- # 遍历每一个text_block
- for text_block in text_blocks:
- lines = text_block['lines']
- num_lines = len(lines)
- num_satisfying_lines = 0
-
- for i in range(num_lines - 1):
- current_line = lines[i]
- next_line = lines[i + 1]
-
- # 获取当前line和下一个line的bbox属性
- current_bbox = current_line['bbox']
- next_bbox = next_line['bbox']
-
- # 检查是否满足条件
- if next_bbox[0] > current_bbox[2] or next_bbox[2] < current_bbox[0]:
- num_satisfying_lines += 1
- # 如果有一半以上的line满足条件，就drop
- # print("num_satisfying_lines:", num_satisfying_lines, "num_lines:", num_lines)
- if num_lines > 20:
- radio = num_satisfying_lines / num_lines
- if radio >= 0.5:
- extra_info = f"{{num_lines: {num_lines}, num_satisfying_lines: {num_satisfying_lines}}}"
- block_text = []
- for line in lines:
- if line['spans']:
- for span in line['spans']:
- block_text.append(span['text'])
- logger.warning(f"pseudo_single_column block_text: {block_text}")
- return True, extra_info
-
- return False, None
-
-
-def pdf_post_filter(page_info) -> tuple:
- """
- return:(True|False, err_msg)
- True, 如果pdf符合要求
- False, 如果pdf不符合要求
-
- """
- bool_is_pseudo_single_column, extra_info = __is_pseudo_single_column(page_info)
- if bool_is_pseudo_single_column:
- return False, {"_need_drop": True, "_drop_reason": DropReason.PSEUDO_SINGLE_COLUMN, "extra_info": extra_info}
-
- return True, None
\ No newline at end of file
diff --git a/magic_pdf/post_proc/remove_footnote.py b/magic_pdf/post_proc/remove_footnote.py
deleted file mode 100644
index 976d3a6a4..000000000
--- a/magic_pdf/post_proc/remove_footnote.py
+++ /dev/null
@@ -1,153 +0,0 @@
-from magic_pdf.libs.boxbase import _is_in, _is_in_or_part_overlap
-import collections # 统计库
-
-
-
-def is_below(bbox1, bbox2):
- # 如果block1的上边y坐标大于block2的下边y坐标，那么block1在block2下面
- return bbox1[1] > bbox2[3]
-
-
-def merge_bboxes(bboxes):
- # 找出所有blocks的最小x0，最大y1，最大x1，最小y0，这就是合并后的bbox
- x0 = min(bbox[0] for bbox in bboxes)
- y0 = min(bbox[1] for bbox in bboxes)
- x1 = max(bbox[2] for bbox in bboxes)
- y1 = max(bbox[3] for bbox in bboxes)
- return [x0, y0, x1, y1]
-
-
-def merge_footnote_blocks(page_info, main_text_font):
- page_info['merged_bboxes'] = []
- for layout in page_info['layout_bboxes']:
- # 找出layout中的所有footnote blocks和preproc_blocks
- footnote_bboxes = [block for block in page_info['footnote_bboxes_tmp'] if _is_in(block, layout['layout_bbox'])]
- # 如果没有footnote_blocks，就跳过这个layout
- if not footnote_bboxes:
- continue
-
- preproc_blocks = [block for block in page_info['preproc_blocks'] if _is_in(block['bbox'], layout['layout_bbox'])]
- # preproc_bboxes = [block['bbox'] for block in preproc_blocks]
- font_names = collections.Counter()
- if len(preproc_blocks) > 0:
- # 存储每一行的文本块大小的列表
- line_sizes = []
- # 存储每个文本块的平均行大小
- block_sizes = []
- for block in preproc_blocks:
- block_line_sizes = []
- block_fonts = collections.Counter()
- for line in block['lines']:
- # 提取每个span的size属性，并计算行大小
- span_sizes = [span['size'] for span in line['spans'] if 'size' in span]
- if span_sizes:
- line_size = sum(span_sizes) / len(span_sizes)
- line_sizes.append(line_size)
- block_line_sizes.append(line_size)
- span_font = [(span['font'], len(span['text'])) for span in line['spans'] if
- 'font' in span and len(span['text']) > 0]
- if span_font:
- # # todo main_text_font应该用基于字数最多的字体而不是span级别的统计
- # font_names.append(font_name for font_name in span_font)
- # block_fonts.append(font_name for font_name in span_font)
- for font, count in span_font:
- # font_names.extend([font] * count)
- # block_fonts.extend([font] * count)
- font_names[font] += count
- block_fonts[font] += count
- if block_line_sizes:
- # 计算文本块的平均行大小
- block_size = sum(block_line_sizes) / len(block_line_sizes)
- block_font = block_fonts.most_common(1)[0][0]
- block_sizes.append((block, block_size, block_font))
-
- # 计算main_text_size
- # main_text_font = font_names.most_common(1)[0][0]
- main_text_size = collections.Counter(line_sizes).most_common(1)[0][0]
- else:
- continue
-
- need_merge_bboxes = []
- # 任何一个下面有正文block的footnote bbox都是假footnote
- for footnote_bbox in footnote_bboxes:
- # 检测footnote下面是否有正文block(正文block需满足，block平均size大于等于main_text_size，且block行数大于等于5)
- main_text_bboxes_below = [block['bbox'] for block, size, block_font in block_sizes if
- is_below(block['bbox'], footnote_bbox) and
- sum([size >= main_text_size,
- len(block['lines']) >= 5,
- block_font == main_text_font])
- >= 2]
- # 如果main_text_bboxes_below不为空，说明footnote下面有正文block，这个footnote不成立，跳过
- if len(main_text_bboxes_below) > 0:
- continue
- else:
- # 否则，说明footnote下面没有正文block，这个footnote成立，添加到待merge的footnote_bboxes中
- need_merge_bboxes.append(footnote_bbox)
- if len(need_merge_bboxes) == 0:
- continue
- # 找出最靠上的footnote block
- top_footnote_bbox = min(need_merge_bboxes, key=lambda bbox: bbox[1])
- # 找出所有在top_footnote_block下面的preproc_blocks，并确保这些preproc_blocks的平均行大小小于main_text_size
- bboxes_below = [block['bbox'] for block, size, block_font in block_sizes if is_below(block['bbox'], top_footnote_bbox)]
- # # 找出所有在top_footnote_block下面的preproc_blocks
- # bboxes_below = [bbox for bbox in preproc_bboxes if is_below(bbox, top_footnote_bbox)]
- # 合并top_footnote_block和blocks_below
- merged_bbox = merge_bboxes([top_footnote_bbox] + bboxes_below)
- # 添加到新的footnote_bboxes_tmp中
- page_info['merged_bboxes'].append(merged_bbox)
- return page_info
-
-
-def remove_footnote_blocks(page_info):
- if page_info.get('merged_bboxes'):
- # 从文字中去掉footnote
- remain_text_blocks, removed_footnote_text_blocks = remove_footnote_text(page_info['preproc_blocks'], page_info['merged_bboxes'])
- # 从图片中去掉footnote
- image_blocks, removed_footnote_imgs_blocks = remove_footnote_image(page_info['images'], page_info['merged_bboxes'])
- # 更新page_info
- page_info['preproc_blocks'] = remain_text_blocks
- page_info['images'] = image_blocks
- page_info['droped_text_block'].extend(removed_footnote_text_blocks)
- page_info['droped_image_block'].extend(removed_footnote_imgs_blocks)
- # 删除footnote_bboxes_tmp和merged_bboxes
- del page_info['merged_bboxes']
- del page_info['footnote_bboxes_tmp']
- return page_info
-
-
-def remove_footnote_text(raw_text_block, footnote_bboxes):
- """
- :param raw_text_block: str类型，是当前页的文本内容
- :param footnoteBboxes: list类型，是当前页的脚注bbox
- """
- footnote_text_blocks = []
- for block in raw_text_block:
- text_bbox = block['bbox']
- # TODO 更严谨点在line级别做
- if any([_is_in_or_part_overlap(text_bbox, footnote_bbox) for footnote_bbox in footnote_bboxes]):
- # if any([text_bbox[3]>=footnote_bbox[1] for footnote_bbox in footnote_bboxes]):
- block['tag'] = 'footnote'
- footnote_text_blocks.append(block)
- # raw_text_block.remove(block)
-
- # 移除，不能再内部移除，否则会出错
- for block in footnote_text_blocks:
- raw_text_block.remove(block)
-
- return raw_text_block, footnote_text_blocks
-
-
-def remove_footnote_image(image_blocks, footnote_bboxes):
- """
- :param image_bboxes: list类型，是当前页的图片bbox(结构体)
- :param footnoteBboxes: list类型，是当前页的脚注bbox
- """
- footnote_imgs_blocks = []
- for image_block in image_blocks:
- if any([_is_in(image_block['bbox'], footnote_bbox) for footnote_bbox in footnote_bboxes]):
- footnote_imgs_blocks.append(image_block)
-
- for footnote_imgs_block in footnote_imgs_blocks:
- image_blocks.remove(footnote_imgs_block)
-
- return image_blocks, footnote_imgs_blocks
\ No newline at end of file
diff --git a/magic_pdf/pre_proc/__init__.py b/magic_pdf/pre_proc/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/pre_proc/citationmarker_remove.py b/magic_pdf/pre_proc/citationmarker_remove.py
deleted file mode 100644
index cb69e238a..000000000
--- a/magic_pdf/pre_proc/citationmarker_remove.py
+++ /dev/null
@@ -1,161 +0,0 @@
-"""
-去掉正文的引文引用marker
-https://aicarrier.feishu.cn/wiki/YLOPwo1PGiwFRdkwmyhcZmr0n3d
-"""
-import re
-# from magic_pdf.libs.nlp_utils import NLPModels
-
-
-# __NLP_MODEL = NLPModels()
-
-def check_1(spans, cur_span_i):
- """寻找前一个char,如果是句号，逗号，那么就是角标"""
- if cur_span_i==0:
- return False # 不是角标
- pre_span = spans[cur_span_i-1]
- pre_char = pre_span['chars'][-1]['c']
- if pre_char in ['。', '，', '.', ',']:
- return True
-
- return False
-
-
-# def check_2(spans, cur_span_i):
-# """检查前面一个span的最后一个单词，如果长度大于5，全都是字母，并且不含大写，就是角标"""
-# pattern = r'\b[A-Z]\.\s[A-Z][a-z]*\b' # 形如A. Bcde, L. Bcde, 人名的缩写
-#
-# if cur_span_i==0 and len(spans)>1:
-# next_span = spans[cur_span_i+1]
-# next_txt = "".join([c['c'] for c in next_span['chars']])
-# result = __NLP_MODEL.detect_entity_catgr_using_nlp(next_txt)
-# if result in ["PERSON", "GPE", "ORG"]:
-# return True
-#
-# if re.findall(pattern, next_txt):
-# return True
-#
-# return False # 不是角标
-# elif cur_span_i==0 and len(spans)==1: # 角标占用了整行？谨慎删除
-# return False
-#
-# # 如果这个span是最后一个span,
-# if cur_span_i==len(spans)-1:
-# pre_span = spans[cur_span_i-1]
-# pre_txt = "".join([c['c'] for c in pre_span['chars']])
-# pre_word = pre_txt.split(' ')[-1]
-# result = __NLP_MODEL.detect_entity_catgr_using_nlp(pre_txt)
-# if result in ["PERSON", "GPE", "ORG"]:
-# return True
-#
-# if re.findall(pattern, pre_txt):
-# return True
-#
-# return len(pre_word) > 5 and pre_word.isalpha() and pre_word.islower()
-# else: # 既不是第一个span，也不是最后一个span，那么此时检查一下这个角标距离前后哪个单词更近就属于谁的角标
-# pre_span = spans[cur_span_i-1]
-# next_span = spans[cur_span_i+1]
-# cur_span = spans[cur_span_i]
-# # 找到前一个和后一个span里的距离最近的单词
-# pre_distance = 10000 # 一个很大的数
-# next_distance = 10000 # 一个很大的数
-# for c in pre_span['chars'][::-1]:
-# if c['c'].isalpha():
-# pre_distance = cur_span['bbox'][0] - c['bbox'][2]
-# break
-# for c in next_span['chars']:
-# if c['c'].isalpha():
-# next_distance = c['bbox'][0] - cur_span['bbox'][2]
-# break
-#
-# if pre_distance 5 and pre_word.isalpha() and pre_word.islower()
-
-
-def check_3(spans, cur_span_i):
- """上标里有[], 有*， 有-， 有逗号"""
- # 如[2-3],[22]
- # 如 2,3,4
- cur_span_txt = ''.join(c['c'] for c in spans[cur_span_i]['chars']).strip()
- bad_char = ['[', ']', '*', ',']
-
- if any([c in cur_span_txt for c in bad_char]) and any(character.isdigit() for character in cur_span_txt):
- return True
-
- # 如2-3, a-b
- patterns = [r'\d+-\d+', r'[a-zA-Z]-[a-zA-Z]', r'[a-zA-Z],[a-zA-Z]']
- for pattern in patterns:
- match = re.match(pattern, cur_span_txt)
- if match is not None:
- return True
-
- return False
-
-
-def remove_citation_marker(with_char_text_blcoks):
- for blk in with_char_text_blcoks:
- for line in blk['lines']:
- # 如果span里的个数少于2个，那只能忽略，角标不可能自己独占一行
- if len(line['spans'])<=1:
- continue
-
- # 找到高度最高的span作为位置比较的基准
- max_hi_span = line['spans'][0]['bbox']
- min_font_sz = 10000 # line里最小的字体
- max_font_sz = 0 # line里最大的字体
-
- for s in line['spans']:
- if max_hi_span[3]-max_hi_span[1]s['size']:
- min_font_sz = s['size']
- if max_font_sz0.2 or (base_span_mid_y-span_mid_y>0 and abs(span_font_sz-min_font_sz)/min_font_sz<0.1):
- """
- 1. 它的前一个char如果是句号或者逗号的话，那么肯定是角标而不是公式
- 2. 如果这个角标的前面是一个单词（长度大于5）而不是任何大写或小写的短字母的话 应该也是角标
- 3. 上标里有数字和逗号或者数字+星号的组合，方括号，一般肯定就是角标了
- 4. 这个角标属于前文还是后文要根据距离来判断，如果距离前面的文本太近，那么就是前面的角标，否则就是后面的角标
- """
- if (check_1(line['spans'], i) or
- # check_2(line['spans'], i) or
- check_3(line['spans'], i)
- ):
- """删除掉这个角标：删除这个span, 同时还要更新line的text"""
- span_to_del.append(span)
- if len(span_to_del)>0:
- for span in span_to_del:
- line['spans'].remove(span)
- line['text'] = ''.join([c['c'] for s in line['spans'] for c in s['chars']])
-
- return with_char_text_blcoks
diff --git a/magic_pdf/pre_proc/construct_page_dict.py b/magic_pdf/pre_proc/construct_page_dict.py
deleted file mode 100644
index c2f83c100..000000000
--- a/magic_pdf/pre_proc/construct_page_dict.py
+++ /dev/null
@@ -1,72 +0,0 @@
-def construct_page_component(page_id, image_info, table_info, text_blocks_preproc, layout_bboxes, inline_eq_info,
- interline_eq_info, raw_pymu_blocks,
- removed_text_blocks, removed_image_blocks, images_backup, droped_table_block, table_backup,
- layout_tree,
- page_w, page_h, footnote_bboxes_tmp):
- """
-
- """
- return_dict = {}
-
- return_dict['para_blocks'] = {}
- return_dict['preproc_blocks'] = text_blocks_preproc
- return_dict['images'] = image_info
- return_dict['tables'] = table_info
- return_dict['interline_equations'] = interline_eq_info
- return_dict['inline_equations'] = inline_eq_info
- return_dict['layout_bboxes'] = layout_bboxes
- return_dict['pymu_raw_blocks'] = raw_pymu_blocks
- return_dict['global_statistic'] = {}
-
- return_dict['droped_text_block'] = removed_text_blocks
- return_dict['droped_image_block'] = removed_image_blocks
- return_dict['droped_table_block'] = []
- return_dict['image_backup'] = images_backup
- return_dict['table_backup'] = []
- return_dict['page_idx'] = page_id
- return_dict['page_size'] = [page_w, page_h]
- return_dict['_layout_tree'] = layout_tree # 辅助分析layout作用
- return_dict['footnote_bboxes_tmp'] = footnote_bboxes_tmp
-
- return return_dict
-
-
-def ocr_construct_page_component(blocks, layout_bboxes, page_id, page_w, page_h, layout_tree,
- images, tables, interline_equations, inline_equations,
- dropped_text_block, dropped_image_block, dropped_table_block, dropped_equation_block,
- need_remove_spans_bboxes_dict):
- return_dict = {
- 'preproc_blocks': blocks,
- 'layout_bboxes': layout_bboxes,
- 'page_idx': page_id,
- 'page_size': [page_w, page_h],
- '_layout_tree': layout_tree,
- 'images': images,
- 'tables': tables,
- 'interline_equations': interline_equations,
- 'inline_equations': inline_equations,
- 'droped_text_block': dropped_text_block,
- 'droped_image_block': dropped_image_block,
- 'droped_table_block': dropped_table_block,
- 'dropped_equation_block': dropped_equation_block,
- 'droped_bboxes': need_remove_spans_bboxes_dict,
- }
- return return_dict
-
-
-def ocr_construct_page_component_v2(blocks, layout_bboxes, page_id, page_w, page_h, layout_tree,
- images, tables, interline_equations, discarded_blocks, need_drop, drop_reason):
- return_dict = {
- 'preproc_blocks': blocks,
- 'layout_bboxes': layout_bboxes,
- 'page_idx': page_id,
- 'page_size': [page_w, page_h],
- '_layout_tree': layout_tree,
- 'images': images,
- 'tables': tables,
- 'interline_equations': interline_equations,
- 'discarded_blocks': discarded_blocks,
- 'need_drop': need_drop,
- 'drop_reason': drop_reason,
- }
- return return_dict
diff --git a/magic_pdf/pre_proc/cut_image.py b/magic_pdf/pre_proc/cut_image.py
deleted file mode 100644
index 18ee65129..000000000
--- a/magic_pdf/pre_proc/cut_image.py
+++ /dev/null
@@ -1,71 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.commons import join_path
-from magic_pdf.libs.ocr_content_type import ContentType
-from magic_pdf.libs.pdf_image_tools import cut_image
-
-
-def ocr_cut_image_and_table(spans, page, page_id, pdf_bytes_md5, imageWriter):
- def return_path(type):
- return join_path(pdf_bytes_md5, type)
-
- for span in spans:
- span_type = span['type']
- if span_type == ContentType.Image:
- if not check_img_bbox(span['bbox']):
- continue
- span['image_path'] = cut_image(span['bbox'], page_id, page, return_path=return_path('images'),
- imageWriter=imageWriter)
- elif span_type == ContentType.Table:
- if not check_img_bbox(span['bbox']):
- continue
- span['image_path'] = cut_image(span['bbox'], page_id, page, return_path=return_path('tables'),
- imageWriter=imageWriter)
-
- return spans
-
-
-def txt_save_images_by_bboxes(page_num: int, page, pdf_bytes_md5: str,
- image_bboxes: list, images_overlap_backup: list, table_bboxes: list,
- equation_inline_bboxes: list,
- equation_interline_bboxes: list, imageWriter) -> dict:
- """
- 返回一个dict, key为bbox, 值是图片地址
- """
- image_info = []
- image_backup_info = []
- table_info = []
- inline_eq_info = []
- interline_eq_info = []
-
- # 图片的保存路径组成是这样的： {s3_or_local_path}/{book_name}/{images|tables|equations}/{page_num}_{bbox[0]}_{bbox[1]}_{bbox[2]}_{bbox[3]}.jpg
-
- def return_path(type):
- return join_path(pdf_bytes_md5, type)
-
- for bbox in image_bboxes:
- if not check_img_bbox(bbox):
- continue
- image_path = cut_image(bbox, page_num, page, return_path("images"), imageWriter)
- image_info.append({"bbox": bbox, "image_path": image_path})
-
- for bbox in images_overlap_backup:
- if not check_img_bbox(bbox):
- continue
- image_path = cut_image(bbox, page_num, page, return_path("images"), imageWriter)
- image_backup_info.append({"bbox": bbox, "image_path": image_path})
-
- for bbox in table_bboxes:
- if not check_img_bbox(bbox):
- continue
- image_path = cut_image(bbox, page_num, page, return_path("tables"), imageWriter)
- table_info.append({"bbox": bbox, "image_path": image_path})
-
- return image_info, image_backup_info, table_info, inline_eq_info, interline_eq_info
-
-
-def check_img_bbox(bbox) -> bool:
- if any([bbox[0] >= bbox[2], bbox[1] >= bbox[3]]):
- logger.warning(f"image_bboxes: 错误的box, {bbox}")
- return False
- return True
diff --git a/magic_pdf/pre_proc/detect_equation.py b/magic_pdf/pre_proc/detect_equation.py
deleted file mode 100644
index f395030c4..000000000
--- a/magic_pdf/pre_proc/detect_equation.py
+++ /dev/null
@@ -1,134 +0,0 @@
-from magic_pdf.libs.boxbase import _is_in, calculate_overlap_area_2_minbox_area_ratio # 正则
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-
-
-def __solve_contain_bboxs(all_bbox_list: list):
-
- """将两个公式的bbox做判断是否有包含关系，若有的话则删掉较小的bbox"""
-
- dump_list = []
- for i in range(len(all_bbox_list)):
- for j in range(i + 1, len(all_bbox_list)):
- # 获取当前两个值
- bbox1 = all_bbox_list[i][:4]
- bbox2 = all_bbox_list[j][:4]
-
- # 删掉较小的框
- if _is_in(bbox1, bbox2):
- dump_list.append(all_bbox_list[i])
- elif _is_in(bbox2, bbox1):
- dump_list.append(all_bbox_list[j])
- else:
- ratio = calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2)
- if ratio > 0.7:
- s1 = (bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1])
- s2 = (bbox2[2] - bbox2[0]) * (bbox2[3] - bbox2[1])
- if s2 > s1:
- dump_list.append(all_bbox_list[i])
- else:
- dump_list.append(all_bbox_list[i])
-
- # 遍历需要删除的列表中的每个元素
- for item in dump_list:
-
- while item in all_bbox_list:
- all_bbox_list.remove(item)
- return all_bbox_list
-
-
-def parse_equations(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
- DPI = 72 # use this resolution
- pix = page.get_pixmap(dpi=DPI)
- pageL = 0
- pageR = int(pix.w)
- pageU = 0
- pageD = int(pix.h)
-
-
- #--------- 通过json_from_DocXchain来获取 table ---------#
- equationEmbedding_from_DocXChain_bboxs = []
- equationIsolated_from_DocXChain_bboxs = []
-
- xf_json = json_from_DocXchain_obj
- width_from_json = xf_json['page_info']['width']
- height_from_json = xf_json['page_info']['height']
- LR_scaleRatio = width_from_json / (pageR - pageL)
- UD_scaleRatio = height_from_json / (pageD - pageU)
-
- for xf in xf_json['layout_dets']:
- # {0: 'title', 1: 'figure', 2: 'plain text', 3: 'header', 4: 'page number', 5: 'footnote', 6: 'footer', 7: 'table', 8: 'table caption', 9: 'figure caption', 10: 'equation', 11: 'full column', 12: 'sub column'}
- L = xf['poly'][0] / LR_scaleRatio
- U = xf['poly'][1] / UD_scaleRatio
- R = xf['poly'][2] / LR_scaleRatio
- D = xf['poly'][5] / UD_scaleRatio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- # equation
- img_suffix = f"{page_ID}_{int(L)}_{int(U)}_{int(R)}_{int(D)}"
- if xf['category_id'] == 13 and xf['score'] >= 0.3:
- latex_text = xf.get("latex", "EmptyInlineEquationResult")
- debugable_latex_text = f"{latex_text}|{img_suffix}"
- equationEmbedding_from_DocXChain_bboxs.append((L, U, R, D, latex_text))
- if xf['category_id'] == 14 and xf['score'] >= 0.3:
- latex_text = xf.get("latex", "EmptyInterlineEquationResult")
- debugable_latex_text = f"{latex_text}|{img_suffix}"
- equationIsolated_from_DocXChain_bboxs.append((L, U, R, D, latex_text))
-
- #---------------------------------------- 排序，编号，保存 -----------------------------------------#
- equationIsolated_from_DocXChain_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- equationIsolated_from_DocXChain_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
-
- equationEmbedding_from_DocXChain_names = []
- equationEmbedding_ID = 0
-
- equationIsolated_from_DocXChain_names = []
- equationIsolated_ID = 0
-
- for L, U, R, D, _ in equationEmbedding_from_DocXChain_bboxs:
- if not(L < R and U < D):
- continue
- try:
- # cur_equation = page.get_pixmap(clip=(L,U,R,D))
- new_equation_name = "equationEmbedding_{}_{}.png".format(page_ID, equationEmbedding_ID) # 公式name
- # cur_equation.save(res_dir_path + '/' + new_equation_name) # 把公式存出在新建的文件夹，并命名
- equationEmbedding_from_DocXChain_names.append(new_equation_name) # 把公式的名字存在list中，方便在md中插入引用
- equationEmbedding_ID += 1
- except:
- pass
-
- for L, U, R, D, _ in equationIsolated_from_DocXChain_bboxs:
- if not(L < R and U < D):
- continue
- try:
- # cur_equation = page.get_pixmap(clip=(L,U,R,D))
- new_equation_name = "equationEmbedding_{}_{}.png".format(page_ID, equationIsolated_ID) # 公式name
- # cur_equation.save(res_dir_path + '/' + new_equation_name) # 把公式存出在新建的文件夹，并命名
- equationIsolated_from_DocXChain_names.append(new_equation_name) # 把公式的名字存在list中，方便在md中插入引用
- equationIsolated_ID += 1
- except:
- pass
-
- equationEmbedding_from_DocXChain_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- equationIsolated_from_DocXChain_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
-
-
- """根据pdf可视区域，调整bbox的坐标"""
- cropbox = page.cropbox
- if cropbox[0]!=page.rect[0] or cropbox[1]!=page.rect[1]:
- for eq_box in equationEmbedding_from_DocXChain_bboxs:
- eq_box = [eq_box[0]+cropbox[0], eq_box[1]+cropbox[1], eq_box[2]+cropbox[0], eq_box[3]+cropbox[1], eq_box[4]]
- for eq_box in equationIsolated_from_DocXChain_bboxs:
- eq_box = [eq_box[0]+cropbox[0], eq_box[1]+cropbox[1], eq_box[2]+cropbox[0], eq_box[3]+cropbox[1], eq_box[4]]
-
- deduped_embedding_eq_bboxes = __solve_contain_bboxs(equationEmbedding_from_DocXChain_bboxs)
- return deduped_embedding_eq_bboxes, equationIsolated_from_DocXChain_bboxs
diff --git a/magic_pdf/pre_proc/detect_footer_by_model.py b/magic_pdf/pre_proc/detect_footer_by_model.py
deleted file mode 100644
index 0c1fbf38b..000000000
--- a/magic_pdf/pre_proc/detect_footer_by_model.py
+++ /dev/null
@@ -1,64 +0,0 @@
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-
-
-def parse_footers(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
-
- #--------- 通过json_from_DocXchain来获取 footer ---------#
- footer_bbox_from_DocXChain = []
-
- xf_json = json_from_DocXchain_obj
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(xf_json, page)
-
- # {0: 'title', # 标题
- # 1: 'figure', # 图片
- # 2: 'plain text', # 文本
- # 3: 'header', # 页眉
- # 4: 'page number', # 页码
- # 5: 'footnote', # 脚注
- # 6: 'footer', # 页脚
- # 7: 'table', # 表格
- # 8: 'table caption', # 表格描述
- # 9: 'figure caption', # 图片描述
- # 10: 'equation', # 公式
- # 11: 'full column', # 单栏
- # 12: 'sub column', # 多栏
- # 13: 'embedding', # 嵌入公式
- # 14: 'isolated'} # 单行公式
- for xf in xf_json['layout_dets']:
- L = xf['poly'][0] / horizontal_scale_ratio
- U = xf['poly'][1] / vertical_scale_ratio
- R = xf['poly'][2] / horizontal_scale_ratio
- D = xf['poly'][5] / vertical_scale_ratio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- if xf['category_id'] == 6 and xf['score'] >= 0.3:
- footer_bbox_from_DocXChain.append((L, U, R, D))
-
-
- footer_final_names = []
- footer_final_bboxs = []
- footer_ID = 0
- for L, U, R, D in footer_bbox_from_DocXChain:
- # cur_footer = page.get_pixmap(clip=(L,U,R,D))
- new_footer_name = "footer_{}_{}.png".format(page_ID, footer_ID) # 脚注name
- # cur_footer.save(res_dir_path + '/' + new_footer_name) # 把页脚存储在新建的文件夹，并命名
- footer_final_names.append(new_footer_name) # 把脚注的名字存在list中
- footer_final_bboxs.append((L, U, R, D))
- footer_ID += 1
-
-
- footer_final_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_footer_bboxs = footer_final_bboxs
- return curPage_all_footer_bboxs
-
diff --git a/magic_pdf/pre_proc/detect_footer_header_by_statistics.py b/magic_pdf/pre_proc/detect_footer_header_by_statistics.py
deleted file mode 100644
index 340965d0e..000000000
--- a/magic_pdf/pre_proc/detect_footer_header_by_statistics.py
+++ /dev/null
@@ -1,284 +0,0 @@
-from collections import defaultdict
-
-from magic_pdf.libs.boxbase import calculate_iou
-
-
-def compare_bbox_with_list(bbox, bbox_list, tolerance=1):
- return any(all(abs(a - b) < tolerance for a, b in zip(bbox, common_bbox)) for common_bbox in bbox_list)
-
-def is_single_line_block(block):
- # Determine based on the width and height of the block
- block_width = block["X1"] - block["X0"]
- block_height = block["bbox"][3] - block["bbox"][1]
-
- # If the height of the block is close to the average character height and the width is large, it is considered a single line
- return block_height <= block["avg_char_height"] * 3 and block_width > block["avg_char_width"] * 3
-
-def get_most_common_bboxes(bboxes, page_height, position="top", threshold=0.25, num_bboxes=3, min_frequency=2):
- """
- This function gets the most common bboxes from the bboxes
-
- Parameters
- ----------
- bboxes : list
- bboxes
- page_height : float
- height of the page
- position : str, optional
- "top" or "bottom", by default "top"
- threshold : float, optional
- threshold, by default 0.25
- num_bboxes : int, optional
- number of bboxes to return, by default 3
- min_frequency : int, optional
- minimum frequency of the bbox, by default 2
-
- Returns
- -------
- common_bboxes : list
- common bboxes
- """
- # Filter bbox by position
- if position == "top":
- filtered_bboxes = [bbox for bbox in bboxes if bbox[1] < page_height * threshold]
- else:
- filtered_bboxes = [bbox for bbox in bboxes if bbox[3] > page_height * (1 - threshold)]
-
- # Find the most common bbox
- bbox_count = defaultdict(int)
- for bbox in filtered_bboxes:
- bbox_count[tuple(bbox)] += 1
-
- # Get the most frequently occurring bbox, but only consider it when the frequency exceeds min_frequency
- common_bboxes = [
- bbox for bbox, count in sorted(bbox_count.items(), key=lambda item: item[1], reverse=True) if count >= min_frequency
- ][:num_bboxes]
- return common_bboxes
-
-def detect_footer_header2(result_dict, similarity_threshold=0.5):
- """
- This function detects the header and footer of the document.
-
- Parameters
- ----------
- result_dict : dict
- result dictionary
-
- Returns
- -------
- result_dict : dict
- result dictionary
- """
- # Traverse all blocks in the document
- single_line_blocks = 0
- total_blocks = 0
- single_line_blocks = 0
-
- for page_id, blocks in result_dict.items():
- if page_id.startswith("page_"):
- for block_key, block in blocks.items():
- if block_key.startswith("block_"):
- total_blocks += 1
- if is_single_line_block(block):
- single_line_blocks += 1
-
- # If there are no blocks, skip the header and footer detection
- if total_blocks == 0:
- print("No blocks found. Skipping header/footer detection.")
- return result_dict
-
- # If most of the blocks are single-line, skip the header and footer detection
- if single_line_blocks / total_blocks > 0.5: # 50% of the blocks are single-line
- # print("Skipping header/footer detection for text-dense document.")
- return result_dict
-
- # Collect the bounding boxes of all blocks
- all_bboxes = []
- all_texts = []
-
- for page_id, blocks in result_dict.items():
- if page_id.startswith("page_"):
- for block_key, block in blocks.items():
- if block_key.startswith("block_"):
- all_bboxes.append(block["bbox"])
-
- # Get the height of the page
- page_height = max(bbox[3] for bbox in all_bboxes)
-
- # Get the most common bbox lists for headers and footers
- common_header_bboxes = get_most_common_bboxes(all_bboxes, page_height, position="top") if all_bboxes else []
- common_footer_bboxes = get_most_common_bboxes(all_bboxes, page_height, position="bottom") if all_bboxes else []
-
- # Detect and mark headers and footers
- for page_id, blocks in result_dict.items():
- if page_id.startswith("page_"):
- for block_key, block in blocks.items():
- if block_key.startswith("block_"):
- bbox = block["bbox"]
- text = block["text"]
-
- is_header = compare_bbox_with_list(bbox, common_header_bboxes)
- is_footer = compare_bbox_with_list(bbox, common_footer_bboxes)
- block["is_header"] = int(is_header)
- block["is_footer"] = int(is_footer)
-
- return result_dict
-
-
-def __get_page_size(page_sizes:list):
- """
- 页面大小可能不一样
- """
- w = sum([w for w,h in page_sizes])/len(page_sizes)
- h = sum([h for w,h in page_sizes])/len(page_sizes)
- return w, h
-
-def __calculate_iou(bbox1, bbox2):
- iou = calculate_iou(bbox1, bbox2)
- return iou
-
-def __is_same_pos(box1, box2, iou_threshold):
- iou = __calculate_iou(box1, box2)
- return iou >= iou_threshold
-
-
-def get_most_common_bbox(bboxes:list, page_size:list, page_cnt:int, page_range_threshold=0.2, iou_threshold=0.9):
- """
- common bbox必须大于page_cnt的1/3
- """
- min_occurance_cnt = max(3, page_cnt//4)
- header_det_bbox = []
- footer_det_bbox = []
-
- hdr_same_pos_group = []
- btn_same_pos_group = []
-
- page_w, page_h = __get_page_size(page_size)
- top_y, bottom_y = page_w*page_range_threshold, page_h*(1-page_range_threshold)
-
- top_bbox = [b for b in bboxes if b[3]bottom_y]
- # 然后开始排序，寻找最经常出现的bbox, 寻找的时候如果IOU>iou_threshold就算是一个
- for i in range(0, len(top_bbox)):
- hdr_same_pos_group.append([top_bbox[i]])
- for j in range(i+1, len(top_bbox)):
- if __is_same_pos(top_bbox[i], top_bbox[j], iou_threshold):
- #header_det_bbox = [min(top_bbox[i][0], top_bbox[j][0]), min(top_bbox[i][1], top_bbox[j][1]), max(top_bbox[i][2], top_bbox[j][2]), max(top_bbox[i][3],top_bbox[j][3])]
- hdr_same_pos_group[i].append(top_bbox[j])
-
- for i in range(0, len(bottom_bbox)):
- btn_same_pos_group.append([bottom_bbox[i]])
- for j in range(i+1, len(bottom_bbox)):
- if __is_same_pos(bottom_bbox[i], bottom_bbox[j], iou_threshold):
- #footer_det_bbox = [min(bottom_bbox[i][0], bottom_bbox[j][0]), min(bottom_bbox[i][1], bottom_bbox[j][1]), max(bottom_bbox[i][2], bottom_bbox[j][2]), max(bottom_bbox[i][3],bottom_bbox[j][3])]
- btn_same_pos_group[i].append(bottom_bbox[j])
-
- # 然后看下每一组的bbox，是否符合大于page_cnt一定比例
- hdr_same_pos_group = [g for g in hdr_same_pos_group if len(g)>=min_occurance_cnt]
- btn_same_pos_group = [g for g in btn_same_pos_group if len(g)>=min_occurance_cnt]
-
- # 平铺2个list[list]
- hdr_same_pos_group = [bbox for g in hdr_same_pos_group for bbox in g]
- btn_same_pos_group = [bbox for g in btn_same_pos_group for bbox in g]
- # 寻找hdr_same_pos_group中的box[3]最大值，btn_same_pos_group中的box[1]最小值
- hdr_same_pos_group.sort(key=lambda b:b[3])
- btn_same_pos_group.sort(key=lambda b:b[1])
-
- hdr_y = hdr_same_pos_group[-1][3] if hdr_same_pos_group else 0
- btn_y = btn_same_pos_group[0][1] if btn_same_pos_group else page_h
-
- header_det_bbox = [0, 0, page_w, hdr_y]
- footer_det_bbox = [0, btn_y, page_w, page_h]
- # logger.warning(f"header: {header_det_bbox}, footer: {footer_det_bbox}")
- return header_det_bbox, footer_det_bbox, page_w, page_h
-
-
-def drop_footer_header(pdf_info_dict:dict):
- """
- 启用规则探测,在全局的视角上通过统计的方法。
- """
- header = []
- footer = []
-
- all_text_bboxes = [blk['bbox'] for _, val in pdf_info_dict.items() for blk in val['preproc_blocks']]
- image_bboxes = [img['bbox'] for _, val in pdf_info_dict.items() for img in val['images']] + [img['bbox'] for _, val in pdf_info_dict.items() for img in val['image_backup']]
- page_size = [val['page_size'] for _, val in pdf_info_dict.items()]
- page_cnt = len(pdf_info_dict.keys()) # 一共多少页
- header, footer, page_w, page_h = get_most_common_bbox(all_text_bboxes+image_bboxes, page_size, page_cnt)
-
- """"
- 把范围扩展到页面水平的整个方向上
- """
- if header:
- header = [0, 0, page_w, header[3]+1]
-
- if footer:
- footer = [0, footer[1]-1, page_w, page_h]
-
- # 找到footer, header范围之后，针对每一页pdf，从text、图片中删除这些范围内的内容
- # 移除text block
-
- for _, page_info in pdf_info_dict.items():
- header_text_blk = []
- footer_text_blk = []
- for blk in page_info['preproc_blocks']:
- blk_bbox = blk['bbox']
- if header and blk_bbox[3]<=header[3]:
- blk['tag'] = "header"
- header_text_blk.append(blk)
- elif footer and blk_bbox[1]>=footer[1]:
- blk['tag'] = "footer"
- footer_text_blk.append(blk)
-
- # 放入text_block_droped中
- page_info['droped_text_block'].extend(header_text_blk)
- page_info['droped_text_block'].extend(footer_text_blk)
-
- for blk in header_text_blk:
- page_info['preproc_blocks'].remove(blk)
- for blk in footer_text_blk:
- page_info['preproc_blocks'].remove(blk)
-
- """接下来把footer、header上的图片也删除掉。图片包括正常的和backup的"""
- header_image = []
- footer_image = []
-
- for image_info in page_info['images']:
- img_bbox = image_info['bbox']
- if header and img_bbox[3]<=header[3]:
- image_info['tag'] = "header"
- header_image.append(image_info)
- elif footer and img_bbox[1]>=footer[1]:
- image_info['tag'] = "footer"
- footer_image.append(image_info)
-
- page_info['droped_image_block'].extend(header_image)
- page_info['droped_image_block'].extend(footer_image)
-
- for img in header_image:
- page_info['images'].remove(img)
- for img in footer_image:
- page_info['images'].remove(img)
-
- """接下来吧backup的图片也删除掉"""
- header_image = []
- footer_image = []
-
- for image_info in page_info['image_backup']:
- img_bbox = image_info['bbox']
- if header and img_bbox[3]<=header[3]:
- image_info['tag'] = "header"
- header_image.append(image_info)
- elif footer and img_bbox[1]>=footer[1]:
- image_info['tag'] = "footer"
- footer_image.append(image_info)
-
- page_info['droped_image_block'].extend(header_image)
- page_info['droped_image_block'].extend(footer_image)
-
- for img in header_image:
- page_info['image_backup'].remove(img)
- for img in footer_image:
- page_info['image_backup'].remove(img)
-
- return header, footer
diff --git a/magic_pdf/pre_proc/detect_footnote.py b/magic_pdf/pre_proc/detect_footnote.py
deleted file mode 100644
index 4f903c855..000000000
--- a/magic_pdf/pre_proc/detect_footnote.py
+++ /dev/null
@@ -1,170 +0,0 @@
-from collections import Counter
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-
-
-def parse_footnotes_by_model(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict, md_bookname_save_path=None, debug_mode=False):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
-
- #--------- 通过json_from_DocXchain来获取 footnote ---------#
- footnote_bbox_from_DocXChain = []
-
- xf_json = json_from_DocXchain_obj
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(xf_json, page)
-
- # {0: 'title', # 标题
- # 1: 'figure', # 图片
- # 2: 'plain text', # 文本
- # 3: 'header', # 页眉
- # 4: 'page number', # 页码
- # 5: 'footnote', # 脚注
- # 6: 'footer', # 页脚
- # 7: 'table', # 表格
- # 8: 'table caption', # 表格描述
- # 9: 'figure caption', # 图片描述
- # 10: 'equation', # 公式
- # 11: 'full column', # 单栏
- # 12: 'sub column', # 多栏
- # 13: 'embedding', # 嵌入公式
- # 14: 'isolated'} # 单行公式
- for xf in xf_json['layout_dets']:
- L = xf['poly'][0] / horizontal_scale_ratio
- U = xf['poly'][1] / vertical_scale_ratio
- R = xf['poly'][2] / horizontal_scale_ratio
- D = xf['poly'][5] / vertical_scale_ratio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- # if xf['category_id'] == 5 and xf['score'] >= 0.3:
- if xf['category_id'] == 5 and xf['score'] >= 0.43: # 新的footnote阈值
- footnote_bbox_from_DocXChain.append((L, U, R, D))
-
-
- footnote_final_names = []
- footnote_final_bboxs = []
- footnote_ID = 0
- for L, U, R, D in footnote_bbox_from_DocXChain:
- if debug_mode:
- # cur_footnote = page.get_pixmap(clip=(L,U,R,D))
- new_footnote_name = "footnote_{}_{}.png".format(page_ID, footnote_ID) # 脚注name
- # cur_footnote.save(md_bookname_save_path + '/' + new_footnote_name) # 把脚注存储在新建的文件夹，并命名
- footnote_final_names.append(new_footnote_name) # 把脚注的名字存在list中
- footnote_final_bboxs.append((L, U, R, D))
- footnote_ID += 1
-
-
- footnote_final_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_footnote_bboxs = footnote_final_bboxs
- return curPage_all_footnote_bboxs
-
-
-def need_remove(block):
- if 'lines' in block and len(block['lines']) > 0:
- # block中只有一行，且该行文本全是大写字母，或字体为粗体bold关键词，SB关键词，把这个block捞回来
- if len(block['lines']) == 1:
- if 'spans' in block['lines'][0] and len(block['lines'][0]['spans']) == 1:
- font_keywords = ['SB', 'bold', 'Bold']
- if block['lines'][0]['spans'][0]['text'].isupper() or any(keyword in block['lines'][0]['spans'][0]['font'] for keyword in font_keywords):
- return True
- for line in block['lines']:
- if 'spans' in line and len(line['spans']) > 0:
- for span in line['spans']:
- # 检测"keyword"是否在span中，忽略大小写
- if "keyword" in span['text'].lower():
- return True
- return False
-
-def parse_footnotes_by_rule(remain_text_blocks, page_height, page_id, main_text_font):
- """
- 根据给定的文本块、页高和页码，解析出符合规则的脚注文本块，并返回其边界框。
-
- Args:
- remain_text_blocks (list): 包含所有待处理的文本块的列表。
- page_height (float): 页面的高度。
- page_id (int): 页面的ID。
-
- Returns:
- list: 符合规则的脚注文本块的边界框列表。
-
- """
- # if page_id > 20:
- if page_id > 2: # 为保证精确度，先只筛选前3页
- return []
- else:
- # 存储每一行的文本块大小的列表
- line_sizes = []
- # 存储每个文本块的平均行大小
- block_sizes = []
- # 存储每一行的字体信息
- # font_names = []
- font_names = Counter()
- if len(remain_text_blocks) > 0:
- for block in remain_text_blocks:
- block_line_sizes = []
- # block_fonts = []
- block_fonts = Counter()
- for line in block['lines']:
- # 提取每个span的size属性，并计算行大小
- span_sizes = [span['size'] for span in line['spans'] if 'size' in span]
- if span_sizes:
- line_size = sum(span_sizes) / len(span_sizes)
- line_sizes.append(line_size)
- block_line_sizes.append(line_size)
- span_font = [(span['font'], len(span['text'])) for span in line['spans'] if 'font' in span and len(span['text']) > 0]
- if span_font:
- # main_text_font应该用基于字数最多的字体而不是span级别的统计
- # font_names.append(font_name for font_name in span_font)
- # block_fonts.append(font_name for font_name in span_font)
- for font, count in span_font:
- # font_names.extend([font] * count)
- # block_fonts.extend([font] * count)
- font_names[font] += count
- block_fonts[font] += count
- if block_line_sizes:
- # 计算文本块的平均行大小
- block_size = sum(block_line_sizes) / len(block_line_sizes)
- # block_font = collections.Counter(block_fonts).most_common(1)[0][0]
- block_font = block_fonts.most_common(1)[0][0]
- block_sizes.append((block, block_size, block_font))
-
- # 计算main_text_size
- main_text_size = Counter(line_sizes).most_common(1)[0][0]
- # 计算main_text_font
- # main_text_font = collections.Counter(font_names).most_common(1)[0][0]
- # main_text_font = font_names.most_common(1)[0][0]
- # 删除一些可能被误识别为脚注的文本块
- block_sizes = [(block, block_size, block_font) for block, block_size, block_font in block_sizes if not need_remove(block)]
-
- # 检测footnote_block 并返回 footnote_bboxes
- # footnote_bboxes = [block['bbox'] for block, block_size, block_font in block_sizes if
- # block['bbox'][1] > page_height * 0.6 and block_size < main_text_size
- # and (len(block['lines']) < 5 or block_font != main_text_font)]
- # and len(block['lines']) < 5]
- footnote_bboxes = [block['bbox'] for block, block_size, block_font in block_sizes if
- block['bbox'][1] > page_height * 0.6 and
- # 较为严格的规则
- block_size < main_text_size and
- (len(block['lines']) < 5 or
- block_font != main_text_font)]
-
- # 较为宽松的规则
- # sum([block_size < main_text_size,
- # len(block['lines']) < 5,
- # block_font != main_text_font])
- # >= 2]
-
-
- return footnote_bboxes
- else:
- return []
-
-
-
diff --git a/magic_pdf/pre_proc/detect_header.py b/magic_pdf/pre_proc/detect_header.py
deleted file mode 100644
index 670eccd3d..000000000
--- a/magic_pdf/pre_proc/detect_header.py
+++ /dev/null
@@ -1,64 +0,0 @@
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-
-
-def parse_headers(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
-
- #--------- 通过json_from_DocXchain来获取 header ---------#
- header_bbox_from_DocXChain = []
-
- xf_json = json_from_DocXchain_obj
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(xf_json, page)
-
- # {0: 'title', # 标题
- # 1: 'figure', # 图片
- # 2: 'plain text', # 文本
- # 3: 'header', # 页眉
- # 4: 'page number', # 页码
- # 5: 'footnote', # 脚注
- # 6: 'footer', # 页脚
- # 7: 'table', # 表格
- # 8: 'table caption', # 表格描述
- # 9: 'figure caption', # 图片描述
- # 10: 'equation', # 公式
- # 11: 'full column', # 单栏
- # 12: 'sub column', # 多栏
- # 13: 'embedding', # 嵌入公式
- # 14: 'isolated'} # 单行公式
- for xf in xf_json['layout_dets']:
- L = xf['poly'][0] / horizontal_scale_ratio
- U = xf['poly'][1] / vertical_scale_ratio
- R = xf['poly'][2] / horizontal_scale_ratio
- D = xf['poly'][5] / vertical_scale_ratio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- if xf['category_id'] == 3 and xf['score'] >= 0.3:
- header_bbox_from_DocXChain.append((L, U, R, D))
-
-
- header_final_names = []
- header_final_bboxs = []
- header_ID = 0
- for L, U, R, D in header_bbox_from_DocXChain:
- # cur_header = page.get_pixmap(clip=(L,U,R,D))
- new_header_name = "header_{}_{}.png".format(page_ID, header_ID) # 页眉name
- # cur_header.save(res_dir_path + '/' + new_header_name) # 把页眉存储在新建的文件夹，并命名
- header_final_names.append(new_header_name) # 把页面的名字存在list中
- header_final_bboxs.append((L, U, R, D))
- header_ID += 1
-
-
- header_final_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_header_bboxs = header_final_bboxs
- return curPage_all_header_bboxs
-
diff --git a/magic_pdf/pre_proc/detect_images.py b/magic_pdf/pre_proc/detect_images.py
deleted file mode 100644
index fe72f60c3..000000000
--- a/magic_pdf/pre_proc/detect_images.py
+++ /dev/null
@@ -1,647 +0,0 @@
-import collections # 统计库
-import re
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-
-
-#--------------------------------------- Tool Functions --------------------------------------#
-# 正则化，输入文本，输出只保留a-z,A-Z,0-9
-def remove_special_chars(s: str) -> str:
- pattern = r"[^a-zA-Z0-9]"
- res = re.sub(pattern, "", s)
- return res
-
-def check_rect1_sameWith_rect2(L1: float, U1: float, R1: float, D1: float, L2: float, U2: float, R2: float, D2: float) -> bool:
- # 判断rect1和rect2是否一模一样
- return L1 == L2 and U1 == U2 and R1 == R2 and D1 == D2
-
-def check_rect1_contains_rect2(L1: float, U1: float, R1: float, D1: float, L2: float, U2: float, R2: float, D2: float) -> bool:
- # 判断rect1包含了rect2
- return (L1 <= L2 <= R2 <= R1) and (U1 <= U2 <= D2 <= D1)
-
-def check_rect1_overlaps_rect2(L1: float, U1: float, R1: float, D1: float, L2: float, U2: float, R2: float, D2: float) -> bool:
- # 判断rect1与rect2是否存在重叠（只有一条边重叠，也算重叠）
- return max(L1, L2) <= min(R1, R2) and max(U1, U2) <= min(D1, D2)
-
-def calculate_overlapRatio_between_rect1_and_rect2(L1: float, U1: float, R1: float, D1: float, L2: float, U2: float, R2: float, D2: float) -> (float, float):
- # 计算两个rect，重叠面积各占2个rect面积的比例
- if min(R1, R2) < max(L1, L2) or min(D1, D2) < max(U1, U2):
- return 0, 0
- square_1 = (R1 - L1) * (D1 - U1)
- square_2 = (R2 - L2) * (D2 - U2)
- if square_1 == 0 or square_2 == 0:
- return 0, 0
- square_overlap = (min(R1, R2) - max(L1, L2)) * (min(D1, D2) - max(U1, U2))
- return square_overlap / square_1, square_overlap / square_2
-
-def calculate_overlapRatio_between_line1_and_line2(L1: float, R1: float, L2: float, R2: float) -> (float, float):
- # 计算两个line，重叠区间各占2个line长度的比例
- if max(L1, L2) > min(R1, R2):
- return 0, 0
- if L1 == R1 or L2 == R2:
- return 0, 0
- overlap_line = min(R1, R2) - max(L1, L2)
- return overlap_line / (R1 - L1), overlap_line / (R2 - L2)
-
-
-# 判断rect其实是一条line
-def check_rect_isLine(L: float, U: float, R: float, D: float) -> bool:
- width = R - L
- height = D - U
- if width <= 3 or height <= 3:
- return True
- if width / height >= 30 or height / width >= 30:
- return True
-
-
-
-def parse_images(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict, junk_img_bojids=[]):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
- #### 通过fitz获取page信息
- ## 超越边界
- DPI = 72 # use this resolution
- pix = page.get_pixmap(dpi=DPI)
- pageL = 0
- pageR = int(pix.w)
- pageU = 0
- pageD = int(pix.h)
-
- #----------------- 保存每一个文本块的LURD ------------------#
- textLine_blocks = []
- blocks = page.get_text(
- "dict",
- flags=fitz.TEXTFLAGS_TEXT,
- #clip=clip,
- )["blocks"]
- for i in range(len(blocks)):
- bbox = blocks[i]['bbox']
- # print(bbox)
- for tt in blocks[i]['lines']:
- # 当前line
- cur_line_bbox = None # 当前line，最右侧的section的bbox
- for xf in tt['spans']:
- L, U, R, D = xf['bbox']
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- textLine_blocks.append((L, U, R, D))
- textLine_blocks.sort(key = lambda LURD: (LURD[1], LURD[0]))
-
-
- #---------------------------------------------- 保存img --------------------------------------------------#
- raw_imgs = page.get_images() # 获取所有的图片
- imgs = []
- img_names = [] # 保存图片的名字，方便在md中插入引用
- img_bboxs = [] # 保存图片的location信息。
- img_visited = [] # 记忆化，记录该图片是否在md中已经插入过了
- img_ID = 0
-
- ## 获取、保存每张img的location信息(x1, y1, x2, y2， UL, DR坐标)
- for i in range(len(raw_imgs)):
- # 如果图片在junklist中则跳过
- if raw_imgs[i][0] in junk_img_bojids:
- continue
- else:
- try:
- tt = page.get_image_rects(raw_imgs[i][0], transform = True)
-
- rec = tt[0][0]
- L, U, R, D = int(rec[0]), int(rec[1]), int(rec[2]), int(rec[3])
-
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- if not(pageL <= L < R <= pageR and pageU <= U < D <= pageD):
- continue
- if pageL == L and R == pageR:
- continue
- if pageU == U and D == pageD:
- continue
- # pix1 = page.get_Pixmap(clip=(L,U,R,D))
- new_img_name = "{}_{}.png".format(page_ID, i) # 图片name
- # pix1.save(res_dir_path + '/' + new_img_name) # 把图片存出在新建的文件夹，并命名
- img_names.append(new_img_name)
- img_bboxs.append((L, U, R, D))
- img_visited.append(False)
- imgs.append(raw_imgs[i])
- except:
- continue
-
- #-------- 如果img之间有重叠。说明获取的img大小有问题，位置也不一定对。就扔掉--------#
- imgs_ok = [True for _ in range(len(imgs))]
- for i in range(len(imgs)):
- L1, U1, R1, D1 = img_bboxs[i]
- for j in range(i + 1, len(imgs)):
- L2, U2, R2, D2 = img_bboxs[j]
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- if ratio_1 > 0 and ratio_2 > 0:
- if ratio_1 == 1 and ratio_2 > 0.8:
- imgs_ok[i] = False
- elif ratio_1 > 0.8 and ratio_2 == 1:
- imgs_ok[j] = False
- elif s1 > 20000 and s2 > 20000 and ratio_1 > 0.4 and ratio_2 > 0.4:
- imgs_ok[i] = False
- imgs_ok[j] = False
- elif s1 / s2 > 5 and ratio_2 > 0.5:
- imgs_ok[j] = False
- elif s2 / s1 > 5 and ratio_1 > 0.5:
- imgs_ok[i] = False
-
- imgs = [imgs[i] for i in range(len(imgs)) if imgs_ok[i] == True]
- img_names = [img_names[i] for i in range(len(imgs)) if imgs_ok[i] == True]
- img_bboxs = [img_bboxs[i] for i in range(len(imgs)) if imgs_ok[i] == True]
- img_visited = [img_visited[i] for i in range(len(imgs)) if imgs_ok[i] == True]
- #*******************************************************************************#
-
- #---------------------------------------- 通过fitz提取svg的信息 -----------------------------------------#
- #
- svgs = page.get_drawings()
- #------------ preprocess, check一些大框，看是否是合理的 ----------#
- ## 去重。有时候会遇到rect1和rect2是完全一样的情形。
- svg_rect_visited = set()
- available_svgIdx = []
- for i in range(len(svgs)):
- L, U, R, D = svgs[i]['rect'].irect
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- tt = (L, U, R, D)
- if tt not in svg_rect_visited:
- svg_rect_visited.add(tt)
- available_svgIdx.append(i)
-
- svgs = [svgs[i] for i in available_svgIdx] # 去重后，有效的svgs
- svg_childs = [[] for _ in range(len(svgs))]
- svg_parents = [[] for _ in range(len(svgs))]
- svg_overlaps = [[] for _ in range(len(svgs))] #svg_overlaps[i]是一个list，存的是与svg_i有重叠的svg的index。e.g., svg_overlaps[0] = [1, 2, 7, 9]
- svg_visited = [False for _ in range(len(svgs))]
- svg_exceedPage = [0 for _ in range(len(svgs))] # 是否超越边界（artbox），很大，但一般是一个svg的底。
-
-
- for i in range(len(svgs)):
- L, U, R, D = svgs[i]['rect'].irect
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L, U, R, D, pageL, pageU, pageR, pageD)
- if (pageL + 20 < L <= R < pageR - 20) and (pageU + 20 < U <= D < pageD - 20):
- if ratio_2 >= 0.7:
- svg_exceedPage[i] += 4
- else:
- if L <= pageL:
- svg_exceedPage[i] += 1
- if pageR <= R:
- svg_exceedPage[i] += 1
- if U <= pageU:
- svg_exceedPage[i] += 1
- if pageD <= D:
- svg_exceedPage[i] += 1
-
- #### 如果有≥2个的超边界的框，就不要手写规则判断svg了。很难写对。
- if len([x for x in svg_exceedPage if x >= 1]) >= 2:
- svgs = []
- svg_childs = []
- svg_parents = []
- svg_overlaps = []
- svg_visited = []
- svg_exceedPage = []
-
- #---------------------------- build graph ----------------------------#
- for i, p in enumerate(svgs):
- L1, U1, R1, D1 = svgs[i]["rect"].irect
- for j in range(len(svgs)):
- if i == j:
- continue
- L2, U2, R2, D2 = svgs[j]["rect"].irect
- ## 包含
- if check_rect1_contains_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- svg_childs[i].append(j)
- svg_parents[j].append(i)
- else:
- ## 交叉
- if check_rect1_overlaps_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- svg_overlaps[i].append(j)
-
- #---------------- 确定最终的svg。连通块儿的外围 -------------------#
- eps_ERROR = 5 # 给识别出的svg，四周留白（为了防止pyMuPDF的rect不准）
- svg_ID = 0
- svg_final_names = []
- svg_final_bboxs = []
- svg_final_visited = [] # 为下面，text识别左准备。作用同img_visited
-
- svg_idxs = [i for i in range(len(svgs))]
- svg_idxs.sort(key = lambda i: -(svgs[i]['rect'].irect[2] - svgs[i]['rect'].irect[0]) * (svgs[i]['rect'].irect[3] - svgs[i]['rect'].irect[1])) # 按照面积，从大到小排序
-
- for i in svg_idxs:
- if svg_visited[i] == True:
- continue
- svg_visited[i] = True
- L, U, R, D = svgs[i]['rect'].irect
- width = R - L
- height = D - U
- if check_rect_isLine(L, U, R, D) == True:
- svg_visited[i] = False
- continue
- # if i == 4:
- # print(i, L, U, R, D)
- # print(svg_parents[i])
-
- cur_block_element_cnt = 0 # 当前要判定为svg的区域中，有多少elements，最外围的最大svg框除外。
- if len(svg_parents[i]) == 0:
- ## 是个普通框的情形
- cur_block_element_cnt += len(svg_childs[i])
- if svg_exceedPage[i] == 0:
- ## 误差。可能已经包含在某个框里面了
- neglect_flag = False
- for pL, pU, pR, pD in svg_final_bboxs:
- if pL <= L <= R <= pR and pU <= U <= D <= pD:
- neglect_flag = True
- break
- if neglect_flag == True:
- continue
-
- ## 搜索连通域, bfs+记忆化
- q = collections.deque()
- for j in svg_overlaps[i]:
- q.append(j)
- while q:
- j = q.popleft()
- svg_visited[j] = True
- L2, U2, R2, D2 = svgs[j]['rect'].irect
- # width2 = R2 - L2
- # height2 = D2 - U2
- # if width2 <= 2 or height2 <= 2 or (height2 / width2) >= 30 or (width2 / height2) >= 30:
- # continue
- L = min(L, L2)
- R = max(R, R2)
- U = min(U, U2)
- D = max(D, D2)
- cur_block_element_cnt += 1
- cur_block_element_cnt += len(svg_childs[j])
- for k in svg_overlaps[j]:
- if svg_visited[k] == False and svg_exceedPage[k] == 0:
- svg_visited[k] = True
- q.append(k)
- elif svg_exceedPage[i] <= 2:
- ## 误差。可能已经包含在某个svg_final_bbox框里面了
- neglect_flag = False
- for sL, sU, sR, sD in svg_final_bboxs:
- if sL <= L <= R <= sR and sU <= U <= D <= sD:
- neglect_flag = True
- break
- if neglect_flag == True:
- continue
-
- L, U, R, D = pageR, pageD, pageL, pageU
- ## 所有孩子元素的最大边界
- for j in svg_childs[i]:
- if svg_visited[j] == True:
- continue
- if svg_exceedPage[j] >= 1:
- continue
- svg_visited[j] = True #### 这个位置考虑一下
- L2, U2, R2, D2 = svgs[j]['rect'].irect
- L = min(L, L2)
- R = max(R, R2)
- U = min(U, U2)
- D = max(D, D2)
- cur_block_element_cnt += 1
-
- # 如果是条line，就不用保存了
- if check_rect_isLine(L, U, R, D) == True:
- continue
- # 如果当前的svg，连2个elements都没有，就不用保存了
- if cur_block_element_cnt < 3:
- continue
-
- ## 当前svg，框住了多少文本框。如果框多了，可能就是错了
- contain_textLineBlock_cnt = 0
- for L2, U2, R2, D2 in textLine_blocks:
- if check_rect1_contains_rect2(L, U, R, D, L2, U2, R2, D2) == True:
- contain_textLineBlock_cnt += 1
- if contain_textLineBlock_cnt >= 10:
- continue
-
- # L -= eps_ERROR * 2
- # U -= eps_ERROR
- # R += eps_ERROR * 2
- # D += eps_ERROR
- # # cur_svg = page.get_pixmap(matrix=fitz.Identity, dpi=None, colorspace=fitz.csRGB, clip=(U,L,R,D), alpha=False, annots=True)
- # cur_svg = page.get_pixmap(clip=(L,U,R,D))
- new_svg_name = "svg_{}_{}.png".format(page_ID, svg_ID) # 图片name
- # cur_svg.save(res_dir_path + '/' + new_svg_name) # 把图片存出在新建的文件夹，并命名
- svg_final_names.append(new_svg_name) # 把图片的名字存在list中，方便在md中插入引用
- svg_final_bboxs.append((L, U, R, D))
- svg_final_visited.append(False)
- svg_ID += 1
-
- ## 识别出的svg，可能有 包含，相邻的情形。需要进一步合并
- svg_idxs = [i for i in range(len(svg_final_bboxs))]
- svg_idxs.sort(key = lambda i: (svg_final_bboxs[i][1], svg_final_bboxs[i][0])) # (U, L)
- svg_final_names_2 = []
- svg_final_bboxs_2 = []
- svg_final_visited_2 = [] # 为下面，text识别左准备。作用同img_visited
- svg_ID_2 = 0
- for i in range(len(svg_final_bboxs)):
- L1, U1, R1, D1 = svg_final_bboxs[i]
- for j in range(i + 1, len(svg_final_bboxs)):
- L2, U2, R2, D2 = svg_final_bboxs[j]
- # 如果 rect1包含了rect2
- if check_rect1_contains_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- svg_final_visited[j] = True
- continue
- # 水平并列
- ratio_1, ratio_2 = calculate_overlapRatio_between_line1_and_line2(U1, D1, U2, D2)
- if ratio_1 >= 0.7 and ratio_2 >= 0.7:
- if abs(L2 - R1) >= 20:
- continue
- LL = min(L1, L2)
- UU = min(U1, U2)
- RR = max(R1, R2)
- DD = max(D1, D2)
- svg_final_bboxs[i] = (LL, UU, RR, DD)
- svg_final_visited[j] = True
- continue
- # 竖直并列
- ratio_1, ratio_2 = calculate_overlapRatio_between_line1_and_line2(L1, R2, L2, R2)
- if ratio_1 >= 0.7 and ratio_2 >= 0.7:
- if abs(U2 - D1) >= 20:
- continue
- LL = min(L1, L2)
- UU = min(U1, U2)
- RR = max(R1, R2)
- DD = max(D1, D2)
- svg_final_bboxs[i] = (LL, UU, RR, DD)
- svg_final_visited[j] = True
-
- for i in range(len(svg_final_bboxs)):
- if svg_final_visited[i] == False:
- L, U, R, D = svg_final_bboxs[i]
- svg_final_bboxs_2.append((L, U, R, D))
-
- L -= eps_ERROR * 2
- U -= eps_ERROR
- R += eps_ERROR * 2
- D += eps_ERROR
- # cur_svg = page.get_pixmap(clip=(L,U,R,D))
- new_svg_name = "svg_{}_{}.png".format(page_ID, svg_ID_2) # 图片name
- # cur_svg.save(res_dir_path + '/' + new_svg_name) # 把图片存出在新建的文件夹，并命名
- svg_final_names_2.append(new_svg_name) # 把图片的名字存在list中，方便在md中插入引用
- svg_final_bboxs_2.append((L, U, R, D))
- svg_final_visited_2.append(False)
- svg_ID_2 += 1
-
- ## svg收尾。识别为drawing，但是在上面没有拼成一张图的。
- # 有收尾才comprehensive
- # xxxx
- # xxxx
- # xxxx
- # xxxx
-
-
- #--------- 通过json_from_DocXchain来获取，figure, table, equation的bbox ---------#
- figure_bbox_from_DocXChain = []
-
- figure_from_DocXChain_visited = [] # 记忆化
- figure_bbox_from_DocXChain_overlappedRatio = []
-
- figure_only_from_DocXChain_bboxs = [] # 存储
- figure_only_from_DocXChain_names = []
- figure_only_from_DocXChain_visited = []
- figure_only_ID = 0
-
- xf_json = json_from_DocXchain_obj
- width_from_json = xf_json['page_info']['width']
- height_from_json = xf_json['page_info']['height']
- LR_scaleRatio = width_from_json / (pageR - pageL)
- UD_scaleRatio = height_from_json / (pageD - pageU)
-
- for xf in xf_json['layout_dets']:
- # {0: 'title', 1: 'figure', 2: 'plain text', 3: 'header', 4: 'page number', 5: 'footnote', 6: 'footer', 7: 'table', 8: 'table caption', 9: 'figure caption', 10: 'equation', 11: 'full column', 12: 'sub column'}
- L = xf['poly'][0] / LR_scaleRatio
- U = xf['poly'][1] / UD_scaleRatio
- R = xf['poly'][2] / LR_scaleRatio
- D = xf['poly'][5] / UD_scaleRatio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- # figure
- if xf["category_id"] == 1 and xf['score'] >= 0.3:
- figure_bbox_from_DocXChain.append((L, U, R, D))
- figure_from_DocXChain_visited.append(False)
- figure_bbox_from_DocXChain_overlappedRatio.append(0.0)
-
- #---------------------- 比对上面识别出来的img,svg 与DocXChain给的figure -----------------------#
-
- ## 比对imgs
- for i, b1 in enumerate(figure_bbox_from_DocXChain):
- # print('--------- DocXChain的图片', b1)
- L1, U1, R1, D1 = b1
- for b2 in img_bboxs:
- # print('-------- igms得到的图', b2)
- L2, U2, R2, D2 = b2
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- # 相同
- if check_rect1_sameWith_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- figure_from_DocXChain_visited[i] = True
- # 包含
- elif check_rect1_contains_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- if s2 / s1 > 0.8:
- figure_from_DocXChain_visited[i] = True
- elif check_rect1_contains_rect2(L2, U2, R2, D2, L1, U1, R1, D1) == True:
- if s1 / s2 > 0.8:
- figure_from_DocXChain_visited[i] = True
- else:
- # 重叠了相当一部分
- # print('进入第3部分')
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- if (ratio_1 >= 0.6 and ratio_2 >= 0.6) or (ratio_1 >= 0.8 and s1/s2>0.8) or (ratio_2 >= 0.8 and s2/s1>0.8):
- figure_from_DocXChain_visited[i] = True
- else:
- figure_bbox_from_DocXChain_overlappedRatio[i] += ratio_1
- # print('图片的重叠率是{}'.format(ratio_1))
-
-
- ## 比对svgs
- svg_final_bboxs_2_badIdxs = []
- for i, b1 in enumerate(figure_bbox_from_DocXChain):
- L1, U1, R1, D1 = b1
- for j, b2 in enumerate(svg_final_bboxs_2):
- L2, U2, R2, D2 = b2
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- # 相同
- if check_rect1_sameWith_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- figure_from_DocXChain_visited[i] = True
- # 包含
- elif check_rect1_contains_rect2(L1, U1, R1, D1, L2, U2, R2, D2) == True:
- figure_from_DocXChain_visited[i] = True
- elif check_rect1_contains_rect2(L2, U2, R2, D2, L1, U1, R1, D1) == True:
- if s1 / s2 > 0.7:
- figure_from_DocXChain_visited[i] = True
- else:
- svg_final_bboxs_2_badIdxs.append(j) # svg丢弃。用DocXChain的结果。
- else:
- # 重叠了相当一部分
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- if (ratio_1 >= 0.5 and ratio_2 >= 0.5) or (min(ratio_1, ratio_2) >= 0.4 and max(ratio_1, ratio_2) >= 0.6):
- figure_from_DocXChain_visited[i] = True
- else:
- figure_bbox_from_DocXChain_overlappedRatio[i] += ratio_1
-
- # 丢掉错误的svg
- svg_final_bboxs_2 = [svg_final_bboxs_2[i] for i in range(len(svg_final_bboxs_2)) if i not in set(svg_final_bboxs_2_badIdxs)]
-
- for i in range(len(figure_from_DocXChain_visited)):
- if figure_bbox_from_DocXChain_overlappedRatio[i] >= 0.7:
- figure_from_DocXChain_visited[i] = True
-
- # DocXChain识别出来的figure，但是没被保存的。
- for i in range(len(figure_from_DocXChain_visited)):
- if figure_from_DocXChain_visited[i] == False:
- figure_from_DocXChain_visited[i] = True
- cur_bbox = figure_bbox_from_DocXChain[i]
- # cur_figure = page.get_pixmap(clip=cur_bbox)
- new_figure_name = "figure_only_{}_{}.png".format(page_ID, figure_only_ID) # 图片name
- # cur_figure.save(res_dir_path + '/' + new_figure_name) # 把图片存出在新建的文件夹，并命名
- figure_only_from_DocXChain_names.append(new_figure_name) # 把图片的名字存在list中，方便在md中插入引用
- figure_only_from_DocXChain_bboxs.append(cur_bbox)
- figure_only_from_DocXChain_visited.append(False)
- figure_only_ID += 1
-
- img_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- svg_final_bboxs_2.sort(key = lambda LURD: (LURD[1], LURD[0]))
- figure_only_from_DocXChain_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_fig_bboxs = img_bboxs + svg_final_bboxs + figure_only_from_DocXChain_bboxs
-
- #--------------------------- 最后统一去重 -----------------------------------#
- curPage_all_fig_bboxs.sort(key = lambda LURD: ( (LURD[2]-LURD[0])*(LURD[3]-LURD[1]) , LURD[0], LURD[1]) )
-
- #### 先考虑包含关系的小块
- final_duplicate = set()
- for i in range(len(curPage_all_fig_bboxs)):
- L1, U1, R1, D1 = curPage_all_fig_bboxs[i]
- for j in range(len(curPage_all_fig_bboxs)):
- if i == j:
- continue
- L2, U2, R2, D2 = curPage_all_fig_bboxs[j]
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- if check_rect1_contains_rect2(L2, U2, R2, D2, L1, U1, R1, D1) == True:
- final_duplicate.add((L1, U1, R1, D1))
- else:
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- if ratio_1 >= 0.8 and ratio_2 <= 0.6:
- final_duplicate.add((L1, U1, R1, D1))
-
- curPage_all_fig_bboxs = [LURD for LURD in curPage_all_fig_bboxs if LURD not in final_duplicate]
-
- #### 再考虑重叠关系的块
- final_duplicate = set()
- final_synthetic_bboxs = []
- for i in range(len(curPage_all_fig_bboxs)):
- L1, U1, R1, D1 = curPage_all_fig_bboxs[i]
- for j in range(len(curPage_all_fig_bboxs)):
- if i == j:
- continue
- L2, U2, R2, D2 = curPage_all_fig_bboxs[j]
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- union_ok = False
- if (ratio_1 >= 0.8 and ratio_2 <= 0.6) or (ratio_1 > 0.6 and ratio_2 > 0.6):
- union_ok = True
- if (ratio_1 > 0.2 and s2 / s1 > 5):
- union_ok = True
- if (L1 <= (L2+R2)/2 <= R1) and (U1 <= (U2+D2)/2 <= D1):
- union_ok = True
- if (L2 <= (L1+R1)/2 <= R2) and (U2 <= (U1+D1)/2 <= D2):
- union_ok = True
- if union_ok == True:
- final_duplicate.add((L1, U1, R1, D1))
- final_duplicate.add((L2, U2, R2, D2))
- L3, U3, R3, D3 = min(L1, L2), min(U1, U2), max(R1, R2), max(D1, D2)
- final_synthetic_bboxs.append((L3, U3, R3, D3))
-
- # print('---------- curPage_all_fig_bboxs ---------')
- # print(curPage_all_fig_bboxs)
- curPage_all_fig_bboxs = [b for b in curPage_all_fig_bboxs if b not in final_duplicate]
- final_synthetic_bboxs = list(set(final_synthetic_bboxs))
-
-
- ## 再再考虑重叠关系。极端情况下会迭代式地2进1
- new_images = []
- droped_img_idx = []
- image_bboxes = [[b[0], b[1], b[2], b[3]] for b in final_synthetic_bboxs]
- for i in range(0, len(image_bboxes)):
- for j in range(i+1, len(image_bboxes)):
- if j not in droped_img_idx:
- L2, U2, R2, D2 = image_bboxes[j]
- s1 = abs(R1 - L1) * abs(D1 - U1)
- s2 = abs(R2 - L2) * abs(D2 - U2)
- ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- union_ok = False
- if (ratio_1 >= 0.8 and ratio_2 <= 0.6) or (ratio_1 > 0.6 and ratio_2 > 0.6):
- union_ok = True
- if (ratio_1 > 0.2 and s2 / s1 > 5):
- union_ok = True
- if (L1 <= (L2+R2)/2 <= R1) and (U1 <= (U2+D2)/2 <= D1):
- union_ok = True
- if (L2 <= (L1+R1)/2 <= R2) and (U2 <= (U1+D1)/2 <= D2):
- union_ok = True
- if union_ok == True:
- # 合并
- image_bboxes[i][0], image_bboxes[i][1],image_bboxes[i][2],image_bboxes[i][3] = min(image_bboxes[i][0], image_bboxes[j][0]), min(image_bboxes[i][1], image_bboxes[j][1]), max(image_bboxes[i][2], image_bboxes[j][2]), max(image_bboxes[i][3], image_bboxes[j][3])
- droped_img_idx.append(j)
-
- for i in range(0, len(image_bboxes)):
- if i not in droped_img_idx:
- new_images.append(image_bboxes[i])
-
-
- # find_union_FLAG = True
- # while find_union_FLAG == True:
- # find_union_FLAG = False
- # final_duplicate = set()
- # tmp = []
- # for i in range(len(final_synthetic_bboxs)):
- # L1, U1, R1, D1 = final_synthetic_bboxs[i]
- # for j in range(len(final_synthetic_bboxs)):
- # if i == j:
- # continue
- # L2, U2, R2, D2 = final_synthetic_bboxs[j]
- # s1 = abs(R1 - L1) * abs(D1 - U1)
- # s2 = abs(R2 - L2) * abs(D2 - U2)
- # ratio_1, ratio_2 = calculate_overlapRatio_between_rect1_and_rect2(L1, U1, R1, D1, L2, U2, R2, D2)
- # union_ok = False
- # if (ratio_1 >= 0.8 and ratio_2 <= 0.6) or (ratio_1 > 0.6 and ratio_2 > 0.6):
- # union_ok = True
- # if (ratio_1 > 0.2 and s2 / s1 > 5):
- # union_ok = True
- # if (L1 <= (L2+R2)/2 <= R1) and (U1 <= (U2+D2)/2 <= D1):
- # union_ok = True
- # if (L2 <= (L1+R1)/2 <= R2) and (U2 <= (U1+D1)/2 <= D2):
- # union_ok = True
- # if union_ok == True:
- # find_union_FLAG = True
- # final_duplicate.add((L1, U1, R1, D1))
- # final_duplicate.add((L2, U2, R2, D2))
- # L3, U3, R3, D3 = min(L1, L2), min(U1, U2), max(R1, R2), max(D1, D2)
- # tmp.append((L3, U3, R3, D3))
- # if find_union_FLAG == True:
- # tmp = list(set(tmp))
- # final_synthetic_bboxs = tmp[:]
-
-
- # curPage_all_fig_bboxs += final_synthetic_bboxs
- # print('--------- final synthetic')
- # print(final_synthetic_bboxs)
- #**************************************************************************#
- images1 = [[img[0], img[1], img[2], img[3]] for img in curPage_all_fig_bboxs]
- images = images1 + new_images
- return images
-
diff --git a/magic_pdf/pre_proc/detect_page_number.py b/magic_pdf/pre_proc/detect_page_number.py
deleted file mode 100644
index 35920a99b..000000000
--- a/magic_pdf/pre_proc/detect_page_number.py
+++ /dev/null
@@ -1,64 +0,0 @@
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-
-
-def parse_pageNos(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
-
- #--------- 通过json_from_DocXchain来获取 pageNo ---------#
- pageNo_bbox_from_DocXChain = []
-
- xf_json = json_from_DocXchain_obj
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(xf_json, page)
-
- # {0: 'title', # 标题
- # 1: 'figure', # 图片
- # 2: 'plain text', # 文本
- # 3: 'header', # 页眉
- # 4: 'page number', # 页码
- # 5: 'footnote', # 脚注
- # 6: 'footer', # 页脚
- # 7: 'table', # 表格
- # 8: 'table caption', # 表格描述
- # 9: 'figure caption', # 图片描述
- # 10: 'equation', # 公式
- # 11: 'full column', # 单栏
- # 12: 'sub column', # 多栏
- # 13: 'embedding', # 嵌入公式
- # 14: 'isolated'} # 单行公式
- for xf in xf_json['layout_dets']:
- L = xf['poly'][0] / horizontal_scale_ratio
- U = xf['poly'][1] / vertical_scale_ratio
- R = xf['poly'][2] / horizontal_scale_ratio
- D = xf['poly'][5] / vertical_scale_ratio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- if xf['category_id'] == 4 and xf['score'] >= 0.3:
- pageNo_bbox_from_DocXChain.append((L, U, R, D))
-
-
- pageNo_final_names = []
- pageNo_final_bboxs = []
- pageNo_ID = 0
- for L, U, R, D in pageNo_bbox_from_DocXChain:
- # cur_pageNo = page.get_pixmap(clip=(L,U,R,D))
- new_pageNo_name = "pageNo_{}_{}.png".format(page_ID, pageNo_ID) # 页码name
- # cur_pageNo.save(res_dir_path + '/' + new_pageNo_name) # 把页码存储在新建的文件夹，并命名
- pageNo_final_names.append(new_pageNo_name) # 把页码的名字存在list中
- pageNo_final_bboxs.append((L, U, R, D))
- pageNo_ID += 1
-
-
- pageNo_final_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_pageNo_bboxs = pageNo_final_bboxs
- return curPage_all_pageNo_bboxs
-
diff --git a/magic_pdf/pre_proc/detect_tables.py b/magic_pdf/pre_proc/detect_tables.py
deleted file mode 100644
index fc2992eec..000000000
--- a/magic_pdf/pre_proc/detect_tables.py
+++ /dev/null
@@ -1,62 +0,0 @@
-from magic_pdf.libs.commons import fitz # pyMuPDF库
-
-
-def parse_tables(page_ID: int, page: fitz.Page, json_from_DocXchain_obj: dict):
- """
- :param page_ID: int类型，当前page在当前pdf文档中是第page_D页。
- :param page :fitz读取的当前页的内容
- :param res_dir_path: str类型，是每一个pdf文档，在当前.py文件的目录下生成一个与pdf文档同名的文件夹，res_dir_path就是文件夹的dir
- :param json_from_DocXchain_obj: dict类型，把pdf文档送入DocXChain模型中后，提取bbox，结果保存到pdf文档同名文件夹下的 page_ID.json文件中了。json_from_DocXchain_obj就是打开后的dict
- """
- DPI = 72 # use this resolution
- pix = page.get_pixmap(dpi=DPI)
- pageL = 0
- pageR = int(pix.w)
- pageU = 0
- pageD = int(pix.h)
-
-
- #--------- 通过json_from_DocXchain来获取 table ---------#
- table_bbox_from_DocXChain = []
-
- xf_json = json_from_DocXchain_obj
- width_from_json = xf_json['page_info']['width']
- height_from_json = xf_json['page_info']['height']
- LR_scaleRatio = width_from_json / (pageR - pageL)
- UD_scaleRatio = height_from_json / (pageD - pageU)
-
-
- for xf in xf_json['layout_dets']:
- # {0: 'title', 1: 'figure', 2: 'plain text', 3: 'header', 4: 'page number', 5: 'footnote', 6: 'footer', 7: 'table', 8: 'table caption', 9: 'figure caption', 10: 'equation', 11: 'full column', 12: 'sub column'}
- # 13: 'embedding', # 嵌入公式
- # 14: 'isolated'} # 单行公式
- L = xf['poly'][0] / LR_scaleRatio
- U = xf['poly'][1] / UD_scaleRatio
- R = xf['poly'][2] / LR_scaleRatio
- D = xf['poly'][5] / UD_scaleRatio
- # L += pageL # 有的页面，artBox偏移了。不在（0,0）
- # R += pageL
- # U += pageU
- # D += pageU
- L, R = min(L, R), max(L, R)
- U, D = min(U, D), max(U, D)
- if xf['category_id'] == 7 and xf['score'] >= 0.3:
- table_bbox_from_DocXChain.append((L, U, R, D))
-
-
- table_final_names = []
- table_final_bboxs = []
- table_ID = 0
- for L, U, R, D in table_bbox_from_DocXChain:
- # cur_table = page.get_pixmap(clip=(L,U,R,D))
- new_table_name = "table_{}_{}.png".format(page_ID, table_ID) # 表格name
- # cur_table.save(res_dir_path + '/' + new_table_name) # 把表格存出在新建的文件夹，并命名
- table_final_names.append(new_table_name) # 把表格的名字存在list中，方便在md中插入引用
- table_final_bboxs.append((L, U, R, D))
- table_ID += 1
-
-
- table_final_bboxs.sort(key = lambda LURD: (LURD[1], LURD[0]))
- curPage_all_table_bboxs = table_final_bboxs
- return curPage_all_table_bboxs
-
diff --git a/magic_pdf/pre_proc/equations_replace.py b/magic_pdf/pre_proc/equations_replace.py
deleted file mode 100644
index 289fbf3fe..000000000
--- a/magic_pdf/pre_proc/equations_replace.py
+++ /dev/null
@@ -1,559 +0,0 @@
-"""
-对pymupdf返回的结构里的公式进行替换，替换为模型识别的公式结果
-"""
-
-from magic_pdf.libs.commons import fitz
-import json
-import os
-from pathlib import Path
-from loguru import logger
-from magic_pdf.libs.ocr_content_type import ContentType
-
-TYPE_INLINE_EQUATION = ContentType.InlineEquation
-TYPE_INTERLINE_EQUATION = ContentType.InterlineEquation
-
-
-def combine_chars_to_pymudict(block_dict, char_dict):
- """
- 把block级别的pymupdf 结构里加入char结构
- """
- # 因为block_dict 被裁剪过，因此先把他和char_dict文字块对齐，才能进行补充
- char_map = {tuple(item["bbox"]): item for item in char_dict}
-
- for i in range(len(block_dict)): # blcok
- block = block_dict[i]
- key = block["bbox"]
- char_dict_item = char_map[tuple(key)]
- char_dict_map = {tuple(item["bbox"]): item for item in char_dict_item["lines"]}
- for j in range(len(block["lines"])):
- lines = block["lines"][j]
- with_char_lines = char_dict_map[lines["bbox"]]
- for k in range(len(lines["spans"])):
- spans = lines["spans"][k]
- try:
- chars = with_char_lines["spans"][k]["chars"]
- except Exception as e:
- logger.error(char_dict[i]["lines"][j])
-
- spans["chars"] = chars
-
- return block_dict
-
-
-def calculate_overlap_area_2_minbox_area_ratio(bbox1, min_bbox):
- """
- 计算box1和box2的重叠面积占最小面积的box的比例
- """
- # Determine the coordinates of the intersection rectangle
- x_left = max(bbox1[0], min_bbox[0])
- y_top = max(bbox1[1], min_bbox[1])
- x_right = min(bbox1[2], min_bbox[2])
- y_bottom = min(bbox1[3], min_bbox[3])
-
- if x_right < x_left or y_bottom < y_top:
- return 0.0
-
- # The area of overlap area
- intersection_area = (x_right - x_left) * (y_bottom - y_top)
- min_box_area = (min_bbox[3] - min_bbox[1]) * (min_bbox[2] - min_bbox[0])
- if min_box_area == 0:
- return 0
- else:
- return intersection_area / min_box_area
-
-
-def _is_xin(bbox1, bbox2):
- area1 = abs(bbox1[2] - bbox1[0]) * abs(bbox1[3] - bbox1[1])
- area2 = abs(bbox2[2] - bbox2[0]) * abs(bbox2[3] - bbox2[1])
- if area1 < area2:
- ratio = calculate_overlap_area_2_minbox_area_ratio(bbox2, bbox1)
- else:
- ratio = calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2)
-
- return ratio > 0.6
-
-
-def remove_text_block_in_interline_equation_bbox(interline_bboxes, text_blocks):
- """消除掉整个块都在行间公式块内部的文本块"""
- for eq_bbox in interline_bboxes:
- removed_txt_blk = []
- for text_blk in text_blocks:
- text_bbox = text_blk["bbox"]
- if (
- calculate_overlap_area_2_minbox_area_ratio(eq_bbox["bbox"], text_bbox)
- >= 0.7
- ):
- removed_txt_blk.append(text_blk)
- for blk in removed_txt_blk:
- text_blocks.remove(blk)
-
- return text_blocks
-
-
-def _is_in_or_part_overlap(box1, box2) -> bool:
- """
- 两个bbox是否有部分重叠或者包含
- """
- if box1 is None or box2 is None:
- return False
-
- x0_1, y0_1, x1_1, y1_1 = box1
- x0_2, y0_2, x1_2, y1_2 = box2
-
- return not (
- x1_1 < x0_2 # box1在box2的左边
- or x0_1 > x1_2 # box1在box2的右边
- or y1_1 < y0_2 # box1在box2的上边
- or y0_1 > y1_2
- ) # box1在box2的下边
-
-
-def remove_text_block_overlap_interline_equation_bbox(
- interline_eq_bboxes, pymu_block_list
-):
-
- """消除掉行行内公式有部分重叠的文本块的内容。
- 同时重新计算消除重叠之后文本块的大小"""
- deleted_block = []
- for text_block in pymu_block_list:
- deleted_line = []
- for line in text_block["lines"]:
- deleted_span = []
- for span in line["spans"]:
- deleted_chars = []
- for char in span["chars"]:
- if any(
- [
- (calculate_overlap_area_2_minbox_area_ratio(eq_bbox["bbox"], char["bbox"]) > 0.5)
- for eq_bbox in interline_eq_bboxes
- ]
- ):
- deleted_chars.append(char)
- # 检查span里没有char则删除这个span
- for char in deleted_chars:
- span["chars"].remove(char)
- # 重新计算这个span的大小
- if len(span["chars"]) == 0: # 删除这个span
- deleted_span.append(span)
- else:
- span["bbox"] = (
- min([b["bbox"][0] for b in span["chars"]]),
- min([b["bbox"][1] for b in span["chars"]]),
- max([b["bbox"][2] for b in span["chars"]]),
- max([b["bbox"][3] for b in span["chars"]]),
- )
-
- # 检查这个span
- for span in deleted_span:
- line["spans"].remove(span)
- if len(line["spans"]) == 0: # 删除这个line
- deleted_line.append(line)
- else:
- line["bbox"] = (
- min([b["bbox"][0] for b in line["spans"]]),
- min([b["bbox"][1] for b in line["spans"]]),
- max([b["bbox"][2] for b in line["spans"]]),
- max([b["bbox"][3] for b in line["spans"]]),
- )
-
- # 检查这个block是否可以删除
- for line in deleted_line:
- text_block["lines"].remove(line)
- if len(text_block["lines"]) == 0: # 删除block
- deleted_block.append(text_block)
- else:
- text_block["bbox"] = (
- min([b["bbox"][0] for b in text_block["lines"]]),
- min([b["bbox"][1] for b in text_block["lines"]]),
- max([b["bbox"][2] for b in text_block["lines"]]),
- max([b["bbox"][3] for b in text_block["lines"]]),
- )
-
- # 检查text block删除
- for block in deleted_block:
- pymu_block_list.remove(block)
- if len(pymu_block_list) == 0:
- return []
-
- return pymu_block_list
-
-
-def insert_interline_equations_textblock(interline_eq_bboxes, pymu_block_list):
- """在行间公式对应的地方插上一个伪造的block"""
- for eq in interline_eq_bboxes:
- bbox = eq["bbox"]
- latex_content = eq["latex"]
- text_block = {
- "number": len(pymu_block_list),
- "type": 0,
- "bbox": bbox,
- "lines": [
- {
- "spans": [
- {
- "size": 9.962599754333496,
- "type": TYPE_INTERLINE_EQUATION,
- "flags": 4,
- "font": TYPE_INTERLINE_EQUATION,
- "color": 0,
- "ascender": 0.9409999847412109,
- "descender": -0.3050000071525574,
- "latex": latex_content,
- "origin": [bbox[0], bbox[1]],
- "bbox": bbox,
- }
- ],
- "wmode": 0,
- "dir": [1.0, 0.0],
- "bbox": bbox,
- }
- ],
- }
- pymu_block_list.append(text_block)
-
-
-def x_overlap_ratio(box1, box2):
- a, _, c, _ = box1
- e, _, g, _ = box2
-
- # 计算重叠宽度
- overlap_x = max(min(c, g) - max(a, e), 0)
-
- # 计算box1的宽度
- width1 = g - e
-
- # 计算重叠比例
- overlap_ratio = overlap_x / width1 if width1 != 0 else 0
-
- return overlap_ratio
-
-
-def __is_x_dir_overlap(bbox1, bbox2):
- return not (bbox1[2] < bbox2[0] or bbox1[0] > bbox2[2])
-
-
-def __y_overlap_ratio(box1, box2):
- """"""
- _, b, _, d = box1
- _, f, _, h = box2
-
- # 计算重叠高度
- overlap_y = max(min(d, h) - max(b, f), 0)
-
- # 计算box1的高度
- height1 = d - b
-
- # 计算重叠比例
- overlap_ratio = overlap_y / height1 if height1 != 0 else 0
-
- return overlap_ratio
-
-
-def replace_line_v2(eqinfo, line):
- """
- 扫描这一行所有的和公式框X方向重叠的char,然后计算char的左、右x0, x1,位于这个区间内的span删除掉。
- 最后与这个x0,x1有相交的span0, span1内部进行分割。
- """
- first_overlap_span = -1
- first_overlap_span_idx = -1
- last_overlap_span = -1
- delete_chars = []
- for i in range(0, len(line["spans"])):
- if "chars" not in line["spans"][i]:
- continue
-
- if line["spans"][i].get("_type", None) is not None:
- continue # 忽略，因为已经是插入的伪造span公式了
-
- for char in line["spans"][i]["chars"]:
- if __is_x_dir_overlap(eqinfo["bbox"], char["bbox"]):
- line_txt = ""
- for span in line["spans"]:
- span_txt = ""
- for ch in span["chars"]:
- span_txt = span_txt + ch["c"]
-
- span_txt = span_txt + ""
-
- line_txt = line_txt + span_txt
-
- if first_overlap_span_idx == -1:
- first_overlap_span = line["spans"][i]
- first_overlap_span_idx = i
- last_overlap_span = line["spans"][i]
- delete_chars.append(char)
-
- # 第一个和最后一个char要进行检查，到底属于公式多还是属于正常span多
- if len(delete_chars) > 0:
- ch0_bbox = delete_chars[0]["bbox"]
- if x_overlap_ratio(eqinfo["bbox"], ch0_bbox) < 0.51:
- delete_chars.remove(delete_chars[0])
- if len(delete_chars) > 0:
- ch0_bbox = delete_chars[-1]["bbox"]
- if x_overlap_ratio(eqinfo["bbox"], ch0_bbox) < 0.51:
- delete_chars.remove(delete_chars[-1])
-
- # 计算x方向上被删除区间内的char的真实x0, x1
- if len(delete_chars):
- x0, x1 = min([b["bbox"][0] for b in delete_chars]), max(
- [b["bbox"][2] for b in delete_chars]
- )
- else:
- # logger.debug(f"行内公式替换没有发生，尝试下一行匹配, eqinfo={eqinfo}")
- return False
-
- # 删除位于x0, x1这两个中间的span
- delete_span = []
- for span in line["spans"]:
- span_box = span["bbox"]
- if x0 <= span_box[0] and span_box[2] <= x1:
- delete_span.append(span)
- for span in delete_span:
- line["spans"].remove(span)
-
- equation_span = {
- "size": 9.962599754333496,
- "type": TYPE_INLINE_EQUATION,
- "flags": 4,
- "font": TYPE_INLINE_EQUATION,
- "color": 0,
- "ascender": 0.9409999847412109,
- "descender": -0.3050000071525574,
- "latex": "",
- "origin": [337.1410153102337, 216.0205245153934],
- "bbox": eqinfo["bbox"]
- }
- # equation_span = line['spans'][0].copy()
- equation_span["latex"] = eqinfo['latex']
- equation_span["bbox"] = [x0, equation_span["bbox"][1], x1, equation_span["bbox"][3]]
- equation_span["origin"] = [equation_span["bbox"][0], equation_span["bbox"][1]]
- equation_span["chars"] = delete_chars
- equation_span["type"] = TYPE_INLINE_EQUATION
- equation_span["_eq_bbox"] = eqinfo["bbox"]
- line["spans"].insert(first_overlap_span_idx + 1, equation_span) # 放入公式
-
- # logger.info(f"==>text is 【{line_txt}】, equation is 【{eqinfo['latex_text']}】")
-
- # 第一个、和最后一个有overlap的span进行分割,然后插入对应的位置
- first_span_chars = [
- char
- for char in first_overlap_span["chars"]
- if (char["bbox"][2] + char["bbox"][0]) / 2 < x0
- ]
- tail_span_chars = [
- char
- for char in last_overlap_span["chars"]
- if (char["bbox"][0] + char["bbox"][2]) / 2 > x1
- ]
-
- if len(first_span_chars) > 0:
- first_overlap_span["chars"] = first_span_chars
- first_overlap_span["text"] = "".join([char["c"] for char in first_span_chars])
- first_overlap_span["bbox"] = (
- first_overlap_span["bbox"][0],
- first_overlap_span["bbox"][1],
- max([chr["bbox"][2] for chr in first_span_chars]),
- first_overlap_span["bbox"][3],
- )
- # first_overlap_span['_type'] = "first"
- else:
- # 删掉
- if first_overlap_span not in delete_span:
- line["spans"].remove(first_overlap_span)
-
- if len(tail_span_chars) > 0:
- min_of_tail_span_x0 = min([chr["bbox"][0] for chr in tail_span_chars])
- min_of_tail_span_y0 = min([chr["bbox"][1] for chr in tail_span_chars])
- max_of_tail_span_x1 = max([chr["bbox"][2] for chr in tail_span_chars])
- max_of_tail_span_y1 = max([chr["bbox"][3] for chr in tail_span_chars])
-
- if last_overlap_span == first_overlap_span: # 这个时候应该插入一个新的
- tail_span_txt = "".join([char["c"] for char in tail_span_chars])
- last_span_to_insert = last_overlap_span.copy()
- last_span_to_insert["chars"] = tail_span_chars
- last_span_to_insert["text"] = "".join(
- [char["c"] for char in tail_span_chars]
- )
- if equation_span["bbox"][2] >= last_overlap_span["bbox"][2]:
- last_span_to_insert["bbox"] = (
- min_of_tail_span_x0,
- min_of_tail_span_y0,
- max_of_tail_span_x1,
- max_of_tail_span_y1
- )
- else:
- last_span_to_insert["bbox"] = (
- min([chr["bbox"][0] for chr in tail_span_chars]),
- last_overlap_span["bbox"][1],
- last_overlap_span["bbox"][2],
- last_overlap_span["bbox"][3],
- )
- # 插入到公式对象之后
- equation_idx = line["spans"].index(equation_span)
- line["spans"].insert(equation_idx + 1, last_span_to_insert) # 放入公式
- else: # 直接修改原来的span
- last_overlap_span["chars"] = tail_span_chars
- last_overlap_span["text"] = "".join([char["c"] for char in tail_span_chars])
- last_overlap_span["bbox"] = (
- min([chr["bbox"][0] for chr in tail_span_chars]),
- last_overlap_span["bbox"][1],
- last_overlap_span["bbox"][2],
- last_overlap_span["bbox"][3],
- )
- else:
- # 删掉
- if (
- last_overlap_span not in delete_span
- and last_overlap_span != first_overlap_span
- ):
- line["spans"].remove(last_overlap_span)
-
- remain_txt = ""
- for span in line["spans"]:
- span_txt = ""
- for char in span["chars"]:
- span_txt = span_txt + char["c"]
-
- span_txt = span_txt + ""
-
- remain_txt = remain_txt + span_txt
-
- # logger.info(f"<== succ replace, text is 【{remain_txt}】, equation is 【{eqinfo['latex_text']}】")
-
- return True
-
-
-def replace_eq_blk(eqinfo, text_block):
- """替换行内公式"""
- for line in text_block["lines"]:
- line_bbox = line["bbox"]
- if (
- _is_xin(eqinfo["bbox"], line_bbox)
- or __y_overlap_ratio(eqinfo["bbox"], line_bbox) > 0.6
- ): # 定位到行, 使用y方向重合率是因为有的时候，一个行的宽度会小于公式位置宽度：行很高，公式很窄，
- replace_succ = replace_line_v2(eqinfo, line)
- if (
- not replace_succ
- ): # 有的时候，一个pdf的line高度从API里会计算的有问题，因此在行内span级别会替换不成功，这就需要继续重试下一行
- continue
- else:
- break
- else:
- return False
- return True
-
-
-def replace_inline_equations(inline_equation_bboxes, raw_text_blocks):
- """替换行内公式"""
- for eqinfo in inline_equation_bboxes:
- eqbox = eqinfo["bbox"]
- for blk in raw_text_blocks:
- if _is_xin(eqbox, blk["bbox"]):
- if not replace_eq_blk(eqinfo, blk):
- logger.warning(f"行内公式没有替换成功：{eqinfo} ")
- else:
- break
-
- return raw_text_blocks
-
-
-def remove_chars_in_text_blocks(text_blocks):
- """删除text_blocks里的char"""
- for blk in text_blocks:
- for line in blk["lines"]:
- for span in line["spans"]:
- _ = span.pop("chars", "no such key")
- return text_blocks
-
-
-def replace_equations_in_textblock(
- raw_text_blocks, inline_equation_bboxes, interline_equation_bboxes
-):
- """
- 替换行间和和行内公式为latex
- """
- raw_text_blocks = remove_text_block_in_interline_equation_bbox(
- interline_equation_bboxes, raw_text_blocks
- ) # 消除重叠：第一步，在公式内部的
-
- raw_text_blocks = remove_text_block_overlap_interline_equation_bbox(
- interline_equation_bboxes, raw_text_blocks
- ) # 消重，第二步，和公式覆盖的
-
- insert_interline_equations_textblock(interline_equation_bboxes, raw_text_blocks)
- raw_text_blocks = replace_inline_equations(inline_equation_bboxes, raw_text_blocks)
- return raw_text_blocks
-
-
-def draw_block_on_pdf_with_txt_replace_eq_bbox(json_path, pdf_path):
- """ """
- new_pdf = f"{Path(pdf_path).parent}/{Path(pdf_path).stem}.step3-消除行内公式text_block.pdf"
- with open(json_path, "r", encoding="utf-8") as f:
- obj = json.loads(f.read())
-
- if os.path.exists(new_pdf):
- os.remove(new_pdf)
- new_doc = fitz.open("")
-
- doc = fitz.open(pdf_path)
- new_doc = fitz.open(pdf_path)
- for i in range(len(new_doc)):
- page = new_doc[i]
- inline_equation_bboxes = obj[f"page_{i}"]["inline_equations"]
- interline_equation_bboxes = obj[f"page_{i}"]["interline_equations"]
- raw_text_blocks = obj[f"page_{i}"]["preproc_blocks"]
- raw_text_blocks = remove_text_block_in_interline_equation_bbox(
- interline_equation_bboxes, raw_text_blocks
- ) # 消除重叠：第一步，在公式内部的
- raw_text_blocks = remove_text_block_overlap_interline_equation_bbox(
- interline_equation_bboxes, raw_text_blocks
- ) # 消重，第二步，和公式覆盖的
- insert_interline_equations_textblock(interline_equation_bboxes, raw_text_blocks)
- raw_text_blocks = replace_inline_equations(
- inline_equation_bboxes, raw_text_blocks
- )
-
- # 为了检验公式是否重复，把每一行里，含有公式的span背景改成黄色的
- color_map = [fitz.pdfcolor["blue"], fitz.pdfcolor["green"]]
- j = 0
- for blk in raw_text_blocks:
- for i, line in enumerate(blk["lines"]):
-
- # line_box = line['bbox']
- # shape = page.new_shape()
- # shape.draw_rect(line_box)
- # shape.finish(color=fitz.pdfcolor['red'], fill=color_map[j%2], fill_opacity=0.3)
- # shape.commit()
- # j = j+1
-
- for i, span in enumerate(line["spans"]):
- shape_page = page.new_shape()
- span_type = span.get("_type")
- color = fitz.pdfcolor["blue"]
- if span_type == "first":
- color = fitz.pdfcolor["blue"]
- elif span_type == "tail":
- color = fitz.pdfcolor["green"]
- elif span_type == TYPE_INLINE_EQUATION:
- color = fitz.pdfcolor["black"]
- else:
- color = None
-
- b = span["bbox"]
- shape_page.draw_rect(b)
-
- shape_page.finish(color=None, fill=color, fill_opacity=0.3)
- shape_page.commit()
-
- new_doc.save(new_pdf)
- logger.info(f"save ok {new_pdf}")
- final_json = json.dumps(obj, ensure_ascii=False, indent=2)
- with open("equations_test/final_json.json", "w") as f:
- f.write(final_json)
-
- return new_pdf
-
-
-if __name__ == "__main__":
- # draw_block_on_pdf_with_txt_replace_eq_bbox(new_json_path, equation_color_pdf)
- pass
diff --git a/magic_pdf/pre_proc/fix_image.py b/magic_pdf/pre_proc/fix_image.py
deleted file mode 100644
index d2f83570d..000000000
--- a/magic_pdf/pre_proc/fix_image.py
+++ /dev/null
@@ -1,244 +0,0 @@
-
-
-
-import re
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap, _is_part_overlap, find_bottom_nearest_text_bbox, find_left_nearest_text_bbox, find_right_nearest_text_bbox, find_top_nearest_text_bbox
-
-from magic_pdf.libs.textbase import get_text_block_base_info
-
-def fix_image_vertical(image_bboxes:list, text_blocks:list):
- """
- 修正图片的位置
- 如果图片与文字block发生一定重叠（也就是图片切到了一部分文字），那么减少图片边缘，让文字和图片不再重叠。
- 只对垂直方向进行。
- """
- for image_bbox in image_bboxes:
- for text_block in text_blocks:
- text_bbox = text_block["bbox"]
- if _is_part_overlap(text_bbox, image_bbox) and any([text_bbox[0]>=image_bbox[0] and text_bbox[2]<=image_bbox[2], text_bbox[0]<=image_bbox[0] and text_bbox[2]>=image_bbox[2]]):
- if text_bbox[1] < image_bbox[1]:#在图片上方
- image_bbox[1] = text_bbox[3]+1
- elif text_bbox[3]>image_bbox[3]:#在图片下方
- image_bbox[3] = text_bbox[1]-1
-
- return image_bboxes
-
-def __merge_if_common_edge(bbox1, bbox2):
- x_min_1, y_min_1, x_max_1, y_max_1 = bbox1
- x_min_2, y_min_2, x_max_2, y_max_2 = bbox2
-
- # 检查是否有公共的水平边
- if y_min_1 == y_min_2 or y_max_1 == y_max_2:
- # 确保一个框的x范围在另一个框的x范围内
- if max(x_min_1, x_min_2) <= min(x_max_1, x_max_2):
- return [min(x_min_1, x_min_2), min(y_min_1, y_min_2), max(x_max_1, x_max_2), max(y_max_1, y_max_2)]
-
- # 检查是否有公共的垂直边
- if x_min_1 == x_min_2 or x_max_1 == x_max_2:
- # 确保一个框的y范围在另一个框的y范围内
- if max(y_min_1, y_min_2) <= min(y_max_1, y_max_2):
- return [min(x_min_1, x_min_2), min(y_min_1, y_min_2), max(x_max_1, x_max_2), max(y_max_1, y_max_2)]
-
- # 如果没有公共边
- return None
-
-def fix_seperated_image(image_bboxes:list):
- """
- 如果2个图片有一个边重叠，那么合并2个图片
- """
- new_images = []
- droped_img_idx = []
-
- for i in range(0, len(image_bboxes)):
- for j in range(i+1, len(image_bboxes)):
- new_img = __merge_if_common_edge(image_bboxes[i], image_bboxes[j])
- if new_img is not None:
- new_images.append(new_img)
- droped_img_idx.append(i)
- droped_img_idx.append(j)
- break
-
- for i in range(0, len(image_bboxes)):
- if i not in droped_img_idx:
- new_images.append(image_bboxes[i])
-
- return new_images
-
-
-def __check_img_title_pattern(text):
- """
- 检查文本段是否是表格的标题
- """
- patterns = [r"^(fig|figure).*", r"^(scheme).*"]
- text = text.strip()
- for pattern in patterns:
- match = re.match(pattern, text, re.IGNORECASE)
- if match:
- return True
- return False
-
-def __get_fig_caption_text(text_block):
- txt = " ".join(span['text'] for line in text_block['lines'] for span in line['spans'])
- line_cnt = len(text_block['lines'])
- txt = txt.replace("Ž . ", '')
- return txt, line_cnt
-
-
-def __find_and_extend_bottom_caption(text_block, pymu_blocks, image_box):
- """
- 继续向下方寻找和图片caption字号，字体，颜色一样的文字框，合并入caption。
- text_block是已经找到的图片catpion（这个caption可能不全，多行被划分到多个pymu block里了）
- """
- combined_image_caption_text_block = list(text_block.copy()['bbox'])
- base_font_color, base_font_size, base_font_type = get_text_block_base_info(text_block)
- while True:
- tb_add = find_bottom_nearest_text_bbox(pymu_blocks, combined_image_caption_text_block)
- if not tb_add:
- break
- tb_font_color, tb_font_size, tb_font_type = get_text_block_base_info(tb_add)
- if tb_font_color==base_font_color and tb_font_size==base_font_size and tb_font_type==base_font_type:
- combined_image_caption_text_block[0] = min(combined_image_caption_text_block[0], tb_add['bbox'][0])
- combined_image_caption_text_block[2] = max(combined_image_caption_text_block[2], tb_add['bbox'][2])
- combined_image_caption_text_block[3] = tb_add['bbox'][3]
- else:
- break
-
- image_box[0] = min(image_box[0], combined_image_caption_text_block[0])
- image_box[1] = min(image_box[1], combined_image_caption_text_block[1])
- image_box[2] = max(image_box[2], combined_image_caption_text_block[2])
- image_box[3] = max(image_box[3], combined_image_caption_text_block[3])
- text_block['_image_caption'] = True
-
-
-def include_img_title(pymu_blocks, image_bboxes: list):
- """
- 向上方和下方寻找符合图片title的文本block，合并到图片里
- 如果图片上下都有fig的情况怎么办？寻找标题距离最近的那个。
- ---
- 增加对左侧和右侧图片标题的寻找
- """
-
-
- for tb in image_bboxes:
- # 优先找下方的
- max_find_cnt = 3 # 向上，向下最多找3个就停止
- temp_box = tb.copy()
- while max_find_cnt>0:
- text_block_btn = find_bottom_nearest_text_bbox(pymu_blocks, temp_box)
- if text_block_btn:
- txt, line_cnt = __get_fig_caption_text(text_block_btn)
- if len(txt.strip())>0:
- if not __check_img_title_pattern(txt) and max_find_cnt>0 and line_cnt<3: # 设置line_cnt<=2目的是为了跳过子标题，或者有时候图片下方文字没有被图片识别模型放入图片里
- max_find_cnt = max_find_cnt - 1
- temp_box[3] = text_block_btn['bbox'][3]
- continue
- else:
- break
- else:
- temp_box[3] = text_block_btn['bbox'][3] # 宽度不变，扩大
- max_find_cnt = max_find_cnt - 1
- else:
- break
-
- max_find_cnt = 3 # 向上，向下最多找3个就停止
- temp_box = tb.copy()
- while max_find_cnt>0:
- text_block_top = find_top_nearest_text_bbox(pymu_blocks, temp_box)
- if text_block_top:
- txt, line_cnt = __get_fig_caption_text(text_block_top)
- if len(txt.strip())>0:
- if not __check_img_title_pattern(txt) and max_find_cnt>0 and line_cnt <3:
- max_find_cnt = max_find_cnt - 1
- temp_box[1] = text_block_top['bbox'][1]
- continue
- else:
- break
- else:
- b = text_block_top['bbox']
- temp_box[1] = b[1] # 宽度不变，扩大
- max_find_cnt = max_find_cnt - 1
- else:
- break
-
- if text_block_btn and text_block_top and text_block_btn.get("_image_caption", False) is False and text_block_top.get("_image_caption", False) is False :
- btn_text, _ = __get_fig_caption_text(text_block_btn)
- top_text, _ = __get_fig_caption_text(text_block_top)
- if __check_img_title_pattern(btn_text) and __check_img_title_pattern(top_text):
- # 取距离图片最近的
- btn_text_distance = text_block_btn['bbox'][1] - tb[3]
- top_text_distance = tb[1] - text_block_top['bbox'][3]
- if btn_text_distance= 5:
- cur_line = (LL, UU, RR, DD)
- res.append(cur_line)
- LL = L1
- else:
- RR = max(RR, R1)
- cur_line = (LL, UU, RR, DD)
- res.append(cur_line)
- return res
-
-def fix_tables(page: fitz.Page, table_bboxes: list, include_table_title: bool, scan_line_num: int):
- """
- :param page :fitz读取的当前页的内容
- :param table_bboxes: list类型，每一个元素是一个元祖 (L, U, R, D)
- :param include_table_title: 是否将表格的标题也圈进来
- :param scan_line_num: 在与表格框临近的上下几个文本框里扫描搜索标题
- """
-
- drawings_lines = get_merged_line(page)
- fix_table_bboxes = []
-
- for table in table_bboxes:
- (L, U, R, D) = table
- fix_table_L = []
- fix_table_U = []
- fix_table_R = []
- fix_table_D = []
- width = R - L
- width_range = width * 0.1 # 只看距离表格整体宽度10%之内偏差的线
- height = D - U
- height_range = height * 0.1 # 只看距离表格整体高度10%之内偏差的线
- for line in drawings_lines:
- if (L - width_range) <= line[0] <= (L + width_range) and (R - width_range) <= line[2] <= (R + width_range): # 相近的宽度
- if (U - height_range) < line[1] < (U + height_range): # 上边界，在一定的高度范围内
- fix_table_U.append(line[1])
- fix_table_L.append(line[0])
- fix_table_R.append(line[2])
- elif (D - height_range) < line[1] < (D + height_range): # 下边界，在一定的高度范围内
- fix_table_D.append(line[1])
- fix_table_L.append(line[0])
- fix_table_R.append(line[2])
-
- if fix_table_U:
- U = min(fix_table_U)
- if fix_table_D:
- D = max(fix_table_D)
- if fix_table_L:
- L = min(fix_table_L)
- if fix_table_R:
- R = max(fix_table_R)
-
- if include_table_title: # 需要将表格标题包括
- text_blocks = page.get_text("dict", flags=fitz.TEXTFLAGS_TEXT)["blocks"] # 所有的text的block
- incolumn_text_blocks = [block for block in text_blocks if not ((block['bbox'][0] < L and block['bbox'][2] < L) or (block['bbox'][0] > R and block['bbox'][2] > R))] # 将与表格完全没有任何遮挡的文字筛除掉（比如另一栏的文字）
- upper_text_blocks = [block for block in incolumn_text_blocks if (U - block['bbox'][3]) > 0] # 将在表格线以上的text block筛选出来
- sorted_filtered_text_blocks = sorted(upper_text_blocks, key=lambda x: (U - x['bbox'][3], x['bbox'][0])) # 按照text block的下边界距离表格上边界的距离升序排序，如果是同一个高度，则先左再右
-
- for idx in range(scan_line_num):
- if idx+1 <= len(sorted_filtered_text_blocks):
- line_temp = sorted_filtered_text_blocks[idx]['lines']
- if line_temp:
- text = line_temp[0]['spans'][0]['text'] # 提取出第一个span里的text内容
- check_en = re.match('Table', text) # 检查是否有Table开头的(英文）
- check_ch = re.match('表', text) # 检查是否有Table开头的(中文）
- if check_en or check_ch:
- if sorted_filtered_text_blocks[idx]['bbox'][1] < D: # 以防出现负的bbox
- U = sorted_filtered_text_blocks[idx]['bbox'][1]
-
- fix_table_bboxes.append([L-2, U-2, R+2, D+2])
-
- return fix_table_bboxes
-
-def __check_table_title_pattern(text):
- """
- 检查文本段是否是表格的标题
- """
- patterns = [r'^table\s\d+']
-
- for pattern in patterns:
- match = re.match(pattern, text, re.IGNORECASE)
- if match:
- return True
- else:
- return False
-
-
-def fix_table_text_block(pymu_blocks, table_bboxes: list):
- """
- 调整table, 如果table和上下的text block有相交区域，则将table的上下边界调整到text block的上下边界
- 例如 tmp/unittest/unittest_pdf/纯2列_ViLT_6_文字 表格.pdf
- """
- for tb in table_bboxes:
- (L, U, R, D) = tb
- for block in pymu_blocks:
- if _is_in_or_part_overlap((L, U, R, D), block['bbox']):
- txt = " ".join(span['text'] for line in block['lines'] for span in line['spans'])
- if not __check_table_title_pattern(txt) and block.get("_table", False) is False: # 如果是table的title，那么不调整。因为下一步会统一调整，如果这里进行了调整，后面的调整会造成调整到其他table的title上（在连续出现2个table的情况下）。
- tb[0] = min(tb[0], block['bbox'][0])
- tb[1] = min(tb[1], block['bbox'][1])
- tb[2] = max(tb[2], block['bbox'][2])
- tb[3] = max(tb[3], block['bbox'][3])
- block['_table'] = True # 占位，防止其他table再次占用
-
- """如果是个table的title，但是有部分重叠，那么修正这个title,使得和table不重叠"""
- if _is_part_overlap(tb, block['bbox']) and __check_table_title_pattern(txt):
- block['bbox'] = list(block['bbox'])
- if block['bbox'][3] > U:
- block['bbox'][3] = U-1
- if block['bbox'][1] < D:
- block['bbox'][1] = D+1
-
-
- return table_bboxes
-
-
-def __get_table_caption_text(text_block):
- txt = " ".join(span['text'] for line in text_block['lines'] for span in line['spans'])
- line_cnt = len(text_block['lines'])
- txt = txt.replace("Ž . ", '')
- return txt, line_cnt
-
-
-def include_table_title(pymu_blocks, table_bboxes: list):
- """
- 把表格的title也包含进来，扩展到table_bbox上
- """
- for tb in table_bboxes:
- max_find_cnt = 3 # 上上最多找3次
- temp_box = tb.copy()
- while max_find_cnt>0:
- text_block_top = find_top_nearest_text_bbox(pymu_blocks, temp_box)
- if text_block_top:
- txt, line_cnt = __get_table_caption_text(text_block_top)
- if len(txt.strip())>0:
- if not __check_table_title_pattern(txt) and max_find_cnt>0 and line_cnt<3:
- max_find_cnt = max_find_cnt -1
- temp_box[1] = text_block_top['bbox'][1]
- continue
- else:
- break
- else:
- temp_box[1] = text_block_top['bbox'][1] # 宽度不变，扩大
- max_find_cnt = max_find_cnt - 1
- else:
- break
-
- max_find_cnt = 3 # 向下找
- temp_box = tb.copy()
- while max_find_cnt>0:
- text_block_bottom = find_bottom_nearest_text_bbox(pymu_blocks, temp_box)
- if text_block_bottom:
- txt, line_cnt = __get_table_caption_text(text_block_bottom)
- if len(txt.strip())>0:
- if not __check_table_title_pattern(txt) and max_find_cnt>0 and line_cnt<3:
- max_find_cnt = max_find_cnt - 1
- temp_box[3] = text_block_bottom['bbox'][3]
- continue
- else:
- break
- else:
- temp_box[3] = text_block_bottom['bbox'][3]
- max_find_cnt = max_find_cnt - 1
- else:
- break
-
- if text_block_top and text_block_bottom and text_block_top.get("_table_caption", False) is False and text_block_bottom.get("_table_caption", False) is False :
- btn_text, _ = __get_table_caption_text(text_block_bottom)
- top_text, _ = __get_table_caption_text(text_block_top)
- if __check_table_title_pattern(btn_text) and __check_table_title_pattern(top_text): # 上下都有一个tbale的caption
- # 取距离最近的
- btn_text_distance = text_block_bottom['bbox'][1] - tb[3]
- top_text_distance = tb[1] - text_block_top['bbox'][3]
- text_block = text_block_bottom if btn_text_distance 0]
- if span_font:
- # main_text_font应该用基于字数最多的字体而不是span级别的统计
- # font_names.append(font_name for font_name in span_font)
- # block_fonts.append(font_name for font_name in span_font)
- for font, count in span_font:
- font_names[font] += count
- main_text_font = font_names.most_common(1)[0][0]
- return main_text_font
-
diff --git a/magic_pdf/pre_proc/ocr_detect_all_bboxes.py b/magic_pdf/pre_proc/ocr_detect_all_bboxes.py
deleted file mode 100644
index 77f242b7d..000000000
--- a/magic_pdf/pre_proc/ocr_detect_all_bboxes.py
+++ /dev/null
@@ -1,244 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.boxbase import get_minbox_if_overlap_by_ratio, calculate_overlap_area_in_bbox1_area_ratio, \
- calculate_iou, calculate_vertical_projection_overlap_ratio
-from magic_pdf.libs.drop_tag import DropTag
-from magic_pdf.libs.ocr_content_type import BlockType
-from magic_pdf.pre_proc.remove_bbox_overlap import remove_overlap_between_bbox_for_block
-
-
-def ocr_prepare_bboxes_for_layout_split(img_blocks, table_blocks, discarded_blocks, text_blocks,
- title_blocks, interline_equation_blocks, page_w, page_h):
- all_bboxes = []
- all_discarded_blocks = []
- for image in img_blocks:
- x0, y0, x1, y1 = image['bbox']
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.Image, None, None, None, None, image["score"]])
-
- for table in table_blocks:
- x0, y0, x1, y1 = table['bbox']
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.Table, None, None, None, None, table["score"]])
-
- for text in text_blocks:
- x0, y0, x1, y1 = text['bbox']
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.Text, None, None, None, None, text["score"]])
-
- for title in title_blocks:
- x0, y0, x1, y1 = title['bbox']
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.Title, None, None, None, None, title["score"]])
-
- for interline_equation in interline_equation_blocks:
- x0, y0, x1, y1 = interline_equation['bbox']
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.InterlineEquation, None, None, None, None, interline_equation["score"]])
-
- '''block嵌套问题解决'''
- '''文本框与标题框重叠，优先信任文本框'''
- all_bboxes = fix_text_overlap_title_blocks(all_bboxes)
- '''任何框体与舍弃框重叠，优先信任舍弃框'''
- all_bboxes = remove_need_drop_blocks(all_bboxes, discarded_blocks)
-
- # interline_equation 与title或text框冲突的情况，分两种情况处理
- '''interline_equation框与文本类型框iou比较接近1的时候，信任行间公式框'''
- all_bboxes = fix_interline_equation_overlap_text_blocks_with_hi_iou(all_bboxes)
- '''interline_equation框被包含在文本类型框内，且interline_equation比文本区块小很多时信任文本框，这时需要舍弃公式框'''
- # 通过后续大框套小框逻辑删除
-
- '''discarded_blocks中只保留宽度超过1/3页面宽度的，高度超过10的，处于页面下半50%区域的（限定footnote）'''
- for discarded in discarded_blocks:
- x0, y0, x1, y1 = discarded['bbox']
- all_discarded_blocks.append([x0, y0, x1, y1, None, None, None, BlockType.Discarded, None, None, None, None, discarded["score"]])
- # 将footnote加入到all_bboxes中，用来计算layout
- if (x1 - x0) > (page_w / 3) and (y1 - y0) > 10 and y0 > (page_h / 2):
- all_bboxes.append([x0, y0, x1, y1, None, None, None, BlockType.Footnote, None, None, None, None, discarded["score"]])
-
- '''经过以上处理后，还存在大框套小框的情况，则删除小框'''
- all_bboxes = remove_overlaps_min_blocks(all_bboxes)
- all_discarded_blocks = remove_overlaps_min_blocks(all_discarded_blocks)
- '''将剩余的bbox做分离处理，防止后面分layout时出错'''
- all_bboxes, drop_reasons = remove_overlap_between_bbox_for_block(all_bboxes)
-
- return all_bboxes, all_discarded_blocks, drop_reasons
-
-
-def add_bboxes(blocks, block_type, bboxes):
- for block in blocks:
- x0, y0, x1, y1 = block['bbox']
- if block_type in [
- BlockType.ImageBody, BlockType.ImageCaption, BlockType.ImageFootnote,
- BlockType.TableBody, BlockType.TableCaption, BlockType.TableFootnote
- ]:
- bboxes.append([x0, y0, x1, y1, None, None, None, block_type, None, None, None, None, block["score"], block["group_id"]])
- else:
- bboxes.append([x0, y0, x1, y1, None, None, None, block_type, None, None, None, None, block["score"]])
-
-
-def ocr_prepare_bboxes_for_layout_split_v2(
- img_body_blocks, img_caption_blocks, img_footnote_blocks,
- table_body_blocks, table_caption_blocks, table_footnote_blocks,
- discarded_blocks, text_blocks, title_blocks, interline_equation_blocks, page_w, page_h
-):
- all_bboxes = []
-
- add_bboxes(img_body_blocks, BlockType.ImageBody, all_bboxes)
- add_bboxes(img_caption_blocks, BlockType.ImageCaption, all_bboxes)
- add_bboxes(img_footnote_blocks, BlockType.ImageFootnote, all_bboxes)
- add_bboxes(table_body_blocks, BlockType.TableBody, all_bboxes)
- add_bboxes(table_caption_blocks, BlockType.TableCaption, all_bboxes)
- add_bboxes(table_footnote_blocks, BlockType.TableFootnote, all_bboxes)
- add_bboxes(text_blocks, BlockType.Text, all_bboxes)
- add_bboxes(title_blocks, BlockType.Title, all_bboxes)
- add_bboxes(interline_equation_blocks, BlockType.InterlineEquation, all_bboxes)
-
- '''block嵌套问题解决'''
- '''文本框与标题框重叠，优先信任文本框'''
- all_bboxes = fix_text_overlap_title_blocks(all_bboxes)
- '''任何框体与舍弃框重叠，优先信任舍弃框'''
- all_bboxes = remove_need_drop_blocks(all_bboxes, discarded_blocks)
-
- # interline_equation 与title或text框冲突的情况，分两种情况处理
- '''interline_equation框与文本类型框iou比较接近1的时候，信任行间公式框'''
- all_bboxes = fix_interline_equation_overlap_text_blocks_with_hi_iou(all_bboxes)
- '''interline_equation框被包含在文本类型框内，且interline_equation比文本区块小很多时信任文本框，这时需要舍弃公式框'''
- # 通过后续大框套小框逻辑删除
-
- '''discarded_blocks'''
- all_discarded_blocks = []
- add_bboxes(discarded_blocks, BlockType.Discarded, all_discarded_blocks)
-
- '''footnote识别：宽度超过1/3页面宽度的，高度超过10的，处于页面下半50%区域的'''
- footnote_blocks = []
- for discarded in discarded_blocks:
- x0, y0, x1, y1 = discarded['bbox']
- if (x1 - x0) > (page_w / 3) and (y1 - y0) > 10 and y0 > (page_h / 2):
- footnote_blocks.append([x0, y0, x1, y1])
-
- '''移除在footnote下面的任何框'''
- need_remove_blocks = find_blocks_under_footnote(all_bboxes, footnote_blocks)
- if len(need_remove_blocks) > 0:
- for block in need_remove_blocks:
- all_bboxes.remove(block)
- all_discarded_blocks.append(block)
-
- '''经过以上处理后，还存在大框套小框的情况，则删除小框'''
- all_bboxes = remove_overlaps_min_blocks(all_bboxes)
- all_discarded_blocks = remove_overlaps_min_blocks(all_discarded_blocks)
- '''将剩余的bbox做分离处理，防止后面分layout时出错'''
- all_bboxes, drop_reasons = remove_overlap_between_bbox_for_block(all_bboxes)
-
- return all_bboxes, all_discarded_blocks
-
-
-def find_blocks_under_footnote(all_bboxes, footnote_blocks):
- need_remove_blocks = []
- for block in all_bboxes:
- block_x0, block_y0, block_x1, block_y1 = block[:4]
- for footnote_bbox in footnote_blocks:
- footnote_x0, footnote_y0, footnote_x1, footnote_y1 = footnote_bbox
- # 如果footnote的纵向投影覆盖了block的纵向投影的80%且block的y0大于等于footnote的y1
- if block_y0 >= footnote_y1 and calculate_vertical_projection_overlap_ratio((block_x0, block_y0, block_x1, block_y1), footnote_bbox) >= 0.8:
- if block not in need_remove_blocks:
- need_remove_blocks.append(block)
- break
- return need_remove_blocks
-
-
-def fix_interline_equation_overlap_text_blocks_with_hi_iou(all_bboxes):
- # 先提取所有text和interline block
- text_blocks = []
- for block in all_bboxes:
- if block[7] == BlockType.Text:
- text_blocks.append(block)
- interline_equation_blocks = []
- for block in all_bboxes:
- if block[7] == BlockType.InterlineEquation:
- interline_equation_blocks.append(block)
-
- need_remove = []
-
- for interline_equation_block in interline_equation_blocks:
- for text_block in text_blocks:
- interline_equation_block_bbox = interline_equation_block[:4]
- text_block_bbox = text_block[:4]
- if calculate_iou(interline_equation_block_bbox, text_block_bbox) > 0.8:
- if text_block not in need_remove:
- need_remove.append(text_block)
-
- if len(need_remove) > 0:
- for block in need_remove:
- all_bboxes.remove(block)
-
- return all_bboxes
-
-
-def fix_text_overlap_title_blocks(all_bboxes):
- # 先提取所有text和title block
- text_blocks = []
- for block in all_bboxes:
- if block[7] == BlockType.Text:
- text_blocks.append(block)
- title_blocks = []
- for block in all_bboxes:
- if block[7] == BlockType.Title:
- title_blocks.append(block)
-
- need_remove = []
-
- for text_block in text_blocks:
- for title_block in title_blocks:
- text_block_bbox = text_block[:4]
- title_block_bbox = title_block[:4]
- if calculate_iou(text_block_bbox, title_block_bbox) > 0.8:
- if title_block not in need_remove:
- need_remove.append(title_block)
-
- if len(need_remove) > 0:
- for block in need_remove:
- all_bboxes.remove(block)
-
- return all_bboxes
-
-
-def remove_need_drop_blocks(all_bboxes, discarded_blocks):
- need_remove = []
- for block in all_bboxes:
- for discarded_block in discarded_blocks:
- block_bbox = block[:4]
- if calculate_overlap_area_in_bbox1_area_ratio(block_bbox, discarded_block['bbox']) > 0.6:
- if block not in need_remove:
- need_remove.append(block)
- break
-
- if len(need_remove) > 0:
- for block in need_remove:
- all_bboxes.remove(block)
- return all_bboxes
-
-
-def remove_overlaps_min_blocks(all_bboxes):
- # 重叠block，小的不能直接删除，需要和大的那个合并成一个更大的。
- # 删除重叠blocks中较小的那些
- need_remove = []
- for block1 in all_bboxes:
- for block2 in all_bboxes:
- if block1 != block2:
- block1_bbox = block1[:4]
- block2_bbox = block2[:4]
- overlap_box = get_minbox_if_overlap_by_ratio(block1_bbox, block2_bbox, 0.8)
- if overlap_box is not None:
- block_to_remove = next((block for block in all_bboxes if block[:4] == overlap_box), None)
- if block_to_remove is not None and block_to_remove not in need_remove:
- large_block = block1 if block1 != block_to_remove else block2
- x1, y1, x2, y2 = large_block[:4]
- sx1, sy1, sx2, sy2 = block_to_remove[:4]
- x1 = min(x1, sx1)
- y1 = min(y1, sy1)
- x2 = max(x2, sx2)
- y2 = max(y2, sy2)
- large_block[:4] = [x1, y1, x2, y2]
- need_remove.append(block_to_remove)
-
- if len(need_remove) > 0:
- for block in need_remove:
- all_bboxes.remove(block)
-
- return all_bboxes
diff --git a/magic_pdf/pre_proc/ocr_detect_layout.py b/magic_pdf/pre_proc/ocr_detect_layout.py
deleted file mode 100644
index 4dad3593d..000000000
--- a/magic_pdf/pre_proc/ocr_detect_layout.py
+++ /dev/null
@@ -1,133 +0,0 @@
-import fitz
-
-from magic_pdf.layout.layout_sort import get_bboxes_layout
-from magic_pdf.libs.boxbase import _is_part_overlap, _is_in
-from magic_pdf.libs.coordinate_transform import get_scale_ratio
-
-
-def get_center_point(bbox):
- """
- 根据边界框坐标信息，计算出该边界框的中心点坐标。
- Args:
- bbox (list): 边界框坐标信息，包含四个元素，分别为左上角x坐标、左上角y坐标、右下角x坐标、右下角y坐标。
- Returns:
- list: 中心点坐标信息，包含两个元素，分别为x坐标和y坐标。
- """
- return [(bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2]
-
-
-def get_area(bbox):
- """
- 根据边界框坐标信息，计算出该边界框的面积。
- Args:
- bbox (list): 边界框坐标信息，包含四个元素，分别为左上角x坐标、左上角y坐标、右下角x坐标、右下角y坐标。
- Returns:
- float: 该边界框的面积。
- """
- return (bbox[2] - bbox[0]) * (bbox[3] - bbox[1])
-
-
-def adjust_layouts(layout_bboxes, page_boundry, page_id):
- # 遍历所有布局框
- for i in range(len(layout_bboxes)):
- # 遍历当前布局框之后的布局框
- for j in range(i + 1, len(layout_bboxes)):
- # 判断两个布局框是否重叠
- if _is_part_overlap(layout_bboxes[i], layout_bboxes[j]):
- # 计算每个布局框的中心点坐标和面积
- area_i = get_area(layout_bboxes[i])
- area_j = get_area(layout_bboxes[j])
-
- # 较大布局框和较小布局框的赋值
- if area_i > area_j:
- larger_layout, smaller_layout = layout_bboxes[i], layout_bboxes[j]
- else:
- larger_layout, smaller_layout = layout_bboxes[j], layout_bboxes[i]
-
- center_large = get_center_point(larger_layout)
- center_small = get_center_point(smaller_layout)
- # 计算横向和纵向的距离差
- distance_x = center_large[0] - center_small[0]
- distance_y = center_large[1] - center_small[1]
-
- # 根据距离差判断重叠方向并修正边界
- if abs(distance_x) > abs(distance_y): # 左右重叠
- if distance_x > 0 and larger_layout[0] < smaller_layout[2]:
- larger_layout[0] = smaller_layout[2]+1
- if distance_x < 0 and larger_layout[2] > smaller_layout[0]:
- larger_layout[2] = smaller_layout[0]-1
- else: # 上下重叠
- if distance_y > 0 and larger_layout[1] < smaller_layout[3]:
- larger_layout[1] = smaller_layout[3]+1
- if distance_y < 0 and larger_layout[3] > smaller_layout[1]:
- larger_layout[3] = smaller_layout[1]-1
- # 排序调整布局边界框列表
- new_bboxes = []
- for layout_bbox in layout_bboxes:
- new_bboxes.append([layout_bbox[0], layout_bbox[1], layout_bbox[2], layout_bbox[3], None, None, None, None, None, None, None, None, None])
-
- layout_bboxes, layout_tree = get_bboxes_layout(new_bboxes, page_boundry, page_id)
-
- # 返回排序调整后的布局边界框列表
- return layout_bboxes, layout_tree
-
-
-def layout_detect(layout_info, page: fitz.Page, ocr_page_info):
- """
- 对输入的布局信息进行解析，提取出每个子布局的边界框，并对所有子布局进行排序调整。
-
- Args:
- layout_info (list): 包含子布局信息的列表，每个子布局信息为字典类型，包含'poly'字段，表示子布局的边界框坐标信息。
-
- Returns:
- list: 经过排序调整后的所有子布局边界框信息的列表，每个边界框信息为字典类型，包含'layout_bbox'字段，表示边界框的坐标信息。
-
- """
- page_id = ocr_page_info['page_info']['page_no']-1
- horizontal_scale_ratio, vertical_scale_ratio = get_scale_ratio(ocr_page_info, page)
- # 初始化布局边界框列表
- layout_bboxes = []
- # 遍历每个子布局
- for sub_layout in layout_info:
- # 提取子布局的边界框坐标信息
- x0, y0, _, _, x1, y1, _, _ = sub_layout['poly']
- bbox = [int(x0 / horizontal_scale_ratio), int(y0 / vertical_scale_ratio),
- int(x1 / horizontal_scale_ratio), int(y1 / vertical_scale_ratio)]
-
- # 将子布局的边界框添加到列表中
- layout_bboxes.append(bbox)
-
- # 初始化新的布局边界框列表
- new_layout_bboxes = []
- # 遍历每个布局边界框
- for i in range(len(layout_bboxes)):
- # 初始化标记变量，用于判断当前边界框是否需要保留
- keep = True
- # 获取当前边界框的坐标信息
- box_i = layout_bboxes[i]
-
- # 遍历其他边界框
- for j in range(len(layout_bboxes)):
- # 排除当前边界框自身
- if i != j:
- # 获取其他边界框的坐标信息
- box_j = layout_bboxes[j]
- # 检测box_i是否被box_j包含
- if _is_in(box_i, box_j):
- # 如果当前边界框被其他边界框包含，则标记为不需要保留
- keep = False
- # 跳出内层循环
- break
-
- # 如果当前边界框需要保留，则添加到新的布局边界框列表中
- if keep:
- new_layout_bboxes.append(layout_bboxes[i])
-
- # 对新的布局边界框列表进行排序调整
- page_width = page.rect.width
- page_height = page.rect.height
- page_boundry = [0, 0, page_width, page_height]
- layout_bboxes, layout_tree = adjust_layouts(new_layout_bboxes, page_boundry, page_id)
-
- # 返回排序调整后的布局边界框列表
- return layout_bboxes, layout_tree
diff --git a/magic_pdf/pre_proc/ocr_dict_merge.py b/magic_pdf/pre_proc/ocr_dict_merge.py
deleted file mode 100644
index 1b5539787..000000000
--- a/magic_pdf/pre_proc/ocr_dict_merge.py
+++ /dev/null
@@ -1,362 +0,0 @@
-from magic_pdf.libs.boxbase import (__is_overlaps_y_exceeds_threshold,
- _is_in_or_part_overlap_with_area_ratio,
- calculate_overlap_area_in_bbox1_area_ratio)
-from magic_pdf.libs.drop_tag import DropTag
-from magic_pdf.libs.ocr_content_type import BlockType, ContentType
-
-
-# 将每一个line中的span从左到右排序
-def line_sort_spans_by_left_to_right(lines):
- line_objects = []
- for line in lines:
- # 按照x0坐标排序
- line.sort(key=lambda span: span['bbox'][0])
- line_bbox = [
- min(span['bbox'][0] for span in line), # x0
- min(span['bbox'][1] for span in line), # y0
- max(span['bbox'][2] for span in line), # x1
- max(span['bbox'][3] for span in line), # y1
- ]
- line_objects.append({
- 'bbox': line_bbox,
- 'spans': line,
- })
- return line_objects
-
-
-def merge_spans_to_line(spans):
- if len(spans) == 0:
- return []
- else:
- # 按照y0坐标排序
- spans.sort(key=lambda span: span['bbox'][1])
-
- lines = []
- current_line = [spans[0]]
- for span in spans[1:]:
- # 如果当前的span类型为"interline_equation" 或者 当前行中已经有"interline_equation"
- # image和table类型，同上
- if span['type'] in [
- ContentType.InterlineEquation, ContentType.Image,
- ContentType.Table
- ] or any(s['type'] in [
- ContentType.InterlineEquation, ContentType.Image,
- ContentType.Table
- ] for s in current_line):
- # 则开始新行
- lines.append(current_line)
- current_line = [span]
- continue
-
- # 如果当前的span与当前行的最后一个span在y轴上重叠，则添加到当前行
- if __is_overlaps_y_exceeds_threshold(span['bbox'], current_line[-1]['bbox'], 0.5):
- current_line.append(span)
- else:
- # 否则，开始新行
- lines.append(current_line)
- current_line = [span]
-
- # 添加最后一行
- if current_line:
- lines.append(current_line)
-
- return lines
-
-
-def merge_spans_to_line_by_layout(spans, layout_bboxes):
- lines = []
- new_spans = []
- dropped_spans = []
- for item in layout_bboxes:
- layout_bbox = item['layout_bbox']
- # 遍历spans,将每个span放入对应的layout中
- layout_sapns = []
- for span in spans:
- if calculate_overlap_area_in_bbox1_area_ratio(
- span['bbox'], layout_bbox) > 0.6:
- layout_sapns.append(span)
- # 如果layout_sapns不为空，则放入new_spans中
- if len(layout_sapns) > 0:
- new_spans.append(layout_sapns)
- # 从spans删除已经放入layout_sapns中的span
- for layout_sapn in layout_sapns:
- spans.remove(layout_sapn)
-
- if len(new_spans) > 0:
- for layout_sapns in new_spans:
- layout_lines = merge_spans_to_line(layout_sapns)
- lines.extend(layout_lines)
-
- # 对line中的span进行排序
- lines = line_sort_spans_by_left_to_right(lines)
-
- for span in spans:
- span['tag'] = DropTag.NOT_IN_LAYOUT
- dropped_spans.append(span)
-
- return lines, dropped_spans
-
-
-def merge_lines_to_block(lines):
- # 目前不做block拼接,先做个结构,每个block中只有一个line,block的bbox就是line的bbox
- blocks = []
- for line in lines:
- blocks.append({
- 'bbox': line['bbox'],
- 'lines': [line],
- })
- return blocks
-
-
-def sort_blocks_by_layout(all_bboxes, layout_bboxes):
- new_blocks = []
- sort_blocks = []
- for item in layout_bboxes:
- layout_bbox = item['layout_bbox']
-
- # 遍历blocks,将每个blocks放入对应的layout中
- layout_blocks = []
- for block in all_bboxes:
- # 如果是footnote则跳过
- if block[7] == BlockType.Footnote:
- continue
- block_bbox = block[:4]
- if calculate_overlap_area_in_bbox1_area_ratio(
- block_bbox, layout_bbox) > 0.8:
- layout_blocks.append(block)
-
- # 如果layout_blocks不为空，则放入new_blocks中
- if len(layout_blocks) > 0:
- new_blocks.append(layout_blocks)
- # 从all_bboxes删除已经放入layout_blocks中的block
- for layout_block in layout_blocks:
- all_bboxes.remove(layout_block)
-
- # 如果new_blocks不为空，则对new_blocks中每个block进行排序
- if len(new_blocks) > 0:
- for bboxes_in_layout_block in new_blocks:
- bboxes_in_layout_block.sort(
- key=lambda x: x[1]) # 一个layout内部的box，按照y0自上而下排序
- sort_blocks.extend(bboxes_in_layout_block)
-
- # sort_blocks中已经包含了当前页面所有最终留下的block，且已经排好了顺序
- return sort_blocks
-
-
-def fill_spans_in_blocks(blocks, spans, radio):
- """将allspans中的span按位置关系，放入blocks中."""
- block_with_spans = []
- for block in blocks:
- block_type = block[7]
- block_bbox = block[0:4]
- block_dict = {
- 'type': block_type,
- 'bbox': block_bbox,
- }
- if block_type in [
- BlockType.ImageBody, BlockType.ImageCaption, BlockType.ImageFootnote,
- BlockType.TableBody, BlockType.TableCaption, BlockType.TableFootnote
- ]:
- block_dict["group_id"] = block[-1]
- block_spans = []
- for span in spans:
- span_bbox = span['bbox']
- if calculate_overlap_area_in_bbox1_area_ratio(
- span_bbox, block_bbox) > radio:
- block_spans.append(span)
- '''行内公式调整, 高度调整至与同行文字高度一致(优先左侧, 其次右侧)'''
- # displayed_list = []
- # text_inline_lines = []
- # modify_y_axis(block_spans, displayed_list, text_inline_lines)
- '''模型识别错误的行间公式, type类型转换成行内公式'''
- # block_spans = modify_inline_equation(block_spans, displayed_list, text_inline_lines)
- '''bbox去除粘连''' # 去粘连会影响span的bbox，导致后续fill的时候出错
- # block_spans = remove_overlap_between_bbox_for_span(block_spans)
-
- block_dict['spans'] = block_spans
- block_with_spans.append(block_dict)
-
- # 从spans删除已经放入block_spans中的span
- if len(block_spans) > 0:
- for span in block_spans:
- spans.remove(span)
-
- return block_with_spans, spans
-
-
-def fix_block_spans(block_with_spans, img_blocks, table_blocks):
- """1、img_block和table_block因为包含caption和footnote的关系，存在block的嵌套关系
- 需要将caption和footnote的text_span放入相应img_block和table_block内的
- caption_block和footnote_block中 2、同时需要删除block中的spans字段."""
- fix_blocks = []
- for block in block_with_spans:
- block_type = block['type']
-
- if block_type == BlockType.Image:
- block = fix_image_block(block, img_blocks)
- elif block_type == BlockType.Table:
- block = fix_table_block(block, table_blocks)
- elif block_type in [BlockType.Text, BlockType.Title]:
- block = fix_text_block(block)
- elif block_type == BlockType.InterlineEquation:
- block = fix_interline_block(block)
- else:
- continue
- fix_blocks.append(block)
- return fix_blocks
-
-
-def fix_block_spans_v2(block_with_spans):
- """1、img_block和table_block因为包含caption和footnote的关系，存在block的嵌套关系
- 需要将caption和footnote的text_span放入相应img_block和table_block内的
- caption_block和footnote_block中 2、同时需要删除block中的spans字段."""
- fix_blocks = []
- for block in block_with_spans:
- block_type = block['type']
-
- if block_type in [BlockType.Text, BlockType.Title,
- BlockType.ImageCaption, BlockType.ImageFootnote,
- BlockType.TableCaption, BlockType.TableFootnote
- ]:
- block = fix_text_block(block)
- elif block_type in [BlockType.InterlineEquation, BlockType.ImageBody, BlockType.TableBody]:
- block = fix_interline_block(block)
- else:
- continue
- fix_blocks.append(block)
- return fix_blocks
-
-
-def fix_discarded_block(discarded_block_with_spans):
- fix_discarded_blocks = []
- for block in discarded_block_with_spans:
- block = fix_text_block(block)
- fix_discarded_blocks.append(block)
- return fix_discarded_blocks
-
-
-def merge_spans_to_block(spans: list, block_bbox: list, block_type: str):
- block_spans = []
- # 如果有img_caption，则将img_block中的text_spans放入img_caption_block中
- for span in spans:
- if calculate_overlap_area_in_bbox1_area_ratio(span['bbox'],
- block_bbox) > 0.6:
- block_spans.append(span)
- block_lines = merge_spans_to_line(block_spans)
- # 对line中的span进行排序
- sort_block_lines = line_sort_spans_by_left_to_right(block_lines)
- block = {'bbox': block_bbox, 'type': block_type, 'lines': sort_block_lines}
- return block, block_spans
-
-
-def make_body_block(span: dict, block_bbox: list, block_type: str):
- # 创建body_block
- body_line = {
- 'bbox': block_bbox,
- 'spans': [span],
- }
- body_block = {'bbox': block_bbox, 'type': block_type, 'lines': [body_line]}
- return body_block
-
-
-def fix_image_block(block, img_blocks):
- block['blocks'] = []
- # 遍历img_blocks,找到与当前block匹配的img_block
- for img_block in img_blocks:
- if _is_in_or_part_overlap_with_area_ratio(block['bbox'],
- img_block['bbox'], 0.95):
-
- # 创建img_body_block
- for span in block['spans']:
- if span['type'] == ContentType.Image and img_block[
- 'img_body_bbox'] == span['bbox']:
- # 创建img_body_block
- img_body_block = make_body_block(
- span, img_block['img_body_bbox'], BlockType.ImageBody)
- block['blocks'].append(img_body_block)
-
- # 从spans中移除img_body_block中已经放入的span
- block['spans'].remove(span)
- break
-
- # 根据list长度，判断img_block中是否有img_caption
- if img_block['img_caption_bbox'] is not None:
- img_caption_block, img_caption_spans = merge_spans_to_block(
- block['spans'], img_block['img_caption_bbox'],
- BlockType.ImageCaption)
- block['blocks'].append(img_caption_block)
-
- if img_block['img_footnote_bbox'] is not None:
- img_footnote_block, img_footnote_spans = merge_spans_to_block(
- block['spans'], img_block['img_footnote_bbox'],
- BlockType.ImageFootnote)
- block['blocks'].append(img_footnote_block)
- break
- del block['spans']
- return block
-
-
-def fix_table_block(block, table_blocks):
- block['blocks'] = []
- # 遍历table_blocks,找到与当前block匹配的table_block
- for table_block in table_blocks:
- if _is_in_or_part_overlap_with_area_ratio(block['bbox'],
- table_block['bbox'], 0.95):
-
- # 创建table_body_block
- for span in block['spans']:
- if span['type'] == ContentType.Table and table_block[
- 'table_body_bbox'] == span['bbox']:
- # 创建table_body_block
- table_body_block = make_body_block(
- span, table_block['table_body_bbox'],
- BlockType.TableBody)
- block['blocks'].append(table_body_block)
-
- # 从spans中移除img_body_block中已经放入的span
- block['spans'].remove(span)
- break
-
- # 根据list长度，判断table_block中是否有caption
- if table_block['table_caption_bbox'] is not None:
- table_caption_block, table_caption_spans = merge_spans_to_block(
- block['spans'], table_block['table_caption_bbox'],
- BlockType.TableCaption)
- block['blocks'].append(table_caption_block)
-
- # 如果table_caption_block_spans不为空
- if len(table_caption_spans) > 0:
- # 一些span已经放入了caption_block中，需要从block['spans']中删除
- for span in table_caption_spans:
- block['spans'].remove(span)
-
- # 根据list长度，判断table_block中是否有table_note
- if table_block['table_footnote_bbox'] is not None:
- table_footnote_block, table_footnote_spans = merge_spans_to_block(
- block['spans'], table_block['table_footnote_bbox'],
- BlockType.TableFootnote)
- block['blocks'].append(table_footnote_block)
-
- break
- del block['spans']
- return block
-
-
-def fix_text_block(block):
- # 文本block中的公式span都应该转换成行内type
- for span in block['spans']:
- if span['type'] == ContentType.InterlineEquation:
- span['type'] = ContentType.InlineEquation
- block_lines = merge_spans_to_line(block['spans'])
- sort_block_lines = line_sort_spans_by_left_to_right(block_lines)
- block['lines'] = sort_block_lines
- del block['spans']
- return block
-
-
-def fix_interline_block(block):
- block_lines = merge_spans_to_line(block['spans'])
- sort_block_lines = line_sort_spans_by_left_to_right(block_lines)
- block['lines'] = sort_block_lines
- del block['spans']
- return block
diff --git a/magic_pdf/pre_proc/ocr_span_list_modify.py b/magic_pdf/pre_proc/ocr_span_list_modify.py
deleted file mode 100644
index 9c2d6cc81..000000000
--- a/magic_pdf/pre_proc/ocr_span_list_modify.py
+++ /dev/null
@@ -1,263 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.boxbase import calculate_overlap_area_in_bbox1_area_ratio, get_minbox_if_overlap_by_ratio, \
- __is_overlaps_y_exceeds_threshold, calculate_iou
-from magic_pdf.libs.drop_tag import DropTag
-from magic_pdf.libs.ocr_content_type import ContentType, BlockType
-
-
-def remove_overlaps_low_confidence_spans(spans):
- dropped_spans = []
- # 删除重叠spans中置信度低的的那些
- for span1 in spans:
- for span2 in spans:
- if span1 != span2:
- # span1 或 span2 任何一个都不应该在 dropped_spans 中
- if span1 in dropped_spans or span2 in dropped_spans:
- continue
- else:
- if calculate_iou(span1['bbox'], span2['bbox']) > 0.9:
- if span1['score'] < span2['score']:
- span_need_remove = span1
- else:
- span_need_remove = span2
- if span_need_remove is not None and span_need_remove not in dropped_spans:
- dropped_spans.append(span_need_remove)
-
- if len(dropped_spans) > 0:
- for span_need_remove in dropped_spans:
- spans.remove(span_need_remove)
- span_need_remove['tag'] = DropTag.SPAN_OVERLAP
-
- return spans, dropped_spans
-
-
-def remove_overlaps_min_spans(spans):
- dropped_spans = []
- # 删除重叠spans中较小的那些
- for span1 in spans:
- for span2 in spans:
- if span1 != span2:
- overlap_box = get_minbox_if_overlap_by_ratio(span1['bbox'], span2['bbox'], 0.65)
- if overlap_box is not None:
- span_need_remove = next((span for span in spans if span['bbox'] == overlap_box), None)
- if span_need_remove is not None and span_need_remove not in dropped_spans:
- dropped_spans.append(span_need_remove)
-
- if len(dropped_spans) > 0:
- for span_need_remove in dropped_spans:
- spans.remove(span_need_remove)
- span_need_remove['tag'] = DropTag.SPAN_OVERLAP
-
- return spans, dropped_spans
-
-
-def remove_spans_by_bboxes(spans, need_remove_spans_bboxes):
- # 遍历spans, 判断是否在removed_span_block_bboxes中
- # 如果是, 则删除该span 否则, 保留该span
- need_remove_spans = []
- for span in spans:
- for removed_bbox in need_remove_spans_bboxes:
- if calculate_overlap_area_in_bbox1_area_ratio(span['bbox'], removed_bbox) > 0.5:
- if span not in need_remove_spans:
- need_remove_spans.append(span)
- break
-
- if len(need_remove_spans) > 0:
- for span in need_remove_spans:
- spans.remove(span)
-
- return spans
-
-
-def remove_spans_by_bboxes_dict(spans, need_remove_spans_bboxes_dict):
- dropped_spans = []
- for drop_tag, removed_bboxes in need_remove_spans_bboxes_dict.items():
- # logger.info(f"remove spans by bbox dict, drop_tag: {drop_tag}, removed_bboxes: {removed_bboxes}")
- need_remove_spans = []
- for span in spans:
- # 通过判断span的bbox是否在removed_bboxes中, 判断是否需要删除该span
- for removed_bbox in removed_bboxes:
- if calculate_overlap_area_in_bbox1_area_ratio(span['bbox'], removed_bbox) > 0.5:
- need_remove_spans.append(span)
- break
- # 当drop_tag为DropTag.FOOTNOTE时, 判断span是否在removed_bboxes中任意一个的下方，如果是,则删除该span
- elif drop_tag == DropTag.FOOTNOTE and (span['bbox'][1] + span['bbox'][3]) / 2 > removed_bbox[3] and \
- removed_bbox[0] < (span['bbox'][0] + span['bbox'][2]) / 2 < removed_bbox[2]:
- need_remove_spans.append(span)
- break
-
- for span in need_remove_spans:
- spans.remove(span)
- span['tag'] = drop_tag
- dropped_spans.append(span)
-
- return spans, dropped_spans
-
-
-def adjust_bbox_for_standalone_block(spans):
- # 对tpye=["interline_equation", "image", "table"]进行额外处理,如果左边有字的话,将该span的bbox中y0调整至不高于文字的y0
- for sb_span in spans:
- if sb_span['type'] in [ContentType.InterlineEquation, ContentType.Image, ContentType.Table]:
- for text_span in spans:
- if text_span['type'] in [ContentType.Text, ContentType.InlineEquation]:
- # 判断span2的纵向高度是否被span所覆盖
- if sb_span['bbox'][1] < text_span['bbox'][1] and sb_span['bbox'][3] > text_span['bbox'][3]:
- # 判断span2是否在span左边
- if text_span['bbox'][0] < sb_span['bbox'][0]:
- # 调整span的y0和span2的y0一致
- sb_span['bbox'][1] = text_span['bbox'][1]
- return spans
-
-
-def modify_y_axis(spans: list, displayed_list: list, text_inline_lines: list):
- # displayed_list = []
- # 如果spans为空,则不处理
- if len(spans) == 0:
- pass
- else:
- spans.sort(key=lambda span: span['bbox'][1])
-
- lines = []
- current_line = [spans[0]]
- if spans[0]["type"] in [ContentType.InterlineEquation, ContentType.Image, ContentType.Table]:
- displayed_list.append(spans[0])
-
- line_first_y0 = spans[0]["bbox"][1]
- line_first_y = spans[0]["bbox"][3]
- # 用于给行间公式搜索
- # text_inline_lines = []
- for span in spans[1:]:
- # if span.get("content","") == "78.":
- # print("debug")
- # 如果当前的span类型为"interline_equation" 或者 当前行中已经有"interline_equation"
- # image和table类型，同上
- if span['type'] in [ContentType.InterlineEquation, ContentType.Image, ContentType.Table] or any(
- s['type'] in [ContentType.InterlineEquation, ContentType.Image, ContentType.Table] for s in
- current_line):
- # 传入
- if span["type"] in [ContentType.InterlineEquation, ContentType.Image, ContentType.Table]:
- displayed_list.append(span)
- # 则开始新行
- lines.append(current_line)
- if len(current_line) > 1 or current_line[0]["type"] in [ContentType.Text, ContentType.InlineEquation]:
- text_inline_lines.append((current_line, (line_first_y0, line_first_y)))
- current_line = [span]
- line_first_y0 = span["bbox"][1]
- line_first_y = span["bbox"][3]
- continue
-
- # 如果当前的span与当前行的最后一个span在y轴上重叠，则添加到当前行
- if __is_overlaps_y_exceeds_threshold(span['bbox'], current_line[-1]['bbox']):
- if span["type"] == "text":
- line_first_y0 = span["bbox"][1]
- line_first_y = span["bbox"][3]
- current_line.append(span)
-
- else:
- # 否则，开始新行
- lines.append(current_line)
- text_inline_lines.append((current_line, (line_first_y0, line_first_y)))
- current_line = [span]
- line_first_y0 = span["bbox"][1]
- line_first_y = span["bbox"][3]
-
- # 添加最后一行
- if current_line:
- lines.append(current_line)
- if len(current_line) > 1 or current_line[0]["type"] in [ContentType.Text, ContentType.InlineEquation]:
- text_inline_lines.append((current_line, (line_first_y0, line_first_y)))
- for line in text_inline_lines:
- # 按照x0坐标排序
- current_line = line[0]
- current_line.sort(key=lambda span: span['bbox'][0])
-
- # 调整每一个文字行内bbox统一
- for line in text_inline_lines:
- current_line, (line_first_y0, line_first_y) = line
- for span in current_line:
- span["bbox"][1] = line_first_y0
- span["bbox"][3] = line_first_y
-
- # return spans, displayed_list, text_inline_lines
-
-
-def modify_inline_equation(spans: list, displayed_list: list, text_inline_lines: list):
- # 错误行间公式转行内公式
- j = 0
- for i in range(len(displayed_list)):
- # if i == 8:
- # print("debug")
- span = displayed_list[i]
- span_y0, span_y = span["bbox"][1], span["bbox"][3]
-
- while j < len(text_inline_lines):
- text_line = text_inline_lines[j]
- y0, y1 = text_line[1]
- if (
- span_y0 < y0 < span_y or span_y0 < y1 < span_y or span_y0 < y0 and span_y > y1
- ) and __is_overlaps_y_exceeds_threshold(
- span['bbox'], (0, y0, 0, y1)
- ):
- # 调整公式类型
- if span["type"] == ContentType.InterlineEquation:
- # 最后一行是行间公式
- if j + 1 >= len(text_inline_lines):
- span["type"] = ContentType.InlineEquation
- span["bbox"][1] = y0
- span["bbox"][3] = y1
- else:
- # 行间公式旁边有多行文字或者行间公式比文字高3倍则不转换
- y0_next, y1_next = text_inline_lines[j + 1][1]
- if not __is_overlaps_y_exceeds_threshold(span['bbox'], (0, y0_next, 0, y1_next)) and 3 * (
- y1 - y0) > span_y - span_y0:
- span["type"] = ContentType.InlineEquation
- span["bbox"][1] = y0
- span["bbox"][3] = y1
- break
- elif span_y < y0 or span_y0 < y0 < span_y and not __is_overlaps_y_exceeds_threshold(span['bbox'],
- (0, y0, 0, y1)):
- break
- else:
- j += 1
-
- return spans
-
-
-def get_qa_need_list(blocks):
- # 创建 images, tables, interline_equations, inline_equations 的副本
- images = []
- tables = []
- interline_equations = []
- inline_equations = []
-
- for block in blocks:
- for line in block["lines"]:
- for span in line["spans"]:
- if span["type"] == ContentType.Image:
- images.append(span)
- elif span["type"] == ContentType.Table:
- tables.append(span)
- elif span["type"] == ContentType.InlineEquation:
- inline_equations.append(span)
- elif span["type"] == ContentType.InterlineEquation:
- interline_equations.append(span)
- else:
- continue
- return images, tables, interline_equations, inline_equations
-
-
-def get_qa_need_list_v2(blocks):
- # 创建 images, tables, interline_equations, inline_equations 的副本
- images = []
- tables = []
- interline_equations = []
-
- for block in blocks:
- if block["type"] == BlockType.Image:
- images.append(block)
- elif block["type"] == BlockType.Table:
- tables.append(block)
- elif block["type"] == BlockType.InterlineEquation:
- interline_equations.append(block)
- return images, tables, interline_equations
diff --git a/magic_pdf/pre_proc/pdf_pre_filter.py b/magic_pdf/pre_proc/pdf_pre_filter.py
deleted file mode 100644
index 1704d0efb..000000000
--- a/magic_pdf/pre_proc/pdf_pre_filter.py
+++ /dev/null
@@ -1,74 +0,0 @@
-from magic_pdf.libs.commons import fitz
-from magic_pdf.libs.boxbase import _is_in, _is_in_or_part_overlap
-from magic_pdf.libs.drop_reason import DropReason
-
-
-def __area(box):
- return (box[2] - box[0]) * (box[3] - box[1])
-
-def __is_contain_color_background_rect(page:fitz.Page, text_blocks, image_bboxes) -> bool:
- """
- 检查page是包含有颜色背景的矩形
- """
- color_bg_rect = []
- p_width, p_height = page.rect.width, page.rect.height
-
- # 先找到最大的带背景矩形
- blocks = page.get_cdrawings()
- for block in blocks:
-
- if 'fill' in block and block['fill']: # 过滤掉透明的
- fill = list(block['fill'])
- fill[0], fill[1], fill[2] = int(fill[0]), int(fill[1]), int(fill[2])
- if fill==(1.0,1.0,1.0):
- continue
- rect = block['rect']
- # 过滤掉特别小的矩形
- if __area(rect) < 10*10:
- continue
- # 为了防止是svg图片上的色块，这里过滤掉这类
-
- if any([_is_in_or_part_overlap(rect, img_bbox) for img_bbox in image_bboxes]):
- continue
- color_bg_rect.append(rect)
-
- # 找到最大的背景矩形
- if len(color_bg_rect) > 0:
- max_rect = max(color_bg_rect, key=lambda x:__area(x))
- max_rect_int = (int(max_rect[0]), int(max_rect[1]), int(max_rect[2]), int(max_rect[3]))
- # 判断最大的背景矩形是否包含超过3行文字，或者50个字 TODO
- if max_rect[2]-max_rect[0] > 0.2*p_width and max_rect[3]-max_rect[1] > 0.1*p_height:#宽度符合
- #看是否有文本块落入到这个矩形中
- for text_block in text_blocks:
- box = text_block['bbox']
- box_int = (int(box[0]), int(box[1]), int(box[2]), int(box[3]))
- if _is_in(box_int, max_rect_int):
- return True
-
- return False
-
-
-def __is_table_overlap_text_block(text_blocks, table_bbox):
- """
- 检查table_bbox是否覆盖了text_blocks里的文本块
- TODO
- """
- for text_block in text_blocks:
- box = text_block['bbox']
- if _is_in_or_part_overlap(table_bbox, box):
- return True
- return False
-
-
-def pdf_filter(page:fitz.Page, text_blocks, table_bboxes, image_bboxes) -> tuple:
- """
- return:(True|False, err_msg)
- True, 如果pdf符合要求
- False, 如果pdf不符合要求
-
- """
- if __is_contain_color_background_rect(page, text_blocks, image_bboxes):
- return False, {"_need_drop": True, "_drop_reason": DropReason.COLOR_BACKGROUND_TEXT_BOX}
-
-
- return True, None
\ No newline at end of file
diff --git a/magic_pdf/pre_proc/post_layout_split.py b/magic_pdf/pre_proc/post_layout_split.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/pre_proc/remove_bbox_overlap.py b/magic_pdf/pre_proc/remove_bbox_overlap.py
deleted file mode 100644
index 2afb07119..000000000
--- a/magic_pdf/pre_proc/remove_bbox_overlap.py
+++ /dev/null
@@ -1,98 +0,0 @@
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap, _is_in, _is_part_overlap
-from magic_pdf.libs.drop_reason import DropReason
-
-def _remove_overlap_between_bbox(bbox1, bbox2):
- if _is_part_overlap(bbox1, bbox2):
- ix0, iy0, ix1, iy1 = bbox1
- x0, y0, x1, y1 = bbox2
-
- diff_x = min(x1, ix1) - max(x0, ix0)
- diff_y = min(y1, iy1) - max(y0, iy0)
-
- if diff_y > diff_x:
- if x1 >= ix1:
- mid = (x0 + ix1) // 2
- ix1 = min(mid - 0.25, ix1)
- x0 = max(mid + 0.25, x0)
- else:
- mid = (ix0 + x1) // 2
- ix0 = max(mid + 0.25, ix0)
- x1 = min(mid - 0.25, x1)
- else:
- if y1 >= iy1:
- mid = (y0 + iy1) // 2
- y0 = max(mid + 0.25, y0)
- iy1 = min(iy1, mid-0.25)
- else:
- mid = (iy0 + y1) // 2
- y1 = min(y1, mid-0.25)
- iy0 = max(mid + 0.25, iy0)
-
- if ix1 > ix0 and iy1 > iy0 and y1 > y0 and x1 > x0:
- bbox1 = [ix0, iy0, ix1, iy1]
- bbox2 = [x0, y0, x1, y1]
- return bbox1, bbox2, None
- else:
- return bbox1, bbox2, DropReason.NEGATIVE_BBOX_AREA
- else:
- return bbox1, bbox2, None
-
-
-def _remove_overlap_between_bboxes(arr):
- drop_reasons = []
- N = len(arr)
- keeps = [True] * N
- res = [None] * N
- for i in range(N):
- for j in range(N):
- if i == j:
- continue
- if _is_in(arr[i]["bbox"], arr[j]["bbox"]):
- keeps[i] = False
-
- for idx, v in enumerate(arr):
- if not keeps[idx]:
- continue
- for i in range(N):
- if res[i] is None:
- continue
-
- bbox1, bbox2, drop_reason = _remove_overlap_between_bbox(v["bbox"], res[i]["bbox"])
- if drop_reason is None:
- v["bbox"] = bbox1
- res[i]["bbox"] = bbox2
- else:
- if v["score"] > res[i]["score"]:
- keeps[i] = False
- res[i] = None
- else:
- keeps[idx] = False
- drop_reasons.append(drop_reasons)
- if keeps[idx]:
- res[idx] = v
- return res, drop_reasons
-
-
-def remove_overlap_between_bbox_for_span(spans):
- arr = [{"bbox": span["bbox"], "score": span.get("score", 0.1)} for span in spans ]
- res, drop_reasons = _remove_overlap_between_bboxes(arr)
- ret = []
- for i in range(len(res)):
- if res[i] is None:
- continue
- spans[i]["bbox"] = res[i]["bbox"]
- ret.append(spans[i])
- return ret, drop_reasons
-
-
-def remove_overlap_between_bbox_for_block(all_bboxes):
- arr = [{"bbox": bbox[:4], "score": bbox[-1]} for bbox in all_bboxes ]
- res, drop_reasons = _remove_overlap_between_bboxes(arr)
- ret = []
- for i in range(len(res)):
- if res[i] is None:
- continue
- all_bboxes[i][:4] = res[i]["bbox"]
- ret.append(all_bboxes[i])
- return ret, drop_reasons
-
diff --git a/magic_pdf/pre_proc/remove_colored_strip_bbox.py b/magic_pdf/pre_proc/remove_colored_strip_bbox.py
deleted file mode 100644
index 17be73aa5..000000000
--- a/magic_pdf/pre_proc/remove_colored_strip_bbox.py
+++ /dev/null
@@ -1,79 +0,0 @@
-from magic_pdf.libs.boxbase import _is_in, _is_in_or_part_overlap, calculate_overlap_area_2_minbox_area_ratio
-from loguru import logger
-
-from magic_pdf.libs.drop_tag import COLOR_BG_HEADER_TXT_BLOCK
-
-
-def __area(box):
- return (box[2] - box[0]) * (box[3] - box[1])
-
-
-def rectangle_position_determination(rect, p_width):
- """
- 判断矩形是否在页面中轴线附近。
-
- Args:
- rect (list): 矩形坐标，格式为[x1, y1, x2, y2]。
- p_width (int): 页面宽度。
-
- Returns:
- bool: 若矩形在页面中轴线附近则返回True，否则返回False。
- """
- # 页面中轴线x坐标
- x_axis = p_width / 2
- # 矩形是否跨越中轴线
- is_span = rect[0] < x_axis and rect[2] > x_axis
- if is_span:
- return True
- else:
- # 矩形与中轴线的距离，只算近的那一边
- distance = rect[0] - x_axis if rect[0] > x_axis else x_axis - rect[2]
- # 判断矩形与中轴线的距离是否小于页面宽度的20%
- if distance < p_width * 0.2:
- return True
- else:
- return False
-
-def remove_colored_strip_textblock(remain_text_blocks, page):
- """
- 根据页面中特定颜色和大小过滤文本块，将符合条件的文本块从remain_text_blocks中移除，并返回移除的文本块列表colored_strip_textblock。
-
- Args:
- remain_text_blocks (list): 剩余文本块列表。
- page (Page): 页面对象。
-
- Returns:
- tuple: 剩余文本块列表和移除的文本块列表。
- """
- colored_strip_textblocks = [] # 先构造一个空的返回
- if len(remain_text_blocks) > 0:
- p_width, p_height = page.rect.width, page.rect.height
- blocks = page.get_cdrawings()
- colored_strip_bg_rect = []
- for block in blocks:
- is_filled = 'fill' in block and block['fill'] and block['fill'] != (1.0, 1.0, 1.0) # 过滤掉透明的
- rect = block['rect']
- area_is_large_enough = __area(rect) > 100 # 过滤掉特别小的矩形
- rectangle_position_determination_result = rectangle_position_determination(rect, p_width)
- in_upper_half_page = rect[3] < p_height * 0.3 # 找到位于页面上半部分的矩形，下边界小于页面高度的30%
- aspect_ratio_exceeds_4 = (rect[2] - rect[0]) > (rect[3] - rect[1]) * 4 # 找到长宽比超过4的矩形
-
- if is_filled and area_is_large_enough and rectangle_position_determination_result and in_upper_half_page and aspect_ratio_exceeds_4:
- colored_strip_bg_rect.append(rect)
-
- if len(colored_strip_bg_rect) > 0:
- for colored_strip_block_bbox in colored_strip_bg_rect:
- for text_block in remain_text_blocks:
- text_bbox = text_block['bbox']
- if _is_in(text_bbox, colored_strip_block_bbox) or (_is_in_or_part_overlap(text_bbox, colored_strip_block_bbox) and calculate_overlap_area_2_minbox_area_ratio(text_bbox, colored_strip_block_bbox) > 0.6):
- logger.info(f'remove_colored_strip_textblock: {text_bbox}, {colored_strip_block_bbox}')
- text_block['tag'] = COLOR_BG_HEADER_TXT_BLOCK
- colored_strip_textblocks.append(text_block)
-
- if len(colored_strip_textblocks) > 0:
- for colored_strip_textblock in colored_strip_textblocks:
- if colored_strip_textblock in remain_text_blocks:
- remain_text_blocks.remove(colored_strip_textblock)
-
- return remain_text_blocks, colored_strip_textblocks
-
diff --git a/magic_pdf/pre_proc/remove_footer_header.py b/magic_pdf/pre_proc/remove_footer_header.py
deleted file mode 100644
index 9e04817c3..000000000
--- a/magic_pdf/pre_proc/remove_footer_header.py
+++ /dev/null
@@ -1,117 +0,0 @@
-import re
-
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap
-from magic_pdf.libs.drop_tag import CONTENT_IN_FOOT_OR_HEADER, PAGE_NO
-
-
-def remove_headder_footer_one_page(text_raw_blocks, image_bboxes, table_bboxes, header_bboxs, footer_bboxs,
- page_no_bboxs, page_w, page_h):
- """
- 删除页眉页脚，页码
- 从line级别进行删除，删除之后观察这个text-block是否是空的，如果是空的，则移动到remove_list中
- """
- header = []
- footer = []
- if len(header) == 0:
- model_header = header_bboxs
- if model_header:
- x0 = min([x for x, _, _, _ in model_header])
- y0 = min([y for _, y, _, _ in model_header])
- x1 = max([x1 for _, _, x1, _ in model_header])
- y1 = max([y1 for _, _, _, y1 in model_header])
- header = [x0, y0, x1, y1]
- if len(footer) == 0:
- model_footer = footer_bboxs
- if model_footer:
- x0 = min([x for x, _, _, _ in model_footer])
- y0 = min([y for _, y, _, _ in model_footer])
- x1 = max([x1 for _, _, x1, _ in model_footer])
- y1 = max([y1 for _, _, _, y1 in model_footer])
- footer = [x0, y0, x1, y1]
-
- header_y0 = 0 if len(header) == 0 else header[3]
- footer_y0 = page_h if len(footer) == 0 else footer[1]
- if page_no_bboxs:
- top_part = [b for b in page_no_bboxs if b[3] < page_h / 2]
- btn_part = [b for b in page_no_bboxs if b[1] > page_h / 2]
-
- top_max_y0 = max([b[1] for b in top_part]) if top_part else 0
- btn_min_y1 = min([b[3] for b in btn_part]) if btn_part else page_h
-
- header_y0 = max(header_y0, top_max_y0)
- footer_y0 = min(footer_y0, btn_min_y1)
-
- content_boundry = [0, header_y0, page_w, footer_y0]
-
- header = [0, 0, page_w, header_y0]
- footer = [0, footer_y0, page_w, page_h]
-
- """以上计算出来了页眉页脚的边界，下面开始进行删除"""
- text_block_to_remove = []
- # 首先检查每个textblock
- for blk in text_raw_blocks:
- if len(blk['lines']) > 0:
- for line in blk['lines']:
- line_del = []
- for span in line['spans']:
- span_del = []
- if span['bbox'][3] < header_y0:
- span_del.append(span)
- elif _is_in_or_part_overlap(span['bbox'], header) or _is_in_or_part_overlap(span['bbox'], footer):
- span_del.append(span)
- for span in span_del:
- line['spans'].remove(span)
- if not line['spans']:
- line_del.append(line)
-
- for line in line_del:
- blk['lines'].remove(line)
- else:
- # if not blk['lines']:
- blk['tag'] = CONTENT_IN_FOOT_OR_HEADER
- text_block_to_remove.append(blk)
-
- """有的时候由于pageNo太小了，总是会有一点和content_boundry重叠一点，被放入正文，因此对于pageNo，进行span粒度的删除"""
- page_no_block_2_remove = []
- if page_no_bboxs:
- for pagenobox in page_no_bboxs:
- for block in text_raw_blocks:
- if _is_in_or_part_overlap(pagenobox, block['bbox']): # 在span级别删除页码
- for line in block['lines']:
- for span in line['spans']:
- if _is_in_or_part_overlap(pagenobox, span['bbox']):
- # span['text'] = ''
- span['tag'] = PAGE_NO
- # 检查这个block是否只有这一个span，如果是，那么就把这个block也删除
- if len(line['spans']) == 1 and len(block['lines']) == 1:
- page_no_block_2_remove.append(block)
- else:
- # 测试最后一个是不是页码：规则是，最后一个block仅有1个line,一个span,且text是数字，空格，符号组成，不含字母,并且包含数字
- if len(text_raw_blocks) > 0:
- text_raw_blocks.sort(key=lambda x: x['bbox'][1], reverse=True)
- last_block = text_raw_blocks[0]
- if len(last_block['lines']) == 1:
- last_line = last_block['lines'][0]
- if len(last_line['spans']) == 1:
- last_span = last_line['spans'][0]
- if last_span['text'].strip() and not re.search('[a-zA-Z]', last_span['text']) and re.search('[0-9]',
- last_span[
- 'text']):
- last_span['tag'] = PAGE_NO
- page_no_block_2_remove.append(last_block)
-
- for b in page_no_block_2_remove:
- text_block_to_remove.append(b)
-
- for blk in text_block_to_remove:
- if blk in text_raw_blocks:
- text_raw_blocks.remove(blk)
-
- text_block_remain = text_raw_blocks
- image_bbox_to_remove = [bbox for bbox in image_bboxes if not _is_in_or_part_overlap(bbox, content_boundry)]
-
- image_bbox_remain = [bbox for bbox in image_bboxes if _is_in_or_part_overlap(bbox, content_boundry)]
- table_bbox_to_remove = [bbox for bbox in table_bboxes if not _is_in_or_part_overlap(bbox, content_boundry)]
- table_bbox_remain = [bbox for bbox in table_bboxes if _is_in_or_part_overlap(bbox, content_boundry)]
-
- return image_bbox_remain, table_bbox_remain, text_block_remain, text_block_to_remove, image_bbox_to_remove, table_bbox_to_remove
diff --git a/magic_pdf/pre_proc/remove_rotate_bbox.py b/magic_pdf/pre_proc/remove_rotate_bbox.py
deleted file mode 100644
index d47aa51c5..000000000
--- a/magic_pdf/pre_proc/remove_rotate_bbox.py
+++ /dev/null
@@ -1,188 +0,0 @@
-import math
-
-from magic_pdf.libs.boxbase import is_vbox_on_side
-from magic_pdf.libs.drop_tag import EMPTY_SIDE_BLOCK, ROTATE_TEXT, VERTICAL_TEXT
-
-
-def detect_non_horizontal_texts(result_dict):
- """
- This function detects watermarks and vertical margin notes in the document.
-
- Watermarks are identified by finding blocks with the same coordinates and frequently occurring identical texts across multiple pages.
- If these conditions are met, the blocks are highly likely to be watermarks, as opposed to headers or footers, which can change from page to page.
- If the direction of these blocks is not horizontal, they are definitely considered to be watermarks.
-
- Vertical margin notes are identified by finding blocks with the same coordinates and frequently occurring identical texts across multiple pages.
- If these conditions are met, the blocks are highly likely to be vertical margin notes, which typically appear on the left and right sides of the page.
- If the direction of these blocks is vertical, they are definitely considered to be vertical margin notes.
-
-
- Parameters
- ----------
- result_dict : dict
- The result dictionary.
-
- Returns
- -------
- result_dict : dict
- The updated result dictionary.
- """
- # Dictionary to store information about potential watermarks
- potential_watermarks = {}
- potential_margin_notes = {}
-
- for page_id, page_content in result_dict.items():
- if page_id.startswith("page_"):
- for block_id, block_data in page_content.items():
- if block_id.startswith("block_"):
- if "dir" in block_data:
- coordinates_text = (block_data["bbox"], block_data["text"]) # Tuple of coordinates and text
-
- angle = math.atan2(block_data["dir"][1], block_data["dir"][0])
- angle = abs(math.degrees(angle))
-
- if angle > 5 and angle < 85: # Check if direction is watermarks
- if coordinates_text in potential_watermarks:
- potential_watermarks[coordinates_text] += 1
- else:
- potential_watermarks[coordinates_text] = 1
-
- if angle > 85 and angle < 105: # Check if direction is vertical
- if coordinates_text in potential_margin_notes:
- potential_margin_notes[coordinates_text] += 1 # Increment count
- else:
- potential_margin_notes[coordinates_text] = 1 # Initialize count
-
- # Identify watermarks by finding entries with counts higher than a threshold (e.g., appearing on more than half of the pages)
- watermark_threshold = len(result_dict) // 2
- watermarks = {k: v for k, v in potential_watermarks.items() if v > watermark_threshold}
-
- # Identify margin notes by finding entries with counts higher than a threshold (e.g., appearing on more than half of the pages)
- margin_note_threshold = len(result_dict) // 2
- margin_notes = {k: v for k, v in potential_margin_notes.items() if v > margin_note_threshold}
-
- # Add watermark information to the result dictionary
- for page_id, blocks in result_dict.items():
- if page_id.startswith("page_"):
- for block_id, block_data in blocks.items():
- coordinates_text = (block_data["bbox"], block_data["text"])
- if coordinates_text in watermarks:
- block_data["is_watermark"] = 1
- else:
- block_data["is_watermark"] = 0
-
- if coordinates_text in margin_notes:
- block_data["is_vertical_margin_note"] = 1
- else:
- block_data["is_vertical_margin_note"] = 0
-
- return result_dict
-
-
-"""
-1. 当一个block里全部文字都不是dir=(1,0)，这个block整体去掉
-2. 当一个block里全部文字都是dir=(1,0)，但是每行只有一个字，这个block整体去掉。这个block必须出现在页面的四周，否则不去掉
-"""
-import re
-
-def __is_a_word(sentence):
- # 如果输入是中文并且长度为1，则返回True
- if re.fullmatch(r'[\u4e00-\u9fa5]', sentence):
- return True
- # 判断是否为单个英文单词或字符（包括ASCII标点）
- elif re.fullmatch(r'[a-zA-Z0-9]+', sentence) and len(sentence) <=2:
- return True
- else:
- return False
-
-
-def __get_text_color(num):
- """获取字体的颜色RGB值"""
- blue = num & 255
- green = (num >> 8) & 255
- red = (num >> 16) & 255
- return red, green, blue
-
-
-def __is_empty_side_box(text_block):
- """
- 是否是边缘上的空白没有任何内容的block
- """
- for line in text_block['lines']:
- for span in line['spans']:
- font_color = span['color']
- r,g,b = __get_text_color(font_color)
- if len(span['text'].strip())>0 and (r,g,b)!=(255,255,255):
- return False
-
- return True
-
-
-def remove_rotate_side_textblock(pymu_text_block, page_width, page_height):
- """
- 返回删除了垂直，水印，旋转的textblock
- 删除的内容打上tag返回
- """
- removed_text_block = []
-
- for i, block in enumerate(pymu_text_block): # 格式参考test/assets/papre/pymu_textblocks.json
- lines = block['lines']
- block_bbox = block['bbox']
- if not is_vbox_on_side(block_bbox, page_width, page_height, 0.2): # 保证这些box必须在页面的两边
- continue
-
- if all([__is_a_word(line['spans'][0]["text"]) for line in lines if len(line['spans'])>0]) and len(lines)>1 and all([len(line['spans'])==1 for line in lines]):
- is_box_valign = (len(set([int(line['spans'][0]['bbox'][0] ) for line in lines if len(line['spans'])>0]))==1) and (len([int(line['spans'][0]['bbox'][0] ) for line in lines if len(line['spans'])>0])>1) # 测试bbox在垂直方向是不是x0都相等，也就是在垂直方向排列.同时必须大于等于2个字
-
- if is_box_valign:
- block['tag'] = VERTICAL_TEXT
- removed_text_block.append(block)
- continue
-
- for line in lines:
- if line['dir']!=(1,0):
- block['tag'] = ROTATE_TEXT
- removed_text_block.append(block) # 只要有一个line不是dir=(1,0)，就把整个block都删掉
- break
-
- for block in removed_text_block:
- pymu_text_block.remove(block)
-
- return pymu_text_block, removed_text_block
-
-def get_side_boundry(rotate_bbox, page_width, page_height):
- """
- 根据rotate_bbox，返回页面的左右正文边界
- """
- left_x = 0
- right_x = page_width
- for x in rotate_bbox:
- box = x['bbox']
- if box[2]0.05:
- # text_block['tag'] = "belong-to-interline-equation"
- # text_block_removed_2.append(text_block)
- # break
-
- # for tb in text_block_removed_2:
- # if tb in text_raw_blocks:
- # text_raw_blocks.remove(tb)
-
- # text_block_removed = text_block_removed + text_block_removed_2
-
- return images, tables, interline_equations, inline_equations, text_raw_blocks, text_block_removed, images_backup, text_block_removed_2
-
-
-def check_text_block_horizontal_overlap(text_blocks: list, header, footer) -> bool:
- """
- 检查文本block之间的水平重叠情况，这种情况如果发生，那么这个pdf就不再继续处理了。
- 因为这种情况大概率发生了公式没有被检测出来。
-
- """
- if len(text_blocks) == 0:
- return False
-
- page_min_y = 0
- page_max_y = max(yy['bbox'][3] for yy in text_blocks)
-
- def __max_y(lst: list):
- if len(lst) > 0:
- return max([item[1] for item in lst])
- return page_min_y
-
- def __min_y(lst: list):
- if len(lst) > 0:
- return min([item[3] for item in lst])
- return page_max_y
-
- clip_y0 = __max_y(header)
- clip_y1 = __min_y(footer)
-
- txt_bboxes = []
- for text_block in text_blocks:
- bbox = text_block["bbox"]
- if bbox[1] >= clip_y0 and bbox[3] <= clip_y1:
- txt_bboxes.append(bbox)
-
- for i in range(len(txt_bboxes)):
- for j in range(i + 1, len(txt_bboxes)):
- if _is_left_overlap(txt_bboxes[i], txt_bboxes[j]) or _is_left_overlap(txt_bboxes[j], txt_bboxes[i]):
- return True
-
- return False
-
-
-def check_useful_block_horizontal_overlap(useful_blocks: list) -> bool:
- """
- 检查文本block之间的水平重叠情况，这种情况如果发生，那么这个pdf就不再继续处理了。
- 因为这种情况大概率发生了公式没有被检测出来。
-
- """
- if len(useful_blocks) == 0:
- return False
-
- page_min_y = 0
- page_max_y = max(yy['bbox'][3] for yy in useful_blocks)
-
- useful_bboxes = []
- for text_block in useful_blocks:
- bbox = text_block["bbox"]
- if bbox[1] >= page_min_y and bbox[3] <= page_max_y:
- useful_bboxes.append(bbox)
-
- for i in range(len(useful_bboxes)):
- for j in range(i + 1, len(useful_bboxes)):
- area_i = (useful_bboxes[i][2] - useful_bboxes[i][0]) * (useful_bboxes[i][3] - useful_bboxes[i][1])
- area_j = (useful_bboxes[j][2] - useful_bboxes[j][0]) * (useful_bboxes[j][3] - useful_bboxes[j][1])
- if _is_left_overlap(useful_bboxes[i], useful_bboxes[j]) or _is_left_overlap(useful_bboxes[j], useful_bboxes[i]):
- if area_i > area_j:
- return True, useful_bboxes[j], useful_bboxes[i]
- else:
- return True, useful_bboxes[i], useful_bboxes[j]
-
- return False, None, None
diff --git a/magic_pdf/pre_proc/solve_line_alien.py b/magic_pdf/pre_proc/solve_line_alien.py
deleted file mode 100644
index 966fb89ea..000000000
--- a/magic_pdf/pre_proc/solve_line_alien.py
+++ /dev/null
@@ -1,29 +0,0 @@
-def solve_inline_too_large_interval(pdf_info_dict: dict) -> dict: # text_block -> json中的preproc_block
- """解决行内文本间距过大问题"""
- for i in range(len(pdf_info_dict)):
-
- text_blocks = pdf_info_dict[f'page_{i}']['preproc_blocks']
-
- for block in text_blocks:
-
- x_pre_1, y_pre_1, x_pre_2, y_pre_2 = 0, 0, 0, 0
-
- for line in block['lines']:
-
- x_cur_1, y_cur_1, x_cur_2, y_cur_2 = line['bbox']
- # line_box = [x1, y1, x2, y2]
- if int(y_cur_1) == int(y_pre_1) and int(y_cur_2) == int(y_pre_2):
- # if len(line['spans']) == 1:
- line['spans'][0]['text'] = ' ' + line['spans'][0]['text']
-
- x_pre_1, y_pre_1, x_pre_2, y_pre_2 = line['bbox']
-
- return pdf_info_dict
-
-
-
-
-
-
-
-
diff --git a/magic_pdf/pre_proc/statistics.py b/magic_pdf/pre_proc/statistics.py
deleted file mode 100644
index 5bf7c78fe..000000000
--- a/magic_pdf/pre_proc/statistics.py
+++ /dev/null
@@ -1,12 +0,0 @@
-
-"""
-统计处需要跨页、全局性的数据
-- 统计出字号从大到小
-- 正文区域占比最高的前5
-- 正文平均行间距
-- 正文平均字间距
-- 正文平均字符宽度
-- 正文平均字符高度
-
-"""
-
diff --git a/magic_pdf/resources/model_config/UniMERNet/demo.yaml b/magic_pdf/resources/model_config/UniMERNet/demo.yaml
deleted file mode 100644
index de83b07ae..000000000
--- a/magic_pdf/resources/model_config/UniMERNet/demo.yaml
+++ /dev/null
@@ -1,46 +0,0 @@
-model:
- arch: unimernet
- model_type: unimernet
- model_config:
- model_name: ./models/unimernet_base
- max_seq_len: 1536
-
- load_pretrained: True
- pretrained: './models/unimernet_base/pytorch_model.pth'
- tokenizer_config:
- path: ./models/unimernet_base
-
-datasets:
- formula_rec_eval:
- vis_processor:
- eval:
- name: "formula_image_eval"
- image_size:
- - 192
- - 672
-
-run:
- runner: runner_iter
- task: unimernet_train
-
- batch_size_train: 64
- batch_size_eval: 64
- num_workers: 1
-
- iters_per_inner_epoch: 2000
- max_iters: 60000
-
- seed: 42
- output_dir: "../output/demo"
-
- evaluate: True
- test_splits: [ "eval" ]
-
- device: "cuda"
- world_size: 1
- dist_url: "env://"
- distributed: True
- distributed_type: ddp # or fsdp when train llm
-
- generate_cfg:
- temperature: 0.0
\ No newline at end of file
diff --git a/magic_pdf/resources/model_config/layoutlmv3/layoutlmv3_base_inference.yaml b/magic_pdf/resources/model_config/layoutlmv3/layoutlmv3_base_inference.yaml
deleted file mode 100644
index 3e0fd309f..000000000
--- a/magic_pdf/resources/model_config/layoutlmv3/layoutlmv3_base_inference.yaml
+++ /dev/null
@@ -1,351 +0,0 @@
-AUG:
- DETR: true
-CACHE_DIR: ~/cache/huggingface
-CUDNN_BENCHMARK: false
-DATALOADER:
- ASPECT_RATIO_GROUPING: true
- FILTER_EMPTY_ANNOTATIONS: false
- NUM_WORKERS: 4
- REPEAT_THRESHOLD: 0.0
- SAMPLER_TRAIN: TrainingSampler
-DATASETS:
- PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000
- PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000
- PROPOSAL_FILES_TEST: []
- PROPOSAL_FILES_TRAIN: []
- TEST:
- - scihub_train
- TRAIN:
- - scihub_train
-GLOBAL:
- HACK: 1.0
-ICDAR_DATA_DIR_TEST: ''
-ICDAR_DATA_DIR_TRAIN: ''
-INPUT:
- CROP:
- ENABLED: true
- SIZE:
- - 384
- - 600
- TYPE: absolute_range
- FORMAT: RGB
- MASK_FORMAT: polygon
- MAX_SIZE_TEST: 1333
- MAX_SIZE_TRAIN: 1333
- MIN_SIZE_TEST: 800
- MIN_SIZE_TRAIN:
- - 480
- - 512
- - 544
- - 576
- - 608
- - 640
- - 672
- - 704
- - 736
- - 768
- - 800
- MIN_SIZE_TRAIN_SAMPLING: choice
- RANDOM_FLIP: horizontal
-MODEL:
- ANCHOR_GENERATOR:
- ANGLES:
- - - -90
- - 0
- - 90
- ASPECT_RATIOS:
- - - 0.5
- - 1.0
- - 2.0
- NAME: DefaultAnchorGenerator
- OFFSET: 0.0
- SIZES:
- - - 32
- - - 64
- - - 128
- - - 256
- - - 512
- BACKBONE:
- FREEZE_AT: 2
- NAME: build_vit_fpn_backbone
- CONFIG_PATH: ''
- DEVICE: cuda
- FPN:
- FUSE_TYPE: sum
- IN_FEATURES:
- - layer3
- - layer5
- - layer7
- - layer11
- NORM: ''
- OUT_CHANNELS: 256
- IMAGE_ONLY: true
- KEYPOINT_ON: false
- LOAD_PROPOSALS: false
- MASK_ON: true
- META_ARCHITECTURE: VLGeneralizedRCNN
- PANOPTIC_FPN:
- COMBINE:
- ENABLED: true
- INSTANCES_CONFIDENCE_THRESH: 0.5
- OVERLAP_THRESH: 0.5
- STUFF_AREA_LIMIT: 4096
- INSTANCE_LOSS_WEIGHT: 1.0
- PIXEL_MEAN:
- - 127.5
- - 127.5
- - 127.5
- PIXEL_STD:
- - 127.5
- - 127.5
- - 127.5
- PROPOSAL_GENERATOR:
- MIN_SIZE: 0
- NAME: RPN
- RESNETS:
- DEFORM_MODULATED: false
- DEFORM_NUM_GROUPS: 1
- DEFORM_ON_PER_STAGE:
- - false
- - false
- - false
- - false
- DEPTH: 50
- NORM: FrozenBN
- NUM_GROUPS: 1
- OUT_FEATURES:
- - res4
- RES2_OUT_CHANNELS: 256
- RES5_DILATION: 1
- STEM_OUT_CHANNELS: 64
- STRIDE_IN_1X1: true
- WIDTH_PER_GROUP: 64
- RETINANET:
- BBOX_REG_LOSS_TYPE: smooth_l1
- BBOX_REG_WEIGHTS:
- - 1.0
- - 1.0
- - 1.0
- - 1.0
- FOCAL_LOSS_ALPHA: 0.25
- FOCAL_LOSS_GAMMA: 2.0
- IN_FEATURES:
- - p3
- - p4
- - p5
- - p6
- - p7
- IOU_LABELS:
- - 0
- - -1
- - 1
- IOU_THRESHOLDS:
- - 0.4
- - 0.5
- NMS_THRESH_TEST: 0.5
- NORM: ''
- NUM_CLASSES: 10
- NUM_CONVS: 4
- PRIOR_PROB: 0.01
- SCORE_THRESH_TEST: 0.05
- SMOOTH_L1_LOSS_BETA: 0.1
- TOPK_CANDIDATES_TEST: 1000
- ROI_BOX_CASCADE_HEAD:
- BBOX_REG_WEIGHTS:
- - - 10.0
- - 10.0
- - 5.0
- - 5.0
- - - 20.0
- - 20.0
- - 10.0
- - 10.0
- - - 30.0
- - 30.0
- - 15.0
- - 15.0
- IOUS:
- - 0.5
- - 0.6
- - 0.7
- ROI_BOX_HEAD:
- BBOX_REG_LOSS_TYPE: smooth_l1
- BBOX_REG_LOSS_WEIGHT: 1.0
- BBOX_REG_WEIGHTS:
- - 10.0
- - 10.0
- - 5.0
- - 5.0
- CLS_AGNOSTIC_BBOX_REG: true
- CONV_DIM: 256
- FC_DIM: 1024
- NAME: FastRCNNConvFCHead
- NORM: ''
- NUM_CONV: 0
- NUM_FC: 2
- POOLER_RESOLUTION: 7
- POOLER_SAMPLING_RATIO: 0
- POOLER_TYPE: ROIAlignV2
- SMOOTH_L1_BETA: 0.0
- TRAIN_ON_PRED_BOXES: false
- ROI_HEADS:
- BATCH_SIZE_PER_IMAGE: 512
- IN_FEATURES:
- - p2
- - p3
- - p4
- - p5
- IOU_LABELS:
- - 0
- - 1
- IOU_THRESHOLDS:
- - 0.5
- NAME: CascadeROIHeads
- NMS_THRESH_TEST: 0.5
- NUM_CLASSES: 10
- POSITIVE_FRACTION: 0.25
- PROPOSAL_APPEND_GT: true
- SCORE_THRESH_TEST: 0.05
- ROI_KEYPOINT_HEAD:
- CONV_DIMS:
- - 512
- - 512
- - 512
- - 512
- - 512
- - 512
- - 512
- - 512
- LOSS_WEIGHT: 1.0
- MIN_KEYPOINTS_PER_IMAGE: 1
- NAME: KRCNNConvDeconvUpsampleHead
- NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: true
- NUM_KEYPOINTS: 17
- POOLER_RESOLUTION: 14
- POOLER_SAMPLING_RATIO: 0
- POOLER_TYPE: ROIAlignV2
- ROI_MASK_HEAD:
- CLS_AGNOSTIC_MASK: false
- CONV_DIM: 256
- NAME: MaskRCNNConvUpsampleHead
- NORM: ''
- NUM_CONV: 4
- POOLER_RESOLUTION: 14
- POOLER_SAMPLING_RATIO: 0
- POOLER_TYPE: ROIAlignV2
- RPN:
- BATCH_SIZE_PER_IMAGE: 256
- BBOX_REG_LOSS_TYPE: smooth_l1
- BBOX_REG_LOSS_WEIGHT: 1.0
- BBOX_REG_WEIGHTS:
- - 1.0
- - 1.0
- - 1.0
- - 1.0
- BOUNDARY_THRESH: -1
- CONV_DIMS:
- - -1
- HEAD_NAME: StandardRPNHead
- IN_FEATURES:
- - p2
- - p3
- - p4
- - p5
- - p6
- IOU_LABELS:
- - 0
- - -1
- - 1
- IOU_THRESHOLDS:
- - 0.3
- - 0.7
- LOSS_WEIGHT: 1.0
- NMS_THRESH: 0.7
- POSITIVE_FRACTION: 0.5
- POST_NMS_TOPK_TEST: 1000
- POST_NMS_TOPK_TRAIN: 2000
- PRE_NMS_TOPK_TEST: 1000
- PRE_NMS_TOPK_TRAIN: 2000
- SMOOTH_L1_BETA: 0.0
- SEM_SEG_HEAD:
- COMMON_STRIDE: 4
- CONVS_DIM: 128
- IGNORE_VALUE: 255
- IN_FEATURES:
- - p2
- - p3
- - p4
- - p5
- LOSS_WEIGHT: 1.0
- NAME: SemSegFPNHead
- NORM: GN
- NUM_CLASSES: 10
- VIT:
- DROP_PATH: 0.1
- IMG_SIZE:
- - 224
- - 224
- NAME: layoutlmv3_base
- OUT_FEATURES:
- - layer3
- - layer5
- - layer7
- - layer11
- POS_TYPE: abs
- WEIGHTS:
-OUTPUT_DIR:
-SCIHUB_DATA_DIR_TRAIN: ~/publaynet/layout_scihub/train
-SEED: 42
-SOLVER:
- AMP:
- ENABLED: true
- BACKBONE_MULTIPLIER: 1.0
- BASE_LR: 0.0002
- BIAS_LR_FACTOR: 1.0
- CHECKPOINT_PERIOD: 2000
- CLIP_GRADIENTS:
- CLIP_TYPE: full_model
- CLIP_VALUE: 1.0
- ENABLED: true
- NORM_TYPE: 2.0
- GAMMA: 0.1
- GRADIENT_ACCUMULATION_STEPS: 1
- IMS_PER_BATCH: 32
- LR_SCHEDULER_NAME: WarmupCosineLR
- MAX_ITER: 20000
- MOMENTUM: 0.9
- NESTEROV: false
- OPTIMIZER: ADAMW
- REFERENCE_WORLD_SIZE: 0
- STEPS:
- - 10000
- WARMUP_FACTOR: 0.01
- WARMUP_ITERS: 333
- WARMUP_METHOD: linear
- WEIGHT_DECAY: 0.05
- WEIGHT_DECAY_BIAS: null
- WEIGHT_DECAY_NORM: 0.0
-TEST:
- AUG:
- ENABLED: false
- FLIP: true
- MAX_SIZE: 4000
- MIN_SIZES:
- - 400
- - 500
- - 600
- - 700
- - 800
- - 900
- - 1000
- - 1100
- - 1200
- DETECTIONS_PER_IMAGE: 100
- EVAL_PERIOD: 1000
- EXPECTED_RESULTS: []
- KEYPOINT_OKS_SIGMAS: []
- PRECISE_BN:
- ENABLED: false
- NUM_ITER: 200
-VERSION: 2
-VIS_PERIOD: 0
diff --git a/magic_pdf/resources/model_config/model_configs.yaml b/magic_pdf/resources/model_config/model_configs.yaml
deleted file mode 100644
index e56d6ee19..000000000
--- a/magic_pdf/resources/model_config/model_configs.yaml
+++ /dev/null
@@ -1,7 +0,0 @@
-weights:
- layoutlmv3: Layout/LayoutLMv3/model_final.pth
- doclayout_yolo: Layout/YOLO/doclayout_yolo_ft.pt
- yolo_v8_mfd: MFD/YOLO/yolo_v8_ft.pt
- unimernet_small: MFR/unimernet_small
- struct_eqtable: TabRec/StructEqTable
- tablemaster: TabRec/TableMaster
\ No newline at end of file
diff --git a/magic_pdf/rw/AbsReaderWriter.py b/magic_pdf/rw/AbsReaderWriter.py
deleted file mode 100644
index 78e455af3..000000000
--- a/magic_pdf/rw/AbsReaderWriter.py
+++ /dev/null
@@ -1,17 +0,0 @@
-from abc import ABC, abstractmethod
-
-
-class AbsReaderWriter(ABC):
- MODE_TXT = "text"
- MODE_BIN = "binary"
- @abstractmethod
- def read(self, path: str, mode=MODE_TXT):
- raise NotImplementedError
-
- @abstractmethod
- def write(self, content: str, path: str, mode=MODE_TXT):
- raise NotImplementedError
-
- @abstractmethod
- def read_offset(self, path: str, offset=0, limit=None) -> bytes:
- raise NotImplementedError
diff --git a/magic_pdf/rw/DiskReaderWriter.py b/magic_pdf/rw/DiskReaderWriter.py
deleted file mode 100644
index dd627fde4..000000000
--- a/magic_pdf/rw/DiskReaderWriter.py
+++ /dev/null
@@ -1,74 +0,0 @@
-import os
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from loguru import logger
-
-
-class DiskReaderWriter(AbsReaderWriter):
- def __init__(self, parent_path, encoding="utf-8"):
- self.path = parent_path
- self.encoding = encoding
-
- def read(self, path, mode=AbsReaderWriter.MODE_TXT):
- if os.path.isabs(path):
- abspath = path
- else:
- abspath = os.path.join(self.path, path)
- if not os.path.exists(abspath):
- logger.error(f"file {abspath} not exists")
- raise Exception(f"file {abspath} no exists")
- if mode == AbsReaderWriter.MODE_TXT:
- with open(abspath, "r", encoding=self.encoding) as f:
- return f.read()
- elif mode == AbsReaderWriter.MODE_BIN:
- with open(abspath, "rb") as f:
- return f.read()
- else:
- raise ValueError("Invalid mode. Use 'text' or 'binary'.")
-
- def write(self, content, path, mode=AbsReaderWriter.MODE_TXT):
- if os.path.isabs(path):
- abspath = path
- else:
- abspath = os.path.join(self.path, path)
- directory_path = os.path.dirname(abspath)
- if not os.path.exists(directory_path):
- os.makedirs(directory_path)
- if mode == AbsReaderWriter.MODE_TXT:
- with open(abspath, "w", encoding=self.encoding, errors="replace") as f:
- f.write(content)
-
- elif mode == AbsReaderWriter.MODE_BIN:
- with open(abspath, "wb") as f:
- f.write(content)
- else:
- raise ValueError("Invalid mode. Use 'text' or 'binary'.")
-
- def read_offset(self, path: str, offset=0, limit=None):
- abspath = path
- if not os.path.isabs(path):
- abspath = os.path.join(self.path, path)
- with open(abspath, "rb") as f:
- f.seek(offset)
- return f.read(limit)
-
-
-if __name__ == "__main__":
- if 0:
- file_path = "io/test/example.txt"
- drw = DiskReaderWriter("D:\projects\papayfork\Magic-PDF\magic_pdf")
-
- # 写入内容到文件
- drw.write(b"Hello, World!", path="io/test/example.txt", mode="binary")
-
- # 从文件读取内容
- content = drw.read(path=file_path)
- if content:
- logger.info(f"从 {file_path} 读取的内容: {content}")
- if 1:
- drw = DiskReaderWriter("/opt/data/pdf/resources/test/io/")
- content_bin = drw.read_offset("1.txt")
- assert content_bin == b"ABCD!"
-
- content_bin = drw.read_offset("1.txt", offset=1, limit=2)
- assert content_bin == b"BC"
-
diff --git a/magic_pdf/rw/S3ReaderWriter.py b/magic_pdf/rw/S3ReaderWriter.py
deleted file mode 100644
index ac1cc1f17..000000000
--- a/magic_pdf/rw/S3ReaderWriter.py
+++ /dev/null
@@ -1,142 +0,0 @@
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.libs.commons import parse_aws_param, parse_bucket_key, join_path
-import boto3
-from loguru import logger
-from botocore.config import Config
-
-
-class S3ReaderWriter(AbsReaderWriter):
- def __init__(
- self,
- ak: str,
- sk: str,
- endpoint_url: str,
- addressing_style: str = "auto",
- parent_path: str = "",
- ):
- self.client = self._get_client(ak, sk, endpoint_url, addressing_style)
- self.path = parent_path
-
- def _get_client(self, ak: str, sk: str, endpoint_url: str, addressing_style: str):
- s3_client = boto3.client(
- service_name="s3",
- aws_access_key_id=ak,
- aws_secret_access_key=sk,
- endpoint_url=endpoint_url,
- config=Config(
- s3={"addressing_style": addressing_style},
- retries={"max_attempts": 5, "mode": "standard"},
- ),
- )
- return s3_client
-
- def read(self, s3_relative_path, mode=AbsReaderWriter.MODE_TXT, encoding="utf-8"):
- if s3_relative_path.startswith("s3://"):
- s3_path = s3_relative_path
- else:
- s3_path = join_path(self.path, s3_relative_path)
- bucket_name, key = parse_bucket_key(s3_path)
- res = self.client.get_object(Bucket=bucket_name, Key=key)
- body = res["Body"].read()
- if mode == AbsReaderWriter.MODE_TXT:
- data = body.decode(encoding) # Decode bytes to text
- elif mode == AbsReaderWriter.MODE_BIN:
- data = body
- else:
- raise ValueError("Invalid mode. Use 'text' or 'binary'.")
- return data
-
- def write(self, content, s3_relative_path, mode=AbsReaderWriter.MODE_TXT, encoding="utf-8"):
- if s3_relative_path.startswith("s3://"):
- s3_path = s3_relative_path
- else:
- s3_path = join_path(self.path, s3_relative_path)
- if mode == AbsReaderWriter.MODE_TXT:
- body = content.encode(encoding) # Encode text data as bytes
- elif mode == AbsReaderWriter.MODE_BIN:
- body = content
- else:
- raise ValueError("Invalid mode. Use 'text' or 'binary'.")
- bucket_name, key = parse_bucket_key(s3_path)
- self.client.put_object(Body=body, Bucket=bucket_name, Key=key)
- logger.info(f"内容已写入 {s3_path} ")
-
- def read_offset(self, path: str, offset=0, limit=None) -> bytes:
- if path.startswith("s3://"):
- s3_path = path
- else:
- s3_path = join_path(self.path, path)
- bucket_name, key = parse_bucket_key(s3_path)
-
- range_header = (
- f"bytes={offset}-{offset+limit-1}" if limit else f"bytes={offset}-"
- )
- res = self.client.get_object(Bucket=bucket_name, Key=key, Range=range_header)
- return res["Body"].read()
-
-
-if __name__ == "__main__":
- if 0:
- # Config the connection info
- ak = ""
- sk = ""
- endpoint_url = ""
- addressing_style = "auto"
- bucket_name = ""
- # Create an S3ReaderWriter object
- s3_reader_writer = S3ReaderWriter(
- ak, sk, endpoint_url, addressing_style, "s3://bucket_name/"
- )
-
- # Write text data to S3
- text_data = "This is some text data"
- s3_reader_writer.write(
- text_data,
- s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json",
- mode=AbsReaderWriter.MODE_TXT,
- )
-
- # Read text data from S3
- text_data_read = s3_reader_writer.read(
- s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=AbsReaderWriter.MODE_TXT
- )
- logger.info(f"Read text data from S3: {text_data_read}")
- # Write binary data to S3
- binary_data = b"This is some binary data"
- s3_reader_writer.write(
- text_data,
- s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json",
- mode=AbsReaderWriter.MODE_BIN,
- )
-
- # Read binary data from S3
- binary_data_read = s3_reader_writer.read(
- s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=AbsReaderWriter.MODE_BIN
- )
- logger.info(f"Read binary data from S3: {binary_data_read}")
-
- # Range Read text data from S3
- binary_data_read = s3_reader_writer.read_offset(
- path=f"s3://{bucket_name}/ebook/test/test.json", offset=0, limit=10
- )
- logger.info(f"Read binary data from S3: {binary_data_read}")
- if 1:
- import os
- import json
-
- ak = os.getenv("AK", "")
- sk = os.getenv("SK", "")
- endpoint_url = os.getenv("ENDPOINT", "")
- bucket = os.getenv("S3_BUCKET", "")
- prefix = os.getenv("S3_PREFIX", "")
- key_basename = os.getenv("S3_KEY_BASENAME", "")
- s3_reader_writer = S3ReaderWriter(
- ak, sk, endpoint_url, "auto", f"s3://{bucket}/{prefix}"
- )
- content_bin = s3_reader_writer.read_offset(key_basename)
- assert content_bin[:10] == b'{"track_id'
- assert content_bin[-10:] == b'r":null}}\n'
-
- content_bin = s3_reader_writer.read_offset(key_basename, offset=424, limit=426)
- jso = json.dumps(content_bin.decode("utf-8"))
- print(jso)
diff --git a/magic_pdf/rw/__init__.py b/magic_pdf/rw/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/spark/__init__.py b/magic_pdf/spark/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/spark/spark_api.py b/magic_pdf/spark/spark_api.py
deleted file mode 100644
index bf0045ac7..000000000
--- a/magic_pdf/spark/spark_api.py
+++ /dev/null
@@ -1,51 +0,0 @@
-from loguru import logger
-
-from magic_pdf.libs.drop_reason import DropReason
-
-
-def get_data_source(jso: dict):
- data_source = jso.get("data_source")
- if data_source is None:
- data_source = jso.get("file_source")
- return data_source
-
-
-def get_data_type(jso: dict):
- data_type = jso.get("data_type")
- if data_type is None:
- data_type = jso.get("file_type")
- return data_type
-
-
-def get_bookid(jso: dict):
- book_id = jso.get("bookid")
- if book_id is None:
- book_id = jso.get("original_file_id")
- return book_id
-
-
-def exception_handler(jso: dict, e):
- logger.exception(e)
- jso["_need_drop"] = True
- jso["_drop_reason"] = DropReason.Exception
- jso["_exception"] = f"ERROR: {e}"
- return jso
-
-
-def get_bookname(jso: dict):
- data_source = get_data_source(jso)
- file_id = jso.get("file_id")
- book_name = f"{data_source}/{file_id}"
- return book_name
-
-
-def spark_json_extractor(jso: dict) -> dict:
-
- """
- 从json中提取数据，返回一个dict
- """
-
- return {
- "_pdf_type": jso["_pdf_type"],
- "model_list": jso["doc_layout_result"],
- }
diff --git a/magic_pdf/tools/__init__.py b/magic_pdf/tools/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/tools/cli.py b/magic_pdf/tools/cli.py
deleted file mode 100644
index c19176fa3..000000000
--- a/magic_pdf/tools/cli.py
+++ /dev/null
@@ -1,119 +0,0 @@
-import os
-from pathlib import Path
-
-import click
-from loguru import logger
-
-import magic_pdf.model as model_config
-from magic_pdf.libs.version import __version__
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-from magic_pdf.tools.common import do_parse, parse_pdf_methods
-
-
-@click.command()
-@click.version_option(__version__,
- '--version',
- '-v',
- help='display the version and exit')
-@click.option(
- '-p',
- '--path',
- 'path',
- type=click.Path(exists=True),
- required=True,
- help='local pdf filepath or directory',
-)
-@click.option(
- '-o',
- '--output-dir',
- 'output_dir',
- type=click.Path(),
- required=True,
- help='output local directory',
-)
-@click.option(
- '-m',
- '--method',
- 'method',
- type=parse_pdf_methods,
- help="""the method for parsing pdf.
-ocr: using ocr technique to extract information from pdf.
-txt: suitable for the text-based pdf only and outperform ocr.
-auto: automatically choose the best method for parsing pdf from ocr and txt.
-without method specified, auto will be used by default.""",
- default='auto',
-)
-@click.option(
- '-l',
- '--lang',
- 'lang',
- type=str,
- help="""
- Input the languages in the pdf (if known) to improve OCR accuracy. Optional.
- You should input "Abbreviation" with language form url:
- https://paddlepaddle.github.io/PaddleOCR/latest/en/ppocr/blog/multi_languages.html#5-support-languages-and-abbreviations
- """,
- default=None,
-)
-@click.option(
- '-d',
- '--debug',
- 'debug_able',
- type=bool,
- help='Enables detailed debugging information during the execution of the CLI commands.',
- default=False,
-)
-@click.option(
- '-s',
- '--start',
- 'start_page_id',
- type=int,
- help='The starting page for PDF parsing, beginning from 0.',
- default=0,
-)
-@click.option(
- '-e',
- '--end',
- 'end_page_id',
- type=int,
- help='The ending page for PDF parsing, beginning from 0.',
- default=None,
-)
-def cli(path, output_dir, method, lang, debug_able, start_page_id, end_page_id):
- model_config.__use_inside_model__ = True
- model_config.__model_mode__ = 'full'
- os.makedirs(output_dir, exist_ok=True)
-
- def read_fn(path):
- disk_rw = DiskReaderWriter(os.path.dirname(path))
- return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN)
-
- def parse_doc(doc_path: str):
- try:
- file_name = str(Path(doc_path).stem)
- pdf_data = read_fn(doc_path)
- do_parse(
- output_dir,
- file_name,
- pdf_data,
- [],
- method,
- debug_able,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- lang=lang
- )
-
- except Exception as e:
- logger.exception(e)
-
- if os.path.isdir(path):
- for doc_path in Path(path).glob('*.pdf'):
- parse_doc(doc_path)
- else:
- parse_doc(path)
-
-
-if __name__ == '__main__':
- cli()
diff --git a/magic_pdf/tools/cli_dev.py b/magic_pdf/tools/cli_dev.py
deleted file mode 100644
index c037f262f..000000000
--- a/magic_pdf/tools/cli_dev.py
+++ /dev/null
@@ -1,152 +0,0 @@
-import json as json_parse
-import os
-from pathlib import Path
-
-import click
-
-import magic_pdf.model as model_config
-from magic_pdf.libs.config_reader import get_s3_config
-from magic_pdf.libs.path_utils import (parse_s3_range_params, parse_s3path,
- remove_non_official_s3_args)
-from magic_pdf.libs.version import __version__
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-from magic_pdf.rw.S3ReaderWriter import S3ReaderWriter
-from magic_pdf.tools.common import do_parse, parse_pdf_methods
-
-
-def read_s3_path(s3path):
- bucket, key = parse_s3path(s3path)
-
- s3_ak, s3_sk, s3_endpoint = get_s3_config(bucket)
- s3_rw = S3ReaderWriter(s3_ak, s3_sk, s3_endpoint, 'auto',
- remove_non_official_s3_args(s3path))
- may_range_params = parse_s3_range_params(s3path)
- if may_range_params is None or 2 != len(may_range_params):
- byte_start, byte_end = 0, None
- else:
- byte_start, byte_end = int(may_range_params[0]), int(
- may_range_params[1])
- return s3_rw.read_offset(
- remove_non_official_s3_args(s3path),
- byte_start,
- byte_end,
- )
-
-
-@click.group()
-@click.version_option(__version__, '--version', '-v', help='显示版本信息')
-def cli():
- pass
-
-
-@cli.command()
-@click.option(
- '-j',
- '--jsonl',
- 'jsonl',
- type=str,
- help='输入 jsonl 路径，本地或者 s3 上的文件',
- required=True,
-)
-@click.option(
- '-m',
- '--method',
- 'method',
- type=parse_pdf_methods,
- help='指定解析方法。txt: 文本型 pdf 解析方法， ocr: 光学识别解析 pdf, auto: 程序智能选择解析方法',
- default='auto',
-)
-@click.option(
- '-o',
- '--output-dir',
- 'output_dir',
- type=click.Path(),
- required=True,
- help='输出到本地目录',
-)
-def jsonl(jsonl, method, output_dir):
- model_config.__use_inside_model__ = False
- if jsonl.startswith('s3://'):
- jso = json_parse.loads(read_s3_path(jsonl).decode('utf-8'))
- else:
- with open(jsonl) as f:
- jso = json_parse.loads(f.readline())
- os.makedirs(output_dir, exist_ok=True)
- s3_file_path = jso.get('file_location')
- if s3_file_path is None:
- s3_file_path = jso.get('path')
- pdf_file_name = Path(s3_file_path).stem
- pdf_data = read_s3_path(s3_file_path)
-
- print(pdf_file_name, jso, method)
- do_parse(
- output_dir,
- pdf_file_name,
- pdf_data,
- jso['doc_layout_result'],
- method,
- False,
- f_dump_content_list=True,
- f_draw_model_bbox=True,
- )
-
-
-@cli.command()
-@click.option(
- '-p',
- '--pdf',
- 'pdf',
- type=click.Path(exists=True),
- required=True,
- help='本地 PDF 文件',
-)
-@click.option(
- '-j',
- '--json',
- 'json_data',
- type=click.Path(exists=True),
- required=True,
- help='本地模型推理出的 json 数据',
-)
-@click.option('-o',
- '--output-dir',
- 'output_dir',
- type=click.Path(),
- required=True,
- help='本地输出目录')
-@click.option(
- '-m',
- '--method',
- 'method',
- type=parse_pdf_methods,
- help='指定解析方法。txt: 文本型 pdf 解析方法， ocr: 光学识别解析 pdf, auto: 程序智能选择解析方法',
- default='auto',
-)
-def pdf(pdf, json_data, output_dir, method):
- model_config.__use_inside_model__ = False
- full_pdf_path = os.path.realpath(pdf)
- os.makedirs(output_dir, exist_ok=True)
-
- def read_fn(path):
- disk_rw = DiskReaderWriter(os.path.dirname(path))
- return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN)
-
- model_json_list = json_parse.loads(read_fn(json_data).decode('utf-8'))
-
- file_name = str(Path(full_pdf_path).stem)
- pdf_data = read_fn(full_pdf_path)
- do_parse(
- output_dir,
- file_name,
- pdf_data,
- model_json_list,
- method,
- False,
- f_dump_content_list=True,
- f_draw_model_bbox=True,
- )
-
-
-if __name__ == '__main__':
- cli()
diff --git a/magic_pdf/tools/common.py b/magic_pdf/tools/common.py
deleted file mode 100644
index c98da4c37..000000000
--- a/magic_pdf/tools/common.py
+++ /dev/null
@@ -1,152 +0,0 @@
-import copy
-import json as json_parse
-import os
-
-import click
-from loguru import logger
-
-import magic_pdf.model as model_config
-from magic_pdf.libs.draw_bbox import (draw_layout_bbox, draw_line_sort_bbox,
- draw_model_bbox, draw_span_bbox)
-from magic_pdf.libs.MakeContentConfig import DropMode, MakeMode
-from magic_pdf.pipe.OCRPipe import OCRPipe
-from magic_pdf.pipe.TXTPipe import TXTPipe
-from magic_pdf.pipe.UNIPipe import UNIPipe
-from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
-from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
-
-
-def prepare_env(output_dir, pdf_file_name, method):
- local_parent_dir = os.path.join(output_dir, pdf_file_name, method)
-
- local_image_dir = os.path.join(str(local_parent_dir), 'images')
- local_md_dir = local_parent_dir
- os.makedirs(local_image_dir, exist_ok=True)
- os.makedirs(local_md_dir, exist_ok=True)
- return local_image_dir, local_md_dir
-
-
-def do_parse(
- output_dir,
- pdf_file_name,
- pdf_bytes,
- model_list,
- parse_method,
- debug_able,
- f_draw_span_bbox=True,
- f_draw_layout_bbox=True,
- f_dump_md=True,
- f_dump_middle_json=True,
- f_dump_model_json=True,
- f_dump_orig_pdf=True,
- f_dump_content_list=True,
- f_make_md_mode=MakeMode.MM_MD,
- f_draw_model_bbox=False,
- f_draw_line_sort_bbox=False,
- start_page_id=0,
- end_page_id=None,
- lang=None,
- layout_model=None,
- formula_enable=None,
- table_enable=None,
-):
- if debug_able:
- logger.warning('debug mode is on')
- f_draw_model_bbox = True
- f_draw_line_sort_bbox = True
-
- orig_model_list = copy.deepcopy(model_list)
- local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name,
- parse_method)
-
- image_writer, md_writer = DiskReaderWriter(
- local_image_dir), DiskReaderWriter(local_md_dir)
- image_dir = str(os.path.basename(local_image_dir))
-
- if parse_method == 'auto':
- jso_useful_key = {'_pdf_type': '', 'model_list': model_list}
- pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer, is_debug=True,
- start_page_id=start_page_id, end_page_id=end_page_id, lang=lang,
- layout_model=layout_model, formula_enable=formula_enable, table_enable=table_enable)
- elif parse_method == 'txt':
- pipe = TXTPipe(pdf_bytes, model_list, image_writer, is_debug=True,
- start_page_id=start_page_id, end_page_id=end_page_id, lang=lang,
- layout_model=layout_model, formula_enable=formula_enable, table_enable=table_enable)
- elif parse_method == 'ocr':
- pipe = OCRPipe(pdf_bytes, model_list, image_writer, is_debug=True,
- start_page_id=start_page_id, end_page_id=end_page_id, lang=lang,
- layout_model=layout_model, formula_enable=formula_enable, table_enable=table_enable)
- else:
- logger.error('unknown parse method')
- exit(1)
-
- pipe.pipe_classify()
-
- if len(model_list) == 0:
- if model_config.__use_inside_model__:
- pipe.pipe_analyze()
- orig_model_list = copy.deepcopy(pipe.model_list)
- else:
- logger.error('need model list input')
- exit(2)
-
- pipe.pipe_parse()
- pdf_info = pipe.pdf_mid_data['pdf_info']
- if f_draw_layout_bbox:
- draw_layout_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name)
- if f_draw_span_bbox:
- draw_span_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name)
- if f_draw_model_bbox:
- draw_model_bbox(copy.deepcopy(orig_model_list), pdf_bytes, local_md_dir, pdf_file_name)
- if f_draw_line_sort_bbox:
- draw_line_sort_bbox(pdf_info, pdf_bytes, local_md_dir, pdf_file_name)
-
- md_content = pipe.pipe_mk_markdown(image_dir,
- drop_mode=DropMode.NONE,
- md_make_mode=f_make_md_mode)
- if f_dump_md:
- md_writer.write(
- content=md_content,
- path=f'{pdf_file_name}.md',
- mode=AbsReaderWriter.MODE_TXT,
- )
-
- if f_dump_middle_json:
- md_writer.write(
- content=json_parse.dumps(pipe.pdf_mid_data,
- ensure_ascii=False,
- indent=4),
- path=f'{pdf_file_name}_middle.json',
- mode=AbsReaderWriter.MODE_TXT,
- )
-
- if f_dump_model_json:
- md_writer.write(
- content=json_parse.dumps(orig_model_list,
- ensure_ascii=False,
- indent=4),
- path=f'{pdf_file_name}_model.json',
- mode=AbsReaderWriter.MODE_TXT,
- )
-
- if f_dump_orig_pdf:
- md_writer.write(
- content=pdf_bytes,
- path=f'{pdf_file_name}_origin.pdf',
- mode=AbsReaderWriter.MODE_BIN,
- )
-
- content_list = pipe.pipe_mk_uni_format(image_dir, drop_mode=DropMode.NONE)
- if f_dump_content_list:
- md_writer.write(
- content=json_parse.dumps(content_list,
- ensure_ascii=False,
- indent=4),
- path=f'{pdf_file_name}_content_list.json',
- mode=AbsReaderWriter.MODE_TXT,
- )
-
- logger.info(f'local output dir is {local_md_dir}')
-
-
-parse_pdf_methods = click.Choice(['ocr', 'txt', 'auto'])
diff --git a/magic_pdf/user_api.py b/magic_pdf/user_api.py
deleted file mode 100644
index 2a4bd59e5..000000000
--- a/magic_pdf/user_api.py
+++ /dev/null
@@ -1,130 +0,0 @@
-"""
-用户输入：
- model数组，每个元素代表一个页面
- pdf在s3的路径
- 截图保存的s3位置
-
-然后：
- 1）根据s3路径，调用spark集群的api,拿到ak,sk,endpoint，构造出s3PDFReader
- 2）根据用户输入的s3地址，调用spark集群的api,拿到ak,sk,endpoint，构造出s3ImageWriter
-
-其余部分至于构造s3cli, 获取ak,sk都在code-clean里写代码完成。不要反向依赖！！！
-
-"""
-import re
-
-from loguru import logger
-
-from magic_pdf.libs.version import __version__
-from magic_pdf.model.doc_analyze_by_custom_model import doc_analyze
-from magic_pdf.rw import AbsReaderWriter
-from magic_pdf.pdf_parse_by_ocr import parse_pdf_by_ocr
-from magic_pdf.pdf_parse_by_txt import parse_pdf_by_txt
-
-PARSE_TYPE_TXT = "txt"
-PARSE_TYPE_OCR = "ocr"
-
-
-def parse_txt_pdf(pdf_bytes: bytes, pdf_models: list, imageWriter: AbsReaderWriter, is_debug=False,
- start_page_id=0, end_page_id=None, lang=None,
- *args, **kwargs):
- """
- 解析文本类pdf
- """
- pdf_info_dict = parse_pdf_by_txt(
- pdf_bytes,
- pdf_models,
- imageWriter,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- debug_mode=is_debug,
- )
-
- pdf_info_dict["_parse_type"] = PARSE_TYPE_TXT
-
- pdf_info_dict["_version_name"] = __version__
-
- if lang is not None:
- pdf_info_dict["_lang"] = lang
-
- return pdf_info_dict
-
-
-def parse_ocr_pdf(pdf_bytes: bytes, pdf_models: list, imageWriter: AbsReaderWriter, is_debug=False,
- start_page_id=0, end_page_id=None, lang=None,
- *args, **kwargs):
- """
- 解析ocr类pdf
- """
- pdf_info_dict = parse_pdf_by_ocr(
- pdf_bytes,
- pdf_models,
- imageWriter,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- debug_mode=is_debug,
- )
-
- pdf_info_dict["_parse_type"] = PARSE_TYPE_OCR
-
- pdf_info_dict["_version_name"] = __version__
-
- if lang is not None:
- pdf_info_dict["_lang"] = lang
-
- return pdf_info_dict
-
-
-def parse_union_pdf(pdf_bytes: bytes, pdf_models: list, imageWriter: AbsReaderWriter, is_debug=False,
- input_model_is_empty: bool = False,
- start_page_id=0, end_page_id=None, lang=None,
- *args, **kwargs):
- """
- ocr和文本混合的pdf，全部解析出来
- """
-
- def parse_pdf(method):
- try:
- return method(
- pdf_bytes,
- pdf_models,
- imageWriter,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- debug_mode=is_debug,
- )
- except Exception as e:
- logger.exception(e)
- return None
-
- pdf_info_dict = parse_pdf(parse_pdf_by_txt)
- if pdf_info_dict is None or pdf_info_dict.get("_need_drop", False):
- logger.warning(f"parse_pdf_by_txt drop or error, switch to parse_pdf_by_ocr")
- if input_model_is_empty:
- layout_model = kwargs.get("layout_model", None)
- formula_enable = kwargs.get("formula_enable", None)
- table_enable = kwargs.get("table_enable", None)
- pdf_models = doc_analyze(
- pdf_bytes,
- ocr=True,
- start_page_id=start_page_id,
- end_page_id=end_page_id,
- lang=lang,
- layout_model=layout_model,
- formula_enable=formula_enable,
- table_enable=table_enable,
- )
- pdf_info_dict = parse_pdf(parse_pdf_by_ocr)
- if pdf_info_dict is None:
- raise Exception("Both parse_pdf_by_txt and parse_pdf_by_ocr failed.")
- else:
- pdf_info_dict["_parse_type"] = PARSE_TYPE_OCR
- else:
- pdf_info_dict["_parse_type"] = PARSE_TYPE_TXT
-
- pdf_info_dict["_version_name"] = __version__
-
- if lang is not None:
- pdf_info_dict["_lang"] = lang
-
- return pdf_info_dict
diff --git a/magic_pdf/utils/__init__.py b/magic_pdf/utils/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/magic_pdf/utils/annotations.py b/magic_pdf/utils/annotations.py
deleted file mode 100644
index 898d88033..000000000
--- a/magic_pdf/utils/annotations.py
+++ /dev/null
@@ -1,11 +0,0 @@
-
-from loguru import logger
-
-
-def ImportPIL(f):
- try:
- import PIL # noqa: F401
- except ImportError:
- logger.error('Pillow not installed, please install by pip.')
- exit(1)
- return f
diff --git a/mineru.template.json b/mineru.template.json
new file mode 100644
index 000000000..7d979ec8c
--- /dev/null
+++ b/mineru.template.json
@@ -0,0 +1,30 @@
+{
+ "bucket_info":{
+ "bucket-name-1":["ak", "sk", "endpoint"],
+ "bucket-name-2":["ak", "sk", "endpoint"]
+ },
+ "latex-delimiter-config": {
+ "display": {
+ "left": "$$",
+ "right": "$$"
+ },
+ "inline": {
+ "left": "$",
+ "right": "$"
+ }
+ },
+ "llm-aided-config": {
+ "title_aided": {
+ "api_key": "your_api_key",
+ "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
+ "model": "qwen3.5-plus",
+ "enable_thinking": false,
+ "enable": false
+ }
+ },
+ "models-dir": {
+ "pipeline": "",
+ "vlm": ""
+ },
+ "config_version": "1.3.1"
+}
\ No newline at end of file
diff --git a/mineru/__init__.py b/mineru/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/__init__.py b/mineru/backend/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/hybrid/__init__.py b/mineru/backend/hybrid/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/hybrid/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/hybrid/hybrid_analyze.py b/mineru/backend/hybrid/hybrid_analyze.py
new file mode 100644
index 000000000..cfa76f3d7
--- /dev/null
+++ b/mineru/backend/hybrid/hybrid_analyze.py
@@ -0,0 +1,793 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import time
+from collections import defaultdict
+
+import cv2
+import numpy as np
+import pypdfium2 as pdfium
+from loguru import logger
+from mineru_vl_utils import MinerUClient
+from mineru_vl_utils.structs import BlockType
+from tqdm import tqdm
+
+from mineru.backend.hybrid.hybrid_model_output_to_middle_json import (
+ append_page_model_list_to_middle_json,
+ finalize_middle_json,
+ init_middle_json,
+)
+from mineru.backend.utils.runtime_utils import exclude_progress_bar_idle_time
+from mineru.backend.pipeline.model_init import HybridModelSingleton
+from mineru.backend.vlm.vlm_analyze import (
+ ModelSingleton,
+ aio_predictor_execution_guard,
+ predictor_execution_guard,
+ _maybe_enable_serial_execution,
+)
+from mineru.data.data_reader_writer import DataWriter
+from mineru.utils.config_reader import get_device, get_processing_window_size
+from mineru.utils.enum_class import ImageType, NotExtractType
+from mineru.utils.model_utils import crop_img, get_vram, clean_memory
+from mineru.utils.ocr_utils import get_adjusted_mfdetrec_res, get_ocr_result_list, sorted_boxes, merge_det_boxes, \
+ update_det_boxes, OcrConfidence
+from mineru.utils.pdf_classify import classify
+from mineru.utils.pdf_image_tools import load_images_from_pdf_doc
+from mineru.utils.pdfium_guard import (
+ close_pdfium_document,
+ get_pdfium_document_page_count,
+ open_pdfium_document,
+)
+
+os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1' # 让mps可以fallback
+os.environ['NO_ALBUMENTATIONS_UPDATE'] = '1' # 禁止albumentations检查更新
+
+LAYOUT_BASE_BATCH_SIZE = 1
+MFR_BASE_BATCH_SIZE = 16
+OCR_DET_BASE_BATCH_SIZE = 8
+
+not_extract_list = [item.value for item in NotExtractType]
+
+def ocr_classify(pdf_bytes, parse_method: str = 'auto',) -> bool:
+ # 确定OCR设置
+ _ocr_enable = False
+ if parse_method == 'auto':
+ if classify(pdf_bytes) == 'ocr':
+ _ocr_enable = True
+ elif parse_method == 'ocr':
+ _ocr_enable = True
+ return _ocr_enable
+
+def ocr_det(
+ hybrid_pipeline_model,
+ np_images,
+ model_list,
+ mfd_res,
+ _ocr_enable,
+ batch_ratio: int = 1,
+):
+ def _set_temp_pixel_bbox(res, pixel_bbox):
+ res["_normalized_bbox"] = list(res["bbox"])
+ res["bbox"] = pixel_bbox
+
+ def _restore_normalized_bbox(res):
+ normalized_bbox = res.pop("_normalized_bbox", None)
+ if normalized_bbox is not None:
+ res["bbox"] = normalized_bbox
+
+ ocr_res_list = []
+ if not hybrid_pipeline_model.enable_ocr_det_batch:
+ # 非批处理模式 - 逐页处理
+ for np_image, page_mfd_res, page_results in tqdm(
+ zip(np_images, mfd_res, model_list),
+ total=len(np_images),
+ desc="OCR-det"
+ ):
+ ocr_res_list.append([])
+ img_height, img_width = np_image.shape[:2]
+ for res in page_results:
+ if res['type'] not in not_extract_list:
+ continue
+ x0 = max(0, int(res['bbox'][0] * img_width))
+ y0 = max(0, int(res['bbox'][1] * img_height))
+ x1 = min(img_width, int(res['bbox'][2] * img_width))
+ y1 = min(img_height, int(res['bbox'][3] * img_height))
+ if x1 <= x0 or y1 <= y0:
+ continue
+ _set_temp_pixel_bbox(res, [x0, y0, x1, y1])
+ try:
+ new_image, useful_list = crop_img(
+ res, np_image, crop_paste_x=50, crop_paste_y=50
+ )
+ finally:
+ _restore_normalized_bbox(res)
+ adjusted_mfdetrec_res = get_adjusted_mfdetrec_res(
+ page_mfd_res, useful_list
+ )
+ bgr_image = cv2.cvtColor(new_image, cv2.COLOR_RGB2BGR)
+ ocr_res = hybrid_pipeline_model.ocr_model.ocr(
+ bgr_image, mfd_res=adjusted_mfdetrec_res, rec=False
+ )[0]
+ if ocr_res:
+ ocr_result_list = get_ocr_result_list(
+ ocr_res, useful_list, _ocr_enable, bgr_image, hybrid_pipeline_model.lang
+ )
+
+ ocr_res_list[-1].extend(ocr_result_list)
+ else:
+ # 批处理模式 - 按语言和分辨率分组
+ # 收集所有需要OCR检测的裁剪图像
+ all_cropped_images_info = []
+
+ for np_image, page_mfd_res, page_results in zip(
+ np_images, mfd_res, model_list
+ ):
+ ocr_res_list.append([])
+ img_height, img_width = np_image.shape[:2]
+ for res in page_results:
+ if res['type'] not in not_extract_list:
+ continue
+ x0 = max(0, int(res['bbox'][0] * img_width))
+ y0 = max(0, int(res['bbox'][1] * img_height))
+ x1 = min(img_width, int(res['bbox'][2] * img_width))
+ y1 = min(img_height, int(res['bbox'][3] * img_height))
+ if x1 <= x0 or y1 <= y0:
+ continue
+ _set_temp_pixel_bbox(res, [x0, y0, x1, y1])
+ try:
+ new_image, useful_list = crop_img(
+ res, np_image, crop_paste_x=50, crop_paste_y=50
+ )
+ finally:
+ _restore_normalized_bbox(res)
+ adjusted_mfdetrec_res = get_adjusted_mfdetrec_res(
+ page_mfd_res, useful_list
+ )
+ bgr_image = cv2.cvtColor(new_image, cv2.COLOR_RGB2BGR)
+ all_cropped_images_info.append((
+ bgr_image, useful_list, adjusted_mfdetrec_res, ocr_res_list[-1]
+ ))
+
+ # 按分辨率分组并同时完成padding
+ RESOLUTION_GROUP_STRIDE = 64 # 32
+
+ resolution_groups = defaultdict(list)
+ for crop_info in all_cropped_images_info:
+ cropped_img = crop_info[0]
+ h, w = cropped_img.shape[:2]
+ # 直接计算目标尺寸并用作分组键
+ target_h = ((h + RESOLUTION_GROUP_STRIDE - 1) // RESOLUTION_GROUP_STRIDE) * RESOLUTION_GROUP_STRIDE
+ target_w = ((w + RESOLUTION_GROUP_STRIDE - 1) // RESOLUTION_GROUP_STRIDE) * RESOLUTION_GROUP_STRIDE
+ group_key = (target_h, target_w)
+ resolution_groups[group_key].append(crop_info)
+
+ # 对每个分辨率组进行批处理
+ for (target_h, target_w), group_crops in tqdm(resolution_groups.items(), desc=f"OCR-det"):
+ # 对所有图像进行padding到统一尺寸
+ batch_images = []
+ for crop_info in group_crops:
+ img = crop_info[0]
+ h, w = img.shape[:2]
+ # 创建目标尺寸的白色背景
+ padded_img = np.ones((target_h, target_w, 3), dtype=np.uint8) * 255
+ padded_img[:h, :w] = img
+ batch_images.append(padded_img)
+
+ # 批处理检测
+ det_batch_size = min(len(batch_images), batch_ratio * OCR_DET_BASE_BATCH_SIZE)
+ batch_results = hybrid_pipeline_model.ocr_model.text_detector.batch_predict(batch_images, det_batch_size)
+
+ # 处理批处理结果
+ for crop_info, (dt_boxes, _) in zip(group_crops, batch_results):
+ bgr_image, useful_list, adjusted_mfdetrec_res, ocr_page_res_list = crop_info
+
+ if dt_boxes is not None and len(dt_boxes) > 0:
+ # 处理检测框
+ dt_boxes_sorted = sorted_boxes(dt_boxes)
+ dt_boxes_merged = merge_det_boxes(dt_boxes_sorted) if dt_boxes_sorted else []
+
+ # 根据公式位置更新检测框
+ dt_boxes_final = (update_det_boxes(dt_boxes_merged, adjusted_mfdetrec_res)
+ if dt_boxes_merged and adjusted_mfdetrec_res
+ else dt_boxes_merged)
+
+ if dt_boxes_final:
+ ocr_res = [box.tolist() if hasattr(box, 'tolist') else box for box in dt_boxes_final]
+ ocr_result_list = get_ocr_result_list(
+ ocr_res, useful_list, _ocr_enable, bgr_image, hybrid_pipeline_model.lang
+ )
+ ocr_page_res_list.extend(ocr_result_list)
+ return ocr_res_list
+
+def mask_image_regions(np_images, model_list):
+ # 根据vlm返回的结果，在每一页中将image、table、equation块mask成白色背景图像
+ for np_image, vlm_page_results in zip(np_images, model_list):
+ img_height, img_width = np_image.shape[:2]
+ # 收集需要mask的区域
+ mask_regions = []
+ for block in vlm_page_results:
+ if block['type'] in [BlockType.IMAGE, BlockType.TABLE, BlockType.EQUATION]:
+ bbox = block['bbox']
+ # 批量转换归一化坐标到像素坐标,并进行边界检查
+ x0 = max(0, int(bbox[0] * img_width))
+ y0 = max(0, int(bbox[1] * img_height))
+ x1 = min(img_width, int(bbox[2] * img_width))
+ y1 = min(img_height, int(bbox[3] * img_height))
+ # 只添加有效区域
+ if x1 > x0 and y1 > y0:
+ mask_regions.append((y0, y1, x0, x1))
+ # 批量应用mask
+ for y0, y1, x0, x1 in mask_regions:
+ np_image[y0:y1, x0:x1, :] = 255
+ return np_images
+
+
+def normalize_bbox_to_unit(item, page_width, page_height):
+ """将像素级bbox归一化为[0, 1]区间"""
+ bbox = item.get('bbox')
+ if bbox is None or len(bbox) != 4:
+ return False
+
+ x0, y0, x1, y1 = [float(v) for v in bbox]
+ if (
+ 0.0 <= x0 <= 1.0
+ and 0.0 <= y0 <= 1.0
+ and 0.0 <= x1 <= 1.0
+ and 0.0 <= y1 <= 1.0
+ ):
+ normalized_bbox = [x0, y0, x1, y1]
+ else:
+ normalized_bbox = [
+ x0 / page_width,
+ y0 / page_height,
+ x1 / page_width,
+ y1 / page_height,
+ ]
+ item['bbox'] = [round(min(max(v, 0), 1), 3) for v in normalized_bbox]
+ return True
+
+
+def _formula_item_to_pixel_bbox(item):
+ bbox = item.get('bbox')
+ if bbox is not None and len(bbox) == 4:
+ return [int(float(v)) for v in bbox]
+
+ return None
+
+
+def _build_inline_formula_inputs(images_layout_res):
+ inline_formula_inputs = []
+ for layout_res in images_layout_res:
+ page_inline_formula_inputs = []
+ for res in layout_res:
+ if res.get('label') not in ['inline_formula', 'display_formula']:
+ continue
+ bbox = res.get('bbox')
+ if bbox is None or len(bbox) != 4:
+ continue
+ page_inline_formula_inputs.append(
+ {
+ "label": "inline_formula",
+ "bbox": list(bbox),
+ "score": float(res.get('score', 0.0)),
+ "latex": res.get('latex', ''),
+ }
+ )
+ inline_formula_inputs.append(page_inline_formula_inputs)
+ return inline_formula_inputs
+
+
+def _process_ocr_and_formulas(
+ images_pil_list,
+ model_list,
+ language,
+ inline_formula_enable,
+ _ocr_enable,
+ batch_ratio: int = 1,
+):
+ """处理OCR和公式识别"""
+
+ # 遍历model_list,对文本块截图交由OCR识别
+ # 根据_ocr_enable决定ocr只开det还是det+rec
+ # 根据inline_formula_enable决定是使用mfd和ocr结合的方式,还是纯ocr方式
+
+ # 将PIL图片转换为numpy数组
+ np_images = [np.asarray(pil_image).copy() for pil_image in images_pil_list]
+
+ # 获取混合模型实例
+ hybrid_model_singleton = HybridModelSingleton()
+ hybrid_pipeline_model = hybrid_model_singleton.get_model(
+ lang=language,
+ formula_enable=inline_formula_enable,
+ )
+
+ if inline_formula_enable:
+ # 在进行`行内`公式检测和识别前，先将图像中的图片、表格、`行间`公式区域mask掉
+ np_images = mask_image_regions(np_images, model_list)
+ # 使用layout模型提供行内公式检测框
+ images_layout_res = hybrid_pipeline_model.layout_model.batch_predict(
+ np_images,
+ batch_size=min(8, batch_ratio * LAYOUT_BASE_BATCH_SIZE),
+ )
+ images_mfd_res = _build_inline_formula_inputs(images_layout_res)
+ # 公式识别
+ inline_formula_list = hybrid_pipeline_model.mfr_model.batch_predict(
+ images_mfd_res,
+ np_images,
+ batch_size=batch_ratio * MFR_BASE_BATCH_SIZE,
+ interline_enable=True,
+ )
+ else:
+ inline_formula_list = [[] for _ in range(len(images_pil_list))]
+
+ mfd_res = []
+ for page_inline_formula_list in inline_formula_list:
+ page_mfd_res = []
+ for formula in page_inline_formula_list:
+ bbox = _formula_item_to_pixel_bbox(formula)
+ if bbox is None:
+ continue
+ page_mfd_res.append({"bbox": bbox})
+ mfd_res.append(page_mfd_res)
+
+ # vlm没有执行ocr，需要ocr_det
+ ocr_res_list = ocr_det(
+ hybrid_pipeline_model,
+ np_images,
+ model_list,
+ mfd_res,
+ _ocr_enable,
+ batch_ratio=batch_ratio,
+ )
+
+ # 如果需要ocr则做ocr_rec
+ if _ocr_enable:
+ need_ocr_list = []
+ img_crop_list = []
+ for page_ocr_res_list in ocr_res_list:
+ for ocr_res in page_ocr_res_list:
+ if 'np_img' in ocr_res:
+ need_ocr_list.append((page_ocr_res_list, ocr_res))
+ img_crop_list.append(ocr_res.pop('np_img'))
+ if len(img_crop_list) > 0:
+ # Process OCR
+ ocr_result_list = hybrid_pipeline_model.ocr_model.ocr(img_crop_list, det=False, tqdm_enable=True)[0]
+
+ # Verify we have matching counts
+ assert len(ocr_result_list) == len(need_ocr_list), f'ocr_result_list: {len(ocr_result_list)}, need_ocr_list: {len(need_ocr_list)}'
+
+ items_to_remove = []
+ # Process OCR results for this language
+ for index, (page_ocr_res_list, need_ocr_res) in enumerate(need_ocr_list):
+ ocr_text, ocr_score = ocr_result_list[index]
+ need_ocr_res['text'] = ocr_text
+ need_ocr_res['score'] = float(f"{ocr_score:.3f}")
+ should_remove = False
+ if ocr_score < OcrConfidence.min_confidence:
+ should_remove = True
+ else:
+ layout_res_bbox = need_ocr_res.get("bbox")
+ if layout_res_bbox is None and need_ocr_res.get("poly") is not None:
+ layout_res_bbox = [
+ need_ocr_res['poly'][0],
+ need_ocr_res['poly'][1],
+ need_ocr_res['poly'][4],
+ need_ocr_res['poly'][5],
+ ]
+ if layout_res_bbox is None:
+ should_remove = True
+ continue
+ layout_res_width = layout_res_bbox[2] - layout_res_bbox[0]
+ layout_res_height = layout_res_bbox[3] - layout_res_bbox[1]
+ if (
+ ocr_text in [
+ '（204号', '（20', '（2', '（2号', '（20号', '号','（204',
+ '(cid:)', '(ci:)', '(cd:1)', 'cd:)', 'c)', '(cd:)', 'c', 'id:)',
+ ':)', '√:)', '√i:)', '−i:)', '−:' , 'i:)',
+ ]
+ and ocr_score < 0.8
+ and layout_res_width < layout_res_height
+ ):
+ should_remove = True
+
+ if should_remove:
+ items_to_remove.append((page_ocr_res_list, need_ocr_res))
+
+ for page_ocr_res_list, need_ocr_res in items_to_remove:
+ if need_ocr_res in page_ocr_res_list:
+ page_ocr_res_list.remove(need_ocr_res)
+
+ _normalize_bbox(inline_formula_list, ocr_res_list, images_pil_list)
+ merged_model_list = _merge_page_sidecar_items(
+ model_list,
+ inline_formula_list,
+ ocr_res_list,
+ )
+ return merged_model_list, hybrid_pipeline_model
+
+
+def _normalize_bbox(
+ inline_formula_list,
+ ocr_res_list,
+ images_pil_list,
+):
+ """归一化坐标并生成最终结果"""
+ for page_inline_formula_list, page_ocr_res_list, page_pil_image in zip(
+ inline_formula_list, ocr_res_list, images_pil_list
+ ):
+ if page_inline_formula_list or page_ocr_res_list:
+ page_width, page_height = page_pil_image.size
+ # 处理公式列表
+ for formula in page_inline_formula_list:
+ normalize_bbox_to_unit(formula, page_width, page_height)
+ # 处理OCR结果列表
+ for ocr_res in page_ocr_res_list:
+ normalize_bbox_to_unit(ocr_res, page_width, page_height)
+
+
+def _build_inline_formula_model_item(formula):
+ return {
+ "type": "inline_formula",
+ "bbox": list(formula["bbox"]),
+ "latex": formula.get("latex", ""),
+ "score": float(formula.get("score", 0.0)),
+ }
+
+
+def _build_ocr_text_model_item(ocr_res):
+ return {
+ "type": "ocr_text",
+ "bbox": list(ocr_res["bbox"]),
+ "text": ocr_res.get("text", ""),
+ "score": float(ocr_res.get("score", 0.0)),
+ }
+
+
+def _merge_page_sidecar_items(
+ model_list,
+ inline_formula_list,
+ ocr_res_list,
+):
+ merged_model_list = []
+ for page_model_list, page_inline_formula_list, page_ocr_res_list in zip(
+ model_list, inline_formula_list, ocr_res_list
+ ):
+ merged_page_model_list = list(page_model_list)
+ merged_page_model_list.extend(
+ _build_inline_formula_model_item(formula)
+ for formula in page_inline_formula_list
+ if formula.get("bbox") is not None
+ )
+ merged_page_model_list.extend(
+ _build_ocr_text_model_item(ocr_res)
+ for ocr_res in page_ocr_res_list
+ if ocr_res.get("bbox") is not None
+ )
+ merged_model_list.append(merged_page_model_list)
+ return merged_model_list
+
+
+def get_batch_ratio(device):
+ """
+ 根据显存大小或环境变量获取 batch ratio
+ """
+ # 1. 优先尝试从环境变量获取
+ """
+ c/s架构分离部署时，建议通过设置环境变量 MINERU_HYBRID_BATCH_RATIO 来指定 batch ratio
+ 建议的设置值如如下，以下配置值已考虑一定的冗余，单卡多终端部署时为了保证稳定性，可以额外保留一个client端的显存作为整体冗余
+ 单个client端显存大小 | MINERU_HYBRID_BATCH_RATIO
+ ------------------|------------------------
+ <= 6 GB | 8
+ <= 4 GB | 4
+ <= 3 GB | 2
+ <= 2 GB | 1
+ 例如：
+ export MINERU_HYBRID_BATCH_RATIO=4
+ """
+ env_val = os.getenv("MINERU_HYBRID_BATCH_RATIO")
+ if env_val:
+ try:
+ batch_ratio = int(env_val)
+ logger.info(f"hybrid batch ratio (from env): {batch_ratio}")
+ return batch_ratio
+ except ValueError as e:
+ logger.warning(f"Invalid MINERU_HYBRID_BATCH_RATIO value: {env_val}, switching to auto mode. Error: {e}")
+
+ # 2. 根据显存自动推断
+ """
+ 根据总显存大小粗略估计 batch ratio，需要排除掉vllm等推理框架占用的显存开销
+ """
+ gpu_memory = get_vram(device)
+ if gpu_memory >= 32:
+ batch_ratio = 16
+ elif gpu_memory >= 16:
+ batch_ratio = 8
+ elif gpu_memory >= 12:
+ batch_ratio = 4
+ elif gpu_memory >= 8:
+ batch_ratio = 2
+ else:
+ batch_ratio = 1
+
+ logger.info(f"hybrid batch ratio (auto, vram={gpu_memory}GB): {batch_ratio}")
+ return batch_ratio
+
+
+def _should_enable_vlm_ocr(ocr_enable: bool, language: str, inline_formula_enable: bool) -> bool:
+ """判断是否启用VLM OCR"""
+ force_enable = os.getenv("MINERU_FORCE_VLM_OCR_ENABLE", "0").lower() in ("1", "true", "yes")
+ if force_enable:
+ return True
+
+ force_pipeline = os.getenv("MINERU_HYBRID_FORCE_PIPELINE_ENABLE", "0").lower() in ("1", "true", "yes")
+ return (
+ ocr_enable
+ and language in ["ch", "en"]
+ and inline_formula_enable
+ and not force_pipeline
+ )
+
+
+def _close_images(images_list):
+ for image_dict in images_list or []:
+ pil_img = image_dict.get("img_pil")
+ if pil_img is not None:
+ try:
+ pil_img.close()
+ except Exception:
+ pass
+
+
+def doc_analyze(
+ pdf_bytes,
+ image_writer: DataWriter | None,
+ predictor: MinerUClient | None = None,
+ backend="transformers",
+ parse_method: str = 'auto',
+ language: str = 'ch',
+ inline_formula_enable: bool = True,
+ model_path: str | None = None,
+ server_url: str | None = None,
+ **kwargs,
+):
+ if predictor is None:
+ predictor = ModelSingleton().get_model(backend, model_path, server_url, **kwargs)
+ predictor = _maybe_enable_serial_execution(predictor, backend)
+
+ device = get_device()
+ _ocr_enable = ocr_classify(pdf_bytes, parse_method=parse_method)
+ _vlm_ocr_enable = _should_enable_vlm_ocr(_ocr_enable, language, inline_formula_enable)
+
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ middle_json = init_middle_json(_ocr_enable, _vlm_ocr_enable)
+ model_list = []
+ doc_closed = False
+ hybrid_pipeline_model = None
+ try:
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ configured_window_size = get_processing_window_size(default=64)
+ effective_window_size = min(page_count, configured_window_size) if page_count else 0
+ total_windows = (
+ (page_count + effective_window_size - 1) // effective_window_size
+ if effective_window_size
+ else 0
+ )
+ logger.info(
+ f'Hybrid processing-window run. page_count={page_count}, '
+ f'window_size={configured_window_size}, total_windows={total_windows}'
+ )
+
+ batch_ratio = get_batch_ratio(device) if not _vlm_ocr_enable else 1
+
+ infer_start = time.time()
+ progress_bar = None
+ last_append_end_time = None
+ try:
+ for window_index, window_start in enumerate(range(0, page_count, effective_window_size or 1)):
+ window_end = min(page_count - 1, window_start + effective_window_size - 1)
+ images_list = load_images_from_pdf_doc(
+ pdf_doc,
+ start_page_id=window_start,
+ end_page_id=window_end,
+ image_type=ImageType.PIL,
+ pdf_bytes=pdf_bytes,
+ )
+ try:
+ images_pil_list = [image_dict["img_pil"] for image_dict in images_list]
+ logger.info(
+ f'Hybrid processing window {window_index + 1}/{total_windows}: '
+ f'pages {window_start + 1}-{window_end + 1}/{page_count} '
+ f'({len(images_pil_list)} pages)'
+ )
+ if _vlm_ocr_enable:
+ with predictor_execution_guard(predictor):
+ window_model_list = predictor.batch_two_step_extract(images=images_pil_list)
+ else:
+ with predictor_execution_guard(predictor):
+ window_model_list = predictor.batch_two_step_extract(
+ images=images_pil_list,
+ not_extract_list=not_extract_list
+ )
+ window_model_list, hybrid_pipeline_model = _process_ocr_and_formulas(
+ images_pil_list,
+ window_model_list,
+ language,
+ inline_formula_enable,
+ _ocr_enable,
+ batch_ratio=batch_ratio,
+ )
+
+ model_list.extend(window_model_list)
+ if progress_bar is None:
+ progress_bar = tqdm(total=page_count, desc="Processing pages")
+ else:
+ exclude_progress_bar_idle_time(
+ progress_bar,
+ last_append_end_time,
+ now=time.time(),
+ )
+ append_page_model_list_to_middle_json(
+ middle_json,
+ window_model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=window_start,
+ _ocr_enable=_ocr_enable,
+ _vlm_ocr_enable=_vlm_ocr_enable,
+ progress_bar=progress_bar,
+ )
+ last_append_end_time = time.time()
+ finally:
+ _close_images(images_list)
+ finally:
+ if progress_bar is not None:
+ progress_bar.close()
+
+ infer_time = round(time.time() - infer_start, 2)
+ if infer_time > 0 and page_count > 0:
+ logger.debug(
+ f"processing-window infer finished, cost: {infer_time}, "
+ f"speed: {round(len(model_list) / infer_time, 3)} page/s"
+ )
+
+ finalize_middle_json(
+ middle_json["pdf_info"],
+ hybrid_pipeline_model,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ )
+ close_pdfium_document(pdf_doc)
+ doc_closed = True
+ clean_memory(device)
+ return middle_json, model_list, _vlm_ocr_enable
+ finally:
+ if not doc_closed:
+ close_pdfium_document(pdf_doc)
+
+
+async def aio_doc_analyze(
+ pdf_bytes,
+ image_writer: DataWriter | None,
+ predictor: MinerUClient | None = None,
+ backend="transformers",
+ parse_method: str = 'auto',
+ language: str = 'ch',
+ inline_formula_enable: bool = True,
+ model_path: str | None = None,
+ server_url: str | None = None,
+ **kwargs,
+):
+ if predictor is None:
+ predictor = ModelSingleton().get_model(backend, model_path, server_url, **kwargs)
+ predictor = _maybe_enable_serial_execution(predictor, backend)
+
+ device = get_device()
+ _ocr_enable = ocr_classify(pdf_bytes, parse_method=parse_method)
+ _vlm_ocr_enable = _should_enable_vlm_ocr(_ocr_enable, language, inline_formula_enable)
+
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ middle_json = init_middle_json(_ocr_enable, _vlm_ocr_enable)
+ model_list = []
+ doc_closed = False
+ hybrid_pipeline_model = None
+ try:
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ configured_window_size = get_processing_window_size(default=64)
+ effective_window_size = min(page_count, configured_window_size) if page_count else 0
+ total_windows = (
+ (page_count + effective_window_size - 1) // effective_window_size
+ if effective_window_size
+ else 0
+ )
+ logger.info(
+ f'Hybrid processing-window run. page_count={page_count}, '
+ f'window_size={configured_window_size}, total_windows={total_windows}'
+ )
+
+ batch_ratio = get_batch_ratio(device) if not _vlm_ocr_enable else 1
+
+ infer_start = time.time()
+ progress_bar = None
+ last_append_end_time = None
+ try:
+ for window_index, window_start in enumerate(range(0, page_count, effective_window_size or 1)):
+ window_end = min(page_count - 1, window_start + effective_window_size - 1)
+ images_list = load_images_from_pdf_doc(
+ pdf_doc,
+ start_page_id=window_start,
+ end_page_id=window_end,
+ image_type=ImageType.PIL,
+ pdf_bytes=pdf_bytes,
+ )
+ try:
+ images_pil_list = [image_dict["img_pil"] for image_dict in images_list]
+ logger.info(
+ f'Hybrid processing window {window_index + 1}/{total_windows}: '
+ f'pages {window_start + 1}-{window_end + 1}/{page_count} '
+ f'({len(images_pil_list)} pages)'
+ )
+ if _vlm_ocr_enable:
+ async with aio_predictor_execution_guard(predictor):
+ window_model_list = await predictor.aio_batch_two_step_extract(images=images_pil_list)
+ else:
+ async with aio_predictor_execution_guard(predictor):
+ window_model_list = await predictor.aio_batch_two_step_extract(
+ images=images_pil_list,
+ not_extract_list=not_extract_list
+ )
+ window_model_list, hybrid_pipeline_model = _process_ocr_and_formulas(
+ images_pil_list,
+ window_model_list,
+ language,
+ inline_formula_enable,
+ _ocr_enable,
+ batch_ratio=batch_ratio,
+ )
+
+ model_list.extend(window_model_list)
+ if progress_bar is None:
+ progress_bar = tqdm(total=page_count, desc="Processing pages")
+ else:
+ exclude_progress_bar_idle_time(
+ progress_bar,
+ last_append_end_time,
+ now=time.time(),
+ )
+ append_page_model_list_to_middle_json(
+ middle_json,
+ window_model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=window_start,
+ _ocr_enable=_ocr_enable,
+ _vlm_ocr_enable=_vlm_ocr_enable,
+ progress_bar=progress_bar,
+ )
+ last_append_end_time = time.time()
+ finally:
+ _close_images(images_list)
+ finally:
+ if progress_bar is not None:
+ progress_bar.close()
+
+ infer_time = round(time.time() - infer_start, 2)
+ if infer_time > 0 and page_count > 0:
+ logger.debug(
+ f"processing-window infer finished, cost: {infer_time}, "
+ f"speed: {round(len(model_list) / infer_time, 3)} page/s"
+ )
+
+ finalize_middle_json(
+ middle_json["pdf_info"],
+ hybrid_pipeline_model,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ )
+ close_pdfium_document(pdf_doc)
+ doc_closed = True
+ clean_memory(device)
+ return middle_json, model_list, _vlm_ocr_enable
+ finally:
+ if not doc_closed:
+ close_pdfium_document(pdf_doc)
diff --git a/mineru/backend/hybrid/hybrid_magic_model.py b/mineru/backend/hybrid/hybrid_magic_model.py
new file mode 100644
index 000000000..a88563b47
--- /dev/null
+++ b/mineru/backend/hybrid/hybrid_magic_model.py
@@ -0,0 +1,487 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import copy
+import re
+
+from loguru import logger
+
+from mineru.utils.boxbase import calculate_overlap_area_in_bbox1_area_ratio
+from mineru.utils.enum_class import ContentType, BlockType, NotExtractType
+from mineru.utils.guess_suffix_or_lang import guess_language_by_text
+from mineru.utils.span_block_fix import fix_text_block
+from mineru.utils.span_pre_proc import txt_spans_extract
+from mineru.utils.visual_magic_model_utils import (
+ GENERIC_CHILD_TYPES,
+ IMAGE_BLOCK_BODY,
+ VISUAL_MAIN_TYPES,
+ clean_content,
+ code_content_clean,
+ isolated_formula_clean,
+ regroup_visual_blocks,
+)
+not_extract_list = [item.value for item in NotExtractType] + [
+ BlockType.CAPTION,
+ BlockType.FOOTNOTE,
+]
+
+
+def _copy_raw_text_block_metadata(raw_block_type, block_info, block):
+ if raw_block_type != BlockType.TEXT:
+ return
+ if "merge_prev" in block_info:
+ block["merge_prev"] = block_info["merge_prev"]
+
+
+class MagicModel:
+ def __init__(
+ self,
+ page_model_list: list,
+ page,
+ scale,
+ page_pil_img,
+ width,
+ height,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ ):
+ (
+ self.page_blocks,
+ self.page_inline_formula,
+ self.page_ocr_res,
+ ) = self._split_page_model_list(copy.deepcopy(page_model_list))
+
+ self.width = width
+ self.height = height
+
+ blocks = []
+ self.all_spans = []
+
+ page_text_inline_formula_spans = []
+ if not _vlm_ocr_enable:
+ for inline_formula in self.page_inline_formula:
+ inline_formula["bbox"] = self.cal_real_bbox(inline_formula["bbox"])
+ inline_formula_latex = inline_formula.pop("latex", "")
+ if inline_formula_latex:
+ page_text_inline_formula_spans.append(
+ {
+ "bbox": inline_formula["bbox"],
+ "type": ContentType.INLINE_EQUATION,
+ "content": inline_formula_latex,
+ "score": inline_formula["score"],
+ }
+ )
+ for ocr_res in self.page_ocr_res:
+ ocr_res["bbox"] = self.cal_real_bbox(ocr_res["bbox"])
+ page_text_inline_formula_spans.append(
+ {
+ "bbox": ocr_res["bbox"],
+ "type": ContentType.TEXT,
+ "content": ocr_res["text"],
+ "score": ocr_res["score"],
+ }
+ )
+ if not _ocr_enable:
+ virtual_block = [0, 0, width, height, None, None, None, "text"]
+ page_text_inline_formula_spans = txt_spans_extract(
+ page,
+ page_text_inline_formula_spans,
+ page_pil_img,
+ scale,
+ [virtual_block],
+ [],
+ )
+
+ # 解析每个块
+ for index, block_info in enumerate(self.page_blocks):
+ try:
+ block_bbox = self.cal_real_bbox(block_info["bbox"])
+ block_type = block_info["type"]
+ raw_block_type = block_type
+ block_content = block_info.get("content")
+ block_angle = block_info.get("angle", 0)
+ block_sub_type = (
+ block_info.get("sub_type")
+ if raw_block_type in ["image", "chart"]
+ else None
+ )
+ except Exception as e:
+ # 如果解析失败，可能是因为格式不正确，跳过这个块
+ logger.warning(f"Invalid block format: {block_info}, error: {e}")
+ continue
+
+ span_type = "unknown"
+ code_block_sub_type = None
+ guess_lang = None
+
+ if block_type in [
+ "text",
+ "title",
+ "ref_text",
+ "phonetic",
+ "header",
+ "footer",
+ "page_number",
+ "aside_text",
+ "page_footnote",
+ "list",
+ ]:
+ span_type = ContentType.TEXT
+ elif block_type in ["image_caption", "table_caption", "code_caption"]:
+ block_type = BlockType.CAPTION
+ span_type = ContentType.TEXT
+ elif block_type in ["image_footnote", "table_footnote"]:
+ block_type = BlockType.FOOTNOTE
+ span_type = ContentType.TEXT
+ elif block_type == "image":
+ block_type = BlockType.IMAGE_BODY
+ span_type = ContentType.IMAGE
+ elif block_type == "image_block":
+ block_type = IMAGE_BLOCK_BODY
+ span_type = ContentType.IMAGE
+ elif block_type == "table":
+ block_type = BlockType.TABLE_BODY
+ span_type = ContentType.TABLE
+ elif block_type == "chart":
+ block_type = BlockType.CHART_BODY
+ span_type = ContentType.CHART
+ elif block_type in ["code", "algorithm"]:
+ block_content = code_content_clean(block_content)
+ code_block_sub_type = block_type
+ block_type = BlockType.CODE_BODY
+ span_type = ContentType.TEXT
+ guess_lang = guess_language_by_text(block_content)
+ elif block_type == "equation":
+ block_type = BlockType.INTERLINE_EQUATION
+ span_type = ContentType.INTERLINE_EQUATION
+
+ # code 和 algorithm 类型的块，如果内容中包含行内公式，则需要将块类型切换为 algorithm
+ switch_code_to_algorithm = False
+
+ span = None
+ if span_type in [ContentType.IMAGE, ContentType.TABLE, ContentType.CHART]:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ }
+ if span_type == ContentType.TABLE:
+ span["html"] = block_content
+ elif raw_block_type in ["image", "chart"] and block_content is not None:
+ span["content"] = block_content
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ "content": isolated_formula_clean(block_content),
+ }
+ elif _vlm_ocr_enable or block_type not in not_extract_list:
+ # vlm_ocr_enable 模式下，所有文本块都直接使用 block 的内容
+ # 非 vlm_ocr_enable 模式下，非提取块仍沿用直接内容模式
+ if block_content:
+ block_content = clean_content(block_content)
+
+ if block_type == "title" and block_content:
+ block_content = re.sub(r"\n\s*", " ", block_content).strip()
+
+ if (
+ block_content
+ and block_content.count("\\(") == block_content.count("\\)")
+ and block_content.count("\\(") > 0
+ ):
+ switch_code_to_algorithm = True
+
+ spans = []
+ last_end = 0
+ for match in re.finditer(r"\\\((.+?)\\\)", block_content):
+ start, end = match.span()
+
+ if start > last_end:
+ text_before = block_content[last_end:start]
+ if text_before.strip():
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.TEXT,
+ "content": text_before,
+ }
+ )
+
+ formula = match.group(1)
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.INLINE_EQUATION,
+ "content": formula.strip(),
+ }
+ )
+
+ last_end = end
+
+ if last_end < len(block_content):
+ text_after = block_content[last_end:]
+ if text_after.strip():
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.TEXT,
+ "content": text_after,
+ }
+ )
+
+ span = spans
+ else:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ "content": block_content,
+ }
+
+ if (
+ span_type
+ in [
+ ContentType.IMAGE,
+ ContentType.TABLE,
+ ContentType.CHART,
+ ContentType.INTERLINE_EQUATION,
+ ]
+ or (_vlm_ocr_enable or block_type not in not_extract_list)
+ ):
+ if span is None:
+ continue
+ if isinstance(span, dict) and "bbox" in span:
+ self.all_spans.append(span)
+ spans = [span]
+ elif isinstance(span, list):
+ self.all_spans.extend(span)
+ spans = span
+ else:
+ raise ValueError(
+ f"Invalid span type: {span_type}, expected dict or list, got {type(span)}"
+ )
+
+ if block_type == BlockType.CODE_BODY:
+ if switch_code_to_algorithm and code_block_sub_type == "code":
+ code_block_sub_type = "algorithm"
+ line = {
+ "bbox": block_bbox,
+ "spans": spans,
+ "extra": {
+ "type": code_block_sub_type,
+ "guess_lang": guess_lang,
+ },
+ }
+ else:
+ line = {"bbox": block_bbox, "spans": spans}
+
+ block = {
+ "bbox": block_bbox,
+ "type": block_type,
+ "angle": block_angle,
+ "lines": [line],
+ "index": index,
+ }
+ if block_sub_type:
+ block["sub_type"] = block_sub_type
+ if raw_block_type == "table" and "cell_merge" in block_info:
+ block["cell_merge"] = block_info["cell_merge"]
+ _copy_raw_text_block_metadata(raw_block_type, block_info, block)
+ else:
+ block_spans = []
+ for span in page_text_inline_formula_spans:
+ if (
+ calculate_overlap_area_in_bbox1_area_ratio(
+ span["bbox"],
+ block_bbox,
+ )
+ > 0.5
+ ):
+ block_spans.append(span)
+
+ if block_spans:
+ for span in block_spans:
+ page_text_inline_formula_spans.remove(span)
+
+ block = {
+ "bbox": block_bbox,
+ "type": block_type,
+ "angle": block_angle,
+ "spans": block_spans,
+ "index": index,
+ }
+ block = fix_text_block(block)
+ _copy_raw_text_block_metadata(raw_block_type, block_info, block)
+
+ blocks.append(block)
+
+ self.image_blocks = []
+ self.table_blocks = []
+ self.chart_blocks = []
+ self.interline_equation_blocks = []
+ self.text_blocks = []
+ self.title_blocks = []
+ self.code_blocks = []
+ self.discarded_blocks = []
+ self.ref_text_blocks = []
+ self.phonetic_blocks = []
+ self.list_blocks = []
+
+ for block in blocks:
+ if block["type"] in VISUAL_MAIN_TYPES or block["type"] in GENERIC_CHILD_TYPES:
+ continue
+ elif block["type"] == BlockType.INTERLINE_EQUATION:
+ self.interline_equation_blocks.append(block)
+ elif block["type"] == BlockType.TEXT:
+ self.text_blocks.append(block)
+ elif block["type"] == BlockType.TITLE:
+ self.title_blocks.append(block)
+ elif block["type"] == BlockType.REF_TEXT:
+ self.ref_text_blocks.append(block)
+ elif block["type"] == BlockType.PHONETIC:
+ self.phonetic_blocks.append(block)
+ elif block["type"] in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_NUMBER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ self.discarded_blocks.append(block)
+ elif block["type"] == BlockType.LIST:
+ self.list_blocks.append(block)
+
+ self.list_blocks, self.text_blocks, self.ref_text_blocks = fix_list_blocks(
+ self.list_blocks,
+ self.text_blocks,
+ self.ref_text_blocks,
+ )
+
+ visual_groups, unmatched_child_blocks = regroup_visual_blocks(blocks)
+ self.image_blocks = visual_groups[BlockType.IMAGE]
+ self.table_blocks = visual_groups[BlockType.TABLE]
+ self.chart_blocks = visual_groups[BlockType.CHART]
+ self.code_blocks = visual_groups[BlockType.CODE]
+
+ for code_block in self.code_blocks:
+ for block in code_block["blocks"]:
+ if block["type"] == BlockType.CODE_BODY:
+ if block["lines"]:
+ line = block["lines"][0]
+ code_block["sub_type"] = line["extra"]["type"]
+ if code_block["sub_type"] == "code":
+ code_block["guess_lang"] = line["extra"]["guess_lang"]
+ del line["extra"]
+ else:
+ code_block["sub_type"] = "code"
+ code_block["guess_lang"] = "txt"
+
+ for block in unmatched_child_blocks:
+ block["type"] = BlockType.TEXT
+ self.text_blocks.append(block)
+
+ @staticmethod
+ def _split_page_model_list(page_model_list):
+ page_blocks = []
+ page_inline_formula = []
+ page_ocr_res = []
+
+ for item in page_model_list:
+ item_type = item.get("type") or item.get("label")
+ if item_type == "inline_formula":
+ page_inline_formula.append(item)
+ elif item_type == "ocr_text":
+ page_ocr_res.append(item)
+ else:
+ page_blocks.append(item)
+
+ return page_blocks, page_inline_formula, page_ocr_res
+
+ def cal_real_bbox(self, bbox):
+ x1, y1, x2, y2 = bbox
+ x_1, y_1, x_2, y_2 = (
+ int(x1 * self.width),
+ int(y1 * self.height),
+ int(x2 * self.width),
+ int(y2 * self.height),
+ )
+ if x_2 < x_1:
+ x_1, x_2 = x_2, x_1
+ if y_2 < y_1:
+ y_1, y_2 = y_2, y_1
+ return (x_1, y_1, x_2, y_2)
+
+ def get_list_blocks(self):
+ return self.list_blocks
+
+ def get_image_blocks(self):
+ return self.image_blocks
+
+ def get_table_blocks(self):
+ return self.table_blocks
+
+ def get_chart_blocks(self):
+ return self.chart_blocks
+
+ def get_code_blocks(self):
+ return self.code_blocks
+
+ def get_ref_text_blocks(self):
+ return self.ref_text_blocks
+
+ def get_phonetic_blocks(self):
+ return self.phonetic_blocks
+
+ def get_title_blocks(self):
+ return self.title_blocks
+
+ def get_text_blocks(self):
+ return self.text_blocks
+
+ def get_interline_equation_blocks(self):
+ return self.interline_equation_blocks
+
+ def get_discarded_blocks(self):
+ return self.discarded_blocks
+
+ def get_all_spans(self):
+ return self.all_spans
+def fix_list_blocks(list_blocks, text_blocks, ref_text_blocks):
+ for list_block in list_blocks:
+ list_block["blocks"] = []
+ if "lines" in list_block:
+ del list_block["lines"]
+
+ temp_text_blocks = text_blocks + ref_text_blocks
+ need_remove_blocks = []
+ for block in temp_text_blocks:
+ for list_block in list_blocks:
+ if (
+ calculate_overlap_area_in_bbox1_area_ratio(
+ block["bbox"],
+ list_block["bbox"],
+ )
+ >= 0.8
+ ):
+ list_block["blocks"].append(block)
+ need_remove_blocks.append(block)
+ break
+
+ for block in need_remove_blocks:
+ if block in text_blocks:
+ text_blocks.remove(block)
+ elif block in ref_text_blocks:
+ ref_text_blocks.remove(block)
+
+ list_blocks = [lb for lb in list_blocks if lb["blocks"]]
+
+ for list_block in list_blocks:
+ type_count = {}
+ for sub_block in list_block["blocks"]:
+ sub_block_type = sub_block["type"]
+ if sub_block_type not in type_count:
+ type_count[sub_block_type] = 0
+ type_count[sub_block_type] += 1
+
+ if type_count:
+ list_block["sub_type"] = max(type_count, key=type_count.get)
+ else:
+ list_block["sub_type"] = "unknown"
+
+ return list_blocks, text_blocks, ref_text_blocks
diff --git a/mineru/backend/hybrid/hybrid_model_output_to_middle_json.py b/mineru/backend/hybrid/hybrid_model_output_to_middle_json.py
new file mode 100644
index 000000000..a197976b2
--- /dev/null
+++ b/mineru/backend/hybrid/hybrid_model_output_to_middle_json.py
@@ -0,0 +1,344 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+import os
+import time
+
+import numpy as np
+from loguru import logger
+from tqdm import tqdm
+
+from mineru.backend.utils.html_image_utils import replace_inline_table_images
+from mineru.backend.utils.ocr_det_utils import (
+ detect_ocr_boxes_from_padded_crop,
+ get_ch_lite_ocr_det_model,
+)
+from mineru.backend.utils.para_block_utils import (
+ annotate_hybrid_cross_page_merge_prev,
+ build_para_blocks_from_preproc,
+ cleanup_internal_para_block_metadata,
+ edge_text_line_hints_key,
+ iter_block_spans,
+ merge_para_text_blocks,
+)
+from mineru.backend.hybrid.hybrid_magic_model import MagicModel
+from mineru.backend.utils.runtime_utils import cross_page_table_merge
+from mineru.utils.config_reader import get_table_enable, get_llm_aided_config
+from mineru.utils.cut_image import cut_image_and_table
+from mineru.utils.enum_class import ContentType
+from mineru.utils.hash_utils import bytes_md5
+from mineru.utils.ocr_utils import OcrConfidence, rotate_vertical_crop_if_needed
+from mineru.utils.pdfium_guard import close_pdfium_document, pdfium_guard
+from mineru.version import __version__
+
+from mineru.utils.llm_aided import llm_aided_title
+title_aided_enable = False
+llm_aided_config = get_llm_aided_config()
+if llm_aided_config:
+ title_aided_config = llm_aided_config.get('title_aided', {})
+ title_aided_enable = title_aided_config.get('enable', False)
+
+
+def blocks_to_page_info(
+ page_model_list,
+ image_dict,
+ page,
+ image_writer,
+ page_index,
+ _ocr_enable,
+ _vlm_ocr_enable,
+) -> dict:
+ """将blocks转换为页面信息"""
+
+ scale = image_dict["scale"]
+ page_pil_img = image_dict["img_pil"]
+ page_img_md5 = bytes_md5(page_pil_img.tobytes())
+ with pdfium_guard():
+ width, height = map(int, page.get_size())
+
+ magic_model = MagicModel(
+ page_model_list,
+ page,
+ scale,
+ page_pil_img,
+ width,
+ height,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ )
+ image_blocks = magic_model.get_image_blocks()
+ table_blocks = magic_model.get_table_blocks()
+ chart_blocks = magic_model.get_chart_blocks()
+ title_blocks = magic_model.get_title_blocks()
+ discarded_blocks = magic_model.get_discarded_blocks()
+ code_blocks = magic_model.get_code_blocks()
+ ref_text_blocks = magic_model.get_ref_text_blocks()
+ phonetic_blocks = magic_model.get_phonetic_blocks()
+ list_blocks = magic_model.get_list_blocks()
+
+ # 如果有标题优化需求，计算标题的平均行高
+ if title_aided_enable:
+ if _vlm_ocr_enable: # vlm_ocr导致没有line信息，需要重新det获取平均行高
+ ocr_model = get_ch_lite_ocr_det_model()
+ for title_block in title_blocks:
+ ocr_det_res, _ = detect_ocr_boxes_from_padded_crop(
+ title_block.get('bbox'),
+ page_pil_img,
+ scale,
+ ocr_model=ocr_model,
+ )
+ if len(ocr_det_res) > 0:
+ # 计算所有res的平均高度
+ avg_height = np.mean([box[2][1] - box[0][1] for box in ocr_det_res])
+ title_block['line_avg_height'] = round(avg_height/scale)
+ else: # 有line信息，直接计算平均行高
+ for title_block in title_blocks:
+ lines = title_block.get('lines', [])
+ if lines:
+ # 使用列表推导式和内置函数,一次性计算平均高度
+ avg_height = sum(line['bbox'][3] - line['bbox'][1] for line in lines) / len(lines)
+ title_block['line_avg_height'] = round(avg_height)
+ else:
+ title_block['line_avg_height'] = title_block['bbox'][3] - title_block['bbox'][1]
+
+ text_blocks = magic_model.get_text_blocks()
+ interline_equation_blocks = magic_model.get_interline_equation_blocks()
+
+ all_spans = magic_model.get_all_spans()
+ # 对image/table/chart/interline_equation的span截图
+ for span in all_spans:
+ if span["type"] in [ContentType.IMAGE, ContentType.TABLE, ContentType.CHART, ContentType.INTERLINE_EQUATION]:
+ span = cut_image_and_table(span, page_pil_img, page_img_md5, page_index, image_writer, scale=scale)
+
+ replace_inline_table_images(table_blocks, image_writer, page_index)
+
+ page_blocks = []
+ page_blocks.extend([
+ *image_blocks,
+ *table_blocks,
+ *chart_blocks,
+ *code_blocks,
+ *ref_text_blocks,
+ *phonetic_blocks,
+ *title_blocks,
+ *text_blocks,
+ *interline_equation_blocks,
+ *list_blocks,
+ ])
+ # 对page_blocks根据index的值进行排序
+ page_blocks.sort(key=lambda x: x["index"])
+
+ page_info = {
+ "preproc_blocks": page_blocks,
+ "discarded_blocks": discarded_blocks,
+ "page_size": [width, height],
+ "page_idx": page_index,
+ }
+ if _vlm_ocr_enable:
+ edge_text_line_hints = _detect_edge_text_line_hints(page_blocks, page_pil_img, scale)
+ if edge_text_line_hints:
+ page_info[edge_text_line_hints_key()] = edge_text_line_hints
+ return page_info
+
+
+def _apply_post_ocr(pdf_info_list, hybrid_pipeline_model):
+ need_ocr_list = []
+ img_crop_list = []
+ for page_info in pdf_info_list:
+ for block in page_info.get('preproc_blocks', []):
+ for span in iter_block_spans(block):
+ if 'np_img' in span:
+ need_ocr_list.append(span)
+ img_crop_list.append(rotate_vertical_crop_if_needed(span['np_img']))
+ span.pop('np_img')
+ for block in page_info.get('discarded_blocks', []):
+ for span in iter_block_spans(block):
+ if 'np_img' in span:
+ need_ocr_list.append(span)
+ img_crop_list.append(rotate_vertical_crop_if_needed(span['np_img']))
+ span.pop('np_img')
+ if len(img_crop_list) > 0:
+ ocr_res_list = hybrid_pipeline_model.ocr_model.ocr(img_crop_list, det=False, tqdm_enable=True)[0]
+ assert len(ocr_res_list) == len(
+ need_ocr_list), f'ocr_res_list: {len(ocr_res_list)}, need_ocr_list: {len(need_ocr_list)}'
+ for index, span in enumerate(need_ocr_list):
+ ocr_text, ocr_score = ocr_res_list[index]
+ if ocr_score > OcrConfidence.min_confidence:
+ span['content'] = ocr_text
+ span['score'] = float(f"{ocr_score:.3f}")
+ else:
+ span['content'] = ''
+ span['score'] = 0.0
+
+
+def init_middle_json(_ocr_enable, _vlm_ocr_enable):
+ return {
+ "pdf_info": [],
+ "_backend": "hybrid",
+ "_ocr_enable": _ocr_enable,
+ "_vlm_ocr_enable": _vlm_ocr_enable,
+ "_version_name": __version__
+ }
+
+
+def append_page_results_to_middle_json(
+ middle_json,
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=0,
+ _ocr_enable=False,
+ _vlm_ocr_enable=False,
+ progress_bar=None,
+):
+ for offset, (page_model_list, image_dict) in enumerate(
+ zip(model_list, images_list)
+ ):
+ page_index = page_start_index + offset
+ with pdfium_guard():
+ page = pdf_doc[page_index]
+ page_info = blocks_to_page_info(
+ page_model_list,
+ image_dict,
+ page,
+ image_writer,
+ page_index,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ )
+ middle_json["pdf_info"].append(page_info)
+ if progress_bar is not None:
+ progress_bar.update(1)
+
+
+def append_page_model_list_to_middle_json(
+ middle_json,
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=0,
+ _ocr_enable=False,
+ _vlm_ocr_enable=False,
+ progress_bar=None,
+):
+ append_page_results_to_middle_json(
+ middle_json,
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=page_start_index,
+ _ocr_enable=_ocr_enable,
+ _vlm_ocr_enable=_vlm_ocr_enable,
+ progress_bar=progress_bar,
+ )
+
+
+def finalize_middle_json(pdf_info_list, hybrid_pipeline_model, _ocr_enable, _vlm_ocr_enable):
+ if not (_vlm_ocr_enable or _ocr_enable):
+ _apply_post_ocr(pdf_info_list, hybrid_pipeline_model)
+
+ build_para_blocks_from_preproc(pdf_info_list)
+ annotate_hybrid_cross_page_merge_prev(
+ pdf_info_list,
+ prefer_edge_line_hints=_vlm_ocr_enable,
+ )
+ merge_para_text_blocks(pdf_info_list, allow_cross_page=True)
+
+ table_enable = get_table_enable(os.getenv('MINERU_VLM_TABLE_ENABLE', 'True').lower() == 'true')
+ if table_enable:
+ cross_page_table_merge(pdf_info_list)
+
+ if title_aided_enable:
+ llm_aided_title_start_time = time.time()
+ llm_aided_title(pdf_info_list, title_aided_config)
+ logger.info(f'llm aided title time: {round(time.time() - llm_aided_title_start_time, 2)}')
+
+ cleanup_internal_para_block_metadata(pdf_info_list)
+
+
+def _detect_edge_text_line_hints(page_blocks, page_pil_img, scale):
+ text_blocks = [block for block in page_blocks if block.get("type") == "text"]
+ if not text_blocks:
+ return {}
+
+ edge_blocks = {}
+ edge_blocks["first"] = text_blocks[0]
+ edge_blocks["last"] = text_blocks[-1]
+
+ ocr_model = get_ch_lite_ocr_det_model()
+
+ edge_line_hints = {}
+ for edge_name, block in edge_blocks.items():
+ line_bboxes = _detect_block_line_bboxes(block, page_pil_img, scale, ocr_model)
+ if line_bboxes:
+ edge_line_hints[edge_name] = {
+ "index": block.get("index"),
+ "lines": [{"bbox": bbox, "spans": []} for bbox in line_bboxes],
+ }
+
+ return edge_line_hints
+
+
+def _detect_block_line_bboxes(block, page_pil_img, scale, ocr_model):
+ block_bbox = block.get("bbox")
+ if not block_bbox:
+ return []
+
+ ocr_det_res, padding = detect_ocr_boxes_from_padded_crop(
+ block_bbox,
+ page_pil_img,
+ scale,
+ ocr_model=ocr_model,
+ )
+ if not ocr_det_res:
+ return []
+
+ block_x0, block_y0 = block_bbox[0], block_bbox[1]
+ line_bboxes = []
+ for box in ocr_det_res:
+ x_coords = [point[0] - padding for point in box]
+ y_coords = [point[1] - padding for point in box]
+ line_bboxes.append([
+ block_x0 + min(x_coords) / scale,
+ block_y0 + min(y_coords) / scale,
+ block_x0 + max(x_coords) / scale,
+ block_y0 + max(y_coords) / scale,
+ ])
+
+ line_bboxes.sort(key=lambda bbox: bbox[1])
+ return line_bboxes
+
+
+def result_to_middle_json(
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ hybrid_pipeline_model,
+):
+ middle_json = init_middle_json(_ocr_enable, _vlm_ocr_enable)
+
+ with tqdm(total=len(model_list), desc="Processing pages") as progress_bar:
+ append_page_model_list_to_middle_json(
+ middle_json,
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ _ocr_enable=_ocr_enable,
+ _vlm_ocr_enable=_vlm_ocr_enable,
+ progress_bar=progress_bar,
+ )
+
+ finalize_middle_json(
+ middle_json["pdf_info"],
+ hybrid_pipeline_model,
+ _ocr_enable,
+ _vlm_ocr_enable,
+ )
+ close_pdfium_document(pdf_doc)
+ return middle_json
diff --git a/mineru/backend/office/__init__.py b/mineru/backend/office/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/office/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/office/docx_analyze.py b/mineru/backend/office/docx_analyze.py
new file mode 100644
index 000000000..a37e73946
--- /dev/null
+++ b/mineru/backend/office/docx_analyze.py
@@ -0,0 +1,70 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import time
+from io import BytesIO
+
+from loguru import logger
+from mineru.backend.office.model_output_to_middle_json import result_to_middle_json
+
+from mineru.model.docx.main import convert_binary
+
+
+def office_docx_analyze(
+ file_bytes,
+ image_writer=None
+):
+ infer_start = time.time()
+
+ file_stream = BytesIO(file_bytes)
+ results = convert_binary(file_stream)
+
+ infer_time = round(time.time() - infer_start, 2)
+ safe_time = max(infer_time, 0.01)
+ logger.debug(f"infer finished, cost: {infer_time}, speed: {round(len(results) / safe_time, 3)} page/s")
+
+ middle_json = result_to_middle_json(
+ results,
+ image_writer,
+ )
+
+ return middle_json, results
+
+if __name__ == '__main__':
+ # Resolve a default docx file relative to this script so the example
+ # works no matter what the current working directory is when the
+ # module is executed. Allow the user to override the path via a
+ # command-line argument for even greater flexibility.
+ from pathlib import Path
+ import argparse
+
+ script_root = Path(__file__).resolve().parent.parent.parent.parent
+ default_docx = script_root / "demo" / "office_docs" / "docx_01.docx"
+
+ parser = argparse.ArgumentParser(
+ description="Quick demo runner for office_docx_analyze"
+ )
+ parser.add_argument(
+ "docx",
+ nargs="?",
+ default=str(default_docx),
+ help="path to docx file (defaults to demo/office_docs/docx_01.docx relative to project root)"
+ )
+ parser.add_argument(
+ "--output-images",
+ help="directory to write image outputs",
+ default="./output_images"
+ )
+ args = parser.parse_args()
+
+ docx_path = Path(args.docx)
+ from mineru.data.data_reader_writer import FileBasedDataWriter
+
+ with open(docx_path, 'rb') as f:
+ file_bytes = f.read()
+ image_writer = FileBasedDataWriter(args.output_images)
+ middle_json, results = office_docx_analyze(
+ file_bytes,
+ image_writer=image_writer,
+ )
+
+ import json
+ logger.info(json.dumps(middle_json, indent=2, ensure_ascii=False))
diff --git a/mineru/backend/office/model_output_to_middle_json.py b/mineru/backend/office/model_output_to_middle_json.py
new file mode 100644
index 000000000..208cf0c4b
--- /dev/null
+++ b/mineru/backend/office/model_output_to_middle_json.py
@@ -0,0 +1,171 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+from collections import defaultdict
+
+from mineru.backend.utils.html_image_utils import replace_inline_table_images, save_span_image_if_needed
+from mineru.backend.office.office_magic_model import MagicModel
+from mineru.utils.enum_class import BlockType
+from mineru.version import __version__
+
+
+def blocks_to_page_info(page_blocks, image_writer, page_index) -> dict:
+ """将blocks转换为页面信息"""
+
+ magic_model = MagicModel(page_blocks)
+ image_blocks = magic_model.get_image_blocks()
+ table_blocks = magic_model.get_table_blocks()
+ chart_blocks = magic_model.get_chart_blocks()
+
+ if image_writer:
+
+ # Write embedded images to local storage via image_writer
+ for img_block in image_blocks:
+ for sub_block in img_block.get("blocks", []):
+ if sub_block.get("type") != "image_body":
+ continue
+ for line in sub_block.get("lines", []):
+ for span in line.get("spans", []):
+ save_span_image_if_needed(span, image_writer, page_index)
+
+ replace_inline_table_images(table_blocks, image_writer, page_index)
+
+ # Replace inline base64 images inside chart content with local paths
+ for chart_block in chart_blocks:
+ for sub_block in chart_block.get("blocks", []):
+ if sub_block.get("type") != "chart_body":
+ continue
+ for line in sub_block.get("lines", []):
+ for span in line.get("spans", []):
+ if span.get("type") != "chart":
+ continue
+ save_span_image_if_needed(span, image_writer, page_index)
+
+ title_blocks = magic_model.get_title_blocks()
+ discarded_blocks = magic_model.get_discarded_blocks()
+ list_blocks = magic_model.get_list_blocks()
+ index_blocks = magic_model.get_index_blocks()
+ text_blocks = magic_model.get_text_blocks()
+ interline_equation_blocks = magic_model.get_interline_equation_blocks()
+
+ page_blocks = []
+ page_blocks.extend([
+ *image_blocks,
+ *chart_blocks,
+ *table_blocks,
+ *title_blocks,
+ *text_blocks,
+ *interline_equation_blocks,
+ *list_blocks,
+ *index_blocks,
+ ])
+ # 对page_blocks根据index的值进行排序
+ page_blocks.sort(key=lambda x: x["index"])
+
+ page_info = {"para_blocks": page_blocks, "discarded_blocks": discarded_blocks, "page_idx": page_index}
+ return page_info
+
+
+def _extract_section_parts_from_content(content: str, level: int):
+ """Try to extract a leading section number (e.g. '1.2.1') from title content.
+
+ Returns a list of ints [n1, n2, ..., nL] when the number of parts equals
+ `level`, otherwise None. Handles formats like:
+ '1心肌特异性...' (no separator)
+ '1.2.1建立...' (Chinese text immediately after number)
+ '2.2.1 ALKBH5 ...' (space separator)
+ """
+ match = re.match(r'^(\d+(?:\.\d+)*)', content.strip())
+ if match:
+ parts = [int(p) for p in match.group(1).split('.')]
+ if len(parts) == level:
+ return parts
+ return None
+
+
+def _collect_index_text_blocks(index_block: dict, result: list[dict]) -> None:
+ """Depth-first collect TOC leaf text blocks."""
+ for child in index_block.get("blocks", []):
+ if child.get("type") == BlockType.INDEX:
+ _collect_index_text_blocks(child, result)
+ elif child.get("type") == BlockType.TEXT:
+ result.append(child)
+
+
+def _link_index_entries_by_anchor(middle_json: dict) -> None:
+ """Keep TOC anchors only when they exist on parsed body blocks."""
+ pdf_info = middle_json.get("pdf_info", [])
+ valid_anchors: set[str] = set()
+
+ for page_info in pdf_info:
+ for block in page_info.get("para_blocks", []):
+ anchor = block.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ valid_anchors.add(anchor.strip())
+
+ if not valid_anchors:
+ return
+
+ for page_info in pdf_info:
+ for block in page_info.get("para_blocks", []):
+ if block.get("type") != BlockType.INDEX:
+ continue
+ toc_text_blocks: list[dict] = []
+ _collect_index_text_blocks(block, toc_text_blocks)
+ for text_block in toc_text_blocks:
+ anchor = text_block.get("anchor")
+ if not isinstance(anchor, str):
+ text_block.pop("anchor", None)
+ continue
+ anchor = anchor.strip()
+ if not anchor or anchor not in valid_anchors:
+ text_block.pop("anchor", None)
+ continue
+ text_block["anchor"] = anchor
+
+
+def result_to_middle_json(model_output_blocks_list, image_writer):
+ middle_json = {"pdf_info": [], "_backend":"office", "_version_name": __version__}
+ for index, page_blocks in enumerate(model_output_blocks_list):
+ page_info = blocks_to_page_info(page_blocks, image_writer, index)
+ middle_json["pdf_info"].append(page_info)
+
+ section_counters: dict[int, int] = defaultdict(int)
+ for page_info in middle_json["pdf_info"]:
+ for block in page_info.get("para_blocks", []):
+ if block.get("type") != BlockType.TITLE:
+ continue
+ level = block.get("level", 1)
+ if block.get("is_numbered_style", False):
+ # Ensure all ancestor levels start at 1 (never 0)
+ for ancestor in range(1, level):
+ if section_counters[ancestor] == 0:
+ section_counters[ancestor] = 1
+ # Increment current level counter and reset all deeper levels
+ section_counters[level] += 1
+ for deeper in list(section_counters.keys()):
+ if deeper > level:
+ section_counters[deeper] = 0
+ # Build section number string, e.g. "1.2.1."
+ section_number = ".".join(
+ str(section_counters[l]) for l in range(1, level + 1)
+ ) + "."
+ block["section_number"] = section_number
+ else:
+ # Some documents embed the section number directly in the content
+ # (is_numbered_style=False). Parse it and sync the counters so
+ # that subsequent numbered blocks continue from the right base.
+ lines = block.get("lines", [])
+ content = ""
+ if lines and lines[0].get("spans"):
+ content = lines[0]["spans"][0].get("content", "")
+ parts = _extract_section_parts_from_content(content, level)
+ if parts:
+ for k, v in enumerate(parts, start=1):
+ section_counters[k] = v
+ # Reset all deeper levels
+ for deeper in list(section_counters.keys()):
+ if deeper > level:
+ section_counters[deeper] = 0
+
+ _link_index_entries_by_anchor(middle_json)
+ return middle_json
diff --git a/mineru/backend/office/office_magic_model.py b/mineru/backend/office/office_magic_model.py
new file mode 100644
index 000000000..1450d75e6
--- /dev/null
+++ b/mineru/backend/office/office_magic_model.py
@@ -0,0 +1,781 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+from typing import Literal
+
+from loguru import logger
+
+from mineru.utils.enum_class import ContentType, BlockType
+from mineru.utils.magic_model_utils import tie_up_category_by_index
+
+
+class MagicModel:
+ def __init__(self, page_blocks: list):
+ self.page_blocks = page_blocks
+
+ blocks = []
+ self.all_spans = []
+
+ # 对caption块进行分类，将其分类为image_caption, table_caption, chart_caption
+ page_blocks = classify_caption_blocks(page_blocks)
+
+ # 解析每个块
+ for index, block_info in enumerate(page_blocks):
+
+ block_type = block_info["type"]
+ block_content = block_info.get("content", "")
+ if not block_content and block_type != BlockType.CHART:
+ continue
+
+ if block_type in [
+ "text",
+ "title",
+ "image_caption",
+ "table_caption",
+ "chart_caption",
+ "header",
+ "footer",
+ "page_footnote",
+ ]:
+ span = parse_text_block_spans(block_content)
+
+ elif block_type in ["image"]:
+ block_type = BlockType.IMAGE_BODY
+ span = {
+ "type": ContentType.IMAGE,
+ "image_base64": block_content,
+ }
+ elif block_type in ["table"]:
+ block_type = BlockType.TABLE_BODY
+ span = {
+ "type": ContentType.TABLE,
+ "html": clean_table_html(block_content),
+ }
+ elif block_type in ["chart"]:
+ block_type = BlockType.CHART_BODY
+ span = {
+ "type": ContentType.CHART,
+ "content": block_content,
+ }
+ if block_info.get("image_base64"):
+ span["image_base64"] = block_info["image_base64"]
+ elif block_type in ["equation"]:
+ block_type = BlockType.INTERLINE_EQUATION
+ span = {
+ "type": ContentType.INTERLINE_EQUATION,
+ "content": block_content,
+ }
+ elif block_type in ["list"]:
+ # 解析嵌套列表结构，生成与VLM一致的blocks结构
+ parsed_list = parse_list_block(block_info)
+ if parsed_list:
+ # 使用外层index作为列表block的index
+ parsed_list["index"] = index
+ blocks.append(parsed_list)
+ continue
+ elif block_type in ["index"]:
+ # 解析嵌套索引结构（目录），生成与list一致的blocks结构
+ parsed_index = parse_index_block(block_info)
+ if parsed_index:
+ parsed_index["index"] = index
+ blocks.append(parsed_index)
+ continue
+ else:
+ # 未知类型，跳过
+ continue
+
+ # 处理span类型并添加到all_spans
+ if isinstance(span, dict):
+ line = {
+ "spans": [span]
+ }
+ elif isinstance(span, list):
+ line = {
+ "spans":span
+ }
+ else:
+ raise ValueError(f"Unsupported span type: {type(span)}")
+
+ block = {
+ "type": block_type,
+ "lines": [line],
+ "index": index,
+ }
+ anchor = block_info.get("anchor")
+ if (
+ isinstance(anchor, str)
+ and anchor.strip()
+ and block_type in [BlockType.TITLE, BlockType.TEXT, BlockType.INTERLINE_EQUATION]
+ ):
+ block["anchor"] = anchor.strip()
+ if block_type == BlockType.TITLE:
+ block["is_numbered_style"] = block_info.get("is_numbered_style", False)
+ block["level"] = block_info.get("level", 1)
+ blocks.append(block)
+
+ self.image_blocks = []
+ self.table_blocks = []
+ self.chart_blocks = []
+ self.interline_equation_blocks = []
+ self.text_blocks = []
+ self.title_blocks = []
+ self.discarded_blocks = []
+ self.list_blocks = []
+ self.index_blocks = []
+ for block in blocks:
+ if block["type"] in [BlockType.IMAGE_BODY, BlockType.IMAGE_CAPTION, BlockType.IMAGE_FOOTNOTE]:
+ self.image_blocks.append(block)
+ elif block["type"] in [BlockType.TABLE_BODY, BlockType.TABLE_CAPTION, BlockType.TABLE_FOOTNOTE]:
+ self.table_blocks.append(block)
+ elif block["type"] in [BlockType.CHART_BODY, BlockType.CHART_CAPTION]:
+ self.chart_blocks.append(block)
+ elif block["type"] == BlockType.INTERLINE_EQUATION:
+ self.interline_equation_blocks.append(block)
+ elif block["type"] == BlockType.TEXT:
+ self.text_blocks.append(block)
+ elif block["type"] == BlockType.TITLE:
+ self.title_blocks.append(block)
+ elif block["type"] in [BlockType.REF_TEXT]:
+ self.ref_text_blocks.append(block)
+ elif block["type"] in [BlockType.PHONETIC]:
+ self.phonetic_blocks.append(block)
+ elif block["type"] in [BlockType.HEADER, BlockType.FOOTER, BlockType.PAGE_NUMBER, BlockType.ASIDE_TEXT, BlockType.PAGE_FOOTNOTE]:
+ self.discarded_blocks.append(block)
+ elif block["type"] == BlockType.LIST:
+ self.list_blocks.append(block)
+ elif block["type"] == BlockType.INDEX:
+ self.index_blocks.append(block)
+ else:
+ continue
+
+ self.image_blocks, not_include_image_blocks = fix_two_layer_blocks(self.image_blocks, BlockType.IMAGE)
+ self.table_blocks, not_include_table_blocks = fix_two_layer_blocks(self.table_blocks, BlockType.TABLE)
+ self.chart_blocks, not_include_chart_blocks = fix_two_layer_blocks(self.chart_blocks, BlockType.CHART)
+
+ for block in not_include_image_blocks + not_include_table_blocks + not_include_chart_blocks:
+ block["type"] = BlockType.TEXT
+ self.text_blocks.append(block)
+
+
+ def get_list_blocks(self):
+ return self.list_blocks
+
+ def get_index_blocks(self):
+ return self.index_blocks
+
+ def get_image_blocks(self):
+ return self.image_blocks
+
+ def get_table_blocks(self):
+ return self.table_blocks
+
+ def get_chart_blocks(self):
+ return self.chart_blocks
+
+ def get_title_blocks(self):
+ return self.title_blocks
+
+ def get_text_blocks(self):
+ return self.text_blocks
+
+ def get_interline_equation_blocks(self):
+ return self.interline_equation_blocks
+
+ def get_discarded_blocks(self):
+ return self.discarded_blocks
+
+
+def parse_text_block_spans(content: str) -> list:
+ """
+ 解析文本类block的content，提取其中的文本、行内公式、超链接和字体样式。
+
+ 支持的标签格式：
+ - ... : 行内公式
+ - ... ... ... : 带字体样式的普通文本
+
+ 字体样式值（逗号分隔）：bold, italic, underline, strikethrough
+
+ Args:
+ content: 文本块的content字符串，可能包含特殊标签
+
+ Returns:
+ 包含多个span的列表，每个span是一个字典，包含type和content等字段。
+ 带样式的文本span额外包含 style 字段（list类型）。
+ """
+ if not content:
+ return []
+
+ # 匹配 或 开始标签
+ _text_tag_re = re.compile(r'')
+
+ spans = []
+ last_end = 0
+ pos = 0
+
+ while pos < len(content):
+ # 查找行内公式标签 ...
+ eq_start = content.find('', pos)
+ # 查找超链接标签
+ hyperlink_start = content.find('', pos)
+ # 查找带样式的文本标签 （顶层，不在 hyperlink 内部）
+ text_tag_match = _text_tag_re.search(content, pos)
+ text_tag_start = text_tag_match.start() if text_tag_match else -1
+
+ # 收集所有有效的标签位置
+ candidates = []
+ if eq_start != -1:
+ candidates.append((eq_start, 'eq'))
+ if hyperlink_start != -1:
+ candidates.append((hyperlink_start, 'hyperlink'))
+ if text_tag_start != -1:
+ candidates.append((text_tag_start, 'text'))
+
+ # 没有找到任何标签，处理剩余文本
+ if not candidates:
+ remaining_text = content[last_end:]
+ if remaining_text:
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": remaining_text
+ })
+ break
+
+ # 取位置最小的标签
+ next_tag_pos, next_tag_type = min(candidates, key=lambda x: x[0])
+
+ # 处理标签前的文本
+ if next_tag_pos > last_end:
+ text_before = content[last_end:next_tag_pos]
+ if text_before:
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": text_before
+ })
+
+ # 处理行内公式
+ if next_tag_type == 'eq':
+ eq_end = content.find(' ', next_tag_pos)
+ if eq_end != -1:
+ formula_content = content[next_tag_pos + 4:eq_end]
+ spans.append({
+ "type": ContentType.INLINE_EQUATION,
+ "content": formula_content
+ })
+ pos = eq_end + 5 # 跳过
+ last_end = pos
+ else:
+ # 未找到闭合标签，将作为普通文本处理
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": content[last_end:]
+ })
+ break
+
+ # 处理带样式的文本标签
+ elif next_tag_type == 'text':
+ text_end = content.find(' ', next_tag_pos)
+ if text_end != -1:
+ # text_tag_match 对应当前 next_tag_pos 的匹配
+ # 重新匹配确保位置对齐
+ tag_open_end = content.find('>', next_tag_pos) + 1
+ text_content = content[tag_open_end:text_end]
+ style_str = text_tag_match.group(1) if text_tag_match and text_tag_match.start() == next_tag_pos else None
+ span = {
+ "type": ContentType.TEXT,
+ "content": text_content
+ }
+ if style_str:
+ span["style"] = [s.strip() for s in style_str.split(',') if s.strip()]
+ spans.append(span)
+ pos = text_end + 7 # 跳过
+ last_end = pos
+ else:
+ # 未找到闭合标签，作为普通文本处理
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": content[last_end:]
+ })
+ break
+
+ # 处理超链接
+ elif next_tag_type == 'hyperlink':
+ hyperlink_end = content.find('', next_tag_pos)
+ if hyperlink_end != -1:
+ # 提取超链接内容
+ hyperlink_content = content[next_tag_pos + 11:hyperlink_end]
+
+ # 解析内部的 和 标签
+ inner_text_match = _text_tag_re.search(hyperlink_content)
+ text_end_in_hl = hyperlink_content.find(' ')
+ url_start = hyperlink_content.find('')
+ url_end = hyperlink_content.find(' ')
+
+ if inner_text_match and text_end_in_hl != -1 and url_start != -1 and url_end != -1:
+ style_str = inner_text_match.group(1)
+ link_text_start = inner_text_match.end() # 开始标签结束后的位置
+ link_text = hyperlink_content[link_text_start:text_end_in_hl]
+ link_url = hyperlink_content[url_start + 5:url_end]
+
+ span = {
+ "type": ContentType.HYPERLINK,
+ "content": link_text,
+ "url": link_url
+ }
+ if style_str:
+ span["style"] = [s.strip() for s in style_str.split(',') if s.strip()]
+ spans.append(span)
+ pos = hyperlink_end + 12 # 跳过
+ last_end = pos
+ else:
+ # 超链接格式不正确，作为普通文本处理
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": content[last_end:]
+ })
+ break
+ else:
+ # 未找到闭合标签，将作为普通文本处理
+ spans.append({
+ "type": ContentType.TEXT,
+ "content": content[last_end:]
+ })
+ break
+
+ return spans
+
+
+def parse_list_block(list_block: dict):
+ """
+ 递归解析嵌套列表结构，生成与VLM一致的blocks结构。
+
+ Args:
+ list_block: 列表块字典
+
+ Returns:
+ tuple: (解析后的列表block, 下一个可用索引)
+ """
+ content = list_block.get("content", [])
+ if not content:
+ return None
+
+ blocks = []
+
+ for item in content:
+ item_type = item.get("type", "")
+
+ if item_type == "text":
+ # 解析文本项（可能包含行内公式和超链接）
+ text_content = item.get("content", "")
+ spans = parse_text_block_spans(text_content)
+ text_block = {
+ "type": BlockType.TEXT,
+ "lines": [{"spans": spans}]
+ }
+ blocks.append(text_block)
+
+ elif item_type == "list":
+ # 递归解析嵌套列表
+ nested_list = parse_list_block(item)
+ if nested_list:
+ blocks.append(nested_list)
+
+ # 构建当前列表block
+ result = {
+ "type": BlockType.LIST,
+ "attribute": list_block.get("attribute", "unordered"),
+ "ilevel": list_block.get("ilevel", 0),
+ "blocks": blocks
+ }
+
+ return result
+
+
+def parse_index_block(index_block: dict):
+ """
+ 递归解析嵌套索引结构（目录），生成与list一致的blocks结构。
+
+ Args:
+ index_block: 索引块字典
+
+ Returns:
+ 解析后的索引block字典，若内容为空则返回 None
+ """
+ content = index_block.get("content", [])
+ if not content:
+ return None
+
+ blocks = []
+
+ for item in content:
+ item_type = item.get("type", "")
+
+ if item_type == "text":
+ text_content = item.get("content", "")
+ spans = parse_text_block_spans(text_content)
+ text_block = {
+ "type": BlockType.TEXT,
+ "lines": [{"spans": spans}]
+ }
+ anchor = item.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ text_block["anchor"] = anchor.strip()
+ blocks.append(text_block)
+
+ elif item_type == "index":
+ nested_index = parse_index_block(item)
+ if nested_index:
+ blocks.append(nested_index)
+
+ result = {
+ "type": BlockType.INDEX,
+ "ilevel": index_block.get("ilevel", 0),
+ "blocks": blocks
+ }
+
+ return result
+
+
+def clean_table_html(html: str) -> str:
+ """
+ 清洗表格HTML，只保留对表格结构表示有用的信息。
+
+ 保留的属性：
+ - colspan: 列合并
+ - rowspan: 行合并
+
+ 清洗的内容：
+ - 移除所有style属性
+ - 移除所有class属性
+ - 移除border等其他属性
+ - 保持表格结构标签（table, thead, tbody, tr, th, td等）
+
+ Args:
+ html: 原始表格HTML字符串
+
+ Returns:
+ 清洗后的HTML字符串
+ """
+ if not html:
+ return ""
+
+ # 需要保留的属性（对表格结构有用）
+ preserved_attrs = {'colspan', 'rowspan'}
+ # img 标签需要额外保留的属性（内联 base64 图片内容）
+ img_preserved_attrs = {'src', 'alt', 'width', 'height'}
+
+ def clean_tag(match):
+ """清洗单个标签，只保留结构相关的属性"""
+ full_tag = match.group(0)
+ tag_name = match.group(1).lower()
+
+ # 自闭合标签的处理
+ is_self_closing = full_tag.rstrip().endswith('/>')
+
+ # img 标签额外保留图片相关属性（如内联 base64 src）
+ current_preserved = preserved_attrs | (img_preserved_attrs if tag_name == 'img' else set())
+
+ # 提取需要保留的属性
+ kept_attrs = []
+
+ # 匹配所有属性: attr="value" 或 attr='value' 或 attr=value 或单独的attr
+ attr_pattern = r'(\w+)\s*=\s*(?:"([^"]*)"|\'([^\']*)\'|(\S+))|(\w+)(?=\s|>|/>)'
+ for attr_match in re.finditer(attr_pattern, full_tag):
+ if attr_match.group(5):
+ # 单独的属性（如 disabled），跳过
+ continue
+
+ attr_name = attr_match.group(1)
+ if attr_name is None:
+ continue
+ attr_name = attr_name.lower()
+ attr_value = attr_match.group(2) or attr_match.group(3) or attr_match.group(4) or ""
+
+ # 只保留指定属性（表格结构属性，img 标签还额外保留图片内容属性）
+ if attr_name in current_preserved:
+ kept_attrs.append(f'{attr_name}="{attr_value}"')
+
+ # 重建标签
+ if kept_attrs:
+ attrs_str = ' ' + ' '.join(kept_attrs)
+ else:
+ attrs_str = ''
+
+ if is_self_closing:
+ return f'<{tag_name}{attrs_str}/>'
+ else:
+ return f'<{tag_name}{attrs_str}>'
+
+ # 匹配开始标签（包括自闭合标签），捕获标签名
+ # 匹配 或 {content}'
+
+ if 'underline' in style:
+ content = f'{content}'
+
+ return content
+
+
+def _get_office_style_render_mode() -> str:
+ mode = os.getenv(
+ OFFICE_STYLE_RENDER_MODE_ENV,
+ OFFICE_STYLE_RENDER_MODE_MARKDOWN,
+ ).strip().lower()
+ if mode in {
+ OFFICE_STYLE_RENDER_MODE_HTML,
+ OFFICE_STYLE_RENDER_MODE_MARKDOWN,
+ }:
+ return mode
+ logger.warning(
+ f"Invalid {OFFICE_STYLE_RENDER_MODE_ENV}={mode!r}, "
+ f"fallback to {OFFICE_STYLE_RENDER_MODE_MARKDOWN!r}"
+ )
+ return OFFICE_STYLE_RENDER_MODE_MARKDOWN
+
+
+def _apply_configured_style(content: str, style: list) -> str:
+ if _get_office_style_render_mode() == OFFICE_STYLE_RENDER_MODE_MARKDOWN:
+ return _apply_markdown_style(content, style)
+ return _apply_html_style(content, style)
+
+
+def _render_link(text: str, url: str) -> str:
+ if _get_office_style_render_mode() == OFFICE_STYLE_RENDER_MODE_MARKDOWN:
+ return f'[{text}]({url})'
+ return f'{text}'
+
+
+def _prefix_table_img_src(html: str, img_buket_path: str) -> str:
+ """Prefix local-path img src attributes in table HTML with img_buket_path."""
+ if not html or not img_buket_path:
+ return html
+ return re.sub(
+ r'src="(?!data:)([^"]+)"',
+ lambda m: f'src="{img_buket_path}/{m.group(1)}"',
+ html,
+ )
+
+
+def _replace_eq_tags_in_table_html(html: str) -> str:
+ """Replace ... tags in table HTML with inline math delimiters."""
+ if not html:
+ return html
+ return re.sub(
+ r'(.*?) ',
+ lambda m: f' {inline_left_delimiter}{m.group(1)}{inline_right_delimiter} ',
+ html,
+ flags=re.DOTALL,
+ )
+
+
+def _format_embedded_html(html: str, img_buket_path: str) -> str:
+ """Apply image-path prefixing and equation replacement for HTML-like content."""
+ return _replace_eq_tags_in_table_html(_prefix_table_img_src(html, img_buket_path))
+
+
+def _build_media_path(img_buket_path: str, image_path: str) -> str:
+ """Build a display path while keeping empty image references empty."""
+ if not image_path:
+ return ''
+ if not img_buket_path:
+ return image_path
+ return f"{img_buket_path}/{image_path}"
+
+
+def _escape_office_markdown_text(content: str) -> str:
+ """Escape plain-text Office content before applying Markdown wrappers."""
+ if not content:
+ return content
+ if _get_office_style_render_mode() != OFFICE_STYLE_RENDER_MODE_MARKDOWN:
+ return content
+ return escape_conservative_markdown_text(content)
+
+
+def get_title_level(para_block):
+ title_level = para_block.get('level', 2)
+ return title_level
+
+
+def _make_rendered_part(
+ span_type,
+ rendered_content: str,
+ raw_content: str = '',
+ style: list | None = None,
+ has_markdown_wrapper: bool = False,
+):
+ return {
+ 'span_type': span_type,
+ 'rendered_content': rendered_content,
+ 'raw_content': raw_content,
+ 'style': style or [],
+ 'has_markdown_wrapper': has_markdown_wrapper,
+ }
+
+
+def _has_markdown_wrapper(style: list) -> bool:
+ if _get_office_style_render_mode() != OFFICE_STYLE_RENDER_MODE_MARKDOWN:
+ return False
+ if not style or 'underline' in style:
+ return False
+ return any(name in OFFICE_MARKDOWN_WRAPPER_STYLES for name in style)
+
+
+def _get_first_non_whitespace_char(text: str):
+ for ch in text:
+ if not ch.isspace():
+ return ch
+ return None
+
+
+def _get_last_non_whitespace_char(text: str):
+ for ch in reversed(text):
+ if not ch.isspace():
+ return ch
+ return None
+
+
+def _is_punctuation_or_symbol(ch: str) -> bool:
+ return unicodedata.category(ch).startswith(('P', 'S'))
+
+
+def _is_boundary_text_char(ch: str) -> bool:
+ if ch.isspace():
+ return False
+ return not _is_punctuation_or_symbol(ch)
+
+
+def _needs_markdown_it_boundary_space(prev_part: dict, next_part: dict) -> bool:
+ if _get_office_style_render_mode() != OFFICE_STYLE_RENDER_MODE_MARKDOWN:
+ return False
+ if not prev_part.get('has_markdown_wrapper', False):
+ return False
+ if next_part.get('span_type') in {
+ ContentType.HYPERLINK,
+ ContentType.INLINE_EQUATION,
+ ContentType.INTERLINE_EQUATION,
+ }:
+ return False
+
+ prev_raw = prev_part.get('raw_content', '')
+ next_raw = next_part.get('raw_content', '')
+ if not prev_raw.strip() or not next_raw.strip():
+ return False
+ if prev_raw[-1].isspace() or next_raw[0].isspace():
+ return False
+
+ prev_char = _get_last_non_whitespace_char(prev_raw)
+ next_char = _get_first_non_whitespace_char(next_raw)
+ if prev_char is None or next_char is None:
+ return False
+ if not _is_punctuation_or_symbol(prev_char):
+ return False
+ if not _is_boundary_text_char(next_char):
+ return False
+ return True
+
+
+def _join_rendered_parts(parts: list[dict]) -> str:
+ para_text = ''
+ prev_part = None
+
+ for i, part in enumerate(parts):
+ span_type = part['span_type']
+ content = part['rendered_content']
+ is_last = i == len(parts) - 1
+
+ if span_type == ContentType.INLINE_EQUATION:
+ if para_text and not para_text.endswith(' '):
+ para_text += ' '
+ para_text += content
+ if not is_last:
+ para_text += ' '
+ else:
+ if prev_part is not None and _needs_markdown_it_boundary_space(prev_part, part):
+ para_text += ' '
+ para_text += content
+
+ prev_part = part
+
+ return para_text
+
+
+def _append_text_part(parts: list[dict], original_content: str, span_style: list):
+ escaped_content = _escape_office_markdown_text(original_content)
+ content_stripped = escaped_content.strip()
+ if content_stripped:
+ styled = _apply_configured_style(content_stripped, span_style)
+ leading = escaped_content[:len(escaped_content) - len(escaped_content.lstrip())]
+ trailing = escaped_content[len(escaped_content.rstrip()):]
+ parts.append(
+ _make_rendered_part(
+ ContentType.TEXT,
+ leading + styled + trailing,
+ raw_content=original_content,
+ style=span_style,
+ has_markdown_wrapper=_has_markdown_wrapper(span_style),
+ )
+ )
+ elif original_content:
+ visible_styles = {'underline', 'strikethrough'}
+ if span_style and any(s in visible_styles for s in span_style):
+ rendered_content = original_content.replace(" ", " ")
+ rendered_content = _apply_configured_style(rendered_content, span_style)
+ else:
+ rendered_content = original_content
+ parts.append(
+ _make_rendered_part(
+ ContentType.TEXT,
+ rendered_content,
+ raw_content=original_content,
+ style=span_style,
+ )
+ )
+
+
+def _append_hyperlink_part(
+ parts: list[dict],
+ original_content: str,
+ span_style: list,
+ url: str = '',
+ plain_text_only: bool = False,
+):
+ link_text = _escape_office_markdown_text(original_content.strip())
+ if not link_text:
+ return
+
+ styled_text = _apply_configured_style(link_text, span_style)
+ if plain_text_only:
+ leading = original_content[:len(original_content) - len(original_content.lstrip())]
+ trailing = original_content[len(original_content.rstrip()):]
+ rendered_content = leading + styled_text + trailing
+ has_markdown_wrapper = _has_markdown_wrapper(span_style)
+ else:
+ rendered_content = _render_link(styled_text, url)
+ has_markdown_wrapper = False
+
+ parts.append(
+ _make_rendered_part(
+ ContentType.HYPERLINK,
+ rendered_content,
+ raw_content=original_content,
+ style=span_style,
+ has_markdown_wrapper=has_markdown_wrapper,
+ )
+ )
+
+
+def merge_para_with_text(para_block, escape_text_block_prefix=True):
+ # First pass: collect rendered parts with raw boundary metadata.
+ parts = []
+ if para_block['type'] == BlockType.TITLE:
+ if para_block.get('is_numbered_style', False):
+ section_number = para_block.get('section_number', '')
+ if section_number:
+ parts.append(
+ _make_rendered_part(
+ ContentType.TEXT,
+ f"{section_number} ",
+ raw_content=f"{section_number} ",
+ )
+ )
+
+ for line in para_block['lines']:
+ for span in line['spans']:
+ span_type = span['type']
+ span_style = span.get('style', [])
+
+ if span_type == ContentType.TEXT:
+ _append_text_part(parts, span['content'], span_style)
+ elif span_type == ContentType.INLINE_EQUATION:
+ content = f"{inline_left_delimiter}{span['content']}{inline_right_delimiter}"
+ content = content.strip()
+ if content:
+ parts.append(
+ _make_rendered_part(
+ span_type,
+ content,
+ raw_content=span['content'],
+ )
+ )
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ content = f"\n{display_left_delimiter}\n{span['content']}\n{display_right_delimiter}\n"
+ content = content.strip()
+ if content:
+ parts.append(
+ _make_rendered_part(
+ span_type,
+ content,
+ raw_content=span['content'],
+ )
+ )
+ elif span_type == ContentType.HYPERLINK:
+ _append_hyperlink_part(
+ parts,
+ span['content'],
+ span_style,
+ url=span.get('url', ''),
+ )
+
+ para_text = _join_rendered_parts(parts)
+ if escape_text_block_prefix and para_block.get('type') == BlockType.TEXT:
+ para_text = escape_text_block_markdown_prefix(para_text)
+ return para_text
+
+
+def _flatten_list_items(list_block):
+ """Recursively flatten nested list blocks into a list of prefixed item strings."""
+ items = []
+ ilevel = list_block.get('ilevel', 0)
+ attribute = list_block.get('attribute', 'unordered')
+ indent = ' ' * ilevel
+ ordered_counter = 1
+
+ for block in list_block.get('blocks', []):
+ if block['type'] in [BlockType.LIST, BlockType.INDEX]:
+ items.extend(_flatten_list_items(block))
+ else:
+ item_text = merge_para_with_text(block, escape_text_block_prefix=False)
+ if item_text.strip():
+ if attribute == 'ordered':
+ items.append(f"{indent}{ordered_counter}. {item_text}")
+ ordered_counter += 1
+ else:
+ items.append(f"{indent}- {item_text}")
+
+ return items
+
+
+def _flatten_list_items_v2(list_block):
+ """Recursively flatten nested list blocks into v2-structured item dicts."""
+ items = []
+ ilevel = list_block.get('ilevel', 0)
+ attribute = list_block.get('attribute', 'unordered')
+ ordered_counter = 1
+
+ for block in list_block.get('blocks', []):
+ if block['type'] in [BlockType.LIST, BlockType.INDEX]:
+ items.extend(_flatten_list_items_v2(block))
+ else:
+ item_content = merge_para_with_text_v2(block)
+ if item_content:
+ if attribute == 'ordered':
+ prefix = f"{' ' * ilevel}{ordered_counter}."
+ ordered_counter += 1
+ else:
+ prefix = f"{' ' * ilevel}-"
+ item = {
+ 'item_type': 'text',
+ 'ilevel': ilevel,
+ 'prefix': prefix,
+ 'item_content': item_content,
+ }
+ anchor = block.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ item["anchor"] = anchor.strip()
+ items.append(item)
+
+ return items
+
+
+def merge_list_to_markdown(list_block):
+ """Recursively convert a nested list block to markdown text."""
+ return '\n'.join(_flatten_list_items(list_block)) + '\n'
+
+
+def _flatten_index_items(index_block):
+ """Recursively flatten index (TOC) blocks into markdown list items.
+
+ Strips the trailing tab+page-number from span content and, when target
+ location fields are present on the leaf text block, wraps the text in
+ a markdown hyperlink pointing to the body-block anchor.
+
+ Styling (bold, italic, underline, strikethrough) is applied via the
+ configured office style render mode. HYPERLINK spans are rendered as
+ plain styled text (without the URL) because TOC entries use
+ document-internal bookmark links, not external URLs.
+
+ The tab+page-number is stripped from the raw content BEFORE markdown
+ style markers are applied, so that closing markers (e.g. ``**``) are
+ never inadvertently removed by the tab-stripping step.
+ """
+ items = []
+ ilevel = index_block.get('ilevel', 0)
+ indent = ' ' * ilevel
+
+ for child in index_block.get('blocks', []):
+ if child.get('type') == BlockType.INDEX:
+ items.extend(_flatten_index_items(child))
+ elif child.get('type') == BlockType.TEXT:
+ span_items = [] # list of (content, span_type, span_style)
+ anchor = child.get('anchor')
+ if not isinstance(anchor, str) or not anchor.strip():
+ anchor = None
+ else:
+ anchor = anchor.strip()
+
+ for line in child.get('lines', []):
+ for span in line.get('spans', []):
+ content = span.get('content', '')
+ span_style = span.get('style', [])
+ span_type = span.get('type')
+ span_items.append((content, span_type, span_style))
+
+ if not span_items:
+ continue
+
+ # ----------------------------------------------------------
+ # Step 1: Strip the trailing tab+page-number from the raw
+ # (unstyled) content BEFORE applying markdown markers.
+ #
+ # Find the last non-equation span that contains a tab; strip
+ # everything after its last tab ONLY when the trailing token
+ # actually looks like a page number.
+ # Then replace any remaining internal tabs with spaces so that
+ # "1.1\t研究对象" → "1.1 研究对象".
+ # ----------------------------------------------------------
+ def _looks_like_page_token(token: str) -> bool:
+ token = token.strip()
+ if not token:
+ return False
+ # Page tokens are usually short and contain no CJK characters.
+ if len(token) > 12:
+ return False
+ if re.search(r'[\u4e00-\u9fff]', token):
+ return False
+ # Arabic / Roman / single-letter page styles.
+ if re.fullmatch(r'\d+', token):
+ return True
+ if re.fullmatch(r'[ivxlcdm]+', token.lower()):
+ return True
+ if re.fullmatch(r'[a-zA-Z]', token):
+ return True
+ return False
+
+ last_tab_span_idx = -1
+ for i, (content, span_type, _) in enumerate(span_items):
+ if span_type != ContentType.INLINE_EQUATION and '\t' in content:
+ last_tab_span_idx = i
+
+ should_strip_page_tail = False
+ if last_tab_span_idx != -1:
+ last_tab_content = span_items[last_tab_span_idx][0]
+ tab_tail = last_tab_content.rsplit('\t', 1)[1]
+ should_strip_page_tail = _looks_like_page_token(tab_tail)
+
+ # Build stripped span_items
+ stripped_span_items = []
+ for i, (content, span_type, span_style) in enumerate(span_items):
+ if span_type != ContentType.INLINE_EQUATION:
+ if i == last_tab_span_idx and should_strip_page_tail:
+ # Strip from last tab onwards (removes tab + page number)
+ content = content.rsplit('\t', 1)[0]
+ # Replace remaining internal tabs with spaces
+ content = content.replace('\t', ' ')
+ stripped_span_items.append((content, span_type, span_style))
+
+ # ----------------------------------------------------------
+ # Step 2: Apply markdown styles and build the final text.
+ #
+ # If all non-equation spans share the same non-empty style
+ # (common in TOC entries like all-bold), apply style once to
+ # the whole item to avoid fragmented markers such as
+ # "**foo****bar**".
+ # ----------------------------------------------------------
+ non_eq_styles = [
+ tuple(span_style)
+ for content, span_type, span_style in stripped_span_items
+ if content and span_type != ContentType.INLINE_EQUATION
+ ]
+ uniform_style = None
+ if non_eq_styles:
+ first_style = non_eq_styles[0]
+ if first_style and all(s == first_style for s in non_eq_styles):
+ uniform_style = list(first_style)
+
+ if uniform_style:
+ raw_parts = []
+ for content, span_type, _span_style in stripped_span_items:
+ if not content:
+ continue
+ if span_type == ContentType.INLINE_EQUATION:
+ raw_parts.append(
+ f'{inline_left_delimiter}{content}{inline_right_delimiter}'
+ )
+ else:
+ # For TOC rendering, hyperlink spans output as plain text.
+ raw_parts.append(_escape_office_markdown_text(content))
+ item_text = ''.join(raw_parts).strip()
+ if item_text:
+ item_text = _apply_configured_style(item_text, uniform_style)
+ else:
+ rendered_parts = []
+ for content, span_type, span_style in stripped_span_items:
+ if not content:
+ continue
+ if span_type == ContentType.INLINE_EQUATION:
+ rendered_parts.append(
+ _make_rendered_part(
+ span_type,
+ f'{inline_left_delimiter}{content}{inline_right_delimiter}',
+ raw_content=content,
+ )
+ )
+ elif span_type == ContentType.HYPERLINK:
+ _append_hyperlink_part(
+ rendered_parts,
+ content,
+ span_style,
+ plain_text_only=True,
+ )
+ else:
+ _append_text_part(rendered_parts, content, span_style)
+
+ item_text = _join_rendered_parts(rendered_parts).strip()
+ if not item_text:
+ continue
+
+ if anchor is not None:
+ item_text = _render_link(item_text, f"#{anchor}")
+
+ items.append(f"{indent}- {item_text}")
+
+ return items
+
+
+def merge_index_to_markdown(index_block):
+ """Convert a nested index (TOC) block to markdown with hyperlinks."""
+ return '\n'.join(_flatten_index_items(index_block)) + '\n'
+
+
+def mk_blocks_to_markdown(para_blocks, make_mode, img_buket_path='', page_idx=None):
+ page_markdown = []
+ for para_block in para_blocks:
+ para_text = ''
+ para_type = para_block['type']
+ if para_type in [BlockType.TEXT, BlockType.INTERLINE_EQUATION]:
+ para_text = merge_para_with_text(para_block)
+ if para_type == BlockType.TEXT:
+ bookmark_anchor = para_block.get("anchor")
+ if (
+ isinstance(bookmark_anchor, str)
+ and bookmark_anchor.strip()
+ and bookmark_anchor.strip().startswith("_Toc")
+ ):
+ para_text = f'\n{para_text}'
+ elif para_type == BlockType.LIST:
+ para_text = merge_list_to_markdown(para_block)
+ elif para_type == BlockType.INDEX:
+ para_text = merge_index_to_markdown(para_block)
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ title_text = merge_para_with_text(para_block)
+ bookmark_anchor = para_block.get("anchor")
+ if isinstance(bookmark_anchor, str) and bookmark_anchor.strip():
+ para_text = f'\n{"#" * title_level} {title_text}'
+ else:
+ para_text = f'{"#" * title_level} {title_text}'
+ elif para_type == BlockType.IMAGE:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ for block in para_block['blocks']: # 1st.拼image_body
+ if block['type'] == BlockType.IMAGE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.IMAGE:
+ if span.get('image_path', ''):
+ para_text += f""
+ for block in para_block['blocks']: # 2nd.拼image_caption
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ para_text += ' \n' + merge_para_with_text(block)
+
+ elif para_type == BlockType.TABLE:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ for block in para_block['blocks']: # 1st.拼table_body
+ if block['type'] == BlockType.TABLE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.TABLE:
+ para_text += f"\n{_format_embedded_html(span['html'], img_buket_path)}\n"
+ for block in para_block['blocks']: # 2nd.拼table_caption
+ if block['type'] == BlockType.TABLE_CAPTION:
+ para_text += ' \n' + merge_para_with_text(block)
+ elif para_type == BlockType.CHART:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ image_path, chart_content = get_body_data(para_block)
+ if chart_content:
+ para_text += f"\n{_format_embedded_html(chart_content, img_buket_path)}\n"
+ elif image_path:
+ para_text += f"})"
+ else:
+ continue
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CHART_CAPTION:
+ para_text += ' \n' + merge_para_with_text(block)
+ if para_text.strip() == '':
+ continue
+ else:
+ # page_markdown.append(para_text.strip())
+ page_markdown.append(para_text.strip('\r\n'))
+
+ return page_markdown
+
+
+def make_blocks_to_content_list(para_block, img_buket_path, page_idx):
+ para_type = para_block['type']
+ para_content = {}
+ if para_type in [
+ BlockType.TEXT,
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ para_content = {
+ 'type': para_type,
+ 'text': merge_para_with_text(para_block),
+ }
+ elif para_type == BlockType.LIST:
+ attribute = para_block.get('attribute', 'unordered')
+ para_content = {
+ 'type': para_type,
+ 'list_items': _flatten_list_items(para_block),
+ }
+ elif para_type == BlockType.INDEX:
+ para_content = {
+ 'type': para_type,
+ 'list_items': _flatten_index_items(para_block),
+ }
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ para_content = {
+ 'type': ContentType.TEXT,
+ 'text': merge_para_with_text(para_block),
+ }
+ if title_level != 0:
+ para_content['text_level'] = title_level
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ para_content = {
+ 'type': ContentType.EQUATION,
+ 'text': merge_para_with_text(para_block),
+ 'text_format': 'latex',
+ }
+ elif para_type == BlockType.IMAGE:
+ para_content = {'type': ContentType.IMAGE, 'img_path': '', BlockType.IMAGE_CAPTION: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.IMAGE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.IMAGE:
+ if span.get('image_path', ''):
+ para_content['img_path'] = f"{img_buket_path}/{span['image_path']}"
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ para_content[BlockType.IMAGE_CAPTION].append(merge_para_with_text(block))
+ elif para_type == BlockType.TABLE:
+ para_content = {'type': ContentType.TABLE, BlockType.TABLE_CAPTION: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.TABLE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.TABLE:
+ if span.get('html', ''):
+ para_content[BlockType.TABLE_BODY] = _format_embedded_html(span['html'], img_buket_path)
+ if block['type'] == BlockType.TABLE_CAPTION:
+ para_content[BlockType.TABLE_CAPTION].append(merge_para_with_text(block))
+ elif para_type == BlockType.CHART:
+ para_content = {
+ 'type': ContentType.CHART,
+ 'img_path': '',
+ 'content': '',
+ BlockType.CHART_CAPTION: [],
+ }
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CHART_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.CHART:
+ para_content['img_path'] = _build_media_path(
+ img_buket_path,
+ span.get('image_path', ''),
+ )
+ if span.get('content', ''):
+ para_content['content'] = _format_embedded_html(
+ span['content'],
+ img_buket_path,
+ )
+ if block['type'] == BlockType.CHART_CAPTION:
+ para_content[BlockType.CHART_CAPTION].append(merge_para_with_text(block))
+
+ para_content['page_idx'] = page_idx
+ anchor = para_block.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ para_content["anchor"] = anchor.strip()
+
+ return para_content
+
+
+def make_blocks_to_content_list_v2(para_block, img_buket_path):
+ para_type = para_block['type']
+ para_content = {}
+ if para_type in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ if para_type == BlockType.HEADER:
+ content_type = ContentTypeV2.PAGE_HEADER
+ elif para_type == BlockType.FOOTER:
+ content_type = ContentTypeV2.PAGE_FOOTER
+ elif para_type == BlockType.PAGE_FOOTNOTE:
+ content_type = ContentTypeV2.PAGE_FOOTNOTE
+ else:
+ raise ValueError(f"Unknown para_type: {para_type}")
+ para_content = {
+ 'type': content_type,
+ 'content': {
+ f"{content_type}_content": merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ if title_level != 0:
+ para_content = {
+ 'type': ContentTypeV2.TITLE,
+ 'content': {
+ "title_content": merge_para_with_text_v2(para_block),
+ "level": title_level
+ }
+ }
+ else:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ "paragraph_content": merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type in [
+ BlockType.TEXT,
+ ]:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ 'paragraph_content': merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ _, math_content = get_body_data(para_block)
+ para_content = {
+ 'type': ContentTypeV2.EQUATION_INTERLINE,
+ 'content': {
+ 'math_content': math_content,
+ 'math_type': 'latex',
+ }
+ }
+ elif para_type == BlockType.IMAGE:
+ image_caption = []
+ image_path, _ = get_body_data(para_block)
+ image_source = {
+ 'path': f"{img_buket_path}/{image_path}",
+ }
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ image_caption.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.IMAGE,
+ 'content': {
+ 'image_source': image_source,
+ 'image_caption': image_caption,
+ }
+ }
+ elif para_type == BlockType.TABLE:
+ table_caption = []
+ _, html = get_body_data(para_block)
+ if html.count(" 1:
+ table_nest_level = 2
+ else:
+ table_nest_level = 1
+ if (
+ "colspan" in html or
+ "rowspan" in html or
+ table_nest_level > 1
+ ):
+ table_type = ContentTypeV2.TABLE_COMPLEX
+ else:
+ table_type = ContentTypeV2.TABLE_SIMPLE
+
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.TABLE_CAPTION:
+ table_caption.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.TABLE,
+ 'content': {
+ 'table_caption': table_caption,
+ 'html': _format_embedded_html(html, img_buket_path),
+ 'table_type': table_type,
+ 'table_nest_level': table_nest_level,
+ }
+ }
+ elif para_type == BlockType.CHART:
+ chart_caption = []
+ image_path, chart_content = get_body_data(para_block)
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CHART_CAPTION:
+ chart_caption.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.CHART,
+ 'content': {
+ 'image_source': {
+ 'path': _build_media_path(img_buket_path, image_path),
+ },
+ 'content': _format_embedded_html(chart_content, img_buket_path),
+ 'chart_caption': chart_caption,
+ }
+ }
+ elif para_type == BlockType.LIST:
+ list_type = ContentTypeV2.LIST_TEXT
+ attribute = para_block.get('attribute', 'unordered')
+ para_content = {
+ 'type': ContentTypeV2.LIST,
+ 'content': {
+ 'list_type': list_type,
+ 'attribute': attribute,
+ 'list_items': _flatten_list_items_v2(para_block),
+ }
+ }
+ elif para_type == BlockType.INDEX:
+ para_content = {
+ 'type': ContentTypeV2.INDEX,
+ 'content': {
+ 'list_type': ContentTypeV2.LIST_TEXT,
+ 'list_items': _flatten_list_items_v2(para_block),
+ }
+ }
+
+ anchor = para_block.get("anchor")
+ if isinstance(anchor, str) and anchor.strip():
+ para_content["anchor"] = anchor.strip()
+
+ return para_content
+
+
+def get_body_data(para_block):
+ """
+ Extract image_path and body content from para_block
+ Returns:
+ - For IMAGE/INTERLINE_EQUATION: (image_path, '')
+ - For TABLE: (image_path, html)
+ - For CHART: (image_path, content)
+ - Default: ('', '')
+ """
+
+ def get_data_from_spans(lines):
+ for line in lines:
+ for span in line.get('spans', []):
+ span_type = span.get('type')
+ if span_type == ContentType.TABLE:
+ return span.get('image_path', ''), span.get('html', '')
+ elif span_type == ContentType.CHART:
+ return span.get('image_path', ''), span.get('content', '')
+ elif span_type == ContentType.IMAGE:
+ return span.get('image_path', ''), ''
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ return span.get('image_path', ''), span.get('content', '')
+ elif span_type == ContentType.TEXT:
+ return '', span.get('content', '')
+ return '', ''
+
+ # 处理嵌套的 blocks 结构
+ if 'blocks' in para_block:
+ for block in para_block['blocks']:
+ block_type = block.get('type')
+ if block_type in [BlockType.IMAGE_BODY, BlockType.TABLE_BODY, BlockType.CHART_BODY, BlockType.CODE_BODY]:
+ result = get_data_from_spans(block.get('lines', []))
+ if result != ('', ''):
+ return result
+ if block_type == BlockType.CHART_BODY:
+ return result
+ return '', ''
+
+ # 处理直接包含 lines 的结构
+ return get_data_from_spans(para_block.get('lines', []))
+
+
+def merge_para_with_text_v2(para_block):
+ _visible_styles = {'underline', 'strikethrough'}
+ para_content = []
+ for i, line in enumerate(para_block['lines']):
+ for j, span in enumerate(line['spans']):
+ content = span.get("content", '')
+ span_style = span.get('style', [])
+ has_visible_style = bool(
+ span_style and any(s in _visible_styles for s in span_style)
+ )
+ if content.strip() or (content and has_visible_style):
+ if span['type'] == ContentType.INLINE_EQUATION:
+ span['type'] = ContentTypeV2.SPAN_EQUATION_INLINE
+ para_content.append(span)
+ return para_content
+
+
+def union_make(pdf_info_dict: list,
+ make_mode: str,
+ img_buket_path: str = '',
+ ):
+
+ output_content = []
+ for page_info in pdf_info_dict:
+ paras_of_layout = page_info.get('para_blocks')
+ paras_of_discarded = page_info.get('discarded_blocks')
+ page_idx = page_info.get('page_idx')
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ if not paras_of_layout:
+ continue
+ page_markdown = mk_blocks_to_markdown(paras_of_layout, make_mode, img_buket_path,
+ page_idx=page_idx)
+ output_content.extend(page_markdown)
+ elif make_mode == MakeMode.CONTENT_LIST:
+ para_blocks = (paras_of_layout or []) + (paras_of_discarded or [])
+ if not para_blocks:
+ continue
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list(para_block, img_buket_path, page_idx)
+ output_content.append(para_content)
+ elif make_mode == MakeMode.CONTENT_LIST_V2:
+ # https://github.com/drunkpig/llm-webkit-mirror/blob/dev6/docs/specification/output_format/content_list_spec.md
+ para_blocks = (paras_of_layout or []) + (paras_of_discarded or [])
+ page_contents = []
+ if para_blocks:
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list_v2(para_block, img_buket_path)
+ page_contents.append(para_content)
+ output_content.append(page_contents)
+
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ return '\n\n'.join(output_content)
+ elif make_mode in [MakeMode.CONTENT_LIST, MakeMode.CONTENT_LIST_V2]:
+ return output_content
+ return None
diff --git a/mineru/backend/office/pptx_analyze.py b/mineru/backend/office/pptx_analyze.py
new file mode 100644
index 000000000..22cac4f0b
--- /dev/null
+++ b/mineru/backend/office/pptx_analyze.py
@@ -0,0 +1,71 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import time
+from io import BytesIO
+
+from loguru import logger
+from mineru.backend.office.model_output_to_middle_json import result_to_middle_json
+
+from mineru.model.pptx.main import convert_binary
+
+
+def office_pptx_analyze(file_bytes, image_writer=None):
+ infer_start = time.time()
+
+ file_stream = BytesIO(file_bytes)
+ results = convert_binary(file_stream)
+
+ infer_time = round(time.time() - infer_start, 2)
+ safe_time = max(infer_time, 0.01)
+ logger.debug(
+ f"infer finished, cost: {infer_time}, speed: {round(len(results) / safe_time, 3)} page/s"
+ )
+
+ middle_json = result_to_middle_json(
+ results,
+ image_writer,
+ )
+
+ return middle_json, results
+
+
+if __name__ == "__main__":
+ # Resolve a default pptx file relative to this script so the example
+ # works no matter what the current working directory is when the
+ # module is executed. Allow the user to override the path via a
+ # command-line argument for even greater flexibility.
+ from pathlib import Path
+ import argparse
+
+ script_root = Path(__file__).resolve().parent.parent.parent.parent
+ default_pptx = script_root / "demo" / "office_docs" / "pptx_01.pptx"
+
+ parser = argparse.ArgumentParser(
+ description="Quick demo runner for office_pptx_analyze"
+ )
+ parser.add_argument(
+ "pptx",
+ nargs="?",
+ default=str(default_pptx),
+ help="path to pptx file (defaults to demo/office_docs/pptx_01.pptx relative to project root)",
+ )
+ parser.add_argument(
+ "--output-images",
+ help="directory to write image outputs",
+ default="./output_images",
+ )
+ args = parser.parse_args()
+
+ pptx_path = Path(args.pptx)
+ from mineru.data.data_reader_writer import FileBasedDataWriter
+
+ with open(pptx_path, "rb") as f:
+ file_bytes = f.read()
+ image_writer = FileBasedDataWriter(args.output_images)
+ middle_json, results = office_pptx_analyze(
+ file_bytes,
+ image_writer=image_writer,
+ )
+
+ import json
+
+ logger.info(json.dumps(middle_json, indent=2, ensure_ascii=False))
diff --git a/mineru/backend/office/xlsx_analyze.py b/mineru/backend/office/xlsx_analyze.py
new file mode 100644
index 000000000..3aae75d79
--- /dev/null
+++ b/mineru/backend/office/xlsx_analyze.py
@@ -0,0 +1,71 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import time
+from io import BytesIO
+
+from loguru import logger
+from mineru.backend.office.model_output_to_middle_json import result_to_middle_json
+
+from mineru.model.xlsx.main import convert_binary
+
+
+def office_xlsx_analyze(file_bytes, image_writer=None):
+ infer_start = time.time()
+
+ file_stream = BytesIO(file_bytes)
+ results = convert_binary(file_stream)
+
+ infer_time = round(time.time() - infer_start, 2)
+ safe_time = max(infer_time, 0.01)
+ logger.debug(
+ f"infer finished, cost: {infer_time}, speed: {round(len(results) / safe_time, 3)} page/s"
+ )
+
+ middle_json = result_to_middle_json(
+ results,
+ image_writer,
+ )
+
+ return middle_json, results
+
+
+if __name__ == "__main__":
+ # Resolve a default xlsx file relative to this script so the example
+ # works no matter what the current working directory is when the
+ # module is executed. Allow the user to override the path via a
+ # command-line argument for even greater flexibility.
+ from pathlib import Path
+ import argparse
+
+ script_root = Path(__file__).resolve().parent.parent.parent.parent
+ default_xlsx = script_root / "demo" / "office_docs" / "xlsx_01.xlsx"
+
+ parser = argparse.ArgumentParser(
+ description="Quick demo runner for office_xlsx_analyze"
+ )
+ parser.add_argument(
+ "xlsx",
+ nargs="?",
+ default=str(default_xlsx),
+ help="path to xlsx file (defaults to demo/office_docs/xlsx_01.xlsx relative to project root)",
+ )
+ parser.add_argument(
+ "--output-images",
+ help="directory to write image outputs",
+ default="./output_images",
+ )
+ args = parser.parse_args()
+
+ xlsx_path = Path(args.xlsx)
+ from mineru.data.data_reader_writer import FileBasedDataWriter
+
+ with open(xlsx_path, "rb") as f:
+ file_bytes = f.read()
+ image_writer = FileBasedDataWriter(args.output_images)
+ middle_json, results = office_xlsx_analyze(
+ file_bytes,
+ image_writer=image_writer,
+ )
+
+ import json
+
+ logger.info(json.dumps(middle_json, indent=2, ensure_ascii=False))
diff --git a/mineru/backend/pipeline/__init__.py b/mineru/backend/pipeline/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/pipeline/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/pipeline/batch_analyze.py b/mineru/backend/pipeline/batch_analyze.py
new file mode 100644
index 000000000..1d88eb425
--- /dev/null
+++ b/mineru/backend/pipeline/batch_analyze.py
@@ -0,0 +1,878 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import base64
+import html
+
+import cv2
+from loguru import logger
+from tqdm import tqdm
+from collections import defaultdict
+import numpy as np
+
+from .model_init import AtomModelSingleton
+from .model_list import AtomicModel
+from ...utils.config_reader import (
+ get_formula_enable,
+ get_ocr_det_mask_inline_formula_enable,
+ get_table_enable,
+)
+from ...utils.bbox_utils import normalize_to_int_bbox
+from ...utils.model_utils import crop_img, get_res_list_from_layout_res, clean_vram
+from ...utils.ocr_utils import merge_det_boxes, update_det_boxes, sorted_boxes
+from ...utils.ocr_utils import (
+ get_adjusted_mfdetrec_res,
+ get_ocr_result_list,
+ OcrConfidence,
+ get_rotate_crop_image_for_text_rec,
+)
+from ...utils.pdf_image_tools import get_crop_np_img
+
+LAYOUT_BASE_BATCH_SIZE = 1
+MFR_BASE_BATCH_SIZE = 16
+OCR_DET_BASE_BATCH_SIZE = 8
+TABLE_ORI_CLS_BATCH_SIZE = 16
+TABLE_Wired_Wireless_CLS_BATCH_SIZE = 16
+
+
+class BatchAnalyze:
+ def __init__(
+ self,
+ model_manager,
+ batch_ratio: int,
+ formula_enable,
+ table_enable,
+ enable_ocr_det_batch: bool = True,
+ table_ori_cls_batch_enabled: bool | None = None,
+ text_ocr_det_batch_enabled: bool | None = None,
+ mask_inline_formula_for_ocr_det: bool = True,
+ ):
+ self.batch_ratio = batch_ratio
+ self.formula_enable = get_formula_enable(formula_enable)
+ self.table_enable = get_table_enable(table_enable)
+ self.model_manager = model_manager
+ self.enable_ocr_det_batch = enable_ocr_det_batch
+ self.table_ori_cls_batch_enabled = (
+ enable_ocr_det_batch if table_ori_cls_batch_enabled is None else table_ori_cls_batch_enabled
+ )
+ self.text_ocr_det_batch_enabled = (
+ enable_ocr_det_batch if text_ocr_det_batch_enabled is None else text_ocr_det_batch_enabled
+ )
+ self.mask_inline_formula_for_ocr_det = (
+ get_ocr_det_mask_inline_formula_enable(mask_inline_formula_for_ocr_det)
+ )
+
+ @staticmethod
+ def _apply_mask_boxes_to_image(
+ bgr_image: np.ndarray,
+ mask_boxes: list[dict] | None,
+ ) -> np.ndarray:
+ if not mask_boxes:
+ return bgr_image
+
+ masked_image = bgr_image.copy()
+ image_h, image_w = masked_image.shape[:2]
+ for mask_box in mask_boxes:
+ bbox = mask_box.get("bbox")
+ if bbox is None:
+ continue
+
+ int_bbox = normalize_to_int_bbox(bbox, image_size=(image_h, image_w))
+ if int_bbox is None:
+ continue
+
+ x0, y0, x1, y1 = int_bbox
+ masked_image[y0:y1, x0:x1] = 255
+
+ return masked_image
+
+ def _get_masked_det_image(
+ self,
+ bgr_image: np.ndarray,
+ mask_boxes: list[dict] | None,
+ ) -> np.ndarray:
+ if not self.mask_inline_formula_for_ocr_det:
+ return bgr_image
+ return self._apply_mask_boxes_to_image(bgr_image, mask_boxes)
+
+ @staticmethod
+ def _prune_empty_ocr_text_blocks(layout_res: list[dict], ocr_enable: bool) -> None:
+ if not ocr_enable or not layout_res:
+ return
+
+ def keep_item(item: dict) -> bool:
+ if item.get("label") != "ocr_text":
+ return True
+
+ text = item.get("text")
+ if isinstance(text, str):
+ return bool(text.strip())
+ return bool(text)
+
+ layout_res[:] = [item for item in layout_res if keep_item(item)]
+
+ @staticmethod
+ def _bbox_center(bbox: list[float]) -> tuple[float, float]:
+ return (float(bbox[0] + bbox[2]) / 2.0, float(bbox[1] + bbox[3]) / 2.0)
+
+ @staticmethod
+ def _is_point_in_bbox(point: tuple[float, float], bbox: list[float]) -> bool:
+ x, y = point
+ return bbox[0] <= x <= bbox[2] and bbox[1] <= y <= bbox[3]
+
+ @staticmethod
+ def _bbox_intersection(bbox1: list[float], bbox2: list[float]) -> list[float] | None:
+ x0 = max(float(bbox1[0]), float(bbox2[0]))
+ y0 = max(float(bbox1[1]), float(bbox2[1]))
+ x1 = min(float(bbox1[2]), float(bbox2[2]))
+ y1 = min(float(bbox1[3]), float(bbox2[3]))
+ if x1 <= x0 or y1 <= y0:
+ return None
+ return [x0, y0, x1, y1]
+
+ @classmethod
+ def _bbox_intersection_area(cls, bbox1: list[float], bbox2: list[float]) -> float:
+ overlap_bbox = cls._bbox_intersection(bbox1, bbox2)
+ if overlap_bbox is None:
+ return 0.0
+ return float(overlap_bbox[2] - overlap_bbox[0]) * float(overlap_bbox[3] - overlap_bbox[1])
+
+ @staticmethod
+ def _bbox_to_relative_bbox(bbox: list[float], base_bbox: list[float]) -> list[float]:
+ return [
+ float(bbox[0]) - float(base_bbox[0]),
+ float(bbox[1]) - float(base_bbox[1]),
+ float(bbox[2]) - float(base_bbox[0]),
+ float(bbox[3]) - float(base_bbox[1]),
+ ]
+
+ @staticmethod
+ def _bbox_to_quad(bbox: list[float]) -> np.ndarray:
+ x0, y0, x1, y1 = bbox
+ return np.asarray([[x0, y0], [x1, y0], [x1, y1], [x0, y1]], dtype=np.float32)
+
+ @staticmethod
+ def _encode_table_inline_image(np_img: np.ndarray, bbox: list[float]) -> str:
+ image_h, image_w = np_img.shape[:2]
+ image_bbox = normalize_to_int_bbox(bbox, image_size=(image_h, image_w))
+ if image_bbox is None:
+ return ""
+
+ x0, y0, x1, y1 = image_bbox
+ if x1 <= x0 or y1 <= y0:
+ return ""
+
+ crop_rgb = np_img[y0:y1, x0:x1]
+ if crop_rgb.size == 0:
+ return ""
+
+ crop_bgr = cv2.cvtColor(crop_rgb, cv2.COLOR_RGB2BGR)
+ success, encoded = cv2.imencode(".jpg", crop_bgr)
+ if not success:
+ return ""
+
+ b64_str = base64.b64encode(encoded.tobytes()).decode("ascii")
+ return f"data:image/jpg;base64,{b64_str}"
+
+ @staticmethod
+ def _get_virtual_image_bbox(bbox: list[float], box_size: float = 10.0) -> list[float]:
+ center_x, center_y = BatchAnalyze._bbox_center(bbox)
+ half_size = box_size / 2.0
+ return [
+ center_x - half_size,
+ center_y - half_size,
+ center_x + half_size,
+ center_y + half_size,
+ ]
+
+ @staticmethod
+ def _table_supports_inline_objects(table_res_dict: dict) -> bool:
+ return str(table_res_dict.get("rotate_label", "0")) == "0"
+
+ @staticmethod
+ def _sort_table_ocr_result(ocr_result: list[list]) -> None:
+ if not ocr_result:
+ return
+
+ sorted_result = sorted(
+ ocr_result,
+ key=lambda item: (float(np.asarray(item[0])[0][1]), float(np.asarray(item[0])[0][0])),
+ )
+
+ for i in range(len(sorted_result) - 1):
+ for j in range(i, -1, -1):
+ cur_box = np.asarray(sorted_result[j][0], dtype=np.float32)
+ next_box = np.asarray(sorted_result[j + 1][0], dtype=np.float32)
+ if (
+ abs(float(next_box[0][1]) - float(cur_box[0][1])) < 10
+ and float(next_box[0][0]) < float(cur_box[0][0])
+ ):
+ sorted_result[j], sorted_result[j + 1] = sorted_result[j + 1], sorted_result[j]
+ else:
+ break
+
+ ocr_result[:] = sorted_result
+
+ @classmethod
+ def _extract_table_inline_objects(
+ cls,
+ layout_res: list[dict],
+ np_img: np.ndarray,
+ formula_enable: bool,
+ ) -> dict[int, list[dict]]:
+ image_h, image_w = np_img.shape[:2]
+ image_size = (image_h, image_w)
+
+ tables = []
+ for res in layout_res:
+ if res.get("label") != "table":
+ continue
+ table_bbox = normalize_to_int_bbox(res.get("bbox"), image_size=image_size)
+ if table_bbox is None:
+ continue
+ tables.append((res, table_bbox))
+
+ if not tables:
+ return {}
+
+ table_inline_objects = {id(table_res): [] for table_res, _ in tables}
+ remove_ids = set()
+ candidate_labels = {"image"}
+ if formula_enable:
+ candidate_labels.update({"inline_formula", "display_formula"})
+
+ for layout_item in layout_res:
+ label = layout_item.get("label")
+ if label not in candidate_labels:
+ continue
+
+ item_bbox = normalize_to_int_bbox(layout_item.get("bbox"), image_size=image_size)
+ if item_bbox is None:
+ continue
+
+ item_center = cls._bbox_center(item_bbox)
+ matched_tables = []
+ for table_res, table_bbox in tables:
+ if not cls._is_point_in_bbox(item_center, table_bbox):
+ continue
+ overlap_area = cls._bbox_intersection_area(item_bbox, table_bbox)
+ matched_tables.append((overlap_area, table_res, table_bbox))
+
+ if not matched_tables:
+ continue
+
+ matched_tables.sort(key=lambda item: item[0], reverse=True)
+ _, table_res, table_bbox = matched_tables[0]
+ overlap_bbox = cls._bbox_intersection(item_bbox, table_bbox)
+ if overlap_bbox is None:
+ continue
+
+ rel_overlap_bbox = cls._bbox_to_relative_bbox(overlap_bbox, table_bbox)
+ score = float(layout_item.get("score", 1.0))
+
+ if label == "image":
+ image_src = cls._encode_table_inline_image(np_img, item_bbox)
+ if not image_src:
+ continue
+ content = f'
'
+ token_bbox = cls._get_virtual_image_bbox(rel_overlap_bbox)
+ kind = "image"
+ else:
+ latex = layout_item.get("latex", "")
+ if not latex:
+ continue
+ content = f"{html.escape(latex)} "
+ token_bbox = rel_overlap_bbox
+ kind = "formula"
+
+ table_inline_objects[id(table_res)].append(
+ {
+ "kind": kind,
+ "page_bbox": item_bbox,
+ "table_rel_mask_bbox": rel_overlap_bbox,
+ "table_token_bbox": token_bbox,
+ "content": content,
+ "score": score,
+ }
+ )
+ remove_ids.add(id(layout_item))
+
+ if remove_ids:
+ layout_res[:] = [item for item in layout_res if id(item) not in remove_ids]
+
+ return table_inline_objects
+
+
+ def __call__(self, images_with_extra_info: list) -> list:
+ if len(images_with_extra_info) == 0:
+ return []
+
+ images_layout_res = []
+
+ self.model = self.model_manager.get_model(
+ lang=None,
+ formula_enable=self.formula_enable,
+ table_enable=self.table_enable,
+ )
+ atom_model_manager = AtomModelSingleton()
+
+ pil_images = [image for image, _, _ in images_with_extra_info]
+
+ np_images = [np.asarray(image) for image, _, _ in images_with_extra_info]
+
+ # pp-doclayout_v2
+ images_layout_res += self.model.layout_model.batch_predict(
+ pil_images,
+ batch_size=min(8, self.batch_ratio * LAYOUT_BASE_BATCH_SIZE)
+ )
+ # 清理显存
+ clean_vram(self.model.device, vram_threshold=8)
+
+ if self.formula_enable:
+ images_mfd_res = []
+ for layout_res in images_layout_res:
+ page_formula_res = []
+ for res in layout_res:
+ if res.get("label") in ["display_formula", "inline_formula"]:
+ res.setdefault("latex", "")
+ page_formula_res.append(res)
+ images_mfd_res.append(page_formula_res)
+
+ # 公式识别
+ images_formula_list = self.model.mfr_model.batch_predict(
+ images_mfd_res,
+ np_images,
+ batch_size=self.batch_ratio * MFR_BASE_BATCH_SIZE,
+ )
+ mfr_count = 0
+ for image_index in range(len(np_images)):
+ mfr_count += len(images_formula_list[image_index])
+ for formula_res, formula_with_latex in zip(
+ images_mfd_res[image_index], images_formula_list[image_index]
+ ):
+ formula_res["latex"] = formula_with_latex.get("latex", "")
+
+ # 清理显存
+ clean_vram(self.model.device, vram_threshold=8)
+
+ else:
+ for layout_res in images_layout_res:
+ # 移除所有的"inline_formula"
+ layout_res[:] = [res for res in layout_res if res.get("label") != "inline_formula"]
+
+
+
+ ocr_res_list_all_page = []
+ table_res_list_all_page = []
+ for index in range(len(np_images)):
+ _, ocr_enable, _lang = images_with_extra_info[index]
+ layout_res = images_layout_res[index]
+ np_img = np_images[index]
+ table_inline_objects = (
+ self._extract_table_inline_objects(
+ layout_res,
+ np_img,
+ formula_enable=self.formula_enable,
+ )
+ if self.table_enable
+ else {}
+ )
+
+ ocr_res_list, table_res_list, single_page_mfdetrec_res = (
+ get_res_list_from_layout_res(layout_res)
+ )
+
+ ocr_res_list_all_page.append({'ocr_res_list':ocr_res_list,
+ 'lang':_lang,
+ 'ocr_enable':ocr_enable,
+ 'np_img':np_img,
+ 'single_page_mfdetrec_res':single_page_mfdetrec_res,
+ 'layout_res':layout_res,
+ })
+
+ for table_res in table_res_list:
+ def get_crop_table_img(scale):
+ bbox = normalize_to_int_bbox(
+ [float(v) / float(scale) for v in table_res["bbox"]]
+ )
+ if bbox is None:
+ return np_img[0:0, 0:0]
+ return get_crop_np_img(bbox, np_img, scale=scale)
+
+ wireless_table_img = get_crop_table_img(scale = 1)
+ wired_table_img = get_crop_table_img(scale = 10/3)
+ table_page_bbox = normalize_to_int_bbox(
+ table_res.get("bbox"),
+ image_size=np_img.shape[:2],
+ ) or [0, 0, 0, 0]
+
+ table_res_list_all_page.append({'table_res':table_res,
+ 'lang':_lang,
+ 'table_img':wireless_table_img,
+ 'wired_table_img':wired_table_img,
+ 'table_page_bbox':table_page_bbox,
+ 'table_inline_objects':table_inline_objects.get(id(table_res), []),
+ })
+
+ # 表格识别 table recognition
+ if self.table_enable:
+
+ # 图片旋转批量处理
+ img_orientation_cls_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.ImgOrientationCls,
+ )
+ try:
+ if self.table_ori_cls_batch_enabled:
+ img_orientation_cls_model.batch_predict(table_res_list_all_page,
+ det_batch_size=self.batch_ratio * OCR_DET_BASE_BATCH_SIZE,
+ batch_size=TABLE_ORI_CLS_BATCH_SIZE)
+ else:
+ for table_res in table_res_list_all_page:
+ rotate_label = img_orientation_cls_model.predict(table_res['table_img'])
+ img_orientation_cls_model.img_rotate(table_res, rotate_label)
+ except Exception as e:
+ logger.warning(
+ f"Image orientation classification failed: {e}, using original image"
+ )
+
+ # 表格分类
+ table_cls_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.TableCls,
+ )
+ try:
+ table_cls_model.batch_predict(table_res_list_all_page,
+ batch_size=TABLE_Wired_Wireless_CLS_BATCH_SIZE)
+ except Exception as e:
+ logger.warning(
+ f"Table classification failed: {e}, using default model"
+ )
+
+ # OCR det 过程，顺序执行
+ rec_img_lang_group = defaultdict(list)
+ det_ocr_engine = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.5,
+ det_db_unclip_ratio=1.6,
+ enable_merge_det_boxes=False,
+ )
+ for index, table_res_dict in enumerate(
+ tqdm(table_res_list_all_page, desc="Table-ocr det")
+ ):
+ bgr_image = cv2.cvtColor(table_res_dict["table_img"], cv2.COLOR_RGB2BGR)
+ table_inline_objects = (
+ table_res_dict.get("table_inline_objects", [])
+ if self._table_supports_inline_objects(table_res_dict)
+ else []
+ )
+ inline_mask_boxes = [
+ {"bbox": inline_object["table_rel_mask_bbox"]}
+ for inline_object in table_inline_objects
+ ]
+ formula_mask_boxes = [
+ {"bbox": inline_object["table_rel_mask_bbox"]}
+ for inline_object in table_inline_objects
+ if inline_object["kind"] == "formula"
+ ]
+ det_image = (
+ self._apply_mask_boxes_to_image(bgr_image, inline_mask_boxes)
+ if inline_mask_boxes
+ else bgr_image
+ )
+ ocr_result = det_ocr_engine.ocr(det_image, rec=False)[0]
+ if ocr_result and formula_mask_boxes:
+ ocr_result = update_det_boxes(ocr_result, formula_mask_boxes)
+ if ocr_result:
+ ocr_result = sorted_boxes(ocr_result)
+ # 构造需要 OCR 识别的图片字典，包括cropped_img, dt_box, table_id，并按照语言进行分组
+ for dt_box in ocr_result:
+ rec_img_lang_group[table_res_dict["lang"]].append(
+ {
+ "cropped_img": get_rotate_crop_image_for_text_rec(
+ bgr_image, np.asarray(dt_box, dtype=np.float32)
+ ),
+ "dt_box": np.asarray(dt_box, dtype=np.float32),
+ "table_id": index,
+ }
+ )
+
+ # OCR rec，按照语言分批处理
+ for _lang, rec_img_list in rec_img_lang_group.items():
+ if not rec_img_list:
+ continue
+ ocr_engine = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.5,
+ det_db_unclip_ratio=1.6,
+ lang=_lang,
+ enable_merge_det_boxes=False,
+ )
+ cropped_img_list = [item["cropped_img"] for item in rec_img_list]
+ ocr_res_list = ocr_engine.ocr(cropped_img_list, det=False, tqdm_enable=True, tqdm_desc=f"Table-ocr rec {_lang}")[0]
+ # 按照 table_id 将识别结果进行回填
+ for img_dict, ocr_res in zip(rec_img_list, ocr_res_list):
+ if table_res_list_all_page[img_dict["table_id"]].get("ocr_result"):
+ table_res_list_all_page[img_dict["table_id"]]["ocr_result"].append(
+ [img_dict["dt_box"], html.escape(ocr_res[0]), ocr_res[1]]
+ )
+ else:
+ table_res_list_all_page[img_dict["table_id"]]["ocr_result"] = [
+ [img_dict["dt_box"], html.escape(ocr_res[0]), ocr_res[1]]
+ ]
+
+ # 先对所有表格使用无线表格模型，然后对分类为有线的表格使用有线表格模型
+ for table_res_dict in table_res_list_all_page:
+ if not self._table_supports_inline_objects(table_res_dict):
+ continue
+
+ table_inline_objects = table_res_dict.get("table_inline_objects", [])
+ if not table_inline_objects:
+ continue
+
+ table_ocr_result = table_res_dict.setdefault("ocr_result", [])
+ for inline_object in table_inline_objects:
+ table_ocr_result.append(
+ [
+ self._bbox_to_quad(inline_object["table_token_bbox"]),
+ inline_object["content"],
+ inline_object["score"],
+ ]
+ )
+
+ self._sort_table_ocr_result(table_ocr_result)
+
+ wireless_table_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.WirelessTable,
+ )
+ wireless_table_model.batch_predict(table_res_list_all_page)
+
+ # 单独拿出有线表格进行预测
+ wired_table_res_list = []
+ for table_res_dict in table_res_list_all_page:
+ # logger.debug(f"Table classification result: {table_res_dict["table_res"]["cls_label"]} with confidence {table_res_dict["table_res"]["cls_score"]}")
+ if (
+ (table_res_dict["table_res"]["cls_label"] == AtomicModel.WirelessTable and table_res_dict["table_res"]["cls_score"] < 0.9)
+ or table_res_dict["table_res"]["cls_label"] == AtomicModel.WiredTable
+ ):
+ wired_table_res_list.append(table_res_dict)
+ del table_res_dict["table_res"]["cls_label"]
+ del table_res_dict["table_res"]["cls_score"]
+ if wired_table_res_list:
+ for table_res_dict in tqdm(
+ wired_table_res_list, desc="Table-wired Predict"
+ ):
+ if not table_res_dict.get("ocr_result", None):
+ continue
+
+ wired_table_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.WiredTable,
+ lang=table_res_dict["lang"],
+ )
+ table_res_dict["table_res"]["html"] = wired_table_model.predict(
+ table_res_dict["wired_table_img"],
+ table_res_dict["ocr_result"],
+ table_res_dict["table_res"].get("html", None)
+ )
+
+ # 表格格式清理
+ for table_res_dict in table_res_list_all_page:
+ html_code = table_res_dict["table_res"].get("html", "") or ""
+
+ # 检查html_code是否包含''和'
'
+ if "" in html_code and "
" in html_code:
+ # 选用到
的内容，放入table_res_dict['table_res']['html']
+ start_index = html_code.find("")
+ end_index = html_code.rfind("
") + len("")
+ table_res_dict["table_res"]["html"] = html_code[start_index:end_index]
+
+
+ # OCR det
+ if self.text_ocr_det_batch_enabled:
+ # 批处理模式 - 按语言和分辨率分组
+ # 收集所有需要OCR检测的裁剪图像
+ all_cropped_images_info = []
+
+ for ocr_res_list_dict in ocr_res_list_all_page:
+ _lang = ocr_res_list_dict['lang']
+
+ for res in ocr_res_list_dict['ocr_res_list']:
+ new_image, useful_list = crop_img(
+ res, ocr_res_list_dict['np_img'], crop_paste_x=50, crop_paste_y=50
+ )
+ adjusted_mfdetrec_res = get_adjusted_mfdetrec_res(
+ ocr_res_list_dict['single_page_mfdetrec_res'], useful_list
+ )
+
+ # BGR转换
+ bgr_image = cv2.cvtColor(new_image, cv2.COLOR_RGB2BGR)
+ det_image = self._get_masked_det_image(
+ bgr_image,
+ adjusted_mfdetrec_res,
+ )
+
+ all_cropped_images_info.append((
+ bgr_image,
+ det_image,
+ useful_list,
+ ocr_res_list_dict,
+ adjusted_mfdetrec_res,
+ _lang,
+ ))
+
+ # 按语言分组
+ lang_groups = defaultdict(list)
+ for crop_info in all_cropped_images_info:
+ lang = crop_info[5]
+ lang_groups[lang].append(crop_info)
+
+ # 对每种语言按分辨率分组并批处理
+ for lang, lang_crop_list in lang_groups.items():
+ if not lang_crop_list:
+ continue
+
+ # logger.info(f"Processing OCR detection for language {lang} with {len(lang_crop_list)} images")
+
+ # 获取OCR模型
+ ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.3,
+ lang=lang
+ )
+
+ # 按分辨率分组并同时完成padding
+ # RESOLUTION_GROUP_STRIDE = 32
+ RESOLUTION_GROUP_STRIDE = 64
+
+ resolution_groups = defaultdict(list)
+ for crop_info in lang_crop_list:
+ cropped_img = crop_info[1]
+ h, w = cropped_img.shape[:2]
+ # 直接计算目标尺寸并用作分组键
+ target_h = ((h + RESOLUTION_GROUP_STRIDE - 1) // RESOLUTION_GROUP_STRIDE) * RESOLUTION_GROUP_STRIDE
+ target_w = ((w + RESOLUTION_GROUP_STRIDE - 1) // RESOLUTION_GROUP_STRIDE) * RESOLUTION_GROUP_STRIDE
+ group_key = (target_h, target_w)
+ resolution_groups[group_key].append(crop_info)
+
+ # 对每个分辨率组进行批处理
+ for (target_h, target_w), group_crops in tqdm(resolution_groups.items(), desc=f"OCR-det {lang}"):
+ # 对所有图像进行padding到统一尺寸
+ batch_images = []
+ for crop_info in group_crops:
+ img = crop_info[1]
+ h, w = img.shape[:2]
+ # 创建目标尺寸的白色背景
+ padded_img = np.ones((target_h, target_w, 3), dtype=np.uint8) * 255
+ padded_img[:h, :w] = img
+ batch_images.append(padded_img)
+
+ # 批处理检测
+ det_batch_size = min(len(batch_images), self.batch_ratio * OCR_DET_BASE_BATCH_SIZE)
+ batch_results = ocr_model.text_detector.batch_predict(batch_images, det_batch_size)
+
+ # 处理批处理结果
+ for crop_info, (dt_boxes, _) in zip(group_crops, batch_results):
+ (
+ bgr_image,
+ _det_image,
+ useful_list,
+ ocr_res_list_dict,
+ adjusted_mfdetrec_res,
+ _lang,
+ ) = crop_info
+
+ if dt_boxes is not None and len(dt_boxes) > 0:
+ # 处理检测框
+ dt_boxes_sorted = sorted_boxes(dt_boxes)
+ dt_boxes_merged = merge_det_boxes(dt_boxes_sorted) if dt_boxes_sorted else []
+
+ # 根据公式位置更新检测框
+ dt_boxes_final = (update_det_boxes(dt_boxes_merged, adjusted_mfdetrec_res)
+ if dt_boxes_merged and adjusted_mfdetrec_res
+ else dt_boxes_merged)
+
+ if dt_boxes_final:
+ ocr_res = [box.tolist() if hasattr(box, 'tolist') else box for box in dt_boxes_final]
+ ocr_result_list = get_ocr_result_list(
+ ocr_res,
+ useful_list,
+ ocr_res_list_dict['ocr_enable'],
+ bgr_image,
+ _lang,
+ )
+ ocr_res_list_dict['layout_res'].extend(ocr_result_list)
+
+ # 清理显存
+ clean_vram(self.model.device, vram_threshold=8)
+
+ else:
+ # 原始单张处理模式
+ for ocr_res_list_dict in tqdm(ocr_res_list_all_page, desc="OCR-det Predict"):
+ # Process each area that requires OCR processing
+ _lang = ocr_res_list_dict['lang']
+ # Get OCR results for this language's images
+ ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ ocr_show_log=False,
+ det_db_box_thresh=0.3,
+ lang=_lang
+ )
+ for res in ocr_res_list_dict['ocr_res_list']:
+ new_image, useful_list = crop_img(
+ res, ocr_res_list_dict['np_img'], crop_paste_x=50, crop_paste_y=50
+ )
+ adjusted_mfdetrec_res = get_adjusted_mfdetrec_res(
+ ocr_res_list_dict['single_page_mfdetrec_res'], useful_list
+ )
+ # OCR-det
+ bgr_image = cv2.cvtColor(new_image, cv2.COLOR_RGB2BGR)
+ det_image = self._get_masked_det_image(
+ bgr_image,
+ adjusted_mfdetrec_res,
+ )
+ ocr_res = ocr_model.ocr(
+ det_image, mfd_res=adjusted_mfdetrec_res, rec=False
+ )[0]
+
+ # Integration results
+ if ocr_res:
+ ocr_result_list = get_ocr_result_list(
+ ocr_res,
+ useful_list,
+ ocr_res_list_dict['ocr_enable'],
+ bgr_image,
+ _lang,
+ )
+
+ ocr_res_list_dict['layout_res'].extend(ocr_result_list)
+
+ # OCR rec
+ # Create dictionaries to store items by language
+ need_ocr_lists_by_lang = {} # Dict of lists for each language
+ img_crop_lists_by_lang = {} # Dict of lists for each language
+
+ for layout_res in images_layout_res:
+ for layout_res_item in layout_res:
+ if not layout_res_item.get("_need_ocr_rec"):
+ continue
+ if 'np_img' in layout_res_item and 'lang' in layout_res_item:
+ lang = layout_res_item['lang']
+
+ # Initialize lists for this language if not exist
+ if lang not in need_ocr_lists_by_lang:
+ need_ocr_lists_by_lang[lang] = []
+ img_crop_lists_by_lang[lang] = []
+
+ # Add to the appropriate language-specific lists
+ need_ocr_lists_by_lang[lang].append((layout_res, layout_res_item))
+ img_crop_lists_by_lang[lang].append(layout_res_item['np_img'])
+
+ # Remove temporary fields after collecting
+ layout_res_item.pop('np_img', None)
+ layout_res_item.pop('lang', None)
+ layout_res_item.pop('_need_ocr_rec', None)
+
+ if len(img_crop_lists_by_lang) > 0:
+
+ # Process OCR by language
+ total_processed = 0
+
+ # Process each language separately
+ for lang, img_crop_list in img_crop_lists_by_lang.items():
+ if len(img_crop_list) > 0:
+ # Get OCR results for this language's images
+
+ ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.3,
+ lang=lang
+ )
+ ocr_res_list = ocr_model.ocr(img_crop_list, det=False, tqdm_enable=True)[0]
+
+ # Verify we have matching counts
+ assert len(ocr_res_list) == len(
+ need_ocr_lists_by_lang[lang]), f'ocr_res_list: {len(ocr_res_list)}, need_ocr_list: {len(need_ocr_lists_by_lang[lang])} for lang: {lang}'
+
+ items_to_remove = []
+ # Process OCR results for this language
+ for index, (page_layout_res, layout_res_item) in enumerate(need_ocr_lists_by_lang[lang]):
+ ocr_text, ocr_score = ocr_res_list[index]
+ layout_res_item['text'] = ocr_text
+ layout_res_item['score'] = float(f"{ocr_score:.3f}")
+ should_remove = False
+ if ocr_score < OcrConfidence.min_confidence:
+ should_remove = True
+ else:
+ layout_res_bbox = layout_res_item['bbox']
+ layout_res_width = layout_res_bbox[2] - layout_res_bbox[0]
+ layout_res_height = layout_res_bbox[3] - layout_res_bbox[1]
+ if (
+ ocr_text in [
+ '（204号', '（20', '（2', '（2号', '（20号', '号', '（204',
+ '(cid:)', '(ci:)', '(cd:1)', 'cd:)', 'c)', '(cd:)', 'c', 'id:)',
+ ':)', '√:)', '√i:)', '−i:)', '−:', 'i:)',
+ ]
+ and ocr_score < 0.8
+ and layout_res_width < layout_res_height
+ ):
+ should_remove = True
+
+ if should_remove:
+ items_to_remove.append((page_layout_res, layout_res_item))
+
+ for page_layout_res, layout_res_item in items_to_remove:
+ if layout_res_item in page_layout_res:
+ page_layout_res.remove(layout_res_item)
+
+ total_processed += len(img_crop_list)
+
+ seal_ocr_items = []
+ for ocr_res_list_dict in ocr_res_list_all_page:
+ for layout_res_item in ocr_res_list_dict['layout_res']:
+ if layout_res_item.get("label") == "seal":
+ seal_ocr_items.append((ocr_res_list_dict, layout_res_item))
+
+ seal_ocr_model = None
+ for ocr_res_list_dict, layout_res_item in tqdm(seal_ocr_items, desc="Seal Predict"):
+ np_img = ocr_res_list_dict['np_img']
+ image_h, image_w = np_img.shape[:2]
+ layout_res_item["text"] = ""
+ seal_bbox = normalize_to_int_bbox(
+ layout_res_item.get("bbox"),
+ image_size=(image_h, image_w),
+ )
+ if seal_bbox is None:
+ continue
+
+ x0, y0, x1, y1 = seal_bbox
+ seal_crop_rgb = np_img[y0:y1, x0:x1]
+ if seal_crop_rgb.size == 0:
+ continue
+
+ if seal_ocr_model is None:
+ seal_ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ lang="seal",
+ )
+
+ seal_crop_bgr = cv2.cvtColor(seal_crop_rgb, cv2.COLOR_RGB2BGR)
+ seal_ocr_res = seal_ocr_model.ocr(seal_crop_bgr, det=True, rec=True)[0]
+ if not seal_ocr_res:
+ continue
+
+ seal_texts = []
+ for seal_item in seal_ocr_res:
+ if not seal_item or len(seal_item) != 2:
+ continue
+ rec_result = seal_item[1]
+ if not rec_result or len(rec_result) < 1:
+ continue
+ rec_text = rec_result[0]
+ if rec_text:
+ seal_texts.append(rec_text)
+
+ layout_res_item["text"] = seal_texts
+
+ for ocr_res_list_dict in ocr_res_list_all_page:
+ self._prune_empty_ocr_text_blocks(
+ ocr_res_list_dict["layout_res"],
+ ocr_res_list_dict["ocr_enable"],
+ )
+
+ return images_layout_res
diff --git a/mineru/backend/pipeline/model_init.py b/mineru/backend/pipeline/model_init.py
new file mode 100644
index 000000000..3c76a6596
--- /dev/null
+++ b/mineru/backend/pipeline/model_init.py
@@ -0,0 +1,363 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import threading
+
+import torch
+from loguru import logger
+
+from .model_list import AtomicModel
+from ...model.layout.pp_doclayoutv2 import PPDocLayoutV2LayoutModel
+from ...model.mfr.unimernet.Unimernet import UnimernetModel
+from ...model.mfr.pp_formulanet_plus_m.predict_formula import FormulaRecognizer
+from mineru.model.ocr.pytorch_paddle import PytorchPaddleOCR
+from ...model.ori_cls.paddle_ori_cls import PaddleOrientationClsModel
+from ...model.table.cls.paddle_table_cls import PaddleTableClsModel
+from ...model.table.rec.slanet_plus.main import PaddleTableModel
+from ...model.table.rec.unet_table.main import UnetTableModel
+from ...utils.config_reader import get_device
+from ...utils.enum_class import ModelPath
+from ...utils.models_download_utils import auto_download_and_get_model_root_path
+
+PIPELINE_MODEL_INIT_LOCK = threading.RLock()
+
+MFR_MODEL = os.getenv('MINERU_FORMULA_CH_SUPPORT', 'False')
+if MFR_MODEL.lower() in ['true', '1', 'yes']:
+ MFR_MODEL = "pp_formulanet_plus_m"
+elif MFR_MODEL.lower() in ['false', '0', 'no']:
+ MFR_MODEL = "unimernet_small"
+else:
+ logger.warning(f"Invalid MINERU_FORMULA_CH_SUPPORT value: {MFR_MODEL}, set to default 'False'")
+ MFR_MODEL = "unimernet_small"
+
+
+def img_orientation_cls_model_init():
+ atom_model_manager = AtomModelSingleton()
+ ocr_engine = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.5,
+ det_db_unclip_ratio=1.6,
+ lang="ch_lite",
+ enable_merge_det_boxes=False
+ )
+ cls_model = PaddleOrientationClsModel(ocr_engine)
+ return cls_model
+
+
+def table_cls_model_init():
+ return PaddleTableClsModel()
+
+
+def wired_table_model_init(lang=None):
+ atom_model_manager = AtomModelSingleton()
+ ocr_engine = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.5,
+ det_db_unclip_ratio=1.6,
+ lang=lang,
+ enable_merge_det_boxes=False
+ )
+ table_model = UnetTableModel(ocr_engine)
+ return table_model
+
+
+def wireless_table_model_init(lang=None):
+ atom_model_manager = AtomModelSingleton()
+ ocr_engine = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.5,
+ det_db_unclip_ratio=1.6,
+ lang=lang,
+ enable_merge_det_boxes=False
+ )
+ table_model = PaddleTableModel(ocr_engine)
+ return table_model
+
+
+def mfr_model_init(weight_dir, device='cpu'):
+ if MFR_MODEL == "unimernet_small":
+ mfr_model = UnimernetModel(weight_dir, device)
+ elif MFR_MODEL == "pp_formulanet_plus_m":
+ mfr_model = FormulaRecognizer(weight_dir, device)
+ else:
+ logger.error('MFR model name not allow')
+ exit(1)
+ return mfr_model
+
+
+def pp_doclayout_v2_model_init(weight, device='cpu'):
+ if str(device).startswith('npu'):
+ device = torch.device(device)
+ model = PPDocLayoutV2LayoutModel(weight, device)
+ return model
+
+def ocr_model_init(det_db_box_thresh=0.3,
+ lang=None,
+ det_db_unclip_ratio=1.8,
+ enable_merge_det_boxes=True
+ ):
+ if lang is not None and lang != '':
+ model = PytorchPaddleOCR(
+ det_db_box_thresh=det_db_box_thresh,
+ lang=lang,
+ use_dilation=True,
+ det_db_unclip_ratio=det_db_unclip_ratio,
+ enable_merge_det_boxes=enable_merge_det_boxes,
+ )
+ else:
+ model = PytorchPaddleOCR(
+ det_db_box_thresh=det_db_box_thresh,
+ use_dilation=True,
+ det_db_unclip_ratio=det_db_unclip_ratio,
+ enable_merge_det_boxes=enable_merge_det_boxes,
+ )
+ return model
+
+
+class AtomModelSingleton:
+ _instance = None
+ _models = {}
+ _lock = PIPELINE_MODEL_INIT_LOCK
+
+ def __new__(cls, *args, **kwargs):
+ with cls._lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ return cls._instance
+
+ def get_atom_model(self, atom_model_name: str, **kwargs):
+
+ lang = kwargs.get('lang', None)
+
+ if atom_model_name in [AtomicModel.WiredTable, AtomicModel.WirelessTable]:
+ key = (
+ atom_model_name,
+ lang
+ )
+ elif atom_model_name in [AtomicModel.OCR]:
+ key = (
+ atom_model_name,
+ kwargs.get('det_db_box_thresh', 0.3),
+ lang,
+ kwargs.get('det_db_unclip_ratio', 1.8),
+ kwargs.get('enable_merge_det_boxes', True)
+ )
+ elif atom_model_name in [AtomicModel.Layout, AtomicModel.MFR]:
+ key = (
+ atom_model_name,
+ kwargs.get('device'),
+ )
+ else:
+ key = atom_model_name
+
+ with self._lock:
+ if key not in self._models:
+ self._models[key] = atom_model_init(model_name=atom_model_name, **kwargs)
+ return self._models[key]
+
+def atom_model_init(model_name: str, **kwargs):
+ atom_model = None
+ if model_name == AtomicModel.Layout:
+ atom_model = pp_doclayout_v2_model_init(
+ kwargs.get('pp_doclayout_v2_weights'),
+ kwargs.get('device')
+ )
+ elif model_name == AtomicModel.MFR:
+ atom_model = mfr_model_init(
+ kwargs.get('mfr_weight_dir'),
+ kwargs.get('device')
+ )
+ elif model_name == AtomicModel.OCR:
+ atom_model = ocr_model_init(
+ kwargs.get('det_db_box_thresh', 0.3),
+ kwargs.get('lang'),
+ kwargs.get('det_db_unclip_ratio', 1.8),
+ kwargs.get('enable_merge_det_boxes', True)
+ )
+ elif model_name == AtomicModel.WirelessTable:
+ atom_model = wireless_table_model_init(
+ kwargs.get('lang'),
+ )
+ elif model_name == AtomicModel.WiredTable:
+ atom_model = wired_table_model_init(
+ kwargs.get('lang'),
+ )
+ elif model_name == AtomicModel.TableCls:
+ atom_model = table_cls_model_init()
+ elif model_name == AtomicModel.ImgOrientationCls:
+ atom_model = img_orientation_cls_model_init()
+ else:
+ logger.error('model name not allow')
+ exit(1)
+
+ if atom_model is None:
+ logger.error('model init failed')
+ exit(1)
+ else:
+ return atom_model
+
+
+class MineruPipelineModel:
+ def __init__(self, **kwargs):
+ self.formula_config = kwargs.get('formula_config')
+ self.apply_formula = self.formula_config.get('enable', True)
+ self.table_config = kwargs.get('table_config')
+ self.apply_table = self.table_config.get('enable', True)
+ self.lang = kwargs.get('lang', None)
+ self.device = kwargs.get('device', 'cpu')
+ logger.info(
+ 'DocAnalysis init, this may take some times......'
+ )
+ atom_model_manager = AtomModelSingleton()
+
+ if self.apply_formula:
+ # 初始化公式解析模型
+ if MFR_MODEL == "unimernet_small":
+ mfr_model_path = ModelPath.unimernet_small
+ elif MFR_MODEL == "pp_formulanet_plus_m":
+ mfr_model_path = ModelPath.pp_formulanet_plus_m
+ else:
+ logger.error('MFR model name not allow')
+ exit(1)
+
+ self.mfr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.MFR,
+ mfr_weight_dir=str(os.path.join(auto_download_and_get_model_root_path(mfr_model_path), mfr_model_path)),
+ device=self.device,
+ )
+
+ # 初始化layout模型
+ self.layout_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.Layout,
+ pp_doclayout_v2_weights=str(
+ os.path.join(auto_download_and_get_model_root_path(ModelPath.pp_doclayout_v2), ModelPath.pp_doclayout_v2)
+ ),
+ device=self.device,
+ )
+ # 初始化ocr
+ self.ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.3,
+ lang=self.lang
+ )
+ # init table model
+ if self.apply_table:
+ self.wired_table_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.WiredTable,
+ lang=self.lang,
+ )
+ self.wireless_table_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.WirelessTable,
+ lang=self.lang,
+ )
+ self.table_cls_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.TableCls,
+ )
+ self.img_orientation_cls_model = atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.ImgOrientationCls,
+ lang=self.lang,
+ )
+
+ logger.info('DocAnalysis init done!')
+
+
+class HybridModelSingleton:
+ _instance = None
+ _models = {}
+ _lock = PIPELINE_MODEL_INIT_LOCK
+
+ def __new__(cls, *args, **kwargs):
+ with cls._lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ return cls._instance
+
+ def get_model(
+ self,
+ lang=None,
+ formula_enable=None,
+ ):
+ key = (lang, formula_enable)
+ with self._lock:
+ if key not in self._models:
+ self._models[key] = MineruHybridModel(
+ lang=lang,
+ formula_enable=formula_enable,
+ )
+ return self._models[key]
+
+def ocr_det_batch_setting():
+ import torch
+ from packaging import version
+ device_type = os.getenv("MINERU_LMDEPLOY_DEVICE", "")
+ if device_type.lower() in ["corex"]:
+ enable_ocr_det_batch = False
+ else:
+ if version.parse(torch.__version__) >= version.parse("2.8.0"):
+ os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+ enable_ocr_det_batch = True
+
+ return enable_ocr_det_batch
+
+class MineruHybridModel:
+ def __init__(
+ self,
+ device=None,
+ lang=None,
+ formula_enable=True,
+ ):
+ if device is not None:
+ self.device = device
+ else:
+ self.device = get_device()
+
+ self.lang = lang
+
+ self.enable_ocr_det_batch = ocr_det_batch_setting()
+
+ if str(self.device).startswith('npu'):
+ try:
+ import torch_npu
+ if torch_npu.npu.is_available():
+ torch_npu.npu.set_compile_mode(jit_compile=False)
+ except Exception as e:
+ raise RuntimeError(
+ "NPU is selected as device, but torch_npu is not available. "
+ "Please ensure that the torch_npu package is installed correctly."
+ ) from e
+
+ self.atom_model_manager = AtomModelSingleton()
+
+ # 初始化OCR模型
+ self.ocr_model = self.atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.OCR,
+ det_db_box_thresh=0.3,
+ lang=self.lang
+ )
+
+ if formula_enable:
+ # 初始化layout模型，用于提供行内公式检测框
+ self.layout_model = self.atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.Layout,
+ pp_doclayout_v2_weights=str(
+ os.path.join(
+ auto_download_and_get_model_root_path(ModelPath.pp_doclayout_v2),
+ ModelPath.pp_doclayout_v2,
+ )
+ ),
+ device=self.device,
+ )
+
+ # 初始化公式解析模型
+ if MFR_MODEL == "unimernet_small":
+ mfr_model_path = ModelPath.unimernet_small
+ elif MFR_MODEL == "pp_formulanet_plus_m":
+ mfr_model_path = ModelPath.pp_formulanet_plus_m
+ else:
+ logger.error('MFR model name not allow')
+ exit(1)
+
+ self.mfr_model = self.atom_model_manager.get_atom_model(
+ atom_model_name=AtomicModel.MFR,
+ mfr_weight_dir=str(os.path.join(auto_download_and_get_model_root_path(mfr_model_path), mfr_model_path)),
+ device=self.device,
+ )
diff --git a/mineru/backend/pipeline/model_json_to_middle_json.py b/mineru/backend/pipeline/model_json_to_middle_json.py
new file mode 100644
index 000000000..d99a207dc
--- /dev/null
+++ b/mineru/backend/pipeline/model_json_to_middle_json.py
@@ -0,0 +1,315 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import copy
+import os
+import time
+
+from loguru import logger
+from tqdm import tqdm
+
+from mineru.backend.utils.html_image_utils import replace_inline_table_images
+from mineru.backend.utils.runtime_utils import cross_page_table_merge
+from mineru.utils.config_reader import get_device, get_llm_aided_config
+from mineru.backend.pipeline.model_init import AtomModelSingleton
+from mineru.backend.pipeline.para_split import para_split
+from mineru.utils.char_utils import full_to_half
+from mineru.utils.cut_image import cut_image_and_table
+from mineru.utils.enum_class import ContentType, BlockType
+from mineru.utils.llm_aided import llm_aided_title
+from mineru.utils.model_utils import clean_memory
+from mineru.backend.pipeline.pipeline_magic_model import MagicModel
+from mineru.utils.ocr_utils import OcrConfidence, rotate_vertical_crop_if_needed
+from mineru.version import __version__
+from mineru.utils.hash_utils import bytes_md5
+from mineru.utils.pdfium_guard import close_pdfium_document, pdfium_guard
+
+
+def page_model_info_to_page_info(page_model_info, image_dict, page, image_writer, page_index, ocr_enable=False):
+ scale = image_dict["scale"]
+ page_pil_img = image_dict["img_pil"]
+ page_img_md5 = bytes_md5(page_pil_img.tobytes())
+ with pdfium_guard():
+ page_w, page_h = map(int, page.get_size())
+ magic_model = MagicModel(
+ page_model_info,
+ page,
+ scale,
+ page_pil_img,
+ page_w,
+ page_h,
+ ocr_enable
+ )
+
+ """从magic_model对象中获取后面会用到的区块信息"""
+ preproc_blocks = magic_model.get_preproc_blocks()
+ discarded_blocks = magic_model.get_discarded_blocks()
+ all_image_spans = magic_model.get_all_image_spans()
+
+ # 对image/table/chart/interline_equation的span截图
+ for span in all_image_spans:
+ if span["type"] in [
+ ContentType.IMAGE,
+ ContentType.TABLE,
+ ContentType.CHART,
+ ContentType.SEAL,
+ ContentType.INTERLINE_EQUATION
+ ]:
+ span = cut_image_and_table(span, page_pil_img, page_img_md5, page_index, image_writer, scale=scale)
+
+ """构造page_info"""
+ replace_inline_table_images(preproc_blocks, image_writer, page_index)
+
+ page_info = make_page_info_dict(preproc_blocks, page_index, page_w, page_h, discarded_blocks)
+
+ return page_info
+
+
+def build_page_model_info(page_layout_dets, page_index, pil_img):
+ page_info_dict = {'page_no': page_index, 'width': pil_img.width, 'height': pil_img.height}
+ return {'layout_dets': page_layout_dets, 'page_info': page_info_dict}
+
+
+def append_page_model_infos_to_middle_json(
+ middle_json,
+ page_model_infos,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=0,
+ ocr_enable=False,
+ progress_bar=None,
+):
+ for offset, (page_model_info, image_dict) in enumerate(zip(page_model_infos, images_list)):
+ page_index = page_start_index + offset
+ with pdfium_guard():
+ page = pdf_doc[page_index]
+ page_info = page_model_info_to_page_info(
+ copy.deepcopy(page_model_info),
+ image_dict,
+ page,
+ image_writer,
+ page_index,
+ ocr_enable=ocr_enable,
+ )
+ if page_info is None:
+ with pdfium_guard():
+ page_w, page_h = map(int, pdf_doc[page_index].get_size())
+ page_info = make_page_info_dict([], page_index, page_w, page_h, [])
+ middle_json["pdf_info"].append(page_info)
+ if progress_bar is not None:
+ progress_bar.update(1)
+
+
+def append_batch_results_to_middle_json(
+ middle_json,
+ batch_results,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=0,
+ ocr_enable=False,
+ model_list=None,
+ progress_bar=None,
+):
+ page_model_infos = []
+ for offset, (image_dict, page_layout_dets) in enumerate(zip(images_list, batch_results)):
+ page_index = page_start_index + offset
+ page_model_info = build_page_model_info(page_layout_dets, page_index, image_dict['img_pil'])
+ page_model_infos.append(page_model_info)
+
+ if model_list is not None:
+ model_list.extend(page_model_infos)
+
+ append_page_model_infos_to_middle_json(
+ middle_json,
+ page_model_infos,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=page_start_index,
+ ocr_enable=ocr_enable,
+ progress_bar=progress_bar,
+ )
+
+
+def _extract_text_from_block(block):
+ text_parts = []
+ for line in block.get("lines", []):
+ for span in line.get("spans", []):
+ if span.get("type") == ContentType.TEXT:
+ text_parts.append(span.get("content", ""))
+ return "".join(text_parts).strip()
+
+
+def _iter_block_spans(block):
+ for line in block.get("lines", []):
+ for span in line.get("spans", []):
+ yield span
+
+ for sub_block in block.get("blocks", []):
+ yield from _iter_block_spans(sub_block)
+
+
+def _normalize_formula_tag_content(tag_content):
+ tag_content = full_to_half(tag_content.strip())
+ if tag_content.startswith("("):
+ tag_content = tag_content[1:].strip()
+ if tag_content.endswith(")"):
+ tag_content = tag_content[:-1].strip()
+ return tag_content
+
+
+def _get_interline_equation_span(block):
+ for line in block.get("lines", []):
+ for span in line.get("spans", []):
+ if span.get("type") == ContentType.INTERLINE_EQUATION:
+ return span
+ return None
+
+
+def _append_formula_number_tag(equation_block, formula_number_block):
+ equation_span = _get_interline_equation_span(equation_block)
+ tag_content = _normalize_formula_tag_content(_extract_text_from_block(formula_number_block))
+ if equation_span is not None:
+ formula = equation_span.get("content", "")
+ equation_span["content"] = f"{formula}\\tag{{{tag_content}}}"
+
+
+def _optimize_formula_number_blocks(pdf_info_list):
+ for page_info in pdf_info_list:
+ optimized_blocks = []
+ blocks = page_info.get("preproc_blocks", [])
+ for index, block in enumerate(blocks):
+ if block.get("type") != BlockType.FORMULA_NUMBER:
+ optimized_blocks.append(block)
+ continue
+
+ prev_block = blocks[index - 1] if index > 0 else None
+ if prev_block and prev_block.get("type") == BlockType.INTERLINE_EQUATION:
+ _append_formula_number_tag(prev_block, block)
+ continue
+
+ next_block = blocks[index + 1] if index + 1 < len(blocks) else None
+ next_next_block = blocks[index + 2] if index + 2 < len(blocks) else None
+ if (
+ next_block
+ and next_block.get("type") == BlockType.INTERLINE_EQUATION
+ and (next_next_block is None or next_next_block.get("type") != BlockType.FORMULA_NUMBER)
+ ):
+ _append_formula_number_tag(next_block, block)
+ continue
+
+ block["type"] = BlockType.TEXT
+ optimized_blocks.append(block)
+
+ page_info["preproc_blocks"] = optimized_blocks
+
+
+def _apply_post_ocr(pdf_info_list, lang=None):
+ need_ocr_list = []
+ img_crop_list = []
+
+ for page_info in pdf_info_list:
+ for block in page_info.get('preproc_blocks', []):
+ for span in _iter_block_spans(block):
+ if 'np_img' in span:
+ need_ocr_list.append(span)
+ # Keep post-OCR rec aligned with the main OCR pipeline for vertical tall crops.
+ img_crop_list.append(rotate_vertical_crop_if_needed(span['np_img']))
+ span.pop('np_img')
+
+ for block in page_info.get('discarded_blocks', []):
+ for span in _iter_block_spans(block):
+ if 'np_img' in span:
+ need_ocr_list.append(span)
+ # Keep post-OCR rec aligned with the main OCR pipeline for vertical tall crops.
+ img_crop_list.append(rotate_vertical_crop_if_needed(span['np_img']))
+ span.pop('np_img')
+
+ if len(img_crop_list) == 0:
+ return
+
+ atom_model_manager = AtomModelSingleton()
+ ocr_model = atom_model_manager.get_atom_model(
+ atom_model_name='ocr',
+ det_db_box_thresh=0.3,
+ lang=lang
+ )
+ ocr_res_list = ocr_model.ocr(img_crop_list, det=False, tqdm_enable=True)[0]
+ assert len(ocr_res_list) == len(
+ need_ocr_list), f'ocr_res_list: {len(ocr_res_list)}, need_ocr_list: {len(need_ocr_list)}'
+ for index, span in enumerate(need_ocr_list):
+ ocr_text, ocr_score = ocr_res_list[index]
+ if ocr_score > OcrConfidence.min_confidence:
+ span['content'] = ocr_text
+ span['score'] = float(f"{ocr_score:.3f}")
+ else:
+ span['content'] = ''
+ span['score'] = 0.0
+
+
+def _post_block_process(pdf_info_list):
+ for page_info in pdf_info_list:
+ for block_key in ["preproc_blocks", "para_blocks"]:
+ for block in page_info.get(block_key, []):
+ block_type = block.get("type")
+ if block_type == BlockType.DOC_TITLE:
+ block["type"] = BlockType.TITLE
+ block["level"] = 1
+ elif block_type == BlockType.PARAGRAPH_TITLE:
+ block["type"] = BlockType.TITLE
+ block["level"] = 2
+ elif block_type == BlockType.VERTICAL_TEXT:
+ block["type"] = BlockType.TEXT
+
+
+def finalize_middle_json(pdf_info_list, lang=None, ocr_enable=False):
+ """Apply document-level post processing once all page_info entries are ready."""
+ _apply_post_ocr(pdf_info_list, lang=lang)
+ _optimize_formula_number_blocks(pdf_info_list)
+ para_split(pdf_info_list)
+ cross_page_table_merge(pdf_info_list)
+
+ llm_aided_config = get_llm_aided_config()
+ if llm_aided_config is not None:
+ title_aided_config = llm_aided_config.get('title_aided', None)
+ if title_aided_config is not None and title_aided_config.get('enable', False):
+ llm_aided_title_start_time = time.time()
+ llm_aided_title(pdf_info_list, title_aided_config)
+ logger.info(f'llm aided title time: {round(time.time() - llm_aided_title_start_time, 2)}')
+
+ _post_block_process(pdf_info_list)
+
+ if os.getenv('MINERU_DONOT_CLEAN_MEM') is None and len(pdf_info_list) >= 10:
+ clean_memory(get_device())
+
+
+def init_middle_json():
+ return {"pdf_info": [], "_backend": "pipeline", "_version_name": __version__}
+
+
+def result_to_middle_json(model_list, images_list, pdf_doc, image_writer, lang=None, ocr_enable=False, formula_enable=None):
+ middle_json = init_middle_json()
+ with tqdm(total=len(model_list), desc="Processing pages") as progress_bar:
+ append_page_model_infos_to_middle_json(
+ middle_json,
+ model_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ ocr_enable=ocr_enable,
+ progress_bar=progress_bar,
+ )
+
+ finalize_middle_json(middle_json["pdf_info"], lang=lang, ocr_enable=ocr_enable)
+ close_pdfium_document(pdf_doc)
+ return middle_json
+
+
+def make_page_info_dict(blocks, page_id, page_w, page_h, discarded_blocks):
+ return_dict = {
+ 'preproc_blocks': blocks,
+ 'page_idx': page_id,
+ 'page_size': [page_w, page_h],
+ 'discarded_blocks': discarded_blocks,
+ }
+ return return_dict
diff --git a/mineru/backend/pipeline/model_list.py b/mineru/backend/pipeline/model_list.py
new file mode 100644
index 000000000..6e635b59c
--- /dev/null
+++ b/mineru/backend/pipeline/model_list.py
@@ -0,0 +1,10 @@
+# Copyright (c) Opendatalab. All rights reserved.
+class AtomicModel:
+ Layout = "layout"
+ MFD = "mfd"
+ MFR = "mfr"
+ OCR = "ocr"
+ WirelessTable = "wireless_table"
+ WiredTable = "wired_table"
+ TableCls = "table_cls"
+ ImgOrientationCls = "img_ori_cls"
diff --git a/mineru/backend/pipeline/para_split.py b/mineru/backend/pipeline/para_split.py
new file mode 100644
index 000000000..0d70ad4b9
--- /dev/null
+++ b/mineru/backend/pipeline/para_split.py
@@ -0,0 +1,443 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import copy
+from loguru import logger
+from mineru.utils.enum_class import ContentType, BlockType, SplitFlag
+from mineru.utils.language import detect_lang
+
+
+LINE_STOP_FLAG = ('.', '!', '?', '。', '！', '？', ')', '）', '"', '”', ':', '：', ';', '；')
+LIST_END_FLAG = ('.', '。', ';', '；')
+
+
+class ListLineTag:
+ IS_LIST_START_LINE = 'is_list_start_line'
+ IS_LIST_END_LINE = 'is_list_end_line'
+
+
+def __process_blocks(blocks):
+ # 对所有block预处理
+ # 1.通过title和interline_equation将block分组
+ # 2.bbox边界根据line信息重置
+
+ result = []
+ current_group = []
+
+ for i in range(len(blocks)):
+ current_block = blocks[i]
+
+ # 如果当前块是 text 类型
+ if current_block['type'] in [BlockType.TEXT, BlockType.INDEX, BlockType.VERTICAL_TEXT]:
+ current_block['bbox_fs'] = copy.deepcopy(current_block['bbox'])
+ if 'lines' in current_block and len(current_block['lines']) > 0:
+ current_block['bbox_fs'] = [
+ min([line['bbox'][0] for line in current_block['lines']]),
+ min([line['bbox'][1] for line in current_block['lines']]),
+ max([line['bbox'][2] for line in current_block['lines']]),
+ max([line['bbox'][3] for line in current_block['lines']]),
+ ]
+ current_group.append(current_block)
+
+ # 检查下一个块是否存在
+ if i + 1 < len(blocks):
+ next_block = blocks[i + 1]
+ # 如果下一个块不是 text 类型且是 title 或 interline_equation 类型
+ if next_block['type'] in [
+ BlockType.ABSTRACT,
+ BlockType.INTERLINE_EQUATION,
+ BlockType.DOC_TITLE,
+ BlockType.PARAGRAPH_TITLE,
+ ]:
+ result.append(current_group)
+ current_group = []
+
+ # 处理最后一个 group
+ if current_group:
+ result.append(current_group)
+
+ return result
+
+
+def __is_list_or_index_block(block):
+ if block['type'] == BlockType.VERTICAL_TEXT:
+ return BlockType.VERTICAL_TEXT
+ if block['type'] == BlockType.INDEX:
+ for line in block['lines']:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ return BlockType.INDEX
+ # 一个block如果是list block 应该同时满足以下特征
+ # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.block内有多个line 右侧不顶格（狗牙状）
+ # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.多个line以endflag结尾
+ # 1.block内有多个line 2.block 内有多个line左侧顶格写 3.block内有多个line 左侧不顶格
+
+ # index block 是一种特殊的list block
+ # 一个block如果是index block 应该同时满足以下特征
+ # 1.block内有多个line 2.block 内有多个line两侧均顶格写 3.line的开头或者结尾均为数字
+ if len(block['lines']) >= 2:
+ first_line = block['lines'][0]
+ line_height = first_line['bbox'][3] - first_line['bbox'][1]
+ block_weight = block['bbox_fs'][2] - block['bbox_fs'][0]
+ block_height = block['bbox_fs'][3] - block['bbox_fs'][1]
+ page_weight, page_height = block['page_size']
+
+ left_close_num = 0
+ left_not_close_num = 0
+ right_not_close_num = 0
+ right_close_num = 0
+ lines_text_list = []
+ center_close_num = 0
+ external_sides_not_close_num = 0
+ multiple_para_flag = False
+ last_line = block['lines'][-1]
+
+ if page_weight == 0:
+ block_weight_ratio = 0
+ else:
+ block_weight_ratio = block_weight / page_weight
+ # logger.info(f"block_weight_ratio: {block_weight_ratio}")
+
+ # 如果首行左边不顶格而右边顶格,末行左边顶格而右边不顶格 （第一行可能可以右边不顶格）
+ if (
+ first_line['bbox'][0] - block['bbox_fs'][0] > line_height / 2
+ and abs(last_line['bbox'][0] - block['bbox_fs'][0]) < line_height / 2
+ and block['bbox_fs'][2] - last_line['bbox'][2] > line_height
+ ):
+ multiple_para_flag = True
+
+ block_text = ''
+
+ for line in block['lines']:
+ line_text = ''
+
+ for span in line['spans']:
+ span_type = span['type']
+ if span_type == ContentType.TEXT:
+ line_text += span['content'].strip()
+ # 添加所有文本，包括空行，保持与block['lines']长度一致
+ lines_text_list.append(line_text)
+ block_text = ''.join(lines_text_list)
+
+ block_lang = detect_lang(block_text)
+ # logger.info(f"block_lang: {block_lang}")
+
+ for line in block['lines']:
+ line_mid_x = (line['bbox'][0] + line['bbox'][2]) / 2
+ block_mid_x = (block['bbox_fs'][0] + block['bbox_fs'][2]) / 2
+ if (
+ line['bbox'][0] - block['bbox_fs'][0] > 0.7 * line_height
+ and block['bbox_fs'][2] - line['bbox'][2] > 0.7 * line_height
+ ):
+ external_sides_not_close_num += 1
+ if abs(line_mid_x - block_mid_x) < line_height / 2:
+ center_close_num += 1
+
+ # 计算line左侧顶格数量是否大于2，是否顶格用abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height/2 来判断
+ if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
+ left_close_num += 1
+ elif line['bbox'][0] - block['bbox_fs'][0] > line_height:
+ left_not_close_num += 1
+
+ # 计算右侧是否顶格
+ if abs(block['bbox_fs'][2] - line['bbox'][2]) < line_height:
+ right_close_num += 1
+ else:
+ # 类中文没有超长单词的情况，可以用统一的阈值
+ if block_lang in ['zh', 'ja', 'ko']:
+ closed_area = 0.26 * block_weight
+ else:
+ # 右侧不顶格情况下是否有一段距离，拍脑袋用0.3block宽度做阈值
+ # block宽的阈值可以小些，block窄的阈值要大
+ if block_weight_ratio >= 0.5:
+ closed_area = 0.26 * block_weight
+ else:
+ closed_area = 0.36 * block_weight
+ if block['bbox_fs'][2] - line['bbox'][2] > closed_area:
+ right_not_close_num += 1
+
+ # 判断lines_text_list中的元素是否有超过80%都以LIST_END_FLAG结尾
+ line_end_flag = False
+ # 判断lines_text_list中的元素是否有超过80%都以数字开头或都以数字结尾
+ line_num_flag = False
+ num_start_count = 0
+ num_end_count = 0
+ flag_end_count = 0
+
+ if len(lines_text_list) > 0:
+ for line_text in lines_text_list:
+ if len(line_text) > 0:
+ if line_text[-1] in LIST_END_FLAG:
+ flag_end_count += 1
+ if line_text[0].isdigit():
+ num_start_count += 1
+ if line_text[-1].isdigit():
+ num_end_count += 1
+
+ if (
+ num_start_count / len(lines_text_list) >= 0.8
+ or num_end_count / len(lines_text_list) >= 0.8
+ ):
+ line_num_flag = True
+ if flag_end_count / len(lines_text_list) >= 0.8:
+ line_end_flag = True
+
+ # 有的目录右侧不贴边, 目前认为左边或者右边有一边全贴边，且符合数字规则极为index
+ if (
+ left_close_num / len(block['lines']) >= 0.8
+ or right_close_num / len(block['lines']) >= 0.8
+ ) and line_num_flag:
+ for line in block['lines']:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ return BlockType.INDEX
+
+ # 全部line都居中的特殊list识别，每行都需要换行，特征是多行，且大多数行都前后not_close,每line中点x坐标接近
+ # 补充条件block的长宽比有要求
+ elif (
+ external_sides_not_close_num >= 2
+ and center_close_num == len(block['lines'])
+ and external_sides_not_close_num / len(block['lines']) >= 0.5
+ and block_height / block_weight > 0.4
+ ):
+ for line in block['lines']:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ return BlockType.LIST
+
+ elif (
+ left_close_num >= 2
+ and (right_not_close_num >= 2 or line_end_flag or left_not_close_num >= 2)
+ and not multiple_para_flag
+ # and block_weight_ratio > 0.27
+ ):
+ # 处理一种特殊的没有缩进的list，所有行都贴左边，通过右边的空隙判断是否是item尾
+ if left_close_num / len(block['lines']) > 0.8:
+ # 这种是每个item只有一行，且左边都贴边的短item list
+ if flag_end_count == 0 and right_close_num / len(block['lines']) < 0.5:
+ for line in block['lines']:
+ if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ # 这种是大部分line item 都有结束标识符的情况，按结束标识符区分不同item
+ elif line_end_flag:
+ for i, line in enumerate(block['lines']):
+ if (
+ len(lines_text_list[i]) > 0
+ and lines_text_list[i][-1] in LIST_END_FLAG
+ ):
+ line[ListLineTag.IS_LIST_END_LINE] = True
+ if i + 1 < len(block['lines']):
+ block['lines'][i + 1][
+ ListLineTag.IS_LIST_START_LINE
+ ] = True
+ # line item基本没有结束标识符，而且也没有缩进，按右侧空隙判断哪些是item end
+ else:
+ line_start_flag = False
+ for i, line in enumerate(block['lines']):
+ if line_start_flag:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ line_start_flag = False
+
+ if (
+ abs(block['bbox_fs'][2] - line['bbox'][2])
+ > 0.1 * block_weight
+ ):
+ line[ListLineTag.IS_LIST_END_LINE] = True
+ line_start_flag = True
+ # 一种有缩进的特殊有序list,start line 左侧不贴边且以数字开头，end line 以 IS_LIST_END_FLAG 结尾且数量和start line 一致
+ elif num_start_count >= 2 and num_start_count == flag_end_count:
+ for i, line in enumerate(block['lines']):
+ if len(lines_text_list[i]) > 0:
+ if lines_text_list[i][0].isdigit():
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ if lines_text_list[i][-1] in LIST_END_FLAG:
+ line[ListLineTag.IS_LIST_END_LINE] = True
+ else:
+ # 正常有缩进的list处理
+ for line in block['lines']:
+ if abs(block['bbox_fs'][0] - line['bbox'][0]) < line_height / 2:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ if abs(block['bbox_fs'][2] - line['bbox'][2]) > line_height:
+ line[ListLineTag.IS_LIST_END_LINE] = True
+
+ return BlockType.LIST
+ else:
+ return BlockType.TEXT
+ else:
+ return BlockType.TEXT
+
+
+def __merge_2_text_blocks(block1, block2):
+ if len(block1['lines']) > 0 and len(block2['lines']) > 0:
+ first_line = block1['lines'][0]
+ line_height = first_line['bbox'][3] - first_line['bbox'][1]
+ block1_weight = block1['bbox'][2] - block1['bbox'][0]
+ block2_weight = block2['bbox'][2] - block2['bbox'][0]
+ min_block_weight = min(block1_weight, block2_weight)
+ if abs(block1['bbox_fs'][0] - first_line['bbox'][0]) < line_height / 2:
+ last_line = block2['lines'][-1]
+ if len(last_line['spans']) > 0:
+ last_span = last_line['spans'][-1]
+ line_height = last_line['bbox'][3] - last_line['bbox'][1]
+ if len(first_line['spans']) > 0:
+ first_span = first_line['spans'][0]
+ if len(first_span['content']) > 0:
+ span_start_with_num = first_span['content'][0].isdigit()
+ span_start_with_big_char = first_span['content'][0].isupper()
+ if (
+ # 上一个block的最后一个line的右边界和block的右边界差距不超过line_height
+ abs(block2['bbox_fs'][2] - last_line['bbox'][2]) < line_height
+ # 上一个block的最后一个span不是以特定符号结尾
+ and not last_span['content'].endswith(LINE_STOP_FLAG)
+ # 两个block宽度差距超过2倍也不合并
+ and abs(block1_weight - block2_weight) < min_block_weight
+ # 下一个block的第一个字符是数字
+ and not span_start_with_num
+ # 下一个block的第一个字符是大写字母
+ and not span_start_with_big_char
+ # 下一个块的y0要比上一个块的y1小
+ and block1['bbox'][1] < block2['bbox'][3]
+ # 两个块任意一个块需要大于1行
+ and (len(block1['lines']) > 1 or len(block2['lines']) > 1)
+ ):
+ if block1['page_num'] != block2['page_num']:
+ for line in block1['lines']:
+ for span in line['spans']:
+ span[SplitFlag.CROSS_PAGE] = True
+ block2['lines'].extend(block1['lines'])
+ block1['lines'] = []
+ block1[SplitFlag.LINES_DELETED] = True
+
+ return block1, block2
+
+
+def __merge_2_vertical_text_blocks(block1, block2):
+ if len(block1['lines']) > 0 and len(block2['lines']) > 0:
+ first_line = block1['lines'][0]
+ line_width = first_line['bbox'][2] - first_line['bbox'][0]
+ block1_height = block1['bbox'][3] - block1['bbox'][1]
+ block2_height = block2['bbox'][3] - block2['bbox'][1]
+ min_block_height = min(block1_height, block2_height)
+ if line_width > 0 and abs(block1['bbox_fs'][1] - first_line['bbox'][1]) < line_width / 2:
+ last_line = block2['lines'][-1]
+ if len(last_line['spans']) > 0:
+ last_span = last_line['spans'][-1]
+ line_width = last_line['bbox'][2] - last_line['bbox'][0]
+ if line_width > 0 and len(first_line['spans']) > 0:
+ first_span = first_line['spans'][0]
+ first_content = first_span.get('content', '')
+ last_content = last_span.get('content', '')
+ if len(first_content) > 0:
+ span_start_with_num = first_content[0].isdigit()
+ span_start_with_big_char = first_content[0].isupper()
+ if (
+ abs(block2['bbox_fs'][3] - last_line['bbox'][3]) < line_width
+ and not last_content.endswith(LINE_STOP_FLAG)
+ and abs(block1_height - block2_height) < min_block_height
+ and not span_start_with_num
+ and not span_start_with_big_char
+ ):
+ if block1['page_num'] != block2['page_num']:
+ for line in block1['lines']:
+ for span in line['spans']:
+ span[SplitFlag.CROSS_PAGE] = True
+ block2['lines'].extend(block1['lines'])
+ block1['lines'] = []
+ block1[SplitFlag.LINES_DELETED] = True
+
+ return block1, block2
+
+
+def __merge_2_list_blocks(block1, block2):
+ if block1['page_num'] != block2['page_num']:
+ for line in block1['lines']:
+ for span in line['spans']:
+ span[SplitFlag.CROSS_PAGE] = True
+ block2['lines'].extend(block1['lines'])
+ block1['lines'] = []
+ block1[SplitFlag.LINES_DELETED] = True
+
+ return block1, block2
+
+
+def __is_list_group(text_blocks_group):
+ # list group的特征是一个group内的所有block都满足以下条件
+ # 1.每个block都不超过3行 2. 每个block 的左边界都比较接近(逻辑简单点先不加这个规则)
+ for block in text_blocks_group:
+ if block['type'] == BlockType.VERTICAL_TEXT:
+ return False
+ if len(block['lines']) > 3:
+ return False
+ return True
+
+
+def __para_merge_page(blocks):
+ page_text_blocks_groups = __process_blocks(blocks)
+ for text_blocks_group in page_text_blocks_groups:
+ if len(text_blocks_group) > 0:
+ # 需要先在合并前对所有block判断是否为list or index block
+ for block in text_blocks_group:
+ block_type = __is_list_or_index_block(block)
+ block['type'] = block_type
+ # logger.info(f"{block['type']}:{block}")
+
+ if len(text_blocks_group) > 1:
+ # 在合并前判断这个group 是否是一个 list group
+ is_list_group = __is_list_group(text_blocks_group)
+
+ # 倒序遍历
+ for i in range(len(text_blocks_group) - 1, -1, -1):
+ current_block = text_blocks_group[i]
+
+ # 检查是否有前一个块
+ if i - 1 >= 0:
+ prev_block = text_blocks_group[i - 1]
+
+ if (
+ current_block['type'] == BlockType.VERTICAL_TEXT
+ and prev_block['type'] == BlockType.VERTICAL_TEXT
+ ):
+ __merge_2_vertical_text_blocks(current_block, prev_block)
+ elif (
+ current_block['type'] == BlockType.TEXT
+ and prev_block['type'] == BlockType.TEXT
+ and not is_list_group
+ ):
+ __merge_2_text_blocks(current_block, prev_block)
+ elif (
+ current_block['type'] == BlockType.LIST
+ and prev_block['type'] == BlockType.LIST
+ ) or (
+ current_block['type'] == BlockType.INDEX
+ and prev_block['type'] == BlockType.INDEX
+ ):
+ __merge_2_list_blocks(current_block, prev_block)
+
+ else:
+ continue
+
+
+def para_split(page_info_list):
+ all_blocks = []
+ for page_info in page_info_list:
+ blocks = copy.deepcopy(page_info['preproc_blocks'])
+ for block in blocks:
+ block['page_num'] = page_info['page_idx']
+ block['page_size'] = page_info['page_size']
+ all_blocks.extend(blocks)
+
+ __para_merge_page(all_blocks)
+ for page_info in page_info_list:
+ page_info['para_blocks'] = []
+ for block in all_blocks:
+ if 'page_num' in block:
+ if block['page_num'] == page_info['page_idx']:
+ if block['type'] == BlockType.VERTICAL_TEXT:
+ block['type'] = BlockType.TEXT
+ page_info['para_blocks'].append(block)
+ # 从block中删除不需要的page_num和page_size字段
+ del block['page_num']
+ del block['page_size']
+
+
+if __name__ == '__main__':
+ input_blocks = []
+ # 调用函数
+ groups = __process_blocks(input_blocks)
+ for group_index, group in enumerate(groups):
+ print(f'Group {group_index}: {group}')
diff --git a/mineru/backend/pipeline/pipeline_analyze.py b/mineru/backend/pipeline/pipeline_analyze.py
new file mode 100644
index 000000000..aa309145f
--- /dev/null
+++ b/mineru/backend/pipeline/pipeline_analyze.py
@@ -0,0 +1,348 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import time
+from typing import List, Tuple
+
+import pypdfium2 as pdfium
+from PIL import Image
+from loguru import logger
+from tqdm import tqdm
+
+from .model_init import MineruPipelineModel, PIPELINE_MODEL_INIT_LOCK
+from .model_json_to_middle_json import (
+ append_batch_results_to_middle_json,
+ finalize_middle_json,
+ init_middle_json,
+)
+from ..utils.runtime_utils import exclude_progress_bar_idle_time
+from mineru.utils.config_reader import get_device, get_processing_window_size
+from ...utils.enum_class import ImageType
+from ...utils.pdf_classify import classify
+from ...utils.pdf_image_tools import load_images_from_pdf_doc
+from ...utils.model_utils import get_vram, clean_memory
+from ...utils.pdfium_guard import (
+ close_pdfium_document,
+ get_pdfium_document_page_count,
+ open_pdfium_document,
+)
+
+
+os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1' # 让mps可以fallback
+os.environ['NO_ALBUMENTATIONS_UPDATE'] = '1' # 禁止albumentations检查更新
+
+class ModelSingleton:
+ _instance = None
+ _models = {}
+ _lock = PIPELINE_MODEL_INIT_LOCK
+
+ def __new__(cls, *args, **kwargs):
+ with cls._lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ return cls._instance
+
+ def get_model(
+ self,
+ lang=None,
+ formula_enable=None,
+ table_enable=None,
+ ):
+ key = (lang, formula_enable, table_enable)
+ with self._lock:
+ if key not in self._models:
+ self._models[key] = custom_model_init(
+ lang=lang,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ )
+ return self._models[key]
+
+
+def custom_model_init(
+ lang=None,
+ formula_enable=True,
+ table_enable=True,
+):
+ model_init_start = time.time()
+ # 从配置文件读取model-dir和device
+ device = get_device()
+
+ formula_config = {"enable": formula_enable}
+ table_config = {"enable": table_enable}
+
+ model_input = {
+ 'device': device,
+ 'table_config': table_config,
+ 'formula_config': formula_config,
+ 'lang': lang,
+ }
+
+ custom_model = MineruPipelineModel(**model_input)
+
+ model_init_cost = time.time() - model_init_start
+ logger.info(f'model init cost: {model_init_cost}')
+
+ return custom_model
+
+
+def _get_ocr_enable(pdf_bytes, parse_method: str) -> bool:
+ if parse_method == 'auto':
+ return classify(pdf_bytes) == 'ocr'
+ if parse_method == 'ocr':
+ return True
+ return False
+
+
+def _close_images(images_list):
+ for image_dict in images_list or []:
+ pil_img = image_dict.get('img_pil')
+ if pil_img is not None:
+ try:
+ pil_img.close()
+ except Exception:
+ pass
+
+
+def _format_doc_slices(batch_slices):
+ return ",".join(
+ f"doc{item['doc_index']}:{item['page_start'] + 1}-{item['page_end'] + 1}"
+ for item in batch_slices
+ )
+
+
+def _finalize_processing_window_context(context, on_doc_ready):
+ if context['closed']:
+ return
+ finalize_middle_json(
+ context['middle_json']['pdf_info'],
+ lang=context['lang'],
+ ocr_enable=context['ocr_enable'],
+ )
+ logger.debug(
+ f"Pipeline doc ready: doc{context['doc_index']} pages={context['page_count']}"
+ )
+ on_doc_ready(
+ context['doc_index'],
+ context['model_list'],
+ context['middle_json'],
+ context['ocr_enable'],
+ )
+ close_pdfium_document(context['pdf_doc'])
+ context['closed'] = True
+
+
+def _emit_zero_page_contexts(doc_contexts, on_doc_ready):
+ for context in doc_contexts:
+ if context['page_count'] == 0 and not context['closed']:
+ _finalize_processing_window_context(context, on_doc_ready)
+
+
+def doc_analyze_streaming(
+ pdf_bytes_list,
+ image_writer_list,
+ lang_list,
+ on_doc_ready,
+ parse_method: str = 'auto',
+ formula_enable=True,
+ table_enable=True,
+):
+ if not (len(pdf_bytes_list) == len(image_writer_list) == len(lang_list)):
+ raise ValueError("pdf_bytes_list, image_writer_list, and lang_list must have the same length")
+
+ doc_contexts = []
+ total_pages = 0
+ for doc_index, (pdf_bytes, image_writer, lang) in enumerate(
+ zip(pdf_bytes_list, image_writer_list, lang_list)
+ ):
+ _ocr_enable = _get_ocr_enable(pdf_bytes, parse_method)
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ total_pages += page_count
+ doc_contexts.append(
+ {
+ 'doc_index': doc_index,
+ 'pdf_bytes': pdf_bytes,
+ 'pdf_doc': pdf_doc,
+ 'page_count': page_count,
+ 'next_page_idx': 0,
+ 'middle_json': init_middle_json(),
+ 'model_list': [],
+ 'image_writer': image_writer,
+ 'lang': lang,
+ 'ocr_enable': _ocr_enable,
+ 'closed': False,
+ }
+ )
+
+ if total_pages == 0:
+ _emit_zero_page_contexts(doc_contexts, on_doc_ready)
+ return
+
+ window_size = get_processing_window_size(default=64)
+ total_batches = (total_pages + window_size - 1) // window_size
+ logger.info(
+ f'Pipeline processing-window multi-file run. doc_count={len(doc_contexts)}, '
+ f'total_pages={total_pages}, window_size={window_size}, total_batches={total_batches}'
+ )
+
+ _emit_zero_page_contexts(doc_contexts, on_doc_ready)
+ processed_pages = 0
+ infer_start = time.time()
+ try:
+ progress_bar = None
+ last_append_end_time = None
+ try:
+ batch_index = 0
+ while processed_pages < total_pages:
+ batch_index += 1
+ batch_capacity = window_size
+ batch_images = []
+ batch_slices = []
+ batch_payloads = []
+
+ for context in doc_contexts:
+ if batch_capacity == 0:
+ break
+ page_start = context['next_page_idx']
+ if page_start >= context['page_count']:
+ continue
+ take_count = min(batch_capacity, context['page_count'] - page_start)
+ page_end = page_start + take_count - 1
+ images_list = load_images_from_pdf_doc(
+ context['pdf_doc'],
+ start_page_id=page_start,
+ end_page_id=page_end,
+ image_type=ImageType.PIL,
+ pdf_bytes=context['pdf_bytes'],
+ )
+ images_with_extra_info = [
+ (image_dict['img_pil'], context['ocr_enable'], context['lang'])
+ for image_dict in images_list
+ ]
+ batch_images.extend(images_with_extra_info)
+ batch_slices.append(
+ {
+ 'doc_index': context['doc_index'],
+ 'page_start': page_start,
+ 'page_end': page_end,
+ 'count': take_count,
+ }
+ )
+ batch_payloads.append((context, images_list, page_start, take_count))
+ context['next_page_idx'] = page_end + 1
+ batch_capacity -= take_count
+
+ logger.info(
+ f'Pipeline processing window batch {batch_index}/{total_batches}: '
+ f'{processed_pages + len(batch_images)}/{total_pages} pages, '
+ f'batch_pages={len(batch_images)}, doc_slices={_format_doc_slices(batch_slices)}'
+ )
+
+ batch_results = batch_image_analyze(
+ batch_images,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ )
+ if progress_bar is None:
+ progress_bar = tqdm(total=total_pages, desc="Processing pages")
+ else:
+ exclude_progress_bar_idle_time(
+ progress_bar,
+ last_append_end_time,
+ now=time.time(),
+ )
+
+ result_offset = 0
+ for context, images_list, page_start, take_count in batch_payloads:
+ result_slice = batch_results[result_offset: result_offset + take_count]
+ append_batch_results_to_middle_json(
+ context['middle_json'],
+ result_slice,
+ images_list,
+ context['pdf_doc'],
+ context['image_writer'],
+ page_start_index=page_start,
+ ocr_enable=context['ocr_enable'],
+ model_list=context['model_list'],
+ progress_bar=progress_bar,
+ )
+ result_offset += take_count
+ _close_images(images_list)
+ images_list.clear()
+
+ if context['next_page_idx'] >= context['page_count'] and not context['closed']:
+ _finalize_processing_window_context(context, on_doc_ready)
+
+ last_append_end_time = time.time()
+ processed_pages += len(batch_images)
+ finally:
+ if progress_bar is not None:
+ progress_bar.close()
+
+ infer_time = round(time.time() - infer_start, 2)
+ if infer_time > 0:
+ logger.debug(
+ f"processing-window multi-file infer finished, cost: {infer_time}, "
+ f"speed: {round(total_pages / infer_time, 3)} page/s"
+ )
+ finally:
+ for context in doc_contexts:
+ if not context['closed']:
+ close_pdfium_document(context['pdf_doc'])
+ context['closed'] = True
+
+
+def batch_image_analyze(
+ images_with_extra_info: List[Tuple[Image.Image, bool, str]],
+ formula_enable=True,
+ table_enable=True):
+
+ from .batch_analyze import BatchAnalyze
+
+ model_manager = ModelSingleton()
+
+ device = get_device()
+
+ if str(device).startswith('npu'):
+ try:
+ import torch_npu
+ if torch_npu.npu.is_available():
+ torch_npu.npu.set_compile_mode(jit_compile=False)
+ except Exception as e:
+ raise RuntimeError(
+ "NPU is selected as device, but torch_npu is not available. "
+ "Please ensure that the torch_npu package is installed correctly."
+ ) from e
+
+ gpu_memory = get_vram(device)
+ if gpu_memory >= 32:
+ batch_ratio = 16
+ elif gpu_memory >= 16:
+ batch_ratio = 8
+ elif gpu_memory >= 8:
+ batch_ratio = 4
+ elif gpu_memory >= 6:
+ batch_ratio = 2
+ else:
+ batch_ratio = 1
+ logger.info(
+ f'GPU Memory: {gpu_memory} GB, Batch Ratio: {batch_ratio}. '
+ )
+
+ # 检测torch的版本号
+ import torch
+ from packaging import version
+ device_type = os.getenv("MINERU_LMDEPLOY_DEVICE", "")
+ if device_type.lower() in ["corex"]:
+ enable_ocr_det_batch = False
+ else:
+ if version.parse(torch.__version__) >= version.parse("2.8.0"):
+ os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+ enable_ocr_det_batch = True
+
+ batch_model = BatchAnalyze(model_manager, batch_ratio, formula_enable, table_enable, enable_ocr_det_batch)
+ results = batch_model(images_with_extra_info)
+
+ clean_memory(get_device())
+
+ return results
diff --git a/mineru/backend/pipeline/pipeline_magic_model.py b/mineru/backend/pipeline/pipeline_magic_model.py
new file mode 100644
index 000000000..3bc04c093
--- /dev/null
+++ b/mineru/backend/pipeline/pipeline_magic_model.py
@@ -0,0 +1,553 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from mineru.backend.pipeline.para_split import ListLineTag
+from mineru.backend.pipeline.pipeline_middle_json_mkcontent import _merge_para_text
+from mineru.utils.boxbase import (
+ bbox_center_distance,
+ bbox_distance,
+ calculate_overlap_area_2_minbox_area_ratio,
+ calculate_overlap_area_in_bbox1_area_ratio,
+)
+from mineru.utils.enum_class import ContentType, BlockType
+from mineru.utils.guess_suffix_or_lang import guess_language_by_text
+from mineru.utils.span_block_fix import merge_spans_to_vertical_line, vertical_line_sort_spans_from_top_to_bottom, \
+ merge_spans_to_line, line_sort_spans_by_left_to_right
+from mineru.utils.span_pre_proc import txt_spans_extract
+
+
+class MagicModel:
+
+ PP_DOCLAYOUT_V2_LABELS_TO_BLOCK_TYPES = {
+ "abstract": BlockType.ABSTRACT,
+ "algorithm": BlockType.CODE,
+ "aside_text": BlockType.ASIDE_TEXT,
+ "chart": BlockType.CHART,
+ "content": BlockType.INDEX,
+ "display_formula": BlockType.INTERLINE_EQUATION,
+ "doc_title": BlockType.DOC_TITLE,
+ "figure_title": BlockType.CAPTION,
+ "footer": BlockType.FOOTER,
+ "footer_image": BlockType.FOOTER,
+ "footnote": BlockType.PAGE_FOOTNOTE,
+ "formula_number": BlockType.FORMULA_NUMBER,
+ "header": BlockType.HEADER,
+ "header_image": BlockType.HEADER,
+ "image": BlockType.IMAGE,
+ "number": BlockType.PAGE_NUMBER,
+ "paragraph_title": BlockType.PARAGRAPH_TITLE,
+ "reference_content": BlockType.REF_TEXT,
+ "seal": BlockType.SEAL,
+ "table": BlockType.TABLE,
+ "text": BlockType.TEXT,
+ "vertical_text": BlockType.VERTICAL_TEXT,
+ "vision_footnote": BlockType.FOOTNOTE,
+ }
+
+ VISUAL_MAIN_TYPES = (BlockType.IMAGE, BlockType.TABLE, BlockType.CHART, BlockType.CODE)
+ VISUAL_CHILD_TYPES = (BlockType.CAPTION, BlockType.FOOTNOTE)
+ VISUAL_TYPE_MAPPING = {
+ BlockType.IMAGE: {
+ "body": BlockType.IMAGE_BODY,
+ "caption": BlockType.IMAGE_CAPTION,
+ "footnote": BlockType.IMAGE_FOOTNOTE,
+ },
+ BlockType.TABLE: {
+ "body": BlockType.TABLE_BODY,
+ "caption": BlockType.TABLE_CAPTION,
+ "footnote": BlockType.TABLE_FOOTNOTE,
+ },
+ BlockType.CHART: {
+ "body": BlockType.CHART_BODY,
+ "caption": BlockType.CHART_CAPTION,
+ "footnote": BlockType.CHART_FOOTNOTE,
+ },
+ BlockType.CODE: {
+ "body": BlockType.CODE_BODY,
+ "caption": BlockType.CODE_CAPTION,
+ "footnote": BlockType.CODE_FOOTNOTE,
+ }
+ }
+
+ def __init__(
+ self,
+ page_model_info: dict,
+ page=None,
+ scale=1,
+ page_pil_img=None,
+ page_w=None,
+ page_h=None,
+ ocr_enable=False,
+ ):
+ self.__page_model_info = page_model_info
+ self.page_inline_formula = []
+ self.page_ocr_res = []
+ self.page_blocks = []
+ self.image_groups = []
+ self.table_groups = []
+ self.chart_groups = []
+ self.all_image_spans = []
+ self.__layout_det_by_index = {}
+ self.__scale = scale
+ self.__fix_axis() # bbox坐标修正，删除高度或者宽度小于等于0的spans
+ self.__post_process() # index重排，填充行内公式和文本span
+ if not ocr_enable:
+ virtual_block = [0, 0, page_w, page_h, None, None, None, "text"]
+ self.page_ocr_res = txt_spans_extract(
+ page,
+ self.page_ocr_res,
+ page_pil_img,
+ scale,
+ [virtual_block],
+ [],
+ )
+ self.page_text_inline_formula_spans = self.page_inline_formula + self.page_ocr_res
+
+ for layout_det in self.__page_model_info['layout_dets']:
+ if layout_det.get('label') in self.PP_DOCLAYOUT_V2_LABELS_TO_BLOCK_TYPES:
+ block_bbox = layout_det['bbox']
+ block_type = self.PP_DOCLAYOUT_V2_LABELS_TO_BLOCK_TYPES[layout_det['label']]
+ block_index = layout_det['index']
+ block_score = layout_det['score']
+ block = self.__copy_block_fields(
+ layout_det,
+ type=block_type,
+ bbox=block_bbox,
+ index=block_index,
+ score=block_score,
+ )
+ self.page_blocks.append(block)
+
+ self.page_blocks.sort(key=lambda x: x["index"])
+ self.__build_page_blocks()
+ self.__classify_visual_blocks()
+ self.__build_return_blocks()
+
+
+ @staticmethod
+ def __fix_text_block(block):
+ if block["type"] == BlockType.VERTICAL_TEXT:
+ # 如果是纵向文本块，则按纵向lines处理
+ block_lines = merge_spans_to_vertical_line(block['spans'])
+ sort_block_lines = vertical_line_sort_spans_from_top_to_bottom(block_lines)
+ else:
+ block_lines = merge_spans_to_line(block['spans'])
+ sort_block_lines = line_sort_spans_by_left_to_right(block_lines)
+
+ if block["type"] == BlockType.CODE:
+ for line in sort_block_lines:
+ line[ListLineTag.IS_LIST_START_LINE] = True
+ code_content = _merge_para_text(
+ {'lines': sort_block_lines},
+ False,
+ '\n'
+ )
+ guess_lang = guess_language_by_text(code_content)
+ if guess_lang not in ["txt", "unknown"]:
+ block["sub_type"] = "code"
+ block["guess_lang"] = guess_lang
+
+ block['lines'] = sort_block_lines
+ del block['spans']
+ return block
+
+ @staticmethod
+ def __copy_block_fields(block, **overrides):
+ copied_block = {
+ key: value
+ for key, value in block.items()
+ if key not in {"cls_id", "label"}
+ }
+ copied_block.update(overrides)
+ return copied_block
+
+
+ @staticmethod
+ def __is_inline_formula_block(layout_det: dict) -> bool:
+ return (
+ layout_det.get("label") == "inline_formula"
+ or layout_det.get("cls_id") == 15
+ )
+
+ @staticmethod
+ def __is_ocr_text_block(layout_det: dict) -> bool:
+ return layout_det.get("label") == "ocr_text"
+
+ def __build_return_blocks(self):
+ self.preproc_blocks = []
+ self.discarded_blocks = []
+ for block in self.page_blocks:
+ if block["type"] in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_NUMBER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_FOOTNOTE
+ ]:
+ self.discarded_blocks.append(block)
+ else:
+ # 单独处理code block
+ if block["type"] in [BlockType.CODE]:
+ for sub_block in block["blocks"]:
+ if sub_block["type"] == BlockType.CODE_BODY:
+ block["sub_type"] = sub_block.pop("sub_type", "algorithm")
+ if block["sub_type"] == "code":
+ block["guess_lang"] = sub_block.pop("guess_lang", "txt")
+
+ self.preproc_blocks.append(block)
+
+ def __build_page_blocks(self):
+ span_type = "unknown"
+ for block in self.page_blocks:
+ if block["type"] in [
+ BlockType.ABSTRACT,
+ BlockType.CODE,
+ BlockType.ASIDE_TEXT,
+ BlockType.INDEX,
+ BlockType.DOC_TITLE,
+ BlockType.CAPTION,
+ BlockType.FOOTER,
+ BlockType.PAGE_FOOTNOTE,
+ BlockType.FORMULA_NUMBER,
+ BlockType.HEADER,
+ BlockType.PAGE_NUMBER,
+ BlockType.PARAGRAPH_TITLE,
+ BlockType.REF_TEXT,
+ BlockType.TEXT,
+ BlockType.VERTICAL_TEXT,
+ BlockType.FOOTNOTE,
+ ]:
+ span_type = ContentType.TEXT
+ elif block["type"] in [BlockType.IMAGE]:
+ span_type = ContentType.IMAGE
+ elif block["type"] in [BlockType.TABLE]:
+ span_type = ContentType.TABLE
+ elif block["type"] in [BlockType.CHART]:
+ span_type = ContentType.CHART
+ elif block["type"] in [BlockType.INTERLINE_EQUATION]:
+ span_type = ContentType.INTERLINE_EQUATION
+ elif block["type"] in [BlockType.SEAL]:
+ span_type = ContentType.SEAL
+
+ if span_type in [
+ ContentType.IMAGE,
+ ContentType.TABLE,
+ ContentType.CHART,
+ ContentType.INTERLINE_EQUATION,
+ ContentType.SEAL
+ ]:
+ span = {
+ "bbox": block["bbox"],
+ "type": span_type,
+ }
+ if span_type == ContentType.TABLE:
+ span["html"] = block.get("html", "")
+ block.pop("html", None)
+ if span_type == ContentType.INTERLINE_EQUATION:
+ span["content"] = block.get("latex", "")
+ block.pop("latex", None)
+ if span_type == ContentType.SEAL:
+ span["content"] = block.get("text")
+ block.pop("text", None)
+
+ self.all_image_spans.append(span)
+ # 构造line对象
+ spans = [span]
+ line = {"bbox": block["bbox"], "spans": spans}
+ block["lines"] = [line]
+ else:
+ # span填充
+ block_spans = []
+ for span in self.page_text_inline_formula_spans:
+ overlap_ratio = calculate_overlap_area_in_bbox1_area_ratio(
+ span['bbox'], block["bbox"]
+ )
+ if block["type"] == BlockType.FORMULA_NUMBER:
+ # OCR 检测框通常会比公式编号框更大，使用最小框重叠比避免编号文字无法回填。
+ overlap_ratio = max(
+ overlap_ratio,
+ calculate_overlap_area_2_minbox_area_ratio(
+ span['bbox'], block["bbox"]
+ ),
+ )
+ if overlap_ratio > 0.5:
+ block_spans.append(span)
+ # 从spans删除已经放入block_spans中的span
+ if len(block_spans) > 0:
+ for span in block_spans:
+ self.page_text_inline_formula_spans.remove(span)
+
+ block["spans"] = block_spans
+ block = self.__fix_text_block(block)
+
+ def __fix_axis(self):
+ need_remove_list = []
+ layout_dets = self.__page_model_info["layout_dets"]
+ for layout_det in layout_dets:
+ x0, y0, x1, y1 = layout_det["bbox"]
+ bbox = [
+ int(x0 / self.__scale),
+ int(y0 / self.__scale),
+ int(x1 / self.__scale),
+ int(y1 / self.__scale),
+ ]
+ layout_det["bbox"] = bbox
+ # 删除高度或者宽度小于等于2的spans
+ if bbox[2] - bbox[0] <= 2 or bbox[3] - bbox[1] <= 2:
+ need_remove_list.append(layout_det)
+ for need_remove in need_remove_list:
+ layout_dets.remove(need_remove)
+
+ def __post_process(self):
+ next_index = 1
+ layout_dets = self.__page_model_info["layout_dets"]
+ for layout_det in layout_dets:
+ if self.__is_inline_formula_block(layout_det):
+ layout_det.pop("index", None)
+ self.page_inline_formula.append({
+ "bbox": layout_det["bbox"],
+ "type": ContentType.INLINE_EQUATION,
+ "content": layout_det["latex"],
+ "score": layout_det["score"],
+ })
+ continue
+
+ if self.__is_ocr_text_block(layout_det):
+ self.page_ocr_res.append({
+ "bbox": layout_det["bbox"],
+ "type": ContentType.TEXT,
+ "content": layout_det["text"],
+ "score": layout_det["score"],
+ })
+ continue
+
+ if "index" in layout_det:
+ layout_det["index"] = next_index
+ next_index += 1
+
+ def __classify_visual_blocks(self):
+ if not self.page_blocks:
+ return
+
+ ordered_blocks = sorted(self.page_blocks, key=lambda x: x["index"])
+ original_type_by_index = {
+ block["index"]: block["type"] for block in ordered_blocks
+ }
+ position_by_index = {
+ block["index"]: pos for pos, block in enumerate(ordered_blocks)
+ }
+ main_blocks = [
+ block
+ for block in ordered_blocks
+ if original_type_by_index[block["index"]] in self.VISUAL_MAIN_TYPES
+ ]
+ child_blocks = [
+ block
+ for block in ordered_blocks
+ if original_type_by_index[block["index"]] in self.VISUAL_CHILD_TYPES
+ ]
+
+ child_parent_map = {}
+ grouped_children = {
+ main_block["index"]: {"captions": [], "footnotes": []}
+ for main_block in main_blocks
+ }
+
+ for child_block in child_blocks:
+ parent_block = self.__find_best_visual_parent(
+ child_block,
+ main_blocks,
+ ordered_blocks,
+ original_type_by_index,
+ position_by_index,
+ )
+ child_parent_map[child_block["index"]] = (
+ None if parent_block is None else parent_block["index"]
+ )
+
+ for child_block in child_blocks:
+ original_child_type = original_type_by_index[child_block["index"]]
+ parent_index = child_parent_map[child_block["index"]]
+
+ if parent_index is None:
+ child_block["type"] = BlockType.TEXT
+ self.__sync_layout_det_type(child_block["index"], BlockType.TEXT)
+ continue
+
+ parent_type = original_type_by_index[parent_index]
+ child_kind = self.__child_kind(original_child_type)
+ mapped_type = self.VISUAL_TYPE_MAPPING[parent_type][child_kind]
+ child_block["type"] = mapped_type
+ self.__sync_layout_det_type(child_block["index"], mapped_type)
+ grouped_children[parent_index][f"{child_kind}s"].append(child_block)
+
+ self.image_groups = []
+ self.table_groups = []
+ self.chart_groups = []
+
+ rebuilt_page_blocks = []
+ for block in ordered_blocks:
+ original_block_type = original_type_by_index[block["index"]]
+
+ if original_block_type in self.VISUAL_CHILD_TYPES:
+ if child_parent_map[block["index"]] is None:
+ rebuilt_page_blocks.append(block)
+ continue
+
+ if original_block_type not in self.VISUAL_MAIN_TYPES:
+ rebuilt_page_blocks.append(block)
+ continue
+
+ mapping = self.VISUAL_TYPE_MAPPING[original_block_type]
+ body_block = self.__make_child_block(block, mapping["body"])
+ captions = sorted(
+ [
+ self.__make_child_block(caption, mapping["caption"])
+ for caption in grouped_children[block["index"]]["captions"]
+ ],
+ key=lambda x: x["index"],
+ )
+ footnotes = sorted(
+ [
+ self.__make_child_block(footnote, mapping["footnote"])
+ for footnote in grouped_children[block["index"]]["footnotes"]
+ ],
+ key=lambda x: x["index"],
+ )
+
+ self.__sync_layout_det_type(block["index"], mapping["body"])
+
+ group_info = {
+ f"{original_block_type}_body": body_block,
+ f"{original_block_type}_caption_list": captions,
+ f"{original_block_type}_footnote_list": footnotes,
+ }
+ if original_block_type == BlockType.IMAGE:
+ self.image_groups.append(group_info)
+ elif original_block_type == BlockType.TABLE:
+ self.table_groups.append(group_info)
+ else:
+ self.chart_groups.append(group_info)
+
+ two_layer_block = {
+ "type": original_block_type,
+ "bbox": block["bbox"],
+ "blocks": [body_block, *captions, *footnotes],
+ "index": block["index"],
+ "score": block.get("score"),
+ }
+ # 对blocks按index排序
+ two_layer_block["blocks"].sort(key=lambda x: x["index"])
+ rebuilt_page_blocks.append(two_layer_block)
+
+ self.page_blocks = rebuilt_page_blocks
+
+ def __find_best_visual_parent(
+ self,
+ child_block,
+ main_blocks,
+ ordered_blocks,
+ original_type_by_index,
+ position_by_index,
+ ):
+ child_type = original_type_by_index[child_block["index"]]
+ if child_type not in self.VISUAL_CHILD_TYPES:
+ return None
+
+ best_parent = None
+ best_key = None
+ for main_block in main_blocks:
+ if not self.__is_visual_neighbor(
+ child_block,
+ main_block,
+ ordered_blocks,
+ original_type_by_index,
+ position_by_index,
+ ):
+ continue
+
+ candidate_key = (
+ abs(child_block["index"] - main_block["index"]),
+ bbox_distance(child_block["bbox"], main_block["bbox"]),
+ bbox_center_distance(child_block["bbox"], main_block["bbox"]),
+ main_block["index"],
+ )
+
+ if best_key is None or candidate_key < best_key:
+ best_key = candidate_key
+ best_parent = main_block
+
+ return best_parent
+
+ def __is_visual_neighbor(
+ self,
+ child_block,
+ main_block,
+ ordered_blocks,
+ original_type_by_index,
+ position_by_index,
+ ):
+ child_type = original_type_by_index[child_block["index"]]
+ if child_type == BlockType.FOOTNOTE and child_block["index"] < main_block["index"]:
+ return False
+
+ child_pos = position_by_index[child_block["index"]]
+ main_pos = position_by_index[main_block["index"]]
+ start_pos = min(child_pos, main_pos) + 1
+ end_pos = max(child_pos, main_pos)
+
+ for pos in range(start_pos, end_pos):
+ between_block = ordered_blocks[pos]
+ if original_type_by_index[between_block["index"]] != child_type:
+ return False
+
+ return True
+
+ @staticmethod
+ def __child_kind(block_type):
+ if block_type == BlockType.CAPTION:
+ return "caption"
+ return "footnote"
+
+ @staticmethod
+ def __make_child_block(block, block_type):
+ return MagicModel.__copy_block_fields(block, type=block_type)
+
+ def __sync_layout_det_type(self, block_index, block_type):
+ layout_det = self.__layout_det_by_index.get(block_index)
+ if layout_det is not None:
+ layout_det["type"] = block_type
+
+ def get_page_blocks(self):
+ return self.page_blocks
+
+ def get_all_image_spans(self):
+ return self.all_image_spans
+
+ def get_preproc_blocks(self):
+ return self.preproc_blocks
+
+ def get_discarded_blocks(self):
+ return self.discarded_blocks
+
+ def get_imgs(self):
+ return self.image_groups
+
+ def get_tables(self):
+ return self.table_groups
+
+ def get_charts(self):
+ return self.chart_groups
+
+ def get_image_blocks(self):
+ return [
+ block for block in self.page_blocks if block["type"] == BlockType.IMAGE
+ ]
+
+ def get_table_blocks(self):
+ return [
+ block for block in self.page_blocks if block["type"] == BlockType.TABLE
+ ]
+
+ def get_chart_blocks(self):
+ return [
+ block for block in self.page_blocks if block["type"] == BlockType.CHART
+ ]
diff --git a/mineru/backend/pipeline/pipeline_middle_json_mkcontent.py b/mineru/backend/pipeline/pipeline_middle_json_mkcontent.py
new file mode 100644
index 000000000..660064120
--- /dev/null
+++ b/mineru/backend/pipeline/pipeline_middle_json_mkcontent.py
@@ -0,0 +1,1028 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+from html import unescape
+
+from loguru import logger
+
+from mineru.utils.char_utils import full_to_half_exclude_marks, is_hyphen_at_line_end
+from mineru.utils.config_reader import get_latex_delimiter_config
+from mineru.backend.pipeline.para_split import ListLineTag
+from mineru.utils.enum_class import BlockType, ContentType, ContentTypeV2, MakeMode
+from mineru.utils.language import detect_lang
+from mineru.backend.utils.markdown_utils import (
+ escape_conservative_markdown_text,
+ escape_text_block_markdown_prefix,
+)
+
+
+def make_blocks_to_markdown(paras_of_layout,
+ mode,
+ img_buket_path='',
+ ):
+ page_markdown = []
+ for para_block in paras_of_layout:
+ para_text = ''
+ para_type = para_block['type']
+ if para_type in [
+ BlockType.TEXT,
+ BlockType.LIST,
+ BlockType.INDEX,
+ BlockType.ABSTRACT,
+ BlockType.REF_TEXT
+ ]:
+ para_text = merge_para_with_text(para_block)
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ para_text = f'{"#" * title_level} {merge_para_with_text(para_block)}'
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ if len(para_block['lines']) == 0 or len(para_block['lines'][0]['spans']) == 0:
+ continue
+ if para_block['lines'][0]['spans'][0].get('content', ''):
+ para_text = merge_para_with_text(para_block)
+ else:
+ para_text = f""
+ elif para_type == BlockType.SEAL:
+ if len(para_block['lines']) == 0 or len(para_block['lines'][0]['spans']) == 0:
+ continue
+ para_text = f""
+ if para_block['lines'][0]['spans'][0].get('content', []):
+ content = " ".join(para_block['lines'][0]['spans'][0]['content'])
+ para_text += f" \n{content}"
+ elif para_type == BlockType.IMAGE:
+ if mode == MakeMode.NLP_MD:
+ continue
+ elif mode == MakeMode.MM_MD:
+ para_text = merge_visual_blocks_to_markdown(para_block, img_buket_path)
+ elif para_type == BlockType.CHART:
+ if mode == MakeMode.NLP_MD:
+ continue
+ elif mode == MakeMode.MM_MD:
+ para_text = merge_visual_blocks_to_markdown(para_block, img_buket_path)
+ elif para_type == BlockType.TABLE:
+ if mode == MakeMode.NLP_MD:
+ continue
+ elif mode == MakeMode.MM_MD:
+ para_text = merge_visual_blocks_to_markdown(para_block, img_buket_path)
+ elif para_type == BlockType.CODE:
+ para_text = merge_visual_blocks_to_markdown(para_block)
+
+ if para_text.strip() == '':
+ continue
+ else:
+ page_markdown.append(para_text.strip())
+
+ return page_markdown
+
+
+def merge_visual_blocks_to_markdown(para_block, img_buket_path=''):
+ # 将 image/table/chart/code 这类视觉块的子 block 按阅读顺序拼接成 markdown。
+ # 这里不再写死 caption/body/footnote 的优先级，而是先展开成 segment，
+ # 再根据 markdown_line / html_block 两类片段决定分隔方式。
+ rendered_segments = []
+
+ for block in get_blocks_in_index_order(para_block.get('blocks', [])):
+ render_block = _inherit_parent_code_render_metadata(block, para_block)
+ rendered_segments.extend(render_visual_block_segments(render_block, img_buket_path))
+
+ para_text = ''
+ prev_segment_kind = None
+ for segment_text, segment_kind in rendered_segments:
+ if para_text:
+ para_text += get_visual_block_separator(prev_segment_kind, segment_kind)
+ para_text += segment_text
+ prev_segment_kind = segment_kind
+
+ return para_text
+
+
+def get_blocks_in_index_order(blocks):
+ # 按 middle json 中的 index 排序子 block；
+ # 如果 index 缺失，则退化为保持原始顺序。
+ return [
+ block
+ for _, block in sorted(
+ enumerate(blocks),
+ key=lambda item: (item[1].get('index', float('inf')), item[0]),
+ )
+ ]
+
+
+def _inherit_parent_code_render_metadata(block, parent_block):
+ # pipeline_magic_model 会把 code_body 的 sub_type/guess_lang 提升到父 code block。
+ # markdown 渲染 code_body 时需要把这两个字段临时透传回来，但不能修改原始输入。
+ if block.get('type') != BlockType.CODE_BODY:
+ return block
+ if parent_block.get('type') != BlockType.CODE:
+ return block
+
+ needs_sub_type = 'sub_type' not in block and 'sub_type' in parent_block
+ needs_guess_lang = 'guess_lang' not in block and 'guess_lang' in parent_block
+ if not needs_sub_type and not needs_guess_lang:
+ return block
+
+ render_block = dict(block)
+ if needs_sub_type:
+ render_block['sub_type'] = parent_block['sub_type']
+ if needs_guess_lang:
+ render_block['guess_lang'] = parent_block['guess_lang']
+ return render_block
+
+
+def render_visual_block_segments(block, img_buket_path=''):
+ # 将单个视觉子 block 渲染成一个或多个 segment。
+ # 文本类子块统一输出 markdown_line；
+ # table 的 html 输出为 html_block，供后续决定是否需要空行隔开。
+ block_type = block['type']
+
+ if block_type in [
+ BlockType.IMAGE_CAPTION,
+ BlockType.IMAGE_FOOTNOTE,
+ BlockType.TABLE_CAPTION,
+ BlockType.TABLE_FOOTNOTE,
+ BlockType.CODE_BODY,
+ BlockType.CODE_CAPTION,
+ BlockType.CODE_FOOTNOTE,
+ BlockType.CHART_CAPTION,
+ BlockType.CHART_FOOTNOTE,
+ ]:
+ block_text = merge_para_with_text(block)
+ if block_text.strip():
+ return [(block_text, 'markdown_line')]
+ return []
+
+ if block_type == BlockType.IMAGE_BODY:
+ return [
+ (f"", 'markdown_line')
+ for line in block['lines']
+ for span in line['spans']
+ if span['type'] == ContentType.IMAGE and span.get('image_path', '')
+ ]
+
+ if block_type == BlockType.CHART_BODY:
+ return [
+ (f"", 'markdown_line')
+ for line in block['lines']
+ for span in line['spans']
+ if span['type'] == ContentType.CHART and span.get('image_path', '')
+ ]
+
+ if block_type == BlockType.TABLE_BODY:
+ rendered_segments = []
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] != ContentType.TABLE:
+ continue
+ if span.get('html', ''):
+ rendered_segments.append((
+ _format_embedded_html(span['html'], img_buket_path),
+ 'html_block',
+ ))
+ elif span.get('image_path', ''):
+ rendered_segments.append((f"", 'markdown_line'))
+ return rendered_segments
+
+ return []
+
+
+def get_visual_block_separator(prev_segment_kind, current_segment_kind):
+ # 根据前后 segment 类型决定分隔符：
+ # 1. 普通 markdown 行之间用 hard break（" \\n"）
+ # 2. 进入 html block 前只换一行
+ # 3. html block 后必须留空行，否则后续文本仍会被当作 html 块内容
+ if prev_segment_kind == 'html_block':
+ # Raw HTML blocks need a blank line after them, otherwise the following
+ # markdown text is still treated as part of the HTML block.
+ return '\n\n'
+ if current_segment_kind == 'html_block':
+ return '\n'
+ return ' \n'
+
+
+latex_delimiters_config = get_latex_delimiter_config()
+
+default_delimiters = {
+ 'display': {'left': '$$', 'right': '$$'},
+ 'inline': {'left': '$', 'right': '$'}
+}
+
+delimiters = latex_delimiters_config if latex_delimiters_config else default_delimiters
+
+display_left_delimiter = delimiters['display']['left']
+display_right_delimiter = delimiters['display']['right']
+inline_left_delimiter = delimiters['inline']['left']
+inline_right_delimiter = delimiters['inline']['right']
+
+CJK_LANGS = {'zh', 'ja', 'ko'}
+
+
+def _prefix_table_img_src(html, img_buket_path):
+ """Prefix non-data image sources in table HTML with img_buket_path."""
+ if not html or not img_buket_path:
+ return html
+
+ return re.sub(
+ r'src="(?!data:)([^"]+)"',
+ lambda match: f'src="{img_buket_path}/{match.group(1)}"',
+ html,
+ )
+
+
+def _replace_eq_tags_in_table_html(html):
+ """Replace ... tags in table HTML with inline math delimiters."""
+ if not html:
+ return html
+
+ return re.sub(
+ r'(.*?) ',
+ lambda match: (
+ f" {inline_left_delimiter}{unescape(match.group(1))}{inline_right_delimiter} "
+ ),
+ html,
+ flags=re.DOTALL,
+ )
+
+
+def _format_embedded_html(html, img_buket_path):
+ """Normalize embedded table HTML for markdown/content outputs."""
+ return _replace_eq_tags_in_table_html(_prefix_table_img_src(html, img_buket_path))
+
+
+def merge_para_with_text(para_block):
+ if _is_fenced_code_block(para_block):
+ code_text = _merge_para_text(
+ para_block,
+ escape_markdown=False,
+ list_line_break='\n',
+ )
+ if not code_text:
+ return ''
+ code_text = '\n'.join(line.rstrip() for line in code_text.split('\n'))
+ guess_lang = para_block.get('guess_lang', 'txt') or 'txt'
+ return f"```{guess_lang}\n{code_text}\n```"
+
+ para_text = _merge_para_text(para_block)
+ if para_block.get('type') == BlockType.TEXT:
+ para_text = escape_text_block_markdown_prefix(para_text)
+ return para_text
+
+
+def _merge_para_text(para_block, escape_markdown=True, list_line_break=' \n'):
+ # 将普通文本段落 block 渲染成 markdown 字符串。
+ # 处理流程分为三层：
+ # 1. 先收集文本内容做语言检测
+ # 2. 再把每个 span 渲染成 markdown 片段
+ # 3. 最后按语言和上下文决定 span 之间如何补空格/换行
+ block_lang = detect_lang(_collect_text_for_lang_detection(para_block))
+ para_parts = []
+
+ for line_idx, line in enumerate(para_block['lines']):
+ line_prefix = _line_prefix(line_idx, line, list_line_break)
+ if line_prefix:
+ para_parts.append(line_prefix)
+
+ for span_idx, span in enumerate(line['spans']):
+ rendered_span = _render_span(span, escape_markdown=escape_markdown)
+ if rendered_span is None:
+ continue
+
+ span_type, content = rendered_span
+ content, suffix = _join_rendered_span(
+ para_block,
+ block_lang,
+ line,
+ line_idx,
+ span_idx,
+ span_type,
+ content,
+ )
+ para_parts.append(content)
+ if suffix:
+ para_parts.append(suffix)
+
+ return ''.join(para_parts).rstrip()
+
+
+def _is_fenced_code_block(para_block):
+ return (
+ para_block.get('type') == BlockType.CODE_BODY
+ and para_block.get('sub_type') == BlockType.CODE
+ )
+
+
+def _collect_text_for_lang_detection(para_block):
+ # 只收集 TEXT span 的内容，用于语言检测。
+ # 这里会先做全角转半角，但不会修改原始输入数据。
+ block_text_parts = []
+ for line in para_block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.TEXT:
+ block_text_parts.append(_normalize_text_content(span.get('content', '')))
+ return ''.join(block_text_parts)
+
+
+def _normalize_text_content(content):
+ # 对原始文本做统一归一化，当前只负责全角转半角。
+ # 单独拆出来是为了让语言检测和渲染阶段复用同一套文本预处理。
+ return full_to_half_exclude_marks(content or '')
+
+
+def _render_span(span, escape_markdown=True):
+ # 将单个 span 渲染成 markdown 片段。
+ # 这里只负责“渲染成什么文本”，不决定后面是否补空格。
+ span_type = span['type']
+ content = ''
+
+ if span_type == ContentType.TEXT:
+ content = _normalize_text_content(span.get('content', ''))
+ if escape_markdown:
+ content = escape_special_markdown_char(content)
+ elif span_type == ContentType.INLINE_EQUATION:
+ if span.get('content', ''):
+ content = f"{inline_left_delimiter}{span['content']}{inline_right_delimiter}"
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ if span.get('content', ''):
+ content = f"\n{display_left_delimiter}\n{span['content']}\n{display_right_delimiter}\n"
+ else:
+ return None
+
+ content = content.strip()
+ if not content:
+ return None
+
+ return span_type, content
+
+
+def _join_rendered_span(para_block, block_lang, line, line_idx, span_idx, span_type, content):
+ # 根据语言和上下文决定当前 span 后面的分隔符。
+ # 这里集中处理：
+ # 1. CJK 与西文的空格差异
+ # 2. 西文行尾连字符是否需要跨行合并
+ # 3. 独立公式是否作为块内容直接插入
+ if span_type == ContentType.INTERLINE_EQUATION:
+ return content, ''
+
+ is_last_span = span_idx == len(line['spans']) - 1
+
+ if block_lang in CJK_LANGS:
+ if is_last_span and span_type != ContentType.INLINE_EQUATION:
+ return content, ''
+ return content, ' '
+
+ if span_type not in [ContentType.TEXT, ContentType.INLINE_EQUATION]:
+ return content, ''
+
+ if (
+ is_last_span
+ and span_type == ContentType.TEXT
+ and is_hyphen_at_line_end(content)
+ ):
+ if _next_line_starts_with_lowercase_text(para_block, line_idx):
+ return content[:-1], ''
+ return content, ''
+
+ return content, ' '
+
+
+def _line_prefix(line_idx, line, list_line_break=' \n'):
+ # 处理进入新 list item 前的 block 级换行。
+ # 这里保留历史语义：list 起始行前插入一个 hard break。
+ if line_idx >= 1 and line.get(ListLineTag.IS_LIST_START_LINE, False):
+ return list_line_break
+ return ''
+
+
+def _next_line_starts_with_lowercase_text(para_block, line_idx):
+ # 判断下一行是否以小写英文文本开头。
+ # 这个条件用于决定西文行尾的连字符是否应与下一行合并。
+ if line_idx + 1 >= len(para_block['lines']):
+ return False
+
+ next_line_spans = para_block['lines'][line_idx + 1].get('spans')
+ if not next_line_spans:
+ return False
+
+ next_span = next_line_spans[0]
+ if next_span.get('type') != ContentType.TEXT:
+ return False
+
+ next_content = _normalize_text_content(next_span.get('content', ''))
+ return bool(next_content) and next_content[0].islower()
+
+
+def merge_adjacent_ref_text_blocks_for_content(para_blocks):
+ merged_blocks = []
+ ref_group = []
+
+ def flush_ref_group():
+ nonlocal ref_group
+ if not ref_group:
+ return
+ if len(ref_group) == 1:
+ merged_blocks.append(ref_group[0])
+ else:
+ merged_blocks.append({
+ 'type': BlockType.REF_TEXT,
+ 'blocks': list(ref_group),
+ 'bbox': ref_group[0].get('bbox'),
+ })
+ ref_group = []
+
+ for para_block in para_blocks or []:
+ if para_block.get('type') == BlockType.REF_TEXT:
+ ref_group.append(para_block)
+ continue
+
+ flush_ref_group()
+ merged_blocks.append(para_block)
+
+ flush_ref_group()
+ return merged_blocks
+
+
+def _build_bbox(para_bbox, page_size):
+ if not para_bbox or not page_size:
+ return None
+
+ page_width, page_height = page_size
+ x0, y0, x1, y1 = para_bbox
+ return [
+ int(x0 * 1000 / page_width),
+ int(y0 * 1000 / page_height),
+ int(x1 * 1000 / page_width),
+ int(y1 * 1000 / page_height),
+ ]
+
+
+def _get_seal_span(para_block):
+ for line in para_block.get('lines', []):
+ for span in line.get('spans', []):
+ if span.get('type') == ContentType.SEAL:
+ return span
+ return None
+
+
+def _get_seal_text(para_block):
+ seal_span = _get_seal_span(para_block)
+ if not seal_span:
+ return ''
+
+ content = seal_span.get('content', '')
+ if isinstance(content, list):
+ return ' '.join(str(item) for item in content if str(item).strip())
+ if isinstance(content, str):
+ return content.strip()
+ return ''
+
+
+def _get_ref_text_item_blocks(para_block):
+ return para_block.get('blocks') or [para_block]
+
+
+def _split_list_item_blocks(para_block):
+ item_blocks = []
+ current_lines = []
+
+ for line_idx, line in enumerate(para_block.get('lines', [])):
+ if line_idx > 0 and line.get(ListLineTag.IS_LIST_START_LINE, False) and current_lines:
+ item_blocks.append({
+ 'type': BlockType.TEXT,
+ 'lines': current_lines,
+ })
+ current_lines = []
+ current_lines.append(line)
+
+ if current_lines:
+ item_blocks.append({
+ 'type': BlockType.TEXT,
+ 'lines': current_lines,
+ })
+
+ return item_blocks
+
+
+def _get_body_data(para_block):
+ def get_data_from_spans(lines):
+ for line in lines:
+ for span in line.get('spans', []):
+ span_type = span.get('type')
+ if span_type == ContentType.TABLE:
+ return span.get('image_path', ''), span.get('html', '')
+ if span_type == ContentType.CHART:
+ return span.get('image_path', ''), span.get('content', '')
+ if span_type == ContentType.IMAGE:
+ return span.get('image_path', ''), ''
+ if span_type == ContentType.INTERLINE_EQUATION:
+ return span.get('image_path', ''), span.get('content', '')
+ return '', ''
+
+ if 'blocks' in para_block:
+ for block in para_block['blocks']:
+ block_type = block.get('type')
+ if block_type in [
+ BlockType.IMAGE_BODY,
+ BlockType.TABLE_BODY,
+ BlockType.CHART_BODY,
+ BlockType.CODE_BODY,
+ ]:
+ result = get_data_from_spans(block.get('lines', []))
+ if result != ('', '') or block_type == BlockType.CHART_BODY:
+ return result
+ return '', ''
+
+ return get_data_from_spans(para_block.get('lines', []))
+
+
+def merge_para_with_text_v2(para_block):
+ block_lang = detect_lang(_collect_text_for_lang_detection(para_block))
+ para_content = []
+ para_type = para_block.get('type')
+
+ for line_idx, line in enumerate(para_block.get('lines', [])):
+ for span_idx, span in enumerate(line.get('spans', [])):
+ span_type = span.get('type')
+
+ if span_type == ContentType.TEXT:
+ content = _normalize_text_content(span.get('content', ''))
+ if not content.strip():
+ continue
+
+ output_type = (
+ ContentTypeV2.SPAN_PHONETIC
+ if para_type == BlockType.PHONETIC
+ else ContentTypeV2.SPAN_TEXT
+ )
+ is_last_span = span_idx == len(line['spans']) - 1
+
+ if block_lang in CJK_LANGS:
+ rendered_content = content if is_last_span else f"{content} "
+ else:
+ if (
+ is_last_span
+ and is_hyphen_at_line_end(content)
+ and _next_line_starts_with_lowercase_text(para_block, line_idx)
+ ):
+ rendered_content = content[:-1]
+ elif is_last_span and is_hyphen_at_line_end(content):
+ rendered_content = content
+ else:
+ rendered_content = f"{content} "
+
+ if para_content and para_content[-1]['type'] == output_type:
+ para_content[-1]['content'] += rendered_content
+ else:
+ para_content.append({
+ 'type': output_type,
+ 'content': rendered_content,
+ })
+ elif span_type == ContentType.INLINE_EQUATION:
+ content = span.get('content', '').strip()
+ if content:
+ para_content.append({
+ 'type': ContentTypeV2.SPAN_EQUATION_INLINE,
+ 'content': content,
+ })
+
+ if para_content and para_content[-1]['type'] in [
+ ContentTypeV2.SPAN_TEXT,
+ ContentTypeV2.SPAN_PHONETIC,
+ ]:
+ para_content[-1]['content'] = para_content[-1]['content'].rstrip()
+
+ return para_content
+
+
+def make_blocks_to_content_list(para_block, img_buket_path, page_idx, page_size):
+ para_type = para_block['type']
+ para_content = None
+ if para_type in [
+ BlockType.TEXT,
+ BlockType.INDEX,
+ BlockType.LIST,
+ BlockType.ABSTRACT,
+ ]:
+ para_content = {
+ 'type': ContentType.TEXT,
+ 'text': merge_para_with_text(para_block),
+ }
+ elif para_type in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_NUMBER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ para_content = {
+ 'type': para_type,
+ 'text': merge_para_with_text(para_block),
+ }
+ elif para_type == BlockType.REF_TEXT:
+ para_content = {
+ 'type': BlockType.LIST,
+ 'sub_type': BlockType.REF_TEXT,
+ 'list_items': [],
+ }
+ for block in _get_ref_text_item_blocks(para_block):
+ item_text = merge_para_with_text(block)
+ if item_text.strip():
+ para_content['list_items'].append(item_text)
+ elif para_type == BlockType.TITLE:
+ para_content = {
+ 'type': ContentType.TEXT,
+ 'text': merge_para_with_text(para_block),
+ }
+ title_level = get_title_level(para_block)
+ if title_level != 0:
+ para_content['text_level'] = title_level
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ if len(para_block['lines']) == 0 or len(para_block['lines'][0]['spans']) == 0:
+ return None
+ para_content = {
+ 'type': ContentType.EQUATION,
+ 'img_path': f"{img_buket_path}/{para_block['lines'][0]['spans'][0].get('image_path', '')}",
+ }
+ if para_block['lines'][0]['spans'][0].get('content', ''):
+ para_content['text'] = merge_para_with_text(para_block)
+ para_content['text_format'] = 'latex'
+ elif para_type == BlockType.SEAL:
+ seal_span = _get_seal_span(para_block)
+ if not seal_span:
+ return None
+ para_content = {
+ 'type': ContentType.SEAL,
+ 'img_path': f"{img_buket_path}/{seal_span.get('image_path', '')}",
+ 'text': _get_seal_text(para_block),
+ }
+ elif para_type == BlockType.IMAGE:
+ para_content = {'type': ContentType.IMAGE, 'img_path': '', BlockType.IMAGE_CAPTION: [], BlockType.IMAGE_FOOTNOTE: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.IMAGE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.IMAGE:
+ if span.get('image_path', ''):
+ para_content['img_path'] = f"{img_buket_path}/{span['image_path']}"
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ para_content[BlockType.IMAGE_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.IMAGE_FOOTNOTE:
+ para_content[BlockType.IMAGE_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.TABLE:
+ para_content = {'type': ContentType.TABLE, 'img_path': '', BlockType.TABLE_CAPTION: [], BlockType.TABLE_FOOTNOTE: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.TABLE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.TABLE:
+ if span.get('html', ''):
+ para_content[BlockType.TABLE_BODY] = _format_embedded_html(
+ span['html'],
+ img_buket_path,
+ )
+
+ if span.get('image_path', ''):
+ para_content['img_path'] = f"{img_buket_path}/{span['image_path']}"
+
+ if block['type'] == BlockType.TABLE_CAPTION:
+ para_content[BlockType.TABLE_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.TABLE_FOOTNOTE:
+ para_content[BlockType.TABLE_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.CHART:
+ para_content = {
+ 'type': ContentType.CHART,
+ 'img_path': '',
+ 'content': '',
+ BlockType.CHART_CAPTION: [],
+ BlockType.CHART_FOOTNOTE: [],
+ }
+ for block in para_block.get('blocks', []):
+ if block['type'] == BlockType.CHART_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.CHART and span.get('image_path', ''):
+ para_content['img_path'] = f"{img_buket_path}/{span['image_path']}"
+ if block['type'] == BlockType.CHART_CAPTION:
+ para_content[BlockType.CHART_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.CHART_FOOTNOTE:
+ para_content[BlockType.CHART_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.CODE:
+ para_content = {
+ 'type': BlockType.CODE,
+ 'sub_type': para_block['sub_type'],
+ BlockType.CODE_CAPTION: [],
+ BlockType.CODE_FOOTNOTE: [],
+ }
+ for block in para_block.get('blocks', []):
+ render_block = _inherit_parent_code_render_metadata(block, para_block)
+ if block['type'] == BlockType.CODE_BODY:
+ para_content[BlockType.CODE_BODY] = merge_para_with_text(render_block)
+ if block['type'] == BlockType.CODE_CAPTION:
+ para_content[BlockType.CODE_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.CODE_FOOTNOTE:
+ para_content[BlockType.CODE_FOOTNOTE].append(merge_para_with_text(block))
+
+ if not para_content:
+ return None
+
+ bbox = _build_bbox(para_block.get('bbox'), page_size)
+ if bbox:
+ para_content['bbox'] = bbox
+ para_content['page_idx'] = page_idx
+
+ return para_content
+
+
+def make_blocks_to_content_list_v2(para_block, img_buket_path, page_size):
+ para_type = para_block['type']
+ para_content = None
+
+ if para_type in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_NUMBER,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ if para_type == BlockType.HEADER:
+ content_type = ContentTypeV2.PAGE_HEADER
+ elif para_type == BlockType.FOOTER:
+ content_type = ContentTypeV2.PAGE_FOOTER
+ elif para_type == BlockType.ASIDE_TEXT:
+ content_type = ContentTypeV2.PAGE_ASIDE_TEXT
+ elif para_type == BlockType.PAGE_NUMBER:
+ content_type = ContentTypeV2.PAGE_NUMBER
+ elif para_type == BlockType.PAGE_FOOTNOTE:
+ content_type = ContentTypeV2.PAGE_FOOTNOTE
+ else:
+ raise ValueError(f"Unknown para_type: {para_type}")
+ para_content = {
+ 'type': content_type,
+ 'content': {
+ f"{content_type}_content": merge_para_with_text_v2(para_block),
+ },
+ }
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ if title_level != 0:
+ para_content = {
+ 'type': ContentTypeV2.TITLE,
+ 'content': {
+ 'title_content': merge_para_with_text_v2(para_block),
+ 'level': title_level,
+ },
+ }
+ else:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ 'paragraph_content': merge_para_with_text_v2(para_block),
+ },
+ }
+ elif para_type in [
+ BlockType.TEXT,
+ BlockType.ABSTRACT,
+ ]:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ 'paragraph_content': merge_para_with_text_v2(para_block),
+ },
+ }
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ image_path, math_content = _get_body_data(para_block)
+ para_content = {
+ 'type': ContentTypeV2.EQUATION_INTERLINE,
+ 'content': {
+ 'math_content': math_content,
+ 'math_type': 'latex',
+ 'image_source': {'path': f"{img_buket_path}/{image_path}"},
+ },
+ }
+ elif para_type == BlockType.IMAGE:
+ image_caption = []
+ image_footnote = []
+ image_path, _ = _get_body_data(para_block)
+ for block in para_block.get('blocks', []):
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ image_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.IMAGE_FOOTNOTE:
+ image_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.IMAGE,
+ 'content': {
+ 'image_source': {'path': f"{img_buket_path}/{image_path}"},
+ 'image_caption': image_caption,
+ 'image_footnote': image_footnote,
+ },
+ }
+ elif para_type == BlockType.TABLE:
+ table_caption = []
+ table_footnote = []
+ image_path, html = _get_body_data(para_block)
+ table_html = _format_embedded_html(html, img_buket_path)
+ table_nest_level = 2 if table_html.count(' 1 else 1
+ if 'colspan' in table_html or 'rowspan' in table_html or table_nest_level > 1:
+ table_type = ContentTypeV2.TABLE_COMPLEX
+ else:
+ table_type = ContentTypeV2.TABLE_SIMPLE
+ for block in para_block.get('blocks', []):
+ if block['type'] == BlockType.TABLE_CAPTION:
+ table_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.TABLE_FOOTNOTE:
+ table_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.TABLE,
+ 'content': {
+ 'image_source': {'path': f"{img_buket_path}/{image_path}"},
+ 'table_caption': table_caption,
+ 'table_footnote': table_footnote,
+ 'html': table_html,
+ 'table_type': table_type,
+ 'table_nest_level': table_nest_level,
+ },
+ }
+ elif para_type == BlockType.CHART:
+ chart_caption = []
+ chart_footnote = []
+ image_path, _ = _get_body_data(para_block)
+ for block in para_block.get('blocks', []):
+ if block['type'] == BlockType.CHART_CAPTION:
+ chart_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.CHART_FOOTNOTE:
+ chart_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.CHART,
+ 'content': {
+ 'image_source': {'path': f"{img_buket_path}/{image_path}"},
+ 'content': '',
+ 'chart_caption': chart_caption,
+ 'chart_footnote': chart_footnote,
+ },
+ }
+ elif para_type == BlockType.CODE:
+ code_caption = []
+ code_footnote = []
+ code_content = []
+ for block in para_block.get('blocks', []):
+ if block['type'] == BlockType.CODE_CAPTION:
+ code_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.CODE_FOOTNOTE:
+ code_footnote.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.CODE_BODY:
+ code_content = merge_para_with_text_v2(block)
+
+ sub_type = para_block['sub_type']
+ if sub_type == BlockType.CODE:
+ para_content = {
+ 'type': ContentTypeV2.CODE,
+ 'content': {
+ 'code_caption': code_caption,
+ 'code_content': code_content,
+ 'code_footnote': code_footnote,
+ 'code_language': para_block.get('guess_lang', 'txt'),
+ },
+ }
+ elif sub_type == BlockType.ALGORITHM:
+ para_content = {
+ 'type': ContentTypeV2.ALGORITHM,
+ 'content': {
+ 'algorithm_caption': code_caption,
+ 'algorithm_content': code_content,
+ 'algorithm_footnote': code_footnote,
+ },
+ }
+ else:
+ raise ValueError(f"Unknown code sub_type: {sub_type}")
+ elif para_type == BlockType.REF_TEXT:
+ list_items = []
+ for block in _get_ref_text_item_blocks(para_block):
+ item_content = merge_para_with_text_v2(block)
+ if item_content:
+ list_items.append({
+ 'item_type': 'text',
+ 'item_content': item_content,
+ })
+ para_content = {
+ 'type': ContentTypeV2.LIST,
+ 'content': {
+ 'list_type': ContentTypeV2.LIST_REF,
+ 'list_items': list_items,
+ },
+ }
+ elif para_type == BlockType.LIST:
+ list_items = []
+ for block in _split_list_item_blocks(para_block):
+ item_content = merge_para_with_text_v2(block)
+ if item_content:
+ list_items.append({
+ 'item_type': 'text',
+ 'item_content': item_content,
+ })
+ para_content = {
+ 'type': ContentTypeV2.LIST,
+ 'content': {
+ 'list_type': ContentTypeV2.LIST_TEXT,
+ 'attribute': para_block.get('attribute', 'unordered'),
+ 'list_items': list_items,
+ },
+ }
+ elif para_type == BlockType.INDEX:
+ list_items = []
+ for block in _split_list_item_blocks(para_block):
+ item_content = merge_para_with_text_v2(block)
+ if item_content:
+ list_items.append({
+ 'item_type': 'text',
+ 'item_content': item_content,
+ })
+ para_content = {
+ 'type': ContentTypeV2.INDEX,
+ 'content': {
+ 'list_type': ContentTypeV2.LIST_TEXT,
+ 'list_items': list_items,
+ },
+ }
+ elif para_type == BlockType.SEAL:
+ seal_span = _get_seal_span(para_block)
+ if not seal_span:
+ return None
+ seal_text = _get_seal_text(para_block)
+ para_content = {
+ 'type': ContentTypeV2.SEAL,
+ 'content': {
+ 'image_source': {
+ 'path': f"{img_buket_path}/{seal_span.get('image_path', '')}",
+ },
+ 'seal_content': (
+ [{'type': ContentTypeV2.SPAN_TEXT, 'content': seal_text}]
+ if seal_text else []
+ ),
+ },
+ }
+
+ if not para_content:
+ return None
+
+ bbox = _build_bbox(para_block.get('bbox'), page_size)
+ if bbox:
+ para_content['bbox'] = bbox
+
+ return para_content
+
+
+def union_make(pdf_info_dict: list,
+ make_mode: str,
+ img_buket_path: str = '',
+ ):
+ output_content = []
+ for page_info in pdf_info_dict:
+ paras_of_layout = page_info.get('para_blocks')
+ paras_of_discarded = page_info.get('discarded_blocks')
+ page_idx = page_info.get('page_idx')
+ page_size = page_info.get('page_size')
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ if not paras_of_layout:
+ continue
+ page_markdown = make_blocks_to_markdown(paras_of_layout, make_mode, img_buket_path)
+ output_content.extend(page_markdown)
+ elif make_mode == MakeMode.CONTENT_LIST:
+ para_blocks = merge_adjacent_ref_text_blocks_for_content(
+ (paras_of_layout or []) + (paras_of_discarded or [])
+ )
+ if not para_blocks:
+ continue
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list(para_block, img_buket_path, page_idx, page_size)
+ if para_content:
+ output_content.append(para_content)
+ elif make_mode == MakeMode.CONTENT_LIST_V2:
+ para_blocks = merge_adjacent_ref_text_blocks_for_content(
+ (paras_of_layout or []) + (paras_of_discarded or [])
+ )
+ page_contents = []
+ if para_blocks:
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list_v2(para_block, img_buket_path, page_size)
+ if para_content:
+ page_contents.append(para_content)
+ output_content.append(page_contents)
+
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ return '\n\n'.join(output_content)
+ elif make_mode in [MakeMode.CONTENT_LIST, MakeMode.CONTENT_LIST_V2]:
+ return output_content
+ else:
+ logger.error(f"Unsupported make mode: {make_mode}")
+ return None
+
+
+def get_title_level(block):
+ title_level = block.get('level', 1)
+ if title_level < 1:
+ title_level = 0
+ return title_level
+
+
+def escape_special_markdown_char(content):
+ """
+ 转义正文里对markdown语法有特殊意义的字符
+ """
+ return escape_conservative_markdown_text(content)
diff --git a/mineru/backend/utils/__init__.py b/mineru/backend/utils/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/utils/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/utils/html_image_utils.py b/mineru/backend/utils/html_image_utils.py
new file mode 100644
index 000000000..979caaa36
--- /dev/null
+++ b/mineru/backend/utils/html_image_utils.py
@@ -0,0 +1,106 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import base64
+import re
+
+from loguru import logger
+
+from mineru.utils.enum_class import BlockType, ContentType
+from mineru.utils.hash_utils import str_sha256
+
+
+def _normalize_image_extension(fmt: str) -> str:
+ normalized = fmt.lower().split("+", 1)[0]
+ return "jpg" if normalized == "jpeg" else normalized
+
+
+def save_base64_image(b64_data_uri: str, image_writer, page_index: int):
+ """Persist a data-URI image via image_writer and return a relative path."""
+ if not image_writer:
+ return None
+
+ match = re.match(r"data:image/([^;]+);base64,(.+)", b64_data_uri, re.DOTALL)
+ if not match:
+ logger.warning(f"Unrecognized image_base64 format in page {page_index}, skipping.")
+ return None
+
+ fmt = _normalize_image_extension(match.group(1))
+ try:
+ img_bytes = base64.b64decode(match.group(2))
+ except Exception as exc:
+ logger.warning(f"Failed to decode image_base64 on page {page_index}: {exc}")
+ return None
+
+ img_path = f"{str_sha256(b64_data_uri)}.{fmt}"
+ image_writer.write(img_path, img_bytes)
+ return img_path
+
+
+def replace_inline_base64_img_src(markup: str, image_writer, page_index: int) -> str:
+ """Replace inline base64 image sources in HTML-like markup with local paths."""
+ if not markup or not image_writer or "base64," not in markup:
+ return markup
+
+ def _replace_src(match, _writer=image_writer, _idx=page_index):
+ img_path = save_base64_image(match.group(1), _writer, _idx)
+ if img_path:
+ return f'src="{img_path}"'
+ return match.group(0)
+
+ return re.sub(
+ r'src="(data:image/[^"]+)"',
+ _replace_src,
+ markup,
+ )
+
+
+def replace_inline_table_images(
+ blocks: list[dict],
+ image_writer,
+ page_index: int,
+ table_block_type=BlockType.TABLE,
+ table_body_type=BlockType.TABLE_BODY,
+ table_span_type=ContentType.TABLE,
+) -> None:
+ """Persist inline base64 images embedded inside table HTML."""
+ if not image_writer:
+ return
+
+ for block in blocks:
+ if block.get("type") != table_block_type:
+ continue
+
+ for sub_block in block.get("blocks", []):
+ if sub_block.get("type") != table_body_type:
+ continue
+
+ for line in sub_block.get("lines", []):
+ for span in line.get("spans", []):
+ if span.get("type") != table_span_type:
+ continue
+ span["html"] = replace_inline_base64_img_src(
+ span.get("html", ""),
+ image_writer,
+ page_index,
+ )
+
+
+def save_span_image_if_needed(
+ span: dict,
+ image_writer,
+ page_index: int,
+ source_key: str = "image_base64",
+ target_key: str = "image_path",
+) -> None:
+ """Persist a span-level base64 image and normalize the target path field."""
+ image_base64 = span.get(source_key, "")
+ if not image_base64:
+ span.setdefault(target_key, "")
+ return
+
+ img_path = save_base64_image(image_base64, image_writer, page_index)
+ if img_path:
+ span[target_key] = img_path
+ del span[source_key]
+ return
+
+ span.setdefault(target_key, "")
diff --git a/mineru/backend/utils/markdown_utils.py b/mineru/backend/utils/markdown_utils.py
new file mode 100644
index 000000000..296078a50
--- /dev/null
+++ b/mineru/backend/utils/markdown_utils.py
@@ -0,0 +1,32 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+import re
+
+CONSERVATIVE_MARKDOWN_SPECIAL_CHARS = ("*", "_", "`", "~", "$")
+TEXT_BLOCK_MARKDOWN_PREFIX_RE = re.compile(
+ r"^(?P[ \t]{0,3})(?P#{1,6}|[+-])(?=[ \t])"
+)
+
+
+def escape_conservative_markdown_text(content: str) -> str:
+ """Escape plain-text characters that carry inline Markdown semantics."""
+ if not content:
+ return content
+
+ for char in CONSERVATIVE_MARKDOWN_SPECIAL_CHARS:
+ content = content.replace(char, "\\" + char)
+
+ return content
+
+
+def escape_text_block_markdown_prefix(content: str) -> str:
+ """Escape a leading Markdown block marker in an assembled text block."""
+ if not content:
+ return content
+
+ match = TEXT_BLOCK_MARKDOWN_PREFIX_RE.match(content)
+ if not match:
+ return content
+
+ marker_start = match.start("marker")
+ return f"{content[:marker_start]}\\{content[marker_start:]}"
diff --git a/mineru/backend/utils/ocr_det_utils.py b/mineru/backend/utils/ocr_det_utils.py
new file mode 100644
index 000000000..28f1d69f8
--- /dev/null
+++ b/mineru/backend/utils/ocr_det_utils.py
@@ -0,0 +1,50 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+import cv2
+import numpy as np
+from loguru import logger
+
+from mineru.utils.pdf_image_tools import get_crop_img
+
+OCR_DET_PADDING = 50
+
+
+def get_ch_lite_ocr_det_model():
+ try:
+ from mineru.backend.pipeline.model_init import AtomModelSingleton
+ except Exception as e:
+ logger.error(
+ "Failed to import AtomModelSingleton, OCR detection will not work. If you want to use OCR features, "
+ "please execute `pip install mineru[core]` to install the required packages."
+ )
+ raise e
+
+ atom_model_manager = AtomModelSingleton()
+ return atom_model_manager.get_atom_model(
+ atom_model_name='ocr',
+ ocr_show_log=False,
+ det_db_box_thresh=0.3,
+ lang='ch_lite'
+ )
+
+
+def detect_ocr_boxes_from_padded_crop(bbox, page_pil_img, scale, ocr_model=None, padding=OCR_DET_PADDING):
+ if not bbox:
+ return [], padding
+
+ crop_pil_img = get_crop_img(bbox, page_pil_img, scale)
+ crop_np_img = np.array(crop_pil_img)
+ if crop_np_img.size == 0:
+ return [], padding
+
+ if padding > 0:
+ crop_np_img = cv2.copyMakeBorder(
+ crop_np_img, padding, padding, padding, padding, cv2.BORDER_CONSTANT, value=[255, 255, 255]
+ )
+
+ crop_img = cv2.cvtColor(crop_np_img, cv2.COLOR_RGB2BGR)
+ if ocr_model is None:
+ ocr_model = get_ch_lite_ocr_det_model()
+
+ ocr_det_res = ocr_model.ocr(crop_img, rec=False)[0]
+ return ocr_det_res or [], padding
diff --git a/mineru/backend/utils/office_chart.py b/mineru/backend/utils/office_chart.py
new file mode 100644
index 000000000..9c016c235
--- /dev/null
+++ b/mineru/backend/utils/office_chart.py
@@ -0,0 +1,31 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+from io import BytesIO
+
+import pandas as pd
+
+
+def minify_html(html: str) -> str:
+ """Strip formatting whitespace from HTML while preserving content."""
+ if not html:
+ return html
+
+ html = re.sub(r'>\s+<', '><', html)
+ html = re.sub(r'\n\s*', '', html)
+ return html
+
+
+def html_table_from_excel_bytes(excel_bytes: bytes) -> str:
+ """Convert the first non-empty worksheet in an embedded workbook to HTML."""
+ if not excel_bytes:
+ return ""
+
+ worksheets = pd.read_excel(BytesIO(excel_bytes), sheet_name=None)
+ for dataframe in worksheets.values():
+ if dataframe is None:
+ continue
+ if dataframe.empty and len(dataframe.columns) == 0:
+ continue
+ return minify_html(dataframe.to_html(index=False, header=True))
+
+ return ""
diff --git a/mineru/backend/utils/office_image.py b/mineru/backend/utils/office_image.py
new file mode 100644
index 000000000..57a9f6aab
--- /dev/null
+++ b/mineru/backend/utils/office_image.py
@@ -0,0 +1,116 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from PIL import Image, ImageDraw, ImageFont
+from loguru import logger
+
+from mineru.utils.check_sys_env import is_windows_environment
+from mineru.utils.pdf_reader import image_to_b64str
+
+
+VECTOR_IMAGE_FORMATS = frozenset({"WMF", "EMF"})
+
+
+def is_vector_image(pil_image: Image.Image) -> bool:
+ return (getattr(pil_image, "format", None) or "").upper() in VECTOR_IMAGE_FORMATS
+
+
+def _load_placeholder_font(font_size: int) -> ImageFont.ImageFont:
+ for font_name in (
+ "DejaVuSans.ttf",
+ "Arial.ttf",
+ "LiberationSans-Regular.ttf",
+ ):
+ try:
+ return ImageFont.truetype(font_name, font_size)
+ except OSError:
+ continue
+ return ImageFont.load_default()
+
+
+def create_text_placeholder(
+ size: tuple[int, int], lines: list[str]
+) -> Image.Image:
+ width = max(int(size[0]), 1)
+ height = max(int(size[1]), 1)
+ placeholder = Image.new("RGB", (width, height), (240, 240, 240))
+ draw = ImageDraw.Draw(placeholder)
+
+ border_width = max(1, min(width, height) // 80)
+ draw.rectangle(
+ (0, 0, width - 1, height - 1),
+ outline=(190, 190, 190),
+ width=border_width,
+ )
+
+ max_text_width = max(width - 16, 1)
+ max_text_height = max(height - 16, 1)
+ fallback_text = "WMF/EMF"
+ text = "\n".join(line for line in lines if line)
+ if not text:
+ text = fallback_text
+
+ font = None
+ spacing = 4
+ bbox = None
+ for font_size in range(max(min(width, height) // 7, 10), 7, -1):
+ font = _load_placeholder_font(font_size)
+ spacing = max(2, font_size // 4)
+ bbox = draw.multiline_textbbox(
+ (0, 0), text, font=font, spacing=spacing, align="center"
+ )
+ text_width = bbox[2] - bbox[0]
+ text_height = bbox[3] - bbox[1]
+ if text_width <= max_text_width and text_height <= max_text_height:
+ break
+ else:
+ text = fallback_text
+ font = _load_placeholder_font(max(min(width, height) // 5, 10))
+ spacing = 2
+ bbox = draw.multiline_textbbox(
+ (0, 0), text, font=font, spacing=spacing, align="center"
+ )
+
+ text_width = bbox[2] - bbox[0]
+ text_height = bbox[3] - bbox[1]
+ origin = ((width - text_width) / 2, (height - text_height) / 2)
+ draw.multiline_text(
+ origin,
+ text,
+ fill=(90, 90, 90),
+ font=font,
+ spacing=spacing,
+ align="center",
+ )
+ return placeholder
+
+
+def serialize_vector_image_with_placeholder(
+ pil_image: Image.Image, image_format_override: str | None = None
+) -> str:
+ image_format = (
+ image_format_override or getattr(pil_image, "format", None) or "WMF/EMF"
+ ).upper()
+
+ if is_windows_environment():
+ try:
+ pil_image.load()
+ return image_to_b64str(pil_image, image_format="PNG")
+ except OSError as e:
+ logger.warning(
+ f"Failed to render {image_format} image: {e}, size: {pil_image.size}. Using placeholder instead."
+ )
+ placeholder_lines = [
+ f"{image_format} placeholder",
+ "Windows rendering failed",
+ ]
+ else:
+ logger.warning(
+ f"Skipping {image_format} image on non-Windows environment, size: {pil_image.size}"
+ )
+ placeholder_lines = [
+ f"{image_format} placeholder",
+ "Use Windows to parse",
+ "the original image",
+ ]
+
+ placeholder = create_text_placeholder(pil_image.size, placeholder_lines)
+ return image_to_b64str(placeholder, image_format="JPEG")
diff --git a/mineru/backend/utils/para_block_utils.py b/mineru/backend/utils/para_block_utils.py
new file mode 100644
index 000000000..84583e16f
--- /dev/null
+++ b/mineru/backend/utils/para_block_utils.py
@@ -0,0 +1,258 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import copy
+
+from mineru.utils.enum_class import BlockType, SplitFlag
+
+
+LINE_STOP_FLAG = ('.', '!', '?', '。', '！', '？', ')', '）', '"', '”', ':', '：', ';', '；')
+MERGE_BARRIER_TYPES = {
+ BlockType.TITLE,
+ BlockType.INTERLINE_EQUATION,
+ BlockType.LIST,
+}
+_CROSS_PAGE_MERGE_KEY = "_cross_page_merge_prev"
+_EDGE_TEXT_LINE_HINTS_KEY = "_edge_text_line_hints"
+
+
+def iter_block_spans(block):
+ for line in block.get("lines", []):
+ for span in line.get("spans", []):
+ yield span
+
+ for sub_block in block.get("blocks", []):
+ yield from iter_block_spans(sub_block)
+
+
+def build_para_blocks_from_preproc(pdf_info_list):
+ for page_info in pdf_info_list:
+ page_info["para_blocks"] = copy.deepcopy(page_info.get("preproc_blocks", []))
+
+
+def merge_para_text_blocks(pdf_info_list, allow_cross_page=False):
+ ordered_blocks = []
+ for page_info in pdf_info_list:
+ page_idx = page_info.get("page_idx")
+ for order_idx, block in enumerate(page_info.get("para_blocks", [])):
+ ordered_blocks.append((page_idx, order_idx, block))
+
+ for current_index in range(len(ordered_blocks) - 1, -1, -1):
+ current_page_idx, _, current_block = ordered_blocks[current_index]
+ if current_block.get("type") != BlockType.TEXT:
+ continue
+ if not current_block.get("merge_prev"):
+ continue
+ if not _block_has_lines(current_block):
+ continue
+
+ previous_block = _find_previous_text_block(
+ ordered_blocks,
+ current_index,
+ current_block,
+ current_page_idx,
+ allow_cross_page=allow_cross_page,
+ )
+ if previous_block is None:
+ continue
+
+ previous_page_idx, _, previous_text_block = previous_block
+ _merge_text_block(
+ current_block,
+ previous_text_block,
+ is_cross_page=current_page_idx != previous_page_idx,
+ )
+
+
+def annotate_hybrid_cross_page_merge_prev(pdf_info_list, prefer_edge_line_hints=False):
+ for page_index in range(1, len(pdf_info_list)):
+ previous_page_info = pdf_info_list[page_index - 1]
+ current_page_info = pdf_info_list[page_index]
+
+ previous_text_block = _find_last_page_edge_text_block(previous_page_info)
+ current_text_block = _find_first_page_edge_text_block(current_page_info)
+ if previous_text_block is None or current_text_block is None:
+ continue
+
+ previous_metric_lines = _resolve_metric_lines(
+ previous_page_info,
+ previous_text_block,
+ edge_name="last",
+ prefer_edge_line_hints=prefer_edge_line_hints,
+ )
+ current_metric_lines = _resolve_metric_lines(
+ current_page_info,
+ current_text_block,
+ edge_name="first",
+ prefer_edge_line_hints=prefer_edge_line_hints,
+ )
+ if not previous_metric_lines or not current_metric_lines:
+ continue
+
+ if can_merge_text_blocks(
+ current_text_block,
+ previous_text_block,
+ current_metric_lines=current_metric_lines,
+ previous_metric_lines=previous_metric_lines,
+ ):
+ current_text_block["merge_prev"] = True
+ current_text_block[_CROSS_PAGE_MERGE_KEY] = True
+
+
+def can_merge_text_blocks(current_block, previous_block, current_metric_lines=None, previous_metric_lines=None):
+ current_lines = current_block.get("lines", [])
+ previous_lines = previous_block.get("lines", [])
+ if not current_lines or not previous_lines:
+ return False
+
+ current_metric_lines = current_metric_lines or current_lines
+ previous_metric_lines = previous_metric_lines or previous_lines
+ if not current_metric_lines or not previous_metric_lines:
+ return False
+
+ first_metric_line = current_metric_lines[0]
+ first_line_height = _line_height(first_metric_line)
+ if first_line_height <= 0:
+ return False
+
+ current_bbox_fs = _build_bbox_fs(current_block, current_metric_lines)
+ if abs(current_bbox_fs[0] - first_metric_line["bbox"][0]) >= first_line_height / 2:
+ return False
+
+ last_metric_line = previous_metric_lines[-1]
+ last_line_height = _line_height(last_metric_line)
+ if last_line_height <= 0:
+ return False
+
+ previous_bbox_fs = _build_bbox_fs(previous_block, previous_metric_lines)
+
+ first_span = _first_span(current_lines[0])
+ last_span = _last_span(previous_lines[-1])
+ if first_span is None or last_span is None:
+ return False
+
+ first_content = first_span.get("content", "")
+ last_content = last_span.get("content", "")
+ if not first_content:
+ return False
+
+ current_block_width = current_block["bbox"][2] - current_block["bbox"][0]
+ previous_block_width = previous_block["bbox"][2] - previous_block["bbox"][0]
+ min_block_width = min(current_block_width, previous_block_width)
+ if min_block_width <= 0:
+ return False
+
+ if abs(previous_bbox_fs[2] - last_metric_line["bbox"][2]) >= last_line_height:
+ return False
+ if last_content.endswith(LINE_STOP_FLAG):
+ return False
+ if abs(current_block_width - previous_block_width) >= min_block_width:
+ return False
+ if first_content[0].isdigit() or first_content[0].isupper():
+ return False
+ if current_block["bbox"][1] >= previous_block["bbox"][3]:
+ return False
+ if len(current_metric_lines) <= 1 and len(previous_metric_lines) <= 1:
+ return False
+
+ return True
+
+
+def cleanup_internal_para_block_metadata(pdf_info_list):
+ for page_info in pdf_info_list:
+ page_info.pop(_EDGE_TEXT_LINE_HINTS_KEY, None)
+ for block in page_info.get("para_blocks", []):
+ block.pop(_CROSS_PAGE_MERGE_KEY, None)
+
+
+def edge_text_line_hints_key():
+ return _EDGE_TEXT_LINE_HINTS_KEY
+
+
+def _find_previous_text_block(ordered_blocks, current_index, current_block, current_page_idx, allow_cross_page):
+ cross_page_allowed = allow_cross_page and current_block.get(_CROSS_PAGE_MERGE_KEY, False)
+
+ for previous_index in range(current_index - 1, -1, -1):
+ previous_page_idx, _, previous_block = ordered_blocks[previous_index]
+ if previous_page_idx != current_page_idx and not cross_page_allowed:
+ return None
+
+ previous_type = previous_block.get("type")
+ if previous_type in MERGE_BARRIER_TYPES:
+ return None
+ if previous_type != BlockType.TEXT:
+ continue
+ return ordered_blocks[previous_index]
+
+ return None
+
+
+def _find_first_page_edge_text_block(page_info):
+ for block in page_info.get("para_blocks", []):
+ block_type = block.get("type")
+ if block_type in MERGE_BARRIER_TYPES:
+ return None
+ if block_type == BlockType.TEXT and _block_has_lines(block):
+ return block
+ return None
+
+
+def _find_last_page_edge_text_block(page_info):
+ for block in reversed(page_info.get("para_blocks", [])):
+ block_type = block.get("type")
+ if block_type in MERGE_BARRIER_TYPES:
+ return None
+ if block_type == BlockType.TEXT and _block_has_lines(block):
+ return block
+ return None
+
+
+def _resolve_metric_lines(page_info, block, edge_name, prefer_edge_line_hints):
+ if prefer_edge_line_hints:
+ edge_line_hints = page_info.get(_EDGE_TEXT_LINE_HINTS_KEY, {})
+ edge_hint = edge_line_hints.get(edge_name)
+ if edge_hint and edge_hint.get("index") == block.get("index"):
+ return edge_hint.get("lines", [])
+ return []
+ return block.get("lines", [])
+
+
+def _merge_text_block(current_block, previous_block, is_cross_page):
+ if is_cross_page:
+ for line in current_block.get("lines", []):
+ for span in line.get("spans", []):
+ span[SplitFlag.CROSS_PAGE] = True
+
+ previous_block.setdefault("lines", []).extend(current_block.get("lines", []))
+ current_block["lines"] = []
+ current_block[SplitFlag.LINES_DELETED] = True
+
+
+def _line_height(line):
+ bbox = line.get("bbox")
+ if not bbox:
+ return 0
+ return bbox[3] - bbox[1]
+
+
+def _build_bbox_fs(block, lines):
+ if lines:
+ return [
+ min(line["bbox"][0] for line in lines),
+ min(line["bbox"][1] for line in lines),
+ max(line["bbox"][2] for line in lines),
+ max(line["bbox"][3] for line in lines),
+ ]
+ return list(block.get("bbox", []))
+
+
+def _block_has_lines(block):
+ return any(line.get("spans") for line in block.get("lines", []))
+
+
+def _first_span(line):
+ spans = line.get("spans", [])
+ return spans[0] if spans else None
+
+
+def _last_span(line):
+ spans = line.get("spans", [])
+ return spans[-1] if spans else None
diff --git a/mineru/backend/utils/runtime_utils.py b/mineru/backend/utils/runtime_utils.py
new file mode 100644
index 000000000..43f677436
--- /dev/null
+++ b/mineru/backend/utils/runtime_utils.py
@@ -0,0 +1,46 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import time
+
+from loguru import logger
+
+from mineru.utils.table_merge import merge_table
+
+
+def cross_page_table_merge(pdf_info: list[dict]):
+ """Merge tables that span across multiple pages in a PDF document.
+
+ Args:
+ pdf_info (list[dict]): A list of dictionaries containing information about each page in the PDF.
+
+ Returns:
+ None
+ """
+ is_merge_table = os.getenv('MINERU_TABLE_MERGE_ENABLE', 'true')
+ if is_merge_table.lower() in ['true', '1', 'yes']:
+ merge_table(pdf_info)
+ elif is_merge_table.lower() in ['false', '0', 'no']:
+ pass
+ else:
+ logger.warning(f'unknown MINERU_TABLE_MERGE_ENABLE config: {is_merge_table}, pass')
+ pass
+
+
+def exclude_progress_bar_idle_time(progress_bar, idle_since: float | None, now: float | None = None):
+ """Exclude non-processing idle time from a reused tqdm progress bar."""
+ if progress_bar is None or idle_since is None:
+ return
+
+ if now is None:
+ now = time.time()
+
+ idle_duration = now - idle_since
+ if idle_duration <= 0:
+ return
+
+ if hasattr(progress_bar, "start_t"):
+ progress_bar.start_t += idle_duration
+ if hasattr(progress_bar, "last_print_t"):
+ progress_bar.last_print_t = now
+ if hasattr(progress_bar, "last_print_n") and hasattr(progress_bar, "n"):
+ progress_bar.last_print_n = progress_bar.n
diff --git a/mineru/backend/vlm/__init__.py b/mineru/backend/vlm/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/backend/vlm/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/backend/vlm/model_output_to_middle_json.py b/mineru/backend/vlm/model_output_to_middle_json.py
new file mode 100644
index 000000000..78ba54fbc
--- /dev/null
+++ b/mineru/backend/vlm/model_output_to_middle_json.py
@@ -0,0 +1,162 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import time
+
+import numpy as np
+from loguru import logger
+from tqdm import tqdm
+
+from mineru.backend.utils.html_image_utils import replace_inline_table_images
+from mineru.backend.utils.para_block_utils import (
+ build_para_blocks_from_preproc,
+ cleanup_internal_para_block_metadata,
+ merge_para_text_blocks,
+)
+from mineru.backend.utils.runtime_utils import cross_page_table_merge
+from mineru.backend.vlm.vlm_magic_model import MagicModel
+from mineru.utils.config_reader import get_table_enable, get_llm_aided_config
+from mineru.utils.cut_image import cut_image_and_table
+from mineru.utils.enum_class import ContentType
+from mineru.utils.hash_utils import bytes_md5
+from mineru.utils.pdfium_guard import close_pdfium_document, pdfium_guard
+from mineru.version import __version__
+
+
+heading_level_import_success = False
+llm_aided_config = get_llm_aided_config()
+if llm_aided_config:
+ title_aided_config = llm_aided_config.get('title_aided', {})
+ if title_aided_config.get('enable', False):
+ from mineru.utils.llm_aided import llm_aided_title
+ from mineru.backend.utils.ocr_det_utils import (
+ detect_ocr_boxes_from_padded_crop,
+ get_ch_lite_ocr_det_model,
+ )
+ heading_level_import_success = True
+
+
+def blocks_to_page_info(page_blocks, image_dict, page, image_writer, page_index) -> dict:
+ """将blocks转换为页面信息"""
+
+ scale = image_dict["scale"]
+ page_pil_img = image_dict["img_pil"]
+ page_img_md5 = bytes_md5(page_pil_img.tobytes())
+ with pdfium_guard():
+ width, height = map(int, page.get_size())
+
+ magic_model = MagicModel(page_blocks, width, height)
+ image_blocks = magic_model.get_image_blocks()
+ table_blocks = magic_model.get_table_blocks()
+ chart_blocks = magic_model.get_chart_blocks()
+ title_blocks = magic_model.get_title_blocks()
+ discarded_blocks = magic_model.get_discarded_blocks()
+ code_blocks = magic_model.get_code_blocks()
+ ref_text_blocks = magic_model.get_ref_text_blocks()
+ phonetic_blocks = magic_model.get_phonetic_blocks()
+ list_blocks = magic_model.get_list_blocks()
+
+ # 如果有标题优化需求，则对title_blocks截图det
+ if heading_level_import_success:
+ ocr_model = get_ch_lite_ocr_det_model()
+ for title_block in title_blocks:
+ ocr_det_res, _ = detect_ocr_boxes_from_padded_crop(
+ title_block.get('bbox'),
+ page_pil_img,
+ scale,
+ ocr_model=ocr_model,
+ )
+ if len(ocr_det_res) > 0:
+ # 计算所有res的平均高度
+ avg_height = np.mean([box[2][1] - box[0][1] for box in ocr_det_res])
+ title_block['line_avg_height'] = round(avg_height/scale)
+
+ text_blocks = magic_model.get_text_blocks()
+ interline_equation_blocks = magic_model.get_interline_equation_blocks()
+
+ all_spans = magic_model.get_all_spans()
+ # 对image/table/chart/interline_equation的span截图
+ for span in all_spans:
+ if span["type"] in [ContentType.IMAGE, ContentType.TABLE, ContentType.CHART, ContentType.INTERLINE_EQUATION]:
+ span = cut_image_and_table(span, page_pil_img, page_img_md5, page_index, image_writer, scale=scale)
+
+ replace_inline_table_images(table_blocks, image_writer, page_index)
+
+ page_blocks = []
+ page_blocks.extend([
+ *image_blocks,
+ *table_blocks,
+ *chart_blocks,
+ *code_blocks,
+ *ref_text_blocks,
+ *phonetic_blocks,
+ *title_blocks,
+ *text_blocks,
+ *interline_equation_blocks,
+ *list_blocks,
+ ])
+ # 对page_blocks根据index的值进行排序
+ page_blocks.sort(key=lambda x: x["index"])
+
+ page_info = {
+ "preproc_blocks": page_blocks,
+ "discarded_blocks": discarded_blocks,
+ "page_size": [width, height],
+ "page_idx": page_index,
+ }
+ return page_info
+
+
+def init_middle_json():
+ return {"pdf_info": [], "_backend": "vlm", "_version_name": __version__}
+
+
+def append_page_blocks_to_middle_json(
+ middle_json,
+ model_output_blocks_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=0,
+ progress_bar=None,
+):
+ for offset, (page_blocks, image_dict) in enumerate(zip(model_output_blocks_list, images_list)):
+ page_index = page_start_index + offset
+ with pdfium_guard():
+ page = pdf_doc[page_index]
+ page_info = blocks_to_page_info(page_blocks, image_dict, page, image_writer, page_index)
+ middle_json["pdf_info"].append(page_info)
+ if progress_bar is not None:
+ progress_bar.update(1)
+
+
+def finalize_middle_json(pdf_info_list):
+ build_para_blocks_from_preproc(pdf_info_list)
+ merge_para_text_blocks(pdf_info_list, allow_cross_page=False)
+
+ table_enable = get_table_enable(os.getenv('MINERU_VLM_TABLE_ENABLE', 'True').lower() == 'true')
+ if table_enable:
+ cross_page_table_merge(pdf_info_list)
+
+ if heading_level_import_success:
+ llm_aided_title_start_time = time.time()
+ llm_aided_title(pdf_info_list, title_aided_config)
+ logger.info(f'llm aided title time: {round(time.time() - llm_aided_title_start_time, 2)}')
+
+ cleanup_internal_para_block_metadata(pdf_info_list)
+
+
+def result_to_middle_json(model_output_blocks_list, images_list, pdf_doc, image_writer):
+ middle_json = init_middle_json()
+ with tqdm(total=len(model_output_blocks_list), desc="Processing pages") as progress_bar:
+ append_page_blocks_to_middle_json(
+ middle_json,
+ model_output_blocks_list,
+ images_list,
+ pdf_doc,
+ image_writer,
+ progress_bar=progress_bar,
+ )
+
+ finalize_middle_json(middle_json["pdf_info"])
+ close_pdfium_document(pdf_doc)
+ return middle_json
diff --git a/mineru/backend/vlm/utils.py b/mineru/backend/vlm/utils.py
new file mode 100644
index 000000000..bf05f685a
--- /dev/null
+++ b/mineru/backend/vlm/utils.py
@@ -0,0 +1,235 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+
+from loguru import logger
+from packaging import version
+
+from mineru.utils.check_sys_env import is_windows_environment, is_linux_environment
+from mineru.utils.config_reader import get_device
+from mineru.utils.model_utils import get_vram
+
+
+def enable_custom_logits_processors() -> bool:
+ import torch
+ from vllm import __version__ as vllm_version
+
+ if torch.cuda.is_available():
+ major, minor = torch.cuda.get_device_capability()
+ # 正确计算Compute Capability
+ compute_capability = f"{major}.{minor}"
+ elif hasattr(torch, 'npu') and torch.npu.is_available():
+ compute_capability = "8.0"
+ elif hasattr(torch, 'gcu') and torch.gcu.is_available():
+ compute_capability = "8.0"
+ elif hasattr(torch, 'musa') and torch.musa.is_available():
+ compute_capability = "8.0"
+ elif hasattr(torch, 'mlu') and torch.mlu.is_available():
+ compute_capability = "8.0"
+ elif hasattr(torch, 'sdaa') and torch.sdaa.is_available():
+ compute_capability = "8.0"
+
+ else:
+ logger.info("CUDA not available, disabling custom_logits_processors")
+ return False
+
+ # 安全地处理环境变量
+ vllm_use_v1_str = os.getenv('VLLM_USE_V1', "1")
+ if vllm_use_v1_str.isdigit():
+ vllm_use_v1 = int(vllm_use_v1_str)
+ else:
+ vllm_use_v1 = 1
+
+ if vllm_use_v1 == 0:
+ logger.info("VLLM_USE_V1 is set to 0, disabling custom_logits_processors")
+ return False
+ elif version.parse(vllm_version) < version.parse("0.10.1"):
+ logger.info(f"vllm version: {vllm_version} < 0.10.1, disable custom_logits_processors")
+ return False
+ elif version.parse(compute_capability) < version.parse("8.0"):
+ if version.parse(vllm_version) >= version.parse("0.10.2"):
+ logger.info(f"compute_capability: {compute_capability} < 8.0, but vllm version: {vllm_version} >= 0.10.2, enable custom_logits_processors")
+ return True
+ else:
+ logger.info(f"compute_capability: {compute_capability} < 8.0 and vllm version: {vllm_version} < 0.10.2, disable custom_logits_processors")
+ return False
+ else:
+ logger.info(f"compute_capability: {compute_capability} >= 8.0 and vllm version: {vllm_version} >= 0.10.1, enable custom_logits_processors")
+ return True
+
+
+def set_lmdeploy_backend(device_type: str) -> str:
+ if device_type.lower() in ["ascend", "maca", "camb"]:
+ lmdeploy_backend = "pytorch"
+ elif device_type.lower() in ["cuda"]:
+ import torch
+ if not torch.cuda.is_available():
+ raise ValueError("CUDA is not available.")
+ if is_windows_environment():
+ lmdeploy_backend = "turbomind"
+ elif is_linux_environment():
+ major, minor = torch.cuda.get_device_capability()
+ compute_capability = f"{major}.{minor}"
+ if version.parse(compute_capability) >= version.parse("8.0"):
+ lmdeploy_backend = "pytorch"
+ else:
+ lmdeploy_backend = "turbomind"
+ else:
+ raise ValueError("Unsupported operating system.")
+ else:
+ raise ValueError(f"Unsupported lmdeploy device type: {device_type}")
+ return lmdeploy_backend
+
+
+def set_default_gpu_memory_utilization() -> float:
+ from vllm import __version__ as vllm_version
+ device = get_device()
+ gpu_memory = get_vram(device)
+ default_gpu_memory_utilization = 0.5
+ if version.parse(vllm_version) >= version.parse("0.11.0") and gpu_memory <= 8:
+ default_gpu_memory_utilization = 0.7
+
+ logger.debug(f"vllm_version: {vllm_version}, gpu_memory: {gpu_memory} GB, default_gpu_memory_utilization: {default_gpu_memory_utilization}")
+ return default_gpu_memory_utilization
+
+
+def set_default_batch_size() -> int:
+ try:
+ device = get_device()
+ gpu_memory = get_vram(device)
+
+ if gpu_memory >= 16:
+ batch_size = 8
+ elif gpu_memory >= 8:
+ batch_size = 4
+ else:
+ batch_size = 1
+ logger.info(f'gpu_memory: {gpu_memory} GB, batch_size: {batch_size}')
+
+ except Exception as e:
+ logger.warning(f'Error determining VRAM: {e}, using default batch_ratio: 1')
+ batch_size = 1
+ return batch_size
+
+
+def _get_device_config(device_type: str) -> dict | None:
+ """获取不同设备类型的配置参数"""
+
+ # 各设备类型的配置定义
+ DEVICE_CONFIGS = {
+ # "musa": {
+ # "compilation_config_dict": {
+ # "cudagraph_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 18, 20, 24, 28, 30],
+ # "simple_cuda_graph": True
+ # },
+ # "block_size": 32,
+ # },
+ "corex": {
+ "compilation_config_dict": {
+ "cudagraph_mode": "FULL_DECODE_ONLY",
+ "level": 0
+ },
+ },
+ "kxpu": {
+ "compilation_config_dict": {
+ "splitting_ops": [
+ "vllm.unified_attention", "vllm.unified_attention_with_output",
+ "vllm.unified_attention_with_output_kunlun", "vllm.mamba_mixer2",
+ "vllm.mamba_mixer", "vllm.short_conv", "vllm.linear_attention",
+ "vllm.plamo2_mamba_mixer", "vllm.gdn_attention", "vllm.sparse_attn_indexer"
+ ]
+ },
+ "block_size": 128,
+ "dtype": "float16",
+ "distributed_executor_backend": "mp",
+ "enable_chunked_prefill": False,
+ "enable_prefix_caching": False,
+ },
+ }
+
+ return DEVICE_CONFIGS.get(device_type.lower())
+
+
+def _check_server_arg_exists(args: list, arg_name: str) -> bool:
+ """检查命令行参数列表中是否已存在指定参数"""
+ return any(arg == f"--{arg_name}" or arg.startswith(f"--{arg_name}=") for arg in args)
+
+
+def _add_server_arg_if_missing(args: list, arg_name: str, value: str) -> None:
+ """如果参数不存在，则添加到命令行参数列表"""
+ if not _check_server_arg_exists(args, arg_name):
+ args.extend([f"--{arg_name}", value])
+
+
+def _add_server_flag_if_missing(args: list, flag_name: str) -> None:
+ """如果 flag 不存在，则添加到命令行参数列表"""
+ if not _check_server_arg_exists(args, flag_name):
+ args.append(f"--{flag_name}")
+
+
+def _add_engine_kwarg_if_missing(kwargs: dict, key: str, value) -> None:
+ """如果参数不存在，则添加到 kwargs 字典"""
+ if key not in kwargs:
+ kwargs[key] = value
+
+
+def mod_kwargs_by_device_type(kwargs_or_args: dict | list, vllm_mode: str) -> dict | list:
+ """根据设备类型修改 vllm 配置参数
+
+ Args:
+ kwargs_or_args: 配置参数，server 模式为 list，engine 模式为 dict
+ vllm_mode: vllm 运行模式 ("server", "sync_engine", "async_engine")
+
+ Returns:
+ 修改后的配置参数
+ """
+ device_type = os.getenv("MINERU_VLLM_DEVICE", "")
+ config = _get_device_config(device_type)
+
+ if config is None:
+ return kwargs_or_args
+
+ if vllm_mode == "server":
+ _apply_server_config(kwargs_or_args, config)
+ else:
+ _apply_engine_config(kwargs_or_args, config, vllm_mode)
+
+ return kwargs_or_args
+
+
+def _apply_server_config(args: list, config: dict) -> None:
+ """应用 server 模式的配置"""
+ import json
+
+ for key, value in config.items():
+ if key == "compilation_config_dict":
+ _add_server_arg_if_missing(
+ args, "compilation-config",
+ json.dumps(value, separators=(',', ':'))
+ )
+ else:
+ # 转换 key 格式: block_size -> block-size
+ arg_name = key.replace("_", "-")
+ if arg_name in {"enable-chunked-prefill", "enable-prefix-caching"} and value is False:
+ _add_server_flag_if_missing(args, f"no-{arg_name}")
+ continue
+ _add_server_arg_if_missing(args, arg_name, str(value))
+
+
+def _apply_engine_config(kwargs: dict, config: dict, vllm_mode: str) -> None:
+ """应用 engine 模式的配置"""
+ try:
+ from vllm.config import CompilationConfig
+ except ImportError:
+ raise ImportError("Please install vllm to use the vllm-async-engine backend.")
+
+ for key, value in config.items():
+ if key == "compilation_config_dict":
+ if vllm_mode == "sync_engine":
+ compilation_config = value
+ elif vllm_mode == "async_engine":
+ compilation_config = CompilationConfig(**value)
+ else:
+ continue
+ _add_engine_kwarg_if_missing(kwargs, "compilation_config", compilation_config)
+ else:
+ _add_engine_kwarg_if_missing(kwargs, key, value)
diff --git a/mineru/backend/vlm/vlm_analyze.py b/mineru/backend/vlm/vlm_analyze.py
new file mode 100644
index 000000000..cb2e72c41
--- /dev/null
+++ b/mineru/backend/vlm/vlm_analyze.py
@@ -0,0 +1,586 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
+import atexit
+import gc
+import os
+import time
+import json
+import threading
+from contextlib import asynccontextmanager, contextmanager
+
+import pypdfium2 as pdfium
+from loguru import logger
+from tqdm import tqdm
+
+from .utils import enable_custom_logits_processors, set_default_gpu_memory_utilization, set_default_batch_size, \
+ set_lmdeploy_backend, mod_kwargs_by_device_type
+from .model_output_to_middle_json import (
+ append_page_blocks_to_middle_json,
+ finalize_middle_json,
+ init_middle_json,
+)
+from mineru.backend.utils.runtime_utils import exclude_progress_bar_idle_time
+from ...data.data_reader_writer import DataWriter
+from mineru.utils.pdf_image_tools import load_images_from_pdf_doc
+from ...utils.check_sys_env import is_mac_os_version_supported
+from ...utils.config_reader import get_device, get_processing_window_size
+
+from ...utils.enum_class import ImageType
+from ...utils.pdfium_guard import (
+ close_pdfium_document,
+ get_pdfium_document_page_count,
+ open_pdfium_document,
+)
+from ...utils.models_download_utils import auto_download_and_get_model_root_path
+
+from mineru_vl_utils import MinerUClient
+from packaging import version
+
+
+class ModelSingleton:
+ _instance = None
+ _models = {}
+ _lock = threading.RLock()
+
+ def __new__(cls, *args, **kwargs):
+ with cls._lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ return cls._instance
+
+ def get_model(
+ self,
+ backend: str,
+ model_path: str | None,
+ server_url: str | None,
+ **kwargs,
+ ) -> MinerUClient:
+ key = (backend, model_path, server_url)
+ with self._lock:
+ if key not in self._models:
+ start_time = time.time()
+ model = None
+ processor = None
+ vllm_llm = None
+ lmdeploy_engine = None
+ vllm_async_llm = None
+ batch_size = kwargs.get("batch_size", 0) # for transformers backend only
+ max_concurrency = kwargs.get("max_concurrency", 100) # for http-client backend only
+ http_timeout = kwargs.get("http_timeout", 600) # for http-client backend only
+ server_headers = kwargs.get("server_headers", None) # for http-client backend only
+ max_retries = kwargs.get("max_retries", 3) # for http-client backend only
+ retry_backoff_factor = kwargs.get("retry_backoff_factor", 0.5) # for http-client backend only
+ # 从kwargs中移除这些参数，避免传递给不相关的初始化函数
+ for param in ["batch_size", "max_concurrency", "http_timeout", "server_headers", "max_retries", "retry_backoff_factor"]:
+ if param in kwargs:
+ del kwargs[param]
+ if backend not in ["http-client"] and not model_path:
+ model_path = auto_download_and_get_model_root_path("/","vlm")
+ if backend == "transformers":
+ try:
+ from transformers import (
+ AutoProcessor,
+ Qwen2VLForConditionalGeneration,
+ )
+ from transformers import __version__ as transformers_version
+ except ImportError:
+ raise ImportError("Please install transformers to use the transformers backend.")
+
+ if version.parse(transformers_version) >= version.parse("4.56.0"):
+ dtype_key = "dtype"
+ else:
+ dtype_key = "torch_dtype"
+ device = get_device()
+ model = Qwen2VLForConditionalGeneration.from_pretrained(
+ model_path,
+ device_map={"": device},
+ **{dtype_key: "auto"}, # type: ignore
+ )
+ processor = AutoProcessor.from_pretrained(
+ model_path,
+ use_fast=True,
+ )
+ if batch_size == 0:
+ batch_size = set_default_batch_size()
+ elif backend == "mlx-engine":
+ mlx_supported = is_mac_os_version_supported()
+ if not mlx_supported:
+ raise EnvironmentError("mlx-engine backend is only supported on macOS 13.5+ with Apple Silicon.")
+ from mineru_vl_utils.mlx_compat import load_mlx_model
+ model, processor = load_mlx_model(model_path)
+ else:
+ if os.getenv('OMP_NUM_THREADS') is None:
+ os.environ["OMP_NUM_THREADS"] = "1"
+
+ if backend == "vllm-engine":
+ try:
+ import vllm
+ except ImportError:
+ raise ImportError("Please install vllm to use the vllm-engine backend.")
+
+ kwargs = mod_kwargs_by_device_type(kwargs, vllm_mode="sync_engine")
+
+ if "compilation_config" in kwargs:
+ if isinstance(kwargs["compilation_config"], str):
+ try:
+ kwargs["compilation_config"] = json.loads(kwargs["compilation_config"])
+ except json.JSONDecodeError:
+ logger.warning(
+ f"Failed to parse compilation_config as JSON: {kwargs['compilation_config']}")
+ del kwargs["compilation_config"]
+ if "gpu_memory_utilization" not in kwargs:
+ kwargs["gpu_memory_utilization"] = set_default_gpu_memory_utilization()
+ if "model" not in kwargs:
+ kwargs["model"] = model_path
+ if enable_custom_logits_processors() and ("logits_processors" not in kwargs):
+ from mineru_vl_utils import MinerULogitsProcessor
+ kwargs["logits_processors"] = [MinerULogitsProcessor]
+ # 使用kwargs为 vllm初始化参数
+ vllm_llm = vllm.LLM(**kwargs)
+ elif backend == "vllm-async-engine":
+ try:
+ from vllm.engine.arg_utils import AsyncEngineArgs
+ from vllm.v1.engine.async_llm import AsyncLLM
+ from vllm.config import CompilationConfig
+ except ImportError:
+ raise ImportError("Please install vllm to use the vllm-async-engine backend.")
+
+ kwargs = mod_kwargs_by_device_type(kwargs, vllm_mode="async_engine")
+
+ if "compilation_config" in kwargs:
+ if isinstance(kwargs["compilation_config"], dict):
+ # 如果是字典，转换为 CompilationConfig 对象
+ kwargs["compilation_config"] = CompilationConfig(**kwargs["compilation_config"])
+ elif isinstance(kwargs["compilation_config"], str):
+ # 如果是 JSON 字符串，先解析再转换
+ try:
+ config_dict = json.loads(kwargs["compilation_config"])
+ kwargs["compilation_config"] = CompilationConfig(**config_dict)
+ except (json.JSONDecodeError, TypeError) as e:
+ logger.warning(
+ f"Failed to parse compilation_config: {kwargs['compilation_config']}, error: {e}")
+ del kwargs["compilation_config"]
+ if "gpu_memory_utilization" not in kwargs:
+ kwargs["gpu_memory_utilization"] = set_default_gpu_memory_utilization()
+ if "model" not in kwargs:
+ kwargs["model"] = model_path
+ if enable_custom_logits_processors() and ("logits_processors" not in kwargs):
+ from mineru_vl_utils import MinerULogitsProcessor
+ kwargs["logits_processors"] = [MinerULogitsProcessor]
+ # 使用kwargs为 vllm初始化参数
+ vllm_async_llm = AsyncLLM.from_engine_args(AsyncEngineArgs(**kwargs))
+ elif backend == "lmdeploy-engine":
+ try:
+ from lmdeploy import PytorchEngineConfig, TurbomindEngineConfig
+ from lmdeploy.serve.vl_async_engine import VLAsyncEngine
+ except ImportError:
+ raise ImportError("Please install lmdeploy to use the lmdeploy-engine backend.")
+ if "cache_max_entry_count" not in kwargs:
+ kwargs["cache_max_entry_count"] = 0.5
+
+ device_type = os.getenv("MINERU_LMDEPLOY_DEVICE", "")
+ if device_type == "":
+ if "lmdeploy_device" in kwargs:
+ device_type = kwargs.pop("lmdeploy_device")
+ if device_type not in ["cuda", "ascend", "maca", "camb"]:
+ raise ValueError(f"Unsupported lmdeploy device type: {device_type}")
+ else:
+ device_type = "cuda"
+ lm_backend = os.getenv("MINERU_LMDEPLOY_BACKEND", "")
+ if lm_backend == "":
+ if "lmdeploy_backend" in kwargs:
+ lm_backend = kwargs.pop("lmdeploy_backend")
+ if lm_backend not in ["pytorch", "turbomind"]:
+ raise ValueError(f"Unsupported lmdeploy backend: {lm_backend}")
+ else:
+ lm_backend = set_lmdeploy_backend(device_type)
+ logger.info(f"lmdeploy device is: {device_type}, lmdeploy backend is: {lm_backend}")
+
+ if lm_backend == "pytorch":
+ kwargs["device_type"] = device_type
+ backend_config = PytorchEngineConfig(**kwargs)
+ elif lm_backend == "turbomind":
+ backend_config = TurbomindEngineConfig(**kwargs)
+ else:
+ raise ValueError(f"Unsupported lmdeploy backend: {lm_backend}")
+
+ log_level = 'ERROR'
+ from lmdeploy.utils import get_logger
+ lm_logger = get_logger('lmdeploy')
+ lm_logger.setLevel(log_level)
+ if os.getenv('TM_LOG_LEVEL') is None:
+ os.environ['TM_LOG_LEVEL'] = log_level
+
+ lmdeploy_engine = VLAsyncEngine(
+ model_path,
+ backend=lm_backend,
+ backend_config=backend_config,
+ )
+ predictor = MinerUClient(
+ backend=backend,
+ model=model,
+ processor=processor,
+ lmdeploy_engine=lmdeploy_engine,
+ vllm_llm=vllm_llm,
+ vllm_async_llm=vllm_async_llm,
+ server_url=server_url,
+ batch_size=batch_size,
+ max_concurrency=max_concurrency,
+ http_timeout=http_timeout,
+ server_headers=server_headers,
+ max_retries=max_retries,
+ retry_backoff_factor=retry_backoff_factor,
+ enable_table_formula_eq_wrap=True,
+ image_analysis=True,
+ enable_cross_page_table_merge=True,
+ )
+ predictor._mineru_runtime_handles = {
+ "backend": backend,
+ "model": model,
+ "processor": processor,
+ "vllm_llm": vllm_llm,
+ "vllm_async_llm": vllm_async_llm,
+ "lmdeploy_engine": lmdeploy_engine,
+ }
+ _maybe_enable_serial_execution(predictor, backend)
+ self._models[key] = predictor
+ elapsed = round(time.time() - start_time, 2)
+ logger.info(f"get {backend} predictor cost: {elapsed}s")
+ return self._models[key]
+
+ def shutdown(self) -> None:
+ with self._lock:
+ predictors = list(self._models.values())
+ self._models.clear()
+
+ for predictor in predictors:
+ _shutdown_predictor_runtime(predictor)
+
+ gc.collect()
+
+
+def _iter_shutdown_candidates(predictor: MinerUClient):
+ runtime_handles = getattr(predictor, "_mineru_runtime_handles", {})
+ client = getattr(predictor, "client", None)
+
+ seen_ids = set()
+
+ def _yield_candidate(candidate):
+ if candidate is None:
+ return
+ candidate_id = id(candidate)
+ if candidate_id in seen_ids:
+ return
+ seen_ids.add(candidate_id)
+ yield candidate
+
+ for key in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "model"):
+ yield from _yield_candidate(runtime_handles.get(key))
+
+ if client is not None:
+ for key in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "model"):
+ yield from _yield_candidate(getattr(client, key, None))
+
+
+def _call_nested_shutdown(target, method_path: str, label: str) -> bool:
+ current = target
+ for attr in method_path.split("."):
+ current = getattr(current, attr, None)
+ if current is None:
+ return False
+
+ if not callable(current):
+ return False
+
+ try:
+ current()
+ logger.debug(f"Shutdown {label} via `{method_path}`")
+ return True
+ except TypeError:
+ logger.debug(f"Skip unsupported shutdown call {label}.{method_path}")
+ return False
+ except Exception as exc:
+ logger.debug(f"Failed to shutdown {label} via `{method_path}`: {exc}")
+ return False
+
+
+def _shutdown_runtime_handle(handle) -> None:
+ for method_path in (
+ "shutdown",
+ "close",
+ "stop",
+ "terminate",
+ "destroy",
+ "engine.shutdown",
+ "engine.close",
+ "engine_core.shutdown",
+ "engine_core.close",
+ "llm_engine.shutdown",
+ "llm_engine.close",
+ "llm_engine.model_executor.shutdown",
+ "llm_engine.model_executor.close",
+ "model_executor.shutdown",
+ "model_executor.close",
+ ):
+ if _call_nested_shutdown(handle, method_path, type(handle).__name__):
+ return
+
+
+def _clear_predictor_references(predictor: MinerUClient) -> None:
+ runtime_handles = getattr(predictor, "_mineru_runtime_handles", {})
+ for key in tuple(runtime_handles.keys()):
+ runtime_handles[key] = None
+
+ client = getattr(predictor, "client", None)
+ if client is not None:
+ for attr in ("vllm_llm", "vllm_async_llm", "lmdeploy_engine", "model", "processor"):
+ if hasattr(client, attr):
+ setattr(client, attr, None)
+
+
+def _shutdown_predictor_runtime(predictor: MinerUClient) -> None:
+ for handle in _iter_shutdown_candidates(predictor):
+ _shutdown_runtime_handle(handle)
+ _clear_predictor_references(predictor)
+
+
+def shutdown_cached_models() -> None:
+ ModelSingleton().shutdown()
+
+
+atexit.register(shutdown_cached_models)
+
+
+def _predictor_uses_mlx(predictor: MinerUClient, backend: str | None = None) -> bool:
+ if backend == "mlx-engine":
+ return True
+ client = getattr(predictor, "client", None)
+ return type(client).__module__.endswith(".mlx_client")
+
+
+def _maybe_enable_serial_execution(
+ predictor: MinerUClient,
+ backend: str | None = None,
+) -> MinerUClient:
+ if _predictor_uses_mlx(predictor, backend) and not hasattr(
+ predictor, "_mineru_execution_lock"
+ ):
+ predictor._mineru_execution_lock = threading.Lock()
+ return predictor
+
+
+@contextmanager
+def predictor_execution_guard(predictor: MinerUClient):
+ lock = getattr(predictor, "_mineru_execution_lock", None)
+ if lock is None:
+ yield
+ return
+ with lock:
+ yield
+
+
+@asynccontextmanager
+async def aio_predictor_execution_guard(predictor: MinerUClient):
+ lock = getattr(predictor, "_mineru_execution_lock", None)
+ if lock is None:
+ yield
+ return
+ await asyncio.to_thread(lock.acquire)
+ try:
+ yield
+ finally:
+ lock.release()
+
+
+def _close_images(images_list):
+ for image_dict in images_list or []:
+ pil_img = image_dict.get("img_pil")
+ if pil_img is not None:
+ try:
+ pil_img.close()
+ except Exception:
+ pass
+
+
+def doc_analyze(
+ pdf_bytes,
+ image_writer: DataWriter | None,
+ predictor: MinerUClient | None = None,
+ backend="transformers",
+ model_path: str | None = None,
+ server_url: str | None = None,
+ **kwargs,
+):
+ if predictor is None:
+ predictor = ModelSingleton().get_model(backend, model_path, server_url, **kwargs)
+ predictor = _maybe_enable_serial_execution(predictor, backend)
+
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ middle_json = init_middle_json()
+ results = []
+ doc_closed = False
+ try:
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ configured_window_size = get_processing_window_size(default=64)
+ effective_window_size = min(page_count, configured_window_size) if page_count else 0
+ total_windows = (
+ (page_count + effective_window_size - 1) // effective_window_size
+ if effective_window_size
+ else 0
+ )
+ logger.info(
+ f'VLM processing-window run. page_count={page_count}, '
+ f'window_size={configured_window_size}, total_windows={total_windows}'
+ )
+
+ infer_start = time.time()
+ progress_bar = None
+ last_append_end_time = None
+ try:
+ for window_index, window_start in enumerate(range(0, page_count, effective_window_size or 1)):
+ window_end = min(page_count - 1, window_start + effective_window_size - 1)
+ images_list = load_images_from_pdf_doc(
+ pdf_doc,
+ start_page_id=window_start,
+ end_page_id=window_end,
+ image_type=ImageType.PIL,
+ pdf_bytes=pdf_bytes,
+ )
+ try:
+ images_pil_list = [image_dict["img_pil"] for image_dict in images_list]
+ logger.info(
+ f'VLM processing window {window_index + 1}/{total_windows}: '
+ f'pages {window_start + 1}-{window_end + 1}/{page_count} '
+ f'({len(images_pil_list)} pages)'
+ )
+ with predictor_execution_guard(predictor):
+ window_results = predictor.batch_two_step_extract(images=images_pil_list)
+ results.extend(window_results)
+ if progress_bar is None:
+ progress_bar = tqdm(total=page_count, desc="Processing pages")
+ else:
+ exclude_progress_bar_idle_time(
+ progress_bar,
+ last_append_end_time,
+ now=time.time(),
+ )
+ append_page_blocks_to_middle_json(
+ middle_json,
+ window_results,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=window_start,
+ progress_bar=progress_bar,
+ )
+ last_append_end_time = time.time()
+ finally:
+ _close_images(images_list)
+ finally:
+ if progress_bar is not None:
+ progress_bar.close()
+ infer_time = round(time.time() - infer_start, 2)
+ if infer_time > 0 and page_count > 0:
+ logger.debug(
+ f"processing-window infer finished, cost: {infer_time}, "
+ f"speed: {round(len(results) / infer_time, 3)} page/s"
+ )
+ finalize_middle_json(middle_json["pdf_info"])
+ close_pdfium_document(pdf_doc)
+ doc_closed = True
+ return middle_json, results
+ finally:
+ if not doc_closed:
+ close_pdfium_document(pdf_doc)
+
+
+async def aio_doc_analyze(
+ pdf_bytes,
+ image_writer: DataWriter | None,
+ predictor: MinerUClient | None = None,
+ backend="transformers",
+ model_path: str | None = None,
+ server_url: str | None = None,
+ **kwargs,
+):
+ if predictor is None:
+ predictor = ModelSingleton().get_model(backend, model_path, server_url, **kwargs)
+ predictor = _maybe_enable_serial_execution(predictor, backend)
+
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ middle_json = init_middle_json()
+ results = []
+ doc_closed = False
+ try:
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ configured_window_size = get_processing_window_size(default=64)
+ effective_window_size = min(page_count, configured_window_size) if page_count else 0
+ total_windows = (
+ (page_count + effective_window_size - 1) // effective_window_size
+ if effective_window_size
+ else 0
+ )
+ logger.info(
+ f'VLM processing-window run. page_count={page_count}, '
+ f'window_size={configured_window_size}, total_windows={total_windows}'
+ )
+
+ infer_start = time.time()
+ progress_bar = None
+ last_append_end_time = None
+ try:
+ for window_index, window_start in enumerate(range(0, page_count, effective_window_size or 1)):
+ window_end = min(page_count - 1, window_start + effective_window_size - 1)
+ images_list = load_images_from_pdf_doc(
+ pdf_doc,
+ start_page_id=window_start,
+ end_page_id=window_end,
+ image_type=ImageType.PIL,
+ pdf_bytes=pdf_bytes,
+ )
+ try:
+ images_pil_list = [image_dict["img_pil"] for image_dict in images_list]
+ logger.info(
+ f'VLM processing window {window_index + 1}/{total_windows}: '
+ f'pages {window_start + 1}-{window_end + 1}/{page_count} '
+ f'({len(images_pil_list)} pages)'
+ )
+ async with aio_predictor_execution_guard(predictor):
+ window_results = await predictor.aio_batch_two_step_extract(images=images_pil_list)
+ results.extend(window_results)
+ if progress_bar is None:
+ progress_bar = tqdm(total=page_count, desc="Processing pages")
+ else:
+ exclude_progress_bar_idle_time(
+ progress_bar,
+ last_append_end_time,
+ now=time.time(),
+ )
+ append_page_blocks_to_middle_json(
+ middle_json,
+ window_results,
+ images_list,
+ pdf_doc,
+ image_writer,
+ page_start_index=window_start,
+ progress_bar=progress_bar,
+ )
+ last_append_end_time = time.time()
+ finally:
+ _close_images(images_list)
+ finally:
+ if progress_bar is not None:
+ progress_bar.close()
+ infer_time = round(time.time() - infer_start, 2)
+ if infer_time > 0 and page_count > 0:
+ logger.debug(
+ f"processing-window infer finished, cost: {infer_time}, "
+ f"speed: {round(len(results) / infer_time, 3)} page/s"
+ )
+ finalize_middle_json(middle_json["pdf_info"])
+ close_pdfium_document(pdf_doc)
+ doc_closed = True
+ return middle_json, results
+ finally:
+ if not doc_closed:
+ close_pdfium_document(pdf_doc)
diff --git a/mineru/backend/vlm/vlm_magic_model.py b/mineru/backend/vlm/vlm_magic_model.py
new file mode 100644
index 000000000..32ecb66bd
--- /dev/null
+++ b/mineru/backend/vlm/vlm_magic_model.py
@@ -0,0 +1,375 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+
+from loguru import logger
+
+from mineru.utils.boxbase import calculate_overlap_area_in_bbox1_area_ratio
+from mineru.utils.enum_class import ContentType, BlockType
+from mineru.utils.guess_suffix_or_lang import guess_language_by_text
+from mineru.utils.visual_magic_model_utils import (
+ GENERIC_CHILD_TYPES,
+ IMAGE_BLOCK_BODY,
+ VISUAL_MAIN_TYPES,
+ clean_content,
+ code_content_clean,
+ isolated_formula_clean,
+ regroup_visual_blocks,
+)
+
+
+def _copy_raw_text_block_metadata(raw_block_type, block_info, block):
+ if raw_block_type != BlockType.TEXT:
+ return
+ if "merge_prev" in block_info:
+ block["merge_prev"] = block_info["merge_prev"]
+
+
+class MagicModel:
+ def __init__(self, page_blocks: list, width, height):
+ self.page_blocks = page_blocks
+
+ blocks = []
+ self.all_spans = []
+ # 解析每个块
+ for index, block_info in enumerate(page_blocks):
+ block_bbox = block_info["bbox"]
+ try:
+ x1, y1, x2, y2 = block_bbox
+ x_1, y_1, x_2, y_2 = (
+ int(x1 * width),
+ int(y1 * height),
+ int(x2 * width),
+ int(y2 * height),
+ )
+ if x_2 < x_1:
+ x_1, x_2 = x_2, x_1
+ if y_2 < y_1:
+ y_1, y_2 = y_2, y_1
+ block_bbox = (x_1, y_1, x_2, y_2)
+ block_type = block_info["type"]
+ raw_block_type = block_type
+ block_content = block_info.get("content")
+ block_angle = block_info.get("angle", 0)
+ block_sub_type = (
+ block_info.get("sub_type")
+ if raw_block_type in ["image", "chart"]
+ else None
+ )
+ except Exception as e:
+ # 如果解析失败，可能是因为格式不正确，跳过这个块
+ logger.warning(f"Invalid block format: {block_info}, error: {e}")
+ continue
+
+ span_type = "unknown"
+ code_block_sub_type = None
+ guess_lang = None
+
+ if block_type in [
+ "text",
+ "title",
+ "ref_text",
+ "phonetic",
+ "header",
+ "footer",
+ "page_number",
+ "aside_text",
+ "page_footnote",
+ "list",
+ ]:
+ span_type = ContentType.TEXT
+ elif block_type in ["image_caption", "table_caption", "code_caption"]:
+ block_type = BlockType.CAPTION
+ span_type = ContentType.TEXT
+ elif block_type in ["image_footnote", "table_footnote"]:
+ block_type = BlockType.FOOTNOTE
+ span_type = ContentType.TEXT
+ elif block_type == "image":
+ block_type = BlockType.IMAGE_BODY
+ span_type = ContentType.IMAGE
+ elif block_type == "image_block":
+ block_type = IMAGE_BLOCK_BODY
+ span_type = ContentType.IMAGE
+ elif block_type == "table":
+ block_type = BlockType.TABLE_BODY
+ span_type = ContentType.TABLE
+ elif block_type == "chart":
+ block_type = BlockType.CHART_BODY
+ span_type = ContentType.CHART
+ elif block_type in ["code", "algorithm"]:
+ block_content = code_content_clean(block_content)
+ code_block_sub_type = block_type
+ block_type = BlockType.CODE_BODY
+ span_type = ContentType.TEXT
+ guess_lang = guess_language_by_text(block_content)
+ elif block_type == "equation":
+ block_type = BlockType.INTERLINE_EQUATION
+ span_type = ContentType.INTERLINE_EQUATION
+
+ # code 和 algorithm 类型的块，如果内容中包含行内公式，则需要将块类型切换为 algorithm
+ switch_code_to_algorithm = False
+
+ if span_type in [ContentType.IMAGE, ContentType.TABLE, ContentType.CHART]:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ }
+ if span_type == ContentType.TABLE:
+ span["html"] = block_content
+ elif raw_block_type in ["image", "chart"] and block_content is not None:
+ span["content"] = block_content
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ "content": isolated_formula_clean(block_content),
+ }
+ else:
+ if block_content:
+ block_content = clean_content(block_content)
+
+ if block_type == "title" and block_content:
+ block_content = re.sub(r"\n\s*", " ", block_content).strip()
+
+ if (
+ block_content
+ and block_content.count("\\(") == block_content.count("\\)")
+ and block_content.count("\\(") > 0
+ ):
+ switch_code_to_algorithm = True
+
+ # 生成包含文本和公式的span列表
+ spans = []
+ last_end = 0
+
+ # 查找所有公式
+ for match in re.finditer(r"\\\((.+?)\\\)", block_content):
+ start, end = match.span()
+
+ # 添加公式前的文本
+ if start > last_end:
+ text_before = block_content[last_end:start]
+ if text_before.strip():
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.TEXT,
+ "content": text_before,
+ }
+ )
+
+ # 添加公式（去除\(和\)）
+ formula = match.group(1)
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.INLINE_EQUATION,
+ "content": formula.strip(),
+ }
+ )
+
+ last_end = end
+
+ # 添加最后一个公式后的文本
+ if last_end < len(block_content):
+ text_after = block_content[last_end:]
+ if text_after.strip():
+ spans.append(
+ {
+ "bbox": block_bbox,
+ "type": ContentType.TEXT,
+ "content": text_after,
+ }
+ )
+
+ span = spans
+ else:
+ span = {
+ "bbox": block_bbox,
+ "type": span_type,
+ "content": block_content,
+ }
+
+ # 处理span类型并添加到all_spans
+ if isinstance(span, dict) and "bbox" in span:
+ self.all_spans.append(span)
+ spans = [span]
+ elif isinstance(span, list):
+ self.all_spans.extend(span)
+ spans = span
+ else:
+ raise ValueError(
+ f"Invalid span type: {span_type}, expected dict or list, got {type(span)}"
+ )
+
+ # 构造 line 对象
+ if block_type == BlockType.CODE_BODY:
+ if switch_code_to_algorithm and code_block_sub_type == "code":
+ code_block_sub_type = "algorithm"
+ line = {
+ "bbox": block_bbox,
+ "spans": spans,
+ "extra": {"type": code_block_sub_type, "guess_lang": guess_lang},
+ }
+ else:
+ line = {"bbox": block_bbox, "spans": spans}
+
+ block = {
+ "bbox": block_bbox,
+ "type": block_type,
+ "angle": block_angle,
+ "lines": [line],
+ "index": index,
+ }
+ if block_sub_type:
+ block["sub_type"] = block_sub_type
+ if raw_block_type == "table" and "cell_merge" in block_info:
+ block["cell_merge"] = block_info["cell_merge"]
+ _copy_raw_text_block_metadata(raw_block_type, block_info, block)
+
+ blocks.append(block)
+
+ self.image_blocks = []
+ self.table_blocks = []
+ self.chart_blocks = []
+ self.interline_equation_blocks = []
+ self.text_blocks = []
+ self.title_blocks = []
+ self.code_blocks = []
+ self.discarded_blocks = []
+ self.ref_text_blocks = []
+ self.phonetic_blocks = []
+ self.list_blocks = []
+
+ for block in blocks:
+ if block["type"] in VISUAL_MAIN_TYPES or block["type"] in GENERIC_CHILD_TYPES:
+ continue
+ elif block["type"] == BlockType.INTERLINE_EQUATION:
+ self.interline_equation_blocks.append(block)
+ elif block["type"] == BlockType.TEXT:
+ self.text_blocks.append(block)
+ elif block["type"] == BlockType.TITLE:
+ self.title_blocks.append(block)
+ elif block["type"] == BlockType.REF_TEXT:
+ self.ref_text_blocks.append(block)
+ elif block["type"] == BlockType.PHONETIC:
+ self.phonetic_blocks.append(block)
+ elif block["type"] in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_NUMBER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ self.discarded_blocks.append(block)
+ elif block["type"] == BlockType.LIST:
+ self.list_blocks.append(block)
+
+ self.list_blocks, self.text_blocks, self.ref_text_blocks = fix_list_blocks(
+ self.list_blocks,
+ self.text_blocks,
+ self.ref_text_blocks,
+ )
+
+ visual_groups, unmatched_child_blocks = regroup_visual_blocks(blocks)
+ self.image_blocks = visual_groups[BlockType.IMAGE]
+ self.table_blocks = visual_groups[BlockType.TABLE]
+ self.chart_blocks = visual_groups[BlockType.CHART]
+ self.code_blocks = visual_groups[BlockType.CODE]
+
+ for code_block in self.code_blocks:
+ for block in code_block["blocks"]:
+ if block["type"] == BlockType.CODE_BODY:
+ if block["lines"]:
+ line = block["lines"][0]
+ code_block["sub_type"] = line["extra"]["type"]
+ if code_block["sub_type"] == "code":
+ code_block["guess_lang"] = line["extra"]["guess_lang"]
+ del line["extra"]
+ else:
+ code_block["sub_type"] = "code"
+ code_block["guess_lang"] = "txt"
+
+ for block in unmatched_child_blocks:
+ block["type"] = BlockType.TEXT
+ self.text_blocks.append(block)
+
+ def get_list_blocks(self):
+ return self.list_blocks
+
+ def get_image_blocks(self):
+ return self.image_blocks
+
+ def get_table_blocks(self):
+ return self.table_blocks
+
+ def get_chart_blocks(self):
+ return self.chart_blocks
+
+ def get_code_blocks(self):
+ return self.code_blocks
+
+ def get_ref_text_blocks(self):
+ return self.ref_text_blocks
+
+ def get_phonetic_blocks(self):
+ return self.phonetic_blocks
+
+ def get_title_blocks(self):
+ return self.title_blocks
+
+ def get_text_blocks(self):
+ return self.text_blocks
+
+ def get_interline_equation_blocks(self):
+ return self.interline_equation_blocks
+
+ def get_discarded_blocks(self):
+ return self.discarded_blocks
+
+ def get_all_spans(self):
+ return self.all_spans
+def fix_list_blocks(list_blocks, text_blocks, ref_text_blocks):
+ for list_block in list_blocks:
+ list_block["blocks"] = []
+ if "lines" in list_block:
+ del list_block["lines"]
+
+ temp_text_blocks = text_blocks + ref_text_blocks
+ need_remove_blocks = []
+ for block in temp_text_blocks:
+ for list_block in list_blocks:
+ if (
+ calculate_overlap_area_in_bbox1_area_ratio(
+ block["bbox"],
+ list_block["bbox"],
+ )
+ >= 0.8
+ ):
+ list_block["blocks"].append(block)
+ need_remove_blocks.append(block)
+ break
+
+ for block in need_remove_blocks:
+ if block in text_blocks:
+ text_blocks.remove(block)
+ elif block in ref_text_blocks:
+ ref_text_blocks.remove(block)
+
+ # 移除blocks为空的list_block
+ list_blocks = [lb for lb in list_blocks if lb["blocks"]]
+
+ for list_block in list_blocks:
+ # 统计list_block["blocks"]中所有block的type，用众数作为list_block的sub_type
+ type_count = {}
+ for sub_block in list_block["blocks"]:
+ sub_block_type = sub_block["type"]
+ if sub_block_type not in type_count:
+ type_count[sub_block_type] = 0
+ type_count[sub_block_type] += 1
+
+ if type_count:
+ list_block["sub_type"] = max(type_count, key=type_count.get)
+ else:
+ list_block["sub_type"] = "unknown"
+
+ return list_blocks, text_blocks, ref_text_blocks
diff --git a/mineru/backend/vlm/vlm_middle_json_mkcontent.py b/mineru/backend/vlm/vlm_middle_json_mkcontent.py
new file mode 100644
index 000000000..439e054c3
--- /dev/null
+++ b/mineru/backend/vlm/vlm_middle_json_mkcontent.py
@@ -0,0 +1,895 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import re
+from html import escape, unescape
+
+from loguru import logger
+
+from mineru.utils.char_utils import full_to_half_exclude_marks, is_hyphen_at_line_end
+from mineru.utils.config_reader import get_latex_delimiter_config, get_formula_enable, get_table_enable
+from mineru.utils.enum_class import MakeMode, BlockType, ContentType, ContentTypeV2
+from mineru.utils.language import detect_lang
+from mineru.backend.utils.markdown_utils import (
+ escape_conservative_markdown_text,
+ escape_text_block_markdown_prefix,
+)
+
+latex_delimiters_config = get_latex_delimiter_config()
+
+default_delimiters = {
+ 'display': {'left': '$$', 'right': '$$'},
+ 'inline': {'left': '$', 'right': '$'}
+}
+
+delimiters = latex_delimiters_config if latex_delimiters_config else default_delimiters
+
+display_left_delimiter = delimiters['display']['left']
+display_right_delimiter = delimiters['display']['right']
+inline_left_delimiter = delimiters['inline']['left']
+inline_right_delimiter = delimiters['inline']['right']
+
+
+def _prefix_table_img_src(html, img_buket_path):
+ """Prefix non-data image sources in table HTML with img_buket_path."""
+ if not html or not img_buket_path:
+ return html
+
+ return re.sub(
+ r'src="(?!data:)([^"]+)"',
+ lambda match: f'src="{img_buket_path}/{match.group(1)}"',
+ html,
+ )
+
+
+def _replace_eq_tags_in_table_html(html):
+ """Replace ... tags in table HTML with inline math delimiters."""
+ if not html:
+ return html
+
+ return re.sub(
+ r'(.*?) ',
+ lambda match: (
+ f" {inline_left_delimiter}{unescape(match.group(1))}{inline_right_delimiter} "
+ ),
+ html,
+ flags=re.DOTALL,
+ )
+
+
+def _format_embedded_html(html, img_buket_path):
+ """Normalize embedded table HTML for markdown/content outputs."""
+ return _replace_eq_tags_in_table_html(_prefix_table_img_src(html, img_buket_path))
+
+
+def _build_media_path(img_buket_path, image_path):
+ if not image_path:
+ return ''
+ if not img_buket_path:
+ return image_path
+ return f"{img_buket_path}/{image_path}"
+
+
+def _apply_visual_sub_type(para_content, para_block):
+ sub_type = para_block.get('sub_type')
+ if sub_type:
+ para_content['sub_type'] = sub_type
+
+
+def _build_visual_details_block(content, span_type, summary_override=''):
+ if not isinstance(content, str) or not content.strip():
+ return ''
+
+ if span_type == ContentType.CHART:
+ summary = summary_override or "chart content"
+ else:
+ summary = summary_override or "image content"
+
+ return (
+ " 1:
+ table_nest_level = 2
+ else:
+ table_nest_level = 1
+ if (
+ "colspan" in table_html or
+ "rowspan" in table_html or
+ table_nest_level > 1
+ ):
+ table_type = ContentTypeV2.TABLE_COMPLEX
+ else:
+ table_type = ContentTypeV2.TABLE_SIMPLE
+
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.TABLE_CAPTION:
+ table_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.TABLE_FOOTNOTE:
+ table_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.TABLE,
+ 'content': {
+ 'image_source': image_source,
+ 'table_caption': table_caption,
+ 'table_footnote': table_footnote,
+ 'html': table_html,
+ 'table_type': table_type,
+ 'table_nest_level': table_nest_level,
+ }
+ }
+ elif para_type == BlockType.CHART:
+ chart_caption = []
+ chart_footnote = []
+ image_path, chart_content = get_body_data(para_block)
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CHART_CAPTION:
+ chart_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.CHART_FOOTNOTE:
+ chart_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.CHART,
+ 'content': {
+ 'image_source': {
+ 'path': _build_media_path(img_buket_path, image_path),
+ },
+ 'content': chart_content if chart_content else '',
+ 'chart_caption': chart_caption,
+ 'chart_footnote': chart_footnote,
+ }
+ }
+ _apply_visual_sub_type(para_content, para_block)
+ elif para_type == BlockType.CODE:
+ code_caption = []
+ code_content = []
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CODE_CAPTION:
+ code_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.CODE_BODY:
+ code_content = merge_para_with_text_v2(block)
+ sub_type = para_block["sub_type"]
+ if sub_type == BlockType.CODE:
+ para_content = {
+ 'type': ContentTypeV2.CODE,
+ 'content': {
+ 'code_caption': code_caption,
+ 'code_content': code_content,
+ 'code_language': para_block.get('guess_lang', 'txt'),
+ }
+ }
+ elif sub_type == BlockType.ALGORITHM:
+ para_content = {
+ 'type': ContentTypeV2.ALGORITHM,
+ 'content': {
+ 'algorithm_caption': code_caption,
+ 'algorithm_content': code_content,
+ }
+ }
+ else:
+ raise ValueError(f"Unknown code sub_type: {sub_type}")
+ elif para_type == BlockType.REF_TEXT:
+ para_content = {
+ 'type': ContentTypeV2.LIST,
+ 'content': {
+ 'list_type': ContentTypeV2.LIST_REF,
+ 'list_items': [
+ {
+ 'item_type': 'text',
+ 'item_content': merge_para_with_text_v2(para_block),
+ }
+ ],
+ }
+ }
+ elif para_type == BlockType.LIST:
+ if 'sub_type' in para_block:
+ if para_block['sub_type'] == BlockType.REF_TEXT:
+ list_type = ContentTypeV2.LIST_REF
+ elif para_block['sub_type'] == BlockType.TEXT:
+ list_type = ContentTypeV2.LIST_TEXT
+ else:
+ raise ValueError(f"Unknown list sub_type: {para_block['sub_type']}")
+ else:
+ list_type = ContentTypeV2.LIST_TEXT
+ list_items = []
+ for block in para_block['blocks']:
+ item_content = merge_para_with_text_v2(block)
+ if item_content:
+ list_items.append({
+ 'item_type': 'text',
+ 'item_content': item_content,
+ })
+ para_content = {
+ 'type': ContentTypeV2.LIST,
+ 'content': {
+ 'list_type': list_type,
+ 'list_items': list_items,
+ }
+ }
+
+ page_width, page_height = page_size
+ para_bbox = para_block.get('bbox')
+ if para_bbox:
+ x0, y0, x1, y1 = para_bbox
+ para_content['bbox'] = [
+ int(x0 * 1000 / page_width),
+ int(y0 * 1000 / page_height),
+ int(x1 * 1000 / page_width),
+ int(y1 * 1000 / page_height),
+ ]
+
+ return para_content
+
+
+
+
+
+def get_body_data(para_block):
+ """
+ Extract image_path and body content from para_block
+ Returns:
+ - For IMAGE: (image_path, content)
+ - For TABLE: (image_path, html)
+ - For CHART: (image_path, content)
+ - For INTERLINE_EQUATION: (image_path, content)
+ - Default: ('', '')
+ """
+
+ def get_data_from_spans(lines):
+ for line in lines:
+ for span in line.get('spans', []):
+ span_type = span.get('type')
+ if span_type == ContentType.TABLE:
+ return span.get('image_path', ''), span.get('html', '')
+ elif span_type == ContentType.CHART:
+ return span.get('image_path', ''), span.get('content', '')
+ elif span_type == ContentType.IMAGE:
+ return span.get('image_path', ''), span.get('content', '')
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ return span.get('image_path', ''), span.get('content', '')
+ elif span_type == ContentType.TEXT:
+ return '', span.get('content', '')
+ return '', ''
+
+ # 处理嵌套的 blocks 结构
+ if 'blocks' in para_block:
+ for block in para_block['blocks']:
+ block_type = block.get('type')
+ if block_type in [BlockType.IMAGE_BODY, BlockType.TABLE_BODY, BlockType.CHART_BODY, BlockType.CODE_BODY]:
+ result = get_data_from_spans(block.get('lines', []))
+ if result != ('', ''):
+ return result
+ return '', ''
+
+ # 处理直接包含 lines 的结构
+ return get_data_from_spans(para_block.get('lines', []))
+
+
+def merge_para_with_text_v2(para_block):
+ block_text = ''
+ for line in para_block['lines']:
+ for span in line['spans']:
+ if span['type'] in [ContentType.TEXT]:
+ span['content'] = full_to_half_exclude_marks(span['content'])
+ block_text += span['content']
+ block_lang = detect_lang(block_text)
+
+ para_content = []
+ para_type = para_block['type']
+ for i, line in enumerate(para_block['lines']):
+ for j, span in enumerate(line['spans']):
+ span_type = span['type']
+ if span.get("content", '').strip():
+ if span_type == ContentType.TEXT:
+ if para_type == BlockType.PHONETIC:
+ span_type = ContentTypeV2.SPAN_PHONETIC
+ else:
+ span_type = ContentTypeV2.SPAN_TEXT
+ if span_type == ContentType.INLINE_EQUATION:
+ span_type = ContentTypeV2.SPAN_EQUATION_INLINE
+ if span_type in [
+ ContentTypeV2.SPAN_TEXT,
+ ]:
+ # 定义CJK语言集合(中日韩)
+ cjk_langs = {'zh', 'ja', 'ko'}
+ # logger.info(f'block_lang: {block_lang}, content: {content}')
+
+ # 判断是否为行末span
+ is_last_span = j == len(line['spans']) - 1
+
+ if block_lang in cjk_langs: # 中文/日语/韩文语境下，换行不需要空格分隔,但是如果是行内公式结尾，还是要加空格
+ if is_last_span:
+ span_content = span['content']
+ else:
+ span_content = f"{span['content']} "
+ else:
+ # 如果span是line的最后一个且末尾带有-连字符，那么末尾不应该加空格,同时应该把-删除
+ if (
+ is_last_span
+ and is_hyphen_at_line_end(span['content'])
+ ):
+ # 如果下一行的第一个span是小写字母开头，删除连字符
+ if (
+ i + 1 < len(para_block['lines'])
+ and para_block['lines'][i + 1].get('spans')
+ and para_block['lines'][i + 1]['spans'][0].get('type') == ContentType.TEXT
+ and para_block['lines'][i + 1]['spans'][0].get('content', '')
+ and para_block['lines'][i + 1]['spans'][0]['content'][0].islower()
+ ):
+ span_content = span['content'][:-1]
+ else: # 如果没有下一行，或者下一行的第一个span不是小写字母开头，则保留连字符但不加空格
+ span_content = span['content']
+ else:
+ # 西方文本语境下content间需要空格分隔
+ span_content = f"{span['content']} "
+
+ if para_content and para_content[-1]['type'] == span_type:
+ # 合并相同类型的span
+ para_content[-1]['content'] += span_content
+ else:
+ span_content = {
+ 'type': span_type,
+ 'content': span_content,
+ }
+ para_content.append(span_content)
+
+ elif span_type in [
+ ContentTypeV2.SPAN_PHONETIC,
+ ContentTypeV2.SPAN_EQUATION_INLINE,
+ ]:
+ span_content = {
+ 'type': span_type,
+ 'content': span['content'],
+ }
+ para_content.append(span_content)
+ else:
+ logger.warning(f"Unknown span type in merge_para_with_text_v2: {span_type}")
+ return para_content
+
+
+def union_make(pdf_info_dict: list,
+ make_mode: str,
+ img_buket_path: str = '',
+ ):
+
+ formula_enable = get_formula_enable(os.getenv('MINERU_VLM_FORMULA_ENABLE', 'True').lower() == 'true')
+ table_enable = get_table_enable(os.getenv('MINERU_VLM_TABLE_ENABLE', 'True').lower() == 'true')
+
+ output_content = []
+ for page_info in pdf_info_dict:
+ paras_of_layout = page_info.get('para_blocks')
+ paras_of_discarded = page_info.get('discarded_blocks')
+ page_idx = page_info.get('page_idx')
+ page_size = page_info.get('page_size')
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ if not paras_of_layout:
+ continue
+ page_markdown = mk_blocks_to_markdown(paras_of_layout, make_mode, formula_enable, table_enable, img_buket_path)
+ output_content.extend(page_markdown)
+ elif make_mode == MakeMode.CONTENT_LIST:
+ para_blocks = (paras_of_layout or []) + (paras_of_discarded or [])
+ if not para_blocks:
+ continue
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list(para_block, img_buket_path, page_idx, page_size)
+ output_content.append(para_content)
+ elif make_mode == MakeMode.CONTENT_LIST_V2:
+ # https://github.com/drunkpig/llm-webkit-mirror/blob/dev6/docs/specification/output_format/content_list_spec.md
+ para_blocks = (paras_of_layout or []) + (paras_of_discarded or [])
+ page_contents = []
+ if para_blocks:
+ for para_block in para_blocks:
+ para_content = make_blocks_to_content_list_v2(para_block, img_buket_path, page_size)
+ page_contents.append(para_content)
+ output_content.append(page_contents)
+
+ if make_mode in [MakeMode.MM_MD, MakeMode.NLP_MD]:
+ return '\n\n'.join(output_content)
+ elif make_mode in [MakeMode.CONTENT_LIST, MakeMode.CONTENT_LIST_V2]:
+ return output_content
+ return None
+
+
+def get_title_level(block):
+ title_level = block.get('level', 1)
+ if title_level > 4:
+ title_level = 4
+ elif title_level < 1:
+ title_level = 0
+ return title_level
diff --git a/mineru/cli/__init__.py b/mineru/cli/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/cli/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/cli/api_client.py b/mineru/cli/api_client.py
new file mode 100644
index 000000000..9370f5f00
--- /dev/null
+++ b/mineru/cli/api_client.py
@@ -0,0 +1,730 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
+import atexit
+import json
+import mimetypes
+import os
+import signal
+import socket
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+import zipfile
+from contextlib import ExitStack
+from dataclasses import dataclass
+from pathlib import Path, PurePosixPath
+from typing import Callable, Optional, Sequence
+
+import click
+import httpx
+from loguru import logger
+
+from mineru.cli.api_protocol import (
+ API_PROTOCOL_VERSION,
+ DEFAULT_MAX_CONCURRENT_REQUESTS,
+)
+from mineru.utils.config_reader import (
+ get_max_concurrent_requests as read_max_concurrent_requests,
+)
+
+HEALTH_ENDPOINT = "/health"
+TASKS_ENDPOINT = "/tasks"
+TASK_STATUS_POLL_INTERVAL_SECONDS = 1.0
+TASK_RESULT_TIMEOUT_SECONDS = 3600
+LOCAL_API_SHUTDOWN_TIMEOUT_SECONDS = 10
+LOCAL_API_CLEANUP_RETRIES = 8
+LOCAL_API_CLEANUP_RETRY_INTERVAL_SECONDS = 0.25
+PROCESS_TREE_CLEANUP_GRACE_SECONDS = 0.1
+
+
+def get_float_env(name: str, default: float, minimum: float = 0.0) -> float:
+ value = os.getenv(name)
+ if value is None:
+ return default
+ try:
+ resolved = float(value)
+ except ValueError:
+ logger.warning(
+ "Invalid {} value: {}. Expected a number, using default {}.",
+ name,
+ value,
+ default,
+ )
+ return default
+ if resolved < minimum:
+ logger.warning(
+ "Invalid {} value: {}. Expected a number >= {}, using default {}.",
+ name,
+ value,
+ minimum,
+ default,
+ )
+ return default
+ return resolved
+
+
+def get_local_api_startup_timeout_seconds(default: float = 300.0) -> float:
+ return get_float_env(
+ "MINERU_LOCAL_API_STARTUP_TIMEOUT_SECONDS",
+ default,
+ minimum=1.0,
+ )
+
+
+LOCAL_API_STARTUP_TIMEOUT_SECONDS = get_local_api_startup_timeout_seconds()
+
+
+def build_managed_process_popen_kwargs() -> dict[str, object]:
+ if os.name == "nt":
+ creationflags = getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
+ if creationflags:
+ return {"creationflags": creationflags}
+ return {}
+ return {"start_new_session": True}
+
+
+def _signal_process_tree(process: subprocess.Popen[bytes], *, force: bool) -> None:
+ if os.name == "nt":
+ command = ["taskkill", "/PID", str(process.pid), "/T"]
+ if force:
+ command.append("/F")
+ try:
+ subprocess.run(
+ command,
+ check=False,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ )
+ except Exception as exc:
+ logger.debug(
+ "Failed to signal managed MinerU process tree {} on Windows: {}",
+ process.pid,
+ exc,
+ )
+ return
+
+ sig = signal.SIGKILL if force else signal.SIGTERM
+ try:
+ os.killpg(process.pid, sig)
+ except ProcessLookupError:
+ return
+ except OSError as exc:
+ logger.debug(
+ "Failed to signal managed MinerU process group {}: {}",
+ process.pid,
+ exc,
+ )
+
+
+def cleanup_process_tree_descendants(
+ process: subprocess.Popen[bytes],
+ *,
+ grace_seconds: float = PROCESS_TREE_CLEANUP_GRACE_SECONDS,
+) -> None:
+ if os.name == "nt":
+ return
+
+ _signal_process_tree(process, force=False)
+ if grace_seconds > 0:
+ time.sleep(grace_seconds)
+ _signal_process_tree(process, force=True)
+
+
+def stop_managed_process(
+ process: subprocess.Popen[bytes] | None,
+ *,
+ shutdown_timeout_seconds: float,
+ use_stdin_shutdown_watcher: bool,
+) -> None:
+ if process is None:
+ return
+
+ was_running_at_entry = process.poll() is None
+ exited_via_stdin_eof = False
+ tree_signaled = False
+
+ if not was_running_at_entry:
+ return
+
+ if use_stdin_shutdown_watcher:
+ if process.stdin is not None and not process.stdin.closed:
+ process.stdin.close()
+ try:
+ process.wait(timeout=shutdown_timeout_seconds)
+ exited_via_stdin_eof = True
+ except subprocess.TimeoutExpired:
+ logger.debug(
+ "Managed MinerU process did not stop after stdin EOF within {}s. Falling back to process-tree termination.",
+ shutdown_timeout_seconds,
+ )
+
+ if process.poll() is None:
+ _signal_process_tree(process, force=False)
+ tree_signaled = True
+ try:
+ process.wait(timeout=shutdown_timeout_seconds)
+ except subprocess.TimeoutExpired:
+ pass
+
+ if process.poll() is None:
+ _signal_process_tree(process, force=True)
+ tree_signaled = True
+ try:
+ process.wait(timeout=shutdown_timeout_seconds)
+ except subprocess.TimeoutExpired:
+ logger.warning(
+ "Managed MinerU process {} did not exit after forceful stop.",
+ process.pid,
+ )
+
+ if exited_via_stdin_eof and not tree_signaled:
+ cleanup_process_tree_descendants(process)
+
+
+@dataclass(frozen=True)
+class UploadAsset:
+ path: Path
+ upload_name: str
+
+
+@dataclass(frozen=True)
+class ServerHealth:
+ base_url: str
+ max_concurrent_requests: int
+ processing_window_size: int
+
+
+@dataclass(frozen=True)
+class SubmitResponse:
+ task_id: str
+ status_url: str
+ result_url: str
+ file_names: tuple[str, ...] = ()
+ queued_ahead: int | None = None
+
+
+@dataclass(frozen=True)
+class TaskStatusSnapshot:
+ status: str
+ queued_ahead: int | None = None
+
+
+class LocalAPIServer:
+ def __init__(self, extra_cli_args: Sequence[str] = ()):
+ self.temp_dir = tempfile.TemporaryDirectory(prefix="mineru-api-client-")
+ self.temp_root = Path(self.temp_dir.name)
+ self.output_root = self.temp_root / "output"
+ self.base_url: str | None = None
+ self.process: subprocess.Popen[bytes] | None = None
+ self._atexit_registered = False
+ self.extra_cli_args = tuple(extra_cli_args)
+ # On Windows, the temporary FastAPI child process can stall during parsing
+ # startup when launched with stdin=PIPE and an EOF-based shutdown watcher.
+ # Use explicit process termination there instead of stdin-driven shutdown.
+ self._use_stdin_shutdown_watcher = os.name != "nt"
+
+ def start(self) -> str:
+ if self.process is not None:
+ raise RuntimeError("Local API server is already running")
+
+ resolved_port = find_free_port()
+ remaining_cli_args = strip_local_api_network_args(self.extra_cli_args)
+ self.base_url = f"http://127.0.0.1:{resolved_port}"
+ env = os.environ.copy()
+ env["MINERU_API_OUTPUT_ROOT"] = str(self.output_root)
+ env["MINERU_API_MAX_CONCURRENT_REQUESTS"] = str(
+ read_max_concurrent_requests(default=DEFAULT_MAX_CONCURRENT_REQUESTS)
+ )
+ env["MINERU_API_DISABLE_ACCESS_LOG"] = "1"
+ if self._use_stdin_shutdown_watcher:
+ env["MINERU_API_SHUTDOWN_ON_STDIN_EOF"] = "1"
+ stdin_target = subprocess.PIPE
+ else:
+ env.pop("MINERU_API_SHUTDOWN_ON_STDIN_EOF", None)
+ stdin_target = subprocess.DEVNULL
+ self.output_root.mkdir(parents=True, exist_ok=True)
+
+ command = [
+ sys.executable,
+ "-m",
+ "mineru.cli.fast_api",
+ "--host",
+ "127.0.0.1",
+ "--port",
+ str(resolved_port),
+ *remaining_cli_args,
+ ]
+ self.process = subprocess.Popen(
+ command,
+ cwd=os.getcwd(),
+ env=env,
+ stdin=stdin_target,
+ **build_managed_process_popen_kwargs(),
+ )
+
+ if not self._atexit_registered:
+ atexit.register(self.stop)
+ self._atexit_registered = True
+ return self.base_url
+
+ def stop(self) -> None:
+ process = self.process
+ self.process = None
+ try:
+ if process is not None:
+ stop_managed_process(
+ process,
+ shutdown_timeout_seconds=LOCAL_API_SHUTDOWN_TIMEOUT_SECONDS,
+ use_stdin_shutdown_watcher=self._use_stdin_shutdown_watcher,
+ )
+ finally:
+ if self._atexit_registered:
+ try:
+ atexit.unregister(self.stop)
+ except Exception:
+ pass
+ self._atexit_registered = False
+ self._cleanup_temp_dir()
+
+ def _cleanup_temp_dir(self) -> None:
+ last_error: Exception | None = None
+ for attempt in range(LOCAL_API_CLEANUP_RETRIES):
+ try:
+ self.temp_dir.cleanup()
+ return
+ except FileNotFoundError:
+ return
+ except Exception as exc:
+ last_error = exc
+ if attempt + 1 < LOCAL_API_CLEANUP_RETRIES:
+ time.sleep(LOCAL_API_CLEANUP_RETRY_INTERVAL_SECONDS)
+
+ if last_error is not None:
+ logger.warning(
+ "Failed to clean up temporary MinerU API directory {}: {}. "
+ "You can remove it manually after processes release any open handles.",
+ self.temp_root,
+ last_error,
+ )
+
+
+class ReusableLocalAPIServer:
+ def __init__(self, extra_cli_args: Sequence[str] = ()):
+ self._lock = threading.Lock()
+ self._server: LocalAPIServer | None = None
+ self._extra_cli_args = tuple(extra_cli_args)
+
+ def configure(self, extra_cli_args: Sequence[str]) -> None:
+ with self._lock:
+ self._extra_cli_args = tuple(extra_cli_args)
+ server = self._server
+ if server is None:
+ return
+ if server.process is not None and server.process.poll() is None:
+ return
+ self._server = None
+
+ def ensure_started(self) -> tuple[LocalAPIServer, bool]:
+ with self._lock:
+ server = self._server
+ if server is not None and server.process is not None and server.process.poll() is None:
+ return server, False
+
+ if server is not None:
+ server.stop()
+
+ server = LocalAPIServer(extra_cli_args=self._extra_cli_args)
+ server.start()
+ self._server = server
+ return server, True
+
+ def stop(self) -> None:
+ with self._lock:
+ server = self._server
+ self._server = None
+ if server is not None:
+ server.stop()
+
+
+def build_http_timeout() -> httpx.Timeout:
+ return httpx.Timeout(connect=10, read=60, write=300, pool=30)
+
+
+def find_free_port() -> int:
+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+ sock.bind(("127.0.0.1", 0))
+ sock.listen(1)
+ return int(sock.getsockname()[1])
+
+
+def strip_local_api_network_args(extra_cli_args: Sequence[str]) -> tuple[str, ...]:
+ remaining_args: list[str] = []
+ i = 0
+
+ while i < len(extra_cli_args):
+ arg = extra_cli_args[i]
+ if arg == "--host":
+ next_index = i + 1
+ if next_index < len(extra_cli_args) and not extra_cli_args[next_index].startswith("--"):
+ i += 2
+ else:
+ i += 1
+ continue
+
+ if arg == "--port":
+ next_index = i + 1
+ if next_index < len(extra_cli_args) and not extra_cli_args[next_index].startswith("--"):
+ i += 2
+ else:
+ i += 1
+ continue
+
+ if arg.startswith("--host="):
+ i += 1
+ continue
+
+ if arg.startswith("--port="):
+ i += 1
+ continue
+
+ remaining_args.append(arg)
+ i += 1
+
+ return tuple(remaining_args)
+
+
+def normalize_base_url(url: str) -> str:
+ return url.rstrip("/")
+
+
+def resolve_effective_max_concurrent_requests(
+ local_max: int,
+ server_max: int,
+) -> int:
+ if local_max <= 0 or server_max <= 0:
+ raise ValueError(
+ "local_max and server_max must both be positive integers"
+ )
+ return min(local_max, server_max)
+
+
+def response_detail(response: httpx.Response) -> str:
+ try:
+ payload = response.json()
+ except Exception:
+ text = response.text.strip()
+ return text or response.reason_phrase
+
+ if isinstance(payload, dict):
+ detail = payload.get("detail")
+ if isinstance(detail, str):
+ return detail
+ error = payload.get("error")
+ if isinstance(error, str):
+ return error
+ message = payload.get("message")
+ if isinstance(message, str):
+ return message
+ return json.dumps(payload, ensure_ascii=False)
+
+
+def validate_server_health_payload(payload: dict, base_url: str) -> ServerHealth:
+ status = payload.get("status")
+ if status != "healthy":
+ raise click.ClickException(
+ f"MinerU API at {base_url} is not healthy: {json.dumps(payload, ensure_ascii=False)}"
+ )
+
+ protocol_version = payload.get("protocol_version")
+ if protocol_version != API_PROTOCOL_VERSION:
+ raise click.ClickException(
+ f"MinerU API at {base_url} returned protocol_version={protocol_version}, "
+ f"expected {API_PROTOCOL_VERSION}"
+ )
+
+ max_concurrent_requests = payload.get("max_concurrent_requests")
+ processing_window_size = payload.get("processing_window_size")
+ if not isinstance(max_concurrent_requests, int) or max_concurrent_requests <= 0:
+ raise click.ClickException(
+ f"MinerU API at {base_url} did not return a valid positive max_concurrent_requests"
+ )
+ if not isinstance(processing_window_size, int):
+ raise click.ClickException(
+ f"MinerU API at {base_url} did not return a valid processing_window_size"
+ )
+
+ return ServerHealth(
+ base_url=base_url,
+ max_concurrent_requests=max_concurrent_requests,
+ processing_window_size=max(1, processing_window_size),
+ )
+
+
+async def fetch_server_health(
+ client: httpx.AsyncClient,
+ base_url: str,
+) -> ServerHealth:
+ response = await client.get(f"{base_url}{HEALTH_ENDPOINT}")
+ if response.status_code != 200:
+ raise click.ClickException(
+ f"Failed to query MinerU API health from {base_url}: "
+ f"{response.status_code} {response_detail(response)}"
+ )
+ return validate_server_health_payload(response.json(), base_url)
+
+
+async def wait_for_local_api_ready(
+ client: httpx.AsyncClient,
+ local_server: LocalAPIServer,
+ timeout_seconds: float = LOCAL_API_STARTUP_TIMEOUT_SECONDS,
+) -> ServerHealth:
+ assert local_server.base_url is not None
+ deadline = asyncio.get_running_loop().time() + timeout_seconds
+ last_error: str | None = None
+
+ while asyncio.get_running_loop().time() < deadline:
+ process = local_server.process
+ if process is not None and process.poll() is not None:
+ raise click.ClickException(
+ "Local mineru-api exited before becoming healthy."
+ )
+ try:
+ return await fetch_server_health(client, local_server.base_url)
+ except click.ClickException as exc:
+ last_error = str(exc)
+ except httpx.HTTPError as exc:
+ last_error = str(exc)
+ await asyncio.sleep(TASK_STATUS_POLL_INTERVAL_SECONDS)
+
+ message = "Timed out waiting for local mineru-api to become healthy."
+ if last_error:
+ message = f"{message} {last_error}"
+ raise click.ClickException(message)
+
+
+def build_parse_request_form_data(
+ lang_list: Sequence[str],
+ backend: str,
+ parse_method: str,
+ formula_enable: bool,
+ table_enable: bool,
+ server_url: Optional[str],
+ start_page_id: int,
+ end_page_id: Optional[int],
+ *,
+ return_md: bool,
+ return_middle_json: bool,
+ return_model_output: bool,
+ return_content_list: bool,
+ return_images: bool,
+ response_format_zip: bool,
+ return_original_file: bool,
+) -> dict[str, str | list[str]]:
+ effective_lang_list = list(lang_list) or ["ch"]
+ data: dict[str, str | list[str]] = {
+ "lang_list": effective_lang_list,
+ "backend": backend,
+ "parse_method": parse_method,
+ "formula_enable": str(formula_enable).lower(),
+ "table_enable": str(table_enable).lower(),
+ "return_md": str(return_md).lower(),
+ "return_middle_json": str(return_middle_json).lower(),
+ "return_model_output": str(return_model_output).lower(),
+ "return_content_list": str(return_content_list).lower(),
+ "return_images": str(return_images).lower(),
+ "response_format_zip": str(response_format_zip).lower(),
+ "return_original_file": str(return_original_file).lower(),
+ "start_page_id": str(start_page_id),
+ "end_page_id": str(99999 if end_page_id is None else end_page_id),
+ }
+ if server_url:
+ data["server_url"] = server_url
+ return data
+
+
+async def submit_parse_task(
+ base_url: str,
+ upload_assets: Sequence[UploadAsset],
+ form_data: dict[str, str | list[str]],
+) -> SubmitResponse:
+ return await asyncio.to_thread(
+ submit_parse_task_sync,
+ base_url,
+ upload_assets,
+ form_data,
+ )
+
+
+def submit_parse_task_sync(
+ base_url: str,
+ upload_assets: Sequence[UploadAsset],
+ form_data: dict[str, str | list[str]],
+) -> SubmitResponse:
+ with httpx.Client(timeout=build_http_timeout(), follow_redirects=True) as sync_client:
+ with ExitStack() as stack:
+ files = []
+ for upload_asset in upload_assets:
+ mime_type = (
+ mimetypes.guess_type(upload_asset.upload_name)[0]
+ or "application/octet-stream"
+ )
+ file_handle = stack.enter_context(open(upload_asset.path, "rb"))
+ files.append(
+ (
+ "files",
+ (
+ upload_asset.upload_name,
+ file_handle,
+ mime_type,
+ ),
+ )
+ )
+
+ response = sync_client.post(
+ f"{base_url}{TASKS_ENDPOINT}",
+ data=form_data,
+ files=files,
+ )
+
+ if response.status_code != 202:
+ raise click.ClickException(
+ f"Failed to submit parsing task: "
+ f"{response.status_code} {response_detail(response)}"
+ )
+
+ payload = response.json()
+ task_id = payload.get("task_id")
+ status_url = payload.get("status_url")
+ result_url = payload.get("result_url")
+ file_names = payload.get("file_names")
+ queued_ahead = payload.get("queued_ahead")
+ if (
+ not isinstance(task_id, str)
+ or not isinstance(status_url, str)
+ or not isinstance(result_url, str)
+ ):
+ raise click.ClickException("MinerU API returned an invalid task payload")
+
+ normalized_file_names: tuple[str, ...] = ()
+ if isinstance(file_names, list) and all(isinstance(name, str) for name in file_names):
+ normalized_file_names = tuple(file_names)
+ if not isinstance(queued_ahead, int):
+ queued_ahead = None
+
+ return SubmitResponse(
+ task_id=task_id,
+ status_url=status_url,
+ result_url=result_url,
+ file_names=normalized_file_names,
+ queued_ahead=queued_ahead,
+ )
+
+
+async def wait_for_task_result(
+ client: httpx.AsyncClient,
+ submit_response: SubmitResponse,
+ task_label: str,
+ *,
+ status_callback: Optional[Callable[[str], None]] = None,
+ status_snapshot_callback: Optional[Callable[[TaskStatusSnapshot], None]] = None,
+ timeout_seconds: float = TASK_RESULT_TIMEOUT_SECONDS,
+) -> None:
+ deadline = asyncio.get_running_loop().time() + timeout_seconds
+ while asyncio.get_running_loop().time() < deadline:
+ try:
+ response = await client.get(submit_response.status_url)
+ except httpx.ReadTimeout:
+ logger.warning(
+ "Timed out while polling task status for {} (task_id={}). "
+ "This can happen during cold start; retrying until the task deadline.",
+ task_label,
+ submit_response.task_id,
+ )
+ await asyncio.sleep(TASK_STATUS_POLL_INTERVAL_SECONDS)
+ continue
+ if response.status_code != 200:
+ raise click.ClickException(
+ f"Failed to query task status for {task_label}: "
+ f"{response.status_code} {response_detail(response)}"
+ )
+
+ payload = response.json()
+ status = payload.get("status")
+ if status in {"pending", "processing"}:
+ queued_ahead = payload.get("queued_ahead")
+ if not isinstance(queued_ahead, int):
+ queued_ahead = None
+ if status_snapshot_callback is not None:
+ status_snapshot_callback(
+ TaskStatusSnapshot(
+ status=status,
+ queued_ahead=queued_ahead,
+ )
+ )
+ if status_callback is not None:
+ status_callback(status)
+ await asyncio.sleep(TASK_STATUS_POLL_INTERVAL_SECONDS)
+ continue
+ if status == "completed":
+ return
+ raise click.ClickException(
+ f"Task {submit_response.task_id} failed for {task_label}: "
+ f"{json.dumps(payload, ensure_ascii=False)}"
+ )
+
+ raise click.ClickException(
+ f"Timed out waiting for result of task {submit_response.task_id} "
+ f"for {task_label}"
+ )
+
+
+async def download_result_zip(
+ client: httpx.AsyncClient,
+ submit_response: SubmitResponse,
+ task_label: str,
+) -> Path:
+ response = await client.get(submit_response.result_url)
+ if response.status_code != 200:
+ raise click.ClickException(
+ f"Failed to download result ZIP for task {submit_response.task_id}: "
+ f"{response.status_code} {response_detail(response)}"
+ )
+ content_type = response.headers.get("content-type", "")
+ if "application/zip" not in content_type:
+ raise click.ClickException(
+ f"Expected a ZIP result for {task_label}, "
+ f"got content-type={content_type or 'unknown'}"
+ )
+
+ zip_fd, zip_path = tempfile.mkstemp(suffix=".zip", prefix="mineru_cli_result_")
+ os.close(zip_fd)
+ Path(zip_path).write_bytes(response.content)
+ return Path(zip_path)
+
+
+def safe_extract_zip(zip_path: Path, output_dir: Path) -> None:
+ output_dir.mkdir(parents=True, exist_ok=True)
+ output_root = output_dir.resolve()
+
+ with zipfile.ZipFile(zip_path, "r") as zip_file:
+ for member in zip_file.infolist():
+ member_path = PurePosixPath(member.filename)
+ if member_path.is_absolute() or ".." in member_path.parts:
+ raise click.ClickException(
+ f"Refusing to extract unsafe ZIP entry: {member.filename}"
+ )
+ target_path = (output_root / Path(*member_path.parts)).resolve()
+ if target_path != output_root and output_root not in target_path.parents:
+ raise click.ClickException(
+ f"Refusing to extract unsafe ZIP entry: {member.filename}"
+ )
+
+ if member.is_dir():
+ target_path.mkdir(parents=True, exist_ok=True)
+ continue
+
+ target_path.parent.mkdir(parents=True, exist_ok=True)
+ with zip_file.open(member, "r") as source, open(target_path, "wb") as handle:
+ handle.write(source.read())
diff --git a/mineru/cli/api_protocol.py b/mineru/cli/api_protocol.py
new file mode 100644
index 000000000..dee49f098
--- /dev/null
+++ b/mineru/cli/api_protocol.py
@@ -0,0 +1,4 @@
+# Copyright (c) Opendatalab. All rights reserved.
+API_PROTOCOL_VERSION = 1
+DEFAULT_MAX_CONCURRENT_REQUESTS = 3
+DEFAULT_PROCESSING_WINDOW_SIZE = 64
diff --git a/mineru/cli/client.py b/mineru/cli/client.py
new file mode 100644
index 000000000..bc7a58159
--- /dev/null
+++ b/mineru/cli/client.py
@@ -0,0 +1,1124 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
+import os
+import sys
+import threading
+from concurrent.futures import Future, ProcessPoolExecutor
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Awaitable, Callable, Optional, TextIO
+
+import click
+import httpx
+import pypdfium2 as pdfium
+from loguru import logger
+
+from mineru.cli.api_protocol import (
+ DEFAULT_MAX_CONCURRENT_REQUESTS,
+ DEFAULT_PROCESSING_WINDOW_SIZE,
+)
+from mineru.utils.config_reader import (
+ get_max_concurrent_requests as read_max_concurrent_requests,
+)
+from mineru.utils.guess_suffix_or_lang import guess_suffix_by_path
+from mineru.utils.pdf_page_id import get_end_page_id
+from mineru.utils.pdfium_guard import (
+ close_pdfium_document,
+ get_pdfium_document_page_count,
+ open_pdfium_document,
+)
+
+from mineru.version import __version__
+from mineru.cli.common import (
+ HybridDependencyError,
+ ensure_backend_dependencies,
+ image_suffixes,
+ office_suffixes,
+ pdf_suffixes,
+ uniquify_task_stems,
+)
+from mineru.cli import api_client as _api_client
+from mineru.cli.output_paths import resolve_parse_dir
+from mineru.cli.visualization import (
+ VisualizationJob,
+ run_visualization_job,
+)
+
+os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+log_level = os.getenv("MINERU_LOG_LEVEL", "INFO").upper()
+
+@dataclass(frozen=True)
+class InputDocument:
+ path: Path
+ suffix: str
+ stem: str
+ effective_pages: int
+ order: int
+
+
+@dataclass
+class PlannedTask:
+ index: int
+ documents: list[InputDocument]
+ total_pages: int
+
+
+@dataclass
+class TaskExecutionProgress:
+ total_tasks: int
+ total_pages: int
+ completed_tasks: int
+ completed_pages: int
+ lock: asyncio.Lock
+
+
+@dataclass
+class VisualizationContext:
+ executor: ProcessPoolExecutor
+ futures: list[Future]
+
+
+ServerHealth = _api_client.ServerHealth
+SubmitResponse = _api_client.SubmitResponse
+LocalAPIServer = _api_client.LocalAPIServer
+
+
+@dataclass(frozen=True)
+class TaskFailure:
+ task_index: int
+ document_stems: tuple[str, ...]
+ message: str
+
+
+@dataclass
+class LiveTaskStatusState:
+ task_index: int
+ task_id: str
+ status: str
+ queued_ahead: int | None = None
+ frame_step: int = 0
+
+
+class LiveAwareStderrSink:
+ def __init__(self, stream: TextIO):
+ self.stream = stream
+ self.lock = threading.RLock()
+ self.renderer: LiveTaskStatusRenderer | None = None
+
+ def set_renderer(self, renderer: "LiveTaskStatusRenderer | None") -> None:
+ with self.lock:
+ self.renderer = renderer
+
+ def isatty(self) -> bool:
+ return bool(getattr(self.stream, "isatty", lambda: False)())
+
+ def write(self, message: str) -> None:
+ with self.lock:
+ renderer = self.renderer
+ if renderer is not None:
+ renderer.clear_locked()
+ self.stream.write(message)
+ self.stream.flush()
+ if renderer is not None:
+ renderer.render_locked()
+
+ def flush(self) -> None:
+ self.stream.flush()
+
+ def stop(self) -> None:
+ self.flush()
+
+
+class LiveTaskStatusRenderer:
+ BAR_WIDTH = 12
+ RUNNER_WIDTH = 4
+ ACTIVE_STATUSES = {"pending", "processing"}
+
+ def __init__(self, sink: LiveAwareStderrSink):
+ self.sink = sink
+ self._rendered_line_count = 0
+ self._task_states: dict[str, LiveTaskStatusState] = {}
+
+ def register_task(
+ self,
+ task: PlannedTask,
+ task_id: str,
+ queued_ahead: int | None = None,
+ ) -> None:
+ with self.sink.lock:
+ self._task_states[task_id] = LiveTaskStatusState(
+ task_index=task.index,
+ task_id=task_id,
+ status="pending",
+ queued_ahead=queued_ahead,
+ )
+ self.render_locked()
+
+ def update_status(
+ self,
+ task_id: str,
+ status_update: _api_client.TaskStatusSnapshot | str,
+ ) -> None:
+ with self.sink.lock:
+ state = self._task_states.get(task_id)
+ if state is None:
+ return
+ if isinstance(status_update, _api_client.TaskStatusSnapshot):
+ status = status_update.status
+ queued_ahead = (
+ status_update.queued_ahead if status == "pending" else None
+ )
+ else:
+ status = status_update
+ queued_ahead = None
+
+ state.status = status
+ state.queued_ahead = queued_ahead
+ if status in self.ACTIVE_STATUSES:
+ state.frame_step += 1
+ self.render_locked()
+
+ def remove_task(self, task_id: str) -> None:
+ with self.sink.lock:
+ if self._task_states.pop(task_id, None) is None:
+ return
+ self.render_locked()
+
+ def close(self) -> None:
+ with self.sink.lock:
+ self._task_states.clear()
+ self.clear_locked()
+
+ def clear_locked(self) -> None:
+ if self._rendered_line_count <= 0:
+ return
+
+ self.sink.stream.write(f"\x1b[{self._rendered_line_count}A\r")
+ for index in range(self._rendered_line_count):
+ self.sink.stream.write("\x1b[2K")
+ if index + 1 < self._rendered_line_count:
+ self.sink.stream.write("\x1b[1B\r")
+ if self._rendered_line_count > 1:
+ self.sink.stream.write(f"\x1b[{self._rendered_line_count - 1}A\r")
+ self.sink.stream.flush()
+ self._rendered_line_count = 0
+
+ def render_locked(self) -> None:
+ self.clear_locked()
+ lines = self._build_render_lines_locked()
+ if lines:
+ self.sink.stream.write("\n".join(lines))
+ self.sink.stream.write("\n")
+ self.sink.stream.flush()
+ self._rendered_line_count = len(lines)
+
+ def _build_render_lines_locked(self) -> list[str]:
+ states = sorted(
+ self._task_states.values(),
+ key=lambda state: (state.task_index, state.task_id),
+ )
+ return [
+ self._build_render_line_locked(state)
+ for state in states
+ ]
+
+ @staticmethod
+ def _build_render_line_locked(state: LiveTaskStatusState) -> str:
+ parts = [f"status={state.status}"]
+ if state.status == "pending" and state.queued_ahead is not None:
+ parts.append(f"ahead={state.queued_ahead}")
+ parts.append(f"task_id={state.task_id}")
+ return f"{LiveTaskStatusRenderer._build_bar(state.frame_step)} {' | '.join(parts)}"
+
+ @classmethod
+ def _build_bar(cls, frame_step: int) -> str:
+ cells = [" "] * cls.BAR_WIDTH
+ runner_start = frame_step % cls.BAR_WIDTH
+ for offset in range(cls.RUNNER_WIDTH):
+ position = (runner_start + offset) % cls.BAR_WIDTH
+ cells[position] = "="
+ head_position = (runner_start + cls.RUNNER_WIDTH - 1) % cls.BAR_WIDTH
+ cells[head_position] = ">"
+ return f"[{''.join(cells)}]"
+
+
+def create_live_task_status_renderer(
+ api_url: Optional[str],
+) -> Optional[LiveTaskStatusRenderer]:
+ if api_url is None or not _stderr_sink.isatty():
+ _stderr_sink.set_renderer(None)
+ return None
+
+ renderer = LiveTaskStatusRenderer(_stderr_sink)
+ _stderr_sink.set_renderer(renderer)
+ return renderer
+
+
+_stderr_sink = LiveAwareStderrSink(sys.stderr)
+logger.remove()
+logger.add(_stderr_sink, level=log_level)
+
+
+def build_http_timeout() -> httpx.Timeout:
+ return _api_client.build_http_timeout()
+
+def find_free_port() -> int:
+ return _api_client.find_free_port()
+
+
+def normalize_base_url(url: str) -> str:
+ return _api_client.normalize_base_url(url)
+
+
+def format_task_label(task: PlannedTask) -> str:
+ doc_names = ", ".join(doc.stem for doc in task.documents)
+ return f"task#{task.index} [{doc_names}]"
+
+
+def format_task_log_label(task: PlannedTask, max_documents: int = 3) -> str:
+ if len(task.documents) <= max_documents:
+ return format_task_label(task)
+
+ visible_names = ", ".join(doc.stem for doc in task.documents[:max_documents])
+ hidden_count = len(task.documents) - max_documents
+ return f"task#{task.index} [{visible_names}, +{hidden_count} more]"
+
+
+def format_count(value: int, singular: str, plural: Optional[str] = None) -> str:
+ unit = singular if value == 1 else (plural or f"{singular}s")
+ return f"{value} {unit}"
+
+
+def format_task_submission_message(
+ task: PlannedTask,
+ progress: TaskExecutionProgress,
+) -> str:
+ return (
+ f"Submitting batch {task.index}/{progress.total_tasks} | "
+ f"{format_count(len(task.documents), 'document')}, "
+ f"{format_count(task.total_pages, 'page')} in this batch | "
+ f"{format_count(progress.total_pages, 'page')} total | "
+ f"{format_task_log_label(task)}"
+ )
+
+
+def format_task_completion_message(
+ task: PlannedTask,
+ progress: TaskExecutionProgress,
+ completed_tasks: int,
+ completed_pages: int,
+) -> str:
+ batch_word = "batch" if progress.total_tasks == 1 else "batches"
+ page_word = "page" if progress.total_pages == 1 else "pages"
+ return (
+ f"Completed batch {task.index}/{progress.total_tasks} | "
+ f"Processed {completed_pages}/{progress.total_pages} {page_word} | "
+ f"{completed_tasks} of {progress.total_tasks} {batch_word} finished | "
+ f"{format_task_log_label(task)}"
+ )
+
+
+def build_task_execution_progress(
+ planned_tasks: list[PlannedTask],
+) -> TaskExecutionProgress:
+ return TaskExecutionProgress(
+ total_tasks=len(planned_tasks),
+ total_pages=sum(task.total_pages for task in planned_tasks),
+ completed_tasks=0,
+ completed_pages=0,
+ lock=asyncio.Lock(),
+ )
+
+
+async def mark_task_completed(
+ progress: TaskExecutionProgress,
+ completed_pages: int,
+) -> tuple[int, int]:
+ async with progress.lock:
+ progress.completed_tasks += 1
+ progress.completed_pages += completed_pages
+ return progress.completed_tasks, progress.completed_pages
+
+
+def create_visualization_context() -> Optional[VisualizationContext]:
+ try:
+ return VisualizationContext(
+ executor=ProcessPoolExecutor(max_workers=1),
+ futures=[],
+ )
+ except Exception as exc:
+ logger.warning(f"Failed to start visualization worker process: {exc}")
+ return None
+
+
+def build_visualization_jobs(
+ planned_task: PlannedTask,
+ output_dir: Path,
+ backend: str,
+ parse_method: str,
+) -> list[VisualizationJob]:
+ draw_span = backend.startswith("pipeline")
+ return [
+ VisualizationJob(
+ document_stem=document.stem,
+ backend=backend,
+ parse_method=parse_method,
+ parse_dir=resolve_parse_dir(
+ output_dir,
+ document.stem,
+ backend,
+ parse_method,
+ is_office=document.suffix in office_suffixes,
+ ),
+ draw_span=draw_span,
+ )
+ for document in planned_task.documents
+ ]
+
+
+def log_visualization_future_result(
+ job: VisualizationJob,
+ future: Future,
+) -> None:
+ try:
+ result = future.result()
+ except Exception as exc:
+ logger.warning(f"Skipping visualization for {job.document_stem}: {exc}")
+ return
+
+ if result.status != "finished":
+ logger.warning(
+ f"Skipping visualization for {result.document_stem}: {result.message}"
+ )
+
+
+def queue_visualization_jobs(
+ visualization_context: Optional[VisualizationContext],
+ jobs: list[VisualizationJob],
+ planned_task: PlannedTask,
+) -> int:
+ if visualization_context is None or not jobs:
+ return 0
+
+ del planned_task
+ queued_jobs = 0
+ for job in jobs:
+ try:
+ future = visualization_context.executor.submit(run_visualization_job, job)
+ except Exception as exc:
+ logger.warning(f"Skipping visualization for {job.document_stem}: {exc}")
+ continue
+
+ future.add_done_callback(
+ lambda completed_future, job=job: log_visualization_future_result(
+ job,
+ completed_future,
+ )
+ )
+ visualization_context.futures.append(future)
+ queued_jobs += 1
+
+ return queued_jobs
+
+
+async def wait_for_visualization_jobs(
+ visualization_context: Optional[VisualizationContext],
+) -> None:
+ if visualization_context is None:
+ return
+
+ try:
+ if visualization_context.futures:
+ await asyncio.gather(
+ *(
+ asyncio.wrap_future(future)
+ for future in visualization_context.futures
+ ),
+ return_exceptions=True,
+ )
+ finally:
+ visualization_context.executor.shutdown(wait=True)
+
+
+def response_detail(response: httpx.Response) -> str:
+ return _api_client.response_detail(response)
+
+
+def validate_server_health_payload(payload: dict, base_url: str) -> ServerHealth:
+ return _api_client.validate_server_health_payload(payload, base_url)
+
+
+async def fetch_server_health(
+ client: httpx.AsyncClient,
+ base_url: str,
+) -> ServerHealth:
+ return await _api_client.fetch_server_health(client, base_url)
+
+
+async def wait_for_local_api_ready(
+ client: httpx.AsyncClient,
+ local_server: LocalAPIServer,
+ timeout_seconds: float = _api_client.LOCAL_API_STARTUP_TIMEOUT_SECONDS,
+) -> ServerHealth:
+ return await _api_client.wait_for_local_api_ready(
+ client,
+ local_server,
+ timeout_seconds=timeout_seconds,
+ )
+
+
+def probe_pdf_effective_pages(
+ path: Path,
+ start_page_id: int,
+ end_page_id: Optional[int],
+) -> int:
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, str(path))
+ try:
+ page_count = get_pdfium_document_page_count(pdf_doc)
+ finally:
+ close_pdfium_document(pdf_doc)
+
+ if page_count <= 0:
+ raise click.ClickException(f"PDF has no pages: {path}")
+
+ effective_end_page_id = get_end_page_id(end_page_id, page_count)
+ if start_page_id > effective_end_page_id:
+ raise click.ClickException(
+ f"Requested page range is empty for PDF {path}: "
+ f"start={start_page_id}, end={end_page_id}"
+ )
+ return effective_end_page_id - start_page_id + 1
+
+
+def collect_input_documents(
+ input_path: Path,
+ start_page_id: int,
+ end_page_id: Optional[int],
+) -> list[InputDocument]:
+ documents: list[Path]
+ if input_path.is_dir():
+ documents = [path for path in sorted(input_path.glob("*")) if path.is_file()]
+ else:
+ documents = [input_path]
+
+ collected: list[InputDocument] = []
+ for order, path in enumerate(documents):
+ suffix = guess_suffix_by_path(path)
+ if suffix not in pdf_suffixes + image_suffixes + office_suffixes:
+ continue
+
+ if suffix in pdf_suffixes:
+ effective_pages = probe_pdf_effective_pages(
+ path,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+ else:
+ effective_pages = 1
+
+ collected.append(
+ InputDocument(
+ path=path,
+ suffix=suffix,
+ stem=path.stem,
+ effective_pages=effective_pages,
+ order=order,
+ )
+ )
+
+ if not collected:
+ raise click.ClickException(f"No supported documents found under {input_path}")
+
+ normalized_stems, renamed_stems = uniquify_task_stems(
+ [document.stem for document in collected]
+ )
+ if renamed_stems:
+ rename_details = ", ".join(
+ f"{document.path.name} -> {effective_stem}"
+ for document, effective_stem in zip(collected, normalized_stems)
+ if document.stem != effective_stem
+ )
+ logger.warning(
+ f"Normalized duplicate document stems within this run: {rename_details}"
+ )
+ return [
+ InputDocument(
+ path=document.path,
+ suffix=document.suffix,
+ stem=effective_stem,
+ effective_pages=document.effective_pages,
+ order=document.order,
+ )
+ for document, effective_stem in zip(collected, normalized_stems)
+ ]
+
+ return collected
+
+
+def plan_pipeline_tasks(
+ documents: list[InputDocument],
+ processing_window_size: int,
+) -> list[PlannedTask]:
+ bins: list[PlannedTask] = []
+ sorted_docs = sorted(
+ documents,
+ key=lambda doc: (-doc.effective_pages, doc.order),
+ )
+
+ for document in sorted_docs:
+ if document.effective_pages > processing_window_size:
+ bins.append(
+ PlannedTask(
+ index=len(bins) + 1,
+ documents=[document],
+ total_pages=document.effective_pages,
+ )
+ )
+ continue
+
+ candidates = [
+ task
+ for task in bins
+ if task.total_pages + document.effective_pages <= processing_window_size
+ ]
+ if candidates:
+ selected = min(candidates, key=lambda task: (task.total_pages, task.index))
+ selected.documents.append(document)
+ selected.total_pages += document.effective_pages
+ continue
+
+ bins.append(
+ PlannedTask(
+ index=len(bins) + 1,
+ documents=[document],
+ total_pages=document.effective_pages,
+ )
+ )
+
+ for index, task in enumerate(bins, start=1):
+ task.index = index
+ return bins
+
+
+def plan_tasks(
+ documents: list[InputDocument],
+ backend: str,
+ processing_window_size: int,
+) -> list[PlannedTask]:
+ if backend == "pipeline":
+ return plan_pipeline_tasks(documents, processing_window_size)
+ return [
+ PlannedTask(index=index, documents=[document], total_pages=document.effective_pages)
+ for index, document in enumerate(documents, start=1)
+ ]
+
+
+def build_request_form_data(
+ lang: str,
+ backend: str,
+ method: str,
+ formula_enable: bool,
+ table_enable: bool,
+ server_url: Optional[str],
+ start_page_id: int,
+ end_page_id: Optional[int],
+) -> dict[str, str | list[str]]:
+ return _api_client.build_parse_request_form_data(
+ lang_list=[lang],
+ backend=backend,
+ parse_method=method,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ return_md=True,
+ return_middle_json=True,
+ return_model_output=True,
+ return_content_list=True,
+ return_images=True,
+ response_format_zip=True,
+ return_original_file=True,
+ )
+
+
+async def submit_task(
+ client: httpx.AsyncClient,
+ base_url: str,
+ planned_task: PlannedTask,
+ form_data: dict[str, str | list[str]],
+) -> SubmitResponse:
+ del client
+ upload_assets = [
+ _api_client.UploadAsset(
+ path=document.path,
+ upload_name=f"{document.stem}{document.path.suffix}",
+ )
+ for document in planned_task.documents
+ ]
+ return await _api_client.submit_parse_task(
+ base_url=base_url,
+ upload_assets=upload_assets,
+ form_data=form_data,
+ )
+
+
+async def wait_for_task_result(
+ client: httpx.AsyncClient,
+ submit_response: SubmitResponse,
+ planned_task: PlannedTask,
+ live_renderer: Optional[LiveTaskStatusRenderer] = None,
+ timeout_seconds: float = _api_client.TASK_RESULT_TIMEOUT_SECONDS,
+) -> None:
+ return await _api_client.wait_for_task_result(
+ client=client,
+ submit_response=submit_response,
+ task_label=format_task_label(planned_task),
+ status_snapshot_callback=(
+ None
+ if live_renderer is None
+ else lambda snapshot: live_renderer.update_status(
+ submit_response.task_id,
+ snapshot,
+ )
+ ),
+ timeout_seconds=timeout_seconds,
+ )
+
+
+async def download_result_zip(
+ client: httpx.AsyncClient,
+ submit_response: SubmitResponse,
+ planned_task: PlannedTask,
+) -> Path:
+ return await _api_client.download_result_zip(
+ client=client,
+ submit_response=submit_response,
+ task_label=format_task_label(planned_task),
+ )
+
+
+def safe_extract_zip(zip_path: Path, output_dir: Path) -> None:
+ _api_client.safe_extract_zip(zip_path, output_dir)
+
+
+def resolve_submit_concurrency(max_concurrent_requests: int, task_count: int) -> int:
+ if max_concurrent_requests <= 0:
+ raise ValueError("max_concurrent_requests must be a positive integer")
+ return max(1, min(max_concurrent_requests, task_count))
+
+
+def resolve_effective_max_concurrent_requests(
+ local_max: int,
+ server_max: int,
+) -> int:
+ return _api_client.resolve_effective_max_concurrent_requests(
+ local_max=local_max,
+ server_max=server_max,
+ )
+
+
+async def execute_planned_tasks(
+ planned_tasks: list[PlannedTask],
+ concurrency: int,
+ task_runner: Callable[[PlannedTask], Awaitable[None]],
+) -> list[TaskFailure]:
+ queue: asyncio.Queue[PlannedTask | None] = asyncio.Queue()
+ failures: list[TaskFailure] = []
+
+ for task in planned_tasks:
+ await queue.put(task)
+ for _ in range(concurrency):
+ await queue.put(None)
+
+ async def worker() -> None:
+ while True:
+ planned_task = await queue.get()
+ try:
+ if planned_task is None:
+ return
+ await task_runner(planned_task)
+ except Exception as exc:
+ assert planned_task is not None
+ failures.append(
+ TaskFailure(
+ task_index=planned_task.index,
+ document_stems=tuple(doc.stem for doc in planned_task.documents),
+ message=str(exc),
+ )
+ )
+ finally:
+ queue.task_done()
+
+ workers = [asyncio.create_task(worker()) for _ in range(concurrency)]
+ await queue.join()
+ await asyncio.gather(*workers, return_exceptions=True)
+ return failures
+
+
+async def run_planned_task(
+ client: httpx.AsyncClient,
+ server_health: ServerHealth,
+ planned_task: PlannedTask,
+ progress: TaskExecutionProgress,
+ backend: str,
+ parse_method: str,
+ visualization_context: Optional[VisualizationContext],
+ form_data: dict[str, str],
+ output_dir: Path,
+ live_renderer: Optional[LiveTaskStatusRenderer] = None,
+) -> None:
+ logger.info(format_task_submission_message(planned_task, progress))
+ submit_response = await submit_task(
+ client=client,
+ base_url=server_health.base_url,
+ planned_task=planned_task,
+ form_data=form_data,
+ )
+ if live_renderer is not None:
+ live_renderer.register_task(
+ planned_task,
+ submit_response.task_id,
+ queued_ahead=submit_response.queued_ahead,
+ )
+ try:
+ await wait_for_task_result(
+ client=client,
+ submit_response=submit_response,
+ planned_task=planned_task,
+ live_renderer=live_renderer,
+ )
+ finally:
+ if live_renderer is not None:
+ live_renderer.remove_task(submit_response.task_id)
+ zip_path = await download_result_zip(
+ client=client,
+ submit_response=submit_response,
+ planned_task=planned_task,
+ )
+ try:
+ safe_extract_zip(zip_path, output_dir)
+ finally:
+ zip_path.unlink(missing_ok=True)
+ completed_tasks, completed_pages = await mark_task_completed(
+ progress,
+ planned_task.total_pages,
+ )
+ logger.info(
+ format_task_completion_message(
+ planned_task,
+ progress,
+ completed_tasks,
+ completed_pages,
+ )
+ )
+ try:
+ visualization_jobs = build_visualization_jobs(
+ planned_task,
+ output_dir,
+ backend,
+ parse_method,
+ )
+ except Exception as exc:
+ logger.warning(
+ f"Skipping visualization for {format_task_log_label(planned_task)}: {exc}"
+ )
+ else:
+ queue_visualization_jobs(
+ visualization_context,
+ visualization_jobs,
+ planned_task,
+ )
+
+
+async def run_orchestrated_cli(
+ input_path: Path,
+ output_dir: Path,
+ method: str,
+ backend: str,
+ lang: str,
+ server_url: Optional[str],
+ api_url: Optional[str],
+ start_page_id: int,
+ end_page_id: Optional[int],
+ formula_enable: bool,
+ table_enable: bool,
+ extra_cli_args: tuple[str, ...] = (),
+) -> None:
+ if start_page_id < 0:
+ raise click.ClickException("--start must be greater than or equal to 0")
+ if end_page_id is not None and end_page_id < 0:
+ raise click.ClickException("--end must be greater than or equal to 0")
+ if api_url is None:
+ try:
+ ensure_backend_dependencies(backend)
+ except HybridDependencyError as exc:
+ raise click.ClickException(str(exc)) from exc
+
+ output_dir.mkdir(parents=True, exist_ok=True)
+ documents = collect_input_documents(
+ input_path=input_path,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+
+ timeout = build_http_timeout()
+ local_server: LocalAPIServer | None = None
+ visualization_context: Optional[VisualizationContext] = None
+ live_renderer: Optional[LiveTaskStatusRenderer] = None
+ async with httpx.AsyncClient(timeout=timeout, follow_redirects=True) as http_client:
+ try:
+ if api_url is None:
+ local_server = LocalAPIServer(extra_cli_args=extra_cli_args)
+ base_url = local_server.start()
+ logger.info(f"Started local mineru-api at {base_url}")
+ server_health = await wait_for_local_api_ready(http_client, local_server)
+ effective_max_concurrent_requests = (
+ server_health.max_concurrent_requests
+ )
+ else:
+ server_health = await fetch_server_health(
+ http_client,
+ normalize_base_url(api_url),
+ )
+ effective_max_concurrent_requests = (
+ resolve_effective_max_concurrent_requests(
+ read_max_concurrent_requests(
+ default=DEFAULT_MAX_CONCURRENT_REQUESTS
+ ),
+ server_health.max_concurrent_requests,
+ )
+ )
+ live_renderer = create_live_task_status_renderer(api_url)
+
+ planned_tasks = plan_tasks(
+ documents=documents,
+ backend=backend,
+ processing_window_size=server_health.processing_window_size
+ if backend == "pipeline"
+ else DEFAULT_PROCESSING_WINDOW_SIZE,
+ )
+ progress = build_task_execution_progress(planned_tasks)
+ concurrency = resolve_submit_concurrency(
+ effective_max_concurrent_requests,
+ len(planned_tasks),
+ )
+ form_data = build_request_form_data(
+ lang=lang,
+ backend=backend,
+ method=method,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+ visualization_context = create_visualization_context()
+ failures = await execute_planned_tasks(
+ planned_tasks=planned_tasks,
+ concurrency=concurrency,
+ task_runner=lambda planned_task: run_planned_task(
+ client=http_client,
+ server_health=server_health,
+ planned_task=planned_task,
+ progress=progress,
+ backend=backend,
+ parse_method=method,
+ visualization_context=visualization_context,
+ form_data=form_data,
+ output_dir=output_dir,
+ live_renderer=live_renderer,
+ ),
+ )
+ if failures:
+ details = "\n".join(
+ f"- task#{failure.task_index} ({', '.join(failure.document_stems)}): {failure.message}"
+ for failure in sorted(failures, key=lambda item: item.task_index)
+ )
+ raise click.ClickException(
+ f"{len(failures)} task(s) failed while processing documents:\n{details}"
+ )
+ finally:
+ try:
+ if local_server is not None:
+ local_server.stop()
+ finally:
+ try:
+ await wait_for_visualization_jobs(visualization_context)
+ finally:
+ if live_renderer is not None:
+ live_renderer.close()
+ _stderr_sink.set_renderer(None)
+
+
+@click.command(context_settings=dict(ignore_unknown_options=True, allow_extra_args=True))
+@click.pass_context
+@click.version_option(__version__, "--version", "-v", help="display the version and exit")
+@click.option(
+ "-p",
+ "--path",
+ "input_path",
+ type=click.Path(exists=True, path_type=Path),
+ required=True,
+ help="local filepath or directory. support pdf, image, docx, pptx, xlsx files",
+)
+@click.option(
+ "-o",
+ "--output",
+ "output_dir",
+ type=click.Path(path_type=Path),
+ required=True,
+ help="output local directory",
+)
+@click.option(
+ "--api-url",
+ "api_url",
+ type=str,
+ default=None,
+ help="MinerU FastAPI base URL. If omitted, mineru starts a temporary local mineru-api service.",
+)
+@click.option(
+ "-m",
+ "--method",
+ "method",
+ type=click.Choice(["auto", "txt", "ocr"]),
+ default="auto",
+ help="""\b
+ the method for parsing pdf:
+ auto: Automatically determine the method based on the file type.
+ txt: Use text extraction method.
+ ocr: Use OCR method for image-based PDFs.
+ Without method specified, 'auto' will be used by default.
+ Adapted only for the case where the backend is set to 'pipeline' and 'hybrid-*'.""",
+)
+@click.option(
+ "-b",
+ "--backend",
+ "backend",
+ type=click.Choice(
+ [
+ "pipeline",
+ "vlm-http-client",
+ "hybrid-http-client",
+ "vlm-auto-engine",
+ "hybrid-auto-engine",
+ ]
+ ),
+ default="hybrid-auto-engine",
+ help="""\b
+ the backend for parsing pdf:
+ pipeline: More general.
+ vlm-auto-engine: High accuracy via local computing power.
+ vlm-http-client: High accuracy via remote computing power(client suitable for openai-compatible servers).
+ hybrid-auto-engine: Next-generation high accuracy solution via local computing power.
+ hybrid-http-client: High accuracy but requires a little local computing power(client suitable for openai-compatible servers).
+ Without method specified, hybrid-auto-engine will be used by default.""",
+)
+@click.option(
+ "-l",
+ "--lang",
+ "lang",
+ type=click.Choice(
+ [
+ "ch",
+ "ch_server",
+ "ch_lite",
+ "en",
+ "korean",
+ "japan",
+ "chinese_cht",
+ "ta",
+ "te",
+ "ka",
+ "th",
+ "el",
+ "latin",
+ "arabic",
+ "east_slavic",
+ "cyrillic",
+ "devanagari",
+ ]
+ ),
+ default="ch",
+ help="""
+ Input the languages in the pdf (if known) to improve OCR accuracy.
+ Without languages specified, 'ch' will be used by default.
+ Adapted only for the case where the backend is set to 'pipeline' and 'hybrid-*'.
+ """,
+)
+@click.option(
+ "-u",
+ "--url",
+ "server_url",
+ type=str,
+ default=None,
+ help="""
+ When the backend is `-http-client`, you need to specify the server_url, for example:`http://127.0.0.1:30000`
+ """,
+)
+@click.option(
+ "-s",
+ "--start",
+ "start_page_id",
+ type=int,
+ default=0,
+ help="The starting page for PDF parsing, beginning from 0.",
+)
+@click.option(
+ "-e",
+ "--end",
+ "end_page_id",
+ type=int,
+ default=None,
+ help="The ending page for PDF parsing, beginning from 0.",
+)
+@click.option(
+ "-f",
+ "--formula",
+ "formula_enable",
+ type=bool,
+ default=True,
+ help="Enable formula parsing. Default is True. ",
+)
+@click.option(
+ "-t",
+ "--table",
+ "table_enable",
+ type=bool,
+ default=True,
+ help="Enable table parsing. Default is True. ",
+)
+def main(
+ ctx: click.Context,
+ input_path: Path,
+ output_dir: Path,
+ api_url: Optional[str],
+ method: str,
+ backend: str,
+ lang: str,
+ server_url: Optional[str],
+ start_page_id: int,
+ end_page_id: Optional[int],
+ formula_enable: bool,
+ table_enable: bool,
+) -> None:
+ asyncio.run(
+ run_orchestrated_cli(
+ input_path=input_path,
+ output_dir=output_dir,
+ method=method,
+ backend=backend,
+ lang=lang,
+ server_url=server_url,
+ api_url=api_url,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ extra_cli_args=tuple(ctx.args),
+ )
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/mineru/cli/common.py b/mineru/cli/common.py
new file mode 100644
index 000000000..96d559fa9
--- /dev/null
+++ b/mineru/cli/common.py
@@ -0,0 +1,814 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import importlib
+import importlib.util
+import json
+import os
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from typing import Sequence
+
+from loguru import logger
+
+from mineru.data.data_reader_writer import FileBasedDataWriter
+from mineru.utils.draw_bbox import draw_layout_bbox, draw_span_bbox
+from mineru.utils.engine_utils import get_vlm_engine
+from mineru.utils.enum_class import MakeMode
+from mineru.utils.guess_suffix_or_lang import guess_suffix_by_bytes
+from mineru.utils.pdf_image_tools import images_bytes_to_pdf_bytes
+from mineru.backend.vlm.vlm_middle_json_mkcontent import union_make as vlm_union_make
+from mineru.backend.office.office_middle_json_mkcontent import union_make as office_union_make
+from mineru.backend.vlm.vlm_analyze import doc_analyze as vlm_doc_analyze
+from mineru.backend.vlm.vlm_analyze import aio_doc_analyze as aio_vlm_doc_analyze
+from mineru.backend.office.pptx_analyze import office_pptx_analyze
+from mineru.backend.office.xlsx_analyze import office_xlsx_analyze
+from mineru.backend.office.docx_analyze import office_docx_analyze
+from mineru.utils.pdfium_guard import rewrite_pdf_bytes_with_pdfium
+
+os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+if os.getenv("MINERU_LMDEPLOY_DEVICE", "") == "maca":
+ import torch
+ torch.backends.cudnn.enabled = False
+
+
+pdf_suffixes = ["pdf"]
+image_suffixes = ["png", "jpeg", "jp2", "webp", "gif", "bmp", "jpg", "tiff"]
+docx_suffixes = ["docx"]
+pptx_suffixes = ["pptx"]
+xlsx_suffixes = ["xlsx"]
+office_suffixes = docx_suffixes + pptx_suffixes + xlsx_suffixes
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+# Maximum UTF-8 byte length allowed for task stems used in filenames.
+# 200 bytes is chosen to stay well below common filesystem limits (e.g. 255 bytes)
+# and to prevent generating excessively long or incompatible filenames.
+MAX_TASK_STEM_BYTES = 200
+
+
+class HybridDependencyError(RuntimeError):
+ pass
+
+
+def build_hybrid_dependency_error_message(backend: str) -> str:
+ return (
+ f"`{backend}` requires local pipeline dependencies (`mineru[pipeline]`, "
+ "including `torch`). Install `mineru[pipeline]` or `mineru[core]`. "
+ "If you need a lightweight remote client without local `torch`, "
+ "use `vlm-http-client` instead."
+ )
+
+
+def ensure_backend_dependencies(backend: str) -> None:
+ if not backend.startswith("hybrid-"):
+ return
+ if importlib.util.find_spec("torch") is None:
+ raise HybridDependencyError(build_hybrid_dependency_error_message(backend))
+
+
+def _load_hybrid_analyze_entrypoint(entrypoint_name: str, backend: str):
+ ensure_backend_dependencies(backend)
+ try:
+ hybrid_analyze = importlib.import_module("mineru.backend.hybrid.hybrid_analyze")
+ except (ImportError, ModuleNotFoundError) as exc:
+ raise HybridDependencyError(
+ build_hybrid_dependency_error_message(backend)
+ ) from exc
+ return getattr(hybrid_analyze, entrypoint_name)
+
+
+def utf8_byte_length(value: str) -> int:
+ return len(value.encode("utf-8"))
+
+
+def truncate_to_utf8_bytes(value: str, max_bytes: int) -> str:
+ if max_bytes <= 0:
+ return ""
+
+ encoded = value.encode("utf-8")
+ if len(encoded) <= max_bytes:
+ return value
+
+ truncated = encoded[:max_bytes]
+ while truncated:
+ try:
+ return truncated.decode("utf-8")
+ except UnicodeDecodeError as exc:
+ truncated = truncated[:exc.start]
+ return ""
+
+
+def normalize_task_stem(stem: str, max_bytes: int = MAX_TASK_STEM_BYTES) -> str:
+ return truncate_to_utf8_bytes(stem, max_bytes)
+
+
+def normalize_upload_filename(upload_name: str) -> str:
+ sanitized_name = Path(upload_name).name
+ sanitized_path = Path(sanitized_name)
+ normalized_stem = normalize_task_stem(sanitized_path.stem)
+ return f"{normalized_stem}{sanitized_path.suffix}"
+
+
+def build_task_stem_candidate(
+ stem: str,
+ suffix: str = "",
+ max_bytes: int = MAX_TASK_STEM_BYTES,
+) -> str:
+ if utf8_byte_length(f"{stem}{suffix}") <= max_bytes:
+ return f"{stem}{suffix}"
+ suffix_bytes = utf8_byte_length(suffix)
+ if suffix_bytes >= max_bytes:
+ return truncate_to_utf8_bytes(suffix, max_bytes)
+ return f"{truncate_to_utf8_bytes(stem, max_bytes - suffix_bytes)}{suffix}"
+
+
+def uniquify_task_stems(
+ stems: Sequence[str],
+) -> tuple[list[str], list[tuple[str, str]]]:
+ """Assign task-local unique stems while preserving input order."""
+ normalized_inputs = [normalize_task_stem(stem) for stem in stems]
+ raw_keys = {stem.casefold() for stem in normalized_inputs}
+ occurrence_counts: dict[str, int] = {}
+ assigned_keys: set[str] = set()
+ unique_stems: list[str] = []
+ renamed: list[tuple[str, str]] = []
+
+ for stem, normalized_stem in zip(stems, normalized_inputs):
+ stem_base = normalized_stem or stem
+ stem_key = stem_base.casefold()
+ seen_count = occurrence_counts.get(stem_key, 0)
+ occurrence_counts[stem_key] = seen_count + 1
+
+ if seen_count == 0 and stem_key not in assigned_keys:
+ effective_stem = stem_base
+ else:
+ suffix = seen_count + 1
+ while True:
+ candidate = build_task_stem_candidate(stem_base, f"_{suffix}")
+ candidate_key = candidate.casefold()
+ if candidate_key not in raw_keys and candidate_key not in assigned_keys:
+ effective_stem = candidate
+ break
+ suffix += 1
+
+ assigned_keys.add(effective_stem.casefold())
+ unique_stems.append(effective_stem)
+ if effective_stem != stem:
+ renamed.append((stem, effective_stem))
+
+ return unique_stems, renamed
+
+
+def read_fn(path, file_suffix: str | None = None):
+ if not isinstance(path, Path):
+ path = Path(path)
+ with open(str(path), "rb") as input_file:
+ file_bytes = input_file.read()
+ if file_suffix is None:
+ file_suffix = guess_suffix_by_bytes(file_bytes, path)
+ if file_suffix in image_suffixes:
+ return images_bytes_to_pdf_bytes(file_bytes)
+ elif file_suffix in pdf_suffixes + office_suffixes:
+ return file_bytes
+ else:
+ raise Exception(f"Unknown file suffix: {file_suffix}")
+
+
+def prepare_env(output_dir, pdf_file_name, parse_method):
+ local_md_dir = str(os.path.join(output_dir, pdf_file_name, parse_method))
+ local_image_dir = os.path.join(str(local_md_dir), "images")
+ os.makedirs(local_image_dir, exist_ok=True)
+ os.makedirs(local_md_dir, exist_ok=True)
+ return local_image_dir, local_md_dir
+
+
+def convert_pdf_bytes_to_bytes(pdf_bytes, start_page_id=0, end_page_id=None):
+ try:
+ rebuilt_pdf_bytes = rewrite_pdf_bytes_with_pdfium(
+ pdf_bytes,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+ if rebuilt_pdf_bytes:
+ return rebuilt_pdf_bytes
+ logger.warning("PDFium rewrite returned empty bytes, using original PDF bytes.")
+ except Exception as fallback_error:
+ logger.warning(
+ f"Error in converting PDF bytes with pdfium: {fallback_error}, "
+ "using original PDF bytes."
+ )
+ return pdf_bytes
+
+
+def _prepare_pdf_bytes(pdf_bytes_list, start_page_id, end_page_id):
+ """准备处理PDF字节数据"""
+ result = []
+ for pdf_bytes in pdf_bytes_list:
+ new_pdf_bytes = convert_pdf_bytes_to_bytes(pdf_bytes, start_page_id, end_page_id)
+ result.append(new_pdf_bytes)
+ return result
+
+
+def _process_output(
+ pdf_info,
+ pdf_bytes,
+ pdf_file_name,
+ local_md_dir,
+ local_image_dir,
+ md_writer,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_orig_pdf,
+ f_dump_md,
+ f_dump_content_list,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_make_md_mode,
+ middle_json,
+ model_output=None,
+ process_mode="vlm",
+):
+ from mineru.backend.pipeline.pipeline_middle_json_mkcontent import union_make as pipeline_union_make
+ if process_mode == "pipeline":
+ make_func = pipeline_union_make
+ elif process_mode == "vlm":
+ make_func = vlm_union_make
+ elif process_mode in office_suffixes:
+ make_func = office_union_make
+ else:
+ raise Exception(f"Unknown process_mode: {process_mode}")
+ """处理输出文件"""
+ if f_draw_layout_bbox:
+ try:
+ draw_layout_bbox(pdf_info, pdf_bytes, local_md_dir, f"{pdf_file_name}_layout.pdf")
+ except Exception as exc:
+ logger.warning(f"Skipping layout bbox visualization for {pdf_file_name}: {exc}")
+
+ if f_draw_span_bbox:
+ try:
+ draw_span_bbox(pdf_info, pdf_bytes, local_md_dir, f"{pdf_file_name}_span.pdf")
+ except Exception as exc:
+ logger.warning(f"Skipping span bbox visualization for {pdf_file_name}: {exc}")
+
+ if f_dump_orig_pdf:
+ if process_mode in ["pipeline", "vlm"]:
+ md_writer.write(
+ f"{pdf_file_name}_origin.pdf",
+ pdf_bytes,
+ )
+ elif process_mode in office_suffixes:
+ md_writer.write(
+ f"{pdf_file_name}_origin.{process_mode}",
+ pdf_bytes,
+ )
+
+ image_dir = str(os.path.basename(local_image_dir))
+
+ if f_dump_md:
+ md_content_str = make_func(pdf_info, f_make_md_mode, image_dir)
+ md_writer.write_string(
+ f"{pdf_file_name}.md",
+ md_content_str,
+ )
+
+ if f_dump_content_list:
+
+ content_list = make_func(pdf_info, MakeMode.CONTENT_LIST, image_dir)
+ md_writer.write_string(
+ f"{pdf_file_name}_content_list.json",
+ json.dumps(content_list, ensure_ascii=False, indent=4),
+ )
+
+ content_list_v2 = make_func(pdf_info, MakeMode.CONTENT_LIST_V2, image_dir)
+ md_writer.write_string(
+ f"{pdf_file_name}_content_list_v2.json",
+ json.dumps(content_list_v2, ensure_ascii=False, indent=4),
+ )
+
+
+ if f_dump_middle_json:
+ md_writer.write_string(
+ f"{pdf_file_name}_middle.json",
+ json.dumps(middle_json, ensure_ascii=False, indent=4),
+ )
+
+ if f_dump_model_output:
+ md_writer.write_string(
+ f"{pdf_file_name}_model.json",
+ json.dumps(model_output, ensure_ascii=False, indent=4),
+ )
+
+ logger.debug(f"local output dir is {local_md_dir}")
+
+
+def _process_pipeline(
+ output_dir,
+ pdf_file_names,
+ pdf_bytes_list,
+ p_lang_list,
+ parse_method,
+ p_formula_enable,
+ p_table_enable,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_md,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_dump_orig_pdf,
+ f_dump_content_list,
+ f_make_md_mode,
+):
+ """处理pipeline后端逻辑"""
+ from mineru.backend.pipeline.pipeline_analyze import doc_analyze_streaming as pipeline_doc_analyze_streaming
+
+ image_writer_list = []
+ md_writer_list = []
+ local_output_info = []
+ for idx, pdf_bytes in enumerate(pdf_bytes_list):
+ pdf_file_name = pdf_file_names[idx]
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, parse_method)
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+ image_writer_list.append(image_writer)
+ md_writer_list.append(md_writer)
+ local_output_info.append((pdf_file_name, local_image_dir, local_md_dir))
+
+ output_futures = []
+
+ def run_output_task(doc_index, middle_json, model_list):
+ pdf_file_name, local_image_dir, local_md_dir = local_output_info[doc_index]
+ md_writer = md_writer_list[doc_index]
+ pdf_bytes = pdf_bytes_list[doc_index]
+ logger.debug(f"Pipeline output start: doc{doc_index}")
+ try:
+ _process_output(
+ middle_json["pdf_info"], pdf_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_pdf,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, model_list, process_mode="pipeline"
+ )
+ logger.debug(f"Pipeline output complete: doc{doc_index}")
+ except Exception:
+ logger.exception(f"Pipeline output failed: doc{doc_index}")
+ raise
+
+ with ThreadPoolExecutor(max_workers=1) as output_executor:
+ def on_doc_ready(doc_index, model_list, middle_json, ocr_enable):
+ logger.debug(
+ f"Pipeline doc ready: doc{doc_index} pages={len(middle_json['pdf_info'])} output_submitted=1"
+ )
+ future = output_executor.submit(run_output_task, doc_index, middle_json, model_list)
+ output_futures.append(future)
+
+ pipeline_doc_analyze_streaming(
+ pdf_bytes_list,
+ image_writer_list,
+ p_lang_list,
+ on_doc_ready,
+ parse_method=parse_method,
+ formula_enable=p_formula_enable,
+ table_enable=p_table_enable,
+ )
+
+ for future in output_futures:
+ future.result()
+ return
+
+
+async def _async_process_vlm(
+ output_dir,
+ pdf_file_names,
+ pdf_bytes_list,
+ backend,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_md,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_dump_orig_pdf,
+ f_dump_content_list,
+ f_make_md_mode,
+ server_url=None,
+ **kwargs,
+):
+ """异步处理VLM后端逻辑"""
+ parse_method = "vlm"
+ f_draw_span_bbox = False
+ if not backend.endswith("client"):
+ server_url = None
+
+ for idx, pdf_bytes in enumerate(pdf_bytes_list):
+ pdf_file_name = pdf_file_names[idx]
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, parse_method)
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+
+ middle_json, infer_result = await aio_vlm_doc_analyze(
+ pdf_bytes, image_writer=image_writer, backend=backend, server_url=server_url, **kwargs,
+ )
+
+ pdf_info = middle_json["pdf_info"]
+
+ _process_output(
+ pdf_info, pdf_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_pdf,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, infer_result, process_mode="vlm"
+ )
+
+
+def _process_vlm(
+ output_dir,
+ pdf_file_names,
+ pdf_bytes_list,
+ backend,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_md,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_dump_orig_pdf,
+ f_dump_content_list,
+ f_make_md_mode,
+ server_url=None,
+ **kwargs,
+):
+ """同步处理VLM后端逻辑"""
+ parse_method = "vlm"
+ f_draw_span_bbox = False
+ if not backend.endswith("client"):
+ server_url = None
+
+ for idx, pdf_bytes in enumerate(pdf_bytes_list):
+ pdf_file_name = pdf_file_names[idx]
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, parse_method)
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+
+ middle_json, infer_result = vlm_doc_analyze(
+ pdf_bytes, image_writer=image_writer, backend=backend, server_url=server_url, **kwargs,
+ )
+
+ pdf_info = middle_json["pdf_info"]
+
+ _process_output(
+ pdf_info, pdf_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_pdf,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, infer_result, process_mode="vlm"
+ )
+
+
+def _process_hybrid(
+ output_dir,
+ pdf_file_names,
+ pdf_bytes_list,
+ h_lang_list,
+ parse_method,
+ inline_formula_enable,
+ backend,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_md,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_dump_orig_pdf,
+ f_dump_content_list,
+ f_make_md_mode,
+ server_url=None,
+ **kwargs,
+):
+ hybrid_doc_analyze = _load_hybrid_analyze_entrypoint(
+ "doc_analyze",
+ f"hybrid-{backend}",
+ )
+ """同步处理hybrid后端逻辑"""
+ if not backend.endswith("client"):
+ server_url = None
+
+ for idx, (pdf_bytes, lang) in enumerate(zip(pdf_bytes_list, h_lang_list)):
+ pdf_file_name = pdf_file_names[idx]
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, f"hybrid_{parse_method}")
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+
+ middle_json, infer_result, _vlm_ocr_enable = hybrid_doc_analyze(
+ pdf_bytes,
+ image_writer=image_writer,
+ backend=backend,
+ parse_method=parse_method,
+ language=lang,
+ inline_formula_enable=inline_formula_enable,
+ server_url=server_url,
+ **kwargs,
+ )
+
+ pdf_info = middle_json["pdf_info"]
+
+ # f_draw_span_bbox = not _vlm_ocr_enable
+ f_draw_span_bbox = False
+
+ _process_output(
+ pdf_info, pdf_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_pdf,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, infer_result, process_mode="vlm"
+ )
+
+
+async def _async_process_hybrid(
+ output_dir,
+ pdf_file_names,
+ pdf_bytes_list,
+ h_lang_list,
+ parse_method,
+ inline_formula_enable,
+ backend,
+ f_draw_layout_bbox,
+ f_draw_span_bbox,
+ f_dump_md,
+ f_dump_middle_json,
+ f_dump_model_output,
+ f_dump_orig_pdf,
+ f_dump_content_list,
+ f_make_md_mode,
+ server_url=None,
+ **kwargs,
+):
+ aio_hybrid_doc_analyze = _load_hybrid_analyze_entrypoint(
+ "aio_doc_analyze",
+ f"hybrid-{backend}",
+ )
+ """异步处理hybrid后端逻辑"""
+ if not backend.endswith("client"):
+ server_url = None
+
+ for idx, (pdf_bytes, lang) in enumerate(zip(pdf_bytes_list, h_lang_list)):
+ pdf_file_name = pdf_file_names[idx]
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, f"hybrid_{parse_method}")
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+
+ middle_json, infer_result, _vlm_ocr_enable = await aio_hybrid_doc_analyze(
+ pdf_bytes,
+ image_writer=image_writer,
+ backend=backend,
+ parse_method=parse_method,
+ language=lang,
+ inline_formula_enable=inline_formula_enable,
+ server_url=server_url,
+ **kwargs,
+ )
+
+ pdf_info = middle_json["pdf_info"]
+
+ # f_draw_span_bbox = not _vlm_ocr_enable
+ f_draw_span_bbox = False
+
+ _process_output(
+ pdf_info, pdf_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_pdf,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, infer_result, process_mode="vlm"
+ )
+
+
+def _process_office_doc(
+ output_dir,
+ pdf_file_names: list[str],
+ pdf_bytes_list: list[bytes],
+ f_dump_md=True,
+ f_dump_middle_json=True,
+ f_dump_model_output=True,
+ f_dump_orig_file=True,
+ f_dump_content_list=True,
+ f_make_md_mode=MakeMode.MM_MD,
+):
+ need_remove_index = []
+ for i, file_bytes in enumerate(pdf_bytes_list):
+ pdf_file_name = pdf_file_names[i]
+ file_suffix = guess_suffix_by_bytes(file_bytes)
+ if file_suffix in office_suffixes:
+
+ need_remove_index.append(i)
+
+ local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, f"office")
+ image_writer, md_writer = FileBasedDataWriter(local_image_dir), FileBasedDataWriter(local_md_dir)
+
+ if file_suffix in docx_suffixes:
+ office_analyze = office_docx_analyze
+ elif file_suffix in pptx_suffixes:
+ office_analyze = office_pptx_analyze
+ elif file_suffix in xlsx_suffixes:
+ office_analyze = office_xlsx_analyze
+ else:
+ raise ValueError(f"Unsupported office suffix: {file_suffix}")
+
+ middle_json, infer_result = office_analyze(
+ file_bytes,
+ image_writer=image_writer,
+ )
+
+ f_draw_layout_bbox = False
+ f_draw_span_bbox = False
+ pdf_info = middle_json["pdf_info"]
+
+ _process_output(
+ pdf_info, file_bytes, pdf_file_name, local_md_dir, local_image_dir,
+ md_writer, f_draw_layout_bbox, f_draw_span_bbox, f_dump_orig_file,
+ f_dump_md, f_dump_content_list, f_dump_middle_json, f_dump_model_output,
+ f_make_md_mode, middle_json, infer_result, process_mode=file_suffix
+ )
+
+ return need_remove_index
+
+
+def do_parse(
+ output_dir,
+ pdf_file_names: list[str],
+ pdf_bytes_list: list[bytes],
+ p_lang_list: list[str],
+ backend="pipeline",
+ parse_method="auto",
+ formula_enable=True,
+ table_enable=True,
+ server_url=None,
+ f_draw_layout_bbox=True,
+ f_draw_span_bbox=True,
+ f_dump_md=True,
+ f_dump_middle_json=True,
+ f_dump_model_output=True,
+ f_dump_orig_pdf=True,
+ f_dump_content_list=True,
+ f_make_md_mode=MakeMode.MM_MD,
+ start_page_id=0,
+ end_page_id=None,
+ **kwargs,
+):
+ need_remove_index = _process_office_doc(
+ output_dir,
+ pdf_file_names=pdf_file_names,
+ pdf_bytes_list=pdf_bytes_list,
+ f_dump_md=f_dump_md,
+ f_dump_middle_json=f_dump_middle_json,
+ f_dump_model_output=f_dump_model_output,
+ f_dump_orig_file=f_dump_orig_pdf,
+ f_dump_content_list=f_dump_content_list,
+ f_make_md_mode=f_make_md_mode,
+ )
+ for index in sorted(need_remove_index, reverse=True):
+ del pdf_bytes_list[index]
+ del pdf_file_names[index]
+ del p_lang_list[index]
+ if not pdf_bytes_list:
+ logger.warning("No valid PDF or image files to process.")
+ return
+
+ # 预处理PDF字节数据
+ pdf_bytes_list = _prepare_pdf_bytes(pdf_bytes_list, start_page_id, end_page_id)
+
+ if backend == "pipeline":
+ _process_pipeline(
+ output_dir, pdf_file_names, pdf_bytes_list, p_lang_list,
+ parse_method, formula_enable, table_enable,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode
+ )
+ else:
+ if backend.startswith("vlm-"):
+ backend = backend[4:]
+
+ if backend == "vllm-async-engine":
+ raise Exception("vlm-vllm-async-engine backend is not supported in sync mode, please use vlm-vllm-engine backend")
+
+ if backend == "auto-engine":
+ backend = get_vlm_engine(inference_engine='auto', is_async=False)
+
+ os.environ['MINERU_VLM_FORMULA_ENABLE'] = str(formula_enable)
+ os.environ['MINERU_VLM_TABLE_ENABLE'] = str(table_enable)
+
+ _process_vlm(
+ output_dir, pdf_file_names, pdf_bytes_list, backend,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode,
+ server_url, **kwargs,
+ )
+ elif backend.startswith("hybrid-"):
+ ensure_backend_dependencies(backend)
+ backend = backend[7:]
+
+ if backend == "vllm-async-engine":
+ raise Exception(
+ "hybrid-vllm-async-engine backend is not supported in sync mode, please use hybrid-vllm-engine backend")
+
+ if backend == "auto-engine":
+ backend = get_vlm_engine(inference_engine='auto', is_async=False)
+
+ os.environ['MINERU_VLM_TABLE_ENABLE'] = str(table_enable)
+ os.environ['MINERU_VLM_FORMULA_ENABLE'] = "true"
+
+ _process_hybrid(
+ output_dir, pdf_file_names, pdf_bytes_list, p_lang_list, parse_method, formula_enable, backend,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode,
+ server_url, **kwargs,
+ )
+
+
+async def aio_do_parse(
+ output_dir,
+ pdf_file_names: list[str],
+ pdf_bytes_list: list[bytes],
+ p_lang_list: list[str],
+ backend="pipeline",
+ parse_method="auto",
+ formula_enable=True,
+ table_enable=True,
+ server_url=None,
+ f_draw_layout_bbox=True,
+ f_draw_span_bbox=True,
+ f_dump_md=True,
+ f_dump_middle_json=True,
+ f_dump_model_output=True,
+ f_dump_orig_pdf=True,
+ f_dump_content_list=True,
+ f_make_md_mode=MakeMode.MM_MD,
+ start_page_id=0,
+ end_page_id=None,
+ **kwargs,
+):
+ need_remove_index = _process_office_doc(
+ output_dir,
+ pdf_file_names=pdf_file_names,
+ pdf_bytes_list=pdf_bytes_list,
+ f_dump_md=f_dump_md,
+ f_dump_middle_json=f_dump_middle_json,
+ f_dump_model_output=f_dump_model_output,
+ f_dump_orig_file=f_dump_orig_pdf,
+ f_dump_content_list=f_dump_content_list,
+ f_make_md_mode=f_make_md_mode,
+ )
+ for index in sorted(need_remove_index, reverse=True):
+ del pdf_bytes_list[index]
+ del pdf_file_names[index]
+ del p_lang_list[index]
+ if not pdf_bytes_list:
+ logger.warning("No valid PDF or image files to process.")
+ return
+
+ # 预处理PDF字节数据
+ pdf_bytes_list = _prepare_pdf_bytes(pdf_bytes_list, start_page_id, end_page_id)
+
+ if backend == "pipeline":
+ # pipeline模式暂不支持异步，使用同步处理方式
+ _process_pipeline(
+ output_dir, pdf_file_names, pdf_bytes_list, p_lang_list,
+ parse_method, formula_enable, table_enable,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode
+ )
+ else:
+ if backend.startswith("vlm-"):
+ backend = backend[4:]
+
+ if backend == "vllm-engine":
+ raise Exception("vlm-vllm-engine backend is not supported in async mode, please use vlm-vllm-async-engine backend")
+
+ if backend == "auto-engine":
+ backend = get_vlm_engine(inference_engine='auto', is_async=True)
+
+ os.environ['MINERU_VLM_FORMULA_ENABLE'] = str(formula_enable)
+ os.environ['MINERU_VLM_TABLE_ENABLE'] = str(table_enable)
+
+ await _async_process_vlm(
+ output_dir, pdf_file_names, pdf_bytes_list, backend,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode,
+ server_url, **kwargs,
+ )
+ elif backend.startswith("hybrid-"):
+ ensure_backend_dependencies(backend)
+ backend = backend[7:]
+
+ if backend == "vllm-engine":
+ raise Exception("hybrid-vllm-engine backend is not supported in async mode, please use hybrid-vllm-async-engine backend")
+
+ if backend == "auto-engine":
+ backend = get_vlm_engine(inference_engine='auto', is_async=True)
+
+ os.environ['MINERU_VLM_TABLE_ENABLE'] = str(table_enable)
+ os.environ['MINERU_VLM_FORMULA_ENABLE'] = "true"
+
+ await _async_process_hybrid(
+ output_dir, pdf_file_names, pdf_bytes_list, p_lang_list, parse_method, formula_enable, backend,
+ f_draw_layout_bbox, f_draw_span_bbox, f_dump_md, f_dump_middle_json,
+ f_dump_model_output, f_dump_orig_pdf, f_dump_content_list, f_make_md_mode,
+ server_url, **kwargs,
+ )
+
+
+if __name__ == "__main__":
+ # pdf_path = "../../demo/pdfs/demo3.pdf"
+ pdf_path = "C:/Users/zhaoxiaomeng/Downloads/4546d0e2-ba60-40a5-a17e-b68555cec741.pdf"
+
+ try:
+ do_parse("./output", [Path(pdf_path).stem], [read_fn(Path(pdf_path))],["ch"],
+ end_page_id=10,
+ backend='vlm-huggingface'
+ # backend = 'pipeline'
+ )
+ except Exception as e:
+ logger.exception(e)
diff --git a/mineru/cli/fast_api.py b/mineru/cli/fast_api.py
new file mode 100644
index 000000000..edcbaae64
--- /dev/null
+++ b/mineru/cli/fast_api.py
@@ -0,0 +1,1620 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
+import mimetypes
+import multiprocessing
+import os
+import shutil
+import sys
+import tempfile
+import threading
+import uuid
+import zipfile
+from contextlib import asynccontextmanager, suppress
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Annotated, Any, Optional
+
+import click
+import uvicorn
+from fastapi import (
+ BackgroundTasks,
+ Depends,
+ FastAPI,
+ File,
+ Form,
+ HTTPException,
+ Request,
+ UploadFile,
+)
+from fastapi.middleware.gzip import GZipMiddleware
+from fastapi.responses import FileResponse, JSONResponse, Response
+from loguru import logger
+
+from base64 import b64encode
+
+from mineru.cli.common import (
+ aio_do_parse,
+ do_parse,
+ image_suffixes,
+ normalize_upload_filename,
+ office_suffixes,
+ pdf_suffixes,
+ normalize_task_stem,
+ read_fn,
+ uniquify_task_stems,
+)
+from mineru.cli.public_http_client_policy import (
+ configure_public_http_client_policy,
+ is_public_bind_host,
+ validate_public_http_client_request,
+ warn_if_public_http_client_policy as _warn_if_public_http_client_policy,
+)
+from mineru.cli.output_paths import resolve_parse_dir
+from mineru.cli.api_protocol import (
+ API_PROTOCOL_VERSION,
+ DEFAULT_MAX_CONCURRENT_REQUESTS,
+ DEFAULT_PROCESSING_WINDOW_SIZE,
+)
+from mineru.cli.vlm_preload import (
+ maybe_preload_vlm_model,
+ split_service_and_model_config,
+)
+from mineru.backend.vlm.vlm_analyze import shutdown_cached_models
+from mineru.utils.cli_parser import arg_parse
+from mineru.utils.check_sys_env import is_mac_environment
+from mineru.utils.config_reader import (
+ get_max_concurrent_requests as read_max_concurrent_requests,
+ get_processing_window_size,
+)
+from mineru.utils.guess_suffix_or_lang import guess_suffix_by_path
+from mineru.utils.pdf_image_tools import shutdown_pdf_render_executor
+from mineru.version import __version__
+
+os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+log_level = os.getenv("MINERU_LOG_LEVEL", "INFO").upper()
+logger.remove()
+logger.add(sys.stderr, level=log_level)
+
+TASK_PENDING = "pending"
+TASK_PROCESSING = "processing"
+TASK_COMPLETED = "completed"
+TASK_FAILED = "failed"
+TASK_TERMINAL_STATES = {TASK_COMPLETED, TASK_FAILED}
+SUPPORTED_UPLOAD_SUFFIXES = pdf_suffixes + image_suffixes + office_suffixes
+DEFAULT_TASK_RETENTION_SECONDS = 24 * 60 * 60
+DEFAULT_TASK_CLEANUP_INTERVAL_SECONDS = 5 * 60
+DEFAULT_OUTPUT_ROOT = "./output"
+ALLOWED_PARSE_METHODS = {"auto", "txt", "ocr"}
+FILE_PARSE_TASK_ID_HEADER = "X-MinerU-Task-Id"
+FILE_PARSE_TASK_STATUS_HEADER = "X-MinerU-Task-Status"
+FILE_PARSE_TASK_STATUS_URL_HEADER = "X-MinerU-Task-Status-Url"
+FILE_PARSE_TASK_RESULT_URL_HEADER = "X-MinerU-Task-Result-Url"
+MINERU_API_PUBLIC_BIND_EXPOSED_ENV = "MINERU_API_PUBLIC_BIND_EXPOSED"
+MINERU_API_ALLOW_PUBLIC_HTTP_CLIENT_ENV = "MINERU_API_ALLOW_PUBLIC_HTTP_CLIENT"
+SWAGGER_UI_FILE_ARRAY_SCHEMA_EXTRA = {
+ # Swagger UI 5 currently fails to render a usable multi-file picker when
+ # FastAPI emits OpenAPI 3.1 byte arrays with contentMediaType.
+ "items": {"type": "string", "format": "binary"}
+}
+
+# 并发控制器
+_request_semaphore: Optional[asyncio.Semaphore] = None
+_configured_max_concurrent_requests = 1
+
+
+def env_flag_enabled(name: str, default: bool = False) -> bool:
+ value = os.getenv(name)
+ if value is None:
+ return default
+ return value.lower() in ("1", "true", "yes", "on")
+
+
+def is_main_multiprocessing_process() -> bool:
+ try:
+ return multiprocessing.current_process().name == "MainProcess"
+ except Exception:
+ return True
+
+
+def install_stdin_shutdown_watcher(server: uvicorn.Server) -> None:
+ if not env_flag_enabled("MINERU_API_SHUTDOWN_ON_STDIN_EOF"):
+ return
+
+ def _watch_stdin_for_eof() -> None:
+ stdin_stream = getattr(sys.stdin, "buffer", sys.stdin)
+ try:
+ stdin_stream.read()
+ except Exception:
+ return
+ server.should_exit = True
+
+ watcher = threading.Thread(
+ target=_watch_stdin_for_eof,
+ name="mineru-api-stdin-shutdown",
+ daemon=True,
+ )
+ watcher.start()
+
+
+@dataclass
+class ParseRequestOptions:
+ files: list[UploadFile]
+ lang_list: list[str]
+ backend: str
+ parse_method: str
+ formula_enable: bool
+ table_enable: bool
+ server_url: Optional[str]
+ return_md: bool
+ return_middle_json: bool
+ return_model_output: bool
+ return_content_list: bool
+ return_images: bool
+ response_format_zip: bool
+ return_original_file: bool
+ start_page_id: int
+ end_page_id: int
+
+
+@dataclass
+class StoredUpload:
+ original_name: str
+ stem: str
+ path: str
+
+
+@dataclass
+class AsyncParseTask:
+ task_id: str
+ status: str
+ backend: str
+ file_names: list[str]
+ created_at: str
+ output_dir: str
+ parse_method: str
+ lang_list: list[str]
+ formula_enable: bool
+ table_enable: bool
+ server_url: Optional[str]
+ return_md: bool
+ return_middle_json: bool
+ return_model_output: bool
+ return_content_list: bool
+ return_images: bool
+ response_format_zip: bool
+ return_original_file: bool
+ start_page_id: int
+ end_page_id: int
+ upload_names: list[str]
+ uploads: list[str]
+ submit_order: int = 0
+ started_at: Optional[str] = None
+ completed_at: Optional[str] = None
+ error: Optional[str] = None
+
+ def to_status_payload(
+ self,
+ request: Request,
+ queued_ahead: int | None = None,
+ ) -> dict[str, Any]:
+ payload = {
+ "task_id": self.task_id,
+ "status": self.status,
+ "backend": self.backend,
+ "file_names": self.file_names,
+ "created_at": self.created_at,
+ "started_at": self.started_at,
+ "completed_at": self.completed_at,
+ "error": self.error,
+ "status_url": str(
+ request.url_for("get_async_task_status", task_id=self.task_id)
+ ),
+ "result_url": str(
+ request.url_for("get_async_task_result", task_id=self.task_id)
+ ),
+ }
+ if queued_ahead is not None:
+ payload["queued_ahead"] = queued_ahead
+ return payload
+
+
+class TaskWaitAbortedError(RuntimeError):
+ """Raised when a synchronous file_parse request cannot keep waiting safely."""
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+ await startup_app_state(app)
+ try:
+ yield
+ finally:
+ await shutdown_app_state(app)
+
+
+def create_app():
+ # By default, the OpenAPI documentation endpoints (openapi_url, docs_url, redoc_url) are enabled.
+ # To disable the FastAPI docs and schema endpoints, set the environment variable MINERU_API_ENABLE_FASTAPI_DOCS=0.
+ enable_docs = env_flag_enabled("MINERU_API_ENABLE_FASTAPI_DOCS", default=True)
+ app = FastAPI(
+ openapi_url="/openapi.json" if enable_docs else None,
+ docs_url="/docs" if enable_docs else None,
+ redoc_url="/redoc" if enable_docs else None,
+ lifespan=lifespan,
+ )
+
+ global _request_semaphore, _configured_max_concurrent_requests
+
+ if is_mac_environment():
+ max_concurrent_requests = 1
+ else:
+ max_concurrent_requests = read_max_concurrent_requests(
+ default=DEFAULT_MAX_CONCURRENT_REQUESTS
+ )
+
+ _configured_max_concurrent_requests = max_concurrent_requests
+ app.state.max_concurrent_requests = max_concurrent_requests
+ _request_semaphore = asyncio.Semaphore(max_concurrent_requests)
+ if is_main_multiprocessing_process():
+ logger.info(f"Request concurrency limited to {max_concurrent_requests}")
+
+ app.add_middleware(GZipMiddleware, minimum_size=1000)
+ app.state.public_bind_exposed = env_flag_enabled(
+ MINERU_API_PUBLIC_BIND_EXPOSED_ENV,
+ default=False,
+ )
+ app.state.allow_public_http_client = env_flag_enabled(
+ MINERU_API_ALLOW_PUBLIC_HTTP_CLIENT_ENV,
+ default=False,
+ )
+ default_service_config, default_model_config = split_service_and_model_config(
+ {
+ "enable_vlm_preload": env_flag_enabled(
+ "MINERU_API_ENABLE_VLM_PRELOAD",
+ default=False,
+ )
+ }
+ )
+ app.state.service_config = default_service_config
+ app.state.config = default_model_config
+ app.state.task_manager = None
+ return app
+
+
+app = create_app()
+
+
+async def startup_app_state(app: FastAPI) -> "AsyncTaskManager":
+ task_manager = AsyncTaskManager(app)
+ await task_manager.start()
+ try:
+ service_config = getattr(app.state, "service_config", {})
+ model_config = getattr(app.state, "config", {})
+ maybe_preload_vlm_model(
+ bool(service_config.get("enable_vlm_preload", False)),
+ model_kwargs=model_config,
+ )
+ except Exception:
+ await task_manager.shutdown()
+ app.state.task_manager = None
+ raise
+
+ app.state.task_manager = task_manager
+ return task_manager
+
+
+async def shutdown_app_state(app: FastAPI) -> None:
+ current_task_manager = getattr(app.state, "task_manager", None)
+ if current_task_manager is not None:
+ await current_task_manager.shutdown()
+ app.state.task_manager = None
+ shutdown_runtime_resources()
+
+
+def shutdown_runtime_resources() -> None:
+ try:
+ shutdown_cached_models()
+ except Exception as exc:
+ logger.warning(f"Failed to shutdown cached VLM models: {exc}")
+
+ try:
+ shutdown_pdf_render_executor()
+ except Exception as exc:
+ logger.warning(f"Failed to shutdown PDF render executor: {exc}")
+
+
+def utc_now_iso() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def get_int_env(name: str, default: int, minimum: int = 0) -> int:
+ try:
+ value = int(os.getenv(name, str(default)))
+ except ValueError:
+ return default
+ if value < minimum:
+ return default
+ return value
+
+
+def get_max_concurrent_requests() -> int:
+ return _configured_max_concurrent_requests
+
+
+def get_task_retention_seconds() -> int:
+ return get_int_env(
+ "MINERU_API_TASK_RETENTION_SECONDS",
+ DEFAULT_TASK_RETENTION_SECONDS,
+ minimum=0,
+ )
+
+
+def get_task_cleanup_interval_seconds() -> int:
+ return get_int_env(
+ "MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS",
+ DEFAULT_TASK_CLEANUP_INTERVAL_SECONDS,
+ minimum=1,
+ )
+
+
+def get_output_root() -> Path:
+ root = Path(os.getenv("MINERU_API_OUTPUT_ROOT", DEFAULT_OUTPUT_ROOT)).expanduser()
+ root.mkdir(parents=True, exist_ok=True)
+ return root.resolve()
+
+
+def warn_if_public_http_client_policy(host: str, allow_public_http_client: bool) -> None:
+ _warn_if_public_http_client_policy(
+ service_name="API",
+ host=host,
+ allow_public_http_client=allow_public_http_client,
+ )
+
+
+def validate_parse_method(parse_method: str) -> str:
+ if parse_method not in ALLOWED_PARSE_METHODS:
+ raise HTTPException(
+ status_code=400,
+ detail=(
+ "Invalid parse_method. Allowed values: "
+ + ", ".join(sorted(ALLOWED_PARSE_METHODS))
+ ),
+ )
+ return parse_method
+
+
+def cleanup_file(file_path: str) -> None:
+ """清理临时文件或目录"""
+ try:
+ if os.path.exists(file_path):
+ if os.path.isfile(file_path):
+ os.remove(file_path)
+ elif os.path.isdir(file_path):
+ shutil.rmtree(file_path)
+ except Exception as e:
+ logger.warning(f"fail clean file {file_path}: {e}")
+
+
+def build_upload_destination(upload_dir: str, filename: str) -> Path:
+ destination = Path(upload_dir) / filename
+ if not destination.exists():
+ return destination
+
+ base_name = Path(filename).stem
+ suffix = Path(filename).suffix
+ index = 2
+ while True:
+ candidate = Path(upload_dir) / f"{base_name}__upload_{index}{suffix}"
+ if not candidate.exists():
+ return candidate
+ index += 1
+
+
+def encode_image(image_path: str) -> str:
+ """Encode image using base64"""
+ with open(image_path, "rb") as f:
+ return b64encode(f.read()).decode()
+
+
+def get_images_dir_image_paths(images_dir: str) -> list[str]:
+ """Return all supported image files directly under images_dir."""
+ if not os.path.isdir(images_dir):
+ return []
+
+ return sorted(
+ str(path)
+ for path in Path(images_dir).iterdir()
+ if path.is_file() and path.suffix.lstrip(".").lower() in image_suffixes
+ )
+
+
+def get_image_mime_type(image_path: str) -> str:
+ mime_type, _ = mimetypes.guess_type(image_path)
+ if mime_type:
+ return mime_type
+ return "image/jpeg"
+
+
+def get_infer_result(
+ file_suffix_identifier: str, pdf_name: str, parse_dir: str
+) -> Optional[str]:
+ """从结果文件中读取推理结果"""
+ result_file_path = os.path.join(parse_dir, f"{pdf_name}{file_suffix_identifier}")
+ if os.path.exists(result_file_path):
+ with open(result_file_path, "r", encoding="utf-8") as fp:
+ return fp.read()
+ return None
+
+
+def normalize_lang_list(lang_list: list[str], file_count: int) -> list[str]:
+ if len(lang_list) == file_count:
+ return lang_list
+ base_lang = lang_list[0] if lang_list else "ch"
+ return [base_lang] * file_count
+
+
+def get_parse_dir(output_dir: str, pdf_name: str, backend: str, parse_method: str) -> str:
+ return str(
+ resolve_parse_dir(
+ output_dir,
+ pdf_name,
+ backend,
+ parse_method,
+ allow_office_fallback=True,
+ )
+ )
+
+
+def is_task_terminal(status: str) -> bool:
+ return status in TASK_TERMINAL_STATES
+
+
+def build_result_dict(
+ output_dir: str,
+ pdf_file_names: list[str],
+ backend: str,
+ parse_method: str,
+ return_md: bool,
+ return_middle_json: bool,
+ return_model_output: bool,
+ return_content_list: bool,
+ return_images: bool,
+) -> dict[str, dict[str, Any]]:
+ result_dict: dict[str, dict[str, Any]] = {}
+ for pdf_name in pdf_file_names:
+ result_dict[pdf_name] = {}
+ data = result_dict[pdf_name]
+
+ try:
+ parse_dir = get_parse_dir(output_dir, pdf_name, backend, parse_method)
+ except ValueError:
+ logger.warning(f"Unknown backend type: {backend}, skipping {pdf_name}")
+ continue
+
+ if not os.path.exists(parse_dir):
+ continue
+
+ if return_md:
+ data["md_content"] = get_infer_result(".md", pdf_name, parse_dir)
+ if return_middle_json:
+ data["middle_json"] = get_infer_result("_middle.json", pdf_name, parse_dir)
+ if return_model_output:
+ data["model_output"] = get_infer_result("_model.json", pdf_name, parse_dir)
+ if return_content_list:
+ data["content_list"] = get_infer_result(
+ "_content_list.json", pdf_name, parse_dir
+ )
+ if return_images:
+ images_dir = os.path.join(parse_dir, "images")
+ image_paths = get_images_dir_image_paths(images_dir)
+ data["images"] = {
+ os.path.basename(
+ image_path
+ ): f"data:{get_image_mime_type(image_path)};base64,{encode_image(image_path)}"
+ for image_path in image_paths
+ }
+ return result_dict
+
+
+def build_zip_arcname(
+ pdf_name: str,
+ parse_dir: str,
+ relative_path: str,
+) -> str:
+ return os.path.join(pdf_name, os.path.basename(parse_dir), relative_path)
+
+
+def create_result_zip(
+ output_dir: str,
+ pdf_file_names: list[str],
+ backend: str,
+ parse_method: str,
+ return_md: bool,
+ return_middle_json: bool,
+ return_model_output: bool,
+ return_content_list: bool,
+ return_images: bool,
+ return_original_file: bool,
+) -> str:
+ zip_fd, zip_path = tempfile.mkstemp(suffix=".zip", prefix="mineru_results_")
+ os.close(zip_fd)
+
+ with zipfile.ZipFile(zip_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
+ for pdf_name in pdf_file_names:
+ try:
+ parse_dir = get_parse_dir(output_dir, pdf_name, backend, parse_method)
+ except ValueError:
+ logger.warning(f"Unknown backend type: {backend}, skipping {pdf_name}")
+ continue
+
+ if not os.path.exists(parse_dir):
+ continue
+
+ if return_md:
+ path = os.path.join(parse_dir, f"{pdf_name}.md")
+ if os.path.exists(path):
+ zf.write(
+ path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ f"{pdf_name}.md",
+ ),
+ )
+
+ if return_middle_json:
+ path = os.path.join(parse_dir, f"{pdf_name}_middle.json")
+ if os.path.exists(path):
+ zf.write(
+ path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ f"{pdf_name}_middle.json",
+ ),
+ )
+
+ if return_model_output:
+ path = os.path.join(parse_dir, f"{pdf_name}_model.json")
+ if os.path.exists(path):
+ zf.write(
+ path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ f"{pdf_name}_model.json",
+ ),
+ )
+
+ if return_content_list:
+ path = os.path.join(parse_dir, f"{pdf_name}_content_list.json")
+ if os.path.exists(path):
+ zf.write(
+ path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ f"{pdf_name}_content_list.json",
+ ),
+ )
+
+ path = os.path.join(parse_dir, f"{pdf_name}_content_list_v2.json")
+ if os.path.exists(path):
+ zf.write(
+ path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ f"{pdf_name}_content_list_v2.json",
+ ),
+ )
+
+ if return_images:
+ images_dir = os.path.join(parse_dir, "images")
+ image_paths = get_images_dir_image_paths(images_dir)
+ for image_path in image_paths:
+ zf.write(
+ image_path,
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ os.path.join("images", os.path.basename(image_path)),
+ ),
+ )
+
+ if return_original_file:
+ origin_pattern = f"{pdf_name}_origin."
+ for path in sorted(Path(parse_dir).iterdir()):
+ if not path.is_file():
+ continue
+ if not path.name.startswith(origin_pattern):
+ continue
+ zf.write(
+ str(path),
+ arcname=build_zip_arcname(
+ pdf_name,
+ parse_dir,
+ path.name,
+ ),
+ )
+ return zip_path
+
+
+def build_result_response(
+ background_tasks: BackgroundTasks,
+ status_code: int,
+ output_dir: str,
+ pdf_file_names: list[str],
+ backend: str,
+ parse_method: str,
+ return_md: bool,
+ return_middle_json: bool,
+ return_model_output: bool,
+ return_content_list: bool,
+ return_images: bool,
+ response_format_zip: bool,
+ return_original_file: bool,
+ zip_filename: str = "results.zip",
+) -> Response:
+ if response_format_zip:
+ zip_path = create_result_zip(
+ output_dir=output_dir,
+ pdf_file_names=pdf_file_names,
+ backend=backend,
+ parse_method=parse_method,
+ return_md=return_md,
+ return_middle_json=return_middle_json,
+ return_model_output=return_model_output,
+ return_content_list=return_content_list,
+ return_images=return_images,
+ return_original_file=return_original_file,
+ )
+ background_tasks.add_task(cleanup_file, zip_path)
+ return FileResponse(
+ path=zip_path,
+ media_type="application/zip",
+ filename=zip_filename,
+ status_code=status_code,
+ )
+
+ result_dict = build_result_dict(
+ output_dir=output_dir,
+ pdf_file_names=pdf_file_names,
+ backend=backend,
+ parse_method=parse_method,
+ return_md=return_md,
+ return_middle_json=return_middle_json,
+ return_model_output=return_model_output,
+ return_content_list=return_content_list,
+ return_images=return_images,
+ )
+ return JSONResponse(
+ status_code=status_code,
+ content={
+ "backend": backend,
+ "version": __version__,
+ "results": result_dict,
+ },
+ )
+
+
+def build_task_submission_response(
+ task: AsyncParseTask,
+ request: Request,
+ task_manager: "AsyncTaskManager",
+) -> JSONResponse:
+ payload = task_manager.build_status_payload(task, request)
+ payload["message"] = "Task submitted successfully"
+ return JSONResponse(status_code=202, content=payload)
+
+
+def build_sync_file_parse_response(
+ background_tasks: BackgroundTasks,
+ task: AsyncParseTask,
+ request: Request,
+) -> Response:
+ task_payload = task.to_status_payload(request)
+ if task.response_format_zip:
+ response = build_result_response(
+ background_tasks=background_tasks,
+ status_code=200,
+ output_dir=task.output_dir,
+ pdf_file_names=task.file_names,
+ backend=task.backend,
+ parse_method=task.parse_method,
+ return_md=task.return_md,
+ return_middle_json=task.return_middle_json,
+ return_model_output=task.return_model_output,
+ return_content_list=task.return_content_list,
+ return_images=task.return_images,
+ response_format_zip=task.response_format_zip,
+ return_original_file=task.return_original_file,
+ zip_filename=f"{task.task_id}.zip",
+ )
+ response.headers[FILE_PARSE_TASK_ID_HEADER] = task.task_id
+ response.headers[FILE_PARSE_TASK_STATUS_HEADER] = task.status
+ response.headers[FILE_PARSE_TASK_STATUS_URL_HEADER] = task_payload["status_url"]
+ response.headers[FILE_PARSE_TASK_RESULT_URL_HEADER] = task_payload["result_url"]
+ return response
+
+ result_dict = build_result_dict(
+ output_dir=task.output_dir,
+ pdf_file_names=task.file_names,
+ backend=task.backend,
+ parse_method=task.parse_method,
+ return_md=task.return_md,
+ return_middle_json=task.return_middle_json,
+ return_model_output=task.return_model_output,
+ return_content_list=task.return_content_list,
+ return_images=task.return_images,
+ )
+ return JSONResponse(
+ status_code=200,
+ content={
+ **task_payload,
+ "backend": task.backend,
+ "version": __version__,
+ "results": result_dict,
+ },
+ )
+
+
+async def parse_request_form(
+ request: Request,
+ files: Annotated[
+ list[UploadFile],
+ File(
+ description="Upload PDF, image, DOCX, PPTX, or XLSX files for parsing",
+ json_schema_extra=SWAGGER_UI_FILE_ARRAY_SCHEMA_EXTRA,
+ ),
+ ],
+ lang_list: Annotated[
+ list[str],
+ Form(
+ description="""(Adapted only for pipeline and hybrid backend)Input the languages in the pdf to improve OCR accuracy.Options:
+- ch: Chinese, English, Chinese Traditional.
+- ch_lite: Chinese, English, Chinese Traditional, Japanese.
+- ch_server: Chinese, English, Chinese Traditional, Japanese.
+- en: English.
+- korean: Korean, English.
+- japan: Chinese, English, Chinese Traditional, Japanese.
+- chinese_cht: Chinese, English, Chinese Traditional, Japanese.
+- ta: Tamil, English.
+- te: Telugu, English.
+- ka: Kannada.
+- th: Thai, English.
+- el: Greek, English.
+- latin: French, German, Afrikaans, Italian, Spanish, Bosnian, Portuguese, Czech, Welsh, Danish, Estonian, Irish, Croatian, Uzbek, Hungarian, Serbian (Latin), Indonesian, Occitan, Icelandic, Lithuanian, Maori, Malay, Dutch, Norwegian, Polish, Slovak, Slovenian, Albanian, Swedish, Swahili, Tagalog, Turkish, Latin, Azerbaijani, Kurdish, Latvian, Maltese, Pali, Romanian, Vietnamese, Finnish, Basque, Galician, Luxembourgish, Romansh, Catalan, Quechua.
+- arabic: Arabic, Persian, Uyghur, Urdu, Pashto, Kurdish, Sindhi, Balochi, English.
+- east_slavic: Russian, Belarusian, Ukrainian, English.
+- cyrillic: Russian, Belarusian, Ukrainian, Serbian (Cyrillic), Bulgarian, Mongolian, Abkhazian, Adyghe, Kabardian, Avar, Dargin, Ingush, Chechen, Lak, Lezgin, Tabasaran, Kazakh, Kyrgyz, Tajik, Macedonian, Tatar, Chuvash, Bashkir, Malian, Moldovan, Udmurt, Komi, Ossetian, Buryat, Kalmyk, Tuvan, Sakha, Karakalpak, English.
+- devanagari: Hindi, Marathi, Nepali, Bihari, Maithili, Angika, Bhojpuri, Magahi, Santali, Newari, Konkani, Sanskrit, Haryanvi, English.
+""",
+ ),
+ ] = ["ch"],
+ backend: Annotated[
+ str,
+ Form(
+ description="""The backend for parsing:
+- pipeline: More general, supports multiple languages, hallucination-free.
+- vlm-auto-engine: High accuracy via local computing power, supports Chinese and English documents only.
+- vlm-http-client: High accuracy via remote computing power(client suitable for openai-compatible servers), supports Chinese and English documents only.
+- hybrid-auto-engine: Next-generation high accuracy solution via local computing power, supports multiple languages.
+- hybrid-http-client: High accuracy via remote computing power but requires a little local computing power(client suitable for openai-compatible servers), supports multiple languages.""",
+ ),
+ ] = "hybrid-auto-engine",
+ parse_method: Annotated[
+ str,
+ Form(
+ description="""(Adapted only for pipeline and hybrid backend)The method for parsing PDF:
+- auto: Automatically determine the method based on the file type
+- txt: Use text extraction method
+- ocr: Use OCR method for image-based PDFs
+""",
+ ),
+ ] = "auto",
+ formula_enable: Annotated[
+ bool,
+ Form(description="Enable formula parsing."),
+ ] = True,
+ table_enable: Annotated[
+ bool,
+ Form(description="Enable table parsing."),
+ ] = True,
+ server_url: Annotated[
+ Optional[str],
+ Form(
+ description="(Adapted only for -http-client backend)openai compatible server url, e.g., http://127.0.0.1:30000",
+ ),
+ ] = None,
+ return_md: Annotated[
+ bool,
+ Form(description="Return markdown content in response"),
+ ] = True,
+ return_middle_json: Annotated[
+ bool,
+ Form(description="Return middle JSON in response"),
+ ] = False,
+ return_model_output: Annotated[
+ bool,
+ Form(description="Return model output JSON in response"),
+ ] = False,
+ return_content_list: Annotated[
+ bool,
+ Form(description="Return content list JSON in response"),
+ ] = False,
+ return_images: Annotated[
+ bool,
+ Form(description="Return extracted images in response"),
+ ] = False,
+ response_format_zip: Annotated[
+ bool,
+ Form(description="Return results as a ZIP file instead of JSON"),
+ ] = False,
+ return_original_file: Annotated[
+ bool,
+ Form(
+ description=(
+ "Include the processed original input file in the ZIP result; "
+ "ignored unless response_format_zip=true"
+ ),
+ ),
+ ] = False,
+ start_page_id: Annotated[
+ int,
+ Form(description="The starting page for PDF parsing, beginning from 0"),
+ ] = 0,
+ end_page_id: Annotated[
+ int,
+ Form(description="The ending page for PDF parsing, beginning from 0"),
+ ] = 99999,
+) -> ParseRequestOptions:
+ validate_public_http_client_request(
+ public_bind_exposed=bool(
+ getattr(request.app.state, "public_bind_exposed", False)
+ ),
+ allow_public_http_client=bool(
+ getattr(request.app.state, "allow_public_http_client", False)
+ ),
+ backend=backend,
+ server_url=server_url,
+ )
+ effective_return_original_file = return_original_file and response_format_zip
+ return ParseRequestOptions(
+ files=files,
+ lang_list=lang_list,
+ backend=backend,
+ parse_method=validate_parse_method(parse_method),
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=server_url,
+ return_md=return_md,
+ return_middle_json=return_middle_json,
+ return_model_output=return_model_output,
+ return_content_list=return_content_list,
+ return_images=return_images,
+ response_format_zip=response_format_zip,
+ return_original_file=effective_return_original_file,
+ start_page_id=start_page_id,
+ end_page_id=end_page_id,
+ )
+
+
+async def save_upload_files(upload_dir: str, files: list[UploadFile]) -> list[StoredUpload]:
+ os.makedirs(upload_dir, exist_ok=True)
+ uploads: list[StoredUpload] = []
+
+ for upload in files:
+ original_name = upload.filename or f"upload-{uuid.uuid4()}"
+ filename = normalize_upload_filename(original_name)
+ normalized_stem = normalize_task_stem(Path(filename).stem)
+ destination = build_upload_destination(upload_dir, filename)
+ try:
+ with open(destination, "wb") as handle:
+ while True:
+ chunk = await upload.read(1 << 20)
+ if not chunk:
+ break
+ handle.write(chunk)
+
+ file_suffix = guess_suffix_by_path(destination)
+ if file_suffix not in SUPPORTED_UPLOAD_SUFFIXES:
+ cleanup_file(str(destination))
+ raise HTTPException(
+ status_code=400,
+ detail=f"Unsupported file type: {file_suffix}",
+ )
+
+ uploads.append(
+ StoredUpload(
+ original_name=original_name,
+ stem=normalized_stem,
+ path=str(destination),
+ )
+ )
+ except Exception:
+ cleanup_file(str(destination))
+ raise
+ finally:
+ await upload.close()
+
+ normalized_stems, renamed_stems = uniquify_task_stems(
+ [upload.stem for upload in uploads]
+ )
+ if renamed_stems:
+ rename_details = ", ".join(
+ f"{Path(upload.original_name).name} -> {effective_stem}"
+ for upload, effective_stem in zip(uploads, normalized_stems)
+ if upload.stem != effective_stem
+ )
+ logger.warning(
+ f"Normalized duplicate upload stems within request: {rename_details}"
+ )
+ uploads = [
+ StoredUpload(
+ original_name=upload.original_name,
+ stem=effective_stem,
+ path=upload.path,
+ )
+ for upload, effective_stem in zip(uploads, normalized_stems)
+ ]
+ return uploads
+
+
+def load_parse_inputs(uploads: list[StoredUpload]) -> tuple[list[str], list[bytes]]:
+ pdf_file_names = []
+ pdf_bytes_list = []
+
+ for upload in uploads:
+ try:
+ pdf_bytes = read_fn(Path(upload.path))
+ except Exception as exc:
+ raise RuntimeError(f"Failed to load file {upload.original_name}: {exc}") from exc
+ pdf_file_names.append(upload.stem)
+ pdf_bytes_list.append(pdf_bytes)
+ return pdf_file_names, pdf_bytes_list
+
+
+async def run_parse_job(
+ output_dir: str,
+ uploads: list[StoredUpload],
+ request_options: ParseRequestOptions | AsyncParseTask,
+ config: dict[str, Any],
+) -> list[str]:
+ pdf_file_names, pdf_bytes_list = await asyncio.to_thread(load_parse_inputs, uploads)
+ actual_lang_list = normalize_lang_list(request_options.lang_list, len(pdf_file_names))
+ response_file_names = list(pdf_file_names)
+
+ parse_kwargs = dict(
+ output_dir=output_dir,
+ pdf_file_names=list(pdf_file_names),
+ pdf_bytes_list=list(pdf_bytes_list),
+ p_lang_list=list(actual_lang_list),
+ backend=request_options.backend,
+ parse_method=request_options.parse_method,
+ formula_enable=request_options.formula_enable,
+ table_enable=request_options.table_enable,
+ server_url=request_options.server_url,
+ f_draw_layout_bbox=False,
+ f_draw_span_bbox=False,
+ f_dump_md=request_options.return_md,
+ f_dump_middle_json=request_options.return_middle_json,
+ f_dump_model_output=request_options.return_model_output,
+ f_dump_orig_pdf=(
+ request_options.return_original_file and request_options.response_format_zip
+ ),
+ f_dump_content_list=request_options.return_content_list,
+ start_page_id=request_options.start_page_id,
+ end_page_id=request_options.end_page_id,
+ **config,
+ )
+
+ if request_options.backend == "pipeline":
+ await asyncio.to_thread(do_parse, **parse_kwargs)
+ else:
+ await aio_do_parse(**parse_kwargs)
+ return response_file_names
+
+
+def create_task_output_dir(task_id: str) -> str:
+ output_root = get_output_root()
+ task_output_dir = output_root / task_id
+ task_output_dir.mkdir(parents=True, exist_ok=True)
+ return str(task_output_dir)
+
+
+async def create_async_parse_task(
+ request_options: ParseRequestOptions,
+) -> AsyncParseTask:
+ task_id = str(uuid.uuid4())
+ task_output_dir = create_task_output_dir(task_id)
+ uploads_dir = os.path.join(task_output_dir, "uploads")
+ task_manager = get_task_manager()
+
+ try:
+ uploads = await save_upload_files(uploads_dir, request_options.files)
+ request_options.files.clear()
+ file_names = [upload.stem for upload in uploads]
+ task = AsyncParseTask(
+ task_id=task_id,
+ status=TASK_PENDING,
+ backend=request_options.backend,
+ file_names=file_names,
+ created_at=utc_now_iso(),
+ output_dir=task_output_dir,
+ parse_method=request_options.parse_method,
+ lang_list=request_options.lang_list,
+ formula_enable=request_options.formula_enable,
+ table_enable=request_options.table_enable,
+ server_url=request_options.server_url,
+ return_md=request_options.return_md,
+ return_middle_json=request_options.return_middle_json,
+ return_model_output=request_options.return_model_output,
+ return_content_list=request_options.return_content_list,
+ return_images=request_options.return_images,
+ response_format_zip=request_options.response_format_zip,
+ return_original_file=request_options.return_original_file,
+ start_page_id=request_options.start_page_id,
+ end_page_id=request_options.end_page_id,
+ upload_names=[upload.original_name for upload in uploads],
+ uploads=[upload.path for upload in uploads],
+ )
+ await task_manager.submit(task)
+ return task
+ except HTTPException:
+ cleanup_file(task_output_dir)
+ raise
+ except Exception:
+ cleanup_file(task_output_dir)
+ raise
+
+
+class AsyncTaskManager:
+ def __init__(self, fastapi_app: FastAPI):
+ self.app = fastapi_app
+ self.tasks: dict[str, AsyncParseTask] = {}
+ self.task_events: dict[str, asyncio.Event] = {}
+ self.queue: asyncio.Queue[str] = asyncio.Queue()
+ self.dispatcher_task: Optional[asyncio.Task[Any]] = None
+ self.cleanup_task: Optional[asyncio.Task[Any]] = None
+ self.active_tasks: set[asyncio.Task[Any]] = set()
+ self.last_worker_error: Optional[str] = None
+ self.is_shutting_down = False
+ self.task_retention_seconds = get_task_retention_seconds()
+ self.task_cleanup_interval_seconds = get_task_cleanup_interval_seconds()
+ self.manager_wakeup = asyncio.Event()
+ self._next_submit_order = 1
+
+ async def start(self) -> None:
+ self.is_shutting_down = False
+ self.last_worker_error = None
+ self.manager_wakeup = asyncio.Event()
+ if self.dispatcher_task is None or self.dispatcher_task.done():
+ self.dispatcher_task = asyncio.create_task(
+ self._dispatcher_loop(), name="mineru-fastapi-task-dispatcher"
+ )
+ if (
+ self.task_retention_seconds > 0
+ and (self.cleanup_task is None or self.cleanup_task.done())
+ ):
+ self.cleanup_task = asyncio.create_task(
+ self._cleanup_loop(), name="mineru-fastapi-task-cleanup"
+ )
+
+ async def shutdown(self) -> None:
+ self.is_shutting_down = True
+ self._wake_waiters()
+ if self.dispatcher_task is not None:
+ self.dispatcher_task.cancel()
+ with suppress(asyncio.CancelledError):
+ await self.dispatcher_task
+ self.dispatcher_task = None
+ if self.cleanup_task is not None:
+ self.cleanup_task.cancel()
+ with suppress(asyncio.CancelledError):
+ await self.cleanup_task
+ self.cleanup_task = None
+
+ pending = list(self.active_tasks)
+ for processor in pending:
+ processor.cancel()
+ if pending:
+ await asyncio.gather(*pending, return_exceptions=True)
+ self.active_tasks.clear()
+
+ async def submit(self, task: AsyncParseTask) -> None:
+ task.submit_order = self._next_submit_order
+ self._next_submit_order += 1
+ self.tasks[task.task_id] = task
+ self.task_events[task.task_id] = asyncio.Event()
+ await self.queue.put(task.task_id)
+
+ def get(self, task_id: str) -> Optional[AsyncParseTask]:
+ return self.tasks.get(task_id)
+
+ def get_queued_ahead(self, task_id: str) -> int | None:
+ task = self.tasks.get(task_id)
+ if task is None:
+ return None
+ if task.status != TASK_PENDING:
+ return 0
+
+ return sum(
+ 1
+ for other_task in self.tasks.values()
+ if (
+ other_task.task_id != task_id
+ and other_task.status == TASK_PENDING
+ and 0 < other_task.submit_order < task.submit_order
+ )
+ )
+
+ def build_status_payload(
+ self,
+ task: AsyncParseTask,
+ request: Request,
+ ) -> dict[str, Any]:
+ return task.to_status_payload(
+ request,
+ queued_ahead=self.get_queued_ahead(task.task_id),
+ )
+
+ async def wait_for_terminal_state(self, task_id: str) -> AsyncParseTask:
+ task = self.tasks.get(task_id)
+ if task is None:
+ raise TaskWaitAbortedError("Task not found")
+ if is_task_terminal(task.status):
+ return task
+
+ task_event = self.task_events.get(task_id)
+ if task_event is None:
+ raise TaskWaitAbortedError("Task wait handle is unavailable")
+
+ event_wait_task = asyncio.create_task(task_event.wait())
+ manager_wait_task = asyncio.create_task(self.manager_wakeup.wait())
+ done: set[asyncio.Task[Any]] = set()
+ pending: set[asyncio.Task[Any]] = set()
+ try:
+ done, pending = await asyncio.wait(
+ {event_wait_task, manager_wait_task},
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+ finally:
+ for waiter in pending:
+ waiter.cancel()
+ if pending:
+ await asyncio.gather(*pending, return_exceptions=True)
+ for waiter in done:
+ with suppress(asyncio.CancelledError):
+ waiter.result()
+
+ task = self.tasks.get(task_id)
+ if task is None:
+ if self.is_shutting_down:
+ raise TaskWaitAbortedError("Task manager is shutting down")
+ raise TaskWaitAbortedError("Task was removed before completion")
+ if is_task_terminal(task.status):
+ return task
+ if self.is_shutting_down:
+ raise TaskWaitAbortedError("Task manager is shutting down")
+ raise TaskWaitAbortedError(
+ self.last_worker_error or "Task manager became unavailable while waiting"
+ )
+
+ def get_stats(self) -> dict[str, int]:
+ stats = {
+ TASK_PENDING: 0,
+ TASK_PROCESSING: 0,
+ TASK_COMPLETED: 0,
+ TASK_FAILED: 0,
+ }
+ for task in self.tasks.values():
+ if task.status in stats:
+ stats[task.status] += 1
+ return stats
+
+ def is_healthy(self) -> bool:
+ if self.dispatcher_task is None:
+ return False
+ if self.dispatcher_task.done() and not self.is_shutting_down:
+ return False
+ if self.task_retention_seconds > 0 and self.cleanup_task is None:
+ return False
+ if (
+ self.task_retention_seconds > 0
+ and self.cleanup_task is not None
+ and self.cleanup_task.done()
+ and not self.is_shutting_down
+ ):
+ return False
+ return self.last_worker_error is None
+
+ def _wake_waiters(self) -> None:
+ self.manager_wakeup.set()
+ for task_event in self.task_events.values():
+ task_event.set()
+
+ def _signal_task_event(self, task_id: str) -> None:
+ task_event = self.task_events.get(task_id)
+ if task_event is not None:
+ task_event.set()
+
+ async def _dispatcher_loop(self) -> None:
+ try:
+ while True:
+ task_id = await self.queue.get()
+ processor = asyncio.create_task(
+ self._process_task(task_id),
+ name=f"mineru-fastapi-task-{task_id}",
+ )
+ self.active_tasks.add(processor)
+ processor.add_done_callback(self._on_processor_done)
+ self.queue.task_done()
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ self.last_worker_error = str(exc)
+ self._wake_waiters()
+ logger.exception("Async task dispatcher crashed")
+ raise
+
+ async def _cleanup_loop(self) -> None:
+ try:
+ while True:
+ await asyncio.sleep(self.task_cleanup_interval_seconds)
+ self.cleanup_expired_tasks()
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ self.last_worker_error = str(exc)
+ logger.exception("Async task cleanup loop crashed")
+ raise
+
+ def _on_processor_done(self, processor: asyncio.Task[Any]) -> None:
+ self.active_tasks.discard(processor)
+ if processor.cancelled():
+ return
+ exception = processor.exception()
+ if exception is not None:
+ logger.error(f"Async task processor crashed: {exception}")
+ self.last_worker_error = str(exception)
+
+ async def _process_task(self, task_id: str) -> None:
+ task = self.tasks.get(task_id)
+ if task is None:
+ return
+
+ try:
+ if _request_semaphore is not None:
+ async with _request_semaphore:
+ await self._run_task(task)
+ else:
+ await self._run_task(task)
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ task.status = TASK_FAILED
+ task.error = str(exc)
+ task.completed_at = utc_now_iso()
+ self._signal_task_event(task_id)
+ logger.exception(f"Async task failed: {task_id}")
+
+ async def _run_task(self, task: AsyncParseTask) -> None:
+ task.status = TASK_PROCESSING
+ task.started_at = utc_now_iso()
+ task.error = None
+
+ uploads = [
+ StoredUpload(
+ original_name=upload_name,
+ stem=file_name,
+ path=upload_path,
+ )
+ for upload_name, file_name, upload_path in zip(
+ task.upload_names,
+ task.file_names,
+ task.uploads,
+ )
+ ]
+ config = getattr(self.app.state, "config", {})
+ await run_parse_job(
+ output_dir=task.output_dir,
+ uploads=uploads,
+ request_options=task,
+ config=config,
+ )
+ task.status = TASK_COMPLETED
+ task.completed_at = utc_now_iso()
+ self._signal_task_event(task.task_id)
+
+ def cleanup_expired_tasks(self) -> int:
+ if self.task_retention_seconds <= 0:
+ return 0
+
+ now = datetime.now(timezone.utc)
+ expired_task_ids = [
+ task_id
+ for task_id, task in self.tasks.items()
+ if self._is_task_expired(task, now)
+ ]
+
+ for task_id in expired_task_ids:
+ task = self.tasks.pop(task_id, None)
+ if task is None:
+ continue
+ task_event = self.task_events.pop(task_id, None)
+ if task_event is not None:
+ task_event.set()
+ cleanup_file(task.output_dir)
+ logger.info(f"Cleaned expired async task: {task_id}")
+ return len(expired_task_ids)
+
+ def _is_task_expired(self, task: AsyncParseTask, now: datetime) -> bool:
+ if task.status not in (TASK_COMPLETED, TASK_FAILED):
+ return False
+ if not task.completed_at:
+ return False
+ try:
+ completed_at = datetime.fromisoformat(task.completed_at)
+ except ValueError:
+ logger.warning(f"Invalid completed_at for task {task.task_id}: {task.completed_at}")
+ return False
+ if completed_at.tzinfo is None:
+ completed_at = completed_at.replace(tzinfo=timezone.utc)
+ return (now - completed_at).total_seconds() >= self.task_retention_seconds
+
+
+def get_task_manager() -> AsyncTaskManager:
+ task_manager = getattr(app.state, "task_manager", None)
+ if task_manager is None:
+ raise HTTPException(status_code=503, detail="Task manager is not initialized")
+ return task_manager
+
+
+@app.post(
+ path="/file_parse",
+ status_code=200,
+ summary="Synchronously parse uploaded files",
+ description=(
+ "Submit a parsing task to the shared async task manager, wait for it to "
+ "finish, and return the final parsing result in the same response."
+ ),
+)
+async def parse_pdf(
+ http_request: Request,
+ background_tasks: BackgroundTasks,
+ request_options: Annotated[
+ ParseRequestOptions, Depends(parse_request_form)
+ ],
+):
+ task = await create_async_parse_task(request_options)
+ request_options = None
+ task_manager = get_task_manager()
+
+ try:
+ task = await task_manager.wait_for_terminal_state(task.task_id)
+ except TaskWaitAbortedError as exc:
+ return JSONResponse(
+ status_code=503,
+ content={
+ **task.to_status_payload(http_request),
+ "message": "Task manager became unavailable while waiting for result",
+ "error": str(exc),
+ },
+ )
+
+ if task.status == TASK_FAILED:
+ return JSONResponse(
+ status_code=409,
+ content={
+ **task.to_status_payload(http_request),
+ "message": "Task execution failed",
+ },
+ )
+
+ return build_sync_file_parse_response(
+ background_tasks=background_tasks,
+ task=task,
+ request=http_request,
+ )
+
+
+@app.post(
+ path="/tasks",
+ status_code=202,
+ summary="Submit an asynchronous parse task",
+ description=(
+ "Submit files for parsing and return immediately with a task id that can be "
+ "checked via the task status and result endpoints."
+ ),
+)
+async def submit_parse_task(
+ http_request: Request,
+ request_options: Annotated[
+ ParseRequestOptions, Depends(parse_request_form)
+ ],
+):
+ task_manager = get_task_manager()
+ task = await create_async_parse_task(request_options)
+ return build_task_submission_response(task, http_request, task_manager)
+
+
+@app.get(path="/tasks/{task_id}", name="get_async_task_status")
+async def get_async_task_status(task_id: str, request: Request):
+ task_manager = get_task_manager()
+ task = task_manager.get(task_id)
+ if task is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+ return task_manager.build_status_payload(task, request)
+
+
+@app.get(path="/tasks/{task_id}/result", name="get_async_task_result")
+async def get_async_task_result(
+ task_id: str,
+ request: Request,
+ background_tasks: BackgroundTasks,
+):
+ task_manager = get_task_manager()
+ task = task_manager.get(task_id)
+ if task is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+
+ if task.status in (TASK_PENDING, TASK_PROCESSING):
+ return JSONResponse(
+ status_code=202,
+ content={
+ **task.to_status_payload(request),
+ "message": "Task result is not ready yet",
+ },
+ )
+
+ if task.status == TASK_FAILED:
+ return JSONResponse(
+ status_code=409,
+ content={
+ **task.to_status_payload(request),
+ "message": "Task execution failed",
+ },
+ )
+
+ return build_result_response(
+ background_tasks=background_tasks,
+ status_code=200,
+ output_dir=task.output_dir,
+ pdf_file_names=task.file_names,
+ backend=task.backend,
+ parse_method=task.parse_method,
+ return_md=task.return_md,
+ return_middle_json=task.return_middle_json,
+ return_model_output=task.return_model_output,
+ return_content_list=task.return_content_list,
+ return_images=task.return_images,
+ response_format_zip=task.response_format_zip,
+ return_original_file=task.return_original_file,
+ zip_filename=f"{task.task_id}.zip",
+ )
+
+
+@app.get(path="/health")
+async def health_check():
+ task_manager = getattr(app.state, "task_manager", None)
+ if task_manager is None or not task_manager.is_healthy():
+ return JSONResponse(
+ status_code=503,
+ content={
+ "status": "unhealthy",
+ "version": __version__,
+ "error": (
+ "Task manager is not initialized"
+ if task_manager is None
+ else task_manager.last_worker_error
+ or (
+ "Task cleanup loop is not running"
+ if (
+ task_manager.task_retention_seconds > 0
+ and task_manager.cleanup_task is None
+ )
+ else "Task dispatcher is not running"
+ )
+ ),
+ },
+ )
+
+ stats = task_manager.get_stats()
+ return {
+ "status": "healthy",
+ "version": __version__,
+ "protocol_version": API_PROTOCOL_VERSION,
+ "queued_tasks": stats[TASK_PENDING],
+ "processing_tasks": stats[TASK_PROCESSING],
+ "completed_tasks": stats[TASK_COMPLETED],
+ "failed_tasks": stats[TASK_FAILED],
+ "max_concurrent_requests": get_max_concurrent_requests(),
+ "processing_window_size": get_processing_window_size(
+ default=DEFAULT_PROCESSING_WINDOW_SIZE
+ ),
+ "task_retention_seconds": task_manager.task_retention_seconds,
+ "task_cleanup_interval_seconds": task_manager.task_cleanup_interval_seconds,
+ }
+
+
+@click.command(
+ context_settings=dict(ignore_unknown_options=True, allow_extra_args=True)
+)
+@click.pass_context
+@click.option("--host", default="127.0.0.1", help="Server host (default: 127.0.0.1)")
+@click.option("--port", default=8000, type=int, help="Server port (default: 8000)")
+@click.option("--reload", is_flag=True, help="Enable auto-reload (development mode)")
+@click.option(
+ "--allow-public-http-client",
+ is_flag=True,
+ help=(
+ "Allow *-http-client backends and server_url even when binding the API to "
+ "0.0.0.0 or ::."
+ ),
+)
+@click.option(
+ "--enable-vlm-preload",
+ "enable_vlm_preload",
+ type=bool,
+ default=False,
+ help="Preload the local VLM model during mineru-api startup.",
+)
+def main(
+ ctx,
+ host,
+ port,
+ reload,
+ allow_public_http_client,
+ enable_vlm_preload,
+ **kwargs,
+):
+ del kwargs
+ raw_config = arg_parse(ctx)
+ raw_config["enable_vlm_preload"] = enable_vlm_preload
+ service_config, model_config = split_service_and_model_config(raw_config)
+ public_bind_exposed = is_public_bind_host(host)
+
+ app.state.service_config = service_config
+ app.state.config = model_config
+ configure_public_http_client_policy(
+ app,
+ public_bind_exposed=public_bind_exposed,
+ allow_public_http_client=allow_public_http_client,
+ )
+ os.environ["MINERU_API_ENABLE_VLM_PRELOAD"] = (
+ "1" if service_config["enable_vlm_preload"] else "0"
+ )
+ os.environ[MINERU_API_PUBLIC_BIND_EXPOSED_ENV] = "1" if public_bind_exposed else "0"
+ os.environ[MINERU_API_ALLOW_PUBLIC_HTTP_CLIENT_ENV] = (
+ "1" if allow_public_http_client else "0"
+ )
+ warn_if_public_http_client_policy(host, allow_public_http_client)
+ access_log = not env_flag_enabled("MINERU_API_DISABLE_ACCESS_LOG")
+
+ print(f"Start MinerU FastAPI Service: http://{host}:{port}")
+ print(f"API documentation: http://{host}:{port}/docs")
+
+ if reload:
+ uvicorn.run(
+ "mineru.cli.fast_api:app",
+ host=host,
+ port=port,
+ reload=True,
+ access_log=access_log,
+ )
+ else:
+ config = uvicorn.Config(
+ app,
+ host=host,
+ port=port,
+ reload=False,
+ access_log=access_log,
+ )
+ server = uvicorn.Server(config)
+ install_stdin_shutdown_watcher(server)
+ server.run()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/mineru/cli/gradio_app.py b/mineru/cli/gradio_app.py
new file mode 100644
index 000000000..6724157c1
--- /dev/null
+++ b/mineru/cli/gradio_app.py
@@ -0,0 +1,1412 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+import base64
+import asyncio
+import httpx
+import os
+import re
+import sys
+import threading
+import time
+import uuid
+import zipfile
+from contextlib import asynccontextmanager
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Callable
+
+import click
+import gradio as gr
+from gradio_pdf import PDF
+from loguru import logger
+
+os.environ["TORCH_CUDNN_V8_API_DISABLED"] = "1"
+# 检测 Gradio 版本，用于兼容 Gradio 5 和 Gradio 6
+_gradio_major_version = int(gr.__version__.split('.')[0])
+IS_GRADIO_6 = _gradio_major_version >= 6
+
+log_level = os.getenv("MINERU_LOG_LEVEL", "INFO").upper()
+logger.remove() # 移除默认handler
+logger.add(sys.stderr, level=log_level) # 添加新handler
+
+from mineru.cli.common import (
+ docx_suffixes,
+ image_suffixes,
+ normalize_task_stem,
+ office_suffixes,
+ pdf_suffixes,
+ read_fn,
+)
+from mineru.cli import api_client as _api_client
+from mineru.cli.output_paths import resolve_parse_dir
+from mineru.cli.vlm_preload import resolve_gradio_local_api_cli_args
+from mineru.cli.visualization import VisualizationJob, run_visualization_job
+
+_gradio_local_api_server = _api_client.ReusableLocalAPIServer()
+
+
+@dataclass(frozen=True)
+class GradioConcurrencyWaitSnapshot:
+ limit: int
+ active: int
+ waiting: int
+ ahead: int
+
+
+@dataclass
+class _LimiterState:
+ semaphore: asyncio.Semaphore
+ active: int = 0
+ waiters: list[object] = field(default_factory=list)
+
+
+class GradioRequestConcurrencyLimiter:
+ def __init__(self):
+ self._lock = threading.Lock()
+ self._states: dict[int, _LimiterState] = {}
+
+ def _get_state(self, limit: int):
+ if limit <= 0:
+ return None
+ with self._lock:
+ state = self._states.get(limit)
+ if state is None:
+ state = _LimiterState(semaphore=asyncio.Semaphore(limit))
+ self._states[limit] = state
+ return state
+
+ def _build_wait_snapshot(
+ self,
+ state: _LimiterState,
+ limit: int,
+ wait_token: object,
+ ) -> GradioConcurrencyWaitSnapshot | None:
+ if wait_token not in state.waiters:
+ return None
+
+ return GradioConcurrencyWaitSnapshot(
+ limit=limit,
+ active=state.active,
+ waiting=len(state.waiters),
+ ahead=state.waiters.index(wait_token),
+ )
+
+ def _remove_waiter(self, state: _LimiterState, wait_token: object) -> None:
+ if wait_token in state.waiters:
+ state.waiters.remove(wait_token)
+
+ async def _cleanup_acquire_interruption(
+ self,
+ state: _LimiterState,
+ acquire_task: asyncio.Task[bool],
+ wait_token: object,
+ should_wait: bool,
+ ) -> None:
+ if not acquire_task.done():
+ acquire_task.cancel()
+ await asyncio.gather(acquire_task, return_exceptions=True)
+ elif not acquire_task.cancelled():
+ try:
+ acquired = acquire_task.result()
+ except Exception:
+ acquired = False
+ if acquired:
+ state.semaphore.release()
+
+ if should_wait:
+ with self._lock:
+ self._remove_waiter(state, wait_token)
+
+ @asynccontextmanager
+ async def acquire(
+ self,
+ limit: int,
+ on_wait: Callable[[GradioConcurrencyWaitSnapshot], None] | None = None,
+ ):
+ state = self._get_state(limit)
+ if state is None:
+ yield
+ return
+
+ wait_token = object()
+ should_wait = False
+ snapshot = None
+ with self._lock:
+ if state.active >= limit or state.waiters:
+ state.waiters.append(wait_token)
+ should_wait = True
+ snapshot = self._build_wait_snapshot(state, limit, wait_token)
+
+ acquire_task: asyncio.Task[bool] = asyncio.create_task(state.semaphore.acquire())
+ last_wait_ahead = None
+ if should_wait and on_wait is not None and snapshot is not None:
+ on_wait(snapshot)
+ last_wait_ahead = snapshot.ahead
+
+ try:
+ if should_wait:
+ while True:
+ done, _ = await asyncio.wait(
+ {acquire_task},
+ timeout=STATUS_TIMER_INTERVAL_SECONDS,
+ )
+ if acquire_task in done:
+ acquire_task.result()
+ break
+
+ if on_wait is None:
+ continue
+
+ with self._lock:
+ snapshot = self._build_wait_snapshot(state, limit, wait_token)
+
+ if snapshot is None or snapshot.ahead == last_wait_ahead:
+ continue
+
+ on_wait(snapshot)
+ last_wait_ahead = snapshot.ahead
+ else:
+ await acquire_task
+ except BaseException:
+ await self._cleanup_acquire_interruption(
+ state=state,
+ acquire_task=acquire_task,
+ wait_token=wait_token,
+ should_wait=should_wait,
+ )
+ raise
+
+ with self._lock:
+ if should_wait:
+ self._remove_waiter(state, wait_token)
+ state.active += 1
+ try:
+ yield
+ finally:
+ with self._lock:
+ state.active = max(0, state.active - 1)
+ state.semaphore.release()
+
+
+_gradio_request_concurrency_limiter = GradioRequestConcurrencyLimiter()
+
+STATUS_BOX_AUTOSCROLL_JS = """
+(value) => {
+ const scrollToBottom = () => {
+ const textarea = document.querySelector(".convert-status-box textarea");
+ if (!textarea) {
+ return;
+ }
+ textarea.scrollTop = textarea.scrollHeight;
+ };
+
+ requestAnimationFrame(() => {
+ scrollToBottom();
+ requestAnimationFrame(scrollToBottom);
+ });
+
+ return [];
+}
+"""
+
+STATUS_TIMER_INTERVAL_SECONDS = 0.1
+STATUS_QUEUE_ANIMATION_INTERVAL_SECONDS = 1.0
+STATUS_QUEUE_ANIMATION_MAX_DOTS = 10
+
+STATUS_PREPARING_REQUEST = "Preparing request..."
+STATUS_CHECKING_SERVER = "Checking server status..."
+STATUS_SUBMITTING_TASK = "Submitting task..."
+STATUS_DOWNLOADING_RESULT = "Task completed, downloading result..."
+STATUS_PROCESSING_OUTPUT = "Preparing outputs..."
+STATUS_COMPLETED = "Completed"
+STATUS_QUEUED_ON_SERVER = "Queued on server"
+STATUS_PROCESSING_ON_SERVER = "Processing on server"
+STATUS_QUEUED_LOCALLY_PREFIX = "Queued locally:"
+
+
+@dataclass
+class StatusPanelState:
+ lines: list[str] = field(default_factory=list)
+ processing_index: int | None = None
+ processing_started_at: float | None = None
+ queue_index: int | None = None
+ queue_started_at: float | None = None
+ queue_base_message: str | None = None
+
+ def append(self, message: str) -> bool:
+ if not message:
+ return False
+
+ if self.is_queue_message(message):
+ self.finalize_processing()
+ return self.update_queue(message)
+
+ if message == STATUS_PROCESSING_ON_SERVER:
+ self.finalize_queue()
+ return self.start_processing()
+
+ self.finalize_processing()
+ self.finalize_queue()
+ if not self.lines or self.lines[-1] != message:
+ self.lines.append(message)
+ return True
+ return False
+
+ def start_processing(self) -> bool:
+ if self.processing_started_at is not None:
+ return self.tick_processing()
+
+ self.processing_started_at = time.monotonic()
+ self.processing_index = len(self.lines)
+ self.lines.append(format_processing_status(0.0))
+ return True
+
+ def tick_processing(self) -> bool:
+ if self.processing_started_at is None or self.processing_index is None:
+ return False
+
+ updated = format_processing_status(
+ max(0.0, time.monotonic() - self.processing_started_at)
+ )
+ if self.lines[self.processing_index] != updated:
+ self.lines[self.processing_index] = updated
+ return True
+ return False
+
+ def finalize_processing(self) -> bool:
+ if self.processing_started_at is None or self.processing_index is None:
+ return False
+
+ self.tick_processing()
+ self.processing_started_at = None
+ self.processing_index = None
+ return True
+
+ def update_queue(self, message: str) -> bool:
+ if (
+ self.queue_index is None
+ or self.queue_started_at is None
+ or self.queue_base_message is None
+ ):
+ self.queue_started_at = time.monotonic()
+ self.queue_index = len(self.lines)
+ self.queue_base_message = message
+ self.lines.append(format_queue_status(message, 0.0))
+ return True
+
+ self.queue_base_message = message
+ updated = format_queue_status(
+ message,
+ max(0.0, time.monotonic() - self.queue_started_at),
+ )
+ if self.lines[self.queue_index] != updated:
+ self.lines[self.queue_index] = updated
+ return True
+ return False
+
+ def tick_queue(self) -> bool:
+ if (
+ self.queue_index is None
+ or self.queue_started_at is None
+ or self.queue_base_message is None
+ ):
+ return False
+
+ updated = format_queue_status(
+ self.queue_base_message,
+ max(0.0, time.monotonic() - self.queue_started_at),
+ )
+ if self.lines[self.queue_index] != updated:
+ self.lines[self.queue_index] = updated
+ return True
+ return False
+
+ def finalize_queue(self) -> bool:
+ if (
+ self.queue_index is None
+ or self.queue_started_at is None
+ or self.queue_base_message is None
+ ):
+ return False
+
+ self.tick_queue()
+ self.queue_index = None
+ self.queue_started_at = None
+ self.queue_base_message = None
+ return True
+
+ def tick(self) -> bool:
+ if self.is_processing:
+ return self.tick_processing()
+ if self.is_queueing:
+ return self.tick_queue()
+ return False
+
+ @property
+ def is_processing(self) -> bool:
+ return self.processing_started_at is not None
+
+ @property
+ def is_queueing(self) -> bool:
+ return self.queue_started_at is not None
+
+ @property
+ def animation_interval_seconds(self) -> float | None:
+ if self.is_processing:
+ return STATUS_TIMER_INTERVAL_SECONDS
+ if self.is_queueing:
+ return STATUS_QUEUE_ANIMATION_INTERVAL_SECONDS
+ return None
+
+ @staticmethod
+ def is_queue_message(message: str) -> bool:
+ return (
+ message.startswith(STATUS_QUEUED_LOCALLY_PREFIX)
+ or message.startswith(STATUS_QUEUED_ON_SERVER)
+ )
+
+ def render(self) -> str:
+ return "\n".join(self.lines)
+
+
+def format_failed_status(error: Exception | str) -> str:
+ return f"Failed: {error}"
+
+
+def format_processing_status(elapsed_seconds: float) -> str:
+ return f"{STATUS_PROCESSING_ON_SERVER} ({elapsed_seconds:.1f}s)"
+
+
+def format_queue_status(base_message: str, elapsed_seconds: float) -> str:
+ dots = "." * (
+ (int(max(0.0, elapsed_seconds)) % STATUS_QUEUE_ANIMATION_MAX_DOTS) + 1
+ )
+ return f"{base_message}{dots}"
+
+
+def format_concurrency_wait_message(snapshot: GradioConcurrencyWaitSnapshot) -> str:
+ return f"{STATUS_QUEUED_LOCALLY_PREFIX} {snapshot.ahead} request(s) ahead"
+
+
+def format_remote_status_message(
+ status_snapshot: _api_client.TaskStatusSnapshot | str,
+) -> str:
+ if isinstance(status_snapshot, _api_client.TaskStatusSnapshot):
+ status = status_snapshot.status
+ queued_ahead = status_snapshot.queued_ahead
+ else:
+ status = status_snapshot
+ queued_ahead = None
+
+ if status == "pending":
+ if queued_ahead is not None:
+ return f"{STATUS_QUEUED_ON_SERVER}: {queued_ahead} request(s) ahead"
+ return STATUS_QUEUED_ON_SERVER
+ if status == "processing":
+ return STATUS_PROCESSING_ON_SERVER
+ if status == "completed":
+ return STATUS_COMPLETED
+ if status == "failed":
+ return format_failed_status("server task failed")
+ return f"Task status: {status}"
+
+
+def compress_directory_to_zip(directory_path, output_zip_path):
+ """压缩指定目录到一个 ZIP 文件。
+
+ :param directory_path: 要压缩的目录路径
+ :param output_zip_path: 输出的 ZIP 文件路径
+ """
+ try:
+ with zipfile.ZipFile(output_zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
+
+ # 遍历目录中的所有文件和子目录
+ for root, dirs, files in os.walk(directory_path):
+ for file in files:
+ # 构建完整的文件路径
+ file_path = os.path.join(root, file)
+ # 计算相对路径
+ arcname = os.path.relpath(file_path, directory_path)
+ # 添加文件到 ZIP 文件
+ zipf.write(file_path, arcname)
+ return 0
+ except Exception as e:
+ logger.exception(e)
+ return -1
+
+
+def image_to_base64(image_path):
+ with open(image_path, 'rb') as image_file:
+ return base64.b64encode(image_file.read()).decode('utf-8')
+
+
+def replace_image_with_base64(markdown_text, image_dir_path):
+ # MIME类型映射
+ mime_types = {
+ '.jpg': 'image/jpeg',
+ '.jpeg': 'image/jpeg',
+ '.png': 'image/png',
+ '.gif': 'image/gif',
+ '.webp': 'image/webp',
+ '.svg': 'image/svg+xml',
+ }
+
+ def _path_to_data_uri(relative_path):
+ file_ext = os.path.splitext(relative_path)[1].lower()
+ if file_ext not in mime_types:
+ return None
+ try:
+ full_path = os.path.join(image_dir_path, relative_path)
+ base64_image = image_to_base64(full_path)
+ return f'data:{mime_types[file_ext]};base64,{base64_image}'
+ except Exception as e:
+ logger.warning(f"Failed to convert image {relative_path} to base64: {e}")
+ return None
+
+ # 匹配Markdown中的图片标签 
+ def replace_md(match):
+ relative_path = match.group(1)
+ data_uri = _path_to_data_uri(relative_path)
+ if data_uri:
+ return f''
+ return match.group(0)
+
+ result = re.sub(r'\!\[(?:[^\]]*)\]\(([^)]+)\)', replace_md, markdown_text)
+
+ # 匹配HTML表格中的 
(跳过已有的data: URI)
+ def replace_html_src(match):
+ relative_path = match.group(1)
+ data_uri = _path_to_data_uri(relative_path)
+ if data_uri:
+ return f'src="{data_uri}"'
+ return match.group(0)
+
+ result = re.sub(r'src="(?!data:)([^"]+)"', replace_html_src, result)
+
+ return result
+
+
+def normalize_language(language):
+ if '(' in language and ')' in language:
+ return language.split('(')[0].strip()
+ return language
+
+
+def resolve_parse_method(file_path, is_ocr, backend):
+ file_suffix = Path(file_path).suffix.lower().lstrip('.')
+ if file_suffix in office_suffixes:
+ return "auto"
+ if backend.startswith("vlm"):
+ return "auto"
+ return "ocr" if is_ocr else "auto"
+
+
+def create_gradio_run_paths(file_path, output_root="./output"):
+ run_id = f"{time.strftime('%y%m%d_%H%M%S')}_{uuid.uuid4().hex[:8]}_{safe_stem(Path(file_path).stem)}"
+ run_root = Path(output_root) / "gradio" / run_id
+ extract_root = run_root / "result"
+ archive_zip_path = run_root / f"{safe_stem(Path(file_path).stem)}.zip"
+ return run_root, extract_root, archive_zip_path
+
+
+def build_gradio_upload_name(file_path):
+ path = Path(file_path)
+ return f"{normalize_task_stem(path.stem)}{path.suffix}"
+
+
+def resolve_result_file_name(submit_response, extract_root, file_path):
+ if submit_response.file_names:
+ return submit_response.file_names[0]
+
+ candidate_dirs = sorted(path.name for path in Path(extract_root).iterdir() if path.is_dir())
+ if len(candidate_dirs) == 1:
+ return candidate_dirs[0]
+ return normalize_task_stem(Path(file_path).stem)
+
+
+async def resolve_server_health(http_client, api_url):
+ if api_url:
+ return await _api_client.fetch_server_health(
+ http_client,
+ _api_client.normalize_base_url(api_url),
+ )
+
+ local_server, started_now = _gradio_local_api_server.ensure_started()
+ if started_now:
+ logger.info(f"Started local mineru-api at {local_server.base_url}")
+ return await _api_client.wait_for_local_api_ready(http_client, local_server)
+
+
+async def ensure_local_api_ready_for_gradio_startup(
+ timeout_seconds: float = _api_client.LOCAL_API_STARTUP_TIMEOUT_SECONDS,
+):
+ local_server, started_now = _gradio_local_api_server.ensure_started()
+ if started_now:
+ logger.info(f"Started local mineru-api at {local_server.base_url}")
+
+ async with httpx.AsyncClient(
+ timeout=_api_client.build_http_timeout(),
+ follow_redirects=True,
+ ) as http_client:
+ return await _api_client.wait_for_local_api_ready(
+ http_client,
+ local_server,
+ timeout_seconds=timeout_seconds,
+ )
+
+
+def maybe_prepare_local_api_for_gradio_startup(
+ *,
+ api_url: str | None,
+ enable_vlm_preload: bool,
+):
+ if api_url is not None or not enable_vlm_preload:
+ return None
+
+ try:
+ return asyncio.run(ensure_local_api_ready_for_gradio_startup())
+ except Exception:
+ _gradio_local_api_server.stop()
+ raise
+
+
+def resolve_gradio_max_concurrent_requests(api_url, server_health):
+ if api_url is None:
+ return server_health.max_concurrent_requests
+
+ return _api_client.resolve_effective_max_concurrent_requests(
+ local_max=_api_client.read_max_concurrent_requests(
+ default=_api_client.DEFAULT_MAX_CONCURRENT_REQUESTS
+ ),
+ server_max=server_health.max_concurrent_requests,
+ )
+
+
+def maybe_generate_local_preview(extract_root, file_name, file_suffix, backend, parse_method):
+ if file_suffix in office_suffixes:
+ return None
+
+ parse_dir = resolve_parse_dir(
+ extract_root,
+ file_name,
+ backend,
+ parse_method,
+ allow_office_fallback=True,
+ )
+ visualization_job = VisualizationJob(
+ document_stem=file_name,
+ backend=backend,
+ parse_method=parse_method,
+ parse_dir=parse_dir,
+ draw_span=backend.startswith("pipeline"),
+ )
+ result = run_visualization_job(visualization_job)
+ if result.status != "finished":
+ logger.warning(
+ f"Skipping visualization for {visualization_job.document_stem}: {result.message}"
+ )
+ return resolve_preview_pdf_path(parse_dir, file_name)
+
+
+async def _run_to_markdown_job(
+ file_path,
+ end_pages=10,
+ is_ocr=False,
+ formula_enable=True,
+ table_enable=True,
+ language="ch",
+ backend="pipeline",
+ url=None,
+ api_url=None,
+ status_callback: Callable[[str], None] | None = None,
+):
+ if file_path is None:
+ return "", "", None, None
+
+ def emit_status(message: str) -> None:
+ if status_callback is not None:
+ status_callback(message)
+
+ normalized_language = normalize_language(language)
+ file_path = str(file_path)
+ file_suffix = Path(file_path).suffix.lower().lstrip('.')
+ parse_method = resolve_parse_method(file_path, is_ocr, backend)
+ run_root, extract_root, archive_zip_path = create_gradio_run_paths(file_path)
+ run_root.mkdir(parents=True, exist_ok=True)
+
+ form_data = _api_client.build_parse_request_form_data(
+ lang_list=[normalized_language],
+ backend=backend,
+ parse_method=parse_method,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ server_url=url,
+ start_page_id=0,
+ end_page_id=end_pages - 1,
+ return_md=True,
+ return_middle_json=True,
+ return_model_output=True,
+ return_content_list=True,
+ return_images=True,
+ response_format_zip=True,
+ return_original_file=True,
+ )
+ upload_assets = [
+ _api_client.UploadAsset(
+ path=Path(file_path),
+ upload_name=build_gradio_upload_name(file_path),
+ )
+ ]
+
+ async with httpx.AsyncClient(
+ timeout=_api_client.build_http_timeout(),
+ follow_redirects=True,
+ ) as http_client:
+ emit_status(STATUS_PREPARING_REQUEST)
+ emit_status(STATUS_CHECKING_SERVER)
+ server_health = await resolve_server_health(http_client, api_url)
+ effective_max_concurrent_requests = resolve_gradio_max_concurrent_requests(
+ api_url=api_url,
+ server_health=server_health,
+ )
+ async with _gradio_request_concurrency_limiter.acquire(
+ effective_max_concurrent_requests,
+ on_wait=lambda snapshot: emit_status(
+ format_concurrency_wait_message(snapshot)
+ ),
+ ):
+ emit_status(STATUS_SUBMITTING_TASK)
+ submit_response = await _api_client.submit_parse_task(
+ base_url=server_health.base_url,
+ upload_assets=upload_assets,
+ form_data=form_data,
+ )
+ emit_status(f"Task submitted：task_id={submit_response.task_id}")
+
+ last_task_snapshot = None
+
+ def handle_task_status(
+ status_snapshot: _api_client.TaskStatusSnapshot,
+ ) -> None:
+ nonlocal last_task_snapshot
+ if status_snapshot == last_task_snapshot:
+ return
+ last_task_snapshot = status_snapshot
+ emit_status(format_remote_status_message(status_snapshot))
+
+ await _api_client.wait_for_task_result(
+ client=http_client,
+ submit_response=submit_response,
+ task_label=Path(file_path).name,
+ status_snapshot_callback=handle_task_status,
+ )
+ emit_status(STATUS_DOWNLOADING_RESULT)
+ result_zip_path = await _api_client.download_result_zip(
+ client=http_client,
+ submit_response=submit_response,
+ task_label=Path(file_path).name,
+ )
+
+ try:
+ _api_client.safe_extract_zip(result_zip_path, extract_root)
+ finally:
+ result_zip_path.unlink(missing_ok=True)
+
+ file_name = resolve_result_file_name(submit_response, extract_root, file_path)
+ local_md_dir = resolve_parse_dir(
+ extract_root,
+ file_name,
+ backend,
+ parse_method,
+ allow_office_fallback=True,
+ )
+ preview_pdf_path = maybe_generate_local_preview(
+ extract_root=extract_root,
+ file_name=file_name,
+ file_suffix=file_suffix,
+ backend=backend,
+ parse_method=parse_method,
+ )
+
+ emit_status(STATUS_PROCESSING_OUTPUT)
+ zip_archive_success = compress_directory_to_zip(local_md_dir, archive_zip_path)
+ if zip_archive_success == 0:
+ logger.info('Compression successful')
+ else:
+ logger.error('Compression failed')
+
+ md_path = Path(local_md_dir) / f"{file_name}.md"
+ with open(md_path, 'r', encoding='utf-8') as f:
+ txt_content = f.read()
+ md_content = replace_image_with_base64(txt_content, local_md_dir)
+
+ if file_suffix in office_suffixes:
+ preview_pdf_path = None
+
+ emit_status(STATUS_COMPLETED)
+ return md_content, txt_content, str(archive_zip_path), preview_pdf_path
+
+
+async def stream_to_markdown(
+ file_path,
+ end_pages=10,
+ is_ocr=False,
+ formula_enable=True,
+ table_enable=True,
+ language="ch",
+ backend="pipeline",
+ url=None,
+ api_url=None,
+):
+ status_state = StatusPanelState()
+ job_task: asyncio.Task | None = None
+ queue_get_task: asyncio.Task | None = None
+ timer_task: asyncio.Task | None = None
+ yield status_state.render(), None, "", "", gr.skip()
+
+ if file_path is None:
+ return
+
+ status_queue: asyncio.Queue[str] = asyncio.Queue()
+ loop = asyncio.get_running_loop()
+
+ def enqueue_status(message: str) -> None:
+ loop.call_soon_threadsafe(status_queue.put_nowait, message)
+
+ try:
+ job_task = asyncio.create_task(
+ _run_to_markdown_job(
+ file_path=file_path,
+ end_pages=end_pages,
+ is_ocr=is_ocr,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ language=language,
+ backend=backend,
+ url=url,
+ api_url=api_url,
+ status_callback=enqueue_status,
+ )
+ )
+
+ while True:
+ if job_task.done() and status_queue.empty():
+ status_state.finalize_processing()
+ status_state.finalize_queue()
+ break
+
+ queue_get_task = asyncio.create_task(status_queue.get())
+ wait_tasks: set[asyncio.Task] = {job_task, queue_get_task}
+ timer_task = None
+ animation_interval = status_state.animation_interval_seconds
+ if animation_interval is not None:
+ timer_task = asyncio.create_task(
+ asyncio.sleep(animation_interval)
+ )
+ wait_tasks.add(timer_task)
+
+ done, pending = await asyncio.wait(
+ wait_tasks,
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+
+ if queue_get_task in done:
+ message = queue_get_task.result()
+ if status_state.append(message):
+ yield status_state.render(), None, "", "", gr.skip()
+ elif timer_task is not None and timer_task in done:
+ if status_state.tick():
+ yield status_state.render(), None, "", "", gr.skip()
+ else:
+ queue_get_task.cancel()
+ await asyncio.gather(queue_get_task, return_exceptions=True)
+
+ for pending_task in pending:
+ if pending_task is job_task:
+ continue
+ pending_task.cancel()
+ await asyncio.gather(pending_task, return_exceptions=True)
+ queue_get_task = None
+ timer_task = None
+
+ while not status_queue.empty():
+ status_state.append(status_queue.get_nowait())
+ except Exception as exc:
+ status_state.append(format_failed_status(exc))
+ yield status_state.render(), None, "", "", gr.skip()
+ raise
+ finally:
+ for task in (queue_get_task, timer_task, job_task):
+ if task is None or task.done():
+ continue
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+ try:
+ md_content, txt_content, archive_zip_path, preview_pdf_path = await job_task
+ except Exception as exc:
+ status_state.append(format_failed_status(exc))
+ yield status_state.render(), None, "", "", gr.skip()
+ raise
+
+ status_state.append(STATUS_COMPLETED)
+ yield (
+ status_state.render(),
+ archive_zip_path,
+ md_content,
+ txt_content,
+ preview_pdf_path,
+ )
+
+
+def resolve_preview_pdf_path(local_md_dir, file_name):
+ layout_pdf_path = os.path.join(local_md_dir, file_name + '_layout.pdf')
+ if os.path.exists(layout_pdf_path):
+ return layout_pdf_path
+
+ origin_pdf_path = os.path.join(local_md_dir, file_name + '_origin.pdf')
+ if os.path.exists(origin_pdf_path):
+ logger.warning(
+ f"Layout preview PDF not found for {file_name}, "
+ f"falling back to origin PDF: {origin_pdf_path}"
+ )
+ return origin_pdf_path
+
+ logger.warning(f"No preview PDF found for {file_name} under {local_md_dir}")
+ return None
+
+
+latex_delimiters_type_a = [
+ {'left': '$$', 'right': '$$', 'display': True},
+ {'left': '$', 'right': '$', 'display': False},
+]
+latex_delimiters_type_b = [
+ {'left': '\\(', 'right': '\\)', 'display': False},
+ {'left': '\\[', 'right': '\\]', 'display': True},
+]
+latex_delimiters_type_all = latex_delimiters_type_a + latex_delimiters_type_b
+
+header_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'resources', 'header.html')
+with open(header_path, mode='r', encoding='utf-8') as header_file:
+ header = header_file.read()
+
+other_lang = [
+ 'ch (Chinese, English, Chinese Traditional)',
+ 'ch_lite (Chinese, English, Chinese Traditional, Japanese)',
+ 'ch_server (Chinese, English, Chinese Traditional, Japanese)',
+ 'en (English)',
+ 'korean (Korean, English)',
+ 'japan (Chinese, English, Chinese Traditional, Japanese)',
+ 'chinese_cht (Chinese, English, Chinese Traditional, Japanese)',
+ 'ta (Tamil, English)',
+ 'te (Telugu, English)',
+ 'ka (Kannada)',
+ 'el (Greek, English)',
+ 'th (Thai, English)'
+]
+add_lang = [
+ 'latin (French, German, Afrikaans, Italian, Spanish, Bosnian, Portuguese, Czech, Welsh, Danish, Estonian, Irish, Croatian, Uzbek, Hungarian, Serbian (Latin), Indonesian, Occitan, Icelandic, Lithuanian, Maori, Malay, Dutch, Norwegian, Polish, Slovak, Slovenian, Albanian, Swedish, Swahili, Tagalog, Turkish, Latin, Azerbaijani, Kurdish, Latvian, Maltese, Pali, Romanian, Vietnamese, Finnish, Basque, Galician, Luxembourgish, Romansh, Catalan, Quechua)',
+ 'arabic (Arabic, Persian, Uyghur, Urdu, Pashto, Kurdish, Sindhi, Balochi, English)',
+ 'east_slavic (Russian, Belarusian, Ukrainian, English)',
+ 'cyrillic (Russian, Belarusian, Ukrainian, Serbian (Cyrillic), Bulgarian, Mongolian, Abkhazian, Adyghe, Kabardian, Avar, Dargin, Ingush, Chechen, Lak, Lezgin, Tabasaran, Kazakh, Kyrgyz, Tajik, Macedonian, Tatar, Chuvash, Bashkir, Malian, Moldovan, Udmurt, Komi, Ossetian, Buryat, Kalmyk, Tuvan, Sakha, Karakalpak, English)',
+ 'devanagari (Hindi, Marathi, Nepali, Bihari, Maithili, Angika, Bhojpuri, Magahi, Santali, Newari, Konkani, Sanskrit, Haryanvi, English)'
+]
+all_lang = [*other_lang, *add_lang]
+
+
+def safe_stem(file_path):
+ stem = Path(file_path).stem
+ # 只保留字母、数字、下划线和点，其他字符替换为下划线
+ return re.sub(r'[^\w.]', '_', stem)
+
+
+def to_pdf(file_path):
+
+ if file_path is None:
+ return None
+
+ pdf_bytes = read_fn(file_path)
+
+ # unique_filename = f'{uuid.uuid4()}.pdf'
+ unique_filename = f'{safe_stem(file_path)}.pdf'
+
+ # 构建完整的文件路径
+ tmp_file_path = os.path.join(os.path.dirname(file_path), unique_filename)
+
+ # 将字节数据写入文件
+ with open(tmp_file_path, 'wb') as tmp_pdf_file:
+ tmp_pdf_file.write(pdf_bytes)
+
+ return tmp_file_path
+
+
+def to_pdf_preview(file_path):
+ """用于 PDF 预览的转换函数，office 文件不支持预览，返回 None。"""
+ if file_path is None:
+ return None
+ file_suffix = Path(file_path).suffix.lower().lstrip('.')
+ if file_suffix in office_suffixes:
+ return None
+ return to_pdf(file_path)
+
+
+def update_file_options_html(file_path, request: gr.Request):
+ """处理文件上传第一阶段：根据文件类型更新 options_group 和 office_html。
+ 将 doc_show（gradio_pdf.PDF）的更新拆分到独立的 .then() 事件中，
+ 以规避 gradio_pdf 0.0.24 在 Gradio 6 中对 value=None 处理不当导致的
+ 整个事件 processing 状态卡死的兼容性问题。
+ """
+ if file_path is None:
+ return (
+ gr.update(visible=True), # options_group - 恢复显示
+ gr.update(value="", visible=False), # office_html - 隐藏
+ )
+
+ file_suffix = Path(file_path).suffix.lower().lstrip('.')
+ is_office = file_suffix in office_suffixes
+
+ if is_office:
+ # 构建可公开访问的文件 URL，供 Microsoft 在线预览使用
+ host = (request.headers.get('x-forwarded-host')
+ or request.headers.get('host', 'localhost:7860'))
+ proto = request.headers.get('x-forwarded-proto', 'http')
+ base_url = f"{proto}://{host}"
+ public_url = f"{base_url}/gradio_api/file={file_path}"
+ viewer_url = f"https://view.officeapps.live.com/op/embed.aspx?src={public_url}"
+ html_content = (
+ f''
+ )
+ return (
+ gr.update(visible=False), # options_group - 隐藏
+ gr.update(value=html_content, visible=True), # office_html - 显示
+ )
+ else:
+ return (
+ gr.update(visible=True), # options_group - 显示
+ gr.update(value="", visible=False), # office_html - 隐藏
+ )
+
+
+def update_doc_show(file_path):
+ """处理文件上传第二阶段：单独更新 doc_show（gradio_pdf.PDF）组件。
+ 对 office 文件仅改变 visible，避免传递 value=None 触发
+ gradio_pdf 0.0.24 在 Gradio 6 中无法完成的加载周期。
+ """
+ if file_path is None:
+ # 无文件时恢复显示并清空（clear 按钮路径）
+ return gr.update(value=None, visible=True)
+
+ file_suffix = Path(file_path).suffix.lower().lstrip('.')
+ is_office = file_suffix in office_suffixes
+
+ if is_office:
+ # 仅隐藏，不改变 value，避免触发 gradio_pdf 加载周期导致事件 pending 卡死
+ return gr.update(visible=False)
+ else:
+ pdf_path = to_pdf_preview(file_path)
+ return gr.update(value=pdf_path, visible=True)
+
+
+@click.command(context_settings=dict(ignore_unknown_options=True, allow_extra_args=True))
+@click.pass_context
+@click.option(
+ '--enable-example',
+ 'example_enable',
+ type=bool,
+ help="Enable example files for input."
+ "The example files to be input need to be placed in the `example` folder within the directory where the command is currently executed.",
+ default=True,
+)
+@click.option(
+ '--enable-http-client',
+ 'http_client_enable',
+ type=bool,
+ help="Enable http-client backend to link openai-compatible servers.",
+ default=False,
+)
+@click.option(
+ '--enable-api',
+ 'api_enable',
+ type=bool,
+ help="Enable gradio API for serving the application.",
+ default=True,
+)
+@click.option(
+ '--max-convert-pages',
+ 'max_convert_pages',
+ type=int,
+ help="Set the maximum number of pages to convert from PDF to Markdown.",
+ default=1000,
+)
+@click.option(
+ '--server-name',
+ 'server_name',
+ type=str,
+ help="Set the server name for the Gradio app.",
+ default=None,
+)
+@click.option(
+ '--server-port',
+ 'server_port',
+ type=int,
+ help="Set the server port for the Gradio app.",
+ default=None,
+)
+@click.option(
+ '--api-url',
+ 'api_url',
+ type=str,
+ help="MinerU FastAPI base URL. If omitted, gradio starts a reusable local mineru-api service.",
+ default=None,
+)
+@click.option(
+ '--enable-vlm-preload',
+ 'enable_vlm_preload',
+ type=bool,
+ help="Preload the local VLM model when gradio starts a local mineru-api service.",
+ default=False,
+)
+@click.option(
+ '--latex-delimiters-type',
+ 'latex_delimiters_type',
+ type=click.Choice(['a', 'b', 'all']),
+ help="Set the type of LaTeX delimiters to use in Markdown rendering:"
+ "'a' for type '$', 'b' for type '()[]', 'all' for both types.",
+ default='all',
+)
+def main(ctx,
+ example_enable,
+ http_client_enable,
+ api_enable, max_convert_pages,
+ server_name, server_port, api_url, enable_vlm_preload, latex_delimiters_type, **kwargs
+):
+
+ # 创建 i18n 实例，支持中英文
+ i18n = gr.I18n(
+ en={
+ "upload_file": "Please select a file to upload (PDF, image, DOCX, PPTX, or XLSX)",
+ "max_pages": "Max convert pages",
+ "backend": "Backend",
+ "server_url": "Server URL",
+ "server_url_info": "OpenAI-compatible server URL for http-client backend.",
+ "recognition_options": "**Recognition Options:**",
+ "table_enable": "Enable table recognition",
+ "table_info": "If disabled, tables will be shown as images.",
+ "formula_label_vlm": "Enable display formula recognition",
+ "formula_label_pipeline": "Enable formula recognition",
+ "formula_label_hybrid": "Enable inline formula recognition",
+ "formula_info_vlm": "If disabled, display formulas will be shown as images.",
+ "formula_info_pipeline": "If disabled, display formulas will be shown as images, and inline formulas will not be detected or parsed.",
+ "formula_info_hybrid": "If disabled, inline formulas will not be detected or parsed.",
+ "ocr_language": "OCR Language",
+ "ocr_language_info": "Select the OCR language for image-based PDFs and images.",
+ "force_ocr": "Force enable OCR",
+ "force_ocr_info": "Enable only if the result is extremely poor. Requires correct OCR language.",
+ "convert": "Convert",
+ "clear": "Clear",
+ "doc_preview": "Document preview",
+ "examples": "Examples:",
+ "convert_status": "Conversion Status",
+ "convert_result": "Convert result",
+ "md_rendering": "Markdown rendering",
+ "md_text": "Markdown text",
+ "backend_info_vlm": "High-precision parsing via VLM, supports Chinese and English documents only.",
+ "backend_info_pipeline": "Traditional Multi-model pipeline parsing, supports multiple languages, hallucination-free.",
+ "backend_info_hybrid": "High-precision hybrid parsing, supports multiple languages.",
+ "backend_info_default": "Select the backend engine for document parsing.",
+ },
+ zh={
+ "upload_file": "请选择要上传的文件（PDF、图片、DOCX、PPTX 或 XLSX）",
+ "max_pages": "最大转换页数",
+ "backend": "解析后端",
+ "server_url": "服务器地址",
+ "server_url_info": "http-client 后端的 OpenAI 兼容服务器地址。",
+ "recognition_options": "**识别选项：**",
+ "table_enable": "启用表格识别",
+ "table_info": "禁用后，表格将显示为图片。",
+ "formula_label_vlm": "启用行间公式识别",
+ "formula_label_pipeline": "启用公式识别",
+ "formula_label_hybrid": "启用行内公式识别",
+ "formula_info_vlm": "禁用后，行间公式将显示为图片。",
+ "formula_info_pipeline": "禁用后，行间公式将显示为图片，行内公式将不会被检测或解析。",
+ "formula_info_hybrid": "禁用后，行内公式将不会被检测或解析。",
+ "ocr_language": "OCR 语言",
+ "ocr_language_info": "为扫描版 PDF 和图片选择 OCR 语言。",
+ "force_ocr": "强制启用 OCR",
+ "force_ocr_info": "仅在识别效果极差时启用，需选择正确的 OCR 语言。",
+ "convert": "转换",
+ "clear": "清除",
+ "doc_preview": "文档预览",
+ "examples": "示例：",
+ "convert_status": "转换状态",
+ "convert_result": "转换结果",
+ "md_rendering": "Markdown 渲染",
+ "md_text": "Markdown 文本",
+ "backend_info_vlm": "多模态大模型高精度解析，仅支持中英文文档。",
+ "backend_info_pipeline": "传统多模型管道解析，支持多语言，无幻觉。",
+ "backend_info_hybrid": "高精度混合解析，支持多语言。",
+ "backend_info_default": "选择文档解析的后端引擎。",
+ },
+ )
+
+ # 根据后端类型获取公式识别标签（闭包函数以支持 i18n）
+ def get_formula_label(backend_choice):
+ if backend_choice.startswith("vlm"):
+ return i18n("formula_label_vlm")
+ elif backend_choice == "pipeline":
+ return i18n("formula_label_pipeline")
+ elif backend_choice.startswith("hybrid"):
+ return i18n("formula_label_hybrid")
+ else:
+ return i18n("formula_label_pipeline")
+
+ def get_formula_info(backend_choice):
+ if backend_choice.startswith("vlm"):
+ return i18n("formula_info_vlm")
+ elif backend_choice == "pipeline":
+ return i18n("formula_info_pipeline")
+ elif backend_choice.startswith("hybrid"):
+ return i18n("formula_info_hybrid")
+ else:
+ return ""
+
+ def get_backend_info(backend_choice):
+ if backend_choice.startswith("vlm"):
+ return i18n("backend_info_vlm")
+ elif backend_choice == "pipeline":
+ return i18n("backend_info_pipeline")
+ elif backend_choice.startswith("hybrid"):
+ return i18n("backend_info_hybrid")
+ else:
+ return i18n("backend_info_default")
+
+ # 更新界面函数
+ def update_interface(backend_choice):
+ formula_label_update = gr.update(label=get_formula_label(backend_choice), info=get_formula_info(backend_choice))
+ backend_info_update = gr.update(info=get_backend_info(backend_choice))
+ if "http-client" in backend_choice:
+ client_options_update = gr.update(visible=True)
+ else:
+ client_options_update = gr.update(visible=False)
+ if "vlm" in backend_choice:
+ ocr_options_update = gr.update(visible=False)
+ else:
+ ocr_options_update = gr.update(visible=True)
+
+ return client_options_update, ocr_options_update, formula_label_update, backend_info_update
+
+
+ del kwargs
+ _gradio_local_api_server.configure(
+ resolve_gradio_local_api_cli_args(
+ ctx.args,
+ api_url=api_url,
+ enable_vlm_preload=enable_vlm_preload,
+ )
+ )
+
+ if latex_delimiters_type == 'a':
+ latex_delimiters = latex_delimiters_type_a
+ elif latex_delimiters_type == 'b':
+ latex_delimiters = latex_delimiters_type_b
+ elif latex_delimiters_type == 'all':
+ latex_delimiters = latex_delimiters_type_all
+ else:
+ raise ValueError(f"Invalid latex delimiters type: {latex_delimiters_type}.")
+
+
+ async def convert_to_markdown_stream(
+ file_path,
+ end_pages=10,
+ is_ocr=False,
+ formula_enable=True,
+ table_enable=True,
+ language="ch",
+ backend="pipeline",
+ url=None,
+ ):
+ async for update in stream_to_markdown(
+ file_path=file_path,
+ end_pages=end_pages,
+ is_ocr=is_ocr,
+ formula_enable=formula_enable,
+ table_enable=table_enable,
+ language=language,
+ backend=backend,
+ url=url,
+ api_url=api_url,
+ ):
+ yield update
+
+ suffixes = [f".{suffix}" for suffix in pdf_suffixes + image_suffixes + office_suffixes]
+ with gr.Blocks() as demo:
+ gr.HTML(header)
+ with gr.Row():
+ with gr.Column(variant='panel', scale=5):
+ with gr.Row():
+ input_file = gr.File(label=i18n("upload_file"), file_types=suffixes)
+ # 下面这些选项在上传 office 文件时会被自动隐藏
+ with gr.Group() as options_group:
+ with gr.Row():
+ max_pages = gr.Slider(1, max_convert_pages, max_convert_pages, step=1, label=i18n("max_pages"))
+ with gr.Row():
+ drop_list = ["pipeline", "vlm-auto-engine", "hybrid-auto-engine"]
+ preferred_option = "hybrid-auto-engine"
+ if http_client_enable:
+ drop_list.extend(["vlm-http-client", "hybrid-http-client"])
+ backend = gr.Dropdown(drop_list, label=i18n("backend"), value=preferred_option, info=get_backend_info(preferred_option))
+ with gr.Row(visible=False) as client_options:
+ url = gr.Textbox(label=i18n("server_url"), value='http://localhost:30000', placeholder='http://localhost:30000', info=i18n("server_url_info"))
+ with gr.Row(equal_height=True):
+ with gr.Column():
+ gr.Markdown(i18n("recognition_options"))
+ table_enable = gr.Checkbox(label=i18n("table_enable"), value=True, info=i18n("table_info"))
+ formula_enable = gr.Checkbox(label=get_formula_label(preferred_option), value=True, info=get_formula_info(preferred_option))
+ with gr.Column() as ocr_options:
+ language = gr.Dropdown(all_lang, label=i18n("ocr_language"), value='ch (Chinese, English, Chinese Traditional)', info=i18n("ocr_language_info"))
+ is_ocr = gr.Checkbox(label=i18n("force_ocr"), value=False, info=i18n("force_ocr_info"))
+ with gr.Row():
+ change_bu = gr.Button(i18n("convert"))
+ clear_bu = gr.ClearButton(value=i18n("clear"))
+ _doc_preview_label = "doc preview" if IS_GRADIO_6 else i18n("doc_preview")
+ doc_show = PDF(label=_doc_preview_label, interactive=False, visible=True, height=800)
+ office_html = gr.HTML(value="", visible=False)
+ if example_enable:
+ example_root = os.path.join(os.getcwd(), 'examples')
+ if os.path.exists(example_root):
+ gr.Examples(
+ label=i18n("examples"),
+ examples=[os.path.join(example_root, _) for _ in os.listdir(example_root) if
+ _.endswith(tuple(suffixes))],
+ inputs=input_file
+ )
+
+ with gr.Column(variant='panel', scale=5):
+ status_box = gr.TextArea(
+ label=i18n("convert_status"),
+ value="",
+ lines=4,
+ max_lines=4,
+ interactive=False,
+ autoscroll=True,
+ elem_classes=["convert-status-box"],
+ )
+ output_file = gr.File(label=i18n("convert_result"), interactive=False)
+ with gr.Blocks():
+ with gr.Tab(i18n("md_rendering")):
+ _md_copy_kwargs = {"buttons": ["copy"]} if IS_GRADIO_6 else {"show_copy_button": True}
+ md = gr.Markdown(
+ label=i18n("md_rendering"),
+ height=1200,
+ latex_delimiters=latex_delimiters,
+ line_breaks=True,
+ **_md_copy_kwargs
+ )
+ with gr.Tab(i18n("md_text")):
+ _textarea_copy_kwargs = {"buttons": ["copy"]} if IS_GRADIO_6 else {"show_copy_button": True}
+ md_text = gr.TextArea(
+ lines=45,
+ label=i18n("md_text"),
+ **_textarea_copy_kwargs
+ )
+
+ # 添加事件处理
+ _private_api_kwargs = (
+ {"api_visibility": "private", "queue": False}
+ if IS_GRADIO_6
+ else {"api_name": False, "queue": False}
+ )
+ backend.change(
+ fn=update_interface,
+ inputs=[backend],
+ outputs=[client_options, ocr_options, formula_enable, backend],
+ **_private_api_kwargs
+ )
+ # 添加demo.load事件，在页面加载时触发一次界面更新
+ demo.load(
+ fn=update_interface,
+ inputs=[backend],
+ outputs=[client_options, ocr_options, formula_enable, backend],
+ **_private_api_kwargs
+ )
+ status_box.change(
+ fn=None,
+ inputs=[status_box],
+ outputs=[],
+ js=STATUS_BOX_AUTOSCROLL_JS,
+ **_private_api_kwargs
+ )
+ clear_bu.add([input_file, md, doc_show, md_text, output_file, is_ocr, office_html, status_box])
+
+ # 清除按钮额外重置 UI 可见性（ClearButton 不一定触发 input_file.change）
+ clear_bu.click(
+ fn=lambda: (
+ gr.update(visible=True),
+ gr.update(value=None, visible=True),
+ gr.update(value="", visible=False),
+ gr.update(value=""),
+ ),
+ inputs=[],
+ outputs=[options_group, doc_show, office_html, status_box],
+ **_private_api_kwargs
+ )
+
+ # 第一阶段：快速更新 options_group 和 office_html，不涉及 gradio_pdf 组件
+ # 第二阶段（.then）：单独更新 doc_show，使 office_html 的 processing 遮罩
+ # 在第一阶段完成后立即消失，规避 gradio_pdf 0.0.24 与 Gradio 6 的兼容性问题。
+ input_file.change(
+ fn=update_file_options_html,
+ inputs=input_file,
+ outputs=[options_group, office_html],
+ **_private_api_kwargs
+ ).then(
+ fn=update_doc_show,
+ inputs=input_file,
+ outputs=[doc_show],
+ **_private_api_kwargs
+ )
+ _to_md_api_kwargs = (
+ {
+ "api_visibility": "public" if api_enable else "private",
+ "queue": True,
+ "show_progress": "hidden",
+ }
+ if IS_GRADIO_6
+ else {
+ "api_name": "to_markdown" if api_enable else False,
+ "queue": True,
+ "show_progress": "hidden",
+ }
+ )
+ change_bu.click(
+ fn=convert_to_markdown_stream,
+ inputs=[input_file, max_pages, is_ocr, formula_enable, table_enable, language, backend, url],
+ outputs=[status_box, output_file, md, md_text, doc_show],
+ **_to_md_api_kwargs
+ )
+
+ demo.queue(default_concurrency_limit=None)
+
+ if IS_GRADIO_6:
+ footer_links = ["gradio", "settings"]
+ if api_enable:
+ footer_links.append("api")
+ _launch_kwargs = {"footer_links": footer_links}
+ else:
+ _launch_kwargs = {"show_api": api_enable}
+ maybe_prepare_local_api_for_gradio_startup(
+ api_url=api_url,
+ enable_vlm_preload=enable_vlm_preload,
+ )
+ demo.launch(
+ server_name=server_name,
+ server_port=server_port,
+ i18n=i18n,
+ **_launch_kwargs,
+ )
+
+
+if __name__ == '__main__':
+ main()
diff --git a/mineru/cli/models_download.py b/mineru/cli/models_download.py
new file mode 100644
index 000000000..4179e01fc
--- /dev/null
+++ b/mineru/cli/models_download.py
@@ -0,0 +1,188 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from contextlib import contextmanager
+import json
+import os
+import sys
+import click
+import requests
+from loguru import logger
+
+from mineru.utils.enum_class import ModelPath
+from mineru.utils.models_download_utils import auto_download_and_get_model_root_path
+
+MODEL_SOURCE_ENV_VAR = 'MINERU_MODEL_SOURCE'
+REMOTE_MODEL_SOURCES = ('huggingface', 'modelscope')
+
+
+def download_json(url):
+ """下载JSON文件"""
+ response = requests.get(url)
+ response.raise_for_status()
+ return response.json()
+
+
+def download_and_modify_json(url, local_filename, modifications):
+ """下载JSON并修改内容"""
+ if os.path.exists(local_filename):
+ data = json.load(open(local_filename))
+ config_version = data.get('config_version', '0.0.0')
+ if config_version < '1.3.1':
+ data = download_json(url)
+ else:
+ data = download_json(url)
+
+ # 修改内容
+ for key, value in modifications.items():
+ if key in data:
+ if isinstance(data[key], dict):
+ # 如果是字典，合并新值
+ data[key].update(value)
+ else:
+ # 否则直接替换
+ data[key] = value
+
+ # 保存修改后的内容
+ with open(local_filename, 'w', encoding='utf-8') as f:
+ json.dump(data, f, ensure_ascii=False, indent=4)
+
+
+def configure_model(model_dir, model_type):
+ """配置模型"""
+ json_url = 'https://gcore.jsdelivr.net/gh/opendatalab/MinerU@master/mineru.template.json'
+ config_file_name = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json')
+ home_dir = os.path.expanduser('~')
+ config_file = os.path.join(home_dir, config_file_name)
+
+ json_mods = {
+ 'models-dir': {
+ f'{model_type}': model_dir
+ }
+ }
+
+ download_and_modify_json(json_url, config_file, json_mods)
+ logger.info(f'The configuration file has been successfully configured, the path is: {config_file}')
+
+
+def download_pipeline_models():
+ """下载Pipeline模型"""
+ model_paths = [
+ ModelPath.pp_doclayout_v2,
+ ModelPath.unimernet_small,
+ ModelPath.pytorch_paddle,
+ ModelPath.slanet_plus,
+ ModelPath.unet_structure,
+ ModelPath.paddle_table_cls,
+ ModelPath.paddle_orientation_classification,
+ ModelPath.pp_formulanet_plus_m,
+ ]
+ download_finish_path = ""
+ for model_path in model_paths:
+ logger.info(f"Downloading model: {model_path}")
+ download_finish_path = auto_download_and_get_model_root_path(model_path, repo_mode='pipeline')
+ logger.info(f"Pipeline models downloaded successfully to: {download_finish_path}")
+ configure_model(download_finish_path, "pipeline")
+
+
+def download_vlm_models():
+ """下载VLM模型"""
+ download_finish_path = auto_download_and_get_model_root_path("/", repo_mode='vlm')
+ logger.info(f"VLM models downloaded successfully to: {download_finish_path}")
+ configure_model(download_finish_path, "vlm")
+
+
+def get_effective_download_model_source(requested_model_source):
+ """获取本次下载命令实际使用的模型源。"""
+ current_model_source = os.getenv(MODEL_SOURCE_ENV_VAR)
+ if current_model_source == 'local':
+ logger.warning(
+ f"{MODEL_SOURCE_ENV_VAR}=local means using pre-downloaded local models. "
+ f"`mineru-models-download` will temporarily use '{requested_model_source}' "
+ f"to perform a real download."
+ )
+ return requested_model_source
+
+ if current_model_source is None:
+ return requested_model_source
+
+ return current_model_source
+
+
+@contextmanager
+def temporary_model_source(model_source):
+ """在命令执行期间临时设置模型源，并在结束后恢复。"""
+ original_model_source = os.getenv(MODEL_SOURCE_ENV_VAR)
+ os.environ[MODEL_SOURCE_ENV_VAR] = model_source
+ try:
+ yield
+ finally:
+ if original_model_source is None:
+ os.environ.pop(MODEL_SOURCE_ENV_VAR, None)
+ else:
+ os.environ[MODEL_SOURCE_ENV_VAR] = original_model_source
+
+
+@click.command()
+@click.option(
+ '-s',
+ '--source',
+ 'model_source',
+ type=click.Choice(REMOTE_MODEL_SOURCES),
+ help="""
+ The source of the model repository.
+ """,
+ default=None,
+)
+@click.option(
+ '-m',
+ '--model_type',
+ 'model_type',
+ type=click.Choice(['pipeline', 'vlm', 'all']),
+ help="""
+ The type of the model to download.
+ """,
+ default=None,
+)
+def download_models(model_source, model_type):
+ """Download MinerU model files.
+
+ Supports downloading pipeline or VLM models from ModelScope or HuggingFace.
+ """
+ # 如果未显式指定则交互式输入下载来源
+ if model_source is None:
+ model_source = click.prompt(
+ "Please select the model download source: ",
+ type=click.Choice(REMOTE_MODEL_SOURCES),
+ default='huggingface'
+ )
+
+ effective_model_source = get_effective_download_model_source(model_source)
+
+ # 如果未显式指定则交互式输入模型类型
+ if model_type is None:
+ model_type = click.prompt(
+ "Please select the model type to download: ",
+ type=click.Choice(['pipeline', 'vlm', 'all']),
+ default='all'
+ )
+
+ logger.info(f"Downloading {model_type} model from {effective_model_source}...")
+
+ try:
+ with temporary_model_source(effective_model_source):
+ if model_type == 'pipeline':
+ download_pipeline_models()
+ elif model_type == 'vlm':
+ download_vlm_models()
+ elif model_type == 'all':
+ download_pipeline_models()
+ download_vlm_models()
+ else:
+ click.echo(f"Unsupported model type: {model_type}", err=True)
+ sys.exit(1)
+
+ except Exception as e:
+ logger.exception(f"An error occurred while downloading models: {str(e)}")
+ sys.exit(1)
+
+if __name__ == '__main__':
+ download_models()
diff --git a/mineru/cli/output_paths.py b/mineru/cli/output_paths.py
new file mode 100644
index 000000000..4cb478144
--- /dev/null
+++ b/mineru/cli/output_paths.py
@@ -0,0 +1,58 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from pathlib import Path
+
+
+OFFICE_PARSE_DIR_NAME = "office"
+VLM_PARSE_DIR_NAME = "vlm"
+
+
+def build_parse_dir(
+ output_dir: str | Path,
+ pdf_name: str,
+ backend: str,
+ parse_method: str,
+ *,
+ is_office: bool = False,
+) -> Path:
+ output_root = Path(output_dir)
+ if is_office:
+ return output_root / pdf_name / OFFICE_PARSE_DIR_NAME
+ if backend.startswith("pipeline"):
+ return output_root / pdf_name / parse_method
+ if backend.startswith("vlm"):
+ return output_root / pdf_name / VLM_PARSE_DIR_NAME
+ if backend.startswith("hybrid"):
+ return output_root / pdf_name / f"hybrid_{parse_method}"
+ raise ValueError(f"Unknown backend type: {backend}")
+
+
+def resolve_parse_dir(
+ output_dir: str | Path,
+ pdf_name: str,
+ backend: str,
+ parse_method: str,
+ *,
+ is_office: bool = False,
+ allow_office_fallback: bool = False,
+) -> Path:
+ parse_dir = build_parse_dir(
+ output_dir,
+ pdf_name,
+ backend,
+ parse_method,
+ is_office=is_office,
+ )
+ if is_office:
+ return parse_dir
+
+ if allow_office_fallback and not parse_dir.exists():
+ office_dir = build_parse_dir(
+ output_dir,
+ pdf_name,
+ backend,
+ parse_method,
+ is_office=True,
+ )
+ if office_dir.exists():
+ return office_dir
+ return parse_dir
diff --git a/mineru/cli/public_http_client_policy.py b/mineru/cli/public_http_client_policy.py
new file mode 100644
index 000000000..e14420303
--- /dev/null
+++ b/mineru/cli/public_http_client_policy.py
@@ -0,0 +1,65 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from fastapi import FastAPI, HTTPException
+from loguru import logger
+
+
+PUBLIC_HTTP_CLIENT_DISABLED_DETAIL = (
+ "Publicly exposed API disables *-http-client backends and server_url by "
+ "default. Rebind to 127.0.0.1 or start with "
+ "--allow-public-http-client if you understand the SSRF risk."
+)
+
+
+def is_public_bind_host(host: str) -> bool:
+ return host in {"0.0.0.0", "::"}
+
+
+def configure_public_http_client_policy(
+ app: FastAPI,
+ *,
+ public_bind_exposed: bool,
+ allow_public_http_client: bool,
+) -> None:
+ app.state.public_bind_exposed = public_bind_exposed
+ app.state.allow_public_http_client = allow_public_http_client
+
+
+def validate_public_http_client_request(
+ *,
+ public_bind_exposed: bool,
+ allow_public_http_client: bool,
+ backend: str,
+ server_url: str | None,
+) -> None:
+ if not public_bind_exposed or allow_public_http_client:
+ return
+ if backend.endswith("-http-client") or bool(server_url and server_url.strip()):
+ raise HTTPException(status_code=400, detail=PUBLIC_HTTP_CLIENT_DISABLED_DETAIL)
+
+
+def warn_if_public_http_client_policy(
+ *,
+ service_name: str,
+ host: str,
+ allow_public_http_client: bool,
+) -> None:
+ if not is_public_bind_host(host):
+ return
+ if allow_public_http_client:
+ logger.warning(
+ "MinerU {} is listening on {} with --allow-public-http-client enabled. "
+ "Requests may supply remote HTTP inference endpoints and turn the service "
+ "into an externally driven outbound request primitive, creating SSRF and "
+ "internal network probing risk.",
+ service_name,
+ host,
+ )
+ return
+ logger.warning(
+ "MinerU {} is listening on {}. Disabling *-http-client backends and "
+ "server_url by default because these inputs let callers choose remote HTTP "
+ "inference endpoints; when the API is publicly reachable, that creates SSRF "
+ "and internal network probing risk.",
+ service_name,
+ host,
+ )
diff --git a/mineru/cli/router.py b/mineru/cli/router.py
new file mode 100644
index 000000000..5fb6f7d34
--- /dev/null
+++ b/mineru/cli/router.py
@@ -0,0 +1,1532 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import asyncio
+import json
+import os
+import random
+import shutil
+import subprocess
+import sys
+import tempfile
+import uuid
+from contextlib import ExitStack, asynccontextmanager, suppress
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any, Optional, Sequence
+
+import click
+import httpx
+import uvicorn
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.middleware.gzip import GZipMiddleware
+from fastapi.responses import JSONResponse, Response, StreamingResponse
+from loguru import logger
+from starlette.background import BackgroundTask
+from starlette.datastructures import UploadFile as StarletteUploadFile
+
+from mineru.cli.api_client import (
+ LOCAL_API_CLEANUP_RETRIES,
+ LOCAL_API_CLEANUP_RETRY_INTERVAL_SECONDS,
+ LOCAL_API_STARTUP_TIMEOUT_SECONDS,
+ TASK_RESULT_TIMEOUT_SECONDS,
+ TASK_STATUS_POLL_INTERVAL_SECONDS,
+ build_managed_process_popen_kwargs,
+ build_http_timeout,
+ find_free_port,
+ normalize_base_url,
+ stop_managed_process,
+ strip_local_api_network_args,
+ response_detail,
+)
+from mineru.cli.api_protocol import API_PROTOCOL_VERSION
+from mineru.cli.common import normalize_upload_filename
+from mineru.cli.public_http_client_policy import (
+ configure_public_http_client_policy,
+ is_public_bind_host,
+ validate_public_http_client_request,
+ warn_if_public_http_client_policy as _warn_if_public_http_client_policy,
+)
+from mineru.cli.vlm_preload import build_local_api_cli_args
+from mineru.version import __version__
+
+TASK_PENDING = "pending"
+TASK_PROCESSING = "processing"
+TASK_COMPLETED = "completed"
+TASK_FAILED = "failed"
+TASK_TERMINAL_STATES = {TASK_COMPLETED, TASK_FAILED}
+DEFAULT_TASK_RETENTION_SECONDS = 24 * 60 * 60
+DEFAULT_TASK_CLEANUP_INTERVAL_SECONDS = 5 * 60
+FILE_PARSE_TASK_ID_HEADER = "X-MinerU-Task-Id"
+FILE_PARSE_TASK_STATUS_HEADER = "X-MinerU-Task-Status"
+FILE_PARSE_TASK_STATUS_URL_HEADER = "X-MinerU-Task-Status-Url"
+FILE_PARSE_TASK_RESULT_URL_HEADER = "X-MinerU-Task-Result-Url"
+HEALTH_ENDPOINT = "/health"
+TASKS_ENDPOINT = "/tasks"
+SOURCE_LOCAL = "local"
+SOURCE_REMOTE = "remote"
+LOCAL_GPU_AUTO = "auto"
+LOCAL_GPU_NONE = "none"
+HTTP_RETRYABLE_STATUS_CODES = {500, 502, 503, 504}
+UPSTREAM_FAILURE_THRESHOLD = 3
+WORKER_REFRESH_INTERVAL_SECONDS = 2.0
+MIN_HEALTHY_PROCESSING_WINDOW_SIZE = 1
+MINERU_ROUTER_PUBLIC_BIND_EXPOSED_ENV = "MINERU_ROUTER_PUBLIC_BIND_EXPOSED"
+MINERU_ROUTER_ALLOW_PUBLIC_HTTP_CLIENT_ENV = "MINERU_ROUTER_ALLOW_PUBLIC_HTTP_CLIENT"
+
+
+def utc_now_iso() -> str:
+ return datetime.now(timezone.utc).isoformat()
+
+
+def env_flag_enabled(name: str, default: bool = False) -> bool:
+ value = os.getenv(name)
+ if value is None:
+ return default
+ return value.lower() in ("1", "true", "yes", "on")
+
+
+def get_int_env(name: str, default: int, minimum: int = 0) -> int:
+ try:
+ value = int(os.getenv(name, str(default)))
+ except ValueError:
+ return default
+ if value < minimum:
+ return default
+ return value
+
+
+def _parse_json_object_response(
+ response: httpx.Response,
+ payload_name: str,
+) -> dict[str, Any]:
+ try:
+ payload = response.json()
+ except ValueError as exc:
+ raise ValueError(f"{payload_name} is not valid JSON") from exc
+ if not isinstance(payload, dict):
+ raise ValueError(f"{payload_name} must be a JSON object")
+ return payload
+
+
+def get_task_retention_seconds() -> int:
+ return get_int_env(
+ "MINERU_API_TASK_RETENTION_SECONDS",
+ DEFAULT_TASK_RETENTION_SECONDS,
+ minimum=0,
+ )
+
+
+def get_task_cleanup_interval_seconds() -> int:
+ return get_int_env(
+ "MINERU_API_TASK_CLEANUP_INTERVAL_SECONDS",
+ DEFAULT_TASK_CLEANUP_INTERVAL_SECONDS,
+ minimum=1,
+ )
+
+
+def is_task_terminal(status: str) -> bool:
+ return status in TASK_TERMINAL_STATES
+
+
+def warn_if_public_http_client_policy(host: str, allow_public_http_client: bool) -> None:
+ _warn_if_public_http_client_policy(
+ service_name="router",
+ host=host,
+ allow_public_http_client=allow_public_http_client,
+ )
+
+
+def cleanup_path(path: str) -> None:
+ try:
+ if os.path.isfile(path):
+ os.remove(path)
+ elif os.path.isdir(path):
+ shutil.rmtree(path)
+ except FileNotFoundError:
+ return
+ except Exception as exc:
+ logger.warning("Failed to clean up {}: {}", path, exc)
+
+
+def cleanup_temporary_directory(temp_dir: tempfile.TemporaryDirectory[str]) -> None:
+ last_error: Exception | None = None
+ for attempt in range(LOCAL_API_CLEANUP_RETRIES):
+ try:
+ temp_dir.cleanup()
+ return
+ except FileNotFoundError:
+ return
+ except Exception as exc:
+ last_error = exc
+ if attempt + 1 < LOCAL_API_CLEANUP_RETRIES:
+ import time
+
+ time.sleep(LOCAL_API_CLEANUP_RETRY_INTERVAL_SECONDS)
+
+ if last_error is not None:
+ logger.warning("Failed to clean up temporary directory {}: {}", temp_dir.name, last_error)
+
+
+def parse_json_env(name: str, default: Sequence[str] = ()) -> tuple[str, ...]:
+ raw = os.getenv(name)
+ if not raw:
+ return tuple(default)
+ try:
+ payload = json.loads(raw)
+ except json.JSONDecodeError:
+ logger.warning("Invalid {} value: {}", name, raw)
+ return tuple(default)
+ if not isinstance(payload, list) or not all(isinstance(item, str) for item in payload):
+ logger.warning("Invalid {} value: {}", name, raw)
+ return tuple(default)
+ return tuple(payload)
+
+
+def resolve_connect_host(host: str) -> str:
+ if host in {"0.0.0.0", "::"}:
+ return "127.0.0.1"
+ return host
+
+
+def detect_visible_cuda_devices() -> list[str]:
+ configured_visible_devices = os.getenv("CUDA_VISIBLE_DEVICES")
+ if configured_visible_devices is not None:
+ return [
+ item.strip()
+ for item in configured_visible_devices.split(",")
+ if item.strip()
+ ]
+
+ try:
+ import torch # type: ignore
+ except ImportError:
+ return []
+ if not torch.cuda.is_available():
+ return []
+ return [str(index) for index in range(torch.cuda.device_count())]
+
+
+def parse_local_gpus(local_gpus: str) -> list[str | None]:
+ value = local_gpus.strip().lower()
+ if value == LOCAL_GPU_NONE:
+ return []
+ if value == LOCAL_GPU_AUTO:
+ detected = detect_visible_cuda_devices()
+ if detected:
+ return detected
+ return [None]
+
+ resolved: list[str | None] = []
+ for item in local_gpus.split(","):
+ normalized = item.strip()
+ if not normalized:
+ continue
+ resolved.append(normalized)
+ return resolved
+
+
+@dataclass(frozen=True)
+class RouterSettings:
+ upstream_urls: tuple[str, ...] = ()
+ local_gpus: str = LOCAL_GPU_AUTO
+ worker_host: str = "127.0.0.1"
+ enable_vlm_preload: bool = False
+ worker_extra_args: tuple[str, ...] = ()
+ task_retention_seconds: int = DEFAULT_TASK_RETENTION_SECONDS
+ task_cleanup_interval_seconds: int = DEFAULT_TASK_CLEANUP_INTERVAL_SECONDS
+ worker_refresh_interval_seconds: float = WORKER_REFRESH_INTERVAL_SECONDS
+
+ @classmethod
+ def from_env(cls) -> "RouterSettings":
+ return cls(
+ upstream_urls=parse_json_env("MINERU_ROUTER_UPSTREAM_URLS_JSON"),
+ local_gpus=os.getenv("MINERU_ROUTER_LOCAL_GPUS", LOCAL_GPU_AUTO),
+ worker_host=os.getenv("MINERU_ROUTER_WORKER_HOST", "127.0.0.1"),
+ enable_vlm_preload=env_flag_enabled(
+ "MINERU_ROUTER_ENABLE_VLM_PRELOAD",
+ default=False,
+ ),
+ worker_extra_args=parse_json_env("MINERU_ROUTER_WORKER_ARGS_JSON"),
+ task_retention_seconds=get_task_retention_seconds(),
+ task_cleanup_interval_seconds=get_task_cleanup_interval_seconds(),
+ )
+
+
+@dataclass
+class StagedUpload:
+ field_name: str
+ upload_name: str
+ content_type: str
+ path: str
+
+
+@dataclass
+class MultipartPayload:
+ temp_dir: str
+ fields: list[tuple[str, str]]
+ uploads: list[StagedUpload]
+
+ def cleanup(self) -> None:
+ cleanup_path(self.temp_dir)
+
+ def get_field_value(self, name: str) -> Optional[str]:
+ for key, value in reversed(self.fields):
+ if key == name:
+ return value
+ return None
+
+
+@dataclass
+class RouterTaskRecord:
+ task_id: str
+ upstream_server_id: str
+ upstream_task_id: str
+ upstream_base_url: str
+ backend: str
+ file_names: list[str]
+ created_at: str
+ status: str
+ started_at: Optional[str] = None
+ completed_at: Optional[str] = None
+ error: Optional[str] = None
+ queued_ahead: int | None = None
+ upstream_error_count: int = 0
+
+ def to_status_payload(self, request: Request) -> dict[str, Any]:
+ payload = {
+ "task_id": self.task_id,
+ "status": self.status,
+ "backend": self.backend,
+ "file_names": self.file_names,
+ "created_at": self.created_at,
+ "started_at": self.started_at,
+ "completed_at": self.completed_at,
+ "error": self.error,
+ "status_url": str(request.url_for("get_router_task_status", task_id=self.task_id)),
+ "result_url": str(request.url_for("get_router_task_result", task_id=self.task_id)),
+ }
+ if self.queued_ahead is not None:
+ payload["queued_ahead"] = self.queued_ahead
+ return payload
+
+
+@dataclass
+class ManagedLocalServer:
+ server_id: str
+ worker_host: str
+ gpu: str | None
+ enable_vlm_preload: bool
+ extra_cli_args: tuple[str, ...]
+ connect_host: str = field(init=False)
+ base_url: str | None = None
+ process: subprocess.Popen[bytes] | None = None
+ temp_dir: tempfile.TemporaryDirectory[str] | None = None
+
+ def __post_init__(self) -> None:
+ self.connect_host = resolve_connect_host(self.worker_host)
+
+ def is_running(self) -> bool:
+ return self.process is not None and self.process.poll() is None
+
+ async def start(self, client: httpx.AsyncClient) -> None:
+ if self.is_running():
+ return
+
+ self.temp_dir = tempfile.TemporaryDirectory(prefix=f"{self.server_id}-")
+ output_root = Path(self.temp_dir.name) / "output"
+ output_root.mkdir(parents=True, exist_ok=True)
+
+ resolved_port = find_free_port()
+ remaining_cli_args = strip_local_api_network_args(self.extra_cli_args)
+ worker_cli_args = build_local_api_cli_args(
+ remaining_cli_args,
+ enable_vlm_preload=self.enable_vlm_preload,
+ )
+ self.base_url = f"http://{self.connect_host}:{resolved_port}"
+ env = os.environ.copy()
+ env["MINERU_API_OUTPUT_ROOT"] = str(output_root)
+ env["MINERU_API_DISABLE_ACCESS_LOG"] = "1"
+ if self.gpu is not None:
+ env["CUDA_VISIBLE_DEVICES"] = str(self.gpu)
+
+ command = [
+ sys.executable,
+ "-m",
+ "mineru.cli.fast_api",
+ "--host",
+ self.worker_host,
+ "--port",
+ str(resolved_port),
+ *worker_cli_args,
+ ]
+ self.process = subprocess.Popen(
+ command,
+ cwd=os.getcwd(),
+ env=env,
+ **build_managed_process_popen_kwargs(),
+ )
+
+ try:
+ await self.wait_until_ready(client)
+ except Exception:
+ self.stop()
+ raise
+
+ async def wait_until_ready(
+ self,
+ client: httpx.AsyncClient,
+ timeout_seconds: float = LOCAL_API_STARTUP_TIMEOUT_SECONDS,
+ ) -> None:
+ assert self.base_url is not None
+ deadline = asyncio.get_running_loop().time() + timeout_seconds
+ last_error: str | None = None
+ while asyncio.get_running_loop().time() < deadline:
+ if self.process is not None and self.process.poll() is not None:
+ raise RuntimeError(f"Local worker {self.server_id} exited before becoming healthy")
+ try:
+ response = await client.get(f"{self.base_url}{HEALTH_ENDPOINT}")
+ if response.status_code == 200:
+ return
+ last_error = response_detail(response)
+ except httpx.HTTPError as exc:
+ last_error = str(exc)
+ await asyncio.sleep(TASK_STATUS_POLL_INTERVAL_SECONDS)
+
+ message = f"Timed out waiting for local worker {self.server_id} to become healthy"
+ if last_error:
+ message = f"{message}: {last_error}"
+ raise RuntimeError(message)
+
+ async def restart(self, client: httpx.AsyncClient) -> None:
+ self.stop()
+ await self.start(client)
+
+ def stop(self) -> None:
+ process = self.process
+ self.process = None
+ try:
+ if process is not None:
+ stop_managed_process(
+ process,
+ shutdown_timeout_seconds=5,
+ use_stdin_shutdown_watcher=False,
+ )
+ finally:
+ temp_dir = self.temp_dir
+ self.temp_dir = None
+ self.base_url = None
+ if temp_dir is not None:
+ cleanup_temporary_directory(temp_dir)
+
+
+@dataclass
+class WorkerState:
+ server_id: str
+ source: str
+ base_url: str
+ gpu: str | None
+ local_server: ManagedLocalServer | None = None
+ healthy: bool = False
+ queued_tasks: int = 0
+ processing_tasks: int = 0
+ completed_tasks: int = 0
+ failed_tasks: int = 0
+ max_concurrent_requests: int = 0
+ processing_window_size: int = MIN_HEALTHY_PROCESSING_WINDOW_SIZE
+ last_error: Optional[str] = None
+ last_checked_at: Optional[str] = None
+ pending_assignments: int = 0
+ consecutive_health_failures: int = 0
+
+ def score(self) -> float:
+ denominator = max(1, self.max_concurrent_requests)
+ numerator = self.queued_tasks + self.processing_tasks + self.pending_assignments
+ return numerator / denominator
+
+ def snapshot(self) -> dict[str, Any]:
+ return {
+ "server_id": self.server_id,
+ "base_url": self.base_url,
+ "source": self.source,
+ "healthy": self.healthy,
+ "gpu": self.gpu,
+ "queued_tasks": self.queued_tasks,
+ "processing_tasks": self.processing_tasks,
+ "completed_tasks": self.completed_tasks,
+ "failed_tasks": self.failed_tasks,
+ "max_concurrent_requests": self.max_concurrent_requests,
+ "processing_window_size": self.processing_window_size,
+ "last_error": self.last_error,
+ "last_checked_at": self.last_checked_at,
+ }
+
+
+class WorkerPool:
+ def __init__(
+ self,
+ settings: RouterSettings,
+ client: httpx.AsyncClient,
+ *,
+ randomizer: Optional[random.Random] = None,
+ ):
+ self.settings = settings
+ self.client = client
+ self.randomizer = randomizer or random.Random()
+ self._selection_lock = asyncio.Lock()
+ self._monitor_task: asyncio.Task[Any] | None = None
+ self._servers: dict[str, WorkerState] = {}
+ self._build_servers()
+
+ def _build_servers(self) -> None:
+ for index, url in enumerate(dict.fromkeys(normalize_base_url(item) for item in self.settings.upstream_urls), start=1):
+ server_id = f"remote-{index}"
+ self._servers[server_id] = WorkerState(
+ server_id=server_id,
+ source=SOURCE_REMOTE,
+ base_url=url,
+ gpu=None,
+ )
+
+ local_specs = parse_local_gpus(self.settings.local_gpus)
+ for index, gpu in enumerate(local_specs, start=1):
+ if gpu is None:
+ server_id = "local-cpu-1"
+ else:
+ server_id = f"local-gpu-{gpu}"
+ local_server = ManagedLocalServer(
+ server_id=server_id,
+ worker_host=self.settings.worker_host,
+ gpu=gpu,
+ enable_vlm_preload=self.settings.enable_vlm_preload,
+ extra_cli_args=self.settings.worker_extra_args,
+ )
+ self._servers[server_id] = WorkerState(
+ server_id=server_id,
+ source=SOURCE_LOCAL,
+ base_url="",
+ gpu=gpu,
+ local_server=local_server,
+ )
+
+ @property
+ def servers(self) -> list[WorkerState]:
+ return list(self._servers.values())
+
+ async def start(self) -> None:
+ for server in self.servers:
+ if server.local_server is None:
+ continue
+ try:
+ await server.local_server.start(self.client)
+ server.base_url = normalize_base_url(server.local_server.base_url or "")
+ except Exception as exc:
+ server.healthy = False
+ server.last_error = str(exc)
+ server.last_checked_at = utc_now_iso()
+
+ await self.refresh_all()
+ if self._monitor_task is None or self._monitor_task.done():
+ self._monitor_task = asyncio.create_task(self._monitor_loop(), name="mineru-router-worker-monitor")
+
+ async def shutdown(self) -> None:
+ if self._monitor_task is not None:
+ self._monitor_task.cancel()
+ with suppress(asyncio.CancelledError):
+ await self._monitor_task
+ self._monitor_task = None
+ for server in self.servers:
+ if server.local_server is not None:
+ server.local_server.stop()
+
+ async def _monitor_loop(self) -> None:
+ while True:
+ await asyncio.sleep(self.settings.worker_refresh_interval_seconds)
+ await self.refresh_all()
+
+ async def refresh_all(self) -> None:
+ for server in self.servers:
+ await self._refresh_server(server)
+
+ def _update_server_from_health_payload(
+ self,
+ server: WorkerState,
+ payload: dict[str, Any],
+ ) -> None:
+ protocol_version = payload.get("protocol_version")
+ if protocol_version != API_PROTOCOL_VERSION:
+ raise ValueError(
+ f"Unsupported protocol_version={protocol_version}, expected {API_PROTOCOL_VERSION}"
+ )
+
+ server.queued_tasks = int(payload.get("queued_tasks", 0))
+ server.processing_tasks = int(payload.get("processing_tasks", 0))
+ server.completed_tasks = int(payload.get("completed_tasks", 0))
+ server.failed_tasks = int(payload.get("failed_tasks", 0))
+ server.max_concurrent_requests = int(payload.get("max_concurrent_requests", 0))
+ if server.max_concurrent_requests <= 0:
+ raise ValueError("max_concurrent_requests must be a positive integer")
+ server.processing_window_size = max(
+ MIN_HEALTHY_PROCESSING_WINDOW_SIZE,
+ int(
+ payload.get(
+ "processing_window_size",
+ MIN_HEALTHY_PROCESSING_WINDOW_SIZE,
+ )
+ ),
+ )
+ server.healthy = payload.get("status") == "healthy"
+ server.last_error = (
+ None if server.healthy else json.dumps(payload, ensure_ascii=False)
+ )
+ server.consecutive_health_failures = (
+ 0 if server.healthy else server.consecutive_health_failures + 1
+ )
+
+ async def _refresh_server(self, server: WorkerState) -> None:
+ if server.local_server is not None:
+ if not server.local_server.is_running():
+ try:
+ await server.local_server.restart(self.client)
+ server.base_url = normalize_base_url(server.local_server.base_url or "")
+ except Exception as exc:
+ server.healthy = False
+ server.last_error = str(exc)
+ server.last_checked_at = utc_now_iso()
+ server.consecutive_health_failures += 1
+ return
+ elif server.local_server.base_url is not None:
+ server.base_url = normalize_base_url(server.local_server.base_url)
+
+ if not server.base_url:
+ server.healthy = False
+ server.last_error = "Upstream base_url is not configured"
+ server.last_checked_at = utc_now_iso()
+ server.consecutive_health_failures += 1
+ return
+
+ try:
+ response = await self.client.get(f"{server.base_url}{HEALTH_ENDPOINT}")
+ except httpx.HTTPError as exc:
+ server.healthy = False
+ server.last_error = str(exc)
+ server.last_checked_at = utc_now_iso()
+ server.consecutive_health_failures += 1
+ if server.local_server is not None and server.consecutive_health_failures >= 2:
+ await self._restart_local_server(server)
+ return
+
+ server.last_checked_at = utc_now_iso()
+ if response.status_code != 200:
+ server.healthy = False
+ server.last_error = response_detail(response)
+ server.consecutive_health_failures += 1
+ if server.local_server is not None and server.consecutive_health_failures >= 2:
+ await self._restart_local_server(server)
+ return
+
+ try:
+ payload = _parse_json_object_response(response, "health payload")
+ self._update_server_from_health_payload(server, payload)
+ except (TypeError, ValueError) as exc:
+ server.healthy = False
+ server.last_error = f"Invalid health payload: {exc}"
+ server.consecutive_health_failures += 1
+ if server.local_server is not None and server.consecutive_health_failures >= 2:
+ await self._restart_local_server(server)
+ return
+
+ async def _restart_local_server(self, server: WorkerState) -> None:
+ local_server = server.local_server
+ if local_server is None:
+ return
+ try:
+ await local_server.restart(self.client)
+ server.base_url = normalize_base_url(local_server.base_url or "")
+ response = await self.client.get(f"{server.base_url}{HEALTH_ENDPOINT}")
+ if response.status_code == 200:
+ server.last_checked_at = utc_now_iso()
+ payload = _parse_json_object_response(response, "health payload")
+ self._update_server_from_health_payload(server, payload)
+ except (TypeError, ValueError) as exc:
+ server.healthy = False
+ server.last_error = f"Invalid health payload: {exc}"
+ server.last_checked_at = utc_now_iso()
+ except Exception as exc:
+ server.healthy = False
+ server.last_error = str(exc)
+ server.last_checked_at = utc_now_iso()
+
+ async def acquire_submission_server(self, excluded_server_ids: Optional[set[str]] = None) -> WorkerState | None:
+ excluded = excluded_server_ids or set()
+ async with self._selection_lock:
+ candidates = [
+ server
+ for server in self.servers
+ if server.healthy and server.server_id not in excluded
+ ]
+ if not candidates:
+ return None
+
+ randomized = list(candidates)
+ self.randomizer.shuffle(randomized)
+ randomized.sort(
+ key=lambda item: (
+ item.score(),
+ item.pending_assignments,
+ 0 if item.source == SOURCE_LOCAL else 1,
+ )
+ )
+ selected = randomized[0]
+ selected.pending_assignments += 1
+ return selected
+
+ async def release_submission_server(self, server_id: str) -> None:
+ async with self._selection_lock:
+ server = self._servers.get(server_id)
+ if server is None:
+ return
+ if server.pending_assignments > 0:
+ server.pending_assignments -= 1
+
+ async def mark_submission_failure(self, server_id: str, error: str) -> None:
+ async with self._selection_lock:
+ server = self._servers.get(server_id)
+ if server is None:
+ return
+ server.healthy = False
+ server.last_error = error
+ server.last_checked_at = utc_now_iso()
+
+ def get_server(self, server_id: str) -> WorkerState | None:
+ return self._servers.get(server_id)
+
+ def health_payload(self) -> tuple[bool, dict[str, Any]]:
+ servers = [server.snapshot() for server in self.servers]
+ healthy_servers = [server for server in self.servers if server.healthy]
+ payload = {
+ "status": "healthy" if healthy_servers else "unhealthy",
+ "version": __version__,
+ "protocol_version": API_PROTOCOL_VERSION,
+ "queued_tasks": sum(server.queued_tasks for server in self.servers),
+ "processing_tasks": sum(server.processing_tasks for server in self.servers),
+ "completed_tasks": sum(server.completed_tasks for server in self.servers),
+ "failed_tasks": sum(server.failed_tasks for server in self.servers),
+ "max_concurrent_requests": sum(
+ server.max_concurrent_requests for server in healthy_servers
+ ),
+ "processing_window_size": min(
+ (server.processing_window_size for server in healthy_servers),
+ default=MIN_HEALTHY_PROCESSING_WINDOW_SIZE,
+ ),
+ "servers": servers,
+ }
+ if not healthy_servers:
+ payload["error"] = "No healthy upstream MinerU API servers are available"
+ return bool(healthy_servers), payload
+
+
+class RouterTaskRegistry:
+ def __init__(
+ self,
+ *,
+ task_retention_seconds: int,
+ cleanup_interval_seconds: int,
+ ):
+ self.task_retention_seconds = task_retention_seconds
+ self.cleanup_interval_seconds = cleanup_interval_seconds
+ self._tasks: dict[str, RouterTaskRecord] = {}
+ self._lock = asyncio.Lock()
+ self._cleanup_task: asyncio.Task[Any] | None = None
+
+ async def start(self) -> None:
+ if self.task_retention_seconds <= 0:
+ return
+ if self._cleanup_task is None or self._cleanup_task.done():
+ self._cleanup_task = asyncio.create_task(self._cleanup_loop(), name="mineru-router-task-cleanup")
+
+ async def shutdown(self) -> None:
+ if self._cleanup_task is not None:
+ self._cleanup_task.cancel()
+ with suppress(asyncio.CancelledError):
+ await self._cleanup_task
+ self._cleanup_task = None
+
+ async def _cleanup_loop(self) -> None:
+ while True:
+ await asyncio.sleep(self.cleanup_interval_seconds)
+ await self.cleanup_expired_tasks()
+
+ async def register(
+ self,
+ *,
+ upstream_server_id: str,
+ upstream_base_url: str,
+ upstream_task_id: str,
+ backend: str,
+ file_names: list[str],
+ created_at: str,
+ status: str,
+ started_at: Optional[str],
+ completed_at: Optional[str],
+ error: Optional[str],
+ queued_ahead: int | None,
+ ) -> RouterTaskRecord:
+ task = RouterTaskRecord(
+ task_id=str(uuid.uuid4()),
+ upstream_server_id=upstream_server_id,
+ upstream_task_id=upstream_task_id,
+ upstream_base_url=upstream_base_url,
+ backend=backend,
+ file_names=file_names,
+ created_at=created_at,
+ status=status,
+ started_at=started_at,
+ completed_at=completed_at,
+ error=error,
+ queued_ahead=queued_ahead,
+ )
+ async with self._lock:
+ self._tasks[task.task_id] = task
+ return task
+
+ async def get(self, task_id: str) -> RouterTaskRecord | None:
+ async with self._lock:
+ return self._tasks.get(task_id)
+
+ async def update_from_upstream_payload(
+ self,
+ task_id: str,
+ payload: dict[str, Any],
+ ) -> RouterTaskRecord | None:
+ async with self._lock:
+ task = self._tasks.get(task_id)
+ if task is None:
+ return None
+ task.status = str(payload.get("status", task.status))
+ task.backend = str(payload.get("backend", task.backend))
+ file_names = payload.get("file_names")
+ if isinstance(file_names, list) and all(isinstance(item, str) for item in file_names):
+ task.file_names = list(file_names)
+ task.created_at = str(payload.get("created_at", task.created_at))
+ task.started_at = payload.get("started_at") if payload.get("started_at") is None else str(payload.get("started_at"))
+ task.completed_at = payload.get("completed_at") if payload.get("completed_at") is None else str(payload.get("completed_at"))
+ task.error = payload.get("error") if payload.get("error") is None else str(payload.get("error"))
+ queued_ahead = payload.get("queued_ahead")
+ task.queued_ahead = queued_ahead if isinstance(queued_ahead, int) else None
+ task.upstream_error_count = 0
+ return task
+
+ async def increment_upstream_error(
+ self,
+ task_id: str,
+ error: str,
+ ) -> RouterTaskRecord | None:
+ async with self._lock:
+ task = self._tasks.get(task_id)
+ if task is None:
+ return None
+ task.upstream_error_count += 1
+ if task.upstream_error_count >= UPSTREAM_FAILURE_THRESHOLD:
+ task.status = TASK_FAILED
+ task.error = error
+ task.completed_at = utc_now_iso()
+ return task
+
+ async def mark_failed(self, task_id: str, error: str) -> RouterTaskRecord | None:
+ async with self._lock:
+ task = self._tasks.get(task_id)
+ if task is None:
+ return None
+ task.status = TASK_FAILED
+ task.error = error
+ task.completed_at = utc_now_iso()
+ return task
+
+ async def cleanup_expired_tasks(self) -> int:
+ if self.task_retention_seconds <= 0:
+ return 0
+
+ now = datetime.now(timezone.utc)
+ async with self._lock:
+ expired_ids = [
+ task_id
+ for task_id, task in self._tasks.items()
+ if self._is_task_expired(task, now)
+ ]
+ for task_id in expired_ids:
+ self._tasks.pop(task_id, None)
+ return len(expired_ids)
+
+ def _is_task_expired(self, task: RouterTaskRecord, now: datetime) -> bool:
+ if task.status not in TASK_TERMINAL_STATES or not task.completed_at:
+ return False
+ try:
+ completed_at = datetime.fromisoformat(task.completed_at)
+ except ValueError:
+ return False
+ if completed_at.tzinfo is None:
+ completed_at = completed_at.replace(tzinfo=timezone.utc)
+ return (now - completed_at).total_seconds() >= self.task_retention_seconds
+
+
+def warn_if_router_preload_ignored(settings: RouterSettings) -> None:
+ if settings.enable_vlm_preload and not parse_local_gpus(settings.local_gpus):
+ logger.warning(
+ "Ignoring --enable-vlm-preload because mineru-router is not launching any local mineru-api workers."
+ )
+
+
+async def startup_router_state(app: FastAPI, settings: RouterSettings) -> None:
+ http_client = httpx.AsyncClient(
+ timeout=build_http_timeout(),
+ follow_redirects=True,
+ )
+ worker_pool = WorkerPool(settings, http_client)
+ registry = RouterTaskRegistry(
+ task_retention_seconds=settings.task_retention_seconds,
+ cleanup_interval_seconds=settings.task_cleanup_interval_seconds,
+ )
+
+ try:
+ await registry.start()
+ await worker_pool.start()
+ healthy, payload = worker_pool.health_payload()
+ if not healthy:
+ raise RuntimeError(
+ payload.get(
+ "error",
+ "No healthy upstream MinerU API servers are available",
+ )
+ )
+ except Exception:
+ await registry.shutdown()
+ await worker_pool.shutdown()
+ await http_client.aclose()
+ raise
+
+ app.state.http_client = http_client
+ app.state.worker_pool = worker_pool
+ app.state.router_task_registry = registry
+
+
+async def shutdown_router_state(app: FastAPI) -> None:
+ registry = getattr(app.state, "router_task_registry", None)
+ worker_pool = getattr(app.state, "worker_pool", None)
+ http_client = getattr(app.state, "http_client", None)
+
+ if registry is not None:
+ await registry.shutdown()
+ if worker_pool is not None:
+ await worker_pool.shutdown()
+ if http_client is not None:
+ await http_client.aclose()
+
+ app.state.router_task_registry = None
+ app.state.worker_pool = None
+ app.state.http_client = None
+
+
+class UpstreamSubmissionUnavailable(RuntimeError):
+ pass
+
+
+class UpstreamSubmissionRejected(RuntimeError):
+ def __init__(self, status_code: int, detail: str):
+ self.status_code = status_code
+ self.detail = detail
+ super().__init__(detail)
+
+
+def build_upload_destination(upload_dir: str, filename: str) -> Path:
+ destination = Path(upload_dir) / filename
+ if not destination.exists():
+ return destination
+
+ base_name = Path(filename).stem
+ suffix = Path(filename).suffix
+ index = 2
+ while True:
+ candidate = Path(upload_dir) / f"{base_name}__upload_{index}{suffix}"
+ if not candidate.exists():
+ return candidate
+ index += 1
+
+
+async def stage_multipart_request(request: Request) -> MultipartPayload:
+ temp_dir = tempfile.mkdtemp(prefix="mineru-router-request-")
+ uploads: list[StagedUpload] = []
+ fields: list[tuple[str, str]] = []
+
+ try:
+ form = await request.form()
+ for key, value in form.multi_items():
+ if isinstance(value, StarletteUploadFile):
+ original_name = value.filename or f"upload-{uuid.uuid4()}"
+ filename = normalize_upload_filename(original_name)
+ destination = build_upload_destination(temp_dir, filename)
+ with open(destination, "wb") as handle:
+ while True:
+ chunk = await value.read(1 << 20)
+ if not chunk:
+ break
+ handle.write(chunk)
+ uploads.append(
+ StagedUpload(
+ field_name=key,
+ upload_name=original_name,
+ content_type=value.content_type or "application/octet-stream",
+ path=str(destination),
+ )
+ )
+ await value.close()
+ else:
+ fields.append((key, str(value)))
+ except Exception:
+ cleanup_path(temp_dir)
+ raise
+
+ return MultipartPayload(temp_dir=temp_dir, fields=fields, uploads=uploads)
+
+
+def parse_submit_response(payload: Any) -> dict[str, Any]:
+ if not isinstance(payload, dict):
+ raise ValueError("MinerU upstream returned an invalid submit payload")
+ task_id = payload.get("task_id")
+ status = payload.get("status")
+ backend = payload.get("backend")
+ created_at = payload.get("created_at")
+ if not isinstance(task_id, str) or not isinstance(status, str) or not isinstance(backend, str):
+ raise ValueError("MinerU upstream returned an invalid submit payload")
+ if created_at is not None and not isinstance(created_at, str):
+ raise ValueError("MinerU upstream returned an invalid submit payload")
+ return {
+ "task_id": task_id,
+ "status": status,
+ "backend": backend,
+ "file_names": payload.get("file_names", []),
+ "created_at": created_at or utc_now_iso(),
+ "started_at": payload.get("started_at"),
+ "completed_at": payload.get("completed_at"),
+ "error": payload.get("error"),
+ "queued_ahead": payload.get("queued_ahead") if isinstance(payload.get("queued_ahead"), int) else None,
+ }
+
+
+def submit_payload_to_upstream_sync(
+ base_url: str,
+ payload: MultipartPayload,
+) -> dict[str, Any]:
+ with ExitStack() as stack, httpx.Client(
+ timeout=build_http_timeout(),
+ follow_redirects=True,
+ ) as client:
+ multipart = [
+ (
+ field_name,
+ (None, field_value),
+ )
+ for field_name, field_value in payload.fields
+ ]
+ multipart.extend(
+ [
+ (
+ upload.field_name,
+ (
+ upload.upload_name,
+ stack.enter_context(open(upload.path, "rb")),
+ upload.content_type,
+ ),
+ )
+ for upload in payload.uploads
+ ]
+ )
+ try:
+ response = client.post(
+ f"{base_url}{TASKS_ENDPOINT}",
+ files=multipart,
+ )
+ except httpx.HTTPError as exc:
+ raise UpstreamSubmissionUnavailable(str(exc)) from exc
+
+ if response.status_code == 202:
+ try:
+ submit_response = _parse_json_object_response(response, "submit payload")
+ return parse_submit_response(submit_response)
+ except ValueError as exc:
+ raise UpstreamSubmissionUnavailable(f"Invalid submit payload: {exc}") from exc
+ if response.status_code in HTTP_RETRYABLE_STATUS_CODES:
+ raise UpstreamSubmissionUnavailable(
+ f"{response.status_code} {response_detail(response)}"
+ )
+ raise UpstreamSubmissionRejected(response.status_code, response_detail(response))
+
+
+async def submit_payload_to_upstream(
+ base_url: str,
+ payload: MultipartPayload,
+) -> dict[str, Any]:
+ return await asyncio.to_thread(submit_payload_to_upstream_sync, base_url, payload)
+
+
+async def submit_router_task(
+ request: Request,
+ payload: MultipartPayload,
+) -> RouterTaskRecord:
+ validate_public_http_client_request(
+ public_bind_exposed=bool(
+ getattr(request.app.state, "public_bind_exposed", False)
+ ),
+ allow_public_http_client=bool(
+ getattr(request.app.state, "allow_public_http_client", False)
+ ),
+ backend=payload.get_field_value("backend") or "",
+ server_url=payload.get_field_value("server_url"),
+ )
+ worker_pool: WorkerPool = request.app.state.worker_pool
+ registry: RouterTaskRegistry = request.app.state.router_task_registry
+ attempted_servers: set[str] = set()
+ last_error: str | None = None
+
+ while True:
+ server = await worker_pool.acquire_submission_server(excluded_server_ids=attempted_servers)
+ if server is None:
+ if last_error is None:
+ raise HTTPException(status_code=503, detail="No healthy upstream MinerU API servers are available")
+ raise HTTPException(status_code=503, detail=last_error)
+
+ try:
+ upstream_payload = await submit_payload_to_upstream(server.base_url, payload)
+ file_names = upstream_payload["file_names"]
+ normalized_file_names = (
+ list(file_names)
+ if isinstance(file_names, list) and all(isinstance(item, str) for item in file_names)
+ else []
+ )
+ return await registry.register(
+ upstream_server_id=server.server_id,
+ upstream_base_url=server.base_url,
+ upstream_task_id=upstream_payload["task_id"],
+ backend=upstream_payload["backend"],
+ file_names=normalized_file_names,
+ created_at=upstream_payload["created_at"],
+ status=upstream_payload["status"],
+ started_at=upstream_payload["started_at"] if isinstance(upstream_payload["started_at"], str) else None,
+ completed_at=upstream_payload["completed_at"] if isinstance(upstream_payload["completed_at"], str) else None,
+ error=upstream_payload["error"] if isinstance(upstream_payload["error"], str) else None,
+ queued_ahead=upstream_payload["queued_ahead"],
+ )
+ except UpstreamSubmissionRejected as exc:
+ raise HTTPException(status_code=exc.status_code, detail=exc.detail) from exc
+ except UpstreamSubmissionUnavailable as exc:
+ attempted_servers.add(server.server_id)
+ last_error = f"Failed to submit task via {server.server_id}: {exc}"
+ await worker_pool.mark_submission_failure(server.server_id, str(exc))
+ finally:
+ await worker_pool.release_submission_server(server.server_id)
+
+
+async def fetch_router_task_status(
+ request: Request,
+ task: RouterTaskRecord,
+) -> RouterTaskRecord:
+ if is_task_terminal(task.status):
+ return task
+
+ registry: RouterTaskRegistry = request.app.state.router_task_registry
+ client: httpx.AsyncClient = request.app.state.http_client
+ url = f"{task.upstream_base_url}{TASKS_ENDPOINT}/{task.upstream_task_id}"
+ try:
+ response = await client.get(url)
+ except httpx.HTTPError as exc:
+ updated = await registry.increment_upstream_error(task.task_id, str(exc))
+ if updated is None:
+ raise HTTPException(status_code=404, detail="Task not found") from exc
+ return updated
+
+ if response.status_code != 200:
+ error = f"{response.status_code} {response_detail(response)}"
+ updated = await registry.increment_upstream_error(task.task_id, error)
+ if updated is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+ return updated
+
+ try:
+ status_payload = _parse_json_object_response(response, "task status payload")
+ except ValueError as exc:
+ updated = await registry.increment_upstream_error(
+ task.task_id,
+ f"Invalid task status payload: {exc}",
+ )
+ if updated is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+ return updated
+
+ updated = await registry.update_from_upstream_payload(task.task_id, status_payload)
+ if updated is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+ return updated
+
+
+async def wait_for_router_task_terminal_state(
+ request: Request,
+ task: RouterTaskRecord,
+ timeout_seconds: float = TASK_RESULT_TIMEOUT_SECONDS,
+) -> RouterTaskRecord:
+ deadline = asyncio.get_running_loop().time() + timeout_seconds
+ current_task = task
+ while asyncio.get_running_loop().time() < deadline:
+ current_task = await fetch_router_task_status(request, current_task)
+ if is_task_terminal(current_task.status):
+ return current_task
+ await asyncio.sleep(TASK_STATUS_POLL_INTERVAL_SECONDS)
+
+ raise HTTPException(
+ status_code=504,
+ detail=f"Timed out waiting for result of task {task.task_id}",
+ )
+
+
+async def proxy_router_task_result(
+ request: Request,
+ task: RouterTaskRecord,
+) -> Response:
+ client: httpx.AsyncClient = request.app.state.http_client
+ result_url = f"{task.upstream_base_url}{TASKS_ENDPOINT}/{task.upstream_task_id}/result"
+ try:
+ upstream_response = await client.send(
+ client.build_request("GET", result_url),
+ stream=True,
+ )
+ except httpx.HTTPError as exc:
+ await request.app.state.router_task_registry.increment_upstream_error(task.task_id, str(exc))
+ raise HTTPException(status_code=502, detail=str(exc)) from exc
+
+ if upstream_response.status_code != 200:
+ body = await upstream_response.aread()
+ await upstream_response.aclose()
+ detail = body.decode("utf-8", errors="replace").strip() or upstream_response.reason_phrase
+ await request.app.state.router_task_registry.increment_upstream_error(
+ task.task_id,
+ f"{upstream_response.status_code} {detail}",
+ )
+ raise HTTPException(
+ status_code=upstream_response.status_code,
+ detail=detail,
+ )
+
+ content_type = upstream_response.headers.get("content-type", "")
+ if "application/json" in content_type:
+ body = await upstream_response.aread()
+ await upstream_response.aclose()
+ return Response(
+ content=body,
+ status_code=200,
+ media_type="application/json",
+ )
+
+ headers: dict[str, str] = {}
+ content_disposition = upstream_response.headers.get("content-disposition")
+ if content_disposition:
+ headers["content-disposition"] = content_disposition
+ return StreamingResponse(
+ upstream_response.aiter_bytes(),
+ status_code=200,
+ media_type=content_type or "application/octet-stream",
+ headers=headers,
+ background=BackgroundTask(upstream_response.aclose),
+ )
+
+
+async def build_sync_router_task_result_response(
+ request: Request,
+ task: RouterTaskRecord,
+) -> Response:
+ client: httpx.AsyncClient = request.app.state.http_client
+ result_url = f"{task.upstream_base_url}{TASKS_ENDPOINT}/{task.upstream_task_id}/result"
+ try:
+ upstream_response = await client.send(
+ client.build_request("GET", result_url),
+ stream=True,
+ )
+ except httpx.HTTPError as exc:
+ await request.app.state.router_task_registry.increment_upstream_error(task.task_id, str(exc))
+ raise HTTPException(status_code=502, detail=str(exc)) from exc
+
+ if upstream_response.status_code != 200:
+ body = await upstream_response.aread()
+ await upstream_response.aclose()
+ detail = body.decode("utf-8", errors="replace").strip() or upstream_response.reason_phrase
+ await request.app.state.router_task_registry.increment_upstream_error(
+ task.task_id,
+ f"{upstream_response.status_code} {detail}",
+ )
+ raise HTTPException(status_code=upstream_response.status_code, detail=detail)
+
+ sync_headers = build_sync_task_headers(task, request)
+ content_type = upstream_response.headers.get("content-type", "")
+ if "application/json" in content_type:
+ try:
+ await upstream_response.aread()
+ payload_data = _parse_json_object_response(
+ upstream_response,
+ "task result payload",
+ )
+ except ValueError as exc:
+ detail = f"Invalid task result payload: {exc}"
+ await request.app.state.router_task_registry.increment_upstream_error(
+ task.task_id,
+ detail,
+ )
+ raise HTTPException(status_code=502, detail=detail) from exc
+ finally:
+ await upstream_response.aclose()
+
+ merged_payload = {
+ **task.to_status_payload(request),
+ "backend": payload_data.get("backend", task.backend),
+ "version": payload_data.get("version", __version__),
+ "results": payload_data.get("results", {}),
+ }
+ return JSONResponse(status_code=200, content=merged_payload, headers=sync_headers)
+
+ headers: dict[str, str] = {}
+ content_disposition = upstream_response.headers.get("content-disposition")
+ if content_disposition:
+ headers["content-disposition"] = content_disposition
+ return StreamingResponse(
+ upstream_response.aiter_bytes(),
+ status_code=200,
+ media_type=content_type or "application/octet-stream",
+ headers={**headers, **sync_headers},
+ background=BackgroundTask(upstream_response.aclose),
+ )
+
+
+def build_sync_task_headers(task: RouterTaskRecord, request: Request) -> dict[str, str]:
+ payload = task.to_status_payload(request)
+ return {
+ FILE_PARSE_TASK_ID_HEADER: task.task_id,
+ FILE_PARSE_TASK_STATUS_HEADER: task.status,
+ FILE_PARSE_TASK_STATUS_URL_HEADER: payload["status_url"],
+ FILE_PARSE_TASK_RESULT_URL_HEADER: payload["result_url"],
+ }
+
+
+def create_app(settings: RouterSettings | None = None) -> FastAPI:
+ resolved_settings = settings or RouterSettings.from_env()
+ enable_docs = env_flag_enabled("MINERU_API_ENABLE_FASTAPI_DOCS", default=True)
+
+ @asynccontextmanager
+ async def lifespan(app: FastAPI):
+ await startup_router_state(app, resolved_settings)
+ try:
+ yield
+ finally:
+ await shutdown_router_state(app)
+
+ app = FastAPI(
+ openapi_url="/openapi.json" if enable_docs else None,
+ docs_url="/docs" if enable_docs else None,
+ redoc_url="/redoc" if enable_docs else None,
+ lifespan=lifespan,
+ )
+ app.state.router_settings = resolved_settings
+ configure_public_http_client_policy(
+ app,
+ public_bind_exposed=env_flag_enabled(
+ MINERU_ROUTER_PUBLIC_BIND_EXPOSED_ENV,
+ default=False,
+ ),
+ allow_public_http_client=env_flag_enabled(
+ MINERU_ROUTER_ALLOW_PUBLIC_HTTP_CLIENT_ENV,
+ default=False,
+ ),
+ )
+ app.add_middleware(GZipMiddleware, minimum_size=1000)
+
+ @app.post(path="/tasks", status_code=202)
+ async def submit_parse_task(http_request: Request):
+ payload = await stage_multipart_request(http_request)
+ try:
+ router_task = await submit_router_task(http_request, payload)
+ finally:
+ payload.cleanup()
+ response_payload = router_task.to_status_payload(http_request)
+ response_payload["message"] = "Task submitted successfully"
+ return JSONResponse(status_code=202, content=response_payload)
+
+ @app.get(path="/tasks/{task_id}", name="get_router_task_status")
+ async def get_router_task_status(task_id: str, request: Request):
+ registry: RouterTaskRegistry = request.app.state.router_task_registry
+ task = await registry.get(task_id)
+ if task is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+ task = await fetch_router_task_status(request, task)
+ return task.to_status_payload(request)
+
+ @app.get(path="/tasks/{task_id}/result", name="get_router_task_result")
+ async def get_router_task_result(task_id: str, request: Request):
+ registry: RouterTaskRegistry = request.app.state.router_task_registry
+ task = await registry.get(task_id)
+ if task is None:
+ raise HTTPException(status_code=404, detail="Task not found")
+
+ task = await fetch_router_task_status(request, task)
+ if task.status in {TASK_PENDING, TASK_PROCESSING}:
+ return JSONResponse(
+ status_code=202,
+ content={
+ **task.to_status_payload(request),
+ "message": "Task result is not ready yet",
+ },
+ )
+ if task.status == TASK_FAILED:
+ return JSONResponse(
+ status_code=409,
+ content={
+ **task.to_status_payload(request),
+ "message": "Task execution failed",
+ },
+ )
+ return await proxy_router_task_result(request, task)
+
+ @app.post(path="/file_parse", status_code=200)
+ async def file_parse(request: Request):
+ payload = await stage_multipart_request(request)
+ try:
+ router_task = await submit_router_task(request, payload)
+ finally:
+ payload.cleanup()
+
+ router_task = await wait_for_router_task_terminal_state(request, router_task)
+ if router_task.status == TASK_FAILED:
+ return JSONResponse(
+ status_code=409,
+ content={
+ **router_task.to_status_payload(request),
+ "message": "Task execution failed",
+ },
+ )
+
+ return await build_sync_router_task_result_response(request, router_task)
+
+ @app.get(path="/health")
+ async def health_check(request: Request):
+ worker_pool: WorkerPool = request.app.state.worker_pool
+ healthy, payload = worker_pool.health_payload()
+ if healthy:
+ return payload
+ return JSONResponse(status_code=503, content=payload)
+
+ return app
+
+
+app = create_app()
+
+
+@click.command(
+ context_settings=dict(ignore_unknown_options=True, allow_extra_args=True)
+)
+@click.pass_context
+@click.option("--host", default="127.0.0.1", help="Server host (default: 127.0.0.1)")
+@click.option("--port", default=8002, type=int, help="Server port (default: 8002)")
+@click.option("--reload", is_flag=True, help="Enable auto-reload (development mode)")
+@click.option(
+ "--allow-public-http-client",
+ is_flag=True,
+ help=(
+ "Allow *-http-client backends and server_url even when binding the router "
+ "to 0.0.0.0 or ::."
+ ),
+)
+@click.option(
+ "--upstream-url",
+ "upstream_urls",
+ multiple=True,
+ help="Existing MinerU FastAPI base URL. Repeat to add multiple upstream servers.",
+)
+@click.option(
+ "--local-gpus",
+ default=LOCAL_GPU_AUTO,
+ help="Local GPU workers to launch: auto, none, or CSV such as 0,1,2.",
+)
+@click.option(
+ "--worker-host",
+ default="127.0.0.1",
+ help="Host for router-managed mineru-api workers (default: 127.0.0.1).",
+)
+@click.option(
+ "--enable-vlm-preload",
+ "enable_vlm_preload",
+ type=bool,
+ default=False,
+ help="Preload the local VLM model in router-managed mineru-api workers.",
+)
+def main(
+ ctx: click.Context,
+ host: str,
+ port: int,
+ reload: bool,
+ allow_public_http_client: bool,
+ upstream_urls: tuple[str, ...],
+ local_gpus: str,
+ worker_host: str,
+ enable_vlm_preload: bool,
+):
+ settings = RouterSettings(
+ upstream_urls=tuple(upstream_urls),
+ local_gpus=local_gpus,
+ worker_host=worker_host,
+ enable_vlm_preload=enable_vlm_preload,
+ worker_extra_args=tuple(ctx.args),
+ task_retention_seconds=get_task_retention_seconds(),
+ task_cleanup_interval_seconds=get_task_cleanup_interval_seconds(),
+ )
+ public_bind_exposed = is_public_bind_host(host)
+ warn_if_router_preload_ignored(settings)
+ configure_public_http_client_policy(
+ app,
+ public_bind_exposed=public_bind_exposed,
+ allow_public_http_client=allow_public_http_client,
+ )
+ os.environ["MINERU_ROUTER_UPSTREAM_URLS_JSON"] = json.dumps(list(settings.upstream_urls))
+ os.environ["MINERU_ROUTER_LOCAL_GPUS"] = settings.local_gpus
+ os.environ["MINERU_ROUTER_WORKER_HOST"] = settings.worker_host
+ os.environ["MINERU_ROUTER_ENABLE_VLM_PRELOAD"] = (
+ "1" if settings.enable_vlm_preload else "0"
+ )
+ os.environ["MINERU_ROUTER_WORKER_ARGS_JSON"] = json.dumps(list(settings.worker_extra_args))
+ os.environ[MINERU_ROUTER_PUBLIC_BIND_EXPOSED_ENV] = (
+ "1" if public_bind_exposed else "0"
+ )
+ os.environ[MINERU_ROUTER_ALLOW_PUBLIC_HTTP_CLIENT_ENV] = (
+ "1" if allow_public_http_client else "0"
+ )
+ warn_if_public_http_client_policy(host, allow_public_http_client)
+
+ access_log = not env_flag_enabled("MINERU_API_DISABLE_ACCESS_LOG")
+ print(f"Start MinerU Router Service: http://{host}:{port}")
+ print(f"API documentation: http://{host}:{port}/docs")
+
+ if reload:
+ uvicorn.run(
+ "mineru.cli.router:app",
+ host=host,
+ port=port,
+ reload=True,
+ access_log=access_log,
+ )
+ return
+
+ configured_app = create_app(settings)
+ uvicorn.run(
+ configured_app,
+ host=host,
+ port=port,
+ reload=False,
+ access_log=access_log,
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/mineru/cli/visualization.py b/mineru/cli/visualization.py
new file mode 100644
index 000000000..cbb3ac8f8
--- /dev/null
+++ b/mineru/cli/visualization.py
@@ -0,0 +1,90 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+from mineru.utils.draw_bbox import draw_layout_bbox, draw_span_bbox
+
+
+VISUALIZATION_FINISHED = "finished"
+VISUALIZATION_SKIPPED = "skipped"
+
+
+@dataclass(frozen=True)
+class VisualizationJob:
+ document_stem: str
+ backend: str
+ parse_method: str
+ parse_dir: Path
+ draw_span: bool
+
+
+@dataclass(frozen=True)
+class VisualizationResult:
+ document_stem: str
+ parse_dir: Path
+ status: str
+ message: str
+ generated_files: tuple[str, ...] = ()
+
+
+def run_visualization_job(job: VisualizationJob) -> VisualizationResult:
+ middle_json_path = job.parse_dir / f"{job.document_stem}_middle.json"
+ origin_pdf_path = job.parse_dir / f"{job.document_stem}_origin.pdf"
+
+ if not middle_json_path.exists():
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_SKIPPED,
+ message=f"missing middle.json: {middle_json_path.name}",
+ )
+ if not origin_pdf_path.exists():
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_SKIPPED,
+ message=f"missing origin.pdf: {origin_pdf_path.name}",
+ )
+
+ try:
+ payload = json.loads(middle_json_path.read_text(encoding="utf-8"))
+ except Exception as exc:
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_SKIPPED,
+ message=f"failed to read middle.json: {exc}",
+ )
+
+ pdf_info = payload.get("pdf_info")
+ if not isinstance(pdf_info, list):
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_SKIPPED,
+ message="invalid middle.json: missing pdf_info",
+ )
+
+ try:
+ pdf_bytes = origin_pdf_path.read_bytes()
+ generated_files = [f"{job.document_stem}_layout.pdf"]
+ draw_layout_bbox(pdf_info, pdf_bytes, str(job.parse_dir), generated_files[0])
+ if job.draw_span:
+ generated_files.append(f"{job.document_stem}_span.pdf")
+ draw_span_bbox(pdf_info, pdf_bytes, str(job.parse_dir), generated_files[1])
+ except Exception as exc:
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_SKIPPED,
+ message=f"visualization failed: {exc}",
+ )
+
+ return VisualizationResult(
+ document_stem=job.document_stem,
+ parse_dir=job.parse_dir,
+ status=VISUALIZATION_FINISHED,
+ message="generated visualization pdf(s)",
+ generated_files=tuple(generated_files),
+ )
diff --git a/mineru/cli/vlm_preload.py b/mineru/cli/vlm_preload.py
new file mode 100644
index 000000000..6464c42c0
--- /dev/null
+++ b/mineru/cli/vlm_preload.py
@@ -0,0 +1,85 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from loguru import logger
+
+from mineru.utils.engine_utils import get_vlm_engine
+
+SERVICE_CONFIG_DEFAULTS: dict[str, Any] = {
+ "enable_vlm_preload": False,
+}
+
+
+def split_service_and_model_config(
+ config: Mapping[str, Any] | None,
+) -> tuple[dict[str, Any], dict[str, Any]]:
+ raw_config = dict(config or {})
+ service_config: dict[str, Any] = {}
+
+ for key, default in SERVICE_CONFIG_DEFAULTS.items():
+ service_config[key] = bool(raw_config.pop(key, default))
+
+ return service_config, raw_config
+
+
+def build_local_api_cli_args(
+ extra_cli_args: Sequence[str],
+ *,
+ enable_vlm_preload: bool,
+) -> tuple[str, ...]:
+ args = tuple(extra_cli_args)
+ if not enable_vlm_preload:
+ return args
+
+ if "--enable-vlm-preload" in args or any(
+ arg.startswith("--enable-vlm-preload=") for arg in args
+ ):
+ return args
+
+ return args + ("--enable-vlm-preload", "true")
+
+
+def resolve_gradio_local_api_cli_args(
+ extra_cli_args: Sequence[str],
+ *,
+ api_url: str | None,
+ enable_vlm_preload: bool,
+) -> tuple[str, ...]:
+ if enable_vlm_preload and api_url:
+ logger.warning(
+ "Ignoring --enable-vlm-preload because --api-url points to an existing MinerU FastAPI service."
+ )
+ return tuple(extra_cli_args)
+
+ return build_local_api_cli_args(
+ extra_cli_args,
+ enable_vlm_preload=enable_vlm_preload,
+ )
+
+
+def preload_vlm_model(*, model_kwargs: Mapping[str, Any] | None = None) -> str:
+ vlm_engine = get_vlm_engine("auto", is_async=True)
+ logger.info(f"Start init {vlm_engine}...")
+
+ from mineru.backend.vlm.vlm_analyze import ModelSingleton
+
+ model_singleton = ModelSingleton()
+ model_singleton.get_model(
+ vlm_engine,
+ None,
+ None,
+ **dict(model_kwargs or {}),
+ )
+ logger.info(f"{vlm_engine} init successfully.")
+ return vlm_engine
+
+
+def maybe_preload_vlm_model(
+ enable_vlm_preload: bool,
+ *,
+ model_kwargs: Mapping[str, Any] | None = None,
+) -> str | None:
+ if not enable_vlm_preload:
+ return None
+ return preload_vlm_model(model_kwargs=model_kwargs)
diff --git a/mineru/cli/vlm_server.py b/mineru/cli/vlm_server.py
new file mode 100644
index 000000000..0155199f1
--- /dev/null
+++ b/mineru/cli/vlm_server.py
@@ -0,0 +1,62 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import click
+import sys
+
+from loguru import logger
+
+
+def vllm_server():
+ from mineru.model.vlm.vllm_server import main
+ main()
+
+
+def lmdeploy_server():
+ from mineru.model.vlm.lmdeploy_server import main
+ main()
+
+
+@click.command(context_settings=dict(ignore_unknown_options=True, allow_extra_args=True))
+@click.option(
+ '-e',
+ '--engine',
+ 'inference_engine',
+ type=click.Choice(['auto', 'vllm', 'lmdeploy']),
+ default='auto',
+ help='Select the inference engine used to accelerate VLM inference, default is "auto".',
+)
+@click.pass_context
+def openai_server(ctx, inference_engine):
+ sys.argv = [sys.argv[0]] + ctx.args
+ if inference_engine == 'auto':
+ try:
+ import vllm
+ inference_engine = 'vllm'
+ logger.info("Using vLLM as the inference engine for VLM server.")
+ except ImportError:
+ logger.info("vLLM not found, attempting to use LMDeploy as the inference engine for VLM server.")
+ try:
+ import lmdeploy
+ inference_engine = 'lmdeploy'
+ # Success message moved after successful import
+ logger.info("Using LMDeploy as the inference engine for VLM server.")
+ except ImportError:
+ logger.error("Neither vLLM nor LMDeploy is installed. Please install at least one of them.")
+ sys.exit(1)
+
+ if inference_engine == 'vllm':
+ try:
+ import vllm
+ except ImportError:
+ logger.error("vLLM is not installed. Please install vLLM or choose LMDeploy as the inference engine.")
+ sys.exit(1)
+ vllm_server()
+ elif inference_engine == 'lmdeploy':
+ try:
+ import lmdeploy
+ except ImportError:
+ logger.error("LMDeploy is not installed. Please install LMDeploy or choose vLLM as the inference engine.")
+ sys.exit(1)
+ lmdeploy_server()
+
+if __name__ == "__main__":
+ openai_server()
\ No newline at end of file
diff --git a/mineru/data/__init__.py b/mineru/data/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/data/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/data/data_reader_writer/__init__.py b/mineru/data/data_reader_writer/__init__.py
new file mode 100644
index 000000000..9cc042f68
--- /dev/null
+++ b/mineru/data/data_reader_writer/__init__.py
@@ -0,0 +1,18 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from .base import DataReader, DataWriter
+from .dummy import DummyDataWriter
+from .filebase import FileBasedDataReader, FileBasedDataWriter
+from .multi_bucket_s3 import MultiBucketS3DataReader, MultiBucketS3DataWriter
+from .s3 import S3DataReader, S3DataWriter
+
+__all__ = [
+ "DataReader",
+ "DataWriter",
+ "FileBasedDataReader",
+ "FileBasedDataWriter",
+ "S3DataReader",
+ "S3DataWriter",
+ "MultiBucketS3DataReader",
+ "MultiBucketS3DataWriter",
+ "DummyDataWriter",
+]
diff --git a/mineru/data/data_reader_writer/base.py b/mineru/data/data_reader_writer/base.py
new file mode 100644
index 000000000..38b6be9ed
--- /dev/null
+++ b/mineru/data/data_reader_writer/base.py
@@ -0,0 +1,64 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+from abc import ABC, abstractmethod
+
+
+class DataReader(ABC):
+
+ def read(self, path: str) -> bytes:
+ """Read the file.
+
+ Args:
+ path (str): file path to read
+
+ Returns:
+ bytes: the content of the file
+ """
+ return self.read_at(path)
+
+ @abstractmethod
+ def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
+ """Read the file at offset and limit.
+
+ Args:
+ path (str): the file path
+ offset (int, optional): the number of bytes skipped. Defaults to 0.
+ limit (int, optional): the length of bytes want to read. Defaults to -1.
+
+ Returns:
+ bytes: the content of the file
+ """
+ pass
+
+
+class DataWriter(ABC):
+ @abstractmethod
+ def write(self, path: str, data: bytes) -> None:
+ """Write the data to the file.
+
+ Args:
+ path (str): the target file where to write
+ data (bytes): the data want to write
+ """
+ pass
+
+ def write_string(self, path: str, data: str) -> None:
+ """Write the data to file, the data will be encoded to bytes.
+
+ Args:
+ path (str): the target file where to write
+ data (str): the data want to write
+ """
+
+ def safe_encode(data: str, method: str):
+ try:
+ bit_data = data.encode(encoding=method, errors='replace')
+ return bit_data, True
+ except: # noqa
+ return None, False
+
+ for method in ['utf-8', 'ascii']:
+ bit_data, flag = safe_encode(data, method)
+ if flag:
+ self.write(path, bit_data)
+ break
diff --git a/mineru/data/data_reader_writer/dummy.py b/mineru/data/data_reader_writer/dummy.py
new file mode 100644
index 000000000..c79417348
--- /dev/null
+++ b/mineru/data/data_reader_writer/dummy.py
@@ -0,0 +1,12 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from .base import DataWriter
+
+
+class DummyDataWriter(DataWriter):
+ def write(self, path: str, data: bytes) -> None:
+ """Dummy write method that does nothing."""
+ pass
+
+ def write_string(self, path: str, data: str) -> None:
+ """Dummy write_string method that does nothing."""
+ pass
diff --git a/magic_pdf/data/data_reader_writer/filebase.py b/mineru/data/data_reader_writer/filebase.py
similarity index 88%
rename from magic_pdf/data/data_reader_writer/filebase.py
rename to mineru/data/data_reader_writer/filebase.py
index 40e9e4ff1..0ad4f79c8 100644
--- a/magic_pdf/data/data_reader_writer/filebase.py
+++ b/mineru/data/data_reader_writer/filebase.py
@@ -1,6 +1,7 @@
+# Copyright (c) Opendatalab. All rights reserved.
import os
-from magic_pdf.data.data_reader_writer.base import DataReader, DataWriter
+from .base import DataReader, DataWriter
class FileBasedDataReader(DataReader):
@@ -55,5 +56,8 @@ def write(self, path: str, data: bytes) -> None:
if not os.path.isabs(fn_path) and len(self._parent_dir) > 0:
fn_path = os.path.join(self._parent_dir, path)
+ if not os.path.exists(os.path.dirname(fn_path)) and os.path.dirname(fn_path) != "":
+ os.makedirs(os.path.dirname(fn_path), exist_ok=True)
+
with open(fn_path, 'wb') as f:
f.write(data)
diff --git a/mineru/data/data_reader_writer/multi_bucket_s3.py b/mineru/data/data_reader_writer/multi_bucket_s3.py
new file mode 100644
index 000000000..e4e7e7c91
--- /dev/null
+++ b/mineru/data/data_reader_writer/multi_bucket_s3.py
@@ -0,0 +1,145 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+from ..utils.exceptions import InvalidConfig, InvalidParams
+from .base import DataReader, DataWriter
+from ..io.s3 import S3Reader, S3Writer
+from ..utils.schemas import S3Config
+from ..utils.path_utils import parse_s3_range_params, parse_s3path, remove_non_official_s3_args
+
+
+class MultiS3Mixin:
+ def __init__(self, default_prefix: str, s3_configs: list[S3Config]):
+ """Initialized with multiple s3 configs.
+
+ Args:
+ default_prefix (str): the default prefix of the relative path. for example, {some_bucket}/{some_prefix} or {some_bucket}
+ s3_configs (list[S3Config]): list of s3 configs, the bucket_name must be unique in the list.
+
+ Raises:
+ InvalidConfig: default bucket config not in s3_configs.
+ InvalidConfig: bucket name not unique in s3_configs.
+ InvalidConfig: default bucket must be provided.
+ """
+ if len(default_prefix) == 0:
+ raise InvalidConfig('default_prefix must be provided')
+
+ arr = default_prefix.strip('/').split('/')
+ self.default_bucket = arr[0]
+ self.default_prefix = '/'.join(arr[1:])
+
+ found_default_bucket_config = False
+ for conf in s3_configs:
+ if conf.bucket_name == self.default_bucket:
+ found_default_bucket_config = True
+ break
+
+ if not found_default_bucket_config:
+ raise InvalidConfig(
+ f'default_bucket: {self.default_bucket} config must be provided in s3_configs: {s3_configs}'
+ )
+
+ uniq_bucket = set([conf.bucket_name for conf in s3_configs])
+ if len(uniq_bucket) != len(s3_configs):
+ raise InvalidConfig(
+ f'the bucket_name in s3_configs: {s3_configs} must be unique'
+ )
+
+ self.s3_configs = s3_configs
+ self._s3_clients_h: dict = {}
+
+
+class MultiBucketS3DataReader(DataReader, MultiS3Mixin):
+ def read(self, path: str) -> bytes:
+ """Read the path from s3, select diffect bucket client for each request
+ based on the bucket, also support range read.
+
+ Args:
+ path (str): the s3 path of file, the path must be in the format of s3://bucket_name/path?offset,limit.
+ for example: s3://bucket_name/path?0,100.
+
+ Returns:
+ bytes: the content of s3 file.
+ """
+ may_range_params = parse_s3_range_params(path)
+ if may_range_params is None or 2 != len(may_range_params):
+ byte_start, byte_len = 0, -1
+ else:
+ byte_start, byte_len = int(may_range_params[0]), int(may_range_params[1])
+ path = remove_non_official_s3_args(path)
+ return self.read_at(path, byte_start, byte_len)
+
+ def __get_s3_client(self, bucket_name: str):
+ if bucket_name not in set([conf.bucket_name for conf in self.s3_configs]):
+ raise InvalidParams(
+ f'bucket name: {bucket_name} not found in s3_configs: {self.s3_configs}'
+ )
+ if bucket_name not in self._s3_clients_h:
+ conf = next(
+ filter(lambda conf: conf.bucket_name == bucket_name, self.s3_configs)
+ )
+ self._s3_clients_h[bucket_name] = S3Reader(
+ bucket_name,
+ conf.access_key,
+ conf.secret_key,
+ conf.endpoint_url,
+ conf.addressing_style,
+ )
+ return self._s3_clients_h[bucket_name]
+
+ def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
+ """Read the file with offset and limit, select diffect bucket client
+ for each request based on the bucket.
+
+ Args:
+ path (str): the file path.
+ offset (int, optional): the number of bytes skipped. Defaults to 0.
+ limit (int, optional): the number of bytes want to read. Defaults to -1 which means infinite.
+
+ Returns:
+ bytes: the file content.
+ """
+ if path.startswith('s3://'):
+ bucket_name, path = parse_s3path(path)
+ s3_reader = self.__get_s3_client(bucket_name)
+ else:
+ s3_reader = self.__get_s3_client(self.default_bucket)
+ if self.default_prefix:
+ path = self.default_prefix + '/' + path
+ return s3_reader.read_at(path, offset, limit)
+
+
+class MultiBucketS3DataWriter(DataWriter, MultiS3Mixin):
+ def __get_s3_client(self, bucket_name: str):
+ if bucket_name not in set([conf.bucket_name for conf in self.s3_configs]):
+ raise InvalidParams(
+ f'bucket name: {bucket_name} not found in s3_configs: {self.s3_configs}'
+ )
+ if bucket_name not in self._s3_clients_h:
+ conf = next(
+ filter(lambda conf: conf.bucket_name == bucket_name, self.s3_configs)
+ )
+ self._s3_clients_h[bucket_name] = S3Writer(
+ bucket_name,
+ conf.access_key,
+ conf.secret_key,
+ conf.endpoint_url,
+ conf.addressing_style,
+ )
+ return self._s3_clients_h[bucket_name]
+
+ def write(self, path: str, data: bytes) -> None:
+ """Write file with data, also select diffect bucket client for each
+ request based on the bucket.
+
+ Args:
+ path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
+ data (bytes): the data want to write.
+ """
+ if path.startswith('s3://'):
+ bucket_name, path = parse_s3path(path)
+ s3_writer = self.__get_s3_client(bucket_name)
+ else:
+ s3_writer = self.__get_s3_client(self.default_bucket)
+ if self.default_prefix:
+ path = self.default_prefix + '/' + path
+ return s3_writer.write(path, data)
diff --git a/mineru/data/data_reader_writer/s3.py b/mineru/data/data_reader_writer/s3.py
new file mode 100644
index 000000000..57312cbb6
--- /dev/null
+++ b/mineru/data/data_reader_writer/s3.py
@@ -0,0 +1,73 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from .multi_bucket_s3 import MultiBucketS3DataReader, MultiBucketS3DataWriter
+from ..utils.schemas import S3Config
+
+
+class S3DataReader(MultiBucketS3DataReader):
+ def __init__(
+ self,
+ default_prefix_without_bucket: str,
+ bucket: str,
+ ak: str,
+ sk: str,
+ endpoint_url: str,
+ addressing_style: str = 'auto',
+ ):
+ """s3 reader client.
+
+ Args:
+ default_prefix_without_bucket: prefix that not contains bucket
+ bucket (str): bucket name
+ ak (str): access key
+ sk (str): secret key
+ endpoint_url (str): endpoint url of s3
+ addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
+ refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
+ """
+ super().__init__(
+ f'{bucket}/{default_prefix_without_bucket}',
+ [
+ S3Config(
+ bucket_name=bucket,
+ access_key=ak,
+ secret_key=sk,
+ endpoint_url=endpoint_url,
+ addressing_style=addressing_style,
+ )
+ ],
+ )
+
+
+class S3DataWriter(MultiBucketS3DataWriter):
+ def __init__(
+ self,
+ default_prefix_without_bucket: str,
+ bucket: str,
+ ak: str,
+ sk: str,
+ endpoint_url: str,
+ addressing_style: str = 'auto',
+ ):
+ """s3 writer client.
+
+ Args:
+ default_prefix_without_bucket: prefix that not contains bucket
+ bucket (str): bucket name
+ ak (str): access key
+ sk (str): secret key
+ endpoint_url (str): endpoint url of s3
+ addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
+ refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
+ """
+ super().__init__(
+ f'{bucket}/{default_prefix_without_bucket}',
+ [
+ S3Config(
+ bucket_name=bucket,
+ access_key=ak,
+ secret_key=sk,
+ endpoint_url=endpoint_url,
+ addressing_style=addressing_style,
+ )
+ ],
+ )
diff --git a/mineru/data/io/__init__.py b/mineru/data/io/__init__.py
new file mode 100644
index 000000000..43bfe5ee4
--- /dev/null
+++ b/mineru/data/io/__init__.py
@@ -0,0 +1,7 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+from .base import IOReader, IOWriter
+from .http import HttpReader, HttpWriter
+from .s3 import S3Reader, S3Writer
+
+__all__ = ['IOReader', 'IOWriter', 'HttpReader', 'HttpWriter', 'S3Reader', 'S3Writer']
\ No newline at end of file
diff --git a/mineru/data/io/base.py b/mineru/data/io/base.py
new file mode 100644
index 000000000..ede371f01
--- /dev/null
+++ b/mineru/data/io/base.py
@@ -0,0 +1,43 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from abc import ABC, abstractmethod
+
+
+class IOReader(ABC):
+ @abstractmethod
+ def read(self, path: str) -> bytes:
+ """Read the file.
+
+ Args:
+ path (str): file path to read
+
+ Returns:
+ bytes: the content of the file
+ """
+ pass
+
+ @abstractmethod
+ def read_at(self, path: str, offset: int = 0, limit: int = -1) -> bytes:
+ """Read at offset and limit.
+
+ Args:
+ path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
+ offset (int, optional): the number of bytes skipped. Defaults to 0.
+ limit (int, optional): the length of bytes want to read. Defaults to -1.
+
+ Returns:
+ bytes: the content of file
+ """
+ pass
+
+
+class IOWriter(ABC):
+
+ @abstractmethod
+ def write(self, path: str, data: bytes) -> None:
+ """Write file with data.
+
+ Args:
+ path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
+ data (bytes): the data want to write
+ """
+ pass
diff --git a/magic_pdf/data/io/http.py b/mineru/data/io/http.py
similarity index 91%
rename from magic_pdf/data/io/http.py
rename to mineru/data/io/http.py
index 3b08271f0..15340f1bd 100644
--- a/magic_pdf/data/io/http.py
+++ b/mineru/data/io/http.py
@@ -1,9 +1,10 @@
+# Copyright (c) Opendatalab. All rights reserved.
import io
import requests
-from magic_pdf.data.io.base import IOReader, IOWriter
+from .base import IOReader, IOWriter
class HttpReader(IOReader):
diff --git a/mineru/data/io/s3.py b/mineru/data/io/s3.py
new file mode 100644
index 000000000..f1dc94a47
--- /dev/null
+++ b/mineru/data/io/s3.py
@@ -0,0 +1,115 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import boto3
+from botocore.config import Config
+
+from ..io.base import IOReader, IOWriter
+
+
+class S3Reader(IOReader):
+ def __init__(
+ self,
+ bucket: str,
+ ak: str,
+ sk: str,
+ endpoint_url: str,
+ addressing_style: str = 'auto',
+ ):
+ """s3 reader client.
+
+ Args:
+ bucket (str): bucket name
+ ak (str): access key
+ sk (str): secret key
+ endpoint_url (str): endpoint url of s3
+ addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
+ refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
+ """
+ self._bucket = bucket
+ self._ak = ak
+ self._sk = sk
+ self._s3_client = boto3.client(
+ service_name='s3',
+ aws_access_key_id=ak,
+ aws_secret_access_key=sk,
+ endpoint_url=endpoint_url,
+ config=Config(
+ s3={'addressing_style': addressing_style},
+ retries={'max_attempts': 5, 'mode': 'standard'},
+ ),
+ )
+
+ def read(self, key: str) -> bytes:
+ """Read the file.
+
+ Args:
+ path (str): file path to read
+
+ Returns:
+ bytes: the content of the file
+ """
+ return self.read_at(key)
+
+ def read_at(self, key: str, offset: int = 0, limit: int = -1) -> bytes:
+ """Read at offset and limit.
+
+ Args:
+ path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
+ offset (int, optional): the number of bytes skipped. Defaults to 0.
+ limit (int, optional): the length of bytes want to read. Defaults to -1.
+
+ Returns:
+ bytes: the content of file
+ """
+ if limit > -1:
+ range_header = f'bytes={offset}-{offset+limit-1}'
+ res = self._s3_client.get_object(
+ Bucket=self._bucket, Key=key, Range=range_header
+ )
+ else:
+ res = self._s3_client.get_object(
+ Bucket=self._bucket, Key=key, Range=f'bytes={offset}-'
+ )
+ return res['Body'].read()
+
+
+class S3Writer(IOWriter):
+ def __init__(
+ self,
+ bucket: str,
+ ak: str,
+ sk: str,
+ endpoint_url: str,
+ addressing_style: str = 'auto',
+ ):
+ """s3 reader client.
+
+ Args:
+ bucket (str): bucket name
+ ak (str): access key
+ sk (str): secret key
+ endpoint_url (str): endpoint url of s3
+ addressing_style (str, optional): Defaults to 'auto'. Other valid options here are 'path' and 'virtual'
+ refer to https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html
+ """
+ self._bucket = bucket
+ self._ak = ak
+ self._sk = sk
+ self._s3_client = boto3.client(
+ service_name='s3',
+ aws_access_key_id=ak,
+ aws_secret_access_key=sk,
+ endpoint_url=endpoint_url,
+ config=Config(
+ s3={'addressing_style': addressing_style},
+ retries={'max_attempts': 5, 'mode': 'standard'},
+ ),
+ )
+
+ def write(self, key: str, data: bytes):
+ """Write file with data.
+
+ Args:
+ path (str): the path of file, if the path is relative path, it will be joined with parent_dir.
+ data (bytes): the data want to write
+ """
+ self._s3_client.put_object(Bucket=self._bucket, Key=key, Body=data)
diff --git a/mineru/data/utils/__init__.py b/mineru/data/utils/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/data/utils/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/data/utils/exceptions.py b/mineru/data/utils/exceptions.py
new file mode 100644
index 000000000..0b0c52c64
--- /dev/null
+++ b/mineru/data/utils/exceptions.py
@@ -0,0 +1,40 @@
+# Copyright (c) Opendatalab. All rights reserved.
+
+class FileNotExisted(Exception):
+
+ def __init__(self, path):
+ self.path = path
+
+ def __str__(self):
+ return f'File {self.path} does not exist.'
+
+
+class InvalidConfig(Exception):
+ def __init__(self, msg):
+ self.msg = msg
+
+ def __str__(self):
+ return f'Invalid config: {self.msg}'
+
+
+class InvalidParams(Exception):
+ def __init__(self, msg):
+ self.msg = msg
+
+ def __str__(self):
+ return f'Invalid params: {self.msg}'
+
+
+class EmptyData(Exception):
+ def __init__(self, msg):
+ self.msg = msg
+
+ def __str__(self):
+ return f'Empty data: {self.msg}'
+
+class CUDA_NOT_AVAILABLE(Exception):
+ def __init__(self, msg):
+ self.msg = msg
+
+ def __str__(self):
+ return f'CUDA not available: {self.msg}'
\ No newline at end of file
diff --git a/magic_pdf/libs/path_utils.py b/mineru/data/utils/path_utils.py
similarity index 95%
rename from magic_pdf/libs/path_utils.py
rename to mineru/data/utils/path_utils.py
index 15fff01b5..fc321f6f0 100644
--- a/magic_pdf/libs/path_utils.py
+++ b/mineru/data/utils/path_utils.py
@@ -1,3 +1,4 @@
+# Copyright (c) Opendatalab. All rights reserved.
def remove_non_official_s3_args(s3path):
diff --git a/magic_pdf/data/schemas.py b/mineru/data/utils/schemas.py
similarity index 83%
rename from magic_pdf/data/schemas.py
rename to mineru/data/utils/schemas.py
index 3adb6760d..00634735f 100644
--- a/magic_pdf/data/schemas.py
+++ b/mineru/data/utils/schemas.py
@@ -1,8 +1,11 @@
+# Copyright (c) Opendatalab. All rights reserved.
from pydantic import BaseModel, Field
class S3Config(BaseModel):
+ """S3 config
+ """
bucket_name: str = Field(description='s3 bucket name', min_length=1)
access_key: str = Field(description='s3 access key', min_length=1)
secret_key: str = Field(description='s3 secret key', min_length=1)
@@ -11,5 +14,7 @@ class S3Config(BaseModel):
class PageInfo(BaseModel):
+ """The width and height of page
+ """
w: float = Field(description='the width of page')
h: float = Field(description='the height of page')
diff --git a/mineru/model/__init__.py b/mineru/model/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/model/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/model/docx/__init__.py b/mineru/model/docx/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/model/docx/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/model/docx/docx_converter.py b/mineru/model/docx/docx_converter.py
new file mode 100644
index 000000000..4fce717c5
--- /dev/null
+++ b/mineru/model/docx/docx_converter.py
@@ -0,0 +1,2794 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+import zipfile
+from io import BytesIO
+from pathlib import Path
+from typing import BinaryIO, Optional, Union, Any, Final, Iterator
+
+from PIL import Image
+from loguru import logger
+from docx import Document
+from docx.document import Document as DocxDocument
+from docx.oxml.xmlchemy import BaseOxmlElement
+from docx.text.paragraph import Paragraph
+from docx.text.hyperlink import Hyperlink
+from docx.text.run import Run
+from lxml import etree
+from pydantic import AnyUrl
+from mammoth.conversion import convert_document_element_to_html
+from mammoth.docx import body_xml
+
+from mineru.model.docx.tools.office_xml import read_str
+from mineru.model.docx.tools.math.omml import oMath2Latex
+from mineru.utils.docx_formatting import Formatting, Script
+from mineru.utils.enum_class import BlockType, ContentType
+from mineru.backend.utils.office_image import (
+ is_vector_image,
+ serialize_vector_image_with_placeholder,
+)
+from mineru.backend.utils.office_chart import html_table_from_excel_bytes
+from mineru.utils.pdf_reader import image_to_b64str
+
+class DocxConverter:
+ _BLIP_NAMESPACES: Final = {
+ "a": "http://schemas.openxmlformats.org/drawingml/2006/main",
+ "r": "http://schemas.openxmlformats.org/officeDocument/2006/relationships",
+ "w": "http://schemas.openxmlformats.org/wordprocessingml/2006/main",
+ "wp": "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing",
+ "mc": "http://schemas.openxmlformats.org/markup-compatibility/2006",
+ "v": "urn:schemas-microsoft-com:vml",
+ "wps": "http://schemas.microsoft.com/office/word/2010/wordprocessingShape",
+ "w10": "urn:schemas-microsoft-com:office:word",
+ "a14": "http://schemas.microsoft.com/office/drawing/2010/main",
+ "c": "http://schemas.openxmlformats.org/drawingml/2006/chart",
+ }
+ _PARAGRAPH_TRANSPARENT_INLINE_CONTAINERS: Final = {
+ "bdo",
+ "customXml",
+ "dir",
+ "fldSimple",
+ "ins",
+ "moveTo",
+ "smartTag",
+ }
+ """
+ Word 文档中使用的 XML 命名空间映射。
+
+ 这些命名空间用于解析 DOCX 文件中的各种元素，包括：
+ - a: DrawingML 主命名空间
+ - r: Office 文档关系命名空间
+ - w: WordprocessingML 主命名空间
+ - wp: Wordprocessing Drawing 命名空间
+ - mc: 标记兼容性命名空间
+ - v: VML (Vector Markup Language) 命名空间
+ - wps: Wordprocessing Shape 命名空间
+ - w10: Office Word 命名空间
+ - a14: Office 2010 Drawing 命名空间
+ """
+
+ def __init__(self):
+ self.XML_KEY = (
+ "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}val"
+ )
+ self.xml_namespaces = {
+ "w": "http://schemas.microsoft.com/office/word/2003/wordml"
+ }
+ self.blip_xpath_expr = etree.XPath(
+ ".//a:blip", namespaces=DocxConverter._BLIP_NAMESPACES
+ )
+
+ # 存放文档字节数据，用于需要重读 ZIP 的辅助方法
+ self._file_bytes: bytes = b''
+ self.docx_obj = None
+ self.pages = []
+ self.cur_page = []
+ self._mammoth_tables_html: list = [] # 完整文档 mammoth 预解析的表格 HTML 列表
+ self._mammoth_table_idx: int = 0 # 当前预解析表格游标
+ self.pre_num_id: int = -1 # 上一个处理元素的 numId
+ self.pre_ilevel: int = -1 # 上一个处理元素的缩进等级, 用于判断列表层级
+ self.list_block_stack: list = [] # 列表块堆栈
+ self.list_counters: dict[tuple[int, int], int] = (
+ {}
+ ) # 列表计数器 (numId, ilvl) -> count
+ self.index_block_stack: list = [] # 目录索引块堆栈
+ self.pre_index_ilevel: int = -1 # 上一个目录项的缩进等级
+ self.heading_list_numids: set = set() # 用作章节标题的列表numId集合
+ self.equation_bookends: str = "{EQ} " # 公式标记格式
+ self.chart_list = [] # 图表列表
+ self.processed_textbox_elements: list = []
+ self.toc_anchor_set: set[str] = set() # TOC 超链接目标锚点集合
+ self._numbering_root: Optional[BaseOxmlElement] = None
+ self._numbering_root_loaded: bool = False
+ self._numbering_level_cache: dict[
+ tuple[int, int], Optional[BaseOxmlElement]
+ ] = {}
+
+ @staticmethod
+ def _escape_hyperlink_text(text: str) -> str:
+ """
+ 转义超链接文本中的方括号。
+
+ Args:
+ text: 要转义的文本
+
+ Returns:
+ str: 转义后的文本
+ """
+ if not text:
+ return text
+ # 转义方括号
+ text = text.replace("[", "\\[").replace("]", "\\]")
+ return text
+
+ @staticmethod
+ def _escape_hyperlink_url(url: str) -> str:
+ """
+ 转义超链接 URL 中的括号。
+
+ Args:
+ url: 要转义的 URL
+
+ Returns:
+ str: 转义后的 URL
+ """
+ if not url:
+ return url
+ # 对括号进行 URL 编码
+ url = url.replace("(", "%28").replace(")", "%29")
+ return url
+
+ @staticmethod
+ def _get_style_str_from_format(format_obj) -> Optional[str]:
+ """
+ 从 Formatting 对象提取样式字符串。
+
+ Args:
+ format_obj: Formatting 对象
+
+ Returns:
+ Optional[str]: 样式字符串（如 "bold,italic"），无样式时返回 None
+ """
+ if format_obj is None:
+ return None
+ styles = []
+ if format_obj.bold:
+ styles.append('bold')
+ if format_obj.italic:
+ styles.append('italic')
+ if format_obj.underline:
+ styles.append('underline')
+ if format_obj.strikethrough:
+ styles.append('strikethrough')
+ return ','.join(styles) if styles else None
+
+ @staticmethod
+ def _has_visible_style(format_obj) -> bool:
+ """
+ 检查格式是否包含可见样式（下划线或删除线）。
+
+ 空白文本在有这些样式时仍然是可见的，应当保留。
+
+ Args:
+ format_obj: Formatting 对象
+
+ Returns:
+ bool: 是否包含可见样式
+ """
+ if format_obj is None:
+ return False
+ return bool(format_obj.underline or format_obj.strikethrough)
+
+ @staticmethod
+ def _is_hidden_run(run: Run) -> bool:
+ """Check whether a run is marked as hidden text in Word."""
+ _W = "http://schemas.openxmlformats.org/wordprocessingml/2006/main"
+ rpr = run._element.find(f"{{{_W}}}rPr")
+ if rpr is None:
+ return False
+ # webHidden: commonly used by TOC page-number field runs
+ if rpr.find(f"{{{_W}}}webHidden") is not None:
+ return True
+ # vanish: generic hidden text
+ if rpr.find(f"{{{_W}}}vanish") is not None:
+ return True
+ return False
+
+ @classmethod
+ def _format_text_with_hyperlink(
+ cls,
+ text: str,
+ hyperlink: Optional[Union[AnyUrl, Path, str]],
+ style_str: Optional[str] = None,
+ ) -> str:
+ """
+ 将文本和超链接格式化，支持字体样式标记。
+
+ 无超链接时：有样式包裹为 文本 ，无样式直接返回文本。
+ 有超链接时：格式化为 文本 链接 标签
+ if style_str:
+ return f'{text} '
+ return text
+
+ hyperlink_str = str(hyperlink)
+ if not hyperlink_str or hyperlink_str.strip() == "" or hyperlink_str == ".":
+ if style_str:
+ return f'{text} '
+ return text
+
+ # 有超链接：构建 标签（含可选样式）
+ if style_str:
+ text_tag = f'{text} '
+ else:
+ text_tag = f'{text} '
+
+ return f"{text_tag}{hyperlink_str} "
+
+ def _build_text_from_elements(
+ self,
+ paragraph_elements: list[
+ tuple[str, Optional[Formatting], Optional[Union[AnyUrl, Path, str]]]
+ ],
+ ) -> str:
+ """
+ 从 paragraph_elements 重组文本，应用超链接格式和字体样式。
+
+ Args:
+ paragraph_elements: 段落元素列表
+
+ Returns:
+ str: 重组后的文本
+ """
+ result_parts = []
+ for text, format_obj, hyperlink in paragraph_elements:
+ if text:
+ style_str = self._get_style_str_from_format(format_obj)
+ formatted_text = self._format_text_with_hyperlink(text, hyperlink, style_str)
+ result_parts.append(formatted_text)
+ return "".join(result_parts) if result_parts else ""
+
+ @staticmethod
+ def _split_paragraph_elements_at_eq_boundaries(
+ paragraph_elements: list,
+ non_eq_segments: list,
+ ) -> list:
+ """
+ 在公式边界处拆分段落元素，解决格式标注跨公式边界失效的问题。
+
+ 当 _get_paragraph_elements 处理含公式（oMath）的段落时，python-docx 的
+ iter_inner_content() 不会遍历 oMath 元素。如果公式前后的文本格式相同，
+ 它们会被合并为单个元素，导致文本跨越 标签两侧。
+ _replace_text_outside_equations 只在单个非公式片段中搜索，无法找到跨片段的文本，
+ 从而导致样式替换失败。
+
+ 本方法通过将这些跨边界的合并元素重新拆分为多个片段来修复此问题，
+ 使每个元素都对应 text_with_equations 中唯一的非公式片段。
+
+ Args:
+ paragraph_elements: (text, format, hyperlink) 元组的列表
+ non_eq_segments: 从 text_with_equations 中提取的非公式文本片段列表
+
+ Returns:
+ 重新拆分后的 (text, format, hyperlink) 列表，每个元素均位于单个公式片段内
+ """
+ if len(non_eq_segments) <= 1:
+ return paragraph_elements
+
+ # 计算各非公式片段的累积结束位置，作为分割边界
+ boundaries: set[int] = set()
+ pos = 0
+ for seg in non_eq_segments[:-1]: # 最后一个片段后无需分割
+ pos += len(seg)
+ boundaries.add(pos)
+
+ if not boundaries:
+ return paragraph_elements
+
+ # 验证段落元素的拼接文本与非公式片段的拼接文本一致
+ concat_elem_text = "".join(text for text, _, _ in paragraph_elements)
+ concat_seg_text = "".join(non_eq_segments)
+ if concat_elem_text != concat_seg_text:
+ # 文本不匹配时安全降级：原样返回
+ return paragraph_elements
+
+ # 在边界处分割元素
+ result = []
+ text_pos = 0
+ for (text, fmt, hyperlink) in paragraph_elements:
+ if not text:
+ result.append((text, fmt, hyperlink))
+ text_pos += len(text)
+ continue
+
+ elem_start = text_pos
+ elem_end = elem_start + len(text)
+ text_pos = elem_end
+
+ # 找到落在该元素内部的分割点
+ splits_in_elem = sorted(
+ b - elem_start for b in boundaries if elem_start < b < elem_end
+ )
+
+ if not splits_in_elem:
+ result.append((text, fmt, hyperlink))
+ else:
+ prev = 0
+ for split_pos in splits_in_elem:
+ fragment = text[prev:split_pos]
+ if fragment:
+ result.append((fragment, fmt, hyperlink))
+ prev = split_pos
+ fragment = text[prev:]
+ if fragment:
+ result.append((fragment, fmt, hyperlink))
+
+ return result
+
+ def _build_text_with_equations_and_hyperlinks(
+ self,
+ paragraph_elements: list[
+ tuple[str, Optional[Formatting], Optional[Union[AnyUrl, Path, str]]]
+ ],
+ text_with_equations: str,
+ equations: list,
+ ) -> str:
+ """
+ 构建同时包含公式、超链接和字体样式的文本。
+
+ Args:
+ paragraph_elements: 段落元素列表，包含格式和超链接信息
+ text_with_equations: 包含公式标记的原始文本
+ equations: 公式列表
+
+ Returns:
+ str: 包含公式标记、超链接格式和字体样式的文本
+ """
+ if not equations:
+ # 没有公式，直接返回带超链接和样式的文本
+ return self._build_text_from_elements(paragraph_elements)
+
+ # 检查是否有超链接
+ has_hyperlink = any(
+ hyperlink is not None and str(hyperlink).strip() not in ("", ".")
+ for _, _, hyperlink in paragraph_elements
+ )
+
+ # 检查是否有字体样式
+ has_style = any(
+ fmt is not None and (fmt.bold or fmt.italic or fmt.underline or fmt.strikethrough)
+ for _, fmt, _ in paragraph_elements
+ )
+
+ if not has_hyperlink and not has_style:
+ # 没有超链接也没有样式，直接返回带公式的文本
+ return text_with_equations
+
+ # 同时有公式和超链接/样式，需要合并处理
+ # 策略：在带公式的文本基础上，将样式/超链接标记插入到正确的位置
+
+ # 0. 拆分 text_with_equations，获取各非公式片段，用于解决跨公式边界的元素合并问题
+ eq_split_pattern = re.compile(r'.*? ', re.DOTALL)
+ non_eq_segments = eq_split_pattern.split(text_with_equations)
+
+ # 在公式边界处重新拆分段落元素，避免单个元素跨越多个非公式片段
+ paragraph_elements = self._split_paragraph_elements_at_eq_boundaries(
+ paragraph_elements, non_eq_segments
+ )
+
+ # 1. 记录每个元素的原始文本和对应的格式化结果
+ element_mappings = []
+ for text, format_obj, hyperlink in paragraph_elements:
+ if text:
+ style_str = self._get_style_str_from_format(format_obj)
+ formatted_text = self._format_text_with_hyperlink(text, hyperlink, style_str)
+ element_mappings.append((text, formatted_text))
+
+ # 2. 在 text_with_equations 中定位每个元素的原始文本，然后替换为格式化后的文本
+ result_text = text_with_equations
+ for original_text, formatted_text in element_mappings:
+ if original_text != formatted_text:
+ # 只有当文本被格式化（添加样式或超链接）时才需要替换
+ result_text = self._replace_text_outside_equations(
+ result_text, original_text, formatted_text
+ )
+
+ return result_text
+
+ def _replace_text_outside_equations(
+ self, text: str, old_text: str, new_text: str
+ ) -> str:
+ """
+ 在公式标记外替换文本。
+
+ Args:
+ text: 原始文本
+ old_text: 要替换的文本
+ new_text: 替换后的文本
+
+ Returns:
+ str: 替换后的文本
+ """
+ # 分割文本为公式和非公式部分
+ eq_pattern = re.compile(r"(.*? )")
+ parts = eq_pattern.split(text)
+
+ result_parts = []
+ for part in parts:
+ if part.startswith("") and part.endswith(" "):
+ # 公式部分，保持不变
+ result_parts.append(part)
+ else:
+ # 非公式部分，进行替换
+ result_parts.append(part.replace(old_text, new_text, 1))
+
+ return "".join(result_parts)
+
+ def convert(
+ self,
+ file_stream: BinaryIO,
+ ):
+ # 重置所有实例状态，确保同一实例多次调用 convert() 时不会残留上次的数据
+ self.pages = []
+ self.cur_page = []
+ self.pre_num_id = -1
+ self.pre_ilevel = -1
+ self.list_block_stack = []
+ self.list_counters = {}
+ self.index_block_stack = []
+ self.pre_index_ilevel = -1
+ self.heading_list_numids = set()
+ self.chart_list = []
+ self.processed_textbox_elements = []
+ self.toc_anchor_set = set()
+ self._numbering_root = None
+ self._numbering_root_loaded = False
+ self._numbering_level_cache = {}
+
+ # 读取文件字节，以便 mammoth 和 python-docx 各自使用独立读取流
+ file_bytes = file_stream.read()
+ # 保存一份字节副本用于后续需要重新打开 ZIP 的方法
+ self._file_bytes = file_bytes
+ # 使用完整文档 mammoth 转换预解析所有表格，获得完整上下文（编号/图片/样式等）
+ self._mammoth_tables_html = self._preparse_tables_with_mammoth(file_bytes)
+ self._mammoth_table_idx = 0
+ self.docx_obj = Document(BytesIO(file_bytes))
+ self.toc_anchor_set = self._collect_toc_anchor_set()
+ # 预扫描文档，识别用作章节标题的列表numId
+ self.heading_list_numids = self._detect_heading_list_numids()
+ self.pages.append(self.cur_page)
+ self._walk_linear(self.docx_obj.element.body)
+ self._add_header_footer(self.docx_obj)
+ self._add_chart_table()
+
+ def _collect_toc_anchor_set(self) -> set[str]:
+ """Collect TOC hyperlink anchors from the entire document body."""
+ anchor_attr = (
+ "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}anchor"
+ )
+ anchors: set[str] = set()
+ for hl in self.docx_obj.element.body.findall(
+ ".//w:hyperlink", namespaces=DocxConverter._BLIP_NAMESPACES
+ ):
+ anchor = hl.get(anchor_attr, "").strip()
+ if anchor and anchor.startswith("_Toc"):
+ anchors.add(anchor)
+ return anchors
+
+ def _walk_linear(
+ self,
+ body: BaseOxmlElement,
+ ):
+ for element in body:
+ # 获取元素的标签名（去除命名空间前缀）
+ tag_name = etree.QName(element).localname
+ # 检查是否存在内联图像（blip元素）
+ drawing_blip = self.blip_xpath_expr(element)
+
+ # 查找所有绘图元素（用于处理DrawingML）
+ drawingml_els = element.findall(
+ ".//w:drawing", namespaces=DocxConverter._BLIP_NAMESPACES
+ )
+ if drawingml_els:
+ self._handle_drawingml(drawingml_els)
+
+ # 检查文本框内容（支持多种文本框格式）
+ # 仅当该元素之前未被处理时才处理
+ if element not in self.processed_textbox_elements:
+ # 现代 Word 文本框
+ txbx_xpath = etree.XPath(
+ ".//w:txbxContent|.//v:textbox//w:p",
+ namespaces=DocxConverter._BLIP_NAMESPACES,
+ )
+ textbox_elements = txbx_xpath(element)
+
+ # 未找到现代文本框，检查替代/旧版文本框格式
+ if not textbox_elements and tag_name in ["drawing", "pict"]:
+ # 额外检查 DrawingML 和 VML 格式中的文本框
+ alt_txbx_xpath = etree.XPath(
+ ".//wps:txbx//w:p|.//w10:wrap//w:p|.//a:p//a:t",
+ namespaces=DocxConverter._BLIP_NAMESPACES,
+ )
+ textbox_elements = alt_txbx_xpath(element)
+
+ # 检查不在标准文本框内的形状文本
+ if not textbox_elements:
+ shape_text_xpath = etree.XPath(
+ ".//a:bodyPr/ancestor::*//a:t|.//a:txBody//a:t",
+ namespaces=DocxConverter._BLIP_NAMESPACES,
+ )
+ shape_text_elements = shape_text_xpath(element)
+ if shape_text_elements:
+ # 从形状文本创建自定义文本元素
+ text_content = " ".join(
+ [t.text for t in shape_text_elements if t.text]
+ )
+ if text_content.strip():
+ logger.debug(
+ f"Found shape text: {text_content[:50]}..."
+ )
+ self.cur_page.append(
+ {
+ "type": BlockType.TEXT,
+ "content": text_content,
+ }
+ )
+ if textbox_elements:
+ self.processed_textbox_elements.append(element)
+ for tb_element in textbox_elements:
+ self.processed_textbox_elements.append(tb_element)
+
+ logger.debug(
+ f"Found textbox content with {len(textbox_elements)} elements"
+ )
+ self._handle_textbox_content(textbox_elements)
+
+ if tag_name == "tbl":
+ # 表格是顶层块级元素，会中断活跃列表的上下文。
+ # 若不重置列表状态，后续列表项会被追加到表格之前创建的列表块中，
+ # 导致表格在 cur_page 中出现在那些列表项之后，产生顺序错乱。
+ if self.pre_num_id != -1:
+ self.pre_num_id = -1
+ self.pre_ilevel = -1
+ self.list_block_stack = []
+ self.list_counters = {}
+ try:
+ # 处理表格元素
+ self._handle_tables(element)
+ except Exception:
+ # 如果表格解析失败，记录调试信息
+ logger.debug("could not parse a table, broken docx table")
+ # 检查图片元素
+ elif drawing_blip:
+ # 判断图片是否为锚定（浮动）图片
+ is_anchored = bool(
+ element.findall(
+ ".//wp:anchor",
+ namespaces=DocxConverter._BLIP_NAMESPACES,
+ )
+ )
+ # 锚定图片在段落中浮动定位，段落文本应出现在图片之前
+ if is_anchored and tag_name == "p":
+ self._handle_text_elements(element)
+ self._handle_pictures(drawing_blip)
+ else:
+ # 处理图片元素
+ self._handle_pictures(drawing_blip)
+ # 如果是段落元素，同时处理其中的文本内容（如描述性文字）
+ if tag_name == "p":
+ self._handle_text_elements(element)
+ # 检查 sdt 元素
+ elif tag_name == "sdt":
+ sdt_content = element.find(
+ ".//w:sdtContent", namespaces=DocxConverter._BLIP_NAMESPACES
+ )
+ if sdt_content is not None:
+ if self._is_toc_sdt(element):
+ # 处理目录SDT，转换为INDEX块
+ self._handle_sdt_as_index(sdt_content)
+ else:
+ # 其他SDT元素，按普通文本处理
+ paragraphs = sdt_content.findall(
+ ".//w:p", namespaces=DocxConverter._BLIP_NAMESPACES
+ )
+ for p in paragraphs:
+ self._handle_text_elements(p)
+ # 检查文本段落元素
+ elif tag_name == "p":
+ # 处理文本元素（包括段落属性如"tcPr", "sectPr"等）
+ self._handle_text_elements(element)
+
+ # 忽略其他未知元素并记录日志
+ else:
+ logger.debug(f"Ignoring element in DOCX with tag: {tag_name}")
+
+ def _preparse_tables_with_mammoth(self, file_bytes: bytes) -> list:
+ """
+ 使用 mammoth 完整文档转换预解析所有顶层表格的 HTML。
+
+ 孤立模式下（仅传入 XML 片段），mammoth 缺少编号定义
+ （word/numbering.xml）、样式（word/styles.xml）和关系
+ （word/_rels/document.xml.rels）等上下文，在遇到含列表项或图片
+ 的单元格时会抛出 AttributeError。通过完整文档转换，mammoth 可
+ 获得完整上下文，从而正确处理这些情况。
+
+ 图片会被 mammoth 转换为内联 data-URI base64 格式（
）。
+
+ 注意：mammoth 不支持 OMML（Office Math Markup Language）公式，会静默丢弃
+ 表格单元格内的公式。本方法在获取 mammoth HTML 后，会同步遍历原始 DOCX XML，
+ 将丢失的公式重新注入对应的 HTML 单元格。
+
+ Returns:
+ list[str]: 文档中所有顶层表格的 HTML 字符串列表，按文档顺序排列
+ """
+ try:
+ import mammoth as _mammoth
+ from bs4 import BeautifulSoup as _BeautifulSoup
+
+ result = _mammoth.convert_to_html(BytesIO(file_bytes))
+ soup = _BeautifulSoup(result.value, 'html.parser')
+
+ # 仅保留顶层表格，排除嵌套在其他表格单元格内的子表格
+ all_tables = soup.find_all('table')
+ top_level_tables = [t for t in all_tables if not t.find_parent('table')]
+
+ # 同步加载 DOCX XML，获取所有顶层表格元素，用于公式注入
+ docx_obj = Document(BytesIO(file_bytes))
+ xml_top_tables = [
+ elem for elem in docx_obj.element.body
+ if etree.QName(elem).localname == 'tbl'
+ ]
+
+ logger.debug(
+ f"Pre-parsed {len(top_level_tables)} top-level tables via full mammoth conversion"
+ )
+
+ # 将 XML 表格中的 OMML 公式注入到 mammoth HTML 表格中
+ result_tables = []
+ for idx, html_table in enumerate(top_level_tables):
+ if idx < len(xml_top_tables):
+ html_table = self._inject_equations_into_table(
+ html_table, xml_top_tables[idx]
+ )
+ result_tables.append(str(html_table))
+ return result_tables
+ except Exception as e:
+ logger.debug(f"Could not pre-parse tables with full mammoth conversion: {e}")
+ return []
+
+ def _inject_equations_into_table(self, html_table, xml_table):
+ """
+ 将 DOCX XML 表格中的 OMML 公式注入到 mammoth 生成的 HTML 表格中。
+
+ mammoth 会静默丢弃 OMML（Office Math Markup Language）公式，导致含公式
+ 的表格单元格在 HTML 中为空。本方法并行遍历 HTML 表格（BeautifulSoup 对象）
+ 和 XML 表格（lxml 元素），对含有 OMML 公式的单元格用包含公式占位符的内容
+ 替换原来的空内容。
+
+ Args:
+ html_table: BeautifulSoup 的 Tag 对象，代表 mammoth 生成的 元素
+ xml_table: lxml 的 Element 对象，代表原始 DOCX 中对应的
| Site | Percentile | |||||||||
| 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 | |
| Traralgon Ck | P | P,* | P | P | P, | P, | P, | P, | P | P |
| Redhill | P,T | P,T | ，* | ** | P.T | P,* | P* | P* | * | ，* |
| Pine Ck | P,T | P,T | P,T | P,T | T | T | T | na | na | |
| Stewarts Ck 5 | P,T | P,T | P,T | P,T | P.T | P.T | P,T | na | na | na |
| Glendhu 2 | P | P,T | P,* | P,T | P.T | P,ns | P,T | P,T | P,T | P,T |
| Cathedral Peak 2 | P,T | P,T | P,T | P,T | P,T | *,T | P,T | P,T | P,T | T |
| Cathedral Peak 3 | P.T | P.T | P,T | P,T | P,T | T | P,T | P,T | P,T | T |
| Lambrechtsbos A | P,T | P | P | P,T | *,T | *,T | *,T | *,T | *,T | T |
| Lambrechtsbos B | P,T | P,T | P,T | P,T | P,T | P,T | P,T | P,T | T | T |
| Biesievlei | P,T | P.T | P,T | P,T | *,T | *,T | T | T | P,T | P,T |
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +lmdeploy | +||
| 命令行工具(mineru) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| fastapi服务(mineru-api) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| openai-server服务（mineru-openai-server） | +🟢 | +🟢 | +|
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +lmdeploy | +||
| 命令行工具(mineru) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟡 | +🟡 | +|
| <vlm/hybrid>-http-client | +🟡 | +🟢 | +|
| fastapi服务(mineru-api) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🔴 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟡 | +🟢 | +|
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🔴 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟡 | +🟢 | +|
| openai-server服务（mineru-openai-server） | +🟡 | +🟢 | +|
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +lmdeploy | +||
| 命令行工具(mineru) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| fastapi服务(mineru-api) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| openai-server服务（mineru-openai-server） | +🟢 | +🟢 | +|
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🔴 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🔴 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +lmdeploy | +||
| 命令行工具(mineru) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| fastapi服务(mineru-api) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +🟢 | +
| <vlm/hybrid>-auto-engine | +🟢 | +🟢 | +|
| <vlm/hybrid>-http-client | +🟢 | +🟢 | +|
| openai-server服务（mineru-openai-server） | +🟢 | +🟢 | +|
| 使用场景 | +容器环境 | +||
|---|---|---|---|
| vllm | +|||
| 命令行工具(mineru) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| fastapi服务(mineru-api) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| gradio界面(mineru-gradio) | +pipeline | +🟢 | +|
| <vlm/hybrid>-auto-engine | +🟢 | +||
| <vlm/hybrid>-http-client | +🟢 | +||
| openai-server服务（mineru-openai-server） | +🟢 | +||
| 使用场景 | +支持情况 | +|
|---|---|---|
| 命令行工具(mineru) | +pipeline | +🔴 | +
| hybrid-http-client | +🔴 | +|
| hybrid-auto-engine | +🔴 | +|
| vlm-auto-engine | +🟢 | +|
| vlm-http-client | +🟢 | +|
| fastapi服务(mineru-api) | +pipeline | +🔴 | +
| hybrid-http-client | +🔴 | +|
| hybrid-auto-engine | +🔴 | +|
| vlm-auto-engine | +🟢 | +|
| vlm-http-client | +🟢 | +|
| gradio界面(mineru-gradio) | +pipeline | +🔴 | +
| hybrid-http-client | +🔴 | +|
| hybrid-auto-engine | +🔴 | +|
| vlm-auto-engine | +🟢 | +|
| vlm-http-client | +🟢 | +|
| openai-server服务（mineru-openai-server） | +🟢 | +|
\n"
+ f"
"
+ )
+
+
+def _build_visual_body_segments(image_path, content, img_buket_path, span_type, summary_override=''):
+ body_segments = []
+ media_path = _build_media_path(img_buket_path, image_path)
+ if media_path:
+ body_segments.append((f"", 'markdown_line'))
+
+ details_block = _build_visual_details_block(
+ content,
+ span_type,
+ summary_override=summary_override,
+ )
+ if details_block:
+ body_segments.append((details_block, 'html_block'))
+
+ return body_segments
+
+
+def _get_blocks_in_index_order(blocks):
+ return [
+ block
+ for _, block in sorted(
+ enumerate(blocks),
+ key=lambda item: (item[1].get('index', float('inf')), item[0]),
+ )
+ ]
+
+
+def _render_code_block_markdown(block, para_block):
+ code_text = merge_para_with_text(block)
+ if para_block.get('sub_type') == BlockType.CODE:
+ guess_lang = para_block.get('guess_lang', 'txt')
+ return f"```{guess_lang}\n{code_text}\n```"
+ return code_text
+
+
+def _render_visual_block_segments(block, para_block, img_buket_path='', table_enable=True):
+ block_type = block['type']
+
+ if block_type in [
+ BlockType.IMAGE_CAPTION,
+ BlockType.IMAGE_FOOTNOTE,
+ BlockType.TABLE_CAPTION,
+ BlockType.TABLE_FOOTNOTE,
+ BlockType.CODE_CAPTION,
+ BlockType.CODE_FOOTNOTE,
+ BlockType.CHART_CAPTION,
+ BlockType.CHART_FOOTNOTE,
+ ]:
+ block_text = merge_para_with_text(block)
+ if block_text.strip():
+ return [(block_text, 'markdown_line')]
+ return []
+
+ if block_type == BlockType.IMAGE_BODY:
+ rendered_segments = []
+ for line in block.get('lines', []):
+ for span in line.get('spans', []):
+ if span.get('type') != ContentType.IMAGE:
+ continue
+ rendered_segments.extend(
+ _build_visual_body_segments(
+ span.get('image_path', ''),
+ span.get('content', ''),
+ img_buket_path,
+ ContentType.IMAGE,
+ summary_override=para_block.get('sub_type', ''),
+ )
+ )
+ return rendered_segments
+
+ if block_type == BlockType.CHART_BODY:
+ rendered_segments = []
+ for line in block.get('lines', []):
+ for span in line.get('spans', []):
+ if span.get('type') != ContentType.CHART:
+ continue
+ rendered_segments.extend(
+ _build_visual_body_segments(
+ span.get('image_path', ''),
+ span.get('content', ''),
+ img_buket_path,
+ ContentType.CHART,
+ summary_override=para_block.get('sub_type', ''),
+ )
+ )
+ return rendered_segments
+
+ if block_type == BlockType.TABLE_BODY:
+ rendered_segments = []
+ for line in block.get('lines', []):
+ for span in line.get('spans', []):
+ if span.get('type') != ContentType.TABLE:
+ continue
+ if table_enable and span.get('html', ''):
+ rendered_segments.append((
+ _format_embedded_html(span['html'], img_buket_path),
+ 'html_block',
+ ))
+ elif span.get('image_path', ''):
+ rendered_segments.append((
+ f"})",
+ 'markdown_line',
+ ))
+ return rendered_segments
+
+ if block_type == BlockType.CODE_BODY:
+ block_text = _render_code_block_markdown(block, para_block)
+ if block_text.strip():
+ return [(block_text, 'markdown_line')]
+ return []
+
+ return []
+
+
+def _get_visual_block_separator(prev_segment_kind, current_segment_kind):
+ if prev_segment_kind == 'html_block' or current_segment_kind == 'html_block':
+ return '\n\n'
+ return ' \n'
+
+
+def _merge_visual_blocks_to_markdown(para_block, img_buket_path='', table_enable=True):
+ rendered_segments = []
+ for block in _get_blocks_in_index_order(para_block.get('blocks', [])):
+ rendered_segments.extend(
+ _render_visual_block_segments(
+ block,
+ para_block,
+ img_buket_path,
+ table_enable=table_enable,
+ )
+ )
+
+ para_text = ''
+ prev_segment_kind = None
+ for segment_text, segment_kind in rendered_segments:
+ if para_text:
+ para_text += _get_visual_block_separator(prev_segment_kind, segment_kind)
+ para_text += segment_text
+ prev_segment_kind = segment_kind
+
+ return para_text
+
+
+def merge_para_with_text(
+ para_block,
+ formula_enable=True,
+ img_buket_path='',
+ escape_text_block_prefix=True,
+):
+ block_text = ''
+ for line in para_block['lines']:
+ for span in line['spans']:
+ if span['type'] in [ContentType.TEXT]:
+ span['content'] = full_to_half_exclude_marks(span['content'])
+ block_text += span['content']
+ block_lang = detect_lang(block_text)
+ escape_markdown_text = para_block.get('type') != BlockType.CODE_BODY
+
+ para_text = ''
+ for i, line in enumerate(para_block['lines']):
+ for j, span in enumerate(line['spans']):
+ span_type = span['type']
+ content = ''
+ if span_type == ContentType.TEXT:
+ content = span['content']
+ if escape_markdown_text:
+ content = escape_conservative_markdown_text(content)
+ elif span_type == ContentType.INLINE_EQUATION:
+ content = f"{inline_left_delimiter}{span['content']}{inline_right_delimiter}"
+ elif span_type == ContentType.INTERLINE_EQUATION:
+ if formula_enable:
+ content = f"\n{display_left_delimiter}\n{span['content']}\n{display_right_delimiter}\n"
+ else:
+ if span.get('image_path', ''):
+ content = f""
+
+ content = content.strip()
+ if content:
+
+ if span_type == ContentType.INTERLINE_EQUATION:
+ para_text += content
+ continue
+
+ # 定义CJK语言集合(中日韩)
+ cjk_langs = {'zh', 'ja', 'ko'}
+ # logger.info(f'block_lang: {block_lang}, content: {content}')
+
+ # 判断是否为行末span
+ is_last_span = j == len(line['spans']) - 1
+
+ if block_lang in cjk_langs: # 中文/日语/韩文语境下，换行不需要空格分隔,但是如果是行内公式结尾，还是要加空格
+ if is_last_span and span_type != ContentType.INLINE_EQUATION:
+ para_text += content
+ else:
+ para_text += f'{content} '
+ else:
+ # 西方文本语境下 每行的最后一个span判断是否要去除连字符
+ if span_type in [ContentType.TEXT, ContentType.INLINE_EQUATION]:
+ # 如果span是line的最后一个且末尾带有-连字符，那么末尾不应该加空格,同时应该把-删除
+ if (
+ is_last_span
+ and span_type == ContentType.TEXT
+ and is_hyphen_at_line_end(content)
+ ):
+ # 如果下一行的第一个span是小写字母开头，删除连字符
+ if (
+ i+1 < len(para_block['lines'])
+ and para_block['lines'][i + 1].get('spans')
+ and para_block['lines'][i + 1]['spans'][0].get('type') == ContentType.TEXT
+ and para_block['lines'][i + 1]['spans'][0].get('content', '')
+ and para_block['lines'][i + 1]['spans'][0]['content'][0].islower()
+ ):
+ para_text += content[:-1]
+ else: # 如果没有下一行，或者下一行的第一个span不是小写字母开头，则保留连字符但不加空格
+ para_text += content
+ else: # 西方文本语境下 content间需要空格分隔
+ para_text += f'{content} '
+ if escape_text_block_prefix and para_block.get('type') == BlockType.TEXT:
+ para_text = escape_text_block_markdown_prefix(para_text)
+ return para_text
+
+
+def mk_blocks_to_markdown(para_blocks, make_mode, formula_enable, table_enable, img_buket_path=''):
+ page_markdown = []
+ for para_block in para_blocks:
+ para_text = ''
+ para_type = para_block['type']
+ if para_type in [BlockType.TEXT, BlockType.INTERLINE_EQUATION, BlockType.PHONETIC, BlockType.REF_TEXT]:
+ para_text = merge_para_with_text(para_block, formula_enable=formula_enable, img_buket_path=img_buket_path)
+ elif para_type == BlockType.LIST:
+ for block in para_block['blocks']:
+ item_text = merge_para_with_text(
+ block,
+ formula_enable=formula_enable,
+ img_buket_path=img_buket_path,
+ escape_text_block_prefix=False,
+ )
+ para_text += f"{item_text} \n"
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ para_text = f'{"#" * title_level} {merge_para_with_text(para_block)}'
+ elif para_type == BlockType.IMAGE:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ para_text = _merge_visual_blocks_to_markdown(
+ para_block,
+ img_buket_path,
+ table_enable=table_enable,
+ )
+
+ elif para_type == BlockType.TABLE:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ para_text = _merge_visual_blocks_to_markdown(
+ para_block,
+ img_buket_path,
+ table_enable=table_enable,
+ )
+ elif para_type == BlockType.CHART:
+ if make_mode == MakeMode.NLP_MD:
+ continue
+ elif make_mode == MakeMode.MM_MD:
+ para_text = _merge_visual_blocks_to_markdown(
+ para_block,
+ img_buket_path,
+ table_enable=table_enable,
+ )
+ elif para_type == BlockType.CODE:
+ para_text = _merge_visual_blocks_to_markdown(
+ para_block,
+ img_buket_path,
+ table_enable=table_enable,
+ )
+
+ if para_text.strip() == '':
+ continue
+ else:
+ # page_markdown.append(para_text.strip() + ' ')
+ page_markdown.append(para_text.strip())
+
+ return page_markdown
+
+
+def make_blocks_to_content_list(para_block, img_buket_path, page_idx, page_size):
+ para_type = para_block['type']
+ para_content = {}
+ if para_type in [
+ BlockType.TEXT,
+ BlockType.REF_TEXT,
+ BlockType.PHONETIC,
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.PAGE_NUMBER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ para_content = {
+ 'type': para_type,
+ 'text': merge_para_with_text(para_block),
+ }
+ elif para_type == BlockType.LIST:
+ para_content = {
+ 'type': para_type,
+ 'sub_type': para_block.get('sub_type', ''),
+ 'list_items':[],
+ }
+ for block in para_block['blocks']:
+ item_text = merge_para_with_text(block, escape_text_block_prefix=False)
+ if item_text.strip():
+ para_content['list_items'].append(item_text)
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ para_content = {
+ 'type': ContentType.TEXT,
+ 'text': merge_para_with_text(para_block),
+ }
+ if title_level != 0:
+ para_content['text_level'] = title_level
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ para_content = {
+ 'type': ContentType.EQUATION,
+ 'text': merge_para_with_text(para_block),
+ 'text_format': 'latex',
+ }
+ elif para_type == BlockType.IMAGE:
+ para_content = {'type': ContentType.IMAGE, 'img_path': '', BlockType.IMAGE_CAPTION: [], BlockType.IMAGE_FOOTNOTE: []}
+ image_path, _ = get_body_data(para_block)
+ para_content['img_path'] = _build_media_path(img_buket_path, image_path)
+ _apply_visual_sub_type(para_content, para_block)
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ para_content[BlockType.IMAGE_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.IMAGE_FOOTNOTE:
+ para_content[BlockType.IMAGE_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.TABLE:
+ para_content = {'type': ContentType.TABLE, 'img_path': '', BlockType.TABLE_CAPTION: [], BlockType.TABLE_FOOTNOTE: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.TABLE_BODY:
+ for line in block['lines']:
+ for span in line['spans']:
+ if span['type'] == ContentType.TABLE:
+
+ if span.get('html', ''):
+ para_content[BlockType.TABLE_BODY] = _format_embedded_html(
+ span['html'],
+ img_buket_path,
+ )
+
+ if span.get('image_path', ''):
+ para_content['img_path'] = f"{img_buket_path}/{span['image_path']}"
+
+ if block['type'] == BlockType.TABLE_CAPTION:
+ para_content[BlockType.TABLE_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.TABLE_FOOTNOTE:
+ para_content[BlockType.TABLE_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.CHART:
+ image_path, chart_content = get_body_data(para_block)
+ para_content = {
+ 'type': ContentType.CHART,
+ 'img_path': _build_media_path(img_buket_path, image_path),
+ 'content': chart_content if chart_content else '',
+ BlockType.CHART_CAPTION: [],
+ BlockType.CHART_FOOTNOTE: [],
+ }
+ _apply_visual_sub_type(para_content, para_block)
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CHART_CAPTION:
+ para_content[BlockType.CHART_CAPTION].append(merge_para_with_text(block))
+ if block['type'] == BlockType.CHART_FOOTNOTE:
+ para_content[BlockType.CHART_FOOTNOTE].append(merge_para_with_text(block))
+ elif para_type == BlockType.CODE:
+ para_content = {'type': BlockType.CODE, 'sub_type': para_block["sub_type"], BlockType.CODE_CAPTION: []}
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.CODE_BODY:
+ code_text = merge_para_with_text(block)
+ if para_block['sub_type'] == BlockType.CODE:
+ guess_lang = para_block.get("guess_lang", "txt")
+ code_text = f"```{guess_lang}\n{code_text}\n```"
+ para_content[BlockType.CODE_BODY] = code_text
+ if block['type'] == BlockType.CODE_CAPTION:
+ para_content[BlockType.CODE_CAPTION].append(merge_para_with_text(block))
+
+ page_width, page_height = page_size
+ para_bbox = para_block.get('bbox')
+ if para_bbox:
+ x0, y0, x1, y1 = para_bbox
+ para_content['bbox'] = [
+ int(x0 * 1000 / page_width),
+ int(y0 * 1000 / page_height),
+ int(x1 * 1000 / page_width),
+ int(y1 * 1000 / page_height),
+ ]
+
+ para_content['page_idx'] = page_idx
+
+ return para_content
+
+
+def make_blocks_to_content_list_v2(para_block, img_buket_path, page_size):
+ para_type = para_block['type']
+ para_content = {}
+ if para_type in [
+ BlockType.HEADER,
+ BlockType.FOOTER,
+ BlockType.ASIDE_TEXT,
+ BlockType.PAGE_NUMBER,
+ BlockType.PAGE_FOOTNOTE,
+ ]:
+ if para_type == BlockType.HEADER:
+ content_type = ContentTypeV2.PAGE_HEADER
+ elif para_type == BlockType.FOOTER:
+ content_type = ContentTypeV2.PAGE_FOOTER
+ elif para_type == BlockType.ASIDE_TEXT:
+ content_type = ContentTypeV2.PAGE_ASIDE_TEXT
+ elif para_type == BlockType.PAGE_NUMBER:
+ content_type = ContentTypeV2.PAGE_NUMBER
+ elif para_type == BlockType.PAGE_FOOTNOTE:
+ content_type = ContentTypeV2.PAGE_FOOTNOTE
+ else:
+ raise ValueError(f"Unknown para_type: {para_type}")
+ para_content = {
+ 'type': content_type,
+ 'content': {
+ f"{content_type}_content": merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type == BlockType.TITLE:
+ title_level = get_title_level(para_block)
+ if title_level != 0:
+ para_content = {
+ 'type': ContentTypeV2.TITLE,
+ 'content': {
+ "title_content": merge_para_with_text_v2(para_block),
+ "level": title_level
+ }
+ }
+ else:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ "paragraph_content": merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type in [
+ BlockType.TEXT,
+ BlockType.PHONETIC
+ ]:
+ para_content = {
+ 'type': ContentTypeV2.PARAGRAPH,
+ 'content': {
+ 'paragraph_content': merge_para_with_text_v2(para_block),
+ }
+ }
+ elif para_type == BlockType.INTERLINE_EQUATION:
+ image_path, math_content = get_body_data(para_block)
+ para_content = {
+ 'type': ContentTypeV2.EQUATION_INTERLINE,
+ 'content': {
+ 'math_content': math_content,
+ 'math_type': 'latex',
+ 'image_source': {'path': f"{img_buket_path}/{image_path}"},
+ }
+ }
+ elif para_type == BlockType.IMAGE:
+ image_caption = []
+ image_footnote = []
+ image_path, _ = get_body_data(para_block)
+ image_source = {
+ 'path': _build_media_path(img_buket_path, image_path),
+ }
+ for block in para_block['blocks']:
+ if block['type'] == BlockType.IMAGE_CAPTION:
+ image_caption.extend(merge_para_with_text_v2(block))
+ if block['type'] == BlockType.IMAGE_FOOTNOTE:
+ image_footnote.extend(merge_para_with_text_v2(block))
+ para_content = {
+ 'type': ContentTypeV2.IMAGE,
+ 'content': {
+ 'image_source': image_source,
+ 'image_caption': image_caption,
+ 'image_footnote': image_footnote,
+ }
+ }
+ _apply_visual_sub_type(para_content, para_block)
+ elif para_type == BlockType.TABLE:
+ table_caption = []
+ table_footnote = []
+ image_path, html = get_body_data(para_block)
+ table_html = _format_embedded_html(html, img_buket_path)
+ image_source = {
+ 'path': f"{img_buket_path}/{image_path}",
+ }
+ if table_html.count("{summary}
\n\n" + f"{content}\n" + "| ，其他使用 | + tag = "th" if row_idx == 0 else "td" + + # 构建属性字符串 + attrs = [] + if row_span > 1: + attrs.append(f'rowspan="{row_span}"') + if col_span > 1: + attrs.append(f'colspan="{col_span}"') + + attr_str = " " + " ".join(attrs) if attrs else "" + + # 获取单元格文本内容 + cell_text = cell.text.strip() if cell.text else "" + # 转义HTML特殊字符，防止XSS + cell_text = ( + cell_text.replace("&", "&") + .replace("<", "<") + .replace(">", ">") + ) + + html_parts.append(f" <{tag}{attr_str}>{cell_text}") + + html_parts.append(" |
|---|
+ * An enhanced XY-Cut implementation that handles: + *
-
+ *
- Cross-layout elements (headers, footers spanning multiple columns) + *
- Adaptive axis selection based on density ratios + *
- L-shaped region handling + *
+ * This is a simplified geometric implementation without semantic type priorities. + *
+ * Algorithm overview: + *
-
+ *
- Pre-mask: Identify cross-layout elements (width > beta * maxWidth, overlaps >= 2) + *
- Compute density ratio to determine split direction preference + *
- Recursive segmentation with adaptive XY/YX-Cut + *
- Merge cross-layout elements at appropriate positions + *
| " + else: + i, row_start, row_end, col_start, col_end = grid[row][col] + if not cell_box_map.get(i): + continue + if row == row_start and col == col_start: + ocr_rec_text = cell_box_map.get(i) + # text = " | {text} | " + ) + temp += cell_content + + table_html = table_html + temp + "
| ") + + lines.append(" |
") + ) + + @staticmethod + def _get_cell_hyperlink_target(cell) -> str: + hyperlink = getattr(cell, "hyperlink", None) + if not hyperlink: + return "" + + target = getattr(hyperlink, "target", None) + if target: + return str(target) + + location = getattr(hyperlink, "location", None) + if location: + return f"#{location}" + + return "" + + @staticmethod + def _sanitize_hyperlink_target(target: str) -> str: + href = target.strip() + if not href: + return "" + + if href.lower().startswith(("javascript:", "data:", "vbscript:")): + return "" + + parsed = urlparse(href) + allowed_schemes = {"http", "https", "mailto", "ftp"} + scheme = parsed.scheme.lower() if parsed.scheme else "" + if scheme and scheme not in allowed_schemes: + return "" + + return html.escape(href, quote=True) + + @staticmethod + def _apply_inline_font_tags(text_html: str, inline_font) -> str: + if not text_html or inline_font is None: + return text_html + + wrapped = text_html + vert_align = getattr(inline_font, "vertAlign", None) + if vert_align == "superscript": + wrapped = f"{wrapped}" + elif vert_align == "subscript": + wrapped = f"{wrapped}" + + if getattr(inline_font, "strike", False): + wrapped = f"
{content}
" + + def _cell_value_to_html(self, cell) -> tuple[str, bool]: + if cell.value is None: + return "", False + + link_target = self._sanitize_hyperlink_target( + self._get_cell_hyperlink_target(cell) + ) + + if isinstance(cell.value, CellRichText): + html_parts = [] + for part in cell.value: + if hasattr(part, "text"): + part_text = self._escape_text_with_line_breaks( + str(getattr(part, "text", "")) + ) + html_parts.append( + self._apply_inline_font_tags( + part_text, + getattr(part, "font", None), + ) + ) + else: + html_parts.append(self._escape_text_with_line_breaks(str(part))) + + rich_text_html = "".join(html_parts) + if link_target and rich_text_html: + rich_text_html = f'{rich_text_html}' + return rich_text_html, True + + plain_text = str(cell.value) + if link_target and plain_text: + escaped_text = self._escape_text_with_line_breaks(plain_text) + return f'{escaped_text}', True + + return plain_text, False + + def _extract_cell_style(self, cell): + """Extract styles from an openpyxl cell.""" + style = {} + if cell.font: + if cell.font.b: + style["font-weight"] = "bold" + if cell.font.i: + style["font-style"] = "italic" + if cell.font.u: + style["text-decoration"] = "underline" + if cell.font.strike: + style["text-decoration"] = "line-through" + if ( + cell.font.color + and hasattr(cell.font.color, "rgb") + and cell.font.color.rgb + ): + # Color might be ARGB "FF000000" + color = cell.font.color.rgb + if isinstance(color, str) and len(color) == 8: + style["color"] = "#" + color[2:] + elif isinstance(color, str): + style["color"] = "#" + color + + if cell.alignment: + if cell.alignment.horizontal: + style["text-align"] = cell.alignment.horizontal + if cell.alignment.vertical: + style["vertical-align"] = cell.alignment.vertical + + if cell.fill and cell.fill.patternType == "solid" and cell.fill.fgColor: + # handle bg color + color = cell.fill.fgColor.rgb + if ( + hasattr(cell.fill.fgColor, "type") + and cell.fill.fgColor.type == "rgb" + and color + ): + if isinstance(color, str) and len(color) == 8: + style["background-color"] = "#" + color[2:] + return style + + @staticmethod + def _get_sheet_content_layer(sheet: Worksheet): + """根据工作表的可见性返回对应的内容层。 + + 若工作表可见，返回 None（默认层）；否则返回 INVISIBLE 层。 + + 参数： + sheet: 待检查的工作表。 + + 返回： + ContentLayer.INVISIBLE 或 None。 + """ + return ( + None if sheet.sheet_state == Worksheet.SHEETSTATE_VISIBLE else "INVISIBLE" + ) diff --git a/magic_pdf/resources/fasttext-langdetect/lid.176.ftz b/mineru/resources/fasttext-langdetect/lid.176.ftz similarity index 100% rename from magic_pdf/resources/fasttext-langdetect/lid.176.ftz rename to mineru/resources/fasttext-langdetect/lid.176.ftz diff --git a/mineru/resources/header.html b/mineru/resources/header.html new file mode 100644 index 000000000..d7f9be856 --- /dev/null +++ b/mineru/resources/header.html @@ -0,0 +1,141 @@ + + + + + +
+
+
+
+
diff --git a/mineru/utils/__init__.py b/mineru/utils/__init__.py
new file mode 100644
index 000000000..1e17167ce
--- /dev/null
+++ b/mineru/utils/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Opendatalab. All rights reserved.
diff --git a/mineru/utils/bbox_utils.py b/mineru/utils/bbox_utils.py
new file mode 100644
index 000000000..717c65946
--- /dev/null
+++ b/mineru/utils/bbox_utils.py
@@ -0,0 +1,51 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import math
+
+import numpy as np
+
+
+def normalize_to_int_bbox(box, image_size: tuple[int, int] | None = None) -> list[int] | None:
+ if box is None:
+ return None
+
+ arr = np.asarray(box, dtype=np.float64)
+ if arr.size == 0:
+ return None
+
+ if arr.ndim == 2 and arr.shape[-1] == 2:
+ xs = arr[:, 0]
+ ys = arr[:, 1]
+ xmin = float(np.min(xs))
+ ymin = float(np.min(ys))
+ xmax = float(np.max(xs))
+ ymax = float(np.max(ys))
+ else:
+ flat = arr.reshape(-1)
+ if flat.size == 4:
+ xmin, ymin, xmax, ymax = [float(v) for v in flat]
+ elif flat.size >= 8:
+ xs = flat[0::2]
+ ys = flat[1::2]
+ xmin = float(np.min(xs))
+ ymin = float(np.min(ys))
+ xmax = float(np.max(xs))
+ ymax = float(np.max(ys))
+ else:
+ return None
+
+ xmin = math.floor(xmin)
+ ymin = math.floor(ymin)
+ xmax = math.ceil(xmax)
+ ymax = math.ceil(ymax)
+
+ if image_size is not None:
+ height, width = image_size
+ xmin = max(0, min(int(width), xmin))
+ ymin = max(0, min(int(height), ymin))
+ xmax = max(0, min(int(width), xmax))
+ ymax = max(0, min(int(height), ymax))
+
+ if xmax <= xmin or ymax <= ymin:
+ return None
+
+ return [int(xmin), int(ymin), int(xmax), int(ymax)]
diff --git a/mineru/utils/boxbase.py b/mineru/utils/boxbase.py
new file mode 100644
index 000000000..ea96c5260
--- /dev/null
+++ b/mineru/utils/boxbase.py
@@ -0,0 +1,227 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import math
+
+
+def is_in(box1, box2) -> bool:
+ """box1是否完全在box2里面."""
+ x0_1, y0_1, x1_1, y1_1 = box1
+ x0_2, y0_2, x1_2, y1_2 = box2
+
+ return (
+ x0_1 >= x0_2 # box1的左边界不在box2的左边外
+ and y0_1 >= y0_2 # box1的上边界不在box2的上边外
+ and x1_1 <= x1_2 # box1的右边界不在box2的右边外
+ and y1_1 <= y1_2
+ ) # box1的下边界不在box2的下边外
+
+
+def bbox_relative_pos(bbox1, bbox2):
+ """判断两个矩形框的相对位置关系.
+
+ Args:
+ bbox1: 一个四元组，表示第一个矩形框的左上角和右下角的坐标，格式为(x1, y1, x1b, y1b)
+ bbox2: 一个四元组，表示第二个矩形框的左上角和右下角的坐标，格式为(x2, y2, x2b, y2b)
+
+ Returns:
+ 一个四元组，表示矩形框1相对于矩形框2的位置关系，格式为(left, right, bottom, top)
+ 其中，left表示矩形框1是否在矩形框2的左侧，right表示矩形框1是否在矩形框2的右侧，
+ bottom表示矩形框1是否在矩形框2的下方，top表示矩形框1是否在矩形框2的上方
+ """
+ x1, y1, x1b, y1b = bbox1
+ x2, y2, x2b, y2b = bbox2
+
+ left = x2b < x1
+ right = x1b < x2
+ bottom = y2b < y1
+ top = y1b < y2
+ return left, right, bottom, top
+
+
+def bbox_distance(bbox1, bbox2):
+ """计算两个矩形框的距离。
+
+ Args:
+ bbox1 (tuple): 第一个矩形框的坐标，格式为 (x1, y1, x2, y2)，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
+ bbox2 (tuple): 第二个矩形框的坐标，格式为 (x1, y1, x2, y2)，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
+
+ Returns:
+ float: 矩形框之间的距离。
+ """
+
+ def dist(point1, point2):
+ return math.sqrt((point1[0] - point2[0]) ** 2 + (point1[1] - point2[1]) ** 2)
+
+ x1, y1, x1b, y1b = bbox1
+ x2, y2, x2b, y2b = bbox2
+
+ left, right, bottom, top = bbox_relative_pos(bbox1, bbox2)
+
+ if top and left:
+ return dist((x1, y1b), (x2b, y2))
+ elif left and bottom:
+ return dist((x1, y1), (x2b, y2b))
+ elif bottom and right:
+ return dist((x1b, y1), (x2, y2b))
+ elif right and top:
+ return dist((x1b, y1b), (x2, y2))
+ elif left:
+ return x1 - x2b
+ elif right:
+ return x2 - x1b
+ elif bottom:
+ return y1 - y2b
+ elif top:
+ return y2 - y1b
+ return 0.0
+
+
+def bbox_center_distance(bbox1, bbox2):
+ """计算两个矩形框中心点之间的欧氏距离。
+
+ Args:
+ bbox1 (tuple): 第一个矩形框的坐标，格式为 (x1, y1, x2, y2)
+ bbox2 (tuple): 第二个矩形框的坐标，格式为 (x1, y1, x2, y2)
+
+ Returns:
+ float: 两个矩形框中心点之间的距离
+ """
+ x1, y1, x1b, y1b = bbox1
+ x2, y2, x2b, y2b = bbox2
+
+ # 计算中心点
+ center1_x = (x1 + x1b) / 2
+ center1_y = (y1 + y1b) / 2
+ center2_x = (x2 + x2b) / 2
+ center2_y = (y2 + y2b) / 2
+
+ # 计算欧氏距离
+ return math.sqrt((center1_x - center2_x) ** 2 + (center1_y - center2_y) ** 2)
+
+
+def get_minbox_if_overlap_by_ratio(bbox1, bbox2, ratio):
+ """通过calculate_overlap_area_2_minbox_area_ratio计算两个bbox重叠的面积占最小面积的box的比例
+ 如果比例大于ratio，则返回小的那个bbox, 否则返回None."""
+ x1_min, y1_min, x1_max, y1_max = bbox1
+ x2_min, y2_min, x2_max, y2_max = bbox2
+ area1 = (x1_max - x1_min) * (y1_max - y1_min)
+ area2 = (x2_max - x2_min) * (y2_max - y2_min)
+ overlap_ratio = calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2)
+ if overlap_ratio > ratio:
+ if area1 <= area2:
+ return bbox1
+ else:
+ return bbox2
+ else:
+ return None
+
+
+def calculate_overlap_area_2_minbox_area_ratio(bbox1, bbox2):
+ """计算box1和box2的重叠面积占最小面积的box的比例."""
+ # Determine the coordinates of the intersection rectangle
+ x_left = max(bbox1[0], bbox2[0])
+ y_top = max(bbox1[1], bbox2[1])
+ x_right = min(bbox1[2], bbox2[2])
+ y_bottom = min(bbox1[3], bbox2[3])
+
+ if x_right < x_left or y_bottom < y_top:
+ return 0.0
+
+ # The area of overlap area
+ intersection_area = (x_right - x_left) * (y_bottom - y_top)
+ min_box_area = min([(bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1]),
+ (bbox2[3] - bbox2[1]) * (bbox2[2] - bbox2[0])])
+ if min_box_area == 0:
+ return 0
+ else:
+ return intersection_area / min_box_area
+
+
+def calculate_iou(bbox1, bbox2):
+ """计算两个边界框的交并比(IOU)。
+
+ Args:
+ bbox1 (list[float]): 第一个边界框的坐标，格式为 [x1, y1, x2, y2]，其中 (x1, y1) 为左上角坐标，(x2, y2) 为右下角坐标。
+ bbox2 (list[float]): 第二个边界框的坐标，格式与 `bbox1` 相同。
+
+ Returns:
+ float: 两个边界框的交并比(IOU)，取值范围为 [0, 1]。
+ """
+ # Determine the coordinates of the intersection rectangle
+ x_left = max(bbox1[0], bbox2[0])
+ y_top = max(bbox1[1], bbox2[1])
+ x_right = min(bbox1[2], bbox2[2])
+ y_bottom = min(bbox1[3], bbox2[3])
+
+ if x_right < x_left or y_bottom < y_top:
+ return 0.0
+
+ # The area of overlap area
+ intersection_area = (x_right - x_left) * (y_bottom - y_top)
+
+ # The area of both rectangles
+ bbox1_area = (bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1])
+ bbox2_area = (bbox2[2] - bbox2[0]) * (bbox2[3] - bbox2[1])
+
+ if any([bbox1_area == 0, bbox2_area == 0]):
+ return 0
+
+ # Compute the intersection over union by taking the intersection area
+ # and dividing it by the sum of both areas minus the intersection area
+ iou = intersection_area / float(bbox1_area + bbox2_area - intersection_area)
+
+ return iou
+
+
+def calculate_overlap_area_in_bbox1_area_ratio(bbox1, bbox2):
+ """计算box1和box2的重叠面积占bbox1的比例."""
+ # Determine the coordinates of the intersection rectangle
+ x_left = max(bbox1[0], bbox2[0])
+ y_top = max(bbox1[1], bbox2[1])
+ x_right = min(bbox1[2], bbox2[2])
+ y_bottom = min(bbox1[3], bbox2[3])
+
+ if x_right < x_left or y_bottom < y_top:
+ return 0.0
+
+ # The area of overlap area
+ intersection_area = (x_right - x_left) * (y_bottom - y_top)
+ bbox1_area = (bbox1[2] - bbox1[0]) * (bbox1[3] - bbox1[1])
+ if bbox1_area == 0:
+ return 0
+ else:
+ return intersection_area / bbox1_area
+
+
+def calculate_vertical_projection_overlap_ratio(block1, block2):
+ """
+ Calculate the proportion of the x-axis covered by the vertical projection of two blocks.
+
+ Args:
+ block1 (tuple): Coordinates of the first block (x0, y0, x1, y1).
+ block2 (tuple): Coordinates of the second block (x0, y0, x1, y1).
+
+ Returns:
+ float: The proportion of the x-axis covered by the vertical projection of the two blocks.
+ """
+ x0_1, _, x1_1, _ = block1
+ x0_2, _, x1_2, _ = block2
+
+ # Calculate the intersection of the x-coordinates
+ x_left = max(x0_1, x0_2)
+ x_right = min(x1_1, x1_2)
+
+ if x_right < x_left:
+ return 0.0
+
+ # Length of the intersection
+ intersection_length = x_right - x_left
+
+ # Length of the x-axis projection of the first block
+ block1_length = x1_1 - x0_1
+
+ if block1_length == 0:
+ return 0.0
+
+ # Proportion of the x-axis covered by the intersection
+ # logger.info(f"intersection_length: {intersection_length}, block1_length: {block1_length}")
+ return intersection_length / block1_length
\ No newline at end of file
diff --git a/mineru/utils/char_utils.py b/mineru/utils/char_utils.py
new file mode 100644
index 000000000..c6a752450
--- /dev/null
+++ b/mineru/utils/char_utils.py
@@ -0,0 +1,55 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import re
+
+
+def is_hyphen_at_line_end(line):
+ """Check if a line ends with one or more letters followed by a hyphen.
+
+ Args:
+ line (str): The line of text to check.
+
+ Returns:
+ bool: True if the line ends with one or more letters followed by a hyphen, False otherwise.
+ """
+ # Use regex to check if the line ends with one or more letters followed by a hyphen
+ return bool(re.search(r'[A-Za-z]+-\s*$', line))
+
+
+def full_to_half_exclude_marks(text: str) -> str:
+ """Convert full-width characters to half-width characters using code point manipulation.
+
+ Args:
+ text: String containing full-width characters
+
+ Returns:
+ String with full-width characters converted to half-width
+ """
+ result = []
+ for char in text:
+ code = ord(char)
+ # Full-width letters and numbers (FF21-FF3A for A-Z, FF41-FF5A for a-z, FF10-FF19 for 0-9)
+ if (0xFF21 <= code <= 0xFF3A) or (0xFF41 <= code <= 0xFF5A) or (0xFF10 <= code <= 0xFF19):
+ result.append(chr(code - 0xFEE0)) # Shift to ASCII range
+ else:
+ result.append(char)
+ return ''.join(result)
+
+
+def full_to_half(text: str) -> str:
+ """Convert full-width characters to half-width characters using code point manipulation.
+
+ Args:
+ text: String containing full-width characters
+
+ Returns:
+ String with full-width characters converted to half-width
+ """
+ result = []
+ for char in text:
+ code = ord(char)
+ # Full-width letters, numbers and punctuation (FF01-FF5E)
+ if 0xFF01 <= code <= 0xFF5E:
+ result.append(chr(code - 0xFEE0)) # Shift to ASCII range
+ else:
+ result.append(char)
+ return ''.join(result)
diff --git a/mineru/utils/check_sys_env.py b/mineru/utils/check_sys_env.py
new file mode 100644
index 000000000..02b56bc8d
--- /dev/null
+++ b/mineru/utils/check_sys_env.py
@@ -0,0 +1,38 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import platform
+
+from packaging import version
+
+
+def is_windows_environment() -> bool:
+ return platform.system() == "Windows"
+
+
+# Detect if the current environment is a Mac computer
+def is_mac_environment() -> bool:
+ return platform.system() == "Darwin"
+
+
+def is_linux_environment() -> bool:
+ return platform.system() == "Linux"
+
+
+# Detect if CPU is Apple Silicon architecture
+def is_apple_silicon_cpu() -> bool:
+ return platform.machine() in ["arm64", "aarch64"]
+
+
+# If Mac computer with Apple Silicon architecture, check if macOS version is 13.5 or above
+def is_mac_os_version_supported(min_version: str = "13.5") -> bool:
+ if not is_mac_environment() or not is_apple_silicon_cpu():
+ return False
+ mac_version = platform.mac_ver()[0]
+ if not mac_version:
+ return False
+ # print("Mac OS Version:", mac_version)
+ return version.parse(mac_version) >= version.parse(min_version)
+
+if __name__ == "__main__":
+ print("Is Mac Environment:", is_mac_environment())
+ print("Is Apple Silicon CPU:", is_apple_silicon_cpu())
+ print("Is Mac OS Version Supported (>=13.5):", is_mac_os_version_supported())
\ No newline at end of file
diff --git a/mineru/utils/cli_parser.py b/mineru/utils/cli_parser.py
new file mode 100644
index 000000000..2278a0014
--- /dev/null
+++ b/mineru/utils/cli_parser.py
@@ -0,0 +1,56 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from collections.abc import Sequence
+
+import click
+
+
+def _coerce_cli_value(raw_value: str) -> bool | float | int | str:
+ lowered = raw_value.lower()
+ if lowered == "true":
+ return True
+ if lowered == "false":
+ return False
+
+ try:
+ return int(raw_value)
+ except ValueError:
+ pass
+
+ try:
+ return float(raw_value)
+ except ValueError:
+ return raw_value
+
+
+def parse_unknown_args(args: Sequence[str]) -> dict:
+ """Parse unknown click args into keyword arguments."""
+ extra_kwargs = {}
+ i = 0
+ while i < len(args):
+ arg = args[i]
+ if not arg.startswith("--"):
+ i += 1
+ continue
+
+ raw_option = arg[2:]
+ if "=" in raw_option:
+ param_name, raw_value = raw_option.split("=", 1)
+ extra_kwargs[param_name.replace("-", "_")] = _coerce_cli_value(raw_value)
+ i += 1
+ continue
+
+ param_name = raw_option.replace("-", "_")
+ next_index = i + 1
+ if next_index < len(args) and not args[next_index].startswith("--"):
+ extra_kwargs[param_name] = _coerce_cli_value(args[next_index])
+ i += 2
+ continue
+
+ extra_kwargs[param_name] = True
+ i += 1
+
+ return extra_kwargs
+
+
+def arg_parse(ctx: "click.Context") -> dict:
+ return parse_unknown_args(ctx.args)
diff --git a/mineru/utils/config_reader.py b/mineru/utils/config_reader.py
new file mode 100644
index 000000000..a04d882e7
--- /dev/null
+++ b/mineru/utils/config_reader.py
@@ -0,0 +1,196 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import json
+import os
+from loguru import logger
+
+try:
+ import torch
+ import torch_npu
+except ImportError:
+ pass
+
+
+# 定义配置文件名常量
+CONFIG_FILE_NAME = os.getenv('MINERU_TOOLS_CONFIG_JSON', 'mineru.json')
+
+
+def read_config():
+ if os.path.isabs(CONFIG_FILE_NAME):
+ config_file = CONFIG_FILE_NAME
+ else:
+ home_dir = os.path.expanduser('~')
+ config_file = os.path.join(home_dir, CONFIG_FILE_NAME)
+
+ if not os.path.exists(config_file):
+ # logger.warning(f'{config_file} not found, using default configuration')
+ return None
+ else:
+ with open(config_file, 'r', encoding='utf-8') as f:
+ config = json.load(f)
+ return config
+
+
+def get_s3_config(bucket_name: str):
+ """~/magic-pdf.json 读出来."""
+ config = read_config()
+
+ bucket_info = config.get('bucket_info')
+ if bucket_name not in bucket_info:
+ access_key, secret_key, storage_endpoint = bucket_info['[default]']
+ else:
+ access_key, secret_key, storage_endpoint = bucket_info[bucket_name]
+
+ if access_key is None or secret_key is None or storage_endpoint is None:
+ raise Exception(f'ak, sk or endpoint not found in {CONFIG_FILE_NAME}')
+
+ # logger.info(f"get_s3_config: ak={access_key}, sk={secret_key}, endpoint={storage_endpoint}")
+
+ return access_key, secret_key, storage_endpoint
+
+
+def get_s3_config_dict(path: str):
+ access_key, secret_key, storage_endpoint = get_s3_config(get_bucket_name(path))
+ return {'ak': access_key, 'sk': secret_key, 'endpoint': storage_endpoint}
+
+
+def get_bucket_name(path):
+ bucket, key = parse_bucket_key(path)
+ return bucket
+
+
+def parse_bucket_key(s3_full_path: str):
+ """
+ 输入 s3://bucket/path/to/my/file.txt
+ 输出 bucket, path/to/my/file.txt
+ """
+ s3_full_path = s3_full_path.strip()
+ if s3_full_path.startswith("s3://"):
+ s3_full_path = s3_full_path[5:]
+ if s3_full_path.startswith("/"):
+ s3_full_path = s3_full_path[1:]
+ bucket, key = s3_full_path.split("/", 1)
+ return bucket, key
+
+
+def get_device():
+ device_mode = os.getenv('MINERU_DEVICE_MODE', None)
+ if device_mode is not None:
+ return device_mode
+ else:
+ if torch.cuda.is_available():
+ return "cuda"
+ elif torch.backends.mps.is_available():
+ return "mps"
+ else:
+ try:
+ if torch_npu.npu.is_available():
+ return "npu"
+ except Exception as e:
+ try:
+ if torch.gcu.is_available():
+ return "gcu"
+ except Exception as e:
+ try:
+ if torch.musa.is_available():
+ return "musa"
+ except Exception as e:
+ try:
+ if torch.mlu.is_available():
+ return "mlu"
+ except Exception as e:
+ try:
+ if torch.sdaa.is_available():
+ return "sdaa"
+ except Exception as e:
+ pass
+
+ return "cpu"
+
+
+def get_formula_enable(formula_enable):
+ formula_enable_env = os.getenv('MINERU_FORMULA_ENABLE')
+ formula_enable = formula_enable if formula_enable_env is None else formula_enable_env.lower() == 'true'
+ return formula_enable
+
+
+def get_table_enable(table_enable):
+ table_enable_env = os.getenv('MINERU_TABLE_ENABLE')
+ table_enable = table_enable if table_enable_env is None else table_enable_env.lower() == 'true'
+ return table_enable
+
+
+def get_ocr_det_mask_inline_formula_enable(enable):
+ enable_env = os.getenv('MINERU_OCR_DET_MASK_INLINE_FORMULA_ENABLE')
+ enable = enable if enable_env is None else enable_env.lower() == 'true'
+ return enable
+
+
+def get_processing_window_size(default: int = 64) -> int:
+ value = os.getenv('MINERU_PROCESSING_WINDOW_SIZE')
+ if value is None:
+ return default
+ try:
+ window_size = int(value)
+ except ValueError:
+ logger.warning(
+ f"Invalid MINERU_PROCESSING_WINDOW_SIZE value: {value}, use default {default}"
+ )
+ return default
+ return max(1, window_size)
+
+
+def get_max_concurrent_requests(default: int = 3) -> int:
+ if default <= 0:
+ raise ValueError(
+ f"default max_concurrent_requests must be a positive integer, got {default}"
+ )
+ value = os.getenv('MINERU_API_MAX_CONCURRENT_REQUESTS')
+ if value is None:
+ return default
+ try:
+ max_concurrent_requests = int(value)
+ except ValueError as exc:
+ raise ValueError(
+ "Invalid MINERU_API_MAX_CONCURRENT_REQUESTS value: "
+ f"{value}. Expected a positive integer."
+ ) from exc
+ if max_concurrent_requests <= 0:
+ raise ValueError(
+ "Invalid MINERU_API_MAX_CONCURRENT_REQUESTS value: "
+ f"{value}. Expected a positive integer."
+ )
+ return max_concurrent_requests
+
+
+def get_latex_delimiter_config():
+ config = read_config()
+ if config is None:
+ return None
+ latex_delimiter_config = config.get('latex-delimiter-config', None)
+ if latex_delimiter_config is None:
+ # logger.warning(f"'latex-delimiter-config' not found in {CONFIG_FILE_NAME}, use 'None' as default")
+ return None
+ else:
+ return latex_delimiter_config
+
+
+def get_llm_aided_config():
+ config = read_config()
+ if config is None:
+ return None
+ llm_aided_config = config.get('llm-aided-config', None)
+ if llm_aided_config is None:
+ # logger.warning(f"'llm-aided-config' not found in {CONFIG_FILE_NAME}, use 'None' as default")
+ return None
+ else:
+ return llm_aided_config
+
+
+def get_local_models_dir():
+ config = read_config()
+ if config is None:
+ return None
+ models_dir = config.get('models-dir')
+ if models_dir is None:
+ logger.warning(f"'models-dir' not found in {CONFIG_FILE_NAME}, use None as default")
+ return models_dir
diff --git a/mineru/utils/cut_image.py b/mineru/utils/cut_image.py
new file mode 100644
index 000000000..57bfa4acf
--- /dev/null
+++ b/mineru/utils/cut_image.py
@@ -0,0 +1,28 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from loguru import logger
+
+from .pdf_image_tools import cut_image
+
+
+def cut_image_and_table(span, page_pil_img, page_img_md5, page_id, image_writer, scale=2):
+
+ def return_path(path_type):
+ return f"{path_type}/{page_img_md5}"
+
+ span_type = span["type"]
+
+ if not check_img_bbox(span["bbox"]) or not image_writer:
+ span["image_path"] = ""
+ else:
+ span["image_path"] = cut_image(
+ span["bbox"], page_id, page_pil_img, return_path=return_path(span_type), image_writer=image_writer, scale=scale
+ )
+
+ return span
+
+
+def check_img_bbox(bbox) -> bool:
+ if any([bbox[0] >= bbox[2], bbox[1] >= bbox[3]]):
+ logger.warning(f"image_bboxes: 错误的box, {bbox}")
+ return False
+ return True
diff --git a/mineru/utils/docx_formatting.py b/mineru/utils/docx_formatting.py
new file mode 100644
index 000000000..7e44840be
--- /dev/null
+++ b/mineru/utils/docx_formatting.py
@@ -0,0 +1,22 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from enum import Enum
+
+from pydantic import BaseModel
+
+
+class Script(str, Enum):
+ """Text script position."""
+
+ BASELINE = "baseline"
+ SUB = "sub"
+ SUPER = "super"
+
+
+class Formatting(BaseModel):
+ """Formatting."""
+
+ bold: bool = False
+ italic: bool = False
+ underline: bool = False
+ strikethrough: bool = False
+ script: Script = Script.BASELINE
diff --git a/mineru/utils/draw_bbox.py b/mineru/utils/draw_bbox.py
new file mode 100644
index 000000000..55cc337df
--- /dev/null
+++ b/mineru/utils/draw_bbox.py
@@ -0,0 +1,421 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import json
+from io import BytesIO
+
+from loguru import logger
+from pypdf import PdfReader, PdfWriter, PageObject
+from reportlab.pdfgen import canvas
+
+from .enum_class import BlockType, ContentType, SplitFlag
+
+
+def cal_canvas_rect(page, bbox):
+ """
+ Calculate the rectangle coordinates on the canvas based on the original PDF page and bounding box.
+
+ Args:
+ page: A PyPDF2 Page object representing a single page in the PDF.
+ bbox: [x0, y0, x1, y1] representing the bounding box coordinates.
+
+ Returns:
+ rect: [x0, y0, width, height] representing the rectangle coordinates on the canvas.
+ """
+ page_width, page_height = float(page.cropbox[2]), float(page.cropbox[3])
+
+ actual_width = page_width # The width of the final PDF display
+ actual_height = page_height # The height of the final PDF display
+
+ rotation_obj = page.get("/Rotate", 0)
+ try:
+ rotation = int(rotation_obj) % 360 # cast rotation to int to handle IndirectObject
+ except (ValueError, TypeError) as e:
+ logger.warning(f"Invalid /Rotate value {rotation_obj!r} on page; defaulting to 0. Error: {e}")
+ rotation = 0
+
+ if rotation in [90, 270]:
+ # PDF is rotated 90 degrees or 270 degrees, and the width and height need to be swapped
+ actual_width, actual_height = actual_height, actual_width
+
+ x0, y0, x1, y1 = bbox
+ rect_w = abs(x1 - x0)
+ rect_h = abs(y1 - y0)
+
+ if rotation == 270:
+ rect_w, rect_h = rect_h, rect_w
+ x0 = actual_height - y1
+ y0 = actual_width - x1
+ elif rotation == 180:
+ x0 = page_width - x1
+ # y0 stays the same
+ elif rotation == 90:
+ rect_w, rect_h = rect_h, rect_w
+ x0, y0 = y0, x0
+ else:
+ # rotation == 0
+ y0 = page_height - y1
+
+ rect = [x0, y0, rect_w, rect_h]
+ return rect
+
+
+def draw_bbox_without_number(i, bbox_list, page, c, rgb_config, fill_config):
+ new_rgb = [float(color) / 255 for color in rgb_config]
+ page_data = bbox_list[i]
+
+ for bbox in page_data:
+ rect = cal_canvas_rect(page, bbox) # Define the rectangle
+
+ if fill_config: # filled rectangle
+ c.setFillColorRGB(new_rgb[0], new_rgb[1], new_rgb[2], 0.3)
+ c.rect(rect[0], rect[1], rect[2], rect[3], stroke=0, fill=1)
+ else: # bounding box
+ c.setStrokeColorRGB(new_rgb[0], new_rgb[1], new_rgb[2])
+ c.rect(rect[0], rect[1], rect[2], rect[3], stroke=1, fill=0)
+ return c
+
+
+def draw_bbox_with_number(i, bbox_list, page, c, rgb_config, fill_config, draw_bbox=True):
+ new_rgb = [float(color) / 255 for color in rgb_config]
+ page_data = bbox_list[i]
+ # 强制转换为 float
+ page_width, page_height = float(page.cropbox[2]), float(page.cropbox[3])
+
+ for j, bbox in enumerate(page_data):
+ # 确保bbox的每个元素都是float
+ rect = cal_canvas_rect(page, bbox) # Define the rectangle
+
+ if draw_bbox:
+ if fill_config:
+ c.setFillColorRGB(*new_rgb, 0.3)
+ c.rect(rect[0], rect[1], rect[2], rect[3], stroke=0, fill=1)
+ else:
+ c.setStrokeColorRGB(*new_rgb)
+ c.rect(rect[0], rect[1], rect[2], rect[3], stroke=1, fill=0)
+ c.setFillColorRGB(*new_rgb, 1.0)
+ c.setFontSize(size=10)
+
+ c.saveState()
+ rotation_obj = page.get("/Rotate", 0)
+ try:
+ rotation = int(rotation_obj) % 360 # cast rotation to int to handle IndirectObject
+ except (ValueError, TypeError):
+ logger.warning(f"Invalid /Rotate value: {rotation_obj!r}, defaulting to 0")
+ rotation = 0
+
+ if rotation == 0:
+ c.translate(rect[0] + rect[2] + 2, rect[1] + rect[3] - 10)
+ elif rotation == 90:
+ c.translate(rect[0] + 10, rect[1] + rect[3] + 2)
+ elif rotation == 180:
+ c.translate(rect[0] - 2, rect[1] + 10)
+ elif rotation == 270:
+ c.translate(rect[0] + rect[2] - 10, rect[1] - 2)
+
+ c.rotate(rotation)
+ c.drawString(0, 0, str(j + 1))
+ c.restoreState()
+
+ return c
+
+
+def draw_layout_bbox(pdf_info, pdf_bytes, out_path, filename):
+ dropped_bbox_list = []
+ tables_body_list, tables_caption_list, tables_footnote_list = [], [], []
+ imgs_body_list, imgs_caption_list, imgs_footnote_list = [], [], []
+ codes_body_list, codes_caption_list, codes_footnote_list = [], [], []
+ titles_list = []
+ texts_list = []
+ interline_equations_list = []
+ lists_list = []
+ list_items_list = []
+ indexs_list = []
+
+ for page in pdf_info:
+ page_dropped_list = []
+ tables_body, tables_caption, tables_footnote = [], [], []
+ imgs_body, imgs_caption, imgs_footnote = [], [], []
+ codes_body, codes_caption, codes_footnote = [], [], []
+ titles = []
+ texts = []
+ interline_equations = []
+ lists = []
+ list_items = []
+ indices = []
+
+ for dropped_bbox in page['discarded_blocks']:
+ page_dropped_list.append(dropped_bbox['bbox'])
+ dropped_bbox_list.append(page_dropped_list)
+ for block in page["para_blocks"]:
+ bbox = block["bbox"]
+ if block["type"] == BlockType.TABLE:
+ for nested_block in block["blocks"]:
+ bbox = nested_block["bbox"]
+ if nested_block["type"] == BlockType.TABLE_BODY:
+ tables_body.append(bbox)
+ elif nested_block["type"] == BlockType.TABLE_CAPTION:
+ tables_caption.append(bbox)
+ elif nested_block["type"] == BlockType.TABLE_FOOTNOTE:
+ if nested_block.get(SplitFlag.CROSS_PAGE, False):
+ continue
+ tables_footnote.append(bbox)
+ elif block["type"] == BlockType.IMAGE:
+ for nested_block in block["blocks"]:
+ bbox = nested_block["bbox"]
+ if nested_block["type"] == BlockType.IMAGE_BODY:
+ imgs_body.append(bbox)
+ elif nested_block["type"] == BlockType.IMAGE_CAPTION:
+ imgs_caption.append(bbox)
+ elif nested_block["type"] == BlockType.IMAGE_FOOTNOTE:
+ imgs_footnote.append(bbox)
+ elif block["type"] == BlockType.CODE:
+ for nested_block in block["blocks"]:
+ if nested_block["type"] == BlockType.CODE_BODY:
+ bbox = nested_block["bbox"]
+ codes_body.append(bbox)
+ elif nested_block["type"] == BlockType.CODE_CAPTION:
+ bbox = nested_block["bbox"]
+ codes_caption.append(bbox)
+ elif nested_block["type"] == BlockType.CODE_FOOTNOTE:
+ bbox = nested_block["bbox"]
+ codes_footnote.append(bbox)
+ elif block["type"] == BlockType.CHART:
+ for nested_block in block["blocks"]:
+ if nested_block["type"] == BlockType.CHART_BODY:
+ bbox = nested_block["bbox"]
+ imgs_body.append(bbox)
+ elif nested_block["type"] == BlockType.CHART_CAPTION:
+ bbox = nested_block["bbox"]
+ imgs_caption.append(bbox)
+ elif nested_block["type"] == BlockType.CHART_FOOTNOTE:
+ bbox = nested_block["bbox"]
+ imgs_footnote.append(bbox)
+ elif block["type"] == BlockType.SEAL:
+ imgs_body.append(bbox)
+ elif block["type"] == BlockType.TITLE:
+ titles.append(bbox)
+ elif block["type"] in [BlockType.TEXT, BlockType.REF_TEXT, BlockType.ABSTRACT]:
+ texts.append(bbox)
+ elif block["type"] == BlockType.INTERLINE_EQUATION:
+ interline_equations.append(bbox)
+ elif block["type"] == BlockType.LIST:
+ lists.append(bbox)
+ if "blocks" in block:
+ for sub_block in block["blocks"]:
+ list_items.append(sub_block["bbox"])
+ elif block["type"] == BlockType.INDEX:
+ indices.append(bbox)
+
+ tables_body_list.append(tables_body)
+ tables_caption_list.append(tables_caption)
+ tables_footnote_list.append(tables_footnote)
+ imgs_body_list.append(imgs_body)
+ imgs_caption_list.append(imgs_caption)
+ imgs_footnote_list.append(imgs_footnote)
+ titles_list.append(titles)
+ texts_list.append(texts)
+ interline_equations_list.append(interline_equations)
+ lists_list.append(lists)
+ list_items_list.append(list_items)
+ indexs_list.append(indices)
+ codes_body_list.append(codes_body)
+ codes_caption_list.append(codes_caption)
+ codes_footnote_list.append(codes_footnote)
+
+ layout_bbox_list = []
+
+ for page in pdf_info:
+ page_block_list = []
+ for block in page["para_blocks"]:
+ if block["type"] in [
+ BlockType.TEXT,
+ BlockType.REF_TEXT,
+ BlockType.ABSTRACT,
+ BlockType.TITLE,
+ BlockType.INTERLINE_EQUATION,
+ BlockType.LIST,
+ BlockType.INDEX,
+ BlockType.SEAL,
+ ]:
+ bbox = block["bbox"]
+ page_block_list.append(bbox)
+ elif block["type"] in [BlockType.IMAGE, BlockType.CHART, BlockType.CODE, BlockType.TABLE]:
+ for sub_block in block["blocks"]:
+ if sub_block.get(SplitFlag.CROSS_PAGE, False):
+ continue
+ bbox = sub_block["bbox"]
+ page_block_list.append(bbox)
+
+ layout_bbox_list.append(page_block_list)
+
+ pdf_bytes_io = BytesIO(pdf_bytes)
+ pdf_docs = PdfReader(pdf_bytes_io)
+ output_pdf = PdfWriter()
+
+ for i, page in enumerate(pdf_docs.pages):
+ # 获取原始页面尺寸
+ page_width, page_height = float(page.cropbox[2]), float(page.cropbox[3])
+ custom_page_size = (page_width, page_height)
+
+ packet = BytesIO()
+ # 使用原始PDF的尺寸创建canvas
+ c = canvas.Canvas(packet, pagesize=custom_page_size)
+
+ c = draw_bbox_without_number(i, codes_body_list, page, c, [102, 0, 204], True)
+ c = draw_bbox_without_number(i, codes_caption_list, page, c, [204, 153, 255], True)
+ c = draw_bbox_without_number(i, codes_footnote_list, page, c, [229, 204, 255], True)
+ c = draw_bbox_without_number(i, dropped_bbox_list, page, c, [158, 158, 158], True)
+ c = draw_bbox_without_number(i, tables_body_list, page, c, [204, 204, 0], True)
+ c = draw_bbox_without_number(i, tables_caption_list, page, c, [255, 255, 102], True)
+ c = draw_bbox_without_number(i, tables_footnote_list, page, c, [229, 255, 204], True)
+ c = draw_bbox_without_number(i, imgs_body_list, page, c, [153, 255, 51], True)
+ c = draw_bbox_without_number(i, imgs_caption_list, page, c, [102, 178, 255], True)
+ c = draw_bbox_without_number(i, imgs_footnote_list, page, c, [255, 178, 102], True)
+ c = draw_bbox_without_number(i, titles_list, page, c, [102, 102, 255], True)
+ c = draw_bbox_without_number(i, texts_list, page, c, [153, 0, 76], True)
+ c = draw_bbox_without_number(i, interline_equations_list, page, c, [0, 255, 0], True)
+ c = draw_bbox_without_number(i, lists_list, page, c, [40, 169, 92], True)
+ c = draw_bbox_without_number(i, list_items_list, page, c, [40, 169, 92], False)
+ c = draw_bbox_without_number(i, indexs_list, page, c, [40, 169, 92], True)
+ c = draw_bbox_with_number(i, layout_bbox_list, page, c, [255, 0, 0], False, draw_bbox=False)
+
+ c.save()
+ packet.seek(0)
+ overlay_pdf = PdfReader(packet)
+
+ # 添加检查确保overlay_pdf.pages不为空
+ if len(overlay_pdf.pages) > 0:
+ new_page = PageObject(pdf=None)
+ new_page.update(page)
+ page = new_page
+ page.merge_page(overlay_pdf.pages[0])
+ else:
+ # 记录日志并继续处理下一个页面
+ # logger.warning(f"layout.pdf: 第{i + 1}页未能生成有效的overlay PDF")
+ pass
+
+ output_pdf.add_page(page)
+
+ # 保存结果
+ with open(f"{out_path}/{filename}", "wb") as f:
+ output_pdf.write(f)
+
+
+def draw_span_bbox(pdf_info, pdf_bytes, out_path, filename):
+ text_list = []
+ inline_equation_list = []
+ interline_equation_list = []
+ image_list = []
+ table_list = []
+ dropped_list = []
+
+ def get_span_info(span):
+ if span['type'] == ContentType.TEXT:
+ page_text_list.append(span['bbox'])
+ elif span['type'] == ContentType.INLINE_EQUATION:
+ page_inline_equation_list.append(span['bbox'])
+ elif span['type'] == ContentType.INTERLINE_EQUATION:
+ page_interline_equation_list.append(span['bbox'])
+ elif span['type'] in [ContentType.IMAGE, ContentType.CHART, ContentType.SEAL]:
+ page_image_list.append(span['bbox'])
+ elif span['type'] == ContentType.TABLE:
+ page_table_list.append(span['bbox'])
+
+ for page in pdf_info:
+ page_text_list = []
+ page_inline_equation_list = []
+ page_interline_equation_list = []
+ page_image_list = []
+ page_table_list = []
+ page_dropped_list = []
+
+
+ # 构造dropped_list
+ for block in page['discarded_blocks']:
+ for line in block['lines']:
+ for span in line['spans']:
+ page_dropped_list.append(span['bbox'])
+ dropped_list.append(page_dropped_list)
+ # 构造其余useful_list
+ # for block in page['para_blocks']: # span直接用分段合并前的结果就可以
+ for block in page['preproc_blocks']:
+ if block['type'] in [
+ BlockType.TEXT,
+ BlockType.TITLE,
+ BlockType.INTERLINE_EQUATION,
+ BlockType.LIST,
+ BlockType.INDEX,
+ BlockType.REF_TEXT,
+ BlockType.ABSTRACT,
+ BlockType.SEAL,
+ ]:
+ for line in block['lines']:
+ for span in line['spans']:
+ get_span_info(span)
+ elif block['type'] in [BlockType.IMAGE, BlockType.TABLE, BlockType.CHART, BlockType.CODE]:
+ for sub_block in block['blocks']:
+ for line in sub_block['lines']:
+ for span in line['spans']:
+ get_span_info(span)
+ text_list.append(page_text_list)
+ inline_equation_list.append(page_inline_equation_list)
+ interline_equation_list.append(page_interline_equation_list)
+ image_list.append(page_image_list)
+ table_list.append(page_table_list)
+
+ pdf_bytes_io = BytesIO(pdf_bytes)
+ pdf_docs = PdfReader(pdf_bytes_io)
+ output_pdf = PdfWriter()
+
+ for i, page in enumerate(pdf_docs.pages):
+ # 获取原始页面尺寸
+ page_width, page_height = float(page.cropbox[2]), float(page.cropbox[3])
+ custom_page_size = (page_width, page_height)
+
+ packet = BytesIO()
+ # 使用原始PDF的尺寸创建canvas
+ c = canvas.Canvas(packet, pagesize=custom_page_size)
+
+ # 获取当前页面的数据
+ draw_bbox_without_number(i, text_list, page, c,[255, 0, 0], False)
+ draw_bbox_without_number(i, inline_equation_list, page, c, [0, 255, 0], False)
+ draw_bbox_without_number(i, interline_equation_list, page, c, [0, 0, 255], False)
+ draw_bbox_without_number(i, image_list, page, c, [255, 204, 0], False)
+ draw_bbox_without_number(i, table_list, page, c, [204, 0, 255], False)
+ draw_bbox_without_number(i, dropped_list, page, c, [158, 158, 158], False)
+
+ c.save()
+ packet.seek(0)
+ overlay_pdf = PdfReader(packet)
+
+ # 添加检查确保overlay_pdf.pages不为空
+ if len(overlay_pdf.pages) > 0:
+ new_page = PageObject(pdf=None)
+ new_page.update(page)
+ page = new_page
+ page.merge_page(overlay_pdf.pages[0])
+ else:
+ # 记录日志并继续处理下一个页面
+ # logger.warning(f"span.pdf: 第{i + 1}页未能生成有效的overlay PDF")
+ pass
+
+ output_pdf.add_page(page)
+
+ # Save the PDF
+ with open(f"{out_path}/{filename}", "wb") as f:
+ output_pdf.write(f)
+
+
+if __name__ == "__main__":
+ # 读取PDF文件
+ pdf_path = "examples/demo1.pdf"
+ with open(pdf_path, "rb") as f:
+ pdf_bytes = f.read()
+
+ # 从json文件读取pdf_info
+
+ json_path = "examples/demo1_1746005777.0863056_middle.json"
+ with open(json_path, "r", encoding="utf-8") as f:
+ pdf_ann = json.load(f)
+ pdf_info = pdf_ann["pdf_info"]
+ # 调用可视化函数,输出到examples目录
+ draw_layout_bbox(pdf_info, pdf_bytes, "examples", "output_with_layout.pdf")
diff --git a/mineru/utils/engine_utils.py b/mineru/utils/engine_utils.py
new file mode 100644
index 000000000..4eaa88f41
--- /dev/null
+++ b/mineru/utils/engine_utils.py
@@ -0,0 +1,76 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+
+from loguru import logger
+
+from mineru.utils.check_sys_env import is_mac_os_version_supported, is_windows_environment, is_mac_environment, \
+ is_linux_environment
+
+
+def get_vlm_engine(inference_engine: str, is_async: bool = False) -> str:
+ """
+ 自动选择或验证 VLM 推理引擎
+
+ Args:
+ inference_engine: 指定的引擎名称或 'auto' 进行自动选择
+ is_async: 是否使用异步引擎(仅对 vllm 有效)
+
+ Returns:
+ 最终选择的引擎名称
+ """
+ if inference_engine == 'auto':
+ # 根据操作系统自动选择引擎
+ if is_windows_environment():
+ inference_engine = _select_windows_engine()
+ elif is_linux_environment():
+ inference_engine = _select_linux_engine(is_async)
+ elif is_mac_environment():
+ inference_engine = _select_mac_engine()
+ else:
+ logger.warning("Unknown operating system, falling back to transformers")
+ inference_engine = 'transformers'
+
+ formatted_engine = _format_engine_name(inference_engine)
+ logger.info(f"Using {formatted_engine} as the inference engine for VLM.")
+ return formatted_engine
+
+
+def _select_windows_engine() -> str:
+ """Windows 平台引擎选择"""
+ try:
+ import lmdeploy
+ return 'lmdeploy'
+ except ImportError:
+ return 'transformers'
+
+
+def _select_linux_engine(is_async: bool) -> str:
+ """Linux 平台引擎选择"""
+ try:
+ import vllm
+ return 'vllm-async' if is_async else 'vllm'
+ except ImportError:
+ try:
+ import lmdeploy
+ return 'lmdeploy'
+ except ImportError:
+ return 'transformers'
+
+
+def _select_mac_engine() -> str:
+ """macOS 平台引擎选择"""
+ try:
+ from mlx_vlm import load as mlx_load
+ if is_mac_os_version_supported():
+ return 'mlx'
+ else:
+ return 'transformers'
+ except ImportError:
+ return 'transformers'
+
+
+def _format_engine_name(engine: str) -> str:
+ """统一格式化引擎名称"""
+ if engine != 'transformers':
+ return f"{engine}-engine"
+ return engine
diff --git a/mineru/utils/enum_class.py b/mineru/utils/enum_class.py
new file mode 100644
index 000000000..34f5055a3
--- /dev/null
+++ b/mineru/utils/enum_class.py
@@ -0,0 +1,136 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from enum import Enum
+
+class BlockType:
+ IMAGE = 'image'
+ TABLE = 'table'
+ CHART = 'chart'
+ IMAGE_BODY = 'image_body'
+ TABLE_BODY = 'table_body'
+ CHART_BODY = 'chart_body'
+ CAPTION = 'caption' # generic caption type (e.g., for Word documents)
+ IMAGE_CAPTION = 'image_caption'
+ TABLE_CAPTION = 'table_caption'
+ CHART_CAPTION = 'chart_caption'
+ ALGORITHM_CAPTION = 'algorithm_caption'
+ FOOTNOTE = 'footnote' # pp_layout中的vision_footnote
+ IMAGE_FOOTNOTE = 'image_footnote'
+ TABLE_FOOTNOTE = 'table_footnote'
+ CHART_FOOTNOTE = 'chart_footnote'
+ TEXT = 'text'
+ TITLE = 'title'
+ INTERLINE_EQUATION = 'interline_equation'
+ EQUATION = "equation" # 公式(独立公式)
+ LIST = 'list'
+ INDEX = 'index'
+ DISCARDED = 'discarded'
+
+ # Added in vlm 2.5
+ CODE = "code"
+ CODE_BODY = "code_body"
+ CODE_CAPTION = "code_caption"
+ CODE_FOOTNOTE = "code_footnote"
+ ALGORITHM = "algorithm"
+ REF_TEXT = "ref_text"
+ PHONETIC = "phonetic"
+ HEADER = "header"
+ FOOTER = "footer"
+ PAGE_NUMBER = "page_number"
+ ASIDE_TEXT = "aside_text"
+ PAGE_FOOTNOTE = "page_footnote"
+
+ # Added in pp_doclayout_v2
+ ABSTRACT = "abstract"
+ DOC_TITLE = "doc_title"
+ PARAGRAPH_TITLE = "paragraph_title"
+ VERTICAL_TEXT = "vertical_text"
+ SEAL = "seal"
+ HEADER_IMAGE = "header_image"
+ FOOTER_IMAGE = "footer_image"
+ FORMULA_NUMBER = "formula_number"
+
+class ContentType:
+ IMAGE = 'image'
+ TABLE = 'table'
+ CHART = 'chart'
+ TEXT = 'text'
+ INTERLINE_EQUATION = 'interline_equation'
+ INLINE_EQUATION = 'inline_equation'
+ EQUATION = 'equation'
+ HYPERLINK = 'hyperlink'
+ SEAL = 'seal'
+
+
+class ContentTypeV2:
+ CODE = 'code'
+ ALGORITHM = "algorithm"
+ EQUATION_INTERLINE = 'equation_interline'
+ IMAGE = 'image'
+ SEAL = 'seal'
+ TABLE = 'table'
+ CHART = 'chart'
+ TABLE_SIMPLE = 'simple_table'
+ TABLE_COMPLEX = 'complex_table'
+ LIST = 'list'
+ LIST_TEXT = 'text_list'
+ LIST_REF = 'reference_list'
+ INDEX = 'index'
+ TITLE = 'title'
+ PARAGRAPH = 'paragraph'
+ SPAN_TEXT = 'text'
+ SPAN_EQUATION_INLINE = 'equation_inline'
+ SPAN_PHONETIC = 'phonetic'
+ SPAN_MD = 'md'
+ SPAN_CODE_INLINE = 'code_inline'
+ PAGE_HEADER = "page_header"
+ PAGE_FOOTER = "page_footer"
+ PAGE_NUMBER = "page_number"
+ PAGE_ASIDE_TEXT = "page_aside_text"
+ PAGE_FOOTNOTE = "page_footnote"
+
+
+class MakeMode:
+ MM_MD = 'mm_markdown'
+ NLP_MD = 'nlp_markdown'
+ CONTENT_LIST = 'content_list'
+ CONTENT_LIST_V2 = 'content_list_v2'
+
+
+class ModelPath:
+ vlm_root_hf = "opendatalab/MinerU2.5-Pro-2604-1.2B"
+ vlm_root_modelscope = "OpenDataLab/MinerU2.5-Pro-2604-1.2B"
+ pipeline_root_modelscope = "OpenDataLab/PDF-Extract-Kit-1.0"
+ pipeline_root_hf = "opendatalab/PDF-Extract-Kit-1.0"
+ pp_doclayout_v2 = "models/Layout/PP-DocLayoutV2"
+ unimernet_small = "models/MFR/unimernet_hf_small_2503"
+ pp_formulanet_plus_m = "models/MFR/pp_formulanet_plus_m"
+ pytorch_paddle = "models/OCR/paddleocr_torch"
+ slanet_plus = "models/TabRec/SlanetPlus/slanet-plus.onnx"
+ unet_structure = "models/TabRec/UnetStructure/unet.onnx"
+ paddle_table_cls = "models/TabCls/paddle_table_cls/PP-LCNet_x1_0_table_cls.onnx"
+ paddle_orientation_classification = "models/OriCls/paddle_orientation_classification/PP-LCNet_x1_0_doc_ori.onnx"
+
+
+class SplitFlag:
+ CROSS_PAGE = 'cross_page'
+ LINES_DELETED = 'lines_deleted'
+
+
+class ImageType:
+ PIL = 'pil_img'
+ BASE64 = 'base64_img'
+
+
+class NotExtractType(Enum):
+ TEXT = BlockType.TEXT
+ TITLE = BlockType.TITLE
+ HEADER = BlockType.HEADER
+ FOOTER = BlockType.FOOTER
+ PAGE_NUMBER = BlockType.PAGE_NUMBER
+ PAGE_FOOTNOTE = BlockType.PAGE_FOOTNOTE
+ REF_TEXT = BlockType.REF_TEXT
+ TABLE_CAPTION = BlockType.TABLE_CAPTION
+ IMAGE_CAPTION = BlockType.IMAGE_CAPTION
+ TABLE_FOOTNOTE = BlockType.TABLE_FOOTNOTE
+ IMAGE_FOOTNOTE = BlockType.IMAGE_FOOTNOTE
+ CODE_CAPTION = BlockType.CODE_CAPTION
diff --git a/mineru/utils/guess_suffix_or_lang.py b/mineru/utils/guess_suffix_or_lang.py
new file mode 100644
index 000000000..3a26cbfa5
--- /dev/null
+++ b/mineru/utils/guess_suffix_or_lang.py
@@ -0,0 +1,76 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from pathlib import Path
+
+from loguru import logger
+from magika import Magika
+
+
+DEFAULT_LANG = "txt"
+PDF_SIG_BYTES = b'%PDF'
+magika = Magika()
+
+def _normalize_text_for_language_guess(code: str) -> str:
+ if not code:
+ return ""
+
+ normalized = []
+ index = 0
+ while index < len(code):
+ current_char = code[index]
+ current_ord = ord(current_char)
+
+ if 0xD800 <= current_ord <= 0xDBFF:
+ if index + 1 < len(code):
+ next_char = code[index + 1]
+ next_ord = ord(next_char)
+ if 0xDC00 <= next_ord <= 0xDFFF:
+ pair = current_char + next_char
+ normalized.append(pair.encode("utf-16", "surrogatepass").decode("utf-16"))
+ index += 2
+ continue
+ index += 1
+ continue
+
+ if 0xDC00 <= current_ord <= 0xDFFF:
+ index += 1
+ continue
+
+ normalized.append(current_char)
+ index += 1
+
+ return "".join(normalized)
+
+
+def guess_language_by_text(code):
+ normalized_code = _normalize_text_for_language_guess(code)
+ if not normalized_code:
+ return DEFAULT_LANG
+
+ try:
+ codebytes = normalized_code.encode("utf-8", errors="replace")
+ lang = magika.identify_bytes(codebytes).prediction.output.label
+ except Exception:
+ return DEFAULT_LANG
+
+ return lang if lang != "unknown" else DEFAULT_LANG
+
+
+def guess_suffix_by_bytes(file_bytes, file_path=None) -> str:
+ suffix = magika.identify_bytes(file_bytes).prediction.output.label
+ if file_path and suffix in ["ai", "html"] and Path(file_path).suffix.lower() in [".pdf"] and file_bytes[:4] == PDF_SIG_BYTES:
+ suffix = "pdf"
+ return suffix
+
+
+def guess_suffix_by_path(file_path) -> str:
+ if not isinstance(file_path, Path):
+ file_path = Path(file_path)
+ suffix = magika.identify_path(file_path).prediction.output.label
+ if suffix in ["ai", "html"] and file_path.suffix.lower() in [".pdf"]:
+ try:
+ with open(file_path, 'rb') as f:
+ if f.read(4) == PDF_SIG_BYTES:
+ suffix = "pdf"
+ except Exception as e:
+ logger.warning(f"Failed to read file {file_path} for PDF signature check: {e}")
+ return suffix
diff --git a/mineru/utils/hash_utils.py b/mineru/utils/hash_utils.py
new file mode 100644
index 000000000..2b82c78dd
--- /dev/null
+++ b/mineru/utils/hash_utils.py
@@ -0,0 +1,30 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import hashlib
+import json
+
+
+def bytes_md5(file_bytes):
+ hasher = hashlib.md5()
+ hasher.update(file_bytes)
+ return hasher.hexdigest().upper()
+
+
+def str_md5(input_string):
+ hasher = hashlib.md5()
+ # 在Python3中，需要将字符串转化为字节对象才能被哈希函数处理
+ input_bytes = input_string.encode('utf-8')
+ hasher.update(input_bytes)
+ return hasher.hexdigest()
+
+
+def str_sha256(input_string):
+ hasher = hashlib.sha256()
+ # 在Python3中，需要将字符串转化为字节对象才能被哈希函数处理
+ input_bytes = input_string.encode('utf-8')
+ hasher.update(input_bytes)
+ return hasher.hexdigest()
+
+
+def dict_md5(d):
+ json_str = json.dumps(d, sort_keys=True, ensure_ascii=False)
+ return hashlib.md5(json_str.encode('utf-8')).hexdigest()
\ No newline at end of file
diff --git a/mineru/utils/language.py b/mineru/utils/language.py
new file mode 100644
index 000000000..d90c00d6f
--- /dev/null
+++ b/mineru/utils/language.py
@@ -0,0 +1,49 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import unicodedata
+
+if not os.getenv("FTLANG_CACHE"):
+ current_file_path = os.path.abspath(__file__)
+ current_dir = os.path.dirname(current_file_path)
+ root_dir = os.path.dirname(current_dir)
+ ftlang_cache_dir = os.path.join(root_dir, 'resources', 'fasttext-langdetect')
+ os.environ["FTLANG_CACHE"] = str(ftlang_cache_dir)
+ # print(os.getenv("FTLANG_CACHE"))
+
+from fast_langdetect import detect_language
+
+
+def remove_invalid_surrogates(text):
+ # 移除无效的 UTF-16 代理对
+ return ''.join(c for c in text if not (0xD800 <= ord(c) <= 0xDFFF))
+
+
+def detect_lang(text: str) -> str:
+
+ if len(text) == 0:
+ return ""
+
+ text = text.replace("\n", "")
+ text = remove_invalid_surrogates(text)
+
+ # print(text)
+ try:
+ lang_upper = detect_language(text)
+ except:
+ html_no_ctrl_chars = ''.join([l for l in text if unicodedata.category(l)[0] not in ['C', ]])
+ lang_upper = detect_language(html_no_ctrl_chars)
+
+ try:
+ lang = lang_upper.lower()
+ except:
+ lang = ""
+ return lang
+
+
+if __name__ == '__main__':
+ print(os.getenv("FTLANG_CACHE"))
+ print(detect_lang("This is a test."))
+ print(detect_lang("This is a test"))
+ print(detect_lang("这个是中文测试。"))
+ print(detect_lang("这个是中文测试。"))
+ print(detect_lang("〖\ud835\udc46\ud835〗这是个包含utf-16的中文测试"))
\ No newline at end of file
diff --git a/mineru/utils/llm_aided.py b/mineru/utils/llm_aided.py
new file mode 100644
index 000000000..aaff369b6
--- /dev/null
+++ b/mineru/utils/llm_aided.py
@@ -0,0 +1,373 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from concurrent.futures import ThreadPoolExecutor
+
+import json_repair
+from loguru import logger
+from openai import OpenAI
+
+from mineru.backend.pipeline.pipeline_middle_json_mkcontent import merge_para_with_text
+from mineru.utils.enum_class import BlockType
+
+
+TITLE_BLOCK_TYPES = {
+ BlockType.TITLE,
+ BlockType.DOC_TITLE,
+ BlockType.PARAGRAPH_TITLE,
+}
+MAX_TITLE_GROUP_WORKERS = 4
+
+
+def _get_title_line_avg_height(block):
+ if "line_avg_height" in block:
+ return block["line_avg_height"]
+
+ title_block_line_height_list = []
+ for line in block.get("lines", []):
+ bbox = line["bbox"]
+ title_block_line_height_list.append(int(bbox[3] - bbox[1]))
+
+ if len(title_block_line_height_list) > 0:
+ return sum(title_block_line_height_list) / len(title_block_line_height_list)
+
+ return int(block["bbox"][3] - block["bbox"][1])
+
+
+def _collect_title_block_refs(page_info_list):
+ title_block_refs = []
+ title_types = set()
+
+ for page_info in page_info_list:
+ for block in page_info.get("para_blocks", []):
+ block_type = block.get("type")
+ if block_type in TITLE_BLOCK_TYPES:
+ title_block_refs.append((page_info, block))
+ title_types.add(block_type)
+
+ return title_block_refs, title_types
+
+
+def _build_title_dict(title_block_refs):
+ title_dict = {}
+
+ for i, (page_info, block) in enumerate(title_block_refs):
+ title_dict[str(i)] = [
+ merge_para_with_text(block),
+ _get_title_line_avg_height(block),
+ int(page_info["page_idx"]) + 1,
+ ]
+
+ return title_dict
+
+
+def _build_title_optimize_prompt(title_dict):
+ return f"""输入的内容是一篇文档中所有标题组成的字典，请根据以下指南优化标题的结果，使结果符合正常文档的层次结构：
+
+1. 字典中每个value均为一个list，包含以下元素：
+ - 标题文本
+ - 文本行高是标题所在块的平均行高
+ - 标题所在的页码
+
+2. 保留原始内容：
+ - 输入的字典中所有元素都是有效的，不能删除字典中的任何元素
+ - 请务必保证输出的字典中元素的数量和输入的数量一致
+
+3. 保持字典内key-value的对应关系不变
+
+4. 优化层次结构：
+ - 根据标题内容的语义为每个标题元素添加适当的层次结构
+ - 行高较大的标题一般是更高级别的标题
+ - 标题从前至后的层级必须是连续的，不能跳过层级
+ - 标题层级最多为4级，不要添加过多的层级
+ - 优化后的标题只保留代表该标题的层级的整数，不要保留其他信息
+
+5. 合理性检查与微调：
+ - 在完成初步分级后，仔细检查分级结果的合理性
+ - 根据上下文关系和逻辑顺序，对不合理的分级进行微调
+ - 确保最终的分级结果符合文档的实际结构和逻辑
+
+IMPORTANT:
+请直接返回优化过的由标题层级组成的字典，格式为{{标题id:标题层级}}，如下：
+{{
+ 0:1,
+ 1:2,
+ 2:2,
+ 3:3
+}}
+不需要对字典格式化，不需要返回任何其他信息。
+
+Input title list:
+{title_dict}
+
+Corrected title list:
+"""
+
+
+def _build_relative_title_optimize_prompt(title_dict):
+ return f"""输入内容是某一篇文档中除文章标题外的全部章节/段落标题组成的字典。
+
+请注意：
+- 文章标题不在本次输入中，已经由系统单独识别并设置为1级标题
+
+1. 字典中每个value均为一个list，包含以下元素：
+ - 标题文本
+ - 文本行高是标题所在块的平均行高
+ - 标题所在的页码
+
+2. 保留原始内容：
+ - 输入的字典中所有元素都是有效的，不能删除字典中的任何元素
+ - 请务必保证输出的字典中元素的数量和输入的数量一致
+
+3. 保持字典内key-value的对应关系不变
+
+4. 优化层次结构：
+ - 根据标题内容的语义为每个标题元素添加适当的层次结构
+ - 行高较大的标题一般是更高级别的标题
+ - 标题从前至后的层级必须是连续的，不能跳过层级
+ - 标题层级最多为4级，不要添加过多的层级
+ - 优化后的标题只保留代表该标题的层级的整数，不要保留其他信息
+
+5. 合理性检查与微调：
+ - 在完成初步分级后，仔细检查分级结果的合理性
+ - 根据上下文关系和逻辑顺序，对不合理的分级进行微调
+ - 确保最终的分级结果符合文档的实际结构和逻辑
+
+IMPORTANT:
+请直接返回优化后的标题层级字典，格式为{{标题id:标题层级}}，如下：
+{{
+ 0:1,
+ 1:2,
+ 2:2,
+ 3:3
+}}
+不要返回 Markdown，不要返回代码块，不要返回任何解释文字。
+
+Input title list:
+{title_dict}
+
+Corrected title list:
+"""
+
+
+def _request_title_levels(title_aided_config, title_dict, prompt_builder=None):
+ if len(title_dict) == 0:
+ return {}
+
+ client = OpenAI(
+ api_key=title_aided_config["api_key"],
+ base_url=title_aided_config["base_url"],
+ )
+
+ retry_count = 0
+ max_retries = 3
+ expected_keys = set(range(len(title_dict)))
+ if prompt_builder is None:
+ prompt_builder = _build_title_optimize_prompt
+ title_optimize_prompt = prompt_builder(title_dict)
+
+ logger.debug(f"Requesting LLM for title optimization with prompt: {title_optimize_prompt}")
+
+ api_params = {
+ "model": title_aided_config["model"],
+ "messages": [{"role": "user", "content": title_optimize_prompt}],
+ "temperature": 0.7,
+ "stream": True,
+ }
+ if "enable_thinking" in title_aided_config:
+ api_params["extra_body"] = {
+ "enable_thinking": title_aided_config["enable_thinking"]
+ }
+
+ while retry_count < max_retries:
+ try:
+ completion = client.chat.completions.create(**api_params)
+ content_pieces = []
+ for chunk in completion:
+ if chunk.choices and chunk.choices[0].delta.content is not None:
+ content_pieces.append(chunk.choices[0].delta.content)
+
+ content = "".join(content_pieces).strip()
+ if "" in content:
+ idx = content.index("") + len("")
+ content = content[idx:].strip()
+
+ logger.debug(f"Raw LLM output for title levels: {content}")
+ dict_completion = json_repair.loads(content)
+ dict_completion = {int(k): int(v) for k, v in dict_completion.items()}
+
+ if set(dict_completion.keys()) == expected_keys:
+ return dict_completion
+
+ logger.warning(
+ "The keys in the optimized title result do not match the input titles."
+ )
+ except Exception as e:
+ logger.exception(e)
+
+ retry_count += 1
+
+ logger.error("Failed to decode dict after maximum retries.")
+ return None
+
+
+def _apply_levels_to_blocks(title_block_refs, levels_by_index):
+ if levels_by_index is None:
+ return
+
+ for i, (_, block) in enumerate(title_block_refs):
+ block["level"] = int(levels_by_index[i])
+
+
+def _normalize_title_types(title_block_refs):
+ for _, block in title_block_refs:
+ if block.get("type") in [BlockType.DOC_TITLE, BlockType.PARAGRAPH_TITLE]:
+ block["type"] = BlockType.TITLE
+
+
+def _get_title_block_identity(block):
+ block_index = block.get("index")
+ if block_index is not None:
+ return ("index", block_index)
+
+ return (
+ "bbox_text",
+ tuple(block.get("bbox", [])),
+ merge_para_with_text(block),
+ )
+
+
+def _sync_para_titles_to_preproc(page_info_list):
+ for page_info in page_info_list:
+ para_title_map = {}
+ for block in page_info.get("para_blocks", []):
+ if block.get("type") in TITLE_BLOCK_TYPES:
+ para_title_map[_get_title_block_identity(block)] = block
+
+ if len(para_title_map) == 0:
+ continue
+
+ for block in page_info.get("preproc_blocks", []):
+ if block.get("type") not in TITLE_BLOCK_TYPES:
+ continue
+
+ para_block = para_title_map.get(_get_title_block_identity(block))
+ if para_block is None:
+ continue
+
+ block["type"] = para_block.get("type", block.get("type"))
+ if "level" in para_block:
+ block["level"] = para_block["level"]
+
+
+def _run_single_pass_title_leveling(title_block_refs, title_aided_config):
+ title_dict = _build_title_dict(title_block_refs)
+ levels_by_index = _request_title_levels(title_aided_config, title_dict)
+ _apply_levels_to_blocks(title_block_refs, levels_by_index)
+
+
+def _split_paragraph_title_groups(title_block_refs):
+ groups = []
+ current_group = []
+
+ for title_ref in title_block_refs:
+ _, block = title_ref
+ if block.get("type") == BlockType.DOC_TITLE:
+ if current_group:
+ groups.append(current_group)
+ current_group = []
+ elif block.get("type") == BlockType.PARAGRAPH_TITLE:
+ current_group.append(title_ref)
+
+ if current_group:
+ groups.append(current_group)
+
+ return groups
+
+
+def _offset_paragraph_title_levels(levels_by_index):
+ if not levels_by_index:
+ return levels_by_index
+
+ return {
+ index: 2 if level == 1 else level
+ for index, level in levels_by_index.items()
+ }
+
+
+def _request_paragraph_group_levels(title_block_refs, title_aided_config):
+ title_dict = _build_title_dict(title_block_refs)
+ levels_by_index = _request_title_levels(
+ title_aided_config,
+ title_dict,
+ prompt_builder=_build_relative_title_optimize_prompt,
+ )
+ return _offset_paragraph_title_levels(levels_by_index)
+
+
+def _run_grouped_title_leveling(title_block_refs, title_aided_config):
+ doc_title_refs = []
+ for title_ref in title_block_refs:
+ _, block = title_ref
+ if block.get("type") == BlockType.DOC_TITLE:
+ block["level"] = 1
+ doc_title_refs.append(title_ref)
+
+ paragraph_title_groups = _split_paragraph_title_groups(title_block_refs)
+ group_levels = []
+
+ if len(paragraph_title_groups) > 1:
+ max_workers = min(len(paragraph_title_groups), MAX_TITLE_GROUP_WORKERS)
+ with ThreadPoolExecutor(max_workers=max_workers) as executor:
+ futures = [
+ executor.submit(
+ _request_paragraph_group_levels,
+ title_group,
+ title_aided_config,
+ )
+ for title_group in paragraph_title_groups
+ ]
+ group_levels = [future.result() for future in futures]
+ else:
+ group_levels = [
+ _request_paragraph_group_levels(title_group, title_aided_config)
+ for title_group in paragraph_title_groups
+ ]
+
+ for title_group, levels_by_index in zip(paragraph_title_groups, group_levels):
+ _apply_levels_to_blocks(title_group, levels_by_index)
+
+ _normalize_title_types(doc_title_refs)
+ for title_group in paragraph_title_groups:
+ _normalize_title_types(title_group)
+
+
+def llm_aided_title(page_info_list, title_aided_config):
+ title_block_refs, title_types = _collect_title_block_refs(page_info_list)
+ if len(title_block_refs) == 0:
+ logger.info("No titles detected, skipping LLM-aided title optimization.")
+ return
+
+ has_doc_title = BlockType.DOC_TITLE in title_types
+ has_paragraph_title = BlockType.PARAGRAPH_TITLE in title_types
+ has_generic_title = BlockType.TITLE in title_types
+
+ if has_doc_title and has_paragraph_title and not has_generic_title:
+ _run_grouped_title_leveling(title_block_refs, title_aided_config)
+ _sync_para_titles_to_preproc(page_info_list)
+ return
+
+ doc_title_refs = []
+ title_refs_for_llm = []
+ for title_ref in title_block_refs:
+ _, block = title_ref
+ if block.get("type") == BlockType.DOC_TITLE:
+ block["level"] = 1
+ doc_title_refs.append(title_ref)
+ else:
+ title_refs_for_llm.append(title_ref)
+
+ if len(title_refs_for_llm) > 0:
+ _run_single_pass_title_leveling(title_refs_for_llm, title_aided_config)
+
+ _normalize_title_types(doc_title_refs)
+ _normalize_title_types(title_refs_for_llm)
+ _sync_para_titles_to_preproc(page_info_list)
diff --git a/mineru/utils/magic_model_utils.py b/mineru/utils/magic_model_utils.py
new file mode 100644
index 000000000..105dcba88
--- /dev/null
+++ b/mineru/utils/magic_model_utils.py
@@ -0,0 +1,163 @@
+# Copyright (c) Opendatalab. All rights reserved.
+"""
+包含两个MagicModel类中重复使用的方法和逻辑
+"""
+from typing import List, Dict, Any, Callable
+
+from loguru import logger
+from mineru.utils.boxbase import bbox_distance, bbox_center_distance, is_in
+
+
+def reduct_overlap(bboxes: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+ """
+ 去除重叠的bbox，保留不被其他bbox包含的bbox
+
+ Args:
+ bboxes: 包含bbox信息的字典列表
+
+ Returns:
+ 去重后的bbox列表
+ """
+ N = len(bboxes)
+ keep = [True] * N
+ for i in range(N):
+ for j in range(N):
+ if i == j:
+ continue
+ if is_in(bboxes[i]['bbox'], bboxes[j]['bbox']):
+ keep[i] = False
+ return [bboxes[i] for i in range(N) if keep[i]]
+
+
+def tie_up_category_by_index(
+ get_subjects_func: Callable,
+ get_objects_func: Callable,
+ extract_subject_func: Callable = None,
+ extract_object_func: Callable = None,
+ object_block_type: str = "object",
+ include_bbox: bool = True,
+):
+ """
+ 基于index的类别关联方法，用于将主体对象与客体对象进行关联
+ 客体优先匹配给index最接近的主体，匹配优先级为：
+ 1. index差值（最高优先级）
+ 2. bbox边缘距离（相邻边距离）
+ 3. bbox中心点距离（最低优先级，作为最终tiebreaker）
+
+ 参数:
+ get_subjects_func: 函数，提取主体对象
+ get_objects_func: 函数，提取客体对象
+ extract_subject_func: 函数，自定义提取主体属性（默认使用bbox和其他属性）
+ extract_object_func: 函数，自定义提取客体属性（默认使用bbox和其他属性）
+
+ 返回:
+ 关联后的对象列表，按主体index升序排列
+ """
+ subjects = get_subjects_func()
+ objects = get_objects_func()
+
+ # 如果没有提供自定义提取函数，使用默认函数
+ if extract_subject_func is None:
+ extract_subject_func = lambda x: x
+ if extract_object_func is None:
+ extract_object_func = lambda x: x
+
+ # 初始化结果字典，key为主体索引，value为关联信息
+ result_dict = {}
+
+ # 初始化所有主体
+ for i, subject in enumerate(subjects):
+ result_dict[i] = {
+ "sub_bbox": extract_subject_func(subject),
+ "obj_bboxes": [],
+ "sub_idx": i,
+ }
+
+ # 提取所有客体的index集合，用于计算有效index差值
+ object_indices = set(obj["index"] for obj in objects)
+
+ def calc_effective_index_diff(obj_index: int, sub_index: int) -> int:
+ """
+ 计算有效的index差值
+ 有效差值 = 绝对差值 - 区间内其他客体的数量
+ 即：如果obj_index和sub_index之间的差值是由其他客体造成的，则应该扣除这部分差值
+ """
+ if obj_index == sub_index:
+ return 0
+
+ start, end = min(obj_index, sub_index), max(obj_index, sub_index)
+ abs_diff = end - start
+
+ # 计算区间(start, end)内有多少个其他客体的index
+ other_objects_count = 0
+ for idx in range(start + 1, end):
+ if idx in object_indices:
+ other_objects_count += 1
+
+ return abs_diff - other_objects_count
+
+ # 为每个客体找到最匹配的主体
+ for obj in objects:
+ if len(subjects) == 0:
+ # 如果没有主体，跳过客体
+ continue
+
+ obj_index = obj["index"]
+ min_index_diff = float("inf")
+ best_subject_indices = []
+
+ # 找出有效index差值最小的所有主体
+ for i, subject in enumerate(subjects):
+ sub_index = subject["index"]
+ index_diff = calc_effective_index_diff(obj_index, sub_index)
+
+ if index_diff < min_index_diff:
+ min_index_diff = index_diff
+ best_subject_indices = [i]
+ elif index_diff == min_index_diff:
+ best_subject_indices.append(i)
+
+ if len(best_subject_indices) == 1:
+ best_subject_idx = best_subject_indices[0]
+ # 如果有多个主体的index差值相同（最多两个），根据边缘距离进行筛选
+ elif len(best_subject_indices) == 2:
+ # 只有在包含bbox信息时才进行边缘距离的计算和比较，否则直接匹配第一个主体
+ if include_bbox:
+ # 计算所有候选主体的边缘距离
+ edge_distances = [(idx, bbox_distance(obj["bbox"], subjects[idx]["bbox"])) for idx in best_subject_indices]
+ edge_dist_diff = abs(edge_distances[0][1] - edge_distances[1][1])
+
+ for idx, edge_dist in edge_distances:
+ logger.debug(f"Obj index: {obj_index}, Sub index: {subjects[idx]['index']}, Edge distance: {edge_dist}")
+
+ if edge_dist_diff > 2:
+ # 边缘距离差值大于2，匹配边缘距离更小的主体
+ best_subject_idx = min(edge_distances, key=lambda x: x[1])[0]
+ logger.debug(f"Obj index: {obj_index}, edge_dist_diff > 2, matching to subject with min edge distance, index: {subjects[best_subject_idx]['index']}")
+ elif object_block_type == "table_caption":
+ # 边缘距离差值<=2且为table_caption，匹配index更大的主体
+ best_subject_idx = max(best_subject_indices, key=lambda idx: subjects[idx]["index"])
+ logger.debug(f"Obj index: {obj_index}, edge_dist_diff <= 2 and table_caption, matching to later subject with index: {subjects[best_subject_idx]['index']}")
+ elif object_block_type.endswith("footnote"):
+ # 边缘距离差值<=2且为footnote，匹配index更小的主体
+ best_subject_idx = min(best_subject_indices, key=lambda idx: subjects[idx]["index"])
+ logger.debug(f"Obj index: {obj_index}, edge_dist_diff <= 2 and footnote, matching to earlier subject with index: {subjects[best_subject_idx]['index']}")
+ else:
+ # 边缘距离差值<=2 且不适用特殊匹配规则，使用中心点距离匹配
+ center_distances = [(idx, bbox_center_distance(obj["bbox"], subjects[idx]["bbox"])) for idx in best_subject_indices]
+ for idx, center_dist in center_distances:
+ logger.debug(f"Obj index: {obj_index}, Sub index: {subjects[idx]['index']}, Center distance: {center_dist}")
+ best_subject_idx = min(center_distances, key=lambda x: x[1])[0]
+ else:
+ best_subject_idx = best_subject_indices[0]
+ else:
+ raise ValueError("More than two subjects have the same minimal index difference, which is unexpected.")
+
+ # 将客体添加到最佳主体的obj_bboxes中
+ result_dict[best_subject_idx]["obj_bboxes"].append(extract_object_func(obj))
+
+ # 转换为列表并按主体index排序
+ ret = list(result_dict.values())
+ ret.sort(key=lambda x: x["sub_idx"])
+
+ return ret
diff --git a/mineru/utils/model_utils.py b/mineru/utils/model_utils.py
new file mode 100644
index 000000000..471f70f64
--- /dev/null
+++ b/mineru/utils/model_utils.py
@@ -0,0 +1,253 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import math
+import os
+import time
+import gc
+from PIL import Image
+from loguru import logger
+import numpy as np
+
+try:
+ import torch
+ import torch_npu
+except ImportError:
+ pass
+
+TEXT_REGION_LABELS = {
+ "abstract",
+ "algorithm",
+ "aside_text",
+ "content",
+ "doc_title",
+ "figure_title",
+ "footer",
+ "footer_image",
+ "footnote",
+ "formula_number",
+ "header",
+ "header_image",
+ "number",
+ "paragraph_title",
+ "reference_content",
+ "text",
+ "vertical_text",
+ "vision_footnote",
+}
+
+
+def _get_bbox(item):
+ bbox = item.get("bbox")
+ if bbox is not None:
+ xmin, ymin, xmax, ymax = bbox
+ return float(xmin), float(ymin), float(xmax), float(ymax)
+
+
+def _get_int_bbox(item):
+ xmin, ymin, xmax, ymax = _get_bbox(item)
+ return math.floor(xmin), math.floor(ymin), math.ceil(xmax), math.ceil(ymax)
+
+
+def crop_img(input_res, input_img, crop_paste_x=0, crop_paste_y=0):
+ crop_xmin, crop_ymin, crop_xmax, crop_ymax = _get_int_bbox(input_res)
+
+ # Calculate new dimensions
+ crop_new_width = crop_xmax - crop_xmin + crop_paste_x * 2
+ crop_new_height = crop_ymax - crop_ymin + crop_paste_y * 2
+
+ if isinstance(input_img, np.ndarray):
+
+ # Create a white background array
+ return_image = np.ones((crop_new_height, crop_new_width, 3), dtype=np.uint8) * 255
+
+ # Crop the original image using numpy slicing
+ cropped_img = input_img[crop_ymin:crop_ymax, crop_xmin:crop_xmax]
+
+ # Paste the cropped image onto the white background
+ return_image[crop_paste_y:crop_paste_y + (crop_ymax - crop_ymin),
+ crop_paste_x:crop_paste_x + (crop_xmax - crop_xmin)] = cropped_img
+ else:
+ # Create a white background array
+ return_image = Image.new('RGB', (crop_new_width, crop_new_height), 'white')
+ # Crop image
+ crop_box = (crop_xmin, crop_ymin, crop_xmax, crop_ymax)
+ cropped_img = input_img.crop(crop_box)
+ return_image.paste(cropped_img, (crop_paste_x, crop_paste_y))
+
+ return_list = [crop_paste_x, crop_paste_y, crop_xmin, crop_ymin, crop_xmax, crop_ymax, crop_new_width,
+ crop_new_height]
+ return return_image, return_list
+
+
+def get_coords_and_area(block_with_poly):
+ """Extract coordinates and area from a table."""
+ xmin, ymin, xmax, ymax = _get_bbox(block_with_poly)
+ area = (xmax - xmin) * (ymax - ymin)
+ return xmin, ymin, xmax, ymax, area
+
+
+def calculate_intersection(box1, box2):
+ """Calculate intersection coordinates between two boxes."""
+ intersection_xmin = max(box1[0], box2[0])
+ intersection_ymin = max(box1[1], box2[1])
+ intersection_xmax = min(box1[2], box2[2])
+ intersection_ymax = min(box1[3], box2[3])
+
+ # Check if intersection is valid
+ if intersection_xmax <= intersection_xmin or intersection_ymax <= intersection_ymin:
+ return None
+
+ return intersection_xmin, intersection_ymin, intersection_xmax, intersection_ymax
+
+
+def is_inside(small_box, big_box, overlap_threshold=0.8):
+ """Check if small_box is inside big_box by at least overlap_threshold."""
+ intersection = calculate_intersection(small_box[:4], big_box[:4])
+
+ if not intersection:
+ return False
+
+ intersection_xmin, intersection_ymin, intersection_xmax, intersection_ymax = intersection
+ intersection_area = (intersection_xmax - intersection_xmin) * (intersection_ymax - intersection_ymin)
+
+ # Check if overlap exceeds threshold
+ return intersection_area >= overlap_threshold * small_box[4]
+
+
+def remove_nested_ocr_text_blocks(
+ ocr_res_list,
+ layout_res,
+ overlap_threshold=0.8,
+ min_area_ratio=1.01,
+):
+ """Remove OCR candidate text blocks that are contained by any larger layout block."""
+ if not ocr_res_list or len(layout_res) < 2:
+ return ocr_res_list, []
+
+ layout_info = [(block, get_coords_and_area(block)) for block in layout_res]
+ blocks_to_remove = []
+
+ for text_block in ocr_res_list:
+ text_box = get_coords_and_area(text_block)
+ text_area = text_box[4]
+ for parent_block, parent_box in layout_info:
+ if parent_block is text_block:
+ continue
+ if parent_box[4] <= text_area * min_area_ratio:
+ continue
+ if is_inside(text_box, parent_box, overlap_threshold):
+ blocks_to_remove.append(text_block)
+ break
+
+ remove_ids = {id(block) for block in blocks_to_remove}
+ filtered_ocr_res_list = [
+ block for block in ocr_res_list if id(block) not in remove_ids
+ ]
+ return filtered_ocr_res_list, blocks_to_remove
+
+
+def get_res_list_from_layout_res(layout_res, overlap_threshold=0.8):
+ """Extract OCR, table and other regions from layout results."""
+ ocr_res_list = []
+ text_res_list = []
+ table_res_list = []
+ single_page_mfdetrec_res = []
+
+ # Categorize regions
+ for i, res in enumerate(layout_res):
+ label = res.get("label")
+
+ if label in ["display_formula", "inline_formula"]:
+ xmin, ymin, xmax, ymax = _get_bbox(res)
+ single_page_mfdetrec_res.append({
+ "bbox": [xmin, ymin, xmax, ymax],
+ })
+ elif label == "table":
+ table_res_list.append(res)
+ elif label in TEXT_REGION_LABELS:
+ text_res_list.append(res)
+
+ ocr_res_list.extend(text_res_list)
+
+ ocr_res_list, nested_text_need_remove = remove_nested_ocr_text_blocks(
+ ocr_res_list,
+ layout_res,
+ overlap_threshold=overlap_threshold,
+ )
+ nested_remove_ids = {id(res) for res in nested_text_need_remove}
+ if nested_remove_ids:
+ layout_res[:] = [res for res in layout_res if id(res) not in nested_remove_ids]
+
+ return ocr_res_list, table_res_list, single_page_mfdetrec_res
+
+
+def clean_memory(device='cuda'):
+ if str(device).startswith("cuda"):
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+ torch.cuda.ipc_collect()
+ elif str(device).startswith("npu"):
+ if torch_npu.npu.is_available():
+ torch_npu.npu.empty_cache()
+ elif str(device).startswith("mps"):
+ torch.mps.empty_cache()
+ elif str(device).startswith("gcu"):
+ if torch.gcu.is_available():
+ torch.gcu.empty_cache()
+ elif str(device).startswith("musa"):
+ if torch.musa.is_available():
+ torch.musa.empty_cache()
+ elif str(device).startswith("mlu"):
+ if torch.mlu.is_available():
+ torch.mlu.empty_cache()
+ elif str(device).startswith("sdaa"):
+ if torch.sdaa.is_available():
+ torch.sdaa.empty_cache()
+ gc.collect()
+
+
+def clean_vram(device, vram_threshold=8):
+ total_memory = get_vram(device)
+ if total_memory and total_memory <= vram_threshold:
+ gc_start = time.time()
+ clean_memory(device)
+ gc_time = round(time.time() - gc_start, 2)
+ logger.debug(f"gc time: {gc_time}")
+
+
+def get_vram(device) -> int:
+ env_vram = os.getenv("MINERU_VIRTUAL_VRAM_SIZE")
+
+ # 如果环境变量已配置,尝试解析并返回
+ if env_vram is not None:
+ try:
+ total_memory = int(env_vram)
+ if total_memory > 0:
+ return total_memory
+ else:
+ logger.warning(
+ f"MINERU_VIRTUAL_VRAM_SIZE value '{env_vram}' is not positive, falling back to auto-detection")
+ except ValueError:
+ logger.warning(
+ f"MINERU_VIRTUAL_VRAM_SIZE value '{env_vram}' is not a valid integer, falling back to auto-detection")
+
+ # 环境变量未配置或配置错误,根据device自动获取
+ total_memory = 1
+ if torch.cuda.is_available() and str(device).startswith("cuda"):
+ total_memory = round(torch.cuda.get_device_properties(device).total_memory / (1024 ** 3)) # 将字节转换为 GB
+ elif str(device).startswith("npu"):
+ if torch_npu.npu.is_available():
+ total_memory = round(torch_npu.npu.get_device_properties(device).total_memory / (1024 ** 3)) # 转为 GB
+ elif str(device).startswith("gcu"):
+ if torch.gcu.is_available():
+ total_memory = round(torch.gcu.get_device_properties(device).total_memory / (1024 ** 3)) # 转为 GB
+ elif str(device).startswith("musa"):
+ if torch.musa.is_available():
+ total_memory = round(torch.musa.get_device_properties(device).total_memory / (1024 ** 3)) # 转为 GB
+ elif str(device).startswith("mlu"):
+ if torch.mlu.is_available():
+ total_memory = round(torch.mlu.get_device_properties(device).total_memory / (1024 ** 3)) # 转为 GB
+ elif str(device).startswith("sdaa"):
+ if torch.sdaa.is_available():
+ total_memory = round(torch.sdaa.get_device_properties(device).total_memory / (1024 ** 3)) # 转为 GB
+
+ return total_memory
diff --git a/mineru/utils/models_download_utils.py b/mineru/utils/models_download_utils.py
new file mode 100644
index 000000000..7c9e20d1e
--- /dev/null
+++ b/mineru/utils/models_download_utils.py
@@ -0,0 +1,76 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+from huggingface_hub import snapshot_download as hf_snapshot_download
+from modelscope import snapshot_download as ms_snapshot_download
+
+from mineru.utils.config_reader import get_local_models_dir
+from mineru.utils.enum_class import ModelPath
+
+def auto_download_and_get_model_root_path(relative_path: str, repo_mode='pipeline') -> str:
+ """
+ 支持文件或目录的可靠下载。
+ - 如果输入文件: 返回本地文件绝对路径
+ - 如果输入目录: 返回本地缓存下与 relative_path 同结构的相对路径字符串
+ :param repo_mode: 指定仓库模式，'pipeline' 或 'vlm'
+ :param relative_path: 文件或目录相对路径
+ :return: 本地文件绝对路径或相对路径
+ """
+ model_source = os.getenv('MINERU_MODEL_SOURCE', "huggingface")
+
+ if model_source == 'local':
+ local_models_config = get_local_models_dir()
+ root_path = local_models_config.get(repo_mode, None)
+ if not root_path:
+ raise ValueError(f"Local path for repo_mode '{repo_mode}' is not configured.")
+ return root_path
+
+ # 建立仓库模式到路径的映射
+ repo_mapping = {
+ 'pipeline': {
+ 'huggingface': ModelPath.pipeline_root_hf,
+ 'modelscope': ModelPath.pipeline_root_modelscope,
+ 'default': ModelPath.pipeline_root_hf
+ },
+ 'vlm': {
+ 'huggingface': ModelPath.vlm_root_hf,
+ 'modelscope': ModelPath.vlm_root_modelscope,
+ 'default': ModelPath.vlm_root_hf
+ }
+ }
+
+ if repo_mode not in repo_mapping:
+ raise ValueError(f"Unsupported repo_mode: {repo_mode}, must be 'pipeline' or 'vlm'")
+
+ # 如果没有指定model_source或值不是'modelscope'，则使用默认值
+ repo = repo_mapping[repo_mode].get(model_source, repo_mapping[repo_mode]['default'])
+
+
+ if model_source == "huggingface":
+ snapshot_download = hf_snapshot_download
+ elif model_source == "modelscope":
+ snapshot_download = ms_snapshot_download
+ else:
+ raise ValueError(f"未知的仓库类型: {model_source}")
+
+ cache_dir = None
+
+ if repo_mode == 'pipeline':
+ relative_path = relative_path.strip('/')
+ cache_dir = snapshot_download(repo, allow_patterns=[relative_path, relative_path+"/*"])
+ elif repo_mode == 'vlm':
+ # VLM 模式下，根据 relative_path 的不同处理方式
+ if relative_path == "/":
+ cache_dir = snapshot_download(repo)
+ else:
+ relative_path = relative_path.strip('/')
+ cache_dir = snapshot_download(repo, allow_patterns=[relative_path, relative_path+"/*"])
+
+ if not cache_dir:
+ raise FileNotFoundError(f"Failed to download model: {relative_path} from {repo}")
+ return cache_dir
+
+
+if __name__ == '__main__':
+ path1 = "models/README.md"
+ root = auto_download_and_get_model_root_path(path1)
+ print("本地文件绝对路径:", os.path.join(root, path1))
\ No newline at end of file
diff --git a/mineru/utils/ocr_utils.py b/mineru/utils/ocr_utils.py
new file mode 100644
index 000000000..b80081dac
--- /dev/null
+++ b/mineru/utils/ocr_utils.py
@@ -0,0 +1,487 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import copy
+import cv2
+import numpy as np
+
+from .bbox_utils import normalize_to_int_bbox
+
+
+class OcrConfidence:
+ min_confidence = 0.5
+ min_width = 3
+
+LINE_WIDTH_TO_HEIGHT_RATIO_THRESHOLD = 4 # 一般情况下，行宽度超过高度4倍时才是一个正常的横向文本块
+TEXT_REC_ROTATE_RATIO = 1.5
+
+
+def merge_spans_to_line(spans, threshold=0.6):
+ if len(spans) == 0:
+ return []
+ else:
+ # 按照y0坐标排序
+ spans.sort(key=lambda span: span['bbox'][1])
+
+ lines = []
+ current_line = [spans[0]]
+ for span in spans[1:]:
+ # 如果当前的span与当前行的最后一个span在y轴上重叠，则添加到当前行
+ if _is_overlaps_y_exceeds_threshold(span['bbox'], current_line[-1]['bbox'], threshold):
+ current_line.append(span)
+ else:
+ # 否则，开始新行
+ lines.append(current_line)
+ current_line = [span]
+
+ # 添加最后一行
+ if current_line:
+ lines.append(current_line)
+
+ return lines
+
+def _is_overlaps_y_exceeds_threshold(bbox1,
+ bbox2,
+ overlap_ratio_threshold=0.8):
+ """检查两个bbox在y轴上是否有重叠，并且该重叠区域的高度占两个bbox高度更低的那个超过80%"""
+ _, y0_1, _, y1_1 = bbox1
+ _, y0_2, _, y1_2 = bbox2
+
+ overlap = max(0, min(y1_1, y1_2) - max(y0_1, y0_2))
+ height1, height2 = y1_1 - y0_1, y1_2 - y0_2
+ # max_height = max(height1, height2)
+ min_height = min(height1, height2)
+
+ return (overlap / min_height) > overlap_ratio_threshold if min_height > 0 else False
+
+
+def _is_overlaps_x_exceeds_threshold(bbox1,
+ bbox2,
+ overlap_ratio_threshold=0.8):
+ """检查两个bbox在x轴上是否有重叠，并且该重叠区域的宽度占两个bbox宽度更低的那个超过指定阈值"""
+ x0_1, _, x1_1, _ = bbox1
+ x0_2, _, x1_2, _ = bbox2
+
+ overlap = max(0, min(x1_1, x1_2) - max(x0_1, x0_2))
+ width1, width2 = x1_1 - x0_1, x1_2 - x0_2
+ min_width = min(width1, width2)
+
+ return (overlap / min_width) > overlap_ratio_threshold if min_width > 0 else False
+
+
+def img_decode(content: bytes):
+ np_arr = np.frombuffer(content, dtype=np.uint8)
+ return cv2.imdecode(np_arr, cv2.IMREAD_UNCHANGED)
+
+def check_img(img):
+ if isinstance(img, bytes):
+ img = img_decode(img)
+ if isinstance(img, np.ndarray) and len(img.shape) == 2:
+ img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
+ return img
+
+
+def alpha_to_color(img, alpha_color=(255, 255, 255)):
+ if len(img.shape) == 3 and img.shape[2] == 4:
+ B, G, R, A = cv2.split(img)
+ alpha = A / 255
+
+ R = (alpha_color[0] * (1 - alpha) + R * alpha).astype(np.uint8)
+ G = (alpha_color[1] * (1 - alpha) + G * alpha).astype(np.uint8)
+ B = (alpha_color[2] * (1 - alpha) + B * alpha).astype(np.uint8)
+
+ img = cv2.merge((B, G, R))
+ return img
+
+
+def preprocess_image(_image):
+ alpha_color = (255, 255, 255)
+ _image = alpha_to_color(_image, alpha_color)
+ return _image
+
+
+def sorted_boxes(dt_boxes):
+ """
+ Sort text boxes in order from top to bottom, left to right
+ args:
+ dt_boxes(array):detected text boxes with shape [4, 2]
+ return:
+ sorted boxes(array) with shape [4, 2]
+ """
+ num_boxes = len(dt_boxes)
+ sorted_boxes = sorted(dt_boxes, key=lambda x: (x[0][1], x[0][0]))
+ _boxes = list(sorted_boxes)
+
+ for i in range(num_boxes - 1):
+ for j in range(i, -1, -1):
+ if abs(_boxes[j + 1][0][1] - _boxes[j][0][1]) < 10 and \
+ (_boxes[j + 1][0][0] < _boxes[j][0][0]):
+ tmp = _boxes[j]
+ _boxes[j] = _boxes[j + 1]
+ _boxes[j + 1] = tmp
+ else:
+ break
+ return _boxes
+
+
+def bbox_to_points(bbox):
+ """ 将bbox格式转换为四个顶点的数组 """
+ x0, y0, x1, y1 = bbox
+ return np.array([[x0, y0], [x1, y0], [x1, y1], [x0, y1]]).astype('float32')
+
+
+def points_to_bbox(points):
+ """ 将四个顶点的数组转换为bbox格式 """
+ x0, y0 = points[0]
+ x1, _ = points[1]
+ _, y1 = points[2]
+ return [x0, y0, x1, y1]
+
+
+def merge_intervals(intervals):
+ # Sort the intervals based on the start value
+ intervals.sort(key=lambda x: x[0])
+
+ merged = []
+ for interval in intervals:
+ # If the list of merged intervals is empty or if the current
+ # interval does not overlap with the previous, simply append it.
+ if not merged or merged[-1][1] < interval[0]:
+ merged.append(interval)
+ else:
+ # Otherwise, there is overlap, so we merge the current and previous intervals.
+ merged[-1][1] = max(merged[-1][1], interval[1])
+
+ return merged
+
+
+def remove_intervals(original, masks):
+ # Merge all mask intervals
+ merged_masks = merge_intervals(masks)
+
+ result = []
+ original_start, original_end = original
+
+ for mask in merged_masks:
+ mask_start, mask_end = mask
+
+ # If the mask starts after the original range, ignore it
+ if mask_start > original_end:
+ continue
+
+ # If the mask ends before the original range starts, ignore it
+ if mask_end < original_start:
+ continue
+
+ # Remove the masked part from the original range
+ if original_start < mask_start:
+ result.append([original_start, mask_start - 1])
+
+ original_start = max(mask_end + 1, original_start)
+
+ # Add the remaining part of the original range, if any
+ if original_start <= original_end:
+ result.append([original_start, original_end])
+
+ return result
+
+
+def update_det_boxes(dt_boxes, mfd_res):
+ new_dt_boxes = []
+ angle_boxes_list = []
+ for text_box in dt_boxes:
+
+ if calculate_is_angle(text_box):
+ angle_boxes_list.append(text_box)
+ continue
+
+ text_bbox = points_to_bbox(text_box)
+ masks_list = []
+ for mf_box in mfd_res:
+ mf_bbox = mf_box['bbox']
+ if _is_overlaps_y_exceeds_threshold(text_bbox, mf_bbox):
+ masks_list.append([mf_bbox[0], mf_bbox[2]])
+ text_x_range = [text_bbox[0], text_bbox[2]]
+ text_remove_mask_range = remove_intervals(text_x_range, masks_list)
+ temp_dt_box = []
+ for text_remove_mask in text_remove_mask_range:
+ temp_dt_box.append(bbox_to_points([text_remove_mask[0], text_bbox[1], text_remove_mask[1], text_bbox[3]]))
+ if len(temp_dt_box) > 0:
+ new_dt_boxes.extend(temp_dt_box)
+
+ new_dt_boxes.extend(angle_boxes_list)
+
+ return new_dt_boxes
+
+
+def merge_overlapping_spans(spans):
+ """
+ Merges overlapping spans on the same line.
+
+ :param spans: A list of span coordinates [(x1, y1, x2, y2), ...]
+ :return: A list of merged spans
+ """
+ # Return an empty list if the input spans list is empty
+ if not spans:
+ return []
+
+ # Sort spans by their starting x-coordinate
+ spans.sort(key=lambda x: x[0])
+
+ # Initialize the list of merged spans
+ merged = []
+ for span in spans:
+ # Unpack span coordinates
+ x1, y1, x2, y2 = span
+ # If the merged list is empty or there's no horizontal overlap, add the span directly
+ if not merged or merged[-1][2] < x1:
+ merged.append(span)
+ else:
+ # If there is horizontal overlap, merge the current span with the previous one
+ last_span = merged.pop()
+ # Update the merged span's top-left corner to the smaller (x1, y1) and bottom-right to the larger (x2, y2)
+ x1 = min(last_span[0], x1)
+ y1 = min(last_span[1], y1)
+ x2 = max(last_span[2], x2)
+ y2 = max(last_span[3], y2)
+ # Add the merged span back to the list
+ merged.append((x1, y1, x2, y2))
+
+ # Return the list of merged spans
+ return merged
+
+
+def merge_det_boxes(dt_boxes):
+ """
+ Merge detection boxes.
+
+ This function takes a list of detected bounding boxes, each represented by four corner points.
+ The goal is to merge these bounding boxes into larger text regions.
+
+ Parameters:
+ dt_boxes (list): A list containing multiple text detection boxes, where each box is defined by four corner points.
+
+ Returns:
+ list: A list containing the merged text regions, where each region is represented by four corner points.
+ """
+ # Convert the detection boxes into a dictionary format with bounding boxes and type
+ dt_boxes_dict_list = []
+ angle_boxes_list = []
+ for text_box in dt_boxes:
+ text_bbox = points_to_bbox(text_box)
+
+ if calculate_is_angle(text_box):
+ angle_boxes_list.append(text_box)
+ continue
+
+ text_box_dict = {'bbox': text_bbox}
+ dt_boxes_dict_list.append(text_box_dict)
+
+ # Merge adjacent text regions into lines
+ lines = merge_spans_to_line(dt_boxes_dict_list)
+
+ # Initialize a new list for storing the merged text regions
+ new_dt_boxes = []
+ for line in lines:
+ line_bbox_list = []
+ for span in line:
+ line_bbox_list.append(span['bbox'])
+
+ # 计算整行的宽度和高度
+ min_x = min(bbox[0] for bbox in line_bbox_list)
+ max_x = max(bbox[2] for bbox in line_bbox_list)
+ min_y = min(bbox[1] for bbox in line_bbox_list)
+ max_y = max(bbox[3] for bbox in line_bbox_list)
+ line_width = max_x - min_x
+ line_height = max_y - min_y
+
+ # 只有当行宽度超过高度4倍时才进行合并
+ if line_width > line_height * LINE_WIDTH_TO_HEIGHT_RATIO_THRESHOLD:
+
+ # Merge overlapping text regions within the same line
+ merged_spans = merge_overlapping_spans(line_bbox_list)
+
+ # Convert the merged text regions back to point format and add them to the new detection box list
+ for span in merged_spans:
+ new_dt_boxes.append(bbox_to_points(span))
+ else:
+ # 不进行合并，直接添加原始区域
+ for bbox in line_bbox_list:
+ new_dt_boxes.append(bbox_to_points(bbox))
+
+ new_dt_boxes.extend(angle_boxes_list)
+
+ return new_dt_boxes
+
+
+def get_adjusted_mfdetrec_res(single_page_mfdetrec_res, useful_list):
+ paste_x, paste_y, xmin, ymin, xmax, ymax, new_width, new_height = useful_list
+ # Adjust the coordinates of the formula area
+ adjusted_mfdetrec_res = []
+ for mf_res in single_page_mfdetrec_res:
+ mf_xmin, mf_ymin, mf_xmax, mf_ymax = mf_res["bbox"]
+ # Adjust the coordinates of the formula area to the coordinates relative to the cropping area
+ x0 = mf_xmin - xmin + paste_x
+ y0 = mf_ymin - ymin + paste_y
+ x1 = mf_xmax - xmin + paste_x
+ y1 = mf_ymax - ymin + paste_y
+ # Filter formula blocks outside the graph
+ if any([x1 < 0, y1 < 0]) or any([x0 > new_width, y0 > new_height]):
+ continue
+ else:
+ adjusted_mfdetrec_res.append({
+ "bbox": [x0, y0, x1, y1],
+ })
+ return adjusted_mfdetrec_res
+
+
+def get_ocr_result_list(
+ ocr_res,
+ useful_list,
+ ocr_enable,
+ bgr_image,
+ lang,
+):
+ paste_x, paste_y, xmin, ymin, xmax, ymax, new_width, new_height = useful_list
+ ocr_result_list = []
+ ori_im = bgr_image.copy()
+ for box_ocr_res in ocr_res:
+ img_crop = None
+ need_ocr_rec = False
+
+ if len(box_ocr_res) == 2:
+ p1, p2, p3, p4 = box_ocr_res[0]
+ text, score = box_ocr_res[1]
+ # logger.info(f"text: {text}, score: {score}")
+ if score < OcrConfidence.min_confidence: # 过滤低置信度的结果
+ continue
+ else:
+ p1, p2, p3, p4 = box_ocr_res
+ text, score = "", 1
+
+ if ocr_enable:
+ tmp_box = copy.deepcopy(np.array([p1, p2, p3, p4]).astype('float32'))
+ img_crop = get_rotate_crop_image_for_text_rec(ori_im, tmp_box)
+ need_ocr_rec = True
+
+ # average_angle_degrees = calculate_angle_degrees(box_ocr_res[0])
+ # if average_angle_degrees > 0.5:
+ poly = [p1, p2, p3, p4]
+
+ if (p3[0] - p1[0]) < OcrConfidence.min_width:
+ # logger.info(f"width too small: {p3[0] - p1[0]}, text: {text}")
+ continue
+
+ if calculate_is_angle(poly):
+ # logger.info(f"average_angle_degrees: {average_angle_degrees}, text: {text}")
+ # 与x轴的夹角超过0.5度，对边界做一下矫正
+ # 计算几何中心
+ x_center = sum(point[0] for point in poly) / 4
+ y_center = sum(point[1] for point in poly) / 4
+ new_height = ((p4[1] - p1[1]) + (p3[1] - p2[1])) / 2
+ new_width = p3[0] - p1[0]
+ p1 = [x_center - new_width / 2, y_center - new_height / 2]
+ p2 = [x_center + new_width / 2, y_center - new_height / 2]
+ p3 = [x_center + new_width / 2, y_center + new_height / 2]
+ p4 = [x_center - new_width / 2, y_center + new_height / 2]
+
+ # Convert the coordinates back to the original coordinate system
+ p1 = [p1[0] - paste_x + xmin, p1[1] - paste_y + ymin]
+ p2 = [p2[0] - paste_x + xmin, p2[1] - paste_y + ymin]
+ p3 = [p3[0] - paste_x + xmin, p3[1] - paste_y + ymin]
+ p4 = [p4[0] - paste_x + xmin, p4[1] - paste_y + ymin]
+
+ bbox = normalize_to_int_bbox([p1, p2, p3, p4])
+ if bbox is None:
+ continue
+
+ ocr_item = {
+ "label": "ocr_text",
+ "bbox": bbox,
+ "score": 1.0 if ocr_enable else float(round(score, 2)),
+ "text": text,
+ }
+ if need_ocr_rec:
+ ocr_item["np_img"] = img_crop
+ ocr_item["lang"] = lang
+ ocr_item["_need_ocr_rec"] = True
+ ocr_result_list.append(ocr_item)
+
+ return ocr_result_list
+
+
+def calculate_is_angle(poly):
+ p1, p2, p3, p4 = poly
+ height = ((p4[1] - p1[1]) + (p3[1] - p2[1])) / 2
+ if 0.8 * height <= (p3[1] - p1[1]) <= 1.2 * height:
+ return False
+ else:
+ # logger.info((p3[1] - p1[1])/height)
+ return True
+
+def is_bbox_aligned_rect(points):
+ x_coords = points[:, 0]
+ y_coords = points[:, 1]
+ unique_x = np.unique(x_coords)
+ unique_y = np.unique(y_coords)
+ return len(unique_x) == 2 and len(unique_y) == 2
+
+def get_rotate_crop_image(img, points):
+ '''
+ img_height, img_width = img.shape[0:2]
+ left = int(np.min(points[:, 0]))
+ right = int(np.max(points[:, 0]))
+ top = int(np.min(points[:, 1]))
+ bottom = int(np.max(points[:, 1]))
+ img_crop = img[top:bottom, left:right, :].copy()
+ points[:, 0] = points[:, 0] - left
+ points[:, 1] = points[:, 1] - top
+ '''
+ assert len(points) == 4, "shape of points must be 4*2"
+
+ if is_bbox_aligned_rect(points):
+ xmin = int(np.min(points[:, 0]))
+ xmax = int(np.max(points[:, 0]))
+ ymin = int(np.min(points[:, 1]))
+ ymax = int(np.max(points[:, 1]))
+ new_img = img[ymin:ymax, xmin:xmax].copy()
+ if new_img.shape[0] > 0 and new_img.shape[1] > 0:
+ return new_img
+
+ img_crop_width = int(
+ max(
+ np.linalg.norm(points[0] - points[1]),
+ np.linalg.norm(points[2] - points[3])))
+ img_crop_height = int(
+ max(
+ np.linalg.norm(points[0] - points[3]),
+ np.linalg.norm(points[1] - points[2])))
+ pts_std = np.float32([[0, 0], [img_crop_width, 0],
+ [img_crop_width, img_crop_height],
+ [0, img_crop_height]])
+ M = cv2.getPerspectiveTransform(points, pts_std)
+ dst_img = cv2.warpPerspective(
+ img,
+ M, (img_crop_width, img_crop_height),
+ borderMode=cv2.BORDER_REPLICATE,
+ flags=cv2.INTER_CUBIC)
+ dst_img_height, dst_img_width = dst_img.shape[0:2]
+ if dst_img_height * 1.0 / dst_img_width >= TEXT_REC_ROTATE_RATIO:
+ dst_img = np.rot90(dst_img)
+ return dst_img
+
+
+def rotate_vertical_crop_if_needed(crop_img, rotate_ratio=TEXT_REC_ROTATE_RATIO):
+ if crop_img is None or crop_img.size == 0:
+ return crop_img
+
+ crop_height, crop_width = crop_img.shape[:2]
+ if crop_width == 0:
+ return crop_img
+
+ if crop_height * 1.0 / crop_width >= rotate_ratio:
+ return np.rot90(crop_img)
+ return crop_img
+
+
+def get_rotate_crop_image_for_text_rec(img, points):
+ crop_img = get_rotate_crop_image(img, points)
+ return rotate_vertical_crop_if_needed(crop_img)
diff --git a/mineru/utils/os_env_config.py b/mineru/utils/os_env_config.py
new file mode 100644
index 000000000..5ee35d9b2
--- /dev/null
+++ b/mineru/utils/os_env_config.py
@@ -0,0 +1,36 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+
+
+def get_op_num_threads(env_name: str) -> int:
+ env_value = os.getenv(env_name, None)
+ return get_value_from_string(env_value, -1)
+
+
+def get_load_images_timeout() -> int:
+ env_value = os.getenv('MINERU_PDF_RENDER_TIMEOUT', None)
+ return get_value_from_string(env_value, 300)
+
+
+def get_load_images_threads() -> int:
+ env_value = os.getenv('MINERU_PDF_RENDER_THREADS', None)
+ return get_value_from_string(env_value, 4)
+
+
+def get_value_from_string(env_value: str, default_value: int) -> int:
+ if env_value is not None:
+ try:
+ num_threads = int(env_value)
+ if num_threads > 0:
+ return num_threads
+ except ValueError:
+ return default_value
+ return default_value
+
+
+if __name__ == '__main__':
+ print(get_value_from_string('1', -1))
+ print(get_value_from_string('0', -1))
+ print(get_value_from_string('-1', -1))
+ print(get_value_from_string('abc', -1))
+ print(get_load_images_timeout())
\ No newline at end of file
diff --git a/mineru/utils/pdf_classify.py b/mineru/utils/pdf_classify.py
new file mode 100644
index 000000000..93bbf9198
--- /dev/null
+++ b/mineru/utils/pdf_classify.py
@@ -0,0 +1,563 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import os
+import re
+from io import BytesIO
+
+import numpy as np
+import pypdfium2 as pdfium
+import pypdfium2.raw as pdfium_c
+from loguru import logger
+from pypdf import PdfReader
+from pdfminer.converter import PDFPageAggregator
+from pdfminer.high_level import extract_text
+from pdfminer.layout import LAParams, LTFigure, LTImage
+from pdfminer.pdfdocument import PDFDocument
+from pdfminer.pdfinterp import PDFPageInterpreter, PDFResourceManager
+from pdfminer.pdfpage import PDFPage
+from pdfminer.pdfparser import PDFParser
+from mineru.utils.pdfium_guard import (
+ close_pdfium_document,
+ open_pdfium_document,
+ pdfium_guard,
+)
+
+PDF_CLASSIFY_STRATEGY_ENV = "MINERU_PDF_CLASSIFY_STRATEGY"
+PDF_CLASSIFY_STRATEGY_HYBRID = "hybrid"
+PDF_CLASSIFY_STRATEGY_LEGACY = "legacy"
+
+MAX_SAMPLE_PAGES = 10
+CHARS_THRESHOLD = 50
+HIGH_IMAGE_COVERAGE_THRESHOLD = 0.8
+CID_RATIO_THRESHOLD = 0.05
+TEXT_QUALITY_MIN_CHARS = 300
+TEXT_QUALITY_BAD_THRESHOLD = 0.03
+TEXT_QUALITY_GOOD_THRESHOLD = 0.005
+MAX_PAGE_ASPECT_RATIO = 10.0
+
+_ALLOWED_CONTROL_CODES = {9, 10, 13}
+_PRIVATE_USE_AREA_START = 0xE000
+_PRIVATE_USE_AREA_END = 0xF8FF
+
+
+def _is_disallowed_control_unicode(unicode_code: int) -> bool:
+ return (
+ (
+ 0 <= unicode_code < 32
+ or 127 <= unicode_code <= 159
+ )
+ and unicode_code not in _ALLOWED_CONTROL_CODES
+ )
+
+
+def classify(pdf_bytes):
+ """
+ Classify a PDF as text-based or OCR-based.
+
+ Returns:
+ "txt" if the PDF can be parsed as text, otherwise "ocr".
+ """
+
+ strategy = get_pdf_classify_strategy()
+ if strategy == PDF_CLASSIFY_STRATEGY_LEGACY:
+ return classify_legacy(pdf_bytes)
+ return classify_hybrid(pdf_bytes)
+
+
+def get_pdf_classify_strategy() -> str:
+ strategy = os.getenv(
+ PDF_CLASSIFY_STRATEGY_ENV, PDF_CLASSIFY_STRATEGY_HYBRID
+ ).strip().lower()
+ if strategy not in {
+ PDF_CLASSIFY_STRATEGY_HYBRID,
+ PDF_CLASSIFY_STRATEGY_LEGACY,
+ }:
+ logger.warning(
+ f"Invalid {PDF_CLASSIFY_STRATEGY_ENV} value: {strategy}, "
+ f"fall back to {PDF_CLASSIFY_STRATEGY_HYBRID}"
+ )
+ return PDF_CLASSIFY_STRATEGY_HYBRID
+ return strategy
+
+
+def classify_hybrid(pdf_bytes):
+ """
+ Fast PDF classification path.
+
+ The hybrid path uses pdfium + pypdf as the main path and falls back to
+ pdfminer only for gray-zone samples.
+ """
+
+ pdf = None
+ page_indices = []
+ should_run_pdfminer_fallback = False
+
+ try:
+ with pdfium_guard():
+ pdf = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ page_count = len(pdf)
+ if page_count == 0:
+ return "ocr"
+
+ page_indices = get_sample_page_indices(page_count, MAX_SAMPLE_PAGES)
+ if not page_indices:
+ return "ocr"
+
+ extreme_page_index, extreme_ratio = get_extreme_aspect_ratio_page_pdfium(
+ pdf,
+ page_indices,
+ )
+ if extreme_page_index is not None:
+ logger.info(
+ "Classify PDF as OCR due to extreme sampled-page aspect ratio: "
+ f"page={extreme_page_index + 1}, ratio={extreme_ratio:.2f}"
+ )
+ return "ocr"
+
+ if (
+ get_avg_cleaned_chars_per_page_pdfium(pdf, page_indices)
+ < CHARS_THRESHOLD
+ ):
+ return "ocr"
+
+ if detect_cid_font_signal_pypdf(pdf_bytes, page_indices):
+ return "ocr"
+
+ text_quality_signal = get_text_quality_signal_pdfium(pdf, page_indices)
+ total_chars = text_quality_signal["total_chars"]
+ abnormal_ratio = text_quality_signal["abnormal_ratio"]
+
+ if total_chars >= TEXT_QUALITY_MIN_CHARS:
+ if abnormal_ratio >= TEXT_QUALITY_BAD_THRESHOLD:
+ return "ocr"
+ should_run_pdfminer_fallback = abnormal_ratio > TEXT_QUALITY_GOOD_THRESHOLD
+ else:
+ should_run_pdfminer_fallback = True
+
+ if (
+ get_high_image_coverage_ratio_pdfium(pdf, page_indices)
+ >= HIGH_IMAGE_COVERAGE_THRESHOLD
+ ):
+ return "ocr"
+
+ except Exception as e:
+ logger.error(f"Failed to classify PDF with hybrid strategy: {e}")
+ return "ocr"
+
+ finally:
+ close_pdfium_document(pdf)
+
+ if should_run_pdfminer_fallback:
+ sample_pdf_bytes = extract_selected_pages(pdf_bytes, page_indices)
+ if not sample_pdf_bytes:
+ return "ocr"
+ if detect_invalid_chars_pdfminer_fallback(sample_pdf_bytes):
+ return "ocr"
+
+ return "txt"
+
+
+def classify_legacy(pdf_bytes):
+ """
+ Legacy classification path kept for rollback and A/B comparison.
+ """
+
+ sample_pdf_bytes = extract_pages(pdf_bytes)
+ if not sample_pdf_bytes:
+ return "ocr"
+ pdf = None
+ try:
+ with pdfium_guard():
+ pdf = open_pdfium_document(pdfium.PdfDocument, sample_pdf_bytes)
+ page_count = len(pdf)
+ if page_count == 0:
+ return "ocr"
+
+ pages_to_check = min(page_count, MAX_SAMPLE_PAGES)
+
+ if (
+ get_avg_cleaned_chars_per_page(pdf, pages_to_check) < CHARS_THRESHOLD
+ ) or detect_invalid_chars(sample_pdf_bytes):
+ return "ocr"
+
+ if (
+ get_high_image_coverage_ratio(sample_pdf_bytes, pages_to_check)
+ >= HIGH_IMAGE_COVERAGE_THRESHOLD
+ ):
+ return "ocr"
+
+ return "txt"
+
+ except Exception as e:
+ logger.warning(f"Failed to classify PDF with legacy strategy: {e}")
+ return "ocr"
+
+ finally:
+ close_pdfium_document(pdf)
+
+
+def get_sample_page_indices(page_count: int, max_pages: int = MAX_SAMPLE_PAGES):
+ if page_count <= 0 or max_pages <= 0:
+ return []
+
+ sample_count = min(page_count, max_pages)
+ if sample_count == page_count:
+ return list(range(page_count))
+ if sample_count == 1:
+ return [0]
+
+ indices = []
+ seen = set()
+ for i in range(sample_count):
+ page_index = round(i * (page_count - 1) / (sample_count - 1))
+ page_index = max(0, min(page_count - 1, page_index))
+ if page_index not in seen:
+ indices.append(page_index)
+ seen.add(page_index)
+
+ if len(indices) < sample_count:
+ for page_index in range(page_count):
+ if page_index in seen:
+ continue
+ indices.append(page_index)
+ seen.add(page_index)
+ if len(indices) == sample_count:
+ break
+
+ return sorted(indices)
+
+
+def get_extreme_aspect_ratio_page_pdfium(
+ pdf_doc,
+ page_indices,
+ max_page_aspect_ratio: float = MAX_PAGE_ASPECT_RATIO,
+):
+ for page_index in page_indices:
+ page = pdf_doc[page_index]
+ page_width, page_height = page.get_size()
+ if page_width <= 0 or page_height <= 0:
+ continue
+
+ aspect_ratio = max(page_width / page_height, page_height / page_width)
+ if aspect_ratio > max_page_aspect_ratio:
+ return page_index, aspect_ratio
+
+ return None, None
+
+
+def get_avg_cleaned_chars_per_page(pdf_doc, pages_to_check):
+ total_chars = 0
+ cleaned_total_chars = 0
+
+ for i in range(pages_to_check):
+ page = pdf_doc[i]
+ text_page = page.get_textpage()
+ text = text_page.get_text_bounded()
+ total_chars += len(text)
+ cleaned_text = re.sub(r"\s+", "", text)
+ cleaned_total_chars += len(cleaned_text)
+
+ avg_cleaned_chars_per_page = cleaned_total_chars / pages_to_check
+ return avg_cleaned_chars_per_page
+
+
+def get_avg_cleaned_chars_per_page_pdfium(pdf_doc, page_indices):
+ cleaned_total_chars = 0
+
+ for page_index in page_indices:
+ page = pdf_doc[page_index]
+ text_page = page.get_textpage()
+ text = text_page.get_text_bounded()
+ cleaned_total_chars += len(re.sub(r"\s+", "", text))
+
+ if not page_indices:
+ return 0.0
+ return cleaned_total_chars / len(page_indices)
+
+
+def get_text_quality_signal_pdfium(pdf_doc, page_indices):
+ total_chars = 0
+ null_char_count = 0
+ replacement_char_count = 0
+ control_char_count = 0
+ private_use_char_count = 0
+
+ for page_index in page_indices:
+ page = pdf_doc[page_index]
+ text_page = page.get_textpage()
+ char_count = text_page.count_chars()
+ total_chars += char_count
+
+ for char_index in range(char_count):
+ unicode_code = pdfium_c.FPDFText_GetUnicode(text_page, char_index)
+ if unicode_code == 0:
+ null_char_count += 1
+ elif unicode_code == 0xFFFD:
+ replacement_char_count += 1
+ elif _is_disallowed_control_unicode(unicode_code):
+ control_char_count += 1
+ elif _PRIVATE_USE_AREA_START <= unicode_code <= _PRIVATE_USE_AREA_END:
+ private_use_char_count += 1
+
+ abnormal_chars = (
+ null_char_count
+ + replacement_char_count
+ + control_char_count
+ + private_use_char_count
+ )
+
+ abnormal_ratio = 0.0
+ if total_chars > 0:
+ abnormal_ratio = abnormal_chars / total_chars
+
+ return {
+ "total_chars": total_chars,
+ "abnormal_ratio": abnormal_ratio,
+ "null_char_count": null_char_count,
+ "replacement_char_count": replacement_char_count,
+ "control_char_count": control_char_count,
+ "private_use_char_count": private_use_char_count,
+ }
+
+
+def detect_cid_font_signal_pypdf(pdf_bytes, page_indices):
+ reader = PdfReader(BytesIO(pdf_bytes))
+
+ for page_index in page_indices:
+ page = reader.pages[page_index]
+ resources = _resolve_pdf_object(page.get("/Resources"))
+ if not resources:
+ continue
+
+ fonts = _resolve_pdf_object(resources.get("/Font"))
+ if not fonts:
+ continue
+
+ for _, font_ref in fonts.items():
+ font = _resolve_pdf_object(font_ref)
+ if not font:
+ continue
+
+ subtype = str(font.get("/Subtype"))
+ encoding = str(font.get("/Encoding"))
+ has_descendant_fonts = "/DescendantFonts" in font
+ has_to_unicode = "/ToUnicode" in font
+
+ if (
+ subtype == "/Type0"
+ and encoding in ("/Identity-H", "/Identity-V")
+ and has_descendant_fonts
+ and not has_to_unicode
+ ):
+ return True
+
+ return False
+
+
+def _resolve_pdf_object(obj):
+ if hasattr(obj, "get_object"):
+ return obj.get_object()
+ return obj
+
+
+def get_high_image_coverage_ratio(sample_pdf_bytes, pages_to_check):
+ pdf_stream = BytesIO(sample_pdf_bytes)
+ parser = PDFParser(pdf_stream)
+ document = PDFDocument(parser)
+
+ if not document.is_extractable:
+ return 1.0
+
+ rsrcmgr = PDFResourceManager()
+ laparams = LAParams(
+ line_overlap=0.5,
+ char_margin=2.0,
+ line_margin=0.5,
+ word_margin=0.1,
+ boxes_flow=None,
+ detect_vertical=False,
+ all_texts=False,
+ )
+ device = PDFPageAggregator(rsrcmgr, laparams=laparams)
+ interpreter = PDFPageInterpreter(rsrcmgr, device)
+
+ high_image_coverage_pages = 0
+ page_count = 0
+
+ for page in PDFPage.create_pages(document):
+ if page_count >= pages_to_check:
+ break
+
+ interpreter.process_page(page)
+ layout = device.get_result()
+
+ page_width = layout.width
+ page_height = layout.height
+ page_area = page_width * page_height
+
+ image_area = 0
+ for element in layout:
+ if isinstance(element, (LTImage, LTFigure)):
+ img_width = element.width
+ img_height = element.height
+ image_area += img_width * img_height
+
+ coverage_ratio = min(image_area / page_area, 1.0) if page_area > 0 else 0
+ if coverage_ratio >= HIGH_IMAGE_COVERAGE_THRESHOLD:
+ high_image_coverage_pages += 1
+
+ page_count += 1
+
+ pdf_stream.close()
+
+ if page_count == 0:
+ return 0.0
+
+ return high_image_coverage_pages / page_count
+
+
+def get_high_image_coverage_ratio_pdfium(pdf_doc, page_indices):
+ high_image_coverage_pages = 0
+
+ for page_index in page_indices:
+ page = pdf_doc[page_index]
+ page_bbox = page.get_bbox()
+ page_area = abs(
+ (page_bbox[2] - page_bbox[0]) * (page_bbox[3] - page_bbox[1])
+ )
+ image_area = 0.0
+
+ for page_object in page.get_objects(
+ filter=[pdfium_c.FPDF_PAGEOBJ_IMAGE], max_depth=3
+ ):
+ left, bottom, right, top = page_object.get_pos()
+ image_area += max(0.0, right - left) * max(0.0, top - bottom)
+
+ coverage_ratio = min(image_area / page_area, 1.0) if page_area > 0 else 0.0
+ if coverage_ratio >= HIGH_IMAGE_COVERAGE_THRESHOLD:
+ high_image_coverage_pages += 1
+
+ if not page_indices:
+ return 0.0
+ return high_image_coverage_pages / len(page_indices)
+
+
+def extract_pages(src_pdf_bytes: bytes) -> bytes:
+ """
+ Extract up to 10 random pages and return them as a new PDF.
+ """
+
+ pdf = None
+ sample_docs = None
+ try:
+ with pdfium_guard():
+ pdf = open_pdfium_document(pdfium.PdfDocument, src_pdf_bytes)
+ total_page = len(pdf)
+ if total_page == 0:
+ logger.warning("PDF is empty, return empty document")
+ return b""
+
+ if total_page <= MAX_SAMPLE_PAGES:
+ return src_pdf_bytes
+
+ select_page_cnt = min(MAX_SAMPLE_PAGES, total_page)
+ page_indices = np.random.choice(
+ total_page, select_page_cnt, replace=False
+ ).tolist()
+
+ sample_docs = open_pdfium_document(pdfium.PdfDocument.new)
+ sample_docs.import_pages(pdf, page_indices)
+
+ output_buffer = BytesIO()
+ sample_docs.save(output_buffer)
+ return output_buffer.getvalue()
+ except Exception as e:
+ logger.exception(e)
+ return src_pdf_bytes
+ finally:
+ close_pdfium_document(pdf)
+ close_pdfium_document(sample_docs)
+
+
+def extract_selected_pages(src_pdf_bytes: bytes, page_indices) -> bytes:
+ """
+ Extract specific pages and return them as a new PDF.
+ """
+
+ selected_page_indices = sorted(set(page_indices))
+ if not selected_page_indices:
+ return b""
+
+ pdf = None
+ sample_docs = None
+ try:
+ with pdfium_guard():
+ pdf = open_pdfium_document(pdfium.PdfDocument, src_pdf_bytes)
+ total_page = len(pdf)
+ if total_page == 0:
+ logger.warning("PDF is empty, return empty document")
+ return b""
+
+ selected_page_indices = [
+ page_index
+ for page_index in selected_page_indices
+ if 0 <= page_index < total_page
+ ]
+ if not selected_page_indices:
+ return b""
+
+ if selected_page_indices == list(range(total_page)):
+ return src_pdf_bytes
+
+ sample_docs = open_pdfium_document(pdfium.PdfDocument.new)
+ sample_docs.import_pages(pdf, selected_page_indices)
+
+ output_buffer = BytesIO()
+ sample_docs.save(output_buffer)
+ return output_buffer.getvalue()
+ except Exception as e:
+ logger.exception(e)
+ return src_pdf_bytes
+ finally:
+ close_pdfium_document(pdf)
+ close_pdfium_document(sample_docs)
+
+
+def detect_invalid_chars(sample_pdf_bytes: bytes) -> bool:
+ """
+ Detect whether a PDF contains invalid CID-style extracted text.
+ """
+
+ sample_pdf_file_like_object = BytesIO(sample_pdf_bytes)
+ laparams = LAParams(
+ line_overlap=0.5,
+ char_margin=2.0,
+ line_margin=0.5,
+ word_margin=0.1,
+ boxes_flow=None,
+ detect_vertical=False,
+ all_texts=False,
+ )
+ text = extract_text(pdf_file=sample_pdf_file_like_object, laparams=laparams)
+ text = text.replace("\n", "")
+
+ cid_pattern = re.compile(r"\(cid:\d+\)")
+ matches = cid_pattern.findall(text)
+ cid_count = len(matches)
+ cid_len = sum(len(match) for match in matches)
+ text_len = len(text)
+ if text_len == 0:
+ cid_chars_ratio = 0
+ else:
+ cid_chars_ratio = cid_count / (cid_count + text_len - cid_len)
+
+ return cid_chars_ratio > CID_RATIO_THRESHOLD
+
+
+def detect_invalid_chars_pdfminer_fallback(sample_pdf_bytes: bytes) -> bool:
+ return detect_invalid_chars(sample_pdf_bytes)
+
+
+if __name__ == "__main__":
+ with open("/Users/myhloli/pdf/luanma2x10.pdf", "rb") as f:
+ p_bytes = f.read()
+ logger.info(f"PDF classify result: {classify(p_bytes)}")
diff --git a/mineru/utils/pdf_image_tools.py b/mineru/utils/pdf_image_tools.py
new file mode 100644
index 000000000..fb11649f8
--- /dev/null
+++ b/mineru/utils/pdf_image_tools.py
@@ -0,0 +1,437 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import atexit
+import multiprocessing
+import os
+import signal
+import threading
+import time
+from io import BytesIO
+
+import numpy as np
+import pypdfium2 as pdfium
+from loguru import logger
+from PIL import Image, ImageOps
+
+from mineru.data.data_reader_writer import FileBasedDataWriter
+from mineru.utils.check_sys_env import is_windows_environment
+from mineru.utils.bbox_utils import normalize_to_int_bbox
+from mineru.utils.os_env_config import get_load_images_timeout, get_load_images_threads
+from mineru.utils.pdf_reader import image_to_b64str, image_to_bytes, page_to_image
+from mineru.utils.enum_class import ImageType
+from mineru.utils.hash_utils import str_sha256
+from mineru.utils.pdf_page_id import get_end_page_id
+from mineru.utils.pdfium_guard import (
+ close_pdfium_document,
+ get_pdfium_document_page_count,
+ open_pdfium_document,
+ pdfium_guard,
+)
+
+from concurrent.futures import ProcessPoolExecutor, wait, ALL_COMPLETED
+from concurrent.futures.process import BrokenProcessPool
+
+
+DEFAULT_PDF_IMAGE_DPI = 200
+# DEFAULT_PDF_IMAGE_DPI = 144
+MAX_PDF_RENDER_PROCESSES = 4
+MIN_PAGES_PER_RENDER_PROCESS = 30
+
+_pdf_render_executor: ProcessPoolExecutor | None = None
+_pdf_render_executor_lock = threading.Lock()
+
+
+def pdf_page_to_image(
+ page: pdfium.PdfPage,
+ dpi=DEFAULT_PDF_IMAGE_DPI,
+ image_type=ImageType.PIL,
+) -> dict:
+ """Convert pdfium.PdfDocument to image, Then convert the image to base64.
+
+ Args:
+ page (_type_): pdfium.PdfPage
+ dpi (int, optional): reset the dpi of dpi. Defaults to DEFAULT_PDF_IMAGE_DPI.
+ image_type (ImageType, optional): The type of image to return. Defaults to ImageType.PIL.
+
+ Returns:
+ dict: {'img_base64': str, 'img_pil': pil_img, 'scale': float }
+ """
+ pil_img, scale = page_to_image(page, dpi=dpi)
+ image_dict = {
+ "scale": scale,
+ }
+ if image_type == ImageType.BASE64:
+ image_dict["img_base64"] = image_to_b64str(pil_img)
+ else:
+ image_dict["img_pil"] = pil_img
+
+ return image_dict
+
+
+def _load_images_from_pdf_worker(
+ pdf_bytes, dpi, start_page_id, end_page_id, image_type
+):
+ """用于进程池的包装函数"""
+ return load_images_from_pdf_core(
+ pdf_bytes, dpi, start_page_id, end_page_id, image_type
+ )
+
+
+def _calculate_render_process_count(total_pages: int, threads: int, cpu_count=None) -> int:
+ requested_threads = max(1, threads)
+ available_cpus = max(1, cpu_count if cpu_count is not None else (os.cpu_count() or 1))
+ page_limited_threads = max(1, total_pages // MIN_PAGES_PER_RENDER_PROCESS)
+ return min(
+ available_cpus,
+ requested_threads,
+ MAX_PDF_RENDER_PROCESSES,
+ page_limited_threads,
+ )
+
+
+def _build_render_page_ranges(
+ start_page_id: int,
+ end_page_id: int,
+ process_count: int,
+) -> list[tuple[int, int]]:
+ total_pages = end_page_id - start_page_id + 1
+ base_pages, remainder = divmod(total_pages, process_count)
+ page_ranges = []
+ current_page = start_page_id
+
+ for process_idx in range(process_count):
+ pages_in_range = base_pages + (1 if process_idx < remainder else 0)
+ range_end = current_page + pages_in_range - 1
+ page_ranges.append((current_page, range_end))
+ current_page = range_end + 1
+
+ return page_ranges
+
+
+def _get_render_process_plan(
+ start_page_id: int,
+ end_page_id: int,
+ threads: int,
+ cpu_count=None,
+) -> tuple[int, list[tuple[int, int]]]:
+ total_pages = end_page_id - start_page_id + 1
+ actual_threads = _calculate_render_process_count(total_pages, threads, cpu_count)
+ return actual_threads, _build_render_page_ranges(
+ start_page_id, end_page_id, actual_threads
+ )
+
+
+def _get_pdf_render_pool_capacity(cpu_count=None) -> int:
+ available_cpus = max(1, cpu_count if cpu_count is not None else (os.cpu_count() or 1))
+ configured_threads = max(1, get_load_images_threads())
+ return min(
+ available_cpus,
+ configured_threads,
+ MAX_PDF_RENDER_PROCESSES,
+ )
+
+
+def _create_pdf_render_executor(max_workers: int) -> ProcessPoolExecutor:
+ if is_windows_environment():
+ return ProcessPoolExecutor(max_workers=max_workers)
+
+ start_method = multiprocessing.get_start_method()
+ if start_method == "fork":
+ logger.debug(
+ "PDF image rendering switches multiprocessing start method from fork to spawn"
+ )
+ return ProcessPoolExecutor(
+ max_workers=max_workers,
+ mp_context=multiprocessing.get_context("spawn"),
+ )
+
+ return ProcessPoolExecutor(max_workers=max_workers)
+
+
+def _get_pdf_render_executor() -> ProcessPoolExecutor:
+ global _pdf_render_executor
+
+ with _pdf_render_executor_lock:
+ if _pdf_render_executor is None:
+ max_workers = _get_pdf_render_pool_capacity()
+ _pdf_render_executor = _create_pdf_render_executor(max_workers=max_workers)
+ logger.debug(
+ f"Created persistent PDF render executor with max_workers={max_workers}"
+ )
+ return _pdf_render_executor
+
+
+def _recycle_pdf_render_executor(
+ executor: ProcessPoolExecutor | None,
+ *,
+ terminate_processes: bool,
+) -> None:
+ global _pdf_render_executor
+
+ if executor is None:
+ return
+
+ with _pdf_render_executor_lock:
+ if _pdf_render_executor is executor:
+ _pdf_render_executor = None
+
+ if terminate_processes:
+ _terminate_executor_processes(executor)
+ executor.shutdown(wait=False, cancel_futures=True)
+
+
+def shutdown_pdf_render_executor() -> None:
+ global _pdf_render_executor
+
+ with _pdf_render_executor_lock:
+ executor = _pdf_render_executor
+ _pdf_render_executor = None
+
+ if executor is not None:
+ _terminate_executor_processes(executor)
+ executor.shutdown(wait=False, cancel_futures=True)
+
+
+atexit.register(shutdown_pdf_render_executor)
+
+
+def _load_images_from_pdf_bytes_range(
+ pdf_bytes: bytes,
+ dpi=DEFAULT_PDF_IMAGE_DPI,
+ start_page_id=0,
+ end_page_id=0,
+ image_type=ImageType.PIL,
+ timeout=None,
+ threads=None,
+):
+ if end_page_id < start_page_id:
+ return []
+
+ if timeout is None:
+ timeout = get_load_images_timeout()
+ if threads is None:
+ threads = get_load_images_threads()
+
+ actual_threads, page_ranges = _get_render_process_plan(
+ start_page_id,
+ end_page_id,
+ threads,
+ )
+
+ logger.debug(
+ f"PDF image rendering uses {actual_threads} processes for pages "
+ f"{start_page_id + 1}-{end_page_id + 1}: {page_ranges}"
+ )
+
+ executor = _get_pdf_render_executor()
+ recycle_executor = False
+ try:
+ futures = []
+ future_to_range = {}
+ for range_start, range_end in page_ranges:
+ future = executor.submit(
+ _load_images_from_pdf_worker,
+ pdf_bytes,
+ dpi,
+ range_start,
+ range_end,
+ image_type,
+ )
+ futures.append(future)
+ future_to_range[future] = range_start
+
+ _, not_done = wait(futures, timeout=timeout, return_when=ALL_COMPLETED)
+ if not_done:
+ recycle_executor = True
+ raise TimeoutError(
+ f"PDF image rendering timeout after {timeout}s "
+ f"for pages {start_page_id + 1}-{end_page_id + 1}"
+ )
+
+ all_results = []
+ for future in futures:
+ range_start = future_to_range[future]
+ images_list = future.result()
+ all_results.append((range_start, images_list))
+
+ all_results.sort(key=lambda x: x[0])
+ images_list = []
+ for _, imgs in all_results:
+ images_list.extend(imgs)
+
+ return images_list
+ except BrokenProcessPool:
+ recycle_executor = True
+ raise
+ finally:
+ if recycle_executor:
+ logger.warning("Recycling persistent PDF render executor after render failure")
+ _recycle_pdf_render_executor(
+ executor,
+ terminate_processes=True,
+ )
+
+
+def _terminate_executor_processes(executor):
+ """强制终止 ProcessPoolExecutor 中的所有子进程"""
+ if hasattr(executor, '_processes'):
+ for pid, process in executor._processes.items():
+ if process.is_alive():
+ try:
+ # 先发送 SIGTERM 允许优雅退出
+ os.kill(pid, signal.SIGTERM)
+ except (ProcessLookupError, OSError):
+ pass
+
+ # 给子进程一点时间响应 SIGTERM
+ time.sleep(0.1)
+
+ # 对仍然存活的进程发送 SIGKILL 强制终止
+ for pid, process in executor._processes.items():
+ if process.is_alive():
+ try:
+ os.kill(pid, signal.SIGKILL)
+ except (ProcessLookupError, OSError):
+ pass
+
+
+def load_images_from_pdf_core(
+ pdf_bytes: bytes,
+ dpi=DEFAULT_PDF_IMAGE_DPI,
+ start_page_id=0,
+ end_page_id=None,
+ image_type=ImageType.PIL, # PIL or BASE64
+):
+ images_list = []
+ pdf_doc = None
+ try:
+ with pdfium_guard():
+ pdf_doc = open_pdfium_document(pdfium.PdfDocument, pdf_bytes)
+ pdf_page_num = len(pdf_doc)
+ end_page_id = get_end_page_id(end_page_id, pdf_page_num)
+
+ for index in range(start_page_id, end_page_id + 1):
+ # logger.debug(f"Converting page {index}/{pdf_page_num} to image")
+ page = pdf_doc[index]
+ image_dict = pdf_page_to_image(page, dpi=dpi, image_type=image_type)
+ images_list.append(image_dict)
+ finally:
+ close_pdfium_document(pdf_doc)
+
+ return images_list
+
+
+def load_images_from_pdf_doc(
+ pdf_doc: pdfium.PdfDocument,
+ dpi=DEFAULT_PDF_IMAGE_DPI,
+ start_page_id=0,
+ end_page_id=None,
+ image_type=ImageType.PIL,
+ pdf_bytes: bytes | None = None,
+ timeout=None,
+ threads=None,
+):
+ pdf_page_num = get_pdfium_document_page_count(pdf_doc)
+ normalized_end_page_id = get_end_page_id(end_page_id, pdf_page_num)
+
+ if pdf_bytes is not None and not is_windows_environment():
+ return _load_images_from_pdf_bytes_range(
+ pdf_bytes,
+ dpi=dpi,
+ start_page_id=start_page_id,
+ end_page_id=normalized_end_page_id,
+ image_type=image_type,
+ timeout=timeout,
+ threads=threads,
+ )
+
+ images_list = []
+ with pdfium_guard():
+ for index in range(start_page_id, normalized_end_page_id + 1):
+ page = pdf_doc[index]
+ image_dict = pdf_page_to_image(page, dpi=dpi, image_type=image_type)
+ images_list.append(image_dict)
+
+ return images_list
+
+
+def cut_image(
+ bbox: tuple,
+ page_num: int,
+ page_pil_img,
+ return_path,
+ image_writer: FileBasedDataWriter,
+ scale=2,
+):
+ """从第page_num页的page中，根据bbox进行裁剪出一张jpg图片，返回图片路径 save_path：需要同时支持s3和本地,
+ 图片存放在save_path下，文件名是:
+ {page_num}_{bbox[0]}_{bbox[1]}_{bbox[2]}_{bbox[3]}.jpg , bbox内数字取整。"""
+
+ # 拼接文件名
+ filename = f"{page_num}_{int(bbox[0])}_{int(bbox[1])}_{int(bbox[2])}_{int(bbox[3])}"
+
+ # 老版本返回不带bucket的路径
+ img_path = f"{return_path}_{filename}" if return_path is not None else None
+
+ # 新版本生成平铺路径
+ img_hash256_path = f"{str_sha256(img_path)}.jpg"
+ # img_hash256_path = f'{img_path}.jpg'
+
+ crop_img = get_crop_img(bbox, page_pil_img, scale=scale)
+
+ img_bytes = image_to_bytes(crop_img, image_format="JPEG")
+
+ image_writer.write(img_hash256_path, img_bytes)
+ return img_hash256_path
+
+
+def get_crop_img(bbox: tuple, pil_img, scale=2):
+ scale_bbox = normalize_to_int_bbox([float(v) * scale for v in bbox])
+ if scale_bbox is None:
+ return pil_img.crop((0, 0, 0, 0))
+ return pil_img.crop(tuple(scale_bbox))
+
+
+def get_crop_np_img(bbox: tuple, input_img, scale=2):
+ if isinstance(input_img, Image.Image):
+ np_img = np.asarray(input_img)
+ elif isinstance(input_img, np.ndarray):
+ np_img = input_img
+ else:
+ raise ValueError("Input must be a pillow object or a numpy array.")
+
+ height, width = np_img.shape[:2]
+ scale_bbox = normalize_to_int_bbox(
+ [float(v) * scale for v in bbox],
+ image_size=(height, width),
+ )
+ if scale_bbox is None:
+ return np_img[0:0, 0:0]
+
+ return np_img[scale_bbox[1] : scale_bbox[3], scale_bbox[0] : scale_bbox[2]]
+
+
+def images_bytes_to_pdf_bytes(image_bytes):
+ # 内存缓冲区
+ pdf_buffer = BytesIO()
+
+ # 载入并转换所有图像为 RGB 模式
+ image = Image.open(BytesIO(image_bytes))
+ # 根据 EXIF 信息自动转正（处理手机拍摄的带 Orientation 标记的图片）
+ image = ImageOps.exif_transpose(image) or image
+ # 只在必要时转换
+ if image.mode != "RGB":
+ image = image.convert("RGB")
+
+ # 第一张图保存为 PDF，其余追加
+ image.save(
+ pdf_buffer,
+ format="PDF",
+ resolution=DEFAULT_PDF_IMAGE_DPI,
+ quality=95,
+ subsampling=0,
+ )
+
+ # 获取 PDF bytes 并重置指针（可选）
+ pdf_bytes = pdf_buffer.getvalue()
+ pdf_buffer.close()
+ return pdf_bytes
diff --git a/mineru/utils/pdf_page_id.py b/mineru/utils/pdf_page_id.py
new file mode 100644
index 000000000..5471297ff
--- /dev/null
+++ b/mineru/utils/pdf_page_id.py
@@ -0,0 +1,10 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from loguru import logger
+
+
+def get_end_page_id(end_page_id, pdf_page_num):
+ end_page_id = end_page_id if end_page_id is not None and end_page_id >= 0 else pdf_page_num - 1
+ if end_page_id > pdf_page_num - 1:
+ logger.debug("end_page_id is out of range, use images length")
+ end_page_id = pdf_page_num - 1
+ return end_page_id
diff --git a/mineru/utils/pdf_reader.py b/mineru/utils/pdf_reader.py
new file mode 100644
index 000000000..f34b382e8
--- /dev/null
+++ b/mineru/utils/pdf_reader.py
@@ -0,0 +1,49 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import base64
+from io import BytesIO
+
+from loguru import logger
+from PIL import Image
+from pypdfium2 import PdfBitmap, PdfPage
+from mineru.utils.pdfium_guard import pdfium_guard
+
+
+def page_to_image(
+ page: PdfPage,
+ dpi: int = 200,
+ max_width_or_height: int = 3500, # changed from 4500 to 3500
+) -> (Image.Image, float):
+ with pdfium_guard():
+ scale = dpi / 72
+
+ long_side_length = max(*page.get_size())
+ if (long_side_length*scale) > max_width_or_height:
+ scale = max_width_or_height / long_side_length
+
+ bitmap: PdfBitmap = page.render(scale=scale) # type: ignore
+
+ image = bitmap.to_pil()
+ try:
+ bitmap.close()
+ except Exception as e:
+ logger.error(f"Failed to close bitmap: {e}")
+ return image, scale
+
+
+def image_to_bytes(
+ image: Image.Image,
+ # image_format: str = "PNG", # 也可以用 "JPEG"
+ image_format: str = "JPEG",
+) -> bytes:
+ with BytesIO() as image_buffer:
+ image.save(image_buffer, format=image_format)
+ return image_buffer.getvalue()
+
+
+def image_to_b64str(
+ image: Image.Image,
+ # image_format: str = "PNG", # 也可以用 "JPEG"
+ image_format: str = "JPEG",
+) -> str:
+ image_bytes = image_to_bytes(image, image_format)
+ return f"data:image/{image_format.lower()};base64,{base64.b64encode(image_bytes).decode('utf-8')}"
diff --git a/mineru/utils/pdf_text_tool.py b/mineru/utils/pdf_text_tool.py
new file mode 100644
index 000000000..999827e9a
--- /dev/null
+++ b/mineru/utils/pdf_text_tool.py
@@ -0,0 +1,52 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import math
+from typing import List
+
+import pypdfium2 as pdfium
+from pdftext.pdf.chars import deduplicate_chars, get_chars
+from pdftext.pdf.pages import assign_scripts, get_blocks, get_lines, get_spans
+
+from mineru.utils.pdfium_guard import pdfium_guard
+
+
+def get_page(
+ page: pdfium.PdfPage,
+ quote_loosebox: bool = True,
+ superscript_height_threshold: float = 0.7,
+ line_distance_threshold: float = 0.1,
+) -> dict:
+ with pdfium_guard():
+ textpage = page.get_textpage()
+ page_bbox: List[float] = page.get_bbox()
+ page_width = math.ceil(abs(page_bbox[2] - page_bbox[0]))
+ page_height = math.ceil(abs(page_bbox[1] - page_bbox[3]))
+
+ page_rotation = 0
+ try:
+ page_rotation = page.get_rotation()
+ except Exception:
+ pass
+
+ chars = deduplicate_chars(
+ get_chars(textpage, page_bbox, page_rotation, quote_loosebox)
+ )
+ spans = get_spans(
+ chars,
+ superscript_height_threshold=superscript_height_threshold,
+ line_distance_threshold=line_distance_threshold,
+ )
+ lines = get_lines(spans)
+ assign_scripts(
+ lines,
+ height_threshold=superscript_height_threshold,
+ line_distance_threshold=line_distance_threshold,
+ )
+ blocks = get_blocks(lines)
+
+ return {
+ "bbox": page_bbox,
+ "width": page_width,
+ "height": page_height,
+ "rotation": page_rotation,
+ "blocks": blocks,
+ }
diff --git a/mineru/utils/pdfium_guard.py b/mineru/utils/pdfium_guard.py
new file mode 100644
index 000000000..08e90fc62
--- /dev/null
+++ b/mineru/utils/pdfium_guard.py
@@ -0,0 +1,85 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import threading
+from io import BytesIO
+from contextlib import contextmanager
+from typing import Any, Callable, Sequence, TypeVar
+
+from mineru.utils.pdf_page_id import get_end_page_id
+
+
+_pdfium_lock = threading.RLock()
+
+T = TypeVar("T")
+
+
+@contextmanager
+def pdfium_guard():
+ with _pdfium_lock:
+ yield
+
+
+def open_pdfium_document(
+ opener: Callable[..., T],
+ *args: Any,
+ **kwargs: Any,
+) -> T:
+ with pdfium_guard():
+ return opener(*args, **kwargs)
+
+
+def get_pdfium_document_page_count(pdf_doc) -> int:
+ with pdfium_guard():
+ return len(pdf_doc)
+
+
+def close_pdfium_document(pdf_doc) -> None:
+ if pdf_doc is None:
+ return
+ with pdfium_guard():
+ pdf_doc.close()
+
+
+def rewrite_pdf_bytes_with_pdfium(
+ src_pdf_bytes: bytes,
+ start_page_id: int = 0,
+ end_page_id: int | None = None,
+ page_indices: Sequence[int] | None = None,
+) -> bytes:
+ import pypdfium2 as pdfium
+
+ pdf_doc = None
+ output_doc = None
+ try:
+ with pdfium_guard():
+ pdf_doc = pdfium.PdfDocument(src_pdf_bytes)
+ total_page_count = len(pdf_doc)
+ if total_page_count == 0:
+ return b""
+
+ if page_indices is not None:
+ normalized_page_indices = sorted(
+ {
+ page_index
+ for page_index in page_indices
+ if 0 <= page_index < total_page_count
+ }
+ )
+ if not normalized_page_indices:
+ return b""
+ else:
+ normalized_end_page_id = get_end_page_id(end_page_id, total_page_count)
+ normalized_page_indices = list(
+ range(start_page_id, normalized_end_page_id + 1)
+ )
+
+ output_doc = pdfium.PdfDocument.new()
+ output_doc.import_pages(pdf_doc, normalized_page_indices)
+
+ output_buffer = BytesIO()
+ output_doc.save(output_buffer)
+ return output_buffer.getvalue()
+ finally:
+ if output_doc is not None:
+ close_pdfium_document(output_doc)
+ if pdf_doc is not None:
+ close_pdfium_document(pdf_doc)
diff --git a/mineru/utils/span_block_fix.py b/mineru/utils/span_block_fix.py
new file mode 100644
index 000000000..430330ff9
--- /dev/null
+++ b/mineru/utils/span_block_fix.py
@@ -0,0 +1,153 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from mineru.utils.enum_class import ContentType
+from mineru.utils.ocr_utils import _is_overlaps_y_exceeds_threshold, _is_overlaps_x_exceeds_threshold
+
+VERTICAL_SPAN_HEIGHT_TO_WIDTH_RATIO_THRESHOLD = 2
+VERTICAL_SPAN_IN_BLOCK_THRESHOLD = 0.8
+
+def fix_text_block(block):
+ # 文本block中的公式span都应该转换成行内type
+ for span in block['spans']:
+ if span['type'] == ContentType.INTERLINE_EQUATION:
+ span['type'] = ContentType.INLINE_EQUATION
+
+ # 假设block中的span超过80%的数量高度是宽度的两倍以上，则认为是纵向文本块
+ vertical_span_count = sum(
+ 1 for span in block['spans']
+ if (span['bbox'][3] - span['bbox'][1]) / (span['bbox'][2] - span['bbox'][0]) > VERTICAL_SPAN_HEIGHT_TO_WIDTH_RATIO_THRESHOLD
+ )
+ total_span_count = len(block['spans'])
+ if total_span_count == 0:
+ vertical_ratio = 0
+ else:
+ vertical_ratio = vertical_span_count / total_span_count
+
+ if vertical_ratio > VERTICAL_SPAN_IN_BLOCK_THRESHOLD:
+ # 如果是纵向文本块，则按纵向lines处理
+ block_lines = merge_spans_to_vertical_line(block['spans'])
+ sort_block_lines = vertical_line_sort_spans_from_top_to_bottom(block_lines)
+ else:
+ block_lines = merge_spans_to_line(block['spans'])
+ sort_block_lines = line_sort_spans_by_left_to_right(block_lines)
+
+ block['lines'] = sort_block_lines
+ del block['spans']
+ return block
+
+
+def merge_spans_to_line(spans, threshold=0.6):
+ if len(spans) == 0:
+ return []
+ else:
+ # 按照y0坐标排序
+ spans.sort(key=lambda span: span['bbox'][1])
+
+ lines = []
+ current_line = [spans[0]]
+ for span in spans[1:]:
+ # 如果当前的span类型为"interline_equation" 或者 当前行中已经有"interline_equation"
+ # image和table类型，同上
+ if span['type'] in [
+ ContentType.INTERLINE_EQUATION, ContentType.IMAGE,
+ ContentType.TABLE
+ ] or any(s['type'] in [
+ ContentType.INTERLINE_EQUATION, ContentType.IMAGE,
+ ContentType.TABLE
+ ] for s in current_line):
+ # 则开始新行
+ lines.append(current_line)
+ current_line = [span]
+ continue
+
+ # 如果当前的span与当前行的最后一个span在y轴上重叠，则添加到当前行
+ if _is_overlaps_y_exceeds_threshold(span['bbox'], current_line[-1]['bbox'], threshold):
+ current_line.append(span)
+ else:
+ # 否则，开始新行
+ lines.append(current_line)
+ current_line = [span]
+
+ # 添加最后一行
+ if current_line:
+ lines.append(current_line)
+
+ return lines
+
+
+def merge_spans_to_vertical_line(spans, threshold=0.6):
+ """将纵向文本的spans合并成纵向lines（从右向左阅读）"""
+ if len(spans) == 0:
+ return []
+ else:
+ # 按照x2坐标从大到小排序（从右向左）
+ spans.sort(key=lambda span: span['bbox'][2], reverse=True)
+
+ vertical_lines = []
+ current_line = [spans[0]]
+
+ for span in spans[1:]:
+ # 特殊类型元素单独成列
+ if span['type'] in [
+ ContentType.INTERLINE_EQUATION, ContentType.IMAGE,
+ ContentType.TABLE
+ ] or any(s['type'] in [
+ ContentType.INTERLINE_EQUATION, ContentType.IMAGE,
+ ContentType.TABLE
+ ] for s in current_line):
+ vertical_lines.append(current_line)
+ current_line = [span]
+ continue
+
+ # 如果当前的span与当前行的最后一个span在y轴上重叠，则添加到当前行
+ if _is_overlaps_x_exceeds_threshold(span['bbox'], current_line[-1]['bbox'], threshold):
+ current_line.append(span)
+ else:
+ vertical_lines.append(current_line)
+ current_line = [span]
+
+ # 添加最后一列
+ if current_line:
+ vertical_lines.append(current_line)
+
+ return vertical_lines
+
+
+# 将每一个line中的span从左到右排序
+def line_sort_spans_by_left_to_right(lines):
+ line_objects = []
+ for line in lines:
+ # 按照x0坐标排序
+ line.sort(key=lambda span: span['bbox'][0])
+ line_bbox = [
+ min(span['bbox'][0] for span in line), # x0
+ min(span['bbox'][1] for span in line), # y0
+ max(span['bbox'][2] for span in line), # x1
+ max(span['bbox'][3] for span in line), # y1
+ ]
+ line_objects.append({
+ 'bbox': line_bbox,
+ 'spans': line,
+ })
+ return line_objects
+
+
+def vertical_line_sort_spans_from_top_to_bottom(vertical_lines):
+ line_objects = []
+ for line in vertical_lines:
+ # 按照y0坐标排序（从上到下）
+ line.sort(key=lambda span: span['bbox'][1])
+
+ # 计算整个列的边界框
+ line_bbox = [
+ min(span['bbox'][0] for span in line), # x0
+ min(span['bbox'][1] for span in line), # y0
+ max(span['bbox'][2] for span in line), # x1
+ max(span['bbox'][3] for span in line), # y1
+ ]
+
+ # 组装结果
+ line_objects.append({
+ 'bbox': line_bbox,
+ 'spans': line,
+ })
+ return line_objects
diff --git a/mineru/utils/span_pre_proc.py b/mineru/utils/span_pre_proc.py
new file mode 100644
index 000000000..cd441c5a3
--- /dev/null
+++ b/mineru/utils/span_pre_proc.py
@@ -0,0 +1,275 @@
+# Copyright (c) Opendatalab. All rights reserved.
+import collections
+import math
+import re
+import statistics
+
+import cv2
+import numpy as np
+from loguru import logger
+
+from mineru.utils.boxbase import calculate_overlap_area_in_bbox1_area_ratio
+from mineru.utils.enum_class import BlockType, ContentType
+from mineru.utils.pdf_image_tools import get_crop_img
+from mineru.utils.pdf_text_tool import get_page
+
+MAX_NATIVE_TEXT_CHARS_PER_PAGE = 65535
+
+
+def __replace_ligatures(text: str):
+ ligatures = {
+ 'fi': 'fi', 'fl': 'fl', 'ff': 'ff', 'ffi': 'ffi', 'ffl': 'ffl', 'ſt': 'ft', 'st': 'st'
+ }
+ return re.sub('|'.join(map(re.escape, ligatures.keys())), lambda m: ligatures[m.group()], text)
+
+def __replace_unicode(text: str):
+ ligatures = {
+ '\r\n': '', '\u0002': '-',
+ }
+ return re.sub('|'.join(map(re.escape, ligatures.keys())), lambda m: ligatures[m.group()], text)
+
+
+"""pdf_text dict方案 char级别"""
+def txt_spans_extract(pdf_page, spans, pil_img, scale, all_bboxes, all_discarded_blocks):
+ page_char_count = None
+ try:
+ page_char_count = pdf_page.get_textpage().count_chars()
+ except Exception as exc:
+ logger.debug(f"Failed to get page char count before txt extraction: {exc}")
+
+ if page_char_count is not None and page_char_count > MAX_NATIVE_TEXT_CHARS_PER_PAGE:
+ logger.info(
+ "Fallback to post-OCR in txt_spans_extract due to high char count: "
+ f"count_chars={page_char_count}"
+ )
+ need_ocr_spans = [
+ span for span in spans if span.get('type') == ContentType.TEXT
+ ]
+ return _prepare_post_ocr_spans(need_ocr_spans, spans, pil_img, scale)
+
+ page_dict = get_page(pdf_page)
+
+ page_all_chars = []
+ page_all_lines = []
+ for block in page_dict['blocks']:
+ for line in block['lines']:
+ rotation_degrees = math.degrees(line['rotation'])
+ # 旋转角度不为0, 90, 180, 270的行，直接跳过（rotation_degrees的值可能不为整数）
+ if not any(abs(rotation_degrees - angle) < 0.1 for angle in [0, 90, 180, 270]):
+ continue
+ page_all_lines.append(line)
+ for span in line['spans']:
+ for char in span['chars']:
+ page_all_chars.append(char)
+
+ # 计算所有sapn的高度的中位数
+ span_height_list = []
+ for span in spans:
+ if span['type'] in [ContentType.TEXT]:
+ span_height = span['bbox'][3] - span['bbox'][1]
+ span['height'] = span_height
+ span['width'] = span['bbox'][2] - span['bbox'][0]
+ span_height_list.append(span_height)
+ if len(span_height_list) == 0:
+ return spans
+ else:
+ median_span_height = statistics.median(span_height_list)
+
+ useful_spans = []
+ unuseful_spans = []
+ # 纵向span的两个特征：1. 高度超过多个line 2. 高宽比超过某个值
+ vertical_spans = []
+ for span in spans:
+ if span['type'] in [ContentType.TEXT]:
+ for block in all_bboxes + all_discarded_blocks:
+ if block[7] in [BlockType.IMAGE_BODY, BlockType.TABLE_BODY, BlockType.INTERLINE_EQUATION]:
+ continue
+ if calculate_overlap_area_in_bbox1_area_ratio(span['bbox'], block[0:4]) > 0.5:
+ if span['height'] > median_span_height * 2.3 and span['height'] > span['width'] * 2.3:
+ vertical_spans.append(span)
+ elif block in all_bboxes:
+ useful_spans.append(span)
+ else:
+ unuseful_spans.append(span)
+ break
+
+ """垂直的span框直接用line进行填充"""
+ if len(vertical_spans) > 0:
+ for pdfium_line in page_all_lines:
+ for span in vertical_spans:
+ if calculate_overlap_area_in_bbox1_area_ratio(pdfium_line['bbox'].bbox, span['bbox']) > 0.5:
+ for pdfium_span in pdfium_line['spans']:
+ span['content'] += pdfium_span['text']
+ break
+
+ for span in vertical_spans:
+ if len(span['content']) == 0:
+ spans.remove(span)
+
+ """水平的span框先用char填充，再用ocr填充空的span框"""
+ new_spans = []
+
+ for span in useful_spans + unuseful_spans:
+ if span['type'] in [ContentType.TEXT]:
+ span['chars'] = []
+ new_spans.append(span)
+
+ need_ocr_spans = fill_char_in_spans(new_spans, page_all_chars, median_span_height)
+
+ return _prepare_post_ocr_spans(need_ocr_spans, spans, pil_img, scale)
+
+
+def _prepare_post_ocr_spans(need_ocr_spans, spans, pil_img, scale):
+ if len(need_ocr_spans) == 0:
+ return spans
+
+ for span in need_ocr_spans:
+ # 对span的bbox截图再ocr
+ span_pil_img = get_crop_img(span['bbox'], pil_img, scale)
+ span_img = cv2.cvtColor(np.array(span_pil_img), cv2.COLOR_RGB2BGR)
+ # 计算span的对比度，低于0.17的span不进行ocr
+ if calculate_contrast(span_img, img_mode='bgr') <= 0.17:
+ if span in spans:
+ spans.remove(span)
+ continue
+
+ span['content'] = ''
+ span['score'] = 1.0
+ span['np_img'] = span_img
+
+ return spans
+
+
+def fill_char_in_spans(spans, all_chars, median_span_height):
+ # 简单从上到下排一下序
+ spans = sorted(spans, key=lambda x: x['bbox'][1])
+
+ grid_size = median_span_height
+ grid = collections.defaultdict(list)
+ for i, span in enumerate(spans):
+ start_cell = int(span['bbox'][1] / grid_size)
+ end_cell = int(span['bbox'][3] / grid_size)
+ for cell_idx in range(start_cell, end_cell + 1):
+ grid[cell_idx].append(i)
+
+ for char in all_chars:
+ char_center_y = (char['bbox'][1] + char['bbox'][3]) / 2
+ cell_idx = int(char_center_y / grid_size)
+
+ candidate_span_indices = grid.get(cell_idx, [])
+
+ for span_idx in candidate_span_indices:
+ span = spans[span_idx]
+ if calculate_char_in_span(char['bbox'], span['bbox'], char['char']):
+ span['chars'].append(char)
+ break
+
+ need_ocr_spans = []
+ for span in spans:
+ chars_to_content(span)
+ # 有的span中虽然没有字但有一两个空的占位符，用宽高和content长度过滤
+ if len(span['content']) * span['height'] < span['width'] * 0.5:
+ # logger.info(f"maybe empty span: {len(span['content'])}, {span['height']}, {span['width']}")
+ need_ocr_spans.append(span)
+ del span['height'], span['width']
+ return need_ocr_spans
+
+
+LINE_STOP_FLAG = ('.', '!', '?', '。', '！', '？', ')', '）', '"', '”', ':', '：', ';', '；', ']', '】', '}', '}', '>', '》', '、', ',', '，', '-', '—', '–',)
+LINE_START_FLAG = ('(', '（', '"', '“', '【', '{', '《', '<', '「', '『', '【', '[',)
+
+Span_Height_Ratio = 0.33 # 字符的中轴和span的中轴高度差不能超过1/3span高度
+def calculate_char_in_span(char_bbox, span_bbox, char, span_height_ratio=Span_Height_Ratio):
+ char_center_x = (char_bbox[0] + char_bbox[2]) / 2
+ char_center_y = (char_bbox[1] + char_bbox[3]) / 2
+ span_center_y = (span_bbox[1] + span_bbox[3]) / 2
+ span_height = span_bbox[3] - span_bbox[1]
+
+ if (
+ span_bbox[0] < char_center_x < span_bbox[2]
+ and span_bbox[1] < char_center_y < span_bbox[3]
+ and abs(char_center_y - span_center_y) < span_height * span_height_ratio # 字符的中轴和span的中轴高度差不能超过Span_Height_Ratio
+ ):
+ return True
+ else:
+ # 如果char是LINE_STOP_FLAG，就不用中心点判定，换一种方案（左边界在span区域内，高度判定和之前逻辑一致）
+ # 主要是给结尾符号一个进入span的机会，这个char还应该离span右边界较近
+ if char in LINE_STOP_FLAG:
+ if (
+ (span_bbox[2] - span_height) < char_bbox[0] < span_bbox[2]
+ and char_center_x > span_bbox[0]
+ and span_bbox[1] < char_center_y < span_bbox[3]
+ and abs(char_center_y - span_center_y) < span_height * span_height_ratio
+ ):
+ return True
+ elif char in LINE_START_FLAG:
+ if (
+ span_bbox[0] < char_bbox[2] < (span_bbox[0] + span_height)
+ and char_center_x < span_bbox[2]
+ and span_bbox[1] < char_center_y < span_bbox[3]
+ and abs(char_center_y - span_center_y) < span_height * span_height_ratio
+ ):
+ return True
+ else:
+ return False
+
+
+def chars_to_content(span):
+ # 检查span中的char是否为空
+ if len(span['chars']) != 0:
+ # 给chars按char_idx排序
+ chars = sorted(span['chars'], key=lambda x: x['char_idx'])
+
+ # Calculate the width of each character
+ char_widths = [char['bbox'][2] - char['bbox'][0] for char in chars]
+ # Calculate the median width
+ median_width = statistics.median(char_widths)
+
+ parts = []
+ for idx, char1 in enumerate(chars):
+ char2 = chars[idx + 1] if idx + 1 < len(chars) else None
+
+ # 如果下一个char的x0和上一个char的x1距离超过0.25个字符宽度，则需要在中间插入一个空格
+ if (
+ char2
+ and char2['bbox'][0] - char1['bbox'][2] > median_width * 0.25
+ and char1['char'] != ' '
+ and char2['char'] != ' '
+ ):
+ parts.append(char1['char'])
+ parts.append(' ')
+ else:
+ parts.append(char1['char'])
+
+ content = ''.join(parts)
+ content = __replace_unicode(content)
+ content = __replace_ligatures(content)
+ content = __replace_ligatures(content)
+ span['content'] = content.strip()
+
+ del span['chars']
+
+
+def calculate_contrast(img, img_mode) -> float:
+ """
+ 计算给定图像的对比度。
+ :param img: 图像，类型为numpy.ndarray
+ :Param img_mode = 图像的色彩通道，'rgb' 或 'bgr'
+ :return: 图像的对比度值
+ """
+ if img_mode == 'rgb':
+ # 将RGB图像转换为灰度图
+ gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
+ elif img_mode == 'bgr':
+ # 将BGR图像转换为灰度图
+ gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+ else:
+ raise ValueError("Invalid image mode. Please provide 'rgb' or 'bgr'.")
+
+ # 计算均值和标准差
+ mean_value = np.mean(gray_img)
+ std_dev = np.std(gray_img)
+ # 对比度定义为标准差除以平均值（加上小常数避免除零错误）
+ contrast = std_dev / (mean_value + 1e-6)
+ # logger.debug(f"contrast: {contrast}")
+ return round(contrast, 2)
diff --git a/mineru/utils/table_merge.py b/mineru/utils/table_merge.py
new file mode 100644
index 000000000..07bc9d67c
--- /dev/null
+++ b/mineru/utils/table_merge.py
@@ -0,0 +1,904 @@
+# Copyright (c) Opendatalab. All rights reserved.
+from copy import deepcopy
+from dataclasses import dataclass
+from typing import Any
+
+from bs4 import BeautifulSoup
+
+from mineru.backend.vlm.vlm_middle_json_mkcontent import merge_para_with_text
+from mineru.utils.char_utils import full_to_half
+from mineru.utils.enum_class import BlockType, SplitFlag
+
+
+CONTINUATION_END_MARKERS = [
+ "(续)",
+ "(续表)",
+ "(续上表)",
+ "(continued)",
+ "(cont.)",
+ "(cont’d)",
+ "(…continued)",
+ "续表",
+]
+
+CONTINUATION_INLINE_MARKERS = [
+ "(continued)",
+]
+
+MAX_HEADER_ROWS = 5
+
+
+@dataclass
+class RowMetrics:
+ row_idx: int
+ effective_cols: int
+ actual_cols: int
+ visual_cols: int
+
+
+@dataclass
+class RowSignature:
+ effective_cols: int
+ colspans: tuple[int, ...]
+ rowspans: tuple[int, ...]
+ normalized_texts: tuple[str, ...]
+ display_texts: tuple[str, ...]
+
+ @property
+ def cell_count(self) -> int:
+ return len(self.colspans)
+
+
+@dataclass
+class RowScanResult:
+ row_effective_cols: list[int]
+ row_metrics: list[RowMetrics]
+ total_cols: int
+ last_nonempty_row_metrics: RowMetrics | None
+ tail_occupied: dict[int, set[int]]
+
+
+@dataclass
+class TableMergeState:
+ owner_block: dict[str, Any]
+ body_span: dict[str, Any]
+ soup: Any
+ tbody: Any
+ rows: list[Any]
+ total_cols: int
+ front_header_info: list[RowSignature]
+ front_first_data_row_metrics: dict[int, RowMetrics]
+ last_data_row_metrics: RowMetrics | None
+ row_effective_cols: list[int]
+ tail_occupied: dict[int, set[int]]
+ dirty: bool = False
+
+
+def _normalize_cell_text(cell) -> str:
+ return "".join(full_to_half(cell.get_text()).split())
+
+
+def _display_cell_text(cell) -> str:
+ return full_to_half(cell.get_text().strip())
+
+
+def _scan_rows(rows, initial_occupied: dict[int, set[int]] | None = None, start_row_idx: int = 0) -> RowScanResult:
+ """Scan rows once and cache effective-column metrics.
+
+ initial_occupied stores future-row occupancy relative to the first scanned row
+ and preserves rowspans that cross a merge boundary.
+ """
+ occupied: dict[int, dict[int, bool]] = {}
+ max_cols = 0
+
+ for row_offset, cols in (initial_occupied or {}).items():
+ if not cols:
+ continue
+ occupied[row_offset] = {col: True for col in cols}
+ max_cols = max(max_cols, max(cols) + 1)
+
+ row_effective_cols: list[int] = []
+ row_metrics: list[RowMetrics] = []
+ last_nonempty_row_metrics: RowMetrics | None = None
+
+ for local_idx, row in enumerate(rows):
+ occupied_row = occupied.setdefault(local_idx, {})
+ col_idx = 0
+ cells = row.find_all(["td", "th"])
+ actual_cols = 0
+
+ for cell in cells:
+ while col_idx in occupied_row:
+ col_idx += 1
+
+ colspan = int(cell.get("colspan", 1))
+ rowspan = int(cell.get("rowspan", 1))
+ actual_cols += colspan
+
+ for row_offset in range(rowspan):
+ target_idx = local_idx + row_offset
+ occupied_target = occupied.setdefault(target_idx, {})
+ for col in range(col_idx, col_idx + colspan):
+ occupied_target[col] = True
+
+ col_idx += colspan
+ max_cols = max(max_cols, col_idx)
+
+ effective_cols = max(occupied_row.keys()) + 1 if occupied_row else 0
+ row_effective_cols.append(effective_cols)
+ max_cols = max(max_cols, effective_cols)
+
+ metrics = RowMetrics(
+ row_idx=start_row_idx + local_idx,
+ effective_cols=effective_cols,
+ actual_cols=actual_cols,
+ visual_cols=len(cells),
+ )
+ row_metrics.append(metrics)
+ if cells:
+ last_nonempty_row_metrics = metrics
+
+ tail_occupied = {
+ row_idx - len(rows): set(cols.keys())
+ for row_idx, cols in occupied.items()
+ if row_idx >= len(rows) and cols
+ }
+
+ return RowScanResult(
+ row_effective_cols=row_effective_cols,
+ row_metrics=row_metrics,
+ total_cols=max_cols,
+ last_nonempty_row_metrics=last_nonempty_row_metrics,
+ tail_occupied=tail_occupied,
+ )
+
+
+def _build_row_signature(row, effective_cols: int) -> RowSignature:
+ cells = row.find_all(["td", "th"])
+ return RowSignature(
+ effective_cols=effective_cols,
+ colspans=tuple(int(cell.get("colspan", 1)) for cell in cells),
+ rowspans=tuple(int(cell.get("rowspan", 1)) for cell in cells),
+ normalized_texts=tuple(_normalize_cell_text(cell) for cell in cells),
+ display_texts=tuple(_display_cell_text(cell) for cell in cells),
+ )
+
+
+def _build_front_cache(rows, max_header_rows: int = MAX_HEADER_ROWS) -> tuple[list[RowSignature], dict[int, RowMetrics]]:
+ front_limit = min(len(rows), max_header_rows + 1)
+ front_rows = rows[:front_limit]
+ front_scan = _scan_rows(front_rows)
+
+ front_header_info = [
+ _build_row_signature(front_rows[idx], front_scan.row_effective_cols[idx])
+ for idx in range(min(len(front_rows), max_header_rows))
+ ]
+ front_first_data_row_metrics = {
+ idx: metrics for idx, metrics in enumerate(front_scan.row_metrics)
+ }
+ return front_header_info, front_first_data_row_metrics
+
+
+def _find_table_body_span(table_block):
+ for block in table_block["blocks"]:
+ if block["type"] == BlockType.TABLE_BODY and block["lines"] and block["lines"][0]["spans"]:
+ return block["lines"][0]["spans"][0]
+ return None
+
+
+def _refresh_table_state_metrics(state: TableMergeState) -> None:
+ scan = _scan_rows(state.rows)
+ state.row_effective_cols = scan.row_effective_cols
+ state.total_cols = scan.total_cols
+ state.last_data_row_metrics = scan.last_nonempty_row_metrics
+ state.tail_occupied = scan.tail_occupied
+ state.front_header_info, state.front_first_data_row_metrics = _build_front_cache(state.rows)
+
+
+def build_table_state_from_html(
+ html: str,
+ max_header_rows: int = MAX_HEADER_ROWS,
+) -> TableMergeState | None:
+ """从原始 HTML 构建 TableMergeState，不依赖 MinerU block 结构。
+
+ 供外部工具（如 mineru-vl-utils）调用，用于跨页表格结构检测。
+ 返回的 state 仅可用于 can_merge_by_structure()，不可传入 can_merge_tables()。
+ """
+ if not html:
+ return None
+
+ soup = BeautifulSoup(html, "html.parser")
+ tbody = soup.find("tbody") or soup.find("table")
+ rows = soup.find_all("tr")
+ if not rows:
+ return None
+
+ scan = _scan_rows(rows)
+ front_header_info, front_first_data_row_metrics = _build_front_cache(rows, max_header_rows=max_header_rows)
+
+ return TableMergeState(
+ owner_block={},
+ body_span={},
+ soup=soup,
+ tbody=tbody,
+ rows=rows,
+ total_cols=scan.total_cols,
+ front_header_info=front_header_info,
+ front_first_data_row_metrics=front_first_data_row_metrics,
+ last_data_row_metrics=scan.last_nonempty_row_metrics,
+ row_effective_cols=scan.row_effective_cols,
+ tail_occupied=scan.tail_occupied,
+ )
+
+
+def _build_table_state(table_block, max_header_rows: int = MAX_HEADER_ROWS) -> TableMergeState | None:
+ body_span = _find_table_body_span(table_block)
+ if body_span is None:
+ return None
+
+ html = body_span.get("html", "")
+ if not html:
+ return None
+
+ soup = BeautifulSoup(html, "html.parser")
+ tbody = soup.find("tbody") or soup.find("table")
+ rows = soup.find_all("tr")
+ scan = _scan_rows(rows)
+ front_header_info, front_first_data_row_metrics = _build_front_cache(rows, max_header_rows=max_header_rows)
+
+ return TableMergeState(
+ owner_block=table_block,
+ body_span=body_span,
+ soup=soup,
+ tbody=tbody,
+ rows=rows,
+ total_cols=scan.total_cols,
+ front_header_info=front_header_info,
+ front_first_data_row_metrics=front_first_data_row_metrics,
+ last_data_row_metrics=scan.last_nonempty_row_metrics,
+ row_effective_cols=scan.row_effective_cols,
+ tail_occupied=scan.tail_occupied,
+ )
+
+
+def _get_or_create_table_state(
+ table_block,
+ state_cache: dict[int, TableMergeState],
+ max_header_rows: int = MAX_HEADER_ROWS,
+) -> TableMergeState | None:
+ cache_key = id(table_block)
+ state = state_cache.get(cache_key)
+ if state is not None:
+ return state
+
+ state = _build_table_state(table_block, max_header_rows=max_header_rows)
+ if state is not None:
+ state_cache[cache_key] = state
+ return state
+
+
+def _serialize_table_state_html(state: TableMergeState) -> None:
+ state.body_span["html"] = str(state.soup)
+ state.dirty = False
+
+
+def calculate_table_total_columns(soup):
+ """计算表格的总列数，通过分析整个表格结构来处理rowspan和colspan."""
+ rows = soup.find_all("tr")
+ return _scan_rows(rows).total_cols if rows else 0
+
+
+def build_table_occupied_matrix(soup):
+ """构建表格的占用矩阵，返回每行的有效列数."""
+ rows = soup.find_all("tr")
+ if not rows:
+ return {}
+
+ scan = _scan_rows(rows)
+ return {
+ row_idx: effective_cols
+ for row_idx, effective_cols in enumerate(scan.row_effective_cols)
+ }
+
+
+def calculate_row_effective_columns(soup, row_idx):
+ """计算指定行的有效列数（考虑rowspan占用）."""
+ row_effective_cols = build_table_occupied_matrix(soup)
+ return row_effective_cols.get(row_idx, 0)
+
+
+def calculate_row_columns(row):
+ """计算表格行的实际列数，考虑colspan属性."""
+ cells = row.find_all(["td", "th"])
+ column_count = 0
+
+ for cell in cells:
+ colspan = int(cell.get("colspan", 1))
+ column_count += colspan
+
+ return column_count
+
+
+def calculate_visual_columns(row):
+ """计算表格行的视觉列数（实际td/th单元格数量，不考虑colspan）."""
+ cells = row.find_all(["td", "th"])
+ return len(cells)
+
+
+def _scan_row_visual_sources(rows, target_row_index: int) -> tuple[dict[int, tuple[int, int]], int]:
+ """扫描到目标行，记录每个视觉列当前由哪个源单元格占据。"""
+ if target_row_index < 0:
+ target_row_index += len(rows)
+ if target_row_index < 0 or target_row_index >= len(rows):
+ return {}, 0
+
+ # occupied[row_idx][col_idx] = (source_row_idx, source_cell_idx)
+ occupied: dict[int, dict[int, tuple[int, int]]] = {}
+ total_cols = 0
+
+ for r_idx in range(target_row_index + 1):
+ occupied_row = occupied.setdefault(r_idx, {})
+ col_idx = 0
+ cells = rows[r_idx].find_all(["td", "th"])
+ for cell_idx, cell in enumerate(cells):
+ while col_idx in occupied_row:
+ col_idx += 1
+ colspan = int(cell.get("colspan", 1))
+ rowspan = int(cell.get("rowspan", 1))
+ source_marker = (r_idx, cell_idx)
+ for ro in range(rowspan):
+ target_idx = r_idx + ro
+ occ = occupied.setdefault(target_idx, {})
+ for c in range(col_idx, col_idx + colspan):
+ occ[c] = source_marker
+ col_idx += colspan
+ total_cols = max(total_cols, col_idx)
+
+ return occupied.get(target_row_index, {}), total_cols
+
+
+def build_visual_col_mapping(rows, target_row_index: int) -> list[int]:
+ """构建目标行中每个显式
+
+
+
+
+ + MinerU 3: Document Extraction Demo +
+
+ Open-source document extraction for PDF, DOCX, PPTX, XLSX, and images to Markdown and JSON.
+ If you found our project helpful, please give us a ⭐️ to support us!
+
+ 
+
+
+
+
+
+
+
+
+
+
+
+
+ Code
+
+
+
+
+
+
+
+
+ Model
+
+
+
+
+
+
+
+
+ Paper
+
+
+
+
+
+
+
+
+
+ Homepage
+
+
+
+
+
+
+
+
+
+ Download
+
+
+
+
+ - A one-stop, open-source, high-quality data extraction tool - -
- - - - -API Reference -------------- - -If you are looking for information on a specific function, class or -method, this part of the documentation is for you. - -.. toctree:: - :maxdepth: 2 - - api diff --git a/next_docs/en/make.bat b/next_docs/en/make.bat deleted file mode 100644 index 954237b9b..000000000 --- a/next_docs/en/make.bat +++ /dev/null @@ -1,35 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=. -set BUILDDIR=_build - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.https://www.sphinx-doc.org/ - exit /b 1 -) - -if "%1" == "" goto help - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% - -:end -popd diff --git a/next_docs/requirements.txt b/next_docs/requirements.txt deleted file mode 100644 index ddb5027a6..000000000 --- a/next_docs/requirements.txt +++ /dev/null @@ -1,11 +0,0 @@ -boto3>=1.28.43 -loguru>=0.6.0 -myst-parser -Pillow==8.4.0 -pydantic>=2.7.2,<2.8.0 -PyMuPDF>=1.24.9 -sphinx -sphinx-argparse -sphinx-book-theme -sphinx-copybutton -sphinx_rtd_theme diff --git a/next_docs/zh_cn/.readthedocs.yaml b/next_docs/zh_cn/.readthedocs.yaml deleted file mode 100644 index 1f93a4d7b..000000000 --- a/next_docs/zh_cn/.readthedocs.yaml +++ /dev/null @@ -1,16 +0,0 @@ -version: 2 - -build: - os: ubuntu-22.04 - tools: - python: "3.10" - -formats: - - epub - -python: - install: - - requirements: docs/requirements.txt - -sphinx: - configuration: docs/zh_cn/conf.py diff --git a/next_docs/zh_cn/Makefile b/next_docs/zh_cn/Makefile deleted file mode 100644 index d4bb2cbb9..000000000 --- a/next_docs/zh_cn/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line, and also -# from the environment for the first two. -SPHINXOPTS ?= -SPHINXBUILD ?= sphinx-build -SOURCEDIR = . -BUILDDIR = _build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/next_docs/zh_cn/_static/image/logo.png b/next_docs/zh_cn/_static/image/logo.png deleted file mode 100644 index 2e6fdf3a9..000000000 Binary files a/next_docs/zh_cn/_static/image/logo.png and /dev/null differ diff --git a/next_docs/zh_cn/conf.py b/next_docs/zh_cn/conf.py deleted file mode 100644 index 4eccd497e..000000000 --- a/next_docs/zh_cn/conf.py +++ /dev/null @@ -1,122 +0,0 @@ -# Configuration file for the Sphinx documentation builder. -# -# This file only contains a selection of the most common options. For a full -# list see the documentation: -# https://www.sphinx-doc.org/en/master/usage/configuration.html - -# -- Path setup -------------------------------------------------------------- - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. - -import os -import subprocess -import sys - -from sphinx.ext import autodoc - - -def install(package): - subprocess.check_call([sys.executable, '-m', 'pip', 'install', package]) - - -requirements_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'requirements.txt')) -if os.path.exists(requirements_path): - with open(requirements_path) as f: - packages = f.readlines() - for package in packages: - install(package.strip()) - -sys.path.insert(0, os.path.abspath('../..')) - -# -- Project information ----------------------------------------------------- - -project = 'MinerU' -copyright = '2024, OpenDataLab' -author = 'MinerU Contributors' - -# The full version, including alpha/beta/rc tags -version_file = '../../magic_pdf/libs/version.py' -with open(version_file) as f: - exec(compile(f.read(), version_file, 'exec')) -__version__ = locals()['__version__'] -# The short X.Y version -version = __version__ -# The full version, including alpha/beta/rc tags -release = __version__ - -# -- General configuration --------------------------------------------------- - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.napoleon', - 'sphinx.ext.viewcode', - 'sphinx.ext.intersphinx', - 'sphinx_copybutton', - 'sphinx.ext.autodoc', - 'sphinx.ext.autosummary', - 'myst_parser', - 'sphinxarg.ext', -] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -# This pattern also affects html_static_path and html_extra_path. -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] - -# Exclude the prompt "$" when copying code -copybutton_prompt_text = r'\$ ' -copybutton_prompt_is_regexp = True - -language = 'zh_CN' - -# -- Options for HTML output ------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -# -html_theme = 'sphinx_book_theme' -html_logo = '_static/image/logo.png' -html_theme_options = { - 'path_to_docs': 'docs/zh_cn', - 'repository_url': 'https://github.com/opendatalab/MinerU', - 'use_repository_button': True, -} -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] - -# Mock out external dependencies here. -autodoc_mock_imports = [ - 'cpuinfo', - 'torch', - 'transformers', - 'psutil', - 'prometheus_client', - 'sentencepiece', - 'vllm.cuda_utils', - 'vllm._C', - 'numpy', - 'tqdm', -] - - -class MockedClassDocumenter(autodoc.ClassDocumenter): - """Remove note about base class when a class is derived from object.""" - - def add_line(self, line: str, source: str, *lineno: int) -> None: - if line == ' Bases: :py:class:`object`': - return - super().add_line(line, source, *lineno) - - -autodoc.ClassDocumenter = MockedClassDocumenter - -navigation_with_keys = False diff --git a/next_docs/zh_cn/index.rst b/next_docs/zh_cn/index.rst deleted file mode 100644 index 7b86aeb0c..000000000 --- a/next_docs/zh_cn/index.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. xtuner documentation master file, created by - sphinx-quickstart on Tue Jan 9 16:33:06 2024. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -欢迎来到 MinerU 的中文文档 -============================================== - -.. figure:: ./_static/image/logo.png - :align: center - :alt: mineru - :class: no-scaled-link - -.. raw:: html - -- 一站式开源高质量数据提取工具 - -
- - diff --git a/next_docs/zh_cn/make.bat b/next_docs/zh_cn/make.bat deleted file mode 100644 index 954237b9b..000000000 --- a/next_docs/zh_cn/make.bat +++ /dev/null @@ -1,35 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=. -set BUILDDIR=_build - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.https://www.sphinx-doc.org/ - exit /b 1 -) - -if "%1" == "" goto help - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% - -:end -popd diff --git a/projects/README.md b/projects/README.md index 3eca3cec5..3a46e7bb2 100644 --- a/projects/README.md +++ b/projects/README.md @@ -1,9 +1,7 @@ -# Welcome to the MinerU Project List +# Welcome to the MinerU Project (Archived) -## Project List +>[!NOTE] +> To consolidate resources, this repository has stopped accepting new project submissions and maintenance requests for existing projects. +> If you have an excellent project based on MinerU that you'd like to share, please submit the project link to the well-maintained community resource repository [awesome-mineru](https://github.com/opendatalab/awesome-mineru). +> Thank you for your support and contribution to the MinerU ecosystem! -- [llama_index_rag](./llama_index_rag/README.md): Build a lightweight RAG system based on llama_index -- [gradio_app](./gradio_app/README.md): Build a web app based on gradio -- [web_demo](./web_demo/README.md): MinerU online [demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/) localized deployment version -- [web_api](./web_api/README.md): Web API Based on FastAPI -- [multi_gpu](./multi_gpu/README.md): Multi-GPU parallel processing based on LitServe diff --git a/projects/README_zh-CN.md b/projects/README_zh-CN.md index 96374cd38..08bd1dee4 100644 --- a/projects/README_zh-CN.md +++ b/projects/README_zh-CN.md @@ -1,9 +1,7 @@ -# 欢迎来到 MinerU 项目列表 +# 欢迎来到 MinerU 项目（Archived） -## 项目列表 +>[!NOTE] +> 为整合资源，本仓库目前已停止接收新项目提交和已有项目的维护请求。 +> 如果您有基于 MinerU 的优秀项目希望分享，欢迎将项目链接提交至精心维护的社区资源库 [awesome-mineru](https://github.com/opendatalab/awesome-mineru)。 +> 感谢您对 MinerU 生态的支持与贡献！ -- [llama_index_rag](./llama_index_rag/README_zh-CN.md): 基于 llama_index 构建轻量级 RAG 系统 -- [gradio_app](./gradio_app/README_zh-CN.md): 基于 Gradio 的 Web 应用 -- [web_demo](./web_demo/README_zh-CN.md): MinerU在线[demo](https://opendatalab.com/OpenSourceTools/Extractor/PDF/)本地化部署版本 -- [web_api](./web_api/README.md): 基于 FastAPI 的 Web API -- [multi_gpu](./multi_gpu/README.md): 基于 LitServe 的多 GPU 并行处理 diff --git a/projects/gradio_app/README.md b/projects/gradio_app/README.md deleted file mode 100644 index c04b068e1..000000000 --- a/projects/gradio_app/README.md +++ /dev/null @@ -1,24 +0,0 @@ -## Installation - -MinerU(>=0.8.0) - > If you already have a functioning MinerU environment, you can skip this step. - > -[Deploy in CPU environment](https://github.com/opendatalab/MinerU?tab=readme-ov-file#quick-cpu-demo) - -[Deploy in GPU environment](https://github.com/opendatalab/MinerU?tab=readme-ov-file#using-gpu) - -Third-party Software - -```bash -pip install gradio gradio-pdf -``` - -## Start Gradio App - -```bash -python app.py -``` - -## Use Gradio App - -Access http://127.0.0.1:7860 in your web browser \ No newline at end of file diff --git a/projects/gradio_app/README_zh-CN.md b/projects/gradio_app/README_zh-CN.md deleted file mode 100644 index ce9b92c5e..000000000 --- a/projects/gradio_app/README_zh-CN.md +++ /dev/null @@ -1,24 +0,0 @@ -## 安装 - -MinerU(>=0.8.0) - >如已有正常运行的MinerU环境则可以跳过此步骤 -> -[在CPU环境部署](https://github.com/opendatalab/MinerU/blob/master/README_zh-CN.md#%E4%BD%BF%E7%94%A8cpu%E5%BF%AB%E9%80%9F%E4%BD%93%E9%AA%8C) - -[在GPU环境部署](https://github.com/opendatalab/MinerU/blob/master/README_zh-CN.md#%E4%BD%BF%E7%94%A8gpu) - -第三方软件 - -```bash -pip install gradio gradio-pdf -``` - -## 启动gradio应用 - -```bash -python app.py -``` - -## 使用gradio应用 - -在浏览器中访问 http://127.0.0.1:7860 \ No newline at end of file diff --git a/projects/gradio_app/app.py b/projects/gradio_app/app.py deleted file mode 100644 index a1d0484b1..000000000 --- a/projects/gradio_app/app.py +++ /dev/null @@ -1,228 +0,0 @@ -# Copyright (c) Opendatalab. All rights reserved. - -import base64 -import os -import time -import uuid -import zipfile -from pathlib import Path -import re - -import pymupdf -from loguru import logger - -from magic_pdf.libs.hash_utils import compute_sha256 -from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter -from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter -from magic_pdf.tools.common import do_parse, prepare_env - -import gradio as gr -from gradio_pdf import PDF - - -def read_fn(path): - disk_rw = DiskReaderWriter(os.path.dirname(path)) - return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN) - - -def parse_pdf(doc_path, output_dir, end_page_id, is_ocr, layout_mode, formula_enable, table_enable, language): - os.makedirs(output_dir, exist_ok=True) - - try: - file_name = f"{str(Path(doc_path).stem)}_{time.time()}" - pdf_data = read_fn(doc_path) - if is_ocr: - parse_method = "ocr" - else: - parse_method = "auto" - local_image_dir, local_md_dir = prepare_env(output_dir, file_name, parse_method) - do_parse( - output_dir, - file_name, - pdf_data, - [], - parse_method, - False, - end_page_id=end_page_id, - layout_model=layout_mode, - formula_enable=formula_enable, - table_enable=table_enable, - lang=language, - ) - return local_md_dir, file_name - except Exception as e: - logger.exception(e) - - -def compress_directory_to_zip(directory_path, output_zip_path): - """ - 压缩指定目录到一个 ZIP 文件。 - - :param directory_path: 要压缩的目录路径 - :param output_zip_path: 输出的 ZIP 文件路径 - """ - try: - with zipfile.ZipFile(output_zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf: - - # 遍历目录中的所有文件和子目录 - for root, dirs, files in os.walk(directory_path): - for file in files: - # 构建完整的文件路径 - file_path = os.path.join(root, file) - # 计算相对路径 - arcname = os.path.relpath(file_path, directory_path) - # 添加文件到 ZIP 文件 - zipf.write(file_path, arcname) - return 0 - except Exception as e: - logger.exception(e) - return -1 - - -def image_to_base64(image_path): - with open(image_path, "rb") as image_file: - return base64.b64encode(image_file.read()).decode('utf-8') - - -def replace_image_with_base64(markdown_text, image_dir_path): - # 匹配Markdown中的图片标签 - pattern = r'\!\[(?:[^\]]*)\]\(([^)]+)\)' - - # 替换图片链接 - def replace(match): - relative_path = match.group(1) - full_path = os.path.join(image_dir_path, relative_path) - base64_image = image_to_base64(full_path) - return f"" - - # 应用替换 - return re.sub(pattern, replace, markdown_text) - - -def to_markdown(file_path, end_pages, is_ocr, layout_mode, formula_enable, table_enable, language): - # 获取识别的md文件以及压缩包文件路径 - local_md_dir, file_name = parse_pdf(file_path, './output', end_pages - 1, is_ocr, - layout_mode, formula_enable, table_enable, language) - archive_zip_path = os.path.join("./output", compute_sha256(local_md_dir) + ".zip") - zip_archive_success = compress_directory_to_zip(local_md_dir, archive_zip_path) - if zip_archive_success == 0: - logger.info("压缩成功") - else: - logger.error("压缩失败") - md_path = os.path.join(local_md_dir, file_name + ".md") - with open(md_path, 'r', encoding='utf-8') as f: - txt_content = f.read() - md_content = replace_image_with_base64(txt_content, local_md_dir) - # 返回转换后的PDF路径 - new_pdf_path = os.path.join(local_md_dir, file_name + "_layout.pdf") - - return md_content, txt_content, archive_zip_path, new_pdf_path - - -latex_delimiters = [{"left": "$$", "right": "$$", "display": True}, - {"left": '$', "right": '$', "display": False}] - - -def init_model(): - from magic_pdf.model.doc_analyze_by_custom_model import ModelSingleton - try: - model_manager = ModelSingleton() - txt_model = model_manager.get_model(False, False) - logger.info(f"txt_model init final") - ocr_model = model_manager.get_model(True, False) - logger.info(f"ocr_model init final") - return 0 - except Exception as e: - logger.exception(e) - return -1 - - -model_init = init_model() -logger.info(f"model_init: {model_init}") - - -with open("header.html", "r") as file: - header = file.read() - - -latin_lang = [ - 'af', 'az', 'bs', 'cs', 'cy', 'da', 'de', 'es', 'et', 'fr', 'ga', 'hr', - 'hu', 'id', 'is', 'it', 'ku', 'la', 'lt', 'lv', 'mi', 'ms', 'mt', 'nl', - 'no', 'oc', 'pi', 'pl', 'pt', 'ro', 'rs_latin', 'sk', 'sl', 'sq', 'sv', - 'sw', 'tl', 'tr', 'uz', 'vi', 'french', 'german' -] -arabic_lang = ['ar', 'fa', 'ug', 'ur'] -cyrillic_lang = [ - 'ru', 'rs_cyrillic', 'be', 'bg', 'uk', 'mn', 'abq', 'ady', 'kbd', 'ava', - 'dar', 'inh', 'che', 'lbe', 'lez', 'tab' -] -devanagari_lang = [ - 'hi', 'mr', 'ne', 'bh', 'mai', 'ang', 'bho', 'mah', 'sck', 'new', 'gom', - 'sa', 'bgc' -] -other_lang = ['ch', 'en', 'korean', 'japan', 'chinese_cht', 'ta', 'te', 'ka'] - -all_lang = [""] -all_lang.extend([*other_lang, *latin_lang, *arabic_lang, *cyrillic_lang, *devanagari_lang]) - - -def to_pdf(file_path): - with pymupdf.open(file_path) as f: - if f.is_pdf: - return file_path - else: - pdf_bytes = f.convert_to_pdf() - # 将pdfbytes 写入到uuid.pdf中 - # 生成唯一的文件名 - unique_filename = f"{uuid.uuid4()}.pdf" - - # 构建完整的文件路径 - tmp_file_path = os.path.join(os.path.dirname(file_path), unique_filename) - - # 将字节数据写入文件 - with open(tmp_file_path, 'wb') as tmp_pdf_file: - tmp_pdf_file.write(pdf_bytes) - - return tmp_file_path - - -if __name__ == "__main__": - with gr.Blocks() as demo: - gr.HTML(header) - with gr.Row(): - with gr.Column(variant='panel', scale=5): - file = gr.File(label="Please upload a PDF or image", file_types=[".pdf", ".png", ".jpeg", "jpg"]) - max_pages = gr.Slider(1, 10, 5, step=1, label="Max convert pages") - with gr.Row(): - layout_mode = gr.Dropdown(["layoutlmv3", "doclayout_yolo"], label="Layout model", value="layoutlmv3") - language = gr.Dropdown(all_lang, label="Language", value="") - with gr.Row(): - formula_enable = gr.Checkbox(label="Enable formula recognition", value=True) - is_ocr = gr.Checkbox(label="Force enable OCR", value=False) - table_enable = gr.Checkbox(label="Enable table recognition(test)", value=False) - with gr.Row(): - change_bu = gr.Button("Convert") - clear_bu = gr.ClearButton(value="Clear") - pdf_show = PDF(label="PDF preview", interactive=True, height=800) - with gr.Accordion("Examples:"): - example_root = os.path.join(os.path.dirname(__file__), "examples") - gr.Examples( - examples=[os.path.join(example_root, _) for _ in os.listdir(example_root) if - _.endswith("pdf")], - inputs=pdf_show - ) - - with gr.Column(variant='panel', scale=5): - output_file = gr.File(label="convert result", interactive=False) - with gr.Tabs(): - with gr.Tab("Markdown rendering"): - md = gr.Markdown(label="Markdown rendering", height=900, show_copy_button=True, - latex_delimiters=latex_delimiters, line_breaks=True) - with gr.Tab("Markdown text"): - md_text = gr.TextArea(lines=45, show_copy_button=True) - file.upload(fn=to_pdf, inputs=file, outputs=pdf_show) - change_bu.click(fn=to_markdown, inputs=[pdf_show, max_pages, is_ocr, layout_mode, formula_enable, table_enable, language], - outputs=[md, md_text, output_file, pdf_show]) - clear_bu.add([file, md, pdf_show, md_text, output_file, is_ocr, table_enable, language]) - - demo.launch(server_name="0.0.0.0") \ No newline at end of file diff --git a/projects/gradio_app/examples/2list_1table.pdf b/projects/gradio_app/examples/2list_1table.pdf deleted file mode 100644 index dd9650bfd..000000000 Binary files a/projects/gradio_app/examples/2list_1table.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/3list_1table.pdf b/projects/gradio_app/examples/3list_1table.pdf deleted file mode 100644 index 5782751ad..000000000 Binary files a/projects/gradio_app/examples/3list_1table.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/academic_paper_formula.pdf b/projects/gradio_app/examples/academic_paper_formula.pdf deleted file mode 100644 index f1381cd27..000000000 Binary files a/projects/gradio_app/examples/academic_paper_formula.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/academic_paper_img_formula.pdf b/projects/gradio_app/examples/academic_paper_img_formula.pdf deleted file mode 100644 index ab8ce7ea0..000000000 Binary files a/projects/gradio_app/examples/academic_paper_img_formula.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/academic_paper_list.pdf b/projects/gradio_app/examples/academic_paper_list.pdf deleted file mode 100644 index ab1d86b50..000000000 Binary files a/projects/gradio_app/examples/academic_paper_list.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/complex_layout.pdf b/projects/gradio_app/examples/complex_layout.pdf deleted file mode 100755 index a4fc9c0f7..000000000 Binary files a/projects/gradio_app/examples/complex_layout.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/complex_layout_para_split_list.pdf b/projects/gradio_app/examples/complex_layout_para_split_list.pdf deleted file mode 100644 index ce34c6404..000000000 Binary files a/projects/gradio_app/examples/complex_layout_para_split_list.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/garbled_formula.pdf b/projects/gradio_app/examples/garbled_formula.pdf deleted file mode 100644 index a2c11939d..000000000 Binary files a/projects/gradio_app/examples/garbled_formula.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/magazine_complex_layout_images_list.pdf b/projects/gradio_app/examples/magazine_complex_layout_images_list.pdf deleted file mode 100644 index 8718fc0fe..000000000 Binary files a/projects/gradio_app/examples/magazine_complex_layout_images_list.pdf and /dev/null differ diff --git a/projects/gradio_app/examples/scanned.pdf b/projects/gradio_app/examples/scanned.pdf deleted file mode 100755 index f35e53dd9..000000000 Binary files a/projects/gradio_app/examples/scanned.pdf and /dev/null differ diff --git a/projects/gradio_app/header.html b/projects/gradio_app/header.html deleted file mode 100644 index 99ec40ea4..000000000 --- a/projects/gradio_app/header.html +++ /dev/null @@ -1,119 +0,0 @@ - - - - - -
-
-
-
-
\ No newline at end of file
diff --git a/projects/gradio_app/requirements.txt b/projects/gradio_app/requirements.txt
deleted file mode 100644
index f23b6756d..000000000
--- a/projects/gradio_app/requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-magic-pdf[full]>=0.8.0
-gradio
-gradio-pdf
\ No newline at end of file
diff --git a/projects/llama_index_rag/README.md b/projects/llama_index_rag/README.md
deleted file mode 100644
index 57cd9aeb4..000000000
--- a/projects/llama_index_rag/README.md
+++ /dev/null
@@ -1,160 +0,0 @@
-## Installation
-
-MinerU
-
-```bash
-git clone https://github.com/opendatalab/MinerU.git
-cd MinerU
-
-conda create -n MinerU python=3.10
-conda activate MinerU
-pip install .[full] --extra-index-url https://wheels.myhloli.com
-```
-
-Third-party software
-
-```bash
-# install
-pip install llama-index-vector-stores-elasticsearch==0.2.0
-pip install llama-index-embeddings-dashscope==0.2.0
-pip install llama-index-core==0.10.68
-pip install einops==0.7.0
-pip install transformers-stream-generator==0.0.5
-pip install accelerate==0.33.0
-
-# uninstall
-pip uninstall transformer-engine
-```
-
-## Environment Configuration
-
-```
-export DASHSCOPE_API_KEY={some_key}
-export ES_USER={some_es_user}
-export ES_PASSWORD={some_es_password}
-export ES_URL=http://{es_url}:9200
-```
-For instructions on obtaining a DASHSCOPE_API_KEY, refer to [documentation](https://help.aliyun.com/zh/dashscope/opening-service)
-
-## Usage
-
-### Data Ingestion
-
-```bash
-python data_ingestion.py -p some.pdf # load data from pdf
-
- or
-
-python data_ingestion.py -p /opt/data/some_pdf_directory/ # load data from multiples pdf which under the directory of {some_pdf_directory}
-```
-
-### Query
-
-```bash
-python query.py --question '{the_question_you_want_to_ask}'
-```
-
-## Example
-
-````bash
-# Start the es service
-docker compose up -d
-
-or
-
-docker-compose up -d
-
-
-# Set environment variables
-export ES_USER=elastic
-export ES_PASSWORD=llama_index
-export ES_URL=http://127.0.0.1:9200
-export DASHSCOPE_API_KEY={some_key}
-
-
-# Ingest data
-python data_ingestion.py example/data/declaration_of_the_rights_of_man_1789.pdf
-
-
-# Ask a question
-python query.py -q 'how about the rights of men'
-
-## outputs
-Please answer the question based on the content within ```:
- ```
- I. Men are born, and always continue, free and equal in respect of their rights. Civil distinctions, therefore, can be founded only on public utility.
- ```
- My question is：how about the rights of men。
-
-question: how about the rights of men
-answer: The statement implies that men are born free and equal in terms of their rights. Civil distinctions should only be based on public utility. However, it does not specify what those rights are. It is up to society and individual countries to determine and protect the specific rights of their citizens.
-
-````
-
-## Development
-
-`MinerU` provides a `RAG` integration interface, allowing users to specify a single input `pdf` file or a directory. `MinerU` will automatically parse the input files and return an iterable interface for retrieving the data.
-
-
-### API Interface
-
-```python
-from magic_pdf.integrations.rag.type import Node
-
-class RagPageReader:
- def get_rel_map(self) -> list[ElementRelation]:
- # Retrieve the relationships between nodes
- pass
- ...
-
-class RagDocumentReader:
- ...
-
-class DataReader:
- def __init__(self, path_or_directory: str, method: str, output_dir: str):
- pass
-
- def get_documents_count(self) -> int:
- """Get the number of pdf documents"""
- pass
-
- def get_document_result(self, idx: int) -> RagDocumentReader | None:
- """Retrieve the parsed content of a specific pdf"""
- pass
-
-
- def get_document_filename(self, idx: int) -> Path:
- """Retrieve the path of a specific pdf"""
- pass
-
-
-```
-
-Type Definitions
-
-```python
-
-
-class Node(BaseModel):
- category_type: CategoryType = Field(description='Category') # Category
- text: str | None = Field(description='Text content', default=None)
- image_path: str | None = Field(description='Path to image or table (table may be stored as an image)', default=None)
- anno_id: int = Field(description='Unique ID', default=-1)
- latex: str | None = Field(description='LaTeX output for equations or tables', default=None)
- html: str | None = Field(description='HTML output for tables', default=None)
-
-
-
-```
-
-Tables can be stored in one of three formats: image, LaTeX, or HTML.
-`anno_id` is a globally unique ID for each Node. It can be used later to match this Node with other Nodes. The relationships between nodes can be retrieved using the `get_rel_map` method. Users can use `anno_id` to link nodes and construct a RAG index that includes node relationships.
-
-
-### Node Relationship Matrix
-
-| | image_body | table_body |
-| -------------- | ---------- | ---------- |
-| image_caption | sibling | |
-| table_caption | | sibling |
-| table_footnote | | sibling |
diff --git a/projects/llama_index_rag/README_zh-CN.md b/projects/llama_index_rag/README_zh-CN.md
deleted file mode 100644
index 4d3bf005b..000000000
--- a/projects/llama_index_rag/README_zh-CN.md
+++ /dev/null
@@ -1,199 +0,0 @@
-
-
-## 介绍
-
-`MinerU` 提供数据 `API接口` 以支持用户导入数据到 `RAG` 系统。本项目将基于`通义千问`展示如何构建一个轻量级的 `RAG` 系统。
-
-
-
-
-
-
- - MinerU: PDF Extraction Demo -
-
- A one-stop, open-source, high-quality data extraction tool, supports
- PDF/webpage/e-book extraction.
-
-
-
-
-
-
- 
-