Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# LLM Configuration
LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=your-api-key-here
LLM_MODEL=gpt-4o
USE_LLM=True

# Debug
METRICS_DEBUG=False
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ coverage.xml

webmainbench.egg-info/*

# cache files
webmainbench/.cache/

# PyPI packaging
build/
dist/
Expand Down
25 changes: 11 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,31 +148,28 @@ hf_hub_download(
)
```

### Configure LLM (Optional)

LLM-enhanced content splitting improves formula/table/code extraction accuracy. To enable it, copy `.env.example` to `.env` and fill in your API credentials:

```bash
cp .env.example .env
# Edit .env and set LLM_BASE_URL, LLM_API_KEY, LLM_MODEL
```

### Run an Evaluation

```python
from webmainbench import DataLoader, Evaluator, ExtractorFactory

dataset = DataLoader.load_jsonl("data/WebMainBench_545.jsonl")
extractor = ExtractorFactory.create("trafilatura")
result = Evaluator().evaluate(dataset, ExtractorFactory.create("trafilatura"))

evaluator = Evaluator(llm_config={
"use_llm": True,
"llm_base_url": "https://api.openai.com/v1",
"llm_api_key": "sk-xxxxxxxxxxxx",
"llm_model": "gpt-4o",
})
result = evaluator.evaluate(dataset, extractor)
m = result.overall_metrics

print(f"Overall Score: {result.overall_metrics['overall']:.4f}")
```

If you don't need LLM-enhanced content splitting (for formula/table/code extraction), disable it explicitly:

```python
evaluator = Evaluator(llm_config={"use_llm": False})
```

### Compare Multiple Extractors

```python
Expand Down
25 changes: 11 additions & 14 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,31 +148,28 @@ hf_hub_download(
)
```

### 配置 LLM(可选)

LLM 增强内容拆分可提升公式/表格/代码的抽取精度。如需启用,将 `.env.example` 复制为 `.env` 并填写 API 信息:

```bash
cp .env.example .env
# 编辑 .env,设置 LLM_BASE_URL、LLM_API_KEY、LLM_MODEL
```

### 运行评测

```python
from webmainbench import DataLoader, Evaluator, ExtractorFactory

dataset = DataLoader.load_jsonl("data/WebMainBench_545.jsonl")
extractor = ExtractorFactory.create("trafilatura")
result = Evaluator().evaluate(dataset, ExtractorFactory.create("trafilatura"))

evaluator = Evaluator(llm_config={
"use_llm": True,
"llm_base_url": "https://api.openai.com/v1",
"llm_api_key": "sk-xxxxxxxxxxxx",
"llm_model": "gpt-4o",
})
result = evaluator.evaluate(dataset, extractor)
m = result.overall_metrics

print(f"Overall Score: {result.overall_metrics['overall']:.4f}")
```

如不需要 LLM 增强内容拆分(用于公式/表格/代码抽取),可显式关闭:

```python
evaluator = Evaluator(llm_config={"use_llm": False})
```

### 多抽取器对比

```python
Expand Down
Loading
Loading