This Repo contains our benchmarking effort to evaluate VLMs on their chip design knowledge. Partial of the collection is displayed (Analog and Digital design, Semiconductor Manufacturing), and the full data will be accessible at final manuscript release stage because we are undergoing necessary legal approval.
| Data | Total | Multiple Choice | Short Answer |
|---|---|---|---|
| Number | 142 | 99 | 43 |
| Visual | Type | Count | |
| schematic | 53 | ||
| diagram | 29 | ||
| layout | 16 | ||
| table | 5 | ||
| mixed | 5 | ||
| structure | 3 | ||
| figure | 3 | ||
| curve | 4 | ||
| flow | 1 | ||
| equations | 1 | ||
| neuralnets | 1 | ||
| equation | 1 | ||
| Prompt Token | mean | 33.48 | |
| std | 54.87 | ||
| min | 5 | ||
| 25% | 11 | ||
| 50% | 16 | ||
| 75% | 28.75 | ||
| max | 370 |