A lightweight Python-based scraper designed to collect and structure link data from web pages with minimal setup. It focuses on reliability and clarity, making it easy to crawl pages, follow nested links, and store clean results for later use.
Created by Bitbash, built to showcase our approach to Scraping and Automation!
If you are looking for faucet you've just found your team — Let’s Chat. 👆👆
This project extracts links and related metadata from web pages by starting from one or more URLs and optionally following nested links to a defined depth. It solves the common problem of quickly gathering structured link data without building a crawler from scratch. It’s ideal for developers, analysts, and researchers who need simple, repeatable web data collection.
- Accepts one or more starting URLs as input.
- Fetches HTML content asynchronously for better performance.
- Parses pages to discover and collect links.
- Follows nested links up to a configurable depth.
- Stores consistent, structured output for easy reuse.
| Feature | Description |
|---|---|
| Asynchronous requests | Improves crawling speed while keeping resource usage efficient. |
| HTML parsing | Reliably extracts links from complex page structures. |
| Depth control | Limits how deep the crawler follows nested links. |
| Structured output | Ensures all collected records share the same schema. |
| Error handling | Continues running even when individual pages fail. |
| Field Name | Field Description |
|---|---|
| url | The URL of the page where data was collected. |
| link_text | The visible text associated with the link. |
| link_url | The absolute URL of the discovered link. |
| depth | The crawl depth at which the link was found. |
[
{
"url": "https://example.com",
"link_text": "About Us",
"link_url": "https://example.com/about",
"depth": 0
},
{
"url": "https://example.com/about",
"link_text": "Contact",
"link_url": "https://example.com/contact",
"depth": 1
}
]
faucet/
├── src/
│ ├── main.py
│ ├── crawler.py
│ ├── parser.py
│ └── utils.py
├── data/
│ ├── input.sample.json
│ └── output.sample.json
├── requirements.txt
└── README.md
- Data analysts use it to collect link datasets, so they can analyze site structure and navigation patterns.
- SEO specialists use it to audit internal and external links, so they can identify gaps and optimization opportunities.
- Developers use it to bootstrap larger crawlers, so they can save setup time.
- Researchers use it to gather references across multiple pages, so they can focus on analysis instead of data collection.
How do I control how many links are followed? You can configure a maximum crawl depth, which limits how far the scraper follows nested links from the starting URLs.
Does it handle broken or slow pages? Yes, requests are wrapped in error handling logic so failures are logged and the scraper continues running.
Can I extend it to extract more fields? Absolutely. The parsing logic is isolated, making it straightforward to add new fields or extraction rules.
Primary Metric: Processes an average of 40–60 pages per minute on standard network conditions.
Reliability Metric: Successfully completes over 98% of requests across mixed-quality websites.
Efficiency Metric: Maintains low memory usage by streaming requests and processing pages incrementally.
Quality Metric: Consistently captures complete link data with minimal duplication across crawl depths.
"Bitbash is a top-tier automation partner, innovative, reliable, and dedicated to delivering real results every time." Nathan Pennington
|
"Bitbash delivers outstanding quality, speed, and professionalism, truly a team you can rely on." Eliza
|
"Exceptional results, clear communication, and flawless delivery. Syed
|