Scrape Twitter/X profiles at scale — no API keys needed. Collect profile links and usernames from any page.
Quick Start • Features • Configuration • FAQ • Contributing
Twitter Data Scraper is a free, open-source Twitter/X profile extraction tool built with Python and Selenium. It collects profile links and usernames from any Twitter/X page — followers, search results, lists, and more — and exports them to clean CSV files.
Most Twitter scraper tools either require expensive API access, break frequently, or are bloated with unnecessary dependencies. This scraper takes a different approach: zero API cost, lightweight (4 dependencies), and built for reliability with auto-save and graceful shutdown.
- Growth Hackers building follower lists for outreach campaigns
- Digital Marketers analyzing competitor audiences on Twitter/X
- Data Analysts collecting social media profile datasets
- Researchers studying Twitter/X user networks and engagement
- Sales Teams building prospect lists from industry conversations
- Developers learning Selenium browser automation
- Zero API Cost - Scrapes directly from the Twitter/X web interface
- Headless Mode - Runs invisibly in the background for servers and CI/CD
- Smart Deduplication - O(1) set-based lookups ensure zero duplicate profiles
- Auto-Save - Periodically saves progress to CSV so you never lose data
- Graceful Shutdown - Press Ctrl+C anytime; data is saved before exit
- Human-Like Behavior - Randomized scroll pauses (1.5s–3.0s) to mimic natural browsing
- Anti-Detection - Stealth browser fingerprint to reduce automation detection
- Configurable - Adjust timeouts, scroll speed, and output file via CLI flags
- Clean Output - Full Twitter profile URLs and @handles exported to CSV
- Free & Open Source - MIT license, no API key required
| Requirement | Details |
|---|---|
| Python | Version 3.9 or higher (Download) |
| Google Chrome | Latest version (Download) |
| Twitter/X Account | Required for accessing follower lists and search results |
# 1. Clone the repository
git clone https://github.com/SoCloseSociety/TwitterDataScraper.git
cd TwitterDataScraper
# 2. (Recommended) Create a virtual environment
python -m venv venv
# Activate it:
# Windows:
venv\Scripts\activate
# macOS / Linux:
source venv/bin/activate
# 3. Install dependencies
pip install -r requirements.txtpython main.py --visible -o my_profiles.csv- A Chrome window opens at the Twitter login page
- Log in to your Twitter/X account manually
- Navigate to the page you want to scrape (followers, search results, lists...)
- Press ENTER in the terminal to start scraping
- Press Ctrl+C to stop — your data is automatically saved
python main.py -o my_profiles.csvNote: Headless mode requires pre-authenticated session cookies or a publicly accessible page.
python main.py --help| Option | Description | Default |
|---|---|---|
-o, --output FILE |
Output CSV filename | scraped_profiles.csv |
--visible |
Run browser in visible mode (required for manual login) | Off |
-m, --max-iterations N |
Maximum scroll iterations | Unlimited |
-v, --verbose |
Enable verbose debug logging | Off |
# Scrape profiles with visible browser
python main.py --visible -o tech_influencers.csv
# Limit to 100 scroll iterations
python main.py --visible -m 100 -o results.csv
# Debug mode
python main.py --visible -v -o debug_output.csv┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Open Chrome │───>│ Login manually │───>│ Navigate to │
│ via Selenium │ │ (visible mode) │ │ target page │
└──────────────────┘ └──────────────────┘ └──────────────────┘
│
▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Export to CSV │<───│ Deduplicate │<───│ Scroll & extract│
│ (auto-save) │ │ with set() │ │ profile links │
└──────────────────┘ └──────────────────┘ └──────────────────┘
Clean CSV output with full URLs — ready for analysis, CRM import, or further processing:
| Column | Description | Example |
|---|---|---|
profile_link |
Full URL to the Twitter profile | https://twitter.com/elonmusk |
profile_username |
Twitter handle with @ prefix | @elonmusk |
See examples/sample_output.csv for a complete sample.
Tune the scraper in twitter_scraper/config.py:
| Setting | Default | Description |
|---|---|---|
SCROLL_PIXELS |
800 |
Pixels to scroll per iteration |
SCROLL_PAUSE_MIN |
1.5s |
Minimum pause between scrolls |
SCROLL_PAUSE_MAX |
3.0s |
Maximum pause between scrolls |
MAX_STALE_ITERATIONS |
60 |
Iterations without new data before auto-stop |
CSV_BATCH_INTERVAL |
10 |
Save to CSV every N iterations |
PAGE_LOAD_TIMEOUT |
30s |
Page load timeout |
TwitterDataScraper/
├── twitter_scraper/ # Core package
│ ├── __init__.py # Package exports (v1.0.0)
│ ├── browser.py # Chrome WebDriver lifecycle management
│ ├── config.py # Centralized configuration constants
│ ├── scraper.py # Profile extraction engine
│ └── utils.py # Connectivity check, logging, helpers
├── examples/ # Sample output files
├── .github/ # Issue & PR templates
├── main.py # CLI entry point
├── requirements.txt # Python dependencies
├── pyproject.toml # Package metadata & build config
├── assets/
│ └── banner.svg # Project banner
├── LICENSE # MIT License
├── README.md # This file
├── CONTRIBUTING.md # Contribution guidelines
├── CODE_OF_CONDUCT.md # Community standards
└── .gitignore # Git ignore rules
The scraper uses webdriver-manager to automatically download the correct ChromeDriver. If you encounter issues:
pip install --upgrade webdriver-managerIf the scraper scrolls but doesn't find profiles:
- Make sure you navigated to a page with profile links (followers, search results, lists)
- Try using
--visiblemode to see what's happening - Twitter/X may have changed its HTML structure — open an issue
If you can't log in:
- Use
--visiblemode for manual login - Complete any security challenges (CAPTCHA, 2FA) manually
- Press ENTER only after fully logged in
chmod +x main.pyQ: Is this free? A: Yes. Twitter Data Scraper is 100% free and open source under the MIT license.
Q: Do I need a Twitter API key? A: No. This tool uses browser automation (Selenium), so no API key or developer account is needed.
Q: How many profiles can I scrape?
A: No hard limit. The scraper runs until no new profiles are found for 60 consecutive stale iterations. Use -m to set a maximum.
Q: Can I scrape followers of a specific account? A: Yes. Navigate to the followers page of any account, press ENTER, and the scraper will collect all visible profiles.
Q: Does it work on Mac / Linux? A: Yes. Fully cross-platform on Windows, macOS, and Linux.
Q: Can I run it without a browser window?
A: Yes. Headless mode is the default. Use --visible only when you need to log in manually.
| Feature | Twitter Data Scraper | Twitter API v2 | Manual Copy-Paste | Paid Tools |
|---|---|---|---|---|
| Price | Free | Free (limited) | Free | $50-200/mo |
| API key required | No | Yes | No | Yes |
| Rate limits | Browser-based | 500K tweets/mo | N/A | Varies |
| Profile extraction | Yes | Yes | Manual | Yes |
| Auto-save | Yes | N/A | N/A | Varies |
| Open source | Yes | N/A | N/A | No |
| Cross-platform | Yes | Any | Yes | Web only |
Contributions are welcome! Please read the Contributing Guide before submitting a pull request.
- Found a bug? Open an issue
- Have an idea? Request a feature
This project is licensed under the MIT License.
This tool is provided for educational and research purposes only. Users are responsible for ensuring their use complies with Twitter/X's Terms of Service and all applicable laws. The authors are not responsible for any misuse of this software. Always scrape responsibly and respect rate limits.
If this project helps you, please give it a star!
It helps others discover this tool.
Built with purpose by SoClose — Digital Innovation Through Automation & AI
Website •
LinkedIn •
Twitter •
Contact