A comprehensive system for discovering and cataloging Luanti (formerly Minetest) mods, games, and modpacks from multiple sources including ContentDB, forum posts, and git repositories.
- ContentDB API: Search the official Luanti Content Database
- Forum Scraping: Extract mod information from Luanti forum posts
- Git Repository Analysis: Automatically detect Luanti mods in git repositories
- Work Queue System: Efficiently process large numbers of items
- SQLite Databases: Separate databases for mod data, work queues, and metadata
- Work Queues: Forum threads and git repositories are processed asynchronously
- Git Host Discovery: Automatically discovers and tracks self-hosted git instances
- Duplicate Prevention: Avoids reprocessing already known items
- Unit tests for all major components
- Mock-based testing for external API calls
- Test coverage for work queue management
- Clone the repository
- Install required dependencies:
pip install requests beautifulsoup4python mod_search.pyThis will:
- Initialize all databases
- Search ContentDB for mods, games, and texture packs
- Fetch forum thread lists and add them to work queues
- Search GitHub for Luanti-related repositories
# Check work queue status
python work_queue_manager.py status
# Refresh forum thread lists
python work_queue_manager.py refresh-forum
# Process forum threads (finds git repositories)
python work_queue_manager.py process-forum --batch-size 10 --max-batches 5
# Process git repositories (validates as Luanti mods)
python work_queue_manager.py process-git --batch-size 10 --max-batches 3Stores discovered mods, games, and modpacks with metadata:
- Basic info: name, title, description, author, type
- URLs: ContentDB, forum, repository, website
- Metadata: tags, dependencies, license, version requirements
- Source tracking: where the item was discovered
forum_queue.db: Forum threads to be processedgit_queue.db: Git repositories to be validatedgit_hosts.db: Discovered self-hosted git instancesnon_mod_repos.db: Repositories known to not contain Luanti content
from forum_search import fetch_forum_thread_list, process_forum_work_queue
# Fetch forum threads and add to work queue
threads = fetch_forum_thread_list("https://forum.luanti.org/viewforum.php?f=11")
# Process work queue
results = process_forum_work_queue(batch_size=10)from git_utils import is_git_repository_url, check_luanti_mod_repository
# Check if URL is a git repository
is_git, host_type = is_git_repository_url("https://github.com/user/repo")
# Validate if repository contains a Luanti mod
is_mod, metadata = check_luanti_mod_repository("https://github.com/user/luanti-mod")from db_utils import (init_db, add_forum_thread_to_queue,
get_unprocessed_forum_threads, save_result)
# Initialize all databases
init_db()
# Add items to work queues
add_forum_thread_to_queue(url, title, thread_type)
# Get unprocessed items
threads = get_unprocessed_forum_threads(limit=10)
# Save discovered mod/game
save_result(mod_data, source="forum")The system can detect and validate Luanti content in repositories from multiple git hosting providers:
- GitHub (github.com)
- GitLab (gitlab.com and self-hosted instances)
- Codeberg (codeberg.org)
- Bitbucket (bitbucket.org)
- Gitea/Forgejo (self-hosted instances)
- SourceForge (sourceforge.net)
- Generic .git URLs
- mod.conf Detection: Looks for
mod.conffile (Luanti mods) - game.conf Detection: Looks for
game.conffile (Luanti games) - Modpack Detection: Checks for
mods/directory structure - Metadata Extraction: Parses configuration files for mod information
The system uses work queues to efficiently process large numbers of forum threads and git repositories:
- Scrape forum pages for thread lists
- Add threads to work queue (avoiding duplicates)
- Process threads to extract git repository links
- Validate repositories as Luanti content
- Save results to main database
- Receive repositories from forum processing or direct search
- Add to git work queue
- Check if repository contains Luanti content
- Extract mod/game metadata
- Save to main database or mark as non-mod
Run the comprehensive test suite:
# Test forum search functionality
python -m unittest test_forum_search.py -v
# Test git utilities
python -m unittest test_git_utils.py -v
# Test ContentDB integration
python -m unittest test_contentdb_api.py -v- Mod Releases:
https://forum.luanti.org/viewforum.php?f=11 - Games:
https://forum.luanti.org/viewforum.php?f=15
- Query terms:
["luanti", "minetest"] - Topics:
["luanti-mod", "minetest-mod"]
- Default batch size: 10 items
- Configurable via command line arguments
- Rate limiting to avoid overwhelming servers
- Network Timeouts: Configurable timeouts for API calls
- Rate Limiting: Built-in delays to respect server limits
- Duplicate Detection: Prevents reprocessing known items
- Error Logging: Tracks failed processing attempts
- Graceful Degradation: Continues processing if individual items fail
- Add URL pattern to
git_utils.py - Implement API integration if available
- Add to test cases
- Update documentation
- Create new search module
- Implement work queue integration
- Add database schema updates
- Write comprehensive tests
- Fork the repository
- Create a feature branch
- Add tests for new functionality
- Ensure all tests pass
- Submit a pull request
[Add your license information here]
- ✅ Work queue system for forum threads and git repositories
- ✅ Multi-source mod discovery (ContentDB, forums, GitHub)
- ✅ Git repository validation for Luanti content
- ✅ Self-hosted git instance discovery
- ✅ Comprehensive test suite
- ✅ Command-line work queue manager
- ✅ Duplicate detection and prevention
- ✅ Support for mods, games, and modpacks
- Web interface for browsing discovered mods
- Integration with additional git hosting providers
- Mod dependency analysis and visualization
- Automated mod compatibility checking
- RSS/webhook notifications for new discoveries
- Integration with mod managers and launchers