diff --git a/README.md b/README.md index 59eade6..eb5b651 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,8 @@ WhatsAppWizard empowers users to easily download online content and create perso - [Running the Project](#running-the-project) 5. [Usage Examples πŸ’‘](#usage-examples-) 6. [Project Structure πŸ“](#project-structure-) -7. [Contributing 🀝](#contributing-) +7. [Download Algorithm πŸ“₯](#download-algorithm-) +8. [Contributing 🀝](#contributing-) 9. [Deployment πŸš€](#deployment-) 10. [API Documentation πŸ“š](#api-documentation-) 11. [Roadmap πŸ—ΊοΈ](#roadmap-) @@ -170,6 +171,8 @@ WhatsAppWizard/ β”œβ”€β”€ .github/workflows/ # GitHub Actions workflows β”œβ”€β”€ .dockerignore # Files to ignore in Docker builds β”œβ”€β”€ .env.example # Example environment variables +β”œβ”€β”€ docs/ # Documentation files +β”‚ └── DOWNLOAD_ALGORITHM.md # Detailed download algorithm documentation β”œβ”€β”€ ecosystem.config.js # PM2 configuration file β”œβ”€β”€ nodemon.json # Nodemon configuration file β”œβ”€β”€ package.json # Project dependencies and scripts @@ -185,6 +188,18 @@ WhatsAppWizard/ β”œβ”€β”€ docker-compose.yml # Docker Compose configuration ``` +## Download Algorithm πŸ“₯ + +The WhatsAppWizard bot uses a sophisticated multi-platform download system that supports TikTok, Instagram, Facebook, YouTube, and Twitter. The algorithm implements: + +- **Platform Detection**: Automatic identification of content source +- **Queue-Based Processing**: Asynchronous download handling using BullMQ +- **Retry Mechanisms**: Robust error handling with platform-specific retry logic +- **File Management**: Organized storage with timestamp-based naming +- **Real-time Status Updates**: Live feedback to users via WhatsApp + +For detailed technical documentation about the download algorithm, including flow diagrams, platform-specific implementations, and error handling strategies, see: **[πŸ“– Download Algorithm Documentation](docs/DOWNLOAD_ALGORITHM.md)** + ## Contributing 🀝 We welcome contributions to WhatsAppWizard! Please follow these guidelines: diff --git a/docs/DOWNLOAD_ALGORITHM.md b/docs/DOWNLOAD_ALGORITHM.md new file mode 100644 index 0000000..ad05186 --- /dev/null +++ b/docs/DOWNLOAD_ALGORITHM.md @@ -0,0 +1,336 @@ +# Download Algorithm Documentation + +## Overview + +The WhatsAppWizard bot implements a sophisticated multi-platform download system that allows users to download media content from popular social media platforms directly through WhatsApp. The system is designed with scalability, reliability, and platform flexibility in mind. + +## Supported Platforms + +- **TikTok** - Video downloads +- **Instagram** - Photos and videos +- **Facebook** - Photos and videos +- **YouTube** - Video downloads (returns direct URL) +- **Twitter** - Photos and videos + +## Architecture Components + +### 1. DownloadService (`src/services/Download.ts`) +The core service responsible for orchestrating the download process across different platforms. + +### 2. QueueService (`src/services/Queue.ts`) +Manages asynchronous job processing using BullMQ with Redis backend for scalability. + +### 3. Platform-Specific Downloaders +Each platform has its own implementation optimized for that platform's API and content structure. + +### 4. File Management System +Handles file storage, organization, and metadata management on disk. + +## Algorithm Flow + +For detailed visual flow diagrams, see: **[πŸ“Š Download Flow Diagrams](DOWNLOAD_FLOW_DIAGRAM.md)** + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ User sends β”‚ +β”‚ URL via β”‚ +β”‚ WhatsApp β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ URL Validation β”‚ +β”‚ & Platform β”‚ +β”‚ Detection β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Job Queue β”‚ +β”‚ Creation β”‚ +β”‚ (BullMQ) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Platform- β”‚ +β”‚ Specific β”‚ +β”‚ Download β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ File Storage β”‚ +β”‚ & Metadata β”‚ +β”‚ Processing β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Response β”‚ +β”‚ to User β”‚ +β”‚ via WhatsApp β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## Detailed Process Breakdown + +### Phase 1: URL Processing and Platform Detection + +1. **URL Reception**: User sends a URL through WhatsApp +2. **Platform Detection**: Uses `detectPlatformFromURL()` from snapsaver-downloader +3. **Validation**: Ensures the URL is from a supported platform +4. **Job Creation**: Creates a download job with unique ID + +```typescript +const platform = this.detectPlatform(url); +if (!platform) { + throw new Error("Invalid URL or unsupported platform."); +} +``` + +### Phase 2: Queue Management + +The system uses BullMQ for robust job processing: + +- **Concurrency**: 4 concurrent workers +- **Job Persistence**: Redis-backed storage +- **Status Tracking**: PENDING β†’ DOWNLOADING β†’ COMPLETED/FAILED +- **Cleanup**: Automatic removal of old jobs (24 hours) + +```typescript +this.downloaderWorker = new Worker( + "whatsapp-bot-downloader-queue", + async (job) => { + // Download processing logic + }, + { + connection: this.redis, + concurrency: 4, + removeOnComplete: { + age: 60 * 60 * 24, // 24 hours + count: 100, // Keep last 100 jobs + } + } +); +``` + +### Phase 3: Platform-Specific Download Strategies + +#### TikTok Downloads +- **API**: nayan-video-downloader +- **Retry Logic**: 3 attempts with 2-second delays +- **Content**: Direct video downloads + +```typescript +private async TikTok(url: string): Promise { + const endpoint = `https://nayan-video-downloader.vercel.app/tikdown?url=${encodeURIComponent(url)}`; + + for (let attempt = 1; attempt <= 3; attempt++) { + try { + const res = await axios.get(endpoint); + return res.data.data.video; + } catch (error) { + if (attempt < 3) { + await new Promise(resolve => setTimeout(resolve, 2000)); + } + } + } +} +``` + +#### Instagram & Facebook Downloads +- **API**: SnapSaver downloader +- **Resolution Handling**: Prioritizes SD resolution for mobile compatibility +- **Content**: Supports both images and videos + +```typescript +private async genericDownloader( + url: string, + platform: "Instagram" | "Facebook", + errorClass: any +): Promise { + const result = await SnapSaver(url); + + // Facebook-specific resolution handling + if (platform === "Facebook") { + const SDResolution = result.data.media.filter( + item => item.resolution?.includes("SD") + )[0]; + // Use SD resolution or fallback to last available + } +} +``` + +#### YouTube Downloads +- **API**: nayan-video-downloader +- **Special Handling**: Returns URL instead of direct download +- **Content**: Video URLs for streaming + +#### Twitter Downloads +- **API**: nayan-video-downloader +- **Quality**: Supports both SD and HD with fallback +- **Content**: Photos and videos + +### Phase 4: File Storage and Management + +1. **Download to Memory**: Content is first downloaded to buffer +2. **File Type Detection**: Uses `file-type` library for MIME type detection +3. **File Organization**: Platform-specific folder structure +4. **Naming Convention**: Timestamp-based unique filenames + +```typescript +private async DownloadOnDisk( + url: string, + platform: string +): Promise { + const res = await axios.get(url, { responseType: "arraybuffer" }); + const buffer = Buffer.from(res.data); + const fileType = await fileTypeFromBuffer(buffer); + + const DownloadPath = ConfigService.getDownloadPaths(platform); + const timestamp = Date.now(); + const fileName = `${timestamp}.${fileType?.ext ?? "bin"}`; + const filePath = `${DownloadPath}/${fileName}`; + + await FileService.saveFile(filePath, buffer); + return { path: filePath, type: mediaType, platform }; +} +``` + +### File Organization Structure +``` +public/ +└── media/ + β”œβ”€β”€ TikTok/ + β”œβ”€β”€ Instagram/ + β”œβ”€β”€ Facebook/ + β”œβ”€β”€ YouTube/ + └── Twitter/ +``` + +## Error Handling and Resilience + +### Retry Mechanisms +- **TikTok**: 3 attempts with exponential backoff +- **Network Failures**: Automatic retry for transient errors +- **Platform-Specific Errors**: Custom error classes for each platform + +### Status Tracking +- **Database Integration**: Prisma ORM for status persistence +- **Real-time Updates**: Status changes are tracked throughout the process +- **Error Logging**: Comprehensive error reporting and analytics + +### Error Types +```typescript +// Platform-specific error classes +- InstagramError +- FacebookError +- Generic network errors +- File system errors +``` + +## Performance Optimizations + +### Concurrency Management +- **Worker Concurrency**: 4 simultaneous downloads +- **Queue Prioritization**: FIFO job processing +- **Resource Management**: Automatic cleanup of completed jobs + +### Memory Management +- **Streaming Downloads**: Large files handled via streams +- **Buffer Management**: Efficient memory usage for file processing +- **Cleanup**: Automatic removal of temporary files + +### Caching Strategy +- **Redis Integration**: Job state persistence +- **File Deduplication**: Prevents redundant downloads +- **Metadata Caching**: Platform detection results + +## Monitoring and Analytics + +### Event System +```typescript +export enum DownloadEvents { + DownloadCompleted = "downloadCompleted", + DownloadFailed = "downloadFailed", + DownloadProgress = "downloadProgress" +} +``` + +### Metrics Tracking +- **Success/Failure Rates**: Per platform statistics +- **Performance Metrics**: Download times and throughput +- **Error Analytics**: PostHog integration for error tracking + +### Queue Monitoring +```typescript +public async getDownloaderQueueCount(): Promise { + return { + waiting: await this.downloaderQueue.getWaitingCount(), + active: await this.downloaderQueue.getActiveCount(), + completed: await this.downloaderQueue.getCompletedCount(), + failed: await this.downloaderQueue.getFailedCount(), + delayed: await this.downloaderQueue.getDelayedCount(), + lastSuccessfulDownload: await this.getLastSuccessfulDownload() + } +} +``` + +## Integration with WhatsApp + +### Message Processing +1. **URL Detection**: Automatic link extraction from messages +2. **User Feedback**: Real-time status updates via WhatsApp +3. **Media Delivery**: Direct file sending or URL sharing (YouTube) +4. **Error Communication**: User-friendly error messages + +### Rate Limiting +- **Per-user Limits**: Prevents abuse +- **Platform Limits**: Respects external API constraints +- **Queue Management**: Graceful handling of high load + +## Security Considerations + +### Input Validation +- **URL Sanitization**: Prevents malicious URL injection +- **Platform Verification**: Ensures URLs are from supported platforms +- **File Type Validation**: MIME type verification before storage + +### Resource Protection +- **File Size Limits**: Prevents storage exhaustion +- **Download Timeouts**: Prevents hanging operations +- **Access Control**: Secure file storage and retrieval + +## Future Enhancements + +### Planned Improvements +- [ ] Additional platform support (Pinterest, Reddit, etc.) +- [ ] Advanced file compression +- [ ] CDN integration for faster delivery +- [ ] Enhanced analytics and reporting +- [ ] API rate limiting improvements + +### Scalability Considerations +- **Horizontal Scaling**: Multiple worker instances +- **Database Optimization**: Enhanced query performance +- **Caching Layer**: Redis-based content caching +- **Load Balancing**: Request distribution across workers + +## Troubleshooting + +### Common Issues +1. **Platform API Changes**: Regular monitoring and updates required +2. **Network Timeouts**: Retry mechanisms handle transient failures +3. **Storage Issues**: Automatic cleanup and monitoring +4. **Queue Overload**: Graceful degradation and user notification + +### Debugging Tools +- **Comprehensive Logging**: All operations are logged +- **Queue Inspection**: Real-time queue status monitoring +- **Error Tracking**: PostHog integration for error analytics +- **Performance Metrics**: Download time and success rate tracking + +--- + +This download algorithm provides a robust, scalable solution for multi-platform media downloading through WhatsApp, with comprehensive error handling, monitoring, and user experience optimization. \ No newline at end of file diff --git a/docs/DOWNLOAD_FLOW_DIAGRAM.md b/docs/DOWNLOAD_FLOW_DIAGRAM.md new file mode 100644 index 0000000..1e2fc6b --- /dev/null +++ b/docs/DOWNLOAD_FLOW_DIAGRAM.md @@ -0,0 +1,227 @@ +# Download Algorithm Flow Diagram + +## High-Level System Flow + +``` +WhatsApp User Request + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ URL Processing β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Extract │───▢│ Platform │───▢│ Validate β”‚ β”‚ +β”‚ β”‚ URL from β”‚ β”‚ Detection β”‚ β”‚ Support β”‚ β”‚ +β”‚ β”‚ Message β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Queue Management β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Create │───▢│ Add to │───▢│ Worker β”‚ β”‚ +β”‚ β”‚ Download β”‚ β”‚ BullMQ β”‚ β”‚ Processing β”‚ β”‚ +β”‚ β”‚ Job β”‚ β”‚ Queue β”‚ β”‚ (4 workers) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Platform-Specific Download β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ TikTok β”‚ β”‚ Instagram β”‚ β”‚ Facebook β”‚ β”‚ ... β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β€’ 3 Retry β”‚ β”‚ β€’ SnapSaver β”‚ β”‚ β€’ SnapSaver β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β€’ nayan-api β”‚ β”‚ β€’ Multi β”‚ β”‚ β€’ SD Res β”‚ β”‚ β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ Media β”‚ β”‚ Priority β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ File Processing β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Download │───▢│ File Type │───▢│ Save to β”‚ β”‚ +β”‚ β”‚ to Buffer β”‚ β”‚ Detection β”‚ β”‚ Disk β”‚ β”‚ +β”‚ β”‚ β”‚ β”‚ (MIME) β”‚ β”‚ (/platform/) β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Response Delivery β”‚ +β”‚ β”‚ +β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ +β”‚ β”‚ Update │───▢│ Send Media │───▢│ Cleanup & β”‚ β”‚ +β”‚ β”‚ Status in β”‚ β”‚ to WhatsApp β”‚ β”‚ Analytics β”‚ β”‚ +β”‚ β”‚ Database β”‚ β”‚ User β”‚ β”‚ β”‚ β”‚ +β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## Platform-Specific Processing Details + +### TikTok Flow +``` +TikTok URL + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ nayan-video- β”‚ +β”‚ downloader API β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Retry Logic β”‚ +β”‚ (3 attempts β”‚ +β”‚ with 2s delay) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Direct Video β”‚ +β”‚ Download β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Instagram/Facebook Flow +``` +Instagram/Facebook URL + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ SnapSaver API β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Resolution β”‚ +β”‚ Processing β”‚ +β”‚ (SD Priority) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Multi-Media β”‚ +β”‚ Download β”‚ +β”‚ (Images/Videos)β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### YouTube Flow +``` +YouTube URL + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ nayan-video- β”‚ +β”‚ downloader API β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Return Stream β”‚ +β”‚ URL (No local β”‚ +β”‚ download) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Send URL to β”‚ +β”‚ User via β”‚ +β”‚ WhatsApp β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## Error Handling Flow + +``` +Download Request + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Try Download β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + Success? ◄─────┐ + β”‚ β”‚ β”‚ + Yesβ”‚ β”‚No β”‚ + β”‚ β”‚ β”‚ + β–Ό β–Ό β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Deliver β”‚ β”‚ Check β”‚ +β”‚ to User β”‚ β”‚ Retry β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Count β”‚ + β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Retry β”‚ + β”‚ Available? β”‚ + β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + Yesβ”€β”€β”€β”€β”˜ No + β”‚ β”‚ + β”‚ β–Ό + β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ β”‚ Report β”‚ + β”‚ β”‚ Error β”‚ + β”‚ β”‚ to User β”‚ + β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + └─▢ Wait & Retry +``` + +## Database Status Tracking + +``` +Job Creation + β”‚ + β–Ό + PENDING + β”‚ + β–Ό + DOWNLOADING + β”‚ + β”œβ”€β–Ά COMPLETED ─▢ Media Sent + β”‚ + └─▢ FAILED ────▢ Error Report +``` + +## Queue Management + +``` +New Download Request + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Add to Queue β”‚ +β”‚ (BullMQ) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Worker Pool β”‚ +β”‚ (4 workers) β”‚ +β”‚ Pick up job β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Process β”‚ +β”‚ Download β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Emit Events β”‚ +β”‚ (Completed/ β”‚ +β”‚ Failed) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` \ No newline at end of file diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..8b0a85a --- /dev/null +++ b/docs/README.md @@ -0,0 +1,49 @@ +# Documentation + +Welcome to the WhatsAppWizard documentation! This directory contains detailed technical documentation for various aspects of the system. + +## Available Documentation + +### πŸ“₯ [Download Algorithm](DOWNLOAD_ALGORITHM.md) +Comprehensive technical documentation explaining the multi-platform download system, including: +- Architecture overview and components +- Platform-specific download strategies +- Error handling and retry mechanisms +- Performance optimizations +- Monitoring and analytics +- Security considerations + +### πŸ“Š [Download Flow Diagrams](DOWNLOAD_FLOW_DIAGRAM.md) +Visual flow diagrams showing: +- High-level system flow +- Platform-specific processing flows +- Error handling flow +- Database status tracking +- Queue management + +## Quick Reference + +### Supported Platforms +- **TikTok** - Video downloads with 3-retry logic +- **Instagram** - Photos and videos via SnapSaver +- **Facebook** - Photos and videos with SD resolution priority +- **YouTube** - Video URL streaming (no local download) +- **Twitter** - Photos and videos with SD/HD fallback + +### Key Technologies +- **BullMQ** - Job queue management +- **Redis** - Queue persistence and caching +- **Prisma** - Database ORM +- **WhatsApp Web.js** - WhatsApp integration +- **SnapSaver** - Instagram/Facebook API +- **nayan-video-downloader** - TikTok/YouTube/Twitter API + +### Performance Characteristics +- **Concurrency**: 4 simultaneous download workers +- **Queue Cleanup**: 24-hour retention policy +- **Retry Logic**: Platform-specific retry mechanisms +- **File Organization**: Platform-based folder structure + +--- + +For general setup and usage information, see the main [README.md](../README.md). \ No newline at end of file