american-dedup

A terminal user interface (TUI) application for finding and managing duplicate files using fdupes.

Key Principles

Priority-based deduplication: You decide which folders have priority. Duplicates are moved FROM target folders while files in non-target folders are always kept. This gives you full control over which copies to preserve.

Safe by design: This application never deletes files. Duplicates are moved to a timestamped folder (__dup_YYYYMMDD_HHMMSS), preserving the original directory structure. You can review the moved files and delete them manually when you're confident they're no longer needed.

Screenshots

Features

• Two-panel folder selection: Select folders to scan and mark target folders for deduplication
• Smart duplicate handling: Files in target folders are moved, files in other folders are kept
• Preview before action: See exactly which files will be moved and which will be kept
• Real-time progress tracking: See percentage and elapsed time during scan
• Save/load configurations: Save folder selections for repeated use
• Internal duplicates option: Optionally handle duplicates within target folders only

Requirements

• Python 3.10+
• fdupes (must be installed on your system)
• Linux (tested on Ubuntu/Debian)

Installing fdupes

# Debian/Ubuntu
sudo apt install fdupes

# Fedora
sudo dnf install fdupes

# Arch Linux
sudo pacman -S fdupes

Installation

From source

git clone https://github.com/elmisi/american-dedup.git
cd american-dedup
./install.sh

This creates a virtual environment and installs the command american-dedup to ~/.local/bin/.

Using pip

pip install american-dedup

Usage

american-dedup [starting_path]

Workflow

1. Add folders to scan: Click "+ Add" to select folders containing files to check for duplicates
2. Mark target folders: Check the boxes next to folders from which duplicates should be moved
3. Configure options: Optionally enable "Include duplicates within target folders"
4. Scan: Click "Scan" to run fdupes and analyze duplicates
5. Preview: Review the files that will be moved vs kept
6. Execute: Click "Execute Move" to move duplicates to a timestamped folder

Keyboard shortcuts

• q - Quit the application
• F1 - Open help screen
• Escape - Go back to previous screen
• Arrow keys - Navigate tree in preview screen

How it works

The application uses a priority system:

• Files in non-target folders have high priority (they are KEPT)
• Files in target folders have low priority (they are MOVED)

When a duplicate group contains files from both:

• All copies in target folders are moved
• All copies in non-target folders are kept

When duplicates exist only within target folders:

• If "Include internal duplicates" is OFF: nothing is moved
• If "Include internal duplicates" is ON: first copy is kept, others are moved

Moved files are placed in a __dup_YYYYMMDD_HHMMSS folder, preserving the original folder structure.

Restoring Files

If you need to restore moved files, they are preserved in the timestamped folder with the original directory structure intact:

1. Navigate to the __dup_YYYYMMDD_HHMMSS folder
2. Browse the preserved directory structure
3. Copy or move files back to their original locations as needed

This manual approach gives you full control over what to restore and when.

Configuration

Configurations are saved to ~/.config/american-dedup/saved_configs.json

Logging

Detailed logs are written to /tmp/american-dedup.log for debugging.

Uninstallation

./uninstall.sh

Or if installed via pip:

pip uninstall american-dedup

Development

Regenerating Screenshots

After UI changes, regenerate the documentation screenshots:

.venv/bin/python scripts/take_screenshots.py

Screenshots are saved as SVG files in the screenshots/ directory.

License

MIT License - see LICENSE for details.

Acknowledgments

• fdupes by Adrian Lopez for the duplicate detection engine
• Textual for the TUI framework