Smart Downloader

Smart file downloader with multi-threading, resumable downloads, and progress display

Overview

Smart file downloader with multi-threaded concurrent downloads, resumable download support, progress display, and file integrity verification.

Core Features

• Multi-threaded concurrent downloads: Use Semaphore to control concurrency, improve download speed
• Resumable downloads: Support resuming from breakpoint, avoid repeated downloads
• Progress display: Rich progress bar showing download progress, speed, remaining time
• File integrity verification: Automatically verify file integrity after download
• Custom request headers: Support custom User-Agent, Referer, etc.
• Error retry: Automatically retry failed downloads, default max 5 times
• Proxy support: Support HTTP/HTTPS/SOCKS5 proxies
• Temporary file management: Download to temporary directory, automatically move to target directory after completion

Usage

Provide a list of download URLs, and the script will automatically download all files.

Parameters

• urls: List of download URLs (required, comma separated or file path)
• output_dir: Output directory (required)
• max_workers: Max concurrency (optional, default 4)
• chunk_size: Chunk size (optional, default 2MB)
• timeout: Request timeout (optional, default 10 seconds)
• retry: Max retry times (optional, default 5)
• headers: Custom request headers (optional, JSON format)
• proxy: Proxy address (optional)

Example

bash

python scripts/smart_download.py urls.txt output/
python scripts/smart_download.py "https://example.com/file1.jpg,https://example.com/file2.jpg" output/
python scripts/smart_download.py urls.txt output/ --max-workers 8 --timeout 30
python scripts/smart_download.py urls.txt output/ --proxy http://127.0.0.1:7890

URL File Format

Supports two ways to provide URLs:

1. Command line comma separated:

   text

   python scripts/smart_download.py "url1,url2,url3" output/
   

2. File path (one URL per line):

   text

   https://example.com/file1.jpg
   https://example.com/file2.jpg
   https://example.com/file3.mp4
   

Output Format

• Files are saved in output_dir directory
• Temporary files are saved in {output_dir}/.temp/ directory
• Temporary directory is automatically cleaned after download

Progress Display

During download, the following information is displayed:

• File name
• Download progress (percentage)
• Download speed (MB/s)
• Remaining time
• Downloaded/Total size

Tech Stack

• Python 3.11+
• httpx 0.28.1
• aiofiles 25.1.0
• rich 14.3.3

Performance Optimization Tips

1. Concurrency: Adjust based on network and target server (recommended 4-8)
2. Chunk size: For large files (>100MB), consider increasing to 4-8MB
3. Timeout: For slow networks, consider increasing to 30-60 seconds
4. Retry times: For unstable networks, consider increasing to 10 times

Error Handling

• Automatically retry failed downloads
• Display detailed error information
• Failed download files are logged
• Support manual cleanup and re-download

Legal Disclaimer

This skill is for learning and research purposes only.

Users must comply with the following principles when using this skill:

1. Not for any commercial use
2. Comply with target platform's terms of service and robots.txt rules
3. Do not conduct large-scale downloads or cause operational disruption to the platform
4. Reasonably control request frequency to avoid burdening the target platform
5. Do not use for any illegal or improper purposes

By using this skill, you agree to comply with the above principles.