Wayback Machine Download Website Files

To pull a full archived website from the Wayback Machine, the wayback-machine-downloader Ruby gem is the standard tool. Install it with gem install wayback_machine_downloader, then run it with the target domain.

It walks the Wayback Machine's CDX index, finds every archived URL under the domain, fetches each one with the id_ suffix to strip Wayback's UI, and saves the result to a local folder with the directory structure preserved.

Useful flags

The tool supports date constraints with --to and --from flags for limiting the snapshot range. Use --only with a path pattern to pull only matching sections (for example, only the /blog/ directory). Use --exclude with a regex to skip certain file types. The --all flag includes error pages and redirects, and --threads controls parallel download threads.

Do not push threads too high. The Wayback Machine throttles aggressive crawlers, and the tool's defaults are tuned to stay under the threshold. For larger sites, expect the run to take hours.

The CDX-server alternative

If the Ruby gem is not an option, the CDX server combined with wget works. Query the CDX API at https://web.archive.org/cdx/search/cdx with the domain, output format, and a collapse parameter to get a list of every archived URL. From there, a shell loop with wget and the id_ URL pattern downloads each one. More work, but more controllable.

What to expect from the result

Old sites had different file structures. The downloader reconstructs what Wayback captured, which is rarely 100% of the original. JavaScript-rendered content, dynamic asset loading, and anything behind auth are often missing or partial. Wayback's coverage is also patchy — small sites may have only a handful of snapshots, large sites may have millions but with gaps.

Before downloading an entire archived site, it is worth knowing what was actually captured. Arkibber lets you explore what the Internet Archive holds for a given domain, so you can assess coverage and identify the most useful snapshots before committing to a full download.