Local copy of Directory Listing with wget

2012.07.26

Well, sometime You need to get local copy (or not local:) ) of some folders with Directory Listing enabled. It usually looks like basic webpage with “Index of …” at title

wget for each folder that mirrored, create multiple file copies of dynamically created Index page:

index.html 
index.html?C=D;O=A 
index.html?C=D;O=D 
index.html?C=M;O=A 
index.html?C=M;O=D 
index.html?C=N;O=A 
etc... 

Annoying, a lot of useless request to server (that already overloaded by Your good will) and not nice for eye to watch Your local copy.
Don't found simple solution on Google, so here is mine:

 wget -r -p -np -e robots=off -U mozilla -R index.html* http://website/file_archive/ 

Explanation: 
-r - recursive 
-p - get all 
-np - don’t ascend to the parent directory 
-e robots=off - don't care about what robots.txt say
-U mozilla - I am Mozilla! :))
-R index.html* - reject files index.html* (dangerous if files in subfolder include index.html files, but I am talking about archives)
Have fun and try to respect those whom sites You dump.

Leave a comment