Monday, August 06, 2007

Using a download accelerator with apt-get

If dist-upgrading your Ubuntu or Debian GNU/Linux distro is taking a considerable amount of time (or says it will), then this might be a useful tip to speed things up a bit.

This is nothing new really; something I used to frequently back in the day when Debian was my primary desktop disto. You see I got sick of waiting for debs to complete downloading, not to mention the apt database being locked, so I wrote a small perl script which used prozilla to quickly download the debs into /var/cache/apt/archives.

Recently, my g/f told me to dist-upgrade her Ubuntu box, since she didn't have broadband @ home and thats when I remembered that I have forgotten where I'd put that script. So I wrote new one that was an even smaller script.

But before that, I wrote a tiny bash "one liner" which can also be used download debs but it may not work with some debs that have to be renamed by escaping some characters. But nevertheless it too can be useful for downloading one or two packages with dependencies.

I'm assuming your running this as root

# cd /var/cache/apt/archives
# apt-get -y --print-uris install package_name > debs.list
# egrep -o -e "(ht|f)tp://[^\']+" debs.list | xargs -l1 axel -a

Instead of using prozilla, I now use axel which is also another download accelerator. You should be able to easily apt-get install axel. Replace package_name with either a package name you are installing or dist-upgrade if you want to upgrade the system altogether. Be warned that you may have to rename some files before apt sees the file as being downloaded completely.

Now on to the "slightly longer, works better", perl script:
# cd /var/cache/apt/archives
# apt-get -y --print-uris install package_name > debs.list
# vi download-debs.pl


#!/usr/bin/perl

while(<>){

if(/([^ ]+)[ ]+([^ ]+)[ ]+/){
($url,$rendeb)=($1,$2);
$url=~s/[' ]//g;
$deb=$url; $deb=~s/.*\///;
$rendeb=~s/[ ]//g;
print "Downloading $deb \n";
system("axel -a $url");
if($deb ne $rendeb){
print "Renaming $deb to $rendeb \n";
system("mv $deb $rendeb");
}
}
}



# ./download-debs.pl debs.list

While your waiting for the download you might even be able to squeeze in a few new smaller packages, as the apt database isn't locked. Once the download is complete, just run the same command without the -print-uris option and your good to go.

Be warned that using a download accelerator might bring your sys admin running to your desk screaming before you can say "apt-get are we done yet!".

10 comments:

Sandaruwan said...

oh!! i wanted the same thing, and i didn't know about the print-uri option.. i ended up hacking the apt-get code : http://www.sandaru1.com/2007/02/14/enhance-apt-get-downloading-using-an-external-download-acceleratoraxelprozillaaget/

Genesis said...

Hi Geek with an attitude,
I like your Blog.
Stay tunned with the SL SFD 2007.
http://nightskymine.blogspot.com/2007/08/software-freedom-day-2007-sep-15h.html
Thank You.

Anonymous said...

There is a problem with the guide.

http://www.linuxquestions.org/questions/linux-software-2/using-a-download-accelerator-with-apt-get-595443/?posted=1#post2940945

linux666 said...

The apt-get command is wrong.

Bud said...

Nope works for me all the time :) What distribution are you trying this on? I am not sure if apt rpm supports the --print-uris option.

linux666 said...

I'm using Ubuntu 7.10 64 bit

Should some package be substituted for package_name? Which?

$ sudo apt-get -y --print-uris package_name
E: Invalid operation package_name

If I try this command with a specific package, it still doesn't work.

$ sudo apt-get -y --print-uris firefox
E: Invalid operation firefox

The syntax of the commands are wrong. This is from the man page of apt-get.

SYNOPSIS
apt-get [-hvs] [-o=config string] [-c=file] {[update] | [upgrade] |
[dselect-upgrade] | [install pkg...] | [remove pkg...] |
[purge pkg...] | [source pkg...] | [build-dep pkg...] | [check]
| [clean] | [autoclean] | [autoremove]}

vukko said...

Bud, you've left out the 'install':

apt-get -y --print-uris install package_name > debs.list

NEOLiVZ said...

Any way yours wont work, but it gave inspiration to write something workable, check it out here http://neolivz.blogspot.com/

NEOLiVZ said...
This comment has been removed by the author.
Bud said...

Ooops sorry I just realized I left out the install(as vukko pointed out) :)

@NEOLiVZ the perl script should work. I remember using it. Whats the error you get? The reason for the perl script was to handle the fact certain filenames needed to be url encoded or they end up getting re downloaded.