Thursday, September 3, 2009

Multi thread debian package download script

This script uses the wget download script that can be generated in synaptic. It requires sed, wget, synaptic(for generation), split, and bc.

The basic idea behind it is to utilize the internet connection by multithreading the apt package download. You can add as many mirriors as you like to get it up to the threshhold. This script should also lessen the load on your primary mirrior.

Things I'd like to add:
  • progress indicator (would be a big +).
  • download speed indicator (a little difficult due to the process backgrounding).
  • download control.
  • ability to adjust download ques based on rate of completion.
  • kill and resume.
  • installation on download completion.
  • threaded installation.

Just a funny note, I decided I didn't like wget's verbosity in the log files so I wanted to pass the -nv flag. Well alias's only work on interactive shells so I made a script that calls wget... called wget and added the dir to the system path... well lets just say I wasn't paying close enough attention the first time I tested it, and I fork bombed my Linux box. I fixed it by referencing the real wget by direct path as you can see in my script.

Script:
#!/bin/bash
##requires wget,bc,grep,
FILE='download.sh'
FILEPRE='download'
SOURCE0='debian.osuosl.org' #sourc0 needs to be your primary mirrior
SOURCE1='linux.csua.berkeley.edu'
SOURCE2='mirror.rit.edu'

NUM=3
##set the wget command params for the wget scripts

SPLIT=`wc -l $FILE | grep -o -E "[0-9]{1,5}"`
SPLIT=`echo "$SPLIT / $NUM" |bc`

split -d --lines=$SPLIT $FILE $FILEPRE

## start the building download env.
if [ ! -d "cache" ]; then
mkdir 'cache'
fi

cd './cache'

x=0
while [ $x -lt $NUM ]; do
CURRF="$FILEPRE"0$x
mv "../$CURRF" "./"

## prepend an alias to the file
cp $CURRF 1$CURRF
WGET=`which wget`
echo "PATH='../:$PATH'
export WGET=$WGET" > $CURRF
cat 1$CURRF >> $CURRF
rm 1$CURRF

if [ $x -ge 1 ]; then
ACTIVE=`eval 'echo $SOURCE'$x`
sed -i "s/$SOURCE0/$ACTIVE/g" "$CURRF"

fi

bash $CURRF &>out$x &

let x=x+1
done

exit



The wget replacement file:
Save this as wget in the same directory as the above script. Make it executable.

#!/bin/bash
$WGET -nv $@

1 comment:

  1. --Update For a multi-threaded downloader check out apt-fast. I downloads from a single mirror though.

    ReplyDelete