cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: multi-threaded/segmented downloads w/ libcurl

From: Anthony Bryan <albryan_at_comcast.net>
Date: Mon, 18 Dec 2006 02:40:23 -0500

Hi,

This is my last attempt at harassing up interest in metalink with libcurl in
2006. :)

metalink is an XML file that lists mirrors and checksums for downloading and
looks like this:

   <file name="openSUSE-10.2-GM-DVD-i386.iso">
    <os>Linux-x86</os>
    <size>3880814592</size>
    <verification>
     <hash type="md5">12b0f2fabb4d41586601f787d33ebc34</hash>
    </verification>
    <resources>
      <url type="http"
           location="jp"
           preference="10">
 
http://ftp.kddilabs.jp/Linux/packages/opensuse/distribution/openSUSE-current
/iso/dvd/openSUSE-10.2-GM-DVD-i386.iso
      </url>
      <url type="http"
           location="ch"
           preference="10">
 
http://mirror.switch.ch/ftp/mirror/opensuse/distribution/openSUSE-current/is
o/dvd/openSUSE-10.2-GM-DVD-i386.iso
      </url>
    </resources>
   </file>

it can also list partial file checksums. this is nice for large files, where
if there is an error in transfer, only the part with the error would need to
be re-downloaded, instead of the whole file.

                <verification>
                        <pieces length="1048576" type="sha1">
                                <hash
piece="0">e8230ecbc94375b8a4b8e3b8da7d376c09d14d8e</hash>
                                <hash
piece="1">37517b42ed7353eb4b0c59f901d33197778343d9</hash>
                </verification>

metalink is used in a few places, by openSUSE, OpenOffice.org, and other
distributions, mostly for ISO downloads:

http://en.opensuse.org/Released_Version#Metalink
http://distribution.openoffice.org/p2p/magnet.html

there is already a command line download utility (aria2) for metalink, but I
think it would be great if there was a way to use libcurl too.

if anyone is interested, please let me know.

(( Anthony Bryan
 )) Metalink [ http://www.metalinker.org ]
Received on 2006-12-18