cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: ftp serial transfers slow ... parallel fast

From: <farr_at_metservice.com>
Date: Thu, 11 Mar 2004 12:25:33 +1300

At 11:58 AM 3/11/2004, you wrote:
> > The obvious question is, how do I make it go faster without transferring
> > everything in parallel? If not already implicit in the answer to the
> > previous question, why would it seem slow from windows (perhaps I am just
> > imagining things ...) ?
>The first thing, ftp is an inherently slow protocol. That's why you'll
>notice curl has special support to add additional timeouts for FTP because
>FTP is notoriously slow. The other thing, when you are comparing Linux and
>Windows, is this the same machine (i.e. dual-boot)? I mean, if you're
>comparing your Windows machine on a 56K and say a Linux shell account on a
>T1, well then obviously the Linux download is going to be faster :)

They are on the same subnet plugged into the same switch. The windows box
is a P4 3.2G and the linux box is a P3 1G, they have the same intel network
card. It is a 100mbit switched network. FTP might be slow ... but that is
just rediculous.

>Also I noticed you mentioned smb. I know someone said they were working on a
>patch for libcurl to support smb, maybe that would be a better alternative
>for you? But, I don't know if that patch was ever completed.

cool ... didn't know that. That would be really useful ... but some lamer
put the requirement for ftp in this project.

> > By the way, if I dont add and remove the easy_curl handle from the multi
> > group between file transfers it seems to crash ... how bizarre. I am
>doing
> > something else wrong, or is that the way it is supposed to work?
>I'm not totally sure what you mean, you might need to be a bit more specific
>about your code.

I have a C++ CurlSocket (which represents an curl_easy_handle). I also
have a SocketSet (which represents a set of CurlSockets, TCPSockets (for
rpc and other nasty things), UDPSockets etc ...). Another object will be
asked to do a set of downloads from a particular host, it will try to use 1
CurlSocket to do all transfers from a particular host. When a particular
download has finished it will ask the CurlSocket to do a new download like
this:

    void CurlSocket::NextTransfer()
     {
         CURLM *curlMultiHandle = getCurlMultiHandle();
         curl_multi_remove_handle(curlMultiHandle, GetCurlHandle());
         curl_multi_add_handle(curlMultiHandle, GetCurlHandle());
           curl_easy_setopt(_curlHandle, CURLOPT_URL, _remoteURLFile.c_str());
              if (_isSending)
             {
               curl_easy_setopt(GetCurlHandle(), CURLOPT_INFILESIZE,
fileSize(localURL.c_str())) ;
                 _localStream = fopen(localURL.c_str(), "rb");
             }
             else
             {
                 _localStream = fopen(localURL.c_str(), "wb");
             }
     }

A little while later I call processCurlMessages, and curl_multi_perform and
my write/read callbacks proceed as normal. I do not understand why I
should need the curl_multi_remove_handle and curl_multi_add_handle calls.
Received on 2004-03-11