cURL / Mailing Lists / curl-library / Single Mail

curl-library

How to avoid multiple CWD for each file download

From: Anders Havn <anders.havn_at_gmail.com>
Date: Wed, 19 Jun 2013 12:48:13 +0200

Hi,

I'm downloading multiple files in a directory. The code run for each file
looks like this:

curl_global_init(CURL_GLOBAL_DEFAULT);
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, callBack);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, data);
curl_easy_setopt(curl, CURLOPT_URL, fileUrl);
curl_easy_setopt(curl, CURLOPT_POSTQUOTE, headerList);
CURLcode result = curl_easy_perform(curl);
curl_slist_free_all (headerList);
curl_easy_cleanup(curl);

The log from one file download looks the following:

2013-06-18 21:08:21> 220 FRITZ!Box6360Cable(kdg) FTP server ready.
2013-06-18 21:08:21> USER userX
2013-06-18 21:08:21> 331 Password required for userX.
2013-06-18 21:08:21> PASS XXX
2013-06-18 21:08:22> 230 User userX logged in.
2013-06-18 21:08:22> PWD
2013-06-18 21:08:22> 257 "/" is current directory.
2013-06-18 21:08:22> CWD WD-Elements1042-02
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> CWD Pictures
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> CWD Standesamt
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> PASV
2013-06-18 21:08:22> 227 Entering Passive Mode (31,18,112,36,221,85)
2013-06-18 21:08:22> TYPE I
2013-06-18 21:08:22> 200 Type set to I.
2013-06-18 21:08:22> SIZE DSC01654.JPG
2013-06-18 21:08:22> 213 2007180
2013-06-18 21:08:22> RETR DSC01654.JPG
2013-06-18 21:08:22> 150 Opening BINARY mode data connection for
'DSC01654.JPG' (2007180 bytes).
2013-06-18 21:08:23> 226 Transfer complete.

The log looks like this for each file. This means that the command CWD is
done three times for each file. This is properly not good for performace,
so how do I avoid this?

Another problem is that after downloading about 50 files then it fails with
an error:

2013-06-18 21:08:21> 220 FRITZ!Box6360Cable(kdg) FTP server ready.
2013-06-18 21:08:21> USER userX
2013-06-18 21:08:21> 331 Password required for userX.
2013-06-18 21:08:21> PASS XXX
2013-06-18 21:08:22> 230 User userX logged in.
2013-06-18 21:08:22> PWD
2013-06-18 21:08:22> 257 "/" is current directory.
2013-06-18 21:08:22> CWD WD-Elements1042-02
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> CWD Pictures
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> CWD Standesamt
2013-06-18 21:08:22> 250 CWD command successful.
2013-06-18 21:08:22> PASV
2013-06-18 21:08:22> 227 Entering Passive Mode (31,18,112,36,221,85)
2013-06-18 21:08:22> TYPE I
2013-06-18 21:08:22> 200 Type set to I.
2013-06-18 21:08:22> SIZE DSC01654.JPG
2013-06-18 21:08:22> 213 2007180
2013-06-18 21:08:22> RETR DSC01654.JPG
2013-06-18 21:08:22> 150 Opening BINARY mode data connection for
'DSC01654.JPG' (2007180 bytes).
2013-06-18 21:08:23> 226 Transfer complete.

2013-06-18 21:08:23> response reading failed 2013-06-18 21:08:23> Error
2013-06-18 21:08:23> couldn't connect to host 2013-06-18 21:08:23> Error
2013-06-18 21:08:23> couldn't connect to host 2013-06-18 21:08:23> Error

Any idea why this is happening or how I can avoid this? Can I retry the
command that fails somehow?

Best Regards,
Anders Havn

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-06-19