Hi,

I would like to download a bunch of files. Therefore I use curl_multi which is faster than downloading all files separately.

I measured the elapsed time during downloading 100 files (each about 5kB), using curl_multi I halved the time. Unfortunately I still need about 65 seconds to load all 100 files.

I thought the use of several threads can reduce the time. So I created 10 threads and distributed the 100 files to all the threads equally, now I only need around 50 seconds. But isn't this still too long? I though I can gain more time. Why is it much faster to download only 10 files in one single thread than downloading 100 files in 10 threads?

Is there a way to increase the speed?

Thank you very much!

This is the code which downloads the files:

// urls is a vector<string> containing all urls of the files
// the corresponding filenames is localpaths

        CURL *handles[urls.size()];
        CURLM *multi_handle;
        int still_running = -1;
        CURLMsg *msg;
        int msgs_left;
        
        multi_handle = curl_multi_init();
        curl_multi_setopt(multi_handle, CURLMOPT_PIPELINING, 1);
        
        FILE* fp [urls.size()];
        for (int i = 0; i < urls.size(); i++){
                fp[i] = fopen(Converter().atocstring(&(localpaths[i])),"wb");
                handles[i] = curl_easy_init();
                curl_easy_setopt(handles[i], CURLOPT_URL, Converter().atocstring(&(urls[i])));
                curl_easy_setopt(handles[i], CURLOPT_WRITEFUNCTION, write_data);
                curl_easy_setopt(handles[i], CURLOPT_WRITEDATA, fp[i]);
                curl_multi_add_handle(multi_handle, handles[i]);
        }
        
        while (still_running){
                curl_multi_perform(multi_handle, &still_running);
                usleep(100000);
        }
        
        while ((msg = curl_multi_info_read(multi_handle, &msgs_left))){
                if (msg->msg == CURLMSG_DONE){
                        if (msg->data.result != 0){
                                int handle_id = 0;
                                for (handle_id = 0; handle_id < urls.size(); handle_id++)
                                        if(msg->easy_handle == handles[handle_id])
                                                break;

                                found_error = true;
                                if (arg->print_error_msg)
                                        cout << "[CURL ERROR] CODE " << msg->data.result << " for " << urls[handle_id] << endl;
                        }
                }
        }
        
        curl_multi_cleanup(multi_handle);
        for (int i = 0; i < urls.size(); i++){
                curl_easy_cleanup(handles[i]);
                fclose(fp[i]);
        }

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2012-09-23