cURL / Mailing Lists / curl-library / Single Mail


Re: maximizing performance when issuing a large number of GET/PUT requests to a single server

From: Alex Loukissas <>
Date: Thu, 11 Aug 2011 11:29:57 -0700

On Thu, Aug 11, 2011 at 10:18 AM, Daniel Stenberg <> wrote:
> On Thu, 11 Aug 2011, Alex Loukissas wrote:
>>> In many situations you won't gain any performance by doing parallell
>>> uploads to a single server, and you'll get a simpler implementation by doing
>>> serial uploads so I'd recommend that. At least for a first shot.
>>> Just make sure you re-use the same CURL handle _without_ doing
>>> cleanup/init between each individual transfer.
>> Seems like a good plan of attack to use the multi interface as a first
>> shot.
> Well, I meant to use the easy interface single-threaded as a first attempt,
> but by all means use the multi interface!

I did try reusing the same handle, which (as expected) had a
tremendous positive effect in the number of TIME_WAIT sockets and the
initial tests on performance show some improvement (although not
drastic). Is there a benefit over what I'm doing now (i.e. looping
through the URIs and issuing a curl_easy_perform) versus having a
number of handles in a curl_multi_handle? From what I understand, the
multi interface is doing the same thing (i.e. serially doing a
curl_easy_perform for each handle).

>> Another option would be HTTP pipelining but from what I see it's not in
>> its most mature stage in libcurl. The reason I'm considering pipelining is
>> that the files I'm up/downloading are small size (few to couple hundred KB)
>> and I'm guessing that squeezing many HTTP requests in a single round trip
>> I'll gain some performance. Any thoughts on that?
> It is not as mature, no, but perhaps more importantly it is only enabled for
> downloading. If you use the multi interface, you can easily just enable
> pipelining and see how it works/performs.

Great thanks. Reads are more important than writes, so that's OK. I'll
give it a shot.

> --
>  /
> -------------------------------------------------------------------
> List admin:
> Etiquette:

List admin:
Received on 2011-08-11