On 7/10/2015 2:06 AM, Rajalakshmi Iyer wrote:
> Thanks for getting back. Here is some background to the application in
> its current state -
>
> The application has X worker threads that receives 1000s of concurrent
> requests per second.
>
> As part of handling these incoming requests, the X worker threads need
> to make a callout to an external HTTP server. This callout typically
> takes up to 100 ms.
>
> Currently, the X worker threads have implemented this by using a
> curl_easy_handle per thread, i.e. a connection per thread that is
> continuously making callouts and handling the responses in the
> callback function CURLOPT_WRITEFUNCTION.
>
> However, this also means that the application is not doing much except
> waiting on external HTTP server. The CPU is hardly utilised.
>
> In order to overcome this, I am planning to use the curl multi
> interface. So each of the X curl easy handles from X threads will be
> added to the curl multi handle.
>
> Does that sound a reasonable thing to do?

Yeah it seems beneficial assuming you don't need to block each thread
waiting for the server reply (which is the way curl_easy_perform works
as you've seen).

Generally there is something in libcurl I'm unclear about (I've asked on
the list before but didn't get a reply addressing it) which is it seems
that it's acceptable behavior for curl_multi_perform to always be
performing and not return for extended periods of time and I thought in
that case you can't call multi_add_handle from another thread during
that time. Whether that's even true (my reading of the code at the time
was yes) and you will encounter that in your situation is for Daniel or
someone more knowledgeable to say.

Assuming so you have to account for that in your design because as you
add easy handles to the multi handle from different threads you could
have a lot of locking if thread 1 holds the lock because it's is in the
middle of using the multi handle (ie you called multi_perform) and
you've got thread {2,9} locked waiting to add easy handles but can't
because the multi is in use for extended periods by thread 1. It seems
to me the way to work this is create a global queue list with a lock and
each thread holds the lock as it adds to the queue. Then back in thread
1 in your multi loop you hold the lock only while you move from the
queue to multi. Note in this case if multi_perform is out for extended
periods of time nothing is added for extended periods of time.

In other words
multi_perform
lock
add from queue
unlock
multi_wait
multi_info_read

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-07-10