cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: questions regarding libcurl's multi interface

From: Adam Glick <savantaskew_at_gmail.com>
Date: Thu, 20 Oct 2011 08:34:46 -0700

Good day, and thanks for your help, Daniel. This is AdamG, a collegue of Jon's.

Perhaps some background would help. The network communication is for
what is essentially a web service. Clients will call at various points
in time, at some points frequently, for information from the web
service. The plan is to keep an open connection to the web service and
to pump requests to the web service as quickly as possible.

Pseudo code might look like
  Initialize headers/options, set up multi handle

  While (not shutting down)
  {
    While (more pending requests)
    {
      Create a new easy handle. The headers, URL, and options will all
be the same for all easy handles. Only the payload in the post data
will differ.
      Add the easy handle to a multi handle
    }
    curl_multi_perform
    while (still active and not shutting down)
    {
      determine an appropriate timeout
      Call curl_multi_fdset to get working handles
      Call select() to wait on working handles or timeout
      handle available results with curl_mult_info_read
      Add new pending requests // Same as “While (more pending
requests) loop above
      Curl_multi_perform
    }
    Sleep until more requests come in or shutdown.
  }

The goals are to get all requests satisfied as quickly as possible
while minimizing use of server resources (e.g. number of connections
per client machine).

Things that are still unknown that we could use your help with
1) Assuming say 10 incoming requests and thus 10 simultaneous easy
handles, how many server connections will there be? Said another way,
does each easy handle have its own connection to the server or do they
share connections?
2) Again assuming multiple simultaneous requests and that individual
requests/responses may be large enough to fit into multiple TCP
packets, will the simultaneous requests/replies reassemble properly?

Your advice is greatly appreciated.

-- Adam

-----Original Message-----
On Wed, 19 Oct 2011, Jonathon wrote:

Your top-posting makes it very hard to follow what you're talking
about. I've unfolded some of the mysteries to allow people to actually
track the subjects.
>>> 2) Is there a way to use the multi interface with several easy
>>> handles, but have them all share the same connection?

> For #2, that's a good question.  I'd like to have my client only
> initiate one HTTP connection with the server at all times, but able to
> submit several posts on that connection.  After reading your previous
> comments, I am guessing that I probably only need one easy-handle for
> that and should use the HTTP pipelining functionality you referred to.

You don't strictly need pipelining to just do a series of requests
over the same connection. libcurl will do that by default if you just
keep doing requests to the same host. Pipelining will only make it
slightly more efficient, especially in the cases where you have larger
RTT.

>>> 3) How would I use the multi interface to keep the connections for
>>> my easy handles alive?
>
> For #3, once I've submitted all my posts to the server, I'd like to
> keep that connection open so that I can immediately send more posts in
> the future without setting up the connection again.  Is there a way to
> do this?  (i.e. specify "keep-alive" in the HTTP header).

If you don't kill the multi handle, libcurl will leave the connection
alive for possible future re-use and it'll re-use or re-create the
connection at next request transparently. After one request you just
add the handle again and if that uses the same protocol+host+port then
the existing connection will be used again.

There's not even any need for any keep-alive in a HTTP header since
HTTP 1.1 has persistent connections by default.

--
  / daniel.haxx.se
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2011-10-20