cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: parallel transfer techniques

From: Richard Atterer <richard_at_2009.atterer.net>
Date: Tue, 16 Jun 2009 22:01:27 +0200

Hi Mohun,

On Tue, Jun 16, 2009 at 02:30:02PM -0400, Mohun Biswas wrote:
> My current call to select uses only the "readfds" parameter, since I do
> only reads of the incoming data. Since the uploads I want to do via the
> multi interface are all writes, it seems to me that I can easily
> partition my fd sets. I.e. I should be able to take the fdset returned by
> curl_multi_fdset(), pass that as "writefds" to select, and only call
> curl_multi_perform() if it has any bits set (assuming it gets FD_ZERO'ed
> before each select call). Does this make sense?

Yes and no - you should keep in mind that HTTP always involves sending a
request and interpreting the response; bytes flow in both directions. So I
would expect the fds that you get by libcurl to be both in the read and
write sets. If you ignore libcurl's read_fd_set, the HTTP upload will
probably hang after all data has been sent.

> Also, is there an idiom for checking whether an fd_set has any bits set,
> other than looping up to maxfd and using FD_SET on each? No problem
> doing it but it seems a bit inelegant. I was looking for a macro like
> FD_ISEMPTY but haven't found one.

There is no such thing AFAIK. The whole select() API has been criticized
due to aspects like this, but that's how it was conceived in BSD... There
are other APIs which fix the scalability problems - under Linux it's
epoll().

I seem to remember that someone once worked on scalability patches for
libcurl which used something other than select(). Was this the case? Did
anything useful come out of it??

(*Some web searches later*) It seems that you would have to create any
epoll() glue code yourself, and then call curl_multi_socket_action() for
any activity on the sockets.
<http://curl.haxx.se/libcurl/c/curl_multi_socket_action.html>

Anyway, unless you have >1000 connections or so, I think using select()
should be OK.

Cheers,

  Richard

-- 
  __   _
  |_) /|  Richard Atterer     |  GnuPG key: 888354F7
  | \/¯|  http://atterer.net  |  08A9 7B7D 3D13 3EF2 3D25  D157 79E6 F6DC 8883 54F7
  ¯ '` ¯
Received on 2009-06-16