Re: Project hiper - High Performance libcurl
Date: Wed, 9 Nov 2005 12:58:36 +0100 (CET)
On Tue, 8 Nov 2005, Cory Nelson wrote:
> The current multi interface of having the user wait for events is not an
> efficient design when it comes to IOCP.
> IOCP uses threading to get full scalabilty on SMP systems.
But does it really scale that well on the ordinary plain non-SMP system? Is
your normal Windows box up to serving 10,000 threads? (if you have the RAM for
it). And I would expect that using a thread for each connection will be more
memory consuming that using a single thread for all transfers (due to stack
and thread context overhead).
> It could certainly be done from one thread, but then you would be losing
> many of the benefits. This is aimed at being high performance so libcurl
> (imho) should definately add the minimal threading awareness for IOCP, if
> only for this hiper API.
> The API I gave was complete. You open a hiper handle and push your easy
> handles into it. The hiper handle would open one or more threads in the
> background and callback when something significant happens to an easy
> handle. Simple as that. If you wanted to wait until all the easy handles
> are finished, you can call the wait function.
(I see a problem to merge a Windows-optimzed concept with the currently
planned event-optimized concept...)
If that is what you want (using a thread for each transfer), won't it suffice
to simply start a new thread and fire off a separate curl_easy_perform() in
there? In what way would this suggested interface be an improvement to that?
HTTP pipelining might be hard to add nicely for such a use case.
> If hiper is meant to abstract away all the stuff needed for high performance
> http, it should also be in charge of threading efficiently as needed.
I want libcurl to abstract away all protocol and transfer related matters. I
want it to know as little as possible about event systems and threading
> That's where the confusion is :) My proposal moves the blocking into
> libcurl where it could be efficiently managed and out of the users reach.
Now I believe I understand your proposal a lot better.
I'll have to give this some more thoughts...
-- Commercial curl and libcurl Technical Support: http://haxx.se/curl.htmlReceived on 2005-11-09