cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: libcurl 7.9.8: Bottleneck on multhithread HTTP fetches?

From: Daniel Stenberg <daniel_at_haxx.se>
Date: Wed, 15 Oct 2003 08:52:34 +0200 (CEST)

On Tue, 14 Oct 2003, Andrew Fuller wrote:

> By the way, if it's any use, this time the longest path (timewise) according
> to Rational Quantify was :
>
> curl_easy_perform->Curl_perform->Curl_connect->SetupConnection->ConnectPlease->Curl_connecthost->socketerror->getsockopt->EnterCriticalSection
>
> Next longest was the same to above except ended :
>
> ...->getsockopt->NtRemoveIoCompletion

Thanks for the provided info. Here's my take on analyzing it:

The VERBOSE output log (from your previous mail) is a mess to read and really
doesn't help much since all the different threads' output mix in a seemingly
random way. You would need to prefix all output lines with a thread id/number
or something for it to start becoming useful. (Thus you'd need to use
DEBUGFUNCTION.)

How many threads do you need to get this problem? You used 80 here, but what's
the minimum amount? What's your network connection to the test server?

We could try to learn from history. See what Chicco wrote back in 2002:
http://curl.haxx.se/mail/lib-2002-02/0057.html

Bjorn Reese filled in some more info on why libcurl does this:
http://curl.haxx.se/mail/lib-2002-02/0058.html

The relevant question here and now is of course: can we modify this behavior
on Windows to make it run better? Chicco said that he didn't need to invoke
'socketerror' if connect() returns 0. Perhaps this is always true on windows?
This page on microsoft.com seems to claim this:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/connect_2.asp

I don't use windows enough myself to try this out.

-- 
 Daniel Stenberg -- curl: been grokking URLs since 1998
-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
Received on 2003-10-15