Re: libcurl 7.9.8: Bottleneck on multhithread HTTP fetches?
Date: Tue, 14 Oct 2003 15:00:11 +0200 (CEST)
On Mon, 13 Oct 2003, Andrew Fuller wrote:
> I've been running some loadtesting on my application and I've narrowed a
> high-load bottleneck down to curl. When a large number of threads (>50 for
> example) attempt to HTTP GET a document simultaneously, the CPU on my
> computer maxes out for a few minutes.
I don't understand how curl would behave differently depending on the amount
of threads. If you have 50, 200 or 10, curl will behave the same.
You're also using a version that is many releases old that we have fixed
numerous bugs in since its release. Have you verified if a later version makes
> I've turned on CURLOPT_VERBOSE as per the docs, and I find : Connection #0
> seems to be dead. Could somebody help explain this to me please? My first
> reaction was that my server was being overloaded, but I've eliminated this
If you get a more recent version and you get the same output, then I would be
interested in digging into this. curl 7.9.8 is history to me, and I don't care
very much about possible bugs in it. Sorry, but my time is too limited for
> If it's of any use, I ran the program with the same arguments through
> Rational Quantify. It identified the longest path (in terms of time) as
I find that very strange.
> Of course, I understand that there'll be an inherent delay when 50 curl
> instances attempt to grab a connection simultaneously! However, I'm
> concerned about the high CPU usage and the delays associated with it. It
> seems to me from curl's verbose output (see results.txt) that it only ever
> uses one connection (#0) - is this correct? Is the approach in my code
If all threads do two curl_easy_perform() to the same host, then both calls
will most likely use the same connection, so yes, then the second one would
re-use connection #0.
-- Daniel Stenberg -- curl: been grokking URLs since 1998 ------------------------------------------------------- This SF.net email is sponsored by: SF.net Giveback Program. SourceForge.net hosts over 70,000 Open Source Projects. See the people who have HELPED US provide better services: Click here: http://sourceforge.net/supporters.phpReceived on 2003-10-14