cURL / Mailing Lists / curl-library / Single Mail


How to achieve high bandwidth utilization when downloading web-pages with libcurl?

From: Alex <>
Date: Sat, 17 Dec 2011 23:57:19 +0400

I'm new to libcurl and network programming, so I think there are some techniques of which I'm not aware. I did some searching, though, and didn't find anything useful so far, so any advices are much appreciated.
So, I have a program written in C++ with libcurl, that basically crawls the web, downloading the pages. I also have a pretty good internet connection, which easily gives me 4-5 MB/s download speed. However, I can't squeeze any decent network utilization out of my program. Downloading a single page at a time, I rarely see speeds in excsess of 300-500 KB/s. The only idea I have so far is to download multiple pages from different servers simultaneously from multiple threads. I've implemented that, but network utilization graph still doesn't look good: it has rare short spikes up to 1.5-2 MB/s with 0 in between the spikes. I've tried different number of threads, from 5 to 15, but didn't see any visible changes whatsoever.
What I would like to see is network bandwidth utilization of >2 MB/s on average, not only in short spikes. How can I achieve that? I know my connection can handle that, and I don't need for simultaneously downloading pages to be from the same site (server).

Thanks in advance!
List admin:
Received on 2011-12-17