On Thu, 11 Dec 2008, scott_py_at_itsanadventure.com wrote:

> With a slightly-modified retriever-multi.py set to 20 concurrent connections
> retrieving the same 100KB file, it generates slightly over 5 MEGABITS per
> second, and the CPU is maxed out.

That sounds terribly wrong.

> Is PyCurl the right tool for this job?

I think it is.

> Is PyCurl's concurrent sessions the right way to do this, or might it work
> better with multiple threads each having a single session?

I'm not sure exactly how the performance will differ between using the multi
interface with 20 transfers and 20 threads doing one transfer each, but I
can't imagine it being noticably difference with that low number of transfers.

> If you want more info, let me know, but the main idea of the program is too
> retrieve MANY files as fast as possible, over and over again.

curl-loader is a tool that does similar stuff, using the curl multi interface
(although not the python binding) and I know they've reached very high speed.

I think your problem lies either in your implementation/usage of pycurl, or
possibly a bug in pycurl's interface against libcurl.

-- 
  / daniel.haxx.se
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python