curl-and-python
RE: Pycurl and the curl_multi_socket_action API
From: Daniel Stenberg <daniel_at_haxx.se>
Date: Fri, 6 Apr 2012 14:04:40 +0200 (CEST)
Date: Fri, 6 Apr 2012 14:04:40 +0200 (CEST)
On Fri, 6 Apr 2012, Utsav Sabharwal wrote:
> But I am not sure if pycurl multi gives similar non blocking feature as in
> libcurl. I prefer using curl easy in threaded mode. The following example
> creates 100 parent threads and I was easily able to crawl around 5M urls per
> day on Amazon EC2
100 threads is probably fine. I would say the problem becomes slightly
different when you go up to 1000 or 10000 connections...
-- / daniel.haxx.se
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2012-04-06