I had problems with blocking dns requests in my crawler and I switched
to c-ares. I have to say the performance gain is impresive.

On the other hand I'm trying to run some experiments using

CURLOPT_LOW_SPEED_LIMIT 512
CURLOPT_LOW_SPEED_TIME 180

In combination with
CURLOPT_CONNECTTIMEOUT 60

But I'm not observing any timeouts (using curl_multi_info_read). As
the handles have been for some minutes stuck with the same url, with
no activity checked by tcpdump.

My questions are:
1 - Any ideas what to try next?
2 - Is there a way to interrupt a transfer, I don't see any call in
the easy interface for this. Since I'm accounting the time and
transfer I could manually kill those stuck connections.

Thanks.

Pedro.

PD: debug output from my program showing the problem:
handle 0: ROBOTS: 0 h 4 m: 0 k: http://blog.omnigroup.com/robots.txt
handle 1: ROBOTS: 0 h 4 m: 0 k: http://zoetrope.stripgenerator.com/robots.txt
handle 2: ROBOTS: 0 h 4 m: 0 k: http://zoomit.wordpress.com/robots.txt
handle 3: ROBOTS: 0 h 2 m: 0 k: http://zangre.blogspot.com/robots.txt
handle 4: ROBOTS: 0 h 4 m: 0 k: http://zozo2k3.blogspot.com/robots.txt
handle 5: ROBOTS: 0 h 4 m: 0 k: http://zumodevidrio.blogspot.com/robots.txt
handle 6: ROBOTS: 0 h 4 m: 0 k:
http://losrojosnousabansombrero.blogspot.com/robots.txt
handle 7: ROBOTS: 0 h 4 m: 0 k: http://zhaojunzheblog.blog.sohu.com/robots.txt
handle 8: ROBOTS: 0 h 4 m: 0 k: http://mariettamorales.blogspot.com/robots.txt
handle 9: ROBOTS: 0 h 4 m: 0 k: http://mediciland.blogspot.com/robots.txt
handle 10: ROBOTS: 0 h 4 m: 0 k: http://zoetrope101.spaces.live.com/robots.txt
handle 11: ROBOTS: 0 h 4 m: 0 k: http://zettwoch.blogspot.com/robots.txt
handle 12: ROBOTS: 0 h 4 m: 0 k: http://zvonnews.sourceforge.net/robots.txt
handle 13: ROBOTS: 0 h 4 m: 0 k: http://zoetrope101.blogspot.com/robots.txt
handle 14: ROBOTS: 0 h 4 m: 0 k: http://zerozofoke.blogspot.com/robots.txt
handle 15: ROBOTS: 0 h 4 m: 0 k: http://zoetrope.tumblr.com/robots.txt
handle 16: ROBOTS: 0 h 4 m: 0 k: http://paulauster.blogspot.com/robots.txt
handle 17: ROBOTS: 0 h 4 m: 0 k: http://zonda.wordpress.com/robots.txt
handle 18: ROBOTS: 0 h 4 m: 0 k: http://zoemini.blogspot.com/robots.txt
handle 19: ROBOTS: 0 h 4 m: 0 k: http://romavictrix.blogspot.com/robots.txt
down: 6.46 MiB 0.00 KB/s
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-12-27