curl-and-python

RE: want to maintain pycurl?

From: Utsav Sabharwal <tashywashysachy_at_live.in>
Date: Tue, 6 Mar 2012 17:42:06 +0530

> I personally have no objections or preferences to where the pycurl project 
> goes or is hosted. It is on sourceforge now, so the info there would need to 
> be updated but other than so... nothing.

> (I personally prefer github to sourceforge any day.)
a

Github sounds great. And I would love to take complete responsibility with the team or alone for "Pycurl binding" 

Recently, I have been implementing pycurl for crawler development around big companies. I found some shortcomings of pycurl over libcurl(the c thing). One of my current project is understanding and resolving limitations of pycurl over libcurl. (This was sad to see that even with same logics and environment pycurl performs slower than libcurl)

I also wrote a Pycurl based crawler to crawl around 5 ~ 10M urls on Amazon EC2 small. I wish to make this opensource and add to docs examples as many new crawler developers wish to use python and pycurl but they lack experience using pycurl to its right efficiency. 

Moreover, I am trying to develop a totally optimize pycurl crawler which can consume complete bandwidth avalaible and be good with CPU. (with an option to control per domain hit)And somewhere at stackoverflow some one posted a benchmarking study: httplib / httlib2 providing better performance than pycurl.We must see what is keeping us behind ;)There is a desperate need to improve the documentation and make pycurl user interactions active again. (People are loosing interest because of lack of activeness from our end.)bottom line: I am ready to put my time and energy   and wish to know who all are in and how and when to start.. . Cheers!!

d_r_a_g_o_s

                                               
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2012-03-06