curl-and-python

Re: Downloading straight to memory

From: Nathan E. Moore <nate_at_redtetrahedron.org>
Date: Fri, 04 Sep 2009 23:37:20 -0400

James Webber wrote:
> You just need to change pycurl's writefunction to that of your file-like
> object:
>
> crl = pycurl.Curl()
> output = cStringIO.StringIO()
> crl.setopt(pycurl.WRITEFUNCTION, output.write)
> ...
> crl.perform()
>
> response = output.getvalue()
>
> You could also use a custom write function if you wanted to parse the
> response specially, but you aren't guaranteed to get the response in
> useful pieces--i.e. it could be split up mid-line or mid-word. I just
> store the whole response and read it out afterward, so that I can be
> sure that it's complete.
>
> - James
>

The minimal requirements for pycurl.WRITEFUNCTION are

takes one argument, a string (may contain null bytes)
returns either None or the length of the string argument on success
any other return value indicates an error

so a minimal impl is
def callback(str):
        pass

don't limit yourself to the StringIO stuff. You can feed the output into
an xmlreader, or list.append

Nathan
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2009-09-05