cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: pyCURL/Libcurl problem with large files

From: Daniel Stenberg <daniel-curl_at_haxx.se>
Date: Mon, 22 Mar 2004 22:24:33 +0100 (CET)

On Mon, 22 Mar 2004, Jesse Noller wrote:

(please don't reply to an existing thread when you want to post an entirely
new thread using a new subject, it messes up threading in mail clients and
the web archive - post a fresh new mail instead)

> I am hoping that someone else working with pycurl catches this - I just
> upgraded to 7.11.1 release (for both curl and pycurl) and I am trying to
> perform file PUTs against a server with 2GB files.
>
> The problem I am having is that with the normal INFILESIZE setopt, the size
> of the file I am feeding libcurl via pyCURL has to be converted to an INT -
> files up to 1GB work fine with:

CURLOPT_INFILESIZE in the libcurl level takes a 'long', which is equal to
'int' on 32bit systems.

> fs = os.path.getsize('file')
> c.setopt(c.INFILESIZE, int(fs))
>
> FS is always a long with these bigger files, so I convert it to an INT
> before feeding it to libcurl/pycurl.

How big is a 'long' and 'int' in python? Don't they match the host's native
longs and ints?

> Now, the problem comes up when doing 2Gb files, the number spit out by
> getsize is a long, and can not be converted to an INT (a known python
> limitation) - as the number is already a long, I skip conversion, and try:
>
> if fs > 524288000: #Set the limit to 500 Megs just to be safe.
> c.setopt(c.INFILESIZE_LARGE, long(fs))

CURLOPT_INFILESIZE_LARGE wants the size passed in as a curl_off_t variable,
not just a long.

> This results in an "Error 55, ''" from Libcurl - I grepped the C code, and
> found:
>
> hugehelp.c:" 55 Failed sending network data\n"

55 is CURLE_SEND_ERROR, yes. 'man libcurl-errors' is a nice way to figure out
libcurl error codes.

> I'm in the dark here (still learning) - I am assuming it's a problem with
> pycurl interfacing with libcurl, but I don't know how to prove that/fix it.

I think so too, and I don't have any clues either! ;-)

-- 
    Daniel Stenberg -- http://curl.haxx.se/ -- http://daniel.haxx.se/
   [[ Do not send mails to this email address. They won't reach me. ]]
Received on 2004-03-22