On Tue, 20 Apr 2004, Jesse Noller wrote:

> We have a 3 GB file, we want to upload via a form POST, for instance:
>
> curl -i -F path=/home/foo/tmp/3GB.dat -F
> uploadedFile=@/home/foo/tmp/3GB.dat
>
> The process runs for awhile after execution, and then linux throws:
> Killed

The formpost code in libcurl will create the whole post in memory before
attempting to post it. This is of course rendering the -F feature useless if
you want to post something that is bigger than your amount of available
memory.

(Yes I want that fixed, but my work load is high already.)

> We're assuming that the URL encoding required for a form POST

This is a multipart formpost, it doesn't need any URL encoding. It just needs
RFC1867-formatting.

> The question is this - does curl itself have a method to url encode on disk,
> and then pass a pre-encoded file to the server, and secondarily, how do we
> avoid this with libcurl?

You can avoid this by producing a full pre-formatted multipart chunk on disk
and then using the "standard" POST function with (lib)curl as that can be done
from a file/using the plain read callback functionality.

There is however no support in libcurl to produce such a pre-formatted
multipart post file, you would need to read up on rfc1867 details and do it
yourself. But personally, I consider the fix to make it not produce the full
post in memory before sending to be the most sensible action to circumvent
this problem.

> In python/pycurl's case, it would be a matter of importing httplib and url
> encoding the file, but how do you pass libcurl the handle to say "this file
> is already encoded?"

That's easy! libcurl doesn't encode anything you tell it to post! :-)

-- 
     Daniel Stenberg -- http://curl.haxx.se -- http://daniel.haxx.se
      Dedicated custom curl help for hire: http://haxx.se/curl.html