cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: libcurl and Perl's WWW::Curl::Easy slower than LWP on small HTTP POSTs

From: Martin J. Evans <martin.evans_at_easysoft.com>
Date: Thu, 15 Sep 2011 11:32:36 +0100

On 14/09/11 19:25, Daniel Stenberg wrote:
> On Wed, 14 Sep 2011, Martin J. Evans wrote:
>
>> I'm not sure I can send the form in the same way from WWW::Curl::Easy. It only has:
>>
>> use WWW::Curl::Form;
>> my $curlf = WWW::Curl::Form->new;
>> $curlf->formaddfile($filename, 'attachment', "multipart/form-data");
>> $curlf->formadd("FIELDNAME", "VALUE");
>>
>> $curl->setopt(CURLOPT_HTTPPOST, $curlf);
>>
>> See - http://search.cpan.org/~szbalint/WWW-Curl-4.15/lib/WWW/Curl.pm
>>
>> I don't see a way to control how the form data is added
>
> That is because formadd and CURLOPT_HTTPPOST imply multipart formpost.
>
> If you want plain old application/x-www-form-urlencoded POST then you use CURLOPT_POSTFIELDS and pass in the string you want sent, pre-formatted by you.
>

Having changed my code to do a URL encoded POST with CURLOPT_POST and CURLOPT_POSTFIELDS my results are now:

Benchmark: timing 1000 iterations of curl, lwp...
       curl: 20 wallclock secs ( 0.34 usr + 0.13 sys = 0.47 CPU) @ 2127.66/s (n=1000)
        lwp: 26 wallclock secs ( 4.41 usr + 0.58 sys = 4.99 CPU) @ 200.40/s (n=1000)

This is largely because there is only one write to the socket now.

Just in case it is useful to anyone else reading this, my test example ended up like something like this:

my $curl_hdr = '';
my $curl_hdr_fh = undef;
open $curl_hdr_fh, ">", \$curl_hdr;
my $curlp = WWW::Curl::Easy->new();
$curlp->setopt(CURLOPT_NOPROGRESS, 1); # shut off the built-in progress meter
$curlp->setopt(CURLOPT_HEADER, 0); # don't include hdr in body
$curlp->setopt(CURLOPT_ENCODING, 'gzip');
$curlp->setopt(CURLOPT_PROXY, ''); # no proxy
open $curl_hdr_fh, "+>", \$curl_hdr;
$curlp->setopt(CURLOPT_WRITEHEADER, $curl_hdr_fh);
$curlp->setopt(CURLOPT_COOKIEFILE, 'cookiescurl');
$curlp->setopt(CURLOPT_COOKIEJAR, 'cookiescurl');

sub postcurl3 {
     seek $curl_hdr_fh, 0, 0;
     truncate $curl_hdr_fh, 0;

     $curlp->setopt(CURLOPT_POST, 1);
     my $url = 'http://xxx.yyy.zzz:82/v1/user';
     $curlp->setopt(CURLOPT_URL, $url);

     my %args = (method => 'send_keep_alive');
     my $rp='';
     while (my ($key, $value) = each %args) {
         $rp .= uri_escape_utf8($key) . '=' . uri_escape_utf8($value) . '&'
     }
     chop $rp;
     $curlp->setopt(CURLOPT_POSTFIELDS, $rp);

     my @hdrs = ('Expect:',
                 'Content-Type: application/x-www-form-urlencoded;charset=utf-8');
     $curlp->setopt(CURLOPT_HTTPHEADER, \@hdrs);

     my $response_body;
     $curlp->setopt(CURLOPT_WRITEDATA,\$response_body);

     my $retcode = $curlp->perform;

     my $response_code = $curlp->getinfo(CURLINFO_HTTP_CODE);
     if ($retcode != 0 || ($response_code != 200 && $response_code != 304)) {
         die "Failed to get url - $response_code, ", $curlp->errbuf;
     }
     #print $response_body, "\n";
}

Many thanks to Daniel for spotting that and the other useful input I've received.

Martin

-- 
Martin J. Evans
Easysoft Limited
http://www.easysoft.com
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2011-09-15