cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: Curl retry without notifying application?

From: Mark Aldred <mark_at_twinstrata.com>
Date: Thu, 27 Aug 2009 10:44:04 -0400

>
>
>
> I am using libcurl 7.19.6.
>>
>> My application uses CURLOPT_READFUNCTION and CURLOPT_WRITEFUNCTION copy
>> the data into and out of curl. My callback functions us structures to track
>> how much data has been copied to maintain sane pointers as the copy
>> progresses. Fairly standard stuff. When curl return an error to our
>> application my code resets the pointers used in the data copy and retires
>> the curl request (for retryable errors). All this works fine.
>>
>> Today I encountered a situation where it appears I issued a PUT request to
>> curl. Curl got the 100-continue from the host, invoked by copy function to
>> get all the data then encountered an error. It seems that curl then retried
>> the operation without returning an error to my application so that I can
>> reset the copy pointers in my structures. The result is that when curl
>> invoked the copy function in my application for the retry, my copy function
>> returned zero because it thought (correctly) that all the data had been
>> copied already. At this point the request hangs.
>>
>> Does curl retry requests without returning an error to the application so
>> that it can prepare for the retry?
>>
>> Here is an excerpt from my log.
>>
>> 0x14f7ae0 2009-Aug-26 23:22:11.905625 CURLINFO_HEADER_OUT 23006848 PUT
>> /some_url HTTP/1.1
>> Accept: */*
>> Authorization: AWS 1VVN4RW1K6ACA2246MR2:jONA2hLltDtR7WSR84/L0xsGH8g=
>> content-length: 131072
>> content-type: application/octet-stream
>> date: Thu, 27 Aug 2009 03:22:11 GMT
>> host: s3.amazonaws.com
>> Expect: 100-continue
>>
>> 0x14f7ae0 2009-Aug-26 23:22:12.912369 CURLINFO_TEXT 23006848 Done waiting
>> for 100-continue
>> 0x14f7ae0 2009-Aug-26 23:22:12.912508 _OutboundDataCopy COPY eh:23006848
>> copy_size:40960 iovec size:131072 copy size:40960 remaining:131072
>> 0x14f7ae0 2009-Aug-26 23:22:12.913001 _OutboundDataCopy COPY eh:23006848
>> copy_size:40960 iovec size:131072 copy size:40960 remaining:90112
>> 0x14f7ae0 2009-Aug-26 23:22:12.913454 _OutboundDataCopy COPY eh:23006848
>> copy_size:40960 iovec size:131072 copy size:40960 remaining:49152
>> 0x14f7ae0 2009-Aug-26 23:22:12.913904 _OutboundDataCopy COPY eh:23006848
>> copy_size:8192 iovec size:131072 copy size:8192 remaining:8192
>> 0x14f7ae0 2009-Aug-26 23:22:12.914067 _OutboundDataCopy DONE bytes
>> processed:131072 size:131072 eh:23006848
>>
>> 0x14f7ae0 2009-Aug-26 23:22:24.665157 CURLINFO_TEXT 23006848 SSL read:
>> error:00000000:lib(0):func(0):reason(0), errno 104
>> 0x14f7ae0 2009-Aug-26 23:22:24.665243 CURLINFO_TEXT 23006848 Connection
>> died, retrying a fresh connect
>> 0x14f7ae0 2009-Aug-26 23:22:24.665283 CURLINFO_TEXT 23006848 Closing
>> connection #0
>> 0x14f7ae0 2009-Aug-26 23:22:24.668640 CURLINFO_TEXT 23006848 Issue another
>> request to this URL: 'https://s3.amazonaws.com:443/some_url<https://s3.amazonaws.com/some_url>'
>>
>> 0x14f7ae0 2009-Aug-26 23:22:24.668719 CURLINFO_TEXT 23006848 Re-using
>> existing connection! (#1) with host s3.amazonaws.com
>> 0x14f7ae0 2009-Aug-26 23:22:24.668757 CURLINFO_TEXT 23006848 Connected to
>> s3.amazonaws.com (72.21.202.97) port 443 (#1)
>>
>> 0x14f7ae0 2009-Aug-26 23:22:24.741157 CURLINFO_HEADER_IN 23006848 HTTP/1.1
>> 100 Continue
>> 0x14f7ae0 2009-Aug-26 23:22:24.753208 _OutboundDataCopy DONE bytes
>> processed:131072 size:131072 eh:23006848
>>
>> One last thing. What is this SSL read error?
>>
>> Thank you for any help you can provide.
>>
>>
> This comment in Curl_retry_request() indicates that curl is doing exactly
what I thought. Sounds like it is retrying a PUT without returning an
error. This behavior is fine for GETs and DELETEs, but not PUTs.

    conn->bits.retry = TRUE; /* mark this as a connection we're about

>
> to retry. Marking it this way should
> prevent i.e HTTP transfers to return
> error just because nothing has been
> transfered! */
>
>
> I'm trying this fix in Curl_http_done(). does this make sense?
>
> if(!premature && /* this check is pointless when DONE is called before
> the
> entire operation is complete */
> !conn->bits.retry &&
> data->req.upload_present != 0 && ///////////////////////// ADDED
> THIS LINE
> ((http->readbytecount +
> data->req.headerbytecount -
> data->req.deductheadercount)) <= 0) {
> /* If this connection isn't simply closed to be retried, AND nothing
> was
> read from the HTTP server (that counts), this can't be right so we
> return an error here */
> failf(data, "Empty reply from server");
> return CURLE_GOT_NOTHING;
> }
>

Sorry for the Top Post

-- 
Mark
Received on 2009-08-27