Bugs item #3208760, was opened at 2011-03-13 12:05
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3208760&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: wrong behaviour
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Chris Smowton ()
Assigned to: Daniel Stenberg (bagder)
Summary: Bad HTTP POST retry with READFUNCTION or READDATA set

Initial Comment:
cURL version: curl 7.21.0 (x86_64-pc-linux-gnu) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18

The scenario:

1. cURL sends an HTTP POST to a server
2. Server realised on reading that (e.g.) the backend has hung up, and sends a FIN
3. cURL notices the socket is dead after sending, but before receiving anything
4. cURL attempts a retransmit from Curl_retry_request
4a. If the socket we were using was fresh (i.e., not re-used), at this point we die with error 52 / server sent empty reply
5. cURL neglects to try to rewind the input file
6. On the retry, cURL calls fread or a custom READFUNCTION and immediately gets EOF
7. cURL sends an empty request body which doesn't match the Content-Length header
8. Server waits a while for the promised POST body
9. Server gives up; silently hangs up the connection
10. cURL decides that since it transmitted a POST but received nothing... it had better retry!
11. Go to step 4

Note that this doesn't happen with the cURL command-line program, as it seems to use POSTFIELDS instead of READDATA or READFUNCTION.

I attach an example libcurl client program which exhibits the problem as follows:
1. It makes a small POST request against the server localhost:9000. This is to leave a re-usable socket around.
2. It makes another small POST against the same server. The socket gets re-used.

If invoked without arguments it does this using a small file on disk and READDATA; if invoked with argument 'func' it will use READFUNCTION and a fake fread function that prints what it's doing.

I also attach a dummy server that exhibits the problem. It behaves as follows:
1. Read the first request
2. Send a dummy response
3. Read a second request
4. Hang up the socket
5. Accept a second socket
6. Read the request

If client and server are used together, we see the following:
1. cURL connects
2. cURL sends POST
3. Server responds
4. cURL reuses connection for second POST
5. Server hangs up
6. cURL notes that the connection died and goes for a retry
7. cURL forms a new connection
8. Server accepts
9. Server reads
10. We note that the request body does not appear in the retried request.
11. Server hangs up
12. cURL fails with server-sent-empty-document.

Finally I attach

I'd suggest that the answer here is that we retry an HTTP POST for which we have called fread and don't try to rewind. If it's a FILE* or we have a SEEKFUNCTION this could work; otherwise we should probably fail with cant-rewind. The function http_perhaps_rewind looks like the right thing, but isn't used here.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-03-20 23:33

Message:
Thanks for the report, this problem is now fixed in the git repository.

To try it out, you either checkout/update your git clone:
http://curl.haxx.se/source.html

or you try tomorrow's daily snapshot: http://curl.haxx.se/snapshots/

----------------------------------------------------------------------

Comment By: Chris Smowton ()
Date: 2011-03-16 01:40

Message:
I've run a good few tests now and the bug hasn't manifested in this way
since I applied the patch. Other failures did occur, but they turned out to
be caused by another, different bug -- I've filed a separate ticket for
that (#3214223). It looks like I'm exploring all the dark corners of
libcurl that don't get a lot of use :)

----------------------------------------------------------------------

Comment By: Chris Smowton ()
Date: 2011-03-13 23:57

Message:
The patch looks like the right idea, and certainly fixes my test case; in
my actual application the problem would only manifest every hundred runs or
so, so it'll take a while to see if there's another corner case here.

I'll let you know if I find anything :)

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-03-13 23:13

Message:
Thanks for your detailed report and analysis of the problem. I think you're
perfectly spot-on and I can see how this is a problem.

I haven't tested this yet, but I figure something simple such as the
3208760.patch I just uploaded here could work. (http_perhaps_rewind is a
bit too focused on auth-related problems)

Is it too simple or does it work for you?

----------------------------------------------------------------------

Comment By: Chris Smowton ()
Date: 2011-03-13 12:10

Message:
BTW, my workaround at the moment: assume that a further fread() after EOF
means that libcurl is retrying the connection and rewind myself.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3208760&group_id=976
Received on 2011-03-20