Bugs item #3076808, was opened at 2010-09-27 14:08
Message generated for change (Comment added) made by dfandrich
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3076808&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: http
Group: wrong behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Chris Smowton ()
Assigned to: Daniel Stenberg (bagder)
Summary: Requests fail silently following a 416 error

Initial Comment:
curl 7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15

The problem: I'm using libcurl to conduct a group of transfers with Range: headers. These might result in HTTP error 416: Requested Range Not Satisfiable.

The author of lib/http.c's comments suggest s/he thought that 416 responses cannot have a body, and carry a Content-Length which describes the length of the real document. However, this isn't the case. The RFC says nothing about 416 under Content-Length, but does state that Content-Range may be used for this purpose. In a few places it mentions response codes which preclude a body; it never includes 416 in such a list.

Real servers in fact *do* include a body, and don't treat Content-Length in an unusual way. Specifically:
* Apache/2.3.8 does not give a Content-Length at all, and includes an error document with chunked encoding.
* Microsoft IIS 7.5 gives a Content-Length describing the length of the error document, and uses Content-Range to indicate the valid ranges which could have been requested as suggested by the RFC
* CherryPy 3.1.2 (a Python WGSI server) uses Content-Length and an error document, but doesn't include a Content-Range header.

Full conversations with these servers are attached.

Ordinarily this is just annoying, but it becomes a serious problem when using libcurl's multi-handles and a server which may delay appreciably between sending its headers and its error document. As far as I can tell from packet traces, what's happening is:

* I add two easy-handles to a multi-handle which make requests of the same server.
* libcurl connects and issues the first request
* The server sends the headers for the 416 reply
* libcurl ignores the Content-Length header, assuming it to be irrelevant and that 416s don't have bodies
* libcurl re-uses the socket to send the second easy-handle's request
* The server sends the error document for the first 416, followed by the headers and document for the second request
* libcurl becomes confused as it gets HTML where it was expecting an HTTP reply

I'm not sure exactly what happens at the end, as I never see the second request complete. However, packet traces show that the second request is definitely sent before the first error document has arrived, and the code comments indicating false expectations about 416 replies suggest this is the cause. I'll try to track it down more carefully soon.

----------------------------------------------------------------------

>Comment By: Dan Fandrich (dfandrich)
Date: 2010-09-30 16:28

Message:
I've uploaded the patch I used.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2010-09-30 14:49

Message:
Dan, Where did you insert that delay to trigger the problem? I might give
it a go in my end as well!

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2010-09-27 23:50

Message:
I tried putting a delay in the http testsuite server and curl ends up
hanging. I haven't tried getting to the bottom of it yet.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2010-09-27 23:42

Message:
A connection that's used for a request is left untouched exactly as it is
until it re-used again. There is no draining done and it actually can't be
any such.

----------------------------------------------------------------------

Comment By: Chris Smowton ()
Date: 2010-09-27 17:20

Message:
So I was surprised to find that I wasn't having the problem more often --
after all, I'm frequently getting 416 errors, and cURL often reuses
connections without trouble.

My theory therefore is that it's the delay that matters. When cURL goes to
reuse an HTTP connection, does it make any effort to drain the socket
before issuing the second request, in case there's any trailing bytes from
the previous one? If so, I'd guess that code will be sucking up the
errordoc so the second request works properly.

That's the goal of my test case simple server -- it deliberately inserts a
large delay between the headers and the body such that cURL will have a
chance to send its request before the errordoc arrives.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2010-09-27 16:46

Message:
I've checked in test case 1204 into git in an attempt to reproduce this
issue, but the test case succeeds! Can you see the difference between the
test case and the conditions under which you're seeing this problem?

----------------------------------------------------------------------

Comment By: Chris Smowton ()
Date: 2010-09-27 15:07

Message:
I've developed a simple test case -- here's a little C program which acts
as a dead dumb server that pauses between sending its headers and its
errordoc for show. To see cURL go crazy, spin it up then do curl --trace -
http://localhost:9000/ http://localhost:9000/

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2010-09-27 14:23

Message:
Thanks for the detailed and well expressed report.

I can only explain this weirdness as a confused mixup of Content-Range and
Content-Length somehow (done by me). This change was introduced in commit
a5e22867c7ec back in October 2003 but as you say there's nothing in the
HTTP spec that backs this up.

We need to clear this up...

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3076808&group_id=976
Received on 2010-10-01