Bugs item #1631566, was opened at 2007-01-09 15:53
Message generated for change (Comment added) made by sdac
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1631566&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: http
Group: wrong content
Status: Closed
Resolution: Works For Me
Priority: 6
Private: No
Submitted By: sdac (sdac)
Assigned to: Daniel Stenberg (bagder)
Summary: HTTP Pipelining, NULL content

Initial Comment:
I have a multi interface, wherein I have put three easy interfaces each pointing to a separate URL. Two of the URLs are to documents with text/html as mime-type and the third is to a audio/mpeg. When I list the returned mime types I get two "text/html" and one "\0", just null. Switching CURLMOPT_PIPELINING to off without changing anything else in the code, and then running, returns two "text/html" and one "audio/mpeg", just like it is supposed to do.

Now I haven't read anything about how http-pipelining works, I don't even know how the client is supposed to separate each respons and tell what request each response corresponds to without using separate connections. I can't see how this is done. Maybe that explains the behavior I see, so it might not be a bug after all.

I will attach a test program for the case.

Also, I quickly just wrote this program to reproduce this case, and I get a segmentation fault in the end when cleaning up. It's probably because I don't clean up properly? I don't know, I don't get the segmentation fault in my other programs.

libcurl version is 7.16.0.

----------------------------------------------------------------------

>Comment By: sdac (sdac)
Date: 2007-03-07 02:46

Message:
Logged In: YES
user_id=1586503
Originator: YES

Sorry for the delay. Yep, it works now with libcurl 7.16.2 cvs snapshot,
tested on 64-bit Linux 2.6.20, 64-bit FreeBSD 7.0 and 32-bit Win XP. :)

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-02-23 11:08

Message:
Logged In: YES
user_id=1110
Originator: NO

No response, works for me.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-02-20 23:05

Message:
Logged In: YES
user_id=1110
Originator: NO

I tried your test snippet with my current CVS (7.16.2-CVS) and it seems to
work just fine...

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-01-23 10:27

Message:
Logged In: YES
user_id=1110
Originator: NO

Indeed lots of useful info, just that I'm loaded with work and probably
won't be able to address this problem before the 7.16.1 release (due to be
released before end of January).

----------------------------------------------------------------------

Comment By: sdac (sdac)
Date: 2007-01-13 15:16

Message:
Logged In: YES
user_id=1586503
Originator: YES

Did some tracing on the packets today, this is what seems to happen.

>> HEAD / HTTP/1.1 (google)
>> HEAD / HTTP/1.1 (sapht.com)
>> HEAD /coffebreak.mp3 HTTP/1.1 (sapht.com)
<< HTTP/1.1 200 OK (google)
<< HTTP/1.1 200 OK (sapht.com) (response to first request)
>> TCP packet with FIN-flag set (google)
<< Server acknowledges FIN (google)
>> TCP packet with FIN-flag set (sapht.com)
<< Server acknowledges FIN (sapht.com)
* Exits

So our client seems to close the connection after the first response is
received, forgetting the fact that the very same connection was used for
the second request as well.

I switched CURLOPT_NOBODY to 0 again to try with GET instead of HEAD. The
program acted identically, except for one detail:

* Connection #1 to host sapht.com left intact
text/html
* Received problem 5 in the chunky parser
(null)

Don't know what problem 5 is, but it's probably related to the fact that
it gets NULL information because the connection seems to have closed
already.

Tried back and forth with NOBODY, and noticed that this actually happens
randomly. Sometimes, disregarded whether NOBODY is off or on, it works,
sometimes not. So I compared packet dumps.

>> HEAD / HTTP/1.1 (sapht.com)
>> HEAD /coffeebreak.mp3 HTTP/1.1 (sapht.com)
>> HEAD / HTTP/1.1 (google)
<< HTTP/1.1 200 OK (google)
... receive everything

So the obvious difference is that it this time sends the, what I thought
was irrelevant, request to google.com AFTER it has sent the two requests to
sapht.com. The only explanation for this behaviour must be that it sort of
depends on host name lookup, and sapht.com delayed more. Though what's
really strange is how this can result in success of the beforementioned
problem with pipelined requests on a completely different host.

Did yet another shot at dumping, this time was like the first. One request
to google, followed by two requests to sapht, followed by response from
google and one single response from sapht.com. Thereafter two TCP packets
with FIN-flag is sent, one to google and one to sapht.com. Instantly
thereafter sapht.com starts spitting out the mp3-file we requested, loads
of packets each which we respond with a TCP packet with the reset-flag
(RST) set.

I looked through the information recieved (those packets which libcurl is
responding RST to), trying to find something that indicated that it
actually was the mp3-file it was spitting. Found nothing, and thought, what
the hell is this if it's not the mp3 file? So I did a search for the string
"TAG" which should be found in MP3-files. And yep, It was found... But
BEFORE our client sent the FIN-signal. This is how it looks:

</body></html>\n\n\r\n0\r\nHTTP/1.1
followed by the rest of all headers, including content-type and all, and
then the mp3-file itself

I believe the zero (not a nullbyte) indicates the end of a chunk, and
instantly after it begins sending the MP3-file. Later on the response from
google comes, and when that happens libcurl instantly FIN's both
connections, in the middle of receiving the mp3-file. And then comes the
scenario where the rest of the mp3-file packets get RST-responses from
libcurl.

Conclusion: The mp3-file is sent, but libcurl closes the connection before
it's fully received.

Duh, hope this is useful information. =)

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-01-11 13:24

Message:
Logged In: YES
user_id=1110
Originator: NO

pipelining should have no special/different effect in the view of the app,
it should only make libcurl re-use the connection

I intend to investigate this, but it'll take a little while before I get a
chance

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1631566&group_id=976
Received on 2007-03-08