I thought about this issue a little:-

>We can't blindly assume that the proxy will always first send us the
>headers
>in one chunk and then give us the rest of the reply with a separate pause.
>Even if, since we're doing TCP we can't assume anything in how the stream
>is
>being read from the proxy. Slow-start and the Nagle algorithm could very
>well
>merge lots of data to get read at once, both the initial header and the
>following data stream.

The only thing that I could think of was the way in which I have set it up -
that is:

At termination of the httptunnel proc - the read buffer can be left in
either of two states.
a) Nothing more to be read
b) unread data in buffer <-- see point 2

Since curl is not issuing any tcp request after the initial proxy connect,
the data in the buffer can only have been placed there as part of the
acknowledgement from the HTTP proxy.

In priciple if the proxy is operating according to the RFC state b) should
never occur! - so if we detect a state b) then the proxy is in error and we
need to do something about it.

I therefore concluded that it was safe to clear any unread data from the
read buffer, since it would have been placed there as part of the
acknowledgement.

my routine first checks if there is any unread data - if there is none then
it does nothing - if there is unread data it flushes it. Note that it clears
only as much data as has been placed in the buffer and left by the previous
read.

2.) I see your point that if for example the connect ack/data returned from
the server your connecting to etc.. are sent together then you might get a
state where - after connect there is data in the buffer and this is valid
data.

I don't think that this will occur in practice but it is theoretically
possible.

Perhaps the best thing to do is this:-

a. set up curl to behave by default as before strictly according to RFC
b. have an option that allows you to enable readbufferflushing behaviour if
you so desire. eg an option -bp (stands for broken proxy)

Then if user detects a problem he can try -bp option to see if it works with
his proxy. This is good becuase it puts the user in control of the
behaviour. Setting up such an option would be easy.

or you could set the default behaviour to flush and have a flag for strict
operations eg -sp (strict proxy)

or the option could be -ptcf=ON/OFF (proxy tunnel connect flush)
or -ptc=FLUSH/NOFLUSH (proxy tunnel connect =FLUSH/NOFLUSH)
    and in the build script you could have PTC_DEFAULT=FLUSH/NOFLUSH
    and then it would be up to the person compiling curl what kind of
    deafault behaviour is exibited.

I personaly like the last option as it puts the user in control.

your thoughts ..

>From: Daniel Stenberg <daniel_at_haxx.se>
>To: john lask <johnlask_at_hotmail.com>
>CC: libcurl Mailing list <curl-library_at_lists.sourceforge.net>
>Subject: Re: problems connecting to HTTPS server
>Date: Mon, 5 Nov 2001 12:05:35 +0100 (MET)
>
>On Mon, 5 Nov 2001, john lask wrote:
>
>Splendid work John, thanks a lot! Comments follow...
>
> > 1. changes to files to allow curl to recognise --tlsv1 as an option
>
>Goodie. But they weren't required for your problem to work out, were they?
>
> > 2. correction to ssl/proxy problem
>
>I would however like to see the implementation of this made a bit
>different.
>I'll explain in detail below.
>
> > 3. new makefile for vc6 for libcurl
>
>Great! I know people will like this.
>
> > note that 1/2 code should be tested on linux before release. I have
> > tested on NT.
>
>I'll personally try to make sure it runs fine on at least Solaris and
>Linux.
>
> > so the loop reads from the buffer until a \r\n\r\n sequence is found,
> > anything after that is left in the buffer i.e. for 0D 0A 0D 0A 00 the 00
> > is left.
> >
> > one could argue that this is correct, nethertheless other clients,
>namely
> > pavuk, etc can cope with the above acknowledgement. This is because they
> > do a block read from the socket buffer, whereas the routine GetLine does
> > a one character at a time read.
> >
> > In principle I believe that http-tunnel connect should leave the socket
> > read buffer clear when it exits. For this reason I have made change #2.
>
>Ok. I understand that this fixes your problem and that other clients do
>this
>and it makes things work.
>
>Unfortunately, it is the wrong thing to do!
>
>This is my view of things. Please correct me or object if you (anyone
>reading
>this) think differently.
>
>We can't blindly assume that the proxy will always first send us the
>headers
>in one chunk and then give us the rest of the reply with a separate pause.
>Even if, since we're doing TCP we can't assume anything in how the stream
>is
>being read from the proxy. Slow-start and the Nagle algorithm could very
>well
>merge lots of data to get read at once, both the initial header and the
>following data stream.
>
>According to the standards, your proxy is doing wrong when it sends a
>trailing zero byte after the final blank line. The data is supposed to
>start
>immediately after the blank line.
>
>That said, I don't think "fix your proxy" is a good answer. I think making
>a
>good "work-round for bad proxies" must first detect the error and then
>step-toe around it. Now we know what the error condition looks like, now we
>must figure out how to be able to detect that and work-around it without
>affecting all the compliant cases. To be frank, I don't currently know how
>to
>do this (automatically).
>
>The initial zero byte in the data stream might signal an error that could
>be
>used to detect this situation, but I'm not sure what to do if we read it
>and
>don't find a zero as that should then have been dealt with by OpenSSL...
>
>Documents used as input for my conclusions:
>
> - Ari Luotonen's internet draft "Tunneling TCP based protocols through
>Web
> proxy servers". It expired in Februrary 1999, but is still available
>all
> over the net. One URL:
>http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt
> (I don't know why this was never made an official RFC or something.)
>
> - RFC2817 "HTTP Upgrade to TLS" section 5.2. It mainly agrees with the
>above
> draft, and doesn't go into the details quite as much.
>
>--
> Daniel Stenberg -- curl groks URLs -- http://curl.haxx.se/
>
>

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp
Received on 2001-11-07