cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: HTTP Post with digest

From: Raf Nulens <raf.nulens_at_androme.com>
Date: Thu, 13 Mar 2008 15:00:08 +0100

Daniel Stenberg wrote:
> On Thu, 13 Mar 2008, Raf Nulens wrote:
>
>>> This is a silly server/site that does this, it isn't libcurl's fault.
>>> It can but adapt to the circumstances.
>>
>> But in my traces i've never seen that server return a 100-continue...
>
> If that's the case, then your application has turned off the "Expect:
> 100-continue" header libcurl normally uses. But of course, as we've
> found out, that header is pointless on this site anyway.
>
>> The first header we send (using CURLOPT_POSTFIELDS) looks like this:
>>
>> POST /ws/1/track/ HTTP/1.1
>> Host: musicbrainz.org
>> Accept: */*
>> Content-Length: 0
>> Content-Type: application/x-www-form-urlencoded
>>
>> Server responds with 401
>>
>> Second header:
>>
>> POST /ws/1/track/ HTTP/1.1
>> Authorization: Digest username="doesntmatter_at_example.com",
>> realm="musicbrainz.org",
>> nonce="fa87522a9a8ca783ebd634ec209375e41205399036",
>> uri="/ws/1/track/", response="ecf90cae4325454acf213a7cf9d10c28"
>> Host: musicbrainz.org
>> Accept: */*
>> Content-Length: 11
>> Content-Type: application/x-www-form-urlencoded
>>
>> Server responds with 401
>>
>> this is the way it should work.
>
> Again you're speaking in "should" terms and while I could possibly agree
> to this statement, this is not how this server works and not how most of
> other deployed servers "out there" work. And in fact it isn't how HTTP
> works.
>
> The server doesn't respond with a 401 until *AFTER* the entire posted
> data (as specified in the Content-Length:) has been sent. There's no
> opportunity for libcurl to stop sending this data in vain. There really
> isn't.
>
> Just to really really make sure you hadn't found a weird state in
> libcurl or in this server that I didn't spot with my previous test, I
> modified my code now to also work with POSTFIELDS and even with
> disabling the Expect: header just to make my test code able to match
> both your use cases in which you say one of them works and one doesn't.
>
> I claim none of the cases work the way you want them to, but instead
> both of them work the way I say they work and I also claim that there's
> nothing libcurl can do to avoid this extra sending of the data in case
> of authentication errors.
>
>>> Disabling 100-continue will only take away the only means the server
>>> would have to deny the data to get posted when it hasn't been
>>> authorized properly. So no, we don't expect removing the header to
>>> improve this situation.
>>
>> But its not an obligation to use the 100-continue header when using
>> digest.
>
> There's never an obligation to use it, but it is the only real chance a
> client can offer a server to deny the sending of the data that is
> otherwise possibly sent in vain to the server even though the request is
> denied authentication.
>
>> Libcurl should just wait with sending binary data until the 401
>> response is received from the server.
>
> No. That's not how HTTP works in general, that's not how HTTP POSTs are
> made according to the spec and that's not how this server works.
>
> It is *OBVIOUS* and quite clear when you run my test program to this
> server that it simply doesn't respond _anything_ (apart from the dummy
> 100 Continue) until the *ENTIRE* post is sent.
>
> Are you denying this? It seems you're hanging on to your stand-point
> without absorbing what I say even though I've confirmed my statements
> with multiple actual live tests against this server.
>
>> And this _does_ work this way when we're using CURLOPT_POSTFIELDS.
>
> Nope. You can easily verify that I'm right and that you're wrong by
> using wireshark or CURLOPT_DEBUGFUNCTION. Or simply compiling and
> running my test code.
>
> It works the same way with CURLOPT_POSTFIELDS as with a read callback.
> I've tried CURLOPT_POSTFIELDS with 10K POSTs and 150K posts and they all
> work identically.
>

I'm afraid i havent made the problem clear... Here are the two traces,
first with CURLOPT_POSTFIELDS, second with read_callback:

Data we're sending is <xml></xml>

=======================================

POST /ws/1/track/ HTTP/1.1
Host: musicbrainz.org
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

HTTP/1.1 401 Authorization Required
Date: Thu, 13 Mar 2008 13:41:24 GMT
Server: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
WWW-Authenticate: Digest realm="musicbrainz.org",
nonce="b1acf1b5fc07197670a5e3be3d43a84b1205415684"
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 musicbrainz.org
Connection: close
Transfer-Encoding: chunked

1d5

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>401 Authorization Required</TITLE>
</HEAD><BODY>
<H1>Authorization Required</H1>
This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.<P>
<HR>
<ADDRESS>Apache/1.3.33 Server at musicbrainz.org Port 80</ADDRESS>
</BODY></HTML>

POST /ws/1/track/ HTTP/1.1
Authorization: Digest username="doesntmatter_at_example.com",
realm="musicbrainz.org",
nonce="b1acf1b5fc07197670a5e3be3d43a84b1205415684", uri="/ws/1/track/",
response="8ab6c01bcaf602ad82ebbafb11a6d59c"
Host: musicbrainz.org
Accept: */*
Content-Length: 11
Content-Type: application/x-www-form-urlencoded

<xml></xml>HTTP/1.1 401 Authorization Required
Date: Thu, 13 Mar 2008 13:41:24 GMT
Server: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
WWW-Authenticate: Digest realm="musicbrainz.org",
nonce="b1acf1b5fc07197670a5e3be3d43a84b1205415684"
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 musicbrainz.org
Connection: close
Transfer-Encoding: chunked

1d5

=======================================
second example (read-callback)

POST /ws/1/track/ HTTP/1.1
Host: musicbrainz.org
Accept: */*
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

<xml></xml>HTTP/1.1 401 Authorization Required
Date: Thu, 13 Mar 2008 13:41:21 GMT
Server: Apache/1.3.33 (Debian GNU/Linux) mod_perl/1.29
WWW-Authenticate: Digest realm="musicbrainz.org",
nonce="eeae99f18f6e74a1abbbdb8b055045901205415681"
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 musicbrainz.org
Connection: close
Transfer-Encoding: chunked

1d5

=======================================

In the second example the data is sent without waiting for the 401
Authentication Required. Notice content-length 0, but still libcurl is
sending data.

In the first example everything goes as it should:
-> POST with content length = 0
-> receive 401
-> resend POST header with digest & authentication and immediately send
the data
-> receive 401 because of auth failed

If your traces tell something else, could you please post them here to
see if there is anything different sent in the headers?

kind regards

Raf Nulens
Received on 2008-03-13

This message: [ Message body ]
Next message: Dan Petitt: "Re: Progress During Large Posts"
Previous message: Brian Ulm: "Re: Re-using SFTP connection and CURLOPT_FTP_CREATE_MISSING_DIRS issue"
In reply to: Daniel Stenberg: "Re: HTTP Post with digest"
Next in thread: Daniel Stenberg: "Re: HTTP Post with digest"
Reply: Daniel Stenberg: "Re: HTTP Post with digest"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]