Bugs item #2799008, was opened at 2009-05-31 08:06
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2799008&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: http
Group: wrong behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Aaron Oneal (incarnadine)
Assigned to: Daniel Stenberg (bagder)
Summary: POST + NTLM broken

Initial Comment:
Using 7.19.5, it is not possible to use NTLM and do a POST against a server like IIS7.

Flow:
1. Request w/ content length but no body! <== bad libcurl! must send body or 0 content length
2. Response 401 authorization required
3. Request (NTLM authorized) w/ content length + body
4. Response 400 bad request

IIS correctly tells libcurl to go away with its authorized request because it didn't finish sending the body from the first request.

I discovered this while trying to do what I would expect is a very common scenario -- use libcurl (via Axis2/C) to send a SOAP request to a web service.

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2009-06-02 23:14

Message:
scrap that patch, was far too simple

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2009-06-02 19:50

Message:
The "weirdo" part is that the app you're using is _not_ following the 1-2-3
steps. Instead it has decided to use its own header in step (2) that's
against our general recommendations and in general is a weird thing to do.
I say this. I designed the API. I don't understand why you feel a need to
contradict this. They don't "Send a content length of X" the way the API is
designed to be used.

I understand that you want your app to work, but that doesn't defend what
they do in their libcurl use.

The fact that we can (and should?) fix this particular problem doesn't
make this a sensible use of libcurl.

See my suggested fix for this mess in the uplodaed
content-length-post-ntlm.patch

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-06-02 18:02

Message:
The caller is instructing libcurl to:
1. Use NTLM authentication
2. Send a content length of X
3. Send a body of Y

That's a perfectly valid request and not weirdo at all. Libcurl is
behaviing incorrectly.

It either needs to:
1. Honor the request as the caller made it, send the body, and let the
caller handle the 401.
2. Send the pre-auth request but with 0 content length and then use the
caller's length and body information for the subsequent request.

I prefer #2 as it's consistent with what happens when you don't specify a
content-length, and it's just darn conveninent.

In no case would it be expected behavior for a caller to make a request
like above and have libcurl respond by sending a content-length and not
body. If that's what the caller wanted, that's what they would have
specified.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2009-06-02 09:51

Message:
Sending a custom Content-Length: is almost always fishy and somewhat of a
weird thing to do. Thus: when an app does it, it should indicate that it
truly know what its doing and it needs to do it. It then makes it a whole
lot harder for libcurl to make any decisions on exactly what the app wants
to achieve. How can libcurl tell the app doesn't want this weirdo request?

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-06-02 09:24

Message:
As I mentioned, this is in the Axis2/C library (not my code), so I don't
really know why the particular design was chosen. I don't see how
CURLOPT_POSTFIELDSIZE_LARGE would particularly help either. The problem is
in sending the pre-auth request when the content-length header is specified
(and it still would be with CURLOPT_POSTFIELDSIZE_LARGE), so the two
solutions I'm aware of are:
1. Stop sending the header from Axis2/C
2. Fix Curl to not do something obviously silly or at least detect the
condition and fail with a more informative message

I filed a bug with the Axis2/C team for #1, and welcome to #2.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2009-06-02 08:45

Message:
And now you know why "it isn't very wise to actually set your own"
Content-Length header. What's the matter with just setting
CURLOPT_POSTFIELDSIZE_LARGE?

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-06-02 08:23

Message:
I got a debug build going and figured out what's going on. Axis2/C is
actually setting a content-length header, and Curl is using it on the first
request even though it knows it's not going to send the body. So, that's
not right. Here's the problem area starting on line 2774 of http.c (and
there are similar problems with PUT)

  case HTTPREQ_POST:
    /* this is the simple POST, using x-www-form-urlencoded style */

    if(conn->bits.authneg)
      postsize = 0;
    else {
      /* figure out the size of the postfields */
      postsize = (data->set.postfieldsize != -1)?
        data->set.postfieldsize:
        (data->set.postfields?
(curl_off_t)strlen(data->set.postfields):0);
    }
    if(!data->req.upload_chunky) {
      /* We only set Content-Length and allow a custom Content-Length if
         we don't upload data chunked, as RFC2616 forbids us to set both
         kinds of headers (Transfer-Encoding: chunked and Content-Length)
*/

      if(!checkheaders(data, "Content-Length:")) {
        /* we allow replacing this header, although it isn't very wise to
           actually set your own */
        result = add_bufferf(req_buffer,
                             "Content-Length: %" FORMAT_OFF_T"\r\n",
                             postsize);
        if(result)
          return result;
      }
    }

You can see that it correctly sets the postsize to 0 because of
conn->bits.authneg, but because the caller specified content-length, it
uses that value instead. Since Curl knows it's not going to send the body,
it should remove the caller specified content-length on the first request
and use it on the subsequent one when it actually intends to send the body.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2009-06-01 13:56

Message:
isn't this then as test 267? It _does_ send a Content-Length: 0 btw so your
case must obviously be different. The question is probably How?

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-05-31 21:31

Message:
I checked out lib553.c and see it's using CURLOPT_READFUNCTION instead of
CURLOPT_POSTFILEDS, but otherwise I don't notice anything out of the
ordinary (besides it not using NTLM). The other tests are in a format I'm
not familiar with.

It could be the server the test suite runs against isn't waiting around
for the body of the first request once it sends the 401 and so the test is
passing.

You can see from the traces though what's going across the wire and this
would seem to indicate a problem.

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-05-31 18:07

Message:
The Expect header is not being sent in the request and the server responds
immediately with a 401. I looked at the Axis2/C code and it explicitly sets
the header to "Expect:" which I presume is how you tell libcurl not to send
the header, though I don't know why they would do that. I'll have a look at
those tests to see if I can compare it with what the Axis2/C code is doing
and will also attach the relevant file here.

I have attached a NetMon 3.3 trace which demonstrates the problem.

You can grab NetMon here if you don't have it handy:
http://www.microsoft.com/downloads/details.aspx?FamilyID=983b941d-06cb-4658-b7f6-3088333d062f&displaylang=en

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2009-05-31 08:26

Message:
This is a common scenario and it has several tests in the curl test suite
(e.g. 267, 553, 1100). How does your scenario differ from those? Are you
sure you're not running into libcurl's Expect: 100-continue behaviour,
which is desired? Can you post the log of your failing session so we can
see exactly what's going on?

----------------------------------------------------------------------

Comment By: Aaron Oneal (incarnadine)
Date: 2009-05-31 08:14

Message:
Also worth mentioning, I'm using MinGW and building as follows:
mingw32-make -f Makefile.m32 libcurl.a SSL=1 ZLIB=1 SSPI=1 IPV6=1

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2799008&group_id=976
Received on 2009-06-02