Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP POST with Negotiate fails #1261

Closed
dhoelzl opened this issue Feb 13, 2017 · 11 comments
Closed

HTTP POST with Negotiate fails #1261

dhoelzl opened this issue Feb 13, 2017 · 11 comments

Comments

@dhoelzl
Copy link
Contributor

dhoelzl commented Feb 13, 2017

I did this

On Windows I want to perform a POST (or another custom) request (CURLOPT_CUSTOMREQUEST) with some data specified with CURLOPT_UPLOAD/CURLOPT_INFILESIZE_LARGE/CURLOPT_READFUNCTION/CURLOPT_SEEKFUNCTION by using negotiate authentication (CURLOPT_HTTPAUTH CURLAUTH_NEGOTIATE, CURLOPT_USERNAME (empty for Windows credentials), CURLOPT_PASSWORD (empty for Windows credentials)), but the authentication fails. Performing a GET request with the same setup succeeds.

Explicitly suppressing the "Expect: 100-continue" header seems to get the request working, but this is not an acceptable solution.

Using CURLOPT_POSTFIELDS instead of CURLOPT_UPLOAD, CURLOPT_READFUNCTION, CURLOPT_SEEKFUNCTION also works. But this is not the intended way to upload large amounts of data.
Using CURLOPT_POSTFIELDS with explicitly defining the "Expect: 100-continue" header fails.

Using CURLAUTH_NTLM instead of CURLAUTH_NEGOTIATE with the same cURL configuration works (also with the "Expect: 100-continue" mechanism). But this is less secure.

Connecting to an IIS with all authentication methods disabled but "Windows Authentication"

I expected the following

Negotiate authentication works also for POST/PUT-requests with CURLOPT_UPLOAD/CURLOPT_READFUNCTION. Handles with a GET request which performed a negotiate challenge can be reused to avoid challenge requests on every new request re-using a CURL handle.

curl/libcurl version

Used cURL Library: Version 7.52.1 64 Bit on Windows built with
-DBUILDING_LIBCURL -DHTTP_ONLY _DWITHOUT_MM_LIB -DUSE_SSLEAY -DUSE_OPENSSL -DCURL_WANTS_CA_BUNDLE_ENV -DENABLE_IPV6 -DUSE_WINDOWS_SSPI -DHAVE_STRUCT_POLLFD

operating system

Client: Tested on Windows 7 and Windows 2008 R2

connection close after negotiate challenge?

In cURL source code I found the following which seems to kill the connection after each negotiation. Why is this? Shouldn't the connection be kept after a (successful) negotiation for re-use?

If I comment out the "streamclose" part and I perform a GET request with negotiate authentication and reuse the same CURL handle for a following POST request with CURLOPT_UPLOAD the POST request succeeds as the authenticated connection will not be killed after the GET. Of course I cannot use this experimental trial as a workaround. I just wanted to find out how the connection handling of cURL works.

CURLcode Curl_http_done(struct connectdata *conn,
                        CURLcode status, bool premature)
{
  struct Curl_easy *data = conn->data;
  struct HTTP *http = data->req.protop;

  infof(data, "Curl_http_done: called premature == %d\n", premature);

  Curl_unencode_cleanup(conn);

#ifdef USE_SPNEGO
  if(data->state.proxyneg.state == GSS_AUTHSENT ||
     data->state.negotiate.state == GSS_AUTHSENT) {
    /* add forbid re-use if http-code != 401/407 as a WA only needed for
     * 401/407 that signal auth failure (empty) otherwise state will be RECV
     * with current code.
     * Do not close CONNECT_ONLY connections. */
    if((data->req.httpcode != 401) && (data->req.httpcode != 407) && // <-- uncommenting this
       !data->set.connect_only) // <-- uncommenting this
      streamclose(conn, "Negotiate transfer completed"); // <-- uncommenting this
    Curl_cleanup_negotiate(data);
  }
#endif
...

code to reproduce the problem:

#define CURL_STATICLIB
#include <curl/curl.h>

//
// Select one of the test cases:
//

// #1 Negotiate upload with CURLOPT_UPLOAD (which I expect to be working)

#define TEST_NEGOTIATE
#define TEST_UPLOAD

// ------------------------------------------------

// #2 Negotiate upload with CURLOPT_POSTFIELDS (works)
/*
#define TEST_NEGOTIATE
#define TEST_POSTFIELDS
*/
// ------------------------------------------------

// #3 NTLM upload (works)
/*
#define TEST_NTLM
#define TEST_UPLOAD
*/
// ------------------------------------------------

// #4 Negotiate GET and UPLOAD with same handle
// (after successful GET the connection is killed, PUT fails)
/*
#define TEST_NEGOTIATE
#define TEST_GET_WITH_UPLOAD
*/
// ------------------------------------------------


static int CURLdebugfunction(CURL *, curl_infotype infotype, char *str, size_t len, void *)
{
  const char *infostring = "CURLINFO_???";
  switch (infotype) {
  #define ENTRY(NAME) case NAME: infostring = TEXT(#NAME); break;
    ENTRY(CURLINFO_TEXT)
      ENTRY(CURLINFO_HEADER_IN)
      ENTRY(CURLINFO_HEADER_OUT)
      ENTRY(CURLINFO_DATA_IN)
      ENTRY(CURLINFO_DATA_OUT)
    #undef ENTRY
  }
  printf("CURLDEBUGINFO %s: %*.*s\n",
    (const char *)infostring, (int)len, (int)len, (const char *)str);
  return 0;
}
size_t CURLheaderfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  char *hdr = (char *)malloc(size*nmemb + 1);
  if (hdr) {
    memcpy(hdr, ptr, size*nmemb);
    hdr[size*nmemb] = '\0';
    printf("RECEIVED HEADER: %s", (const char *)hdr);
    free(hdr);
  }
  else {
    printf("HEADER: OUT OF MEMORY\n");
  }
  return size*nmemb;
}
static size_t CURLreadpos = 0;
size_t CURLreadfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t read = 0;
  for (; CURLreadpos < 10 && CURLreadpos < size * nmemb; CURLreadpos++) {
    ((char *)ptr)[CURLreadpos] = 'a';
    read++;
  }
  printf("READ REQUEST DATA: %llu -> %llu\n", size*nmemb, (UINT64)read);
  return read;
}
int CURLseekfunction(void *ptr, curl_off_t offset, int origin)
{
  printf("SEEK: %lld, %d\n", (INT64)offset, origin);
  if (origin == SEEK_SET) {
    CURLreadpos = offset;
    return CURL_SEEKFUNC_OK;
  }
  printf("SEEK: NOT IMPL\n");
  return CURL_SEEKFUNC_CANTSEEK;
}
int main(void)
{
  CURL *curl;
  CURLcode res;
  curl_global_init(CURL_GLOBAL_ALL);
  curl = curl_easy_init();
  if (curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "http://url-to-server-with-windows-auth-goes-here");
    curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, CURLheaderfunction);
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, CURLreadfunction);
    curl_easy_setopt(curl, CURLOPT_SEEKFUNCTION, CURLseekfunction);
    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, CURLdebugfunction);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  #if defined(TEST_NEGOTIATE)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE);
    curl_easy_setopt(curl, CURLOPT_USERNAME, "");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "");
  #endif
  #if defined(TEST_NTLM)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
    curl_easy_setopt(curl, CURLOPT_USERNAME, "<NTLM-User>");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "<NTLM-Password>");
  #endif

  #if defined(TEST_UPLOAD)
      curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
      curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
      curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, 10);
  #endif
  #if defined(TEST_POSTFIELDS)
    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "aaaaaaaaaa");
  #endif
  #if defined(TEST_GET_WITH_UPLOAD)
    curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
  #endif

    res = curl_easy_perform(curl);

  #if defined(TEST_GET_WITH_UPLOAD)
    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, 10);
    res = curl_easy_perform(curl);
  #endif

    if (res != CURLE_OK) {
      printf("curl_easy_perform() failed: %s\n",
        curl_easy_strerror(res));
    }
    else {
      long http_code = 0;
      curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
      printf("CURL STATUS: %ld\n", http_code);
    }
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

test program output

Output test case # 1 (Negotiate upload)
#1_negotiate_upload_failed.txt

Output test case # 2 (Negotiate postfields)
#2_negotiate_postfields_succeeded.txt

Output test case # 3 (NTLM upload)
#3_ntlm_upload_succeeded.txt

Output test case # 4 (Negotiate upload with same handle after GET)
#4_negotiate_upload_after_successful_get_failed.txt

@bagder bagder changed the title HTTP POST/PUT with Negotiate Authentication (CURLAUTH_NEGOTIATE) and data specified with CURLOPT_UPLOAD/CURLOPT_INFILESIZE_LARGE/CURLOPT_READFUNCTION/CURLOPT_SEEKFUNCTION fails HTTP POST with Negotiate fails Feb 13, 2017
@dhoelzl
Copy link
Contributor Author

dhoelzl commented Feb 22, 2017

Can anybody reproduce this?
Do you need some additional information?

@jzakrzewski
Copy link
Contributor

It happens to our system also. Unless the "Expect: ..." is suppressed it doesn't work. Due to lack of time, setting this header is our official workaround. Although not ideal as curl has to re-send the data after auth.

@dhoelzl
Copy link
Contributor Author

dhoelzl commented Feb 24, 2017

For me suppressing the Expect-Header works if directly connected to the HTTP server for small requests, but if going through a proxy (CCProxy 8.0 on Windows in my test), the first request which is a POST request in my case succeeds, but the second request (on the same cURL handle) which is a GET request does not get any response and hangs/times out. Could be that this is a CCProxy issue, but I have also seen that problem without proxy after some more requests have already been performed with cURL, maybe depending on the state of the cURL's connection cache.
Suppressing the Expect-Header does also not work for me if the request body is more than 32kb (or randomly already at 16kb). After providing that data via the cURL read function, the server seems to stop the request and and sends a 401 somewhere in the middle of the challenge handshake and cURL aborts the challenge/request. The cURL seek function seems to be called properly in that case.

@iboukris
Copy link
Contributor

iboukris commented Feb 24, 2017

I'd expect Negotiate POST to be not optimal in case of multi-round negotiation (usually NTLM inside), because there is no special handling like the 'authneg' logic for NTLM, see more in this mail thread:
https://curl.haxx.se/mail/lib-2016-11/0196.html

However, it sounds like some flows (like with Expect) behave better than others so perhaps it could be fixed up to at least work.

@dhoelzl
Copy link
Contributor Author

dhoelzl commented Feb 27, 2017

No, I can't get it working with uploading big files. cURL calls the read function multiple times (for each challenge request) twice, in combination with the seek function and and after that the server responds always with 401; the full content is never transferred to the server.
What can I do to upload big files with Negotiate?

#define CURL_STATICLIB
#include <curl/curl.h>

// Test case #1
#define TEST_NEGOTIATE
#define UPLOAD_SIZE (1 * 1024 * 1024)
#define TEST_SUPPRESS_EXPECT_CONTINUE

// Test case #2
/*
#define TEST_NTLM
#define UPLOAD_SIZE (1 * 1024 * 1024)
*/

// Test case #3
/*
#define TEST_NEGOTIATE
#define UPLOAD_SIZE 10
#define TEST_SUPPRESS_EXPECT_CONTINUE
*/

static int CURLdebugfunction(CURL *, curl_infotype infotype, char *str, size_t len, void *)
{
  const char *infostring = "CURLINFO_???";
  switch (infotype) {
  #define ENTRY(NAME) case NAME: infostring = TEXT(#NAME); break;
    ENTRY(CURLINFO_TEXT)
      ENTRY(CURLINFO_HEADER_IN)
      ENTRY(CURLINFO_HEADER_OUT)
      ENTRY(CURLINFO_DATA_IN)
      ENTRY(CURLINFO_DATA_OUT)
    #undef ENTRY
  }
  printf("CURLDEBUGINFO %s: %*.*s\n",
    (const char *)infostring, (int)len, (int)len, (const char *)str);
  return 0;
}
size_t CURLheaderfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  char *hdr = (char *)malloc(size*nmemb + 1);
  if (hdr) {
    memcpy(hdr, ptr, size*nmemb);
    hdr[size*nmemb] = '\0';
    printf("RECEIVED HEADER: %s", (const char *)hdr);
    free(hdr);
  }
  else {
    printf("HEADER: OUT OF MEMORY\n");
  }
  return size*nmemb;
}
static size_t CURLreadpos = 0;
size_t CURLreadfunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
  size_t read = 0;
  for (; CURLreadpos < UPLOAD_SIZE && read < size * nmemb;) {
    ((char *)ptr)[read] = 'a';
    read++;
    CURLreadpos++;
  }
  printf("READ REQUEST DATA: %llu -> %llu\n", size*nmemb, (UINT64)read);
  return read;
}
int CURLseekfunction(void *ptr, curl_off_t offset, int origin)
{
  printf("SEEK: %lld, %d\n", (INT64)offset, origin);
  if (origin == SEEK_SET) {
    CURLreadpos = offset;
    return CURL_SEEKFUNC_OK;
  }
  printf("SEEK: NOT IMPL\n");
  return CURL_SEEKFUNC_CANTSEEK;
}
int main(void)
{
  CURL *curl;
  CURLcode res;
  curl_global_init(CURL_GLOBAL_ALL);
  curl = curl_easy_init();
  if (curl) {
    curl_easy_setopt(curl, CURLOPT_URL, "http://url-to-server-with-windows-auth-goes-here");
    curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, CURLheaderfunction);
    curl_easy_setopt(curl, CURLOPT_READFUNCTION, CURLreadfunction);
    curl_easy_setopt(curl, CURLOPT_SEEKFUNCTION, CURLseekfunction);
    curl_easy_setopt(curl, CURLOPT_DEBUGFUNCTION, CURLdebugfunction);
    curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);

  #if defined(TEST_NEGOTIATE)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NEGOTIATE);
  #endif
  #if defined(TEST_NTLM)
    curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM);
  #endif
    curl_easy_setopt(curl, CURLOPT_USERNAME, "");
    curl_easy_setopt(curl, CURLOPT_PASSWORD, "");

    curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
    curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
    curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, UPLOAD_SIZE);

  #if defined(TEST_SUPPRESS_EXPECT_CONTINUE)
    struct curl_slist *headers = NULL;
    headers = curl_slist_append(headers, "Expect:");
    curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
  #endif

    res = curl_easy_perform(curl);

    if (res != CURLE_OK) {
      printf("curl_easy_perform() failed: %s\n",
        curl_easy_strerror(res));
    }
    else {
      long http_code = 0;
      curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);
      printf("CURL STATUS: %ld\n", http_code);
    }
    curl_easy_cleanup(curl);
  }
  curl_global_cleanup();
  return 0;
}

test program output

Negotiate with suppressing expect/continue and uploading 1 MiB
#1_negotiate_suppress_expect_1mib_failed.txt

NTLM with uploading 1 MiB (not suppressing expect/continue)
#2_ntlm_1mib_succeeded.txt

Negotiate with suppressing expect/continue and uploading 10 bytes
#3_negotiate_supporess_expect_10bytes_succeeded.txt

@Nitrael86
Copy link

We have the same problem in our environment with negotiate, is there any possability to get it fixed?

@dhoelzl
Copy link
Contributor Author

dhoelzl commented Oct 9, 2017

I have created a fix which solves the following problems with CURLAUTH_NEGOTIATE:

  • Immediately initialize negotiate challenge on the first challenge request (This was not done so far)
  • Do not send content while negotiating (Content-Length must be zero in this phase)
  • Fix Expect: 100-continue handling
  • Enable reusing easy handles to avoid re-challlenging every request
  • Enable restarting negotiate authentication on a already authenticated connection

I changed the following:

  • Added GSS_AUTHDONE state to negotiatedata which represents a connection with completed negotiation
  • Added GSS_AUTHSUCC state to negotiatedata which represents a fully authenticated negotiated connection
  • Moved and adjusted authentication state handling from output_auth_headers into Curl_output_negotiate and Curl_auth_spnego_cleanup (All other challenge based auth methods do this)
  • Commented out the code which closed the connection after a negotiated request
  • Add read stream rewind logic analogical to the NTLM stream rewind handling.
  • Moved negotiatedata members from UrlState to connectdata
  • Consider authproblem state for CURLAUTH_NEGOTIATE
  • Consider reuse_forbid for CURLAUTH_NEGOTIATE
  • Consider negotiate authentication restart on a already fully authenticated connection

I have tested this code (only on Windows!) by simultaneously downloading and uploading small and big amounts of data and reusing already created connections and either directly connecting to a server or doing negotiate authentication through a proxy server. I have not tested proxy negotiate authentication.
Regards,
Dominik

@stale
Copy link

stale bot commented Apr 15, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 15, 2018
@dhoelzl
Copy link
Contributor Author

dhoelzl commented Apr 16, 2018

PR is still active

@stale
Copy link

stale bot commented Mar 12, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Mar 12, 2019
@MarcelRaad MarcelRaad removed the stale label Mar 12, 2019
@MarcelRaad
Copy link
Member

The PR is still active.

@bagder bagder closed this as completed in 6c60355 Mar 14, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

6 participants