Bugs item #1879375, was opened at 2008-01-24 20:10
Message generated for change (Comment added) made by knreed
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1879375&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: https
Group: wrong behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Kevin Reed (knreed)
Assigned to: Daniel Stenberg (bagder)
Summary: HTTPS Post over NTLM Proxy when using callback

Initial Comment:
On December 5 there is the following fix reported:

-----------------------------------------------------
Daniel S (5 Dec 2007)
- Spacen Jasset reported a problem with doing POST (with data read with a callback) over a proxy when NTLM is used as auth with the proxy. The bug also concerned Digest and was limited to using callback only. Spacen worked with us to provide a useful patch. I added the test case 547 and 548 to verify two variations of POST over proxy with NTLM.
-----------------------------------------------------

We have a situation that matches this very closely. And in fact when testing with the CVS snapshot from Jan 20th it is fixed (Great work!). However, we found that it does not work correctly when using https. I suspect that if test cases 547 and 548 were run using https instead of http you would see what our applications are seeing.

Here is the output from our application:

1/24/2008 7:52:58 PM||Libraries: libcurl/7.18.0-20080120 OpenSSL/0.9.8g zlib/1.2.3
1/24/2008 8:04:25 PM|World Community Grid|Sending scheduler request: Requested by user. Requesting 0 seconds of work, reporting 2 completed tasks
1/24/2008 8:04:25 PM||[http_debug] HTTP_OP::init_post(): https://secure.worldcommunitygrid.org/boinc/wcg_cgi/fcgi
1/24/2008 8:04:25 PM||[http_debug] [ID#127] info: About to connect() to proxy www.romwnet.org port 8080 (#1)
1/24/2008 8:04:25 PM||[http_debug] [ID#127] info: Trying 75.144.221.229...
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Connected to www.romwnet.org (75.144.221.229) port 8080 (#1)
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Establish HTTP proxy tunnel to secure.worldcommunitygrid.org:443
1/24/2008 8:04:26 PM||[http_debug] [ID#127] Sent header to server: CONNECT secure.worldcommunitygrid.org:443 HTTP/1.0
Host: secure.worldcommunitygrid.org:443
User-Agent: BOINC client (windows_intelx86 5.10.38)
Proxy-Connection: Keep-Alive

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: HTTP/1.1 407 Proxy Authentication Required

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Server: FreeProxy/4.0

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Date: Fri, 25 Jan 2008 02:04:29 GMT

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Content-Type: text/html

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Transfer-Encoding: Chunked

1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: CONNECT responded chunked
1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Proxy-Authenticate: NTLM

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Proxy-Authenticate: Basic realm="ROMWNET"

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server: Proxy-Connection: Close

1/24/2008 8:04:26 PM||[http_debug] [ID#127] Received header from server:

1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: 1 bytes of chunk left
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Read 0 bytes of chunk, continue
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Read 1455 bytes of chunk, continue
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Read 567 bytes of chunk, continue
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Read 0 bytes of chunk, continue
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: chunk reading DONE
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Received HTTP code 407 from proxy after CONNECT
1/24/2008 8:04:26 PM||[http_debug] HTTP error: Failure when receiving data from the peer
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Expire cleared
1/24/2008 8:04:26 PM||[http_debug] [ID#127] info: Connection #1 to host www.romwnet.org left intact
1/24/2008 8:04:27 PM||Project communication failed: attempting access to reference site

----------------------------------------------------------------------

>Comment By: Kevin Reed (knreed)
Date: 2008-01-25 09:15

Message:
Logged In: YES
user_id=1991134
Originator: YES

Your question made me wonder what happens if I force the proxy auth to be
NTLM. I changed this line:

curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYAUTH, CURLAUTH_ANY);

to

curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYAUTH, CURLAUTH_NTLM);

to see if forcing the NTLM bit would work. With the NTLM bit forced it
works great. However, we are deploying this to an environment where the
proxy type is not known so we need to be able to set CURLAUTH_ANY.

thanks

----------------------------------------------------------------------

Comment By: Kevin Reed (knreed)
Date: 2008-01-25 09:03

Message:
Logged In: YES
user_id=1991134
Originator: YES

Thank you for taking a look at this so quickly. Here is the code that is
used to set the curl options. This code sets it up and it works (with the
fix from Dec 5) great for posts that use http. However, when using https
it fails.

    curlEasy = curl_easy_init(); // get a curl_easy handle to use
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_URL, m_url);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_SSL_VERIFYHOST, 2L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_SSL_VERIFYPEER, 1L);
    if (boinc_file_exists(CA_BUNDLE_FILENAME)) {
        // call this only if a local copy of ca-bundle.crt exists;
        // otherwise, let's hope that it exists in the default place
        //
        curlErr = curl_easy_setopt(curlEasy, CURLOPT_CAINFO,
CA_BUNDLE_FILENAME);
    }

    // set the user agent as this boinc client & version
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_USERAGENT,
g_user_agent_string);

    // bypass any signal handlers that curl may want to install
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_NOSIGNAL, 1L);
    // bypass progress meter
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_NOPROGRESS, 1L);

    // setup timeouts
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_TIMEOUT, 0L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_LOW_SPEED_LIMIT, 10L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_LOW_SPEED_TIME, 300L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_CONNECTTIMEOUT, 120L);
    
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_MAXREDIRS, 50L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_AUTOREFERER, 1L);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_FOLLOWLOCATION, 1L);

    // if we tell Curl to accept any encoding (e.g. deflate)
    // it seems to accept them all, which screws up projects that
    // use gzip at the application level.
    // So, detect this and don't accept any encoding in that case
    //
    if (!out || !ends_with(std::string(out), std::string(".gz"))) {
        curlErr = curl_easy_setopt(curlEasy, CURLOPT_ENCODING, "");
    }

    // setup a basic http proxy
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYTYPE,
CURLPROXY_HTTP);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYPORT, (long)
pi.http_server_port);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXY, (char*)
pi.http_server_name);

        curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYAUTH, CURLAUTH_ANY);
    sprintf(szCurlProxyUserPwd, "%s:%s", pi.http_user_name,
pi.http_user_passwd);
    curlErr = curl_easy_setopt(curlEasy, CURLOPT_PROXYUSERPWD,
szCurlProxyUserPwd);

    static const char g_content_type[] = {"Content-Type:
application/x-www-form-urlencoded"};
    pcurlList = curl_slist_append(pcurlList, g_content_type);

   curlErr = curl_easy_setopt(curlEasy, CURLOPT_WRITEFUNCTION,
libcurl_write);
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_WRITEDATA, this);

   curl_off_t fs = (curl_off_t) content_length;

   pByte = NULL;
   lSeek = 0; // initialize the vars we're going to use for byte
transfers

   // CMC Note: we can make the libcurl_read "fancier" in the future,
   // for now it just fwrite's to the file request, which is sufficient
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_POSTFIELDS, NULL);
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_POSTFIELDSIZE_LARGE, fs);
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_READFUNCTION,
libcurl_read);
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_READDATA, this);

   // callback function to rewind input file
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_IOCTLFUNCTION,
libcurl_ioctl);
   curlErr = curl_easy_setopt(curlEasy, CURLOPT_IOCTLDATA, this);

   curlErr = curl_easy_setopt(curlEasy, CURLOPT_POST, 1L);

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-01-25 06:08

Message:
Logged In: YES
user_id=1110
Originator: NO

And CURLOPT_PROXYUSERPWD is set to something and CURLOPT_PROXYAUTH
includes the CURLAUTH_NTLM bit?

This output looks like libcurl simply doesn't trigger a remade request
with the authentication and thus it exits the loop and returns error since
it fails to go through the proxy.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1879375&group_id=976
Received on 2008-01-25