Password containing newline fails when following http 302 #5582

jonjohnsonjr · 2020-06-18T19:50:20Z

This started happening in 7.62.0 (7.61.1 is fine). I narrowed it down to 46e1640 using git bisect, but I'm not familiar enough with curl to dig much deeper than that (it seems like a big change).

Using a password containing a newline works fine on the first request, but if the server returns a 302, something goes wrong when curl is reconstructing the right URL to follow, and fails with:

curl: (3) URL using bad/illegal format or missing URL

I reproduced this with a simple go server that listens on 8080 and redirects from /foo to /bar.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Location", "/bar")
		w.WriteHeader(http.StatusFound)
	})

	http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "bar\n")
	})

	log.Fatal(http.ListenAndServe(":8080", nil))
}

I have a ~/key.json file containing some "credentials" for a basic password:

{
  "hello": "world"
}

This fails because (I believe) key.json contains a newline:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:22:43 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

But using jq to drop newlines with --compact-output succeeds:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json | jq --compact-output .)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%22hello%22%3a%22world%22%7d@localhost:8080/bar'
* Found bundle for host localhost: 0x5d2f3589c0e0 [serially]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /bar HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 4
< Content-Type: text/plain; charset=utf-8
< 
bar
* Connection #0 to host localhost left intact

I did this

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
curl: (3) URL using bad/illegal format or missing URL

I expected the following

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
bar

curl/libcurl version

$ ./src/curl -V
curl 7.71.0-DEV (x86_64-pc-linux-gnu) libcurl/7.71.0-DEV OpenSSL/1.1.0l zlib/1.2.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

operating system

$ uname -a
Linux penguin 4.19.113-08528-g5803a1c7e9f9 #1 SMP PREEMPT Thu Apr 2 15:21:14 PDT 2020 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

sterchelen · 2020-06-18T22:04:43Z

Hey,
First issue for me as trying to explain the why.
Look at lib/escape.c , we are authorizing anything except characters with an hexa code inferior to 0x20

curl/lib/escape.c

Lines 179 to 182 in fa4fbc5

    
           if(reject_ctrl && (in < 0x20)) { 
        
             free(ns); 
        
             return CURLE_URL_MALFORMAT; 
        
           }

Here is the commit that introduces this behavior 75ca568
As @bagder 's commit mentioned, this is for security reason. Here is the secu advisory https://curl.haxx.se/docs/CVE-2012-0036.html

Hope that helps. I am sure that @bagder could add more context and information to my comment.

bagder · 2020-06-18T22:09:36Z

This looks like a duplicate of the issue we fixed with #5400 (600a8cd).

Can you try current master and see if the issue is fixed?

jonjohnsonjr · 2020-06-19T16:23:57Z

First issue for me as trying to explain the why.

This makes sense, thanks for the explanation.

Can you try current master and see if the issue is fixed?

Trying with current master still reproduces:

$ git rev-parse HEAD
fa4fbc533f6ce7e76fc005972a3100ef2a26cc6c

$ ./buildconf && ./configure --with-ssl && make

 $ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Fri, 19 Jun 2020 15:54:47 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

It looks like the fix you linked is being applied, per this line:

Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'

FWIW, we managed to work around this by just constructing the Authorization header directly instead of using -u:

$ ./src/curl -L -H "Authorization: Basic $(echo _json_key:$(cat ~/key.json) | base64 -w0)" localhost:8080/foo
bar

Given we have a workaround, this isn't a big issue for us anymore, especially since the relevant code is a CVE mitigation. Feel free to close this issue if it's too much trouble.

(except for the path part) Reported-by: Jon Johnson Jr Fixes #5582

bagder · 2020-06-23T13:53:57Z

This is an extention of a bug in libcurl's URL API: setting a perfectly valid URL like http://user%0aname:pass%0aword@www.example will return an error for curl_url_get() when trying to extract the user name and password - and have then URL decoded.

Update: I notice this is in fact even documented functionality...

Reported-by: Jon Johnson Jr Fixes #5582

Reported-by: Jon Johnson Jr Fixes #5582 Closes #5592

jonjohnsonjr · 2020-06-25T16:41:25Z

I can confirm this is now fixed, thanks @bagder !

bagder · 2020-06-25T17:19:21Z

thanks for confirming @jonjohnsonjr !

bagder added authentication HTTP labels Jun 18, 2020

bagder self-assigned this Jun 19, 2020

bagder added a commit that referenced this issue Jun 19, 2020

urlapi: allow URL parts to have "control" bytes when decoded

47e8484

(except for the path part) Reported-by: Jon Johnson Jr Fixes #5582

bagder linked a pull request Jun 19, 2020 that will close this issue

Fix newline password handling #5586

Closed

bagder added a commit that referenced this issue Jun 23, 2020

url: allow user + password to contain "control codes" for HTTP(S)

1e15189

Reported-by: Jon Johnson Jr Fixes #5582

bagder mentioned this issue Jun 23, 2020

http: allow user and password to contain "control codes" #5592

Closed

bagder added a commit that referenced this issue Jun 24, 2020

url: allow user + password to contain "control codes" for HTTP(S)

a6cbf00

Reported-by: Jon Johnson Jr Fixes #5582 Closes #5592

bagder closed this as completed in d5ed571 Jun 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Password containing newline fails when following http 302 #5582

Password containing newline fails when following http 302 #5582

jonjohnsonjr commented Jun 18, 2020

sterchelen commented Jun 18, 2020

bagder commented Jun 18, 2020

jonjohnsonjr commented Jun 19, 2020

bagder commented Jun 23, 2020 •

edited

jonjohnsonjr commented Jun 25, 2020

bagder commented Jun 25, 2020

Password containing newline fails when following http 302 #5582

Password containing newline fails when following http 302 #5582

Comments

jonjohnsonjr commented Jun 18, 2020

I did this

I expected the following

curl/libcurl version

operating system

sterchelen commented Jun 18, 2020

bagder commented Jun 18, 2020

jonjohnsonjr commented Jun 19, 2020

bagder commented Jun 23, 2020 • edited

jonjohnsonjr commented Jun 25, 2020

bagder commented Jun 25, 2020

bagder commented Jun 23, 2020 •

edited