--etag-save failing with error "(23) Failed writing header" #5610

bhartshorn · 2020-06-25T16:04:53Z

I did this

Against a specific server (get.helm.sh), curl returns an error when using the --etag-save flag. I have used this feature with other servers (specifically github releases) without issue. I'm not familiar enough with http to troubleshoot this further. I tested on two machines: latest MacOS and Arch Linux, each with fully updated curl versions (7.71 & 7.70).

MacOS (installed via homebrew) output:

> uname -a
Darwin [masked hostname] 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

> /usr/local/opt/curl/bin/curl -V
curl 7.71.0 (x86_64-apple-darwin19.5.0) libcurl/7.71.0 SecureTransport zlib/1.2.11
Release-Date: 2020-06-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

> /usr/local/opt/curl/bin/curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering http/1.1
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: sni114fcgl.wpc.edgecastcdn.net
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/1.1
> Host: get.helm.sh
> User-Agent: curl/7.71.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Age: 43
< Content-MD5: LntYfUTjxG3ILF3WkCeKzQ==
< Content-Type: application/x-tar
< Date: Thu, 25 Jun 2020 15:24:11 GMT
< Etag: 0x8D8117F40DD72AC
* Failed writing header
* Closing connection 0
curl: (23) Failed writing header

Arch Linux output:

[brandon@[masked hostname] ~]$ uname -a
Linux [masked hostname] 5.7.5-arch1-1 #1 SMP PREEMPT Mon, 22 Jun 2020 08:10:02 +0000 x86_64 GNU/Linux

[brandon@[masked hostname] ~]$ curl --version
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1g zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.41.0
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

[brandon@[masked hostname] ~]$ curl -vvv -OLsSf  --etag-save test.etag https://get.helm.sh/helm-v3.2.4-darwin-amd64.tar.gz
*   Trying 2606:2800:11f:1cb7:261b:1f9c:2074:3c:443...
* Connected to get.helm.sh (2606:2800:11f:1cb7:261b:1f9c:2074:3c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [88 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [155 bytes data]                                                                                                                                              [26/47]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2971 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Los Angeles; O=Verizon Digital Media Services, Inc.; OU=SecOps; CN=sni114fcgl.wpc.edgecastcdn.net
*  start date: May  8 00:00:00 2019 GMT
*  expire date: May 12 12:00:00 2021 GMT
*  subjectAltName: host "get.helm.sh" matched cert's "get.helm.sh"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55b2b261b8b0)
} [5 bytes data]
> GET /helm-v3.2.4-darwin-amd64.tar.gz HTTP/2
> Host: get.helm.sh
> user-agent: curl/7.70.0
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [233 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< accept-ranges: bytes
< content-md5: LntYfUTjxG3ILF3WkCeKzQ==
< content-type: application/x-tar
< date: Thu, 25 Jun 2020 15:40:59 GMT
< etag: 0x8D8117F40DD72AC
* Failed writing header
} [5 bytes data]
* stopped the pause stream!
* Connection #0 to host get.helm.sh left intact
curl: (23) Failed writing header

I expected the following

I would expect curl to succesfully download the file, and write the etag value to the specified file.

curl/libcurl version

Found in output above ^^

operating system

Found in output above ^^

Other

Happy to help troubleshoot further if needed.

The text was updated successfully, but these errors were encountered:

jay · 2020-06-26T06:17:34Z

You've silenced the warnings which is why you don't see:

< Etag: 0x8D8117F40DD72AC
Warning:
Received header etag is missing double quote/s

curl/src/tool_cb_hdr.c

Lines 115 to 128 in e9db32a

    
                 /* 
        
                  * if server side messed with the etag header and doesn't include 
        
                  * double quotes around the etag, kindly exit with a warning 
        
                  */ 
        
                 if(!first) { 
        
                   warnf(per->config->global, 
        
                         "\nReceived header etag is missing double quote/s\n"); 
        
                   return 1; 
        
                 } 
        
                 else { 
        
                   /* discard first double quote */ 
        
                   first++; 
        
                 }

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

/cc @s-3ntinel

- Return 'failure' on failure, to follow the existing style. - Put Warning: and the warning message on the same line. Ref: curl#5610

- Return 'failure' on failure, to follow the existing style. - Put Warning: and the warning message on the same line. Ref: #5610 Closes #5612

jay · 2020-06-26T13:52:55Z

I'm not sure why the warning message starts on a separate line, and also I think it should return failure for the fail conditions instead of return 1

I fixed that but I think we should change the behavior to not fail in this case and instead ignore the invalid etag (or ignore the double quotes if there's some precedent for it).

bagder · 2020-06-26T22:42:57Z

Yes, I too think it should rather ignore it when invalid (but probably say something in the verbose output).

sterchelen · 2020-06-27T09:48:18Z

To start playing with the code, can I take care of your proposal?

bagder · 2020-06-27T10:14:28Z

@sterchelen please go ahead, this could certainly be a good entry step into the code. This is entirely cmdline tool-side code. My advice: submit the first take on your PR early to solicit feedback on your approach!

bhartshorn · 2020-06-29T17:07:04Z

You've silenced the warnings which is why you don't see:

D'oh. I figured there was some PEBCAK at play here also. Thanks for looking at this.

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

sterchelen · 2020-06-29T18:33:30Z

Just for confirmation, an etag header is invalid without double-quotes according to this spec? https://tools.ietf.org/html/rfc7232#section-2.3

In fact, as the rfc specify:

An entity-tag consists of an opaque quoted string

Combine E-Tags, If-Modified-Since in one section. Also mention the curl --compression option for typically GZIP decompression. Note that E-Tags were broken in curl <7.73 due to a bug with "weak" e-tags. curl/curl#5610 From a question/feedback by e-mail from Hadrien Lacour, thanks.

jay added cmdline tool HTTP labels Jun 26, 2020

jay added a commit to jay/curl that referenced this issue Jun 26, 2020

tool_cb_hdr: Fix etag warning output and return code

31ae5bd

- Return 'failure' on failure, to follow the existing style. - Put Warning: and the warning message on the same line. Ref: curl#5610

jay mentioned this issue Jun 26, 2020

tool_cb_hdr: Fix etag warning output and return code #5612

Closed

jay added a commit that referenced this issue Jun 26, 2020

tool_cb_hdr: Fix etag warning output and return code

cb85275

- Return 'failure' on failure, to follow the existing style. - Put Warning: and the warning message on the same line. Ref: #5610 Closes #5612

sterchelen mentioned this issue Jun 29, 2020

tool_cb_hdr: Continue when etag is not btw double quotes #5626

Closed

bhartshorn mentioned this issue Jun 29, 2020

get.helm.sh returns invalid etag header helm/helm-www#735

Open

stokito mentioned this issue Aug 16, 2020

add suppport for weak etags #5700

Closed

bagder linked a pull request Aug 24, 2020 that will close this issue

etag: save and use the full received contents #5833

Closed

bagder closed this as completed in 98c9459 Aug 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--etag-save failing with error "(23) Failed writing header" #5610

--etag-save failing with error "(23) Failed writing header" #5610

bhartshorn commented Jun 25, 2020

jay commented Jun 26, 2020

jay commented Jun 26, 2020

bagder commented Jun 26, 2020

sterchelen commented Jun 27, 2020

bagder commented Jun 27, 2020

bhartshorn commented Jun 29, 2020

sterchelen commented Jun 29, 2020 •

edited

--etag-save failing with error "(23) Failed writing header" #5610

--etag-save failing with error "(23) Failed writing header" #5610

Comments

bhartshorn commented Jun 25, 2020

I did this

I expected the following

curl/libcurl version

operating system

Other

jay commented Jun 26, 2020

jay commented Jun 26, 2020

bagder commented Jun 26, 2020

sterchelen commented Jun 27, 2020

bagder commented Jun 27, 2020

bhartshorn commented Jun 29, 2020

sterchelen commented Jun 29, 2020 • edited

sterchelen commented Jun 29, 2020 •

edited