Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using --etag-compare with --output truncate existing output #5181

Closed
kwon-young opened this issue Apr 4, 2020 · 2 comments
Closed

Using --etag-compare with --output truncate existing output #5181

kwon-young opened this issue Apr 4, 2020 · 2 comments
Labels
cmdline tool

Comments

@kwon-young
Copy link
Contributor

I did this

$ curl --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                              
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...                                                                                                                                                  
* Connected to myurl.com (ip) port 80 (#0)                                                                                                                                                                                                     
> GET /test.ext HTTP/1.1                                                                                                                    
> Host: myurl.com                                                                                                                                                                                                                                    
> User-Agent: curl/7.69.1                                                                                                                                                                                                                                    
> Accept: */*                                                                                                                                                                                                                                                
>                                                                                                                                                                                                                                                            
* Mark bundle as not supporting multiuse                                                                                                                                                                                                                     
< HTTP/1.1 200 OK                                                                                                                                                                                                                                            < Date: Sat, 04 Apr 2020 00:21:28 GMT                                                                                                                                                                                                                        
< Content-Type: application/ext                                                                                                                                                                                                                             
< Content-Length: 3840872                                                                                                                                                                                                                                    
< Connection: keep-alive                                                                                                                                                                                                                                     
< Set-Cookie: __cfduid=d55f833dcabfc0685a31a6437ba68ad091585959688; expires=Mon, 04-May-20 00:21:28 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax                                                                                                   
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT                                                                                                                                                                                                               
< ETag: "530bff48-3a9b68"                                                                                                                                                                                                                                    
< Cache-Control: max-age=14400                                                                                                                                                                                                                               
< CF-Cache-Status: HIT                                                                                                                                                                                                                                       
< Age: 6791                                                                                                                                                                                                                                                  
< Accept-Ranges: bytes                                                                                                                                                                                                                                       
< Server: cloudflare                                                                                                                                                                                                                                         
< CF-RAY: 57e6de961b60ee4c-CDG                                                                                                                                                                                                                               
<                                                                                                                                                                                                                                                            
{ [964 bytes data]                                                                                                                                                                                                                                           
100 3750k  100 3750k    0     0  5138k      0 --:--:-- --:--:-- --:--:-- 5131k                                                                                                                                                                               
* Connection #0 to host myurl.com left intact

This download test.ext as expected.

If we try to update test.ext but only if the file changed on the server using --etag-compare and the server respond that the file has not been updated, test.ext is truncated:

$ curl --etag-compare test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                                                              
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                                                                
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...                                                                                                                                                  
* Connected to myurl.com (ip) port 80 (#0)                                                                                                                                                                                                     
> GET /test.ext HTTP/1.1                                                                                                                    
> Host: myurl.com                                                                                                                                                                                                                                    
> User-Agent: curl/7.69.1                                                                                                                                                                                                                                    
> Accept: */*                                                                                                                                                                                                                                                
> If-None-Match: "530bff48-3a9b68"                                                                                                                                                                                                                           
>                                                                                                                                                                                                                                                            
* Mark bundle as not supporting multiuse                                                                                                                                                                                                                     
< HTTP/1.1 304 Not Modified                                                                                                                                                                                                                                  
< Date: Sat, 04 Apr 2020 00:23:30 GMT                                                                                                                                                                                                                        < Connection: keep-alive                                                                                                                                                                                                                                     
< Set-Cookie: __cfduid=d582f0a38586003e65053440888ac938f1585959810; expires=Mon, 04-May-20 00:23:30 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax                                                                                                   
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT                                                                                                                                                                                                               
< ETag: "530bff48-3a9b68"                                                                                                                                                                                                                                    
< Cache-Control: max-age=14400                                                                                                                                                                                                                               
< CF-Cache-Status: HIT                                                                                                                                                                                                                                       
< Age: 6913                                                                                                                                                                                                                                                  
< Server: cloudflare                                                                                                                                                                                                                                         
< CF-RAY: 57e6e18f2f0da84f-CDG                                                                                                                                                                                                                               
<                                                                                                                                                                                                                                                              0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0                                                                                                                                                                               
* Connection #0 to host myurl.com left intact
$ file test.ext
test.ext: empty

curl/libcurl version

curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1e zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Linux edwin 5.5.13-1-MANJARO #1 SMP PREEMPT Wed Mar 25 17:14:28 UTC 2020 x86_64 GNU/Linux

Discussion

I would like to help resolve this issue, but I am having a hard writing a test case of this.

I would need the ability to specify existing files like test.etag and test.ext prior running the curl command.
Then assert that test.ext is still the same after running the curl command.

Is there some documentation on the language used for the test cases?
@bagder
Copy link
Member

bagder commented Apr 4, 2020

I would need the ability to specify existing files like test.etag and test.ext prior running the curl command.

You can create files for the test in <client><file> (where file can also have 1,2,3,4 suffixes I believe).

Then assert that test.ext is still the same after running the curl command.

<verify><file>, also with suffixes if you need more than one

Is there some documentation on the language used for the test cases?

Yes tests/FILEFORMAT but I will also recommend grepping existing test cases in case the language in the docs isn't clear!

@kwon-young kwon-young mentioned this issue Apr 4, 2020
Closed
bagder pushed a commit that referenced this issue Apr 4, 2020
@kwon-young @bagder
Do not write output file if server respond with http code 304
eb65e86 
Using the --etag-compare option, we can ask the server to either send
the file if it was updated or respond with a 304 response code to
signify that the file did not change.
This commit recognize this response code and avoid writing the output
file, which avoid truncating a previously downloaded file.

This fix issue #5181.
@kwon-young
Copy link
Contributor Author

I saw that you wrote the test case for this.

Thank you for your help.

@kwon-young kwon-young reopened this Apr 5, 2020
@bagder bagder closed this as completed in 54ecc11 Apr 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cmdline tool
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bagder @kwon-young