Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Honor 429 HTTP Code / Retry-After header #3794

Closed
john-hascall opened this issue Apr 19, 2019 · 9 comments
Closed

Honor 429 HTTP Code / Retry-After header #3794

john-hascall opened this issue Apr 19, 2019 · 9 comments
Labels
cmdline tool HTTP

Comments

@john-hascall
Copy link

The command line client's --retry option should recognize and honor the 429 HTTP status return and the accompanying 'Retry-After' header.

Affects all versions, but I happen to be running:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.28.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets

on:

Linux 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
@bagder
Copy link
Member

bagder commented Apr 19, 2019

I reasonable suggestion methinks, even if that's a very rarely used response code and header...

@john-hascall
Copy link
Author

john-hascall commented Apr 19, 2019 via email

@bagder
Copy link
Member

bagder commented May 3, 2019

I want to address this issue after #3804 has gone in, just to save me from having to do double work.

@mark-kubacki
Copy link
Contributor

At least one server written in Golang does sent this. It's used when it runs out of file descriptors.

Another sensible use is in anything that runs in the cloud, hits a resource limit (clouds are made of rigid boxes), and cold-start times are known.

@bagder
Copy link
Member

bagder commented Aug 6, 2019

It seems basically no browser supports it:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#Browser_compatibility

@john-hascall
Copy link
Author

john-hascall commented Aug 6, 2019 via email

@bagder bagder mentioned this issue Aug 6, 2019
Closed
bagder added a commit that referenced this issue Aug 6, 2019
@bagder
CURLINFO_RETRY_AFTER: parse the Retry-After header value
5075756 
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
bagder added a commit that referenced this issue Aug 6, 2019
@bagder
curl: make use of CURLINFO_RETRY_AFTER when retrying
9564c2d 
If a Retry-After: header was used in the response, that value overrides
other retry timing options.

Closes #3794
bagder added a commit that referenced this issue Aug 7, 2019
@bagder
CURLINFO_RETRY_AFTER: parse the Retry-After header value
f933449 
This is only the libcurl part that provides the information. There's no
user of the parsed value. This change includes three new tests for the
parser.

Ref: #3794
@bagder bagder closed this as completed in 640b973 Aug 7, 2019
@john-hascall
Copy link
Author

john-hascall commented Aug 8, 2019 via email

@stain
Copy link
Contributor

stain commented Oct 2, 2019 •
edited
Loading

Thanks @bagder - but when I now try to use this in 7.66.0 against for instance Zenodo API it seems curl is still fails early on 429 HTTP status code - is the retry-logic triggered for 429 just like the 5xx status codes?

@stain
Copy link
Contributor

stain commented Oct 2, 2019

Try for instance

    curl \
    --show-error \
    --fail-early \
    --retry \
    10 \
    --retry-connrefused \
    --location \
    --remote-name-all \
    --dump-header \
    - \
    --header \
    'Accept: application/ld+json' \
    https://zenodo.org/api/records/1299320 \
    https://zenodo.org/api/records/1299783 \
    https://zenodo.org/api/records/1484496 \
    https://zenodo.org/api/records/1308997 \
    https://zenodo.org/api/records/1309962 \
    https://zenodo.org/api/records/1484341 \
    https://zenodo.org/api/records/1310621 \
    https://zenodo.org/api/records/1442453 \
    https://zenodo.org/api/records/1484403 \
    https://zenodo.org/api/records/1421066 \
    https://zenodo.org/api/records/1312623 \
    https://zenodo.org/api/records/1311933 \
    https://zenodo.org/api/records/3343031 \
    https://zenodo.org/api/records/3334688 \
    https://zenodo.org/api/records/3380592 \
    https://zenodo.org/api/records/1484442 \
    https://zenodo.org/api/records/1314672 \
    https://zenodo.org/api/records/3458522 \
    https://zenodo.org/api/records/3445531 \
    https://zenodo.org/api/records/3357806 \
    https://zenodo.org/api/records/3362412 \
    https://zenodo.org/api/records/3458560 \
    https://zenodo.org/api/records/3268809 \
    https://zenodo.org/api/records/1313218 \
    https://zenodo.org/api/records/1312652 \
    https://zenodo.org/api/records/1472385 \
    https://zenodo.org/api/records/1323831 \
    https://zenodo.org/api/records/3336124 \
    https://zenodo.org/api/records/3336148 \
    https://zenodo.org/api/records/1208478 \
    https://zenodo.org/api/records/3338176 \
    https://zenodo.org/api/records/1410052 \
    https://zenodo.org/api/records/3384296 \
    https://zenodo.org/api/records/3270620 \
    https://zenodo.org/api/records/1484480 \
    https://zenodo.org/api/records/3196309 \
    https://zenodo.org/api/records/1484502 \
    https://zenodo.org/api/records/1310034 \
    https://zenodo.org/api/records/1410238 \
    https://zenodo.org/api/records/1408170 \
    https://zenodo.org/api/records/1484424 \
    https://zenodo.org/api/records/1326284 \
    https://zenodo.org/api/records/1473154 \
    https://zenodo.org/api/records/1312685 \
    https://zenodo.org/api/records/1420531 \
    https://zenodo.org/api/records/3459770 \
    https://zenodo.org/api/records/3382423 \
    https://zenodo.org/api/records/1215611 \
    https://zenodo.org/api/records/1311612 \
    https://zenodo.org/api/records/3382259

stain added a commit to stain/curl that referenced this issue Oct 4, 2019
@stain
Ensure HTTP 429 triggers --retry
f294160 
This completes curl#3794

Also make sure the new tests from curl#4195 are enabled
@stain stain mentioned this issue Oct 4, 2019
Closed
bagder pushed a commit that referenced this issue Oct 5, 2019
@stain @bagder
curl: ensure HTTP 429 triggers --retry
f7f0b00 
This completes #3794.

Also make sure the new tests from #4195 are enabled

Closes #4465
@stain stain mentioned this issue Oct 16, 2019
Closed
@lock lock bot locked as resolved and limited conversation to collaborators Jan 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
cmdline tool HTTP
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bagder @mark-kubacki @stain @john-hascall