Re: working around status code 429

From: Dirk Hünniger via curl-users <curl-users_at_lists.haxx.se>
Date: Wed, 24 Jan 2024 17:10:48 +0100

Hi Daniel,

I got a simplified test case:

calling

dirk_at_dirk-System-Product-Name:~/hs1$ curl --retry 30 -s -w "%{http_code}
%{url} %{filename_effective}\n" --compressed -K ./curlrun1 --parallel -L
| grep "429 "

result in

429
https://commons.wikimedia.org/wiki/File:1-Acylglycerin-3-phosphocholin.svg
/tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/601.4.html
429 https://commons.wikimedia.org/wiki/File:Glycerin-3-phosphocholin.svg
/tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/603.4.html
429 https://de.wikibooks.org/wiki/File:Phosphatidylglycerin.svg
/tmp/MediaWiki2LaTeXImages-3943f5a7379c6219/591.1.html

and more...

where curlrun1 is in 7zip file I attached

Yours Dirk

On 21.01.24 12:54, Daniel Stenberg wrote:
> On Sun, 21 Jan 2024, Dirk Hünniger via curl-users wrote:
>
>> I am downloading about 1000 URLs in parallel with a singe curl call.
>> Some of them return status code 429 and need to be retried.
>> Unfortunately curl does not seem to be able to retry them itself with
>> the --retry-all-errors --retry 30 options.
>
> curl *should* retry on 429 with just --retry. I believe we even have
> test cases verifying this...
>

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2024-01-24