Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: working around status code 429
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Dirk Hünniger via curl-users <curl-users_at_lists.haxx.se>
Date: Sun, 21 Jan 2024 13:26:17 +0100
Hi Daniel,
the problem I observed goes away when I try to observe it for too long.
I think that the server just caches the requests and does not respond
with 429 anymore, once everything is cached. I got the following test
case (see attached files). And this console log.
dirk_at_dirk-System-Product-Name:~/hs1$ ghc --make 1.hs
[1 of 1] Compiling Main ( 1.hs, 1.o )
Linking 1 ...
dirk_at_dirk-System-Product-Name:~/hs1$ ./1
Number of Failures 395
Number of Failures 371
Number of Failures 356
Number of Failures 341
^C
dirk_at_dirk-System-Product-Name:~/hs1$
I hope you can still reproduce the problem. Otherwise wait for 24 hours
and try again then.
Yours Dirk
On 21.01.24 12:54, Daniel Stenberg wrote:
> On Sun, 21 Jan 2024, Dirk Hünniger via curl-users wrote:
>
>> I am downloading about 1000 URLs in parallel with a singe curl call.
>> Some of them return status code 429 and need to be retried.
>> Unfortunately curl does not seem to be able to retry them itself with
>> the --retry-all-errors --retry 30 options.
>
> curl *should* retry on 429 with just --retry. I believe we even have
> test cases verifying this...
>
Date: Sun, 21 Jan 2024 13:26:17 +0100
Hi Daniel,
the problem I observed goes away when I try to observe it for too long.
I think that the server just caches the requests and does not respond
with 429 anymore, once everything is cached. I got the following test
case (see attached files). And this console log.
dirk_at_dirk-System-Product-Name:~/hs1$ ghc --make 1.hs
[1 of 1] Compiling Main ( 1.hs, 1.o )
Linking 1 ...
dirk_at_dirk-System-Product-Name:~/hs1$ ./1
Number of Failures 395
Number of Failures 371
Number of Failures 356
Number of Failures 341
^C
dirk_at_dirk-System-Product-Name:~/hs1$
I hope you can still reproduce the problem. Otherwise wait for 24 hours
and try again then.
Yours Dirk
On 21.01.24 12:54, Daniel Stenberg wrote:
> On Sun, 21 Jan 2024, Dirk Hünniger via curl-users wrote:
>
>> I am downloading about 1000 URLs in parallel with a singe curl call.
>> Some of them return status code 429 and need to be retried.
>> Unfortunately curl does not seem to be able to retry them itself with
>> the --retry-all-errors --retry 30 options.
>
> curl *should* retry on 429 with just --retry. I believe we even have
> test cases verifying this...
>
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-01-21