Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Better resolver callbacks.
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Daniel Stenberg via curl-library <curl-library_at_cool.haxx.se>
Date: Fri, 23 Oct 2020 15:59:44 +0200 (CEST)
On Fri, 23 Oct 2020, John Wiseman via curl-library wrote:
> It's the number of max concurrent dns queries. Without such limit, my
> crawler will easyily blow up the dns server of my home network with almost 1
> thousand threads doing dns query on just several names.
Ouch. Yes, we clearly need a way to ratelimit that. It could probably be made
to work similarly to how we limit the total number of concurrent connections
and connections to the same host name. Maybe set the maximum number of name
resolves per second to allow or something.
> I've set connection limits on multi handle, but seems it won't limit these
> dns queries.
It should do it somewhat, and only indirectly. If it reaches the maximum
number of connections, in total or per host name, it will not resolve the new
host name until that connection can be allowed to get setup. But if you have
short-lived connections or HTTP requests doing redirects etc, that might not
be a very affective rate limit.
Date: Fri, 23 Oct 2020 15:59:44 +0200 (CEST)
On Fri, 23 Oct 2020, John Wiseman via curl-library wrote:
> It's the number of max concurrent dns queries. Without such limit, my
> crawler will easyily blow up the dns server of my home network with almost 1
> thousand threads doing dns query on just several names.
Ouch. Yes, we clearly need a way to ratelimit that. It could probably be made
to work similarly to how we limit the total number of concurrent connections
and connections to the same host name. Maybe set the maximum number of name
resolves per second to allow or something.
> I've set connection limits on multi handle, but seems it won't limit these
> dns queries.
It should do it somewhat, and only indirectly. If it reaches the maximum
number of connections, in total or per host name, it will not resolve the new
host name until that connection can be allowed to get setup. But if you have
short-lived connections or HTTP requests doing redirects etc, that might not
be a very affective rate limit.
-- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://www.wolfssl.com/contact/ ------------------------------------------------------------------- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2020-10-23