Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Handling Cloudfare issues
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Fabian Keil via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 27 Sep 2023 07:22:36 +0200
Mac-Fly via curl-library <curl-library_at_lists.haxx.se> wrote on 2023-09-26 at 23:29:16:
> since Monday, many of my tiny monitoring checks (analyse web pages for
> changes or specific content) where I used curl just fine over _years_
> suddenly fail with HTTP 403 access denied. While investigating, I found
> out that these pages are all "protected" by Cloudfare. If I use a
> browser (Firefox) go to one of these webpages I see the message as in
> the image attached.
>
> It basically says:
> "[Homepage name] must check the security of your connection"
> And then, at the bottom a Cloudfare reference "Ray ID" is shown as well
> as a Cloudfare slogan.
>
> As an example to reproduce: You can use any project page on sourceforge.net
>
> To rant a little: I don't now whats wrong with the internet these days
> and why such checks are required at all. I am sure they break a lot of
> applications like mine! (Rant off.)
As a Tor user I'm used to Cloudflare breaking the web and even
added an entry to the ElectroBSD FAQ [0].
> Now, I found many reports and "solutions" that are related to that issue
> (like manipulating the header sent, setting cookies etc.) but they are
> _all_ not working in these cases.
Sometimes Clouflare expects the client to execute proprietary JavaScript
to solve captures or do some other proof-of-work "for security reasons",
so manipulating headers isn't always sufficient.
My understanding is that Cloudflare "customers" can configure the type
of "protection" they get and can, for example, disable captures etc.
for Tor users.
Unless I'm mistaken, Cloudflare "customers" frequently don't pay with
their money but with their private data and the data of the visitors
and I wouldn't be surprised if Cloudflare is using the collected
data for nefarious purposes but this is getting off-topic for this
list.
> I am sure I am not the only one and now I am searching here for answers
> because I believe many curl users are affected, too. Please help me! :-)
You could contact the owners of the website and request that they
instruct Cloudflare to allow requests with curl again, but I suspect
that many website owners don't want you to access their website with
curl anyway, so they may not do that willingly ...
In case of SourceForge you could also argue that SourceForge is
(or used to be) a free software site, so they shouldn't require
their visitors to execute proprietary JavaScript to access the
site.
I've used this argument in the past for other free-software-related
sites and sometimes it worked and the site owners even thanked
me as they were unaware of this issue.
Unfortunately SourceForge changed owners a couple of times in the
past and they "lost" some staff, so it's possible that you don't
get any response from a human ...
Fabian
[0] <https://www.fabiankeil.de/gehacktes/electrobsd/#cloudflare-garbage>
Date: Wed, 27 Sep 2023 07:22:36 +0200
Mac-Fly via curl-library <curl-library_at_lists.haxx.se> wrote on 2023-09-26 at 23:29:16:
> since Monday, many of my tiny monitoring checks (analyse web pages for
> changes or specific content) where I used curl just fine over _years_
> suddenly fail with HTTP 403 access denied. While investigating, I found
> out that these pages are all "protected" by Cloudfare. If I use a
> browser (Firefox) go to one of these webpages I see the message as in
> the image attached.
>
> It basically says:
> "[Homepage name] must check the security of your connection"
> And then, at the bottom a Cloudfare reference "Ray ID" is shown as well
> as a Cloudfare slogan.
>
> As an example to reproduce: You can use any project page on sourceforge.net
>
> To rant a little: I don't now whats wrong with the internet these days
> and why such checks are required at all. I am sure they break a lot of
> applications like mine! (Rant off.)
As a Tor user I'm used to Cloudflare breaking the web and even
added an entry to the ElectroBSD FAQ [0].
> Now, I found many reports and "solutions" that are related to that issue
> (like manipulating the header sent, setting cookies etc.) but they are
> _all_ not working in these cases.
Sometimes Clouflare expects the client to execute proprietary JavaScript
to solve captures or do some other proof-of-work "for security reasons",
so manipulating headers isn't always sufficient.
My understanding is that Cloudflare "customers" can configure the type
of "protection" they get and can, for example, disable captures etc.
for Tor users.
Unless I'm mistaken, Cloudflare "customers" frequently don't pay with
their money but with their private data and the data of the visitors
and I wouldn't be surprised if Cloudflare is using the collected
data for nefarious purposes but this is getting off-topic for this
list.
> I am sure I am not the only one and now I am searching here for answers
> because I believe many curl users are affected, too. Please help me! :-)
You could contact the owners of the website and request that they
instruct Cloudflare to allow requests with curl again, but I suspect
that many website owners don't want you to access their website with
curl anyway, so they may not do that willingly ...
In case of SourceForge you could also argue that SourceForge is
(or used to be) a free software site, so they shouldn't require
their visitors to execute proprietary JavaScript to access the
site.
I've used this argument in the past for other free-software-related
sites and sometimes it worked and the site owners even thanked
me as they were unaware of this issue.
Unfortunately SourceForge changed owners a couple of times in the
past and they "lost" some staff, so it's possible that you don't
get any response from a human ...
Fabian
[0] <https://www.fabiankeil.de/gehacktes/electrobsd/#cloudflare-garbage>
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-09-27