Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

url: Fix NO_PROXY env var to work properly with --proxy option. #1140

Closed

Conversation

eramoto
Copy link
Contributor

@eramoto eramoto commented Nov 25, 2016

The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.

@mention-bot
Copy link

@eramoto, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bagder, @dfandrich and @rousskov to be potential reviewers.

@bagder
Copy link
Member

bagder commented Nov 27, 2016

allows us to access non-proxied hosts even if using NO_PROXY env var with --proxy option

It isn't very clear to me exactly how this works (after this patch). Are you saying that the proxy given with --proxy now is used even if the host in the URL is mentioned in the NO_PROXY environment variable?

http_proxy, no_proxy, --noproxy and --proxy all interact here. Do you think you can clarify how in the docs, like in the --proxy or --noproxy section?

@notok
Copy link

notok commented Nov 28, 2016

@bagder eramoto is saying that NO_PROXY environment variable is ignored when used with --proxy option.

So, for example, before this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed through proxy ** NOT INTENDED **

after this patch

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://somewhere.example.com/
# -> Accessed through proxy

no_proxy=127.0.0.1,localhost,192.168.1.1 curl --proxy http://proxy.example.com:8080/  http://192.168.1.1/
# -> Accessed directly, as intended.

@eramoto please tell me if this is not what you meant.

@bagder
Copy link
Member

bagder commented Nov 29, 2016

That's a crystal clear explanation that I understand perfectly. @eramoto, do you think you could also add something to the docs in this vein that lets future users understand the priorities and what overrides what?

@eramoto
Copy link
Contributor Author

eramoto commented Dec 2, 2016

@bagder

I add the following descriptions to the curl man-page:

  • --noproxy option does not override NO_PROXY env var.
  • If specify both of --noproxy option and NO_PROXY env var, both are effective.
  • Since 7.52.0, NO_PROXY env var is effective even if specify --proxy option.

Cloud you confirm this commit.

@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.

Since 7.52.0, this einvironment variable disable the proxy even if specify
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spelling: "environment"

@@ -2312,6 +2317,16 @@ Sets the proxy server to use if no protocol-specific proxy is set.
.IP "NO_PROXY <comma-separated list of hosts>"
list of host names that shouldn't go through any proxy. If set to a asterisk
\&'*' only, it matches all hosts.

Since 7.52.0, this einvironment variable disable the proxy even if specify
\fI--porxy\fP option. That is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spelling:" --proxy"

Copy link
Member

@bagder bagder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just two spelling errors, apart from that I think it looks ready to merge!

@eramoto
Copy link
Contributor Author

eramoto commented Dec 5, 2016

Thank you for your reviews. @bagder

I fixed these typos.

@bagder
Copy link
Member

bagder commented Dec 9, 2016

Do we really need the environment variable for NOPROXY to override the command line option --noproxy? It feels like against how several other options works. We generally let command line options override environment variables so that if you know what you want to can specify that exactly with command line options and if you rather want to let the environment control you don't use the command line option.

@eramoto
Copy link
Contributor Author

eramoto commented Dec 14, 2016

Because it is better to change to the behavior that are similar to other options,
I now rewrite lib/url.c and docs/curl.1 to override the environment variable NO_PROXY
due to the command line option --noproxy.

Cloud you wait a few days. @bagder

@eramoto
Copy link
Contributor Author

eramoto commented Dec 22, 2016

I changed as following:

  • The command line option --noproxy overrides the environment variable NO_PROXY.
  • Rewrite this behavior into the individual option documentation page noproxy.d.

Cloud you check this commits. @bagder

@bagder
Copy link
Member

bagder commented Dec 27, 2016

I thought we agreed that the command line options should override the environment variables? Yet your man page edit says:

this environment variable disable the proxy even if specify
--proxy option. That is
NO_PROXY=direct.example.com curl -x http://proxy.example.com http://direct.example.com
accesses the target URL directly.

@notok
Copy link

notok commented Dec 28, 2016

@bagder
I think an option should override the corresponding env var, like --noproxy overrides NO_PROXY,
but not for different env var, like --proxy(-x) option does not override NO_PROXY.

@bagder
Copy link
Member

bagder commented Dec 28, 2016

Hm, yes I think you're right. Thanks.

@bagder
Copy link
Member

bagder commented Dec 28, 2016

@eramoto , Can you please rebase your patch set (and squash what you think is reasonable), and update the release version number mentioned in the docs where this change is included (7.52.2 is now the next possible release number) and I promise to do my best to merge it when done?

@bagder
Copy link
Member

bagder commented Dec 28, 2016

(basically, you can skip changing curl.1 completely, we'll generate that based on the cmdline-opts/* docs anyway)

The combination of --noproxy option and http_proxy env var works well
both for proxied hosts and non-proxied hosts.

However, when combining NO_PROXY env var with --proxy option,
non-proxied hosts are not reachable while proxied host is OK.

This patch allows us to access non-proxied hosts even if using NO_PROXY
env var with --proxy option.
If defined CURL_DISABLE_HTTP, detect_proxy() returned NULL. If not
defined CURL_DISABLE_HTTP, detect_proxy() checked noproxy list.

Thus refactor to set proxy to NULL instead of calling detect_proxy() if
define CURL_DISABLE_HTTP, and refactor to call detect_proxy() if not
define CURL_DISABLE_HTTP and the host is not in the noproxy list.
Under condition using http_proxy env var, noproxy list was the combination
of --noproxy option and NO_PROXY env var previously. Since this commit,
--noproxy option overrides NO_PROXY environment variable even if use
http_proxy env var.
@eramoto eramoto force-pushed the fix-noproxy-env-with-proxy-option branch from c211af4 to 59fa2ed Compare January 6, 2017 13:01
@eramoto
Copy link
Contributor Author

eramoto commented Jan 10, 2017

I rebased and squashed my patch set (, and updated the release version number in the docs).

@bagder
Could you please merge this new patch set?

@bagder bagder closed this in 2ac1942 Jan 13, 2017
@bagder
Copy link
Member

bagder commented Jan 13, 2017

Thanks a lot for your work and patience on this!

houglum added a commit to houglum/httplib2 that referenced this pull request May 25, 2017
This allows users to specify an override value for the noproxy list, in
addition to overriding the proxy url, in proxy_info_from_url(). If no
override is given, we should respect the values in the no_proxy/NO_PROXY
environment variables, if present. This behavior can be seen in other
tools like curl (see curl/curl#1140).

I would also tweak this for the python3 module, but its ProxyInfo class
doesn't even seem to have a bypass_hosts attribute.
galvez pushed a commit to httplib2/httplib2 that referenced this pull request Jun 21, 2017
This allows users to specify an override value for the noproxy list, in
addition to overriding the proxy url, in proxy_info_from_url(). If no
override is given, we should respect the values in the no_proxy/NO_PROXY
environment variables, if present. This behavior can be seen in other
tools like curl (see curl/curl#1140).

I would also tweak this for the python3 module, but its ProxyInfo class
doesn't even seem to have a bypass_hosts attribute.
@lock lock bot locked as resolved and limited conversation to collaborators Jan 19, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

Successfully merging this pull request may close these issues.

None yet

4 participants