curl / Mailing Lists / curl-users / Single Mail


Re: redirect URL

From: Ray Satiro via curl-users <>
Date: Tue, 21 Mar 2017 15:46:44 -0400

On 3/21/2017 1:59 PM, Claudio Piga wrote:
> Hi Nicholas, sorry if I reply this way, but I do not seem to be able
> to find a "reply to all" button in the mailing list page on the browser.
> However your command
> curl -sLvvvD- -o example.html "<url>"
> produces a file with some content ( a list of hotels in the right
> destination), but not the same content that is shown when the url is
> directly used on a browser, especially in terms of the dates indicated
> in the link, which I report here again for convenience:
> So somehow curl is not mimicking the browser environment.

Please do not top-post it makes the conversation harder to follow. That
website does not show the hotel listings when you use the curl user
agent. I can show the results using another user agent and setting the
language, like this:

curl -v -L -H "Accept-Language: en-US,en;q=0.8" -A "Mozilla/5.0
(compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)" -o out

What user agent you use can determine how much detail is returned, for
example IE9 user agent above booking doesn't return GPS coordinates. I
would ask them the best way to get information like that. You could
retrieve all 600k and parse the DOM but I suspect there's probably some
JSON you could tap into if they allow it.


Received on 2017-03-21