curl / Mailing Lists / curl-users / Single Mail

curl-users

Re: Not able to replicate what a browser (javascript disabled) is doing

From: bruce via curl-users <curl-users_at_cool.haxx.se>
Date: Tue, 19 Dec 2017 17:34:51 -0500

Aha!

Hi Ray. Thanks for your input. Using the following, I was able to get
output similar to what the browser generated.

Now my issue is how to get the "Career" values. The target url in the
browser with javascript lists "Undergrad/Graduate" as the
select/option list. In inspecting the FF/Debug data for the
page/javascript I can't figure out where this is derived from. I can
easily see the terms/subjects from the api calls. The career data
(undergrad/graduate) isn't happening for me though!

curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp --compressed -H
'Connection: keep-alive' -H 'Host: mymobile.icc.edu' -H
'Upgrade-Insecure-Requests: 1' -L
'https://mymobile.icc.edu/app/catalog/classSearch/'

Thoughts/comments???

thanks!

On Tue, Dec 19, 2017 at 3:09 PM, Ray Satiro via curl-users
<curl-users_at_cool.haxx.se> wrote:
> On 12/19/2017 8:02 AM, bruce via curl-users wrote:
>
> Trying to test out a couple of sites that implement javascript. The
> test case for the browser - Opera is to run the browser/target site
> with javascript disabled. In this case the browser returns partial
> content.
>
> I copied/modified the cmd used by the browser using the "copy as Curl"
> function and have the following curl that I'm testing. However, I'm
> getting a situation where the process timesout, and doesn't return any
> data.
>
> Using " https://mymobile.icc.edu/app/catalog/classSearch/" as the
> target url works in different test browsers.
>
> The test curl follows:
>
> curl -v -A "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
> Firefox/38.0" --cookie-jar a.lwp --cookie a.lwp -H
> 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language:
> en-US,en;q=0.5' -H 'Connection: keep-alive' -H 'Host:
> mymobile.icc.edu' -H 'Upgrade-Insecure-Requests: 1' -L
> 'https://mymobile.icc.edu/app/catalog/classSearch/'
>
> At this point, I'm curious as to why the process hangs but the browser
> (with javascript disabled) doesn't.
>
>
> I can't reproduce this. I went as far back as 7.39.0 and tried a number of
> ssl backends including openssl, winssl, wolfssl and mbedtls. If you can
> still reproduce this please give your curl -V and the verbose output from
> the command so we can see where it hangs. Also, your use of the
> Accept-Encoding header will, if the response is compressed, prevent curl
> from automatically decompressing the response using the supported
> decompression methods. I suggest use --compressed [1] instead.
>
> [1]: https://curl.haxx.se/docs/manpage.html#--compressed
>
>
> -----------------------------------------------------------
> Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>
-----------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-users
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2017-12-19