cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: simple/weird question

From: bruce <badouglas_at_gmail.com>
Date: Mon, 30 Nov 2015 07:04:23 -0500

As an update..

Using the --trace-ascii option shows the actual/complete content,
which has the missing content, however the returned results has the
mix of the trace results and the actual text content.

So the issue, is what's really causing the weird behavior, I suspect
that there's something within curl that can be used to resolve the
'strange chars..

The following curl shows the missing/weird results for the section of
the content around the "select/option" area for the term/quarter..

curl -vvv -A "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0)
Gecko/20100101 Firefox/38.0" -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H
'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate'
 -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H
'Host: www.foothill.edu' --cookie-jar aa.lwp --cookie aa.lwp -L
"http://www.foothill.edu/schedule/schedule.php"

curl -vvv --trace-ascii - --cookie-jar aa.lwp --cookie aa.lwp
-L "http://www.foothill.edu/schedule/schedule.php"

        <strong>Quarter:</strong>
        <span class='indent'>
            <select nam<= Recv data, 1440 bytes (0x5a0)
0000: e="Quarter" id="Quarter">....<option value="" selected="selected
0040: ">Select Quarter</option>.<option value="2015F">Fall 2015</optio
0080: n>.<option value="2016W">Winter 2016</option>...</select> &nbsp;
00c0: <i class='small'>(optional)</i>...</span><br />...<div class='i
0100: ndent1'>...<i class='small'>Current quarter results will be disp
0140: layed if no quarter is selected....<br />...Future quarter listi
0180: ngs will appear when the scheduling office determines that they
01c0: are ready for release.</i>...<br /><br />...</div>....<strong>Co
0200: urse Number (CRN):</strong>...<span class='small'>....<input typ
0240: e="text" name="crn" id="crn" size="8" maxlength="6"/> <i class=
0280: 'small'>(optional) &nbsp;&nbsp;&nbsp;(Example: 21015)</i>...</sp
02c0: an>...<br /><br />..... <span class='indent1'><input type="submi
0300: t" name="search" value="Search" />... &nbsp;&nbsp;&nbsp;&nbsp;&n
0340: bsp;&nbsp;<input type="reset" value="Clear Form" />... &nbsp;&nb
0380: sp;&nbsp;&nbsp;<i class='small'>(Search Results limited to 300.)
03c0: </i>...</span><br /><br />....<div class='colhead2' style="backg
0400: round-color: #dbddc5; width:550px; padding:5px;">....<strong>Adv
0440: anced Search</strong>...</div>...<span class='small'>....<em>Sel
0480: ect a Department and one or more options from below for Advanced
04c0: Search.</em>...</span>....<table width='540'>....<tr>.....<td v
0500: align="top" colspan="2" style='line-height: 4px;'>&nbsp;</td>...
0540: .</tr>....<tr>.....<td valign="top">......Course Type:.....</td>
0580: .....<td valign="top">......<inp
e="Quarter" id="Quarter">
            <option value="" selected="selected">Select Quarter</option>
<option value="2</select> &nbsp; <i class='small'>(optional)</i>
        </span><br />
        <div class='indent1'>
        <i class='small'>Current quarter results will be displayed if
no quarter is selected.
        <br />
        Future quarter listings will appear when the scheduling office
determines that they are ready for release.</i>
        <br /><br />
        </div>

On Sun, Nov 29, 2015 at 9:21 PM, bruce <badouglas_at_gmail.com> wrote:
> Hi.
>
> Trying to fetch a simple page.
>
> The target is:
> http://www.foothill.edu/schedule/schedule.php
>
> In different browsers, without javascript or referer, results come
> back as expected.
>
> Using different curls settings, can't seem to get the complete
> returned content. The content seems to be missing the select for the
> terms!!
> Almost as though the returned data is corrupted.. But it happens
> consistently, regardless of trying different headers, etc..
>
> I've also tried wget, and get the same results.
>
> my test curl is:
> curl -vvv -k -A "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0)
> Gecko/20100101 Firefox/38.0" -H 'Accept:
> text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H
> 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate'
> --cookie-jar aa.lwp --cookie aa.lwp -L
> "http://www.foothill.edu/schedule/schedule.php"
>
>
> any pointers would be cool...
>
> thanks
>
>
>
> this should have a fall/winter term in the select/options..
>
> <strong>Quarter:</strong>
> <span class='indent'>
> <select name="Quarter" id="Quarter">
> <option value="" selected="selected">Select Quarter</option>
> <option value="2</select> &nbsp; <i class='small'>(optional)</i>
> </span><br />
> <div class='indent1'>
> <i class='small'>Current quarter results will be displayed if
> no quarter is selected.
> <br />
> Future quarter listings will appear when the scheduling office
> determines that they are ready for release.</i>
> <br /><br />
> </div>
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-11-30