Hello,

I'm developing a web crawler using the curl library. While fetching this
page:

http://travel.aol.com/travel-guide/Oistins+Fish+Festival-Barbados-events--Barbados:164:282944899

I noticed it did not return the full page (40617 bytes) as it does in a web
browser.

I believe the cause of the issue to be the Content Length returned by the
web server (8192 bytes). Obviously this is incorrect, but I have no control
over it.

I tried the --ignore-content-length flag and this did not help. I also need
to fetch from the PHP implementation where this flag doesn't (seem to)
exist.

Is there a way to force curl into downloading the entire page?

Thanks in advance,

Dave

-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2009-02-22