curl-library
Re: http redirect problem....
Date: Thu, 5 Mar 2009 11:27:03 -0600
On Thu, Mar 5, 2009 at 12:47 AM, Dan Fandrich <dan_at_coneharvesters.com>wrote:
> On Thu, Mar 05, 2009 at 11:52:01AM +0530, Mohan gupta wrote:
> > I am new to libcurl and is using it to make a web crawler . The problem i
> faced
> > using it is that it is not following redirect properly .
> > I tried to download the page www.gmail.com and the returned file is
> >
> > <HTML><HEAD><meta http-equiv="content-type"
> content="text/html;charset=utf-7">
> > <TITLE>301 Moved</TITLE></HEAD><BODY>
> > <H1>301 Moved</H1>
> > The document has moved
> > here.^M
> > </BODY></HTML>^M
> >
> >
> > I did use curl_easy_setopt(easy_handle,CURLOPT_AUTOREFERER,1L); but it
> has not
> > helping.
>
> You want the CURLOPT_FOLLOWLOCATION option.
>
And even that only works if the page is accompanied by a Location header.
Curl won't follow redirects like these:
<meta http-equiv="refresh" content="0;url=http://some.other.webpage/">
<script> top.location=http://some.other.place </script>
I think I've also seen "top.href=xxx" & "window.location" in that last one,
but maybe not... And then there's the self-submitting form pages, which
submit themselves in different ways.
Ralph Mitchell
Received on 2009-03-05