Hi,
I'm starting using curl library in my application that search a list of urls in google, and then download documents form this pages.

I have a problem with the google search: in some cases when I try to download the page with urls matching my search, I have a CURLcode.CURLE_OK as result form my easy.perform() but then the page downloaded has the following content:

        Request denied by WatchGuard HTTP proxy.
        Reason: request protocol version denied line='GET /search?as_ft=e&as_filetype=pdf&num=100&filter=0&hl=it&q=%2Bvolpe+costruzione navale+palo+nautico%29HTTP/1.1\\x0d\\x0a'
        Method: GET
        Host: 216.239.59.147
        Path: search?as_ft=e&as_filetype=pdf&num=100&filter=0&hl=it&q=%2Bvolpe+costruzione

I do not set any particular proxy for this download because I did no use proxy here.

This is my code (I do curl globalInit on my main):

        MemoryStream header_stream = new MemoryStream();
        Easy easy = new Easy();
        easy.SetOpt(CURLoption.CURLOPT_URL, url);
        easy.SetOpt(CURLoption.CURLOPT_TIMEOUT, timeOut * 1000);

        CURLcode code = easy.SetOpt(CURLoption.CURLOPT_HEADERFUNCTION, new Easy.HeaderFunction(OnHeaderData));
        easy.SetOpt(CURLoption.CURLOPT_HEADERDATA, header_stream);
        // Imposta la callback di scrittura del corpo della pagina
        easy.SetOpt(CURLoption.CURLOPT_WRITEFUNCTION, new Easy.WriteFunction(OnWriteData));
        MemoryStream data_stream = new MemoryStream();
        easy.SetOpt(CURLoption.CURLOPT_WRITEDATA, data_stream);

        // download: proceed!
        code = easy.Perform();

        if (code != CURLcode.CURLE_OK)
        {
          return "";
        }

        byte[] page_data = data_stream.GetBuffer();
        string page_data_string = Encoding.Default.GetString(page_data);

        easy.Cleanup();

Any suggestion? I then use curl to download each single page (when the google search run correctly) and I have no problems, so maybe I simply forgot some setting...

Thanks,
Antonia.
Received on 2007-03-01