Hello,

I am scraping a website for all its products (1000 different pages), and
I've read somewhere that cURL reads 2000 pages / minute. That is NOT my
case!
It reads very slow compared to 2000 pages / minute.

Am I doing anything wrong? Can I optimize it so it does this faster?

Here is the Code:

<?php

// Hello, cURL =)
$ch = curl_init();

// Loop through a array with 1000 records
foreach ($result as $gamedata) {

        // Get URL source in to string $gamedata_src
        $url = 'http://www.foo.com' . $gamedata['url'];
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
        $gamedata_src = curl_exec ($ch);

        // This is the stuff I normally uses, and it is just about as
fast as cURL?!???
# $gamedata_src = file_get_contents('http://www.foo.com' .
$gamedata['url']);

        // Get the stuff we want
        preg_match('/<li id=\'tabMarked\'>(.*?)<\/a>/s', $gamedata_src,
$gamedata_currency);

        // Remove unwanted stuff
        $gamedata_currency = strip_tags($gamedata_currency[1]);
        $gamedata['currency'] = trim($gamedata_currency);

        // Print out results
        echo $gamedata['url'] . " = " . $gamedata['currency'] . "\n";
}

// Bye bye cURL
curl_close ($ch);

?>

Thanks for any help I can get!

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2006-03-30