Hello,

I am using cURL to grab data from a website. The data contains about
6,000 items
in a osCommerce webshop.

The webshop is Mogs.com and I have made a scraper application that grabs
ALL games,
servers and realms aswell as the items for each games.

Now here is my problem! If I do a scrape on ALL games (one by one)
through my application
using this code more or less:

> # $server_url: Contains the URL where items should be grabbed from
>
>
> $curl_url = $server_url;
> curl_setopt($curl_handle, CURLOPT_URL, $curl_url);
> curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
> curl_setopt($curl_handle, CURLOPT_TIMEOUT, 20);
> curl_setopt($curl_handle, CURLOPT_NOPROGRESS, TRUE);
> curl_setopt($curl_handle, CURLOPT_MAXCONNECTS, 1);
> curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 20);
> $itemdata_src = curl_exec ($curl_handle);

Okay so when I run this for each game on Mogs.com (automatically), cURL
always skips
the same game server. It returns nothing... It finds the game name,
server name but no items.

Though, If I do a scrape using cURL on only that game that gets messed
up when I try to scrape
all the games at once, it works fine! All items are found and peace
everywhere!!?!

Any ideas on what may could be wrong? It has to be something with cURL.
There are no bugs inside my code since it finds every items if i scrape
the games manually one by one.

Best regards,
Niklas

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2006-05-31