curl-and-php
Re: Robots protocol
From: Daniel Stenberg <daniel_at_haxx.se>
Date: Tue, 25 Apr 2006 08:31:47 +0200 (CEST)
Date: Tue, 25 Apr 2006 08:31:47 +0200 (CEST)
On Tue, 25 Apr 2006, vivekanandan wrote:
> Yeah ... I am refering the robots.txt and also some sites are using meta tag
> for robots is it possible find this..
robots.txt - just GET it and parse it
meta tags - parse the HTML you got
-- Commercial curl and libcurl Technical Support: http://haxx.se/curl.html _______________________________________________ http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-phpReceived on 2006-04-25