Yeah ... I am refering the robots.txt and also some sites are using meta
tag for robots is it possible find this..

Daniel Stenberg wrote:

>On Tue, 25 Apr 2006, vivekanandan wrote:
>
>
>
>>how can we find a site which is using robots protocol or not..
>>
>>
>
>Are you referring to the robots.txt file?
>
>
>

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2006-04-25