Jake Posted June 1, 2007 Share Posted June 1, 2007 I've been browsing through my webstats and found that bots are searching for the page robots.txt in order to scan my site. I was on other sites like google/paypal, and say that they have robots.txt pages and i was wondering what mine should consist of, have any of you got this file that i could upload to my site. Quote Link to comment Share on other sites More sharing options...
Decepti0n Posted June 1, 2007 Share Posted June 1, 2007 Re: Robots.txt You dont really need it unless you dont want search engines to index some of your pages (like admin pages etc), but it doesn't really matter because you'd still need to log in to see it Quote Link to comment Share on other sites More sharing options...
waffles Posted June 27, 2007 Share Posted June 27, 2007 Re: Robots.txt It should basically consist of this: User-agent: ia_archiver Disallow: / Save it as robots.txt in your public_html fodler and it stops webcrawlers and robots from going through your site. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.