Searching...

phpnut

I have a few pages that a user clicks on that adds a 1 to a database and then is redirected back to ther previous page. Google and Yahoo keep spidering these pages (i know from the IPs) and I am getting numerous false rows in my databases. How can I stop google and yahoo from doing this?

Nuvo

You should be able to do this with the robots.txt file for your site, if you're using one.
Something like:

# All robots should use these rules.
User-agent: *

# The site is left without crawling.
Disallow: /

# You shalt not index my CGI!
Disallow: /cgi-bin/

# This is my secret directory, don't look here!
Disallow: /mysecretdir/

#Leave /tharg/donotrun/, /tharg/donotrun.php and any other donotrun's alone please.
Disallow: /tharg/donotrun

Should do it.

Also, to prevent indexing and link locating in pages that the bot does manage to find, you can use:

<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />

xPureNLx

Very interesting reply Nuvo. Thank you for explaining

- xPureNLx