mdshare Posted February 10, 2009 Posted February 10, 2009 We all know the importance of indexed pages at google. Yet games fail misserable at it due to lack of public pages and having a private ingame forum. Now I wonder if anyone allows to let google bot crawl the game itself, shouldn't be a problem as the google bot IPs are fixed ones. if not, here is a guess how it could be done ... didn't dig in it how to, just a thought At the main index page (login/register) having a check if IP = googlebot if yes then login as guest account, allowing to crawl almost the whole game, especially the ingame forums. Quote
Haunted Dawg Posted February 10, 2009 Posted February 10, 2009 Re: google and your game Can you provide us with some of the spider's IP's? Quote
mdshare Posted February 10, 2009 Author Posted February 10, 2009 Re: google and your game Verifying GooglebotPrint You can verify that a bot accessing your server really is Googlebot by using a reverse DNS lookup, verifying that the name is in the googlebot.com domain, and then doing a forward DNS lookup using that googlebot name. This is useful if you're concerned that spammers or other troublemakers are accessing your site while claiming to be Googlebot. For example: > host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. > host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1Google doesn't post a public list of IP addresses for webmasters to whitelist. This is because these IP address ranges can change, causing problems for any webmasters who have hard coded them. The best way to identify accesses by Googlebot is to use the user-agent (Googlebot). taken from http://www.google.com/support/webmaster ... swer=80553 so as they say check on user-agent as I made a mistake in my previous post that they DO change IPs Quote
Haunted Dawg Posted February 10, 2009 Posted February 10, 2009 Re: google and your game So to check weather they are a spider is best to check there host name? On my new game i am developing a different user base. ID 1 => Me ID 2 - 15 => NPC's ID 16 => Guest ID 17 => Bot How ever. The guest, and npc's are easy do able. The bot's i need to track down. I was wondering if you could send me or post here the spider's IP's and such. I have noticed on SMF 2.0 you can track down spider's and such. Maybe you could help me out and post that info? It's not like i can harm anyone with it since they are just bot's. The bot's name's goes something like this: Alexa => ia_archiver Ask => Teoma Gigablast => Gigabot Google => googlebot There is a long list which can be found by going like this on SMF Admin => Search Engines => Spiders Quote
mdshare Posted February 10, 2009 Author Posted February 10, 2009 Re: google and your game check on user agent ( http://www.user-agents.org/ ) and there is more to it you will need a guest/demo account also for visitors that they can login so removing forum posting/mail send from that account would be advisable why ? It is EXPRESSLY against Googles Terms & Conditions that you must NOT show content to 'spider' guests which you don't show to human guests. also known as Cloaking. http://www.google.com/support/webmaster ... swer=35769 Google would penalize you, as content that gets fetched needs to be available to everyone. So with a guest/demo account you would follow their guidelines. Quote
Haunted Dawg Posted February 10, 2009 Posted February 10, 2009 Re: google and your game Only thing a guest won't be allow to do is attack, train, get item's, and change password name etc. And if by user they want to show there email on there profile then it will not be shown to guests. Guest's wont also be able to send a mail, post on the forum's and post in the shoutbox. A normal thing a bot wouldn't do. Would they still penalize me for that? Quote
mdshare Posted February 10, 2009 Author Posted February 10, 2009 Re: google and your game btw about tracking the bots on SMF Results today 2009-02-10 Google 65535 2009-02-10 Alexa 1013 2009-02-10 Yahoo! 65535 2009-02-10 Google (AdSense) 65535 2009-02-10 Google (Mobile) 65535 2009-02-10 MSN 5622 2009-02-10 CE Network Crawler 65535 so if I would not have that part enabled CE would have +300k pageviews today due to the spiders as it would be seen as a pageview, by having it enabled it doesn't count them as pageviews seems a lot but not all gets indexed, from googlewebmaster tools Total URLs: 97817 Indexed URLs: 38452 Quote
CtrlFreq Posted February 10, 2009 Posted February 10, 2009 Re: google and your game Now I wonder if anyone allows to let google bot crawl the game itself, shouldn't be a problem as the google bot IPs are fixed ones. We don't allow their bots to crawl the game directly, but we've made our persistent forums, and a number of other pages viewable to the public, and we link to them from our sitemap.xml file. It's constructed so it always reports the current time as last updated to google each time they catalog our site. For a bonus, it also has the effect of providing a number of hot links into the site when we're searched for on google. <?PHP Header ( 'Content-type:text/xml' ); print('<?xml version="1.0" encoding="UTF-8"?>'); ?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>http://www.themobstergame.com/system/index</loc> <lastmod><?PHP print(date("Y-m-d") . "T" . date("H:i:s.0P")); ?></lastmod> <changefreq>always</changefreq> <priority>1.0</priority> </url> <url> <loc>http://www.themobstergame.com/system/forum</loc> <lastmod><?PHP print(date("Y-m-d") . "T" . date("H:i:s.0P")); ?></lastmod> <changefreq>always</changefreq> <priority>0.4</priority> </url> [other URL nodes with different priorities] </urlset> <?PHP die(); ?> Quote
yuriorlov Posted February 21, 2009 Posted February 21, 2009 Re: google and your game check on user agent ( http://www.user-agents.org/ ) and there is more to it you will need a guest/demo account also for visitors that they can login so removing forum posting/mail send from that account would be advisable why ? It is EXPRESSLY against Googles Terms & Conditions that you must NOT show content to 'spider' guests which you don't show to human guests. also known as Cloaking. http://www.google.com/support/webmaster ... swer=35769 Google would penalize you, as content that gets fetched needs to be available to everyone. So with a guest/demo account you would follow their guidelines. can someone explain to me how you get the bot to login to your demo account? Quote
shedh Posted February 21, 2009 Posted February 21, 2009 Re: google and your game easy, create an account for the google bot, create a script which will allow the google bots ip login automatically into your game and then it will go through your game Quote
POG1 Posted February 21, 2009 Posted February 21, 2009 Re: google and your game check on user agent ( http://www.user-agents.org/ ) and there is more to it you will need a guest/demo account also for visitors that they can login so removing forum posting/mail send from that account would be advisable why ? It is EXPRESSLY against Googles Terms & Conditions that you must NOT show content to 'spider' guests which you don't show to human guests. also known as Cloaking. http://www.google.com/support/webmaster ... swer=35769 Google would penalize you, as content that gets fetched needs to be available to everyone. So with a guest/demo account you would follow their guidelines. So maybe mailing and events would be of limits to bot accounts. This is a good idea though because with some games all the bots will see is a few words, images and a form. @shedh - I wouldn't say it is easy. Quote
shedh Posted February 22, 2009 Posted February 22, 2009 Re: google and your game lol true the concept is easy but the scripting i think will be hard don't know how the mccodes engine actually work so i wouldn't know Quote
yuriorlov Posted February 22, 2009 Posted February 22, 2009 Re: google and your game easy, create an account for the google bot, create a script which will allow the google bots ip login automatically into your game and then it will go through your game it says in the google page not to count on the ip address... Quote
Haunted Dawg Posted February 22, 2009 Posted February 22, 2009 Re: google and your game Well aparantly. Over 4 year's the ip has been the same. I don't think it's going to change. Just look at how many site's google has indexed that need's a login form ;) Quote
yuriorlov Posted February 22, 2009 Posted February 22, 2009 Re: google and your game ok the question remains how to get it to login... I use sessions (but that requires a cookie to hold the session id) so how if the bot cant accept cookies? Quote
Haunted Dawg Posted February 22, 2009 Posted February 22, 2009 Re: google and your game It accept's cookie's and session's. Quote
POG1 Posted February 22, 2009 Posted February 22, 2009 Re: google and your game Maybe something like this? $IP = $_SERVER['REMOTE_ADDR']; if($IP == '66.249.64.47' OR $IP == '66.249.66.129') { $_SESSION['id'] = 3; /* ID of the google bot. */ } Quote
Haunted Dawg Posted February 22, 2009 Posted February 22, 2009 Re: google and your game Pog, you have to add a redirection to the logged in page also. This is due to the fact that it will never go to a page like explore.php because there is no link to it. $IP = $_SERVER['REMOTE_ADDR']; if($IP == 'googlebot') { $_SESSION['userid'] = 3; $_SESSION['loggedin'] = 1; header("location: explore.php"); } Quote
POG1 Posted February 22, 2009 Posted February 22, 2009 Re: google and your game Yeah i thought about that but it will then not add the pages not in the game. You could have a link into the game as well though maybe :) Quote
Haunted Dawg Posted February 22, 2009 Posted February 22, 2009 Re: google and your game Yeah you could but only it must show if it is the google bot ;) Quote
Eternal Posted February 24, 2009 Posted February 24, 2009 Re: google and your game what about this $IP = $_SERVER['REMOTE_ADDR']; if($IP == '66.249.64.47' OR $IP == '66.249.66.129') {echo '[url="gbot.php?username=googlebot&password=whatever"]login[/url]'; } Then in gbot.php have something simulating a user login attempt like $IP = $_SERVER['REMOTE_ADDR']; if($IP == '66.249.64.47' OR $IP == '66.249.66.129') { $username = $_GET['username']; $password =$_GET['password']; $uq=$db->query("SELECT userid FROM users WHERE login_name='".$username."' AND `userpass`=md5('".$password."')"); } else { header("Location: login.php"); } // Other details saying a successfull login then instead of header(); //Cause if you put a re-direct bots will cancel the sites veiwing... // put [url="index.php"]index[/url] Since i herd google don't use javascript so it can't use a login page. That would be optimum for it to be included into your game. Then disable attacks and preferences from that user and not able to be attacked... Quote
POG1 Posted February 24, 2009 Posted February 24, 2009 Re: google and your game Since i herd google don't use javascript so it can't use a login page. That would be optimum for it to be included into your game. Then disable attacks and preferences from that user and not able to be attacked... you don't need JS to login :S Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.