The Jeffster Posted 29, November 2006 Report Share Posted 29, November 2006 Its not a cost issue, its a configuration issue as we keep saying, googlebot is eating up nearly 2-3gb of bandwidth every day or two .. we are trying different ways to get it to stop being so greedy with little luck yet, but we hope we can find a solution very soon, we are working hard to get it resolved. Thanks for your patience. ex Thanks heaps for the feedback Exstatic, that is an annoying issue I assume that you have a file called robot.txt in the root and it's set to index only the root of /forums ? If it keeps becoming a recurring issue push it to Google to solve, you can say it's costing money and it's definitely disruptive. Good luck, let me know if I can help EDIT: I had a look in the root and there is no robot.txt file in the root, here is a link to some information on how to format this file: http://www.robotstxt.org/wc/norobots.html Quote Link to comment Share on other sites More sharing options...
exstatic Posted 29, November 2006 Report Share Posted 29, November 2006 Jeff, We have a robots.txt file (http://www.ozrenaultsport.com/robots.txt) and google is reading it, and only browsing the pages we want it to.. but this still chews a whole heap of bandwidth. A few of the other phpbb admins who were having this problem basically said the only way to stop the little bugger was to ban him completely.. which is what I've had to do until I can get feedback from google on another way around it, i.e; custom crawl rates etc.. Cheers for your help ex Quote Link to comment Share on other sites More sharing options...
The Jeffster Posted 29, November 2006 Report Share Posted 29, November 2006 Jeff, We have a robots.txt file (http://www.ozrenaultsport.com/robots.txt) and google is reading it, and only browsing the pages we want it to.. but this still chews a whole heap of bandwidth. A few of the other phpbb admins who were having this problem basically said the only way to stop the little bugger was to ban him completely.. which is what I've had to do until I can get feedback from google on another way around it, i.e; custom crawl rates etc.. Cheers for your help ex No worries mate I must of typed in robot.txt instead of robots.txt Would suck to ban the googlebot all together as we won't get indexed. How about an XML feed to a specific directory, the robot is only allowed to browse that. The forum bizz rules pushes only new posts to this directory. Possibly have the directory cleared every 30 days so the XML is reduced in size. Just an idea, problem is writing the logic. I do know that we are looking at doing that for our flash sites to help with indexing Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.