Jump to content

Bandwidth Problem


ClioF1

Recommended Posts

Its not a cost issue, its a configuration issue as we keep saying, googlebot is eating up nearly 2-3gb of bandwidth every day or two .. we are trying different ways to get it to stop being so greedy with little luck yet, but we hope we can find a solution very soon, we are working hard to get it resolved.

 

Thanks for your patience.

ex

 

Thanks heaps for the feedback Exstatic, that is an annoying issue I assume that you have a file called robot.txt in the root and it's set to index only the root of /forums ?

 

If it keeps becoming a recurring issue push it to Google to solve, you can say it's costing money and it's definitely disruptive.

 

Good luck, let me know if I can help :P

 

EDIT:

 

I had a look in the root and there is no robot.txt file in the root, here is a link to some information on how to format this file:

 

http://www.robotstxt.org/wc/norobots.html

Link to comment
Share on other sites

Jeff,

 

We have a robots.txt file (http://www.ozrenaultsport.com/robots.txt) and google is reading it, and only browsing the pages we want it to.. but this still chews a whole heap of bandwidth.

 

A few of the other phpbb admins who were having this problem basically said the only way to stop the little bugger was to ban him completely.. which is what I've had to do until I can get feedback from google on another way around it, i.e; custom crawl rates etc..

 

Cheers for your help

ex

Link to comment
Share on other sites

Jeff,

 

We have a robots.txt file (http://www.ozrenaultsport.com/robots.txt) and google is reading it, and only browsing the pages we want it to.. but this still chews a whole heap of bandwidth.

 

A few of the other phpbb admins who were having this problem basically said the only way to stop the little bugger was to ban him completely.. which is what I've had to do until I can get feedback from google on another way around it, i.e; custom crawl rates etc..

 

Cheers for your help

ex

 

No worries mate I must of typed in robot.txt instead of robots.txt 8)

 

Would suck to ban the googlebot all together as we won't get indexed.

 

How about an XML feed to a specific directory, the robot is only allowed to browse that. The forum bizz rules pushes only new posts to this directory. Possibly have the directory cleared every 30 days so the XML is reduced in size.

 

Just an idea, problem is writing the logic. I do know that we are looking at doing that for our flash sites to help with indexing

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...