xovibot

[wserra]

The board has been lagging somewhat over the last several days, at least for me. When that happened, it seemed as though frequently there were a large number of guests. Just now there were 43. About 25 of them were iterations of the xovibot crawler. Rude behavior. Moreover, most of them were actually shown as "replying to message".

WTF? Is it time to keep these obnoxious bots off the board? web?

[wserra]

And now - two minutes later - every one of them is gone. There are a handful of baidu spiders, but at least they're reading. Strange, no? Almost looks like a DOS.

[fortinbras]

I am going to wait for a translation into English.

[The Observer]

http://blocklistpro.com/content-scraper ... -gmbh.html

[webhick]

Rude little data aggregator. We had a similar problem with 80legs. I added it to the bot list. It usually clears up after they crawl the whole site.

Also, the whole "replying to post" thing is simply because it's following every stupid link. Rude.

[wserra]

Now we've got 30 or so Ahrefs. When you get a second, web, please throw them in the bot list as well. Better yet, if I can do it without the risk of FUBARing the board, tell me how.

Thanks.

[Philistine]

Look up robots.txt for your public-html (www) root directory...
You can define user-agents (bots) and directories to omit from scan by bots.

[webhick]

Look up robots.txt for your public-html (www) root directory...
You can define user-agents (bots) and directories to omit from scan by bots.

We only block malicious bots, like the screen scraper from a few years back. Others, we add to the phpbb bots list so it doesn't muddy the waters.