Board index Public Relations Site Related Crawlers + OS info

Crawlers + OS info

Posts that are directly related to the site.

Post November 19th, 2003, 11:34 am

Posts: 1830
Points on hand: 3,960.00 Points
Location: London, United Kingdom

What the hell are the crawlers? what do they do? (sounds like something from the matrix)Active Guests : 33
CRAWLER SEARCH ENGINE SPIDER
INKTOMI cal 11/19/2003
08:40:52 AM 43 Minutes
CRAWLER SEARCH ENGINE SPIDER
INKTOMI detail 11/19/2003
09:10:27 AM 13 Minutes
Guest #1 Windows XP
Microsoft IE 6.x Forums 11/19/2003
08:27:29 AM 56 Minutes

And os and browser info? cool, but weird!

Post November 19th, 2003, 11:44 am

Posts: 4533
Points on hand: 3,318.00 Points
Location: Kettering, England / Northamptonshire, United Kingdom

im not exactly sure but i know there computer bots, and moniter the site for stuff. There mainly search engines bots from sites like google and the other mentioned here. Also to my knowledge we have quite abit of problem with them as they take up a hell of a lot of bandwidth. I geuss wws will fill you in more to the point

Post November 19th, 2003, 1:52 pm
Oscar User avatar
Founding Member
Founding Member

Posts: 14409
Points on hand: 11,989.60 Points
Bank: 187,052.60 Points
Location: California, USA

what the edge said, they index our website, though we have a problem with them that we get them on the site all day long not one, but we've had up to 60 of them stay here all day. I could simply write a robots.txt file to kick all of them off for good, but them I loose rankings on all search engines, which currently ranks us as #1 But the reason you can now see that info is because I have a couple of developers looking into a problem that I have that page that was caused by a fix of another on the same page, and to not give them admin access, I gave them guest access to view the problem as so.


Return to Site Related

 


  • Related topics
    Replies
    Views
    Last post