Are You Being Crawled?
I've noticed that much of my web traffic has come from Google and MSN bots crawling my site - they count for almost fifty percent of my traffic!
So I found this interesting script that you can include on your site that will email you a notification everytime a googlebot crawls your site:
if(eregi("googlebot",$HTTP_USER_AGENT)) { if ($QUERY_STRING != "") {$url = "http://".$SERVER_NAME.$PHP_SELF.'?'.$QUERY_STRING;} else {$url = "http://".$SERVER_NAME.$PHP_SELF;} $today = date("F j, Y, g:i a"); mail("you@youremail.com", "Googlebot detected on http://$SERVER_NAME", "$today - Google crawled $url"); }
InterAction:
YourThoughts?
(Minutia)
This entry was written by Jesse on Tuesday, February 15, 2005 at 11:07 AM and appears in the Blogging chapter. The previous article was entitled, "Happy Valentine's Day!", and the next entry is called, "CBS (Crash, Burn, Sink)". Bookmark the permalink, save it to del.icio.us or Digg it.
GetUpdated
ElseWhere
AllChapters
- 50 articles in the chapter 0's and 1's
- 35 articles in the chapter America
- 54 articles in the chapter Blogging
- 34 articles in the chapter Ears
- 35 articles in the chapter Eyes
- 21 articles in the chapter Family Ties
- 18 articles in the chapter Friends
- 3 articles in the chapter Great-Grandmother
- 58 articles in the chapter Heart
- 4 articles in the chapter Intercessional
- 61 articles in the chapter Journeys
- 44 articles in the chapter Kicks and Giggles
- 47 articles in the chapter Mind
- 28 articles in the chapter Miscellany
- 16 articles in the chapter Movies
- 2 articles in the chapter Podcasts
- 30 articles in the chapter Retrospection
- 63 articles in the chapter Savior
- 2 articles in the chapter Tech Tips
- 32 articles in the chapter The Pen
- 6 articles in the chapter Wedding/Honeymoon
- 19 articles in the chapter World
BiteSize.journal
ReaderFavorites
- The Mosquito Tone Test
- The Girl And The Boy: An Engagement Story
- Would Jesus Have Cared About The Cake?
- 10 Signs of a Jesus Follower
- Erego Ergo Sum (I Pay, Therefore I Am)
- A Libertarian View of Pensacola Christian College
- Situational Ethics
- Rich White People
- Fundamentalism and the Gentleness of Wisdom
- 10 Tips For Creating Website Mockups In Photoshop
20 March 20051. JDG:
A good friend of mine made these comments:
1) This script would be trivial to abuse and fill your email box with false positives. See LWP::Simple.
2) I think you invite trouble by publishing code you did not write that has no firm license. ("This script is completely free for you to use and modify however you see fit, but if you make any cool changes, please share them with us :)" does not count really as a license.
3) Why in heavens name would you care when Google was actually indexing your pages, and aren't there easier ways to tell after it has happened than filling your in-box with Junk from yourself?
Points definitely to note.