Drunkmenworkhere.org just published the evaluation of an interesting experiment. Over the timespan of one year, they put a tree of two million files online in a sub folder of the homepage. After that year is over, the author has now evaluated the crawlers conduct graphically. Even though the findings do not necessarily coincide with my own experience, the article is still an interesting read.