Just about everybody effective blocks (or throttles to near zero) all but the top 3 or so crawlers. For a popular commerce site a significant fraction (like 1/3rd) of the infrastructure capacity is just responding to crawler traffic.