> What I don't get is why the newspapers don't shut Google down in their robots.txt...
That's because their real motives have nothing to do with what they are saying.
You can't assume that they want what they say they want because, of course, they are completely capable of making it happens -- as you say, with robots.txt. It's trivial and I can guarantee you that they're aware of that fact.
What they are probably angling for is some kind of legal settlement, either a lump sum or royalties for when their search results are shown, or Google uses their content to compute rank, or some kind of special treatment, a special relationship with Google.
To really remove themselves from Google would be shooting themselves in the foot, and doubtless they know it.
Or, like you said, they want Google to pay for a firehose. But why would Google pay for a firehose when the content is right there on their sites, which Google indexes anyway?
People, you've got to be more multi-layered in your thinking. I know nerds are known to be more literal than most, but you can't go through the world thinking that other people are like you. You have to compare people's actions, with what they say, and figure out what the gestalt means. And you have to question a person's motives for saying a thing.
Speaking, writing and publicizing are not just low-bandwidth means of transferring pure information.
Newspapers should stop them from indexing and offer up this API instead, at a price.
Look, a web site is a web site.
Facebook doesn't let Google in to index all of it's data, this created an opportunity to eventually sell access to index it.
Newspapers could do the same, stop them from indexing their sites and sell them access to this firehose api, just like _every_ other data-driven startup.
No one flinches when Yelp offers and API with specific usage conditions. Newspapers should do the exact same thing.
That's because their real motives have nothing to do with what they are saying.
You can't assume that they want what they say they want because, of course, they are completely capable of making it happens -- as you say, with robots.txt. It's trivial and I can guarantee you that they're aware of that fact.
What they are probably angling for is some kind of legal settlement, either a lump sum or royalties for when their search results are shown, or Google uses their content to compute rank, or some kind of special treatment, a special relationship with Google.
To really remove themselves from Google would be shooting themselves in the foot, and doubtless they know it.
Or, like you said, they want Google to pay for a firehose. But why would Google pay for a firehose when the content is right there on their sites, which Google indexes anyway?
People, you've got to be more multi-layered in your thinking. I know nerds are known to be more literal than most, but you can't go through the world thinking that other people are like you. You have to compare people's actions, with what they say, and figure out what the gestalt means. And you have to question a person's motives for saying a thing.
Speaking, writing and publicizing are not just low-bandwidth means of transferring pure information.
No human communication ever is.