Don't
Try Crawling Google News Comments
By David A. Utter
When AOL broke down its subscriber
walls, it looked like the end of walled
garden content models on the Internet.
They have been coming back in some ways,
Facebook being one less-restrictive
example.
Google News doesn't seem like the place
to find high walls and ivy creeping
up the bricks. TechCrunch thinks it
is, citing Techmeme's Gabe Rivera about
the policy for comments on Google News:
One thing that bugs me: they’re
now hosting original news content, yet
they prohibit other aggregators from
crawling it (per robots.txt restrictions
and TOS). Of course Google News relies
on the openness of other organizations
with original news content.
Cynthia Brumfield at IP Democracy showed
how determining news participants promises
a whole bunch of headaches for whoever
ends up doing the verification of comments
and commenters:...
Click to Read More