Notes from my terminal

Scraping news.google.de

Ah, Google, endless source of joy and anger! Google's German version recently launched news.google.de, a news aggregator collecting headlines from online news sources.

I'd really like to know how they do it since they claim that it works without human intervention and it does an impressive job at recognizing common subjects and clustering them. But my guess is that Google won't publish any details about their technology. The best we can hope for is a joke, like in their "explanation" of PageRank.

Another thing Google doesn't provide us with is a RSS feed for the news. However, it's easy to roll your own. I did a version in Perl on the day news.google.de started, but it can also be done in a more exotic language.

#!/usr/bin/ruby
require 'net/http'
google_s = Net::HTTP.get('news.google.de', '/news/de/de/mainlite.html')
google_re = Regexp.new('<a class=y href="(.*?)">(.*?)<\/a>.*?<\/b><br>(.*?)<br>')
puts "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
puts "<rdf:RDF"
puts "  xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\""
puts "  xmlns:dc=\"http://purl.org/dc/elements/1.1/\""
puts "  xmlns:sy=\"http://purl.org/rss/1.0/modules/syndication/\""
puts "  xmlns=\"http://purl.org/rss/1.0/\">"
puts "<channel rdf:about=\"http://news.google.de/\">"
puts "<title>news.google.de</title>"
puts "<link>http://news.google.de/</link>"
puts "<description>Schlagzeilen von Google (auf deutsch)</description>"
puts "<dc:language>de-de</dc:language>"
puts "<dc:creator></dc:creator>"
puts "</channel>"
while google_m = google_re.match(google_s)
  puts "<item>"
  puts "  <link>http://news.google.de/#{google_m[1]}</link>"
  puts "  <title>#{google_m[2]}</title>"
  puts "  <description>#{google_m[3]}</description>"
  puts "</item>"
  google_s.sub!(google_re, '')
end
puts "</rdf:RDF>"