Ruby - Using Ruby to easily scrape/search/spider a web page for "things" - Part 1

Hello,

This is part 1 of the "Using Ruby to easily scrape/search/spider a web page for "things"" multi-part post.

This part contains the code necessary to scrape a web site for a specific thing. In the case of this example, a heading.

The other parts of this multi-part post will explain specific sections of the code and will demonstrate how to spider through a website by scraping for links.

The following code can be used to scrape/search/spider a web site for headings:

#!/sw/bin/ruby
require 'open-uri'
require 'pp'

spider_url = "http://robertpyke.com/"
pp "Looking up #{spider_url}"

# The parentheses mark what we want to capture
heading_pattern = /<h[0-9]>(.*?)<\/h[0-9]>/  

headings = []
open(spider_url) do |f|
  f.each do |line|
    matchdata = line.scan(heading_pattern)
    matchdata.each do |match|  # Each match, match is an array
      match.each do |string|   # Each string within a match
        headings << string     # Store the heading we found
      end
    end
  end
end
pp headings  # Print the headings we found

Note: It should be noted that the regex used to capture the heading is by no means perfect. It has been provided as a simple starting point for people wanting to scrape web pages. Both the limitations of this regex, and a more advanced regex example, will be provided in a later part of this multi-part blog post.