View on GitHub

Wombat

Web crawler/scraper with an elegant DSL which extracts structured data from web pages

Download this project as a .zip file Download this project as a tar.gz file

Wombat

Gem Version CI Build Status Dependency Status Code Climate Coverage Status FOSSA Status

Web scraper with an elegant DSL that parses structured data from web pages.

Usage:

gem install wombat

Scraping a page:

The simplest way to use Wombat is by calling Wombat.crawl and passing it a block:

require 'wombat'

Wombat.crawl do
  base_url "https://www.github.com"
  path "/"

  headline xpath: "//h1"
  subheading css: "p.alt-lead"

  what_is({ css: ".one-fourth h4" }, :list)

  links do
    explore xpath: '/html/body/header/div/div/nav[1]/a[4]' do |e|
      e.gsub(/Explore/, "Love")
    end

    features css: '.nav-item-opensource'
    business css: '.nav-item-business'
  end
end
The code above is gonna return the following hash:
{
  "headline"=>"How people build software",
  "subheading"=>"Millions of developers use GitHub to build personal projects, support their businesses, and work together on open source technologies.",
  "what_is"=>[
    "For everything you build",
    "A better way to work",
    "Millions of projects",
    "One platform, from start to finish"
  ],
  "links"=>{
    "explore"=>"Love",
    "features"=>"Open source",
    "business"=>"Business"
  }
}

Wiki

API Documentation

Changelog

Contributing to Wombat

Contributors

Copyright (c) 2012 Felipe Lima. See LICENSE.txt for further details.

License

FOSSA Status