Headless Chrome Ruby - Conquering Automation with Ferrum

In the ever-evolving landscape of web automation, Ruby developers have long relied on the robust Selenium framework. But for those seeking a more streamlined, lightweight approach, a new contender has emerged: Ferrum.

Ferrum in a Nutshell

Ferrum, affectionately dubbed the Headless Chrome Ruby API, is a gem that allows Ruby programmers to interact with Chrome and Chromium browsers in a simple, efficient, and protocol-driven manner. Unlike Selenium, which relies on WebDriver and external drivers, Ferrum leverages the Chrome DevTools Protocol (CDP) directly, offering several advantages:

• Headless by Default: Ferrum prioritizes headless automation, minimizing resource consumption and making it ideal for server-side tasks and testing environments.

• Ruby-Centric API: Designed with Ruby developers in mind, Ferrum offers a clean, intuitive API that feels natural and familiar.

• Raw Power of CDP: By accessing the CDP directly, Ferrum unlocks the full potential of Chrome's capabilities, enabling control over various aspects like network, security, and performance.

• Lightweight and Fast: Ferrum boasts a minimal footprint, making it ideal for projects requiring efficient resource utilization.

Working with Ferrum

Installation

Add the below line to your Gemfile and execute the bundle install.

gem "ferrum"

> bundle install

If you are writing a script, execute the below command in your terminal to install the ferrum gem.

> gem install ferrum

Where ferrum can be used

Web Testing

Ferrum simplifies writing comprehensive tests for web applications by letting you control Chrome and interact with its elements programmatically. You can test user flows, verify pages, and validate functionalities. You can run tests without a visible browser window, enhancing efficiency and resource usage. This is particularly useful for CI/CD pipelines or large test suites.

While Ferrum directly controls Chrome, it can be used within browser testing frameworks like Capybara to achieve cross-browser compatibility with other Chromium-based browsers like Edge and Brave.

As shown in the screenshot below, let's say you have a simple Rails application with a HomeController rendering the index action.

You enter the name and click Submit. It redirects you to the HomeController show page.

Note:

To ensure you can access /home, add the below routes to the config/routes file.

Rails.application.routes.draw do
  get "home" => "home#index"
  get "submit" => "home#show"
end

You can test this page using the Ferrum gem by writing a system spec. For this, you must ensure the ferrum gem is installed in your Rails application.

1. Create a home_page_test.rb file under the test/system directory.

2. Add the code below to this file.

   require 'test_helper'
   
   class HomePageTest < ActionDispatch::SystemTestCase
     def setup
       @browser = Ferrum::Browser.new
     end
   
     def teardown
       @browser.quit
     end
   
     test "visiting the homepage and interacting with elements" do
        @page = @browser.create_page
        @page.goto "http://localhost:3000/home"
   
        # Find an element using a CSS selector
        header = @page.at_css('h1')
        assert_equal "Welcome to Rails!", header.text
   
        # Fill in a form and submit it
        input_field = @page.at_xpath("//*[@id='name']")
        input_field.focus.type("Sam Example")
        @page.at_xpath("/html/body/form/input[2]").click
   
        # Assert the response
        assert @page.body.include?("Welcome Sam Example")
     end
   end

3. Run the spec in your terminal as below and it should pass.

Note: When working with Ferrum, you need to start the rails server, while executing system specs.

> demo_ferrum % rails s

# on a new tab in the terminal run the system spec
> demo_ferrum % rails test test/system/home_page_test.rb
  Running 1 tests in a single process (parallelization threshold is 50)
  Run options: --seed 22870

  # Running:

  .

  Finished in 1.157573s, 0.8639 runs/s, 1.7278 assertions/s.
  1 runs, 2 assertions, 0 failures, 0 errors, 0 skips

When you execute the spec, a headless Chrome version is running. If you don't want to run the spec in the headless version, you need to pass headless: false to the new method as shown:

class HomePageTest < ActionDispatch::SystemTestCase
  def setup
    @browser = Ferrum::Browser.new(headless: false)
  end
end

Web Scraping and Crawling

Ferrum can navigate web pages, find specific elements, and programmatically extract text, images, or other data. This can be helpful for market research, competitor analysis, or building datasets.

You can add a rake task for website scraping in your Rails application. Let's say you want to scrape Books to Scrape website.

1. Create a file scraping.rake in the lib/tasks directory.

2. Add the following code to the file.

   # lib/tasks/scraping.rake
   # This file is used for scraping websites using Ferrum
   
   namespace :scraping do
     desc "Crawl the Books To Scrape website"
     task :book_to_scrape do
       browser = Ferrum::Browser.new
   
       begin
         page = browser.create_page
         page.go_to "https://books.toscrape.com/"
   
         # Extract book titles and prices
         book_elements = page.css("article.product_pod")
         books = book_elements.map do |book_element|
           title = book_element.at_css("h3 a").text
           price = book_element.at_css("p.price_color").text
           { title: title, price: price }
         end
   
         puts books
       ensure
         browser.quit
       end
     end  
   end

   Note:

   The .css function will return an Array of Node objects. The .at_css will return the first node that matches the CSS. Please go through the Finders section in the README of the gem. It has a comprehensive list of all the functions available on the browser and page objects.

3. Execute the rake task in your terminal: rake scraping:book_to_scrape. You might see the following output in your terminal.

   > demo_ferrum % rake scraping:book_to_scrape
    {:title => "A Light in the ...", :price => "£51.77"}
    {:title => "Tipping the Velvet", :price => "£53.74"}
    {:title => "Soumission", :price => "£50.10"}
    {:title => "Sharp Objects", :price => "£47.82"}
    {:title => "Sapiens: A Brief History ...", :price => "£54.23"}
    {:title => "The Requiem Red", :price => "£22.65"}
    {:title => "The Dirty Little Secrets ...", :price => "£33.34"}
    {:title => "The Coming Woman: A ...", :price => "£17.93"}
    {:title => "The Boys in the ...", :price => "£22.60"}
    {:title => "The Black Maria", :price => "£52.15"}
    {:title => "Starving Hearts (Triangular Trade ...", :price => "£13.99"}
    {:title => "Shakespeare's Sonnets", :price => "£20.66"}
    {:title => "Set Me Free", :price => "£17.46"}
    {:title => "Scott Pilgrim's Precious Little ...", :price => "£52.29"}
    {:title => "Rip it Up and ...", :price => "£35.02"}
    {:title => "Our Band Could Be ...", :price => "£57.25"}
    {:title => "Olio", :price => "£23.88"}
    {:title => "Mesaerion: The Best Science ...", :price => "£37.59"}
    {:title => "Libertarianism for Beginners", :price => "£51.33"}
    {:title => "It's Only the Himalayas", :price => "£45.17"}

Other Automation Tasks:

1. Automate repetitive tasks:

   Here's a detailed breakdown of how Ferrum empowers you to automate various tasks across different websites.

   • Filling Forms

     Imagine automatically filling online forms with pre-defined information, like customer details or shipping addresses. No more tedious typing, typos, or wasted time! Whether Chrome, Firefox, or Chromium-based, Ferrum adapts to handle forms across various platforms.

   • Submitting Requests

     Automate submitting data requests, feedback forms, or even registration forms on multiple websites with Ferrum and let the browser handle the tedious clicks and navigation.

     If you want to submit reports or forms at specific times or intervals, Ferrum can be integrated with scheduling tools to automate recurring tasks.

2. Create PDF reports from web pages:

   With Ferrum, you can generate PDF reports from web pages in several ways, depending on your desired level of control and complexity.

   • Using Chromium's built-in PDF printing

     This is the simplest method and requires minimal code. Use the pdf method on the Ferrum browser instance. You can create a rake task as below:

     # lib/tasks/pdf.rake
     
     desc "Convert to PDF using Ferrum"
     task :print_to_pdf do
       browser = Ferrum::Browser.new
       browser.go_to("https://books.toscrape.com/")
     
       pdf_data = browser.pdf(margin: 0)
     
       File.open('report.pdf', 'wb') { |file| file.write(pdf_data) }
     end

3. Control browser extensions:

   While Ferrum doesn't directly control extensions within the browser, you can preload the extensions using the :extensions option.

   browser = Ferrum::Browser.new(
     extensions: [
       "/path/to/extension1.crx", { source: "window.secret = 'top'" },
       "/path/to/extension2.crx"
     ]
   )

4. Screenshots:

   The Ferrum gem offers a comprehensive screenshot feature with various options to capture specific web page portions or viewports.

   Use the screenshot method on your Ferrum::Browser object:

   desc "Take screenshots"
   task :screenshots do
     browser = Ferrum::Browser.new
   
     begin
       page = browser.create_page
       page.go_to "https://books.toscrape.com/"
        
       # take screenshot of full page
       page.screenshot(path: "books.png", full: true)
     ensure
       browser.quit
     end   
   end

   This case is handy to debug the issue with flaky specs.

   You can pass a few more options to the screenshot method. A few of them are:

   1. path: Specify the path to save the screenshot as a file.

   2. encoding: Choose between :binary (default) or :base64 for returning the image data.

   3. format: Select the image format ("jpeg" or "png").

   4. quality: Set the JPEG quality (0-100).

   5. full: Capture the entire page (default is visible viewport).

   6. selector: Target a specific element within the viewport for capture.

   7. area: Define a custom area for the screenshot using coordinates (x, y, width, height).

   8. scale: Zoom in/out the page before capturing.

   9. background_color: Set a custom background color for the screenshot.

Why Choose Ferrum?

Sure, other gems handle automation, but Ferrum offers a unique blend of benefits:

• Simplicity: Its Ruby-flavored API feels natural and intuitive, making even complex tasks a breeze.

• Speed: Forget clunky overhead. Ferrum's direct CDP connection translates to blazing-fast execution, especially for repetitive tasks.

• Power: Unleash the full potential of Chrome DevTools. Ferrum grants access to features often hidden behind WebDriver layers, opening a world of possibilities.

• Flexibility: Need a hybrid of head full and headless? Ferrum can handle it. Want to customize the browser window size or user-agent? No problem. Its flexibility lets you bend automation to your will.

• Lightweight: Ditch the bulky dependencies and enjoy a gem with a minimal footprint, freeing up resources for your actual code.

Does Ferrum uses Capybara?

No, Ferrum does not use Capybara under the hood. While it works well with Capybara, they differ in their approach and goals:

Ferrum is designed as a low-level, pure-Ruby Chrome/Chromium driver and offers direct communication with Chrome DevTools Protocol (CDP) for fine-grained control and flexibility. It is primarily used for complex browser automation tasks, web scraping, performance profiling, and browser scripting.

Capybara acts as a higher-level, abstraction layer for browser testing. It supports multiple drivers, including WebDriver and Ferrum (through the Cuprite gem). It is focused on simulating user interactions and testing web applications in various scenarios.

Cuprite is a driver for Capybara that specifically uses Ferrum. It allows you to leverage Capybara's familiar API with Ferrum's underlying headless Chrome capabilities. This is ideal if you already use Capybara for testing and want to benefit from Ferrum's performance and flexibility.

Conclusion

Whether you're a seasoned automation veteran or a curious newbie, Ferrum deserves a spot in your Ruby toolbox. Its intuitive API, raw power, and flexible approach to headless Chrome make it a compelling choice for conquering any web automation challenge. So, saddle up, partner, and let Ferrum guide you through the wild west of web automation!

I have created a Rails application for the above steps. Please refer to this repository.