Yak Shaving Failing Integration Tests with React and Rails
This Thanksgiving weekend, I decided to a major refresh of the react-webpack-rails-tutorial, including:
- Update all Ruby and NPM libraries to the latest, including updating the linters to the latest and some big updates to the react-bootstrap API
- Get CI working on both Travis and Codeship.
- Yarn, Webpack V2, a react-native client, and Action Cable.
To save others the pain in getting the integration tests working again, here’s my tips. I’m assuming you’re using Capybara and Ruby on Rails.
Use the latest versions, so long as they are supported. Since I recently upgraded to Mac OS Sierra, I generally needed the latest versions. It’s worth noting that selenium-webdriver 3.x is not yet supported by Capybara.
Here’s a few general tips:
- Update your JavaScript code to clear out any warnings and errors from the console.
- When you have a problem try to narrow it down and make sure it’s reproducible. Does the failing tests pass on it’s own, without the test suite? Maybe there is some order dependency of your specs? One thing to definitely do is to use the random order option for rspec. The order is not totally random. After each run, you’ll see the “seed” and you can use that re-run the specs in exactly the same order. config.order = :random
- Consider using the rspec-retry gem. This will allow you to automatically retry any failing tests. If you do use this, when doing local development, you can set your retry count to 1, so you don’t see the same failure N times! export RSPEC_RETRY_RETRY_COUNT=1
- Try your best to tweak one thing at a time, such as a spec running configuration, and assess if that causes more or less failures. Yes, this is super tedious when tracking down failures on a CI environment.
- Take care that your setup of database cleaner matches up with the current docs.
- On the react-webpack-rails-tutorial, we’ve got both Travis and Codeship setups. You’re free to look at those and ask me any questions. The travis setup is in the .travis.yml file. You can see our Codeship setup at the bottom of this article. The only driver that seems to reliably pass every time is selenium. I had high hopes that poltergeist or selenium_chrome would pass, but the wouldn’t. Locally, on my Mac, all 3 main drivers pass.
- When testing locally, I prefer poltergeist because of how it will fail if there are any warnings or errors in the logs.
Configure Your Integration Tests to Support Multiple Drivers
If you test setup allows you to switch drivers via an environment variable, that can help identify if a test failure is due to a real issue, or an issue with one of the test drivers.
If you look at our spec/rails_helper.rb, you’ll see support for:
- PhantomJS via the poltergeist gem, with options for skipping animations and ignoring errors.
- Selenium, both Chrome and Firefox.
- Webkit.
Of these, my favorites are PhantomJS and Selenium Chrome. Webkit appears to poorly support the modern JavaScript and React in my React on Rails apps.
PhantomJS
Why It Rocks!
- PhantomJS fails a test for any error or warning in the JavaScript logs. Plus you can usually see log issue that caused the test to fail.
- PhantomJS, being headless, is fast, especially if animations are turned off. To turn off animations, see this file and this line.
Why It’s Frustrating!
PhantomJS seems riddled with some race conditions that cause it to crash. Some of the times that PhantomJS crashed, there was a real issue in the test. However, some to the times, there was no issue at all. I reported this crashing in this Poltergeist Github Issue. The issues seem especially related to screens with animations, using ReactCSSTransitionGroup and animations regarding opacity. The errors you’ll see include: Capybara::Poltergeist::DeadClient, Errno::EPIPE, and Errno::ECONNRESET.
Workaround to the Crashes
With much fiddling, I developed the code to switch from PhantomJS to Selenium Chrome when PhantomJS crashes. If you’ve got issues with PhantomJS crashing, try adding this file, poltergeist.rb, to your spec/support directory. Here’s a gist of this where you can comment.
# See in use here:
# https://github.com/shakacode/react-webpack-rails-tutorial/blob/master/spec/support/poltergeist.rb
# This file supports 2 strategies:
# 1. switch_to_selenium: switch drivers
# 2. restart_poltergeist
RESTART_PHANTOMJS = ENV["RESTART_PHANTOMJS"] &&
%w(TRUE YES).include?(ENV["RESTART_PHANTOMJS"].upcase)
puts "RESTART_PHANTOMJS = #{RESTART_PHANTOMJS}"
CAPYBARA_TIMEOUT_RETRIES = 5
# HACK: workaround for Capybara Poltergeist StatusFailErrors, simply retries
# based on https://gist.github.com/afn/c04ccfe71d648763b306
RSpec.configure do |config|
config.around(:each, type: :feature) do |ex|
example = RSpec.current_example
use_selenium = false
original_driver = Capybara.default_driver
CAPYBARA_TIMEOUT_RETRIES.times do
example.instance_variable_set("@exception", nil)
# Private method in rspec:
# rspec-core-3.5.4/lib/rspec/core/memoized_helpers.rb:139
__init_memoized
if use_selenium
puts "Switching to selenium from #{Capybara.current_driver}"
Capybara.current_driver = js_selenium_driver
Capybara.javascript_driver = js_selenium_driver
end
ex.run
example_ex = example.exception
break unless example_ex
is_multiple_exception = example_ex.is_a?(RSpec::Core::MultipleExceptionError)
break unless example_ex.is_a?(Capybara::Poltergeist::StatusFailError) ||
example_ex.is_a?(Capybara::Poltergeist::DeadClient) ||
is_multiple_exception
if is_multiple_exception
m_exceptions = example_ex.all_exceptions
idx = m_exceptions.find_index do |exception|
exception.is_a?(Capybara::Poltergeist::StatusFailError) ||
exception.is_a?(Capybara::Poltergeist::DeadClient) ||
exception.class < SystemCallError
end
break unless idx
end
puts "\n"
puts "=" * 80
puts "Exception caught! #{example_ex.ai}"
puts "when running example:\n #{example.full_description}"
puts " at #{example.location} with driver #{Capybara.current_driver}."
if RESTART_PHANTOMJS
PhantomJSRestart.call
else
use_selenium = true
end
puts "=" * 80
end
Capybara.current_driver = original_driver
Capybara.javascript_driver = original_driver
Capybara.use_default_driver
end
end
# Rather than using switching to use selenium, we could have restarted Phantomjs
module PhantomJSRestart
def self.call
puts "Restarting phantomjs: iterating through capybara sessions..."
session_pool = Capybara.send("session_pool")
session_pool.each do |mode, session|
msg = " => #{mode} -- "
driver = session.driver
if driver.is_a?(Capybara::Poltergeist::Driver)
msg += "restarting"
driver.restart
else
msg += "not poltergeist: #{driver.class}"
end
puts msg
end
end
end
Dependency Updates
You will want to ensure you have the current versions of both PhantomJS and the Chrome Driver.
Phantomjs
Update to 2.1.1 with:
brew update && brew upgrade phantomjsYour CI system should use these steps, from .travis.yml:
**before_install**:
- mkdir $PWD/travis-phantomjs
- curl -sSL https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2 -o $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
- tar -xvf $PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2 -C $PWD/travis-phantomjs
- export PATH=$PWD/travis-phantomjs/phantomjs-2.1.1-linux-x86_64/bin:$PATHOn CodeShip, you will need to set the PHANTOMJS_VERSION environment variable to 2.1.1.
Chrome Driver
If you’re using the Rubygem chromedriver-helper, then you should run the command chromedriver-update. You can run that command after you run bundle in your CI setup. However, you might have installed via brew, so you should update via brew update && brew upgrade chromedriver.
Codeship Setup
Setup Commands
export PHANTOMJS_HOST=”https://s3.amazonaws.com/codeship-packages"
export PHANTOMJS_VERSION=2.1.1
\curl -sSL https://raw.githubusercontent.com/codeship/scripts/master/packages/phantomjs.sh | bash -s
# We support all major ruby versions: 1.9.3, 2.0.0, 2.1.x, 2.2.x and JRuby
rvm use 2.3.1
bundle config build.nokogiri — use-system-libraries
bundle install
chromedriver-update
nvm install stable && nvm alias default stable
npm install npm@latest -g
npm install
cd client && npm run build:client && npm run build:server
cd ..
export RAILS_ENV=test
bundle exec rake db:schema:loadTest Commands
COVERALLS_REPO_TOKEN=YynC2SEiNITamhqXRCpzDzlmeuoVqdFgp DRIVER=selenium_chrome bundle exec rake
bundle exec rake lint
