How do ensure that your application properly handles errors, especially when relying on third parties, such as payment processors? Is it easy to verify that the right things happen when the wrong things happen? Last week's article Strategies for Rails Logging and Error Handling discussed some techniques to setup a good error handling strategy. Here's some techniques to verify that your application does what you expect it to do when things go wrong. The key message is to check how your application handles errors, before your customers do.

Your Code Depends on Outside Systems (That Might Raise Errors)

Suppose you've created the super-duper Rails storefront application that takes online payments. You may even have some unit tests that verify the code. Then you get the dreaded call that customers are being charged twice and their orders are not processed. WTF?

It's not entirely obvious how to verify proper error handling when outside systems fail, or even when odd errors are raised from your own code. Payment processing deserves some special attention because it's a dependency on an outside service (the payment processor) and will typically require database updates based on the result of the payment processing. If you're updating several tables, then you'll want to use a transaction to ensure that all or nothing saves. While code review and manual testing are good first steps, you should consider a few extra steps with error handling for sensitive parts of your application.

Verification of Error Handling Strategy

Typically, error handling code is not well tested. It's much more common to test the "happy path" of everything going right.

Let's look at hypothetical example and some tests that can flush out some errors.

class Order
  def purchase_cart
    error_message = nil
    Order.transaction do
      # self.user record has charge info, and self.total is the order total
      # PaymentGateway.charge returns either error_message if failed or charge_details if success
      error_message, charge_details = PaymentGateway.charge user, total
      # update the order and the user records with the charge_details
      set_charge_fields_and_save user, charge_details unless error_message # update the order to indicated purchased
      fulfill_order # do lots of complicated stuff to fulfill the order
    end
    error_message # return any error message if there is one
  end
end

So what can go wrong?

Payment Processing is Like a 2-Phase Commit

Conceptually, you want a transaction, such that it's all or nothing. If the charge goes through, then so does everything else. Payment processing like a 2-phase commit, except one has to handle all the what-ifs to be sure that it's handled correctly.

The general steps of payment processing are like this:

1. Connect to outside resource to make charge.
2. Update database records indicating charge successful.
3. Fulfill the order.

Rails transactions work such than any exception in the block will cause the transaction to be rolled back. The problem with the above code is what happens if fulfill_order throws an exception? The customer has been charged, the order was updated to reflect payment, but then ka-boom and an exception is raised, and any database updates to the order are rolled back, but the payment is not refunded. The customer is confused as there is a charge but nothing else. How could you have tested (and avoided) this?

Brute Force Methodology

You can simulate error conditions by manually placing =raise "any error message"= statements in your code, and then testing, say in the UI manually. This is a good first step to verify that your error handling is working correctly. You might raise a specific error, if say your payment processor throws a specific type of error.

For the above example, the different methods referenced, such as process_order can get modified with a single line at the beginning, which would be:

def process_order
  raise "Any error message"
  # Lots of other code that can be commented out
end

Then go into the UI and test placing an order. Consider the following questions:

1. Was the right error message displayed to the user?
2. Was the right information logged at the correct log level?
3. Was an automatic email sent regarding the error?

See my prior article Saner Rails Logging for the answers to #2 and #3.

By applying this technique to each of the components of completing a purchase, one can flush out (and handle) nearly all of the different possible errors that could affect a purchase. Give this technique a try in some critical section of the code. You'll be surprised how well it works. Before giving you the fix to the above code, let's see if we can write unit and feature tests on our error handling.

RSpec Unit Testing of Errors

It turns out that with stubbing in rspec, it's easy to test error handling! RSpec provides a nice mocking library. The test code would look something like this. Pay attention to the call to stub.

describe Order do
  describe "#purchase_cart" do
    context "process_order fails" do
      let(:order) { create :order } # factory_girl creation of order and related objects
      before do
        # The magic stubbing of every instance
        Order.any_instance.stub(:fulfill_order) { raise ArgumentError, "test error" }
        # The call to purchase_cart will first call 'charge'
        PaymentGateway.should_receive(:charge).and_return([nil, "charge_details"])
        # The error from within purchase_cart should do a refund
        PaymentGateway.should_receive(:refund).and_return("refund_details")
      end
      it "should throw an error" do
        expect {
          order.purchase_cart
        }.to raise_error
        order.reload
        order.purchased.should_not be
        # charge refunded verified in mock
      end
    end

This test code ensures that the error handling of purchase_cart will catch an error from fulfill_order, and properly refund the payment and rollback any changes to the order record.

Here's an improved version of the Order#payment_method above:

class Order
  def purchase_cart
    error_message = nil
    begin
      Order.transaction do
        # user has a credit card info, returns either error_message if failed or charge_details if success
        error_message, charge_details = PaymentGateway.charge user, total
        set_charge_fields_and_save user, charge_details unless error_message # update the order to indicated purchased
      end
      fulfill_order # do lots of complicated stuff to process the order, do this outside of the original tx, so that the payment info can be committed.
    rescue => e
      Utility.log_exception e # Unified strategy for error handling including email notification, see below
      refund_charge if charge_details # If there's an error here, then sys admins will have to manually refund the charge.
      throw e
    end
    error_message # return any error message if there is one
  end
end

Here are the key points to the improved code:

1. There's a block to catch the exception which is separate from the transaction block. The rescue properly handles the case of an a charge being made and needing to be refunded. Utility.log_exception will ensure that all the right things happen with this sort of error (see code for Utlity.logException).
2. fulfill_order is moved outside of the transaction block. This allows the transaction to complete, and then the order_fulfillment takes place. If there's an issue in fulfilling the order, that can be dealt with separately from the original charge. In other words, the customer can successfully pay for the order, and the store can deal with the failure to fulfill the order.

RSpec Capybara Feature (Integration) Tests of UI Errors

It's possibly more important and sometimes easier to do the verification at the integration level in RSpec feature specs using Capybara with PhantomJs and Poltergeist. The secret sauce is the same use of the same stubbing technique as above to replace some key methods such that they throw an exception. This sort of technique works amazingly well to ensure that application will do the right then when an unexpected failure occurs, from the logging and emailing of the error message to the browser display to then end user.

I tend to develop such a test in an iterative manner:

1. Make sure you've got tests on the "happy" case where the story goes as planned.

2. Then introduce test cases where have bits of code like this that will raise an error at an opportune time.

   Order.any_instance.stub(:fulfill_order) { raise ArgumentError, "test error" }

3. Allow the test cases to fail, and put in screen shots (in Capybara with phantomjs, that looks like this:

   render_page "a-descriptive-name"

   Setup this method render_page in a spec helper file like this:

   def render_page name
     path = File.join Rails.application.config.integration_test_render_dir, "#{name}.png"
     page.driver.render(path)
   end

4. Put in some assertions that the page shows the correct error and the records in the database have the right values.

5. You can even

Here's an example that tests a failure of the Stripe payment API, including verification that an email was sent signifying an error:

 # using gem vcr to record http communication for faster performance
 let(:order) { create :order } # lots of setup in factory girl for non-purchased order

 scenario "Purchase cart, Strip payment error", :vcr do
   # Setup the stub -- the secret sauce to this test
   error_content = "Testing error handling exception message"
   PaymentGateway.stub(:charge) { raise Stripe::InvalidRequestError.new(error_content, 'id') }
   place_order
   page.should have_content error_content
   page.should have_content "Error purchasing"
   order.reload
   order.purchased.should_not be
 end

 def place_order
  login_as(user, :scope => :user)
  visit shopping_cart_path
  page.should have_selector('.total .price', :text => in_dollars(order.total))
  page.render_page("purchase-cart-1")
  click_link "CHECKOUT"
  fill_in_credit_card_info # utility test method to fill in credit card data
  page.should have_selector('.total .price', :text => in_dollars(order.total))
  render_page("purchase-cart-with-payment-info-2")
  click_on "PURCHASE"
  wait_for_spinners # method to wait for the busy spinner to stop
  render_page("purchase-cart-after-click-purchase-3")
  validate_error_emailed
end

# example of how you verify that an error was emailed
def validate_error_emailed
  email = ActionMailer::Base.deliveries.last
  email.should_not be_nil
  email.to.should_not include(order.user.email)
  email.to.should include('[email protected]')
end

Conclusion

If you aren't simulating how your application responds to errors, then you'll eventually find out, and the result might not be as good as you'd prefer. You can simulate errors with the very simple and quick technique of a well placed =raise "some error"=, and then testing in a UI. Or you might prefer the robustness of unit or feature tests using stubbing. Either way, the key message is to check how your application handles errors, before your customers do.

Related Post: Strategies for Rails Logging and Error Handling