Unreliable/Flakey Capybara/AngularJS Integration T

问题:

How do I make these tests reliably pass?

Currently these tests are flakey.
Sometimes they pass. Sometimes they fail.
Below is the setup, code and output demonstrating this issue.
Suggestions to overcome this issue will be greatly appreciated and I am sure will help many others, so please comment!

Test Code Environment

1. Rails 3.2
2. RSpec 2.x
3. Capybara
4. Poltergeist
5. PhantomJS
6. AngularJS
7. Google Chrome Version 47.0.2526.106 (64-bit)

Testing Gems from Gemfile.lock

capybara (2.1.0)
database_cleaner (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-rspec (2.1.2)
jasminerice (0.0.10)
pg (0.17.1)
phantomjs (2.1.1.0)
poltergeist (1.4.1)
protractor-rails (0.0.17)
pry (0.9.12)
rack (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.3.20)
rspec-rails (2.11.4)
simplecov (0.8.2)
sprockets (2.2.3)
zeus (0.13.3)
zeus-parallel_tests (0.2.1)

Things I have tried

1. Ensure that I use Capybara's waiting DSL matchers
2. Ensure that my database cleaner is correctly setup
3. Test every page item assuming that it might not be on the page and could still be loading
4. Narrow down inconsistent tests
5. Run inconsistent test alone

6. Identify code Capybara DSLs that are the trigger to inconsistent test results.

   • i.e. creating a new record and assuming the page has redirected and that the record is on the page click_on

   or

   • .click not consistently 'working'
7. Upgrade Capybara to the latest version (in a seperate branch)
8. Upgraded Poltergeist and RSpec to the latest version (in a seperate branch, still working on this)

Resources I used

[1] Capybara The DSL
[2] Capybara, PhantomJs, Poltergeist, and Rspec Tips
And many more...

How tests were run

rspec spec/integration/costings/show_costing_spec.rb --format documentation

Test Code

show_costing_spec.rb

require "spec_helper"

RSpec.describe "Show a new costing in the listing," do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page).to have_css("#name", text: "test1")
    end
  end

  it "shows the details of the new costing after creation" do
    expect(page).to have_content("Costings")
    within "#costings_table" do
      expect(page).to have_content("test1")
      all("#show").last.click
    end

    expect(page).to have_content("Costing Details")
    expect(page).to have_css("#name", text: "test1")
  end
end  

spec_helper.rb

# This file is copied to spec/ when you run 'rails generate r spec:install'  
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")
require 'rspec/rails'
require 'rspec/autorun'

# Integration Testing
require 'capybara/poltergeist'
Capybara.register_driver :poltergeist_debug do |app|
 Capybara::Poltergeist::Driver.new(app, :inspector => true)  
end
Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Capybara Integration Test Helpers
def admin_sign_in
  visit "/login"
  #Create staff member in database
  Staff.make!(:admin)
  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"
end

def create_costing(item)
  visit "/api#/costings"
  click_on "new_btn"
  within "#form_costing" do
    find("#name", match: :first).set("#{item}")
    find("#description", match: :first).set("test description")    
    find("#from_date", match: :first).set("15/02/2016")
    find("#cost_hourly_cents", match: :first).set("1.00")
    click_on "create_btn"
  end
end

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
  end

  # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
  config.fixture_path = "#{::Rails.root}/spec/fixtures"

  # Allow a 'focus' tag so that we can run just a few tests which we are currently working on
  config.treat_symbols_as_metadata_keys_with_true_values = true
  config.filter_run focus: true
  config.run_all_when_everything_filtered = true
  config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

  # Defer Garbage Collection
  config.before(:all) { DeferredGarbageCollection.start }
  config.after(:all)  { DeferredGarbageCollection.reconsider }

  # If you're not using ActiveRecord, or you'd prefer not to run each of your
  # examples within a transaction, remove the following line or assign false
  # instead of true.
  config.use_transactional_fixtures = false
  # config.infer_spec_type_from_file_location!

  # Configure Database Cleaner
  config.include Capybara::DSL
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end
end

Test Results

Test Run 1: Failing

Run options: include {:focus=>true} exclude {:slow=>true}

All examples were filtered out; ignoring {:focus=>true}

Show a new costing in the listing, shows the costing after creation shows the details of the new costing after creation (FAILED - 1)

Failures:

1) Show a new costing in the listing,
shows the details of the new costing after creation
Failure/Error: expect(page).to have_content("test1")
expected #has_content?("test1") to return true, got false
# ./spec/integration/costings/show_costing_spec.rb:20:in block (3 levels) in
# ./spec/integration/costings/show_costing_spec.rb:19:in block (2 levels) in

Finished in 5.46 seconds 2 examples, 1 failure

Test Run 2: Passing

Run options: include {:focus=>true} exclude {:slow=>true}

All examples were filtered out; ignoring {:focus=>true}

Show a new costing in the listing,
shows the costing after creation
shows the details of the new costing after creation

Finished in 3.57 seconds 2 examples, 0 failures

Update 1

Upgraded testing gems to the following versions:
capybara (2.6.2) from (2.1.0)
database_cleaner (1.5.1) from (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-spec (2.1.2)
jasminerice (0.0.10)
pg (0.17.1)
phantomjs (2.1.1.0)
poltergeist (1.9.0) from (1.4.1)
protractor-rails (0.0.17)
pry (0.10.3) from (0.9.12)
rack (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.4.9) from (1.3.20)
rspec-rails (3.4.2) from (2.11.4)
simplecov (0.8.2)
sprockets (2.2.3)
zeus (0.13.3)
zeus-parallel_tests (0.2.1)

Result 1

Unfortunately upgrading these gems did not seem to make a difference and my tests were still flakey.

Update 2

I implemented Tom Walpole's suggestions. Ensured that my admin_sign_in waits for sign_in to complete.

Also updated my database_cleaner setup as Tom suggested.

Result 2

For my stack these changes did not seem to have an effect.

Note: If one is not using AngularJS I feel these changes would have made a difference. So thank you Tom for your suggestions.

Update 3

I needed to get more information on what was happening during my test runs. There are suggestions on the net to log, use screen shot saving gems and the like but I did not feel like these would be the most time efficient. I wanted to specify where I wanted the test to pause at and view the contents of variables and form fields. In a browser would be ideal.

What I Used
I was using "save_and_open_page" and "print page.html" to debug.

What I Moved To
As I was running RSpec 3.4.2 I added a debug helper method:

rails_helper.rb

def debugit
  puts current_url
  require 'pry'
  binding.pry
end

Result 3

A URL would be printed in the console and the test would be paused. At this stage I would be able to navigate to the URL, login, navigate to the test page and view what the Capybara test had done.

This enabled me to identify that the source of my problems arose when the test was using capybara's fill_in DSL. In some test runs the fields would be populated correctly and the form would be submitted. In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted.

Update 4

I discovered that because I was using AngularJS input forms and tables, AngularJS required a tiny bit of time to bind to the input fields. If it was not allowed this time the input data would not be saved.

Capybara provides waiting methods such as "within" and "find". I used these but they did not help with the AngularJS binding time issue. I found ng-if could be used to create an if statement to wait for particular item that would signify the AngularJS bindings to the form fields complete.

So I used Capybara waiting methods to wait for the fields I wanted to fill and I used AngularJS' ng-if not to show the fields until they are ready.

Implementation
index.html.erb

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
  </table>
</div>

Result 4

Tests finally pass! However I have all these find methods with xpath ensuring specific and difficult to target items are waited on...

Update 5

Even though in my gemfile I was running the gem phantomJS version 2.1.1 my command line version was only 1.X. This proved to be significant.

I updated my command line phantomJS version to 2.1.1. At the same time I ensured all my input boxes, buttons, tables, headings had unique ids. I then was able to remove all find(:xpath) occurances without breaking the tests.

Result 5

This suite of tests now reliably pass all the time! Exactly what I wanted! Yes!

回答1:

The immediate thing that jumps out is that your admin_sign_in doesn't actually wait for the sign_in to complete. This means that your call to create_costing can occur without the session cookie having been set in your browser. The last line in your admin_sign_in method should be something like

expect(page).to have_text('You are signed in') # whatever message is shown upon sign in

or

expect(page).to have_current_path('/') # whatever path an admin is redirected to upon signing in

That will make sure the login has actually completed and therefore the session cookies have been set in your browser.

Also your database cleaner config should use an append_after block rather than after - see https://github.com/DatabaseCleaner/database_cleaner#rspec-with-capybara-example

回答2:

The Issue

Problems arose when the test was using capybara's fill_in DSL. In some test runs the fields would be populated correctly and the form would be submitted. In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted.

1. When filling in forms, ensure AngularJS bindings are complete and Capybara waiting methods are used

AngularJS' ng-if statements needed to be used not to show form fields until they are ready.
This needed to be done in conjunction with the use of Capybara waiting methods to ensure fill_in field are only submitted once form load has completed.

index.html.erb or equivalent:

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
   </table>
</div>

2. Updated the command line version of PhantomJS to the latest (2.1.1)

This seemed to enable tests to run without as many Capybara waiting methods in order to achieve reliable tests.

Updated Test Code
show_costing_spec.rb

require "rails_helper"

RSpec.describe "Show a new costing in the listing,", :type => :feature do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page.find("#code2")).to have_content("2")
      expect(page.find("#name2")).to have_content("test1")
    end
  end

  it "shows the details of the new costing after creation" do
    within "#costings_table" do
      click_on "show2"
    end

    expect(page.find("#page_title")).to have_content("Costing Details")
    expect(page.find("#code")).to have_content("2")
    expect(page.find("#name")).to have_content("test1") 
    expect(page.find("#description")).to have_content("test description")
  end
end

rails_helper.rb

# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)

# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")

require 'rspec/rails'
require 'devise'

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

    # Setup Devise before it is used in rails_helper
    config.include Devise::TestHelpers, :type => :controller
    Devise.stretches = 1 # Improves speed.
end

config.include Capybara::DSL, :type => :feature
  config.mock_with :rspec

# Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
config.fixture_path = "#{::Rails.root}/spec/fixtures"

# Allow a 'focus' tag so that we can run just a few tests which we are currently working on
config.filter_run focus: true
config.run_all_when_everything_filtered = true
config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

# Defer Garbage Collection
config.before(:all) { DeferredGarbageCollection.start }
config.after(:all)  { DeferredGarbageCollection.reconsider }

# Integration Testing
require 'capybara/rspec'
require 'capybara/poltergeist'

Capybara.register_driver :poltergeist_debug do |app|
  Capybara::Poltergeist::Driver.new(app, {:inspector => true, js_errors: false })  
end

Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Debugging tools
def debugit
  puts current_url
  require 'pry'
  binding.pry
end

# If you're not using ActiveRecord, or you'd prefer not to run each of your
# examples within a transaction, remove the following line or assign false
# instead of true.
config.use_transactional_fixtures = false

#Show Deprications As Errors with full backtracing
config.raise_errors_for_deprecations!

#rest of the file....
# Final part of Configure Database Cleaner

Capybara.default_max_wait_time = 5
config.use_transactional_fixtures = false

config.before(:suite) do
  if config.use_transactional_fixtures?
    raise(<<-MSG)
      Delete line `config.use_transactional_fixtures = true` from
      rails_helper.rb (or set it to false) to prevent uncommitted
      transactions being used in JavaScript-dependent specs. During 
      testing, the app-under-test that the browser driver connects to 
      uses a different database connection to the database connection 
      used by the spec. The app's database connection would not be 
      able to access uncommitted transaction data setup over the 
      spec's database connection.
     MSG
  end
  DatabaseCleaner.clean_with(:truncation)
end  

config.before(:each) do
  DatabaseCleaner.strategy = :transaction
end

config.before(:each, type: :feature) do
  # :rack_test driver's Rack app under test shares database connection
  # with the specs, so continue to use transaction strategy for speed.
  driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test

    if !driver_shares_db_connection_with_specs
      # Driver is probably for an external browser with an app
      # under test that does *not* share a database connection with the
      # specs, so use truncation strategy.
      DatabaseCleaner.strategy = :truncation
    end
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    DatabaseCleaner.clean
  end
end


def admin_sign_in
  visit "/login"

  #Create staff member in database
  Staff.make!(:admin)

  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"

  expect(page).to have_text('Logout')
end

def create_costing(item)
  @item = item
  visit "/api#/costings"

  expect(page).to have_selector("#new_btn")
  click_on "new_btn"

  expect(page).to have_text("New Costing")
  within "#form_costing" do
    fill_in "name", with: "#{@item}"
    fill_in "description", with: "test description"
    fill_in "from_date1", with: "15/02/2015" 
    fill_in "cost_hourly_cents1", with: "12.00"

    expect(page).to have_selector("#create_btn")
    click_on "create_btn"
  end
  expect(page.find("#page_title")).to have_content("Costings")
end