Poltergeist/PhantomJS crashing in Gitlab CI
<h1>Background:</h1>

<p>My testing suite did not handle any JS functionality until recently. I started out
only on my machine with a selenium-webdriver javascript driver and firefox 47.0.1
with no issues. I then migrated over to <code>poltergeist</code> supporting <code>PhantomJS</code> as
my javascript driver. I had to make a few changes to some broken tests, but have
very much enjoyed the changeover and am 'sold' on PhantomJS/poltergeist.</p>

<h1>Expected Behavior:</h1>

<p>I anticipated the tests to pass on my Gitlab-ci just as they had on my machine
(hey, we can dream), with maybe a few configuration tweaks needed.</p>

<h1>Versions:</h1>

<p>Ruby - 2.3.1</p>

<p>Rails - 5.0.0.1</p>

<p>poltergeist - 1.12.0</p>

<p>rspec - 3.5.0</p>

<p>rspec-rails - 3.5.1</p>

<p>capybara - 2.11.0</p>

<p>cliver - 0.3.2</p>

<p>factory_girl_rails - 4.7.0</p>

<p>phantomjs - 2.1.1 (on both my machine and CI server)</p>

<h1>Capybara Config:</h1>

<pre class="lang-rb prettyprint-override"><code>require 'capybara/rspec'
require 'capybara/poltergeist'

Capybara.default_driver = :rack_test
Capybara.register_driver :poltergeist do |app|
   options = {
     :js_errors =&gt; false,
     :timeout =&gt; 360,
     :phantomjs_options =&gt; ['--load-images=no', '--ignore-ssl-errors=yes', '--disk-cache=false']
   }
   Capybara::Poltergeist::Driver.new(app, options)
 end
 Capybara.javascript_driver = :poltergeist
</code></pre>

<h1>DatabaseCleaner Config</h1>

<pre class="lang-rb prettyprint-override"><code>  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = Capybara.current_driver == :rack_test ? :transaction : :truncation
    DatabaseCleaner.clean
    DatabaseCleaner.start
  end


  config.after(:each) do |example|
    puts 'RAM USAGE: ' + `pmap #{Process.pid} | tail -1`[10,40].strip

    if defined?(page)
      Capybara.reset_sessions!
      page.driver.restart
    end
    DatabaseCleaner.clean
  end
</code></pre>

<h1>Actual Behavior/Backtraces:</h1>

<h2>1. <strong>Failing Tests, PhantomJS Crash</strong></h2>

<p>To get a full stacktrace on my failing tests before I ate up all of my memory I
used the rspec <code>--fail-fast</code> option to kick back results for me on my first error.
This is what I returned:</p>

<pre class="lang-rb prettyprint-override"><code> ArgumentError:
                The detector #&lt;struct Cliver::Detector command_arg=nil, version_pattern=nil&gt; failed to detect theversion of the executable at '/usr/bin/phantomjs'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:191:in `detect_version'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:87:in `block in installed_versions'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:214:in `block in find_executables'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:208:in `map'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:208:in `find_executables'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:86:in `each'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:86:in `installed_versions'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:108:in `each'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:108:in `detect!'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:97:in `detect'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver.rb:35:in `detect'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:47:in `initialize'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:14:in `new'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:14:in `start'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:44:in `client'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:25:in `browser'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:183:in `reset!'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara/session.rb:110:in `reset!'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `block in reset_sessions!'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `reverse_each'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `reset_sessions!'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara/rspec.rb:21:in `block (2 levels) in &lt;top (required)&gt;'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:357:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:507:in `block in run_owned_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `each'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `run_owned_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:593:in `block in run_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `each'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `run_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:463:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:503:in `run_after_example'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:269:in `block in run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:496:in `block in with_around_and_singleton_context_hooks'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:453:in `block in with_around_example_hooks'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:464:in `block in run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:604:in `block in run_around_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-rails-3.5.1/lib/rspec/rails/adapters.rb:127:in `block (2 levels) in &lt;module:MinitestLifecycleAdapter&gt;'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:375:in `execute_with'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:98:in `block in run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:88:in `loop'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:88:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:22:in `block (2 levels) in setup'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:375:in `execute_with'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:607:in `run_around_example_hooks_for'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:464:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:453:in `with_around_example_hooks'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:496:in `with_around_and_singleton_context_hooks'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:251:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:627:in `block in run_examples'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:623:in `map'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:623:in `run_examples'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:589:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `block (3 levels) in run_specs'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `map'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `block (2 levels) in run_specs'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1835:in `with_suite_hooks'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:112:in `block in run_specs'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/reporter.rb:77:in `report'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:111:in `run_specs'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:87:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/exe/rspec:4:in `&lt;top (required)&gt;'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `load'
              # /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `&lt;main&gt;'
              #
              #   Showing full backtrace because every line was filtered out.
              #   See docs for RSpec::Configuration#backtrace_exclusion_patterns and
              #   RSpec::Configuration#backtrace_inclusion_patterns for more information.
</code></pre>

<p>Also, in my <code>gitlab-ci.yml</code> file, if I call <code>phantomjs --version</code> in <code>before_script</code> I receive:</p>

<pre><code>QXcbConnection: Could not connect to display
PhantomJS has crashed. Please read the bug reporting guide at
&lt;http://phantomjs.org/bug-reporting.html&gt; and file a bug report.
</code></pre>

<p>Any help on this would be much appreciated. </p>

<h2>2. <strong>Memory Blowup</strong> <em>SOLVED: see update 2</em></h2>

<p>The tests would eat up the memory, even after we increased memory capacity on the
servers, and crash the test suite before it would finish. To track memory usage
in my RSpec configuration after each test I ran:</p>

<p><code>puts '</code>enter code here<code>RAM USAGE: ' +</code>pmap #{Process.pid} | tail -1<code>[10,40].strip</code></p>

<p>The memory would be minimally affected under the model tests, but grow rapidly
under the integration tests. Like so:</p>

<pre><code>Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 568220K
Model Test-
RAM USAGE: 568220K
Model Test-
RAM USAGE: 568360K
Model Test-
RAM USAGE: 568500K
Model Test-
RAM USAGE: 568652K
Model Test-
RAM USAGE: 568788K

First Feature Test:
RAM USAGE: 1100628K
</code></pre>

<p>And blowing up from there.</p>

<p>To handle this I have made sure to use appropriate DatabaseCleaner strategies,
also cycled through trying tools such as <code>Capybara.reset_sessions!</code>,
<code>Capybara.current_session.driver.quit</code>, <code>page.driver.quit</code>, <code>page.driver.restart</code>
with no real joy.</p>

<h1>UPDATE:</h1>

<p><em>5:06 PM UTC, 15/12/16</em></p>

<h2>1. Crashes</h2>

<p>Gitlab support has suggested investigating whether our packages and dependencies are resulting in a non-headless situation with PhantomJS. We are investigating this now.</p>

<h2>2. Memory</h2>

<p>I've run a memory tracker against a few different iterations here, and it seems to be simply enabling our <code>js: true</code> feature tests with a javascript driver to be eating up the memory,  going from 1G for about 1,750 tests to surpassing our 4G limit on 1,800-1,850 tests. This seems to occur, I believe, with either <code>selenium-webdriver</code> OR <code>poltergeist</code> as the js driver. </p>

<h1>UPDATE 2:</h1>

<p><em>11:45 PM UTC, 16/12/16</em></p>

<h2>1. Crashing</h2>

<p>Still awaiting proper installation of packages on our server. Will update and award answer if this is the issue. </p>

<h2>2. Memory</h2>

<p>On the branch where we were migrating over to the Javascript-driver enabled batch of tests, part of the configuration to speed up the tests implemented a DeferredGarbageCollector which had settings that gobbled up the memory. PhantomJS and Poltergeist are not leaking. <strong>SOLVED</strong></p>