Protractor Exits with code 0 even though tests are

Question

I am running protractor v4.0.9 via npm, and noticed that my build on Visual Studio Team Services does not fail even when tests are reported as failing. Looking into this further, I introduced an error so that tests would fail and then ran them locally. As the screenshot below shows, I have 17/17 failing tests, but the exit code is still 0, which I understand is a success. It even says chrome #01 passed, which I assume is wrong.

Here is my protractor config:

"use strict";
exports.config = {
    baseUrl: process.env.npm_package_config_baseUrl,
    framework: 'jasmine',
    capabilities: {
        browserName: 'chrome'
    },
    specs: ['test/*.spec.js'],
    rootElement: 'se-app',
    directConnect: true,
    ignoreUncaughtExceptions: false,
    onPrepare: function () {
        var globals = require('protractor');
        var browser = globals.browser;
        browser.ignoreSynchronization = true;
        browser.manage().window().maximize();
        var specReporter = require('jasmine-spec-reporter');

        // add jasmine spec reporter
        jasmine.getEnv().clearReporters();
        jasmine.getEnv().addReporter(new specReporter({
            displayStacktrace: false
        }));

        var reporters = require('jasmine-reporters');
        jasmine.getEnv().addReporter(new reporters.JUnitXmlReporter({
            savePath: 'junit/'
        }));
    },
};

I checked the documentation for the Protractor config but could not see any obvious settings that would cause this to happen. I even added in ignoreUncaughtExceptions:false (shown above) to be explicit but it made no difference.

Answer 1

Ok, I found the problem. By default, Jasmine includes the console Reporter and the CompletionReporter. The latter is what exits the process if the tests fail. By clearing the reporters, I was removing this reporter.

I could just remove that line, but I want to remove the console reporter so that the spec reporter and console reporter aren't both doing the same job.

So I had to add the completion reporter back in. I'm not sure if this is the "correct" way to do it, but it works. Here is my new onPrepare function:

onPrepare: function() {
    let globals = require('protractor');
    let reporters = require('jasmine-reporters');
    let CompletionReporter = require('jasmine/lib/reporters/completion_reporter');
    let SpecReporter = require('jasmine-spec-reporter');

    let browser: ProtractorBrowser = globals.browser;
    browser.ignoreSynchronization = true;
    browser.manage().window().maximize();

    let specReporter = new SpecReporter({
        displayStacktrace: false
    });

    let junitReporter = new reporters.JUnitXmlReporter({
        savePath: 'junit/'
    });

    let completionReporter = new CompletionReporter();
    completionReporter.onComplete((success: boolean) => {
        if (!success) {
            process.exit(1);
        }
    });

    // Now clear all existing reporters and add the ones I want in the order I want
    jasmine.getEnv().clearReporters();
    jasmine.getEnv().addReporter(specReporter);
    jasmine.getEnv().addReporter(junitReporter);
    jasmine.getEnv().addReporter(completionReporter);
}

Answer 2

In case you are using grunt-protractor-runner module to run protractor, make sure that the option keepAlive in Gruntfile.js, under protractor.options is set to false.

Answer 3

In case anyone runs into this on Jasmine 2+, Maloric's answer won't cover you. Instead, add the onComplete function to the protractor.conf itself:

...other conf...,
onComplete: function (success) {
    if (!success) {
      process.exit(1);
    }
}

For some reason, the default onComplete action appears to be empty, but this'll rectify it. If you already have other onComplete actions specified, just add a process.exit(code) call.