Test Pyramid

The software test pyramid comes up a lot in Agile testing circles. In particular a common problem is that teams conflate the concepts of end-to-end tests, UI tests, and customer facing tests like acceptance, system and integration tests. These are all orthogonal characteristics. The unit tests and integration tests which use API’s are integrated and automated. GUI-based tools are used for creating the system and acceptance level tests.

Test Pyramid:

Test Driven Development (TDD)

TDD is based on the test-first concept of Extreme Programming (XP) that encourages simple design with high level of confidence. The procedure of doing TDD is following:

1. Write a test
2. Run all tests
3. Write the implementation code
4. Run all tests
5. Refactor

Test Tools

Test tools can be divided into the following functionalities. Some provide us with only one functionality, and some provide us with a combination.

1. Test framework: Provide a testing structure (Mocha, Jasmine, Jest, Cucumber)
2. Provide a test environment (Karma)
3. Provide mocks, spies, and stubs (Sinon)
4. Provide an assertions functions (Chai)
5. First layer of smoke testing, use a Headless browser, WebKit (PhantomJS, Headless Chrome)
6. Multiple browsers to ensure cross-browser consistency (Selenium)
7. WebdriverIO is a nodejs implementation of the Selenium WebDriver API
8. Sauce Labs is a provider of cloud-base cross-browser testing.

WebKit is an open source browser engine for rendering web pages in Safari, Chrome, and Opera. Gecko is the Firefox browser engine.

Note that the line between 5 and 6 gets slightly blurred with the recent WebDriver support in the latest PhantomJS. It is now possible to quickly run the tests first using PhantomJS and then (assuming there is no serious error encountered) continue to execute the same tests thoroughly in a Selenium setup.

The rumor is that the new headless mode of Chrome will replace PhantomJS.

Test frameworks and Test runners

The following table summarizes the list of various test frameworks and the corresponding test runners. If the framework does not need an external/third-party runner, it is marked as “built-in”.

Testing structure

Testing structure refers to the organization of your tests. Tests are usually organized in BDD structure that supports BDD (behavior-driven development).

PhantomJS

For a lot of JavaScript code, you can test without needing a DOM, but there are times when your tests need to work with HTML elements. This is where PhantomJS comes into play.

PhantomJS can execute JavaScript just like a browser. It is suitable for general command-line based testing.

PhantomJS is is a headless WebKit with JavaScript API.

Webkit is the layout engine that Chrome, Safari, and a few other niche browsers use. So PhantomJS is a browser, but a headless browser.

PhantomJS is used to launch the tests via a suitable test runner. Itself is not a testing library, but many of the other popular testing libraries can run on top of PhantomJS. e.g. Jasmine and Mocha.

There isn't a good PhantomJS runner for Jasmine at this time, it's much easier to integrate Mocha and PhantomJS with mocha-phantomjs. For this reason, we use Mocha for DOM-related tests.

npm install -g mocha-phantomjs

Using PhantomJS with CI system such as Jenkins or TeamCity does not require special setup. Make sure PhantomJS is installed properly on the slave/build agent and it is ready to go.

Since PhantomJS is purely headless on Linux, the agent can run on an installation without any GUI. This means, a barebone Linux system without X11 is not a problem for PhantomJS. It makes it possible to spawn light build agents on Amazon EC2 or Heroku instances.

Replace PhantomJS with headless Chrome

This needs to be done in two places:

1. Update docker images to include headless Chrome pre-installed, remove PhantomJS from these images.
2. Update Karma config to utilize Chrome instead of PhantomJS (a simple change to karma-chrome-launcher).

Selenium and WebDriverIO with Mocha

Capturing screenshots is one of many tests that can be run with Selenium. Usually, the tests are run in sequence with a test runner. WebDriverIO has its own test runner, compatible with frameworks such as Mocha, Jasmine or Cucumber.

In order to run the Selenium scripts locally you first need to set up a standalone server:

1. Download Selenium server
2. Download and unpack ChromeDriver
3. Launch the standalone Selenium server:

    $ java -jar -Dwebdriver.chrome.driver=’/path/to/chromedriver’ selenium-server-standalone-3.0.1.jar
   

4. When the server is up and running, all that remains is running the tests:

    $ node selenium.js
   

   Once executed, the screenshot will be captured into the local directory.