Selection Criteria of Test Framework

Selection Criteria of Test Framework

There are dozens of tools for testing web applications on JavaScript, TypeScript and Node.js.

  • Ava — test runner;
  • CodeceptJS — end-to-end testing framework;
  • Cypress — end-to-end browser testing framework;
  • Hermione — test runner for browser testing based on Mocha and WebdriverIO;
  • Jasmine — testing framework;
  • Jest — testing framework;
  • Karma — test runner;
  • Mocha — test runner;
  • Nightwatch — end-to-end testing framework;
  • Playwright — end-to-end testing framework;
  • Protractor — deprecated end-to-end test framework for Angular applications;
  • Puppeteer — headless Chrome Node.js API (actually, it is not a testing tool);
  • SelenideJS — browser testing framework;
  • TestCafe — end-to-end browser testing framework;
  • WebdriverIO — end-to-end testing framework.

Some of these tools can run out-of-the-box and consist of a test runner, assertion library and much more. Some of these can be used as building blocks, for instance, Mocha as a test runner + Puppeteer as an API for browser + Chai as an assertion library + Allure as a test reporter.

Anyway, there should be a list of criteria by which to make a choice in favor of a particular tool. The last time when I took part in a choosing a test framework on a new project, we were guided by the following requirements:

  1. Browser support (including mobile browsers);
  2. Documentation;
  3. Community (articles, video tutorials, stackoverflow answers);
  4. Support by developers (reactions on GitHub issues) and frequency of updates;
  5. Parallelisation (the power of test runner);
  6. Retries (the power of test runner);
  7. Skips and conditional skips (the power of test runner);
  8. Assertions (the power of assertion library);
  9. Reports (variety of console and HTML reports);
  10. WebSocket support;
  11. Selenium (WebDriver) support? DevTools Protocol support? Or both?
  12. Working with tabs (for example Cypress does not support that);
  13. Mocks (stubs);
  14. Proxy HTTP requests/responses;
  15. Request HTTP API inside UI tests;
  16. Setting cookies, localStorage and user-agents;
  17. Screenshot testing;
  18. Debug mode;
  19. Headless and headed modes;
  20. Integration with IDE (for example Playwright can run test in VSCode);
  21. Run in localhost;
  22. Run in CI/CD systems;
  23. Run a single test;
  24. Run only a bunch of tests (ability to grep);
  25. Logging the test execution process;
  26. Making screenshots and videos in case of failure (for example Playwright even can record traces);
  27. Distribution size (the number of dependencies in package.json);
  28. Open Source, license.

In recent times (according to my own observations), the short list of test frameworks for browser testing usually looks like this:

  • Cypress;
  • Playwright;
  • WebdriverIO.

And for unit or API testing:

  • Jest;
  • Mocha.

After all, criteria and tools listed above can be expanded depending on the context of your project.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store