Selection Criteria of JavaScript Test Framework

When a frontend team starts building a testing infrastructure, there is always a debate about the choice of testing tools.

There are dozens of tools for testing web applications on JavaScript, TypeScript and Node.js.

• Ava — test runner;
• CodeceptJS — end-to-end testing framework;
• Cypress — end-to-end browser testing framework;
• Hermione — test runner for browser testing based on Mocha and WebdriverIO;
• Jasmine — testing framework;
• Jest — testing framework;
• Karma — test runner;
• Mocha — test runner;
• Nightwatch — end-to-end testing framework;
• Node Tap — a Test-Anything-Protocol library (mostly for any kind of unit tests);
• Playwright — end-to-end testing framework;
• Protractor — deprecated end-to-end test framework for Angular applications;
• Puppeteer — headless Chrome Node.js API (actually, it is not a testing tool);
• SelenideJS — browser testing framework;
• TestCafe — end-to-end browser testing framework;
• WebdriverIO — end-to-end testing framework.

Some of these tools can run out-of-the-box and consist of a test runner, assertion library and much more. Some of these can be used as building blocks, for instance, Mocha as a test runner + Puppeteer as an API for browser + Chai as an assertion library + Allure as a test reporter.

Anyway, there should be a list of criteria by which to make a choice in favor of a particular tool. The last time when I took part in a choosing a test framework on a new project, we were guided by the following requirements:

1. Do you need unit or integration/end-to-end testing?
2. Browser support (including mobile browsers);
3. Documentation;
4. Community (articles, video tutorials, stackoverflow answers);
5. Support by developers (reactions on GitHub issues) and frequency of updates;
6. Parallelisation (the power of test runner);
7. Retries (the power of test runner);
8. Skips and conditional skips (the power of test runner);
9. Assertions (the power of assertion library);
10. Reports (variety of console and HTML reports);
11. WebSocket support;
12. Selenium (WebDriver) support? DevTools Protocol support? Or both?
13. Working with tabs (for example Cypress does not support that);
14. Mocks/stubs;
15. Proxy HTTP requests/responses;
16. Support HTTP API requests;
17. Setting cookies, localStorage and user-agents;
18. Visual/screenshot testing;
19. Debug mode;
20. Headless and headed modes (for browsers);
21. Integration with IDE (for example Playwright can run test in VSCode);
22. Run on localhost;
23. Run in CI/CD systems;
24. Run a single test;
25. Filter tests (ability to grep a bunch of tests);
26. Logging the test execution process;
27. Making screenshots and videos in case of failure (for example Playwright can even record traces);
28. Distribution size (the number of dependencies in package.json);
29. Maintainability (and how quickly new team members will get used to the project);
30. Open Source and/or license.

In recent times (according to my own observations), the short list of test frameworks for browser testing usually looks like this:

• Cypress;
• Playwright;
• WebdriverIO.

And for unit or API testing:

• Jest;
• Mocha;
• And an additional HTTP request library for Node.js, like GOT.

After all, criteria and tools listed above can be expanded depending on the context of your project.

Moreover, the stated criteria can be used for another (non JS) testing stack.