Selection Criteria of JavaScript Test Framework

When a frontend team starts building a testing infrastructure, there is always a debate about the choice of testing tools.

There are dozens of tools for testing web applications on JavaScript, TypeScript, and Node.js.

• Ava — test runner;
• CodeceptJS — end-to-end testing framework;
• Cypress — end-to-end browser testing framework;
• Hermione — test runner for browser testing based on Mocha and WebdriverIO;
• Jasmine — testing framework;
• Jest — testing framework;
• Karma — test runner;
• Mocha — test runner;
• Nightwatch — end-to-end testing framework;
• Node Tap — a Test-Anything-Protocol library (mostly for any kind of unit tests);
• Playwright — end-to-end testing framework;
• Protractor — deprecated end-to-end test framework for Angular applications;
• Puppeteer — headless Chrome Node.js API (actually, it is not a testing tool);
• SelenideJS — browser testing framework;
• TestCafe — end-to-end browser testing framework;
• Vitest — Jest-compatible modern testing framework;
• WebdriverIO — end-to-end testing framework.

Some of these tools can run out-of-the-box and consist of a test runner, assertion library, and much more. Some are dying, though, and you need to choose wisely. Some can be used as building blocks, for instance, Mocha as a test runner + Puppeteer as an API for a browser + Chai as an assertion library + Allure as a test reporter.

Anyway, there should be a list of criteria by which to make a choice in favor of a particular tool. The last time when I took part in choosing a test framework for a new project, we were guided by the following requirements:

1. Do you need unit or integration/end-to-end testing?
2. Browser support (including mobile browsers);
3. Documentation;
4. Community (articles, video tutorials, stackoverflow answers);
5. Support by developers (reactions on GitHub issues) and frequency of updates;
6. Parallelization (the power of test runner);
7. Retries (the power of test runner);
8. Skips and conditional skips (the power of test runner);
9. Assertions (the power of assertion library);
10. Reports (variety of console and HTML reports);
11. WebSocket support;
12. Selenium (WebDriver) support? DevTools Protocol support? Or both?
13. Working with tabs (for example, Cypress does not support that);
14. Mocks/stubs;
15. Proxy HTTP requests/responses;
16. Support HTTP API requests;
17. Setting cookies, localStorage, and user-agents;
18. Visual/screenshot testing;
19. Debug mode;
20. Headless and headed modes (for browsers);
21. Integration with IDE (for example, Playwright can run test in VSCode);
22. Run on localhost;
23. Run in CI/CD systems;
24. Run a single test;
25. Filter tests (ability to grep a bunch of tests);
26. Logging the test execution process;
27. Making screenshots and videos in case of failure (for example, Playwright can even record traces);
28. Distribution size (the number of dependencies in package.json);
29. Maintainability (and how quickly new team members will get used to the project);
30. Open Source and/or license.

In recent times (according to my own observations), the short list of test frameworks for browser testing usually looks like this:

• Playwright;
• Cypress;
• WebdriverIO.

And for unit or API testing:

• Jest;
• Mocha;
• And an additional HTTP request library for Node.js, like GOT or Axios.

UPDATE for 2023 year:

• Vitest is a preferable option for unit testing;

• Fetch API starts work out of the box from Node.js 21, so you may skip adding a library for HTTP requests (see an example).

After all, criteria and tools listed above can be expanded depending on the context of your project.

Moreover, the stated criteria can be used for another (non JS) testing stack.