Selection Criteria of Test Framework
When a dev team starts building a testing infrastructure, there is always a debate about the choice of testing tools.
- Ava — test runner;
- CodeceptJS — end-to-end testing framework;
- Cypress — end-to-end browser testing framework;
- Hermione — test runner for browser testing based on Mocha and WebdriverIO;
- Jasmine — testing framework;
- Jest — testing framework;
- Karma — test runner;
- Mocha — test runner;
- Nightwatch — end-to-end testing framework;
- Playwright — end-to-end testing framework;
- Protractor — deprecated end-to-end test framework for Angular applications;
- Puppeteer — headless Chrome Node.js API (actually, it is not a testing tool);
- SelenideJS — browser testing framework;
- TestCafe — end-to-end browser testing framework;
- WebdriverIO — end-to-end testing framework.
Some of these tools can run out-of-the-box and consist of a test runner, assertion library and much more. Some of these can be used as building blocks, for instance, Mocha as a test runner + Puppeteer as an API for browser + Chai as an assertion library + Allure as a test reporter.
Anyway, there should be a list of criteria by which to make a choice in favor of a particular tool. The last time when I took part in a choosing a test framework on a new project, we were guided by the following requirements:
- Browser support (including mobile browsers);
- Community (articles, video tutorials, stackoverflow answers);
- Support by developers (reactions on GitHub issues) and frequency of updates;
- Parallelisation (the power of test runner);
- Retries (the power of test runner);
- Skips and conditional skips (the power of test runner);
- Assertions (the power of assertion library);
- Reports (variety of console and HTML reports);
- WebSocket support;
- Selenium (WebDriver) support? DevTools Protocol support? Or both?
- Working with tabs (for example Cypress does not support that);
- Mocks (stubs);
- Proxy HTTP requests/responses;
- Request HTTP API inside UI tests;
- Setting cookies, localStorage and user-agents;
- Screenshot testing;
- Debug mode;
- Headless and headed modes;
- Integration with IDE (for example Playwright can run test in VSCode);
- Run in localhost;
- Run in CI/CD systems;
- Run a single test;
- Run only a bunch of tests (ability to grep);
- Logging the test execution process;
- Making screenshots and videos in case of failure (for example Playwright even can record traces);
- Distribution size (the number of dependencies in package.json);
- Open Source, license.
In recent times (according to my own observations), the short list of test frameworks for browser testing usually looks like this:
And for unit or API testing:
After all, criteria and tools listed above can be expanded depending on the context of your project.