We have been using and promoting TestCafe at platformOS for the past couple of years with great success. Because a lot of people will write tests and maintain them over a long time, an End-to-End framework has to come with some specific requirements:
- Easy to remember and type out API
- Good waiting mechanisms (for XHR requests, animations)
- Extendibility, page object support, helpers support
- Good search in documentation to quickly reference less used APIs
- Run properly in Docker and/or GitHub Actions
TestCafe is scoring high on the above areas, I would say averaging around 7.5/10, which means there is still room for improvement.
Even though we have been happy with TestCafe, last year when I stumbled upon a new contender, CodeceptJS, I decided to give it a shot on our documentation and marketing sites. It delivered excellent developer performance. It was enough to dive deeper into its documentation and expand our test suites to include some more test cases.
1. Test API
Very often when writing TestCafe tests, we had to resort to vanilla JS and DOM operations. One of the most frustrating examples was to get some text from an element and then compare it to another. It was too much work and I never could see a reason why TestCafe had no API for that. CodeceptJS has a lot more API helpers to avoid these complications and diverging into vanilla JS. Below, I give you some examples of TestCafe scenarios converted to CodeceptJS ones.
Checking if correct breadcrumbs links are present on a page
// TestCafe
test('Breadcrumbs are showing up', async t => {
await t.navigateTo('/api-reference/liquid/introduction');
await t.expect(Selector('.breadcrumbs a').withText('Documentation').exists).ok();
await t.expect(Selector('.breadcrumbs a').withText('API Reference').exists).ok();
await t.expect(Selector('.breadcrumbs a').withText('Introduction').exists).ok();
});
// CodeceptJS
Scenario('Are showing up', ({ I }) => {
I.amOnPage('/api-reference/liquid/introduction');
I.see('Documentation', '.breadcrumbs');
I.see('API Reference', '.breadcrumbs');
I.see('Introduction', '.breadcrumbs');
});
Checking if automatically generated content links have the same text as headings in the article
// TestCafe
test('Generated steps have correct text', async t => {
const container = await Selector('[data-autosteps]');
const firstEl = await container.find('a').nth(0);
const secondEl = await container.find('a').nth(1);
await t.expect(firstEl.textContent).eql('Step 1: Fetch JWT token for a user');
await t.expect(secondEl.textContent).eql('Step 2: Create a page with a policy that checks the JWT token');
});
// CodeceptJS
const stepsLinks = 'ul.content__autosteps a';
Scenario('Steps have correct texts', async ({ I }) => {
I.see('Step 1: Fetch JWT token for a user', stepsLinks);
I.see('Step 2: Create a page with a policy that checks the JWT token', stepsLinks);
I.see('Step 3: Send signed request', stepsLinks);
});
Checking if there is the same number of generated links as headings in the content
// TestCafe
test('Generate as many links as there are headings with steps', async t => {
const container = await Selector('[data-autosteps]');
const stepHeadings = await Selector('h3').filter(h => /^Step \\d+:/.test(h.textContent));
const generatedLinks = await container.find('a');
await t.expect(await stepHeadings.count).eql(await generatedLinks.count);
await t.expect(await stepHeadings.count).eql(3);
});
// CodeceptJS
Scenario('Generate same number of links as there are steps', async ({ I }) => {
let stepsNumber = await I.grabNumberOfVisibleElements(stepsLinks);
let headersNumber = await I.grabNumberOfVisibleElements('h3[id]');
assert.ok(stepsNumber === headersNumber);
});
In my opinion, the CodeceptJS API is more human-like, reads like a sentence, and does not require as much JavaScript knowledge as TestCafe, which was a barrier of entry for our new testers, who sometimes were new to JavaScript. Additionally, tests almost always have fewer lines (or characters in lines) of code, thus are easier to read.
Easy to extend
TestCafe was already very easy to extend, because of its "just JavaScript" approach and good documentation. CodeceptJS is pretty similar in that respect, but documentation teaches us how to properly use different ways of abstracting complexity and reuse code to maximize the maintainability of the code over the long run.
CodeceptJS supports:
Documentation
CodeceptJS has good documentation. It has some dead links and a lot of long menus, but that's because it supports so many test runners and features. In that sense, it is similar to TestCafe — it uses Algolia for searching, which returns excellent results in a very short time.
Additionally, one of my favorite parts of their documentation is that they promote best practices. Not only do they describe how the software works, but they also educate visitors on how to write good tests, or at least well organized and readable ones, so they don't have to be rewritten every six months when the files get too long, abstractions too complicated, and page objects too big.
Read more on CodeceptJS Best Practices at https://codecept.io/best/
Apart from having extensive documentation on their website, they also cultivate community-driven channels like a Slack chat, Discourse forum, GitHub issues, Twitter, Stack Overflow. This means there is a way to contact both developers and users should you need it.
Many runners supported
- WebDriver (Selenium)
- Puppeteer - Google API for controlling chromium-based browsers. We chose to use it as it tends to work reliably and fast
- Playwright - Microsoft's take on the cross-browser API, a competitor to puppeteer
- TestCafe - yes, it even can run tests using TestCafe
- Appium - for testing mobile/hybrid applications
- BrowserStack - for testing remotely on many different browsers
In this aspect, we were happy with TestCafe, but it is good to know that we do not lose any features that we use by switching to CodeceptJS.
Plugins
CodeceptJS has a modular structure. Most of the features most people will use are in the core. TestCafe is structured similarly and arguably has more plugins provided by the community.
Just to name a few plugins that will make some specific use cases easier:
autoLogin - exposes simple API for you to log in before running scenario, and caches cookies for future test runstryTo - allows you to write tests that will not fail if the operation was not successfully done. For example, you can save a variable if a particular text has been visible, without failing a test if it didn't. const result = await tryTo(() => I.see('Welcome'));retryFailedStep - allows you to retry failed steps X number of times to minimize the flakiness of tests. Sometimes animations did not end in time, sometimes a request came back too slow - this plugin is great, because if everything is fast and predictable, it's invisible, when something is wrong, it will try to salvage the test runcustomLocator - it makes it very easy to mark HTML tags with data-id so it can be picked up by CodeceptJS when selecting a particular item that might be problematic. Sometimes it is the cleanest way to add data-id="my-element" rather than create some long, convoluted, and often fragile CSS/xpath selectors
Nice to have features
Initialize project wizard
npx create-codeceptjs . and npx codeceptjs init
Very often when starting a new project, the first steps can be frustrating. It was refreshing to see that the CodeceptJS team anticipated that and created a project to help me go from zero to a working project using Playwright (which is later switched to Puppeteer).
Parallel execution
Just like when using TestCafe, you can execute your tests in parallel using CodeceptJS. We never managed to write a test suite independent enough to be able to run tests in parallel in TestCafe, but it looks like CodeceptJS is running whole scenarios (sets of tests) in parallel, so this time we might be able to speed up our tests by a couple of times.
We will be exploring this feature deeper in the future because this way of splitting tests into threads is much better than alternatives. As a side effect, it will force us to implement stricter test organization because parallel execution requires scenarios to be independent when run on the same application server.
Email testing
Using end-to-end testing for testing emails is notoriously hard because there is no other way than actually sending a real email, receiving it, and inspecting it - and for that, you need to have a mailbox online. And if you are testing a registration process with confirmation links, it gets pretty complex very fast. CodeceptJS developers provide an official helper for the MailSlurp service, and the community provides helpers for testmailapp, mailosaurus, and tempmail.
I'm very happy that CodeceptJS developers thought about that and the community added their own plugins so it can become easier to test emails.
Read more at https://codecept.io/email/
Visual testing
If you decide to test for visual breaking changes, CodeceptJS has this feature supported in two ways: Resemble.js and Applitools. We never used visual testing in TestCafe so my experience is very limited (we used in it only GhostInspector, and we were experimenting with percy.io), but this is something we might do for projects that are established graphically and do not change often (are in maintenance mode).
Read more at https://codecept.io/visual/
Graphical interface
Even though we do not use it every day, it is a welcome feature that might lower the barrier of entry and make running tests locally more approachable. Some things are much easier to do in the GUI for people that are not fluent in CLI or in large test suites. Repeating one test is just one click, instead of typing it out.
Migrating
Migrating from the TestCafe syntax to CodeceptJS is pretty easy and repetitive, the harder part is to actually rewrite the test. While I was doing that in our experimental projects I discovered that the resulting test was often more comprehensive and better thought out than the original. I don't know if that's because I was writing those tests again (having more experience), or working with CodeceptJS just encourages me to write more, because I enjoy it more.
Every time we discover a new technology that we want to switch to we do it gradually. We start with small projects, then we use it in new projects, and finally, if there is time, we migrate the biggest legacy projects. This allows us to minimize the risk, get more experience in the technology before using it in more complicated projects, and showcase the benefits to other team members on a real example, instead of in theory.
Example
If you are interested in an example project that uses CodeceptJS run on GitHub Actions our documentation repository should be a good starting point.
Take note of the codecept.conf.js file and the test_staging step in build.yml.
If you are more interested in the migration process from TestCafe to CodeceptJS take a look at this commit. Keep in mind that first we tested it out with Playwright (inside the official Microsoft Github Action), and in later commits switched to Puppeteer to keep it consistent with TestCafe in case any issues arise.