NightwatchJS for Desktop Browser testing
12 min readAug 2, 2023
Preparing environment for Desktop Browser test automation
Requirements
- Create a NightwatchJS project for desktop browser:
- npm init nightwatch@latest <project-directory>
npm init nightwatch@latest nightwatch-desktop-browser
Need to install the following packages:
create-nightwatch@3.1.2
Ok to proceed? (y) y
// End-2-End, Component or Mobile App
? Select testing type to setup for your project
> End-to-End testing
? Select language + test runner variant
> TypeScript / default
? Select target browsers
> Chrome (or Firefox or Edge or Safari for Mac)
? Enter source folder where test files are stored
> test
? Enter the base_url of the project
> http://localhost
? Select where to run Nightwatch tests
> Both (localhost and remote/cloude service)
? (Remote) Select cloud provider:
> BrowserStack (or Sauce Labs or Other provider / remote selenium-server)
? Allow Nightwatch to collect completely anonymous usage metrics?
> No
? Setup testing on Mobile devices as well?
> No, skip for nowNPM will install nightwatch, create the config file nightwatch.conf.js and some test examples at nightwatch/github.ts
Installing the following packages:
- nightwatch
- typescript
- @swc/core
- ts-node
Installing nightwatch
Done!
Installing typescript
Done!
Installing @swc/core
Done!
Installing ts-node
Done!
Created a new tsconfig.json with:
TS
target: es2016
module: commonjs
strict: true
esModuleInterop: true
skipLibCheck: true
forceConsistentCasingInFileNames: true
You can learn more at https://aka.ms/tsconfig
Generating example files...
✔ Success! Generated some example files at 'nightwatch'.
Generating a configuration file based on your responses...
✔ Success! Configuration file generated at: "/home/popos/Projetos/nightwatch-desktop-browser/nightwatch.conf.js".
IMPORTANT
Please set the credentials required to run tests on your cloud provider, by setting the below env variables:
- BROWSERSTACK_USERNAME
- BROWSERSTACK_ACCESS_KEY
(.env files are also supported)
✨ SETUP COMPLETE
Nightwatch:
version: 3.1.2
changelog: https://github.com/nightwatchjs/nightwatch/releases/tag/v3.1.2
💬 Join our Discord community to find answers to your issues or queries. Or just join and say hi.
https://discord.gg/SN8Da2X
🚀 RUN EXAMPLE TESTS
To run all examples, run:
npx nightwatch ./nightwatch
To run a single example (github.ts), run:
npx nightwatch ./nightwatch/github.ts4. Configure file nightwatch.conf.js
- src_folders: [‘test’,’nightwatch’], — the path to test cases
- test_workers: {enabled: true}, — parallel test cases enabled
- screenshots: {enabled: false, path: ‘screens’, on_failure: true}, — to enable screenshot on failure
- desiredCapabilities: {browserName: ‘chrome’}, — browser selected for testing
- webdriver: {start_process: true,server_path: ‘’}, — if you want to use a desired webdriver
- firefox: { — settings for firefox environment
- chrome: { — settings for chrome environment
- edge: { — setings for edge environment
- safari: { — setings for safari environment
- browserstack: { — settings for browserstack environment
- ‘browserstack.chrome’: { —extends enviroment for browserstack to use chrome browser
- ‘browserstack.firefox’: { — extends enviroment for browserstack to use firefox browser
more information about it at https://nightwatchjs.org/guide/reference/settings.html
4. Verify the environment by running a sample test case:
single test:
npx nightwatch ./nightwatch/<test suite>.<ext> --env <enviroments,..>multi tests:
npx nightwatch ./nightwatch/<test suite path> --env <enviroments,..>Let`s run a single test (github.ts) into a single environment (firefox):
cd nightwatch-desktop-browser
npx nightwatch ./nightwatch/github.ts --env firefox
[Nightwatch/Github] Test Suite
────────────────────────────────────────────────────────────
⠋ Starting ChromeDriver on port 9515...
ℹ Connected to ChromeDriver on port 9515 (985ms).
Using: chrome (115.0.5790.110) on LINUX.
Running Github Title test:
───────────────────────────────────────────────────────────────────────────────────────────────────
ℹ Loaded url https://github.com in 1968ms
✔ Testing if the page title contains 'GitHub' (24ms)
✨ PASSED. 1 assertions. (2.017s)
Running Github search for nightwatch repository:
───────────────────────────────────────────────────────────────────────────────────────────────────
ℹ Loaded url https://github.com/search in 1437ms
✔ Element <.header-search-button> was visible after 80 milliseconds.
✔ Testing if element´s <.header-search-button> inner text equals 'nightwatch' (60ms)
✔ Element <div[data-testid="results-list"]:first-child> was visible after 599 milliseconds.
✔ Testing if element <div[data-testid="results-list"]:first-child> contains text 'End-to-end testing framework written in Node.js and using the W3C Webdriver API' (235ms)
✨ PASSED. 4 assertions. (3.158s)
Running Github login with fake credentials:
───────────────────────────────────────────────────────────────────────────────────────────────────
ℹ Loaded url https://github.com/login in 516ms
✔ Element <[value='Sign in']> was visible after 32 milliseconds.
✔ Testing if element <#js-flash-container .flash.flash-error> contains text 'Incorrect username or password.' (24ms)
✨ PASSED. 2 assertions. (1.764s)
✨ PASSED. 7 total assertions (8.685s)
Wrote HTML report file to: /home/popos/Projetos/nightwatch-desktop-browser/tests_output/nightwatch-html-report/index.htmlAnd now let`s run multiple suites in multiple environments:
It will first open 4 sessions for Firefox to run each one of the 4 test suites, and after that open another 4 sessions for Chrome to run the same 4 test suites, and create a test report.
npx nightwatch ./nightwatch --env firefox,chrome
Running firefox: nightwatch/duckDuckGo.ts
Running firefox: nightwatch/ecosia.ts
Running firefox: nightwatch/github.ts
Running firefox: nightwatch/google.ts
Running chrome: nightwatch/duckDuckGo.ts
Running chrome: nightwatch/ecosia.ts
Running chrome: nightwatch/github.ts
Running chrome: nightwatch/google.ts
┌ ────────────────── ✔ firefox: nightwatch/google.ts ───────────────────────────────┐
│ │
│ [Nightwatch/Google] Test Suite │
│ ──────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: firefox (115.0.3) on LINUX. │
│ – Google title test │
│ - Loading url: https://google.com/ncr │
│ ℹ Loaded url https://google.com/ncr in 3601ms │
│ ✔ Testing if the page title equals 'Google' (97ms) │
│ ✔ firefox [Nightwatch/Google] Google title test (3.771s) │
│ – Google search test │
│ ✔ Element <#main> was visible after 1721 milliseconds. │
│ ✔ Testing if element <#main> contains text 'Nightwatch.js' (3878ms) │
│ ✔ firefox [Nightwatch/Google] Google search test (6.587s) │
│ │
└─────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ firefox: nightwatch/ecosia.ts ───────────────────────────────┐
│ │
│ [Ecosia.org Demo] Test Suite │
│ ──────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: firefox (115.0.3) on LINUX. │
│ – Demo test ecosia.org │
│ ✔ Element <body> was visible after 860 milliseconds. │
│ ✔ Testing if the page title contains 'Ecosia' (73ms) │
│ ✔ Testing if element <input[type=search]> is visible (372ms) │
│ ✔ Testing if element <button[type=submit]> is visible (132ms) │
│ ✔ Testing if element <.layout__content> contains text 'Nightwatch.js' (1134ms) │
│ ✔ firefox [Ecosia.org Demo] Demo test ecosia.org (6.802s) │
│ │
└─────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ firefox: nightwatch/duckDuckGo.ts ──────────────────────────────┐
│ │
│ [duckduckgo example] Test Suite │
│ ────────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: firefox (115.0.3) on LINUX. │
│ – Search Nightwatch.js and check results │
│ ✔ Element <input[name=q]> was visible after 557 milliseconds. │
│ ✔ Testing if element <.react-results--main> is visible (352ms) │
│ ✔ Testing if element <.react-results--main> contains text 'Nightwatch.js' (1538ms) │
│ ✔ firefox [duckduckgo example] Search Nightwatch.js and check results (10.425s) │
│ │
└────────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ firefox: nightwatch/github.ts ────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ [Nightwatch/Github] Test Suite │
│ ──────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: firefox (115.0.3) on LINUX. │
│ – Github Title test │
│ - Loading url: https://github.com │
│ ℹ Loaded url https://github.com in 3197ms │
│ ✔ Testing if the page title contains 'GitHub' (122ms) │
│ ✔ firefox [Nightwatch/Github] Github Title test (3.376s) │
│ – Github search for nightwatch repository │
│ - Loading url: https://github.com/search │
│ ℹ Loaded url https://github.com/search in 4220ms │
│ ✔ Element <.header-search-button> was visible after 284 milliseconds. │
│ ✔ Testing if elements <.header-search-button> inner text equals 'nightwatch' (1501ms) │
│ ✔ Element <div[data-testid="results-list"]:first-child> was visible after 356 milliseconds. │
│ ✔ Testing if element <div[data-testid="results-list"]:first-child> contains text 'End-to-end testing framework written in Node.js and using the W3C │
│ Webdriver API' (1714ms) │
│ ✔ firefox [Nightwatch/Github] Github search for nightwatch repository (9.603s) │
│ – Github login with fake credentials │
│ - Loading url: https://github.com/login │
│ ℹ Loaded url https://github.com/login in 699ms │
│ ✔ Element <[value='Sign in']> was visible after 38 milliseconds. │
│ ✔ Testing if element <#js-flash-container .flash.flash-error> contains text 'Incorrect username or password.' (45ms) │
│ ✔ firefox [Nightwatch/Github] Github login with fake credentials (2.925s) │
│ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ chrome: nightwatch/duckDuckGo.ts ────────────────────────────────────┐
│ │
│ [duckduckgo example] Test Suite │
│ ────────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Error getting version of chromedriver 115. Retrying with chromedriver 114 (attempt 1/5) │
│ Using: chrome (115.0.5790.110) on LINUX. │
│ – Search Nightwatch.js and check results │
│ ✔ Element <input[name=q]> was visible after 264 milliseconds. │
│ ✔ Testing if element <.react-results--main> is visible (2712ms) │
│ ✔ Testing if element <.react-results--main> contains text 'Nightwatch.js' (600ms) │
│ ✔ chrome [duckduckgo example] Search Nightwatch.js and check results (7.342s) │
│ │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ chrome: nightwatch/ecosia.ts ────────────────────────────────────────┐
│ │
│ [Ecosia.org Demo] Test Suite │
│ ──────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Error getting version of chromedriver 115. Retrying with chromedriver 114 (attempt 1/5) │
│ Using: chrome (115.0.5790.110) on LINUX. │
│ – Demo test ecosia.org │
│ ✔ Element <body> was visible after 298 milliseconds. │
│ ✔ Testing if the page title contains 'Ecosia' (33ms) │
│ ✔ Testing if element <input[type=search]> is visible (201ms) │
│ ✔ Testing if element <button[type=submit]> is visible (212ms) │
│ ✔ Testing if element <.layout__content> contains text 'Nightwatch.js' (606ms) │
│ ✔ chrome [Ecosia.org Demo] Demo test ecosia.org (6.399s) │
│ │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ chrome: nightwatch/google.ts ────────────────────────────────┐
│ │
│ [Nightwatch/Google] Test Suite │
│ ──────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: chrome (115.0.5790.110) on LINUX. │
│ – Google title test │
│ - Loading url: https://google.com/ncr │
│ ℹ Loaded url https://google.com/ncr in 1684ms │
│ ✔ Testing if the page title equals 'Google' (47ms) │
│ ✔ chrome [Nightwatch/Google] Google title test (1.751s) │
│ – Google search test │
│ ✔ Element <#main> was visible after 350 milliseconds. │
│ ✔ Testing if element <#main> contains text 'Nightwatch.js' (489ms) │
│ ✔ chrome [Nightwatch/Google] Google search test (9.219s) │
│ │
└─────────────────────────────────────────────────────────────────────────────────────┘
┌ ────────────────── ✔ chrome: nightwatch/github.ts ─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ [Nightwatch/Github] Test Suite │
│ ──────────────────────────────────────────────────────────── │
│ Applicable driver not found; attempting to install with Selenium Manager (Beta) │
│ Using: chrome (115.0.5790.110) on LINUX. │
│ – Github Title test │
│ - Loading url: https://github.com │
│ ℹ Loaded url https://github.com in 6921ms │
│ ✔ Testing if the page title contains 'GitHub' (106ms) │
│ ✔ chrome [Nightwatch/Github] Github Title test (7.037s) │
│ – Github search for nightwatch repository │
│ - Loading url: https://github.com/search │
│ ℹ Loaded url https://github.com/search in 1608ms │
│ ✔ Element <.header-search-button> was visible after 962 milliseconds. │
│ ✔ Testing if elements <.header-search-button> inner text equals 'nightwatch' (348ms) │
│ ✔ Element <div[data-testid="results-list"]:first-child> was visible after 34 milliseconds. │
│ ✔ Testing if element <div[data-testid="results-list"]:first-child> contains text 'End-to-end testing framework written in Node.js and using the W3C │
│ Webdriver API' (293ms) │
│ ✔ chrome [Nightwatch/Github] Github search for nightwatch repository (4.037s) │
│ – Github login with fake credentials │
│ - Loading url: https://github.com/login │
│ ℹ Loaded url https://github.com/login in 268ms │
│ ✔ Element <[value='Sign in']> was visible after 29 milliseconds. │
│ ✔ Testing if element <#js-flash-container .flash.flash-error> contains text 'Incorrect username or password.' (29ms) │
│ ✔ chrome [Nightwatch/Github] Github login with fake credentials (1.344s) │
│ │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
✨ PASSED. 36 total assertions (38.452s)
Wrote HTML report file to: /home/popos/Projetos/nightwatch-desktop-browser/tests_output/nightwatch-html-report/index.html5. Test reports
Report will be overwrite at /tests_output/nightwatch-html-report/index.html
- At top right — tests by environment
- At left side — the test flow for each test suite
- At center — the test case details
6. Parallel testing
To enable parallell test execution, update the nightwatch.conf.js file again, set test_workers to true. You can set the number of threads (works) or let it be auto (not declared will use auto as default — auto = number of CPU cores available):
test_workers: {
enabled: false,
workers: 'auto'
},Or use the CLI command line --workers :
npx nightwatch ./nightwatch --env firefox,chrome --workes 4The parallel execution test cases are executed at the same time, it consumes much more resources from your desktop, but the execution takes less time:
To prevent fake negatives, let`s add some retries to the test suites that uses the BDD describe test syntax (export test syntax are more limited) :
describe('duckduckgo example', function(this: ExtendDescribeThis<CustomThis>) {
this.retries(2)
...
describe('Ecosia.org Demo', function() {
this.retries(2)
...- this.retries(3); — how many time to retry a failed testcase inside this test suite — retest right after the test case that failed
- this.suiteRetries(2); — how many times to retry the current test suite in case of an assertion failure or error — retest only after completes the test suite that failed
Or use the CLI command line --retries (3 = original execution + 3 more)
npx nightwatch ./nightwatch --env firefox,chrome --retries 3Another trick is the use of hedless, when enable it the graphic interface of website are downloaded but not rendered, improving the time and performance.
At each environment we could enable the headless flag…
firefox: {
desiredCapabilities: {
browserName: 'firefox',
alwaysMatch: {
acceptInsecureCerts: true,
'moz:firefoxOptions': {
args: [
'-headless',
// '-verbose'
]
}
}
},
...
chrome: {
desiredCapabilities: {
browserName: 'chrome',
'goog:chromeOptions': {
// More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
//
// w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
w3c: true,
args: [
//'--no-sandbox',
//'--ignore-certificate-errors',
//'--allow-insecure-localhost',
'--headless'
]
}
},
...… but instead, let`s run the test cases using the CLI (command line) flag --headless:
npx nightwatch ./nightwatch --env firefox,chrome --headlessFor more command line settings https://nightwatchjs.org/guide/nightwatch-cli/command-line-options.html
Time consuming and performance comparison:
- no headless and serial execution = 37.051s
npx nightwatch ./nightwatch --env firefox,chrome- no headless and parallel execution = 30.089s
npx nightwatch ./nightwatch --env firefox,chrome --workers 4- headless and serial execution = 26.288s
npx nightwatch ./nightwatch --env firefox,chrome --headless --serial- headless and parallel execution = 22.477s
npx nightwatch ./nightwatch --env firefox,chrome --headless --workers 4We achieved a difference of 14,574s comparing the least performant to the very performant
7. Debug mode and Inspect web elements
Add a break point into the test case with .debug()
ecosia.js test case
describe('Ecosia.org Demo', function() {
this.retries(2)
this.tags = ['demo'];
before(browser => browser.navigateTo('https://www.ecosia.org/'));
it('Demo test ecosia.org', function(browser) {
browser
.waitForElementVisible('body')
.assert.titleContains('Ecosia')
.debug()
.assert.visible('input[type=search]')
.setValue('input[type=search]', 'nightwatch js')
.assert.visible('button[type=submit]')
.click('button[type=submit]')
.assert.textContains('.layout__content', 'Nightwatch.js');
});
after(browser => browser.end());
});And run the test case with debug parameter:
npx nightwatch ./nightwatch/ecosia.js --env firfefox --debugNote: You can`t debug multiple environments, debug it each one easch time.
Firebox driver will starts and initializes Firefox Browser to start the test case.
Now open the Inspect Mode to inspect the web elements!
Now it`s possible to inspect the elements from mobile browser, just open the Inspect Mode (F12 or Right Click Inspect Element (Q)):
Create a selector for the button accept all at cookie notifier:
button[data-test-id="cookie-notice-accept"]It has a data-test-id attribute with value cookie-notice-accept
Update the test case to accept the terms:
describe('Ecosia.org Demo', function() {
this.retries(2)
this.tags = ['demo'];
before(browser => browser.navigateTo('https://www.ecosia.org/'));
it('Demo test ecosia.org', function(browser) {
browser
.waitForElementVisible('body')
.assert.titleContains('Ecosia')
.waitForElementVisible('button[data-test-id="cookie-notice-accept"]')
.assert.visible('button[data-test-id="cookie-notice-accept"]')
.click('button[data-test-id="cookie-notice-accept"]')
.assert.visible('input[type=search]')
.setValue('input[type=search]', 'nightwatch js')
.assert.visible('button[type=submit]')
.click('button[type=submit]')
.assert.textContains('.layout__content', 'Nightwatch.js');
});
after(browser => browser.end());
});Exit debug mode typing .exit command (test case will continue to run after the break point and finish), remove the .debug() break point and run the same test case again without debug:
npx nightwatch ./nightwatch/ecosia.js --env firefoxNow it will accept cookies before complete the execute the test case!
Now you know how to test your website into a desktop browser!
For more information about Native Mobile testing instead of Desktop one, check it out our last post:
- Native Application Automation: https://medium.com/@dmaioni/nightwatchjs-9a526ea984ad
- Mobile Browser Automation: https://medium.com/@dmaioni/nightwatchjs-for-browser-mobile-testing-54676158de64
- GitHub Project: https://github.com/maioni/nightwatch-desktop-browser
