How to Take Screenshots during Selenium Test Execution

This article talks about how to modify the Selenium source code to take screenshots/snapshots while Selenium test suites are running, and more importantly when a test case fails.

We're clamoring for this feature at my company to make it easier to diagnose those not-so-obvious integration test failures. There is an open issue on Selenium's JIRA pertaining to this, and another blog posting that offers a solution if you are using the C# Selenium Driver as your base for executing tests and not using the Selenium Remote Control like we do.

I've been able to come up with a way to take screenshots after each step of each test by specifying an optional command line parameter.

A folder is placed adjacent to the test results file with the screenshots, and each command (step) of each test in the results file now has a link to the screenshot for that particular step, as shown in the sample below:


EDIT: Obsolete solution - please read: While this was a good idea to get something working, the notion of adjusting the engine code is shaky at best. Instead, we wrote a native test runner for Selenese that can execute a single test suite, or tap into Selenium Grid to run multiple test suites at the same time. I haven't blogged about it but if you'd like to give it a try please let me know!

I've published how to do it here to have a reference point for maintaining this functionality in Selenium as we update the code, and so others can learn from it. Unfortunately, no matter how much I edit this post, the steps are a little confusing so I am happy to e-mail you source code that will do this, and update this post where it is unclear. I'll also look into contributing this back to Selenium although it will need some polishing to meet their standards.

Summary

The approach is relatively straight forward. If you want to apply these changes on your own, you'll need to check out the Selenium source code and familiarize yourself with the different pieces. In a nutshell, here is what I did to add the functionality:

1. Modified Selenium Remote Control to take an additional command line argument to tell Selenium to take screenshots.
2. Update the HTML Launcher code as needed to pass the directory as a query line parameter that is used for the screenshots to Selenium Core when invoking the HTML Test Suite.
3. Modify Selenium Core to acknowledge the additional query line parameter.
4. Take the screenshots and update the results file with the hyperlink of the screenshot prior to posting the result back to Selenium Server.

Each step is described more thoroughly in subsequent sections.

Assumptions

• The screenshot is taken with a command invoked just like calling captureEntirePageScreenshot in a normal test, which only works on Firefox in Chrome mode.
  
• The directory that the screenshots are saved in is located in the /path/to/results/screenshots/ directory. There is no separate command line argument to customize this since seemed to be the the easiest way to know how to create a link to the screenshots in the results file.

Part 1: Modify Selenium Remote Control to take an additional command line argument to tell Selenium to take screenshots

A new command line argument to tell Selenium to take screenshots was needed, and is modeled after the following example:

java -jar selenium-server.jar -take-screenshots -port 4445 -htmlSuite *chrome http://www.google.com "C:\selenium\testsuite\TestSuite.html" "\\BUILD\IntegrationTestResults\run001\results.html"

Add the new configuration parameter as well as the parsing logic to set it.

Add this field and its corresponding properties to RemoteControlConfiguration:

private boolean takeScreenshots;
public boolean shouldTakeScreenshots() {
  return this.takeScreenshots;
}
public void setTakeScreenshots(boolean takeScreenshots) {
  this.takeScreenshots = takeScreenshots;
}

Insert in RemoteControlLauncher:parseLauncherOptions():

} else if ("-take-screenshots".equals(arg)) {
  configuration.setTakeScreenshots(true);
}


Part 2: Pass the directory that Selenium Core is to save screenshots in as a query line parameter when invoking the HTML Test Suite.

First, modify HTML Launcher to acknowledge the configuration parameter created in Step 1.

Open SeleniumServer and find the call to runHTHMLSuite, and make this line look like the following:

result = launcher.runHTMLSuite(getRequiredSystemProperty("htmlSuite.browserString"), startURL, suiteFile, resultFile, configuration.getTimeoutInSeconds(), configuration.isMultiWindow(), configuration.shouldTakeScreenshots());

You'll need to either overload or update the runHTMLSuite() method of the HTMLLauncher class to take this new configuration option.

Modify HTMLLauncher's private runHTMLSuite() method, and replace the call to launchHTMLSuite() with all of the following code:

String screenshotDir = null;
if (takeScreenshots){
  screenshotDir = outputFile.getParent();
  new File(screenshotDir + File.separator + "screenshots").mkdir();
  log.info("Taking screenshots; will attempt to save them in " + screenshotDir);
}
launcher.launchHTMLSuite(suiteURL, browserURL, multiWindow, defaultLogLevel, screenshotDir);

This screenshot directory parameter that was now just added needs to be added to BrowserLauncher and all of the classes that implement it. The easiest way to do it is to refactor with Eclipse to add the additional parameter to all of the classes at once.

LauncherUtils is where the screenshot directory is used to build the query string when launching the test suite. I added a new overloaded method that takes the screenshot directory and appends it to the query line:

protected static String getDefaultHTMLSuiteUrl(String browserURL, String suiteUrl, boolean multiWindow, int serverPort, String defaultLogLevel, String screenshotBaseDir) {
  String url = LauncherUtils.stripStartURL(browserURL);
  String resultsUrl;
  if (serverPort == 0) {
    resultsUrl = "../postResults";
  } else {
    resultsUrl = "http://localhost:" + serverPort + "/selenium-server/postResults";
  }

  url += "/selenium-server/core/TestRunner.html?auto=true"
    + "&multiWindow=" + multiWindow
    + "&defaultLogLevel=" + defaultLogLevel
    + "&baseUrl=" + urlEncode(browserURL) + "/selenium-server/tests/"
    + "&resultsUrl=" + resultsUrl
    + "&test=" + urlEncode(suiteUrl);

  if (null != screenshotBaseDir && !screenshotBaseDir.isEmpty()){
    url += "&screenshotBaseDir=" + urlEncode(screenshotBaseDir);
  }

  return url;
}

Finally, modify FirefoxChromeLauncher to call the newly enhanced LauncherUtils.getDefaultHTMLStringUrl() method.

Now, Selenium Core will receive an additional Query Line parameter screenshotBaseDir when screenshots are requested.

Step 3: Modify Selenium Core to acknowledge the additional query line parameter.

First, add an easy way to access the passed in query parameter:

Note that it is helpful to understand Prototype classes for following these instructions.

In selenium-testrunner.js add to the HtmlTestRunnerControlPanel class:

getScreenshotBaseDirectory: function() {
  return this._getQueryParameter("screenshotBaseDir");
}

I added methods to lookup the test suite name and currently executing test name, so they can be used as a part of the screenshot file names. The reason for doing this is sometimes multiple test suites run the same test so it reduces the chance of a file name collision. These appear in HtmlTestCase class, also in selenium-testrunner.js:

extractTestSuiteName: function() {
  try {
    var testsuiteNameWithPath = htmlTestRunner.controlPanel.getTestSuiteName();
    var testsuiteRegex = /[^\/\\]+.$/;
    var regexResults = testsuiteRegex.exec(testsuiteNameWithPath);
    return regexResults[0].replace(".html", "");
  } catch (e) {}
  return 'unknown';
},

extractTestName: function() {
  try {
    var testsuiteNameWithPath = this.testWindow.location.pathname;
    var testsuiteRegex = /[^\/\\]+.$/;
    var regexResults = testsuiteRegex.exec(testsuiteNameWithPath);
    return regexResults[0].replace(".html", "");
  } catch (e) {}
    return 'unknown';
}

Finally, the HtmlRunnerTestLoop class initialize() function is annotated to set two fields that can be accessed to construct the file name for the screenshots in the next step:

var screenshotBaseDir = htmlTestRunner.controlPanel.getScreenshotBaseDirectory();
if (null != screenshotBaseDir) {
  this.screenshotDirectory = screenshotBaseDir;// + "\\" + htmlTestCase.extractTestSuiteName() + "_" + htmlTestCase.extractTestName();
  this.screenshotNamingPrefix = htmlTestCase.extractTestSuiteName() + "_" + htmlTestCase.extractTestName();
}

Step 4: Take the screenshot and update the results file with the hyperlink of the screenshot prior to posting the result back to the Selenium Server

I added a new function to HtmlTestCaseRow in selenium-testrunner.js to create the hyperlink to the screenshot in the results file:

setScreenshotLink: function(screenshotHref) {
  this.trElement.cells[0].innerHTML = "" + getText(this.trElement.cells[0]) + "";
}

Add this function to the TestLoop class that actually takes the screenshots:


takeScreenshot : function() {
  if (null != this.screenshotDirectory && this.screenshotDirectory != '')
  {
    var justExecutedCommand = this.currentCommand;
    if (null != justExecutedCommand) //Don't snapshot before test begins
    {
      var screenshotHref = "screenshots\\" + this.screenshotNamingPrefix + "_" + this.commandCounter + ".jpg";
      var screenshotFilename = this.screenshotDirectory + "\\" + screenshotHref;
      var screenshotCommand = new Command('captureEntirePageScreenshot', screenshotFilename, null);
      var handler = this.commandFactory.getCommandHandler(screenshotCommand.command);
      if (handler == null) {
        throw new SeleniumError("Unknown command: '" + screenshotCommand.command + "'");
      }

      screenshotCommand.target = selenium.preprocessParameter(screenshotCommand.target);
      screenshotCommand.value = selenium.preprocessParameter(screenshotCommand.value);
      handler.execute(selenium, screenshotCommand);

      currentTest.currentRow.setScreenshotLink(screenshotHref.replace("\\", "/"));
    }
  }
}

Almost done: just need to call the function to take the screenshots and initialize the counter (for which there is probably a better way of doing):

In the TestLoop class:

• call this.commandCounter = 0 at the top of the start() function
• call this.takeScreenshot() at the top of the continueTest() function
• call this.commandCounter++ at the top of the resume() function
• call this.takeScreenshot() for unhandled errors (right above the call for takeScreenshot())

Wrap Up

There is a lot here, but the steps should be straightfoward save a few steps to checkout and build the source code. Please feel free to leave a comment if you have suggestions or feedback, particularly if there is a better way to accomplish this!