The most important part of whatever testing is a test results report. There is no sense to do anything with tested application if you cannot deliver to your customer understandable and easy for reading document where all steps of a test case, expected and actual results of all provided actions are described in details. That’s why I believe that the work must be started from creation of classes for generating such report.
I decided to develop classes necessary for reporting in a separate project named Logger. For now the project is already added to the solution. It will provide classes and methods which should to write down into the memory all the events occurred during a test session, and then, at the end of its work, the classes will create summary report with the tests run results. Another reason the project should be developed first is that many of its classes will be used in other projects very often and almost all over the solution.
Below is the description of how the reporting will work in real life. I use such approach because, on my opinion, it is most correct and wide-spread:
● The most high-level unit of software testing is a test run session. A test session is a set of all tests (test cases) which should be run in the scope of a single verification of the product quality. When all the tests are finished, the main document with the list of all test suites (test collections) and other statistics is created. The customer can understand in general the result of the test session and current status of the product quality with this document. For details he can see other documents, links to which are listed in the document.
● All test cases must be grouped in test suites – collections of tests joined together by their logic of similar functionality. Every test suite is represented in programming code as a single class with attribute [TestFixture] or [TestClass] and contains test methods which actually are the tests. After all the tests in the test suite has finished the work, a document with the list of all tests and their statuses is created. A link to this document is placed in the main test session results document. So, the customer can switch from the main document to the report of every single test suite and see its status and statistics in details.
● Every test is represented in the programming code as a method with attribute [Test] or [TestMethod] located in a class of a test suite. It consists of a test scope (description of the test actions – what the test does and what it expects to be) and test steps (actions which are performed during the test execution). New document with description and results of the test execution is created every time a method is invoked. Therefore there will be as many documents with test results as test methods have been invoked. Links to these documents are listed in appropriate test suite report. So, the customer can switch not only from main document to a test collection results document but a bit deeper – to a report of a single test case.
● Every test case consists of test steps – actions which interact with the tested application and provide necessary verifications. A step consists of a description (description of the action), expected and actual results (descriptions of what is expected to occur after the step is done and what actually has happened). All test case steps are listed in test case report.
The diagram shows the structure of a test result summary report:
I propose to create classes for generating summary report basing on this scheme. It must work this way:
● Every class with attribute [TestFixture] or [TestClass] is a test suite (collection) and it contains test methods marked with attribute [Test] or [TestMethod], that are the tests
● Since a report is the only one for all tests its instance may be declared as static. The reporter must be also public because it is used all over the entire solution. It creates a list of test cases in the memory and adds a test case instance with appropriate attributes, steps and substeps each time any other test have been executed.
● During a test session every single step is added to currently running test case. When the test is finished it is added to the test cases list as mentioned above.
● When all the tests finished running the reporter generates a report in HTML format located in a separate uniquely named folder. The report consists of a set of files: main document named index.html, a separate document for every test suite and a separate document for every test case that has been executed. The main document contains links to documents of test suites; every test suite document contains links to documents of test cases which have been run under this test suite. Test case documents consist of a list of test steps with their statuses, descriptions, expected and actual results, additional technical information and screenshots.
I began from the very lowest level. This is a class which describes a test step – one single action. Every step consists of description, expected and actual results, status, start time and related screenshot (if necessary). Also it may have a list of substeps.
The next level is a test case. Its attributes are: name, name of test suite (collection) which the test belongs to, description, start time and duration time, status, test scope – a description of steps which should be performed in the scope of the test case, and, of course, a list of test steps. Also I added a couple of auxiliary variables: current step and current step index.
The upper level is a test run session. This is one session of testing the application. It must have a name, description, start time and a list of test suites (which contain test cases for running in the scope of the session). There is also a current test case as an auxiliary variable.
The highest level of test results reporting is a report. This is the class which has methods for conducting the test runs and test cases. The report class generates final documents and saves them.
Thus I created four classes in the Logger project. The first class is Report. All report creation logic is described inside it. It’s declared as public static. I also added the enumerator Status to this class. The Status is a list of possible statuses for test steps and test cases.
Report:
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
namespace Logger
{
public static class Report
{
internal enum Status
{
Pass, Fail,
NoRun, Warning
}
}
}
TestStep:
using System;
using System.Collections.Generic;
namespace Logger
{
internal class TestStep
{
internal TestStep() { SubSteps = new List<TestStep>(); }
internal List<TestStep> SubSteps;
internal Report.Status Status { get; set; }
internal string Description { get; set; }
internal string Tooltip { get; set; }
internal string ExpectedResult { get; set; }
internal string ActualResult { get; set; }
internal string ScreenshotPath { get; set; }
internal DateTime StartTime { get; set; }
}
}
TestCase:
using System;
using System.Collections.Generic;
namespace Logger
{
internal class TestCase
{
internal TestCase()
{
TestScope = new Dictionary<string, string>();
TestSteps = new List<TestStep>();
CurrentStepIndex = 0;
}
internal TestStep CurrentStep;
internal Dictionary<string, string> TestScope;
internal List<TestStep> TestSteps;
internal Report.Status Status { get; set; }
internal string TestCollectionName { get; set; }
internal string Description { get; set; }
internal TimeSpan Duration { get; set; }
internal string Name { get; set; }
internal DateTime StartTime { get; set; }
internal int CurrentStepIndex { get; set; }
}
}
TestSession:
using System;
using System.Collections.Generic;
namespace Logger
{
internal class TestSession
{
internal TestSession() { TestCases = new List<TestCase>(); }
internal TestCase CurrentTestCase;
internal List<TestCase> TestCases;
internal string Name { get; set; }
internal string Description { get; set; }
internal DateTime StartTime { get; set; }
}
}
This is the description of how tests execution process should work:
1. Since the Report class is static, its static constructor is invoked (I’m going to add it later)
2. GlobalSetup.SetUpAssembly() method is invoked. It is executed only once. I set up here the instance of the report.
3. BaseTest.TestFixtureSetUp() or BaseTest.TestCollectionSetUp(). These methods are invoked before start of the first test inside current test suite (class marked with [TestFixture] / [TestClass]). At the moment the methods do nothing.
4. BaseTest.Setup() is kicked off. The method starts right before every single test execution. The instance of IWebDriver interface is initialized here (new browser window is launched).
5. Example.ATestMethod() marked as [Test] or [TestMethod] starts. I perform the testing here.
6. BaseTest.Teardown() is invoked. It does its work after current test is finished. The IWebDriver is destroyed (the browser is closed)
7. Clauses 4, 5 and 6 are repeated as many time as many test methods have been kicked off for execution from inside Example class
8. BaseTest.TestFixtureTearDown() or BaseTest.TestCollectionTearDown() method is performed. For now both have nothing to do.
9. If there are other test classes (similar to Example) and their tests are sent for running, the clauses 3-8 will repeat.
10. GlobalSetup.TearDownAssemnly() finishes the testing. The test report is generated and saved.
I must say that the inner logic of Report class puts definite rules on the context of test methods:
● Test methods contain a test scope. It is not obligatory but recommended to add. This is a description of the test case steps in the following format – “step description”, “expected result” – Dictionary. It is done for few reasons: first, a test case will contain a description of its flow that is good and useful for anyone who will read the test in future; second, the scope is used for counting performed test steps and allows to determine which steps are performed and which are stayed in NoRun state.
● Every new step inside a test must be invoked explicitly to make the step being logged into the report correctly. Therefore, I have to invoke next test step as many times as many steps are declared in the scope.
If you have more elegant solution please let me know. Below is my implementation of the Report class.
First of all I added the following variables to the class:
/// <summary>
/// an instance of a test execution model
/// </summary>
private static readonly TestSession TestSession = new TestSession();
/// <summary>
/// path to a folder on a local drive for saving the report
/// </summary>
public static string ReportFolderPath;
Then I overrode class constructor by default. In C# there may be only one instance of a static class which is created automatically by CLR when the application starts. But C# allows creating custom constructor by default for static classes and managing the class state as well.
/// <summary>
/// constructor by default
/// </summary>
static Report()
{
TestSession.StartTime = DateTime.Now;
}
The constructor remembers the time the test session starts.
Report class methods:
StartTestSession() method should be visible outside the project and will be invoked explicitly before any test has started – from inside SetUpAssembly() method. It sets values to TestSession fields, initializes path to a folder for saving the report and creates the folder.
/// <summary>
/// Initialize variables general for current test session
/// </summary>
/// <param name=”TestSessionName”>Name of current test session</param>
/// <param name=”TestSessionDescription”>Description of current test session</param>
public static void StartTestSession(string TestSessionName, string TestSessionDescription = null)
{
TestSession.Description = TestSessionDescription ?? string.Empty;
TestSession.Name = string.IsNullOrEmpty(TestSessionName) ? “Undefined” : TestSessionName;
ReportFolderPath = Directory.GetCurrentDirectory() + “\\Reports\\” + TestSession.Name + “_” + TestSession.StartTime.ToString(CultureInfo.InvariantCulture).Replace(“:”, string.Empty).Replace(“/”, string.Empty).Replace(” “, string.Empty);
Directory.CreateDirectory(ReportFolderPath);
}
StartTestCase() method initializes TestSession.CurrentTestCase variable with new TestCase instance. This instance is a model of currently started test. The method should be explicitly invoked from a test case method because the method parameters are unique for every single test.
/// <summary>
/// Initialize variables of current test case
/// </summary>
/// <param name=”collection”>Name of test collection which the test case belongs to</param>
/// <param name=”name”>Name of test case</param>
/// <param name=”description”>Summary description of the test case</param>
/// <param name=”testScope”>The test scope (description of test case steps in format ‘step action, expected result’)</param>
public static void StartTestCase(string collection, string name, string description, Dictionary<string, string> testScope)
{
TestSession.CurrentTestCase = new TestCase
{
StartTime = DateTime.Now,
TestCollectionName = collection,
Name = string.IsNullOrEmpty(name) ? DateTime.Now.ToString(CultureInfo.InvariantCulture).Replace(“:”, string.Empty).Replace(“/”, string.Empty).Replace(” “, string.Empty) : name,
Description = description ?? string.Empty,
TestScope = testScope ?? new Dictionary<string, string>{ {“No test scope is declared for the test case”, string.Empty} }
};
}
SaveCurrentTestStep() is private method used only inside the Report class. It adds current test step to the list of test steps of current test case. When the step is saved in the list of steps the CurrentStep variable may be re-initialized with new value (new step becomes current).
/// <summary>
/// adds current test step to the list of steps of current test case
/// </summary>
private static void SaveCurrentTestStep()
{
var currentStep = TestSession.CurrentTestCase.CurrentStep;
if (currentStep == null)
return; //if there is no current step so there is nothing to save
//currently this path is empty because there is no method for saving screenshots yet.
currentStep.ScreenshotPath = string.Empty;
//Status.NoRun is the initial status of a step.
//If there is no substeps it means the step haven’t been performed and its state is No run.
if (!currentStep.Status.Equals(Status.Warning))
currentStep.Status = currentStep.SubSteps.Count > 0 ? Status.Pass : Status.NoRun;
//Sets the screenshot of the last substep as the step’s screenshot which displays status of the application after step is executed
if (currentStep.SubSteps.Count > 0)
currentStep.ScreenshotPath = currentStep.SubSteps.Last().ScreenshotPath;
//looking through all substeps.
//If any of the substeps is marked as Warning the step is marked the same,
//if any is marked Fail, the step status is Fail.
//Otherwise the step status remains Pass.
foreach (var subStep in currentStep.SubSteps)
{
if (!subStep.Status.Equals(Status.Pass))
{
if (subStep.Status.Equals(Status.Warning))
{
currentStep.ActualResult = subStep.ActualResult ?? “Warning”;
currentStep.Status = Status.Warning;
if (!TestSession.CurrentTestCase.Status.Equals(Status.Fail))
TestSession.CurrentTestCase.Status = Status.Warning;
}
else
{
currentStep.ActualResult = subStep.ActualResult ?? “Error”;
currentStep.Status = Status.Fail;
TestSession.CurrentTestCase.Status = Status.Fail;
//If any of the substeps is failed its screenshot will be displayed as the step’s screenshot
currentStep.ScreenshotPath = subStep.ScreenshotPath;
break;
}
}
}
if (string.IsNullOrEmpty(currentStep.ActualResult))
{
currentStep.ActualResult = “As expected”;
}
//current step is saved to the list of steps of current test case
TestSession.CurrentTestCase.TestSteps.Add(currentStep);
}
RunStep() method creates new current test step instead of the step that have been saved into the list of test case steps. The method is invoked from test methods. It must be invoked the same number of times as there is number of items in the test scope.
/// <summary>
/// creates new test step and makes it current
/// </summary>
/// <param name=”customDescription”>Description of test step. Overrides the value stored in the test case scope</param>
public static void RunStep(string customDescription = null)
{
SaveCurrentTestStep();
var testCase = TestSession.CurrentTestCase;
//verifies if the index of the step is within the number of steps defined in the test scope.
//If True, the values of the scope item with the same index are used.
//Apart the step is described with warning message
bool isTestScopeValid = testCase.CurrentStepIndex <= testCase.TestScope.Count – 1;
//new step
testCase.CurrentStep = new TestStep
{
Description = isTestScopeValid ? testCase.TestScope.ElementAt(testCase.CurrentStepIndex).Key : “No description provided for the test step”,
Tooltip = customDescription ?? string.Empty,
ExpectedResult = isTestScopeValid ? testCase.TestScope.ElementAt(testCase.CurrentStepIndex).Value : string.Empty,
StartTime = DateTime.Now,
Status = Status.NoRun
};
testCase.CurrentStepIndex++;
}
Methods AddInfo(), AddWarning(), AddError() add a passed, marked as Warning or failed substep to the current test step. They may be invoked all over the solution.
/// <summary>
/// adds new passed substep to the list of substeps of current step
/// </summary>
/// <param name=”description”>Substep description</param>
/// <param name=”expectedResult”>Expected result</param>
/// <param name=”screenshotPath”>Path to related screenshot file</param>
public static void AddInfo(string description, string expectedResult = null, string screenshotPath = null)
{
TestSession.CurrentTestCase.CurrentStep.SubSteps.Add(new TestStep
{
Description = description ?? string.Empty,
ExpectedResult = expectedResult ?? string.Empty,
ScreenshotPath = screenshotPath ?? string.Empty,
ActualResult = “As expected”,
StartTime = DateTime.Now,
Status = Status.Pass
});
}
/// <summary>
/// adds new substep with warning status to the list of substeps of current step
/// </summary>
/// <param name=”description”>Substep description</param>
/// <param name=”expectedResult”>Expected result</param>
/// <param name=”actualResult”>Actual result</param>
/// <param name=”screenshotPath”>Path to related screenshot file</param>
public static void AddWarning(string description, string expectedResult = null, string actualResult = null, string screenshotPath = null)
{
TestSession.CurrentTestCase.CurrentStep.SubSteps.Add(new TestStep
{
Description = description ?? string.Empty,
ExpectedResult = expectedResult ?? string.Empty,
ScreenshotPath = screenshotPath ?? string.Empty,
ActualResult = actualResult ?? string.Empty,
StartTime = DateTime.Now,
Status = Status.Warning
});
}
/// <summary>
/// adds new failed substep to the list of substeps of current step
/// </summary>
/// <param name=”description”>Substep description</param>
/// <param name=”expectedResult”>Expected result</param>
/// <param name=”actualResult”>Actual result</param>
/// <param name=”screenshotPath”>Path to related screenshot file</param>
public static void AddError(string description, string expectedResult = null, string actualResult = null, string screenshotPath = null)
{
TestSession.CurrentTestCase.CurrentStep.SubSteps.Add(new TestStep
{
Description = description ?? string.Empty,
ExpectedResult = expectedResult ?? string.Empty,
ScreenshotPath = screenshotPath ?? string.Empty,
ActualResult = actualResult ?? string.Empty,
StartTime = DateTime.Now,
Status = Status.Fail
});
}
FinishTestCase() adds current test to the list of tests that have been executed during current test session.
/// <summary>
/// saves current test case to the list of test cases of the test session
/// </summary>
public static void FinishTestCase()
{
SaveCurrentTestStep();
TestSession.CurrentTestCase.Duration = DateTime.Now – TestSession.CurrentTestCase.StartTime;
TestSession.TestCases.Add(TestSession.CurrentTestCase);
}
ReportStep(), ReportTestCase(), ReportTestSuite()
These three methods are private and are used by SaveReport() method for generating report. The methods takes data from the instance of the TestSession and generate HTML table with these data. Every method is designed for different kind of data: test steps, test cases and test suites (test collections).
/// <summary>
/// Saves data of certain test step as text in HTML format
/// </summary>
/// <param name=”report”>target HTML document to which the formatted data will be appended</param>
/// <param name=”step”>test step for save</param>
/// <param name=”isSubstep”>indicated is the step parent (main) step or a substep</param>
/// <param name=”stepName”>step name</param>
/// <returns>String: modified input parameter report with test step data appended</returns>
private static string ReportStep(string report, TestStep step, bool isSubstep = false, string stepName = null)
{
string color, status;
string background = isSubstep ? “#F1F1F1” : “#DBFAFC”;
string display = isSubstep ? “display: none; ” : string.Empty;
string onclick = step.SubSteps.Count > 0 ? ” onclick=’toggle(\”” + step.Description + “\”);'” : string.Empty;
string className = string.IsNullOrEmpty(stepName) ? string.Empty : ” class='” + stepName + “‘”;
report += “<tr” + className + ” style='” + display + “background: ” + background + “;'” + onclick + “>” + “\r\n”;
if (step.Status == Status.Fail)
{
color = “red”;
status = “Fail”;
}
else if (step.Status == Status.Warning)
{
color = “orange”;
status = “Warning”;
}
else if (step.Status == Status.Pass)
{
color = “green”;
status = “Pass”;
}
else
{
color = “grey”;
status = “No run”;
}
report += “<td>” + step.StartTime + “</td>” + “\r\n”;
report += “<td style=’color: ” + color + “‘>” + status + “</td>” + “\r\n”;
report += “<td title='” + step.Tooltip + “‘>” + step.Description + “</td>” + “\r\n”;
report += “<td>” + step.ExpectedResult + “</td>” + “\r\n”;
report += “<td>” + step.ActualResult + “</td>” + “\r\n”;
report += “<td>” + “\r\n”;
if (!string.IsNullOrEmpty(step.ScreenshotPath))
report += “<a href='” + step.ScreenshotPath + “‘>Screenshot</a>” + “\r\n”;
report += “</td>” + “\r\n” + “</tr>” + “\r\n”;
return step.SubSteps.Aggregate(report, (current, substep) => ReportStep(current, substep, true, step.Description));
}
/// <summary>
/// Saves data of certain test case as text in HTML format
/// </summary>
/// <param name=”report”>target HTML document to which the formatted data will be appended</param>
/// <param name=”testCase”>test case for save</param>
/// <returns>String: modified input parameter report with test case data appended</returns>
private static string ReportTestCase(string report, TestCase testCase)
{
string color, status;
const string background = “#F1F1F1”;
report += “<tr style=’background: ” + background + “;’>” + “\r\n”;
if (testCase.Status == Status.Fail)
{
color = “red”;
status = “Fail”;
}
else if (testCase.Status == Status.Warning)
{
color = “orange”;
status = “Warning”;
}
else
{
color = “green”;
status = “Pass”;
foreach (var step in testCase.TestSteps)
{
if (step.Status.Equals(Status.NoRun))
{
color = “orange”;
status = “Warning”;
break;
}
}
}
report += “<td>” + testCase.StartTime + “</td>” + “\r\n”;
report += “<td style=’color: ” + color + “‘>” + status + “</td>” + “\r\n”;
report += “<td>” + “\r\n”;
report += “<a href='” + testCase.Name + “.html” + “‘style=’color: ” + color + “;’>” + testCase.Name + “</a>” + “\r\n”;
report += “</td>” + “\r\n” + “</tr>” + “\r\n”;
return report;
}
/// <summary>
/// Saves data of certain test collection as text in HTML format
/// </summary>
/// <param name=”report”>target HTML document to which the formatted data will be appended</param>
/// <param name=”collectionName”>test collection name</param>
/// <returns>String: modified input parameter report with test collection data appended </returns>
private static string ReportTestSuite(string report, string collectionName)
{
string color;
const string background = “#F1F1F1”;
report += “<tr style=’background: ” + background + “;’>” + “\r\n”;
var collection = from t in TestSession.TestCases where t.TestCollectionName == collectionName select t;
var list = collection as IList<TestCase> ?? collection.ToList();
var passed = from l in list where l.Status.Equals(Status.Pass) select l;
var warning = from l in list where l.Status.Equals(Status.Warning) select l;
var failed = from l in list where l.Status.Equals(Status.Fail) select l;
var numberOfFailed = failed.Count();
if ((double)numberOfFailed / list.Count >= 0.5)
color = “red”;
else if ((double)numberOfFailed / list.Count >= 0.25)
color = “orange”;
else if ((double)numberOfFailed / list.Count >= 0.1)
color = “yellow”;
else
color = “green”;
report += “<td>” + “\r\n”;
report += “<a href='” + collectionName + “.html” + “‘ style=’color: ” + color + “;’>” + collectionName + “</a>” + “\r\n”;
report += “</td>” + “\r\n”;
report += “<td>” + list.Count() + “</td>” + “\r\n”;
report += “<td>” + (passed.Count() + warning.Count()) + “</td>” + “\r\n”;
report += “<td>” + failed.Count() + “</td>” + “\r\n” + “</tr>” + “\r\n”;
return report;
}
And finally the method which generates the report and saves it to local drive – SaveReport().This method is invoked once in the very ending of the test session from TearDownAssembly() method.
/// <summary>
/// generates final summary report and saves it to local drive
/// </summary>
public static void SaveReport()
{
string reportBody, filePath;
//distinct list of test collections’ names in the report
List<string> testCollectionsInReport = TestSession.TestCases.GroupBy(x => x.TestCollectionName).Select(y => y.First().TestCollectionName).ToList();
StreamWriter sw;
foreach (var testCollectionName in testCollectionsInReport)
{
string name = testCollectionName;
var testCases = from row in TestSession.TestCases where row.TestCollectionName == name select row;
//creates document for every test case
var enumerable = testCases as IList<TestCase> ?? testCases.ToList();
foreach (var testCase in enumerable)
{
if ((testCase.TestScope.Count > testCase.TestSteps.Count) && !testCase.Status.Equals(Status.Fail))
testCase.Status = Status.Warning;
reportBody =
“<!DOCTYPE HTML PUBLIC ‘ -//W3C//DTD HTML 4.01//EN’ ‘http://www.w3.org/TR/html4/strict.dtd’>” + “\r\n” +
“<html>” + “\r\n” +
“<head>” + “\r\n” +
“<meta charset=’utf-8′ />” + “\r\n” +
“<meta content=’text/html; charset=utf-8′ />” + “\r\n” +
“<title>” + testCase.Name + ” test result” + “</title>” + “\r\n” +
“<style>td{border: 0; padding: 0 1em; }</style>” + “\r\n” +
“</head>” + “\r\n”;
reportBody +=
“<body>” + “\r\n” +
“<h2>” + testCase.Name + ” test result – ” + testCase.StartTime + “</h2>” + “\r\n” +
“<div style=’border: solid #DBFAFC 2px;” + ” padding: 1em; width:35%; border-radius:10px;’>” + “\r\n” +
“<h4>” + “Test steps:” + “</h4>” + “\r\n” +
“<ul>” + “\r\n”;
reportBody = testCase.TestScope.Aggregate(reportBody, (current, scope) => current + (“<li>” + scope.Key + ” – ” + scope.Value + “</li>” + “\r\n”));
reportBody += “</ul>” + “\r\n” +
“<br/>” + “\r\n” +
“<h5>” + “Duration: ” + testCase.Duration.ToString(@”hh\:mm\:ss”) + “</h5>” + “\r\n” +
“</div>” + “\r\n” +
“<br/><br/>” + “\r\n” +
“<div style=’padding: 1em;’>” + “\r\n” +
“<table>” + “\r\n” +
“<tr style=’font-weight: bold; font-size:large; align: center;’>” + “\r\n” +
“<td>Time</td>” + “\r\n” +
“<td>Status</td>” + “\r\n” +
“<td>Description</td>” + “\r\n” +
“<td>Expected result</td>” + “\r\n” +
“<td>Actual result</td>” + “\r\n” +
“<td>Screenshot</td>” + “\r\n” +
“</tr>” + “\r\n”;
reportBody = testCase.TestSteps.Aggregate(reportBody, (current, step) => ReportStep(current, step));
reportBody = (from scope in testCase.TestScope let stepDone = testCase.TestSteps.FirstOrDefault(x => x.Description == scope.Key) where stepDone == null select scope).Aggregate(reportBody, (current, scope) => ReportStep(current, new TestStep { Description = scope.Key, Status = Status.NoRun }));
reportBody +=
“</table>” + “\r\n” +
“</div>” + “\r\n” +
“<script type=’text/javascript’>function toggle(id){var element = document.getElementsByClassName(id);i = element.length;while(i–){element[i].display = \”none\”;if (element[i]) {var display = element[i].style.display;if (display == \”none\”){element[i].style.display = \”\”;} else{element[i].style.display = \”none\”;}}}}</script>” + “\r\n” +
“<script type=’text/javascript’>function addImgAndReplaceA(){ var links = document.getElementsByTagName(‘a’); for(var i = 0; i < links.length; i++){ var a = document.createElement(‘a’); var pathToFolder = links[i].getAttribute(‘href’).replace(/.+(?=\\ScreenShots)/, ‘.\\\\’); a.innerHTML = ‘<img src=\”‘+ pathToFolder +’\” width=90 height=60>’; links[i].parentNode.replaceChild(a, links[i]); links[i].addEventListener(‘mouseover’, showImg, false); links[i].addEventListener(‘mouseout’, unshowImg, false); links[i].addEventListener(‘click’, openImgInTab, false); } } function openImgInTab(e){ e = e || event; window.open(e.target.src,’_blank’); } function showImg(e){ e = e || event; var div = document.createElement(‘div’); div.id = ‘screenshot’; div.style.cssText=’display: block; position: fixed; right: 10%; top:10%; border: solid 3px black;’; div.innerHTML = ‘<img src=’ + e.target.src + ‘ style=\” max-width:’ + document.body.clientWidth * 0.8 + ‘px; max-height:’ + document.body.clientHeight * 0.8 + ‘px;\”>’; document.body.appendChild(div); } function unshowImg(){ var element = document.getElementById(‘screenshot’); element.parentNode.removeChild(element);} window.onload = addImgAndReplaceA; </script>” + “\r\n” +
“</body>” + “\r\n”;
reportBody +=
“</html>” + “\r\n”;
filePath = ReportFolderPath + “\\” + testCase.Name + “.html”;
sw = File.CreateText(filePath);
sw.Write(reportBody);
sw.Close();
}
//creates document for every test suite
List<string> testCaseNames = enumerable.Select(x => x.Name).ToList();
testCaseNames.Sort();
var startTime = enumerable.Select(x => x.StartTime).Min();
reportBody =
“<!DOCTYPE HTML PUBLIC ‘ -//W3C//DTD HTML 4.01//EN’ ‘http://www.w3.org/TR/html4/strict.dtd’>” + “\r\n” +
“<html>” + “\r\n” +
“<head>” + “\r\n” +
“<meta charset=’utf-8′ />” + “\r\n” +
“<meta content=’text/html; charset=utf-8′ />” + “\r\n” +
“<title>” + testCollectionName + ” test result” + “</title>” + “\r\n” +
“<style>td{border: 0; padding: 0 1em; }</style>” + “\r\n” +
“</head>” + “\r\n”;
reportBody +=
“<body>” + “\r\n” +
“<h2>” + testCollectionName + ” test result – ” + startTime + “</h2>” + “\r\n” +
“<br/><br/>” + “\r\n” +
“<div style=’padding: 1em;’>” + “\r\n” +
“<table>” + “\r\n” +
“<tr style=’font-weight: bold; font-size:large; align: center;’>” + “\r\n” +
“<td>Time</td>” + “\r\n” +
“<td>Status</td>” + “\r\n” +
“<td>Test case</td>” + “\r\n” +
“</tr>” + “\r\n”;
reportBody = enumerable.Aggregate(reportBody, ReportTestCase);
reportBody +=
“</table>” + “\r\n” +
“</div>” + “\r\n” +
“<script type=’text/javascript’>function toggle(id){var element = document.getElementsByClassName(id);i = element.length;while(i–){element[i].display = \”none\”;if (element[i]) {var display = element[i].style.display;if (display == \”none\”){element[i].style.display = \”\”;} else{element[i].style.display = \”none\”;}}}}</script>” + “\r\n” +
“</body>” + “\r\n” +
“</html>” + “\r\n”;
filePath = ReportFolderPath + “\\” + testCollectionName + “.html”;
sw = File.CreateText(filePath);
sw.Write(reportBody);
sw.Close();
}
reportBody =
“<!DOCTYPE HTML PUBLIC ‘ -//W3C//DTD HTML 4.01//EN’ ‘http://www.w3.org/TR/html4/strict.dtd’>” + “\r\n” +
“<html>” + “\r\n” +
“<head>” + “\r\n” +
“<meta charset=’utf-8′ />” + “\r\n” +
“<meta content=’text/html; charset=utf-8′ />” + “\r\n” +
“<title>” + TestSession.Name + ” test result” + “</title>” + “\r\n” +
“<style>td{border: 0; padding: 0 1em; }</style>” + “\r\n” +
“<script type=’text/javascript’ src=’https://www.google.com/jsapi’></script>” + “\r\n” +
“<script type=’text/javascript’>google.load(‘visualization’, ‘1.0’, {‘packages’:[‘corechart’]});google.setOnLoadCallback(drawChart);function drawChart() {var data = new google.visualization.DataTable();data.addColumn(‘string’, ‘Topping’);data.addColumn(‘number’, ‘Slices’);data.addRows([[”, 0],[‘Failed’, ” + TestSession.TestCases.Count(x => x.Status == Status.Fail) + “],[‘Warnings’, ” + TestSession.TestCases.Count(x => x.Status == Status.Warning) + “],[‘Passed’, ” + TestSession.TestCases.Count(x => x.Status == Status.Pass) + “]]);var options = {‘title’:’Logger 1.0 test result – 5/5/2015 8:13:04 PM’, ‘width’:400, ‘height’:300};var chart = new google.visualization.PieChart(document.getElementById(‘chart_div’));chart.draw(data, options);}</script>” + “\r\n” +
“</head>” + “\r\n”;
reportBody +=
“<body>” + “\r\n” +
“<h2>” + TestSession.Name + ” test result – ” + TestSession.StartTime + “</h2>” + “\r\n” +
“<br/><br/>” + “\r\n” +
“<div id=’chart_div’></div>” + “\r\n” +
“<div style=’padding: 1em;’>” + “\r\n” +
“<table>” + “\r\n” +
“<tr style=’font-weight: bold; font-size:large; align: center;’>” + “\r\n” +
“<td>Test collection</td>” + “\r\n” +
“<td>Tests executed</td>” + “\r\n” +
“<td>Tests passed</td>” + “\r\n” +
“<td>Tests failed</td>” + “\r\n” +
“</tr>” + “\r\n”;
reportBody = testCollectionsInReport.Aggregate(reportBody, ReportTestSuite);
reportBody +=
“</table>” + “\r\n” +
“</div>” + “\r\n” +
“<script type=’text/javascript’>function toggle(id){var element = document.getElementsByClassName(id);i = element.length;while(i–){element[i].display = \”none\”;if (element[i]) {var display = element[i].style.display;if (display == \”none\”){element[i].style.display = \”\”;} else{element[i].style.display = \”none\”;}}}}</script>”;
reportBody +=
“</body>” + “\r\n” +
“</html>” + “\r\n”;
filePath = ReportFolderPath + “\\index.html”;
sw = File.CreateText(filePath);
sw.Write(reportBody);
sw.Close();
}
After the solution is built the file named Logger.dll appears in the bin/Debug folder of the Logger project. This is the entire Logger project compiled in a single file. For being able to use this library in any other project you must have the Logger.dll fiel and add reference to it in the project. Thus, I added the reference in the Tests project to Logger.dll library.
Usage of Report class
Below the explanation of how to use Report class and its methods in tests is given.
At the moment I have the following three classes in the Tests project developed earlier:
● GlobalSetup – container for methods TearDownAssembly() and SetUpAssembly() which are invoked once in the very beginning and the very end of the test session.
● BaseTest – base class for all test classes (marked with [TestFixture]/[TestClass] attribute and with test methods inside). Contains auxiliary methods used in testing which are not test cases.
● Example – test class which represents a test collection, child of the BaseTest. It contains two test methods (test cases) – GoogleSearch() and GoogleAccountTitleVerification() marked with [Test]/[TestMethod] attributes.
I want to update a bit these test cases to show how the reporting works. First step is updating of GlobalSetup class:
using Logger;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using MSTest = Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Tests
{
[SetUpFixture]
[TestClass]
public class GlobalSetup
{
[SetUp]
public static void SetUpAssembly()
{
//is invoked once in the very beginning of a test session.
//Logger 1.0 is a name of the test session.
//The name will be displayed in the title of main report document.
Report.StartTestSession(“Logger 1.0”);
}
[AssemblyInitialize()]
public static void SetUpAssembly(MSTest.TestContext context)
{
SetUpAssembly();
}
[TearDown]
[AssemblyCleanup()]
public static void TearDownAssembly()
{
//is invoked once in the very end of a test run
Report.SaveReport();
}
}
}
Second step – update of BaseTest class.
I put FinishTestCase() method to TearDown() method. The method is common for all test case methods in the project and works similar with any of them, therefore it may be invoked from TearDown() which is invoked every time a test method finishes the work.
public class BaseTest
{
. . .
[TearDown]
[TestCleanup]
public void Teardown()
{
Report.FinishTestCase();
driver.Quit();
}
}
Third step. The Example class requires changes of its test methods to be more suitable for new requirements provided by Logger project.
using System;
using NUnit.Framework;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
using nUnit = NUnit.Framework;
using MSTest = Microsoft.VisualStudio.TestTools.UnitTesting;
using System.Collections.Generic;
using Logger;
using System.Reflection;
namespace Tests
{
[TestFixture][TestClass]
public class Example : BaseTest
{
[Test][TestMethod]
public void GoogleSearch()
{
string searchTerm = “selenium”;
//test scope
Dictionary<string, string> scope = new Dictionary<string, string>();
scope.Add(“Navigate to Google search page”, “Google search page is loaded”);
scope.Add(“Do nothing”, “”);
scope.Add(“Perform search on the term “ + searchTerm, “Title of search result page starts with “ + searchTerm);
// Run StartTestCase method and pass test suite name, test case name, test case description and declared above test scope to it as parameters. The method is invoked inside the test method because its parameters are unique for every single test. That’s why it cannot be invoked from BaseTest.SetUp() method which is automatically invoked before every test method start. Parameters: this.GetType().Name – name of class and test collection – Example, MethodBase.GetCurrentMethod().Name – name of test case – GoogleSearch
Report.StartTestCase(this.GetType().Name, MethodBase.GetCurrentMethod().Name, “Google Search”, scope);
// Run the first test step. This method creates new instance of a test step and all further actions and events will be logged into this test steps as its substeps until the method is invoked one more time (and new current test step is created). The number of steps must match the number of test scope items.
Report.RunStep();
driver.Navigate().GoToUrl(“http://www.google.com”);
// add a substep to the current step. You can add as many as you need substeps to the current step
Report.AddInfo(“Navigation to Google Search page”);
// Run the next test step
Report.RunStep();
// Add a substep
Report.AddInfo(“No action”);
// Run the last test step
Report.RunStep();
IWebElement query, btnSearch;
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
query = driver.FindElement(By.Name(“q”));
query.Clear();
query.SendKeys(“selenium”);
btnSearch = driver.FindElement(By.Name(“btnG”));
btnSearch.Click();
// Add a substep
Report.AddInfo(“Search is performed”, “Search result page is opened”);
wait.Until(d => { return d.Title.StartsWith(“selenium”); });
nUnit.Assert.IsTrue(driver.Title.StartsWith(“selenium”));
// Add a substep
Report.AddInfo(“Assertion”);
}
[Test][TestMethod]
public void GoogleAccountTitleVerification()
{
// Invokation of StartTestCase with the test scope declared anonymously
Report.StartTestCase(this.GetType().Name, MethodBase.GetCurrentMethod().Name, “Google Account Verification”, new Dictionary<string, string> {{ “Open Google Accounts page”, “Title of Google Accounts page is ‘Sign in – Google Accounts'” }});
Report.RunStep();
driver.Navigate().GoToUrl(“https://accounts.google.com/”);
nUnit.Assert.AreEqual(“Sign in – Google Accounts”, driver.Title);
Report.AddInfo(“nUnit verification”);
}
}
}
As an example I added one more test class to the Tests project and named it TestingLogger. Cap, this is for testing the Logger class. Here is its code:
using System;
using System.Collections.Generic;
using System.Reflection;
using Logger;
using NUnit.Framework;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Tests
{
[TestFixture][TestClass]
public class TestingLogger : BaseTest
{
[Test][TestMethod]
public void ErrorTest()
{
Report.StartTestCase(this.GetType().Name, MethodBase.GetCurrentMethod().Name, “Error test”, new Dictionary<string, string> { { “Test with error message”, “Test fails” } });
Report.RunStep();
Report.AddError(“Error message”, “Error is logged”);
Report.AddInfo(“nUnit verification”);
Report.AddInfo(“MSTest verification”);
}
[Test][TestMethod]
public void WarningTest()
{
Report.StartTestCase(this.GetType().Name, MethodBase.GetCurrentMethod().Name, “Warning test”, new Dictionary<string, string>
{
{ “Test with warning message”, “Test fails” }
});
Report.RunStep();
Report.AddWarning(“Warning message”, “Warning is logged”);
Report.AddInfo(“nUnit verification”);
Report.AddInfo(“MSTest verification”);
}
[Test][TestMethod]
public void NoRunStepsTest()
{
Report.StartTestCase(this.GetType().Name, MethodBase.GetCurrentMethod().Name, “Test with NoRun steps”, new Dictionary<string, string>
{
{ “Step 1”, “Pass” },
{ “Step 2”, “No run” },
{ “Step 3”, “No run” },
{ “Step 4”, “No run” }
});
Report.RunStep();
Report.AddInfo(“message”, “”);
Report.AddInfo(“nUnit verification”); Report.AddInfo(“MSTest verification”);
}
}
}
I believe the code does not require additional comments.
Now you can build the solution (Build -> Build Solution menu item) and run the tests. I think there are mustn’t be any problems with running the tests under Visual Studio with Test Explorer (Test -> Windows -> Test Explorer).
After the solution is built, new file Tests.dll is compiled in bin/Debug folder of the Tests project.
NUnit tool. Select File -> Open Project. In appeared explorer window browse to the Tests.dll file and open it. The project will be opened in NUnit tool and you will see all your test suites with all their tests inside.
Run all the tests – Tests -> Run All or F5. After all the tests are finished you can see test result report. It is located in bin/Debug/Reports folder of the Tests project.
The name of the folder consists of the last test session name and time it has started. Inside the folder you will find all the documents related to this test session: index.html – main document, one document per every test collection and one document per every test case. Open index.html and you can see general results of the tests run. If you navigate to a test collection document you will see the results of running tests within this collection. From here you can navigate to any test case document and see result of a concrete test case run.
Zipping summary report and sending it by email
After the tests execution is complete and summary report is generated it must be delivered to stakeholders – other testers, customer, and superiors. I decided that sending the report by email is the easiest way to do this. With email I can send the report whatever I wish. It is very useful because all of the stakeholders may promptly see the report and also you do not need to search for the report on the local drive, especially if the tests have been run on remote machine with limited access. Just check your inbox and you will see the report.
In C# there are special classes for sending emails – SmtpClient and MailMessage. They are delivered with .NET library and are located in the namespace System.Net.Mail. System.Net namespace must be included to the code for using the class NetworkCredentials.
Since the report is packed to ZIP archive before sending by email, a reference to System.IO.Compression.FileSystem library must be added as well.
For the purpose I created the following method and put it into the Report class:
using System;
using System.Collections.Generic;
using System.Globalization;
using System.IO;
using System.Linq;
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Net.Mail;
namespace Logger
{
public static class Report
{
. . .
/// <summary>
/// Compresses summary test report into zip archive
/// </summary>
/// <returns>String: path to created archive</returns>
private static string ZipReport()
{
string zipFilePath = ReportFolderPath + “.zip”;
try
{
ZipFile.CreateFromDirectory(ReportFolderPath, zipFilePath);
}
catch (Exception ex)
{
Report.AddWarning(“An error has occurred during archiving of summary report. Error message: ” + ex.Message);
zipFilePath = string.Empty;
}
return zipFilePath;
}
//emailSubject – subject of the email
//emalTo – list of recipients’ addresses whom the letter must be sent to
//emailBody – content of the email, in my case it will be index.html file with summary result for all test collections (though you may write here whatever you want, for example “Hi, guys! Here are the tests execution results. Happy analyzing! :))
//emailAttachmentPaths – optional parameter, list of paths to other attachments if you want to put them into the email
/// <summary>
/// Sends summary test report to specified e-mail addresses
/// </summary>
/// <param name=”emailSubject”>e-mail subject</param>
/// <param name=”emailTo”>List of addresses of recipients</param>
/// <param name=”emailBody”>e-mail body</param>
/// <param name=”emailAttachmentPaths”>List of paths to files to be attached to the e-mail</param>
public static void SendReportByEmail(string emailSubject, ICollection<string> emailTo, string emailBody, IEnumerable<string> emailAttachmentPaths = null)
{
//this is an address which the letter will be sent from. The address must be real and actual
const string senderEmail = “[email protected]”;
const string senderPassword = “password_to_sender_email”;
if (emailTo == null || emailTo.Count == 0)
{
Report.AddWarning(“Sending test result report by email”, “Email has been sent to specified addresses”, “Email address is not specified”);
return;
}
try
{
//referring to smtp client of the email provider (I use Google). You can know the host and port names from help documentation or support service. The sender’s address and password are sent to smtp client here for establishing the connection
var smtp = new SmtpClient(“s mtp.gmail.com”, 587)
{
Credentials = new NetworkCredential(senderEmail, senderPassword),
EnableSsl = true
};
//an instance of the email is created
var email = new MailMessage
{
From = new MailAddress(senderEmail),
Subject = emailSubject,
Body = emailBody,
IsBodyHtml = true,
SubjectEncoding = System.Text.Encoding.UTF8,
BodyEncoding = System.Text.Encoding.UTF8,
};
//adding recipients’ addresses
foreach (var address in emailTo)
email.To.Add(new MailAddress(address));
//adding attachments
if (emailAttachmentPaths != null)
foreach (var path in emailAttachmentPaths)
email.Attachments.Add(new Attachment(path));
email.Attachments.Add(new Attachment(ZipReport()));
smtp.Send(email);
}
catch (Exception ex)
{
Report.AddWarning(“Sending summary report by email”, “Email has been sent to specified addresses”, “An error has occurred while sending the email. Error message: ” + ex.Message);
}
}
}
}
The method is invoked after all the tests have been executed and the report is created and saved. Therefore, it must be invoked in GlobalSetup.TearDownAssembly() method right after the Report.SaveReport() method is worked off.
To be able to use the last changes in the Tests project the Logger project must be recompiled.
[SetUpFixture][TestClass]
public class GlobalSetup
{
. . .
[TearDown][AssemblyCleanup()]
public static void TearDownAssembly()
{
Report.SaveReport();
string subject = “Test report – session name”;
string[] addresses = { “[email protected]” };
string body = File.ReadAllText(Report.ReportFolderPath + “\\index.html”);
Report.SendReportByEmail(subject, addresses, body);
}
}
This is a report I’ve got on my email:
Report’s main page is displayed as email body and the rest of the pages and files are delivered in the attachment. So I can see test execution result in general directly from email message and if I want to know details I can download the attachment.
[row]
[column lg=”4″ md=”12″ sm=”12″ xs=”12″ ]
Simple Example of a Test [/column]
[column lg=”4″ md=”12″ sm=”12″ xs=”12″ ]
Table Of Content
[/column]
[column lg=”4″ md=”12″ sm=”12″ xs=”12″ ]
Event Logging
[/column]
[/row]