The standard JUnit tests are fine, but sometimes you want to try something specific. Here’s how.
Repeat after us: JUnit 5 is not a test runner. JUnit 5 is a platform. The JUnit 5 platform was designed from the ground up to solve one problem: to separate the development of test runners from their integration with IDEs and build tools. To this end, JUnit 5 introduces the concept of a test engine. This two-part series will answer the following questions:
◉ What is a JUnit test engine, and why would you want to build one?
◉ How do you build a very simple one?
◉ What do you have to do to integrate your engine with IDEs and build tools? (spoiler: almost nothing)
In part 1 of this “build your own test engine” (BYOTE) series, you will see how a minimal test engine can be implemented, and in part 2, you will see how it can be tested. To learn from the authors of several well-known test engines, part 2 of this series will contain interviews about their experiences in building real-world engines.
Why would anyone build a custom test engine?
Have you ever wanted to build your own test engine? On the JUnit platform, this is easier than you might think. To show off the flexibility of the JUnit platform, you will develop a completely declarative test engine.
Developing functionality is all fun and games, but how do you actually test a test engine? It turns out that the JUnit platform brings tools for exactly that.
But what good is your custom engine if it is hard to execute and get reports for the results? No one will want to use it—probably not even you. Fortunately, the JUnit platform was designed from the ground up to solve this exact problem: Your test engine will be executable in all major IDEs and build tools without almost any effort on your part!
So, apart from curiosity, why would you even want to create a custom engine? Here are three possible reasons.
◉ You want a different testing model than what you can find elsewhere, such as one that focuses on performance, mutation, or property-based testing.
◉ You want improved support for the idioms of a different JVM language such as Groovy, Kotlin, or Clojure.
◉ You want a Java-based tool but with a custom syntax to support the specific requirements of a certain problem domain.
You, of course, might have other reasons, so let’s see how to build a custom engine. Before that, however, a bit of architectural background about JUnit 5 is essential.
JUnit 5 101
Contrary to public opinion, JUnit 5 is a platform rather than a simple test runner. To understand the need for such a platform, it is helpful to take a look at some JUnit history.
Early versions of JUnit were not designed with tool integration in mind. Due to the wild success of those early versions, however, IDEs and build tools began integrating JUnit into their environments quite soon. However, because JUnit was not designed to be executed from third-party processes, integrators sometimes had to rely on dangerous mechanisms, such as reflection, to make things work.
Such integrations were naturally brittle and were very hard to change and maintain because the outside world relied upon, and was coupled to, JUnit’s internals—even private members. An infamous example for this coupling is the breaking of the integration in a widely used IDE when a JUnit 4.x version changed the name of a private field. This situation was only exacerbated by other test libraries and frameworks mimicking JUnit’s structure to leverage JUnit’s IDE integration.
Therefore, while planning and designing the new generation of JUnit, the JUnit team took special care to avoid such coupling. As a matter of fact, the team invited representatives from all major IDEs and build tools to a kick-off meeting in October 2015 to discuss a solid foundation for an integration architecture. The main goal was to provide different APIs for different groups of users, namely
◉ An engine service provider interface (SPI) (junit-platform-engine) for integrators of existing test engines and authors of new test engines
◉ A launcher API (junit-platform-launcher) for IDEs and build tools to provide stable and transparent means of integrating all test engines
◉ An annotation-based API (junit-jupiter-api) for test authors, which would be similar in look and feel to the JUnit 4.x generation but with a more flexible extension model
The first two are the core components of the JUnit 5 platform, and Figure 1 shows the essential parts of the JUnit 5 platform and several engines built upon it.
Figure 1. The JUnit 5 platform architecture
This clear separation of concerns is fundamental to the new architecture and has so far served well to avoid the problematic coupling. As a consequence, the JUnit platform has been integrated into all major IDEs and build tools using the launcher API.
Similarly, the platform engine SPI has been used to integrate various test engines with the JUnit 5 platform. This allows test engine authors to concentrate on the test definition and execution concerns and completely disregard the aspect of launching their test cases from IDEs and build tools. A special case is, of course, JUnit’s own Jupiter test engine, which uses the new JUnit 5 test model. However, even this built-in engine uses only the public API and has no secret access to other parts of JUnit 5.
To show how easy it is for engine authors to integrate tools with the JUnit platform, you’ll develop a (very) small test engine from scratch, named WebSmokeTest.
WebSmokeTest: The world’s smallest test engine
For most JVM languages, basing the test model on classes and methods seems a natural fit. However, other test definition models are possible in principle—as long as individual tests can be described unambiguously by an org.junit.platform.engine.TestDescriptor implementation.
Typically, a test descriptor implementation has to be provided by the test engine author and represents a testable element of the given programming model, such as a single test (FieldTestDescriptor) or a container of tests (ClassTestDescriptor).
For the purpose of this article, we want to implement an engine that is both small and lightweight. Hence, we could contemplate alternatives to a method-based test model. One option might be to use lambdas assigned to fields—because lambdas can be considered a form of lightweight methods. This is possible, since we could use java.util.function.Predicate implementations using the aptly named method boolean test(T t) for test execution and the boolean as a success/failure indicator.
While such lambdas might be a bit more lightweight than full-blown methods, maybe we can take this a bit further. We will do so soon, but let’s consider the domain in detail first.
The problem domain: Web smoke testing
As mentioned in the introduction, there can be different reasons for creating a custom engine. Here, you want to support the requirements of a special domain in such a way that creating tests becomes very simple for the test author. As a simple yet still useful example, we pick the domain of HTTP-based smoke tests—in this case, a test that merely tests whether a URL works. Specifically, you need
◉ A succinct way to define the URL against which the smoke test should be executed
◉ A simple means to specify the expected HTTP status code
Consider the test successful if an HTTP GET request against the specified URL results in the specified HTTP status code. As for behavior, the degrees of freedom are extremely limited: WebSmokeTest always does the same thing, namely, execute an HTTP request against a certain server.
Since this test does not need variations in behavior, you can get rid of methods with explicitly programmed statements entirely (or lambdas, for that matter). You can choose a completely declarative approach: The HTTP URLs will be fields of type String. Because you might want to add support for POST and other verbs later, model the HTTP verb as an annotation, and model the expected status code as an argument of the annotation. Hence, a complete smoke test would look like as follows:
@GET(expected = 200)
String shouldReturn200 = "https://blogs.oracle.com/javamagazine/";
The actual implementation described below adds the public and static modifiers—but this is for implementation simplicity only and has no real bearing on the test model.
Agreements must be kept
When you design a custom test engine, it is important to understand the contract between a test engine and the JUnit platform. This contract is defined by the org.junit.platform.engine.TestEngine interface, would looks as follows:
String getId();
TestDescriptor discover(EngineDiscoveryRequest discoveryRequest, UniqueId uniqueId);
void execute(ExecutionRequest request);
Apart from some optional metadata, this interface contains three core responsibilities of a test engine. These core responsibilities are
◉ Identification: The engine must provide a unique string by which it can be referenced and differentiated from other engines on the classpath. For example, Jupiter’s ID is junit-jupiter.
◉ Discovery: The engine must be able to inform the JUnit platform about everything it considers its own test cases. These are arranged in a tree structure, with the returned TestDescriptor being the root node.
◉ Execution: When requested by the platform, the engine must be able to execute a given ExecutionRequest. This request contains a TestDescriptor referencing the tests to be run and provides an EngineExecutionListener to the engine. The former was previously returned by the engine’s own discover method. The latter is then used by the engine’s implementation to fire test lifecycle events, such as executionStarted() or executionFinished().
In many cases, test engines will support a hierarchical test definition model—mirroring Java’s hierarchical structure of packages, classes, and methods. The JUnit platform provides special support for such hierarchical engines in the form of the org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine base class. JUnit’s own Jupiter test engine is itself an extension of this base class. By using this support, advanced features, such as parallel execution, can be readily reused by custom engines. However, for the sake of simplicity and clarity WebSmokeTest will not use this base class; it will implement the required interfaces directly.
Implementing the minimal test engine
A test engine usually consists of the engine class itself and one or more TestDescriptor implementations. In this example, there are also two annotations. This is in no way mandatory—when designing this exercise, we happened to choose an annotation-based testing model. For such implementations, the JUnit platform provides good support with the org.junit.platform.commons.support.AnnotationSupport utility. Listing 1 shows the implementation of the custom test engine.
Listing 1. Implementation of the custom test engine
public class WebSmokeTestEngine implements TestEngine {
private static final Predicate<Class<?>> IS_WEBSMOKE_TEST_CONTAINER
= classCandidate -> AnnotationSupport.isAnnotated(classCandidate, WebSmokeTest.class);
@Override
public String getId() {
return "websmoke-test";
}
@Override
public TestDescriptor discover(EngineDiscoveryRequest request, UniqueId uniqueId) {
TestDescriptor engineDescriptor = new EngineDescriptor(uniqueId, "Web Smoke Test");
request.getSelectorsByType(ClasspathRootSelector.class).forEach(selector -> {
appendTestsInClasspathRoot(selector.getClasspathRoot(), engineDescriptor);
});
request.getSelectorsByType(PackageSelector.class).forEach(selector -> {
appendTestsInPackage(selector.getPackageName(), engineDescriptor);
});
request.getSelectorsByType(ClassSelector.class).forEach(selector -> {
appendTestsInClass(selector.getJavaClass(), engineDescriptor);
});
return engineDescriptor;
}
private void appendTestsInClasspathRoot(URI uri, TestDescriptor engineDescriptor) {
ReflectionSupport.findAllClassesInClasspathRoot(uri, IS_WEBSMOKE_TEST_CONTAINER, name -> true) //
.stream() //
.map(aClass -> new ClassTestDescriptor(aClass, engineDescriptor)) //
.forEach(engineDescriptor::addChild);
}
private void appendTestsInPackage(String packageName, TestDescriptor engineDescriptor) {
ReflectionSupport.findAllClassesInPackage(packageName, IS_WEBSMOKE_TEST_CONTAINER, name -> true) //
.stream() //
.map(aClass -> new ClassTestDescriptor(aClass, engineDescriptor)) //
.forEach(engineDescriptor::addChild);
}
private void appendTestsInClass(Class<?> javaClass, TestDescriptor engineDescriptor) {
if (AnnotationSupport.isAnnotated(javaClass, WebSmokeTest.class)) {
engineDescriptor.addChild(new ClassTestDescriptor(javaClass, engineDescriptor));
}
}
@Override
public void execute(ExecutionRequest request) {
TestDescriptor root = request.getRootTestDescriptor();
new SmokeTestExecutor().execute(request, root);
}
}
In Listing 1, you can see integration, discovery, and execution at work. The discover(EngineDiscoveryRequest, UniqueId) method creates an engine descriptor and adds test descriptors hierarchically, while the EngineDiscoveryRequest gives access to several implementations of org.junit.platform.engine.DiscoverySelector. Such selectors can reference various structural elements of Java (such as methods, classes, packages, and the whole classpath) or the file system (files, directories), as well as JUnit’s own UniqueId instances. The test engine uses these to indicate to the requesting tool (such as an IDE) what it considers test cases associated with every such element according to its specific test model.
The last method in Listing 1 takes care of test execution: execute(ExecutionRequest request) accepts an ExecutionRequest and delegates most of the actual work to a helper class named SmokeTestExecutor. In this class, the various TestDescriptor variants are handled and the individual tests are executed. The most important parts are shown in Listing 2.
Listing 2. The execution of a single test
private void executeTest(ExecutionRequest request, FieldTestDescriptor fieldTestDescriptor) {
request.getEngineExecutionListener().executionStarted(fieldTestDescriptor);
TestExecutionResult executionResult = executeTestField(fieldTestDescriptor);
request.getEngineExecutionListener().executionFinished(fieldTestDescriptor, executionResult);
}
private TestExecutionResult executeTestField(FieldTestDescriptor descriptor) {
Field testField = descriptor.getTestField();
try {
int expected = getExpectedStatusCode(testField);
String url = getUrl(testField);
HttpResponse<String> response = this.executeHttpRequest(url);
int actual = response.statusCode();
if (expected != actual) {
var message = String.format("expected HTTP status code %d but received %d from server", expected, actual);
return TestExecutionResult.failed(new AssertionFailedError(message, expected, actual));
}
} catch (Exception e) {
return TestExecutionResult.failed(new RuntimeException("Failed to execute HTTP request", e));
}
return TestExecutionResult.successful();
}
The executeTest method is responsible for sandwiching the actual execution call between lifecycle events. The executeTestField method retrieves the URL and the expected status code, and then it actually executes the HTTP request via a (purely technical) helper method. If a response is received, the method checks the actual status code and creates a TestExecutionResult.failed() with an AssertionFailedError in case the response does not meet the expectation. If sending the request throws an exception, it is wrapped in a more technical RuntimeException and a TestExecutionResult.failed() result. If all goes well, the method returns a TestExecutionResult.successful() result. This basically is the complete custom test engine.
The following example annotates the test class with @WebSmokeTest to identify the test classes. It contains three separate test cases defined by simple fields, and each field is annotated with @GET denoting the need to execute an HTTP GET request. (Supporting POST or other HTTP verbs would be just as simple.)
The target URL of the request is specified in the value of the field. As mentioned above, you have complete freedom in the new test engine regarding how to design a test model. You might have used methods, as most traditional test engines do. Or you might have specified the expected value as a method return type. All are perfectly valid options. Listing 3 shows three test cases.
Listing 3. Three test cases for the new engine
@WebSmokeTest
public class ExampleWebSmokeTests {
@GET(expected = 200)
public static String shouldReturn200 = "https://blogs.oracle.com/javamagazine/";
@GET(expected = 401)
public static String shouldReturn401 = "https://httpstat.us/401";
@GET(expected = 201)
public static String expect201ButGet404 = "https://httpstat.us/404";
}
Now that you have an engine implementation and a number of sample tests in place, you only need to let the platform (and hence, IDEs and build tools) know about the new test engine.
Test engine integration
This test engine class can be immediately executed in an IDE such as IntelliJ IDEA if two conditions are fulfilled. First, you need the test engine code on the classpath (typically in a Maven/Gradle dependency). Second, the new engine must be registered with the JUnit platform. This is done via Java’s well-known
SPI. To this end, a special file must be present on the classpath, as you can see in Figure 2.
Figure 2. The JUnit configuration file using the Java SPI mechanism
For the SPI mechanism to work, the filename must specify the SPI, here named org.junit.platform.engine.TestEngine. The file contains only one line, the fully qualified name (FQN) of the main engine class, which in this case is org.example.websmoke.engine.WebSmokeTestEngine. This FQN must refer to a class implementing the interface specified by the filename. When the main engine class is executed in the IDE, the test result should look as shown in Figure 3.
Figure 3. Execution of the main engine class in the IDE
As can be seen in Figure 3, the project uses domain-specific custom display names to reference the individual test cases in the GUI of the executing IDE. Thus, in this case, you can even display the whole test definition (the URL and the expected status code) right next to the IDE symbol indicating success or failure. Such flexibility is enabled by the TestDescriptor model, freeing the IDE from the need to mechanically display plain method or field names in all cases.
Source: oracle.com
Posts
0 comments:
Post a Comment