Posted by Daniel Bartl

Adaptive Test Execution With Gradle And Spring

One of the customizations that we wanted to have as part of our regular build process is to be able to detect which required external services are available in the concrete build execution environment at build time and to adapt the test execution accordingly. This is especially important in regards to the medium and large tests (as defined by google) you might want to execute as part of your build.

Test Classification By Google

It’s medium and large tests that will fail and lead to a broken build in case some of the required external services are not available at build execution time. Having a broken build of this kind actually reflects 2 things:

  1. That your build process, and not necessarily your software package that it’s trying to build, is currently broken.
  2. That the software package that could be built in case you skip the failing medium and large tests possibly does not fulfill all quality requirements that your build process usually requires and should be marked as such accordingly.

We simply wanted to have a build process that is able to adapt according to the execution environment at hand and always provide us with the best possible build under given circumstances. Maybe it’s a beautiful sunny day and you want to work in a park on your notebook where your build execution environment might possibly be limited (your “wellness” environment might likely be improved though ;) ) or you simply don’t want to have a completely broken build while some other member of your team is trying to get the database online again.

Buildtime Environment Detection

Thanks to the flexibility of Gradle and a single nice feature of Spring TestContext framework creating this kind of adaptive build process was very simple indeed.

The following example will show how you can easily verify whether your database instance (in our case MongoDB) is available at build execution time and how you can adapt your build execution accordingly.

First step is to define a dependency of our build script to a library that will be used in order to verify whether required database instance is currently running:

buildscript {

    dependencies {
        classpath group: 'com.gmongo', name: 'gmongo', version: '0.7'
    }

    repositories {
        mavenCentral()
    }
}

We then define an additional method as part of our build script that will use previously defined library accordingly:


/**
 * Verifies what environment services required for successful test execution are available at buildtime:
 * <ul>
 *     <li>
 *         Verifies whether there is an according MongoDB instance running in the local test environment
 *         listening on the default port of MongoDB: 27017.
 *         Note: GMongo driver initialization fails with IOException wrapped inside of
 *         com.mongodb.MongoInternalException in case there is no running MongoDB instance found.
 *     </li>
 * <ul>
 *
 * @return a Map with according entry for each of the required environment services.
 */
Map defineEnvironmentForTestExecution() {
    def env = new HashMap()
    try {
        new GMongo()
        env.put("CI-MongoDBAvailable", "true")
    } catch (Throwable throwable) {
        env.put("CI-MongoDBAvailable", "false")
    }
    return env
}

Now a Map of identified environment properties has to be included into a Map of system properties defined for each test execution (here defined for multi-module gradle build as part of the subprojects configuration):


subprojects {
   envProps = defineEnvironmentForTestExecution()

    test {
        systemProperties = envProps
    }
}

Additionally, we customize the manifest creation since we want to include the information about the environment that the package has been built in into the MANIFEST file of the package itself (here defined for one of the subprojects being packaged as a WAR archive):


    war {
        manifest {
            attributes envProps
        }
    }

Since envProps has been defined in subprojects section previously, it can be referenced in each of the subprojects accordingly. This will ensure that all information about the environment that the artifact has been created in, will be described in its META-INF/MANIFEST.MF file:

Manifest-Version: 1.0
CI-MongoDBAvailable: false

The only code change that we have to apply in order to get our adaptive test execution that reacts upon the envProps that have been calculated and included into a list of system properties is to annotate our tests with the according @IfProfileValue annotation provided by Spring framework:


@ContextConfiguration(locations = "classpath:WEB-INF/spring-mvc-context.xml")
@RunWith(SpringJUnit4ClassRunner.class)
public class SpringRestContextTest {

    @Autowired
    MongoTemplate mongoTemplate;

    @Test
    @IfProfileValue(name = "CI-MongoDBAvailable", value = "true")
    public void testSpringRestContextInitialization() {
        Assert.assertNotNull(mongoTemplate);
    }
}

All tests annotated as described above will now be executed only in case the according MongoDB instance is available in the current build execution environment.

Additionally to providing detailed information about the build environment in the MANIFEST file of the artifact itself, one could easily enough additionally mark the created artifact by modifying its very name to something like $YourArtifactBaseName.PARTIALLY.TESTED.ONLY.war or simply changing its version to 1.0.0.PARTIALLY.TESTED.SNAPSHOT for example.

You could basically mark the artifact in whatever ways you like, given of course that you have already migrated your build process to Gradle ;) .

Lost In Redirection With Apache Wicket

Quite a few teams have already got stuck into the following problem when working with wicket forms in a clustered environment while having 2 (or more) tomcat server with enabled session replication running.

In case of invalid data being submitted with a form instance for example, it seemed like according error messages wouldn’t be presented when the same form page gets displayed again. Sometimes! And sometimes they would! One of those nightmares of rather deterministic programmer’s life. This so called Lost In Redirection problem, even if it looks like a wicket bug at first, is rather a result of a default setting in wicket regarding the processing of form submissions in general. In order to prevent another wide known problem of double form submissions, Wicket uses a so called REDIRECT_TO_BUFFER strategy for dealing with rendering a page after web form’s processing (@see IRequestCycleSettings#RenderStrategy).

What does the default RenderStrategy actually do?

Both logical parts of a single HTTP request, an action and a render part get processed within the same request, but instead of streaming the render result to the browser directly, the result is cached on the server first.

Default Form Processing in Wicket

Wicket will create an according BufferedHttpServletResponse instance that will be used to cache the resulting HttpServletResponse within the WebApplication.

WebApplication Caching Buffered Responses

After the buffered response is cached the HTTP status code of 302 get’s provided back to the browser resulting in an additional GET request to the redirect URL (which Wicket sets to the URL of the Form itself). There is a special handling code for this case in the WicketFilter instance that then looks up a Map of buffered responses within the WebApplication accordingly. If an appropriate already cached response for the current request is found, it get’s streamed back to the browser immediately. No additional form processing happens now. The following is a code snippet taken from WicketFilter:

			// Are we using REDIRECT_TO_BUFFER?
			if (webApplication.getRequestCycleSettings().getRenderStrategy() == IRequestCycleSettings.REDIRECT_TO_BUFFER)
			{
				// Try to see if there is a redirect stored
				// try get an existing session
				ISessionStore sessionStore = webApplication.getSessionStore();
				String sessionId = sessionStore.getSessionId(request, false);
				if (sessionId != null)
				{
					BufferedHttpServletResponse bufferedResponse = null;
					String queryString = servletRequest.getQueryString();
					// look for buffered response
					if (!Strings.isEmpty(queryString))
					{
						bufferedResponse = webApplication.popBufferedResponse(sessionId,
							queryString);
					}
					else
					{
						bufferedResponse = webApplication.popBufferedResponse(sessionId,
							relativePath);
					}
					// if a buffered response was found
					if (bufferedResponse != null)
					{
						bufferedResponse.writeTo(servletResponse);
						// redirect responses are ignored for the request
						// logger...
						return true;
					}
				}
			}

So what happens in case you have 2 server running your application with session replication and load balancing turned on while using the default RenderStrategy described above?

Since a Map of buffered responses is cached within a WebApplication instance that does not get replicated between the nodes obviously, a redirect request that is suppose to pick up the previously cached response (having possibly form violation messages inside) potentially get’s directed to the second node in your cluster by the load balancer. The second node does not have any responses already prepared and cached for your user. The node therefore handles the request as a completely new request for the same form page and displays a fresh new form page instance to the user accordingly.

Clustering Without Sticky Sessions

Unfortunately, there is currently no ideal solution to the problem described above. The default RenderStrategy used by Apache Wicket simply does not work well in a fully clustered environment with load balancing and session replication turned on. One possibility is to change the default render strategy for your application to a so called ONE_PASS_RENDER RenderStrategy which is the more suitable option to use when you want to do sophisticated (non-sticky session) clustering. This is easily done in the init method of your own subclass of Wicket’s WebApplication :

    @Override
    protected void init() {
        getRequestCycleSettings().setRenderStrategy(
            IRequestCycleSettings.ONE_PASS_RENDER);
    }

ONE_PASS_RENDER RenderStrategy does not solve the double submit problem though! So this way you’d only be trading one problem for another one actually.

You could of course turn on the session stickiness between your load balancer (apache server) and your tomcat server additionally to the session replication which would be the preferred solution in my opinion.

Clustering With Sticky Sessions Turned On

Session replication would still provide you with failover in case one of the tomcat server dies for whatever reason and sticky sessions would ensure that the Lost In Redirection problem does not occur any more.

New Job Offer for München: JEE SPRING & APACHE WICKET ENTERPRISE SPECIALIST

When I was 5 years old, my mom always told me that happiness was the key to life.
When I went to school, they asked me what I wanted to be when I grew up.
I wrote down “happy.”
They told me I didn’t understand the assignment and I told them that they didn’t understand life.

Your Geek Crowd

Do you have a passion for java based open source software and related frameworks? Would you love to work in a small team of software experts striving to provide great experience to its users, longing for that final touch of expertise that will make the difference in your customer’s eyes. Do you enjoy learning about new technology and frameworks, going innovative ways together with your colleagues, coming up with great software solutions and sharing your knowledge with others frequently? We should get in touch!

As a JEE Spring & Apache Wicket Enterprise Specialist at comSysto, you will be working in a small team with talented UI designers and skilled developers, with experienced business intelligence experts and with Oracle database masters. You will be involved in all the technical discussions regarding our own product, searching for new innovative ideas of designing web 2.0 systems and making decisions about what is feasible and what is not. We require everyone to be open-minded and always be willing to learn new things. One of the many things you could do is to pick up good Spring framework skills over time. We will help you become SpringSource Certified Spring Professional with our training program and will help you develop your skills even further.

Your tasks and responsibility at comSysto:
• Work together with other developers, web UI designers, product owners and database masters
• Cooperate with responsible designers and architects to ensure great software design and quality
• Drive innovation by coming up with brave and surprising ideas for our own products and our customer projects
• Provide cool new e2e features as part of a team
• Identify and resolve issues with the existing architecture and code

Your basic skills:
• Excellent Java coding skills
• Good knowledge in popular open source technologies like Spring, Hibernate & Apache Wicket
• Ability to work in a team with excellent communication, advanced
problem solving and analytical skills

Your advanced skills:
• Knowledge of open source libraries, tools and frameworks
• Extensive understanding of web architectures, dependency injection containers and aspect-oriented programming
• Knowledge in some of the standard web UI technologies like CSS, JavaScript (JQuery, Spring JS, Dojo), REST, JSON, XML, Adobe Flash, Adobe Flex, Adobe BlazeDS
• Experience with agile software development methodologies like XP, TDD and Scrum
• Interest to learn more about new languages and frameworks, e.g. Adobe Flex, Groovy & Grails, Gradle, iPhone & Android Development
• Interest to learn more about several SpringSource sub-projects

Your experience:

• Ideally 3 or more years in a professional JEE role designing and developing java web applications
• Applications of interested graduates are welcome as well (given you already have some working knowledge of Java!)
• German language skills are essential for this position!

Please simply send your application to job[at]comsysto.com.

Geeks@Sailing

Java 4-ever

One of the most compelling dramas of our time ;)

Agiler Firmenausflug zum Bodensee

Anbei ein paar Fotos von unserem letzten Firmenausflug zum Bodensee. Es war ein wunderschöner Segelausflug und ein entspannendes, nicht allzu windiges, aber dafür ein sehr sonniges Wochenende, das wir auf 3 Segelbooten verbringen durften. Ungefähr genauso wie es auch sein sollte, damit alle Geeks wieder heil nach Hause zurückkommen ;)

Leider dürfen wir keine unserer vielen Videoaufnahmen hier veröffentlichen, weil wir inzwischen alle Rechte an diesen an Disney abgetreten haben. Als Gegenleistung hat Disney daraus aber diesen tollen Film für uns gemacht.

Test Driven Development With Apache Wicket And Spring Framework

Since the development of our own web applications is mostly based on a Spring framework for dependency injection and application configuration in general and Apache Wicket framework as a truly component-oriented web framework for the MVC part of building a web application, it’s especially important for us to get these two frameworks running together smoothly not only when deployed on a running server instance itself but rather during the execution of our JUnit based integration tests as well. Thanks to the WicketTester API provided by the Wicket framework itself, one can easily build high-quality web applications while practicing test driven development and providing a decent set of unit and integration tests to be executed with each build. As already mentioned previously, integration and configuration of our web applications is based on a lightweight Spring container meaning that the integration of Spring’s ApplicationContext and a WicketTester API is essential to get our integration tests running. In order to best explain how to achieve that integration in an easy and elegant fashion in your integration test environment, we’ll first take a look at a configuration of these 2 framework beauties in a runtime environment.

Configuration of the runtime environment

In order to get Wicket framework up to speed when your server is up and running, you usually configure an according WicketFilter instance in your web application deployment descriptor file (web.xml) while passing it a single init parameter called applicationClassName that points to your main implementation class extending org.apache.wicket.protocol.http.WebApplication where all of your application-wide settings and initialization requirements are dealt with:

<filter>
    <filter-name>wicketfilter</filter-name>
    <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
    <init-param>
        <param-name>applicationClassName</param-name>
        <param-value>com.comsysto.webapp.MyWebApplication</param-value>
    </init-param>
</filter>

In case you want to get Wicket application up and running while leaving the application configuration and dependency injection issues to the Spring container, the configuration to be provided within the deployment descriptor looks slightly different though:

<web-app>
    <filter>
        <filter-name>wicketfilter</filter-name>
        <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
        <init-param>
            <param-name>applicationFactoryClassName</param-name>
            <param-value>org.apache.wicket.spring.SpringWebApplicationFactory</param-value>
        </init-param>
    </filter>
    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>
    <context-param>
        <param-name>contextConfigLocation<param-name>
        <param-value>/WEB-INF/applicationContext.xml<param-value>
    </context-param>
</web-app>

The additional configuration part containing listener and context parameter definition is a usual, purely Spring container related configuration detail. ContextLoaderListener is an implementation of standard Servlet API ServletContextListener interface provided by the Spring framework itself and is responsible for looking up an according bean definition file(s) specified by the context param above and creating an ApplicationContext instance during servlet context initialization accordingly. When integrating an ApplicationContext instance with Wicket framework, one of the beans defined in the above mentioned Spring bean definition file has to be your own specific extension of org.apache.wicket.protocol.http.WebApplication. You can either define an according bean in the bean definition file itself:

<beans>
    <bean id="myWebApp" class="com.comsysto.webapp.MyWebApplication"/>
</beans>

or use powerful classpath scanning feature of the Spring framework and annotate the MyWebApplication implementation with the appropriate @Component annotation accordingly while enabling the Spring container to scan the according package(s) of your application for relevant bean definitions:

<beans>
    <context:component-scan base-package="com.comsysto.webapp" />
    <context:component-scan base-package="com.comsysto.webapp.service" />
    <context:component-scan base-package="com.comsysto.webapp.repository" />
</beans>

Either way, if everything goes well, you’ll get a pre-configured ApplicationContext all set up during the startup of your web container. One of the beans in the ApplicationContext will be your own extension of Wicket’s WebApplication type. SpringWebApplicationFactory implementation provided by the Wicket framework itself that you have defined as the applicationFactoryClassName in the configuration of your WicketFilter will then be used in order to retrieve that very same WebApplication bean out of your Spring ApplicationContext. The Factory expects one and only one extension of Wicket’s very own WebApplication type to be found within the ApplicationContext instance at runtime. If no such bean or more than one bean extending WebApplication is found in the given ApplicationContext an according IllegalStateException will be raised and initialization of your web application will fail:

Map<? , ?> beans = BeanFactoryUtils.beansOfTypeIncludingAncestors(ac,WebApplication.class, false, false);
if (beans.size() == 0)
{
	throw new IllegalStateException("bean of type [" + WebApplication.class.getName() +
			"] not found");
}
if (beans.size() > 1)
{
	throw new IllegalStateException("more than one bean of type [" +
			WebApplication.class.getName() + "] found, must have only one");
}

After the WebApplication bean has been successfully retrieved from the ApplicationContext via SpringWebApplicationFactory, WicketFilter will then, as part of its own initialization process, trigger both internalInit() and init() methods of the WebApplication bean. The latter one is the exact spot where the last piece of the runtime configuration puzzle between Wicket and Spring is to be placed :

@Component
public class MyWebApplication extends WebApplication {
    @Override
    protected void init() {
        addComponentInstantiationListener(new SpringComponentInjector(this));
    }

}

SpringComponentInjector provided by the Wicket framework enables you to get dependencies from the ApplicationContext directly injected into your Wicket components by simply annotating these with the according @SpringBean annotation.

Configuration of the JUnit based integration test environment

One of the main features of Apache Wicket framework is the ability to easily write and run plain unit tests for your Pages and all other kinds of Components that even include the verification of the rendering process itself by using JUnit framework and the WicketTester API only. When using Spring framework for application configuration together with Wicket, as we do, you can even use the same tools to easily write and run full blown integration tests for your web application as well. All you have to do is use Spring’s TestContext framework additionally to configure and run your JUnit based integration tests. The Spring Framework provides a set of Spring specific annotations that you can use in your integration tests in conjunction with the TestContext framework itself in order to easily configure an according ApplicationContext instance for your tests as well as for appropriate transaction management before, during and after your test execution. Following code snippet represents a simple JUnit 4 based test case using Spring’s specific annotations in order to initialize an ApplicationContext instance prior to executing the test itself:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:WEB-INF/applicationContext.xml"})
@TransactionConfiguration(transactionManager = "txManager", defaultRollback = false)
public class LoginPageTest {

    private WicketTester tester;

    @Autowired
    private ApplicationContext ctx;

    @Autowired
    private MyWebApplication myWebApplication;

    @Before
    public void setUp() {
        tester = new WicketTester(myWebApplication);
    }

    @Test
    @Transactional
    @Rollback(true)
    public void testRenderMyPage() {
        tester.startPage(LoginPage.class);
        tester.assertRenderedPage(LoginPage.class);
        tester.assertComponent("login", LoginComponent.class);
    }
}

By defining three annotations on the class level (see code snippet above) in your test, Spring’s TestContext framework takes care of preparing and initializing an ApplicationContext instance having all the beans defined in the according Spring context file as well as the transaction management in case your integration test includes some kind of database access. Fields marked with @Autowired annotation will be automatically dependency injected as well so that you can easily access and use these for your testing purposes. Since MyWebApplication, which extends Wicket’s WebApplication type and represents the main class of our web application, is also a bean within the ApplicationContext managed by Spring, it will also be provided to us by the test framework itself and can be easily used in order to initialize a WicketTester instance later on during the execution of the test’s setUp() method. With this kind of simple, annotation based test configuration we are able to run an integration test that verifies whether a LoginPage gets started and initialized, whether the rendering of the page runs smoothly and whether the page itself contains a LoginComponent that we possibly need in order to process user’s login successfully.

When you run this test though, you’ll get the following exception raised unfortunately:


java.lang.IllegalStateException: No WebApplicationContext found: no ContextLoaderListener registered?
	at org.springframework.web.context.support.WebApplicationContextUtils.getRequiredWebApplicationContext(WebApplicationContextUtils.java:84)
	at org.apache.wicket.spring.injection.annot.SpringComponentInjector.<init>(SpringComponentInjector.java:72)
	at com.comsysto.serviceplatform.uiwebapp.MyWebApplication.initializeSpringComponentInjector(MyWebApplication.java:59)
	at com.comsysto.serviceplatform.uiwebapp.MyWebApplication.init(MyWebApplication.java:49)
	at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:719)
	at org.apache.wicket.protocol.http.MockWebApplication.<init>(MockWebApplication.java:168)
	at org.apache.wicket.util.tester.BaseWicketTester.<init>(BaseWicketTester.java:219)
	at org.apache.wicket.util.tester.WicketTester.<init>(WicketTester.java:325)
	at org.apache.wicket.util.tester.WicketTester.<init>(WicketTester.java:308)

As you can see above, the Exception gets raised during the initialization of the WicketTester instance even before the actual test method gets executed. Even though we have applied rather cool and simple annotation based test configuration already described and passed in perfectly well prepared ApplicationContext instance to the WicketTester instance in the constructor, somewhere down the rabbit hole someone complained that no WebApplicationContext instance could have been found which seems to be required in order to initialize the WicketTester properly.

Illegal State Sequence

Illegal State Sequence

The problem that we run against here is due to the fact that SpringComponentInjector during its own initialization is trying to get hold of an according Spring’s ApplicationContext instance that would normally be there in a runtime environment but does not find any since we are running in a test environment currently. SpringComponentInjector delegates to Spring’s own WebApplicationContextUtils class to retrieve the instance of ApplicationContext out of the ServletContext which is perfectly fine for a runtime environment but is unfortunately failing in a test environment:

public static WebApplicationContext getRequiredWebApplicationContext(ServletContext sc)
		throws IllegalStateException {

	WebApplicationContext wac = getWebApplicationContext(sc);
	if (wac == null) {
		throw new IllegalStateException("No WebApplicationContext found: no ContextLoaderListener registered?");
	}
	return wac;
}

If you still remember we defined a ContextLoaderListener in our web.xml file as part of the configuration of our runtime environment that makes sure an according WebApplicationContext instance gets initialized and registered against the ServletContext properly. Luckily, this problem can easily be solved if we slightly change the way we initialize SpringComponentInjector in our main MyWebApplication class. Apart from the constructor that we have used so far, there is another constructor in the SpringComponentInjector class that expects the caller to provide it with an according ApplicationContext instance rather than trying to resolve one on its own:

public SpringComponentInjector(WebApplication webapp, ApplicationContext ctx,
		boolean wrapInProxies)
{
	if (webapp == null)
	{
		throw new IllegalArgumentException("Argument [[webapp]] cannot be null");
	}

	if (ctx == null)
	{
		throw new IllegalArgumentException("Argument [[ctx]] cannot be null");
	}

	// store context in application's metadata ...
	webapp.setMetaData(CONTEXT_KEY, new ApplicationContextHolder(ctx));

	// ... and create and register the annotation aware injector
	InjectorHolder.setInjector(new AnnotSpringInjector(new ContextLocator(), wrapInProxies));
}

In order to use this constructor instead of the one we used previously, we now obviously need to get hold of the ApplicationContext instance on our own in our MyWebApplication implementation. The easiest way to do this is to use Spring’s own concept of lifecycle callbacks provided to the beans managed by the Spring container. Since our MyWebApplication is also a bean managed by the Spring container at runtime (enabled by the classpath scanning and @Component annotation on a type level), we can declare it to implement ApplicationContextAware interface which ensures that it gets provided with the ApplicationContext instance that it runs in by the Spring container itself during startup.

public interface ApplicationContextAware {

	void setApplicationContext(ApplicationContext applicationContext) throws BeansException;

}

So the relevant parts of MyWebApplication type will now look something like the following code snippet:

@Component
public class MyWebApplication extends WebApplication implements ApplicationContextAware {
    @Override
    protected void init() {
        addComponentInstantiationListener(new SpringComponentInjector(this, ctx, true));
    }

    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        this.ctx = applicationContext;
    }
}

For additional clarification of how MyWebApplication now relates to both Wicket and Spring framework here is an according class diagram:

MyWebApplication class diagram

MyWebApplication Class Diagram

And that’s all folks! No additional modifications are required to the test itself. It’s gonna turn green now. This way you can use exactly the same Spring context configuration that you’d use in your runtime environment for running your JUnit based integration tests as well. Just grab the colleague you like most and try a pair programming and test first approach with Wicket and Spring. You’ll be delighted! And if you’re not, go buy yourself one of these ;) .

Tagged , , ,
Follow

Get every new post delivered to your Inbox.