Testing distributed JAVA enterprise applications is a big challenge. This difficulty is due to the fact that this kind of application is in most situations implemented using many architectural layers. There are:
The different components of the business logic layer are very likely to run in some sort of managed environment – an EJB container, a SERVLET container, a JMS listener or an RMI server – and access resources that are only available in that environment. Your testing procedure must absolutely take that into account in order to obtain reliable test results.
When testing components of a distributed software application, one must constantly have the following questions in mind:
In the paragraphs bellow I'll try to explain why these questions are important and how to deal with each situation. During the analysis, I'll also highlight requirements a good testing framework should satisfy in order to adapt to specific testing conditions.
In order to be able to better explain distributed testing requirements and patterns I'll consider a simple case study. This is a useful exercise because your testing strategy should be strongly based by the project's global architecture.
Let's consider you develop a product that targets two client categories, say SOHO and corporate. In order to address the two markets, you may want to ship two different configurations of your product:
The business logic is exactly the same, just the supported data volume and number of users differ. In both cases your product offers a browser based user interface, implemented using a SERVLET, to enable users to use the application via the web.
Since the business logic is the same you should only implement once. In this particular case, a good choice for implementing business services is as stateless singleton. This way, the implementation can be reused in different environments. You could call your services directly from a SERVLET or you could wrap an EJB around it.
Now, in order to address your different volumetric requirements, you must choose the right infrastructure for each of it.
In both cases, your business services – implemented as singletons – will delegate persistence operations – CRUD, finders – to persistence classes.
At application deployment time you'll create two packages, one for the SOHO clients (a WAR file) and one for the corporate clients (an EAR file).
The SOHO package will contain only the presentation layer, the service implementation factory, the singleton business service implementation and the persistence classes. The packaging procedure will also include a configuration file for the factory, to instruct it to return the singleton implementation of the services.
The corporate package will include, in addition to that, the EJB components and a configuration file to instruct the service implementation factory to return the EJB implementation of the business services. In order to provide for better performance, you may also want to provide a packaging that allows the SERVLET based presentation layer to run on one machine and the EJB based business services on another machine.
Before moving any further, I'd like to make sure we all agree on general testing principles. Let's assume we have a method that processes a data set, throws one or more checked exceptions, and may or may not return a result value. How many test do you have to write in order to properly test this method? Well, you'll need:
In order to determine how many success test you must write, you must analyze the algorithm your code fragment implements.
If your algorithm is “linear”, you'll need just one success test. I say an algorithm is linear if has the same behavior regardless of the value of its input parameters and regardless of the state of the processed data set.
If your algorithm “branches” you'll need one success test per possible path in your algorithm's decision graph. I say an algorithm “branches” if there is at least one decision point in it which induces a variation in the algorithm's behavior, based on the value of its input parameters or the state of the processed data set. A decision point is typically an if / else or switch / case construct.
The code fragment granularity we're usually testing at is a class method, so I'll use this term from now on.
The goal of success testing is to make sure that tested code works correctly under “normal” conditions, that is, when executed against a valid combination of input parameters and data items to be processed. Valid means that the data set and the input parameters would not cause your code fragment to throw any checked exception and not return any error code, if the algorithm is functioning properly.
If the code fragment you must test implements a linear algorithm, you can run a test case as described bellow:
If the method you must test implements a branching algorithm, the following pattern can be applied:
identify every possible path in your algorithm's decisional graph
for each possible path in the graph, run a test case as described bellow:
create a valid dataset in the execution environment and choose a set if input parameters which combined together will trigger that execution path in the algorithm. You can do this by combining multiple fixtures
invoke the method to be tested with the specified input parameters
analyze the returned value and/or the processed data set and decide if the algorithm was successful or not. If the results are not those expected, force the test to fail. If a checked exception was thrown, force the test to fail. If an error code was returned, for the test to fail.
remove from the execution environment every data item that was created for testing purposes – your fixtures should take care of that.
Failure testing only makes sense if the tested code fragment throws checked exceptions or returns error codes, the goal being to verify that exceptions and error codes are thrown / returned only when they're supposed to.
Failure testing is as important as success testing, as it insures that checked exceptions are thrown in every situation in which the algorithm requires it.
You'll notice that failure testing strategies are based on analyzing checked exceptions declared by your method and by recreating in the testing environment the conditions that would cause those exceptions to be thrown.
Sometimes, however, a method is forced to declare checked exceptions but you never throw that exception yourself. This is the case if your code is executed in some managed environment. The most obvious example that comes to mind is code running in EJB container or RMI servers. In these cases your methods must declare java.rmi.RemoteException, but you'll never throw RemoteException yourself, the container will throw it instead if an unexpected error or any kind is caught while your code is executed. The container will catch that error and throw a Remote exception instead, giving you a chance to deal with it. Such exceptions should not be taken into account in your failure testing analysis and should be treated as unchecked throwables.
When testing a linear algorithm implementation for failure, the following pattern can be applied:
identify every checked exception thrown by the code fragment and / or every error code that is returned
for each checked exception and for each error code, identify every combination of environment data set and input parameters that should cause that exception to be thrown or that error code to be returned. This shouldn't be very complicated in a linear algorithm: usually exceptions are thrown and error codes are returned inside “if” or “switch/case” constructs. By looking at the “if” and “switch” conditions, you should easily be able to identify data sets and input parameters that trigger it.
For each combination that you have identified
create the data set in the environment where the test will be executed – you can this by using fixtures
invoke the method to be tested with the specified input parameters
make sure the expected exception is thrown. If no exception is thrown, force the test to fail. If an exception other then the expected one is thrown, for the test to fail.
remove from the execution environment every data item that was created for testing purposes – your fixtures should take care of that.
Testing for failures in branching algorithms is a more complicated task, as the analysis phase required to identify data set / input parameters combinations that trigger exceptions or error code returns can be quite elaborated.
However, once you have identified data set and input parameters combinations that trigger exceptions or error codes, you should proceed as for an linear algorithm.
If your algorithm is really complicated and has a lot of decision points, it may be worth trying to adopt a “divide and conquer” strategy. The idea is to isolate code fragments from your tested method using “extract method” refactoring and test each extracted method independently. Then perform the remaining tests on the main method. Depending on how you code is designed and how your data is organized, this strategy may or may not be possible. If you feel it could help you, a good way to deal with it is to try to extract methods from within major if / else or switch / case constructs.
Client side tests should cover all code that is executed in the client JVM, as well as all code running in remote containers that is invoked from the client side.
Testing code that runs exclusively on the client side is the most common scenario, used by most test writers around the world. It's not a very complicated task because the tested code is completely contained within the JVM running the test and the data the tested code processes is accessible from within the client JVM. Because the test case has access to any resource the tested code has, the impact of the tested code on the computing environment can be easily introspected. As a result, it is very easy to determine if the code behaved as expected.
Remote facade testing is a very common pattern that appears when testing distributed applications and it appears when testing code that that provides services to remote user interfaces or are otherwise invoked by remote clients. Such code can be implemented as:
Session EJBs
Web services
RMI exposed classes
JMS listeners
etc
The particularity of this situation is that while the test case is executed within the client JVM, the actual tested code must run in a remote environment. This scenario has an impact on two phases of the test case execution:
when the testing environment is set up and teared down, data required by the test to run must be created /deleted on the remote environment. This means either data has to be accessed using data source configured on the application server, or a file has to be written to the local file system on the machine where the tested code runs, or something else has to be performed remotely.
After the tested code has returned, the test case must determine if the execution was successful. If the call returns a value or an object, you'll be able to compare that with the expected result and decide. But what if your code fragment doesn't return any value but just alters data in the remote environment and returns void? In this case the only way you can decide if the code was successful or not is to introspect that remote environment and make sure data was altered as expected.
There are two methods to introspect a remote environment:
fetch remote data that was supposed to be modified using public services published by the remote component you're testing and analyze it
send a “probe” in the remote environment, have it analyze the data and send back the result to the test case.
The first method is really simple and works with any framework. It has, however, an important disadvantage, which is it's lack of accuracy. Let's take the following example: you must test the “createObject” service. You call it, you don't get any exception or error code. Is that sufficient to make us reasonably sure the service works as it should? Not at all! Nothing tells us the object was really created on the database. From what we know if the method's body could be empty, we'd get the same results. We have to check if the object was properly persisted. Let's assume we call the “findObject” service for that. If the call is successful and the returned object is identical to the one we wanted to create, the test has succeeded. But what if the object is not the same or it's not found? Well, in this case, we've obviously detected an error, and the test case has fulfilled half of its mission. But are we able to tell where the problem lies? Not at all. The problem could be located in the create method, maybe it didn't create the object properly. Or maybe the error lies in the finder method, maybe it contains an error and doesn't fetch objects correctly. Or maybe the error list in the O/R mapping layer, and the O/R mapping framework is mapping fields on the wrong columns, or doesn't map it at all.
In order to obtain accurate results we must find a way to introspect the remote environment and see how it was modified. To achieve this, we could send in a “probe” to decide if the environment was modified by the “createObject” method as expected.
But what is this probe? Its a serializable class which implements a simple interface, like the following:
public interface Probe {public void check() throws ProbeException;}
It works like this: you write a class that implements the Probe interface and in the “check” method you write the code that investigates the environment and decides if it is in a coherent state. Since the check method will be executed in the remote environment it can access every resource available in that environment, such as files on the local file system, or data sources. If the result check is OK you just return, otherwise, you throw the exception to describe what is wrong. If your check method needs parameters, you can add those either by using a contructor or by implementing setters on the probe class.
In order to achieve this functionality you'll need a support agent in the remote environment. This support agent will receive the serialized probe, will deserialize it, execute it and return the results. For an EJB container, this support agent can be a method of a session bean that takes the probe as parameter and throws the exception that signals the error. The EJB method would just invoke the check method on its parameter and that's it.
This, however, looks quite complicated and you definitely shouldn't have to deal with all this complexity in your test cases. You shouldn't have to worry about writing the support agent and shouldn't even have to call the support agent yourself. The testing framework should do it for you. In a later chapter I'll explain what I expect from a good testing framework from this point of view.
“In container” testing is a technique that enables us to execute tests and test artifacts in remote managed environments, similar to the one where the code will be executed during production. This technique is necessary because code running in a managed environment – in an EJB container, for example – can have its behavior influenced by container settings, and it can also depend on resources that are accessible only from within the container.
In the case of an EJB container such resources can be data sources, JMS queues, the local file system and so on.
In order to perform “in container testing” we have to use the same principles as described for probes: an agent must be present in the remote environment. This agent will receive test cases, fixtures and guards in serialized form, execute it and send the results back to the test case runner for analysis.
Of course, just like for probes, all this complexity should be taken care of by the testing framework. If we want to test in an EJB container, for example, the testing framework should provide agents to reproduce different possible settings, like running the tests from an EJB in bean or container managed transactions mode.
The persistence layer is a very important aspect of any enterprise application. Since the business logic layer makes heavy use of it, it is very important to test it properly – and independently. If the persistence layer is properly tested it makes it easier to write business logic tests, because the analysis of potential errors is simplified – because we know problems can only come from the business logic layer and not from the persistence layer, which was already tested.
In order to agree on how a persistence layer should be tested, we should first agree on what a persistence layer is. Well, in the following chapters I'll assume a persistence layer is a collection of data and utility classes that conserve a time persistent state of your application. This way the application can be saved on a reliable storage system – a relational database, a directory system, a file on the file system. In a 100% EJB system, the persistence layer would be composed of the entity java beans and their home interfaces . Of course, those familiar with EJB will jump up to the sealing asking: but what about the remote interface? I don't consider the remote interface to be part of the persistence layer as the remote interface is just an accessory, an interface implemented by a stubs to invoke methods on persistent data objects – in this case, the entity beans themselves.
If we generalize, the persistence layer would be composed of persistent classes – these are classes that contain data that will actually be persisted – and classes that contain utility methods to manage the life cycle of the persistent data and method that enable us to find persistent classes according to various criteria.
If we maintain the comparison with the EJB technology, we can see that entity beans are part of the persistence layer, because they are persistent objects, as they contain the data which must be persisted. Home interfaces are part of the persistence layer, because stubs implementing these interfaces enable us to manage the persistent data's life cycle – create, read, update, delete procedures – and also enable us to retrieve persistent data according to various criteria dictated by the application's business logic.
Unless your application is a GUI one that talks directly to the database the persistence layer will be executed in a remote container and it will be called from within that container too. This kinda makes sense if you think about it: after all if your business logic is implemented in EJBs, that's where you're going to call your persistence layer from. And if your business logic is embedded in a SERVLET, then you'll probably call your persistence layer from the SERVLET.
In order to properly test your persistence layer, you must reproduce at testing time the same conditions as in production, and this is where in container testing comes in.
We can globally identify two main persistence strategies: in relational databases and in XML files. Relational databases are widely used to persist large volumes of relatively fine grained information items, which need frequent updates or very selective reads. XML files are generally used to persist small amounts of relatively static data. Good examples of such data are reference tables or application configuration data. The persistence testing strategy you should choose depends on these specificities.
The testing strategy for this kind of persistence layer should take into consideration two main aspects: the number of entities that must be persisted and for each entity, the number of persistence operations that can be performed. The typical persistence operations are: create, read, update, delete and a certain number of finders. The difference between the read operation and finder operations is that read always takes the entity's primary key as parameter and if successful it always returns exactly one entity. Finders, on the other hand, can take any number of parameters – generally not the primary key – and can return one or many entities.
First you should test for success: given a test environment in a known state, you must choose a set of parameters for the create method which would cause it to create create an entity that would not violate any integrity rule.
If any checked exception is raised, you should force your test to fail. If not, you should investigate the test environment to see if the entity was created as expected. If the environment in which the test has executed is remote – such as an EJB container – you may want to consider sending a probe.
If the entity was properly created, you can consider the test was a success. Otherwise you should force your test to fail and issue a meaningful error message. If you're really paranoid about your tests, you can even make sure that exactly one entity was created, and not many of it. This makes a good regression test as it could protect you against stupid errors such as accidentally calling the database library twice.
What exactly does it mean that the entity was created as expected? Well, you create the entity from a set of data – the set of parameters you've chosen at the beginning of this chapter. To test that your entity was properly created, you should compare every atomic information item – field – in the initial data set with its persisted counterpart. If every field matches, then everything is fine. If your persisted data set contains calculated values, you should make sure the result is of the required precision.
Don't forget the tear-down phase: if your entity was created – correctly or not – you must remove it at the end of your test case execution and leave the test execution environment in the same state as it was before the test started to execute.
With “create”, you should also have one or more failure tests. The most obvious one is to try to create an entity with a primary key that already exists. In this case you would use the test setup phase to create an entity and then, in the test itself, try to create another entity by setting the same primary key. Then you should check that the proper exception is raised or the proper error code is returned. If the expected exception is not raised or even worth, the entity is created the second time, you should force your test to fail.
Then you should use the tear-down method to clean the test environment.
Your entity may have to comply to a variety of database constraints: unique values, not null values, foreign key constraints, and so on. If so, you should have a failure test per such constraint and use the same pattern to test it: if necessary, use the set-up phase to create a test environment that would trigger a constraint violation, try to create your entity and make sure this attempt fails as expected and then tear-down the test environment.
The “read” tests set is typically more simple then the “create” one. For the success test, you have to prepare the test environment by creating a number of entities and remember one of it. Your test should then try to read that particular entity and check the following points:
the primary key of the retrieved entity is the same as the one used as a search key.
The attributes of the retrieved entity are the same as those of the entity that was created – except for special attributes such as time and data stamps.
If the entity was not found or if any of the checkpoints are not satisfied, you should force your test to fail.
In any case, don't forget to tear-down your test environment.
The failure test is quite simple: use the same fixture to create the same entities as for the success test and then try to read an entity using a primary key that you know doesn't exist. If “read” raises the expected exception or error code, the test is successful. Otherwise, you should force your test to fail. And again, tear-down your test environment.
Testing “update” is a little more complicated. You must create a certain number of entities in your test set-up phase. For that, you should reuse the fixtures from the “read” tests. You should pick one of the created entities, make a copy of it and then modify its attributes using legal value. Legal values means valid with respect to the constraints to which the entity must comply. The test shall then call update.
If any exception is raised, you should force your test to fail. If no exception is raised or error code returned, you should re-fetch the entity that you have updated, and compare its values with those you have modified. If the values are identical, the test was a success. Otherwise, you should force it to fail.
You should also make sure that the target entity is the only one that was modified. If other entities were modified too, your test should be forced to fail.
Failure test patterns are very close to those for testing “create”. The difference is that “update” should typically not let you change the primary key attribute. If that's true for your design, you should run a test to verify it. Then you should have at least one failure test case for each database constraint that applies to your entity, as discussed for “create”.
You should also have a failure test for the case where you try to update an entity that doesn't exist. In this case you should make sure that a proper exception / error code is raised and – if you feel its necessary – that no other entity was updated.
In the environment in which the test was executed is remote you might need to implement these checks using the probe mechanism.
At the end, you should tear down your test environment.
In order to test “delete” for success, you should create a number of entities, as for read and update. The test should pick an entity and call delete. If no checked exception is raised, you should make sure the targeted entity – and only that one – was deleted. If this is true, the test is successful, otherwise you should force it to fail.
At the end, you should tear down your test environment.
To test for failure, you should try to delete an entity that doesn't exist. If the proper exception is thrown and no other entity was removed, the test can be considered a success. Otherwise, you should force your test to fail.
Also, it may be possible that database constraints would make deleting an entity illegal. If this is the case, you should reproduce this situation and make sure the delete method informs you correctly about the situation.
Finders are usually quite tricky to test. The principle behind is the following: create a data set on which you will perform the find operation. The dataset should be composed of some data that satisfies your test's search criteria and some that doesn't. Remember the set of entities that are supposed to match your criteria. In your test, call the finder and make sure that the set of returned entities is exactly the same as set of entities that were created and which satisfy the search criteria. In order to do that you must check that the number of entities in both sets is the same and also that each created entity appears exactly once in the set of returned entities. For each found entity, should also check that its business values have the same attributes as the entities that were created. If this is true, your test is successful, otherwise you should force it to fail.
At the end of the test, you should not forget to tear down your test environment.
As previously explained, XML based persistence layers are typically more simple then relational database based ones. This technology is typically chosen to persist data that is completely loaded into memory when the application starts, and saved periodically. The CRUD operation and finders only modify “in memory” data. This is why this kind of persistence layer doesn't have complicated finders or granular CRUD operations. They mainly provide two methods: load save. These are typically more simple to test.
If the XML file resides on a remote machine – like the one where the application server or the RMI server runs, you may want to send a probe to introspect the remote environment.
In order to test “save” you should build a realistic data structure and ask the persistence layer to save it to disc. The you should investigate the XML file and make it contains all the entities that were saved and that all the entities have the right values. If this is true, the test was successful, otherwise you should force it to fail.
If your XML file is validated by an XML schema, you should also test for failure by creating a data structure that violates schema rules, and make sure the persistence layer reports the errors in an accurate way.
In order to test “load” you should have a valid XML file containing a realistic data structure. The test should try to load the XML file. If the load operation is successful and if the reconstructed data structure is valid – that is, you can find in it all the entities that you expect, and those entities have the right values – the test was successful. Otherwise you should force your test to fail.
You can also imagine failure tests where the input XML file is not properly formatted or contains out of range values, and make sure the persistence layer reports the errors as it should.
Business layers a more difficult to test than persistence layers. Persistence layer logic is simple and linear, which means it doesn't offer many variation possibilities. Business logic procedures, however, can introduce important algorithm variations based on values of input data. These variations – branches – are often quite difficult to identify and they augment the number of both success and failure tests required to properly test your code. In order to identify all meaningful tests that must be written for a business algorithm you need good analytical skills, and most of all, you need to know that algorithm very well. This is quite different from testing persistence layers, where a new person on the project could write the tests without much difficulty
Often, business logic layers are themselves layered. Lower layers implement basic business processing procedures while upper layers contain implement more complex procedure. A simple example from the personal finance domain could be the following: suppose you have a system that optimizes placements in accounts with various interest rates. For such a system we can imagine a three level business logic layer:
Layer 0: implements simple business operations such as debit account and credit account.
Layer 1: implements slightly more complicated procedures such as transfer funds, which call debit account and credit account
Layer 3: implements the optimize placement procedure, which by far the most complicated procedure of the system.
In such a case, you should begin by testing the procedures in layer 0. Once you're confident enough that debit and credit operations work fine, you can begin testing fund transfers. The fact that credit and debit are already tested lets you concentrate on testing the specificities of the transfer operation. The same principle can be applied as you move up the layers.
Typically, only the highest level business logic layers are exposed as an application facade. In a J2EE environment the highest level layers would be implemented as EJB with remote interfaces while lower level layers would be implemented as EJB with local interfaces. This can raise a few testing challenges, because only the remote EJB are accessible from remote locations – and as such, are testable using the remote facade testing pattern. Lower level layers can only be invoked from within the EJB container. This is where in container testing comes in and plays an important role. Problem is no testing framework today supports in container testing out of the box. You can either hack around your favorite testing framework to implement this feature or rely on an existing extension. Problem is, existing extensions don't generally have enough features, so in most cases you'll end up hacking.
Application facade testing consists in testing the top layers of your application – typically administration and business processing services exposed to clients.
Technically, the application facade testing pattern is very simple to implement, because it can be done with a regular testing framework, no extension being required. Even though it does offer some assurance regarding the quality of public services exposed by your enterprise application, it's far from being enough, especially if errors appear. If an error appears while testing the facade, you will not be able to determine where exactly the error happened. It may be in the persistence layer or it may be in one of the underlying business processing layers. Since you only test the facade, it will be difficult for you to determine where exactly the error comes from. When used alone, facade testing is black box testing.
When used in conjunction with in container testing, however, it becomes extremely powerful. This is because your underlying layers are already tested, which means that if the facade tests fail, errors either come form the facade logic itself or some more indication if offered by underlying layers tests. Also, in container testing lets you create data sets in a facade API independent way, by executing server side fixtures.
Now, remember our case study, form the beginning of this paper? What if our application has two types of clients: a heavy client and a web application client? Well in this case, in order to make sure our API works well we should have two sets of application facade tests: one that executes the tests in a client JVM, reproduce the conditions of the swing application and one that executed the test in a SERVLET container, to reproduce the conditions of the web application.
Most test frameworks are capable of testing client side code. Most popular choices today are JUnit, TestNG and JTiger.
JUnit was the first JAVA unit testing framework around and as a result of that its seen as a standard setter and it benefits from largest user base. It also integrates with every meaningful IDE or build tool.
TestNG and JTiger are the challengers. They offer everything JUnit offers and even more, but as newcomers they don't have the same adoption rate as JUnit, nor do they integrate with as many IDEs and build tools.
TestNG, however, is making very big progress and once it will be integrated with all meaningfull IDEs and build tools, it will be able to compete with JUnit from equal positions.
If you want to obtain reliable results from your tests, the tested code must be executed in the container in which it is expected to run during production. If your code is expected to support more then one container – let's say: Weblogic, JBoss and Websphere – and more then one database – say Oracle and DB2 – then you can only consider your code to be fully tested after having executed your test against every combination of container / database you claim to support, as differences exist between various standard implementations. A few examples of issues I've ran into are:
differences in container behavior: the “supports” transactional attribute on EJB methods. It works as expected on weblogic and jboss, but doesn't quite work on websphere. What happens on websphere is that if there is no ongoing transaction and you call your “supports” method, and within it you query your database, the JDBC connection hangs.
differences in database behavior: if you insert an empty string value in a varchar column in DB2 or MS-SQL and then you query the record, you get back an empty string (as expected). If you do the same thing on Oracle, you get null back.
JVM differences: application servers can use one of the three mainstream JAVA virtual machines: the SUN JVM, Weblogic's JRockit and the IBM JVM. Subtle, yet important differences exist between different implementations of the JVM which may cause your code to behave differently when executed in one or another. These differences generally appear when using the JIT – meaning in 99% of the situations. For example, when performing introspection, the IBM virtual machine returns the list of methods and attributes in reversed alphabetic order, while the sun JVM returns it in alphabetic order. If in your code you assumed one or the other, it wouldn't work on the other platform. Also, I remember that when we testing on JRockit, some open source libraries we were using just refused to work.
I hope you found this paper useful. I tried to put in it all the hints I would have liked to be aware of when I first started to write tests extensively.
all content (c) 1998 - 2005 Emil & Maria Kirschner