Video Post - Java Code Coverage with Emma

Here you go, for the first time, I have recorded my voice for a tutorial on Java Code Coverage with Emma. Please have a look and let me know whether it is useful to you. I m planning to slowly but consistently release similar tutorials on Java, Tools, Operating Systems and Network Security. Watch the first presentation on Emma by clicking this links

1. Emma Code Coverage - Part 1
2. Emma for IDEs - Eclipse
3. Emma for IDEs - Netbeans
4. Emma - Offline Instrumentation
5. Emma - On-the-fly Instrumentation
6. Emma - Reporting
7. Emma with Ant
8. Emma with Junit
   

Emma - Java Code Coverage Made Easy (Part 3)

Before reading this post, I would suggest the readers to go through the previous posts (Part 1and Part 2) for better understanding even though this post alone can be of use but when combined with other previous will help you to understand the whole context of code coverage. In the first post, we understood the importance of code coverage and the second post talked about Emma's on-the-fly instrumentation. Before dealing with other possible ways of using Emma, in this post, I would like to give an overview about Emma. The intention of having previous posts is to help someone to get off the ground and this post will help to understand Emma. This post will be predominantly theoretical (which is very much needed to understand how tools are written for Java in general and how Emma is implemented in particular). With the information given in this post, you will be able to come to decision on how to use Emma effectively.

In the world of Java, you can write tools in whichever way you want. However, Java gives you at least two standard way of writing tools. These are defined as specification. The first method is something to do with "monitoring and managing" runtime behavior. Java Platform Debug Architecture, Java Debug Wire Protocol, Java Virtual Machine Tool Interface and Java Debug Interface primarily do "runtime monitoring and managing". The concept can be put simply put as "watching what JVM does and taking actions based on JVM behavior". The tool should listen what JVM is doing through events and write logic - what the tool intended to do. The main disadvantage of this is approach is the overhead in receiving the events and processing the events.

Another standard way is to add the logic to the class files. Let us consider code coverage. The objective of any code coverage tool is to measure the coverage line by line. In order to do this, the coverage tool injects the code coverage logic in all class files. Whenever the class is loaded and executed, along with the application logic, the code coverage logic also runs and records the coverage metrics. This is called as bytecode instrumentation. Again, there are two ways of doing this. First is to modify the application class in the hard disk (or any secondary storage) and the second method is add the bytecode on-the-fly when the classes are loaded.

Emma exploits both the methods and hence it lets users to measure the code coverage both offline instrumentation and on-the-fly instrumentation. The caveat is that during the on-the-fly and offline, the way the application is measured for coverage is not going to vary. Only the method used to inject or instrument bytecodes is going to vary (which we described already).

Emma employs on-the-fly instrumentation if you have integrated Emma with IDEs. The command line version of Emma has the capability to run both in offline and on-the-fly instrumentation mode. Most of the times, the developers/testers use on-the-fly instrumentation since the risk of changing the application classes is not there as bytecode instrumentation happens on-the-fly. But the life is not easy as it seems. When you are using certain application servers or servlet container, you cannot use on-the-fly because the application servers/servlet containers have custom classloaders. Here offline instrumentation is the only option. So, before using Emma, you may need to study little bit about your application and use the suitable methods.

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link

Emma - Java Code Coverage Made Easy (Part 2)

This is the second post on Java code coverage with Emma. In previous post, we discussed about Code Coverage and introduction to Emma. In this post, we will discuss about how to install Emma and do the basic code coverage with an example and screenshots. After you complete the reading of this post and previous post, you should be able to download Emma, install it and run Emma from command line. Towards the end, we will also discuss about how to get HTML report and interpret simple Emma report. It will be good if you can download Emma from here and use it side by side as you read this post. Please make sure that you download the binary of Emma and not the source version. The binary version of Emma should be sufficient to do this. Along with this, you need to have Java compiler and Java Runtime Environment to compile your Java classes. In order to give a picture of how Emma works and process involved, let us take the toughest approach of running Emma from command line rather than integrating with IDE. In later posts, we will discuss about IDE integration (Eclipse/Netbeans) to make developers life easier.

Downloading and Installing Emma
Emma comes in three different versions - binaries with examples, source and libraries. In order to do this example, we need to download "binaries with examples and documentation". You can directly download it form Sourceforge.net or you can access Sourceforge project via Emma home page. Irrespective of the way you choose, please make sure that you download Emma binaries with example version. After downloading emma..zip, unzip in a directory. You find three directories "docs", "examples" and "libs". Inside the directory "lib", you will find two JAR files emma.jar and emma_ant.jar. For time being, we will not be requiring emma_ant.jar as it is used to integrate with "ant" which we will cover in subsequent posts. The JAR "emma.jar" is the tool that we will be using throughout this example. Apart from the examples given in this post, you can also try to play with the examples that comes with Emma. Take "emma.jar" from the extracted directory and copy it to the directory where we have our examples. The following picture shows my setup.

Emma.jar and Examples

Java Program Example
Consider the following Java classes. Greater.java is the main class that we are going to test and measure coverage with the help of GreaterTestDriver.java. The class "GreaterTestDriver.java" mimics the testing of the class "Greater.java".

GreaterTestDriver.java

Greater.java

Greater.java has compare method and it prints out the largest of two numbers. GreaterTestDriver.java creates an object of "Greater.java" and invokes compare method passing values 10 and 5 respectively. For values 10 and 5, if block in the method compare is executed and else part will not be executed which our coverage has to pin point.

Running Emma
Running Emma is simple and you only need to specify emma.jar in the classpath and run your application using "emmarun" command. In our case, it is running the test driver "GreaterTestDriver" class. Once you run Emma and after you application exits, Emma prints the coverage results in a text file "converage.txt".

Running Emma and Getting output in text file

If you look at the report "coverage.txt", it gives the coverage data at package, block, method, class and line level.

You can also run Emma and get report in HTML format. Again it is very simple and you need to specify that in command line options (refer the last boxitem to know more about command line options used in this post).

Emma with HTML Reporting

So far you have run Emma and got the HTML report. But the data given by Emma shows the coverage results but the information is not sufficient. It will be helpful to merge the results with source code. Emma can also pin point which lines are executed and which lines are not executed. Yet again, the step is simple and following example is self explanatory.

Emma Reporting with Source Code

In the above report, the lines that are highlighted in green color are the ones that are tested at least once and the lines that are in pink color are not tested. In ideal case, you need to improve your test suite to cover the lines that are highlighted in pink color. As an exercise, can you achieve 100% coverage in Greater.java?

Command options used in this example

# In order to run Emma from command line and to get report in test file

java -cp emma.jar emmarun -cp . example.GreaterTestDriver

# In order to run Emma from command line and get report in HTML but without Source Code

java -cp emma.jar emmarun -r html -cp . example.GreaterTestDriver

# In order to run Emma from command line, get report in HTML and merge the result with Source Code

java -cp emma.jar emmarun -sp . -r html -cp . example.GreaterTestDrive

In this post we saw only on-the-fly instrumentation and in upcoming posts we will discuss other features of Emma such as Eclipse (and Netbeans) integration, off-line instrumentation, integrating with ant and merging data of multiple sessions, integrating with Tomcat.

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link

Emma - Java Code Coverage Made Easy (Part 1)

Irrespective of the number of lines of code you write and if you are a good (and honest) tester or developer, you accept the fact that it is impossible to test the entire software. I am not talking about those simple programs like printing "Hello, World". I am talking about software that is decent in size. Testing software is brain intensive. I would like to give you a simple scenario how I used to think when I was in testing team (of course, that was 6 years back when I was fresher). When I was executing a test case or writing a test case, I generally used to get few parallel thoughts about what are the other related test cases. It never ceases there. It went on and on until I felt comfortable. If I did not cover some corner cases, something would pinch me from inside to do more, to spawn many parallel thoughts and eventually I would end up writing more test cases. While this approach is good, but this is very subjective and depends on the person who is testing. It depends on the knowledge level and expertise of the person who is doing the testing. The testing should be much more and also driven by data. Even if we take that most of the testers of are effective, how are we going to convince and be sure that the test cases are effective?

As a tester, we generally used to get requirements, design documents (?) and finally code in binary format. If you apply general system thinking, tester is above developer simply because he supervises developers' work and validates it. The confidence level of the tester has to impeccable so that the project managers, product managers can have sound sleep and sound sleep every night. This confidence comes with thoroughness of testing and this ability is only possible if you eat, sleep, think with testing in your head and practice testing as art. You might be watching YouTube, but yet you will think like a tester. You might be doing test strategy, test planning, test plan, sophisticated test case design. All these comes in the following ways - requirements, design document, your understanding about how the code will look like and ad hoc testing. But is that sufficient. We are living in the world of doing more and wanting more, testing effectiveness is no exception. Is there a mechanism by which I can measure my test suite effectiveness. Yes, it is there. The rest of the post deals with doing code coverage for Java applications using Emma, an open source code coverage tool.

First, what is code coverage? The process of measuring or seeing which part of code that are tested and the areas that are not at all tested is called code coverage. Second, why is it important? The answer is read the definition again. Period. If you are unit testing you code, it helps you to see the coverage immediately and hence you can fix it "then and there" and butterflies will not fly in your stomach when the product goes to testing team. If you are a tester, it helps you to measure - first your test plan and then the code. Naturally, you will write more test cases or change the existing test cases to cover more. The point is "more coverage". So, no more butterflies in your stomach too when your product goes out to production.

Emma is an open source software which is simple, powerful and flexible. Emma uses bytecode instrumentation technique and measures the code coverage. It has the ability to instrument the classes on-the-fly and also supports offline instrumentation. In on-the-fly instrumentation, you just give the location of your Jar file or class files, Java Virtual Machine when it loads the classes it instruments the logic for code coverage. In offline mode, you need to do an additional step of adding the logic to Jar/class files and copy it in your hard disk and load the instrumented classes to Java Virtual Machine. But irrespective the method you choose, the code coverage will be same. The offline mode is particularly helpful when you have custom classloaders like application servers or servlet container.

You can run Emma directly from command line and you can also integrate with nightly build as a part of continous improvement strategy. Ideally, in nightly build, code coverage works hand in hand with Unit Testing Framework like JUnit. After instrumentation, you have carry out your testing. When you are testing, Emma does the code coverage. Once you are done with the testing, you can safely stop the application. Emma also comes with reporting and it can produce reports in HTML and XML. The report can also be merged along with source code and by just looking at this report, you can identify the lines that not covered.

Apart from running Emma from command line and including with ant, it can also be used at developer's desktop. Emma has plugins for famous IDEs - Eclipse and Netbeans. With this, the developer's can run their unit test cases with Emma immediately after implementing a unit of code. Based on the coverage, they can work on their unit test cases to achieve better coverage. Better coverage, better quality and improved confidence. With this, we come to an end on introducing Emma - Open Source Java Code Coverage Solution.

In the next post, we will be discussing how to install Emma and use it.

Have a Great Day

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link