Irrespective of the number of lines of code you write and if you are a good (and honest) tester or developer, you accept the fact that it is impossible to test the entire software. I am not talking about those simple programs like printing "Hello, World". I am talking about software that is decent in size. Testing software is brain intensive. I would like to give you a simple scenario how I used to think when I was in testing team (of course, that was 6 years back when I was fresher). When I was executing a test case or writing a test case, I generally used to get few parallel thoughts about what are the other related test cases. It never ceases there. It went on and on until I felt comfortable. If I did not cover some corner cases, something would pinch me from inside to do more, to spawn many parallel thoughts and eventually I would end up writing more test cases. While this approach is good, but this is very subjective and depends on the person who is testing. It depends on the knowledge level and expertise of the person who is doing the testing. The testing should be much more and also driven by data. Even if we take that most of the testers of are effective, how are we going to convince and be sure that the test cases are effective?
As a tester, we generally used to get requirements, design documents (?) and finally code in binary format. If you apply general system thinking, tester is above developer simply because he supervises developers' work and validates it. The confidence level of the tester has to impeccable so that the project managers, product managers can have sound sleep and sound sleep every night. This confidence comes with thoroughness of testing and this ability is only possible if you eat, sleep, think with testing in your head and practice testing as art. You might be watching YouTube, but yet you will think like a tester. You might be doing test strategy, test planning, test plan, sophisticated test case design. All these comes in the following ways - requirements, design document, your understanding about how the code will look like and ad hoc testing. But is that sufficient. We are living in the world of doing more and wanting more, testing effectiveness is no exception. Is there a mechanism by which I can measure my test suite effectiveness. Yes, it is there. The rest of the post deals with doing code coverage for Java applications using Emma, an open source code coverage tool.
First, what is code coverage? The process of measuring or seeing which part of code that are tested and the areas that are not at all tested is called code coverage. Second, why is it important? The answer is read the definition again. Period. If you are unit testing you code, it helps you to see the coverage immediately and hence you can fix it "then and there" and butterflies will not fly in your stomach when the product goes to testing team. If you are a tester, it helps you to measure - first your test plan and then the code. Naturally, you will write more test cases or change the existing test cases to cover more. The point is "more coverage". So, no more butterflies in your stomach too when your product goes out to production.
Emma is an open source software which is simple, powerful and flexible. Emma uses bytecode instrumentation technique and measures the code coverage. It has the ability to instrument the classes on-the-fly and also supports offline instrumentation. In on-the-fly instrumentation, you just give the location of your Jar file or class files, Java Virtual Machine when it loads the classes it instruments the logic for code coverage. In offline mode, you need to do an additional step of adding the logic to Jar/class files and copy it in your hard disk and load the instrumented classes to Java Virtual Machine. But irrespective the method you choose, the code coverage will be same. The offline mode is particularly helpful when you have custom classloaders like application servers or servlet container.
You can run Emma directly from command line and you can also integrate with nightly build as a part of continous improvement strategy. Ideally, in nightly build, code coverage works hand in hand with Unit Testing Framework like JUnit. After instrumentation, you have carry out your testing. When you are testing, Emma does the code coverage. Once you are done with the testing, you can safely stop the application. Emma also comes with reporting and it can produce reports in HTML and XML. The report can also be merged along with source code and by just looking at this report, you can identify the lines that not covered.
Apart from running Emma from command line and including with ant, it can also be used at developer's desktop. Emma has plugins for famous IDEs - Eclipse and Netbeans. With this, the developer's can run their unit test cases with Emma immediately after implementing a unit of code. Based on the coverage, they can work on their unit test cases to achieve better coverage. Better coverage, better quality and improved confidence. With this, we come to an end on introducing Emma - Open Source Java Code Coverage Solution.
In the next post, we will be discussing how to install Emma and use it.
Have a Great Day
I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link
As a tester, we generally used to get requirements, design documents (?) and finally code in binary format. If you apply general system thinking, tester is above developer simply because he supervises developers' work and validates it. The confidence level of the tester has to impeccable so that the project managers, product managers can have sound sleep and sound sleep every night. This confidence comes with thoroughness of testing and this ability is only possible if you eat, sleep, think with testing in your head and practice testing as art. You might be watching YouTube, but yet you will think like a tester. You might be doing test strategy, test planning, test plan, sophisticated test case design. All these comes in the following ways - requirements, design document, your understanding about how the code will look like and ad hoc testing. But is that sufficient. We are living in the world of doing more and wanting more, testing effectiveness is no exception. Is there a mechanism by which I can measure my test suite effectiveness. Yes, it is there. The rest of the post deals with doing code coverage for Java applications using Emma, an open source code coverage tool.
First, what is code coverage? The process of measuring or seeing which part of code that are tested and the areas that are not at all tested is called code coverage. Second, why is it important? The answer is read the definition again. Period. If you are unit testing you code, it helps you to see the coverage immediately and hence you can fix it "then and there" and butterflies will not fly in your stomach when the product goes to testing team. If you are a tester, it helps you to measure - first your test plan and then the code. Naturally, you will write more test cases or change the existing test cases to cover more. The point is "more coverage". So, no more butterflies in your stomach too when your product goes out to production.
Emma is an open source software which is simple, powerful and flexible. Emma uses bytecode instrumentation technique and measures the code coverage. It has the ability to instrument the classes on-the-fly and also supports offline instrumentation. In on-the-fly instrumentation, you just give the location of your Jar file or class files, Java Virtual Machine when it loads the classes it instruments the logic for code coverage. In offline mode, you need to do an additional step of adding the logic to Jar/class files and copy it in your hard disk and load the instrumented classes to Java Virtual Machine. But irrespective the method you choose, the code coverage will be same. The offline mode is particularly helpful when you have custom classloaders like application servers or servlet container.
You can run Emma directly from command line and you can also integrate with nightly build as a part of continous improvement strategy. Ideally, in nightly build, code coverage works hand in hand with Unit Testing Framework like JUnit. After instrumentation, you have carry out your testing. When you are testing, Emma does the code coverage. Once you are done with the testing, you can safely stop the application. Emma also comes with reporting and it can produce reports in HTML and XML. The report can also be merged along with source code and by just looking at this report, you can identify the lines that not covered.
Apart from running Emma from command line and including with ant, it can also be used at developer's desktop. Emma has plugins for famous IDEs - Eclipse and Netbeans. With this, the developer's can run their unit test cases with Emma immediately after implementing a unit of code. Based on the coverage, they can work on their unit test cases to achieve better coverage. Better coverage, better quality and improved confidence. With this, we come to an end on introducing Emma - Open Source Java Code Coverage Solution.
In the next post, we will be discussing how to install Emma and use it.
Have a Great Day
I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link
