Knowing Linux Processes

Sometime in 2005, I happened to write two articles on Linux Processes which was published Linux for You. For someone who is relatively new to Linux/Unix programming, the following two articles should be a curtain raiser as it explains basics of processes in Linux. While the topic is Linux processes, the concept should apply to any multi-tasking operating systems. Sharing it here hoping that it will be useful for some of you.

Access the following links for the PDFs

An Introduction to Linux Processes

Processes in Linux (Part -2)

Happy Reading.

Emma - Java Code Coverage Made Easy (Part 3)

Before reading this post, I would suggest the readers to go through the previous posts (Part 1and Part 2) for better understanding even though this post alone can be of use but when combined with other previous will help you to understand the whole context of code coverage. In the first post, we understood the importance of code coverage and the second post talked about Emma's on-the-fly instrumentation. Before dealing with other possible ways of using Emma, in this post, I would like to give an overview about Emma. The intention of having previous posts is to help someone to get off the ground and this post will help to understand Emma. This post will be predominantly theoretical (which is very much needed to understand how tools are written for Java in general and how Emma is implemented in particular). With the information given in this post, you will be able to come to decision on how to use Emma effectively.

In the world of Java, you can write tools in whichever way you want. However, Java gives you at least two standard way of writing tools. These are defined as specification. The first method is something to do with "monitoring and managing" runtime behavior. Java Platform Debug Architecture, Java Debug Wire Protocol, Java Virtual Machine Tool Interface and Java Debug Interface primarily do "runtime monitoring and managing". The concept can be put simply put as "watching what JVM does and taking actions based on JVM behavior". The tool should listen what JVM is doing through events and write logic - what the tool intended to do. The main disadvantage of this is approach is the overhead in receiving the events and processing the events.

Another standard way is to add the logic to the class files. Let us consider code coverage. The objective of any code coverage tool is to measure the coverage line by line. In order to do this, the coverage tool injects the code coverage logic in all class files. Whenever the class is loaded and executed, along with the application logic, the code coverage logic also runs and records the coverage metrics. This is called as bytecode instrumentation. Again, there are two ways of doing this. First is to modify the application class in the hard disk (or any secondary storage) and the second method is add the bytecode on-the-fly when the classes are loaded.

Emma exploits both the methods and hence it lets users to measure the code coverage both offline instrumentation and on-the-fly instrumentation. The caveat is that during the on-the-fly and offline, the way the application is measured for coverage is not going to vary. Only the method used to inject or instrument bytecodes is going to vary (which we described already).

Emma employs on-the-fly instrumentation if you have integrated Emma with IDEs. The command line version of Emma has the capability to run both in offline and on-the-fly instrumentation mode. Most of the times, the developers/testers use on-the-fly instrumentation since the risk of changing the application classes is not there as bytecode instrumentation happens on-the-fly. But the life is not easy as it seems. When you are using certain application servers or servlet container, you cannot use on-the-fly because the application servers/servlet containers have custom classloaders. Here offline instrumentation is the only option. So, before using Emma, you may need to study little bit about your application and use the suitable methods.

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link

Emma - Java Code Coverage Made Easy (Part 2)

This is the second post on Java code coverage with Emma. In previous post, we discussed about Code Coverage and introduction to Emma. In this post, we will discuss about how to install Emma and do the basic code coverage with an example and screenshots. After you complete the reading of this post and previous post, you should be able to download Emma, install it and run Emma from command line. Towards the end, we will also discuss about how to get HTML report and interpret simple Emma report. It will be good if you can download Emma from here and use it side by side as you read this post. Please make sure that you download the binary of Emma and not the source version. The binary version of Emma should be sufficient to do this. Along with this, you need to have Java compiler and Java Runtime Environment to compile your Java classes. In order to give a picture of how Emma works and process involved, let us take the toughest approach of running Emma from command line rather than integrating with IDE. In later posts, we will discuss about IDE integration (Eclipse/Netbeans) to make developers life easier.

Downloading and Installing Emma
Emma comes in three different versions - binaries with examples, source and libraries. In order to do this example, we need to download "binaries with examples and documentation". You can directly download it form Sourceforge.net or you can access Sourceforge project via Emma home page. Irrespective of the way you choose, please make sure that you download Emma binaries with example version. After downloading emma..zip, unzip in a directory. You find three directories "docs", "examples" and "libs". Inside the directory "lib", you will find two JAR files emma.jar and emma_ant.jar. For time being, we will not be requiring emma_ant.jar as it is used to integrate with "ant" which we will cover in subsequent posts. The JAR "emma.jar" is the tool that we will be using throughout this example. Apart from the examples given in this post, you can also try to play with the examples that comes with Emma. Take "emma.jar" from the extracted directory and copy it to the directory where we have our examples. The following picture shows my setup.

Emma.jar and Examples

Java Program Example
Consider the following Java classes. Greater.java is the main class that we are going to test and measure coverage with the help of GreaterTestDriver.java. The class "GreaterTestDriver.java" mimics the testing of the class "Greater.java".

GreaterTestDriver.java

Greater.java

Greater.java has compare method and it prints out the largest of two numbers. GreaterTestDriver.java creates an object of "Greater.java" and invokes compare method passing values 10 and 5 respectively. For values 10 and 5, if block in the method compare is executed and else part will not be executed which our coverage has to pin point.

Running Emma
Running Emma is simple and you only need to specify emma.jar in the classpath and run your application using "emmarun" command. In our case, it is running the test driver "GreaterTestDriver" class. Once you run Emma and after you application exits, Emma prints the coverage results in a text file "converage.txt".

Running Emma and Getting output in text file

If you look at the report "coverage.txt", it gives the coverage data at package, block, method, class and line level.

You can also run Emma and get report in HTML format. Again it is very simple and you need to specify that in command line options (refer the last boxitem to know more about command line options used in this post).

Emma with HTML Reporting

So far you have run Emma and got the HTML report. But the data given by Emma shows the coverage results but the information is not sufficient. It will be helpful to merge the results with source code. Emma can also pin point which lines are executed and which lines are not executed. Yet again, the step is simple and following example is self explanatory.

Emma Reporting with Source Code

In the above report, the lines that are highlighted in green color are the ones that are tested at least once and the lines that are in pink color are not tested. In ideal case, you need to improve your test suite to cover the lines that are highlighted in pink color. As an exercise, can you achieve 100% coverage in Greater.java?

Command options used in this example

# In order to run Emma from command line and to get report in test file

java -cp emma.jar emmarun -cp . example.GreaterTestDriver

# In order to run Emma from command line and get report in HTML but without Source Code

java -cp emma.jar emmarun -r html -cp . example.GreaterTestDriver

# In order to run Emma from command line, get report in HTML and merge the result with Source Code

java -cp emma.jar emmarun -sp . -r html -cp . example.GreaterTestDrive

In this post we saw only on-the-fly instrumentation and in upcoming posts we will discuss other features of Emma such as Eclipse (and Netbeans) integration, off-line instrumentation, integrating with ant and merging data of multiple sessions, integrating with Tomcat.

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link

Emma - Java Code Coverage Made Easy (Part 1)

Irrespective of the number of lines of code you write and if you are a good (and honest) tester or developer, you accept the fact that it is impossible to test the entire software. I am not talking about those simple programs like printing "Hello, World". I am talking about software that is decent in size. Testing software is brain intensive. I would like to give you a simple scenario how I used to think when I was in testing team (of course, that was 6 years back when I was fresher). When I was executing a test case or writing a test case, I generally used to get few parallel thoughts about what are the other related test cases. It never ceases there. It went on and on until I felt comfortable. If I did not cover some corner cases, something would pinch me from inside to do more, to spawn many parallel thoughts and eventually I would end up writing more test cases. While this approach is good, but this is very subjective and depends on the person who is testing. It depends on the knowledge level and expertise of the person who is doing the testing. The testing should be much more and also driven by data. Even if we take that most of the testers of are effective, how are we going to convince and be sure that the test cases are effective?

As a tester, we generally used to get requirements, design documents (?) and finally code in binary format. If you apply general system thinking, tester is above developer simply because he supervises developers' work and validates it. The confidence level of the tester has to impeccable so that the project managers, product managers can have sound sleep and sound sleep every night. This confidence comes with thoroughness of testing and this ability is only possible if you eat, sleep, think with testing in your head and practice testing as art. You might be watching YouTube, but yet you will think like a tester. You might be doing test strategy, test planning, test plan, sophisticated test case design. All these comes in the following ways - requirements, design document, your understanding about how the code will look like and ad hoc testing. But is that sufficient. We are living in the world of doing more and wanting more, testing effectiveness is no exception. Is there a mechanism by which I can measure my test suite effectiveness. Yes, it is there. The rest of the post deals with doing code coverage for Java applications using Emma, an open source code coverage tool.

First, what is code coverage? The process of measuring or seeing which part of code that are tested and the areas that are not at all tested is called code coverage. Second, why is it important? The answer is read the definition again. Period. If you are unit testing you code, it helps you to see the coverage immediately and hence you can fix it "then and there" and butterflies will not fly in your stomach when the product goes to testing team. If you are a tester, it helps you to measure - first your test plan and then the code. Naturally, you will write more test cases or change the existing test cases to cover more. The point is "more coverage". So, no more butterflies in your stomach too when your product goes out to production.

Emma is an open source software which is simple, powerful and flexible. Emma uses bytecode instrumentation technique and measures the code coverage. It has the ability to instrument the classes on-the-fly and also supports offline instrumentation. In on-the-fly instrumentation, you just give the location of your Jar file or class files, Java Virtual Machine when it loads the classes it instruments the logic for code coverage. In offline mode, you need to do an additional step of adding the logic to Jar/class files and copy it in your hard disk and load the instrumented classes to Java Virtual Machine. But irrespective the method you choose, the code coverage will be same. The offline mode is particularly helpful when you have custom classloaders like application servers or servlet container.

You can run Emma directly from command line and you can also integrate with nightly build as a part of continous improvement strategy. Ideally, in nightly build, code coverage works hand in hand with Unit Testing Framework like JUnit. After instrumentation, you have carry out your testing. When you are testing, Emma does the code coverage. Once you are done with the testing, you can safely stop the application. Emma also comes with reporting and it can produce reports in HTML and XML. The report can also be merged along with source code and by just looking at this report, you can identify the lines that not covered.

Apart from running Emma from command line and including with ant, it can also be used at developer's desktop. Emma has plugins for famous IDEs - Eclipse and Netbeans. With this, the developer's can run their unit test cases with Emma immediately after implementing a unit of code. Based on the coverage, they can work on their unit test cases to achieve better coverage. Better coverage, better quality and improved confidence. With this, we come to an end on introducing Emma - Open Source Java Code Coverage Solution.

In the next post, we will be discussing how to install Emma and use it.

Have a Great Day

I have also come up with a little interactive presentation with flash using Wink on Emma. It can be accessed through this link

Speaking Experience @ ILUG Chennai Monthly Meet

This is the first time that I am speaking in Indian Linux User Group, Chennai. ILUG is most active LUGs in India. The meeting generally happens in IIT Madras (the place I wanted to study). Again, I want to reiterate the fact that speaking about Open Source in Open Source meeting gives me enough motivation and abundance of energy. I registered the topics with Mr. Bharathi Subramainan and he gave be slot immediately. Today, I also noticed what happens when people with passion sign up to do something. Unlike other meetings, the members were very punctual and I could see considerable number of people waiting in the hall and some people working in computers in another room. Today's incident enlightened me about valuing others time and I will never forget this in my life. Apart from that we also had good knowledge sharing session.

The talks started at 3.15pm and the first talk was hosted by Selvakumar K from NRC-FOSS on KTechLab, an Open Source IDE for Electronics and PIC Micro Controller Circuit Design and Simulation. He talked about analog and digital simulation tools in Linux. I feel that the tool will help any electronics student to do experiment in his/her laptop/computer. The colleges should seriously consider to have this tool in their department and encourage students to use when they are doing practicals or learning theory.

Next, I started my talk on Java Code coverage and explained about need for code coverage and features of Emma. I talked about various ways of using Emma such as with IDE, nightly build, from command line, online instrumentation and off-line instrumentation. Towards the end, I gave a quick demo on using Emma with Eclipse, on-the-fly instrumentation from command and HTML reporting.

Towards the end, Prof Dr C N Krishnan talked about NRC-FOSS and its Phase 2 initiatives. He requested the help from ILUG-Chennai. He also expressed his concerns on giving feedback to NRC-FOSS.

We have also been given some stickers, pens and Fedora CDs sponsored by Ubuntu and the speakers have been given caps (so, I got one) from Ubuntu. This Saturday was fulfilling and exciting as well. Looking forward for more such knowledge sharing session.

Couple of Workshops Next Month

The next month (Apr 2009), I have signed up for couple of workshops on open source/Linux. The first workshop is to the students of Vellore Institute of Technology and I will be speaking to them on Linux Process Management, POSIX Threads and Linux Scheduler. It has been a while since I read on these areas. I have started to refresh and testing my memory power. I m quite happy with my memory power. I am able to come out with presentation slides without refering to famous Robert Love and Cesati books. Of course, I will be refering these books after completing my presentation to validate/verify the content. I would say, even preparing the presentation slides for Linux presentation itself is fulfilling. This event will be on 4-Apr-2009. Some well known dignitaries of Indian OSS community are speaking as well.

The next workshop is still conceptual stage and I am discussing with ILUC, Chennai co-ordinator Mr Bharathi Subramanian. I proposed a talk on SNORT and he came up with a brilliant idea on conduting a full day workshop on SNORT. I am going to team up with my friend and colleague S. Badhrinath. Badhri and myself have already started to have set up ready for a basic workshop on SNORT. We just downloaded SNORT source code and some attack tools. We are yet to start the preparation of presentation slides. Still, we have time for this event. This is likely to happen on 25-Apr.

Since there is a holiday season coming up, I am planning to conduct few full day workshops for students in Software Engineering, Linux, Networking, Network Security in Chennai. Let me see how far I m doing this.

Another fulfilling moment in my life

During last September, I happen to visit Paavai Engineering College at Salem, Tamilnadu for a guest lecture on Linux. It is quite a while after that and couple of weeks before, I got an invitation through my colleague Vinoth to speak students of IT department at Sona College of Technology, Selam, Tamilnadu on Open Source and how they can learn with the help of Open Source. My friend and colleague, Sundar also been invited to speak about "How to handle change", the change from college to company.

We (Sundar, Vinoth and myself) started from Chennai on Friday night and reached Salem Saturday early morning. Vinoth's family (Parents and his brother) gave a very warm welcome. After relaxing a bit after the journey, we reached the college around 8.30 am and met HoD of IT department. We discussed and understood his expectation from us and he insisted to have an informal and interactive session. It didn't surprise us as we prepared our lectures that way and I was more than happy because I do not know to speak formally.

At 9.30am, we started the proceedings with a warm welcome by the students and staff of IT department. I started to speak about Open Source. I planned my session for about 2 hours but wrapped by session within 30-45 minutes as I sensed that the students needed much awareness on taking some areas interest and specialize in a particular area rather than focus too much on Open Source. I feel that Open Source is of much use to the students only when they specialize in a particular area. For example, one needs to know the basics of Networking and Network Security before downloading or using SNORT intrusion detection system. While anyone can download it and do ".configure; make; make install", a lot of theoretical and systematic learning is required to make their learning meaningful. With this in mind, I thought that I would give the floor to Sundar who needed to speak on "change and how to respond to it". It would give me sometime to think about what we should do. Apart from that, I thought Sundar's talk would give students enough courage to speak up and participate. It happened exactly the same way.

Around 10.15am, Sundar talked about change and it was well received and I believe he spoke for more than an hour. Then we left for a break for 15 minutes. The students were impressive on keeping up the punctuality and they were back to their seats exactly after 15 minutes. I was inspired by it. Next, we asked the students to pick an area (Networking, Network Security, Operating System, Database, Embedded) and form groups. Asked them to write down what they already knew and what they would like to know and how they would achieve it.

The view about gaining technical knowledge is not through lectures and trainings. The lectures and trainings will give only awareness. During my session, i thought I should give that awareness and motivate the students to work consistently and specialize in their area of interests. I insisted upon having good theoretical knowledge and expand the gained theoretical knowledge with the help of Open Source Software. A long term goal would be contributing to some Open Source Products and at the same time achieving the purpose of joining an Engineering/Technology course.

We had this session for about 90 minutes and then 15 minutes Q/A session. Few students were around there for another 15-20 minutes to get their questions clarified. We could see some sparks in student's mind. Overall, I had another fulfilling day at Sona College of Technology. It was also quite a learning experience for me and yet again the students showed me so many good qualities that are needed for human beings such as mutual respect and trust. Also, I would like to emphasize and inspired by the humbleness, modesty and professionalism of HoD of Department of IT and hospitality of Mr. Iyyanar, NSS Co-ordinator of the college.

Open Source Solution Series

When I was a schoolboy, we used to buy fruits from a lady almost for 10 years. Over a period of time, she won the confidence of many families and become loyal to each and every family. She used to sell fruits, some times vegetables, chillies, groundnuts and many other things. Most of the families in the campus bought things from her not only because she was loyal but also because of open business models. Whenever, she brought things to sell, she would give the specimen and only then would disclose the price. She had a simple and successful business model of making things open. She did not give something as free to sell her goods, she also thought that it was buyer's right to buy right and good things. There is no question on why this business model as successful. You can find similar business people in your life too.

This makes me to think and correlate with Open Source and the success of Open Source business models. When you are selling a software, how can you ensure that the software is good. And secondly, how will you convince your customer that you are infact giving them something good not crap. At a very surface level, you can do this by exposing your bug backlog, the bug trend, the reports from your dev/test team. However, this does not give enough information to your customers. This question is not about the confidence you have on your product. It is about the confidence of the customer you win. When make we the software open, we can see it in at least two perspectives - any can view/modify the source code and secondly we are open for comments.

By being open for receiving feedbacks gives us opportunity to correct something which is not good. The customers will be getting better things. The success of open source is not "free of cost" but "open for improvement". This should give confidence to the users that there is a community of developers who are working on making things better.

In this blog, I am planning to share my experience on Open Source Software, particularly quality enterprise class software which are much better than any proprietary software. I am also hopeful that in down the line, we will be discussing solutions from Open Source solving particular problem rather than software. This will give you a good picture about Open Source and its quality and innovation. This journey will also give me a learning by trying some open source solution which I never tried.

Hoping for the best