It has been a while that I wrote some code. Been learning Python using Udemy for some time. Python is truly expressive and batteries included language. I read about Hashmaps and thought I should give it a try. Within a very short duration (say 1 hour), I was kind of able to complete full hashmap implementation along with basic tests using PyTest. My coding screenshots. You can also check out my GitHub (that has Jupyter notebook file)
Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts
18 June 2019
05 June 2019
Visualization in Python - Case Study: Urban Accidents in India
I was reading a news item related to accidents and wanted to explore more. With a bit of search, I landed in Govt of India Data website. The website had so much data and visualization in different industries/areas. With another double down, I managed to download the data for Urban accidents in all Indian states.
With a couple of hours of staring at the data and coding, managed to pull out the following charts. The data had absolute numbers for each state and I had come up with a ranking on the following KPIs (had to download population data from Wikipedia according to 2011 census report)
- Number of urban accidents
- Percentage of accidents with a weighted urban population
Feedback loop: Data -> Information -> Insights -> Business Transformation
Tools Used:
- Python (no Numpy or anything else)
- Matplotlib for visualization
- Jupyter to cook Python code quickly
Source code and visualization can be found in my github repo
Questions:
- Why GA has more accidents than TN? (why??)
- Why HP has more accidents than PY? (terrain?)
- What can be done to minimize accidents?
Visit www.geekshub.in to know more and learn more on Python/Go/Cloud/Cloud Native/Javascript
09 January 2018
Unit Tests - Are they important?
While I was going through some of the projects in GitHub, I found that almost all the projects have automated unit tests (I hope you will also see many of your projects have that. These days it is quite common to start off by writing tests first). These tests are run most of the times - before every commit, after every commit, nightly tests and at various stages. These tests are a critical and important piece in CI/CD. Tests, in general, are the cornerstone to software development. The tests are proof that the specific part of the software is working correctly and consistently. In this post, I am going to talk about unit testing and in subsequent posts show some examples (in Python but you can find similar tools in other modern languages)
What is testing?
It is the systematic exploration of the subject and verifying all the aspects of the subject is intact. It is the philosophy that a faulty brick is responsible for bringing down a massive building. The main purpose of testing is building quality (correctness and consistency) and the side effect is "defects".
What is a test case?
A test case is a scenario which ensures that a certain aspect of the subject works as intended (or does not work which becomes a defect). When you change the condition, it is altogether a different scenario. A test case with the correct result is the proof that the system works fine in a specific scenario. A test case is narrow (in the sense that it verifies only a specific part/function), to the point and repeatable. It is concrete, it improves confidence and it builds quality.
Top-down approach
When you test a bigger piece, it is often challenging to find out the boundary of the piece. As you explore, you tend to get a feeling that the bigger piece is expanding as you explore (and often never-ending as you come to know more about the system). It quickly gets out of control. The top-down approach (knowing the bigger system first and then decomposing it further, keep working on it until you come to a unit) is time-consuming and tedious to do in all situations (someone fixing a small bug needs to communicate why she has fixed that brick), and delaying the validation of a unit until bigger piece is built is ineffective. The top-down approach should focus on the problem domain and wider aspects/functions much at a higher level.
Bottom-up Approach
This is not to say that top-down approach is wrong. Rather the top down is not sufficient or it is sufficient but fixing it when the system is developed fully is going to hurt the product in many ways (very likely). It needs to be complemented by the bottom-up approach. Write your "unit of code" and while you are writing your "unit of code", write your own tests. It is even better if someone writes unit test code for your unit of code. The top-down and bottom-up are two different perspectives of the same system (the first perspective is system has components and the second perspective is the components make the system)
What is a unit of code and unit testing?
This is a very vague term, often confusing and means different things to different people. We can consider a unit of code as "smallest possible amount of code that can be tested". You can call a function as the unit (like the ones that we are going to see tomorrow) but then the entire main() function where your application runs is too big to be called the unit of code. So, apply your mind when deciding how smaller or bigger is the unit of code.
A unit test is a scenario (setup, input, output, validation, cleanup, and reporting). Each unit test will have certain environment or conditions that are assumed to be present, input values, output returned, the validation that proves that the test is passed or failed, cleanup and finally last but not least reporting the success or failure.
When the unit test suites/cases can be run?
It is an absolute must that a developer writes unit tests before she writes production code. It is the test code that is to be written first and runs or at least while you are writing your code (writing test code after you commit your code is the biggest sin that you can do to your code). The idea is to break the software and keep fixing it. With that philosophy, it is a critical piece in software development. The unit tests have to be run successfully during the development stage (numerous stages and numerous times), before the code is committed, after the commit and nightly builds.
Why automated unit tests?
If you look at the above point, it stresses the importance of running them all the times. So, it becomes easier (and productive too) to write/automate once and run millions of times throughout the life of the product. So prefer automated unit tests over manual.
In the above analogy, unit tests are certainly examining each and every brick that they are good. The other forms are testing ensures that the joints between bricks are correct, the walls are correct, all the room comply with the requirements and overall the building is a masterpiece.
In the next post, we will start off with a simple example, learn the trades of unit testing and learn Python's built-in "unittest" module.
A unit test is a scenario (setup, input, output, validation, cleanup, and reporting). Each unit test will have certain environment or conditions that are assumed to be present, input values, output returned, the validation that proves that the test is passed or failed, cleanup and finally last but not least reporting the success or failure.
When the unit test suites/cases can be run?
It is an absolute must that a developer writes unit tests before she writes production code. It is the test code that is to be written first and runs or at least while you are writing your code (writing test code after you commit your code is the biggest sin that you can do to your code). The idea is to break the software and keep fixing it. With that philosophy, it is a critical piece in software development. The unit tests have to be run successfully during the development stage (numerous stages and numerous times), before the code is committed, after the commit and nightly builds.
Why automated unit tests?
If you look at the above point, it stresses the importance of running them all the times. So, it becomes easier (and productive too) to write/automate once and run millions of times throughout the life of the product. So prefer automated unit tests over manual.
In the above analogy, unit tests are certainly examining each and every brick that they are good. The other forms are testing ensures that the joints between bricks are correct, the walls are correct, all the room comply with the requirements and overall the building is a masterpiece.
In the next post, we will start off with a simple example, learn the trades of unit testing and learn Python's built-in "unittest" module.
07 January 2018
"Learning Python by Mark Lutz" - Reading Journal 6 [Chapter 10, Chapter 11 and Chapter 12]
![]() |
| Learning Python by Mark Lutz |
For next few months, I would be posting my experiences with Python more specifically the learning I am going through from the book - "Learning Python by Mark Lutz". You can find all of my reading journal of this book.
The rest of this post covers what I learned in Chapter 10, Chapter 11 and Chapter 12.
Chapter 10: Introducing Python Statements
This chapter is fairly easy to read. By now you should have written some code with statements which should make this chapter even more easier. But an important take away is "why Python is Pythonic" with to respect statements, code blocks, and indentation. Towards the end, we also see few nice and quick examples.
Chapter 11: Assignments, Expressions, and Prints
This is another great chapter that covers assignments, expressions, and prints in great detail. It details various ways of assignments, sequence unpacking, sequence assignments and augmented assignments. Variable naming rules, conventions, and expression statements are discussed too. Towards the end, there is a good section on prints and various ways of printing covering both 2.x, 3.x, and printing in version neutral way. Another great chapter.
Chapter 12: if Tests and Syntax Rules
This chapter gives an in-depth treatment of if constructs & its optional extension (elif, else) giving examples of nested and deeply nested if constructs, multiway branching. The most valuable content is the discussion on indentation and philosophy of Python with respect to code readability in the light of software maintenance. Truth values, boolean tests and if/else ternary operations are discussed. I suggest to stop here for a while and then try out examples so that the concepts are understood well.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
Chapter 10: Introducing Python Statements
This chapter is fairly easy to read. By now you should have written some code with statements which should make this chapter even more easier. But an important take away is "why Python is Pythonic" with to respect statements, code blocks, and indentation. Towards the end, we also see few nice and quick examples.
Chapter 11: Assignments, Expressions, and Prints
This is another great chapter that covers assignments, expressions, and prints in great detail. It details various ways of assignments, sequence unpacking, sequence assignments and augmented assignments. Variable naming rules, conventions, and expression statements are discussed too. Towards the end, there is a good section on prints and various ways of printing covering both 2.x, 3.x, and printing in version neutral way. Another great chapter.
Chapter 12: if Tests and Syntax Rules
This chapter gives an in-depth treatment of if constructs & its optional extension (elif, else) giving examples of nested and deeply nested if constructs, multiway branching. The most valuable content is the discussion on indentation and philosophy of Python with respect to code readability in the light of software maintenance. Truth values, boolean tests and if/else ternary operations are discussed. I suggest to stop here for a while and then try out examples so that the concepts are understood well.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
05 January 2018
"Learning Python by Mark Lutz" - Reading Journal 5 [Chapter 8 and Chapter 9]
![]() |
| Learning Python by Mark Lutz |
For next few months, I would be posting my experiences with Python more specifically the learning I am going through from the book - "Learning Python by Mark Lutz". You can find all of my reading journal of this book.
The rest of this post covers what I learned in Chapter 8 and Chapter 9.
Chapter 8: List and Dictionaries
This chapter has given a lot of information in lists and directories. The key thing is that each operation is shown with examples. If you have gone through the chapter on "string", the list should be similar and you can just zip it through. The operations such as create, modify, slice, sequencing, iteration, comprehension and in place modification with loads of examples. Once you are through this chapter, you will understand (and relate to) if someone says you need not memorize things in Python. It was a very good read.
Chapter 9: Tuples, Files, and Everything Else
This chapter gives a nice exposition on tuples, named tuples, files and overview/summary of types that are discussed so far. At this point in time, I have to admit that Python is becoming easier to comprehend. Since we have seen lists/dictionary already, tuples and named tuples are similar except that they are immutable. With respect to files, I learned how to read, write, seek and flush the buffers, iterate over lines, opening them in the binary mode which turns off encoding/decoding Unicode, introduction to pickle, JSON & struct and file context manager. The chapter closes with a lengthy overview of types giving us a treat.
With this, I have completed Part 2 of the book roughly 20% of the book. Still, I have 80% to cover but it is exciting :-). Overall, my impression is - part 2 is deeper and insightful than part1 (with no offense meant to part1)
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
Chapter 8: List and Dictionaries
This chapter has given a lot of information in lists and directories. The key thing is that each operation is shown with examples. If you have gone through the chapter on "string", the list should be similar and you can just zip it through. The operations such as create, modify, slice, sequencing, iteration, comprehension and in place modification with loads of examples. Once you are through this chapter, you will understand (and relate to) if someone says you need not memorize things in Python. It was a very good read.
Chapter 9: Tuples, Files, and Everything Else
This chapter gives a nice exposition on tuples, named tuples, files and overview/summary of types that are discussed so far. At this point in time, I have to admit that Python is becoming easier to comprehend. Since we have seen lists/dictionary already, tuples and named tuples are similar except that they are immutable. With respect to files, I learned how to read, write, seek and flush the buffers, iterate over lines, opening them in the binary mode which turns off encoding/decoding Unicode, introduction to pickle, JSON & struct and file context manager. The chapter closes with a lengthy overview of types giving us a treat.
With this, I have completed Part 2 of the book roughly 20% of the book. Still, I have 80% to cover but it is exciting :-). Overall, my impression is - part 2 is deeper and insightful than part1 (with no offense meant to part1)
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
04 January 2018
"Learning Python by Mark Lutz" - Reading Journal 4 [Chapter 6 and Chapter 7]
![]() |
| Learning Python by Mark Lutz |
For next few months, I would be posting my experiences with Python more specifically the learning I am going through from the book - "Learning Python by Mark Lutz". You can find all of my reading journal of this book.
The rest of this post covers what I learned in Chapter 6 and Chapter 7.
Chapter 6: The Dynamic Typing Interlude
This chapter provides an excellent coverage of dynamic types, objects, variables and a picture on how the objects are created in memory. Dynamic typing - how Python detects the type (or rather where it stores the type of the object), the significance of variable names, object reference counts and where the variable names are stored are discussed. A brief walkthrough on automatic reclaim of memory (aka Garbage Collection) and GC in the context of shared references, shared references and In-place changes and shared references and equality are discussed. If you are from the language like Java, you should be knowing most of the concepts already. Overall a nice theoretical view of PVM memory and object space. Nice and neat chapter
Chapter 7: String Fundamentals
If you like strings and string manipulation - I bet you will like this chapter. What appears as simple (when we do no know) becomes quite overwhleming when we go deeper. This chapter is written in such a way that it never overwhelms with a lot of friendly advices. Do not miss this chapter if you are a fan of strings. Let us come what I learned. String, why string is a sequence, string liternals, escape sequences, raw strings, unicode representation, how to express strings (three different ways), indexing, slicing, extended slicing, conversion, changing strings, immutablity, string methods, string module (which you should be using), formatting expressions. After reading the chapter, I feel that this is one of the chapters that I should read few times. I am sure that strings will take some time to master. Overall, the chapter is nice, informative. Kudos to the author again.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
Chapter 6: The Dynamic Typing Interlude
This chapter provides an excellent coverage of dynamic types, objects, variables and a picture on how the objects are created in memory. Dynamic typing - how Python detects the type (or rather where it stores the type of the object), the significance of variable names, object reference counts and where the variable names are stored are discussed. A brief walkthrough on automatic reclaim of memory (aka Garbage Collection) and GC in the context of shared references, shared references and In-place changes and shared references and equality are discussed. If you are from the language like Java, you should be knowing most of the concepts already. Overall a nice theoretical view of PVM memory and object space. Nice and neat chapter
Chapter 7: String Fundamentals
If you like strings and string manipulation - I bet you will like this chapter. What appears as simple (when we do no know) becomes quite overwhleming when we go deeper. This chapter is written in such a way that it never overwhelms with a lot of friendly advices. Do not miss this chapter if you are a fan of strings. Let us come what I learned. String, why string is a sequence, string liternals, escape sequences, raw strings, unicode representation, how to express strings (three different ways), indexing, slicing, extended slicing, conversion, changing strings, immutablity, string methods, string module (which you should be using), formatting expressions. After reading the chapter, I feel that this is one of the chapters that I should read few times. I am sure that strings will take some time to master. Overall, the chapter is nice, informative. Kudos to the author again.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
02 January 2018
Python Tricks -
I came across a good website on Python tricks - https://dbader.org/python-tricks. I recommend you to subscribe to Dan Bader's Improve your Python skills.
Relatively an easier post in Python. I am sure that this is going to be much more useful to you than what I write on my own :-)
Tomorrow, I am going to post on unit testing, a function to find prime number and test code using Python's unittest
Tomorrow, I am going to post on unit testing, a function to find prime number and test code using Python's unittest
01 January 2018
"Learning Python by Mark Lutz" - Reading Journal 3 [Chapter 4 and Chapter 5]
![]() |
| Learning Python by Mark Lutz |
For next few months, I would be posting my experiences with Python more specifically the learning I am going through from the book - "Learning Python by Mark Lutz". You can find all of my reading journal of this book
The rest of this post covers what I learned in Chapter 4 and Chapter 5.
Chapter 4: Introducing Python Object Types
As the chapter title says, this chapter introduces core data types that are built-in to Python - Numeric, String, Lists, Dictionaries. It also gives an overview of Files, Set, and user-defined classes. It talks about sequences, mappings, and sets. How to iterate over collections and access mappings/dictionaries. What are type-specific functions available. On files, a simple read/write with encoding/decoding is discussed at a higher level. At the end of this chapter, you will realize that everything is object in Python (and dynamic/strick typing system). It was very interesting to read. On the downside, I am still curious as to when I will write some code. I do not see coding exercises/puzzles at the end of the chapter yet.
Chapter 5: Numeric Types
What we think as just integers and floats are mind-boggling when we try to go little deeper. This chapter is no exception especially when you have two versions of Python (2.x and 3.x) behave little differently. I am pretty sure that numeric types and differences in two major versions of Python are going to be little tough to remember and one has to keep referring the material until it becomes second nature. Let us come to the learning.
This chapter covers numeric types - integers, floats, decimal, fraction, complex numbers, operators, building expressions, precedence, bitwise operators, comprehension, sets, booleans, evaluation of tests, chained tests, representing numbers in other bases (hex, octal and binary) and finally some introduction on number crunching with popular modules. I strongly suggest practicing while you read to get the feel of what the author says. The chapter was longer with so many details but never boring though. Truly great stuff.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
Chapter 4: Introducing Python Object Types
As the chapter title says, this chapter introduces core data types that are built-in to Python - Numeric, String, Lists, Dictionaries. It also gives an overview of Files, Set, and user-defined classes. It talks about sequences, mappings, and sets. How to iterate over collections and access mappings/dictionaries. What are type-specific functions available. On files, a simple read/write with encoding/decoding is discussed at a higher level. At the end of this chapter, you will realize that everything is object in Python (and dynamic/strick typing system). It was very interesting to read. On the downside, I am still curious as to when I will write some code. I do not see coding exercises/puzzles at the end of the chapter yet.
Chapter 5: Numeric Types
What we think as just integers and floats are mind-boggling when we try to go little deeper. This chapter is no exception especially when you have two versions of Python (2.x and 3.x) behave little differently. I am pretty sure that numeric types and differences in two major versions of Python are going to be little tough to remember and one has to keep referring the material until it becomes second nature. Let us come to the learning.
This chapter covers numeric types - integers, floats, decimal, fraction, complex numbers, operators, building expressions, precedence, bitwise operators, comprehension, sets, booleans, evaluation of tests, chained tests, representing numbers in other bases (hex, octal and binary) and finally some introduction on number crunching with popular modules. I strongly suggest practicing while you read to get the feel of what the author says. The chapter was longer with so many details but never boring though. Truly great stuff.
All posts related to "Learning Python by Mark Lutz"
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
31 December 2017
"Learning Python by Mark Lutz" - Reading Journal 2 [Chapter 2 and Chapter 3]
![]() |
| Learning Python by Mark Lutz |
For next few months, I would be posting my experiences with Python more specifically the learning I am going through from the book - "Learning Python by Mark Lutz". You can find all of my reading journal of this book and accompanying code in GitHub
The rest of this post covers what I learned in Chapter 2 and Chapter 3.
Chapter 2: How Python Runs Programs
This chapter introduces you a needed theoretical overview of Python runtime (interpreter) and Python bytecode. The execution of the program is explained in two viewpoints - programmer's view and Python's view. The programmers' view is what code to be written following the language constructs generating source code and Python's view is how to execute the program converting source code to actual functionality (the output of programmer view is input to Python's view). The chapter offers execution model variations (different runtime models - CPython, Jython, Stackless, PyPy) and optimization tools (Psyco JIT, Shed Skin, Cthyon). This chapter is wonderfully written and there will be some learning to everyone. I thoroughly enjoyed the content :-). Kudos to the author for covering a variety of insights.
Chapter 3: How You Run Programs
I need to start with a warning. This chapter has so much information and ways to run your code like "from the command line", "interactive", "IDLE" and "IDE". I personally suggest that we should try all the methods at least once and choose one of the methods. I find IDE convenient. Yes, I do use PyCharm community edition. On the good side, the chapter introduces modules, importing, namespaces, module reloads which is quite interesting. Good round up of exercises at the end. I do not want to bog you down with a lot of details. One little advice I have for you is to take breaks while reading this chapter and be prepared for a long haul. Without the break, I feel this chapter is little tough to complete and might appear daunting (and might be demotivating to complete). The good thing about such details are completeness
BTW, with this, we complete the first part of the book and we have eight more parts.
Looking forward to the Part 2
All posts related to "Learning Python by Mark Lutz"
Source Code@GitHub (exercises of this book)
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
Chapter 2: How Python Runs Programs
This chapter introduces you a needed theoretical overview of Python runtime (interpreter) and Python bytecode. The execution of the program is explained in two viewpoints - programmer's view and Python's view. The programmers' view is what code to be written following the language constructs generating source code and Python's view is how to execute the program converting source code to actual functionality (the output of programmer view is input to Python's view). The chapter offers execution model variations (different runtime models - CPython, Jython, Stackless, PyPy) and optimization tools (Psyco JIT, Shed Skin, Cthyon). This chapter is wonderfully written and there will be some learning to everyone. I thoroughly enjoyed the content :-). Kudos to the author for covering a variety of insights.
Chapter 3: How You Run Programs
I need to start with a warning. This chapter has so much information and ways to run your code like "from the command line", "interactive", "IDLE" and "IDE". I personally suggest that we should try all the methods at least once and choose one of the methods. I find IDE convenient. Yes, I do use PyCharm community edition. On the good side, the chapter introduces modules, importing, namespaces, module reloads which is quite interesting. Good round up of exercises at the end. I do not want to bog you down with a lot of details. One little advice I have for you is to take breaks while reading this chapter and be prepared for a long haul. Without the break, I feel this chapter is little tough to complete and might appear daunting (and might be demotivating to complete). The good thing about such details are completeness
BTW, with this, we complete the first part of the book and we have eight more parts.
Looking forward to the Part 2
All posts related to "Learning Python by Mark Lutz"
Source Code@GitHub (exercises of this book)
Friendly Note: I am posting the images and links from the publisher's website. I am not an affiliate and I do not get paid for writing this. If my journal is useful for someone to learn Python, I am more than happy :-)
30 December 2017
"Learning Python by Mark Lutz" - Reading Journal 1 [Intro and Chapter 1]
For next several months, I would be posting my experiences with Python more specifically the learning I am going to have from the book - "Learning Python by Mark Lutz". My first impression after reading the table of contents is very promising. It looks to me that it is one of the books that provide complete coverage of Python and exhaustive too. I am eager and yet at the same time little afraid because I never learned any programming language reading a book completely. Since it is massive in size containing 1500+ pages, I feel it is going to test my consistency. And that is one of the reasons why I want to write my reading journal.
The rest of the post contains what I learned from the chapter 1 (Initially I thought I would write journal of first three chapters. But I could not do it due to merits and the amount of content in this chapter)Chapter 1: A Python Q & A Session
The book starts with a bang providing enough material and motivation as to why someone should invest time learning Python and reading/working-out this book. This chapter provides an introduction to Python in the form of question and answer. These are kind of questions that were eating my mind and wanting an answer because the websites or courses talk only on the language but this book seems to address the gap as to "why". I also feel that the author has set up an important tone - question everything when you learn. You will find answers to the following questions
- Why people use Python?
- Is Python scripting language?
- Is there any downside of Python?
- Who uses Python?
- What can I do with Python
- How is Python developed and supported?
- What are its technical strengths?
- How Python stacks up to other languages?
This chapter gives deeper insights with solid reasonings and you will understand what you will get by learning Python and what is the amount of knowledge that you will get by reading this book - "Learning Python by Mark Lutz"
Needless to say, I am all excited to continue to learn Python :-)
29 December 2017
Simple ProcMon - Putting all things together
I have written few posts on process management in Python - fork, exec, wait, and exit. These are exactly same Python APIs around system APIs provided by Linux/Unix/POSIX. In this post, I am going to talk about putting things together and create a simple process monitor.
Functionality of Simple Process Monitor
- Have the list of processes that need to be monitored in the INI file. Let us call this configuration
- Load the configuration file
- Start the process
- Exec the new program
- Loop #3 and #4 until all the processes are loaded
- Monitor the process (wait). A simple call to "wait" would listen for a child to exit.
- When a child dies, handle it either by restarting the process again or do nothing (depending on how it is configured).
- Go back to #6 and continue the monitoring
It is going to tough to post all the code here. Refer my GITHub for the code. Feel free to extend or write tests
Experience: It is another personal experience on higher productivity claim of Python. I wrote the code in a couple of hours and spent around 2 hours on refactoring. Given my (lack of) experience in Python, Python seems to fulfill its promise.
25 December 2017
Process Management in Python - os._exit()
This is a continuation of series of posts on Process management in Python. Read this, this and this before continuing further. In this post, we are going to see how os._exit works and how it is related to "wait".
os._exit when invoked, the process that is calling it terminates immediately without doing any clean-ups (however system.exit() can be handled by the Python program, a much safer/clean way). The parent process receives the exit status when the parent process calls "wait" or "waitpid. os._exit calls _exit of OS
The following code demonstrates it. Beware, os._exit exits the process abruptly.
os._exit when invoked, the process that is calling it terminates immediately without doing any clean-ups (however system.exit() can be handled by the Python program, a much safer/clean way). The parent process receives the exit status when the parent process calls "wait" or "waitpid. os._exit calls _exit of OS
The following code demonstrates it. Beware, os._exit exits the process abruptly.
For more information on exit and _exit refer - https://en.wikipedia.org/wiki/Exit_(system_call)
Refer the Code in GIT Hub
Here is the code in Python (being Python, it is self-explanatory :-))
24 December 2017
Process Management in Python - os.wait()
This is a continuation of series of posts on Process management in Python. Read this and this before continuing further. In this post, we are going to see
- How the parent process can get the status of its child after the child has exited using "os.wait"
- What happens to the child processes after they exit but the parent is still alive and did not get the status (defunct - meaning that userspace is cleared but a small about of information in kernel space is retained so as to give the information when the parent requests the status which may potentially lead to inability to create new processes)
Getting the status of the child processes is simple. Just call os.wait() to get status of any child processes or waitpid to get status of a specific process. Note, this is blocking call meaning that the parent (thread) waits here until the child is exited/terminated.
For more information on wait/waitpid, refer - https://en.wikipedia.org/wiki/Wait_(system_call)
Refer the Code in GIT Hub
For more information on wait/waitpid, refer - https://en.wikipedia.org/wiki/Wait_(system_call)
Refer the Code in GIT Hub
Here is the code in Python (being Python, it is self-explanatory :-))
23 December 2017
Process Management in Python - os.exec
This is continuation of the previous post - Process Management in Python - os.fork()
In Linux, running a command from the shell is a two-step process. The shell first creates a process using fork (posted earlier) and overlays a new program using "exec". For example, when you want to run the command "ls" from the shell - the shell forks a copy of its own and immediately loads the binary of "ls" and starts the execution again.
Refer https://en.wikipedia.org/wiki/Exec_(system_call) for more details.
Beware, you cannot find a function exec() but rather it is a family of functions with names such as - execl, execle, execlp, execv. The functions are almost same and they differ only in parameters that are being passed. I have used one such functions in the following code. When you run this code, you should see a directory listing - the current process upon the successful calling of exec, loads "ls" and executes it. Once "ls" is executed, it exits (like it normally used to do)
As an experiment, you can do fork to create a child process and in the child process execution flow, you can overlay a new program.
Refer the Code in GIT Hub
Try running this script from Linux shell and see the output. Do not forget to share, like, comment if you like this post :-)
22 December 2017
Process Management in Python - os.fork()
Inspired by https://pymotw.com/3/signal/index.html, I thought of exploring os package in Python. The next few posts would be on process management and how to do it in Python. I would be showing you the code that is tiny. The goal is to provide basic code and I leave the improvision to you.
There are four important system calls - fork, exec, wait and exit that are related to process management. Interestingly Python provides equivalent APIs as a part of os package so that you can do a lot of system programming (and use it in a cool way). In this post, let us see some code that uses "os.fork" to create a new process.
os.fork() - creates a new process (often called as parent-child processes). The following code has enough comments and some explanation embedded as comments. Suggest you read the code and run it in Linux box.
Refer the Code in GIT Hub
Refer the Code in GIT Hub
If you want to know more about fork, refer - https://en.wikipedia.org/wiki/Fork_(system_call)
Share, comment and contribute if you like this post :-)
21 December 2017
Alarm and Signal in Python
Continuation to my previous post in Signals in Python. Again the credits go to Python Module of the week. The following code shows how easier it is to work with Python. This simple script registers for an alarm using SIGALRM which is configured to fire after "n" seconds. After "n" seconds, the handler function gets called.
In this code, I set alarm once as a part of main control flow and register again in the alarm handler. This program does not exit unless one kills the process or stops the execution of the program. (Python code seems to be more beautiful)
20 December 2017
Signal Handling in Python
If you are system programmer, you would have heard about signals and probably would have done lots of coding using signal. It might also be a second nature to you. But when you started to get your hands dirty with signals especially in C, it might have been quite involved. Now moving to signal handling in Python
Recently (last week), there was a post from Python Module of the week on signals and I thought I would try it. Here is the code that does some signal handling. You might want to check the full post (BTW, it is very interesting one). I feel that it easy to write code (and beautiful too) and I was able to do C-equivalent code in less than 30 minutes (which means that Python makes us productive). Here is the script that I wrote. You can find in GIT Hub and feel free to play around.
Credits - https://pymotw.com/3/signal/index.html
17 December 2017
"Python Crash Course by Eric Matthes" - Final Post and Review
If you are new to Python and want to quickly ramp-up to become no non-sense developer in Python, you might want to read this post, these posts that are related to this book and probably all of these posts to become Python Super Star.
This is going to be the final post on this book
Review Comments: I accidentally stumbled on this book when I was looking for the books on Python. As the name suggests, it met the purpose of offering the crash course in Python. I could see the chapters are shorter as that one can complete a chapter (or two) in a single sitting. and logically arranged. The code snippets (examples) and "try it yourself" offered a great insight into the programming language. If you are in a hurry to get yourself exposed to Python, grab a copy of this book and you should be making a lot of good progress in booting up in Python. I should also alarm you that this book does not cover the language and its features in great detail. If you are looking for a deeper coverage of each feature, there are other books.
This is going to be the final post on this book
Review Comments: I accidentally stumbled on this book when I was looking for the books on Python. As the name suggests, it met the purpose of offering the crash course in Python. I could see the chapters are shorter as that one can complete a chapter (or two) in a single sitting. and logically arranged. The code snippets (examples) and "try it yourself" offered a great insight into the programming language. If you are in a hurry to get yourself exposed to Python, grab a copy of this book and you should be making a lot of good progress in booting up in Python. I should also alarm you that this book does not cover the language and its features in great detail. If you are looking for a deeper coverage of each feature, there are other books.
I have solved all the exercises in GitHub. I do not intend to solve the projects now.
Ratings: I would rate this book a 4.5 Stars.
Recommendation: Apt for beginners of Python (If someone is new to programming, they might feel left behind in later chapters as the book covers quite a lot of ground in every chapter or might need to spend more time referring the materials online). If you are new to programming, maybe a MOOC course on Python would be an excellent start.
You can find all posts related to this book - here
Next Goal:
It is time to learn Python little deeper using "Learning Python" by Mark Lurtz
Want to challenge yourself?
Do you want to take part in the challenge of completing "Learning Python" in two months? Add comments below.
13 December 2017
"Python Crash Course by Eric Matthes" - Code Journal 2
If you are new to Python and want to quickly ramp-up to become no non-sense developer in Python, you might want to read this post, these posts that are related to this book and probably all of these posts to become Python Super Star.
Today's update:
Today (13-Dec-17, coded from 6am to 7.45am), I committed the coding solutions for Chapter 8. While solving the exercises, I realized that how easy and lovely is Python. When people say that Python is highly expressive, I did not relate to it. Now, I can :-). The amount of features that you can build with the amount of code is no way related. You will end up writing lesser code (less code means less time to develop means high productivity. Less code means fewer bugs which leads to high quality). If you want to experience it, try reading Chapter 8 and try solving examples. Lists, dictionaries along with various ways of using functions, passing arguments, passing parameters dynamically, passing a variable number of arguments are cool features of Python. Functions in Python is very powerful and yet flexible (I urge you to experience it by yourself)
Solutions to Chapter 8 can be found in GIT Hub Repository. View it and Share it
Today's update:
Today (13-Dec-17, coded from 6am to 7.45am), I committed the coding solutions for Chapter 8. While solving the exercises, I realized that how easy and lovely is Python. When people say that Python is highly expressive, I did not relate to it. Now, I can :-). The amount of features that you can build with the amount of code is no way related. You will end up writing lesser code (less code means less time to develop means high productivity. Less code means fewer bugs which leads to high quality). If you want to experience it, try reading Chapter 8 and try solving examples. Lists, dictionaries along with various ways of using functions, passing arguments, passing parameters dynamically, passing a variable number of arguments are cool features of Python. Functions in Python is very powerful and yet flexible (I urge you to experience it by yourself)
Do not underestimate Python and this book (because it is a crash course). Solving some of the exercises can be very involved and serves the purpose. I would give 5/5 for the exercises in this chapter. Great selection of exercises. Kudos to the author
Solutions to Chapter 8 can be found in GIT Hub Repository. View it and Share it
Want to contribute/collaborate?
If you want to contribute, you can clone the repository, create a branch and give me a pull request. If you want to know how to collaborate, email/call me. Together, we can make things better.
If you wish to join me in the journey of a lifelong student, add yourself to my blog list and I will not disappoint you :-).
12 December 2017
"Python Crash Course by Eric Matthes" - Code Journal 1
If you are new to Python and want to quickly ramp-up to become no non-sense developer in Python, you might want to read this post, these posts that are related to this book and probably all of these posts to become Python Super Star.
I am archiving all the code that I have written so far in GIT. Just in case if you want to see it live, check this out - https://github.com/grabyourfreedom/pythoncrashcourse. I have committed all the code that I have written till chapter 5. I will keep working on it - fix bugs, add comments, refactor and add solutions to more exercises. For next few days, I will work on the code until chapter 11 and start reading the other chapters from Saturday (for more posts and code). The code journal posts are likely to be short as I am planning to post only updates here and commit the code to GIT hub.
It is very exciting to ramp in a language - reading, coding, and sharing.
Want to contribute/collaborate?
If you want to contribute, you can clone the repository, create a branch and give me a pull request.
If you wish to join me in the journey of a lifelong student, add yourself to my blog list and I will not disappoint you :-).
Subscribe to:
Posts (Atom)
