One question I have been pondering over for a while. We often focus on high code coverage and use it as guide for measuring the quality of our code, but is it appropriate or the right thing?
Recently I saw infoq video Beauty Is in the Eye of the Beholder, so here are my thoughts.
Its bad because its percentage of your lines of code covered by tests. But how can that be bad. Like any metric developers/development managers/team leader often find it easier it as a measure of code quality. Now there is nothing wrong with that if that meant that the quality of tests you were writing were excellent. Now let say if you as team agree that the tests are doing their job why do you measure some thing you already know, right. So often its used by teams where the quality does not meet their expectation. If the code is not meeting your expectation how will a measure which is absolute help you improve it. In reality it won’t. What will happen often team will focus on getting a very high coverage figure. Once you have achieved the high figure it will feel like a big stepping stone achieved. In some case this may be right, but it does not guarantee that you have
- You have not over engineered your code to achieve the high figure you were aiming for.
- What about coupling and cohesion? Still may be awful in your code but your test stats will be green.
- There is good chance you may written tests for the sake of high coverage figure.
- When it come to changing your code you are likely have with brittle tests which need multiple changes every time you change your code.
Now it does not feel very good when on hand you have high coverage, but brittle tests.
The real issue here is that fact that even if you have excellent code coverage its does not guarantee that you would have written maintainable code.
In a mature team, where there is common understanding of quality & maintainability, the most important measure is the gut feel, every time you have make a small change to the code the number tests you have to change, how brittle are the tests and more importantly how confident that the the change will work without and side-effect.
I am not proposing that don’t measure your code coverage. What I am proposing is that only use the measure if it really helps your team develop quality code and release value early. One final thought, your best feedback is the rate of change and time to test & release a small change though to production and getting early feedback from your real testers i.e. the users of the system.
More to come on this …