The pitfall of code analysis tooling

Introduction

I have worked at different companies and have seen many of them struggle to attain a sufficient quality level for their automated systems. A large part of this quality can be attributed to quality aspects from software that was created in-house. Some companies cope better with these aspects than others and many employ code analysis tools to measure code quality. Measuring code quality is by no means a bad thing to do as part of a more elaborate quality assurance and control system employed during software development. The problem lies more in the fact that in certain circumstances people tend to answer more difficult questions by substituting them with easier ones (Khaneman, 2011). An observant reader may have noticed that I have mentioned three levels of quality in this introduction and that I have used code, software and system quality seemingly interchangeably. Obviously, these levels are not the same and even encompass each other. System quality encompasses software quality which encompasses code quality. However, answering the question about system quality is much more difficult than answering the question about software quality which in turn is more difficult than answering the question about code quality.

Difficulties with code analysis tooling

Struggling with desired system quality, and any actions that should be taken to attain this quality level, is difficult and leaves many companies wandering about how to proceed. This goes for software quality as well. In the article ‘A Research Study on Software Quality Attributes’ (Dr.A.Chandrasekar, 2014) there is no mention of code quality aspects. In software quality, architecture and design and especially the suitability and application of these to the underlying business problem is important but difficult to measure and time consuming. Besides the effort involved, any assessment of these aspects is confounded by changing business conditions and other more subtle factors. Because of these difficulties, companies sometimes tend to substitute the answer of system quality by getting answers from the results of automated code analysis tools. At this level it is easier to achieve some insights in the code written because commercial tooling is readily available. However, the level at which quality is now measured is only a small part of the quality of software (Wikipedia, 2020) let alone the entire system (Figure 1). When applying code analysis tooling, other dynamics may emerge in the organization that may or may not be favorable depending on the way that the tool is employed.

Code analysis tools often come with a default rule set. This default rule set should be tuned for the organization in order to consider the context in which the software is developed. But this is often neglected for different reasons like time constraints and lacking knowledge about the ruleset or its rules and their importance. So, what remains is a default ruleset that has made assumptions about certain aspects of the software that is written that may not be appropriate within the set context. First, a ruleset determines what is measured and as such determines what is important. Second the tool has a built-in judgement system about what value is acceptable. The latter is where things start to get troublesome because the judgement then lacks context in the broadest sense of the word as elaborately indicated by the anonymous and unfortunately unfinished blog on the CQSE site (Continuous Quality in Software Engineering, 2020). Generated code is an example that comes to mind and is something that usually should be excluded from code coverage. It takes time and effort to exclude or ‘fix’ this code and sometimes one needs to do this time and again. One could argue that such generated code should comply to the ruleset of the code analysis tool, but changing the code generator is not always possible or will otherwise take effort on the side of the developer. Besides the contextual problem, changing code to comply with the set rules does not always improve it. I have encountered refactoring of larger methods to reduce cyclomatic complexity into numerous smaller methods. This resolved the large cyclomatic complexity but changed the code into a procedural call chain of methods that made the intention of the code more difficult to assess and harder to maintain. This brings us to the fact that resolving code quality issues does not guarantee an improvement in the readability or maintainability of the code base. This depends heavily on the way the issue was resolved and that means the involvement and judgement of a member of the development team. It also tells us that using code analysis tooling on the code level is not enough to guarantee an easy or cost-effective code base to maintain.

Figure 1: Code analysis tooling in system quality
(Ratios or scales are arbitrary)

In case automated build pipelines have been configured to comply strictly with the rule set, resulting in failed builds, effort must be spent to correct any code to abide the rules of the tool. Code must be rewritten to make the build pass or rules must be suppressed. I have seen from first hand that code adjustments to comply with certain rules can result in absurd code just to get code checked in. A review should catch these kinds of absurdities but this again means the involvement and judgement of a team member. Even with a properly configured ruleset, any effort spent in changing code to comply with this set would be great in case this would automatically and significantly improve the overall system quality. In my experience however, the effort spent usually does not improve the overall software or system in an equal way. The number of severe bugs does not go down significantly and features do not get released any quicker. The effort spent in getting the code to conform to a rule set can even be a distraction from other, possibly more important, quality aspects that are now left unaddressed.

About Reporting

Lots of tools for code analysis have some option to report on metrics and found issues. These metrics and issues are the boon for many business or development managers. The accompanying dashboards can become so important that system quality is substituted by code quality. This can be especially problematic in case managers have limited knowledge about software development and fail to address other quality aspects in the development process. A dashboard gives them something concrete to report about especially when there is pressure on time and resources which is usually the case. This kind of reporting will then move to the forefront of any report about quality of the software or even the system that development teams produce. The reason is often that no other metrics or means to measure system quality are available. A tool and a dashboard are then all that are available which reminds me of the ‘What You See Is All There Is’ principle (WYSIATI) as explained by Khaneman (Thinking, fast and slow, 2011). In our context, the report is all there is and the investment in code quality, its significance in, and contribution to system quality will often be misjudged.

Conclusion

Don’t get me wrong. I think there definitively is a place for code analysis tooling and if properly used they do guard against code errors that may be hard to catch during reviews. But too many times, these tools take the prime or sole place in system quality measurements. It is understandable to revert to this means of operation because of the difficult questions pertaining software or system quality. But one can start with certain simple aspects of quality that can be measured and are almost always relevant. The number of bugs not found during development that leak to acceptance or production are an example (Osherove, 2019). Another could be the time it takes to get a feature from inception to production or the amount of usage of certain functionality of the system.

Avoid the trap of having a code analysis tool as the sovereign ruler of system quality thereby claiming a false sense of security. Make sure that multiple processes and metrics are combined to allow for proper quality control in software development. Even if this may take some effort and discipline to implement and measure. Make sure that the most significant quality aspects of your system and development process get the highest attention. In case these are in order, a code analysis tool can be a proper complementary aid in measuring and eventually enforcing certain code quality aspects.

(Latest edit: Feb 6, 2021 - Minor textual changes)

References

[1]	D. Khaneman, Thinking, fast and slow, Amsterdam / Antwerp: Business Contact, 2011.
[2]	M. M. Dr.A.Chandrasekar, "A Research Study on Software Quality Attributes," International Journal of Scientific and Research Publications, vol. 4, no. 1, 2014.
[3]	Wikipedia, "Software quality," 2020. [Online]. Available: https://en.wikipedia.org/wiki/Software_quality. [Accessed 16 2 2020].
[4]	Continuous Quality in Software Engineering, "Facts & Fallacies of Software Measurement," 16 2 2020. [Online]. Available: https://www.cqse.eu/en/blog_drafts/facts-and-fallacies-of-software-measurement/post/.
[5]	R. Osherove, "Lies, Damned Lies, and Metrics • Roy Osherove," Copenhagen, 2019.

16-02-2020