Statistics & Studies: The Benefits Of Test Driven Development
In this article, I review some test driven development statistics and studies to understand how it's been employed, the benefits, and the challenges teams face with this approach.
Traditionally, the process of software development proceeds linearly. However, in the last few decades, as agile systems are becoming more popular (as many as 87% of teams follow an agile or agile-like approach), software development has started to embrace different methodologies that take the project's requirements and character into account.
The test-driven development (TDD) technique is one of the methods that has been attracting attention in the agile software development area.
In a research paper published by the Institute of Electrical and Electronics Engineers, authors Yahya Rafique and Vojislav Misic say that “Test-Driven Development (TDD) is among the cornerstone practices of the Extreme Programming (XP) development process” (Source).
The man who is said to have developed TDD is reported to have “stated in 2003 that TDD encourages simple designs and inspires confidence” (Source). However, questions remain regarding the productivity and quality claims made about TDD.
We took some time to gather the latest TDD statistics to test the claims made.
This article starts by defining the concept of TDD, and how it differs from the traditional approach. We then look at some of the statistics which either validate or reject the claims made about TDD.
What Is Test-Driven Development?
Test-driven development is an approach where a test is written before the software developer creates the production code to fulfill the test. This technique's basic idea is to allow the writer of code to take some time to consider their design or requirements before writing functional code.
TDD Process
- Writing the Test: In TDD, all new features begin with the writing of a test. The programmer must understand the specification and requirements of the feature. To accomplish this, they will need to go through user stories and use cases to understand the aim of the new code they are developing.
- Test Fails: Once the programmer has written the test, they run it. Since there is no code to implement it yet, the test will fail. This confirms that the automated test framework works correctly and rules out the possibility that the new test will always pass because it is defective.
- Write the Code: Now, the programmer knows that the feature works according to the design. They now write the code that passes the test. The code might not be perfect, or pass the test excellently, but that doesn't matter. The developer isn't expected to write codes beyond the functionality the test was created to check for.
- Run Tests: Knowing that the feature works as designed, whenever they release new code, the programmer can use a test management tool to run that test again, providing them with the validation that their most recent update has not broken the previous feature.
- Refactor Code: In TDD, as the code base grows, it has to be cleaned continuously. Considering that the programmer’s main focus in the above stages was to write the code only, this stage ensures efficiency. It allows the program's source code's internal structure to be improved while its external characteristics are preserved. This stage can remove duplications and add new functionalities.
- Repeat: The above steps are automatically repeated to ensure that the TDD cycles cover all the features.
Differences Between TDD And Traditional Development
To understand TDD, it seems appropriate to determine how it differs from the traditional approaches to programming.
The main difference is that the traditional methods follow a linear process, while TDD goes through a cyclical process.
Programmers who use the traditional testing methods start by creating the code and only concentrate on the test at the end of the development process. On the other hand, the one who follows the TDD model starts by creating the test and then develops the code that meets the test. This approach shares a lot of principles from the shift left movement in software testing.
While the programmer using the traditional approach can pay attention to the code's correctness, they run the risk of failing to detect all the code's failures. The programmer who uses the TDD method works on the code until it passes the test through refactoring the code. This is done until the code meets functionality, a process likely to result in fewer bugs.
Is TDD Slower Or Faster Than Traditional Test Development?
Concerning whether TDD makes the programming process move faster, there are conflicting statistics from various sources.
A case study involving Microsoft and IBM teams of software engineers concluded that the “teams experienced a 15-35% increase in initial development time” when they used the TDD technique. However, the study notes that these are numbers “subjectively estimated by management” (Source).
When looked at from the perspective that Microsoft and IBM studies indicate that there were improvements in quality, it can be argued that, in the long run, TDD saves the time that would have been required to fix problems. The teams, at Microsoft and IBM, agreed with this view (Source).
A study focusing on the initial perceptions of experienced professionals when using TDD concludes that “after overcoming the initial difficulties to understand where to start, and know-how to create a test for a feature that does not yet exist, participants gain greater confidence to implement new features and make changes due to broad test coverage” (Source). What this seems to suggest is that things improve with time.
Writing for the online publishing platform, Medium.com, programmer and author of “Composing Software” and “Programming JavaScript Applications,” Erick Elliot focuses on how TDD changed his life. Elliot aggress that the process can be slow in the beginning, but says, “somewhere around the 2-years in the mark, something magical started to happen: I started coding faster with unit tests than I ever did without them” (Source).
It looks like using TDD may make things slower initially. However, if looked at from a long-term view, the time saved by better quality code may compensate for the time lost in the beginning. Also, it can be expected that as programmers get better at TDD, they are likely to move faster.
There are also many other factors to consider when trying to measure the time it takes to get a high quality result. For more on this, tune into Niall Lynch's episode on The QA Lead podcast on measuring T2Q (Time To Quality).
Does TTD Result In Fewer Bugs?
In the discussion above, one of the main advantages of TDD presented is that it results in fewer bugs. But do the statistics agree?
The same studies involving the engineering teams at Microsoft and IBM above concluded that “the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice.” Specifically, the IBM teams reported a drop of 40% in defect density, and those at Microsoft reported a 60-90% drop (Source).
Do Test Driven Development Statistics Support The Conclusion That TDD Produces Better Quality?
Based on the results of a study presented at the 2007 First International Symposium on IEEE in Finland, Maria Siniaalto and Pekka Abrahamsson report that TDD has been shown to produce better code quality when compared to non-TTD developed software (Source).
In their paper, Siniaalto and Abrahamsson cite a study conducted in China that concluded that TDD improved process tracking and task estimation. The same study concludes that “TDD also enhances the following of consistent practices and guidelines.” This results in better quality with fewer defects. Also, the teams that used TDD were better able to fix their defects more rapidly (Source).
A study carried out amongst developers, with about ten years of professional experience (on average), to investigate their perceptions when employing TTD, quotes a developer who says, “TDD has helped me to improve the code, making it more readable.” Another participant reports that “TDD allows greater maintainability” (Source).
Does TDD Promote Simpler Design?
Boby George and Laurie Williams, both working in the Department of Computer Science at North Carolina State University, ran an experiment where 24 programmers were put into two groups: one used TDD and the other the linear approach.
George and Williams report that out of the participants, “92% of developers believed that TDD yields higher quality code, 79% thought that TDD promotes simpler design and 71% thought the approach was noticeably effective” (Source).
These test driven development statistics about quality give a strong indication that TDD does, in fact, result in higher quality code and simpler design.
In an article published by the free learning portal, Guru99.com, Kanchan Kulkarni, says, “TDD makes the code simpler and clear. It allows the developer to maintain less documentation” (Source).
Is it Easy to Adopt TDD Design?
From George and Williams's experiment, 56% of the professional developers believed it was difficult getting into a TDD mindset, while 23% claim that the lack of an upfront design phase is the reason for this difficulty. Of the total responses, 40% believed that the adoption of TDD is difficult (Source).
These test driven development statistics about adoption signal that TDD is seen as difficult to adopt.
Is TDD Overrated?
In an article published on Medium.com, Tylor Borgeson, who calls himself a Full Stack Software Developer interested in Machine Learning, AI, Infrastructure, DevOps, and Agile, uses the headline “Test-Driven Development is Overrated.” However, the fact that he has placed the headline in quotation marks shows that this is not a statement he is making.
Borgeson then addresses those who say that the method is overrated and slow, telling them that most people who hold this view haven't used the method long enough. Ending his article, he says, “Now go practice Test-Driven Development until it doesn’t hurt anymore” (Source).
What's Next?
Learn QA approaches from experts in The QA Lead podcast
Sign up for The QA Lead newsletter to get our latest how-to guides and podcast episodes
Join the waitlist for The QA Lead online community forum where you can share best practices with other QA and software testing professionals.
Hope to see you there!
