The uncertainty principle – Technology adoptive product development

Introduction

Every company that produces software is faced with the uncertainty surrounding new technologies or new versions of existing ones. The rate at which technologies, or cloud services today, emerge and change is at an all-time high (**). Besides, there is a tendency to reduce the lifetime of a certain technology version by reducing the time that the version will be officially supported. This goes hand in hand with the relatively short release cycle of major releases, each superseding the previous one. In lots of companies there is not enough time for development teams to investigate the application of emerging new technologies. This gradually slides the development team and its company into maintenance mode (Figure 1).

Figure 1: Different modes of development

But these new and unknown technologies sometimes pose considerable risk because of lack of experience or knowledge within the team. There are several ways to overcome these risks.

Training

The first one is obvious. Just send some team members to a training. This is not always the best option. The team may be provided with a broad initial training that may not be needed for your project. Besides, the training can be an investment that a company or the developer is unwilling to make. This reluctance will especially be prevalent in case it is not yet certain that the technology will be committed to. So, this can become a chicken and egg problem. And even with the training of a development team, there may still be plenty of remaining risks in the application of this new technology to your business problem.

Prototyping

Prototyping [1] is another way to mitigate risks and probably the best-known form is ‘throw away’ prototyping. This type of prototyping can be used to prove that the implementation can successfully solve the problem at hand and identify risks by uncovering unintended and unknown behavior. In case there is no or limited knowledge of this specific technology, this method can also be used to ‘test’ the technology to see whether it would be suitable for adoption. The advantage is that one can focus more on the initial business problem, or development cycle that is to be completed. However, the amount of knowledge that can be attained during these short development cycles, is limited and can lead to too limited analysis [2]. This is especially the case when the technology that is to be applied has many aspects or is inherently complex. The ratio of the amount learned against the amount unknown can remain too low which makes it difficult to make an informed decision about whether or not to commit to the new. Besides that, real world business problems may prove more intricate than can be tested during throw away prototyping and may manifest other requirements that could easily be missed due to the limited prototype phase. There are several ways that one can react or respond to this situation. The first reaction could be that there is no time for anything besides the ‘throw away’ prototype and that any risky investment in new technologies should be avoided at this point in time. Another is to place the ‘burden’ on the development team to investigate technology in their own time. These kinds of reactions can be provoked because of planning pressure or bad management. This may even be the case if the expected benefits of the new technology are perceived to eventually outweigh its drawbacks. The time constraint may be a viable reason to avoid investment in any new technologies but often is a symptom of a larger cultural problem where deadlines play the upper hand and software development is sliding into an ever-greater unsustainable exercise.

Hiring Experts

The third option is to temporarily hire the knowledge in the form of an external consultant. He or she may even help with the initial application of the new technology. Be aware though that a consultant may bring its own bias into the decision that you are going to need to make and be responsible for. Besides that, there is the risk that in case the consultant is hired, new knowledge and experience are not gained by current team members. This is especially prevalent when there is lots of pressure on the development team to focus on existing features. This may pose a risk after the ‘hand-off’ when the new product must be maintained by the original team that had limited or no exposure to the new technology and the consultant has left for a new project. Exposure to, or even better involvement of the existing team in, the application of new technology should still be a priority. Another issue can be that the consultant in question may need to learn a lot about the installed software base and may need to learn about the company’s problem domain and software development principles before being able to satisfactory apply the technology. This may take an extensive amount of time. But this option may not be suitable. For new technologies, especially certain new cloud services, there will initially be limited experience in the market. So, one has to do without any technology expert or consultant. Of course, there is the option to postpone the application of the new technology service until there is more knowledge in the market (e.g., late adopters)

Technology Ad o ptive Product Development

Definition

There is a fourth option that holds the middle between making a throw away prototype and creating the final product. This approach has parallels to what was called ‘Betting on Evolutionary Architecture’ by James Lewis [3] To avoid any confusion between this approach and other similar terms or prototyping approaches, I will call this method technology adoptive product development. The main differences with throw away prototyping is that the result will not be thrown away but that eventually, the result will be taken into production. During this approach the feedback to the developers and testers comes from the behavior of the technology when used to solve a certain business problem. The main goal in technology adoptive product development is to accommodate developers in learning and applying a new technology to a limited but real business problem. With each development cycle the software gets closer to something that can and will eventually be taken in production. To accommodate this way of development there are certain preconditions that are necessary to maximize the chances of success.

Success factors

Perhaps the most important one is that you will need talented and experienced developers. The talent that is needed is that a developer is good at adopting the technology in his or her development work [4]. The experience is needed to make better decisions with the uncertainty of the moment [3]. This is important in order to make more correct decisions during the choices that are to be made during development. Figure 2 shows the choices made (white dashed line) over het possible choices (purple tree). The purple tree represents a tree where some choices lead to a dead end and where one should revert a previous decision. Eventually a sufficient quality product from both a technical and business perspective is reached. Another aspect that is important in order to proceed, is that the business problem should be relatively small or can be solved in small steps. The third, and this one is sometimes hard to predict, is that a first version of the business problem can ideally be solved with a limited set of the new technology. This one is crucial because it means that the learning curve will usually take less time but still in the end will result in a functioning piece of software. Fourth is that the new technology is the only new piece of technology that will be applied and that the technology can be relatively easily integrated into the existing system; no other new technologies should be necessary for the development to proceed. Microservices or a system composed of small components can be a tremendous help in this aspect. The last is that a technically oriented tester is involved from start to end. The reason for this is that testing is of paramount importance. In case the new software or component is thoroughly tested in all relevant aspects, aspects that go beyond testing the required functionality, there is more confidence that the new technology has been properly applied. It may even help to uncover certain behavior of said technology that could otherwise be overlooked.

Figure 2: The road to sufficient quality

Results

The end result will be that the team has been given a chance to acquire the new skills and learn the new technology while at the same time tackling a business problem, or a small part of it. The fact that the team was able to do this can give a big confidence boost to the team itself, the technology used and any business managers. They may have been sceptic about the introduction of new technology and the instability that it could cause. This achievement paves the way for evaluation of the technology and new iterations of the previously introduced technology or new, other technologies. This result can not be underestimated because it may be the start of a cultural change where new technology aversion will eventually subside. Adoption of new technologies may become mainstream during development cycles. Note that evaluation after the successful first implementation is still critical. As said before, during development of a solution for a real business problem, issues may arise that may not have come to light during development of a proof of concept or throw away prototype. This means that, in hindsight, the team may have learned that some things could have better been done differently, some things could have been done better. There is also still a small chance that eventually the decision could be made NOT to continue with said technology and an alternative should be examined. Assuming that the new technology is mature enough to proceed with, technology adoptive product development has another advantage. Because the new technology was adopted quickly and efficiently, the business benefits from a higher ROI because of the fast technology adoption. This means that the time to the next maintenance or major release of the applied technology is optimized.

Conclusion

With the current relentless release cycles of major technology providers(**), it is too expensive to perform lengthy analysis trajectories of months or years to try to find out how exactly a certain technology would fit with the currently installed software and systems base. New major releases will already be looming on the horizon by the time any lengthy analysis is complete, and a new version could have emerged with new important features that could, and often will, invalidate any lengthy analysis that was performed. Other and faster ways of adopting new technologies should be evaluated and applied. The approach described here gives just another option to mitigate risks of adopting new technologies besides training, prototyping and hiring knowledge. Technology adoptive product development will integrate technology adoption in your software process. The astute reader will have no doubt noticed that all that was applied here was the fail-fast principle while at the same time reducing major risks and using fast feedback cycles. The important thing here is though, that failure has been taken into account while maximizing for success while adopting new technologies.

(Latest edit: Feb 6, 2021 - Minor textual changes)

------------------------------------

Notes:

(*) The author is well aware that the 'uncertainty principle' usually refers to the one defined in quantum mechanics as described by several source inluding Wikipedia [5]. However, every endeavour in software development is ultimately uncertain to a specific level.

(**) Examples at the time of writing are Angular with a release cycle for major releases of every six months and a support lifetime of eighteen months [6]. Microsoft DotNet currently has a new major version every year with some of these versions having a ‘Long Term Support’ of about three years [7].

References:

[1]	Wikipedia, "Software prototyping," [Online]. Available: https://en.wikipedia.org/wiki/Software_prototyping.
[2]	Wikipedia, "Software prototyping - Disadvantages of prototyping," [Online]. Available: https://en.wikipedia.org/wiki/Software_prototyping#Disadvantages_of_prototyping. [Accessed 31 01 2021].
[3]	J. Lewis, "Betting on Evolutionary Architecture as Code - a note on hypothesis-driven architecture," Oslo, 2018.
[4]	K. Tate, "Chapter 3: The Principles of Sustainable Software Development," in Sustainable Software Development, Upper Saddle River, NJ, Pearson Education Inc., 2005, p. 39.
[5]	Wikipedia, "Uncertainty principle," [Online]. Available: https://en.wikipedia.org/wiki/Uncertainty_principle. [Accessed 31 01 2021].
[6]	Angular team, "Angular versioning and releases," 31 01 2021. [Online]. Available: https://angular.io/guide/releases#release-frequency.
[7]	Microsoft, ".NET Core and .NET 5 Support Policy," [Online]. Available: https://dotnet.microsoft.com/platform/support/policy/dotnet-core. [Accessed 31 01 2021].

31-01-2021