A Framework for Making Informed Tooling Choices in Software Delivery Systems
The success of any software delivery system heavily relies on the tools and technologies chosen to build it. Selecting the right tooling is a critical decision that impacts efficiency, quality, and the overall project success. It is a common sight that decision making of tooling choices for software delivery systems is overlooked and carried out without a systematic approach.
In this article we explore a comprehensive guide to making informed tooling choices for software delivery systems, and formulate a framework to evaluate available tools for building a software delivery system.
Once the fundamentals of this framework are laid down, we will further explore this framework in action by taking a hypothetical use case. For this matter, we will look into upgrading the performance testing tool and infrastructure of a distributed system of a ride hailing app to meet its current demands.
Introduction
Effective software delivery systems are critical for modern organisations to deliver high quality software swiftly by improving the developer experience. The choice of tools plays a pivotal role in shaping these systems. The selection process can be overwhelming due to the abundance of tools available, each with its strengths and weaknesses.
This article intended to guide software delivery teams in making informed tooling choices by following a systematic approach that involves,
- Defining goals and requirements
- Analysing current state
- Prioritising tooling needs
- Evaluating tool options
- Conducting trials and proof of concepts
- Seeking feedback and collaboration
- Monitoring and adoption
By following this systematic approach software delivery teams could make confident tooling choices in software delivery systems that are well aligned with specific needs and goals.
Define Goals and Requirements
Start by clearly outlining software delivery goals and requirements. Define the key objectives; rapid development, high performance, or robust security. Identifying these priorities helps establish the foundation for evaluating tools.
Goals and requirements can be categorised based on the quantitative nature of the goals, non functional requirements, and technology stack. What are the key performance indicators for lead time, deployment frequency, defect escape rate, and mean time to repair (MTTR)? Are there any specific non functional requirements related to performance, security, scalability, and resilience? Consider the compatibility and integration needs of the existing technology stack as well, to ensure seamless adoption of new tools.
For our hypothetical scenario, the goals and requirements could be outlined as follows.
+------------------------------------------------------------------------+
| Goal | KPI | Target | Priority |
+------------------------------------------------------------------------+
| Increase performance|Percentage of critical | > 90% | High |
| test coverage | system components | | |
| | covered | | |
+------------------------------------------------------------------------+
| Reduce performance | Average execution time | < 10 min | Medium |
| test execution time | per test suite | | |
+------------------------------------------------------------------------+
| Improved test | Ability to simulate | Up to | High |
| environment | realistic user loads | 10,000 | |
| scalability | | concurrent | |
| | | users | |
+------------------------------------------------------------------------+
| Enhance performance | Real-time insights into | Sub-second | High |
| visibility | system performance | response | |
| | | time | |
+------------------------------------------------------------------------+
| Facilitate root | Identify bottlenecks and| Within | High |
| cause analysis | performance issues | minutes | |
+------------------------------------------------------------------------+Analyse Current State
Create a comprehensive list of the current technology stack and tooling, including each tool’s capabilities, limitations, and usage patterns. Analyse the current software delivery workflow to identify bottlenecks and areas for improvement. Leverage historical data on CI/CD pipeline performance, deployment frequency, and incident resolution times to gain valuable insights in the process.
In our hypothetical ride hailing app, we could consider that our current test setup is based on Selenium WebDriver for UI automation and JMeter for API load testing.
The current test setup is characterised by several identified bottlenecks. These include the manual creation of test scripts, which not only consumes time but also introduces the potential for human error. The test environment scalability is limited, hindering the ability to conduct comprehensive and large scale testing. Additionally, the execution times are slow, impacting overall efficiency. Real time performance visibility is lacking, making it challenging to promptly identify and address issues as they arise. Furthermore, troubleshooting proves to be a time consuming process in the current setup. Addressing these bottlenecks is crucial for enhancing the efficiency, reliability, and agility of the testing process.
In addition to the previously identified bottlenecks in the current test setup, there are further challenges due to the unavailability of historical performance test data and insufficient insights into system bottlenecks. The absence of historical performance data makes it difficult to analyse and compare past testing outcomes, limiting the ability to track improvements or identify recurring issues.
Prioritise Tooling Needs
With a clear understanding of the goals and current state, prioritise the tooling needs. Utilise the Pareto principle (80/20 rule) to identify 20% of tools contributing to 80% of the workflow pain points. Calculate the potential return on investment (ROI) for each shortlisted tool, considering both cost and potential efficiency gains. Finally, identify and evaluate potential risks associated with adopting each shortlisted tool.
By considering above identified goals and current state, we could prioritise the tooling needs of the ride hailing system’s performance test setup similar to below mentioned table.
+-----------------------------------------------------------------------+
| Tooling Need | Priority | Cost-Benefit Analysis | Risk |
| | | | Analysis |
+-----------------------------------------------------------------------+
| Performance test | High | Increased test | Moderate |
| automation framework | | converage | |
| | | Reduced execution time | |
| | | Automated regression | |
| | | tests | |
+-----------------------------------------------------------------------+
| Performance test | High | Improved performance | High |
| monitoring and | | visibility | |
| analysis tools | | Faster root cause | |
| | | analysis | |
| | | Proactive performance | |
| | | optimisation | |
+-----------------------------------------------------------------------+
| Scalable test | High | Ability to simulate | Moderate |
| environment | | realistic user loads | |
| infrastructure | | Identify potential | |
| | | performance issues | |
| | | before deployment | |
+-----------------------------------------------------------------------+
| Centralised test | Medium | Improved collaboration | Low |
| result management | | Easier access to | |
| | | historic data | |
| | | Better decision making | |
+-----------------------------------------------------------------------+
| Continuous | Medium | Faster feedback loop | |
| performance test | | Earlier identification | |
| integration | | of performance | |
| | | regressions | |
+-----------------------------------------------------------------------+Evaluate Tool Options
Once potential tools are shortlisted, search deeper into their features and functionalities. Create detailed comparison matrices that outline each tool’s capabilities. Compare technical specifications such as supported platforms, integration capabilities, and API availability. Additionally, conduct thorough security audits for any tool that handles sensitive data or introduces potential vulnerabilities.
Research and evaluate potential tool options based on the prioritised needs. Consider factors such as:
- Features and functionality: Does the tool meet the specific requirements and address the identified pain points?
- Ease of use and learning curve: Can the team easily adopt and integrate the tool into their workflow?
- Scalability and performance: Can the tool scale with the growing needs and maintain performance under load?
- Integration capabilities: Does the tool seamlessly integrate with the existing tools and ecosystem?
- Cost and licensing: Does the cost of the tool align with the budget and are the licensing terms flexible enough for your needs?
- Community and support: Does the tool have a strong and active community offering support and resources?
Based on our assessment, suppose that we outlined below tools for further investigation via a proof of concept.
+------------------------------------------------------------------------+
| Tool | Features | Technical Specifications | Security| Cost |
+------------------------------------------------------------------------+
| Gatling | Open source | Supports various | Moderate| Free |
| | framework for | protocols | | |
| | load and | Integrates well with CI | | |
| | stress testing| and CD pipelines | | |
+------------------------------------------------------------------------+
| K6 | Open source | Highly scalable | High | Free |
| | cloud based | Distributed testing | | |
| | load testing | | | |
| | platform | | | |
+------------------------------------------------------------------------+
| Dynatrace| Performance | Realtime application | High | Subscri|
| | monitoring | performance insights | | ption |
| | and analysis | Root cause analysis | | |
| | platform | | | |
+------------------------------------------------------------------------+
| AppDynam | Performance | Code level insights | High | Subscri|
| ics | monitoring | User experience | | ption |
| | and analysis | monitoring | | |
| | platform | | | |
+------------------------------------------------------------------------+
| LoadRunne| Commercial | Comprehensive reporting | High | Paid |
| r | load | | | |
| | testing tool | | | |
+------------------------------------------------------------------------+Conduct Trials and Proof of Concepts
The best way to evaluate tools is through practical experience. Don’t rely solely on theoretical evaluations. Conduct proof of concept exercises and trials with shortlisted tools to assess their real world effectiveness and suitability for the specific context. Develop test cases that simulate real world scenarios and assess each tool’s effectiveness. Measure the performance of shortlisted tools under load and compare them against the performance objectives. Finally, conduct user acceptance testing with key stakeholders to gather feedback and ensure the tools are user friendly and integrate seamlessly into existing workflows.
Seeking Feedback and Collaboration
Involve stakeholders across the software delivery lifecycle in the decision making process. Gather feedback from developers, testers, operations personnel, and other team members who will directly use the tools. This collaborative approach ensures everyone’s needs are considered and promotes buy in for the chosen tools. Diversity in perspective is crucial for informed decision making. It is crucial to avoid overlooking institutional knowledge during the process. Establish a tooling selection committee representing different stakeholders across the software delivery lifecycle. Organise technical workshops to educate team members about shortlisted tools and address their concerns. Engage with online communities and forums dedicated to the shortlisted tools to gain insights from experienced users.
Based on the stakeholder feedback, evaluation and analysis, we could arrive at the following prioritisation and decisions for our hypothetical scenario.
Immediate actions:
- Gatling: Implement Gatling as the primary performance test automation framework due to its high test coverage, reduced execution times, CI/CD integration, and open source nature.
- Dynatrace: Adopt Dynatrace for performance monitoring and analysis due to its real time insights, root cause analysis capabilities, and comprehensive performance visibility.
Actionable in the next quarter:
- Scalable test environment infrastructure: Investigate on a cloud based compute fleet for large scale load testing requirements.
- Centralised test result management: Evaluate a tool like Perfecto Test Cloud to manage and analyse performance test results efficiently.
Monitor and evaluate:
- K6: While K6 offers promising features, its suitability for our complex distributed system requires further evaluation and comparison with Gatling.
Monitor and Adopt
Remember, the tooling needs will evolve over time as the system grows. Be prepared to adapt and adjust these tooling strategies as needed to stay ahead of the curve. This tooling strategy is a living document, not static. Define and track key metrics that measure the impact of the software delivery performance. Conduct periodic evaluations of the tooling strategy to identify opportunities for improvement and adapt to changing needs. Stay up to date with industry trends and continuously learn about new tools and technologies emerging in the software delivery landscape.
For our hypothetical scenario we could monitor below metrics and continuously evaluate the suitability of our performance test tooling in the software delivery system.
- Increased test coverage: Ensure comprehensive coverage of critical system components with automated performance tests, leading to improved software quality and fewer performance issues.
- Reduced test execution time: Ensure speeds up test cycles, enabling faster feedback loops and quicker releases.
- Improved performance visibility: Ensure that the system provides real time insights into application performance and identifies bottlenecks quickly, allowing for proactive performance optimisation.
- Scalable test environments: Ensuring the performance testing system can handle peak user demands.
- Better collaboration: Ensure that the standardised tools improve the collaboration and reduce the learning curve across teams.
By following this framework and prioritising the right tools, we can ensure that this data driven approach leads to better decision making, ultimately contributing to a more successful software delivery system.
