Unlocking generative AIs true value: a guide to measuring ROI

This includes aspects of generative AI systems such as models, deployment pipelines, and various interactions within the broader system context. The true value of gen AI goes beyond numbers, and companies must balance financial metrics with qualitative assessments. Improved decision-making, accelerated innovation and enhanced customer experiences often play a crucial role in determining the success of gen AI initiatives—yet these benefits don’t easily fit into traditional ROI models. Despite strong adoption and business benefits, some leaders highlight the risks of AI code assistance. Organizations adopting AI for devops and software development should define non-negotiables, train teams on safe utilization, identify practices to validate the quality of AI results, and capture metrics that reveal AI-delivered business value. Small time savings during the agile development sprints can yield larger benefits when aggregated across functional release cycles.

CSO Executive Sessions: How AI and LLMs are affecting security in the financial services industry

Last year, I wrote about the 10 ways generative AI would transform software development, including early use cases in code generation, code validation, and other improvements in the software development process. Over the past year, I’ve also covered how genAI impacts low-code development, using genAI for quality assurance in continuous testing, and using AI and machine learning for dataops. In the race to harness the transformative power of gen AI, enthusiasm alone won’t generate returns. As companies confront the complexities of measuring impact, they must move beyond traditional metrics to embrace a more nuanced understanding of value—one that accounts for both tangible and intangible outcomes. The path to success lies not in grand, sweeping implementations but in focused, high-impact initiatives that align with business objectives and evolve over time.

Subscribe to Newsletter to get latest insights & analysis in your inbox.

With the right strategies and investments in 2024, we can continue to build on the strong foundation we have established in enabling secure and seamless work from anywhere. In 2023, the cybersecurity industry experienced massive shifts when it comes to the technology we use and how we use it. Quality assurance practices, including test automation and code reviews, are another area where genAI provides value to devops teams. In the 2024 State of Software Quality report, 58% of respondents said that time constraints were their most significant challenge when performing code reviews. According to the report, more than 50% of respondents were using AI in some aspects of code reviews.

Training employees on how to leverage new technologies safely and responsibly is crucial for fostering an environment of true innovation. As businesses adopt and adapt, forward-thinking technology leaders and CIOs will face new questions and challenges to prepare their technology stacks, platforms and organizations to take advantage of this unprecedented technology wave. Seemingly overnight, this revolutionary technology has dropped millions of jaws by auto-assembling volumes of structurally sound sentences and fully functional lines of code. It’s become such a hot topic that even the Kardashians must be getting jealous. I believe generative AI will bring massive changes in how companies run their business, the technology solutions they need to compete and the skill sets required of their employees. To move from AI hype to real-world productivity gains, they must lead the charge in reimagining the digital workplace.

Alternatively, teams may decide they want to forgo buying and instead build their solution in-house. First, however, they’ll have to assess the specific infrastructure needed, navigate commercial licensing and resource the team correctly to train the models (among other steps). With most of the unstructured data stored as notes in case management systems, the federal CIO should be looking for a strategy to house unstructured data and leverage it for future knowledge management and self-service needs. • Deploy virtual assistants to support employees with administrative tasks such as scheduling, procurement requests and IT troubleshooting.

• Bogdan Raduta, head of AI at FlowX.AI, raises questions about quality and innovation when businesses rely too heavily on generic user experiences and AI defaults to patterns and conventions.
• CIOs need to rethink operating models to balance democracy with governance.
• “We’ve had to be intentional about piloting solutions like ambient voice documentation, ensuring measurable outcomes, and supporting adoption through training and provider input — not just rolling out tools for the sake of innovation,” he said.

Premier Health reports data breach

This technology holds the potential to revolutionise productivity by transforming how organisations personalise the employee experience. And 90% of CIOs, IT directors and VPs of IT believe digital workplace transformation is essential for employees to use AI effectively. Developers should continue to explore AI capabilities for building software and developing experiences, especially because these capabilities are evolving quickly. While experimentation is needed, devops teams and IT departments should create target goals and metrics for AI benefits while seeking benchmarks for where other organizations are delivering value. Even when SaaS platforms announce agentic experiences, data teams should evaluate whether data volume and quality on the platform are sufficient to support the AI models.

CIOs are always under pressure to rationalize their software usage and total spend to their organizations. Mobile apps for the field usually consist of forms, checklists, access to information, dashboards, and reports. They can inform field operations about work that needs to be done, answer implementation questions, and provide information to planning and scheduling teams working at the office. The OWASP generative AI red teaming guide closes out by listing some key best practices organizations should consider more broadly.

These indirect and intangible benefits, while potentially transformative, are notoriously difficult to capture in conventional ROI calculations. Join leaders from Block, GSK, and SAP for an exclusive look at how autonomous agents are reshaping enterprise workflows – from real-time decision-making to end-to-end automation. What matters most is preparing your workforce, thinking through the change management process, reshaping business workflows, and acquiring new skills. This change process should be underway now so your team members will be ready to run with the full potential of the technology at scale — safely and ethically. AI raises profound ethical questions that extend beyond any single organization, and CIOs also have a responsibility for building guardrails, advocating for standards, and promoting responsible AI development and deployment. The real value of technology investments lies in their “option value” — the pathways they open for future innovation.

Want to know how the bad guys attack AI systems? MITRE’S ATLAS can show you

• Teams often reach peak performance just as the project ends and they split up — throwing away their hard-won collective intelligence.
• Areas like time tracking, communications, and job reporting with minimal industry-specific business needs are early use cases that will appear in vendor applications.
• They also track the number of accurately flagged high-risk accounts as a key measure of gen AI’s predictive power.
• This 12-step approach balances quantitative metrics like cost savings and revenue generation with qualitative benefits such as improved customer experience and enhanced decision-making.
• As companies confront the complexities of measuring impact, they must move beyond traditional metrics to embrace a more nuanced understanding of value—one that accounts for both tangible and intangible outcomes.

Building scalable systems and adaptable talent strategies ensures readiness for the next wave of transformation. If you’re not investing for both the short and long term, you’re designing for obsolescence. While generative AI is exciting, we also must acknowledge that cybersecurity should remain mission-critical. Customers and partners trust us to secure their data and operations, and it is on us to ensure we are maturing our cyber defenses through leading technology, automation and best practices.