What do we want from our AI?
In collaboration with The National DevOps Conference and Awards, we interviewed Jody Bailey, CTO at Stack Overflow, on his view on Generative AI and AI tools.
The National DevOps Conference & Awards takes place in London on the 22nd and 23rd of October 2024. To exhibit your products at the event, please get in touch here.
In mid-2024, Generative AI has moved beyond the early adopter phase and its hype cycle moment, so questions about its staying power have now been replaced with discussion of how technologists and organisations can and should best harness its efficacy and power. One of the benefits of GenAI tools most frequently touted by advocates is increased productivity and efficiency.
However, recent data suggests that how enterprises measure productivity is often murky at best. Despite this, AI tools are here to stay, so it is in the best interest of technologists and companies to understand which tools are being used the most and how AI impacts the overall developer experience.
Top AI Tools for Developer Productivity in 2024
A recent survey by Stack Overflow on coding assistants and developer productivity found that 76% of developers are using or are planning to use coding assistants, and of those using them, ChatGPT leads the way as the most often used tool with 84% of professional developers using it, followed by GitHub Copilot at 49% and Visual Studio IntelliCode at 11%. Developers also cited that the tools they found most satisfying to use were also the easiest to use, with Codeium ranking highest for both satisfaction and ease, with GitHub Copilot and ChatGPT coming in second and third respectively.
However, GenAI tools are not without their shortcomings, as technologists relayed that coding assistants also consistently struggle with context, complexity, and more obscure technical concepts or instances. For example, one user wrote that code assistants have trouble with “[m]ost high-level or architectural questions. Or questions regarding niche components, or resources behind NDAs”. Another user wrote that their coding tool of choice provided incorrect responses when working with lesser-known “programming tools (or tools of [a] specific language).”
Challenges and Limitations of AI Tools in Software Development
Issues of trust and accuracy linger when it comes to generative AI. Stack Overflow’s 2024 Developer Survey demonstrates a growing gap between increasing AI usage and stagnating levels of trust and favourability. 76% of all respondents are using or planning to use AI tools up from 70% in 2023, while AI’s favourability rating decreased from 77% last year to 72%, while only 43% of developers trust the accuracy of AI tools’ output (only one percentage point higher than a year ago).
When it came to using AI tech specifically on their teams at work, respondents to the coding assistant and productivity survey also indicated an inability to handle a higher level of complexity and a lack of trust in AI as the top two reasons preventing their team from adopting these tools, and those in organisations with lower adoption rates also say their colleagues not using coding assistants is a challenge to integrating AI into a larger team’s workflow. Adding to the confusion and potentially widespread use, nearly three-quarters of respondents also claim they are unsure if their company has an AI policy or guidance in place.
The future for Generative AI
Developers value both satisfaction and ease of use when it comes to GenAI. Although code gen tools may not be the saving grace to technologists some heralded them to be, they are nonetheless helpful for automating tedious, time-consuming tasks, summarising large amounts of text or data, and even empowering creativity among teams. Despite this, inaccuracies and a lack of trust remain a barrier. How can we move forward to a future where AI is used to empower workers?
Ultimately, improving trust in AI is all about improving the quality of data available to LLMs. As the old adage goes, garbage in, garbage out. Human oversight is also critical because AI tools are just that – tools. The nature of working as a developer is complex and dynamic and it’s the vast community of people that bring experience, context, and nuance to the table.
Developers are highly satisfied with AI tools that are easy to use and are increasingly incorporating them into their workflow, despite productivity gains not being clearly articulated as a business KPI. Should this continue, we may just find ourselves in an era where the developers’ own perception of productivity and ease of use become more important than organisational productivity metrics.
Explore the Future of Generative AI at the National DevOps Conference in London
Join us for an in-depth discussion on the scope and future of Generative AI at The National DevOps Conference and Awards, happening in London on October 22nd and 23rd, 2024. This premier event will feature expert insights into how AI is transforming DevOps practices and the broader tech industry.
View the Full Agenda: The National DevOps Conference and Awards Agenda
Exclusive Offer: Gain free entry to the conference by submitting your project to the DevOps Awards before the September 16th deadline. Don’t miss this opportunity to showcase your innovation and network with industry leaders.
For exhibit at the conference, please contact calum.budge@31media.co.uk
Foe media enquiries, please contact vaishnavi.nashte@31media.co.uk
