We're still unprepared to fight the security bugs we already encounter, let alone new AI-borne issues.

5 Min Read
an image of someone holding a phone with a screen that says "ChatGPT" in front of something that says "OpenAI"
Source: Greg Guy via Alamy Stock Photo

From the first rumbles of hype for the latest culture-shattering AI tools, developers and the coding-curious alike have been using them to generate code at the touch of a button. Security experts quickly pointed out that, in many cases, the code being produced was poor quality and vulnerable and, in the hands of those with little security awareness, could cause an avalanche of insecure apps and Web development to hit unsuspecting consumers.

And then there are those who have enough security knowledge to use it for, well, evil. For every mind-blowing AI feat, it seems there is a counter-punch of the same technology being used for nefarious purposes. Phishing, deep fake scam videos, malware creation, general script kiddie shenanigans — these disruptive activities are now achievable much faster, with lower barriers to entry.

There is certainly a lot of clickbait touting this tooling as revolutionary, or at least coming out on top when matched with "average" human skill. While it is looking inevitable that large language models–style (LLM) AI technology will change the way we approach many aspects of work — not just software development — we need to take a step back and consider the risks beyond the headlines.

And as a coding companion, its flaws are perhaps its most "human" attribute.

Poor Coding Patterns Dominate Its Go-To Solutions

With ChatGPT trained on decades of existing code and knowledge bases, it's no surprise that for all its marvel and mystery, it too suffers from the same common pitfalls people face when navigating code. Poor coding patterns are the go-to, and it still takes a security-aware driver to generate secure coding examples by asking the right questions and delivering the right prompt engineering.

Even then, there is no guarantee that the code snippets given are accurate and functional from a security perspective; the technology is prone to hallucination, even making up nonexistent libraries when asked to perform some specific JSON operations. This could lead to "hallucination squatting" by threat actors, who would be all too happy to spin up some malware disguised as the fabricated library recommended with full confidence by ChatGPT.

Ultimately, we have to face the reality that, in general, we have not expected developers to be sufficiently security-aware, nor have we as an industry adequately prepared them to write secure code as a default state. This will be evident in the enormous amount of training data fed into ChatGPT, and we can expect similar lackluster security results from its output, at least initially. Developers would have to be able to identify the security bugs and either fix them themselves or design better prompts for a more robust outcome.

The first large-scale user study examining how users interact with an AI coding assistant to solve a variety of security-related functions — conducted by researchers at Stanford University — supports this notion.

Between this and the inevitable AI-borne threats that will permeate our future, developers, now more than ever, must hone their security skills and raise the bar for code quality, no matter its origin.

The Road to a Data Breach Disaster Is Paved With Good Intentions

It should come as no surprise that AI coding companions are popular, especially as developers are faced with increasing responsibility, tighter deadlines, and the ambitions of a company's innovation resting on their shoulders. However, even with the best intentions, a lack of actionable security awareness when using AI for coding will inevitably lead to glaring security problems. All developers with AI/ML tooling will generate more code, and its level of security risk will depend on their skill level. Organizations need to be acutely aware that untrained people will certainly generate code faster, but so too will they increase the speed of technical security debt.

Even our preliminary test with ChatGPT in April revealed it will generate very basic mistakes that could have devastating consequences. When we asked it to build a login routine in PHP using a MySQL database, functional code was generated quickly. However, it defaulted to storing passwords in plaintext in a database, storing database connection credentials in code, and using a coding pattern that could result in SQL injection (although, it did do some level of filtering on the input parameters and spitting out database errors). All rookie errors by any measure:

a basic login routine in PH using a MySQL database.

Further prompting ensured the mistakes were amended, but it takes significant security knowledge to course-correct. Unchecked and widespread use of these tools is no better than unleashing junior developers onto your projects, and if this code is building sensitive infrastructure or processing personal data, then we're looking at a ticking time bomb.

Of course, just like junior developers undoubtedly increase their skills over time, we expect AI/ML capabilities to improve. A year from now, it may not make such obvious and simple security mistakes. However, that will have the effect of dramatically increasing the security skill required to track down the more serious, hidden, non-trivial security errors it will still be in danger of producing.

We Remain Ill-Prepared to Find and Fix Security Vulnerabilities, and AI Widens the Gap

While there has been much talk of "shifting left" for many years, the fact remains that, for most organizations, there is a significant lack of practical security knowledge among the development cohort, and we must work harder to provide the right-fit tools and education to help them on their way.

As it stands, we're not prepared for the security bugs we already encounter, not to mention the new AI-borne issues like prompt injection and hallucination squatting that represent entirely new attack vectors that are set to take off like wildfire. AI coding tools do represent the future of a developer's coding arsenal, but the education to wield these productivity weapons safely must come now.

About the Author(s)

Pieter Danhieux

CEO, Chairman, & Co-Founder, Secure Code Warrior

Pieter Danhieux is a globally recognised security expert, with over 12 years experience as a security consultant and 8 years as a Principal Instructor for SANS teaching offensive techniques on how to target and assess organisations, systems and individuals for security weaknesses. In 2016, he was recognised as one of the Coolest Tech people in Australia (Business Insider), awarded Cyber Security Professional of the Year (AISA - Australian Information Security Association) and holds GSE, CISSP, GCIH, GCFA, GSEC, GPEN, GWAPT, GCIA certifications.

Matias Madou

Co-Founder & CTO, Secure Code Warrior

Matias Madou is a researcher and developer with more than 15 years of hands-on software security experience. He has developed solutions for companies such as Fortify Software and his own company, Sensei Security. Over his career, Matias has led multiple application security research projects which have led to commercial products and boasts over 10 patents under his belt. When he is away from his desk, Matias has served as an instructor for advanced application security training courses and regularly speaks at global conferences, including RSA Conference, Black Hat, DEF CON, BSIMM, OWASP AppSec, and BruCon.

Matias holds a Ph.D. in Computer Engineering from Ghent University, where he studied application security through program obfuscation to hide the inner workings of an application.

Keep up with the latest cybersecurity threats, newly discovered vulnerabilities, data breach information, and emerging trends. Delivered daily or weekly right to your email inbox.

You May Also Like


More Insights