AI code copilots are backwards-facing tools in a novelty-seeking industry

Language models should lend themselves to creating a variety of useful tools for software development and have the potential to revolutionise programming. Those tools don’t exist yet. What we have today might be the first step in that direction, but will need more improvement before they can be safely adopted by the field in general.

Code has a number of qualities that make it uniquely suited for language models. The structure and syntax of a programming language is much more consistent than any natural language. Its meaning usually doesn’t change based on context or culture. Hallucinations should be immediately discovered as errors.^[1] Train the model exclusively on permissively licensed open source code and you both make memorisation and overfitting non-issues, and minimise potential antagonism^[2] with the Free Software and Open Source communities.^[3]

Programming is the one field where the potential of language models is obvious to even the most ardent critic.^[4]

Potential, because current tools aren’t there yet, even though they feel ‘there’ to many programmers who use them. They still have a few issues.

These tools come in a few different varieties:

• Fancy autocompletes—sometimes labelled “code copilots” or “code assistants”, but they’re really just fancy autocompletes. You prompt it with a phrase, query, or code, and it will complete it with its best guess.
• Code modification and conversion tools. Tools that generate tests for your code fall into this category.^[5]
• Chat-based interfaces that are used to generate or explain code in an interactive manner.

Code modification and conversion tools build on the core strengths of large language models and are as close to a risk-free application of this technology as you can get. The other two, however, have issues.

Autocompletes were the first to gain serious adoption in the form of GitHub Copilot. Unfortunately, accidentally copying GPL-licenced code into your project is a real possibility with GitHub Copilot, which has serious consequences for projects whose licence is either proprietary or incompatible with the GPL.^[6]

That doesn’t mean other “code copilots” or the chat-based interfaces such as ChatGPT are free from risk. Autocompletes and chat-based interfaces create a dynamic that interferes with the programmer’s ability to assess the code it generates.^[7] Their design triggers two biases that make the coder more likely to accept code they would not find acceptable in other contexts.

1. Automation bias: People have a tendency to trust what the computer says over their own judgement. Or, to be more specific, when presented with a cognitive aid intended to help them think less, that’s exactly what they do: they think less. This leads them to be much less critical of the outcome of the cognitive aid than they would have of that same output in other circumstances.^[8] This has lead to serious issues in other industries, such as aerospace, where pilot trust in the capabilities of a plane’s autopilot over their own, causing catastrophe.^[9] This is a known problem with how we, as humans, interact with machines or any form of cognitive automation.^[10]

2. Anchoring bias: Our mind fixates on the first thing that anchors it in that context. For example, when it comes to shopping, the first price you see will be the anchor that you judge other prices from.^[11] The first result from an autocomplete tool is likely to trigger our anchoring bias.

These two biases combined mean that users of code assistants are extremely likely to accept the first suggestion the tool makes that doesn’t cause errors.

That means autocompletes need to be substantially better than an average coder to avoid having a detrimental effect on overall code quality. Obviously, if programmers are going to be picking the first suggestion that works, that suggestion needs to be at least as good as what they would have written themselves unaided. What’s less obvious is that the lack of familiarity—not having written the generated code by hand themselves—is likely to lead them to miss bugs and integration issues that would have been trivially obvious if they had typed it out themselves. To balance that out, the generated code needs to be better than average, which is a tricky thing to ask of a system that’s specifically trained on mountains of average code.

Unfortunately, that mediocrity seems to be reflected in the output. GitHub Copilot, for example seems to regularly generate vulnerable code with security issues.^[12]

Another issue is that AI coding tools will never push back against bad ideas. They will always try to generate a solution, even when it has no hope of generating safely working code.^[13] Many of the first ideas we have, as coders, are ill-advised. It comes with the territory. You have a problem. Think of a solution. Do some research to see how it would be done. Discover that the solution was a bad idea. Type “how to implement your own cryptography” into Stack Overflow and you will get answers that tell you, unequivocally, that you absolutely should not implement your own cryptography. Programming is full of these pitfalls and AI tools, with their hallucinations and a tendency to generate confident answers no matter what, are almost purpose-designed to trip you into them.

Some of these pitfalls have to do with age. Platforms, libraries, and APIs become obsolete, insecure, and outdated, but language models are, because of their design, stuck in the past. Newer methods are going to be underrepresented in their training data set if they appear in it at all. You are likely, for example, to run into instances where they would recommend you use outdated open source libraries that have since been replaced with faster and more reliable platform implementations. Or, it might generate code that uses APIs that have been deprecated or even removed in current versions of the platform. If the AI copilot isn’t ‘fresh’ enough, it’s going to cause problems.^[14]

Code assistant language models will need to be updated much more frequently than natural language models if they are to remain useful in the long term.

A related issue is cost: the biggest expense for most software projects is maintenance.

Software rots. Platforms change from under them. Software libraries switch up their APIs, requiring changes in your project. The understanding a programmer has of the problem changes, revealing flaws in the initial implementation. Requirements change. Most of the work of programming is in updating and changing code, not in writing code from scratch.

Autocompletion tools are of no help in this context. Chat-based tools are only helpful if you need your own code explained back to you, which is now more likely to happen because you might not have written the code yourself in the first place.

Code generation tools can even make this–the most expensive part of programming—even harder and more expensive. By making it incredibly easy to generate a lot of bug-filled, insecure, but broadly functional code, they are likely to lead to code base inflation.

Projects will be bigger while doing less, making them harder to fix, update, and maintain. By going faster, we will have slowed ourselves down. The cost-saving automation of generative AI is likely to be very expensive in the long run.

This was an excerpt from The Intelligence Illusion