Cory Doctorow on code liability:
Code is a liability. Code’s capabilities are assets. The goal of a tech shop is to have code whose capabilities generate more revenue than the costs associated with keeping that code running. For a long time, firms have nurtured a false belief that code costs less to run over time: after an initial shakedown period in which the bugs in the code are found and addressed, code ceases to need meaningful maintenance. After all, code is a machine without moving parts – it does not wear out; it doesn’t even wear down.
[…]
“Writing code” is an incredibly useful, fun, and engrossing pastime. It involves breaking down complex tasks into discrete steps that are so precisely described that a computer can reliably perform them, and optimising that performance by finding clever ways of minimizing the demands the code puts on the computer’s resources, such as RAM and processor cycles.
Meanwhile, “software engineering” is a discipline that subsumes “writing code,” but with a focus on the long-term operations of the system the code is part of. Software engineering concerns itself with the upstream processes that generate the data the system receives. It concerns itself with the downstream processes that the system emits processed information to. It concerns itself with the adjacent systems that are receiving data from the same upstream processes and/or emitting data to the same downstream processes the system is emitting to.
“Writing code” is about making code that runs well. “Software engineering” is about making code that fails well. It’s about making code that is legible – whose functions can be understood by third parties who might be asked to maintain it, or might be asked to adapt the processes downstream, upstream or adjacent to the system to keep the system from breaking. It’s about making code that can be adapted, for example, when the underlying computer architecture it runs on is retired and has to be replaced, either with a new kind of computer, or with an emulated version of the old computer.
Because that’s the thing: any nontrivial code has to interact with the outside world, and the outside world isn’t static, it’s dynamic. The outside world busts through the assumptions made by software authors all the time and every time it does, the software needs to be fixed. Remember Y2K? That was a day when perfectly functional code, running on perfectly functional hardware, would stop functioning – not because the code changed, but because time marched on.
We’re 12 years away from the Y2038 problem, when 32-bit flavors of Unix will all cease to work, because they, too, will have run out of computable seconds. These computers haven’t changed, their software hasn’t changed, but the world – by dint of ticking over, a second at a time, for 68 years – will wear through their seams, and they will rupture.
[…]
AI can write code, but AI can’t do software engineering. Software engineering is all about thinking through context – what will come before this system? What will come after it? What will sit alongside of it? How will the world change? Software engineering requires a very wide “context window,” the thing that AI does not, and cannot have. AI has a very narrow and shallow context window, and linear expansions to AI’s context window requires geometric expansions in the amount of computational resources the AI consumes.
Can AI be trained in such a way it would handle constantly changing world? Can it be trusted to make right decisions in unexpected situations? Humans are incredibly complex, and even small deviations in situations could make previous decision undesirable. Since AI is not a deterministic system, how can it be trusted to make right choices?
If I have to choose between a strict algorithm that fails well in unexpected situations, and AI that attempts to come up with any solution it can “think” of, I’ll always choose the algorithm. I’d sleep better at night knowing my system will not make some catastrophic decision because neural networks decided that deleting all data that it’s causing issues would get rid of the problem.
Can AI become good enough that it could navigate complexities of the real world and make correct decisions in novel situations? I guess we still yet to see. But one thing is clear - having people involved is not going anywhere any time soon.