Understanding Software

We don’t understand how AI works. We need to slow down.

That’s a fairly common sentiment today but my thought is, “Do we really understand any of the software we’ve built so far?”

In my experience, no. To me it seems an almost impossible task to fully understand software. We can have tests to verify and check correctness but there’s no guarantee we have written all the tests, and all of the right tests.

If I understand the software I’ve built then I should be able to explain all of its possible states, but I can’t. More than that I should be able to predict all of its possible states but you already know how difficult that is.

A common misunderstanding about software is that it is defined by the code that we write. The code is merely instructions for the CPU to execute. The software that you use, experience, is emergent. It is the outcome of THAT CPU, in YOUR computer, running THOSE instructions, in THAT specific order. What complicates this matter further is the fact that it’s not just your instructions running in the computer.

Over the years we as an industry have found ways to merely contain this complexity and make some sense of it. Operating systems, privileges, memory boundaries, processes and threads, etc. These are all but abstractions and help us to both express our intent and also contain the complexity behind it.