Software Decays. AI Doesn't. (At least, not in the way we think.)

Software Decays. AI Doesn't. (At least, not in the way we think.)

"Perhaps the thing we're really maintaining was never the software."

Software decay is one of those ideas that feels almost too obvious to discuss.

Ask any engineer whether software ages and the answer is immediate. Of course it does. Operating systems reach end of support. Third-party libraries become unsupported. Java will always be Java. 

Sony decides to no longer make games on physical media. 

Hardware disappears from vendor catalogues.  Security vulnerabilities are discovered in code that once seemed perfectly acceptable. Even successful software slowly accumulates technical debt as the world around it evolves.

As an introduction. None of that feels controversial because we've spent decades building an engineering discipline around the problem.

We don't expect software to remain unchanged forever. We expect to patch it, test it, upgrade it and, eventually, replace it. Entire professions exist because software decay is both inevitable and manageable.

That strikes me as one of engineering's quieter successes. Cyber security exists as a profession because software deteriorates at a different rate than our cyber defenses. Almost everything does. Even my knees. Mostly from kneeling on hard cold server room floor tiles. 

As cyber security analysts and architects we've become remarkably good at recognising deterioration before it becomes failure.

We become better at cyber security when we understand the difference that a vulnerability scanner doesn't tell us whether a system has been compromised. It tells us where confidence should begin to diminish. 

Regression testing doesn't prove software is correct. It increases our confidence that recent changes haven't introduced unexpected behaviour. Disaster recovery planning isn't really about disaster. It's about confidence that the organisation can continue operating when assumptions eventually prove wrong.

Cyber security has to understand that software engineering has always been less about maintaining software than maintaining confidence in software. Which is why as cyber security we should ask our software engineering co-workers if they also want anything from the vending machine when we walk past their desks.

Here’s where those decades of knowledge is adjusting, because now we have to also think about AI.

AI doesn't age in isolation.

Software ages largely because technology changes around it.

AI ages largely because reality changes around it.

At first glance, AI appears to inherit the same maintenance story.

Models require infrastructure. Infrastructure requires patching. APIs evolve. Hardware fails. Dependencies need updating. There is comfort in believing that AI simply extends software engineering rather than changing it.

After all, a model is still software executing on a computer, or at least, that's how I initially thought about it.

Story Time.

I was standing in a McDonald's in London during one of AWS's major outages affecting the US East region.

The menu boards had become erratic. Some were blank. Others were only partially rendering. Staff were apologising to customers whilst trying to explain why ordering lunch had suddenly become difficult.

It took me a moment to realise what had happened.

The menu boards in London weren't really talking to London.

They were talking to the AWS data centre in Northern Virginia.

Had the McDonald's explained that to the waiting customers that would have sounded strangely absurd, to everyone waiting for their food.

Many of the people standing in that restaurant had probably been there dozens of times. Some might have lived in London their entire lives. Yet whether they could order lunch depended, in part, on systems operating in Northern Virginia.

The software hadn't changed. Neither had the physical menu display screens failed.

The restaurant's local network hadn't collapsed.

One design assumption had quietly become false.

Somewhere, someone had decided that a restaurant in central London could depend on services operating thousands of miles away. Most days, that assumption held perfectly. On that particular lunch hour day, it didn't.

The menu boards hadn't really failed. It was the architecture's assumptions had.

Assumptions are like infrastructure. When they're working, they're almost invisible. We only discover them the day they fail.

Are we thinking about AI maintenance in entirely the wrong way.

Maybe the interesting question isn't whether AI systems decay differently from traditional software. The interesting question is what it is we're actually maintaining.

The McDonald's example stayed with me because it wasn't really about cloud computing.

It was about assumptions.

The software continued behaving exactly as it had been designed to behave. What changed was one of the assumptions that made that behaviour useful.

That made me wonder whether we've been slightly misleading ourselves when we talk about software maintenance.

We often describe maintenance as patching operating systems, upgrading libraries, replacing hardware, or fixing vulnerabilities. Those activities are certainly important, but they all seem to serve a larger purpose.

They preserve confidence.

When we install a security update, we're not really maintaining the patch. We're maintaining confidence that the software can continue to be trusted.

When we perform regression testing, we're not proving the software is perfect. We're increasing confidence that recent changes haven't introduced unexpected behaviour.

When we build redundant systems, we aren't preserving servers. We're preserving confidence that the service remains available when something inevitably fails.

The software is the artefact.

Confidence is the asset.

The software and systems for AI will decay in a whole different way. 

If today you work with traditional software. The questions to ask to maintain confidence in the implementation and in defense against cyber-attacks are these:

• Does the software still behave as specified?

• Does it still execute correctly?

• Does it still meet its availability, performance and security requirements?

If you’re not working with AI systems those questions haven't disappeared. However, if you are they’ve simply become insufficient, because AI systems introduce another question, which is:

Are the assumptions that initially justified trusting AI within, do they still resemble the world in which it is making decisions today.

That's a very different maintenance problem. A machine learning model may execute flawlessly, and the infrastructure may remain resilient. Every dashboard may stay green, yet confidence can still erode because the relationship between the model and reality has changed.

In traditional software we ask, has the software become less reliable, but with AI those assumptions have become less representative. The maintenance problem that AI is introducing is that do we have confidence that the assumptions embedded within those models still deserve our trust.

Let’s land this thing

The software is the artefact.

Confidence is the asset.

For decades we've become remarkably good at maintaining confidence in software.

We patch operating systems, when we can we replace hardware.

We upgrade libraries. And monitor performance.

We do our best to write sensible disaster recovery plans.

These tasks exist because they help us answer a simple timeless question. Can this system still be trusted?

AI doesn't replace those questions. It adds another.

Do the assumptions that justified trusting it still resemble reality?

That's the subtle difference.

….. because ???

AI increasingly asks us to defend systems against changes in the world itself.

Reality has always been moving, and now that is what AI governance really is.

Steal this line.

We haven't stopped maintaining software. We've started maintaining the relationship between software and reality.

Today’s cyber security is a discipline built around continuously asking whether the assumptions embedded within our systems still deserve to be trusted.