> we have to get updated database software from a vendor, and to install it we have to update the API the billing software uses
And so why haven't you updated the API the billing software uses, long before now? Why haven't you updated the database software before now?
CIOs create risk when they don't prioritize keeping their products up to date. When you can't even install security updates without breaking your installations, you have a problem. And your problem is more than some technical problem, it's a cultural problem.
Yes it's risky to update a large number of machines. But as a CIO, it is your job to mitigate that risk. There's no such real thing as "unexpected security fixes" in this day and age. They are entirely expected, and if you cannot deal with predictable occurrences then you are quite simply incompetent.
I think the reason that the billing software's API hasn't been updated is right there in it's name: "billing". Billing is one of those things in an organization that can't suffer much downtime. Nobody wants to be that guy who who endangered revenue because there was no possible way that the security patch should have been able to break the billing software.
Does billing work? Yes? Then don't mess with it. Billing is only to be messed with when the cost of not messing with it is that it will never work again and all prior data will be lost forever. Nothing short of that risk is sufficient justification to touch anything related to billing.
I'm being hyperbolic there, but conservatism around systems that presently work shouldn't be terribly surprising, something like billing especially.
> Does [a service with high uptime requirements] work? Yes? Then don't mess with it
Ah yes, the "don't fix what ain't broken" canard. And it would be completely understandable if we didn't understand that, in general, the best way to ensure overall uptime is to encourage small, frequent updates over large, infrequent updates, because change is inevitable and the risk of the update is proportional to its size.
I understand that you're being hyperbolic, but that kind of conservatism is born of ignorance. Expecting CIOs to be educated about mitigating risk in the systems they are in control of is not a high expectation for someone in a CIO role.
A lot of this is a social issue too - it's IT professionals over-committing with SLAs and being too passive when it comes to discussing terms to set realistic RPOs for fragile systems when the resources aren't available for proper patch testing.
It's very difficult to explain to end-operators of systems the importance of having things like redundancies, test systems, and the ability to have downtime for patching, but it's something that IT Professionals need to be way better about. It's very tempting to throw out goals like 99.9% uptime, but many operations run with an employee bandwidth that in no way can support such a goal for the number of systems they need to deal with.
To be fair, sometimes the end-operator needs require some absolutely antiquated pieces of technology that rely on voodoo like rituals to keep the systems running, and trying to shift organizations off this technology is the diplomatic equivalent of a land war in Russian during winter, and IT administrators [1]want to avoid getting into such a battle.
Hopefully, this Ransomware outbreak will help provide disruption to such pieces of technology that are stuck in the past, but part of it is going to require that the new technology makers be willing to respect why so many organizations hang on to older technology. (This tends to revolve around pricing models) I think there is going to be a lot of opportunity to review major systems that have big restrictions on legacy software and hardware and overtake the incumbents that aren't willing to shore up their products.
[1] Edit: removed too many mixed metaphors from one sentence O.O
Care to elaborate? Do you mean that newer software comes with mandatory maintenance costs that users are unable to unwilling to bear? In that case, paying for security patches and maintenance should be palatable to customers in this context, shouldn't it? Or did you mean something else?
All of that thinking is just a bet, against the future possibility of an essential need to patch/upgrade for security reasons.
They are reaping the short-term benefit of guaranteed uptime, in exchange for being able to do anything easily when you really have to in future.
Because catastrophic events are rare, many companies think they're just being really clever and there are no such consequences.
And so why haven't you updated the API the billing software uses, long before now? Why haven't you updated the database software before now?
CIOs create risk when they don't prioritize keeping their products up to date. When you can't even install security updates without breaking your installations, you have a problem. And your problem is more than some technical problem, it's a cultural problem.
Yes it's risky to update a large number of machines. But as a CIO, it is your job to mitigate that risk. There's no such real thing as "unexpected security fixes" in this day and age. They are entirely expected, and if you cannot deal with predictable occurrences then you are quite simply incompetent.