Clever patching keeps the system serviceable

It was the kind of day most systems administrators would like to forget. A customer of Canadian security consultant David Lewis, founder of the Liquidmatrix Security Digest, had decided to roll out a software patch to a Symantec product.

Unfortunately, the firm didn’t check the patch as well as it could have and the tweak disabled its firewalls.

Patch management looks easy but can cause nightmares if not handled properly, says Lewis, who warns that companies should never rely completely on automation.

“You will always need a human element,” he says.

The patch management challenge intensifies as the number of applications in an enterprise grows. Microsoft’s update service does a good job of looking after its own applications, but takes you only so far.

Third-party applications are harder to pinpoint and manage, and they represent roughly two-thirds of the problem. In 2010, 69 per cent of the sources of vulnerabilities on endpoints were found to have originated with third-party programs.

In 2006, patching Microsoft applications and the operating system on the average endpoint would have eliminated 55 per cent of vulnerabilities. In 2010, it got rid of just 31 per cent.

Take Adobe, for example. The company has suffered from several serious vulnerability exploits over the years, one of which appeared in September. A zero-day in the Flash player makes it possible for attackers to take control of a machine and the firm admitted that it was being exploited in the wild.

Adobe’s PDF reader has also had critical vulnerabilities, and fleeing to alternatives such as FoxIT’s PDF Reader doesn’t help. It, too, has suffered from vulnerability issues.

Fast work

In addition to patches that break systems in weird ways, time management can also be an issue. In many companies, the window available to take down systems for planned maintenance is shrinking, so patches must be rolled out faster.

However, Kamel Patel, a UK practice manager at giant IT services company Dimension Data, claims the last time he had to install a patch on a machine that needed a mandatory reboot was a while back. The move to the cloud, he argues, has made patch management easier.

“Some of the issues when you installed a patch and it overrode another file are reduced,” he says.

Not everyone buys the Utopian idea of patch-free IT departments. “So, why did Google and Adobe get nailed using IE 6?" asks Lewis.

Both companies were compromised during 2009 by zero-day attacks that exploited Internet Explorer 6 in an onslaught known as Operation Aurora. These companies were running a browser a couple of generations older than the one currently available.

“Why?” asks Emerson Tan, founder of PacketStorm, an online community that collects vulnerabilities and exploits. “Because nobody has bothered to fix their corporate intranets. Upgrading to something with most of the flaws fixed will simply break their internal apps."

Enveloping cloud

Brian Bourne, founder of Sector, a security conference taking place in Toronto in October, is equally sceptical that cloud-based apps escape patch management issues.

“You have less control because you have to move forward when they say so," he says.

Cloud-based application vendors update their software regularly without customer input. As an enterprise user, you may be able to stay on an earlier revision for a while by negotiating with the vendor, but that won’t last forever.

“You might have written something that interfaces with its application. Or there may be some feature it removed or altered that you were dependent on but which it figured no customers were using," says Bourne.

Other challenges include the consumerisation of IT, which encourages employees and contractors to bring in devices such as tablets and smartphones.

Making sure these are adequately patched creates a whole new set of problems, landing us in the sticky area of network access control, network quarantine and policy servers to manage the whole tangled mess.

Smaller businesses have an easier time, according to Patel. “It's pretty straightforward," he says. “Just accept everything from Windows Update."

For many small companies, this will be adequate. But every so often, a patch appears that takes down a piece of software. For example, Microsoft's recent gaffe, in which it accidentally decided that Google Chrome was a piece of malware, caused problems for many users.

For many companies the cost of setting up a proper test bed may be prohibitive

Ideally, customers will test everything before deploying a patch. But for many companies the cost of setting up a proper test bed and maintaining a configuration management database may be prohibitive, if not from a capital expenditure perspective, then simply because they don't have the internal nous to get the job done.

Examination fatigue

Many companies are settling for a compromise, Patel suggests. Rather than testing a patch to death with a variety of different configurations they give it quick once-over.

“You might try it out on test machines and if after a week users aren’t experiencing problems, you release it to the whole estate," he says.

Some companies may simply wait for two weeks to see if any adverse reactions to new patches turn up elsewhere, and if not, they deploy. It all depends on the level of risk that the company is comfortable with.

Ultimately, any patching strategy involves at least some human interaction, but the key lies in minimising fuss by adopting a mature approach to IT.

For example, any change management process can be made simpler by adopting just one or two images for corporate desktops, rather than juggling many desktop builds. Reporting software can also illustrate the effects of changes and help ensure that a deployment has succeeded, with minimal impact on the infrastructure.

Maintaining the reliability of your systems involves attention to detail and a refined approach to change management. Do you have what it takes? ®