How secure is your shell? At many enterprises, not very

In 1995, young Finnish computer scientist Tatu Ylonen invented the SSH data in-transit security protocol, or "Secure Shell" for short. His goal was to make it more secure for authorized users to access the information stored on a server.

If you've been using a computer sometime over the last decade and a half, chances are high that you've stumbled across the protocol somehow. It's a favorite of large companies.

The enterprise (particularly the IT organization) is vastly different from what it was in 1995, yet Ylonen continues to work on the same key issue: secure access to sensitive information. (His company, SSH Communications Security, just released a new SSH key identification and removal service.) It turns out that it matters more than ever in an always-on world.

We spoke to him about what companies are doing to keep their data secure -- and how they're failing.

ZDNet: What are you working on? What problem are you trying to solve?

TY: The problem we are trying to solve is these keys that grant access to servers in the IT infrastructure that haven't really been managed properly. We've implemented services that mitigate these risks; we just last week closed a $2.5 million deal with a major bank.

What we're basically providing them is expertise on how to get SSH keys under control -- knowledge on back doors they have into the IT environment -- and then help them eliminate all those keys. There are thousands of such keys today in the environment. It's to make sure that traders or anybody else can't get unauthorized access to systems.

We want to start changing keys automatically so that if there are any copies or an employee leaves the organization, access is terminated.

We've been digging into the SSH key management issue for the last couple of years. We were getting more requests around it from customers. We have one of the top three retail chains [as a customer] and began digging through the problem with them. None of these major customers actually knew how many SSH keys they had in the environment. They didn't know who could access what. They had no control.

ZD: Which is a crazy thing to say, really.

TY: This problem has been hidden inside IT. These keys are used typically for automating administrative responsibilities. These companies have something like 20,000 servers in their environments. There's nothing wrong with [corporate] protocol, it's just that the management of these controls has been sloppy.

One bank estimates they have 400,000 access keys that they have been accumulating over the last 10 years. They don't know which ones to remove because they don't know which keys allow what. They don't want to close them down because they don't want to shut the door on access to a critical system.

At one global bank, we found over a million keys that grant access to the server. Ten percent gave access to the root. The big risk here is that...imagine that these keys connect different servers to each other in a kind of mesh. If you break into one server, there's a substantial risk that you'd be able to get into other servers -- you could continue to attack all of the other servers. You could potentially knock down a very substantial part of an organizations infrastructure in a matter of minutes. It seems silly, but an extremely similar attack happened in 1988.

ZDNet: How proactive, or reactive, are enterprises on this issue? Do they wait for a crisis before they're compelled to address these things?

TY: An IT administrator only sees a small corner of the infrastructure. They only see a part of the problem; no knowledge of the size of the problem or how many keys they have. And they're so extremely busy, especially with the downsizing going on over the last few years. They just don't have the bandwidth. Even auditors haven't woken up. Some have.

There are lots of administrators involved in creating these keys. They can read and copy private keys stored on that account. Some keys are short -- you could write them on the back of your hand.

There's a cost for creating keys. At one business, 200 administrators were spending 10 percent of their time doing this. Another company had a 15-person dedicated team set up for security administration. If you estimate what's the cost of doing that kind of work per year, it amounts to something near the $2 to $3 million range for these companies. That's easily paid back through software.

It helps people justify doing something.

The problem exists in every Fortune 500 company. Every one of them has Linux and Unix servers. It's also mainframes. There are routers that use SSH underneath for management. This applies to the largest cloud services providers. More than half of the world's websites run on Apache, which runs on Linux or BSD, typically, and those machines are typically managed using SSH.

ZDNet: Are they listening?

TY: If we talk to the right people, they almost always recognize that they probably have the problem. They almost always never realize the scope of the problem. There's a lot of educating that has to be done.

I believe that most public companies in the U.S. are out of compliance with Sarbanes-Oxley [The U.S. law that tightened regulation on the accuracy of corporate finances. --Ed.]  because they're not properly controlling who accesses what, and they're not properly terminating access when an employee leaves, and they're not properly auditing those keys.

We have customer cases starting with 20 servers. There's a substantial service component in the process. We need to engage other consulting companies and educate internal IT departments so that they would be able to do this.

The problem is so large that it goes into how auditing guidelines are implemented. Auditors don't know that SSH keys exist.

It's a challenging problem. For us, it helps that we have so many large customers using SSH commercially -- more than half of the 10 largest U.S. banks and seven of the global 10 companies use our commercial SSH clients and servers. We can talk to them about what the real issues are.

ZDNet: In my recent experience, the attitude toward cybersecurity has veered toward, "It's not a matter of if you're hacked, but when." Do you find that corporate executives are thinking differently about security?

TY: The boundaries of organizations have become so complex. Multiple sites. People working from home using mobile devices. Bringing their personal laptops to the office.

You won't be able to control everything. You must have multiple lines of defense. You can't allow another server to be compromised because one person broke into a server. IT infrastructure is becoming an amoeba tied into all sorts of partners. The protocols of that have become so complex. Javascript, Flash, even PDFs have become extremely complex documents. Even in-house, the protocols...it's pretty complex semantics. The more complexity you have, the more potential exposure you have. The more code, the bigger the server, the more holes.

The approach we've developed with customers is discovering the keys in the environment -- itself pretty tricky to do properly -- and tracking the environment -- that is, which keys can be used -- so that we'll know which keys are never removed. When we want to remove the keys to the root -- ordinary users can't create keys for other users, which especially applies to automated processes -- we automate the management of keys, eliminating 15 or 20 man-years in the organization, giving them very substantial cost savings. Then we start rotating the keys so that every key gets changed every one or three months.

You get cost savings. You reduce the number of admins. You get operational efficiency savings. The goal is to minimize the risk.