Tag Archives: Best Practices

When VMs Get Pwned: Real Security in Virtual Environments

This week I wrote an article for a few of our partners, include the well-known SpiceWorks IT pro forum. If you’re interesting in learning about the additional risks introduced by hardware vritualization and in securing your virtual environment, read on.

Spinning up a virtual machine (VM) without a good security policy, a hardened hypervisor, and virtual security controls is like diving into a public pool with your mouth wide open—you may not contract a disease immediately, but eventually you’ll have a very bad day.

I’m not sure why some administrators forget to harden their virtual environment. Perhaps they imagine it like the virtual realm of “World of Warcraft“; if a VM gets pwned they can just return, ghost-like, and resurrect it. Personally, I think it’s more like the “Tron” universe; if data on your VM gets popped, it disappears “IRL” too. This article discusses the additional security risks hardware virtualization introduces, and shares a few quick virtual security best practices to mitigate those risks.

A brief history of virtualization
The concept of virtualization has existed since the inception of computing. Back in the mainframe days, and again when personal computers emerged, we virtualized our computers’ input/output (I/O) system via the operating system (OS). The OS (also called the supervisor) abstracts I/O functions so individual applications don’t have to “re-invent the wheel” and figure out how to handle them.

Next, we wanted to run many programs at once, so we virtualized memory. Multi-tasking OSs like Windows present virtual memory pages to each application. Every application thinks it has all your computer’s memory resources, but your OS secretly virtualizes it.

Finally, we come to virtualization as we know it today; specifically, hardware virtualization. Hardware virtualization allows us to abstract our computer’s hardware and run multiple OSs on one physical machine. In order to do this, hardware virtualization requires an additional authority to manage everything, called the hypervisor.

Hardware virtualization presents many benefits, most of which you’re probably aware of. It saves us time, money, space, HVAC costs, and may even help us go green. So why hasn’t everyone virtualized everything? While there is no simple answer to that question, security is one major inhibitor.

Virtual environments increase real-world risks
Hardware virtualization introduces new security risks and concerns. I’ve heard people generalize this risk as, “the same security issues as physical computers, just on virtual ones.” There’s some truth to that, but hardware virtualization also adds additional layers of concern. Let me share a few of those issues:

  • Hypervisor security issues – I mentioned the hypervisor — the higher-level authority that abstracts your computer’s hardware, and manages multiple VMs. A hypervisor is just software, albeit complex software. Like all software, it can suffer from development bugs (buffer overflows and the like), which may result in security vulnerabilities. The hypervisor is a high-privileged system, so vulnerabilities in its software pose dire consequences. Hypervisors also introduce new management interfaces and services to your infrastructure, both of which present new targets for attackers. In short, hardware virtualization increases the potential attack surface of your network.
  • Multi-tenant security issues – Running multiple computers on one physical platform introduces new problems. Information security has the concept of security domains. You segment resources by what they do, and how much you trust them. In the physical world, servers typically do one thing, so separating them is easy. In the virtual world, you can place many servers on one machine, making it more difficult to separate them by domain. Furthermore, VMs can communicate instantly with one another, sans a physical network. This leads to the “bad neighbor” syndrome, where one infected VM can quickly infect others on the same hardware, if configured improperly. Multi-tenancy also introduces data remnant risks, where one VM might access the data left in memory by another VM.
  • Virtual network blind spot – I mentioned VMs can communicate with each other over a virtual network, never hitting a physical network. Though we have many great physical network visibility and security tools, they can’t see anything happening on virtual networks. Unless you implement virtual security controls, you won’t see the attacks until it’s too late.
  • It’s relatively new and complex – While virtualization (even hardware virtualization) has been around for decades, many small business administrators only dove into it recently. Some of us are just starting to wrap our heads around virtual networking and switching, let alone learning how to harden hypervisors and implement virtual security controls. Additionally, some virtualization benefits complicate security. Its increased flexibility and mobility means you can copy and paste a VM instantly, but also allows you to quickly replicate, multiple, and further expose security mis-configurations. In short, virtual servers often move quicker than security policies can keep up.

Those are just a few unique virtualization security concerns. VMs also suffer from “normal” computer security risks too – only these risks are further complicated by the liquid nature of virtual environments.

Virtual attacks: From theory to reality
You might ask, “Well that sounds scary in theory, but are attackers really targeting virtualization?” In a word, yes.

For instance, security researchers have demonstrated virtualization rootkits (Blue Pill), attackers have designed malware that avoids researcher VMs, and criminals have leveraged VMs as Command and Control (C&C) channels for botnets. Most recently, the Crisis malware included a spreading mechanism that searched for and infected virtual images, making it the first wild malware that specifically targets VMs.

Security best practices for a virtual world
At this point, you might be balking at the idea of virtualizing anything, but don’t worry. Virtualization’s benefits far outweigh its risks, and there are defenses for all these dangers. Here are a few virtualization best practices to get you started:

  • Favor bare metal hypervisors over hosted ones – Without going into full detail, there are two types of hypervisors: those that run directly on hardware (Type 1), and those that run on top of an OS (Type 2). I recommend using a Type1 (or bare metal) hypervisor like VMware ESXi; the benefit being that it limits the scope of hypervisor attacks. For instance, an elevation of privilege attack against a Type 2 hypervisor may give an attacker full control of your native OS, whereas there is no OS to access on a Type 1 hypervisor.
  • Segment your hypervisor’s management network – Hypervisors have management interfaces which you can access over a network. Make sure to separate this management network from other virtual and physical networks you don’t trust, and limit access to management interfaces.
  • Disable unnecessary hypervisor services – Hypervisors provide many services to virtual machines, including access to USB devices, Bluetooth, disc drives, file services, physical network cards, and so on. They also provide ease-of-use features, like the ability to drag and drop files between virtual machines. Each of these services exposes more attack surface. If you don’t use it, disable it.
  • Patch your hypervisor regularly – It’s software; it has vulnerabilities; patch it. Hypervisors fall into the class of things people tend to deploy and forget. Don’t let that be the case for you.
  • Apply least privilege principles to hypervisors and guest VMs – You already know the least privilege principle, now extend it to your virtual environment. Who can manage the hypervisor, build new VMs, launch VMs, and what privileges do users on guest VMs have? These are all things you can control.
  • Standardize your VM images – VMs move fast. People clone, copy, and redeploy them at the speed of file transfer. Badly configured VMs can spread security mis-configurations quickly. On the same token, creating a well-hardened, properly configured VM to use as a base image ensures your newly deployed virtual servers will at least start with secure defaults.
  • Deploy virtual security controls – Today, virtual networks are a huge blind spot in our infrastructure. Though hypervisor vendors have started to provide some basic virtual security controls, they lack high-end security and visibility tools. Some third parties, like WatchGuard,  have started to provide these virtual security solutions. Consider using them.
  • Extend normal security controls to VMs – The tips above help you specifically harden your virtual environment, but virtual machines still act like normal computers. They need OS and software patches, as well as normal host-based security controls like antivirus (AV). Luckily, there are solutions which help you apply such controls across multiple virtual machines at once.

At the end of the day, all the information and network security practices we use to protect physical devices still apply to protecting the virtual ones. The only difference is we have to extend these practices across new surfaces, and introduce our familiar tools to new environments. Once you understand the new attack surface that hypervisors and virtual networks present, you can start to mitigate these risks using familiar defenses. It’s just a matter of diligently doing what you already know. — Corey Nachreiner, CISSP (@SecAdept)

Use the Zappos Breach to Renew Your Password Strategy

Last Sunday, Zappos (a popular, Amazon-owned, online shoe reseller) warned its employees and customers that an attacker had gained access to their internal network, and made off with a bunch of sensitive customer information. The good news? The attacker did not gain access to any customer credit card info. The bad news? He or she did steal over 24 million users’ names, addresses, phone numbers, email addresses, and encrypted or hashed passwords.

Zappos hasn’t released any technical details about the attack, and I don’t expect them to. If forced to guess, I’d assume it probably originated from some web application flaw, which is a pretty common vector these days. That’s why I often suggest that IT and web administrators focus their security resources on their web applications; both by encouraging secure web coding practices, and by leveraging security controls with application-layer inspection capabilities (such as the HTTP and HTTPS proxies that WatchGuard’s XTM appliances offer). However, that’s not what I’m here to talk about today. Today, I want to talk about passwords.

I’ve talked about passwords many times before, but as a core principle of security (technically part of Authentication), the advice bears repeating. Here are some password-related tips; both general and related to password security breaches:

  • Change your password(s) after a security breach – If a site you use ever has a security breach where attackers gain access to passwords (hashed or not), change your password immediately. In Zappos case, they are forcing this advice by terminating old passwords. If you use Zappos, be sure to change your password now, before a bad guy does it for you.
  • Use strong passwords – I believe passwords should be greater than 10 characters. One easy way you can create long passwords, with enough entropy, is by using passphrases, or more specifically something I call pass-sentences. WatchGuard’s Bud Logs In video talks about these concepts in more detail (and is good for basic endusers).
  • Use different passphrases on different web sites – This is crucial aspect of password security, especially when considering these types of web breaches. If you, like most people, use the same password for many different web sites, the attacker that has Zappos’ password archive now may have your password for all web sites. If you have been using the same password everywhere, not only should you change your Zappos password, but you should change your password on every site (and make it different this time). This breach situation is exactly why experts recommend you use different passwords everywhere. That said, many people find this advice hard to implement in practice; which brings me to the next tip…
  • Leverage password vault software – Password vaults make it easier for you to manage multiple passwords securely. They are not perfect. If you use multiple machines and OSs, you may have trouble finding password management software that meets all your needs. Plus, password vaults become a single point of potential failure, as they almost literally store all the keys to your kingdom. It’s extremely important to use secure password vaults, and protect them. That said, they offer the only practical solution to managing multiple passwords today. This article suggests a few good ones to use (I have used 1password myself).
None of this advice is ground -breaking.  I’ve mentioned it many times before, including during the HBGary hacking incident. However, some aspects of password security — particular the part about not reusing passwords — are admittedly hard for normal people to follow in the real world, because they can slow things down. I hope you use this Zappos breach to remind you of the benefits of following certain security best practices, even if they put small speed bumps in front of your typical business processes. Sometimes we need these speed bumps to prevent ourselves from crashing headlong into a brick wall.  — Corey Nachreiner, CISSP (@SecAdept)

The HBGary vs. Anonymous Saga: What can we learn?

If you follow security news, then you’ve surely heard about the recent drama between “Anonymous” and the HBGary security firm (more on who Anonymous is below), which took place over the past few weeks. While I’ve personally followed this fiasco with great interest, I’ve avoided commenting about it here, since most of our customers and readers are network administrators who are more concerned with practical business solutions than melodramatic cyber-quarrels. However, recently I read a fantastic article sharing the technical details of the HBGary breach, which I believe is a must-read for any computer security practitioner.

I’ll come back to that article in a minute, but first let me summarize the Anonymous/HBGary saga for those that may not have heard about it (if you have heard about it, feel free to skip to “Learning from Others’ Mistakes”).

I assume most of you are aware of the Wikileaks Cablegate story, since it’s made worldwide news. You know, that incident where Wikileaks — a non-profit organization that shares private or classified information with the press — publicly released some embarrassing U.S. diplomatic cables and royally peeved off the U.S. government. I don’t really want to recap the whole Wikileaks incident. I only bring it up to remind you that some camps oppose Wikileaks’ mission of “outing” sensitive information, while others camps fully endorse it.

Enter the mysterious Internet entity called “Anonymous.”

Who are Anonymous?

If you follow technology news, you’ve probably heard the “Anonymous” name in headlines before. They’re a group attributed for a wide-range of Internet incidents; from attacking Scientology to YouTube porn day (and many in between). However, in my talks with peers, I’ve found that many IT folks don’t really know who or what “Anonymous” is. Some may imagine “Anonymous” as a specific group of attackers, but that’s not really the case.  In a nutshell, Anonymous is a random group of users tied strongly with popular image forums, like 4chan.

Occasionally, this group of random anonymous users decides to take on some fight, and loosely organizes what is essentially a virtual flash mob. For example, someone might post that they dislike Scientology, and ask other users to start figuring out ways to mess with Scientology, online. From there, chaos ensues. This means, the “Anonymous” group is not a specific group of people; rather it is a random group of users that happens to rally behind one cause or another; in other words, “hacktivists.”

Over the last few weeks, Wikileaks was one of those causes. I won’t pretend to speak for Anonymous, but I think it’s pretty safe to say that most 4chan users are pro-Wikileaks. Once the U.S. government started going after Wikileaks and its founder, an “Anonymous” group formed to start “fighting back” in Operation Avenge Assange. Using some very basic attack tools (Low Orbit Ion Canon), Anonymous began launching fairly successful Distributed Denial of Service (DDoS) attacks against various high-profile targets like Visa and MasterCard.

What’s this have to do with HBGary?

That’s the background story, but you might now be wondering where HBGary comes into the situation. HBGary is a security company that provides various security services to customers. It also has an offshoot company called HBGary Federal, which provides those services to the government.

Early on in the Wikileaks battle, HBGary threw its gauntlet into the fight, going after Wikileaks donors. Furthermore, the COO of HBGary Federal, Aaron Barr, thought it would be interesting to infiltrate the Anonymous group and try to find who its leaders were (assuming it has any). He seems to have attempted this by lurking on forums and IRC. Eventually, Barr started sharing his findings with the press, and intended to present them at a security conference. This, of course, set Anonymous off. They had a new target and cause… take out HBGary.

And that they did! Not to mince words, but Anonymous pretty much decimated HBGary’s defenses one brick at a time. By leveraging some very basic security issues in a number of systems, Anonymous was able to deface HBGary’s web site, delete 1 TB of backups, and steal tens of thousands of critical and sensitive emails (including some very embarrassing ones). I’ll get into more detail on how Anonymous did this below, but suffice to say a company, especially one focused on security, couldn’t suffer a more embarrassing public breach. In fact, HBGary was so affected by this attack that they even pulled out of the RSA conference.

So now you’re up-to-date with HBGary internet soap-opera, but why should you care? Well, this incident leads to a fairly obvious question. How did a respected security firm get hacked so quickly and easily?

Learning from Others’ Mistakes

That’s brings us full circle, to the article I mentioned at the beginning of this post. Last week, Ars Technica published an article detailing exactly how Anonymous broke into HBGary’s network (which they learned by talking to those who participated in the attack). This real world incident is a perfect example of how seemingly small chinks in different parts of your defenses can add up to gaping holes that totally compromise your system. Furthermore, it illustrates how not following some of the most basic best practices could land you in a heap of trouble. If you haven’t read the article, I highly recommend you go do so now. I’ll wait…

Ok, you’re back?

As you read, HBGary surprisingly fell victim to some of the most basic security mistakes one could make. To accomplish all of the mayhem I mentioned earlier, Anonymous’ attack included the following components:

  • A SQL injection on a badly coded custom CMS
  • A cracking attack (using rainbow tables) on badly encrypted passwords
  • The discovery of some embarrassingly weak passwords used by high value targets
  • The discovery of rampant password reuse (again, by high value targets)
  • An elevation of privilege attack due to an very unpatched system
  • …and some basic social engineering

None of those attacks are new, nor particularly extraordinary or complex.  In fact, some are as old as hacking itself. All security professionals know basic security best practices to safeguard against them. That’s why this incident should wave a big red flag in the security community. How could such a well-respected security firm, who knows the right things to do, fall victim to such basic attacks? In his article, Peter Bright offers a potential answer to that question. He suggests that, “the standard advice isn’t good enough.”

I don’t think Bright means that the industry’s best practices are wrong; especially considering he also says standard advice would have protected HBGary. Rather, I believe Bright means that if our standard advice is too hard or time consuming for normal people to follow, they will ignore it. I agree with this sentiment. Few will follow technically sound best practices if they are impractical.

Let’s take the whole password reuse issue. Every security practitioner knows you should not reuse the same password at multiple sites. If you do reuse your password and an attacker gains access to it via one insecure site, then the attacker has the keys to your entire kingdom. Obviously, you should use different passwords everywhere, which is the industry best practice. However, following this best practice isn’t easy. At the very least, it takes extra time and thought. Most normal users don’t know about the password vault or keychain software that might help them manage multiple passwords, Even when they do, users don’t always use them because they adds extra steps, or roadblocks to their daily processes. As a result, many people reuse passwords.

This is the crux of the problem; the industry’s technically correct advice may not be “good enough” if normal people find it impractical. Security experts, in their white towers, often forget that security is not the core mission of most businesses. Many administrators consider security a necessary chore; something they have to do, but don’t really want to spend time on. The average user cares even less. No one like roadblocks that make doing their job harder, and users often see security controls as roadblocks.

Unfortunately, there is no easy answer to this dilemma. In order to secure things, you have to put access controls in front of them. However, I see Bright’s comment as a call to arms for the security industry. The best security mechanism in the world won’t do a thing if your users turn it off.  So we need to design our security controls with ease of use in mind, which is something WatchGuard is focused on. We need to protect networks, while still facilitating business.

My second takeaway seems obvious in its simplicity, yet many people don’t really do it. That is, “Do what you know.” Over the last few years, my cohorts and I have ended many of our security presentations sharing a statistic we learned from a study done by the Verizon RISK team. Over the years, the RISK team has researched real world security breaches to study why the breach happened, and how it could have been prevented. They found that in almost 90% of the cases, the victim organization had the proper policies and technologies to have prevented the breach; they just didn’t follow their own policies, or configure their technology properly. This is what happened with HBGary. They obviously know how to prevent the simple attacks that succeeded against them; they just didn’t.

I’m not pointing fingers at HBGary. As the Verizon RISK team found, it seems like most organizations don’t follow through with best security practice. However, if we want to avoid security incidents, this is something we need to improve. When I was a kid, I remember fondly watching the G.I. Joe cartoon that always ended by saying, “and knowing is half the battle.” We need to remember that doing what you know is the other, arguably more important, half of that battle.

Learn from HBGary, and do what you know.


Get every new post delivered to your Inbox.

Join 8,238 other followers

%d bloggers like this: