Mitigating the Threats of Rogue Machines—802.1X or IPsec?

Published: August 9, 2005

Security Management

By Steve Riley
Sr. Program Manager
Security Business and Technology Unit

See other Security Management columns.

(Note: I've updated the original article published in August. After receiving numerous comments from readers, I wanted to clarify my true thinking of 802.1X and also provide more information about the differences between the capabilities of 802.1X and IPsec.)

Rogue computers. Aren’t these some of the scariest things to infest your network? You go to great lengths to build a protected network, you keep your clients current with updates and anti-malware signatures…yet you still suspect that these non-conforming machines are causing problems in your network. Forthcoming technologies like Network Access Protection (NAP) promise to help control this soon, but what can you do right now to mitigate—at least somewhat—the risks posed by rogue machines?

Using 802.1X to control who can access a network is an increasingly popular solution. 802.1X ( is a port-based access control method defined by the Institute of Electrical and Electronic Engineers (IEEE) that can be configured to require mutual authentication between the client and the network. If there's no authentication, no communications are permitted. 802.1X works with Extensible Authentication Protocol (EAP, to authenticate the client to the network and the network to the client, ensuring that both sides are communicating with recognized entities.

As a technology to control admission of authenticated computers into a network, 802.1X works very well. Unless your computer is authorized to communicate on the network, it simply can’t. Your computer must possess the correct authorization—and if you do, then you are allowed to communicate on the network. This was the design goal of 802.1X, and the goal was met. However, it’s insufficient for proper, complete protection from rogue machines because it lacks a full end-to-end authentication and authorization mechanism. There’s a shortcoming in wired 802.1X that reduces its effectiveness at keeping out rogue machines. I’ll describe how an attack can take advantage of a weakness in the way the protocol operates.

How 802.1X operates

802.1X is designed to work over just about any kind of network: wired, wireless, barbed wire, even bongo drum ( 802.1X requires a supporting infrastructure, namely clients, LAN switches, and wireless access points that can participate in 802.1X, a RADIUS server, and some sort of account database (such as Active Directory).

A client, called a supplicant, makes an initial connection to an authenticator, the LAN switch or a wireless access point. The authenticator is configured to require 802.1X from all supplicants and will ignore any incoming connection that doesn't conform. The authenticator asks the supplicant for its identity, which it will forward to the authentication server (RADIUS).

RADIUS follows whatever mechanism it needs to authenticate the incoming client. Generally this involves setting up an EAP conversation between the supplicant and the authentication server (the authenticator is just a pass-through device here) and establishing an authentication method inside the EAP conversation. Note that EAP itself doesn’t define any kind of security on its own—the authentication protocols used must incorporate their own security. Out of the box, Windows supports three EAP methods:

  • EAP-MD5. The authentication server sends a challenge to the supplicant. The supplicant combines the challenge with its identity and its password, creates an MD5 hash of all that, and sends this back to the authentication server. The authentication server decrypts what it received and if that matches the original challenge, authentication succeeds. Note that this isn't secure, since it sends actual passwords; EAP-MD5 is disallowed by Windows for 802.1X wireless connections.

  • EAP-TLS. The authentication server sets up a transport layer security (TLS) session with the supplicant. The server sends its digital certificate to the supplicant, which the supplicant validates. The supplicant then sends its digital certificate to the authentication server, which the server validates. Thus the client and the network have mutually authenticated each other, and so long as each side trusts the other's certificate and the certificate is valid, authentication succeeds.

  • Protected EAP (PEAP). PEAP begins like EAP—the authentication server sets up a TLS session with the supplicant and sends its digital certificate to the supplicant for validation. If the supplicant trusts the certificate, it uses one of several methods to authenticate itself to the server. Right now the only supplicant-side authentication method available in Windows is MS-CHAPv2, in which the supplicant uses traditional accounts (user or computer IDs and passwords) to authenticate. This is called PEAP-EAP-MS-CHAPv2. Note that PEAP-EAP-TLS is also a configurable option, although there is really no reason to choose this. It establishes a completely separate second TLS session inside the first; this doubling of TLS sessions is slower than pure EAP-TLS.

Once RADIUS has authenticated the supplicant, the supplicant is allowed to communicate on the network behind the authenticator (remember, this is the LAN switch or the wireless access point). Although an authenticator has one physical network port, think of it as having two virtual ports. Traffic from authenticated supplicants passes over the controlled port, which blocks traffic from unauthenticated supplicants. During the authentication process the authenticator must communicate with the RADIUS server, which occurs over the uncontrolled port. After a supplicant authenticates, the controlled port transitions into a connected state for that supplicant.

Using 802.1X for wired security

For 802.1X to be effective in wired networks, the infrastructure needs to be fairly up-to-date. All of the switches in your network—or at least those that clients and servers connect to—must support 802.1X. You also need a PKI if you don’t have one: PEAP requires a digital certificate for the RADIUS server; EAP-TLS requires digital certificates for all computers. (You don’t need a PKI if you use EAP-MD5, but I discourage that.) That could be a pretty expensive proposition if you use certificates from public authorities, so save yourself some money and build a Windows enterprise certificate authority. All of your domain-joined computers will trust this CA and will therefore trust the certificates it issues.

All of your clients need to have an 802.1X-capable IP stack, such as the stack built into Windows XP. If you can’t upgrade to Windows XP yet, there is an option: Microsoft has released an 802.1X stack for Windows 2000 ( However, in both Windows XP and Windows 2000 wired 802.1X networks, there’s a known problem that can occur if all of these conditions are true:

  • You've configured 802.1X only for machine (not machine and user) authentication

  • You're using PEAP (not EAP-TLS)

  • Your computer's machine account password has expired

The result? The computer won’t be able to log onto the domain. (It’s a DLL issue that we can’t really fix now without re-engineering the stack. It will be fixed in Windows Vista and Windows Server “Longhorn.”) If this describes your environment, then consider longer expiration times on machine passwords. This really isn’t all that bad since they’re randomly generated and very strong. If you’re using EAP-TLS, the problem doesn’t exist because authentication is handled through a different set of DLLs. If you’re using PEAP with machine and computer authentication, then the machine password will get reset when user authentication succeeds.

What about network devices that can’t participate in 802.1X, like printers, network storage devices, or that old DOS PC running some ancient, creaky, totally unsupportable yet mission-critical application? Remember, your reason for implementing 802.1X in the first place was to make sure that only authorized devices can communicate. Now you need to create an exception. Before you do, though, does your security policy allow that? Check first. You’ll also need to create exceptions for bootstrapping new systems (perhaps on a physically-isolated segment that’s exempt from 802.1X). Note that requiring 802.1X will eliminate your ability to use PXE boot in your network.

Why 802.1X on wired networks is insufficient

802.1X is an ideal foundation for wireless security, which we’ve explored many times in the past at conferences and in documentation on Microsoft’s Web site ( But deploying 802.1X alone, as a method of securing wired networks against rogue machines, has some significant drawbacks. Working with non-participating devices, as we discussed above, is one. Lack of manageability is another: in AD group policy, several group policy objects exist for you to manage 802.1X on wireless networks. These GPOs don’t exist for wired interfaces, and there are no published APIs for managing wired 802.1X client computers. Some architectural reasons prevent adding GPOs to Windows 2000 and Windows XP for wired 802.1X. Because of this lack of centralized management capability, large-scale 802.1X deployment on Windows isn’t feasible currently and will be expensive to maintain.

Finally, there’s a major weakness in the protocol: it authenticates only at the establishment of a connection (in some implementations, there is a periodic reauthentication, but this doesn’t mitigate the attack I describe below). Once a supplicant authenticates and the switch port opens, further communications between the supplicant and the switch aren’t authenticated. This creates a situation in which it’s possible for an attacker to join the network. (Thanks to Svyatoslav Pidgorny, Microsoft MVP for security, for showing me this vulnerability.)

Setting up the attack does require physical access to the network, so in some respects this attack is a bit esoteric. An attacker needs to disconnect a computer (let’s call this the “victim”) from its 802.1X-protected network switch port, connect a hub to the port, connect the victim to the hub, and connect an attack computer (which we’ll call the “shadow”) to the hub. This is trivially easy if the attacker is physically inside your facility and if your Ethernet jacks are accessible. Or the attacker could connect an unmanaged access point to the hub and then conduct the attack from your parking lot. (Of course, the attacker will probably try to hide by disabling this AP’s SSID broadcast.)

The brief disconnection of the victim from the network won’t interfere with the attack’s success. When the victim computer is reconnected, it authenticates to the switch again. It doesn’t matter that a hub is in the way now, because a hub is little more than a wire with ports in it. Electrically, the victim is still connected to the switch. Hubs are invisible to 802.1X.

Next the attacker configures the shadow computer’s MAC and IP addresses to be the same as those on the victim computer. A little network sniffing will quickly reveal this. The attacker also configures a host firewall to drop all inbound traffic that isn’t a reply to communications that it initiated.

Now, here’s how the attack works. After the victim computer has authenticated and the switch port is open, the attacker can connect to resources on the protected network. This is because there is no per-packet authentication of the traffic once the port is open. Since the shadow computer has the same MAC and IP addresses as the victim computer, from the point of view of the switch it appears only as if there’s a single computer connected to the port. 802.1X’s lack of follow-on per-packet authentication creates the situation for this man-in-the-middle attack I’ve just described. 802.1X only authenticates the connection; it assumes all traffic that’s flowing over the connection is legitimate.

Note that the attacker can communicate only over stateless protocols like ICMP or UDP. So the attacker could ping computers on the network and receive a DHCP lease (the same IP address as the victim). But the attacker can’t communicate over TCP to the network—the victim computer will reset any connections initiated by the shadow host. Here’s the sequence:

  1. The shadow computer sends a SYN packet to a server on the protected network.

  2. The server returns an ACK-SYN, which both the shadow and the victim receive.

  3. The victim computer isn't expecting this ACK-SYN so it returns a RST to the server. Meanwhile, the shadow returns an ACK to the server.

  4. If the victim's RST arrives at the server first, the server shuts down the connection and ignores the shadow's ACK. If the shadow's ACK arrives at the server first, the connection completes but the victim's subsequent RST will terminate it anyway.

There’s one very interesting exception to the rule. If the victim computer is running a firewall that drops unsolicited inbound ACK-SYNs, which most do, the victim won’t process the received ACK-SYN in step 2 and therefore won’t send the RST to the server. The rest of the above sequence won’t happen and the shadow computer can have complete access to the protected network. This is the only instance I know of where a personal firewall on a computer can reduce the security of the rest of the network! Of course this is no reason not to deploy personal firewalls; their benefits strongly outweigh the likelihood of this attack actually happening.

So what should you do?

First, understand the scope of the attack. Wireless networks aren’t vulnerable because 802.1X+EAP creates mutually-authenticated sessions with per-supplicant encryption keys (this is called “dynamic WEP”). Because the keys are never sent over the air, the shadow computer attack won’t work—it has no way of acquiring the key. The shadow is unable to connect to the access point where the victim computer is already connected. So, in a sense, wireless networks with 802.1X+EAP are actually more secure than wired ones.

But for wired networks consider what it is that you’re trying to protect. If the goal is to prevent rogue machines from infecting other machines on the network, then an end-to-end protection mechanism is necessary. 802.1X alone doesn’t give you this. It will stop an attacker from conducting bandwidth denial-of-service attacks, which might be important to you, although generally it’s computers, not networks, that are the target of DoS attacks. But the unstated premise of 802.1X is that all potential computers allowed to connect to the network have the same trust level. While this might be true in some organizations, generally there are varying levels of computer trust.

So to address the stated threat, I strongly encourage IPsec. IPsec stops an attacker from getting to other IPsec-protected resources and allows you to define “zones” of trust. It’s the protocol’s connection and packet authentication that make it better suited to addressing the threat. If Alice’s computer is unable to create the proper IPsec security association with Bob’s server, then Alice can try all she likes but she won’t be able to compromise Bob. It could be that Alice’s computer is completely untrusted, or that her computer is in a different trust zone than the server. And this works even if Alice’s computer is mobile and not directly connected to the network, since IPsec controls can move with the host.

IPsec isn’t a silver bullet, of course. It does require hosts that are capable of performing IPsec. This shouldn’t be a problem for most organizations, though, since IPsec has been in Windows for some time now and is also supported by many other popular operating systems. (Windows IPsec interoperates with stacks that perform IKE negotiation, not manual keying.) IPsec won’t stop an attacker from getting an IP address or from communicating over your LAN—but you could, of course, just disable the switch port of someone trying to DoS your network. Or you might consider deploying both 802.1X and IPsec if you want to address a range of threats; some of our customers have begun doing this now.

Is it an IPsec vs. 802.1X battle?

It shouldn’t be. When you think of 802.1X as a way of “preventing entry” and IPsec as a way of “preventing access,” it becomes evident that they attempt to solve the rogue machine problems in different ways. You have to analyze your threats, consider your risk stance, and understand the differences in the technologies:



Authenticates connections and packets between computers

Authenticates connections to the network

Protects computers

Protects networks

Allows defining zones of varying trust levels (directions and types of traffic)

Treats all computers as having the same trust level

Supports movement between networks

Applies only to computers connected to the network

Can’t prevent network communications

Stops network bandwidth denial of service attacks

Stops rogue computers from doing anything useful

Can’t prevent attacks from malicious rogue machines

Won’t protect against “trusted” attackers

Won’t protect against “trusted” attackers

However, you can’t rely only on 802.1X to stop rogue computers from damaging other computers on your network. The rise of mobile computers and wireless networking has increased the need for having each computer become responsible for protecting itself from other computers in the network. Host protection is now no longer an option, but it can be complemented with network protection. IPsec is the minimum requirement for addressing the threat; 802.1X can add protection against additional threats.

Learn more

At Microsoft we reassessed the risks against our information assets. From this we identified two overarching security requirements on our network:

  • Protect high-value intellectual property and meet compliance requirements

  • Protect managed hosts from attacks and unauthorized access

To satisfy these requirements we’ve used IPsec to implement a notion we call domain isolation on our corporate network. It works well for us and I suggest you try it, too. It gives you the ability to require domain membership of all your computers and even to limit access to some resources to specific groups of especially trusted computers. Learn more about it here:

Search for “domain isolation” to find even more information. The principles we describe in the documents on domain isolation are the foundation for the upcoming network access protection (NAP) technologies in Windows Server “Longhorn.” NAP defines four levels of quarantine: DHCP, VPN, 802.1X, and IPsec. It’s important to start planning for it now. Implementing IPsec-based domain isolation in your network today will put you far down the path for even better client isolation and health-checking in the future.