ESXi Security Lab

Late last year, I set about building a new virtualization platform to serve as my security lab. Since the advent of Microsoft Hyper-V and VMware ESXi, I had been eager to start using a pure hypervisor solution rather than just VMware server and workstation.

I started my research on VMware ESXi and soon had it running nested within VMware workstation so I could get a sense of how to install and configure the basics. Reading blogs and forums (HardOCP's virtualized computing forum was especially helpful), helped me assemble a list of parts that would make up my ESXi security lab. I had a few requirements, but the most important one was stability. I had read reports of problems with non-compatible hardware and I wanted to ensure that my time would be spent productively working on my virtual machines rather than fighting with ESXi.

After assembling the system hardware and testing basic functionality, it was time to install ESXi. This took no more than 15 minutes. It was incredibly simple. It involved booting from the ISO and pointing the installer to the USB drive for the OS, and then let setup take care of the rest. ESXi was up and running and I was logging into virtual center about 20 minutes after I had started the installation. I initially installed ESXi 4.x, but soon upgraded to ESXi 5. This process was equally straightforward.

The diagram below is a logical representation of the current state of my security lab. Its not 100% technically accurate but provides an overview of the architecture. I created a self-contained, deliberately insecure set of networks populated with vulnerable hosts and systems, while keeping them segregated from my internal "production" systems and home network.

I have three separate host-only networks configured. Each network is assigned to a virtual switch and has a Endian Firewall with two interfaces. The firewalls are precisely configured to allow and deny strategic ports and protocols to pass between the three malnet networks enabling a variety of attack scenarios. The front-end Endian firewall that separates the virtual and physical networks is configured to avoid any contamination from the vulnerable/exploited/infected hosts on the malnet networks.

Each of the malnet networks and the DMZ honeynet are monitored by Snort intrusion detection sensors. I use the excellent open source IDS package, Security Onion, as the lab's core IDS. Security Onion combines open source security packages Snort and OSSEC with the security management consoles squil, Snorby and SQuert to make one great security monitoring Linux distro. ESXi virtual interfaces and switches make it easy to assign IDS monitoring interfaces to any of the virtual networks on the fly.

The malnet networks are populated with some of the deliberately vulnerable OS images available on the internet, such as metasploitable and Kioptix. I am also creating my own set of vulnerable virtual hosts and services which are designed to be exploited in interesting and clever ways. The malnet02 network, for example, contains a poorly configured and largely unpatched Windows 2003 AD domain controller that is running several vulnerable services. Attacking and exploiting hosts like this is a great way to keep your offensive skills sharpened while deepening your understanding of how to defend.

The internet firewall for my home network is a Cisco ASA 5505 with an enterprise security license. This enables me to assign a physical port on the firewall as a DMZ interface. I connected this interface to one of the ethernet interfaces in ESXi and assigned it to a virtual switch. I can now connect any virtual machine to the DMZ switch and with a few clicks can place it in the DMZ honeynet network. The network currently has one host running a honeypot. The ASA is configured to allow inbound traffic on a variety of ports (TCP 21,22,110,1433,3389) to the DMZ from the internet.

For wireless hacking research and practice, I added a WRT54G Linksys wireless router configured as an access point with default Linksys firmware. I can use the ALFA USB card attached to my laptop or I can connect the ALFA attached to the ESXi box to any virtual host and attack the wireless network from a VM.

Here is a screenshot of the ESXi Summary page in Virtual Center:

Hardware:
A complete list of the hardware I used for the ESXi server is listed below.

The hardware I chose was one generation behind when I purchased it last November. The parts I chose are widely in use and generally accepted to be compatible with ESXi. This was the most important factor - building the system with proven hardware components and avoiding any incompatibilities.

CPU:
Intel Xeon X3450 2.67GHz. This CPU is a quad core. With hyper-threading enabled, it provides eight usable logical processors in ESXi. I let ESXi manage the CPU distribution but I can alocate virtual machines to individual logical processors if I need to. I have been impressed with how efficiently ESXi allocates memory and CPU; aside from assigning memory to virtual machines, I let ESXi handle all the resource allocation.

Motherboard:
Supermicro X8SIL. This board and its well proven with ESXi. It has an integrated IPMI interface which enables "lights-out" style management of the system. IPMI is essentially a web interface to control the system via software running on the motherboard. This enables me to power cycle the system or view the local console remotely. I looked at it once and have not had reason to look again.

Memory:
32GB (4 x 8) Kingston ECC registered DDR3.
I spent a lot of time researching memory for the system making absolutely sure it was compatible with ESXi and the Supermicro motherboard. This memory is on the VMware HCL.

Storage:
2 x Seagate Barracuda 1 TB 7200 rpm hard drives. I have always found Seagate drives to be quiet and reliable and these drives are no exception. I have no need for RAID in my lab so I installed both as storage in ESXi. My VMs sit on one, and my ISOs and some backups are on the other. A simple and effective arrangement.

ESXi runs from an 8 GB USB key. The Supermicro has a USB port directly on the motherboard, so it's connected there and hidden away in the case. Its easy to back up my ESXi OS and configuration by simply imaging the USB key.

Networking:
The motherboard has two physical ethernet interfaces (three if you count IPMI) and I added an Intel dual port Gigabit NIC (82546EB) I had from another system for a total of four usable physical interfaces. An ALFA networks wireless adapter is attached to ESXi via USB.

The ESXi adapters hosting the malnet networks are connected to a Cisco SG300 10 port managed switch. I have enabled layer three routing on this device so I can route between VLANs on the switch. The switch can be managed with a web interface or via the command line.

Power Supply:
I used a Cooler Master GX 450W for a power supply as it was reasonably priced and had excellent reviews. It's also whisper quiet, which was another one of my requirements for the system.

Overall, I am very happy with my ESXi lab. A sandbox to run attacks and test defenses is an essential tool for anyone who wants to learn and keep their offensive and defensive security skills sharp.

OSCP - My review

The truism "anything worth having doesn't come easy" is one I have often remembered when on a particularly difficult path to a goal. Never have the words rung quite so true when applied to my quest for the OSCP certification. This phrase, along with several other quotes and snips of wisdom helped to motivate me though the PWB (Penetration Testing with Backtrack) course and final 24 hour exam.

The OSCP certification is an offensive security course which teaches the attacking side of Information Security and is largely aimed at those wanting to become penetration testers. My personal motivation for taking the course and exam were to better understand the methodology, tools and techniques that attackers employ to breach networks and systems. I have been a dabbler with offensive security practices for several years, have read several books on the subject, taken courses and run my own lab of vulnerable systems to practice on. I wanted to consolidate, formalize and measure the basic knowledge I had gained though my own exercises and the PWB course seemed like a perfect way to do this.

Obtaining the OSCP certification requires taking a self-paced course "Penetration testing with Backtrack" and passing a final exam. The course materials consist of a PDF manual, a lab full of vulnerable systems and a set of videos which complement and enhance the exercises in the PDF. The student is expected to review each section of the course and in some cases complete a given exercise and document it at the end of the given module. The course starts off fairly easy with some simple scripting but soon ramps up to scanning, buffer overflows, web application hacking, client side attacks, password attacks etc. Each module in the course is well laid out and presented. The videos are voiced by the Backtrack CIO himself "Mutts" and he does a great job of explaining the material at hand. Some of the videos and modules are worth repeating to really digest the concepts being explained.

The student is expected to follow up the modules with their own research and where necessary seek answers to questions or expand on the topic which may not have been fully understood in the text or video. For example, I went through the buffer overflow section of the course twice and then practiced on some vulnerable applications to really digest and understand the subject. Eventually, after much frustration the concepts clicked and I soon found myself writing some of my own simple buffer overflow exploits. The feeling of accomplishment I got from this was tremendous and underlines the Offensive Security mantra of "Try harder".

Once I started to get comfortable with the exercises and documentation it was time to move on to the lab. The PWB lab is comprised of multiple networks and systems which contain wide range of vulnerable applications and systems spread across several networks. The student connects to the lab via a VPN connection from Backtrack. Once the student starts working in the PWB lab they are expected to document each system they manage to break into, and in the case of root or administrator access retrieve a key from the administrator's desktop as proof of compromise. Some of the systems in the lab are relatively easy to get access to, but many are not and present challenges that would frustrate a trappist monk. I spent many late nights trying harder and battling to gain access to a system, breaking one barrier only to encounter another. Applying "Try harder" often worked in these cases and forced me to think and approach problems in new and novel ways. After extended periods of study and practice I found myself able to slip into a hacker's mindset far more easily.

The PWB lab is really well designed. There are multiple ways of gaining access to many of the systems and some systems lead to other networks. For example, some systems are dual homed and have access to other networks which also contain vulnerable systems. The dual homed systems are great for practicing pivoting and attacking systems and networks though intermediary hosts. This often involves tunneling attacks through hosts you already control to circumvent firewall rules. Many of the vulnerabilities in the lab require you to download, fix and compile exploit code. Often in these cases the devil is in the very minor details and absolute focus and concentration is required to get an exploit to work the way you want it to.

After several months in the lab I managed to break into more than 35 systems. I had root or administrator access on almost all of them. As I had taken a couple of extensions generously paid for by my employer I decided to book the PWB 24 hour challenge and make an attempt at gaining the full certification. The OSCP challenge requires that the student connect to a new network containing hosts they have never seen and to compromise enough of them to gain enough points to pass. The student is given 24 hours to complete the challenge and then a further 24 hours to submit their final report for review.

I took two days off work and told my team to only call me if something was on fire or someone was dead. I started the challenge at 11 AM on Thursday morning. I spent the first couple of hours just getting a lay of the land and planning my attacks. The rest of the day was a blur, I remember my wife bringing me food a couple of times and my dogs wondering why I was still up typing furiously at 4 AM. I had made good progress throughout the day but was stuck needing 10 points to pass and my weary mind was starting to demand sleep. I considered packing it in and taking the exam again at later date when I decided to give it one final push. Sometime around 7 AM on Friday morning I was finally done, I had owned everything with the exception of one box, a box that I had user privileges on and tried so hard to elevate. I probably spent 5 hours on it alone. I stumbled into bed as my wife was getting up for work. I drifted off to sleep with a big smile on my face. I had really done it and it was over. I was both elated and sad as I had grown attached to my late night study and hacking sessions in the lab, listening to the inception soundtrack or just silence save for my typing.

Documenting each system I hacked was probably my least favorite part of the course, but absolutely necessary as part of the process. As I worked my way through the lab systems I took notes, console output and screenshots of each compromise to use later in my final report. If I had the course over again I would have documented each system completely as I rooted them, rather than waiting to near the end to compile all my notes into a cohesive report. This ended up taking me almost a week and another day for my final exam report. Counting my lab exercises my final report was 350 pages.

When I received the official word from Offensive Security that I had passed I was also given access to a discussion forum restricted to those who had also passed the PWB challenge. The forum contains war stories from the labs and solutions to some of the exam systems. I looked up the host that I had tried so hard to elevate from user to admin and found that I was extremely close the whole time, a minor change in one parameter would have done the trick. ;-)

I would highly recommend the PWB course to anyone who is serious about Information Security. More than just a hands on technical challenge, it's also a test of determination and perseverance.

Try Harder!