I lost my voice before speaking at DEFCON… and went on anyways

Thanks to my open source work, I have a lot of opportunities to speak. In 2011, I gave 30 talks between conferences, professional groups, and invitations to private organizations. This week, much of my industry has collectively gone on vacation to Las Vegas for BlackHat USA, BSides Las Vegas, and of course… DEFCON.

I was fortunate to have an opportunity to speak or demonstrate my work at all three events. Wednesday I spoke at the BlackHat arsenal. This event involves standing at a pod in the hallway and speaking over the noise of the conference for one hour. It’s one of my favorite events because it’s focused on demonstrations and I get a chance to interact with everyone in the audience.

Arsenal

Thursday was a mad rush. I spent the morning at the BSides conference. I gave my talk, hung around to answer questions, hopped a cab, and found myself back at Caesar’s Palace to demonstrate Armitage again at BlackHat.

For me, the big show was DEFCON though. This is where I would release the results of my 7-month DARPA effort, Cortana. And, despite the many times I’ve had the opportunity to share my work, something happened that I have never experienced before a talk: I lost my voice.

Thursday night, while I practiced my talk, my voice stopped working. It went to a very harsh whisper and my throat felt sore. I decided to stop practicing and focus on testing my live demonstrations instead.

I woke up at 7:30am Friday morning and I couldn’t speak.

Seriously.

My talk was at noon. I first paid the $6/bottle minibar fee and drank every bottle of water I could find in my room.  I then tried putting a hot towel on my neck.

No voice.

I walked to the Walgreens Pharmacy. I used my phone and pointing to communicate with the pharmacist. She wasn’t too enthusiastic. She told me to take some ibuprofen and suck on cough drops.

As I checked out, I used gestures to communicate with the cashier. She switched to American Sign Language to communicate with me. I don’t know American Sign Language beyond hello. I smiled and left.

At this point, it’s time to head to the Rio. I’m at a loss for options. I thought about emailing the lead speaker liaison and asking to switch my time or give my spot back to an alternate. I couldn’t get behind that option mentally. I had no idea what I would do at this point, but I knew I would make the show go on.

Downstairs, I’m waiting in line for a taxi cab. I exchange several glances with the guy behind me. Finally, he asks if I’m headed to the same place and asks if I want to share a cab. I gesture yes. Thankfully… I was saved the trouble of finding a way to communicate with the driver.

Once we were in the car, I pulled out my laptop and wrote a message explaining my strange behavior at the moment.

I justified continuing to “speak” despite having no voice. I figured at the very least… I had live demonstrations and possibly, I could croak out parts of my talk by keeping the microphone really close.

We get to the hotel and pay the cab. I go to the speaker registration area. I think there is something truly ironic about registering at the speaker registration area with no voice. Again, I used my laptop to communicate with the DEFCON staff.

At this time, it’s 11:15am. I’m on at noon. I have no voice beyond a very harsh sounding whisper that I dare not use for fear of losing even that.

I go to the speaker ready room. The staff manning the room does the usual, “can I help you?” I pull out my laptop and communicate through a text editor. They laugh and send me to the next room to sit with the other speakers. Several folks from the EFF were discussing their Q&A panel that would happen at the same time. The room also had buzz as General Alexander was in our same suite. The tight security kept us planted at our table.

I used the text-to-speech feature on my Macintosh to converse with my fellow speakers. It was funny to type a thought, press enter, and watch for reactions. After a few rounds of this, I felt like I was participating somewhat in the conversation.

I converted my presentation to PDF. I knew, at worse, I could maybe try using the text to speech feature to communicate some things verbally. Having a PDF would allow me to keep my slides up and a terminal for typing text at the same time. I still had no idea what I would do, but a plan was forming.

My DEFCON speaker goon, Bushy, introduced himself and I explained my situation in a text editor. The speaker goon’s role is to make get speaker’s where they’re going, help them watch the clock, and make sure everything goes smooth. I attempted to speak, close to Bushy’s ear, to test my voice and indicate that I still had something left. Sadly… I didn’t. My voice was still a terribly hoarse whisper.

We then started the trek to the Penn and Teller theater where I would deliver my presentation.

Bushy asked if I tried hot tea with honey yet. I had tried a few things, but not the hot tea. I signed that I had not. We went to Starbucks where there was an incredibly long line. Bushy jumps right to the front and states “I’ll pay for all of your drinks if you just order a hot tea”. He immediately followed it up with “I have a speaker here who is on in 10 minutes and he lost his voice”. The nice couple at the front of the line immediately put the order in and they refused anyone’s offer to compensate them for the tea.

I added three packets of honey to the tea and drank such a big initial swig that I… burnt my tongue. Desperate times call for desperate measures?

We then proceed to the Penn and Teller theater. As we walk, hotel security keeps stopping me and explaining that I can’t carry a drink. Each time, I had to look back and get Bushy’s attention to intervene on my behalf. I didn’t have the voice to explain myself after all.

The final guard, right before the theater, shook his head and laughed when he learned about the situation.

We then enter the Penn and Teller theater. This theater has two levels and seats about 1,500 people. I don’t know if it was configured differently for DEFCON. This part’s a blur to me. I do know that I approached a real stage, saw the sophisticated lighting gear around me, and as I looked at the tiered seating to my left and right… the theater was nearly full.

I’ve never spoken at DEFCON before and I did not know what to expect. I didn’t expect I would see such a full room or find myself escorted to such a prominent stage. I felt like a musician who was expected to perform… but couldn’t.

I see friends in the audience who try to greet me. From stage, I point to my voice and make a motion with my hands to indicate that I had no voice. They got the message and I could see the “Oh… my…” look in their face.

I wasn’t nervous at this point but I still had no idea what would happen.

I setup my laptop and opened a text editor. I wrote:

“Life is an adventure. Here I am speaking to ~1,000 of you. We’re going to have fun today, but you should know… I lost my voice”

Some people noticed it and I could immediately here the “haha, oh my” reaction.

I kept drinking my tea and honey.

Bushy introduces me and states that this is the first time in DEFCON history a speaker has… not had their voice.

By this time, I knew I could croak some things. At worse, I could let everyone read my slides and make a short comment about each. This would still get the content across.

I brought the microphone close and found I could croak enough to speak. My tempo was slow at first. As I kept going (and drinking my tea) my voice slowly came back. It never came back all the way, but it was 100x better than I expected.

Defcon 2012
I was able to deliver the entire 50 minute lecture as I intended to. My demonstrations worked. And I had a great experience.

Thank you Bushy for saving the day for me. You went above and beyond as a speaker goon and while I had no idea what I would do, I thank you for helping me get up to the stage where everything worked out.

This is one speaking experience that I will never forget.

Coming to Vegas: My Picks and Schedule

And, like most folks in the security industry, I’m getting ready to head to Las Vegas for the week. I’ll arrive in Sin City on Monday, 23 July 12.

I’m armed with Cobalt Strike comic fliers, business cards, and boring sell sheets. I also have a pile of Armitage stickers to give away.

In this (long) blog post, I recommend a few talks and provide my schedule. If you’d like to meet for a meal or a beverage, I’m definitely open to this. Contact me and we’ll discuss a way to sync up.

My Picks

Hacking into Smartphones

On Wednesday and Thursday at 11:45am in the BlackHat USA Arsenal, Georgia Weidman will demo her Smartphone Penetration Testing framework. This is not a tool for “hacking from” smartphones. It’s a tool set for hacking into smartphones. I highly recommend attending one of these demo sessions. I’m demoing Armitage at the same time, but if you go see Georgia twice, I’ll give you twice as many stickers.

Ambush

At BSides Las Vegas on Wednesday at 11am, Matt Weeks will reveal a new defensive technology called Ambush to the world. We’ve had a few discussions about this technology. As usual, Matt is up to something incredibly novel.

This presentation will demonstrate the ability to configure signatures on everything from the highest to the lowest level API’s, catching whole classes of malware. It can prevent exploitation of certain vulnerabilities and identify shellcode, keylogging, remote control, and HTTPS-encrypted communications regardless of code obfuscation. Pentesters, red teams, and malware authors used to worry about getting caught while writing to disk. Now, no action is safe. The implementation, the Ambush Host Intrusion Prevention System, will be released open-source.

SNSCAT: Social Media Sites as Covert Command and Control

I’m really excited about this. My friends Dan Gunter and Solomon Sonya will reveal SNSCat at BlackHat USA at 2:15pm on Thursday.

“SNSCat provides a simple to use post-penetration data exfiltration/infiltration and C2 (Command and Control) platform using images and documents on social media sites (Facebook, Google Apps, twitter, imgur, etc).”

“will introduce our tool and show how one can easily move files in and out of a network using social media sites. We will next demonstrate how one can use SNSCat along with the implants we have created to establish full command and control between the controller and the listening agents.”

Grab thy Hashes

And, finally my friends Jon Claudius and Ryan Reynolds will present a survey of how different tools extract password hashes on Windows. The twist–most of them do it in a semi-broken way. They’ve analyzed the problem and they’re revealing fixes for key tools that penetration testers take advantage of.

This is extremely important. I feel like a lot of tools released at conferences are one-time things that will never see an update later on. It makes working with them frustrating as the work may be novel, make a great demo, but if it doesn’t work a year from now–what’s the benefit? It’s great to see someone looking at what we use every day, figuring out what’s wrong, and contributing back in a way that will benefit a lot of people immediately.

This talk is happening at 11am on Saturday at DEFCON.

My Schedule

Tuesday, 24 July 12

I’m hanging out at the Adaptive Penetration Testing course taught at BlackHat USA by the Veris Group LLC.

Wednesday, 25 July 12 (BlackHat USA)

I will demo Armitage in the BlackHat Arsenal at 11:45am. My goal during the demo is to explain Armitage to those who haven’t seen it and capture some of the cool tricks few people know about. For example, Ctrl+T takes a screenshot of the current tab and saves it to a preset place.

Thursday, 26 July 12. Morning (BSides Las Vegas)

At 10am, I will present Force Multipliers for Red Team Operations. Each year, in March and April, I spend most of these two months on the road hacking in several exercises. I treat these events as a laboratory for trying out ideas and making observations about how hackers work together. I will break down what I learned from this year’s season with a focus on how we organized ourselves, what worked, and offer ideas of what I’d like to see next.

Thursday, 26 July 12. Afternoon (BlackHat USA)

I’m back at BlackHat at 11:45am demoing in the Arsenal again. If you missed me on Wednesday, come by on Thursday and get a sticker. I really dig these kiosk style demos. It’s easier to connect with you and have a dialog.

Friday, 27 July 12 (DEFCON)

At noon on Friday, I’m presenting Cortana: Rise of the Automated Red Team. During this talk, I will reveal the fully scriptable version of Armitage and its stand-alone interpreter Cortana. You’ll learn how to add bots to your red team or add new features to Armitage. This project was a big effort to put together and I was very fortunate that DARPA’s Cyber Fast Track program helped make it possible.

Here’s a Hak5 segment from last year where I first talked about this next iteration for Armitage:

I also noticed that I’m speaking opposite of General Alexander from US Cyber Command and the NSA. I guarantee I will give far more live demos than he will. That said, I wish I wasn’t speaking at noon, I’d love to see his talk too.

Saturday, 28 July 12 and Sunday 29 July 12 (DEFCON)

I’m at DEFCON all weekend and I fly out Monday morning.

Cobalt Strike 1.44 / 19 Jul 12 Update

Another Cobalt Strike update is available. This update makes Cobalt Strike compatible with version 4.4 of the Metasploit Framework.

Here are the new features in this update:

  • Cobalt Strike now has a USB attack generator built in. This was fun to put together and I can imagine, even more fun to deploy. The default settings emulate the social engineering attack used by the Conficker worm to spread itself.

  • For those of you hiding behind a NAT device, this release also adds a means to notify Cobalt Strike and the Metasploit Framework of your proper external IP address. Go to Cobalt Strike -> Listeners -> set LHOST to try it out. The old solution was to set up a Cobalt Strike team server and specify the right IP address during startup. Not everyone uses a team server during their engagements (why not?), so for you–I’ve added this ability.

And two notable bug fixes:

  • Recent changes to the Metasploit Framework database schema caused some issues with Cobalt Strike’s vulnerability descriptions. This issue has been fixed. Your hosts and vulnerability reports should look lovely again.
  • This release addresses a configuration issue preventing permanent reverse_http and reverse_https listeners from functioning. Previously, Cobalt Strike would bind to 0.0.0.0 to accept a connection on any interface. Unfortunately, this means the http/https payloads will try to communicate with 0.0.0.0 instead of your system after the initial handshake. If you were experiencing trouble with these payloads using Cobalt Strike before, this fix addresses the issue.

If you’d like to learn more, take a glance at the full release notes. A 7-day trial with these updates is available too. Licensed users may run the update program included with Cobalt Strike to get the latest.

Use Armitage and Cobalt Strike on Amazon’s EC2

James Webb has an interesting blog post on how to use Armitage to manage a pen test through Amazon’s Elastic Computing Cloud.

He does a good job articulating the benefits which include using Amazon’s EC2 to test your security from an outside in perspective or using it as a central point for a distributed red team to work from.

He also explains how to obtain authorization for penetration testing activities from Amazon. They do have a process for this and they’re very good about responding to these requests.

You can use Cobalt Strike or Armitage to work with Amazon’s EC2. If you use Cobalt Strike, I recommend using the quick-msf-setup script included with Cobalt Strike to quickly setup your environment. This process is described in the Cobalt Strike Linux Installation Instructions.

Also, when you run the teamserver, make sure you specify the external IP address of the EC2 node and not the private address bound to the network interface on the system. By specifying an external IP address, you’re telling the Metasploit Framework where it should send reverse connections to by default. It’s really important that this IP address is something your target systems can talk to.

Link

Cobalt Strike Interviews

On Cobalt Strike launch day, I had a couple of opportunities to tell the Cobalt Strike story and share what’s happening with the world. First, I was on PaulDotCom episode 292. In this interview, the PaulDotCom crew asks about Cobalt Strike, what it does, and Carlos Perez posts crazy PowerShell recipes to the Skype chat.

http://youtu.be/umXJdd2w_RA

I was also on the McAfee AudioParasitics podcast discussing Cobalt Strike. Here’s the episode description:

AudioParasitics is on the Offense – Dave and Jim welcome special guest Raphael Mudge!! Known far and wide as the creator and developer of Armitage, Raphael gives us the history behind the tool and touches on some exciting new efforts. We dig deep into the new Cobalt Strike tool, as well upcoming release of Cortana at DC20.

This is a great interview. In it, I get to share how Cortana was funded through the DARPA Cyber Fast Track program. I talk about the history of Armitage and I get a chance to discuss what Cobalt Strike is trying to accomplish. Check out the podcast for more:

Cobalt Strike Video Review

Ryan Linn created a video review of Cobalt Strike for the Ethical Hacker Network. Unfortunately, I can’t embed the video into the blog post, but I encourage you to check it out. It’s 20 minutes with a well-regarded expert taking Cobalt Strike through its paces.

Overall, I enjoyed getting to learn Cobalt Strike. It’s a new release, and it wasn’t perfect. On the other hand, it did all of the things that I needed to do quickly, and it made pass-the-hash a lot easier than going through the console. Having different tables was another nice feature, so that multiple tasks could be done at once and compartmentalized so that the text wasn’t intermixed. As it continues to mature and add features, Cobalt Strike is going to be a good tool for individual testers and teams who aren’t looking to spend $100k on tools.

http://www.ethicalhacker.net/content/view/433/1/

Update 11/27/12: Don at ethicalhacker.net has put the video review on YouTube. Thanks Don! Here it is:

Meet Cobalt Strike: Adaptive Pen Testing

If you’re reading this, you’re likely aware of the Armitage project. Fed by your enthusiasm and feedback, Armitage has enjoyed a rapid pace of development since its inception. I left a security engineer role one year ago to search out how to properly nurture this project and its ideas going forward. This search led to some exciting initiatives, one that I’m announcing, right now.

I’d like to introduce you to Armitage’s big brother: Cobalt Strike

Cobalt Strike is a penetration testing suite built for threat emulation. I say suite, because it’s not just software. It’s documentation, online training, and a set of tools to help you execute an adaptive penetration test.

Cobalt Strike adds client-side reconnaissance, spear phishing, web drive-by attacks, and reporting to Armitage’s red team collaboration and post-exploitation capabilities.

Now that you’ve met Cobalt Strike, here are the next steps:

1. Watch the Cobalt Strike trailer to get a taste of Cobalt Strike

2. Visit the Cobalt Strike website and request a trial to try Cobalt Strike

3. Get Cobalt Strike into your organization: buy online or request a quote.

Live Training at BlackHat USA

If you’re ready to add Adaptive Penetration Testing to your organization’s skill set, I recommend signing up for the BlackHat USA course run by the Veris Group. This course is a vendor neutral offering, but those who attend will have an opportunity to play with Cobalt Strike under the guidance of a seasoned instructor team.

The instructors David, Jason, and Chris are among the early adopters who helped shape this product.

And, what about Armitage?

Armitage, Cobalt Strike, and my security research initiatives are now under the banner of Strategic Cyber LLC. The formation of this company is an exciting opportunity. I can now work more formally with many of you and strengthen new and existing relationships.

Armitage will enjoy the same development pace and it will stay open source, always. Even better, I’m releasing something really big for Armitage at DEFCON 20.

I hope to see you there!

— Raphael


Raphael Mudge
Principal, Strategic Cyber LLC
http://www.advancedpentest.com/
1-888-761-7773

Bloggers and Journalists: More information about Strategic Cyber LLC and Cobalt Strike is available in our press kit.

DARPA’s Cyber Fast Track: My Experience

Last week, I received a grant from DARPA through the Cyber Fast Track program. I consider this a big milestone in my personal career. If you’re an independent researcher or entrepreneur, bent on making your ideas real, then this program is for you.

This blog post will give you my experience applying and getting funded by this program. I’ve chosen a question and answer format because I had a ton of questions when I applied. I hope answering these questions encourages you to take advantage of this amazing opportunity.

Before we begin, please remember: none of this is the official word of DARPA. This blog post merely reflects my understanding. Also, if you arrived here with a Google search, my last name is Mudge, but I am not the Program Manager Mudge at DARPA.

What is Cyber Fast Track?

Cyber Fast Track is a DARPA program to fund us, the hacker community. Here’s the description from the Cyber Fast Track page:

The Defense Advanced Research Projects Agency’s Cyber Fast Track program is aimed at improving Cyber Security. This program will rely on the skills of small organizations, boutiques, hacker spaces and maker labs to address cyber security issues.

According to DARPA program manager, Peiter “Mudge” Zatko, instead of engaging in traditional programs that don’t produce results for years, we envision results within months by harnessing teams or individuals on the back of short, fixed-price DARPA contracts.

If you’re an individual who is trying to advance the security community through independent research, DARPA wants to hear from you. You don’t have to work for a big defense contractor, you don’t have to work for anyone. I applied as an individual with no formal organization behind me.

What is the process?

If you want to apply, I recommend reading the research announcement at FedBizOpps. This will give you all the details on the program.

Next, go to the Cyber Fast Track resources page and get the proposal template. I started my proposal without the template. This was a mistake. The template makes writing the proposal much easier. Plus, it’s better to give DARPA what they expect.

The technical meat of your proposal is 10-15 pages. I pushed the upper end of this. It took me about four days spread out over two weeks to create a proposal I was happy with.

Submitting the proposal is easy. I created an encrypted ZIP file with the proposal. I uploaded the proposal ZIP to a website. And I emailed the password to open the ZIP file to DARPA. The details on how to do this are in the research announcement.

I submitted my proposal on Friday, 21 Oct 11. I received a phone call of acceptance on Thursday, 3 Nov 11. I had a signed contract on Friday, 4 Nov 11. This is a mind blowingly fast turn-around to fund a program.

Do I need an LLC, DBA, C Corp, S Corp, LLP, or GmbH?

Nope. You can apply as an individual. I put down Raphael Mudge as my company.

Do I need a lawyer?

The contract is one page. My contract listed the milestones I proposed, the dates I said I would deliver them by, and the price I proposed for each milestone. Since I applied as an individual, I also had to affirm that I am an independent contractor, not an employee, and that I am responsible for all taxes on what I receive.

If you’ve dealt with contracts, you may know the pain of reading a “we own your first child” clause, raising the point, and cringing as some sleaze asserts “it’s boiler-plate, all contracts have it” while breathing their lunch cocktails in your face. There is none of that here. This is the simplest contract I have signed.

Who gets the rights to my work?

You keep all commercial rights. The Cyber Fast Track FAQ has a thorough answer to this question.

What does DARPA get?

DARPA recognizes that big contributions may come from fringe thinkers doing what they love without constraints. This program allows those who are motivated to pursue their wacky vision. Your project may not change the world, but with hundreds of these, something big is bound to happen. It’s kind of like the Y Combinator model for defense contracts. Or in short–they’re spending their money to advance the state of the art.

How much do I ask for?

This depends. How long will your effort take? I recommend that you figure out what you want to do, list several milestones, and then estimate how long each milestone will take.

Now you should have some number of hours. Make sure your time estimate is realistic. Cyber Fast Track contracts are firm-fixed price. You’re on the hook to deliver what you propose in the amount of time you claim.

Your next task is to figure out an hourly bill rate. Your bill rate must fit the government accepted rate for someone at your career level. In my last position, I worked as a Senior Security Engineer.

I used Google to search for “Senior Security Engineer” hourly rate site:gsaadvantage.gov. This yielded price lists for various defense contractors. Pick one that works for you and multiply that by the number of hours you estimated. Now you know how much to ask for.

Optionally, find someone who consults for the government or owns a defense contracting company and ask them for advice. I followed both of these approaches and they each yielded the same numbers.

Do I need a security clearance, CAGE code, or a DUNS number?

No.

What should I apply with?

The research announcement has the DARPA answer on what they’re looking for. Short answer though, they’re looking for interesting security research. Apply with what you’re interested in. Don’t worry about what the government wants or what they need. Take your thread of research, explain why it’s valuable, who it’s valuable to, and explain what’s new in your approach.

I also recommend scoping your idea as tightly as possible. I used to review proposals when I worked as a researcher and I never backed a proposal that was all over the place. The proposal reviewer should understand the problem you’re solving and your plan to solve it after they read the first paragraphs of your executive summary.

Ideally, the reviewer should understand your project from the title alone. This isn’t always possible, but do your best.

Keep in mind that Cyber Fast Track does not fund improvements to existing technologies. I have a research interest in red team organization and tactics. Armitage is my current vehicle to explore this research interest. I cut a stand-alone project out of my long-term road map. I emphasized the research questions this stand-alone project would address and this became my proposal.

What are my chances of getting funded?

DARPA has seen 30 proposals and funded 8 so far. A ~25% acceptance rate. This is better than some conferences I’ve applied to. Network World has these numbers and the titles of the current efforts.

How do I stack the odds in my favor?

Make your proposal easy to read. If your proposal is poorly written, you will torture your reviewer(s). Good proposals are short and they inform the reader.

  • Use bullets when listing several ideas. This will make them stand out.
  • Write in the active voice.
  • Use simple words over complex ones

I recommend that you visit plainlanguage.gov for Plain English writing tips. I also wrote a writing style checker that may help you. If you want to read a book, try Bill Stott’s Write to the Point. It’s my favorite book on writing.

Where to go from here

Cyber Fast Track takes away all the friction for valid ideas to receive funding. This is the first time in my career I have seen something like this. If it fits you, take advantage of it!

If you’re in New York City on November 9, go to the Cyber Fast Track Town Hall at NYU Poly. Watch the Cyber Fast Track Events page for future events.

My VirtualBox Penetration Testing Lab

Last week I taught an Advanced Threat Tactics course at the Lonestar Application Security conference. I like to provide ample hands-on opportunities in my courses. The students retain much more this way. I decided to use the class proceeds to build a killer virtual machine server for my students to hack on.

Requirements

My requirements were as follows:

  • Run a lot of virtual machines at once (this is not a very specific requirement)
  • Headless. I live in a Washington, DC apartment. I do not want to waste room providing a keyboard, monitor, and mouse to administer it. I do not want to travel with these items either.
  • Travel friendly. I used this server to teach a class. It needed to travel with me.
  • MacOS X friendly. I am not a Windows user.

Hardware

I initially built my server to run VMWare ESXi. I built my server using parts from newegg.com. I read that newegg.com has a terrible record with dead on arrival hard drives, so I bought these from staples.com.

Here’s the part list:

  • Case: Shuttle SH67H3 [ $240 ]
  • Network Card: Intel EXPI9301CTBLK 10/100/1000Mbps PCI-Express Network Adapter [ $30 ]
  • RAM: CORSAIR XMS3 16GB (4 x 4GB) [ $100 ]
  • Processor: Intel Core i7-2600 Sandy Bridge 3.4Ghz (3.7GhZ Turbo Boost) Quad-Core [ $300 ]
  • Hard drive: Two 1TB Seagate Barracuda disks, 7200 RPM with 32MB cache [ $140 ] *

Total cost? $810

  • I picked up two hard drives to allow a RAID-1 configuration

Software

The operating systems problem is an easy one. I own an MSDN subscription. This gives me access to all of Microsoft’s operating systems going back to Windows 3.x and DOS. Missing is Windows 95, 98, and 2000. I believe this is because of a court ruling related to their crippled Java many years ago. I’ve had MSDN since June and I am extremely happy with it.

VMWare ESXi

The virtual machine software was not such an easy story. I tried VMWare ESXi first. It doesn’t need a host operating system, meaning more system resources go to powering my virtual machines. I own VMWare Fusion for MacOS X and I love it. VMWare ESXi seems like the natural choice.

I installed VMWare ESXi, a process that was not without its trials. At some point it no longer recognized my USB keyboard. I found plugging my keyboard into a port in the back of the system allowed the install process to execute smoothly. I quickly learned that the best way to manage ESXi is through the vSphere client. vSphere is only available for Windows. This immediately went against my MacOS X friendly requirement.

Also, I did not know what functionality was missing from VMWare ESXi/vSphere without their paid package. I quickly learned that the missing functionality included the ability to quickly clone virtual machines. For someone setting up a penetration testing lab, this feature is a must. Many people on Twitter came to the rescue with an assortment of hacks to get around this limitation. I don’t like hacks for simple things like cloning a virtual machine.

Also, using vSphere in a Windows virtual machine was painfully slow for me. VMWare ESXi is out for me.

VMWare Workstation

Because I’m already familiar with VMWare Fusion and Workstation for Windows, I opted to try VMWare Workstation. I installed Ubuntu 10.04LTS. I tried export the user interface using X windows and I tried interacting with it through VNC. Even on my local network, both options were too slow. VMWare Workstation quickly failed the test for me.

VirtualBox

The last option I tried was Oracle’s VirtualBox. I should have tried it first. I installed VirtualBox on Ubuntu 10.04LTS server. I didn’t install X windows at all. The entire install was done while logged in remotely from Terminal.app on MacOS X. I then set up phpVirtualBox to administer the system. phpVirtualBox is fantastic. I can easily configure, make linked clones of, snapshot/restore, and start/stop my virtual machines.

Through phpVirtualBox it’s also trivial to create multiple “host-only” networks and assign them to your virtual machines. This is great for creating isolated enclaves bridged only by a dual-homed virtual machine I set up.

VirtualBox also has an RDP server built-in. phpVirtualBox exposes a flash RDP client to allow me to interact with the virtual machines in my browser. The VirtualBox Guest Additions also fixed wonky mouse issues for me.

In the end I went with VirtualBox and phpVirtualBox as my headless virtual machine solution of choice.

One tip: when running VirtualBox, make sure the kvm-intel and kvm-amd kernel modules are not loaded. These modules conflict with VirtualBox and your VMs will not start. Just because they don’t exist on one boot doesn’t mean you won’t see them later. I was bit by this. Save yourself from my pain.

Conclusion

So, with my hardware in place I set up a penetration testing lab. I had a Windows 2008 domain controller, several Windows 2003 servers, several Windows XP and Windows 7 workstations, and a garden variety of Linux boxes for various purposes. I’ve had up to 15 virtual machines running simultaneously and can go higher. This lab passed my “run lots of virtual machines test”.

Using my browser and phpVirtualBox I was able to administer every aspect of the virtual machine creation and configuration process. This system passed the headless requirement.

This box fit into my carry on luggage and TSA let me through the checkpoint with it no problem. At BWI two or three personnel looked at it quizzically, but they let it through. I was quite happy as I didn’t want to check it for fear of what that might do to the system.  So this system passed the travel requirement too.

And, thanks to the headless admin through a browser, this system is definitely MacOS X user friendly as well.

Setting up a penetration testing lab is a good way to hone your system administration skills. It’s also a lot of fun and makes a great place to experiment with different attack techniques at a greater scale.

If you have any questions, leave them in the comments.