Archive for the ‘Strategic Cyber LLC’ Category

h1

Cobalt Strike in 2013 – Closing the Gap Between Pen Testers and Advanced Threats

December 30, 2013

2013 was a good year for Cobalt Strike. From a business perspective, I notice that the understanding of the product is much different from when I put it on the market in June 2012. That’s very helpful. :) From a technical perspective, great strides were made closing the gap between penetration testing tools and advanced threat malware.

This year, I pushed twelve Cobalt Strike releases. Here are some of the highlights:

February 2013, Cobalt Strike introduced a distributed red team operations capability. This feature allows one Cobalt Strike client to connect to multiple team servers and coordinate their actions in an attack. Other penetration testing tools are still single server focused. This was an important move to bring our tools closer to how real threats operate.

Through most of this year, there was a lot of work on Cobalt Strike’s Beacon. This feature really evolved in a big way. It started out as a lifeline to request a Meterpreter session as needed. This year, Beacon has evolved into a multi-protocol communication layer for Meterpreter and the Metasploit Framework. It’s also functional as a remote administration tool. I’ve enhanced Beacon’s ability to stay low and slow, but also added the flexibility to use it interactively and tunnel traffic through it. This year, I also added the ability for Beacon to communicate over DNS and SMB.

October 2013, I introduced browser pivoting. This is a man-in-the-browser attack to hijack authenticated HTTP sessions and use them in an attacker’s browser. This has a lot of implications for government and financial institutions as it demonstrates how a motivated attacker defeats strong two-factor authentication. Conceptually, a lot of us are comfortable with the idea that once the end-point is owned, an attacker can do anything. When it comes to the prove it phase, we sometimes come up short on capability (fixing this is why I’m in business). Browser Pivoting is a risk demonstration tool to show that, without a doubt, once an attacker owns a system, they can access anything else that user has access to.

And, while it’s not a technical change, I cut Tradecraft, a free 9-part online course on red team operations. I took Strategic Cyber’s two-day Advanced Threat Tactics course and cut a video for each lecture. I didn’t hold anything back. I see documentation and code as equally important in a product. Cool insights and new features do no good if they’re not communicated. Cobalt Strike’s freely available educational materials and documentation are one of its great strengths. Tradecraft replaced the Penetration Testing with Cobalt Strike course from January 2012.

Overall 2013 was a pretty rocking year. I expect more of the same in 2014.

h1

Strategic Cyber Heads to Vegas

July 28, 2013

Once each year, the security industry collectively takes a vacation in Las Vegas, NV. I didn’t start going to conferences until a few years ago, but this yearly pilgrimage has grown on me. I greatly enjoy putting faces to names, seeing old friends, and making new ones. I always learn something too.

During the week, here’s where I plan to be:

Tuesday, I’m spending the day between the Veris Group‘s Adaptive Penetration Testing and Adaptive Red Team Tactics courses at BlackHat USA. David McGuire and Jason Frank are the lead instructors of these courses. They’re both awesome guys and I’m a supporter of their courses. I appreciate their mission to help our industry think about the complete attack process… not just exploitation.

Wednesday and Thursday I’m at BlackHat USA.

On Thursday at 10am, I’m demoing “Armitage – A Scriptable Red Team Collaboration Tool” in Station 2 of the BlackHat Arsenal. This demo will emphasize Cortana and its ability to integrate third-party tools into Armitage. This is my third year in the arsenal and I am eternally grateful to Nabil of ToolsWatch for giving me the opportunity to demonstrate Armitage at BlackHat again.

Raffi and Nabil… sitting in a tree… K I…

If crowding around a kiosk is not your thing, on Thursday, at 2:50pm, I will give a BlackHat Arsenal Turbo Presentation on the same topic. I don’t know the location of the Turbo presentations, so do check the schedule once you’re there.

Friday through Sunday, I will be in the DEF CON vendor area giving demonstrations and answering questions about Cobalt Strike. As usual, I like give aways. I have new batches of Armitage and Cobalt Strike stickers. The new Armitage stickers look something like this:

I am also giving away the latest cut of the Cobalt Strike Pen Test Lab DVD. This DVD is a self-contained course on executing targeted attacks as an external actor. It has virtual machines, self-guided labs, and a mapping to my online course. The best way to get it is to come see me and say hi at a conference. So, please stop by. I plan to bring a lot of DVDs with me, but if previous conferences are a good indicator, I will give all of them away–so do stop by before I run out of DVDs. I’ll keep a few with me too, so you may snag one if you run into me casually.

See you in Vegas.

h1

Waging Cyber War with Cobalt Strike at the Collegiate Cyber Defense Competition

March 18, 2013

The 2013 season for the Collegiate Cyber Defense Competition (CCDC) is well underway. These CCDC events put student blue teams in charge of a corporate network. One hour of competition time simulates a week of real life. On top of system administration and business injects, students must defend their networks against a constant barrage of attacks from a professional red team.

In the past, different vendors have made extended trials of their products available for use by the CCDC red teams. In 2012, Rapid7 made Metasploit Pro available. Several years ago, Immunity offered their Canvas product as well. Keeping with this tradition, Strategic Cyber has made Cobalt Strike available to the 2013 red teams.

This offer is more than an extended trial though. I believe a well-prepared red team will help the students get the most out of their CCDC experience. To help CCDC red teams prepare, Strategic Cyber has mailed its pen test lab DVDs to all red team members that requested one. This DVD includes target VMs and self-guided labs on exploitation, social engineering, post-exploitation, pivoting, and collaboration.

Cobalt Strike is a collection of threat emulation tools added to Armitage and the Metasploit Framework. While Cobalt Strike was built for a client-side attack surface, it offers several capabilities CCDC red teams will find useful. Here’s a few of them:

  • Collaboration – While most commercial penetration testing products offer collaboration features now, Cobalt Strike’s little sister Armitage pioneered some of these ideas. Armitage was made to meet CCDC red team needs. With Cobalt Strike, CCDC red teams will have the ability to simultaneously interact with compromised hosts, share data, and track events through a shared event log. Cobalt Strike’s host labels feature also allows the red team to add notes to hosts and to create arbitrary groups of targets.hackers
  • Distributed Operations – A known CCDC Red Team best practice is to setup multiple attack servers, each with a specific role. Red Team members should perform noisy actions, such as attacks and scans, on their local system. Compromised systems should actively communicate with a server dedicated to long-term persistence.  Red Team members should use another server for active post-exploitation and pivoting. This is a lot of attack servers to keep track of!

    Cobalt Strike embraces this idea by enabling distributed operations. One Cobalt Strike client may control multiple attack servers. Cobalt Strike’s distributed ops features make it seamless to send sessions between servers, use all known credentials in a brute force attack, and to set up client-side attacks that span multiple servers.distops_phase3

  • APT-style Command and Control – Years ago, the CCDC red team activity resembled sport fishing. We would exploit a host, marvel at our accomplishment, and throw it back for more exploitation later. Now, CCDC red teams try to mimic a well-embedded adversary. A well-embedded attacker does not maintain an active connection to their victim at all times. They install agents that periodically phone home, request tasks, and execute them.

    Cobalt Strike’s Beacon gives CCDC red teams this asynchronous style command and control. Beacon uses DNS to ask if tasks are available. When tasked, Beacon will download its tasks over HTTP and execute them. Beacon is a first-class payload, like Meterpreter. It’s trivial to deliver it with a client-side exploit, embed it in an executable, and inject it into a process. Beacon will log keystrokes, execute commands, and spawn Meterpreter sessions for active post-exploitation. Beacon is Cobalt Strike’s agent for long-term command and control.

  • Cortana Scripting – One of the hardest parts of CCDC is managing 10+ simultaneous engagements. The CCDC Red Team has to try all attacks against all teams for them to count. Fortunately, it’s trivial to write scripts to automate most red team actions including launching exploits and installing persistence. All Cortana scripts written for use with Armitage will work just fine with Cobalt Strike.

I know we had a lot of fun with Cobalt Strike at the North East and Rocky Mountain CCDC regions. I’m looking forward to the war stories that come from this season.

h1

My Software Development Practices: The Joel Test

February 21, 2013

Joel Spolsky is one of my favorite writers on the topic of software development. He coined a 12-step Joel Test to determine if your company had sane development practices. A lot of these are very common sense, but I’d like to share a little about how I work and this seems a good way to do it.

1. Do you use source control? Yes

I have a private git repository for development of Cobalt Strike. Armitage uses Subversion still (mostly because I’m too lazy to move it from Google Code).

2. Can you make a build in one step? Yes

Yes. I use a fairly standard Apache Ant build file for my Java projects. I think Ant is cumbersome for some things, so I tie multiple build steps together with a shell script that carries out all steps to create trial and production Cobalt Strike packages to deliver to my end users.

Cobalt Strike has a couple of sub-projects written in C for Windows and Linux. These sub-projects each have their own build process for their respective development environments. I’m toying with using a cross-compiler to build the Windows code where I can get away with it.

One example where I can’t use a cross compiler is Beacon. It is tied to a specific Visual Studio environment because of the Reflective DLL injection code it depends on.

3. Do you make daily builds? Sort of

I make builds when I finish a bug fix or make intermediate progress on a feature. On the days I write code, I am doing full builds of Cobalt Strike multiple times.

4. Do you have a bug database? No

I don’t have a formal database. I still track bugs and notes in a text file. When I start growing development beyond myself, I will pick a simple solution to work with. I do not have a bug database for customers to report bugs into. I still handle this over email.

Armitage uses the Google Code bug tracker and I stay on top of this.

5. Do you fix bugs before writing new code? Yes

If I can reproduce the problem and I’m confident I know what the bug is, I will fix it before I work on a new feature. I prefer having a few features that work extremely well over a myriad of features that half work. When I have a bug report, I will go quite far to try to reproduce it. I have an MSDN subscription and I use it to replicate environments when I need to. However, I’ve had bugs that are too hard to reproduce. Sometimes, I’m put in a situation where I have to wait for more clues before I can track down the bug.

6. Do you have an up-to-date schedule? No

I do not work on a schedule. Generally, I aim for a release every 1-3 weeks. I aim to have ~10 logged changes per release. I know which features I intend to build and they have a prioritization. I do not tie these features to specific dates because some genius suggestion, customer request, or bug report may come along and take priority.

In November, I was contacted about Cobalt Strike’s spear phishing tool. I had a trial user who really needed the ability to authenticate to an SMTP server and he was willing to provide access to his infrastructure for testing. I dropped my current development tasks and took advantage of the opportunity to add SMTP authentication to Cobalt Strike’s spear phishing tool. I had always planned to add this, but it became a higher priority when someone asked for it.

7. Do you have a spec? Yes

Another classic Joel Spolsky series is Painless Functional Specifications. In this short series, Joel describes how to write a functional specification to describe a product. I wrote a specification for what would become Cobalt Strike and built the product from it. This spec helped me build my initial product, but I don’t use specs for maintenance.  Sometimes, when I’m planning to build a significant feature, I will write a spec and send it to my trusted board of users who have the opportunity to chime in on it.

8. Do programmers have quiet working conditions? Yes

I work from home where I have a comfortable and quiet setup. Sometimes I listen to music, but often times I prefer to work in crisp silence. I once worked for a defense contractor where phones would ring and people would have meetings on their speakerphone in the cubes around me. This was a miserable experience. I will never do this to a programmer. :)

Sometimes, I work from Affinity Lab. Affinity Lab is a coworking space in Washington, DC with ~60 companies. It’s Strategic Cyber’s official business address too. Affinity Lab is less quiet, but I go there when I need to be around people and accomplish administrative tasks. I sometimes write code there too, when the change of scenery is enough to jumpstart the brain.

9. Do you use the best tools money can buy? Yes

This is a very strong yes. Anything I can spend to grow my business, improve my product, or make myself more efficient is a write-off. Taxes incentivize me to spend! If I don’t spend the money, I get to keep a portion of it. If I spend it, the full amount goes to grow my business. I commit money to software, hardware, contractors, and licensed technology quite regularly. I try not to be foolish with this though. For example, I’m on the fence about upgrading my MSDN Operating Systems subscription to a full MSDN subscription. The full subscription is quite expensive, so I don’t know if I will extract that value out of it. Generally though, when a case can be made, I’m quick to invest back into my business. I work off of a maxed out MacBrook Pro and an Apple Thunderbolt display.

10. Do you have testers? No

Yes, I have testers, they’re called users. When I have enough changes to cut a release, I do so. Cobalt Strike changes make it to customers and trial users very quickly. When the product was under private development, I had a team of beta testers who acted as a stand-in for the customers I would eventually get. I still use this team as an advisory board.

Generally, my releases focus on one feature area making them easier to test. I don’t believe in monolithic point releases. Sometimes, I will introduce a bug or error, and if it’s a show stopper, I recut the release or create a special build for the customer that needs it. These cases are very rare.

In terms of user feedback, Cobalt Strike benefits greatly from building on Armitage. I try to keep the code for the two interfaces as close as possible. This gives Armitage’s significantly larger user base a chance to chime in on something that will improve both products.

I spend a lot of time testing the foundation I build on too. Cobalt Strike builds on the Metasploit Framework which is one of the fastest moving projects I’ve ever touched. Something that works today, could change, and ripple into my product in an unexpected way. I also have the pleasure of serving a user community that likes to use the latest development version of the Metasploit Framework. Talk about a catch-22!

I mitigate this scenario with unit tests. Before I cut a Cobalt Strike release, I update to the latest version of the Metasploit Framework. I run several Cortana-based unit tests to exercise the Metasploit Framework, automatically hack into hosts, and do basic post exploitation. The unit tests help me test the Metasploit Framework and exercise my interfaces to it. I also exercise a few basic scenarios. Occasionally, I stage a node on Amazon’s EC2 and conduct a penetration test against an enterprise network lab environment I host on a Shuttle PC.

Each release, I publish the version I tested with my release notes for customers to match their environment to. If someone chooses to update to the latest version of the framework’s code. That’s up to them. If they encounter a problem, they can downgrade to the last tested version.

Of course the best testing is real world use. I don’t do services / pen testing work now (development is a full time job!), so >exercises are the closest I come to real world use with my tools. When invited, I play red team in exercises and war games. These opportunities provide valuable feedback that helps me make my tools better. Generally, I spend March and April doing nothing but exercises. These months are so busy that I sometimes leave at the tail end of one exercise to catch a flight to the next one.

11. Do new candidates write code during their interview? Not Applicable

I’m not interviewing developers yet. When that time comes, I will not hire from a typical interview. I will carve out a small project, hire someone on a contract basis, and see how well we work together. This will answer the questions that an interview is ill-suited for. I’m okay risking money on a test project to see how well someone works out.

12. Do you do hallway usability testing? No

The Joel Test asks, how often do you observe a stranger trying to complete a random task through your software? I do usability testing, but I don’t do it by pulling people out of a hallway. I do it through classes. I teach two classes. I teach a free 4-hour workshop on Armitage and Cobalt Strike at various conferences. When hired to, I teach the Advanced Threat Tactics course.

I don’t get a lot of usability feedback from the 4-hour workshop. Generally there isn’t a lot of time for the labs and the labs are very scripted. However, I sometimes receive a gem of a suggestion or see where something could be made intuitive.

The usability goldmine for me is the Advanced Threat Tactics course. The class ends with a capstone exercise. The exercise takes place in my enterprise network lab environment. The environment is seeded with data and services to create the sense of a living world. I put my students into teams and I assign each team one of four goals. The students are expected to get a foothold in the lab environment and iterate through the network attack process until they accomplish their goal. Some of the goals are very open-ended (e.g., you’re a hacktivist, expose ACME for their wrong doing). The exercise is where I observe how well my toolset and its workflow clicks with users.

The host labels feature added to Cobalt Strike and Armitage last month came from a January run of the Advanced Threat Tactics course.

Advanced Threat Tactics is so effective for usability testing, that I also have a private invite-only session I run. I ran it twice last year and I plan to run it later this year. During this private session, I invite a mix of people. I always make sure I have two people who have never hacked before. I also invite some of the most experienced penetration testers and researchers I know. This is an intimidating group to “teach”, but it’s a useful way to see how different skill levels approach the toolset.

My goal is to make sure Cobalt Strike is valuable to experienced pen testers without forcing them to learn a new way to do everything. The fact that Cobalt Strike provides full access to the Metasploit Framework console helps greatly with this. I also watch to make sure that novice users are able to get things done with Cobalt Strike after a reasonable amount of instruction.

h1

Cobalt Strike Boxed Set comes to ShmooCon

February 13, 2013

It’s the middle of February, love is in the air, and… I’m busy preparing for my favorite hacker conference ShmooCon.

This year, for the second year in a row, Strategic Cyber LLC is sponsoring ShmooCon.

Last year, I had intended to launch Cobalt Strike. Except, it wasn’t called Cobalt Strike and someone else beat me on filing a trademark application on the original name–by about five days. Pure coincidence and I learned a lesson about retaining an IP lawyer early in the business formation process. Anyways…

Cobalt Strike is having its first year at ShmooCon and I plan to make it a good one. I’m unveiling a Limited Edition Boxed Set and giving away more of the popular Pen Test Lab DVDs. Read on…

Cobalt Strike Boxed Set

Limited Edition Boxed Set (Seriously)

If you haven’t bought Cobalt Strike yet, now is your opportunity. Leading up to and during ShmooCon, a few Limited Edition Boxed Sets are available. If you buy a Cobalt Strike license now through this weekend and present the key at the Cobalt Strike table, I will issue a boxed set to you (while supplies last).

These sets are beautiful. They include a professionally bound copy of the Cobalt Strike manual, a DVD with the Cobalt Strike software, and a Cobalt Strike sticker.

Most big software companies ask for a big check. In exchange, you get some 1s and 0s transmitted to you over the internet. When’s the last time someone bothered to put those 1s and 0s into a box? I rest my case.

Penetration Testing Lab DVD

If you haven’t tried Cobalt Strike yet, we have a slight problem. I don’t want you to buy without putting the software through its paces. I’m quite serious about this. If you want to try Cobalt Strike, stop by the table and get a Penetration Testing Lab DVD.

This DVD has everything you need to put Cobalt Strike through its paces from the comforts of your laptop. This DVD includes an attack virtual machine, a Cobalt Strike trial package, and two victim virtual machines with self-guided hacking labs. I think of it as a chemistry kit for learning hacking. You can follow the steps or invent your own experiments.

I plan to burn a few hundred of these. I’m doing it now. I will run out. I always do. If you want one, come get it as early into the conference as you can.

Come say Hi!

I work the Strategic Cyber LLC table the entire time. If you have questions about Armitage or Cobalt Strike or if you’d like to see a demonstration, come on by. I’m looking forward to seeing you at ShmooCon!

h1

Advanced Threat Tactics Training

October 11, 2012

I share a lot from my experiences playing on exercise red teams. I talk about the tactics to collaborate, persist on systems, and challenge network defenders in an artificial environment. Armitage was built for this role.

I speak little about my experience working as a penetration tester. I used to work for a security consulting firm providing “red team services to a DoD customer”. My job was threat emulation. My partner and I would plan and execute actions over a long period of time. All of our activities were double-blind. To protect our work, my boss would meet with our contact in a public area set aside for smokers, hand over our plan, and gain approval to execute at that time.

Last October, I was asked by the LASCON organizers in Austin, TX to teach a one day course at their conference. I opted to teach a course on threat emulation. This is when I wrote Advanced Threat Tactics with Armitage. The course briefly introduced Armitage and the Metasploit Framework. A lot of time was spent on how to get a foothold using tactics these tools don’t directly support. The lecture portion ended with two talks on post-exploitation and how to move inside of a network.

The capabilities missing from our tools made up the Advanced Threat Tactics portion of the course. In these three lectures and labs, I taught:

  • All attacks start with reconnaissance. How do you perform reconnaissance before a targeted phishing campaign? I introduced the concept of a system profiler and how to build one.
  • What do you do if client-side applications are patched? Think like a criminal–you care about the end and not the means. Here, I introduced the idea of hacking with features. It’s important to know how to look at an attack surface and recognize opportunities to get code execution. Sometimes the simple ways work best.
  • Once you have an attack, you need to make sure it passes anti-virus. You also need to think about command and control and how you will go through a restrictive firewall. In this portion of the lecture, I introduced students to these ideas and tools available (at the time) to help them with this process.
  • Once you have your attack put together, it’s important to package it in a convincing way and get it to your target. Here I taught how to send a pixel perfect phishing message. I made students do these steps by hand. Nothing says fun quite like stripping headers from a message in a text editor and then typing SMTP commands by hand to exchange email with the target’s mail server.

My course helped students think creatively about how to get a foothold in a network and use that foothold to achieve a goal. The missing capabilities in the penetration tester’s toolbox have become the road map for Cobalt Strike.

Fast forward one year later. I’m teaching a two-day Advanced Threat Tactics course at OWASP AppSec USA. The heart of the course is still the same. It’s a two-day opportunity to learn how to think creatively about the hacking process and execute the tactics through several guided labs. The two-day time frame allows me to add a lab and lecture on evading defenses. I have also expanded the post-exploitation and maneuver lectures.

The best part of the course is the exercise though. The course ends with an exercise that lasts several hours. You have the opportunity to work with a team and assume the role of different threat actors attacking a simulated enterprise.

  • As a hacktivist, you’ll break into the ACME corporation, discover their dirty secrets, deface their website, and publish their email spool
  • As an actor interested in economic espionage, you will gain access to the ACME corporation, find the source code to their secret project, and steal it.
  • And, as a nation we face risk of sabotage through cyberspace. As this threat, you’ll find and manipulate a control system that leads to the destruction of a nuclear reactor.

I wrote this course for a broad audience to include novice to experienced penetration testers and network defenders. I teach the Advanced Threat Tactics by request to organizations who have the resources for 12-15 students. For individuals, the best opportunity is to attend Advanced Threat Tactics at a conference. The next run of Advanced Threat Tactics is at AppSec USA in Austin, TX. The course is Tuesday, 10/23/12 and Wednesday, 10/24/12. If you’d like to sign up, there’s still space available.

h1

Go Down the Stack Young Man – Story of a Bug

August 8, 2012

Last week, I released a big improvement to the responsiveness of the Armitage team server on congested networks. This particular case of poor responsiveness was extremely difficult to reproduce. Despite my continued attempts to optimize around the real cause, I failed time and again to nail it. I thought I solved the problem, until I received a “bug report” indicating otherwise. I’m certain I figured it out this time. Here’s the back story:

Armitage is a collaborative hacking tool built on the Metasploit Framework. The collaborative piece is made possible by a team server. This server acts as a proxy between the remote Armitage clients and the one Metasploit Framework server. Through this proxy, I’m able to deconflict multiple clients interacting with a session and offer additional APIs to my clients.

Armitage Collaboration Architecture (New)

As long as I’ve had a team server, I noticed clients connecting from Windows 7 clients always felt slow *. As an experiment, I opted to disable Nagle’s algorithm. Nagle’s algorithm is built into most TCP stacks. It reduces network congestion by holding onto small packets and attempting to combine them into one larger packet. For protocols that generate small packets naturally (e.g., telnet), Nagle’s algorithm may add unnecessary latency. Most socket APIs include a means to disable it.

Disabling Nagle’s algorithm resulted in a big responsiveness boost on Windows 7. Linux and MacOS X clients connected to a team server were snappier too. I noticed that my local unit tests completed two minutes faster with Nagle’s disabled too.

I was pleased with this change until I received a “report” three weeks ago. Some folks were using Armitage during an exercise and… apparently the collaboration piece was very slow for them.

I was about ready to tear my hair out when I received this report. Performance is the one thing I put the most time into. I went back and forth with the user to understand their configuration and environment. He had everything setup as I would have requested it.

This report especially frustrated me because of the amount of testing I do. About once per quarter, I will connect 12 Armitage clients to a node on Amazon’s Elastic Computing Cloud. I will then populate the database with about 5,000 hosts worth of data. From this point, I then proceed to carry out a simulated external engagement against my local test lab.

Here’s a screencast demonstrating this particular test:

So, what could the problem be?

I opted to do, what I should have done a long time ago…  I ran tcpdump to better understand how the team server looked on the network.

tcpdump -i eth2 | grep 55553 | grep -v "length 0"

With this running, I noticed that Armitage placed many small packets on the network, about 20-24 bytes consistently. I expected this because I disabled Nagle’s algorithm. Anything small would go out immediately

15:09:19.132609 IP 192.168.95.241.60153 > 192.168.95.241.55553: Flags [P.], seq 7961:7986, ack 5943, win 770, options [nop,nop,TS val 15794750 ecr 15794750], length 25

15:09:19.132714 IP 192.168.95.241.60153 > 192.168.95.241.55553: Flags [P.], seq 7986:8008, ack 5943, win 770, options [nop,nop,TS val 15794750 ecr 15794750], length 22

I then enabled Nagle’s algorithm and watched the same traffic dump. The result? All of the packets were the same size as before. With Nagle’s enabled, I was paying the penalty of having small packets with the additional latency of Nagle’s holding on to them. Great.

I scratched my head and decided to dig deeper into my code. None of this seemed right. As I dug through, I learned that there is no buffer between my SSL code and the code that serializes an object and writes it to a socket. The team server was serializing Java objects and writing them to the socket one byte at a time, rather than sending them as one byte buffer.

Doh!

I updated my code to write serialized objects to a buffer before sending them to a socket. This reduced the number of packets by a factor of 10-20. I also reenabled Nagle’s algorithm.

14:34:05.179461 IP 192.168.95.241.52289 > 192.168.95.241.55553: Flags [P.], seq 978092:978547, ack 804839, win 770, options [nop,nop,TS val 15266262 ecr 15266137], length 455

14:34:05.180174 IP 192.168.95.241.55553 > 192.168.95.241.52289: Flags [P.], seq 804839:805218, ack 978547, win 770, options [nop,nop,TS val 15266262 ecr 15266262], length 379

At this point, I tested on Windows 7 and noticed performance was good to go. I also ran my unit tests and noticed no performance change.

Here’s likely what happened. I play in a lot of exercises with Armitage. Exercise networks are usually congested. There’s a lot of activity happening. All of the team clients flooding the network with small packets probably made the congestion much worse.

I’m embarrassed that this problem slipped past my radar, but I’m happy that it’s finally fixed.

Lesson learned: when it comes to performance, I can’t treat the network as an invisible abstraction that delivers my data. I have to give my interaction with the network as much attention as I give to optimizing my software.

* Note: Armitage clients used to connect to both the Metasploit Framework and a team server. Only packets sent to the team server were victim to this problem. In May 2012, I changed Armitage’s collaboration setup to proxy everything through the team server. This made the problem noticeable and forced me to start looking at it. This is when I made the change to disable Nagle’s algorithm. 

Follow

Get every new post delivered to your Inbox.

Join 13,803 other followers