Phone Bill a CC NC ND image from Nikita Kashner's Flickr stream

By Darren Anstee of Arbor Networks

Why is it a good idea to us flow information?

• You don’t need to invest in new equipment to get flow information
• It can be used to detect malware infected hosts, DDoS, zero-day exploits, attack and abuse
• Network flows information is generated regardless if there was symmetric or a-symmetic routing

Network flow information is like a phone bill, you cannot tell what has been said, but you can use it to prove who talked to who.

So what does a flow record contain?

• Source IP address
• Destination IP address
• Source port
• Destination port
• Input IfIndex
• Protocol
• Type of Service
• packet count
• Byte count
• First packet time
• Last packet time
• Output ifIndex
• Etc…

Flow information allows you to monitor large geographically dispersed networks.

So how can flow information be used for security purposes?

Flow information helps you understand how you network normally behaves. Unusual behavior might indicate DDoS attacks of malware infections.

One could look at the flow information manually, but it does make more sense to install a collection and analysis system. These systems often give the benefit of providing historical data that can help us understand current data and allow us to use this information for forensic purposes.

There are a lot of open source and commercial flow collection and analysis systems available.

Next Darren showed demonstrations of how flow information can be used.

First example is how to detect malware infected hosts in an enterprise environment.

How? One of two ways:

• Looking for abnormal behavior
• Looking for known bad behavior, e.g. communication to known Command can Control servers

So what is typical unusual behavior?

• Unusual outbound SMTP
• Off-net DNS queries
• Scan detection
• Unusual outbound behavior
• etc.

Finding more then one anomalies increases the likelihood of these systems being infected.

One of the bonuses of flow information is that routers and switches still generate flow information even if firewalls drop the traffic.

Darren showed us how tools like nfdump can be used to detect systems with various abnormal behavior such as connecting to external mail servers or DNS servers too much or generating classic DDoS attacks.

Naturally you can also use flow information to detect DDoS attacks.

How do tools, like those Arbor makes, detect DDoS attacks?

• Baseline detection and baseline deviation
• Misuse flow detection (SYN-flood, UDP-flood)
• Detect bursts in the network
• Use thresholds

Why would you use flow information over firewall logs? Routers and switches are much more omnipresent and switches and routers do generate flows even if the firewall drops the traffic.

The slides for this talk with links to whitepapers and open source tools can be downloaded from the first.org website.

But what if you have an issue with NFS and you need a network dump? 

In ESXi tipically you don’t have a local datastore where you can write files from the network dump and your datastore over NFS is not availabe!

Before running into the Data Centre and stick a USB disk or even better a SCSI disk you might want to try this.

One trick I used that worked out pretty well for me, with a little help of my a linux machine, is to send the tcpdump output to a FIFO and from a remote host (might be a VM in a different ESXi host) over SSH cat the FIFO to a local file.

How To:
On the ESXi host logon via SSH as root and create a named pipe:

root@yourESXihost# mkfifo /tmp/pipe.dmp

and from a remote linux machine launch the following:

you@yourlinuxhost > ssh root@youresxihost "cat /tmp/pipe.dmp" > capture-for-wireshark.cap

Now from a new ssh session to ESXi as root lauch

root@yourESXihost# tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp

OR even better from the remote machine:

you@yourlinuxhost > ssh root@youresxihost "tcpdump-uw -n -s 1524 -i vmk# -w /tmp/pipe.dmp"
(replace the # with the proper vmk port number)

Reproduce your issue and when you finished just hit  “Cotrol+C” to stop the network dump and the cat.
Now you can open your file directly in wireshark (that’s what I use at least!)

This little trick of course can be used to troubleshoot network problems in a VM as well, dumping the traffic from a VMK# nic for the entire dvPortGroup. You just need to make sure that the the VM’s vNIC and the vmk# nic are connected to the same dvPortGroup and you must remember to allow promiscuous mode (not allowed by default)

Good Luck!

Please note: your network can be very chatty so the file can grow very fast and/or your ESXi host might not like the tcpdump so use it at your own risk and only if you really know what you are doing!

Dmytri’s talk compares peer to peer and client server with communism and captalism. It is important that these terms refer to their original meanings:

• Communism is a theoretical society with no classes and no state.
• Capitalism refers to a society in which the owners of Capital are able to abstain from direct-production by appropriating the products of workers who employ their property in production.

Mesh networks can be compared to communism:

• Participants can interact directly
• No Toll Gate, No Prices

Star networks can be compared to capitalism:

• The Capitalist is the Operator
• Roles and credentials create classes
• Mediation is needed to charge a price

Historically the internet itself was created by universities and could be more compared to communism. The big online services (CompuServe, AOL, etc) have a capitalistic background.

The enclosure of the internet shows a number of phases:

Enclosure 1.0: Capitalism bought all ISP, so ISP’s are now run by big corporations.

Enclosure 2.0: The return of Client-Server

The start topology is is being re-imposed on the Internet:

• Usenet being replaced by Webforums
• Email being replaced by Social Medea
• IRC being replaced by Twitter, etc

In fact P2P is being criminalized: there is not positive media coverage for it.

Capitalist will Not Fund P2P

The first part of his presentation explained why routers are interesting targets (they are in the core), but also why routers are not actually exploited that much. One of the reasons is that the attack surface of router is quite small because routers don’t expose that much services to a truly remote attacker and are rarely used as clients.

The exception to the rule is “cisco-sa-20070124-crafted-ip-option” which is a remotely exploitable bug that causes a stack overflow on the router. Since “nobody ever updates router software” this vulnerability is still very much alive.

But routers need to support more and more, like IPv6, VoIP, XML configuration interface, luckily most services off.

Writing exploits for Cisco IOS is hard because it is not a real OS, but a single ELF binary. It is not based on a real OS we know hoe to exploit. Its only option to recover from a critical fault is a full reboot.

Another thing that makes exploitation hard is the memory layout. It is different from each single IOS version that it out there, and there are quite a few, currently there are over 270,000 different IOS images known by Cisco and you cannot get the version number remotely.

Best bet for getting a reliable return address for router exploitation is Rommon, the routers bios which loads the IOS and then remains in memory. It is at a fix address and there are big pools of the same versions present on the internet.

Unlike his talk at BlackHat Felix actually showed how the crafted ip option exploit can be used to get working reliable exploit. But since IOS is not an OS you need to get away with it without killing the router. If the stack is not completely overwritten, the return registers remain in tack and thus can be used to reliably return. His method has one drawback, in order for it to work, you need to know the version, but it is not remotely identifiable.

As an alternative there are code similarities in IOS images, but this still has problems.

Felix also made progress on shell code, he showed code that would cause the password evaluation function to always return true.

How do you protect your router?
•    Have faith.
•    Don’t allow people to talk to your router
•    Protect your routing protocols
•    Don’t run services on routers
•    Treat your service cards as the linux machines they are

Running Rancid helps, modification of the data structures show up here.

Turn crash dumping on, this will make sure you keep evidence of any attacks.