By Michael Morris, Director of Global Technologies Alliances and Business Development, Endace
A common challenge for security analysts, network operations and application teams is lacking the right data to troubleshoot security or performance issues quickly and conclusively. Typically, analysts are overloaded with alerts. They need to find ways to reduce the noise and identify the root cause of issues more efficiently.
Investigations often start with log data or network metadata. But this data often doesn’t provide enough detail to see exactly what happened or only provides a limited view of activity for a specific user, IP address, application or server. Log data doesn’t provide broader insight into related activity and also may potentially be manipulated by malicious actors covering their tracks. Network metadata can provide broader insight but lacks the detail needed for full event reconstruction.
This lack of concrete, definitive evidence slows issue investigation, resulting in stressed employees and unexamined issues that represent an unknown risk to the organization. Recording full packet data offers a potential solution. Packets hold clues that can be vital for investigating and resolving issues, providing comprehensive evidence across an entire IT environment that can definitively identify lateral movement or command and control traffic, and enable the accurate reconstruction of exfiltrated data.
With today’s regulatory environment increasingly requiring mandatory breach notification, relying on log data and metadata alone for investigating security issues doesn’t cut it any longer. Packet capture is the only way to be sure you have the data necessary to understand the full depth and breadth of a breach.
Packet capture solutions can be integrated with SIEMS, IDS, NPMs/APMs and AI technologies (such as Cisco, IBM, Palo Alto Networks, Splunk and many others) making it easy to locate packets of interest. SecOps, NetOps or DevOps analysts can quickly pivot from alerts in their tools directly to the relevant packet data with a single click, making locating suspect packets inside petabytes of network traffic easy. Once the packets have been located, built-in packet decodes in tools such as Wireshark provide a wealth of useful information for analysts.
Let’s take a look at a typical investigation example. In this case, it’s an application performance issue that’s being investigated, but the same workflow can be used to investigate security alerts, hunt for threats or reconstruct exfiltrated data from a security breach.
The Case of the Retransmissions
Betty DuBois is an industry-renowned SharkFest educator and network investigator who has created a video that demonstrates how easy investigations can be if you have access to full packet data. Let’s walk through how, in just a couple of minutes, you can resolve two specific issues to address in an application performance degradation that generated tickets and alarms due to due to customer complaints.
She starts by zooming in on recorded traffic using the time, application, and client IP of impacted users based on reported event tickets. (Time 00:36)
Examining the filtered traffic uncovers unusually high retransmission counts for MS OneDrive traffic – which could be part of the cause. Retransmissions are high on a single subnet, indicating a potential issue on the service provider’s side. (Time:58)
Drilling into the packet details uncovers lengthy delays for both standard server response packets and retransmissions (Time 2:33)
Further analysis quickly identifies two issues: the service provider’s server is taking too long to retransmit packets and way too long to start sending data to the client. (Time 3:45). In just a few clicks, packet data has enabled Betty to precisely pinpoint the root causes of the issue and identify who is responsible for resolving them – in this case, the service provider.
This example demonstrates how access to full packet data can accelerate the process of isolating, investigating and remediating IT issues and security threats with 100% confidence. For the full video, visit Packet Detectives Episode 1: The Case of the Retransmissions: (https://blog.endace.com/2019/12/17/demystifying-network-investigations-packet-data-part-1/).
About the Author
Michael Morris is the director of global technology alliances and business development at Endace. He has more than 20 years of experience in software and hardware solutions for infrastructure and network management and security. Contact: www.Endace.com, [email protected], @endace.
