Pentester Syndrome

Making things appear worse than they are

by Alex Haynes, Information Security Manager, CDL

If you work in Enterprise security or in any technical domain you may have at one point had the pleasure of reading a report from a recent penetration test (otherwise known as ‘pentesting’). These tests are usually conducted against websites or exposed infrastructure to simulate a real-life attacker and are genuinely useful at evaluating your security posture at a particular point in time. That being said, there are also many misinterpretations of pentesting as an activity and pentesting results and reports that lead to a skewed perspective on risk in the Enterprise as a result of pentesting itself. I’ll cover off the main downsides to pentesting and how to compensate for these in an Enterprise environment so you don’t fall victim to ‘pentester syndrome’ yourself and get the best value out of your pentests.

Pentests are frozen in time

One of the major downsides of pentesting today is that they don’t match the speed of development of modern applications. Most companies pentest annually but rarely will you now find an application or website that is only updated once a year. In today’s environments, applications and websites are updated frequently, sometimes once a day, and occasionally even more than that. Like I mentioned earlier, a pentest is a snapshot of your security posture at a particular point in time. That’s it. Once you’ve updated your website or application, those findings are potentially out of date, as you may have already introduced new configuration changes into your environment, which means potential new vulnerabilities.

Hear ye, my pentest begins today

As a pentest is supposed to be a simulation of a real-life attack, it’s always interesting to note the number of companies I’ve seen announce to all and sundry that they are beginning a pentest. This eliminates one of the primary advantages of the test – simulating a real-life attack. Telling everyone in advance of a test means everyone will discount logs, reports, and alerts generated by that test, effectively robbing you of a simulated incident response scenario. It’s a surprising discovery that once you stop announcing your pentests, no one detects them. If you cannot even detect a legitimate pentest with your current enterprise security setup, then it’s unlikely you’ll be able to detect a malicious attacker doing the same.

Time-limited

Pentesters don’t have the luxury of time in any engagement. A website pentest may go from 3-5 days, plus an extra day or two to write up the report. Because of this limitation, pentesters will gravitate towards low-hanging fruit – vulnerabilities that are easy to find, especially with automated tools. Vulnerabilities that required complex custom scripting or time investment won’t be looked at in depth simply because they don’t have time and this is completely normal. Attackers today don’t have any such time constraints so can spend as long as they like focusing on a single asset, dredging up obscure but nonetheless critical vulnerabilities that would require more time investment than a pentest can provide.

Pentester syndrome

Pentesting companies often have to prove their value to their customers – after all, they are paid lucrative rates to provide a niche service. Unfortunately for many, the ‘value’ derived from the point of view of the customer has been degraded to how many vulnerabilities were found. After all, if a company uses two different pentesting companies in a row, but one of them found more of them than the other company – are they not better?

This is where ‘pentester syndrome’ occurs – making things worse than they appear. I’ve had the pleasure (or misfortune – depending on your point of view) of having written or read hundreds of pentesting reports in my career. When a pentester doesn’t find any vulnerabilities of note, it becomes the norm to ‘talk up’ issues that can be classed as ‘informational’ to vulnerability status. This is damaging for many reasons, the first being, that you are being given a false assessment of your security posture. If you aren’t experienced or technical enough to translate these findings into quantifiable risk, you will then spend resource and time blindly chasing down remediations for these ‘vulnerabilities’ that have absolutely no bearing on your security posture and will never cause you any trouble. I’ve listed the kind of findings you don’t really need to spend too much resource on unless you need a very hardened security posture.

What kind of findings don’t I need to be worried about?

So a bit of a caveat: I don’t know what your risk profile is, so these might not all be applicable to you. What I can tell you, is that no one has suffered breaches due to the below by virtue of the fact that they only aid an attacker in very specific contrived situations, and then only for a specific kind of vulnerability or require a man-in-the-middle positioning which is, despite what you may hear, difficult to achieve.

No HSTS headers, any kind of SSL/TLS issues

A common one that finds itself onto pentest reports. If you have HSTS then client-side certificate errors will always be fatal (the user can’t click through the warnings) and that side can only be negotiated in HTTPS. What this means if someone tries to man-in-the-middle you, well they can’t. This also means it’s hardly likely to affect many users since a man-in-the-middle position for an attacker is hard to get. You can lump all SSL/TLS issues into this bucket, so things like weak ciphers and outdated versions of SSL are here. As the attack vector is identical (you need man-in-the-middle) these are never likely to affect a great number of individuals at once.

Lack of Secure/HTTPOnly flags on cookies

This one’s another common culprit. The lack of the HTTPOnly flag on cookies will only ever help an attacker who’s found a cross-site scripting vulnerability and is only leveraging it to extract the cookie itself. As now we have things like the browser exploitation framework (BEEF) and many other vectors to exploit cross-site scripting apart from cookie theft, it is of limited use and is effective ‘hardening’ – it’s even less useful if they find no cross-site scripting vulnerabilities on your site. The ‘secure’ cookie only implies cookies can only be sent over HTTPS, so it also falls into ‘hardening’ like the HSTS issues above

Use of a vulnerable library

I see this one a lot – usually affecting things like out of date third party libraries in products (things like jquery, reacts, etc.). A vulnerable library only indicates a method in a specific library is vulnerable if it is used and then only in a specific way. If it’s not used, no way you can be vulnerable. If you see this in a pentesting report, just ask for a proof of concept, and the finding usually gets taken off the report.

And many more

Lack of x-frame-options header, user or account ID enumeration, lack of CAPTCHA, lack of password complexity requirements, inadequate SPF/DMARC/DKIM configs are all things which may appear but which will rarely cause you any trouble since they are not ‘vulnerabilities’.

Getting the most out of your pentests

So now that we’ve exposed the issues with pentesting today, how can we make them better. Some of them are down to pentesting being the incorrect tool for the job but a lot of it can be improved by engaging with your pentester provider a bit more.

Challenge them

Even if you are not a technical individual or feel out of your depth when people start talking about XXE’s and Cross-origin resource sharing, you can always ask the pentester for proof of concept, especially if he’s written ‘X is vulnerable because of Y’. Pentesters will provide this by default for major vulnerabilities but may sometimes gloss over this part. Feel free to challenge the reports and have them modified. If they can’t provide proof of concept, then you can have it removed from the report.

Don’t announce them

To get the ‘simulation’ get prior approval to stop announcing to the world when pentests are occurring. The data you gather will be more valuable and if your incident response procedures and operational teams detect them, that data can be fed back into a virtuous cycle of improvement. This is just as valuble if they aren’t detected. What failed? Why?

Match them to major functionality improvements and updates

When your application or website has new functionality added or improved, try and cycle the pentest to coincide right after this. Now, this will largely be budget dependent. Alternatively, if you have frequent functionality updates, consider switching to crowdsourced security to replace some or all of your pentests, as this will mitigate the main weakness of pentests – that they occur at a single point in time.

Cycle pentesters

One pentester will see vulnerabilities that a previous pentester missed, and this is entirely normal. To remedy this the best practice is to cycle your pentesters, be it by asking the vendor you use to send a different individual each time, or by just picking a different vendor each time. As offensive security skills are at a premium this will sometimes backfire and you’ll end up with the same pentester at a different company (this happens more often than you think). Switching to a crowdsourced security model remedies this entirely, as your pool of testers inflates so you have dozens if not hundreds of people testing your application.

Just another tool

Finally, pentesting is just another tool in your arsenal. You cannot get by with just a pentest, nor can you really get by without one. They are useful tools but are a complement to your arsenal, not the only thing. Whether you preach holistic security or ascribe to a defense in depth model then pentesting is just another aspect of that model. Just make sure you get the most out of it.

About the Author
Alex Haynes is Information Security Manager at CDL. He has a background in offensive security and is credited for discovering vulnerabilities in products by Microsoft, Adobe, Pinterest, Amazon Web Services, IBM and many more. He is a former top 10 ranked researcher on Bugcrowd – a vulnerability disclosure platform with over 200 vulnerabilities to his name. Alex can be reached online at [email protected]