Black box penetration testing: realistic or redundant?

This post was written by Jack, one of our Cyber Security Consultants.

What is Black Box Testing?

Black box testing, in the context of penetration testing, is a method of vulnerability assessment whereby the tester attacks a target with the same level of knowledge and permissions as a genuine malicious actor might have. Typically, this means the test begins with the tester receiving no information on internal workings, and no credentials or permissions.

When considering and researching penetration testing, it is common for IT professionals to believe that the most effective test would be one that more accurately mirrors a real-life scenario, concluding that, for this reason, black box testing is the right method for them.

While black box testing can offer valuable insight by using the same methodologies that a real attacker might, it is important to understand that the expected result from such a test is unlikely to include details of the potentially-critical vulnerabilities that could be detected during a white box test.

What is White Box testing?

Conversely, white box testing involves the provisioning of a tester with the very knowledge and permissions denied from black box testers. This provides the assessor with the ability to probe the target for vulnerabilities present within the application’s core; an area hidden from a black box tester.

Performing this type of testing allows organisations to more accurately understand the risks present in their applications and services by revealing vulnerabilities that are presented only to end users with permissions greater than nil, and real attackers who have managed to crack the outermost layer of security.

Crucially, white box testing is not exclusive of black box testing; meaning a white box assessment includes everything involved with a black box test; but also includes testing of otherwise unreachable areas.

Benefits of Black Box Testing

When commissioning testing to be performed by an external organisation, black box testing may be considered safer as it does not require giving elevated permissions or access to a user outside of the business. Additionally, there is the fear that white box testing often reveals more critical vulnerabilities, meaning that the external tester could, in theory, either exploit these to the detriment of the business or sell this threat intelligence on the black market to those that might seek to do harm to the organisation.

Typically, as this test type involves a smaller scope, black box tests take fewer days to complete and, for this reason, are ordinarily cheaper.

Benefits of White Box Testing

In the six most recent white box penetration tests performed by Securious, 52% of all vulnerabilities found would have been revealed using black box testing methods only. This means that 48% of the discovered vulnerabilities would not have been detected in a black box scenario.

The average criticality of the vulnerabilities we found through white box testing techniques is 37% greater than otherwise, meaning that the additional vulnerabilities uncovered as a result of a white box penetration test are also the vulnerabilities that pose a greater threat to the confidentiality, integrity or availability of critical business data and infrastructure.

Our clients communicate to us that interest in white box testing typically stems from a need for access control / user privilege testing, but it is also clear that it is important to them to understand the threats presented to them by their users and/or employees. These aspects are not typically tested in a black box assessment.

Conclusion

The argument of White Box vs Black Box testing can draw many parallels to the argument of ‘Security: built in vs bolted on’. The main difference in both cases being the method of risk reduction. When considering ‘risk’ as a function of likelihood and impact, we can reduce risk in two distinct manners: in one, by performing security ‘hardening’, increasing the level of security on the applications perimeter and thereby decreasing the likelihood that an attack will be successful; and in another by increasing security ‘resiliency’, in doing so reducing the impact of an attack and thereby the damage that a successful attack may cause.

Black box testing effectively tests only an application’s security ‘hardness’ and, notwithstanding a massive security hole, has little opportunity to test anything else. White Box, however, is able to test every aspect of an application’s function, ensuring all forms and input fields are not vulnerable to code injections, reflections or storage.

Before undertaking penetration testing, it is important to understand what the goals of the test are, what needs testing (hardening vs resiliency & hardening) and consider what actionable, useful insight might look like to you.