High Performance Application Security Testingv1.0

1. Summary

Data, web, and application security have evolved dramatically over the past few years. Just as new threats abound, the architecture of applications—how we build and deploy them—has changed. We’ve traded monolithic applications for microservices running in containers and communicating via application programming interfaces (APIs)—and all of it deployed through automated continuous integration/continuous deployment (CI/CD) pipelines. The frameworks we have established to build and deploy applications are optimized for time to market—yet security remains of utmost importance.

The challenge of securing and innovating is profound and requires a lightweight and integrated security solution that won’t impede performance and delivery. For example, DevOps teams need security controls that work across distributed environments without invasively slowing down or burdening the release cycle. The maturation of these controls and processes ultimately transitions into the realm of DevSecOps, where security is built into the CI/CD pipeline.

The multitude of deployed apps, APIs, and microservices produces a constant flow of communication and data among applications that require active management—both internal and external. Apps themselves can vary greatly in the protocols, allowed methods, authorization/authentication schemes, and usage patterns. Perhaps most important, IT departments need granular control over the entire application ecosystem to prevent security breaches and attacks, be they man-in-the-middle, distributed denial of service, or script/code/SQL injection attacks.

While security is of utmost importance, the pace of modern business demands high performance, and this is especially true in application- and microservice-enabled enterprises. The conventional approach—deploying a perimeter Web Application Firewall (WAF) to protect applications by filtering and monitoring traffic between the app and the internet—is no longer enough. Even internal communication between apps and microservices on the trusted corporate network can be compromised and must be addressed. A defense-in-depth strategy is needed with multiple WAFs.

This report focuses on web application security mechanisms deployed in the cloud and closer to your apps. The cloud enables enterprises to differentiate and innovate with microservices at a rapid pace and allows microservice endpoints to be cloned and scaled in a matter of minutes. The cloud also offers elastic scalability compared to on-premises deployments, enabling faster server deployment and application development and less costly compute. However, the cloud is just as vulnerable, if not more so, to attacks and breaches as on-premises APIs and apps are.

Our focus is specifically on approaches to securing apps, APIs, and microservices that are tuned for high performance and availability. We define “high performance” as companies that experience workloads of more than 1,000 transactions per second (tps) and require a maximum latency below 30 milliseconds across the landscape.

For many organizations, performance is a big deal—they need to ensure secured transactions at rates that keep pace with the speed of their business. A WAF or application security solution cannot be a performance bottleneck. Many of these companies seek a solution that can load balance across redundant microservices and enable high transaction volumes.

The numbers add up. If a business experiences 1,000 transactions per second, that translates into 3 billion API calls in a month. And it is not uncommon for large companies with high-end traffic levels to experience 10 billion or more API calls in a 30-day period. Make no mistake, performance is a critical factor when choosing an API security solution.

In this report, we performance tested security mechanisms on NGINX, AWS, and Azure: ModSecurity, NGINX App Protect WAF, AWS Web Application Firewall (WAF), and Azure WAF. This last product was tested as a fully managed security offering. Note, ModSecurity is commercially distributed by NGINX and will be referred to as “ModSecurity” throughout the rest of this report.

In our benchmarks, NGINX App Protect WAF outperformed ModSecurity at all tested attack rates. NGINX App Protect WAF produced 4.7x lower latency than NGINX running ModSecurity at the 99^th percentile at 1,000 transactions per second (tps) on the 5% bad request test. In our tests, the latencies for App Protect and ModSecurity diverged at the higher percentiles, becoming pronounced at the 95^th percentile and above.

For fully managed offerings, NGINX App Protect WAF produced 128x lower latency than AWS WAF at 1,000 tps on the 5% bad request test at the 99th percentile. Also, NGINX App Protect WAF produced 82x lower latency than Azure WAF at 1,000 tps on the 5% bad request test at the 99th percentile. Since AWS and Azure’s WAF is fully managed, we do not know what underlying compute resources are working behind the scenes, which makes an apples-to-apples performance comparison difficult. Once again, latency differences were minimal until the 90^th percentile, with a significant difference witnessed at the 99^th percentile and above.

On a single small 2 CPU and 5.25GB of RAM EC2 instance, we captured the maximum transaction throughput achieved with 100% success (no 5xx or 429 errors) and less than 30ms maximum latency. NGINX App Protect WAF produced about 5,000 requests per second, compared to only 2,000 requests per second with ModSecurity. App Protect provides the same level of throughput as hitting the API directly without a WAF in between.

Testing hardware and software in the cloud is very challenging. Configurations may favor one vendor over another in feature availability, virtual machine processor generations, memory amounts, storage configurations for optimal input/output, network latencies, software and operating system versions, and the workload itself. Even more challenging is testing the fully managed, as-a-service offerings where the underlying configurations (processing power, memory, networking, and the like) are unknown. Our testing demonstrates a narrow slice of potential configurations and workloads.

As the sponsor of the report, NGINX opted for a default NGINX installation and API gateway configuration out of the box—the solution was not tuned or altered for performance. GigaOm selected identical hardware configurations for both App Protect and ModSecurity. The fully managed AWS WAF and Azure WAF were used “as-is,” since, by virtue of being fully managed, we have no access, visibility, or control over their infrastructure.

We leave the issue of fairness for the reader to determine. We strongly encourage you to look past marketing messages and discern for yourself what is of value. We hope this report is informative and helpful in uncovering some of the challenges and nuances of security architecture selection.

We have provided enough information in the report for anyone to reproduce this test. You are encouraged to compile your own representative workloads and test compatible configurations applicable to your requirements.