Table of Contents
1. Executive Summary
This GigaOm benchmark report was commissioned by Microsoft.
Transactional applications supported by databases are used by organizations to run their daily operations. By enabling the recording and tracking of transactions, these databases ensure that all business is carried out effectively and efficiently.
Database engines built and tuned for high-throughput transactions handle these transactional workloads. The creation of transactional data is accelerating quickly due to the 24/7 nature and variety of sources of transactions. The data is produced in high volumes and at a high rate, regardless of whether it is used to process financial transactions, analyze video game activities, or track health conditions.
As a result, transaction throughput is essential for read- and write-intensive cloud applications and must be sustained despite database latency, network throughput, API/microservice calls, and other processes.
The evolution of PostgreSQL from conventional, single-node database deployments to distributed implementations of PostgreSQL has elevated its capabilities in the areas of scalability, fault tolerance, and performance. Yet self-managed implementations can be difficult to maintain. Many enterprises therefore turn to fully managed, cloud-distributed implementations of PostgreSQL to leverage the advantages of a distributed database without the headaches of management and maintenance.
In our experience, a fully-managed distributed PostgreSQL implementation should be capable of:
- Global distribution: able to replicate your data across multiple regions to provide low-latency access for users across the world
- High availability and durability: with at least 99.9% uptime SLA and automatic failover capabilities
- Scalability without disruption: scale out your database as needed without downtime to your application
- Integration with other cloud services: such as analytics, streaming data, and machine learning
- Developer productivity: comes with a variety of tools and APIs, such as SDKs for popular languages, a REST API, and support for protocols like JDBC and ODBC
- Cost-effective: offering a range of pricing tiers and pay-for-what-you-use billing to best fit your budget
To evaluate distributed databases, we conducted a GigaOm Transactional Field Test derived from the industry-standard TPC Benchmark™ C (TPC-C). We compared fully-managed as-a-service offerings of cloud-distributed databases:
- Azure Cosmos DB for PostgreSQL
- CockroachDB Dedicated
- YugabyteDB Managed
Our monthly cost for similar CPU and memory configurations were as follows: Cosmos DB for PostgreSQL, $25,484; Managed YugabyteDB (YugabyteDB), $42,067; and Managed CockroachDB (CockroachDB), $45,384. In terms of new orders per minute (NOPM) performance—analogous to TpmC—the results measured for TPC-C were 1.05 million for Cosmos DB for PostgreSQL, 178,000 for CockroachDB, and 136,000 for YugabyteDB.
We also showed that the CockroachDB and YugabyteDB average CPU utilization rose substantially as the number of TPC-C warehouses increased—from less than 20% at 1,000 warehouses to more than 50% at 10,000 warehouses to 80% or more at 20,000 warehouses. This indicates that NOPM performance is unlikely to improve significantly, even with higher warehouse counts, as available CPU becomes a bottleneck.
Finally, the three-year total system price-performance measurements showed Cosmos DB for PostgreSQL to be the most cost-effective option at $0.87 per throughput (three-year total cost divided by transactions per minute), while CockroachDB was $9.19 and YugabyteDB was $11.13. The lower the price-performance metric, the better the throughput for money the database system provides.
Testing hardware and software across cloud vendors is very challenging. Configurations can favor one cloud vendor over another in feature availability, virtual machine processor generations, memory amounts, storage configurations for optimal input/output, network latencies, software and operating system versions, and the benchmarking workload itself. Our testing demonstrates a slice of potential configurations and workloads. As the sponsor of the report, Microsoft selected the specific Azure configuration it wanted to test. GigaOm then selected the configurations for CockroachDB and YugabyteDB closest to the Azure configuration in terms of CPU and memory configuration.
We leave the benchmark’s fairness for the reader to determine. We strongly encourage you to look past marketing messages and discern what is of value. We hope this report is informative and helpful in uncovering some of the challenges and nuances in platform selection.
The parameters to replicate this benchmark are provided in this document. We used HammerDB v4.6, a well-known and widely available TPC-C workload test kit for Cosmos DB for PostgreSQL and YugabyteDB. We used CockroachDB’s own TPC-C implementation for their tests.
The reason CoackroachDB was tested using its own tooling, rather than the HammerDB test kit, is that CockroachDB does not support HammerDB’s stored procedures. The CockroachDB TPC-C implementation we employed offered more flexibility in testing, which allowed us to create custom workloads and employ parallelized tests for improved performance.
You are encouraged to compile your own representative queries, data sets, data sizes, and test compatible configurations applicable to your requirements.
2. Distributed DBaaS Offerings
The world of distributed database-as-a-service (DBaaS) is rapidly evolving. DBaaS is a cloud-based service that allows users to access and manage databases without having to install and maintain the underlying hardware and software. This makes it an attractive option for businesses that need to quickly and easily access and manage large amounts of data. We chose three of the leading distributed DBaaS offerings for our test.
Azure Cosmos DB for PostgreSQL
Microsoft offers Azure Cosmos DB, a multi-model, globally distributed database service for both NoSQL and Relational workloads. Azure Cosmos DB for PostgreSQL is a cloud-based database service that is powered by the Citus open source extension to Postgres. The service enables users to easily create and manage relational PostgreSQL databases that are scalable and secure. It provides users with a fully managed and globally distributed database service that can be used to store and query relational data. The service is designed to provide low latency, high availability, and automatic scale-out of your data. With Azure Cosmos DB for PostgreSQL, users have access to a powerful and resilient database service that is fully managed, secure, and globally available.
CockroachDB Dedicated
CockroachDB offers a fully managed, distributed SQL database as a managed service called CockroachDB Dedicated. It is optimized for mission-critical applications and is made to be highly available, resilient, and secure. A number of features are available, including secure access control, automatic scaling, and automated backups. Additionally, it offers numerous integrations with other services and various tools for managing and monitoring the database.
YugabyteDB Managed
YugabyteDB Managed is a cloud-based database service that provides a distributed SQL database for modern applications. It is designed to provide high performance, scalability, and reliability for mission-critical applications. It is a fully managed service that automates the deployment, scaling, and management of YugabyteDB clusters, allowing users to focus on their applications instead of managing the underlying infrastructure. It also provides a range of features, such as automatic backups, monitoring, and alerting.
Note: Google Spanner Postgres Interface
We had planned to test Google Spanner Postgres Interface; however, the product did not provide the level of Postgres compatibility required to run the benchmark. As a result, Spanner Postgres was left out of this evaluation.
3. Field Test Setup
GigaOm Transactional Field Test
The GigaOm Transactional Field Test is a workload derived from the well-regarded, industry-standard TPC Benchmark™ C (TPC-C). The benchmark is described as follows on the TPC website:
Approved in July of 1992, TPC Benchmark C is an online transaction processing (OLTP) benchmark. TPC-C involves a mix of five concurrent transactions of different types and complexity, either executed online or queued for deferred execution. The database is comprised of nine types of tables with a wide range of record and population sizes. TPC-C is measured in transactions per minute (tpmC). While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment but rather represents any industry that must manage, sell, or distribute a product or service.
The results of TPC-C are valuable to any organization that frequently uses its transactional databases for modern application development.
Database Population
In the case of Cosmos DB for PostgreSQL and YugabyteDB, the data sets used in the benchmark were generated by HammerDB, based on the information provided in the TPC-C specification. For CockroachDB, due to interoperability issues with HammerDB, we used the datasets provided by the CockroachDB TPC-C implementation and imported the data according to the instructions provided. We tested the databases scaled for 1,000 and 10,000 warehouses, which determined the initial data volume of the database. For example, a total of 1,000 warehouses yields 30 million customers, 100 million items, 30 million orders, and 300 million order line items.
We completed three runs per test on each platform with 1,000 virtual users, with each test run lasting 60 minutes after a 5-minute ramp-up. Also, after each run, a full backup was restored to reset the database back to its original state. Table 1 details the HammerDB settings we used for Cosmos DB for PostgreSQL and YugabyteDB. The results of the testing are shared in the Field Test Results section.
Table 1. HammerDB parameters used for Cosmos DB for PostgreSQL and YugabyteDB
| Phase | Parameter | Setting |
|---|---|---|
| Build / Run | pg_count_ware | 1,000 and 10,000 |
| Build / Run | pg_raiseerror | True |
| Build / Run | pg_storedprocs | True |
| Build | pg_partition | False |
| Build | pg_cituscompat | True |
| Build | pg_vacuum | True |
| Run | pg_allwarehouse | True |
| Run | pg_driver | Timed |
| Run | pg_timeprofile | True |
| Run | pg_rampup | 5 |
| Run | pg_duration | 60 |
| Run | vuset delay | 20 |
| Run | vuset vu | 1,000* |
| Source: GigaOm 2023 | ||
Note that Cosmos DB for PostgreSQL has a default limit of 1,000 concurrent connections. Thus, we tested 999 users since one of the connections is needed for administration. This limit can be increased with a support request to Azure.
Database Environments
Selecting and sizing the infrastructure for comparison can be challenging, particularly across two vendor platforms. There are various offerings between Cosmos DB for PostgreSQL, CockroachDB, and YugabyteDB for these types of workloads. There was not an exact match across the offerings for either processors or memory, as can be seen in Table 2.
Table 2. Configurations of the Systems Under Test (SUT)
| Cosmos DB for PostgreSQL | CockroachDB Dedicated | YugabyteDB Managed | |
|---|---|---|---|
| Version | PostgreSQL 15.1 Citus 11.2.0 | 22.2.4 | 2.14.6.0-b30 |
| Cloud | Azure | AWS | AWS |
| Region | East US | US East 1 | US East 1 |
| Fault Tolerance | High Availability Off | Single Region | Node-level |
| Worker Nodes | 12x 16 vCores, 128 GB RAM 2,048 GB Storage | 14x 16 vCPU, 64 GiB RAM 2,048 GB Storage | 14x 16 vCPU, 64 GiB RAM 2,048 GB Storage |
| Coordinator Node | 1x32 vCores, 128 GB RAM512 GB Storage | N/A | N/A |
| Total vCPU | 224 | 224 | 224 |
| Cost Per Hour | $34.91 | $62.17 | $57.63 |
| Cost Per Month | $25,484 | $45,384 | $42,067 |
| Source: GigaOm 2023 | |||
Additionally, taking guidance from HammerDB’s best practice recommendations for PostgreSQL, we changed the default Cosmos DB for PostgreSQL worker memory to 16MB and set pg_stat_statements.track to “none”. These settings are not configurable for the fully-managed versions of YugabyteDB and CockroachDB. Also, the services offered by these platforms have several high availability (HA) options that are not directly comparable. We selected the minimal fault tolerance option for each, which offers the best performance.
We calculated the monthly cost of each configuration to help us determine the relative price-performance of each tested platform (Figure 1). Of course, results may vary across different configurations, and again, you are encouraged to compile your own representative queries, data sets, data sizes, and test compatible configurations applicable to your requirements.
Figure 1. Monthly Cost of Systems Under Test with 224 Total vCPU. (lower is better)
4. Field Test Results
This section analyzes the New Orders per minute (NOPM) from the three 60-minute runs of each of the scaled GigaOm Transactional Field Tests described above. A higher NOPM is better—meaning more transactions are being processed every minute.
Azure Cosmos DB for PostgreSQL achieved over five times more throughput than the CockroachDB Dedicated and YugabyteDB Managed configurations described in the previous section.
Figure 2. Best New Orders per Minute (NOPM) Observed by Each Platform (higher is better)
Results Validation
Comparing and testing distributed databases across clouds and vendors is very challenging. Configurations can favor one platform over another, and we always strive to be as fair as possible. To internally assess the fairness of our field tests, we attempted to validate our results by asking two questions:
- Did we push each competitive platform as hard as possible?
- Are there other published benchmarking results against which to compare our results?
First, we recognize the three main bottlenecks for system performance concerning the TPC-C-like workload we used are CPU, memory, and disk I/O. While executing tests, we monitor these metrics as closely as the managed platform allows. After running the CockroachDB and YugabyteDB configurations through the 1,000-warehouse test, we saw less than 20% average CPU utilization. We knew they could be pushed harder. We increased the number of warehouses and repeated the tests at 10,000 and 20,000.
Figure 3 shows the number of new orders per minute (NOPM) produced in our tests. In Figure 4, we track the CPU utilization curve across the three test tiers. As it “flattened” into the 80% range, we felt confident we were nearing the CPU bounds. Also, adding more warehouses would likely introduce significant memory pressure with the cluster sizes we used. In terms of disk I/O, it is difficult to speculate.
Figure 3. CockroachDB and YugabyteDB NOPM as Number of TPC-C Warehouses Increases (higher is better)
Figure 4. CockroachDB and YugabyteDB Average CPU Utilization as Number of TPC-C Warehouses Increases
We also attempted to validate our results by comparing what we observed with other published results. For comparable CockroachDB results, we reference a 2020 study published by the Association for Computing Machinery (ACM) that reported CockroachDB on three nodes of c5d.4xlarge achieved 12,474 TpmC using 1,000 warehouses, and 15 nodes of c5d.4xlarge achieved 124,036 TpmC using 10,000 warehouses. These results were very similar to ours.
For comparable YugabyteDB results, we found published TPC-C-like results on their website that reported that three nodes of c5d.4xlarge achieved 12,563 TpmC while using 1,000 warehouses, and 125,163 TpmC while using 10,000 warehouses. Our results in our tests of YugabyteDB were about 40% lower than the published figures on the website. While this is concerning, we must note that the YugabyteDB results were achieved by running their own TPC-C-like workload driver installed locally on the database nodes. We could not get their TPC-C driver to work against their managed service, which is why we used HammerDB instead. Obviously, this HammerDB driver was installed on an EC2 instance in the same region as our cluster, but it’s difficult to know how running it against the managed service impacted our results.
Figures 5 and 6 show the results for testing at 1,000 warehouses and 10,000 warehouses, respectively.
Figure 5. CockroachDB and YugabyteDB TpmC Benchmark Comparison: 1K Warehouses
Figure 6. CockroachDB and YugabyteDB TpmC Benchmark Comparison: 10K Warehouses
However, the fact remains that even if we had achieved similar results as Yugabyte’s reported numbers, the overall throughput would have been much lower than Cosmos DB. As you will see in the next section, the resulting price-per-performance would have remained much more expensive as well.
5. Price Per Performance
The price-performance metric is price/throughput. This is defined as the cost of running each cloud platform continuously for three years, divided by the transactions per second throughput uncovered in the previously described tests. The calculation is as follows:
Price Per Performance = 3-Year Cost for System Under Test ÷ NOPM =
Service Hourly Rate × 24 hours/day × 365 days/year × 3 years
New Orders Per Minute (NOPM)
When evaluating price-per-performance, the lower the number, the better. This means you get more compute power, storage I/O, and capacity for your budget.
Pricing Used:
To keep pricing comparisons on a level playing field, we used the same type of on-demand, pay-as-you-go pricing across all three vendors. The prices used to calculate the results in Figure 7 were at the time of testing and reflect the US East 1 region on AWS and East US region on Azure.
NOTE: Each platform has different pricing options. Buyers should evaluate all available pricing choices, such as discounts for multi-year commitments, not just the on-demand/pay-as-you-go rates presented in this paper.
Figure 7. Price Per Performance (3-Year Total System Under Test Cost ÷ NOPM) Lower is Better
As you can see, the price-per-performance of Azure Cosmos DB for PostgreSQL was significantly lower than CockroachDB Dedicated and YugabyteDB Managed.
6. Conclusion
This report outlines the results from a TPC-C-like test to compare Azure Cosmos DB for PostgreSQL, CockroachDB Dedicated, and YugabyteDB Managed. On this day, for this particular workload, with these specific configurations, Azure Cosmos DB for PostgreSQL had higher throughput than CockroachDB and YugabyteDB. Furthermore, it was more cost-effective from a price-per-performance standpoint to run Azure Cosmos DB for PostgreSQL than CockroachDB and YugabyteDB managed services.
For Cosmos DB for PostgreSQL, YugabyteDB, and CockroachDB, our monthly costs were about $25K, $42K, and $45K, respectively. The best observed NOPM (TpmC) performance measurement for TPC-C was 1.05 million for Cosmos DB for PostgreSQL, 178,000 for CockroachDB, and 136,000 for YugabyteDB. We also demonstrated that as the number of TPC-C warehouses increased, the average CPU utilization for CockroachDB and YugabyteDB significantly increased.
Based on our analysis, the three-year price-performance of Azure Cosmos DB for PostgreSQL was the least expensive at $0.87, followed by CockroachDB at $9.19 and by YugabyteDB at $11.13.
This means that Cosmos DB for PostgreSQL was not only the most performant of the three databases but also the most cost-effective regarding price, performance, and total system costs.
Keep in mind, more favorable configurations (such as self-managed deployment) and optimizations on Azure Cosmos DB for PostgreSQL, CockroachDB, and YugabyteDB may be possible. All three of these managed service distributed databases are evolving their offerings.
7. Disclaimer
Performance is important, but it is only one criterion for a business-critical database platform selection. This test is a point-in-time check into specific performance. There are numerous other factors to consider in selection across factors of Administration, Integration, Workload Management, User Interface, Scalability, Vendor, Reliability, and numerous other criteria. It is also our experience that performance changes over time and is competitively different for different workloads. Also, a performance leader can hit up against the point of diminishing returns, and viable contenders can quickly close the gap.
The benchmark setup was informed by the TPC Benchmark™ C (TPC-C) specification. The workload was derived from TPC-C and is not an official TPC benchmark, nor should the results be compared to official TPC-C publications.
GigaOm runs all of its performance tests to strict ethical standards. The report’s results are the objective results of applying queries to the simulations described. The report clearly defines the selected criteria and process used to establish the field test. The report also clearly states the data set sizes, the platforms, the queries, etc. used. The reader is left to determine how to qualify the information for their needs. The report does not make any claim regarding the third-party certification and presents the objective results received from the application of the process to the criteria as described in the report. The report strictly measures performance and does not purport to evaluate other factors that potential customers may find relevant when making a purchase decision.
This is a sponsored report. Microsoft chose the competitors, the test, and the Microsoft configuration. GigaOm chose the most compatible configurations for the other tested platform and ran the testing workloads. Choosing compatible configurations is subject to judgment. We have attempted to describe our decisions in this paper.
8. About Microsoft
Microsoft (Nasdaq “MSFT” @microsoft) enables digital transformation for the era of an intelligent cloud and an intelligent edge. Its mission is to empower every person and every organization on the planet to achieve more.
Microsoft offers Cosmos DB on Azure—a globally-distributed, multi-model database service for managing data “at planet-scale”. To learn more about Azure Cosmos DB, visit https://azure.microsoft.com/en-us/products/cosmos-db.
9. About William McKnight
William McKnight is a former Fortune 50 technology executive and database engineer. An Ernst & Young Entrepreneur of the Year finalist and frequent best practices judge, he helps enterprise clients with action plans, architectures, strategies, and technology tools to manage information.
Currently, William is an analyst for GigaOm Research who takes corporate information and turns it into a bottom-line-enhancing asset. He has worked with Dong Energy, France Telecom, Pfizer, Samba Bank, ScotiaBank, Teva Pharmaceuticals, and Verizon, among many others. William focuses on delivering business value and solving business problems utilizing proven approaches in information management.
10. About Jake Dolezal
Jake Dolezal is a contributing analyst at GigaOm. He has two decades of experience in the information management field, with expertise in analytics, data warehousing, master data management, data governance, business intelligence, statistics, data modeling and integration, and visualization. Jake has solved technical problems across a broad range of industries, including healthcare, education, government, manufacturing, engineering, hospitality, and restaurants. He has a doctorate in information management from Syracuse University.
11. About GigaOm
GigaOm provides technical, operational, and business advice for IT’s strategic digital enterprise and business initiatives. Enterprise business leaders, CIOs, and technology organizations partner with GigaOm for practical, actionable, strategic, and visionary advice for modernizing and transforming their business. GigaOm’s advice empowers enterprises to successfully compete in an increasingly complicated business atmosphere that requires a solid understanding of constantly changing customer demands.
GigaOm works directly with enterprises both inside and outside of the IT organization to apply proven research and methodologies designed to avoid pitfalls and roadblocks while balancing risk and innovation. Research methodologies include but are not limited to adoption and benchmarking surveys, use cases, interviews, ROI/TCO, market landscapes, strategic trends, and technical benchmarks. Our analysts possess 20+ years of experience advising a spectrum of clients from early adopters to mainstream enterprises.
GigaOm’s perspective is that of the unbiased enterprise practitioner. Through this perspective, GigaOm connects with engaged and loyal subscribers on a deep and meaningful level.
12. Copyright
© Knowingly, Inc. 2023 "Transaction Processing & Price-Performance Testing" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact sales@gigaom.com.