Key Criteria for Evaluating Data Pipelinesv1.0

1. Summary

In a data-obsessed world, the ability to leverage data for quick business insights is vital. At the same time, organizations must be in control of data flow to ensure data quality and achieve valid insights.

Data pipelines offer a solution to this challenge. They take raw data, transform it so it’s format is suitable for the target system, and then load it to a unified data store. That process normally takes weeks and requires writing chunks of code, but can be streamlined to take less time and require little-to-no coding with data pipeline tools. Considering the constantly growing amounts of data and the competitive advantage it can provide when analyzed efficiently, adopting a data pipeline tool to automate the data integration process can offer an unbeatable edge.

In an environment that offers a broad array of solutions, users should consider features like scalability, performance, monitoring, alerts, ease of maintenance, and collaboration before adopting a data pipeline solution and getting into technical debt.

In this report, we explore the data pipeline solution sector and outline the key criteria by which platforms should be evaluated, providing a decision making framework for IT leaders seeking to adopt a data pipeline platform.

Key findings:

• Most platforms offer codeless solutions and enable an environment for citizen data professionals to utilize data.
• The most complete offerings are AI-powered, provide support for any combination of cloud and hybrid infrastructures, and offer serverless processing.
• The marketplace for classic on-premises data pipeline solutions is mature and stable. The newest developments are premised on a single streamlined tool for data integration and increasingly, machine learning. Vendors are also providing richer built-in tools for complex data transformations.

  How to Read this Report

  This GigaOm report is one of a series of documents that helps IT organizations assess competing solutions in the context of well-defined features and criteria. For a fuller understanding consider reviewing the following reports:

  Key Criteria report: A detailed market sector analysis that assesses the impact that key product features and criteria have on top-line solution characteristics—such as scalability, performance, and TCO—that drive purchase decisions.

  GigaOm Radar report: A forward-looking analysis that plots the relative value and progression of vendor solutions along multiple axes based on strategy and execution. The Radar report includes a breakdown of each vendor’s offering in the sector.

  Solution Profile: An in-depth vendor analysis that builds on the framework developed in the Key Criteria and Radar reports to assess a company’s engagement within a technology sector. This analysis includes forward-looking guidance around both strategy and product.