Key Criteria for Evaluating Edge AI Processorsv1.0

1. Summary

The market for artificial intelligence (AI) accelerator processors for edge computing is estimated to be currently worth some $20 billion and is set to double in five years, surpassing the spend on AI accelerators in the data center (across private and public clouds). The edge has distinct constraints that impact AI processors working in that environment: latency limits, power availability, safety-critical use cases, privacy and security concerns, and not least, data throughput in a typically small chip that falls within cost limits. A large but fragmented group of chip suppliers has emerged in a market that’s highly competitive but still in the early stages and yet to rationalize, making the right investment decision challenging.

This GigaOm Key Criteria report is designed to be a foundational resource for product manufacturers building edge technology solutions, to help them make decisions about selecting an edge AI processor. In this report, we provide an overview of edge AI processor capabilities, including those that are commonplace and those that stand out. We define specific criteria and metrics for selecting an edge AI processor, enabling engineering organizations to make better decisions from among available options.

This evaluation extends to table stakes that are common to virtually all products in this sector, key criteria that define differentiating features to focus on, and emerging technologies that point to ongoing innovation in this space. Finally, we describe a set of evaluation metrics—high-level characteristics that help determine the impact processor choice can have on implementation and are useful in assessing specific products. Whether you are looking to extend existing capabilities in AI edge computing or have yet to adopt edge AI processors, this report lays the groundwork for informing the selection and implementation of an edge AI processor for your needs.

Findings reached in this report include:

• The edge computing market is set to grow on the back of a number of converging technologies: AI, 5G, digital transformation, cloud native technology, and expansion of IOT.
• A useful criterion for determining an edge (and local) application from one that is merely distributed between edge and cloud is that latency is less than 20ms, meaning the application does not rely on a network to the cloud to perform its main function; instead, it has local resources that allow it to fulfil its prime function and operate in near real time. Many edge applications are connected to the cloud or gateways mid-way for additional functionality and services.
• The requirements for an AI processor at the edge are markedly different from accelerating AI in the cloud or data center: In the latter case, AI applications are typically trained and run large data sets for large numbers of users or they process complex numerically heavy applications in high-performance computing (HPC). At the edge, in contrast, the AI applications are typically running in inference mode, and are constrained by limitations in power, chip size, latency, bandwidth, and cost.
• AI training at the edge is a differentiator across the edge AI processors, with local continuous learning. This is in contrast to wired or over-the-air model updating, for which the edge processor is always operating in inference mode.
• Selecting an edge AI processor simply on its tera operations per second (TOPS) rating or even TOPS per watt misses important considerations. We provide a full and rounded set of assessment criteria that considers preprocessing optimization, latencies, software maturity, and more.
• We take the view that the type of processor architecture is less important than its characteristics and suitability for edge AI applications.