IBD 1 : The metrics and evaluation method for DNN processor benchmark while doing Inference task

Abstract

With the many varieties of AI hardware prevailing on the market, it is often hard to decide which one is the most suitable to use but not only with the best performance. As there is an industry-wide trend demand for deep learning deployment, the inference benchmark for the effectiveness of DNN processor becomes important and is of great help to select and optimize AI hardware. To systematically benchmark deep learning deployment platforms, and give more objective and useful metrics comparison. In this paper, an end to end benchmark evaluation system was brought up called IBD, it combined 4 steps include three components with 6 metrics. The performance comparison results are obtained from the chipsets from Qualcomm, HiSilicon, and NVIDIA, which can provide hardware acceleration for AI inference. To comprehensively reflect the current status of the DNN processor deploying performance, we chose six devices from three kinds of deployment scenarios which are cloud, desktop and mobile, ten models from three different kinds of applications with diverse characteristics are selected, and all these models are trained from three major training frameworks. Several important observations were made by using our methodologies. Experimental results showed that workload diversity should focus on the difference came from training frameworks, inference frameworks with specific processors, input size and precision (floating and quantized).