Abstract
Choosing the appropriate deployment strategy for any Deep Learning (DL) project in a production environment has always been the most challenging problem for industrial practitioners. There are several conflicting constraints and controversial approaches when it comes to deployment. Among these problems, the deployment on cloud versus the deployment on edge represents a common dilemma. In a nutshell, each approach provides benefits where the other would have limitations. This paper presents a real-world case study on deploying a face recognition application using MTCNN detector and FaceNet recognizer. We report the challenges faced to decide on the best deployment strategy. We propose three inference architectures for the deployment, including cloud-based, edge-based, and hybrid. Furthermore, we evaluate the performance of face recognition inference on different cloud-based and edge-based GPU platforms. We consider different models of Jetson boards for the edge (Nano, TX2, Xavier NX, Xavier AGX) and various GPUs for the cloud (GTX 1080, RTX 2080Ti, RTX 2070, and RTX 8000). We also investigate the effect of deep learning model optimization using TensorRT and TFLite compared to a standard Tensorflow GPU model, and the effect of input resolution. We provide a benchmarking study for all these devices in terms of frames per second, execution times, energy and memory usages. After conducting a total of 294 experiments, the results demonstrate that the TensorRT optimization provides the fastest execution on all cloud and edge devices, at the expense of significantly larger energy consumption (up to +40% and +35% for edge and cloud devices, respectively, compared to Tensorflow). Whereas TFLite is the most efficient framework in terms of memory and power consumption, while providing significantly less (-4% to -62%) processing acceleration than TensorRT. Practitioners Note: The study reported in this paper presents the real-challenges that we faced during our development and deployment of a face-recognition application both on the edge and on the cloud, and the solutions we have developed to solve these problems. The code, results, and interactive analytic dashboards of this paper will be put public upon publication.