基於3D人臉辨識之擴增實境技術改善臉盲症社交輔助系統

Abstract

本論文目標為開發一套 AR(Augmented Reality，擴增實境)眼鏡輔助系 統，協助臉盲症患者在社交上、對於生活中他人的辨識。本研究主要貢獻為 提出以三維人臉模型作為人臉辨識之資料擴增基礎，滿足對於臉盲症患者實 際社交情境之實用性，並且將各軟體與硬體平台之優勢進行系統整合與設 計，實現可讓患者立即投入使用之社交輔助工具。具體架構包含以下內容: 第一，以結構光技術(Structured Light)結合立體視覺攝影機，經由 Structured Light 與2D RGB 輸入，2D 資料通過深度神經網路(Deep Neural Network)進行 人臉的提取，並確認三維空間中人臉之座標，運用深度學習將3D 點雲資訊 和2D 影像進行實時三維人臉密集重建，並取得人臉正面、側面 等7個角度 之人臉資訊，提高人臉對於側臉與大動態辨識的準確度。第二，藉由第一部 分產生之人臉資訊，輸入卷積神經網路進行運算，卷積神經網路以輸出128 維之特徵向量取代傳統高維分類器作為人臉特徵依據，將計算之特徵向量與 系統內 SQL(Structured Query Language)資料庫，進行歐式距離計算並比對， 取得最小歐式距離並對應該人臉的姓名資料。第三，將人臉標籤資訊、空間 中之人臉座標點，藉由相機模型投影，實現 AR 眼鏡中顯示即時人臉辨識標 籤及人臉 Bounding Box。本論文希望臉盲症患者在戴上 AR眼鏡後，AR 眼 鏡能夠即時從環境中掃描人臉，並從既有資料庫之中，辨別出對應之身分， 將該人的位置與人名標註至 AR 眼鏡中，幫助臉盲症患者能夠辨別出生活中 每個人之身分，不因認不出臉而產生困惑，突破社交上的阻礙，降低而因社 交上之阻礙，導致產生自閉症等心理疾病之可能性。The paper aims to develop an AR (Augmented Reality) eyewear system to help people with face blindness to recognize others in their life socially. This specific framework consists of the following: First, Structured Light technology is used to combine stereoscopic cameras with 2D RGB input through Structured Light, 2D data is extracted from the face through Deep Neural Network, and the coordinates of the face are identified in 3D space . In addition, the 3D point cloud information and 2D images are reconstructed bvin real time by deep learning, and the face information is obtained from seven angles, including front and side faces, to improve the accuracy of face recognition for side faces and large movements. Second, the face information generated in the first part is input to the convolutional neural network for computation. The convolutional neural network replaces the traditional high-dimensional classifier with a 128-dimensional feature vector as the basis for face characteristics. The minimum European distance is obtained and the name of the face is matched. This paper hopes that the AR glasses can instantly scan the face from the environment and identify the corresponding identity from the existing database, and label the location and name of the person into the AR glasses after the face blindness patient wears the AR glasses. The AR glasses help people with face blindness to identify everyone in their lives, so that they will not be confused by their inability to recognize faces, and to break through social barriers and reduce the possibility of autism and other psychological disorders caused by social barriers.