人臉疲勞檢測應(yīng)用-米爾基于RK3576核心板/開發(fā)板

本篇源自：優(yōu)秀創(chuàng)作者 lulugl

本文將介紹基于米爾電子MYD-LR3576開發(fā)板（米爾基于瑞芯微 RK3576開發(fā)板）的人臉疲勞檢測方案測試。

米爾基于RK3576核心板/開發(fā)板

【前言】

人臉疲勞檢測：一種通過分析人臉特征來判斷一個人是否處于疲勞狀態(tài)的技術(shù)。其原理主要基于計算機(jī)視覺和機(jī)器學(xué)習(xí)方法。當(dāng)人疲勞時，面部會出現(xiàn)一些特征變化，如眼睛閉合程度增加、眨眼頻率變慢、打哈欠、頭部姿態(tài)改變等。
例如，通過檢測眼睛的狀態(tài)來判斷疲勞程度是一個關(guān)鍵部分。正常情況下，人的眨眼頻率相對穩(wěn)定，而當(dāng)疲勞時，眨眼頻率會降低，并且每次眨眼時眼睛閉合的時間可能會延長。同時，頭部可能會不自覺地下垂或者搖晃，這些特征都可以作為疲勞檢測的依據(jù)。米爾MYC-LR3576采用8核CPU+搭載6 TOPS的NPU加速器，3D GPU，能夠非常輕松的實現(xiàn)這個功能，下面就如何實現(xiàn)這一功能分享如下：

【硬件】

1、米爾MYC-LR3576開發(fā)板
2、USB攝像頭

【軟件】

1、v4l2
2、openCV
3、dlib庫：dlib 是一個現(xiàn)代化的 C++ 工具包，它包含了許多用于機(jī)器學(xué)習(xí)、圖像處理、數(shù)值計算等多種任務(wù)的算法和工具。它的設(shè)計目標(biāo)是提供高性能、易于使用的庫，并且在開源社區(qū)中被廣泛應(yīng)用。

【實現(xiàn)步驟】

1、安裝python-opencv
2、安裝dlib庫
3、安裝v4l2庫

【代碼實現(xiàn)】

1、引入cv2、dlib以及線程等：

importcv2importdlibimportnumpyasnpimporttimefromconcurrent.futuresimportThreadPoolExecutorimport threading

2、初始化dlib的面部檢測器和特征點(diǎn)預(yù)測器

detector=dlib.get_frontal_face_detector()predictor = dlib.shape_predictor('shape_predictor_68_face_landmarks.dat')

3、定義計算眼睛縱橫比的函數(shù)

defeye_aspect_ratio(eye):A=np.linalg.norm(np.array(eye[1])-np.array(eye[5]))B=np.linalg.norm(np.array(eye[2])-np.array(eye[4]))C=np.linalg.norm(np.array(eye[0])-np.array(eye[3])) ear = (A + B) / (2.0 * C) return ear

4、定義計算頭部姿勢的函數(shù)

defget_head_pose(shape):

#定義面部特征點(diǎn)的三維坐標(biāo)object_points=np.array([(0.0,0.0,0.0), #鼻尖(0.0,-330.0,-65.0),

#下巴(-225.0,170.0,-135.0),

#左眼左眼角 (225.0, 170.0, -135.0), # 右眼右眼角(-150.0,-150.0,-125.0), #左嘴角(150.0,-150.0,-125.0)

#右嘴角],dtype=np.float32)
image_pts=np.float32([shape[i]foriin[30,8,36,45,48,54]])size=frame.shapefocal_length=size[1]center=(size[1]//2,size[0]//2)camera_matrix=np.array([[focal_length,0,center[0]],[0,focal_length,center[1]],

[0,0,1]],dtype="double")
dist_coeffs=np.zeros((4,1))(success,rotation_vector,translation_vector)=cv2.solvePnP(object_points,image_pts,camera_matrix,dist_coeffs,flags=cv2.SOLVEPNP_ITERATIVE)
rmat,_=cv2.Rodrigues(rotation_vector)angles,_,_,_,_,_=cv2.RQDecomp3x3(rmat) return angles

5、定義眼睛縱橫比閾值和連續(xù)幀數(shù)閾值

EYE_AR_THRESH=0.3EYE_AR_CONSEC_FRAMES = 48

6、打開攝像頭
我們先使用v4l2-ctl --list-devices來例出接在開發(fā)板上的列表信息：

USBCamera:USBCamera(usb-xhci-hcd.0.auto-1.2): /dev/video60/dev/video61/dev/media7

在代碼中填入60為攝像頭的編號：

cap=cv2.VideoCapture(60)cap.set(cv2.CAP_PROP_FRAME_WIDTH,480) #降低分辨率cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 320)

7、創(chuàng)建多線程處理函數(shù)，實現(xiàn)采集與分析分離：

#多線程處理函數(shù)defprocess_frame(frame): globalCOUNTER,TOTALgray=cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)faces=detector(gray,0) #第二個參數(shù)為0，表示不使用upsampling
forfaceinfaces:landmarks=predictor(gray,face)shape=[(landmarks.part(i).x,landmarks.part(i).y)foriinrange(68)]
left_eye=shape[36:42]right_eye=shape[42:48]
left_ear=eye_aspect_ratio(left_eye)right_ear=eye_aspect_ratio(right_eye)ear=(left_ear+right_ear)/2.0
ifear=EYE_AR_CONSEC_FRAMES:TOTAL+=1COUNTER=0
#繪制68個特征點(diǎn)

forninrange(0,68):x,y=shape[n]cv2.circle(frame,(x,y),2,(0,255,0),-1)
cv2.putText(frame,f"EyeAR:{ear:.2f}",(10,30),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)cv2.putText(frame,f"BlinkCount:{TOTAL}",(10,60),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)
#計算頭部姿勢

angles=get_head_pose(shape)pitch,yaw,roll=anglescv2.putText(frame,f"Pitch:{pitch:.2f}",(10,120),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)cv2.putText(frame,f"Yaw:{yaw:.2f}",(10,150),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)cv2.putText(frame,f"Roll:{roll:.2f}",(10,180),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)
#判斷疲勞狀態(tài) ifCOUNTER>=EYE_AR_CONSEC_FRAMESorabs(pitch)>30orabs(yaw)>30orabs(roll)>30:cv2.putText(frame,"FatigueDetected!",(10,210),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)
return frame

8、創(chuàng)建圖像顯示線程：

withThreadPoolExecutor(max_workers=2)asexecutor:future_to_frame={} whileTrue:ret,frame=cap.read() ifnotret: break

#提交當(dāng)前幀到線程池

future=executor.submit(process_frame,frame.copy())future_to_frame[future]=frame

#獲取已完成的任務(wù)結(jié)果

forfutureinlist(future_to_frame.keys()): iffuture.done():processed_frame=future.result()cv2.imshow("Frame",processed_frame) delfuture_to_frame[future] break

#計算幀數(shù)

fps_counter+=1elapsed_time=time.time()-start_time ifelapsed_time>1.0:fps=fps_counter/elapsed_timefps_counter=0start_time=time.time()cv2.putText(processed_frame,f"FPS:{fps:.2f}",(10,90),cv2.FONT_HERSHEY_SIMPLEX,0.7,(0,0,255),2)
if cv2.waitKey(1) & 0xFF == ord('q'):

實現(xiàn)效果：

根據(jù)檢測的結(jié)果，我們就可以來實現(xiàn)疲勞提醒等等的功能。
整體代碼如下：

importcv2importdlibimportnumpyasnpimporttimefromconcurrent.futuresimportThreadPoolExecutorimportthreading
#初始化dlib的面部檢測器和特征點(diǎn)預(yù)測器

detector=dlib.get_frontal_face_detector()predictor=dlib.shape_predictor('shape_predictor_68_face_landmarks.dat')
#修改字體大小font_scale=0.5 #原來的字體大小是0.7，現(xiàn)在改為0.5
#定義計算眼睛縱橫比的函數(shù)

defeye_aspect_ratio(eye):A=np.linalg.norm(np.array(eye[1])-np.array(eye[5]))B=np.linalg.norm(np.array(eye[2])-np.array(eye[4]))C=np.linalg.norm(np.array(eye[0])-np.array(eye[3]))ear=(A+B)/(2.0*C) returnear
#定義計算頭部姿勢的函數(shù)

defget_head_pose(shape):

#定義面部特征點(diǎn)的三維坐標(biāo)

object_points=np.array([(0.0,0.0,0.0),

#鼻尖(0.0,-330.0,-65.0),

#下巴(-225.0,170.0,-135.0),

#左眼左眼角(225.0,170.0,-135.0),

#右眼右眼角(-150.0,-150.0,-125.0),

#左嘴角(150.0,-150.0,-125.0)

#右嘴角],dtype=np.float32)
image_pts=np.float32([shape[i]foriin[30,8,36,45,48,54]])size=frame.shapefocal_length=size[1]center=(size[1]//2,size[0]//2)camera_matrix=np.array([[focal_length,0,center[0]],[0,focal_length,center[1]],[0,0,1]],dtype="double")
dist_coeffs=np.zeros((4,1))(success,rotation_vector,translation_vector)=cv2.solvePnP(object_points,image_pts,camera_matrix,dist_coeffs,flags=cv2.SOLVEPNP_ITERATIVE)
rmat,_=cv2.Rodrigues(rotation_vector)angles,_,_,_,_,_=cv2.RQDecomp3x3(rmat) returnangles
#定義眼睛縱橫比閾值和連續(xù)幀數(shù)閾值

EYE_AR_THRESH=0.3EYE_AR_CONSEC_FRAMES=48
#初始化計數(shù)器

COUNTER=0TOTAL=0
#創(chuàng)建鎖對象lock=threading.Lock()
#打開攝像頭cap=cv2.VideoCapture(60)cap.set(cv2.CAP_PROP_FRAME_WIDTH,480) #降低分辨率cap.set(cv2.CAP_PROP_FRAME_HEIGHT,320)
#初始化幀計數(shù)器和時間戳

fps_counter=0start_time=time.time()
#多線程處理函數(shù)

defprocess_frame(frame): globalCOUNTER,TOTALgray=cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY) faces = detector(gray, 0) # 第二個參數(shù)為0，表示不使用upsampling
forfaceinfaces:landmarks=predictor(gray,face) shape = [(landmarks.part(i).x, landmarks.part(i).y) for i in range(68)]
left_eye=shape[36:42]right_eye=shape[42:48]
left_ear=eye_aspect_ratio(left_eye)right_ear=eye_aspect_ratio(right_eye)ear=(left_ear+right_ear)/2.0
ifear=EYE_AR_CONSEC_FRAMES:TOTAL+=1COUNTER=0
#繪制68個特征點(diǎn) forninrange(0,68):x,y=shape[n]cv2.circle(frame,(x,y),2,(0,255,0),-1)
cv2.putText(frame,f"EyeAR:{ear:.2f}",(10,30),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2) cv2.putText(frame, f"Blink Count: {TOTAL}", (10, 60), cv2.FONT_HERSHEY_SIMPLEX, font_scale, (0, 0, 255), 2)
#計算頭部姿勢

angles=get_head_pose(shape)pitch,yaw,roll=anglescv2.putText(frame,f"Pitch:{pitch:.2f}",(10,120),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)cv2.putText(frame,f"Yaw:{yaw:.2f}",(10,150),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2) cv2.putText(frame, f"Roll: {roll:.2f}", (10, 180), cv2.FONT_HERSHEY_SIMPLEX, font_scale, (0, 0, 255), 2)
#判斷疲勞狀態(tài)

ifCOUNTER>=EYE_AR_CONSEC_FRAMESorabs(pitch)>30orabs(yaw)>30orabs(roll)>30:cv2.putText(frame,"FatigueDetected!",(10,210),cv2.FONT_HERSHEY_SIMPLEX,font_scale,(0,0,255),2)
returnframe
withThreadPoolExecutor(max_workers=2)asexecutor:future_to_frame={} whileTrue:ret,frame=cap.read() ifnotret: break
#提交當(dāng)前幀到線程池

future=executor.submit(process_frame,frame.copy())future_to_frame[future]=frame
#獲取已完成的任務(wù)結(jié)果

forfutureinlist(future_to_frame.keys()): iffuture.done():processed_frame=future.result()cv2.imshow("Frame",processed_frame) delfuture_to_frame[future] break
#計算幀數(shù)

fps_counter+=1elapsed_time=time.time()-start_time ifelapsed_time>1.0:fps=fps_counter/elapsed_timefps_counter=0start_time=time.time() cv2.putText(processed_frame, f"FPS: {fps:.2f}", (10, 90), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
ifcv2.waitKey(1)&0xFF==ord('q'): break
#釋放攝像頭并關(guān)閉所有窗口

cap.release()cv2.destroyAllWindows()

【總結(jié)】

【米爾MYC-LR3576核心板及開發(fā)板】
這塊開發(fā)板性能強(qiáng)大，能輕松實現(xiàn)對人臉的疲勞檢測，通過計算結(jié)果后進(jìn)入非常多的工業(yè)、人工智能等等的實用功能。