Journal of Ergonomics
Open Access

Short Communication on Detection of Operator Fatigue in the Main Control Room of a Nuclear Power Plant Based on Eye Blink Rate, PERCLOS and Mouse Velocity

Licao Dai^{* *}Correspondence: Licao Dai, Institute of Human Factors, University of South China, Hengyang 421001, China, Email:

Author info »

About the Study

Automatic and non-invasive real-time detection of operator fatigue in the main control room of a nuclear power plant is of great importance. The research is for this purpose. It is composed of three parts. The first part is about collecting data through simulation tasks as well as feature engineering. In the second part, unsupervised learning is used to generate different clusters. Trends in eye and mouse feature changes are used to determine the fatigue levels corresponding to different clusters. In the third part, a model for detecting fatigue is trained by machine learning, and the model is tested for its speed in different time windows.

In the first part, on the one hand, data was obtained through simulator experiments. The data consisted of eye and mouse data captured via camera and python script. The simulated experiments were divided into 5 phases according to the task time period paradigm [1], each phase lasting approximately 12 minutes. At the end of each stage, the experimenter was asked to fill in the Karolinska Sleepiness Scale (KSS) and the Stanford Sleepiness Scale (SSS). On the other hand, feature engineering was performed on the obtained data. Data pre-processing and feature extraction were carried out separately. Data preprocessing included the processing of outliers, duplicates and missing values. Four features were extracted: blink rate, number of frames with eyes closed in a given time (PERCLOS), Average Mouse Velocity (AMV) and Average of Mouse Velocity (AOV).

For the second part, there are several widely-used methods of fatigue labelling: subjective assessment methods, task phase times and physiological trends [2-4]. In order to avoid the disadvantages of these labelling methods, firstly, multivariate time series clustering is used. The sample data are divided into multiple clusters. As operator fatigue increases, there is a tendency for blink rate and the proportion of eyes closed for a specified period of time to increase [5] and mouse velocity tends to decrease [6]. Then, these clusters are identified as fatigue levels by trends in subjective scores (KSS and SSS), eye features (blink rate and PERCLOS), and mouse features (AMV and AOV). The advantages of using this unsupervised learning approach to determine fatigue levels are automatic annotation, objective, robust and highly accurate (fine-grained annotation).

In the third part, an automatic fatigue detection model combining eye features and mouse features is constructed by training a large sample data of labelled fatigue levels. In comparative experiments with three feature sets of eye, mouse and a combination of eye and mouse, the best algorithm is KNearest Neighbor (KNN). The sampling rate is 0.1 Hz. For the first time window 60 s is needed to detect fatigue and an additive 10 s is required for a fast detection.

Conclusion

In all about Human Computer Interaction (HCI) mental state detection, we used studies targeting blink rate, PERCLOS, average mouse velocity and average mouse velocity to be able to provide some suggestions for further acquisition of even better features. We addressed the issue of fatigue detection for operators in nuclear power plant control room and developed unsupervised learning methods to improve laborious sample labeling. Additionally, two non-intrusive modalities, eye tracking and mouse movement were used to improve the performance of the fatigue detection algorithm. Within about 10 seconds of data, our algorithm can accurately detect the operator's fatigue state.

References

1.  Li J, Li H, Umer W, Wang H, Xing X, Zhao S, et al. Identification and Classification of Construction Equipment Operators’Mental Fatigue Using Wearable Eye-Tracking Technology. Autom. Constr. 2020; 109.103000.

   [Google Scholar]

2.  McDonald AD, Lee JD, Schwarz C, Brown, TL. A Contextual and Temporal Algorithm for Driver Drowsiness Detection. Accid. Anal. Prev. 2018;113:25–37.

   [Crossref][Google Scholar]

3.  Yamada Y, Kobayashi M. Detecting Mental Fatigue from Eye-Tracking Data Gathered While Watching Video: Evaluation in Younger and Older Adults. Artif. Intell. Med. 2018; 91: 39–48.

   [Crossref][Google Scholar][PubMed]

4.  Zandi AS, Quddus A, Prest L, Comeau, FJE. Non-Intrusive Detection of Drowsy Driving Based on Eye Tracking Data. T ransp. Res. Rec. 2019; 2673:247–257.

   [Crossref][Google Scholar]

5.  Nie B, Huang X, Chen Y, Li A, Zhang R, Huang J. Experimental Study on Visual Detection for Fatigue of Fixed-Position Staff. Appl. Ergon. 2017; 65:1–11.

   [Crossref][Google Scholar][PubMed]

6.  Pimenta A, Carneiro D, Novais P, Neves J. Monitoring Mental Fatigue through the Analysis of Keyboard and Mouse Interaction Patterns. In Hybrid Artificial Intelligent Systems; Pan, J.-S., Polycarpou, M.M., Wo´ zniak, M., de Carvalho, A.C.P.L.F., Quintián, H., Corchado, E., Eds.; Lecture Notes in Computer Science; Springer: Berlin, Heidelberg, 2013; 8073:222–231.

   [Crossref][Google Scholar]