Using machine learning method to recognize abnormal patterns covers the shortage of traditional control charts for autocorrelation processes, which violate the applicable conditions of the control chart, i.e., the independent identically distributed (IID) assumption. In this study, we propose a recognition model based on support vector machine (SVM) for the AR (1) type of autocorrelation process. For achieving a higher recognition performance, the cuckoo search algorithm (CS) is used to optimize the two hyper-parameters of SVM, namely the penalty parameter

Nowadays, the integration concept of informatization and industrialization is gradually being proposed. Many new technologies emerge continuously, such as Virtual Manufacturing (VM), Computer Integrated Manufacturing (CIM), and Radio-frequency Identification (RFID), Concurrent Engineering (CE), [

Academia has carried out much studies on process quality control with autocorrelation phenomena and proposed some approaches. The main category is to use statistical methods to eliminate the influence of autocorrelation on process data. The most representative one is the residual control chart. Montgomery and Woodall [

SVM is a major machine learning method for classification, which has been also used frequently in control chart pattern recognition [

Due to the superiorities of the SVM and the CS, we proposed an autocorrelation process pattern recognition model based on SVM optimized by CS (CS-SVM), which is used as the pattern classifier. In this model, the hyper-parameters of the SVM are optimized by the CS to get higher generalization performance. To verify this model’s effectiveness, the data sets of basic patterns in AR (1) type of autocorrelation process, including normal pattern and seven types of abnormal pattern, are generated by Monte Carlo simulation method. Based on the data sets, some verification experiments are conducted. The experiment results show that the proposed model has apparent advantages over some other methods, both in recognition accuracy and training efficiency. At the same time, it has an acceptable on-line detecting performance.

The paper is structured as follows. In Section 2, some related theories and methods are reviewed. In Section 3, the model is established, and its structure graph is presented. In Section 4, the environment and data of the simulation experiments are given and results are discussed. Section 5 concludes the paper with a summary and remarks. Description of the Monte Carlo simulation functions of the eight patterns is displayed in the Appendix A.

In the modern manufacturing process, the production tempo is getting faster and faster, and the automatic data acquisition technology is used more and more widely. As a result, the observed values of a specific variable get at different moment frequently present certain type of dependence with each other. That is called the autocorrelation phenomenon [

In

In actual production process, affected by certain assignable cause, process variable may takes on certain type of fluctuation, which can be seen from the control chart. Academia has defined these fluctuations as control chart patterns (CCP). There are eight major types of CCP, namely the Normal (NOR), the Upward Shift (US), the Downward Shift (DS), the Increasing Trend (IT), the Decreasing Trend (DT), the Cycle (CYC), the Systematic (SYS), and the Mixture (MIX). These CCPs can still take place in the process as there is autocorrelation exists. The

The classification problem using support vector machines can be described as follows. Given a sample set

SVM is essentially a hyper-plane

In

The most crucial characteristic of SVM is that it can transfer nonlinear separable samples in lower dimensional space to linear separable samples in higher dimensional space through using kernel function. The kernel function

There are many types of kernel function, including the radial basis function (RBF), the sigmoid kernel, the linear kernel and the polynomial kernel, etc. Literature reviews show that the SVM with RBF has good performance in control chart pattern recognition. Therefore, the SVM with RBF is used as the pattern classifier in this study. The formula of RBF is as follows:

The parameter

CS is a new meta-heuristic search optimization algorithm illuminated by the biological characteristics in nature [

among them

In

After updating the position by

The recognition model based on CS-SVM is shown in

Step 1: Initialize the basic parameters. When the probability that the host bird finds a non-self bird egg is 0.25, it is sufficient for most optimization problems, so

Step 2: The position and state of the non-optimal nest are updated by using

Step 3: After the nest position is updated, the random number

Step 4: If the maximum number of iterations or required accuracy is reached, the obtained optimal bird nest position is accepted as the global optimal solution; otherwise, turn back to step 2 and continue to iterate and update.

Step 5: Output the position of the global optimal nest, namely the best hyper-parameters

In order to verify the performance of the proposed model, which is established through programming in the MATLAB2018, and hereinto, the SVM is realized by the Libsvm toolbox. Then, some simulation experiments with this model are carried out in the MATLAB2018. The performance indicators of the computer used are CPU2.4GHZ, RAM12.0G. Following the conventional research methods in this field, eight types of basic CCP sample (as described in 2.1) of AR (1) process are generated by Monte carol simulation method. These samples are divided into three mutually exclusive sets, including the training set, the validation set and the test set. The number of samples for each pattern in each set is shown in

Data set | NOR | US | DS | IT | DT | CYC | SYS | MIX |
---|---|---|---|---|---|---|---|---|

Training set | 500 | 720 | 720 | 720 | 720 | 720 | 720 | 720 |

Validation set | 90 | 90 | 90 | 90 | 90 | 90 | 90 | 90 |

Test set | 90 | 90 | 90 | 90 | 90 | 90 | 90 | 90 |

Pattern type | Parameter value | Starting points |
---|---|---|

NOR | ||

US | ||

DS | ||

IT | ||

DT | ||

CYC | t = 0 | |

SYS | t = 0 | |

MIX | t = 0 |

For the patterns of US, DS, IT, and DT, different starting points of exception are set for different abnormity amplitudes. Small amplitudes have smaller abnormal starting points, for example, in US,

In each iteration of the CS, the CS-SVM model is trained after being given the hyper-parameters, and calculated recognition accuracy with the verification set. The result is taken as the fitness value. Without loss of generality, the hyper-parameters are optimized when the autocorrelation coefficient

Recognition accuracy test of this model is carried out on the test set. For the case of

The average accuracy of processes with some other different level of autocorrelation coefficients (

In order to further verify the superiority of this model, a comparative experiment was carried out based on the same data sets. The chosen comparative objects are the model based on SVM optimized by particle swarm optimization (PSO-SVM) and the model based on SVM optimized by genetic algorithm (GA-SVM). The reason is that both PSO and GA are intelligent optimization algorithms, which are often used to optimize the hyper-parameters of SVM. Five times of optimization are conducted for the case of autocorrelation coefficients

Model type | The setting of main parameters |
---|---|

CS-SVM | |

PSO-SVM | _{1} = 1.5, _{2} = 1.7, max |

GA-SVM | Max |

It can be seen from

Average run length (ARL) is a key indicator of the online performance of control chart or other process anomaly detectors. ARL can be measured through simulation method. The usual practice is to simulate lots of data streams first, and then fetch data from each data stream with a sliding window, till a normal pattern sample is misrecognized as abnormal (ARL_{0}) or an abnormal pattern is identified (ARL). In this study, 3000 in-control data streams and 3000 × 9 × 7 out-of-control data streams are separately generated for each autocorrelation level. The ARL_{0} reflexes the probability of occurrence of the type I error in the model, and should be controlled as close as possible to the theoretical value (370 for the univariate process) before measuring ARL. For that reason, the try and error method is applied, i.e., the number of the NOR sample in the train data set is adjusted according to the achieved ARL_{0} value till it reaches slightly higher than 370. In addition, the SRL (standard deviation of ARL) is also calculated to measure the stability of ARL performance.

In order to verify the online performance of the proposed CS-SVM based model, an SVM based model in reference [

ϕ | Model | US | DS | IT | DT | CYC | SYS | MIX | Aggregated average | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

ARL | SRL | ARL | SRL | ARL | SRL | ARL | SRL | ARL | SRL | ARL | SRL | ARL | SRL | ARL | SRL | ||

0.9 | CS-SVM based | 2.5 | 2.6 | 2.95 | 3.1 | 13.3 | 3.9 | 12.3 | 3.5 | 12.4 | 2.3 | 3.0 | |||||

SVM based | 9.8 | 3.6 | 9.6 | 4.0 | 12.3 | 3.90 | 12.4 | 4.2 | 10.7 | 1.7 | 10.8 | 2.3 | 10.7 | 2.8 | 10.9 | 3.2 | |

0.7 | CS-SVM based | 2.5 | 2.5 | 2.96 | 3.0 | 12.7 | 3.6 | 12.2 | 2.2 | 13.3 | 2.9 | 2.8 | |||||

SVM based | 9.2 | 3.4 | 9.1 | 3.0 | 13.6 | 2.90 | 12.5 | 3.1 | 11.3 | 3.4 | 10.3 | 2.1 | 10.2 | 2.6 | 10.9 | 2.9 | |

0.5 | CS-SVM based | 2.6 | 2.6 | 2.97 | 2.0 | 13.0 | 4.5 | 11.9 | 1.9 | 12.4 | 2.2 | 2.7 | |||||

SVM based | 8.9 | 2.2 | 8.5 | 1.8 | 13.5 | 2.20 | 13.4 | 2.2 | 12.4 | 3.2 | 10.1 | 1.7 | 10.3 | 2.0 | 11.0 | 2.2 | |

0.3 | CS-SVM based | 2.8 | 2.8 | 2.94 | 1.5 | 12.3 | 3.3 | 10.7 | 3.7 | 12.1 | 2.4 | 2.8 | |||||

SVM based | 8.5 | 1.4 | 8.2 | 1.5 | 11.4 | 1.70 | 13.5 | 1.7 | 12.2 | 2.7 | 10.5 | 1.6 | 10.0 | 1.8 | 10.6 | 1.8 | |

0.1 | CS-SVM based | 2.7 | 2.8 | 2.81 | 1.9 | 14.3 | 2.6 | 10.4 | 3.7 | 11.9 | 1.9 | 11.1 | 2.6 | ||||

SVM based | 7.9 | 1.3 | 7.7 | 1.1 | 13.0 | 1.30 | 13.2 | 1.2 | 13.0 | 2.8 | 11.1 | 1.5 | 10.7 | 1.9 | 10.9 | 1.6 | |

0 | CS-SVM based | 1.8 | 1.8 | 1.94 | 12.7 | 2.0 | 14.4 | 2.7 | 1.7 | 1.5 | 10.7 | 1.9 | |||||

SVM based | 8.1 | 1.1 | 8.4 | 1.2 | 11.3 | 1.30 | 10.4 | 1.3 | 9.5 | 2.4 | 10.6 | 1.6 | 10.4 | 1.8 | 9.8 | 1.5 | |

–0.1 | CS-SVM based | 8.1 | 2.8 | 7.1 | 2.8 | 13.6 | 2.82 | 12.8 | 2.9 | 15.1 | 2.9 | 1.4 | 1.4 | 10.9 | 2.4 | ||

SVM based | 7.3 | 1.1 | 7.5 | 1.1 | 13.8 | 1.20 | 13.7 | 1.2 | 12.1 | 2.5 | 10.3 | 1.9 | 10.7 | 1.5 | 10.8 | 1.5 | |

–0.3 | CS-SVM based | 7.7 | 2.8 | 7.8 | 3.0 | 13.1 | 2.90 | 13.3 | 3.1 | 15.5 | 2.2 | 3.1 | 1.7 | 2.7 | |||

SVM based | 7.0 | 1.1 | 7.2 | 1.0 | 13.2 | 1.10 | 13.5 | 1.1 | 12.5 | 2.3 | 12.0 | 2.1 | 12.2 | 1.8 | 11.1 | 1.5 | |

–0.5 | CS-SVM based | 7.0 | 2.9 | 8.4 | 3.0 | 14.1 | 3.08 | 13.6 | 3.2 | 16.2 | 2.2 | 2.9 | 3.5 | 3.0 | |||

SVM based | 7.2 | 1.0 | 7.1 | 1.0 | 13.5 | 1.00 | 13.8 | 0.9 | 13.2 | 2.4 | 12.4 | 2.9 | 12.1 | 2.3 | 11.3 | 1.6 | |

–0.7 | CS-SVM based | 8.8 | 2.9 | 8.5 | 3.0 | 13.9 | 3.00 | 13.6 | 3.0 | 17.0 | 6.1 | 2.9 | 4.6 | 3.6 | |||

SVM based | 6.6 | 0.9 | 6.9 | 0.9 | 13.4 | 1.10 | 13.5 | 1.1 | 12.8 | 2.3 | 14.1 | 4.9 | 15.2 | 4.6 | 11.8 | 2.3 | |

–0.9 | CS-SVM based | 8.8 | 3.0 | 8.8 | 3.1 | 13.9 | 2.97 | 13.8 | 3.0 | 18.2 | 6.8 | 2.6 | 8.4 | 4.3 | |||

SVM based | 7.0 | 1.1 | 6.8 | 1.0 | 13.3 | 1.20 | 13.0 | 1.3 | 12.4 | 2.1 | 17.6 | 7.3 | 17.6 | 6.5 | 12.5 | 2.9 |

It can be seen that the proposed model performs better for the US, DS, IT and DT pattern as

This study uses CS to optimize the two hyper-parameters of SVM (CS-SVM), and then based on which to establish a recognition model for abnormal patterns in autocorrelation process. A series of simulation experiments have been conducted to test the performances of this model. The experiment results show that the established model can achieve higher recognition accuracy in comparison with the model based on SVM optimized by the PSO or the GA. In the meantime, it takes much less time to optimize the hyper-parameters for this model. That means this model has higher training efficiency, considering that the parameter optimization procedure is involved in the training process. Furthermore, the model shows good recognition accuracy for each tested autocorrelation level (whether positive or negative autocorrelation), indicating its broad applicability. At last, the ARL values of the model at different autocorrelation levels are measured, which are generally better than those of the comparative model from the reference. That indicates the model also possesses an acceptable online performance. Identification of abnormal patterns in autocorrelation process is still in the exploratory stage, and the proposed model provides a new way for it. Currently, we have verified the model’s effectiveness through simulation experiments. Testing the model’s effectiveness for other types of autocorrelation processes except AR (1) and studying how to use it in the actual manufacturing process will be our next work.

I want to take this chance to thanks my tutor----Bo Zhu. In composing this paper, he gives me much academic and constructive advice and helps me correct my essay. Besides these, he also allowed me to do my teaching practice. At the same time, I want to thank my friends Chunmei Chen, Kaimin Pang and Yuwei Wan. They participated much in this research. Finally, I’d like to thank all my friends, especially my three lovely roommates, for their encouragement and support.