Kernel learning based on structure risk minimum can be employed to build a soft measuring model for analyzing small samples. However, it is difficult to select learning parameters, such as kernel parameter (KP) and regularization parameter (RP). In this paper, a soft measuring method is investigated to select learning parameters, which is based on adaptive multi-layer selective ensemble (AMLSEN) and least-square support vector machine (LSSVM). First, candidate kernels and RPs with K and R numbers are preset based on prior knowledge, and candidate sub-sub-models with K*R numbers are constructed through utilizing LSSVM. Second, the candidate sub-sub-models with same KPs and different RPs are selectively fused by using the branch and bound SEN (BBSEN) to obtain K SEN-sub-models. Third, these SEN-sub-models are selectively combined through using BBSEN again to obtain SEN models with different ensemble sizes, and then a new metric index is defined to determine the final AMLSEN-LSSVM-based soft measuring model. Finally, the learning parameters and ensemble sizes of different SEN layers are obtained adaptively. Simulation results based on the UCI benchmark and practical DXN datasets are conducted to validate the effectiveness of the proposed approach.

Data-driven soft measuring techniques can be used for online estimation of offline assay process parameters and experts’ estimation quantity variables [

Selective ensemble (SEN) modeling can selectively fuse multiple sub-models in a linear or nonlinear method and achieve better prediction performances than single modeling. However, SEN modeling still exerts several limitations. One of them is ensemble construction, which creates a set of candidate sub-models for the same training dataset. The ensemble construction method based on the resample of training samples validates the ensemble method, and many available sub-models can obtain better performances than the ensemble of all the sub-models [

In this paper, the SEN kernel leaning algorithm for small data-driven samples is adopted to softly measure the DXN emission. Although several SEN-LSSVMs such as fuzzy C-means cluster-based SEN-LSSVM [

Motivated by the above problems, this paper proposes a new adaptive multi-layer SEN-LSSVM (AMLSEN-LSSVM) method for modeling small samples with complex characteristics. Compared to the existing literatures, the distinctive contributions of this study are listed as follows. (1) A new SEN-LSSVM modeling framework is proposed to model small samples according to the candidate learning parameters at the first time; (2) The proposed method can adaptively select the kernel and RPs simultaneously. Moreover, the ensemble sizes of different SEN layers are also adaptively determined in implicit pattern; (3) A new metric index is defined for selecting the final AMLSEN-LSSVM soft measuring model to achieve tradeoff between model complexity and prediction performance.

The remainder of this paper is organized as follows. Section 2 describes the proposed modeling strategy. In Section 3, a detailed realization of the proposed approach is demonstrated. Section 4 presents the experimental results on two UCI benchmarks and practical DXN datasets for references. Finally, Section 5 concludes this paper.

Based on the above analysis, an adaptive multi-layer SEN-LSSVM (AMLSEN-LSSVM) soft measuring strategy is proposed, which consists of candidate sub-sub-models, candidate SEN-sub-models, SEN-models, and decision selection modules. The process is shown in

In

The functions of the different modules are illustrated as follows:

(1) Candidate sub-sub-model module: Construct the

(2) Candidate SEN-sub-model module: Construct the K SEN-sub-models by using the prediction outputs of

(3) SEN-model module: Construct the SEN-models with an ensemble size from 2 to (

(4) Decision selection module: Select the final soft measuring model from SEN-models with the different ensemble sizes by defining a new metric index and making a trade-off between prediction accuracy and model complexity.

As the candidate kernels and RPs of the LSSVM model are respectively denoted as

where

Taken the

where

where

where

Accordingly, the above problem becomes a linear equation system:

By solving the above system,

For a concise expression, (7) can also be expressed as

Therefore, all these candidate sub-sub-models are denoted as

The prediction outputs of these candidate sub-sub-models can be rewritten as

where

In (9), the

The candidate SEN-sub-models are built for each row of (10) by selecting and combining the candidate sub-sub-models based on different

where

where

where the root mean square error (RMSE) is used to evaluate the generalization performance of the SEN-sub-model with ensemble size

In order to solve the above problem, the optimized ensemble sub-sub-models and their weighted coefficients are obtained through repeating the BBSEN optimization algorithm

where

For simplification, all the RPs of the

Based on the above sub-sections, the SEN-sub-models with the same KPs and different RPs are obtained. Then,

In this context, BBSEN is further employed to obtain the prediction output

where

where

where

Accordingly, the SEN model

In

Supposed that

where

According to the above metric index, the SEN-model with the lowest RMSE is selected as the final DXN soft measuring model. Thus, the final prediction output is obtained by

where

In this section, two UCI benchmarks and a practical DXN datasets are exploited to validate the proposed methods. Primarily, each dataset is divided into two parts: training data and testing data. Then radius basis function (RBF) is used for the kernel type of the LSSVM. Finally, the candidate kernel and RP datasets are selected as Can_1{0.1, 1, 100, 1000, 2000, 4000, 6000, 8000, 10000, 20000, 40000, 60000, 80000, 160000} and Can_2{0.1, 1, 6, 12, 25, 50, 100, 200, 400, 800, 1600, 3200, 6400, 12800, 25600, 51200, 102400}, respectively. Seen from the samples, Can_1 and Can_2 illustrate that the candidate learning parameters have a wide range. The order number and their values are shown in

0.1 | 1 | 100 | 1000 | 2000 | 4000 | |

0.1 | 1 | 6 | 12 | 25 | 50 | |

6000 | 8000 | 10000 | 20000 | 40000 | 60000 | |

100 | 200 | 400 | 800 | 1600 | 3200 | |

80000 | 160000 | -- | -- | -- | -- | |

6400 | 12800 | 25600 | 51200 | 102400 | -- |

In order to present preliminary results, two UCI benchmark datasets, Boston housing and concrete compressive strength, are used to validate the proposed method, which are listed as follows.

For Boston housing data, the inputs include: (1) the

For concrete compressive strength data, the inputs include: (1) cement; (2) blast furnace slag; (3) fly ash; (4) water; (5) superplasticizer; (6) coarse aggregate; (7) fine aggregate of the various ingredients of concrete placement per cubic meter; and (8) conserved days. The output is concrete compressive strength with data size 1030.

According to the candidate learning parameters, 14 × 17 = 238 sub-sub-models based on the LSSVM are constructed. Correspondingly, the BBSEN method is employed to build 14 SEN-sub-models. The statistical results are shown in

Order | Kernel parameter(KP) | Models | Boston housing data | Concrete compressive strength data | ||||
---|---|---|---|---|---|---|---|---|

RMSE |
Regularization parameter (RP,Order number) | Ensemble size | RMSE |
Regularization parameter (RP,Order number) | Ensemble size | |||

1 | 0.1 | EnAll-sub-sub | 8.9735 | all | 17 | 16.1059 | all | 17 |

SEN-sub | 8.5902 | 16–17 | 2 | 15.0883 | 15–17 | 2 | ||

Best-sub-sub | 8.5902 | 17 | 1 | 15.0883 | 17 | 1 | ||

2 | 1 | EnAll-sub-sub | 6.9940 | all | 17 | 12.8921 | all | 17 |

SEN-sub | 5.1031 | 16–17 | 2 | 12.4234 | 2–17 | 6 | ||

Best-sub-sub | 5.1031 | 17 | 1 | 11.8064 | 6 | 1 | ||

3 | 100 | EnAll-sub-sub | all | 17 | all | 17 | ||

SEN-sub | 4–17 | 14 | 4–17 | 4 | ||||

Best-sub-sub | 8 | 1 | 10 | 1 | ||||

4 | 1000 | EnAll-sub-sub | 6.5492 | all | 17 | 15.9703 | all | 17 |

SEN-sub | 3.2483 | 10–17 | 8 | 8.1098 | 11–17 | 7 | ||

Best-sub-sub | 3.2637 | 13 | 1 | 8.1362 | 13 | 1 | ||

5 | 2000 | EnAll-sub-sub | 7.8296 | all | 17 | 16.4346 | all | 17 |

SEN-sub | 3.2653 | 13–17 | 5 | 8.1475 | 14–17 | 4 | ||

Best-sub-sub | 3.2681 | 15 | 1 | 8.1459 | 15 | 1 | ||

6 | 4000 | EnAll-sub-sub | 8.5535 | all | 17 | 16.6256 | all | 17 |

SEN-sub | 3.2763 | 16–17 | 2 | 8.1642 | 16–17 | 2 | ||

Best-sub-sub | 3.2705 | 17 | 1 | 8.1511 | 17 | 1 | ||

7 | 6000 | EnAll-sub-sub | 8.7715 | all | 17 | 16.6813 | all | 17 |

SEN-sub | 3.3318 | 16–17 | 2 | 8.3123 | 16–17 | 2 | ||

Best-sub-sub | 3.3022 | 17 | 1 | 8.2182 | 17 | 1 | ||

8 | 8000 | EnAll-sub-sub | 8.8700 | all | 17 | 16.7074 | all | 17 |

SEN-sub | 3.4023 | 16–17 | 2 | 8.5411 | 16–17 | 2 | ||

Best-sub-sub | 3.3586 | 17 | 1 | 8.3811 | 17 | 1 | ||

9 | 10000 | EnAll-sub-sub | 8.9245 | all | 17 | 16.7225 | all | 17 |

SEN-sub | 3.4733 | 16–17 | 2 | 8.7862 | 16–17 | 2 | ||

Best-sub-sub | 3.4187 | 17 | 1 | 8.5831 | 17 | 1 | ||

10 | 20000 | EnAll-sub-sub | 9.0216 | all | 17 | 16.7514 | all | 17 |

SEN-sub | 3.7978 | 16–17 | 2 | 9.7130 | 16–17 | 2 | ||

Best-sub-sub | 3.7030 | 17 | 1 | 9.4900 | 17 | 1 | ||

11 | 40000 | EnAll-sub-sub | 9.0632 | all | 17 | 16.7652 | all | 17 |

SEN-sub | 4.2073 | 16–17 | 2 | 10.4129 | 16–17 | 2 | ||

Best-sub-sub | 4.1183 | 17 | 1 | 10.2943 | 17 | 1 | ||

12 | 60000 | EnAll-sub-sub | 9.0759 | all | 17 | 16.7697 | all | 17 |

SEN-sub | 4.3858 | 16–17 | 2 | 10.6267 | 16–17 | 2 | ||

Best-sub-sub | 4.3268 | 17 | 1 | 10.5610 | 17 | 1 | ||

13 | 80000 | EnAll-sub-sub | 9.0821 | all | 17 | 16.7719 | all | 17 |

SEN-sub | 4.4689 | 16–17 | 2 | 10.7145 | 16–17 | 2 | ||

Best-sub-sub | 4.4297 | 17 | 1 | 10.6731 | 17 | 1 | ||

14 | 16000 | EnAll-sub-sub | 9.0910 | all | 17 | 16.7752 | all | 17 |

SEN-sub | 4.5524 | 16–17 | 2 | 10.8124 | 16–17 | 2 | ||

Best-sub-sub | 4.5645 | 17 | 1 | 10.7968 | 17 | 1 |

For the 14 SEN-sub-models, the BBSEN method is used again to obtain the SEN-model with a different ensemble size, and the detailed statistical results of the SEN-models are illustrated in

Ensemble Size | SEN-sub-model (KP) number | RMSE | Ensemble size | Metric index |
---|---|---|---|---|

2 | 2 1 | 6.5189 | 4 | 0.07497 |

3 | 3 2 1 | 4.7546 | 18 | 0.07358 |

4 | 4 3 2 1 | 4.0554 | 26 | 0.07610 |

5 4 3 2 1 | 3.7270 | 31 | 0.07884 | |

6 | 6 5 4 3 2 1 | 3.5499 | 33 | 0.07944 |

7 | 7 6 5 4 3 2 1 | 3.4546 | 35 | 0.08093 |

8 | 8 7 6 5 4 3 2 1 | 3.4036 | 37 | 0.08289 |

9 | ||||

10 | 10 9 8 7 6 5 4 3 2 1 | 3.3827 | 41 | 0.08767 |

11 | 11 10 9 8 7 6 5 4 3 2 1 | 3.4121 | 43 | 0.09050 |

12 | 12 11 10 9 8 7 6 5 4 3 2 1 | 3.4538 | 45 | 0.09345 |

13 | 13 12 11 10 9 8 7 6 5 4 3 2 1 | 3.5011 | 47 | 0.09646 |

Ensemble Size | SEN-sub-model (KP) number | RMSE | Ensemble size | Metric index |
---|---|---|---|---|

2 | 2 1 | 12.8102 | 8 | 0.07228 |

3 | 3 2 1 | 9.9116 | 12 | 0.06506 |

4 | 4 3 2 1 | 8.9488 | 19 | 0.07158 |

5 4 3 2 1 | 8.5528 | 23 | 0.07602 | |

6 | 6 5 4 3 2 1 | 8.3599 | 25 | 0.07827 |

7 | 7 6 5 4 3 2 1 | 8.2641 | 27 | 0.08097 |

8 | 8 7 6 5 4 3 2 1 | 8.2247 | 29 | 0.08393 |

9 | ||||

10 | 10 9 8 7 6 5 4 3 2 1 | 8.2942 | 33 | 0.09054 |

11 | 11 10 9 8 7 6 5 4 3 2 1 | 8.4228 | 35 | 0.09429 |

12 | 12 11 10 9 8 7 6 5 4 3 2 1 | 8.5617 | 37 | 0.09808 |

13 | 13 12 11 10 9 8 7 6 5 4 3 2 1 | 8.6963 | 39 | 0.10185 |

The above results show that it is essential to make a tradeoff between prediction performance and model complexity. The proposed indices of different SEN-models with

Seen form

The above results show that the proposed method can model the two UCI benchmark datasets effectively, which can make an adaptive selection of the learning parameters.

The proposed method is compared with the PLS, KPLS, GASEN-BPNN, and GASEN-LSSVM approaches. Concretely, the number of latent variables (LVs) and kernel LVs (KLVs) are decided by the leave-one-out cross-validation, and the number of hidden nodes of the GASEN-BPNN is set to two times plus one of the original inputs’ number. Thus, the structure of BPNN is 13-27-1 for the housing data, and 8-16-1 for concrete data. Meanwhile, the learning parameters of GASEN-LSSVM are used as ones of the best sub-sub-model in the proposed method. For GASEN-BPNN and GASEN-LSSVM, the modelling process is repeated 10 times to overcome their randomness.

Boston housing data | Concrete compressive strength data | Denote | |||||
---|---|---|---|---|---|---|---|

RNSEs | RMSEs | ||||||

Max | Mean | Min | Max | Mean | Min | ||

PLS | -- | 4.681 | -- | -- | 10.92 | -- | LVs = 4/ LVs = 7 |

KPLS | -- | 3.195 | -- | -- | 8.179 | -- | KLVs = 8/ KLVs = 8 |

GASEN-BPNN | 11.2478 | 8.9307 | 5.0509 | 14.8580 | 12.0756 | 10.3971 | (13-27-1)/(8-17-1) |

GASEN-LSSVM | 3.3480 | 3.2899 | 3.2393 | 10.1777 | 10.0149 | 9.7490 | (100,200)/(100,800) |

This paper | 3.027 | 7.163 |

Although the GASEN-based prediction results are disturbed within a certain range due to the random initialization of the input weights and the bias of BPNN and population GA, the PLS/KPLS methods can activate the extracted latent variables to construct a linear or nonlinear model with a single KP for stabilization.

KP determines the geometry of the feature space, which can be calculated by using the Fisher discrimination for classification problem. Actually, due to the deficiency of the range obtained for the regression problem, the candidate KP set in our proposed method serves as the key factor to obtain an effective soft sensor model. In addition, a wide range of KPs must be used for very small sample data with complex characteristics, such as DXN.

RP is always decided by the cross validation or other optimization methods. In terms of results, a small number of modeling samples require more RPs, that is, the ensemble size increases with decrease of the number of samples. Moreover, a large KP requires a small number of RPs. As a result, the reasonable range should also be optimized in the future studies.

Selecting a suitable number of ensemble sub-models from candidates in the SEN-modeled process means that more ensemble sub-models leads to the more complex method, which implies sub-sub-models can measure the complexity of the proposed AMLSEN method. In practice, the ensemble size of the SEN-model in the proposed method is selected to make a trade-off between prediction accuracy and model complexity. Additionally, the ensemble size of the SEN-sub-model is implicitly determined by using the BBSEN algorithm in terms of prediction performance. Thus, the proposed method has a flexible structure to demonstrate the conclusion that the ensemble size increases with the decrease of the number of training samples.

A new soft measuring method is proposed based on the AMLSEN-LSSVM algorithm, and many sub-sub-models based on the different candidate learning parameters are constructed by using the LSSVM. According to the same KP, these candidate sub-sub-models are selectively fused to obtain SEN-sub-models by using the BBSEN, which are selectively combined by using the BBSEN again for building SEN models with different ensemble sizes. Ultimately, the final soft measuring model is determined based on a newly defined metric index. The simulation results based on the benchmark datasets show the effectiveness of the proposed method, which demonstrate that not only the proposed method can be used for softly measuring other difficult-to-measure process parameters in different industrial processes, but also a more common adaptive multi-layer SEN modeling framework can be explored.