In this paper, we proposed a combined PCA-LPP algorithm to improve 3D face reconstruction performance. Principal component analysis (PCA) is commonly used to compress images and extract features. One disadvantage of PCA is local feature loss. To address this, various studies have proposed combining a PCA-LPP-based algorithm with a locality preserving projection (LPP). However, the existing PCA-LPP method is unsuitable for 3D face reconstruction because it focuses on data classification and clustering. In the existing PCA-LPP, the adjacency graph, which primarily shows the connection relationships between data, is composed of the e-or k-nearest neighbor techniques. By contrast, in this study, complex and detailed parts, such as wrinkles around the eyes and mouth, can be reconstructed by composing the topology of the 3D face model as an adjacency graph and extracting local features from the connection relationship between the 3D model vertices. Experiments verified the effectiveness of the proposed method. When the proposed method was applied to the 3D face reconstruction evaluation set, a performance improvement of 10% to 20% was observed compared with the existing PCA-based method.

Facial modeling has a wide range of applications in visual and graphic computing, including facial tracking, emotion recognition, and interactive image and video-editing tasks related to multimedia. A three-stage development process has been implemented for face modeling: face modeling using parametric surfaces, face reconstruction using 3D data interpolation, and linear hybrid-face modeling [

The PCA [

In this study, a PCA algorithm that can effectively express global information and an LPP method that can preserve local information were combined. The disadvantages of the two algorithms were supplemented, and the PCA-LPP algorithm [

PCA is considered only a global feature. The disadvantage of PCA is that it does not consider regional features; thus, the reconstruction rate of regional features, such as lips, is fairly low. To compensate for this, we also considered local features by combining PCA and LPP. Thus, the reconstruction efficiency rate for the eyes and lips of the 3D face model improved.

A previous study used 68 landmarks, but the reconstruction rate of the nose was unnoticeable. We improved the recovery rate using the 74 landmarks provided in the FacewareHouse Database.

A high demand for face modeling technology exists in a wide range of multimedia applications. The face modeling process is complex, difficult, and time consuming. To improve this face modeling process, several studies have been conducted on techniques for reconstructing a 3D face model from an image. Representative face reconstruction methods include 3DMM for parameterizing and reconstructing face information and end-to-end face reconstruction techniques. The 3DMM method was first proposed by Blanz and Vetter at the University of Basel in Switzerland in 1999 [

Among end-to-end 3D face reconstruction methods, the most famous are VRNet [

Although PCA has the advantage of effectively representing global information, representing the local information of complex data, such as multidimensional manifold structures, is difficult. However, LPP can effectively represent the data of a multidimensional manifold structure. LPP is advantageous in preserving local information by projecting data while preserving connected neighbor information after identifying connections between the data. However, LPP has the disadvantage of requiring a large number of eigenvectors to represent global information. The PCA-LPP combination algorithm can be used to compensate for the shortcomings of these two methods. The existing PCA-LPP algorithm was implemented for data classification or clustering; however, it did not obtain good results for use in 3D face reconstruction. This section introduces the PCA, LPP, and PCA-LPP methods and presents the proposed method as a combined PCA-LPP algorithm.

PCA is the best-known dimensionality reduction technique for determining the basis vector with the greatest variance when projecting data. These basis vectors are identical to the eigenvectors of the data-covariance matrix. Therefore, easily extracting feature vectors from data is possible by decomposing the covariance matrix of the data into singular value decomposition (SVD) [

LPP is a technique that preserves and projects neighboring relationships between data to reduce dimensions, which can effectively represent regional characteristics. In the next section, we introduce how to extract the feature vectors from the LPP.

The adjacency graph [

A weight matrix W [

We calculate eigenvectors and eigenvalues using

However, LPP is not widely used for 3D face reconstruction. Because LPP preserves local information well, it presents good results in reconstructing detailed geometric information; however, this is because numerous feature vectors are required to reconstruct the overall silhouette.

The PCA-LPP is an algorithm that complements each method’s weaknesses by combining the advantages of effectively containing the global information of the PCA and LPP, which can effectively represent the local characteristics. The authors of [

The PCA-LPP implementation used in [

LPP is an effective method for finding feature vectors when data are nonlinear manifolds. However, when a feature vector is constructed using only LPP, most feature vectors must be used for reconstruction because feature values are almost identical. As PCA considers only the overall characteristics of the data, the reconstruction of detailed parts may be insufficient. Therefore, we solved this problem by combining PCA and LPP. Additionally, traditional PCA-LPP combination methods [

Data matrix X is transposed as the common connection relation of the data and used as the weight matrix. After solving the LPP problem, data matrix T is constructed using

X = Data Matrix (vertex count * model count)

F = Topology information in face model

A = Adjacency matrix created by connecting the vertices of the mesh in F

D = diag (A’s sum)

L = D−A

L,

T

S = T–T’ mean

v,

v =

To test the performance of our proposed PCA-LPP method, we modified it to fit the open-source published Facewarehouse data [

Detecting face in a single image

Face detection is performed using a multitask cascaded convolutional neural network (MTCNN) [

Detecting face landmarks on the detected face

Dlib [

Rigid tracking

Face reconstruction consists of two processes: rigid and non-rigid tracking. Rigid tracking is the process of aligning a face in an image with the prepared template model. The template model is the average face model of the Facewarehouse database, and when aligning the model and face in the image, it is sorted based on the landmark of the face. The template model is rotated and translated until the L2 norm between the landmark of the image acquired using dlib and the landmark of the projected template model is minimized.

Rigid tracking uses landmark loss with the L2 norm as a process to match the original image’s size and orientation to the template model.

As shown in

Non-rigid tracking

Non-rigid tracking optimizes the aligned template model equally to the face in the image. In this step, the identity, texture, and expression coefficients that determine the appearance of the model are estimated using the L2 loss as follows:

Here,

This section introduces the experimental environment used to reconstruct a face from a single image. The operating system performed deep learning and experimentation using MS Windows 10. One Nvidia GeForce RTX 3080Ti GPU was used for deep learning, as well as Python 3.8. Pytorch3D [

To evaluate the proposed method, face reconstruction was performed using an image provided by Facewarehouse, and the reconstructed model was evaluated by comparing it to the original face model provided by Facewarehouse. For comparisons between algorithms, the error of the reconstruction results using the proposed method and those using the PCA or LPP algorithms alone were compared with the original model. Landmark and photo losses acquired during tracking are the results to be compared with the image input for reconstruction; therefore, they are unsuitable for the evaluation and excluded from the evaluation process. We calculated the L2 norm between the vector and original model using the face image in Facewarehouse as input.

The evaluation method measures and compares the model loss, which is given by

Model number | PCA | LPP | Proposed method |
---|---|---|---|

Model No. 6 | 1.44e−3 | 1.68e−3 | 0.96e−3 |

Model No. 16 | 2.73e−3 | 4.41e−3 | 2.24e−3 |

Model No. 17 | 3.21e−3 | 4.60e−3 | 2.04e−3 |

Model No. 48 | 1.90e−3 | 2.29e−3 | 1.05e−3 |

Model No. 95 | 1.93e−3 | 2.32e−3 | 16e−3 |

Model No. 107 | 2.22e−3 | 3.37e−3 | 2.16e−3 |

Model No. 117 | 1.50e−3 | 1.71e−3 | 1.25e−3 |

Model No. 142 | 2.50e−3 | 3.51e−3 | 2.18e−3 |

Model No. 150 | 1.99e−3 | 2.72e−3 | 1.52e−3 |

We compared the reconstructed face models using the PCA and LPP algorithms with the proposed method. The reconstruction results in

However, because the proposed method includes both global and local feature, it is easier to get stuck in local minima problem than the PCA, which values global information. In addition, the reconstruction performance is notably reduced in cases where the face direction exceeds 30 degrees in a large pose or concealed part of the face. Thus, to obtain good results using the proposed method, using input image that face the front and avoiding the local minima problem, such as regularization, is important.

In this paper, we proposed a combined PCA-LPP algorithm that improves the existing feature vector extraction method to improve the 3D face reconstruction performance. Unlike the existing PCA-LPP method, the focus was on 3D face reconstruction, and the combined PCA-LPP algorithm was implemented using the 3D face mesh as an adjacency graph. The combined PCA-LPP algorithm exhibited a better reconstruction performance than the existing 3D face reconstruction methods implemented with feature vectors extracted from PCA and LPP. In particular, the performance improved by 10%–20% compared with the PCA algorithm, which is often used in the existing face reconstruction field. Therefore, the combined PCA-LPP algorithm yielded better results than the conventional method for 3D face reconstruction, as demonstrated by our results.

However, the proposed method was used only for 3D face reconstruction and is valid only for extracting the parameters of the feature vectors without training on the Facewarehouse dataset. In future work, we plan to compare the loss using feature vectors extracted from databases other than Facewarehouse. In this research, because the reconstruction rate of large pose face images was not excellent, we intend to search for a landmark method that is good for large poses and increases the face reconstruction rate by applying a neural network structure.

Thanks to Professor Kun Zhou for supporting Facewarehouse at Zhejiang University.