|
Published Articles >> Table of Contents >> Abstract
March 2004 (Vol. 26, No. 3)
pp. 322-336
Automatic Eyeglasses Removal from Face Images
Chenyu Wu
Ce Liu
Heung-Yueng Shum, IEEE
Ying-Qing Xu
Zhengyou Zhang, IEEE
Full Article Text:
   
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TPAMI.2004.1262319
Send link to a friend
| Abstract |
|
In this paper, we present an intelligent image editing and face synthesis system that automatically removes eyeglasses from an input frontal face image. Although conventional image editing tools can be used to remove eyeglasses by pixel-level editing, filling in the deleted eyeglasses region with the right content is a difficult problem. Our approach works at the object level where the eyeglasses are automatically located, removed as one piece, and the void region filled. Our system consists of three parts: eyeglasses detection, eyeglasses localization, and eyeglasses removal. First, an eye region detector, trained offline, is used to approximately locate the region of eyes, thus the region of eyeglasses. A Markov-chain Monte Carlo method is then used to accurately locate key points on the eyeglasses frame by searching for the global optimum of the posterior. Subsequently, a novel sample-based approach is used to synthesize the face image without the eyeglasses. Specifically, we adopt a statistical analysis and synthesis approach to learn the mapping between pairs of face images with and without eyeglasses from a database. Extensive experiments demonstrate that our system effectively removes eyeglasses.
|
 References
|
[1] S. Baker and T. Kanade, “Hallucinating Faces,” Proc. Fourth Int'l Conf. Automatic Face and Gesture Recognition, 2000.
[2] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballsester, Image Inpainting Proc. SIGGRAPH 2000 Conf., pp. 417-424, 2000.
[3] L. Bookstein, "Principal Warps: Thin-Plate Splines and the Decomposition of Deformations," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 6, pp. 567-585, June 1989.
[4] H. Chen, Y.Q. Xu, H.Y. Shum, S.C. Zhu, and N.N. Zhen, Example-Based Facial Sketch Generation with Non-Parametric Sampling Proc. Int'l Conf. Computer Vision, pp. 433-438, July 2001.
[5] T. Cootes and C. Taylor, Statistical Models of Appearance for Computer Vision technical report, Univ. of Manchester, 2000.
[6] E. Mortensen and W. Barrett, Intelligent Scissor for Image Composition Proc. SIGGRAPH 1995 Conf., pp. 191-198, 1995.
[7] D.A. Forsyth, J. Haddon, and S. Ioffe, The Joy of Sampling Int'l J. Computer Vision, vol. 41, nos. 1/2, pp. 109-134, 2001.
[8] W.T. Freeman and E.C. Pasztor, Learning Low-Level Vision Proc. Seventh IEEE Int'l Conf. Computer Vision, pp. 1182-1189, 1999.
[9] S. Geman and D. Geman, Stochastic Relaxation, Gibbs Distribution and the Bayesian Restoration of Images IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, no. 6, pp. 721-741, June 1984.
[10] A. Hertzmann, C.E. Jacobs, N. Oliver, B. Curless, and D.H. Salesin, Image Analogies Proc. SIGGRAPH '01 Conf., Aug. 2001.
[11] X. Jiang, M. Binkert, B. Achermann, and H. Bunke, Towards Detection of Glasses in Facial Images Proc. Int'l Conf. Pattern Recognition, pp. 1071-1973, 1998.
[12] Z. Jing and R. Mariani, Glasses Detection and Extraction by Deformable Contour Proc. Int'l Conf. Pattern Recognition, 2000.
[13] T. Leung, M. Burl, and P. Perona, "Finding Faces in Cluttered Scenes Using Labeled Random Graph Matching," Int'l Conf. Computer Vision, 1995, pp. 637-644.
[14] C. Liu and H.Y. Shum, Kullback-Leibler Boosting and Its Applications in Face Detection Proc. Computer Vision and Pattern Recognition Conf., 2003.
[15] C. Liu, H. Shum, and C. Zhang, "A Two-Step Approach to Hallucinating Faces: Global Parametric Model and Local Non-Parametric Model," Proc. Int'l Conf. Computer Vision (ICCV), vol. I,IEEE CS Press, Los Alamitos, Calif., 2001, pp. 192-198.
[16] C. Liu, S.C. Zhu, and H.Y. Shum, Learning Inhomogeneous Gibbs Model of Face by Minimax Entropy Proc. Int'l Conf. Computer Vision, 2001.
[17] Z.C. Liu, Y. Shan, and Z. Zhang, Expressive Expression Mapping with Ratio Images Proc. SIGGRAPH '01 Conf., Aug. 2001.
[18] N. Metropolis, A.W. Rosenbluth, M.N. Rosenbluth, A.H. Teller, and E. Teller, Equations of State Calculations by Fast Computing Machines J. Chemical Physics, vol. 21, pp. 1087-1091, 1953.
[19] E. Osuna, R. Freund, and F. Girosi, Training Support Vector Machines: An Application to Face Detection Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 17-19, 1997.
[20] P. Pérez, A. Blake, and M. Gangnet, Jetstream: Probabilistic Contour Extraction with Particles Proc. Int'l Conf. Computer Vision, July 2001.
[21] P.J. Phillips, H. Moon, P. Rauss, and S.A. Rizvi, "The FERET Evaluation Methodology for Face-Recognition Algorithms," Computer Vision and Pattern Recognition, pp. 137-143, 1997.
[22] H. Rowley, S. Baluja, and T. Kanade, "Neural Network-Based Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, Jan. 1998, pp. 23-38.
[23] Y. Saito, Y. Kenmochi, and K. Kotani, “Estimation of Eyeglassless Facial Images Using Principal Component Analysis,” Proc. Int'l Conf. Image Processing, vol. 4, pp. 197-201, Oct. 1999.
[24] H. Schneiderman and T. Kanade, A Statistical Method for 3D Object Detection Applied to Faces and Cars Proc. Seventh IEEE Int'l Conf. Computer Vision, May 2000.
[25] K.K. Sung and T. Poggio, "Example-Based Learning for View-Based Human Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, pp. 39-50, Jan. 1998.
[26] Z. Tu, S. Zhu, and H. Shum, Image Segmentation by Data Driven Markov Chain Monte Carlo Proc. Int'l Conf. Computer Vision, vol. 2, pp. 131-138 July 2001.
[27] P. Viola and M. Jones, Robust Real-Time Face Detection Proc. Eighth IEEE Int'l Conf. Computer Vision, July 2001.
[28] Z. Wen, T. Huang, and Z. Liu, Face Relighting with Radiance Environment Maps Proc. Computer Vision and Pattern Recognition Conf., 2003.
[29] C.Y. Wu, C. Liu, and J. Zhou, Eyeglass Existence Verification by Support Vector Machine Proc. Second IEEE Pacific-Rim Conf. Multimedia, Oct. 2001.
[30] A.L. Yuille, D.S. Cohen, and P. Hallinan, Feature Extraction from Faces Using Deformable Templates Int'l J. Computer Vision, vol. 8, no. 2, pp. 99-112, 1992.
[31] S.C. Zhu, Y.N. Wu, and D. Mumford, Filter, Random Field and Maximum Entropy Neural Computation, vol. 9, no. 7, Nov. 1997.
|
 Additional Information
|
Index Terms- Intelligent image editing, find-and-replace, eye region detection, eyeglasses localization, eyeglasses removal.
Citation:
Chenyu Wu, Ce Liu, Heung-Yueng Shum, Ying-Qing Xu, Zhengyou Zhang,
"Automatic Eyeglasses Removal from Face Images,"
IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 26,
no. 3,
pp. 322-336,
Mar.,
2004
|
|
RSS Feed