Sergey Tulyakov
Welcome! I am a Principal Research Scientist heading the Creative Vision team at Snap Research. My work focuses on creating methods for manipulating the world via computer vision and machine learning. This includes 2D and 3D methods for photorealistic object manipulation and animation, video synthesis, prediction and retargeting. My work has been published as 30+ top conference papers, journals and patents resulting in multiple tech transfers, including Snapchat Pet Tracking and Real-time Neural Lenses (gender swap, baby face).
Before joining Snap Inc., I built the real-time face tracking and reconstruction engine behind facify.me. 2016, I was a research intern at Microsoft Research, Cambridge, UK, and worked with Sebastian Nowozin and Andrew Fitzgibbon on improving generative models. In 2017, I interned at NVIDIA with Ming-Yu Liu, Xiaodong Yang and Jan Kautz and worked on video generation.
I am a PhD graduate from the University of Trento, Italy, where I worked with Nicu Sebe at the Multimedia and Human Understanding Group MHUG. Before joining the MHUG group, I worked as a software engineer for six years.
I am looking for prospective interns and collaborators. If you are interested send me an email.
Media coverage:
- Pet Tracking: The Sun, The Verge, Variety, CNet and others
- Real-time Neural Lenses: CNet, BoredPanda, Newsweek, The Telegraph and others
- Time Machine Lens: elite daily, Engadget, and others
- Image Animation: Two-minute papers, Vice, The Next Web, and others
Contact: stulyakov@snap.com
CV Google Scholar
Research
The ultimate goal of the Creative Vision group is to develop cutting edge computer vision, machine learning, and graphics technology to unlock creativity of users and creators in a variety of products at Snap. Our work enables machines to see, understand and manipulate the world. We focus on the following broad directions:
- Understanding Humans: Human digitization (segmentation, 3D pose and geometry of face, body, hand, clothes)
- Understanding the World: Digitization of the world and objects in it (recognition, segmentation, 3D reconstruction, scene understanding)
- Tools for Creativity: Arbitrary AR manipulation of the world, including style transfer for shape and texture of faces, objects and scenes, adding and removing objects from the scene
- Efficient Algorithms and Architectures: Technologies for faster, smaller models and more accurate inference
Work Experience
July 2017 - Present:
Principal Research Scientist at Snap Research, Santa Monica, CA
Jan 2017 - Apr 2017:
Research Intern at NVIDIA Research, Santa Clara, CA
Aug 2016 - Nov 2016:
Research Intern at Microsoft Research, Cambridge, UK
July 2010 - Sept 2012:
Team and Project Lead at HiQo-Solutions, Minsk, Belarus
June 2006 - June 2010:
Software Engineer at Todes Ltd, Minsk, Belarus
Education
Nov 2012 - Apr 2017:
PhD Canditate at The Univesity of Trento, Italy
Sept 2014 - Feb 2015:
Research intern at the Robotics Institute, Carnegie-Mellon University
Sept 2009 - July 2010:
MSc at the Belarusian State University of Intormatics and Radioelectronics
Sept 2004 - July 2009:
B.Eng at the Belarusian State University of Intormatics and Radioelectronics
Teaching
Teaching Computers to Imagine with Deep Generative Models
Sergey Tulyakov, Stéphane Lathuilière
University of Trento, Italy. November 19-26, 2019
Course material
Unlocking Creativity with Computer Vision: Representations for Animation, Stylization and Manipulation
June 20, CVPR Virtual 2021
Tutorial webpagePublications
R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis
European Conference on Computer Vision, 2022
Huan Wang, Jian Ren, Zeng Huang, Menglei Chai, Kyle Olszewski, Yun Fu, Sergey Tulyakov
Project page Paper Code
NeROIC: Neural Rendering of Objects from Online Image Collections
Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, Sergey Tulyakov
ACM Transactions on Graphics (SIGGRAPH), 2022
Project page Paper Code
Playable Environments: Video Manipulation in Space and Time
Willi Menapace, Stéphane Lathuilière, Aliaksandr Siarohin, Christian Theobalt, Sergey Tulyakov, Vladislav Golyanik, Elisa Ricci
Computer Vision and Pattern Recognition, 2022
Project page Paper Code
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski,Shervin Minaee, Dimitris Metaxas, Sergey Tulyakov
Computer Vision and Pattern Recognition, 2022
Project page Paper Code
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2
Ivan Skorokhodov, Sergey Tulyakov, Mohamed Elhoseiny
Computer Vision and Pattern Recognition, 2022
Project page Paper
In&Out: Diverse Image Outpainting via GAN Inversion
Yen-Chi Cheng, Chieh Hubert Lin, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Ming-Hsuan Yang
Computer Vision and Pattern Recognition, 2022
Project page Paper Code
[Oral] F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization
Qing Jin, Jian Ren, Richard Zhuang, Sumant Hanumante, Zhengang Li, Zhiyu Chen, Yanzhi Wang, Kaiyuan Yang, Sergey Tulyakov
International Conference on Learning Representations, 2022
Openreview Paper Code
InfinityGAN: Towards Infinite-Resolution Image Synthesis
Chieh Hubert Lin, Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, Ming-Hsuan Yang
International Conference on Learning Representations, 2022
Project page Paper Code
Flow Guided Transformable Bottleneck Networks for Motion Retargeting
Jian Ren, Menglei Chai, Oliver Woodford, Kyle Olszewski, Sergey Tulyakov
Computer Vision and Pattern Recognition, 2021
Paper
Motion Representations for Articulated Animation
Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, Sergey Tulyakov
Computer Vision and Pattern Recognition, 2021
Project page Paper Code
[Oral] Playable Video Generation
Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, Elisa Ricci
Computer Vision and Pattern Recognition, 2021
Project page Paper Code Demo
[Spotlight] A Good Image Generator Is What You Need for High-Resolution Video Synthesis
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, Sergey Tulyakov
International Conference on Learning Representations, 2021
Project page Paper Code
Interactive video stylization using few-shot patch-based training Best in Show Award at SIGGRAPH Real-Time Live!
ACM Transactions on Grahpics (SIGGRAPH), 2020
Paper Code Project page
Neural Hair Rendering
Menglei Chai, Jian Ren, Sergey Tulyakov
European Conference on Computer Vision, 2020
Paper
Towards Photo-Realistic Facial Expression Manipulation
Zhenglin Geng, Chen Cao, Sergey Tulyakov
International Journal of Computer Vision, 2020
Paper
Human Motion Transfer from Poses in the Wild
Jian Ren, Menglei Chai, Sergey Tulyakov, Chen Fang, Xiaohui Shen, Jianchao Yang
Image Manipulation Workshop, Computer Vision and Pattern Recognition, 2020
Paper
Laplace Landmark Localization
Joseph P Robinson, Yuncheng Li, Ning Zhang, Yun Fu, and Sergey Tulyakov
International Conference on Computer Vision, 2019
Paper
3D Guided Fine-Grained Face Manipulation
Zhenglin Geng, Chen Cao, Sergey Tulyakov
Computer Vision and Pattern Recognition, 2019
Paper
Train One Get One Free: Partially Supervised Neural Network for Bug Report Duplicate Detection and Clustering
Lahari Poddar, Leonardo Neves, William Brendel, Luis Marujo, Sergey Tulyakov, Pradeep Karuturi
North American Chapter of the Association for Computational Linguistics, 2019
Paper
Recurrent Convoltutional Shape Regression
Wei Wang, Sergey Tulyakov and Nicu Sebe
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
Paper
Hybrid VAE: Improving Deep Generative Models using Partial Observations
Sergey Tulyakov, Andrew Fitzgibbon, Sebastian Nowozin
Neural Information Processing Systems Workshops, 2017
Paper
Viewpoint-consistent 3D Face Alignment
Sergey Tulyakov, Laszlo A. Jeni, Jeffrey F. Cohn and Nicu Sebe
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
Paper
[Oral] Recurrent Convolutional Face Alignment
Wei Wang, Sergey Tulyakov and Nicu Sebe
Asian Conference on Computer Vision, 2016
Paper
Sergey Tulyakov, Xavier Alameda Pineda, Elisa Ricci, Jijun Yin, Jeffrey Cohn and Nicu Sebe
Computer Vision and Pattern Recognition, 2016
Read more
Regressing a 3D Face Shape from a Single Image
Sergey Tulyakov and Nicu Sebe
International Conference on Computer Vision, 2015
Read more
FaceCept3D: Real Time 3D Face Tracking and Analysis
Sergey Tulyakov, Radu Vieriu, Enver Sangineto and Nicu Sebe
International Conference on Computer Vision Workshops, 2015
Paper
