In addition to the exciting competition, the new technology involved in the 2022 Beijing Winter Olympics has also attracted widespread attention. The Visual & Data Intelligence Center (VDIC) of SIST and its incubated startup DGene, developed the “Free-Viewpoint Video Effects Live Application System” to serve the video broadcast of the Winter Olympic Games, providing viewers with a live, free-view of the games. For the characteristics of ice and snow sports in the Winter Olympics, ShanghaiTech’s team has created a faster and more reliable free-view system based on the existing research results of free-view technology, and participated in the application of free-view technology for ice curling and speed skating competitions in the Winter Olympics as one of the technical support units.
The Free-Viewpoint Video Effects Live Application System not only allows viewers to watch the game from a fixed viewpoint, but also features the time freeze effects that freeze exciting moments. In the broadcasting of ice curling and speed skating events, for example, the images of the athletes at the moment of throwing the pots and crossing the curves will be frozen, as if time has stopped. At the same time, the viewing angle starts to rotate, allowing viewers to have a more comprehensive understanding of the athletes’ position and trajectory at that instantaneous moment.
In order to meet the high requirements of the system application for the broadcast of the Winter Olympics, VDIC made a quick transfer of the previous research achievements in the fields of computer vision and deep learning, and they solved the multiple challenges in the implementation process, and achieved technology breakthroughs. This new technology solves the efficiency problem of traditional multi-camera calibration. With the help of the 3D calibration algorithm, rapid calibration can be performed under limited scene feature point information. The system can then quickly respond to the camera position changes during the game, and solve one challenge of the application related to the fact that features in the ice and snow event venues are simple, making it difficult to accurately and quickly calibrate the athelets.
At present, VDIC is working on combining the emerging neural network rendering technology and the traditional video layered rendering strategy, to realize the virtual perspective generation under the condition of sparse camera deployment with the support of the deep learning method and a large number of moving scene data. The goal of this work is to render editable free-viewpoint video with a wide viewing angle range. The related research results of this work have been published in the 2021 special issue of the journal ACM ToG - SIGGRAPH 2021, in an article entitled “Editable Free-Viewpoint Video Using a Layered Neural Representation”.