The development of powerful deep learning technologies has brought about some new and innovative ways to impact both society and individuals. But it has also led to the emergence of fake media and deepfakes. To tackle the issue, AI Singapore organized the Trusted Media Challenge (TMC) to explore how Artificial Intelligence (AI) technologies could be leveraged to combat fake media.
ByteDance's researcher - Weimin Wang's - own submission won the top prize at this competition! He attended the awarding ceremony today together with the BytePlus team to give a presentation in the cover image (Photo: Courtesy of AI Singapore (AISG))
ByteDance's researcher - Weimin Wang's - own submission won the top prize at this competition! He attended the awarding ceremony today together with the BytePlus team to give a presentation in the cover image (Photo: Courtesy of AI Singapore (AISG))
Combating misinformation
Deepfakes have become a worldwide concern since they can be incredibly convincing. This technology can be abused to create realistic videos that serve a negative purpose, such as spreading misinformation.
The team's winning solution to tackling this challenge is best described as a divide-and-conquer approach. The team divided the overall task of deepfake detection into two sub-tasks:
Deepfakes have become a worldwide concern since they can be incredibly convincing. This technology can be abused to create realistic videos that serve a negative purpose, such as spreading misinformation.
The team's winning solution to tackling this challenge is best described as a divide-and-conquer approach. The team divided the overall task of deepfake detection into two sub-tasks:
- 1. Identifying any manipulation to the face, audio, or a combination of both in the dataset.
- 2. Identifying lip motions inconsistent with the audio in the dataset.
In the first task, the team trained multimodal deep learning classifiers based on the original binary labels from the dataset using both facial and speech cues from each video, and to generate the baseline predictions of whether each video is fake.
The more challenging task 2 was then further divided into sub-tasks that can be easily solved using 2 independent, single-modal classifiers for each style of Professional or Amateur videos. Finally, if any test video whose classifiers do not agree with the same style, it will be marked as a potential fake thus given a penalty score.
The aggregate score from both tasks 1 and 2 were then used to predict the final fake-ness of the videos. It turns out that the judges agreed with the effectiveness of this approach, which allowed Weimin's team to take a decent lead in the final leaderboard score and ultimately take home the top prize.
Weimin is no stranger to data science competitions. He previously earned the title of "Kaggle Competition Grandmaster" after achieving the highest ranking out of 3000+ teams in another Computer Vision competition. He currently works at ByteDance as a Research Scientist where his team contributes to the development of multimodal capabilities across multiple ByteDance products. "This competition aligns well with my background in Deep Learning & Multi-modals, as well as with my vision to build AI applications that address real-world challenges. I took part in this challenge with hopes to boost AI awareness in Singapore, and to connect with both industrial & academic AI enthusiasts within the community" Weimin shared.
At ByteDance, we empower people to be themselves, we encourage their curiosity, and to search for inspiration in the world around them so we stay ahead of the curve. We want to create tech that inspires creativity and sparks curiosity within everyone. That's where we hope you come in. We're looking for forward-thinking individuals to work with some of the best tech in the game.
The more challenging task 2 was then further divided into sub-tasks that can be easily solved using 2 independent, single-modal classifiers for each style of Professional or Amateur videos. Finally, if any test video whose classifiers do not agree with the same style, it will be marked as a potential fake thus given a penalty score.
The aggregate score from both tasks 1 and 2 were then used to predict the final fake-ness of the videos. It turns out that the judges agreed with the effectiveness of this approach, which allowed Weimin's team to take a decent lead in the final leaderboard score and ultimately take home the top prize.
Weimin is no stranger to data science competitions. He previously earned the title of "Kaggle Competition Grandmaster" after achieving the highest ranking out of 3000+ teams in another Computer Vision competition. He currently works at ByteDance as a Research Scientist where his team contributes to the development of multimodal capabilities across multiple ByteDance products. "This competition aligns well with my background in Deep Learning & Multi-modals, as well as with my vision to build AI applications that address real-world challenges. I took part in this challenge with hopes to boost AI awareness in Singapore, and to connect with both industrial & academic AI enthusiasts within the community" Weimin shared.
At ByteDance, we empower people to be themselves, we encourage their curiosity, and to search for inspiration in the world around them so we stay ahead of the curve. We want to create tech that inspires creativity and sparks curiosity within everyone. That's where we hope you come in. We're looking for forward-thinking individuals to work with some of the best tech in the game.
Check out our R&D openings here
