Here we provided a google colab implementation of HeyGem.
HeyGem is a free and open-source AI avatar project developed by Duix.com. It enables anyone to create ultra-realistic digital human avatars and produce AI-driven videos easily and at almost zero cost, using ordinary computer hardware.
HeyGem uses artificial intelligence to train digital avatars from real-person video data, bypassing the need for expensive and complex 3D modeling.
Case 1: Text + Orignal Video to Video (Text2Video.ipynb)
| Input 1 | Input 2 (Original) | Output |
| In order to produce effective targeted therapies for cancer, scientists need to isolate the genetic and phenotypic characteristics of cancer cells ... |
text2videoVideo.mp4 |
text2video.mp4 |
Case 2: Text + Orignal Video to Video (Text2Video.ipynb)
| Input 1 | Input 2 (Original) | Output |
| Large language model is gradually changing people's lives... |
HeygemTest3Video.mp4 |
HeygemTest3.mp4 |
Case 3: Audio + Orignal Video to Video (Audio2Video.ipynb)
| Input 1 | Input 2 (Original) | Output |
| Audio |
audio2videoVideo.mp4 |
audio2video.mp4 |
Case 4: Audio + Orignal Video to Video (Audio2Video.ipynb)
| Input 1 | Input 2 (Original) | Output |
| Audio |
HeygemTest1Video.mp4 |
HeygemTest1.mp4 |
Note 1: We have observed that the model performs suboptimally when tested on animation videos. This is likely because the underlying model was primarily trained on real, human-centric datasets. As a result, its ability to generalize to animation videos is limited, and the outputs in these cases may be less accurate or realistic.
Note 2 (used A100GPU in Colab): Audio + Orignal Video to Video - Generating a 3-minute 54-second video takes approximately 4 minutes and 31 seconds. In comparison, generating an 18-second video requires about 1 minute and 18 seconds.
Setting in Colab: In the implementation, used A100GPU in Colab as follows
-
Step 1: Download the ipynb file https://github.com/xinxingwu-uk/Colab_Implementation-HeyGem/blob/main/DownloadModel.ipynb from the GitHub repository, then upload it to your Google Drive.
-
Step 2: In Google Drive, open the ipynb file by Google Colab
-
Step 3: Implement the ipynb file in Google Colab - Run all cells in the notebook to set up and download the project.
-
Step 4: After execution, check your Google Drive in the same folder where the notebook is located. The whole project should now be available there - Google Drive folder: HeyGem-Linux-Python-Hack.
-
Step 5: Upload the ipynb files Audio2Video.ipynb (https://github.com/xinxingwu-uk/Colab_Implementation-HeyGem/blob/main/Audio2Video.ipynb) and Text2Video.ipynb (https://github.com/xinxingwu-uk/Colab_Implementation-HeyGem/blob/main/Text2Video.ipynb) in your Google Drive, right-click, and open with Google Colab. Then, run through the notebook step by step:
Take *Text + Orignal Video to Video (Text2Video.ipynb) as an example,
(a) Find the uploaded Text2Video.ipynb, open it
(b) Mount Google Drive
(c) Execute all cells in order for Text2Video.ipynb. After completing implementation, a file 1004-r.mp4 will appear in your Google Drive folder
Note: All related materials, including notebooks, models, and output files, are provided in the Google Drive
This project is based on Original Projects Duix.Heygem and HeyGem-Linux-Python-Hack built with DUIX.COM.
See LICENSE for details.