Time embedding scale + 1 / classifier-free guidance

[ajrheng]

According to #77, FiLM conditioning is used for the time embeddings instead of just simple shifting. May I know what is the reason for the +1 added to the scaling?
x = x * (scale + 1) + shift

On a related note, if I wanted to implement classifier-free guidance to do conditional generation, would you recommend implementing the class-conditioning embedding with FiLM in the same way as time? Would appreciate if anyone could point me to a github implemention of classifier-free guidance for DDPM--looks like the original paper does not have open-source code.

Thank you!

[lucidrains]

@ajrheng Hi Alvin, so the rationale for the +1 is to basically make sure the FiLM initial output is ~ identity function to start off with, assuming that the scale and shift will be centered around 0

for examples of classifier free guidance, just study this function

[ajrheng]

@lucidrains Thanks Phil for the quick response.

1. Is there a reason why we want FiLM to be identity at the start? Was it found to work better empirically perhaps?
2. Thanks for the link. That's equation 6 of the paper I presume, where you combine conditional and unconditional outputs. My question was exactly how should we implement the class-conditioning signal. It seems for that repo the signal is text tokens and it looks rather involved. For a simple class signal (like CIFAR10 classes), would it be appropriate to process it sinusoidally (like time) and just concatenate it with time signal before passing into FiLM?

Edit: seems like in Imagen, adding and concatenate the two signals are used. May I know which would be better for a simple class signal?

t = t + text_hiddens
c = time_tokens if not exists(text_tokens) else torch.cat((time_tokens, text_tokens), dim = -2)

[lucidrains]

1. yea, it is just a better initialization
2. i would just give each class its own embedding, and do the conditional dropout by swapping it out for a null embedding

maybe if i find some time this week, i'll build it out in a separate file

[ajrheng]

Are you suggesting to fix the class embeddings and not train them? Presumably these embeddings are still inserted into the UNet layers in a FiLM manner?

I would greatly appreciate if you have the time to build out a demo. Thank you very much, Phil!

[lucidrains]

@ajrheng ok i'll build it today, i think it'll help a lot of students

[ajrheng]

@lucidrains Thank you! That will indeed be extremely helpful. Please let me know where I can find it when you are finished!

[lucidrains]

@ajrheng oh sorry, didn't get to it today 😢 and i'm going to go see Wakanda Forever

tomorrow morning!

[ajrheng]

@lucidrains No problem. Enjoy the movie! I was almost supposed to catch it today too xD

[lucidrains]

@ajrheng hey Alvin, let me know if https://github.com/lucidrains/denoising-diffusion-pytorch/blob/main/denoising_diffusion_pytorch/classifier_free_guidance.py is helpful (example script at the bottom)

[ajrheng]

Thank you so much Phil! This will help with my project so much. I will get about testing it immediately. Will close this issue now!

[Mat-Po]

@ajrheng Hi Alvin! I am trying to follow your path toward class conditioning diffusion models here. I saw the file in the previous comment by Phil which was super useful. I was trying to adapt the Trainer class here to speed up the training but i am having some difficulties. did you menage to speed up your training in this or in any other ways?

I also saw this other repo from Phil but i felt that in my case, (conditioning on 6 classes) is much better the solution suggested here by Phil. Have you tried it or have any comment on this?