Extend CleanVision to run on video data

[jwmueller]

Some discussion of this here: #214

Making CleanVision runnable on video would be super useful to many folks!

Should be fairly easy:

• just extract every k-th frame from the video,
• run cleanvision on all those images,
• aggregate the results across frames per video

[LemurPwned]

I'd be super interested in this -- might come in handy when sampling for annotations etc. Potentially doing some on-the-fly deduplication using the perceptual hash to avoid sampling visually identical scenes, before passing in to do further analysis, could improve efficiency of the process.

[jwmueller]

@LemurPwned This would be an awesome contribution! Nobody is working on this actively AFAIK.

@LaFemme was interested in trying CleanVision for video and could test out your PR code too!

[LemurPwned]

@jwmueller Great, I started playing around with the idea and for simplicity sake I created a separate repo for the sampling part: https://github.com/LemurPwned/video-sampler.
It seems that depending on the settings a significant reduction can take place. It's also pretty damn quick!

I'll try to integrate the sampling with the clean vision api in a PR soon.

[jwmueller]

awesome super excited about this!!

[LemurPwned]

@lafemmephile thanks, just to be clear, you want to pick up from here?

[LemurPwned]

@lafemmephile thanks for clarifying :) -- I'm fine with this, let me know if you need any help.
As to the outline you mentioned, makes sense -- the trickest part is probably avoiding reimplementation of the features from Imagelab.

[LemurPwned]

@lafemmephile I'm not a maintainer for this project, so I'm not sure to what extent my insight is useful here. But, I was essentially thinking of something similar to spacy's pipeline. A simple, minimal object which takes in the config for each step in the pipeline and then executes them in sequence.
So, I'd create a small video sampler object as a separate API, define its config, then write a simple pipeline executor instance that runs the sampler, saves the output and points image lab at it. In principle, prepping the config input correctly (as a kwargs dict or something to that extent) would decouple those objects from most changes in Imagelab's object api.
And, it allows you to add more steps before/after if needed.

[sanjanag]

Hi @LemurPwned ! The video sampler works great. Could you explain in more details about the approach you're suggesting for integrating video sampler in cleanvision? More specifically a pseudocode for what the classes would look like?

@lafemmephile We could start with a VideoLab class, that'd take care of sampling frames from videos, saving them and running cleanvision on those frames. I'd suggest to also think about a few finer points while working on this extension.

• What issues should VideoLab check for? For example, odd_size would be rendered useless in the case of a video since all images will be the same size.
• The VideoLab code is decoupled from the Imagelab code as much as possible to avoid any breaking changes in Imagelab.
• What are the preprocessing steps needed for VideoLab? For instance, sampling frames is a preprocessing step.
• What's the structure of the data object that will carried through VideoLab. For example in Imagelab we use the Dataset object to store all information related to dataset. Can we use the same or need to make a new one?
• How would the visualizations of results look like for Videolab?
• How would one detect duplicate videos? We can start with image property issues to make it simple.

@lafemmephile Feel free to start more discussions on these points. I think we could start with a layout of the code first and then start filling in details for specific methods.