Research Priority Queue

[iejMac]

• More Benchmarks (benchmarking aggregation models #33)
• Test Different CLIP backbones (Test out different CLIP backbones #36)
• Hyperparameter sweep for simple aggregator like self attention (Hyperparameter sweep: SelfAttentionPooler #37)
• Architecture tuning (Architecture Tuning: Try out other poolers #38)
• video-clip guided stable diffusion (Better Inference API + Fun examples #39)

[iejMac]

Test Different CLIP backbones - H/14 gets much better results and also isn't much slower (due to how slow video decoding is) so we will likely shift to H/14 embeddings (or maybe L/14) while video decoding is still the bottleneck. If we decide to change the architecture of clip-video-encode to alleviate this bottleneck we should revisit this question.