Getting Embedding from CLIP model

[lelefontaa]

Is there a way to save the embeddings of images generated by the CLIP model (in chat_handler) so that they can be reused without recalculating them?

[onestardao]

Yup — definitely doable.

You can cache the CLIP embeddings easily after generation — just serialize the tensor (or numpy array) and save it to disk with a hashed filename (based on the image path or content hash).
Then next time, check if that file exists before running the encoder again.

Here's a very quick sketch of the idea:

import os, hashlib, torch

def get_cache_path(image_path):
    h = hashlib.md5(open(image_path, 'rb').read()).hexdigest()
    return f"./clip_cache/{h}.pt"

def get_clip_embedding(image, model):
    cache_path = get_cache_path(image)
    if os.path.exists(cache_path):
        return torch.load(cache_path)
    else:
        embedding = model.encode_image(image)
        torch.save(embedding, cache_path)
        return embedding

That’ll save you time and GPU sanity.

PS:
I build semantic systems that run on top of embedding pipelines like this — if you ever hit weird drift or hallucination when using CLIP + chat models, WFGY (open-source) might help.