Skip to content

plot_stimuli crashes on silent / speechless video clips #46

@sepehrheravi

Description

@sepehrheravi

Describe the bug

PlotBrain.plot_timesteps(..., show_stimuli=True) crashes when the input
video clip has no speech or no audio track, because
BasePlotBrain.plot_stimuli assumes both (a) get_audio(...) returns a
non-None object and (b) every segment.events dataframe contains a
type column (i.e. at least one Word event).

Both assumptions break in realistic workflows — e.g. screen recordings,
silent clips, or any video where WhisperX produces zero words. The model
itself handles the missing-modality case correctly (main.py auto-drops
the text extractor when no Word events exist), so this is strictly
a plotting-side bug.

Reproduction

import pandas as pd
from pathlib import Path
from tribev2.demo_utils import TribeModel, get_audio_and_text_events

model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = get_audio_and_text_events(
    pd.DataFrame([{
        "type": "Video", "filepath": "silent_clip.mp4",
        "start": 0, "timeline": "default", "subject": "default",
    }]),
    audio_only=True,
)
preds, segments = model.predict(events=df)
model._plotter.plot_timesteps(preds[:5], segments=segments[:5], show_stimuli=True)
# AttributeError: 'NoneType' object has no attribute 'to_soundarray'
# OR  KeyError: 'type'  when events is None / missing the column 

Expected behaviour

plot_stimuli should gracefully skip the audio waveform / word-overlay
subplots when those modalities are not present in the segments, mirroring
the model's existing "auto-drop missing extractor" behaviour.

Environment

OS: Ubuntu 24.04 aarch64 (NVIDIA DGX Spark, GB10)
Python 3.12.3
torch 2.11.0+cu130 (Blackwell sm_121)

Proposed fix

Two defensive guards in: tribev2/plotting/base.py::BasePlotBrain.plot_stimuli

Only plot the audio waveform when get_audio(...) returns non-None.
Only iterate words when segment.events is not None and the dataframe has a type column.
No change in behaviour when audio / words ARE present.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions