There is a bug in train_maskrcnn.py. # Line 74, in filter_roidb function valid = len(fg_inds) > 0 and len(bg_inds) > 0 should be: valid = len(fg_inds) > 0 or len(bg_inds) > 0