How to infer my own dataset?

[shitengyuan]

Excuse me.I am very interested in your work and would like to use your model to train and inference on my own dataset, but I don’t know how to do it? Can you give me some advice?

[yeshwanth95]

Hi @shitengyuan . Thank you for your interest in our work. The Pix2Poly training and inference scripts should work with any dataset that follows the MS-COCO format for annotations. You will need to prepare your data in the MS-COCO format.

Then you will simply need to implement a Pytorch Dataset class for your dataset in the datasets directory and add a get_<custom_dataset>_loaders function in ddp_utils.py. You can have a look at the implementations for existing datasets for reference. After this you should be able to use the exisitng training and inference scripts simply by importing the dataset and dataloaders in the corresponding scripts.

I hope that is helpful. Feel free to let me know if your need help with anything specific.

[shitengyuan]

Hello, I am using my own dataset (image size 800*800) with the same training set as the test machine for training 50epoch and found that the training loss is basically not decreasing.
The test result is
miou: 0.0
biou: 0.0
macc: 0.861035943031311
bacc: 0.0
May I ask what is the reason for this? Thank you for your advice.

Add:
Because my input is 800*800, and the encoder used is vit_patch8_224, so I did the following when encoding the position:

`class Encoder(nn.Module):
def init(self, model_name='deit3_small_patch16_384_in21ft1k', pretrained=False, out_dim=256) -> None:
super().init()
self.model = timm.create_model(
model_name=model_name,
num_classes=0,
global_pool='',
pretrained=pretrained,
# image_size=(800,800)
# image_size=CFG.INPUT_SIZE
)
# 替换 patch_embed
self.model.patch_embed = FlexiblePatchEmbed(
img_size=(800, 800), # 这里只是初始化用，forward 中不再做 assert
patch_size=self.model.patch_embed.patch_size,
in_chans=self.model.patch_embed.proj.in_channels,
embed_dim=self.model.patch_embed.proj.out_channels,
)

    # 保存原始 patch 信息和 pos_embed
    self.patch_size = self.model.patch_embed.patch_size
    self.grid_size = self.model.patch_embed.grid_size
    self.pos_embed = self.model.pos_embed  # shape = (1, num_patches + 1, dim)
    
    self.bottleneck = nn.AdaptiveAvgPool1d(out_dim)



def interpolate_pos_encoding(self, x, w, h):
    """
    x: (B, num_patches+1, C), w: 输入宽度, h: 输入高度
    """
    npatch = x.shape[1] - 1
    N = self.pos_embed.shape[1] - 1
    if npatch == N and w == h:
        return self.pos_embed
    class_pos_embed = self.pos_embed[:, 0]
    patch_pos_embed = self.pos_embed[:, 1:]
    dim = x.shape[-1]
    h0 = h // self.patch_size[0]
    w0 = w // self.patch_size[1]
    patch_pos_embed = patch_pos_embed.reshape(1, int(N**0.5), int(N**0.5), dim).permute(0, 3, 1, 2)
    patch_pos_embed = F.interpolate(patch_pos_embed, size=(h0, w0), mode='bilinear', align_corners=False)
    patch_pos_embed = patch_pos_embed.permute(0, 2, 3, 1).reshape(1, h0 * w0, dim)
    return torch.cat((class_pos_embed.unsqueeze(0), patch_pos_embed), dim=1)

def forward(self, x):
    B, C, H, W = x.shape
    # 1. Patch embedding
    x = self.model.patch_embed(x)  # (B, num_patches, C)
    cls_token = self.model.cls_token.expand(B, -1, -1)
    x = torch.cat((cls_token, x), dim=1)

    # 2. Interpolate pos embedding
    x = x + self.interpolate_pos_encoding(x, W, H)
    x = self.model.pos_drop(x)

    # 3. Transformer blocks
    for blk in self.model.blocks:
        x = blk(x)
    x = self.model.norm(x)

    # 4. 去掉 cls token，送入 bottleneck
    return self.bottleneck(x[:, 1:])`