Regarding Model Training 关于模型训练

[tanggang1997]

请问后续有计划放出训练代码吗，非常好的工作，但是自己想微调

May I ask if there are plans to release the training code later? It's very good work, but I want to fine-tune it myself.

[viewfinder-annn]

Thank you for your interest in our work and for pointing out the desire to fine-tune the model!

We do plan to release the training code eventually. However, it will take some time to properly organize, clean up, and document the code before we can make it publicly available. We appreciate your patience as we prepare this for release.

In the meantime, to give you an initial idea of the training process, especially for the Flow Matching (FM) part:

1. Noise Generation Location: The noise $\mathbf{z}$ used in our Flow Matching process is generated internally within the model's forward pass. This means that after calling forward(), you will have the necessary components (noise $\mathbf{z}$) readily available to calculate the loss. See anyaccomp/fmt_model.py L216

2. Necessary components for loss calculation, are returned around line 251 in the provided image snippet (specifically in the else block of if output_hidden_states:). See anyaccomp/fmt_model.py L251

3. Flow Matching Loss Snippet Example:
   For reference on how the loss is calculated using the outputs, here is a direct code snippet illustrating the core logic (similar to what you might implement for fine-tuning):

   # Example of a typical forward pass and loss calculation in FM
   noise, x, flow_pred, final_mask, prompt_len = self.model(
       x=x_mel,
       x_mask=x_mask,
       cond=cond_mel,
       cond_mask=cond_mask,
   )
   
   final_mask = final_mask.squeeze(-1)
   # The ground truth flow, often calculated using the generated noise z (represented here by 'noise')
   flow_gt = x - (1 - self.cfg.model.flow_matching_transformer.sigma) * noise
   
   # [B, n_frames, D]
   diff_loss = F.l1_loss(
       flow_pred, flow_gt, reduction="none"
   ).float() * final_mask.unsqueeze(-1)
   diff_loss = torch.mean(diff_loss, dim=2).sum() / final_mask.sum()
   
   total_loss += diff_loss
   train_losses["diff_loss"] = diff_loss
   
   # ... optimization steps follow

We hope this gives you a clearer picture of the process while we finalize the code release. We will update the issue once the training code is ready!

[tanggang1997]

Thank you for your interest in our work and for pointing out the desire to fine-tune the model!

We do plan to release the training code eventually. However, it will take some time to properly organize, clean up, and document the code before we can make it publicly available. We appreciate your patience as we prepare this for release.

In the meantime, to give you an initial idea of the training process, especially for the Flow Matching (FM) part:

1. Noise Generation Location: The noise

   z
   

   used in our Flow Matching process is generated internally within the model's forward pass. This means that after calling forward(), you will have the necessary components (noise

   z
   

   ) readily available to calculate the loss. See anyaccomp/fmt_model.py L216

2. Necessary components for loss calculation, are returned around line 251 in the provided image snippet (specifically in the else block of if output_hidden_states:). See anyaccomp/fmt_model.py L251

3. Flow Matching Loss Snippet Example:
   For reference on how the loss is calculated using the outputs, here is a direct code snippet illustrating the core logic (similar to what you might implement for fine-tuning):

   Example of a typical forward pass and loss calculation in FM

   noise, x, flow_pred, final_mask, prompt_len = self.model(
   x=x_mel,
   x_mask=x_mask,
   cond=cond_mel,
   cond_mask=cond_mask,
   )

   final_mask = final_mask.squeeze(-1)

   The ground truth flow, often calculated using the generated noise z (represented here by 'noise')

   flow_gt = x - (1 - self.cfg.model.flow_matching_transformer.sigma) * noise

   [B, n_frames, D]

   diff_loss = F.l1_loss(
   flow_pred, flow_gt, reduction="none"
   ).float() * final_mask.unsqueeze(-1)
   diff_loss = torch.mean(diff_loss, dim=2).sum() / final_mask.sum()

   total_loss += diff_loss
   train_losses["diff_loss"] = diff_loss

   ... optimization steps follow

We hope this gives you a clearer picture of the process while we finalize the code release. We will update the issue once the training code is ready!

thank you for your reply