Bugs in logging functions and _prepare_inputs

[cnyvfang]

if self.log_completions and self.state.global_step % self.args.logging_steps == 0:
            prompts_to_log = gather_object(prompts_text)
            completions_to_log = gather_object(completions_text)
            rewards = gather_object(rewards)
            rewards_to_log = rewards.tolist()

            if self.accelerator.is_main_process:
                if is_rich_available():
                    print_prompt_completions_sample(
                    prompts_to_log,
                    completions_to_log,
                    rewards_to_log,
                    self.state.global_step,
                    )

The print_prompt_completions_sample function's functionality requires rewards to be accepted as a dict (containing key, value) to achieve its purpose, however, the input rewards at this step seems to be a list (Even if it was not converted to a list).

[cnyvfang]

def _prepare_inputs(self, inputs: dict[str, Union[torch.Tensor, Any]]) -> dict[str, Union[torch.Tensor, Any]]:
        mode = "eval" if self.control.should_evaluate else "train"
        if mode == "train":
            if not hasattr(self, '_epoch_iterator'):
                frame = inspect.currentframe().f_back.f_back
                print("frame keys ", frame.f_locals.keys())
                epoch_iterator = frame.f_locals.get('epoch_iterator')
                assert epoch_iterator is not None
                self._epoch_iterator = epoch_iterator
            if self.state.global_step % self.num_iterations == 0:
                inputs = self._generate_and_score_completions(inputs)
                self._buffered_inputs[self._step % self.args.gradient_accumulation_steps] = inputs
            else:
                inputs = self._buffered_inputs[self._step % self.args.gradient_accumulation_steps]
            self._step += 1
        else:
            # In evaluation, we don't reuse completions across multiple updates, so we don't need to buffer inputs.
            inputs = self._generate_and_score_completions(inputs)
        return inputs

And it seems that self._buffered_inputs is not initialized in gpg_trainer.py.