How do I reproduce the ablation study for Fig3(a)?

[kun1224]

First of all, thank you very much for open-sourcing your work!

I want to reproduce the ablation study for Figure 3(a). I didn't change the network structure，I used the parameters you trained for SCRIMP，then，I only modified the input to the communication part， specifically the value of c1 in the forward function of the SCRIMPNet class.。 Below is the code after my modifications.

  @autocast()
  def forward(self, obs, vector, input_state, message):
       ....
        if self.modelType == ModelType.SCRIMP:
            c1 = self.communication_layer(message)
        elif self.modelType == ModelType.SCRIMP_WITHOUT_COM:
            c1 = torch.zeros_like(message)
        elif self.modelType == ModelType.SCRIMP_WITHOUT_ENCODER:
            c1 = message.clone()

After making these changes and running eval_model.py, the results for SCRIMP_WITHOUT_COM and SCRIMP_WITHOUT_ENCODER are poor. Does this mean there's an issue with my approach? Do I need to change the network architecture and retrain it, and then perform the ablation study for evaluation?or Could you tell me how to conduct the experiments for SCRIMP_WITHOUT_COM and SCRIMP_WITHOUT_ENCODER specifically?

I am looking forward to your response.Thank you!