some SUSPICION of the result

[David-Ttao]

I used the patchtst source code, ETTh1 data set, trained for 5 rounds, and modified the scaler in dataloader to MinMax (consistent with the setting in the article) and the datasets split ratio is 7:2:1. The results are shown below. Among them, mae is lower than FreTS0.065 only when lookback is 192. The rest exceed the results provided by the authors. So I suspect that the result is not solid enough, the story is fancy, but the actual model is worth discussing.

[aikunyi]

this is our results

[David-Ttao]

Hello There is no convincing reason why my baseline results are better than yours. And can you explain this? The second line of code does not use traindata's data features to normalize the global features. Instead, it normalizes with global features. The first line of code does not make any sense, which will cause the test set information to leak into the training set.

In the normal benchmark, normalization is done like this:

only fit train_data, and transform on df_data. NOT fit_transform on df_data

[aikunyi]

Thanks for your reminder, we've revised it. If you have time, you are welcome to run FreTS with the normalization method like the normal benchmark. Thanks.

[Joeland4]

Hello There is no convincing reason why my baseline results are better than yours. And can you explain this? The second line of code does not use traindata's data features to normalize the global features. Instead, it normalizes with global features. The first line of code does not make any sense, which will cause the test set information to leak into the training set. In the normal benchmark, normalization is done like this: only fit train_data, and transform on df_data. NOT fit_transform on df_data

I have another two problems. first is why the results MAE and RMSE are not consistent with that in other SOTA papers such as DLinear or iTransformer？

The second is Did you really do the frequency MLP of Channel， the most intuitive thing is that there should be permutation invariance in channels. The biggest problem in your code is
in run_longExp.py
in FreTS.py
You can't do channel MLP in your model code at all, This is obviously not a mistake that can go undetected during the experiment! @aikunyi @David-Ttao

[ambitious121]

您好：没有令人信服的理由说明我的基准结果比您的好。您能解释一下吗？第二行代码不使用 traindata 的数据特征来规范化全局特征。相反，它使用全局特征进行规范化。第一行代码没有任何意义，这将导致测试集信息泄漏到训练集中。在正常的基准测试中，归一化是这样的：只拟合 train_data，并在 df_data 上转换。NOT fit_transform on df_data

我还有另外两个问题。首先是为什么 MAE 和 RMSE 的结果与其他 SOTA 论文（如 DLinear 或 iTransformer）中的结果不一致？第二个是你真的做过 Channel 的频率 MLP 吗，最直观的是 channels 中应该有排列不变性。你的代码中最大的问题是run_longExp.py FreTS.py 你根本不能在你的模型代码中做通道 MLP，这显然不是一个在实验过程中可以不被发现的错误！

他这个确实没有做通道的mlp我把代码改的可以进行通道mlp之后效果反而变差了。

[2ySong]

您好，我也有些疑问。他这个指标为什么没有和Autoformer类对齐，怎么都0.00+了？

[ManishIISc]

this is our results

There is a disparity in the results, even though the same hyperparameters and exact code were used