Maintenance status of matbench

[ml-evs]

Code snippet

No response

What happened?

Hi matbench devs, we're thinking of resubmitting a MODNet benchmark for 2024, but would like to query the development status of this repo. Can it still be considered active? I can see there are quite a few outstanding benchmark PRs (#356, #353, #352), plus dependency issues (#269) rearing their heads.

I'd be happy to help out on the maintenance front, but perhaps it would be good to get a couple of developers from other submissions too? Although the benchmarks maybe seen as "old hat" now, there are still lots of papers that refer to them, reporting their own models as SoTA without actually submitting here, or only comparing to a subset of other models...

Let me know what you think -- tagging @hrushikesh-s @ardunn and @computron as the last people involved in this.

Version

latest

Which OS?

• MacOS
• Windows
• Linux

Log output

No response

[duchieuphan2k1]

@ml-evs
It has been approximately one year since the last active update from the Matbench development team. Some recent research papers claim to have achieved SoTA results on Matbench but have not submitted them to the official benchmark, possibly due to the lack of active maintenance. This situation may lead to unfair comparisons.

In your opinion, which research works can be considered as fair SoTA results at present? Additionally, could you kindly share any recent updates on ModNet's performance?

I am currently working on developing a model for Matbench, and having access to the most recent results would be incredibly valuable.

Thank you in advance for your support and insights. I am looking forward to your response.

[ml-evs]

In your opinion, which research works can be considered as fair SoTA results at present? Additionally, could you kindly share any recent updates on ModNet's performance?

I can't say I've really kept up with the literature (if thats even possible these days) on pure property prediction models, obviously the trend is now for foundational models that are very hard to assess with regards to more old-school challenge datasets like these. Our MODNet resubmission was mostly going to be for reproducibility (and perhaps minor improvement) rather than a step change in performance.

I am currently working on developing a model for Matbench, and having access to the most recent results would be incredibly valuable.

Indeed, unfortunately it seems like this repo no longer a priority. Depending on the scope of your model, matbench-discovery might also be appropriate.

[duchieuphan2k1]

Thank you for getting back to me so quickly.

I'll take a look at matbench-discovery,

Thank you again for your recommendation