Skip to content

Conversation

@ChinChangYang
Copy link
Contributor

Fix the high numerical error in mish activation #2359.

Algorithm:

e = exp(x)
mish = x / (1 + 2 / (e * (e + 2)))

Evaluation:

In the following experiments, the mean absolute errors are evaluated by the method in #2359 (comment).

Before this change, NE generates high numerical error:

Mean Absolute Errors Across Samples:
  var_17:
    NE:  2.955052
    GPU: 0.000998

With the new algorithm, NE generates low numerical error:

Mean Absolute Errors Across Samples:
  var_17:
    NE:  0.001744
    GPU: 0.001516

A tester reported that the new mish function generates NaN only when x is -Inf in the float16 format.

Performance:

This change has been adopted in KataGo Core ML backend ChinChangYang/KataGo#7. The performance of the KataGo model with the new mish activation (7.15 ms) is similar to the original mish implementation (7.03 ms).

Conclusion:

Overall, the change enhances the accuracy and reliability of the mish activation in Core ML models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant