After poking around in the softap-setup Javascript I assume that the provided WiFi password is always hex-encoded before sending it to the setup API.
Line 39405 in softap-browser.js
securePass = this.__publicKey.encrypt(opts.password, 'hex');
I am, however, using a 256 bit (64 characters) hex-key as my WiFi password.
This is within specification and the password needs to be passed on directly without further encoding in order to work.
This was already an issue with Linux where the WiFi setup GUI does not support it and I had to edit config files directly for it to work.