Support for hash files more than 512Mb

[nf404]

Now hashers support hashing only 32bit message length in bits.
This is about 512Mb. After that it calculates wrong hash.
I'm ignoring this, but now https://nf404.github.io/crypto-api/example/hasher-file.html calculates 1Mb per second on my core I5, so 600Mb calculates about 10min. This may be not so rare case to calculate hash from 1Gb file in browser )))

[n1474335]

This would be a useful feature in CyberChef as we improve our support for large files.

It would also be useful if your hashers could accept data from an ArrayBuffer instead of having to convert to a string first as this increases processing time.

[nf404]

When I tested ArrayBuffers was slowly than Arrays and Arrays much slowly than strings.
I understand that I need to migrate from strings to ArrayBuffers at least for webasm but for now this would decrease speed.
So the solution will be to create encoder fromArrayBuffer. I created issue #7 for this.

[n1474335]

Thanks for the swift response.

I would argue that the best way to read a large binary file into a browser is by using FileReader.readAsArrayBuffer(). At this point, the data is already in ArrayBuffer format and converting it to a string will take a substantial amount of time. This conversion overhead probably means that hashing the file as a string will take longer than hashing it as an ArrayBuffer, even if the ArrayBuffer method is currently slightly slower. I would hope that ArrayBuffer performance will improve in future. In theory they should probably be faster than normal arrays and strings due to their fixed sizes, but obviously a lot of optimisation has gone into arrays and strings in the past.

[nf404]

I use readAsBinaryString for example https://github.com/nf404/crypto-api/blob/master/example/hasher-file.js
I don't compare speed of this two functions but it work pretty fast