Make ironclad modular

[zen-wq]

I know it's a somewhat daunting task but the absolute majority of Ironclad uses I've seen only need MD5, SHA1, and byte-array-to-hex-string.

Frankly, I have no idea how to go about it. Splitting it into ironclad.digest and ironclad.cypher would be a good start, but more than half of the digests are also dead weight.

[luismbo]

This would be useful. Perhaps the ironclad system can continue loading everything by default, but having the possibility to load core functionality and specific algorithms would be nice since ironclad does take a while to compile.

[zen-wq]

Yes, that's how I did it with cl-charms. ironclad system should just load all the separate ironclad modules so existing users don't break.

[glv2]

Frankly, I have no idea how to go about it. Splitting it into ironclad.digest and ironclad.cypher would be a good start, but more than half of the digests are also dead weight.

The "modules" that take the most time to compile are ciphers and digests, the rest compiles pretty fast in comparison. And as some digests are based on ciphers, having ironclad.cipher and ironclad.digest packages would not reduce the compilation time a lot. For example, using just one digest would still require compiling all ciphers and all digests.

To have a really smaller compilation time, we would need to have one system/package per cipher and per digest. That will take some work.

[luismbo]

@glv2 why is that? Couldn't multiple systems share the same package?

[zen-wq]

They can. That would be one system per cipher/digest.

[luismbo]

Here's a lightly tested first try: luismbo@879ab77

[glv2]

Here's a lightly tested first try: luismbo@879ab77

I took a look at your patch, here are a few notes:

keystream is not a cipher, it's a function to jump anywhere in the stream of pseudo-random bytes generated by some ciphers (not all ciphers support this operation).

The dependency list of some of the systems will have to be modified:

• xchacha depends on chacha
• xsalsa20 depends on salsa
• skein depends on threefish
• tree-hash depends on tiger
• there might be others...

Also, aead, kdf, macs, prng and public-key depend on digests and ciphers, so I think we'll need to organize the systems in the following way:

• ironclad/core, with just the basic stuff (common, conditions, generic, util, etc...)
• ironclad/cipher/xxx depending on ironclad/core and sometimes on ironclad/cipher/yyy
• ironclad/digest/xxx depending on ironclad/core and sometimes on ironclad/cipher/yyy or ironclad/digest/zzz
• ironclad depending on all the previous systems, and containing the other modules (aead, kdf, macs, etc...)

Maybe it could be useful to also have the systems ironclad/ciphers, ironclad/digests, ironclad/aead, ironclad/macs, etc...

[luismbo]

Regarding that last suggestion, would ironclad/ciphers load all the the ciphers?

[glv2]

Yes, and ironclad/digests would load all the digests. These would in fact be almost empty system definitions with a long list of dependencies.

[luismbo]

Here's another go, even more untested than the previous one: luismbo@781d638. It adds a simple way to define dependencies and I've included the ones you've mentioned, but haven't reviewed the compiler warnings or anything like that to see if more are missing.

Not sure about ironclad/public-key vs ironclad/public-keys (likewise for aead and kdf).

A potential issue about this split is that changing a file, e.g. updating a macro, within ironclad/core does not trigger the recompilation of ironclad/cipher/foo, IIRC. It'd be nice to tell ASDF to do such recompilations.

[glv2]

@luismbo I created a split-systems branch with some fixes on top of your patch.
When loading the global ironclad system, it looks like things work fine and the tests pass.
I made a few tests loading only some subsystems, and they worked. Now, let's tests this more thoroughly before it can be merged into the master branch...
Please report any issue or weird behavior than you can find.

[luismbo]

@glv2 thanks for picking this up. In terms of testing, perhaps we could load each subsystem individually (in a fresh lisp, let's say SBCL) then run the tests? (It'd be convenient if we could get away with detecting undefined ciphers/digests/etc rather than explicitly spliting the tests, but perhaps that could fail to detect missing dependencies, so perhaps explicitly splitting the tests is the way to go.) Does this seem like an approach worth exploring?

[glv2]

I guess the best solution will be for each subsystem to have its tests, and the test target for the global ironclad system will run the tests of all the subsystems. This will require an overhaul of our test system.

Do you think we should make these subsystems available to users in a release soon (as experimental feature), or wait until the new test system is done?

[luismbo]

Releasing this as an experimental feature sounds like a good idea!

[glv2]

Subsystems are available as experimental feature in Ironclad 0.56.