dod integration with blindml

[benclifford]

It took me a long time to get dod installed and running alongside blindml. Here are some potential action items based on that experience. This was based on @snowgy's blindml-integration branch. There is a corresponding issue in the blindml repo for action items there: TheDataStation/blindml#5

Installability:

This was a lot of work for me: lots of documentation, but lots of it slowly rotten by the passage of time. My install environment ended up as installing an old version of Debian with very old versions of python and java installed on top of that.

python installability

There is a requirements.txt file which pins packages to specific versions; however these versions are wrong often. I've got updates from Yue's working installation that work for me.

• update the requirements.txt file to versions that work

I also ended up installing an old python version (3.6) because of dependency version requirements

• find out what still requires specific python versions and liberalise

docker:

There are docker files which try to build a working set of docker images, however these seem to have rotten with the passage of time too. They should be fairly straightforward to bring up to date and would be a good way to codify the install process.

Java:

the java code did not work with the recent version of Java that I had installed by default. This stems from a really old version of gradle used to build the java components. I ended installing a really old JDK from Oracle to get this working.

• make this code buildable with recent java

Documentation/testing

• the quickstart guide should have enough information to actually run and see results - a small extra section on the end "now get this URL, run this command, you should see this result"
• the quickstart guide needs a once-over to update versions of things based on my experiences

Parameterisation

• The lsh_threshold value in networkbuildercoordinator is hard-coded, but I've needed to change this during my work on this project. This should be parameterised properly.

Integration with blindml

I made dod installable as a library (as well as making blindml installable as a library). This allows pip to discover dependencies that satisfy the requirements of both blindml and dod (rather than both components asserting specific requirement versions which are different).

• add setup.py to show pip how to install this
• make requirements.txt looser rather than asserting specific versions, and remove requirements that aren't directly required. (also see how blindml generates a version lock file to list specific versions, while also having a liberal requirements.txt)

output csv files are called 'view_n' without an extension.

• dod should output view_n.csv files with a csv extension.

Continuous Integration

The installation of this stack is sufficiently complicated and fragile that it would benefit from having automated testing in CI, I think.

• setup CI to test installation process and run basic tests