2024 Cohort 2 - Group 21 Discussion

[sharon-digipres]

This is the discussion thread for those in Group 21 of the July-December 2024 Cohort

[SMurrayDoc]

Hello Group 21

[tomensom]

Google Doc for meeting notes: https://docs.google.com/document/d/1y77HyDSBEF-z8DEbN_xWNBljQYEqCY4mKZKyvXAtJlo/edit?usp=sharing

[sprater]

Could I get permission to edit the document?

[eliu-ual]

I've changed the permissions so anyone with the link can edit

[tomensom]

Thinking about existing (non-python) solutions and their pluses/minuses, with an idea to defining what we want our own to do. TeraCopy is a Windows/Mac GUI tool I've used before for achieving something like Jack's use case - copying files from one place to another and making sure those copies match the source.
Good: GUI is friendly to less techy people; it's free (sort of).
Bad: much slower than regular OS copying; no Linux version; a bit cryptic about what it's doing (and closed source); free version is non-commercial use only.

[sg-archives]

I wanted to suggest rclone, which is something that is open source and primarily used as a tool for dealing with cloud materials, but could be of great use here. See more https://rclone.org/ or https://github.com/rclone/rclone

[sg-archives]

Forgot to say we use this with a software partner (Artefactual) currently and have only heard good things, though I am not using it personally and it's not Python-based. It's written in Go.

[sprater]

Hello, everyone. I'm Scott Prater, and I've volunteered to be a mentor for this group. I'm sorry I couldn't make it to the meeting this morning, but I'm looking forward to joining the group at the next meetup in August. In the meantime, I'll monitor this discussion board along with everyone, and chime in when I can be of assistance.

[eliu-ual]

Something we didn't get a chance to discuss yesterday was Python environment, if there is a preferred one for our purposes or if we'd like to attempt to use the same one - I don't have one installed at the moment, and in the past have only used browser-based platforms like Google Collab and Jupyter Notebooks to run scripts but haven't otherwise used anything else. I think Sharon shared a list of options in her presentation but thought I'd ask among ourselves too.

[sprater]

I use the free version of PyCharm: https://www.jetbrains.com/pycharm/. Too many nice features to list, but it's not too difficult to get running right out of the box.

Here's a basic tutorial for PyCharm to write a "Hello, World!" Python script: https://www.guru99.com/creating-your-first-python-program.html

[eliu-ual]

I will try this, thank you!

[sg-archives]

For Erin's topic, I thought I would suggest using the brilliant https://ifiscripts.readthedocs.io/en/latest/index.html from Kieran O'Leary of the Irish Film Institute. Many of these are related to the work that I do, but my work is much less at the batch level, so I can't wholly apply them. They seem like a good starting point for some of your issues, especially the fixity scripts.

[sg-archives]

also on github: https://github.com/kieranjol/IFIscripts

[migel-wimtore]

For Jack's topic...

Like sg-archives i immediately came across IFI Scripts. Specifically i found this nice post where someone actually talks about using them:

https://digitensions.home.blog/2019/09/16/fixity/

Furthermore they have written a few bullet points that i think could be useful for us in thinking about the steps involved in copying our artist-supplied files and validating them with checksums. Note this is, according to article author, the flow that IFI Script 'copyit.py' follows -- could be helpful of course when we design our script. I added some comments in caps:

- Check the (DESTINATION) drive has write access and if the source is a file or directory

- Deletes random files ‘.DS_Store‘, ‘Thumbs.db‘, ‘desktop.ini‘, ‘Desktop.ini‘ (CAREFUL WITH THIS KIND OF THING AND ARTIST' PROJECTS!)

- Using hashlib will produce an md5 checksum for the file or directory of files to be copied, and store that in a folder

- Checks the destination has enough available space to start the copy

- The files begin copying, using script system tools specific to the operating system running the script, eg Win32 uses robocopy or Linux uses cp

- Creates a destination checksum manifest alongside the copied file(s)

- Counts for extra files in the destination and compares with those in destination manifest (STRETCH GOAL?)

- Checks if you want to overwrite duplicate files or manifests as they’re encountered (STRETCH GOAL?)

- Runs a verification check with the two checksum manifests and tells you if your files match or not

[migel-wimtore]

So just ahead of our meeting today i thought i would flesh out and make a bit more explicit what i imagine could be involved in each of the above steps. Caveat: i might be totally wrong! :)

Check the (DESTINATION) drive has write access and if the source is a file or directory

This could be an attempt to mount the disk and create a file, but this seems a bit hacky.
I came across this forum thread:

https://serverfault.com/questions/193971/determine-if-filesystem-or-partition-is-mounted-ro-or-rw-via-bash-script

It has a number of suggestions for checking write status of disks. Unfortunately i don't have access to Linux at work so untested.

    Basically, use grep or shell commands to find read/write status of attached disks.

Deletes random files ‘.DS_Store‘, ‘Thumbs.db‘, ‘desktop.ini‘, ‘Desktop.ini‘ (CAREFUL WITH THIS KIND OF THING AND ARTIST' PROJECTS!)

This could be a fairly simple loop where for each file in source, if file equates to known temporary file, delete file.

Using hashlib will produce an md5 checksum for the file or directory of files to be copied, and store that in a folder

Found a script that uses Python and Hashlib to generate checksums for all files in a folder -- might useful for reference:

https://gist.github.com/vinovator/d864555d9e82d25e52fd

Whatever we do, probably we will want to create a single text file with the checksums and file names (and paths). This we can then compare to the checksums for the copied files later.

Checks the destination has enough available space to start the copy

I suppose we would want to calculate the size of all files to be copied and then run and OS specific command (or perhaps a Python built in....) that returns free space and destination disk and compare the two values.

We will need to take care that we are comparing like for like, e.g. ensure that both values are in KB.

There are a number of ways to do this on Linux. See:

https://opensource.com/article/18/7/how-check-free-disk-space-linux

Whichever way we do it, i suppose we will want to extract just the values of interest from whatever tools we use. So we might pipe the output of those tool to another tool like grep to extract just specic value, which we can then compare.

The files begin copying, using script system tools specific to the operating system running the script, eg Win32 uses robocopy or Linux uses cp

Once we have established it is safe to copy, we could simply use the Linux command 'cp', but there are also other options, such as rsync. Advantage of cp is that it is very basic and, i imagine, faster than something like rsync (which will perform some more advanced operations before copying). The great thing about rsync is that it can do fancy things like only copy files that don't exist on source...

Creates a destination checksum manifest alongside the copied file(s)

After files have been copied we will want to run a similar process as we did for the source to create checksums for all copied files and store those in a text file alongside file names (and paths).

Counts for extra files in the destination and compares with those in destination manifest (STRETCH GOAL?)

Not sure if this is really relevant to us or whether we will just assume we are not interested in existing files at destination (probably just ignoring them), but perhaps i am misunderstanding something...

Checks if you want to overwrite duplicate files or manifests as they’re encountered (STRETCH GOAL?)

Rsync could handle this for us...

Runs a verification check with the two checksum manifests and tells you if your files match or not

The money. Assuming our two manifests files (for source and destination) are in the exact same format, we could do something simple like using Linux's 'diff' command to see if there are any differences in the files.

Possibly we would something for granular....

BASIC FLOW

So i suppose basically what we want to happen is:

	create checksum manifest for source

	copy files from source to destination

	create checksum manifest for destination

	compare to checksum manifests for differences

[migel-wimtore]

Sorry for any side-scrolling; i don't know why the text didn't wrap property :(

[sprater]

Goal for next time: get all the infrastructure pieces in place to have an environment set up in Mac or Windows or Linux (a virtual environment is fine) to begin examining and modifying the IFI scripts.

1. Start with a Unix-like terminal with a command line. Mac Terminal works well, as does Git for Windows.
   Goal: be able to open the terminal, and run a command: echo $PATH is a good command to run.

2. Install git. Git bash for Windows comes bundled with git; on Mac, you may need to install the Git package. In Ubuntu Linux, you can run sudo apt install git.
   Goal: verify that git is installed, by running git --version in your terminal:

   git --version  
   -> git version 2.45.2.windows.1
   

3. Install Python 3. Preferably install it as a package for your operating system, using the package or application manager for your device. Make sure it is in your application path, so that you can run it at the command line without having to specify the full directory where the python program executable is installed. Install Pip (the Python dependency/package manager) too.
   Goal: verify that Python 3 and pip are installed, by running python --version and pip --version in your terminal:

   python --version
   -> Python 3.12.5
   pip --version
   -> pip 24.2 from C:\Program Files\Python312\Lib\site-packages\pip (python 3.12)
   

4. Fork the https://github.com/kieranjol/IFIscripts Github repository into your own github space. This simply copies the source repository into your Github area, and makes you the owner. Any changes you make to your own copy will be your own henceforth, and will not affect the upstream repository (unless you ask the upstream owner to merge your changes into their repo).
   Goal: you should have a complete copy of the IFIscripts repository in your github account, at https://github.com/<your github id>/IFIscripts. Example: https://github.com/sprater/IFIscripts

5. Clone your copy of the IFIscripts repo to your local workstation, where you have Python 3 and Git installed. Make a directory where you can put all your projects, then clone the repo in that directory. For example, I use the directory Projects/.
   Goal: verify that you have a local copy of the IFIscripts on your workstation.

   mkdir -p Projects
   cd Projects/
   git clone git@github.com:sprater/IFIscripts.git
   -> Cloning into 'IFIscripts'...
      remote: Enumerating objects: 4687, done.
      remote: Counting objects: 100% (82/82), done.
      remote: Compressing objects: 100% (68/68), done.
      remote: Total 4687 (delta 37), reused 36 (delta 14), pack-reused 4605 (from 1)
      Receiving objects: 100% (4687/4687), 1.72 MiB | 9.63 MiB/s, done.
      Resolving deltas: 100% (2978/2978), done.
    cd IFIscripts
    git status
    -> On branch master
       Your branch is up to date with 'origin/master'.
         
       nothing to commit, working tree clean
   

6. Install the Python module dependencies for IFIscripts. Use pip to install the dependency defined in the repo's setup.py file.
   Goal: verify that all the dependencies are installed, by running pip install <path/to/IFIscripts/directory>.

   pip install ~/Projects/IFIscripts
   -> Defaulting to user installation because normal site-packages is not writeable
      Processing c:\users\sprater\projects\ifiscripts
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      [...]
      Successfully installed bagit-1.8.1 cffi-1.17.0 clairmeta-1.5.2 cryptography-41.0.7 dicttoxml-1.7.16 freetype-py-2.4.0 future-1.0.0 ifiscripts-2020.4.3.1 lxml-4.9.4 pycountry-22.3.5 pycparser-2.22 pyopenssl-23.3.0 python-dateutil-2.9.0.post0 setuptools-73.0.1 shutilwhich-1.1.0 six-1.16.0
   

7. Run the IFIScript copyit.py without any arguments to verify that Python and all the module dependencies are found. You should get a usage error, telling you that you are missing source and destination paths; this indicates that the script is working as designed.
   Goal: verify that the copyit.py script runs correctly in your environment:

   cd ~/Projects/IFIscripts
   ./copyit.py
   -> usage: copyit.py [-h] [-l] [-move] [-justcopy] [-y] source destination
      copyit.py: error: the following arguments are required: source, destination
      
   

I did run into some issues in Git bash: the IFIscripts are set to explicitly run under the executable python3: #!/usr/bin/env python3, which does not exist on my workstation, although python does. If you get the error that python3 cannot be found, let me know, and we'll sort it out.

[SMurrayDoc]

Hello! For those who don't have access to emails at the moment, I've popped a meeting in for the 26th to replace the meeting next week which we agreed would be easier due to iPres Conflicts.
I'm not going to make it to the October meeting, but I should see you all on the 26th of September.

[sg-archives]

Much appreciated!

[migel-wimtore]

Hi, all. I just created a little 'scratch' file where we can play around with ideas. One thing that is conspicuously absent from it is proper method documentation -- where we describe what the method does, and stipulate what it expects and what it returns. Basically adding all that just made the file much longer and more intimidating so i left them out.

Here is the repo. Hopefully the link is all you need to get access to it:

https://github.com/migel-wimtore/file_copy_checker

[SMurrayDoc]

Ello! 🐱
I'm not going to make it to the next meeting, but I've put a pull request onto the file_copy_checker repo with my attempts at python coding. Might be total gibberish 🤷 . Hopefully not...
I don't think I'd get the time to run and test it before, but I wanted to give you guys something so you weren't stalled!
Hope everyone is well,
Suzy

[tomensom]

Hi all! In my absence tomorrow, some brief notes follow on the section of code which will compare source hash(es) with hash(es) of copied set. I'm afraid I'm far from solving this one..

I tried playing around with the difflib python module to do the comparison (doc here https://docs.python.org/3/library/difflib.html). The ndiff class works well for a single file/hash where a delta is returned indicating a change. But with a set of files/hashes (i.e. multi-line manifest) it compares using line position and so fails to find matches where the order of hashing is different. In the end, I kind of concluded that maybe this not the right approach. Instead would it work to just check if the two hashes generated are the same ("==") per file and if not output an error? Then make this iterative for multi-line manifests (no idea how hard that is!). I'm not sure and maybe I'm missing something here!

[sprater]

Can post a link here to the section of code where you are trying this out?

[sprater]

Stub python script:

#!/usr/bin/python

import os.path

def check_file_exists(file_path: str) -> bool:
    print('\n Checking for existence of "' + file_path + "'")
    return os.path.isfile(file_path)

def main():

    name_of_file_to_be_copied: str = 'somesrcfile'
    source_path: str = '/my/source/folder/'
    destination_path: str = '/my/destination/folder/'

    # Step 1: check if the file exists
    if check_file_exists(source_path + name_of_file_to_be_copied):
        print('Yes, the file exists\n')
    else:
        print('No, the file does not exist\n')
        exit()

[sprater]

The first line, which is the path to your python executable, will vary from system to system. To see where your python is: which python or which python3

[sprater]

Our updated version of this script is now checked into this repo: https://github.com/Digital-Preservation-Coalition/PythonStudyGroups/tree/main/2024%20Cohort%202/Group%2021
If you'd like to expand on it, check it out, then change it and commit it in a branch. We can look at the branches, work through merges, together.

[migel-wimtore]

@spater Hi, Scott. It seems i can't edit secureCopyFile.py directly, but have to fork. I gather from our conversation that this is not what we expected (hoped).... Is it just a case of getting me the right permissions or should i try and do a fork and merge, do you think? Thanks

[sprater]

@Sharon-DPC @mutanthumb can you verify that @migel-wimtore has commit permissions on this repo?

[migel-wimtore]

Hm, i'm in two minds as to whether it is worth overwriting your file so that it also contains the structure i created in my file. I feel like we have mine there as a working reference, but that it's OK to build up your file step by step as we go. It strikes me there is no reason to implicitly codify the structure i came up with -- not doing so allows us to be less encumbered and more flexible

[sprater]

Works for me.

[migel-wimtore]

Same

[sg-archives]

Just marking some ways we can calculate hashes recursively using hashlib and the os modules:

import hashlib
import os

def find_md5_hashes(directory):
#Find MD5 hashes of files recursively in a directory.
hashes = {}
for root, _, files in os.walk(directory):
for file in files:
file_path = os.path.join(root, file)
hashes[file_path] = md5(file_path)
return hashes

This approach is entirely lifted from https://www.geeksforgeeks.org/finding-md5-of-files-recursively-in-directory-in-python/

[migel-wimtore]

Really sorry, everyone, not going to make it today. Hope you all have a lovely time. I'll miss it!

[sprater]

Hi, everyone -- thanks for a great meeting. For those who couldn't attend, please run a "git pull" on the repo and take a look at the changes we made today, and see if you can get the secureCopyFile.py script to run. You may need to change the path or name of your python executable on the very first line of the script to be able to run it.

@SMurrayDoc has agreed to write a check to see if the destination file exists, and commit the update to the the repo.

[sprater]

I also have a recording for the meeting (1 hr 40 mins long). Please PM me if you would like me to send it to you.

[SMurrayDoc]

Hi Team! I've been quite unwell for the last few weeks and I haven't had a chance to sit down and look at the destination file existing coding. Unfortunately, I don't think things are going to change in the short term and rather than leaving us in the lurch at the last minute I was wondering if anyone would be able to take on this task? I will still try to endeavor to get this done, I just don't have a lot of faith in the actuality of this at this time!

[SMurrayDoc]

https://stackoverflow.com/questions/82831/how-do-i-check-whether-a-file-exists-without-exceptions

and

https://stackoverflow.com/questions/8933237/how-do-i-check-if-a-directory-exists-in-python

were my jumping off points.

[tomensom]

Hey Suzy! Sorry to hear that - hope you feel better soon. I'm not sure I'll have time to take this on either, but don't worry I'm sure we can figure it out in the next session either way, no stress 😌

[eliu-ual]

Hi Suzy so sorry to hear this - hope you feel better soon!

[eliu-ual]

Apologies all, don't have the device that I normally do this work on so may not be able to participate as fulsomely due to restrictions on software installation - trying to get Python set up on the device I'm using today, but might not have everything in order by the time of our meeting - very sorry about this!

[migel-wimtore]

Really sorry, but not going to make it AGAIN :| Hope you all learn lots and wishing happy holidays to everyone

[sprater]

Hi, everyone! Today we worked through the secureCopyFile.py script, and arrived at the point where we actually copied a file from the source to destination, then did some simple checks to make sure the copy was successful. If you have a chance, please run git pull to pull down the latest version of our code, and give it a look and spin.

There was some interest in continuing our work in the spring, outside the DPC. In mid-January I'll send out a poll to all of us to find some meeting times in the spring to keep moving forward.