2024 Cohort 2 - Group 21 Discussion #47
Replies: 21 comments 21 replies
-
|
Hello Group 21 |
Beta Was this translation helpful? Give feedback.
-
|
Google Doc for meeting notes: https://docs.google.com/document/d/1y77HyDSBEF-z8DEbN_xWNBljQYEqCY4mKZKyvXAtJlo/edit?usp=sharing |
Beta Was this translation helpful? Give feedback.
-
|
Thinking about existing (non-python) solutions and their pluses/minuses, with an idea to defining what we want our own to do. TeraCopy is a Windows/Mac GUI tool I've used before for achieving something like Jack's use case - copying files from one place to another and making sure those copies match the source. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone. I'm Scott Prater, and I've volunteered to be a mentor for this group. I'm sorry I couldn't make it to the meeting this morning, but I'm looking forward to joining the group at the next meetup in August. In the meantime, I'll monitor this discussion board along with everyone, and chime in when I can be of assistance. |
Beta Was this translation helpful? Give feedback.
-
|
Something we didn't get a chance to discuss yesterday was Python environment, if there is a preferred one for our purposes or if we'd like to attempt to use the same one - I don't have one installed at the moment, and in the past have only used browser-based platforms like Google Collab and Jupyter Notebooks to run scripts but haven't otherwise used anything else. I think Sharon shared a list of options in her presentation but thought I'd ask among ourselves too. |
Beta Was this translation helpful? Give feedback.
-
|
For Erin's topic, I thought I would suggest using the brilliant https://ifiscripts.readthedocs.io/en/latest/index.html from Kieran O'Leary of the Irish Film Institute. Many of these are related to the work that I do, but my work is much less at the batch level, so I can't wholly apply them. They seem like a good starting point for some of your issues, especially the fixity scripts. |
Beta Was this translation helpful? Give feedback.
-
|
For Jack's topic... Like sg-archives i immediately came across IFI Scripts. Specifically i found this nice post where someone actually talks about using them: https://digitensions.home.blog/2019/09/16/fixity/ Furthermore they have written a few bullet points that i think could be useful for us in thinking about the steps involved in copying our artist-supplied files and validating them with checksums. Note this is, according to article author, the flow that IFI Script 'copyit.py' follows -- could be helpful of course when we design our script. I added some comments in caps: |
Beta Was this translation helpful? Give feedback.
-
|
Goal for next time: get all the infrastructure pieces in place to have an environment set up in Mac or Windows or Linux (a virtual environment is fine) to begin examining and modifying the IFI scripts.
I did run into some issues in Git bash: the IFIscripts are set to explicitly run under the executable |
Beta Was this translation helpful? Give feedback.
-
|
Hello! For those who don't have access to emails at the moment, I've popped a meeting in for the 26th to replace the meeting next week which we agreed would be easier due to iPres Conflicts. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, all. I just created a little 'scratch' file where we can play around with ideas. One thing that is conspicuously absent from it is proper method documentation -- where we describe what the method does, and stipulate what it expects and what it returns. Basically adding all that just made the file much longer and more intimidating so i left them out. Here is the repo. Hopefully the link is all you need to get access to it: |
Beta Was this translation helpful? Give feedback.
-
|
Ello! 🐱 |
Beta Was this translation helpful? Give feedback.
-
|
Hi all! In my absence tomorrow, some brief notes follow on the section of code which will compare source hash(es) with hash(es) of copied set. I'm afraid I'm far from solving this one.. I tried playing around with the difflib python module to do the comparison (doc here https://docs.python.org/3/library/difflib.html). The ndiff class works well for a single file/hash where a delta is returned indicating a change. But with a set of files/hashes (i.e. multi-line manifest) it compares using line position and so fails to find matches where the order of hashing is different. In the end, I kind of concluded that maybe this not the right approach. Instead would it work to just check if the two hashes generated are the same ("==") per file and if not output an error? Then make this iterative for multi-line manifests (no idea how hard that is!). I'm not sure and maybe I'm missing something here! |
Beta Was this translation helpful? Give feedback.
-
|
Stub python script: |
Beta Was this translation helpful? Give feedback.
-
|
@spater Hi, Scott. It seems i can't edit secureCopyFile.py directly, but have to fork. I gather from our conversation that this is not what we expected (hoped).... Is it just a case of getting me the right permissions or should i try and do a fork and merge, do you think? Thanks |
Beta Was this translation helpful? Give feedback.
-
Just marking some ways we can calculate hashes recursively using hashlib and the os modules:import hashlib def find_md5_hashes(directory): This approach is entirely lifted from https://www.geeksforgeeks.org/finding-md5-of-files-recursively-in-directory-in-python/ |
Beta Was this translation helpful? Give feedback.
-
|
Really sorry, everyone, not going to make it today. Hope you all have a lovely time. I'll miss it! |
Beta Was this translation helpful? Give feedback.
-
|
Hi, everyone -- thanks for a great meeting. For those who couldn't attend, please run a "git pull" on the repo and take a look at the changes we made today, and see if you can get the secureCopyFile.py script to run. You may need to change the path or name of your python executable on the very first line of the script to be able to run it. @SMurrayDoc has agreed to write a check to see if the destination file exists, and commit the update to the the repo. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Team! I've been quite unwell for the last few weeks and I haven't had a chance to sit down and look at the destination file existing coding. Unfortunately, I don't think things are going to change in the short term and rather than leaving us in the lurch at the last minute I was wondering if anyone would be able to take on this task? I will still try to endeavor to get this done, I just don't have a lot of faith in the actuality of this at this time! |
Beta Was this translation helpful? Give feedback.
-
|
Apologies all, don't have the device that I normally do this work on so may not be able to participate as fulsomely due to restrictions on software installation - trying to get Python set up on the device I'm using today, but might not have everything in order by the time of our meeting - very sorry about this! |
Beta Was this translation helpful? Give feedback.
-
|
Really sorry, but not going to make it AGAIN :| Hope you all learn lots and wishing happy holidays to everyone |
Beta Was this translation helpful? Give feedback.
-
|
Hi, everyone! Today we worked through the There was some interest in continuing our work in the spring, outside the DPC. In mid-January I'll send out a poll to all of us to find some meeting times in the spring to keep moving forward. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
This is the discussion thread for those in Group 21 of the July-December 2024 Cohort
Beta Was this translation helpful? Give feedback.
All reactions