2024 Cohort 2 - Group 12 Discussion

[sharon-digipres]

This is the discussion thread for those in Group 12 of the July-December 2024 Cohort

[Karen-NMRN]

Hi, so first step accomplished and created a GitHub account! Good to meet you all earlier.

[DeanAyotte]

Hi all, I've attached the brief notes we discussed today with links etc to resources people mentioned. Nice to meet you all, and see you next month.
DPC Python Study Group Meeting Notes 17.7.24.docx

[DeanAyotte]

Hey all, I've emailed the DPC about possible scripts we could work on over the next few months but I also found this example from the first cohort in their folders: c93d633

[StephCornwell]

Instructions for command line python install for future reference: python for archivists, pg. 7

[hdeakyne]

Thanks for this help! Note: Not sure this works if you don't have administrator privileges. That's where I got stuck. I'm going to wait until I really need it to bother IT. Maybe I'll see if I can get it to work at home.

[hdeakyne]

Update: When I was installing at home (Windows), I noticed the install screen had a box to check to add to PATH. So I just reinstalled at work with that checked off. Much easier! Doesn't require admin.

[hdeakyne]

In case you had this question while looking at sample scripts, "How do I choose the right module?", here is a pretty well structured explanation of the modules: https://pymotw.com/3/index.html. It was the clearest one I saw while trying to figure out what "import os", etc meant.

[hdeakyne]

Also, my department is doing group ArchivesSpace API learning. The trainer in the workshop video we are watching listed the modules/libraries she uses the most for archival work: JSON, CSV (for working with spreadsheets), and Request (for handling the request/response cycle).

Others she mentioned that she uses, but less frequently are pymarc (for MARC21), sqlalchemy (working with SQL in ASpace), lxml, and ArchivesSnake which is ASpace specific.

[DeanAyotte]

Hey all, been using this resource recently which has been useful for some basics: https://www.codecademy.com/catalog/language/python

[StephCornwell]

I've been using codedex, nice easy to follow basic lessons and I enjoy the fun aesthetic: https://www.codedex.io/python

[DeanAyotte]

This one is great, thanks for the link!

[hdeakyne]

Ah, it's so cute!

[hdeakyne]

I went through the Codedex (so fun!) until I got to the paid lessons. Here is a little script I made afterwards; most of it came from Googling how to do the things I wanted to do. I'm traveling soon and wanted to make a list of outfits to wear from the clothing I plan to bring. It runs in the Codedex screen, but can't get it to work right in the command line or Python IDLE on my computer.
outfits.txt

Also, maybe I can add on a loop or something so that it will pick 6 outfits. We'll see!

[StephCornwell]

Wrong about the f{ }! Here's the same thing expressed without having to use as much syntax:

print (f"Day {outfits+1}: {selected_top}, {selected_layer}, {selected_pant}, {selected_shoes}")

https://www.geeksforgeeks.org/formatted-string-literals-f-strings-python/

[hdeakyne]

Yes, amazing, thank you! I tried so many different ways to getting the f{} to work, but couldn't find the right syntax.

[Alea-bot]

Very fun!

[hdeakyne]

P.S. just to see if it would work, I made the script even smaller.
outfits2.txt

[StephCornwell]

Nice!

[harrylloyd-bl]

Hi Group 12 if you need anything @ me or use the questions section 😄

[hdeakyne]

Related to our possible group project which involves referring to CSV files to do things, the API video I shared discusses at the end (3:26) writing script that pulls info from CSV to create new records in ASpace.

[DeanAyotte]

I've had a go at the file renamer we talked about trying out in our last meeting: https://github.com/Digital-Preservation-Coalition/PythonStudyGroups/blob/main/How-Tos/Use%20Case%20-%20How%20to%20batch%20rename%20files.md?plain=1

Attached is my version of the script with some slight variations from the code that is suggested in the step by step guide.

I thought maybe something that might be useful in the file renamer script is to have an output file which acts as a log without having manually export text file from Terminal or Command Line using manual inputs. I've been trying to figure out a way to do this as part of the python script into a specific directory folder but I can't find a straight answer online to the issues I'm running into. Anyone got any suggestions?

renamer.txt

[hdeakyne]

Cool! I'm going on vacation soon so don't have time to work on Python for the next few weeks :( But I'm very excited to look at this all when I get back.

[hdeakyne]

Yay, I finally had a chance to try it out. I had it search for "IMG" and change it to another word to better describe the images. It only changed images ending in .JPG (I updated that too, I think case might matter??) and skipped over all the .AAE and .MOV files.

Initially when I tried it out, I got an error (pictured). Looking it up meant I needed to add an r in front of the source path to produce a raw string: https://stackoverflow.com/questions/1347791/unicode-error-unicodeescape-codec-cant-decode-bytes-when-writing-windows. I'm just quoting from this link, I don't actually fully understand the reasoning.

[hdeakyne]

Now I'm curious if there is a way to get it to search within folders within the source folder. Or can we add multiple source paths?

[DeanAyotte]

Interesting. I'm wondering if the source folder error is a difference between Mac (which I am using at home) and Windows.

It would be good if multiple source folder paths could be opened at the same time. WIth this code I imagine you'd have to pull all the files into a single ingest/working folder which introudces possible errors and extra time.

[hdeakyne]

Yes, I now see that the How To discusses the other solution for Mac to Windows, using double backslashes.
I've tried a few solutions for doing multiple folders at once, but nothing yet.

[DeanAyotte]

Here is a list of all the functions that Python can do: https://docs.python.org/3/library/functions.html

[hdeakyne]

Meeting 3, September 18, 2024, Notes. The meeting was just Dean and me! So we didn't really move forward much. Decided to continue with the decisions made in Meeting 2 on what to work on (working through various tutorials) and see if we can come up with a way to have the renaming program work on multiple folders at once.

[DeanAyotte]

Hey all,

I've been learning about reading/writing files and creating csv files in a python script and I have tried to apply this to extracting and converting data in an xml file into a csv spreadsheet.

I am using metadata from the National Library of Scotland's Moving Image Archive that is available on their Data Foundry here: https://data.nls.uk/data/metadata-collections/moving-image-archive/

The script I have written to do this is attached. It currently creates a csv file with the different DC elements as row headers, but does not fill in any of the data from the xml file.

I've tried running it through Chat GPT 4 for some assistance but can't work out what is happening. Any ideas?

Thanks!

data_extractor.txt

[DeanAyotte]

I've had a bit more of a test and this version seems to work. Open to more suggestions and tips though 😃

data_extractor_v2.txt

[harrylloyd-bl]

Looks like you got the answer! To do with setting the right namespaces and doing the right findall()s?

For debugging generally I can really recommend Julia Evans' poster and zine (if your budget can spare $12) on debugging as a good framework to follow. I do pretty much everything in there any time I find a bug. Specific Python debuggers vary but the VS Code/Spyder/Pycharm/Jupyter notebook ones are good and the interactive Python shell or a notebook are also helpful tools!

[DeanAyotte]

Thanks @harrylloyd-bl, I'll grab those next pay day. I've been using VS Code for a bit, sometimes been finding the debugger a bit confusing, but that'll probably come with experience.

[DeanAyotte]

Hey all,

I was having a look at using python to try and write a script to extract metadata from AV files and I ended up finding this information with lots of scripts for extracting metadata from different file formats: https://www.linkedin.com/pulse/mastering-metadata-extracting-file-information-python-yamil-garcia-wurwe

I'm going to try running and tweaking the AV one, so I'll let you know if there are any issues/solutions.

[harrylloyd-bl]

Hi group 12, sorry to not drop in to your session today, if there's anything you need help with let us know :)