Group 12 discussion

[pinaryesil]

[dorawaugh]

24th January, Meeting notes:

• We will use this discussion page for communication and questions between sessions;
• We will take it in turns to facilitate each of the remaining five meetings;
• Between now and our next meeting, we will try to test out one of the how-tos in the GitHub guide in our own work and come to the next session ready to discuss how that went;
• We will also come to the next session with one example of something we would like to be able to do in our work using Python;
• Anna will be our February facilitator, thank you, Anna!

[dorawaugh]

Course that I've found really useful: https://www.youtube.com/watch?v=8DvywoWv6fI

[digitensions]

Hi there, just wanted to say hi! I'm about this month if you need any python help, will try to assist. Just give me an @ and wishing you luck! Joanna

[dorawaugh]

Wed 28 Feb (meeting notes)

Anna:

• Worked on batch renaming files;
• Ran the script over test files, script renames files with 'foo' in name to 'bar';
• Would like to learn more about os module and has some ideas of use cases that module would be useful for;
• Had a look at some additional tutorials.

Dorothy:

• Worked on deleting files, this is something I currently do using multiple commands in bash;
• After using the how-to guide to delete thumbs.db files, used some trial and error to figure out how to find and delete files ending with a specific file extension;
• Plan to keep refining this over coming month to identify hidden files as well, do a better job of commenting to explain what I'm doing;
• Would also like to better understand libraries and how I might use them...

Dina:

• Worked on deleting files and adapted to work on born digital collections to identify and delete other kinds of unwanted files;
• Also created a file listing what files have been deleted, with file path and size;
• Script also lists file size to help ensure larger files aren't being deleted;
• Archivist can review that list before deleting files;
• Two version of the script to accommodate both files on a server and files on external media;
• Has also been working through the Python For Everyone tutorial;
• Noticed that script wouldn't work on a Mac, Mac wouldn't find some of the files identified by the script; could debug by printing out the file listing immediately following line 19 on script--that will help identify whether the Mac isn't seeing them at all or if the Mac just isn't able to match against the if statement provided on line 20;
• Probably a library that can be included to normalise a path depending on your OS;

Libraries:

• Some other code that someone else created that has been made publicly available and that you can use in your own code;
• Because Python has been a round for so long, there are lots of libraries out there;
• Some core libraries that get included in the install, but also some out there on Git;
• Using Google is a good way to search for libraries;
• Any library part of core Python distribution are reliable and should be preferred, well-maintained and distributed;
• That said, some libraries that you do need to install with pip but they are well-used and have large user communities.

There is a space on the Study Group GitHub pages to drop in scripts that we are working on, so please do that.

Anna's use case:

• Unzips all the folders, renames all the files using the name of the parent folder, merges the XML files together to create a single XML file with multiple records.
• Planning on looking at os module and XML libraries;
• Would also like to look into API how-to guide.

Ceilan:

• Would like to look at API how-to as well, looking at libraries that integrate MARC fields.

Dina:

• Would like to look at developing script that would compare provided checksums with generated checksums to make sure that they match.

Amy:

• Would like to get started on some tutorials;
• Look at their workflows to see if scripting could be used to improve the workflow or experience for the archivist

Other resources:
Lots of free lessons on the Programming Historian.

Next meeting: 27th March. Can use this discussion board to chat between meetings.
Amy will facilitate next meeting (thank you!)

[dvsokolova]

Here is the script I wrote for deleting hidden and other unwanted files from the external media:

# Import the OS module that allows interacting with files and directories on the system
import os

# Enter directory path:
path = input("Enter directory path: ")

# Return object orientation in string format using function repr(), which generates the object orientation
    # in string format, but also gives single quotes around the text
dir_path = repr(path)[1:-1]

# Open a text file that will be automatically closed when writing is finished
with open('list_of_unwanted_files.txt', 'w') as df:
	# Use os.walk to list files in folders and subfolders
	for root, dirs, files in os.walk(dir_path):
		# When using os.walk() iteratively change the working directory as working the way through the sub-folders.
		# To do this change the working directory to the root variable
		os.chdir(root)
		# Look for files that match the condition
		for file in files:
			if file.startswith('.') or file.startswith('~$') or file == 'Thumbs.db' or file == 'desktop.ini':
				# Get file sizes
				file_size = os.path.getsize(file)
				# Create a list of files including paths and file sizes
				line = (root + '\\' + file + ', ' + str(round(file_size / (1024), 2)) + ' MB')
				# Replace two slashes with one slash for Windows paths
				single_backslash_line = line.replace('\\\\', '\\')
				# Write the list of files into the text file
				df.write(single_backslash_line + '\n')
				# os.remove(file) # - use for deleting the listed files

[dvsokolova]

I tried to add the script to https://github.com/Digital-Preservation-Coalition/PythonStudyGroups/tree/main/Cohort1/Group12, but got a message " Uploads are disabled. File uploads require push access to this repository. "

[agoslen]

Wed 27 March (meeting notes)

Dorothy:

• Use Archivematica. When you ingest, it creates a METS file, information on file formats, checksums, validation, etc. End up with a rich METS file that is not really usable. Interested in using python to pull out some information from the METS files.
• Tried to write a script to pull out a single component but had trouble.
• Made some simpler test XML files.Two books with titles and prices. Script prints out title of books of test XML files.
• Will spend more time to see if can get the script to work on METS.

Dina:

• Remediate filenames and file paths.
• Special characters in filenames. Preservation workflow needs to have filenames normalized.
• Archivematica filename normalization script - tried to adapt for their purposes. Remediate special characters in filenames and also file paths.
• Exclude hidden files from being renamed
• Print list of files to be revised
• Renaming sometimes results in two files with the same name. Need to come up with how to handle those cases of filename collisions, such as adding “_1”.
• Q: Will this happen as part of Archivematica workflow or will use this script outside of that workflow?
• Problem with the METS files may have to do with the various namespaces

Anna:

• Looked at tutorial for working with APIs
• Replaced placeholder geographic data with local data to get local weather data
• Would like to review script more closely to better understand what each step is doing
• Still hoping to work on unzipped metadata exports and merging

Discussion:

• Dina had trouble adding script to https://github.com/Digital-Preservation-Coalition/PythonStudyGroups/tree/main/Cohort1/Group12
• Dorothy will ask about permissions for this space

[agoslen]

Hello Group 12! I am not sure if I will make it to our meeting tomorrow (May 29th) as I will be traveling. If I'm not able to be there I wanted to share a brief update. I've been continuing to work through PY4E, and I also experimented with a script for extracting several zip files. Below is that script.

import os
import zipfile

path = 'C:\\Users\\goslen\\python_study_group\\dcr_exports\\zips'

os.chdir(path)

for file in os.listdir('.'):
    with zipfile.ZipFile(file, 'r') as zip_ref:
        zip_ref.extractall('.')