This project is from when I was trying to learn webscraping. I wanted to scrape and store the class feedback question-by-question and make a tool to view and compare class-specific professor evaluations.
There are 2 different "scraping scripts" contained within this and a front end file for user-interaction:
-
Codes Crawler -- This script opens up the UChicago course catalog main page and uses Beautiful Soup to:
a) Determine which links on the page lead to a department catalog page
b) Open these pages (using requests) and pull the page html
c) Find and save the unique course codes for every class -
Course Evals -- This scripts uses the course codes returned by the codes crawler, Beautiful Soup, and Selenium to:
a) Navigate to the course feedback portal (all login information sanitized in uploaded files)
b) Search each unique course code
c) Go through the professors for each class and identify their most recent evaluation
d) Open the most recent evalutions for each professor who taught each class and pull the data on each question and response
e) Serialize the class, professor, question, and response data in HTML format (lots of nesting in tree data structure) and write it into a file -
Front End -- This script uses the file returned by Course Evals (stored locally), Beautiful Soup, Pandas, Matplotlib, and a touch of regex to:
a) Guide a user through inputs to select the class and professors they want to see the evaluations of
b) Determine which questions they'd like to see of those that appear in the evaluation surveys for the class / professors they've selected
c) Visualize the student survey responses either as
i) A dataframe (formatted using tabulate)
ii) A graph representing the feedback pulled from the UChicago feedback portal
I hope to return to this project in the future and use a library like React to build out the user-interface as an online platform.
Note -- The Util file is used to help the Codes Crawler determine which of the links leads to a department catalog and which are not useful for my purposes.