AP IB GT enrollment

[HarishC727]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @HarishC727, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a new, automated pipeline for integrating Advanced Placement, International Baccalaureate, and Gifted and Talented enrollment data from the Civil Rights Data Collection. The system includes specialized scripts for downloading and processing data from various years, handling structural differences, and preparing it for statistical variable generation, thereby enriching our educational datasets.

Highlights

• New Data Ingestion System: Introduced a comprehensive system for downloading, processing, and importing Advanced Placement (AP), International Baccalaureate (IB), and Gifted and Talented (GT) enrollment data from the Civil Rights Data Collection (CRDC).
• Dedicated Download Scripts: Added two Python scripts: download_ap_ib_gt.py for general multi-year data acquisition and download_2015_16.py for specific handling of the uniquely structured 2015-16 CRDC data.
• Automated Data Processing: Implemented run_process.sh scripts for each data type (AP, IB, GT) to automate the processing of downloaded files, including adding 'YEAR' and 'ncesid' columns, and generating statistical variables.
• Configuration and Metadata: Included new configuration files (pvmap.csv, common_metadata.csv) and manifest files (manifest.json) to define data mapping, import specifications, and scheduling for each enrollment type.
• Robust File Handling: Enhanced data processing with robust error handling for empty files, missing columns, and special logic for specific AP subject files (Mathematics, Science) in older datasets.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces new scripts and configurations for downloading and processing AP, IB, and GT enrollment data. The changes are extensive, adding new functionality for data ingestion. My review focuses on improving script robustness, fixing a critical configuration error, and enhancing documentation clarity. I've identified a critical typo in a configuration file that needs to be addressed. Additionally, I've suggested replacing the use of eval in shell scripts with a safer alternative to prevent potential security vulnerabilities and recommended cleaning up the formatting in the README file for better readability.

[manishvats2]

approved