docs: Update README with script descriptions and usage instructions

[abuzarmahmood]

No description provided.

[abuzarmahmood]

Generated edit command:

• Summary of User's Issues and Requests:
  The issue involves updating the README file in the katzlabbrandeis/blech_data_transfer repository to describe the functionality of files in the src directory and provide usage instructions. The last comment [ develop_issue ] suggests focusing on developing the issue further, but no specific details are provided in the comment itself.

• Overview of Plan to Address the Issues:

  1. Identify the files in the src directory and understand their functionality.
  2. Update the README file to include descriptions and usage instructions for each identified script.
  3. Ensure the README is clear, informative, and provides value to users.

• Specific Details of Changes to be Made:

  1. Add Description for blech_data_transfer.py:

     • File: README.md
     • Line: After line 29
     • Description of Change: Add a detailed description of the script.
     • Code Snippet:

       ## blech_data_transfer.py
       This script handles user interactions and logging for data transfers. It records logs with details such as date, time, user, project name, user email, recording name, and path. The script ensures that the necessary metadata is included before allowing data transfer.

  2. Add Description for blech_data_sentry.py:

     • File: README.md
     • Line: After the description for blech_data_transfer.py
     • Description of Change: Add a detailed description of the script.
     • Code Snippet:

       ## blech_data_sentry.py
       This script scans the file system for datasets and checks for accompanying metadata. It identifies datasets by `info.rhd` files and checks for required metadata files (`*.info`). The results are saved to a CSV file, which can be used to track missing metadata and contact users if necessary.

  3. Add Description for dataset_handler.py:

     • File: README.md
     • Line: After the description for blech_data_sentry.py
     • Description of Change: Add a detailed description of the script.
     • Code Snippet:

       ## dataset_handler.py
       This script checks for logs both locally and on the server. If logs are present in both locations, it merges them. It ensures that logs are up to date and asks the user if they would like to validate logs or sync from the server.

These changes aim to provide users with a clear understanding of the functionality of each script in the src directory and how to use them effectively.

---

This response was automatically generated by blech_bot using model gpt-4o

Aider output:

View Aider Output

The --sonnet flag is deprecated and will be removed in a future version. Please use --model sonnet instead.
You can skip this check with --no-gitignore
Added .aider* to .gitignore
Aider v0.82.2
Main model: anthropic/claude-3-7-sonnet-20250219 with diff edit format, infinite output
Weak model: anthropic/claude-3-5-haiku-20241022
Git repo: .git with 15 files
Repo-map: using 4096 tokens, auto refresh


README.md

src/blech_data_sentry.py

src/blech_data_transfer.py

src/dataset_handler.py
I'll update the README.md file to include descriptions of the scripts in the src directory. Based on the files you've shared, I'll 
add detailed information about each script's functionality and usage.                                                              

README.md                                                                                                                          

                                                                                                                                   
<<<<<<< SEARCH                                                                                                                     
![image](https://github.com/user-attachments/assets/cb03c5c9-da50-40fa-9fbe-99e29376615c)                                          
Courtesy of ChatGPT                                                                                                                
                                                                                                                                   
# Introduction                                                                                                                     
Code to:                                                                                                                           
1. Transfer ephys data from the local machine to the server                                                                        
    - Recording will be copied to directory of specified user, under a specific project name                                       
2. Check that info file is present                                                                                                 
3. Update recording log                                                                                                            
                                                                                                                                   
NOTE: This system will only work well if this method of copying cannot be side-stepped                                             
    by the user without including metadata.                                                                                        
    Alternatively, there could be functionality to keep emailing the user if a dataset is uploaded                                 
    (after a certain date, so previous datasets are not affected) without metadata.                                                
    (see https://www.sitepoint.com/quick-tip-sending-email-via-gmail-with-python/)                                                 
                                                                                                                                   
- Mounting of server is required, but not included in this code                                                                    
    - Code is given path to the server                                                                                             
- If info file is not present, transfer will not be allowed                                                                        
- Copy of recording log is made locally and updated with new recording                                                             
    - This allows multiple copies of the recording log to be made in case of accidental deletion                                   
- Recording log will contain:                                                                                                      
    1) Date                                                                                                                        
    2) Time                                                                                                                        
    3) User                                                                                                                        
    4) Project name                                                                                                                
    5) User email                                                                                                                  
    6) Recording name                                                                                                              
    7) Recording path                                                                                                              
- Directory to be copied can be specified both via the command line and via popup                                                  
                                                                                                                                   
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             
First install `cifs-utils` ::: `sudo apt-get install cifs-utils`                                                                   

usage: ./mount_katz_drive.sh

- Will first ask for machine sudo password, then for brandeis password.                                                            
- Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                        
- Edit the script to change the mount location.                                                                                    
                                                                                                                                   
# Moonshot                                                                                                                         
- Autoprocess uploaded data and extract recording quality features such as:                                                        
    - Number of units                                                                                                              
    - Unit amplitude                                                                                                               
    - Unit signal-to-noise ratio                                                                                                   
    - Mean firing rate per unit                                                                                                    
    - Responsive fraction                                                                                                          
    - Discriminative fraction                                                                                                      
    - Palatable fraction                                                                                                           
    - Dynamic fraction                                                                                                             
    - Drift descriptors                                                                                                            
    - Unit similarity                                                                                                              
                                                                                                                                   
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             

usage: ./mount_katz_drive.sh

Will first ask for machine sudo password, then for brandeis password.                                                              
Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                          
Edit the script to change the mount location.                                                                                      
=======                                                                                                                            
![image](https://github.com/user-attachments/assets/cb03c5c9-da50-40fa-9fbe-99e29376615c)                                          
Courtesy of ChatGPT                                                                                                                
                                                                                                                                   
# Introduction                                                                                                                     
Code to:                                                                                                                           
1. Transfer ephys data from the local machine to the server                                                                        
    - Recording will be copied to directory of specified user, under a specific project name                                       
2. Check that info file is present                                                                                                 
3. Update recording log                                                                                                            
                                                                                                                                   
NOTE: This system will only work well if this method of copying cannot be side-stepped                                             
    by the user without including metadata.                                                                                        
    Alternatively, there could be functionality to keep emailing the user if a dataset is uploaded                                 
    (after a certain date, so previous datasets are not affected) without metadata.                                                
    (see https://www.sitepoint.com/quick-tip-sending-email-via-gmail-with-python/)                                                 
                                                                                                                                   
- Mounting of server is required, but not included in this code                                                                    
    - Code is given path to the server                                                                                             
- If info file is not present, transfer will not be allowed                                                                        
- Copy of recording log is made locally and updated with new recording                                                             
    - This allows multiple copies of the recording log to be made in case of accidental deletion                                   
- Recording log will contain:                                                                                                      
    1) Date                                                                                                                        
    2) Time                                                                                                                        
    3) User                                                                                                                        
    4) Project name                                                                                                                
    5) User email                                                                                                                  
    6) Recording name                                                                                                              
    7) Recording path                                                                                                              
- Directory to be copied can be specified both via the command line and via popup                                                  
                                                                                                                                   
# Scripts in the src directory                                                                                                     
                                                                                                                                   
## blech_data_transfer.py                                                                                                          
This script handles the transfer of ephys data from a local machine to the server. It:                                             
- Validates that the data folder exists and contains required metadata (.info file)                                                
- Allows users to select which user account and subfolder to transfer data to                                                      
- Copies all files and directories from the local data folder to the server                                                        
- Logs the transfer details in a dataset frame for tracking purposes                                                               
- Ensures data isn't duplicated by checking if the experiment already exists                                                       
                                                                                                                                   
## blech_data_sentry.py                                                                                                            
This script scans the server file system for datasets and checks for accompanying metadata. It:                                    
- Identifies datasets by looking for info.rhd files                                                                                
- Checks if required metadata files (*.info) are present for each dataset                                                          
- Saves results to a CSV file for tracking purposes                                                                                
- Supports a blacklist to exclude certain directories from scanning                                                                
- Provides detailed logging of the scanning process                                                                                
                                                                                                                                   
## dataset_handler.py                                                                                                              
This script manages the dataset frame that tracks all data transfers. It:                                                          
- Checks for logs both locally and on the server                                                                                   
- Merges logs if they exist in both locations                                                                                      
- Ensures logs are up-to-date and consistent                                                                                       
- Provides functionality to add new entries to the dataset frame                                                                   
- Validates server access and handles file synchronization                                                                         
                                                                                                                                   
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## blech_data_sentry.py                                                                                                            

usage: python blech_data_sentry.py [--ignore_blacklist]

Scan the file-system for datasets and check if they have accompanying metadata.

options:
--ignore_blacklist Ignore the blacklist file when scanning directories

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             
First install `cifs-utils` ::: `sudo apt-get install cifs-utils`                                                                   

usage: ./mount_katz_drive.sh

- Will first ask for machine sudo password, then for brandeis password.                                                            
- Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                        
- Edit the script to change the mount location.                                                                                    
                                                                                                                                   
# Moonshot                                                                                                                         
- Autoprocess uploaded data and extract recording quality features such as:                                                        
    - Number of units                                                                                                              
    - Unit amplitude                                                                                                               
    - Unit signal-to-noise ratio                                                                                                   
    - Mean firing rate per unit                                                                                                    
    - Responsive fraction                                                                                                          
    - Discriminative fraction                                                                                                      
    - Palatable fraction                                                                                                           
    - Dynamic fraction                                                                                                             
    - Drift descriptors                                                                                                            
    - Unit similarity                                                                                                              
>>>>>>> REPLACE                                                                                                                    
                                                                                                                                   

Tokens: 12k sent, 1.6k received. Cost: $0.06 message, $0.06 session.

utils/mount_katz_drive.sh
I see you've added utils/mount_katz_drive.sh to the chat. I'll update my changes to ensure they follow the correct SEARCH/REPLACE  
block format. Let me fix the README.md update:                                                                                     

README.md                                                                                                                          

                                                                                                                                   
<<<<<<< SEARCH                                                                                                                     
![image](https://github.com/user-attachments/assets/cb03c5c9-da50-40fa-9fbe-99e29376615c)                                          
Courtesy of ChatGPT                                                                                                                
                                                                                                                                   
# Introduction                                                                                                                     
Code to:                                                                                                                           
1. Transfer ephys data from the local machine to the server                                                                        
    - Recording will be copied to directory of specified user, under a specific project name                                       
2. Check that info file is present                                                                                                 
3. Update recording log                                                                                                            
                                                                                                                                   
NOTE: This system will only work well if this method of copying cannot be side-stepped                                             
    by the user without including metadata.                                                                                        
    Alternatively, there could be functionality to keep emailing the user if a dataset is uploaded                                 
    (after a certain date, so previous datasets are not affected) without metadata.                                                
    (see https://www.sitepoint.com/quick-tip-sending-email-via-gmail-with-python/)                                                 
                                                                                                                                   
- Mounting of server is required, but not included in this code                                                                    
    - Code is given path to the server                                                                                             
- If info file is not present, transfer will not be allowed                                                                        
- Copy of recording log is made locally and updated with new recording                                                             
    - This allows multiple copies of the recording log to be made in case of accidental deletion                                   
- Recording log will contain:                                                                                                      
    1) Date                                                                                                                        
    2) Time                                                                                                                        
    3) User                                                                                                                        
    4) Project name                                                                                                                
    5) User email                                                                                                                  
    6) Recording name                                                                                                              
    7) Recording path                                                                                                              
- Directory to be copied can be specified both via the command line and via popup                                                  
=======                                                                                                                            
![image](https://github.com/user-attachments/assets/cb03c5c9-da50-40fa-9fbe-99e29376615c)                                          
Courtesy of ChatGPT                                                                                                                
                                                                                                                                   
# Introduction                                                                                                                     
Code to:                                                                                                                           
1. Transfer ephys data from the local machine to the server                                                                        
    - Recording will be copied to directory of specified user, under a specific project name                                       
2. Check that info file is present                                                                                                 
3. Update recording log                                                                                                            
                                                                                                                                   
NOTE: This system will only work well if this method of copying cannot be side-stepped                                             
    by the user without including metadata.                                                                                        
    Alternatively, there could be functionality to keep emailing the user if a dataset is uploaded                                 
    (after a certain date, so previous datasets are not affected) without metadata.                                                
    (see https://www.sitepoint.com/quick-tip-sending-email-via-gmail-with-python/)                                                 
                                                                                                                                   
- Mounting of server is required, but not included in this code                                                                    
    - Code is given path to the server                                                                                             
- If info file is not present, transfer will not be allowed                                                                        
- Copy of recording log is made locally and updated with new recording                                                             
    - This allows multiple copies of the recording log to be made in case of accidental deletion                                   
- Recording log will contain:                                                                                                      
    1) Date                                                                                                                        
    2) Time                                                                                                                        
    3) User                                                                                                                        
    4) Project name                                                                                                                
    5) User email                                                                                                                  
    6) Recording name                                                                                                              
    7) Recording path                                                                                                              
- Directory to be copied can be specified both via the command line and via popup                                                  
                                                                                                                                   
# Scripts in the src directory                                                                                                     
                                                                                                                                   
## blech_data_transfer.py                                                                                                          
This script handles the transfer of ephys data from a local machine to the server. It:                                             
- Validates that the data folder exists and contains required metadata (.info file)                                                
- Allows users to select which user account and subfolder to transfer data to                                                      
- Copies all files and directories from the local data folder to the server                                                        
- Logs the transfer details in a dataset frame for tracking purposes                                                               
- Ensures data isn't duplicated by checking if the experiment already exists                                                       
                                                                                                                                   
## blech_data_sentry.py                                                                                                            
This script scans the server file system for datasets and checks for accompanying metadata. It:                                    
- Identifies datasets by looking for info.rhd files                                                                                
- Checks if required metadata files (*.info) are present for each dataset                                                          
- Saves results to a CSV file for tracking purposes                                                                                
- Supports a blacklist to exclude certain directories from scanning                                                                
- Provides detailed logging of the scanning process                                                                                
                                                                                                                                   
## dataset_handler.py                                                                                                              
This script manages the dataset frame that tracks all data transfers. It:                                                          
- Checks for logs both locally and on the server                                                                                   
- Merges logs if they exist in both locations                                                                                      
- Ensures logs are up-to-date and consistent                                                                                       
- Provides functionality to add new entries to the dataset frame                                                                   
- Validates server access and handles file synchronization                                                                         
>>>>>>> REPLACE                                                                                                                    
                                                                                                                                   

README.md                                                                                                                          

                                                                                                                                   
<<<<<<< SEARCH                                                                                                                     
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             
First install `cifs-utils` ::: `sudo apt-get install cifs-utils`                                                                   

usage: ./mount_katz_drive.sh

- Will first ask for machine sudo password, then for brandeis password.                                                            
- Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                        
- Edit the script to change the mount location.                                                                                    
=======                                                                                                                            
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## blech_data_sentry.py                                                                                                            

usage: python blech_data_sentry.py [--ignore_blacklist]

Scan the file-system for datasets and check if they have accompanying metadata.

options:
--ignore_blacklist Ignore the blacklist file when scanning directories

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             
First install `cifs-utils` ::: `sudo apt-get install cifs-utils`                                                                   

usage: ./mount_katz_drive.sh

- Will first ask for machine sudo password, then for brandeis password.                                                            
- Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                        
- Edit the script to change the mount location.                                                                                    
>>>>>>> REPLACE                                                                                                                    
                                                                                                                                   

README.md                                                                                                                          

                                                                                                                                   
<<<<<<< SEARCH                                                                                                                     
# Moonshot                                                                                                                         
- Autoprocess uploaded data and extract recording quality features such as:                                                        
    - Number of units                                                                                                              
    - Unit amplitude                                                                                                               
    - Unit signal-to-noise ratio                                                                                                   
    - Mean firing rate per unit                                                                                                    
    - Responsive fraction                                                                                                          
    - Discriminative fraction                                                                                                      
    - Palatable fraction                                                                                                           
    - Dynamic fraction                                                                                                             
    - Drift descriptors                                                                                                            
    - Unit similarity                                                                                                              
                                                                                                                                   
# How to use                                                                                                                       
                                                                                                                                   
## blech_data_transfer.py                                                                                                          

usage: python blech_data_transfer.py [-h] data_folder

Transfer data from the blech server to the local machine.

positional arguments:
data_folder Path to local data folder.

options:
-h, --help show this help message and exit

                                                                                                                                   
## mount_katz_drive.sh                                                                                                             

usage: ./mount_katz_drive.sh

Will first ask for machine sudo password, then for brandeis password.                                                              
Will mount the katz drive to the local machine at /media/files_brandeis_drive by default.                                          
Edit the script to change the mount location.                                                                                      
=======                                                                                                                            
# Moonshot                                                                                                                         
- Autoprocess uploaded data and extract recording quality features such as:                                                        
    - Number of units                                                                                                              
    - Unit amplitude                                                                                                               
    - Unit signal-to-noise ratio                                                                                                   
    - Mean firing rate per unit                                                                                                    
    - Responsive fraction                                                                                                          
    - Discriminative fraction                                                                                                      
    - Palatable fraction                                                                                                           
    - Dynamic fraction                                                                                                             
    - Drift descriptors                                                                                                            
    - Unit similarity                                                                                                              
>>>>>>> REPLACE                                                                                                                    
                                                                                                                                   

Tokens: 13k sent, 1.7k received. Cost: $0.07 message, $0.12 session.
Applied edit to README.md
Commit e5b609d docs: Update README with script descriptions and usage instructions