Skip to content

Shortfin documentation #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jturmelle wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Shortfin documentation #451

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5c3e97e
Updated for IRI servers
jturmelle Jan 8, 2025
96c0e0d
fixed github ssh key instructions
jturmelle Jan 8, 2025
6982fe2
added alternative git clone
jturmelle Jan 8, 2025
22256b3
modified conda create notes
jturmelle Jan 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 78 additions & 38 deletions fbfmaproom/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,66 +1,118 @@
# FbF Maproom
# Setting up a development FBF Maproom
Copy link
Collaborator

@aaron-kaplan aaron-kaplan Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this change is appropriate. This is the title for the whole document, which includes not only setting up a development environment, but also updating datasets, modifying the python environment, and adding a foreign table.


## Setting up a development environment
These instructions are specific to developers working on an IRI server.

For now, these instructions are specific to developers working on an IRI server.
## Log into the server, with port forwarding enabled

### Log into the server, with port forwarding enabled
### Valid servers are

Choose port on which to run your development server. You must choose one that isn't in use by another developer on the same server. Log into the server (shortfin01 in the example) and forward your chosen port (1234 in the example).
- shortfin.iri.columbia.edu
- storm.iri.columbia.edu

Choose port on which to run your development server. You must choose one that isn't in use by another developer on the same server. Log into the server and forward your chosen port (1234 in the example, but don’t use this number make up your own random number).

```
ssh -L 1234:localhost:1234 shortfin01.iri.columbia.edu
ssh -L 1234:localhost:1234 shortfin.iri.columbia.edu
```

### On the server, create an ssh key to use with GitHub
## GitHub Account

The repository is public, so unless you are making changes to the code, you don't need a github account.
Copy link
Collaborator

@aaron-kaplan aaron-kaplan Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reading further, I see that for users without a github account you give the instructions for cloning over https further down, but when I first read to here it wasn't clear that that would be coming. I still would have followed the instructions to create an ssh key, and then I would have gotten stuck at "Configure GitHub to use that ssh key."

I think most readers of this document will have a github account already. Have you encountered anyone who didn't? I'm hesitant to complicate things by introducing a variant workflow for users that may not even exist.

Copy link
Contributor Author

@jturmelle jturmelle Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting up a GitHub account and key always seems to elicit problems. The number of times we've spent the first 45 minutes getting them set up with git repositories is not small. I would argue setting up git shouldn't even be in this document.

Copy link
Collaborator

@aaron-kaplan aaron-kaplan Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason they need to run a test server is so they can test configuration changes before commiting and pushing them. So even if they manage to get a test server running without a github account, they need a github account for the next step.


### On the server, create an ssh key to use with GitHub if you don't already have one.

```
ssh-keygen -f ~/.ssh/id_rsa
ssh-keygen -f ~/.ssh/id_rsa_github -t rsa
```

Provide a strong password when prompted. Then

```
cat ~/.ssh/id_rsa.pub
cat ~/.ssh/id_rsa_github.pub
```

and copy the output of that command for use in the next one. It should start with `ssh-rsa`.

### Configure GitHub to use that ssh key
In a browser, visit [https://github.com/settings/ssh/new]. For the title put e.g. "IRI development server," and in the Key box paste the output from the previous comand. Then click the green "Add SSH key" button.

### Clone this repository and enter the fbfmaproom directory
In a browser, visit [https://github.com/settings/ssh/new](https://github.com/settings/ssh/new). For the title put e.g. "IRI development server," and in the Key box paste the output from the previous comand. Then click the green "Add SSH key" button.

### Configure your IRI account to use that ssh key (Mac, Linux)

```
cat << EOF >> ~/.ssh/config

Host github.com
HostName github.com
IdentityFile ~/.ssh/id_rsa_github
PreferredAuthentications publickey
IdentitiesOnly yes

EOF
chmod 600 ~/.ssh/config
```

## Clone this repository and enter the fbfmaproom directory

If you have a github account and have set up your ssh key:

```
git clone git@github.com:iridl/python-maprooms
cd python-maprooms/fbfmaproom
```

If you don't have a github account and will not be making any code changes:

```
git clone https://github.com/iridl/python-maprooms.git
cd python-maprooms/fbfmaproom
```

## Python miniconda environment

### Enable conda

This could be added to your ~/.bash_profile to be loaded every time you login.

```
module load python/miniconda3.9.5
module load python/miniconda3
source /software/rhel8/x86_64/miniconda/3/etc/profile.d/conda.sh
```

### Create a conda environment containing this project's dependencies
### Optionally create a conda environment containing this project's dependencies

On the IRI servers, this environment is pre-built, so skip this step unless you're using an environment where this doesn't exist.

```
cond env list
```

If fbmaproom2 doesn't exist:

```
conda create -n fbfmaproom2 --file conda-linux-64.lock
```
(substituting osx or win for linux as appropriate)

You don't need to install conda-lock for this.
(substituting conda-osx-64.lock or conda-win-64.lock appropriate)

Do not use conda-lock for this.

Note that the command is `conda create`, not `conda env create`. Both exist, and they're different :-(

### Create a local configuration file
Copy link
Collaborator

@aaron-kaplan aaron-kaplan Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A pre-existing problem: this is part of the "Python miniconda environment" section, but has nothing to do with miniconda. Should be bumped out one level.


Create a file called `config-local.yaml` with the following contents, modified as noted in the comments.

```
db:
password: itsasecret # get the real db password from another developer
dev_server_port: 1234 # the port you chose earlier
```

## Running the application in your development environment

### Activate the conda environment

```
conda activate fbfmaproom2
```
Expand All @@ -71,7 +123,7 @@ conda activate fbfmaproom2
CONFIG=fbfmaproom-sample.yaml:config-local.yaml python fbfmaproom.py
```

### Test the application in a browser
### Test the application in a browser

On your laptop, connect to e.g. `http://localhost:1234/fbfmaproom/ethiopia`, substituting your chosen port number for `1234`. ssh port forwarding will forward the connection to the application running on the server.

Expand All @@ -82,11 +134,12 @@ All datasets used by the application are stored in zarr format. There are two ca
### Datasets from the Data Library

The script `fbf-update-data.py` pulls data from the DL and saves it as zarr. First read the script and find the name of the dataset you want to update, e.g. `ethiopia/rain-mam`. Then run the script as follows, substituting the chosen dataset name. Note that we run this script in the enactsmaproom conda environment, not the fbfmaproom environment. The fbfmaproom environment is missing the netcdf library, which the script requires; using the enactsmaproom environment is a lazy workaround.

```
conda activate /home/aaron/miniconda3/envs/enactsmaproom
CONFIG=fbfmaproom-sample.py python fbf-update-data.py ethiopia/pnep
```

### Datasets not from the Data Library

Scripts for converting non-Data Library datasets to zarr are contained in the `data-conversion-scripts` directory. One of these scripts, `zarrify-forecast.py`, is used for all PyCPT forecast datasets, for all countries. Other scripts are specific to a single dataset and are kept in per-country subdirectories.
Expand All @@ -98,6 +151,7 @@ For `zarrify-forecast.py`, edit the end of the script to indicate which dataset
## Adding or removing python dependencies

Edit `environment.yml`, then regenerate the lock files as follows:

```
conda install conda-lock
conda-lock lock -f environment.yml -f environment-dev.yml
Expand All @@ -108,24 +162,10 @@ conda-lock render

This maproom makes use of foreign tables in Postgres. Here's a brief explanation of how to add one:

1. Login in the pgdb12 server using psql. Make sure the user/role you use has the appropriate permissions

psql -h pgdb12.iri.columbia.edu -U fist -W DesignEngine

2. Create a server. Note that this has probably already been done so should be unnecessary

CREATE SERVER dlcomputemon1_iridb FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'dlcomputemon1.iri.columbia.edu', port '5432', dbname 'iridb');

3. Import the specific table(s) you want. It is advised to use the command below and not `IMPORT FOREIGN TABLE` so that the
definition of the table does not have to be (re)specified by hand. The schema can be arbitrarily chosen, I've used public here
but it might be advisable to create a schema for foreign tables specifically. Make sure you have the right schema of the table in the foreign server.

IMPORT FOREIGN SCHEMA public LIMIT TO (table_name) FROM SERVER dlcomputemon1_iridb INTO public;

4. Create a user mapping for every user in pgdb12 you want to have access to tables in the foreign server

CREATE USER MAPPING FOR dero SERVER dlcomputemon1_iridb OPTIONS (user 'ingrid_ro', password 'PASSWORD');

5. Grant select privileges to the local user if necessary

GRANT SELECT ON table_name TO dero;
1. Login in the pgdb12 server using psql. Make sure the user/role you use has the appropriate permissionspsql -h pgdb12.iri.columbia.edu -U fist -W DesignEngine
Copy link
Collaborator

@aaron-kaplan aaron-kaplan Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You seem to have made some unintended changes in this section.

2. Create a server. Note that this has probably already been done so should be unnecessaryCREATE SERVER dlcomputemon1\_iridb FOREIGN DATA WRAPPER postgres\_fdw OPTIONS (host 'dlcomputemon1.iri.columbia.edu', port '5432', dbname 'iridb');
3. Import the specific table(s) you want. It is advised to use the command below and not `IMPORT FOREIGN TABLE` so that the
definition of the table does not have to be (re)specified by hand. The schema can be arbitrarily chosen, I've used public here
but it might be advisable to create a schema for foreign tables specifically. Make sure you have the right schema of the table in the foreign server.IMPORT FOREIGN SCHEMA public LIMIT TO (table\_name) FROM SERVER dlcomputemon1\_iridb INTO public;
4. Create a user mapping for every user in pgdb12 you want to have access to tables in the foreign serverCREATE USER MAPPING FOR dero SERVER dlcomputemon1\_iridb OPTIONS (user 'ingrid\_ro', password 'PASSWORD');
5. Grant select privileges to the local user if necessaryGRANT SELECT ON table\_name TO dero;