Merge pull request #25 from SD2E/devel

Devel

Author: wjallen

docs/authorization.md docs/01.authorization.mddocs/authorization.md renamed to docs/01.authorization.md

@@ -11,7 +11,7 @@ for generating their own authorization tokens and, optionally, saving them in th
 <br>
 #### Generate authorization token
 
-Generate a token with the following command (the `-S` flag stores the token in 
+Generate a token with the following command (the `-S` flag stores the token in
 your Agave configuration file):
 ```
 % auth-tokens-create -S

docs/create_client.md docs/01.create_client.mddocs/create_client.md renamed to docs/01.create_client.md

@@ -4,14 +4,15 @@ title: Create the Client API Keys
 tagline:
 ---
 
-The Agave CLI is used to configure your user environment to interact with the 
-SD2E platform. If you do not yet have the CLI installed and in your `$PATH`, first
-go [here](install_cli.md).
+[The Agave API](http://developer.tacc.cloud/), (a component of the SD2E API), is
+used to configure your user environment to interact with the SD2E platform. If
+you do not yet have the CLI installed and in your `$PATH`, first go
+[here](01.install_cli.md) to install it.
 
 <br>
 #### Initiate with the SD2E Agave tenant
 
-On the command line, perform: 
+On the command line, perform:
 ```
 % tenants-init -t sd2e
 
@@ -24,7 +25,7 @@ home directory: `~/.agave/current`
 <br>
 #### Create API keys
 
-API keys are used to authenticate and store your user credentials for a specific 
+API keys are used to authenticate and store your user credentials for a specific
 client application. To generate API Keys:
 ```
 % clients-create -S -N sd2e_client -D "My client used for interacting with SD2E"
@@ -60,7 +61,7 @@ can delete then regenerate client API keys with the following:
 ```
 
 <br>
-#### The ~/.agave/current configuration file
+#### The `~/.agave/current` configuration file
 
 In addition to the tenant ID, your API keys are stored in the Agave configuration
 file. To view the contents of the file:

docs/install_cli.md docs/01.install_cli.mddocs/install_cli.md renamed to docs/01.install_cli.md

@@ -1,21 +1,35 @@
 ---
 layout: page
-title: Install and Configure the CLI
+title: Install and Configure the SD2E CLI
 tagline:
 ---
 
-The primary method for interacting with the SD2E platform is the 
-[Agave command line interface (CLI)](https://agaveapi.co/).
-The CLI is a collection of ~150 shell scripts for managing
-authorization, files, systems, applications, jobs, and more. 
+The primary method for interacting with the SD2E platform is the
+[SD2E command line interface (CLI)](https://github.com/SD2E/sd2e-cli).
+The SD2E CLI is a collection of several other CLIs including Agave, Abaco,
+TACCLab, and TACCReg. In total, they comprise  ~150 shell scripts and other
+resources for managing authorization, files, systems, applications, jobs, and
+more.
 
-There are three methods to install the CLI:
+There are three methods to install the CLI, choose whichever fits your preferred
+working environment:
 1. Run an installer script to install in ~/sd2e-cloud-cli/bin/
 2. Manual install to a location of your choosing
 3. Pull a CLI Docker image
 
 <br>
-#### Run an installer script to install in ~/sd2e-cloud-cli/bin/
+#### Requirements
+
+Make sure in your environment, you have:
+1. [Bash](https://www.gnu.org/software/bash/)
+2. [cURL](https://curl.haxx.se/)
+3. [Git](https://git-scm.com/)
+4. [Docker](https://www.docker.com/get-docker)
+5. [jq](https://stedolan.github.io/jq/)
+6. [python 2.7+ or 3.x+](https://www.python.org/downloads/)
+
+<br>
+#### Install method 1: Run an installer script to install in `~/sd2e-cloud-cli/bin/`
 
 Open a terminal window and run the installer script:
 ```
@@ -41,7 +55,7 @@ TACC Cloud API versions:
 ```
 
 <br>
-#### Manual install to a location of your choosing
+#### Install method 2: Manual install to a location of your choosing
 
 Open a terminal window and navigate to your preferred location for installation:
 
@@ -51,7 +65,8 @@ Open a terminal window and navigate to your preferred location for installation:
 
 Download and unpack the CLI:
 ```
-% curl -L https://raw.githubusercontent.com/sd2e/sd2e-cli/master/sd2e-cloud-cli.tgz -o sd2e-cloud-cli.tgz
+% curl -L https://raw.githubusercontent.com/sd2e/sd2e-cli/master/sd2e-cloud-cli.tgz \
+       -o sd2e-cloud-cli.tgz
 % tar -xvzf sd2e-cloud-cli.tgz
 ```
 
@@ -74,7 +89,7 @@ TACC Cloud API versions:
 ```
 
 <br>
-#### Pull a CLI Docker image
+#### Install method 3: Pull a CLI Docker image *(outdated)*
 
 The CLI is also available as a Docker image. For this to work, it is assumed you
 have a reasonably recent version of [Docker](https://www.docker.com/) installed.
@@ -99,15 +114,23 @@ TACC Cloud API versions:
 <br>
 #### Influential environment variables
 
+When working with the SD2E platform, you may optionally set some of the following
+environment variables. In bash, it will take the form:
+```
+% export VAR_NAME=option
+e.g.
+% export AGAVE_JSON_PARSER=jq
+```
+
 <table style="width:100%" border="1px" cellpadding="5">
   <tr>
-    <th align="left">environment variable</th>
-    <th align="left">options</th>
-    <th align="left">description</th>
+    <th align="left">Environment variable</th>
+    <th align="left">Options</th>
+    <th align="left">Description</th>
   </tr>
   <tr>
     <td>AGAVE_JSON_PARSER</td>
-    <td>json_mirror, jq, json, python, native</td>
+    <td>jq, json_mirror, json, python, native</td>
     <td>Set this to your favorite json parser (if installed)</td>
   <tr>
   </tr>

docs/01.request_access.md

@@ -0,0 +1,35 @@
+---
+layout: page
+title: Request Access to the SD2E Platform
+tagline:
+---
+
+#### SD2E Access
+Access is restricted to SD2 project members. Your username and password will be
+the same as your TACC username and password. To gain access, please follow the
+instructions on this page:
+
+[Request SD2E Access](https://sd2e.org/accounts/request-access/)
+
+Note: If you are already logged in to SD2E.org, you will have to first logout
+by [clicking here](https://sd2e.org/accounts/logout), then retry the link above.
+Account confirmation may take up to 1-2 business days.
+
+<br>
+#### GitLab
+Depending on how you plan to interact with and use the SD2E platform, you may
+also need access to the [SD2E GitLab](https://gitlab.sd2e.org/). Once your TACC
+account and SD2E access are confirmed, request access to Gitlab by posting in
+the `#cyberinfrastructure` channel of the
+[SD2E Team Slack](https://sd2e.slack.com/messages).
+
+<br>
+#### Docker Hub
+Several components of the SD2E platform utilize Docker images. To make full use
+of this platform, create a personal [Docker Hub account](https://hub.docker.com/)
+if you do not have one already.
+
+
+
+---
+Return to the [API Documentation Overview](../index.md)

docs/ssh_keys.md docs/01.ssh_keys.mddocs/ssh_keys.md renamed to docs/01.ssh_keys.md

@@ -5,15 +5,15 @@ tagline:
 ---
 
 In certain circumstances, it is advantageous to use public / private SSH keys
-to authenticate with Agave. Specifically, this is important when
-[creating your own private systems](create_systems.md). Having SSH keys will make it 
-easier to interact with systems that do not allow password authentication or that
-require frequent password changes, and it will increase security by preventing
-the need to store your password in a plain text file.
+to authenticate with the Agave API. Specifically, this is important when
+[creating your own private systems](02.create_systems.md). Having SSH keys
+will make it easier to interact with systems that do not allow password
+authentication or that require frequent password changes, and it will increase
+security by preventing the need to store your password in a plain text file.
 
 If you are new to the Agave CLI and the SD2E Agave tenant, it is recommended to
 skip this section of the documents until you need to
-[create your own private systems](create_systems.md).
+[create your own private systems](02.create_systems.md).
 
 <br>
 #### Prerequisites
@@ -40,7 +40,7 @@ Welcome to
  / ___ \ |_| | | | | | (_) \__ \ |_) | | | |  __/ | |  __/
 /_/   \_\__|_| |_| |_|\___/|___/ .__/|_| |_|\___|_|  \___|
                                |_|
-[js-104-39] % 
+[js-104-39] %
 ```
 
 <br>
@@ -76,8 +76,8 @@ Finally, add your public key to the `authorized_keys` file on your remote system
 [js-104-39] % cat agave_rsa.pub >> ~/.ssh/authorized_keys
 ```
 Your private key (`agave_rsa`) will be used in the future when it is time to
-[create your own private systems](create_systems.md). Never share your private
-key with anyone else.
+[create your own private systems](02.create_systems.md). Never share your
+private key with anyone else.
 
 
 ---

docs/advanced_job.md docs/02.advanced_job.mddocs/advanced_job.md renamed to docs/02.advanced_job.md

@@ -57,7 +57,7 @@ Modify the file to include the following:
   "maxRunTime": "05:00:00",
   "archive": false,
   "inputs": {
-    "input": "agave://data-sd2e-community/sample/sailfish/test/read1.fastq"
+    "input": "agave://data-sd2e-community/sample/sailfish/test/reads1.fastq.gz"
   },
   "parameters": {
   }
@@ -81,7 +81,7 @@ number of cores, and other parameters can be modified in a similar way.
 
 A useful feature of Agave and the SD2E tenant is the ability to use the output
 of one job as the input of a second job. It saves time and avoids unnecessary
-file transfers to keep the output/input in Agave "Job API space. For example,
+file transfers to keep the output/input in Agave "job API space". For example,
 perhaps you want to filter out ribosomal RNA from your raw sequence data using
 `SortMeRNA`, then perform a quality control check on the remaining RNA seq data
 using `FastQC`. For the first step, run `SortMeRNA` on sample data:
@@ -154,16 +154,16 @@ URI to the `input_file` will remain the same:
 ```
 https://api.sd2e.org/        # base URL - DO NOT CHANGE
 jobs/v2/                     # refers to jobs API - DO NOT CHANGE
-68373037840494.../           # previous job ID
+68373037840494.../           # job ID from step 1
 outputs/media/               # location of output data - DO NOT CHANGE
-4311028...free_reads.fastq   # name of output file
+4311028...free_reads.fastq   # name of output file from step 1
 ```
 
 <br>
 #### Parameter sweeps
 
 Most public SD2E apps are designed to take one input file or configuration, run
-an analysis, and return a result. With some simple scripting, it is possible to 
+an analysis, and return a result. With some simple scripting, it is possible to
 perform parameter sweeps using multiple Agave jobs. For example, imagine you would
 like to run FastQC on a series of fastq files named: `fastq_01.fq`, `fastq_02.fq`,
 `fastq_03.fq`, etc:
@@ -180,7 +180,7 @@ fastq_01.fq  fastq_02.fq  fastq_03.fq
 % jobs-template fastqc-0.11.5u2 > fastqc.json
 ```
 
-The next  step is to write a short script around the `fastqc.json` template 
+The next  step is to write a short script around the `fastqc.json` template
 that populates the file name into a new job json file, then submits the job.
 Here is an example script:
 

docs/create_systems.md docs/02.create_systems.mddocs/create_systems.md renamed to docs/02.create_systems.md

@@ -6,7 +6,7 @@ tagline:
 
 Most standard SD2E API users will not need to create custom execution or storage
 systems. By default, you have access to some private and public execution and
-storage systems that should connect you to [most of TACC resources](systems_basics.md).
+storage systems that should connect you to [most of TACC resources](02.systems_basics.md).
 
 Even so, there are circumstances where it is advantageous to create your own systems.
 For example, they can help you connect to private lab servers, license servers,
@@ -19,21 +19,21 @@ and execution system.
 
 For any system you register with Agave, you need the hostname and login credentials.
 The preferred login credentials are SSH key pairs. For instructions on creating
-appropriate public / private keys, see [here](ssh_keys.md).
+appropriate public / private keys, see [here](01.ssh_keys.md).
 
 You also need some basic information on the compute resources which will be
 registered as systems. For storage systems, you need to know the desired path
 for storing files. For execution systems, you need to first decide if it will be
-an HPC or CLI execution system. If an HPC execution system, you need some 
+an HPC or CLI execution system. If an HPC execution system, you need some
 information about the queueing system (queue names, limits, etc.). For both types
 of execution system you need a path for running jobs.
 
-The Jetstream instance we will register has hostname `129.114.104.39`. We will 
-assume the user '`username`' has already created [SSH keys](ssh_keys.md) and added
-the public key to `~/.ssh/known_hosts/`. The path for storing data will be the 
-home directory, `/home/username/`. Since this instance does not have a queuing 
-system, we will register it as a CLI execution system and run jobs from 
-the home directory, `/home/username/`.
+The Jetstream instance we will register has hostname `129.114.104.39`. We will
+assume the user '`username`' has already created [SSH keys](01.ssh_keys.md)
+and added the public key to `~/.ssh/known_hosts/`. The path for storing data
+will be the home directory, `/home/username/`. Since this instance does not have
+a queuing system, we will register it as a CLI execution system and run jobs
+from the home directory, `/home/username/`.
 
 <br>
 #### Register a storage system
@@ -97,7 +97,7 @@ To register this system with Agave, use the following command:
 <br>
 #### Register an execution system
 
-An execution system contains many of the same fields as a storage system. Use 
+An execution system contains many of the same fields as a storage system. Use
 the following template to begin:
 ```
 {
@@ -151,7 +151,7 @@ Again, because keys will be stored in plain text in this file, do not share this
 file with anyone and keep it secure. More fields, including information about how to submit jobs to an `HPC` system
 using a queue can be found in the [Agave documentation](http://developer.tacc.cloud/).
 
-Save the template in a file called `jetstream-execution-username.json`. 
+Save the template in a file called `jetstream-execution-username.json`.
 To register this system with Agave, use the following command:
 ```
 % systems-addupdate -F jetstream-execution-username.json