From 920245a1cc4b9c7b8c425136ffa95961547e16ed Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Thu, 9 Jun 2016 16:21:30 -0700 Subject: [PATCH 01/21] Added server environment example doc --- serverenvironmentexample.adoc | 130 ++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 serverenvironmentexample.adoc diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc new file mode 100644 index 0000000..6996c25 --- /dev/null +++ b/serverenvironmentexample.adoc @@ -0,0 +1,130 @@ +{\rtf1\ansi\ansicpg1252\cocoartf1404\cocoasubrtf470 +{\fonttbl\f0\fswiss\fcharset0 Helvetica;} +{\colortbl;\red255\green255\blue255;} +\margl1440\margr1440\vieww10800\viewh8400\viewkind0 +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 + +\f0\fs24 \cf0 GRR Server Environment Example Setup\ +====================================\ +:toc:\ +:toc-placement: preamble\ +:icons:\ +\ +A GRR Server Environment can be whatever you make it. This example goes over the steps necessary to a three server environment: one to house the front end HTTP service, one to house the master datastore, and one to house the admin ui and workers.\ +\ +Assumptions Made\ +----------------\ +Below are the assumptions made to make use of this example.\ +\ +* GRR Servers will be running Ubuntu 16.04 LTS with all updates already installed\ +* There will be 3 servers in the configuration as outlined above. They will be called Master Dataserver, HTTP Server, and Main Server\ +* Bleeding edge GRR is desired and will be installed via pip\ +* The virtual environment will be called GRR_NEW\ +* All networking has been properly set up\ +\ +Setting up the servers\ +----------------------\ +1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git protobuf-compiler`\ +2. `pip install --upgrade pip`\ +3. `pip install virtualenv`\ +4. `git clone https://github.com/google/grr.git`\ +5. `virtualenv GRR_NEW`\ +6. `source GRR_NEW/bin/activate`\ +7. `cd grr/`\ +8. `pip install --editable .`\ +9. `pip install --editable grr/config/grr-response-server`\ +10. `pip install --no-cache-dir -f https://storage.googleapis.com/releases.grr-response.com/index.html grr-response-templates`\ +11. `pip install --editable grr/config/grr-response-test`\ +\ +Configuring GRR on the Main Server\ +----------------------------------\ +1. Set up a MTA of your choice\ +2. Run `grr_config_updater initialize`\ +* Step 1: Key Generation: - this step autocompletes and needs no user input\ +* Step 2: Setting Basic Configuration Parameters:\ + - GRR Datastore: Choose `1` for a SQLite Datastore. That is the datastore all these tutorials assume you have\ + - GRR URLs: Enter hostname: `
`\ + - Server URL: Frontend URL `http://:8080/`\ + - AdminUI URL: leave as default, press enter\ + - GRR Emails: Email Domain: ``\ + - GRR Emails: Alert Email Address: ``\ + - GRR Emails: Emergency Email Access Email Address: ``\ +* Step 3: Adding Admin User: ``\ +* Step 4: Installing template package: `n`\ +* Step 5: - Completes on its own\ +3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move `Client.executable_signing_public_key` to a new file and change permissions based on your executable signing structure policy\ +4. _Optional_ - Set up the Approval Process\ +* Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now.\ +* https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing]\ +\ +Initial Configuration on HTTP Server\ +------------------------------------\ +1. Replace `grr/install_data/etc/server.local.yaml` and `grr/install_data/etc/grr-server.yaml` with the files from main server\ +\ +Configuration on Master Dataserver\ +----------------------------------\ +1. Replace `grr/install_data/etc/server.local.yaml` and `grr/install_data/etc/grr-server.yaml` with the files from main server\ +2. Edit `grr/install_data/etc/server.local.yaml` and add the following:\ ++\ +[source,yaml]\ +Datastore.location: \ +Dataserver.server_list:\ + - http://:7000\ +Dataserver.client_credentials:\ + - ::rw\ +Dataserver.server_username: \ +Dataserver.server_password: \ +\ +* Terms:\ + - `server_list` is a list of all database servers with the master listed first\ + - `client_credentials` are used for the http server and main server to communicate with the database\ + - `server_username` and `server_password` are used for any slave dataservers to talk to the master dataserver.\ +3. Start the dataserver service by running `grr_server --component dataserver_master`\ +* It should start with no errors and not show any information. If you would like to see more information add in `--verbose`\ +\ +Finish the Configuration on the Main Server\ +-------------------------------------------\ +1. Edit install_data/etc/server.local.yaml\ +* Dataserver Configuration:\ ++\ +[source,yaml]\ +Dataserver.server_list:\ + - http://:7000\ +Datastore.implementation: HTTPServer\ +HTTPDatastore.username: \ +HTTPDatastore.password: \ +\ +* Email Configuration:\ ++\ +[source,yaml]\ +Worker.smtp_server: \ +Worker.smtp_port: 587\ +Worker.smtp_starttle: True\ +Worker.smtp_user: \ +Worker.smtp_password: \ +Email.approval_cc_address: \ +\ +2. Add users who should be able to access the admin interface.\ +* *Note* Users cannot change their password in the admin ui, they must change it in the command line\ +* https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User Management]\ +3. Run `grr_config_updater repack_clients`\ +* *Note* -If you decided to implement #3 in Configuring GRR on the Main Server add `--secondary_configs ` after `grr_config_updater`\ +4. Start services with `grr_server --component worker and grr_server --component ui`\ +\ +Finish the configuration on HTTP Server\ +---------------------------------------\ +1. Edit `install_data/etc/server.local.yaml`\ ++\ +[source,yaml]\ +Dataserver Configuration:\ +Dataserver.server_list:\ + - http://:7000\ +Datastore.implementation: HTTPServer\ +HTTPDatastore.username: \ +HTTPDatastore.password: \ +\ +2. Start services with `grr_server --component http_server`\ +\ +Post Setup\ +---------\ +At this point, you should be able to visit your admin interface, download the appropriate client installer binary. Once that is installed on the client, it will communicate back to your HTTP server and be visible through the admin ui.} \ No newline at end of file From c5b54719c7f1647d9f415c9bc42aafa5cd11f136 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Thu, 9 Jun 2016 16:22:54 -0700 Subject: [PATCH 02/21] Fixed env doc --- serverenvironmentexample.adoc | 277 ++++++++++++++++++---------------- 1 file changed, 148 insertions(+), 129 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 6996c25..fc8ea06 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -1,130 +1,149 @@ -{\rtf1\ansi\ansicpg1252\cocoartf1404\cocoasubrtf470 -{\fonttbl\f0\fswiss\fcharset0 Helvetica;} -{\colortbl;\red255\green255\blue255;} -\margl1440\margr1440\vieww10800\viewh8400\viewkind0 -\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 +GRR Server Environment Example Setup +==================================== +:toc: +:toc-placement: preamble +:icons: -\f0\fs24 \cf0 GRR Server Environment Example Setup\ -====================================\ -:toc:\ -:toc-placement: preamble\ -:icons:\ -\ -A GRR Server Environment can be whatever you make it. This example goes over the steps necessary to a three server environment: one to house the front end HTTP service, one to house the master datastore, and one to house the admin ui and workers.\ -\ -Assumptions Made\ -----------------\ -Below are the assumptions made to make use of this example.\ -\ -* GRR Servers will be running Ubuntu 16.04 LTS with all updates already installed\ -* There will be 3 servers in the configuration as outlined above. They will be called Master Dataserver, HTTP Server, and Main Server\ -* Bleeding edge GRR is desired and will be installed via pip\ -* The virtual environment will be called GRR_NEW\ -* All networking has been properly set up\ -\ -Setting up the servers\ -----------------------\ -1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git protobuf-compiler`\ -2. `pip install --upgrade pip`\ -3. `pip install virtualenv`\ -4. `git clone https://github.com/google/grr.git`\ -5. `virtualenv GRR_NEW`\ -6. `source GRR_NEW/bin/activate`\ -7. `cd grr/`\ -8. `pip install --editable .`\ -9. `pip install --editable grr/config/grr-response-server`\ -10. `pip install --no-cache-dir -f https://storage.googleapis.com/releases.grr-response.com/index.html grr-response-templates`\ -11. `pip install --editable grr/config/grr-response-test`\ -\ -Configuring GRR on the Main Server\ -----------------------------------\ -1. Set up a MTA of your choice\ -2. Run `grr_config_updater initialize`\ -* Step 1: Key Generation: - this step autocompletes and needs no user input\ -* Step 2: Setting Basic Configuration Parameters:\ - - GRR Datastore: Choose `1` for a SQLite Datastore. That is the datastore all these tutorials assume you have\ - - GRR URLs: Enter hostname: `
`\ - - Server URL: Frontend URL `http://:8080/`\ - - AdminUI URL: leave as default, press enter\ - - GRR Emails: Email Domain: ``\ - - GRR Emails: Alert Email Address: ``\ - - GRR Emails: Emergency Email Access Email Address: ``\ -* Step 3: Adding Admin User: ``\ -* Step 4: Installing template package: `n`\ -* Step 5: - Completes on its own\ -3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move `Client.executable_signing_public_key` to a new file and change permissions based on your executable signing structure policy\ -4. _Optional_ - Set up the Approval Process\ -* Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now.\ -* https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing]\ -\ -Initial Configuration on HTTP Server\ -------------------------------------\ -1. Replace `grr/install_data/etc/server.local.yaml` and `grr/install_data/etc/grr-server.yaml` with the files from main server\ -\ -Configuration on Master Dataserver\ -----------------------------------\ -1. Replace `grr/install_data/etc/server.local.yaml` and `grr/install_data/etc/grr-server.yaml` with the files from main server\ -2. Edit `grr/install_data/etc/server.local.yaml` and add the following:\ -+\ -[source,yaml]\ -Datastore.location: \ -Dataserver.server_list:\ - - http://:7000\ -Dataserver.client_credentials:\ - - ::rw\ -Dataserver.server_username: \ -Dataserver.server_password: \ -\ -* Terms:\ - - `server_list` is a list of all database servers with the master listed first\ - - `client_credentials` are used for the http server and main server to communicate with the database\ - - `server_username` and `server_password` are used for any slave dataservers to talk to the master dataserver.\ -3. Start the dataserver service by running `grr_server --component dataserver_master`\ -* It should start with no errors and not show any information. If you would like to see more information add in `--verbose`\ -\ -Finish the Configuration on the Main Server\ --------------------------------------------\ -1. Edit install_data/etc/server.local.yaml\ -* Dataserver Configuration:\ -+\ -[source,yaml]\ -Dataserver.server_list:\ - - http://:7000\ -Datastore.implementation: HTTPServer\ -HTTPDatastore.username: \ -HTTPDatastore.password: \ -\ -* Email Configuration:\ -+\ -[source,yaml]\ -Worker.smtp_server: \ -Worker.smtp_port: 587\ -Worker.smtp_starttle: True\ -Worker.smtp_user: \ -Worker.smtp_password: \ -Email.approval_cc_address: \ -\ -2. Add users who should be able to access the admin interface.\ -* *Note* Users cannot change their password in the admin ui, they must change it in the command line\ -* https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User Management]\ -3. Run `grr_config_updater repack_clients`\ -* *Note* -If you decided to implement #3 in Configuring GRR on the Main Server add `--secondary_configs ` after `grr_config_updater`\ -4. Start services with `grr_server --component worker and grr_server --component ui`\ -\ -Finish the configuration on HTTP Server\ ----------------------------------------\ -1. Edit `install_data/etc/server.local.yaml`\ -+\ -[source,yaml]\ -Dataserver Configuration:\ -Dataserver.server_list:\ - - http://:7000\ -Datastore.implementation: HTTPServer\ -HTTPDatastore.username: \ -HTTPDatastore.password: \ -\ -2. Start services with `grr_server --component http_server`\ -\ -Post Setup\ ----------\ -At this point, you should be able to visit your admin interface, download the appropriate client installer binary. Once that is installed on the client, it will communicate back to your HTTP server and be visible through the admin ui.} \ No newline at end of file +A GRR Server Environment can be whatever you make it. This example goes over the +steps necessary to a three server environment: one to house the front end HTTP +service, one to house the master datastore, and one to house the admin ui and +workers. + +Assumptions Made +---------------- +Below are the assumptions made to make use of this example. + +* GRR Servers will be running Ubuntu 16.04 LTS with all updates already +* installed +* There will be 3 servers in the configuration as outlined above. They will be +* called Master Dataserver, HTTP Server, and Main Server +* Bleeding edge GRR is desired and will be installed via pip +* The virtual environment will be called GRR_NEW +* All networking has been properly set up + +Setting up the servers +---------------------- +1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git +protobuf-compiler` +2. `pip install --upgrade pip` +3. `pip install virtualenv` +4. `git clone https://github.com/google/grr.git` +5. `virtualenv GRR_NEW` +6. `source GRR_NEW/bin/activate` +7. `cd grr/` +8. `pip install --editable .` +9. `pip install --editable grr/config/grr-response-server` +10. `pip install --no-cache-dir -f +https://storage.googleapis.com/releases.grr-response.com/index.html +grr-response-templates` +11. `pip install --editable grr/config/grr-response-test` + +Configuring GRR on the Main Server +---------------------------------- +1. Set up a MTA of your choice +2. Run `grr_config_updater initialize` +* Step 1: Key Generation: - this step autocompletes and needs no user input +* Step 2: Setting Basic Configuration Parameters: + - GRR Datastore: Choose `1` for a SQLite Datastore. That is the datastore + all these tutorials assume you have + - GRR URLs: Enter hostname: `
` + - Server URL: Frontend URL `http://:8080/` + - AdminUI URL: leave as default, press enter + - GRR Emails: Email Domain: `` + - GRR Emails: Alert Email Address: `` + - GRR Emails: Emergency Email Access Email Address: `` +* Step 3: Adding Admin User: `` +* Step 4: Installing template package: `n` +* Step 5: - Completes on its own +3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move +`Client.executable_signing_public_key` to a new file and change permissions +based on your executable signing structure policy +4. _Optional_ - Set up the Approval Process +* Users listed in the approval process will be added later. However this +* configuration must be pushed out to all servers, so it should be set up now. +* https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing] + +Initial Configuration on HTTP Server +------------------------------------ +1. Replace `grr/install_data/etc/server.local.yaml` and +`grr/install_data/etc/grr-server.yaml` with the files from main server + +Configuration on Master Dataserver +---------------------------------- +1. Replace `grr/install_data/etc/server.local.yaml` and +`grr/install_data/etc/grr-server.yaml` with the files from main server +2. Edit `grr/install_data/etc/server.local.yaml` and add the following: ++ +[source,yaml] +Datastore.location: +Dataserver.server_list: + - http://:7000 +Dataserver.client_credentials: + - ::rw +Dataserver.server_username: +Dataserver.server_password: + +* Terms: + - `server_list` is a list of all database servers with the master listed + first + - `client_credentials` are used for the http server and main server to + communicate with the database + - `server_username` and `server_password` are used for any slave dataservers + to talk to the master dataserver. +3. Start the dataserver service by running `grr_server --component +dataserver_master` +* It should start with no errors and not show any information. If you would like +* to see more information add in `--verbose` + +Finish the Configuration on the Main Server +------------------------------------------- +1. Edit install_data/etc/server.local.yaml +* Dataserver Configuration: ++ +[source,yaml] +Dataserver.server_list: + - http://:7000 +Datastore.implementation: HTTPServer +HTTPDatastore.username: +HTTPDatastore.password: + +* Email Configuration: ++ +[source,yaml] +Worker.smtp_server: +Worker.smtp_port: 587 +Worker.smtp_starttle: True +Worker.smtp_user: +Worker.smtp_password: +Email.approval_cc_address: + +2. Add users who should be able to access the admin interface. +* *Note* Users cannot change their password in the admin ui, they must change it +* in the command line +* https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User +* Management] +3. Run `grr_config_updater repack_clients` +* *Note* -If you decided to implement #3 in Configuring GRR on the Main Server +* add `--secondary_configs ` after `grr_config_updater` +4. Start services with `grr_server --component worker and grr_server --component +ui` + +Finish the configuration on HTTP Server +--------------------------------------- +1. Edit `install_data/etc/server.local.yaml` ++ +[source,yaml] +Dataserver Configuration: +Dataserver.server_list: + - http://:7000 +Datastore.implementation: HTTPServer +HTTPDatastore.username: +HTTPDatastore.password: + +2. Start services with `grr_server --component http_server` + +Post Setup +--------- +At this point, you should be able to visit your admin interface, download the +appropriate client installer binary. Once that is installed on the client, it +will communicate back to your HTTP server and be visible through the admin ui. From 147b7f510a5772a149ddf52de131343d2306af65 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Thu, 9 Jun 2016 16:23:52 -0700 Subject: [PATCH 03/21] Fixed typos --- serverenvironmentexample.adoc | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index fc8ea06..6179a55 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -13,10 +13,8 @@ Assumptions Made ---------------- Below are the assumptions made to make use of this example. -* GRR Servers will be running Ubuntu 16.04 LTS with all updates already -* installed -* There will be 3 servers in the configuration as outlined above. They will be -* called Master Dataserver, HTTP Server, and Main Server +* GRR Servers will be running Ubuntu 16.04 LTS with all updates already installed +* There will be 3 servers in the configuration as outlined above. They will be called Master Dataserver, HTTP Server, and Main Server * Bleeding edge GRR is desired and will be installed via pip * The virtual environment will be called GRR_NEW * All networking has been properly set up From 134c8fb1ee6f86d357a88214b05b283897546c81 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Thu, 9 Jun 2016 16:26:27 -0700 Subject: [PATCH 04/21] Fixed typos --- serverenvironmentexample.adoc | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 6179a55..8a1ba66 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -57,8 +57,7 @@ Configuring GRR on the Main Server `Client.executable_signing_public_key` to a new file and change permissions based on your executable signing structure policy 4. _Optional_ - Set up the Approval Process -* Users listed in the approval process will be added later. However this -* configuration must be pushed out to all servers, so it should be set up now. +* Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now. * https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing] Initial Configuration on HTTP Server @@ -90,8 +89,8 @@ Dataserver.server_password: to talk to the master dataserver. 3. Start the dataserver service by running `grr_server --component dataserver_master` -* It should start with no errors and not show any information. If you would like -* to see more information add in `--verbose` +* It should start with no errors and not show any information. If you would + like to see more information add in `--verbose` Finish the Configuration on the Main Server ------------------------------------------- From e5dcd6020d5aeaeeb8056478e9debb78ec48f859 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Thu, 9 Jun 2016 16:27:29 -0700 Subject: [PATCH 05/21] Fixed typos --- serverenvironmentexample.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 8a1ba66..c834a5a 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -115,13 +115,13 @@ Worker.smtp_password: Email.approval_cc_address: 2. Add users who should be able to access the admin interface. -* *Note* Users cannot change their password in the admin ui, they must change it -* in the command line +* *Note* Users cannot change their password in the admin ui, they must change + it in the command line * https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User -* Management] + Management] 3. Run `grr_config_updater repack_clients` * *Note* -If you decided to implement #3 in Configuring GRR on the Main Server -* add `--secondary_configs ` after `grr_config_updater` + add `--secondary_configs ` after `grr_config_updater` 4. Start services with `grr_server --component worker and grr_server --component ui` From 7dbf56dfc87f2d625f1665696759968b357cd055 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Fri, 10 Jun 2016 10:55:46 -0700 Subject: [PATCH 06/21] Fixed approval_cc_address section --- serverenvironmentexample.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index c834a5a..f1e0212 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -114,6 +114,9 @@ Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: +* *Note* `approval_cc_address` should be within the configured domain. This means if +your email domain is configured to `localhost`, and you want the email to go to +`alerts@localhost` you should put `Email.approval_cc_address: alerts` 2. Add users who should be able to access the admin interface. * *Note* Users cannot change their password in the admin ui, they must change it in the command line From 5d99b82b79c16b9fa8b76b9b7c89f4a8e19b92d3 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Fri, 10 Jun 2016 11:19:27 -0700 Subject: [PATCH 07/21] Updated approval_cc section --- serverenvironmentexample.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index f1e0212..052ca89 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -114,9 +114,8 @@ Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: -* *Note* `approval_cc_address` should be within the configured domain. This means if -your email domain is configured to `localhost`, and you want the email to go to -`alerts@localhost` you should put `Email.approval_cc_address: alerts` +* *Note* `approval_cc_address` should only be the account name, your email + domain will be appended later. 2. Add users who should be able to access the admin interface. * *Note* Users cannot change their password in the admin ui, they must change it in the command line From 5001d23213a84738b7295ca24c96014570fd9440 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:23:01 -0700 Subject: [PATCH 08/21] Added reference to install from pip --- serverenvironmentexample.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 052ca89..4a90cc8 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -21,6 +21,9 @@ Below are the assumptions made to make use of this example. Setting up the servers ---------------------- +The steps below are based off of +https://github.com/google/grr-doc/blob/master/installfrompip.adoc[Install from +Pip] documentation. If the below does not work, please refer back to there. 1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git protobuf-compiler` 2. `pip install --upgrade pip` From cac31f1d6a3bd1d13394ac6c6e9ef5ed8c40ce4e Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:24:00 -0700 Subject: [PATCH 09/21] Fixed typo --- serverenvironmentexample.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 4a90cc8..b1a07f2 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -24,6 +24,7 @@ Setting up the servers The steps below are based off of https://github.com/google/grr-doc/blob/master/installfrompip.adoc[Install from Pip] documentation. If the below does not work, please refer back to there. + 1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git protobuf-compiler` 2. `pip install --upgrade pip` From c0902fcb82415d506601fe28b570c7204e30a771 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:27:34 -0700 Subject: [PATCH 10/21] Updated Pulling out the private key --- serverenvironmentexample.adoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index b1a07f2..4eab413 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -58,7 +58,7 @@ Configuring GRR on the Main Server * Step 4: Installing template package: `n` * Step 5: - Completes on its own 3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move -`Client.executable_signing_public_key` to a new file and change permissions +`PrivateKeys.executable_signing_private_key` key to a new file and change permissions based on your executable signing structure policy 4. _Optional_ - Set up the Approval Process * Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now. @@ -118,6 +118,11 @@ Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: +* Private Key configuration ++ +[source,yaml] +PrivateKeys.executable_signing_private_key: "%(|file)" + * *Note* `approval_cc_address` should only be the account name, your email domain will be appended later. 2. Add users who should be able to access the admin interface. From 7ce8e248c0ffb1dd1f1ab53acf96e4de19407f40 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:29:20 -0700 Subject: [PATCH 11/21] Fixed order in configuring main server --- serverenvironmentexample.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 4eab413..4fd2036 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -118,13 +118,13 @@ Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: +* *Note* `approval_cc_address` should only be the account name, your email + domain will be appended later. * Private Key configuration + [source,yaml] PrivateKeys.executable_signing_private_key: "%(|file)" -* *Note* `approval_cc_address` should only be the account name, your email - domain will be appended later. 2. Add users who should be able to access the admin interface. * *Note* Users cannot change their password in the admin ui, they must change it in the command line From 62b30474b2ebd80402d50b3b1690591adc2b4aba Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:30:11 -0700 Subject: [PATCH 12/21] Increased indent on a note --- serverenvironmentexample.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 4fd2036..f100fbb 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -118,7 +118,7 @@ Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: -* *Note* `approval_cc_address` should only be the account name, your email +- *Note* `approval_cc_address` should only be the account name, your email domain will be appended later. * Private Key configuration + From bfe8a3ba6c6d369f2b7ebb8fcd74cde0308b7274 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:32:26 -0700 Subject: [PATCH 13/21] Fixed smtp port to be more general --- serverenvironmentexample.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index f100fbb..4e90795 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -112,7 +112,7 @@ HTTPDatastore.password: + [source,yaml] Worker.smtp_server: -Worker.smtp_port: 587 +Worker.smtp_port: Worker.smtp_starttle: True Worker.smtp_user: Worker.smtp_password: From 0e5214022193d3568f771a09375e834c4da08e9c Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 10:33:08 -0700 Subject: [PATCH 14/21] fixed typo --- serverenvironmentexample.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 4e90795..866a38d 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -113,7 +113,7 @@ HTTPDatastore.password: [source,yaml] Worker.smtp_server: Worker.smtp_port: -Worker.smtp_starttle: True +Worker.smtp_starttls: True Worker.smtp_user: Worker.smtp_password: Email.approval_cc_address: From 0e0f8c3114229fc0861d629dd12c0f85e1d46346 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 15:05:33 -0700 Subject: [PATCH 15/21] Added additional info re private key --- serverenvironmentexample.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 866a38d..eac23b8 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -60,6 +60,7 @@ Configuring GRR on the Main Server 3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move `PrivateKeys.executable_signing_private_key` key to a new file and change permissions based on your executable signing structure policy +* When you move the key to a new file, remove all extra whitespace as well. 4. _Optional_ - Set up the Approval Process * Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now. * https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing] From a6232fe6e35dc95fce137ebf9d859c94f1126f70 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 15:06:58 -0700 Subject: [PATCH 16/21] Updated Steps for privatekey process --- serverenvironmentexample.adoc | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index eac23b8..59248a2 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -131,10 +131,7 @@ PrivateKeys.executable_signing_private_key: "%(|file)" it in the command line * https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User Management] -3. Run `grr_config_updater repack_clients` -* *Note* -If you decided to implement #3 in Configuring GRR on the Main Server - add `--secondary_configs ` after `grr_config_updater` -4. Start services with `grr_server --component worker and grr_server --component +3. Start services with `grr_server --component worker and grr_server --component ui` Finish the configuration on HTTP Server From 99fe831b25ec83c8d8c808c68a57ebffed8eba1e Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Mon, 13 Jun 2016 15:18:04 -0700 Subject: [PATCH 17/21] Fixed typo --- serverenvironmentexample.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index 59248a2..c41a78b 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -5,7 +5,7 @@ GRR Server Environment Example Setup :icons: A GRR Server Environment can be whatever you make it. This example goes over the -steps necessary to a three server environment: one to house the front end HTTP +steps necessary to create a three server environment: one to house the front end HTTP service, one to house the master datastore, and one to house the admin ui and workers. From f8abf3bf3bfe870faa4213ef3d999cf6f7cf334b Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Wed, 13 Jul 2016 14:19:52 -0700 Subject: [PATCH 18/21] Added http datastore doc --- httpDatastore.adoc | 102 ++++++++++++++++++++++++++++++++++ serverenvironmentexample.adoc | 2 +- 2 files changed, 103 insertions(+), 1 deletion(-) create mode 100644 httpDatastore.adoc diff --git a/httpDatastore.adoc b/httpDatastore.adoc new file mode 100644 index 0000000..eb67dd7 --- /dev/null +++ b/httpDatastore.adoc @@ -0,0 +1,102 @@ +How to create an HTTP datastore environment +=========================================== +Assumptions Made +---------------- +* GRR servers will be running on Ubuntu 16.04 LTS following +https://github.com/google/grr-doc/blob/master/installfrompip.adoc#installing-grr-server-for-dev-ie-tracking-head[Install +from Pip] documentation +* A “main server” will be hosting all the other services in the infrastructure. +Virtual Environment created is called “GRR_NEW” +* It is desired to have every data server host through port tcp:7000 (this is not +a necessity) +* All services are down for this configuration change. +* On every server you work with in this document, you activated your virtualenv +and changed into grr/ + +Set up the master dataserver +---------------------------- +1. Copy the /install_data/etc/server.local.yaml file from your main server to your +master dataserver +2. Edit the server.local.yaml with either changing or adding the following: ++ +[source,yaml] +Datastore.location: +Datastore.implementation: SQLiteDataStore +Dataserver.server_list: + - http://:7000 + - http://:7000 + - http://:7000 +Dataserver.client_credentials: + - ::rw +Dataserver.server_username: +Dataserver.server_password: + +3. Run `python grr/server/data_server/data_server.py +--config=install_data/etc/grr-server.yaml --master` to start the service and +verify it starts correctly. Keep this running + +Set up the slave dataservers +---------------------------- +1. Copy install_data/etc/server.local.yaml from the main server to your slave +dataserver +2. Edit grr/install_data/etc/server.local.yaml. Change or add the following: ++ +[source,yaml] +Dataserver.server_list: + - :7000 +Dataserver.server_username: +Dataserver.server_password: +Datastore.implementation: SqliteDataStore +Datastore.location: + +* Notes: + - The server_username and server_password were configured in the setup of the +master dataserver. + - The configuration lines may not be in a sensible order, make sure to search for + them within the document to avoid duplicates. Dataserver.server_username and + Dataserver.server_password will need to be added in. The others will be modified +3. Run the command qgrr_server --component dataserver_slave --verbose` to see the +data server slave connect. +4. Repeat for each slave dataserver you wish to create +5. *On the master server* in the management interface, after adding and connecting +all the data servers, run rebalance and confirm that you would like to reshard +the database. + +Update the main server configuration +------------------------------------ +1. Edit install_data/etc/server.local.yaml ++ +[source,yaml] +Dataserver.server_list: + - http://:7000 + - http://:7000 +Datastore.implementation: HTTPDataStore +HTTPDataStore.username: +HTTPDataStore.password: + + +Add more slave dataservers after initial setup +---------------------------------------------- +Change configuration on the master dataserver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +1. Run the command `python grr/server/data_server/manager.py +--config=install_data/etc/grr-server.yaml` to start the dataserver management +interface + - While in the interface, run `addserver 7000` and confirm that +you would like to add it. Repeat for all dataserver slaves. + - Keep this interface up +2. In a separate tab edit install_data/etc/server.local.yaml. Change the +`Dataserver.server_list section` to add any dataserver slaves you want to add ++ +[source,yaml] +Dataserver.server_list: + - :7000 + - :7000 + - :7000 + +3. Run the command `python grr/server/data_server/data_server.py +--config=install_data/etc/grr-server.yaml --master` + - Leave this command running, keep the screen somewhere you can monitor it. +4. Set up the slave dataservers as you did above +5. Update the main server configuration as you did above with the new +dataserver slaves diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc index c41a78b..c4e1d7a 100644 --- a/serverenvironmentexample.adoc +++ b/serverenvironmentexample.adoc @@ -26,7 +26,7 @@ https://github.com/google/grr-doc/blob/master/installfrompip.adoc[Install from Pip] documentation. If the below does not work, please refer back to there. 1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git -protobuf-compiler` +protobuf-compiler prelink` 2. `pip install --upgrade pip` 3. `pip install virtualenv` 4. `git clone https://github.com/google/grr.git` From ea40b294f199c343c80d477b6818d13e4b84ab5d Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Wed, 13 Jul 2016 14:23:56 -0700 Subject: [PATCH 19/21] Fixed typos --- httpDatastore.adoc | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/httpDatastore.adoc b/httpDatastore.adoc index eb67dd7..d30442e 100644 --- a/httpDatastore.adoc +++ b/httpDatastore.adoc @@ -15,9 +15,9 @@ and changed into grr/ Set up the master dataserver ---------------------------- -1. Copy the /install_data/etc/server.local.yaml file from your main server to your +1. Copy the `/install_data/etc/server.local.yaml` file from your main server to your master dataserver -2. Edit the server.local.yaml with either changing or adding the following: +2. Edit the `server.local.yaml` with either changing or adding the following: + [source,yaml] Datastore.location: @@ -37,9 +37,9 @@ verify it starts correctly. Keep this running Set up the slave dataservers ---------------------------- -1. Copy install_data/etc/server.local.yaml from the main server to your slave -dataserver -2. Edit grr/install_data/etc/server.local.yaml. Change or add the following: +1. Copy `install_data/etc/server.local.yaml` from the main server to your +dataserver slave +2. Edit `install_data/etc/server.local.yaml`. Change or add the following: + [source,yaml] Dataserver.server_list: @@ -50,21 +50,21 @@ Datastore.implementation: SqliteDataStore Datastore.location: * Notes: - - The server_username and server_password were configured in the setup of the + - The `server_username` and `server_password` were configured in the setup of the master dataserver. - The configuration lines may not be in a sensible order, make sure to search for - them within the document to avoid duplicates. Dataserver.server_username and - Dataserver.server_password will need to be added in. The others will be modified -3. Run the command qgrr_server --component dataserver_slave --verbose` to see the + them within the document to avoid duplicates. `Dataserver.server_username` and + `Dataserver.server_password` will need to be added in. The others will be modified +3. Run the command `grr_server --component dataserver_slave --verbose` to see the data server slave connect. -4. Repeat for each slave dataserver you wish to create +4. Repeat for each dataserver slave you wish to create 5. *On the master server* in the management interface, after adding and connecting -all the data servers, run rebalance and confirm that you would like to reshard +all the data servers, run `rebalance` and confirm that you would like to reshard the database. Update the main server configuration ------------------------------------ -1. Edit install_data/etc/server.local.yaml +1. Edit `install_data/etc/server.local.yaml` + [source,yaml] Dataserver.server_list: @@ -75,7 +75,7 @@ HTTPDataStore.username: HTTPDataStore.password: -Add more slave dataservers after initial setup +Add more dataserver slaves after initial setup ---------------------------------------------- Change configuration on the master dataserver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -85,7 +85,7 @@ interface - While in the interface, run `addserver 7000` and confirm that you would like to add it. Repeat for all dataserver slaves. - Keep this interface up -2. In a separate tab edit install_data/etc/server.local.yaml. Change the +2. In a separate tab edit `install_data/etc/server.local.yaml`. Change the `Dataserver.server_list section` to add any dataserver slaves you want to add + [source,yaml] @@ -97,6 +97,6 @@ Dataserver.server_list: 3. Run the command `python grr/server/data_server/data_server.py --config=install_data/etc/grr-server.yaml --master` - Leave this command running, keep the screen somewhere you can monitor it. -4. Set up the slave dataservers as you did above +4. Set up the dataserver slaves as you did above 5. Update the main server configuration as you did above with the new dataserver slaves From 813f0d46530a345ffb0ff367bc39e133fad46d53 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Wed, 13 Jul 2016 14:26:01 -0700 Subject: [PATCH 20/21] Fixed typo --- httpDatastore.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/httpDatastore.adoc b/httpDatastore.adoc index d30442e..1c93bb7 100644 --- a/httpDatastore.adoc +++ b/httpDatastore.adoc @@ -86,7 +86,7 @@ interface you would like to add it. Repeat for all dataserver slaves. - Keep this interface up 2. In a separate tab edit `install_data/etc/server.local.yaml`. Change the -`Dataserver.server_list section` to add any dataserver slaves you want to add +`Dataserver.server_list` section to add any dataserver slaves you want to add + [source,yaml] Dataserver.server_list: From 1b7417267a7c8126e3d5ac08723551a99feb8171 Mon Sep 17 00:00:00 2001 From: Jessica Wilson Date: Wed, 13 Jul 2016 14:27:38 -0700 Subject: [PATCH 21/21] Removed serverenvironmentexample.adoc --- serverenvironmentexample.adoc | 155 ---------------------------------- 1 file changed, 155 deletions(-) delete mode 100644 serverenvironmentexample.adoc diff --git a/serverenvironmentexample.adoc b/serverenvironmentexample.adoc deleted file mode 100644 index c4e1d7a..0000000 --- a/serverenvironmentexample.adoc +++ /dev/null @@ -1,155 +0,0 @@ -GRR Server Environment Example Setup -==================================== -:toc: -:toc-placement: preamble -:icons: - -A GRR Server Environment can be whatever you make it. This example goes over the -steps necessary to create a three server environment: one to house the front end HTTP -service, one to house the master datastore, and one to house the admin ui and -workers. - -Assumptions Made ----------------- -Below are the assumptions made to make use of this example. - -* GRR Servers will be running Ubuntu 16.04 LTS with all updates already installed -* There will be 3 servers in the configuration as outlined above. They will be called Master Dataserver, HTTP Server, and Main Server -* Bleeding edge GRR is desired and will be installed via pip -* The virtual environment will be called GRR_NEW -* All networking has been properly set up - -Setting up the servers ----------------------- -The steps below are based off of -https://github.com/google/grr-doc/blob/master/installfrompip.adoc[Install from -Pip] documentation. If the below does not work, please refer back to there. - -1. `apt-get install debhelper dpkg-dev libssl-dev python-dev python-pip rpm git -protobuf-compiler prelink` -2. `pip install --upgrade pip` -3. `pip install virtualenv` -4. `git clone https://github.com/google/grr.git` -5. `virtualenv GRR_NEW` -6. `source GRR_NEW/bin/activate` -7. `cd grr/` -8. `pip install --editable .` -9. `pip install --editable grr/config/grr-response-server` -10. `pip install --no-cache-dir -f -https://storage.googleapis.com/releases.grr-response.com/index.html -grr-response-templates` -11. `pip install --editable grr/config/grr-response-test` - -Configuring GRR on the Main Server ----------------------------------- -1. Set up a MTA of your choice -2. Run `grr_config_updater initialize` -* Step 1: Key Generation: - this step autocompletes and needs no user input -* Step 2: Setting Basic Configuration Parameters: - - GRR Datastore: Choose `1` for a SQLite Datastore. That is the datastore - all these tutorials assume you have - - GRR URLs: Enter hostname: `
` - - Server URL: Frontend URL `http://:8080/` - - AdminUI URL: leave as default, press enter - - GRR Emails: Email Domain: `` - - GRR Emails: Alert Email Address: `` - - GRR Emails: Emergency Email Access Email Address: `` -* Step 3: Adding Admin User: `` -* Step 4: Installing template package: `n` -* Step 5: - Completes on its own -3. _Optional_ - Edit `grr/install_data/etc/server.local.yaml` and move -`PrivateKeys.executable_signing_private_key` key to a new file and change permissions -based on your executable signing structure policy -* When you move the key to a new file, remove all extra whitespace as well. -4. _Optional_ - Set up the Approval Process -* Users listed in the approval process will be added later. However this configuration must be pushed out to all servers, so it should be set up now. -* https://github.com/google/grr-doc/blob/master/admin.adoc#auditing[Auditing] - -Initial Configuration on HTTP Server ------------------------------------- -1. Replace `grr/install_data/etc/server.local.yaml` and -`grr/install_data/etc/grr-server.yaml` with the files from main server - -Configuration on Master Dataserver ----------------------------------- -1. Replace `grr/install_data/etc/server.local.yaml` and -`grr/install_data/etc/grr-server.yaml` with the files from main server -2. Edit `grr/install_data/etc/server.local.yaml` and add the following: -+ -[source,yaml] -Datastore.location: -Dataserver.server_list: - - http://:7000 -Dataserver.client_credentials: - - ::rw -Dataserver.server_username: -Dataserver.server_password: - -* Terms: - - `server_list` is a list of all database servers with the master listed - first - - `client_credentials` are used for the http server and main server to - communicate with the database - - `server_username` and `server_password` are used for any slave dataservers - to talk to the master dataserver. -3. Start the dataserver service by running `grr_server --component -dataserver_master` -* It should start with no errors and not show any information. If you would - like to see more information add in `--verbose` - -Finish the Configuration on the Main Server -------------------------------------------- -1. Edit install_data/etc/server.local.yaml -* Dataserver Configuration: -+ -[source,yaml] -Dataserver.server_list: - - http://:7000 -Datastore.implementation: HTTPServer -HTTPDatastore.username: -HTTPDatastore.password: - -* Email Configuration: -+ -[source,yaml] -Worker.smtp_server: -Worker.smtp_port: -Worker.smtp_starttls: True -Worker.smtp_user: -Worker.smtp_password: -Email.approval_cc_address: - -- *Note* `approval_cc_address` should only be the account name, your email - domain will be appended later. -* Private Key configuration -+ -[source,yaml] -PrivateKeys.executable_signing_private_key: "%(|file)" - -2. Add users who should be able to access the admin interface. -* *Note* Users cannot change their password in the admin ui, they must change - it in the command line -* https://github.com/google/grr-doc/blob/master/admin.adoc#user-management[User - Management] -3. Start services with `grr_server --component worker and grr_server --component -ui` - -Finish the configuration on HTTP Server ---------------------------------------- -1. Edit `install_data/etc/server.local.yaml` -+ -[source,yaml] -Dataserver Configuration: -Dataserver.server_list: - - http://:7000 -Datastore.implementation: HTTPServer -HTTPDatastore.username: -HTTPDatastore.password: - -2. Start services with `grr_server --component http_server` - -Post Setup ---------- -At this point, you should be able to visit your admin interface, download the -appropriate client installer binary. Once that is installed on the client, it -will communicate back to your HTTP server and be visible through the admin ui.