Admin docs contains deprecated info

[pettai]

Hi,

We're getting ready to deploy GRR on a larger scale, but repacking clients as described in the docs is not working because the docs are deprecated:
https://github.com/google/grr-doc/blob/master/admin.adoc#repacking-clients-with-custom-labels-multi-organization-deployments
(many of the CLI options are either renamed or doesn't exists anymore)

I also found this thread that describes the a newer way of doing it which actually is sane to the latest stable GRR release (3.1.0.2)
https://groups.google.com/forum/#!searchin/grr-users/repacking|sort:relevance/grr-users/PUpiGT9zPuw/7G5OM-yLQwAJ

For example:
root@grr:~# grr_client_build buildandrepack --template /usr/share/grr-server/grr-response-templates/templates/
Building installers for: ['AllPlatforms Context']
Repacking AllPlatforms Context as grr with labels: []
Repacking template: /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.deb.zip
Loading configuration from /tmp/tmp5If28L/grrd.yaml
Configuration writeback is set to /tmp/tmp5If28L/grrd.yaml
Writing back configuration to file /tmp/tmp5If28L/grrd.yaml
Build Config Error: Empty Client.server_urls
Build Config Error: Missing Client.executable_signing_public_key.
Build Config Error: CA certificate missing from config.
Repacking template /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.deb.zip failed: Bad configuration generated. Terminating.
Failed to repack /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.deb.zip.
Complete, installers for ['AllPlatforms_Context'] are in /usr/share/grr-server/executables/2017-04-24T20:55:15Z/linux_amd64_deb

I can't find grrd.yaml in the docs nor in the grr installations (done with the provided script)

[mbushkov]

Hi! Quick question - how did you install the GRR server? From PIP or from a DEB package?

If you need to repack the client, then you likely need "grr_client_build repack" command, not "buildandrepack". You don't need to provide grrd.yaml or anything like this - grr_client_build will use the current config of the GRR server.

Here's an example of repacking a client with a grr server installed from PIP:

# virtualenv .
# source  bin/activate
# pip install grr-response-server
# pip install --no-cache-dir -f https://storage.googleapis.com/releases.grr-response.com/index.html grr-response-templates
# grr_config_updater initialize
# grr_client_build repack --template grr_3.1.0.2_amd64.xar.zip --outputdir .

[pettai]

Ah, sorry for not being clear, "the (bash) script" that installs GRR on ubuntu Xenial using the DEB package.
I also tried the same thing you suggested earlier, but with the same result as before (but uses grr.yaml instead of grrd.yaml)

root@grr:~# grr_client_build repack --template /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.xar.zip --outputdir=./grr_installers
Repacking template: /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.xar.zip
Loading configuration from /tmp/tmpQzfOXj/grr.yaml
Configuration writeback is set to /tmp/tmpQzfOXj/grr.yaml
Writing back configuration to file /tmp/tmpQzfOXj/grr.yaml
Build Config Error: Empty Client.server_urls
Build Config Error: Missing Client.executable_signing_public_key.
Build Config Error: CA certificate missing from config.
Repacking template /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.xar.zip failed: Bad configuration generated. Terminating.
Failed to repack /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.xar.zip.

[mbushkov]

Did you run "grr_config_updater initialize" on a machine where you're repacking the clients? Is GRR service actually running on this machine?

[pettai]

Yes, I'm pretty sure the DEB package calls that command in the postinstall script, because then I run "grr_config_updater initialize" manually I just get the same questions I answered during the installation using "install_script_ubuntu.sh".

root@grr:~# grr_config_updater initialize
Checking write access on config /etc/grr//server.local.yaml

Step 0: Importing Configuration from previous installation.
No old config file found.

Step 1: Key Generation
You already have keys in your config, do you want to overwrite them? [yN]:

Step 2: Setting Basic Configuration Parameters
We are now going to configure the server using a bunch of questions.


-=GRR Datastore=-
For GRR to work each GRR server has to be able to communicate with the
datastore.  To do this we need to configure a datastore.

Found existing settings:
  Datastore: SqliteDataStore
  Datastore Location: /usr/share/grr-server/lib/python2.7/site-packages/grr/var/grr-datastore

Do you want to keep this configuration? [Yn]:

[...]

Anyway, I ran thru "grr_config_updater initialize" just to be sure, and tried to issue the repacking command once again, but with the same result:

root@grr:~# grr_client_build repack --template /usr/share/grr-server/grr-response-templates/templates/GRR_3.1.0.2_amd64.exe.zip --outputdir=./grr_installers
Repacking template: /usr/share/grr-server/grr-response-templates/templates/GRR_3.1.0.2_amd64.exe.zip
Loading configuration from /tmp/tmpOReo3N/GRR.exe.yaml
Configuration writeback is set to /tmp/tmpOReo3N/GRR.exe.yaml
Writing back configuration to file /tmp/tmpOReo3N/GRR.exe.yaml
Build Config Error: Empty Client.server_urls
Build Config Error: Missing Client.executable_signing_public_key.
Build Config Error: CA certificate missing from config.
Repacking template /usr/share/grr-server/grr-response-templates/templates/GRR_3.1.0.2_amd64.exe.zip failed: Bad configuration generated. Terminating.
Failed to repack /usr/share/grr-server/grr-response-templates/templates/GRR_3.1.0.2_amd64.exe.zip.

GRR is running AFAICT:

root@grr:~# ps -ef | grep grr
root     11281     1  0 Apr24 ?        00:00:00 /bin/bash /usr/bin/grr_server --component http_server --disallow_missing_config_definitions -p StatsStore.process_id=http_server_122d7fea6cc7429eb4a9a59b76530b4a
root     11282     1  0 Apr24 ?        00:00:00 /bin/bash /usr/bin/grr_server --component ui --disallow_missing_config_definitions -p StatsStore.process_id=ui_122d7fea6cc7429eb4a9a59b76530b4a
root     11283     1  0 Apr24 ?        00:00:00 /bin/bash /usr/bin/grr_server --component worker --disallow_missing_config_definitions -p StatsStore.process_id=worker_122d7fea6cc7429eb4a9a59b76530b4a
root     11288     1  0 Apr24 ?        00:00:00 /bin/bash /usr/bin/grr_server --component worker2 --disallow_missing_config_definitions -p StatsStore.process_id=worker2_122d7fea6cc7429eb4a9a59b76530b4a
root     11291 11281  0 Apr24 ?        00:06:08 /usr/share/grr-server/bin/python /usr/share/grr-server//bin/grr_server --context Global Install Context --component http_server --disallow_missing_config_definitions -p StatsStore.process_id=http_server_122d7fea6cc7429eb4a9a59b76530b4a
root     11292 11288  0 Apr24 ?        00:10:23 /usr/share/grr-server/bin/python /usr/share/grr-server//bin/grr_server --context Global Install Context --component worker2 --disallow_missing_config_definitions -p StatsStore.process_id=worker2_122d7fea6cc7429eb4a9a59b76530b4a
root     11303 11283  0 Apr24 ?        00:10:08 /usr/share/grr-server/bin/python /usr/share/grr-server//bin/grr_server --context Global Install Context --component worker --disallow_missing_config_definitions -p StatsStore.process_id=worker_122d7fea6cc7429eb4a9a59b76530b4a
root     11304 11282  0 Apr24 ?        00:04:18 /usr/share/grr-server/bin/python /usr/share/grr-server//bin/grr_server --context Global Install Context --component ui --disallow_missing_config_definitions -p StatsStore.process_id=ui_122d7fea6cc7429eb4a9a59b76530b4a

[mbushkov]

I reproduced the issue. Seems that grr_client_build loads secondary configuration file from /usr/share/grr-server/install_data/etc/server.local.yaml and not from /etc/grr/server.local.yaml. /etc/grr/server.local.yaml is the deployment-specific configuration file that contains configuration parameters necessary to repack the client: Client.server_urls, Client.executable_signing_public_key and CA.

This behavior seems to be different from grr_client_build's behavior when GRR server is installed from pip. I'll take a closer look at it.

In the meantime, this should work:

sudo /usr/share/grr-server/bin/grr_client_build --secondary_configs /etc/grr/server.local.yaml --verbose repack --template grr_3.1.0.2_amd64.deb.zip --outputdir=/grr-installers

[pettai]

Yes finally repack works, thanks for looking into this.
But the example command only repack one template once (and AFAICT, it doesn't add any label in grrd.yaml too). The initial issue with the documentation was that doing multiple repacking of clients with labels, as described here:
https://groups.google.com/forum/#!searchin/grr-users/repacking%7Csort:relevance/grr-users/scEpaarIiOI/DdUTOZ4BPQAJ

Hence, my initial cli command was using option "buildandrepack" (like in the email thread above). But by adding --secondary_configs /etc/grr/server.local.yaml it now fails because it can't build chipsec. So this seems like the wrong way for just repacking multiple clients with different labels?

My buildconfig.yaml (additionally) contains:

[...]
Customer1 Context:
  Client.labels: [customer1]

Customer2 Context:
  Client.labels: [customer2]
  ClientBuilder.target_platforms:
    - windows_amd64_exe
    - windows_i386_exe

I tested adding --context Customer1 then doing "repack"
/usr/share/grr-server/bin/grr_client_build --secondary_configs /etc/grr/server.local.yaml --verbose --context Customer1 repack --template GRR_3.1.0.2_amd64.exe.zip --outputdir=./grr-installers

However, it seems that it has no knowledge of the context (I guess it doesn't consult buildconfig.yaml)

Traceback (most recent call last):
  File "/usr/share/grr-server/bin/grr_client_build", line 11, in <module>
    sys.exit(ClientBuild())
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/distro_entry.py", line 119, in ClientBuild
    flags.StartMain(client_build.main)
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/flags.py", line 121, in StartMain
    main([sys.argv[0]])
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/client/client_build.py", line 364, in main
    startup.ClientInit()
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/startup.py", line 57, in ClientInit
    ConfigInit()
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/startup.py", line 37, in ConfigInit
    config_lib.ParseConfigCommandLine()
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/config_lib.py", line 1682, in ParseConfigCommandLine
    CONFIG.AddContext(context)
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/grr/lib/config_lib.py", line 879, in AddContext
    context_string)
grr.lib.config_lib.InvalidContextError: Invalid context specified: Customer1

Is doing multiple template repacking and adding labels not supported anymore?

[mbushkov]

It is very much supported. Will take a look into that and get back to you.

[mbushkov]

I had to experiment a bit, since the version of GRR released as a DEB package is pretty old and the way repacking works is changed in HEAD.

The command above should be:

/usr/share/grr-server/bin/grr_client_build --secondary_configs /etc/grr/server.local.yaml \
  --context "Customer1 Context" repack \
  --template GRR_3.1.0.2_amd64.exe.zip --outputdir ./grr-installers

Note that "Context" is actually a part of the context name, so you have to specify it as "Customer1 Context".

But, basically instructions in the Google Groups thread you linked should work (including repacking with multiple contexts), provided that you use --secondary_configs argument with grr_client_build. I.e.:

/usr/share/grr-server/bin/grr_client_build --platform windows --arch amd64 \
  --secondary_configs /etc/grr/server.local.yaml \
  buildandrepack --template /usr/share/grr-server/grr-response-templates/templates/

Please tell me if this works for you.

[pettai]

Thanks.

Both commands works in the sense that they actually repack client(s), but none of them produce repacked client with labels :(

/usr/share/grr-server/bin/grr_client_build --secondary_configs /etc/grr/server.local.yaml   --context "Customer1 Context" repack   --template GRR_3.1.0.2_amd64.exe.zip --outputdir ./grr_installers
Repacking template: /usr/share/grr-server/grr-response-templates/templates/GRR_3.1.0.2_amd64.exe.zip
Loading configuration from /tmp/tmpQkTNb3/GRR.exe.yaml
Configuration writeback is set to /tmp/tmpQkTNb3/GRR.exe.yaml
Writing back configuration to file /tmp/tmpQkTNb3/GRR.exe.yaml
Deployable binary generated at ./grr_installers/GRR_3.1.0.2_amd64.exe
Repacked into ./grr_installers/GRR_3.1.0.2_amd64.exe

After installing this client it shows up in GRR, but without a label.
(I tried with bogus context too, but that (correctly) causes the repack to fail...)

Second command only produces "AllPlatforms_Context" (which what you get default anyway). No (custom) contexts (with labels) are generated.

I tried adding --context "Customer1 Context", but that doesn't seems be caught, and again "AllPlatforms_Context" is used:

root@grr:~# /usr/share/grr-server/bin/grr_client_build --context "Customer1 Context"  --secondary_configs /etc/grr/server.local.yaml   buildandrepack --template /usr/share/grr-server/grr-response-templates/templates/
Building installers for: ['AllPlatforms Context']
Repacking AllPlatforms Context as grr with labels: []
Repacking template: /usr/share/grr-server/grr-response-templates/templates/grr_3.1.0.2_amd64.deb.zip
Loading configuration from /tmp/tmppO_zud/grrd.yaml
Configuration writeback is set to /tmp/tmppO_zud/grrd.yaml
Writing back configuration to file /tmp/tmppO_zud/grrd.yaml
Created package /usr/share/grr-server/executables/2017-04-26T20:03:39Z/linux_amd64_deb/AllPlatforms_Context/grr_3.1.0.2_amd64.deb
Repacked into /usr/share/grr-server/executables/2017-04-26T20:03:39Z/linux_amd64_deb/AllPlatforms_Context/grr_3.1.0.2_amd64.deb
Complete, installers for ['AllPlatforms_Context'] are in /usr/share/grr-server/executables/2017-04-26T20:03:39Z/linux_amd64_deb

[mbushkov]

Just checking - did you add a list of contexts to /usr/share/grr-server/install_data/etc/buildconfig.yaml like this?

ClientBuilder.BuildTargets:
  - AllPlatforms Context
  - Customer1 Context

Can you please also send the output of your command with the --verbose flag?

[pettai]

No, I put that in /etc/grr/buildconfig.yaml, but after I copied the file to /usr/share/grr-server/install_data/etc/buildconfig.yaml it finally works!

(btw. Why having the same (redundant) config at two places?)

[pettai]

Thx for the help sorting this issue out

[pettai]

#75

[ntdsutil]

Hey GRR Guys!
I don't want to start a new issue because my question belongs to the documentation.
I'm actually writing my Master Thesis about remote forensics and GRR is part of it.
Now I'm a bit disappointed because this part of the docs ist still TBD. https://github.com/google/grr-doc/blob/master/implementation.adoc#client-communication-and-encryption
Sure there is a general overview of your security implementation but I need some more specific information about client - server communication and encryption. Are there some slides or further informations?
Regards!

[mbushkov]

Hi! We have a bunch of publications that may be of interest here: https://github.com/google/grr-doc/blob/master/publications.adoc
I'm pretty sure that some of them have info about client-server communication, although it's likely that none of them dives into details.