Skip to content

Commit 072e9f8

Browse files
jenworthingtonjenworthington
authored
Update README.md
1 parent ed3a24b commit 072e9f8

File tree

1 file changed

+255
-36
lines changed

1 file changed

+255
-36
lines changed

‎README.md‎

Lines changed: 255 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,273 @@
1-
# Splunk Log-driver plugin for Docker
21

3-
Splunk Logging Plugin allows docker containers to send their logs directly to a Splunk Enterprise service or a Splunk
4-
Cloud deployment.
2+
# What does Splunk Connect for Docker do?
3+
Splunk Connect for Docker is a plug-in that extends and expands Docker's logging capabilities so that customers can push their Docker and container logs to their Splunk on-premise or cloud deployment.
54

6-
## Getting Started
5+
Splunk Connect for Docker is a supported open source product and customers with and active Splunk support contract are able to receive Splunk Extension support under the Splunk Support Policy. Please refer to Docker Engine managed plugin system documentation on support for Microsoft Windows and other platforms as well as Prerequisites in this document.
76

8-
You need to install Docker Engine >= 1.12.
7+
Splunk Connect for Docker is not supported on windows.
98

10-
Additional information about Docker plugins [can be found here.](https://docs.docker.com/engine/extend/plugins_logging/)
9+
# Prerequisites
10+
Before you install Splunk Connect for Docker, make sure your system meets the following minimum prerequisites:
1111

12+
* Docker Engine: Version 17.05 or later. If you plan to configure Splunk Connect for Docker via 'daemon.json', you must have the Docker Community Edition (Docker-ce) 18.03 equivalent or later installed.
13+
* Splunk Enterprise, Splunk Light, or Splunk Cloud version 6.6 or later. Splunk Connect for Docker plugin is not currently supported on Windows.
14+
* For customers deploying to Splunk Cloud, HEC must be enabled and a token must be generated by Splunk Support before logs can be ingested.
15+
* Configure an HEC token on Splunk Enterprise or Splunk Light (either single instance or distributed environment). Refer to the set up and use HTTP Event Collector documentation for more details.
16+
* Operating System Platform support as defined in Docker Engine managed plugin system documentation.
1217

13-
### Developing
18+
# Install and configure Splunk Connect for Docker
1419

15-
For development, you can clone and run make
20+
## Step 1: Get an HTTP Event Collector token
21+
You must configure the Splunk HTTP Event Collector (HEC) to send your Docker container logging data to Splunk Enterprise or Splunk Cloud. HEC uses tokens as an alternative to embedding Splunk Enterprise or Splunk Cloud credentials in your app or supporting files. For more about how the HTTP event collector works, see http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector
1622

17-
```
18-
git clone git@github.com:splunk/docker-logging-plugin.git
19-
cd docker-logging-plugin
20-
make
21-
```
23+
1. Enable your HTTP Event collector:
24+
http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/HECWalkthrough#Enable_HEC
25+
2. Create an HEC token:
26+
http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector
27+
http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UseHECusingconffiles
2228

23-
### Installing
29+
Note the following when you generate your token:
30+
* Make sure that indexer acknowledgement is disabled for your token.
31+
* Optionally, enable the indexer acknowledgement functionality by clicking the Enable indexer management checkbox.
32+
* Do not generate your token using the default TLS cert provided by Splunk. For information about configuring Splunk to use self-signed or third-party certs, see http://docs.splunk.com/Documentation/Splunk/7.0.3/Security/AboutsecuringyourSplunkconfigurationwithSSL.
33+
* Splunk Cloud customers must file a support request in order to have a token generated.
2434

25-
To install the plugin, you can run
35+
## Step 2: Install the plugin
36+
There are multiple ways to install Splunk Connect for Docker, Splunk recommends installing from Docker Store (option 1) to ensure the most current and stable build.
2637

27-
```
28-
docker plugin install splunk/docker-logging-driver:latest --alias splunk
29-
docker plugin ls
30-
```
38+
### Install the Plugin from Docker Store
3139

32-
This command will pull and enable the plugin
40+
1. Create the plugin from the package.
3341

34-
### Using
42+
$ docker plugin install splunk/docker-logging-plugin --alias splunk-logging-plugin
3543

36-
The plugin uses the same parameters as the [splunk logging driver](https://docs.docker.com/engine/admin/logging/splunk/).
44+
2. Enable the plugin:
3745

46+
$ docker plugin enable splunk-logging-plugin:latest
3847

39-
#### Splunk Enterprise Example
48+
### Install the plugin from the tar file
4049

41-
```
42-
$ docker run --log-driver=splunk \
43-
--log-opt splunk-url=https://your-splunkhost:8088 \
44-
--log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
45-
--log-opt splunk-capath=/path/to/cert/cacert.pem \
46-
--log-opt splunk-caname=SplunkServerDefaultCert \
47-
--log-opt tag="{{.Name}}/{{.FullID}}" \
48-
--log-opt labels=location \
49-
--log-opt env=TEST \
50-
--env "TEST=false" \
51-
--label location=west \
52-
-it ubuntu bash
50+
1. Clone the repository and check out release branch
5351

54-
```
52+
$ git clone https://github.com/splunk/docker-logging-plugin.git
53+
54+
$ git checkout release/2.0.0
55+
56+
2. Create the plugin package
57+
58+
$ cd docker-logging-plugin
59+
$ make package # this creates a splunk-logging-plugin.tar.gz
60+
61+
3. unzip the package
62+
63+
$ tar -xzf splunk-logging-plugin.tar.gz
64+
65+
4. Create the plugin
66+
67+
$ docker plugin create splunk-logging-plugin:latest splunk-logging-plugin/
68+
69+
5. Verify that the plugin is installed by running the following command:
70+
71+
$ docker plugin ls
72+
73+
6. Enable the plugin
74+
75+
$ docker plugin enable splunk-logging-plugin:latest
76+
77+
## Step 3: Run containers with the plugin installed
78+
79+
Splunk Connect for Docker continually listens for logs, but your containers must also be running so that the container logs are forwarded to Splunk Connect for Docker. The following examples describe how to configure containers to run with Splunk Connect for Docker.
80+
81+
To start your containers, refer to the Docker Documentation found at:
82+
83+
https://docs.docker.com/config/containers/logging/configure/
84+
https://docs.docker.com/config/containers/logging/configure/#configure-the-delivery-mode-of-log-messages-from-container-to-log-driver
85+
86+
## Examples
87+
88+
This sample <addr>daemon.json</addr> command configures Splunk Connect for Docker for all containers on the docker engine. Splunk Software recommends that when working in a production environment, you pass your HEC token through <addr>daemon.json</addr> as opposed to the command line.
89+
90+
{
91+
"log-driver": "splunk-logging-plugin",
92+
"log-opts": {
93+
"splunk-url": "<splunk_hec_endpoint>",
94+
"splunk-token": "<splunk-hec-token>",
95+
"splunk-insecureskipverify": "true"
96+
}
97+
}
98+
99+
This sample command configures Splunk Connect for Docker for a single container.
100+
101+
$ docker run --log-driver=splunk-logging-plugin --log-opt splunk-url=<splunk_hec_endpoint> --log-opt splunk-token=<splunk-hec_token> --log-opt splunk-insecureskipverify=true -d <docker_image>
102+
103+
## Step 4: Set Configuration variables
104+
105+
Use the configuration variables to configure the behaviors and rules for Splunk Connect for Docker. For example you can confiugre your certificate security or how messages are formatted and distributed. Note the following:
106+
107+
* Configurations that pass though docker <addr>run --log-opt <opt><addr> are effective instantaneously.
108+
* You must restart the Docker engine after configuring through `<addr>`daemon.json`<addr>`
109+
110+
### How to use the variables
111+
112+
The following is an example of the logging options specified for the Splunk Enterprise instance. In this example:
113+
114+
The path to the root certificate and Common Name is specified using an HTTPS scheme to be used for verification.
115+
116+
$ docker run --log-driver=splunk-logging-plugin\
117+
--log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 \
118+
--log-opt splunk-url=https://splunkhost:8088 \
119+
--log-opt splunk-capath=/path/to/cert/cacert.pem \
120+
--log-opt splunk-caname=SplunkServerDefaultCert \
121+
--log-opt tag="{{.Name}}/{{.FullID}}" \
122+
--log-opt labels=location \
123+
--log-opt env=TEST \
124+
--env "TEST=false" \
125+
--label location=west \
126+
<docker_image>
127+
128+
### Required Variables
129+
130+
Variable | Description
131+
------------ | -------------
132+
splunk-token | Splunk HTTP Event Collector token.
133+
splunk-url | Path to your Splunk Enterprise, self-service Splunk Cloud instance, or Splunk Cloud managed cluster (including port and scheme used by HTTP Event Collector) in one of the following formats: https://your_splunk_instance:8088 or https://input-prd-p-XXXXXXX.cloud.splunk.com:8088 or https://http-inputs-XXXXXXXX.splunkcloud.com
134+
135+
136+
### Optional Variables
137+
138+
Variable | Description | Default
139+
------------ | ------------- | -------------
140+
splunk-source | Event source |
141+
splunk-sourcetype | Event source type |
142+
splunk-index | Event index. (Note that HEC token must be configured to accept the specified index) |
143+
splunk-capath | Path to root certificate. (Must be specified if splunk-insecureskipverify is false) |
144+
splunk-caname | Name to use for validating server certificate; by default the hostname of the splunk-url is used. |
145+
splunk-insecureskipverify| "false" means that the service certificates are validated and "true" means that server certificates are not validated. | false
146+
splunk-format | Message format. Values can be inline, json, or raw. For more infomation about formats see the Messageformats option. | inline
147+
splunk-verify-connection| Upon plug-in startup, verify that Splunk Connect for Docker can connect to Splunk HEC endpoint. False indicates that Splunk Connect for Docker will start up and continue to try to connect to HEC and will push logs to buffer until connection has been establised. Logs will roll off buffer once buffer is full. True indicates that Splunk Connect for Docker will not start up if connection to HEC cannot be established. | false
148+
splunk-gzip | Enable/disable gzip compression to send events to Splunk Enterprise or Splunk Cloud instance. | false
149+
splunk-gzip-level | Set compression level for gzip. Valid values are -1 (default), 0 (no compression), 1 (best speed) … 9 (best compression). | DefaultCompression
150+
tag | Specify tag for message, which interpret some markup. Default value is {{.ID}} (12 characters of the container ID). Refer to the log tag option documentation for customizing the log tag format. https://docs.docker.com/v17.09/engine/admin/logging/log_tags/ |
151+
labels | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container. |
152+
env | Comma-separated list of keys of environment variables to be included in message if they specified for a container. |
153+
env-regex | A regular expression to match logging-related environment variables. Used for advanced log tag options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the attributes of a logging message. |
154+
155+
156+
### Advanced options - Environment Variables
157+
158+
Variable | Description | Default
159+
------------ | ------------- | -------------
160+
SPLUNK_LOGGING_DRIVER_POST_MESSAGES_FREQUENCY | How often plug-in posts messages when there is nothing to batch, i.e., the maximum time to wait for more messages to batch. The internal buffer used for batching is flushed either when the buffer is full (the disgnated batch size is reached) or the buffer timesout (specified by this frequency) | | 5s
161+
SPLUNK_LOGGING_DRIVER_POST_MESSAGES_BATCH_SIZE | The number of messages the plug-in should collect before sending them in one batch. | 1000
162+
SPLUNK_LOGGING_DRIVER_BUFFER_MAX | The maximum amount of messages to hold in buffer and retry when the plug-in cannot connect to remote server. | 10 * 1000
163+
SPLUNK_LOGGING_DRIVER_CHANNEL_SIZE | How many pending messages can be in the channel used to send messages to background logger worker, which batches them. | 4 * 1000
164+
SPLUNK_LOGGING_DRIVER_TEMP_MESSAGES_HOLD_DURATION | Appends logs that are chunked by docker with 16kb limit. It specifies how long the system can wait for the next message to come. | 100ms
165+
SPLUNK_LOGGING_DRIVER_TEMP_MESSAGES_BUFFER_SIZE | Appends logs that are chunked by docker with 16kb limit. It specifies the biggest message that the system can reassemble. The value provided here should be smaller than or equal to the Splunk HEC limit. 1 MB is the default HEC setting. | 1mb
166+
167+
168+
### Message formats
169+
There are three logging plug-in messaging formats set under the optional variable splunk-format:
170+
171+
* inline (this is the default format)
172+
* json
173+
* raw
174+
175+
The default format is inline, where each log message is embedded as a string and is assigned to "line" field. For example:
176+
177+
{
178+
"attrs": {
179+
"env1": "val1",
180+
"label1": "label1"
181+
},
182+
"tag": "MyImage/MyContainer",
183+
"source": "stdout",
184+
"line": "my message"
185+
}
186+
{
187+
"attrs": {
188+
"env1": "val1",
189+
"label1": "label1"
190+
},
191+
"tag": "MyImage/MyContainer",
192+
"source": "stdout",
193+
"line": "{\"foo\": \"bar\"}" //though this is a string that can be marshaled to json, it is still treated as a string
194+
}
195+
196+
When messages are JSON objects, you may want to embed them in the message sent to Splunk.
197+
198+
To format messages as json objects, set --log-opt splunk-format=json. The plug-in will try to parse every line as a JSON object and embed the json object to "line" field. If it cannot parse the message, it is sent inline. For example:
199+
200+
{
201+
"attrs": {
202+
"env1": "val1",
203+
"label1": "label1"
204+
},
205+
"tag": "MyImage/MyContainer",
206+
"source": "stdout",
207+
"line": "my message" //fall back to a string
208+
}
209+
{
210+
"attrs": {
211+
"env1": "val1",
212+
"label1": "label1"
213+
},
214+
"tag": "MyImage/MyContainer",
215+
"source": "stdout",
216+
"line": {
217+
"foo": "bar"
218+
}
219+
}
220+
221+
If --log-opt splunk-format=raw, each message together with attributes (environment variables and labels) and tags are combined in a raw string. Attributes and tags are prefixed to the message. For example:
222+
223+
#<tag> <env=value> <label=value> <logging_messaage>
224+
MyImage/MyContainer env1=val1 label1=label1 my message
225+
MyImage/MyContainer env1=val1 label1=label1 {"foo": "bar"}
226+
227+
# Troubleshooting
228+
229+
If your Splunk Connector for Docker does not behave as expected, use the debug functionality and then refer to the following tips included in output.
230+
231+
## Enable Debug Mode to find log errors
232+
233+
Plugin logs can be found as docker daemon log. To enable debug mode, export environment variable LOGGIN_LEVEL=DEBUG in docker engine environment. See the Docker documentation for information about how to enable debug mode in your docker environment: https://docs.docker.com/config/daemon/
234+
235+
## Use the debugger to check your debug the Splunk HEC connection
236+
237+
Check HEC endpoint accessibility Docker environment. If the endpoint cannot be reached, debug logs are not sent to Splunk, or the logs or will buffer and drop as they roll off the buffer.
238+
239+
Test HEC endpoint is accessible
240+
$ curl -k https://<ip_address>:8088/services/collector/health
241+
{"text":"HEC is healthy","code":200}
242+
243+
## Check your HEC configuration for clusters
244+
245+
If you are using an Indexer Cluster, the current plugin accepts a single splunk-url value. We recommend that you configure a load balancer in front of your Indexer tier. Make sure the load balancer can successfully tunnel the HEC requests to the indexer tier. If HEC is configured in an Indexer Cluster environment, all indexers should have same HEC token configured. See http://docs.splunk.com/Documentation/Splunk/7.0.3/Data/UsetheHTTPEventCollector.
246+
247+
## Check your heavy forwarder connection
248+
249+
If you ae using a heavy forwarder to preprocess the events (e.g: funnel multiple log lines to a single event), make sure that the heavy forwarder is properly connecting to the indexers. To troubleshoot the forwarder and receiver connection, see: https://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Forwarding/Receiverconnection.
250+
251+
## Check the plugin's debug log in docker
252+
253+
Stdout of a plugin is redirected to Docker logs. Such entries have a plugin=<ID> suffix.
254+
255+
To find out the plugin ID of Splunk Connect for Docker, use the command below and look for Splunk Logging Plugin entry.
256+
257+
# list all the plugins
258+
$ docker plugin ls
259+
260+
Depending on your system, location of Docker daemon logging may vary. Refer to Docker documentation for Docker daemon log location for your specific platform. Here are a few examples:
261+
262+
* Ubuntu (old using upstart ) - /var/logging/upstart/docker.logging
263+
* Ubuntu (new using systemd ) - sudo journalctl -fu docker.service
264+
* Boot2Docker - /var/logging/docker.logging
265+
* Debian GNU/Linux - /var/logging/daemon.logging
266+
* CentOS - /var/logging/daemon.logging | grep docker
267+
* CoreOS - journalctl -u docker.service
268+
* Fedora - journalctl -u docker.service
269+
* Red Hat Enterprise Linux Server - /var/logging/messages | grep docker
270+
* OpenSuSE - journalctl -u docker.service
271+
* OSX - ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/logging/d‌ocker.logging
272+
* Windows - Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time, as mentioned here.
273+

0 commit comments

Comments
 (0)