Skip to content

Commit 9be27f7

Browse files
Add custom docker build option to logstash documentation (#10879)
Signed-off-by: Fanit Kolchina <[email protected]> (cherry picked from commit 57b1251) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
1 parent 4239a8c commit 9be27f7

File tree

1 file changed

+56
-11
lines changed

1 file changed

+56
-11
lines changed

_tools/logstash/index.md

Lines changed: 56 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@ Make sure you have [Java Development Kit (JDK)](https://www.oracle.com/java/tech
6767
```bash
6868
tar -zxvf logstash-8.8.2-linux-x86_64.tar.gz
6969
```
70+
{% include copy.html %}
7071

7172
3. Navigate to the `logstash-8.8.2` directory.
7273

@@ -75,6 +76,7 @@ Make sure you have [Java Development Kit (JDK)](https://www.oracle.com/java/tech
7576
```bash
7677
bin/logstash-plugin install logstash-output-opensearch
7778
```
79+
{% include copy.html %}
7880

7981
You should receive the following output:
8082

@@ -91,23 +93,48 @@ You can add your pipeline configurations to the `config` directory. Logstash sav
9193

9294
### Docker
9395

96+
You can either use a custom Dockerfile to build a Logstash image or use the standard Logstash image.
97+
98+
#### Option 1: Using a custom Dockerfile (recommended)
99+
100+
1. Create a custom Dockerfile to build a Logstash image with the required OpenSearch plugins:
101+
102+
```
103+
FROM logstash:<LATEST_VERSION>
104+
RUN bin/logstash-plugin install logstash-output-opensearch
105+
RUN bin/logstash-plugin install logstash-input-opensearch
106+
```
107+
{% include copy.html %}
108+
109+
1. Build the image:
110+
111+
```
112+
docker build -t logstash-with-opensearch-plugins .
113+
```
114+
{% include copy.html %}
115+
116+
#### Option 2: Using the standard Logstash image
117+
94118
1. Pull the latest Logstash image as stated in the [Logstash downloads](https://www.elastic.co/downloads/logstash).
95119

96120
```
97121
docker pull docker.elastic.co/logstash/logstash:8.8.2
98122
```
123+
{% include copy.html %}
99124

100125
1. Create a Docker network:
101126

102127
```
103128
docker network create test
104129
```
130+
{% include copy.html %}
105131

106132
1. Start OpenSearch with this network:
107133

108134
```
109135
docker run -p 9200:9200 -p 9600:9600 --name opensearch --net test -e "discovery.type=single-node" opensearchproject/opensearch:1.2.0
110136
```
137+
{% include copy.html %}
111138

112139
1. Start Logstash:
113140

@@ -123,6 +150,7 @@ You can add your pipeline configurations to the `config` directory. Logstash sav
123150
}
124151
}'
125152
```
153+
{% include copy.html %}
126154

127155
## Process text from the terminal
128156

@@ -135,6 +163,8 @@ To enter some text in the terminal and see the event data in the output:
135163
```bash
136164
bin/logstash -e "input { stdin { } } output { stdout { } }"
137165
```
166+
{% include copy.html %}
167+
138168
Add the `—debug` flag to see a more detailed output.
139169

140170
2. Enter "hello world" in your terminal. Logstash processes the text and outputs it back to the terminal:
@@ -162,6 +192,7 @@ If you already have a Logstash process running, you’ll get an error. To fix th
162192
cd data
163193
rm -rf .lock
164194
```
195+
{% include copy.html %}
165196

166197
2. Restart Logstash.
167198

@@ -171,7 +202,7 @@ To define a pipeline that handles JSON requests:
171202

172203
1. Open the `config/pipeline.conf` file in any text editor you like. You can create a pipeline configuration file with any extension, the `.conf` extension is a Logstash convention. Add the `json` codec to accept JSON as the input and the `file` plugin to output the processed events to a `.txt` file:
173204

174-
```yml
205+
```json
175206
input {
176207
stdin {
177208
codec => json
@@ -183,22 +214,25 @@ To define a pipeline that handles JSON requests:
183214
}
184215
}
185216
```
217+
{% include copy.html %}
186218

187219
To process inputs from a file, add an input file to the `events-data` directory and then pass its path to the `file` plugin at the input:
188220

189-
```yml
221+
```json
190222
input {
191223
file {
192224
path => "events-data/input_data.log"
193225
}
194226
}
195227
```
228+
{% include copy.html %}
196229

197230
2. Start Logstash:
198231

199232
```bash
200-
$ bin/logstash -f config/pipeline.conf
233+
bin/logstash -f config/pipeline.conf
201234
```
235+
{% include copy.html %}
202236

203237
`config/pipeline.conf` is a relative path to the `pipeline.conf` file. You can use an absolute path as well.
204238

@@ -207,14 +241,18 @@ To define a pipeline that handles JSON requests:
207241
```json
208242
{ "amount": 10, "quantity": 2}
209243
```
244+
{% include copy.html %}
210245
211246
The pipeline only handles a single line of input. If you paste some JSON that spans multiple lines, you’ll get an error.
212247
213248
4. Check that the fields from the JSON object are added to the `output.txt` file:
214249
215-
```json
216-
$ cat output.txt
250+
```bash
251+
cat output.txt
252+
```
253+
{% include copy.html %}
217254
255+
```json
218256
{
219257
"@version": "1",
220258
"@timestamp": "2021-05-30T05:52:52.421Z",
@@ -230,7 +268,7 @@ To define a pipeline that handles HTTP requests:
230268
231269
1. Use the `http` plugin to send events to Logstash through HTTP:
232270
233-
```yml
271+
```json
234272
input {
235273
http {
236274
host => "127.0.0.1"
@@ -244,31 +282,34 @@ To define a pipeline that handles HTTP requests:
244282
}
245283
}
246284
```
285+
{% include copy.html %}
247286
248287
If you don’t specify any options, the `http` plugin binds to `localhost` and listens on port 8080.
249288
250289
2. Start Logstash:
251290
252291
```bash
253-
$ bin/logstash -f config/pipeline.conf
292+
bin/logstash -f config/pipeline.conf
254293
```
294+
{% include copy.html %}
255295
256296
3. Use Postman to send an HTTP request. Set `Content-Type` to an HTTP header with a value of `application/json`:
257297
258298
```json
259299
PUT 127.0.0.1:8080
260-
261300
{
262301
"amount": 10,
263302
"quantity": 2
264303
}
265304
```
305+
{% include copy.html %}
266306
267307
Or, you can use the `curl` command:
268308
269309
```bash
270310
curl -XPUT -H "Content-Type: application/json" -d ' {"amount": 7, "quantity": 3 }' http://localhost:8080 (http://localhost:8080/)
271311
```
312+
{% include copy.html %}
272313
273314
Even though we haven't added the `json` plugin to the input, the pipeline configuration still works because the HTTP plugin automatically applies the appropriate codec based on the `Content-Type` header.
274315
If you specify a value of `applications/json`, Logstash parses the request body as JSON.
@@ -309,14 +350,15 @@ The `stdin` plugin doesn’t supporting automatic reloading.
309350
310351
1. Add an option named `start_position` with a value of `beginning` to the input plugin:
311352
312-
```yml
353+
```json
313354
input {
314355
file {
315356
path => "/Users/<user>/Desktop/logstash7-12.1/events-data/input_file.log"
316357
start_position => "beginning"
317358
}
318359
}
319360
```
361+
{% include copy.html %}
320362
321363
Logstash only processes any new events added to the input file and ignores the ones that it has already processed to avoid processing the same event more than once on restart.
322364
@@ -334,26 +376,29 @@ The `stdin` plugin doesn’t supporting automatic reloading.
334376
335377
51575938 1 4 7727
336378
```
379+
{% include copy.html %}
337380
338381
The last number in the `sinceDB` file (7727) is the byte offset of the last known event processed.
339382
340383
5. To process the input file from the beginning, delete the `sinceDB` file:
341384
342-
```yml
385+
```bash
343386
rm .sincedb_*
344387
```
388+
{% include copy.html %}
345389
346390
2. Start Logstash with a `—-config.reload.automatic` argument:
347391
348392
```bash
349393
bin/logstash -f config/pipeline.conf --config.reload.automatic
350394
```
395+
{% include copy.html %}
351396
352397
The `reload` option only reloads if you add a new line at the end of the pipeline configuration file.
353398
354399
Sample output:
355400
356-
```yml
401+
```json
357402
{
358403
"message" => "216.243.171.38 - - [20/Sep/2017:19:11:52 +0200] \"GET /products/view/123 HTTP/1.1\" 200 12798 \"https://codingexplained.com/products\" \"Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)\"",
359404
"@version" => "1",

0 commit comments

Comments
 (0)