diff --git a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md index b461b08f..e4b80546 100644 --- a/crowdsec-docs/docs/appsec/alerts_and_scenarios.md +++ b/crowdsec-docs/docs/appsec/alerts_and_scenarios.md @@ -115,7 +115,7 @@ We can now create a scenario that will trigger when a single IPs triggers this r type: leaky format: 3.0 name: crowdsecurity/foobar-enum -description: "Ban IPs repeateadly triggering out of band rules" +description: "Ban IPs repeatedly triggering out of band rules" filter: "evt.Meta.log_type == 'appsec-info' && evt.Meta.rule_name == 'crowdsecurity/foobar-access'" distinct: evt.Meta.target_uri leakspeed: "60s" diff --git a/crowdsec-docs/docs/appsec/benchmark.md b/crowdsec-docs/docs/appsec/benchmark.md index a6181447..c4e771f8 100644 --- a/crowdsec-docs/docs/appsec/benchmark.md +++ b/crowdsec-docs/docs/appsec/benchmark.md @@ -15,7 +15,7 @@ sidebar_position: 80 --> -The Application Security Component benchmarks have been run on a AWS EC2 Instance `t2.medium` (2vCPU/4Go RAM). +The Application Security Component benchmarks have been run on a AWS EC2 Instance `t2.medium` (2vCPU/4GiB RAM). All the benchmarks have been run with only one `routine` configured for the Application Security Component. diff --git a/crowdsec-docs/docs/appsec/configuration.md b/crowdsec-docs/docs/appsec/configuration.md index e612a643..e873f003 100644 --- a/crowdsec-docs/docs/appsec/configuration.md +++ b/crowdsec-docs/docs/appsec/configuration.md @@ -6,7 +6,7 @@ sidebar_position: 6 ## Overview -This page explains the interraction between various files involved in AppSec configuration and the details about the processing pipeline AppSec request processing. +This page explains the interaction between various files involved in AppSec configuration and the details about the processing pipeline AppSec request processing. **Prerequisites**: - Familiarity with [AppSec concepts](/appsec/intro.md) @@ -24,7 +24,7 @@ The goals of the acquisition file are: - To specify the **address** and **port** where the AppSec-enabled Remediation Component(s) will forward the requests to. - And specify one or more [AppSec configuration files](#appsec-configuration) to use as definition of what rules to apply and how. -Details can be found in the [AppSec Datasource page](/log_processor/data_sources/apps). +Details can be found in the [AppSec Datasource page](/log_processor/data_sources/appsec.md). ### Defining Multiple AppSec Configurations diff --git a/crowdsec-docs/docs/appsec/quickstart/traefik.mdx b/crowdsec-docs/docs/appsec/quickstart/traefik.mdx index 898dfc50..926707ca 100644 --- a/crowdsec-docs/docs/appsec/quickstart/traefik.mdx +++ b/crowdsec-docs/docs/appsec/quickstart/traefik.mdx @@ -25,7 +25,7 @@ Additionally, we'll show how to monitor these alerts through the [console](https - Traefik Plugin **[Remediation Component](/u/bouncers/intro)**: Thanks to [maxlerebourg](https://github.com/maxlerebourg) and team they created a [Traefik Plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) that allows you to block requests directly from Traefik. :::info -Prior to starting the guide ensure you are using the [Traefik Plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) and **NOT** the older [traefik-crowdsec-bouncer](https://app.crowdsec.net/hub/author/fbonalair/remediation-components/traefik-crowdsec-bouncer) as it hasnt recieved updates to use the new AppSec Component. +Prior to starting the guide ensure you are using the [Traefik Plugin](https://plugins.traefik.io/plugins/6335346ca4caa9ddeffda116/crowdsec-bouncer-traefik-plugin) and **NOT** the older [traefik-crowdsec-bouncer](https://app.crowdsec.net/hub/author/fbonalair/remediation-components/traefik-crowdsec-bouncer) as it hasnt received updates to use the new AppSec Component. ::: :::warning @@ -77,7 +77,7 @@ If you have a folder in which you are persisting the configuration files, you ca There steps will change depending on how you are running the Security Engine. If you are running via `docker run` then you should launch the container within the same directory as the `appsec.yaml` file. If you are using `docker-compose` you can use a relative file mount to mount the `appsec.yaml` file. Steps: - 1. Change to the location where you exectued the `docker run` or `docker compose` command. + 1. Change to the location where you executed the `docker run` or `docker compose` command. 2. Create a `appsec.yaml` file at the base of the directory. 3. Add the following content to the `appsec.yaml` file. @@ -96,11 +96,11 @@ Since CrowdSec is running inside a container you must set the `listen_addr` to ` diff --git a/crowdsec-docs/docs/getting_started/crowdsec_tour.mdx b/crowdsec-docs/docs/getting_started/crowdsec_tour.mdx index 6230d19c..1d0c90d1 100644 --- a/crowdsec-docs/docs/getting_started/crowdsec_tour.mdx +++ b/crowdsec-docs/docs/getting_started/crowdsec_tour.mdx @@ -250,7 +250,7 @@ Those metrics are a great way to know if your configuration is correct: The `Acquisition Metrics` is a great way to know if your parsers are setup correctly: - If you have 0 **LINES PARSED** for a source : You are probably *missing* a parser, or you have a custom log format that prevents the parser from understanding your logs. - - However, it's perfectly OK to have a lot of **LINES UNPARSED** : Crowdsec is not a SIEM, and only parses the logs that are relevant to its scenarios. For example, [ssh parser](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs), only cares about failed authentication events (at the time of writting). + - However, it's perfectly OK to have a lot of **LINES UNPARSED** : Crowdsec is not a SIEM, and only parses the logs that are relevant to its scenarios. For example, [ssh parser](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs), only cares about failed authentication events (at the time of writing). - **LINES POURED TO BUCKET** tell you that your scenarios are matching your log sources : it means that some events from this log source made all their way to an actual scenario diff --git a/crowdsec-docs/docs/log_processor/data_sources/introduction.md b/crowdsec-docs/docs/log_processor/data_sources/introduction.md index 8cb0281d..25aff98c 100644 --- a/crowdsec-docs/docs/log_processor/data_sources/introduction.md +++ b/crowdsec-docs/docs/log_processor/data_sources/introduction.md @@ -1,20 +1,25 @@ --- id: intro -title: Acquisition Datasources Introduction +title: Acquisition Datasources sidebar_position: 1 --- -## Datasources +To monitor applications, the Security Engine needs to read logs. +DataSources define where to access them (either as files, or over the network from a centralized logging service). -To be able to monitor applications, the Security Engine needs to access logs. -DataSources are configured via the [acquisition](/configuration/crowdsec_configuration.md#acquisition_path) configuration, or specified via the command-line when performing cold logs analysis. +They can be defined: +- in [Acquisition files](/configuration/crowdsec_configuration.md#acquisition_path). Each file can contain multiple DataSource definitions. This configuration can be generated automatically, please refer to the [Service Discovery documentation](/log_processor/service-discovery-setup/intro.md) +- for cold log analysis, you can also specify acquisitions via the command line. + + +## Datasources modules Name | Type | Stream | One-shot -----|------|--------|---------- [Appsec](/log_processor/data_sources/appsec.md) | expose HTTP service for the Appsec component | yes | no [AWS cloudwatch](/log_processor/data_sources/cloudwatch.md) | single stream or log group | yes | yes -[AWS kinesis](/log_processor/data_sources/kinesis.md)| read logs from a kinesis strean | yes | no +[AWS kinesis](/log_processor/data_sources/kinesis.md)| read logs from a kinesis stream | yes | no [AWS S3](/log_processor/data_sources/s3.md)| read logs from a S3 bucket | yes | yes [docker](/log_processor/data_sources/docker.md) | read logs from docker containers | yes | yes [file](/log_processor/data_sources/file.md) | single files, glob expressions and .gz files | yes | yes @@ -46,6 +51,7 @@ An expression that will run after the acquisition has read one line, and before It allows to modify an event (or generate multiple events from one line) before parsing. For example, if you acquire logs from a file containing a JSON object on each line, and each object has a `Records` array with multiple events, you can use the following to generate one event per entry in the array: + ``` map(JsonExtractSlice(evt.Line.Raw, "Records"), ToJsonString(#)) ``` @@ -62,7 +68,7 @@ By default, when reading logs in real-time, crowdsec will use the time at which Setting this option to `true` will force crowdsec to use the timestamp from the log as the time of the event. -It is mandatory to set this if your application buffers logs before writting them (for example, IIS when writing to a log file, or logs written to S3 from almost any AWS service).
+It is mandatory to set this if your application buffers logs before writing them (for example, IIS when writing to a log file, or logs written to S3 from almost any AWS service).
If not set, then crowdsec will think all logs happened at once, which can lead to some false positive detections. ### `labels` @@ -70,31 +76,39 @@ If not set, then crowdsec will think all logs happened at once, which can lead t A map of labels to add to the event. The `type` label is mandatory, and used by the Security Engine to choose which parser to use. -## Acquisition configuration example +## Acquisition configuration examples -```yaml title="/etc/crowdsec/acquis.yaml" +```yaml title="/etc/crowdsec/acquis.d/nginx.yaml" filenames: - /var/log/nginx/*.log labels: type: nginx ---- +``` + +```yaml title="/etc/crowdsec/acquis.d/linux.yaml" filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog ---- +``` + +```yaml title="/etc/crowdsec/acquis.d/docker.yaml" source: docker container_name_regexp: - .*caddy* labels: type: caddy --- -... +source: docker +container_name_regexp: + - .*nginx* +labels: + type: nginx ``` :::warning The `labels` and `type` fields are necessary to dispatch the log lines to the right parser. -Also note between each datasource is `---` this is needed to separate multiple YAML documents (each datasource) in a single file. +In the last example we defined multiple datasources separated by the line `---`, which is the standard YAML marker. ::: diff --git a/crowdsec-docs/docs/log_processor/data_sources/syslog_service.md b/crowdsec-docs/docs/log_processor/data_sources/syslog_service.md index 691da461..c595bb98 100644 --- a/crowdsec-docs/docs/log_processor/data_sources/syslog_service.md +++ b/crowdsec-docs/docs/log_processor/data_sources/syslog_service.md @@ -51,6 +51,6 @@ This module does not support command-line acquisition. :::warning This syslog datasource is currently intended for small setups, and is at risk of losing messages over a few hundreds events/second. -To process significant amounts of logs, rely on dedicated syslog server such as [rsyslog](https://www.rsyslog.com/), with this server writting logs to files that Security Engine will read from. +To process significant amounts of logs, rely on dedicated syslog server such as [rsyslog](https://www.rsyslog.com/), with this server writing logs to files that Security Engine will read from. This page will be updated with further improvements of this data source. -::: \ No newline at end of file +::: diff --git a/crowdsec-docs/docs/log_processor/data_sources/troubleshoot.md b/crowdsec-docs/docs/log_processor/data_sources/troubleshoot.md index 8eeaef79..c5f9121d 100644 --- a/crowdsec-docs/docs/log_processor/data_sources/troubleshoot.md +++ b/crowdsec-docs/docs/log_processor/data_sources/troubleshoot.md @@ -5,7 +5,7 @@ sidebar_position: 10 --- The [prometheus](/observability/prometheus.md) instrumentation exposes metrics about acquisition and data sources. -Those can as well be view via `cscli metrics` : +Those can as well be viewed via `cscli metrics` : ```bash INFO[19-08-2021 06:33:31 PM] Acquisition Metrics: diff --git a/crowdsec-docs/docs/log_processor/intro.mdx b/crowdsec-docs/docs/log_processor/intro.mdx index 32df210a..d0b98247 100644 --- a/crowdsec-docs/docs/log_processor/intro.mdx +++ b/crowdsec-docs/docs/log_processor/intro.mdx @@ -4,24 +4,23 @@ title: Introduction sidebar_position: 1 --- -The Log Processor is one of the core component of the Security Engine to: +The Log Processor is a core component of the Security Engine. It: -- Read logs from [Data Sources](log_processor/data_sources/introduction.md) in the form of Acquistions. -- Parse the logs and extract relevant information using [Parsers](log_processor/parsers/introduction.mdx). -- Enrich the parsed information with additional context such as GEOIP, ASN using [Enrichers](log_processor/parsers/enricher.md). -- Monitor the logs for patterns of interest known as [Scenarios](log_processor/scenarios/introduction.mdx). -- Push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. - -!TODO: Add diagram of the log processor pipeline +- Reads logs from [Data Sources](log_processor/data_sources/introduction.md) via Acquistions. +- Parses logs and extract relevant information using [Parsers](log_processor/parsers/introduction.mdx). +- Enriches the parsed information with additional context such as GEOIP, ASN using [Enrichers](log_processor/parsers/enricher.md). +- Monitors patterns of interest via [Scenarios](log_processor/scenarios/introduction.mdx). +- Pushes alerts to the Local API (LAPI), where alert/decisions are stored. - Read logs from datasources - Parse the logs - Enrich the parsed information - Monitor the logs for patterns of interest + -## Introduction +## Log Processor -The Log Processor is an internal core component of the Security Engine in charge of reading logs from Data Sources, parsing them, enriching them, and monitoring them for patterns of interest. +The Log Processor reads logs from Data Sources, parses and enriches them, and monitors them for patterns of interest. Once a pattern of interest is detected, the Log Processor will push alerts to the Local API (LAPI) for alert/decisions to be stored within the database. @@ -35,19 +34,19 @@ Data Sources are individual modules that can be loaded at runtime by the Log Pro Acquisitions are the configuration files that define how the Log Processor should read logs from a Data Source. Acquisitions are defined in YAML format and are loaded by the Log Processor at runtime. -We have two ways to define Acquisitions within the [configuration directory](/u/troubleshooting/security_engine#where-is-configuration-stored) : +We support two ways to define Acquisitions in the [configuration directory](/u/troubleshooting/security_engine#where-is-configuration-stored): -- `acquis.yaml` file: This used to be only place to define Acquisitions prior to `1.5.0`. This file is still supported for backward compatibility. -- `acquis.d` folder: This is a directory where you can define multiple Acquisitions in separate files. This is useful when you want to auto generate files using an external application such as ansible. +- `acquis.yaml` file: the legacy, single-file configuration (still supported) +- `acquis.d` directory: a directory of multiple acquisition files (since v1.5.0, recommended for any non-trivial setup) ```yaml title="Example Acquisition Configuration" ## /etc/crowdsec/acquis.d/file.yaml source: file ## The Data Source module to use filenames: - - /tmp/foo/*.log - - /var/log/syslog + - /tmp/foo/*.log + - /var/log/syslog labels: - type: syslog + type: syslog ``` For more information on Data Sources and Acquisitions, see the [Data Sources](log_processor/data_sources/introduction.md) documentation. @@ -87,3 +86,9 @@ You can see more information on Whitelists in the [documentation](log_processor/ Alert Context is additional context that can sent with an alert to the LAPI. This context can be shown locally via `cscli` or within the [CrowdSec Console](https://app.crowdsec.net/signup) if you opt in to share context when you enroll your instance. You can read more about Alert Context in the [documentation](log_processor/alert_context/intro.md). + +### Service Discovery & Setup + +On installation, CrowdSec can automatically detect existing services, download the relevant Hub collections, and generate acquisitions based on discovered log files. + +You can [customize or override these steps](log_processor/service-discovery-setup/intro.md), for example when provisioning multiple systems or using configuration management tools. diff --git a/crowdsec-docs/docs/log_processor/service-discovery-setup/detect-yaml.md b/crowdsec-docs/docs/log_processor/service-discovery-setup/detect-yaml.md new file mode 100644 index 00000000..69d1e59c --- /dev/null +++ b/crowdsec-docs/docs/log_processor/service-discovery-setup/detect-yaml.md @@ -0,0 +1,139 @@ +--- +id: detect-yaml +title: Syntax +sidebar_position: 1 +--- + +# Syntax + +A minimal detection file is a YAML map with a top‐level `detect:` key. + +Under it, each entry describes one service plan: + +```yaml +# detect.yaml +--- +detect: + apache2-file-apache2: + when: + - Systemd.UnitInstalled("apache2.service") or len(Path.Glob("/var/log/apache2/*.log")) > 0 + hub_spec: + collections: + - crowdsecurity/apache2 + acquisition_spec: + filename: apache2.yaml + datasource: + source: file + filenames: + - /var/log/apache2/*.log + labels: + type: apache2 +``` + +## Fields + +### `when` + +A list of expression that must return a boolean. + +If multiple expressions are provided, they must all return `true` for the service to be included. + +```yaml +when: + - Host.OS == "linux" + - Systemd.UnitInstalled("") +``` + +You can use any of the helper referenced [here](/log_processor/service-discovery-setup/expr.md). + +### `hub_spec` + +A map of hub items to install. + +Specifying an invalid item type or item will log an error but will not prevent the detection to continue. + +```yaml +hub_spec: + collections: + - crowdsecurity/linux + parsers: + - crowdsecurity/nginx-logs + scenarios: + - crowdsecurity/http-bf +``` + +### `acquisition_spec` + +This item defines the acquisition that will be written to disk + +```yaml +acquisition_spec: + filename: foobar.yaml + datasource: + source: docker + container_name: foo + labels: + type: bar +``` + +The `filename` attribute will be used to generate the name of file in the form of `acquis.d/setup..yaml`. + +The content of `datasource` will be validated (syntax, required fields depending on the datasource configured) and be written as-is to the file. + +## Examples + +Basic OS / Hub only: + +```yaml +detect: + linux: + when: + - Host.OS == "linux" + hub_spec: + collections: + - crowdsecurity/linux +``` + +`journalctl` source with a filter: + +```yaml +detect: + caddy-journal: + when: + - Systemd.UnitInstalled("caddy.service") + - len(Path.Glob("/var/log/caddy/*.log")) == 0 + hub_spec: + collections: + - crowdsecurity/caddy + acquisition_spec: + filename: caddy.yaml + datasource: + source: journalctl + labels: + type: caddy + journalctl_filter: + - "_SYSTEMD_UNIT=caddy.service" +``` + +Windows event log: + +```yaml +detect: + windows_auth: + when: + - Host.OS == "windows" + hub_spec: + collections: + - crowdsecurity/windows + acquisition_spec: + filename: windows_auth.yaml + datasource: + source: wineventlog + event_channel: Security + event_ids: + - 4625 + - 4623 + event_level: information + labels: + type: eventlog +``` diff --git a/crowdsec-docs/docs/log_processor/service-discovery-setup/expr.md b/crowdsec-docs/docs/log_processor/service-discovery-setup/expr.md new file mode 100644 index 00000000..80bbd4d9 --- /dev/null +++ b/crowdsec-docs/docs/log_processor/service-discovery-setup/expr.md @@ -0,0 +1,157 @@ +--- +id: setup-expr-helpers +title: Expr Helpers +sidebar_position: 1 +--- + +# Expression Helpers Reference + +Various helpers are available for use in the `detect.yaml` file to determine how crowdsec should be configured. + +## Host + + This object gives access to various information about the current state of the operating system + +### `Host.Hostname` + +    Returns the hostname of the machine + +> `Host.Hostname == "mymachine"` + +### `Host.Uptime` + +    Returns the uptime of the machine in seconds. + +### `Host.Boottime` + +    Returns the unix timestamp of the time the machine booted. + +### `Host.Procs` + +    Returns the number of processes on the machine. + +### `Host.OS` + +    Returns the name of the OS (`linux`, `freebsd`, `windows`, ...) + +> `Host.OS == "linux"` + +### `Host.Platform` + +    Returns the variant of the OS (`ubuntu`, `linuxmint`, ....) + +> `Host.Platform == "ubuntu"` + +### `Host.PlatformFamily` + +    Returns the family of the OS (`debian`, `rhel`, ...) + +> `Host.PlatformFamily == "debian"` + +### `Host.PlatformVersion` + +    Returns the version of the OS or distribution (for linux, /etc/os-release) + +> `Host.PlatformVersion == "25.04" + +### `Host.KernelVersion` + +    Returns the current kernel version as returned by `uname -r` + +> `Host.KernelVersion == "6.16.2" + +### `Host.KernelArch` + +    Returns the native architecture of the system (`x86_64`, ...) + +> `Host.KernelArch == "x86_64"` + +### `Host.VirtualizationSystem` + +    Returns the name of the virtualization system in use if any. + +> `Host.VirtualizationSystem == "kvm"` + +### `Host.VirtualizationRole` + +    Returns the virtualization role of the system if any (`guest`, `host`) + +> `Host.VirtualizationRole == "host"` + +### `Host.HostID` + +    Returns a unique ID specific to the system. + +## Path + +This object exposes helpers functions for the filesystem + +### `Exists(path) bool` + +    Returns `true` if the specified path exists. + +> `Path.Exists("/var/log/nginx/access.log") == true` + +### `Glob(pattern) []string` + +    Returns a list of files matching the provided pattern. + +    Returns an empty list if the glob pattern is invalid + +> `len(Path.Glob("/var/log/nginx/*.log")) > 0` + +## System + +### `ProcessRunning(name) bool` + +    Returns `true` if there's any process with the specified name running + +> `System.ProcessRunning("nginx") == true` + +## Systemd + +    This object exposes helpers to get informations about Systemd units. + +    Only available on Linux. + +### `UnitInstalled(unitName) bool` + +    Returns `true` if the provided unit is installed. + +> `Systemd.UnitInstalled("nginx") == true` + +### `UnitConfig(unitName, key) string` + +    Returns the value of the specified key from the specified unit. + +    Returns an empty value if the unit if not installed and an error if the key does not exist. + +> `Systemd.UnitConfig("nginx", "StandardOutput") == "journal"` + +### `UnitLogsToJournal(unitName) bool` + +    Returns `true` if unit stdout/stderr are redirect to journal or journal+console. + +> `Systemd.UnitLogsToJournal("nginx") == true` + +## Windows + +    This object exposes helpers to get informations about Windows services. + +    Only available on Windows. + +### `ServiceEnabled(serviceName) bool` + +    Returns `true` if the specified service exists and is configured to start automatically on boot. + +> `Windows.ServiceEnabled("MSSSQLSERVER") == true` + +## Version + +### `Check(version, constraint) bool` + +    Performs a semantic version check. + +    Constraint supports operators like `=`, `!=`, `<`, `<=`, `>`, `>=`, ranges (1.1.1 - 1.3.4), AND with commas (`>1`, `<3`), and ~ compatible ranges. + +> `Version.Check(Host.KernelVersion, ">=6.24.0")` diff --git a/crowdsec-docs/docs/log_processor/service-discovery-setup/intro.md b/crowdsec-docs/docs/log_processor/service-discovery-setup/intro.md new file mode 100644 index 00000000..9ccb2280 --- /dev/null +++ b/crowdsec-docs/docs/log_processor/service-discovery-setup/intro.md @@ -0,0 +1,176 @@ +--- +id: intro +title: Service Discovery +sidebar_position: 1 +--- + +# Service Discovery + +The goals of service discovery are to automatically: + - Detect services on your machine supported by crowdsec + - Install related hub items + - Generate acquisition configuration + +## Basic Usage + +The main way to use the service discovery is with `cscli setup interactive` or `cscli setup unattended`. + +By default, it will use the detection file provided by crowdsec stored in `/var/lib/crowdsec/data/detect.yaml`. + +In interactive mode, `cscli` will ask you to choose which service to configure based on those that were detected, and will require confirmation before any operation (installing hub items, generating acquisition config, ...). + +It is your responsibility to check the compatibility of the generated acquisitions with the ones you add later or were already on the system. + +:::warning + +While `cscli setup` validates the generated configuration files for syntax errors or invalid configuration, it does *not* check for duplicate acquisition. + +If using a custom `detect.yaml`, make sure no logs are read multiple times (with the same `type` label), as this could lead to false positives. + +::: + + +`cscli` will ask for confirmation before proceeding if: + +- there is an `acquis.yaml` +- there is any non-generated file in `acquis.d` +- you modified the generated files in `acquis.d` (there is a checksum to detect modifications). Proceeding could overwrite them. + +Files composed by comments only are ignored. + +Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation. In the case above, instead of asking for confirmation, unattended mode will just skip the service detection. + + +### Generated acquisition files & coexistence with your own files + +When you generated the acquisition configuration with `cscli setup`, `cscli` writes one file per service as `setup..yaml` in the acquisition directory (typically `/etc/crowdsec/acquis.d`). The content is prefixed with a header that includes a checksum and a comment stating it was generated by `cscli setup`. + +- Files carrying a valid checksum are considered generated and may be overwritten by future runs. +- Files without a valid checksum are treated as manually edited; in interactive mode, `cscli` shows a colorized diff and asks before overwriting. In unattended flows, the command refuses to proceed if manual files are detected. +- Either way, the safest practice is: **don’t edit generated files**. If you need changes, delete the generated `setup..yaml` and create your own hand‑managed file instead or use a custom `detect.yaml` to generate the proper configuration automatically. + +> Tips + +> - The actual on‑disk path is computed as `acquis.d/setup..yaml` where `` comes from `acquisition_spec.filename`. +> - Use `--acquis-dir` to target a different directory. +> - `--dry-run` prints what would be created without writing files. + + +## Advanced Usage + +### Use a custom `detect.yaml` + +You can provide a custom `detect.yaml` in two ways: + +```bash +# Path flag +cscli setup detect --detect-config /path/to/detect.yaml + +# Or via environment variable +CROWDSEC_SETUP_DETECT_CONFIG=/path/to/detect.yaml cscli setup detect +``` + +Helpful flags: + +- `--yaml` – render the setup plan as YAML (easy to review/edit); default output is JSON. +- `--force ` – pretend detection matched for `` (repeatable). +- `--ignore ` – drop `` from the plan even if matched (repeatable). +- `--skip-systemd` – disable systemd‐based detection (default if systemctl can't be run). +- `--list-supported-services` – print the service keys present in your file and exit. + +You can see a list of all the available expr helpers in the [dedicated documentation](/log_processor/service-discovery-setup/expr.md). + +For example, if you have configured nginx to log in a non-standard location, you can use a custom `detect.yaml` to override it. + +This example will generate an acquisition with the pattern `/srv/logs/nginx/*.log` if the nginx service is installed OR if any file matches the glob pattern `/srv/logs/nginx/*.log`: + +```yaml +# detect.yaml +--- +detect: + nginx-custom: + when: + - Systemd.UnitInstalled("nginx.service") or len(Path.Glob("/srv/logs/nginx/*.log")) > 0 + hub_spec: + collections: + - crowdsecurity/nginx + acquisition_spec: + filename: nginx.yaml + datasource: + source: file + filenames: + - /srv/logs/nginx/*.log + labels: + type: nginx +``` + +:::warning + +When using a custom detect configuration, the default one will be fully ignored. + +This means that on top of your custom detection, you will most likely want to add the basic OS detection, for example: + +```yaml +detect: + linux: + when: + - Host.OS == "linux" + hub_spec: + collections: + - crowdsecurity/linux + acquisition_spec: + filename: linux.yaml + datasource: + source: file + labels: + type: syslog + filenames: + - /var/log/messages + - /var/log/syslog + - /var/log/kern.log +``` + +::: + +### Unattended installs with a custom detect file + +Linux packages (deb or rpm) will automatically call `cscli setup unattended` during installation. + +You can specify a custom detection file to use by setting `CROWDSEC_SETUP_DETECT_CONFIG` before installing the package with `apt` or `dnf`. + +Alternatively, if you want to skip the automatic detection completely, you can set the env var `CROWDSEC_SETUP_UNATTENDED_DISABLE` to any value. + +### End-to-end workflow + +Behind the scenes, `cscli setup` use multiple steps to configure crowdsec: + +- Generate a YAML plan that contains the detected services, their associated hub items and acquisition configuration +- Validate this file +- Install the hub items +- Write the acquisition config to disk + +If you wish, you can manually invoke any of those steps (if you only want to install the hub items for example). + +`cscli setup detect` can be used to generate the setup file: + +```bash +cscli setup detect --detect-config ./detect.yaml --yaml > setup.yaml +``` + +You can then validate its content for syntax error or issues with the acquisition configuration: + +```bash +cscli setup validate ./setup.yaml +``` + +Then, install the hub items: + +```bash +cscli setup install-hub ./setup.yaml +``` + +And finally, write the acquisition config: + +```bash +cscli setup install-acquisition ./setup.yaml --acquis-dir /etc/crowdsec/acquis.d +``` diff --git a/crowdsec-docs/sidebars.ts b/crowdsec-docs/sidebars.ts index e55234e7..f03cf491 100644 --- a/crowdsec-docs/sidebars.ts +++ b/crowdsec-docs/sidebars.ts @@ -126,6 +126,18 @@ const sidebarsConfig: SidebarConfig = { }, ], }, + { + type: "category", + label: "Service Discovery", + link: { + type: "doc", + id: "log_processor/service-discovery-setup/intro", + }, + items: [ + "log_processor/service-discovery-setup/detect-yaml", + "log_processor/service-discovery-setup/setup-expr-helpers", + ], + }, { type: "category", label: "Alert Context", diff --git a/crowdsec-docs/unversioned/bouncers/haproxy.mdx b/crowdsec-docs/unversioned/bouncers/haproxy.mdx index c9428dd2..b8ba8537 100644 --- a/crowdsec-docs/unversioned/bouncers/haproxy.mdx +++ b/crowdsec-docs/unversioned/bouncers/haproxy.mdx @@ -43,7 +43,7 @@ This component is compatible with HAProxy version 2.5 and higher. ## How does it work ? -This component leverages haproxy lua's API to check e IP address against the local API. +This component leverages haproxy lua's API to check the IP address against the local API. Supported features: diff --git a/crowdsec-docs/unversioned/bouncers/ingress-nginx.mdx b/crowdsec-docs/unversioned/bouncers/ingress-nginx.mdx index 735f3e91..b6cf50f4 100644 --- a/crowdsec-docs/unversioned/bouncers/ingress-nginx.mdx +++ b/crowdsec-docs/unversioned/bouncers/ingress-nginx.mdx @@ -312,7 +312,7 @@ CAPTCHA_PROVIDER=recaptcha ``` :::info -For backwards compatability reasons `recaptcha` is the default if no value is set. +For backwards compatibility reasons `recaptcha` is the default if no value is set. ::: ### `SECRET_KEY` diff --git a/crowdsec-docs/unversioned/bouncers/nginx.mdx b/crowdsec-docs/unversioned/bouncers/nginx.mdx index d6f86cae..46bcb012 100644 --- a/crowdsec-docs/unversioned/bouncers/nginx.mdx +++ b/crowdsec-docs/unversioned/bouncers/nginx.mdx @@ -515,7 +515,7 @@ CAPTCHA_PROVIDER= note: The ratio of fire to smoke is around 1% at the time of writting +> note: The ratio of fire to smoke is around 1% at the time of writing ## CTI Information diff --git a/crowdsec-docs/unversioned/getting_started/installation/cloudways.mdx b/crowdsec-docs/unversioned/getting_started/installation/cloudways.mdx index 78ab610c..c8a85d78 100644 --- a/crowdsec-docs/unversioned/getting_started/installation/cloudways.mdx +++ b/crowdsec-docs/unversioned/getting_started/installation/cloudways.mdx @@ -291,7 +291,7 @@ We want CrowdSec to run in the background and start at boot. For this we'll add a systemd service in the user level. ### Create the systemd service for user -- At the time of writting (for v1.6.3) you can use the following content: +- At the time of writing (for v1.6.3) you can use the following content: - Create and edit ~/.config/systemd/user/crowdsec.service ```bash [Unit] diff --git a/crowdsec-docs/unversioned/getting_started/installation/docker.mdx b/crowdsec-docs/unversioned/getting_started/installation/docker.mdx index 1eae6a60..4174096e 100644 --- a/crowdsec-docs/unversioned/getting_started/installation/docker.mdx +++ b/crowdsec-docs/unversioned/getting_started/installation/docker.mdx @@ -64,7 +64,7 @@ crowdsec: #### Compose key aspects -If you dont find an example that fits your needs, you can create your own `docker-compose.yml` file. Here are the key aspects: +If you don't find an example that fits your needs, you can create your own `docker-compose.yml` file. Here are the key aspects: ##### Provide Access To Logs diff --git a/crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx b/crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx index 5dcfd1be..12edd487 100644 --- a/crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx +++ b/crowdsec-docs/unversioned/getting_started/post_installation/acquisition.mdx @@ -5,13 +5,14 @@ title: Acquisition # Acquisition -By default when CrowdSec is installed it will attempt to detect the running services and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro). +By default when CrowdSec is installed it will attempt to [detect the running services](https://docs.crowdsec.net/next/log_processor/service-discovery-setup/intro) (for CrowdSec >= 1.7.0) and acquire the appropriate log sources and [Collections](https://docs.crowdsec.net/docs/next/collections/intro). -However, we should check that this detection worked or you may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for other services that are not detected. +However, we should check that this detection worked and the log locations are correct. +You may want to manually acquire additional [Collections](https://docs.crowdsec.net/docs/next/collections/intro) for the services that were not detected. ## What log sources are already detected? -To find what log sources are already detected, you can use the `cscli` command line tool. +To find out which log sources are providing data to crowdsec, you can query the CrowdSec metrics with the `cscli` command line tool. ```bash cscli metrics show acquisition diff --git a/crowdsec-docs/unversioned/troubleshooting/remediation_components.mdx b/crowdsec-docs/unversioned/troubleshooting/remediation_components.mdx index e189be23..df7ce8ae 100644 --- a/crowdsec-docs/unversioned/troubleshooting/remediation_components.mdx +++ b/crowdsec-docs/unversioned/troubleshooting/remediation_components.mdx @@ -59,7 +59,7 @@ You can use the os related commands to filter the logs to only show errors. **Please make sure the log location matches your distribution.** -## My Remediaton Component is not showing any error messages within its log file but its failing to start/work +## My Remediation Component is not showing any error messages within its log file but its failing to start/work Most likely means the bouncer is failing to decode the configuration file provided. To find which line is causing the issue, you can use systemd/journalctl to get the error message: