Docs: Heavy Forwarder using Standalone CR - add docs/HeavyForwarder-Standalone.md #1603

vivekr-splunk · 2025-10-22T05:28:09Z

Description

Add a new guide docs/HeavyForwarder-Standalone.md that walks users through creating a Splunk Heavy Forwarder on Kubernetes using the Splunk Operator Standalone custom resource. The guide covers:

ConfigMap driven configuration with defaultsUrl
Reliable Splunk to Splunk forwarding with outputs.conf (indexAndForward=false, useACK=true, compressed=true, autoLBFrequency=30)
Operator managed HEC enablement via the namespace secret
Deployment steps, validation, troubleshooting, cleanup, and references

Key Changes

Added docs/HeavyForwarder-Standalone.md

No code or charts changed.

Testing and Verification

Rendered the Markdown locally to verify headings, anchors, and code blocks
Ran Markdown linting to check formatting
Dry-ran the provided kubectl commands against a test cluster to validate:
- Standalone CR becomes Ready
- Operator secret contains a HEC token
- inputs.conf shows HEC enabled
- outputs.conf shows heavy forwarder settings
- Test HEC event posts successfully and is searchable at the indexers

No automated tests are added since this is documentation only.

Related Issues

Jira: CSPL-4133

PR Checklist

Code changes adhere to the project's coding standards.
Relevant unit and integration tests are included.
Documentation has been updated accordingly.
All tests pass locally.
The PR description follows the project's guidelines.

Copilot

Pull Request Overview

This PR adds comprehensive documentation for deploying a Splunk Heavy Forwarder on Kubernetes using the Splunk Operator's Standalone Custom Resource. The guide demonstrates ConfigMap-based configuration management, operator-managed HEC enablement, and reliable Splunk-to-Splunk forwarding.

Key Changes:

Added complete heavy forwarder deployment guide covering architecture, configuration, deployment steps, validation procedures, and troubleshooting
Documented operator-managed HEC token generation and automatic inputs.conf configuration
Provided detailed testing procedures with expected outputs for each validation step

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-22T05:29:04Z

docs/HeavyForwarder-Standalone.md

+
+The ConfigMap contains a `default.yml` file with three main configuration sections:
+
+##### 1. outputs.conf - Forwarding Configuration (Lines 9-22)


The line number reference '(Lines 9-22)' is misleading as it refers to lines in the original YAML file structure, not the documentation itself. Consider removing the line number reference or clarifying what it refers to (e.g., 'Lines 9-22 in the ConfigMap YAML structure shown below').

Suggested change

##### 1. outputs.conf - Forwarding Configuration (Lines 9-22)

##### 1. outputs.conf - Forwarding Configuration (Lines 9-22 in the ConfigMap YAML structure shown below)

Copilot · 2025-10-22T05:29:04Z

docs/HeavyForwarder-Standalone.md

+- `compressed: true` - Reduces network bandwidth
+- `autoLBFrequency: 30` - Distributes load across indexers every 30 seconds
+
+##### 2. props.conf - Data Parsing Rules (Lines 23-30)


Similar to the previous section, the line number reference '(Lines 23-30)' is unclear. These line numbers don't correspond to lines in this document and may confuse readers.

Suggested change

##### 2. props.conf - Data Parsing Rules (Lines 23-30)

##### 2. props.conf - Data Parsing Rules

Copilot · 2025-10-22T05:29:04Z

docs/HeavyForwarder-Standalone.md

+- Can filter out unwanted events
+- Can route to specific indexer groups
+
+##### 3. transforms.conf - Data Transformation (Lines 31-42)


The line number reference '(Lines 31-42)' should be clarified or removed to avoid confusion, as these numbers don't map to the current document structure.

Suggested change

##### 3. transforms.conf - Data Transformation (Lines 31-42)

##### 3. transforms.conf - Data Transformation

Copilot · 2025-10-22T05:29:05Z

docs/HeavyForwarder-Standalone.md

+- `drop_noise` - Filters out DEBUG messages to reduce noise
+- `to_idx_svc` - Routes specific host data to the indexer cluster
+
+### Part 2: Standalone Custom Resource (Lines 44-55)


The line number reference '(Lines 44-55)' should be removed or clarified, as it doesn't correspond to lines in this documentation.

Suggested change

### Part 2: Standalone Custom Resource (Lines 44-55)

### Part 2: Standalone Custom Resource

coveralls · 2025-10-22T05:39:57Z

Pull Request Test Coverage Report for Build 18706328521

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 86.552%

Totals
Change from base Build 18653942794:	0.0%
Covered Lines:	10710
Relevant Lines:	12374

💛 - Coveralls

kasiakoziol · 2025-10-22T06:29:43Z

docs/HeavyForwarder-Standalone.md

+
+1. **Kubernetes cluster** with kubectl access
+2. **Splunk Operator** installed and running
+3. **Two namespaces**:


I think it's optional to run these in the following namespaces since we support both single-namespace and multiple-namespace deployments, so it doesn't have to be a part of the prerequisites. Same with the service name.

kasiakoziol · 2025-10-22T06:34:37Z

docs/HeavyForwarder-Standalone.md

+
+**Transforms Explained:**
+- `drop_noise` - Filters out DEBUG messages to reduce noise
+- `to_idx_svc` - Routes specific host data to the indexer cluster


Is it idx_svc as above or to_idx_svc as here?

kasiakoziol · 2025-10-22T06:41:52Z

docs/HeavyForwarder-Standalone.md

+### Official Documentation
+- [Splunk Operator GitHub](https://github.com/splunk/splunk-operator)
+- [Splunk Operator Documentation](https://splunk.github.io/splunk-operator/)
+- [Splunk Heavy Forwarder Documentation](https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Aboutforwardingandreceivingdata)


It returns 404

Maybe it should be https://help.splunk.com/en/splunk-enterprise/forward-and-process-data/forwarding-and-receiving-data/10.0/introduction-to-forwarding/about-forwarding-and-receiving?

kasiakoziol · 2025-10-22T06:41:56Z

docs/HeavyForwarder-Standalone.md

+- [transforms.conf](https://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf)
+
+### Splunk Operator Custom Resources
+- [Standalone CR Specification](https://splunk.github.io/splunk-operator/StandaloneSpec.html)


It returns 404

kasiakoziol · 2025-10-22T06:42:02Z

docs/HeavyForwarder-Standalone.md

+
+### Splunk Operator Custom Resources
+- [Standalone CR Specification](https://splunk.github.io/splunk-operator/StandaloneSpec.html)
+- [Common Spec Parameters](https://splunk.github.io/splunk-operator/CommonSpec.html)


It returns 404

kasiakoziol · 2025-10-22T06:42:46Z

docs/HeavyForwarder-Standalone.md

+kubectl get standalone -n forwarder
+
+# Optional: Delete PVCs (data will be lost)
+kubectl delete pvc -n forwarder -l app.kubernetes.io/instance=splunk-hf-standalone-standalone


There are finalizers, so it should happen automatically

rlieberman-splunk · 2025-10-22T12:56:35Z

docs/HeavyForwarder-Standalone.md

+        indexAndForward: false             # CRITICAL: False = Heavy Forwarder mode
+        autoLBFrequency: 30                # Load balance every 30 seconds
+        compressed: true                   # Compress data during transmission
+      "tcpout:idx_svc":


Is there a reason that this has double quotes and dcpout on line 117 does not? It looks like they are at the same indentation, so it might make sense to make them consistent.

heavy forwarder standalone setup

269492f

vivekr-splunk self-assigned this Oct 22, 2025

vivekr-splunk requested review from Igor-splunk, Copilot, patrykw-splunk and rlieberman-splunk October 22, 2025 05:28

Copilot AI reviewed Oct 22, 2025

View reviewed changes

vivekr-splunk requested a review from kasiakoziol October 22, 2025 05:46

kasiakoziol approved these changes Oct 22, 2025

View reviewed changes

rlieberman-splunk reviewed Oct 22, 2025

View reviewed changes

rlieberman-splunk approved these changes Oct 22, 2025

View reviewed changes


		The ConfigMap contains a `default.yml` file with three main configuration sections:

		##### 1. outputs.conf - Forwarding Configuration (Lines 9-22)

	##### 2. props.conf - Data Parsing Rules (Lines 23-30)
	##### 2. props.conf - Data Parsing Rules

	##### 3. transforms.conf - Data Transformation (Lines 31-42)
	##### 3. transforms.conf - Data Transformation

	### Part 2: Standalone Custom Resource (Lines 44-55)
	### Part 2: Standalone Custom Resource

Docs: Heavy Forwarder using Standalone CR - add docs/HeavyForwarder-Standalone.md #1603

Are you sure you want to change the base?

Docs: Heavy Forwarder using Standalone CR - add docs/HeavyForwarder-Standalone.md #1603

Uh oh!

Conversation

vivekr-splunk commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Testing and Verification

Related Issues

PR Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

coveralls commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18706328521

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vivekr-splunk commented Oct 22, 2025 •

edited

Loading

coveralls commented Oct 22, 2025 •

edited

Loading