[DRAFT] Describe Trace Snapshot Profiling

specification/behaviors.md

@@ -51,3 +51,69 @@
 
 See [integration_context.md](integration_context.md) for specifics about
 exchanging additional context between AppD and splunk-otel based agents.
+
+## Trace Snapshot Profiling
+
+**Status**: [Experimental](../README.md#versioning-and-status-of-the-specification)
+
+This section describes the behavior for Splunk instrumentation libraries 
+that contain trace snapshot profiling features.
+
+### Instrumentation Source
+
+Agents MUST specify set the `profiling.instrumentation.source` value to `snapshot`
+
+### Starting Trace Profiler
+
+The OpenTelemetry Baggage entry for `splunk.trace.snapshot.volume` MUST be used to 
+decide whether to profile a trace. A value of `higest` is the signal to begin 
+profiling where as a value of `off` is an explicit signal to not profile.
+
+When profiling a trace, the profiler SHOULD be started when an entry span is 
+detected. An entry span is defined as either the root span of the trace or 
+any other span within a trace whose parent span is remote.
+
+When a trace is profiled agents MUST add the span attribute `splunk.snapshot.profiling` 
+with a value of `true` to the entry span.
+
+Agents SHOULD take an initial stack trace sample when starting to profile a trace.
+
+### Trace Profiling
+An instrumentation library that has trace snapshot profiling capabilities MUST
+be able to sample call stacks for specific trace ids at a fixed interval.
+
+When a language runtime supports threading, stacks MUST be sampled only for 
+trace ids selected for snapshotting. The samples for profiled threads SHOULD be 
+taken instantaneously and MAY be taken at separate times.
+
+Agents MUST sample threads associated with the entry span for the duration of 
+the span's life.
+
+### Call Stack Span Association
+
+Agents SHOULD keep track of the current span for each profiled thread. Agents 
+are RECOMMENDED to use the OpenTelemetry Context for determining when the current 
+span changes.
+
+When available, agents MUST use the span id from the profiled thread's current span
+as the span id.
+
+### Stopping Trace Profiler
+Trace profiling MUST be stopped when the entry span of a service ends.
+
+Agents SHOULD take a final stack trace sample when stopping profiling 
+for a trace.
+
+### Exporting Stack Traces
+It is RECOMMENDED to export stack traces in batches to take advantage of the pprof 
+data format.
+
+Agents SHOULD attempt to export any remaining stack traces during the Agent shutdown phase. 
+
+### Call Stack Ingest
+
+Call stacks MUST be ingested as [OpenTelemetry
+Logs](https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/logs).
+The logs containing profiling data MUST be sent via OTLP. Instrumentation
+libraries SHOULD reuse persistent OTLP connections from other signals (traces,
+metrics).

specification/configuration.md

@@ -173,6 +173,7 @@ instance using the following environment variables:
 | `SPLUNK_PROFILER_MEMORY_ENABLED`       | false   | Whether memory profiling is enabled. [2] [6]                                             |
 | `SPLUNK_REALM`                         | `none`  | Which realm to send exported data. [3]                                                   |
 | `SPLUNK_TRACE_RESPONSE_HEADER_ENABLED` | true    | Whether `Server-Timing` header is added to HTTP responses. [4]                           |
+| `SPLUNK_SNAPSHOT_PROFILER_ENABLED`     | false   | Whether Trace Snapshot CPU profiling is enabled. [2] [5]                                 |
 
 - [1]: Not user required if another system performs the authentication. For
   example, instrumentation libraries SHOULD send data to a locally running

specification/semantic_conventions.md

@@ -272,3 +272,20 @@ For each `cpu` sample:
   in milliseconds if this sample represents a periodic event
 - label `thread.state` of type `string` OPTIONALLY can be set to describe
   the state of the thread
+
+## Trace Snapshot Profiling
+
+**Status**: [Experimental](../README.md#versioning-and-status-of-the-specification)
+
+Unless stated otherwise Agents MUST follow the `Profiling `ResourceLogs` Message`
+semantic conventions.
+
+Trace Snapshot Profiling Configuration Options
+
+| Name                                         | Type   | Description                                         | Valid Values                 | Default |
+|----------------------------------------------|--------|-----------------------------------------------------|------------------------------| ------- |
+| `splunk.snapshot.profiler.enabled`           | string | Enable or Disable trace snapshot profiling          | `true` or `false`            | `false` |
+| `splunk.snapshot.profiler.sampling.interval` | string | Interval in which to take trace stack trace samples | Any valid duration `string`  | `10ms`  |
+
+The span attribute `splunk.snapshot.profiling` with a value of `true` indicates that 
+a trace within a service has been profiled.