Initial Hybrid Agent Trace implementation #1587

lrafeei · 2025-11-19T00:21:05Z

This PR contains a barebones tracing implementation.

Implementation for different transaction types, Distributed Tracing/context propagation, tracer configuration, instrumentation scope attributes, and meters will come in subsequent PRs

github-actions · 2025-11-19T00:28:03Z

✅MegaLinter analysis: Success

Descriptor	Linter	Files	Fixed	Elapsed time
✅ ACTION	actionlint	7		0.96s
✅ MARKDOWN	markdownlint	7	0	1.33s
✅ PYTHON	ruff	955	1	1.02s
✅ PYTHON	ruff-format	955	1	0.41s
✅ YAML	prettier	15	0	1.54s
✅ YAML	v8r	15		5.72s
✅ YAML	yamllint	15		0.71s

See detailed reports in MegaLinter artifacts

codecov-commenter · 2025-11-19T00:45:44Z

Codecov Report

❌ Patch coverage is 51.11940% with 131 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop-hybrid-core-tracing@afbc9f2). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
newrelic/api/opentelemetry.py	46.15%	83 Missing and 15 partials ⚠️
newrelic/hooks/hybridagent_opentelemetry.py	56.57%	28 Missing and 5 partials ⚠️

Additional details and impacted files

@@                      Coverage Diff                       @@
##             develop-hybrid-core-tracing    #1587   +/-   ##
==============================================================
  Coverage                               ?   79.94%           
==============================================================
  Files                                  ?      210           
  Lines                                  ?    24488           
  Branches                               ?     3886           
==============================================================
  Hits                                   ?    19578           
  Misses                                 ?     3548           
  Partials                               ?     1362

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hmstepanek

Just reviewed this one class so far but wanted to get you the feedback now rather than waiting until next week.

newrelic/api/opentelemetry.py

hmstepanek · 2025-11-20T23:14:27Z

newrelic/api/opentelemetry.py

+            # but for debug purposes, we will raise an error
+            _logger.debug("Otel span and NR trace do not match nor correspond to a remote span")
+            _logger.debug("otel span: %s\nnewrelic trace: %s", self.otel_parent, current_nr_trace)
+            raise ValueError("Unexpected span parent scenario encountered")


Is this error caught somewhere?

In this case, I cannot think of a scenario where this would happen, so I wanted to keep this here for debug purposes.

hmstepanek · 2025-11-21T22:56:34Z

newrelic/api/opentelemetry.py

+            _logger.debug("otel span: %s\nnewrelic trace: %s", self.otel_parent, current_nr_trace)
+            raise ValueError("Unexpected span parent scenario encountered")
+
+        if nr_trace_type == FunctionTrace:


I have a lot of questions about these. Mainly; shouldn't we be pulling all the attributes out of the OTEL attr list and mapping the here into the appropriate NR equivalents unless there isn't a spot for them in which case they are supposed to become user attributes according to the spec?

That part will come in a separate PR. For now, the attributes are added as custom attributes, but the actual attribute mapping per SpanKind will come in separate PRs. Of course, most of the time, the attributes are not added during initialization, but rather throughout the span's operation, so I will be adding logic to add these when the span is ending.

hmstepanek · 2025-11-21T22:58:34Z

newrelic/api/opentelemetry.py

+        elif nr_trace_type == DatastoreTrace:
+            trace_kwargs = {
+                "product": self.instrumenting_module,
+                "target": None,


It looks like target maps to OTEL's db.name.

It does! The issue with the OTel attributes is that the db.name attribute is usually not set during the creation of the span, but rather later on in the span's operation.

newrelic/api/opentelemetry.py

hmstepanek · 2025-11-21T23:37:27Z

newrelic/api/opentelemetry.py

+    def record_exception(self, exception, attributes=None, timestamp=None, escaped=False):
+        if not hasattr(self, "nr_trace"):
+            if exception:
+                notice_error((type(exception), exception, exception.__traceback__))


Should we be passing attributes here too?

hmstepanek · 2025-11-21T23:38:23Z

newrelic/api/opentelemetry.py

+            else:
+                notice_error(sys.exc_info(), attributes=attributes)
+        else:
+            self.nr_trace.notice_error((type(exception), exception, exception.__traceback__), attributes=attributes)


What if there's no exception here?

hmstepanek · 2025-11-21T23:38:43Z

newrelic/api/opentelemetry.py

+        # TODO: not implemented yet
+        pass
+
+    def record_exception(self, exception, attributes=None, timestamp=None, escaped=False):


Should we be doing something with timestamp and escaped here?

It does not seem like we will do anything with the escaped argument, and the timestamp argument will be used for the events/log events (which I have not implemented yet)

hmstepanek · 2025-11-21T23:42:33Z

newrelic/api/opentelemetry.py

+        # We will ignore the end_time parameter and use NR's end_time
+
+        # Check to see if New Relic trace ever existed or,
+        # if it does, that trace has already ended


Maybe store this value and simplify the logic rather than calling hasattr a bunch:

nr_trace = hasattr(self, "nr_trace", None) if not nr_trace or nr_trace and getattr(nr_trace, "end_time", None): return

Co-authored-by: Hannah Stepanek <[email protected]>

lrafeei added 2 commits November 18, 2025 15:56

Initial otel trace/span implementation

ae0cb31

Add some hybrid cross agent tests

acc1607

mergify bot added the merge-conflicts Merge conflicts detected. label Nov 19, 2025

Merge branch 'develop-hybrid-core-tracing' into hybrid-agent-trace-base

b01ac4b

mergify bot removed the merge-conflicts Merge conflicts detected. label Nov 19, 2025

mergify bot added the tests-failing Tests failing in CI. label Nov 19, 2025

Tweak some formatting and merge issues

b45a7c8

lrafeei force-pushed the hybrid-agent-trace-base branch from 13df939 to b45a7c8 Compare November 19, 2025 21:33

[MegaLinter] Apply linters fixes

bf075f1

lrafeei marked this pull request as ready for review November 19, 2025 22:04

lrafeei requested a review from a team as a code owner November 19, 2025 22:04

mergify bot and others added 2 commits November 20, 2025 18:53

Merge branch 'develop-hybrid-core-tracing' into hybrid-agent-trace-base

c33134e

Fix application initialization bug

daa433f

hmstepanek requested changes Nov 21, 2025

View reviewed changes

lrafeei and others added 9 commits November 21, 2025 15:52

Update newrelic/api/opentelemetry.py

9e03321

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

e2bf015

Co-authored-by: Hannah Stepanek <[email protected]>

[MegaLinter] Apply linters fixes

40a58f7

Update newrelic/api/opentelemetry.py

abf1a31

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

22f1906

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

04ceb65

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

08fed22

Co-authored-by: Hannah Stepanek <[email protected]>

[MegaLinter] Apply linters fixes

17aa20f

Reviewer suggestions, part 1

2aafac3

lrafeei requested a review from hmstepanek November 22, 2025 00:45

lrafeei and others added 4 commits November 22, 2025 00:46

[MegaLinter] Apply linters fixes

f8d6cf7

Fix hasattr syntax

98b6285

Update newrelic/api/opentelemetry.py

ea958bb

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

ee66602

Co-authored-by: Hannah Stepanek <[email protected]>

lrafeei and others added 4 commits November 21, 2025 16:55

Update newrelic/api/opentelemetry.py

5deedb8

Co-authored-by: Hannah Stepanek <[email protected]>

Update newrelic/api/opentelemetry.py

9d270d2

Co-authored-by: Hannah Stepanek <[email protected]>

More reviewer suggestions/syntax fixes

fbe86e2

[MegaLinter] Apply linters fixes

af6a853

Initial Hybrid Agent Trace implementation #1587

Are you sure you want to change the base?

Initial Hybrid Agent Trace implementation #1587

Uh oh!

Conversation

lrafeei commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅MegaLinter analysis: Success

Uh oh!

codecov-commenter commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hmstepanek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Nov 19, 2025 •

edited

Loading

codecov-commenter commented Nov 19, 2025 •

edited

Loading