OpenTelemetry tracing: InvalidTraceId

General
1

Hi team,

I am trying to config the elastic ci stack to send tracing data to Xray using OpenTelemetry backend.

I was able to config and send the trace to logging, however, when trying to add xray as another exporter, it seems that the traceId is not in the correct format (i.e. date component is way in the past). For example, one of the trace segement is

{
  "name": "agent environment hook",
  "id": "f228bdc87426ef9f",
  "start_time": 1698107995.2174475,
  "service": {
    "version": "3.56.0"
  },
  "trace_id": "1-470965de-abdd7b05e92bab26e2c9c2c0",
  "end_time": 1698107995.668894,
  "fault": false,
  "error": false,
  "throttle": false,
  "aws": {
    "xray": {
      "auto_instrumentation": false
    }
  },
  "metadata": {
    "default": {
      "analytics.event": "true",
      "hook.command": "/etc/buildkite-agent/hooks/environment",
      "hook.name": "environment",
      "hook.type": "agent"
    }
  },
  "parent_id": "ba4b42ec09e4a565",
  "type": "subsegment"
}

where the date of the trace_id is 470965de, which is 7 October 2007. As a result, xray api response an error

{
buildkite-agent-collector-1  |       ErrorCode: "InvalidTraceId",
buildkite-agent-collector-1  |       Id: "f228bdc87426ef9f",
buildkite-agent-collector-1  |       Message: "Invalid segment. ErrorCode: InvalidTraceId"
buildkite-agent-collector-1  |     }

I wonder if there is something I am missing here, or this could be a bug, could you please help to shed some light, that would be appreciated.

Thank you,
Nam

2

Hello @nam!

Hope you are well - and thanks for the message :wave:

At a glance, this looks like its a conversion error from the OpenTelemetry backend to an AWS XRay event: the trace_id format for such events includes that middle portion which is a 8 digit hexadecimal value representing the time of the original request - and it looks like the switching between the backends seems to be the case here.

I’d suggest in this case to open an issue on the Agent’s GitHub repository so the folks can assess this from an agent perspective (in particular, the OpenTemeletry integration. Any additional information that you can share (in addition to the above) there would be fantastic for assessment.

Cheers!

3

Thanks @james.s . I have submitted an GH issue. Looking forward to any insights from the team.

1 Like