XRay Traces not showing up in AWS Console

Question

I've followed the AWS documentation on setting up XRAY for our Spring Boot application deployed on AWS ECS, but I'm not able to see the traces for my services in the AWS Console. Here's a high level view of the changes I've implemented:

Added Role with Policy permissions to my EC2

"xray:BatchGetTraces",
"xray:GetServiceGraph",
"xray:GetTraceGraph",
"xray:GetTraceSummaries",
"xray:PutTelemetryRecords",
"xray:PutTraceSegments"

Added Tracing filter

@Bean
public Filter TracingFilter() {
    return new AWSXRayServletFilter("myService");
}

Added XRAY dependencies to our POM file and added @XRayEnabled annotation to our Controller method:

Downloaded the XRAY Daemon to our ec2 instance and installed

curl https://s3.dualstack.us-east-1.amazonaws.com/aws-xray-assets.us-east-1/xray-daemon/aws-xray-daemon-3.x.rpm -o /home/ec2-user/xray.rpm
yum install -y /home/ec2-user/xray.rpm

I've verified that we are seeing UDP logging statements for example: com.amazonaws.xray.emitters.UDPEmitter:

 {
  "name" : "myService",
  "id" : "1234",
  "start_time" : 1.546020031234E9,
  "trace_id" : "myTraceId",
  "end_time" : 1.546020031234E9,
  "http" : {
    "request" : {
      "method" : "POST",
      "client_ip" : "myIp",
      "url" : "myURL",
      "user_agent" : "PostmanRuntime/7.4.0",
      "x_forwarded_for" : true
    },
    "response" : {
      "content_length" : 200,
      "status" : 200
    }
  },
  "aws" : {
    "xray" : {
      "sdk_version" : "1.2.1",
      "sdk" : "X-Ray for Java"
    }
  },
  "service" : {
    "runtime" : "OpenJDK 64-Bit Server VM",
    "runtime_version" : "1.8.0_151"
  }
}

And I've also verified that the daemon is running on the ec2 using netstat -tulpn

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
udp        0      0 127.0.0.1:2000          0.0.0.0:*                           14126/xray 

What else is needed to be able to get the XRAY traces to show up in the AWS Console?

I've started the docker daemon with logging enabled, but I'm not seeing any indication that the docker daemon is sending data to AWS, just startup information and that's it:

2018-12-28T23:14:19Z [Info] Initializing AWS X-Ray daemon 3.0.0
2018-12-28T23:14:19Z [Info] Using buffer memory limit of 304 MB
2018-12-28T23:14:19Z [Info] 4864 segment buffers allocated
2018-12-28T23:14:19Z [Info] Using region: us-east-1
2018-12-28T23:14:19Z [Info] Starting proxy http server on 127.0.0.1:2000

Answer 1

I am guessing the issue is because you are running X-Ray Daemon on EC2 host and your java container is trying to send events to 127.0.0.1:2000 by default which is inside the java container itself but not the host address. Docker container sees 127.0.0.1 as within the container scope.

You will need to configure the X-Ray Daemon Address properly on your Java App.

AWS_XRAY_DAEMON_ADDRESS – Set the host and port of the X-Ray daemon listener. By default, the SDK uses 127.0.0.1:2000 for both trace data (UDP) and sampling (TCP). Use this variable if you have configured the daemon to listen on a different port or if it is running on a different host.

Format

Same port – address:port

Different ports – tcp:address:port udp:address:port

https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-configuration.html#xray-sdk-java-configuration-envvars

Since you are instrumenting the ECS based application, I would advise to spin up X-Ray Daemon as Docker Container but not as actual process on EC2 host.

Example -

1. Run the X-Ray Daemon as ECS Container(As Daemon scheduling type). https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html#service_scheduler
2. Based on which networking model you are using on ECS, you should be able to interact with X-Ray container Address and Port from your java container.

Let me know if you have any questions.

Update -

X-RAY Daemon as a Docker Container vs running it on the host?

It's just some opinion and it seems like some recommended way from AWS. Few advantages I can think of are

1. You don't have to maintain scripts/sequence for bringing X-Ray Daemon process part of your EC2 AMI.
2. You don't have to give whole EC2 Role permission to send data to X-Ray but with container, only that particular task role has permissions and not everything else.
3. If the process is stopped due to any reason, you have to manually bring up process or remove EC2 from cluster or maintain complex scripts on AMI. But with being ECS managed container, it will make sure task is running always.
4. ECS Daemon scheduling documentation says your case is why they brought this type of containers.

https://aws.amazon.com/about-aws/whats-new/2018/06/amazon-ecs-adds-daemon-scheduling/

Again, this is just my opinion but you can go with desired way as well.