Datadog APM이 안될 때(적용이 안될 때) 5가지 확인 방법

Datadog Agent yaml을 제대로 구성을 했음에도 불구하고 Datadog APM이 안될 때 아래 5가지 방법에 대해 확인이 가능하다.

---

1) APM Trace 활성화 확인

본인의 datadog agent yaml 파일이 있는 경로에 가서 확인한다.

$ cat /etc/datadog-agent/datadog.yaml

 

그 이후 아래처럼 apm_config 부분에 APM Trace 부분이 활성화되어 있는지 확인한다.

## Provides autodetected defaults, for kubernetes environments,
## please see datadog.yaml.example for all supported options

apm_config:
apm_non_local_traffic: true

# Use java container support
jmx_use_container_support: true

 

 

2) APM 서비스 여부 확인

Agent 상태를 확인하여 APM 서비스가 있는지 확인한다.

Trace Agent 섹터부분을 확인 후 아래처럼 구성되어 있는지 확인한.

$ agent status

=============
Process Agent
=============

Version: 7.58.2
Status date: 2024-12-10 01:55:38.525 UTC (1733795738525)
Process Agent Start: 2024-12-05 07:16:03.024 UTC (1733382963024)
Pid: 1
Go Version: go1.22.7
Build arch: amd64
Log Level: info
Enabled Checks: [process_discovery connections rtcontainer container]
Allocated Memory: 0 bytes
Hostname: ip-192-168-14-121.ap-northeast-2.compute.internal-cloudwiz-eks
System Probe Process Module Status: Not running
Process Language Detection Enabled: True

=================
Process Endpoints
=================
https://process.datadoghq.com - API Key ending with:

=========
Collector
=========
Last collection time: 2024-12-10 01:55:33
Docker socket:
Number of processes: 0
Number of containers: 22
Process Queue length: 0
RTProcess Queue length: 0
Connections Queue length: 0
Event Queue length: 0
Pod Queue length: 0
Process Bytes enqueued: 0
RTProcess Bytes enqueued: 0
Connections Bytes enqueued: 0
Event Bytes enqueued: 0
Pod Bytes enqueued: 0
Drop Check Payloads: []

===========
Trace Agent
===========
Version: 7.58.2
Status date: 2024-12-10 01:55:38.525 UTC (1733795738525)
Trace Agent Start: 2024-12-05 07:16:05.123 UTC (1733382965123)
Pid: 2
Enabled: True
API Key: Authorized
APM Endpoint: https://trace.agent.datadoghq.com
Trace Queue length: 5
Traces enqueued: 1200
Traces sent: 1195
Tracer error count: 0

==========
Extractors
==========

Workloadmeta
============
Cache size: 93
Stale diffs discarded: 0
Diffs dropped: 0

====================
Remote Configuration
====================
Organization enabled: True
API Key: Authorized
Last error: None

============
System Probe
============
Status: Running
Uptime: 114h39m15.000071281s
Last Updated: 2024-12-10 01:55:28 UTC (1733795728000)

USM
===
Status: Disabled

NPM
===
Status: Running
Last Check: 2024-12-10 01:55:13 UTC (1733795713000)

Event Monitor
================
Status: Running

========================
Transport Proxy Warnings
========================
No Transport Proxy Warnings

 

 

3) 올바른 URL 접속 확인

curl http://localhost:8126/info

 

8126 Port에 올바르게 접속이 되는지 확인한다. (안될 경우 문제 있는 것)

root@datadog-agent-kbhcg:/# curl http://localhost:8126/info
{
"version": "7.58.2",
"git_commit": "4ad1243",
"endpoints": [
"/v0.3/traces",
"/v0.3/services",
"/v0.4/traces",
"/v0.4/services",
"/v0.5/traces",
"/v0.7/traces",
"/profiling/v1/input",
"/telemetry/proxy/",
"/v0.6/stats",
"/v0.1/pipeline_stats",
"/evp_proxy/v1/",
"/evp_proxy/v2/",
"/evp_proxy/v3/",
"/evp_proxy/v4/",
"/debugger/v1/input",
"/debugger/v1/diagnostics",
"/symdb/v1/input",
"/dogstatsd/v1/proxy",
"/dogstatsd/v2/proxy",
"/tracer_flare/v1",
"/v0.7/config",
"/config/set"
],
"client_drop_p0s": true,
"span_meta_structs": true,
"long_running_spans": true,
"evp_proxy_allowed_headers": [
"Content-Type",
"Accept-Encoding",
"Content-Encoding",
"User-Agent",
"DD-CI-PROVIDER-NAME"
],
"config": {
"default_env": "none",
"target_tps": 10,
"max_eps": 200,
"receiver_port": 8126,
"receiver_socket": "/var/run/datadog/apm.socket",
"connection_limit": 0,
"receiver_timeout": 0,
"max_request_bytes": 26214400,
"statsd_port": 8125,
"max_memory": 0,
"max_cpu": 0,
"analyzed_spans_by_service": {},
"obfuscation": {
"elastic_search": true,
"mongo": true,
"sql_exec_plan": false,
"sql_exec_plan_normalize": false,
"http": {
"remove_query_string": false,
"remove_path_digits": false
},
"remove_stack_traces": false,
"redis": {
"Enabled": true,
"RemoveAllArgs": false
},
"memcached": {
"Enabled": true,
"KeepCommand": false
}
}
},
"peer_tags": null,
"span_kinds_stats_computed": null

 

 

4) 환경변수 체크

Datadog 환경변수가 잘 적용됐는지 확인

$ printenv | grep DD

 

맨 아래 부분에 해당 환경변수가 설정되어 있는지 확인한다. (없으면 추가)

1) DD_TRACE_ENABLED=true
2) DD_APM_ENABLED=true

root@datadog-agent-kbhcg:/# printenv | grep DD

DD_ORCHESTRATOR_EXPLORER_CONTAINER_SCRUBBING_ENABLED=true
DD_GIT_REPOSITORY_URL=
DD_DOGSTATSD_SOCKET=/var/run/
DATADOG_AGENT_PORT_8125_UDP_ADDR=
DD_PYTHON_VERSION=3
DD_API_KEY=
DD_LEADER_ELECTION=true
DD_LEADER_LEASE_NAME=datadog-leader-election
DD_SYSTEM_PROBE_NETWORK_ENABLED=true
DD_CLUSTER_AGENT_ENABLED=true
DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE=true
DATADOG_AGENT_PORT_8126_TCP_ADDR=
DD_SYSPROBE_SOCKET=/var/runk
DD_IGNORE_AUTOCONF=kubernetes_state
DD_CLUSTER_NAME=
DD_PROCESS_CONFIG_RUN_IN_CORE_AGENT_ENABLED=false
DD_REMOTE_CONFIGURATION_ENABLED=true
DD_LOG_LEVEL=info
DD_KUBERNETES_KUBELET_HOST=
DATADOG_ADMISSION_CONTROLLER_PORT_443_TCP_ADDR=
DD_COLLECT_DNS_STATS=true
DD_EXTRA_CONFIG_PROVIDERS=clusterchecks endpointschecks
DD_CONTAINER_IMAGE_ENABLED=true
DD_GIT_COMMIT_SHA=
DD_SYSTEM_PROBE_CONFIG_ENABLE_CONNTRACK=true
DD_CLUSTER_AGENT_AUTH_TOKEN=
DD_HEALTH_PORT=
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
DD_CLUSTER_AGENT_TOKEN_NAME=
DD_SECRET_BACKEND_COMMAND_ALLOW_GROUP_EXEC_PERM=true
KUBERNETES_PORT_443_TCP_ADDR=
DD_COLLECT_KUBERNETES_EVENTS=true
DD_SYSTEM_PROBE_ENABLED=true
DD_LOGS_ENABLED=true
DD_ORCHESTRATOR_EXPLORER_ENABLED=true
DD_LANGUAGE_DETECTION_ENABLED=true
DD_CLUSTER_AGENT_KUBERNETES_SERVICE_NAME=datadog-cluster-agent
DATADOG_CLUSTER_AGENT_PORT_5005_TCP_ADDR=
DOCKER_DD_AGENT=true
DD_SITE=datadoghq.com
DD_TRACE_ENABLED=true
DD_APM_ENABLED=true

 

 

5) 포트 여부 확인

Datadog와 통신하는 포트가 열려있는지 확인한다.

$ ss -tuln | grep 8126

 

아래 처럼뜨면 열려 있는 것 (안열려 있다면 오픈해주자)

tcp LISTEN 0 4096 *:8126 *:*