What is istio-proxy access log mean?

Question

I am trying to troubleshoot my service by looking at the istio-proxy access log (it logs every access). However, I can't find any documentation that explains the meaning of each entry in the log.

For example

[2018-12-20T11:09:42.302Z] "GET / HTTP/1.1" 200 - 0 614 0 0 "10.32.96.32" "curl/7.54.0" "17b8f245-af00-4379-9f8f-a4dcd2f38c01" "foo.com" "127.0.0.1:8080"

What does log above mean?

Updated

I've tried Vadim's answer, but I couldn't find the log format data. Here's the output json file. Is there anything that I miss? I am using istio-1.0.0

Answer 1

Istio proxy access log's configuration is defined as part of envoy.http_connection_manager or envoy.tcp_proxy filters. To see it's configuration, run:

istioctl proxy-config listeners <your pod> -n <your namespace> -o json

Search for access_log of envoy.http_connection_manager for HTTP and access_log of envoy.tcp_proxy for TCP.

You will see something like this:

 "filters": [
                {
                    "name": "envoy.http_connection_manager",
                    "config": {
                        "access_log": [
                            {
                                "config": {
                                    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME%\n",
                                    "path": "/dev/stdout"

Check the log attributes definitions here

If access_log's format is not specified in the output above, the default format is used.

Answer 2

Istio/Envoy access logs comes with a default format. Here is the default format

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"\n

It matches with the sample log entry that you have given. You can find more details about the fields and generally about envoy's access logs here

Answer 3

Here is the format of log:

\[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?|%{DATA:}) %{DATA:protocol}\" %{NUMBER:status_code} %{DATA:response_flags} \"%{**DATA:mixer_status**}\" %{NUMBER:bytes_received} %{NUMBER:bytes_sent} %{NUMBER:duration} (?:%{NUMBER:upstream_service_time}|%{DATA:tcp_service_time}) \"%{DATA:forwarded_for}\" \"%{DATA:user_agent}\" \"%{DATA:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_service}\" %{DATA:upstream_cluster} %{DATA:upstream_local} %{DATA:downstream_local} %{DATA:downstream_remote} %{**DATA:requested_server**}

Here is real log:

[2019-09-05T06:55:32.008Z] "GET /solutionprofile/api/v1/health HTTP/1.1" 200 - "-" 0 16 10 10 "-" "kube-probe/1.12" "dc9ac3b2-2ee4-4a4b-967e-8f0cc3953e80" "10.228.69.15:3000" "127.0.0.1:3000" inbound|80|http|hp-solutionprofile-service.hp.svc.cluster.local - 10.228.69.15:3000 10.228.69.1:59692 -