502 response coming from errors in Google Cloud Lo

Question

I'm using Google App Engine Flexible env (already migrated to env:flex) with python 3.4 runtime.

Over last month, i noticed multiple times(but less than 5% of requests) that i or automated processes, get a 502 from the server (Bad Gateway). Couldn't reproduce it locally and couldn't find any trace for it under GAE service logs. But looking for 502 error across all services, i realized that they come from Cloud HTTP Load Balancer service.

Going over the jsonPayload of these 502 errors, i see this reason:

statusDetails: "failed_to_pick_backend"
@type:"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I don't have a custom load-balancer service from GCP, i never had to deal with it till now. The GAE VM instance is consistently below 10% of CPU Utilization

Any suggestions on how to solve it?