What are default VPC firewall rules for App Engine

Question

I have a few services deployed in one App Engine flexible environment project. The service instances are running in the same VPC network (not default) which is specified in their app.yaml files and assigned IP addresses from the same subnetwork. By default both services are accessible from outside and I want to configure firewall rules to leave only the default service allowed for incoming traffic.

So as the first step I'm configuring a rule to deny all ingress for all instances in this VPC, with the idea of creating another rule with a higher priority to allow traffic to only one instance which is running the default service.
The problem is that when I create a firewall rule with priority 1001 to block all incoming traffic my default service still get requests. But if I specify the priority 1000 or bellow then the traffic is blocked and I get 502 Server Error.

The question is of course WHY? The VPC documentation states that there are only two default firewall rules used for all manually created VPC networks:

• A default "allow egress" rule.
  Allows all egress connections. Rule has a priority of 65535.
• A default "deny ingress" rule.
  Deny all ingress connection. Rule has a priority of 65535

Then how come that my 1001 priority rule fails to block the ingress and the exactly the same rule with priority 1000 (or below) works as expected? Are there any other default firewall rules for Flex instances or am I missing something else here?

Answer 1

From what I understand, both firewalls must allow the traffic in order for a request to reach your App Engine. If any one has a good reason to block the request, then the request is blocked.

You can know which firewall is blocking by looking at the response code :

• A 502 -> VPC firewall
• A 403 -> App Engine Firewall

If logs are enabled, you can know which rule on the VPC firewall is causing the deny.

Regarding the "1000 against 1001 priority" rule, that's a very strange (and undocumented) behavior. So let's document it here for the folks who might stumble on it :

Rules on the VPC network for flexible App Engine instances seems to work only if priority is below 1000.