Specific to Docker based deployment, what are the differences between those two? Since Google App Engine Flexible now also supports Dockerfile based deployment and it is also fully-managed service, seems it's more preferred option rather than configuring Kubernetes deployment on Container Engine, isn't it?
What are the use cases where it's more preferred to use Google Container Engine over App Engine Flexible?
The main benefit apart from what is mentioned in Javier's answer is that using Kubernetes makes your application portable and independent of the underlying cloud platform. If you ever want to move your application to AWS, Azure, on-prem etc. you can do that with very few changes if you're using Kubernetes, while if you're on App Engine Flex, it'll require a lot more changes.
Another difference is if you want to create lots of different micro-services with complex structure, Kubernetes is more suited to that, instead of App Engine. It's not impossible to implement micro-service architecture in App Engine too. You can certainly use different yaml files, but those doesn't give you a lot of flexibility and any complex application will ultimately need to be ported.
Also, if you want to have stateful containers, you can't do that with App Engine Flex.
Kubernetes being an open-source project has more vibrant community than App Engine. Google is more focused on Kubernetes than App Engine and updates to Kubernetes are more frequent than App Engine.
But that doesn't mean App Engine is any less powerful. I have worked on complex applications serving hundreds of thousands (probably a million now) of users with App Engine (Both standard and flexible environment) and we never thought of migrating it out of GAE. I mean, if it can handle the traffic of Pokemon Go, it can certainly handle your app's too.
Performance wise someone did a comparison here. The summary results are as follows -
They are different things. App Engine Flexible is focused on application development - i.e. you have an application and you want to be deployed and managed by Google). On the other hand, Kubernetes is more about having your own infrastructure. Obviously, you can also deploy applications in Kubernetes but, as it's your "own" infrastructure, you are the one to directly manage how both the and the application will behave (create services, create scalability policies, RBAC, security policies...).
In this sense, Kubernetes is more flexible in what you can achieve. However, as a developer, you may not be interested in the infrastructure at all, only that your application works and scales. For this kind of profile, App Engine Flexible is more suitable.
If, on the other side, want to manage a complete Container infrastructure (more SRE profile), then Kubernetes is for you.
If you are looking to run your application 24/7, GKE also offers more vCPU and RAM resources at a lower price compared to the App Engine as well. You can further reduce your running cost if you run your applications with preemptible VMs (up to 80% cheaper than regular VMs), on Kubernetes, your app will be restarted with seconds of downtime.
Another advantage of GKE is that you can run multiple applications within the same cluster whereas App Engine charges you per app deployed.
PS: I run multiple sites on a GKE cluster. Doing this on App Engine will cost me a few times more than what I am currently paying.