Flink 1.3 running a single job on YARN how to set

问题:

I am running a single flink job on Yarn as descriped here.

flink run -m yarn-cluster -yn 3 -ytm 12000

I can set the number of yarn nodes / task managers with the above parameter -yn. However I want to know whether it is possible to set the number of task slots per task manager. When I use the parallelsim (-p) parameter it only sets the overall parallelism. And the number of task slots is computed by dividing this value by the number of provided task managers. I tried using the dynamic properties (-yD) parameter which is supposed to "allow the user to specify additional configuration values" like this:

 -yD -Dtaskmanager.numberOfTaskSlots=8

But this does not overwrite the value given in the flink-conf.yaml. Is there any way to specify the number of task slots per TaskManager when running a single on flink (other than changing the config file)? Also is there a documentation which dynamic properties are valid using the -yD parameter?

回答1:

You can use the settings of yarn-session, here, prefixed by y to submit Flink job on YARN cluster. For example the command,

flink run -m yarn-cluster -yn 5 -yjm 768 -ytm 1400 -ys 2 -yqu streamQ my_program.jar

will submit my_program.jar Flink application with 5 containers, 768m memory for the jobmanager, 1400m memory and 2 cpu core for taskmanagers, each and will use the resources of nodemanagers on predefined YARN queue streamQ. See my answer to this post for other important information.