Cloud Dataflow job scaling beyond max worker value

问题:

Dataflow job id: 2016-01-13_16_00_09-15016519893798477319

Pipeline was configured with the following worker/scaling config:

• min 2 workers
• max 50 workers

However, the job scaled to 55 workers. Why was the max worker value of 50 not honoured?

Jan 14, 2016, 11:00:10 AM
(77f7e53b4884ba02): Autoscaling: Enabled for job 2016-01-13_16_00_09-15016519893798477319 between 1 and 1000000 worker processes.

Jan 14, 2016, 11:00:17 AM
(374d4f69f65e2506): Worker configuration: n1-standard-1 in us-central1-a.

Jan 14, 2016, 11:00:18 AM
(28acda8454e90ad2): Starting 2 workers...

Jan 14, 2016, 11:01:49 AM
(cf611e5d4ce4784d): Autoscaling: Resizing worker pool from 2 to 50.

Jan 14, 2016, 11:06:20 AM
(36c68efd7f1743cf): Autoscaling: Resizing worker pool from 50 to 55.

回答1:

This turned out to be a bug in our code. We were calling the wrong method. We need to call setMaxNumWorkers, and not setNumWorkers.