Shutting down JVM after 8 consecutive periods of m

2019-07-25 18:25发布

问题:

I am writing Apache beam BAtch dataflow in which I am writing from GCS to BQ. My data contains 4 millions of records. I have specified n1-HighMem-8 machine type.My dataflow works form small amount of data.

I my use case I schema is not fixed so I have used .getFailedInserts() Method to get schema failed records not inserted.

I have grouped them and writing to BQ using BQ load job via GCS in same dataflow.

for this amount of data I am geting following error 7 time and then my dataflow errors out:

E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system

E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system E Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /

Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /

[ContainerManager]: Fail to get rootfs information unable to find data for container /

E PercpuUsage had 0 cpus, but the actual number is 8; ignoring extra CPUs E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities

E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system

E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system E Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /

Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 27662/33436/33436 MB, GC last/max = 93.00/95.00 %, #pushbacks=0, gc thrashing=true. Heap dump not written.