I am writing Apache beam BAtch dataflow in which I am writing from GCS to BQ. My data contains 4 millions of records. I have specified n1-HighMem-8 machine type.My dataflow works form small amount of data.
I my use case I schema is not fixed so I have used .getFailedInserts() Method to get schema failed records not inserted.
I have grouped them and writing to BQ using BQ load job via GCS in same dataflow.
for this amount of data I am geting following error 7 time and then my dataflow errors out:
E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system E Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
[ContainerManager]: Fail to get rootfs information unable to find data for container /
E PercpuUsage had 0 cpus, but the actual number is 8; ignoring extra CPUs E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities
E EXT4-fs (dm-0): couldn't mount as ext3 due to feature incompatibilities E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system
E Error initializing dynamic plugin prober: Error (re-)creating driver directory: mkdir /usr/libexec/kubernetes: read-only file system E Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
Shutting down JVM after 8 consecutive periods of measured GC thrashing. Memory is used/total/max = 27662/33436/33436 MB, GC last/max = 93.00/95.00 %, #pushbacks=0, gc thrashing=true. Heap dump not written.