I am using Bayesian regression for hyperparameter optimization, therefore, for some parameters I am running out of memory and this is throwing an error. Therefore, this error could be generated after many iterations, but might not be generated as well. So, I am looking to catch this error and then continue running the code normally. So, I am not sure where to place try
and catch
block in tensorflow. I guess, the last method throwing the error is session.py
. Here is the error:
---------------------------------------------------------------------------
ResourceExhaustedError Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
1321 try:
-> 1322 return fn(*args)
1323 except errors.OpError as e:
...
ResourceExhaustedError: OOM when allocating tensor with shape[14,128,1,788200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/ExpandDims, PermConstNHWCToNCHW-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Any help is much appreciated!!