Bfloat16 training in GPUs

问题:

Hi I am trying to train a model using the new bfloat16 datatype variables. I know this is supported in Google TPUs. I was wondering if anyone has tried training using GPUs (for example, GTX 1080 Ti). Is that even possible, whether the GPU tensor cores are supportive? If anyone has any experience please share your thoughts. Many thanks!

回答1:

I had posted this question in Tensorflow github community. Here is their response so far - " bfloat16 support isn't complete for GPUs, as it's not supported natively by the devices.

For performance you'll want to use float32 or float16 for GPU execution (though float16 can be difficult to train models with). TPUs support bfloat16 for effectively all operations (but you currently have to migrate your model to work on the TPU). "