TensorFlow 1.13.1 (stable) in ML 5.4
TensorFlow 1.13.1 contains major bugfixes compared to 1.12.0, here are some highlights:
* TensorFlow GPU binaries are now built against CUDA 10 and TensorRT 5.0.
(This is extremely important to leverage the full capabilities of V100 GPUs)
* Fixes a potential security vulnerability where carefully crafted GIF images can produce a null pointer dereference during decoding
* Improve performance of GPU cumsum/cumprod by up to 300x.
Any loads more, but these 3 are enough to warrant an upgrade, not to mention compatibility ops and renaming that make models less prone to breaking in TF 2.0
Hi, this is great, been running 1.13, I wanted to evaluate 1.14.0rc0 and 2.0.0b1 but when I set it to install either:
Then init script return non-zero... Maybe there's a bug, perhaps would be good to investigate ahead of 1.14.0 release?
Can I somehow view the logs?
Yes, we will upgrade to CUDA 10 with TF 1.13. We will provide instructions in the Runtime ML 5.4 blog post, which should show up in a few days. Stay tuned:)
Great! Please consider updating to the latest versions of CUDA, CudNN and TensorRT, on v100 instances the latest version can provide up to a 3x speedup for some ops (depthwise convolution for example).
Any chance I can try 1.13 in the current beta somehow? How would I install it?
If TF 1.14 is released and available in conda before ~July 1st, we will do the upgrade in Runtime 5.5 ML. If not, we will consider patching and upgrading to TF 1.13.
Okay, but what about the upcoming 1.14 release then (which is what master is right now)?
It's supposedly the last 1.x release so it could be good, also contains 2.x compatibility stuff
ML 5.4 will keep TF 1.12 unfortunately. There is a bug in TF 1.13.1 that makes Horovod distributed training fail. See https://github.com/tensorflow/tensorflow/pull/24461. It is fixed in master but not backported to 1.13.x release.
You can still install TF 1.13 or TF 2.0 alpha on ML 5.4. We will provide instructions in the release notes and the companion blog post.