Cluster configuration templates
We use a handful of different cluster configurations (eg 250 gb on demand, 100 gb on demand, 1 tb spot, etc) to run different operations. In the clusters tab there would be 2 areas - live clusters and cluster configurations. If I want to spin up a specific cluster that I've already configured then I can just click a create button for the configuration and it will spin up in the live cluster area.
This feature will become more useful with job scheduling so I can map each scheduled job to a cluster configuration so the correct size cluster is spun up for that job to run.
You can now re-start terminated clusters by clicking on the “play” button on the clusters page. That way, you can keep all your previous parameters and use previous cluster settings as templates. You can also clone a terminated cluster by clicking on the copy button on the clusters page.
Any news on cluster template ? It would be great if there would be a possibility to export the cluster configuration via JSON.
AdminAli Ghodsi (Admin, Databricks) commented
Yes it is. In about a week or two you'll be able to CLONE a cluster's (active or terminated, doesn't matter) configuration. That way you don't have to fill all the settings again and again. We are also working on the actual templates and persistent configs, but that's slightly further out.
Dominick Rocco commented
Is this still in the works? I long for this feature every time I configure the same cluster I configured yesterday.
Gabriele Del Prete commented
Is there any news on this feature? It was promised to be live 6 months ago.
Ophir Cohen commented
This would be something that we definitely need!
As addition to the above, we found ourself use clusters on a daily basis for ad-hoc queries and tests etc...
As it might be sporadic, we don't want to keep idle clusters up and running.
So, I would like to give my team the ability to trigger pre-configured cluster whenever they need and terminate it when they finish.