Need to be able to see output in cell BEFORE it completes
When I'm running Python or other code in a cell, it produces intermediate logs/outputs (e.g. when training a model, it shows how the accuracy is improving after each iteration), but these outputs are only shown AFTER the whole cell completes running. If the cell takes 5 minutes to run, we have no clue what it's doing. In a Jupyter notebook, we see output right away. This is a must-have feature
AdminPM (Product Manager, Databricks) commented
Spark 2.0.2-db3 and later versions support displaying continuous opuputs for cells. You can find more information about this feature here: https://docs.databricks.com/release-notes/product/latest.html#continuous-results-for-notebook-cells
Khurom Kiyani commented
For some reason I switched to a different cluster set-up today [ Spark 2.1 (Auto-updating Scala 2.11) ] and to my surprise what you have requested is now happening :) . I gave you three votes, but have now taken them off since it seems to have been implemented. That was pretty fast from data bricks -- 1 day!!! Maybe other people had already asked this before. In any case, I for one am liking it.