Product Feedback

  • Hot ideas
  • Top ideas
  • New ideas
  • My feedback
  1. 2 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    completed  ·  2 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  2. Define AWS Role to be used on the cluster

    I could set up the AWS Role on my cluster so my access permissions to S3 could be defined on roles and not in access and secret keys, this way I don't need to have access/secret keys harcoded on my workbook.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  3. Ganglia is included in Databricks Cloud

    Metrics: All, if possible. But these are the most important: cpu_*, dfs*, disk_*, disk_total, jvm.*, load_*, network_report,.

    This use case is to help us determine:

    1) The optimal set of configuration options.
    2) The optimal size for the cluster.
    3) Potential bugs in the code.

    The first 2 points are interrelated.

    13 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  4. we could spin up GPU-capable clusters.

    Would love to have access to GPU instances for deep learning.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  5. Ability to spin up clusters with smaller EC2 instance types for prototyping purposes

    It will be great for cost management if we ave the ability to spin up clusters with smaller EC2 instance types. This will allow the use of cheaper clusters for non-production purposes such as prototyping and investigations

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  6. 19 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  7. Improve cluster status visibility

    Subject:
    Improve cluster status visibility

    What is the idea?
    Expose more visibility during the cluster creation process, today there is no indication of the current progress and sometimes and takes a while to spin up instances (more than 10 min), which is fine but it would be better to expose more statuses to improve the system interaction with the user.

    For example, in case of spot instances you can expose the below statuses:
    - fulfilled
    - pending-evaluation
    - pending-fulfillment
    - price-too-low

    (Inspired by AWS spot market, more info here http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html#spot-instance-bid-status-understand)

    Why it matters?

    Better visibility is always better.
    Today,…

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  8. A cluster can scale down to 1 worker if idle for 30 minutes

    If a cluster is idle for > 30 minutes or some number, would be great if it scaled down automatically to just one worker.

    33 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    2 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    Auto-scaling is not ready (both up-scaling and down-scaling). Due to legacy contracts, older customers need to contact their SA to discuss how to get this enabled.

  9. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    Cluster ACLs have been released. In the clusters’ page, you can click on the right-most arrow/more button and click on “Permissions” to control who can run notebooks and other things on the cluster.

  10. A user had more spot pricing functionality

    The ability to set spot pricing when deploying a cluster. Also, the ability to choose an "automatic" or "no preference" for availability zone.

    3 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    You can now set custom spot bidding price when you create a cluster (in clusters page, in jobs page, or through the REST API). In the UI, look under advanced settings and AWS.

  11. "Recent History" for clusters would show system activity (like cluster terminated due to spot price being too high) along with user activity

    "Recent History" for clusters would show system activity (like cluster terminated due to spot price being too high) along with user activity.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  12. Allow terminating of clusters before they are started

    Sometimes it just takes ages to get spot clusters and we give up and create on-demand. In that case we need to remember to kill the spot one once it's up.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  13. auto-shutdown and auto-relaunch Cluster out of office hours and week days

    The idea is that when we create a Cluster that we use only for data/model exploration we can select an option to insure the cluster gets shut down out of business hours/days and get relaunched in time for the next day of work
    Of course we can do this manually but 99% of the time we would forget

    47 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    5 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    Happy to announce that the most voted for feature has been released. You can now setup auto-termination on any cluster that is launched. The timeout time for shutdown is configurable.

  14. Cluster Activity Log

    I'd like to be able to see when clusters were shut down or brought up and by whom.

    7 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  15. Show which notebooks are running specific commands

    It would be nice to find which notebook ran a specific job on a cluster

    22 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    It is now possible to see in the detailed clusters page which notebooks are actively running vs being idle. It is also possible to go into a notebook, click Schedule and see the list of jobs that are using that notebook. If you have additional use cases you’d like us to cover w.r.t. insight into what’s running, please contact us.

  16. clusters could be created programmatically

    in order to create a fully automatic data pipeline where Spark is just one step it is necessary to spin up a cluster, run some Spark job (i.e. Notebook) and terminate the cluster by some API. the web UI is quite limiting.

    5 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    completed  ·  1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  17. Cluster configuration templates

    We use a handful of different cluster configurations (eg 250 gb on demand, 100 gb on demand, 1 tb spot, etc) to run different operations. In the clusters tab there would be 2 areas - live clusters and cluster configurations. If I want to spin up a specific cluster that I've already configured then I can just click a create button for the configuration and it will spin up in the live cluster area.

    This feature will become more useful with job scheduling so I can map each scheduled job to a cluster configuration so the correct size cluster is…

    9 votes
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    4 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →

    You can now re-start terminated clusters by clicking on the “play” button on the clusters page. That way, you can keep all your previous parameters and use previous cluster settings as templates. You can also clone a terminated cluster by clicking on the copy button on the clusters page.

  18. 1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  19. AWS Exceptions/Responses displayed to screen

    When reconfiguring clusters I exceeded the AWS instance limit and no error messages were returned. If AWS Exceptions could be returned and the reconfiguration terminated more quickly it would be appreciated.

    I've noted from a response that this might already be in the works.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    0 comments  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
  20. Maintain history of clusters, even after termination

    * Have history of clusters (name, size, etc) after termination. Ability to clone.

    1 vote
    Vote
    Sign in
    (thinking…)
    Sign in with: Facebook Google
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    1 comment  ·  Cluster management  ·  Flag idea as inappropriate…  ·  Admin →
← Previous 1
  • Don't see your idea?

Feedback and Knowledge Base