Cluster Management

{brickster} has 1:1 mappings with the clusters REST API, enabling full control of Databricks clusters from your R session.

Cluster Creation

Clusters have a number of parameters and can be configured to match to needs of a given workload. db_cluster_create() facilitates creation of a cluster in a Databricks workspace for all cloud platforms (AWS, Azure, GCP).

Depending on the cloud you will need to change the node types and cloud_attrs to be one of; aws_attributes(), azure_attributes(), or gcp_attributes().

Below we will create a cluster on AWS and then step through using the other supporting functions.

library(brickster)

# create a small cluster on AWS with DBR 9.1 LTS
new_cluster <- db_cluster_create(
  name = "brickster-cluster",
  spark_version = "9.1.x-scala2.12",
  num_workers = 2,
  node_type_id = "m5a.xlarge",
  cloud_attrs = aws_attributes(
    ebs_volume_count = 3,
    ebs_volume_size = 100
  )
)

Refer to documentation for details on how to use other parameters not mentioned here (e.g. spark_conf).

Before creating a cluster you may want to check the supported values for a number of the parameters. There are functions to assist with this:

Function Purpose
db_cluster_runtime_versions() List of runtime versions available for the workspace, useful for finding relevant spark_version
db_cluster_list_node_types() List of supported node types available in workspace/region, useful for finding relevant node_type_id/driver_node_type_id
db_cluster_list_zones() AWS Only, lists availability zones (AZ) clusters can occupy

db_cluster_get() will provide details for the cluster we just created, including information such as the state.

This can be useful as you may wish to wait for the cluster to be RUNNING , which is exactly what get_and_start_cluster() uses internally to wait until the cluster is running before completing.

cluster_info <- db_cluster_get(cluster_id = new_cluster$cluster_id)
cluster_info$state

Editing Clusters

You can edit Databricks clusters to change various parameters using db_cluster_edit(). For example, we may decide we want our cluster to autoscale between 2-8 nodes and add some tags.


# we are required to input all parameters
db_cluster_edit(
  cluster_id = new_cluster$cluster_id,
  name = "brickster-cluster",
  spark_version = "9.1.x-scala2.12",
  node_type_id = "m5a.xlarge",
  autoscale = cluster_autoscale(min_workers = 2, max_workers = 8),
  cloud_attrs = aws_attributes(
    ebs_volume_count = 3,
    ebs_volume_size = 100
  ),
  custom_tags = list(
    purpose = "brickster_cluster_demo"
  )
)

However, if the intention was to only change the size of a given cluster the db_cluster_resize() function is a simpler alternative.

I can either adjust the number of workers or change the autoscale range. If the range or workers is adjusted via autoscale the number of workers active on the cluster will be increased/decreased if they are outside the bounds.

# adjust number autoscale range to be between 4-6 workers
db_cluster_resize(
  cluster_id = new_cluster$cluster_id,
  autoscale = cluster_autoscale(min_workers = 4, max_workers = 6)
)

It’s important to note that if specifying num_workers instead of autoscale on a cluster than has an existing autoscale range it will become a fixed number of workers from that point onward.

Databricks clusters can be “pinned” which stops them from being removed after 30 days of termination. db_cluster_pin() and db_cluster_unpin() are the functions used for changing if a cluster is “pinned” or not.

# pin the cluster
db_cluster_pin(cluster_id = new_cluster$cluster_id)

# unpin the cluster
# db_cluster_unpin(cluster_id = new_cluster$cluster_id)

Cluster State

There are a few functions that can be used to to manage the state of an existing cluster

Function Purpose
db_cluster_start() Start a cluster that is inactive
db_cluster_restart() Restart a cluster, cluster must already be running
db_cluster_delete() /db_cluster_terminate() Terminate an active cluster, does not remove the cluster configuration from Databricks
db_cluster_perm_delete() Stops (if active) and permanently deletes a cluster, it will not longer appear in Databricks

Cluster Libraries

Databricks clusters can have libraries installed from a number of sources using db_libs_install() and the associated libs_*() functions:

Function Library Source
lib_cran() CRAN
lib_pypi() PyPi
lib_egg() Python egg (file)
lib_whl() Python wheel (file)
lib_maven() Maven
lib_jar() JAR (file) 
# installing a package from CRAN on cluster
db_libs_install(
  cluster_id = new_cluster$cluster_id,
  libraries = libraries(
    lib_cran(package = "palmerpenguins"),
    lib_cran(package = "dplyr")
  )
)

For convenience the wait_for_lib_installs() function will block until all the libraries for the specified cluster have finished installing.

wait_for_lib_installs(cluster_id = new_cluster$cluster_id)

Installation of libraries is asynchronous and will complete in the background. db_libs_cluster_status() is used to check on the installation status of libraries for a given cluster, db_libs_all_cluster_statuses() is used for getting the status of all libraries across all clusters in the workspace.

db_libs_cluster_status(cluster_id = new_cluster$cluster_id)

Libraries can be uninstalled using db_libs_uninstall().

db_libs_uninstall(
  cluster_id = new_cluster$cluster_id,
  libraries = libraries(
    lib_cran(package = "palmerpenguins")
  )
)

Using db_libs_cluster_status() shows that the library will be uninstalled upon restart (e.g. db_cluster_restart()).

db_libs_cluster_status(cluster_id = new_cluster$cluster_id)

Events

A list of events regarding the clusters activity can be fetched via db_cluster_events(). There are many event types that can occur, and by default the 50 most recent events are returned.

events <- db_cluster_events(cluster_id = new_cluster$cluster_id)
head(events, 1)