Hub
NodeNode / Source

Create Databricks Environment

Tools & Services Apache Spark
Drag & drop
Like
Copy short link

Creates a Databricks Environment connected to an existsing Databricks cluster. See AWS or Azure Databricks documentation for more information.

Note: To avoid an accidental cluster startup, this node creates a dummy DB and Spark port if loaded in executed state from a stored workflow. Reset and execute the node to start the cluster and create a Spark execution context.

Cluster access control : KNIME uploads additional libraries to the cluster. This requires manage cluster-level permissions if your cluster is secured with access control. See the Databricks documentation on how to set up the permission.

Node details

Output ports
  1. Type: DB Session
    DB Connection
    JDBC connection, that can be connected to the KNIME database nodes.
  2. Type: File System
    DBFS Connection
    DBFS connection, that can be connected to the Spark nodes to read/write files.
  3. Type: Spark Context
    Spark Context
    Spark context, that can be connected to all Spark nodes.

Extension

The Create Databricks Environment node is part of this extension:

Related workflows & nodes

  1. Working with Databricks
    Big data Databricks Spark
    +2
    This workflow demonstrates the usage of the Create Databricks Environment node which allo…
    knime > Examples > 10_Big_Data > 01_Big_Data_Connectors > 03_DatabricksExample
  2. Working with Utility Nodes
    File handling Zip Unzip
    +4
    Download compressed file, extract it, read extracted file and finaly delete extracted fil…
    knime > Examples > 01_Data_Access > 01_Common_Type_Files > 11_Working_with_Utility_Nodes
  3. Working with Utility Nodes
    File handling Zip Unzip
    +4
    Download compressed file, extract it, read extracted file and finaly delete extracted fil…
    haoran > Public > 01_Data_Access > 01_Common_Type_Files > 11_Working_with_Utility_Nodes
  4. Incremental Data Processing with Parquet
    Parquet Incremental loading NYC taxi dataset
    +3
    In this workflow, we will use the NYC taxi dataset to show case a continous preprocessing…
    knime > Examples > 01_Data_Access > 01_Common_Type_Files > 12_Incremental_processing_Parquet_file
  5. Incremental Data Processing with Parquet
    Parquet Incremental loading NYC taxi dataset
    +3
    In this workflow, we will use the NYC taxi dataset to show case a continous preprocessing…
    haoran > Public > 01_Data_Access > 01_Common_Type_Files > 12_Incremental_processing_Parquet_file
  6. Google BigQuery meets Databricks
    Google BigQuery DB query Cloud
    +7
    This workflow connects to the Austin Bikeshare dataset, hosted among the Google BigQuery …
    knime > Examples > 10_Big_Data > 01_Big_Data_Connectors > 07_Will_They_Blend_BigQuery_Databricks
  7. Data Transfer between Clouds
    File handling Google Sharepoint
    +1
    This workflow demonstrates the utilization of the new file system connection nodes within…
    knime > Examples > 01_Data_Access > 06_ZIP_and_Remote_Files > 09_Data_Transfer_between_Clouds
  8. Data Transfer between Clouds
    File handling Google Sharepoint
    +1
    This workflow demonstrates the utilization of the new file system connection nodes within…
    haoran > Public > 01_Data_Access > 06_ZIP_and_Remote_Files > 09_Data_Transfer_between_Clouds