Create Local Big Data Environment

Creates a fully functional local big data environment including Apache Hive, Apache Spark and HDFS.

The Spark WebUI of the created local Spark context is available via the Spark context outport view. Simply click on the Click here to open link and the Spark WebUI is opened in the internal web browser.

Note : Executing this node only creates a new Spark context, when no local Spark context with the same Context name currently exists. Resetting the node does not destroy the context. Whether closing the KNIME workflow will destroy the context or not, depends on the configured Action to perform on dispose . Spark contexts created by this node can be shared between KNIME workflows.

Output ports

1. Type: DB Session
   Hive Connection
   JDBC connection to a local Hive instance. This port can be connected to the KNIME database nodes.
2. Type: File System
   File System Connection
   File System connection that points to the local file system. This port can be connected for example to the Spark nodes that read/write files.
3. Type: Spark Context
   Spark Context
   Local Spark context, that can be connected to all Spark nodes.