Create Local Big Data Environment (Legacy)

Source

Creates a fully functional local big data environment including Apache Hive, Apache Spark and HDFS.

The Spark WebUI of the created local Spark context is available via the Spark context outport view. Simply click on the Click here to open link and the Spark WebUI is opened in the internal web browser.

Note: Executing this node only creates a new Spark context, when no local Spark context with the same Context name currently exists. Resetting the node does not destroy the context. Whether closing the KNIME workflow will destroy the context or not, depends on the configured Action to perform on dispose. Spark contexts created by this node can be shared between KNIME workflows.

Output Ports

  1. Type: Database Connection
    JDBC connection to a local Hive instance. This port can be connected to the KNIME database nodes.
  2. Type: Remote Connection
    HDFS connection that points to the local file system. This port can be connected for example to the Spark nodes that read/write files.
  3. Type: Spark Context
    Local Spark context, that can be connected to all Spark nodes.

  1. Interactive Big Data Exploration and Visualization

    big data data exploration visualization webportal
    This workflow shows how to perform data exploration and visualization on a large dataset using KNIME Big Data Extensions…
    Users > knime > Examples > 50_Applications > 49_NYC_Taxi_Visualization

  2. h2o_glm

    Users > knime > Examples > 50_Applications > 36_Guided_Analytics_for_ML_Automation > 01_Guided_Analytics_for_ML_Automation > Models > classification

  3. Big Data Preprocessing

    Hive Hadoop Big Data SQL in-database
    This workflow demonstrates the usage of the database nodes in conjunction with the Hive Connector node, which is part of…
    Users > knime > Examples > 10_Big_Data > 01_Big_Data_Connectors

  4. Fetch and Transform PubChem Data

    BigData Spark JSON REST
    This workflow prepares a data set using Local Big Data Environment for Data Chefs Battle: Chemistry vs Biology. It colle…
    Users > knime > Examples > 08_Other_Analytics_Types > 02_Chemistry_and_Life_Sciences > 02_Fetch_And_Transform_PubChem_Data

  5. h2o_gbm

    Users > knime > Examples > 50_Applications > 36_Guided_Analytics_for_ML_Automation > 01_Guided_Analytics_for_ML_Automation > Models > classification

  6. Performance and Scalability Testing: Example KNIME with Spark

    This workflows show how to learn a random forest using the KNIME Apache Spark nodes. We here are measuring the speed of …
    Users > knime > Examples > 50_Applications > 53_Performance_and_Scalability > 01_workflows

  7. Deployment_workflow

    Demand prediction random forest time series prediction mode Spark cluster NYC taxi datset
    In this use case, we will use the NYC taxi dataset and a Random Forest to train a simple time series prediction model to…
    Users > knime > Examples > 10_Big_Data > 02_Spark_Executor > 11_Taxi_Demand_Prediction

  8. h2o_naive

    Users > knime > Examples > 50_Applications > 36_Guided_Analytics_for_ML_Automation > 01_Guided_Analytics_for_ML_Automation > Models > classification

  9. Local_Big_Data_Irish_Meter

    Hive Spark Spark PCA Spark Pivot Local Big Data Environment Big Data Spark SQL Parquet
    This workflow uses a portion of the Irish Energy Meter dataset, and presents a simple analysis based on the whitepaper "…
    Users > knime > Examples > 10_Big_Data > 02_Spark_Executor

  10. Movie Recommendation Engine with Spark Collaborative Filtering

    retail association rules recommandation engine market basket analysis MBA Apriori algorithm collaborative filtering; Spark
    1. Create local Spark Context; 2. Read ratings.csv and movies.csv from movie-lens dataset into Spark (https://grouplens.…
    Users > knime > Examples > 10_Big_Data > 02_Spark_Executor

Extension

This node is part of the extension

KNIME Extension for Local Big Data Environments

 v4.0.0

Short Link

Drag node into KNIME Analytics Platform