Hub
  • Software
  • Blog
  • Forum
  • Events
  • Documentation
  • About KNIME
  • KNIME Hub
  • Nodes
  • Spark DataFrame Java Snippet
NodeNode / Manipulator

Spark DataFrame Java Snippet

Tools & Services Apache Spark Misc Java Snippet
Drag & drop
Like
Copy short link

This node allows you to execute arbitrary java code to manipulate or create Spark DataFrames. Simply enter the java code in the text area.

Note, that this node also supports flow variables as input to your Spark job. To use a flow variable simply double click on the variable in the "Flow Variable List".

It is also possible to use external java libraries. In order to include such external jar or zip files, add their location in the "Additional Libraries" tab using the control buttons. For details see the "Additional Libraries" tab description below.
The used libraries need to be present on your cluster and added to the class path of your Spark job server. They are not automatically uploaded!

You can define reusable templates with the "Create templates..." button. Templates are stored in the users workspace by default and can be accessed via the "Templates" tab. For details see the "Templates" tab description below.

Node details

Input ports
  1. Type: Spark Data
    First Spark Data
    First input Spark DataFrame.
  2. Type: Spark Data
    Second Spark Data
    Optional second input Spark DataFrame. (optional)
Output ports
  1. Type: Spark Data
    Result Spark Data
    Result Spark DataFrame.

Extension

The Spark DataFrame Java Snippet node is part of this extension:

  1. Go to item

Related workflows & nodes

  1. Go to item
    TryCatchSpark
    mareike.hoeger > Public > TryCatchSpark
  2. Go to item
    NRU0
    khauthanhhieu > Public > NRU0
  3. Go to item
    Fetch and Transform PubChem Data
    BigData Spark JSON
    +2
    This workflow prepares a data set using Local Big Data Environment for Data Chefs Battle:…
    b_eslami > Public > 02_Chemistry_and_Life_Sciences > 02_Fetch_And_Transform_PubChem_Data > 02_Fetch_And_Transform_PubChem_Data
  4. Go to item
    Fetch and Transform PubChem Data
    BigData Spark JSON
    +2
    This workflow prepares a data set using Local Big Data Environment for Data Chefs Battle:…
    knime > Examples > 08_Other_Analytics_Types > 02_Chemistry_and_Life_Sciences > 02_Fetch_And_Transform_PubChem_Data > 02_Fetch_And_Transform_PubChem_Data
  5. Go to item
    Cleaning the NYC taxi dataset on Spark
    Big data Exploration Visualization
    +4
    This workflow handles the preprocessing of the NYC taxi dataset (loading, cleaning, filte…
    knime > Examples > 50_Applications > 49_NYC_Taxi_Visualization > Data_Preparation
  1. Go to item
  2. Go to item
  3. Go to item
  4. Go to item
  5. Go to item
  6. Go to item

KNIME
Open for Innovation

KNIME AG
Hardturmstrasse 66
8005 Zurich, Switzerland
  • Software
  • Getting started
  • Documentation
  • E-Learning course
  • Solutions
  • KNIME Hub
  • KNIME Forum
  • Blog
  • Events
  • Partner
  • Developers
  • KNIME Home
  • KNIME Open Source Story
  • Careers
  • Contact us
Download KNIME Analytics Platform Read more on KNIME Server
© 2022 KNIME AG. All rights reserved.
  • Trademarks
  • Imprint
  • Privacy
  • Terms & Conditions
  • Credits