Hub
Pricing About
  • Software
  • Blog
  • Forum
  • Events
  • Documentation
  • About KNIME
  • KNIME Community Hub
  • Nodes
  • Create Spark Context (Livy)
NodeNode / Source

Create Spark Context (Livy)

Tools & Services Apache Spark
Drag & drop
Like
Copy short link

Creates a new Spark context via Apache Livy .

This node requires access to a remote file system such as HDFS/webHDFs/httpFS or S3/Blob Store/Cloud Store in order to exchange temporary files between KNIME and the Spark context (running on the cluster).

Note : Executing this node always creates a new Spark context. Resetting the node or closing the KNIME workflow will destroy the Spark context. Spark contexts created by this node cannot be shared between KNIME workflows.

Node details

Input ports
  1. Type: File System
    Remote file system connection
    A connection to a remote file system to exchange temporary files between KNIME and the Spark context (running on the cluster). Supported file systems are:
    • HDFS, webHDFS and httpFS. Note that here KNIME must access the remote file system with the same user as Spark, otherwise Spark context creation fails. When authenticating with Kerberos against both HDFS/webHDFs/httpFS and Livy, then usually the same user will be used. Otherwise, this must be ensured manually.
    • Amazon S3, Azure Blob Store and Google Cloud Storage (recommended when using Spark on Amazon EMR/Azure HDInsight/Google Cloud Dataproc). Note that for these file systems a staging area must be specified (see above).
Output ports
  1. Type: Spark Context
    Spark Context
    Spark context.

Extension

The Create Spark Context (Livy) node is part of this extension:

  1. Go to item

Related workflows & nodes

  1. Go to item
  2. Go to item
  3. Go to item

No known nodes available

KNIME
Open for Innovation

KNIME AG
Talacker 50
8001 Zurich, Switzerland
  • Software
  • Getting started
  • Documentation
  • E-Learning course
  • Solutions
  • KNIME Hub
  • KNIME Forum
  • Blog
  • Events
  • Partner
  • Developers
  • KNIME Home
  • KNIME Open Source Story
  • Careers
  • Contact us
Download KNIME Analytics Platform Read more on KNIME Business Hub
© 2023 KNIME AG. All rights reserved.
  • Trademarks
  • Imprint
  • Privacy
  • Terms & Conditions
  • Credits