Hub
  • Software
  • Blog
  • Forum
  • Events
  • Documentation
  • About KNIME
  • KNIME Hub
  • mlauber71
  • Spaces
  • Public
  • kn_example_db_bigdata_nodes
  • m_020_db_access_local_bigdata_tables
WorkflowWorkflow

An overview of KNIME based functions to access big data systems (with KNIME's local big data environment)

Knime Sql Db Base Database
+3

Last edited: 

Drag & drop
Like
Download workflow
Copy short link
Workflow preview
An overview of KNIME based functions to access big data systems (with KNIME's local big data environment) Use SQL with Impala/Hive and Spark and also PySpark to access and manipulate data on a big data system. The example is from the classic MS "Northwind" database. THX to J. Thelen for input from SQL lecture --------------- REMEMBER: Spark is about lazy evaluation. That means it will not do anything besides *planning* and preparing the transformations *until* you force it to do something. So the initial load of Spark may take some time (setting up the environment), the next steps might seem super fast (just structuring RDDs and creating -empty- place holders). The moment you want to get data back Spark springs into action and delivers the results.

External resources

  • A meta collection of KNIME and database related issues, dealing with basic SQL as well as Big Data related workflows with Hive/Impala and Spark (including PySpark)
  • Being Lazy is Useful — Lazy Evaluation in Spark
  • School of Hive - with KNIME's local Big Data environment (SQL for Big Data)
  • Spark DataFrame basic operations
  • Spark SQL built-in functions

Used extensions & nodes

Created with KNIME Analytics Platform version 4.4.1
  • Go to item
    KNIME Base nodes Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.1

  • Go to item
    KNIME Database Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.1

  • Go to item
    KNIME Extension for Apache Spark Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.1

  • Go to item
    KNIME Extension for Local Big Data Environments Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.0

  • Go to item
    KNIME Javasnippet Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.1

  • Go to item
    KNIME Quick Forms Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.4.1

  1. Go to item
  2. Go to item
  3. Go to item
  4. Go to item
  5. Go to item
  6. Go to item

Legal

By using or downloading the workflow, you agree to our terms and conditions.

Discussion
Discussions are currently not available, please try again later.

KNIME
Open for Innovation

KNIME AG
Hardturmstrasse 66
8005 Zurich, Switzerland
  • Software
  • Getting started
  • Documentation
  • E-Learning course
  • Solutions
  • KNIME Hub
  • KNIME Forum
  • Blog
  • Events
  • Partner
  • Developers
  • KNIME Home
  • KNIME Open Source Story
  • Careers
  • Contact us
Download KNIME Analytics Platform Read more on KNIME Server
© 2022 KNIME AG. All rights reserved.
  • Trademarks
  • Imprint
  • Privacy
  • Terms & Conditions
  • Credits