Hub
Pricing About
  • Software
  • Blog
  • Forum
  • Events
  • Documentation
  • About KNIME
  • KNIME Community Hub
  • knime
  • Spaces
  • Examples
  • 10_Big_Data
  • 01_Big_Data_Connectors
  • 04_GoogleCloudExample
WorkflowWorkflow

Working with Google cloud services

Google Bigquery Cloud storage Spark Big data
KNIME profile image

Last edit:

Drag & drop
Like
Download workflow
Copy short link
Workflow preview
This workflow demonstrates how to connect to various Google Cloud Services such as Google BigQuery, Google Dataproc, and Google Cloud Storage from within KNIME Analytics Platform. The Google Authentication (API Key) node allows you to authenticate with the various Google APIs using a p12 key file. The output of the Google Authentication (API Key) node can be used as input for the Google BigQuery Connector node. The Google BigQuery Connector node provides a DB Connection which can be used with the existing DB nodes to visually assemble queries that are executed within your BigQuery cluster. To upload large amounts of data into the BigQuery cluster use the DB Loader node since the JDBC based interface has a lot of restrictions. The Google Cloud Storage Connector node connects KNIME Analytics Platform with your Google Cloud Storage and allows you to work with your files using the file handling nodes. The Google Cloud Storage File Picker node creates a pre-signed URL that can be used in the reader nodes in KNIME to read directly from Google Cloud Storage or that can be shared with other users to access the dedicated files without the need for authentication. Finally, the Create Spark Context (Livy) node can be used to set up a Spark context in your Google Cloud Dataproc. In order to use the node, you need to execute the Apache Livy Initialization Action during cluster creation. For more details see link to the documentation. Once a context is created you can use all the existing Spark nodes to visually assemble your Spark analysis flow.

External resources

  • Google Cloud Dataproc
  • Apache Livy Initialization Action
  • Google BigQuery
  • Google Cloud Storage

Used extensions & nodes

Created with KNIME Analytics Platform version 4.1.0
  • Go to item
    KNIME BigQuery Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

    KNIME profile image
    knime
  • Go to item
    KNIME Core Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

    KNIME profile image
    knime
  • Go to item
    KNIME Database Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

    KNIME profile image
    knime
  • Go to item
    KNIME Extension for Apache Spark Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

    KNIME profile image
    knime
  • Go to item
    KNIME Google Cloud Storage Connectors Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

    KNIME profile image
    knime
  • Go to item
    KNIME Twitter & Google Connectors Unknown extension

    This is an unpublished or unknown extension.

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

  1. Go to item
  2. Go to item
  3. Go to item
  4. Go to item
  5. Go to item
  6. Go to item
Loading deployments
Loading ad hoc executions

Legal

By using or downloading the workflow, you agree to our terms and conditions.

Discussion
Discussions are currently not available, please try again later.

KNIME
Open for Innovation

KNIME AG
Talacker 50
8001 Zurich, Switzerland
  • Software
  • Getting started
  • Documentation
  • E-Learning course
  • Solutions
  • KNIME Hub
  • KNIME Forum
  • Blog
  • Events
  • Partner
  • Developers
  • KNIME Home
  • KNIME Open Source Story
  • Careers
  • Contact us
Download KNIME Analytics Platform Read more on KNIME Business Hub
© 2023 KNIME AG. All rights reserved.
  • Trademarks
  • Imprint
  • Privacy
  • Terms & Conditions
  • Credits