Hub
ComponentComponent

Spark Lag Column

Last edited: 

Drag & drop
Like
Use or download
Copy short link
This component copies column values from preceding rows into the current row in a Spark DataFrame/RDD. The component can be used to - make a copy of the selected column and shift the cells I steps up (I = lag interval) - make L copies of the selected column and shift the cells of each copy I, 2I, 3I, ... L*I steps up (L = lag) Required extensions: KNIME Data Generation (https://hub.knime.com/knime/extensions/org.knime.features.datageneration/latest) KNIME Extension for Apache Spark (https://hub.knime.com/knime/extensions/org.knime.features.bigdata.spark/latest) KNIME Quick Forms (https://hub.knime.com/knime/extensions/org.knime.features.js.quickforms/latest)

Component details

Input ports
  1. Type: Spark Data
    Spark Input data
    Spark DataFrame containing at least the value column and a numeric or timestamp column for sorting purpose.
Output ports
  1. Type: Spark Data
    Lag input table
    Spark DataFrame with input data and additional columns copying the values from preceding rows.

Used extensions & nodes

Created with KNIME Analytics Platform version 4.1.0
  • KNIME Core Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

  • KNIME Data Generation Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

  • KNIME Extension for Apache Spark Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

  • KNIME Math Expression (JEP) Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0

  • KNIME Quick Forms Trusted extension

    KNIME AG, Zurich, Switzerland

    Version 4.1.0