Import CSV - This workflow will act as a meta workflow trying to answer questions about importing data from CSV and Text files

Import CSV - This workflow will act as a meta workflow trying to answer questions about importing data from CSV and Text files

One of the most frequently asked questions is the one how to import data from CSV and Text files under various circumsatnces. Often these files are messy with unbalanced quotes and column separators that would also appear in the data. Best advise would be to avoid CSV at all and resort to formats the would preserve the column types and formats like Parquet, ORC, ARFF or even H2 or SQLite - but often this is not possible

The (new) CSV Reader should be the first stop
https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.io.filehandling.csv.reader.CSVTableReaderNodeFactory
=> please also note that you can configure a lot of settings like "Support changing file schemas" if your CSV structure does change or force data types
https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.io.filehandling.csv.reader.CSVTableReaderNodeFactory#relatedworkflowsnodes

use R to read and write custom CSV files and read them back with different nodes
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_r_csv_read_write
=> also on the right side a collection of several nodes to import CSV and TXT files

Force all columns to be read as strings (use KNIME nodes only)
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_csv_import_force_all_string_pure_knime

Demonstrates how to import several CSV files with the new CSV reader from sub folders at once
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_csv_import_multiple_csv_types

-------------
Often if everything else fails R's "Readr" package might still be able to help you

use R library(readr) to read (messy) CSV file into KNIME
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_r_read_single_csv_file
=> also export files to ARFF, Parquet and SQLite (with and without R) to preserve column types

import 'dirty' CSV files with the help of R's readr
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_r_import_csv_r_readr

Force R’s readr to import all columns of a messy CSV file as strings - allow changing table structures
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_r_import_csv_messy_loop_force_all_string

Import CSV with non-printable UNICODE characters
https://forum.knime.com/t/csv-writer-problem/28064/9?u=mlauber71
https://hub.knime.com/mlauber71/spaces/Public/latest/forum/kn_forum_28064_r_import_csv_r_readr_strange_charaters

-------------
CSV's can be used to append data to an existing file without having to reload it again

Download data from a database in chunks and append them to an CSV file
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_write_csv_loop_chunks/m_001_write_csv_loop_chunks

-------------
You can also move CSV files around

copy files to two target folders using local URI strings - the basic version
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_move_csv_files_basic

copy files to two target folders using local URI strings
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_move_csv_files

-------------

And you often combine CSV files with other formats like MS Access, Excel and Big Data systems

Export huge MS sql table to csv without fetching all records together
https://hub.knime.com/mlauber71/spaces/Public/latest/forum/kn_example_huge_db_to_csv/m_001_write_csv_41

CSV files from subfolders to xlsx sheets
https://hub.knime.com/mlauber71/spaces/Public/latest/forum/kn_example_csv_to_excel/m_001_import

KNIME and Hive - load multiple CSV files at once via external table
https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_bigdata_hive_csv_loader/m_001_import_hive_csv

External resources

• KNIME and Hive - load multiple CSV files at once via external table
• CSV files from subfolders to xlsx sheets
• Export huge MS sql table to csv without fetching all records together
• copy files to two target folders using local URI strings
• copy files to two target folders using local URI strings - the basic version
• Download data from a database in chunks and append them to an CSV file
• Import CSV with non-printable UNICODE characters
• Force R’s readr to import all columns of a messy CSV file as strings - allow changing table structures
• import 'dirty' CSV files with the help of R's readr
• use R library(readr) to read (messy) CSV file into KNIME
• Demonstrates how to import several CSV files with the new CSV reader from sub folders at once
• Force all columns to be read as strings (use KNIME nodes only)
• use R to read and write custom CSV files and read them back with different nodes
• The (new) CSV Reader should be the first stop