Amazon S3 Remote File Example

In order to connect to the service, Amazon S3 credentials are required.

Folders created on root level are created as buckets on S3. The virtual file system functionality is fully supported by the file handling nodes.
In the first step a bucket and a directory inside that bucket is created. After that a file is uploaded to the directory inside the bucket (path: /examplebucket***/exampledirectory). Finally all the files and folders inside the example bucket are listed. The result is a list of URIs which can be used to download files/directories.

In the second step we download the iris.csv which we uploaded in the step before. This saves the file to disk; this file can then be read using a csv reader. Since it might be unnecessary to cache the file to disk, the Amazon S3 File Picker provides the functionality to create URLs from which files can be read directly. This works with all file readeres that support reading files from a URL. These URLs have an expiration date which can be set in the dialog.

Stream reading also becomes useful in conjunction with the KNIME Streaming Executor as the (possibly large) data is read from the remote side and is immediately processed without being downloaded to the local machine.

After having seen how we upload and download files and how we use the File Picker to read files directly we can delete our example bucket from S3.