TextKleaner

This workflow is designed to help you prepare a textual dataset for a bag-of-words style computational analysis. It assumes that you already have your data in a tabular form - that is, a CSV or KNIME table containing a column of plain text documents along with metadata columns.

The workflow performs four types of operations to prepare your text for analysis. First, it scrubs your text, removing or replacing various characters to ensure that the text is formatted cleanly and consistently. Second, the workflow provides various ways to find and exclude documents that are irrelevant to your study. Third, it helps you to find and remove duplicated text, both in the form of highly similar documents and 'boilerplating' that is repeated at the start of documents. Finally, it allows you to enrich your data by tagging names and ngrams, and to refine your data by filtering out terms that are rare or uninformative, and by standardising plurals and other word variants.

While the loading and scrubbing of texts must be performed first, there is some flexibility around the remaining steps. The filtering operations in Step 2 are entirely optional, and can be performed in any order, although there are benefits to detecting duplicates before filtering documents by topic. Duplicate detection and boilerplate removal (Step 3) are also optional, but are highly recommended if you plan to tag ngrams in Step 4 or use topic modelling in your analysis. Duplicate detection should be performed after document filtering, but boilerplate removal can be performed at any stage before Step 4, and indeed may improve the results of the 'Filter by topic' operation. Tagging and filtering (Step 4) must be run last, as it will convert your documents from plain text strings into a tokenised format for subsequent analysis.

Except for duplicate detection (which saves information in a separate table), each operation in Steps 2 and 3 will overwrite the input data with the filtered data. The excluded documents are saved in a separate file, and can be reviewed or restored at any stage.

External resources

• TextKleaner - a Knime workflow for preparing large text datasets for analysis