The Text Preprocessing component uses extremely fast regex-based text processing to remove specific types of characters from a String column and normalize the data as much as possible without over-processing.
This component eliminates the need to convert text to a Document type in order to preprocess it. It also executes extremely quickly compared to various other approaches, promoting scalability.
Each option in the configuration is processed independently of the others. There is an order of operations (which can be audited or edited by drilling into the component).
- Type: TableInput TableA table of input data with at least one String column.