Import set performance

An algorithm is used to transform import sets from their staging table into their final destination.

Importing data via an import set requires the following two phases.
  1. The data is loaded from a data source into a staging table.
  2. The data is transformed from the staging table into a target table.

The transform algorithm operates in blocks of 100 records at a time. This approach allows the application server to pre-fetch a variety of information relevant to each block of records. This action reduces the number of unique interactions with the database and improves throughput.

Performance is improved for all large transformations are using this algorithm. It also improves transformations with many reference or choice type columns. Transformations that with complex or unkeyed coalesce conditions experience a proportionately smaller benefit from the algorithm.