Mapping Details
Last updated
Last updated
You can access the more granular mapping details by clicking on the hyperlink of the mapping name or by clicking on the expand/collapse arrow to the right of it.
All mapping details will be revealed after clicking either the mapping name or the screen expansion arrow.
Kodda splits the editing of various options into different related sets based upon how often data pipeline users change the various options.
Load Type lets you select one of five distinct load types, controlling the data ingest process.
Append New Records Only - Kodda identifies the records that do not yet exist in the target table and adds them based on the identified business key. If a record already exists it is ignored.
Append Only reads the source data and appends it onto the target data, without removing or updating any target data.
Full Refresh removes all old data from the target table and then does a full load from the source. This is the default load type for new maps generated. Removal is done via SQL Truncate for systems that support Truncate otherwise SQL Delete is used. For files, the entire file is replaced.
Incremental Change Tracking leverages change tracking when enabled on SQL Server to detect newly inserted, updated or deleted records. Click the link for an example of how to enable change tracking.
Incremental Insert - This load pattern allows you to select a column to identify new or changed records from the source. A filter on this column is applied on each extract from the source system to limit the number of records read. This is useful for identifying changed records. This requires an incremental column or expression to be defined in order to drive the incremental loads.
Incremental Insert/Update performs inserts and/or updates of source data changes to the target on each mapping execution. This requires an incremental column or expression to be defined in order to drive the incremental loads. This requires a business key to be defined in the mapping.
Insert/Update extracts all records from the source and merges them to the target using the supplied primary key. Incremental Append is useful for loading data into unstructured targets, in which you want to only append new data from the source into the target. This requires an incremental column or expression to be defined in order to drive the incremental loads.
Insert/Update/Delete (Logical) - Kodda will extract all records from the source system and will insert new records, update existing records and set the si_del_flg column to Y when a record no longer exists in the source data set. This load pattern results in a column (si_del_flg) being added to the target table to identify logically deleted records.
Incremental Append w/View, similar to Incremental Append, this only appends new data, but it also creates logical views based on the defined primary key that allows you to query the "current" values. This requires an incremental column or expression to be defined in order to drive the incremental loads and a business key to be defined in the mapping.
Incremental Column: A column that can be used to identify newly created or changed records. Ideally this column is a date or integer type.
Incremental Operator: The comparison operator used when extracting data from the source. This value can be either > (default) or >=.
Next Incremental Value: The maximum value that was identified the last time the mapping ran. This is the value that will be used on the next run to identify new or changed records.
Expression: A userdefined SQL expression that can be used as the incremental column. This field is available when the Incremental Column value is set to <CustomExpression>.
In order to improve query performance you may use the expression $incremental(<column_name>)$ within the query of a map to control the placement of the generated incremental load filter. This can be helpful with certain database optimizers to ensure the incremental filter is pushed down to the lowest possible point of the query.
Data Extractor is a feature which lets you optimize the data extraction from your source system, by selecting how many extractors you want to run in parallel against your source. You then also set the split column(s) used to derive the parallel data sets to extract and load.
Number of Extractors: Based upon your source system's capacity, you can set the number of extractors as allowed in your configuration. This will indicate to the platform that it should sample the data and determine a proper number of rows to extract for each of the extractors you enable. The Split Column(s) will be used to derive these data sets processed by each extractor. The current limit for Number of Extractors is between 1 and 8.
Split Column: To use multiple Extractors, you must select which INTEGER column in your source data to use to divide the data into a relatively even split between extractors. If you do not have an integer column in your source table you can supply a custom expression that can be used that produces an integer.
<CustomExpression>: Allows you set the custom expression in order to split the data across multiple extractors. The expression must result in an integer, produce the same result for each row when selecting a single or multiple rows from the table and be valid for use in a "Where Clause" of a query executed in the selected source system. It is highly recommended that the column or expression results in uniformly distributed data across the entire data set being extracted.
Run-time Options provide a series of parameters that allow you to control mapping execution performance and behavior, including:
Batch Size lets you control the number of rows to insert into the target between commit operations.
Disable Audit Cols disables the use of built-in audit columns in the mappings, This is identical to the Disable Audit Columns option described on the Generate Mapping page.
Recreate Target drops and recreates the target table when you run this mapping. Only use this for full refresh operations on large tables, or you may opt to use this for small 100 row or less tables each and every time you run a mapping for those small tables, if you believe that the structure of these table(s) may be changing often.
Truncate Target truncates the target table when you run this mapping, before reloading it with a full refresh. This will negate any incremental logic, so only use it for smaller tables, or larger tables that, for some reason, you need to truncate before loading.
Disable Target DDL prevents DDL statements from running on the target table when the mapping is run. See Disable Target DDL on the Generate Mapping page.
Detect Target Changes enables auto-detection of target table changes in the map. This can be used in conjunction with Disable Target DDL to ensure delivery of data when the target structure differs from the source. See Detect Target Changes on the Generate Mapping page.
Detect Source Changes enables auto-detection of source schema changes in the map. This allows a map to detect when a source schema has changed from what has been defined in the existing mapping, allowing you to run the mapping and automatically add new columns from the source, or ignore columns that no longer exist in the source. See Detect Source Target Changes on the Generate Mapping page.
Source Connection Name allows you to change the source connection to another database or data set name.
Target Connection Name allows you to change the target connection to another database or data set name.
You can edit all mapping level settings in the top center pane of the Edit Mapping screen, by clicking the edit icon .
Click either save or the cancel icon for each section separately.
Map Name allows you to change the default name to your own convention. Click the undo icon to revert back to the original default name.
