Change Data Capture¶
Change Data Capture (CDC) in IFS Cloud refers to the capability of identifying and handling data changes such as inserts, updates, and deletions, so that only the delta (changed data) is processed or shared, instead of repeatedly transferring full data sets. This approach is essential for modern, event‑driven, and integration‑heavy enterprise landscapes. The mechanism of CDC will replicate data changes from the IFS Cloud Oracle database into the tenant’s own data lake in the IFS.ai platform.
Why Change Data Capture Matters in IFS Cloud¶
In enterprise environments, business data is constantly evolving. For example, orders change, schedules update, and inventory levels fluctuate. Reprocessing full data loads every time is inefficient and slow.
CDC in IFS Cloud enables:
- Efficient integrations by sending only changed records
- Near real‑time change data capture to downstream systems
- Improved performance by avoiding full data reloads
- Accurate synchronization between IFS Cloud and the tenant-specific data lake in the IFS.ai platform
Identifying Tables for Change Data Capture (CDC)¶
There are two primary sources used to identify the tables for which CDC needs to be enabled:
- Tables associated with Data Products created in Data Services Foundation (DSF).
- Tables selected directly by users from IFS Cloud.
The following sections describe how tables are identified and configured for CDC for both sources. This process involves interactions between the CDC Watcher Service and the Table Management Service within the DSF platform, which work together to determine tenant entitlements and the corresponding tables eligible for CDC.
Tables Linked to Data Products¶
IFS Data Product developers (currently internal IFS RnD) can create multiple Data Products within a specific product area or domain. Each Data Product can be linked to one or more tables depending on the use case for which the Data Product is created.
When a tenant obtains entitlement to use specific Data Products, background services in DSF retrieve the tenant’s capability information associated with the Tenant ID from LEC. This process is handled by the Table Maintenance Service.
One of the key responsibilities of the Table Maintenance Service is to identify the list of tables associated with the Data Products that the tenant is entitled to use. These tables are then considered eligible for CDC configuration.
Tables Selected from IFS Cloud¶
In addition to Data Product–based tables, tenants may also be granted the capability to manually select tables from IFS Cloud for CDC. This capability is independent of Data Product entitlements.
The Table Maintenance Service verifies whether the tenant has the required capability to select tables directly from IFS Cloud. If the capability is enabled, users will be provided with the option to select tables for CDC through the IFS Cloud interface.
Note
A maximum of 100 tables can be selected. Thus, ensure that only the required tables are selected.
CDC Configuration Process¶
Once the table entitlement information is gathered by the Table Maintenance Service, the CDC Watcher Service retrieves this information and initiates the required configurations to enable CDC for the identified tables.
These configurations include:
- Updating system tables such as:
CDC_SUP_LOG_ENABLED_TABLE_RTBCDC_EXCLUDED_COLUMN_RTB- Snapshot load triggering tables
- Enabling Supplementary Logging on the identified tables
- Configuring the Debezium server/container to capture database changes
These steps ensure that changes occurring in the database tables are captured and made available for downstream data processing.
Note
The Debezium server is configured to restart every 30 minutes to detect changes in the CDC table constitution. Therefore, when a new table is enabled for CDC, the capture process may not start immediately. In some cases, there may be a delay of up to 30 minutes before the server detects the newly added table. Once the Debezium server restarts and identifies the table, it will begin capturing changes immediately. However, the delay is not always exactly 30 minutes. If a new table is added shortly before the scheduled restart (for example, one minute before), the Debezium server will detect it during that restart and begin capturing changes almost immediately.
Related IFS Cloud Components¶
The primary product component responsible for enabling CDC in IFS Cloud is the CDCFW component.
Additionally, the CDCESG component related to the Data Product released in this release is required, as it it contains the workload jobs necessary for the Data Product.
Note
For any table with CDC enabled, regardless of how the table was identified for CDC enablement, Oracle LOB-type columns (such as CLOB or BLOB) are not captured through CDC. If such columns exist in a table, they will be stored and exposed through a separate table structure.
IFS Cloud Web Interface for Viewing CDC Enabled Tables¶
To view tables that have CDC enabled, or to select additional tables for CDC, navigate to the CDC Enabled Tables page in IFS Cloud.
The fields displayed on this page are explained in the table below.
| Field | Explanation |
|---|---|
| Component | A component in IFS Cloud is a modular business unit that encapsulates related functionality, data, and APIs for a specific business domain. |
| Table | A table in IFS Cloud is a component-owned database structure used to store business data and accessed exclusively through validated business logic. |
| Included in a Data Product | This indicates whether the selected table is included in a Data Product or not. |
| Is enabled | This indicates whether CDC is currently enabled for the selected table or not. |
| Error Message | Displays the error message associated with any issues that occur during the CDC enablement process. |
Note
To access this page in IFS Cloud, users need to have the CDC_ADMINISTRATOR permission set assigned to them.
Adding New Tables to Enable CDC¶
As mentioned above, in addition to tables included in Data Products, users can enable CDC for additional tables from IFS Cloud, provided that the tenant has the required capability entitlement.
When this entitlement is available, the Enable CDC for Tables option will appear on the page. Follow the steps below to select additional tables for CDC.
Step 1: Launch the Table Selection Assistant
Click Enable CDC for Tables to open the Table Selection for Change Data Capture assistant.
Step 2: Select a Component
Within the assistant, select the required Component from the list of values.
Step 3: Select Tables
Once a component is selected, the tables associated with that component will be displayed. Select one or more tables that you want to enable for CDC.
Step 4: Confirm the Selection
Click OK to complete the table selection process and enable CDC for the selected tables.
Additional Options Available for Manually Selected Tables¶
For tables that are manually selected for CDC, several additional configuration options are available. These options allow users to manage CDC behavior and perform specific actions on the selected tables.
1. Disabling CDC¶
CDC can be disabled only for tables that were manually enabled by the user. Tables that are part of Data Products cannot be disabled.
To disable CDC, select the table that should no longer participate in CDC synchronization and click Disable.
When this action is performed:
- The selected table is removed from Change Data Capture tracking.
- Supplementary logging for the table is revoked at the database level.
The Disable CDC action can be applied to one or multiple tables at the same time. Once disabled, the tables will be removed from the CDC Enabled Tables list.
Note
If your entitlement associated with the Data Product capability is removed or no longer exists, the tables linked to that Data Product will automatically be removed from the CDC enabled table list.
2. Column Selection¶
The Column Selection assistant displays all columns included for the selected table.
By default, CLOB and BLOB type columns are excluded. From this interface, users can optionally exclude additional columns from the CDC process.
3. Trigger Snapshot Load¶
When a table is enabled for CDC, the initial synchronization always performs a full load of the table. After the initial load, subsequent data changes are captured through CDC according to the configured synchronization settings.
If a new full snapshot of the table is required at a later stage, it can be triggered using the Trigger Snapshot Load option.
When this option is selected:
- A confirmation prompt will appear.
- Selecting OK initiates a full snapshot load for the selected table.
Note
Currently, you can only select one table at a time to initiate a snapshot load.
Workload Jobs Related to the CDC Flow¶
The following workload jobs are directly or indirectly related to the Change Data Capture (CDC) data flow in DSF. These jobs process CDC-generated data, maintain supporting datasets, and refresh analytical structures used by downstream services such as dashboards and transformations.
| Workload Job | Component | Description | When to Run |
|---|---|---|---|
| RAW_TO_MIRROR_WORKLOAD | CDCFW | Once the CDC-enabled tables are available in the per-tenant Data Lake in the IFS.ai platform, this workload transforms the Raw CDC tables generated by the CDC ingestion pipeline into Mirror tables in the Data Lake. Raw tables contain row-level change events captured from the Oracle source database. The Mirror layer processes these change events and reconstructs the latest state of each table, making the data easier to consume for analytics and downstream workloads. The job uses metadata about CDC-enabled tables, primary keys, and table structures to correctly merge and update records in the Mirror tables. | Run this workload after enabling CDC for new tables, especially when tables are added manually, so the newly enabled tables are included in the Raw-to-Mirror transformation pipeline. The scheduled time for this workload job can be adjusted according to your requirements. |
| ENUMERATION_WORKLOAD | CDCFW | This workload job loads and maintains the enumeration value dataset used in DSF. Enumeration datasets contain code and display value pairs defined in IFS Cloud. The workload processes enumeration metadata and creates a dataset in the Data Lake so that these mappings can be used by analytics, data transformations, and queries that require human-readable values instead of internal codes. | This workload is delivered as part of the solution package and does not normally need to be executed during initial setup. It should be triggered after a Service Update or when enumeration values are updated, to refresh the enumeration dataset. |
Read more about workload jobs->Workload Job Definitions