SAA Consultancy's Data Flow Rule Engine is responsible for transferring data from one source to another. Each "Job" is processed by the Rule Engine, and simultaneously, all actions performed in the user interface (UI) are logged in the Rule Engine. This logging helps maintain a record of all data-related activities.
The Rule Engine consists of the following services
-
Full Load Services
This service runs continuously 24/7, ready to pick up any data transfer request. Once it receives a request, it begins moving data based on the configured settings. The initial data transfer involves a complete data dump from the source to the warehouse. Delta encoding, a method for storing or transmitting data as differences between sequential data sets, is used to reduce data redundancy. During incremental loads, the changes between the target and source data are transferred at regular intervals. The last extraction date is stored to ensure that only records added after this date are loaded.
-
CDC Services (Change Data Capture)
In databases, change data capture (CDC) is a set of software design patterns used to identify and track changes in data, allowing actions to be taken using the updated data.
-
Apply Data Change
The Apply Data Change stage is a processing stage that takes the change data set, containing changes in the before and after data sets, from the Change Capture stage. It applies the encoded change operations to the before data set to compute an after data set.
-
Custom Load Change
Custom services use custom techniques or user-defined columns to detect changes in the database and apply those changes. Currently, this service is employed for custom delta detection in the Cerner Patient Information System.
-
Execution Log and Environment Monitoring Components
The execution log includes components for monitoring server health, providing information about the various servers involved in data movement. The information captured by this service is displayed on a dashboard in the frontend.