Data Migration Use Case and Data flow

USE CASE - Data Migration using ADF (Azure Data Factory)

Client Existing system is running in DB2 Database. Current application use DB2 database for storing data related to their application. Apart from this, current application also fetches few master information from ORACLE DB to complete their own application transactions. Once all the transactions completed few data moving to ORACLE back again, from where few more application uses those data generated by this application.

ORACLE server also receives few data from different system, that continually feeding data for other applications.

OLD System	New System	Description
Main Frame	Cloud Web Application
DB2	Azure SQL Server
ORACLE	ORACLE	This works as centralized server uses by many other applications.

Data Model: New Data Model on Azure SQL Server is completely Re-Design and Normalized form.

Data Migration: All data migration using ADF (Azure Data Factory).

DB2 to Azure SQL Server (One-time migration).

• DB2 team generate .txt files through JOB and store into Azure BLOB storage.
• ADF pipeline read those .txt file and store into Azure SQL Server Stage tables.
• Using Azure SQL Server Store Procedure (Internal processing) all the data moved from Stage to Main application table after transformation. Where new WEB application consumes those data.

ORACLE to Azure SQL Server (One-time migration and then INCREMENTAL Load)

• First time we load all the data whatever required by the new WEB application as One-time migration from ORACLE to Azure SQL Server Stage table called UPStage.
• After UPStage table got populated, Azure SQL start Internal processing for those records and update main application tables.
• Next time onward ADF will fetch only DELTA records from ORACLE and update Azure SQL Server UPStage tables continuously.
• All DELTA fetching logic maintained in SQL control table.

Azure SQL Server to ORACLE (INCREMENTAL Load)

• Azure SQL server push updated records to ORACLE back again.
• First Azure SQL Server populate few Down Stage tables in Azure Itself, after processing the updated records by new WEB application.
• Now once all the Down Stage table got populated then ADF will PUSH those tables into ORACLE DB.
• Then Based on these Stage tables, ORACLE will update their main tables

In next post we will describe the above ADF (Azure Data Factory) flow of different activities to complete the data migration.