Problem Description: We are trying to integrate Oracle Fusion (SaaS) data with our data lake on AWS S3. Our primary use case is to extract data from the SCM module. However, we are encountering the following challenges:
- Hard-Deleted Data: Some data in Oracle Fusion is hard deleted by users, making it impossible to capture changes using standard APIs or reports.
- Raw Data Extraction: We need to extract raw data tables with a 1:1 mapping format, similar to the underlying tables in Oracle Fusion.
- Change Data Capture (CDC): We require a solution to enable CDC or access to historical/deleted data for incremental processing.
Our goal is to set up an efficient pipeline where raw data from Oracle Fusion can be loaded into S3 for further processing and analytics. We need guidance on:
- Accessing raw data tables in Oracle Fusion SaaS.
- Overcoming the limitations caused by hard deletions.
- Implementing a reliable data extraction method compatible with AWS S3.