I have many on-prem Oracle databases, and we are building a Data Lake/Warehouse in ADW.
For the Data Ingestion portion of our architecture, we would like to read from those sources using a CDC-based approach. For the target, since it's ADW in OCI, more of a microbatch approach seems correct. That is, individual inserts/updates from the source are not sent to ADW. Instead, every 20 minutes, something wakes up, and sends them up to OCI object storage, and then a DBMS_CLOUD package does the INSERT or MERGE operation.
However, my understanding from tech support is that GoldenGate OCI does not support this. It just passes along each insert/update, which is not ideal from a cloud db loading pattern perspective. They said we could essentially write our own process (where the GG data is written out to files, and then we do the legwork of getting them loaded into ADW from there), but that is also not ideal.
We also want a code-free solution. Point and click, or minimal configuration. Don't want to have to set up complex ETL flows with one for each of the hundreds of tables we need to ingest.
Most of the other data ingestion tools out there that I looked at – Infa Mass Ingestion, FiveTran, CData Sync, Stitch, Rivery – all support this pattern, but only if you're using one of the more major players like Redshift or Snowflake as the target.
So, given the needs:
- CDC Source
- Microbatch target
- Low-code solution
Does Oracle have other options in this space that fit those needs? ODI? Data Loader tasks? Something else?
Thank you for your advice! :-)