Hi everyone,
I am currently trying to use Data Integration Services to load files from a bucket location.
However, I am facing quite a few challenges with the possible scenarios.
In my bucket location I need to load only the most recent file present in a particular location.
The current approach I am trying to make is using the data asset with a pattern that should capture all files present, then I perform a desceding sorting by the date, assign a row_number and filter only the 1 result…
Still not sure this is actually working fine, but already facing a situation that when there are no files captured by the pattern the process fails and halts the whole thing.
How to actually handle the errors with this tool? And also how to properly debug each step.
Thank you