Skip to Main Content
Forums

Analytics Software

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

How can we execute Spark Streaming in ODI 12c

sinan_ggJul 22 2018

Hi All,

I m trying to use Kafka-Spark Streaming technologies in ODI12c to transferring data instantly with Kafka => Spark Streaming => Hive. I have succesfully configured Kafka and Spark Server Physical and Logical Architecture on topology side. I m working on Oracle BigdataLite 4.11 virtual machine.

On ODI mapping side, i m using following properties, and also i selected the staging location as Spark Async and enable "Streaming".

  • SOURCE_GROUP: Kafka topic
  • TRANS_GROUP: Spark Async Server
  • TARGET_GROUP: Hive table

Mapping.PNG

When i run this mapping, i have an following error.

ODI-1590: The execution of the script failed.

ODIKM-SPARK-ASYNC-20000: Could not detect Spark Live REST API connection details after 12 retries. Values detected: appId:"application_1532283273790_0011" appName:"Kafka_Spark_Hive_Physical" liveBaseUrl:"null" historyBaseUrl:"/user/spark/applicationHistory" yarnUrl:"{5}" yarnAppId:"{6}"

When i run this mapping with Spark Sync execution mode, and enable "Streaming", it has not been generated streaming code. So ODI step is running successfully and finished.

But my main purpose is running spark streaming jobs on ODI. Please let me know how to use spark?

Thanks,

Sinan,
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Aug 19 2018
Added on Jul 22 2018
#business-intelligence-applications, #data-integrator, #odi, #system-management-and-integration
0 comments
152 views