GoldenGate

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Elasticsearch 5.X not work with OGG-BD.

3508998Mar 2 2018 — edited Mar 13 2018

Hello folks. i need to run OGG-BD elasticsearch handler elasticsearch version 5.x. (5.1.2 with xpack)

i have successfully tested with elasticsearch handler 2x on ES v2.4.5 and Oracle Goldengate big data version 12.3.0.1 following this guide. https://www.cnblogs.com/margiex/p/8192673.html

But for elasticsearch 5.x, i can't run replicat process.

is there any mistake on my config? Also any good tutorial for

and my configs is

*****************************************************

cat /home/oracle/.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

# User specific environment and startup programs

#PATH=$PATH:$HOME/.local/bin:$HOME/bin

#export PATH

export JAVA_HOME=/opt/jdk

export PATH=$JAVA_HOME/bin:$PATH

export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:$LD_LIBRARY_PATH

*********************************************************************************************************

pwd

/u01/app/oracle/ogg-bd/dirprm

******************************************

cat client5x.properties

cluster.name=elasticsearch

xpack.security.user=elastic:changeme

****************************************

cat elasticsearch5x.props

gg.handlerlist=elasticsearch

gg.handler.elasticsearch.type=elasticsearch

## Handler properties for Elasticsearch 5.x

gg.handler.elasticsearch.ServerAddressList=localhost:9300

gg.handler.elasticsearch.clientSettingsFile=client5x.properties

gg.handler.elasticsearch.version=5.x

#gg.handler.elasticsearch.bulkWrite=true

gg.classpath=/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/plugins/x-pack*:/usr/share/elasticsearch/modules/transport-netty4/*:/usr/share/elasticsearch/modules/reindex/*

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm

*********************************************************************************************************

cat res.prm

***********************************************************************************************

REPLICAT res

TARGETDB LIBFILE libggjava.so SET property=dirprm/elasticsearch5x.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 1000

MAP QASOURCE.*, TARGET QASOURCE.*;

************************************************************************************************

Here is my output log for ggserror.txt

2018-03-02T00:45:44.795-0500 INFO OGG-00987 Oracle GoldenGate Command Interpreter: GGSCI command (oracle): start res.

2018-03-02T00:45:44.799-0500 INFO OGG-00963 Oracle GoldenGate Manager, mgr.prm: Command received from GGSCI on host [FE80::F39C:D6D8:BEC5:C89C]:23438 (START REPLICAT RES ).

2018-03-02T00:45:44.799-0500 INFO OGG-00975 Oracle GoldenGate Manager, mgr.prm: REPLICAT RES starting.

2018-03-02T00:45:44.829-0500 INFO OGG-00995 Oracle GoldenGate Delivery, res.prm: REPLICAT RES starting.

2018-03-02T00:45:44.829-0500 INFO OGG-03059 Oracle GoldenGate Delivery, res.prm: Operating system character set identified as UTF-8.

2018-03-02T00:45:44.829-0500 INFO OGG-02695 Oracle GoldenGate Delivery, res.prm: ANSI SQL parameter syntax is used for parameter parsing.

2018-03-02T00:45:45.037-0500 INFO OGG-15052 Oracle GoldenGate Delivery, res.prm: Using Java class path: .:ggjava/ggjava.jar:./dirprm:ggjava/resources/lib/optional/log4j-api-2.9.0.jar:ggjava/resources/lib/optional/log4j-core-2.9.0.jar:ggjava/resources/lib/optional/log4j-slf4j-impl-2.9.0.jar.

2018-03-02T00:45:47.472-0500 WARNING OGG-00869 Oracle GoldenGate Delivery, res.prm: java.lang.ClassNotFoundException: org.elasticsearch.xpack.client.PreBuiltXPackTransportClient.

2018-03-02T00:45:47.472-0500 WARNING OGG-00869 Oracle GoldenGate Delivery, res.prm: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient.

2018-03-02T00:45:47.472-0500 WARNING OGG-00869 Oracle GoldenGate Delivery, res.prm: org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient.

2018-03-02T00:45:47.472-0500 WARNING OGG-00869 Oracle GoldenGate Delivery, res.prm: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userExitDataSource' defined in class path resource [oracle/goldengate/datasource/DataSource-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient.

2018-03-02T00:45:47.473-0500 ERROR OGG-15051 Oracle GoldenGate Delivery, res.prm: Java or JNI exception:

oracle.goldengate.util.GGException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userExitDataSource' defined in class path resource [oracle/goldengate/datasource/DataSource-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userExitDataSource' defined in class path resource [oracle/goldengate/datasource/DataSource-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at oracle.goldengate.datasource.DataSourceLauncher.<init>(DataSourceLauncher.java:161)

at oracle.goldengate.datasource.UserExitMain.main(UserExitMain.java:108)

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)

... 11 more

Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264).

2018-03-02T00:45:47.473-0500 ERROR OGG-01668 Oracle GoldenGate Delivery, res.prm: PROCESS ABENDING.

cat dirrpt/RES.log

*****************************************************************************************************

INFO 2018-03-02 00:07:58.000643 [main] - Starting Java user-exit

INFO 2018-03-02 00:07:59.000755 [main] - Loading configuration from properties file: GGS_USEREXIT_CONF='dirprm/elasticsearch5x.props' (Java system property (set to file path))

INFO 2018-03-02 00:07:59.000755 [main] - Adding to classpath, gg.dirprm=./dirprm (GoldenGate Parameter Directory)

INFO 2018-03-02 00:07:59.000756 [main] - Adding to classpath, gg.home=. (GoldenGate Installation Directory)

INFO 2018-03-02 00:07:59.000756 [main] - Adding to classpath, gg.classpath=/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/modules/transport-netty4/*:/usr/share/elasticsearch/modules/reindex/* (Additional runtime classpath setting)

INFO 2018-03-02 00:07:59.000870 [main] -

***********************************************************************

Oracle GoldenGate for Big Data, 12.3.1.1.2.005

Built with Java 1.8.0_131 (class version: 52.0)

***********************************************************************

INFO 2018-03-02 00:08:00.000168 [main] - Memory at Initialization : Max: 494.94 MB, Total: 30.94 MB, Free: 17.33 MB, Used: 13.61 MB

INFO 2018-03-02 00:08:00.000232 [main] - Setting handler=elasticsearch: retry policy defaults. Startup policy: Retry policy: enableRetry=true, numRetry=0, exponentialDelay=no, retryDelayMS=0; Runtime policy:Retry policy: enableRetry=true, numRetry=0, exponentialDelay=no, retryDelayMS=0

ERROR 2018-03-02 00:08:00.000238 [main] - org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userExitDataSource' defined in class path resource [oracle/goldengate/datasource/DataSource-context.xml]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [oracle.goldengate.datasource.GGDataSource]: Factory method 'getDataSource' threw exception; nested exception is java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at oracle.goldengate.datasource.DataSourceLauncher.<init>(DataSourceLauncher.java:161)

at oracle.goldengate.datasource.UserExitMain.main(UserExitMain.java:108)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)

... 11 more

Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at oracle.goldengate.handler.elasticsearch.util.ElasticSearchFactory.getInstance(ElasticSearchFactory.java:39)

at oracle.goldengate.handler.elasticsearch.ElasticSearchHandler.establishConnection(ElasticSearchHandler.java:103)

at oracle.goldengate.handler.elasticsearch.ElasticSearchHandler.init(ElasticSearchHandler.java:42)

at oracle.goldengate.datasource.AbstractDataSource.addDataSourceListener(AbstractDataSource.java:591)

at oracle.goldengate.datasource.factory.DataSourceFactory.getDataSource(DataSourceFactory.java:161)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)

... 12 more

Caused by: java.lang.ClassNotFoundException: org.elasticsearch.xpack.client.PreBuiltXPackTransportClient

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 24 more

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1123) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1018) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1051) ~[spring-context-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at oracle.goldengate.datasource.DataSourceLauncher.<init>(DataSourceLauncher.java:161) ~[ggdbutil-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at oracle.goldengate.datasource.UserExitMain.main(UserExitMain.java:108) [ggcmdui-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

... 11 more

Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/xpack/client/PreBuiltXPackTransportClient

at java.lang.Class.forName0(Native Method) ~[?:1.8.0_162]

at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_162]

at oracle.goldengate.handler.elasticsearch.util.ElasticSearchFactory.getInstance(ElasticSearchFactory.java:39) ~[ggelasticsearch-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at oracle.goldengate.handler.elasticsearch.ElasticSearchHandler.establishConnection(ElasticSearchHandler.java:103) ~[ggelasticsearch-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at oracle.goldengate.handler.elasticsearch.ElasticSearchHandler.init(ElasticSearchHandler.java:42) ~[ggelasticsearch-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at oracle.goldengate.datasource.AbstractDataSource.addDataSourceListener(AbstractDataSource.java:591) ~[ggdbutil-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at oracle.goldengate.datasource.factory.DataSourceFactory.getDataSource(DataSourceFactory.java:161) ~[ggdbutil-12.3.1.1.2.005.jar:12.3.1.1.2.005]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.2.3.RELEASE.jar:4.2.3.RELEASE]

... 11 more

Caused by: java.lang.ClassNotFoundException: org.elasticsearch.xpack.client.PreBuiltXPackTransportClient

at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_162]

at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_162]

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) ~[?:1.8.0_162]

at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_162]