Hi, I'm getting some trouble with capturing data change from Cassandra (successful initial loading before). Extract process works but it does not capture any data from Cassandra. I configure param for extract process as below:
EXTRACT capcass
JVMCLASSPATH ggjava/ggjava.jar:DependencyDownloader/dependencies/cassandra_libs/
*
TRANLOGOPTIONS SFTP KNOWNHOSTSFILE /home/manhnt/known_hosts
TRANLOGOPTIONS CDCREADERSDKVERSION 3.11
TRANLOGOPTIONS CDCLOGDIRTEMPLATE /var/lib/cassandra/cdc_raw
SOURCEDB USERIDALIAS cass
EXTTRAIL /home/manhnt/oggbd/dirdat/ce
TABLE test.*;
Here is the report of this process:
***********************************************************************
Oracle GoldenGate for Big Data
Version 21.4.0.0.0 (Build 002)
Oracle GoldenGate Capture
Version 21.4.0.0.0 OGGCORE_21.4.0.0.0OGGRU_PLATFORMS_211022.1803
Oracle Linux 7, x64, 64bit (optimized), Generic on Oct 22 2021 23:34:57
Copyright (C) 1995, 2021, Oracle and/or its affiliates. All rights reserved.
Starting at 2023-03-09 10:49:59
***********************************************************************
Operating System Version:
Linux
Version #2 SMP Mon Jan 23 18:37:18 PST 2023, Release 5.4.17-2136.316.7.el8uek.x8
6_64
Node: localhost.localdomain
Machine: x86_64
soft limit hard limit
Address Space Size : unlimited unlimited
Heap Size : unlimited unlimited
File Size : unlimited unlimited
CPU Time : unlimited unlimited
Process id: 6065
Description:
***********************************************************************
** Running with the following parameters **
***********************************************************************
2023-03-09 10:49:59 INFO OGG-03059 Operating system character set identifie
d as UTF-8.
2023-03-09 10:49:59 INFO OGG-02695 ANSI SQL parameter syntax is used for pa
rameter parsing.
EXTRACT capcass
JVMCLASSPATH ggjava/ggjava.jar:DependencyDownloader/dependencies/cassandra_libs/
*
TRANLOGOPTIONS SFTP KNOWNHOSTSFILE /home/manhnt/known_hosts
TRANLOGOPTIONS CDCREADERSDKVERSION 3.11
TRANLOGOPTIONS CDCLOGDIRTEMPLATE /var/lib/cassandra/cdc_raw
SOURCEDB USERIDALIAS cass
2023-03-09 10:50:00 INFO OGG-03037 Session character set identified as UTF-
8.
EXTTRAIL /home/manhnt/oggbd/dirdat/ce
TABLE test.*;
2023-03-09 10:50:00 WARNING OGG-02735 No heartbeat table schema available. Hea
rtbeat table is not enabled.
2023-03-09 10:50:00 INFO OGG-01851 filecaching started: thread ID: 13975815
2226560.
2023-03-09 10:50:00 INFO OGG-01815 Virtual Memory Facilities for: COM
anon alloc: mmap(MAP_ANON) anon free: munmap
file alloc: mmap(MAP_SHARED) file free: munmap
target directories:
/home/manhnt/oggbd/dirtmp.
2023-03-09 10:50:00 INFO OGG-25340
Database Version:
Apache Cassandra version[3.11.13]
Cluster name [Test Cluster]
Datacenter [datacenter1]
Rack [rack1]
Driver Information:
Apache Cassandra CPP Driver version[2.14.1].
2023-03-09 10:50:00 INFO OGG-25339 Connected to database Genericdb, CPU inf
ormation not available.
2023-03-09 10:50:00 INFO OGG-01055 Recovery initialization completed for ta
rget file /home/manhnt/oggbd/dirdat/ce000000013, at RBA 1593.
2023-03-09 10:50:00 INFO OGG-01478 Output file /home/manhnt/oggbd/dirdat/ce
is using format RELEASE 19.1/21.1.
2023-03-09 10:50:00 INFO OGG-01026 Rolling over remote file /home/manhnt/og
gbd/dirdat/ce000000013.
2023-03-09 10:50:00 INFO OGG-01053 Recovery completed for target file /home
/manhnt/oggbd/dirdat/ce000000014, at RBA 1593.
2023-03-09 10:50:00 INFO OGG-01057 Recovery completed for all targets.
2023-03-09 10:50:00 INFO OGG-00182 VAM API running in single-threaded mode.
2023-03-09 10:50:00 INFO OGG-01515 Positioning to begin time 2023 M03 4 16:
51:53.
path[./dirdat/ce] seq[0] rba[0]
BigDataVam recovery: recovering from trail file sequence trail file seq[0]
path[/home/manhnt/oggbd/dirdat/ce] seq[13] rba[1593]
BigDataVam recovery: recovering from trail file sequence trail file seq[0]
BigDataVam recovery: startPosition[] lastSequenceID[].
Java VM Option: -DGROUPNAME=CAPCASS
Java VM Option: -DOGG_DIRLOG=dirrpt
Java VM Option: -DOGG_DIRCRD=../../../dircrd
Java VM Option: -Dgg.log=log4j
Java VM Option: -Dlog4j.configurationFile=log4j-default.properties
Java VM Option: -Dgg.log.level=info
Java VM Option: -Dgg.log.file.size=52428800
Java VM Option: -Dgg.log.file.count=7
Java VM Option: -Djava.class.path=ggjava/ggjava.jar:/home/manhnt/oggbd/Dependenc
yDownloader/dependencies/cassandra_libs/netty-codec-4.0.47.Final.jar:/home/manhn
t/oggbd/DependencyDownloader/dependencies/cassandra_libs/logback-core-1.2.9.jar:
/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jna-4.2.2.ja
r:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/reporter-c
onfig-base-3.0.3.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassan
dra_libs/jackson-annotations-2.13.2.jar:/home/manhnt/oggbd/DependencyDownloader/
dependencies/cassandra_libs/gson-2.8.0.jar:/home/manhnt/oggbd/DependencyDownload
er/dependencies/cassandra_libs/metrics-logback-3.1.5.jar:/home/manhnt/oggbd/Depe
ndencyDownloader/dependencies/cassandra_libs/metrics-core-3.1.5.jar:/home/manhnt
/oggbd/DependencyDownloader/dependencies/cassandra_libs/ecj-4.4.2.jar:/home/manh
nt/oggbd/DependencyDownloader/dependencies/cassandra_libs/netty-transport-4.0.47
.Final.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/j
ackson-databind-2.13.2.2.jar:/home/manhnt/oggbd/DependencyDownloader/dependencie
s/cassandra_libs/asm-commons-5.0.3.jar:/home/manhnt/oggbd/DependencyDownloader/d
ependencies/cassandra_libs/asm-util-5.0.3.jar:/home/manhnt/oggbd/DependencyDownl
oader/dependencies/cassandra_libs/jcl-over-slf4j-1.7.25.jar:/home/manhnt/oggbd/D
ependencyDownloader/dependencies/cassandra_libs/ohc-core-j8-0.4.4.jar:/home/manh
nt/oggbd/DependencyDownloader/dependencies/cassandra_libs/asm-5.0.4.jar:/home/ma
nhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/snappy-java-1.1.1.7.
jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/caffeine
-2.2.6.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/n
etty-handler-4.0.47.Final.jar:/home/manhnt/oggbd/DependencyDownloader/dependenci
es/cassandra_libs/thrift-server-0.3.7.jar:/home/manhnt/oggbd/DependencyDownloade
r/dependencies/cassandra_libs/commons-codec-1.9.jar:/home/manhnt/oggbd/Dependenc
yDownloader/dependencies/cassandra_libs/high-scale-lib-1.0.6.jar:/home/manhnt/og
gbd/DependencyDownloader/dependencies/cassandra_libs/reporter-config3-3.0.3.jar:
/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/ohc-core-0.4
.4.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/cassa
ndra-all-3.11.13.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassan
dra_libs/jsch-0.1.54.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/ca
ssandra_libs/netty-buffer-4.0.47.Final.jar:/home/manhnt/oggbd/DependencyDownload
er/dependencies/cassandra_libs/netty-all-4.0.44.Final.jar:/home/manhnt/oggbd/Dep
endencyDownloader/dependencies/cassandra_libs/joda-time-2.4.jar:/home/manhnt/ogg
bd/DependencyDownloader/dependencies/cassandra_libs/asm-analysis-5.0.3.jar:/home
/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/lz4-1.3.0.jar:/ho
me/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jamm-0.3.0.jar:
/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/stream-2.5.2
.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jbcrypt
-0.4.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/dis
ruptor-3.0.1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_
libs/commons-lang3-3.1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/
cassandra_libs/logback-classic-1.2.9.jar:/home/manhnt/oggbd/DependencyDownloader
/dependencies/cassandra_libs/compress-lzf-0.8.4.jar:/home/manhnt/oggbd/Dependenc
yDownloader/dependencies/cassandra_libs/concurrentlinkedhashmap-lru-1.4.jar:/hom
e/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/commons-math3-3.
2.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jctool
s-core-1.2.1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_
libs/snakeyaml-1.26.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cas
sandra_libs/jnr-posix-3.0.27.jar:/home/manhnt/oggbd/DependencyDownloader/depende
ncies/cassandra_libs/log4j-over-slf4j-1.7.25.jar:/home/manhnt/oggbd/DependencyDo
wnloader/dependencies/cassandra_libs/netty-common-4.0.47.Final.jar:/home/manhnt/
oggbd/DependencyDownloader/dependencies/cassandra_libs/HdrHistogram-2.1.9.jar:/h
ome/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jflex-1.6.0.ja
r:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jnr-ffi-2.
0.7.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/ST4-
4.0.8.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/js
on-simple-1.1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra
_libs/cassandra-driver-core-3.3.1.jar:/home/manhnt/oggbd/DependencyDownloader/de
pendencies/cassandra_libs/antlr-runtime-3.5.2.jar:/home/manhnt/oggbd/DependencyD
ownloader/dependencies/cassandra_libs/airline-0.6.jar:/home/manhnt/oggbd/Depende
ncyDownloader/dependencies/cassandra_libs/jffi-1.2.10-native.jar:/home/manhnt/og
gbd/DependencyDownloader/dependencies/cassandra_libs/snowball-stemmer-1.3.0.581.
1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/jackso
n-core-2.13.2.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra
_libs/javax.inject-1.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/ca
ssandra_libs/metrics-jvm-3.1.5.jar:/home/manhnt/oggbd/DependencyDownloader/depen
dencies/cassandra_libs/asm-tree-5.0.3.jar:/home/manhnt/oggbd/DependencyDownloade
r/dependencies/cassandra_libs/jffi-1.2.10.jar:/home/manhnt/oggbd/DependencyDownl
oader/dependencies/cassandra_libs/libthrift-0.9.2.jar:/home/manhnt/oggbd/Depende
ncyDownloader/dependencies/cassandra_libs/guava-18.0.jar:/home/manhnt/oggbd/Depe
ndencyDownloader/dependencies/cassandra_libs/slf4j-api-1.7.25.jar:/home/manhnt/o
ggbd/DependencyDownloader/dependencies/cassandra_libs/commons-cli-1.1.jar:/home/
manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/sigar-1.6.4.jar:/h
ome/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs/concurrent-tre
es-2.4.0.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/cassandra_libs
/jnr-constants-0.9.0.jar:/home/manhnt/oggbd/DependencyDownloader/dependencies/ca
ssandra_libs/jnr-x86asm-1.0.2.jar:
Java VM Option: -Xrs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/manhnt/oggbd/ggjava/resources/lib/log4j-
slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/manhnt/oggbd/DependencyDownloader/depend
encies/cassandra_libs/logback-classic-1.2.9.jar!/org/slf4j/impl/StaticLoggerBind
er.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Table wildcard string[test=*;]
***********************************************************************
** Run Time Messages **
***********************************************************************
Also the log file:
INFO 2023-03-09 10:50:01.000144 [Thread-1] - Invoked CassandraCDCConfigurationManager.readParameters with args: [dirprm/.CAPCASS.properties, cassandra]
INFO 2023-03-09 10:50:01.000146 [Thread-1] - Extract group in use[CAPCASS].
INFO 2023-03-09 10:50:01.000160 [Thread-1] - Start timestamp set in extract [2023-03-04T16:51:53.712979].
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Cassandra Capture properties ***START***
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Key [stopOnActiveLogReadError] Value [true]
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Key [abendOnFullPartitionDelete] Value [true]
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Key [sdkVersion] Value [3.11]
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Key [abendOnEmptyPartitionUpdate] Value [true]
INFO 2023-03-09 10:50:01.000162 [Thread-1] - Key [nodes] Value [127.0.0.1]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [abendOnPartitionUpdate] Value [true]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [readActiveLogs] Value [false]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [cdcDirectoryTemplate] Value [/var/lib/cassandra/cdc_raw]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [sshPort] Value [22]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [skipComplexColumnData] Value [true]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [port] Value [9042]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [stopOnMissingRecord] Value [true]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [withSSL] Value [false]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [sftpKnownHostsFile] Value [/home/manhnt/known_hosts]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [allowCommitLogReprocessing] Value [false]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [username] Value [cassandra://cassandra]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [processLoopMillis] Value [5000]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [wildcardTableEntriesString] Value [test=*;]
INFO 2023-03-09 10:50:01.000163 [Thread-1] - Key [fetchRemoteLogFiles] Value [true]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [cleanupCDCLogs] Value [false]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [vamPositionType] Value [GG_ATTR_DS_TIME]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [deduplicate] Value [true]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [checkCDCFlag] Value [true]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [abendOnPartitionDelete] Value [true]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [moveCDCLogsToStagingDir] Value [true]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Key [startTimestamp] Value [2023-03-04 16:51:53.712979]
INFO 2023-03-09 10:50:01.000164 [Thread-1] - Cassandra Capture properties ***END***
I'm currently using OGG for Big Data 21.4 and Cassandra 3.11
I also read the cause and solution on the Oracle's Doc:
The Cassandra database has not flushed the data from the active commit log files to the CDC commit log files. The flush is dependent on the load of the Cassandra cluster.
I've checked my cdc_raw folder and it has log files.
The Cassandra Extract captures data from the CDC commit log files only.
That means I also read log from commitlog folder and how to config it in param file?
Check the CDC property of the source table. The CDC property of the source table should be set to true.
I set to true by using ALTER command.
Data is not captured if the TRANLOGOPTIONS CDCREADERSDKVERSION 3.9 parameter is in use and the JVMCLASSPATH is configured to point to Cassandra 3.10 or 3.11 JAR files.
I'm using Cassandra 3.11 and point to Cassandra 3.11 JAR file as you can see in the above report.