Oracle Database Discussions

Hi,

Please help me with the below:
we have this procedure running since 2 years on daily basis and it was working fine.
the only change on the systems was the TA application upgrade which includes slight change of the tables structure on the MTZ database and most of the changes were adding new columns to the tables!!
after that the job was run successful for only one day (the day of the installation!!) after that the runs were unsuccessful
the Global coordinator of the DISTRIBUTED TRANSACTIONS is the PROD database and the remote database is the the MTZ database, still I have the below errors in the alert_PROD.log after each failed run:

..
Wed Dec 16 10:27:33 2009
Error 3113 trapped in 2PC on transaction 28.17.485251. Cleaning up.
Wed Dec 16 10:27:33 2009
DISTRIB TRAN PROD.JT.JTGROUP.d163732.28.17.485251
is local tran 28.17.485251 (hex=1c.11.76783)
insert pending prepared tran, scn=136262018215 (hex=1f.b9d964a7)
Wed Dec 16 10:27:33 2009
Error stack returned to user:
ORA-02054: transaction 28.17.485251 in-doubt
ORA-03113: end-of-file on communication channel
Wed Dec 16 10:27:33 2009
DISTRIB TRAN PROD.JT.JTGROUP.d163732.28.17.485251
is local tran 28.17.485251 (hex=1c.11.76783)
change pending prepared tran, scn=136262018215 (hex=1f.b9d964a7)
to pending collecting tran, scn=136262018215 (hex=1f.b9d964a7)
DISTRIB TRAN PROD.JT.JTGROUP.d163732.28.17.485251
is local tran 28.17.485251 (hex=1c.11.76783))
delete pending collecting tran, scn=136262018215 (hex=1f.b9d964a7)
Wed Dec 16 10:31:05 2009
Error 3135 trapped in 2PC on transaction 33.42.522126. Cleaning up.
Wed Dec 16 10:31:05 2009
DISTRIB TRAN PROD.JT.JTGROUP.d163732.33.42.522126
is local tran 33.42.522126 (hex=21.2a.7f78e)
insert pending collecting tran, scn=136262137474 (hex=1f.b9db3682)
Wed Dec 16 10:31:05 2009
Error stack returned to user:
ORA-02050: transaction 33.42.522126 rolled back, some remote DBs may be in-doubt
ORA-03135: connection lost contact

the procedure we run is as follows:

SQL> set serveroutput on size 1000000
SQL> begin
2 xxcust_hr_ta_pkg.transfer_hrms_to_ta;
3 end;
4 /
begin
*
ERROR at line 1:
ORA-02041: client database did not begin a transaction
ORA-02063: preceding line from GERP_MTZ_LINK
ORA-06512: at "APPS.XXCUST_HR_TA_PKG", line 617
ORA-06512: at line 2

the distributed_lock_timeout is set on both dbs to 60

NAME TYPE VALUE
------------------------------------ ----------- -------------------
distributed_lock_timeout integer 60

the DBA_2PC_PENDING and DBA_2PC_NEIGHBORS doesn't have any records, it seems the blocking happens after the start of run time with about 20 mins and I fail to capture these records.

is there any mechanism to trace the problem?
I will try to do it manually but I feel it is hopless as most of the tracing of the code ends up with JAVA code :(
Please advice

Regards.