Our setup is that we have two databases; a SQL Server 2008 database and an Oracle database (11g). I've got the oracle MTS stuff installed and the Oracle MTS Recovery Service is running. I have DTC configured to allow distributed transactions. All access to the Oracle tables takes place via views in the SQL Server database that go against Oracle tables in the linked server.
(With regard to DTC config: Checked-> Network DTC Access, Allow Remote Clients, Allow Inbound, Allow Outbound, Mutual Authentication (tried all 3 options), Enable XA Transactions and Enable SNA LU 6.2 Transactions. DTC logs in as NT AUTHORITY\NetworkService)
Our app is an ASP.NET MVC 4.0 app that calls into a number of WCF services to perform database work. Currently the web app and the WCF service share the same app pool (not sure if it's relevant, but just in case...)
Some of our services are transactional, others are not.
Each WCF service that is transactional has the following attribute on its interface:
[ServiceContract(SessionMode=SessionMode.Required)]
and the following attribute on the method signatures in the interface:
[TransactionFlow(TransactionFlowOption.Allowed)]
and the following attribute on every method implementations:
[OperationBehavior(TransactionScopeRequired = true, TransactionAutoComplete = true)]
In my data access layer, all the transactional methods are set up as follows:
using (IDbConnection conn = DbTools.GetConnection(_configStr, _connStr))
{
using (IDbCommand cmd = DbTools.GetCommand(conn, "SET XACT_ABORT ON"))
{
cmd.ExecuteNonQuery();
}
using (IDbCommand cmd = DbTools.GetCommand(conn, sql))
{
... Perform actual database work ...
}
}
Services that are transactional call transactional DAL code. The idea was to keep the stuff that needs to be transactional (a few cases) separate from the stuff that doesn't need to be transactional (~95% of the cases).
There ought not be cases where transactional and non-transactional WCF methods are called from within a transaction (though I haven't verified this and this may be the cause of my problems. I'm not sure, which is part of why I'm asking here.)
As I mentioned before, in most cases, this all works fine.
Periodically, and I cannot identify what initiates it, I start getting errors. And once they start, pretty much everything starts failing for a while. Eventually things start working again. Not sure why... This is all in a test environment with a single user.
Sometimes the error is:
Unable to start a nested transaction for OLE DB provider "OraOLEDB.Oracle" for linked server "ORACLSERVERNAME". A nested transaction was required because the XACT_ABORT option was set to OFF.
This message, I'm guessing is happening when I have non-transactional stuff within transactions, as I'm not setting XACT_ABORT in the non-transactional code (that's totally doable, if that will fix my issue).
Most often, however, the error is this:
System.Data.SqlClient.SqlException (0x80131904): The operation could not be performed because OLE DB provider "OraOLEDB.Oracle" for linked server "ORACLSERVERNAME" was unable to begin a distributed transaction.
Now, originally we only had transactions on SQL Server tables and that all worked fine. It wasn't until we added transaction support for some of the Oracle tables that things started failing. I know the Oracle transactions work. And as I said, most of the time, everything is just hunky dorey and then sometimes it starts failing and keeps failing for a while until it decides to stop failing and then it all works again.
I noticed that our transactions didn't seem to have a DistributedIdentifier set, so I added the EnsureDistributed() method from this blog post: http://www.make-awesome.com/2010/04/forcibly-creating-a-distributed-net-transaction/
Instead of a hardcoded Guid (which seemed to cause a lot of problems), I have it generating a new Guid for each transaction and that seems to work, but it has not fixed my problem. I'm wondering if the lack of a DistribuedIdentifier is indicative of some other underlying problem. I've never dealt with an environment quite like this before, so I'm not sure what is "normal".
I've also noticed that the DistributedIdentifier doesn't get passed to WCF. From the client, I have a DistributedIdentifier and a LocalIdentifier in Transaction.Current.TransactionInformation. In the WCF server, however there is only a LocalIdentifier set and it is a different Guid from the client side (which makes sense, but I would have expected the DistributedIdentifier to go across).
So I changed the wait the code above works and instead, on the WCF side, I call a method that calls Transaction.Current.EnlistDurable() with the DummyEnlistmentNotification class from the link above (though with a unique Guid for each transaction instead of the hardcoded guid in the link). I now havea DistributedIdentifier on the server-side, but it still doesn't fix the problem.
It appears that when I'm in the midst of transactions failing, even after I shut down IIS, I'm unable to get the DTC service to shutdown and restart. If I go into Component Services and change the security settings, for example, and hit Apply or OK, after a bit of a wait I get a dialgo that says, "Failed ot restart the MS DTC serivce. Please examine the eventlog for further details."
In the eventlog I get a series of events:
1 (from MSDTC): "The MS DTC service is stopping"
2 (From MSSQL$SQLEXPRESS): "The connection has been lost with Microsoft Distributed Transaction Coordinator (MS DTC). Recovery of any in-doubt distributed transactions
involving Microsoft Distributed Transaction Coordinator (MS DTC) will begin once the connection is re-established. This is an informational
message only. No user action is required."
-- Folowed by these 3 identical messages
3 (from MSDTC Client 2): 'MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system GCOVA38.'
4 (from MSDTC Client 2): 'MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system GCOVA38.'
5 (from MSDTC Client 2): 'MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system GCOVA38.'
6 (From MSDTC 2): MSDTC started with the following settings: Security Configuration (OFF = 0 and ON = 1):
Allow Remote Administrator = 0,
Network Clients = 1,
Trasaction Manager Communication:
Allow Inbound Transactions = 1,
Allow Outbound Transactions = 1,
Transaction Internet Protocol (TIP) = 0,
Enable XA Transactions = 1,
Enable SNA LU 6.2 Transactions = 1,
MSDTC Communications Security = Mutual Authentication Required, Account = NT AUTHORITY\NetworkService,
Firewall Exclusion Detected = 0
Transaction Bridge Installed = 0
Filtering Duplicate Events = 1
This makes me wonder if there's something maybe holding a transaction open somewhere?