MS DTC on a cloned VPC

I was trying to debug an issue, where my application was using Distributed Transactions to do database access. The data was to be inserted across 4 databases in a transaction, where it was an all or nothing.

The environment was configured as 2 clusters: 4 VPCs that were all cloned (app layer). This cluster talked to another cluster of DB VPC's (data layer), each being a clone of the original image.

Anyway, here is the error I was getting:

An error occurred while enlisting in a distributed transaction

MS DTC was running on both layers, so this error wasn't very helpful.

The next step was to check the permissions of the MS DTC. MS DTS, by default works locally but not remotely. So the next step is to configure this:

Go to Control Panel -> Administrative Tools -> Component Services.  Then open the settings for MS DTC, right click and choose Properties. Then click on the Security Opens underneath. This window configures the access of the machine. For this I chose:

App layer -> Allow Outbound, Enable TIP transactions.
DB layer -> Allow Inbound, Enable TIP transactions.

This got us further, with the error message changing to:

The partner transaction manager has disabled its support for remote/network transactions.

We were stumped. Searching around Google, I found a tool called DTCPing, which checks communications between machines. The communications seemed to be working except one message caught our eye:

CID Duplicated: RPC may be successful but DTC may not work.

CID? Guid more like! This means that the DTC on both machines have the same GUID - a big no no on MS DTC. This is fixed by running 2 little commands:

msdtc -uninstall
msdtc -install

A reboot later and the GUID will have changed. Therefore, any 2 computers communicating using MS DTC with the same GUID will not work properly.

Only took 8 hours to diagnose that one - phew!