Database not restarting due to race condition after short network incidents
Details
| Detail name | Value |
|---|---|
| Changelog Number | 9786 |
| Type | Bug |
| Status | Open |
| Affected Versions | Exasol 6.1.0, Exasol 6.2.0, Exasol 7.0.alpha1 |
Issue
The internal tracking of processes that is performed on each cluster node is subject to a race condition under the following circumstances:
- A node is briefly not reachable for the other nodes in the cluster (e.g. due to an issue affecting their network connection).
- This node then re-joins the cluster before the problem was recognized on the node itself.
As a result of this bug it is possible that the daemon that manages the restarting of a failed database waits indefinitely for a database to shut down completely after this database failed due to the connection issue.
In on-premise deployments, this bug can be triggered e.g. by faulty networking equipment or changes to its configuration. In a cloud-based deployment, it is believed that the conditions that trigger this bug can be caused by events such as updates to the hypervisor or migrating the virtual machine to a different server, which are performed by the cloud provider.
Workaround
Re-starting the cluster services on the database nodes should remedy the issue and allow re-starting the database.
