Cluster splits during a node failure can cause a database restart to fail

Detail name	Value
Changelog Number	9727
Type	Bug
Status	Open
Affected Versions	Exasol 6.1.0, Exasol 6.2.0, Exasol 7.0.alpha1

Issue

If the network connection between some nodes of the cluster breaks while a database is in the process of being restarted due to a node failure, then the state of one of the nodes can be recorded incorrectly in an internal service.

This causes the restart of the database to fail even when enough nodes are back online.

Indications

So far, this bug has been observed in cloud based systems, where updates performed by the cloud provider on its hypervisor infrastructure can cause this pattern of node-restarts and short network interruptions.

The messages

System marked as stopped.
Pushed system node n0012.c0001.exacluster.local from reserve node list into active node list.

immediately followed by

Mark system active node n0012.c0001.exacluster.local as failed.

can indicate this specific issue if they persist across restart attempts and the mentioned node did not actually fail. (In this example node 12 is affected by the issue).

Workaround

Manually restarting the affected database (preceded by a Force Shutdown if needed) should resolve the issue.