After network issues database can show as offline despite being connectible

Detail name	Value
Changelog Number	9755
Type	Bug
Status	Open
Affected Versions	Exasol 6.1.0, Exasol 6.2.0, Exasol 7.0.alpha1

Issue

Under a set of specific circumstances, a running database can be shown as stopped in EXAoperation and cannot be started via EXAoperation while actually continuing to run without issues.

The circumstances under which this happens are as follows:

• A minority of the cluster nodes, which contain all active nodes for this database are split off from the network connection to the rest of the cluster, while maintaining internal connectivity.
• These nodes then re-join the cluster when network connectivity is restored.

This bug has only been observed in setups with a cluster of at least 3 nodes and databases running on a single node each. Due to the nature of the circumstances that are required to trigger the bug, it is unlikely to be triggered in other setups by pure chance for databases with more active nodes. It could however be triggered by a systemic failure such as a network outage between multiple server racks that each contain the nodes and storage for one database

In a cloud-based deployment, it is believed that the conditions that trigger this bug can be caused by events such as updates to the hypervisor or migrating the virtual machine to a different server, which are performed by the cloud provider.

Workaround

If the database ist still running, fixing the issue can be postponed until a database restart is acceptable.

In most cases, issuing a Force Shutdown command for the affected database in EXAoperation will fix the issue and allow manually starting the database again.
Otherwise, the issue can be fixed by restarting cluster services on the database nodes and then starting the database.