Storage disks can fail to activate on boot

Detail name	Value
Changelog Number	10474
Type	Bug
Status	Resolved
Affected Versions	EXASOL 6.0.0, Exasol 6.1.0, Exasol 6.2.0, Exasol 7.0.alpha1
Fix Versions	Exasol 7.1.0, Exasol 6.2.13, Exasol 7.0.6
Resolution Date	2021-01-13

Description

During the boot process for database nodes, a process runs to ensure that the node's disks are present and partitioned correctly. In some cases, an internal warning from this process can be misinterpreted as an error and cause the disks to not be available for the storage service.

In this case, the following error will be shown in the Exaoperation Log:

HDD mount failed: Disk xyz primary partitions failed

If the node is an active node, this will prevent the database from starting. However, when the node is a reserve node then this may go unnoticed until at some point another node crashes, and the failsafety mechanism fails to restart the database due to the disks being offline.

Verification

The Exaoperation log can be monitored for the error message mentioned above to detect this condition.

After a node is rebooted (either automatically after a crash or manually due to maintenance), you can check that all necessary disks are marked as ONLINE in the storage service. This can be done manually in Exaoperation (EXAStorage -> Space on Disks).

If needed, this check can also be automated via Exaoperation's xmlrpc interface with the getAllDevices method on storage nodes. See here for documentation and below for an example:

>>> server.storage.n0012.getAllDevices()
[{'crc_error': False,
  'disk': 'd03_storage',
  'hid': 'h0000',
  'io_error': False,
  'name': 'disk_d03_storage_0',
  'read_only': False,
  'sector_size': 4096,
  'size': '72937373696',
  'state': 'OFFLINE'},
 {'crc_error': False,
  'disk': 'd03_storage',
  'hid': 'h0001',
  'io_error': False,
  'name': 'disk_d03_storage_1',
  'read_only': False,
  'sector_size': 4096,
  'size': '72937373696',
  'state': 'ONLINE'}]

Workaround

Enable the affected disks in the EXAStorage tab in Exaoperation by clicking on the affected node and clicking "Enable devices"

In some cases, it is possible that the devices still do not appear. In this case, it is necessary to reboot the affected node first.