RAID Admin: Understanding Logical Block Address (LBA) messages in the event log

Typically, these operations do not succeed as the result of one of the following issues:

The target drive's light is orange or red

In the event that an Xserve RAID drive has become physically unreliable, the Apple Drive Module changes from a green to solid orange or red. Please contact AppleCare for assistance; you may need a replacement Apple Drive Module.  Note:  While an Apple Drive Module is "rebuilding", the drive status light alternates between green and orange, which is normal.

It is normal for an Apple Drive Module that has previously been part of another RAID set to have a solid orange LED when inserted in an Xserve RAID. To remove the previous RAID set information and make the Apple Drive Module available for use in the current RAID set, you must use RAID Admin to make the Apple Drive Module available. This feature can be found in RAID Admin 1.5 or later (with firmware version 1.5 or later) under the Utilities tab. This step was not required prior to RAID Admin 1.5.

The target drive is not available

In the event that an Xserve RAID cannot communicate with a disk, a warning message similar to this may appear in the event log:

Warning Upper Controller 06/18/07 02:50:44 PM Disk 13 Failed To Respond. Retrying. RETRY COUNT:0 LBA:0x507600

If you would like to replace the affected Apple Drive Module with a new one, you must determine which physical disk needs to be replaced. To do so, reference the event log message that references the affected disk with a value of 1 to 14. The numeral indicates the position of the Apple Drive Module from left to right, starting with one (1). For instance, in the example 'Disk 13' is the drive that did not respond. This means that the affected drive is the 13th drive from the left. This is the drive that should be replaced. If a replacement Apple Drive Module is needed, please contact AppleCare for assistance.

The target block on the drive is 'bad'

A 'bad block' is a block that can no longer be reliably read from or written to reliably. It is normal for bad blocks to occur on hard drives, especially as the Apple Drive Module's operating hours increase. When a bad block is found, the RAID controller tries to access the bad block 5 times and logs the following event for each of the 5 attempts:

Warning Upper Controller 06/18/07 02:50:44 PM Disk 4 Reported An Error. COMMAND:0x25 ERROR:0x40 STATUS:0x51 LBA:0x91E6980

When a bad block is encountered, the Apple Drive Module disables the bad block and substitutes it with a reserved spare block. As long as there are reserved spare blocks available, the bad blocks are substituted with reserved spare blocks. When all of the reserved spare blocks have been used, the Apple Drive Module no longer substitutes bad blocks with reserved blocks. Depending on whether the Apple Drive Module is a member of a JBOD or a RAID set, the Apple Drive Module is considered reliable or not.

• Apple Drive Module RAID Members:
  In the case of an Apple Drive Module being a member of a RAID set, an Apple Drive Module that has no reserved blocks available is considered reliable; the Xserve RAID will avoid writing to blocks that are known to be bad but have not been replaced with reserved spare blocks. However, when a bad block is encountered, the RAID controller must map the bad block to a reserve block, which results in a pause in communication. Therefore, you can use RAID Admin > Background Array Conditioning to proactively locate bad blocks on the RAID drives and map them out, avoiding communication disruptions during production.
  
  To run Background Array Conditioning, follow these steps:
  
  1. Open RAID Admin and select the Xserve RAID from the list of RAIDs.
  2. Click Utilities.
  3. Select Background Array Conditioning.
  4. Select the array to be conditioned.
  5. Select a priority, which indicates how much RAID controller bandwidth will be dedicated to the conditioning process. 
     (Note that the higher the priority is, more RAID controller resources will be dedicated to the conditioning process, which can lower the performance of typical RAID functionality.)
  6. Click Start Conditioning.
  
  
• Apple Drive Module JBOD Members:
  In the case of an Apple Drive Module being a member of a JBOD, a drive that has no reserved blocks available is considered unable to read and write data reliably. To ensure that bad blocks can be replaced with reserved spare blocks, it is necessary for you to determine if there are reserved spare blocks available. Use one of the following methods to determine if reserved spare blocks are available.

  To determine if the Apple Drive Module has utilized all reserved spare blocks, use the command dd to read data from the bad block.
  
  Note: The following procedure should be performed on the computer that was writing to the JBOD disk when the warning message appeared. If the computer in question has been restarted since the warning message appeared, do not proceed; the device identifier of the affected device changed during restart.

  To read data from the bad block:
  1. Determine which OS device the affected Apple Drive Module corresponds to:
     On the computer that was accessing the affected JBOD when the issue occurred, read the system log entries that correspond to the timestamp of the LBA message in the Xserve RAID Event log. For instance, if the timestamp in the Xserve RAID event log is 6/18/07 2:50:44:44 PM, access the same timestamp in the system log. You should find a log entry similar to this:
     
     Jun 18 14:52:05 hostname kernel[0]: disk8: I/O error
     
     This log entry indicates that the OS has classified the affected Apple Drive Module as Disk 8.
  
  
  2. Convert the LBA value from a hex value to a decimal value. 
     To do so, use the following command in Terminal:
     
     $ printf "%d\\n\\n" [LBA_Value]
     
     Here the LBA_value is the hexadecimal value reported in the warning message. For instance, if the value is reported as 0x1777CD00, use the following command:
     
     $ printf "%d\\n\\n" 0x1777CD00
     
     Using this command will provide the following decimal value: 393727232
  
  
  3. Use the dd command to read data from the affected block:
     
     $ dd if=/dev/[device] of=/dev/null skip=[decimalValue] count=10
     
     Here the [device] is the value found in step 1 and where [decimalValue] is the value found in Step 2.
     
     If you do not receive a warning message when trying to read from the block, the bad block has successfully been replaced with a reserved spare block. If, however, running the command results in a message similar to the one below, then there are no reserved spare blocks available:
     
     dd: /dev/disk8: Input/output error
  
  
  4. If you have determined that an Apple Drive Module has run out of reserved spare blocks and you would like to replace it with a new Apple Drive Module, you must determine which physical drive is to be replaced. 
     
     To do this, reference the event log message that identifies the affected disk with a value of 1 to 14. The numeral indicates the position of the drive from left to right, starting with one. For instance:
     
     Warning Upper Controller 06/18/07 02:50:44 PM Disk 12 Reported An Error. COMMAND:0x25 ERROR:0x40 STATUS:0x51 LBA:0x91E6980
     
     In this example, 'Disk 12' is the drive that reported the message. This means that the affected drive is the 12th drive from the left. This is the affected Apple Drive Module that should be replaced.

If a replacement Apple Drive Module is needed, please contact AppleCare for assistance.