Server chronicles - another repair

 The entropy can only increase, so after some time I had the server shut down, when I restarted something went wrong.
I didn't notice any problem until I did walk close to it.
I did see the Red LED blinking, indicating a problem.

OK, so we need to repair something.

Step 1 - find out the problem

The blinking LED is not much as information, so as the first step I went on the ILo system, looking for clues.

(on my server the ILo is another IP assigned to it, so I just opened a browser on that address)

On the System Information - Health Summary, only a warning sign on the Storage !
Oh no !!!! Now what ?

Ok, Clicking on Storage (where the warning sign is) reported few more details :

So the  problem is on the HPE Smart Array.
It is the module that basically handle the RAID.

But what exactly is the problem ?

The cache, a memory that stores data before to be written on the hard disks, is "degraded".
There are different possible problems, so I started to look around for a more specific way to diagnose the problem.

I run on the server Ubuntu 22.04 LTE.
A search for tools to diagnose the cache for RAID, for the HP Proliant DL360p Gen8, did show I needed to have a tool called ssacli (Smart Storage Administrator Command Line Interface).

Of course is not in the standard repo of Ubuntu.

Step 2 - install ssacli

To install ssacli we need to add an HPE repo to our list

• sudo curl -fsSL https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | sudo gpg --dearmor -o /usr/share/keyrings/hpePublicKey2048_key1.gpg
  
  
• echo "deb [signed-by=/usr/share/keyrings/hpePublicKey2048_key1.gpg] http://downloads.linux.hpe.com/SDR/repo/mcp jammy/current non-free" | sudo tee /etc/apt/sources.list.d/hp-mcp.list

After that a good update and upgrade

• sudo apt update
• sudo apt upgrade

and finally 

• sudo apt install ssacli

Step 3 - diagnose the problem

Now with the new tool, did run a general diagnostic

$ sudo ssacli ctrl all show status

Smart Array P420i in Slot 0 (Embedded)
   Controller Status: OK
   Cache Status: Permanently Disabled
   Battery/Capacitor Status: Failed (Replace Batteries/Capacitors)

Well !!! I would say a clear diagnostic !!

A more deeper search on the web, and I found that the Smart Array P420i (the RAID controller basically) has two components :

• a DIMM memory/controller   (and the status is OK)
• a battery/capacitor - that is the problem

The same search did show what to look on eBay (or other places), the battery/capacitor I need is  this one :

(On eBay for 24$) 

Ordered one, and when it will arrive, I'll change it.

A safety note

Looking for the problem, I found out that the server is still safe to be used.
When this error happens, the RAID is degraded and thus not used with the cache.
It means of course that the server operates under performance, but the data will be safe.

So matter of few days and I could even took the server offline again for a while.

What to expect

Assuming the only problem is the battery/capacitor, when the replacement arrives :

• I will change it
• restart the server

On forum and other places, people said to wait until 24 hours before to consider the repair done.

For a while is possible the warning sign on ILo will remain on until the POST will clear the error.

No need to reset something or issue commands.