Major Crash

[talitha]

Hi,
My new 3U server, with 56 terabytes of disk space, recently just stopped working.
I am seeing (among other things) from the serial link on the back of the unit the following message
System is quarantined due to 'multiple disks with re-read mismatch'

Does anyone know what on earth that means?
thanks in advance

J

 

[J.Green]

The system enters a quarantine state in an attempt to prevent corruption of the filesystem when it discovers that bad RAM blocks exist.

If the system has entered quarantine, you should reach out to Kaleidescape Tech Support as soon as you can. This is not a condition that can be remedied by any user action.

You can reach Tech Support at:
+1 877-352-5343 Toll Free (US Only)
+1 650-625-6160
7:00am - 5:00pm PT (M - F)

or in the UK
(0) 8081 207585 Toll Free (UK Only)
+44 2030 516267 (London)
2:00pm - 12:00am GMT (M - F)

or via email at support@kaleidescape.com

 

[talitha]

I was afraid of that..
thanks for your help..

 

[brodricj]

Unfortunately major crash disease has infected me also....
New 1U server arrived yesterday. Inserted 4TB file system from old server into new server, failed to boot. Inserted 2TB file system from old server into new server, also failed to boot.
How it can be that 2 seemingly good file systems when being transferred from one server to another would suddenly go so bad that not even a 3 disc boot works is exceedingly bad luck.
It looks like the discs need to go back to KHQ to attempt a data recovery (about a 2 month turnaround).

 

[brodricj]

Unfortunately major crash disease has infected me also.....

Very pleased to learn this morning that K support were able to recover virtually all of the data from my degraded 4x4TB file system and are in the process of replicating it onto a new 4x4TB file system. Great Support!

 

[fritzcurtis]

Interested to know how they recovered the data? Can you elaborate as this is a major concern for me with a growing library.

 

[brodricj]

A 4x4TB file system needs 3 working discs to boot. If it expects to see 4 discs but only sees 3 during a boot it will boot in a degraded mode and generate a "failed disc" message (automatically logged with the dealer and K Support). In my situation I had one disc going bad with read/write errors that was unknown to me, and then I had another disc totally fail. So I ended up with only 2 good discs in the file system meaning the server wouldn't boot or produce a "failed disc" message.
K Support were able to get the file system to boot in degraded mode with 2 good discs and 1 bad disc. The options then become to replace the failed disc with an invalidated disc and let the system do a rebuild, or to replicate to another 4x4TB-R file system. A rebuild is stressful on the discs and thus is not recommended when 1 of the discs is producing errors because if it subsequently fails during a rebuild you are down to 2 good discs with no boot possible and loss of data. So they replicate to another file system which is a less stressful process on the bad disc. In my case the failure mode resulted in the loss of only one movie which I will have to re-import after replication. Much better than having to re-import the whole collection.

The crux of the data recovery was K being able to get that bad disc to operate the file system into a degraded state (with the failed disc removed from the array). My server at home would not boot with only 2 good discs, 1 disc with errors, and 1 failed disc. The Kaleidescape magic wand could reach no further than Sunnyvale in this instance (which is why I had to send the discs to them).

So 1U system owners be aware, if a disc goes bad replace it, because if a 2nd one goes bad you're facing a loss of data. Unfortunately for me I had 2 go bad at the same time. Fortunately for me K Support was able to come to the rescue, and double fortunately the discs were in warranty (otherwise it would have been an expensive exercise to do this).

p.s. unfortunately there was no upgrade path 4TB-R>6TB otherwise I would have jumped on that option.

 

[Bchenry]

The system enters a quarantine state in an attempt to prevent corruption of the filesystem when it discovers that bad RAM blocks exist.

If the system has entered quarantine, you should reach out to Kaleidescape Tech Support as soon as you can. This is not a condition that can be remedied by any user action.

You can reach Tech Support at:
+1 877-352-5343 Toll Free (US Only)
+1 650-625-6160
7:00am - 5:00pm PT (M - F)

or in the UK
(0) 8081 207585 Toll Free (UK Only)
+44 2030 516267 (London)
2:00pm - 12:00am GMT (M - F)

or via email at support@kaleidescape.com

I recently had same problem and lost full 3u server with 14 6TB disks full of Blu-ray movies. Not happy that Kaleidescape was aware of this issue and did not have a patch or improved procedure communicated. I was replacing one bad disk and then discovered that the entire 3U server was now corrupt and not recoverable.