| NetHosted - Andrew |
 |
NetHosted Staff
Joined: 22 Mar 2004
Posts: 5684
|
 Posted: Sat Apr 30, 2005 4:13 pm Post subject: [30/04/05] Major Downtime on Pluto |
| |
I'm going to attempt to summarise the course of events that have lead to and caused the downtime all have experienced on Pluto today.
The server went down, the data Data Centre were alerted and attempted to reboot the server without success, they continued to try and diagnose a very hard to diagnose problem. As the server at first continued to function in many aspects, over time it degraded to the state where it would not respond to any requests. The decision was made to awake me when the data centre realised this problem was quite serious. There was a brief amount of time ~30 minutes where the server appeared to be functioning correctly. However after this time it went down with a kernel panic relating to the ext3 file system.
The RAID(1) system we have in place to protect against out-right hard disk failure (hardware failure) worked against us at this point mirroring the corruption on Disk 1 to Disk 2 of the server. By this point the data centre decided to wake up their head admin and he came into work to start to try and fix the situation with Pluto.
The decision was made by ther head admin that the disks were beyond repair. Luckily the root parition was mountable and the data centre techs quickly built a new server to transfer the data from. It may be possible to salvage more than the cPanel backups it depends how well the drive holds up as we are currently pulling the backups off it.
It is not possible to give an ETA on when all accounts will be restored at this time as too many factors are still undecided. As of 17:09 we have a new server running with cPanel installed and we will start attempting to restore accounts as soon as they have been pulled from the broken drive.
I'll keep everyone updated, apologies to all for the obvious inconvience caused by this.
Andrew
_________________
| Andrew Bassett
| Managing Director, NetHosted Ltd.
| Resellers, take a look at overselling  !
| Members, tell us what you think of NetHosted! |
|
| Back to top |
|
| NetHosted - Andrew |
|
NetHosted Staff
Joined: 22 Mar 2004
Posts: 5684
|
| Posted: Sat Apr 30, 2005 7:02 pm Post subject: |
| |
All data has been successfully moved to the new disks apart from one user, who I will get in contact with shortly after we have tried a few more things to move their data. So the restoring begins...
Andrew
_________________
| Andrew Bassett
| Managing Director, NetHosted Ltd.
| Resellers, take a look at overselling !
| Members, tell us what you think of NetHosted! |
|
| Back to top |
|
| NetHosted - Andrew |
|
NetHosted Staff
Joined: 22 Mar 2004
Posts: 5684
|
| Posted: Sun May 01, 2005 10:29 am Post subject: |
| |
Update:
Outstanding issues I'm aware of:
1) SSL certificates aren't installed
2) Addon domains don't work (new ones do, old ones don't, I know why)
3) Still to grab people's files if at all possible.
4) Re-assign dedicated IPs
5) Fantastico
Most of these depend on the DC hooking up the faulty disk so I can get everything I need.
Andrew
_________________
| Andrew Bassett
| Managing Director, NetHosted Ltd.
| Resellers, take a look at overselling !
| Members, tell us what you think of NetHosted! |
|
| Back to top |
|
| NetHosted - Andrew |
|
NetHosted Staff
Joined: 22 Mar 2004
Posts: 5684
|
| Posted: Sun May 01, 2005 3:01 pm Post subject: |
| |
Ok, we're setting up an FTP server due to the nature of the bad drive I'd rather have all the data off it.
We haven't forgotten about all the requests for custom data, we don't want to overly stress the drive in case it gives up completely.
All requets for addon domain fixes are going well, SSL certs will be back as soon as we get access to our FTP server to grab all the certs off the old server.
Fantastico will happen last.
Andrew
_________________
| Andrew Bassett
| Managing Director, NetHosted Ltd.
| Resellers, take a look at overselling !
| Members, tell us what you think of NetHosted! |
|
| Back to top |
|
| NetHosted - Andrew |
|
NetHosted Staff
Joined: 22 Mar 2004
Posts: 5684
|
| Posted: Mon May 02, 2005 6:46 pm Post subject: |
| |
| NetHosted - Andrew wrote: |
1) SSL certificates aren't installed
2) Addon domains don't work (new ones do, old ones don't, I know why)
3) Still to grab people's files if at all possible.
4) Re-assign dedicated IPs
5) Fantastico |
1 - Done for most people.
2 - Done for most people.
3 - The drive is really giving up now it's very unlikely we'll be able to rescue anything else.
4 - Done for most.
5. Done.
Andrew
_________________
| Andrew Bassett
| Managing Director, NetHosted Ltd.
| Resellers, take a look at overselling !
| Members, tell us what you think of NetHosted! |
|
| Back to top |
|
User Permissions |
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum
|
| |
Client Support
