Have Host Server Upgrade in Progress

Off Topic
41

Thank you, Paul, for saying in a “kinder, gentler” way than I would have: WTF!? I transferred four sites today—reloaded files, changed nameservers at godaddy.com, recreated email accounts, etc. I answered many “WTF” emails with phrases gleaned from this FW thread. I left the two biggest sites in hopes that it would be resolved, but I am so disappointed in the lack of response from James that I don’t know if I will leave them there, even after (if ?) this problem is solved.

James, if you’re listening, learn at least one thing from this: communicate with your customers! Own up to what’s happening! God, we are willing to forgive, but not when we are being totally ignored. There is no bigger insult.

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

42

I’m having an early start here in the UK (4.30am) and all my sites are back on-line and everything looks like it’s back to normal at last. Fingers crossed :¬)

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

43

Ah…I should have checked access to cPanel before posting the above. cPanel is NOT working at 5am here in the UK.

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

44

Looks like my sites are up and running now. Not able to check email this morning as I am out for a bit.

No reply from James to my last email. I sent him 7 emails about this yesterday all asking him what was going on and giving him a chance to explain. I had three very short replies, none of which addressed my questions.

He handled this very badly.

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

45

I had this email earlier:

02/11/2010 12:30 AM
My sincerest apologies to everyone affected by this absolutely unintended downtime. What started as a routine maintenance upgrade, turned into a catastrophic nightmare!

What Happened?
The back-up drive was scanned on Monday afternoon looking for corruption or security issues. There was some identified and I was contacted late Monday evening and was told that they would need to do this ASAP and it would only require 4-6 hours to change the drives - though it would require the server to be offline for that time. We discussed the best possible time and decided that 7pm EST was the best time to start as most North American / South American clients would be done for the day and EU clients would be in bed and that by the time they got to work - they wouldn’t know of the downtime - on both sides of the pond.

I agreed that Tuesday night would be best as given the severity (though routine) of the issue, I did not want to jeopardize any of the clients’ data.

First Mistake…
I did not send out an immediate notification… Honestly, we’ve done many such maintenance upgrades, over the years - that I 1) was confident in the techs doing the job and 2) didn’t want to cause you, the client - any undue stress for something that wasn’t really a big deal… and nobody would be really affected based on the time we’d chosen to proceed…

Murphy’s Law
So the maintenance started on schedule at 7pm EST Tuesday night… All seemed to be going well… and hearing nothing from the techs - I went to bed at 11pm… only to wake up to find that all he** had broken loose!

During the changing of the drives, the main and secondary RAID drives were also scanned and they too had the same issue! The techs immediately - without first consulting me - decided to do a complete change of all three drives… Now, had I known that this was the case, I would most asssuredly stopped the process and shifted the maintenance to begin Friday night into the weekend… But as I said, I was not consulted and like you, I woke to a Wednesday morning mess.

Why No Communication?
Unfortunately, the support system that I am sending this announcement with - resides on the same server that was down! So, I was completely unable to do anything but try to respond as quickly as I could to the emails that I did receive from clients. At times, I had no idea what was going on and was on hold for a very long time today - trying to get info about what was going on so that I was at least abreast of the progress (albeit very slow progress) and to keep pushing for the whole mess to be resolved as quickly as possible…

So, now what?
I do hope that you continue to host with me! I absolutely would never do anything to intentionally cause any of you any loss of business!

What’s the Go-Forward Plan, then?
I’ve immediately set-up a 3rd Party, off-server support site. You can see what is coming up for maintenance and also submit tickets if you are experiencing any issues or have questions. Here’s the link: http://havehost.zendesk.com

In addition, you can also just send an email to: email@hidden

As a new iPhone owner, I have also installed the support system’s ticket system app and will be pushed notifications immediately upon your ticket or forum submission directly and immediately to my iPhone.

In addition, I am switching all support email to utilize GMail, instead of the old have-host.com email as in today’s case - if the server is down, I didn’t receive many of your emails until late this evening when the server went back to normalcy.

In Closing…
Once again, I apologize for any inconvenience that the downtime caused you and/or your organization. Rest assured that with the new 3rd Party Support Site you will always have immediate access to a means to communicate with me.

What if my site isn’t working, right now?
Please submit a ticket via either the new site or via the new email and I will have a fix in place as soon as possible.

Best Regards,
James Wilkinson

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

46

On 11 Feb 2010, 7:45 am, Steve Cox wrote, in part:
I had this email earlier:…

Yes, me too, as I’m sure did others. An extremely frustrating day ended with two sites up and running again late last night and the third site back to normal this morning. Just to say I appreciate James’ explanation and thank him for the steps he has taken to improve customer communication.

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

47

My clients’ sites went back up late last night. I was about to send them each a brief email this morning, when James beat me to the draw with the above-quoted detailed explanation. I don’t currently provide on-going site management, so I’ll leave it to my clients to decide whether to stay with Have Host or not.

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

48

To be fair to James, while his communication skills are lacking in this respect, Server outages like this are unusual.

Yes if we had been able to pre-warn our customers a lot of unpleasant calls could have been avoided.

But no matter what you do there would still have been complaints.

I, like a lot of you, have had an initial knee-jerk reaction to this extended outage and was on the point of starting to move sites to other hosting - a lot of hassle you will all agree.

I haven’t and I am positive that this will be a lesson to James and his Tech Support.

What you have to bear in mind is that we are a concentrated bunch here sharing the same hosting and thats why many voices were heard - there would not have been the same noises made if even a much larger host’s servers had gone down affecting far more sites.

You all have to make up your own minds about the way forward - to move sites or not but I for one am staying where I am just now. That doesn’t mean that I will not consider long and hard before adding new sites to James’ servers.

David

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

49

David,

you took the words right out of my mouth. I am also not moving.

THank you

Julie
On Feb 11, 2010, at 8:19 PM, DeltaDave wrote:

To be fair to James, while his communication skills are lacking in this respect, Server outages like this are unusual.

Yes if we had been able to pre-warn our customers a lot of unpleasant calls could have been avoided.

But no matter what you do there would still have been complaints.

I, like a lot of you, have had an initial knee-jerk reaction to this extended outage and was on the point of starting to move sites to other hosting - a lot of hassle you will all agree.

I haven’t and I am positive that this will be a lesson to James and his Tech Support.

What you have to bear in mind is that we are a concentrated bunch here sharing the same hosting and thats why many voices were heard - there would not have been the same noises made if even a much larger host’s servers had gone down affecting far more sites.

You all have to make up your own minds about the way forward - to move sites or not but I for one am staying where I am just now. That doesn’t mean that I will not consider long and hard before adding new sites to James’ servers.

David

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

50

Ditto the last two posts above. I believe James is as upset about this as the rest of us. He’s given pretty good support in the past so I hope as you say, lessons have been learned.

Steve

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options

51

At 20:19 -0500 11/2/10, DeltaDave wrote:

To be fair to James, while his communication skills are lacking in
this respect, Server outages like this are unusual.

Yes if we had been able to pre-warn our customers a lot of
unpleasant calls could have been avoided.

But no matter what you do there would still have been complaints.

I, like a lot of you, have had an initial knee-jerk reaction to this
extended outage and was on the point of starting to move sites to
other hosting - a lot of hassle you will all agree.

I haven’t and I am positive that this will be a lesson to James and
his Tech Support.

What you have to bear in mind is that we are a concentrated bunch
here sharing the same hosting and thats why many voices were heard -
there would not have been the same noises made if even a much larger
host’s servers had gone down affecting far more sites.

You all have to make up your own minds about the way forward - to
move sites or not but I for one am staying where I am just now. That
doesn’t mean that I will not consider long and hard before adding
new sites to James’ servers.

My hosting with James is non-critical - even trivial, so there would
be no point in moving my stuff.

He certainly needs to keep us informed about something a serious as
actual downtime, whatever time of day or night, and he should tell us
in advance about things like PHP and MySQL version changes (but not
all ISPs do that anyway). Also, that communication should be active,
by email, not by monitoring some site somewhere.

The real culprits for the extended downtime seem to be the techies at
the place where his server(s) is/are. They should not have taken it
upon themselves to change both sets of discs at once without checking
with James first. Another worrying point is that those techies don’t
just look after Have Host servers. They will be looking after a hall
full of them. Who knows which other companies and ISPs have their
servers under the watchful eye of these people. Anyone who uses
another Canadian ISP should perhaps check.

David

–
David Ledger - Freelance Unix Sysadmin in the UK.
HP-UX specialist of hpUG technical user group (www.hpug.org.uk)
email@hidden
www.ivdcs.co.uk

offtopic mailing list
email@hidden
Update your subscriptions at:
http://freewaytalk.net/person/options