About the downtime

Started by Darkmoon, August 11, 2009, 09:58:18 AM

Previous topic - Next topic

Darkmoon

CMF was down for 3+ hours this morning. Near as we can tell, the entire system that CMF is hosted on (the company hosting it, all the other websites, and the company's own website) was down for that time period.

While we can't do anything when the site is down, as we aren't the server admins, we are actively working on a new solution for the future. There is discussion of moving the site to a place we can control better (and make easier backups from), in hopes of avoid the stress that was experienced this morning.

Don't feel like this is a plea for help from you, our loyal users. This is simply an fyi and reassurance that we are doing all that we can to maintain the high-level of quality you have come to expect from this site.
In Brightest Day. In Blackest Night...

hapless

*dons his Mr. Obvious hat*
Well, the downtime seemed to span from about 3am to 7am US east cost time, and according to traceroute the hosting company is in the DC area. They probably fixed it once somebody came to work in the morning.

//h
Chaosnet device not responding - check breaker on the Unibus

Tapewolf

Out of interest, how much disruption would switching hosts cause?  Would you be able to seamlessly image the entire database backend onto a new system, or would there be... problems?

J.P. Morris, Chief Engineer DMFA Radio Project * IT-HE * D-T-E


llearch n'n'daCorna

We'd have to take it down for a while.

Basically, we don't have access to the DB, and the access we have is pretty piss-poor. We'd have to shut down the forum, take a complete backup (via the half-arsed interface), transfer it to the new host, import it, unlock the forum, and away we go.

This isn't likely to be a big problem, as we'd be taking things down while the DNS updated anyway. However, we'll see where we go - we might do something intelligent like make it forum.missmab.com or something while we're at it (although I suspect that would annoy a bunch of folks who just happen to be here), perhaps as a temporary measure while we switch over; switching it back once the DNS stabilises.


By "piss-poor", I mean the SMF backup util doesn't work at all, unless you're smaller than 40Mb in size (which we passed a couple years ago) due to some brain-dead configuration on the server. And it's brain-dead because it's intended to make the server less likely to fall over due to one user hammering the crap out of it, doesn't actually achieve that, isn't a good way of doing it, etc etc... if you're gonna share the server, split it between DB and web, and put the two on different servers. However, my personal disagreements with the design decisions of the people who run the server aside, it's just another one of those things...


All this is a bit cart-before-the-horse, though; unless things go seriously south, we'll be staying with montecarlohosting for a while yet. If only because it's been pre-payed. And if they go that far south, we'll be looking at a "build from scratch" scenario...
Thanks for all the images | Unofficial DMFA IRC server
"We found Scientology!" -- The Bad Idea Bears

Succubus_1982

So I joined this forum for nothing?  :U  I mean presuming you're going to move it to a different host therefore requiring folks to join that forum instead.....

(P.S Sorry if this is a mods only thread, if so just let me know and I will delete my post and await official news)
I'm not lazy... I'm motivationally impaired

How to be an evil Overlord Part 1|Part 2

You can Skype me at lucy_locket82 but note me first or I might delete you

Tapewolf

Quote from: Succubus_1982 on August 11, 2009, 02:49:30 PM
So I joined this forum for nothing?  :U  I mean presuming you're going to move it to a different host therefore requiring folks to join that forum instead.....

What he seems to be saying is that they (llearch and Damaris) do have some way of tranferring the forum intact to a new host, but it will take time and be labour-intensive and it probably won't be necessary.  I think the 'rebuild from scratch' scenario is if montecarlo hosting go down hard and it isn't possible to retrieve the database at all.  That would be a rather unhappy event - I'm not sure there's a backup of the Brotherhood of the Machine RP, for example and I'm not sure all of our writers have been particularly conscientious about making backups.
(To say nothing of the damage it would do to the wiki citations!)

Quote from: llearch n'n'daCorna on August 11, 2009, 02:33:38 PM
we might do something intelligent like make it forum.missmab.com or something while we're at it (although I suspect that would annoy a bunch of folks who just happen to be here), perhaps as a temporary measure while we switch over; switching it back once the DNS stabilises.

To be sure that would mess with Turnsky, Jigsaw, Darkmoon and myself...

J.P. Morris, Chief Engineer DMFA Radio Project * IT-HE * D-T-E


llearch n'n'daCorna

Quote from: Succubus_1982 on August 11, 2009, 02:49:30 PM
So I joined this forum for nothing?  :U  I mean presuming you're going to move it to a different host therefore requiring folks to join that forum instead.....

No.

We're discussing what might happen as a result of the hoster (MonteCarloHosting) breaking in a bad way. They broke for 4 hours this morning, which, fortunately, they've recovered from. We're also discussing the possibility of moving from MCH to someone else, and covering all the interesting little details that would be involved in moving that.

Short answer is: if every goes right, we'd get everything shifted within a day or two, but we'd have to have the forum turned off while we did it, so there'd be a bit of an outage. If it goes horribly wrong, we'd have to start from scratch, and that would suck. Badly. Both for you guys, and for me, since I'm the poor bastard who does all the technical setup and maintenance around here.

FWIW, I'm looking into getting a solid, complete backup off MCH; with any luck, I'll be able to do that every month or so, just in case things go horribly wrong. However, I won't be able to automate it, because it's a horrible web form with authentication and stuff on it, so it'll be a "as and when I remember"...

Quote from: Succubus_1982 on August 11, 2009, 02:49:30 PM
(P.S Sorry if this is a mods only thread, if so just let me know and I will delete my post and await official news)

Have you looked to see how to delete your posts? ;-]

No, anyone can join in. It just happens that the mods are the folks who are interested - after all, they're the ones who've put in enough time to keep this thing going this long, just because they care. Anyone else who wants to pay attention is welcome to do so - that's why we posted this here, rather than in the admin-only forum.
Thanks for all the images | Unofficial DMFA IRC server
"We found Scientology!" -- The Bad Idea Bears

Mao

If nothing else I'm reading this.  I've tried to help looking for alternate hosting solutions before, but I've really not found many that would come close to meeting the criteria that I was given.  Though it still stands that I'll help in what ways I can.

----------------
Now playing: Smashing Pumpkins - Tonight Tonight
via FoxyTunes

Succubus_1982

Ah didn't realise there was no delete function (or so it seems) and does that mean if you do have to move it we'll have to rejoin the replacement forum? I'm a bit confused...

Also I dunno if it was universal but about an hour ago I couldn't load this forum page so........it went down again?
I'm not lazy... I'm motivationally impaired

How to be an evil Overlord Part 1|Part 2

You can Skype me at lucy_locket82 but note me first or I might delete you

Mao

I didn't notice any issue. :S

As for rejoining, not necessarily.  If llearch is able to successfully backup the database then once it's migrated to the new server you should be able to login as you would normally (dns stuff aside).

----------------
Now playing: Smashing Pumpkins - Perfect
via FoxyTunes

llearch n'n'daCorna

Quote from: Mao Laoren on August 11, 2009, 04:38:59 PM
If nothing else I'm reading this.  I've tried to help looking for alternate hosting solutions before, but I've really not found many that would come close to meeting the criteria that I was given.  Though it still stands that I'll help in what ways I can.

I _have_ one that meets the criteria. I just need to set it up. And given the timeframe I've now been given, it's fine.

But thanks for looking. ;-]
Thanks for all the images | Unofficial DMFA IRC server
"We found Scientology!" -- The Bad Idea Bears

hapless

Quote from: Succubus_1982 on August 11, 2009, 05:28:40 PM
Also I dunno if it was universal but about an hour ago I couldn't load this forum page so........it went down again?

Confirmed, around 22:15 CET (16:15 EDT) it was down for a short time (half of hour or so).

//h
Chaosnet device not responding - check breaker on the Unibus

Tezkat

Yeah, these guys don't seem too reliable. It's not just today, either. I've noticed numerous occasions over the past few months where CMF would be inaccessible for an hour or so at a time. Often enough to wonder if there weren't serious server load issues or something...

The same thing we do every night, Pinky...

Darkmoon

Well, no server is going yo be up 100% of the time. I've been more or less happy for the most part as when I come here it's normally up, which is better than some hosts I've used.

That said, this site is large enough that a four hour chunk of downtime is too much. A couple of sites attached to here are as much business as hobby now, and our customers (ie, you guys) have expectations (reasonable ones).

Eventually I do forsee us moving on, but don't expect to lose this place when we make the "new" one. We'll get the data to port.
In Brightest Day. In Blackest Night...