Dave Mast – Geek At Large

I’m in the process of moving the blog to its new home right now.  For those of you who visit the blog via web browser, you will eventually get auto-forwarded to http://davemast.net if you’re not already using that URL.

For those of you who are using readers, you can subscribe to the new feed by clicking here.  This is a FeedBurner feed, so if I would ever decide to move again, you’ll still be ok.  (Thanks for the tip, Tony!  I wish I would have done the FeedBurner thing from the get-go.)

Thanks for your patience with me during this move, everyone.

It’s 2:30 and my brain is all over the place.

Some other things happening in my neck of the woods…

We are moving our phone system to a PRI next week.  We’re going to drop our copper (all except one line) and switch to a T1 to carry all of our voice lines.  This is going to allow folks that need it to have a DID (Direct Inbound Dialing) number for themselves.  People that know that number will be able to dial directly in to an extension on our phone system without going through the receptionist.   This system will also provide us with a ridiculous amount of inbound lines.  I forget exactly how many, but I know that it’s more than the 6 we have now.

I am working on a time frame to change the name of our Windows domain.  We still carry the old ’sugarcreekfmc.org’ domain name on our network, and I would like to change it to ‘newpointe.org’ to make it current.  Maybe this can go into our 2nd Quarter goals for the IT Department; I definitely don’t want to rush it.  Everything I’ve read is telling me that you can severely break your domain if things go wrong.  That being said, we will be testing the domain name change in a lab environment before even thinking about pulling it off on the production servers.

After a week of thinking it over, I’ve decided that I’m going to move my blog to a different location.  The nice thing is that I’m going to stay on the WordPress platform, so all of my posts, comments, etc. can be exported from here and imported to the new site.  The bad thing is that feed address is going to change, and you all will have to add the new feed once I move.  I promise I don’t take it hard if you want to drop me for creating extra work.    This isn’t immediate; I’ll give plenty of notice before the blog actually moves to the new server.

That’s all for now… hopefully sleep will catch up with me soon.

There’s been a lot going on this week, and things aren’t winding down anytime soon…I’m just gonna throw out some random thoughts here.

It’s official:  We are going to move our primary internet connection over to Time-Warner cable.  6Mb/512Kb for $140/month and a $300 setup fee…I’m still trying to figure out how the price dropped so much.  This is going to be a huge improvement over the WiFi that we’ve been on since December.  We will probably keep the WiFi around for a backup, though.  DSL isn’t available in our location yet, so right now it’s our only option without spending a ton of money.  That extra outbound bandwidth is going to be huge.

Also, we’ve made a decision on our VPN:  SSL-Explorer is what we will be offering to our users.  I received an overwhelming “YES” vote from our staff when I put a poll out on whether or not they would use a VPN to access their computer and files remotely.  We’re going to start with a license for 10 concurrent users (you can have as many users in the database as you want, but only 10 can be connected at once), and we’ll see how that goes.  That’s like 1/3 of our staff, so we ought to be good for awhile.  You never know though.

On Tuesday night, we removed domain controller status from our file server, and now we only have one DC, and that’s all that machine does.  Seems simple enough, yes?  Well, I came in Wednesday to find that our Exchange servers had stopped talking to each other, and though I could reach our file server just fine, I wasn’t able to make any connections at all FROM the file server.  I did some digging and eventually realized that I never changed the DNS server entries for the former domain controller; it was still looking at itself for DNS records (doh!).  After making the DNS changes ans doing some work in ESM, our file server was talking to the primary DC again, and our Exchange servers are playing nice again as well.  The moral of this story?  Make sure you test your systems THOROUGHLY after a big change like this.

I came into the office this morning to the sound of multiple people calling my name…never good when that’s happening.  It seems that no one was able to authenticate to the first Exchange server.  A look at the machine’s event log revealed that it lost contact with our one-and-only domain controller shortly after midnight.  I tried remoting into the DC and got no response.  Not good.  I went to the server rack and pointed the KVM at the seemingly dead server.  The mouse seemed to work, but as soon as I clicked on an item on the desktop, the whole machine froze instantly.  Oh no, I thought, we make this domain controller the only one on the network, and it craps out in 36 hours!  Reluctantly, I hit the reset button to see if the system will boot up, and I then see the culprit:  It turns out that a drive on the server’s RAID5 array failed and brought the system to a halt.  Luckily, I’ve got 9 drives sitting on the shelf for a Ghetto-Tastic** project, so I remove the failed drive and put a fresh one in its place.  The server boots up, and after a minute or so, the array begins to rebuild using the new drive.  Whew!  Within a few minutes, the domain controller is back up, our servers are talking again, and our users can log in! 

Some observations from this:

1. I’m extremely glad that we decided to RAID this server last year.  Had this been a single-drive server at the time of failure, we would have been extremely hosed, and I would most-likely still be at work.  
2. We need to get a hot spare drive for our domain controller.  According to the docs for the RAID controller, a hot spare would have prevented the system from locking up.  I’m no expert on how RAID setups are supposed to work, so I’m going to read more about this.  I already ordered 2 drives for this system; one to replace the failed unit, and another to plug in as a hot spare.  I’m considering getting hot spare drives for our other servers as well.
3. I miss having 2 domain controllers already.  Yes, it was a security risk serving files and mail off of our secondary DC, but if that machine still had DC status, we wouldn’t have missed a beat this morning.  I’m going to start planning for a 1U server to put in our IDF on the other corner of the building so we can avoid a repeat performance.

This has been a very eventful week.  There’s been some good learning experiences too.  I wish they hadn’t come at the expense of our uptime, but sometimes that’s the way it goes.

**Ghetto-Tastic - ©2006 Jason Powell

I didn’t really feel like troubleshooting between Firefox and IE, so I decided to load up Firefox and use it for the remainder of the evening. 

I was up around 3:00am this morning doing stuff around the house (I’m a total night owl), and I just happened to hear the famous logout.wav coming from my PC.  Windows was apparently shutting down to install updates.  What’s weird is that I had just gone to Windows Update last night (around 7 or 8pm) and loaded all the critical and optional updates for my PC.  Did Microsoft release another update within 7 hours?

I got to my PC this morning and attempted to load Google Reader using IE.  Voila!  Everything is loading just fine, and there are no deaded lock-ups like I was experiencing last night.  But, I don’t see any updates that were installed outside of the ones that I installed manually the previous evening.  Strange…. was this a covert operation?