After some technical difficulties and some long-awaited changes, NewPointe’s podcast is back in good order and ready for your subscription.
You can subscribe to the podcast by clicking below.
Over the past couple weeks, we’ve had intermittent (my favorite) issues with one of our cameras — the signal would start getting jumpy and eventually go back to normal after a few seconds. It looked just like someone had unplugged the genlock connection and put it right back.
I finally had a chance to take a look at it today, and I was able to get a nice glimpse of the problem with my camera:
See how the copper in the center of the left connector is flared out? That’s the camera’s genlock connection, and most-likely where our issue is.
I found a small needle-nosed pliers and closed the copper up as best I could… it’s looking a little better now.
I thought about getting in there with a small soldering tip and dripping some solder on that copper to brace it a little. Going in there makes me a little nervous though. See that white at the back of the connector? That’s plastic. I don’t really want to risk melting it, or worse, accidentally shorting the connector out.
So far, the camera is passing all the stress-tests (or different wire-wiggling techniques) that I’m subjecting it to, so I’m feeling good about this being the cause of our signal issues. We’ll see in a couple weeks…
Well, it would seem that our internet connection seems to be back in order, although I’m not too thrilled with the manner in which it was ultimately fixed.
Here’s a rundown of what’s been going on:
Friday PM-Sunday PM – Our internet connection starts dropping out for about a minute every 33 minutes, almost to the second. In addition, outbound browsing is very slow and showing symptoms of packet loss. I open a ticket with TW tech support, but they claim that my situation can’t be monitored until Monday morning. I guess 24/7/365 support doesn’t apply to Tier 3.
Sunday PM-Monday AM – Our cable modem stops passing traffic completely. It still syncs up, but I’m unable to ping any external addresses with it. Another call to tech support ensues, and the tech finds that something may be wrong with whatever device that provisions are static IP addresses. Apparently cable modems that use DHCP are still working though, because I — well, let’s just say I found a way to get around this issue for a short time.
Monday AM-PM – We get the pleasure of becoming part of a widespread outage. No modem sync — at all. "Everyone who’s got a pulse is working on this one," the tech tells me. Well that’s good to know.
Monday PM – Wednesday – I finally see sync lights on our cable modem after popping my head in the datacenter for about the 15th time. Our internet is back, and it’s no longer dropping out every 30 minutes, but it’s still very slow…almost like DNS isn’t resolving properly (the answer is always DNS). Connections are fast ONCE their made, but it takes forever to get that far.
Wednesday – Slightly frustrated, I call Time-Warner again to see if they can look at the issue and stress-test our modem. The stress test doesn’t show any issues of packet loss. After looking at a few other things, this dialog ensues…
Tech: "What DNS server are you using?"
Me: "We’re using OpenDNS’s servers" (I rattle off the IP addresses)
Tech: <paraphrase>"Well, I’m looking at a bulletin sent out by our network team in response to an issue that’s been going on for about 5 days. (hey that sounds like us!) They’re recommending that anyone seeing speed issues make sure they’re using our DNS servers.
Me: "Well that’s great, but we use OpenDNS for our content filtering, plus we’ve been using them for a year now with no issues."
Tech: "Still, try changing your settings and see if that doesn’t help your speed issue." </paraphrase>
So I logged into our DNS servers and changed the forwarder addresses to match Time-Warner’s DNS servers. BOOM — instant speed increase. It would seem apparent that using TW’s DNS instead of OpenDNS fixed the issue, but now I’m left with just as many questions as answers. What’s up with OpenDNS? Do they have issues of their own? Is TW doing something on their end to interfere? If not, did a simple DNS query to a non-TW DNS server take forever?
Needless to say, I’m not too pleased with this situation. If this is going to be a permanent issue, I now have to find another way to do content filtering. I will most-likely be giving ScrubIT a try now to see if there are any similar issues. We’ll see how that goes.
What started out Friday as a every-33-minute interruption has now become a drop-dead outage as of about 5:45pm today (my FTL drive didn’t make the jump in time, Jason). In an attempt to not go on-site, I’m currently on the phone with Time Warner tech support for the 3rd time in 36 hours. On a Monday morning, a 1-minute drop every 30 minutes would be quite an inconvenience, but it would be doable, but again we’re an F1 church, and F1 Contributions requires an internet connection to work.
The rep that I talked to was able to reset the modem remotely, but unfortunately no data is passing through. So, I’m about to go on-site to power-cycle the modem and follow up on the ticket if any further work is needed. I’m hoping like crazy that this isn’t going to be an issue that bites us on Monday morning.
It doesn’t matter how many situations and problems you’ve dealt with as an IT guy, something new and different will eventually come along to make you scratch your head.
On Friday morning I got a notice that our internet connection had dropped out for a short period of time. After just a minute of being down, the connection came back up. No big deal, right?
Throughout Saturday morning, I’ve received another notice stating that our connection was down, and then up again. This happened twice on Saturday morning. Slightly concerned, I downloaded a little utility (Uptime Scout) to constantly ping our network and log any timeouts that occurred. At the same time, I called Time Warner Tech Support and opened a ticket with them so they at least had it on record.
When I returned later in the evening, what I found in the logs was very interesting. Our internet connection was dropping out every 33 minutes, almost to the second. The outage didn’t last long … a minute at the very most. However, we use F1, and if this happens on Sunday morning things could get a little messy.
Now I’m very intrigued about our problem, and so I decide to go on-site to look at our cable modem and firewall. I’m figuring that SURELY our modem is resetting itself or that there’s something that I’ll be able to see. However, after sitting in the cold datacenter for 50 minutes and watching 2 more outages happen, it’s quite apparent that there’s nothing wrong with the cable modem. It’s not flinching, and the traffic indicators are still flashing despite my inability to ping anything on the internet.
Well, that leaves you with 2 possible causes … faulty equipment directly outside our building, or a faulty network card in our pfSense box. As unlikely as I think it is that our pfSense box would have a bad nic (I just replaced it), it’s definitely not outside the realm of possibility.
So, for the last 40 minutes, my laptop has been connected directly to the cable modem in an attempt to rule out any faulty component on our network, and just as I started typing this paragraph, my pings began to time out and Uptime Scout started making noises at me. From what I can see, whatever is puking every 33 minutes is not connected to our network or even in the building.
I called Time Warner back with this extra tidbit of information, and I must say that their tech support has been very nice to work with, even at 1:00am.
Recent Comments