Hey Checkyourlogs Fans,
Today, I wanted to write to you about a very difficult problem to resolve on any network. Internet Connectivity that keeps going up and down. It can be a very frustrating experience. However, with the right troubleshooting tools and techniques you can work through the issues.
The first place most of us would go is here to something like www.speedtest.net right…
Here is what I was getting when I was testing à now remember that my connection would go up and down throughout the day and I happened to hit this while it was up.
Now, I’m supposed to be paying for a dedicated internet access DIA Service through Xplornet’s Commercial Division which is basically a point to point wireless internet connection.
So, we tried all the things you should always try on your end:
- Ping your local interface of your firewall – Check it was up
- Ping your Default Gateway of the Router à Hmmm.. It was going down.
C:\Users\DKTCLAPTOP>ping 4.2.2.2 -t
Pinging 4.2.2.2 with 32 bytes of data:
Request timed out.
Request timed out.
Reply from 10.21.2.1: TTL expired in transit.
Reply from 10.21.2.9: Destination net unreachable.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 4.2.2.2: bytes=32 time=35ms TTL=54
Reply from 4.2.2.2: bytes=32 time=37ms TTL=54
Reply from 4.2.2.2: bytes=32 time=36ms TTL=54
Reply from 4.2.2.2: bytes=32 time=37ms TTL=54
Reply from 4.2.2.2: bytes=32 time=35ms TTL=54
Reply from 4.2.2.2: bytes=32 time=36ms TTL=54
Reply from 4.2.2.2: bytes=32 time=61ms TTL=54
So, what did this tell me? It told me that I was having issues with the upstream device at the head end or the tower.
Now, you still need to go through all the troubleshooting steps with Tier 1, Tier 2, and Tier 3 support.
My first support call was logged with the general after hours support line at Xplornet on Friday. They were absolutely clueless and had no idea how to deal with the commercial services which I’m supposed to be getting a 99.9% uptime guarantee on my contract. I asked that someone from the on-call support service team get back to be. Outages continue over the weekend and I started to dig in to see just how long this had been happening for.
How did I do this you might ask? Well I have been a www.pingdom.com customer for the past several years and I actively monitor all of my links and my customers. I really like their reporting tools and in this case, I wanted to see just how far back the issue was happening. Remember contractually I am guaranteed 99.9% uptime. (In the mean time we had since replaced our Firewall and Switch at my home office.)
Take a look at the past 7 days in my Pingdom report.
What does this look like to you? To me having done this for a very long time the very first thing that was coming to my mind was I have duplex mismatch or there is a head end device that is in a reboot loop.
I continued my support calls with the Xplornet Commercial support team and even told them I figured that there was an issue upstream from me. The first response I received from them was that I was on the same Radio Frequency as a Weather Radar Station and that it sends out probes and this causes a loss in singnal. It was further explained that according to CRTC the Canadian Regulatory body that if a Weather Probe goes out all other signals and services must give way. So, I said “No problem … Change my signal frequency and we are good” Seemed like a viable answer as this is a wireless service that I’m on and honestly I have no clue as to the back-end gear they are using as an ISP.
Unfortunately, the problem was still happening and I even told the tech that I think you have an issue upstream have you checked to see if things are rebooting. During this conversation I asked if they monitor the endpoint connections like mine for outages and was told they only look at things on a 3-5 minute window so if something was rebooting inside that window it would let them know….
This is where not nice Dave came out… I said you have a contractual SLA with my service and you are telling me that you don’t monitor the logs on your head end switches via Syslog, SolarWinds or what not?? As it turns out I pulled another report and this issue had been going on for way longer than a week. Here have a look:
Do you see any issues with the above? We had just lived with the service over the past year and I hadn’t really paid much attention to things as I figured it was the wireless service, signal interference, or snow storms (rain) or something. Wow was I wrong…
Turns out the issue was 100 % on their end and they had a bad device at their head end that had been rebooting for over 1 year. I had heard customers on other services with Xplornet complaining about the services saying it was horrible and how could I still be with them.
So, once again we escalated the matted and we are now on the Wednesday / Thursday the following week when I get a phone call from the Director of the operations team stating that they had finally found the issue and the it was indeed a head end device that had been rebooting. They had dispatched a team and would work through the weekend to resolve it.
Fast forward to this week. Let’s have a look at what we have now:
Forget about from the time prior to November 7th. Look at the 7th and onward does that look better. I think so! As part of my pain and suffering I was also able to re-negotiate the balance of my contract with the ISP Xplornet dropping my monthly costs from 725.00 per month for a guaranteed 10/10 Symmetrical service to 450 per month + received the credits due per the contract.
I do have to say this about the Commercial support team at Xplornet… They had a chance to do the right thing and they did live up to their obligations contractually and hopefully now I will have a much more stable service and continue to be a loyal customer.
Lessons learned for you should be:
- Frequently check your actual speeds and make sure you are getting what you are paying for
- Ensure that your read and understand your rights contractually especially with ISP’s
- Don’t just believe what they tell you … Use your own tools
- Use External Monitoring Services to keep track of things. Don’t think they have tools or will share their data with you.
If I hadn’t had a service like Pingdom with accurate reports I wouldn’t have had a leg to stand on as a customer.
Lastly have a look at my new connectivity speeds:
I’m now inside of my supported range for my contract service Internet and again very happy.
Hope you enjoyed this post and you find it helpful.
Dave
