When you’ve worked in an industry as long as we have (25+ years!), you’re bound to have a horror story…or two or three. Of course every industry’s horror stories look a little different. For us in the tech industry, when things take a wrong turn for our clients, it can mean potentially catastrophic consequences. Read on – if you dare – for some of our greatest IT nightmares for just one of our clients (These might even be scarier than the IT from Stephen King.)
IT Nightmares and How We Helped Our Customer Overcome Them
IT Nightmare #1:
A year ago someone ripped up the power cables down the street from our client, causing a total power loss. Despite having undergone annual maintenance only two months prior, the backup generator for their building failed to start due to a dead battery. Our work with partners like HPE Nimble Storage can help clients in times like these, by ensuring their storage system is future-proof and down time is limited to none.
IT Nightmare #2:
Sometimes it doesn’t pay to get something new, without checking it twice at least. A new storage shelf that was put in at the end of the day turned out to have a bad drive and needed a replacement…after everyone had already left the building for the evening.
IT Nightmare #3:
A new storage array shelf that was added caused the VMware vCenter to show that the storage system had disappeared for no reason! However all the members in the cluster still remained connected without any issues. Alerts were also sent to the entire team stating that the Production system should be failed over to the disaster recovery (DR) site.
IT Nightmare #4:
Try, try, and sometimes we must try again. There’s no end to our dedication though. When six new switches were installed in a new building, one of them – the last one added to the environment – was dead on arrival when plugged in. When the replacement came, it too was dead on arrival! Finally, the third and final replacement switch worked.
IT Nightmare #5:
A switch firmware upgrade completely freaked out the network. This required support to come in due to a deprecated command, which caused two days of issues. When something like this happens, our partnerships with companies like Arista come into play to ensure our clients have a seamless cloud solution.
IT Nightmare #6:
A water pipe that was located directly under the room where all the switches are installed somehow blew up, flooding the room and building.
IT Nightmare #7:
When deploying new servers for expansion at the DR data center, one of the brand new server power cords failed right out of the box!
IT Nightmare #8:
Two High Availability (HA) firewalls, each supposedly identical, suddenly started to hijack each other’s traffic for no reason whatsoever! The connection between these two firewall peers is meant to ensure seamless failover in the event that one of the peers fails. Despite efforts to identify the issue, no cause could be determined. As a result, new hardware – identical to what was already there – needed to be installed to resolve the issue.
While these are just a few of the craziest IT nightmares we’ve seen over the years, we were nonetheless able to provide a solution to each situation that ensured the client’s ability to continue working without any downtime. When working with our IT team, our goal is to use our knowledge, experience and project delivery capabilities to customize a solution for your IT needs. Whether you’re facing an IT nightmare like the ones listed above, or just need some help solving one of your day-to-day IT issues, our team is ready to serve you!