23

...or sometimes both!

ESXi 5.1 crashing (bug fixed by updates)

Installing XenServer 6.2 on Proliant DL160G6 freezes during "Installing from base pack" (firmware)

Disconnected Network Adapter on VMware - only power cycle solve this (firmware)

Unable to view sensor data in ESXi 5.x on DL380 G7 (bug fixed by updates)

Why is VMware ESXi 5.5 crashing? (bug fixed by updates)

ESXi host losing connection to vCenter (bug fixed by updates)

HP ProLiant with SATA/SAS Drives (firmware)

Is this a basic troubleshooting step that people are missing? Maybe it's that VMware and HP solutions require more massaging and care than others, but my impression is that people simple aren't making the effort to keep their environments updated and run into bugs as a result.

In a time where software updates are mandatory for general purpose operating systems, our phones, televisions, bathroom scales and even my bicycle, why is it missed so often in the server hardware and virtualization space?

Is there a cleaner way to answer these questions or suggest, "are you on a recent revision of software|firmware|BIOS?" at the time of posting?

3
  • 5
    Did you upgrade the firmware before asking complaining about this problem? :)
    – Rex
    Commented Apr 1, 2014 at 16:28
  • 3
    @Rex Yes, I upgraded the firmware and it bricked my server.
    – ewwhite
    Commented Apr 25, 2014 at 13:43
  • Then don't answer them! See that? I solved your problem!
    – usedTobeaMember
    Commented May 21, 2014 at 2:47

4 Answers 4

Reset to default
34

It's very useful to know that a particular version of software/drivers/firmware actually fixes the problem that someone is describing, so these answers can be incredibly helpful to other people facing the same problem (especially people for whom upgrading firmware may be a practical problem, e.g. in production systems, high availability systems, or just people who really want to understand the root cause of their problem instead of randomly gefingerpoking until it goes away)

11
  • 3
    Good point. I just fear that I'm a broken-record by repeating the same thing... Granted, I do try to link the bugfix or release note that applies to the problem. But The frequency with which this occurs on VMware and HP-related items is surprising to me.
    – ewwhite
    Commented Mar 22, 2014 at 0:24
  • 12
    @ewwhite You're surprised that people ignore regular maintenance until the server breaks? I though you'd been around IT for a few years...
    – Chris S Mod
    Commented Mar 23, 2014 at 2:10
  • 1
    @ChrisS What surprises me isn't that people ignore regular server maintenance (hell, we do that because we're very understaffed), it's that when something breaks the first thought isn't maybe I could try updating that firmware/driver it's been running for 6 years...
    – HopelessN00b
    Commented Mar 23, 2014 at 5:06
  • 3
    "It's been running for six years, why would it break now?"
    – Tim
    Commented Mar 26, 2014 at 0:23
  • 2
    "But we have 500 machines all running the same firmware, why does only this one have trouble?"
    – MikeyB
    Commented Mar 26, 2014 at 15:34
  • 1
    @MikeyB Then that's a troubleshooting exercise. Determine what's different with the particular host. I've seem mismatched firmware across similar hardware platforms cause issues with specific nodes.
    – ewwhite
    Commented Mar 26, 2014 at 15:40
  • 3
    @ewwhite Good point. I just fear that I'm a broken-record by repeating the same thing - Welcome to why my SF participation has been drastically reduced in the last 6 months or so. You can only lead so many horses to water before you want to turn them all into glue. The higher SF climbs up the google visibility ladder, the lower the quality of the community, unfortunately. It's why I've replaced my serverfault.com bookmark with chat.serverfault.com
    – MDMarra
    Commented Mar 26, 2014 at 16:06
  • @MDMarra But it's for the love of the game, er... rep-points!
    – ewwhite
    Commented Mar 26, 2014 at 20:25
  • 3
    I blame the vendors for failing to maintain useful communication with their customers. If your product has random crashes in a critical component and all of your customers don't know about it and how to fix it, you failed them. Also, failing to be as up-front and complete as possible about side effects to updates so that people don't avoid updates just because the last two updates they did (three years ago) caused days of outages. That's assuming that all updates are being tested better than they are today.
    – Slartibartfast
    Commented Mar 27, 2014 at 5:08
  • 3
    As you say, upgrading might be a practical problem. We need to be sensitive to the possibility that the upgrade can cause unrelated problems. People administering complex systems in production use do not upgrade unless the fixed problem is more important than the risk of breaking some other function. And they might want to do a minimal upgrade, so it is helpful to know which version fixes the problem.
    – rleir
    Commented Apr 3, 2014 at 12:38
  • At first, it could help if the software/firmware didn't have bugs that need to be patched ;) tests, tests, tests...
    – vn.
    Commented Apr 3, 2014 at 14:00
10

One time I had to rush at a customer's site whose quite impressive Exchange 2010 system (talking about > 20 servers here) kept randomly stopping mail delivery, which definitely isn't something you would like a mail system to do.

The root issue was a random crash of the main mail transport service (edgetransport.exe, for those who know and/or care) on all five of the Internet-facing servers.

The root issue of that... the whole setup was stuck on O.S. and Exchange releases from more than a year and a half before, because after having it designed and installed by Really Expensive Consultants, nobody bothered patching anything, ever. Even if the servers themselves were screaming for that, since they had Internet access and were perfectly able to yell "hey, I need updates, please do them!". This included a missing Windows service pack and all subsequent updates, a missing Exchange service pack and subsequent updates, and even updates for the antimalware software running on those servers which were actually experiencing the problem. I'm quite sure that, had the servers been physical instead of virtual, firmwares and drivers would have been treated in the exact same way. I don't want to know how their real, physical servers were managed (or, more likely, left to themselves).

First thing I did, was giving the customary "please update everything to the latest releases, then we'll talk" speech. That wasn't enough. They wanted to know for sure if that was going to really fix their problem, or they weren't willing to touch anything. Even if their system was at that stage performing no useful function at all, because it just kept randomly ceasing to work. The fact they were running software not even anymore supported by the vendor itself was just that irrelevant to them.

Ok.

Proof they asked for, proof they got: I spent a whole (of course conspicuously billed) afternoon on MSDN, and found at least five different Exchange updates, which of course their servers were all lacking, which had in their release notes "this fixes random crashes of the Edge Transport Service".

They hadn't even bothered with 5 minutes of Google, they just kept restarting those servers and calling more and more consultants which, it seems, were even more clueless than them. And when someone finally told them "maybe you really should run more up-to-date software", they just didn't listen.

Oh, and the icing on the cake is, this was definitely not some cheap understaffed shop. This was Italy's biggest telecommunications company, which was hosting this Exchange system for a government agency (and not a small one).

3
  • 6
    $StrongSwearword
    – miniBill
    Commented Mar 26, 2014 at 23:01
  • @Massimo I can assure you - not only Italians do think that way. "Do not touch a running system" is different than "Do not run a touchy system". The latter was the case here and could have been fixed by patching.
    – Nils
    Commented Apr 1, 2014 at 12:35
  • American telecoms are no different.
    – Colyn1337
    Commented Dec 16, 2015 at 14:18
6

You can always downvote and move on. Leave a comment if you really wanna help, but there's no obligation to help any question asker, let alone the especially clueless, lazy or stupid.

And really, someone who doesn't have the good sense God gave pigeons in not someone worth getting burned out over.

3

The canonical solution to this kind of thing is to create a canonical Q&A then close stuff as a duplicate of it.

The canonical cron Q&A is a good example of this.

For new cron related questions I will often cast a close duplicate vote on it and then leave a comment

Please work your way through the link above it will almost certainly help you solve your problem. If it doesn't, it will help you gather information that will help us to help you. You should add this information to your question.

The intention here is that should someone find the canonical answer lacking then it will have at least given them the tools to gather the information. If someone did do this then we have the additional benefit of being able to add something new to the canonical Q&A.

4
  • 1
    There's no canonical answer here, as the reasons and problems vary. This is more of a troubleshooting step that people should take if they see unexpected behavior on their systems.
    – ewwhite
    Commented Mar 21, 2014 at 12:40
  • 4
    @ewwhite: You can't teach the internet how to diagnose a problem. Just leave a comment rather than an answer - that's what they are there for.
    – user9517
    Commented Mar 21, 2014 at 13:06
  • I'm suggesting a note that asks if firmware, software, etc. have been updated as part of the troubleshooting process. There are two separate issues here. 1). People who run into odd issues due to outdated software and 2). People whose hardware exhibits bad behavior or where toxic firmware combinations are present.
    – ewwhite
    Commented Mar 21, 2014 at 15:36
  • 3
    @ewwhite: if you can use the pro-forma comment script, saves a lot of repeated typing.
    – user9517
    Commented Mar 21, 2014 at 15:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .