I signed up for this session (titled “Advanced ESX 3.0.x Diagnostics Log Analysisâ€) with the hope of getting some in-depth, useful information on log analysis and troubleshooting. As a field person, out there in the trenches installing and supporting this stuff, having detailed information on the information that’s being logged by the system is incredibly helpful. Let’s hope that this session provides exactly that.
The presenter is Mostafa Khalil, a member of VMware Product Support Engineering. His accent isn’t too bad, which is a pleasant surprise. He prefaces his session by saying that log analysis is a topic that could span days, but he was only given 60 minutes. Sounds like a way of saying, “I’m not going to cover everything you want me to cover.â€
We start out with a review of the ESX Server boot process, during which process Mostafa indicates that ESX no longer uses the Linux kernel to boot VMkernel. This is a fairly clear fact now with the release of ESX Server 3i, the embedded version of the hypervisor, but something to note nevertheless.
The easiest way to collect log files is to use the Virtual Infrastructure (VI) Client. Select an ESX server, go to the Administration menu, and select “Export Diagnostics Dataâ€. This will collect the following logs (all paths are relative to /var/log):
- vmkernel
- messages
- dmesg
- boot.log
- initrdlogs/*
- vmksummary
- vmware/hostd.log
- vmware/vpx/vpxa.log
- esxcfg-boot.log
- esxcg-firewall.log
- vmware-cim.log
- esxcfg-linuxnet.log
- esxupdate.log (see my patch management session notes for more on this one)
- oldconf/esx*.conf
- rpmpkgs
- vmkernel-version
Questions after the session appeared to indicate that using the vm-support script is equivalent to gathering logs through the VI Client. If there was anyone else from the session there, can you confirm that’s what you heard as well?
Shortly after this point, my laptop battery died, and so I was unable to transcribe any more information. I will try to update this article later with some of the information I can remember. Sorry everyone! Feel free to chip in for a second laptop battery…
More posts are on the way, so keep reading, and keep the comments and corrections coming!
Tags: ESX, Virtualization, VMware
6 comments
Comments feed for this article
Trackback link
http://blog.scottlowe.org/2007/09/12/vmworld-2007-session-on-advanced-diagnostics-log-analysis/trackback/
Wednesday, September 12, 2007 at 6:02 pm
vmzare
Though I’m far far away from San fransico, I can confirm vm-support is same as running export diagnostic data from VI-Client. When you run this command from GUI, it actually sends the same command back to ESX and get those Tarballs.Please do share if someone disagrees with me.
Thursday, January 31, 2008 at 11:06 am
Suresh Thoppay
Is there any command line to export the diagnostic data instead of using it from GUI?
Thursday, January 31, 2008 at 4:27 pm
slowe
Suresh,
The “vm-support” command may do what you are looking for. Let me know if that helps!
Thursday, February 14, 2008 at 10:30 am
Suresh Thoppay
With vm-support, you need to do it for every server. It can’t collect the details from the VC server.
Thursday, June 5, 2008 at 7:10 pm
Vivien
Hi
I am just wondering have you ever encountered an esx 3.5 server rebooted itself without warning and the logs doesn’t say much? It’s as if someone just press the reboot button and there are not many errors. Do you think this is a hardware error? I went through the vmkernel/vmwarning logs. The only thing on warning is irq 0 is not valid error.
That’s about it.
Thank you!
Friday, June 6, 2008 at 9:16 am
slowe
Vivien,
Is this happening on a regular basis, or was it just a one-time thing? Either way, it shouldn’t have happened (or be happening). I’d recommend opening a case with VMware support and getting down to the root cause with them.