ESXTOP

Section: VMware ESX Manual (1)
Updated: October 1, 2008
Index Return to Main Contents
 

NAME

esxtop, resxtop - display ESX resource utilization statistics

 

SYNOPSIS

esxtop [-] [h] [v] [b] [s] [a] [c configfile] [R vm-support_dir_path] [d delay] [n iter]

resxtop [-] [h] [v] [b] [s] [a] [c configfile] [d delay] [n iter] [-server server-name [-vihost host-name]] [-portnumber socket-port] [-username user-name]  

DESCRIPTION

esxtop provides a fine-grain look at resource utilization in real time. For more information, please refer to the official documentation, available at <http://www.vmware.com/info?id=193>. Esxtop runs on the ESX Service Console. It can only be run by user root. It can be run in three different modes; interactive (default), batch and replay. Worlds in this document refer to ESX VMKernel schedulable entities, similar to processes in other operating systems.

resxtop provides a fine-grain look at resource utilization of a specified server in real time. Resxtop connects to the server using a user name and password. Options --server, --vihost, --portnumber and --username can be used to specify the connection parameters. The resxtop user will be prompted to input the password. If resxtop connects to ESX directly, --server specifies the ESX system name. If resxtop connects to a VC server, --server specifies the VC server name and --vihost must be used to specify the ESX host name. Resxtop can be run in two different modes in the same way as esxtop; interactive (default) and batch.  

Interactive Mode

This is the default mode for esxtop. This mode displays resource usage statistics in the following screens: CPU screen, memory screen, storage adapter screen, storage device screen, storage VM screen, network screen, and interrupt screen. A help menu is available for each screen.
 In this mode there are several command line options available.  

Command-line Options

h Prints help for esxtop command-line options. v Prints esxtop version number. s Tells esxtop to run in secure mode. This disables the interactive command to change delay between screen updates. a Show all statistics. This option overides ~/.esxtop4rc configuration file setups and shows all statistics. c Load user-defined configuration file. Default configuration file is ~/.esxtop4rc. User can specify the filename to be used as configuration file. User-defined configuration file will be default file name for saving configuration with W command. d Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. You can change this with the interactive command 's'. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Update the display 'n' number of times and then exit.  

Common Statistics Description

Several statistics are displayed on the different screens while esxtop is running in interactive mode. Statistics listed in this section are common across all screens. "uptime" This first line found at the top of each of the esxtop screens displays the current time, time since last reboot and number of currently running Worlds.
 On this first line CPU load averages are also displayed for the CPU, storage and network screens. The load averages over the past 1, 5 and 15 minutes are displayed. Load averages take into account both running and ready-to-run Worlds. A load average of 1.00 implies that all the physical CPUs are fully accounted for. A load average of 2.00 implies that ESX may be in need of twice as many physical CPUs as currently available.
 For the memory screen memory overcommit averages are also displayed on the first line. The memory overcommit averages over the past 1, 5 and 15 minutes are displayed. A memory overcommit average of 0.37 implies that the memory is overcommit by 37%.  

Common Interactive Commands

Several single-key commands are recognized while esxtop is running in interactive mode. Commands listed in this section are common across all screens. The command to specify the delay between updates is disabled if the s option has been given on the command line. All sorting interactive commands sort in descending order. "h or ?" Displays a help menu for the current screen giving a brief summary of commands, and the status of secure mode. space Immediately updates the current screen. ^L Erases and redraws the current screen. "f or F" Displays a screen to add or remove statistics columns (fields) to or from the current screen. See below for more information. "o or O" Displays a screen to change order of statistics columns on the current screen. See below for more information. # Change the number of statistics rows to show. You will be prompted to enter the number. Any value greater than 0 overrides automatic determination of the number of rows to show, which is based on window size measurement. This number updated on any one of the esxtop screens will be used for all esxtop screens. L Change the display length of the name fields, which are "NAME" for the CPU, memory and storage vm screens, "DEVICE" for the sotrage device screen, "DNAME" for the network screen, "DEVICES" for the interrupt screen. You will be prompted to enter the number. Any value greater than 0 overrides the default number of characters of the name field to show. A value equals to 0 means the default value applies. s Change the delay between updates. You will be prompted to enter the delay time, in seconds, between updates. Fractional values are recognized down to microseconds. The default value is 5 seconds. The minimum value is 2 seconds. This command is not available in secure mode. W Write current setup to ~/.esxtop4rc, or to user-defined configuration file specified by -c option. This is the recommended way to write an esxtop configuration file. See below for more information. q Quit. c switch to CPU resource utilization screen. m switch to memory resource utilization screen. d switch to storage (disk) adapter resource utilization screen. u switch to storage (disk) device resource utilization screen. v switch to storage (disk) VM resource utilization screen. n switch to network resource utilization screen. i switch to interrupt utilization screen.  

The Statistics Columns (fields) and Order Screens

After pressing f, F, o or O you will be shown a screen specifying the field order on the top line and short descriptions of the field contents. The field order string uses the following syntax: if the letter in the field string corresponding to a field is upper case, the field will be displayed. This is also indicated by an asterisk in front of the field description. The order of the fields corresponds to the order of the letters in the string.
 From the field select screen you can toggle the display of a field by pressing the corresponding letter.
 From the order screen you may move a field to the left by pressing the corresponding upper case letter and to the right by pressing the lower case one.  

Configuration File

esxtop reads its default configuration from ~/.esxtop4rc or from specified file name with -c option. This configuration file contains five lines. The first four lines contain lower and upper case letters to specify which fields in what order are to be displayed on the CPU, memory, storage and network screens. The letters correspond to the letters in the Fields or Order screens for the respective esxtop screens. Editing this file by hand is not recommended. Select fields and order in a running esxtop process and save this file using the W interactive command. The fifth line contains information on the other options. Most important, if you have saved a configuration in secure mode, you will not get an insecure esxtop without removing the lower 's' from this fifth line of your ~/.esxtop4rc. A digit specifies the delay time between updates. As in interactive mode, a lower 'c', 'm', 'd' and 'n' determines the screen esxtop starts on.  

CPU Screen

This screen displays server-wide and per Resource Pool/World CPU utilization statistics. By default, World CPU utilization statistics are aggregated into Resource Pools of Worlds. For Worlds belonging to a Virtual Machine, these Resource Pools correspond to running Virtual Machines. All other Worlds are logically aggregated into Resource Pools to save vertical window space.

Statistics Descriptions
PCPU USED(%) This line displays the percentage CPU usage per PCPU, and the percentage CPU usage averaged over all PCPUs. A PCPU refers to a physical hardware execution context -- a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled. PCPU USED(%) is the percentage of PCPU nominal frequency that was used since the last screen update. PCPU USED(%) is the total sum of %USED for Worlds that ran on this PCPU. Note that if PCPU is running at frequency that is higher than its nominal (rated) frequency, then PCPU USED(%) can be greater than 100%.

PCPU UTIL(%) This line displays the percentage CPU utilization per PCPU, and the percentage CPU utilization averaged over all PCPUs. A PCPU refers to a physical hardware execution context -- a physical CPU core if hyper-threading is unavailable or disabled, or a logical CPU (aka LCPU or SMT thread) if hyper-threading is enabled. UTIL(%) represents the raw PCPU utilization, that is the percentage of real time that PCPU was not idle.

USED(%) and UTIL(%) may be different due to power management technologies or hyper-threading.

CORE UTIL(%) This line displays the percentage utilization of each core and the average over all cores. It is displayed only when hyper-threading is used. In interactive mode, the percentage utilization of a core is aligned with the percentage utilization of its first logical CPU (aka LCPU or SMT thread).

A core is utilized, if either or both of its logical CPUs are utilized. The percentage utilization of a core is not the sum of the percentage utilization of both logical CPUs on this core.

CCPU(%) This line displays the percentages of total CPU time as reported by the ESX Service Console. us is for percentage user time, sy is for percentage system time, id is for percentage idle time and wa is for percentage wait time. cs/sec is for the context switches per second recorded by the ESX Service Console. ID The Resource Pool ID of the running World's Resource Pool or World ID of running World. GID The Resource Pool ID of running World's Resource Pool. NAME The name of running World's Resource Pool or name of running World. NWLD The number of worlds in running World's Resource Pool. If a Resource Pool is expanded using the interactive command e (see interactive commands below) then NWLD for all the resulting Worlds belonging to the Resource Pool will be 1 (some Resource Pool like the console Resource Pool have only 1 member). %STATE TIMES This is a set of CPU statistics made up of the following percentages. Note that for a World, the percentages are a percentage of one physical CPU core. %USED The percentage of physical CPU core cycles used by the Resource Pool/World.

%USED may depend on the frequency with which CPU core is running. When running with lower CPU core frequency, %USED can be smaller than %RUN. On CPUs which support turbo mode, CPU frequency can also be higher than nominal (rated) frequency, and in that case %USED can be larger than %RUN. %SYS The percentage of time spent in the ESX VMKernel on behalf of the Resource Pool/World to process interrupts and to perform other system activities. This time is part of used time used to calculate %USED above. %OVRLP The Percentage of system time that was spent on behalf of some other Resource Pool/World while Resource Pool/World was scheduled. This time is not included in the %SYS for Resource Pool/World. For example, if Resource Pool A was currently scheduled and a network packet for Resource Pool B was being processed by the ESX VMKernel, then the time spent doing so appears as %OVRLP for Resource Pool A and %SYS for Resource Pool B. %RUN Percentage of total time scheduled. This time does not account for hyper-threading and system time. Hence, on a hyper-threading enabled server, the %RUN can be twice as large as %USED. %WAIT The total percentage of time the Resource Pool/World spent in wait state. Note that this percentage includes the percentage of time the Resource Pool/World was idle. %CSTP The percentage of time the Resource Pool/World spent in ready, co-deschedule state. This statistic is for internal to VMware use only. %IDLE The percentage of time the Resource Pool/World was idle. Subtracting this percentage from %WAIT above gives you the percentage of time the Resource Pool/World was waiting on some event. %RDY The percentage of time the Resource Pool/World was ready to run. %MLMTD Percentage of time the ESX VMKernel deliberately didn't run the Resource Pool/World because that would violate the Resource Pool/World's limit setting. Since the Resource Pool/World is ready to run when it is prevented from running in this way, the %MLMTD (max limited) time is included in %RDY time. %SWPWT Percentage of time the Resource Pool/World was waiting for the ESX VMKernel swapping memory. The %SWPWT (swap wait) time is included in the %WAIT time. EVENT COUNTS/s This is a set of CPU statistics made up of the following, per second, event rates. These statistics are for internal to VMware use only. CPU ALLOC This is a set of CPU statistics made up of the following CPU allocation configuration parameters. AMIN Resource Pool/World attribute reservation. AMAX Resource Pool/World attribute limit. ASHRS Resource pool attribute shares. SUMMARY STATS This is a set of CPU statistics made up of the following CPU configuration parameters and statistics. Note that these statistics are only applicable to Worlds and not Resource Pools. AFFINITY BIT MASK Bit mask showing the current scheduling affinity for the World. HTSHARING current hyper-threading configuration. CPU The physical or logical processor the World was found to be running on. EMIN The Effective Min in MHz for the World. The amount of CPU resources guaranteed to the world if all the worlds on the system start contending for CPU resources. ESX VMKernel dynamically calculates the EMIN value for all worlds based on the resource settings (Reservations, Limits and Shares) of all the resource pools and VMs on a system. This statistic is for internal to VMware use only. HTQ World is currently quaratined or not. 'N' implies no and 'Y' implies yes. TIMER/s The timer rate the World is currently requesting for.

Interactive Commands
e Expand/Rollup Resource Pool/Worlds CPU statistics. Allows viewing CPU resource utilization statistics broken down by individual Worlds belonging to a Resource Pool. U Sort Resource Pools/Worlds by Resource Pool %USED column. R Sort Resource Pools/Worlds by Resource Pool %RDY column. N Sort Resource Pools/Worlds by GID column. This is the default sort order. V Display VM instances only.  

Memory Screen

This screen displays server-wide and per Resource Pool memory utilization statistics. As on the CPU screen, Resource Pools correspond to running Virtual Machines or Worlds that consume memory. Note the distinction between machine memory and physical memory in this section.

Statistics Descriptions
PMEM (MB) This line displays the machine memory statistics for the server. All numbers are in megabytes. total is for the total amount of machine memory in the server, cos is for the amount of machine memory allocated to the ESX Service Console, vmk is for the amount of machine memory being used by the ESX VMKernel, other is for the amount of machine memory being used by everything other than the ESX Service Console and ESX VMKernel and free is for the amount of machine memory that is free. VMKMEM (MB) This line displays the machine memory statistics for the ESX Server VMKernel. All numbers are in megabytes. managed is for the total amount of machine memory managed by the ESX Server VMKernel, minfree is for the minimum amount of machine memory that the ESX VMKernel would like to keep free, rsvd is for the amount of machine memory that is currently reserved, ursvd is for the amount of machine memory that is currently unreserved and state is for the memory state as reported by the ESX VMKernel. Possible values are high, soft, hard and low. Here, high implies that the machine memory is not under any pressure and low implies that the machine memory is under pressure. COSMEM (MB) This line displays the memory statistics as reported by the ESX Service Console. All numbers are in megabytes. free is for the amount of idle machine memory, swap_t is for the total swap configured, swap_f is for the amount of swap free, r/s is for the rate at which memory is swapped in from disk and w/s is for the rate at which memory is swapped to disk. NUMA (MB) This line displays the ESX NUMA statistics. This line is only displayed if ESX is running on a NUMA server. All numbers are in megabytes. For each NUMA node in the server there are two statistics that are displayed. The first statistic is the total amount of machine memory in the NUMA node that is managed by ESX. The second statistic, that is displayed within round brackets, is the amount of machine memory in the node that is currently free. PSHARE (MB) This line displays the ESX page-sharing statistics. All numbers are in megabytes. shared is for the amount of physical memory that is being shared, common is for the amount of machine memory that is common across World(s) and saving is for the amount of machine memory that is saved due to page-sharing. SWAP (MB) This line displays the ESX swap usage statistics. All numbers are in megabytes. curr is for the current swap usage, target is for where ESX expects the swap usage to be, r/s is for the rate at which memory is swapped in by ESX from disk and w/s is for the rate at which memory is swapped to disk by the ESX Server. MEMCTL (MB) This line displays the Memory balloon statistics. All numbers are in megabytes. curr is for the total amount of physical memory reclaimed using the vmmemctl modules, target is for the total amount of physical memory ESX would like to reclaim using the vmmemctl modules and max is for the maximum amount of physical memory ESX can reclaim using the vmmemctl modules. AMIN Resource pool attribute reservation. AMAX Resource pool attribute limit. ASHRS Resource pool attribute shares. NHN Current Home Node for Resource Pool. This statistic is only applicable on NUMA systems. NMIG Number of NUMA migrations between two snapshots. It includes balance migration, inter-node VM swaps performed for locality balancing and load balancing. This statistic is only applicable on NUMA systems. NRMEM (MB) Current amount of remote memory being accessed by Resource Pool. This statistic is only applicable on NUMA systems. NLMEM (MB) Current amount of local memory being accessed by Resource Pool. This statistic is only applicable on NUMA systems. N%L Current percentage memory being accessed by Resource Pool that is local. This statistic is only applicable on NUMA systems. GST_NDx (MB) The guest memory being allocated for Resource Pool on NUMA node x. "x" is the node number. This statistic is only applicable on NUMA systems. OVD_NDx (MB) The VMM overhead memory being allocated for Resource Pool on NUMA node x. "x" is the node number. This statistic is only applicable on NUMA systems. MEMSZ (MB) The amount of physical memory allocated to a Resource Pool. GRANT (MB) The amount of physical memory granted to a Resource Pool. It is the mapped guest physical memory. GRANT - SHRDSVD is the consumed host machine memory. SZTGT (MB) The amount of machine memory the ESX VMKernel wants to allocate to Resource Pool. TCHD (MB) The working set estimate for Resource Pool. %ACTV Percentage of guest physical memory that is being referenced by the guest. This is an instantaneous value. %ACTVS Percentage of guest physical memory that is being referenced by the guest. This is a slow moving average. %ACTVF Percentage of guest physical memory that is being referenced by the guest. This is a fast moving average. %ACTVN Percentage of guest physical memory that will be referenced by the guest. This is an estimation. This statistic is for internal to VMware use only. MCTL? memory balloon driver is installed or not. 'N' implies no and 'Y' imples yes. MCTLSZ (MB) The amount of physical memory reclaimed from Resource Pool by way of ballooning. MCTLTGT (MB) The amount of physical memory ESX would like to reclaim from Resource Pool by way of ballooning. MCTLMAX (MB) The maximum amount of physical memory ESX can reclaim from Resource Pool by way of ballooning. This maximum is dependent on guest operating system type. SWCUR (MB) Current swap usage by Resource Pool. SWTGT (MB) Where ESX expectd the Swap usage by the Resource Pool to be. SWR/s (MB) Rate at which memory is being swapped in by ESX from disk for the Resource Pool. SWW/s (MB) Rate at which Resource Pool memory is being swapped to disk by ESX. CPTRD (MB) Amount of data read from checkpoint file. CPTTGT (MB) Size of checkpoint file. ZERO (MB) Resource Pool physical pages that are zeroed. SHRD (MB) Resource Pool physical pages that are shared. SHRDSVD (MB) Machine pages that are saved due to Resource Pool shared pages. COWH (MB) This statistic is for internal to VMware use only. OVHDUW (MB) Current space overhead for the user world. This statistic is for internal to VMware use only. OVHD (MB) Current space overhead for Resource Pool. OVHDMAX (MB) Maximum space overhead that may be incurred by Resource Pool. MEM Committed (MB) This is a set of memory statistics made up of minimum memory commitment target, current memory commitment target, charged memory commitment, and pages per share. These statistics are for internal to VMware use only. MEM Responsive This statistic is for internal to VMware use only.

Interactive Commands
V Display VM instances only. M Sort Resource Pools by Resource Pool MEMSZ column. B Sort Resource Pools by Resource Pool MCTLSZ column. N Sort Resource Pools by GID column. This is the default sort order.  

Storage Adapter Screen

This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per storage adapter by default. Statistics can also be viewed per storage channel, target, or path. See below for more information.

Statistics Descriptions
ADAPTR The name of the storage adapter. CID The storage adapter channel id. This id is only visible if the corresponding adapter is expanded. See interactive command 'e' below. TID The storage adapter channel target id. This id is only visible if the corresponding adapter and channel are expanded. See interactive commands 'e' and 'a' below. LID The storage adapter channel target LUN id. This id is only visible if the corresponding adapter, channel and target are expanded. See interactive commands 'e', 'a' and 't' below. ADAPTR:CID:TID:LID specifies one path. When the statistics are expanded to lun level, esxtop shows the path statistics. NCHNS The number of channels. NTGTS The number of targets. NLUNS The number of LUNs. NWDS The number of Worlds. BLKSZ The block size in bytes. This statistic is only applicable to LUNs. AQLEN The storage adapter queue depth. This is the maximum number of ESX VMKernel active commands that the adapter driver is configured to support. LQLEN The LUN queue depth. This is the maximum number of ESX VMKernel active commands that the LUN is allowed to have. ACTV The number of commands in the ESX VMKernel that are currently active. This statistic is only applicable to LUNs. QUED The number of commands in the ESX VMKernel that are currently queued. This statistic is only applicable to LUNs. %USD The percentage of queue depth used by ESX VMKernel active commands. This statistic is only applicable to LUNs. LOAD The ratio of ESX VMKernel active commands plus ESX VMKernel queued commands to queue depth. This statistic is only applicable to LUNs. CMDS/s The number of commands issued per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. READS/s The number of read commands issued per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. WRITES/s The number of write commands issued per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. MBREAD/s The megabytes read per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. MBWRTN/s The megabytes written per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. DAVG/cmd The average device latency (millisecs) per command. KAVG/cmd The average ESX VMKernel latency (millisecs) per command. GAVG/cmd The average Guest OS latency (millisecs) per command. QAVG/cmd The average queue latency (millisecs) per command. DAVG/rd The average device read latency (millisecs) per read. KAVG/rd The average ESX VMKernel read latency (millisecs) per read. GAVG/rd The average Guest OS read latency (millisecs) per read. QAVG/rd The average queue read latency (millisecs) per read. DAVG/wr The average device write latency (millisecs) per write. KAVG/wr The average ESX VMKernel write latency (millisecs) per write. GAVG/wr The average Guest OS write latency (millisecs) per write. QAVG/wr The average queue write latency (millisecs) per write. ABRTS/s The number of commands aborted per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. RESETS/s The number of commands reset per second. If multiple paths are connected to the same lun, this statistic is per path when it is expanded to luns. PAECMD/s The number of PAE commands per second. PAECP/s The number of PAE copies per second. SPLTCMD/s The number of split commands per second. SPLTCP/s The number of split copies per second.

Interactive Commands
e Expand/Rollup storage adapter statistics. Allows viewing storage resource utilization statistics broken down by individual channels belonging to an expanded storage adapter. You will be prompted to enter the adapter name. P Expand/Rollup storage adapter statistics. Allows viewing storage resource utilization statistics broken down by individual paths belonging to an expanded storage adapter. You will be prompted to enter the adapter name. a Expand/Rollup storage channel statistics. Allows viewing storage resource utilization statistics broken down by individual targets belonging to an expanded storage channel. You will be prompted to enter the adapter name and the channel ID. t Expand/Rollup storage target statistics. Allows viewing storage resource utilization statistics broken down by individual paths belonging to an expanded storage target. You will be prompted to enter the adapter name, the channel ID and the target ID. r Sort by READS/s column. w Sort by WRITES/s column. R Sort by MBREAD/s column. T Sort by MBWRTN/s column. N Sort first by ADAPTER column, then by CID/TID/LID column. This is the default sort order.  

Storage Device Screen

This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per storage device by default. Statistics can also be viewed per path, world, or partition. See below for more information.

Statistics Descriptions
DEVICE The name of the storage device. PATH The path name. This name is only visible if the corresponding device is expanded to paths. See interactive command 'p' below. WORLD The world id. This id is only visible if the corresponding device is expanded to worlds. See interactive command 'e' below. The world statistics are per world per device. PARTITION The partition id. This id is only visible if the corresponding device is expanded to partitions. See interactive command 't' below. NPH The number of paths. NWD The number of worlds. NPN The number of partitions. SHARES The number of shares. This statistic is only applicable to worlds. BLKSZ The block size in bytes. NUMBLKS The number of blocks of the device. DQLEN The storage device queue depth. This is the maximum number of ESX VMKernel active commands that the device is configured to support. WQLEN The World queue depth. This is the maximum number of ESX VMKernel active commands that the World is allowed to have. Note that this is a per device maximum for the World. It is valid only if the corresponding device is expanded to worlds. ACTV The number of commands in the ESX VMKernel that are currently active. This statistic is only applicable to worlds and devices. QUED The number of commands in the ESX VMKernel that are currently queued. This statistic is only applicable to worlds and devices. %USD The percentage of queue depth used by ESX VMKernel active commands. This statistic is only applicable to worlds and devices. LOAD The ratio of ESX VMKernel active commands plus ESX VMKernel queued commands to queue depth. This statistic is only applicable to worlds and devices. CMDS/s The number of commands issued per second. READS/s The number of read commands issued per second. WRITES/s The number of write commands issued per second. MBREAD/s The megabytes read per second. MBWRTN/s The megabytes written per second. DAVG/cmd The average device latency (millisecs) per command. KAVG/cmd The average ESX VMKernel latency (millisecs) per command. GAVG/cmd The average Guest OS latency (millisecs) per command. QAVG/cmd The average queue latency (millisecs) per command. DAVG/rd The average device read latency (millisecs) per read. KAVG/rd The average ESX VMKernel read latency (millisecs) per read. GAVG/rd The average Guest OS read latency (millisecs) per read. QAVG/rd The average queue read latency (millisecs) per read. DAVG/wr The average device write latency (millisecs) per write. KAVG/wr The average ESX VMKernel write latency (millisecs) per write. GAVG/wr The average Guest OS write latency (millisecs) per write. QAVG/wr The average queue write latency (millisecs) per write. ABRTS/s The number of commands aborted per second. RESETS/s The number of commands reset per second. PAECMD/s The number of PAE commands per second. This statistic is only applicable to paths. PAECP/s The number of PAE copies per second. This statistic is only applicable to paths. SPLTCMD/s The number of split commands per second. This statistic is only applicable to paths. SPLTCP/s The number of split copies per second. This statistic is only applicable to paths.

Interactive Commands
e Expand/Rollup storage world statistics. Allows viewing storage resource utilization statistics broken down by individual worlds belonging to an expanded storage device. You will be prompted to enter the device name. The statistics are per world per device. p Expand/Rollup storage path statistics. Allows viewing storage resource utilization statistics broken down by individual paths belonging to an expanded storage device. You will be prompted to enter the device name. t Expand/Rollup storage partition statistics. Allows viewing storage resource utilization statistics broken down by individual partitions belonging to an expanded storage device. You will be prompted to enter the device name. r Sort by READS/s column. w Sort by WRITES/s column. R Sort by MBREAD/s column. T Sort by MBWRTN/s column. N Sort first by DEVICE column, then by PATH/WORLD/PARTITION column. This is the default sort order.  

Storage VM Screen

This screen displays VM-centric storage statistics. On this screen, statistics are aggregated on a per-resource-pool basis by default. One VM has one corresponding resource pool, so, they are equivalent to per-VM statistics. Statistics can also be viewed on a per-world, or a per-world-per-device basis.

Statistics Descriptions
ID The Resource Pool ID of the running World's Resource Pool or ,B World ID of runnng World. GID The Resource Pool ID of running World's Resource Pool. NAME The name of running World's Resource Pool or name of the running World. Device The storage device name. This name is only visible if corresponding world is expanded to devices. See interactive command 'i' below. NWD The number of worlds. NDV The number of devices. It is valid only if the corresponding resource pool is expanded to worlds SHARES The number of shares. This statistic is only applicable to worlds. It is valid only if the corresponding resource pool is expaneded to worlds BLKSZ The block size in bytes. It is valid only if the corresponding world is expanded to devices. NUMBLKS The number of blocks of the device. It is valid only if the corresponding world is expanded to devices. DQLEN The storage device queue depth. This is the maximum number of ESX VMKernel active commands that the device is configured to support. It is valid only if the corresponding world is expanded to devices. WQLEN The World queue depth. This is the maximum number of ESX VMKernel active commands that the World is allowed to have. It is valid only if the corresponding world is expanded to worlds. ACTV The number of commands in the ESX VMKernel that are currently active. This statistic is only applicable to worlds and devices. QUED The number of commands in the ESX VMKernel that are currently queued. This statistic is only applicable to worlds and devices. %USD The percentage of queue depth used by ESX VMKernel active commands. This statistic is only applicable to worlds and devices. LOAD The ratio of ESX VMKernel active commands plus ESX VMKernel queued commands to queue depth. This statistic is only applicable to worlds and devices. CMDS/s The number of commands issued per second. READS/s The number of read commands issued per second. WRITES/s The number of write commands issued per second. MBREAD/s The megabytes read per second. MBWRTN/s The megabytes written per second. DAVG/cmd The average device latency (millisecs) per command. KAVG/cmd The average ESX VMKernel latency (millisecs) per command. GAVG/cmd The average Guest OS latency (millisecs) per command. QAVG/cmd The average queue latency (millisecs) per command. DAVG/rd The average device read latency (millisecs) per read. KAVG/rd The average ESX VMKernel read latency (millisecs) per read. GAVG/rd The average Guest OS read latency (millisecs) per read. QAVG/rd The average queue read latency (millisecs) per read. DAVG/wr The average device write latency (millisecs) per write. KAVG/wr The average ESX VMKernel write latency (millisecs) per write. GAVG/wr The average Guest OS write latency (millisecs) per write. QAVG/wr The average queue write latency (millisecs) per write. ABRTS/s The number of commands aborted per second. RESETS/s The number of commands reset per second.

Interactive Commands
e Expand/Rollup storage world statistics. Allows viewing storage resource utilization statistics broken down by individual worlds belonging to a group. You will be prompted to enter the group id. The statistics are per world. l Expand/Rollup storage device(=lun) statistics. Allows viewing storage resource utilization statistics broken down by individual devices belonging to an expanded world. You will be prompted to enter the world id. V Display VM instances only. r Sort by READS/s column. w Sort by WRITES/s column. R Sort by MBREAD/s column. T Sort by MBWRTN/s column. N Sort first by VM column, then by WORLD column. This is the default sort order.  

Network Screen

This screen displays server-wide network utilization statistics. On this screen statistics are arranged per port per configured virtual network device. For physical NIC statistics, see the row corresponding to the port that the physical NIC is connected to. For virtual NIC configured in a particular Virtual Machine statistics, see the row corresponding to the port that the virtual NIC is connected to.

Statistics Descriptions
PORT-ID The virtual network device port id. UPLINK Y implies the corresponding port is an uplink. N implies it is not. UP Y implies the corresponding link is up. N implies it is not. SPEED The link speed in MegaBits per second. FDUPLX Y implies the corresponding link is operating at full duplex. N implies it is not. USED-BY The virtual network device port user. TEAM-PNIC The physical NIC name for the team uplink. DTYP The virtual network device type. H implies HUB and S implies switch. DNAME The virtual network device name. PKTTX/s The number of packets transmitted per second. PKTRX/s The number of packets received per second. MbTX/s The MegaBits transmitted per second. MbRX/s The MegaBits received per second. %DRPTX The percentage of transmit packets dropped. %DRPRX The percentage of receive packets dropped. ACTN/s Number of actions per second. This statistic is for internal to VMware use only.

Interactive Commands
T Sort by MbTX/s column. R Sort by MbRX/s column. t Sort by PKTTX/s column. r Sort by PKTRX/s column. N Sort by PORT ID column. This is the default sort order.  

Interrupt Screen

This screen displays interrupt utilization statistics. On this screen statistics are arranged per interrupt vector.

Statistics Descriptions
VECTOR The interrupt vector id. COUNT/s The total number of interrupts per second across all the CPUs. E.g., If you have 2 CPUs, COUNT/s = COUNT_0 + COUNT_1. COUNT_0 and COUNT_1 are described below. COUNT_x The number of interrupts per second on CPU 'x'. TIME/int The average processing time in microseconds per interrupt. TIME_x The average processing time in microseconds per interrupt on CPU 'x'. DEVICES The devices that use the interrupt vector. If the interrupt vector is not enabled for the device, its name is enclosed in "<>", e.g. "<VMK device>".

 

Batch Mode

Allows collecting and saving resource utilization statistics in a file. Running in this mode is a two step process.
 Step  one requires running esxtop in interactive mode, switching to each of the four available screens, selecting the columns on each screen you are interested in and saving this configuration in the ~/.esxtop4rc file using the W interactive command. See Interactive Mode above. Note that the global stats are always available in batch mode.
 Step  two requires running esxtop in batch mode and redirecting the output to a file. This may be done as follows: esxtop -b > foo.csv. Note that the file name needs to have a .csv extension. Esxtop does not enforce this, but post-processing tools mentioned next require this.
 Statistics collected in batch mode may be post-processed using such MS Windows applications like Excel and Perfmon.
 In this mode, esxtop will not accept interactive commands. Esxtop in batch mode runs until it produces the number of iterations requested (see command-line option n below for more details) or until killed using CTRL c.
 In this mode there are several command line options available.  

Command-line Options

b Run esxtop in Batch mode. d Specifies the delay between statistics snapshots. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Collect and save statistics this number of times and then exit.  

Replay Mode

Replays resource utilization statistics collected using vm-support (see vm-support man page for more information), resxtop does not support replay mode. Running in this mode is a two step process.
 Step  one requires running vm-support in snapshot mode on the ESX Service Console. This may be done as follows: vm-support -S -d duration -i interval. The resulting zipped tar file needs to be un-zipped and un-tar'ed before use by esxtop replay mode.
 Step  two requires running esxtop in replay mode. This may be done as follows: esxtop -R vm-support_dir_path. Note that it is not required to run esxtop replay mode on the ESX Service Console.
 In replay mode, esxtop needs to be of the same verison as that of ESX to avoid compatibility problem.
 Replay mode can be run to produce Batch Mode style output (see command-line option b below for more information).
 In replay mode, esxtop accepts the same set of interactive commands as in Interactive Mode. Esxtop in replay mode runs until there are no more vm-support collected snapshots to be read or esxtop produces the number of iterations requested (see command-line option n below for more details).
 In this mode there are several command line options available.  

Command-line Options

R Specifies the path to the vm-support collected snapshot's directory. b Run esxtop in Batch mode. d Specifies the delay between screen updates. Default is 5 seconds. Minimum is 2 seconds. If a delay of less than 2 seconds is specified then the delay is set to 2 seconds. n Number of iterations. Update the display this number of times and then exit.  

EXPERIMENTAL FEATURES

 

Export and Import Entities

This feature allows users show only the entities that they are interested in, such as groups, devices, adapters, ports, interrupt vectors, etc. The ability of selecting interesting entities can also reduce the CPU load of esxtop itself. Two command line options are introduced for this purpose: [export-entity entity-file] and [import-entity entity-file].

Users can do three steps to select entities which they want to display.

(1) Users first export the list of current entities using the export-entity option, by running esxtop -export-entity entity-file. The entity-file is generated, including the ids of all the existing entities at that time.

(2) Then, they can edit the generated entity-file to delete the entities by inserting # before their names.

(3) At last step, users can import entity-file using the import-entity option, by running esxtop -import-entity entity-file. Therefore, esxtop will only show the stats for the entities selected in the entity-file.

Note that these two options are only available in esxtop, not in resxtop.

 

Entity Highlight/Expand/Deletion Interactive Operations

Under interactive mode, users can use keys in numpad to highlight, expand and delete entities.

User can press '8' to move up the highlight cursor, and '2' to move down the highlight cursor.

Users can press '6' to can expand/unroll the selected entity when appropriate. In CPU Screen, it expands/rollups Resource Pool/Worlds CPU statistics. In Storage Device Screen, pressing '6' can rotate the three different expanding modes and rollup modes. Pressing '6' the first time expands storage world statistics; pressing '6' the second time expands storage path statistics; pressing '6' the third time expands storage partition statistics; pressing '6' the fourth time rollups to the default view. In Disk VM Screen, pressing '6' can expand/rollup Resource Pool/Worlds Disk statistics.

Users can press '4' to remove the highlighted entity from display, which will decrease the amount of stats we try to collect. (Note that this delete operation is only available in esxtop, not in resxtop.)

Users can press '.' to restore all the entities and we will collect data for all of them again.

 

NOTES:

Note that the disk VM screen shows the storage stats for a VM. The selection of group entities and disk device entities have effect to this screen. If the group is not chosen, it won't show up. If the device is not chosen, all operations on this device will be counted as zeros.

 

FILES

~/.esxtop4rc The personal configuration file.  

SEE ALSO

vm-support(1), cpu(8), mem(8), diskbw(8), numa(8)  

COPYRIGHT

VMware ESX is Copyright 1998-2009 VMware, Inc. All rights reserved.  

DISCLAIMER

Based on top source code from the OpenBSD distribution.


   Copyright (c) 1997, Jason Downs.  All rights reserved.
   Copyright (c) 1984, 1989, William LeFebvre, Rice University
   Copyright (c) 1989, 1990, 1992, William LeFebvre, 
                                   Northwestern University

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


 

Index

NAME
SYNOPSIS
DESCRIPTION
Interactive Mode
Command-line Options
Common Statistics Description
Common Interactive Commands
The Statistics Columns (fields) and Order Screens
Configuration File
CPU Screen
Memory Screen
Storage Adapter Screen
Storage Device Screen
Storage VM Screen
Network Screen
Interrupt Screen
Batch Mode
Command-line Options
Replay Mode
Command-line Options
EXPERIMENTAL FEATURES
Export and Import Entities
Entity Highlight/Expand/Deletion Interactive Operations
NOTES:
FILES
SEE ALSO
COPYRIGHT
DISCLAIMER

This document was created by man2html, using the manual pages. Brought to you by Bouke Groenescheij, www.jume.nl
Time: 15:25:41 GMT, May 26, 2009