Basic commands for Linux OS Performance Monitoring

Monitoring the system performance regularly is very much important to ensure the services are being delivered to the end customers without any latency. OS Performance monitoring is an important layer of the entire system performance, along with other layers including application performance, network performance etc…

OS Performance monitoring tools are used for monitoring, visualising, storing, and analysing system-level performance measurements. It allows the monitoring and management of real-time data, and logging and retrieval of historical data.

Red Hat Enterprise Linux provides several tools that can be used from the command line to monitor a system performance.

We are discussing here some of the built-in command line tools for system monitoring.

top

Read more

The top program provides a dynamic real-time view of a running system.  It can display system summary information as well as a list of processes or threads currently being managed by the Linux kernel.

Top command helps the system administrator to find the process and users who utilize more resource in the system.

——————- advertisements ——————-

———————————————————

Let’s see the example below.

top is provided by the procps-ng package. It gives a dynamic view of the processes in a running system. It displays a variety of information, including a system summary and a list of tasks currently being managed by the Linux kernel

ps

It is the abbreviation of “process status”. ps displays information about a selection of the active processes. The output of ps command may vary depends on the parameters we used with it.

Let’s see the example below

——————- advertisements ——————-

———————————————————

ps is provided by the procps-ng package. It captures a snapshot of a select group of active processes. By default, the examined group is limited to processes that are owned by the current user and associated with the terminal where the ps command is executed.

vmstat

It is the abbreviation of virtual memory statistics. vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

Let’s see the example below

Virtual memory statistics (vmstat) is provided by the procps-ng package. If we use vmstat as a command with no parameters, it will show you the report  which contains the averages for each of the statistics since the last reboot.

sar

It is the abbreviation of System activity reporter.  It collects and reports information about system activity that has occurred so far on the current day.

sar is provided by the sysstat package.  It can be used to monitor Linux system’s resources like CPU usage, Memory utilization, I/O devices consumption, Network monitoring, Disk usage, process and thread allocation and more.

——————- advertisements ——————-

———————————————————

Let’s see the example below

sar command will show only cpu monitoring activity if any flag is not specifies by user. It displays result on the output screen by default , in addition the result can also be stored in the file specified using  -o filename option.

netstat

It is the abbreviation of network statistics. Netstat prints information about the Linux networking subsystem. Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

netstat is provided by the package net-tools .By default, netstat displays a list of open sockets.  If you don’t specify any address families, then the active sockets of all configured address families will be printed.

Below example shows how netstat can be used to print the routing table.

iostat

It is the abbreviation of input/output  statistics .The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

——————- advertisements ——————-

———————————————————

Let’s see the example below

iostat is provided by the package sysstat.The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

That’s some of the very basic and important commands and it’s usage. Hope it will help you to monitor your system effectively. We will discuss more performance related topics in upcoming posts.

Mirror disk replacement in solaris

Let’s here discuss how to replace the failed root mirror disk in solaris under SVM.

==========================================================

1.Identify the faluty disk and its partition

root@soalris /root>metastat -c
d65 m 29GB d63 d61 (maint)
d63 s 29GB c1t1d0s6
d61 s 29GB c1t0d0s6 (maint) <—— Disk showing in maintanace state and it has to be replaced

d30 m 9.8GB d31 d32 (maint)
d31 s 9.8GB c1t0d0s3 (maint) <——
d32 s 9.8GB c1t1d0s3

d55 m 5.9GB d53 d51 (maint)
d53 s 5.9GB c1t1d0s5
d51 s 5.9GB c1t0d0s5 (maint) <——

Read more

d45 m 7.8GB d43 d41 (maint)
d43 s 7.8GB c1t1d0s4
d41 s 7.8GB c1t0d0s4 (maint) <——
d5 m 7.8GB d3 d1 (maint)
d3 s 7.8GB c1t1d0s0
d1 s 7.8GB c1t0d0s0 (maint) <——

d10 m 7.8GB d11 d12 (maint)
d11 s 7.8GB c1t0d0s1 (maint) <——
d12 s 7.8GB c1t1d0s1

2. So we have identified the disk c1t0d0 as faulty which is showing need maintanace in the metastat output.

3. Confirm the disk is having errors in the iostat output and /var/adm/messages also.

root@solaris /root>iostat -En
c1t0d0 Soft Errors: 173 Hard Errors: 0 Transport Errors: 0
Vendor: HITACHI Product: H101473SCSUN72G Revision: SA23 Serial No: 0810DTE8YA
Size: 73.41GB <73407865856 bytes>

4.identify the boot path

prtconf -vp|grep bootpath

bootpath: ‘/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/disk@2,0:a’

echo|format

c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@2,0

So the primary boot disk is c1t1d0. And the faulty disk is its mirror.

Steps to do unconfigure the faulty disk before proceeding with the replacement. (its an online activity)

=========================================================================

1. Take all neccessory pre outputs as below.

df -h,metastat -c,metadb,echo | format,cat /etc/vfstab,swap -l

2. Detach the faluty submirros

metadetach d65 d61

metadetach d30 d31

metadetach d55 d51

metadetach d45 d41

metadetach d5 d1

metadetach d10 d11

3. Delete the metadb information from the disk.

# metadb -d /dev/dsk/c1t0d0s7

4.  Use the cfgadm command to display all the disks in the server

cfgadm -al

root@usoponshpamf4g /root>cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 scsi-bus connected configured unknown

c1::dsk/c1t0d0 disk connected configured unknown <——- Failed Disk

c1::dsk/c1t1d0 disk connected configured unknown
c1::dsk/c1t2d0 disk connected configured unknown
c2 fc-fabric connected configured unknown
c2::500009720822514c disk connected configured unknown
c3 fc-fabric connected configured unknown
c3::5000097208225168 disk connected configured unknown

On identifying the disk to be removed, unconfigure the disk. You may have to use -f along with -c to forcibly remove the disk in some cases.

cfgadm -c unconfigure c1::dsk/c1t0d0

 

5). Verify the status of the disk in cfgadm -al command. It should show unconfigured and unavailable.

# cfgadm -al

c1::dsk/c1t0d0 connected unconfigured unknown

You can safely remove the disk from the server now.

6.  Request FE to insert the new disk into the disk slot of the server and run the below command.

# devfsadm

You should see the new disk detected in the OS:

 

Steps to configure the newly added disk

==============================

1. cfgadm -c configure c1::dsk/c1t0d0

2. verify the disk is configured

c1::dsk/c1t0d0 available connected configured unknown

2. copy the prtvtoc from the primary disk.

prtvtoc /dev/rdsk/c1t1d0s2 | fmthard -s – /dev/rdsk/c1t0d0s2

3. now add the metadb in the disk.

metadb -afc3 /dev/dsk/c1t0d0s7

4.Install the bootblk on slice 0 of the new disk.

installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0

5.Update the device ID in the SVM database

metadvadm -u c1t0d0

6. Attach the detach sub mirrors using the metaattach command. The syntax to do so is :

metattach d65 d61

metattach d30 d31

metattach d55 d51

metattach d45 d41

metattach d5 d1

metattach d10 d11

7. You can verify the resync status by using metastat -c command.

 

Thank You !!!