Tag: monitoring

Getting familiar with Splunk – a brief introduction

2 comments

Are you getting started with your journey towards Splunk ? or are you in the early stages in the Splunk learning path ? If your answer is ‘yes’, this post is for you. We will be uncovering some of the very basics about Splunk in this post.

Splunk is a software used to search and analyse machine data. This machine data can come from web applications, sensors, devices or any data created by user. It serves the needs of IT infrastructure by analysing the logs generated in various processes but it can also analyse any structured or semi-structured data with proper data modelling. Splunk performs capturing, indexing, and correlating the real time data in a searchable container and produces graphs, alerts, dashboards and visualisations. Splunk provides easy to access data over the whole organisation for easy diagnostics and solutions to various business problems.

May be an image of text that says "splunk>"

Let’s dive into the details..

Product categories Read more

Splunk is available in three different product categories as follows −

Splunk Enterprise − It is used by companies which have large IT infrastructure and IT driven business. It helps in gathering and analysing the data from websites, applications, devices and sensors, etc.

——————- advertisements ——————-

———————————————————

Splunk Cloud − It is the cloud hosted platform with same features as the enterprise version. It can be availed from Splunk itself or through the AWS cloud platform.

Splunk Light − It allows search, report and alert on all the log data in real time from one place. It has limited functionalities and features as compared to the other two versions.

Features of Splunk 

Data Types: Splunk supports any format and any amount of data -enables centralised log management.
Dashboards and Visualisations : Customised dashboards and data visualisations. Dashboards integrate charts, reports and re-usable panels to display a comprehensive data
Monitoring and Alerting : Continuous monitoring of events, conditions, and critical KPIs helps to have greater visibility into your operations.
Reporting : Reports can be created in real time, scheduled to run at any interval.
Apps and Add-ons : Splunk base has 1000+ apps and add-ons from Splunk, partners and community.

Components of Splunk

——————- advertisements ——————-

———————————————————

The primary components in the Splunk architecture are the forwarder, the indexer, and the search head.

Splunk Forwarder:
The forwarder is an agent you deploy on IT systems, which collects logs and sends them to the indexer. Splunk has two types of forwarders:

Universal Forwarder – forwards the raw data without any prior treatment. This is faster, and requires less resources on the host, but results in huge quantities of data sent to the indexer.
Heavy Forwarder – performs parsing and indexing at the source, on the host machine and sends only the parsed events to the indexer.
Splunk Indexer
The indexer transforms data into events, stores it to disk and adds it to an index.

Splunk Search Head
The search head provides the UI users can use to interact with Splunk. It allows users to search and query Splunk data.

——————- advertisements ——————-

———————————————————

What Splunk can Index

Alternative to Splunk

Sumo Logic : allows you to monitor and visualize historical and real-time events.
Loggly : helps you to collect data from the system using Syslog compatibility.
ELK stack : ELK Stack allows users to take to data from any source, in any format, and to search, analyze, and visualize that data.

Hope this gave you a brief about the software and it’s functions. We’ll be adding more related contents soon. Please feel free to add your feedback/questions in the comments section.

Basic commands for Linux OS Performance Monitoring

Leave a comment

Monitoring the system performance regularly is very much important to ensure the services are being delivered to the end customers without any latency. OS Performance monitoring is an important layer of the entire system performance, along with other layers including application performance, network performance etc…

OS Performance monitoring tools are used for monitoring, visualising, storing, and analysing system-level performance measurements. It allows the monitoring and management of real-time data, and logging and retrieval of historical data.

Red Hat Enterprise Linux provides several tools that can be used from the command line to monitor a system performance.

We are discussing here some of the built-in command line tools for system monitoring.

top

Read more

The top program provides a dynamic real-time view of a running system.  It can display system summary information as well as a list of processes or threads currently being managed by the Linux kernel.

Top command helps the system administrator to find the process and users who utilize more resource in the system.

——————- advertisements ——————-

———————————————————

Let’s see the example below.

top is provided by the procps-ng package. It gives a dynamic view of the processes in a running system. It displays a variety of information, including a system summary and a list of tasks currently being managed by the Linux kernel

ps

It is the abbreviation of “process status”. ps displays information about a selection of the active processes. The output of ps command may vary depends on the parameters we used with it.

Let’s see the example below

——————- advertisements ——————-

———————————————————

ps is provided by the procps-ng package. It captures a snapshot of a select group of active processes. By default, the examined group is limited to processes that are owned by the current user and associated with the terminal where the ps command is executed.

vmstat

It is the abbreviation of virtual memory statistics. vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity.

Let’s see the example below

Virtual memory statistics (vmstat) is provided by the procps-ng package. If we use vmstat as a command with no parameters, it will show you the report  which contains the averages for each of the statistics since the last reboot.

sar

It is the abbreviation of System activity reporter.  It collects and reports information about system activity that has occurred so far on the current day.

sar is provided by the sysstat package.  It can be used to monitor Linux system’s resources like CPU usage, Memory utilization, I/O devices consumption, Network monitoring, Disk usage, process and thread allocation and more.

——————- advertisements ——————-

———————————————————

Let’s see the example below

sar command will show only cpu monitoring activity if any flag is not specifies by user. It displays result on the output screen by default , in addition the result can also be stored in the file specified using  -o filename option.

netstat

It is the abbreviation of network statistics. Netstat prints information about the Linux networking subsystem. Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.

netstat is provided by the package net-tools .By default, netstat displays a list of open sockets.  If you don’t specify any address families, then the active sockets of all configured address families will be printed.

Below example shows how netstat can be used to print the routing table.

iostat

It is the abbreviation of input/output  statistics .The iostat command is used for monitoring system input/output device loading by observing the time the devices are active in relation to their average transfer rates.

——————- advertisements ——————-

———————————————————

Let’s see the example below

iostat is provided by the package sysstat.The iostat command generates reports that can be used to change system configuration to better balance the input/output load between physical disks.

That’s some of the very basic and important commands and it’s usage. Hope it will help you to monitor your system effectively. We will discuss more performance related topics in upcoming posts.

Linux Swap Space Creation and Monitoring

Leave a comment

Overview

This Post is intended to understand the swap creation, monitoring and extending in Redhat Linux.

Swap space is a restricted amount of physical memory that is allocated for use by the operating system when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory. Read more

Recommended System Swap Space
In years past, the recommended amount of swap space increased linearly with the amount of RAM in the system. But because the amount of memory in modern systems has increased into the hundreds of gigabytes, it is now recognized that the amount of swap space that a system needs is a function of the memory workload running on that system. However, given that swap space is usually designated at install time, and that it can be difficult to determine beforehand the memory workload of a system, Redhat recommend determining system swap using the following table.

Amount of RAM in the System Recommended Amount of Swap Space
4GB of RAM or less a minimum of 2GB of swap space
4GB to 16GB of RAM a minimum of 4GB of swap space
16GB to 64GB of RAM a minimum of 8GB of swap space
64GB to 256GB of RAM a minimum of 16GB of swap space
256GB to 512GB of RAM a minimum of 32GB of swap space

Note : On most distributions of Linux, it is recommended that you set swap space while installing the operating system

 

How to Monitor Swap Space

We shall look at different commands and tools that can help you to monitor your swap space usage in your Linux systems as follows

Using the swapon Command

To view all devices marked as swap in the /etc/fstab file you can use the –all option. Though devices that are already working as swap space are skipped

If you want to view a summary of swap space usage by device, use the – summary (swapon –s) option.

[root@nfsserver ~]# swapon –summary
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2097148 0       -1
[root@nfsserver ~]#
[root@nfsserver ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2097148 0       -1
Note :- Use –help option to view more options and information.
Using /proc/swaps

The /proc filesystem is a process information pseudo-file system. It actually does not contain ‘real’ files but runtime system information, for example system memory, devices mounted, hardware configuration and many more.

[root@nfsserver ~]# cat /proc/swaps

Filename                                Type            Size    Used    Priority

/dev/dm-1                               partition       2097148 0       -1

[root@nfsserver ~]#

Using ‘free’ Command
The free command is used to display the amount of free and used system memory. Using the free command with -h option, which displays output in a human readable format.
[root@nfsserver ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        674M        6.5G        9.8M        507M        6.7G
Swap:          2.0G          0B        2.0G
[root@nfsserver ~]#
 Using top Command
To check swap space usage with the help of ‘top’ command
Using the vmstat Command
This command is used to display information about virtual memory statistics
[root@nfsserver ~]# vmstat
procs ———–memory———- —swap– —–io—- -system– ——cpu—–
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 6791708   2784 516484    0    0     7     0   24   23  0  0 100  0  0
[root@nfsserver ~]#
ADDING SWAP SPACE
Sometimes it is necessary to add more swap space after installation
You have three options: create a new swap partition, create a new swap file, or extend swap on an existing LVM2 logical volume. It is recommended that you extend an existing logical volume
Extending Swap on an LVM2 Logical Volume
To extend an LVM2 swap logical volume(suppose /dev/mapper/centos-swap is our swap volume)
1. Disable swapping for the associated logical volume:
[root@nfsserver ~]# swapoff -v /dev/mapper/centos-swap
swapoff /dev/mapper/centos-swap
[root@nfsserver ~]# swapon -s
[root@nfsserver ~]#
2. Resize the LVM2 logical volume by 256 MB
 [root@nfsserver ~]# lvresize /dev/mapper/centos-swap -L +256M
  Size of logical volume centos/swap changed from 2.00 GiB (512 extents) to 2.25 GiB (576 extents).
  Logical volume centos/swap successfully resized.
 [root@nfsserver ~]#
3. Format the new swap space
[root@nfsserver ~]# mkswap /dev/centos/swap
mkswap: /dev/centos/swap: warning: wiping old swap signature.
Setting up swapspace version 1, size = 2359292 KiB
no label, UUID=5e487401-9ae0-4e1d-adff-2346edfc6244
[root@nfsserver ~]#
4. Enable the extended logical volume
[root@nfsserver ~]# swapon -va
swapon /dev/mapper/centos-swap
swapon: /dev/mapper/centos-swap: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/mapper/centos-swap: pagesize=4096, swapsize=2415919104, devsize=2415919104
[root@nfsserver ~]#
5. Test that the logical volume has been extended properly
[root@nfsserver ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.6G        677M        6.5G        9.8M        507M        6.7G
Swap:          2.2G          0B        2.2G
[root@nfsserver ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2359292 0       -1
[root@nfsserver ~]#
Creating an LVM2 Logical Volume for Swap
To add a swap volume group (suppose /dev/centos/swap2 is the new volume)
1. Create the LVM2 logical volume of size 256 MB
[root@nfsserver ~]# lvcreate centos -n swap2 -L 256M
  Logical volume “swap2” created.
[root@nfsserver ~]#
2. Format the new swap space
[root@nfsserver ~]# mkswap /dev/centos/swap2
Setting up swapspace version 1, size = 262140 KiB
no label, UUID=6ea40455-47a0-46bf-844e-ec0ebd4a4e6a
[root@nfsserver ~]#
3. Add the following entry to the /etc/fstab file
/dev/mapper/centos-swap2 swap                    swap    defaults        0 0
4. Enable the extended logical volume
[root@nfsserver ~]# swapon –va
swapon /dev/mapper/centos-swap2
swapon: /dev/mapper/centos-swap2: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/mapper/centos-swap2: pagesize=4096, swapsize=268435456, devsize=268435456
[root@nfsserver ~]#
5. Verify the swap space
[root@nfsserver ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2097148 0       -1
/dev/dm-3                               partition       262140  0       -2
Creating a Swap File
To Add a swap file
1. Determine the size of the new swap file in megabytes and multiply by 1024 to determine the number of blocks. For example, the block size of a 64 MB swap file is 65536.
2. At a shell prompt as root, type the following command with count being equal to the desired block size:
[root@nfsserver ~]# dd if=/dev/zero of=/swapfile bs=1024 count=65536
65536+0 records in
65536+0 records out
67108864 bytes (67 MB) copied, 0.0893063 s, 751 MB/s
[root@nfsserver ~]#
[root@nfsserver ~]# ls -ld /swapfile
-rw-r–r–. 1 root root 67108864 May 17 16:38 /swapfile
[root@nfsserver ~]# du -sh /swapfile
64M     /swapfile
[root@nfsserver ~]#
3. Change the permissions of the newly created file
[root@nfsserver ~]# chmod 0600 /swapfile
[root@nfsserver ~]#
4. Setup the swap file with the command
[root@nfsserver ~]# mkswap /swapfile
Setting up swapspace version 1, size = 65532 KiB
no label, UUID=8a404550-e8a3-4f2b-9daf-137fc34f7b6d
[root@nfsserver ~]#
5. Edit /etc/fstab and enable the newly added swap space
/swapfile          swap            swap    defaults        0 0
[root@nfsserver ~]# swapon -va
swapon /swapfile
swapon: /swapfile: found swap signature: version 1, page-size 4, same byte order
swapon: /swapfile: pagesize=4096, swapsize=67108864, devsize=67108864
[root@nfsserver ~]#
6. Verify the swap space created.
[root@nfsserver ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       2097148 0       -1
/dev/dm-3                               partition       262140  0       -2
/swapfile                               file    65532   0       -3
[root@nfsserver ~]#
Hope this has helped you ..
Thanks!!!!