Tech Admin, Author at Beginner's Forum

Python Virtual Environments – Why and How ?

February 15, 2022 Tech Admin One comment

You might have heard about Virtual Environments in Python already and may not be using it yet. Or you maybe aware of the advantages, but not sure how to set it up. In this post, we are trying to help you connect the dots.

Why you need Virtual Environments ? How to use it ? Let’s see..

Let’s take an example of 2 Python projects where you have the below requirements.

Project 1: Python version 2.7 and NumPy module version 1.17

Project 2: Python version 3.6 and NumPy module version 1.22 Read more

How you are going to achieve this ? And this is just 2 projects, of course you’ll be working on many projects and those will have their own requirements. Virtual Environments is the way..!

——————- advertisements ——————-

———————————————————

Think of Virtual Environments as a individual sections in a wardrobe. You are keeping your dresses in a partition, your spouse’s in a separate partition and kid’s items in another partition. Your kid is a Python project and his/her dresses are the modules/packages required for the project. The partition in wardrobe is the Virtual Environment containing only the required packages without messing with other project’s files. This helps in easy management, as you can have the required packages with respective versions in it’s own Virtual Environment without worrying about the other projects.

Working with Virtual Environments

You’ll need to install virtualenv (pip install virtualenv) if you are using Python 2.x. venv will be pre-installed in Python 3.x. (Let’s consider Python 3.x for the remaining examples)

python -m venv NewEnv

This command will create a New directory named NewEnv and it will have the subdirectories as below..

Scripts – This directory contains all the scripts including activate, deactivate, pip etc…

Include – C headers for the python packages

Lib – Contains the site-packages folder containing all the dependencies

Now you have created the environment. You have to activate the environment for using it. You’ll have the activate (batch) script inside the scripts folder.

C:\Users\bforum\test>NewEnv\Scripts\activate.bat
(NewEnv) C:\Users\bforum\test>

——————- advertisements ——————-

———————————————————

As you can see in the above example, we have run activation for NewEnv and the next line indicates we are in ‘NewEnv’.

When I do a pip list now, it is listing only pip and setup tools in this enviroment.

(NewEnv) C:\Users\bforum\test> pip list

Package Version
———- ——-
pip 19.2.3
setuptools 41.2.0

(NewEnv) C:\Users\bforum\test>

Now I can use pip install ModuleName==Version to install the specific version of modules in this environment. I just installed NumPy and Pandas (and their dependencies). Now the output looks like below.

(NewEnv) C:\Users\bforum\test> pip list

Package Version
————— ——-
numpy 1.22.0
pandas 1.4.1
pip 19.2.3
python-dateutil 2.8.2
pytz 2021.3
setuptools 41.2.0
six 1.16.0

(NewEnv) C:\Users\bforum\test>

——————- advertisements ——————-

———————————————————

You can deactivate the Virtual Environment to come out of it using deactivate (shell) script.

(NewEnv) C:\Users\bforum\test>NewEnv\Scripts\deactivate.bat

C:\Users\bforum\test>

That’s it in this post, hope this helped you. Comments section is open for any feedback/questions.

Getting familiar with Boto3 – AWS SDK for Python

January 5, 2022 Tech Admin Leave a comment

Let’s discuss about Boto3, AWS Software Development Kit (SDK) for Python in this post. Boto3 enables you to perform most of the tasks in almost all the AWS services via Python scripts. For more details on the available services and actions, I would recommend you to refer the Boto3 documentation.

In this post we will have a brief introduction to the SDK including the installation and a few examples.

Installation

Yes, as you expected it is the pip install command. pip install boto3 will take care of the installation and if you want a specific version you can use the below syntax (where version is the required version 1.20.30 for example).

pip install boto3==version Read more

C:\Users\bforum>python -m pip install boto3
Collecting boto3
Downloading boto3-1.20.31-py3-none-any.whl (131 kB)
|████████████████████████████████| 131 kB 168 kB/s
Collecting botocore<1.24.0,>=1.23.31
Downloading botocore-1.23.31-py3-none-any.whl (8.5 MB)
|████████████████████████████████| 8.5 MB 726 kB/s

==== truncated output ====

Successfully installed boto3-1.20.31 botocore-1.23.31

Setting up Boto3

Now that you have installed the module, you have import the same to your program. import boto3 would do it.

——————- advertisements ——————-

———————————————————

For accessing the AWS services, you have to allow the credentials for Boto3 to connect to the environment. You can have the shared credentials (key_id and secret for AWS access updated in the .aws/credentials file. If you are using an EC2 instance, you can use the IAM roles for accessing the AWS environment.

Not at all recommended, but here I am having my credentials inside my script itself.

Basic S3 tasks

aws_secret_access_key=’YOUR_ACCESS_KEY’
s3 = boto3.client(service_name=”s3″,aws_access_key_id=”test_key”,aws_secret_access_key=aws_secret_access_key)

You can replace the above line with just s3 = boto3.client(‘s3’) if you have your credentials defined in the file or is being taken care by IAM.

You can now invoke the below command to create an S3 bucket.

s3.create_bucket(Bucket=’bucket_name’)

# Where bucket_name is the desired bucket name should be unique and must meeting the naming requirements (lowercase,numbers,periods and dashes only, having length of 3-63 characters etc…)

You will get an HTTP 200 response with bucket name and creation details.

Now let’s see how we can list S3 buckets.

s3 = boto3.client(‘s3’)
s3.list_buckets()

——————- advertisements ——————-

———————————————————

the above list_buckets command will list the existing buckets, but that will be HTTP response in JSON format. To filter only the names, let’s use the below commands (basically a for loop).

out = s3.list_buckets()
for bucket in out[‘Buckets’]:
print(bucket[“Name”])

The output will be the bucket names as below

beginnersforum-bf1
beginnersforum-bf2
beginnersforum1crrs3

You can download a file from S3 using the below commands

bucket_name = ‘beginnersforum-bf1’
keyname = ‘Test Text file1.txt’
output_filename = ‘test1.txt’
s3.download_file(bucket_name, keyname, output_filename)

Few EC2 examples

ec2 = boto3.client(‘ec2’) #will create an EC2 client which can be used in the upcoming commands.

start an instance : ec2.start_instances(InstanceIds=[instance_id])

Reboot an instance : ec2.reboot_instances(InstanceIds=[‘Instance_ID’])

Shutdown an instance : ec2.stop_instances(InstanceIds=[instance_id])

That was just a very basic intro to this Python module. Based on your need, you may refer to the specific service and action details in the SDK documentation. Hope this post helped you in understanding the module, please feel free to have your thoughts/feedback in the comments section.

Machine Learning / Artificial Intelligence – basic pre-requisites to get started

May 30, 2021 Tech Admin Leave a comment

Machine Learning or Artificial Intelligence technologies are booming (or should I say boooooming..!!) nowadays, maybe already. Many of you are planning to get started with your AI or ML studies, but not clear on where to start and what are the pre-requisites. Yes, a bit different area from your current Infrastructure/Software/Database/Coding domain and which requires a bit (not really, a lot more) of additional learning.

Let’s try to take a look at the most common pre-requisites for getting started with your AI/ML journey.

Mathematics

Maths, but how much of maths ??

Not too much, it is basically a refresh required for your high-school maths. You just have to remember linear algebra (linear equations, matrices and vectors) stuff and basics of calculus (rate of change, integration, differentiation etc…). Remember, it is required for you to understand how the models work. You do not have to do all these calculations etc… by your own, or you do not have to code all these in maths terms. The model will do it, maths is for you to understand what is happening in the back-end.

——————- advertisements ——————-

———————————————————

Statistics and Probability

It’s a lot to learn, if you are not too good in it. You should be having a good level of statistics knowledge to understand the AI/ML stuff, as the basics are all based on the statistics terms.

You should be well versed with the descriptive statistics including the mean, median, standard deviation, variance etc… of the data. Same applies with the Probability theory. You have to be too good with it, able to understand the data distribution including distribution functions PDF (Probability Density Function) and CDF (Cumulative Density Function) (and PMF etc…).

——————- advertisements ——————-

———————————————————

Histograms and other common plotting – visualisation – methods, a better understanding of data (covariance, correlation etc…).

The more statistics you know the better you can understand.

Programming

Is it necessary to know coding ??

I would say, yes. There are MLaaS (Machine Learning as a Service) tools which will do the most part for you where you don’t have to write a single line of code. But I would prefer coding it yourself for a better understanding and that maybe necessary based on your job role.

Python – is where everyone gets landed, based on its popularity and it being Machine Learning friendly (just kidding, it has rich set of modules available for ML). If you are good at R, C/C++, Java, Julia, Scala, Ruby… still you are at a better position, 1 item (programming) is already checked.

——————- advertisements ——————-

———————————————————

Useful links

Let’s me add it soon…

I believe that covers the basic requirements to get started with your ML/AI learning. The more you work with data, the better you would be with it. It requires a lot, lot of learning and even more practice. Please feel free to add your suggestions/queries in the comments section.

How to resize a linux filesytem without Data Loss

November 29, 2014 Tech Admin Leave a comment

Today i’m going to show you how to resize a filesystem without data loss.This can be quite useful, if you do not use the LVM technology.

Note:- Before doing this in live enviornment, kindly test it in your testbox
Here i’m using my susebox .
Tools Used :- fdisk, umount, fsck, tune2fs, e2fsck, resize2fs.

I’m going to resize the /dev/sdb1 in this tutorial. ( if your partition is named diferently, kindly replace it) Read more

In the above screenshot, you can able to see the harddisk size, mount point etc.

So /dev/sdb harddisk is of 16GB size

/dev/sdb1 is 8.7 GB . and it’s mounted to /testdir directory.

To extend the size of /dev/sdb1 , the filesystem should be unmounted. Here /dev/sdb1 is mounted on /testdir, run the umount command to unmount the same.

Now run fsck -n /dev/sdb1 for checking the filesystem.

Then we need to remove the journal from the filesystem , this will bring ext2 filesystem to /dev/sdb1

Now run e2fsck to check the filesystem,

Now we can resize the filesystem using resize2fs tool.

Resize2fs can resize ext2 file systems, but not ext3 file systems, that’s why we changed /dev/sdb1 to ext2 by removing journel from it.

8.7GB is the size of /dev/sdb1 (check the above df -h screenshot). Now we are going to extend the /dev/sdb1 from 8.7 GB to 15 GB .

Note :- In case of shrinking data , don’t make it smaller than 1.9GB (space used as per df-kh output), you will lose data!

So we are going to run resize2fs /dev/sdb1 15000M

Kindly note the number of blocks (384000) and their size (4k). We need it later..

Now we are going to delete the partition /dev/sdb1 using fdisk tool

” Dont worry about your data 🙂 ”

Next we will create a new /dev/sdb1 partition. It was a primary partition earlier, so choose p again, and again it is our partition no. 1:

Then comes the important part,

Size of the partition:-

The First Cylinder is not an issue , it’s default (1). But in the case of Last Cylinder , we don’t have the value , So we can specify the size in kilobyes (K) .

We can calculate the size like this ,

Last Cylinder Size = The amount of blocks from the resize2fs output * the size of block

// amount of blocks = 384000 .

// Size of block = 4k

Last Cylinder Size = 384000 * 4k = 1536000K

Apply this with +1536000K and write the partition using ‘w’

Then run fsck -n /dev/sdb1.

Then create the journal on our new /dev/sdb1, thus turning it into an ext3 partition again:

Check the size of filesystem,

Check with df -h and you are good to mount the filesystem.

Hope this helped.

LINUX
UNIX

How to use sudo – root privilege for a normal user

November 16, 2014 Tech Admin 2 comments

Sudo works under Linux / Mac OS X and all UNIX like operating systems. /etc/sudoers (config file that defines or list of who can run what) allows you to delegate authority to give certain users or groups of users the ability to run various commands as the superuser(root) or another user without needing the root password. This is useful for delegating roles and permissions to other users without sharing the root password.

This file must be edited with the visudo command as the root user . The sudo command allows users to do tasks on a Linux system as another user.

How do I use sudo?

Let us take an example here,

If you want to give the user John, access to restart httpd service. First, Login as root user, then use visudo command edit the config file:

# visudo

Append the following lines to file:

John ALL= /etc/init.d/httpd restart

Save and close file . Now John user can restart httpd service by typing the following command:

$ sudo /etc/init.d/httpd restart

Output:

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
Password:
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

The sudo command has logged the attempt to the log file /var/log/secure or /var/log/auth.log file:

# tail -f /var/log/secure

Sample outputs:

Nov 15 06:05:36 localhost sudo: John : TTY=pts/1 ; PWD=/home/John ; USER=root ; COMMAND=/etc/init.d/httpd restart

Before running a command with sudo, users usually supply their password. Once authenticated, and if the /etc/sudoers configuration file permits the user access, the command will be run. sudo logs each command run.
a) If you want to allow John to run various commands:

John ALL=/sbin/halt, /bin/kill, /etc/init.d/httpd

b) Allow user John to restart httpd without any password i.e. as root without authenticating himself:

John ALL= NOPASSWD: /etc/init.d/httpd restart

c) Allow user Alex to run any command from /usr/bin directory:

Alex ALL = /usr/bin/*

d) Allow user Alex to run ALL commands:

Alex ALL =(ALL) ALL

e) If you want to allow system admin user to run all commands (add users to a group eg:- unixsa)
Append the following line:

## Allows people in group unixsa to run all commands
 %unixsa ALL=(ALL) ALL

Save and close the file. Finally, add a group called unixsa:

# groupadd unixsa

Add a user called Alex (existing user) to group unixsa:

# usermod -a -G unixsa Alex

Verify group membership:

# id Alex

Sample Outputs:

uid=5001(Alex) gid=5001(Alex) groups=5001(Alex),110(unixsa)

$ sudo /etc/init.d/network restart

f) If database admin user want to run command as oracle

Append the following lines to /etc/sudoers file:

John ALL=(oracle) /u01/app/oracle/product/11.1.0/db_1/bin/dbstart

In this case, it’s that indicating the user John can execute the dbstart command as oracle occurs. When using sudo to assume the role of a user other than root, use the -u option and give the user name as an argument, followed by the command that should be executed.

$ whoami
John

$ sudo -u oracle /u01/app/oracle/product/11.1.0/db_1/bin/dbstart

Output:

Password:
 Processing Database instance "TEST": log file /u01/app/oracle...

That’s all, we have tried all the common options for the sudo command. You may have your queries in the comments section.

LINUX
UNIX

Red Hat Enterprise Linux specification comparison, RHEL 3/4/5/6 and RHEL 7

November 8, 2014 Tech Admin Leave a comment

Let’s have a look at the specification comparison between different recent versions. Here is list of Hardware and software features (from RedHat) in RHEL 3/4/5/6 and the RHEL 7.

Specification	Version 3	Version 4	Version 5	Version 6	Version 7
Maximum logical CPUs
x86	16	32	32	32	N/A
Itanium 2	8	256 [512]	256 [1024]	N/A	N/A
x86_64	8	64 [64]	160 [255]	240 [4096]	160 [5120]
POWER	8	64 [128]	128	128	128
System z	64 (z900)	64 (z10 EC)	101 (zEC12)	101 (zEC12)	101 (zEC12)
Maximum memory
x86	64GB	64GB	16GB	16GB	N/A
Itanium 2	128GB	2TB	2TB	N/A	N/A
x86_64	128GB	256GB [1TB]	1TB	6TB [64TB]	3TB [64TB]
POWER	64GB	128GB [1TB]	512GB [1TB]	2TB	2TB
System z	256GB (z900)	1.5TB (z10 EC)	3TB (z196)	3TB (z196)	3TB (z196)
Maximum number of device paths (“sd” devices)
	256	256	1,024	8,192	10,000
Required minimums
x86	256MB	256MB	512MB minimum/1 GB/logical CPU recommended	512MB minimum/1 GB/logical CPU recommended	N/A
x86_64	256MB	256MB	512MB minimum/1 GB/logical CPU recommended	1GB minimum/1 GB/logical CPU recommended	1GB minimum/1 GB/logical CPU recommended
Itanium 2	512MB	512MB	512MB/1 GB/logical CPU recommended	N/A	N/A
POWER	512MB	512MB	1GB minimum/2GB recommended	2GB minimum/2GB required per install	2GB minimum/2GB required per install
Minimum diskspace	800MB	800MB	1GB minimum/5GB recommended	1GB minimum/5GB recommended	10GB minimum/20GB recommended
File systems and storage limits
Maximum filesize (Ext3)	2TB	2TB	2TB	2TB	2TB
Maximum file system size (Ext3)	2TB	8TB	16TB	16TB	16TB
Maximum file size (Ext4)	—	—	16TB	16TB	16TB
Maximum file system size (Ext4)	—	—	16TB [1EB]	16TB [1EB]	50TB [1EB]
Maximum file size (GFS)	2TB	16TB [8EB]	16TB [8EB]	N/A	N/A
Maximum file system size (GFS)	2TB	16TB [8EB]	16TB [8EB]	N/A	N/A
Maximum file size (GFS2)	—	—	100TB [8EB]	100TB [8EB]	100TB [8EB]
Maximum file system size (GFS2)	—	—	100TB [8EB]	100TB [8EB]	100TB [8EB]
Maximum file size (XFS)	—	—	100TB [8EB]	100TB [8EB]	500TB [8EB]
Maximum file system size (XFS)	—	—	100TB [16EB]	100TB [16EB]	500TB [16EB]
Maximum Boot LUN size (BIOS)	—	—	2TB	2TB	2TB
Maximum Boot LUN size (UEFI)	—	—	N/A	32bit (i686) – 2TB, 64bit – 16TB (tested limit)	50TB
Maximum x86 per-process virtual address space	Approx. 4GB	Approx. 4GB	Approx. 3GB	Approx. 3GB	N/A
Maximum x86_64 per-process virtual address space		512GB	2TB	128TB	128TB
Kernel and OS features
Kernel foundation	Linux 2.4.21	Linux 2.6.9	Linux 2.6.18	2.6.32 – 2.6.34	3.10
Compiler/toolchain	GCC 3.2	GCC 3.4	GCC 4.1	GCC 4.4	GCC 4.8.2
Languages supported	10	15	19	22	22
NIAP/CC certified	Yes (3+)	Yes (4+)	Yes (4+)	Yes (4+)	Under Evaluation (4+)
Common Criteria certified KVM	—	—	Evaluated	Evaluated	Under Evaluation
IPv6	—	—	Ready Logo Phase 2	Ready Logo Phase 2	Under Evaluation
FIPS certified	—	—	Yes (7 modules)	Yes (8 modules)	Under Evaluation (9 modules)
Common Operating Environment (COE) compliant	Yes	Yes	N/A	N/A	N/A
LSB-compliant	Yes – 1.3	Yes – 3	Yes – 3.1	Yes – 4.0	Under Evaluation (4.1)
GB18030	No	Yes	Yes	Yes	Yes
Client environment
Desktop GUI	Gnome 2.2	Gnome 2.8	Gnome 2.16	Gnome 2.28	Gnome 3.8
Graphics	XFree86	X.org	X.org 7.1.1	X.org 7.4	X.org 7.7
OpenOffice	V1.1	V1.1.2	V2.0.4	V3.2	LibreOffice V4.1.4
Gnome Evolution	V1.4	V2.0	V2.8.0	V2.28	V3.8.5
Default browser	Mozilla	Firefox	Firefox 1.5	Firefox 3.6	Firefox 24.5

LINUX
UNIX

Structure of Linux

November 8, 2014 Tech Admin One comment

Here we will be discussing the basics of LINUX, starting with the structure of the OS. The Basic components will be the Kernel, Shell and FHS (Filesystem Hierarchy Standard) . Let’s go through these components in detail..

The Kernel

The Kernel is the Heart of an operating system (OS). Kernel manages everything of a Linux OS. It performs tasks that create and maintain the Linux environment. It is the communication channel between the hardware and shell. The hardware operations are directed by Shell and are passed through to the right hardware by Kernel.

SHELL

Simply put, the shell is a program that takes your commands from the keyboard and gives them to the operating system to perform. . It is the interface between you and Linux . The commands which we issued , which is interpreted and passed on to the kernel for processing.

There are variable types of shells available in Linux. Most Linux varieties use Bourne-Again shell(bash) but support various others: Korn Shell, Bourne shell, C shell, etc. For all intensive purposes you can just stick with bash but I will show you how to change this if you want to. As you advance you can use shells to create scripts to automate tasks, making your daily routine all the more easier

Filesystem Hierarchy Standard

Linux uses the Filesystem Hierarchy Standard (FHS) file system structure, which defines the names, locations, and permissions for many file types and directories.

/ – The root directory. Everything in Linux is under root directory. The first stage of Linux filesystem structure.

/bin – Most of the command binary files are under this directory. General commands including ls, man, mv etc… are all under /bin.

/boot – Necessary boot files for Linux are kept under /boot.

/dev – The device files are located in /dev

/etc – Configuration files are saved under /etc. This directory is just like the control panel for a Windows host.

/home – For all users except the root user, the home directory will be placed in /home. For e.g, /home/beginnersforum.

/media – /media will be used for mounting external storage devices

/mnt – Temporary mounted file systems

/opt — Application programs directory

/sbin — Additional system binaries.

/tmp – As the name indicates, a temporary directory for files.

/usr – The largest directory in Linux. Application files and config files are also kept under here

/var – var stands for variable , that the Variable data on a system. Data that will change as the system is running (Log files, backups, cache, etc.)

/root – Home directory for root as already discussed

This directory structure is everything in Linux.

We can conclude the basic components of Linux here even-though the topic is very deep. More in coming posts. Enjoy reading…!

LINUX
UNIX