Troubleshooting NFS Mount Issues in Linux

Network File System (NFS) is a protocol which allows a system to share directories and files with others over a network. By using NFS, users and programs can access files on remote systems almost as if they were local files.

This post refers how to mount the network share in our local system and what are all the common issues and how to generally troubleshoot connectivity and config issues.

NFS Client Configuration

1. Install the required nfs packages if not already installed on the server Read more

# rpm -qa | grep nfs-utils


# yum install nfs-util

2. Use the mount command to mount exported file systems. Syntax for the command:

# mount -t nfs -o options host:/remote/export /local/directory 

——————- advertisements ——————-  

———————————————————

Example :

# mount -t nfs -o ro,nosuid remote_host:/home /remote_home

This example does the following:
– It mounts /home from remote host (remote_host) on local mount point /remote_home.
– File system is mounted read-only and users are prevented from running a setuid program (-o ro,nosuid options).

3. Update /etc/fstab to mount NFS shares at boot time.

# vi /etc/fstab


remote_host:/home      /remote_home nfs        ro,nosuid           0            0

Troubleshooting NFS connectivity issues

Depending on the client and the issue, wide range of error messages can appear while trying to mount an NFS share, it might also take forever to mount, or even mount normally but the mount points will be empty.Below are the common errors we face in the client side while mounting the NFS/NAS shares.

——————- advertisements ——————-  

———————————————————

Error 1: 

mount: mount to NFS server 'NFS-Server' failed: System Error: No route to host.

This can be caused by the RPC messages being filtered by either the host firewall, the client firewall, or a network switch. Verify if a firewall is active and if NFS traffic is allowed. Normally nfs is using port 2049.

  1. Check the show mount output of the server to verify the filesystem has exported for the client ip.
# showmount –e <NFS server IP > | grep –I  <clientIP>

Check the port Connectivity of the NFS server using telnet

# telnet <NFS server IP> 2049

 

Error 2:

mount_nfs: can't mount / from 1.2.3.4 onto /mnt: RPC prog. not avail

Error: “mount clntudp_create: RPC: Port mapper failure – RPC: Unable to receive

The Linux NFS implementation requires that both the NFS service and the portmapper (RPC) service be running on both the client and the server. Check it like this:

——————- advertisements ——————-  

———————————————————

            # rpcinfo -p
     program vers proto   port  service

    100000    4   tcp    111  portmapper

    100000    3   tcp    111  portmapper

    100000    2   tcp    111  portmapper

    100000    4   udp    111  portmapper

    100000    3   udp    111  portmapper

    100000    2   udp    111  portmapper...

          # ]# systemctl status rpcbind
  •  rpcbind.service - RPC bind service
             Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; indirect; vendor preset: enabled)

            Active: active (running) since Fri 2018-05-18 12:39:15 IST; 2s ago

            Process: 15222 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited,      status=0/SUCCESS)

 Main PID: 15223 (rpcbind)

            CGroup: /system.slice/rpcbind.service

           └─15223 /sbin/rpcbind -w

 

May 18 12:39:15 nfsserver systemd[1]: Starting RPC bind service...

May 18 12:39:15 nfsserver systemd[1]: Started RPC bind service.

If not, start it with the commands give below.

# systemctl start rpcbind

——————- advertisements ——————-  

———————————————————

Error 3: 

Error: “NFS Stale File Handle”

Unlike traditional Linux file systems that allow an application to access an open file even if the file has been deleted using unlink or rm, NFS does not support this feature. An NFS file is deleted immediately. Any program which attempts to do further I/O on the deleted file will receive the “NFS Stale File Handle” error. For example, if your current working directory is an NFS directory and is deleted, you will see this error at the next shell prompt.

To refresh the client’s state with that of the server you may do a lazy unmount the mount point and remount it

# umount -l /mnt/mount_point

or kill the process, which references the mounted file system:

# fuser -k [mounted-filesystem].

——————- advertisements ——————-  

———————————————————

Error 4:

Error: “Access Denied” or “Permission Denied

Check the export permissions for the NFS file system. You can do this from the client:

# showmount -e server_name

 

Error 5:

Error: “rpc mount export: RPC: Timed out

Unable to access file system at [NFS SERVER]: rpc mount export: RPC: Timed out This is caused by DNS name resolution issue. NFS(RPC) needs reverse name resolution. If NFS server or client cannot resolve their name, this error occurs. In case gets the error message, check DNS configuration and /etc/hosts configuration.

 

Hope we have covered almost all the regular errors and steps for solving those. Please share your thoughts in the comments section. If you want us to add any additional issues-resolution, kindly let us know.

Thanks for reading..!

Mirror disk replacement in solaris

Let’s here discuss how to replace the failed root mirror disk in solaris under SVM.

==========================================================

1.Identify the faluty disk and its partition

[email protected] /root>metastat -c
d65 m 29GB d63 d61 (maint)
d63 s 29GB c1t1d0s6
d61 s 29GB c1t0d0s6 (maint) <—— Disk showing in maintanace state and it has to be replaced

d30 m 9.8GB d31 d32 (maint)
d31 s 9.8GB c1t0d0s3 (maint) <——
d32 s 9.8GB c1t1d0s3

d55 m 5.9GB d53 d51 (maint)
d53 s 5.9GB c1t1d0s5
d51 s 5.9GB c1t0d0s5 (maint) <——

Read more

d45 m 7.8GB d43 d41 (maint)
d43 s 7.8GB c1t1d0s4
d41 s 7.8GB c1t0d0s4 (maint) <——
d5 m 7.8GB d3 d1 (maint)
d3 s 7.8GB c1t1d0s0
d1 s 7.8GB c1t0d0s0 (maint) <——

d10 m 7.8GB d11 d12 (maint)
d11 s 7.8GB c1t0d0s1 (maint) <——
d12 s 7.8GB c1t1d0s1

2. So we have identified the disk c1t0d0 as faulty which is showing need maintanace in the metastat output.

3. Confirm the disk is having errors in the iostat output and /var/adm/messages also.

[email protected] /root>iostat -En
c1t0d0 Soft Errors: 173 Hard Errors: 0 Transport Errors: 0
Vendor: HITACHI Product: H101473SCSUN72G Revision: SA23 Serial No: 0810DTE8YA
Size: 73.41GB <73407865856 bytes>

4.identify the boot path

prtconf -vp|grep bootpath

bootpath: ‘/[email protected],600000/[email protected]/[email protected]/[email protected]/[email protected]/[email protected]/[email protected],0:a’

echo|format

c1t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/[email protected],600000/[email protected]/[email protected]/[email protected]/[email protected]/[email protected]/[email protected],0

So the primary boot disk is c1t1d0. And the faulty disk is its mirror.

Steps to do unconfigure the faulty disk before proceeding with the replacement. (its an online activity)

=========================================================================

1. Take all neccessory pre outputs as below.

df -h,metastat -c,metadb,echo | format,cat /etc/vfstab,swap -l

2. Detach the faluty submirros

metadetach d65 d61

metadetach d30 d31

metadetach d55 d51

metadetach d45 d41

metadetach d5 d1

metadetach d10 d11

3. Delete the metadb information from the disk.

# metadb -d /dev/dsk/c1t0d0s7

4.  Use the cfgadm command to display all the disks in the server

cfgadm -al

[email protected] /root>cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 scsi-bus connected configured unknown

c1::dsk/c1t0d0 disk connected configured unknown <——- Failed Disk

c1::dsk/c1t1d0 disk connected configured unknown
c1::dsk/c1t2d0 disk connected configured unknown
c2 fc-fabric connected configured unknown
c2::500009720822514c disk connected configured unknown
c3 fc-fabric connected configured unknown
c3::5000097208225168 disk connected configured unknown

On identifying the disk to be removed, unconfigure the disk. You may have to use -f along with -c to forcibly remove the disk in some cases.

cfgadm -c unconfigure c1::dsk/c1t0d0

 

5). Verify the status of the disk in cfgadm -al command. It should show unconfigured and unavailable.

# cfgadm -al

c1::dsk/c1t0d0 connected unconfigured unknown

You can safely remove the disk from the server now.

6.  Request FE to insert the new disk into the disk slot of the server and run the below command.

# devfsadm

You should see the new disk detected in the OS:

 

Steps to configure the newly added disk

==============================

1. cfgadm -c configure c1::dsk/c1t0d0

2. verify the disk is configured

c1::dsk/c1t0d0 available connected configured unknown

2. copy the prtvtoc from the primary disk.

prtvtoc /dev/rdsk/c1t1d0s2 | fmthard -s – /dev/rdsk/c1t0d0s2

3. now add the metadb in the disk.

metadb -afc3 /dev/dsk/c1t0d0s7

4.Install the bootblk on slice 0 of the new disk.

installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c1t0d0s0

5.Update the device ID in the SVM database

metadvadm -u c1t0d0

6. Attach the detach sub mirrors using the metaattach command. The syntax to do so is :

metattach d65 d61

metattach d30 d31

metattach d55 d51

metattach d45 d41

metattach d5 d1

metattach d10 d11

7. You can verify the resync status by using metastat -c command.

 

Thank You !!!

How to create a ufs filesystem on a newly added disk in Solaris

Let’s here discuss how to create a new Filesystem on a newly added disk in Solaris. Below is the scenario shown with examples.

HDD

 

1. Once disk is added in the server ,check if it is visible in OS by running the below command

bash-3.00# echo | format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c0d1 <DEFAULT cyl 2607 alt 2 hd 255 sec 63>
/[email protected],0/[email protected],1/[email protected]/[email protected],0
Specify disk (enter its number): Specify disk (enter its number):
bash-3.00# Read more

2. If it is not visible in the OS run the below command to scan the newly added hardware.

bash-3.00# devfsadm          ===>(it works only for Solaris 10 and later versions. The older versions you need to perform a reconfigure reboot)

3. Check the format command again to confirm the disk is showing in the OS.

bash-3.00# echo | format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c0d1 <DEFAULT cyl 2607 alt 2 hd 255 sec 63>
/[email protected],0/[email protected],1/[email protected]/[email protected],0
1. c1t0d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>
/[email protected],0/pci15ad,[email protected]/[email protected],0
Specify disk (enter its number): Specify disk (enter its number):
bash-3.00#


========================================================================================

4. Now we can partition the disk

bash-3.00# format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c0d1 <DEFAULT cyl 2607 alt 2 hd 255 sec 63>
/[email protected],0/[email protected],1/[email protected]/[email protected]1,0
1. c1t0d0 <DEFAULT cyl 1303 alt 2 hd 255 sec 63>
/[email protected],0/pci15ad,[email protected]/[email protected],0
Specify disk (enter its number): 1
selecting c1t0d0
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
fdisk – run the fdisk program
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
!<cmd> – execute <cmd>, then return
quit
format> format> p                         ===> “P” is used for partition
WARNING – This disk may be in use by an application that has
modified the fdisk table. Ensure that this disk is
not currently in use before proceeding to use fdisk.       ===> if we get this error do fdisk on the disk

format> fdisk
No fdisk table exists. The default partition for the disk is:

a 100% “SOLARIS System” partition

Type “y” to accept the default partition, otherwise type “n” to edit the
partition table.
y
format> p
PARTITION MENU:
0 – change `0′ partition
1 – change `1′ partition
2 – change `2′ partition
3 – change `3′ partition
4 – change `4′ partition
5 – change `5′ partition
6 – change `6′ partition
7 – change `7′ partition
select – select a predefined table
modify – modify a predefined partition table
name – name the current table
print – display the current table
label – write partition map and label to the disk
!<cmd> – execute <cmd>, then return
quit
partition>
partition> print                                     ===> print is used to print the current partition table on the disk available
Current partition table (original):
Total disk cylinders available: 1302 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 – 1301 9.97GB (1302/0/0) 20916630
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 – 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0

partition>


partition> 0                                           ===> creating 0th partition in this disk
Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0

Enter partition id tag[unassigned]: ?                     ===> ? mark is for help
Expecting one of the following: (abbreviations ok):
unassigned boot root swap
usr backup stand var
home alternates reserved

Enter partition id tag[unassigned]: var
Enter partition permission flags[wm]: ?
Expecting one of the following: (abbreviations ok):
wm – read-write, mountable
wu – read-write, unmountable
rm – read-only, mountable
ru – read-only, unmountable

Enter partition permission flags[wm]: wm
Enter new starting cyl[1]: ?
Expecting an integer from 0 to 1301
Enter new starting cyl[1]: 1
Enter partition size[0b, 0c, 1e, 0.00mb, 0.00gb]: 4g
partition> print
Current partition table (unnamed):
Total disk cylinders available: 1302 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 var wm 1 – 523 4.01GB (523/0/0) 8401995               ===> we have var partition created in 0th slice now.
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 – 1301 9.97GB (1302/0/0) 20916630
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 – 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0

partition>
partition> label                       ===> label is used to write the partition table with the new entry.
Ready to label disk, continue? yes

partition> q                            ===>q is used to quit the menu.
=======================================================================================


5. Now we can format the disk with UFS file system

bash-3.00# newfs /dev/rdsk/c1t0d0s0                      ===>newfs is the command used to create the filesystem. Default filesystem will be UFS.
newfs: construct a new file system /dev/rdsk/c1t0d0s0: (y/n)? y           ===> we can create filesystem only in rawdisks (rdsk).
Warning: 2998 sector(s) in last cylinder unallocated
/dev/rdsk/c1t0d0s0: 8401994 sectors in 1368 cylinders of 48 tracks, 128 sectors
4102.5MB in 86 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
7472672, 7571104, 7669536, 7767968, 7866400, 7964832, 8063264, 8161696,
8260128, 8358560
bash-3.00#
==========================================================================================
6. Now we can mount the filesystem as /var

bash-3.00# mount /dev/dsk/c1t0d0s0 /var               ===> for mounting you should use “dsk” naming format (/dev/dsk/c1t0d0s0 )
bash-3.00# df -h /var
Filesystem size used avail capacity Mounted on
/dev/dsk/c1t0d0s0 3.9G 4.0M 3.9G 1% /var
bash-3.00#

Now you can make the entry permanent in /etc/vfstab

=============================================================================================

Hope this will help you. For more posts on Solaris you may click here

Thank you!!!