RedHat Certified System Administrator(EX200) is the 1st to acquire for most of Redhat advanced certificates(non-cloud, regular system admin path), Here are some hints and memo for learning and preparing for RHCSA.

ln has hard link and soft link.

ln newfile.txt /tmp/newfile-hlink2.txt

will create a hard link of newfile.txt. hard link means full copy of the original file, the ln created hard link file will exist even if the original file got deleted. Furthermore, when multiple hard link files of the same original exist, any modification done on any of these hard link file will be synced on all hard link files.

ln -s /root/newfile-link2.txt /tmp/newfile-symlink.txt

will create a soft link, it will point to nonexist if the original got deleted.

User Management

User Membership

usermod -l newname oldname will change username. By default, when a new user got created by useradd, it will automatically belongs to itself name group as primary group. To change its primary group, use:

usermod -g newgroup username

To join other sub groups, for example making user join root and enable sudo su:

usermod -aG root username
User Login and Expiration Management

To disable user student login without deleting the user account:

usermod -s /sbin/nologin student

This way, a user can no longer login the system, but still get his account valid to login webservers(when login webserver with local account enabled).

/etc/login.defs defines default login parameters, including password expiration date.

To force a password change on the first login:

sudo chage -d 0 username

File Managment

File Permissions

Each file has 3 digits representing user, group, other. And each of them is a sum of r=4, w=2, and x=1.

The following command will recursively set read and write access on demodir and all its children for their group owner, but will only apply group execute permissions to directories and files which already have execute set for user, group, and/or other.

chmod -R g+rwX demodir

Advanced options for file running ownership replacing x identifier, setuid = u+s; setgid = g+s; sticky = o+t. And can be convert to digits: setuid = 4; setgid = 2; sticky = 1. To make files created in the directory inherits the group affiliation from the directory, rather than inheriting it from the creating user:

 chmod g+s directory

this is equal to:

chmod 2770 directory

as 2 represents setgid which is g+s.

Default File Permissions

umask can change default file permission when user creates files or folders. By default umask=0002 will make files other set without wirte permission, this results in: files

-rw-rw-r--. 1 student student 0 May  9 01:54 newfile1


drwxrwxr-x. 2 student student 0 May  9 01:54 newdir1

if change current folder umask value to 027:

[[email protected] ~]$ umask 027
[[email protected] ~]$ touch newfile4
[[email protected] ~]$ ls -l newfile4
-rw-r-----. 1 student student 0 May  9 01:55 newfile4
[[email protected] ~]$ mkdir newdir4
[[email protected] ~]$ ls -ld newdir4
drwxr-x---. 2 student student 0 May  9 01:54 newdir4

Redhat7 stores this default and persistent umask value under /etc/profile and /etc/bashrc for global system or inside ~/.bashrc for individual users.

ACL File Permission

Another advanced way to control file accessbilities is to user ACl. This way besides basic chmod file permissions, it provides an extra layer of controlling who can access. If you want multiple users(belongs to different group) or groups to be able to access a file, ACL is a good option. To make usergroup sodor be able to access this file even if chmod 700 has been set for default file owner group.

setfacl -m u::rwx,g:sodor:rX,o::- file

Here group sodor must already have x permission on file’s current folder(i.e /redhat/test), so that sodor can bypass /redhat/test and cat file content. Only having rw permission on /redhat/test is not enought and will end with erro Permission Denied.
Same rule applies to individual users as well.

To make all files in a directory created later on have same ACL:

setfacl -m d:u:name:rx directory

To make all existing files in a directory have same ACL:

setfacl -R -m u:name:rX directory

To explicitly deny a user to access a file, but keep user alive and accessibility to other files:

setfacl -Rm u:james:- /shares/steamies


Security Enhanced Linux provides extra layer of file access management, instead of limiting access from user level(chown/setfacl), it bans access from Application side. A good example is that when we install named/bind9 on RHEL7, it comes with SELinux enalbed default values, make /var/named/ as its customized DNS content folder and all files in it will have context named_zone_t, but if we manually put all DNS files in any other folders, named won’t be able to read those files unless changing their context to be named_zone_t.

SELinux labels have several contexts: user, role, type, and sensitivity. We usually focus on 3rd context: type.

Default SELinux mode stores at /etc/selinux/config.

Usually most of the commands have an option -Z to disply SELinux information. E.g: ps -auxZ or ll -Z.

chcon can change SELinux context of any file, while restorecon will restore any file’s context to its default value which inherit from the folder it belongs to.

[[email protected] ~]# mkdir /virtual
[[email protected] ~]# ls -Zd /virtual
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /virtual
[[email protected] ~]# chcon -t httpd_sys_content_t /virtual
[[email protected] ~]# ls -Zd /virtual
drwxr-xr-x. root root unconfined_u:object_r:httpd_sys_content_t:s0 /virtual
[[email protected] ~]# restorecon -v /virtual
restorecon reset /virtual context unconfined_u:object_r:httpd_sys_content_t:s0->
[[email protected] ~]# ls -Zd /virtual
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /virtual

semanage is a python app that can control SELinux context via Regular Expression. Install via yum install policycoreutils-python.

To change all files context type in /virtual/

semanage fcontext -a -t httpd_sys_content_t '/virtual(/.*)?'

getsebool -a will only gives you current status of a context, while semanage boolean -lC can tell what has been changed in the system and if it’s enabled by default.

[[email protected] ~]# getsebool -a
abrt_anon_write --> off
abrt_handle_event --> off
abrt_upload_watch_anon_write --> on
antivirus_can_scan_system --> off
antivirus_use_jit --> off
[[email protected] ~]# getsebool httpd_enable_homedirs
httpd_enable_homedirs --> off
[[email protected] ~]# setsebool httpd_enable_homedirs on
[[email protected] ~]# semanage boolean -l | grep httpd_enable_homedirs
httpd_enable_homedirs          (on   ,  off)  Allow httpd to enable homedirs
[[email protected] ~]# getsebool httpd_enable_homedirs
httpd_enable_homedirs --> on
[[email protected] ~]# setsebool -P httpd_enable_homedirs on
[[email protected] ~]# semanage boolean -l | grep httpd_enable_homedirs
httpd_enable_homedirs          (on   ,   on)  Allow httpd to enable homedirs

To find out what context a file needs to have for named to read, we can use:

semanage fcontext -l | grep named
/etc/rndc.*                                        regular file       system_u:object_r:named_conf_t:s0
/var/named(/.*)?                                   all files          system_u:object_r:named_zone_t:s0
/etc/unbound(/.*)?                                 all files          system_u:object_r:named_conf_t:s0
/var/run/bind(/.*)?                                all files          system_u:object_r:named_var_run_t:s0
/var/log/named.*                                   regular file       system_u:object_r:named_log_t:s0
/var/run/named(/.*)?                               all files          system_u:object_r:named_var_run_t:s0

And you’ll see that ideally files under /var/named should have named_zone_t, so if any file functions as files in /var/named but don’t have correct context will fail.

Process Management

kill can actually do more then just kill.

[[email protected] ~]$ kill -l
 1) SIGHUP      2) SIGINT      3) SIGQUIT     4) SIGILL      5) SIGTRAP
 6) SIGABRT     7) SIGBUS      8) SIGFPE      9) SIGKILL    10) SIGUSR1
11) SIGSEGV    12) SIGUSR2    13) SIGPIPE    14) SIGALRM    15) SIGTERM

So kill -9 is the brutal kill, while kill is default to kill -15 as soft kill.

killall and pkill can kill multiple process at the same time, e.g stop all process that hacker runs, pkill -U hacker.


Few tools used related to linux partition:

  • fdisk: Main partition program to partition MBR size < 2TB.
  • gdisk: Partition GPT dev.
  • parted: Better tool to partition for dev size > 2TB.
  • dd: File convertor, can create image from a dev.
  • mkfs: Format tool, ext4 is used as default in most linux system,xfs is better for storage usage as it supports >16TB.

To create a swap, we can use a dedicated partition to store swap or making a file storing swap caches(/dev/zero can be loopback).

  1. fdisk /dev/vdb to create a new partition for swap, and then mkswap /dev/vdb1 and swapon /dev/vdb1.
  2. dd if=/dev/zero of=/path/to/swapfile bs=1M count=1K to create a 1G swap file, then mkswap /path/to/swapfile and swapon /path/to/swapfile.

To permanently activate a swap, edit /etc/fstab:

#for dd
/swapfile swap swap defaults 0 0
#for fdisk partition
UUID=fbd7fa60-b781-44a8-961b-37ac3ef572bf  swap  swap  defaults  0 0 

Here the last 2 digits 0 0 indicate:

  • dump: used by the dump utility to decide when to make a backup. When installed, dump checks the entry and uses the number to decide if a file system should be backed up. Possible entries are 0 and 1. If 0, dump will ignore the file system, if 1, dump will make a backup. Most users will not have dump installed, so they should put 0 for the dump entry.
  • pass: fsck reads the pass number and determines in which order the file systems should be checked. Possible entries are 0, 1, and 2. The root file system should have the highest priority, 1, all other file systems you want to have checked should get a 2. File systems with a pass value 0 will not be checked by the fsck utility.

Mouting Devices

Dynamically created devices/folders won’t be automacially mounted, and may include complex manual task such as editing /etc/fstab for them to be loaded at boot.

autofs is a tool to automate all these and has been widely used against NFS mounted folders. To use it:

  1. Create /etc/auto.master.d/*.autofs, this file identifies the base directory used for mount points and identifies the mapping file used for creating the automounts. direct.autofs for direct(pre-exist) mounting, shares.autofs for dynamic mouting.

  2. Write content like this:

    • ini
    • ini
          #filename: /etc/auto.master.d/shares.autofs
          # to define which files will contain dynamic mounting points definition
          /shares  /etc/auto.demo

  3. write content:

    • ini
    • ini
          #filename: /etc/auto.home
          # if want to mount work dynamically, meaning creating and deleting /shares/work after use
          work  -rw,sync  serverX:/shares/work
          # if NFS has multiple exports, we can use following
          *  -rw,sync  serverX:/shares/&

RHEL Boot Process

How to Recover Lost Root Password

On Red Hat Enterprise Linux 7, it is possible to have the scripts that run from the initramfs pause at certain points, provide a root shell, and then continue when that shell exits. While this is mostly meant for debugging, it can also be used to recover a lost root password:

  1. Reboot the system.
  2. Interrupt the boot loader countdown by pressing any key.
  3. Move the cursor to the entry that needs to be booted.
  4. Press e to edit the selected entry.
  5. Move the cursor to the kernel command line (the line that starts with linux16).
  6. Append rd.break (this will break just before control is handed from the initramfs to the actual system).
  7. Press Ctrl+x to boot with the changes. At this point, a root shell will be presented, with the root file system for the actual system mounted read-only on /sysroot.

To recover the root password from this point, use the following procedure:

#Remount /sysroot as read-write. /sysroot contains real system disk, /etc, /root, /home, etc.
switch_root:/# mount -o remount,rw /sysroot
#Switch into a chroot jail, where /sysroot is treated as the root of the file system tree. So that you can change root password of real system from pre-boot root user.
switch_root:/# chroot /sysroot        
#Set a new root password:
sh-4.2# passwd root        
#Make sure that all unlabeled files (including /etc/shadow at this point) get relabeled during boot
sh-4.2# touch /.autorelabel
#Type exit twice. The first will exit the chroot jail, and the second will exit the initramfs debug shell.
Emergency and Rescue Targets

By appending either or to the kernel command line from the boot loader, the system will spawn into a special rescue or emergency shell instead of starting normally. Both of these shells require the root password. The emergency target keeps the root file system mounted read-only, while waits for to complete first so that more of the system will be initialized (e.g., logging, file systems, etc.).

To use this method of selecting a different target, use the following procedure for Red Hat Enterprise Linux 7 systems:

  1. (Re)boot the system.
  2. Interrupt the boot loader menu countdown by pressing any key.
  3. Move the cursor to the entry to be started.
  4. Press e to edit the current entry.
  5. Move the cursor to the line that starts with linux16. This is the kernel command line.
  6. Append or
  7. Press Ctrl+x to boot with these changes.

These shells can be used to fix any issues that prevent the system from booting normally; for example, a dependency loop between services, or an incorrect entry in /etc/fstab. Exiting from these shells will continue with the regular boot process.


The way to create a lv is similar as how kubernetes claim storage in the backend.

Steps to create a valid LV:

  1. fdisk /dev/sda to partition a disk, don’t need to format it now.
  2. pvcreate /dev/sda1 to create a Physical Volume, it will be functioning as a label for lvgroup to select.
  3. vgcreate vgname /dev/sda1 /dev/sdb1 to create a Volume Group.
  4. lvcreate -n lvname -L 2G vgname to claim 2G size from above vg.
  5. mkfs -t xfs /dev/vgname/lvname to format newly created lv.

To extend a lv and acquire the extended space:

lvextend -rL +300M /dev/vg-alpha/hercules

Besides using partition as LVM PV, LVM can also use local file. Simply using dd create a file mirror then mount and part:

  1. dd if=/dev/zero of=/path/to/file.img bs=512M count=1 to create a file.
  2. fdisk /path/to/file.img to create new partition and change type to 8e(LVM).
  3. losetup /path/to/file.img /dev/loopx mount file onto empty loop slot.
  4. pvcreate /dev/loopx, vgcreate test /dev/loopx, lvcreate -n test -L 512M test to create lvm using file mounted.