Cheat sheet on Linux administration

In the docs above you will see the following command to view a list of mappings between SELinux and Linux user accounts. As per the docs, you need to have the policycoreutils-python package installed first.

• semanage login -l

ls -Z file1 will show you:

-rw-rw-r--  user1 group1 unconfined_u:object_r:user_home_t:s0 file1

In this example, SELinux provides:

• a user (unconfined_u),
• a role (object_r),
• a type (user_homet), and
• a level (s0).

This information is used to make access control decisions. On DAC systems, access is controlled based on Linux user and group IDs. SELinux policy rules are checked after DAC rules. SELinux policy rules are not used if DAC rules deny access first.

Temporary Changes: chcon The chcon command changes the SELinux context for files.

However, changes made with the chcon command do not survive a file system relabel, or the execution of the restorecon command. SELinux policy controls whether users are able to modify the SELinux context for any given file. When using chcon, users provide all or part of the SELinux context to change. An incorrect file type is a common cause of SELinux denying access.

chcon -t type file-name changes the file type, where type is an SELinux type, such as httpd_sys_content_t, file-name is a file or directory name:

- chcon -t httpd_sys_content_t file-name - chcon -t httpd_sys_content_t file-name

Run the chcon -R -t type directory-name command to change the type of the directory and its contents, where type is an SELinux type, such as httpd_sys_content_t, and directory-name is a directory name:

chcon -R -t httpd_sys_content_t directory-name

If you have files in a directory that you know are setup correctly, you can use one of their contexts as a reference when changing the context of a new file: The syntax is:

• chcon --reference known-file.html newfile-to-match-it.html

But this is temporary, so to do it correctly, run the command:

• sudo semanage fcontext -a -t httpd_sys_content_t test.html

When I ran that command though, the fcontext of the file in the directory actually did NOT change. So what did change??? Maybe nothing, because,… the file context for all .html files was already set. I checked in /etc/selinux/targeted/contexts/files and test.html was not there, but the wildcard *.html was.

• /usr/share/nginx/html(/.*)? system_u:object_r:httpd_sys_content_t:s0

I find that since the fcontext for the /usr/share/nginx/html directory has already been set, when I touch a file in that directory, it is created with the correct fcontext. If I move a file from another user directory, it won't have the correct fcontext, so I then have to change it.

From access.redhat.com

The semanage fcontext command is used to change the SELinux context of
files. When using targeted policy, changes are written to files located in the
/etc/selinux/targeted/contexts/files/ directory: The file_contexts file
specifies default contexts for many files, as well as contexts updated via
semanage fcontext.  The file_contexts.local file stores contexts to newly
created files and directories not found in file_contexts.  Two utilities read
these files. The setfiles utility is used when a file system is relabeled and
the restorecon utility restores the default SELinux contexts. This means that
changes made by semanage fcontext are persistent, even if the file system is
relabeled. SELinux policy controls whether users are able to modify the SELinux
context for any given file.

These examples were used when installing roundcube:

semanage fcontext -a -t httpd_log_t '/var/www/html/webmail/temp(/.*)?'
semanage fcontext -a -t httpd_log_t '/var/www/html/webmail/logs(/.*)?'
restorecon -v -R /var/www/html/webmail

This lets the roundcube service have write access to these temp and log directories that otherwise SELinux would prevent.

When testing roundcube while installing, you can turn off SELinux temporarily, to confirm that SELinux isn't messing you up. Once everything works, turn it back on, and add the fcontext exceptions.

For example, what are my fcontexts for /usr/share/nginx/html/*.html ? They are listed in this directory, so you can check:

• /etc/selinux/targeted/contexts/files/file_contexts

In that file you can see this line:

• /usr/share/nginx/html(/.*)? system_u:object_r:httpd_sys_content_t:s0

And comparing that to the output of ls -Z in directory /usr/share/nginx/html

• unconfined_u:object_r:httpd_sys_content_t:s0 top.html (for all html files)

A good tool to see what you have going with semanage is to list things first. Try semanage fcontext -l | grep httpd_sys_rw_content_t for starters. I had first I semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html(/.*)?" when I was intalling apache.org. Then I wanted to list all related settings so I did the semanage fcontext -l command and grepped for what I was looking for.

semanage can also lists ports that are protected, and add/delete ports as needed. semanage port -l

I tried to create a virtual apache host, that listened to port 7927, for a flask app, that was being moved into a production apache mod_wsgi app.

Everything was correct in my virtual host config, but ss -tulpn would NOT ever say it was listening on port 7927.

I then issued three semanage commands:

• semanage port -l | grep http
• semanage port -a -t http_port_t -p tcp 7927
• semanage port -l | grep http

After that every was copasetic! semanage fcontext -a -t httpd_{sysrwcontentt} "var/www/wordpress(.*)?

If you just type mail and hammer, then you are using the mail client to view your own mbox i.e. in /var/mail/userid messages

If you just type mail user@domain.com and hammer, then you are in interactive mode, and will be prompted for subject, followed by a chance to enter a body of the email message. C-d for EOF

I have setup my own zmailer.py module that uses some common python library modules. You can create your own using this as an example. Below is a subset of that module, showing just the pertinent mailing code. It uses an environment module where I keep my credentials, called env_user_zp.py

  #!/usr/local/bin/python3.8
  ''' Module for sending google emails, utilizing gmail app specific passwords
      Author: Zintis Perkons

      /usr/bin/env python resolved to /usr/local/bin/python3.8.  This may change
      in the future, so we may want to change this to #!/usr/local/bin/python3.8

    The password  is a google "app" password and must be set first via the
    google https://myaccount.google.com/apppasswords page.  The result is an
    alphabetic string that is 16 characters long, for example hrlsnjuqscpekzybk

    Can be called by any python program that wants to use my google mail acct
    to send an email

    Syntax to call this module is:
    from zmailer import sendnow
    sendnow(summary, type_of_report, zbody, whensent, emailaddresses)

    If run directly, will send a test email to technical@....  Syntax to run
    directly is python -m gmail.py

  '''
  import smtplib
  from email.message import EmailMessage
  import env_user_zp as env

  EMAILADDR = env.GMAILUSER
  EMAILPASS = env.PYTHONGOOGLEPASS

  def sendnow(summary: str,
              type_of_report: str,
              whensent: str,
              zbody="No body was included in this message",
              to_whom=["technical@thunderconsulting.simplelogin.com"] ):
      ''' sends an email with the summary data.

      to_whom should be passed to this method as a list of strings, even
      if there is only one element.  For example ['palin@python.uk.co']
      The default recipient (if none provided) is ["technical@..."]

      '''

      msg = EmailMessage()
      msg['Subject'] = 'Zinux ' + type_of_report + ' Sent on ' + whensent
      msg['From'] = EMAILADDR
      msg['To'] = to_whom

      whattosay = zbody + summary
      msg.set_content(whattosay)


      with smtplib.SMTP_SSL('smtp.gmail.com', 465) as smtp:
          smtp.login(EMAILADDR, EMAILPASS)
          smtp.send_message(msg)


  if __name__ == '__main__':
      # Run directly this script will send to technical@thunderconsulting.simplelogin.com
      # (default recipient so not actually included in this __main__ call

      mess_body = '''

      To whom it may concern,

      This message body can be replaced with any string you like by passing
      the key value for the parameter 'zbody'

      Signed,

      Zintis Perkons

      '''

      sendnow(summary="zmailer.py was run directly",
              type_of_report="Daily", 
              zbody=mess_body,
              whensent="Today")  # typically would use an actual date string
#             to_whom=["technical@thunderconsulting.simplelogin.com"] )

To list all of them:

• mount -l

To list just type nfs4:

• mount -l -t nfs4

If the drives are nfs drives you may try:

• nfsstat

You can manually less the file /proc/mounts or grep that for nfs to just see nfs mounts.

If you distro has installed smarttools, you can try to use smartctl.

The old technique of using /etc/fastab still works in CentOS 8 There are more modern tools, but I would start with man 5 fstab, man 8 mount, man 8 findfs

From youtube example, use mount -o loop CentOS-7-livecd-x866.iso /mnt The above link also had good example use of squashfs, chroot, mount and systemctl

To shrink the size of an LVM partiion (logical volume manager) to 10Gig

1. umount /dev/hda2 /mountpt
2. e2fsck -f /dev/hda2
3. resiz32fs /dev/hda2 10G
4. lvreduce -L 10G /dev/hda2
5. mount /dev/hda2 /mountpt

Note that if you used fdisk to partition a disk, you are out of luck. fdisk creates fixed volume partitions and all you can do is erase them and recreate them larger. (could recover from a backup)

• ext4 is a filesystem that you may install onto a partition.

- df -hT to see the size of the storage on each partition.

• e2fsck is needed to check that the filesystem on the partition is not corrupted.

You can mount a fusion share, i.e. a folder that you have configured on the fusion vm settings, under Sharing using the vmhgfs-fuse command on the C8host. /usr/bin/vmhgfs-fuse .host:/ /var/osx-share -o subtype=vmhgfs-fuse,allow_other

Create an alias "share" for this i.e.

alias share='sudo /usr/bin/vmhgfs-fuse .host:/ /var/osx-share \
             -o subtype=vmhgfs-fuse,allow_other'

This will mount the directory .host:/ which is what the VMware fusion GUI has allocated for this VM as a shared folder. (My Fusion GUI was assigned the directory "fusion-share-folder" on my external 250GB Sandisk SSD drive) i.e. /Volumes/ZP-250GB/Virtual Machines/fusion-share-folder

Anyway that is the folder that is designated .host:/ Then my CentOS guest VM sees that folder as /var/osx-share.

So recapping;

sudo /usr/bin/vmhgfs-fuse .host:/ /var/osx-share -o subtype=vmhgfs-fuse,allow_other
cd /var/osx-share/fusion-share-folder

On my macbook pro the fusion share folder is on my exteranl 250GB drive: ZP-250GB/Virtual\ Machines/fusion-share-folder

squashfs and unsquashfs : similar to tar -gvxf So for example unsquashfs squashfs.img

You can run this in a cron job to alert you via email when disk usage is over 90% used. This was taken from cyberciti.biz:

#!/bin/bash
# Shell Script to monitor NAS backup disk space
# Shell script will mount NAS using mount command and look for total used
# disk space. If NAS is running out of disk space an email alert will be sent to
# admin.
# -------------------------------------------------------------------------
# Copyright (c) 2004 nixCraft project <http://cyberciti.biz/fb/>
# This script is licensed under GNU GPL version 2.0 or above
# -------------------------------------------------------------------------
# This script is part of nixCraft shell script collection (NSSC)
# Visit http://bash.cyberciti.biz/ for more information.
# -------------------------------------------------------------------------
#!/bin/bash
#*** SET ME FIRST ***#
NASUSER="Your-User-Name"
NASPASS="Your-Password"
NASIP="nas.yourcorp.com"
NASROOT="/username"
NASMNTPOINT="/mnt/nas"
EMAILID="admin@yourcorp.com"

GETNASIP=$(host ${NASIP} | awk '{ print $4}')

# Default warning limit is set to 17GiB
LIMIT="17"

# Failsafe 
[ ! -d ${NASMNTPOINT} ] && mkdir -p ${NASMNTPOINT}
mount | grep //${GETNASIP}/${NASUSER}

# if not mounted, just mount nas
[ $? -eq 0 ] && : || mount -t cifs //${NASIP}/${NASUSER} -o username=${NASUSER},password=${NASPASS} ${NASMNTPOINT}
cd ${NASMNTPOINT}

# get NAS disk space
nSPACE=$(du -hs|cut -d'G' -f1)
# Bug fix
# get around floating point by rounding off e.g 5.7G stored in $nSPACE 
# as shell cannot do floating point 
SPACE=$(echo $nSPACE | cut -d. -f1)

cd /
umount ${NASMNTPOINT}

# compare and send an email 
if [ $SPACE -ge $LIMIT ]
then
        logger "Warning: NAS Running Out Of Disk Space [${SPACE} G]"
        mail -s 'NAS Server Disk Space' ${EMAILID} <<EOF
NAS server [ mounted at $(hostname) ] is running out of disk space!!!
Current allocation ${SPACE}G @ $(date) 
EOF
else
    logger "$(basename $0) ~ NAS server ${NASIP} has sufficent disk space for backup!"
fi

A simpler script to check local drives also taken from cyberciti.biz

#!/bin/sh
# Shell script to monitor or watch the disk space
# It will send an email to $ADMIN, if the (free avilable) percentage 
# of space is >= 90% 
# -------------------------------------------------------------------------
# Copyright (c) 2005 nixCraft project <http://cyberciti.biz/fb/>
# This script is licensed under GNU GPL version 2.0 or above
# -------------------------------------------------------------------------
# This script is part of nixCraft shell script collection (NSSC)
# Visit http://bash.cyberciti.biz/ for more information.
# ----------------------------------------------------------------------
# Linux shell script to watch disk space (should work on other UNIX oses )
# SEE URL: http://www.cyberciti.biz/tips/shell-script-to-watch-the-disk-space.html
# set admin email so that you can get email
ADMIN="me@somewher.com"
# set alert level 90% is default
ALERT=90
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  #echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge $ALERT ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" | 
     mail -s "Alert: Almost out of disk space $usep" $ADMIN
  fi
done

Simplest of all, still from cyberciti.biz

#!/bin/bash
# Tested Under FreeBSD and OS X
FS="/usr"
THRESHOLD=90
OUTPUT=($(LC_ALL=C df -P ${FS}))
CURRENT=$(echo ${OUTPUT[11]} | sed 's/%//')
[ $CURRENT -gt $THRESHOLD ] && echo "$FS file system usage $CURRENT" | mail -s "$FS file system" you@example.com

But my favourite is taken from, you guessed it, cyberciti.giz

#!/bin/sh
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 90 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
     mail -s "Alert: Almost out of disk space $usep%" you@somewhere.com
  fi
done

This command is needed to take the user edited file, /etc/default/grub and create the system file, /boot/grub2/grub.cfg . By default grub2-mkconfig will send to stdout which is good to check, but after you can override that with the -o option or just redirect with > to /boot/grub2/grub.cfg

Of course, if you stepped away during a boot, and missed the messages, you can always see them all in /var/log/messages

I was getting this in /var/log/messages :

May 17 08:39:13 zintis dracut[1468]: Stored kernel commandline:
May 17 08:39:13 zintis dracut[1468]: rd.driver.pre=iTCO_wdt,lpc_ich
May 17 08:39:13 zintis dracut[1468]: *** Install squash loader ***
May 17 08:39:13 zintis dracut[1468]: *** Stripping files ***
May 17 08:39:13 zintis dracut[1468]: *** Stripping files done ***
May 17 08:39:13 zintis dracut[1468]: *** Squashing the files inside the initramfs ***
May 17 08:39:19 zintis systemd[1]: systemd-hostnamed.service: Succeeded.
May 17 08:39:27 zintis dracut[1468]: *** Squashing the files inside the initramfs done ***
May 17 08:39:27 zintis dracut[1468]: *** Creating image file '/boot/initramfs-4.18.0-372.9.1.el8.x86_64kdump.img' ***
May 17 08:39:27 zintis dracut[1468]: *** Creating initramfs image file '/boot/initramfs-4.18.0-372.9.1.el8.x86_64kdump.img' done ***
May 17 08:39:28 zintis kdumpctl[964]: kdump: kexec: loaded kdump kernel
May 17 08:39:28 zintis kdumpctl[964]: kdump: Starting kdump: [OK]
May 17 08:39:28 zintis systemd[1]: Started Crash recovery kernel arming.
May 17 08:39:28 zintis systemd[1]: man-db-cache-update.service: Succeeded.
May 17 08:39:28 zintis systemd[1]: Started man-db-cache-update.service.
May 17 08:39:28 zintis systemd[1]: Startup finished in 1.014s (kernel) + 3.239s (initrd) + 42.371s (userspace) = 46.625s.
May 17 08:39:28 zintis systemd[1]: run-r459feec8c3c94458af06522936755a9c.service: Succeeded.
May 17 08:40:20 zintis systemd[1]: Starting system activity accounting tool...
May 17 08:40:20 zintis systemd[1]: sysstat-collect.service: Succeeded.
May 17 08:40:20 zintis systemd[1]: Started system activity accounting tool.
May 17 08:41:02 zintis systemd[1009]: Starting Mark boot as successful...
May 17 08:41:02 zintis grub2-set-bootflag[5580]: Error reading from /boot/grub2/grubenv: Invalid argument
May 17 08:41:02 zintis systemd[1009]: grub-boot-success.service: Main process exited, code=exited, status=1/FAILURE
May 17 08:41:02 zintis systemd[1009]: grub-boot-success.service: Failed with result 'exit-code'.
May 17 08:41:02 zintis systemd[1009]: Failed to start Mark boot as successful.
(END)

Gives me:

• Linux hostname.com 4.18.0-348.23.1.el8_5.x86_64 ...

After upgrade it is:

• Linux hostname.com 4.18.0-372.9.1.el8.x86_64 ...

Systemd consists of many things:

• init
• systemctl
• journald (logs)
• journalctl
• networkd
• logind (getty)
• process management

Rather than run levels, systemd uses named targets that can be many.

Systemd uses "targets" instead of runlevels. By default there are two main() targets: multi-user.target and graphical.target

Targets are groups of units Think of them as the old "runlevels". But Multiple targets can be active at once. You also get more meaningful names: See man 7 systemd.special for a list of possible targets, but that does NOT include any custom targets you might have created yourself.

You might have noticed that in among the unit files in /usr/lib/systemd/system files are files with the .target ending. Here for example is my reboot.target file:

[Unit]
Description=Reboot
Documentation=man:systemd.special(7)
DefaultDependencies=no
Requires=systemd-reboot.service
After=systemd-reboot.service
AllowIsolate=yes
JobTimeoutSec=30min
JobTimeoutAction=reboot-force

[Install]
Alias=ctrl-alt-del.target
reboot.target (END)

Another good example called network.target

[Unit]
Description=Network
Documentation=man:systemd.special(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget
After=network-pre.target
RefuseManualStart=yes

Centos 8 uses systemclt set-default <TARGET>.target command where <TARGET> is typically either multi-user for a cli interface, or graphical for a GUI windows mgr. But you can systemctl set-default <TARGET>.target any of the valid targets

So, to tell systemd to boot into a cli (text based)

• systemctl set-default multi-user.target

If you want to boot into a gui again, change it to:

• systemclt set-default graphical.target

Almalinux also uses systemctl set-default multi-user.target

• systemctl default

• systemctl isolate <TARGET>.target

Use the Services Manager or run the following command:

• ln -sf /usr/lib/systemd/system/TARGET-NAME.target /etc/systemd/system/default.target

i.e. you set a symbolic link in /etc/systemd/system/default.target to point to the target file you want as your default.

So, my AlmaLinux VM has these most common targets in /lib/systemd/system/*.target

< get output from terminal here >

check with:

See the (related) dnf groups with:

• sudo dnf grouplist

You can also, correctly?, append rd.systemd.unit=multi-user.target to the /proc/cmdline file: It will looks something like this:

• BOOT_IMAGE=/vmlinuz-3.10.0-327.36.3.el7.x86_64 root=UUID=2cc29b16-fe2b-400f-a39f-3e9048784599 ro vconsole.keymap=us crashkernel=auto vconsole.font=latarcyrheb-sun16 rd.driver.blacklist=radeon LANG=en_US.UTF-8 3

Once in Terminal, you can start the GUI again if you need to by using: systemctl isolate graphical and you will be back in the GUI. Then, back to GUI with systemctl isolate multiuser.target and back again with: systemctl isolate graphical.target Notice that switching to a GUI using startx will mess you up because, running startx still leaves you in the multiuser.target position. So systemctl isolate multiuser.target at this point will do nothing. You have to kill startx. Best to just use systemctl every time and you will avoid this problem.

Just run systemctl get-default to find out. This is what the system will boot to.

• systemctl set-default graphical.target much like the SystemV runlevel5
• systemctl set-default multiuser.target much like the SystemV runlevel3

If you want to do this at run-time, (i.e. you've already booted) you use isolate

• systemctl isolate [target]

systemctl lets you see what is running, and control it. i.e. start/stop By itself (with no arguments) it will list all units under systemctl control. You can pipe to grep if wanted, and you get a nicer output if you do.

sudo systemctl edit NetworkManager.service see also: /usr/lib/sysctl.d/60-libvirtd.conf # but do not edit this file

- systemctl start unit1 unit2, unit3: Start these "globbed" units immediately: - systemctl stop unit1 unit4: Stop these units immediateley. - systemctl restart unit: Restart a unit: - systemctl reload unit: Ask a unit to reload its configuration: - systemctl status unit: Show the status of a unit, including whether running or not: - systemctl is-enabled unit: Check whether a unit is already enabled or not: - systemctl enable unit: Unit will start up on bootup: - systemctl enable unit: Unit will start up on bootup: - systemctl enable --now unit: Unit will start on bootup AND start immediately. - systemctl enable --now unit: Unit will start on bootup AND start immediately. - systemctl disable unit: Disable a unit to not start during bootup: - systemctl mask unit: Mask a unit to make it impossible to start it (both manually and as a dependency, which makes masking dangerous):

• systemctl list-units
• systemctl list-units --type=target
• systemctl list-units --type=service
• systemctl list-unit-files
• systemctl -t service will list all the service units.
• systemctl -t target will list all the target units.
• systemctl -t device will list all the device units.

sudo systemctl list-units --type=help
Available unit types:
service
socket
target
device
mount
automount
swap
timer
path
slice
scope

Unmask a unit: systemctl unmask unit

Show the manual page associated with a unit (this has to be supported by the unit file): systemctl help unit

can be up to 255 characters, this limit does NOT include the path name, which can be altogether 4096 characters. woww woww wee wah… Is very niice.

Very useful to find who/what user/process has open files. The name here is very descriptive. "List Open Files" or lsof. By default, with no other options, will show ALL the open files by the kernel. This will be a very long list.

By default lsof shows list of ALL open files. Any options included on the command are OR'd together so for instance lsof -u zintis -i will show open files by user zintis as well as all open internet files.

If you want to AND several options together you simply

Already mentioned above regarding file descriptors, the /proc filesystem is a special directory where each unix process can keep track of itself. Try a very simple ls /proc and you will see sub-directories labelled with the PID of each open running process.

Useful subdirectories are:

• /proc/PID/fd : file descriptors
• /proc/PID/fd/0 : stdin
• /proc/PID/fd/1 : stdout
• /proc/PID/fd/2 : stderror
• /proc/PID/mem : memory
• /proc/PID/map_files : open files
• /proc/PID/net : network state ?
• /proc/cpuinfo : see what the cpu is doing right now

cat While we are talking about /proc/cpu=, there is another useful cpu command: lscpu to list out details about your cpu.

An example use case:

ls -l /proc/17590/map_files |  awk '/\.so/{print $11}'  | sort -u
# as root

Some applications are linked to libselinux.so directly, which means that the setenforce settings of selinux may not have an affect on the app. You can confirm that by listing open files used by an app, with lsof. For example:

• -a and all the options. i.e. all of the options must be met to display something\
• -p for process with PID (can be a csv list of PIDs)
• -P show numeric port numbers for network files
• +|-r repeat displaying lsof output, +r stop when no more open files, -r stop only when interrupted C-c, or ^c

ps -aux | grep nginx
# get list of PIDS first
sudo lsof -P -p '17589,17590'  | grep -i selinux

/dev/sda, /dev/sdb are names of SCSI devices /dev/sda, /dev/sdb also for SAS drives /dev/hda, /dev/hdb are for IDE/EIDE drives /dev/fd0, /dev/fd1 floppy drives (obsoleted)

to manually change the filesystem table * use caution…

to list the block ids of all your block devices (disks and partitions) This will also show the UUID of the disk $ sudo lsblk -f will show output in a nice tablular form, but $ sudo lsblk works too.

/dev/ttyS0 /dev/ttyS1 and /dev/stty?

/dev/lp1

You can clear your own cache using

• rm -rv ~~/.cache
• ~ ~

Typically two built-in processes, lsof and netstat are the easiest ways to look up which process is listening on which port.

Takes on a signal, as a name or a number, a q value, and the PID or name. I have only used the PID myself. You can also -list the signal names for your system using kill -l

kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
21) SIGTTIN	22) SIGTTOU	23) SIGURG	24) SIGXCPU	25) SIGXFSZ
26) SIGVTALRM	27) SIGPROF	28) SIGWINCH	29) SIGIO	30) SIGPWR
31) SIGSYS	34) SIGRTMIN	35) SIGRTMIN+1	36) SIGRTMIN+2	37) SIGRTMIN+3
38) SIGRTMIN+4	39) SIGRTMIN+5	40) SIGRTMIN+6	41) SIGRTMIN+7	42) SIGRTMIN+8
43) SIGRTMIN+9	44) SIGRTMIN+10	45) SIGRTMIN+11	46) SIGRTMIN+12	47) SIGRTMIN+13
48) SIGRTMIN+14	49) SIGRTMIN+15	50) SIGRTMAX-14	51) SIGRTMAX-13	52) SIGRTMAX-12
53) SIGRTMAX-11	54) SIGRTMAX-10	55) SIGRTMAX-9	56) SIGRTMAX-8	57) SIGRTMAX-7
58) SIGRTMAX-6	59) SIGRTMAX-5	60) SIGRTMAX-4	61) SIGRTMAX-3	62) SIGRTMAX-2
63) SIGRTMAX-1	64) SIGRTMAX	

kill command: kill [-signal|-s signal|-p] [-q value] [-a] [--] pid|name...

See also man 7 signal for info on the signals themselves.

An excerpt follows:

Signal     Value     Action   Comment
──────────────────────────────────────────────────────────────────────
SIGHUP        1       Term    Hangup detected on controlling terminal
                              or death of controlling process
SIGINT        2       Term    Interrupt from keyboard
SIGQUIT       3       Core    Quit from keyboard
SIGILL        4       Core    Illegal Instruction
SIGABRT       6       Core    Abort signal from abort(3)
SIGFPE        8       Core    Floating-point exception
SIGKILL       9       Term    Kill signal
SIGSEGV      11       Core    Invalid memory reference
SIGPIPE      13       Term    Broken pipe: write to pipe with no
                              readers; see pipe(7)
SIGALRM      14       Term    Timer signal from alarm(2)
SIGTERM      15       Term    Termination signal
SIGUSR1   30,10,16    Term    User-defined signal 1
SIGUSR2   31,12,17    Term    User-defined signal 2
SIGCHLD   20,17,18    Ign     Child stopped or terminated
SIGCONT   19,18,25    Cont    Continue if stopped
SIGSTOP   17,19,23    Stop    Stop process
SIGTSTP   18,20,24    Stop    Stop typed at terminal
SIGTTIN   21,21,26    Stop    Terminal input for background process
SIGTTOU   22,22,27    Stop    Terminal output for background process

The signals SIGKILL and SIGSTOP cannot be caught, _block_ed, or ignored.

Next the signals not in the POSIX.1-1990 standard but described in SUSv2 and POSIX.1-2001.

Signal       Value     Action   Comment
────────────────────────────────────────────────────────────────────
SIGBUS      10,7,10     Core    Bus error (bad memory access)
SIGPOLL                 Term    Pollable event (Sys V).
                                Synonym for SIGIO
SIGPROF     27,27,29    Term    Profiling timer expired
SIGSYS      12,31,12    Core    Bad system call (SVr4);
                                see also seccomp(2)
SIGTRAP        5        Core    Trace/breakpoint trap
SIGURG      16,23,21    Ign     Urgent condition on socket (4.2BSD)
SIGVTALRM   26,26,28    Term    Virtual alarm clock (4.2BSD)
SIGXCPU     24,24,30    Core    CPU time limit exceeded (4.2BSD);
                                see setrlimit(2)
SIGXFSZ     25,25,31    Core    File size limit exceeded (4.2BSD);
                                see setrlimit(2)

Common signals to stop a process are SIGHUP, SIGQUIT, SIGKILL, SIGTERM but what they actually do, depends on how the process is written to handle each of these signals. i.e. what the signal handler is written to do for each.

But by convention only, they each do the following: (note, NOT in numeric order):

Sometimes, the process itself is fine, but there is a problem with open files related to the process, or that the process opened and did not close.

1. Find the process ID If not stored in /etc/nginx.pid file, or something similar do this:
   • ps x | grep nginx
2. List the files opened by that process
   • lsof -p <pid from step 1>
3. Use these two steps when troubleshooting where a process might be writing error logs as well.

This unit must be running for sar to be able to report anything. To start this service, run systemclt start sysstat Installing sysstat does NOT mean that it will also start the service. Check with systemctl status sysstat

Once running, it will start writting performance data to the file:

• /var/lop/sa/saDD where DD is the current day. Any existing files will be archived. For example on <2022-03-10 Thu> the file is sa10
• sa logs are binary files. You need to use sar to view data in them. For example: sar -r -f /var/log/sa/sa26
• My Centos also stores the corresponding sar file which is just an ASCII text file of the same output as sar -r -f ... but with all the typical options. It is like running sar repeatedly on the daily data, and saving all the output into the sar file.

On low-load systems, I will probably NOT run sysstat process by default, i.e. so do NOT run systemctl enable sysstat, and remember to stop the service after you have completed any performance analysis. (You could also cron start the service and cron stop the service during low production times)

If you omit all options, and just run sar by itself, it will write the average values since the system was restarted, to STDOUT.

Common options:

True, but one big difference is that top is interactive while sar can collect data over a longer period of time and write to logs.

If the output shows %IO is more than zero for a longer period of time, you have a I/O bottleneck

You could run this in a crontab: sar 3 10 -o ~/troubleshooting-ddyy > /dev/null 2>&1

Other common uses of sar

• sar collects and displays ALL system activities statistics.
• sadc stands for “system activity data collector”. This is the sar backend tool that does the data collection.
• sa1 stores system activities in binary data file. sa1 depends on sadc for this purpose. sa1 runs from cron.
• sa2 creates daily summary of the collected statistics. sa2 runs from cron.
• sadf can generate sar report in CSV, XML, and various other formats. Use this to integrate sar data with other tools.
• iostat generates CPU, I/O statistics
• mpstat displays CPU statistics.
• pidstat reports statistics based on the process id (PID)
• nfsiostat displays NFS I/O statistics.
• cifsiostat generates CIFS statistics.

• /sys
• /proc
• /dev
• modprobe
• lsmod
• lspci
• lsusb

A custom bash script running over time can find leaks. For example

while true
do
echo ".oO0o. .oO0o. .oO0o. .oO0o. .oO0o. .oO0o." >> ~/find-my-leak.txt
date
ps -aux
sleep 180

After this is run for a while, you can analyse the file ~/find-my-leak.txt

I found this suggestion on StackExchange.

1. Find out the PID of the process which causing memory leak. ps -aux
2. capture the /proc/PID/smaps and save into some file like BeforeMemInc.txt.
3. wait till memory gets increased.
4. capture again /proc/PID/smaps and save it has afterMemInc.txt
5. find the difference between first smaps and 2nd smaps, e. g. with diff -u beforeMemInc.txt afterMemInc.txt
6. note down the address range where memory got increased, for example:

   beforeMemInc.txt            afterMemInc.txt
   ---------------------------------------------------
   2b3289290000-2b3289343000   2b3289290000-2b3289343000  #ADDRESS
   Shared_Clean:    0 kB       Shared_Clean:    0 kB          
   Shared_Dirty:    0 kB       Shared_Dirty:    0 kB
   Private_Clean:   0 kB       Private_Clean:   0 kB
   Private_Dirty:  28 kB       Private_Dirty:  36 kB  
   Referenced:     28 kB       Referenced:     36 kB
   Anonymous:      28 kB       Anonymous:      36 kB  #INCREASE MEM
   AnonHugePages:   0 kB       AnonHugePages:   0 kB
   Swap:            0 kB       Swap:            0 kB
   KernelPageSize:  4 kB       KernelPageSize:  4 kB
   MMUPageSize:     4 kB       MMUPageSize:     4 kB
   Locked:          0 kB       Locked:          0 kB
   VmFlags: rd wr mr mw me ac  VmFlags: rd wr mr mw me ac
   

7. use GDB to dump memory on running process or get the coredump using gcore -o process
8. Use gdb on running process to dump the memory to some file.

   gdb -p PID
   dump memory ./dump_outputfile.dump 0x2b3289290000 0x2b3289343000
   

9. now, use strings command or hexdump -C to print the dump_{outputfile.dump}

strings outputfile.dump

1. You get readable form where you can locate those strings into your source code. Analyze your source to find the leak.

To map the memory of a process. See man pmap if my pid is 2785, try pmap 2785 or pmap -X 2795

Running top or htop and sorting by memory usage, the comparing over time is one easy approach.

You could also run top non-interactively, i.e. in batch mode with -b option. Just remember that you also need to specify the -n option to limit the max number of iterations to run. so: top -b -n 10 You could also stretch this out by increasing the delay between updates with the d 5 option for 5 second intervals

Then you could use awk, sort, and other inline Linux commands to search for memory leakages (numbers going up)

On my Alma C8host, top gives me these headings (by default)

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3804 zintis    20   0  749592  53912  36700 S   2.4   0.5   0:10.59 gnome-terminal-

Other column headings can be chosen, but I will here describe just the default column heading meanings:

• PID process ID of the task
• USER the effecive username of the task's owner
• PR priority
• NI nice value. negative means higher priority, positive means lower
• VIRT virtual memory used by the task (includes RES, SHR, SWAP … )
• RES resident memory, a subset of VIRT that represents the non-swapped physical memory currently used.
• SHR shared memory, a subset of RES that may be used by other processes
• S process status:
  • D uninterruptible sleep
  • I idle
  • R running
  • S sleeping
  • T stopped by job control signal
  • t stopped by debugger during trace
  • Z zombie
• %CPU duh
• %MEM duh
• TIME+ duh
• COMMAND duh

From man top:

%MEM - simply RES divided by total physical memory
CODE - the `pgms' portion of quadrant 3
DATA - the entire quadrant 1 portion of VIRT plus all
       explicit mmap file-backed pages of quadrant 3
RES  - anything occupying physical memory which, beginning with
       Linux-4.5, is the sum of the following three fields:
       RSan - quadrant 1 pages, which include any
              former quadrant 3 pages if modified
       RSfd - quadrant 3 and quadrant 4 pages
       RSsh - quadrant 2 pages
RSlk - subset of RES which cannot be swapped out (any quadrant)
SHR  - subset of RES (excludes 1, includes all 2 & 4, some 3)
SWAP - potentially any quadrant except 4
USED - simply the sum of RES and SWAP
VIRT - everything in-use and/or reserved (all quadrants)

On RHEL based systems:

• /usr/lib/httpd/modules on 32 bit systems
• /usr/lib64/httpd/modules/ on 64 bit systems

You will most likely need to sudo yum install module-init-tools to add custom modules to your kernel.

On other Linux systems:

• /lib modules`uname -r`/kernel/drivers

To list all the module installed on a RHEL system, use grubby --info=ALL and possibly: grubby --inf=all | grep kernel if just looking for kernel modules.

On RHEL systems, the file /lib/modules/`uname -r`/modules.dep keeps the list of kernel module dependencies, for that particular kernel version.

You can generate this list using depmod program. You will need the kmod package installed. kmod is short for kernel modules.

1. Write our kernel module, or download the source code for the custom module.
2. Compile the source code
3. Take the resulting .ko file, i.e. zpmod.ko and run it. The module will load and run, but only until next reboot
4. Make it permanent, i.e. auto load on boot up as follows: a) edit /etc/modules and add the name of the module without the .ko extnsn b) copy zpmod.ko file to /lib/modules/`uname -r`/kernel/drivers. Now the module will be in the modprobe database. c) Run depmod that will find all teh dependencies of your module. d) Confirm that the module is loaded at boot with lsmod | grep zpmod

Repeating, custom modules will load on boot if: (depending on the system)

1. they are listed in /etc/modules ( a file, one line per module listed)
2. create a module.conf file in the /etc/mdoules-load.d/ directory. Each module that has a module.conf file will be loaded at boot time.

   i.e. The files in /etc/modules-load.d/ directory are text files that list the modules to be loaded at boot, one per line.

3. list them in /etc/modprobe.d See. .conf files in etc/modprobe.d
4. Ensure the module is configured to get loaded in either:
   • /etc/modprobe.conf,
   • /etc/modprobe.d/*,
   • /etc/rc.modules, or
   • /etc/sysconfig/modules/*

See the man pages for modprobe, lsmod, and depmod

On my AlmaLinux system, the modules are in:

• /lib/modules/4.18.0...

For example my network kernel modules were in:

• /lib/modules/4.18.0-348.12.2.el8_5.x86_64/kernel/net

The /etc/modprobe.d directory contains all the kernel module configuration files, which all end in .conf Any parameters for a kernel module can be specied either on the command line, or as a line in such a .conf file.

modinfo usb-storage Will give you detailed information about the kernel module usb-storage Notice the lack of the .ko ending. When issueing commands, you drop the .ko ending and just use the name of the module itself.

lists the mods currently in your kernel. Often like: lsmod | less. You will see that many modules are loaded at boot time. They are persistent.

Depending on the system , there are two ways to make a module persistent.

1. create a module.conf file in the /etc/mdoules-load.d/ directory. Each module that has a module.conf file will be loaded at boot time.

   i.e. The files in /etc/modules-load.d/ directory are text files that list the modules to be loaded at boot, one per line.

2. The kernel will load all the modules listed in the file /etc/modules, one line per module. i.e. the modules will be persistent.

Running depmod on a module will find all the dependencies of that module. Need this info when setting up persistent modules, so that the dependencies are loaded too.

installs a specific module (does not load dependencies for you).

loads a module and any dependencies. (typically preferred over insmod) If you use modprobe -v zpmod it will load the zpmod.ko module, and all its dependencies and show you which dependencies it loaded.

It looks for .ko files in ~/lib/modules/`uname -r`/ directory to then act upon it.

On redhat systems, modprobe -r is used in place of rmmod to remove a module.

remove a module from the kernel. Typically when you only need a specific hardware device very rarely. So, you can insmod x then use the h/w, then rmmod x.

It does not matter WHAT is in the timestamp file, a date, or some poem. What find -newer looks for is the modification time of that timestamp file So if you look at ls -l /tmp/timestamp you will see the actual time that finder will use.

Extra Packages for Enterprise Linux repository is missing, and will not be installed. If you did want to install it it would be with:

EPEL repository

sudo yum install epel-release sudo yum repolist or sudo dnf ???? this needs finishing…

Once EPEL has been installed above, you can install htop using: turn on EPEL repo

sudo yum search htop
sudo yum install htop    or sudo yum -y htop
sudo yum info htop
sudo yum update htop
sudo yum info htop

You may also consider bashtop along with htop. bashtop uses more resources but is nicer and possibly easier to use.

To change a user to use /usr/bin/bash use the command:

• chsh /usr/bin/bash betty OR
• usermod -s /usr/bin/bash betty

To change a user to use /sbin/nologin use the command:

• chsh /sbin/nologin apache

OR

• usermod -s /sbin/nologin apache
• usermod -s /sbin/nologin nginx
• usermod -s /sbin/nologin mysql

The file /etc/groups shows groups. However the command group <userid> will show what groups a user belongs to.

Use sudo usermod -a -G wheel sally to add sally to the wheel group Or sudo usermod -a -G nginx zintis to make user zintis to sudo group.

To change an existing user's existing group to some new group use: sudo usermod -g newgroup sally Now sally will have a primary group "newgroup"

Use sudo groupadd newestgroup55 to add a new group called newestgroup55

This can be done in one of two ways.

1. using the chown -R userid:groupid <file(s)> for example to change all files in this directory and all subdirectories to have owner and group be nginx:nginx chown -R nginx:nginx *
2. use chgrp -=R nginx * to change the group of each file to nginx and do it for all subdirectories too.

To hit just the filename hosts the command would be chgrp wheel hosts

inodes are the metadata of a file . inodes specify the data structure for file metadata. The size of your inodes are determined when the disk drive or partition is created. Usually defaults are fine. If you know that your partition will be used for either relatively few, but large files, you could save some space by creating less inodes, OR, if you know you will have very many relatively small files, you should partition your disk with many more inodes.

Unless you use zfs, a modern file system, the inodes are created on demand, so you should never run out.

check inode of a file with stat filename

check free inodes using df -i or df -hi

• ls -i

will show you that an inode is paired with every file. An inode contains these things:

• size in bytes
• location on disk
• permissions
• owner
• group owner
• file creation/modification/access times
• reference count (i.e. how many hardlinks reference this file)

You can think of a directory as a table of filenames and their associated inode.

on ext3 and ext4 filesystems, the default number of inodes reserved is one inode per 16 kB of space. So on average if you have files that are 16 kB large you can fill your space. If you have on average files of less than 16 kB, then you may run out of inodes, called inode exhaustion.

inodes belong to the file, not the directory.

To display current settings and time: timedatectl without any arguments.

To set time zone: timedatectl set-timezone America/Toronto (EDT, -0400) If you don't know what the timezone is called, just list them all: timedatectl list-timezones | grep -i america

To adjust the time forward 5 hours:

To determine if you machine has a tsc; cat /proc/cpuinfo | grep constant_tsc If you get any output you have tsc.

If your host was suspended or put to sleep, the time of the guest will be out of sync when they wake up.

The simple solution is to reboot the guest, or not go to sleep on the host.

It turns out you can use less and get more functionality, even when compared with tail -f. Simply use the "command" F while in less and you will get auto updates at the end of the file, yet still be able to scroll back lines, and then go back to the "end" with G etc.

command << HERESTRING
    text1
    text2
    ...
    textn
HERESTRING

The HERESTRING is often EOF or >>EOF or just EOF The command can be any bash command, for example wc -l which would just show you who many lines are in the HEREDOCUMENT.

For a more useful example, you could cat a here document that lists the arguments of the calling command:

#!/usr/bin/env bash
cat << EOF
   0th argument is: $0
   1st argument is: $1
   2nd argument is: $2
EOF

The run that script with Apples oranges peaches you should get

0th argument is: Apples
1st argument is: oranges
2nd argument is: peaches

An even better example for ftp:

ftp -n << MYFTP 2> /dev/null
       open ftp.acme.com
       user anonymous zintis@cisco.com
       ascii
       prompt
       cd folderofgoodstuff
       mget file1 file2 file3
       bye
MYFTP

used for input redirection from text or a variable. The input is included in the same line with single quoatation marks.

wc -w <<< 'Hello World!'

I have also seen here docs starting with the HERESTRING in single quoatation marks, such as

#!/usr/bin/env bash
cat << 'EOF' 
   0th argument is: $0
   1st argument is: $1
   2nd argument is: $2
EOF

This will show lines that contain 'string' as well as 3 lines after that line. Try this: grep "if " -A3 ~/bin/python/bin/*.py

Kind of useful, I think.

Similarily this shows the line and 1 line before that line as well. grep "elif" -B2 ~/bin/python/bin/*.py

grep just looks for the actual string egrep interprets certain characters as grep commands, like | for "or", & for "and"

By default the $IFS is set to a <space>, i.e. ' ' You can change that in a script, but good idea is to change it back to what it was when you started. We need to save that as $OLDIFS. Like this example, that has input that are comma separated i.e. csv

#!/bin/env bash
OLDIFS=$IFS
IFS=","
... do you thing with inputs that are separated with single tics
IFS=$OLDIFS

#!/bin/env bash
OLDIFS=$IFS
IFS=","

while read user job uid location
# first four fields in a csv file
  do
    echo -e "\e[1;33m$user \
    ==================\e[0m\n\
    Role : \t $job\n\
    ID : \t $uid\n\
    Site : \t $location\n"
  done < $1

IFS=$OLDIFS

note that the echo command above could all have been written on one line but the "\" character simply spans the echo statement across to the next line in the code. It has nothting to do with the newline character, which is \n apart from often coming right after it. You could have written

do
   echo -e "\e[1;33m$user  ==================\e[0m\n Role : \t $job\n ID : \t $uid\n Site : \t $location\n"
 done

And gotten the exact same output from the script.

This seciton has not been completed.

My Almalinux was booting into a command line, so I issued the command:

• sudo systemctl set-default graphical

After that my system tried to boot into a GUI, but I got this error:

So now I have to get out of this GUI, then fix what is wrong. It's like I dug a deeper hole.

First step is to realize that in this window, it is the GUI that is failing, and your system may have already booted ok. To get out of the GUI, try:

• Ctrl-Alt-F4

This will get you to a command prompt. I had to change my mac keyboard touchbar settings to show F1, F2, etc. I usually have the touchbar set to Expanded Control Strip. I have to remember to set it back. Especially if you have swapped Caps Lock key with the fn key, like I have. You need to have fn key set to Change Input Source for the caps lock switch to work.

To temporarily change the run level or target, use isolate To make the change permanent use set-default. To read what is currently set, use get-default

To switch from GUI to CLI: systemctl isolate multi-user.target

To switch from CLI to GUI: systemctl isolate graphical.target

To list available targets: systemctl --type=help

To set the CLI as a default runlevel (target in systemd terminology):\ systemctl set-default multi-user.target.

Analogously for GUI: systemctl set-default graphical.target