FANDOM


Basic

User Management

Memory Management

Disk and Filesystem Management

  • mount : mount a filesystem
  • tune2fs : adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems
  • dumpe2fs : dump ext2/ext3/ext4 filesystem information

Ext4 Filesystem

Service Control

  • init
  • systemd : provides a system and service manager that runs as PID 1 and starts the rest of the system
  • Upstart : an event-based replacement for the /sbin/init daemon
  • inetd
  • xinetd : a secure replacement for inetd

Job Control

Network Management

Monitoring & Diagnosis

Process

CPU

Memory

Disk IO

Virtualization

Shell

Bash

When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When a login shell exits, bash reads and executes commands from the files ~/.bash_logout and /etc/bash.bash_logout, if the files exists.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

Bash programming general

Shell expansions

Operators and expressions

String

Arithemetic

Array

Redirections

find ... > found.txt              # write down only output
find ... > out_and_err.txt 2>&1   # write down both output and error
find ... 2> /dev/null             # hide only error
find ... > /dev/null 2>&1         # hide both output and error

If statement

For statement

getopts

X11

GNOME

Linux Distributions

CentOS

Utilities

journalctl

Ubuntu

Repositories

Commands

Fedora

Red Hat Enterprise Linux

UNIX Systems

AIX

AIX Commands

  • oslevel
    • Reports the latest installed level (technology level, maintenance level and service pack) of the system.
  • lsdev
    • Displays devices in the system and their characteristics.
  • prtconf
    • Displays system configuration information.
  • lparstat
    • Reports logical partition (LPAR) related information and statistics.
  • no
    • Manages network tuning parameters.

Solaris

Commands/Utilities

Common

Command Description Readings
find find files POSIX.1-2008/Utilities/find
id return user identity http://en.wikipedia.org/wiki/Id_(Unix)
source read and execute ex commands from file POSIX.1-2008/Utilities/ex/source
. evaluates commands in a computer file in the current execution context [1]
sudo allows users to run programs with the security privileges of another user (normally the superuser, or root) http://en.wikipedia.org/wiki/Sudo
wc word, line, and byte or character count POSIX.1-2008/Utilities/wc
grep search a file for a pattern POSIX.1-2008/Utilities/grep
tee reads standard input and writes it to both standard output and one or more files, effectively duplicating its input. tee
curl command line tool and library for transferring data with URLs Manual
man page
wget non-interactive download of files from the Web GNU Wget Manual
scp secure remote file copy program

su

sudo

tee

curl

scp

Editing

sed

Typical command
Command Syntax Sample Remark
Substitution 's/pattern/replacement/option' sed 's/\r$//g' README.windows.txt > README.linux.txt
Append Lines 'address a text-to-append' sed -i '2 a export GOROOT=/usr/lib/go-1.8\nPATH=$PATH:$GOROOT/bin\n' .profile
sed -i '$ a \\nexport GOROOT=/usr/lib/go-1.8\nexport PATH=$PATH:$GOROOT/bin\n' ~/.bashrc
'$' as an address means the last line.
'\n' at the beginning of replacement string need additional leading backslash to form '\\n'.
Change Lines 'address c text-to-change'
Delete Lines 'address d'
Character escape

The only difference between basic and extended regular expressions is in the behavior of a few characters: ‘?’, ‘+’, parentheses, braces (‘{}’), and ‘|’. While basic regular expressions require these to be escaped if you want them to behave as special characters, when using extended regular expressions you must escape them if you want them to match a literal character.

awk

read

Parsing

Jshon

jq

Administration

useradd

mount

clusterssh

parallel

Terminator

Monitoring & Diagnosis

Command Description Misc
sysctl examining and changing kernel parameters at runtime
top
lsof displays information about files open to Unix processes
iostat collect and show operating system storage input and output statistics
strace
netstat
tcpdump

ps

sysctl

top

lsof

strace

iotop

hdparm

dd

iostat

netstat

tcpdump

iperf

iperf3

Networking

brctl

iptables

traceroute

conntrack

PAC Manager

misc

watch

screen

Tips

Diagnosing and Monitoring Linux

Identifying the product of Linux installed

For Linux, /etc/issues file contains more detailed information on what Linux product it is.

$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Kernel \r on an \m

For most of Linux distributions, /etc/*-release file is provided which contains more detailed and systematic information. The file would be lsb-release, os-release, redhat-release or so on as to the Linux distributions.

$ cat /etc/*-release
...

Identifying kernel version

$ more /proc/version
Linux version 4.4.0-83-generic (buildd@lgw01-29) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017

Identifying kernel parameters

$ sysctl -a | more

Identifying threads of a specific process

$ ps -T -p 31

Identifying the memory capacity and usage

$ cat /proc/meminfo
MemTotal:        3922904 kB
MemFree:         3037280 kB
...

Listing disks

$ lsblk
...

Listing filesystems

To list filesystems with capacity and usage, use df

$ df -ahT
...

To list filesystems with mount options, use mount command

$ mount -l
...

Identifying the details of a certain filesystem

$ sudo dumpe2fs -h /dev/xvda2
...

Identifying the filesystem a certain file belongs to

$ df -h /var   #asking what filesystem contains /var directory
...

Checking disk caching

$ hdparm -W /dev/sda
...

Identifying disk detail information

$ hdparm -i /dev/sda
...

Benchmarking disk performance

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=append,noatime
done   

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=fdatasync,notrunc oflag=append,noatime
done   

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=dsync,append,noatime
done

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=direct,append,noatime
done
References
Each CONV symbol may be:

  ascii     from EBCDIC to ASCII
  ebcdic    from ASCII to EBCDIC
  ibm       from ASCII to alternate EBCDIC
  block     pad newline-terminated records with spaces to cbs-size
  unblock   replace trailing spaces in cbs-size records with newline
  lcase     change upper case to lower case
  ucase     change lower case to upper case
  sparse    try to seek rather than write the output for NUL input blocks
  swab      swap every pair of input bytes
  sync      pad every input block with NULs to ibs-size; when used
            with block or unblock, pad with spaces rather than NULs
  excl      fail if the output file already exists
  nocreat   do not create the output file
  notrunc   do not truncate the output file
  noerror   continue after read errors
  fdatasync  physically write output file data before finishing
  fsync     likewise, but also write metadata

Each FLAG symbol may be:

  append    append mode (makes sense only for output; conv=notrunc suggested)
  direct    use direct I/O for data
  directory  fail unless a directory
  dsync     use synchronized I/O for data
  sync      likewise, but also for metadata
  fullblock  accumulate full blocks of input (iflag only)
  nonblock  use non-blocking I/O
  noatime   do not update access time
  nocache   Request to drop cache.  See also oflag=sync
  noctty    do not assign controlling terminal from file
  nofollow  do not follow symlinks
  count_bytes  treat 'count=N' as a byte count (iflag only)
  skip_bytes  treat 'skip=N' as a byte count (iflag only)
  seek_bytes  treat 'seek=N' as a byte count (oflag only)

Identifying TCP/IP ports currently in use

You can identify TCP/IP ports currently in use using netstat command. The options of netstat is slightly different among operating systems.

For UNIX,

$ netstat -anotu

For Linux,

$ sudo netstat -anoptu

(-a : all sockets, -n : numeric, -o : timers, -p : PID/program, -t : TCP, -u : UDP)

You need root privilege to take effect of -p option To find out whether a given port is being used or not, use grep command.

$ sudo netstat -anoptu | grep -E '(^Proto)|(8080)'

For Windows,

$ netstat -ano

For more about netstat, refer topics in Wikipedia.

Monitoring network traffic by interface card

watch -n 2 netstat -i

Capturing incoming HTTP request using tcpdump

sudo tcpdump -s 0 -A -i eth1 dst port 80

Readings

Diagnosing and Monitoring AIX

Identifying System Configuration

Using Common Shell Commands

Identifying the shell of your current login

To identify what shell a user is set to use by default, you can check SHELL variable.

$ echo $SHELL
/bin/bash
$ bin/ksh
$ echo $SHELL
/bin/bash

As the above example shows, SHELL variable contains the login default shell type not the one currently in use.

Hiding the output of command

To hide both the normal output and error output, redirect stdout and stderr to null device

% npm ls -g json >/dev/null 2>&1
% #or
% npm ls -g y18n &>/dev/null

To hide only the error output, redirect stderr to null device

% npm ls -g json 2>/dev/null

Listing files using find command excluding files with 'Permission denied'

When executing find command in simplest format, you may get lots of lines just saying that 'Permission denied'. Most cases, those are not what you want, and lots of permission denied lines can disturb you identifying the wanted result.

You can use stderr redirection to cut out permission denied files (or directories).

% find / -name '*.jar' 2>/dev/null

Finding files having specified name with full path

If you want to find files with extension of 'jar' and print them with full path, use find command with -exec operator like the following.

% find . -name '*.jar' -exec ls -l {} \;

For more about find command and -exec operator including strange '{}' or '\;' in the above example, refer the followings.

Finding files containing the specified word

% find /home |xargs grep "password"

For more about xargs, refer the followings.

Finding large files

To find large files(not directories) under current directory and list them in pages, use the following command.

% find . -type f -exec du -k {} 2>/dev/null \; | sort -nr | more

To filter out small files, you can use size option with find command, or to filter out some subdirectories you can redirect the result to grep command. The following command will list files whose size are more than 1 mega-byte under current directory recursively except the subdirectories starting with 'svn' in order of their size.

% find . -type f -size +1000000c -exec du -k {} 2>/dev/null \; | sort -nr | grep -E "\./svn.*" -v

Listing all files under a certain directory recursively

Using find command

% find /proc/sys -type f 2>/dev/null | more

Listing distinct file extensions of all files under a directory

% find . -type f -name "*.*" | sed -r 's/^.+(\.\w+)$/\1/' | sort | uniq

Counting files under a directory recurssively

% find . -type f -print | wc -l

Counting files in a tar file

% tar -tvf archive.tar | grep "^-.*" | wc -l

Inverse matching with grep command

To find lines not matching the specified patterns in a file, you can use -v option with grep command.

$ svn list -R http://.../repos1 | grep -v -E '(.*java|.*/)'

You don't need to be bothered to find out how to use complex negative patterns with regex.

Viewing files in octal or hexadecimal format - od

You can view non ascii base files in hexadecimal format using od command.

% od -A d -x journal.log

For more about od, refer the following.

Viewing file contents without line wrapping - less -S

% less -S known_hosts

Viewing the result of ps command without line wrapping

You can redirect the result to cat or less command, or use ww flag.

% ps auxf | cat
...
% ps auxfww
...
% ps auxf | less -+S

Viewing file contents without comments lines (starting with #)

% cat /etc/apt/sources.list | grep -P '^[^#].*'

Sorting the file system usage result from the du command

You can sort the output of du command applying pipe to sort command.

% du -m | sort -n

For more about du and sort, read the followings.

Getting multiple files form the target URL using wget command

wget provide --accept or -A switch which can represent multiple files using comma separated list, wild card, or character class. But it's not that -A switch support regular expression.

$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '198[7-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '199[0-9].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '200[0-8].csv.bz2' http://stat-computing.org/dataexpo/2009/)"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A 'airports.csv, carriers.csv, plane-data.csv' http://stat-computing.org/dataexpo/2009/)"

For more, refer the following

Adding lines to a specific row of a file using sed command

$ sed -i '2 a export GOROOT=/usr/lib/go-1.8' .profile
$ sed -i '3 a export PATH=$PATH:$GOROOT/bin\n' .profile

Bash Programming

Directory containing the current script

readonly dir=$(cd `dirname $0` && pwd)

Looping array

For indexed array

fruits=(Apple Banana Kiwi)

for f in ${fruits[@]}; do
  echo $f
done;

For correlated arrays,

fruits=(Apple Banana Kiwi)
colors=(red yellow green)

for (( i=0; i<${#fruits[@]}; i++ )); do
  echo ${fruits[$i]} is ${colors[$i]}
done;

For associate array,

declare -Ar vm_params=( # kernel parameters for virtual memory
  [swappiness]=1
  [dirty_ratio]=10
  [dirty_background_ratio]=5
  [min_free_kbytes]=262144 #256MB
)

for param in "${!vm_params[@]}"; do
  echo "Updating 'vm.${param}' to '${vm_params[${param}]}'."
  sudo sysctl -wq vm.${param}=${vm_params[${param}]}

  if [ `grep -E "^\\s*vm.${param}\\s*=\\s*${vm_params[${param}]}.*" /etc/sysctl.conf | wc -l` -ne 1 ]; then
    sudo sed -r -i 's/^\s*(vm\.'"${param}"'.*)/#\1/g' /etc/sysctl.conf
    sudo sh -c "echo 'vm.${param}=${vm_params[${param}]}' >> /etc/sysctl.conf"
  fi
done

2 dimentional array

declare -a m
m[0]='a b c d'
m[1]='e f g h'
m[2]='i j k l'
m[3]='m n o p'
m[4]='q r s t'

for r in "${m[@]}"; do
  echo $r
  for c in $r; do
    echo $c
  done
done

awk script typical snippet

Use awk to structure target data table from the input source and process each row using while statement assigning each column to its own variable using read.
So typical script flow is awk ... | while read var1 var2 var3; do ...; done like the following real example.

awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do

  no=${vm##vm}  # remove left 'vm'
  org_no=$(($(($((no - 1))/org_size)) + 1))
  peer_str="{\"requests\": \"grpcs://${ip}:7051\", \"events\": \"grpcs://${ip}:7053\", \"server-hostname\": \"peer${no}\", \"tls_cacerts\": \"tlsCerts/org${org_no}.com/tlsca${org_no}-cert.pem\"}"

  # Defines paired peer no : (vm1-vm2, vm3-vm4, ...)
  if [ $((no % 2)) -eq 1 ]; then
    no2=$((no + 1))
  else
    no2=$((no - 1))
  fi

  # For current peer vm
  sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer1' part of './generated/vm${no}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer1' part of './generated/vm${no}/channelConfig.json' file."
  fi

  # For paired peer vm
  sed -i -r 's#@@peer2@@#\"peer2\": '"$peer_str"'#' ./generated/vm${no2}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
    exit 1
  else
    echo "Successfully updated 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
  fi
done

parallel script typical snippet

For local tasks

dirs=('/boot' '/proc' '/sys' '/var' '/sbin');

for dir in ${dirs[@]}; do echo $dir; done | parallel --no-notice --bar --joblog /dev/stdout '
  files=$(sudo find {} -type f 2>/dev/null | wc -l);
  echo {} ":" ${files} "files"
'

For remote tasks

hosts=('192.168.100.1' '192.168.100.2' '192.168.100.3');

for host in ${hosts[@]}; do echo $host; done | parallel --no-notice --bar --joblog /dev/stdout '
  name=$(ssh -o StrictHostKeyChecking=no paul@{} echo "\`hostname\`");
  echo {} ":" ${name};
'

Managing Packages

Ubuntu

Installing a new software package
  1. Update the package information
  2. Check whether or not the software package is already installed
  3. Search and review the software package to install
  4. Install or upgrade the software package
  5. (Optionally) Confirm all the files installed by the package

Not using apt, which is preferred 'cause apt doesn't support in-script usage well.

$ sudo apt-get update                                 # update package information
...
$ sudo dpkg -l | awk '{print $2}' | grep golang-1.8   # check previous installation
...
$ sudo apt-cache --names-only search ^golang | more   # search available packages
...
$ sudo apt-cache show golang-1.8                      # review the software to install
...
$ sudo apt-get install golang-1.8                     # install the software
...
$ sudo apt-file list golang-1.8                       # confirm all the files installed

Using apt

$ sudo apt update                          # update package information
...
$ sudo apt list *golang-1.8* --installed   # check previous installation
...
$ sudo apt search ^golang | more           # search available packages
...
$ sudo apt show golang-1.8                 # review the software to install
...
$ sudo apt install golang-1.8              # install the software
...
$ sudo apt-file list golang-1.8            # confirm all the files installed
  • References
    • apt-get
    • apt-cache
    • apt-file : searching files in packages for the APT package management system.
    • apt : provides a high-level commandline interface for the package management system.
Installing a software package specifying version with wildcard
$ sudo apt-get install nodejs=6.10.2*
Listing all installed packages
$ sudo dpkg -l | awk '{print $2}'

Or

$ sudo apt list --installed

Ad blocker interference detected!


Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.