File/Directory Description Remarks
Parameter Description Remarks
fs.file-max the maximum number of file-handles that the Linux kernel will allocate
fs.file-nr the number of allocated file handles, the number of free(allocated but unused) file handles, and the maximum number of file handles
fs.nr_open the maximum number of file-handles a process can allocate default: 1048576 (1024*1024)
File Item Description Remarks
/proc/stat ctxt the total number of context switches across all CPUs
btime the time at which the system booted, in seconds since the Unix epoch
processes the number of processes and threads created, which includes (but is not limited to) those created by calls to the fork() and clone() system calls
procs_running the number of processes currently running on CPUs
procs_blocked the number of processes currently blocked, waiting for I/O to complete


Environment Variables

Variable Description Remarks
IFS determines how bash recognizes fields, or word boundaries, when it interprets character strings Internal Field Separator
LC_ALL determines the values for all locale categories
TZ Timezone information

User Management

  • Password filed of shadow file
Value Description Remarks
$id$salt$hashed the printable form of a password hash as produced by crypt
empty string No password
! the account is password locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed
* the account is locked, user will be unable to log in via password authentication but other methods (e.g. ssh key) may be still allowed

Memory Management

Disk and Filesystem Management

  • Logical volume management
    • Physical Volume Group (PVG) = ∑ Physical Volume
    • Physical Volume (PV) = ∑ Physical Extent
    • Logical Extentn = Physical Extentm
    • Logical Volume (LV) = ∑ Logical Extent (in a single PVG)
  • procfs (proc filesystem)
    • a special filesystem in Unix-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory
    • CentOS / /proc/sys/ : provides information about the system and allows the system administrator to immediately enable and disable kernel features.
File/Directory Description Remarks
/proc/stat/ Overall/various statistics about the system, such as the number of page faults since the system was booted
/proc/{pid}/cmdline a file containing full command-line for this process
/proc/{pid}/cwd a symbolic link to the current working directory of this process
/proc/{pid}/exe a symbolic link to the actual executable file for this process
/porc/{pid}/environ a file containing environment variables used by this process
/proc/{pid}/status a file containing basic information for this process including its run state and memory usage
/proc/{pid}/fd/ a directory containing symbolic links for the all the open file descriptors by this process
/proc/{pid}/task/ a directory containing hard links for the all the tasks that have been started by this process
  • tmpfs
    • a common name for a temporary file storage facility on many Unix-like operating systems


command description remarks
mount mount a filesystem
tune2fs adjust tunable filesystem parameters on ext2/ext3/ext4 filesystems
dumpe2fs dump ext2/ext3/ext4 filesystem information

Well-Known Directories

Directory/File Description Remarks
/var/lib/dpkg/info contains scripts (pre-install, post-install, pre-remove, post-rmove) of debian/APT packages

Filesystem Types

Filesystem Description Remarks
SquashFS a compressed read-only filesystem for Linux

Ext4 Filesystem

Service Management

Init system Released when/with Remarks
SysV init 1983
Upstart 2006, Ubuntu 6.10
systemd 2011, Fedora 15


Command Syntax Remarks
List all services sudo service --status-all
Start a service sudo service script start service sshd stop
Check a status of a service sudo service script status service sshd status
Stop a service sudo service script stop service sshd stop
Restart a service sudo service script restart service sshd restart
List all scripts ls /etc/init.d/



Manpage Description Remarks
systemd.unit Unit configuration
systemd.service Service unit configuration
systemd.exec Execution environment configuration
  • If systemd encounters an unknown option, it will write a warning log message but continue loading the unit. If an option or section name is prefixed with X-, it is ignored completely by systemd.
  • Along with a unit file foo.service, a "drop-in" directory foo.service.d/ may exist. All files with the suffix ".conf" from this directory will be parsed after the unit file itself is parsed. This is useful to alter or add configuration settings for a unit, without having to modify unit files. Drop-in files must contain appropriate section headers. For instantiated units, this logic will first look for the instance ".d/" subdirectory (e.g. "foo@bar.service.d/") and read its ".conf" files, followed by the template ".d/" subdirectory (e.g. "foo@.service.d/") and the ".conf" files there. Moreover for units names containing dashes ("-"), the set of directories generated by truncating the unit name after all dashes is searched too. Specifically, for a unit name foo-bar-baz.service not only the regular drop-in directory foo-bar-baz.service.d/ is searched but also both foo-bar-.service.d/ and foo-.service.d/. This is useful for defining common drop-ins for a set of related units, whose names begin with a common prefix.
  • In addition to /etc/systemd/system, the drop-in ".d/" directories for system services can be placed in /usr/lib/systemd/system or /run/systemd/system directories. Drop-in files in /etc take precedence over those in /run which in turn take precedence over those in /usr/lib. Drop-in files under any of these directories take precedence over unit files wherever located. Multiple drop-in files with different names are applied in lexicographic order, regardless of which of the directories they reside in.
    • /etc/systemd/system/foo.service.d > /run/systemd/system/foo.service.d > /usr/lib/systemd/system/foo.service.d
  • systemctl --state=help
Category Enum Description
Load States loaded
Active States active
Specifier Title Description
%i Instance name For instantiated units this is the string between the first "@" character and the type suffix. Empty for non-instantiated units.
%I Unescaped instance name Same as %i, but with escaping undone.
  • Control the systemd system and service manager
Command Description Flags Remarks
systemctl list-units List units that systemd currently has in memory
systemctl list-unit-files List unit files installed on the system, in combination with their enablement state
systemctl enable Enable one or more units or unit instances
systemctl disable Disables one or more units
systemctl start Start (activate) one or more units specified on the command line
systemctl stop Stop (deactivate) one or more units specified on the command line
systemctl restart Stop and then start one or more units specified on the command line
systemctl reload Asks all units listed on the command line to reload their configuration
systemctl reload-or-restart Reload one or more units if they support it. If not, stop and then start them instead.
systemctl mask
systemctl unmask
systemctl status Show terse runtime status information about one or more units, followed by most recent log data from the journal
systemctl show Show properties of one or more units, jobs, or the manager itself
systemctl cat Show backing files of one or more units
systemctl is-enabled Checks whether any of the specified unit files are enabled (as with enable)
systemctl is-active Check whether any of the specified units are active (i.e. running)
systemctl list-dependencies --all, --reverse
systemctl mask
systemctl edit /etc/systemd/system/service_name.d/override.conf
Tips and Tricks
Check whether a service support reload or not and how it is supported

Check CanReload and ExecReload properties using systemctl show command.

$ systemctl show telegraf.service | grep -i reload
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }


  • inetd : a super-server daemon on many Unix systems that provides Internet services
  • xinetd : a secure replacement for inetd

Process Management

Signal Number Description Remarks
SIGHUP 1 Usually means that the controlling pseudo or virtual terminal has been closed
SIGKILL 9 Sent to a process to cause it to terminate immediately

Network Management


Monitoring & Diagnosis




Disk IO





When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When a login shell exits, bash reads and executes commands from the files ~/.bash_logout and /etc/bash.bash_logout, if the files exists.

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc.

Bash programming general

Special Parameters

Parameter Description Remarks
$@ expands to the positional parameters, starting from one When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$@" is equivalent to "$1" "$2" ….
$# expands to the number of positional parameters in decimal
$? expands to the exit status of the most recently executed foreground pipeline.
!$ designates the last argument of the preceding command.

Shell expansions

Expansion Syntax Description Remarks
Brace Expansion {str1,str2,str3, ...}, {x..y[..incr]} $ mkdir /opt/lego/{bin,conf,lib,docs,log,tmp}
Tilde Expansion ~, ~dir(/dir)*, ~/user(/dir)*
Shell Parameter Expansion ${parameter:-word}, ${parameter:=word}, ${parameter:?word}, ${parameter:+word}, ${parameter##word}, ${parameter%%word}, ...
Command Substitution $(command), `command` `echo $CODE` | base64`
$(echo $CODE | base64)
Arithmetic Expansion $(( expression ))
Filename Expansion Pattern syntax in file name expansion is different from general regex patterns.
History Expansion !!, !n, !-n(!-1, !-2, !-3 ...), !!:$, !$, !0
Shell Parameter Expansion
Expression Meaning Remarks
${parameter:-word} If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted.
${parameter/pattern/string} Parameter is expanded and the longest match of pattern against its value is replaced with string.
${parameter#pattern} If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern deleted.
${parameter%pattern} If the pattern matches a trailing portion of the expanded value of parameter, then the result of the expansion is the value of parameter with the shortest matching deleted.
${parameter:offset(:length)} Expands to up to length characters of the value of parameter starting at the character specified by offset.
${#parameter} The length in characters of the expanded value of parameter is substituted.
${#@}, ${#*} the number of positional parameters
${#arr[@]}, ${#arr[*]} the number of elements in the array
${!arr[@]}, ${!arr[*]} Expands to the list of array indices (keys) assigned in array named arr.
Filename Expansion
Basic Globbing
Wildcard Title Description Remark
* Any string matches any number of any characters including none
? Any single character matches any single character
[c1c2c3...] Any one of the enclosed characters matches one character given in the bracket
[s-e] matches one character from the (locale-dependent) range given in the bracket
[!c1c2c3...] matches one character that is not given in the bracket
[!s-e] matches one character that is not from the range given in the bracket
Extended Globbing on Bash

Operators and expressions





find ... > found.txt              # write down only output
find ... > out_and_err.txt 2>&1   # write down both output and error
find ... 2> /dev/null             # hide only error
find ... > /dev/null 2>&1         # hide both output and error

If statement

For statement







Linux-PAM (Pluggable Authentication Modules for Linux) is a suite of shared libraries that enable the local system administrator to choose how applications authenticate users.



command description remarks
sestatus SELinux status tool
setenforce modify the mode SELinux is running in
seinfo allows the user to query the components of a SELinux policy e.g.) seinfo -t, seinfo -r, seinfo -u -x
semanage SELinux Policy Management tool semanage boolean, semanage user, semanage login, semanage module, semanage port, semanage interface, semanage node, semanage fcontext
matchpathcon get the default SELinux security context for the specified path from the file contexts configuration
restorecon restore file(s) default SELinux security contexts
chcat change file SELinux security category
task command-line remarks
List all permissive types $ sudo semanage permissive -l
List all modules $ sudo semodule -l
List all module binaries of default policy $ ls /usr/share/linux/default
View log file $ sudo tail -f -n 300 /var/log/audit/audit.log
List all SELinux users $ sudo semanage user -l
List all login mappings $ sudo semanage login -l
List all roles $ seinfo -r
List all types $ seinfo -t
List types that are accessible for a role $ seinfo -rdbadm_r -x
List all security categories $ chcat -L

Files and Directories

file/directory description remarks
/etc/selinux/default/contexts/default_contexts used by SELinux-aware applications that need to set a security context for user processes (generally the login applications) default_contexts man page
/etc/selinux/default/contexts/users/ overrides rules in /etc/selinux/default/contexts/default_contexts by user


Define user

user user1_r roles { role1_r role2_r ... } level ...


  • SELinux adds type enforcement to standard Linux. This means that both the standard Linux and enhanced SELinux access controls must be satisfied to access an object. So, for example, if we have SELinux write access to a file but we do not have w permission on the file, we cannot write the file.
  • In general, consider domain, domain type, subject type, and process type to be synonymous.
  • Finally, be aware of the differences between the user ID in standard Linux security and the user identifier in a security context. Technically, these are completely orthogonal identifiers, used separately by the standard and security-enhanced access control mechanisms, respectively. Any relationship between these two is strictly provided via the login process according to conventions not directly enforced by the SELinux policy.
  • Remember that a type_transition rule causes a domain transition to be attempted by default, but it does not allow it.
  • SELinux Users can have multiple roles that they can reach, and then in those roles they can reach multiple types.
  • Three users that you will usually see on the system are "user_u", "system_u" and root. The user_u is the default SELinux User for a logged in user on a system. "system_u" is the default User for processes started during the boot up process.
  • The role field on a file is always object_r, and really has no meaning other than as a place holder.
  • RBAC(Roles Based Access Control) is not really used in targeted policy, but becomes more important in Strict and MLS policy.
  • Most of the policy rules in SELinux revolve around what subject types have what access to which object types.
  • LSM provides a set of hooks in the kernel system call logic. These hooks are usually placed after the standard Linux access checks but before the actual resource is accessed by the kernel on behalf of the caller.
  • In standard Linux, if you have a file descriptor, you can use it regardless of the change in file access mode. In SELinux, for objects such as files where access is validated on all attempts to use (for example, every read system call is checked against the policy and not just open calls), access revocation works fine.
  • Using and applying SELinux is all about writing and understanding policies.
  • SELinux dose not change the Linux DAC implementation nor can it override denials made by the Linux DAC permissions. If regular system (without SELinux) prevents a particular access, there is nothing SELinux can do to override this decision. This is because the LSM hooks are triggered after the regular DAC permission checks have been executed, which is a conscious design decision from the LSM project.
  • There are more than 80 classes and over 200 permissions known to SELinux and policy rules need to take into account all these classes and permissions for each interaction between two objects and resources.
  • SELinux has no notion of Linux process ownership and, once running, does not care how the process is called, witch processID it has, and what account the process runs as.
  • The majority of SELinux policy rules(over 99 percent) consists of rules related to the interaction between two types(without mentioning roles, users or sensitivity levels).
  • Multiple Linux users can be assigned to the same SELinux user.
  • When distributing SELinux policy modules, most Linux distributions place the *.pp SELinux policy modules inside /usr/share/selinux, usually withing a subdirectory named after the policy store.

Targeted Policy

  • Unconfined services executed by init end up running in the unconfined_service_t domain.
  • Unconfined services executed by kernel end up running in the kernel_t domain.
  • Unconfined services executed by unconfined Linux users end up running in the unconfined_t domain.

SELinux Reference Policy

Linux Distributions



  • Must Know
    • Creating and deleting partitions can be done from within debian-installer as well as from an existing operating system.
    • During hardware detection debian-installer checks if any of the drivers for the hardware devices in your system require firmware to be loaded. If any firmware is requested but unavailable, a dialog will be displayed that allows the missing firmware to be loaded from a removable medium.
  • Boot parameters
  • Procedure
    1. Check available memory / low memory mode
    2. Selecting Localization Options
      • language, country, locales
    3. Choosing a Keyboard
    4. Looking for the Ubuntu Installer ISO Image
      • When installing via the hd-media method
    5. Configuring the Network
    6. Configuring the Clock and Time Zone
      • /usr/share/zoneinfo/
    7. Setting Up Users And Passwords
    8. Partitioning and Mount Point Selection
      • supports LVM, Software RAID, Serial ATA RAID, and Encryption
    9. Installing the Base System
    10. Installing Additional Software
    11. Making Your System Bootable
    12. Finishing the Installation
Installer Components for 16.04
Components Description Remarks
localechooser Allows the user to select localization options for the installation and the installed system: language, country and locales.
console-setup Shows a list of keyboards, from which the user chooses the model which matches his own.
hw-detect Automatically detects most of the system's hardware, including network cards, disk drives, and PCMCIA.
cdrom-detect Looks for and mounts an Ubuntu installation CD.
netcfg Configures the computer's network connections so it can communicate over the internet.
iso-scan Searches for ISO images (.iso files) on hard drives.
choose-mirror Presents a list of Ubuntu archive mirrors.
cdrom-checker Checks integrity of a CD-ROM.
lowmem Tries to detect systems with low memory and then does various tricks to remove unnecessary parts of debian-installer from the memory
anna Installs packages which have been retrieved from the chosen mirror or CD. Anna's Not Nearly APT
user-setup Sets up the root password, and adds a non-root user.
clock-setup Updates the system clock and determines whether the clock is set to UTC or not.
tzsetup Selects the time zone, based on the location selected earlier.
partman Allows the user to partition disks attached to the system, create file systems on the selected partitions, and attach them to the mountpoints. LVM support
lvmcfg Helps the user with the configuration of the LVM (Logical Volume Manager).
mdcfg Allows the user to set up Software RAID.
base-installer Installs the most basic set of packages which would allow the computer to operate under Ubuntu when rebooted.
apt-setup Configures apt, mostly automatically, based on what media the installer is running from.
pkgsel Uses tasksel to select and install additional software.
shell Allows the user to execute a shell from the menu, or in the second console.
save-logs Provides a way for the user to record information on a floppy disk, network, hard disk, or other media when trouble is encountered.


PPA Title Description Remarks
ppa:jonathonf/backports Backport collection Backports of various (low impact) packages to Trusty and Xenial sqlite3
ppa:wireshark-dev/stable Wireshark stable releases Latest stable Wireshark releases back-ported from Debian package versions.
ppa:vbernat/haproxy-1.8 HAProxy 1.8 Contains packages for HAProxy 1.8.


  • tcp
    • /proc/sys/net/ipv4/

Service Control






Package Description Remarks
procps /proc file system utilities free, kill, ps, sysctl, top, uptime, vmstat, and watch
net-tools includes the important tools for controlling the network subsystem of the Linux kernel ifconfig, netstat, route, ...






Red Hat Enterprise Linux

UNIX Systems


AIX Commands

  • oslevel
    • Reports the latest installed level (technology level, maintenance level and service pack) of the system.
  • lsdev
    • Displays devices in the system and their characteristics.
  • prtconf
    • Displays system configuration information.
  • lparstat
    • Reports logical partition (LPAR) related information and statistics.
  • no
    • Manages network tuning parameters.


Commands, Utilities and Tools


Command Description Readings
id return user identity
set allows you to change the values of shell options and set the positional parameters, or to display the names and values of shell variables. Bash set builtin
source read and execute ex commands from file POSIX.1-2008/Utilities/ex/source
. evaluates commands in a computer file in the current execution context
sudo allows users to run programs with the security privileges of another user (normally the superuser, or root)
kill sends the specified signal to the specified process or process group
date print or set the system date and time



Options Description Remarks
-v Print shell input lines as they are read.set echo on
-x Print a trace of simple commands








Command Description Readings
find find files POSIX.1-2008/Utilities/find
grep search a file for a pattern POSIX.1-2008/Utilities/grep
xargs construct argument lists and invoke utility POSIX.1-2008/Utilities/xargs
wc word, line, and byte or character count POSIX.1-2008/Utilities/wc
tee reads standard input and writes it to both standard output and one or more files, effectively duplicating its input. tee
curl command line tool and library for transferring data with URLs Manual
man page
wget non-interactive download of files from the Web GNU Wget Manual
scp secure remote file copy program



Category Option Description Remarks
Input -E Interpret PATTERN as an extended regular expression.
-v Invert the sense of matching, to select non-matching lines.
Output -m NUM Stop reading a file after NUM matching lines.





Option Description Remarks
-X --request
-s --slient
-S --show-error
--connect-timeout SECONDS Maximum time allowed for connection
-m --max-time
  • Timeouts : --max-time(-m), --connect-timeout




  • [addr] : optional line address - single line number, a range of lines, a regular expression
  • X : single-letter command - a, c, d, D, i, p, s
Command Description Remarks
a text Append text after a line
c text Replace (change) lines with text
i text Insert text before a line
p Print the pattern space
P Print the pattern space, up to the first new line
n next
{ cmd ; cmd ... } Group several commands together.
Command Syntax Sample Remark
Substitution 's/pattern/replacement/option' sed 's/\r$//g' > README.linux.txt
Append Lines 'address a text-to-append' sed -i '2 a export GOROOT=/usr/lib/go-1.8\nPATH=$PATH:$GOROOT/bin\n' .profile
sed -i '$ a \\nexport GOROOT=/usr/lib/go-1.8\nexport PATH=$PATH:$GOROOT/bin\n' ~/.bashrc
'$' as an address means the last line.
'\n' at the beginning of replacement string need additional leading backslash to form '\\n'.
Change Lines 'address c text-to-change'
Delete Lines 'address d'
CLI Options
Option Description Remarks
-e script, --expression=script add the script to the commands to be executed
-i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if SUFFIX supplied)
-E, -r, --regexp-extended use ERE in the script
-n, --quiet, --silent suppress automatic printing of pattern space
Character escape

The only difference between basic and extended regular expressions is in the behavior of a few characters: ‘?’, ‘+’, parentheses, braces (‘{}’), and ‘|’. While basic regular expressions require these to be escaped if you want them to behave as special characters, when using extended regular expressions you must escape them if you want them to match a literal character.

Notation Special Character Literal
BRE \?, \+, \(, \), \{, \}, \| ?, +, (, ), {, }, |
ERE ?, +, (, ), {, }, | \?, \+, \(, \), \{, \}, \|
$ # Simple substitution
$ timedatectl status | grep 'Network time on' | sed  's/.*\(yes\|no\).*/\1/'
$ # Using shell parameter or variable
$ sed -r -i 's/^\s*(vm\.'"${param}"'.*)/#\1/g' /etc/sysctl.conf 
$ # Multiple edit in a single line
$ echo $output | sed -e 's/.*\[validator_token] \(.*\)/\1/; s/\s//g'
$ # Adding a line 
$ sed -i '2 a export GOROOT=/usr/lib/go-1.8' .profile
$ # Using # instead of / for substitution command
$ sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
$ # Using multiple command in a line using { }
$ cat rippled.cfg | sed -n '/^\[validator_token\]/{n;p}'
$ # Print out the sentence like file names such as song title more human readable
$ lowered='A|An|The|In|On|Under|Of|To|For|Like'
$ ls | sed -E 's/(.*)/\L\1/; s/(\b\w)/\U\1/g; s/\b('$lowered')\b/\L\1/g; s/^(.)/\U\1/'


  • Desc. : search files for lines (or other units of text) that contain certain patterns.
  • Syntax :
awk [options] [--] 'pattern { action } pattern { action } ...' file
Variable Type Description Remarks
NR Auto-set The number of input records awk has processed since the beginning of the program’s execution
NF Auto-set The number of fields in the current input record.
$ # Just filtering without any action
$ ls -l | awk '$3 == "root"' 
$ # Filtering using ~ operator and regex
$ netstat -antp | awk '$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)/'
$ # Escape $ when using double quotation
$ watch -n 2 'netstat -antp | awk "\$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)$/"| sort -k 4'
$ # Just action without filtering pattern
$ awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do





Option Description Remarks
-s --slurp Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once
-R --raw-input Don’t parse the input as JSON. Instead, each line of text is passed to the filter as a string. jq -R 'fromjson? | ... '
-r --raw-output If the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes.
Filter Syntax Description Remarks
Identity .
Object Identifier .key
Optional Object Identifier .key?
Generic Object Index .["key"]
Array Index .[n]
Array/String Slice .[n:m]
Array/Object Value Iterator .[], .[]?
Comma , If two filters are separated by a comma, then the same input will be fed into both and the two filters’ output value streams will be concatenated in order
Piple | combines two filters by feeding the output(s) of the one on the left into the input of the one on the right .a | .b | .c == .a.b.c
.[] | .foo == .[].foo (?)
Expression Syntax Description Remarks
Array Constructor [] used to construct arrays [.foo, .bar, .baz], [.items[].name]
Object Constructor {}
Recursive Descent ..
rippled=`echo ${rippled}' '$peers_connected | jq -s '.[0] + { peers_found : [.[1].result.peers[].public_key] }'`

echo $rippleds | jq '.['$i'] | { name: .name, type: .type, peers_defined: .peers }'
Operator Syntax Description Remarks
Addition Operator + takes two filters, applies them both to the same input, and adds the results together
Variable / Symbolic Binding Operator ... as $identifier |
Alternative Operator // defaults
Function Syntax Description Remarks
Has Function has(key)
In Function in(object), in(array) sorts its input, which must be an array
Any Filter any, any(condition)
All Filter all, all(condition)
Map Function map(filter), map(function) run that filter for each element of the input array, and return the outputs in a new array
Select Function select(predicate) predicate = boolean expression
Sort Function sort, sort_by(path_expression)
Flatten Filter flatten, flatten(depth)
echo $rippleds3 | jq 'sort_by(if .type == "validator" then 1 else 2 end, .name)'
Navigating JSON node using dot operator
for (( i = ${start}; i <= ${end}; i++ )); do
  resp=`curl -ksS --header "Content-Type: application/json" \
    --header "Connection: keep-alive" \
    --data "{\"method\": \"ledger\", \"params\": [ { \"ledger_index\": ${i}, \"transactions\": true } ]}" \
    http://${addr}:${port} | jq '.'`

  status=`echo ${resp} | jq '.result.status'`
  close_time=`echo ${resp} | jq '.result.ledger.close_time_human'`
  txs=`echo ${resp} | jq '.result.ledger.transactions'`
  txs_num=`echo ${txs} | jq '. | length'`
Constructing new JSON object
$ curl -sSX POST \
>   http://tracker1/ \
>   -H 'Connection: keep-alive' \
>   -H 'Content-Type: application/json' \
>   -d '{
>     "method": "server_info",
>     "params": [ {} ]
> }' | jq ' | {complete_ledgers: .complete_ledgers, hostid: .hostid, time: .time}'
  "complete_ledgers": "1247544-1440506,1440513-1460692,1460699-1468605,1468609-1479330,1479335-1567817",
  "hostid": "tracker1",
  "time": "2018-Oct-15 06:50:04.525106"
Merging distinct JSON objects using + operator
info=`curl -ksS --data '{"method":"server_info", "params": [{}]}' http://$addr:$port \
  | jq ' | {build_ver: .build_version, complete_ledgers: .complete_ledgers, io_latency_ms: .io_latency_ms, peers: .peers, peer_disconnects: .peer_disconnects, server_state: .server_state, uptime: .uptime, validated_ledger_index: .validated_ledger.seq}'`

logging=`curl -ksS --data '{"method":"log_level", "params": [{}]}' http://$addr:$port \
  | jq '{base_log_level: .result.levels.base}'`

validators=`curl -ksS --data '{"method":"validators", "params": [{}]}' http://$addr:$port \
  | jq '{trusted_validators: .result.trusted_validator_keys | length, validation_quorum: .result.validation_quorum}'`

echo "{\"info\": $info, \"logging\": $logging, \"validators\": $validators}" | jq '.info + .logging + .validators'
Merging distinct JSON string using -s option and + operator
rippled=`echo $rippled' '$peers_connected | jq -s '.[0] + { peers_found : [.[1].result.peers[].public_key] }'`
Using select function
#! /bin/bash

declare script_dir=`dirname -- "$0"`
script_dir=$(cd "$script_dir" && pwd)
declare -r config="$script_dir/../generated/ripple-network-config.json"

# TODO Sort by name
declare -r rippleds=`cat "$config" | jq -r '.rippleds'`
declare -r rippleds_cnt=`echo $rippleds | jq 'length'`
declare -r protocol=https

declare rippled
declare name
declare peers_defined
declare proxy
declare addr
declare port
declare peers_connected
declare validators
for (( i = 0; i < $rippleds_cnt; i++ )); do
  rippled=`echo $rippleds | jq '.['$i']'`

  name=`echo $rippled | jq -r '.name'`
  echo $name
  peers_defined=`echo $rippled | jq -r '.peers'`
  echo $peers_defined

  proxy=`echo $rippled | jq -r '.proxy.frontends[] | select((.backend == "rippled-rpc") and (.protocol == "'$protocol'"))'`
  echo $proxy
  addr=`echo $proxy | jq -r '.address'`
  echo $addr
  port=`echo $proxy | jq '.port'`
  echo $port

  peers_connected=`curl -ksS -X POST \
  --cacert "$script_dir/../files/tls/test-ca.crt" \
  --cert "$script_dir/../files/tls/test-tls-client.crt" \
  --key "$script_dir/../files/tls/test-tls-client.key" \
  --cert-type PEM \
  $protocol'://'$addr':'$port'/' \
  -H 'Connection: close' \
  -H 'Content-Type: application/json' \
  -d '{"method": "peers", "params": [{}]}'`
  echo $peers_connected

  validators=`curl -ksS -X POST \
  --cacert "$script_dir/../files/tls/test-ca.crt" \
  --cert "$script_dir/../files/tls/test-tls-client.crt" \
  --key "$script_dir/../files/tls/test-tls-client.key" \
  --cert-type PEM \
  $protocol'://'$addr':'$port'/' \
  -H 'Connection: close' \
  -H 'Content-Type: application/json' \
  -d '{"method": "validators", "params": [{}]}'`
  echo $validators




Option Description Remarks
-n The maximum number of open file descriptors
-p The pipe buffer size
-s The maximum stack size
-u The maximum number of processes available to a single user
-v The maximum amount of virtual memory available to the process


Mount Options
Option Description Filesystems Remarks
atime The inode access time is controlled by kernel defaults.
noatime Do not update inode access times on this filesystem.
relatime Update inode access times relative to modify or change time.
strictatime Allows to explicitly requesting full atime updates.
ro Mount the filesystem read-only.
rw Mount the filesystem read-write.
default Use default options: rw, suid, dev, exec, auto, nouser, async, and relatime.




Package Management


utility command description remarks
apt-get APT package handling utility -- command-line interface
update resynchronize the package index files from their sources
upgrade install the newest versions of all packages currently installed on the system from the sources enumerated in /etc/apt/sources.list.
remove packages are removed leaving its configuration files on the system.
purge packages are removed and purged (any configuration files are deleted too).
clean clears out the local repository of retrieved package files.
autoclean clears out the local repository of retrieved package files that can no longer be downloaded, and are largely useless.
autoremove remove packages that were automatically installed to satisfy dependencies for other packages and are now no longer needed.
apt-file searching files in packages for the APT package management system
apt provides a high-level commandline interface for the package management system. cann't be used inside script file


Action Syntax Description Remarks
Install dpkg -i package_file... --install
Remove dpkg -r package... remove everything except conffiles --remove
Purge dpkg -P pakcage... remove everything, including conffiles --purge
Status Description Remarks
rc the package has been removed, but that the configuration files remain
  • 1st character : desired state
Character Status Description
u Unknown an unknown state
i Install marked for installation
r Remove marked for removal
p Purge marked for purging
h Hold
  • 2nd character : current state
Character Status Description
n not-installed The package is not installed on your system.
i installed The package is unpacked and configured OK.
c config-files Only the configuration files of the package exist on the system.
u unpacked The package is unpacked, but not configured.
h half-installed The installation of the package has been started, but not completed for some reason.
W triggers-awaited The package awaits trigger processing by another package.
t triggers-pending The package has been triggered.

Monitoring & Diagnosis

Command/Tool Description Remarks
sysctl examining and changing kernel parameters at runtime
lsof displays information about files open to Unix processes
iostat collect and show operating system storage input and output statistics
ss show socket statistics
rsyslog the rocket-fast system for log processing


Category Option Description Remarks
Simple Process Selection a Lift the BSD-style "only yourself" restriction, which is imposed upon the set of all processes when some BSD-style (without "-") options are used or when the ps personality setting is BSD-like. The set of processes selected in this manner is in addition to the set of processes selected by other means. An alternate description is that this option causes ps to list all processes with a terminal (tty), or to list all processes when used together with the x option.
x Lift the BSD-style "must have a tty" restriction, which is imposed upon the set of all processes when some BSD-style (without "-") options are used or when the ps personality setting is BSD-like. The set of processes selected in this manner is in addition to the set of processes selected by other means. An alternate description is that this option causes ps to list all processes owned by you (same EUID as ps), or to list all processes when used together with the a option.
Output Format Control u Display user-oriented format.
Output Modifires c Show the true command name. This is derived from the name of the executable file, rather than from the argv value. Command arguments and any modifications to them are thus not shown. This option effectively turns the args format keyword into the comm format keyword; it is useful with the -f format option and with the various BSD-style format options, which all normally display the command arguments. See the -f option, the format keyword args, and the format keyword comm.
o Specify user-defined format Identical to -o and --format
f ASCII art process hierarchy (forest)
--sort specify sorting order. Sorting syntax is [+|-]key[,[+|-]key[,...]]
Format Specifiers
  • Used with o or --sort options
Code Header Description Aliases Remarks
pid PID a number representing the process ID tgid(TGID)
ppid PPID a number representing the process ID tgid(TGID)
pgid PGID process group ID or, equivalently, the process ID of the process group leader pgrp(PGRP)
sid SID session ID or, equivalently, the process ID of the session leader sess(SESS)
tid TID light weight process (thread) ID of the dispatchable entity spid(SPID), lwp(LWP)
comm COMMAND command name (only the executable name) ucmd(CMD), ucomm(COMMAND)
command COMMAND command with all its arguments as a string args(COMMAND)
uid UID effective user ID euid(EUID)
user USER effective user name (textual user ID) euser(EUSER)
gid GID effective group ID number of the process as a decimal integer egid(EGID)
group GROUP effective group name of the process (textual group ID) egroup(EGROUP)
ruid RUID real user ID
ruser RUSER real user name (textual user ID)
rgid RGID real group ID
rgroup RGROUP real group name (textual group ID)
stat STAT multi-character process state
tname TTY Controlling terminal tt, tty
start STARTED time the command started
lstart STARTED time the command started long format ?
etime ELAPSED elapsed time since the process was started, in the form [[DD-]hh:]mm:ss
etimes ELAPSED elapsed time since the process was started, in seconds
time TIME cumulative CPU time, "[DD-]HH:MM:SS" format cputime
times TIME cumulative CPU time in seconds cputimes
nlwp NLWP number of lwps (threads) in the process thcount
rss RSS the non-swapped physical memory that a task has used (in KB) rsz Resident Set Size
vsz VSZ virtual memory size of the process in KiB vsize
Process State Codes
Category Code Meaning Remarks
Main R running or runnable (on run queue)
S interruptible sleep (waiting for an event to complete)
X dead (should never be seen)
Z defunct ("zombie") process, terminated but not reaped by its parent
Additional < high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group
  • What are “session leaders” in `ps`? (Aug 6 '11)
    • Sessions and process groups are just ways to treat a number of related processes as a unit. All the members of a process group always belong to the same session, but a session may have multiple process groups
$ ps axuww      #full command line
$ ps axuc       #true command name only, used before calling 'killall -r'
$ ps axucf      #true command name in hierarchy
$ ps axufww     #full command line in hirerachy
$ ps axuww -T   #including thread
$ ps axufww | grep -v grep | grep -E '^USER|haproxy' 
$ ps axu --sort=-time | head -n 5
$ ps axww o pid,user,group,command | grep -v grep | grep -E '^PID|rippled'
$ ps axf o pid,ppid,pgid,sid,commad



Predefined file descriptor
Name Description Remarks
cwd current working directory
rtd root directory
txt program text (code and data)
mem memory-mapped file







Option Description Remark
-a, --all Show both listening and non-listening (for TCP this means established connections) sockets.
-n, --numeric Show numerical addresses instead of trying to determine symbolic host, port or user names.
-t, --tcp Display only TCP connections. Linux
-p, --program Show the PID and name of the program to which each socket belongs.
-o, --timers Include information related to networking timers.
-e, --extend Show the PID and name of the program to which each socket belongs.
$ netstat -antpe






  • Severity Level
Severity Keyword Value Description Remarks
Emergency emerg 0 System is unusable
Alert alert 1 Action must be taken immediately
Critical crit 2 Critical conditions
Error err 3 Error conditions
Warning warning 4 Warning conditions
Notice notice 5 Normal but significant conditions
Informational info 6 Informational messages
Debug debug 7 Debug-level messages
Category Properties Description Remarks
Message programname the “static” part of the tag, as defined by BSD syslogd
Configuration Objects
Object Description Remarks
global() used to set global configuration parameters
module() used to load plugins
input() the primary means of describing inputs
action() the primary means of describing actions to be carried out
timezone() used to define timezone settings
Category Module Title Description Parameters Remarks
Input imuxsock Unix Socket Input Module accept syslog messages from applications running on the local system via Unix sockets Socket
imklog Kernel Log Input Module reads messages from the kernel log and submits them to the syslog engine
imtcp TCP Syslog Input Module receives syslog messages via TCP Port
imudp UDP Syslog Input Module receives syslog messages via UDP Port
Reserved Templates
Template Format Description Remarks
RSYSLOG_TraditionalFileFormat "%timegenerated% %HOSTNAME% %syslogtag%%msg%\\n" the old style default log file format
RSYSLOG_FileFormat "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n" a modern-style logfile format



  • ip man page
  • show / manipulate routing, devices, policy routing and tunnels




PAC Manager



  • Desc. : stores, retrieves, generates, and synchronizes passwords securely


TCP Wrapper


Shell Script Test






Tips and Tricks

Diagnosing and Monitoring Linux

Task Commandline Remarks
Identifying the product of Linux installed $ cat /etc/issue
$ cat /etc/*-release
Identifying kernel version $ cat /proc/version
Listing or identifying kernel parameters $ sysctl -a | more
$ sysctl net.ipv4 | more
$ sysctl net.ipv4.tcp_max_syn_backlog
Identifying CPU capacity $ cat /proc/cpuinfo
Identifying memory capacity and usage $ cat /proc/meminfo
Listing processes $ ps auxfww
Identifying threads of a specific process $ ps -T -p 31
Listing disks $ lsblk
Listing filesystems $ df -ahT
$ mount -l
Identifying details of a certain filesystem $ dumpe2fs -h /dev/xvda2
Identifying the filesystem a certain file belongs to $ df -h /var
Checking disk caching $ hdparm -W /dev/sda
Identifying TCP/IP ports in use $ netstat -anotup
Identifying sockets summary $ ss -s
Identifying the user limits of current login or session $ ulimit -a
Identifying the max. number of file handles for the entire system $ cat /proc/sys/fs/file-max
Identifying file handle usage $ cat /proc/sys/fs/file-nr
Counting the number of currently open files to a specific process $ lsof -a -p pid -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l

Identifying the product of Linux installed

For Linux, /etc/issues file contains more detailed information on what Linux product it is.

$ cat /etc/issue
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Kernel \r on an \m

For most of Linux distributions, /etc/*-release file is provided which contains more detailed and systematic information. The file would be lsb-release, os-release, redhat-release or so on as to the Linux distributions.

$ cat /etc/*-release

Identifying kernel version

$ cat /proc/version
Linux version 4.4.0-83-generic (buildd@lgw01-29) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017

Identifying kernel parameters

To print out all kernel parameters

$ sysctl -a | more

To print out a group of kernel parameters

$ sysctl net.ipv4 | more

To print out a specific kernel parameter

$ sysctl net.ipv4.tcp_syncookies
$ sysctl net.ipv4.conf.all.rp_filter
$ sysctl net.ipv4.tcp_max_syn_backlog

Identifying boot-up time

$ uptime -s


$ who -b

Identifying CPU capacity

$ cat /proc/cpuinfo

Identifying memory capacity and usage

$ cat /proc/meminfo
MemTotal:        3922904 kB
MemFree:         3037280 kB

Identifying threads of a specific process

$ ps -T -p 31

Listing disks

$ lsblk

Listing filesystems

To list filesystems with capacity and usage, use df

$ df -ahT

To list filesystems with mount options, use mount command

$ mount -l

Identifying the details of a certain filesystem

$ dumpe2fs -h /dev/xvda2

Identifying the filesystem a certain file belongs to

$ df -h /var   #asking what filesystem contains /var directory

Checking disk caching

$ hdparm -W /dev/sda

Identifying disk detail information

$ hdparm -i /dev/sda

Benchmarking disk performance

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=append,noatime

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=fdatasync,notrunc oflag=append,noatime

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=dsync,append,noatime

for i in {1..3}; do
  rm -f /var/tmp/diskperftest.txt && time dd bs=100K count=5000 if=/dev/zero of=/var/tmp/diskperftest.txt conv=notrunc oflag=direct,append,noatime
Each CONV symbol may be:

  ascii     from EBCDIC to ASCII
  ebcdic    from ASCII to EBCDIC
  ibm       from ASCII to alternate EBCDIC
  block     pad newline-terminated records with spaces to cbs-size
  unblock   replace trailing spaces in cbs-size records with newline
  lcase     change upper case to lower case
  ucase     change lower case to upper case
  sparse    try to seek rather than write the output for NUL input blocks
  swab      swap every pair of input bytes
  sync      pad every input block with NULs to ibs-size; when used
            with block or unblock, pad with spaces rather than NULs
  excl      fail if the output file already exists
  nocreat   do not create the output file
  notrunc   do not truncate the output file
  noerror   continue after read errors
  fdatasync  physically write output file data before finishing
  fsync     likewise, but also write metadata

Each FLAG symbol may be:

  append    append mode (makes sense only for output; conv=notrunc suggested)
  direct    use direct I/O for data
  directory  fail unless a directory
  dsync     use synchronized I/O for data
  sync      likewise, but also for metadata
  fullblock  accumulate full blocks of input (iflag only)
  nonblock  use non-blocking I/O
  noatime   do not update access time
  nocache   Request to drop cache.  See also oflag=sync
  noctty    do not assign controlling terminal from file
  nofollow  do not follow symlinks
  count_bytes  treat 'count=N' as a byte count (iflag only)
  skip_bytes  treat 'skip=N' as a byte count (iflag only)
  seek_bytes  treat 'seek=N' as a byte count (oflag only)

Identifying TCP/UDP ports currently in use

You can identify TCP/IP ports currently in use using netstat command. The options of netstat is slightly different among operating systems.


$ netstat -anotu

For Linux,

$ sudo netstat -anotup   #-a: all sockets, -n: numeric, -t: TCP, -u: UDP, -p: PID/program, -o: timers

You need root privilege to take effect of -p option To find out whether a given port is being used or not, use grep command.

$ sudo netstat -anotup | grep -E '(^Proto)|(8080)'

For Windows,

$ netstat -ano

For more about netstat, refer topics in Wikipedia.

Monitoring network traffic by interface card

watch -n 2 netstat -i

Capturing incoming HTTP request using tcpdump

$ sudo tcpdump -s 0 -A -i eth1 dst port 80

Identifying the user limits of current login or session

$ ulimit -a

Counting the number of currently open files to a specific process

$ lsof -a -p 12345 -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l


$ ls -l /proc/12345/fd | wc -l


Diagnosing and Monitoring AIX

Identifying System Configuration

Using Common Shell Commands

Identifying the shell of your current login

To identify what shell a user is set to use by default, you can check SHELL variable.

$ echo $SHELL
$ bin/ksh
$ echo $SHELL

As the above example shows, SHELL variable contains the login default shell type not the one currently in use.

Hiding the output of command

To hide both the normal output and error output, redirect stdout and stderr to null device

% npm ls -g json >/dev/null 2>&1
% #or
% npm ls -g y18n &>/dev/null

To hide only the error output, redirect stderr to null device

% npm ls -g json 2>/dev/null

Listing files using find command excluding files with 'Permission denied'

When executing find command in simplest format, you may get lots of lines just saying that 'Permission denied'. Most cases, those are not what you want, and lots of permission denied lines can disturb you identifying the wanted result.

You can use stderr redirection to cut out permission denied files (or directories).

% find / -name '*.jar' 2>/dev/null

Finding files having specified name with full path

If you want to find files with extension of 'jar' and print them with full path, use find command with -exec operator like the following.

% find . -name '*.jar' -exec ls -l {} \;

For more about find command and -exec operator including strange '{}' or '\;' in the above example, refer the followings.

Finding files containing the specified word

% find /home |xargs grep "password"

For more about xargs, refer the followings.

Finding old directories or files to remove them

To find old files or directories and then work with them, use find command with -amin, -atime, -cmin, -ctime, -mmin or -mtime and -exec options.

% find . -maxdepth 1 -type d -ctime +10 -exec rm -Rf {} \;

Finding large files

To find large files(not directories) under current directory and list them in pages, use the following command.

% find . -type f -exec du -k {} 2>/dev/null \; | sort -nr | more

To filter out small files, you can use size option with find command, or to filter out some subdirectories you can redirect the result to grep command. The following command will list files whose size are more than 1 mega-byte under current directory recursively except the subdirectories starting with 'svn' in order of their size.

% find . -type f -size +1000000c -exec du -k {} 2>/dev/null \; | sort -nr | grep -E "\./svn.*" -v

Listing all files under a certain directory recursively

Using find command

% find /proc/sys -type f 2>/dev/null | more

Listing distinct file extensions of all files under a directory

% find . -type f -name "*.*" | sed -r 's/^.+(\.\w+)$/\1/' | sort | uniq

Counting files under a directory recurssively

% find . -type f -print | wc -l

Counting files in a tar file

% tar -tvf archive.tar | grep "^-.*" | wc -l

Inverse matching with grep command

To find lines not matching the specified patterns in a file, you can use -v option with grep command.

$ svn list -R http://.../repos1 | grep -v -E '(.*java|.*/)'

You don't need to be bothered to find out how to use complex negative patterns with regex.

Viewing files in octal or hexadecimal format - od

You can view non ascii base files in hexadecimal format using od command.

% od -A d -x journal.log

For more about od, refer the following.

Viewing file contents without line wrapping - less -S

% less -S known_hosts

Viewing the result of ps command without line wrapping

You can redirect the result to cat or less command, or use ww flag.

% ps auxf | cat
% ps auxfww
% ps auxf | less -+S

Viewing file contents without comments lines (starting with #)

% cat /etc/apt/sources.list | grep -P '^[^#].*'

Sorting the file system usage result from the du command

You can sort the output of du command applying pipe to sort command.

% du -m | sort -n

For more about du and sort, read the followings.

Getting multiple files form the target URL using wget command

wget provide --accept or -A switch which can represent multiple files using comma separated list, wild card, or character class. But it's not that -A switch support regular expression.

$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '198[7-9].csv.bz2'"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '199[0-9].csv.bz2'"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A '200[0-8].csv.bz2'"
$ su - hdfs -c "(cd ~; wget -x -P samples/flight/rawdata -A 'airports.csv, carriers.csv, plane-data.csv'"

For more, refer the following

Adding lines to a specific row of a file using sed command

$ sed -i '2 a export GOROOT=/usr/lib/go-1.8' .profile
$ sed -i '3 a export PATH=$PATH:$GOROOT/bin\n' .profile

Repeat command over piped targets using xargs

$ dpkg -l | grep "^rc" | awk '{print $2}' | xargs sudo dpkg --purge

Grep from a specific column

$ ls -l | awk '$3 == "root"'

$ netstat -antp | awk '$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)/'

$ watch -n 2 'netstat -antp | awk "\$4 ~ /.*:(1580|1590|1600|1943|1953|1963|2080|2090|2100|2443|2453|2463|5505|5515|6506|6516)$/"| sort -k 4'

Bash Programming

Directory containing the current script

The following script would work correctly with a directory containing spaces in its path.

script_dir=`dirname -- "$0"`
script_dir=$(cd "$script_dir" && pwd)

When the directory may contain spaces, $script_dir and its derivatives should always be double quoted like the following code

script_dir=`dirname -- "$0"`
script_dir=$(cd "$script_dir" && pwd)
readonly config="$script_dir/../generated/ripple-network-config.json"

cat "$config"

Looping array

For indexed array

fruits=(Apple Banana Kiwi)

for f in ${fruits[@]}; do
  echo $f

# or looping in index order

for (( i = 0; i < ${#fruits[@]}; i++ )); do
  echo ${fruits[${i}]}

For correlated arrays,

fruits=(Apple Banana Kiwi)
colors=(red yellow green)

for (( i=0; i<${#fruits[@]}; i++ )); do
  echo ${fruits[$i]} is ${colors[$i]}

For associate array,

declare -Ar vm_params=( # kernel parameters for virtual memory
  [min_free_kbytes]=262144 #256MB

for param in "${!vm_params[@]}"; do
  echo "Updating 'vm.${param}' to '${vm_params[${param}]}'."
  sudo sysctl -wq vm.${param}=${vm_params[${param}]}

  if [ `grep -E "^\\s*vm.${param}\\s*=\\s*${vm_params[${param}]}.*" /etc/sysctl.conf | wc -l` -ne 1 ]; then
    sudo sed -r -i 's/^\s*(vm\.'"${param}"'.*)/#\1/g' /etc/sysctl.conf
    sudo sh -c "echo 'vm.${param}=${vm_params[${param}]}' >> /etc/sysctl.conf"

Looping arguments

Looping arguments by item,

declare command="curl -sSL ${url_base}/ | bash -s --"

for arg in "$@"; do
  if [ ${arg:0:1} == '-' ]; then
    command=${command}" ${arg}"
    command=${command}" '${arg}'"

Looping arguments by index,

declare -A args
declare arg
for (( i = 1; i <= $#; i++ )); do
  if [ ! ${arg:0:1} == '-' ]; then arg="'${arg}'"; fi
# echo ${args[@]}

2 dimentional array

declare -a m
m[0]='a b c d'
m[1]='e f g h'
m[2]='i j k l'
m[3]='m n o p'
m[4]='q r s t'

for r in "${m[@]}"; do
  echo $r
  for c in $r; do
    echo $c

Read a file line by line

cat ./${infile} | while -r read line; do
  IFS=','; read -r -a fields <<< "${line}"
  # ...

awk script typical snippet

Use awk to structure target data table from the input source and process each row using while statement assigning each column to its own variable using read.
So typical script flow is awk ... | while read var1 var2 var3; do ...; done like the following real example.

awk -F, '{if (($1 ~ /vm[0-9]*/) && (substr($1, 3, length($1) - 1) + 0 < 97)) print $1 " " $2}' ../../vms.csv | while read vm ip; do

  no=${vm##vm}  # remove left 'vm'
  org_no=$(($(($((no - 1))/org_size)) + 1))
  peer_str="{\"requests\": \"grpcs://${ip}:7051\", \"events\": \"grpcs://${ip}:7053\", \"server-hostname\": \"peer${no}\", \"tls_cacerts\": \"tlsCerts/org${org_no}.com/tlsca${org_no}-cert.pem\"}"

  # Defines paired peer no : (vm1-vm2, vm3-vm4, ...)
  if [ $((no % 2)) -eq 1 ]; then
    no2=$((no + 1))
    no2=$((no - 1))

  # For current peer vm
  sed -i -r 's#@@peer1@@#\"peer1\": '"$peer_str"',#' ./generated/vm${no}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer1' part of './generated/vm${no}/channelConfig.json' file."
    exit 1
    echo "Successfully updated 'peer1' part of './generated/vm${no}/channelConfig.json' file."

  # For paired peer vm
  sed -i -r 's#@@peer2@@#\"peer2\": '"$peer_str"'#' ./generated/vm${no2}/channelConfig.json
  if [ $? -ne 0 ]; then
    echo "Fail to update 'peer2' part of './generated/vm${no2}/channelConfig.json' file."
    exit 1
    echo "Successfully updated 'peer2' part of './generated/vm${no2}/channelConfig.json' file."

parallel script typical snippet

For local tasks

dirs=('/boot' '/proc' '/sys' '/var' '/sbin');

for dir in ${dirs[@]}; do echo $dir; done | parallel --no-notice --bar --joblog /dev/stdout '
  files=$(sudo find {} -type f 2>/dev/null | wc -l);
  echo {} ":" ${files} "files"

For remote tasks

hosts=('' '' '');

for host in ${hosts[@]}; do echo $host; done | parallel --no-notice --bar --joblog /dev/stdout '
  name=$(ssh -o StrictHostKeyChecking=no paul@{} echo "\`hostname\`");
  echo {} ":" ${name};

Managing Packages


Installing a new software package
  1. Update the package information
  2. Check whether or not the software package is already installed
  3. Search and review the software package to install
  4. Install or upgrade the software package
  5. (Optionally) Confirm all the files installed by the package

Not using apt, which is preferred 'cause apt doesn't support in-script usage well.

$ sudo apt-get update                                 # update package information
$ sudo dpkg -l | awk '{print $2}' | grep golang-1.8   # check previous installation
$ sudo apt-cache --names-only search ^golang | more   # search available packages
$ sudo apt-cache show golang-1.8                      # review the software to install
$ sudo apt-get install golang-1.8                     # install the software
$ sudo apt-file list golang-1.8                       # confirm all the files installed

Using apt

$ sudo apt update                          # update package information
$ sudo apt list *golang-1.8* --installed   # check previous installation
$ sudo apt search ^golang | more           # search available packages
$ sudo apt show golang-1.8                 # review the software to install
$ sudo apt install golang-1.8              # install the software
$ sudo apt-file list golang-1.8            # confirm all the files installed
  • References
    • apt-get
    • apt-cache
    • apt-file : searching files in packages for the APT package management system.
    • apt : provides a high-level commandline interface for the package management system.
Installing a software package specifying version with wildcard
$ sudo apt-get install nodejs=6.10.2*
Listing all installed packages
$ sudo dpkg -l | awk '{print $2}'


$ sudo apt list --installed


$ sudo dpkg -l | grep -E '\stelegraf'
$ sudo dpkg -i telegraf_1.8.2-1_amd64.deb


Preceed configuration for Ubuntu 16.04 automatic installation

### References

### Localization
# Language, Country and Locales
d-i debian-installer/language string en
d-i debian-installer/country string KR
d-i debian-installer/locale string en_US.UTF-8
d-i localechooser/supported-locales multiselect en_US.UTF-8

# Keyboard
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us

### Network
d-i netcfg/enable boolean false
d-i netcfg/disable_autoconfig boolean true
d-i netcfg/get_ipaddress string
d-i netcfg/get_netmask string
d-i netcfg/get_gateway string
d-i netcfg/get_nameservers string
d-i netcfg/confirm_static boolean true
d-i netcfg/hostname string chainz001

### Clock and Time-zone
d-i clock-setup/utc boolean true
d-i time/zone string Asia/Seoul
d-i clock-setup/ntp boolean false

### Account
d-i passwd/root-login boolean false
d-i passwd/make-user boolean tue

d-i passwd/user-fullname string Blockchain User
d-i passwd/username string chainz
d-i passwd/user-password password skccz
d-i passwd/user-password-crypted password [crypt(3) hash]
d-i passwd/user-uid string 1010
d-i passwd/user-default-groups string audio cdrom video
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false

### Partitioning
d-i partman-auto/disk string /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
d-i partman-auto/method string lvm

d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true

# d-i partman-auto-lvm/guided_size string max

d-i partman-auto/expert_recipe string \
  512 512 512 ext2 \
    $primary{ } \
    $bootable{ } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext2 } \
    label{ boot } \
    mountpoint{ /boot } \
  . \
  100 1000 -1 ext3 \
    $defaultignore{ } \
    $primary{ } \
    method{ lvm } \
    device{ /dev/sda } \
    vg_name{ vg1 } \
  . \
  96 512 300% linux-swap \
    $lvmok{ } \
    in_vg{ vg1 } \
    lv_name{ swap } \
    method{ swap } \
    format { } \
  . \
  10240 10240 10240 ext4 \
    $lvmok{ } in_vg{ vg1 } \
    lv_name{ root } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
    label{ root } \
    mountpoint{ / } \
  . \
  10240 10240 10240 ext4 \
    $lvmok{ } in_vg{ vg1 } \
    lv_name{ var } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
    label{ root } \
    mountpoint{ / } \
  . \
  10240 10240 10240 ext4 \
    $lvmok{ } in_vg{ vg1 } \
    lv_name{ tmp } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
    label{ tmp } \
    mountpoint{ /tmp } \
  . \
  10240 10240 10240 ext4 \
    $lvmok{ } in_vg{ vg1 } \
    lv_name{ home } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext4 } \
    label{ home } \
    mountpoint{ /home } \