Bash-Oneliner

I am glad that you are here! I was working on bioinformatics a few years ago and was amazed by those single-word bash commands which are much faster than my dull scripts, time saved through learning command-line shortcuts and scripting. Recent years I am working on cloud computing and I keep recording those useful commands here. Not all of them is oneliner, but i put effort on making them brief and swift. I am mainly using Ubuntu, Amazon Linux, RedHat, Linux Mint, Mac and CentOS, sorry if the commands don't work on your system.

This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear search engine and Stack Overflow.

English and bash are not my first language, please correct me anytime, thank you. If you know other cool commands, please teach me!

Here's a more stylish version of Bash-Oneliner~

Handy Bash one-liners

• Terminal Tricks
• Variable
• Math
• Grep
• Sed
• Awk
• Xargs
• Find
• Condition and Loop
• Time
• Download
• Random
• Xwindow
• System
• Hardware
• Networking
• Data Wrangling
• Others

Terminal Tricks

Using Ctrl keys

Ctrl + a : move to the beginning of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + e : move to the end of line.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + l : equivalent to clear.
Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
Ctrl + z : stop current running process and keep it in background. You can use `fg` to continue the process in the foreground, or `bg` to continue the process in the background.
Ctrl + _ : undo typing.

Change case

Esc + u
# converts text from cursor to the end of the word to uppercase.
Esc + l
# converts text from cursor to the end of the word to lowercase.
Esc + c
# converts letter under the cursor to uppercase, rest of the word to lowercase.

Run history number (e.g. 53)

!53

Run last command

!!
# run the previous command using sudo
sudo !!

Run last command and change some parameter using caret substitution (e.g. last command: echo 'aaa' -> rerun as: echo 'bbb')

# last command: echo 'aaa'
^aaa^bbb

#echo 'bbb'
#bbb

# Notice that only the first aaa will be replaced, if you want to replace all 'aaa', use ':&' to repeat it:
^aaa^bbb^:&
# or
!!:gs/aaa/bbb/

Run past command that began with (e.g. cat filename)

!cat
# or
!c
# run cat filename again

Bash globbing

# '*' serves as a "wild card" for filename expansion.
/etc/pa*wd    #/etc/passwd

# '?' serves as a single-character "wild card" for filename expansion.
/b?n/?at      #/bin/cat

# '[]' serves to match the character from a range.
ls -l [a-z]*   #list all files with alphabet in its filename.

# '{}' can be used to match filenames with more than one patterns
ls *.{sh,py}   #list all .sh and .py files

Some handy environment variables

$0   :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$#   :number of positional parameters.
$?   :most recent foreground pipeline exit status.
$-   :current options set for the shell.
$$   :pid of the current shell (not subshell).
$!   :is the PID of the most recent background command.
$_   :last argument of the previously executed command, or the path of the bash script.

$DESKTOP_SESSION     current display manager
$EDITOR   preferred text editor.
$LANG   current language.
$PATH   list of directories to search for executable files (i.e. ready-to-run programs)
$PWD    current directory
$SHELL  current shell
$USER   current username
$HOSTNAME   current hostname

Using vi-mode in your shell

set -o vi
# change bash shell to vi mode
# then hit the Esc key to change to vi edit mode (when `set -o vi` is set)
k
# in vi edit mode - previous command
j
# in vi edit mode - next command
0
# in vi edit mode - beginning of the command
R
# in vi edit mode - replace current characters of command
2w
# in vi edit mode - next to 2nd word
b
# in vi edit mode - previous word
i
# in vi edit mode - go to insert mode
v
# in vi edit mode - edit current command in vi
man 3 readline
# man page for complete readline mapping

Variable

[back to top]

Variable substitution within quotes

# foo=bar
echo $foo
# bar
echo "$foo"
# bar
# single quotes cause variables to not be expanded
echo '$foo'
# $foo
# single quotes within double quotes will not cancel expansion and will be part of the output
echo "'$foo'"
# 'bar'
# doubled single quotes act as if there are no quotes at all
echo ''$foo''
# bar

Get the length of variable

var="some string"
echo ${#var}
# 11

Get the first character of the variable

var=string
echo "${var:0:1}"
#s

# or
echo ${var%%"${var#?}"}

Remove the first or last string from variable

var="some string"
echo ${var:2}
#me string

Replacement (e.g. remove the first leading 0 )

var="0050"
echo ${var[@]#0}
#050

Replacement (e.g. replace 'a' with ',')

{var/a/,}

Replace all (e.g. replace all 'a' with ',')

{var//a/,}

Substitute environment variables in a file

export NAME="Alice"
envsubst < template.txt > output.txt
# template.txt:       Hello ${NAME}, your order is shipped.
# output.txt:         Hello Alice, your order is shipped.
# or
export USER="Bob" && echo 'Hi $USER, welcome!' | envsubst
# Hi Bob, welcome!

Grep lines with strings from a file (e.g. lines with 'stringA or 'stringB' or 'stringC')

# with grep
test="stringA stringB stringC"
grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep

To change the case of the string stored in the variable to lowercase (Parameter Expansion)

var=HelloWorld
echo ${var,,}
helloworld

Expand and then execute variable/argument

cmd="bar=foo"
eval "$cmd"
echo "$bar" # foo

Record a terminal session

# https://github.com/asciinema/asciinema
asciinema rec demo.cast

Math

[back to top]

Arithmetic Expansion in Bash (Operators: +, -, *, /, %, etc)

echo $(( 10 + 5 ))  #15
x=1
echo $(( x++ )) #1 , notice that it is still 1, since it's post-increment
echo $(( x++ )) #2
echo $(( ++x )) #4 , notice that it is not 3 since it's pre-increment
echo $(( x-- )) #4
echo $(( x-- )) #3
echo $(( --x )) #1
x=2
y=3
echo $(( x ** y )) #8

Print out the prime factors of a number (e.g. 50)

factor 50
# 50: 2 5 5

Sum up input list (e.g. seq 10)

seq 10|paste -sd+|bc

Sum up a file (each line in file contains only one number)

awk '{s+=$1} END {print s}' filename

Column subtraction

cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'

Simple math with expr

expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)

More math with bc

# Number of decimal digit/ significant figure
echo "scale=2;2/3" | bc

# .66

# Exponent operator
echo "10^2" | bc
# 100

# Using variables
echo "var=5;--var"| bc
# 4

Grep

[back to top]

Type of grep

grep = grep -G # Basic Regular Expression (BRE)
fgrep = grep -F # fixed text, ignoring meta-characters
egrep = grep -E # Extended Regular Expression (ERE)
rgrep = grep -r # recursive
grep -P # Perl Compatible Regular Expressions (PCRE)

Grep and count number of empty lines

grep -c "^$"

Grep and return only integer

grep -o '[0-9]*'
# or
grep -oP '\d*'

Grep integer with certain number of digits (e.g. 3)

grep '[0-9]\{3\}'

# or
grep -E '[0-9]{3}'

# or
grep -P '\d{3}'

Grep only IP address

grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'

Grep whole word (e.g. 'target')

grep -w 'target'

# or using RE
grep '\btarget\b'

Grep returning lines before and after match (e.g. 'bbo')

# return also 3 lines after match
grep -A 3 'bbo'

# return also 3 lines before match
grep -B 3 'bbo'

# return also 3 lines before and after match
grep -C 3 'bbo'

Grep string starting with (e.g. 'S')

grep -o 'S.*'

Extract text between words (e.g. w1,w2)

grep -o -P '(?<=w1).*(?=w2)'

Grep lines without word (e.g. 'bbo')

grep -v bbo filename

Grep lines not begin with string (e.g. #)

grep -v '^#' file.txt

Grep variables with space within it (e.g. myvar="some strings")

grep "$myvar" filename
# remember to quote the variable!

Grep only one/first match (e.g. 'bbo')

grep -m 1 bbo filename

Grep and return number of matching line(e.g. 'bbo')

grep -c bbo filename

Count occurrence (e.g. three times a line count three times)

grep -o bbo filename |wc -l

Case insensitive grep (e.g. 'bbo'/'BBO'/'Bbo')

grep -i "bbo" filename

COLOR the match (e.g. 'bbo')!

grep --color bbo filename

Grep search all files in a directory(e.g. 'bbo')

grep -R bbo /path/to/directory
# or
grep -r bbo /path/to/directory

Search all files in directory, do not ouput the filenames (e.g. 'bbo')

grep -rh bbo /path/to/directory

Search all files in directory, output ONLY the filenames with matches(e.g. 'bbo')

grep -rl bbo /path/to/directory

Grep OR (e.g. A or B or C or D)

grep 'A\|B\|C\|D'

Grep AND (e.g. A and B)

grep 'A.*B'

Regex any single character (e.g. ACB or AEB)

grep 'A.B'

Regex with or without a certain character (e.g. color or colour)

grep 'colou\?r'

Grep all content of a fileA from fileB

grep -f fileA fileB

Grep a tab

grep $'\t'

Grep variable from variable

$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
# grep -q will output 0 if match found
# remember to add space between []!

Grep strings between a bracket()

grep -oP '\(\K[^\)]+'

Grep number of characters with known strings in between(e.g. AAEL000001-RA)

grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character

Skip directory (e.g. 'bbo')

grep -d skip 'bbo' /path/to/files/*

Sed

[back to top]

Remove the 1st line

sed 1d filename

Remove the first 100 lines (remove line 1-100)

sed 1,100d filename

Remove lines with string (e.g. 'bbo')

sed "/bbo/d" filename
# case insensitive:
sed "/bbo/Id" filename

Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2)

sed -E '/^.{5}[^2]/d'
#aaaa2aaa (you can stay)
#aaaa1aaa (delete!)

Edit infile (edit and save to file), (e.g. deleting the lines with 'bbo' and save to file)

sed -i "/bbo/d" filename

When using variable (e.g. $i), use double quotes " "

# e.g. add >$i to the first line (to make a bioinformatics FASTA file)
sed "1i >$i"
# notice the double quotes! in other examples, you can use a single quote, but here, no way!
# '1i' means insert to first line

Using environment variable and end-of-line pattern at the same time.

# Use backslash for end-of-line $ pattern, and double quotes for expressing the variable
sed -e "\$s/\$/\n+--$3-----+/"

Delete/remove empty lines

sed '/^\s*$/d'

# or

sed '/^$/d'

Delete/remove last line

sed '$d'

Delete/remove last character from end of file

sed -i '$ s/.$//' filename

Add string to beginning of file (e.g. "[")

sed -i '1s/^/[/' filename

Add string at certain line number (e.g. add 'something' to line 1 and line 3)

sed -e '1isomething' -e '3isomething'

Add string to end of file (e.g. "]")

sed '$s/$/]/' filename

Add newline to the end

sed '$a\'

Add string to beginning of every line (e.g. 'bbo')

sed -e 's/^/bbo/' filename

Add string to end of each line (e.g. "}")

sed -e 's/$/\}\]/' filename

Add \n every nth character (e.g. every 4th character)

sed 's/.\{4\}/&\n/g'

Add a line after the line that matches the pattern (e.g. add a new line with "world" after the line with "hello")

sed '/hello*/a world' filename
# hello
# world

Concatenate/combine/join files with a separator and next line (e.g separate by ",")

sed -s '$a,' *.json > all.json

Substitution (e.g. replace A by B)

sed 's/A/B/g' filename

Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)

sed "s/aaa=.*/aaa=\/my\/new\/path/g"

Select lines start with string (e.g. 'bbo')

sed -n '/^@S/p'

Delete lines with string (e.g. 'bbo')

sed '/bbo/d' filename

Print/get/trim a range of line (e.g. line 500-5000)

sed -n 500,5000p filename

Print every nth lines

sed -n '0~3p' filename

# catch 0: start; 3: step

Print every odd # lines

sed -n '1~2p'

Print every third line including the first line

sed -n '1p;0~3p'

Remove leading whitespace and tabs

sed -e 's/^[ \t]*//'
# Notice a whitespace before '\t'!!

Remove only leading whitespace

sed 's/ *//'

# notice a whitespace before '*'!!

Remove ending commas

sed 's/,$//g'

Add a column to the end

sed "s/$/\t$i/"
# $i is the valuable you want to add

# To add the filename to every last column of the file
for i in $(ls); do sed -i "s/$/\t$i/" $i; done

Add extension of filename to last column

for i in T000086_1.02.n T000086_1.02.p; do sed "s/$/\t${i/*./}/" $i; done >T000086_1.02.np

Remove newline\ nextline

sed ':a;N;$!ba;s/\n//g'

Print a particular line (e.g. 123th line)

sed -n -e '123p'

Print a number of lines (e.g. line 10th to line 33 rd)

sed -n '10,33p' <filename

Change delimiter

sed 's=/=\\/=g'

Replace with wildcard (e.g A-1-e or A-2-e or A-3-e....)

sed 's/A-.*-e//g' filename

Remove last character of file

sed '$ s/.$//'

Insert character at specified position of file (e.g. AAAAAA --> AAA#AAA)

sed -r -e 's/^.{3}/&#/' filename

Awk

[back to top]

Set tab as field separator

awk -F $'\t'

Output as tab separated (also as field separator)

awk -v OFS='\t'

Pass variable

a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename

Print line number and number of characters on each line

awk '{print NR,length($0);}' filename

Find number of columns

awk '{print NF}'

Reverse column order

awk '{print $2, $1}'

Check if there is a comma in a column (e.g. column $1)

awk '$1~/,/ {print}'

Split and do for loop

awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename

Print all lines before nth occurrence of a string (e.g stop print lines when 'bbo' appears 7 times)

awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'

Print filename and last line of all files in directory

ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' filename

Add string to the beginning of a column (e.g add "chr" to column $3)

awk 'BEGIN{OFS="\t"}$3="chr"$3'

Remove lines with string (e.g. 'bbo')

awk '!/bbo/' filename

Remove last column

awk 'NF{NF-=1};1' filename

Usage and meaning of NR and FNR

# For example there are two files:
# fileA:
# a
# b
# c
# fileB:
# d
# e
awk 'print FILENAME, NR,FNR,$0}' fileA fileB
# fileA    1    1    a
# fileA    2    2    b
# fileA    3    3    c
# fileB    4    1    d
# fileB    5    2    e

AND gate

# For example there are two files:
# fileA:
# 1    0
# 2    1
# 3    1
# 4    0
# fileB:
# 1    0
# 2    1
# 3    0
# 4    1

awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB
# 1    0
# 2    1
# 3    0
# 4    0

Round all numbers of file (e.g. 2 significant figure)

awk '{while (match($0, /[0-9]+\[0-9]+/)){
    \printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
    \$0=substr($0, RSTART+RLENGTH)
    \}
    \print
    \}'

Give number/index to every row

awk '{printf("%s\t%s\n",NR,$0)}'

Break combine column data into rows

# For example, separate the following content:
# David    cat,dog
# into
# David    cat
# David    dog

awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' filename

# Detail here:　http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string

Average a file (each line in file contains only one number)

awk '{s+=$1}END{print s/NR}'

Print field start with string (e.g Linux)

awk '$1 ~ /^Linux/'

Sort a row (e.g. 1 40 35 12 23 --> 1 12 23 35 40)

awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'

Subtract previous row values (add column6 which equal to column4 minus last column5)

awk '{$6 = $4 - prev5; prev5 = $5; print;}'

Xargs

[back to top]

Set tab as delimiter (default:space)

xargs -d\t

Prompt commands before running commands

ls|xargs -L1 -p head

Display 3 items per line

echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3
# 4 5 6

Prompt before execution

echo a b c |xargs -p -n 3

Print command along with output

xargs -t abcd
# bin/echo abcd
# abcd

With find and rm

find . -name "*.html"|xargs rm

# when using a backtick
rm `find . -name "*.html"`

Delete files with whitespace in filename (e.g. "hello 2001")

find . -name "*.c