Bash / Shell - Cheatsheet
Resources
What & Link | Type |
---|---|
SS64 Bash Reference | Docs |
man7: Linux man Pages | Docs |
Wooledge / GreyCat: Bash Reference Sheet | Cheatsheet |
ExplainShell.com - breaks down any given command and explains what it does) |
Interactive Tool |
Current directory:
echo $PWD
Including the "Hash-Bang" / "She-Bang"
#!/bin/bash
-
^ Should go at the top of sh files.
- this is why
- Tells system to run with BASH
Commenting
# My comment
Logic / flow
Test
Before using advanced branching logic, you should know that the shell has a built-in test
check - basically coerces the output of an expression to a boolean that can be used in logic flows. Simply encase the expression/condition in brackets:
[ check-this-condition ]
or double brackets, as the newer version
If / Else
Great guides
- https://dev.to/puritanic/shellscripting-conditional-execution-3kgm
- http://www.compciv.org/topics/bash/conditional-branching/
- https://www.tldp.org/LDP/Bash-Beginners-Guide/html/chap_07.html
As long as the thing you want to do if the conditional is true always exits with exit code 0 (success), this works as simple one-liner if/else:
conditional_test && echo "it worked" || echo "it failed"
Double Pipe vs Double Ampersand, and Other Flow Operators
Quick reference:
-
&&
= Only execute right side if left side succeeds-
Examples:
false && echo "this text will NOT show"
true && echo "this text will show"
-
-
||
= Only execute right side if left side FAILS (non-zero exit code)- Essentially the inverse of
&&
-
Examples:
false || echo "this text will show"
bad_commmand || echo "this text will show"
true || echo "this text will NOT show"
- Essentially the inverse of
-
&
= asynchronously runs both commands on either side, regardless of success of either, in detached (forked) processes- Warning: This can be a hackish way to do things
- Definitely do not use this if the second command is dependent on the output of the first
-
Examples:
slow_command_to_execute & echo "this will appear before the left side is done!"
true & echo "this text will show"
false & echo "this text will also show"
-
;
= Execute both side, sequentially, regardless of success of either-
Examples:
true; echo "this text will show"
bad_command; echo "this text still shows"
-
Since this doesn't work in many Windows environments, an easy workaround to get the same behavior is to replace
CMD_ONE; CMD_TWO
with(CMD_ONE || true) && CMD_TWO
.- This exhibits the same behavior, since
CMD_TWO
will also synchronously execute afterCMD_ONE
, regardless of its success - Great for NPM scripts
- Nice writeup
- This exhibits the same behavior, since
-
-
|
= Not for logic flow, used for redirection
Grep
-
In general, if you are a RegEx power user, you will probably find
sed
much preferable. Orawk
. -
Cheatsheets:
-
(Common) Flags:
Flag Description -E
Extended regex -o
Only output the matching part of the line -p
Treat as perl style regular exp -i
Ignore case -e
Pass explicit patterns, multiple allowed
Grep - Print Only Matching
The -o
flag will do this, but it also adds line breaks. So, to get a clean output:
grep -o '{pattern}' | tr -d '\n'
# Example
echo hello | grep -o '.ll.' | tr -d '\n'
# prints 'ello'
sed
-
Cheatsheets
-
Common flags
Flag Description -n
Silent, suppress printing of pattern space -r
(or-E
on some systems)Use extended regexp - I always prefer -
Syntax
-
print output
-
echo $'hello world\nLine Two\nGoodbye' | sed -E -n '/Line.+/p'
- Prints "Line Two"
-
-
substitute
-
echo $'hello world\nLine Two\nGoodbye' | sed -E 's/Line.+/my substitution/'
-
Prints:
- hello world
my substitution
Goodbye
- hello world
-
-
Example: Replace space with newline
echo 'item_a item_b' | sed -E 's/ /\n/g'
-
Prints:
- itema
itemb
- itema
-
Example: Replace
null
character with new linesed -E 's/\x0/\n/g'
-
-
Print only a specific capture group
-
This is actually a little complicated. Basically, you have to substitute the entire input with the back-reference of the capture.
sed -E -n 's/.*(My_Search).*/\1/p
-
In action:
-
echo $'hello world\nLine Two\nGoodbye' | sed -E -n 's/.*^Line (.+)$.*/\1/p'
-
Prints:
- "Two"
-
-
-
-
Warning:
sed
on your system might have limitations - for example, be warned that if you can't use thePerl
(-p
) mode, you will need to use[0-9]
instead of/d
, for digits.
Capturing and Executing Output
If you simply want to "capture" the value output by a script and store it as a variable, you can use substitution. See "Storing into a variable".
If you want to execute the output of a command as a new command / script, you can use the (dangerous) eval
command, plus substitution: eval $( MY_COMMAND )
.
Here is a full example:
(echo echo \"in test.txt\") > test.txt
eval $( cat test.txt )
# "in test.txt"
Piping and redirection
-
Piping VS Redirection
-
Simple answer:
- Piping: Pass output to another command, program, etc.
- Redirect: Pass output to file or stream
-
-
Pipe
|
-
echo 'hello world' | grep -o 'hello'
- Prints
hello
- Prints
-
Redirection
>
echo "hello" > output.txt
Problems with piping
Piping, in general, is taking the stdout of one process to the stdin of another. If the process you are trying to pipe to is expecting arguments and doesn't care about stdin, or ignores it, piping won't work as you want it to.
The best solution for this is usually to use xargs
, which reads stdin
and converts the input into arguments which are passed to the command of your choice.
Or, you can use substitution
to capture the result of the first part of the pipe and reuse it in the second.
See this S/O answer for details.
If the input you are passing contains special characters or spaces (such as spaces in a filename), take extra care to handle it. For example, see if the thing generating the input can escape it and null terminate the fields (e.g.
git-diff --name-only
-z
), and then you can use the-0
or--null
option withxargs
to tell it to expect null terminated fields.
??? - 2>&1
You see 2>&1
all over the place in bash scripts, because it is very useful. Essentially, it forces errors (stderr
) to be piped to whatever value stdout
is set to.
This has a few really handy and common uses:
-
See both the output and the errors in the console at the same time
- Often errors are routed to
stderr
and not shown in the console.
- Often errors are routed to
-
Suppress errors
-
Since this forces errors to
stdout
, this has the side effect of suppressing them from their normal destination- However, they are still going to show up in
stdout
obviously. If you really want to suppress them entirely, use2> /dev/null
, which essentially sends them to oblivion
- However, they are still going to show up in
-
-
Send both output and errors to file
-
If you redirect to a file before using
2>&1
, then both outputs gets sent to the file.-
ls file-does-not-exist.txt > output.txt 2>&1
output.txt
will now contain "ls: cannot access 'file-does-not-exist.txt': No such file or directory"
-
-
On a more technical level, Unix has descriptors that are kind of like IDs. 2
is the descriptor/id for stderr
, and 1
is the id for stdout
. In the context of redirection, using &
+ ID (&{descriptorId}
) means copy the descriptor given by the ID. This is important for several reasons - one of which is that 2>1
could be interpreted as "send output of 2 to a file named 1", whereas 2>&1
ensures that it is interpreted as "send output of 2 to descriptor with ID=1".
So... kinda...
-
2>&1
- can be broken down into:
-
stderr>&stdout
- ->
-
stderr>value_of_stdout
- ->
stdout = stderr + stdout
Suppress errors
Make sure to see above section about how descriptors work with redirection, but a handy thing to remember is:
# Pretend command 'foobar' is likely to throw errors that we want to completely suppress
foobar 2>/dev/null
This sends error output to /dev/null
, which basically discards all input.
Stderr Redirection - Additional reading
- great breakdown
-
Good SO answers:
- advanced IO redirection
- good forum thread
Using variables
Simple:
VARIABLE_NAME=VARIABLE_VALUE
Example:
MYPATH="C:\Users\Joshua"
cd $MYPATH
echo "I'm in Joshua's folder!"
Storing into a variable
How to store the result of a command into a variable:
-
There are two methods:
-
Command/process substitution (easy to understand)
VARIABLE_NAME=$(command)
- However, this doesn't always work with complex multi-step operations
-
read
command (complicated) - works withredirection / piping
echo "hello" | read VARIABLE_NAME
-
Environment Variables
List all env values
printenv
Set an environment variable - current process and sub-processes
export VARIABLE_NAME=VARIABLE_VALUE
Set an environment variable - permanently
In order for an environment variable to be persisted across sessions and processes, it needs to get saved and exported from a config file.
This is often done by manually editing /etc/environment
:
-
- Launch editor:
sudo -H gedit /etc/environment
- Launch editor:
-
- Append key-value pair:
VAR_NAME="VAR_VAL"
- Append key-value pair:
-
- Save
The difference between setting a variable with
export
vs without, is similar to the difference in MS Windows, for usingsetx
vs justset
->export
persists the value.
Global path
echo $PATH
Triggering / running a SH file
-
Make sure it is "runnable" - that it has the execute permission
chmod +x /scriptfolder/scriptfile.sh
-
Then call it:
/scriptfolder/scriptfile.sh
If you are having issues running from Windows...
- MAKE SURE LINE ENDINGS ARE \n and not \r\n
Also, make sure you specify directory, as in ./myscript.sh
, not myscript.sh
, even if you are currently in the same directory as script.
Keeping file open after execution
Add this to very end of file:
exec $SHELL
Note: this will interfere with scripts handing back control to other scripts; ie resuming flow between multiple scripts.
Strings
Special characters (newline, etc)
You need to prefix with $
before string to use special characters.
Example:
-
echo 'hello\ngoodbye'
-
Prints:
- "hello\ngoodbye"
-
-
echo $'hello\ngoodbye'
-
Prints:
- "hello
goodbye"
- "hello
-
You can also use
printf
for linebreaks:printf '\n\n'
Joining Strings
You can simply put strings together in variable assignment, like this:
FOO="Test"
BAR=$FOO"ing"
echo $BAR
echoes:
testing
Joining Strings with xargs
By default, xargs appends a space to arguments passed through. For example:
echo "Script" | xargs echo "Java"
# "Java Script"
If we want to disable that behavior, we can use the -I
argument, which is really for substitution, but can be applied to this use-case:
echo "Script" | xargs -I {} echo "Java{}"
# Or...
echo "Script" | xargs -I % echo "Java%"
# Etc...
# Output: "JavaScript" - Success!
Converting to and from Base64 Encoding
Just use the base64
utility, which can be piped to, or will take a file input.
If you don't care about presentation, make sure to use
--wrap=0
to disable column limit / wrapping
Skip Lines in Shell Output String
If you have skip lines in output (for example, to omit a summary row), you can use:
tail -n +{NUM_LINES_TO_SKIP + 1}
# Or, another way to think of it:
# tail -n +{LINE_TO_START_AT}
# Example: Skip very first line
printf 'First Line\nSecond Line\nThird Line' | tail -n +2
# Ouput is :
# Second Line
# Third Line
Trim Trailing Line Breaks
There are a bunch of ways to do this, but the first answer provided is probably the best - using command substitution, since it automatically removes trailing newlines:
echo -n "$(printf 'First Line\nSecond Line\nThird Line, plus three trailing newlines!\n\n\n')"
If you want to remove all line breaks, you can use tr
for a easier to remember solution:
printf 'I have three trailing newlines!\n\n\n' | tr -d '\n'
Generate a Random String
There is an excellent StackExchange thread on the topic, and most answers boil down to either using /dev/urandom
as a source, or openssl
, both of which have wide portability and ease of use.
-
/dev/urandom
-
From StackExchange
# OWASP list - https://owasp.org/www-community/password-special-characters head /dev/urandom | tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' | head -c {length}
- I've had some issues with the above command on Windows (with ported utils)...
-
-
OpenSSL
-
SSH Public / Private Pairs: Using ssh-keygen (available on most Unix based OS's, included Android)
-
You can run it with no arguments and it will prompt for file location to save to
ssh-keygen
-
Or, pass arguments, like
-t
for algorithm type, and-f
for filename,-c
for commentssh-keygen -t rsa -C "your_email@example.com"
- Technically, the private/public keys generated by this can also be used with OpenSSL signing utils
-
-
Standard Public / Private Pairs: OpenSSL
How to use Public and Private Key Signing
Generally, the most widely used tool for asymmetric keys with Bash (or even cross-OS, with Windows support) is the OpenSSL CLI utilities.
Here are some resources on how to use OpenSSL for public/private key signing:
- https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl
- https://wiki.openssl.org/index.php/Command_Line_Utilities
Create new files, or update existing file timestamps
-
Touch without any flags will create a file if it does not exist, and if it does already exist, update timestamp of "last accessed" to now
touch "myfolder/myfile.txt"
-
If you want touch to only touch existing and never create new files, use
-c
touch -c "myfile.txt"
-
Specifically update last accessed stamp of file
touch -a "myfolder/myfile.txt"
-
specifically update "Last MODIFIED" stamp of file
touch -m "myfolder/myfile.txt"
-
You can also use wildcard matching
touch -m *.txt
-
and combine flags
touch -m -a *.txt
Verify Files
You can verify that a file exists with test -f {filepath}
. Handy guide here.
Deleting
-
Delete everything in a directory you are CURRENTLY in:
-
Best:
find -mindepth 1 -delete
-
UNSAFE!!!
rm -rf *
-
Better, since it prompts first:
rm -ri *
-
-
Delete everything in a different directory (slightly safer than above)
rm -rf path/to/folder
-
Delete based on pattern
find . -name '*.js' -delete
LS and File Listing
📄 LS - SS64
LS Cheatsheet
How to... | Cmd |
---|---|
Show all files | ls -a |
Show filesizes (human readable) | ls -lh |
Show filesize (MB) | ls --block-size=MB |
Show details (long) | ls -l (or, more commonly, ls -al ) |
Sort by last modified | ls -t |
âš¡ -> Nice combo:
ls -alht --color
(or, easier to rememberls -halt --color
). All files, with details, human readable filesizes, color-coded, and sorted by last modified.
ls - show all files, including hidden files and directories (like .git
)
ls -a
Print a Directory Tree View with Bash
Both Windows and *nix support the tree
command.
If, for some reason, you can't use that command, some users on StackOverflow have posted solutions that emulate tree using find + sed.
Show progress bar / auto-update / keep console updated:
Great SO Q&A
Find executable paths
If you are looking for the bash equivalent of Window's "where" command, to find how a binary is exposed, try using which
. E.g. which node
.
Symlinks (symbolic links)
You can use the ln
command (ss64) to create symbolic links.
# Works for both files and directories
ln -s {realTargetPath} {symbolicFileName}
# If you need to update an existing symlink, you need to use "force"
ln -sf {realTargetPath} {symbolicFileName}
In general, it is best / easiest to always use absolute paths for the targets.
Evaluating symlinks
You can use ls -la
to list all files, including symlinks.
If you just want to see resolved symlinks, you can use grep - ls -la | grep "\->"
Networking
cURL
-
Good cheatsheets
- Show headers only
curl -I http://example.com
-
Search for something
-
You can't just pipe directly to grep or sed, because curl sends progress info
stderr
, so use--silent
flag:-
curl --silent https://joshuatz.com | sed -E -n 's/.*<title>(.+)<\/title>.*/\1/p'
- Prints:
Joshua Tzucker's Site
- Prints:
-
-
-
Download a file
- Specify filename:
curl -o {New_Filename_Or_Path} {URL}
- Reuse online filename:
curl -O {URL_with_filename}
- Specify filename:
Networking - How do I...
-
Resolve DNS hostname to IP
getent hosts HOST_NAME | awk '{ print $1 }'
- Credit goes to this S/O
-
Download a file and save it locally with bash?
-
You can use
wget
orcURL
(S/O):wget -O {New_Filename_Or_Path} {URL}
curl -o {New_Filename_Or_Path} {URL}
- If you want to just use the name of the file as-is, you can drop
-O
with wget
-
Archives
How do I...
-
Extract / unpack a tarball
tar -xvf {filename}
- For more options, here is a helpful page
Handy Commands for Exploring a New OS
Command | What? |
---|---|
uname -a |
Display system OS info |
apt list --installed |
List installed packages |
crontab -l or less /etc/crontab |
View crontab entries |
Echoing out Dates
The main command to be familiar with is the date
utility.
You can use date +FMT_STRING
to specify the format to apply to the output.
Common Formats:
Command | What | Sample |
---|---|---|
date |
Prints current date/time in %c format |
Sat Nov 28 03:56:03 PST 2020 |
date -u +"%Y-%m-%dT%H:%M:%SZ" |
Prints current date, a full ISO-8601 string | 2020-11-28T12:11:27Z |
date +%s |
Seconds since epoch | 1606565661 |
Get Date as MS Since Epoch
If you don't actually need the full precision of milliseconds, but need the format / length, you can use: date +%s000
If you really need as-close-to-real MS timestamps, you can use any of these (some might not work on all systems):
date +%s%3N
date +%s%N | cut -b1-13
echo $(($(date +%s%N)/1000000))
Above solutions were gathered from this S/O question, which has a bunch of great responses.
You could also always use
node -p "Date.now()"
if you have NodeJS installed.