Resources

Current directory:

echo $PWD

Including the "Hash-Bang" / "She-Bang"

#!/bin/bash

• ^ Should go at the top of sh files.
  • this is why
  • Tells system to run with BASH

Sometimes you will see flags included as part of the shebang. For example, you can use -e (errexit) to have the script exit immediately if a command fails:

#!/bin/bash -e

Commenting

# My comment

Logic / flow

Test

Before using advanced branching logic, you should know that the shell has a built-in test check - basically coerces the output of an expression to a boolean that can be used in logic flows. Simply encase the expression/condition in brackets:

[ check-this-condition ]

or double brackets, as the newer version

If / Else

Great guides

• Dev.to - Tasevski - Shellscripting: Conditional Execution
• CompCiv - Conditional Branching
• TLDP - Beginner's Bash Guide - Chapter 7 - Conditional Statements

As long as the thing you want to do if the conditional is true always exits with exit code 0 (success), this works as simple one-liner if/else:

conditional_test && echo "it worked" || echo "it failed"

Helpful S/O's for understanding the above: 1, 2

Double Pipe vs Double Ampersand, and Other Flow Operators

Quick reference:

• && = Only execute right side if left side succeeds
  • Examples:
    • false && echo "this text will NOT show"
    • true && echo "this text will show"
• || = Only execute right side if left side FAILS (non-zero exit code)
  • Essentially the inverse of &&
  • Examples:
    • false || echo "this text will show"
    • bad_command || echo "this text will show"
    • true || echo "this text will NOT show"
• & = asynchronously runs both commands on either side, regardless of success of either, in detached (forked) processes
  • Warning: This can be a hackish way to do things
  • Definitely do not use this if the second command is dependent on the output of the first
  • Examples:
    • slow_command_to_execute & echo "this will appear before the left side is done!"
    • true & echo "this text will show"
    • false & echo "this text will also show"
• ; = Execute both side, sequentially, regardless of success of either
  • Examples:
    • true; echo "this text will show"
    • bad_command; echo "this text still shows"
  • Since this doesn't work in many Windows environments, an easy workaround to get the same behavior is to replace CMD_ONE; CMD_TWO with (CMD_ONE || true) && CMD_TWO.
    • This exhibits the same behavior, since CMD_TWO will also synchronously execute after CMD_ONE, regardless of its success
    • Great for NPM scripts
    • Nice writeup
• | = Not for logic flow, used for redirection
  • See "piping and redirection" section

Grep

• In general, if you are a RegEx power user, you will probably find sed much preferable. Or awk.

  • grep can actually be a bit of a pain when trying to do things like use capture groups (1, 2)

• Cheatsheets:

  • man
  • ss64
  • examples

• (Common) Flags:

  Flag	Description
  -E	Extended regex
  -o	Only output the matching part of the line
  -p	Treat as perl style regular exp
  -i	Ignore case
  -e	Pass explicit patterns, multiple allowed

Grep - Print Only Matching

The -o flag will do this, but it also adds line breaks. So, to get a clean output:

grep -o '{pattern}' | tr -d '\n'

# Example
echo hello | grep -o '.ll.' | tr -d '\n'
# prints 'ello'

sed

• Cheatsheets

  • man
  • ss64
  • full manual
    • Basic Syntax
    • Extended Syntax
    • Escape Sequences
  • Good regex-based guide

• Common flags

  Flag	Description
  -n	Silent, suppress printing of pattern space
  -r (or -E on some systems)	Use extended regexp - I always prefer

• Syntax

  • print output
    • echo $'hello world\nLine Two\nGoodbye' | sed -E -n '/Line.+/p'
      • Prints "Line Two"
  • substitute
    • echo $'hello world\nLine Two\nGoodbye' | sed -E 's/Line.+/my substitution/'
      • Prints:
        • hello world
          my substitution
          Goodbye
    • Example: Replace space with newline
      • echo 'item_a item_b' | sed -E 's/ /\n/g'
      • Prints:
        • item_a
          item_b
    • Example: Replace null character with new line
      • sed -E 's/\x0/\n/g'
  • Print only a specific capture group
    • This is actually a little complicated. Basically, you have to substitute the entire input with the back-reference of the capture.
      • sed -E -n 's/.*(My_Search).*/\1/p
    • In action:
      • echo $'hello world\nLine Two\nGoodbye' | sed -E -n 's/.*^Line (.+)$.*/\1/p'
        • Prints:
          • "Two"

Warning: sed on your system might have limitations - for example, be warned that if you can't use the Perl (-p) mode, you will need to use [0-9] instead of /d, for digits.

Capturing and Executing Output

If you simply want to "capture" the value output by a script and store it as a variable, you can use substitution. See "Storing into a variable".

If you want to execute the output of a command as a new command / script, you can use the (dangerous) eval command, plus substitution: eval $( MY_COMMAND ).

Here is a full example:

(echo echo \"in test.txt\") > test.txt
eval $( cat test.txt )
# "in test.txt"

Capturing Input Arguments in Bash Scripts

To capture arguments (aka positional parameters) within a script, you can use $@, and $# for the number of arguments (these are a form of Special Parameters). Make sure to double-quote when using - e.g.:

exec "$@"

You can use this to pass input arguments to a completely different program / process, which makes it handy for intermediate scripting.

Guide: Bash Hackers Wiki - Handling Positional Parameters

Piping and redirection

• Piping VS Redirection
  • Simple answer:
    • Piping: Pass output to another command, program, etc.
    • Redirect: Pass output to file or stream
• Pipe
  • |
  • echo 'hello world' | grep -o 'hello'
    • Prints hello
• Redirection
  • >
  • echo "hello" > output.txt

Problems with piping

Piping, in general, is taking the stdout of one process to the stdin of another. If the process you are trying to pipe to is expecting arguments and doesn't care about stdin, or ignores it, piping won't work as you want it to.

The best solution for this is usually to use xargs, which reads stdin and converts the input into arguments which are passed to the command of your choice.

Or, you can use substitution to capture the result of the first part of the pipe and reuse it in the second.

See this S/O answer for details.

If the input you are passing contains special characters or spaces (such as spaces in a filename), take extra care to handle it. For example, see if the thing generating the input can escape it and null terminate the fields (e.g. git-diff --name-only -z), and then you can use the -0 or --null option with xargs to tell it to expect null terminated fields.

Copying to Clipboard

There are a bunch of different options, and it largely depends on what you have available on your OS.

This S/O response serves as a good list.

??? - 2>&1

You see 2>&1 all over the place in bash scripts, because it is very useful. Essentially, it forces errors (stderr) to be piped to whatever value stdout is set to.

This has a few really handy and common uses:

1. See both the output and the errors in the console at the same time
   • Often errors are routed to stderr and not shown in the console.
2. Suppress errors
   • Since this forces errors to stdout, this has the side effect of suppressing them from their normal destination
     • However, they are still going to show up in stdout obviously. If you really want to suppress them entirely, use 2> /dev/null, which essentially sends them to oblivion
3. Send both output and errors to file
   • If you redirect to a file before using 2>&1, then both outputs gets sent to the file.
     • ls file-does-not-exist.txt > output.txt 2>&1
       • output.txt will now contain "ls: cannot access 'file-does-not-exist.txt': No such file or directory"

On a more technical level, Unix has descriptors that are kind of like IDs. 2 is the descriptor/id for stderr, and 1 is the id for stdout. In the context of redirection, using & + ID (&{descriptorId}) means copy the descriptor given by the ID. This is important for several reasons - one of which is that 2>1 could be interpreted as "send output of 2 to a file named 1", whereas 2>&1 ensures that it is interpreted as "send output of 2 to descriptor with ID=1".

So... kinda...

• 2>&1
  • can be broken down into:
• stderr>&stdout
  • ->
• stderr>value_of_stdout
  • ->
• stdout = stderr + stdout

Suppress errors

Make sure to see above section about how descriptors work with redirection, but a handy thing to remember is:

# Pretend command 'foobar' is likely to throw errors that we want to completely suppress
foobar 2>/dev/null

This sends error output to /dev/null, which basically discards all input.

Stderr Redirection - Additional reading

• great breakdown
• Good SO answers:
  • examples
• advanced IO redirection
• good forum thread

Checking for Errors

You can use $? to get the last exist status. Here are some examples.

Reminder: Anything other than 0 is an error ("non-zero exit code").

Using variables

Setting Variables

For setting variables, it depends on the variable type.

Normal variables:

VARIABLE_NAME=VARIABLE_VALUE

For this syntax, keep in mind that the key-pair can act like an environment variable if a command is immediately executed within the same process.

For example, MY_VAR=ABC printenv will show that MY_VAR has value ABC, but MY_VAR=ABC && printenv will not - it will show that MY_VAR is unset as an environment variable.

Environment variables (persisted through session):

export VARIABLE_NAME=VARIABLE_VALUE

Escape spaces by enclosing VARIABLE_VALUE in double quotes

Also, see Environment Variables subsection

Reading Variables

Prefix with $.

Example:

MYPATH="/home/joshua"
cd $MYPATH
echo "I'm in Joshua's folder!"

If you want to expand a variable inside a string, you can also use {} (curly braces) around the variable to expand it.

Default / Global Variables

In addition to using printenv to see all defined variables, you can also find lists of variables that usually come default with either the system shell, bash, or globally:

• TLDP - Internal Variables
• SS64 - Shell and environment variables

Storing into a variable

How to store the result of a command into a variable:

• There are two methods:
  • Command/process substitution (easy to understand)

  VARIABLE_NAME=$(command)

    - However, this doesn't always work with complex multi-step operations

  • read command (complicated) - works with redirection / piping

    echo "hello" | read VARIABLE_NAME

Environment Variables

List all env values

printenv

Set an environment variable - current process and sub-processes

export VARIABLE_NAME=VARIABLE_VALUE

Set an environment variable - permanently

In order for an environment variable to be persisted across sessions and processes, it needs to get saved and exported from a config file.

This is often done by manually editing /etc/environment:

• 1. Launch editor: sudo -H gedit /etc/environment
• 2. Append key-value pair: VAR_NAME="VAR_VAL"
• 3. Save

The difference between setting a variable with export vs without, is similar to the difference in MS Windows, for using setx vs just set -> export persists the value.

Global path

echo $PATH

Triggering / running a SH file

• Make sure it is "runnable" - that it has the execute permission
  • chmod +x /scriptfolder/scriptfile.sh
• Then call it:
  • /scriptfolder/scriptfile.sh

If you are having issues running from Windows...

• MAKE SURE LINE ENDINGS ARE \n and not \r\n

Also, make sure you specify directory, as in ./myscript.sh, not myscript.sh, even if you are currently in the same directory as script.

Keeping file open after execution

Add this to very end of file:

exec $SHELL

Note: this will interfere with scripts handing back control to other scripts; ie resuming flow between multiple scripts.

Strings

Special characters (newline, etc)

You need to prefix with $ before string to use special characters.

Example:

• echo 'hello\ngoodbye'
  • Prints:
    • "hello\ngoodbye"
• echo $'hello\ngoodbye'
  • Prints:
    • "hello
      goodbye"

You can also use printf for linebreaks: printf '\n\n'

Joining Strings

You can simply put strings together in variable assignment, like this:

FOO="Test"
BAR=$FOO"ing"
echo $BAR

echoes:

testing

Joining Strings with xargs

By default, xargs appends a space to arguments passed through. For example:

echo "Script" | xargs echo "Java"
# "Java Script"

If we want to disable that behavior, we can use the -I argument, which is really for substitution, but can be applied to this use-case:

echo "Script" | xargs -I {} echo "Java{}"
# Or...
echo "Script" | xargs -I % echo "Java%"
# Etc...

# Output: "JavaScript" - Success!

Converting to and from Base64 Encoding

Just use the base64 utility, which can be piped to, or will take a file input.

If you don't care about presentation, make sure to use --wrap=0 to disable column limit / wrapping

Skip Lines in Shell Output String

If you have skip lines in output (for example, to omit a summary row), you can use:

tail -n +{NUM_LINES_TO_SKIP + 1}
# Or, another way to think of it:
# tail -n +{LINE_TO_START_AT}

# Example: Skip very first line
printf 'First Line\nSecond Line\nThird Line' | tail -n +2
# Ouput is :
#     Second Line
#     Third Line

To skip the last line:

head -n -1

Trim Trailing Line Breaks

There are a bunch of ways to do this, but the first answer provided is probably the best - using command substitution, since it automatically removes trailing newlines:

echo -n "$(printf 'First Line\nSecond Line\nThird Line, plus three trailing newlines!\n\n\n')"

You could also use the head -n -1 trick to remove the very last line.

If you want to remove all line breaks, you can use tr for a easier to remember solution:

printf 'I have three trailing newlines!\n\n\n' | tr -d '\n'

Generate a Random String

There is an excellent StackExchange thread on the topic, and most answers boil down to either using /dev/urandom as a source, or openssl, both of which have wide portability and ease of use.

• /dev/urandom
  • From StackExchange

    # OWASP list - https://owasp.org/www-community/password-special-characters
    head /dev/urandom | tr -dc 'A-Za-z0-9!"#$%&'\''()*+,-./:;<=>?@[\]^_`{|}~' | head -c {length}

  • I've had some issues with the above command on Windows (with ported utils)...
• OpenSSL
  • For close to length:
    • openssl rand -base64 {length}
  • For exactly length:
    • openssl rand -base64 {length} | head -c {length}

Security, Encryption, Hashing

Quick Hash Generation

If you need to quickly generate a hash, checksum, etc. - there are multiple utilities you can use.

• sha256sum
  • Example: echo -n test | sha256sum
  • Example: cat msg.txt | sha256sum
• openssl dgst (-sha256, -sha512, etc.)
  • Example: echo -n test | openssl dgst -r -sha256
  • Example: openssl dgst -r -sha256 msg.txt

I'm using -r with openssl dgst to get its output to match the common standard that things like sha256sum, Window's CertUtil, and other generators use.

🚨 WARNING: Be really wary of how easy it is to accidentally add or include newlines and/or extra spacing in the content you are trying to generate a hash from. If you accidentally add one in the shell that is not present in the content, the hashes won't match.

There are even more solutions offered here.

How to generate keys and certs

• SSH Public / Private Pairs: Using ssh-keygen (available on most Unix based OS's, included Android)
  • You can run it with no arguments and it will prompt for file location to save to
    • ssh-keygen
  • Or, pass arguments, like -t for algorithm type, and -f for filename, -c for comment
    • ssh-keygen -t rsa -C "your_email@example.com"
  • Technically, the private/public keys generated by this can also be used with OpenSSL signing utils
• The standard convention for filenames is:
  • Public key: {name}_{alg}.pub
    • Example: id_rsa.pub
  • Private key:
    • No extension: {name}_{alg}
      • Example: id_rsa
    • Other extensions
      • .key
      • .pem
      • .private
      • Doesn't really matter; just don't use ppk, since that it very specific to putty
• Standard Public / Private Pairs: OpenSSL
  • See: Docs - "Generating an RSA Private Key"

💡 The final text in a public key, which is plain text that looks like username@users-pc actually does nothing, and is just a comment; you can set it on creation with -C mycomment if you like, or edit afterwards. But again, no impact.

How to use Public and Private Key Signing

Generally, the most widely used tool for asymmetric keys with Bash (or even cross-OS, with Windows support) is the OpenSSL CLI utilities.

Here are some resources on how to use OpenSSL for public/private key signing:

• https://www.zimuel.it/blog/sign-and-verify-a-file-using-openssl
• https://wiki.openssl.org/index.php/Command_Line_Utilities

Create new files, or update existing file timestamps

• Touch without any flags will create a file if it does not exist, and if it does already exist, update timestamp of "last accessed" to now
  • touch "myfolder/myfile.txt"
• If you want touch to only touch existing and never create new files, use -c
  • touch -c "myfile.txt"
• Specifically update last accessed stamp of file
  • touch -a "myfolder/myfile.txt"
• specifically update "Last MODIFIED" stamp of file
  • touch -m "myfolder/myfile.txt"
• You can also use wildcard matching
  • touch -m *.txt
• and combine flags
  • touch -m -a *.txt

Verify Files

You can verify that a file exists with test -f {filepath}. Handy guide here.

If you want to check the line-endings of a file, for example to detect the accidental usage of CRLF instead of LF, you can use file MY_FILE.

Hex View

To view the hex of a file, you can use xxd {FILE_PATH}

Deleting

• Delete everything in a directory you are CURRENTLY in:
  • Best:
    • find -mindepth 1 -delete
  • UNSAFE!!!
    • rm -rf *
  • Better, since it prompts first:
    • rm -ri *
• Delete everything in a different directory (slightly safer than above)
  • rm -rf path/to/folder
• Delete based on pattern
  • find . -name '*.js' -delete

LS and File Listing

📄 LS - SS64

LS Cheatsheet

⚡ -> Nice combo: ls -alht --color (or, easier to remember ls -halt --color). All files, with details, human readable filesizes, color-coded, and sorted by last modified.

ls - show all files, including hidden files and directories (like .git)

ls -a

List directory sizes

du -sh *

Print a Directory Tree View with Bash

Both Windows and *nix support the tree command.

If, for some reason, you can't use that command, some users on StackOverflow have posted solutions that emulate tree using find + sed.

Count Matching Files

For a faster file count operation, you can use find's printf option to replace all filenames with dots, and then use wc character count to count them. Like this:

find {PATH} {FILTER} -type f -printf '.' | wc -c

Here is an example, to count all the .md Markdown files in a /docs directory:

find ./docs -iname "*.md" -type f -printf '.' | wc -c

Credit

Show progress bar / auto-update / keep console updated:

Great SO Q&A

Find executable paths

If you are looking for the bash equivalent of Window's "where" command, to find how a binary is exposed, try using which. E.g. which node.

Symlinks (symbolic links)

You can use the ln command (ss64) to create symbolic links.

# Works for both files and directories
ln -s {realTargetPath} {symbolicFileName}

# If you need to update an existing symlink, you need to use "force"
ln -sf {realTargetPath} {symbolicFileName}

In general, it is best / easiest to always use absolute paths for the targets.

Evaluating symlinks

You can use ls -la to list all files, including symlinks.

If you just want to see resolved symlinks, you can use grep - ls -la | grep "\->"

Networking

💡 An excellent package to get for working with network stuff is net-tools. It is also what contains netstat, which is great for watching active connections / ports.

cURL

• Good cheatsheets
  • devhints.io
  • dennyzhang
• Show headers only
  • curl -I http://example.com
• Search for something
  • You can't just pipe directly to grep or sed, because curl sends progress info stderr, so use --silent flag:
    • curl --silent https://joshuatz.com | sed -E -n 's/.*<title>(.+)<\/title>.*/\1/p'
      • Prints: Joshua Tzucker's Site
• Download a file
  • Specify filename: curl -o {New_Filename_Or_Path} {URL}
  • Reuse online filename: curl -O {URL_with_filename}
• Follow redirects: -L
  • Useful for downloading DropBox links (or else you get an empty file):
    • curl -L -o myfile.txt https://www.dropbox.com/s/....?dl=1

Networking - Checking DNS Records and Domain Info

Overview post of a few different methods

• dig
  • Default (A records + NS): dig {DOMAIN}
  • All: dig {DOMAIN} ANY
  • Specific type: dig {DOMAIN} {RECORD_TYPE}
    • dig joshuatz.com cname
• host
  • Default (describes records): host {DOMAIN}
  • All: host -a {DOMAIN}
  • Specific type: host -t {RECORD_TYPE} {DOMAIN}
• nslookup
  • (might not be available on all distros, but useful since this works on Windows too. However, nslookup also seems less reliable...)
  • Default (A record): nslookup {DOMAIN}
  • All: nslookup -d {DOMAIN}
    • Equivalent to nslookup -t ANY {DOMAIN}
  • Specific type: nslookup -querytype {RECORD_TYPE} {DOMAIN}
    • OR:nslookup -t {RECORD_TYPE} {DOMAIN}

Networking - How do I...

• Resolve DNS hostname to IP
  • getent hosts HOST_NAME | awk '{ print $1 }'
  • Credit goes to this S/O
• Download a file and save it locally with bash?
  • You can use wget or cURL (S/O):
    • wget -O {New_Filename_Or_Path} {URL}
    • curl -o {New_Filename_Or_Path} {URL}
  • If you want to just use the name of the file as-is, you can drop -O with wget
• Transfer files across devices using bash?
  • You can transfer over SSH, using the scp command
    • Example: scp my-file.txt joshua@1.1.1.1:/home/joshua
    • Example: scp -i ssh_pkey my-file.txt joshua@1.1.1.1:/home/joshua
    • Example: scp -rp ./my-dir joshua@1.1.1.1:/home/joshua/my-dir
  • Another option good option is rsync, especially for frequent syncs of data where some has stayed the same (it optimizes for syncing only what has changed).
    • Example: rsync -az -P . joshua@domain.com:/home/joshua/my_dir
    • Use --filter=':- .gitignore' to reuse gitignore file for exclusions
    • You can use --filter multiple times and they will be combined
    • Use --exclude to exclude single files, directories, or globs, also allowed multiple times
    • Use --dry-run to preview
    • Guide: "Deploying Code with Rsync"
  • Alternatively, you could use cURL to upload your file, to a service like transfer.sh, and then cURL again on your other device to download the same file via the generated link

Archives

How do I...

• Extract / unpack a tarball
  • tar -xvf {filename}
  • For more options, see docs, and here is a helpful page with examples
• Extract / unpack a .zip archive
  • This is not natively supported on many flavors of Linux, but can be added by installing and using a program such as unzip

Handy Commands for Exploring a New OS

x86_64 == amd64

Get Public IP Address

Easy mode: curl http://icanhazip.com

Lots of different options out there.

Echoing out Dates

The main command to be familiar with is the date utility.

You can use date +FMT_STRING to specify the format to apply to the output.

Common Formats:

Get Date as MS Since Epoch

If you don't actually need the full precision of milliseconds, but need the format / length, you can use: date +%s000

If you really need as-close-to-real MS timestamps, you can use any of these (some might not work on all systems):

• date +%s%3N
• date +%s%N | cut -b1-13
• echo $(($(date +%s%N)/1000000))

Above solutions were gathered from this S/O question, which has a bunch of great responses.

You could also always use node -p "Date.now()" if you have NodeJS installed.

User Management

Adding or Modifying Users

Use adduser {username} to add a new (non-root) user.

If you want to create a new user, but also grant them sudo / admin privileges, you can either:

• Add to sudo group while creating
  • useradd --groups sudo {username}
    • OR:
  • adduser {username] --ingroup sudo
• Create user first, then add to sudo group
  1. Create user:
     • adduser {username]
       • OR:
     • useradd {username}
  2. usermod -a -G sudo {username}

💡 Note: The above commands could also be used for adding to groups other than sudo - just swap out sudo with the group you want to use

🚨 Warning: Creating a new user will not automatically grant them SSH access. See SSH Notes for details.

The adduser {USER} {GROUP} syntax only works if the user already exists.

Add User to Group

usermod -a -G groupname username

(also see above section(s))

Listing User Groups

You can use groups to list all groups you are a part of, or use groups {USER} for a specific user.

For listing all groups on a system, you might be able to use less /etc/group or getent group (see more here).

Deleting a User

Use userdel {USERNAME} to delete a user. Optionally, pass the -r flag to also delete their home directory.

Session, Window, and Screen Management

Screen

If you need to manage multiple sessions, which you can leave and resume at any time, screen is the usual go-to program.

Screen Docs: linux.die.net, SS64

Troubleshooting

• Input has stopped appearing as you type it
  • This can happen for a number of reasons. The quick fix is usually to use reset or stty sane.