A deep dive into Bash scripts to help you automate tasks

Shinichi Okada

·

Follow

Published in

Better Programming

·

15 min read

·

Jan 29, 2021

--

5

Listen

Share

[Updated on 2021–02-18. Code changed to Gist and added links]

Introduction

Bash scripts allow you to automate command-line tasks. For example, watch this video. The video shows how to automate creating a YouTube channel with Bash scripts. After watching it, you may want to create your own YouTube channel. For another example, in this article, I automated common terminal tasks.

In this article, we’ll cover basic Bash scripting for beginners and create simple executable scripts.

Let’s dive into Bash scripting!

Table of Contents· Introduction
· Setting Things Up
  ∘ Mac
  ∘ The bin directory
· VSCode Extensions
· Getting Started
  ∘ Shebang
  ∘ Comment
· Variables
  ∘ Variable assignment
  ∘ Environment variables
  ∘ Internal variables
  ∘ Assigning the command output to a variable
  ∘ Built-in commands
· Tests
  ∘ if statement
  ∘ Spaces Matter
  ∘ Test file expressions
  ∘ Test string expressions
  ∘ Test integer expressions
  ∘ Double parentheses
· How To Use if/else and if/elif/else Statements
· Double Brackets
· How To Use for Loops
  ∘ Using a for loop to rename files
· How To Use Parameters
· How To Accept User Input
· Brace Expansion Using Ranges
· How To Use While Loops
· What Are Exit Status/Return Codes?
  ∘ Checking the exit status
· How To Connect Commands
  ∘ Logical operators and the command exit status
  ∘ Exit command
  ∘ Logical AND (&&)
  ∘ Logical OR (||)
  ∘ The semicolon
  ∘ Pipe |
· Functions
  ∘ Function parameters
· Variable Scope
  ∘ Local variables
· Function Exit Status
· Checklist
· Conclusion
  ∘ Newsletter
  ∘ References

Setting Things Up

Mac

You can check your Bash version:

$ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
Copyright (C) 2007 Free Software Foundation, Inc.

Let’s upgrade the Bash version:

$ brew install bash

You can list all of the instances of executables found.

$ which -a bash
/usr/local/bin/bash
/bin/bash

Let’s check the newly installed version:

$ /usr/local/bin/bash --version
GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.5.0)
Copyright (C) 2019 Free Software Foundation, Inc.
...

Let’s find out which version of Bash you’re using:

$ which bash
/usr/local/bin/bash
# or
$ type bash
bash is /usr/local/bin/bash

If this doesn’t work, you need to export the Bash path in your ~/.bashrc (or ~/.zshrc for Zsh users).

export PATH="/usr/local/bin:$PATH"

Open a new terminal tab or restart your terminal, and check the Bash version.

$ bash --version
GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.5.0)
Copyright (C) 2019 Free Software Foundation, Inc.
...

Add /usr/local/bin/bash to /etc/shells as a trusted shell.

$ vim /etc/shells
# List of acceptable shells for chpass(1).
# Ftpd will not allow users to connect who are not using
# one of these shells./bin/bash
/bin/csh
/bin/ksh
/bin/sh
/bin/tcsh
/bin/zsh
/usr/local/bin/bash

Check the Bash version in a Bash script. We’ll learn how to run this script later.

#!/usr/local/bin/bash
echo $BASH_VERSION

Mac gotcha
Mac uses Zsh as the default shell. For example, echo $BASH_VERSION won't work on the terminal. It works from the shell script if you’re using #!/bin/bash.

Running Linux on Mac
You can use Multipass to run Ubuntu on Mac/Windows. This article will help you get started. An alternative is using Docker and using Linux images.

The bin directory

Let’s create a directory and add the path to the terminal config file (.bashrc, .zshrc). By doing this, you can run your script as myscript rather than ./myscript.

In your home directory, create the bin directory:

$ mkdir bin

Add the path to this directory in your terminal configuration file. (either .bashrc or .zshrc)

export PATH="$PATH:$HOME/bin"

Reload the config file:

# for .zshrc
$ . ~/.zshrc 
# or
$ source ~/.zshrc
# for .bashrc
$ . ~/.bashrc
# or
$ source ~/.bashrc

We’ll create all of our scripts in this bin directory.

Multipass tip
You can mount the local directory to your Multipass instance and export the PATH in the ~/.bashrc.

VSCode Extensions

If you’re using VSCode, I recommend the shellcheck [2] and shell-format [3] extensions. shellcheck adds a linter for shell scripts for Bash, Zsh, and sh. shell-format is a shellscript document format.

Getting Started

It’s a convention to use a .sh extension, but you don’t need to use an extension. Create a file called myscript:

$ cd ~/bin
# create a empty file
$ > myscript
# or
$ touch myscript

Then add the following:

We need to make the file executable. In the terminal:

$ chmod 755 myscript

If you’ve added the bin directory in your terminal config file (~/.bashrc or ~/.zshrc), you can run it as:

$ myscript
Welcome to Shell Scripting.

Shebang

The #! syntax at the beginning of scripts indicates an interpreter for execution under UNIX/Linux operating systems.

#!/bin/bash
echo "I use the newest bash interpreter."
---
#!/bin/csh
echo "use csh interpreter."
---
#!/bin/ksh
echo "use ksh interpreter."
---
#!/bin/zsh
echo "use zsh interpreter."

Use one of the outputs from which -a bash. If your Bash isn’t in the /bin directory, you can alternatively use #!/usr/bin/env bash.

Comment

Use # to comment out a line.

#!/bin/bash
# This is a bash comment.
echo "Hi"

Variables

The Bash variables are case sensitive, and using their names in uppercase is a convention. But you’re free to use a lowercase name or mix cases. The name of a variable can contain only letters (a-z or A-Z); numbers (0-9); or the underscore character ( _), starting with a letter. Don’t start them with a digit and no spaces before and after the = symbol.

Please note that some websites recommend using all-lowercase variable names since system-variable names are usually all in uppercase.

# valid
FIRSTLETTERS="ABC"
FIRST_THREE_LETTERS="ABC"
firstThreeLetters="ABC"
MY_SHELL="bash"
my_another_shell="my another shell"
My_Shell="My shell"# Invalid
3LETTERS="ABC"
first-three-letters="ABC"
first@Thtree@Letters="ABC"
ABC = "ABC "
MY_SHELL = "bash"
My-SHELL="bash"
1MY_SHELL="My shell"

When you use letters before/after the variable, use {}:

Variable assignment

Double quotes preserve the literal value within the quotes except for $, `, and \. The dollar-sign symbol and the backtick keep their special meaning within the double quotes and a backslash, \, is an escape character.

Single quotes preserve the literal value within the quotes.

Environment variables

Linux environment variables have information stored within the system. Global variables are also called environment variables.

You can find some of the environment variables in your terminal:

$ env
# or
$ printenv

You can use these variables in your script:

#!/bin/bash
echo $SHELL, $USER, $HOME # /bin/zsh, shinokada, /Users/shinokada

Internal variables

There are Bourne shell reserved variables, Bash reserved variables and special Bash variables. [1]

Assigning the command output to a variable

You can use $(command) to store the command output in a variable. For example, you can output the ls -l:

#!/bin/bash
# file name ex1
LIST=$(ls -l)
echo "File information: $LIST"

Run it in your terminal (don’t forget to make the file executable).

$ ex1
File information total 40
-rwxr-xr-x  1 shinokada  staff   272 Oct 27 08:52 ex1
-rwxr-xr-x  1 shinokada  staff     0 Oct 26 14:14 ex2
...

The following will save the time and date, a username, and the system uptime to a logfile. > is one of the redirects, and it’ll overwrite a file. You can use >> to append outputs to a file.

#!/bin/bashDATE=$(date -u) # date and time, -u option gives the UTC
WHO=$(whoami) # user name
UPTIME=$(uptime) # shows how long the system has been running
echo "Today is $DATE. You are $WHO. Uptime info: $UPTIME" > logfile

Built-in commands

Shell built-in commands are commands that you can run in a shell. You can find built-in commands:

$ compgen -b | sort
-
.
:
[
alias
autoload
bg
bindkey
break
...

You can use type to find the kind of command:

$ type cd
cd is a shell builtin

The which command returns the file pathname. It works only for executable programs:

$ which ls
/usr/bin/ls

The which command returns either no response or an error message when you use it for some of the built-ins or aliases that are substitutes for actual executable programs:

# no response
$ which cd
$ which umask
$ which alert

You can find more details about builtin by man builtin:

$ man builtin
BUILTIN(1)  BSD General Commands ManualNAME
     builtin, !, %, ., :, @, {, }, alias, alloc, bg, bind, bindkey,
     break, breaksw, builtins, case, cd, chdir, command, complete, 
     continue, default, dirs, do, done, echo, echotc, elif, else, end, 
     ....

Tests

The test command performs a variety of checks and comparisons, and you can use it with if statements.

if statement

The then keyword follows after an if statement:

if [ condition-for-test ]
then
  command
  ...
fi

You can use if and then in one line using ; to terminate the if statement:

if [ condition-for-test ]; then
  command
  ...
fi

(There will be more about the if statement later.)

Generally, the comparison works with double quotes and single quotes when a variable is a single word:

Single quotes won’t work in certain cases:

Without double quotes, you may have a problem:

In the first test, there are four arguments: two arguments from $VAR1, which has "my" and "var", and then = and "my var".

In short, always use double quotes.

Spaces Matter

If you don’t have space before and after the = symbol, the expression is treated as one word, and then it’s treated as true.

Remember that there must be a space between the [ and the variable name and the equality operator, = or ==. If you miss any of the spaces here, you may see an error like ‘unary operator expected’ or missing `]’.

# correct, a space before and after = sign
if [ $VAR2 = 1 ]; then
    echo "\$VAR2 is 1."
else
    echo "It's not 1."
fi# wrong, no space before and after = sign
if [ $VAR2=1 ]; then
    echo "$VAR2 is 1."
else
    echo "It's not 1."
fi

Test file expressions

The table below shows expressions for testing files.

You can find all of the options by using man test.

The test command returns an exit status of 0 when the expression is true and status of 1 when the expression is false.

Test string expressions

The table below shows expressions for testing strings.

#!/bin/bashSTRING=""if [ -z "$STRING" ]; then
  echo "There is no string." >&2 
  exit 1
fi# Output
# There is no string.

>&2 redirects the error message to the standard error: Please read Exit Status for exit 1.

Test integer expressions

The table below shows expressions for testing integers. Use the left side of expressions for POSIX-compliant and the right side for Bash.

Double parentheses

You can use the double parentheses, ((... )), for arithmetic expansion and evaluation.

How To Use if/else and if/elif/else Statements

The following are examples of if...else and if...elif...else statements.

The if/else statement has the following structure:

if [ condition-is-true ]
then
  command A
else
  command B
fi
# or
if [ condition-is-true ]; then
  command A
else
  command B
fi

Example:

The if/elif/else statement has the following structure:

if [ condition-is-true ]
then
  command A
elif [ condition-is-true ]
then
  command B
else
  command C
fi
# or
if [ condition-is-true ]; then
  command A
elif [ condition-is-true ]; then
  command B
else
  command C
fi

Example:

Double Brackets

If a variable is a single word, the double-bracket conditional-compound command [[ condition-for-test ]] doesn’t require a double-quote; but a single bracket, [ condition-for-test ], requires a double-quote for a variable. But it’s a good practice to use a double-quote.

The double bracket has additional functions compared to the single bracket. You can use logical && and || and =~ for regex .

How To Use for Loops

A for loop is used for iterating over a sequence, and it has the following structure:

for VARIABLE_NAME in ITEM_1 ITEM_N
do
  command A
done

Example:

You can use a variable as follows:

Using a for loop to rename files

#!/bin/bashIMGS=$(ls *png)
DATE=$(date +%F)for IMGin $IMGS
do
  echo "Renaming ${IMG} to ${DATE}-${IMG}"
  mv ${IMG} ${DATE}-${IMG}
done

The + sign in +%F signals a user-defined format, and %F shows the full date. For more details about date, use man date.

How To Use Parameters

A Bash script can take multiple arguments as follows:

$ scriptname param1 param2 param3

param1 to param3 are called positional parameters. You can use $0, $1, $2, etc. (often we use ${1}, ${2}, etc) to output parameters. For example:

#!/bin/bashecho "'\$0' is $0"
echo "'\$1' is $1"
echo "'\$2' is $2"
echo "'\$3' is $3"

Outputs:

$ ex1 param1 param2 param3
'$0' is /Users/shinokada/bin/ex1
'$1' is param1
'$2' is param2
'$3' is param3

The $0 outputs the name of the file, including the path.

To access all of the parameters, you can use $@.

#!/bin/bashfor PET in $@
do
  echo "My pet is: $PET"
done

Using this script:

$ ex1 cat dog gorilla
My pet is: cat
My pet is: dog
My pet is: gorilla

How To Accept User Input

User inputs are called STDIN. You can use the read command with the -p (prompt) option to read the user input. It’ll output the prompt strings. The -r option doesn’t allow backslashes to escape any characters.

read -rp "PROMPT" VARIABLE

Example:

#!/bin/bashread -rp "Enter your programming languages: " PROGRAMMESecho "Your programming languages are: "
for PROGRAMME in $PROGRAMMES; do
    echo "$PROGRAMME "
done

Running the script:

$ ex1                     
Enter your programming languages: python js rust 
Your programming languages are: 
python 
js 
rust

Brace Expansion Using Ranges

Brace expansion using ranges is a sequence expression. You can use it with integers and characters.

$ echo {0..3}
$ echo {a..d}
# output: 
# 0 1 2 3
# a b c d

You can use this brace expansion in for loops:

#!/bin/bashfor i in {0..9}; 
do 
  touch file_"$i".txt; 
done

This will create different file names with different modification times.

$ ls -al
-rw-r--r--    1 shinokada  staff      0 Jan 27 08:25 file_0.txt
-rw-r--r--    1 shinokada  staff      0 Jan 27 08:25 file_1.txt
-rw-r--r--    1 shinokada  staff      0 Jan 27 08:25 file_2.txt
-rw-r--r--    1 shinokada  staff      0 Jan 27 08:25 file_3.txt
-rw-r--r--    1 shinokada  staff      0 Jan 27 08:25 file_4.txt
...

How To Use While Loops

While an expression is true, the while loop keeps executing lines of code.

#!/bin/bashi=1
while [ $i -le 5 ]; do
  echo $i
  ((i++))
done

Output:

1
2
3
4
5

What Are Exit Status/Return Codes?

Every command returns an exit status ranging from 0-255. 0 stands for success, and anything other than 0 stands for an error. We can use this for error checking.

Checking the exit status

$? contains the return code of the previously executed command:

$ ls ./no/exist
ls: cannot access './no/exist': No such file or directory
$ echo "$?"
2

Let’s use the exit code in an if statement:

-c 1 stops after sending one request packet. Here we check if the exit status is equal to 0.

Output:

$ ex1
PING google.com (216.58.197.206): 56 data bytes
64 bytes from 216.58.197.206: icmp_seq=0 ttl=114 time=11.045 ms--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 11.045/11.045/11.045/0.000 ms
google.com reachable.

How To Connect Commands

Logical operators and the command exit status

A command outputs an exit status, and we can use && and || to decide the next actions.

Exit command

You can explicitly define the return code using exit.

exit 0
exit 1
exit 2
etc.

Example:

#!/bin/bashHOST="google.com"
ping -c 1 $HOST  
if [ "$?"" -ne "0" ]
then
  echo "$HOST unreachable."
  exit 1
fi
exit 0

Since the script explicitly returns an exit status code, you can call another shell script or command.

$ ex1 && ls          
...
2020-10-03-image-1.png  ex1     myscript1
2020-10-03-image-2.png  ifstatement  
2020-10-03-image-3.png  logfile

Logical AND (&&)

The second command runs if the first command returns the 0 status code, which means it succeeded.

mkdir /tmp/bak && cp test.txt /tmp/bak

Logical OR (||)

The second command runs if the first command returns non0 status code, which means it failed.

cp test.txt /tmp/bak/ || cp test.test.txt /tmp

Examples

If the ping command succeeds the echo statement will run.

#!/bin/bashhost="google.com"
ping -c 1 $host && echo "You can reach ${host}."

If the ping command fails, the echo statement runs.

#!/bin/bashhost="google.com"
ping -c 1 $host || echo "You can't reach ${host}."

The semicolon

A semicolon is not a logical operator, but you can use it to separate commands to ensure they all get executed.

cp text.txt /tmp/bak/ ; cp test.txt /tmp
# This is the same as
cp text.txt /tmp/bak/  
cp test.txt /tmp

Pipe |

The commands on both sides of | run in respective subshells and both start at the same time.

The first command changes the directory to the home directory and lists the files and directories.

The second command just prints the directory where the command was executed.

$ echo "$(cd ~ && ls)"
$ echo "$(cd ~ | ls)"

Functions

In a Bash script, you must define a function before you use it. You can create a function with or without a function keyword.

function function-name(){}
# or
function-name(){}

When you call a function, use only the function name without ().

Functions can call other functions.

The now function is declared before calling the hello function. If you define the now function after calling hello function, it won’t work.

#!/bin/bash
# this won't work
function hello(){
  echo "Hello!"
  now
}
hello
function now(){
  echo "It's $(date +%r)"
}

The date +%r outputs in a format of 05:04:12 PM.

Function parameters

As we discussed in the section on script parameters, $0 isn’t a function name but is the script itself. $1 is the first parameter of a function.

#!/bin/bash
function fullname(){
  echo "$0"
  echo "My name is $1 $2"
}
fullname John Doe# Output
# /Users/shinokada/bin/ex1
# My name is John Doe

$@ contains all of the parameters.

Variable Scope

By default, variables are global, and you must define them before using them. It’s a good practice to declare all of the global variables at the top.

Local variables

You can access local variables within the function using the local keyword. Only functions can have local variables.

Function Exit Status

You can return the exit status explicitly in a function:

return 0

The exit status of the last command executed in the function is implicitly returned. The valid codes range from 0-255. 0 represents success and $? shows the exit status.

$ my_function
$ echo $?
0

You can use $? in an if statement:

The above script backs up /etc/hosts as the default, if there’s no argument. If you give an argument, it’ll check if it’s a file. If it’s a file, it’ll create a copy in the /tmp directory.

$$ is the process ID (PID) of the script itself. The PID is different each time you run the script. So if you’re running the script more than once a day, it’s useful.

basename ${1} returns the file name from your argument. For example, the basename of /etc/hosts is hosts.

An example of date +%F is 2020-10-04.

$ ls /tmp$ ex1           
Backing up /etc/hosts to /tmp/hosts.2020-10-04.77124
Backup succeeded.
$ ls /tmp
hosts.2020-10-04.77124

exit and return keywords

return will cause the current function to go out of scope, while exit will cause the script to end at the point where it’s called.

Checklist

• Start with a shebang.
• Describe the purpose of the script
• Declare all of the global variables first
• Declare all of the functions after the global variables
• Use local variables in functions.
• Write the main body after the functions
• Use an explicit exit status code in your functions, the if statement, and at the end of the script

Conclusion

We covered settings, variables, tests, if statements, double brackets, for loops, parameters, user inputs, while loops, exit statuses, logical operators, functions, and variable scopes.

I hope this article gives you the foundation for the next step in learning Bash scripting. Happy coding, everyone.

Get full access to every story on Medium by becoming a member.

References

1. Variables, Internal variables
2. shellcheck
3. shellcheck wiki
4. shell-format
5. The Bash Hackers Wiki
6. Advanced Bash-Scripting Guide