Bash best practices

I am keeping example bash scripts that demonstrates the points I am making in this page in lessons-learned.

CLI guidelines

Valet is trying to follow the best practices for CLI applications. One good guideline can be found at clig.dev.

Coding style

TODO: copy some of the sections from https://google.github.io/styleguide/shellguide.html.

Calling external programs

Always use command to call external programs. This is to avoid any potential conflicts with functions or aliases that have the same name as the external program.

E.g. instead of mkdir -p /my/dir, we should write command mkdir -p /my/dir.

Naming conventions

Files

• All files should be named using kebab-case (lowercase with hyphens).
• All scripts should have the .sh extension.

Variables

• Use camelCase for local variable names. All variables should be created local to the function by default.
• Use upper SNAKE_CASE for global variables and constants.
  • If the variable is meant to be used outside of the script file, it should be prefixed with GLOBAL_.
  • Otherwise, it can be prefixed with _${SCRIPT_NAME}_ where SCRIPT_NAME is the filename of the script. This is to avoid conflicts with other scripts.

Functions

• Use camelCase for function names.
• For exposed functions (i.e. ‘public’ functions that are used outside of the script in which they are defined), prefix the function name with the script name. E.g. myLib::functionName (where this function is created in the file my-lib.sh).

Library functions

This section describes how to write library functions that can be used in Valet extensions. For functions that are solely use in one command, you don’t have to stick to these strict rules, but it is still a good idea to follow them.

Function parameters

Ideally, we would use positional arguments and flags like we do for our CLI. But it would be too costly to parse these in each function that we call.

After some testing, which can be found in the lessons-learned/passing-function-parameters.sh file, I found that the best way to pass parameters to a function is to use the shell parameters syntax.

While it is not the fastest way, it has the big advantage of making the function calls very readable, with options being passes with the shell parameter syntax option=value and positional arguments being passed as normal arguments.

Function calls would look like this:

normalFunction mandatoryArgument1 mandatoryArgument2 myOption=one myOption3="my value"
functionWithInfiniteArguments mandatoryArgument1 mandatoryArgument2 extraArg1 extraArg2 --- myOption=one myOption3="my value"

Notice that if the function accepts an undetermined number of arguments, we can use the --- separator to indicate that all the following parameters are options and not positional arguments.

The function core::parseFunctionOptions can be used to extract the options from the function arguments. For functions that accept a set number of arguments, the parsing can be done using eval "local a= ${*@Q}" (a= is there in case no options are passed).

A clear example of argument parsing implementation can be found in the coding-style test suite.

Function outputs

When it comes to outputs, we should use the REPLY global variable to store the output of our function. This way, we can easily access the output after the function has been called.

function myFunction() {
  local myOption="${myOption:-default_value}"
  # ...
  REPLY="some output"
}
_myOption="my_value" myFunction
echo "${REPLY}"

• We can use REPLY2, REPLY3, etc… if we need to return multiple values from a function.
• We can use REPLY_ARRAY, REPLY_ARRAY2, etc… to return an array of values from a function.
• We can use REPLY_CODE to return an exit code from a function.

As seen above, we should not use return to return an exit code because it would encourage a bad usage of the function. Instead, we should always return 0 from our functions and use the REPLY_CODE variable to return an exit code.

Function body

If possible, make each function independent and self-contained. This means that the function should not rely on other functions or global variables. A bit of copy-paste is acceptable if it makes the function self-contained. This will make the function easier to test and debug.

Explicitly declare all the parameters of the function at the top of the function body using local declare statements. This will make it easier to understand what parameters it expects and what are the default values. This extra cost is negligible.

Notable pitfalls

errexit bash option and ERR trap in bash

It is crucial to remember that any command executed in a until, while, if, or as part of a !, ||, && pipeline will not trigger the ERR trap if it fails! This behavior is described in the trap builtin documentation as well as in the set builtin documentation for the -e option.

This is also true for the body of a function or for command/functions that runs in this context where the ERR trap is not triggered.

When creating a complex function, do not use return 1 (or any exit code different than 0) because this would encourage a bad usage of the function (i.e. it would encourage if myfunction then...). Instead, use REPLY_CODE and simply return 0 from the function. Of course, for very simple functions, it is acceptable to use return 1 to simplify its usage.

See the lessons-learned/error-handling.sh for a demonstration.

Quote from bash manual: If a compound command or shell function executes in a context where -e is being ignored, none of the commands executed within the compound command or function body will be affected by the -e setting, even if -e is set and a command returns a failure status.

This is the reason why many bash users recommend to not use set -e in bash scripts (see bash pitfalls).

However, Valet has been designed with this pitfall in mind and all the library functions are written in a way that discourages users from using functions in a context where the ERR trap is not triggered.

Moreover, in my opinion, not using the errexit option will cause more issues because it allows the script to continue executing when it should have exited (e.g. after failing to change directory, which leads to run rm -f ./* in the wrong directory).

For these reasons, I think that the errexit option is relevant and that’s why it is enabled by default in Valet.

pipefail

Same as errexit, there are some pitfalls with the pipefail option which are well explained in the (see bash pitfalls) page.

However, contrary to errexit, the errors produced by this option are always defensive. Meaning that sometimes it will cause the script to exit early when it should not, but it will never allow the script to continue when it should have exited.

Moreover, since pipes create subshells, you should always avoid them in your scripts and they are never used in Valet itself.

This is why I think that this option is relevant and should be enabled by default in Valet.

nounset bash option

The bash option nounset will cause bash to exit if a variable is not set.

Not that an unset variable will NOT trigger the ERR trap. This is because the ERR trap is only triggered by commands that return a non-zero exit status, and an unset variable does not return a non-zero exit status.

Instead, it prints an error message to stderr and exits with a non-zero exit status.

However, this error will always be caught during your tests and should never happen during the execution of your scripts by end users (or this means that the tests were not exhaustive enough).

Thus, I believe that there are no good reasons to disable this option when writing new bash scripts.

Jobs and coproc

If the main process exits, the coprocs and jobs will not exit automatically: When we exit an interactive shell, a SIGHUP will be sent to the shell process which forwards it to all its children processes, including the coproc/jobs. But in the case of a script, we start a new bash process for the script which in turns starts processes for coproc/jobs. The SIGHUP is not sent automatically to the coproc/jobs when the script ends (it only happens at the end of an interactive shell session).

Coproc/jobs will no longer have the PPID of the bash script but will be assigned the PPID 1 which is the init process. This is actually a way to ‘daemonize’ a process (detaching it from the terminal). This technical is called ‘double fork’ and can also be replaced by using the builtin ‘disown’ command.

We must check if the main process is still running inside the coproc or we must kill all the coproc/jobs when the main script exits.

Traps and set options are inherited from the parent shell but the exit trap is not executed on error (at least not in bash 5.2 when writing this page)! If we re register the EXIT trap in the coproc/job, it will be executed on error. So it is better to re-register the EXIT trap in the coproc/job for consistent behavior.

Coproc behaves like a background job with the exception that they do not inherit the SIGINT trap. CTRL+C will interrupt both the main process and a background job, but not a coproc.

To make our life easier, Valet only uses coproc and never uses jobs. Coproc are as fast as jobs to start and they come with two way communication channels (stdin/stdout).

Aliases

It sounds like a good idea to use aliases in Valet to speed up the execution time by aliasing functions that should get bypassed in certain conditions.

For example, define alias log::debug=': #' log::trace=': #' to bypass the debug and trace logging functions when they are not needed.

However, aliases are expanded when a function definition is read, not when the function is executed. So if we do that, we would bypass the logging functions even if we unalias them later.