Custom Bash Builtins in C: Lessons From awie-bash

Most posts in the PHP-style function library on this site are pure Bash: parameter expansion and [[ ]], no external commands. Years ago I also wrote some of the same helpers in C, as builtins compiled directly into Bash. They lived in a folder called awie-bash, part of an old project of mine that I pushed to GitHub in 2013 "for history".

Here is how custom builtins work, the two ways to ship one, what I got wrong back then, and a tested modern port with timings.

What a builtin is, and why it is faster

A builtin is a command implemented inside the shell process. When you run ls, Bash calls fork() to copy itself, then the child calls exec() to load /usr/bin/ls. A builtin skips both: Bash looks the name up in its table and calls a C function directly.

$ type -a printf
printf is a shell builtin
printf is /usr/bin/printf
$ type -t ls
file
$ enable -f ./old_strstr.so strstr
$ type -t strstr
builtin

A shell function also runs without a fork, which is why the pure-Bash strstr(), trim() and emptydir() are already fast. A C builtin goes further: no interpreted code, and direct access to system calls such as opendir() that the shell language cannot reach.

Anatomy of a .def file

Bash's own builtins (cd, echo, read) are written as .def files in the builtins/ directory. A small program in the source tree, mkbuiltins, turns each one into a .c file and collects the builtin names and help text into the shell's table. Here is my strstr.def in full:

This file is strstr.def, from which is created strstr.c.
It implements the builtin "strstr" in Bash.
$PRODUCES strstr.c
#include <config.h>
#include <stdio.h>
#include <string.h>
#include "../builtins.h"
#include "../shell.h"
#include "bashgetopt.h"
$BUILTIN strstr
$FUNCTION strstr_builtin
$SHORT_DOC strstr string substring
Shell version of strstr(3).  Returns success (0) or failure (1).
$END
int strstr_builtin(WORD_LIST *list) {
	if(no_options(list)) return(EX_BADUSAGE);
	list=loptend;
	if(list_length(list)!=2) {
		builtin_usage();
		return(EX_BADUSAGE);
	}
	if(strstr (list->word->word, (list->next)->word->word)) {
		return(EXECUTION_SUCCESS);
	}
	return(EXECUTION_FAILURE);
}

• $PRODUCES names the generated C file. The C code outside the $BUILTIN blocks is copied into it.
• $BUILTIN is the name you type. $FUNCTION is the C function behind it. Two $BUILTIN blocks may share one function: strcasestr.def registers both strcasestr and stristr that way.
• $SHORT_DOC is the usage line printed by builtin_usage() and help -s. The lines up to $END become the long text for help strstr.

The function receives a WORD_LIST, a linked list of the expanded arguments without the command name: list->word->word is the first one as a C string, list->next the rest.

no_options() runs Bash's internal getopt with no valid options. It prints an error and returns non-zero for anything that looks like an option. It shows the help text for --help. It also leaves the global loptend pointing at the first real argument, past any --.

The return value is the exit status: EXECUTION_SUCCESS (0), EXECUTION_FAILURE (1) or EX_BADUSAGE (2). EX_USAGE is an internal 258. Returning it gave $? = 2 in my tests.

Helpers sit above the $BUILTIN block as ordinary static C. From empty_dir.def:

static int check_empty(char *dirname) {
        DIR *dir;
        struct dirent *entry;
        int check=0;
        if((dir = opendir(dirname)) == NULL) return(check);
        for(;;) {
                if((entry = readdir(dir)) == NULL) {
                        closedir(dir);
                        return(check);
                }
                if(!strcmp(entry->d_name,".") || !strcmp(entry->d_name,"..")) continue;
                check=1;
                return(check);
        }
        return(check);
}

It has two bugs, described below. To wrap an existing main(), make_builtin_argv() converts the list into a classic argv, which is how ttysize.def reuses a small busybox applet:

int ttysize_builtin(WORD_LIST *list) {
	char **v;
	int c, r;
	v=make_builtin_argv(list, &c);
	r=ttysize_main(c, v);
	FREE(v);
	return r;
}

Shipping option 1: compile it into Bash

This is what awie-bash did. I copied the 22 .def files into builtins/, and the modified builtins/Makefile.in lists them next to the standard ones:

DEFSRC =  $(srcdir)/alias.def $(srcdir)/bind.def $(srcdir)/break.def \
	  ...
	  $(srcdir)/printf.def $(srcdir)/complete.def \
	  $(srcdir)/basename.def $(srcdir)/dirname.def $(srcdir)/sleep.def $(srcdir)/sync.def \
	  $(srcdir)/usleep.def $(srcdir)/iocat.def $(srcdir)/unlink.def $(srcdir)/remove.def \
	  $(srcdir)/empty_dir.def $(srcdir)/xmkdir.def $(srcdir)/crypt.def $(srcdir)/uname.def \
	  $(srcdir)/strftime.def $(srcdir)/strstr.def $(srcdir)/strcasestr.def $(srcdir)/strtrim.def \
	  ...
OFILES = builtins.o \
	...
	basename.o dirname.o sleep.o sync.o usleep.o iocat.o \
	unlink.o remove.o empty_dir.o xmkdir.o crypt.o uname.o strftime.o strstr.o strcasestr.o \
	strtrim.o uccase.o lccase.o tty.o consoletype.o symlink.o ttysize.o

Further down are dependency lines such as strstr.o: strstr.def. The existing .def.o suffix rule runs mkbuiltins and the compiler, so no new build logic is needed. The result is a bash where strstr is as built-in as echo, with no setup in scripts. The cost is a private fork of the shell, which you must rebuild for every upstream fix. The Makefile lists no mapfile.def (new in Bash 4.0), so it came from an older tree. Some of the code also no longer compiles. sleep.def fails against 4.4 with "conflicting types for 'fsleep'", because 4.4 declares its own.

Shipping option 2: a loadable builtin

Bash can also load a builtin from a shared object at run time with enable -f, which uses dlopen(). The source tree's examples/loadables/ has samples, starting with hello.c. Instead of $ directives you define one structure:

struct builtin hello_struct = {
	"hello",		/* builtin name */
	hello_builtin,		/* function implementing the builtin */
	BUILTIN_ENABLED,	/* initial flags for builtin */
	hello_doc,		/* array of long documentation strings. */
	"hello",		/* usage synopsis; becomes short_doc */
	0			/* reserved for internal use */
};

enable -f file.so NAME looks up the symbol NAME_struct. If you ask for the wrong name, you get "cannot find emptydir_struct in shared object". enable -d NAME removes it again. Bash 4.4 added two things. It calls NAME_builtin_load() and NAME_builtin_unload() if they exist. It also searches the colon-separated BASH_LOADABLES_PATH when the file name has no slash. The Bash manual's builtins page documents enable.

Reviewing my old code

I converted several of the old files to loadables in a mechanical way. I changed only the include paths and appended a struct builtin. Then I ran them in this machine's Bash 4.4.20. Some of the code still works well. In my tests strstr behaves like the [[ $1 == *"$2"* ]] shell version, including an empty needle counting as a match. The rest:

empty_dir leaks a directory handle. When it finds an entry, check_empty() returns without closedir(). Every "not empty" answer leaks a descriptor in your shell. Worse, an opendir() failure returns the same 0 as "empty", so a missing path, a plain file, and an exhausted descriptor table all report "empty":

$ enable -f ./old_empty_dir.so empty_dir
$ empty_dir missing; echo $?
0
$ ls /proc/$$/fd | wc -l
4
$ for i in {1..100}; do empty_dir full; done
$ ls /proc/$$/fd | wc -l
104

In a new shell with a low descriptor limit, the leak turns into a wrong answer:

$ ulimit -n 64
$ enable -f ./old_empty_dir.so empty_dir
$ for ((i = 1; i <= 100; i++)); do empty_dir full && break; done
$ echo "full looked empty on call $i"
full looked empty on call 62

A builtin lives as long as the shell. So leaks that do not matter in a short-lived command add up here.

Arguments starting with a dash are rejected. Calling no_options() in a string function means strtrim "-5 " fails with "-5: invalid option". strstr accepts -- because it moves to loptend. strtrim and empty_dir never do, so strtrim -- " hi " treats -- as the string and prints --. strtrim also prints without a trailing newline, unlike everything else in the set.

Copied help text. help strtolower says "convert letter to uppercase", and usleep's usage line reads sleep [number].

The case functions work on bytes and do not match PHP. They overwrite list->word->word in place. That turned out to be harmless, because Bash gives the builtin its own copy (strtolower "$s" left $s unchanged). But strtoupper "café" printed CAFé under en_GB.UTF-8. ucfirst "hELLO wORLD" printed Hello world, where PHP keeps HELLO wORLD. And lcfirst "123 ABC" lowercased the first letter rather than the first character.

No prototypes, and a crash on 64-bit. None of the files include common.h, where no_options(), builtin_usage() and make_builtin_argv() are declared. GCC 8 warns about implicit declarations, and for make_builtin_argv() that is fatal: the returned pointer is treated as a 32-bit int. On this x86_64 machine iocat segfaults the shell:

bash -c 'enable -f ./old_iocat.so iocat; iocat hello.txt'
echo "exit status: $?"

crash.sh: line 1: 284168 Segmentation fault      (core dumped) bash -c 'enable -f ./old_iocat.so iocat; iocat hello.txt'
exit status: 139

With common.h added, it prints the file. But iocat on a directory then hangs until it is killed. The read() call returns -1, and that value is passed to write(). That call also returns -1, and the "short write" check sees the two values as equal. A missing file is skipped silently.

Shadowing a real command. My builtin sleep took fractions but nothing else, so in that bash any script running sleep 1m gets "1m: bad sleep interval" where GNU sleep works. A builtin takes priority over $PATH, so reusing a command's name changes scripts you never wrote.

Also, from reading the code: xmkdir leaks three allocations per call, and unlink ignores errors and always returns success.

A modern port: empty_dir as a loadable

The port fixes the leak, reports errors instead of guessing, honours --, and distinguishes the three outcomes:

/* empty_dir -- loadable Bash builtin: succeed if a directory is empty.
 * Port of awie-bash/empty_dir.def. Exit status:
 * 0 empty, 1 not empty, 2 usage error or directory cannot be read.
 */
#include <config.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <dirent.h>
#include "loadables.h"
static int
dir_is_empty (const char *path)
{
  DIR *dir;
  struct dirent *e;
  int empty = 1;
  if ((dir = opendir (path)) == NULL)
    return -1;
  while ((e = readdir (dir)) != NULL)
    {
      if (strcmp (e->d_name, ".") == 0 || strcmp (e->d_name, "..") == 0)
        continue;
      empty = 0;
      break;
    }
  closedir (dir);               /* the old version leaked this */
  return empty;
}
int
empty_dir_builtin (WORD_LIST *list)
{
  int r;
  if (no_options (list))
    return EX_USAGE;
  list = loptend;               /* skip a leading "--" */
  if (list == 0 || list->next)
    {
      builtin_usage ();
      return EX_USAGE;
    }
  r = dir_is_empty (list->word->word);
  if (r < 0)
    {
      builtin_error ("%s: %s", list->word->word, strerror (errno));
      return 2;
    }
  return r ? EXECUTION_SUCCESS : EXECUTION_FAILURE;
}
char *empty_dir_doc[] = {
  "Test whether a directory is empty.",
  "",
  "Exit status is 0 if DIRECTORY has no entries other than . and ..,",
  "1 if it has entries, and 2 if it cannot be opened.",
  (char *)NULL
};
struct builtin empty_dir_struct = {
  "empty_dir",                  /* name used with enable -f */
  empty_dir_builtin,            /* the C function */
  BUILTIN_ENABLED,              /* initial flags */
  empty_dir_doc,                /* help text */
  "empty_dir [--] directory",   /* usage line */
  0                             /* reserved */
};

Building it

You need the headers of a configured Bash tree. With no development package installed, I used the 4.4.18 source, close to the system's 4.4.20, configured in a temporary directory:

curl -LO https://ftp.gnu.org/gnu/bash/bash-4.4.18.tar.gz
tar xzf bash-4.4.18.tar.gz
cd bash-4.4.18
./configure --without-bash-malloc
make pathnames.h          # shell.h needs this generated header
B=$PWD
cd /path/to/your/builtin
gcc -fPIC -DHAVE_CONFIG_H -DSHELL -O2 -Wall \
    -I"$B" -I"$B/include" -I"$B/lib" -I"$B/builtins" \
    -I"$B/examples/loadables" -c empty_dir.c
gcc -shared -o empty_dir.so empty_dir.o

Note: configure the tree in the same way as the bash you load it into. My first attempt used the default bash malloc. A build that frees memory then failed in Red Hat's /bin/bash with "undefined symbol: sh_xfree", because that bash is built without it. --without-bash-malloc fixed it.

$ enable -f ./empty_dir.so empty_dir
$ type empty_dir
empty_dir is a shell builtin
$ empty_dir empty; echo $?
0
$ empty_dir full; echo $?
1
$ empty_dir missing; echo $?
bash: empty_dir: missing: No such file or directory
2
$ empty_dir full/notes.txt; echo $?
bash: empty_dir: full/notes.txt: Not a directory
2
$ empty_dir; echo $?
empty_dir: usage: empty_dir [--] directory
2
$ enable -d empty_dir
$ type empty_dir
bash: type: empty_dir: not found

A loop of 2,000 calls on a non-empty directory left the shell with 4 open descriptors, the same number it started with.

Timing

I measured on one machine (Intel i7-8565U laptop, Bash 4.4.20), averaged over 100 to 10,000 calls each. I compared the builtin, the pure-Bash emptydir() from the earlier post, and [ -z "$(ls -A -- "$dir")" ]. For strstr I compared the old builtin, the [[ ]] function and printf | grep -qF.

A repeat run gave the same ordering. The external column is mostly fork-and-exec, about 0.8 ms a call here. For strstr the builtin only cuts an already tiny time in half, so the shell function remains the sensible choice. The large directory is different: the glob in emptydir() reads and sorts all 10,000 names into an array, while the builtin stops at the first entry.

When it is worth it

Write a builtin when a loop that runs very often needs a system call the shell cannot make. Also write one when the shell version processes far more data than the answer needs. Make it loadable rather than forking Bash, include loadables.h, compile with -Wall, close what you open, and avoid names of real commands. For string helpers, the pure-Bash library is fast enough.

Related reading

• emptydir(): check whether a directory is empty in pure Bash
• strstr(): substring test in Bash
• trim(): strip whitespace in Bash