One of the conventions in the world of Unix-like OSes is that of the dotfile. This is a file with a period at the beginning of its name (like a file named .zshrc). There are two unique properties about them as far as the OS and the user are concerned; one with good results, the other not so much.

The first property is that by convention they’re used to store per-user application settings, and placed in the user’s home directory. .zshrc is one of mine and configures zsh, the shell I use. There are many others. When used for this purpose they’re generally named after the application they control in some way. They can be shared and their general structure as a plain text file makes it easy for users of software to share settings, tricks, etc. with each other. Many of the files are highly programmable – the shell control files allow arbitrary commands to be executed upon shell startup, for example. Some people even check their files into code repositories, and there are now many examples of dotfile configurations out there available for anyone to use.

The second property that makes the files unique (in most of the OSes in question) is that they’re generally considered invisible files by programs. What this means can vary somewhat, but it typically means software doesn’t process them or show information about them by default. ls, the command to list files in a directory, does not show them in its output unless you pass in the -a flag. Most command shells, when using *, the wildcard that matches all files in a directory, similarly ignores them.1 So what is the justification for doing this? As Unix luminary Rob Pike explained in a wonderful post, there really is none:

A lesson in shortcuts.

Long ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. I’m not sure but I believe .. went in during the Version 2 rewrite, when the file system became hierarchical (it had a very different structure early on). When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this: if (name[0] == ‘.’) continue; This statement was a little shorter than what it should have been, which is if (strcmp(name, “.”) == 0 || strcmp(name, “..”) == 0) continue; but hey, it was easy.

Two things resulted.

First, a bad precedent was set. A lot of other lazy programmers introduced bugs by making the same simplification. Actual files beginning with periods are often skipped when they should be counted.

Second, and much worse, the idea of a “hidden” or “dot” file was created. As a consequence, more lazy programmers started dropping files into everyone’s home directory. I don’t have all that much stuff installed on the machine I’m using to type this, but my home directory has about a hundred dot files and I don’t even know what most of them are or whether they’re still needed. Every file name evaluation that goes through my home directory is slowed down by this accumulated sludge.

I’m pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.

How many bugs and wasted CPU cycles and instances of human frustration (not to mention bad design) have resulted from that one small shortcut about 40 years ago?

Keep that in mind next time you want to cut a corner in your code.

(For those who object that dot files serve a purpose, I don’t dispute that but counter that it’s the files that serve the purpose, not the convention for their names. They could just as easily be in $HOME/cfg or $HOME/lib, which is what we did in Plan 9, which had no dot files. Lessons can be learned.)

Rob Pike

What justifications have others given for this supposed feature? I haven’t found much. Mostly the argument in favor of it is to protect novice users. Changing or removing the files can cause unexpected behavior, the argument goes, and the user often doesn’t know much about them – applications will often create a starter one without the user’s input – so it’s best to hide them by default. Perhaps that argument held water long ago, but certainly not in 2018. Let’s look at the usage of the Unix-like OSes now:

  1. OSes like Android and macOS which take great pains to hide Unixy things like dotfiles from the end user.
  2. Servers, where dotfiles may or may not be used at all depending on the setup but have no real novice users to protect.
  3. Techies like me running Linux or BSDs on PCs, where they are of critical importance and the users are aware of their power.

In none of those cases is there a use in keeping dotfiles hidden, and in the latter two it’s actively harmful. Processing dotfiles is often very important to system functionality. For example a naïve backup solution that decided to copy everything in a directory with the * wildcard would ignore the dotfiles, even though they are often the most important things to back up! And besides just controlling application settings, the files can serve many other important functions such as storing version control metadata (.git, .gitignore), controlling remote authentication into the system (.ssh), and even storing software installed by the user (.opam, .local). There’s really no reason not to see them when you’re working with files. Now there can be a lot of them – especially in your home directory – but that’s a reason to organize them better (like using the .config directory) instead of hiding them.

By the way, many OSes over the years – macOS among them – has used a much nicer solution for this. It added a new flag to the file flags attribute called the hidden flag. This is set on many system files and directories. Such marked files are hidden by default in the macOS GUI, but not the command line. This is a much cleaner separation that protects new users while not making things difficult for experienced users.

It’s far too late to get the dotfile behavior out entirely from Unix-likes, but it’s instructive to look back at the systems we use and see where they could be improved upon. That’s the only way we can build better systems in the future. The legacy of dotfiles provides lessons for making them, both good and bad.

  1. In zsh, putting setopt globdots in your .zshrc removes this misfeature.