Twenty Questions

This is a really basic idea, but let’s say you’re building a service that has lots of knobs you can turn and switches you can flip to control features. Is it better to throw these into a configuration file (say using Jsonnet), or expose them as command line flags? Why not both?

A proposal

The fountain of wisdom known as Stack Overflow has a similar question. Using that as a starting point, but with some of my own thoughts, I’d propose:

• Use environment variables sparingly for things that don’t change much.
• Use command line flags for most simple things you want to specify when starting your app.
• Use config files for larger parameters or things that don’t change often. Values passed from the command line (and perhaps env vars) should override values in a config file.

So in reality, you probably end up using a mix of all three: put defaults and bigger bits into a config file, pass frequently changed values on the command-line, and perhaps use a few environment variables to configure variables during runtime. Even if you start heavily in one direction, I bet you’ll see that pattern emerge over time; however, make sure to do it in the least surprising way. Use a consistent pattern, and even delegate managing configuration to a class. This way, you can evolve to a point where your configuration class loads info from a centrally managed service, allowing your app to be more dynamic.

While I was working at a big tech company, it was actually pretty common to end up with builds that specified lots of flags. But when it came time to run jobs in production, you would specify the job in a configuration language that passed explicit flag values while invoking the binary. There were even times where we kept configuration files in source control, and a low-latency system pushed freshly submitted changes to listeners in our service in an effort to make updates more dynamic.

Inspecting configuration at run time

If you’re building a long-lived service, then it’s probably useful to expose some endpoints with debug information that include configuration values (of course, inaccessible outside your network). This way, you can ensure that everything is working the way you think it is. Tangentially, you could probably use similar pages to feed observability metrics into a system like Prometheus.

Distinguishing configuration and state

I have also noticed instances where state and configuration were mixed together in the same “configuration” files; however, I believe it’s important to distinguish the two:

• Configuration is the set of knobs and switches you can turn to change the behavior of your application.

• State is data that your application creates, reads, updates and deletes as part of its purpose.

Why might you store state in configuration-like files? Well, sometimes you just have tidbits of data that are small in size and static in nature. Managing them in a database would introduce too much overhead, especially when the could live right in memory alongside the application. So, you throw them into version control alongside the application. Imagine things like lists of countries or zip codes.

The line between configuration and state can get blurry because you could argue that an application might change behavior based on some initial state. But as a general rule, I believe the above is a good distinction.

Recap

This is a dry topic, but doing it right can contribute to more robust systems. The two take-aways are:

1. Delegate configuration to a helper that loads config in a non-surprising way using command line flags and config files, with sparing usage of environment variables. This well help you down the road.
2. Distinguish configuration and state.