Make sure that users' changes to the config files are preserved.

Fixes #4440.

Rather than hardcoding a config which we always forget to update, generate it
from the default config.