When being root is not enough

The other day, when I logged into my server, I realised that I hadn’t upgraded it in a while. Fedora 37 had just been released, so it was a good time for an upgrade. The process went smoothly, but I noticed after rebooting that the Caddy service failed to start. At first I thought that maybe something in the config was no longer compatible with the new version of Caddy. Everything checked out fine. When I manually executed the command in the caddy.service file then the service would start up without any issues.

That was strange. The Caddyfile is owned by root, the caddy.service file is owned by root, and the service is executed and run as root. So it wasn’t a permission issue.

Let’s see what systemd has to say for itself.

I ran the command journalctl -u caddy --no-pager | less, and that is when I saw this:

caddy[736]: run: reading config file: open /the/path/to/Caddyfile: permission denied

So it was a permission issue after all. But how? Is being root not enough?

I furiously started searching the web and eventually stumbled upon this StackOverflow post, which suggested that SELinux could cause this type of behaviour.

I temporarily disabled SELinux with sudo setenforce Permissive to see if it would fix my issue ‒ and it did. Cool, but why would SELinux cause the issue in the first place? What didn’t it like about my caddy.service file?

To find out, I started with the semanage fcontext --list command. This is when I realised that I was storing my Caddyfile in a directory that SELinux didn’t approve of. So my final solution was just to move it to a directory that SELinux was happy with. The last thing I needed to do was update the caddy.service file to point to the new location of the Caddyfile and restart the service.