Improve systemd configuration documentation #28453
Description
I must say that Spring Boot documentation regarding application installation under Linux for sysvinit system is well made and I could get it up and running quite fast, so good job there!
Unfortunately I also have to say that the same is not valid for the systemd part. I'll try to describe my today experience to help you improve that.
Just a note: seeking the Internet (mainly StackOverflow and GitHub) the feeling is that you don't want to go too deep with systemd integration because of little knowledge. This is understandable, however IMHO as far as I know most of (not-so) recent versions the main Linux distributions are using systemd now (Debian, Ubuntu, CentOS, Fedore, openSUSE, RHEL...) so if I had to chose where to put more effort in, I would opt for systemd today.
This said...
The documentation at:
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#deployment.installing.nix-services.script-customization
seems to suggest that the subsequent information applies to both sysvinit and systemd cases. In particular, there's no mention to the fact that the config file is completely ignored in the systemd case, unless the service script references it.
With systemd, Environment= lines should be added to the [Service] section of the service script to set up environment variables. Multiple lines could be given, for instance:
[Unit]
Description=myapp
After=syslog.target
[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
Environment=JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Environment=JAVA_OPTS=-Xmx128m
[Install]
WantedBy=multi-user.target
However, since I like to have myapp.conf along with my application, because it allows me to see and change environment variables right there, without having to mess with the systemd service file, I added an EnvironmentFile directive to my myapp.service:
[Unit]
Description=myapp
After=syslog.target
[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
EnvironmentFile=/var/myapp/myapp.conf
[Install]
WantedBy=multi-user.target
It would be useful to add a note, though, that myapp.conf must then be a pure environment file, not a script, so things like the following, which are commonly used by us and do work in the sysvinit case:
JAVA_OPTS=-Xmx64m
# add JMX options
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999"
do not work here, because $JAVA_OPTS is not expanded in the second assignment. If this is needed, then the above must be changed to:
[Unit]
Description=myapp
After=syslog.target
[Service]
User=myappuser
ExecStart=/bin/bash -ac '. /var/myapp/myapp.conf; exec /var/myapp/myapp.jar'
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
In this case, though, the guidelines for securing the config file mentioned in the docs (with no mention to the fact that we're still talking about sysvinit only) do not apply, because the config file must be readable by the myappuser user, so this should be taken into account. Instead, using EnvironmentFile seems to allow that file to be readable by root only, so it might be better from a security point of view. A much less powerful way to split environment variable declarations in this case could be the one mentioned here, which consists of terminating each line with \ and continuing on the next line, but of course it's not equally flexible and may lead to errors when you want to comment/uncomment some lines.
Please note that the same limitation of EnvironmentFile about variable expansion also applies to variable defined as Environment entries directly in the service script.
Last but not least, logging. There's a timid note in the documentation, just before this link, that says that LOG_FOLDER, and LOG_FILENAME are ignored in the systemd case, even if you get to properly set up the environment like the above. That note points the user to https://www.freedesktop.org/software/systemd/man/systemd.service.html, where however I was not able to locate any mention to how to properly redirect logs.
First of all, the documentation may say that, by default, logs with systemd are redirected to the journal, so issuing journald -u myapp would allow to see them. This said, the page at https://www.freedesktop.org/software/systemd/man/systemd.exec.html is more useful. I then found out that the best way to redirect both the standard output and the standard error to a log file is this:
- with systemd >= 240
[Unit]
Description=myapp
After=syslog.target
[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
StandardOutput=append:/var/myapp/logs/myapp.log
StandardError=inherit
[Install]
WantedBy=multi-user.target
- with systemd < 240 (Ubuntu 18.04, for instance), "append:" is not supported, so the best I could find is:
[Unit]
Description=myapp
After=syslog.target
[Service]
User=myappuser
ExecStart=/bin/sh -c 'exec /var/myapp/myapp.jar >>/var/myapp/logs/myapp.log 2>&1'
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
Of course it should be straightforward to change the above to redirect standard out and err to two distinct files.
/bin/sh may be /bin/bash, as well, so the above solutions to include the config file and redirect the log can be combined into:
ExecStart=/bin/bash -ac '. /var/myapp/myapp.conf; exec /var/myapp/myapp.jar >>/var/myapp/logs/myapp.log 2>&1'
I know this is a lot of information, perhaps also a bit out-of-scope. However, unless Spring Boot provides some way to improve the user experience with systemd out-of-the-box, I believe that the documentation should at least give the minimal information required to set up these basic configuration aspects: environment setup and logging.
Hope this is helpful.