SearchSELinux stands for Security-Enhanced Linux. It’s a Linux Security Module (LSM) that enables permissions and restrictions with a much finer granularity than traditional Unix permissions.
In short, SELinux works by labeling resources with types and establishing
permissions for certain operations between types. For example, a process
running the program /usr/sbin/varnishd has the type varnishd_t and the
files inside the /etc/varnish directory have the type varnishd_etc_t.
There exists a rule to allow varnishd_t to read varnishd_etc_t files, for
example /etc/varnish/default.vcl. SELinux also has its own flavor of users
and roles and comes with a significant learning curve.
Once the varnish service is started, you should be able to observe two
processes:
$ ps -C varnishd,cache-main -o comm,label
COMMAND LABEL
varnishd system_u:system_r:varnishd_t:s0
cache-main system_u:system_r:varnishd_t:s0
The label gives us the security context for /usr/sbin/varnishd:
system_u is the built-in system user meant for system servicessystem_r is the built-in system rolevarnishd_t is a types0 is a rangeThe important part in this case is that varnishd is labeled with the
varnishd_t type. Even though varnishd is executed as the root user,
SELinux will deny any operation that a root user could normally perform
if that operation is not allowed for the varnishd_t type.
It used to be almost systematic to solve a permission problem by disabling SELinux and you should avoid that. What SELinux solves is the loophole of the root user: in theory the root user (UID 0) is the privileged user and can do anything. For instance, port numbers below 1024 are privileged ports, so in order to simply run a standard web server on ports 443 and 80 you need root privileges.
Varnish was designed with security in mind, and the workload is split in two processes:
With its jail facility Varnish will permanently drop privileges in the cache process, and temporarily drop them in the management process to only reacquire privileges with the smallest possible window to perform privileged operations.
While Varnish proactively tries to mitigate many attack vectors, what happens when a severe security vulnerability is discovered in any piece of software? If a remote code execution can be achieved on a process running as root then an attacker can do virtually anything. SELinux defends against that and will for example prevent Varnish from reading your mail server configuration if you happen to run both services on the same system. SELinux comes with a policy covering a large part of critical system software like Varnish and can greatly reduce the blast radius of certain classes of vulnerabilities.
You shouldn’t disable SELinux.
When SELinux is enabled it can essentially run in two modes:
In enforcing mode, anything denied by the policy will be blocked and recorded
in the system’s audit log. The application will likely see denied operations
fail with operation not permitted (EPERM), and how this error is handled is
up to the application.
In permissive mode, an unauthorized operation is only logged, not denied, which allows the application to keep making progress while informing system administrators that the policy would have denied certain operations.
Ideally, running a comprehensive workload in permissive mode should allow you to catch most if not all the things that would go wrong in enforcing mode.
You can then search the audit log with ausearch and even generate SELinux
policies with audit2allow. When working with Varnish Enterprise, the best way
to help us study SELinux denials is to send us a varnishgather report after
running your workload in permissive mode.
See varnishgather on GitHub.
You don’t have to jeopardize the security of your entire system by putting it in permissive mode. Instead, you can put a specific type from the policy in permissive mode:
semanage permissive -a varnishd_t # add to the list of permissive types
semanage permissive -d varnishd_t # delete permissive type
semanage permissive -l # list permissive types
As mentioned earlier, SELinux will prevent Varnish from reading your mail server configuration. This works by abstracting resources, such as paths, programs or port numbers with types, and then allowing certain types to perform specific operations on other types.
The SELinux policy contains several types related to Varnish, you may put in permissive mode the one that is denied operations according to the audit log.
It will very likely be one of the following:
varnishd_tvarnishlog_t (this includes varnishncsa)For example, varnishd_t is allowed to read configuration files labeled as
varnishd_etc_t. Some utilities like ls can take a -Z option to print the
security context of files and directories:
$ ls -Z /etc/varnish/*.vcl
unconfined_u:object_r:user_home_t:s0 /etc/varnish/custom.vcl
system_u:object_r:varnishd_etc_t:s0 /etc/varnish/default.vcl
If you observe files with the wrong security context, this can be corrected
with the restorecon utility:
$ restorecon -v -r /etc/varnish
Relabeled /etc/varnish/custom.vcl from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:varnishd_etc_t:s0
If your SELinux troubleshooting reveals something more complicated than an
unexpected label on a file, remember to contact our support if you are a
Varnish Enterprise customer. For the quickest possible resolution, include a
varnishgather report after running your workload with the problematic type
in permissive mode.
Varnish Enterprise supports SELinux on Red Hat Enterprise Linux (RHEL) and compatible systems. The policy is developed and maintained by Red Hat, and we at Varnish Software contribute fixes and enhancements to the policy when we update our downstream policy.
SELinux is modular, so there is a base policy and a set of modules on top of that. There is a varnishd semodule covering several components from Varnish Cache.
In broad strokes, the varnishd program is allowed to do the following:
varnishd_connect_any/var/lib/varnish tree/etc/varnishIt also allows varnishlog and varnishncsa to access Varnish’s shared
memory log and persist logs in /var/log/varnish.
On systems where we support SELinux we disable the varnishd semodule and add
our own varnish-plus semodule instead. We do this because Varnish Enterprise
has more features and some of them would otherwise be denied by default.
We could have added a complementary semodule and leave the varnishd one alone,
but we support both RHEL7 and RHEL8 and their policies target respectively
Varnish 4.0 and 6.0 between which there is a noticeable gap.
It was much simpler to maintain a single policy for both platforms. By
effectively forking the varnishd semodule we also took care of isolating
Enterprise-only permissions, which gives us a better understanding
of whether a change we are making is relevant for the upstream policy and
should be contributed back.
The one noticeable departure from the upstream policy is the split of the
varnishd_port_t in two:
varnishd_http_port_t (data plane)varnishd_cli_port_t (control plane)All VMODs are running inside varnishd, and are subject to the same SELinux
policy. Even our vmod_memcached would likely be denied a connection to a
Memcached server based on Memcached’s default port number.
There are several ways to deal with this. You could add the Memcached port to
the varnishd_http_port_t directly from a command line with the semanage
utility. Or you could create your own semodule to grant varnishd_t access to
memcache_port_t.
Unless you are proficient enough with SELinux to extend the policy, you could simply turn a built-in boolean on:
semanage boolean --modify --on varnishd_connect_any
When in doubt, contact Varnish support.
When we first introduced our downstream varnish-plus policy, it needed to disable an existing varnish SELinux module. Unfortunately, it didn’t handle all scenarios properly. In some cases, it could fail to install the varnish-plus SELinux module and allow the varnish-plus package installation to proceed.
Using off-the-shelf RPM macros to manage SELinux modules solved the original issue. This approach resulted in generally better integration, especially during upgrades, but it still wouldn’t prevent a broken package installation from proceeding. At that time we did not realize that the RPM macros use a different priority level than our initial integration.
The introduction of RHEL8 was a regression from the previous simplification.
Our former varnish-plus-ports SELinux module was installed as CIL code by the
RPM macros. This code would no longer compile on RHEL8, forcing us to manage
varnishd_http_port_t and varnishd_cli_port_t types manually.
Next, the introduction of Amazon Linux 2 brought more complications to the process. It advertises itself as RHEL7 in its RPM setup, but has outdated dependencies, such as SELinux. Because of this, we have had to entirely circumvent our SELinux policy on this platform.
As a result, a series of bug fixes have been implemented to remediate all known failure scenarios and prevent a package installation when a failure would occur. This could result in the following error messages:
Error: could not disable the varnishd SELinux module.
error: %pre(varnish-plus-selinux-X.Y.ZrT-1.elV.noarch) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm
This can occur when there is a dependency to the varnishd SELinux module in
the system. For example, when the varnishd_t type is set to be permissive,
which under the hood creates a permissive_varnishd_t SELinux module.
Or if in order to solve a denial, for example to allow monitoring software to access something owned by Varnish, applying a policy update that was suggested by the audit logs tooling can also take the form of an SELinux module.
In those scenarios you can temporarily disable the offending module:
$ semodule -d permissive_varnishd_t
# perform package update
$ semodule -e permissive_varnishd_t
This should be a one-time workaround, because once the varnishd SELinux
module is properly replaced by our own varnish-plus policy, upgrades can
happen safely even when a module depends on it.
When in doubt, contact our support.
One frequently recurring mistake is to copy files via SSH and move them to the
/etc/varnish directory. Whether it is done manually or through automation,
it should be noted that when a file is effectively moved, it will keep the
security context it inherited when it was created.
So it is common that VCL files in /etc/varnish end up with the user_home_t
type. You can fix this as part of your deployment by adding a restorecon
step or by updating the security context during the move:
$ ls -Z custom.vcl
unconfined_u:object_r:user_home_t:s0 custom.vcl
$ mv -v -Z custom.vcl /etc/varnish
copied 'custom.vcl' -> '/etc/varnish/custom.vcl'
removed 'custom.vcl'
$ ls -Z /etc/varnish/custom.vcl
unconfined_u:object_r:varnishd_etc_t:s0 /etc/varnish/custom.vcl
There are other utilities like ps that can take a -Z option, not just ls
and mv.
Our packaging comes with a logrotate configuration, and the idea is to always
append access logs and rotate them to avoid overwriting and losing logs on
disk as we are building them up. The lack of arbitrary write permission comes
from the upstream varnishd semodule and it looks sensible enough that we
decided to keep it that way.