A varnishncsa logfile per host

Varnish is a caching proxy server in quite widespread use. Its common role is to set in between your load balancer and web servers to lessen the requests ending up at the latter.

A often recurring situation is that your Varnish instance serves multiple different hostnames. If you want to use some of the older tools such as Webalizer or awstats to process your logs you probably need a log file per hostname.

For this we can use varnishncsa a utility that comes with Varnish to retrieve logfiles. This post was written with Ubuntu 18.04 as a base system but should work on your system even if you might have to change some paths.

Normally a varnishncsa.service file is created for systemd to run a default instance of it, these logs end up in /var/log/varnish. We're going to add our own parametrized service files that filter out specific hostnames.

Note that you will end up running multiple instances of varnishncsa, this hasn't been a problem for me as they are quite lightweight.

Let's get started.

Setup

Start by creating a new service file in /lib/systemd/system under the name varnishncsa-per-host@.service.

# cat /lib/systemd/system/varnishncsa-per-host\@.service
[Unit]
Description=Varnish HTTP accelerator log daemon for Host %I
Documentation=https://supakeen.com/weblog/varnishncsa-per-host.html
After=varnish.service

[Service]
Type=forking
PIDFile=/run/varnishncsa/varnishncsa-%I.pid
RuntimeDirectory=varnishncsa
User=varnishlog
Group=varnish
ExecStart=/usr/bin/varnishncsa -a -q "ReqHeader ~ '^Host: %I'" -w /var/log/varnish/varnishncsa-%I.log -D -P /run/varnishncsa/varnishncsa-%I.pid
ExecReload=/bin/kill -HUP $MAINPID
PrivateDevices=true
PrivateNetwork=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=full

[Install]
WantedBy=multi-user.target

In this file you see multiple %I format specifiers. These will be replaced by systemd. We are using varnishncsa's filtering on the Host header to find only the host we're interested in.

After we've created this file we can now create symlinks to enable this service for a hostname. Repeat this step per Host header you want to filter out.

# ln -s /lib/systemd/system/varnishncsa-per-host\@.service ./varnishncsa-per-host@supakeen.com.service
# systemctl daemon-reload
# systemctl enable varnishncsa-per-host@supakeen.com.service
# systemctl start varnishncsa-per-host@supakeen.com.service
# systemctl status varnishncsa-per-host@supakeen.com.service
● varnishncsa-per-host@supakeen.com.service - Varnish HTTP accelerator log daemon for Host supakeen.com
   Loaded: loaded (/lib/systemd/system/varnishncsa-per-host@.service; indirect; vendor preset: enabled)
   Active: active (running) since Sat 2020-04-25 09:26:34 UTC; 18min ago
     Docs: https://supakeen.com/weblog/varnishncsa-per-host.html
 Main PID: 20822 (varnishncsa)
    Tasks: 1 (limit: 1151)
   CGroup: /system.slice/system-varnishncsa\x2dper\x2dhost.slice/varnishncsa-per-host@supakeen.com.service
           └─20822 /usr/bin/varnishncsa -a -q ReqHeader ~ '^Host: supakeen.com' -w /var/log/varnish/varnishncsa-supakeen.com.log -D -P /run/varnishncsa/varnishncsa-supakeen.com.pid

Apr 25 09:26:34 var.tty.cat systemd[1]: Starting Varnish HTTP accelerator log daemon for Host supakeen.com...
Apr 25 09:26:34 var.tty.cat systemd[1]: varnishncsa-per-host@supakeen.com.service: Failed to parse PID from file /run/varnishncsa/varnishncsa-supakeen.com.pid: Invalid argument
Apr 25 09:26:34 var.tty.cat systemd[1]: Started Varnish HTTP accelerator log daemon for Host supakeen.com.
root@var:~#

And a new file now shows up in /var/log/varnish

# stat /var/log/varnish/varnishncsa-supakeen.com.log
  File: /var/log/varnish/varnishncsa-supakeen.com.log
  Size: 1946            Blocks: 8          IO Block: 4096   regular file
Device: fc01h/64513d    Inode: 260830      Links: 1
Access: (0644/-rw-r--r--)  Uid: (  113/varnishlog)   Gid: (  115/ varnish)
Access: 2020-04-25 09:26:34.904152413 +0000
Modify: 2020-04-25 09:34:43.891850255 +0000
Change: 2020-04-25 09:34:43.891850255 +0000
 Birth: -

That's it! You can now use this file any way you want. Good luck.