A varnishncsa logfile per host
Varnish is a caching proxy server in quite widespread use. Its common role is to set in between your load balancer and web servers to lessen the requests ending up at the latter.
A often recurring situation is that your Varnish instance serves multiple different hostnames. If you want to use some of the older tools such as Webalizer or awstats to process your logs you probably need a log file per hostname.
For this we can use varnishncsa a utility that comes with Varnish to retrieve logfiles. This post was written with Ubuntu 18.04 as a base system but should work on your system even if you might have to change some paths.
Normally a varnishncsa.service file is created for systemd to run a default instance of it, these logs end up in /var/log/varnish. We're going to add our own parametrized service files that filter out specific hostnames.
Note that you will end up running multiple instances of varnishncsa, this hasn't been a problem for me as they are quite lightweight.
Let's get started.
Start by creating a new service file in /lib/systemd/system under the name varnishncsa-per-host@.service.
# cat /lib/systemd/system/varnishncsa-per-host\@.service [Unit] Description=Varnish HTTP accelerator log daemon for Host %I Documentation=https://supakeen.com/weblog/varnishncsa-per-host.html After=varnish.service [Service] Type=forking PIDFile=/run/varnishncsa/varnishncsa-%I.pid RuntimeDirectory=varnishncsa User=varnishlog Group=varnish ExecStart=/usr/bin/varnishncsa -a -q "ReqHeader ~ '^Host: %I'" -w /var/log/varnish/varnishncsa-%I.log -D -P /run/varnishncsa/varnishncsa-%I.pid ExecReload=/bin/kill -HUP $MAINPID PrivateDevices=true PrivateNetwork=true PrivateTmp=true ProtectHome=true ProtectSystem=full [Install] WantedBy=multi-user.target
In this file you see multiple %I format specifiers. These will be replaced by systemd. We are using varnishncsa's filtering on the Host header to find only the host we're interested in.
After we've created this file we can now create symlinks to enable this service for a hostname. Repeat this step per Host header you want to filter out.
# ln -s /lib/systemd/system/varnishncsa-per-host\@.service ./email@example.com # systemctl daemon-reload # systemctl enable firstname.lastname@example.org # systemctl start email@example.com # systemctl status firstname.lastname@example.org ● email@example.com - Varnish HTTP accelerator log daemon for Host supakeen.com Loaded: loaded (/lib/systemd/system/varnishncsa-per-host@.service; indirect; vendor preset: enabled) Active: active (running) since Sat 2020-04-25 09:26:34 UTC; 18min ago Docs: https://supakeen.com/weblog/varnishncsa-per-host.html Main PID: 20822 (varnishncsa) Tasks: 1 (limit: 1151) CGroup: /system.slice/system-varnishncsa\x2dper\firstname.lastname@example.org └─20822 /usr/bin/varnishncsa -a -q ReqHeader ~ '^Host: supakeen.com' -w /var/log/varnish/varnishncsa-supakeen.com.log -D -P /run/varnishncsa/varnishncsa-supakeen.com.pid Apr 25 09:26:34 var.tty.cat systemd: Starting Varnish HTTP accelerator log daemon for Host supakeen.com... Apr 25 09:26:34 var.tty.cat systemd: email@example.com: Failed to parse PID from file /run/varnishncsa/varnishncsa-supakeen.com.pid: Invalid argument Apr 25 09:26:34 var.tty.cat systemd: Started Varnish HTTP accelerator log daemon for Host supakeen.com. root@var:~#
And a new file now shows up in /var/log/varnish
# stat /var/log/varnish/varnishncsa-supakeen.com.log File: /var/log/varnish/varnishncsa-supakeen.com.log Size: 1946 Blocks: 8 IO Block: 4096 regular file Device: fc01h/64513d Inode: 260830 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 113/varnishlog) Gid: ( 115/ varnish) Access: 2020-04-25 09:26:34.904152413 +0000 Modify: 2020-04-25 09:34:43.891850255 +0000 Change: 2020-04-25 09:34:43.891850255 +0000 Birth: -
That's it! You can now use this file any way you want. Good luck.