Update 2023-06-23 - This works without stunnel if you can get bind 9.19!

tls myTLS-DoT {
    ca-file "/etc/bind/ca.crt";
    key-file "/etc/bind/my_client.key";
    cert-file "/etc/bind/my_client.crt";
    remote-hostname "remote_server";
};
options {
        forwarders port 853 tls myTLS-DoT {
            1.2.3.4;
	};
};

I was looking to do DNS over TLS for my home environment, but I wanted to run the endpoints on cloud servers that I don't want others to use. My home IP changes very frequently and so I didn't want to do a whitelist approach, so my next thought was TLS client certificate auth.
I already use pihole for add filtering, so I wanted it to tie into that as well.
I came up with something that works, but is super convoluted. There is most likely a better way to do this, but I went through a few iterations and this worked.
The flow ended up looking like this:



I did this all on Rocky Linux 8/9 and Debian 11. The configs for Rocky Linux don't differ between 8/9, but do slightly on Debian 11 as bind is layed out differently.

Pi-hole server configuration

Rocky Linux

dnf install bind stunnel

Debian 11

apt install stunnel4 bind9

Configure bind

I changed/added a few things, but mostly defaults. Changed/added are in bold. This makes bind listen on port 5335 and forward all requests to 127.0.0.1 on port 853 TCP.

Rocky Linux

/etc/named.conf

options {
        listen-on port 5335 { 127.0.0.1; };
        listen-on-v6 port 5335 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { localhost; };

        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        forward only;
        forwarders {
                127.0.0.1 port 853;
        };

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";

        /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
        include "/etc/crypto-policies/back-ends/bind.config";
};

server 127.0.0.1 {
        tcp-only yes;
};

Debian 11

/etc/bind/named.conf.options

options {
        listen-on port 5335 { 127.0.0.1; };
        listen-on-v6 port 5335 { ::1; };
        directory "/var/cache/bind";
        allow-query     { localhost; };

        recursion yes;

        dnssec-enable no;
        dnssec-validation no;


        forward only;
        forwarders {
                127.0.0.1 port 853;
        };
};
server 127.0.0.1 {
        tcp-only yes;
};

systemctl enable named --now

Configure stunnel

You will need certificates for every pihole server involved. You can use Easy-RSA to set up something pretty quickly.
Once you initialize the pki and generate a CA, you can generate a client cert:

./easyrsa  build-client-full server_name nopass

Copy the pki/ca.crt, pki/issued/server_name.crt, and pki/private/server_name.key generated to /etc/stunnel/.
Generate a cert/key for nginx.

./easyrsa  build-client-full remote_server_name nopass

cat pki/issued/remote_server_name.crt ca.crt > remote_server_name_combined.crt

Copy the cert created with cat and the pki/private/remote_server_name.key to the remote server at /etc/nginx/.
Swap nobody for the setuid, setgid, and chown to stunnel4 for Debian 11 in the following commands/config.

mkdir /run/stunnel && chown nobody /run/stunnel

/etc/stunnel/stunnel.conf

sslVersion = TLSv1.3
setuid = nobody
setgid = nobody
socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
pid = /run/stunnel/stunnel.pid


[dns]
client = yes
accept = 127.0.0.1:853
connect = remote_server:853
#connect = another_server:853
#failover = rr
verifyChain = yes
CAFile = /etc/stunnel/ca.crt
cert = /etc/stunnel/pihole.crt
key = /etc/stunnel/pihole.key

Rocky Linux

systemctl enable stunnel --now

Debian 11

systemctl enable stunnel4 --now

Configure pi-hole

Modify the PIHOLE_DNS_1 variable to be localhost.

/etc/pihole/setupVars.conf

PIHOLE_DNS_1=127.0.0.1#5335

Tell pi-hole to reconfigure and select repair as the option.

pihole -r

Remote server configuration

Configure nginx

Rocky Linux

dnf install nginx

Debian 11

apt install nginx

Copy the ca.crt that you put in /etc/stunnel/ca.crt to your remote server in /etc/nginx/
Add this to the bottom of your nginx.conf. This requires the nginx-mod-stream nginx module.

/etc/nginx/nginx.conf

stream {
        upstream dns {
                zone dns 64k;
                server 127.0.0.1:5335;
        }

        server {
                listen 853 ssl;
                ssl_certificate /etc/nginx/remote_server_name_combined.crt;
                ssl_certificate_key /etc/nginx/remote_server_name.key;
                ssl_client_certificate /etc/nginx/ca.crt;
                ssl_verify_client on;
                ssl_session_timeout 5m;
                ssl_protocols TLSv1.3;
                ssl_prefer_server_ciphers off;
                ssl_session_cache shared:SSL2:10m;
                ssl_session_tickets off;
                proxy_pass dns;
        }
}

systemctl enable nginx --now

Configure unbound

Rocky Linux

dnf install unbound

Debian 11

apt install unbound

mkdir /var/log/unbound && chown unbound. /var/log/unbound

server:
    logfile: "/var/log/unbound/unbound.log"
    verbosity: 1
    log-queries: yes
    interface: 127.0.0.1
    port: 5335
    do-ip4: yes
    do-udp: yes
    do-tcp: yes
    do-ip6: yes
    prefer-ip6: no
    harden-glue: yes
    harden-dnssec-stripped: yes
    use-caps-for-id: no
    edns-buffer-size: 1232
    prefetch: yes
    num-threads: 1
    so-rcvbuf: 1m

systemctl enable unbound --now

Testing

Pi-hole server

Test stunnel -> nginx

dig whoami.akamai.net @127.0.0.1 -p 853 +tcp

Test bind -> stunnel

dig google.com @127.0.0.1 -p 5335

Test pihole -> bind

dig google.com @127.0.0.1

Remote server

Test unbound

dig whoami.akamai.net @127.0.0.1 -p 5335