P-A1: Building a simple HTTP ingress system

There are many reverse proxy/LB options available these days, but there is an argument to be made for OpenBSD’s Relayd. It comes in the default base installation, it is relatively simple, yet compelling and powerful enough for everyday use cases. And when needed, also offering tight integration with the PF firewall - for those edge cases which are almost impossible to solve with alternative solutions in simple terms.

This post is a glimpse into a multi node/tenant HTTP ingress system, routing and load balancing requests to various backend web servers while also offering TLS “acceleration” with SNI and automatic Letsencrypt certificate handling.

Desired end state:

flowchart LR

  C(web X)

  A@{ shape: cloud, label: requests }

  A r1@===>|tls| B((relayd))
  r1@{ animation: slow }

  B r2@--->|http| C
  r2@{ animation: slow }


  B r3@--->|http| D(web Y)
  r3@{ animation: slow }

TLS: Letsencrypt, acme-client, http-01

OpenBSD comes with its own builtin acme-client. It is dead simple, supports the http-01 challenge type and that’s it. Its spartan nature is not a limitation but its main feature and in many ways superior to alternatives such as certbot.

TLS: httpd serving http-01

For type http-01 to work, httpd must be running and serving the challenge response file on the same machine where the acme-client is also executed. For this the following /etc/httpd.conf can be used:

server "acme-challenge" {
    log style forwarded
    listen on ixl0 port 80
    location "/.well-known/acme-challenge/*" {
        root "/acme"
        request strip 2
    }
}

Enable and start httpd:

# rcctl enable httpd && rcctl start httpd

TLS: acme-client

The acme-client configuration is done in /etc/acme-client.conf. This file supports include statements. In the example below each domain name is configured in its own include file.

/etc/acme-client.conf:

authority letsencrypt {
	api url "https://acme-v02.api.letsencrypt.org/directory"
	account key "/etc/acme/letsencrypt-privkey.pem"
}

authority letsencrypt-staging {
	api url "https://acme-staging-v02.api.letsencrypt.org/directory"
	account key "/etc/acme/letsencrypt-staging-privkey.pem"
}

include "/etc/acme-client.d/domain-x.conf"
include "/etc/acme-client.d/domain-y.conf"
include "/etc/acme-client.d/domain-z.conf"

/etc/acme-client.d/domain-x.conf:

domain domain-x.tld {
    challengedir "/var/www/acme"
    alternative names { www.domain-x.tld }
    domain key "/etc/ssl/private/domain-x.tld.key"
    domain full chain certificate "/etc/ssl/domain-x.tld.crt"
    sign with letsencrypt
}

The configuration can be verified and tested with just executing (use -v for more details):

# acme-client -n

Then to issue LE cert for a given domain simply execute it with the domain handle:

# acme-client domain-x.tld

Cert renewal

To check and renew certificates which are about to expire a simple daily cron job can be configured for each domain handle.

Enter Relayd

As from now on Relayd will listen on both port 80 and 443 for incoming requests, httpd will require a minor change to /etc/httpd.conf, move it to localhost and port 8080. This is achieved by the following configuration change:

server "acme-challenge" {
    log style forwarded
    listen on lo0 port 8080 # <--- changed
    location "/.well-known/acme-challenge/*" {
        root "/acme"
        request strip 2
    }
}

From this point Relayd must be also acme-client aware and needs to switch http-01 requests from Lestencrypt to the correct httpd service. http-01 challenge resquests will be forwarded to the dedicated acme httpd service as illustrated below:

flowchart LR

  C(acme)

  A@{ shape: cloud, label: http-01 }

  A r1@-->|http| B((relayd))
  r1@{ animation: slow }

  B r2@-->|http| C
  r2@{ animation: slow }

  B r3@-.- D(web X)

  B r4@-.- E(web Y)

The following stanza is used in /etc/relayd.conf to enable the flow depicted above:

pass quick path "/.well-known/acme-challenge/*" forward to <acme>

The net effect is that plain HTTP requests served on port 80 containing the challenge path will be unconditionally forwarded to the dedicated acme httpd service - and never routed to any backend instances. Thus all certificate handling will automatically take place on the Relayd node, enabling this way transparent TLS offloading/acceleration at the ingress point. This greatly simplifies the TLS setup for backend servers.

Full /etc/relayd.conf:

acme="127.0.0.1"
acme_port="8080"

table <acme> { $acme }

# load additional tables
include "/etc/relayd.d/tables/domain-x.tld.conf"
include "/etc/relayd.d/tables/domain-y.tld.conf"
include "/etc/relayd.d/tables/domain-z.tld.conf"

http protocol "http" {
        include "/etc/relayd.d/headers.conf"
        include "/etc/relayd.d/cache-control.conf"
        block
        # pass in Letsencrypt.org ACME validation with priority
        pass quick path "/.well-known/acme-challenge/*" forward to <acme>
        include "/etc/relayd.d/backends/domain-x.tld.conf"
        include "/etc/relayd.d/backends/domain-y.tld.conf"
        include "/etc/relayd.d/backends/domain-z.tld.conf"
}

http protocol "tls" {
        include "/etc/relayd.d/headers.conf"
        include "/etc/relayd.d/cache-control.conf"
        include "/etc/relayd.d/tls-headers.conf"
        block
        include "/etc/relayd.d/backends/domain-x.tld.conf"
        include "/etc/relayd.d/backends/domain-y.tld.conf"
        include "/etc/relayd.d/backends/domain-z.tld.conf"
        include "/etc/relayd.d/tls-keypairs.conf"
        tls { no tlsv1.0, ciphers "HIGH" }
}

relay "http" {
        listen on ixl0 port 80
        protocol "http"
        forward to <acme> port $acme_port check tcp
        include "/etc/relayd.d/forwards/http/domain-x.tld.conf"
        include "/etc/relayd.d/forwards/http/domain-y.tld.conf"
        include "/etc/relayd.d/forwards/http/domain-z.tld.conf"
}

relay "http-tls" {
        listen on ixl0 port 443 tls
        protocol "tls"
        include "/etc/relayd.d/forwards/http/domain-x.tld.conf"
        include "/etc/relayd.d/forwards/http/domain-y.tld.conf"
        include "/etc/relayd.d/forwards/http/domain-z.tld.conf"
}

In the Relayd configuration above a simple pattern is emerging. Each domain is configured in its own include file (the same pattern that has been used in acme-client.conf).

As Letsencrypt is exclusively issuing its certificate challenges over plain HTTP on port 80, Relayd must handle this only under http protocol "http" - there is no need to match this type again under http protocol "tls".

Relayd: TLS SNI

For TLS and SNI to work the following configuration is used.

Each domain’s key is defined in /etc/relayd.d/tls-keypairs.conf:

tls keypair "domain-x.tld"
tls keypair "domain-y.tld"
tls keypair "domain-z.tld"

Relayd will look for a matching key under the /etc/ssl/private directory. Example for domain-x.tld the key name is /etc/ssl/private/domain-x.tld.key.

Relayd: SNI - mapping domain names to web servers

The matching backend web server for a domain is defined by its Host header and matching table entry forward to <domain-x.tld>. Example for domain-x.tld in /etc/relayd.d/backends/domain-x.tld.conf:

pass request header "Host" value "www.domain-x.tld"   forward to <domain-x.tld>
pass request header "Host" value "domain-x.tld"       forward to <domain-x.tld>

The file above is just telling Relayd to match the HTTP request’s Host header against the correct <table> entry.

Relayd: forward request to web server

To complete the HTTP request after the “SNI” Host header mapping step, Relayd needs to relay and forward this request to the correct backend web server which is defined under the relay "http-tls" section in the corresponding “forwards” domain file, example /etc/relayd.d/forwards/http/domain-x.tld.conf:

forward to <domain-x.tld> port 8080 check tcp

The file above is declaring the TCP port and check to be used for health checking and referencing a final table <domain-x.tld> in this case, which is declared at the top of relayd.conf in /etc/relayd.d/tables/domain-x.tld.conf:

table <domain-x.tld> { 10.1.10.123 }

The table can contain one or more IP addresses for the same web server, thus enabling a pool of servers to handle requests for both failover and load distribution.

Conclusion

Relayd comes with many configuration options and deployment scenarios. This post was meant to be an introduction to some of Relayd’s capabilities and its use as a modern HTTP ingress system. The system described in this document is currently in production use with various enhancements. There are a number of other aspects such as security, operational tasks, monitoring, fault tolerance, high availability and additonal service discovery which did not fit into this document. These topics will be covered in the upcoming posts. Stay tuned.

References

relayd(8) relayd.conf(5) acme-client(1) acme-client.conf(5) httpd(8) httpd.conf(5)