TLS and wildcard certificates

If you are hosting a website at example.com, you'd need to do three things:

1. Get the domain example.com.
2. Host a website somewhere, and point your domain at that webpage.
3. Get a "TLS certificate," a cryptographic signature that stops others from pretending to be your website.

The third step is relatively new, having only been effectively required since 2015.

Recently, all three of these steps have become much simpler, with platforms like Cloudflare and Github Pages letting you host your sites while automatically getting TLS certificates.

How providers can offer free TLS certificates

When you use a managed website provider, usually you point one of your domains at their servers.

For example, you might make a DNS CNAME record pointing from example.com to user.github.io if you're using GitHub Pages.

When you tell GitHub about your domain, they're able to ask LetsEncrypt for a free certificate on your behalf:

1. GitHub tells LetsEncrypt, "Please give me a certificate for example.com"
2. LetsEncrypt responds, "Prove you own it, place this file at example.com/.well-known/acme-challenge"
3. GitHub puts that file at user.github.io/.well-known/acme-challenge
4. LetsEncrypt queries example.com, sees your CNAME record which says to check user.github.io, and they see GitHub's record.

In all, this process is relatively easy, and this general mechanism is how almost all site hosting providers generate certificates for free for you.

The problem with wildcard certificates

Regular TLS certificates only work with one domain. If your site has pages at example.com and app.example.com, GitHub would need to go through the process above twice.

Similarly, if your app lets users create profiles, it might have a separate domain for every user: user1.example.com, etc.

Wildcard certificates are of the form *.example.com, they can be used for any of th epages above - app.example.com, user1.example.com, etc.

The problem with wildcard certificates is that your hosting provider can't easily get them from LetsEncrypt. The process above (putting a file at a specific path) isn't possible, the only way to prove you control the entire wildcard domain is with another challenge:

DNS-01 and proving you own someone else's domain

To request a *.example.com certificate from letsencrypt, GitHub Pages would need to use DNS-01, which looks like this:

1. GitHub tells LetsEncrypt, "Please give me a certificate for *.example.com"
2. LetsEncrypt responds, "Prove you own it, create this DNS record: TXT _acme-challenge.example.com (some value)
3. The hard part: GitHub would need to create that record somehow
4. LetsEncrypt queries example.com, sees the record exists, and sends GitHub the *.example.com TLS certificate

Step 3 is the reason GitHub Pages doesn't support wildcard certificates - how could they create certificates within a domain that they don't own?

It's not impossible: Github could run their own DNS server

If you're building something like GitHub pages, and want your users to be able to create wildcard certificates, you'll need to run your own DNS server.

In GitHub's case, they want github.io to work with two kinds of requests:

1. Queries like "Which server is hosting the website for user.github.io?" should be responded to with "one of GitHub's webservers"
2. Queries like "What is the value of the TXT record at user.github.io? should be responded to with "whatever letsencrypt wants"

Take a look at this example, where you've made *.example.com point to user.github.io:

1. GitHub tells LetsEncrypt, "Please give me a certificate for *.example.com"
2. LetsEncrypt responds, "Prove you own it, create this DNS record: TXT _acme-challenge.example.com letsencryptpassword
3. GitHub tells its DNS server to answer DNS queries for TXT user.github.io with letsencryptpassword
4. LetsEncrypt queries the TXT record at _acme-challenge.example.com, is told "see user.github.io" by your CNAME record. It dutifully queries github.io's DNS server for the TXT record at user.github.io and sees  letsencryptpassword, so it gives GitHub the certificate for *.example.com

A sample implementation

We had to solve this problem for webapp.io, where users can add their own domains to host their preview environments.

For example, a user might want branch feature1 to be hosted at feature1.companyenv.com.

This means we need to request the wildcard certificate *.companyenv.com. This example uses the popular miekg/dns go package.

The DNS handler function from the previous section would look like this:

dns.HandleFunc(".", func(writer dns.ResponseWriter, msg *dns.Msg) {
	respMsg := &dns.Msg{
		Answer: []dns.RR{},
	}
	for _, q := range msg.Question {
		if q.Qtype == dns.TypeTXT {
			acmeResponse := getAcmeChallengeFor(q.Name)
			if acmeResponse == "" {
				respMsg.SetRcode(msg, dns.RcodeNameError)
				break
			} else {
				respMsg.SetReply(msg)
				respMsg.Answer = append(respMsg.Answer, &dns.TXT{
					Hdr: dns.RR_Header{
						Name:   q.Name,
						Rrtype: dns.TypeTXT,
						Class:  dns.ClassINET,
						Ttl:    0,
					},
					Txt: []string{acmeResponse},
				})
				respMsg.Authoritative = true
			}
		} else if q.Qtype == dns.TypeA || q.Qtype == dns.TypeAAAA || q.Qtype == dns.TypeCNAME {
			matches := pattern.FindStringSubmatch(q.Name)
			if matches != nil {
				respMsg.SetRcode(msg, dns.RcodeNameError)
				break
			}
				respMsg.SetReply(msg)
			respMsg.Answer = append(respMsg.Answer, &dns.CNAME{
				Hdr: dns.RR_Header{
					Name:   dns.Fqdn(q.Name),
					Rrtype: dns.TypeCNAME,
					Class:  dns.ClassINET,
					Ttl:    86400,
				},
				Target: dns.Fqdn("demotarget.webapp.io"),
			})
			respMsg.Authoritative = true
		}
	}
		err := writer.WriteMsg(respMsg)
	if err != nil {
		klog.Warning(err)
	}
	writer.Close()
})

To reiterate:

• If someone asks for a TXT record, answer with what letsencrypt is expecting
• If someone asks for a website, answer "check out this other page"

Wildcard certificates on-demand

There's one last related problem to solve: a wildcard certificate for *.companyenv.com cannot be used for api.branch1.companyenv.com

That's where on-demand TLS, a concept popularized by the Caddy web server, comes in.

The idea is to generate the TLS certificate when a user visits the page for the first time:

1. Someone requests api.branch1.companyenv.com
2. We use DNS-01 with LetsEncrypt and our DNS server to create a record for *.branch1.companyenv.com
3. The user sees the newly issued certificate, and can successfully view the site.

Combining it with our wildcard TLS certificate flow from the previous section, the final process would look like this:

Alternatives to LetsEncrypt

Some of the teams that use webapp.io request over a hundred unique certificates per day. LetsEncrypt has limits on how many certificates you can issue, for that reason, we split our requests across LetsEncrypt, ZeroSSL, and SSL.com. We've had a particularly good experience with ZeroSSL, where we've issued many thousand certificates.