letsencrypt cheatsheet for AlmaLinux

Home

1 letsencrypt for Apache and nginx

2 Apache

This section is sparse. Just adding some file locations.

2.1 Apache config files for letsencrypt

  • /etc/httpd/conf.d/*.conf
  • /etc/httpd/conf/httpd.conf
  • /etc/httpd/conf/ssl.conf #+ENDSRC

2.2 files

  • /var/log/httpd
  • /var/lib/httpd # contains the httpd state
  • /etc/httpd/conf/httpd.conf # the main config file!
  • /etc/httpd/conf/magic # contains the mime types supported
  • /etc/httpd/conf.d/php.conf
  • /etc/httpd/conf.d/welcome.conf
  • /etc/httpd/conf.d/userdir.conf
  • /etc/httpd/conf.d/autoindex.conf

see man 5 httpd.conf which states that the main config file for apache is:

  • /etc/httpd/conf/httpd.conf

2.3 other possible config files:

2.3.1 specifiying documentroot in httpd.conf

I would check

  • /etc/apache3/httpd.conf
  • /etc/apache2/apache2.conf
  • /etc/httpd/httpd.conf
  • /etc/htpd/conf/httpd.conf

on my centos system it was in:

-/etc/httpd

specifically

-/etc/httpd/

3 Securing apache.

Most of this was gleaned from this youtube link.

3.1 Keep apache up-to-date

dnf upgrade httpd

3.2 Securing apache and wordpress users

Each of these daemons should be run by a least privileged user, and NOT run by root.

  1. Make the user have nologin shell. Here are a few lines of /etc/passwd where you can see that these users have least privileges.
    zintis:x:1000:1000::/home/zintis:/bin/bash
    apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin
    mysql:x:27:27:MySQL Server:/var/lib/mysql:/sbin/nologin
    nginx:x:992:989:Nginx web server:/var/lib/nginx:/sbin/nologin
    

To change a user to use /sbin/nologin use the command:

  • chsh /sbin/nologin apache

OR

  • usermod -s /sbin/nologin apache
  1. Ensure all the files in /var/www are owned by apache with group apache:
    • cd /var/www ; chown apache:apache *
  2. change the home directory of each user. So for apache -> /var/www/html
    • sudo vipw and change the home directory to /var/www/html
    • or, usermod -m -d /var/www/html apache -d is an abbreviation for --home and changes the user's home directory

    Typically you would run these commands as a common sequence:

    • mkdir /home/new_home_directory
    • chown username:username /home/new_home_directory
    • chmod 700 /home/new_home_directory
    • usermod --home /home/new_home_directory username

3.3 Securing httpd.conf lines

You can change global directory security settings, or override these global settings for a particular directory with the AllowOverride yes Otherwise just the global settings will take effect.

Starting from DocumentRoot "/var/www/html"

<Directory>
Require ip 77.68.0.0/16 127.0.0.1/32   
Require ip 2001:db8::a00:20ff:fea7:ccea
Require ip 2001:db8:1:1::a
Require ip 2001:db8:2:1::/64
Require ip 2001:db8:3::/48
</Directory>

4 Linode SSL certificates

From this linode community posting:

"While Linode is not a Certificate Authority and doesn't offer SSLs for purchasing, you can use any 3rd party SSL certificate on a Linode server.

You can install an SSL on many different Linux configurations. Here are a few guides for the most popular distributions and web servers:

CentOS:SSL Certificates with Apache on CentOS 7

Ubuntu: SSL Certificates with Apache on Debian & Ubuntu

Each web server (Apache, Nginx) will need to be configured to work with your third-party certificates, depending on your websites and requirements.

5 Let's Encrpt and Certbot

One of the easiest methods for obtaining and installing an SSL certificate is through Let's Encrypt. The Electronic Frontier Foundation (EFF) has developed Certbot, a free and automated tool for creating Let's Encrypt SSLs and configuring them for different web server software. /Certbot runs on your Linode and can be downloaded on Certbot's website.

ssl-certificates-apache-on-centOS.png

Figure 1: SSL Certificate on Apache on CentOS 7

6 Log of letsencrpyt.org and certbot on apache

Here is the final output of my certbot --apache command:

certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: www.acme.org
2: zinux.acme.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/www.acme.org.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Keeping the existing certificate
Created an SSL vhost at /etc/httpd/conf.d/wordpress-le-ssl.conf
Deploying Certificate to VirtualHost /etc/httpd/conf.d/wordpress-le-ssl.conf
Redirecting vhost in /etc/httpd/conf.d/wordpress.conf to ssl vhost in /etc/httpd/conf.d/wordpress-le-ssl.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://www.acme.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Subscribe to the EFF mailing list (email: admin@acme.org).

IMPORTANT NOTES:
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

root@zinux ~[1037]$

6.1 root "history" of letsencrpt CA install

Run April 23, 2021 Key to this process are the snap install commands, and the certbot –apache command. See certbot.eff.org/docs

dnf install snapd
iptables -L
iptables -S
iptables -A INPUT -p tcp -m tcp --dport 80 -j LOG --log-prefix " Remote web " --log-level 6
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j LOG --log-prefix " Remote https " --log-level 6
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
systemctl enable --now snapd.socket
systemctl start snapd
snap install core
snap refresh core
dnf list --installed | grep certbot
dnf remove certbot
lst /var/lib/snapd/
ln -s /var/lib/snapd/snap /snap
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot
dnf install mod_ssl
which ssl_module
ssr httpd
apachectl -M | grep ssl
certbot --apache

Then you can check if your auto-renew is set up correctly. I found that crontab -l had zero entries, so I looked further with:

  • systemctl list-timers
    systemctl list-timers
    NEXT                         LEFT          LAST                         PASSED       UNIT                         ACTIVATES
    Fri 2021-04-23 11:40:00 EDT  6min left     Fri 2021-04-23 11:30:04 EDT  3min 37s ago sysstat-collect.timer        sysstat-collect.service
    Fri 2021-04-23 12:22:15 EDT  48min left    Fri 2021-04-23 11:22:14 EDT  11min ago    dnf-makecache.timer          dnf-makecache.service
    Fri 2021-04-23 16:23:55 EDT  4h 50min left Thu 2021-04-22 16:23:55 EDT  19h ago      systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.s>
    Fri 2021-04-23 19:01:00 EDT  7h left       n/a                          n/a          snap.certbot.renew.timer     snap.certbot.renew.servi>
    Sat 2021-04-24 00:00:00 EDT  12h left      Fri 2021-04-23 00:00:00 EDT  11h ago      unbound-anchor.timer         unbound-anchor.service
    Sat 2021-04-24 00:07:00 EDT  12h left      Fri 2021-04-23 00:07:00 EDT  11h ago      sysstat-summary.timer        sysstat-summary.service
    Sat 2021-04-24 06:18:24 EDT  18h left      Fri 2021-04-23 06:17:02 EDT  5h 16min ago dnf-automatic.timer          dnf-automatic.service
    
    7 timers listed.
    Pass --all to see loaded but inactive timers, too.
    root@zinux ~[1003]$
    

    Of note is the snap.certbot.renew.timer line. So looks good.

6.2 Monitoring certbot installation

The obvious one is looking at what certificates you have instaslled:

  • certbot certificates
    certbot certificates
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Found the following certs:
      Certificate Name: www.acme.org
        Serial Number: 381beada275ecacc9cb450278104dcacb28
        Key Type: RSA
        Domains: www.acme.org
        Expiry Date: 2021-07-22 14:05:28+00:00 (VALID: 89 days)
        Certificate Path: /etc/letsencrypt/live/www.acme.org/fullchain.pem
        Private Key Path: /etc/letsencrypt/live/www.acme.org/privkey.pem
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    root@zinux /etc/letsencrypt[1073]$
    
    

    Looks good.

6.3 SSL Labs to check installation

You can usewww.ssllabs.com to check your letsencrypt installation:

This failed for me. I believe it is because my certbot –apache gave me the option of creating a cert for www.acme.org which I took. However, my apache2 setup seems to only have acme.org (no www).

  1. fix by recreating a cert for acme.org?
  2. fix by getting apache to use www.acme.org?
  3. The domain name is an alias for a web site whose main name is different, but the alias was not included in the certificate by mistake

I am not sure which way to go…. hmmmmm I will try 1) and include an alias 3) so, I will delete my certbot certificate by running certbot --apache again, and replace it.

I am getting this:

root@zinux /var/www/html[1085]$
certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: www.acme.org
2: zinux.acme.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): c
Please specify --domains, or --installer that will help in domain names autodiscovery, or --cert-name for an existing certificate name.
root@zinux /var/www/html[1086]$
certbot --apache --domains acme.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Requesting a certificate for acme.org
Performing the following challenges:
http-01 challenge for acme.org
Waiting for verification...
Cleaning up challenges

We were unable to find a vhost with a ServerName or Address of acme.org.
Which virtual host would you like to choose?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: wordpress-le-ssl.conf          | Multiple Names        | HTTPS | Enabled
2: wordpress.conf                 | Multiple Names        |       | Enabled
3: ssl.conf                       |                       | HTTPS | Enabled
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): c
No vhost exists with servername or alias of acme.org. No vhost was selected. Please specify ServerName or ServerAlias in the Apache config.
No vhost selected

IMPORTANT NOTES:
 - Unable to install the certificate
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/acme.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/acme.org/privkey.pem
   Your certificate will expire on 2021-07-22. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again with the "certonly" option. To non-interactively
   renew *all* of your certificates, run "certbot renew"
root@zinux /var/www/html[1087]$

So, I then changed the apache httpd.conf file to specify my server correctly from www.acme.org to just acme.org. the restarted httpd, but still I got this:

root@zinux /etc/httpd/conf[1097]$
certbot --apache --domains acme.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/acme.org.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate for acme.org

We were unable to find a vhost with a ServerName or Address of acme.org.
Which virtual host would you like to choose?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: wordpress-le-ssl.conf          | Multiple Names        | HTTPS | Enabled
2: wordpress.conf                 | Multiple Names        |       | Enabled
3: ssl.conf                       |                       | HTTPS | Enabled
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): c
No vhost exists with servername or alias of acme.org. No vhost was selected. Please specify ServerName or ServerAlias in the Apache config.
No vhost selected

IMPORTANT NOTES:
 - Unable to install the certificate
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/acme.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/acme.org/privkey.pem
   Your certificate will expire on 2021-07-22. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again with the "certonly" option. To non-interactively
   renew *all* of your certificates, run "certbot renew"
root@zinux /etc/httpd/conf[1098]$

But???? I checked to browse to acme.org and now got a valid secure certificate?????

Lets check ssllab.com…

7 !%\(@#T^\) getting in trouble now:

trying to get my wordpress config, my apache config, my hostnamectl, my dns all to match acme.org, but now letsencrypt.org giving me this grieve:


certbot --apache --domain acme.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/acme.org.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate for acme.org
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: acme.org: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details.
root@zintis /etc/httpd/conf.d[1019]$

7.1 Discussion on deleting certs

I found this discussion on community.letsencrypt.org that points out the tricky nature of getting your certs straightened out.

Seems that the best solution is let them expire (after 90 days).

Another, slightly later discussion has some better info: here

The user got his error:

 usage:
certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

 Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
 it will attempt to use a webserver both for obtaining and installing the
 certificate.
 certbot: error: unrecognized arguments:

8 Commercial Certificate from GoDaddy

I caved and bought the EV certificate from GoDaddy that expires in 2 years, (April 2023). That has yet to be applied to acme.org, awaiting for confirmation.

9 Removing certificates from Apache config

From a 6 year old post:


The reason why the restart you mention fails is probably because you're
commenting out the certificate in a general config file and leave it in some
other file, probably in a vhost config. You could check for that with this: grep
-E '(domain[12]\.certificate|localhost\.crt) /etc/httpd/sites-enabled/*', which
will search for the strings: domain1.certificate, domain2.certificate and
localhost.crt in all files in /etc/httpd/sites-enabled/ directory. If it finds
either of these strings it will print out the relevant lines prepended with the
filename containing them. You can then proceed to comment out those lines. You
may have to change the certificate file path in order to provide another
certificate or comment out the whole HTTPS section if you only want plain HTTP.

Try to restart (or test configuration) after you're done. If you get through a
restart successfully, then the server has stopped using the certificates.


Another key bit of info on certs: "This file (localhost.crt) is generated with the script /etc/pki/tls/certs/make-dummy-cert on Red Hat, CentOS, and derivatives. You should not need it after you configure your web server to use the DigiCert certificate, but I wanted to add this to the conversation. – Aaron Copley Dec 2 '14 at 21:40 "

And more: You can regenerate the default self-signed certificate using OpenSSL:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/pki/tls/private/localhost.key -out /etc/ssl/certs/localhost.crt

  1. You can search the Apache config files and replace the self-signed cert with the new certificate.

This command will tell you which Apache config files reference the localhost.crt file:

grep -i -r localhost.crt etc/httpd An example output of the above command might be this:

/etc/httpd/conf.d/ssl.conf:SSLCertificateFile /etc/pki/tls/certs/localhost.crt That tells us to look in /etc/httpd/conf.d/ssl.conf and update the SSLCertificateFile, SSLCertificateKeyFile, and SSLCertificateChainFile to their new DigiCert certificate files.

Please feel free to call DigiCert support at 1-801-701-9600 if you have any problems or questions.

From this lin:

The files /etc/pki/tls/certs/localhost.crt and /etc/pki/tls/private/localhost.key are created by the postinstall script of the mod_ssl package. You can find the CentOS 7 spec file here: https://git.centos.org/rpms/httpd/blob/c7/f/SPECS/httpd.spec (check the other branches for different CentOS versions). Here is the script from CentOS 7:

%define sslcert %{_sysconfdir}/pki/tls/certs/localhost.crt
%define sslkey %{_sysconfdir}/pki/tls/private/localhost.key

%post -n mod_ssl
umask 077

if [ -f %{sslkey} -o -f %{sslcert} ]; then
   exit 0
fi

%{_bindir}/openssl genrsa -rand /proc/apm:/proc/cpuinfo:/proc/dma:/proc/filesystems:/proc/interrupts:/proc/ioports:/proc/pci:/proc/rtc:/proc/uptime 2048 > %{sslkey} 2> /dev/null

FQDN=`hostname`
if [ "x${FQDN}" = "x" -o ${#FQDN} -gt 59 ]; then
   FQDN=localhost.localdomain
fi

cat << EOF | %{_bindir}/openssl req -new -key %{sslkey} \
         -x509 -sha256 -days 365 -set_serial $RANDOM -extensions v3_req \
         -out %{sslcert} 2>/dev/null
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
${FQDN}
root@${FQDN}
EOF
So if you delete both localhost.key and localhost.crt, and do yum reinstall mod_ssl, then the postinstall script will recreate them for you.

(I'm posting this answer because searching how to recreate /etc/pki/tls/certs/localhost.crt leads me here, but as others stated, if you got a cert signed by a CA, you don't need the localhost.crt and localhost.key files any more.)

More:

 just had this same problem. There was no default localhost certificate for CentOS 8 so I ran this and it regenerated all the default certs.

/usr/libexec/httpd-ssl-gencerts
Hope it helps someone else.

10 Move to nginx

I ran the godaddy certificates for 1 year. In the meantime, I also switched to nginx. So now I have no inconsistencies on the web server name. It is www.acme.org (not acme.org)

10.1 Run certbot

root@zintis /etc/pki/tls[1011]$
certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): admin@acme.org

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: www.acme.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Requesting a certificate for www.acme.org

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/www.acme.org/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/www.acme.org/privkey.pem
This certificate expires on 2023-07-27.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for www.acme.org to /etc/nginx/conf.d/www.acme.org.conf
Congratulations! You have successfully enabled HTTPS on https://www.acme.org

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@zintis /etc/pki/tls[1012]$


10.2 Home